Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
Use WebSocket on controller isolated agents (CHOO-3013) (#687)
Abel Dantas6ae882645cc1
diff --git a/artifacts.yaml b/artifacts.yamlindex a6090ab16..79f7ea373 100644--- a/artifacts.yaml+++ b/artifacts.yaml@@ -195,7 +195,8 @@ contracts:# window, so `accepts` stays at 1 on both sides. switch-core serves either# transport, and agent-runtime prefers the socket and falls back to the# stream and its beat against a server with no socket. Both go, and- # `accepts` rises to 8, once no client still connects over the stream.+ # `accepts` rises to 8, once `switch.agents.connected{transport:sse}` stays+ # at zero.artifacts:switch-core: { speaks: 8, accepts: 1 }agent-runtime: { speaks: 8, accepts: 1 }diff --git a/console/packages/agent-controller/README.md b/console/packages/agent-controller/README.mdindex ce93c1057..35ad52455 100644--- a/console/packages/agent-controller/README.md+++ b/console/packages/agent-controller/README.md@@ -15,12 +15,12 @@ runs, and the agent hosts do the running.An agent whose definition asks for `isolation: "isolated"` runs instead as an agenthost in a process of its own (the shared-host bundle with `--ensure-watch`, as Console-runs an agent on an SSH host). It hears its events through the relay, which serves it-the per-agent agent protocol (its own event stream, heartbeat and placements), and it-is not stopped when the controller exits: it reconnects when the controller is back.+runs an agent on an SSH host). It hears its events on the controller's **hub**, the+same one in-process agent hosts use, over a WebSocket on the relay's port, and it is+not stopped when the controller exits: it reconnects when the controller is back.Changing an agent's isolation restarts it the other way.-The controller holds **one** event stream to Switch for all of its agents and hands+The controller holds **one** connection to Switch, a WebSocket, for all of its agents and handseach shared agent's events to its agent host directly. Each agent host makes its calls to Switchthrough a **local relay** on a loopback port, which adds the controller'scredentials. No agent holds a Switch credential: each is given a token that only the@@ -158,24 +158,25 @@ session hosts it launches with `process.execPath` run as Node too.### The controller stream- Opens a connection (`POST /v1/controllers/{id}/connection`) with each agent's- cursor (the last sequence its agent host confirmed, or `"head"`), then attaches the- stream (`GET /v1/controllers/{id}/events`). The stream carries every bound agent's+ cursor (the last sequence its agent host confirmed, or `"head"`), then attaches a+ WebSocket to it (`/v1/controllers/{id}/connection/ws`). It carries every bound agent'sevents (`agent.event`, `agent.gap`, `agent.session_command`,`agent.approval_outcome`), its attachment and room membership (`agent.attached`,`agent.detached`, `agent.rooms`), and the management nudges (`assignment.changed`,`operation.pending`, `credential.revoked`).-- Beats the connection (`POST .../connection/beat`) every `heartbeat_interval_s`- (2 s) while the stream is attached, with each agent's confirmed cursor.+- Answers every `ping` Switch sends on the socket (each `heartbeat_interval_s`, 2 s)+ with a `pong` naming each agent's confirmed cursor: that pong is its heartbeat.- The open and every beat also carry `placements`: for each agent whose runningagent host has a session placed in a room, those rooms. It is thewhole current map each time (Switch replaces what it held), and agents with noplacement are left out, so a placement the agent host makes reaches Switch on thenext beat.-- A dropped stream is reattached to the same connection. A connection Switch no+- A dropped socket is reattached to the same connection. A connection Switch nolonger knows (`unknown_connection`, `stale_generation`, or any `evicted` but`taken_over`) is opened afresh, from the cursors as they stand. Reconnects back- off with jitter, and a stream silent for 45 s (Switch writes a keepalive every- 15 s) counts as dropped.+ off with jitter, except after close code 1012 (Switch restarting), which is+ retried within a second. A socket silent for 10 s (Switch pings every 2 s)+ counts as dropped.- `taken_over`, as a frame or a refusal, means another instance of this controlleropened the stream. This one stops its agents and exits with code `4`.- Each agent's events are handed to its agent host one at a time, in order, and only@@ -197,9 +198,18 @@ session hosts it launches with `process.execPath` run as Node too.with a token minted for that agent (`swlr_…`) as `SWITCH_API_TOKEN`. A token therelay did not mint is refused with `401`; while the controller is still starting,with `503`.-- It serves no event stream and no connection bookkeeping- (`GET /agents/{id}/events`, `POST /agents/{id}/connection/…` answer `404`): the+- It serves no event stream and no connection bookkeeping of the agent protocol+ (`GET /agents/{id}/events`, `POST /agents/{id}/connection/…` answer `410`): thecontroller holds each agent's connection to Switch itself.+- **The hub**, at `ws://127.0.0.1:<port>/hub` with the agent's relay token, named in+ the credentials file as `SWITCH_AGENT_HUB`. An agent host in a process of its own+ hears its events there: the hub sends each event, gap, room control and approval+ outcome as a request, and counts it handled when the agent host answers `done`, as+ it does for an agent host in the controller's process. The agent host states its+ sessions' rooms there too (`placements`). One agent host per agent: a newer one+ takes the hub over (close `4409`, the older stands down). When the controller+ stops, the hub closes with `1012` and the agent hosts reconnect when it is back,+ resuming after the last event they handled.- **Forwarded to Switch**, everything else under `/agents/{id}/…`,`/agent-sessions/…`, `/sessions/…`, `/version` and `/health`: operations, media,typing, history, session activity and approvals. The relay sends them with the@@ -229,7 +239,8 @@ session hosts it launches with `process.execPath` run as Node too.reboot, say): the agent host is launched again, once 15 s have passed since itwas last launched, so an agent host still coming up is not launched twice.- **Desired `running`, relay credentials rewritten** (the relay came back on- another port): a running agent host is restarted so it reads them.+ another port, or they were written before the hub): a running agent host is+ restarted so it reads them.- **Agent host failed or was taken over:** it is left down, and reported as`failed`. A new revision or an `agent.restart` brings it back. One that failedbecause the relay refused its token is relaunched once it has a new one.@@ -288,7 +299,7 @@ It holds:|---|---|| `controller.db` | SQLite: identity, cached assignment, per-agent applied revision and local failures, restart times, each agent's stream cursor, the relay's port, status seq. Everything except the identity can be rebuilt from the server. || `secrets/controller-credential` | The controller credential (see below). Absent when the credential is handed over with `--credential-stdin`. |-| `agents/<id>/credentials.json` | Each agent's relay endpoint and relay token, in the layout the shared host reads. No Switch credential. |+| `agents/<id>/credentials.json` | Each agent's relay endpoint, relay token and hub, in the layout the shared host reads. No Switch credential. || `agent hosts/<id>/` | Each agent's agent host state root: `watch.json`, `config.json`, `health.json` (what `status` reads), its journal, and `supervisor/failure.json` once it has failed for good. |An agent whose definition names no directory works indiff --git a/console/packages/agent-controller/package.json b/console/packages/agent-controller/package.jsonindex b793aed26..28147897a 100644--- a/console/packages/agent-controller/package.json+++ b/console/packages/agent-controller/package.json@@ -26,11 +26,13 @@"dependencies": {"@switch-console/agent-providers": "workspace:*","@switch-console/plugins": "workspace:*",+ "ws": "^8.21.3","zod": "^4.4.3"},"devDependencies": {"@sandboxaq/switch-agent-runtime": "workspace:*","@types/node": "^24.13.3",+ "@types/ws": "^8.18.2","@typescript/native-preview": "7.0.0-dev.20260609.1","tsdown": "^0.22.2","vitest": "^5.0.1"diff --git a/console/packages/agent-controller/src/agent-hub.ts b/console/packages/agent-controller/src/agent-hub.tsindex 02ffb720d..0d3877880 100644--- a/console/packages/agent-controller/src/agent-hub.ts+++ b/console/packages/agent-controller/src/agent-hub.ts@@ -140,6 +140,13 @@ export class AgentHub {return cursors;}+ /** Where Core attached the agent on the controller stream, while it is attached. */+ attachment(agentId: string): { fromSeq: number; rooms: string[] } | null {+ const agent = this.agents.get(agentId);+ if (!agent?.attached || agent.head === null) return null;+ return { fromSeq: agent.head, rooms: [...(agent.rooms ?? [])] };+ }+/** The agent's events flow on the controller stream and its agent host is taking them. */attached(agentId: string): boolean {return this.agents.get(agentId)?.host?.connected ?? false;diff --git a/console/packages/agent-controller/src/api.test.ts b/console/packages/agent-controller/src/api.test.tsindex b34f3b95e..6c7e56cc8 100644--- a/console/packages/agent-controller/src/api.test.ts+++ b/console/packages/agent-controller/src/api.test.ts@@ -9,11 +9,7 @@ import {} from './api';import { ConfigurationError } from './errors';import { silentLogger } from './log';-import {- type Assignment,- controllerBeatRequestSchema,- controllerConnectionRequestSchema,-} from './schemas';+import { type Assignment, controllerConnectionRequestSchema } from './schemas';import { FakeCore } from './testing/fake-core';let core: FakeCore;@@ -46,6 +42,7 @@ function client(credential = core.credential) {controllerId: core.controllerId,version: '0.1.0',tokens,+ openWebSocket: core.openWebSocket,}),};}@@ -131,7 +128,7 @@ describe('enroll', () => {});expect(result).toEqual({ controller_id: core.controllerId, credential: core.credential });const request = core.requests.at(-1)!;- expect(request.headers[PROTOCOL_HEADER.toLowerCase()]).toBe('1');+ expect(request.headers[PROTOCOL_HEADER.toLowerCase()]).toBe('2');expect(request.headers['idempotency-key']).toBeTruthy();});@@ -279,8 +276,8 @@ describe('errors', () => {});});-describe('the controller stream routes', () => {- it('opens a connection declaring itself and its cursors, beats it, and attaches the stream', async () => {+describe('the controller connection', () => {+ it('opens a connection declaring itself and its cursors, then attaches its socket', async () => {core.setAssignment(assignment);const { client: api } = client();const signal = new AbortController().signal;@@ -292,43 +289,36 @@ describe('the controller stream routes', () => {client_version: '0.1.0',cursors: { 'agent-1': 41, 'agent-2': 'head' },});- const connection = { connectionId: opened.connection_id, generation: opened.generation };- const stream = new AbortController();- const response = await api.openEvents(connection, stream.signal);- expect(response.headers.get('content-type')).toContain('text/event-stream');- await api.beat(connection, { 'agent-1': 42 }, signal);- expect(controllerBeatRequestSchema.parse(core.requests.at(-1)!.body)).toEqual({- connection_id: opened.connection_id,+ const { socket } = await api.openSocket({+ connectionId: opened.connection_id,generation: opened.generation,- cursors: { 'agent-1': 42 },});- stream.abort();- });-- it('reads the refusals that decide between reopening and stopping', async () => {- const { client: api } = client();- const signal = new AbortController().signal;- await expect(- api.beat({ connectionId: 'nobody', generation: 1 }, {}, signal)- ).rejects.toMatchObject({ status: 404, code: 'unknown_connection' });- const first = await api.openConnection({}, signal);- await api.openConnection({}, signal);- await expect(- api.beat({ connectionId: first.connection_id, generation: first.generation }, {}, signal)- ).rejects.toMatchObject({ status: 409, code: 'taken_over' });+ await new Promise((resolve) => socket.addEventListener('message', resolve, { once: true }));+ const attach = core.requests.at(-1)!;+ socket.close(1000);+ expect(attach.method).toBe('WS');+ expect(attach.path).toBe(`/v1/controllers/${core.controllerId}/connection/ws`);+ expect(Object.fromEntries(new URLSearchParams(attach.search))).toEqual({+ connection_id: opened.connection_id,+ generation: String(opened.generation),+ });+ expect(attach.headers.authorization).toBe('Bearer access-token-1');+ expect(attach.headers['switch-controller-protocol']).toBe('2');});it('reads a refusal in the agent bridge’s own envelope', async () => {const { client: api } = client();core.scripted.push({method: 'POST',- path: `/v1/controllers/${core.controllerId}/connection/beat`,+ path: `/v1/controllers/${core.controllerId}/connection`,+ status: 409,+ body: { detail: { code: 'taken_over', message: 'someone else' } },+ });+ await expect(api.openConnection({}, new AbortController().signal)).rejects.toMatchObject({status: 409,- body: { detail: { code: 'no_stream', message: 'open the stream' } },+ code: 'taken_over',+ message: 'someone else',});- await expect(- api.beat({ connectionId: 'c', generation: 1 }, {}, new AbortController().signal)- ).rejects.toMatchObject({ status: 409, code: 'no_stream', message: 'open the stream' });});});@@ -360,7 +350,7 @@ function statusReport() {return {seq: 1,observed_at: new Date().toISOString(),- controller: { version: '0.1.0', protocol: 1 as const, assignment_revision: 0 },+ controller: { version: '0.1.0', protocol: 2 as const, assignment_revision: 0 },machine: {platform: { os: 'linux', arch: 'x64', os_version: '6.1' },disk_free_bytes: 1,diff --git a/console/packages/agent-controller/src/api.ts b/console/packages/agent-controller/src/api.tsindex fe3c74993..154bdb2c4 100644--- a/console/packages/agent-controller/src/api.ts+++ b/console/packages/agent-controller/src/api.ts@@ -7,7 +7,6 @@ import {type Assignment,assignmentSchema,type ControllerConnection,- controllerBeatResponseSchema,controllerConnectionResponseSchema,credentialRotateResponseSchema,type EnrollRequest,@@ -32,6 +31,36 @@ export const CONTROLLER_CLIENT = 'switch-agent-controller';export type Fetch = typeof fetch;+/** What the controller needs of a WebSocket; Node's own satisfies it. */+export type SocketLike = Pick<+ WebSocket,+ 'addEventListener' | 'removeEventListener' | 'send' | 'close' | 'readyState'+>;++/** Opens a WebSocket with request headers, as Node's does and a browser's does not. */+export type OpenWebSocket = (url: string, headers: Record<string, string>) => SocketLike;++/** Node's WebSocket (Node 22 and newer), which takes request headers. */+export const nodeWebSocket: OpenWebSocket = (url, headers) => {+ const Socket = (+ globalThis as unknown as {+ WebSocket?: new (url: string, init: { headers: Record<string, string> }) => WebSocket;+ }+ ).WebSocket;+ if (!Socket)+ throw new ConfigurationError(+ `The agents controller needs Node 22 or newer to connect to Switch (this is ${process.version}, with no WebSocket).`+ );+ return new Socket(url, { headers });+};++/** The controller's socket, and how to say its token was refused. */+export type OpenedSocket = {+ socket: SocketLike;+ /** The socket was refused with a 401 that is not a revocation: the token went stale. */+ tokenRefused: () => void;+};+/*** A refusal from the server, read from its `{"error": {code, message,* retryable}}` envelope. A response without one is still an error: it gets a@@ -274,6 +303,7 @@ export class ControllerClient {/** This controller's version, declared when the stream connection opens. */version: string;tokens: AccessTokens;+ openWebSocket: OpenWebSocket;}) {}@@ -407,48 +437,24 @@ export class ControllerClient {}/**- * Proves the connection alive, and confirms how far each agent's host has- * read.+ * The socket that carries an open connection: its frames down, the+ * controller's pongs (its beat) up. A refusal arrives on it as a `refused`+ * frame, read by the caller.*/- async beat(- connection: { connectionId: string; generation: number },- cursors: Record<string, number>,- signal: AbortSignal- ): Promise<void> {- const response = await this.request(`${this.streamPath}/connection/beat`, {- method: 'POST',- body: {- connection_id: connection.connectionId,- generation: connection.generation,- cursors,- },- signal,- });- await parsed(response, controllerBeatResponseSchema);- }-- /** Attaches the controller stream to an open connection; the caller reads the body. */- async openEvents(- connection: { connectionId: string; generation: number },- signal: AbortSignal- ): Promise<Response> {+ async openSocket(connection: {+ connectionId: string;+ generation: number;+ }): Promise<OpenedSocket> {const query = new URLSearchParams({connection_id: connection.connectionId,generation: String(connection.generation),});- const response = await this.request(`${this.streamPath}/events?${query}`, {- method: 'GET',- headers: { Accept: 'text/event-stream' },- signal,+ const url = `${this.streamPath.replace(/^http/, 'ws')}/connection/ws?${query}`;+ const token = await this.deps.tokens.get();+ const socket = this.deps.openWebSocket(url, {+ [PROTOCOL_HEADER]: String(PROTOCOL_VERSION),+ Authorization: `Bearer ${token}`,});- if (!response.body)- throw new ControllerApiError(- response.status,- 'invalid_response',- 'The event stream opened with no body.',- true,- null- );- return response;+ return { socket, tokenRefused: () => this.deps.tokens.invalidate(token) };}}diff --git a/console/packages/agent-controller/src/controller.test.ts b/console/packages/agent-controller/src/controller.test.tsindex 9518d60b6..517e1e188 100644--- a/console/packages/agent-controller/src/controller.test.ts+++ b/console/packages/agent-controller/src/controller.test.ts@@ -3,8 +3,9 @@ import { createServer, type Server } from 'node:net';import { tmpdir } from 'node:os';import { join } from 'node:path';import { setTimeout as delay } from 'node:timers/promises';-import { type AgentBridgeEvent, SwitchEventStream } from '@sandboxaq/switch-agent-runtime';+import type { AgentBridgeEvent } from '@sandboxaq/switch-agent-runtime';import { callOperation, SESSION_SELECTOR_HEADERS } from '@sandboxaq/switch-agent-runtime/hosted';+import { openHubStream } from '@switch-console/agent-providers';import { afterEach, beforeEach, describe, expect, it } from 'vitest';import { type ControllerDeps, type ControllerExit, runController } from './controller';import { ConfigurationError } from './errors';@@ -69,6 +70,7 @@ function deps(server = core.url): ControllerDeps {runtime: runtime.build,locator: new FakeLocator(),fetch,+ openWebSocket: core.openWebSocket,log: silentLogger,dataDir: dir,workspacesFor: () => join(dir, 'workspaces'),@@ -83,7 +85,6 @@ function deps(server = core.url): ControllerDeps {streamIdleMs: 2_000,streamInitialBackoffMs: 10,streamMaxBackoffMs: 50,- relay: { heartbeatTtlMs: 6_000, heartbeatIntervalS: 2, sweepMs: 50, keepaliveMs: 15_000 },eventBufferLimit: 100,},};@@ -243,7 +244,7 @@ describe('runController', () => {await expect(fetch(`${credentials.endpoint}/version`)).rejects.toThrow();}, 20_000);- it('serves an isolated agent its events as its own stream through the relay, and moves it in-process when that changes', async () => {+ it('serves an isolated agent its events on the hub over the relay, and moves it in-process when that changes', async () => {const isolatedAgent = agent(1);isolatedAgent.definition.isolation = 'isolated';core.setAssignment({ revision: 1, agents: [isolatedAgent] });@@ -254,13 +255,12 @@ describe('runController', () => {const events: AgentBridgeEvent[] = [];const own = new AbortController();watchers.push(own);- new SwitchEventStream({+ openHubStream(credentials.hub)({creds: { agentId: 'agent-1', apiEndpoint: credentials.endpoint, token: credentials.token },connectionId: 'isolated-host',worker: null,scope: 'all',filter: 'addressed',- spawnCapable: true,rooms: [],onEvent: (event) => void events.push(event),onGap: () => {},@@ -273,7 +273,7 @@ describe('runController', () => {'the isolated host attached');core.pushEvent('agent-1', 1, addressed(1));- await waitFor(() => events.length === 1, 'the event on its own stream');+ await waitFor(() => events.length === 1, 'the event on the hub');await waitFor(() => store.cursors().get('agent-1') === 1, 'its cursor confirmed');core.setAssignment({ revision: 2, agents: [agent(2)] });@@ -328,6 +328,36 @@ describe('runController', () => {expect(await running).toBe('stopped');}, 20_000);+ it('restarts an agent host whose credentials predate the hub, so it comes back on the hub', async () => {+ const port = await freePort();+ core.setAssignment({ revision: 1, agents: [agent(1)] });+ store.saveRelayPort(port);+ store.saveAssignment(core.assignment, '"1"', '2026-01-01T00:00:00Z');+ store.recordApplied('agent-1', 1, '2026-01-01T00:00:00Z');+ await runtime.launch('agent-1', runtimeTemplate(), {+ isolation: 'isolated',+ restart: false,+ replaceIdentity: false,+ clearTakenOver: false,+ });+ // As a controller from before the hub wrote them: the relay and a token, no hub.+ await runtime.writeCredentials('agent-1', {+ endpoint: `http://127.0.0.1:${port}`,+ token: 'swlr_old',+ hub: '',+ });+ runtime.calls.length = 0;+ running = runController(deps(), stop.signal);+ await waitFor(() => runtime.launches('agent-1').length === 1, 'a restart');+ expect(runtime.launches('agent-1')[0]!.options.restart).toBe(true);+ expect(runtime.credentials.get('agent-1')).toMatchObject({+ endpoint: `http://127.0.0.1:${port}`,+ hub: `ws://127.0.0.1:${port}/hub`,+ });+ stop.abort();+ expect(await running).toBe('stopped');+ });+it('points a watcher at the relay’s new port, and restarts it, when its old one is taken', async () => {const port = await freePort();const blocker = createServer();@@ -346,6 +376,7 @@ describe('runController', () => {await runtime.writeCredentials('agent-1', {endpoint: `http://127.0.0.1:${port}`,token: 'swlr_old',+ hub: `ws://127.0.0.1:${port}/hub`,});runtime.calls.length = 0;running = runController(deps(), stop.signal);@@ -445,6 +476,7 @@ describe('runController', () => {await runtime.writeCredentials('agent-1', {endpoint: 'http://127.0.0.1:1',token: 'swlr_k',+ hub: 'ws://127.0.0.1:1/hub',});runtime.calls.length = 0;await core.stop();diff --git a/console/packages/agent-controller/src/controller.ts b/console/packages/agent-controller/src/controller.tsindex 75b9d788c..1c9e69bff 100644--- a/console/packages/agent-controller/src/controller.ts+++ b/console/packages/agent-controller/src/controller.ts@@ -1,12 +1,13 @@import type { OpenAgentStream } from '@switch-console/agent-providers';import { AgentHub } from './agent-hub';-import { AccessTokens, ControllerClient, type Fetch, isRevoked } from './api';+import { AccessTokens, ControllerClient, type Fetch, isRevoked, type OpenWebSocket } from './api';import { ConfigurationError } from './errors';+import { HubSocket } from './hub-socket';import { errorMessage, type Logger } from './log';import { processPendingOperations } from './operations';import { isSafeSegment } from './paths';import { definitionProblem, reconcile, type ReconcileDeps, startAgent } from './reconcile';-import { DEFAULT_RELAY_TIMING, LocalRelay, RELAY_TOKEN_PREFIX, type RelayTiming } from './relay';+import { LocalRelay, RELAY_TOKEN_PREFIX } from './relay';import { UpstreamForwarder } from './relay-forward';import type { AgentRuntime } from './runtime';import {@@ -41,7 +42,6 @@ export type ControllerTiming = {* every agent on the controller is offline while the stream is down.*/streamMaxBackoffMs: number;- relay: RelayTiming;/** The most events held per agent while its agent host is not taking them. */eventBufferLimit: number;};@@ -51,11 +51,10 @@ export const DEFAULT_TIMING: ControllerTiming = {statusPollMs: 5_000,statusMinGapMs: 1_000,defaultReportWithinS: 60,- // Three missed 15 s keepalives.- streamIdleMs: 45_000,+ // Five missed pings, at the 2 s Switch sends them.+ streamIdleMs: 10_000,streamInitialBackoffMs: 1_000,streamMaxBackoffMs: 8_000,- relay: DEFAULT_RELAY_TIMING,eventBufferLimit: 5_000,};@@ -69,6 +68,8 @@ export type ControllerDeps = {runtime: (openStream: (agentId: string) => OpenAgentStream, workspaces: string) => AgentRuntime;locator: ProviderLocator;fetch: Fetch;+ /** Opens the controller's connection to Switch. */+ openWebSocket: OpenWebSocket;log: Logger;/** Where disk space is measured and provider checks run. */dataDir: string;@@ -88,10 +89,11 @@ export type ControllerDeps = {export type ControllerExit = 'stopped' | 'revoked' | 'taken_over';/**- * Makes an agent's credentials file name the relay and a token it accepts:- * the token already in the file when it names the relay as it listens now, a- * new one otherwise. Resolves true when the file was (re)written, which a running- * agent host only reads when it starts.+ * Makes an agent's credentials file name the relay, its hub, and a token the+ * relay accepts: the token already in the file when it names both as they+ * listen now, a new one otherwise. Resolves true when the file was (re)written,+ * which a running agent host only reads when it starts, so it is restarted:+ * that is also what moves an agent host from before the hub onto it.*/export async function ensureRelayCredentials(agentId: string,@@ -99,16 +101,22 @@ export async function ensureRelayCredentials(): Promise<boolean> {const current = await deps.runtime.readCredentials(agentId);const endpoint = deps.relay.endpoint;- if (current && current.endpoint === endpoint && current.token.startsWith(RELAY_TOKEN_PREFIX)) {+ const hub = deps.relay.hubUrl;+ if (+ current &&+ current.endpoint === endpoint &&+ current.hub === hub &&+ current.token.startsWith(RELAY_TOKEN_PREFIX)+ ) {if (!deps.relay.isRegistered(agentId, current.token))deps.relay.register(agentId, current.token);return false;}const token = deps.relay.mint(agentId);- await deps.runtime.writeCredentials(agentId, { endpoint, token });+ await deps.runtime.writeCredentials(agentId, { endpoint, token, hub });deps.log.info('Wrote the agent’s relay credentials', {agentId,- why: current ? 'they named another endpoint or a Switch key' : 'it had none',+ why: current ? 'they named another endpoint, hub or a Switch key' : 'it had none',});return true;}@@ -195,6 +203,7 @@ export async function runController(controllerId: identity.controllerId,version: deps.version,tokens,+ openWebSocket: deps.openWebSocket,});const queue = new SerialQueue();const cached = store.cachedAssignment();@@ -224,14 +233,8 @@ export async function runController();const relay = new LocalRelay({log,- version: deps.version,- now: deps.now,- timing: timing.relay,- bufferLimit: timing.eventBufferLimit,- onCursor: (agentId, cursor) =>- store.saveCursor(agentId, cursor, new Date(deps.now()).toISOString()),- onChange: () => reporter?.request(),- sharedRoomFor: (agentId, sessionId) => hub.roomFor(agentId, sessionId),+ roomFor: (agentId, sessionId) => hub.roomFor(agentId, sessionId),+ hub: new HubSocket({ hub, log }),forwarder: new UpstreamForwarder({server: identity.server,auth: {@@ -242,31 +245,19 @@ export async function runController(log,}),});- /**- * The agents whose host runs in a process of its own: their events go- * through the relay, as their own stream. Every other agent's go to its host- * in this process, through the hub.- */- const isolated = new Set<string>();+ /** The agents whose saved cursor the hub has been given. */const placed = new Set<string>();- const delivery = (agentId: string) => (isolated.has(agentId) ? relay : hub);- /** Sends each agent's events the way its definition asks, moving its cursor when that changes. */+ /** Resumes each newly assigned agent where this controller last confirmed it. */const placeAgents = (held: Assignment) => {const cursors = store.cursors();for (const entry of held.agents) {const agentId = entry.agent_id;- const wanted = entry.definition.isolation === 'isolated';- if (placed.has(agentId) && wanted === isolated.has(agentId)) continue;+ if (placed.has(agentId)) continue;placed.add(agentId);- if (wanted) {- isolated.add(agentId);- hub.forget(agentId);- } else {- isolated.delete(agentId);- relay.forgetStreams(agentId);- }const cursor = cursors.get(agentId);- if (cursor !== undefined) delivery(agentId).setCursor(agentId, cursor);+ // Core may have attached the agent before the assignment that names it+ // was read: where it attached it is then where the hub already stands.+ if (cursor !== undefined && !hub.attachment(agentId)) hub.setCursor(agentId, cursor);}};if (assignment) placeAgents(assignment);@@ -326,7 +317,7 @@ export async function runController(store,runtime,providers,- attached: (agentId) => delivery(agentId).attached(agentId),+ attached: (agentId) => hub.attached(agentId),dataDir: deps.dataDir,workspacesDir: workspaces,version: deps.version,@@ -455,33 +446,29 @@ export async function runController(case 'evicted':return;case 'agent.event':- delivery(frame.data.agent_id).ingest(frame.data);+ hub.ingest(frame.data);return;case 'agent.gap':log.warn('Switch reports events an agent missed', {agentId: frame.data.agent_id,reason: frame.data.reason,});- delivery(frame.data.agent_id).gap(frame.data);+ hub.gap(frame.data);return;case 'agent.session_command':- delivery(frame.data.agent_id).sessionCommand(frame.data);+ hub.sessionCommand(frame.data);return;case 'agent.approval_outcome':- delivery(frame.data.agent_id).approvalOutcome(frame.data);+ hub.approvalOutcome(frame.data);return;case 'agent.attached':- delivery(frame.data.agent_id).attach(- frame.data.agent_id,- frame.data.from_seq,- frame.data.rooms- );+ hub.attach(frame.data.agent_id, frame.data.from_seq, frame.data.rooms);return;case 'agent.detached':- delivery(frame.data.agent_id).detach(frame.data.agent_id, frame.data.reason);+ hub.detach(frame.data.agent_id, frame.data.reason);return;case 'agent.rooms':- delivery(frame.data.agent_id).setRooms(frame.data.agent_id, frame.data.rooms);+ hub.setRooms(frame.data.agent_id, frame.data.rooms);return;case 'assignment.changed':void sync(`assignment.changed to revision ${frame.data.revision}`);@@ -522,9 +509,9 @@ export async function runController(cursors: () => {const cursors: Record<string, AgentCursor> = {};for (const entry of assignment?.agents ?? []) cursors[entry.agent_id] = 'head';- return { ...cursors, ...hub.cursors(), ...relay.cursors() };+ return { ...cursors, ...hub.cursors() };},- confirmed: () => ({ ...hub.cursors(), ...relay.cursors() }),+ confirmed: () => hub.cursors(),onOpened: (connection) =>log.info('Switch has these agents bound to this controller', {agents: connection.agents,@@ -532,13 +519,11 @@ export async function runController(onConnected: () => {log.info('Connected to Switch');hub.streamAttached();- relay.streamAttached();void sync('connected');void runOperations();},onDisconnected: () => {hub.setUpstream(false);- relay.setUpstream(false);},onFrame,signal: stop.signal,diff --git a/console/packages/agent-controller/src/detached-runtime.ts b/console/packages/agent-controller/src/detached-runtime.tsindex 4d2861109..c70b70954 100644--- a/console/packages/agent-controller/src/detached-runtime.ts+++ b/console/packages/agent-controller/src/detached-runtime.ts@@ -34,11 +34,12 @@ const LAUNCH_TIMEOUT_MS = 60_000;* runs an agent on an SSH host: a detached agent host from the agent-providers* shared-host bundle, in a state root of its own, driven through the files it* reads (`watch.json`, `config.json`) and observed through the files it- * writes (`health.json`, `supervisor/failure.json`). It reaches Switch, its- * event stream included, through the controller's relay.+ * writes (`health.json`, `supervisor/failure.json`). It makes its calls to+ * Switch through the controller's relay, and hears its events on the+ * controller's hub, over a WebSocket on the relay's port.** Being its own process, it is not stopped when the controller exits: it- * keeps retrying the relay, and picks up again when the controller is back.+ * keeps retrying the hub, and picks up again when the controller is back.* A restart is a stop and a start: the agent host is turned off, waited out,* and launched again from the new template.*/diff --git a/console/packages/agent-controller/src/hub-socket.ts b/console/packages/agent-controller/src/hub-socket.tsnew file mode 100644index 000000000..113440703--- /dev/null+++ b/console/packages/agent-controller/src/hub-socket.ts@@ -0,0 +1,194 @@+import { randomUUID } from 'node:crypto';+import type { IncomingMessage } from 'node:http';+import type { Duplex } from 'node:stream';+import type { SwitchEventStreamDeps } from '@sandboxaq/switch-agent-runtime';+import {+ type AgentEventStream,+ HUB_CLOSE,+ HUB_PROTOCOL,+ type HubClientMessage,+ hubClientMessageSchema,+ type HubServerMessage,+} from '@switch-console/agent-providers';+import { type RawData, type WebSocket, WebSocketServer } from 'ws';+import type { AgentHub } from './agent-hub';+import { errorMessage, type Logger } from './log';++/** How long an agent host has, once its socket is open, to say hello. */+const HELLO_TIMEOUT_MS = 10_000;+/** How long closing waits for the agent hosts to see the close before cutting them. */+const CLOSE_GRACE_MS = 500;++type Served = { socket: WebSocket; stream: AgentEventStream | null };++/**+ * The hub over a WebSocket, for agent hosts that run in a process of their+ * own: each is an agent host on the hub exactly as one in this process is+ * (`AgentHub.open`), with every callback the hub makes sent to it as a request+ * and resolved when it answers `done`. So an event counts as taken once the+ * agent host has handled it, wherever it runs.+ *+ * Served on the relay's loopback port at `/hub`; the relay authenticates the+ * upgrade with the agent's relay token and hands it here with the agent named.+ */+export class HubSocket {+ private readonly server = new WebSocketServer({ noServer: true });+ /** The agent host connected for each agent; a newer one takes the agent over. */+ private readonly served = new Map<string, Served>();++ constructor(+ private readonly deps: {+ hub: Pick<AgentHub, 'open'>;+ log: Logger;+ }+ ) {}++ accept(agentId: string, req: IncomingMessage, socket: Duplex, head: Buffer): void {+ this.server.handleUpgrade(req, socket, head, (ws) => this.serve(agentId, ws));+ }++ /**+ * Closes every agent host's socket as a restart (1012), so each reconnects+ * soon after the controller is back, and gives them a moment to read it.+ */+ async close(): Promise<void> {+ const closing = [...this.server.clients].map(+ (ws) =>+ new Promise<void>((resolve) => {+ if (ws.readyState === ws.CLOSED) return resolve();+ ws.once('close', () => resolve());+ ws.close(HUB_CLOSE.restarting, 'the agents controller is restarting');+ })+ );+ await Promise.race([+ Promise.all(closing),+ new Promise((resolve) => setTimeout(resolve, CLOSE_GRACE_MS).unref()),+ ]);+ for (const ws of this.server.clients) ws.terminate();+ this.served.clear();+ }++ private serve(agentId: string, ws: WebSocket): void {+ const { log } = this.deps;+ const lifetime = new AbortController();+ const waiting = new Map<number, { resolve: () => void; reject: (error: Error) => void }>();+ let nextId = 0;+ const entry: Served = { socket: ws, stream: null };++ const send = (message: HubServerMessage) => {+ if (ws.readyState === ws.OPEN) ws.send(JSON.stringify(message));+ };+ const request = (message: (id: number) => HubServerMessage) =>+ new Promise<void>((resolve, reject) => {+ if (lifetime.signal.aborted)+ return reject(new Error('The agent host is no longer connected.'));+ const id = ++nextId;+ waiting.set(id, { resolve, reject });+ send(message(id));+ });++ const helloTimer = setTimeout(+ () => ws.close(HUB_CLOSE.refused, 'no hello within 10 s'),+ HELLO_TIMEOUT_MS+ );+ helloTimer.unref();++ const hello = (message: HubClientMessage & { type: 'hello' }) => {+ clearTimeout(helloTimer);+ if (message.protocol !== HUB_PROTOCOL) {+ ws.close(+ HUB_CLOSE.refused,+ `hub protocol ${message.protocol} is not ${HUB_PROTOCOL}; restart the agent host`+ );+ return;+ }+ const previous = this.served.get(agentId);+ if (previous && previous.socket !== ws)+ previous.socket.close(+ HUB_CLOSE.takenOver,+ 'another agent host for this agent connected to the hub'+ );+ this.served.set(agentId, entry);+ const streamDeps: SwitchEventStreamDeps = {+ creds: { agentId, apiEndpoint: '', token: '' },+ connectionId: randomUUID(),+ scope: 'all',+ filter: 'addressed',+ rooms: [],+ ...(message.startCursor === null ? {} : { startCursor: message.startCursor }),+ worker: null,+ log,+ signal: lifetime.signal,+ onEvent: (event) => request((id) => ({ type: 'event', id, event })),+ onGap: (gap) => request((id) => ({ type: 'gap', id, gap })),+ onSessionCommand: (command) => request((id) => ({ type: 'session_command', id, command })),+ onApprovalOutcome: (outcome) =>+ request((id) => ({ type: 'approval_outcome', id, outcome })),+ onConnected: () => send({ type: 'connected' }),+ onDisconnected: ({ error }) => send({ type: 'disconnected', error }),+ onEvicted: () => {},+ };+ const stream = this.deps.hub.open(agentId, streamDeps);+ entry.stream = stream;+ if (Object.keys(message.placements).length)+ stream.replacePlacements(message.placements).catch((error: unknown) =>+ log.warn('Refused the placements an agent host reconnected with', {+ agentId,+ error: errorMessage(error),+ })+ );+ stream.start();+ log.info('An isolated agent host connected to the hub', {+ agentId,+ startCursor: message.startCursor,+ });+ };++ const receive = (raw: RawData) => {+ let message: HubClientMessage;+ try {+ message = hubClientMessageSchema.parse(JSON.parse(raw.toString()));+ } catch (error) {+ log.warn('Closed a hub connection that sent something the hub does not read', {+ agentId,+ error: errorMessage(error),+ });+ ws.close(HUB_CLOSE.refused, 'unreadable message');+ return;+ }+ if (message.type === 'hello') {+ if (entry.stream) ws.close(HUB_CLOSE.refused, 'a second hello');+ else hello(message);+ return;+ }+ if (!entry.stream) {+ ws.close(HUB_CLOSE.refused, 'hello first');+ return;+ }+ if (message.type === 'done') {+ const call = waiting.get(message.id);+ waiting.delete(message.id);+ if (message.error === undefined) call?.resolve();+ else call?.reject(new Error(message.error));+ return;+ }+ entry.stream.replacePlacements(message.placements).then(+ () => send({ type: 'result', id: message.id }),+ (error: unknown) => send({ type: 'result', id: message.id, error: errorMessage(error) })+ );+ };++ ws.on('message', receive);+ ws.on('close', () => {+ clearTimeout(helloTimer);+ lifetime.abort();+ for (const call of waiting.values())+ call.reject(new Error('The agent host disconnected from the hub.'));+ waiting.clear();+ if (this.served.get(agentId) === entry) this.served.delete(agentId);+ });+ ws.on('error', (error) =>+ log.warn('A hub connection failed', { agentId, error: errorMessage(error) })+ );+ }+}diff --git a/console/packages/agent-controller/src/main.ts b/console/packages/agent-controller/src/main.tsindex 21d23915a..b41b0d9e0 100644--- a/console/packages/agent-controller/src/main.ts+++ b/console/packages/agent-controller/src/main.ts@@ -1,7 +1,7 @@import { hostname } from 'node:os';import { parseArgs } from 'node:util';import packageJson from '../package.json' with { type: 'json' };-import { ControllerApiError, enroll, normalizeServerUrl } from './api';+import { ControllerApiError, enroll, normalizeServerUrl, nodeWebSocket } from './api';import { DEFAULT_TIMING, runController } from './controller';import { DetachedRuntime } from './detached-runtime';import { ConfigurationError, UsageError } from './errors';@@ -212,6 +212,7 @@ async function runCommand(args: string[]): Promise<number> {),locator: new PathProviderLocator(process.env.PATH),fetch,+ openWebSocket: nodeWebSocket,log,dataDir,workspacesFor: serverWorkspacesDir,diff --git a/console/packages/agent-controller/src/reconcile.test.ts b/console/packages/agent-controller/src/reconcile.test.tsindex c4209830e..d6e33c138 100644--- a/console/packages/agent-controller/src/reconcile.test.ts+++ b/console/packages/agent-controller/src/reconcile.test.ts@@ -55,7 +55,11 @@ function deps(): ReconcileDeps {ensureCredentials: async (agentId) => {if (runtime.credentials.get(agentId)?.endpoint === endpoint) return false;minted.push(agentId);- await runtime.writeCredentials(agentId, { endpoint, token: `swlr_${minted.length}` });+ await runtime.writeCredentials(agentId, {+ endpoint,+ token: `swlr_${minted.length}`,+ hub: `${endpoint.replace('http', 'ws')}/hub`,+ });return true;},forgetAgent: (agentId) => void forgotten.push(agentId),@@ -85,7 +89,11 @@ describe('reconcile', () => {it('starts a running agent: points it at the relay, launches its watcher', async () => {await reconcile(assignment(agent()), deps());expect(minted).toEqual(['agent-1']);- expect(runtime.credentials.get('agent-1')).toEqual({ endpoint: RELAY, token: 'swlr_1' });+ expect(runtime.credentials.get('agent-1')).toEqual({+ endpoint: RELAY,+ token: 'swlr_1',+ hub: `${RELAY.replace('http', 'ws')}/hub`,+ });const [launch] = runtime.launches();expect(launch!.options).toEqual({isolation: 'shared',diff --git a/console/packages/agent-controller/src/relay.test.ts b/console/packages/agent-controller/src/relay.test.tsindex 423c0bd5d..37d724fb3 100644--- a/console/packages/agent-controller/src/relay.test.ts+++ b/console/packages/agent-controller/src/relay.test.ts@@ -3,15 +3,12 @@ import { request } from 'node:http';import { tmpdir } from 'node:os';import { join } from 'node:path';import { setTimeout as delay } from 'node:timers/promises';-import {- type AgentBridgeEvent,- type ApprovalOutcome,- CONTRACTS,- type Eviction,- PlacementsRefusedError,- type SessionCommand,- SwitchEventStream,- type SwitchEventStreamDeps,+import type {+ AgentBridgeEvent,+ ApprovalOutcome,+ Eviction,+ SessionCommand,+ SwitchEventStreamDeps,} from '@sandboxaq/switch-agent-runtime';import {callOperation,@@ -19,10 +16,14 @@ import {fetchMediaToFile,SESSION_SELECTOR_HEADERS,} from '@sandboxaq/switch-agent-runtime/hosted';+import { type AgentEventStream, openHubStream } from '@switch-console/agent-providers';import { afterEach, beforeEach, describe, expect, it } from 'vitest';+import WebSocket from 'ws';+import { AgentHub } from './agent-hub';import { AccessTokens } from './api';+import { HubSocket } from './hub-socket';import { silentLogger } from './log';-import { LocalRelay, RELAY_AGENT_PROTOCOL } from './relay';+import { LocalRelay } from './relay';import { UpstreamForwarder } from './relay-forward';import type { AgentAssignment } from './schemas';import { FakeCore } from './testing/fake-core';@@ -45,7 +46,7 @@ function assigned(agentId: string): AgentAssignment {instructions: '',auto_approve: false,directory: null,- isolation: 'shared',+ isolation: 'isolated',},};}@@ -68,7 +69,6 @@ function message(sequence: number, addressed: boolean, room = 'room-a') {timestamp: sequence,},sequence,- ...(addressed ? { missed: { count: sequence, reason: null } } : {}),},};}@@ -82,16 +82,14 @@ async function waitFor(condition: () => boolean, what: string, timeoutMs = 15_00}let core: FakeCore;+let hub: AgentHub;let relay: LocalRelay;let token: string;let revokedCalls: number;let cursorsSaved: [string, number][];const stops: (() => void)[] = [];-beforeEach(async () => {- core = new FakeCore();- await core.start();- core.setAssignment({ revision: 1, agents: [assigned(AGENT), assigned('agent-2')] });+function newRelay(): LocalRelay {const tokens = new AccessTokens({fetch,server: core.url,@@ -100,11 +98,10 @@ beforeEach(async () => {now: Date.now,log: silentLogger,});- revokedCalls = 0;- cursorsSaved = [];- relay = new LocalRelay({+ return new LocalRelay({log: silentLogger,- version: '0.1.0',+ roomFor: (agentId, sessionId) => hub.roomFor(agentId, sessionId),+ hub: new HubSocket({ hub, log: silentLogger }),forwarder: new UpstreamForwarder({server: core.url,auth: {@@ -114,18 +111,26 @@ beforeEach(async () => {},log: silentLogger,}),+ });+}++beforeEach(async () => {+ core = new FakeCore();+ await core.start();+ core.setAssignment({ revision: 1, agents: [assigned(AGENT), assigned('agent-2')] });+ revokedCalls = 0;+ cursorsSaved = [];+ hub = new AgentHub({+ log: silentLogger,+ bufferLimit: 100,onCursor: (agentId, cursor) => cursorsSaved.push([agentId, cursor]),onChange: () => {},- sharedRoomFor: (agentId, sessionId) =>- agentId === 'agent-2' && sessionId === 'session-7' ? 'room-shared' : null,- now: Date.now,- timing: { heartbeatTtlMs: 6_000, heartbeatIntervalS: 2, sweepMs: 50, keepaliveMs: 15_000 },- bufferLimit: 100,});+ relay = newRelay();await relay.start(null);token = relay.mint(AGENT);- relay.streamAttached();- relay.attach(AGENT, 0, ['room-a', 'room-b']);+ hub.streamAttached();+ hub.attach(AGENT, 0, ['room-a', 'room-b']);relay.setReady();});@@ -135,37 +140,34 @@ afterEach(async () => {await core.stop();});-type Watcher = ReturnType<typeof watch>;+type Host = ReturnType<typeof host>;-/** The runtime's own protocol client, pointed at the relay as a watcher points it. */-function watch(overrides: Partial<SwitchEventStreamDeps> = {}) {+/** An agent host in a process of its own, as the shared daemon opens its stream: on the hub. */+function host(overrides: Partial<SwitchEventStreamDeps> = {}, as = token) {const seen = {events: [] as AgentBridgeEvent[],gaps: [] as Parameters<SwitchEventStreamDeps['onGap']>[0][],evictions: [] as Eviction[],commands: [] as SessionCommand[],outcomes: [] as ApprovalOutcome[],- released: [] as { roomId: string; sessionId: string | null }[],- rooms: [] as string[][],connected: 0,+ disconnected: [] as string[],};const controller = new AbortController();- const stream = new SwitchEventStream({- creds: { agentId: AGENT, apiEndpoint: relay.endpoint, token },- connectionId: 'watcher-connection',+ const stream: AgentEventStream = openHubStream(relay.hubUrl)({+ creds: { agentId: AGENT, apiEndpoint: relay.endpoint, token: as },+ connectionId: 'host-connection',worker: null,scope: 'all',filter: 'addressed',- spawnCapable: true,rooms: [],onEvent: (event) => void seen.events.push(event),onGap: (gap) => void seen.gaps.push(gap),onEvicted: (eviction) => void seen.evictions.push(eviction),onSessionCommand: (command) => void seen.commands.push(command),onApprovalOutcome: (outcome) => void seen.outcomes.push(outcome),- onRoomReleased: (released) => void seen.released.push(released),- onRooms: (rooms) => void seen.rooms.push(rooms),onConnected: () => void seen.connected++,+ onDisconnected: ({ error }) => void seen.disconnected.push(error),log: quiet,signal: controller.signal,...overrides,@@ -176,18 +178,14 @@ function watch(overrides: Partial<SwitchEventStreamDeps> = {}) {return { stream, seen, stop };}-async function connected(watcher: Watcher, times = 1): Promise<void> {- await waitFor(() => watcher.seen.connected >= times, 'the stream to connect');+async function connected(agentHost: Host, times = 1): Promise<void> {+ await waitFor(() => agentHost.seen.connected >= times, 'the hub to say connected');}-function caller(- watcher: Watcher,- overrides: Partial<CallerContext> = {},- sessionId: string | null = 'session-1'-): CallerContext {+function caller(overrides: Partial<CallerContext> = {}, sessionId: string | null = 'session-1') {return {identity: { endpoint: relay.endpoint, agentId: AGENT, token },- connectionId: 'watcher-connection',+ connectionId: 'host-connection',selector: sessionId? {[SESSION_SELECTOR_HEADERS.sessionId]: sessionId,@@ -198,9 +196,9 @@ function caller(room: null,mediaDir: '/nonexistent',cwd: '/',- deadConnection: (operation) => `dead: ${operation}`,+ deadConnection: (operation: string) => `dead: ${operation}`,...overrides,- };+ } satisfies CallerContext;}async function post(path: string, body: unknown, bearer = token): Promise<Response> {@@ -211,179 +209,86 @@ async function post(path: string, body: unknown, bearer = token): Promise<Respon});}-describe('the relay as the agent protocol, read by the real SwitchEventStream', () => {- it('serves switch-core’s revisions up to the socket, which the runtime still accepts', () => {- const core = CONTRACTS['agent-protocol']['switch-core'];- const runtime = CONTRACTS['agent-protocol']['agent-runtime'];- // The relay serves the event stream, not the socket of agent-protocol 8.- expect(RELAY_AGENT_PROTOCOL).toEqual({ speaks: 7, accepts: core.accepts });- expect(runtime.accepts).toBeLessThanOrEqual(RELAY_AGENT_PROTOCOL.speaks);- expect(RELAY_AGENT_PROTOCOL.accepts).toBeLessThanOrEqual(runtime.speaks);- });-- it('connects, then delivers addressed events in order, with their ids and missed counts', async () => {- const watcher = watch();- await connected(watcher);- // Switch keeps no record of its sessions, so the agent host says when it starts one.- expect(watcher.stream.announcesSessionStarts).toBe(true);- relay.ingest(message(1, true));- relay.ingest(message(2, false));- relay.ingest({- agent_id: AGENT,- seq: 3,- event: {- type: 'command',- room_id: 'room-a',- sequence: 3,- payload: { command: 'reset', args: '', user_id: 'u', user_name: 'U' },- },+/** The status a WebSocket upgrade to the hub is answered with, as a raw client sees it. */+function upgradeStatus(bearer: string, path = '/hub'): Promise<number> {+ return new Promise((resolve, reject) => {+ const socket = new WebSocket(`${relay.hubUrl.replace('/hub', path)}`, {+ headers: { Authorization: `Bearer ${bearer}` },});- relay.ingest(message(4, true, 'room-b'));- await waitFor(() => watcher.seen.events.length === 2, 'two addressed events');- expect(watcher.seen.events.map((event) => event.sequence)).toEqual([1, 4]);- expect(watcher.seen.events[0]).toMatchObject({- type: 'message',- room_id: 'room-a',- bridge_id: null,- missed: { count: 1, reason: null },+ socket.on('unexpected-response', (_req, res) => {+ resolve(res.statusCode ?? 0);+ socket.terminate();});- expect(watcher.seen.events[0]).not.toHaveProperty('agent_id');- expect(watcher.stream.position).toBe(4);- });-- it('delivers every event to a connection filtering nothing', async () => {- const watcher = watch({ filter: 'all', connectionId: 'everything' });- await connected(watcher);- relay.ingest(message(1, true));- relay.ingest(message(2, false));- await waitFor(() => watcher.seen.events.length === 2, 'both events');- });-- it('resumes from Last-Event-ID without replaying what was read, and opens at head', async () => {- const first = watch();- await connected(first);- relay.ingest(message(1, true));- relay.ingest(message(2, true));- await waitFor(() => first.seen.events.length === 2, 'the first two events');- first.stop();- relay.ingest(message(3, true));- relay.ingest(message(4, true));- const resumed = watch({ startCursor: 2 });- await waitFor(() => resumed.seen.events.length === 2, 'the events missed while away');- expect(resumed.seen.events.map((event) => event.sequence)).toEqual([3, 4]);- const fresh = watch({ connectionId: 'fresh' });- await connected(fresh);- relay.ingest(message(5, true));- await waitFor(() => fresh.seen.events.length === 1, 'only what came after opening');- expect(fresh.seen.events[0]!.sequence).toBe(5);+ socket.on('open', () => {+ resolve(101);+ socket.terminate();+ });+ socket.on('error', reject);});+}- it('confirms the cursor the watcher beats, and only that', async () => {- const watcher = watch();- await connected(watcher);- relay.ingest(message(1, true));- relay.ingest(message(2, true));- await waitFor(() => watcher.seen.events.length === 2, 'the events');- await waitFor(() => relay.cursors()[AGENT] === 2, 'the beat confirming sequence 2', 5000);- expect(cursorsSaved.at(-1)).toEqual([AGENT, 2]);- expect(relay.attached(AGENT)).toBe(true);+describe('the hub, over the relay’s WebSocket, read by openHubStream', () => {+ it('serves the hub on the relay’s loopback port', () => {+ expect(relay.hubUrl).toBe(`${relay.endpoint.replace('http', 'ws')}/hub`);});- it('answers the heartbeat as Switch does: 200, 404 unknown, 409 taken_over or no_stream', async () => {- const watcher = watch();- await connected(watcher);- const unknown = await post(`/agents/${AGENT}/connection/beat`, {- connection_id: 'nobody',- cursor: 0,- generation: 1,- });- expect(unknown.status).toBe(404);- expect(await unknown.text()).toContain('nobody is not open');- const stale = await post(`/agents/${AGENT}/connection/beat`, {- connection_id: 'watcher-connection',- cursor: 0,- generation: 999_999,- });- expect(stale.status).toBe(409);- expect((await stale.json()).detail.code).toBe('taken_over');- const unfenced = await post(`/agents/${AGENT}/connection/beat`, {- connection_id: 'watcher-connection',- cursor: 0,- });- expect((await unfenced.json()).detail.code).toBe('unfenced');- watcher.stop();- await delay(50);- const detached = await post(`/agents/${AGENT}/connection/beat`, {- connection_id: 'watcher-connection',- cursor: 0,- generation: null,+ it('connects, then delivers addressed events in order, each confirmed once handled', async () => {+ let release: () => void = () => {};+ const held = new Promise<void>((resolve) => (release = resolve));+ const seen: number[] = [];+ const agentHost = host({+ onEvent: async (event) => {+ seen.push(event.sequence!);+ if (event.sequence === 4) await held;+ },});- expect(detached.status).toBe(409);- });-- it('evicts the stream another client takes over, and the runtime stands down', async () => {- const first = watch();- await connected(first);- const second = watch();- await connected(second);- await waitFor(() => first.seen.evictions.length === 1, 'the first to be evicted');- expect(first.seen.evictions[0]!.code).toBe('taken_over');- relay.ingest(message(1, true));- await waitFor(() => second.seen.events.length === 1, 'the event on the new holder');- expect(first.seen.events).toEqual([]);+ await connected(agentHost);+ hub.ingest(message(1, true));+ hub.ingest(message(2, false));+ hub.ingest(message(4, true));+ hub.ingest(message(5, true));+ await waitFor(() => seen.length === 2, 'the events up to the held one');+ // The event being handled is not confirmed, and the next is not sent.+ expect(seen).toEqual([1, 4]);+ expect(hub.cursors()[AGENT]).toBe(2);+ release();+ await waitFor(() => seen.length === 3, 'the event after the held one');+ expect(seen).toEqual([1, 4, 5]);+ await waitFor(() => hub.cursors()[AGENT] === 5, 'the cursor to move past it');+ expect(cursorsSaved.at(-1)).toEqual([AGENT, 5]);});- it('closes a connection whose heartbeat lapses, with an evicted frame', async () => {- await relay.close();- relay = new LocalRelay({- log: silentLogger,- version: '0.1.0',- forwarder: { forward: async () => {} },- onCursor: () => {},- onChange: () => {},- sharedRoomFor: () => null,- now: Date.now,- timing: { heartbeatTtlMs: 300, heartbeatIntervalS: 2, sweepMs: 20, keepaliveMs: 15_000 },- bufferLimit: 100,- });- await relay.start(null);- token = relay.mint(AGENT);- relay.setReady();- const response = await fetch(- `${relay.endpoint}/agents/${AGENT}/events?connection_id=raw&scope=all&protocol=6`,- { headers: { Authorization: `Bearer ${token}`, Accept: 'text/event-stream' } }- );- const text = await response.text();- expect(text).toContain('event: connection_state');- expect(text).toContain('event: evicted');- expect(text).toContain('"code":"heartbeat_lapsed"');+ it('tells the agent host when the controller loses and regains Switch', async () => {+ const agentHost = host();+ await connected(agentHost);+ hub.setUpstream(false);+ await waitFor(() => agentHost.seen.disconnected.length === 1, 'disconnected');+ expect(agentHost.seen.disconnected[0]).toMatch(/reconnecting to Switch/);+ hub.streamAttached();+ hub.attach(AGENT, 0, ['room-a', 'room-b']);+ await connected(agentHost, 2);});it('tracks placements, names the session’s room on its calls, and routes room controls to it', async () => {- const watcher = watch();- await connected(watcher);- await watcher.stream.replacePlacements({ 'session-1': 'room-a' });- await waitFor(- () => watcher.seen.rooms.some((rooms) => rooms.includes('room-a')),- 'the subscription change naming room-a'- );-- const placed = await callOperation(caller(watcher), 'post_message', { body: 'hi' });+ const agentHost = host();+ await connected(agentHost);+ await agentHost.stream.replacePlacements({ 'session-1': 'room-a' });++ const placed = await callOperation(caller(), 'post_message', { body: 'hi' });expect(placed.isError).toBeFalsy();expect(placed.structuredContent).toMatchObject({ room_id: 'room-a' });const forwarded = core.requests.findLast((r) => r.path.endsWith('/ops/post_message'))!;expect(forwarded.headers['x-switch-room-id']).toBe('room-a');expect(forwarded.headers['x-switch-agent-id']).toBe(AGENT);expect(forwarded.headers['x-switch-session-id']).toBe('session-1');- expect(forwarded.headers['x-switch-connection-id']).toBeUndefined();expect(forwarded.headers.authorization).toMatch(/^Bearer access-token-/);- const unplaced = await callOperation(caller(watcher, {}, 'session-9'), 'post_message', {});+ const unplaced = await callOperation(caller({}, 'session-9'), 'post_message', {});expect(unplaced.structuredContent).toMatchObject({ room_id: null });- const bySoleRoom = await callOperation(caller(watcher, {}, null), 'post_message', {});+ const bySoleRoom = await callOperation(caller({}, null), 'post_message', {});expect(bySoleRoom.structuredContent).toMatchObject({ room_id: 'room-a' });- relay.sessionCommand({+ hub.sessionCommand({agent_id: AGENT,room_id: 'room-a',command: {@@ -402,32 +307,22 @@ describe('the relay as the agent protocol, read by the real SwitchEventStream',requesterName: 'Person',},});- await waitFor(() => watcher.seen.commands.length === 1, 'the room control');- expect(watcher.seen.commands[0]).toMatchObject({- sessionId: 'session-1',+ await waitFor(() => agentHost.seen.commands.length === 1, 'the room control');+ expect(agentHost.seen.commands[0]).toMatchObject({commandId: 'command-1',+ roomId: 'room-a',body: { type: 'session.reset' },- requesterName: 'Person',});- expect(watcher.seen.commands[0]).not.toHaveProperty('room_id');- expect(watcher.seen.commands[0]).not.toHaveProperty('agent_id');- relay.sessionCommand({- agent_id: AGENT,- room_id: null,- command: { commandId: 'command-2', origin: { roomId: 'room-b' } },- });await expect(- watcher.stream.replacePlacements({ 'session-1': 'room-elsewhere' })- ).rejects.toBeInstanceOf(PlacementsRefusedError);- await delay(50);- expect(watcher.seen.commands).toHaveLength(1);+ agentHost.stream.replacePlacements({ 'session-1': 'room-elsewhere' })+ ).rejects.toThrow(/not a member of room room-elsewhere/);});- it('passes approval outcomes to the watcher', async () => {- const watcher = watch();- await connected(watcher);- relay.approvalOutcome({+ it('passes approval outcomes to the agent host', async () => {+ const agentHost = host();+ await connected(agentHost);+ hub.approvalOutcome({agent_id: AGENT,outcome: {session_id: 'session-1',@@ -438,27 +333,16 @@ describe('the relay as the agent protocol, read by the real SwitchEventStream',answered_at: '2026-01-01T00:00:00Z',},});- await waitFor(() => watcher.seen.outcomes.length === 1, 'the outcome');- expect(watcher.seen.outcomes[0]).toMatchObject({ request_id: 'request-1', answer: 'yes' });- });-- it('tells a connection its room was taken by a sibling placing a session there', async () => {- const watcher = watch();- await connected(watcher);- await watcher.stream.replacePlacements({ 'session-1': 'room-a' });- const other = watch({ connectionId: 'other-connection' });- await connected(other);- await other.stream.replacePlacements({ 'session-2': 'room-a' });- await waitFor(() => watcher.seen.released.length === 1, 'room_released');- expect(watcher.seen.released[0]).toEqual({ roomId: 'room-a', sessionId: 'session-1' });+ await waitFor(() => agentHost.seen.outcomes.length === 1, 'the outcome');+ expect(agentHost.seen.outcomes[0]).toMatchObject({ request_id: 'request-1', answer: 'yes' });});it('passes gaps through in order, and a reset as a cursor reset', async () => {- const watcher = watch();- await connected(watcher);- relay.ingest(message(1, true));- await waitFor(() => watcher.seen.events.length === 1, 'the first event');- relay.gap({+ const agentHost = host();+ await connected(agentHost);+ hub.ingest(message(1, true));+ await waitFor(() => agentHost.seen.events.length === 1, 'the first event');+ hub.gap({agent_id: AGENT,from_sequence: 1,resumed_at: 10,@@ -466,15 +350,10 @@ describe('the relay as the agent protocol, read by the real SwitchEventStream',all_rooms: false,reason: 'events older than the retention window were dropped',});- relay.ingest(message(11, true));- await waitFor(() => watcher.seen.events.length === 2, 'the event after the gap');- expect(watcher.seen.gaps[0]).toMatchObject({- fromSequence: 1,- resumedAt: 10,- rooms: ['room-a'],- cursorReset: false,- });- relay.gap({+ hub.ingest(message(11, true));+ await waitFor(() => agentHost.seen.events.length === 2, 'the event after the gap');+ expect(agentHost.seen.gaps[0]).toMatchObject({ fromSequence: 1, resumedAt: 10 });+ hub.gap({agent_id: AGENT,from_sequence: 11,resumed_at: 2,@@ -482,59 +361,68 @@ describe('the relay as the agent protocol, read by the real SwitchEventStream',all_rooms: true,reason: 'the server restarted',});- await waitFor(() => watcher.seen.gaps.length === 2, 'the reset');- expect(watcher.seen.gaps[1]).toMatchObject({ resumedAt: 2, cursorReset: true });- relay.ingest(message(3, true));- await waitFor(() => watcher.seen.events.length === 3, 'numbering from the reset');- expect(relay.cursors()[AGENT]).toBe(2);+ await waitFor(() => agentHost.seen.gaps.length === 2, 'the reset');+ expect(agentHost.seen.gaps[1]).toMatchObject({ resumedAt: 2, cursorReset: true });+ hub.ingest(message(3, true));+ await waitFor(() => agentHost.seen.events.length === 3, 'numbering from the reset');+ await waitFor(() => hub.cursors()[AGENT] === 3, 'the cursor from the reset');});- it('drops what a reattached stream replays, and a gap behind what it holds', async () => {- const watcher = watch();- await connected(watcher);- relay.ingest(message(1, true));- relay.ingest(message(2, true));- relay.streamAttached();- relay.attach(AGENT, 0, ['room-a', 'room-b']);- relay.gap({- agent_id: AGENT,- from_sequence: 0,- resumed_at: 1,- rooms: ['room-a'],- all_rooms: false,- reason: 'dropped',+ it('a handler that fails leaves the event unconfirmed', async () => {+ const agentHost = host({+ onEvent: (event) => {+ if (event.sequence === 2) throw new Error('the journal is full');+ },});- relay.ingest(message(1, true));- relay.ingest(message(2, true));- relay.ingest(message(3, true));- await waitFor(() => watcher.seen.events.length === 3, 'the new event');- expect(watcher.seen.events.map((event) => event.sequence)).toEqual([1, 2, 3]);- expect(watcher.seen.gaps).toEqual([]);+ await connected(agentHost);+ hub.ingest(message(1, true));+ hub.ingest(message(2, true));+ await waitFor(() => hub.cursors()[AGENT] === 1, 'the first event confirmed');+ await delay(100);+ expect(hub.cursors()[AGENT]).toBe(1);});- it('tells a watcher reconnecting from before what the relay still holds that it missed events', async () => {- relay.setCursor('agent-2', 50);- const tokenTwo = relay.mint('agent-2');- const gaps: number[] = [];- const controller = new AbortController();- stops.push(() => controller.abort());- const behind = new SwitchEventStream({- creds: { agentId: 'agent-2', apiEndpoint: relay.endpoint, token: tokenTwo },- connectionId: 'behind',- worker: null,- scope: 'all',- filter: 'addressed',- rooms: [],- startCursor: 40,- onEvent: () => {},- onGap: (gap) => void gaps.push(gap.resumedAt ?? -1),- onEvicted: () => {},- log: quiet,- signal: controller.signal,+ it('reconnects when the controller restarts, resuming after what it handled', async () => {+ const agentHost = host();+ await connected(agentHost);+ hub.ingest(message(1, true));+ await waitFor(() => agentHost.seen.events.length === 1, 'the first event');+ const port = Number(new URL(relay.endpoint).port);+ await relay.close();+ await waitFor(() => agentHost.seen.disconnected.length >= 1, 'the restart close');++ // The controller comes back on the same port, its hub knowing nothing yet.+ hub = new AgentHub({+ log: silentLogger,+ bufferLimit: 100,+ onCursor: (agentId, cursor) => cursorsSaved.push([agentId, cursor]),+ onChange: () => {},});- behind.start();- await waitFor(() => gaps.length === 1, 'the gap');- expect(gaps[0]).toBe(50);+ relay = newRelay();+ await relay.start(port);+ relay.register(AGENT, token);+ relay.setReady();+ hub.streamAttached();+ hub.attach(AGENT, 0, ['room-a', 'room-b']);+ await connected(agentHost, 2);+ // Switch replays from the opened cursor; the agent host said where it was.+ hub.ingest(message(1, true));+ hub.ingest(message(2, true));+ await waitFor(() => agentHost.seen.events.length === 2, 'the new event');+ expect(agentHost.seen.events.map((event) => event.sequence)).toEqual([1, 2]);+ });++ it('a second agent host for the agent takes the hub over, and the first stands down', async () => {+ const first = host();+ await connected(first);+ const second = host();+ await connected(second);+ await waitFor(() => first.seen.evictions.length === 1, 'the first to be evicted');+ expect(first.seen.evictions[0]).toMatchObject({ code: 'taken_over' });+ hub.ingest(message(1, true));+ await waitFor(() => second.seen.events.length === 1, 'the event at the second');+ await delay(50);+ expect(first.seen.events).toEqual([]);});});@@ -544,19 +432,12 @@ describe('what the relay refuses', () => {});it('refuses a token it did not mint with 401, and asks to retry while starting', async () => {- const refused = await post(`/agents/${AGENT}/connection/beat`, {}, 'swlr_not-a-token');+ const refused = await post(`/agents/${AGENT}/ops/post_message`, {}, 'swlr_not-a-token');expect(refused.status).toBe(401);- const starting = new LocalRelay({- log: silentLogger,- version: '0.1.0',- forwarder: { forward: async () => {} },- onCursor: () => {},- onChange: () => {},- sharedRoomFor: () => null,- now: Date.now,- timing: { heartbeatTtlMs: 6_000, heartbeatIntervalS: 2, sweepMs: 50, keepaliveMs: 15_000 },- bufferLimit: 100,- });+ expect(await upgradeStatus('swlr_not-a-token')).toBe(401);+ expect(await upgradeStatus(token)).toBe(101);+ expect(await upgradeStatus(token, '/elsewhere')).toBe(404);+ const starting = newRelay();await starting.start(null);try {const early = await fetch(`${starting.endpoint}/agents/${AGENT}/ops`, {@@ -568,6 +449,18 @@ describe('what the relay refuses', () => {}});+ it('answers the agent connection routes of an agent host from before the hub with 410', async () => {+ for (const path of [`/agents/${AGENT}/connection/beat`, `/agents/${AGENT}/events`]) {+ const response = await fetch(`${relay.endpoint}${path}`, {+ method: path.endsWith('beat') ? 'POST' : 'GET',+ headers: { Authorization: `Bearer ${token}` },+ });+ expect(response.status, path).toBe(410);+ expect(JSON.stringify(await response.json())).toMatch(/\/hub/);+ }+ expect(core.requests.filter((r) => r.path.includes('/connection/'))).toEqual([]);+ });+it('refuses another agent’s routes, and never relays the management routes', async () => {const other = await fetch(`${relay.endpoint}/agents/agent-2/ops`, {headers: { Authorization: `Bearer ${token}` },@@ -612,11 +505,30 @@ describe('forwarding to Switch', () => {expect(forwarded.headers.authorization).toMatch(/^Bearer access-token-/);expect(forwarded.headers['x-switch-agent-id']).toBe(AGENT);expect(forwarded.headers['x-switch-room-id']).toBeUndefined();- expect(forwarded.headers['switch-controller-protocol']).toBe('1');+ expect(forwarded.headers['switch-controller-protocol']).toBe('2');expect(forwarded.body).toEqual({ row: 1 });});- it('names the room of a shared agent’s call from its host in the controller', async () => {+ it('names the room of a call from its session’s placement on the hub', async () => {+ // An agent host in the controller's own process states its placements to the hub directly.+ hub.attach('agent-2', 0, ['room-shared']);+ const inProcess = new AbortController();+ stops.push(() => inProcess.abort());+ await hub+ .open('agent-2', {+ creds: { agentId: 'agent-2', apiEndpoint: '', token: '' },+ connectionId: 'in-process',+ worker: null,+ scope: 'all',+ filter: 'addressed',+ rooms: [],+ onEvent: () => {},+ onGap: () => {},+ onEvicted: () => {},+ log: quiet,+ signal: inProcess.signal,+ })+ .replacePlacements({ 'session-7': 'room-shared' });const shared = relay.mint('agent-2');const response = await fetch(`${relay.endpoint}/agents/agent-2/ops/post_message`, {method: 'POST',diff --git a/console/packages/agent-controller/src/relay.ts b/console/packages/agent-controller/src/relay.tsindex f1adbeb0a..e9dc296a5 100644--- a/console/packages/agent-controller/src/relay.ts+++ b/console/packages/agent-controller/src/relay.ts@@ -1,14 +1,9 @@-import { createHash, randomBytes, randomInt } from 'node:crypto';+import { createHash, randomBytes } from 'node:crypto';import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http';import type { AddressInfo, Socket } from 'node:net';-import { z } from 'zod';+import type { Duplex } from 'node:stream';+import { HUB_PATH } from '@switch-console/agent-providers';import { errorMessage, type Logger } from './log';-import type {- AgentApprovalOutcomeFrame,- AgentEventFrame,- AgentGapFrame,- AgentSessionCommandFrame,-} from './schemas';/*** The controller's local relay: what each managed agent's host uses as@@ -17,52 +12,18 @@ import type {** Every call an agent host makes is forwarded to Switch with the controller's* access token, the agent named in `X-Switch-Agent-Id`, and the calling- * session's room in `X-Switch-Room-Id`.+ * session's room in `X-Switch-Room-Id`. No agent holds a Switch credential.*- * An isolated agent's host runs in a process of its own, so it also hears its- * events here: Switch delivers every agent's events on the one controller- * stream, and the relay serves an isolated agent's to it as its own per-agent- * stream, reproducing the agent protocol (`GET /agents/{id}/events` with its- * frames, ids and resume, the heartbeat, placements and room claims). A shared- * agent's host runs in the controller's process and hears its events from the- * `AgentHub` instead; the room of its calls comes from there too.+ * An agent host in a process of its own hears its events on the controller's+ * hub, at `/hub` on this same port (`HubSocket`). The relay serves no event+ * stream and no connection of its own: Switch holds one connection, the+ * controller's, for every agent on this machine.*/-/**- * The agent-protocol revisions the relay serves: switch-core's, up to the- * socket. Revision 8 moved the connection onto a WebSocket, which the relay- * does not serve; it serves the event stream and its beat, which the runtime- * falls back to against a server that speaks 7 or older.- */-export const RELAY_AGENT_PROTOCOL = { speaks: 7, accepts: 1 } as const;-/** From this revision a client names its connection incarnation on every beat and room request. */-const FENCED_PROTOCOL_REVISION = 2;-const APPROVAL_OUTCOME_PROTOCOL_REVISION = 4;-const SESSION_COMMAND_PROTOCOL_REVISION = 5;-const ROOM_RELEASED_PROTOCOL_REVISION = 6;-const MAX_CONNECTIONS_PER_AGENT = 32;-const MAX_JSON_BODY_BYTES = 256 * 1024;-/** Every token the relay mints starts with this, which tells it apart from a Switch credential. */export const RELAY_TOKEN_PREFIX = 'swlr_';-export type RelayTiming = {- /** A connection with no beat for this long is closed, as Switch closes one. */- heartbeatTtlMs: number;- /** What `connection_state` tells the client to beat on. */- heartbeatIntervalS: number;- sweepMs: number;- keepaliveMs: number;-};--export const DEFAULT_RELAY_TIMING: RelayTiming = {- heartbeatTtlMs: 6_000,- heartbeatIntervalS: 2,- sweepMs: 1_000,- keepaliveMs: 15_000,-};--/** Where a request that is not connection bookkeeping goes. */+/** Where a request that is not the hub goes. */export interface Forwarder {forward(req: IncomingMessage,@@ -71,77 +32,18 @@ export interface Forwarder {): Promise<void>;}+/** Where the hub's upgrades go, with the agent the token named. */+export interface HubEndpoint {+ accept(agentId: string, req: IncomingMessage, socket: Duplex, head: Buffer): void;+ close(): Promise<void>;+}+export type RelayDeps = {log: Logger;- /** The room of a call for a shared agent (one whose host runs in the controller), from the calling session. */- sharedRoomFor: (agentId: string, sessionId: string | null) => string | null;- version: string;+ /** The room of a call for an agent, from the calling session's placement. */+ roomFor: (agentId: string, sessionId: string | null) => string | null;forwarder: Forwarder;- /** An agent's confirmed cursor moved; the controller persists it and beats it upstream. */- onCursor: (agentId: string, cursor: number) => void;- /** Something `attached()` reads changed. */- onChange: () => void;- now: () => number;- timing: RelayTiming;- /** The most events held per agent for local resume; past it the oldest go, and say so. */- bufferLimit: number;-};--type Declaration = {- speaks: number | null;- accepts: number | null;- artifact: string | null;- version: string | null;-};--type LocalStream = {- res: ServerResponse;- generation: number;- /** The rooms the client was last told it covers. */- told: Set<string>;-};--type LocalConnection = {- id: string;- agentId: string;- scope: 'single' | 'all';- filter: 'all' | 'addressed';- spawnCapable: boolean;- /** The last sequence written out (or skipped); null until the agent's head is known. */- cursor: number | null;- /** The last sequence the client confirmed on a beat, never moving back. */- confirmed: number | null;- generation: number;- lastBeat: number;- stream: LocalStream | null;- rooms: Set<string>;- released: Map<string, string | null>;- declaration: Declaration;-};--type Buffered =- | { kind: 'event'; seq: number; type: string; roomId: string; notifiable: boolean; data: object }- | { kind: 'gap'; seq: number; data: Record<string, unknown> };--type AgentState = {- agentId: string;- /** Core attached the agent to the controller stream. */- attached: boolean;- /** The rooms the agent belongs to, as Core last said; null before it has. */- rooms: Set<string> | null;- /** The highest sequence known for this agent. */- head: number | null;- /** Below or at this, the relay holds nothing: a cursor here gets a gap. */- droppedThrough: number;- /** The cursor the controller resumes the agent from upstream. */- confirmed: number | null;- buffer: Buffered[];- /** The last time Core reset the agent's numbering (a Core restart). */- reset: { reason: string; rooms: string[] } | null;- connections: Map<string, LocalConnection>;- /** Each session's room, and the connection that placed it there. */- placements: Map<string, { roomId: string; owner: string }>;- overflowing: boolean;+ hub: HubEndpoint;};class HttpRefusal extends Error {@@ -153,47 +55,16 @@ class HttpRefusal extends Error {}}-function coded(status: number, code: string, message: string): HttpRefusal {- return new HttpRefusal(status, { code, message });-}--function notOpen(connectionId: string): HttpRefusal {- return new HttpRefusal(- 404,- `connection ${connectionId} is not open; reconnect and resume from your cursor`- );-}-function hashToken(token: string): string {return createHash('sha256').update(token).digest('hex');}-/** Whether a client filtering to what addresses the agent is sent this event, as Switch decides. */-export function isNotifiable(type: string, payload: Record<string, unknown>): boolean {- if (type === 'message') return payload.addressed === true;- if (type === 'room_join') return payload.listening === true;- if (type === 'command') return false;- return type.startsWith('task_');-}--const beatSchema = z.object({- connection_id: z.string().min(1),- cursor: z.number().int().nonnegative().default(0),- generation: z.number().int().nullish(),-});-const roomRequestSchema = z.object({- connection_id: z.string().min(1),- room_id: z.string().min(1),- takeover: z.boolean().default(false),- generation: z.number().int().nullish(),-});-const placementsSchema = z.object({- connection_id: z.string().min(1),- placements: z.record(z.string().min(1), z.string().min(1)),- generation: z.number().int().nullish(),-});--const LOCAL_ROUTE = /^\/agents\/([^/]+)\/(events|connection\/[^/]+)$/;+/**+ * The agent protocol's own connection routes, which an agent host from before+ * the hub still calls. Refused rather than forwarded: Switch refuses them to a+ * controller, and the agent host is restarted onto the hub.+ */+const CONNECTION_ROUTE = /^\/agents\/[^/]+\/(events|connection\/[^/]+)$/;/** The prefixes forwarded to Switch. Anything else, the management routes above all, stays here. */const FORWARDED = /^\/(agents\/[^/]+\/.+|agent-sessions\/.+|sessions\/.+|version|health)$/;/** `/agents/<segment>/...` routes whose segment is not an agent. */@@ -205,20 +76,11 @@ export class LocalRelay {private readonly sockets = new Set<Socket>();private readonly tokens = new Map<string, string>();private readonly tokenOf = new Map<string, string>();- private readonly agents = new Map<string, AgentState>();- private readonly connections = new Map<string, LocalConnection>();- /**- * Starts anywhere, as Switch's does, so a client still holding an- * incarnation from before a restart cannot name one this relay issues.- */- private incarnation = randomInt(2 ** 31);private ready = false;- private upstream = false;- private timers: ReturnType<typeof setInterval>[] = [];constructor(private readonly deps: RelayDeps) {}- /** Listens on 127.0.0.1: on `preferredPort` when it is free, so running watchers find it again. */+ /** Listens on 127.0.0.1: on `preferredPort` when it is free, so running agent hosts find it again. */async start(preferredPort: number | null): Promise<number> {const server = createServer((req, res) => {void this.handle(req, res).catch((error: unknown) => {@@ -230,6 +92,7 @@ export class LocalRelay {else res.destroy();});});+ server.on('upgrade', (req, socket, head) => this.upgrade(req, socket, head));server.on('connection', (socket) => {this.sockets.add(socket);socket.on('close', () => this.sockets.delete(socket));@@ -255,11 +118,6 @@ export class LocalRelay {}this.server = server;this.port = (server.address() as AddressInfo).port;- this.timers = [- setInterval(() => this.sweep(), this.deps.timing.sweepMs),- setInterval(() => this.keepalive(), this.deps.timing.keepaliveMs),- ];- for (const timer of this.timers) timer.unref();return this.port;}@@ -268,14 +126,18 @@ export class LocalRelay {return `http://127.0.0.1:${this.port}`;}+ /** Where an agent host in a process of its own hears its events. */+ get hubUrl(): string {+ if (!this.server) throw new Error('The relay is not listening.');+ return `ws://127.0.0.1:${this.port}${HUB_PATH}`;+ }+/**- * Stops serving. Streams are cut without an `evicted` frame: the watchers- * outlive the controller, and a dropped socket is what tells them to keep- * retrying until it is back.+ * Stops serving. The hub's agent hosts are told the controller is+ * restarting, and keep retrying until it is back: they outlive it.*/async close(): Promise<void> {- for (const timer of this.timers) clearInterval(timer);- this.timers = [];+ await this.deps.hub.close();for (const socket of this.sockets) socket.destroy();const server = this.server;this.server = null;@@ -284,7 +146,7 @@ export class LocalRelay {/*** Unknown tokens are told to retry (503) until this is called: right after- * a restart a watcher can reach the relay before the controller has read+ * a restart an agent host can reach the relay before the controller has read* back its token, and a 401 would stop it for good.*/setReady(): void {@@ -300,7 +162,6 @@ export class LocalRelay {const hash = hashToken(token);this.tokens.set(hash, agentId);this.tokenOf.set(agentId, hash);- this.agent(agentId);}/** A new token for the agent, replacing any it had. */@@ -314,263 +175,62 @@ export class LocalRelay {return this.tokenOf.get(agentId) === hashToken(token);}- /**- * Forgets what is held for the agent's own stream, keeping its token: its- * host now runs in the controller's process.- */- forgetStreams(agentId: string): void {- const agent = this.agents.get(agentId);- if (!agent) return;- for (const conn of agent.connections.values()) {- conn.stream?.res.destroy();- this.connections.delete(conn.id);- }- this.agents.delete(agentId);- this.deps.onChange();- }-- /** Forgets the agent: its token stops working and its streams are cut. */+ /** Forgets the agent: its token stops working. */unregister(agentId: string): void {const hash = this.tokenOf.get(agentId);if (hash) this.tokens.delete(hash);this.tokenOf.delete(agentId);- const agent = this.agents.get(agentId);- if (!agent) return;- for (const conn of agent.connections.values()) {- conn.stream?.res.destroy();- this.connections.delete(conn.id);- }- this.agents.delete(agentId);- this.deps.onChange();- }-- /** Where the controller resumes an agent from, as persisted. */- setCursor(agentId: string, cursor: number): void {- const agent = this.agent(agentId);- agent.confirmed = cursor;- agent.head = cursor;- agent.droppedThrough = cursor;- }-- /** Each agent's confirmed cursor, for the upstream beat and reopen. */- cursors(): Record<string, number> {- const cursors: Record<string, number> = {};- for (const agent of this.agents.values())- if (agent.confirmed !== null) cursors[agent.agentId] = agent.confirmed;- return cursors;- }-- /** The agent's events flow on the controller stream and its watcher is connected here. */- attached(agentId: string): boolean {- const agent = this.agents.get(agentId);- if (!agent || !this.upstream || !agent.attached) return false;- return [...agent.connections.values()].some((conn) => conn.stream && this.alive(conn));- }-- // -- What arrives on the controller stream ------------------------------------ setUpstream(connected: boolean): void {- if (this.upstream === connected) return;- this.upstream = connected;- this.deps.onChange();- }-- /**- * The controller stream (re)attached. Switch attaches every bound agent- * afresh on each stream, with `agent.attached`, so none counts as attached- * until it says so again.- */- streamAttached(): void {- for (const agent of this.agents.values()) agent.attached = false;- this.upstream = true;- this.deps.onChange();- }-- attach(agentId: string, fromSeq: number, rooms: string[]): void {- const agent = this.agent(agentId);- agent.attached = true;- this.setRooms(agentId, rooms);- if (agent.head === null) agent.head = fromSeq;- else if (fromSeq > agent.head) {- // Switch starts the agent past what the relay holds: whatever lay- // between is not coming, and a watcher behind it is told so.- agent.head = fromSeq;- agent.droppedThrough = Math.max(agent.droppedThrough, fromSeq);- }- for (const conn of agent.connections.values())- if (conn.cursor === null) {- conn.cursor = agent.head;- conn.confirmed ??= agent.head;- }- this.deps.log.info('Agent attached to the controller stream', { agentId, fromSeq });- this.pumpAll(agent);- this.deps.onChange();- }-- detach(agentId: string, reason: string): void {- const agent = this.agents.get(agentId);- if (!agent) return;- agent.attached = false;- this.deps.log.warn('Agent detached from the controller stream', { agentId, reason });- this.deps.onChange();}- /** The agent's membership changed: rooms it left stop being covered here. */- setRooms(agentId: string, rooms: string[]): void {- const agent = this.agent(agentId);- agent.rooms = new Set(rooms);- for (const conn of agent.connections.values()) {- for (const room of conn.rooms) if (!agent.rooms.has(room)) conn.rooms.delete(room);- this.pump(conn);- }- }-- /**- * A domain event: buffered for local resume, and written to every stream- * that covers it. One the relay already has (Switch replays from where the- * connection opened each time the stream reattaches) is dropped.- */- ingest(frame: AgentEventFrame): void {- const { agent_id: agentId, seq } = frame;- const data = { ...frame.event, sequence: seq };- const agent = this.agent(agentId);- if (agent.head !== null && seq <= agent.head) return;- agent.buffer.push({- kind: 'event',- seq,- type: data.type,- roomId: data.room_id,- notifiable: isNotifiable(data.type, data.payload),- data,- });- for (const conn of agent.connections.values())- if (conn.cursor === null) {- conn.cursor = seq - 1;- conn.confirmed ??= seq - 1;- }- agent.head = seq;- this.pumpAll(agent);- this.trim(agent);- }-- /**- * Switch could not serve the agent's cursor. A reset (Switch restarted, so- * its numbering went back: `all_rooms`, resuming below what the relay- * holds) clears what is held and goes to every stream now. A gap resuming- * past what the relay holds takes its place in the buffer, so each stream- * meets it in order. One resuming at or below it describes events the relay- * already has — a reattached stream starting from where the connection- * opened — and is not passed on.- */- gap(frame: AgentGapFrame): void {- const { agent_id: agentId, ...data } = frame;- const agent = this.agent(agentId);- const resumedAt = data.resumed_at;- if (resumedAt === undefined) {- for (const conn of agent.connections.values()) this.write(conn, 'gap', data);- return;- }- if (agent.head !== null && resumedAt <= agent.head && data.all_rooms !== true) {- this.deps.log.debug('Dropped a gap behind what the relay already holds', {- agentId,- resumedAt,- head: agent.head,- });- return;- }- if (agent.head !== null && resumedAt < agent.head) {- agent.buffer = [];- agent.head = resumedAt;- agent.droppedThrough = resumedAt;- agent.reset = { reason: data.reason, rooms: data.rooms ?? [] };- agent.confirmed = resumedAt;- this.deps.onCursor(agentId, resumedAt);- for (const conn of agent.connections.values()) {- if (conn.confirmed !== null) conn.confirmed = Math.min(conn.confirmed, resumedAt);- if (!conn.stream) continue;- this.write(conn, 'gap', data);- conn.cursor = resumedAt;- }- return;- }- agent.buffer.push({ kind: 'gap', seq: resumedAt, data });- agent.head = resumedAt;- this.pumpAll(agent);- this.trim(agent);- }+ // -- HTTP -------------------------------------------------------------------- /** A room control, for the session placed in its room. */- sessionCommand(frame: AgentSessionCommandFrame): void {- const { agent_id: agentId, command } = frame;- const origin = command.origin as { roomId?: unknown } | undefined;- const roomId = frame.room_id ?? (typeof origin?.roomId === 'string' ? origin.roomId : null);- const agent = this.agents.get(agentId);- const sessionId = agent && roomId ? this.sessionIn(agent, roomId) : null;- if (!agent || !sessionId) {- this.deps.log.warn(- 'Dropped a room control: no session of the agent is placed in its room here',- { agentId, roomId, commandId: command.commandId }- );- return;- }- const watchers = this.watchers(agent, SESSION_COMMAND_PROTOCOL_REVISION);- if (!watchers.length) {- this.deps.log.warn('Dropped a room control: the agent has no watcher connected', {- agentId,- roomId,- commandId: command.commandId,- });- return;- }- for (const conn of watchers) this.write(conn, 'session_command', { ...command, sessionId });+ /** The agent a request's bearer token names, or the refusal for one it does not. */+ private authenticate(req: IncomingMessage): string | HttpRefusal {+ const auth = req.headers.authorization ?? '';+ const token = auth.startsWith('Bearer ') ? auth.slice(7).trim() : '';+ const agentId = token ? this.tokens.get(hashToken(token)) : undefined;+ if (agentId) return agentId;+ if (!this.ready)+ return new HttpRefusal(503, 'The agents controller is starting; retry in a moment.');+ return new HttpRefusal(+ 401,+ 'This token is not one the agents controller issued, or its agent is no longer assigned to this machine.'+ );}- approvalOutcome(frame: AgentApprovalOutcomeFrame): void {- const { agent_id: agentId, outcome } = frame;- const agent = this.agents.get(agentId);- const watchers = agent ? this.watchers(agent, APPROVAL_OUTCOME_PROTOCOL_REVISION) : [];- if (!watchers.length) {- this.deps.log.warn(- 'An approval outcome arrived while the agent has no watcher connected; Switch sends it again until it is delivered',- { agentId, requestId: outcome.request_id }+ private upgrade(req: IncomingMessage, socket: Duplex, head: Buffer): void {+ const path = new URL(req.url ?? '/', 'http://relay.invalid').pathname;+ const refuse = (status: number, reason: string) => {+ socket.end(`HTTP/1.1 ${status} ${reason}\r\nConnection: close\r\nContent-Length: 0\r\n\r\n`);+ };+ if (path !== HUB_PATH) return refuse(404, 'Not Found');+ const agentId = this.authenticate(req);+ if (agentId instanceof HttpRefusal)+ return refuse(+ agentId.status,+ agentId.status === 503 ? 'Service Unavailable' : 'Unauthorized');- return;- }- for (const conn of watchers) this.write(conn, 'approval_outcome', outcome);+ this.deps.hub.accept(agentId, req, socket, head);}- // -- HTTP --------------------------------------------------------------------private async handle(req: IncomingMessage, res: ServerResponse): Promise<void> {const raw = req.url ?? '';if (!raw.startsWith('/') || raw.startsWith('//'))return this.send(res, 400, { detail: 'The relay serves origin-form paths only.' });const url = new URL(raw, 'http://relay.invalid');- const auth = req.headers.authorization ?? '';- const token = auth.startsWith('Bearer ') ? auth.slice(7).trim() : '';- const agentId = token ? this.tokens.get(hashToken(token)) : undefined;- if (!agentId) {- if (!this.ready)- return this.send(res, 503, {- detail: 'The agents controller is starting; retry in a moment.',- });- return this.send(res, 401, {- detail:- 'This token is not one the agents controller issued, or its agent is no longer assigned to this machine.',- });- }+ const agentId = this.authenticate(req);+ if (agentId instanceof HttpRefusal)+ return this.send(res, agentId.status, { detail: agentId.detail });try {- const local = LOCAL_ROUTE.exec(url.pathname);- if (local) {- const named = decodeURIComponent(local[1]!);- if (named !== agentId)- throw new HttpRefusal(403, `authenticated as agent ${agentId}, not ${named}`);- return await this.handleLocal(req, res, agentId, local[2]!, url);- }+ if (CONNECTION_ROUTE.test(url.pathname))+ throw new HttpRefusal(+ 410,+ `The agents controller no longer serves the agent connection; its agent hosts hear their events on ${HUB_PATH}. Restart the agent host.`+ );this.checkForwardable(url, raw, agentId);await this.deps.forwarder.forward(req, res, {agentId,- roomId: this.roomFor(agentId, req),+ roomId: this.deps.roomFor(agentId, header(req, 'x-switch-session-id')),path: `${url.pathname}${url.search}`,});} catch (error) {@@ -590,558 +250,6 @@ export class LocalRelay {throw new HttpRefusal(403, `authenticated as agent ${agentId}, not ${named}`);}- private async handleLocal(- req: IncomingMessage,- res: ServerResponse,- agentId: string,- route: string,- url: URL- ): Promise<void> {- if (route === 'events') {- if (req.method !== 'GET') throw new HttpRefusal(405, 'Method Not Allowed');- if (!(req.headers.accept ?? '').includes('text/event-stream'))- throw new HttpRefusal(- 406,- 'The agents controller relays the event stream only; ask for text/event-stream.'- );- return this.openStream(req, res, agentId, url);- }- if (req.method !== 'POST') throw new HttpRefusal(405, 'Method Not Allowed');- const body = await readJson(req);- switch (route) {- case 'connection/beat':- return this.send(res, 200, this.beat(agentId, parse(beatSchema, body)));- case 'connection/placements':- return this.send(res, 200, this.placements(agentId, parse(placementsSchema, body)));- case 'connection/subscribe':- return this.send(res, 200, this.subscribe(agentId, parse(roomRequestSchema, body)));- case 'connection/unsubscribe':- return this.send(res, 200, this.unsubscribe(agentId, parse(roomRequestSchema, body)));- default:- throw new HttpRefusal(404, 'Not Found');- }- }-- private openStream(req: IncomingMessage, res: ServerResponse, agentId: string, url: URL): void {- const query = url.searchParams;- const connectionId = query.get('connection_id');- if (!connectionId)- throw new HttpRefusal(- 400,- 'connection_id is required to open an event stream; generate a UUID and reuse it when reconnecting so the connection survives the drop'- );- const scope = query.get('scope') ?? 'single';- if (scope !== 'single' && scope !== 'all')- throw new HttpRefusal(400, `scope must be 'single' or 'all', got '${scope}'`);- const filter = query.get('filter') ?? 'all';- if (filter !== 'all' && filter !== 'addressed')- throw new HttpRefusal(400, `filter must be 'all' or 'addressed', got '${filter}'`);- const agent = this.agent(agentId);- const lastEventId = req.headers['last-event-id'];- const rawCursor =- (typeof lastEventId === 'string' && lastEventId) || query.get('start_from') || 'head';- let cursor: number | null;- if (rawCursor === 'head') cursor = agent.head;- else {- cursor = Number(rawCursor);- if (!Number.isSafeInteger(cursor))- throw new HttpRefusal(- 400,- `start_from must be 'head' or a sequence number, got '${rawCursor}'`- );- cursor = Math.max(cursor, 0);- }- const declaration = declarationOf(query);- if (- declaration.speaks !== null &&- declaration.accepts !== null &&- !(- declaration.accepts <= RELAY_AGENT_PROTOCOL.speaks &&- RELAY_AGENT_PROTOCOL.accepts <= declaration.speaks- )- )- throw new HttpRefusal(409, {- message: `agent-protocol ${declaration.accepts}-${declaration.speaks} does not overlap the ${RELAY_AGENT_PROTOCOL.accepts}-${RELAY_AGENT_PROTOCOL.speaks} the agents controller serves`,- contract: 'agent-protocol',- server: { version: null, ...RELAY_AGENT_PROTOCOL },- client: { speaks: declaration.speaks, accepts: declaration.accepts },- remedy: 'Update the agents controller, or the agent runtime, so their ranges overlap.',- });- const expectedRaw = query.get('expected_generation');- const expected = expectedRaw === null ? null : Number(expectedRaw);-- let conn = this.connections.get(connectionId);- if (conn && conn.agentId !== agentId) throw new HttpRefusal(409, notOpen(connectionId).detail);- if (conn && !this.alive(conn)) {- this.closeConnection(conn, null);- conn = undefined;- }- if (conn) {- if (expected !== null && expected !== conn.generation)- throw coded(- 409,- 'taken_over',- `connection ${connectionId} has been reopened since incarnation ${expected} and is now at ${conn.generation}; another client holds it, so this reattach was refused and the connection was left untouched`- );- if (conn.stream)- this.evict(- conn,- 'taken_over',- 'another stream attached to this connection and took it over'- );- } else {- if (agent.connections.size >= MAX_CONNECTIONS_PER_AGENT)- throw new HttpRefusal(- 409,- `agent ${agentId} already holds ${MAX_CONNECTIONS_PER_AGENT} connections; close one before opening another`- );- conn = {- id: connectionId,- agentId,- scope,- filter,- spawnCapable: false,- cursor: null,- confirmed: null,- generation: 0,- lastBeat: 0,- stream: null,- rooms: new Set(),- released: new Map(),- declaration,- };- this.connections.set(connectionId, conn);- agent.connections.set(connectionId, conn);- }- conn.scope = scope;- conn.filter = filter;- conn.spawnCapable = query.get('spawn_capable') === 'true';- conn.declaration = declaration;- conn.cursor = cursor;- conn.confirmed = cursor;- conn.lastBeat = this.deps.now();- conn.generation = ++this.incarnation;-- for (const roomId of (query.get('rooms') ?? '').split(',').filter(Boolean)) {- if (agent.rooms && !agent.rooms.has(roomId)) {- this.closeConnection(conn, null);- throw new HttpRefusal(403, `agent ${agentId} is not a member of room ${roomId}`);- }- this.claim(conn, roomId, true);- }-- res.writeHead(200, {- 'Content-Type': 'text/event-stream',- 'Cache-Control': 'no-cache',- Connection: 'keep-alive',- 'X-Accel-Buffering': 'no',- });- const stream: LocalStream = { res, generation: conn.generation, told: new Set(conn.rooms) };- conn.stream = stream;- const owner = conn;- res.on('close', () => {- if (owner.stream === stream) {- owner.stream = null;- this.deps.onChange();- }- });- this.write(conn, 'connection_state', {- connection_id: conn.id,- agent_id: agentId,- generation: conn.generation,- scope: conn.scope,- filter: conn.filter,- spawn_capable: conn.spawnCapable,- rooms: [...conn.rooms].sort(),- cursor: conn.cursor ?? 0,- protocol: RELAY_AGENT_PROTOCOL.speaks,- heartbeat_interval_seconds: this.deps.timing.heartbeatIntervalS,- // Switch keeps no record of an agent's sessions: the agent host says when it starts one.- announce_session_starts: true,- server: {- version: null,- contracts: { 'agent-protocol': { ...RELAY_AGENT_PROTOCOL } },- relay: { artifact: 'switch-agent-controller', version: this.deps.version },- },- client: declaration,- });- if (agent.reset && conn.cursor !== null && agent.head !== null && conn.cursor > agent.head) {- this.write(conn, 'gap', {- from_sequence: agent.head,- resumed_at: agent.head,- rooms: [...conn.rooms].sort(),- all_rooms: true,- reason: agent.reset.reason,- });- conn.cursor = agent.head;- conn.confirmed = agent.head;- }- this.pump(conn);- this.deps.onChange();- }-- private beat(- agentId: string,- body: z.infer<typeof beatSchema>- ): { ok: true; rooms: string[]; cursor: number } {- const conn = this.live(agentId, body.connection_id);- this.fence(conn, body.generation ?? null, 'heartbeat');- if (!conn.stream)- throw coded(- 409,- 'no_stream',- `connection ${conn.id} has no stream attached; reopen the event stream`- );- conn.lastBeat = this.deps.now();- const agent = this.agent(agentId);- const cursor = agent.head === null ? body.cursor : Math.min(body.cursor, agent.head);- if (conn.cursor !== null && cursor > conn.cursor) conn.cursor = cursor;- if (conn.confirmed !== null && cursor > conn.confirmed) conn.confirmed = cursor;- this.confirm(agent);- this.trim(agent);- return { ok: true, rooms: [...conn.rooms].sort(), cursor: conn.cursor ?? cursor };- }-- private placements(agentId: string, body: z.infer<typeof placementsSchema>) {- const conn = this.current(agentId, body.connection_id, body.generation ?? null);- const agent = this.agent(agentId);- const rooms = Object.values(body.placements);- for (const roomId of new Set(rooms))- if (agent.rooms && !agent.rooms.has(roomId))- throw new HttpRefusal(403, `agent ${agentId} is not a member of room ${roomId}`);- const twice = [...new Set(rooms.filter((room, index) => rooms.indexOf(room) !== index))];- if (twice.length)- throw new HttpRefusal(- 400,- `placements name room(s) ${twice.sort().join(', ')} for more than one session; one session of an agent acts in a room`- );- const released: { connection_id: string; room_id: string; session_id: string | null }[] = [];- const previousRooms = new Set<string>();- for (const [sessionId, placed] of agent.placements)- if (placed.owner === conn.id) {- previousRooms.add(placed.roomId);- agent.placements.delete(sessionId);- }- for (const [sessionId, roomId] of Object.entries(body.placements)) {- agent.placements.delete(sessionId);- for (const [other, placed] of agent.placements) {- if (placed.roomId !== roomId) continue;- agent.placements.delete(other);- const loser = this.connections.get(placed.owner);- if (loser && loser !== conn && loser.agentId === agentId) {- this.notifyReleased(loser, roomId, other);- released.push({ connection_id: loser.id, room_id: roomId, session_id: other });- }- }- agent.placements.set(sessionId, { roomId, owner: conn.id });- }- for (const roomId of new Set(rooms)) {- const evicted = this.claim(conn, roomId, true);- if (evicted && !released.some((r) => r.connection_id === evicted.id && r.room_id === roomId))- released.push({ connection_id: evicted.id, room_id: roomId, session_id: null });- }- for (const roomId of previousRooms) if (!rooms.includes(roomId)) conn.rooms.delete(roomId);- for (const other of agent.connections.values()) this.pump(other);- return {- ok: true,- placements: this.connectionPlacements(conn),- rooms: [...conn.rooms].sort(),- released,- };- }-- private subscribe(agentId: string, body: z.infer<typeof roomRequestSchema>) {- const conn = this.current(agentId, body.connection_id, body.generation ?? null);- const agent = this.agent(agentId);- if (agent.rooms && !agent.rooms.has(body.room_id))- throw new HttpRefusal(403, `agent ${agentId} is not a member of room ${body.room_id}`);- const departing =- conn.scope === 'single' ? [...conn.rooms].filter((r) => r !== body.room_id) : [];- const evicted = this.claim(conn, body.room_id, body.takeover);- for (const roomId of departing) conn.rooms.delete(roomId);- for (const other of agent.connections.values()) this.pump(other);- return {- ok: true,- rooms: [...conn.rooms].sort(),- evicted_connection_id: evicted?.id ?? null,- warning: evicted- ? `Room ${body.room_id} was being attended by connection ${evicted.id} of this agent. That connection no longer receives the room's messages; this one does.`- : null,- };- }-- private unsubscribe(agentId: string, body: z.infer<typeof roomRequestSchema>) {- const conn = this.current(agentId, body.connection_id, body.generation ?? null);- conn.rooms.delete(body.room_id);- for (const other of this.agent(agentId).connections.values()) this.pump(other);- return { ok: true, rooms: [...conn.rooms].sort() };- }-- // -- Connections -------------------------------------------------------------- private agent(agentId: string): AgentState {- let agent = this.agents.get(agentId);- if (!agent) {- agent = {- agentId,- attached: false,- rooms: null,- head: null,- droppedThrough: 0,- confirmed: null,- buffer: [],- reset: null,- connections: new Map(),- placements: new Map(),- overflowing: false,- };- this.agents.set(agentId, agent);- }- return agent;- }-- private alive(conn: LocalConnection): boolean {- return this.deps.now() - conn.lastBeat < this.deps.timing.heartbeatTtlMs;- }-- /** The agent's open connection by id, or the 404 Switch answers for one that is not. */- private live(agentId: string, connectionId: string): LocalConnection {- const conn = this.connections.get(connectionId);- if (!conn || conn.agentId !== agentId) throw notOpen(connectionId);- if (!this.alive(conn)) {- this.closeConnection(conn, 'heartbeat_lapsed');- throw notOpen(connectionId);- }- return conn;- }-- /** Refuses a request from a client that is not, or cannot show it is, the connection's holder. */- private fence(conn: LocalConnection, generation: number | null, what: string): void {- if (generation === null) {- if ((conn.declaration.speaks ?? 0) >= FENCED_PROTOCOL_REVISION)- throw coded(- 409,- 'unfenced',- `connection ${conn.id} is held by a client speaking agent-protocol ${conn.declaration.speaks}, which names the connection incarnation on every ${what}; this one named none`- );- return;- }- if (generation !== conn.generation)- throw coded(- 409,- 'taken_over',- `connection ${conn.id} was reopened since incarnation ${generation} and is now at ${conn.generation}; another client holds it, so this ${what} was refused`- );- }-- private current(agentId: string, connectionId: string, generation: number | null) {- const conn = this.live(agentId, connectionId);- this.fence(conn, generation, 'room request');- return conn;- }-- /** Takes a room for `conn`, from whichever sibling held it. Returns that sibling. */- private claim(conn: LocalConnection, roomId: string, takeover: boolean): LocalConnection | null {- const agent = this.agent(conn.agentId);- let evicted: LocalConnection | null = null;- for (const sibling of agent.connections.values()) {- if (sibling === conn || !sibling.rooms.has(roomId) || !this.alive(sibling)) continue;- if (!takeover)- throw new HttpRefusal(- 409,- `room ${roomId} is already held by connection ${sibling.id} for this agent; close it or pass takeover`- );- sibling.rooms.delete(roomId);- this.notifyReleased(sibling, roomId, this.placedBy(agent, sibling, roomId));- evicted = sibling;- }- conn.released.delete(roomId);- conn.rooms.add(roomId);- return evicted;- }-- private notifyReleased(conn: LocalConnection, roomId: string, sessionId: string | null): void {- if ((conn.declaration.speaks ?? 0) < ROOM_RELEASED_PROTOCOL_REVISION) return;- if (sessionId !== null || !conn.released.has(roomId)) conn.released.set(roomId, sessionId);- }-- private placedBy(agent: AgentState, conn: LocalConnection, roomId: string): string | null {- for (const [sessionId, placed] of agent.placements)- if (placed.roomId === roomId && placed.owner === conn.id) return sessionId;- return null;- }-- private connectionPlacements(conn: LocalConnection): Record<string, string> {- const placements: Record<string, string> = {};- for (const [sessionId, placed] of this.agent(conn.agentId).placements)- if (placed.owner === conn.id) placements[sessionId] = placed.roomId;- return placements;- }-- private sessionIn(agent: AgentState, roomId: string): string | null {- for (const [sessionId, placed] of agent.placements)- if (placed.roomId === roomId) return sessionId;- return null;- }-- /**- * The room a forwarded call is made in: the calling session's placement when- * the call names a session, otherwise the single room of the connection it- * names. Null when neither says, and Switch answers as it does for a caller- * bound to no room.- */- roomFor(agentId: string, req: Pick<IncomingMessage, 'headers'>): string | null {- const agent = this.agents.get(agentId);- const sessionId = header(req, 'x-switch-session-id');- if (!agent || agent.connections.size === 0) return this.deps.sharedRoomFor(agentId, sessionId);- if (sessionId) return agent.placements.get(sessionId)?.roomId ?? null;- const connectionId = header(req, 'x-switch-connection-id');- const conn = connectionId ? agent.connections.get(connectionId) : undefined;- if (conn && conn.rooms.size === 1) return [...conn.rooms][0]!;- return null;- }-- private covers(conn: LocalConnection, roomId: string): boolean {- if (conn.rooms.has(roomId)) return true;- if (conn.scope === 'single') return false;- for (const sibling of this.agent(conn.agentId).connections.values())- if (sibling !== conn && sibling.rooms.has(roomId) && this.alive(sibling)) return false;- return true;- }-- /** The scope-`all` connections with a stream that take frames of this revision. */- private watchers(agent: AgentState, revision: number): LocalConnection[] {- return [...agent.connections.values()].filter(- (conn) =>- conn.scope === 'all' &&- conn.stream !== null &&- this.alive(conn) &&- (conn.declaration.speaks ?? 0) >= revision- );- }-- /** The lowest cursor any live connection has confirmed, never moving back. */- private confirm(agent: AgentState): void {- const confirmed = [...agent.connections.values()]- .filter((conn) => conn.confirmed !== null && this.alive(conn))- .map((conn) => conn.confirmed!);- if (!confirmed.length) return;- const lowest = Math.min(...confirmed);- if (agent.confirmed !== null && lowest <= agent.confirmed) return;- agent.confirmed = lowest;- this.deps.onCursor(agent.agentId, lowest);- }-- /** Drops what every live connection has confirmed, and the oldest past the limit. */- private trim(agent: AgentState): void {- const confirmed = [...agent.connections.values()]- .filter((conn) => this.alive(conn))- .map((conn) => conn.confirmed ?? -1);- const floor = confirmed.length ? Math.min(...confirmed) : -1;- let drop = 0;- while (drop < agent.buffer.length && agent.buffer[drop]!.seq <= floor) drop++;- let overflow = false;- if (agent.buffer.length - drop > this.deps.bufferLimit) {- drop = agent.buffer.length - this.deps.bufferLimit;- overflow = true;- }- if (drop > 0) {- agent.droppedThrough = Math.max(agent.droppedThrough, agent.buffer[drop - 1]!.seq);- agent.buffer.splice(0, drop);- }- if (overflow && !agent.overflowing)- this.deps.log.warn(- 'The relay is holding more events than it keeps for an agent whose watcher is not reading; the oldest are dropped, and the watcher is told when it reconnects',- { agentId: agent.agentId, limit: this.deps.bufferLimit }- );- agent.overflowing = overflow;- }-- // -- Writing ------------------------------------------------------------------ private write(conn: LocalConnection, event: string, data: unknown, id?: number): void {- const stream = conn.stream;- if (!stream) return;- const lines = id === undefined ? '' : `id: ${id}\n`;- stream.res.write(`${lines}event: ${event}\ndata: ${JSON.stringify(data)}\n\n`);- }-- private pumpAll(agent: AgentState): void {- for (const conn of agent.connections.values()) this.pump(conn);- }-- /** Writes what the connection is owed: room changes, then every event past its cursor. */- private pump(conn: LocalConnection): void {- const stream = conn.stream;- if (!stream) return;- for (const [roomId, sessionId] of conn.released)- this.write(conn, 'room_released', { room_id: roomId, session_id: sessionId });- conn.released.clear();- if (conn.rooms.size !== stream.told.size || [...conn.rooms].some((r) => !stream.told.has(r))) {- stream.told = new Set(conn.rooms);- this.write(conn, 'subscription_changed', {- rooms: [...conn.rooms].sort(),- reason: 'subscription updated',- });- }- // A session connection with no room yet must not read: it would skip,- // and so lose, every event before the room it is about to claim.- if (conn.cursor === null || (conn.scope === 'single' && conn.rooms.size === 0)) return;- const agent = this.agent(conn.agentId);- if (conn.cursor < agent.droppedThrough) {- this.write(conn, 'gap', {- from_sequence: conn.cursor,- resumed_at: agent.droppedThrough,- rooms: [...conn.rooms].sort(),- all_rooms: true,- reason: 'the agents controller no longer holds events this far back; re-read room context',- });- conn.cursor = agent.droppedThrough;- }- for (let index = firstAfter(agent.buffer, conn.cursor); index < agent.buffer.length; index++) {- const item = agent.buffer[index]!;- if (item.kind === 'gap')- this.write(conn, 'gap', { ...item.data, from_sequence: conn.cursor });- else if (this.covers(conn, item.roomId) && (conn.filter === 'all' || item.notifiable))- this.write(conn, item.type, item.data, item.seq);- conn.cursor = item.seq;- }- }-- private evict(conn: LocalConnection, code: string, reason: string): void {- const stream = conn.stream;- if (!stream) return;- this.write(conn, 'evicted', { code, reason, room_id: null });- conn.stream = null;- stream.res.end();- }-- /** Ends a connection; with a code, its open stream is told why first. */- private closeConnection(conn: LocalConnection, code: 'heartbeat_lapsed' | null): void {- if (code)- this.evict(conn, code, 'heartbeat lapsed; reopen the stream and resume from your cursor');- else conn.stream?.res.end();- conn.stream = null;- this.connections.delete(conn.id);- this.agents.get(conn.agentId)?.connections.delete(conn.id);- this.deps.onChange();- }-- private sweep(): void {- for (const conn of [...this.connections.values()])- if (!this.alive(conn)) {- this.deps.log.debug('Closing a local connection whose heartbeat lapsed', {- agentId: conn.agentId,- connectionId: conn.id,- });- this.closeConnection(conn, 'heartbeat_lapsed');- }- }-- private keepalive(): void {- for (const conn of this.connections.values()) conn.stream?.res.write(': keepalive\n\n');- }-private send(res: ServerResponse, status: number, body: unknown): void {const text = JSON.stringify(body);res.writeHead(status, {@@ -1157,59 +265,3 @@ function header(req: Pick<IncomingMessage, 'headers'>, name: string): string | nconst text = Array.isArray(value) ? value[0] : value;return text ? text : null;}--function declarationOf(query: URLSearchParams): Declaration {- const integer = (name: string) => {- const raw = query.get(name);- if (raw === null) return null;- const value = Number(raw);- return Number.isSafeInteger(value) ? value : null;- };- const speaks = integer('protocol');- return {- speaks,- accepts: integer('protocol_accepts') ?? speaks,- artifact: query.get('client'),- version: query.get('client_version'),- };-}--/** The index of the first buffered item past `cursor`. */-function firstAfter(buffer: Buffered[], cursor: number): number {- let low = 0;- let high = buffer.length;- while (low < high) {- const middle = (low + high) >> 1;- if (buffer[middle]!.seq <= cursor) low = middle + 1;- else high = middle;- }- return low;-}--async function readJson(req: IncomingMessage): Promise<unknown> {- const chunks: Buffer[] = [];- let size = 0;- for await (const chunk of req) {- size += (chunk as Buffer).length;- if (size > MAX_JSON_BODY_BYTES) throw new HttpRefusal(413, 'Request body too large.');- chunks.push(chunk as Buffer);- }- try {- return JSON.parse(Buffer.concat(chunks).toString('utf8'));- } catch {- throw new HttpRefusal(400, 'The request body is not JSON.');- }-}--function parse<S extends z.ZodType>(schema: S, body: unknown): z.infer<S> {- const result = schema.safeParse(body);- if (!result.success)- throw new HttpRefusal(- 422,- result.error.issues.map((issue) => ({- loc: ['body', ...issue.path.map(String)],- msg: issue.message,- }))- );- return result.data;-}diff --git a/console/packages/agent-controller/src/runtime.test.ts b/console/packages/agent-controller/src/runtime.test.tsindex 808b39d89..dc14be23f 100644--- a/console/packages/agent-controller/src/runtime.test.ts+++ b/console/packages/agent-controller/src/runtime.test.ts@@ -28,7 +28,11 @@ if (args[0] === '--probe') {}`;-const RELAY = { endpoint: 'http://127.0.0.1:43210', token: 'swlr_relay-token-placeholder' };+const RELAY = {+ endpoint: 'http://127.0.0.1:43210',+ token: 'swlr_relay-token-placeholder',+ hub: 'ws://127.0.0.1:43210/hub',+};const LAUNCH = {isolation: 'shared' as const,restart: false,@@ -128,10 +132,7 @@ async function fakeWatcher(root: string): Promise<ChildProcess> {describe('InProcessRuntime', () => {it('writes credentials the shared host reads, owner-only, outside the working directory', async () => {- await runtime.writeCredentials('agent-1', {- endpoint: 'http://127.0.0.1:43210',- token: 'swlr_relay-token-placeholder',- });+ await runtime.writeCredentials('agent-1', RELAY);const path = runtime.credentialsPath('agent-1');expect(path).toBe(join(dir, 'data', 'agents', 'agent-1', 'credentials.json'));expect(statSync(path).mode & 0o777).toBe(0o600);@@ -140,11 +141,9 @@ describe('InProcessRuntime', () => {SWITCH_API_ENDPOINT: 'http://127.0.0.1:43210',SWITCH_API_TOKEN: 'swlr_relay-token-placeholder',SWITCH_AGENT_ID: 'agent-1',+ SWITCH_AGENT_HUB: 'ws://127.0.0.1:43210/hub',});- expect(await runtime.readCredentials('agent-1')).toEqual({- endpoint: 'http://127.0.0.1:43210',- token: 'swlr_relay-token-placeholder',- });+ expect(await runtime.readCredentials('agent-1')).toEqual(RELAY);await runtime.deleteCredentials('agent-1');await runtime.deleteCredentials('agent-1');expect(await runtime.readCredentials('agent-1')).toBeNull();diff --git a/console/packages/agent-controller/src/runtime.ts b/console/packages/agent-controller/src/runtime.tsindex c815c4b73..f903cfb96 100644--- a/console/packages/agent-controller/src/runtime.ts+++ b/console/packages/agent-controller/src/runtime.ts@@ -76,8 +76,11 @@ export type LaunchOptions = {clearTakenOver: boolean;};-/** What an agent host reads to reach Switch: the controller's relay, and a token for it. */-export type RelayCredentials = { endpoint: string; token: string };+/**+ * What an agent host reads to reach Switch: the controller's relay, a token+ * for it, and the hub an agent host in a process of its own hears its events on.+ */+export type RelayCredentials = { endpoint: string; token: string; hub: string };/** Runs agent hosts one way: in this controller's process, or each in a process of its own. */export interface AgentRunner {@@ -286,8 +289,12 @@ export class InProcessRuntime implements AgentRuntime {const env = (JSON.parse(text) as { env?: Record<string, unknown> }).env ?? {};const endpoint = env.SWITCH_API_ENDPOINT;const token = env.SWITCH_API_TOKEN;+ // Written before the hub: an empty one, which no relay names, so the file is written again.+ const hub = typeof env.SWITCH_AGENT_HUB === 'string' ? env.SWITCH_AGENT_HUB : '';if (env.SWITCH_AGENT_ID !== agentId) return null;- return typeof endpoint === 'string' && typeof token === 'string' ? { endpoint, token } : null;+ return typeof endpoint === 'string' && typeof token === 'string'+ ? { endpoint, token, hub }+ : null;} catch {return null;}@@ -302,6 +309,7 @@ export class InProcessRuntime implements AgentRuntime {SWITCH_API_ENDPOINT: credentials.endpoint,SWITCH_API_TOKEN: credentials.token,SWITCH_AGENT_ID: agentId,+ SWITCH_AGENT_HUB: credentials.hub,},}));diff --git a/console/packages/agent-controller/src/schemas.test.ts b/console/packages/agent-controller/src/schemas.test.tsindex 08c967515..0b1b20b56 100644--- a/console/packages/agent-controller/src/schemas.test.ts+++ b/console/packages/agent-controller/src/schemas.test.ts@@ -4,9 +4,8 @@ import { describe, expect, it } from 'vitest';import { z } from 'zod';import {assignmentSchema,- controllerBeatRequestSchema,- controllerBeatResponseSchema,controllerConnectionRequestSchema,+ controllerPongSchema,controllerConnectionResponseSchema,credentialRotateResponseSchema,enrollRequestSchema,@@ -46,8 +45,7 @@ const FIXTURES = join(*/const SCHEMA_FOR: Record<string, z.ZodType> = {'assignment_response.json': assignmentSchema,- 'controller_beat_request.json': controllerBeatRequestSchema,- 'controller_beat_response.json': controllerBeatResponseSchema,+ 'controller_pong.json': controllerPongSchema,'controller_connection_request.json': controllerConnectionRequestSchema,'controller_connection_response.json': controllerConnectionResponseSchema,'credential_rotate_response.json': credentialRotateResponseSchema,@@ -178,7 +176,7 @@ describe('status report', () => {const report = {seq: 1,observed_at: '2026-01-01T00:00:00Z',- controller: { version: '0.1.0', protocol: 1, assignment_revision: 0 },+ controller: { version: '0.1.0', protocol: 2, assignment_revision: 0 },machine: {platform: { os: 'linux', arch: 'x64', os_version: '6.1' },disk_free_bytes: 1,diff --git a/console/packages/agent-controller/src/schemas.ts b/console/packages/agent-controller/src/schemas.tsindex 9254fa2cb..a4f4571b4 100644--- a/console/packages/agent-controller/src/schemas.ts+++ b/console/packages/agent-controller/src/schemas.ts@@ -11,7 +11,7 @@ import { z } from 'zod';* known enum is read as `unknown` rather than failing the whole message.*/-export const PROTOCOL_VERSION = 1;+export const PROTOCOL_VERSION = 2;export const PROVIDERS = ['claude', 'codex', 'opencode', 'antigravity', 'cursor'] as const;export type Provider = (typeof PROVIDERS)[number];@@ -295,14 +295,12 @@ export const controllerConnectionResponseSchema = z.object({});export type ControllerConnection = z.infer<typeof controllerConnectionResponseSchema>;-export const controllerBeatRequestSchema = z.object({- connection_id: id,- generation: z.number().int(),+/** The controller's answer to each `ping`: its beat, with how far each agent's host has read. */+export const controllerPongSchema = z.object({+ type: z.literal('pong'),cursors: z.record(z.string().min(1), sequence),});-export type ControllerBeatRequest = z.infer<typeof controllerBeatRequestSchema>;--export const controllerBeatResponseSchema = z.object({ agents: z.array(id) });+export type ControllerPong = z.infer<typeof controllerPongSchema>;/** The stream's first frame. */export const connectionStateSchema = z.object({diff --git a/console/packages/agent-controller/src/status.test.ts b/console/packages/agent-controller/src/status.test.tsindex 4a98b69f9..79a04f548 100644--- a/console/packages/agent-controller/src/status.test.ts+++ b/console/packages/agent-controller/src/status.test.ts@@ -315,7 +315,7 @@ describe('StatusCollector', () => {const assignment = { revision: 7, agents: [entry()] };const first = await collector.collect(assignment);expect(statusReportSchema.safeParse({ ...first, seq: 1 }).success).toBe(true);- expect(first.controller).toEqual({ version: '0.1.0', protocol: 1, assignment_revision: 7 });+ expect(first.controller).toEqual({ version: '0.1.0', protocol: 2, assignment_revision: 7 });expect(first.providers.map((p) => p.provider)).toEqual(['claude']);expect(first.agents[0]).toMatchObject({agent_id: 'agent-1',diff --git a/console/packages/agent-controller/src/stream.test.ts b/console/packages/agent-controller/src/stream.test.tsindex c3b973c36..05441d721 100644--- a/console/packages/agent-controller/src/stream.test.ts+++ b/console/packages/agent-controller/src/stream.test.ts@@ -2,61 +2,72 @@ import { existsSync, readFileSync } from 'node:fs';import { join } from 'node:path';import { setTimeout as delay } from 'node:timers/promises';import { afterEach, beforeEach, describe, expect, it } from 'vitest';-import { AccessTokens, ControllerApiError, ControllerClient } from './api';+import {+ AccessTokens,+ ControllerApiError,+ ControllerClient,+ type OpenedSocket,+ type SocketLike,+} from './api';import { DEFAULT_TIMING } from './controller';import { silentLogger } from './log';import type { AgentCursor, ControllerConnection } from './schemas';-import {- type ControllerFrame,- type ControllerStreamOptions,- parseSse,- runControllerStream,- type SseItem,-} from './stream';+import { type ControllerFrame, type ControllerStreamOptions, runControllerStream } from './stream';import { FakeCore } from './testing/fake-core';-function streamOf(chunks: string[], keepOpen = false): ReadableStream<Uint8Array> {- const encoder = new TextEncoder();- return new ReadableStream({- start(controller) {- for (const chunk of chunks) controller.enqueue(encoder.encode(chunk));- if (!keepOpen) controller.close();- },- });-}+type Message = { event: string; data: unknown };++/**+ * A socket that opens, delivers `messages` one per turn of the event loop,+ * then closes with `end` (or stays open). What the controller sends is kept+ * in `sent`, and `deliver` adds more as the test goes.+ */+class ScriptedSocket extends EventTarget {+ readyState = 0;+ readonly sent: unknown[] = [];++ constructor(messages: (Message | string)[], end: number | 'open') {+ super();+ setImmediate(() => {+ this.readyState = 1;+ this.dispatchEvent(new Event('open'));+ for (const message of messages) this.deliver(message);+ if (end !== 'open') this.end(end);+ });+ }-async function collect(body: ReadableStream<Uint8Array>): Promise<SseItem[]> {- const items: SseItem[] = [];- for await (const item of parseSse(body)) items.push(item);- return items;-}+ deliver(message: Message | string): void {+ const data = typeof message === 'string' ? message : JSON.stringify(message);+ setImmediate(() => {+ if (this.readyState !== 3) this.dispatchEvent(Object.assign(new Event('message'), { data }));+ });+ }-describe('parseSse', () => {- it('reads events, comments and multi-line data', async () => {- expect(- await collect(- streamOf([': keepalive\n\n', 'event: a\ndata: {"x":1}\n\n', 'data: line1\ndata: line2\n\n'])- )- ).toEqual([- { kind: 'comment', text: 'keepalive' },- { kind: 'event', event: 'a', data: '{"x":1}', id: null },- { kind: 'event', event: 'message', data: 'line1\nline2', id: null },- ]);- });+ end(code: number): void {+ setImmediate(() => {+ if (this.readyState === 3) return;+ this.readyState = 3;+ this.dispatchEvent(Object.assign(new Event('close'), { code }));+ });+ }- it('handles CRLF, fields without a space, and frames split across chunks', async () => {- expect(- await collect(streamOf(['event:b\r', '\ndata:{"y"', ':2}\r\nid: 7\r\n', '\r\n']))- ).toEqual([{ kind: 'event', event: 'b', data: '{"y":2}', id: '7' }]);- });+ send(data: string): void {+ this.sent.push(JSON.parse(data));+ }- it('drops a trailing frame the stream never finished', async () => {- expect(await collect(streamOf(['event: c\ndata: {}\n']))).toEqual([]);- });-});+ close(code = 1000): void {+ this.end(code);+ }+}++function scriptedSocket(messages: (Message | string)[], end: number | 'open'): ScriptedSocket {+ return new ScriptedSocket(messages, end);+}++const state: Message = { event: 'connection_state', data: { connection_id: 'connection-1' } };-function sse(event: string, data: unknown): string {- return `event: ${event}\ndata: ${JSON.stringify(data)}\n\n`;+function refused(status: number, code: string): Message {+ return { event: 'refused', data: { status, detail: { code, message: code } } };}async function waitFor(condition: () => boolean, what: string, timeoutMs = 15_000): Promise<void> {@@ -78,23 +89,24 @@ const opened: ControllerConnection = {class ScriptedClient {opens: Record<string, AgentCursor>[] = [];attaches: { connectionId: string; generation: number }[] = [];- beats: Record<string, number>[] = [];+ opened: ScriptedSocket[] = [];+ staleTokens = 0;open: () => Promise<ControllerConnection> = async () => opened;- events: () => Promise<Response> = async () => new Response(streamOf([], true));- beat: () => Promise<void> = async () => {};+ sockets: () => ScriptedSocket = () => scriptedSocket([], 'open');readonly client: ControllerStreamOptions['client'] = {openConnection: async (cursors) => {this.opens.push(cursors);return this.open();},- openEvents: async (connection) => {+ openSocket: async (connection): Promise<OpenedSocket> => {this.attaches.push(connection);- return this.events();- },- beat: async (_connection, cursors) => {- this.beats.push(cursors);- return this.beat();+ const socket = this.sockets();+ this.opened.push(socket);+ return {+ socket: socket as unknown as SocketLike,+ tokenRefused: () => void this.staleTokens++,+ };},};}@@ -128,15 +140,20 @@ function run(describe('runControllerStream', () => {it('opens with the cursors, attaches, and hands over typed frames in order, skipping bad ones', async () => {const scripted = new ScriptedClient();- scripted.events = async () =>- new Response(- streamOf(- [- ': ping\n\n',- sse('agent.attached', { agent_id: 'agent-1', from_seq: 7, rooms: ['room-a'] }),- sse('agent.event', { agent_id: 'agent-1' }),- sse('agent.unknown', { agent_id: 'agent-1' }),- sse('agent.event', {+ scripted.sockets = () =>+ scriptedSocket(+ [+ { event: 'ping', data: {} },+ 'not json',+ {+ event: 'agent.attached',+ data: { agent_id: 'agent-1', from_seq: 7, rooms: ['room-a'] },+ },+ { event: 'agent.event', data: { agent_id: 'agent-1' } },+ { event: 'agent.unknown', data: { agent_id: 'agent-1' } },+ {+ event: 'agent.event',+ data: {agent_id: 'agent-1',seq: 8,event: {@@ -146,16 +163,15 @@ describe('runControllerStream', () => {payload: { addressed: true },future: 1,},- }),- sse('assignment.changed', { revision: 2 }),- sse('operation.pending', {- operation_id: 'op-1',- kind: 'agent.restart',- agent_id: null,- }),- ],- true- )+ },+ },+ { event: 'assignment.changed', data: { revision: 2 } },+ {+ event: 'operation.pending',+ data: { operation_id: 'op-1', kind: 'agent.restart', agent_id: null },+ },+ ],+ 'open');const stop = new AbortController();const { frames, states, ending } = run(scripted, stop.signal);@@ -174,19 +190,21 @@ describe('runControllerStream', () => {expect(frames[1]!.data).toMatchObject({ seq: 8, event: { future: 1 } });});- it('beats the connection with the confirmed cursors while attached', async () => {+ it('answers every ping with a pong naming the confirmed cursors: its beat', async () => {const scripted = new ScriptedClient();+ const ping = { event: 'ping', data: {} };+ scripted.sockets = () => scriptedSocket([state, ping, ping], 'open');const stop = new AbortController();const { ending } = run(scripted, stop.signal);- await waitFor(() => scripted.beats.length >= 2, 'two beats');+ await waitFor(() => (scripted.opened[0]?.sent.length ?? 0) >= 2, 'two pongs');stop.abort();await ending;- expect(scripted.beats[0]).toEqual({ 'agent-1': 7 });+ expect(scripted.opened[0]!.sent[0]).toEqual({ type: 'pong', cursors: { 'agent-1': 7 } });});- it('reattaches to the same connection after the stream ends', async () => {+ it('reattaches to the same connection after the socket closes', async () => {const scripted = new ScriptedClient();- scripted.events = async () => new Response(streamOf([': ping\n\n']));+ scripted.sockets = () => scriptedSocket([state], 1000);const stop = new AbortController();const { states, ending } = run(scripted, stop.signal);await waitFor(() => scripted.attaches.length >= 3, 'three attaches');@@ -199,6 +217,18 @@ describe('runControllerStream', () => {expect(states).toContain('disconnected');});+ it('comes back within a second, without backing off, when the server is restarting', async () => {+ const scripted = new ScriptedClient();+ scripted.sockets = () => scriptedSocket([state], 1012);+ const stop = new AbortController();+ const started = Date.now();+ const { ending } = run(scripted, stop.signal, { initialBackoffMs: 60_000 });+ await waitFor(() => scripted.attaches.length >= 3, 'three attaches', 3_000);+ stop.abort();+ await ending;+ expect(Date.now() - started).toBeGreaterThanOrEqual(490);+ });+it('reconnects when nothing arrives within the idle timeout', async () => {const scripted = new ScriptedClient();const started = Date.now();@@ -212,13 +242,11 @@ describe('runControllerStream', () => {it('opens a new connection, from the cursors as they stand, when Switch no longer knows it', async () => {const scripted = new ScriptedClient();- let attaches = 0;- scripted.events = async () => {- attaches++;- if (attaches === 2)- throw new ControllerApiError(404, 'unknown_connection', 'gone', false, null);- return new Response(streamOf([]));- };+ scripted.sockets = () =>+ scriptedSocket(+ scripted.attaches.length === 2 ? [refused(404, 'unknown_connection')] : [state],+ scripted.attaches.length === 2 ? 4404 : 1000+ );let cursor = 7;const stop = new AbortController();const { ending } = run(scripted, stop.signal, {@@ -230,13 +258,9 @@ describe('runControllerStream', () => {expect(scripted.opens).toEqual([{ 'agent-1': 7 }, { 'agent-1': 8 }]);});- it('opens a new connection when the heartbeat is refused as stale', async () => {+ it('opens a new connection when an attach is refused as stale', async () => {const scripted = new ScriptedClient();- let beats = 0;- scripted.beat = async () => {- beats++;- if (beats === 1) throw new ControllerApiError(409, 'stale_generation', 'old', false, null);- };+ scripted.sockets = () => scriptedSocket([refused(409, 'stale_generation')], 4409);const stop = new AbortController();const { ending } = run(scripted, stop.signal);await waitFor(() => scripted.opens.length === 2, 'a second open');@@ -244,11 +268,12 @@ describe('runControllerStream', () => {await ending;});- it('opens a new connection when Switch ends the stream for any reason but a takeover', async () => {+ it('opens a new connection when Switch evicts it for any reason but a takeover', async () => {const scripted = new ScriptedClient();- scripted.events = async () =>- new Response(- streamOf([sse('evicted', { code: 'heartbeat_lapsed', reason: 'lapsed' })], true)+ scripted.sockets = () =>+ scriptedSocket(+ [state, { event: 'evicted', data: { code: 'no_stream', reason: 'beat refused' } }],+ 1000);const stop = new AbortController();const { ending } = run(scripted, stop.signal);@@ -259,39 +284,50 @@ describe('runControllerStream', () => {it('stops for good when another instance took the connection over', async () => {const evicted = new ScriptedClient();- evicted.events = async () =>- new Response(streamOf([sse('evicted', { code: 'taken_over', reason: 'x' })], true));+ evicted.sockets = () =>+ scriptedSocket(+ [state, { event: 'evicted', data: { code: 'taken_over', reason: 'x' } }],+ 1000+ );expect(await run(evicted, new AbortController().signal).ending).toBe('taken_over');- const scripted = new ScriptedClient();- scripted.beat = async () => {- throw new ControllerApiError(409, 'taken_over', 'someone else', false, null);- };- expect(await run(scripted, new AbortController().signal).ending).toBe('taken_over');- const refused = new ScriptedClient();- refused.events = async () => {- throw new ControllerApiError(409, 'taken_over', 'someone else', false, null);- };- expect(await run(refused, new AbortController().signal).ending).toBe('taken_over');+ const refusedAttach = new ScriptedClient();+ refusedAttach.sockets = () => scriptedSocket([refused(409, 'taken_over')], 4409);+ expect(await run(refusedAttach, new AbortController().signal).ending).toBe('taken_over');});- it('ends as revoked on the frame, on a refused open, and on a refused beat', async () => {+ it('ends as revoked on the frame, on a refused open, and on a refused attach', async () => {const framed = new ScriptedClient();- framed.events = async () => new Response(streamOf([sse('credential.revoked', {})], true));+ framed.sockets = () => scriptedSocket([{ event: 'credential.revoked', data: {} }], 'open');const first = run(framed, new AbortController().signal);expect(await first.ending).toBe('revoked');expect(first.frames.map((frame) => frame.type)).toEqual(['credential.revoked']);- const refused = new ScriptedClient();- refused.open = async () => {+ const refusedOpen = new ScriptedClient();+ refusedOpen.open = async () => {throw new ControllerApiError(401, 'controller_revoked', 'revoked', false, null);};- expect(await run(refused, new AbortController().signal).ending).toBe('revoked');+ expect(await run(refusedOpen, new AbortController().signal).ending).toBe('revoked');- const beaten = new ScriptedClient();- beaten.beat = async () => {- throw new ControllerApiError(401, 'controller_revoked', 'revoked', false, null);- };- expect(await run(beaten, new AbortController().signal).ending).toBe('revoked');+ const refusedAttach = new ScriptedClient();+ refusedAttach.sockets = () => scriptedSocket([refused(401, 'controller_revoked')], 4401);+ const third = run(refusedAttach, new AbortController().signal);+ expect(await third.ending).toBe('revoked');+ expect(refusedAttach.staleTokens).toBe(0);+ });++ it('drops a token the socket refused as stale, and attaches again', async () => {+ const scripted = new ScriptedClient();+ scripted.sockets = () =>+ scripted.attaches.length === 1+ ? scriptedSocket([refused(401, 'token_expired')], 4401)+ : scriptedSocket([state], 'open');+ const stop = new AbortController();+ const { states, ending } = run(scripted, stop.signal);+ await waitFor(() => states.includes('connected'), 'a connection');+ stop.abort();+ await ending;+ expect(scripted.staleTokens).toBe(1);+ expect(scripted.opens).toHaveLength(1);});it('keeps retrying an open that fails', async () => {@@ -302,6 +338,7 @@ describe('runControllerStream', () => {if (opens < 3) throw new TypeError('fetch failed');return opened;};+ scripted.sockets = () => scriptedSocket([state], 'open');const stop = new AbortController();const { states, ending } = run(scripted, stop.signal);await waitFor(() => states.includes('connected'), 'a connection');@@ -310,7 +347,7 @@ describe('runControllerStream', () => {expect(opens).toBe(3);});- it('waits less each time a stream attached, however briefly, and never long', async () => {+ it('waits less each time a socket attached, however briefly, and never long', async () => {const scripted = new ScriptedClient();let opens = 0;scripted.open = async () => {@@ -318,12 +355,8 @@ describe('runControllerStream', () => {if (opens <= 3) throw new TypeError('fetch failed');return opened;};- let attaches = 0;- scripted.events = async () => {- attaches++;- if (attaches === 1) return new Response(streamOf([]));- throw new TypeError('fetch failed');- };+ scripted.sockets = () =>+ scripted.attaches.length === 1 ? scriptedSocket([state], 1000) : scriptedSocket([], 1006);const waits: number[] = [];const log = {...silentLogger,@@ -340,7 +373,7 @@ describe('runControllerStream', () => {await waitFor(() => waits.length >= 5, 'five reconnect waits');stop.abort();await ending;- // Doubling from the start after the stream that attached, capped, with+ // Doubling from the start after the socket that attached, capped, with// jitter taking half off (random() is 0 here).expect(waits.slice(0, 5)).toEqual([4, 8, 16, 8, 16]);});@@ -370,13 +403,7 @@ describe('runControllerStream', () => {data: unknown;}[];const scripted = new ScriptedClient();- scripted.events = async () =>- new Response(- streamOf(- fixtureFrames.map((frame) => sse(frame.event, frame.data)),- true- )- );+ scripted.sockets = () => scriptedSocket(fixtureFrames, 'open');const stop = new AbortController();const { frames, ending } = run(scripted, stop.signal);await waitFor(@@ -396,7 +423,7 @@ describe('runControllerStream', () => {});});-describe('runControllerStream against the controller stream routes', () => {+describe('runControllerStream against the controller routes', () => {let core: FakeCore;beforeEach(async () => {@@ -414,6 +441,7 @@ describe('runControllerStream against the controller stream routes', () => {server: core.url,controllerId: core.controllerId,version: '0.1.0',+ openWebSocket: core.openWebSocket,tokens: new AccessTokens({fetch,server: core.url,@@ -425,7 +453,7 @@ describe('runControllerStream against the controller stream routes', () => {});}- it('opens, reads, beats, reattaches, and reopens with cursors after the connection lapses', async () => {+ it('opens, reads, pongs, reattaches, and reopens with cursors after the connection lapses', async () => {let confirmed = 3;const frames: ControllerFrame[] = [];const stop = new AbortController();@@ -439,20 +467,23 @@ describe('runControllerStream against the controller stream routes', () => {onFrame: async (frame) => void frames.push(frame),signal: stop.signal,log: silentLogger,- // Long enough that a loaded machine does not lapse the stream before the test does.+ // Long enough that a loaded machine does not lapse the socket before the test does.idleTimeoutMs: 30_000,initialBackoffMs: 5,maxBackoffMs: 20,random: () => 0,});- await waitFor(() => core.streamCount === 1, 'the stream');+ await waitFor(() => core.streamCount === 1, 'the socket');core.push('assignment.changed', { revision: 4 });- await waitFor(() => frames.length === 1, 'the frame');- await waitFor(() => core.beats.length > 0, 'a beat');+ await waitFor(+ () => frames.some((frame) => frame.type === 'assignment.changed'),+ 'the pushed frame'+ );+ await waitFor(() => core.beats.length > 0, 'a pong');expect(core.beats[0]).toEqual({ 'agent-1': 3 });core.closeStreams();- await waitFor(() => core.streamCount === 1, 'the stream reattached');+ await waitFor(() => core.streamCount === 1, 'the socket reattached');expect(core.opens).toHaveLength(1);confirmed = 9;diff --git a/console/packages/agent-controller/src/stream.ts b/console/packages/agent-controller/src/stream.tsindex c845fc660..0285e15ab 100644--- a/console/packages/agent-controller/src/stream.ts+++ b/console/packages/agent-controller/src/stream.ts@@ -1,6 +1,12 @@import { setTimeout as delay } from 'node:timers/promises';import type { z } from 'zod';-import { ControllerApiError, type ControllerClient, isRevoked, isTakenOver } from './api';+import {+ ControllerApiError,+ type ControllerClient,+ isRevoked,+ isTakenOver,+ type OpenedSocket,+} from './api';import { errorMessage, type Logger } from './log';import {type AgentApprovalOutcomeFrame,@@ -29,60 +35,7 @@ import {operationPendingSchema,} from './schemas';-export type SseItem =- | { kind: 'comment'; text: string }- | { kind: 'event'; event: string; data: string; id: string | null };--/**- * Splits an SSE byte stream into events and comments. Comments are surfaced- * rather than skipped because the server's `: keepalive` is how a silent but- * healthy stream is told apart from a dead one.- */-export async function* parseSse(body: ReadableStream<Uint8Array>): AsyncGenerator<SseItem> {- const reader = body.getReader();- const decoder = new TextDecoder();- let buffered = '';- let event = '';- let id: string | null = null;- let data: string[] = [];- try {- for (;;) {- const { done, value } = await reader.read();- if (done) return;- buffered += decoder.decode(value, { stream: true });- let newline: number;- while ((newline = buffered.search(/\r\n|\r|\n/)) !== -1) {- const line = buffered.slice(0, newline);- const width = buffered.startsWith('\r\n', newline) ? 2 : 1;- // A lone CR at the end of the buffer may be the first half of a CRLF.- if (width === 1 && buffered[newline] === '\r' && newline === buffered.length - 1) break;- buffered = buffered.slice(newline + width);- if (line === '') {- if (data.length)- yield { kind: 'event', event: event || 'message', data: data.join('\n'), id };- event = '';- data = [];- continue;- }- if (line.startsWith(':')) {- yield { kind: 'comment', text: line.slice(1).trimStart() };- continue;- }- const colon = line.indexOf(':');- const field = colon === -1 ? line : line.slice(0, colon);- let value = colon === -1 ? '' : line.slice(colon + 1);- if (value.startsWith(' ')) value = value.slice(1);- if (field === 'event') event = value;- else if (field === 'data') data.push(value);- else if (field === 'id') id = value;- }- }- } finally {- void reader.cancel().catch(() => {});- }-}--/** The schema for each SSE event type the controller stream carries, by `event:` name. */+/** The schema for each frame the controller's socket carries, by its `event`. */export const STREAM_FRAME_SCHEMAS = {connection_state: connectionStateSchema,evicted: evictedSchema,@@ -116,10 +69,10 @@ export type ControllerFrame =export type StreamEnding = 'stopped' | 'revoked' | 'taken_over';export type ControllerStreamOptions = {- client: Pick<ControllerClient, 'openConnection' | 'beat' | 'openEvents'>;+ client: Pick<ControllerClient, 'openConnection' | 'openSocket'>;/** Where each agent resumes when a connection is opened. */cursors: () => Record<string, AgentCursor>;- /** How far each agent's watcher has confirmed reading, sent on every beat. */+ /** How far each agent's watcher has confirmed reading, sent with every pong. */confirmed: () => Record<string, number>;/** A connection was opened: Core attached these agents to it. */onOpened: (connection: ControllerConnection) => Promise<void> | void;@@ -131,7 +84,7 @@ export type ControllerStreamOptions = {onFrame: (frame: ControllerFrame) => Promise<void>;signal: AbortSignal;log: Logger;- /** No byte for this long, keepalives included, and the stream is presumed dead. */+ /** Nothing on the socket for this long, pings included, and it is presumed dead. */idleTimeoutMs: number;/** The first reconnect wait, and where the wait returns to once a stream attaches. */initialBackoffMs: number;@@ -141,9 +94,18 @@ export type ControllerStreamOptions = {random: () => number;};+/**+ * The close code a server sends as it begins shutting down (uvicorn sends it+ * to every socket). The server is about to be back, so the reconnect waits+ * are short and do not grow.+ */+const SERVICE_RESTART = 1012;+const RESTART_RETRY_MS = 250;+const RESTART_RETRY_JITTER_MS = 500;+class IdleTimeout extends Error {constructor(ms: number) {- super(`No data on the event stream for ${Math.round(ms / 1000)} s.`);+ super(`Nothing on the controller socket for ${Math.round(ms / 1000)} s.`);}}@@ -153,35 +115,40 @@ function connectionGone(error: unknown): boolean {error instanceof ControllerApiError &&(error.status === 404 ||error.code === 'unknown_connection' ||- error.code === 'stale_generation')+ error.code === 'stale_generation' ||+ error.code === 'no_stream'));}-type Held = { connectionId: string; generation: number; heartbeatMs: number };+type Held = { connectionId: string; generation: number };-/** One attempt at attaching the stream: its socket, and what it learned. */+/** One attempt at attaching the socket, and what it learned. */type Attempt = {- socket: AbortController;+ abort: AbortController;/** Set when the stream stopped for good. */ending: StreamEnding | null;/** When the stream attached, or 0 if it never did. */attachedAt: number;- beating: Promise<void> | null;+ /** The server closed the socket because it is restarting. */+ restarting: boolean;};+type Message = { event: string; data: unknown };+/**- * The controller stream, held open for as long as `signal` lives.+ * The controller's connection to Switch, held open for as long as `signal`+ * lives.** A connection is opened once (`POST .../connection`, with every agent's- * cursor) and the stream attached to it (`GET .../events`). A dropped socket- * reattaches to the same connection and generation, so Core resumes where it- * was; a connection Core no longer knows (its 6 s heartbeat lapsed, or it was- * superseded) is opened afresh, from the cursors as they stand. While the- * stream is attached the connection is beaten every `heartbeat_interval_s`- * with each agent's confirmed cursor and the rooms its sessions work in, so a- * placement made locally reaches Switch on the next beat.+ * cursor) and a WebSocket attached to it (`.../connection/ws`). The socket+ * carries the stream's frames down, and the server's `ping` every heartbeat+ * interval, which is answered with a `pong` naming each agent's confirmed+ * cursor: that pong is the controller's beat. A dropped socket reattaches to+ * the same connection and generation, so Core resumes where it was; a+ * connection Core no longer knows (its heartbeat lapsed, or it was+ * superseded) is opened afresh, from the cursors as they stand.*- * Ends with `'revoked'` on `credential.revoked` or a request refused as+ * Ends with `'revoked'` on `credential.revoked` or a refusal saying* `controller_revoked`, with `'taken_over'` when another instance of this* controller took the connection over (an `evicted` frame or a refusal saying* so: reopening would take it straight back, so this one stops), and with@@ -203,12 +170,12 @@ class ControllerStream {const { signal, log } = this.options;while (!signal.aborted) {const attempt: Attempt = {- socket: new AbortController(),+ abort: new AbortController(),ending: null,attachedAt: 0,- beating: null,+ restarting: false,};- const stop = () => attempt.socket.abort();+ const stop = () => attempt.abort.abort();signal.addEventListener('abort', stop, { once: true });let failure: string | null = null;try {@@ -218,31 +185,32 @@ class ControllerStream {if (isRevoked(error)) attempt.ending = 'revoked';else if (isTakenOver(error)) attempt.ending = 'taken_over';else {- const reason = attempt.socket.signal.reason;+ const reason = attempt.abort.signal.reason;failure = errorMessage(reason instanceof IdleTimeout ? reason : error);}}} finally {signal.removeEventListener('abort', stop);- attempt.socket.abort();- await attempt.beating;+ attempt.abort.abort();}if (attempt.ending) return attempt.ending;if (signal.aborted) break;if (attempt.attachedAt > 0) {- this.options.onDisconnected(failure ?? 'the stream ended');- // Every agent on this controller is offline until the stream is back,- // so a stream that attached at all starts the waits over.+ this.options.onDisconnected(failure ?? 'the socket closed');+ // Every agent on this controller is offline until the socket is back,+ // so a socket that attached at all starts the waits over.this.backoff = this.options.initialBackoffMs;}- const wait = Math.round(this.backoff * (0.5 + this.options.random() * 0.5));+ const wait = attempt.restarting+ ? RESTART_RETRY_MS + Math.round(this.options.random() * RESTART_RETRY_JITTER_MS)+ : Math.round(this.backoff * (0.5 + this.options.random() * 0.5));if (failure !== null)- log.warn('The controller stream is down; reconnecting.', {+ log.warn('The controller socket is down; reconnecting.', {error: failure,retryInMs: wait,});await delay(wait, undefined, { signal }).catch(() => {});- this.backoff = Math.min(this.backoff * 2, this.options.maxBackoffMs);+ if (!attempt.restarting) this.backoff = Math.min(this.backoff * 2, this.options.maxBackoffMs);}return 'stopped';}@@ -250,13 +218,9 @@ class ControllerStream {private async attach(attempt: Attempt): Promise<void> {const { client, log } = this.options;if (!this.held) {- const opened = await client.openConnection(this.options.cursors(), attempt.socket.signal);- this.held = {- connectionId: opened.connection_id,- generation: opened.generation,- heartbeatMs: opened.heartbeat_interval_s * 1000,- };- log.info('Opened the controller stream connection', {+ const opened = await client.openConnection(this.options.cursors(), attempt.abort.signal);+ this.held = { connectionId: opened.connection_id, generation: opened.generation };+ log.info('Opened the controller connection', {connectionId: opened.connection_id,generation: opened.generation,agents: opened.agents.length,@@ -264,135 +228,160 @@ class ControllerStream {await this.options.onOpened(opened);}const current = this.held;- let response: Response;try {- response = await client.openEvents(current, attempt.socket.signal);+ await this.read(await client.openSocket(current), attempt);} catch (error) {if (!connectionGone(error)) throw error;- log.warn('Switch no longer knows the stream connection; opening a new one.', {+ log.warn('Switch no longer knows the controller connection; opening a new one.', {error: errorMessage(error),});- this.held = null;- return;+ if (this.held === current) this.held = null;}- attempt.attachedAt = Date.now();- this.options.onConnected();- attempt.beating = this.beat(current, attempt);- await this.read(response.body!, attempt);}- private async read(body: ReadableStream<Uint8Array>, attempt: Attempt): Promise<void> {- const { socket } = attempt;+ /**+ * Reads the socket until it closes: answers each ping with the confirmed+ * cursors and hands every other frame on, in order. A refusal is thrown as+ * the error an HTTP request would have raised.+ */+ private async read(opened: OpenedSocket, attempt: Attempt): Promise<void> {+ const { socket } = opened;+ const abort = attempt.abort;+ const pending: Message[] = [];+ let refused: { status: number; code: string; message: string } | null = null;+ let closedWith: number | null = null;+ let wake: (() => void) | null = null;+ const notify = () => {+ const resolve = wake;+ wake = null;+ resolve?.();+ };let idle: ReturnType<typeof setTimeout> | null = null;const touch = () => {if (idle) clearTimeout(idle);idle = setTimeout(- () => socket.abort(new IdleTimeout(this.options.idleTimeoutMs)),+ () => abort.abort(new IdleTimeout(this.options.idleTimeoutMs)),this.options.idleTimeoutMs);};- const aborted = new Promise<never>((_resolve, reject) => {- const fail = () => reject(socket.signal.reason ?? new Error('aborted'));- if (socket.signal.aborted) fail();- socket.signal.addEventListener('abort', fail, { once: true });- });- aborted.catch(() => {});+ const onMessage = (message: MessageEvent) => {+ touch();+ let parsed: { event?: unknown; data?: unknown };+ try {+ parsed = JSON.parse(String(message.data)) as typeof parsed;+ } catch {+ this.options.log.error('Dropped a controller socket message that is not JSON');+ return;+ }+ if (parsed.event === 'ping') {+ socket.send(JSON.stringify({ type: 'pong', cursors: this.options.confirmed() }));+ return;+ }+ if (parsed.event === 'refused') {+ refused = refusalOf(parsed.data);+ return;+ }+ if (typeof parsed.event !== 'string') return;+ pending.push({ event: parsed.event, data: parsed.data });+ notify();+ };+ const onClose = (event: CloseEvent) => {+ closedWith = event.code;+ if (event.code === SERVICE_RESTART) attempt.restarting = true;+ notify();+ };+ const onAbort = () => notify();+ socket.addEventListener('message', onMessage);+ socket.addEventListener('close', onClose);+ socket.addEventListener('error', notify);+ abort.signal.addEventListener('abort', onAbort, { once: true });try {touch();- const items = parseSse(body);for (;;) {- const next = await Promise.race([items.next(), aborted]);- if (next.done) break;- touch();- const item = next.value;- if (item.kind === 'comment') continue;- const frame = parseFrame(item, this.options.log);- if (!frame) continue;- await this.options.onFrame(frame);- if (frame.type === 'credential.revoked') {- attempt.ending = 'revoked';- return;- }- if (frame.type === 'evicted') {- if (frame.data.code === 'taken_over') {- this.options.log.error(- 'Another instance of this controller took its stream over; this one stops. Run one controller per data directory.'- );- attempt.ending = 'taken_over';- return;+ while (pending.length > 0) {+ const message = pending.shift() as Message;+ if (attempt.attachedAt === 0) {+ attempt.attachedAt = Date.now();+ this.options.onConnected();}- this.options.log.warn('Switch ended the controller stream; opening a new connection.', {- code: frame.data.code,- reason: frame.data.reason,- });- this.held = null;- return;+ const frame = parseFrame(message, this.options.log);+ if (!frame) continue;+ await this.options.onFrame(frame);+ if (this.ended(frame, attempt)) return;}+ if (closedWith !== null || abort.signal.aborted) break;+ await new Promise<void>((resolve) => {+ wake = resolve;+ });+ }+ const refusal = refused as { status: number; code: string; message: string } | null;+ if (refusal !== null) {+ if (refusal.status === 401 && refusal.code !== 'controller_revoked') opened.tokenRefused();+ throw new ControllerApiError(refusal.status, refusal.code, refusal.message, false, null);}+ if (abort.signal.aborted && abort.signal.reason instanceof IdleTimeout)+ throw abort.signal.reason;+ if (!this.options.signal.aborted && attempt.attachedAt === 0)+ throw new Error(`The controller socket closed before it attached (code ${closedWith}).`);if (!this.options.signal.aborted)- this.options.log.warn('The controller stream ended; reconnecting.');+ this.options.log.warn('The controller socket closed; reconnecting.', { code: closedWith });} finally {if (idle) clearTimeout(idle);+ socket.removeEventListener('message', onMessage);+ socket.removeEventListener('close', onClose);+ socket.removeEventListener('error', notify);+ abort.signal.removeEventListener('abort', onAbort);+ if (socket.readyState === 0 || socket.readyState === 1) socket.close(1000);}}- /** Beats the connection while this attempt's stream is attached. */- private async beat(connection: Held, attempt: Attempt): Promise<void> {- const { client, log } = this.options;- const signal = attempt.socket.signal;- let failures = 0;- while (!signal.aborted) {- await delay(connection.heartbeatMs, undefined, { signal }).catch(() => {});- if (signal.aborted) return;- try {- await client.beat(connection, this.options.confirmed(), signal);- if (failures > 0)- log.info('The controller stream heartbeat recovered', { afterFailures: failures });- failures = 0;- } catch (error) {- if (signal.aborted) return;- if (isRevoked(error) || isTakenOver(error)) {- if (isTakenOver(error))- log.error(- 'Another instance of this controller took its stream connection over; this one stops. Run one controller per data directory.'- );- attempt.ending = isRevoked(error) ? 'revoked' : 'taken_over';- attempt.socket.abort();- return;- }- if (connectionGone(error)) {- log.warn('Switch refused the heartbeat for a connection it no longer holds; reopening.', {- error: errorMessage(error),- });- if (this.held === connection) this.held = null;- attempt.socket.abort();- return;- }- failures++;- // The first failure, then powers of two: an outage costs a few lines.- if ((failures & (failures - 1)) === 0)- log.warn('The controller stream heartbeat failed', {- failures,- error: errorMessage(error),- });- }+ /** Whether this frame ends the attempt: a revocation, or an eviction. */+ private ended(frame: ControllerFrame, attempt: Attempt): boolean {+ if (frame.type === 'credential.revoked') {+ attempt.ending = 'revoked';+ return true;}+ if (frame.type !== 'evicted') return false;+ if (frame.data.code === 'taken_over') {+ this.options.log.error(+ 'Another instance of this controller took its connection over; this one stops. Run one controller per data directory.'+ );+ attempt.ending = 'taken_over';+ return true;+ }+ this.options.log.warn('Switch ended the controller connection; opening a new one.', {+ code: frame.data.code,+ reason: frame.data.reason,+ });+ this.held = null;+ return true;}}-function parseFrame(item: SseItem & { kind: 'event' }, log: Logger): ControllerFrame | null {- const schema = (STREAM_FRAME_SCHEMAS as Record<string, z.ZodType>)[item.event];+function refusalOf(data: unknown): { status: number; code: string; message: string } {+ const body = (data ?? {}) as { status?: unknown; detail?: unknown };+ const detail = (body.detail ?? {}) as { code?: unknown; message?: unknown };+ return {+ status: typeof body.status === 'number' ? body.status : 0,+ code: typeof detail.code === 'string' ? detail.code : 'refused',+ message:+ typeof detail.message === 'string' ? detail.message : 'Switch refused the controller socket.',+ };+}++function parseFrame(message: Message, log: Logger): ControllerFrame | null {+ const schema = (STREAM_FRAME_SCHEMAS as Record<string, z.ZodType>)[message.event];if (!schema) {- log.debug('Ignoring an event type this controller does not know', { event: item.event });+ log.debug('Ignoring a frame this controller does not know', { event: message.event });return null;}- try {- return { type: item.event, data: schema.parse(JSON.parse(item.data)) } as ControllerFrame;- } catch (error) {- log.error('Dropped a controller stream frame that does not match the protocol', {- event: item.event,- error: errorMessage(error),+ const result = schema.safeParse(message.data);+ if (!result.success) {+ log.error('Dropped a controller frame that does not match the protocol', {+ event: message.event,+ error: result.error.message,});return null;}+ return { type: message.event, data: result.data } as ControllerFrame;}diff --git a/console/packages/agent-controller/src/testing/fake-core.ts b/console/packages/agent-controller/src/testing/fake-core.tsindex 58b69ac33..a10238e0e 100644--- a/console/packages/agent-controller/src/testing/fake-core.ts+++ b/console/packages/agent-controller/src/testing/fake-core.ts@@ -1,5 +1,6 @@import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http';import type { AddressInfo } from 'node:net';+import type { OpenWebSocket, SocketLike } from '../api';import type {AgentCursor,Assignment,@@ -19,6 +20,56 @@ type Recorded = {bytes: number;};+/**+ * A WebSocket held in memory: what the controller sends lands in `received`,+ * and what the fake core sends arrives as `message` events, one per turn of+ * the event loop, in order.+ */+class FakeSocket extends EventTarget {+ readyState = 0;+ readonly url: string;+ readonly headers: Record<string, string>;+ onSend: (data: string) => void = () => {};++ constructor(url: string, headers: Record<string, string>) {+ super();+ this.url = url;+ this.headers = headers;+ }++ send(data: string): void {+ if (this.readyState === 1) this.onSend(String(data));+ }++ close(code = 1000): void {+ this.end(code);+ }++ open(): void {+ this.readyState = 1;+ this.dispatchEvent(new Event('open'));+ }++ deliver(event: string, data: unknown): void {+ if (this.readyState !== 1) return;+ const text = JSON.stringify({ event, data });+ setImmediate(() => {+ if (this.readyState !== 3)+ this.dispatchEvent(Object.assign(new Event('message'), { data: text }));+ });+ }++ /** Closes after whatever was sent before it has arrived. */+ end(code: number): void {+ if (this.readyState >= 2) return;+ this.readyState = 2;+ setImmediate(() => {+ this.readyState = 3;+ this.dispatchEvent(Object.assign(new Event('close'), { code }));+ });+ }+}+type StreamConnection = {id: string;generation: number;@@ -63,7 +114,8 @@ export class FakeCore {private generations = 0;private current: StreamConnection | null = null;private readonly connections = new Map<string, StreamConnection>();- private readonly streams = new Map<ServerResponse, StreamConnection>();+ private readonly streams = new Map<FakeSocket, StreamConnection>();+ private readonly pings = new Set<ReturnType<typeof setInterval>>();private server: Server | null = null;url = '';@@ -82,7 +134,8 @@ export class FakeCore {}async stop(): Promise<void> {- for (const stream of this.streams.keys()) stream.destroy();+ for (const ping of this.pings) clearInterval(ping);+ for (const stream of this.streams.keys()) stream.end(1001);this.server?.closeAllConnections();await new Promise<void>((resolve) => this.server?.close(() => resolve()) ?? resolve());}@@ -109,8 +162,7 @@ export class FakeCore {}push(event: string, data: unknown): void {- for (const stream of this.streams.keys())- stream.write(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`);+ for (const stream of this.streams.keys()) stream.deliver(event, data);}/** An agent's domain event on the controller stream, as Switch wraps it. */@@ -119,12 +171,87 @@ export class FakeCore {this.push('agent.event', { agent_id: agentId, seq, event: { ...event, sequence: seq } });}- /** Ends the open streams; the connection itself lives on. */- closeStreams(): void {- for (const stream of this.streams.keys()) stream.end();+ /** Ends the open sockets; the connection itself lives on. */+ closeStreams(code = 1000): void {+ for (const stream of this.streams.keys()) stream.end(code);this.streams.clear();}+ /** The controller's socket, held by this fake core rather than a server. */+ readonly openWebSocket: OpenWebSocket = (url, headers) => {+ const socket = new FakeSocket(url, headers);+ setImmediate(() => this.attach(socket));+ return socket as unknown as SocketLike;+ };++ private attach(socket: FakeSocket): void {+ const url = new URL(socket.url.replace(/^ws/, 'http'));+ this.requests.push({+ method: 'WS',+ path: url.pathname,+ search: url.search,+ headers: Object.fromEntries(+ Object.entries(socket.headers).map(([name, value]) => [name.toLowerCase(), value])+ ),+ body: null,+ bytes: 0,+ });+ socket.open();+ const refuse = (status: number, code: string) => {+ socket.deliver('refused', { status, detail: { code, message: `refused: ${code}` } });+ socket.end(4000 + status);+ };+ if (url.pathname !== `/v1/controllers/${this.controllerId}/connection/ws`)+ return refuse(404, 'not_found');+ const bearer = socket.headers.Authorization?.replace(/^Bearer /, '') ?? '';+ if (this.revoked) return refuse(401, 'controller_revoked');+ if (!this.tokens.has(bearer)) return refuse(401, 'token_expired');+ const connection = this.connections.get(url.searchParams.get('connection_id') ?? '');+ if (!connection) return refuse(404, 'unknown_connection');+ if (connection.superseded) return refuse(409, 'taken_over');+ if (Number(url.searchParams.get('generation')) !== connection.generation)+ return refuse(409, 'stale_generation');+ socket.deliver('connection_state', {+ controller_id: this.controllerId,+ assignment_revision: this.assignment.revision,+ report_within_s: this.reportWithinS,+ connection_id: connection.id,+ generation: connection.generation,+ heartbeat_interval_s: this.heartbeatIntervalS,+ });+ // Every bound agent is attached afresh on each socket, from where the+ // connection was opened.+ for (const agentId of this.bound()) {+ const cursor = connection.cursors[agentId];+ socket.deliver('agent.attached', {+ agent_id: agentId,+ from_seq: typeof cursor === 'number' ? cursor : (this.heads.get(agentId) ?? 0),+ rooms: this.rooms.get(agentId) ?? [],+ });+ }+ this.streams.set(socket, connection);+ const ping = setInterval(() => socket.deliver('ping', {}), this.heartbeatIntervalS * 1000);+ this.pings.add(ping);+ socket.onSend = (data) => {+ const message = JSON.parse(data) as { type?: string; cursors?: Record<string, number> };+ if (message.type !== 'pong') return;+ if (connection.superseded || !this.connections.has(connection.id)) {+ socket.deliver('evicted', {+ code: connection.superseded ? 'taken_over' : 'unknown_connection',+ reason: 'this connection is no longer current',+ });+ socket.end(1000);+ return;+ }+ this.beats.push(message.cursors ?? {});+ };+ socket.addEventListener('close', () => {+ clearInterval(ping);+ this.pings.delete(ping);+ this.streams.delete(socket);+ });+ }+/** Forgets the connection, as a lapsed heartbeat would: the next attach or beat is a 404. */forgetConnection(): void {if (this.current) this.connections.delete(this.current.id);@@ -249,10 +376,11 @@ export class FakeCore {this.current.superseded = true;for (const [stream, connection] of this.streams)if (connection === this.current) {- stream.write(- `event: evicted\ndata: ${JSON.stringify({ code: 'taken_over', reason: 'another connection of this controller took over' })}\n\n`- );- stream.end();+ stream.deliver('evicted', {+ code: 'taken_over',+ reason: 'another connection of this controller took over',+ });+ stream.end(1000);}}const connection: StreamConnection = {@@ -270,52 +398,6 @@ export class FakeCore {agents: this.bound(),});}- if (method === 'GET' && url.pathname === `${streamBase}/events`) {- const connection = this.connections.get(url.searchParams.get('connection_id') ?? '');- if (!connection) return this.refuse(res, 404, 'unknown_connection');- if (connection.superseded) return this.refuse(res, 409, 'taken_over');- if (Number(url.searchParams.get('generation')) !== connection.generation)- return this.refuse(res, 409, 'stale_generation');- res.writeHead(200, { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache' });- const frame = (event: string, data: unknown) =>- res.write(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`);- frame('connection_state', {- controller_id: this.controllerId,- assignment_revision: this.assignment.revision,- report_within_s: this.reportWithinS,- connection_id: connection.id,- generation: connection.generation,- heartbeat_interval_s: this.heartbeatIntervalS,- });- // Every bound agent is attached afresh on each stream, from where the- // connection was opened.- for (const agentId of this.bound()) {- const cursor = connection.cursors[agentId];- frame('agent.attached', {- agent_id: agentId,- from_seq: typeof cursor === 'number' ? cursor : (this.heads.get(agentId) ?? 0),- rooms: this.rooms.get(agentId) ?? [],- });- }- this.streams.set(res, connection);- res.on('close', () => this.streams.delete(res));- return;- }- if (method === 'POST' && url.pathname === `${streamBase}/connection/beat`) {- const beat = body as {- connection_id: string;- generation: number;- cursors: Record<string, number>;- };- const connection = this.connections.get(beat.connection_id);- if (!connection) return this.refuse(res, 404, 'unknown_connection');- if (connection.superseded) return this.refuse(res, 409, 'taken_over');- if (beat.generation !== connection.generation)- return this.refuse(res, 409, 'stale_generation');- if ('placements' in (body as object)) return this.refuse(res, 422, 'validation_error');- this.beats.push(beat.cursors);- return this.json(res, 200, { agents: this.bound() });- }if (method === 'GET' && url.pathname === `${base}/assignment`) {const etag = `"${this.assignment.revision}"`;if (req.headers['if-none-match'] === etag) {diff --git a/console/packages/agent-providers/src/host/hub-stream.ts b/console/packages/agent-providers/src/host/hub-stream.tsnew file mode 100644index 000000000..d66d6dce5--- /dev/null+++ b/console/packages/agent-providers/src/host/hub-stream.ts@@ -0,0 +1,252 @@+import {+ type AgentBridgeEvent,+ type ApprovalOutcome,+ EVICTION_TAKEN_OVER,+ type SessionCommand,+ type SwitchEventStreamDeps,+} from '@sandboxaq/switch-agent-runtime';+import { z } from 'zod';+import type { AgentEventStream, OpenAgentStream } from './agent-host';++/**+ * The agents controller's hub, over a WebSocket on the controller's loopback+ * relay: how an agent host in a process of its own (an `isolated` agent) hears+ * its agent's events. The controller holds the agent's connection to Switch+ * and hands each event to the hub; the hub hands it here, one at a time, and+ * counts it taken once this side's handler has resolved, exactly as it does+ * for an agent host in the controller's own process.+ *+ * Every request the hub makes carries an `id` and is answered with `done`;+ * `placements` from this side is answered with `result`. The agent is the one+ * the bearer token names.+ */+export const HUB_PATH = '/hub';+export const HUB_PROTOCOL = 1;++/** Closes the hub sends. */+export const HUB_CLOSE = {+ /** The controller is stopping or restarting: reconnect soon. */+ restarting: 1012,+ /** A newer agent host for the same agent took the hub over: stand down. */+ takenOver: 4409,+ /** The hello was missing, malformed, or named a protocol the hub does not speak. */+ refused: 4400,+} as const;++const placementsSchema = z.record(z.string().min(1), z.string().min(1));++export const hubClientMessageSchema = z.discriminatedUnion('type', [+ z.object({+ type: z.literal('hello'),+ protocol: z.number().int(),+ startCursor: z.number().int().nonnegative().nullable(),+ placements: placementsSchema,+ }),+ z.object({ type: z.literal('done'), id: z.number().int(), error: z.string().optional() }),+ z.object({ type: z.literal('placements'), id: z.number().int(), placements: placementsSchema }),+]);+export type HubClientMessage = z.infer<typeof hubClientMessageSchema>;++export type HubGap = Parameters<SwitchEventStreamDeps['onGap']>[0];++export type HubServerMessage =+ | { type: 'connected' }+ | { type: 'disconnected'; error: string }+ | { type: 'event'; id: number; event: AgentBridgeEvent }+ | { type: 'gap'; id: number; gap: HubGap }+ | { type: 'session_command'; id: number; command: SessionCommand }+ | { type: 'approval_outcome'; id: number; outcome: ApprovalOutcome }+ | { type: 'result'; id: number; error?: string };++/** Node's WebSocket, which, unlike a browser's, takes request headers. */+type HeaderedWebSocket = new (url: string, init: { headers: Record<string, string> }) => WebSocket;++const INITIAL_BACKOFF_MS = 250;+const MAX_BACKOFF_MS = 5_000;++/**+ * Opens the agent's stream on the controller's hub at `url`+ * (`ws://127.0.0.1:<relay port>/hub`), with the relay token the agent host+ * was given.+ */+export function openHubStream(url: string): OpenAgentStream {+ return (deps) => new HubEventStream(url, deps);+}++class HubEventStream implements AgentEventStream {+ /** The agent host, not Switch, tells a room when it starts a session for it. */+ readonly announcesSessionStarts = true;+ private started = false;+ private socket: WebSocket | null = null;+ /** The last sequence this side has handled; sent on every hello. */+ private cursor: number | null;+ private placements: Record<string, string> = {};+ private nextId = 0;+ private readonly calls = new Map<number, (error: string | undefined) => void>();+ /** Requests are handled one at a time, in the order they came. */+ private handling: Promise<void> = Promise.resolve();+ private standingDown = false;++ constructor(+ private readonly url: string,+ private readonly deps: SwitchEventStreamDeps+ ) {+ this.cursor = deps.startCursor ?? null;+ }++ start(): void {+ if (this.started) return;+ this.started = true;+ void this.run();+ }++ // Switch decides who may start a session from the agent's binding.+ setSpawnCapable(): void {}++ async replacePlacements(placements: Record<string, string>): Promise<void> {+ this.placements = { ...placements };+ const socket = this.socket;+ // Not connected: the next hello carries them.+ if (!socket || socket.readyState !== WebSocket.OPEN) return;+ const id = ++this.nextId;+ const error = await new Promise<string | undefined>((resolve) => {+ this.calls.set(id, resolve);+ socket.send(JSON.stringify({ type: 'placements', id, placements }));+ });+ if (error !== undefined) throw new Error(error);+ }++ workerCall(): Promise<unknown> {+ return Promise.reject(+ new Error('An agent run by an agents controller is not a hosted worker.')+ );+ }++ private async run(): Promise<void> {+ const { signal, log } = this.deps;+ let backoff = INITIAL_BACKOFF_MS;+ let failures = 0;+ while (!signal.aborted && !this.standingDown) {+ const { code, reason, helloed } = await this.connectOnce();+ if (signal.aborted) return;+ if (code === HUB_CLOSE.takenOver) {+ this.standingDown = true;+ this.deps.onEvicted({+ code: EVICTION_TAKEN_OVER,+ reason: reason || 'another agent host for this agent took the controller hub over',+ roomId: null,+ });+ return;+ }+ const error = reason || `the agents controller hub closed (${code})`;+ this.deps.onDisconnected?.({ error });+ if (helloed) {+ backoff = INITIAL_BACKOFF_MS;+ failures = 0;+ }+ failures++;+ // The first failure, then powers of two: a long outage costs a few lines.+ if ((failures & (failures - 1)) === 0)+ log.warn(`The agents controller hub is unreachable (${error}); retrying.`);+ const wait = code === HUB_CLOSE.restarting ? INITIAL_BACKOFF_MS : backoff;+ await sleep(Math.round(wait * (0.5 + Math.random() * 0.5)), signal);+ backoff = Math.min(backoff * 2, MAX_BACKOFF_MS);+ }+ }++ /** One socket, from open to close. */+ private connectOnce(): Promise<{ code: number; reason: string; helloed: boolean }> {+ return new Promise((resolve) => {+ const Socket = (globalThis as { WebSocket?: HeaderedWebSocket }).WebSocket;+ if (!Socket) throw new Error('The agents controller hub needs Node 22 or newer.');+ const socket = new Socket(this.url, {+ headers: { Authorization: `Bearer ${this.deps.creds.token}` },+ });+ this.socket = socket;+ let helloed = false;+ const abort = () => socket.close(1000, 'the agent host is stopping');+ this.deps.signal.addEventListener('abort', abort, { once: true });+ socket.addEventListener('open', () => {+ helloed = true;+ socket.send(+ JSON.stringify({+ type: 'hello',+ protocol: HUB_PROTOCOL,+ startCursor: this.cursor,+ placements: this.placements,+ })+ );+ });+ socket.addEventListener('message', (message) => {+ let parsed: HubServerMessage;+ try {+ parsed = JSON.parse(String(message.data)) as HubServerMessage;+ } catch {+ this.deps.log.error('Dropped a message from the agents controller hub that is not JSON');+ return;+ }+ if (parsed.type === 'result') {+ const call = this.calls.get(parsed.id);+ this.calls.delete(parsed.id);+ call?.(parsed.error);+ return;+ }+ this.handling = this.handling.then(() => this.handle(socket, parsed));+ });+ socket.addEventListener('close', (event) => {+ this.deps.signal.removeEventListener('abort', abort);+ if (this.socket === socket) this.socket = null;+ // The placements a call was sending go again with the next hello.+ for (const call of this.calls.values()) call(undefined);+ this.calls.clear();+ resolve({ code: event.code, reason: event.reason, helloed });+ });+ // A failed handshake also closes; the close says what happened.+ socket.addEventListener('error', () => {});+ });+ }++ private async handle(socket: WebSocket, message: HubServerMessage): Promise<void> {+ const { deps } = this;+ if (message.type === 'connected') return deps.onConnected?.();+ if (message.type === 'disconnected') return deps.onDisconnected?.({ error: message.error });+ if (message.type === 'result') return;+ let error: string | undefined;+ try {+ switch (message.type) {+ case 'event':+ await deps.onEvent(message.event);+ if (typeof message.event.sequence === 'number')+ this.cursor = Math.max(this.cursor ?? 0, message.event.sequence);+ break;+ case 'gap':+ await deps.onGap(message.gap);+ if (message.gap.resumedAt !== undefined) this.cursor = message.gap.resumedAt;+ break;+ case 'session_command':+ await deps.onSessionCommand?.(message.command);+ break;+ case 'approval_outcome':+ await deps.onApprovalOutcome?.(message.outcome);+ break;+ }+ } catch (failure) {+ error = failure instanceof Error ? failure.message : String(failure);+ }+ if (socket.readyState === WebSocket.OPEN)+ socket.send(JSON.stringify({ type: 'done', id: message.id, ...(error ? { error } : {}) }));+ }+}++function sleep(ms: number, signal: AbortSignal): Promise<void> {+ return new Promise((resolve) => {+ if (signal.aborted) return resolve();+ const timer = setTimeout(done, ms);+ function done() {+ clearTimeout(timer);+ signal.removeEventListener('abort', done);+ resolve();+ }+ signal.addEventListener('abort', done, { once: true });+ });+}diff --git a/console/packages/agent-providers/src/host/shared-config.ts b/console/packages/agent-providers/src/host/shared-config.tsindex 8da979f0e..b34d5cc4a 100644--- a/console/packages/agent-providers/src/host/shared-config.ts+++ b/console/packages/agent-providers/src/host/shared-config.ts@@ -117,6 +117,9 @@ export async function readSharedCredentials(config: SharedHostConfig) {SWITCH_API_ENDPOINT: z.string().min(1),SWITCH_API_TOKEN: z.string().min(1),SWITCH_AGENT_ID: z.string().min(1),+ // Set when an agents controller runs the agent host in a process of+ // its own: the hub it hears its events on instead of Switch.+ SWITCH_AGENT_HUB: z.string().min(1).optional(),}),}).parse(JSON.parse(await readFile(config.execution.credentialsPath, 'utf8'))).env;diff --git a/console/packages/agent-providers/src/host/shared-daemon.ts b/console/packages/agent-providers/src/host/shared-daemon.tsindex 50978f147..190e66ab7 100644--- a/console/packages/agent-providers/src/host/shared-daemon.ts+++ b/console/packages/agent-providers/src/host/shared-daemon.ts@@ -7,13 +7,14 @@ import { AttachmentTransfers } from './attachment-transfers';import { type ControlContext, ensureSessions, serveControl } from './control';import { OBSOLETE_BUNDLE_EXIT_CODE, WorkerObsoleteError } from './exit-codes';import { hostedWorker } from './hosted-watcher';+import { openHubStream } from './hub-stream';import { detachedSupervision, ensureSharedProcess, inProcessSupervision } from './launch';import { replaceOwner } from './ownership-lock';import { ownProcessGroup } from './process-fence';import { checkProviderReadiness } from './provider-readiness';import { adapterFor } from './server';import { HOST_EXIT_GRACE_MS, SessionLinks } from './session-channel';-import { sharedConfigSchema } from './shared-config';+import { readSharedCredentials, sharedConfigSchema } from './shared-config';import { hostSessionProcess } from './shared-host';import { superviseSharedHost } from './supervisor';import { recordWatcherHealth } from './watcher-health-file';@@ -135,6 +136,11 @@ async function main(): Promise<void> {transfers,};const hosted = await hostedWorker(config, resolve(root), context);+ // An agents controller running this agent host in a process of its own+ // names its hub: the agent's events come from there, not from Switch.+ const hub = config.execution+ ? (await readSharedCredentials(config)).SWITCH_AGENT_HUB+ : undefined;// Console reads the watcher's connection state from this file, with the// rest of the host's watcher state, rather than from the control port.const stopRecording = recordWatcherHealth(resolve(root), control);@@ -151,7 +157,7 @@ async function main(): Promise<void> {supervision,control,hosted,- openSwitchStream+ hub ? openHubStream(hub) : openSwitchStream).finally(() => stop.abort()),serveControl(resolve(root), context, stop.signal),]);diff --git a/console/packages/agent-providers/src/index.ts b/console/packages/agent-providers/src/index.tsindex 7478443d7..0e9cfb689 100644--- a/console/packages/agent-providers/src/index.ts+++ b/console/packages/agent-providers/src/index.ts@@ -65,6 +65,15 @@ export {openSwitchStream,runAgentHost,} from './host/agent-host';+export {+ HUB_CLOSE,+ HUB_PATH,+ HUB_PROTOCOL,+ type HubClientMessage,+ hubClientMessageSchema,+ type HubServerMessage,+ openHubStream,+} from './host/hub-stream';export { clearTakenOver, readTakenOver, type TakenOver } from './host/taken-over';export {recordWatcherHealth,diff --git a/console/pnpm-lock.yaml b/console/pnpm-lock.yamlindex 91ff45164..67e5f6428 100644--- a/console/pnpm-lock.yaml+++ b/console/pnpm-lock.yaml@@ -305,6 +305,9 @@ importers:'@switch-console/plugins':specifier: workspace:*version: link:../plugins+ ws:+ specifier: ^8.21.3+ version: 8.21.3zod:specifier: ^4.4.3version: 4.4.3@@ -315,6 +318,9 @@ importers:'@types/node':specifier: ^24.13.3version: 24.13.3+ '@types/ws':+ specifier: ^8.18.2+ version: 8.18.2'@typescript/native-preview':specifier: 7.0.0-dev.20260609.1version: 7.0.0-dev.20260609.1@@ -2565,6 +2571,9 @@ packages:'@types/[email protected]':resolution: {integrity: sha512-ko/gIFJRv177XgZsZcBwnqJN5x/Gien8qNOn0D5bQU/zAzVf9Zt3BlcUiLqhV9y4ARk0GbT3tnUiPNgnTXzc/Q==}+ '@types/[email protected]':+ resolution: {integrity: sha512-67MQl+fpWKVTT1NYdnmo3U4sc/xPo/zQBncVnI74qmQa0z/b+1g6iYqNmGCPbxO+zz2aklb08a0oHfegiVd0/w==}+'@typescript/[email protected]':resolution: {integrity: sha512-Yf/zHEadP/yUiWUdM/mZVfEVFJuGMf6nhRSFif0vp+FwtfGU4jmlpNF7BTJJdOHrrcWkwEJKzAoMCtEtyxhuyQ==}engines: {node: '>=16.20.0'}@@ -7423,6 +7432,10 @@ snapshots:'@types/[email protected]': {}+ '@types/[email protected]':+ dependencies:+ '@types/node': 24.13.3+'@typescript/[email protected]':optional: truediff --git a/core/switch_core/bridges/agent/api/handlers.py b/core/switch_core/bridges/agent/api/handlers.pyindex 1b29b2049..4ac585df7 100644--- a/core/switch_core/bridges/agent/api/handlers.py+++ b/core/switch_core/bridges/agent/api/handlers.py@@ -133,6 +133,8 @@ from switch_core.db.stores.api_key_store import ApiKeyStorefrom switch_core.db.stores.feature_flag_store import FeatureFlagStorefrom switch_core.feature_flags import is_known_flagfrom switch_core.gateway.known_agents import KNOWN_AGENTS+from switch_core.observability.catalogue import AGENT_CONNECTIONS_REFUSED+from switch_core.observability.metrics import metricsfrom switch_core.version import switch_core_versionlogger = logging.getLogger(__name__)@@ -193,12 +195,17 @@ async def _resolve_registration_user_id("bootstrap" if key.type == BOOTSTRAP_KEY_TYPE else "personal_key")try:- return await resolve_registration_owner_id(session, protocol.user_store, key)+ owner_id = await resolve_registration_owner_id(+ session, protocol.user_store, key+ )except RuntimeError as exc:logger.error("Agent-registration bootstrap owner resolution failed: %s", exc)raise HTTPException(status_code=503, detail="Agent registration is temporarily unavailable") from exc+ # Not held across registration, which opens its own session and calls every bridge.+ await session.commit()+ return owner_id# How the current registration authenticated. A contextvar rather than a@@ -397,6 +404,8 @@ async def register_known_agents_bulk_endpoint(),)+ # Not held across registration, which opens its own session and calls every bridge.+ await session.commit()results: list[BulkRegisterResult] = []for subagent_name, name, description in derived:# Inherited parent settings are the base; explicit request options@@ -764,8 +773,8 @@ async def poll_events(stream, with its heartbeat on `POST /connection/beat`: that is how an agentruntime built before the WebSocket connects (agent-protocol revision 7 andolder), and it is kept for those clients for a compatibility window. It- goes once no client still connects over it. Anything else falls back to- the long poll, served from the same buffer.+ goes once `switch.agents.connected{transport:sse}` stays at zero. Anything+ else falls back to the long poll, served from the same buffer.The declaration parameters are all optional and all default to None,meaning *unknown* (CHOO-1865): a client that says nothing still connects.@@ -1038,6 +1047,11 @@ async def _open_connection(return conn, frames+def _refusal_reason(exc: HTTPException) -> str:+ """The `reason` a refused open is counted under."""+ return "protocol" if isinstance(exc.__cause__, ProtocolVersionError) else "other"++async def _open_event_stream(*,agent: Agent,@@ -1063,25 +1077,29 @@ async def _open_event_stream(loop, so the two cannot drift while both exist. A refusal is the HTTPerror the socket would have sent as its `refused` frame."""- _conn, frames = await _open_connection(- agent=agent,- protocol=protocol,- config=config,- connection_id=connection_id,- scope=scope,- event_filter=event_filter,- start_from=start_from,- spawn_capable=spawn_capable,- declaration=declaration,- rooms=rooms,- expected_generation=expected_generation,- worker_capability=worker_capability,- host_boot_id=host_boot_id,- host_instance_id=host_instance_id,- worker_state_version=worker_state_version,- last_event_id=last_event_id,- transport="sse",- )+ try:+ _conn, frames = await _open_connection(+ agent=agent,+ protocol=protocol,+ config=config,+ connection_id=connection_id,+ scope=scope,+ event_filter=event_filter,+ start_from=start_from,+ spawn_capable=spawn_capable,+ declaration=declaration,+ rooms=rooms,+ expected_generation=expected_generation,+ worker_capability=worker_capability,+ host_boot_id=host_boot_id,+ host_instance_id=host_instance_id,+ worker_state_version=worker_state_version,+ last_event_id=last_event_id,+ transport="sse",+ )+ except HTTPException as exc:+ metrics().increment(AGENT_CONNECTIONS_REFUSED, {"reason": _refusal_reason(exc)})+ raisereturn StreamingResponse(sse_stream(frames),media_type="text/event-stream",@@ -1236,6 +1254,7 @@ async def connection_socket(worker_state_version=worker_state_version,)except HTTPException as exc:+ metrics().increment(AGENT_CONNECTIONS_REFUSED, {"reason": _refusal_reason(exc)})await websocket.send_json({"event": "refused",diff --git a/core/switch_core/bridges/agent/auth.py b/core/switch_core/bridges/agent/auth.pyindex edf161bbe..b0f64b458 100644--- a/core/switch_core/bridges/agent/auth.py+++ b/core/switch_core/bridges/agent/auth.py@@ -31,6 +31,8 @@ from switch_core.db.tenant_lookup import (tenant_of_api_key,)from switch_core.logging_context import log_context+from switch_core.observability.catalogue import AGENT_CONNECTIONS_REFUSED+from switch_core.observability.metrics import metricsfrom switch_core.tenant_context import tenant_scopelogger = logging.getLogger(__name__)@@ -78,7 +80,7 @@ _NOT_AN_AGENT_SEGMENT = frozenset({"rooms", "feature-flags"})# Registration: a controller registers nothing, so its token is refused here._REGISTRATION_SEGMENTS = frozenset({"register-known", "register-known-bulk"})# The connection surface a controller serves its agents itself, from its own-# stream (`GET /v1/controllers/{id}/events`). A controller-backed agent has no+# connection (`/v1/controllers/{id}/connection/ws`). A controller-backed agent has no# connection of its own, and the legacy heartbeats would make it look live from# a second source._SERVED_ON_THE_CONTROLLER_STREAM = re.compile(@@ -86,6 +88,11 @@ _SERVED_ON_THE_CONTROLLER_STREAM = re.compile()+# The agent connection's socket, whose refusals are counted as connections+# refused. Any other path's 401 is an ordinary failed request.+_AGENT_CONNECTION_PATH = re.compile(r"^/agents/[^/]+/connection/ws$")++class OIDCTokenValidator:def __init__(self,@@ -425,7 +432,7 @@ class BearerAuthMiddleware:await _controller_refusal(MANAGED_BY_CONTROLLER,"A controller receives its agents' events on its own stream, "- "GET /v1/controllers/{id}/events; this agent route is not "+ "/v1/controllers/{id}/connection/ws; this agent route is not ""served to it.",409,)(scope, receive, send)@@ -589,6 +596,8 @@ async def _unauthorized(connection uses for every refusal."""if scope["type"] == "websocket":+ if _AGENT_CONNECTION_PATH.match(scope.get("path", "")):+ metrics().increment(AGENT_CONNECTIONS_REFUSED, {"reason": "unauthorized"})await receive() # the client's websocket.connectawait send({"type": "websocket.accept"})await send({"type": "websocket.close", "code": 4401, "reason": reason})diff --git a/core/switch_core/bridges/agent/protocol/agent_connections.py b/core/switch_core/bridges/agent/protocol/agent_connections.pyindex 182535d25..784299116 100644--- a/core/switch_core/bridges/agent/protocol/agent_connections.py+++ b/core/switch_core/bridges/agent/protocol/agent_connections.py@@ -31,7 +31,7 @@ from contextlib import asynccontextmanagerfrom dataclasses import dataclass, fieldfrom typing import Any, Literal-from switch_core.artifacts import contract_range+from switch_core.artifacts import ARTIFACT_VERSIONS, contract_rangefrom switch_core.bridges.agent.protocol.controller_presence import ControllerPresencefrom switch_core.bridges.agent.protocol.hosted_workers import (IDLE_FRESH_FOR_SECONDS,@@ -55,7 +55,10 @@ from switch_core.bridges.agent.protocol.liveness import TAKEN_OVER as TAKEN_OVERfrom switch_core.bridges.agent.protocol.liveness import CloseCode as CloseCodefrom switch_core.bridges.agent.protocol.liveness import Closure as Closurefrom switch_core.logging_context import log_context-from switch_core.observability.catalogue import AGENT_CONNECTIONS_EXPIRED+from switch_core.observability.catalogue import (+ AGENT_CONNECTIONS_EXPIRED,+ AGENT_CONNECTIONS_OPENED,+)from switch_core.observability.metrics import metricslogger = logging.getLogger(__name__)@@ -64,7 +67,7 @@ Scope = Literal["single", "all"]DeliveryFilter = Literal["all", "addressed"]# How a connection's stream reaches its client. `sse` is the event stream an# agent runtime built before the WebSocket still asks for, served for a-# compatibility window and told apart so we can see those clients drain.+# compatibility window and counted apart so we can see those clients drain.Transport = Literal["websocket", "sse"]# Refuse a client that cannot meet this server's protocol rather than degrading@@ -273,6 +276,11 @@ class WorkerAlreadyAttachedError(ConnectionError_):self.agent_id = agent_id+# How many closed connection ids are remembered for the reconnect counter.+# Far more than the live fleet, so a lapse storm is still recognised when its+# agents come back.+_RECENTLY_CLOSED_LIMIT = 10_000+#: The hosted launch moved to a newer revision than the one this worker was#: attached for. Terminal: the worker's capability is obsolete, and only a#: restart onto the current bundle can attach again.@@ -359,6 +367,20 @@ class ClientDeclaration:}+def client_label(declaration: ClientDeclaration) -> str:+ """The declared artifact as a metric value: a registered name, or a bucket.++ The client says what it is, so only names in the artifact registry pass+ through. The declared version is never a label: it is the client's own+ string and unbounded, and stays in the `[CONN] opened` log line instead.+ """+ if not declaration.artifact:+ return "unknown"+ if declaration.artifact in ARTIFACT_VERSIONS:+ return declaration.artifact+ return "other"++class ProtocolVersionError(ConnectionError_):"""A client's declared agent-protocol range cannot meet this server's.@@ -447,6 +469,18 @@ class AgentConnection:return self.closure is None and (now - self.last_beat) < HEARTBEAT_TTL_SECONDS+# The `(transport, client)` an agent run by an agents controller counts under:+# it travels on its controller's connection, whatever that connection uses.+CONTROLLER_LABEL = ("controller", "agents-controller")+++def connection_transport(conn: AgentConnection) -> str:+ """`websocket` or `sse` while a stream is attached, by what it travels over;+ `detached` while it has dropped and the connection waits out its heartbeat+ window for a reconnect."""+ return conn.stream_transport if conn.stream_attached else "detached"++class AgentConnectionRegistry:"""The live set of agent connections. Authoritative, in memory."""@@ -484,6 +518,10 @@ class AgentConnectionRegistry:# presence is their controller's. Every presence question below asks# it first for those agents, and this registry for the rest.self.controllers = ControllerPresence(on_bound=self._close_for_controller)+ # Why each recently closed connection id closed, so its next open can+ # be counted as a return after a lapse rather than a first connect.+ # Insertion-ordered and capped: only the counter reads it.+ self._recently_closed: dict[str, str] = {}def _new_incarnation(self) -> int:"""The next never-before-used incarnation number.@@ -580,6 +618,7 @@ class AgentConnectionRegistry:# what is on the other end of it need not.existing.declaration = declarationself._reset_worker(existing)+ metrics().increment(AGENT_CONNECTIONS_OPENED, {"kind": "reattach"})logger.info("[CONN] reattached agent=%s connection=%s scope=%s generation=%s ""transport=%s",@@ -612,6 +651,10 @@ class AgentConnectionRegistry:)self._by_id[connection_id] = connowned.add(connection_id)+ metrics().increment(+ AGENT_CONNECTIONS_OPENED,+ {"kind": self._recently_closed.pop(connection_id, "fresh")},+ )logger.info("[CONN] opened agent=%s connection=%s scope=%s filter=%s spawn=%s ""client=%s version=%s protocol=%s transport=%s",@@ -674,6 +717,14 @@ class AgentConnectionRegistry:self._drop_placements(conn)self.relays.fail_connection(conn.id, None)conn.wake.set()+ lapsed = closure is HEARTBEAT_LAPSED+ self._remember_closed(connection_id, "after_lapse" if lapsed else "after_close")+ if lapsed:+ # Counted here rather than in the sweep: a lapse is also noticed+ # when the dead connection is next used (`require`) and by its own+ # event stream, and counting only the sweep under-reported lapses+ # against the reconnects that follow them.+ metrics().increment(AGENT_CONNECTIONS_EXPIRED, {})# `beats` and the age separate the two ways a connection dies, which# otherwise look identical in the log: a client that never beat at all# (beats=0 — it is not running the heartbeat, or cannot reach us) versus@@ -724,6 +775,12 @@ class AgentConnectionRegistry:),)+ def _remember_closed(self, connection_id: str, kind: str) -> None:+ self._recently_closed.pop(connection_id, None)+ self._recently_closed[connection_id] = kind+ while len(self._recently_closed) > _RECENTLY_CLOSED_LIMIT:+ del self._recently_closed[next(iter(self._recently_closed))]+def sweep(self) -> list[AgentConnection]:"""Close connections whose heartbeat has lapsed. Returns those closed."""now = time.monotonic()@@ -737,10 +794,6 @@ class AgentConnectionRegistry:gone = self.close(conn.id, HEARTBEAT_LAPSED)if gone is not None:closed.append(gone)- if closed:- # A gauge of live connections cannot show churn: agents- # reconnecting as fast as they expire hold it perfectly flat.- metrics().increment(AGENT_CONNECTIONS_EXPIRED, {}, float(len(closed)))return closed# ------------------------------------------------------------------@@ -1376,6 +1429,26 @@ class AgentConnectionRegistry:conn.agent_id for conn in self._by_id.values() if conn.is_alive(now)} | self.controllers.live_agent_ids()+ def live_agents_by_transport(self) -> dict[tuple[str, str], int]:+ """Agents with a live connection, per `(transport, client)` label.++ Counted like `live_agent_ids`: an agent is one, however many+ connections it holds, but an agent connected two different ways counts+ once under each, so the labels can sum to more than the agent count.+ An agent run by an agents controller has no connection of its own and+ counts under `CONTROLLER_LABEL` while its controller is live.+ """+ now = time.monotonic()+ agents: dict[tuple[str, str], set[str]] = {}+ for conn in self._by_id.values():+ if conn.is_alive(now):+ key = (connection_transport(conn), client_label(conn.declaration))+ agents.setdefault(key, set()).add(conn.agent_id)+ controlled = self.controllers.live_agent_ids()+ if controlled:+ agents.setdefault(CONTROLLER_LABEL, set()).update(controlled)+ return {key: len(ids) for key, ids in agents.items()}+def live_connection_ids(self) -> set[str]:"""Every connection currently alive, by id.diff --git a/core/switch_core/bridges/agent/protocol/controller_stream.py b/core/switch_core/bridges/agent/protocol/controller_stream.pyindex eee014858..195e350d2 100644--- a/core/switch_core/bridges/agent/protocol/controller_stream.py+++ b/core/switch_core/bridges/agent/protocol/controller_stream.py@@ -1,4 +1,5 @@-"""One event stream per agents controller (`GET /v1/controllers/{id}/events`).+"""One stream per agents controller, sent on its socket+(`/v1/controllers/{id}/connection/ws`).A read-side merge over the per-agent `EventBuffer`: every agent bound to thecontroller is read from its own cursor, with nothing filtered (the controller@@ -28,17 +29,17 @@ of its agents' watchers exactly the stream it would have had from Switch:- `evicted {code, reason}`: the stream ends. `taken_over` is terminal for theclient that receives it; the rest are recovered by opening again.-A keepalive comment is written whenever nothing else has been for the-keepalive interval, only so idle proxies keep the connection.+Frames are `{"event", "data"}` dicts, which the socket sends as JSON. When+nothing has happened for the idle interval the stream yields `IDLE`, which is+not sent: it only lets the loop look at the connection again."""from __future__ import annotationsimport asyncio-import jsonimport loggingimport time-from collections.abc import AsyncIterator, Awaitable, Callable+from collections.abc import AsyncGenerator, Awaitable, Callablefrom dataclasses import dataclass, fieldfrom typing import TYPE_CHECKING, Any, Protocol@@ -59,15 +60,12 @@ if TYPE_CHECKING:logger = logging.getLogger(__name__)-KEEPALIVE_INTERVAL_SECONDS = 15.0-KEEPALIVE = b": keepalive\n\n"+IDLE_INTERVAL_SECONDS = 15.0CATCH_UP_BATCH = 200-STREAM_HEADERS = {- "Cache-Control": "no-cache",- "Connection": "keep-alive",- "X-Accel-Buffering": "no",-}+ControllerFrame = dict[str, Any]+# Yielded when nothing happened for the idle interval; never sent.+IDLE: ControllerFrame = {"event": "idle", "data": {}}# The nudge after which nothing more is written: the controller stops every# agent on it and exits.@@ -85,10 +83,8 @@ class ControllerNudges(Protocol):def close(self) -> None: ...-def frame(event: str, data: dict[str, Any]) -> bytes:- return (- f"event: {event}\ndata: {json.dumps(data, separators=(',', ':'))}\n\n"- ).encode()+def frame(event: str, data: dict[str, Any]) -> ControllerFrame:+ return {"event": event, "data": data}@dataclass@@ -117,8 +113,8 @@ def controller_event_stream(nudges: ControllerNudges,opening: dict[str, Any],rooms_of: Callable[[str], Awaitable[set[str]]],- keepalive_seconds: float,-) -> AsyncIterator[bytes]:+ idle_seconds: float,+) -> AsyncGenerator[ControllerFrame]:"""The stream for an open controller connection whose stream was justattached as `stream_token`. Takes ownership of `nudges` and closes it."""return _ControllerStream(@@ -129,7 +125,7 @@ def controller_event_stream(approvals=approvals,nudges=nudges,rooms_of=rooms_of,- keepalive_seconds=keepalive_seconds,+ idle_seconds=idle_seconds,).run(opening)@@ -144,7 +140,7 @@ class _ControllerStream:approvals: ApprovalOutcomes | None,nudges: ControllerNudges,rooms_of: Callable[[str], Awaitable[set[str]]],- keepalive_seconds: float,+ idle_seconds: float,) -> None:self._conn = connself._token = stream_token@@ -153,11 +149,11 @@ class _ControllerStream:self._approvals = approvalsself._nudges = nudgesself._rooms_of = rooms_of- self._keepalive = keepalive_seconds+ self._idle = idle_secondsself._tenant_id = current_tenant_id()self._attached: dict[str, _Attached] = {}- async def run(self, opening: dict[str, Any]) -> AsyncIterator[bytes]:+ async def run(self, opening: dict[str, Any]) -> AsyncGenerator[ControllerFrame]:conn = self._conntry:yield frame("connection_state", opening)@@ -213,7 +209,7 @@ class _ControllerStream:continueif not await self._wait():- yield KEEPALIVE+ yield IDLEfinally:for attached in self._attached.values():if attached.unsubscribe is not None:@@ -239,11 +235,11 @@ class _ControllerStream:return conn.closurereturn None- def _detach_departed(self) -> list[bytes]:+ def _detach_departed(self) -> list[ControllerFrame]:conn = self._connowed = self._presence.take_detached(conn)bound = self._presence.agents_of(conn.controller_id)- frames: list[bytes] = []+ frames: list[ControllerFrame] = []for agent_id in sorted(self._attached):if agent_id in bound and agent_id not in owed:continue@@ -264,9 +260,9 @@ class _ControllerStream:)return frames- async def _attach_arrivals(self) -> list[bytes]:+ async def _attach_arrivals(self) -> list[ControllerFrame]:conn = self._conn- frames: list[bytes] = []+ frames: list[ControllerFrame] = []for agent_id in sorted(self._presence.agents_of(conn.controller_id) - set(self._attached)):@@ -357,8 +353,8 @@ class _ControllerStream:)attached.resync = True- async def _owed_outcomes(self) -> list[bytes]:- frames: list[bytes] = []+ async def _owed_outcomes(self) -> list[ControllerFrame]:+ frames: list[ControllerFrame] = []for attached in list(self._attached.values()):if attached.resync and self._approvals is not None:attached.resync = False@@ -377,7 +373,7 @@ class _ControllerStream:)return frames- def _deliver(self, attached: _Attached) -> list[bytes]:+ def _deliver(self, attached: _Attached) -> list[ControllerFrame]:agent_id = attached.agent_idfor room_id in self._presence.rooms(agent_id):self._buffer.ensure_counting(@@ -409,7 +405,7 @@ class _ControllerStream:# would find work it can never read and spin.attached.cursor = max(attached.cursor, self._buffer.head(agent_id))return []- frames: list[bytes] = []+ frames: list[ControllerFrame] = []for item in pending:data = item.event.model_dump(mode="json")data["sequence"] = item.seq@@ -441,13 +437,13 @@ class _ControllerStream:async def _wait(self) -> bool:"""Wait for an event, a nudge or a change to the connection. False when- the keepalive interval passed with none of them."""+ the idle interval passed with none of them."""bells = [self._buffer.doorbell(agent_id) for agent_id in self._attached]for bell in bells:bell.clear()self._conn.wake.clear()# Re-checked after clearing: anything that arrived between the last- # read and the clear would otherwise wait for the keepalive timeout.+ # read and the clear would otherwise wait for the idle timeout.if self._has_work() or self._nudges.wake.is_set():return Truewaiters = [@@ -456,7 +452,7 @@ class _ControllerStream:]try:done, _ = await asyncio.wait(- waiters, timeout=self._keepalive, return_when=asyncio.FIRST_COMPLETED+ waiters, timeout=self._idle, return_when=asyncio.FIRST_COMPLETED)return bool(done)finally:diff --git a/core/switch_core/bridges/agent/protocol/event_buffer.py b/core/switch_core/bridges/agent/protocol/event_buffer.pyindex 482dd7dce..66bcd3b73 100644--- a/core/switch_core/bridges/agent/protocol/event_buffer.py+++ b/core/switch_core/bridges/agent/protocol/event_buffer.py@@ -37,7 +37,11 @@ from dataclasses import dataclassfrom switch_core.bridges.agent.protocol.types import AgentEventfrom switch_core.logging_context import log_context-from switch_core.observability.catalogue import AGENT_EVENTS_DROPPED+from switch_core.observability.catalogue import (+ AGENT_BUFFER_SCAN_DURATION,+ AGENT_BUFFER_SCANNED,+ AGENT_EVENTS_DROPPED,+)from switch_core.observability.metrics import metricslogger = logging.getLogger(__name__)@@ -52,6 +56,13 @@ DEFAULT_MAX_EVENTS_PER_AGENT = 2000# later is told it has a gap rather than handed a partial stream.DEFAULT_RETENTION_SECONDS = 15 * 60+# An agent past the cap drops an event on every enqueue, and a warning per drop+# is a log line per delivered message. One per agent per interval says the same.+OVERFLOW_WARNING_INTERVAL_SECONDS = 60.0++_READ = {"operation": "read"}+_UNREAD = {"operation": "unread"}+# Why an unread count is absent or incomplete. Never a bare zero: a reader that# is told nothing went by must be able to trust it.NO_BASELINE = "nothing recorded what you had already seen in this room"@@ -204,6 +215,10 @@ class EventBuffer:# restart is gone from the buffer, so a count taken from what is left# would be a floor presented as a total.self._restarted: set[str] = set()+ # agent -> when its last overflow warning was logged, and how many+ # overflow drops have gone unlogged since.+ self._overflow_warned_at: dict[str, float] = {}+ self._overflow_unlogged: dict[str, int] = {}# ------------------------------------------------------------------# Producing@@ -264,11 +279,14 @@ class EventBuffer:readable when its window ends is relied on to decide how long a recordabout it has to be kept."""+ started = time.perf_counter()self._trim(agent_id)self._check_cursor(agent_id, after_seq)out: list[BufferedEvent] = []- for item in self._events.get(agent_id, ()):+ events = self._events.get(agent_id, ())+ scanned = len(events)+ for index, item in enumerate(events):if item.seq <= after_seq:continueif rooms is not None and item.room_id not in rooms:@@ -277,7 +295,9 @@ class EventBuffer:continueout.append(item)if limit is not None and len(out) >= limit:+ scanned = index + 1break+ self._record_scan(_READ, scanned, started)return outasync def wait(self, agent_id: str, timeout: float) -> None:@@ -469,14 +489,17 @@ class EventBuffer:if baseline is None:return Unread(count=None, reason=RESTARTED)+ started = time.perf_counter()+ events = self._events.get(agent_id, ())count = sum(1- for item in self._events.get(agent_id, ())+ for item in eventsif item.room_id == room_idand baseline < item.seq <= through_seqand item.event.type == "message"and not item.notifiable)+ self._record_scan(_UNREAD, len(events), started)dropped = self._dropped_through.get(agent_id, {}).get(room_id, 0)if dropped > baseline:return Unread(count=count, reason=COUNTED_FROM_A_HOLE)@@ -512,6 +535,8 @@ class EventBuffer:self._dropped_through.pop(agent_id, None)self._counting.pop(agent_id, None)self._restarted.discard(agent_id)+ self._overflow_warned_at.pop(agent_id, None)+ self._overflow_unlogged.pop(agent_id, None)def _begin(self, agent_id: str, room_id: str, occupant: Reader, baseline: int@@ -725,6 +750,16 @@ class EventBuffer:# Internals# ------------------------------------------------------------------+ @staticmethod+ def _record_scan(operation: dict[str, str], scanned: int, started: float) -> None:+ registry = metrics()+ registry.observe(+ AGENT_BUFFER_SCAN_DURATION,+ operation,+ (time.perf_counter() - started) * 1000.0,+ )+ registry.observe(AGENT_BUFFER_SCANNED, operation, scanned)+def _wake(self, agent_id: str) -> None:notify = self._notify.get(agent_id)if notify is not None:@@ -766,23 +801,44 @@ class EventBuffer:item = events.popleft()dropped[item.room_id] = item.seqmetrics().increment(AGENT_EVENTS_DROPPED, {"reason": "overflow"}, overflow)- # The agent is a field as well as being in the message: this fires- # from the buffer's own bookkeeping rather than from anything the- # agent called, so there is no request context to inherit it from,- # and "which agent is falling behind" is the only question this- # line is ever read to answer.- with log_context(agent_id=agent_id):- logger.warning(- "[EVENT-BUF] agent=%s exceeded %s buffered events; dropped "- "through seq=%s in rooms %s — readers resuming from before "- "this will be told they missed events",- agent_id,- self._max_events,- max(dropped.values()),- ", ".join(sorted(dropped)),- )+ self._warn_overflow(agent_id, overflow, dropped)if dropped:markers = self._dropped_through.setdefault(agent_id, {})for room_id, seq in dropped.items():markers[room_id] = max(seq, markers.get(room_id, 0))++ def _warn_overflow(+ self, agent_id: str, overflow: int, dropped: dict[str, int]+ ) -> None:+ """Log that the cap forced events out, at most once per interval per agent.++ The drops in between are not lost: the metric still counts each one, and+ the next line says how many went unlogged.+ """+ now = time.monotonic()+ last = self._overflow_warned_at.get(agent_id)+ if last is not None and now - last < OVERFLOW_WARNING_INTERVAL_SECONDS:+ self._overflow_unlogged[agent_id] = (+ self._overflow_unlogged.get(agent_id, 0) + overflow+ )+ return+ unlogged = self._overflow_unlogged.pop(agent_id, 0)+ self._overflow_warned_at[agent_id] = now+ # The agent is a field as well as being in the message: this fires+ # from the buffer's own bookkeeping rather than from anything the+ # agent called, so there is no request context to inherit it from,+ # and "which agent is falling behind" is the only question this+ # line is ever read to answer.+ with log_context(agent_id=agent_id):+ logger.warning(+ "[EVENT-BUF] agent=%s exceeded %s buffered events; dropped "+ "through seq=%s in rooms %s, plus %s dropped since the last "+ "warning; readers resuming from before this will be told they "+ "missed events",+ agent_id,+ self._max_events,+ max(dropped.values()),+ ", ".join(sorted(dropped)),+ unlogged,+ )diff --git a/core/switch_core/bridges/agent/protocol/stream.py b/core/switch_core/bridges/agent/protocol/stream.pyindex a872e768c..fe101188d 100644--- a/core/switch_core/bridges/agent/protocol/stream.py+++ b/core/switch_core/bridges/agent/protocol/stream.py@@ -103,8 +103,8 @@ async def sse_stream(frames: AsyncGenerator[Frame]) -> AsyncIterator[bytes]:"""The connection's frames as a `text/event-stream` body.Kept for clients built before the WebSocket (agent-protocol revision 7 and- older), for a compatibility window: it goes once no client still connects- over it.+ older), for a compatibility window: it goes once+ `switch.agents.connected{transport:sse}` stays at zero."""try:async for frame in frames:diff --git a/core/switch_core/clients/agent_consumer.py b/core/switch_core/clients/agent_consumer.pyindex 69f9684f1..54dee2e16 100644--- a/core/switch_core/clients/agent_consumer.py+++ b/core/switch_core/clients/agent_consumer.py@@ -708,7 +708,9 @@ class AgentConsumer(Consumer[AgentActor]):if hosted is not None and not hosted.deliver:return- logger.debug("Enqueuing event %s", agent_event.model_dump_json(indent=2))+ # Serialising runs once per agent per message, so only when it will be seen.+ if logger.isEnabledFor(logging.DEBUG):+ logger.debug("Enqueuing event %s", agent_event.model_dump_json(indent=2))self._event_buffer.enqueue(self.agent.id,meta.room_id,@@ -932,7 +934,10 @@ class AgentConsumer(Consumer[AgentActor]):if hosted is not None and not hosted.deliver:return- logger.debug("Enqueuing media event %s", agent_event.model_dump_json(indent=2))+ if logger.isEnabledFor(logging.DEBUG):+ logger.debug(+ "Enqueuing media event %s", agent_event.model_dump_json(indent=2)+ )self._event_buffer.enqueue(self.agent.id,meta.room_id,diff --git a/core/switch_core/clients/client_factory.py b/core/switch_core/clients/client_factory.pyindex 5fd33da4a..8bc782f0f 100644--- a/core/switch_core/clients/client_factory.py+++ b/core/switch_core/clients/client_factory.py@@ -19,6 +19,7 @@ from switch_core.transport import MessageTransportfrom switch_core.transport.ephemeral import EphemeralBusfrom switch_core.transport.invites import InviteBusfrom switch_core.transport.postgres import PostgresTransport+from switch_core.transport.room_cache import RoomDeliveryCachelogger = logging.getLogger(__name__)@@ -37,6 +38,7 @@ class ClientFactory:listener: MessageListener,invites: InviteBus,ephemeral: EphemeralBus,+ room_cache: RoomDeliveryCache,) -> None:self._client_store = client_storeself._session_factory = session_factory@@ -48,6 +50,8 @@ class ClientFactory:self._listener = listenerself._invites = invitesself._ephemeral = ephemeral+ # One for the process, so every member of a room shares it.+ self._room_cache = room_cacheself._registry: dict[str,tuple[@@ -122,4 +126,5 @@ class ClientFactory:listener=self._listener,invites=self._invites,ephemeral=self._ephemeral,+ room_cache=self._room_cache,)diff --git a/core/switch_core/config.py b/core/switch_core/config.pyindex 3bd15fe50..0b4516dde 100644--- a/core/switch_core/config.py+++ b/core/switch_core/config.py@@ -506,6 +506,16 @@ class SwitchConfig(BaseSettings):# `report_within_s`; a controller that has not reported for three of these# is shown as unknown and refused new placements.controller_status_interval_seconds: int = 60+ # Share one read of a room's new messages between every client in it+ # (`transport/room_cache.py`). Without it, a room of N agents reads each+ # new page N times, which is what exhausted the pool in a restart burst.+ # The limits bound memory, not correctness: anything outside them is read+ # from the database, as every client did before the cache.+ room_delivery_cache_max_bytes: int = 64 * 1024 * 1024+ room_delivery_cache_max_rooms: int = 5000+ # At least one delivery page (200), or a fill could not be held.+ room_delivery_cache_max_rows_per_room: int = 1000+ room_delivery_cache_max_age_seconds: float = 300.0# Postgres terminates a connection that sits inside an open transaction# without executing anything for longer than this (a Postgres interval suchdiff --git a/core/switch_core/db/models.py b/core/switch_core/db/models.pyindex 51547f0b7..063502810 100644--- a/core/switch_core/db/models.py+++ b/core/switch_core/db/models.py@@ -2469,12 +2469,25 @@ class Message(TenantScoped, Base):)id: Mapped[str] = mapped_column(Text, primary_key=True, default=_uuid)- # Position within the room, from 1, and the cursor the read path pages on.- # Assigned by MessageStore.create under a per-room lock rather than by a- # sequence: a sequence hands out numbers when a statement runs, not when it- # commits, so a row can commit after one with a higher number and a reader- # paging on `seq > n` would step straight over it. See the store for the- # argument in full.+ # The message's position in its room's log: the room's offset, in Kafka+ # terms (room = partition, seq = offset, a reader's cursor = its consumer+ # offset). It means something only within its room: unique per room, and+ # two rooms can each have a message 5.+ #+ # Live messages (MessageStore.create) count up from 1 with no gaps, in+ # commit order, and a number is never renumbered or reused. Reconstructed+ # history (MessageStore.create_historical) counts down from -1, below the+ # room's oldest, so seq can be negative and ordering by it still walks the+ # room in the order things happened.+ #+ # A reader's cursor is the last seq it has, and it reads `seq > cursor`.+ # Commit order is what makes that safe: nothing can later commit behind a+ # cursor. A Postgres sequence would not give it, because it hands out+ # numbers when a statement runs rather than when it commits. See+ # MessageStore._next_seq for the argument in full.+ #+ # Not the `sequence` an agent sees on its event stream: that one is the+ # event buffer's, numbered per agent across all its rooms.seq: Mapped[int] = mapped_column(BigInteger, nullable=False)room_id: Mapped[str] = mapped_column(Text, nullable=False)# Global on purpose: a random, globally-unique identifier — scoping itdiff --git a/core/switch_core/db/stores/delivery_cursor_store.py b/core/switch_core/db/stores/delivery_cursor_store.pyindex 5506bb06e..5d3bd45b1 100644--- a/core/switch_core/db/stores/delivery_cursor_store.py+++ b/core/switch_core/db/stores/delivery_cursor_store.py@@ -19,9 +19,10 @@ class DeliveryCursorStore:async def position(self, session: AsyncSession, agent_id: str, room_id: str) -> int:"""How far this agent has been delivered, 0 if it never has.- Zero rather than None because `seq` starts at 1: an agent with no- cursor is behind the room's first message, which is the same statement- the number already makes.+ Zero rather than None because live messages are numbered from 1: an+ agent with no cursor is behind the room's first live message, which is+ the same statement the number already makes. Reconstructed history sits+ below zero and is not delivered."""result = await session.execute(select(DeliveryCursor.last_seq).where(diff --git a/core/switch_core/db/stores/message_store.py b/core/switch_core/db/stores/message_store.pyindex 42638fa45..90fa3c7a8 100644--- a/core/switch_core/db/stores/message_store.py+++ b/core/switch_core/db/stores/message_store.py@@ -92,6 +92,10 @@ class MessageStore:async def _next_seq(self, session: AsyncSession, room_id: str) -> int:"""The next position in this room, allocated in commit order.+ One above the room's highest, so live positions run 1, 2, 3 with no+ gaps: a transaction that rolls back takes its number with it, and the+ next writer gets the same one.+A database sequence — `Identity`, `SERIAL`, `nextval` — is the obviousway to number rows and the wrong one here. It hands out a number whenthe INSERT runs, not when the transaction commits, and it does sodiff --git a/core/switch_core/gateway/agents.py b/core/switch_core/gateway/agents.pyindex 3dcceaace..4e83fe3ae 100644--- a/core/switch_core/gateway/agents.py+++ b/core/switch_core/gateway/agents.py@@ -110,6 +110,8 @@ async def delete_agent_by_name()if hosted_launch_of(agent.metadata_) is not None:raise HTTPException(status_code=409, detail=CLOUD_AGENT_DELETE_REFUSED)+ # Not held across the delete, which opens its own sessions and calls every bridge.+ await session.commit()try:await protocol.delete_agent(agent_name=agent_name)except ValueError as exc:@@ -143,6 +145,8 @@ async def delete_agent()if hosted_launch_of(agent.metadata_) is not None:raise HTTPException(status_code=409, detail=CLOUD_AGENT_DELETE_REFUSED)+ # Not held across the delete, which opens its own sessions and calls every bridge.+ await session.commit()try:await protocol.delete_agent(agent_id=agent_id)except ValueError as exc:@@ -299,6 +303,8 @@ async def register_known_subagents(),)+ # Not held across registration, which opens its own session and calls every bridge.+ await session.commit()results: list[BulkRegisterResult] = []for d in derived:try:diff --git a/core/switch_core/gateway/auth.py b/core/switch_core/gateway/auth.pyindex df0a63e7b..08532287e 100644--- a/core/switch_core/gateway/auth.py+++ b/core/switch_core/gateway/auth.py@@ -1,5 +1,6 @@from __future__ import annotations+import asyncioimport datetimeimport loggingfrom collections.abc import AsyncIterator@@ -68,6 +69,23 @@ def verify_password(password: str, password_hash: str | None) -> bool:return result+async def hash_password_off_loop(password: str) -> str:+ """`hash_password` on a worker thread.++ bcrypt is slow on purpose, a few hundred milliseconds of CPU per call, and+ it holds the event loop for all of it: on the loop, every other request in+ the process waits behind one login. Call it with no transaction open, too,+ or the connection waits with them.+ """+ return await asyncio.to_thread(hash_password, password)+++async def verify_password_off_loop(password: str, password_hash: str | None) -> bool:+ """`verify_password` on a worker thread, for the same reason as+ `hash_password_off_loop`."""+ return await asyncio.to_thread(verify_password, password, password_hash)++def create_jwt(user_id: str, email: str, role: str, keyring: Keyring, tenant_id: str | None) -> str:@@ -716,11 +734,15 @@ async def get_tenant_is_admin(should be built from — see ``UserStore.administers`` for what it actuallychecks.- Also callable directly (not just as a FastAPI dependency) from any- handler or service function that already holds a bound `session`, the- caller's `User`, and a `UserStore`.+ Ends the read's transaction before returning, the way `get_current_user`+ does, so the connection is back in the pool before the route body runs. A+ route that went on to open a second session, or to call a messaging+ platform, would otherwise do it holding this one. So it must not be called+ on a session with writes pending."""- return await user_store.administers(session, user)+ is_admin = await user_store.administers(session, user)+ await session.commit()+ return is_adminasync def get_tenant_is_owner(@@ -732,9 +754,12 @@ async def get_tenant_is_owner(Strictly narrower than ``get_tenant_is_admin`` — see``authz.owns_tenant``. A route that changes the ownership set needs both:- ``require_tenant_admin`` to get in, this to make the call.+ ``require_tenant_admin`` to get in, this to make the call. Ends its read's+ transaction for the same reason ``get_tenant_is_admin`` does."""- return await user_store.owns(session, user)+ is_owner = await user_store.owns(session, user)+ await session.commit()+ return is_ownerasync def require_tenant_admin(diff --git a/core/switch_core/gateway/auth_routes.py b/core/switch_core/gateway/auth_routes.pyindex cd8f88bce..1fc547ff2 100644--- a/core/switch_core/gateway/auth_routes.py+++ b/core/switch_core/gateway/auth_routes.py@@ -24,12 +24,12 @@ from switch_core.gateway.auth import (describe_session_state,get_authenticated_session,get_current_user,- hash_password,+ hash_password_off_loop,initial_tenant_claim,list_tenant_memberships,require_admin,set_session_cookie,- verify_password,+ verify_password_off_loop,)from switch_core.gateway.dependencies import (get_bridge_store,@@ -125,11 +125,12 @@ async def login(raise HTTPException(status_code=403, detail="Password login is disabled")user = await user_store.get_by_email(session, req.email)- if user is None or not verify_password(req.password, user.password_hash):- raise HTTPException(status_code=401, detail="Invalid email or password")- # Ends the read so its connection goes back to the pool before the- # membership lookup takes one: a request holds one connection at a time.+ # One connection at a time, and none held across bcrypt.await session.commit()+ if user is None or not await verify_password_off_loop(+ req.password, user.password_hash+ ):+ raise HTTPException(status_code=401, detail="Invalid email or password")set_session_cookie(response,@@ -163,6 +164,30 @@ async def _prewarm(return SignupMachine(status="starting", reason=None)+async def _refuse_signup(+ session: AsyncSession, user_store: UserStore, email: str, config: SwitchConfig+) -> None:+ """Raise if this sign-up must be refused: the hourly cap is reached, or the+ email is taken. Rolls back before raising, so a refusal never leaves the+ transaction, or an advisory lock taken in it, open."""+ created, retry_after = await user_store.created_in_last_hour(session)+ if created >= config.gateway_signup_max_per_hour:+ logger.warning(+ "Refused sign-up: %d users created in the last hour (cap %d)",+ created,+ config.gateway_signup_max_per_hour,+ )+ await session.rollback()+ raise HTTPException(+ status_code=429,+ detail="Too many sign-ups on this server in the last hour. Try again later.",+ headers={"Retry-After": str(retry_after)},+ )+ if await user_store.get_by_email(session, email) is not None:+ await session.rollback()+ raise HTTPException(status_code=409, detail="Email already registered")++@router.post("/auth/signup", status_code=201)async def signup(req: SignupRequest,@@ -185,31 +210,22 @@ async def signup()with tenant_scope(TENANT_ZERO_ID):+ # Checked before the hash so a refusal stays cheap, and again under the+ # lock after it; neither the lock nor a connection is held across bcrypt.+ await _refuse_signup(session, user_store, req.email, config)+ await session.rollback()+ password_hash = await hash_password_off_loop(req.password)+await session.execute(text("SELECT pg_advisory_xact_lock(hashtextextended(:key, 0))"),{"key": "gateway-signup"},)- created, retry_after = await user_store.created_in_last_hour(session)- if created >= config.gateway_signup_max_per_hour:- logger.warning(- "Refused sign-up: %d users created in the last hour (cap %d)",- created,- config.gateway_signup_max_per_hour,- )- await session.rollback()- raise HTTPException(- status_code=429,- detail="Too many sign-ups on this server in the last hour. Try again later.",- headers={"Retry-After": str(retry_after)},- )- if await user_store.get_by_email(session, req.email) is not None:- await session.rollback()- raise HTTPException(status_code=409, detail="Email already registered")+ await _refuse_signup(session, user_store, req.email, config)user = User(name=req.display_name or req.email.split("@")[0],email=req.email,role="user",- password_hash=hash_password(req.password),+ password_hash=password_hash,)try:async with session.begin_nested():@@ -355,10 +371,10 @@ async def change_password(session: Annotated[AsyncSession, Depends(get_session)],user: Annotated[User, Depends(get_current_user)],) -> dict[str, bool]:- if not verify_password(req.current_password, user.password_hash):+ if not await verify_password_off_loop(req.current_password, user.password_hash):raise HTTPException(status_code=403, detail="Current password is incorrect")- user.password_hash = hash_password(req.new_password)+ user.password_hash = await hash_password_off_loop(req.new_password)await session.commit()return {"ok": True}@@ -392,12 +408,15 @@ async def create_user(existing = await user_store.get_by_email(session, req.email)if existing is not None:raise HTTPException(status_code=409, detail="Email already registered")+ # Not held across bcrypt.+ await session.commit()+ password_hash = await hash_password_off_loop(req.password)user = User(name=req.name,email=req.email,role=req.role,- password_hash=hash_password(req.password),+ password_hash=password_hash,)await user_store.create(session, user)await session.commit()diff --git a/core/switch_core/gateway/collaborations.py b/core/switch_core/gateway/collaborations.pyindex f4cf2cc5d..842f0406e 100644--- a/core/switch_core/gateway/collaborations.py+++ b/core/switch_core/gateway/collaborations.py@@ -324,18 +324,6 @@ async def update_bridge(),)- if payload.agent_greetings_enabled is not None:- bridge = await bridge_store.set_agent_greetings_enabled(- session, bridge_id, payload.agent_greetings_enabled- )- if payload.channel_creation_enabled is not None:- bridge = await bridge_store.set_channel_creation_enabled(- session, bridge_id, payload.channel_creation_enabled- )- if payload.preconfigured is not None:- bridge = await bridge_store.set_preconfigured(- session, bridge_id, payload.preconfigured- )if payload.connection_config is not None:current = bridge.connection_config or {}merged = {**current, **payload.connection_config}@@ -347,6 +335,9 @@ async def update_bridge(status_code=422,detail="event_delivery cannot be changed on an existing connection.",)+ # Not held across the platform's credential check; nothing is written+ # until it passes, so a refused edit still changes nothing.+ await session.commit()try:await collab_lifecycle.check_edited_connection_config(bridge_id=bridge_id, bridge_type=bridge.type, connection_config=merged@@ -366,6 +357,20 @@ async def update_bridge(raise HTTPException(status_code=422, detail=str(exc)) from excexcept (BridgeClaimConflict, BridgeCredentialError) as exc:raise HTTPException(status_code=400, detail=str(exc)) from exc++ if payload.agent_greetings_enabled is not None:+ bridge = await bridge_store.set_agent_greetings_enabled(+ session, bridge_id, payload.agent_greetings_enabled+ )+ if payload.channel_creation_enabled is not None:+ bridge = await bridge_store.set_channel_creation_enabled(+ session, bridge_id, payload.channel_creation_enabled+ )+ if payload.preconfigured is not None:+ bridge = await bridge_store.set_preconfigured(+ session, bridge_id, payload.preconfigured+ )+ if payload.connection_config is not None:bridge = await bridge_store.merge_connection_config(session, bridge_id, dict(payload.connection_config))@@ -497,6 +502,8 @@ async def search_bridge_directory(u.external_user_id: ufor u in await external_user_store.get_by_bridge(session, bridge_id)}+ # Not held across the platform search below.+ await session.commit()source: Literal["directory", "known"] = "directory"note: str | None = None@@ -682,6 +689,8 @@ async def claim_bridge_identity(# the account exists. Provisioning mints a human actor, so taking the# request's word for it would let any signed-in user conjure accounts# for people who do not exist.+ # Not held across the directory search or provisioning's own session.+ await session.commit()await _require_directory_account(collab_lifecycle,bridge_id=bridge_id,@@ -844,6 +853,8 @@ async def delete_bridge("display_name": bridge.display_name,"rooms_deleted": len(rooms),}+ # Not held across the deletes, which open their own sessions and call Matrix.+ await session.commit()for room in rooms:await room_service.delete_room(room.id)diff --git a/core/switch_core/gateway/connectors.py b/core/switch_core/gateway/connectors.pyindex 8dd425657..a32b6b587 100644--- a/core/switch_core/gateway/connectors.py+++ b/core/switch_core/gateway/connectors.py@@ -143,7 +143,8 @@ async def delete_connector(raise HTTPException(status_code=403, detail="Not authorized to delete this connector")-+ # Not held across removal, which deletes the agents on sessions of its own.+ await session.commit()try:await connector_lifecycle.remove(connector_id)except ValueError as exc:diff --git a/core/switch_core/gateway/provider_verifications.py b/core/switch_core/gateway/provider_verifications.pyindex 2dead89c8..518bf467a 100644--- a/core/switch_core/gateway/provider_verifications.py+++ b/core/switch_core/gateway/provider_verifications.py@@ -72,6 +72,8 @@ async def worker_session(raise HTTPException(403, "Connection check owner is no longer a member.")+ # Not held while the route reads the worker's body, at the worker's pace.+ await session.commit()yield sessiondiff --git a/core/switch_core/gateway/rooms.py b/core/switch_core/gateway/rooms.pyindex c8b13f180..773cbf240 100644--- a/core/switch_core/gateway/rooms.py+++ b/core/switch_core/gateway/rooms.py@@ -91,6 +91,28 @@ async def _require_room(return room+async def _require_room_released(+ session: AsyncSession,+ room_store: RoomStore,+ room_id: str,+ user: User,+ action: Action,+ is_admin: bool,+) -> Room:+ """`_require_room`, then end the read's transaction.++ For a route that goes on to call `RoomService` or `RoomYamlService`, which+ open sessions of their own and may call Matrix or a messaging platform.+ Holding this read's connection while waiting for another from the same+ pool is how the pool runs dry under load: every connection held by a+ request that is waiting for one more. The room stays readable after the+ commit (`expire_on_commit=False`).+ """+ room = await _require_room(session, room_store, room_id, user, action, is_admin)+ await session.commit()+ return room++async def _external_channel_url(room: Room) -> str | None:"""Native deeplink to open the room's external channel in the messagingapp, built by the live collaboration adapter. None when not bridged, the@@ -149,6 +171,9 @@ async def _build_room_detail(roles = await _list_room_role_details(session, room.id, protocol)join_event_listeners = await room_store.get_join_event_listeners(session, room.id)+ # Not held across the deeplink, which can ask the platform on a cache miss.+ await session.commit()+ external_channel_url = await _external_channel_url(room)return RoomDetail(id=room.id,@@ -162,7 +187,7 @@ async def _build_room_detail(bridge_id=room.bridge_id,bridge_display_name=bridge_display_name,bridge_type=bridge_type,- external_channel_url=await _external_channel_url(room),+ external_channel_url=external_channel_url,group_id=room.group_id,group_name=group_name,read_visibility=room.read_visibility,@@ -478,7 +503,7 @@ async def export_room_yaml() -> Response:"""Export a room to YAML in the same surface ``/rooms/from-yaml`` accepts.Each section can be dropped via its boolean toggle (default included)."""- await _require_room(session, room_store, room_id, user, "read", is_admin)+ await _require_room_released(session, room_store, room_id, user, "read", is_admin)yaml_text = await rooms_yaml.export(room_id,agents=agents,@@ -520,7 +545,9 @@ async def patch_room(user: Annotated[User, Depends(get_current_user)],is_admin: Annotated[bool, Depends(get_tenant_is_admin)],) -> RoomDetail:- target = await _require_room(session, room_store, room_id, user, "write", is_admin)+ target = await _require_room_released(+ session, room_store, room_id, user, "write", is_admin+ )# Changing a room's access permission (read/write visibility) is reserved# for the room's owner and tenant admins — plain write access (which a# publicly-writable room grants to everyone) is not enough. Otherwise any@@ -677,7 +704,7 @@ async def put_protection(user: Annotated[User, Depends(get_current_user)],is_admin: Annotated[bool, Depends(get_tenant_is_admin)],) -> RoomDetail:- await _require_room(session, room_store, room_id, user, "write", is_admin)+ await _require_room_released(session, room_store, room_id, user, "write", is_admin)try:await room_service.update_protection_config(room_id, req.protection_config)except ValueError:@@ -704,7 +731,7 @@ async def put_observe(user: Annotated[User, Depends(get_current_user)],is_admin: Annotated[bool, Depends(get_tenant_is_admin)],) -> RoomDetail:- await _require_room(session, room_store, room_id, user, "write", is_admin)+ await _require_room_released(session, room_store, room_id, user, "write", is_admin)try:await room_service.update_observe_config(room_id, req.observe_config)except ValueError:@@ -731,7 +758,7 @@ async def post_room_agents(user: Annotated[User, Depends(get_current_user)],is_admin: Annotated[bool, Depends(get_tenant_is_admin)],) -> RoomDetail:- await _require_room(session, room_store, room_id, user, "write", is_admin)+ await _require_room_released(session, room_store, room_id, user, "write", is_admin)try:await room_service.add_agents_to_room(room_id,@@ -764,7 +791,7 @@ async def patch_room_agent(user: Annotated[User, Depends(get_current_user)],is_admin: Annotated[bool, Depends(get_tenant_is_admin)],) -> RoomDetail:- await _require_room(session, room_store, room_id, user, "write", is_admin)+ await _require_room_released(session, room_store, room_id, user, "write", is_admin)try:await room_service.set_join_event_listeners(room_id, {agent_id: req.receives_join_events}@@ -792,7 +819,7 @@ async def delete_room_agent(user: Annotated[User, Depends(get_current_user)],is_admin: Annotated[bool, Depends(get_tenant_is_admin)],) -> RoomDetail:- await _require_room(session, room_store, room_id, user, "write", is_admin)+ await _require_room_released(session, room_store, room_id, user, "write", is_admin)try:await room_service.remove_agents_from_room(room_id, [agent_id])except ValueError as e:@@ -819,7 +846,7 @@ async def post_room_users(user: Annotated[User, Depends(get_current_user)],is_admin: Annotated[bool, Depends(get_tenant_is_admin)],) -> RoomDetail:- await _require_room(session, room_store, room_id, user, "write", is_admin)+ await _require_room_released(session, room_store, room_id, user, "write", is_admin)try:await room_service.add_users_to_room(room_id, req.user_names)except ValueError as e:@@ -845,7 +872,7 @@ async def _set_archived(user: User,is_admin: bool,) -> RoomDetail:- await _require_room(session, room_store, room_id, user, "write", is_admin)+ await _require_room_released(session, room_store, room_id, user, "write", is_admin)try:await room_service.set_room_archived(room_id, archived)except ValueError:@@ -922,7 +949,7 @@ async def delete_room(user: Annotated[User, Depends(get_current_user)],is_admin: Annotated[bool, Depends(get_tenant_is_admin)],) -> dict[str, bool]:- await _require_room(session, room_store, room_id, user, "delete", is_admin)+ await _require_room_released(session, room_store, room_id, user, "delete", is_admin)try:await room_service.delete_room(room_id)except ValueError:@@ -951,6 +978,8 @@ async def bulk_delete_rooms("Skipping room %s during bulk delete: not authorized", room_id)continue+ # See `_require_room_released`.+ await session.commit()await room_service.delete_room(room_id)deleted += 1return BulkDeleteResponse(deleted=deleted)@@ -980,6 +1009,8 @@ async def bulk_archive_rooms("Skipping room %s during bulk archive: not authorized", room_id)continue+ # See `_require_room_released`.+ await session.commit()await room_service.set_room_archived(room_id, req.archived)updated += 1return BulkArchiveResponse(updated=updated)diff --git a/core/switch_core/gateway/template_runs.py b/core/switch_core/gateway/template_runs.pyindex c46f4e19a..6f14d05c8 100644--- a/core/switch_core/gateway/template_runs.py+++ b/core/switch_core/gateway/template_runs.py@@ -199,6 +199,8 @@ async def _control("template, or an admin can do that."),)+ # Not held across set_state, which opens its own session and posts in every room.+ await session.commit()try:await service.set_state(root_id,diff --git a/core/switch_core/gateway/templates.py b/core/switch_core/gateway/templates.pyindex 5af5e995e..6baa5ca93 100644--- a/core/switch_core/gateway/templates.py+++ b/core/switch_core/gateway/templates.py@@ -417,6 +417,8 @@ async def patch_template(require_manage(principal, locked.owner_id)if req.content is not None:+ # Not held across linting (up to 1 MiB of YAML); the update re-checks under lock.+ await session.commit()_require_within_size_limit(req.content, config)_require_storable(req.content)try:diff --git a/core/switch_core/gateway/tenants.py b/core/switch_core/gateway/tenants.pyindex cca9d8cf4..c1491eb9a 100644--- a/core/switch_core/gateway/tenants.py+++ b/core/switch_core/gateway/tenants.py@@ -460,6 +460,8 @@ async def create_tenant(user = await user_store.get(session, caller.id)if user is None:raise HTTPException(status_code=401, detail="User not found")+ # Not held across provisioning, which opens a session per tenant.+ await session.commit()tenant = await _provision_workspace(session_factory, user_store, client_lifecycle, caller, req.name)diff --git a/core/switch_core/main.py b/core/switch_core/main.pyindex bc0595b8f..8c59e09c0 100644--- a/core/switch_core/main.py+++ b/core/switch_core/main.py@@ -151,7 +151,12 @@ from switch_core.observability.bootstrap import (RuntimeProbes,start_observability,)-from switch_core.observability.pool import install_pool_watermark, pool_stats+from switch_core.observability.pool import (+ WaitTimedQueuePool,+ install_hold_timer,+ install_pool_watermark,+ pool_stats,+)from switch_core.observability.query import instrument_queriesfrom switch_core.observability.runtime import EventLoopLagfrom switch_core.provisioning import Provisioning@@ -169,6 +174,7 @@ from switch_core.telemetry.setup import build_telemetryfrom switch_core.tenant_context import no_tenantfrom switch_core.transport.ephemeral import EphemeralBusfrom switch_core.transport.invites import InviteBus+from switch_core.transport.room_cache import RoomCacheLimits, RoomDeliveryCachefrom switch_core.version import switch_core_versionlogger = logging.getLogger(__name__)@@ -396,7 +402,7 @@ async def run(config: SwitchConfig) -> None:# the schema owner's connection where one is configured, and neither# belongs on the pooled application engine built below.encrypted_json.configure(config.keyring)- engine = create_engine_from_config(config)+ engine = create_engine_from_config(config, poolclass=WaitTimedQueuePool)tenants_isolated = await _check_tenant_isolation(config, engine)session_factory = create_session_factory(engine)# Wired here rather than inside the engine factory, so the database layer@@ -408,6 +414,7 @@ async def run(config: SwitchConfig) -> None:# not change how queries execute.instrument_queries(engine)pool_watermark = install_pool_watermark(engine)+ install_hold_timer(engine)# Its connection is held rather than borrowed, so it builds its own outside# the pool. Nothing subscribes yet; it starts with the server so that the@@ -531,6 +538,18 @@ async def run(config: SwitchConfig) -> None:# fresh heartbeat row (CHOO-1857 stage B).connections = AgentConnectionRegistry()+ # One shared read per room for every client in this process.+ room_cache = RoomDeliveryCache(+ session_factory=session_factory,+ message_store=message_store,+ limits=RoomCacheLimits(+ max_bytes=config.room_delivery_cache_max_bytes,+ max_rooms=config.room_delivery_cache_max_rooms,+ max_rows_per_room=config.room_delivery_cache_max_rows_per_room,+ max_age_seconds=config.room_delivery_cache_max_age_seconds,+ ),+ )+# ── Client factory ───────────────────────────────────────────────────────client_factory = ClientFactory(client_store=client_store,@@ -543,6 +562,7 @@ async def run(config: SwitchConfig) -> None:listener=message_listener,invites=invites,ephemeral=ephemeral,+ room_cache=room_cache,)client_factory.register("agent",@@ -608,6 +628,7 @@ async def run(config: SwitchConfig) -> None:resource_service=resource_service,session_factory=session_factory,telemetry=telemetry,+ room_cache=room_cache,)collab_lifecycle._room_service = room_service@@ -817,8 +838,9 @@ async def run(config: SwitchConfig) -> None:consumers_running=client_lifecycle.running_count,connectors_running=connector_lifecycle.running_count,connectors_configured=connector_lifecycle.expected_count,- agents_connected=lambda: len(connections.live_agent_ids()),+ agents_connected=connections.live_agents_by_transport,pool_stats=lambda: pool_stats(engine, pool_watermark),+ room_cache_stats=room_cache.stats,)# ── Lifespan: start server-side connectors once HTTP is serving ────────@@ -841,6 +863,7 @@ async def run(config: SwitchConfig) -> None:version=switch_core_version(),session_factory=session_factory,probes=probes,+ db_server_engine=lambda: create_unpooled_engine(config),)asyncio.create_task(connector_lifecycle.start_all())sweep_task = asyncio.create_task(_runtime_state_sweep_loop(protocol))diff --git a/core/switch_core/management/controller_routes.py b/core/switch_core/management/controller_routes.pyindex e28424ca2..66dfb2094 100644--- a/core/switch_core/management/controller_routes.py+++ b/core/switch_core/management/controller_routes.py@@ -6,26 +6,36 @@ names a controller then requires it to be the token's own. Enrollment andtoken exchange carry their secret in the body and resolve their tenant fromit here.-The controller's stream (`/v1/controllers/{id}/...`) is Core's: these routes-authenticate and parse, and Core's `ControllerPresence` and controller stream-do the rest. Management only adds its nudges and the first frame's+The controller's connection (`/v1/controllers/{id}/connection` and its+socket) is Core's: these routes authenticate and parse, and Core's+`ControllerPresence` and controller stream do the rest. Management only adds its nudges and the first frame'sassignment revision.Every failure is answered in the contract's error envelope (`errors.py`).`Switch-Controller-Protocol` is checked when sent, and every response says-which protocol versions this server accepts.+which protocol versions this server accepts. Protocol 2 moved the stream and+the beat onto one WebSocket; a protocol 1 controller is refused with 426."""from __future__ import annotations+import asyncioimport hashlibimport jsonimport logging-from collections.abc import Awaitable, Callable, Coroutine+from collections.abc import AsyncGenerator, Awaitable, Callable, Coroutinefrom typing import Annotated, Any-from fastapi import APIRouter, Depends, Header, Request, Response-from fastapi.responses import JSONResponse, StreamingResponse+from fastapi import (+ APIRouter,+ Depends,+ Header,+ Request,+ Response,+ WebSocket,+ WebSocketDisconnect,+)+from fastapi.responses import JSONResponsefrom sqlalchemy.ext.asyncio import AsyncSession, async_sessionmakerfrom switch_core.bridges.agent.auth import ControllerPrincipal@@ -35,8 +45,9 @@ from switch_core.bridges.agent.protocol.controller_presence import (ControllerConnectionError,)from switch_core.bridges.agent.protocol.controller_stream import (- KEEPALIVE_INTERVAL_SECONDS,- STREAM_HEADERS,+ IDLE,+ IDLE_INTERVAL_SECONDS,+ ControllerFrame,controller_event_stream,)from switch_core.bridges.agent.protocol.liveness import HEARTBEAT_INTERVAL_SECONDS@@ -53,7 +64,6 @@ from switch_core.management.dependencies import (from switch_core.management.errors import ManagementError, ManagementRoute, error_bodyfrom switch_core.management.schemas import (MAX_STATUS_BYTES,- ControllerBeatRequest,ControllerConnectionRequest,DefinitionV1,EnrollRequest,@@ -69,8 +79,8 @@ logger = logging.getLogger(__name__)PROTOCOL_HEADER = "Switch-Controller-Protocol"PROTOCOL_ACCEPTS_HEADER = "Switch-Controller-Protocol-Accepts"-SUPPORTED_PROTOCOLS = "1-1"-_SUPPORTED_PROTOCOL_VERSIONS = frozenset({"1"})+SUPPORTED_PROTOCOLS = "2-2"+_SUPPORTED_PROTOCOL_VERSIONS = frozenset({"2"})class ControllerRoute(ManagementRoute):@@ -315,7 +325,7 @@ async def open_connection(The agents bound to it are attached on the stream, each from its cursorhere; the response names them. Opening again is a takeover: the earlier- connection's stream ends with `taken_over`, and its beats are refused.+ connection's socket ends with `taken_over`, and its beats are refused."""presence = protocol.connections.controllerstry:@@ -341,21 +351,21 @@ async def open_connection(}[email protected]("/v1/controllers/{controller_id}/events")-async def controller_events(- principal: PathController,- management: Management,- protocol: Protocol,- session_factory: Annotated[- async_sessionmaker[AsyncSession], Depends(get_management_session_factory)- ],+async def attach_stream(+ *,+ principal: ControllerPrincipal,+ management: ManagementService,+ protocol: AgentCore,+ session_factory: async_sessionmaker[AsyncSession],connection_id: str,generation: int,-) -> StreamingResponse:- """The controller's one stream: its agents' events and the management nudges.+) -> AsyncGenerator[ControllerFrame]:+ """Attach the stream to an open connection: its agents' events and the+ management nudges, starting with `connection_state`.The nudge subscription is taken before the first frame is built, so achange that commits while the stream is opening is still delivered.+ Raises the refusal when the connection cannot take a stream."""presence = protocol.connections.controllersnudges = management.notifier.subscribe(principal.controller_id)@@ -376,48 +386,40 @@ async def controller_events("heartbeat_interval_s": HEARTBEAT_INTERVAL_SECONDS,}token = presence.attach_stream(conn)- return StreamingResponse(- controller_event_stream(- conn=conn,- stream_token=token,- presence=presence,- buffer=protocol.event_buffer,- approvals=protocol.approval_outcomes,- nudges=nudges,- opening=opening,- rooms_of=_rooms_reader(protocol),- keepalive_seconds=KEEPALIVE_INTERVAL_SECONDS,- ),- media_type="text/event-stream",- headers=STREAM_HEADERS,+ return controller_event_stream(+ conn=conn,+ stream_token=token,+ presence=presence,+ buffer=protocol.event_buffer,+ approvals=protocol.approval_outcomes,+ nudges=nudges,+ opening=opening,+ rooms_of=_rooms_reader(protocol),+ idle_seconds=IDLE_INTERVAL_SECONDS,)[email protected]("/v1/controllers/{controller_id}/connection/beat")-async def connection_beat(- body: ControllerBeatRequest,- principal: PathController,- protocol: Protocol,-) -> dict[str, Any]:+def record_beat(+ *,+ principal: ControllerPrincipal,+ protocol: AgentCore,+ connection_id: str,+ generation: int,+ cursors: dict[str, int],+) -> list[str]:"""Keep the controller, and so every agent on it, live; confirm cursors.- The cursors confirmed here are where a stream reattached to this- connection resumes each agent.-- Every two seconds; six without one and its agents are not live. Refused- with `taken_over` once another connection has replaced this one, which is- terminal for the client, and with `no_stream` or `unknown_connection`- when the stream has to be opened again.+ A pong on the socket, every heartbeat interval; six seconds without one+ and its agents are not live. The cursors confirmed here are where a+ stream reattached to this connection resumes each agent. Raises the+ presence's refusal: `taken_over` once another connection has replaced+ this one, terminal for the client, and `no_stream` or+ `unknown_connection` when the connection has to be opened again."""presence = protocol.connections.controllers- try:- conn = presence.beat(- principal.controller_id, body.connection_id, body.generation- )- except ControllerConnectionError as exc:- raise _connection_refusal(exc) from exc+ conn = presence.beat(principal.controller_id, connection_id, generation)agents = presence.agents_of(principal.controller_id)- for agent_id, cursor in body.cursors.items():+ for agent_id, cursor in cursors.items():binding = presence.binding(agent_id)if agent_id not in agents or binding is None:continue@@ -426,4 +428,180 @@ async def connection_beat(confirmed = min(cursor, protocol.event_buffer.head(agent_id))protocol.event_buffer.confirm(agent_id, presence.holder_id(binding), confirmed)presence.resume_from(conn, agent_id, confirmed)- return {"agents": sorted(agents)}+ return sorted(agents)+++def _refused(error: ManagementError) -> dict[str, Any]:+ """The frame a refused socket gets before it closes with 4000 + status,+ as the bearer middleware sends its own refusals."""+ return {+ "event": "refused",+ "data": {+ "status": error.status_code,+ "detail": {"code": error.code, "message": error.message},+ },+ }+++def _cursors_of(message: dict[str, Any]) -> dict[str, int]:+ raw = message.get("cursors")+ if not isinstance(raw, dict):+ return {}+ return {+ agent_id: cursor+ for agent_id, cursor in raw.items()+ if isinstance(agent_id, str)+ and isinstance(cursor, int)+ and not isinstance(cursor, bool)+ and cursor >= 0+ }++[email protected]("/v1/controllers/{controller_id}/connection/ws")+async def connection_socket(+ websocket: WebSocket,+ controller_id: str,+ management: Management,+ protocol: Protocol,+ session_factory: Annotated[+ async_sessionmaker[AsyncSession], Depends(get_management_session_factory)+ ],+ connection_id: str,+ generation: int,+) -> None:+ """The controller's one connection: its agents' events and the management+ nudges down, its heartbeat up, on one WebSocket.++ Attaches to a connection opened with `POST .../connection`. Frames are+ JSON `{"event", "data"}`, the events the stream carries. The server sends+ `{"event": "ping"}` every heartbeat interval and the controller answers+ `{"type": "pong", "cursors": {agent_id: n}}`, which is its beat. A refused+ attach is a `refused` frame, then a close with 4000 plus the status; a+ refused beat is an `evicted` frame naming why, then a close.+ """+ await websocket.accept()+ try:+ principal = _socket_principal(websocket, controller_id)+ frames = await attach_stream(+ principal=principal,+ management=management,+ protocol=protocol,+ session_factory=session_factory,+ connection_id=connection_id,+ generation=generation,+ )+ except ManagementError as error:+ await websocket.send_json(_refused(error))+ await websocket.close(code=4000 + error.status_code)+ return++ outbox: asyncio.Queue[ControllerFrame] = asyncio.Queue(maxsize=_SOCKET_OUTBOX)+ pump = asyncio.create_task(_pump(frames, outbox))+ pongs = asyncio.create_task(+ _receive_pongs(websocket, principal, protocol, connection_id, generation)+ )+ loop = asyncio.get_running_loop()+ next_ping = loop.time() + HEARTBEAT_INTERVAL_SECONDS+ try:+ # One writer: frames and pings both go out from this loop.+ while not pump.done() or not outbox.empty():+ if pongs.done():+ refusal = pongs.result()+ if refusal is not None:+ await websocket.send_json(+ {+ "event": "evicted",+ "data": {"code": refusal.code, "reason": str(refusal)},+ }+ )+ return+ try:+ frame = await asyncio.wait_for(+ outbox.get(), timeout=max(0.0, next_ping - loop.time())+ )+ except TimeoutError:+ frame = None+ if frame is not None and frame is not IDLE:+ await websocket.send_json(frame)+ if loop.time() >= next_ping:+ await websocket.send_json({"event": "ping", "data": {}})+ next_ping = loop.time() + HEARTBEAT_INTERVAL_SECONDS+ except WebSocketDisconnect:+ return+ except RuntimeError:+ # The server closed the socket under us (uvicorn does on shutdown).+ return+ finally:+ pump.cancel()+ pongs.cancel()+ failure = pump.exception() if pump.done() and not pump.cancelled() else None+ if failure is not None:+ logger.error(+ "Controller %s connection %s: delivery failed, closing its socket",+ principal.controller_id,+ connection_id,+ exc_info=failure,+ )+ try:+ await websocket.close(code=1011 if failure is not None else 1000)+ except RuntimeError:+ pass # the client closed it first+++# Frames buffered between the stream and the socket.+_SOCKET_OUTBOX = 64+++def _socket_principal(websocket: WebSocket, controller_id: str) -> ControllerPrincipal:+ """The controller the bearer middleware authenticated, checked against the+ path and the protocol it declares, as the HTTP routes check theirs."""+ requested = websocket.headers.get(PROTOCOL_HEADER)+ if requested is not None and requested.strip() not in _SUPPORTED_PROTOCOL_VERSIONS:+ raise ManagementError(+ 426,+ reason_codes.PROTOCOL_UNSUPPORTED,+ f"Controller protocol {requested!r} is not supported; "+ f"this server accepts {SUPPORTED_PROTOCOLS}.",+ )+ return require_controller(controller_id, get_controller_principal(websocket))+++async def _pump(+ frames: AsyncGenerator[ControllerFrame], outbox: asyncio.Queue[ControllerFrame]+) -> None:+ # Closed here: cancelled while waiting on the outbox, the stream would+ # otherwise detach only when the generator is garbage collected.+ try:+ async for frame in frames:+ await outbox.put(frame)+ finally:+ await frames.aclose()+++async def _receive_pongs(+ websocket: WebSocket,+ principal: ControllerPrincipal,+ protocol: AgentCore,+ connection_id: str,+ generation: int,+) -> ControllerConnectionError | None:+ """Turn each pong into a beat, until the controller goes or a beat is+ refused. Returns the refusal, so the socket can say which before it+ closes; None when the controller simply went away."""+ while True:+ try:+ message = await websocket.receive_json()+ except (WebSocketDisconnect, ValueError, RuntimeError):+ return None+ if not isinstance(message, dict) or message.get("type") != "pong":+ continue+ try:+ record_beat(+ principal=principal,+ protocol=protocol,+ connection_id=connection_id,+ generation=generation,+ cursors=_cursors_of(message),+ )+ except ControllerConnectionError as refusal:+ return refusaldiff --git a/core/switch_core/management/dependencies.py b/core/switch_core/management/dependencies.pyindex d9f14b50c..593f83967 100644--- a/core/switch_core/management/dependencies.py+++ b/core/switch_core/management/dependencies.py@@ -10,8 +10,8 @@ from __future__ import annotationsfrom typing import Any-from fastapi import Requestfrom sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker+from starlette.requests import HTTPConnectionfrom switch_core.bridges.agent.auth import ControllerPrincipalfrom switch_core.management import reason_codes@@ -45,7 +45,7 @@ def get_management_session_factory() -> async_sessionmaker[AsyncSession]:return _state["session_factory"] # type: ignore[no-any-return]-def get_controller_principal(request: Request) -> ControllerPrincipal:+def get_controller_principal(request: HTTPConnection) -> ControllerPrincipal:"""The controller the bearer middleware authenticated."""principal = request.scope.get("controller")if not isinstance(principal, ControllerPrincipal):diff --git a/core/switch_core/management/schemas.py b/core/switch_core/management/schemas.pyindex 2373d0c19..f815994a8 100644--- a/core/switch_core/management/schemas.py+++ b/core/switch_core/management/schemas.py@@ -320,14 +320,6 @@ class ControllerConnectionRequest(_ControllerStreamBody):}-class ControllerBeatRequest(_ControllerStreamBody):- """A beat: the connection is still there, and these cursors are confirmed."""-- connection_id: str- generation: int- cursors: dict[str, int]--# ── Gateway requests ──────────────────────────────────────────────────────────diff --git a/core/switch_core/observability/bootstrap.py b/core/switch_core/observability/bootstrap.pyindex dfd4966a2..31439dcd9 100644--- a/core/switch_core/observability/bootstrap.py+++ b/core/switch_core/observability/bootstrap.py@@ -14,7 +14,7 @@ from collections.abc import Callable, Iterator, Mappingfrom dataclasses import dataclassimport httpx-from sqlalchemy.ext.asyncio import async_sessionmaker+from sqlalchemy.ext.asyncio import AsyncEngine, async_sessionmakerfrom switch_core.config import SwitchConfigfrom switch_core.logging_context import LogContextFilter@@ -26,7 +26,11 @@ from switch_core.observability.catalogue import (DB_POOL_IN_USE,DB_POOL_OVERFLOW,DB_POOL_SIZE,+ DELIVERY_CACHE_BYTES,+ DELIVERY_CACHE_ROOMS,+ RUNTIME_STARTS,)+from switch_core.observability.db_server import DbServerSamplerfrom switch_core.observability.exporter import MetricsExporterfrom switch_core.observability.health import (HealthMonitor,@@ -55,6 +59,7 @@ from switch_core.observability.runtime import (RuntimeMetrics,log_unreadable_sources,)+from switch_core.transport.room_cache import RoomCacheStatslogger = logging.getLogger(__name__)@@ -83,10 +88,13 @@ class RuntimeProbes:consumers_running: Callable[[], int]connectors_running: Callable[[], int]connectors_configured: Callable[[], int]- agents_connected: Callable[[], int]+ # Agents per (transport, client); see+ # `AgentConnectionRegistry.live_agents_by_transport`.+ agents_connected: Callable[[], Mapping[tuple[str, str], int]]# None when the engine's pool does not keep these — see# :mod:`switch_core.observability.pool`.pool_stats: Callable[[], PoolStats | None]+ room_cache_stats: Callable[[], RoomCacheStats]@dataclass@@ -119,8 +127,19 @@ class Observability:def _state_readings(probes: RuntimeProbes) -> Callable[[], Iterator[GaugeReading]]:+ # Every (transport, client) pair reported so far: one that empties out is+ # reported as 0, or the dashboard would keep showing its last count.+ seen: set[tuple[str, str]] = {("websocket", "unknown")}+def readings() -> Iterator[GaugeReading]:- yield GaugeReading(AGENTS_CONNECTED, float(probes.agents_connected()), {})+ connected = probes.agents_connected()+ seen.update(connected)+ for transport, client in sorted(seen):+ yield GaugeReading(+ AGENTS_CONNECTED,+ float(connected.get((transport, client), 0)),+ {"transport": transport, "client": client},+ )yield GaugeReading(CONSUMERS_RUNNING, float(probes.consumers_running()), {})for platform, running in probes.bridges_running_by_platform().items():yield GaugeReading(@@ -136,6 +155,10 @@ def _state_readings(probes: RuntimeProbes) -> Callable[[], Iterator[GaugeReadingyield GaugeReading(DB_POOL_SIZE, float(stats.size), {})yield GaugeReading(DB_POOL_OVERFLOW, float(stats.overflow), {})+ cache = probes.room_cache_stats()+ yield GaugeReading(DELIVERY_CACHE_BYTES, float(cache.bytes), {})+ yield GaugeReading(DELIVERY_CACHE_ROOMS, float(cache.rooms), {})+return readings@@ -144,10 +167,13 @@ def start_observability(version: str | None,session_factory: async_sessionmaker,probes: RuntimeProbes,+ db_server_engine: Callable[[], AsyncEngine] | None = None,) -> Observability:"""Install the registry, start the loops, and hand back the handle.- Called once, from the server's lifespan.+ Called once, from the server's lifespan. `db_server_engine` builds the+ engine the database sampler takes its one connection from; it must not be+ the application's pooled engine."""monitor = HealthMonitor(checks=[@@ -179,6 +205,7 @@ def start_observability(registry = MetricsRegistry()install(registry)+ registry.increment(RUNTIME_STARTS, {})monitor.install(registry)RuntimeMetrics(lag).install(registry)@@ -209,6 +236,12 @@ def start_observability(tasks.append(asyncio.create_task(exporter.run_forever(), name="metrics-exporter"))+ if db_server_engine is not None:+ sampler = DbServerSampler(db_server_engine)+ registry.register_observer(sampler.readings)+ tasks.append(+ asyncio.create_task(sampler.run_forever(), name="db-server-sampler")+ )logger.info("Reporting metrics to %s every %.0fs as service %r.",client.url_for("metrics"),diff --git a/core/switch_core/observability/catalogue.py b/core/switch_core/observability/catalogue.pyindex 16502642b..48941ec03 100644--- a/core/switch_core/observability/catalogue.py+++ b/core/switch_core/observability/catalogue.py@@ -124,6 +124,33 @@ DB_POOL_TIMEOUTS = _spec("at its ceiling is only suggestive: this is a request that got no connection ""at all within the timeout.",)+# How long a connection stayed borrowed, by who borrowed it. The peak above+# says the pool filled; this says which code was holding it. `caller` is the+# first switch_core function on the stack when the connection was handed out,+# as `module:function`, so the series count is bounded by the code, not the+# traffic. A hold far longer than the queries inside it is code awaiting+# something else with a connection in hand.+DB_POOL_HOLD_DURATION = _spec(+ "switch.db.pool.hold.duration",+ "histogram",+ "ms",+ "How long a pooled connection stayed checked out, by the code that "+ "borrowed it. Compare with query duration: a hold much longer than its "+ "queries is code waiting on something else while holding a connection.",+ "caller",+)+# The timeouts above count requests that waited the full `db_pool_timeout`;+# this is every wait, so a pool that makes requests queue for 200 ms shows up+# long before one makes them give up.+DB_POOL_WAIT_DURATION = _spec(+ "switch.db.pool.wait.duration",+ "histogram",+ "ms",+ "Time from asking the pool for a connection to getting one, including "+ "opening a new one when the pool grows. Requests that timed out are not "+ "here; they are counted in switch.db.pool.timeouts.",+ bounds=SUB_MILLISECOND_BOUNDS_MS,+)# The pool gauges say whether connections are scarce; this says whether the# database is slow, which is the other half and the one a slow request is# usually about. `operation` is the statement's leading keyword mapped through@@ -141,6 +168,30 @@ DB_QUERY_DURATION = _spec(bounds=SUB_MILLISECOND_BOUNDS_MS,)+# What the database itself reports, sampled by this server on a connection of+# its own rather than one from the pool above: the case worth seeing is the pool+# exhausted, when a pooled connection is the one thing a sampler cannot get.+# Only the server role's own sessions in its own database are counted, which is+# also all `pg_stat_activity` shows a role that is not a superuser.+DB_SERVER_CONNECTIONS = _spec(+ "switch.db.server.connections",+ "gauge",+ "{connection}",+ "Sessions the database holds for this server's role, by state, as of the "+ "last sample. `idle_in_transaction` is a connection checked out and held "+ "across work that is not a query, the usual way a pool runs dry. Counts "+ "every replica's sessions, not only this one's.",+ "state",+)+DB_SERVER_TRANSACTIONS = _spec(+ "switch.db.server.transactions",+ "sum",+ "{transaction}",+ "Transactions committed or rolled back in this server's database, from the "+ "database's own counters. Counts every client of the database, every "+ "replica included.",+)+# ── Message transport ────────────────────────────────────────────────────────MESSAGES_SENT = _spec("switch.messages.sent",@@ -191,6 +242,59 @@ DELIVERY_LAG = _spec("actor",)+# ── Shared room reads ────────────────────────────────────────────────────────+# The per-room cache of recent rows (`transport/room_cache.py`). Every value+# below comes from a fixed set in that module.+DELIVERY_CACHE_READS = _spec(+ "switch.delivery_cache.reads",+ "sum",+ "{read}",+ "A transport asking the room cache for the rows after its cursor. "+ "`outcome` is hit (served from memory), filled (served after waiting for "+ "the room's one read), or why it fell back to its own database read: "+ "behind (cursor below what is held), evicted (dropped while it waited, "+ "and the read it waited for could not answer it), "+ "unwatched, or gave_up.",+ "outcome",+)+DELIVERY_CACHE_FILLS = _spec(+ "switch.delivery_cache.fills",+ "sum",+ "{fill}",+ "Database reads made by the room cache on behalf of every member of a "+ "room. `outcome` is ok, failed, or discarded (the room was dropped while "+ "it read).",+ "outcome",+)+DELIVERY_CACHE_ROWS_READ = _spec(+ "switch.delivery_cache.rows_read",+ "sum",+ "{row}",+ "Rows the room cache read from the database. Against "+ "switch.messages.delivered, the number of deliveries each read served.",+)+DELIVERY_CACHE_EVICTIONS = _spec(+ "switch.delivery_cache.evictions",+ "sum",+ "{eviction}",+ "Rows or rooms the room cache let go of, by reason: bytes or rooms (the "+ "process-wide limits), rows or age (one room's), unwatched (its last "+ "member left), invalidated.",+ "reason",+)+DELIVERY_CACHE_BYTES = _spec(+ "switch.delivery_cache.bytes",+ "gauge",+ "By",+ "Estimated memory the room cache holds, against room_delivery_cache_max_bytes.",+)+DELIVERY_CACHE_ROOMS = _spec(+ "switch.delivery_cache.rooms",+ "gauge",+ "{room}",+ "Rooms the room cache holds rows for.",+)+# ── Bridges ──────────────────────────────────────────────────────────────────# Both bridges report the same metrics. `bridge` is "collaboration" or# "agent". For a collaboration bridge `platform` is one of the registered@@ -264,20 +368,82 @@ AGENT_EVENTS_DROPPED = _spec("(retention).","reason",)+# The buffer answers a read and an unread count by walking the agent's retained+# events from the oldest. These say what that walk costs, so whether to index+# the buffer is decided from data. `operation` is "read" or "unread".+AGENT_BUFFER_SCANNED = _spec(+ "switch.agent.buffer.scanned",+ "histogram",+ "{event}",+ "Retained events walked to answer one buffer read or unread count. Bounded "+ "by the per-agent cap, so a p95 near it means every read walks a full buffer.",+ "operation",+ bounds=(0.0, 10.0, 50.0, 100.0, 250.0, 500.0, 1000.0, 1500.0, 2000.0),+)+AGENT_BUFFER_SCAN_DURATION = _spec(+ "switch.agent.buffer.scan.duration",+ "histogram",+ "ms",+ "Wall time of one buffer read or unread count. A read runs on every pass "+ "of a connection's delivery loop, an unread count on every addressed event "+ "it delivers.",+ "operation",+ bounds=(0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 25.0, 100.0),+)+# `reason` is from a fixed set the refusal sites choose. `transport_removed` is+# a client still asking for the retired event stream, the one way to see old+# clients that never manage to connect at all. Nothing sends it while the event+# stream is kept for its compatibility window; it is kept here for the release+# that removes the stream, when it starts firing again.+AGENT_CONNECTIONS_REFUSED = _spec(+ "switch.agent.connections_refused",+ "sum",+ "{connection}",+ "Agent connections refused on opening, by reason: transport_removed (an "+ "old client asking for the event stream), protocol (no agent-protocol "+ "revision in common), unauthorized (no valid credential), other.",+ "reason",+)AGENT_CONNECTIONS_EXPIRED = _spec("switch.agent.connections_expired","sum","{connection}",- "Agent connections closed on a lapsed heartbeat. A steady rate against a "- "flat connection count is churn, which a gauge alone cannot show.",+ "Agent connections closed on a lapsed heartbeat, whichever path noticed: "+ "the sweep, the next use of the dead connection, or its own event stream. "+ "A steady rate against a flat connection count is churn, which a gauge "+ "alone cannot show.",+)++# Every stream open, by what it was. `reattach` is a live connection whose+# socket came back; `after_lapse` and `after_close` are a connection id this+# process closed recently, coming back; `fresh` is one this process has never+# seen, which is a first connect or anything after a server restart. A burst+# of `after_lapse` is the reconnect storm the expiry counter only implies.+AGENT_CONNECTIONS_OPENED = _spec(+ "switch.agent.connections_opened",+ "sum",+ "{connection}",+ "Agent event streams opened, by kind: reattach, after_lapse, after_close or fresh.",+ "kind",)# ── Agents and clients ───────────────────────────────────────────────────────+# `client` is the declared artifact when the registry knows the name, else+# `other` or `unknown`. Never the declared version: the client chooses it.+# `transport` is `websocket`, `sse` (an old client on the event stream, kept+# for a compatibility window that ends when this stays at zero), `detached`+# (the stream dropped, and the connection waits out its heartbeat window), or+# `controller` (run by an agents controller, on that controller's connection).AGENTS_CONNECTED = _spec("switch.agents.connected","gauge","{agent}",- "Agents holding a live protocol connection.",+ "Agents holding a live protocol connection, by transport (websocket, sse, "+ "detached, controller) and client. An agent connected more than one way counts once "+ "under each, so summing across the attributes can exceed the number of "+ "agents.",+ "transport",+ "client",)CONSUMERS_RUNNING = _spec("switch.consumers.running",@@ -326,6 +492,15 @@ RUNTIME_EVENT_LOOP_LAG = _spec("single-threaded, so this is the one number that says whether anything is ""being starved.",)+# One per process start, so a restart lines up against everything else on a+# dashboard. The version is already on every series' resource.+RUNTIME_STARTS = _spec(+ "switch.runtime.starts",+ "sum",+ "{start}",+ "Process starts. A marker: deploys and restarts, to line up against "+ "reconnect bursts and pool peaks.",+)RUNTIME_GC_COLLECTIONS = _spec("switch.runtime.gc_collections","sum",@@ -353,19 +528,33 @@ CATALOGUE: dict[str, MetricSpec] = {DB_POOL_SIZE,DB_POOL_OVERFLOW,DB_POOL_TIMEOUTS,+ DB_POOL_HOLD_DURATION,+ DB_POOL_WAIT_DURATION,DB_QUERY_DURATION,+ DB_SERVER_CONNECTIONS,+ DB_SERVER_TRANSACTIONS,MESSAGES_SENT,MESSAGES_DELIVERED,SEND_FAILURES,DELIVERY_FAILURES,DELIVERY_LAG,+ DELIVERY_CACHE_READS,+ DELIVERY_CACHE_FILLS,+ DELIVERY_CACHE_ROWS_READ,+ DELIVERY_CACHE_EVICTIONS,+ DELIVERY_CACHE_BYTES,+ DELIVERY_CACHE_ROOMS,BRIDGE_EVENTS_IN,BRIDGE_EVENTS_OUT,BRIDGE_ERRORS,BRIDGES_RUNNING,BRIDGE_CALL_DURATION,AGENT_EVENTS_DROPPED,+ AGENT_BUFFER_SCANNED,+ AGENT_BUFFER_SCAN_DURATION,+ AGENT_CONNECTIONS_REFUSED,AGENT_CONNECTIONS_EXPIRED,+ AGENT_CONNECTIONS_OPENED,AGENTS_CONNECTED,CONSUMERS_RUNNING,CONNECTORS_RUNNING,@@ -373,6 +562,7 @@ CATALOGUE: dict[str, MetricSpec] = {RUNTIME_CPU_SECONDS,RUNTIME_OPEN_FDS,RUNTIME_EVENT_LOOP_LAG,+ RUNTIME_STARTS,RUNTIME_GC_COLLECTIONS,HEALTH_CHECK,)diff --git a/core/switch_core/observability/db_server.py b/core/switch_core/observability/db_server.pynew file mode 100644index 000000000..635211ce9--- /dev/null+++ b/core/switch_core/observability/db_server.py@@ -0,0 +1,208 @@+"""The database's own view of this server: its sessions and its transactions.++The pool gauges are this process's side of the connection. What they cannot+show is what the database is holding, and in particular how many sessions sit+`idle in transaction`, a connection checked out and held across work that is+not a query, which is the usual way a pool runs dry. A managed database with no+monitoring integration has nowhere else to read that from.++Sampled on a connection of its own, never one from the application pool: the+moment worth seeing is the pool exhausted, which is exactly when a pooled+connection cannot be had. One connection, held between samples and reopened+when it fails, so the sampler costs the database one idle session.++Only the server role's own sessions in its own database are counted. That is+also all `pg_stat_activity` shows in full to a role that is not a superuser,+so the numbers mean the same under the restricted runtime role as under the+owner.+"""++from __future__ import annotations++import asyncio+import logging+import time+from collections.abc import Callable, Iterator+from dataclasses import dataclass++from sqlalchemy import text+from sqlalchemy.exc import ProgrammingError+from sqlalchemy.ext.asyncio import AsyncConnection, AsyncEngine++from switch_core.observability.catalogue import (+ DB_SERVER_CONNECTIONS,+ DB_SERVER_TRANSACTIONS,+)+from switch_core.observability.metrics import GaugeReading, metrics++logger = logging.getLogger(__name__)++SAMPLE_INTERVAL_SECONDS = 10.0+SAMPLE_TIMEOUT_SECONDS = 2.0+# A database that stays unreachable is reported once a minute, not every sample.+FAILURE_LOG_INTERVAL_SECONDS = 60.0++STATES = ("active", "idle_in_transaction", "idle", "other")++# `pg_stat_activity.state` mapped onto STATES; anything else (fastpath, an+# aborted transaction, a null state) is `other`.+_STATE_OF = {+ "active": "active",+ "idle in transaction": "idle_in_transaction",+ "idle": "idle",+}++_SESSIONS = text(+ "SELECT state, count(*) FROM pg_stat_activity "+ "WHERE usename = current_user AND datname = current_database() "+ "AND pid <> pg_backend_pid() GROUP BY state"+)+_TRANSACTIONS = text(+ "SELECT xact_commit + xact_rollback FROM pg_stat_database "+ "WHERE datname = current_database()"+)+++@dataclass(frozen=True)+class DbServerSample:+ connections: dict[str, int]+ transactions: int+++class DbServerSampler:+ """Samples the database's session and transaction counters on a timer.++ Connection counts are a gauge read from the last sample, and absent once a+ sample has failed: a stale count would draw a healthy database through an+ outage. Transactions are a counter, recorded as the rise between samples.+ """++ def __init__(+ self,+ engine_factory: Callable[[], AsyncEngine],+ *,+ interval_seconds: float = SAMPLE_INTERVAL_SECONDS,+ timeout_seconds: float = SAMPLE_TIMEOUT_SECONDS,+ ) -> None:+ self._engine_factory = engine_factory+ self._interval = interval_seconds+ self._timeout = timeout_seconds+ self._engine: AsyncEngine | None = None+ self._conn: AsyncConnection | None = None+ self._latest: DbServerSample | None = None+ self._last_transactions: int | None = None+ self._last_failure_log = float("-inf")+ self.disabled = False++ def readings(self) -> Iterator[GaugeReading]:+ latest = self._latest+ if latest is None:+ return+ for state in STATES:+ yield GaugeReading(+ DB_SERVER_CONNECTIONS,+ float(latest.connections.get(state, 0)),+ {"state": state},+ )++ async def run_forever(self) -> None:+ try:+ while not self.disabled:+ await self.sample_once()+ await asyncio.sleep(self._interval)+ finally:+ await self.aclose()++ async def sample_once(self) -> DbServerSample | None:+ """Take one sample and record it. None when it could not be taken."""+ if self.disabled or not self._postgres():+ return None+ try:+ sample = await asyncio.wait_for(self._read(), self._timeout)+ except ProgrammingError:+ # The views are missing or unreadable, which no retry will change.+ logger.warning(+ "The database's activity views cannot be read, so "+ "switch.db.server.* will not be reported by this server.",+ exc_info=True,+ )+ self.disabled = True+ self._latest = None+ await self.aclose()+ return None+ except Exception as exc:+ self._latest = None+ await self._drop_connection()+ self._log_failure(exc)+ return None+ self._record(sample)+ return sample++ def _postgres(self) -> bool:+ if self._engine is None:+ self._engine = self._engine_factory()+ if self._engine.dialect.name == "postgresql":+ return True+ logger.info(+ "The database is not Postgres, so switch.db.server.* is not reported."+ )+ self.disabled = True+ return False++ async def _read(self) -> DbServerSample:+ conn = await self._connection()+ sessions = await conn.execute(_SESSIONS)+ connections = dict.fromkeys(STATES, 0)+ for state, count in sessions.all():+ connections[_STATE_OF.get(state, "other")] += int(count)+ transactions = (await conn.execute(_TRANSACTIONS)).scalar_one_or_none()+ return DbServerSample(+ connections=connections, transactions=int(transactions or 0)+ )++ async def _connection(self) -> AsyncConnection:+ if self._conn is None:+ assert self._engine is not None+ conn = await self._engine.connect()+ # Autocommit, so the sampler never sits idle in a transaction itself.+ self._conn = await conn.execution_options(isolation_level="AUTOCOMMIT")+ return self._conn++ def _record(self, sample: DbServerSample) -> None:+ self._latest = sample+ previous = self._last_transactions+ self._last_transactions = sample.transactions+ # The first sample is a baseline, and a fall is a statistics reset.+ if previous is not None and sample.transactions >= previous:+ metrics().increment(+ DB_SERVER_TRANSACTIONS, {}, float(sample.transactions - previous)+ )++ def _log_failure(self, exc: BaseException) -> None:+ now = time.monotonic()+ if now - self._last_failure_log < FAILURE_LOG_INTERVAL_SECONDS:+ return+ self._last_failure_log = now+ logger.warning(+ "Could not sample the database's own connection and transaction "+ "counts (%s: %s); retrying every %.0fs, and saying so at most once a "+ "minute.",+ type(exc).__name__,+ exc,+ self._interval,+ )++ async def _drop_connection(self) -> None:+ conn, self._conn = self._conn, None+ if conn is None:+ return+ try:+ await asyncio.wait_for(conn.close(), self._timeout)+ except Exception:+ logger.debug("Closing the database sampler's connection failed.")++ async def aclose(self) -> None:+ await self._drop_connection()+ engine, self._engine = self._engine, None+ if engine is not None:+ await engine.dispose()diff --git a/core/switch_core/observability/pool.py b/core/switch_core/observability/pool.pyindex 537c075ce..b4177b415 100644--- a/core/switch_core/observability/pool.py+++ b/core/switch_core/observability/pool.py@@ -12,11 +12,27 @@ absent one.from __future__ import annotations+import logging+import os+import sysimport threading+import timefrom dataclasses import dataclass+from types import FrameType+from typing import Any+import greenletfrom sqlalchemy import eventfrom sqlalchemy.ext.asyncio import AsyncEngine+from sqlalchemy.pool import AsyncAdaptedQueuePool, ConnectionPoolEntry++from switch_core.observability.catalogue import (+ DB_POOL_HOLD_DURATION,+ DB_POOL_WAIT_DURATION,+)+from switch_core.observability.metrics import metrics++logger = logging.getLogger(__name__)class PoolInUseWatermark:@@ -99,3 +115,166 @@ def _overflow_beyond_nominal(raw: int) -> int:approaching `size`."""return max(0, raw)+++# ── Who holds a connection, and for how long ─────────────────────────────────++# A hold this long is code that awaited something else with a connection in+# hand: the slowest query on this schema is a few hundred milliseconds.+LONG_HOLD_SECONDS = 1.0+# One warning per caller per window, so a storm of long holds is a line per+# culprit rather than one per request.+_LONG_HOLD_LOG_INTERVAL_SECONDS = 60.0+# Frames shown with a long-hold warning, innermost first.+_LONG_HOLD_STACK_DEPTH = 6++_PACKAGE_MARKER = f"{os.sep}switch_core{os.sep}"+# Frames that are the plumbing every borrow passes through, not the borrower.+_PLUMBING = (+ f"{_PACKAGE_MARKER}observability{os.sep}",+ f"{_PACKAGE_MARKER}db{os.sep}engine.py",+ f"{_PACKAGE_MARKER}db{os.sep}session_scope.py",+ f"{_PACKAGE_MARKER}db{os.sep}tenant_session.py",+)+# A test tree mirrors the package path, so its frames would otherwise pass as ours.+_TESTS_MARKER = f"{os.sep}tests{os.sep}"+_HOLD_KEY = "_switch_hold"+UNKNOWN_CALLER = "unknown"+++@dataclass(frozen=True)+class _Hold:+ started: float+ caller: str+ stack: tuple[str, ...]+++def _frames_of_borrower() -> list[FrameType]:+ """The call stack of the code borrowing a connection, innermost first.++ A checkout on the async engine runs inside SQLAlchemy's greenlet, whose+ own stack starts at the greenlet and knows nothing of the coroutine that+ awaited it. That coroutine is suspended in the parent greenlet, so the+ walk continues from the parent's frame when the local one runs out.+ """+ frames: list[FrameType] = []+ frame: FrameType | None = sys._getframe(1)+ while frame is not None:+ frames.append(frame)+ frame = frame.f_back+ current = greenlet.getcurrent()+ parent = current.parent+ if parent is not None and parent.gr_frame is not None:+ frame = parent.gr_frame+ while frame is not None:+ frames.append(frame)+ frame = frame.f_back+ return frames+++def _ours(frame: FrameType) -> bool:+ filename = frame.f_code.co_filename+ return (+ _PACKAGE_MARKER in filename+ and _TESTS_MARKER not in filename+ and not any(p in filename for p in _PLUMBING)+ )+++def _module(frame: FrameType) -> str:+ filename = frame.f_code.co_filename+ tail = filename.rsplit(_PACKAGE_MARKER, 1)[-1]+ return "switch_core." + tail.removesuffix(".py").replace(os.sep, ".")+++def describe_borrower() -> tuple[str, tuple[str, ...]]:+ """`module:function` of the innermost switch_core frame, and a short stack.++ The label is bounded by the code rather than the traffic, so it can be a+ metric attribute. Asked on every checkout: walking a few dozen frames is+ microseconds against a round trip of a hundred or more.+ """+ ours = [f for f in _frames_of_borrower() if _ours(f)]+ if not ours:+ return UNKNOWN_CALLER, ()+ caller = f"{_module(ours[0])}:{ours[0].f_code.co_name}"+ stack = tuple(+ f"{_module(f)}:{f.f_code.co_name}:{f.f_lineno}"+ for f in ours[:_LONG_HOLD_STACK_DEPTH]+ )+ return caller, stack+++class _LongHoldLog:+ """Rate-limits the long-hold warning per caller."""++ def __init__(self) -> None:+ self._lock = threading.Lock()+ self._last: dict[str, float] = {}+ self._suppressed: dict[str, int] = {}++ def report(self, hold: _Hold, held_seconds: float) -> None:+ now = time.monotonic()+ with self._lock:+ last = self._last.get(hold.caller)+ if last is not None and now - last < _LONG_HOLD_LOG_INTERVAL_SECONDS:+ self._suppressed[hold.caller] = self._suppressed.get(hold.caller, 0) + 1+ return+ self._last[hold.caller] = now+ suppressed = self._suppressed.pop(hold.caller, 0)+ logger.warning(+ "A database connection was held for %.2f s by %s (%d more long holds "+ "by it in the last %d s). Stack: %s",+ held_seconds,+ hold.caller,+ suppressed,+ int(_LONG_HOLD_LOG_INTERVAL_SECONDS),+ " < ".join(hold.stack) or "unknown",+ )+++def install_hold_timer(engine: AsyncEngine) -> None:+ """Time every checkout to its checkin, by the code that borrowed it.++ The checkout event fires in the borrower's call, which is the only moment+ its stack is there to read; the checkin fires wherever the connection+ happens to be returned, so the borrower is stored on the connection+ record in between.+ """+ long_holds = _LongHoldLog()++ @event.listens_for(engine.sync_engine, "checkout")+ def _on_checkout(_dbapi: object, record: Any, _proxy: object) -> None:+ caller, stack = describe_borrower()+ record.info[_HOLD_KEY] = _Hold(time.perf_counter(), caller, stack)++ @event.listens_for(engine.sync_engine, "checkin")+ def _on_checkin(_dbapi: object, record: Any) -> None:+ hold = record.info.pop(_HOLD_KEY, None) if record is not None else None+ if hold is None:+ return+ held = time.perf_counter() - hold.started+ metrics().observe(DB_POOL_HOLD_DURATION, {"caller": hold.caller}, held * 1000.0)+ if held >= LONG_HOLD_SECONDS:+ long_holds.report(hold, held)+++class WaitTimedQueuePool(AsyncAdaptedQueuePool):+ """The default async pool, timing how long each request waits for a+ connection.++ A subclass rather than an event because the pool has no event for the+ request, only for the handover. Passed in by `main.py` as the engine's+ `poolclass`, so the database layer stays unaware of any of this; a pool+ rebuilt by `dispose()` keeps the class, and with it the timing.+ """++ def _do_get(self) -> ConnectionPoolEntry:+ started = time.perf_counter()+ try:+ return super()._do_get()+ finally:+ # A wait that ends in a pool timeout is the one that matters most.+ metrics().observe(+ DB_POOL_WAIT_DURATION, {}, (time.perf_counter() - started) * 1000.0+ )diff --git a/core/switch_core/room_service.py b/core/switch_core/room_service.pyindex b7bdaf06e..cdb385892 100644--- a/core/switch_core/room_service.py+++ b/core/switch_core/room_service.py@@ -52,6 +52,7 @@ if TYPE_CHECKING:from switch_core.bridges.collaboration.lifecycle_service import (CollaborationBridgeLifecycleService,)+ from switch_core.transport.room_cache import RoomDeliveryCachelogger = logging.getLogger(__name__)@@ -167,6 +168,7 @@ class RoomService:collab_bridge_store: CollaborationBridgeStore,resource_service: ResourceService,session_factory: async_sessionmaker[AsyncSession],+ room_cache: RoomDeliveryCache,telemetry: TelemetryService | None = None,) -> None:self._provisioning = provisioning@@ -180,6 +182,8 @@ class RoomService:# Optional because several tests and tooling build a RoomService# without one; `emit_safely` treats None as "report nothing".self._telemetry = telemetry+ # The shared delivery read; told when a room goes.+ self._room_cache = room_cacheasync def _resolve_agent_ids(self, config: RoomCreateConfig) -> list[str]:if config.agent_ids is not None:@@ -806,6 +810,10 @@ class RoomService:await self._room_store.delete(session, room_id)await session.commit()+ # Kicking the members above already emptied it if they were all+ # running here; this covers any that were not.+ self._room_cache.invalidate(room.tenant_id, room_id)+if bridge_id:collaboration_core = self._collab_lifecycle.get(bridge_id)if collaboration_core:diff --git a/core/switch_core/transport/postgres.py b/core/switch_core/transport/postgres.pyindex 9ab82f0c5..0ca83ce15 100644--- a/core/switch_core/transport/postgres.py+++ b/core/switch_core/transport/postgres.py@@ -6,7 +6,7 @@ of that. **The write is the send.** A send is an INSERT, numbered per roomunder an advisory lock (`MessageStore.create`); the `messages_notify` triggerannounces it on commit (`db/notify_ddl.py`); one `MessageListener` wakes thetransports watching that room (`messages/notify.py`); and each transport reads-the rows after its own cursor, `_DELIVERY_PAGE` at a time. There is no second+the rows after its own cursor, `DELIVERY_PAGE` at a time. There is no secondstore to agree with. Switch used to run on a Matrix homeserver; the`matrix_*` column names are what is left of it, kept as stable ids.@@ -40,7 +40,6 @@ from switch_core.db.models import (ClientRoom,MediaBlob,Message,- MessageAttachment,UsageMetric,)from switch_core.db.session_scope import tenant_session@@ -60,6 +59,7 @@ from switch_core.transport.content import media_content, message_contentfrom switch_core.transport.ephemeral import EphemeralBusfrom switch_core.transport.invites import InviteBusfrom switch_core.transport.port import Handler, TransportHandlers+from switch_core.transport.room_cache import DELIVERY_PAGEfrom switch_core.transport.stored import to_inboundfrom switch_core.transport.types import (DownloadResult,@@ -83,13 +83,12 @@ if TYPE_CHECKING:from switch_core.db.stores.room_store import RoomStorefrom switch_core.db.stores.usage_store import UsageStorefrom switch_core.messages.notify import MessageListener+ from switch_core.transport.room_cache import RoomDeliveryCache+ from switch_core.transport.stored import DeliveryAttachments, DeliveryRow-logger = logging.getLogger(__name__)+ DeliveryPage = tuple[list[tuple[DeliveryRow, DeliveryAttachments]], bool]-# How many rows one wake-up reads at a time. A page rather than everything-# outstanding, so a room that moved a long way while a handler was busy is-# delivered in bounded steps instead of one unbounded read.-_DELIVERY_PAGE = 200+logger = logging.getLogger(__name__)# What makes an `m.room.message` a file rather than text._MEDIA_MSGTYPES = frozenset({"m.image", "m.file", "m.video", "m.audio"})@@ -175,6 +174,7 @@ class PostgresTransport:listener: MessageListener,invites: InviteBus,ephemeral: EphemeralBus,+ room_cache: RoomDeliveryCache,) -> None:self.user_id = user_idself.client_id = client_id@@ -204,6 +204,11 @@ class PostgresTransport:self._listener = listenerself._invites = invitesself._ephemeral = ephemeral+ # Shared with every other transport in the process.+ self._room_cache = room_cache+ # When this client was last woken for each room, on the cache's clock.+ # A shared read only counts for this client if it started after this.+ self._woken: dict[str, int] = {}self._handlers = TransportHandlers()# Per-room delivery position, and the transport-side id to hand back# to a handler. Both are keyed by the Switch room id, which is what@@ -435,13 +440,13 @@ class PostgresTransport:# already undone the claim and there is nothing subscribed to undo.returnself._cursors[room_id] = seq+ self._room_cache.attach(self.tenant_id, room_id, self)self._listener.subscribe(room_id, self._on_room_advanced)self._ephemeral.subscribe(transport_room_id, self._on_ephemeral)if from_seq is not None:# The announcement for anything already written went out before# this subscription existed, so nothing will wake the loop for it.- self._pending.add(room_id)- self._wake.set()+ self._mark_pending(room_id)def _unwatch(self, transport_room_id: str) -> None:"""Stop delivering one room. Not watching it is success.@@ -469,6 +474,9 @@ class PostgresTransport:# A wake-up already queued for this room would otherwise be drained# after the subscription was dropped.self._pending.discard(room_id)+ self._woken.pop(room_id, None)+ # The last member out takes the room's cached rows with it.+ self._room_cache.detach(self.tenant_id, room_id, self)def _unwatch_all(self) -> None:"""Drop every room, through the same path a single removal takes."""@@ -496,6 +504,15 @@ class PostgresTransport:"""if room_id not in self._watching:return+ self._mark_pending(room_id)++ def _mark_pending(self, room_id: str) -> None:+ """Queue a room for this client's loop, remembering when.++ The tick is taken here, after the commit that caused the wake, so a+ shared read that started later is known to have seen it.+ """+ self._woken[room_id] = self._room_cache.tick()self._pending.add(room_id)self._wake.set()@@ -516,6 +533,11 @@ class PostgresTransport:Every delivery is a suspension point, so the subscription is re-readper row. A removal landing mid-page has to stop the rows it has notreached, and must not be undone by a cursor write after it.++ A page comes from the room's one shared read when the cache can vouch+ for it from this cursor, and from this client's own read when it+ cannot. Only the read is shared: the rows are still handed over here,+ one at a time, under the same checks."""transport_room_id = self._watching.get(room_id)if transport_room_id is None:@@ -527,6 +549,10 @@ class PostgresTransport:# row; the map only ever held this value, and two sources for one fact# is how they come to disagree.tenant_id = self.tenant_id+ # Taken once per pass: a wake landing during it queues another pass,+ # which takes the newer one. A room queued without a wake time (no+ # path does that today) gets "now", which only ever costs a read.+ woken_at = self._woken.get(room_id) or self._room_cache.tick()# Bound for the read *and* the delivery below: a handler (posting to a# bridge, gating a command) opens its own sessions rather than reusing# this one, and those still need the room's tenant. The contextvar@@ -537,33 +563,72 @@ class PostgresTransport:cursor = self._cursors.get(room_id)if cursor is None or room_id not in self._watching:return- async with tenant_session(self._session_factory, tenant_id) as session:- rows = await self._message_store.list_for_room(- session,- room_id,- after_seq=cursor,- limit=_DELIVERY_PAGE,- )- attachments = await self._message_store.attachments_for(- session, [row.id for row in rows]- )+ rows, done = await self._read_shared_page(+ room_id, tenant_id, cursor, woken_at+ )+ if self._cursors.get(room_id) != cursor:+ # Removed and put back while waiting on the room's+ # shared read: these rows are from the old+ # subscription's position, not the new one's.+ continueif not rows:return- for row in rows:+ for row, attachments in rows:if room_id not in self._watching:returnself._cursors[room_id] = row.seq- await self._deliver(- transport_room_id, row, attachments.get(row.id, [])- )- if len(rows) < _DELIVERY_PAGE:+ await self._deliver(transport_room_id, row, attachments)+ if done:return+ async def _read_page(+ self, room_id: str, tenant_id: str, cursor: int+ ) -> DeliveryPage:+ """This client's own read of the next page, for when the room's+ shared read cannot serve it. `done` when the page came back short."""+ async with tenant_session(self._session_factory, tenant_id) as session:+ rows = await self._message_store.list_for_room(+ session,+ room_id,+ after_seq=cursor,+ limit=DELIVERY_PAGE,+ )+ attachments = await self._message_store.attachments_for(+ session, [row.id for row in rows]+ )+ page: list[tuple[DeliveryRow, DeliveryAttachments]] = [+ (row, attachments.get(row.id, [])) for row in rows+ ]+ return page, len(rows) < DELIVERY_PAGE++ async def _read_shared_page(+ self, room_id: str, tenant_id: str, cursor: int, woken_at: int+ ) -> DeliveryPage:+ """The next page from the room's shared read, or this client's own+ when the cache cannot vouch for it (cursor behind what it holds,+ entry dropped while waiting).++ The key is this client's tenant and a room it resolved under that+ tenant, which is what makes a hit as good as the read it replaces.+ """+ page = await self._room_cache.read(+ tenant_id,+ room_id,+ after_seq=cursor,+ woken_at=woken_at,+ limit=DELIVERY_PAGE,+ )+ if page is None:+ if room_id not in self._watching:+ return [], True+ return await self._read_page(room_id, tenant_id, cursor)+ return [(row, row.attachments) for row in page.rows], page.done+async def _deliver(self,transport_room_id: str,- row: Message,- attachments: list[MessageAttachment],+ row: DeliveryRow,+ attachments: DeliveryAttachments,) -> None:room = RoomRef(room_id=transport_room_id)event = to_inbound(row, attachments, transport_room_id=transport_room_id)diff --git a/core/switch_core/transport/room_cache.py b/core/switch_core/transport/room_cache.pynew file mode 100644index 000000000..7f1ee0c4d--- /dev/null+++ b/core/switch_core/transport/room_cache.py@@ -0,0 +1,587 @@+"""One read of a room's new messages, shared by every client watching it.++Each client has its own `PostgresTransport`, and each transport reads the rows+after its own cursor when the listener says a room moved. A room with six+agents in it therefore reads every new message six times, through six pool+checkouts, at the same moment. Under a restart burst that is what exhausted+the pool. This keeps one bounded copy of a room's recent rows in the process,+so the first transport to need them reads them and the rest are handed the+same rows.++**What an entry claims.** An entry for a room holds every row whose `seq` is+in `(floor, ceiling]`, exactly, and nothing else. A fill reads `seq >+ceiling` and appends, so the range only ever grows at the top and stays+contiguous. Trimming raises `floor`. A reader whose cursor is in+`[floor, ceiling)` is served from memory; one below `floor` reads the+database itself, and comes back once its cursor reaches the range. No read is+ever served across a hole, because the entry never has one.++**When an entry is current enough.** A row is visible to a read only if it+committed before the read's snapshot, and a reader woken for a row must not+be told "nothing new" by a read that started before that row committed. So+time here is a counter, `tick()`: a transport takes one when it is woken+(after the NOTIFY, so after the commit), and a fill takes one before it+queries. A fill that started after a reader's wake has seen what the reader+was woken for. `complete_as_of` is the start of the latest fill that reached+the end of the room: every row committed before it is at or below `ceiling`.+A reader may be told it has caught up only when that tick is later than its+own wake. This is what keeps the missed-notification window closed on the+first read after subscribing, without a database read per transport.++**One fill per room at a time.** It runs as a task of its own, not in any+transport's loop, so a client that is cancelled mid-read does not cancel the+read every other member is waiting on. Never from the listener's fan-out+either: a waker only notes the room, as it always has.++**Rows are immutable snapshots.** A row and its attachments are published+together, in one step with no await, after both reads finish. `content` is+kept as JSON text and parsed afresh for every delivery, so no two clients ever+hold the same dict. Attachment metadata only: the bytes live in `media_blobs`+and are never read here.++**Messages are append-only.** The NOTIFY trigger fires on INSERT only+(`db/notify_ddl.py`), nothing updates a delivered row, and deleting a room+cascades its messages away. A cached row is therefore never stale. A future+edit or redaction path must call `invalidate` for the room, or members would+keep being handed the old text from memory.++**What it does not do.** It checks nothing about who may read. The key is+`(tenant_id, room_id)`, taken from a transport that resolved the room under+its own tenant, and every fill reads under that tenant's session, so row-level+security applies to the fill exactly as it did to each transport's own read.+Whether *this* client is still in the room, and whether it wants the message,+stays with the transport and its consumer, checked after every await.+"""++from __future__ import annotations++import asyncio+import bisect+import json+import logging+import time+from collections import OrderedDict+from dataclasses import dataclass+from datetime import datetime+from typing import TYPE_CHECKING, Any, cast++from switch_core.db.session_scope import tenant_session+from switch_core.observability.catalogue import (+ DELIVERY_CACHE_EVICTIONS,+ DELIVERY_CACHE_FILLS,+ DELIVERY_CACHE_READS,+ DELIVERY_CACHE_ROWS_READ,+)+from switch_core.observability.metrics import metrics+from switch_core.tenant_context import no_tenant++if TYPE_CHECKING:+ from collections.abc import Iterable++ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker++ from switch_core.db.models import Message, MessageAttachment+ from switch_core.db.stores.message_store import MessageStore++logger = logging.getLogger(__name__)++# How many rows one wake-up reads at a time, by a transport or by a fill. A+# page rather than everything outstanding, so a room that moved a long way+# while a handler was busy is delivered in bounded steps instead of one+# unbounded read. Here rather than beside the transport so the process can+# size the cache without naming a transport implementation.+DELIVERY_PAGE = 200++# What a row costs beyond its strings: the object, its tuple slot, the+# datetime. An estimate, so the byte limit is a budget rather than a measure.+_ROW_OVERHEAD_BYTES = 400++# How many fills one read may wait for before it gives up and lets the+# transport read the database. Each fill started after the reader's wake+# either reaches the end of the room or adds a page, so one or two is the+# norm; this bounds the odd case of a cursor far ahead of an old entry.+_MAX_ROUNDS = 4++CacheKey = tuple[str, str]+++@dataclass(frozen=True, slots=True)+class CachedAttachment:+ """A file's metadata, as `to_inbound` reads it off `MessageAttachment`."""++ uri: str+ filename: str | None+ mimetype: str | None+ size: int | None+++@dataclass(frozen=True, slots=True)+class CachedRow:+ """A message row frozen at the moment it was read.++ Carries the attributes `to_inbound` and the delivery metrics read off a+ `Message`, under the same names, so both paths build an event the same+ way.+ """++ id: str+ seq: int+ transport_event_id: str+ sender_id: str+ sender_name: str | None+ event_type: str+ msgtype: str | None+ body: str | None+ formatted_body: str | None+ thread_root_event_id: str | None+ sent_at: datetime+ content_json: str+ attachments: tuple[CachedAttachment, ...]+ size: int++ @property+ def content(self) -> dict[str, Any]:+ """A fresh dict on every access, so one client's edit is its own."""+ return cast("dict[str, Any]", json.loads(self.content_json))++ @classmethod+ def of(cls, row: Message, attachments: Iterable[MessageAttachment]) -> CachedRow:+ files = tuple(+ CachedAttachment(+ uri=file.uri,+ filename=file.filename,+ mimetype=file.mimetype,+ size=file.size,+ )+ for file in attachments+ )+ content_json = json.dumps(row.content, separators=(",", ":"))+ texts = (+ row.transport_event_id,+ row.sender_id,+ row.sender_name,+ row.event_type,+ row.msgtype,+ row.body,+ row.formatted_body,+ row.thread_root_event_id,+ content_json,+ )+ size = _ROW_OVERHEAD_BYTES + sum(len(text) for text in texts if text)+ size += sum(+ len(file.uri) + len(file.filename or "") + len(file.mimetype or "") + 64+ for file in files+ )+ return cls(+ id=row.id,+ seq=row.seq,+ transport_event_id=row.transport_event_id,+ sender_id=row.sender_id,+ sender_name=row.sender_name,+ event_type=row.event_type,+ msgtype=row.msgtype,+ body=row.body,+ formatted_body=row.formatted_body,+ thread_root_event_id=row.thread_root_event_id,+ sent_at=cast("datetime", row.sent_at),+ content_json=content_json,+ attachments=files,+ size=size,+ )+++@dataclass(frozen=True)+class RoomCacheLimits:+ """Process-wide bounds. Whatever falls outside them is read from the+ database, so a limit set too low costs reads, never messages."""++ max_bytes: int+ max_rooms: int+ max_rows_per_room: int+ max_age_seconds: float+++@dataclass(frozen=True)+class RoomCacheStats:+ bytes: int+ rooms: int+ rows: int+++@dataclass(frozen=True)+class CachedPage:+ """Rows after a reader's cursor, in `seq` order with no gaps.++ `done` means nothing else was committed before the reader was woken, so+ it can stop without asking again.+ """++ rows: tuple[CachedRow, ...]+ done: bool+++class _Fill:+ __slots__ = ("after", "rows", "started", "task")++ def __init__(self, started: int, after: int) -> None:+ self.started = started+ # It reads the rows after this, so what it holds is (after, last].+ self.after = after+ # What it read, once checked; kept so its waiters can still use it if+ # the entry is dropped before it lands.+ self.rows: tuple[CachedRow, ...] | None = None+ self.task: asyncio.Task[None] | None = None+++class _Entry:+ __slots__ = (+ "born",+ "bytes",+ "ceiling",+ "complete_as_of",+ "dropped",+ "filling",+ "floor",+ "rows",+ )++ def __init__(self, floor: int) -> None:+ self.floor = floor+ self.ceiling = floor+ self.rows: tuple[CachedRow, ...] = ()+ # When each row was cached, for the age limit. Parallel to `rows`.+ self.born: tuple[float, ...] = ()+ self.bytes = 0+ # No fill has reached the end yet, so no wake is covered.+ self.complete_as_of = -1+ self.filling: _Fill | None = None+ # Why it was dropped, once it has been.+ self.dropped: str | None = None+++class RoomDeliveryCache:+ """The process's shared read of recent rows, one entry per watched room."""++ def __init__(+ self,+ *,+ session_factory: async_sessionmaker[AsyncSession],+ message_store: MessageStore,+ limits: RoomCacheLimits,+ page: int = DELIVERY_PAGE,+ ) -> None:+ if limits.max_rooms < 1 or limits.max_bytes < 1:+ raise ValueError("the room delivery cache needs room for one room")+ if limits.max_rows_per_room < page:+ # A fill reads a page; a room that cannot hold one would trim what+ # it just read before anyone was served from it.+ raise ValueError(+ f"max_rows_per_room ({limits.max_rows_per_room}) must be at "+ f"least the delivery page ({page})"+ )+ self._session_factory = session_factory+ self._message_store = message_store+ self._limits = limits+ self._page = page+ # Least recently read first, so eviction pops from the front.+ self._entries: OrderedDict[CacheKey, _Entry] = OrderedDict()+ # Who is watching each room. An entry exists only while someone is,+ # which is what purges a room every member has left or that was+ # deleted (deleting a room removes its members first).+ self._watchers: dict[CacheKey, set[object]] = {}+ self._bytes = 0+ self._clock = 0++ # ── Time ──────────────────────────────────────────────────────────────────++ def tick(self) -> int:+ """A moment, ordered against every other moment in this process.++ Synchronous and strictly increasing, which is all the handoff needs:+ whether a fill started after a reader was woken.+ """+ self._clock += 1+ return self._clock++ # ── Who is watching ───────────────────────────────────────────────────────++ def attach(self, tenant_id: str, room_id: str, watcher: object) -> None:+ self._watchers.setdefault((tenant_id, room_id), set()).add(watcher)++ def detach(self, tenant_id: str, room_id: str, watcher: object) -> None:+ """Forget a watcher; drop the room's rows when it was the last.++ Safe for a watcher that never attached, like the buses' unsubscribe,+ because `_release_claim` undoes a claim that may never have+ subscribed.+ """+ key = (tenant_id, room_id)+ watchers = self._watchers.get(key)+ if watchers is None:+ return+ watchers.discard(watcher)+ if watchers:+ return+ del self._watchers[key]+ self._drop(key, "unwatched")++ def invalidate(self, tenant_id: str, room_id: str) -> None:+ """Drop everything held for a room.++ The hook for anything that changes a row after it was written (an+ edit, a redaction) or removes the room. Nothing does the former today;+ see the module docstring. A fill in flight for the room is discarded+ when it lands, and its waiters read the database: it may have read+ rows from before the change.+ """+ self._drop((tenant_id, room_id), "invalidated")++ # ── Reading ───────────────────────────────────────────────────────────────++ async def read(+ self,+ tenant_id: str,+ room_id: str,+ *,+ after_seq: int,+ woken_at: int,+ limit: int,+ ) -> CachedPage | None:+ """Rows after `after_seq`, or None for "read the database yourself".++ `woken_at` is the reader's tick from when it was last woken for this+ room. An empty page is only ever answered once a fill that started+ after it has reached the end of the room.+ """+ key = (tenant_id, room_id)+ waited = False+ for round_ in range(_MAX_ROUNDS + 1):+ if key not in self._watchers:+ self._count("unwatched")+ return None+ entry = self._entries.get(key)+ if entry is None:+ entry = self._create(key, floor=after_seq)+ else:+ self._entries.move_to_end(key)+ self._expire(key, entry)+ if after_seq < entry.floor:+ self._count("behind")+ return None+ if after_seq < entry.ceiling:+ start = bisect.bisect_right(entry.rows, after_seq, key=_seq_of)+ rows = entry.rows[start : start + limit]+ done = rows[-1].seq >= entry.ceiling and entry.complete_as_of > woken_at+ self._count("filled" if waited else "hit")+ return CachedPage(rows=rows, done=done)+ if entry.complete_as_of > woken_at:+ self._count("filled" if waited else "hit")+ return CachedPage(rows=(), done=True)+ if round_ == _MAX_ROUNDS:+ break+ fill = entry.filling+ if fill is None:+ fill = self._start_fill(key, entry)+ # A fill that started before this reader was woken may not have+ # seen what woke it. It is still waited for rather than run+ # beside, so a room has one read in flight; the next round starts+ # a fresh one.+ waited = True+ assert fill.task is not None+ await asyncio.shield(fill.task)+ if self._entries.get(key) is not entry:+ return self._from_dropped(+ key,+ entry,+ fill,+ after_seq=after_seq,+ woken_at=woken_at,+ limit=limit,+ )+ self._count("gave_up")+ return None++ def _from_dropped(+ self,+ key: CacheKey,+ entry: _Entry,+ fill: _Fill,+ *,+ after_seq: int,+ woken_at: int,+ limit: int,+ ) -> CachedPage | None:+ """Serve a waiter from a fill whose entry was dropped while it read.++ Evicted for room or bytes, the rows are as good as ever; they are just+ not kept. Sending every waiter to the database instead would put a+ read per member on top of the one already made, which under churn is+ the load this cache exists to remove. Invalidated means a row may have+ changed under the read, so its waiters still read the database.+ """+ if key not in self._watchers:+ self._count("unwatched")+ return None+ if (+ entry.dropped == "invalidated"+ or fill.rows is None+ or after_seq < fill.after+ ):+ self._count("evicted")+ return None+ rows = tuple(row for row in fill.rows if row.seq > after_seq)+ reached_end = len(fill.rows) < self._page and fill.started > woken_at+ if not rows and not reached_end:+ self._count("evicted")+ return None+ served = rows[:limit]+ self._count("filled")+ return CachedPage(rows=served, done=reached_end and len(served) == len(rows))++ def stats(self) -> RoomCacheStats:+ return RoomCacheStats(+ bytes=self._bytes,+ rooms=len(self._entries),+ rows=sum(len(entry.rows) for entry in self._entries.values()),+ )++ # ── Filling ───────────────────────────────────────────────────────────────++ def _create(self, key: CacheKey, *, floor: int) -> _Entry:+ entry = _Entry(floor)+ self._entries[key] = entry+ while len(self._entries) > self._limits.max_rooms:+ oldest = next(iter(self._entries))+ self._drop(oldest, "rooms")+ return entry++ def _start_fill(self, key: CacheKey, entry: _Entry) -> _Fill:+ # Ticked here, before the task can query, so a reader woken after+ # this moment never mistakes this fill for one that saw its row.+ fill = _Fill(started=self.tick(), after=entry.ceiling)+ task = asyncio.create_task(+ self._fill(key, entry, fill), name="room-delivery-cache-fill"+ )+ # Every waiter may have been cancelled; the outcome is still+ # retrieved, so a failed fill does not warn "never retrieved".+ task.add_done_callback(_retrieve)+ fill.task = task+ entry.filling = fill+ return fill++ async def _fill(self, key: CacheKey, entry: _Entry, fill: _Fill) -> None:+ tenant_id, room_id = key+ after = fill.after+ try:+ # The task inherits its creator's context; the session below binds+ # the tenant explicitly, and nothing else here should inherit one.+ with no_tenant():+ async with tenant_session(self._session_factory, tenant_id) as session:+ rows = await self._message_store.list_for_room(+ session, room_id, after_seq=after, limit=self._page+ )+ attachments = await self._message_store.attachments_for(+ session, [row.id for row in rows]+ )+ snapshots = tuple(+ CachedRow.of(row, attachments.get(row.id, ())) for row in rows+ )+ except BaseException:+ if entry.filling is fill:+ entry.filling = None+ metrics().increment(DELIVERY_CACHE_FILLS, {"outcome": "failed"})+ raise+ if entry.filling is fill:+ entry.filling = None+ metrics().increment(DELIVERY_CACHE_ROWS_READ, {}, float(len(snapshots)))+ if snapshots and snapshots[0].seq <= after:+ # `list_for_room` reads `seq > after` in order, so this cannot+ # happen; if it ever does, holding these rows would break the+ # range the entry claims.+ logger.error(+ "A fill for room %s returned seq %d at or below %d; discarding",+ room_id,+ snapshots[0].seq,+ after,+ )+ metrics().increment(DELIVERY_CACHE_FILLS, {"outcome": "discarded"})+ return+ fill.rows = snapshots+ if self._entries.get(key) is not entry or entry.ceiling != after:+ # Evicted, unwatched or invalidated while reading. Publishing into+ # a dropped entry would resurrect it outside the limits; its+ # waiters may still take the rows (`_from_dropped`).+ metrics().increment(DELIVERY_CACHE_FILLS, {"outcome": "discarded"})+ return+ self._publish(key, entry, fill, snapshots)+ metrics().increment(DELIVERY_CACHE_FILLS, {"outcome": "ok"})++ def _publish(+ self,+ key: CacheKey,+ entry: _Entry,+ fill: _Fill,+ snapshots: tuple[CachedRow, ...],+ ) -> None:+ """Make a fill's rows visible. No await, so nobody sees half of it."""+ now = time.monotonic()+ entry.rows = entry.rows + snapshots+ entry.born = entry.born + (now,) * len(snapshots)+ if snapshots:+ entry.ceiling = snapshots[-1].seq+ if len(snapshots) < self._page:+ # It read to the end of the room as of its snapshot.+ entry.complete_as_of = max(entry.complete_as_of, fill.started)+ added = sum(row.size for row in snapshots)+ entry.bytes += added+ self._bytes += added+ excess = len(entry.rows) - self._limits.max_rows_per_room+ if excess > 0:+ self._trim(entry, excess)+ metrics().increment(DELIVERY_CACHE_EVICTIONS, {"reason": "rows"})+ while self._bytes > self._limits.max_bytes and self._entries:+ self._drop(next(iter(self._entries)), "bytes")++ # ── Letting go ────────────────────────────────────────────────────────────++ def _expire(self, key: CacheKey, entry: _Entry) -> None:+ """Trim rows past the age limit. Lazy, on read: the byte and room+ limits are what bound memory, this only stops a quiet room holding+ rows nobody will ask for again."""+ horizon = time.monotonic() - self._limits.max_age_seconds+ stale = bisect.bisect_left(entry.born, horizon)+ if stale:+ self._trim(entry, stale)+ metrics().increment(DELIVERY_CACHE_EVICTIONS, {"reason": "age"})++ def _trim(self, entry: _Entry, count: int) -> None:+ """Drop the oldest `count` rows; the range now starts above them."""+ dropped = entry.rows[:count]+ entry.floor = dropped[-1].seq+ entry.rows = entry.rows[count:]+ entry.born = entry.born[count:]+ freed = sum(row.size for row in dropped)+ entry.bytes -= freed+ self._bytes -= freed++ def _drop(self, key: CacheKey, reason: str) -> None:+ entry = self._entries.pop(key, None)+ if entry is None:+ return+ entry.dropped = reason+ self._bytes -= entry.bytes+ metrics().increment(DELIVERY_CACHE_EVICTIONS, {"reason": reason})++ # ── Counting ──────────────────────────────────────────────────────────────++ def _count(self, outcome: str) -> None:+ metrics().increment(DELIVERY_CACHE_READS, {"outcome": outcome})+++def _seq_of(row: CachedRow) -> int:+ return row.seq+++def _retrieve(task: asyncio.Task[None]) -> None:+ if not task.cancelled():+ task.exception()diff --git a/core/switch_core/transport/stored.py b/core/switch_core/transport/stored.pyindex ca9e93932..6c9aaf528 100644--- a/core/switch_core/transport/stored.py+++ b/core/switch_core/transport/stored.py@@ -2,7 +2,7 @@from __future__ import annotations-from typing import Any+from typing import TYPE_CHECKING, Anyfrom switch_core.attachments import ATTACHMENT_GROUP_KEYfrom switch_core.db.models import Message, MessageAttachment@@ -16,10 +16,20 @@ from switch_core.transport.types import (InboundMessage,)+if TYPE_CHECKING:+ from collections.abc import Sequence++ from switch_core.transport.room_cache import CachedAttachment, CachedRow++ # A row as delivery reads it: straight from the table, or the shared copy+ # the room cache holds. `to_inbound` reads the same attributes off either.+ DeliveryRow = Message | CachedRow+ DeliveryAttachments = Sequence[MessageAttachment] | Sequence[CachedAttachment]+def to_inbound(- row: Message,- attachments: list[MessageAttachment],+ row: DeliveryRow,+ attachments: DeliveryAttachments,*,transport_room_id: str,) -> InboundEvent:@@ -28,6 +38,9 @@ def to_inbound(Which of the four inbound shapes a row becomes is read off the row itself:an arrival, a file, a `com.switch.*` payload, or a message. The row keeps thewhole content dict, so nothing is reconstructed here that was not sent.++ A `CachedRow` is shared by every client in the room, so its `content` is+ parsed afresh on each access and every event built here owns its dict."""content = dict(row.content)room_id = transport_room_iddiff --git a/core/tests/benchmarks/server.py b/core/tests/benchmarks/server.pyindex a87f3e662..e1e7768e2 100644--- a/core/tests/benchmarks/server.py+++ b/core/tests/benchmarks/server.py@@ -73,6 +73,7 @@ from switch_core.sessions.contract import ApprovalResultfrom switch_core.tenant_context import bind_tenant_id, tenant_scopefrom switch_core.transport.ephemeral import EphemeralBusfrom switch_core.transport.invites import InviteBus+from switch_core.transport.room_cache import RoomCacheLimits, RoomDeliveryCachefrom tests.benchmarks.instrumentation import (RequestCounter,TracingMiddleware,@@ -474,6 +475,17 @@ async def _serve(invites=invites,)+ room_cache = RoomDeliveryCache(+ session_factory=session_factory,+ message_store=session_env.message_store,+ limits=RoomCacheLimits(+ max_bytes=config.room_delivery_cache_max_bytes,+ max_rooms=config.room_delivery_cache_max_rooms,+ max_rows_per_room=config.room_delivery_cache_max_rows_per_room,+ max_age_seconds=config.room_delivery_cache_max_age_seconds,+ ),+ )+client_factory = ClientFactory(client_store=session_env.client_store,session_factory=session_factory,@@ -484,6 +496,7 @@ async def _serve(listener=message_listener,invites=invites,ephemeral=ephemeral,+ room_cache=room_cache,)client_factory.register("agent",@@ -524,6 +537,7 @@ async def _serve(collab_bridge_store=session_env.bridge_store,resource_service=resource_service,session_factory=session_factory,+ room_cache=room_cache,)app, protocol = create_agent_bridge_app(diff --git a/core/tests/conftest.py b/core/tests/conftest.pyindex e335daff4..70b5692d4 100644--- a/core/tests/conftest.py+++ b/core/tests/conftest.py@@ -29,6 +29,7 @@ from switch_core.db.models import TENANT_ZERO_ID, Tenantfrom switch_core.db.runtime_role import grant_runtime_rolefrom switch_core.keys import Keyringfrom switch_core.tenant_context import tenant_scope+from tests.switch_core.pool_checkouts import PoolCheckouts# Production configures this in `main.run()`; every test that touches a# connection config needs the same, and none needs a particular key.@@ -174,6 +175,23 @@ async def session_factory(await engine.dispose()+def pool_checkouts(+ session_factory: async_sessionmaker[AsyncSession],+) -> Iterator[PoolCheckouts]:+ """Connections borrowed from `session_factory`'s pool, counted per request.++ Wrap an app under test with `pool_checkouts.wrap(app)`, or a block with+ `pool_checkouts.scope(label)`, and assert `pool_checkouts.over() == []`:+ no request held two connections at once. See `pool_checkouts.py`.+ """+ tracker = PoolCheckouts(session_factory.kw["bind"])+ try:+ yield tracker+ finally:+ tracker.close()++@dataclass(frozen=True)class RLSHarness:"""Two ways into the same database, for row-level-security tests thatdiff --git a/core/tests/integration/conftest.py b/core/tests/integration/conftest.pyindex 600b85dc9..5ddde7db0 100644--- a/core/tests/integration/conftest.py+++ b/core/tests/integration/conftest.py@@ -88,6 +88,7 @@ from switch_core.room_service import RoomServicefrom switch_core.tenant_context import tenant_scopefrom switch_core.transport.ephemeral import EphemeralBusfrom switch_core.transport.invites import InviteBus+from switch_core.transport.room_cache import RoomCacheLimits, RoomDeliveryCache# ── Mirrors deploy/local/docker-compose.yml — keep in sync ──────────────────────POSTGRES_IMAGE = "postgres:16-alpine"@@ -554,6 +555,17 @@ async def harness(session_env: SessionEnv) -> AsyncIterator[Harness]:invites=invites,)+ room_cache = RoomDeliveryCache(+ session_factory=session_factory,+ message_store=session_env.message_store,+ limits=RoomCacheLimits(+ max_bytes=config.room_delivery_cache_max_bytes,+ max_rooms=config.room_delivery_cache_max_rooms,+ max_rows_per_room=config.room_delivery_cache_max_rows_per_room,+ max_age_seconds=config.room_delivery_cache_max_age_seconds,+ ),+ )+client_factory = ClientFactory(client_store=session_env.client_store,session_factory=session_factory,@@ -565,6 +577,7 @@ async def harness(session_env: SessionEnv) -> AsyncIterator[Harness]:listener=message_listener,invites=invites,ephemeral=ephemeral,+ room_cache=room_cache,)client_factory.register("agent",@@ -605,6 +618,7 @@ async def harness(session_env: SessionEnv) -> AsyncIterator[Harness]:collab_bridge_store=session_env.bridge_store,resource_service=resource_service,session_factory=session_factory,+ room_cache=room_cache,)protocol = AgentCore(diff --git a/core/tests/switch_core/bridges/agent/api/test_event_stream_compat.py b/core/tests/switch_core/bridges/agent/api/test_event_stream_compat.pyindex 8e7ed15e1..6e4b6430a 100644--- a/core/tests/switch_core/bridges/agent/api/test_event_stream_compat.py+++ b/core/tests/switch_core/bridges/agent/api/test_event_stream_compat.py@@ -34,6 +34,7 @@ from switch_core.bridges.agent.protocol.agent_connections import ()from switch_core.bridges.agent.protocol.event_buffer import EventBufferfrom switch_core.bridges.agent.protocol.types import AgentEvent, MessagePayload+from switch_core.observability.metrics import MetricsRegistry, install, uninstallAGENT_ID = "agent-1"ROOM = "room-a"@@ -201,6 +202,63 @@ async def test_closing_the_stream_detaches_it_and_keeps_the_connection() -> Noneconn = protocol.connections.get("c1")assert conn is not Noneassert not conn.stream_attached+ assert protocol.connections.live_agents_by_transport() == {+ ("detached", "agent-runtime"): 1+ }+++async def test_a_stream_attached_over_sse_is_reported_as_sse() -> None:+ protocol = _Protocol()+ stream = _Reader((await _open(protocol)).body_iterator)+ await stream.frame()++ assert protocol.connections.live_agents_by_transport() == {+ ("sse", "agent-runtime"): 1+ }+ await stream.close()+++async def test_a_client_that_moves_to_the_socket_is_reported_on_the_socket() -> None:+ """The transport is the last attach's: an upgraded Console reattaching the+ same connection over the socket stops counting as an old client."""+ protocol = _Protocol()+ stream = _Reader((await _open(protocol)).body_iterator)+ await stream.frame()++ protocol.connections.open(+ agent_id=AGENT_ID,+ connection_id="c1",+ scope="single",+ delivery_filter="all",+ spawn_capable=False,+ cursor=0,+ declaration=ClientDeclaration(speaks=8, accepts=1, artifact="agent-runtime"),+ expected_generation=None,+ transport="websocket",+ )++ assert protocol.connections.live_agents_by_transport() == {+ ("websocket", "agent-runtime"): 1+ }+ await stream.close()+++async def test_a_refused_stream_is_counted_like_a_refused_socket() -> None:+ registry = MetricsRegistry()+ install(registry)+ try:+ with pytest.raises(HTTPException) as caught:+ await _open(_Protocol(), protocol_version=99, protocol_accepts=99)+ finally:+ uninstall()++ assert caught.value.status_code == 409+ refused = next(+ p for p in registry.collect() if p.name == "switch.agent.connections_refused"+ )+ assert [(dict(p.attributes), p.value) for p in refused.numbers] == [+ ({"reason": "protocol"}, 1.0)+ ]# ── The declared ranges still meet ──────────────────────────────────────────diff --git a/core/tests/switch_core/bridges/agent/protocol/test_event_buffer_cost.py b/core/tests/switch_core/bridges/agent/protocol/test_event_buffer_cost.pynew file mode 100644index 000000000..444d1c50f--- /dev/null+++ b/core/tests/switch_core/bridges/agent/protocol/test_event_buffer_cost.py@@ -0,0 +1,237 @@+"""What the agent event buffer costs per delivered event, and what it logs.++An agent past the cap drops an event on every enqueue, and used to log a+warning on every one: a line per delivered message for an agent that had read+everything. The warning is now at most once per interval per agent, while the+drop counter and the gap a reader is told about stay exact.++The buffer answers reads by walking the agent's retained events, so the walk is+measured: how many events, and how long, per call.+"""++from __future__ import annotations++import logging+from collections.abc import Iterator++import pytest++from switch_core.bridges.agent.protocol import event_buffer as buffer_module+from switch_core.bridges.agent.protocol.event_buffer import (+ OVERFLOW_WARNING_INTERVAL_SECONDS,+ CursorExpiredError,+ EventBuffer,+ Reader,+)+from switch_core.bridges.agent.protocol.types import AgentEvent, MessagePayload+from switch_core.observability.metrics import MetricsRegistry, install, uninstall+from switch_core.observability.otlp import HistogramPoint++_LOGGER = "switch_core.bridges.agent.protocol.event_buffer"+++def registry() -> Iterator[MetricsRegistry]:+ registry = MetricsRegistry()+ install(registry)+ yield registry+ uninstall()+++class _Clock:+ def __init__(self) -> None:+ self.now = 1000.0++ def __call__(self) -> float:+ return self.now+++def clock(monkeypatch: pytest.MonkeyPatch) -> _Clock:+ clock = _Clock()+ monkeypatch.setattr(buffer_module.time, "monotonic", clock)+ return clock+++def _event(index: int, *, addressed: bool = True) -> AgentEvent:+ return AgentEvent(+ type="message",+ room_id="room-1",+ payload=MessagePayload(+ message_id=f"m{index}",+ sender="@someone:test",+ sender_name="someone",+ body=f"hello {index}",+ addressed=addressed,+ timestamp=index,+ ),+ )+++def _overflow_warnings(caplog: pytest.LogCaptureFixture) -> list[logging.LogRecord]:+ return [+ record+ for record in caplog.records+ if record.name == _LOGGER and "exceeded" in record.getMessage()+ ]+++def _dropped(registry: MetricsRegistry) -> dict[tuple, float]:+ payload = next(+ (p for p in registry.collect() if p.name == "switch.agent.events_dropped"),+ None,+ )+ if payload is None:+ return {}+ return {+ tuple(sorted(point.attributes.items())): point.value+ for point in payload.numbers+ }+++def _histograms(registry: MetricsRegistry, name: str) -> dict[str, HistogramPoint]:+ payload = next((p for p in registry.collect() if p.name == name), None)+ assert payload is not None, f"{name} was not recorded"+ return {point.attributes["operation"]: point for point in payload.histograms}+++def test_overflow_warns_once_per_interval_per_agent(clock, registry, caplog):+ caplog.set_level(logging.WARNING, logger=_LOGGER)+ buffer = EventBuffer(+ max_events_per_agent=3, retention_seconds=3600, sequence_base=0+ )++ for index in range(50):+ buffer.enqueue("agent-1", "room-1", _event(index))+ clock.now += 0.1++ # 47 drops inside five seconds: one line, not 47.+ assert len(_overflow_warnings(caplog)) == 1+ assert _dropped(registry) == {(("reason", "overflow"),): 47.0}+++def test_overflow_warning_resumes_after_the_interval_with_a_count(+ clock, registry, caplog+):+ caplog.set_level(logging.WARNING, logger=_LOGGER)+ buffer = EventBuffer(+ max_events_per_agent=3, retention_seconds=3600, sequence_base=0+ )++ for index in range(4):+ buffer.enqueue("agent-1", "room-1", _event(index))+ for index in range(4, 14):+ clock.now += 1+ buffer.enqueue("agent-1", "room-1", _event(index))+ clock.now += OVERFLOW_WARNING_INTERVAL_SECONDS+ buffer.enqueue("agent-1", "room-1", _event(14))++ warnings = _overflow_warnings(caplog)+ assert len(warnings) == 2+ # The ten drops between the two lines are counted into the second.+ assert "plus 10 dropped since the last warning" in warnings[1].getMessage()+ assert _dropped(registry) == {(("reason", "overflow"),): 12.0}+++def test_overflow_warning_is_per_agent(clock, caplog):+ caplog.set_level(logging.WARNING, logger=_LOGGER)+ buffer = EventBuffer(+ max_events_per_agent=1, retention_seconds=3600, sequence_base=0+ )++ for index in range(5):+ buffer.enqueue("agent-1", "room-1", _event(index))+ buffer.enqueue("agent-2", "room-1", _event(index))++ warned = sorted(+ record.getMessage().split()[1] for record in _overflow_warnings(caplog)+ )+ assert warned == ["agent=agent-1", "agent=agent-2"]+++def test_a_quiet_warning_still_leaves_the_gap_for_readers(clock, caplog):+ caplog.set_level(logging.WARNING, logger=_LOGGER)+ buffer = EventBuffer(+ max_events_per_agent=3, retention_seconds=3600, sequence_base=0+ )++ for index in range(20):+ buffer.enqueue("agent-1", "room-1", _event(index))++ assert len(_overflow_warnings(caplog)) == 1+ # A reader from the start is told about every drop, logged or not.+ with pytest.raises(CursorExpiredError) as exc:+ buffer.read_from("agent-1", 0)+ assert exc.value.oldest == 18+ assert exc.value.rooms == ("room-1",)+ assert [item.seq for item in buffer.read_from("agent-1", 17)] == [18, 19, 20]+++def test_removing_an_agent_forgets_its_warning_state(clock, caplog):+ caplog.set_level(logging.WARNING, logger=_LOGGER)+ buffer = EventBuffer(+ max_events_per_agent=1, retention_seconds=3600, sequence_base=0+ )++ for index in range(3):+ buffer.enqueue("agent-1", "room-1", _event(index))+ buffer.remove("agent-1")+ for index in range(3):+ buffer.enqueue("agent-1", "room-1", _event(index))++ assert len(_overflow_warnings(caplog)) == 2+++def test_a_read_records_how_many_events_it_walked(registry):+ buffer = EventBuffer(+ max_events_per_agent=100, retention_seconds=3600, sequence_base=0+ )+ for index in range(10):+ buffer.enqueue("agent-1", "room-1", _event(index))++ buffer.read_from("agent-1", 0)+ # Stopping at the limit walks only as far as the limit was reached.+ buffer.read_from("agent-1", 0, limit=3)++ scanned = _histograms(registry, "switch.agent.buffer.scanned")["read"]+ assert scanned.count == 2+ assert scanned.total == 13.0+++def test_a_read_and_an_unread_count_record_their_duration(registry):+ buffer = EventBuffer(+ max_events_per_agent=100, retention_seconds=3600, sequence_base=0+ )+ reader = Reader(id="session-1", is_session=True)+ buffer.caught_up("agent-1", reader, "room-1", 0, arrived_on=None)+ for index in range(4):+ buffer.enqueue("agent-1", "room-1", _event(index, addressed=index == 3))++ buffer.read_from("agent-1", 0)+ assert buffer.unread("agent-1", "room-1", 4).count == 3++ collected = registry.collect()+ durations = next(+ p for p in collected if p.name == "switch.agent.buffer.scan.duration"+ )+ scanned = next(p for p in collected if p.name == "switch.agent.buffer.scanned")+ assert {p.attributes["operation"]: p.count for p in durations.histograms} == {+ "read": 1,+ "unread": 1,+ }+ assert all(p.total >= 0 for p in durations.histograms)+ assert {p.attributes["operation"]: p.total for p in scanned.histograms} == {+ "read": 4.0,+ "unread": 4.0,+ }+++def test_an_unread_count_with_nothing_to_count_records_no_scan(registry):+ buffer = EventBuffer(sequence_base=0)+ buffer.enqueue("agent-1", "room-1", _event(0))++ # No baseline: answered without walking the buffer.+ assert buffer.unread("agent-1", "room-1", 1).count is None++ names = {p.name for p in registry.collect()}+ assert "switch.agent.buffer.scanned" not in namesdiff --git a/core/tests/switch_core/clients/test_client_tenant_plumbing.py b/core/tests/switch_core/clients/test_client_tenant_plumbing.pyindex 266ad4ea1..d58d88d0b 100644--- a/core/tests/switch_core/clients/test_client_tenant_plumbing.py+++ b/core/tests/switch_core/clients/test_client_tenant_plumbing.py@@ -65,6 +65,7 @@ def _factory(session_factory: async_sessionmaker[AsyncSession]) -> ClientFactorylistener=MagicMock(),invites=InviteBus(),ephemeral=EphemeralBus(),+ room_cache=MagicMock(),)factory.register("user", Actor)return factorydiff --git a/core/tests/switch_core/clients/test_enqueue_debug_logging.py b/core/tests/switch_core/clients/test_enqueue_debug_logging.pynew file mode 100644index 000000000..2dfd54cce--- /dev/null+++ b/core/tests/switch_core/clients/test_enqueue_debug_logging.py@@ -0,0 +1,135 @@+"""The debug line for an enqueued event costs nothing unless DEBUG is on.++It pretty-prints the whole event, and an argument to `logger.debug` is built+before the level is checked. That ran once per agent per message: on a busy+server, thousands of serialisations a second for a line nobody was logging.+"""++from __future__ import annotations++import logging+from types import SimpleNamespace++import pytest++from switch_core.bridges.agent.protocol.types import AgentEvent+from switch_core.clients.agent_consumer import AgentConsumer+from switch_core.transport import InboundMedia, InboundMessage, RoomRef++_LOGGER = "switch_core.clients.agent_consumer"+++def _consumer(enqueued: list[AgentEvent]) -> SimpleNamespace:+ """An unaddressed delivery: no store is consulted, the event is enqueued."""++ async def _resolve_room_meta(_room_id: str) -> SimpleNamespace:+ return SimpleNamespace(+ room_id="room-1", bridge_id=None, channel_type="channel_public"+ )++ return SimpleNamespace(+ agent=SimpleNamespace(id="agent-1", name="helper"),+ _resolve_room_meta=_resolve_room_meta,+ _addressed_without_lookup=lambda _event, _meta: False,+ _event_buffer=SimpleNamespace(+ enqueue=lambda _agent, _room, event: enqueued.append(event)+ ),+ )+++def _message() -> InboundMessage:+ return InboundMessage(+ room_id="!room:server",+ event_id="$evt",+ sender="@someone:switch.local",+ timestamp=1,+ content={"body": "hello"},+ body="hello",+ sender_name="someone",+ )+++def dumps(monkeypatch: pytest.MonkeyPatch) -> list[int]:+ calls: list[int] = []+ real = AgentEvent.model_dump_json++ def counting(self: AgentEvent, **kwargs: object) -> str:+ calls.append(1)+ return real(self, **kwargs) # type: ignore[arg-type]++ monkeypatch.setattr(AgentEvent, "model_dump_json", counting)+ return calls+++async def test_an_event_is_not_serialised_for_a_debug_line_nobody_logs(dumps, caplog):+ caplog.set_level(logging.INFO, logger=_LOGGER)+ enqueued: list[AgentEvent] = []++ await AgentConsumer.on_message(+ _consumer(enqueued), # type: ignore[arg-type]+ RoomRef(room_id="!room:server"),+ _message(),+ )++ assert len(enqueued) == 1+ assert dumps == []+ assert not [r for r in caplog.records if "Enqueuing event" in r.getMessage()]+++async def test_the_debug_line_is_still_logged_when_debug_is_on(dumps, caplog):+ caplog.set_level(logging.DEBUG, logger=_LOGGER)+ enqueued: list[AgentEvent] = []++ await AgentConsumer.on_message(+ _consumer(enqueued), # type: ignore[arg-type]+ RoomRef(room_id="!room:server"),+ _message(),+ )++ assert len(enqueued) == 1+ assert dumps == [1]+ lines = [r for r in caplog.records if "Enqueuing event" in r.getMessage()]+ assert len(lines) == 1+ assert '"body": "hello"' in lines[0].getMessage()++[email protected]("level", [logging.INFO, logging.DEBUG])+async def test_a_media_event_is_serialised_only_when_debug_is_on(dumps, caplog, level):+ caplog.set_level(level, logger=_LOGGER)+ enqueued: list[AgentEvent] = []+ media = InboundMedia(+ room_id="!room:server",+ event_id="$media",+ sender="@someone:switch.local",+ timestamp=1,+ content={"body": "a file"},+ body="a file",+ uri="mxc://server/file",+ )++ await AgentConsumer._emit_media(+ _consumer(enqueued), # type: ignore[arg-type]+ RoomRef(room_id="!room:server"),+ media,+ SimpleNamespace( # type: ignore[arg-type]+ room_id="room-1", bridge_id=None, channel_type="channel_public"+ ),+ False,+ "someone",+ None,+ [],+ "a file",+ )++ assert len(enqueued) == 1+ logged = [r for r in caplog.records if "Enqueuing media event" in r.getMessage()]+ if level == logging.DEBUG:+ assert dumps == [1]+ assert len(logged) == 1+ else:+ assert dumps == []+ assert logged == []diff --git a/core/tests/switch_core/db/stores/test_delivery_cursor_store.py b/core/tests/switch_core/db/stores/test_delivery_cursor_store.pyindex b640e6a74..7da3a46f1 100644--- a/core/tests/switch_core/db/stores/test_delivery_cursor_store.py+++ b/core/tests/switch_core/db/stores/test_delivery_cursor_store.py@@ -60,8 +60,8 @@ class TestPosition:async def test_an_agent_that_was_never_delivered_is_at_zero(self, session_factory: async_sessionmaker[AsyncSession]) -> None:- """Zero, not an error: `seq` starts at 1, so zero already says- "behind everything"."""+ """Zero, not an error: live messages are numbered from 1, so zero+ already says "behind every live message"."""async with session_factory() as session:agent_id, room_id = await _make_agent_and_room(session)await session.commit()diff --git a/core/tests/switch_core/db/stores/test_message_store.py b/core/tests/switch_core/db/stores/test_message_store.pyindex 182f4b93b..195bc8700 100644--- a/core/tests/switch_core/db/stores/test_message_store.py+++ b/core/tests/switch_core/db/stores/test_message_store.py@@ -369,8 +369,8 @@ class TestListForRoom:async def test_after_seq_from_another_room_does_not_skip_rows(self, session_factory: async_sessionmaker[AsyncSession]) -> None:- """`seq` is global, so a cursor is only meaningful within its own room;- rows interleaved from another room must not consume the page."""+ """`seq` is per room, so a cursor is only meaningful within its own+ room; rows interleaved from another room must not consume the page."""store = MessageStore()async with session_factory() as session:room_a = await _make_room(session, "alpha")diff --git a/core/tests/switch_core/fixtures/agent_controllers/controller_beat_request.json b/core/tests/switch_core/fixtures/agent_controllers/controller_beat_request.jsondeleted file mode 100644index da7deae60..000000000--- a/core/tests/switch_core/fixtures/agent_controllers/controller_beat_request.json+++ /dev/null@@ -1,5 +0,0 @@-{- "connection_id": "00000000-0000-4000-8000-0000000000e1",- "generation": 3141592653,- "cursors": { "00000000-0000-4000-8000-0000000000a1": 42 }-}diff --git a/core/tests/switch_core/fixtures/agent_controllers/controller_beat_response.json b/core/tests/switch_core/fixtures/agent_controllers/controller_beat_response.jsondeleted file mode 100644index 0ed1b2e97..000000000--- a/core/tests/switch_core/fixtures/agent_controllers/controller_beat_response.json+++ /dev/null@@ -1 +0,0 @@-{ "agents": ["00000000-0000-4000-8000-0000000000a1"] }diff --git a/core/tests/switch_core/fixtures/agent_controllers/controller_pong.json b/core/tests/switch_core/fixtures/agent_controllers/controller_pong.jsonnew file mode 100644index 000000000..9a13c948e--- /dev/null+++ b/core/tests/switch_core/fixtures/agent_controllers/controller_pong.json@@ -0,0 +1,4 @@+{+ "type": "pong",+ "cursors": { "00000000-0000-4000-8000-0000000000a1": 42 }+}diff --git a/core/tests/switch_core/fixtures/agent_controllers/status_request.json b/core/tests/switch_core/fixtures/agent_controllers/status_request.jsonindex fd2ceec86..452ae818c 100644--- a/core/tests/switch_core/fixtures/agent_controllers/status_request.json+++ b/core/tests/switch_core/fixtures/agent_controllers/status_request.json@@ -1,7 +1,7 @@{"seq": 42,"observed_at": "2026-01-01T00:00:00Z",- "controller": { "version": "0.1.0", "protocol": 1, "assignment_revision": 3 },+ "controller": { "version": "0.1.0", "protocol": 2, "assignment_revision": 3 },"machine": {"platform": { "os": "linux", "arch": "x64", "os_version": "6.1.0" },"disk_free_bytes": 53687091200,diff --git a/core/tests/switch_core/gateway/test_directory_fallback.py b/core/tests/switch_core/gateway/test_directory_fallback.pyindex 13a973dc3..c4af87eb1 100644--- a/core/tests/switch_core/gateway/test_directory_fallback.py+++ b/core/tests/switch_core/gateway/test_directory_fallback.py@@ -84,6 +84,14 @@ class _StubUserStore:return []+class _StubSession:+ """The route commits before asking the platform, so it is not holding a+ connection while the platform answers."""++ async def commit(self) -> None:+ return None++async def _search(*,adapter_result: list[DirectoryUser] | Exception | None,@@ -93,7 +101,7 @@ async def _search(return await search_bridge_directory("b1",query,- None, # type: ignore[arg-type]+ _StubSession(), # type: ignore[arg-type]_StubBridgeStore(), # type: ignore[arg-type]_StubExternalUserStore(known), # type: ignore[arg-type]_StubUserStore(), # type: ignore[arg-type]diff --git a/core/tests/switch_core/gateway/test_held_connections.py b/core/tests/switch_core/gateway/test_held_connections.pynew file mode 100644index 000000000..b95037ba8--- /dev/null+++ b/core/tests/switch_core/gateway/test_held_connections.py@@ -0,0 +1,698 @@+"""No gateway request holds two pooled connections at once (CHOO-3198).++A request that keeps its first connection while it waits for a second one from+the same pool is the shape that starves the pool under load: every connection+ends up held by a request waiting for another, nobody can give one back, and+after the pool timeout they all fail together. The same goes for a connection+held while the request waits on something that is not the database at all: a+messaging platform, Matrix, or bcrypt.++These drive the real routes through the real authentication and admin+dependencies, against real Postgres, and count connections where every path+meets: the pool (`tests/switch_core/pool_checkouts.py`). Two contracts:++- no request ever holds more than one connection at a time;+- every platform call, Matrix call and bcrypt call happens holding none, and+ bcrypt runs off the event loop.+"""++from __future__ import annotations++import secrets+import threading+from collections.abc import AsyncIterator, Callable+from types import SimpleNamespace+from typing import Any+from unittest.mock import MagicMock++import httpx+import pytest+from fastapi import Depends, FastAPI+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker++from switch_core.db.models import (+ TENANT_ZERO_ID,+ Agent,+ ApiKey,+ Client,+ CollaborationBridge,+ Room,+ TenantMember,+ User,+)+from switch_core.db.stores.agent_store import AgentStore+from switch_core.db.stores.collaboration_bridge_store import CollaborationBridgeStore+from switch_core.db.stores.external_user_store import ExternalUserStore+from switch_core.db.stores.room_role_store import RoomRoleStore+from switch_core.db.stores.room_store import RoomStore+from switch_core.db.stores.user_store import UserStore+from switch_core.gateway import auth as gateway_auth+from switch_core.gateway import dependencies as gw_deps+from switch_core.gateway.agents import router as agents_router+from switch_core.gateway.auth import (+ create_jwt,+ get_current_user,+ get_tenant_is_admin,+ hash_password,+)+from switch_core.gateway.auth_routes import router as auth_router+from switch_core.gateway.collaborations import router as bridges_router+from switch_core.gateway.hosted_launches import hosted_settings+from switch_core.gateway.rooms import router as rooms_router+from switch_core.keys import Keyring+from switch_core.room_service import RoomService+from tests.switch_core.pool_checkouts import PoolCheckouts++# Made for each run, so nothing secret-looking is committed.+_SECRET = secrets.token_hex(32)+_KEYRING = Keyring.parse("test:" + _SECRET, legacy_secret=None)+_PASSWORD = secrets.token_urlsafe(16)+++class _Calls:+ """What the fakes below were asked, with the connections held at the time."""++ def __init__(self, tracker: PoolCheckouts) -> None:+ self._tracker = tracker+ self.held: list[tuple[str, int]] = []++ def record(self, what: str) -> None:+ self.held.append((what, self._tracker.held_now()))+++class _Provisioning:+ """Matrix, as `RoomService` sees it."""++ def __init__(self, calls: _Calls) -> None:+ self._calls = calls++ async def invite_to_room(self, transport_room_id: str, user_id: str) -> None:+ self._calls.record("matrix invite")++ async def kick_user(self, transport_room_id: str, user_id: str) -> None:+ self._calls.record("matrix kick")++ async def delete_room(self, transport_room_id: str) -> None:+ self._calls.record("matrix delete")+++class _RunningClients:+ def __init__(self, by_agent: dict[str, Any]) -> None:+ self._by_agent = by_agent++ def get_by_agent_id(self, agent_id: str) -> Any:+ return self._by_agent.get(agent_id)++ def get(self, client_id: str) -> Any:+ return next(+ (c for c in self._by_agent.values() if c.client_id == client_id), None+ )++ def get_by_type(self, client_type: str, tenant_id: str) -> list[Any]:+ return []+++class _Adapter:+ """A messaging platform's adapter: every call is a network round trip."""++ def __init__(self, calls: _Calls) -> None:+ self._calls = calls++ async def search_directory_users(self, query: str) -> list[Any]:+ self._calls.record("platform directory search")+ return [+ SimpleNamespace(+ external_user_id="U1",+ username=query,+ display_name=query,+ email=None,+ )+ ]++ async def home_deeplink(self) -> str | None:+ return None++ async def install_links(self) -> list[Any]:+ return []++ async def install_note(self) -> str | None:+ return None++ async def channel_deeplink(self, external_channel_id: str) -> str | None:+ self._calls.record("platform channel deeplink")+ return None+++class _CollabLifecycle:+ def __init__(self, calls: _Calls) -> None:+ self._calls = calls+ self.calls = calls+ self._adapter = _Adapter(calls)++ def get_adapter(self, bridge_id: str) -> _Adapter:+ return self._adapter++ def get(self, bridge_id: str) -> Any:+ async def ensure_external_user(**_: Any) -> Any:+ self._calls.record("provision external user")+ raise ValueError("refused by the test")++ return SimpleNamespace(ensure_external_user=ensure_external_user)++ def supports_channel_creation(self, bridge_type: str) -> bool:+ return True++ def supports_directory_search(self, bridge_type: str) -> bool:+ return True++ async def check_edited_connection_config(self, **_: Any) -> None:+ self._calls.record("platform credential check")++ async def check_start_guards(self, **_: Any) -> None:+ self._calls.record("start guards")++ async def restart(self, bridge_id: str) -> None:+ self._calls.record("bridge restart")++ def note_preconfigured(self, bridge_id: str, value: bool) -> None:+ pass++ async def remove(self, bridge_id: str) -> None:+ self._calls.record("bridge remove")+++class _Protocol:+ """The parts of `AgentCore` the routes under test reach.++ `delete_agent` does what the real one does to the pool: opens a session of+ its own, then calls out to every bridge.+ """++ def __init__(+ self, calls: _Calls, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ self.room_role_store = RoomRoleStore()+ self.agent_store = AgentStore()+ self.connections = SimpleNamespace(live_connection_ids=lambda: set())+ self._calls = calls+ self._session_factory = session_factory++ async def delete_agent(+ self, *, agent_id: str | None = None, agent_name: str | None = None+ ) -> None:+ async with self._session_factory() as session:+ await AgentStore().get(session, agent_id or "")+ self._calls.record("bridge identity removal")++ async def get_agent_statuses_by_ids_in_session(+ self, session: AsyncSession, room_id: str, agent_ids: list[str]+ ) -> dict[str, Any]:+ return {}+++def _config() -> SimpleNamespace:+ return SimpleNamespace(+ keyring=_KEYRING,+ gateway_cookie_secure=False,+ gateway_tenant_choice_enabled=False,+ gateway_password_login_enabled=True,+ gateway_signup_open=True,+ gateway_signup_max_per_hour=100,+ )+++def _app(+ session_factory: async_sessionmaker[AsyncSession],+ tracker: PoolCheckouts,+ *,+ room_service: RoomService,+ collab: _CollabLifecycle,+ extra: Callable[[FastAPI], None] | None = None,+) -> Any:+ calls = collab.calls++ async def _session() -> AsyncIterator[AsyncSession]:+ async with session_factory() as session:+ yield session++ app = FastAPI()+ app.include_router(rooms_router, prefix="/rooms")+ app.include_router(bridges_router, prefix="/bridges")+ app.include_router(auth_router)+ app.include_router(agents_router, prefix="/agents")+ if extra is not None:+ extra(app)+ overrides: dict[Callable[..., Any], Callable[..., Any]] = {+ gw_deps.get_session: _session,+ gw_deps.get_system_session: _session,+ gw_deps.get_session_factory: lambda: session_factory,+ gw_deps.get_user_store: UserStore,+ gw_deps.get_room_store: RoomStore,+ gw_deps.get_bridge_store: CollaborationBridgeStore,+ gw_deps.get_external_user_store: ExternalUserStore,+ gw_deps.get_room_service: lambda: room_service,+ gw_deps.get_collab_lifecycle: lambda: collab,+ gw_deps.get_protocol: lambda: _Protocol(calls, session_factory),+ gw_deps.get_agent_store: AgentStore,+ gw_deps.get_config: _config,+ hosted_settings: lambda: None,+ }+ app.dependency_overrides.update(overrides)+ return tracker.wrap(app)+++def _client(app: Any, user: User) -> httpx.AsyncClient:+ token = create_jwt(user.id, user.email, user.role, _KEYRING, TENANT_ZERO_ID)+ return httpx.AsyncClient(+ transport=httpx.ASGITransport(app=app),+ base_url="http://test",+ cookies={"switch_auth": token},+ )+++async def _admin(session_factory: async_sessionmaker[AsyncSession]) -> User:+ """A workspace owner: a plain user whose admin bit comes from membership,+ so `get_tenant_is_admin` has to read it."""+ async with session_factory() as session:+ user = User(+ name="owner",+ email="[email protected]",+ role="user",+ password_hash=hash_password(_PASSWORD),+ )+ session.add(user)+ await session.flush()+ session.add(+ TenantMember(tenant_id=TENANT_ZERO_ID, user_id=user.id, role="owner")+ )+ await session.commit()+ return user+++async def _agent(session: AsyncSession, owner: User, name: str) -> tuple[Agent, Any]:+ key = ApiKey(+ user_id=owner.id,+ key_hash=f"hash-{name}",+ encrypted_key="enc",+ label=name,+ type="agent",+ )+ client = Client(transport_user_id=f"@{name}:test", display_name=name, type="agent")+ session.add_all([key, client])+ await session.flush()+ agent = Agent(+ name=name,+ description="d",+ agent_type="session_addressable",+ connector_type="claude_code",+ integration_profile={},+ client_id=client.id,+ api_key_id=key.id,+ owner_id=owner.id,+ )+ session.add(agent)+ await session.flush()+ running = SimpleNamespace(+ client_id=client.id, transport_user_id=client.transport_user_id+ )+ return agent, running+++async def _room(session: AsyncSession, owner: User, name: str) -> Room:+ room = Room(+ transport_room_id=f"!{name}:test",+ name=name,+ description="d",+ owner_id=owner.id,+ )+ session.add(room)+ await session.flush()+ return room+++def _room_service(+ session_factory: async_sessionmaker[AsyncSession],+ calls: _Calls,+ collab: _CollabLifecycle,+ running: dict[str, Any],+) -> RoomService:+ return RoomService(+ provisioning=_Provisioning(calls), # type: ignore[arg-type]+ room_store=RoomStore(),+ agent_store=AgentStore(),+ client_lifecycle=_RunningClients(running), # type: ignore[arg-type]+ collab_lifecycle=collab, # type: ignore[arg-type]+ collab_bridge_store=CollaborationBridgeStore(),+ resource_service=None, # type: ignore[arg-type]+ session_factory=session_factory,+ room_cache=MagicMock(),+ )+++def _assert_one_at_a_time(tracker: PoolCheckouts, calls: _Calls) -> None:+ assert tracker.scopes, "no request was counted; the tracker is not wired"+ assert [(s.label, s.peak) for s in tracker.over()] == []+ assert [(what, held) for what, held in calls.held if held] == []+++class TestTheGuardItself:+ async def test_it_catches_a_request_that_takes_a_second_connection(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ pool_checkouts: PoolCheckouts,+ ) -> None:+ # The shape this whole file exists to forbid, written out on purpose:+ # read on the request's session, then open another before committing.+ # If this stops reporting 2, every other assertion here is vacuous.+ user = await _admin(session_factory)+ calls = _Calls(pool_checkouts)+ collab = _CollabLifecycle(calls)++ def held_and_wait(app: FastAPI) -> None:+ @app.get("/held-and-wait")+ async def held_and_wait(+ session: AsyncSession = Depends(gw_deps.get_session),+ _user: User = Depends(get_current_user),+ _is_admin: bool = Depends(get_tenant_is_admin),+ ) -> dict[str, bool]:+ await RoomStore().get(session, "nothing")+ async with session_factory() as other:+ await RoomStore().get(other, "nothing")+ return {"ok": True}++ app = _app(+ session_factory,+ pool_checkouts,+ room_service=_room_service(session_factory, calls, collab, {}),+ collab=collab,+ extra=held_and_wait,+ )+ async with _client(app, user) as client:+ assert (await client.get("/held-and-wait")).status_code == 200++ assert [(s.label, s.peak) for s in pool_checkouts.over()] == [+ ("GET /held-and-wait", 2)+ ]+++class TestRoomRoutes:+ async def test_room_writes_hold_one_connection_at_a_time(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ pool_checkouts: PoolCheckouts,+ ) -> None:+ user = await _admin(session_factory)+ async with session_factory() as session:+ room = await _room(session, user, "main")+ doomed = await _room(session, user, "doomed")+ bulk = await _room(session, user, "bulk")+ agent, running = await _agent(session, user, "helper")+ await session.commit()+ calls = _Calls(pool_checkouts)+ collab = _CollabLifecycle(calls)+ app = _app(+ session_factory,+ pool_checkouts,+ room_service=_room_service(+ session_factory, calls, collab, {agent.id: running}+ ),+ collab=collab,+ )++ async with _client(app, user) as client:+ responses = [+ await client.patch(f"/rooms/{room.id}", json={"name": "renamed"}),+ await client.put(+ f"/rooms/{room.id}/protection", json={"protection_config": {}}+ ),+ await client.put(+ f"/rooms/{room.id}/observe", json={"observe_config": {}}+ ),+ await client.post(+ f"/rooms/{room.id}/agents", json={"agent_ids": [agent.id]}+ ),+ await client.patch(+ f"/rooms/{room.id}/agents/{agent.id}",+ json={"receives_join_events": True},+ ),+ await client.delete(f"/rooms/{room.id}/agents/{agent.id}"),+ await client.post(f"/rooms/{room.id}/archive"),+ await client.post(f"/rooms/{room.id}/unarchive"),+ await client.get(f"/rooms/{room.id}"),+ await client.post(+ "/rooms/bulk-archive",+ json={"room_ids": [bulk.id], "archived": True},+ ),+ await client.delete(f"/rooms/{doomed.id}"),+ await client.post("/rooms/bulk-delete", json={"room_ids": [bulk.id]}),+ ]++ assert [r.status_code for r in responses] == [200] * len(responses), [+ r.text for r in responses if r.status_code != 200+ ]+ # The Matrix side was really reached, so "held nothing" means something.+ assert {what for what, _ in calls.held} >= {+ "matrix invite",+ "matrix kick",+ "matrix delete",+ }+ _assert_one_at_a_time(pool_checkouts, calls)+++class TestAgentRoutes:+ async def test_deleting_an_agent_holds_one_connection_at_a_time(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ pool_checkouts: PoolCheckouts,+ ) -> None:+ user = await _admin(session_factory)+ async with session_factory() as session:+ by_id, _ = await _agent(session, user, "by-id")+ await _agent(session, user, "by-name")+ await session.commit()+ calls = _Calls(pool_checkouts)+ collab = _CollabLifecycle(calls)+ app = _app(+ session_factory,+ pool_checkouts,+ room_service=_room_service(session_factory, calls, collab, {}),+ collab=collab,+ )++ async with _client(app, user) as client:+ responses = [+ await client.delete(f"/agents/{by_id.id}"),+ await client.delete("/agents/by-name/by-name"),+ ]++ assert [r.status_code for r in responses] == [200, 200]+ assert [what for what, _ in calls.held] == ["bridge identity removal"] * 2+ _assert_one_at_a_time(pool_checkouts, calls)+++class TestBridgeRoutes:+ async def _bridge(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> CollaborationBridge:+ async with session_factory() as session:+ client = Client(+ transport_user_id="@bridge:test", display_name="bridge", type="bridge"+ )+ session.add(client)+ await session.flush()+ bridge = CollaborationBridge(+ type="slack",+ display_name="Slack",+ connection_config={"bot_token": "xoxb-old"},+ client_id=client.id,+ status="running",+ )+ session.add(bridge)+ await session.commit()+ return bridge++ async def test_platform_calls_are_made_holding_no_connection(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ pool_checkouts: PoolCheckouts,+ ) -> None:+ user = await _admin(session_factory)+ bridge = await self._bridge(session_factory)+ calls = _Calls(pool_checkouts)+ collab = _CollabLifecycle(calls)+ app = _app(+ session_factory,+ pool_checkouts,+ room_service=_room_service(session_factory, calls, collab, {}),+ collab=collab,+ )++ async with _client(app, user) as client:+ edited = await client.patch(+ f"/bridges/{bridge.id}",+ json={+ "agent_greetings_enabled": False,+ "connection_config": {"bot_token": "xoxb-new"},+ },+ )+ searched = await client.get(+ f"/bridges/{bridge.id}/directory", params={"query": "ada"}+ )+ claimed = await client.post(+ f"/bridges/{bridge.id}/identities",+ json={"external_user_id": "U1", "username": "ada"},+ )+ deleted = await client.delete(f"/bridges/{bridge.id}")++ assert edited.status_code == 200, edited.text+ assert searched.status_code == 200, searched.text+ # The fake refuses to provision, so the claim stops at 409 — after the+ # directory search and the provisioning attempt, which is what counts.+ assert claimed.status_code == 409, claimed.text+ assert deleted.status_code == 200, deleted.text+ assert [what for what, _ in calls.held] == [+ "platform credential check",+ "start guards",+ "bridge restart",+ "platform directory search",+ "platform directory search",+ "provision external user",+ "bridge remove",+ ]+ _assert_one_at_a_time(pool_checkouts, calls)++ async def test_a_refused_credential_edit_writes_nothing(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ pool_checkouts: PoolCheckouts,+ ) -> None:+ # The credential check now runs after the read's transaction ends, so+ # the writes that used to precede it inside that transaction moved after+ # it. A refusal must still leave every field as it was.+ from switch_core.bridges.collaboration.models import BridgeCredentialError++ user = await _admin(session_factory)+ bridge = await self._bridge(session_factory)+ calls = _Calls(pool_checkouts)+ collab = _CollabLifecycle(calls)++ async def refuse(**_: Any) -> None:+ raise BridgeCredentialError("invalid_auth")++ collab.check_edited_connection_config = refuse # type: ignore[method-assign]+ app = _app(+ session_factory,+ pool_checkouts,+ room_service=_room_service(session_factory, calls, collab, {}),+ collab=collab,+ )++ async with _client(app, user) as client:+ response = await client.patch(+ f"/bridges/{bridge.id}",+ json={+ "agent_greetings_enabled": False,+ "connection_config": {"bot_token": "xoxb-bad"},+ },+ )++ assert response.status_code == 400+ async with session_factory() as session:+ stored = await CollaborationBridgeStore().get(session, bridge.id)+ assert stored is not None+ assert stored.agent_greetings_enabled is bridge.agent_greetings_enabled+ assert stored.connection_config == {"bot_token": "xoxb-old"}+++class TestPasswordRoutes:+ @pytest.fixture+ def bcrypt_calls(+ self, monkeypatch: pytest.MonkeyPatch, pool_checkouts: PoolCheckouts+ ) -> list[tuple[str, bool, int]]:+ """Each bcrypt call: which, whether it ran on the event loop's thread,+ and the connections the request held at the time."""+ seen: list[tuple[str, bool, int]] = []+ real_hash = gateway_auth.hash_password+ real_verify = gateway_auth.verify_password++ def on_loop() -> bool:+ return threading.current_thread() is threading.main_thread()++ def hash_password(password: str) -> str:+ seen.append(("hash", on_loop(), pool_checkouts.held_now()))+ return real_hash(password)++ def verify_password(password: str, password_hash: str | None) -> bool:+ seen.append(("verify", on_loop(), pool_checkouts.held_now()))+ return real_verify(password, password_hash)++ monkeypatch.setattr(gateway_auth, "hash_password", hash_password)+ monkeypatch.setattr(gateway_auth, "verify_password", verify_password)+ return seen++ async def test_bcrypt_runs_off_the_loop_holding_no_connection(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ pool_checkouts: PoolCheckouts,+ bcrypt_calls: list[tuple[str, bool, int]],+ ) -> None:+ user = await _admin(session_factory)+ async with session_factory() as session:+ operator = User(name="op", email="[email protected]", role="admin")+ session.add(operator)+ await session.flush()+ session.add(+ TenantMember(+ tenant_id=TENANT_ZERO_ID, user_id=operator.id, role="member"+ )+ )+ await session.commit()+ calls = _Calls(pool_checkouts)+ collab = _CollabLifecycle(calls)+ app = _app(+ session_factory,+ pool_checkouts,+ room_service=_room_service(session_factory, calls, collab, {}),+ collab=collab,+ )++ async with _client(app, user) as client:+ login = await client.post(+ "/auth/login", json={"email": user.email, "password": _PASSWORD}+ )+ wrong = await client.post(+ "/auth/login", json={"email": user.email, "password": "nope-nope"}+ )+ changed = await client.put(+ "/auth/me/password",+ json={"current_password": _PASSWORD, "new_password": "a new one!"},+ )+ signed_up = await client.post(+ "/auth/signup",+ json={"email": "[email protected]", "password": "a password!"},+ )+ async with _client(app, operator) as client:+ created = await client.post(+ "/users",+ json={+ "name": "made",+ "email": "[email protected]",+ "password": "a password!",+ "role": "user",+ },+ )++ assert login.status_code == 200, login.text+ assert wrong.status_code == 401, wrong.text+ assert changed.status_code == 200, changed.text+ assert signed_up.status_code == 201, signed_up.text+ assert created.status_code == 200, created.text+ assert bcrypt_calls == [+ ("verify", False, 0),+ ("verify", False, 0),+ ("verify", False, 0),+ ("hash", False, 0),+ ("hash", False, 0),+ ("hash", False, 0),+ ]+ _assert_one_at_a_time(pool_checkouts, calls)diff --git a/core/tests/switch_core/management/harness.py b/core/tests/switch_core/management/harness.pyindex c1b5c2aee..f39bc50c7 100644--- a/core/tests/switch_core/management/harness.py+++ b/core/tests/switch_core/management/harness.py@@ -40,6 +40,10 @@ from switch_core.bridges.agent.auth import BearerAuthMiddleware, ControllerPrincfrom switch_core.bridges.agent.controller_auth_cache import ControllerAuthCachefrom switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistryfrom switch_core.bridges.agent.protocol.agent_core import AgentCore+from switch_core.bridges.agent.protocol.controller_presence import (+ ControllerConnectionError,+)+from switch_core.bridges.agent.protocol.controller_stream import IDLEfrom switch_core.bridges.agent.protocol.event_buffer import EventBufferfrom switch_core.db.models import (TENANT_ZERO_ID,@@ -59,6 +63,7 @@ from switch_core.gateway import dependencies as gw_depsfrom switch_core.gateway.auth import create_jwtfrom switch_core.keys import Keyringfrom switch_core.management import controller_routes+from switch_core.management.errors import ManagementErrorfrom switch_core.management.wiring import Management, build_managementTEST_KEYRING = Keyring.parse("test:" + "x" * 40, legacy_secret=None)@@ -432,26 +437,20 @@ async def add_room(return room.id-def parse_frame(raw: bytes) -> tuple[str, dict[str, Any]]:- event, data = "", "{}"- for line in raw.decode().strip().splitlines():- if line.startswith("event: "):- event = line[len("event: ") :]- elif line.startswith("data: "):- data = line[len("data: ") :]- return event, json.loads(data)+def parse_frame(frame: dict[str, Any]) -> tuple[str, dict[str, Any]]:+ return frame["event"], frame["data"]async def take(- stream: AsyncIterator[bytes], count: int, timeout: float = 3.0+ stream: AsyncIterator[dict[str, Any]], count: int, timeout: float = 3.0) -> list[tuple[str, dict[str, Any]]]:- """The next `count` frames off a stream, keepalives skipped."""+ """The next `count` frames off a stream, idle ticks skipped."""frames: list[tuple[str, dict[str, Any]]] = []async def pump() -> None:while len(frames) < count:raw = await anext(stream)- if raw.startswith(b":"):+ if raw is IDLE:continueframes.append(parse_frame(raw))@@ -459,6 +458,30 @@ async def take(return frames+@dataclass+class Answer:+ """What the socket would have said, shaped like an HTTP response: a beat+ or an attach either succeeds or is refused with the contract's envelope."""++ status_code: int+ body: dict[str, Any]++ @property+ def text(self) -> str:+ return json.dumps(self.body)++ def json(self) -> dict[str, Any]:+ return self.body+++def _principal(controller: EnrolledController) -> ControllerPrincipal:+ return ControllerPrincipal(+ controller_id=controller.controller_id,+ owner_id=controller.owner.id,+ tenant_id=TENANT_ZERO_ID,+ )++async def open_connection(client: httpx.AsyncClient,controller: EnrolledController,@@ -477,22 +500,57 @@ async def open_stream(harness: Harness,controller: EnrolledController,opened: dict[str, Any],-) -> AsyncIterator[bytes]:- """The controller's event stream, from the route itself.-- Called directly rather than over HTTP: the in-process transport collects a- response whole, and this one never ends by itself.- """- response = await controller_routes.controller_events(- principal=ControllerPrincipal(- controller_id=controller.controller_id,- owner_id=controller.owner.id,- tenant_id=TENANT_ZERO_ID,- ),+) -> AsyncIterator[dict[str, Any]]:+ """The frames the controller's socket would send, from the route's own+ stream: the socket adds only pings around them."""+ return await controller_routes.attach_stream(+ principal=_principal(controller),management=harness.management.service,protocol=harness.protocol,session_factory=harness.session_factory,connection_id=opened["connection_id"],generation=opened["generation"],)- return response.body_iterator # type: ignore[return-value]+++async def attach(+ harness: Harness,+ controller: EnrolledController,+ connection_id: str,+ generation: int,+) -> Answer:+ """An attach, answered as the socket's `refused` frame would be."""+ try:+ stream = await open_stream(+ harness,+ controller,+ {"connection_id": connection_id, "generation": generation},+ )+ except ManagementError as error:+ return Answer(error.status_code, error.body())+ await stream.aclose() # type: ignore[attr-defined]+ return Answer(200, {})+++def beat(+ harness: Harness,+ controller: EnrolledController,+ *,+ connection_id: str,+ generation: int,+ cursors: dict[str, int],+) -> Answer:+ """A pong, answered as the socket would: the agents on success, the+ refusal its `evicted` frame would name otherwise."""+ try:+ agents = controller_routes.record_beat(+ principal=_principal(controller),+ protocol=harness.protocol,+ connection_id=connection_id,+ generation=generation,+ cursors=cursors,+ )+ except ControllerConnectionError as exc:+ error = controller_routes._connection_refusal(exc)+ return Answer(error.status_code, error.body())+ return Answer(200, {"agents": agents})diff --git a/core/tests/switch_core/management/test_connection_socket.py b/core/tests/switch_core/management/test_connection_socket.pynew file mode 100644index 000000000..6ab15a04f--- /dev/null+++ b/core/tests/switch_core/management/test_connection_socket.py@@ -0,0 +1,192 @@+"""The controller's connection over one WebSocket: the stream's frames down,+pongs up as beats.++What the stream carries, and what a beat does, are tested against Postgres in+`test_stream.py` through the same `attach_stream` and `record_beat` the socket+calls. These cover what is particular to the socket: framing, the ping the+server drives, and how refusals and evictions are told.+"""++from __future__ import annotations++import asyncio+from collections.abc import AsyncGenerator, Iterator+from typing import Any++import pytest+from fastapi import FastAPI+from fastapi.testclient import TestClient+from starlette.types import ASGIApp, Receive, Scope, Send+from starlette.websockets import WebSocketDisconnect++from switch_core.bridges.agent.auth import ControllerPrincipal+from switch_core.bridges.agent.dependencies import get_protocol+from switch_core.bridges.agent.protocol.controller_presence import (+ ControllerTakenOverError,+ UnknownControllerConnectionError,+)+from switch_core.management import controller_routes+from switch_core.management.dependencies import (+ get_management,+ get_management_session_factory,+)++CONTROLLER = "controller-1"+PATH = f"/v1/controllers/{CONTROLLER}/connection/ws?connection_id=c1&generation=7"+++class _AuthenticatedAs:+ """What the bearer middleware leaves in scope for a controller token."""++ def __init__(self, app: ASGIApp) -> None:+ self.app = app++ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:+ scope["controller"] = ControllerPrincipal(+ controller_id=CONTROLLER, owner_id="owner-1", tenant_id="tenant-1"+ )+ await self.app(scope, receive, send)+++class _Calls:+ def __init__(self) -> None:+ self.frames: list[dict[str, Any]] = [+ {"event": "connection_state", "data": {"connection_id": "c1"}},+ {"event": "agent.attached", "data": {"agent_id": "a1", "from_seq": 4}},+ ]+ self.attach_refusal: Exception | None = None+ self.beat_refusal: Exception | None = None+ self.beats: list[dict[str, Any]] = []+++def calls() -> _Calls:+ return _Calls()+++def client(calls: _Calls, monkeypatch: pytest.MonkeyPatch) -> Iterator[TestClient]:+ # Pings every 50 ms rather than every heartbeat interval.+ monkeypatch.setattr(controller_routes, "HEARTBEAT_INTERVAL_SECONDS", 0.05)++ async def attach_stream(**kwargs: Any) -> AsyncGenerator[dict[str, Any]]:+ if calls.attach_refusal is not None:+ raise calls.attach_refusal++ async def frames() -> AsyncGenerator[dict[str, Any]]:+ for frame in calls.frames:+ yield frame+ while True: # the stream stays open until the socket goes+ yield controller_routes.IDLE+ await _sleep()++ return frames()++ def record_beat(**kwargs: Any) -> list[str]:+ calls.beats.append(kwargs)+ if calls.beat_refusal is not None:+ raise calls.beat_refusal+ return ["a1"]++ monkeypatch.setattr(controller_routes, "attach_stream", attach_stream)+ monkeypatch.setattr(controller_routes, "record_beat", record_beat)+ app = FastAPI()+ app.include_router(controller_routes.router)+ app.dependency_overrides[get_protocol] = lambda: None+ app.dependency_overrides[get_management] = lambda: None+ app.dependency_overrides[get_management_session_factory] = lambda: None+ with TestClient(_AuthenticatedAs(app)) as test_client:+ yield test_client+++async def _sleep() -> None:+ await asyncio.sleep(0.01)+++def _until(socket: Any, event: str) -> dict[str, Any]:+ for _ in range(50):+ frame: dict[str, Any] = socket.receive_json()+ if frame["event"] == event:+ return frame+ raise AssertionError(f"no {event} frame")+++def test_the_stream_comes_down_as_json_and_the_server_pings(client: TestClient) -> None:+ with client.websocket_connect(PATH) as socket:+ first = socket.receive_json()+ second = socket.receive_json()+ ping = _until(socket, "ping")++ assert first == {"event": "connection_state", "data": {"connection_id": "c1"}}+ assert second["event"] == "agent.attached"+ assert ping == {"event": "ping", "data": {}}+++def test_a_pong_is_the_beat_with_every_agents_cursor(+ client: TestClient, calls: _Calls+) -> None:+ with client.websocket_connect(PATH) as socket:+ _until(socket, "ping")+ socket.send_json({"type": "pong", "cursors": {"a1": 9, "a2": -1, "a3": "x"}})+ _until(socket, "ping")++ assert calls.beats[0]["connection_id"] == "c1"+ assert calls.beats[0]["generation"] == 7+ # A cursor that is not a non-negative integer is left out, not trusted.+ assert calls.beats[0]["cursors"] == {"a1": 9}+++def test_a_refused_beat_is_told_as_evicted_then_the_socket_closes(+ client: TestClient, calls: _Calls+) -> None:+ calls.beat_refusal = ControllerTakenOverError("c1")+ with client.websocket_connect(PATH) as socket:+ _until(socket, "ping")+ socket.send_json({"type": "pong", "cursors": {}})+ evicted = _until(socket, "evicted")+ with pytest.raises(WebSocketDisconnect) as closed:+ socket.receive_json()++ assert evicted["data"]["code"] == "taken_over"+ assert closed.value.code == 1000+++def test_a_refused_attach_is_a_refused_frame_then_4000_plus_the_status(+ client: TestClient, calls: _Calls+) -> None:+ calls.attach_refusal = controller_routes._connection_refusal(+ UnknownControllerConnectionError("c1")+ )+ with client.websocket_connect(PATH) as socket:+ refused = socket.receive_json()+ with pytest.raises(WebSocketDisconnect) as closed:+ socket.receive_json()++ assert refused["event"] == "refused"+ assert refused["data"]["status"] == 404+ assert refused["data"]["detail"]["code"] == "unknown_connection"+ assert closed.value.code == 4404+++def test_a_controller_speaking_protocol_1_is_refused(client: TestClient) -> None:+ with client.websocket_connect(+ PATH, headers={"Switch-Controller-Protocol": "1"}+ ) as socket:+ refused = socket.receive_json()+ with pytest.raises(WebSocketDisconnect) as closed:+ socket.receive_json()++ assert refused["data"]["detail"]["code"] == "protocol_unsupported"+ assert closed.value.code == 4426+++def test_another_controllers_socket_is_forbidden(client: TestClient) -> None:+ with client.websocket_connect(+ "/v1/controllers/someone-else/connection/ws?connection_id=c1&generation=7"+ ) as socket:+ refused = socket.receive_json()+ with pytest.raises(WebSocketDisconnect) as closed:+ socket.receive_json()++ assert refused["data"]["detail"]["code"] == "forbidden"+ assert closed.value.code == 4403diff --git a/core/tests/switch_core/management/test_contract_fixtures.py b/core/tests/switch_core/management/test_contract_fixtures.pyindex 550214b47..fc78a651f 100644--- a/core/tests/switch_core/management/test_contract_fixtures.py+++ b/core/tests/switch_core/management/test_contract_fixtures.py@@ -25,8 +25,8 @@ from switch_core.bridges.agent.commands import room_control_framefrom switch_core.bridges.agent.protocol.event_buffer import Readerfrom switch_core.bridges.agent.protocol.types import AgentEvent, MessagePayloadfrom switch_core.db.stores.agent_store import AgentStore+from switch_core.management import controller_routesfrom switch_core.management.schemas import (- ControllerBeatRequest,ControllerConnectionRequest,EnrollRequest,OperationResultRequest,@@ -38,6 +38,7 @@ from tests.switch_core.management.harness import (Harness,add_member,add_room,+ beat,build_harness,cookies_for,create_managed_agent,@@ -96,7 +97,6 @@ class TestRequestFixturesParse:ControllerConnectionRequest.model_validate(_fixture("controller_connection_request.json"))- ControllerBeatRequest.model_validate(_fixture("controller_beat_request.json"))class TestEnrollmentAndTokens:@@ -259,16 +259,15 @@ class TestTheControllerConnection:)stream = await open_stream(harness, controller, opened.json())await take(stream, 2)- beat_body = _fixture("controller_beat_request.json")- beat_body.update(+ pong = _fixture("controller_pong.json")+ beaten = beat(+ harness,+ controller,connection_id=opened.json()["connection_id"],generation=opened.json()["generation"],- cursors={agent_id: 0},- )- beat = await client.post(- f"/v1/controllers/{controller.controller_id}/connection/beat",- json=beat_body,- headers=controller.headers,+ cursors=controller_routes._cursors_of(+ {**pong, "cursors": {agent_id: 0}}+ ),)present = harness.protocol.connections.controllers.live_rooms(agent_id)await stream.aclose()@@ -277,14 +276,11 @@ class TestTheControllerConnection:assert_same_shape(opened.json(), _fixture("controller_connection_response.json"))- assert beat.status_code == 200, beat.text- assert_same_shape(beat.json(), _fixture("controller_beat_response.json"))+ assert beaten.status_code == 200, beaten.textassert harness.protocol.connections.controllers.live_rooms(agent_id) == set()assert present == {room_id}- async def test_placements_are_refused_on_open_and_beat(- self, harness: Harness- ) -> None:+ async def test_placements_are_refused_on_open(self, harness: Harness) -> None:owner = await add_member(harness.session_factory, "ada")async with harness.client() as client:controller = await enroll_console(harness, client, owner)@@ -292,36 +288,15 @@ class TestTheControllerConnection:room_id = await add_room(harness.session_factory, agent_id)body = _fixture("controller_connection_request.json")body["cursors"] = {agent_id: 0}- refused_open = await client.post(+ refused = await client.post(f"/v1/controllers/{controller.controller_id}/connection",json={**body, "placements": {agent_id: [room_id]}},headers=controller.headers,)- opened = await client.post(- f"/v1/controllers/{controller.controller_id}/connection",- json=body,- headers=controller.headers,- )- stream = await open_stream(harness, controller, opened.json())- await take(stream, 2)- beat_body = _fixture("controller_beat_request.json")- beat_body.update(- connection_id=opened.json()["connection_id"],- generation=opened.json()["generation"],- cursors={agent_id: 0},- placements={agent_id: [room_id]},- )- refused_beat = await client.post(- f"/v1/controllers/{controller.controller_id}/connection/beat",- json=beat_body,- headers=controller.headers,- )- await stream.aclose()- for refused in (refused_open, refused_beat):- assert refused.status_code == 422, refused.text- assert refused.json()["error"]["code"] == "validation_error"- assert "placements" in refused.json()["error"]["message"]+ assert refused.status_code == 422, refused.text+ assert refused.json()["error"]["code"] == "validation_error"+ assert "placements" in refused.json()["error"]["message"]class _Outcomes:diff --git a/core/tests/switch_core/management/test_controller_routes.py b/core/tests/switch_core/management/test_controller_routes.pyindex 74162feca..4248b302f 100644--- a/core/tests/switch_core/management/test_controller_routes.py+++ b/core/tests/switch_core/management/test_controller_routes.py@@ -440,18 +440,18 @@ class TestStatus:response = await client.put(f"/v1/management/controllers/{controller.controller_id}/status",json=fixture("status_request.json"),- headers={**controller.headers, "Switch-Controller-Protocol": "2"},+ headers={**controller.headers, "Switch-Controller-Protocol": "1"},)accepted = await client.put(f"/v1/management/controllers/{controller.controller_id}/status",json=fixture("status_request.json"),- headers={**controller.headers, "Switch-Controller-Protocol": "1"},+ headers={**controller.headers, "Switch-Controller-Protocol": "2"},)assert response.status_code == 426assert response.json()["error"]["code"] == "protocol_unsupported"- assert response.headers["Switch-Controller-Protocol-Accepts"] == "1-1"+ assert response.headers["Switch-Controller-Protocol-Accepts"] == "2-2"assert accepted.status_code == 200- assert accepted.headers["Switch-Controller-Protocol-Accepts"] == "1-1"+ assert accepted.headers["Switch-Controller-Protocol-Accepts"] == "2-2"async def _placed_agent(diff --git a/core/tests/switch_core/management/test_middleware.py b/core/tests/switch_core/management/test_middleware.pyindex 47729dac1..e3a68ec49 100644--- a/core/tests/switch_core/management/test_middleware.py+++ b/core/tests/switch_core/management/test_middleware.py@@ -245,7 +245,6 @@ class TestRefusals:"POST",f"/v1/management/controllers/{second.controller_id}/credential/rotate",),- ("GET", f"/v1/controllers/{second.controller_id}/events"),]:response = await client.request(method, path, headers=first.headers)assert response.status_code == 403, (method, path, response.text)diff --git a/core/tests/switch_core/management/test_stream.py b/core/tests/switch_core/management/test_stream.pyindex ee6f90b98..2f5adb9cf 100644--- a/core/tests/switch_core/management/test_stream.py+++ b/core/tests/switch_core/management/test_stream.py@@ -39,6 +39,8 @@ from tests.switch_core.management.harness import (Harness,add_member,add_room,+ attach,+ beat,build_harness,cookies_for,enroll_console,@@ -415,50 +417,36 @@ class TestTheConnection:evicted = await take(old_stream, 1)with pytest.raises(StopAsyncIteration):await anext(old_stream)- beat_path = f"/v1/controllers/{controller.controller_id}/connection/beat"- stale = await client.post(- beat_path,- json={- "connection_id": old["connection_id"],- "generation": old["generation"],- "cursors": {},- },- headers=controller.headers,+ stale = beat(+ harness,+ controller,+ connection_id=old["connection_id"],+ generation=old["generation"],+ cursors={},)- wrong_generation = await client.post(- beat_path,- json={- "connection_id": new["connection_id"],- "generation": new["generation"] + 1,- "cursors": {},- },- headers=controller.headers,+ wrong_generation = beat(+ harness,+ controller,+ connection_id=new["connection_id"],+ generation=new["generation"] + 1,+ cursors={},)- unknown = await client.post(- beat_path,- json={- "connection_id": "nope",- "generation": 1,- "cursors": {},- },- headers=controller.headers,+ unknown = beat(+ harness,+ controller,+ connection_id="nope",+ generation=1,+ cursors={},)- no_stream = await client.post(- beat_path,- json={- "connection_id": new["connection_id"],- "generation": new["generation"],- "cursors": {},- },- headers=controller.headers,+ no_stream = beat(+ harness,+ controller,+ connection_id=new["connection_id"],+ generation=new["generation"],+ cursors={},)- old_events = await client.get(- f"/v1/controllers/{controller.controller_id}/events",- params={- "connection_id": old["connection_id"],- "generation": old["generation"],- },- headers=controller.headers,+ old_events = await attach(+ harness, controller, old["connection_id"], old["generation"])assert new["generation"] != old["generation"]@@ -500,20 +488,18 @@ class TestTheConnection:assert not presence.is_live(agent_id)stream = await open_stream(harness, controller, opened)await take(stream, 2)- beat = await client.post(- f"/v1/controllers/{controller.controller_id}/connection/beat",- json={- "connection_id": opened["connection_id"],- "generation": opened["generation"],- "cursors": {agent_id: 99, "not-mine": 3},- },- headers=controller.headers,+ beaten = beat(+ harness,+ controller,+ connection_id=opened["connection_id"],+ generation=opened["generation"],+ cursors={agent_id: 99, "not-mine": 3},)live = presence.is_live(agent_id)await stream.aclose()- assert beat.status_code == 200, beat.text- assert beat.json() == {"agents": [agent_id]}+ assert beaten.status_code == 200, beaten.text+ assert beaten.json() == {"agents": [agent_id]}assert livebinding = presence.binding(agent_id)assert binding is not None@@ -541,16 +527,14 @@ class TestTheConnection:first = await open_stream(harness, controller, opened)# state, two attaches, three events for `confirmed`await take(first, 6)- beat = await client.post(- f"/v1/controllers/{controller.controller_id}/connection/beat",- json={- "connection_id": opened["connection_id"],- "generation": opened["generation"],- "cursors": {confirmed: 2},- },- headers=controller.headers,+ beaten = beat(+ harness,+ controller,+ connection_id=opened["connection_id"],+ generation=opened["generation"],+ cursors={confirmed: 2},)- assert beat.status_code == 200, beat.text+ assert beaten.status_code == 200, beaten.text# The socket drops; events keep arriving; the same connection is# streamed again.buffer.enqueue(@@ -649,7 +633,7 @@ class TestTheConnection:async def test_a_lapsed_beat_ends_the_stream_and_its_agents_are_not_live(self, harness: Harness, monkeypatch: pytest.MonkeyPatch) -> None:- monkeypatch.setattr(controller_routes, "KEEPALIVE_INTERVAL_SECONDS", 0.05)+ monkeypatch.setattr(controller_routes, "IDLE_INTERVAL_SECONDS", 0.05)owner = await add_member(harness.session_factory, "ada")presence = harness.protocol.connections.controllersasync with harness.client() as client:@@ -663,14 +647,12 @@ class TestTheConnection:conn.last_beat = time.monotonic() - HEARTBEAT_TTL_SECONDS - 1assert not presence.is_live(agent_id)evicted = await take(stream, 1)- beat = await client.post(- f"/v1/controllers/{controller.controller_id}/connection/beat",- json={- "connection_id": opened["connection_id"],- "generation": opened["generation"],- "cursors": {},- },- headers=controller.headers,+ beaten = beat(+ harness,+ controller,+ connection_id=opened["connection_id"],+ generation=opened["generation"],+ cursors={},)assert evicted == [@@ -683,7 +665,7 @@ class TestTheConnection:},)]- assert (beat.status_code, beat.json()["error"]["code"]) == (+ assert (beaten.status_code, beaten.json()["error"]["code"]) == (404,"unknown_connection",)@@ -734,4 +716,4 @@ async def test_a_waiting_stream_wakes_promptly_for_an_event(harness: Harness) ->)raw = await asyncio.wait_for(pending, timeout=2)await stream.aclose()- assert b"agent.event" in raw+ assert raw["event"] == "agent.event"diff --git a/core/tests/switch_core/observability/test_agent_protocol_metrics.py b/core/tests/switch_core/observability/test_agent_protocol_metrics.pyindex 23dcdad70..35b7a1a74 100644--- a/core/tests/switch_core/observability/test_agent_protocol_metrics.py+++ b/core/tests/switch_core/observability/test_agent_protocol_metrics.py@@ -133,3 +133,98 @@ def test_a_quiet_sweep_counts_nothing(registry):assert connections.sweep() == []assert _counts(registry, "switch.agent.connections_expired") == {}+++def _open(connections: AgentConnectionRegistry, connection_id: str = "c1") -> None:+ connections.open(+ agent_id="agent-1",+ connection_id=connection_id,+ scope="single", # type: ignore[arg-type]+ delivery_filter="all", # type: ignore[arg-type]+ spawn_capable=False,+ cursor=0,+ declaration=ClientDeclaration(speaks=PROTOCOL_VERSION),+ expected_generation=None,+ )+++def test_a_first_open_is_fresh(registry):+ _open(AgentConnectionRegistry())++ assert _counts(registry, "switch.agent.connections_opened") == {+ (("kind", "fresh"),): 1.0+ }+++def test_a_socket_coming_back_to_a_live_connection_is_a_reattach(registry):+ connections = AgentConnectionRegistry()+ _open(connections)+ _open(connections)++ assert _counts(registry, "switch.agent.connections_opened") == {+ (("kind", "fresh"),): 1.0,+ (("kind", "reattach"),): 1.0,+ }+++def test_a_return_after_a_lapse_is_told_apart(registry, monkeypatch):+ """The reconnect storm: agents lapse, then come straight back."""+ connections = AgentConnectionRegistry()+ _open(connections)+ later = time_module.monotonic() + conn_module.HEARTBEAT_TTL_SECONDS + 1+ monkeypatch.setattr(conn_module.time, "monotonic", lambda: later)+ assert connections.sweep()++ _open(connections)++ assert _counts(registry, "switch.agent.connections_opened") == {+ (("kind", "fresh"),): 1.0,+ (("kind", "after_lapse"),): 1.0,+ }+++def test_a_return_after_any_other_close_is_after_close(registry):+ connections = AgentConnectionRegistry()+ _open(connections)+ connections.close("c1", conn_module.TAKEN_OVER)+ _open(connections)++ assert (+ _counts(registry, "switch.agent.connections_opened")[(("kind", "after_close"),)]+ == 1.0+ )+++def test_the_memory_of_closed_connections_is_bounded(monkeypatch):+ monkeypatch.setattr(conn_module, "_RECENTLY_CLOSED_LIMIT", 2)+ connections = AgentConnectionRegistry()+ for index in range(4):+ _open(connections, f"c{index}")+ connections.close(f"c{index}", conn_module.HEARTBEAT_LAPSED)++ assert list(connections._recently_closed) == ["c2", "c3"]+++def test_a_lapse_noticed_on_next_use_is_counted_too(registry, monkeypatch):+ """The sweep is one of three ways a lapse is noticed. Counting only it+ under-reported lapses against the reconnects that follow them."""+ connections = AgentConnectionRegistry()+ _open(connections)+ later = time_module.monotonic() + conn_module.HEARTBEAT_TTL_SECONDS + 1+ monkeypatch.setattr(conn_module.time, "monotonic", lambda: later)++ with pytest.raises(conn_module.UnknownConnectionError):+ connections.require("agent-1", "c1")+ _open(connections)++ # One collection: reading the registry resets it.+ payloads = {p.name: p for p in registry.collect()}+ expired = sum(+ point.value for point in payloads["switch.agent.connections_expired"].numbers+ )+ opened = {+ point.attributes["kind"]: point.value+ for point in payloads["switch.agent.connections_opened"].numbers+ }+ assert expired == 1.0+ assert opened == {"fresh": 1.0, "after_lapse": 1.0}diff --git a/core/tests/switch_core/observability/test_bootstrap.py b/core/tests/switch_core/observability/test_bootstrap.pyindex bc677f899..02edafaf9 100644--- a/core/tests/switch_core/observability/test_bootstrap.py+++ b/core/tests/switch_core/observability/test_bootstrap.py@@ -7,11 +7,14 @@ import jsonfrom contextlib import asynccontextmanagerimport pytest+from sqlalchemy.ext.asyncio import create_async_engine+from sqlalchemy.pool import NullPoolfrom switch_core.config import SwitchConfigfrom switch_core.observability.bootstrap import RuntimeProbes, start_observabilityfrom switch_core.observability.metrics import metrics, uninstallfrom switch_core.observability.pool import PoolStats+from switch_core.transport.room_cache import RoomCacheStatsBASE_ENV = {"DB_HOST": "localhost",@@ -74,8 +77,12 @@ def _probes(**overrides) -> RuntimeProbes:consumers_running=lambda: 5,connectors_running=lambda: 2,connectors_configured=lambda: 2,- agents_connected=lambda: 3,+ agents_connected=lambda: {+ ("websocket", "agent-runtime"): 2,+ ("websocket", "unknown"): 1,+ },pool_stats=lambda: PoolStats(in_use=4, size=30, overflow=0),+ room_cache_stats=lambda: RoomCacheStats(bytes=0, rooms=0, rows=0),)return RuntimeProbes(**{**defaults, **overrides})@@ -163,7 +170,14 @@ async def test_an_endpoint_installs_the_registry_and_reports_state(monkeypatch):for name, payload in payloads.items()if payload.numbers and not payload.numbers[0].attributes}- assert values["switch.agents.connected"] == 3.0+ connected = {+ (point.attributes["transport"], point.attributes["client"]): point.value+ for point in payloads["switch.agents.connected"].numbers+ }+ assert connected == {+ ("websocket", "agent-runtime"): 2.0,+ ("websocket", "unknown"): 1.0,+ }assert values["switch.consumers.running"] == 5.0assert values["switch.connectors.running"] == 2.0assert values["switch.db.pool.in_use"] == 4.0@@ -181,6 +195,52 @@ async def test_an_endpoint_installs_the_registry_and_reports_state(monkeypatch):await observability.aclose()+async def test_the_room_cache_reports_what_it_holds(monkeypatch):+ config = _config(+ monkeypatch,+ OTLP_ENDPOINT="https://collector.example",+ DEPLOYMENT_ID=DEPLOYMENT_ID,+ OTLP_EXPORT_INTERVAL_SECONDS="3600",+ )+ observability = start_observability(+ config=config,+ version="1.0.0",+ session_factory=_session_factory(),+ probes=_probes(+ room_cache_stats=lambda: RoomCacheStats(bytes=4096, rooms=3, rows=12)+ ),+ )+ try:+ payloads = {p.name: p for p in metrics().collect()}+ assert payloads["switch.delivery_cache.bytes"].numbers[0].value == 4096.0+ assert payloads["switch.delivery_cache.rooms"].numbers[0].value == 3.0+ finally:+ await observability.aclose()+++async def test_a_start_is_counted_once_so_restarts_line_up(monkeypatch):+ observability = start_observability(+ config=_config(+ monkeypatch,+ OTLP_ENDPOINT="https://collector.example",+ DEPLOYMENT_ID=DEPLOYMENT_ID,+ OTLP_EXPORT_INTERVAL_SECONDS="3600",+ ),+ version="1.0.0",+ session_factory=_session_factory(),+ probes=_probes(),+ )+ try:+ starts = next(+ p for p in metrics().collect() if p.name == "switch.runtime.starts"+ )+ assert [point.value for point in starts.numbers] == [1.0]+ finally:+ await observability.aclose()++@pytest.mark.asyncioasync def test_a_pool_that_reports_nothing_produces_no_reading(monkeypatch):"""A zero would draw an idle pool; absence draws nothing, which is true."""@@ -262,3 +322,72 @@ async def test_closing_stops_the_loops_and_uninstalls(monkeypatch):assert metrics().enabled is Falseassert all(task.done() for task in observability._tasks)+++async def test_the_database_sampler_starts_with_the_metrics_exporter(monkeypatch):+ built: list[object] = []++ def engine():+ built.append(object())+ return create_async_engine(+ "postgresql+asyncpg://u:[email protected]:1/x", poolclass=NullPool+ )++ observability = start_observability(+ config=_config(+ monkeypatch,+ OTLP_ENDPOINT="https://collector.example",+ DEPLOYMENT_ID=DEPLOYMENT_ID,+ OTLP_EXPORT_INTERVAL_SECONDS="3600",+ ),+ version="1.0.0",+ session_factory=_session_factory(),+ probes=_probes(),+ db_server_engine=engine,+ )+ try:+ await asyncio.sleep(0.05)+ names = {task.get_name() for task in observability._tasks}+ assert "db-server-sampler" in names+ assert built, "the sampler never built its engine"+ finally:+ await observability.aclose()+++async def test_a_connection_kind_that_empties_out_is_reported_as_zero(monkeypatch):+ """Otherwise the dashboard keeps drawing its last count, which is exactly+ the line that has to reach zero before an old transport can be removed."""+ connected = {("websocket", "agent-runtime"): 2, ("detached", "agent-runtime"): 1}+ observability = start_observability(+ config=_config(+ monkeypatch,+ OTLP_ENDPOINT="https://collector.example",+ DEPLOYMENT_ID=DEPLOYMENT_ID,+ OTLP_EXPORT_INTERVAL_SECONDS="3600",+ ),+ version="1.0.0",+ session_factory=_session_factory(),+ probes=_probes(agents_connected=lambda: dict(connected)),+ )+ try:++ def reading() -> dict[tuple[str, str], float]:+ payload = {p.name: p for p in metrics().collect()}[+ "switch.agents.connected"+ ]+ return {+ (n.attributes["transport"], n.attributes["client"]): n.value+ for n in payload.numbers+ }++ assert reading()[("detached", "agent-runtime")] == 1.0+ del connected[("detached", "agent-runtime")]+ assert reading() == {+ ("websocket", "agent-runtime"): 2.0,+ ("detached", "agent-runtime"): 0.0,+ ("websocket", "unknown"): 0.0,+ }+ finally:+ await observability.aclose()diff --git a/core/tests/switch_core/observability/test_connection_metrics.py b/core/tests/switch_core/observability/test_connection_metrics.pynew file mode 100644index 000000000..a4dd62d50--- /dev/null+++ b/core/tests/switch_core/observability/test_connection_metrics.py@@ -0,0 +1,347 @@+"""How agents are connected, and why a connection was refused.++The rollout question is which clients are still connecting the old way, so the+connected gauge carries the transport and the client, and the refusals carry a+reason. Both labels come from fixed sets: a client's own string never reaches a+metric.+"""++from __future__ import annotations++from collections.abc import Iterator+from typing import Any++import pytest+from fastapi import FastAPI, HTTPException+from fastapi.testclient import TestClient+from starlette.types import ASGIApp, Receive, Scope, Send+from starlette.websockets import WebSocketDisconnect++from switch_core.bridges.agent.api import handlers+from switch_core.bridges.agent.api.session_reporter import SessionReporter+from switch_core.bridges.agent.api_key_cache import ApiKeyCache+from switch_core.bridges.agent.auth import BearerAuthMiddleware+from switch_core.bridges.agent.dependencies import get_config, get_protocol+from switch_core.bridges.agent.protocol.agent_connections import (+ CONTROLLER_LABEL,+ PROTOCOL_VERSION,+ AgentConnectionRegistry,+ ClientDeclaration,+ client_label,+)+from switch_core.bridges.agent.protocol.controller_presence import Binding+from switch_core.bridges.agent.protocol.event_buffer import EventBuffer+from switch_core.db.models import Agent+from switch_core.db.stores.agent_store import AgentStore+from switch_core.db.stores.api_key_store import ApiKeyStore+from switch_core.observability.bootstrap import RuntimeProbes, _state_readings+from switch_core.observability.metrics import MetricsRegistry, install, uninstall+from switch_core.transport.room_cache import RoomCacheStats++AGENT_ID = "agent-1"+++def registry() -> Iterator[MetricsRegistry]:+ registry = MetricsRegistry()+ install(registry)+ yield registry+ uninstall()+++def _counts(registry: MetricsRegistry, name: str) -> dict[tuple, float]:+ payload = next((p for p in registry.collect() if p.name == name), None)+ if payload is None:+ return {}+ return {+ tuple(sorted(point.attributes.items())): point.value+ for point in payload.numbers+ }+++def _open(+ connections: AgentConnectionRegistry,+ agent_id: str,+ connection_id: str,+ artifact: str | None,+ version: str | None = None,+) -> Any:+ return connections.open(+ agent_id=agent_id,+ connection_id=connection_id,+ scope="all",+ delivery_filter="all",+ spawn_capable=False,+ cursor=0,+ declaration=ClientDeclaration(+ speaks=PROTOCOL_VERSION, artifact=artifact, version=version+ ),+ expected_generation=None,+ )+++# ── Connected, by transport and client ───────────────────────────────────────+++ ("artifact", "label"),+ [+ ("agent-runtime", "agent-runtime"),+ ("switch-console", "switch-console"),+ (None, "unknown"),+ ("", "unknown"),+ ("my-homemade-runtime/9.9", "other"),+ ],+)+def test_the_client_label_is_a_registered_name_or_a_bucket(+ artifact: str | None, label: str+) -> None:+ assert client_label(ClientDeclaration(artifact=artifact)) == label+++def test_agents_are_counted_per_transport_and_client() -> None:+ connections = AgentConnectionRegistry()+ _open(connections, "a1", "c1", "agent-runtime", version="0.7.2")+ # One agent, two sockets from the same client: still one agent.+ _open(connections, "a1", "c2", "agent-runtime", version="0.7.3")+ _open(connections, "a2", "c3", None)+ _open(connections, "a3", "c4", "something-else")+ detached = _open(connections, "a4", "c5", "agent-runtime")+ connections.detach_stream(detached, detached.stream_generation)++ assert connections.live_agents_by_transport() == {+ ("websocket", "agent-runtime"): 1,+ ("websocket", "unknown"): 1,+ ("websocket", "other"): 1,+ ("detached", "agent-runtime"): 1,+ }+++def test_agents_run_by_a_live_controller_count_under_the_controller() -> None:+ connections = AgentConnectionRegistry()+ _open(connections, "a1", "c1", "agent-runtime")+ controllers = connections.controllers+ for agent_id in ("m1", "m2"):+ controllers.bind(+ Binding(+ agent_id=agent_id,+ controller_id="ctl",+ tenant_id="t",+ controller_name="the machine",+ running=True,+ )+ )+ conn = controllers.open(controller_id="ctl", tenant_id="t", resume_cursors={})+ controllers.attach_stream(conn)++ assert connections.live_agents_by_transport() == {+ ("websocket", "agent-runtime"): 1,+ CONTROLLER_LABEL: 2,+ }+++def test_an_agent_connected_two_ways_counts_under_each() -> None:+ connections = AgentConnectionRegistry()+ _open(connections, "a1", "c1", "agent-runtime")+ _open(connections, "a1", "c2", None)++ assert connections.live_agents_by_transport() == {+ ("websocket", "agent-runtime"): 1,+ ("websocket", "unknown"): 1,+ }+++def _probes(agents_connected: Any) -> RuntimeProbes:+ return RuntimeProbes(+ listener_connected=lambda: True,+ session_activity_listener_connected=lambda: True,+ bridges_running=lambda: 0,+ bridges_running_by_platform=lambda: {},+ bridges_configured=lambda: 0,+ consumers_running=lambda: 0,+ connectors_running=lambda: 0,+ connectors_configured=lambda: 0,+ agents_connected=agents_connected,+ pool_stats=lambda: None,+ room_cache_stats=lambda: RoomCacheStats(bytes=0, rooms=0, rows=0),+ )+++def _connected_readings(probes: RuntimeProbes) -> dict[tuple, float]:+ return {+ tuple(sorted(reading.attributes.items())): reading.value+ for reading in _state_readings(probes)()+ if reading.spec.name == "switch.agents.connected"+ }+++def test_the_gauge_carries_transport_and_client_but_never_the_version() -> None:+ connections = AgentConnectionRegistry()+ _open(connections, "a1", "c1", "agent-runtime", version="0.7.2")+ _open(connections, "a2", "c2", "agent-runtime", version="0.0.1-dirty")++ readings = _connected_readings(_probes(connections.live_agents_by_transport))++ # The zero baseline is always there; no series carries a version.+ assert readings == {+ (("client", "agent-runtime"), ("transport", "websocket")): 2.0,+ (("client", "unknown"), ("transport", "websocket")): 0.0,+ }+++def test_the_gauge_still_reports_zero_with_nobody_connected() -> None:+ """A panel over no series reads "no data", which is not the same as none."""+ readings = _connected_readings(_probes(lambda: {}))++ assert list(readings.values()) == [0.0]+++# ── Refused ──────────────────────────────────────────────────────────────────+++async def test_the_event_stream_is_served_so_never_refused_as_removed(+ registry,+) -> None:+ """Kept for old clients for a compatibility window, so `transport_removed`+ does not fire; a stream refused for another reason is counted for it."""+ with pytest.raises(HTTPException):+ await handlers.poll_events(+ agent_id=AGENT_ID,+ agent=Agent(id=AGENT_ID, name="agent"),+ protocol=None, # type: ignore[arg-type]+ config=None, # type: ignore[arg-type]+ accept="text/event-stream",+ )++ assert _counts(registry, "switch.agent.connections_refused") == {+ (("reason", "other"),): 1.0+ }+++def test_a_stream_attached_over_sse_is_counted_as_sse() -> None:+ connections = AgentConnectionRegistry()+ connections.open(+ agent_id="a1",+ connection_id="c1",+ scope="all",+ delivery_filter="all",+ spawn_capable=False,+ cursor=0,+ declaration=ClientDeclaration(speaks=6, accepts=1, artifact="agent-runtime"),+ expected_generation=None,+ transport="sse",+ )+ _open(connections, "a2", "c2", "agent-runtime")++ readings = _connected_readings(_probes(connections.live_agents_by_transport))++ assert readings[(("client", "agent-runtime"), ("transport", "sse"))] == 1.0+ assert readings[(("client", "agent-runtime"), ("transport", "websocket"))] == 1.0+++class _Protocol:+ def __init__(self) -> None:+ self.event_buffer = EventBuffer(sequence_base=0)+ self.connections = AgentConnectionRegistry()+ self.approval_outcomes = None+ self.sessions = SessionReporter(None)++ async def record_client_declaration(self, *args: Any) -> None:+ return None++ async def require_room_member(self, agent_id: str, room_id: str) -> None:+ return None+++class _AuthenticatedAs:+ def __init__(self, app: ASGIApp, agent: Agent) -> None:+ self.app = app+ self.agent = agent++ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:+ scope["agent"] = self.agent+ await self.app(scope, receive, send)+++def client() -> Iterator[TestClient]:+ app = FastAPI()+ app.include_router(handlers.router, prefix="/agents")+ protocol = _Protocol()+ app.dependency_overrides[get_protocol] = lambda: protocol+ app.dependency_overrides[get_config] = lambda: None+ agent = Agent(id=AGENT_ID, name="agent")+ with TestClient(_AuthenticatedAs(app, agent)) as client: # type: ignore[arg-type]+ yield client+++def _refused_on_socket(client: TestClient, query: str) -> int:+ with client.websocket_connect(f"/agents/{AGENT_ID}/connection/ws?{query}") as ws:+ assert ws.receive_json()["event"] == "refused"+ with pytest.raises(WebSocketDisconnect) as closed:+ ws.receive_json()+ return closed.value.code+++def test_a_protocol_mismatch_on_the_socket_is_counted_as_protocol(+ registry, client: TestClient+) -> None:+ code = _refused_on_socket(+ client, f"connection_id=c1&protocol={PROTOCOL_VERSION + 1}"+ )++ assert code == 4409+ assert _counts(registry, "switch.agent.connections_refused") == {+ (("reason", "protocol"),): 1.0+ }+++def test_any_other_refusal_on_the_socket_is_counted_as_other(+ registry, client: TestClient+) -> None:+ code = _refused_on_socket(client, "connection_id=c1&scope=everywhere")++ assert code == 4400+ assert _counts(registry, "switch.agent.connections_refused") == {+ (("reason", "other"),): 1.0+ }+++async def _unauthenticated_socket(path: str) -> None:+ async def app(scope: Any, receive: Any, send: Any) -> None:+ raise AssertionError("an unauthenticated socket reached the app")++ middleware = BearerAuthMiddleware(+ app,+ agent_store=AgentStore(),+ api_key_store=ApiKeyStore(),+ api_key_cache=ApiKeyCache(ttl_seconds=5, max_entries=8),+ session_factory=None, # type: ignore[arg-type]+ )+ inbox = [{"type": "websocket.connect"}]++ async def receive() -> dict:+ return inbox.pop(0)++ async def send(message: dict) -> None:+ return None++ await middleware(+ {"type": "websocket", "path": path, "headers": [], "query_string": b""},+ receive,+ send,+ )+++async def test_an_unauthenticated_connection_socket_is_counted(registry) -> None:+ await _unauthenticated_socket(f"/agents/{AGENT_ID}/connection/ws")++ assert _counts(registry, "switch.agent.connections_refused") == {+ (("reason", "unauthorized"),): 1.0+ }+++async def test_an_unauthenticated_socket_elsewhere_is_not(registry) -> None:+ await _unauthenticated_socket("/mcp")++ assert _counts(registry, "switch.agent.connections_refused") == {}diff --git a/core/tests/switch_core/observability/test_db_server.py b/core/tests/switch_core/observability/test_db_server.pynew file mode 100644index 000000000..b533c02a0--- /dev/null+++ b/core/tests/switch_core/observability/test_db_server.py@@ -0,0 +1,155 @@+"""The database's own view of this server, against a real Postgres.++The sampler exists for the moment the pool is exhausted, so the test that+matters most holds the application pool's only connection and samples anyway.+"""++from __future__ import annotations++from collections.abc import Iterator++import pytest+from sqlalchemy import text+from sqlalchemy.engine import make_url+from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine+from sqlalchemy.pool import NullPool++from switch_core.observability.db_server import DbServerSampler+from switch_core.observability.metrics import MetricsRegistry, install, uninstall+++def registry() -> Iterator[MetricsRegistry]:+ registry = MetricsRegistry()+ install(registry)+ yield registry+ uninstall()+++def _unpooled(url: str) -> AsyncEngine:+ return create_async_engine(url, poolclass=NullPool)+++def _restricted_url(rls_harness) -> str:+ return rls_harness.restricted_engine.url.render_as_string(hide_password=False)+++def _states(sampler: DbServerSampler) -> dict[str, float]:+ return {r.attributes["state"]: r.value for r in sampler.readings()} # type: ignore[misc]+++async def test_it_samples_while_the_app_pool_is_exhausted(postgres_url: str) -> None:+ app_engine = create_async_engine(postgres_url, pool_size=1, max_overflow=0)+ sampler = DbServerSampler(lambda: _unpooled(postgres_url))+ try:+ async with app_engine.connect() as held:+ # The pool's only connection, checked out and left in a transaction.+ await held.execute(text("SELECT 1"))++ sample = await sampler.sample_once()++ assert app_engine.pool.checkedout() == 1 # type: ignore[attr-defined]+ assert sample is not None+ assert sum(sample.connections.values()) >= 1+ assert sample.connections["idle_in_transaction"] >= 1+ assert sample.transactions >= 0+ states = _states(sampler)+ assert set(states) == {"active", "idle_in_transaction", "idle", "other"}+ assert states["idle_in_transaction"] >= 1+ finally:+ await sampler.aclose()+ await app_engine.dispose()+++async def test_it_does_not_count_its_own_session(rls_harness) -> None:+ sampler = DbServerSampler(lambda: _unpooled(_restricted_url(rls_harness)))+ try:+ sample = await sampler.sample_once()+ finally:+ await sampler.aclose()++ # A role with no session but the sampler's own.+ assert sample is not None+ assert sum(sample.connections.values()) == 0+++async def test_transactions_are_recorded_as_the_rise_between_samples(+ postgres_url: str, registry: MetricsRegistry+) -> None:+ sampler = DbServerSampler(lambda: _unpooled(postgres_url))+ other = _unpooled(postgres_url)+ try:+ await sampler.sample_once()+ # The first sample is only a baseline.+ assert not [+ p for p in registry.collect() if p.name == "switch.db.server.transactions"+ ]++ async with other.begin() as conn:+ await conn.execute(text("SELECT 1"))+ await sampler.sample_once()++ payload = next(+ p for p in registry.collect() if p.name == "switch.db.server.transactions"+ )+ assert payload.numbers[0].value >= 1+ finally:+ await sampler.aclose()+ await other.dispose()+++async def test_a_restricted_role_sees_only_its_own_sessions(rls_harness) -> None:+ """The runtime role is not a superuser, and the numbers must still be true."""+ sampler = DbServerSampler(lambda: _unpooled(_restricted_url(rls_harness)))+ try:+ async with rls_harness.owner_engine.connect() as owner:+ await owner.execute(text("SELECT 1"))+ before = await sampler.sample_once()++ async with rls_harness.restricted_engine.connect() as held:+ await held.execute(text("SELECT 1"))+ during = await sampler.sample_once()+ finally:+ await sampler.aclose()++ assert before is not None and during is not None+ # The owner's open transaction is not this role's.+ assert before.connections["idle_in_transaction"] == 0+ assert during.connections["idle_in_transaction"] == 1+++async def test_an_unreachable_database_reports_nothing_and_keeps_trying(+ postgres_url: str,+) -> None:+ unreachable = make_url(postgres_url).set(host="127.0.0.1", port=1)+ sampler = DbServerSampler(+ lambda: _unpooled(unreachable.render_as_string(hide_password=False))+ )+ try:+ assert await sampler.sample_once() is None+ assert list(sampler.readings()) == []+ assert not sampler.disabled+ finally:+ await sampler.aclose()+++async def test_a_sampler_that_switches_itself_off_stops_reporting(+ postgres_url: str, registry: MetricsRegistry+) -> None:+ """The last good reading must not stay on the dashboard after it stops."""+ from sqlalchemy.exc import ProgrammingError++ sampler = DbServerSampler(lambda: _unpooled(postgres_url))+ try:+ assert await sampler.sample_once() is not None+ assert list(sampler.readings())++ async def unreadable() -> None:+ raise ProgrammingError("SELECT ...", {}, Exception("permission denied"))++ sampler._read = unreadable # type: ignore[method-assign]+ assert await sampler.sample_once() is None+ assert sampler.disabled+ assert list(sampler.readings()) == []+ finally:+ await sampler.aclose()diff --git a/core/tests/switch_core/observability/test_pool_holds.py b/core/tests/switch_core/observability/test_pool_holds.pynew file mode 100644index 000000000..5d8ec4c17--- /dev/null+++ b/core/tests/switch_core/observability/test_pool_holds.py@@ -0,0 +1,187 @@+"""Who holds a pooled connection, for how long, and how long others wait.++The pool peak says the pool filled. These say which code was holding it and+what that cost everyone queued behind it, which is what a pool timeout during+a reconnect storm needs before anyone can fix it.+"""++from __future__ import annotations++import logging+from collections.abc import Iterator+from types import SimpleNamespace++import pytest+from sqlalchemy import create_engine, text+from sqlalchemy import exc as sqlalchemy_exc+from sqlalchemy.ext.asyncio import create_async_engine+from sqlalchemy.pool import QueuePool+from sqlalchemy.util import greenlet_spawn++from switch_core.observability import pool as pool_module+from switch_core.observability.metrics import MetricsRegistry, install, uninstall+from switch_core.observability.pool import (+ UNKNOWN_CALLER,+ WaitTimedQueuePool,+ _Hold,+ _LongHoldLog,+ describe_borrower,+ install_hold_timer,+)++# Code compiled under a switch_core path, so the borrower lookup takes it for+# ours: the test module itself is deliberately not counted as a caller.+_FAKE_FILE = "/srv/switch/core/switch_core/rooms/fake_borrower.py"+++def registry() -> Iterator[MetricsRegistry]:+ registry = MetricsRegistry()+ install(registry)+ yield registry+ uninstall()+++def _histograms(registry: MetricsRegistry, name: str) -> dict[tuple, tuple[int, float]]:+ payload = next((p for p in registry.collect() if p.name == name), None)+ if payload is None:+ return {}+ return {+ tuple(sorted(point.attributes.items())): (point.count, point.total)+ for point in payload.histograms+ }+++def _ours(source: str) -> dict[str, object]:+ namespace: dict[str, object] = {+ "describe_borrower": describe_borrower,+ "greenlet_spawn": greenlet_spawn,+ }+ exec(compile(source, _FAKE_FILE, "exec"), namespace)+ return namespace+++class TestTheBorrower:+ def test_names_the_innermost_switch_core_function(self) -> None:+ borrow = _ours("def borrow():\n return describe_borrower()\n")["borrow"]++ caller, stack = borrow() # type: ignore[operator]++ assert caller == "switch_core.rooms.fake_borrower:borrow"+ assert stack[0].startswith("switch_core.rooms.fake_borrower:borrow:")++ async def test_finds_the_coroutine_behind_sqlalchemys_greenlet(self) -> None:+ """An async checkout runs in a greenlet whose own stack starts at the+ greenlet. The coroutine that awaited it is only reachable through the+ parent, and it is the one worth naming."""+ fetch = _ours(+ "async def fetch_members():\n"+ " return await greenlet_spawn(describe_borrower)\n"+ )["fetch_members"]++ caller, _ = await fetch() # type: ignore[operator]++ assert caller == "switch_core.rooms.fake_borrower:fetch_members"++ def test_code_outside_switch_core_is_unknown(self) -> None:+ assert describe_borrower() == (UNKNOWN_CALLER, ())++ def test_a_test_tree_under_a_switch_core_path_is_not_ours(self) -> None:+ source = "def borrow():\n return describe_borrower()\n"+ namespace: dict[str, object] = {"describe_borrower": describe_borrower}+ test_file = "/srv/switch/core/tests/switch_core/rooms/test_fake.py"+ exec(compile(source, test_file, "exec"), namespace)++ assert namespace["borrow"]() == (UNKNOWN_CALLER, ()) # type: ignore[operator]+++class TestHoldTime:+ def test_every_checkout_is_timed_by_its_caller(self, registry) -> None:+ engine = create_engine("sqlite://", poolclass=QueuePool)+ install_hold_timer(SimpleNamespace(sync_engine=engine)) # type: ignore[arg-type]++ with engine.connect() as conn:+ conn.execute(text("select 1"))++ holds = _histograms(registry, "switch.db.pool.hold.duration")+ assert list(holds) == [(("caller", UNKNOWN_CALLER),)]+ count, total = holds[(("caller", UNKNOWN_CALLER),)]+ assert count == 1+ assert total >= 0.0++ def test_a_long_hold_is_logged_with_its_stack(+ self, registry, caplog, monkeypatch+ ) -> None:+ monkeypatch.setattr(pool_module, "LONG_HOLD_SECONDS", 0.0)+ engine = create_engine("sqlite://", poolclass=QueuePool)+ install_hold_timer(SimpleNamespace(sync_engine=engine)) # type: ignore[arg-type]++ with caplog.at_level(logging.WARNING, logger=pool_module.__name__):+ with engine.connect():+ pass++ assert "A database connection was held for" in caplog.text+++class TestTheLongHoldLog:+ def test_one_warning_per_caller_per_window(self, caplog, monkeypatch) -> None:+ clock = iter([100.0, 110.0, 200.0])+ monkeypatch.setattr(pool_module.time, "monotonic", lambda: next(clock))+ log = _LongHoldLog()+ hold = _Hold(started=0.0, caller="switch_core.rooms:slow", stack=())++ with caplog.at_level(logging.WARNING, logger=pool_module.__name__):+ log.report(hold, 2.0)+ log.report(hold, 3.0) # inside the window: counted, not logged+ log.report(hold, 4.0) # after it: logged, with the one it held back++ warnings = [r.getMessage() for r in caplog.records]+ assert len(warnings) == 2+ assert "(0 more long holds" in warnings[0]+ assert "(1 more long holds" in warnings[1]+++class TestWaitTime:+ async def test_every_request_for_a_connection_is_timed(self, registry) -> None:+ engine = create_async_engine(+ "postgresql+asyncpg://u:p@localhost/db", poolclass=WaitTimedQueuePool+ )+ pool = engine.pool++ def get_and_return() -> None:+ # The pool's own handover, without a server to connect to: a+ # placeholder entry the queue hands straight back.+ pool._pool.put(object()) # type: ignore[attr-defined]+ pool._do_get() # type: ignore[attr-defined]++ await greenlet_spawn(get_and_return)++ waits = _histograms(registry, "switch.db.pool.wait.duration")+ assert waits[()][0] == 1+ await engine.dispose()++ async def test_a_wait_that_times_out_is_timed_too(self, registry) -> None:+ engine = create_async_engine(+ "postgresql+asyncpg://u:p@localhost/db",+ poolclass=WaitTimedQueuePool,+ pool_size=1,+ max_overflow=0,+ pool_timeout=0.01,+ )+ pool = engine.pool+ # Every connection the pool may have is out: the request can only wait.+ pool._overflow = pool._max_overflow # type: ignore[attr-defined]++ with pytest.raises(sqlalchemy_exc.TimeoutError):+ await greenlet_spawn(pool._do_get) # type: ignore[attr-defined]++ waits = _histograms(registry, "switch.db.pool.wait.duration")+ assert waits[()][0] == 1+ await engine.dispose()++ def test_the_class_survives_a_rebuild(self) -> None:+ engine = create_async_engine(+ "postgresql+asyncpg://u:p@localhost/db", poolclass=WaitTimedQueuePool+ )++ assert isinstance(engine.pool.recreate(), WaitTimedQueuePool)diff --git a/core/tests/switch_core/pool_checkouts.py b/core/tests/switch_core/pool_checkouts.pynew file mode 100644index 000000000..c89e398c4--- /dev/null+++ b/core/tests/switch_core/pool_checkouts.py@@ -0,0 +1,105 @@+"""Count the pooled connections a request holds, at the pool itself.++A request that holds one connection while it waits for a second from the same+pool can starve it: under load every connection ends up held by a request+waiting for one more, and they all time out together. The ways a request ends+up doing that are many (a service that opens its own session, a lookup helper,+a dependency that reads and never commits), so counting sessions at any one of+them misses the rest. This counts where they all meet, the pool's `checkout`+and `checkin` events, and attributes each connection to the request that+borrowed it.++A request is a `CheckoutScope`, entered with `PoolCheckouts.scope()` or, for an+app under test, `PoolCheckouts.wrap(app)`. The scope lives in a context+variable, which SQLAlchemy carries into the greenlet a checkout runs in, and+which tasks the request starts inherit, so their connections count as the+request's too. A checkin is matched back to its checkout by connection record+rather than by context, because a connection can be returned from somewhere+else, a session closed by the dependency's teardown for one.+"""++from __future__ import annotations++from collections.abc import Iterator+from contextlib import contextmanager+from contextvars import ContextVar+from dataclasses import dataclass+from typing import Any++from sqlalchemy import event+from sqlalchemy.ext.asyncio import AsyncEngine+++@dataclass+class CheckoutScope:+ """One request's connections: how many it holds now, and the most at once."""++ label: str+ held: int = 0+ peak: int = 0+++_current: ContextVar[CheckoutScope | None] = ContextVar(+ "pool_checkout_scope", default=None+)+++class PoolCheckouts:+ def __init__(self, engine: AsyncEngine) -> None:+ self._pool = engine.sync_engine.pool+ self._owners: dict[int, CheckoutScope] = {}+ self.scopes: list[CheckoutScope] = []+ event.listen(self._pool, "checkout", self._on_checkout)+ event.listen(self._pool, "checkin", self._on_checkin)++ def close(self) -> None:+ event.remove(self._pool, "checkout", self._on_checkout)+ event.remove(self._pool, "checkin", self._on_checkin)++ def _on_checkout(self, _dbapi: Any, record: Any, _proxy: Any) -> None:+ scope = _current.get()+ if scope is None:+ return+ scope.held += 1+ scope.peak = max(scope.peak, scope.held)+ self._owners[id(record)] = scope++ def _on_checkin(self, _dbapi: Any, record: Any) -> None:+ scope = self._owners.pop(id(record), None)+ if scope is not None:+ scope.held -= 1++ @contextmanager+ def scope(self, label: str) -> Iterator[CheckoutScope]:+ scope = CheckoutScope(label)+ self.scopes.append(scope)+ token = _current.set(scope)+ try:+ yield scope+ finally:+ _current.reset(token)++ def held_now(self) -> int:+ """Connections the current scope holds at this moment; 0 outside one.++ For a fake standing in for a platform, Matrix or bcrypt to record when+ it is called: the answer there should always be 0.+ """+ scope = _current.get()+ return 0 if scope is None else scope.held++ def wrap(self, app: Any) -> Any:+ """`app`, with every HTTP request in a scope of its own."""++ async def asgi(scope: dict[str, Any], receive: Any, send: Any) -> None:+ if scope["type"] != "http":+ await app(scope, receive, send)+ return+ with self.scope(f"{scope['method']} {scope['path']}"):+ await app(scope, receive, send)++ return asgi++ def over(self, limit: int = 1) -> list[CheckoutScope]:+ """Every scope that held more than `limit` connections at once."""+ return [s for s in self.scopes if s.peak > limit]diff --git a/core/tests/switch_core/telemetry/test_room_lifecycle_failure_reporting.py b/core/tests/switch_core/telemetry/test_room_lifecycle_failure_reporting.pyindex 77f394b20..d4b2b3dc6 100644--- a/core/tests/switch_core/telemetry/test_room_lifecycle_failure_reporting.py+++ b/core/tests/switch_core/telemetry/test_room_lifecycle_failure_reporting.py@@ -49,6 +49,7 @@ def _room_service() -> RoomService:collab_bridge_store=None, # type: ignore[arg-type]resource_service=None, # type: ignore[arg-type]session_factory=_broken_session_factory, # type: ignore[arg-type]+ room_cache=None, # type: ignore[arg-type]telemetry=object(), # type: ignore[arg-type])diff --git a/core/tests/switch_core/test_room_service_delete_room.py b/core/tests/switch_core/test_room_service_delete_room.pyindex 7c8a02674..40e557079 100644--- a/core/tests/switch_core/test_room_service_delete_room.py+++ b/core/tests/switch_core/test_room_service_delete_room.py@@ -70,6 +70,14 @@ class _FakeProvisioning:self._events.append(("delete_room", transport_room_id))+class _FakeRoomCache:+ def __init__(self) -> None:+ self.invalidated: list[tuple[str, str]] = []++ def invalidate(self, tenant_id: str, room_id: str) -> None:+ self.invalidated.append((tenant_id, room_id))++def _build_service(*,room: Any,@@ -85,6 +93,7 @@ def _build_service(svc._collab_lifecycle = _FakeLifecycle(bridges) # type: ignore[assignment]svc._client_lifecycle = _FakeClientLifecycle(clients) # type: ignore[assignment]svc._provisioning = _FakeProvisioning(events) # type: ignore[assignment]+ svc._room_cache = _FakeRoomCache() # type: ignore[assignment]return svc, room_store@@ -198,3 +207,23 @@ class TestDeleteRoom:with pytest.raises(ValueError, match="Room not found"):await svc.delete_room("nope")++ async def test_drops_the_rooms_shared_delivery_reads(self) -> None:+ """Members still running here are kicked, which empties the room's+ cache entry; this is the call that covers any that were not."""+ events: list[Any] = []+ room = SimpleNamespace(+ id="room-1",+ tenant_id="room-tenant",+ transport_room_id="!mx:switch.local",+ bridge_id=None,+ external_channel_id=None,+ )+ svc, room_store = _build_service(+ room=room, client_ids=[], clients={}, bridges={}, events=events+ )++ await svc.delete_room("room-1")++ assert room_store.deleted == ["room-1"]+ assert svc._room_cache.invalidated == [("room-tenant", "room-1")] # type: ignore[attr-defined]diff --git a/core/tests/switch_core/test_room_service_pool_holds.py b/core/tests/switch_core/test_room_service_pool_holds.pyindex 4275d80b3..e0f15cf3a 100644--- a/core/tests/switch_core/test_room_service_pool_holds.py+++ b/core/tests/switch_core/test_room_service_pool_holds.py@@ -16,6 +16,7 @@ written until the transport side is in place.from __future__ import annotations+from types import SimpleNamespacefrom typing import Anyimport pytest@@ -151,6 +152,9 @@ def _service(svc._provisioning = provisioning # type: ignore[assignment]svc._client_lifecycle = _RunningClients(running) # type: ignore[assignment]svc._collab_lifecycle = _RunningClients({}) # type: ignore[assignment]+ svc._room_cache = SimpleNamespace( # type: ignore[assignment]+ invalidate=lambda tenant_id, room_id: None+ )return svc, provisioningdiff --git a/core/tests/switch_core/transport/test_postgres_transport.py b/core/tests/switch_core/transport/test_postgres_transport.pyindex ac14d89aa..394ee2464 100644--- a/core/tests/switch_core/transport/test_postgres_transport.py+++ b/core/tests/switch_core/transport/test_postgres_transport.py@@ -8,6 +8,9 @@ it.What is deliberately not implemented is tested too: a transport that quietlyreturns nothing would look like a working deployment with a silent room, whichis the one failure this codebase refuses to ship.++Every transport here shares the room delivery cache, as in production. The+cache must not change anything a test here can observe."""from __future__ import annotations@@ -50,9 +53,42 @@ from switch_core.transport import ()from switch_core.transport.ephemeral import EphemeralBusfrom switch_core.transport.invites import InviteBus-from switch_core.transport.postgres import _DELIVERY_PAGE, PostgresTransport+from switch_core.transport.postgres import DELIVERY_PAGE, PostgresTransport+from switch_core.transport.room_cache import RoomCacheLimits, RoomDeliveryCachefrom tests.conftest import RLSHarness+# The room cache `_transport` hands out. One per session factory, because that+# is what the process has: one, shared by every transport built on the same+# pool. Emptied per test by `_fresh_caches`.+_CACHES: dict[int, RoomDeliveryCache] = {}++[email protected](autouse=True)+def _fresh_caches() -> Iterator[None]:+ _CACHES.clear()+ yield+ _CACHES.clear()+++def _room_cache(+ session_factory: async_sessionmaker[AsyncSession],+) -> RoomDeliveryCache:+ cache = _CACHES.get(id(session_factory))+ if cache is None:+ cache = RoomDeliveryCache(+ session_factory=session_factory,+ message_store=MessageStore(),+ limits=RoomCacheLimits(+ max_bytes=64 * 1024 * 1024,+ max_rooms=1000,+ max_rows_per_room=1000,+ max_age_seconds=300.0,+ ),+ page=DELIVERY_PAGE,+ )+ _CACHES[id(session_factory)] = cache+ return cache+async def _make_room(session: AsyncSession) -> tuple[str, str, str, str]:"""Insert the Room and Client a send depends on.@@ -202,6 +238,7 @@ def _transport(listener=listener or _FakeListener(),invites=invites or InviteBus(),ephemeral=ephemeral or EphemeralBus(),+ room_cache=_room_cache(session_factory),)@@ -592,6 +629,7 @@ class TestMedia:listener=_FakeListener(),invites=InviteBus(),ephemeral=EphemeralBus(),+ room_cache=_room_cache(session_factory),)@@ -2071,7 +2109,7 @@ class TestBeingRemovedFromARoom:async def _kick_on_the_page_boundary(_room_ref, event) -> None:delivered.append(event.body)- if len(delivered) == _DELIVERY_PAGE:+ if len(delivered) == DELIVERY_PAGE:await provisioning.kick_user(room, leaving[1])removed = _transport(@@ -2090,14 +2128,14 @@ class TestBeingRemovedFromARoom:await _watched_room(member)# More than one page, so the loop would come back for another.- for i in range(_DELIVERY_PAGE + 25):+ for i in range(DELIVERY_PAGE + 25):await member.send_message(room, f"line {i}", sender_name="agent one", metered=True)await listener.announce(room_id)# The whole of page one, and none of page two.- assert delivered == [f"line {i}" for i in range(_DELIVERY_PAGE)]+ assert delivered == [f"line {i}" for i in range(DELIVERY_PAGE)]assert room_id not in removed._watchingassert room_id not in removed._cursorsassert "Delivery failed" not in caplog.text, (diff --git a/core/tests/switch_core/transport/test_room_cache.py b/core/tests/switch_core/transport/test_room_cache.pynew file mode 100644index 000000000..5eea47266--- /dev/null+++ b/core/tests/switch_core/transport/test_room_cache.py@@ -0,0 +1,721 @@+"""The room delivery cache on its own, against a store a test can steer.++These pin the rules the cache rests on without a database in the way: what+range an entry claims, when a shared read counts for a reader, what is+published together, and what happens at every limit. The same rules are+proved again through `PostgresTransport` against Postgres in+`test_room_delivery_cache.py`; here a test can stop a read half-way and look.+"""++from __future__ import annotations++import asyncio+import contextlib+import random+import uuid+from collections.abc import AsyncIterator, Collection+from datetime import UTC, datetime+from typing import Any++import pytest++from switch_core.db.models import Message, MessageAttachment+from switch_core.tenant_context import current_tenant_id+from switch_core.transport.room_cache import (+ CachedAttachment,+ CachedPage,+ RoomCacheLimits,+ RoomDeliveryCache,+)++TENANT = "tenant-a"+ROOM = "room-1"+PAGE = 5+++class _Sessions:+ """Stands in for the session factory: `tenant_session` only enters it."""++ def __call__(self) -> contextlib.AbstractAsyncContextManager[object]:+ return self._session()++ @contextlib.asynccontextmanager+ async def _session(self) -> AsyncIterator[object]:+ yield object()+++class _Store:+ """The two reads a fill makes, over rows a test commits.++ A read takes its snapshot when it is called, like a statement does, and+ can then be held at `gate` so a test can act while it is in flight.+ """++ def __init__(self) -> None:+ self.rows: dict[tuple[str, str], list[Message]] = {}+ self.files: dict[str, list[MessageAttachment]] = {}+ self.reads = 0+ self.tenants: list[str | None] = []+ self.gate: asyncio.Event | None = None+ self.attachments_gate: asyncio.Event | None = None+ self.fail_next = False+ self.yield_randomly: random.Random | None = None++ def commit(+ self,+ room_id: str = ROOM,+ *,+ tenant_id: str = TENANT,+ body: str | None = None,+ files: int = 0,+ ) -> Message:+ log = self.rows.setdefault((tenant_id, room_id), [])+ seq = (log[-1].seq if log else 0) + 1+ row = Message(+ id=uuid.uuid4().hex,+ seq=seq,+ room_id=room_id,+ transport_event_id=f"sw_{uuid.uuid4().hex}",+ sender_id="@a:test",+ sender_name="a",+ event_type="m.room.message",+ msgtype="m.file" if files else "m.text",+ body=body or f"line {seq}",+ formatted_body=None,+ thread_root_event_id=None,+ content={"body": body or f"line {seq}", "nested": {"n": seq}},+ sent_at=datetime.now(UTC),+ )+ log.append(row)+ self.files[row.id] = [+ MessageAttachment(+ message_id=row.id,+ position=i,+ uri=f"switch-media://{row.id}/{i}",+ filename=f"f{i}.txt",+ mimetype="text/plain",+ size=10 + i,+ )+ for i in range(files)+ ]+ return row++ async def _maybe_yield(self) -> None:+ if self.yield_randomly is not None:+ for _ in range(self.yield_randomly.randint(0, 3)):+ await asyncio.sleep(0)++ async def list_for_room(+ self, session: object, room_id: str, *, after_seq: int, limit: int+ ) -> list[Message]:+ self.reads += 1+ tenant = current_tenant_id()+ self.tenants.append(tenant)+ await self._maybe_yield()+ if self.fail_next:+ self.fail_next = False+ raise RuntimeError("pool timeout")+ log = self.rows.get((tenant or "", room_id), [])+ snapshot = [row for row in log if row.seq > after_seq][:limit]+ if self.gate is not None:+ await self.gate.wait()+ await self._maybe_yield()+ return snapshot++ async def attachments_for(+ self, session: object, message_ids: Collection[str]+ ) -> dict[str, list[MessageAttachment]]:+ if self.attachments_gate is not None:+ await self.attachments_gate.wait()+ await self._maybe_yield()+ return {i: self.files[i] for i in message_ids if self.files.get(i)}+++def _limits(**overrides: Any) -> RoomCacheLimits:+ values: dict[str, Any] = {+ "max_bytes": 10_000_000,+ "max_rooms": 100,+ "max_rows_per_room": 50,+ "max_age_seconds": 300.0,+ }+ values.update(overrides)+ return RoomCacheLimits(**values)+++def _cache(store: _Store, page: int = PAGE, **limits: Any) -> RoomDeliveryCache:+ return RoomDeliveryCache(+ session_factory=_Sessions(), # type: ignore[arg-type]+ message_store=store, # type: ignore[arg-type]+ limits=_limits(**limits),+ page=page,+ )+++def _seqs(page: CachedPage | None) -> list[int]:+ assert page is not None+ return [row.seq for row in page.rows]+++async def _read(+ cache: RoomDeliveryCache,+ after_seq: int,+ woken_at: int,+ *,+ tenant_id: str = TENANT,+ room_id: str = ROOM,+) -> CachedPage | None:+ return await cache.read(+ tenant_id, room_id, after_seq=after_seq, woken_at=woken_at, limit=PAGE+ )+++async def _until(predicate: Any) -> None:+ for _ in range(200):+ if predicate():+ return+ await asyncio.sleep(0)+ raise AssertionError("never happened")+++class TestOneFillPerRoom:+ async def test_concurrent_reads_for_one_room_collapse_into_one_fill(self) -> None:+ store = _Store()+ cache = _cache(store)+ watchers = [object() for _ in range(6)]+ for watcher in watchers:+ cache.attach(TENANT, ROOM, watcher)+ for _ in range(3):+ store.commit()+ store.gate = asyncio.Event()+ woken = cache.tick()++ readers = [asyncio.create_task(_read(cache, 0, woken)) for _ in watchers]+ await _until(lambda: store.reads == 1)+ for _ in range(5):+ await asyncio.sleep(0)+ assert store.reads == 1, "a second read started while one was in flight"+ store.gate.set()+ pages = await asyncio.gather(*readers)++ assert store.reads == 1+ assert all(_seqs(page) == [1, 2, 3] for page in pages)+ assert all(page is not None and page.done for page in pages)++ async def test_a_reader_cancelled_while_waiting_does_not_cancel_the_fill(+ self,+ ) -> None:+ """The fill is the room's, not the reader's that happened to start it."""+ store = _Store()+ cache = _cache(store)+ cache.attach(TENANT, ROOM, "a")+ cache.attach(TENANT, ROOM, "b")+ store.commit()+ store.gate = asyncio.Event()+ woken = cache.tick()++ starter = asyncio.create_task(_read(cache, 0, woken))+ await _until(lambda: store.reads == 1)+ other = asyncio.create_task(_read(cache, 0, woken))+ await asyncio.sleep(0)+ starter.cancel()+ with pytest.raises(asyncio.CancelledError):+ await starter+ store.gate.set()++ assert _seqs(await other) == [1]+ assert store.reads == 1++ async def test_a_failed_fill_reaches_its_waiters_and_the_next_read_retries(+ self,+ ) -> None:+ store = _Store()+ cache = _cache(store)+ cache.attach(TENANT, ROOM, "a")+ store.commit()+ store.fail_next = True+ woken = cache.tick()++ with pytest.raises(RuntimeError, match="pool timeout"):+ await _read(cache, 0, woken)+ assert _seqs(await _read(cache, 0, woken)) == [1]+++class TestTheHandoff:+ """The range an entry claims, and when a shared read counts for a reader."""++ async def test_a_reader_inside_the_range_gets_exactly_the_rows_after_it(+ self,+ ) -> None:+ store = _Store()+ cache = _cache(store)+ cache.attach(TENANT, ROOM, "a")+ for _ in range(4):+ store.commit()+ woken = cache.tick()+ assert _seqs(await _read(cache, 0, woken)) == [1, 2, 3, 4]++ for cursor in range(4):+ page = await _read(cache, cursor, woken)+ assert _seqs(page) == list(range(cursor + 1, 5))+ assert page is not None and page.done+ assert store.reads == 1++ async def test_a_fill_that_began_before_a_wake_does_not_answer_for_it(+ self,+ ) -> None:+ """The missed-notification window, closed.++ A read that took its snapshot before row 2 committed cannot say there+ is nothing after 1 to a reader woken for row 2. It is waited for, not+ trusted, and a fresh fill finds the row.+ """+ store = _Store()+ cache = _cache(store)+ cache.attach(TENANT, ROOM, "a")+ cache.attach(TENANT, ROOM, "b")+ store.commit()+ store.gate = asyncio.Event()+ early = asyncio.create_task(_read(cache, 0, cache.tick()))+ await _until(lambda: store.reads == 1)++ store.commit() # row 2, committed after the fill's snapshot+ woken_for_two = cache.tick()+ late = asyncio.create_task(_read(cache, 0, woken_for_two))+ await asyncio.sleep(0)+ store.gate.set()++ assert _seqs(await early) == [1]+ first = await late+ assert _seqs(first) == [1]+ assert first is not None and not first.done, "told it was caught up"+ second = await _read(cache, 1, woken_for_two)+ assert _seqs(second) == [2]+ assert second is not None and second.done+ assert store.reads == 2++ async def test_caught_up_means_a_fill_after_the_wake_reached_the_end(+ self,+ ) -> None:+ store = _Store()+ cache = _cache(store)+ cache.attach(TENANT, ROOM, "a")+ store.commit()+ before = cache.tick()+ assert _seqs(await _read(cache, 0, before)) == [1]++ # Nothing new, and the last fill started after this wake: no read.+ assert _seqs(await _read(cache, 1, before)) == []+ assert store.reads == 1++ # Woken again: the old fill cannot vouch for it, so one more read.+ store.commit()+ later = cache.tick()+ assert _seqs(await _read(cache, 1, later)) == [2]+ assert store.reads == 2++ async def test_a_reader_below_the_range_is_sent_to_the_database(self) -> None:+ store = _Store()+ cache = _cache(store)+ cache.attach(TENANT, ROOM, "a")+ for _ in range(3):+ store.commit()+ woken = cache.tick()+ # The first reader is at 2, so the entry starts there.+ assert _seqs(await _read(cache, 2, woken)) == [3]++ assert await _read(cache, 1, woken) is None+ assert await _read(cache, 0, woken) is None+ # Once its own reads bring it to the range, it is served from it.+ assert _seqs(await _read(cache, 2, woken)) == [3]++ async def test_a_reader_ahead_of_the_entry_extends_it_without_a_gap(+ self,+ ) -> None:+ store = _Store()+ cache = _cache(store)+ cache.attach(TENANT, ROOM, "a")+ store.commit()+ assert _seqs(await _read(cache, 0, cache.tick())) == [1]+ for _ in range(3):+ store.commit()++ woken = cache.tick()+ # This reader read 2 and 3 itself; the entry extends from 1, not 3.+ assert _seqs(await _read(cache, 3, woken)) == [4]+ assert _seqs(await _read(cache, 0, woken)) == [1, 2, 3, 4]++ async def test_a_long_room_is_read_a_page_at_a_time(self) -> None:+ store = _Store()+ cache = _cache(store)+ cache.attach(TENANT, ROOM, "a")+ for _ in range(PAGE * 2 + 1):+ store.commit()+ woken = cache.tick()++ cursor, seen = 0, []+ while True:+ page = await _read(cache, cursor, woken)+ assert page is not None+ seen.extend(row.seq for row in page.rows)+ if page.rows:+ cursor = page.rows[-1].seq+ if page.done:+ break+ assert seen == list(range(1, PAGE * 2 + 2))+++class TestWhatIsPublished:+ async def test_rows_are_published_only_with_their_attachments(self) -> None:+ store = _Store()+ cache = _cache(store)+ cache.attach(TENANT, ROOM, "a")+ cache.attach(TENANT, ROOM, "b")+ store.commit(files=2)+ store.attachments_gate = asyncio.Event()+ woken = cache.tick()++ first = asyncio.create_task(_read(cache, 0, woken))+ await _until(lambda: store.reads == 1)+ # Rows read, attachments not yet: nothing is visible to anyone.+ assert cache.stats().rows == 0+ second = asyncio.create_task(_read(cache, 0, woken))+ await asyncio.sleep(0)+ assert not second.done()+ store.attachments_gate.set()++ for page in (await first, await second):+ assert page is not None+ (row,) = page.rows+ assert [f.uri for f in row.attachments] == [+ f"switch-media://{row.id}/0",+ f"switch-media://{row.id}/1",+ ]++ async def test_attachments_are_metadata_only(self) -> None:+ """Media bytes live in `media_blobs` and are never read or held."""+ assert set(CachedAttachment.__slots__) == {+ "uri",+ "filename",+ "mimetype",+ "size",+ }++ async def test_each_access_to_content_is_a_fresh_dict(self) -> None:+ store = _Store()+ cache = _cache(store)+ cache.attach(TENANT, ROOM, "a")+ store.commit(body="original")+ page = await _read(cache, 0, cache.tick())+ assert page is not None+ (row,) = page.rows++ mine = row.content+ mine["body"] = "edited"+ mine["nested"]["n"] = -1+ theirs = row.content+ assert theirs["body"] == "original"+ assert theirs["nested"] == {"n": 1}++ async def test_a_cached_row_cannot_be_changed(self) -> None:+ store = _Store()+ cache = _cache(store)+ cache.attach(TENANT, ROOM, "a")+ store.commit()+ page = await _read(cache, 0, cache.tick())+ assert page is not None+ with pytest.raises(AttributeError):+ page.rows[0].body = "edited" # type: ignore[misc]+++class TestKeys:+ async def test_an_entry_is_never_served_to_another_tenant(self) -> None:+ """Same room id, two tenants: two entries, each filled under its own+ tenant. A room id alone never finds another tenant's rows."""+ store = _Store()+ cache = _cache(store)+ cache.attach("tenant-a", ROOM, "a")+ cache.attach("tenant-b", ROOM, "b")+ store.commit(tenant_id="tenant-a", body="for a")+ woken = cache.tick()++ mine = await _read(cache, 0, woken, tenant_id="tenant-a")+ theirs = await _read(cache, 0, woken, tenant_id="tenant-b")++ assert mine is not None and [r.body for r in mine.rows] == ["for a"]+ assert theirs is not None and theirs.rows == ()+ assert store.tenants == ["tenant-a", "tenant-b"]+ assert cache.stats().rooms == 2++ async def test_a_room_nobody_is_watching_is_not_cached(self) -> None:+ store = _Store()+ cache = _cache(store)+ store.commit()+ assert await _read(cache, 0, cache.tick()) is None+ assert cache.stats().rooms == 0+ assert store.reads == 0+++class TestLettingGo:+ async def _filled(self, cache: RoomDeliveryCache, store: _Store) -> None:+ cache.attach(TENANT, ROOM, "a")+ store.commit()+ assert _seqs(await _read(cache, 0, cache.tick())) == [1]++ async def test_the_last_watcher_leaving_purges_the_room(self) -> None:+ store = _Store()+ cache = _cache(store)+ cache.attach(TENANT, ROOM, "b")+ await self._filled(cache, store)++ cache.detach(TENANT, ROOM, "a")+ assert cache.stats().rooms == 1, "one member is still watching"+ cache.detach(TENANT, ROOM, "b")+ assert cache.stats() == cache.stats().__class__(bytes=0, rooms=0, rows=0)++ async def test_detaching_what_never_attached_is_harmless(self) -> None:+ cache = _cache(_Store())+ cache.detach(TENANT, ROOM, "never")++ async def test_invalidate_drops_the_entry(self) -> None:+ """The hook for a future edit or redaction, and for room deletion."""+ store = _Store()+ cache = _cache(store)+ await self._filled(cache, store)++ cache.invalidate(TENANT, ROOM)++ assert cache.stats().rooms == 0+ assert cache.stats().bytes == 0+ # Still watched, so the next read fills afresh from the database.+ assert _seqs(await _read(cache, 0, cache.tick())) == [1]+ assert store.reads == 2++ async def test_invalidate_during_a_fill_sends_its_waiters_to_the_database(+ self,+ ) -> None:+ store = _Store()+ cache = _cache(store)+ cache.attach(TENANT, ROOM, "a")+ store.commit()+ store.gate = asyncio.Event()+ reader = asyncio.create_task(_read(cache, 0, cache.tick()))+ await _until(lambda: store.reads == 1)++ cache.invalidate(TENANT, ROOM)+ store.gate.set()++ assert await reader is None+ assert cache.stats().rooms == 0, "a discarded fill came back to life"++ async def test_an_eviction_during_a_fill_still_serves_its_waiters(+ self,+ ) -> None:+ """The rows were read once already; a waiter reading them again+ because the entry was evicted would multiply the load under churn."""+ store = _Store()+ cache = _cache(store, max_rooms=1)+ cache.attach(TENANT, ROOM, "a")+ cache.attach(TENANT, "other", "b")+ store.commit()+ store.commit("other")+ store.gate = asyncio.Event()+ woken = cache.tick()+ waiters = [asyncio.create_task(_read(cache, 0, woken)) for _ in range(3)]+ await _until(lambda: store.reads == 1)+ # Another room's read takes the only slot and evicts this one mid-fill.+ other = asyncio.create_task(_read(cache, 0, cache.tick(), room_id="other"))+ await _until(lambda: store.reads == 2)++ store.gate.set()++ for waiter in waiters:+ page = await waiter+ assert page is not None, "an evicted fill sent its waiter to the database"+ assert _seqs(page) == [1]+ assert page.done+ assert _seqs(await other) == [1]+ assert cache.stats().rooms == 1, "the evicted room came back"+ assert store.reads == 2++ async def test_the_byte_limit_evicts_the_least_recently_read_room(+ self,+ ) -> None:+ store = _Store()+ probe = _cache(store)+ probe.attach(TENANT, "probe", "p")+ store.commit("probe")+ await _read(probe, 0, probe.tick(), room_id="probe")+ one_row = probe.stats().bytes++ cache = _cache(store, max_bytes=one_row * 2 + one_row // 2)+ for room in ("r1", "r2", "r3"):+ cache.attach(TENANT, room, "a")+ store.commit(room)+ woken = cache.tick()+ await _read(cache, 0, woken, room_id="r1")+ await _read(cache, 0, woken, room_id="r2")+ await _read(cache, 0, woken, room_id="r1") # r2 is now the oldest+ await _read(cache, 0, cache.tick(), room_id="r3")++ assert cache.stats().rooms == 2+ assert cache.stats().bytes <= one_row * 2 + one_row // 2+ reads = store.reads+ await _read(cache, 0, woken, room_id="r1")+ assert store.reads == reads, "r1 should have survived"+ await _read(cache, 0, cache.tick(), room_id="r2")+ assert store.reads == reads + 1, "r2 should have been evicted"++ async def test_the_room_limit_evicts_the_least_recently_read_room(+ self,+ ) -> None:+ store = _Store()+ cache = _cache(store, max_rooms=2)+ for room in ("r1", "r2", "r3"):+ cache.attach(TENANT, room, "a")+ store.commit(room)+ await _read(cache, 0, cache.tick(), room_id=room)+ assert cache.stats().rooms == 2++ async def test_the_row_limit_raises_the_floor(self) -> None:+ store = _Store()+ cache = _cache(store, max_rows_per_room=PAGE)+ cache.attach(TENANT, ROOM, "a")+ for _ in range(PAGE):+ store.commit()+ woken = cache.tick()+ assert len(_seqs(await _read(cache, 0, woken))) == PAGE+ store.commit()+ store.commit()+ woken = cache.tick()+ assert _seqs(await _read(cache, PAGE, woken)) == [PAGE + 1, PAGE + 2]++ assert cache.stats().rows == PAGE+ # Rows 1 and 2 were let go: a reader that still needs them reads them.+ assert await _read(cache, 0, woken) is None+ assert await _read(cache, 1, woken) is None+ assert _seqs(await _read(cache, 2, woken))[0] == 3++ async def test_the_age_limit_raises_the_floor(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ clock = [1000.0]+ monkeypatch.setattr(+ "switch_core.transport.room_cache.time.monotonic", lambda: clock[0]+ )+ store = _Store()+ cache = _cache(store, max_age_seconds=60.0)+ cache.attach(TENANT, ROOM, "a")+ store.commit()+ woken = cache.tick()+ await _read(cache, 0, woken)+ clock[0] += 61.0+ store.commit()+ woken = cache.tick()+ assert _seqs(await _read(cache, 1, woken)) == [2]++ assert await _read(cache, 0, woken) is None, "row 1 outlived its age"+ assert cache.stats().rows == 1++ def test_a_room_must_hold_at_least_one_page(self) -> None:+ with pytest.raises(ValueError, match="max_rows_per_room"):+ _cache(_Store(), page=200, max_rows_per_room=199)+++class TestEveryRowOnceInOrder:+ """Readers, writes, evictions and invalidations, interleaved at random.++ Each reader behaves as a transport does: it is woken after a commit, keeps+ its own cursor, takes what the cache gives it and reads the store itself+ when the cache says no. Whatever the interleaving, each must see every+ row exactly once and in order.+ """++ @pytest.mark.parametrize("seed", range(40))+ async def test_random_interleavings(self, seed: int) -> None:+ rng = random.Random(seed)+ store = _Store()+ store.yield_randomly = rng+ page = rng.choice([1, 2, 3, PAGE])+ cache = _cache(+ store,+ page=page,+ max_rows_per_room=page + rng.randint(0, 4),+ max_bytes=rng.choice([10_000_000, 3000, 1500]),+ )+ readers = 4+ total = 40+ received: list[list[int]] = [[] for _ in range(readers)]+ wakes = [asyncio.Event() for _ in range(readers)]+ woken = [0] * readers+ cursors = [0] * readers+ idle = [asyncio.Event() for _ in range(readers)]++ async def own_read(after: int) -> tuple[list[int], bool]:+ # A transport's own database read: snapshot, then a suspension.+ rows = [r.seq for r in store.rows.get((TENANT, ROOM), [])]+ await asyncio.sleep(0)+ found = [s for s in rows if s > after][:page]+ return found, len(found) < page++ async def reader(i: int) -> None:+ cache.attach(TENANT, ROOM, i)+ while True:+ await wakes[i].wait()+ wakes[i].clear()+ at = woken[i]+ while True:+ shared = await cache.read(+ TENANT, ROOM, after_seq=cursors[i], woken_at=at, limit=page+ )+ if shared is None:+ seqs, done = await own_read(cursors[i])+ else:+ seqs, done = [r.seq for r in shared.rows], shared.done+ for seq in seqs:+ assert seq == cursors[i] + 1, (i, cursors[i], seqs)+ cursors[i] = seq+ received[i].append(seq)+ # Reader 3 is slow, so it falls out of the held range.+ for _ in range(rng.randint(0, 6) if i == 3 else 0):+ await asyncio.sleep(0)+ if rng.random() < 0.3:+ await asyncio.sleep(0)+ if done or not seqs:+ break+ if not wakes[i].is_set():+ idle[i].set()++ def wake_all() -> None:+ for i in range(readers):+ woken[i] = cache.tick()+ idle[i].clear()+ wakes[i].set()++ tasks = [asyncio.create_task(reader(i)) for i in range(readers)]+ try:+ for _ in range(total):+ store.commit()+ wake_all()+ for _ in range(rng.randint(0, 4)):+ await asyncio.sleep(0)+ if rng.random() < 0.05:+ cache.invalidate(TENANT, ROOM)+ for i in range(readers):+ settled = asyncio.ensure_future(idle[i].wait())+ done, _ = await asyncio.wait(+ [settled, tasks[i]],+ timeout=5,+ return_when=asyncio.FIRST_COMPLETED,+ )+ settled.cancel()+ if tasks[i] in done:+ tasks[i].result() # a reader's own assertion, raised here+ assert done, f"reader {i} never settled, seed {seed}"+ finally:+ for task in tasks:+ task.cancel()+ await asyncio.gather(*tasks, return_exceptions=True)++ expected = list(range(1, total + 1))+ for i in range(readers):+ assert received[i] == expected, f"reader {i}, seed {seed}"diff --git a/core/tests/switch_core/transport/test_room_delivery_cache.py b/core/tests/switch_core/transport/test_room_delivery_cache.pynew file mode 100644index 000000000..c2f10b304--- /dev/null+++ b/core/tests/switch_core/transport/test_room_delivery_cache.py@@ -0,0 +1,675 @@+"""The room delivery cache through `PostgresTransport`, against Postgres.++`test_room_cache.py` pins the cache's rules with a store a test can stop+half-way. These prove the same guarantees where they matter: real transports,+real rows, real row-level security, and in one case the real LISTEN.++`test_postgres_transport.py` runs every existing transport test on the cache+too.+"""++from __future__ import annotations++import asyncio+import uuid+from collections.abc import Iterator+from typing import Any++import pytest+from sqlalchemy import select+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine+from sqlalchemy.pool import NullPool++from switch_core.db.models import Client, ClientRoom, Message, Room, Tenant+from switch_core.db.session_scope import tenant_session+from switch_core.db.stores.client_store import ClientStore+from switch_core.db.stores.message_store import MessageStore+from switch_core.db.stores.room_store import RoomStore+from switch_core.messages.notify import MessageListener+from switch_core.provisioning.postgres import PostgresProvisioning+from switch_core.transport import InboundMedia, InboundMessage+from switch_core.transport.invites import InviteBus+from switch_core.transport.postgres import DELIVERY_PAGE, PostgresTransport+from switch_core.transport.room_cache import RoomCacheLimits, RoomDeliveryCache+from tests.conftest import RLSHarness+from tests.switch_core.transport.test_postgres_transport import (+ _FakeListener,+ _make_client,+ _make_room,+ _Received,+ _settled,+ _transport,+ _watched_room,+)++DELIVERY_TIMEOUT = 10.0+++def _cache(+ session_factory: async_sessionmaker[AsyncSession], **limits: Any+) -> RoomDeliveryCache:+ values: dict[str, Any] = {+ "max_bytes": 64 * 1024 * 1024,+ "max_rooms": 1000,+ "max_rows_per_room": 1000,+ "max_age_seconds": 300.0,+ }+ values.update(limits)+ return RoomDeliveryCache(+ session_factory=session_factory,+ message_store=MessageStore(),+ limits=RoomCacheLimits(**values),+ page=DELIVERY_PAGE,+ )+++def _with(transport: PostgresTransport, cache: RoomDeliveryCache) -> PostgresTransport:+ """The transport `_transport` built, on the cache this test chose.++ `_transport` hands out that module's shared cache; these tests each decide+ for themselves which cache, and its limits.+ """+ transport._room_cache = cache+ return transport+++class _Room:+ """One room with `size` members, each a receiving transport."""++ def __init__(self) -> None:+ self.room_id = ""+ self.transport_room_id = ""+ self.members: list[PostgresTransport] = []+ self.received: list[_Received] = []+ self.tasks: list[asyncio.Task[None]] = []++ @classmethod+ async def open(+ cls,+ session_factory: async_sessionmaker[AsyncSession],+ size: int,+ *,+ cache: RoomDeliveryCache,+ listener: Any,+ invites: InviteBus | None = None,+ ) -> _Room:+ room = cls()+ async with session_factory() as session:+ room_id, transport_room_id, first_id, first_user = await _make_room(session)+ clients = [(first_id, first_user)]+ for n in range(size - 1):+ clients.append(await _make_client(session, f"member{n}"))+ for client_id, _ in clients:+ # Recorded directly, so no arrival row is in the way.+ await RoomStore().add_client(session, client_id, room_id)+ await session.commit()+ room.room_id, room.transport_room_id = room_id, transport_room_id+ for client_id, user_id in clients:+ transport = _with(+ _transport(+ session_factory,+ client_id=client_id,+ user_id=user_id,+ listener=listener,+ invites=invites,+ ),+ cache,+ )+ received = _Received()+ transport.register_handlers(received.handlers())+ room.members.append(transport)+ room.received.append(received)+ room.tasks.append(asyncio.create_task(transport.receive_forever()))+ for transport in room.members:+ await _watched_room(transport)+ return room++ def close(self) -> None:+ for task in self.tasks:+ task.cancel()++ async def settled(self) -> None:+ for transport in self.members:+ await _settled(transport)+++class _ReadCounter:+ """Counts `list_for_room` calls, which is what a delivery read costs."""++ def __init__(self, monkeypatch: pytest.MonkeyPatch) -> None:+ self.calls = 0+ original = MessageStore.list_for_room++ async def _counting(store, session, room_id, **kwargs): # type: ignore[no-untyped-def]+ self.calls += 1+ return await original(store, session, room_id, **kwargs)++ monkeypatch.setattr(MessageStore, "list_for_room", _counting)+++async def _event_ids_in_order(+ session_factory: async_sessionmaker[AsyncSession], room_id: str+) -> list[str]:+ async with session_factory() as session:+ rows = await session.execute(+ select(Message.transport_event_id)+ .where(Message.room_id == room_id, Message.seq > 0)+ .order_by(Message.seq)+ )+ return list(rows.scalars())+++def _ids(received: _Received) -> list[str]:+ return [event.event_id for event in received.events] # type: ignore[attr-defined]+++def rooms() -> Iterator[list[_Room]]:+ opened: list[_Room] = []+ yield opened+ for room in opened:+ room.close()+++class TestEveryMemberGetsEveryRowOnce:+ async def test_a_room_of_six_reads_each_new_message_once(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ monkeypatch: pytest.MonkeyPatch,+ rooms: list[_Room],+ ) -> None:+ """The case the cache exists for, and the numbers for the PR.++ Six members, twenty messages, each announced as it is sent. Every+ member must get every message once and in order, and the room is read+ once per message instead of once per member.+ """+ cache = _cache(session_factory)+ listener = _FakeListener()+ room = await _Room.open(session_factory, 6, cache=cache, listener=listener)+ rooms.append(room)+ reads = _ReadCounter(monkeypatch)++ messages = 20+ for n in range(messages):+ await room.members[n % 6].send_message(+ room.transport_room_id, f"m{n}", sender_name="a", metered=False+ )+ await listener.announce(room.room_id)++ expected = await _event_ids_in_order(session_factory, room.room_id)+ assert len(expected) == messages+ for received in room.received:+ assert _ids(received) == expected+ per_message = reads.calls / messages+ print( # noqa: T201 - the figure the PR quotes, read with -s+ f"\nroom of 6, {messages} messages: {reads.calls} list_for_room "+ f"calls, {per_message:.2f} per message"+ )+ assert reads.calls == messages++ async def test_a_burst_over_the_real_listener_arrives_once_in_order(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ postgres_schema: str,+ monkeypatch: pytest.MonkeyPatch,+ rooms: list[_Room],+ ) -> None:+ """The real LISTEN, the real fan-out, three writers at once.++ Announcements coalesce and land mid-read here, the way they do in+ production; nothing is stepping the transports one at a time.+ """+ listener = MessageListener(+ lambda: create_async_engine(postgres_schema, poolclass=NullPool)+ )+ await listener.start()+ try:+ await asyncio.wait_for(listener.connected.wait(), DELIVERY_TIMEOUT)+ cache = _cache(session_factory)+ room = await _Room.open(session_factory, 6, cache=cache, listener=listener)+ rooms.append(room)+ await room.settled()+ reads = _ReadCounter(monkeypatch)++ async def write(member: int, count: int) -> None:+ for n in range(count):+ await room.members[member].send_message(+ room.transport_room_id,+ f"{member}-{n}",+ sender_name="a",+ metered=False,+ )++ await asyncio.gather(write(0, 25), write(1, 25), write(2, 25))+ expected = await _event_ids_in_order(session_factory, room.room_id)+ assert len(expected) == 75++ async def all_arrived() -> None:+ while any(len(r.events) < 75 for r in room.received):+ await asyncio.sleep(0.02)++ await asyncio.wait_for(all_arrived(), DELIVERY_TIMEOUT)+ await asyncio.sleep(0.2) # long enough for a duplicate to show+ for received in room.received:+ assert _ids(received) == expected+ print( # noqa: T201 - the figure the PR quotes, read with -s+ f"\nburst over LISTEN, room of 6, 75 messages from 3 writers: "+ f"{reads.calls} list_for_room calls"+ )+ finally:+ await listener.stop()++ async def test_members_at_different_cursors_each_get_their_own_sequence(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ rooms: list[_Room],+ ) -> None:+ """One member far behind, one in the middle, one at the head.++ Whoever reads first decides where the held range starts; the others+ are below it (their own read until they reach it) or above it (the+ range is extended up to them). Each gets exactly what follows its own+ cursor.+ """+ cache = _cache(session_factory)+ listener = _FakeListener()+ room = await _Room.open(session_factory, 3, cache=cache, listener=listener)+ rooms.append(room)+ for n in range(12):+ await room.members[0].send_message(+ room.transport_room_id, f"m{n}", sender_name="a", metered=False+ )+ expected = await _event_ids_in_order(session_factory, room.room_id)++ for start in ([0, 5, 11], [11, 0, 5], [5, 11, 0]):+ cache.invalidate(room.members[0].tenant_id, room.room_id)+ for member, received in zip(room.members, room.received, strict=True):+ received.events.clear()+ for member, cursor in zip(room.members, start, strict=True):+ member._cursors[room.room_id] = cursor+ await listener.announce(room.room_id)+ for received, cursor in zip(room.received, start, strict=True):+ assert _ids(received) == expected[cursor:], start++ async def test_evictions_mid_stream_lose_and_repeat_nothing(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ rooms: list[_Room],+ ) -> None:+ """A byte limit smaller than one row, and invalidations as it runs.++ Every fill is thrown away the moment it lands, so every read falls+ back to the transport's own. Slower, and still exactly once.+ """+ cache = _cache(session_factory, max_bytes=1)+ listener = _FakeListener()+ room = await _Room.open(session_factory, 4, cache=cache, listener=listener)+ rooms.append(room)++ for n in range(15):+ await room.members[n % 4].send_message(+ room.transport_room_id, f"m{n}", sender_name="a", metered=False+ )+ if n % 5 == 0:+ cache.invalidate(room.members[0].tenant_id, room.room_id)+ await listener.announce(room.room_id)++ expected = await _event_ids_in_order(session_factory, room.room_id)+ for received in room.received:+ assert _ids(received) == expected+ assert cache.stats().bytes <= 1+++class TestWhatAMemberIsHanded:+ async def test_files_arrive_with_their_rows(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ rooms: list[_Room],+ ) -> None:+ cache = _cache(session_factory)+ listener = _FakeListener()+ room = await _Room.open(session_factory, 3, cache=cache, listener=listener)+ rooms.append(room)++ await room.members[0].send_media(+ room.transport_room_id,+ "switch-media://abc",+ "notes.txt",+ "text/plain",+ 12,+ sender_name="a",+ msgtype="m.file",+ metered=False,+ )+ await listener.announce(room.room_id)++ for received in room.received:+ (event,) = received.events+ assert isinstance(event, InboundMedia)+ assert (event.uri, event.filename, event.mimetype, event.size) == (+ "switch-media://abc",+ "notes.txt",+ "text/plain",+ 12,+ )++ async def test_changing_a_delivered_message_changes_nobody_elses(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ rooms: list[_Room],+ ) -> None:+ cache = _cache(session_factory)+ listener = _FakeListener()+ room = await _Room.open(session_factory, 3, cache=cache, listener=listener)+ rooms.append(room)++ async def vandal(_room: object, event: InboundMessage) -> None:+ event.content["body"] = "vandalised"+ event.content["nested"]["n"] = -1 # type: ignore[index]+ room.received[0].events.append(event)++ room.members[0].register_handlers(room.received[0].handlers(on_message=vandal))+ for _ in range(2): # twice, so a later reader follows an earlier vandal+ await room.members[1].send_message(+ room.transport_room_id,+ "original",+ sender_name="a",+ metered=False,+ extra_content={"nested": {"n": 1}},+ )+ await listener.announce(room.room_id)++ for received in room.received[1:]:+ assert [e.content["body"] for e in received.events] == ["original"] * 2 # type: ignore[attr-defined]+ assert [e.content["nested"] for e in received.events] == [{"n": 1}] * 2 # type: ignore[attr-defined]+++class TestMembershipIsStillEachClients:+ async def test_a_removed_member_gets_nothing_after_removal(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ monkeypatch: pytest.MonkeyPatch,+ rooms: list[_Room],+ ) -> None:+ """Removed while the room's shared read was in flight.++ The read completes and the remaining member is handed the row; the+ removed one was waiting on the same read and gets nothing from it.+ """+ cache = _cache(session_factory)+ listener = _FakeListener()+ invites = InviteBus()+ room = await _Room.open(+ session_factory, 2, cache=cache, listener=listener, invites=invites+ )+ rooms.append(room)+ leaving, staying = room.members++ await staying.send_message(+ room.transport_room_id, "held", sender_name="a", metered=False+ )+ gate = asyncio.Event()+ original = MessageStore.list_for_room++ async def held(store, session, room_id, **kwargs): # type: ignore[no-untyped-def]+ await gate.wait()+ return await original(store, session, room_id, **kwargs)++ monkeypatch.setattr(MessageStore, "list_for_room", held)+ for waker in list(listener.wakers[room.room_id]):+ await waker(room.room_id)+ for _ in range(20):+ await asyncio.sleep(0)+ assert leaving._delivering and staying._delivering++ provisioning = PostgresProvisioning(+ session_factory=session_factory,+ room_store=RoomStore(),+ client_store=ClientStore(),+ message_store=MessageStore(),+ invites=invites,+ )+ await provisioning.kick_user(room.transport_room_id, leaving.user_id)+ gate.set()+ await room.settled()++ assert room.received[0].events == []+ assert [e.body for e in room.received[1].events] == ["held"] # type: ignore[attr-defined]++ # And nothing after it, while the room is still cached for the other.+ await staying.send_message(+ room.transport_room_id, "after", sender_name="a", metered=False+ )+ await listener.announce(room.room_id)+ assert room.received[0].events == []+ assert [e.body for e in room.received[1].events] == ["held", "after"] # type: ignore[attr-defined]++ async def test_a_room_everyone_has_left_is_not_held(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ rooms: list[_Room],+ ) -> None:+ """Deleting a room removes its members first, so this is also what+ deleting one leaves behind: nothing."""+ cache = _cache(session_factory)+ listener = _FakeListener()+ invites = InviteBus()+ room = await _Room.open(+ session_factory, 2, cache=cache, listener=listener, invites=invites+ )+ rooms.append(room)+ await room.members[0].send_message(+ room.transport_room_id, "hello", sender_name="a", metered=False+ )+ await listener.announce(room.room_id)+ assert cache.stats().rooms == 1++ provisioning = PostgresProvisioning(+ session_factory=session_factory,+ room_store=RoomStore(),+ client_store=ClientStore(),+ message_store=MessageStore(),+ invites=invites,+ )+ for member in room.members:+ await provisioning.kick_user(room.transport_room_id, member.user_id)++ assert cache.stats().rooms == 0+ assert cache.stats().bytes == 0++ async def test_another_tenants_rows_are_never_served(+ self, rls_harness: RLSHarness+ ) -> None:+ """A transport of tenant B pointed at tenant A's room, by force.++ The schema makes this impossible to reach honestly (a client cannot be+ a member of another tenant's room), so the subscription is written+ straight into the transport, standing in for some future bug that+ reaches it. A's rows are already in the cache under A's key. B's read+ is keyed by B and filled under B's tenant, so row-level security+ answers it, and the answer is nothing.+ """+ owner, restricted = rls_harness.owner, rls_harness.restricted+ tenant_a, tenant_b = (f"tenant-{uuid.uuid4().hex[:8]}" for _ in range(2))+ async with owner() as session:+ session.add_all(+ [Tenant(id=t, slug=t, name=t) for t in (tenant_a, tenant_b)]+ )+ await session.flush()+ suffix = uuid.uuid4().hex[:8]+ a_client = Client(+ tenant_id=tenant_a,+ transport_user_id=f"@a-{suffix}:test",+ display_name="a",+ type="agent",+ )+ b_client = Client(+ tenant_id=tenant_b,+ transport_user_id=f"@b-{suffix}:test",+ display_name="b",+ type="agent",+ )+ room = Room(+ tenant_id=tenant_a,+ transport_room_id=f"!room-{suffix}:test",+ name="a's room",+ description="",+ )+ session.add_all([a_client, b_client, room])+ await session.flush()+ session.add(+ ClientRoom(tenant_id=tenant_a, client_id=a_client.id, room_id=room.id)+ )+ await session.commit()+ room_id, transport_room_id = room.id, room.transport_room_id+ a_ids = (a_client.id, a_client.transport_user_id)+ b_ids = (b_client.id, b_client.transport_user_id)++ cache = _cache(restricted)+ listener = _FakeListener()+ a = _with(+ _transport(+ restricted,+ client_id=a_ids[0],+ user_id=a_ids[1],+ tenant_id=tenant_a,+ listener=listener,+ ),+ cache,+ )+ b = _with(+ _transport(+ restricted,+ client_id=b_ids[0],+ user_id=b_ids[1],+ tenant_id=tenant_b,+ listener=listener,+ ),+ cache,+ )+ a_got, b_got = _Received(), _Received()+ a.register_handlers(a_got.handlers())+ b.register_handlers(b_got.handlers())+ tasks = [+ asyncio.create_task(a.receive_forever()),+ asyncio.create_task(b.receive_forever()),+ ]+ try:+ await _watched_room(a)+ await a.send_message(+ transport_room_id, "for tenant a", sender_name="a", metered=False+ )+ await listener.announce(room_id)+ assert [e.body for e in a_got.events] == ["for tenant a"] # type: ignore[attr-defined]+ assert cache.stats().rooms == 1++ # The forced subscription: what `_watch` would record, from 0.+ b._room_ids[transport_room_id] = room_id+ b._watching[room_id] = transport_room_id+ b._cursors[room_id] = 0+ cache.attach(tenant_b, room_id, b)+ b._mark_pending(room_id)+ await _settled(b)++ assert b_got.events == []+ assert cache.stats().rooms == 2, "B shared A's entry"+ async with tenant_session(restricted, tenant_a) as session:+ rows = await MessageStore().list_for_room(+ session, room_id, after_seq=0, limit=10+ )+ assert [row.body for row in rows] == ["for tenant a"]+ finally:+ for task in tasks:+ task.cancel()+++class TestTheListenerIsNeverHeldUp:+ async def test_the_fan_out_finishes_while_a_fill_is_blocked(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ monkeypatch: pytest.MonkeyPatch,+ ) -> None:+ """The single fan-out task wakes every room; a fill must not be in it.++ The room's read is held. The fan-out still wakes both members, moves+ on to the next room, and drains its queue.+ """+ listener = MessageListener(lambda: None) # type: ignore[arg-type,return-value]+ listener._running = True+ fan_out = asyncio.create_task(listener._fan_out_forever())+ cache = _cache(session_factory)+ tasks: list[asyncio.Task[None]] = []+ try:+ async with session_factory() as session:+ room_id, room, first_id, first_user = await _make_room(session)+ second_id, second_user = await _make_client(session, "second")+ for client_id in (first_id, second_id):+ await RoomStore().add_client(session, client_id, room_id)+ await session.commit()+ members = [+ _with(+ _transport(+ session_factory,+ client_id=client_id,+ user_id=user_id,+ listener=listener, # type: ignore[arg-type]+ ),+ cache,+ )+ for client_id, user_id in (+ (first_id, first_user),+ (second_id, second_user),+ )+ ]+ for member in members:+ member.register_handlers(_Received().handlers())+ tasks.append(asyncio.create_task(member.receive_forever()))+ for member in members:+ await _watched_room(member)+ for member in members:+ await _settled(member)++ gate = asyncio.Event()+ fills = 0+ original = MessageStore.list_for_room++ async def held(store, session, room, **kwargs): # type: ignore[no-untyped-def]+ nonlocal fills+ fills += 1+ await gate.wait()+ return await original(store, session, room, **kwargs)++ monkeypatch.setattr(MessageStore, "list_for_room", held)+ other_room_woken = asyncio.Event()++ async def other_room(_room_id: str) -> None:+ other_room_woken.set()++ listener.subscribe("some-other-room", other_room)+ await asyncio.wait_for(other_room_woken.wait(), DELIVERY_TIMEOUT)+ other_room_woken.clear()++ # Wake the room; its one shared read starts and is held.+ listener._mark(room_id, 1)+ for _ in range(500):+ if fills and all(m._delivering for m in members):+ break+ await asyncio.sleep(0.002)+ assert fills == 1, "both members should be waiting on one read"++ # With that read still held, the fan-out goes on waking rooms.+ listener._mark("some-other-room", 1)+ await asyncio.wait_for(other_room_woken.wait(), DELIVERY_TIMEOUT)+ assert not gate.is_set()+ assert fills == 1+ assert listener._pending == {}+ assert all(m._delivering for m in members)+ assert not fan_out.done()+ gate.set()+ for member in members:+ await _settled(member)+ finally:+ for task in tasks:+ task.cancel()+ fan_out.cancel()diff --git a/deploy/observability/README.md b/deploy/observability/README.mdindex c53391e6c..4d6adf18e 100644--- a/deploy/observability/README.md+++ b/deploy/observability/README.md@@ -45,7 +45,9 @@ done## Check these before you trust the dashboard**Histogram panels need two things, not one.** Request latency, delivery lag,-database query duration and platform call duration are OTLP histograms, and the+database query duration, pool hold and wait time (`switch.db.pool.hold.duration`,+`switch.db.pool.wait.duration`), platform call duration and the agent buffer's+scan length and duration are OTLP histograms, and the`p95:`/`p99:` queries here assume the collector exports them to Datadog as*distributions*. Check the collector'shistogram mode — in the older `histograms` mode they arrive as separate@@ -55,9 +57,10 @@ Then check the second thing, which is easy to miss because it is on Datadog'sside rather than the collector's: **percentile aggregations are off by defaulton a distribution metric and are billed separately.** Enable them per metric inMetrics Summary, and add the tag each panel groups by to that metric's-configured tag set — `route` for requests, `operation` for database queries,-`bridge` and `platform` for bridge calls — or `p95: … by {…}` returns nothing on a fresh-account. Empty is the honest outcome either way; a panel is never silently+configured tag set (`route` for requests, `operation` for database queries and+the agent buffer, `caller` for pool hold time but not wait time, `bridge` and+`platform` for bridge calls), or+`p95: … by {…}` returns nothing on a fresh account. Empty is the honest outcome either way; a panel is never silentlyswitched to a different statistic.The two latency **monitors** depend on the same thing, and fail more quietlydiff --git a/deploy/observability/dashboard.json b/deploy/observability/dashboard.jsonindex 51d5ed459..2a07d7448 100644--- a/deploy/observability/dashboard.json+++ b/deploy/observability/dashboard.json@@ -71,7 +71,7 @@"title": "Agents connected","requests": [{- "q": "avg:switch.agents.connected{$env,$service}",+ "q": "sum:switch.agents.connected{$env,$service}","aggregator": "last"}],@@ -714,6 +714,323 @@"width": 12,"height": 4}+ },+ {+ "definition": {+ "type": "timeseries",+ "title": "Agents connected, by transport and client",+ "show_legend": true,+ "requests": [+ {+ "q": "sum:switch.agents.connected{$env,$service} by {transport,client}",+ "display_type": "line"+ }+ ]+ },+ "layout": {+ "x": 0,+ "y": 67,+ "width": 6,+ "height": 4+ }+ },+ {+ "definition": {+ "type": "timeseries",+ "title": "Agent connections refused per second, by reason",+ "show_legend": true,+ "requests": [+ {+ "q": "sum:switch.agent.connections_refused{$env,$service} by {reason}.as_rate()",+ "display_type": "bars",+ "style": {+ "palette": "warm"+ }+ }+ ]+ },+ "layout": {+ "x": 6,+ "y": 67,+ "width": 6,+ "height": 4+ }+ },+ {+ "definition": {+ "type": "timeseries",+ "title": "Database sessions for this server's role, by state (the database's view)",+ "show_legend": true,+ "requests": [+ {+ "q": "max:switch.db.server.connections{$env,$service} by {state}",+ "display_type": "area"+ }+ ]+ },+ "layout": {+ "x": 0,+ "y": 71,+ "width": 6,+ "height": 4+ }+ },+ {+ "definition": {+ "type": "timeseries",+ "title": "Database transactions per second (the database's view)",+ "show_legend": false,+ "requests": [+ {+ "q": "avg:switch.db.server.transactions{$env,$service}.as_rate()",+ "display_type": "line"+ }+ ]+ },+ "layout": {+ "x": 6,+ "y": 71,+ "width": 6,+ "height": 4+ }+ },+ {+ "definition": {+ "type": "timeseries",+ "title": "Agent buffer events walked per scan, p50 / p95 by operation (cap 2000), needs distribution mapping, see README",+ "show_legend": true,+ "requests": [+ {+ "q": "p50:switch.agent.buffer.scanned{$env,$service} by {operation}",+ "display_type": "line"+ },+ {+ "q": "p95:switch.agent.buffer.scanned{$env,$service} by {operation}",+ "display_type": "line"+ }+ ]+ },+ "layout": {+ "x": 0,+ "y": 75,+ "width": 6,+ "height": 4+ }+ },+ {+ "definition": {+ "type": "timeseries",+ "title": "Agent buffer scan duration p95 / p99 (ms) by operation, needs distribution mapping, see README",+ "show_legend": true,+ "requests": [+ {+ "q": "p95:switch.agent.buffer.scan.duration{$env,$service} by {operation}",+ "display_type": "line"+ },+ {+ "q": "p99:switch.agent.buffer.scan.duration{$env,$service} by {operation}",+ "display_type": "line"+ }+ ]+ },+ "layout": {+ "x": 6,+ "y": 75,+ "width": 6,+ "height": 4+ }+ },+ {+ "definition": {+ "type": "timeseries",+ "title": "Database pool: connection hold p95 / p99 (ms) by caller. Needs distribution mapping, see README",+ "show_legend": true,+ "requests": [+ {+ "q": "p95:switch.db.pool.hold.duration{$env,$service} by {caller}",+ "display_type": "line"+ },+ {+ "q": "p99:switch.db.pool.hold.duration{$env,$service} by {caller}",+ "display_type": "line"+ }+ ]+ },+ "layout": {+ "x": 0,+ "y": 79,+ "width": 6,+ "height": 4+ }+ },+ {+ "definition": {+ "type": "timeseries",+ "title": "Database pool: wait for a connection p95 / p99 (ms). Needs distribution mapping, see README",+ "show_legend": true,+ "requests": [+ {+ "q": "p95:switch.db.pool.wait.duration{$env,$service}",+ "display_type": "line"+ },+ {+ "q": "p99:switch.db.pool.wait.duration{$env,$service}",+ "display_type": "line"+ }+ ]+ },+ "layout": {+ "x": 6,+ "y": 79,+ "width": 6,+ "height": 4+ }+ },+ {+ "definition": {+ "type": "timeseries",+ "title": "Agent connections opened per second, by kind (after_lapse = returning from a lapse)",+ "show_legend": true,+ "requests": [+ {+ "q": "sum:switch.agent.connections_opened{$env,$service} by {kind}.as_rate()",+ "display_type": "bars"+ }+ ]+ },+ "layout": {+ "x": 0,+ "y": 83,+ "width": 6,+ "height": 4+ }+ },+ {+ "definition": {+ "type": "timeseries",+ "title": "Process starts (deploys and restarts)",+ "show_legend": false,+ "requests": [+ {+ "q": "sum:switch.runtime.starts{$env,$service}.as_count()",+ "display_type": "bars"+ }+ ]+ },+ "layout": {+ "x": 6,+ "y": 83,+ "width": 6,+ "height": 4+ }+ },+ {+ "definition": {+ "type": "note",+ "content": "**Shared room reads.** Each fill is one database read standing in for one per member of the room, so fills against deliveries is the saving. A fallback (behind, evicted, unwatched, gave_up) is a transport reading the database itself: it costs a read, never a message.",+ "background_color": "gray",+ "font_size": "14",+ "text_align": "left",+ "vertical_align": "top",+ "show_tick": false,+ "tick_pos": "50%",+ "tick_edge": "left",+ "has_padding": true+ },+ "layout": {+ "x": 0,+ "y": 87,+ "width": 12,+ "height": 2+ }+ },+ {+ "definition": {+ "type": "timeseries",+ "title": "Room cache reads per second, by outcome (hit and filled are served from memory)",+ "show_legend": true,+ "requests": [+ {+ "q": "sum:switch.delivery_cache.reads{$env,$service} by {outcome}.as_rate()",+ "display_type": "bars"+ }+ ]+ },+ "layout": {+ "x": 0,+ "y": 89,+ "width": 6,+ "height": 4+ }+ },+ {+ "definition": {+ "type": "timeseries",+ "title": "Room cache fills and rows read per second, against messages delivered",+ "show_legend": true,+ "requests": [+ {+ "q": "sum:switch.delivery_cache.fills{$env,$service} by {outcome}.as_rate()",+ "display_type": "line"+ },+ {+ "q": "sum:switch.delivery_cache.rows_read{$env,$service}.as_rate()",+ "display_type": "line"+ },+ {+ "q": "sum:switch.messages.delivered{$env,$service}.as_rate()",+ "display_type": "line"+ }+ ]+ },+ "layout": {+ "x": 6,+ "y": 89,+ "width": 6,+ "height": 4+ }+ },+ {+ "definition": {+ "type": "timeseries",+ "title": "Room cache memory (bytes) and rooms held",+ "show_legend": true,+ "requests": [+ {+ "q": "max:switch.delivery_cache.bytes{$env,$service}",+ "display_type": "line"+ },+ {+ "q": "max:switch.delivery_cache.rooms{$env,$service}",+ "display_type": "line"+ }+ ]+ },+ "layout": {+ "x": 0,+ "y": 93,+ "width": 6,+ "height": 4+ }+ },+ {+ "definition": {+ "type": "timeseries",+ "title": "Room cache evictions per second, by reason",+ "show_legend": true,+ "requests": [+ {+ "q": "sum:switch.delivery_cache.evictions{$env,$service} by {reason}.as_rate()",+ "display_type": "bars"+ }+ ]+ },+ "layout": {+ "x": 6,+ "y": 93,+ "width": 6,+ "height": 4+ }}]}diff --git a/docs/design/agent-controllers-v1.md b/docs/design/agent-controllers-v1.mdindex 2edbd418c..88ba97923 100644--- a/docs/design/agent-controllers-v1.md+++ b/docs/design/agent-controllers-v1.md@@ -386,10 +386,9 @@ stream. The flag and everything else above stay as they are.### Agent hosts in the controller's process, and the local relay- Each agent's definition says where its agent host runs: `isolation: "shared"` (the default)- in the controller's process, as below; `isolated` in a process of its own, served the- per-agent agent protocol by the relay (its own event stream, heartbeat, placements and room- claims), as before agent hosts moved in-process. Cloud machines run every agent isolated, as- a systemd unit. Changing an agent's isolation restarts it the other way.+ in the controller's process, as below; `isolated` in a process of its own, which reaches+ the same hub over a WebSocket on the relay's port (`/hub`). Cloud machines run every agent+ isolated, as a systemd unit. Changing an agent's isolation restarts it the other way.- One upstream stream. Each agent's agent host runs inside the controller's process and is handedits events from that stream directly (`AgentHub`): in order, filtered to what addresses theagent, with gaps, resets, room controls and approval outcomes. Events that arrive while its@@ -397,13 +396,15 @@ stream. The flag and everything else above stay as they are.event. The agent host states its sessions' rooms in memory, for the relay and for routingsession commands; none of it is sent upstream.- The shared agent host code (`runAgentHost`) takes the function that opens its event stream: the- controller passes its hub, while Console and the shared daemon pass the agent host's own- connection to Switch.+ controller passes its hub, an isolated agent host the hub over its WebSocket+ (`openHubStream`, chosen by `SWITCH_AGENT_HUB` in its credentials), while Console passes the+ agent host's own connection to Switch. Only the controller holds the agent protocol's+ connection to Switch; nothing on the machine imitates it.- A loopback HTTP relay (`127.0.0.1`, a per-agent bearer token minted locally) is what eachagent host uses as `SWITCH_API_ENDPOINT` for its calls to Switch. It forwards everything with thecontroller access token, the `X-Switch-Agent-Id` header, and `X-Switch-Room-Id` resolved from- the placements. It serves no event stream and no connection bookkeeping. The credentials file- names the relay and its local token, never a Switch credential.+ the placements. It serves no event stream and no connection bookkeeping of its own. The+ credentials file names the relay, its hub and its local token, never a Switch credential.### Core implementation notes (decisions the spec left open)diff --git a/docs/design/controller-contract-v1.md b/docs/design/controller-contract-v1.mdindex 2ca401772..45727b353 100644--- a/docs/design/controller-contract-v1.md+++ b/docs/design/controller-contract-v1.md@@ -7,10 +7,11 @@ Step 1 of the roadmap turns this into OpenAPI and JSON Schema files plus recorde## 0. Conventions-- **Transport:** HTTPS with JSON bodies (`application/json`), and one SSE stream (`text/event-stream`).+- **Transport:** HTTPS with JSON bodies (`application/json`), and one WebSocket per controller carrying JSON frames.- **Base paths:** Management is `/v1/management/...`. The Core controller surface is `/v1/controllers/...`. Agent actions use the existing `/agents/{agent_id}/...` routes.-- **Version:** every request sends `Switch-Controller-Protocol: 1`.- - The server answers with the range it supports in `Switch-Controller-Protocol-Accepts: 1-1`.+- **Version:** every request, and the socket's opening request, sends `Switch-Controller-Protocol: 2`.+ - The server answers with the range it supports in `Switch-Controller-Protocol-Accepts: 2-2`.+ - Protocol 2 moved the stream and the beat onto one WebSocket (§6). Protocol 1 (an SSE stream and a beat POST) is no longer accepted.- An unsupported version returns `426` with code `protocol_unsupported`.- **Forward compatibility:** receivers ignore unknown fields. An unknown enum value is read as `unknown`, never as an error.- **Time:** RFC 3339 UTC strings. **IDs:** opaque strings.@@ -222,22 +223,26 @@ POST /v1/controllers/{id}/connection→ 201 { connection_id: string, generation: number, heartbeat_interval_s: number,attached: string[] } // agents Core attached to this connection-GET /v1/controllers/{id}/events?connection_id=…&generation=…- Accept: text/event-stream- → SSE stream (frames below). The server sends ": ping" every 15s--POST /v1/controllers/{id}/connection/beat- { connection_id: string, generation: number, cursors: Record<agent_id, number> }- → 200 { attached: string[] }- → 409 { code: "taken_over" | "stale_generation" } // taken_over is terminal for this client- → 404 { code: "unknown_connection" } // reopen+WS /v1/controllers/{id}/connection/ws?connection_id=…&generation=…+ Authorization: Bearer <access token>+ ← { event: string, data: object } // frames below, one JSON text message each+ ← { event: "ping", data: {} } // every heartbeat_interval_s+ → { type: "pong", cursors: Record<agent_id, number> }+ // the beat: alive, and how far each agent has read+ ← { event: "refused", data: { status, detail: { code, message } } }, then close 4000 + status+ // 404 unknown_connection, 409 taken_over | stale_generation,+ // 401 controller_revoked | token_expired, 426 protocol_unsupported+ ← { event: "evicted", data: { code, reason } }, then close+ // a pong refused: taken_over is terminal for this client,+ // the rest (no_stream, unknown_connection, …) reopen+ close 1012 // the server is restarting: reconnect within a second```**Attachment is automatic.** Core attaches the agents currently bound to this controller (§7), and attaches or detaches them as bindings change. There's no per-agent subscribe call.-**Core knows only whether an agent is connected.** A controller-backed agent is connected while its controller's stream is attached and beating, its owner has it `running`, and the controller is not revoked. Core does not know where the agent's sessions are, or whether one is running: the open and the beat carry no session map, and a body that still sends `placements` is refused with `422 validation_error` (the one exception to ignoring unknown fields, so a controller still reporting sessions is noticed rather than silently dropped). When a connected agent is addressed, Core delivers the message and posts nothing; if a session has to start, the agent's host posts "Starting a session…" itself. When it is not connected, Core tells the room why: stopped by its owner, its machine removed, or its machine offline.+**Core knows only whether an agent is connected.** A controller-backed agent is connected while its controller's socket is attached and answering pings, its owner has it `running`, and the controller is not revoked. Core does not know where the agent's sessions are, or whether one is running: the open and the pong carry no session map, and a body that still sends `placements` is refused with `422 validation_error` (the one exception to ignoring unknown fields, so a controller still reporting sessions is noticed rather than silently dropped). When a connected agent is addressed, Core delivers the message and posts nothing; if a session has to start, the agent's host posts "Starting a session…" itself. When it is not connected, Core tells the room why: stopped by its owner, its machine removed, or its machine offline.-**SSE frames.** `event:` is the type and `data:` is JSON.+**Frames.** `event` is the type and `data` is the JSON object below without its `type`.```tstype StreamEvent =