Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
Limit connection opens per pod, so agents reconnecting after a pod loss don't use up the database pool (CHOO-3667) (#1139)
Abel Dantas6d5af7d16c22
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 0763b72ce..8b9791612 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -85,6 +85,18 @@ version of their own to them without also giving them a release of their own.Now a room the runner cannot read reads as missing.#### Added+- **A reconnect storm is paced instead of taking the pool.** When a pod goes+ away, every agent and controller it held opens a connection on the pods+ left within seconds. On staging (two pods, 1,000 agents, one pod deleted)+ those opens took the whole database pool: requests and the process lease+ timed out with them, and the surviving pod dropped agents of its own. Each+ process now opens at most `AGENT_OPEN_LIMIT` (8) agent and controller+ connections at once. An open that finds no slot within 2 s, which keeps it+ well inside the timeout of the proxy in front, is refused before+ authentication with 503 and `retry_after_s`, its place in the line at the+ pace opens are taking in that process, so the storm comes back as a queue+ spread over what the process can take. A caller that comes back early keeps+ its time. `switch.agent.opens_deferred` counts the refusals, by route.- **The Helm chart can put switch-core on chosen nodes.**`switchCore.nodeSelector` and `switchCore.tolerations` go onto theswitch-core pods as written, so they can run on a node group of their own,@@ -4191,6 +4203,10 @@ tags; see RELEASING.md.### [Unreleased]#### Added+- **Waits its turn when Switch is busy opening connections.** An open or an+ attach refused as `busy` comes with when to try again, the controller's+ place in the server's queue, and the controller now waits that long+ instead of its own backoff.- **A provider report says when signing in would not help.** A providerwhose check says its login command would not fix the problem is reportedwith `login_helps: false` beside `auth_problem`, so Console does not offer@@ -4292,6 +4308,10 @@ The Switch protocol client and MCP runtime### [Unreleased]#### Added+- **Waits its turn when Switch is busy opening connections.** A server+ pacing a reconnect storm refuses an open as `busy` with when to open again,+ which is this agent's place in its queue. The stream now waits that long,+ even right after a restart, instead of retrying on its own schedule.- When the WebSocket never opens and the runtime asks for the event streaminstead, the request carries `fallback=socket_never_opened` and thesocket's close code as `fallback_close_code`, so the server's log can telldiff --git a/console/packages/agent-controller/src/stream.test.ts b/console/packages/agent-controller/src/stream.test.tsindex 15a7d74cc..34660e475 100644--- a/console/packages/agent-controller/src/stream.test.ts+++ b/console/packages/agent-controller/src/stream.test.ts@@ -396,6 +396,26 @@ describe('runControllerStream', () => {expect(opens).toBe(3);});+ it('opens again when a busy server says, not on its own backoff', async () => {+ const scripted = new ScriptedClient();+ let opens = 0;+ scripted.open = async () => {+ opens++;+ if (opens === 1) throw new ControllerApiError(503, 'busy', 'opening at capacity', true, 0.3);+ return opened;+ };+ scripted.sockets = () => scriptedSocket([state], 'open');+ const stop = new AbortController();+ const started = Date.now();+ const { states, ending } = run(scripted, stop.signal);+ await waitFor(() => states.includes('connected'), 'a connection', 3_000);+ stop.abort();+ await ending;+ expect(opens).toBe(2);+ // The backoff alone would have opened again after 5 ms.+ expect(Date.now() - started).toBeGreaterThanOrEqual(290);+ });+it('waits less each time a socket attached, however briefly, and never long', async () => {const scripted = new ScriptedClient();let opens = 0;diff --git a/console/packages/agent-controller/src/stream.ts b/console/packages/agent-controller/src/stream.tsindex 841e1291d..972133bcb 100644--- a/console/packages/agent-controller/src/stream.ts+++ b/console/packages/agent-controller/src/stream.ts@@ -111,6 +111,8 @@ export type ControllerStreamOptions = {* are short and do not grow.*/const SERVICE_RESTART = 1012;+/** The refusal of a server opening as many connections as it can take at once. */+const BUSY = 'busy';const RESTART_RETRY_MS = 250;const RESTART_RETRY_JITTER_MS = 500;@@ -192,9 +194,18 @@ class ControllerStream {const stop = () => attempt.abort.abort();signal.addEventListener('abort', stop, { once: true });let failure: string | null = null;+ // When a busy server said to come back (core's `open_gate.py`): that is+ // this controller's place in its queue, so it outranks the backoff.+ let toldMs: number | null = null;try {await this.attach(attempt);} catch (error) {+ if (+ error instanceof ControllerApiError &&+ error.code === BUSY &&+ error.retryAfterS !== null+ )+ toldMs = Math.round(error.retryAfterS * 1000);if (!attempt.ending && !signal.aborted) {if (isRevoked(error)) attempt.ending = 'revoked';else if (isTakenOver(error)) attempt.ending = 'taken_over';@@ -217,16 +228,20 @@ class ControllerStream {// so a socket that attached at all starts the waits over.this.backoff = this.options.initialBackoffMs;}- const wait = attempt.restarting- ? RESTART_RETRY_MS + Math.round(this.options.random() * RESTART_RETRY_JITTER_MS)- : Math.round(this.backoff * (0.5 + this.options.random() * 0.5));+ const wait =+ toldMs !== null+ ? toldMs+ : attempt.restarting+ ? RESTART_RETRY_MS + Math.round(this.options.random() * RESTART_RETRY_JITTER_MS)+ : Math.round(this.backoff * (0.5 + this.options.random() * 0.5));if (failure !== null)log.warn('The controller socket is down; reconnecting.', {error: failure,retryInMs: wait,});await delay(wait, undefined, { signal }).catch(() => {});- if (!attempt.restarting) this.backoff = Math.min(this.backoff * 2, this.options.maxBackoffMs);+ if (!attempt.restarting && toldMs === null)+ this.backoff = Math.min(this.backoff * 2, this.options.maxBackoffMs);}return 'stopped';}@@ -264,7 +279,7 @@ class ControllerStream {const { socket } = opened;const abort = attempt.abort;const pending: Message[] = [];- let refused: { status: number; code: string; message: string } | null = null;+ let refused: Refusal | null = null;let closedWith: number | null = null;let wake: (() => void) | null = null;const notify = () => {@@ -336,10 +351,16 @@ class ControllerStream {wake = resolve;});}- const refusal = refused as { status: number; code: string; message: string } | null;+ const refusal = refused as Refusal | null;if (refusal !== null) {if (refusal.status === 401 && refusal.code !== 'controller_revoked') opened.tokenRefused();- throw new ControllerApiError(refusal.status, refusal.code, refusal.message, false, null);+ throw new ControllerApiError(+ refusal.status,+ refusal.code,+ refusal.message,+ refusal.retryAfterS !== null,+ refusal.retryAfterS+ );}if (abort.signal.aborted && abort.signal.reason instanceof IdleTimeout)throw abort.signal.reason;@@ -380,14 +401,22 @@ class ControllerStream {}}-function refusalOf(data: unknown): { status: number; code: string; message: string } {+type Refusal = { status: number; code: string; message: string; retryAfterS: number | null };++function refusalOf(data: unknown): Refusal {const body = (data ?? {}) as { status?: unknown; detail?: unknown };- const detail = (body.detail ?? {}) as { code?: unknown; message?: unknown };+ const detail = (body.detail ?? {}) as {+ code?: unknown;+ message?: unknown;+ retry_after_s?: unknown;+ };+ const after = detail.retry_after_s;return {status: typeof body.status === 'number' ? body.status : 0,code: typeof detail.code === 'string' ? detail.code : 'refused',message:typeof detail.message === 'string' ? detail.message : 'Switch refused the controller socket.',+ retryAfterS: typeof after === 'number' && Number.isFinite(after) && after >= 0 ? after : null,};}diff --git a/console/packages/switch-agent-runtime/src/event-stream.test.ts b/console/packages/switch-agent-runtime/src/event-stream.test.tsindex fac1e1877..d3773816e 100644--- a/console/packages/switch-agent-runtime/src/event-stream.test.ts+++ b/console/packages/switch-agent-runtime/src/event-stream.test.ts@@ -510,6 +510,64 @@ describe('a server restart', () => {});});+describe('a busy server', () => {+ it('opens again when the server says, not on its own backoff', async () => {+ vi.useFakeTimers();+ serve((socket, index) => {+ if (index === 0) {+ socket.refuse(503, { code: 'busy', message: 'opening at capacity', retry_after_s: 12 });+ return;+ }+ attach(socket, 0);+ });+ const { abort } = makeStream({ rooms: [] });++ // The backoff alone would have opened again after 1 s.+ await vi.advanceTimersByTimeAsync(11_900);+ expect(server.sockets).toHaveLength(1);+ await vi.advanceTimersByTimeAsync(200);+ expect(server.sockets).toHaveLength(2);+ abort.abort();+ });++ it('stops at once when stopped while waiting its turn', async () => {+ vi.useFakeTimers();+ serve((socket) => {+ socket.refuse(503, { code: 'busy', message: 'opening at capacity', retry_after_s: 300 });+ });+ const { abort } = makeStream({ rooms: [] });+ await vi.advanceTimersByTimeAsync(1_000);+ expect(server.sockets).toHaveLength(1);++ abort.abort();+ await vi.advanceTimersByTimeAsync(0);+ // Nothing is left waiting out the 300 s, to keep the process alive.+ expect(vi.getTimerCount()).toBe(0);+ });++ it('waits its turn even right after a restart', async () => {+ vi.useFakeTimers();+ serve((socket, index) => {+ if (index === 0) {+ attach(socket, 0);+ queueMicrotask(() => socket.drop(1012, 'service restart'));+ } else if (index === 1) {+ socket.refuse(503, { code: 'busy', message: 'opening at capacity', retry_after_s: 30 });+ } else {+ attach(socket, 0);+ }+ });+ const { abort } = makeStream({ rooms: [] });++ // A restart alone retries within a second; the queue's time wins.+ await vi.advanceTimersByTimeAsync(20_000);+ expect(server.sockets).toHaveLength(2);+ await vi.advanceTimersByTimeAsync(12_000);+ expect(server.sockets).toHaveLength(3);+ abort.abort();+ });+});+describe('reopening while an open is still in flight', () => {/*** A server that keeps the incarnation the way the real one does: every opendiff --git a/console/packages/switch-agent-runtime/src/event-stream.ts b/console/packages/switch-agent-runtime/src/event-stream.tsindex f90a33057..daf68b402 100644--- a/console/packages/switch-agent-runtime/src/event-stream.ts+++ b/console/packages/switch-agent-runtime/src/event-stream.ts@@ -74,6 +74,14 @@ const SERVICE_RESTART = 1012;const RESTART_WINDOW_MS = 60_000;const RESTART_RETRY_MIN_MS = 250;const RESTART_RETRY_SPREAD_MS = 500;+/**+ * The refusal of a server that is opening as many connections as it can take+ * at once (core's `open_gate.py`). It names when to open again, which is this+ * client's place in the server's queue, so that is when the next open goes,+ * whatever the backoff or a restart would have said.+ */+const BUSY = 'busy';+/*** How long an open socket has to last before it counts as a working stream.*@@ -169,6 +177,31 @@ export function refusalCode(body: string): string | null {}}+/** The wait a `busy` refusal names, in milliseconds, or null. */+export function busyRetryAfterMs(status: number, body: string): number | null {+ if (status !== 503 || refusalCode(body) !== BUSY) return null;+ try {+ const detail = (JSON.parse(body) as { detail?: { retry_after_s?: unknown } }).detail;+ const seconds = detail?.retry_after_s;+ return typeof seconds === 'number' && Number.isFinite(seconds) && seconds >= 0+ ? Math.round(seconds * 1000)+ : null;+ } catch {+ return null;+ }+}++/** Waits `ms`, or less if any of these signals aborts first, and leaves no timer behind. */+function waitUnless(ms: number, ...signals: AbortSignal[]): Promise<void> {+ let timer: ReturnType<typeof setTimeout> | undefined;+ return Promise.race([+ new Promise<void>((resolve) => {+ timer = setTimeout(resolve, ms);+ }),+ until(...signals),+ ]).finally(() => clearTimeout(timer));+}+/** Resolves when any of these signals aborts, so a wait can be given up on. */export function until(...signals: AbortSignal[]): Promise<void> {const any = AbortSignal.any(signals);@@ -819,6 +852,8 @@ export class SwitchEventStream {if (signal.aborted || this.halt.signal.aborted) return;let openedAt = 0;let failure: unknown = null;+ // The wait a busy server named for this open, if it was refused as busy.+ let toldMs: number | null = null;try {const params = new URLSearchParams({connection_id: connectionId,@@ -868,6 +903,7 @@ export class SwitchEventStream {} catch (error) {if (error instanceof OpenRefused) {const { status, body } = error;+ toldMs = busyRetryAfterMs(status, body);// Reopening without the refused room is a different request from the// one that just failed, and the declared set strictly shrinks, so// this cannot spin: retry now rather than serving the backoff a@@ -922,7 +958,8 @@ export class SwitchEventStream {});}this.deps.onDisconnected?.({ error });- await pace();+ if (toldMs !== null) await waitUnless(toldMs, signal, this.halt.signal);+ else await pace();}}diff --git a/core/switch_core/bridges/agent/app.py b/core/switch_core/bridges/agent/app.pyindex 590fe0baf..7a22cd167 100644--- a/core/switch_core/bridges/agent/app.py+++ b/core/switch_core/bridges/agent/app.py@@ -39,6 +39,7 @@ from switch_core.db.stores.room_store import RoomStorefrom switch_core.gateway.ui import GatewayUi, GatewayUiMiddlewarefrom switch_core.logging_context import log_contextfrom switch_core.observability.http import MetricsMiddleware+from switch_core.open_gate import OpenGatefrom switch_core.refusal_breaker import RefusalBreakerfrom switch_core.request_context import RequestContextMiddlewarefrom switch_core.request_validation import public_errors@@ -192,6 +193,9 @@ def create_agent_bridge_app(session_factory=session_factory, # type: ignore[arg-type]controller_auth=controller_auth,)+ # Outside the bearer middleware, so an open the process cannot take yet is+ # told when to come back before its credential is looked up.+ app.add_middleware(OpenGate, limit=config.agent_open_limit, clock=time.monotonic)# Outside the bearer middleware, so a caller that keeps repeating a refused# request is answered before its credential is looked up again.app.add_middleware(RefusalBreaker, clock=time.monotonic)diff --git a/core/switch_core/config.py b/core/switch_core/config.pyindex 8fcb890b8..dee21b287 100644--- a/core/switch_core/config.py+++ b/core/switch_core/config.py@@ -751,6 +751,17 @@ class SwitchConfig(BaseSettings):# SQLAlchemy's default is 30s, long enough that pool starvation surfaces as# unexplained latency rather than an error. Fail fast and loudly instead.db_pool_timeout: float = 5.0+ # How many agent and controller connections this process opens at once+ # (`open_gate.py`). Opening one takes dozens of transactions from the same+ # pool the process's requests and its lease renewal use, and when a pod+ # goes away every agent it held opens on the pods left within seconds.+ # Past this many, an open waits briefly for a slot and is then told when to+ # come back, in the order it arrived, at the pace opens are taking here,+ # instead of queueing on the pool. 8 is what the pod-kill tests ran with:+ # with 300 agents on two pods and 30 + 10 pools, no pool timeouts and no+ # agent lost, against 161 pool timeouts and 56 agents lost without the+ # gate (dev-abel, 28 Oct 2026). 0 turns it off.+ agent_open_limit: int = Field(default=8, ge=0)# Resolving a bearer token to its agent is the highest-frequency query in# the system — it runs before every authenticated request, and a fleet ofdiff --git a/core/switch_core/observability/catalogue.py b/core/switch_core/observability/catalogue.pyindex 60f934816..5032e7180 100644--- a/core/switch_core/observability/catalogue.py+++ b/core/switch_core/observability/catalogue.py@@ -475,6 +475,18 @@ AGENT_CONNECTIONS_REFUSED = _spec("revision in common), unauthorized (no valid credential), other.","reason",)+# An agent or controller connection told to open again later because this+# process was already opening as many as it takes at once (`open_gate.py`).+# A burst of these is a reconnect storm being paced, after a pod went away.+AGENT_OPENS_DEFERRED = _spec(+ "switch.agent.opens_deferred",+ "sum",+ "{connection}",+ "Agent and controller connection opens refused with a time to come back, "+ "because this process was opening as many as it takes at once. By route: "+ "agent_socket, agent_event_stream, controller_open, controller_socket.",+ "route",+)AGENT_CONNECTIONS_EXPIRED = _spec("switch.agent.connections_expired","sum",@@ -663,6 +675,7 @@ CATALOGUE: dict[str, MetricSpec] = {AGENT_BUFFER_SCAN_DURATION,AGENT_CONNECTIONS_REFUSED,AGENT_CONNECTIONS_EXPIRED,+ AGENT_OPENS_DEFERRED,AGENT_CONNECTIONS_OPENED,AGENTS_CONNECTED,CONSUMERS_RUNNING,diff --git a/core/switch_core/open_gate.py b/core/switch_core/open_gate.pynew file mode 100644index 000000000..ede82e8d0--- /dev/null+++ b/core/switch_core/open_gate.py@@ -0,0 +1,340 @@+"""Paces connection opens, so a reconnect storm becomes a queue.++When a pod goes away, every agent and controller it held opens a connection on+the pods that are left, all within a few seconds. Each open authenticates,+makes a new incarnation of the connection and reads where the stream starts:+dozens of transactions, on the same pool the process's requests and its own+lease renewal use. Measured on staging (two pods, 1,000 agents, one pod+deleted), the opens took the whole pool: thousands of pool timeouts, opens+that failed after their incarnation was made, and the surviving pod's lease+lapsing, which made it drop agents of its own.++This middleware runs at most `limit` opens at once in this process. An open+that finds every slot taken waits up to `WAIT_S` for one. Past that, or when+more opens are waiting already than can get a slot within `WAIT_S`, it is+refused before authentication or the database, with `retry_after_s`: its place+in the line. The line moves one slot every median open time over `limit` (the+median of the last `limit` opens), so a storm comes back spread over the time+the process needs to take it, at the pace opens are going here, and the line+slows down when the database does. A caller refused again before its time+keeps that time, so a client that retries at once cannot push everyone else's+back.++An open is done when its first frame or response body goes out (the+`connection_state`, or a refusal) or the request ends. What follows is the+stream, which the event buffer's own cap paces. The state is this process's+own: each pod paces its own opens, as each has its own pool.+"""++from __future__ import annotations++import asyncio+import contextlib+import json+import logging+import math+import random+import re+import statistics+from collections import OrderedDict, deque+from collections.abc import Callable+from dataclasses import dataclass++from starlette.types import ASGIApp, Message, Receive, Scope, Send++from switch_core.observability.catalogue import AGENT_OPENS_DEFERRED+from switch_core.observability.metrics import metrics+from switch_core.refusal_breaker import bearer_caller++logger = logging.getLogger(__name__)++# How long an open waits for a slot before it is told when to come back. The+# wait and the open after it have to fit well inside the timeout of the+# proxies in front of switch-core: the tailnet ingress gives up on a request+# after about 10 s (measured on dev-abel, 29 Oct 2026). An open that outwaits+# a proxy still runs here after the proxy has given up, which moves the+# connection on behind its client's back, and today's agent runtime gives up+# for good on the refusal that follows. 2 s leaves most of the 10 s to the open+# itself and to proxies that give up sooner.+WAIT_S = 2.0+# The callers whose come-back time is remembered, which bounds this memory to a+# few MB. The gate answers before authentication, so a caller that never+# authenticates takes a place in the line too; the line given out goes no+# further than these callers' worth of slots, which bounds how far such+# callers can push back the times real ones are given. 10,000 is about the+# most agents one deployment has been tested with.+MAX_TRACKED = 10_000++BUSY = "busy"+_STATUS = 503++# The opens it paces, by the label its metric carries.+_ROUTES: tuple[tuple[re.Pattern[str], str], ...] = (+ (re.compile(r"^/agents/[^/]+/connection/ws$"), "agent_socket"),+ (re.compile(r"^/agents/[^/]+/events$"), "agent_event_stream"),+ (re.compile(r"^/v1/controllers/[^/]+/connection$"), "controller_open"),+ (re.compile(r"^/v1/controllers/[^/]+/connection/ws$"), "controller_socket"),+)+# Controllers read refusals from the management error envelope; everything+# else on this app reads FastAPI's `detail`.+_MANAGEMENT_PREFIXES = ("/v1/controllers/", "/v1/management/")+++def _route(scope: Scope) -> str | None:+ if scope["type"] not in ("http", "websocket"):+ return None+ path = scope.get("path", "")+ if scope["type"] == "http" and scope.get("method") not in ("GET", "POST"):+ return None+ for pattern, label in _ROUTES:+ if pattern.match(path):+ return label+ return None+++@dataclass(eq=False)+class _Waiter:+ caller: str | None+ # Set when a slot is handed to this open.+ slot: asyncio.Future[None]+++class OpenGate:+ def __init__(+ self,+ app: ASGIApp,+ *,+ limit: int,+ clock: Callable[[], float],+ rand: Callable[[], float] = random.random,+ ) -> None:+ self.app = app+ self.limit = limit+ self._clock = clock+ self._rand = rand+ self._slots = max(limit, 1)+ self._running = 0+ self._waiting: deque[_Waiter] = deque()+ # How long the last `limit` opens took, from taking a slot to their+ # first frame or response.+ self._open_s: deque[float] = deque(maxlen=self._slots)+ # The end of the line of callers told when to come back.+ self._next_at = 0.0+ self._given: OrderedDict[str, float] = OrderedDict()+ self._refusing_since: float | None = None+ self._refused = 0++ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:+ route = _route(scope) if self.limit > 0 else None+ if route is None:+ await self.app(scope, receive, send)+ return+ retry_after_s = await self._enter(bearer_caller(scope))+ if retry_after_s is not None:+ metrics().increment(AGENT_OPENS_DEFERRED, {"route": route})+ self._note_refused(retry_after_s)+ await _refuse(scope, receive, send, retry_after_s)+ return++ started = self._clock()+ released = False++ def release() -> None:+ nonlocal released+ if not released:+ released = True+ self._release(max(self._clock() - started, 0.0))++ async def done_on_first_output(message: Message) -> None:+ if message["type"] in (+ "websocket.send",+ "websocket.close",+ "http.response.body",+ ):+ release()+ await send(message)++ try:+ await self.app(scope, receive, done_on_first_output)+ finally:+ release()++ async def _enter(self, caller: str | None) -> float | None:+ """Take a slot, waiting up to `WAIT_S` for one. None once the open holds+ a slot; otherwise when it is to come back, in seconds."""+ if self._running < self._slots and not self._waiting:+ self._running += 1+ self._forget(caller)+ return None+ if len(self._waiting) >= self._most_waiting():+ return self._retry_after(caller, self._clock())++ waiter = _Waiter(caller, asyncio.get_running_loop().create_future())+ self._waiting.append(waiter)+ try:+ async with asyncio.timeout(WAIT_S):+ await waiter.slot+ except TimeoutError:+ if not waiter.slot.done() or waiter.slot.cancelled():+ with contextlib.suppress(ValueError):+ self._waiting.remove(waiter)+ return self._retry_after(caller, self._clock())+ # Handed a slot as the wait ran out: it is this open's.+ except asyncio.CancelledError:+ if waiter.slot.cancelled():+ with contextlib.suppress(ValueError):+ self._waiting.remove(waiter)+ else:+ # Handed a slot as the request went away: the next one has it.+ self._pass_slot()+ raise+ self._forget(caller)+ return None++ def _release(self, took: float) -> None:+ self._open_s.append(took)+ self._pass_slot()+ self._note_line_through()++ def _pass_slot(self) -> None:+ while self._waiting:+ waiter = self._waiting.popleft()+ if not waiter.slot.done():+ waiter.slot.set_result(None)+ return+ self._running -= 1++ def _open_time(self) -> float:+ """How long an open takes here: the median of the last `limit`. Before+ any has finished, at least `WAIT_S`, the time an open has just waited+ with every slot taken and none of them done."""+ return statistics.median(self._open_s) if self._open_s else WAIT_S++ def _most_waiting(self) -> float:+ """The most opens worth keeping waiting: as many as get a slot within+ `WAIT_S` at the pace opens are going here. An open past them would+ wait out `WAIT_S` and be refused anyway, so it is refused at once.+ Before any open has finished there is no pace to go by, and `WAIT_S`+ alone bounds the wait."""+ if not self._open_s or self._open_time() <= 0:+ return math.inf+ return WAIT_S * self._slots / self._open_time()++ def _retry_after(self, caller: str | None, now: float) -> float:+ """When this caller is to come back, in seconds: its place in the line.++ The line moves one slot every open time over `limit`. The first caller+ told to come back is behind the opens waiting here, each one after it a+ slot later, at a random point within its slot. The line reaches no+ further than `MAX_TRACKED` callers' worth of slots; past that, a caller+ is given a random point within it. A caller already given a time that+ has not come keeps it."""+ if caller is not None:+ at = self._given.get(caller)+ if at is not None and at > now:+ self._given.move_to_end(caller)+ return at - now+ spacing = self._open_time() / self._slots+ end = now + MAX_TRACKED * spacing+ start = max(now + (len(self._waiting) + 1) * spacing, self._next_at)+ if start < end:+ self._next_at = start + spacing+ at = start + spacing * self._rand()+ else:+ # Past the end of the line: a random point within it, so callers+ # refused together do not come back together.+ self._next_at = end+ at = now + spacing + (end - now - spacing) * self._rand()+ if caller is not None:+ self._given[caller] = at+ self._given.move_to_end(caller)+ while len(self._given) > MAX_TRACKED:+ self._given.popitem(last=False)+ return at - now++ def _forget(self, caller: str | None) -> None:+ if caller is not None:+ self._given.pop(caller, None)++ def _note_refused(self, retry_after_s: float) -> None:+ self._refused += 1+ if self._refusing_since is not None:+ return+ self._refusing_since = self._clock()+ logger.warning(+ "Opening connections more slowly than they arrive: %d opening and %d "+ "waiting for a slot, the rest told when to come back, the first in %.1f s",+ self._running,+ len(self._waiting),+ retry_after_s,+ )++ def _note_line_through(self) -> None:+ now = self._clock()+ if self._refusing_since is None or self._waiting or self._next_at > now:+ return+ logger.warning(+ "Caught up with connection opens: %d were told to come back later "+ "over %.0f s",+ self._refused,+ now - self._refusing_since,+ )+ self._refusing_since = None+ self._refused = 0+++async def _refuse(+ scope: Scope, receive: Receive, send: Send, retry_after_s: float+) -> None:+ """Refuse the open in the form its client reads refusals in, with when to+ come back."""+ # Rounded up to the millisecond clients wait in, so none comes back early.+ seconds = math.ceil(retry_after_s * 1000) / 1000+ message = (+ "This server is opening as many connections as it can take right now. "+ f"Open again in {seconds:g} s."+ )+ path = scope.get("path", "")+ if scope["type"] == "websocket":+ # As every refused agent or controller socket: accepted, a `refused`+ # frame with the status and detail, then closed with 4000 plus it.+ await receive() # the client's websocket.connect+ await send({"type": "websocket.accept"})+ frame = {+ "event": "refused",+ "data": {+ "status": _STATUS,+ "detail": {+ "code": BUSY,+ "message": message,+ "retry_after_s": seconds,+ },+ },+ }+ await send({"type": "websocket.send", "text": json.dumps(frame)})+ await send({"type": "websocket.close", "code": 4000 + _STATUS})+ return+ if path.startswith(_MANAGEMENT_PREFIXES):+ body: dict[str, object] = {+ "error": {+ "code": BUSY,+ "message": message,+ "retryable": True,+ "retry_after_s": seconds,+ }+ }+ else:+ body = {"detail": {"code": BUSY, "message": message, "retry_after_s": seconds}}+ payload = json.dumps(body).encode()+ await send(+ {+ "type": "http.response.start",+ "status": _STATUS,+ "headers": [+ (b"content-type", b"application/json"),+ (b"retry-after", str(math.ceil(retry_after_s)).encode()),+ (b"content-length", str(len(payload)).encode()),+ ],+ }+ )+ await send({"type": "http.response.body", "body": payload})diff --git a/core/switch_core/refusal_breaker.py b/core/switch_core/refusal_breaker.pyindex 077c57623..4a18fc4e7 100644--- a/core/switch_core/refusal_breaker.py+++ b/core/switch_core/refusal_breaker.py@@ -79,7 +79,7 @@ class RefusalBreaker:await self.app(scope, receive, send)returnmethod = scope.get("method", "GET")- caller = _bearer(scope)+ caller = bearer_caller(scope)if caller is None and method == "POST" and _TOKEN_EXCHANGE.match(path):body = await _read_body(receive)caller = hashlib.sha256(body).hexdigest()@@ -144,10 +144,14 @@ class RefusalBreaker:)-def _bearer(scope: Scope) -> str | None:+_BEARER = b"bearer "+++def bearer_caller(scope: Scope) -> str | None:+ """The caller a request names: its bearer token, hashed, never kept."""for name, value in scope.get("headers", []):- if name == b"authorization" and value.lower().startswith(b"bearer "):- return hashlib.sha256(value[7:].strip()).hexdigest()+ if name == b"authorization" and value.lower().startswith(_BEARER):+ return hashlib.sha256(value[len(_BEARER) :].strip()).hexdigest()return Nonediff --git a/core/tests/switch_core/gateway/test_gateway_ui.py b/core/tests/switch_core/gateway/test_gateway_ui.pyindex fc5c9c4c3..7e1303996 100644--- a/core/tests/switch_core/gateway/test_gateway_ui.py+++ b/core/tests/switch_core/gateway/test_gateway_ui.py@@ -154,6 +154,7 @@ def _agent_bridge_app(gateway_ui: GatewayUi | None) -> FastAPI:class _Config:agent_auth_cache_ttl_seconds = 1agent_auth_cache_max_entries = 16+ agent_open_limit = 8keyring = Keyring.parse("test:" + "x" * 40, legacy_secret=None)oauth_issuer_url = Noneoauth_audience = Nonediff --git a/core/tests/switch_core/management/test_middleware.py b/core/tests/switch_core/management/test_middleware.pyindex 06f9626a9..549af63fe 100644--- a/core/tests/switch_core/management/test_middleware.py+++ b/core/tests/switch_core/management/test_middleware.py@@ -296,6 +296,7 @@ class TestTheFlagOff:class _Config:agent_auth_cache_ttl_seconds = 1agent_auth_cache_max_entries = 16+ agent_open_limit = 8keyring = Keyring.parse("test:" + "x" * 40, legacy_secret=None)oauth_issuer_url = Noneoauth_audience = Nonediff --git a/core/tests/switch_core/telemetry/test_emitters_are_live.py b/core/tests/switch_core/telemetry/test_emitters_are_live.pyindex f763415c6..798fcaee3 100644--- a/core/tests/switch_core/telemetry/test_emitters_are_live.py+++ b/core/tests/switch_core/telemetry/test_emitters_are_live.py@@ -75,6 +75,7 @@ def _config() -> Any:class _Config:agent_auth_cache_ttl_seconds = 1agent_auth_cache_max_entries = 16+ agent_open_limit = 8keyring = Keyring.parse("test:" + "x" * 40, legacy_secret=None)oauth_issuer_url = Noneoauth_audience = Nonediff --git a/core/tests/switch_core/test_ingress_agent_api_paths.py b/core/tests/switch_core/test_ingress_agent_api_paths.pyindex 9c525acdb..bad2ad45d 100644--- a/core/tests/switch_core/test_ingress_agent_api_paths.py+++ b/core/tests/switch_core/test_ingress_agent_api_paths.py@@ -36,6 +36,7 @@ def _config() -> Any:class _Config:agent_auth_cache_ttl_seconds = 1agent_auth_cache_max_entries = 16+ agent_open_limit = 8keyring = Keyring.parse("test:" + "x" * 40, legacy_secret=None)oauth_issuer_url = Noneoauth_audience = Nonediff --git a/core/tests/switch_core/test_open_gate.py b/core/tests/switch_core/test_open_gate.pynew file mode 100644index 000000000..f6f69e090--- /dev/null+++ b/core/tests/switch_core/test_open_gate.py@@ -0,0 +1,342 @@+import asyncio+import json+import logging++import pytest+from starlette.types import Message, Receive, Scope, Send++from switch_core import open_gate+from switch_core.open_gate import OpenGate+++class _Socket:+ """An agent socket: accepted, then a first frame once `opened` is set, then+ a stream that runs until `ended` is set. Counts the opens that reached it."""++ def __init__(self) -> None:+ self.opened = asyncio.Event()+ self.ended = asyncio.Event()+ self.calls = 0++ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:+ self.calls += 1+ await receive()+ await send({"type": "websocket.accept"})+ await self.opened.wait()+ await send({"type": "websocket.send", "text": '{"event": "connection_state"}'})+ await self.ended.wait()+ await send({"type": "websocket.close", "code": 1000})+++class _Held:+ """Holds each open until the test lets it answer, and counts the opens that+ reached it: `answers[n]` lets the n-th answer."""++ def __init__(self) -> None:+ self.answers: list[asyncio.Event] = []++ @property+ def calls(self) -> int:+ return len(self.answers)++ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:+ answer = asyncio.Event()+ self.answers.append(answer)+ await answer.wait()+ await send({"type": "http.response.start", "status": 201, "headers": []})+ await send({"type": "http.response.body", "body": b"{}"})+++class _Clock:+ def __init__(self) -> None:+ self.now = 1000.0++ def __call__(self) -> float:+ return self.now+++def _scope(caller: str, path: str = "/v1/controllers/c1/connection") -> Scope:+ headers = [(b"authorization", f"Bearer {caller}".encode())]+ if path.endswith("/ws"):+ return {"type": "websocket", "path": path, "headers": headers}+ return {"type": "http", "method": "POST", "path": path, "headers": headers}+++def _receive(scope: Scope) -> Receive:+ async def receive() -> Message:+ if scope["type"] == "websocket":+ return {"type": "websocket.connect"}+ return {"type": "http.request", "body": b"{}", "more_body": False}++ return receive+++async def _open(gate: OpenGate, scope: Scope) -> list[Message]:+ sent: list[Message] = []++ async def send(message: Message) -> None:+ sent.append(message)++ await gate(scope, _receive(scope), send)+ return sent+++def _told(sent: list[Message]) -> float | None:+ """When a refused open was told to come back, in seconds; None if it opened."""+ if sent[0]["type"] == "websocket.accept":+ assert [m["type"] for m in sent] == [+ "websocket.accept",+ "websocket.send",+ "websocket.close",+ ]+ assert sent[2]["code"] == 4503+ frame = json.loads(sent[1]["text"])+ assert frame["event"] == "refused"+ assert frame["data"]["status"] == 503+ detail = frame["data"]["detail"]+ elif sent[0]["status"] == 503:+ body = json.loads(sent[1]["body"])+ detail = body.get("error") or body["detail"]+ else:+ return None+ assert detail["code"] == "busy"+ return detail["retry_after_s"]+++async def _settle() -> None:+ await asyncio.sleep(0.01)+++async def _measure(gate: OpenGate, app: _Held, clock: _Clock, seconds: float) -> None:+ """Run one open through the gate that takes `seconds`."""+ task = asyncio.create_task(_open(gate, _scope("measured")))+ await _settle()+ clock.now += seconds+ app.answers[-1].set()+ await task+++async def _answer_all(app: _Held, tasks: list[asyncio.Task[list[Message]]]) -> None:+ while not all(task.done() for task in tasks):+ for answer in app.answers:+ answer.set()+ await _settle()+++def short_wait(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setattr(open_gate, "WAIT_S", 0.05)+++async def test_an_open_past_the_limit_waits_for_a_slot_and_then_runs() -> None:+ app = _Socket()+ gate = OpenGate(app, limit=1, clock=_Clock())+ first = asyncio.create_task(_open(gate, _scope("a", "/agents/a1/connection/ws")))+ await _settle()+ second = asyncio.create_task(_open(gate, _scope("b", "/agents/b1/connection/ws")))+ await _settle()+ assert app.calls == 1++ # The first open's first frame frees its slot, though its stream goes on.+ app.opened.set()+ await _settle()+ assert app.calls == 2++ app.ended.set()+ await asyncio.gather(first, second)+++async def test_an_open_that_finds_no_slot_in_time_is_told_when_to_come_back(+ short_wait: None,+) -> None:+ app = _Held()+ gate = OpenGate(app, limit=1, clock=_Clock(), rand=lambda: 0.0)+ running = asyncio.create_task(_open(gate, _scope("a")))+ await _settle()++ told = _told(await _open(gate, _scope("b", "/agents/b1/connection/ws")))+ assert app.calls == 1 # refused before authentication or anything else+ # No open has finished yet, so one is taken to last at least the wait that+ # just ran out with the slot still taken: one slot from now.+ assert told == pytest.approx(open_gate.WAIT_S)++ await _answer_all(app, [running])+++async def test_more_waiting_than_can_get_a_slot_in_time_are_refused_at_once() -> None:+ clock = _Clock()+ app = _Held()+ gate = OpenGate(app, limit=1, clock=clock, rand=lambda: 0.0)+ await _measure(gate, app, clock, 0.5)+ # One slot, opens taking 0.5 s: four get a slot within the 2 s wait.+ line = [asyncio.create_task(_open(gate, _scope(c))) for c in "abcde"]+ await _settle()+ assert sum(task.done() for task in line) == 0++ fifth_waiting = asyncio.create_task(_open(gate, _scope("f")))+ await _settle()+ assert fifth_waiting.done() # refused at once, not after the wait+ # Behind the four waiting, one slot (0.5 s) each.+ assert _told(fifth_waiting.result()) == pytest.approx(2.5)++ await _answer_all(app, line)+++async def test_refused_opens_are_given_times_one_slot_apart_in_arrival_order() -> None:+ clock = _Clock()+ gate = OpenGate(_Held(), limit=4, clock=clock, rand=lambda: 0.0)+ gate._open_s.append(0.8) # a slot frees every 0.8 s over 4+ times = [gate._retry_after(f"caller-{n}", clock.now) for n in range(4)]+ assert times == pytest.approx([0.2, 0.4, 0.6, 0.8])+++async def test_the_slots_follow_the_median_of_the_last_opens() -> None:+ clock = _Clock()+ gate = OpenGate(_Held(), limit=3, clock=clock, rand=lambda: 0.0)+ # One open that hung does not move the line.+ gate._open_s.extend([0.3, 30.0, 0.3])+ assert gate._retry_after("a", clock.now) == pytest.approx(0.1)++ # The last three opens took longer, so the slots are further apart.+ clock.now += 10.0+ gate._open_s.extend([0.6, 0.6, 0.6])+ first = gate._retry_after("b", clock.now)+ assert gate._retry_after("c", clock.now) - first == pytest.approx(0.2)+++async def test_a_caller_refused_again_before_its_time_keeps_its_time() -> None:+ clock = _Clock()+ gate = OpenGate(_Held(), limit=1, clock=clock, rand=lambda: 0.0)+ gate._open_s.append(1.0)+ for n in range(10):+ gate._retry_after(f"caller-{n}", clock.now)+ first = gate._retry_after("late", clock.now)+ clock.now += 1.0+ # Back too early, as a client that ignores the time is: the same moment,+ # and the line behind it does not move.+ assert gate._retry_after("late", clock.now) == pytest.approx(first - 1.0)+ assert gate._retry_after("next", clock.now) == pytest.approx(first)+++async def test_callers_past_the_end_of_the_line_are_spread_within_it(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ monkeypatch.setattr(open_gate, "MAX_TRACKED", 4)+ clock = _Clock()+ draws = iter([0.0, 0.0, 0.0, 0.25, 0.75])+ gate = OpenGate(_Held(), limit=1, clock=clock, rand=lambda: next(draws))+ gate._open_s.append(1.0)+ times = [gate._retry_after(f"caller-{n}", clock.now) for n in range(5)]+ # Four callers' worth of one-second slots, and past that a random point+ # within them, not all of them at the end.+ assert times[:3] == pytest.approx([1.0, 2.0, 3.0])+ assert times[3:] == pytest.approx([1.75, 3.25])+ assert len(gate._given) == 4+++async def test_an_open_that_goes_away_while_waiting_leaves_no_slot_taken() -> None:+ app = _Held()+ gate = OpenGate(app, limit=1, clock=_Clock())+ running = asyncio.create_task(_open(gate, _scope("a")))+ await _settle()+ leaving = asyncio.create_task(_open(gate, _scope("b")))+ staying = asyncio.create_task(_open(gate, _scope("c")))+ await _settle()++ leaving.cancel()+ app.answers[0].set()+ await _settle()+ assert app.calls == 2 # "c" has the slot "b" would have had++ await _answer_all(app, [running, staying])+ # And with every open done, the slot is free for the next.+ nxt = asyncio.create_task(_open(gate, _scope("d")))+ await _settle()+ assert app.calls == 3+ await _answer_all(app, [nxt])+++async def test_an_open_handed_a_slot_as_it_goes_away_passes_the_slot_on() -> None:+ app = _Held()+ gate = OpenGate(app, limit=1, clock=_Clock())+ leaving: asyncio.Task[list[Message]] | None = None++ async def send(message: Message) -> None:+ # By its first output "a" has handed its slot to "b", which goes away+ # before it runs.+ if leaving is not None:+ leaving.cancel()++ scope = _scope("a")+ running = asyncio.create_task(gate(scope, _receive(scope), send))+ await _settle()+ leaving = asyncio.create_task(_open(gate, _scope("b")))+ staying = asyncio.create_task(_open(gate, _scope("c")))+ await _settle()++ app.answers[0].set()+ await _settle()+ assert leaving.cancelled()+ assert app.calls == 2 # "c" has the slot++ app.answers[1].set()+ await asyncio.gather(running, staying)+++async def test_a_controller_open_is_refused_in_the_management_envelope(+ short_wait: None,+) -> None:+ app = _Held()+ gate = OpenGate(app, limit=1, clock=_Clock())+ running = asyncio.create_task(_open(gate, _scope("a")))+ await _settle()++ sent = await _open(gate, _scope("ctl-token"))+ assert sent[0]["status"] == 503+ headers = dict(sent[0]["headers"])+ error = json.loads(sent[1]["body"])["error"]+ assert error["code"] == "busy"+ assert error["retryable"] is True+ assert int(headers[b"retry-after"]) >= error["retry_after_s"]++ await _answer_all(app, [running])+++async def test_other_routes_and_a_limit_of_0_are_not_paced() -> None:+ app = _Held()+ gate = OpenGate(app, limit=1, clock=_Clock())+ running = asyncio.create_task(_open(gate, _scope("a")))+ await _settle()++ # Every slot taken, yet a message post is not an open and goes straight in.+ post = asyncio.create_task(_open(gate, _scope("a", "/agents/a1/message")))+ await _settle()+ assert app.calls == 2++ off = OpenGate(app, limit=0, clock=_Clock())+ opening = asyncio.create_task(_open(off, _scope("b")))+ await _settle()+ assert app.calls == 3++ await _answer_all(app, [running, post, opening])+++async def test_one_line_is_logged_when_refusing_starts_and_one_when_it_is_over(+ short_wait: None, caplog: pytest.LogCaptureFixture+) -> None:+ clock = _Clock()+ app = _Held()+ gate = OpenGate(app, limit=1, clock=clock, rand=lambda: 0.0)+ await _measure(gate, app, clock, 0.5)+ running = asyncio.create_task(_open(gate, _scope("a")))+ await _settle()++ with caplog.at_level(logging.WARNING, logger="switch_core.open_gate"):+ for caller in ("c", "d", "e"):+ assert _told(await _open(gate, _scope(caller))) is not None+ assert len(caplog.records) == 1++ clock.now += 10.0 # every time given has passed+ await _answer_all(app, [running])+ assert len(caplog.records) == 2+ assert "3 were told to come back" in caplog.records[1].getMessage()diff --git a/deploy/observability/dashboard.json b/deploy/observability/dashboard.jsonindex 5c0997923..f892d48e9 100644--- a/deploy/observability/dashboard.json+++ b/deploy/observability/dashboard.json@@ -842,7 +842,29 @@"layout": {"x": 0,"y": 71,- "width": 12,+ "width": 6,+ "height": 4+ }+ },+ {+ "definition": {+ "type": "timeseries",+ "title": "Connection opens told to come back later per second, by route (a reconnect storm being paced)",+ "show_legend": true,+ "requests": [+ {+ "q": "sum:switch.agent.opens_deferred{$env,$service} by {route}.as_rate()",+ "display_type": "bars",+ "style": {+ "palette": "warm"+ }+ }+ ]+ },+ "layout": {+ "x": 6,+ "y": 71,+ "width": 6,"height": 4}},