Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

Limit connection opens per pod, so agents reconnecting after a pod loss don't use up the database pool (CHOO-3667) (#1139)

Abel Dantas6d5af7d16c22
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0763b72ce..8b9791612 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -85,6 +85,18 @@ version of their own to them without also giving them a release of their own.
Now a room the runner cannot read reads as missing.
#### Added
+- **A reconnect storm is paced instead of taking the pool.** When a pod goes
+ away, every agent and controller it held opens a connection on the pods
+ left within seconds. On staging (two pods, 1,000 agents, one pod deleted)
+ those opens took the whole database pool: requests and the process lease
+ timed out with them, and the surviving pod dropped agents of its own. Each
+ process now opens at most `AGENT_OPEN_LIMIT` (8) agent and controller
+ connections at once. An open that finds no slot within 2 s, which keeps it
+ well inside the timeout of the proxy in front, is refused before
+ authentication with 503 and `retry_after_s`, its place in the line at the
+ pace opens are taking in that process, so the storm comes back as a queue
+ spread over what the process can take. A caller that comes back early keeps
+ its time. `switch.agent.opens_deferred` counts the refusals, by route.
- **The Helm chart can put switch-core on chosen nodes.**
`switchCore.nodeSelector` and `switchCore.tolerations` go onto the
switch-core pods as written, so they can run on a node group of their own,
@@ -4191,6 +4203,10 @@ tags; see RELEASING.md.
### [Unreleased]
#### Added
+- **Waits its turn when Switch is busy opening connections.** An open or an
+ attach refused as `busy` comes with when to try again, the controller's
+ place in the server's queue, and the controller now waits that long
+ instead of its own backoff.
- **A provider report says when signing in would not help.** A provider
whose check says its login command would not fix the problem is reported
with `login_helps: false` beside `auth_problem`, so Console does not offer
@@ -4292,6 +4308,10 @@ The Switch protocol client and MCP runtime
### [Unreleased]
#### Added
+- **Waits its turn when Switch is busy opening connections.** A server
+ pacing a reconnect storm refuses an open as `busy` with when to open again,
+ which is this agent's place in its queue. The stream now waits that long,
+ even right after a restart, instead of retrying on its own schedule.
- When the WebSocket never opens and the runtime asks for the event stream
instead, the request carries `fallback=socket_never_opened` and the
socket's close code as `fallback_close_code`, so the server's log can tell
diff --git a/console/packages/agent-controller/src/stream.test.ts b/console/packages/agent-controller/src/stream.test.ts
index 15a7d74cc..34660e475 100644
--- a/console/packages/agent-controller/src/stream.test.ts
+++ b/console/packages/agent-controller/src/stream.test.ts
@@ -396,6 +396,26 @@ describe('runControllerStream', () => {
expect(opens).toBe(3);
});
+ it('opens again when a busy server says, not on its own backoff', async () => {
+ const scripted = new ScriptedClient();
+ let opens = 0;
+ scripted.open = async () => {
+ opens++;
+ if (opens === 1) throw new ControllerApiError(503, 'busy', 'opening at capacity', true, 0.3);
+ return opened;
+ };
+ scripted.sockets = () => scriptedSocket([state], 'open');
+ const stop = new AbortController();
+ const started = Date.now();
+ const { states, ending } = run(scripted, stop.signal);
+ await waitFor(() => states.includes('connected'), 'a connection', 3_000);
+ stop.abort();
+ await ending;
+ expect(opens).toBe(2);
+ // The backoff alone would have opened again after 5 ms.
+ expect(Date.now() - started).toBeGreaterThanOrEqual(290);
+ });
+
it('waits less each time a socket attached, however briefly, and never long', async () => {
const scripted = new ScriptedClient();
let opens = 0;
diff --git a/console/packages/agent-controller/src/stream.ts b/console/packages/agent-controller/src/stream.ts
index 841e1291d..972133bcb 100644
--- a/console/packages/agent-controller/src/stream.ts
+++ b/console/packages/agent-controller/src/stream.ts
@@ -111,6 +111,8 @@ export type ControllerStreamOptions = {
* are short and do not grow.
*/
const SERVICE_RESTART = 1012;
+/** The refusal of a server opening as many connections as it can take at once. */
+const BUSY = 'busy';
const RESTART_RETRY_MS = 250;
const RESTART_RETRY_JITTER_MS = 500;
@@ -192,9 +194,18 @@ class ControllerStream {
const stop = () => attempt.abort.abort();
signal.addEventListener('abort', stop, { once: true });
let failure: string | null = null;
+ // When a busy server said to come back (core's `open_gate.py`): that is
+ // this controller's place in its queue, so it outranks the backoff.
+ let toldMs: number | null = null;
try {
await this.attach(attempt);
} catch (error) {
+ if (
+ error instanceof ControllerApiError &&
+ error.code === BUSY &&
+ error.retryAfterS !== null
+ )
+ toldMs = Math.round(error.retryAfterS * 1000);
if (!attempt.ending && !signal.aborted) {
if (isRevoked(error)) attempt.ending = 'revoked';
else if (isTakenOver(error)) attempt.ending = 'taken_over';
@@ -217,16 +228,20 @@ class ControllerStream {
// so a socket that attached at all starts the waits over.
this.backoff = this.options.initialBackoffMs;
}
- const wait = attempt.restarting
- ? RESTART_RETRY_MS + Math.round(this.options.random() * RESTART_RETRY_JITTER_MS)
- : Math.round(this.backoff * (0.5 + this.options.random() * 0.5));
+ const wait =
+ toldMs !== null
+ ? toldMs
+ : attempt.restarting
+ ? RESTART_RETRY_MS + Math.round(this.options.random() * RESTART_RETRY_JITTER_MS)
+ : Math.round(this.backoff * (0.5 + this.options.random() * 0.5));
if (failure !== null)
log.warn('The controller socket is down; reconnecting.', {
error: failure,
retryInMs: wait,
});
await delay(wait, undefined, { signal }).catch(() => {});
- if (!attempt.restarting) this.backoff = Math.min(this.backoff * 2, this.options.maxBackoffMs);
+ if (!attempt.restarting && toldMs === null)
+ this.backoff = Math.min(this.backoff * 2, this.options.maxBackoffMs);
}
return 'stopped';
}
@@ -264,7 +279,7 @@ class ControllerStream {
const { socket } = opened;
const abort = attempt.abort;
const pending: Message[] = [];
- let refused: { status: number; code: string; message: string } | null = null;
+ let refused: Refusal | null = null;
let closedWith: number | null = null;
let wake: (() => void) | null = null;
const notify = () => {
@@ -336,10 +351,16 @@ class ControllerStream {
wake = resolve;
});
}
- const refusal = refused as { status: number; code: string; message: string } | null;
+ const refusal = refused as Refusal | null;
if (refusal !== null) {
if (refusal.status === 401 && refusal.code !== 'controller_revoked') opened.tokenRefused();
- throw new ControllerApiError(refusal.status, refusal.code, refusal.message, false, null);
+ throw new ControllerApiError(
+ refusal.status,
+ refusal.code,
+ refusal.message,
+ refusal.retryAfterS !== null,
+ refusal.retryAfterS
+ );
}
if (abort.signal.aborted && abort.signal.reason instanceof IdleTimeout)
throw abort.signal.reason;
@@ -380,14 +401,22 @@ class ControllerStream {
}
}
-function refusalOf(data: unknown): { status: number; code: string; message: string } {
+type Refusal = { status: number; code: string; message: string; retryAfterS: number | null };
+
+function refusalOf(data: unknown): Refusal {
const body = (data ?? {}) as { status?: unknown; detail?: unknown };
- const detail = (body.detail ?? {}) as { code?: unknown; message?: unknown };
+ const detail = (body.detail ?? {}) as {
+ code?: unknown;
+ message?: unknown;
+ retry_after_s?: unknown;
+ };
+ const after = detail.retry_after_s;
return {
status: typeof body.status === 'number' ? body.status : 0,
code: typeof detail.code === 'string' ? detail.code : 'refused',
message:
typeof detail.message === 'string' ? detail.message : 'Switch refused the controller socket.',
+ retryAfterS: typeof after === 'number' && Number.isFinite(after) && after >= 0 ? after : null,
};
}
diff --git a/console/packages/switch-agent-runtime/src/event-stream.test.ts b/console/packages/switch-agent-runtime/src/event-stream.test.ts
index fac1e1877..d3773816e 100644
--- a/console/packages/switch-agent-runtime/src/event-stream.test.ts
+++ b/console/packages/switch-agent-runtime/src/event-stream.test.ts
@@ -510,6 +510,64 @@ describe('a server restart', () => {
});
});
+describe('a busy server', () => {
+ it('opens again when the server says, not on its own backoff', async () => {
+ vi.useFakeTimers();
+ serve((socket, index) => {
+ if (index === 0) {
+ socket.refuse(503, { code: 'busy', message: 'opening at capacity', retry_after_s: 12 });
+ return;
+ }
+ attach(socket, 0);
+ });
+ const { abort } = makeStream({ rooms: [] });
+
+ // The backoff alone would have opened again after 1 s.
+ await vi.advanceTimersByTimeAsync(11_900);
+ expect(server.sockets).toHaveLength(1);
+ await vi.advanceTimersByTimeAsync(200);
+ expect(server.sockets).toHaveLength(2);
+ abort.abort();
+ });
+
+ it('stops at once when stopped while waiting its turn', async () => {
+ vi.useFakeTimers();
+ serve((socket) => {
+ socket.refuse(503, { code: 'busy', message: 'opening at capacity', retry_after_s: 300 });
+ });
+ const { abort } = makeStream({ rooms: [] });
+ await vi.advanceTimersByTimeAsync(1_000);
+ expect(server.sockets).toHaveLength(1);
+
+ abort.abort();
+ await vi.advanceTimersByTimeAsync(0);
+ // Nothing is left waiting out the 300 s, to keep the process alive.
+ expect(vi.getTimerCount()).toBe(0);
+ });
+
+ it('waits its turn even right after a restart', async () => {
+ vi.useFakeTimers();
+ serve((socket, index) => {
+ if (index === 0) {
+ attach(socket, 0);
+ queueMicrotask(() => socket.drop(1012, 'service restart'));
+ } else if (index === 1) {
+ socket.refuse(503, { code: 'busy', message: 'opening at capacity', retry_after_s: 30 });
+ } else {
+ attach(socket, 0);
+ }
+ });
+ const { abort } = makeStream({ rooms: [] });
+
+ // A restart alone retries within a second; the queue's time wins.
+ await vi.advanceTimersByTimeAsync(20_000);
+ expect(server.sockets).toHaveLength(2);
+ await vi.advanceTimersByTimeAsync(12_000);
+ expect(server.sockets).toHaveLength(3);
+ abort.abort();
+ });
+});
+
describe('reopening while an open is still in flight', () => {
/**
* A server that keeps the incarnation the way the real one does: every open
diff --git a/console/packages/switch-agent-runtime/src/event-stream.ts b/console/packages/switch-agent-runtime/src/event-stream.ts
index f90a33057..daf68b402 100644
--- a/console/packages/switch-agent-runtime/src/event-stream.ts
+++ b/console/packages/switch-agent-runtime/src/event-stream.ts
@@ -74,6 +74,14 @@ const SERVICE_RESTART = 1012;
const RESTART_WINDOW_MS = 60_000;
const RESTART_RETRY_MIN_MS = 250;
const RESTART_RETRY_SPREAD_MS = 500;
+/**
+ * The refusal of a server that is opening as many connections as it can take
+ * at once (core's `open_gate.py`). It names when to open again, which is this
+ * client's place in the server's queue, so that is when the next open goes,
+ * whatever the backoff or a restart would have said.
+ */
+const BUSY = 'busy';
+
/**
* How long an open socket has to last before it counts as a working stream.
*
@@ -169,6 +177,31 @@ export function refusalCode(body: string): string | null {
}
}
+/** The wait a `busy` refusal names, in milliseconds, or null. */
+export function busyRetryAfterMs(status: number, body: string): number | null {
+ if (status !== 503 || refusalCode(body) !== BUSY) return null;
+ try {
+ const detail = (JSON.parse(body) as { detail?: { retry_after_s?: unknown } }).detail;
+ const seconds = detail?.retry_after_s;
+ return typeof seconds === 'number' && Number.isFinite(seconds) && seconds >= 0
+ ? Math.round(seconds * 1000)
+ : null;
+ } catch {
+ return null;
+ }
+}
+
+/** Waits `ms`, or less if any of these signals aborts first, and leaves no timer behind. */
+function waitUnless(ms: number, ...signals: AbortSignal[]): Promise<void> {
+ let timer: ReturnType<typeof setTimeout> | undefined;
+ return Promise.race([
+ new Promise<void>((resolve) => {
+ timer = setTimeout(resolve, ms);
+ }),
+ until(...signals),
+ ]).finally(() => clearTimeout(timer));
+}
+
/** Resolves when any of these signals aborts, so a wait can be given up on. */
export function until(...signals: AbortSignal[]): Promise<void> {
const any = AbortSignal.any(signals);
@@ -819,6 +852,8 @@ export class SwitchEventStream {
if (signal.aborted || this.halt.signal.aborted) return;
let openedAt = 0;
let failure: unknown = null;
+ // The wait a busy server named for this open, if it was refused as busy.
+ let toldMs: number | null = null;
try {
const params = new URLSearchParams({
connection_id: connectionId,
@@ -868,6 +903,7 @@ export class SwitchEventStream {
} catch (error) {
if (error instanceof OpenRefused) {
const { status, body } = error;
+ toldMs = busyRetryAfterMs(status, body);
// Reopening without the refused room is a different request from the
// one that just failed, and the declared set strictly shrinks, so
// this cannot spin: retry now rather than serving the backoff a
@@ -922,7 +958,8 @@ export class SwitchEventStream {
});
}
this.deps.onDisconnected?.({ error });
- await pace();
+ if (toldMs !== null) await waitUnless(toldMs, signal, this.halt.signal);
+ else await pace();
}
}
diff --git a/core/switch_core/bridges/agent/app.py b/core/switch_core/bridges/agent/app.py
index 590fe0baf..7a22cd167 100644
--- a/core/switch_core/bridges/agent/app.py
+++ b/core/switch_core/bridges/agent/app.py
@@ -39,6 +39,7 @@ from switch_core.db.stores.room_store import RoomStore
from switch_core.gateway.ui import GatewayUi, GatewayUiMiddleware
from switch_core.logging_context import log_context
from switch_core.observability.http import MetricsMiddleware
+from switch_core.open_gate import OpenGate
from switch_core.refusal_breaker import RefusalBreaker
from switch_core.request_context import RequestContextMiddleware
from switch_core.request_validation import public_errors
@@ -192,6 +193,9 @@ def create_agent_bridge_app(
session_factory=session_factory, # type: ignore[arg-type]
controller_auth=controller_auth,
)
+ # Outside the bearer middleware, so an open the process cannot take yet is
+ # told when to come back before its credential is looked up.
+ app.add_middleware(OpenGate, limit=config.agent_open_limit, clock=time.monotonic)
# Outside the bearer middleware, so a caller that keeps repeating a refused
# request is answered before its credential is looked up again.
app.add_middleware(RefusalBreaker, clock=time.monotonic)
diff --git a/core/switch_core/config.py b/core/switch_core/config.py
index 8fcb890b8..dee21b287 100644
--- a/core/switch_core/config.py
+++ b/core/switch_core/config.py
@@ -751,6 +751,17 @@ class SwitchConfig(BaseSettings):
# SQLAlchemy's default is 30s, long enough that pool starvation surfaces as
# unexplained latency rather than an error. Fail fast and loudly instead.
db_pool_timeout: float = 5.0
+ # How many agent and controller connections this process opens at once
+ # (`open_gate.py`). Opening one takes dozens of transactions from the same
+ # pool the process's requests and its lease renewal use, and when a pod
+ # goes away every agent it held opens on the pods left within seconds.
+ # Past this many, an open waits briefly for a slot and is then told when to
+ # come back, in the order it arrived, at the pace opens are taking here,
+ # instead of queueing on the pool. 8 is what the pod-kill tests ran with:
+ # with 300 agents on two pods and 30 + 10 pools, no pool timeouts and no
+ # agent lost, against 161 pool timeouts and 56 agents lost without the
+ # gate (dev-abel, 28 Oct 2026). 0 turns it off.
+ agent_open_limit: int = Field(default=8, ge=0)
# Resolving a bearer token to its agent is the highest-frequency query in
# the system — it runs before every authenticated request, and a fleet of
diff --git a/core/switch_core/observability/catalogue.py b/core/switch_core/observability/catalogue.py
index 60f934816..5032e7180 100644
--- a/core/switch_core/observability/catalogue.py
+++ b/core/switch_core/observability/catalogue.py
@@ -475,6 +475,18 @@ AGENT_CONNECTIONS_REFUSED = _spec(
"revision in common), unauthorized (no valid credential), other.",
"reason",
)
+# An agent or controller connection told to open again later because this
+# process was already opening as many as it takes at once (`open_gate.py`).
+# A burst of these is a reconnect storm being paced, after a pod went away.
+AGENT_OPENS_DEFERRED = _spec(
+ "switch.agent.opens_deferred",
+ "sum",
+ "{connection}",
+ "Agent and controller connection opens refused with a time to come back, "
+ "because this process was opening as many as it takes at once. By route: "
+ "agent_socket, agent_event_stream, controller_open, controller_socket.",
+ "route",
+)
AGENT_CONNECTIONS_EXPIRED = _spec(
"switch.agent.connections_expired",
"sum",
@@ -663,6 +675,7 @@ CATALOGUE: dict[str, MetricSpec] = {
AGENT_BUFFER_SCAN_DURATION,
AGENT_CONNECTIONS_REFUSED,
AGENT_CONNECTIONS_EXPIRED,
+ AGENT_OPENS_DEFERRED,
AGENT_CONNECTIONS_OPENED,
AGENTS_CONNECTED,
CONSUMERS_RUNNING,
diff --git a/core/switch_core/open_gate.py b/core/switch_core/open_gate.py
new file mode 100644
index 000000000..ede82e8d0
--- /dev/null
+++ b/core/switch_core/open_gate.py
@@ -0,0 +1,340 @@
+"""Paces connection opens, so a reconnect storm becomes a queue.
+
+When a pod goes away, every agent and controller it held opens a connection on
+the pods that are left, all within a few seconds. Each open authenticates,
+makes a new incarnation of the connection and reads where the stream starts:
+dozens of transactions, on the same pool the process's requests and its own
+lease renewal use. Measured on staging (two pods, 1,000 agents, one pod
+deleted), the opens took the whole pool: thousands of pool timeouts, opens
+that failed after their incarnation was made, and the surviving pod's lease
+lapsing, which made it drop agents of its own.
+
+This middleware runs at most `limit` opens at once in this process. An open
+that finds every slot taken waits up to `WAIT_S` for one. Past that, or when
+more opens are waiting already than can get a slot within `WAIT_S`, it is
+refused before authentication or the database, with `retry_after_s`: its place
+in the line. The line moves one slot every median open time over `limit` (the
+median of the last `limit` opens), so a storm comes back spread over the time
+the process needs to take it, at the pace opens are going here, and the line
+slows down when the database does. A caller refused again before its time
+keeps that time, so a client that retries at once cannot push everyone else's
+back.
+
+An open is done when its first frame or response body goes out (the
+`connection_state`, or a refusal) or the request ends. What follows is the
+stream, which the event buffer's own cap paces. The state is this process's
+own: each pod paces its own opens, as each has its own pool.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import contextlib
+import json
+import logging
+import math
+import random
+import re
+import statistics
+from collections import OrderedDict, deque
+from collections.abc import Callable
+from dataclasses import dataclass
+
+from starlette.types import ASGIApp, Message, Receive, Scope, Send
+
+from switch_core.observability.catalogue import AGENT_OPENS_DEFERRED
+from switch_core.observability.metrics import metrics
+from switch_core.refusal_breaker import bearer_caller
+
+logger = logging.getLogger(__name__)
+
+# How long an open waits for a slot before it is told when to come back. The
+# wait and the open after it have to fit well inside the timeout of the
+# proxies in front of switch-core: the tailnet ingress gives up on a request
+# after about 10 s (measured on dev-abel, 29 Oct 2026). An open that outwaits
+# a proxy still runs here after the proxy has given up, which moves the
+# connection on behind its client's back, and today's agent runtime gives up
+# for good on the refusal that follows. 2 s leaves most of the 10 s to the open
+# itself and to proxies that give up sooner.
+WAIT_S = 2.0
+# The callers whose come-back time is remembered, which bounds this memory to a
+# few MB. The gate answers before authentication, so a caller that never
+# authenticates takes a place in the line too; the line given out goes no
+# further than these callers' worth of slots, which bounds how far such
+# callers can push back the times real ones are given. 10,000 is about the
+# most agents one deployment has been tested with.
+MAX_TRACKED = 10_000
+
+BUSY = "busy"
+_STATUS = 503
+
+# The opens it paces, by the label its metric carries.
+_ROUTES: tuple[tuple[re.Pattern[str], str], ...] = (
+ (re.compile(r"^/agents/[^/]+/connection/ws$"), "agent_socket"),
+ (re.compile(r"^/agents/[^/]+/events$"), "agent_event_stream"),
+ (re.compile(r"^/v1/controllers/[^/]+/connection$"), "controller_open"),
+ (re.compile(r"^/v1/controllers/[^/]+/connection/ws$"), "controller_socket"),
+)
+# Controllers read refusals from the management error envelope; everything
+# else on this app reads FastAPI's `detail`.
+_MANAGEMENT_PREFIXES = ("/v1/controllers/", "/v1/management/")
+
+
+def _route(scope: Scope) -> str | None:
+ if scope["type"] not in ("http", "websocket"):
+ return None
+ path = scope.get("path", "")
+ if scope["type"] == "http" and scope.get("method") not in ("GET", "POST"):
+ return None
+ for pattern, label in _ROUTES:
+ if pattern.match(path):
+ return label
+ return None
+
+
+@dataclass(eq=False)
+class _Waiter:
+ caller: str | None
+ # Set when a slot is handed to this open.
+ slot: asyncio.Future[None]
+
+
+class OpenGate:
+ def __init__(
+ self,
+ app: ASGIApp,
+ *,
+ limit: int,
+ clock: Callable[[], float],
+ rand: Callable[[], float] = random.random,
+ ) -> None:
+ self.app = app
+ self.limit = limit
+ self._clock = clock
+ self._rand = rand
+ self._slots = max(limit, 1)
+ self._running = 0
+ self._waiting: deque[_Waiter] = deque()
+ # How long the last `limit` opens took, from taking a slot to their
+ # first frame or response.
+ self._open_s: deque[float] = deque(maxlen=self._slots)
+ # The end of the line of callers told when to come back.
+ self._next_at = 0.0
+ self._given: OrderedDict[str, float] = OrderedDict()
+ self._refusing_since: float | None = None
+ self._refused = 0
+
+ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+ route = _route(scope) if self.limit > 0 else None
+ if route is None:
+ await self.app(scope, receive, send)
+ return
+ retry_after_s = await self._enter(bearer_caller(scope))
+ if retry_after_s is not None:
+ metrics().increment(AGENT_OPENS_DEFERRED, {"route": route})
+ self._note_refused(retry_after_s)
+ await _refuse(scope, receive, send, retry_after_s)
+ return
+
+ started = self._clock()
+ released = False
+
+ def release() -> None:
+ nonlocal released
+ if not released:
+ released = True
+ self._release(max(self._clock() - started, 0.0))
+
+ async def done_on_first_output(message: Message) -> None:
+ if message["type"] in (
+ "websocket.send",
+ "websocket.close",
+ "http.response.body",
+ ):
+ release()
+ await send(message)
+
+ try:
+ await self.app(scope, receive, done_on_first_output)
+ finally:
+ release()
+
+ async def _enter(self, caller: str | None) -> float | None:
+ """Take a slot, waiting up to `WAIT_S` for one. None once the open holds
+ a slot; otherwise when it is to come back, in seconds."""
+ if self._running < self._slots and not self._waiting:
+ self._running += 1
+ self._forget(caller)
+ return None
+ if len(self._waiting) >= self._most_waiting():
+ return self._retry_after(caller, self._clock())
+
+ waiter = _Waiter(caller, asyncio.get_running_loop().create_future())
+ self._waiting.append(waiter)
+ try:
+ async with asyncio.timeout(WAIT_S):
+ await waiter.slot
+ except TimeoutError:
+ if not waiter.slot.done() or waiter.slot.cancelled():
+ with contextlib.suppress(ValueError):
+ self._waiting.remove(waiter)
+ return self._retry_after(caller, self._clock())
+ # Handed a slot as the wait ran out: it is this open's.
+ except asyncio.CancelledError:
+ if waiter.slot.cancelled():
+ with contextlib.suppress(ValueError):
+ self._waiting.remove(waiter)
+ else:
+ # Handed a slot as the request went away: the next one has it.
+ self._pass_slot()
+ raise
+ self._forget(caller)
+ return None
+
+ def _release(self, took: float) -> None:
+ self._open_s.append(took)
+ self._pass_slot()
+ self._note_line_through()
+
+ def _pass_slot(self) -> None:
+ while self._waiting:
+ waiter = self._waiting.popleft()
+ if not waiter.slot.done():
+ waiter.slot.set_result(None)
+ return
+ self._running -= 1
+
+ def _open_time(self) -> float:
+ """How long an open takes here: the median of the last `limit`. Before
+ any has finished, at least `WAIT_S`, the time an open has just waited
+ with every slot taken and none of them done."""
+ return statistics.median(self._open_s) if self._open_s else WAIT_S
+
+ def _most_waiting(self) -> float:
+ """The most opens worth keeping waiting: as many as get a slot within
+ `WAIT_S` at the pace opens are going here. An open past them would
+ wait out `WAIT_S` and be refused anyway, so it is refused at once.
+ Before any open has finished there is no pace to go by, and `WAIT_S`
+ alone bounds the wait."""
+ if not self._open_s or self._open_time() <= 0:
+ return math.inf
+ return WAIT_S * self._slots / self._open_time()
+
+ def _retry_after(self, caller: str | None, now: float) -> float:
+ """When this caller is to come back, in seconds: its place in the line.
+
+ The line moves one slot every open time over `limit`. The first caller
+ told to come back is behind the opens waiting here, each one after it a
+ slot later, at a random point within its slot. The line reaches no
+ further than `MAX_TRACKED` callers' worth of slots; past that, a caller
+ is given a random point within it. A caller already given a time that
+ has not come keeps it."""
+ if caller is not None:
+ at = self._given.get(caller)
+ if at is not None and at > now:
+ self._given.move_to_end(caller)
+ return at - now
+ spacing = self._open_time() / self._slots
+ end = now + MAX_TRACKED * spacing
+ start = max(now + (len(self._waiting) + 1) * spacing, self._next_at)
+ if start < end:
+ self._next_at = start + spacing
+ at = start + spacing * self._rand()
+ else:
+ # Past the end of the line: a random point within it, so callers
+ # refused together do not come back together.
+ self._next_at = end
+ at = now + spacing + (end - now - spacing) * self._rand()
+ if caller is not None:
+ self._given[caller] = at
+ self._given.move_to_end(caller)
+ while len(self._given) > MAX_TRACKED:
+ self._given.popitem(last=False)
+ return at - now
+
+ def _forget(self, caller: str | None) -> None:
+ if caller is not None:
+ self._given.pop(caller, None)
+
+ def _note_refused(self, retry_after_s: float) -> None:
+ self._refused += 1
+ if self._refusing_since is not None:
+ return
+ self._refusing_since = self._clock()
+ logger.warning(
+ "Opening connections more slowly than they arrive: %d opening and %d "
+ "waiting for a slot, the rest told when to come back, the first in %.1f s",
+ self._running,
+ len(self._waiting),
+ retry_after_s,
+ )
+
+ def _note_line_through(self) -> None:
+ now = self._clock()
+ if self._refusing_since is None or self._waiting or self._next_at > now:
+ return
+ logger.warning(
+ "Caught up with connection opens: %d were told to come back later "
+ "over %.0f s",
+ self._refused,
+ now - self._refusing_since,
+ )
+ self._refusing_since = None
+ self._refused = 0
+
+
+async def _refuse(
+ scope: Scope, receive: Receive, send: Send, retry_after_s: float
+) -> None:
+ """Refuse the open in the form its client reads refusals in, with when to
+ come back."""
+ # Rounded up to the millisecond clients wait in, so none comes back early.
+ seconds = math.ceil(retry_after_s * 1000) / 1000
+ message = (
+ "This server is opening as many connections as it can take right now. "
+ f"Open again in {seconds:g} s."
+ )
+ path = scope.get("path", "")
+ if scope["type"] == "websocket":
+ # As every refused agent or controller socket: accepted, a `refused`
+ # frame with the status and detail, then closed with 4000 plus it.
+ await receive() # the client's websocket.connect
+ await send({"type": "websocket.accept"})
+ frame = {
+ "event": "refused",
+ "data": {
+ "status": _STATUS,
+ "detail": {
+ "code": BUSY,
+ "message": message,
+ "retry_after_s": seconds,
+ },
+ },
+ }
+ await send({"type": "websocket.send", "text": json.dumps(frame)})
+ await send({"type": "websocket.close", "code": 4000 + _STATUS})
+ return
+ if path.startswith(_MANAGEMENT_PREFIXES):
+ body: dict[str, object] = {
+ "error": {
+ "code": BUSY,
+ "message": message,
+ "retryable": True,
+ "retry_after_s": seconds,
+ }
+ }
+ else:
+ body = {"detail": {"code": BUSY, "message": message, "retry_after_s": seconds}}
+ payload = json.dumps(body).encode()
+ await send(
+ {
+ "type": "http.response.start",
+ "status": _STATUS,
+ "headers": [
+ (b"content-type", b"application/json"),
+ (b"retry-after", str(math.ceil(retry_after_s)).encode()),
+ (b"content-length", str(len(payload)).encode()),
+ ],
+ }
+ )
+ await send({"type": "http.response.body", "body": payload})
diff --git a/core/switch_core/refusal_breaker.py b/core/switch_core/refusal_breaker.py
index 077c57623..4a18fc4e7 100644
--- a/core/switch_core/refusal_breaker.py
+++ b/core/switch_core/refusal_breaker.py
@@ -79,7 +79,7 @@ class RefusalBreaker:
await self.app(scope, receive, send)
return
method = scope.get("method", "GET")
- caller = _bearer(scope)
+ caller = bearer_caller(scope)
if caller is None and method == "POST" and _TOKEN_EXCHANGE.match(path):
body = await _read_body(receive)
caller = hashlib.sha256(body).hexdigest()
@@ -144,10 +144,14 @@ class RefusalBreaker:
)
-def _bearer(scope: Scope) -> str | None:
+_BEARER = b"bearer "
+
+
+def bearer_caller(scope: Scope) -> str | None:
+ """The caller a request names: its bearer token, hashed, never kept."""
for name, value in scope.get("headers", []):
- if name == b"authorization" and value.lower().startswith(b"bearer "):
- return hashlib.sha256(value[7:].strip()).hexdigest()
+ if name == b"authorization" and value.lower().startswith(_BEARER):
+ return hashlib.sha256(value[len(_BEARER) :].strip()).hexdigest()
return None
diff --git a/core/tests/switch_core/gateway/test_gateway_ui.py b/core/tests/switch_core/gateway/test_gateway_ui.py
index fc5c9c4c3..7e1303996 100644
--- a/core/tests/switch_core/gateway/test_gateway_ui.py
+++ b/core/tests/switch_core/gateway/test_gateway_ui.py
@@ -154,6 +154,7 @@ def _agent_bridge_app(gateway_ui: GatewayUi | None) -> FastAPI:
class _Config:
agent_auth_cache_ttl_seconds = 1
agent_auth_cache_max_entries = 16
+ agent_open_limit = 8
keyring = Keyring.parse("test:" + "x" * 40, legacy_secret=None)
oauth_issuer_url = None
oauth_audience = None
diff --git a/core/tests/switch_core/management/test_middleware.py b/core/tests/switch_core/management/test_middleware.py
index 06f9626a9..549af63fe 100644
--- a/core/tests/switch_core/management/test_middleware.py
+++ b/core/tests/switch_core/management/test_middleware.py
@@ -296,6 +296,7 @@ class TestTheFlagOff:
class _Config:
agent_auth_cache_ttl_seconds = 1
agent_auth_cache_max_entries = 16
+ agent_open_limit = 8
keyring = Keyring.parse("test:" + "x" * 40, legacy_secret=None)
oauth_issuer_url = None
oauth_audience = None
diff --git a/core/tests/switch_core/telemetry/test_emitters_are_live.py b/core/tests/switch_core/telemetry/test_emitters_are_live.py
index f763415c6..798fcaee3 100644
--- a/core/tests/switch_core/telemetry/test_emitters_are_live.py
+++ b/core/tests/switch_core/telemetry/test_emitters_are_live.py
@@ -75,6 +75,7 @@ def _config() -> Any:
class _Config:
agent_auth_cache_ttl_seconds = 1
agent_auth_cache_max_entries = 16
+ agent_open_limit = 8
keyring = Keyring.parse("test:" + "x" * 40, legacy_secret=None)
oauth_issuer_url = None
oauth_audience = None
diff --git a/core/tests/switch_core/test_ingress_agent_api_paths.py b/core/tests/switch_core/test_ingress_agent_api_paths.py
index 9c525acdb..bad2ad45d 100644
--- a/core/tests/switch_core/test_ingress_agent_api_paths.py
+++ b/core/tests/switch_core/test_ingress_agent_api_paths.py
@@ -36,6 +36,7 @@ def _config() -> Any:
class _Config:
agent_auth_cache_ttl_seconds = 1
agent_auth_cache_max_entries = 16
+ agent_open_limit = 8
keyring = Keyring.parse("test:" + "x" * 40, legacy_secret=None)
oauth_issuer_url = None
oauth_audience = None
diff --git a/core/tests/switch_core/test_open_gate.py b/core/tests/switch_core/test_open_gate.py
new file mode 100644
index 000000000..f6f69e090
--- /dev/null
+++ b/core/tests/switch_core/test_open_gate.py
@@ -0,0 +1,342 @@
+import asyncio
+import json
+import logging
+
+import pytest
+from starlette.types import Message, Receive, Scope, Send
+
+from switch_core import open_gate
+from switch_core.open_gate import OpenGate
+
+
+class _Socket:
+ """An agent socket: accepted, then a first frame once `opened` is set, then
+ a stream that runs until `ended` is set. Counts the opens that reached it."""
+
+ def __init__(self) -> None:
+ self.opened = asyncio.Event()
+ self.ended = asyncio.Event()
+ self.calls = 0
+
+ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+ self.calls += 1
+ await receive()
+ await send({"type": "websocket.accept"})
+ await self.opened.wait()
+ await send({"type": "websocket.send", "text": '{"event": "connection_state"}'})
+ await self.ended.wait()
+ await send({"type": "websocket.close", "code": 1000})
+
+
+class _Held:
+ """Holds each open until the test lets it answer, and counts the opens that
+ reached it: `answers[n]` lets the n-th answer."""
+
+ def __init__(self) -> None:
+ self.answers: list[asyncio.Event] = []
+
+ @property
+ def calls(self) -> int:
+ return len(self.answers)
+
+ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+ answer = asyncio.Event()
+ self.answers.append(answer)
+ await answer.wait()
+ await send({"type": "http.response.start", "status": 201, "headers": []})
+ await send({"type": "http.response.body", "body": b"{}"})
+
+
+class _Clock:
+ def __init__(self) -> None:
+ self.now = 1000.0
+
+ def __call__(self) -> float:
+ return self.now
+
+
+def _scope(caller: str, path: str = "/v1/controllers/c1/connection") -> Scope:
+ headers = [(b"authorization", f"Bearer {caller}".encode())]
+ if path.endswith("/ws"):
+ return {"type": "websocket", "path": path, "headers": headers}
+ return {"type": "http", "method": "POST", "path": path, "headers": headers}
+
+
+def _receive(scope: Scope) -> Receive:
+ async def receive() -> Message:
+ if scope["type"] == "websocket":
+ return {"type": "websocket.connect"}
+ return {"type": "http.request", "body": b"{}", "more_body": False}
+
+ return receive
+
+
+async def _open(gate: OpenGate, scope: Scope) -> list[Message]:
+ sent: list[Message] = []
+
+ async def send(message: Message) -> None:
+ sent.append(message)
+
+ await gate(scope, _receive(scope), send)
+ return sent
+
+
+def _told(sent: list[Message]) -> float | None:
+ """When a refused open was told to come back, in seconds; None if it opened."""
+ if sent[0]["type"] == "websocket.accept":
+ assert [m["type"] for m in sent] == [
+ "websocket.accept",
+ "websocket.send",
+ "websocket.close",
+ ]
+ assert sent[2]["code"] == 4503
+ frame = json.loads(sent[1]["text"])
+ assert frame["event"] == "refused"
+ assert frame["data"]["status"] == 503
+ detail = frame["data"]["detail"]
+ elif sent[0]["status"] == 503:
+ body = json.loads(sent[1]["body"])
+ detail = body.get("error") or body["detail"]
+ else:
+ return None
+ assert detail["code"] == "busy"
+ return detail["retry_after_s"]
+
+
+async def _settle() -> None:
+ await asyncio.sleep(0.01)
+
+
+async def _measure(gate: OpenGate, app: _Held, clock: _Clock, seconds: float) -> None:
+ """Run one open through the gate that takes `seconds`."""
+ task = asyncio.create_task(_open(gate, _scope("measured")))
+ await _settle()
+ clock.now += seconds
+ app.answers[-1].set()
+ await task
+
+
+async def _answer_all(app: _Held, tasks: list[asyncio.Task[list[Message]]]) -> None:
+ while not all(task.done() for task in tasks):
+ for answer in app.answers:
+ answer.set()
+ await _settle()
+
+
+def short_wait(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr(open_gate, "WAIT_S", 0.05)
+
+
+async def test_an_open_past_the_limit_waits_for_a_slot_and_then_runs() -> None:
+ app = _Socket()
+ gate = OpenGate(app, limit=1, clock=_Clock())
+ first = asyncio.create_task(_open(gate, _scope("a", "/agents/a1/connection/ws")))
+ await _settle()
+ second = asyncio.create_task(_open(gate, _scope("b", "/agents/b1/connection/ws")))
+ await _settle()
+ assert app.calls == 1
+
+ # The first open's first frame frees its slot, though its stream goes on.
+ app.opened.set()
+ await _settle()
+ assert app.calls == 2
+
+ app.ended.set()
+ await asyncio.gather(first, second)
+
+
+async def test_an_open_that_finds_no_slot_in_time_is_told_when_to_come_back(
+ short_wait: None,
+) -> None:
+ app = _Held()
+ gate = OpenGate(app, limit=1, clock=_Clock(), rand=lambda: 0.0)
+ running = asyncio.create_task(_open(gate, _scope("a")))
+ await _settle()
+
+ told = _told(await _open(gate, _scope("b", "/agents/b1/connection/ws")))
+ assert app.calls == 1 # refused before authentication or anything else
+ # No open has finished yet, so one is taken to last at least the wait that
+ # just ran out with the slot still taken: one slot from now.
+ assert told == pytest.approx(open_gate.WAIT_S)
+
+ await _answer_all(app, [running])
+
+
+async def test_more_waiting_than_can_get_a_slot_in_time_are_refused_at_once() -> None:
+ clock = _Clock()
+ app = _Held()
+ gate = OpenGate(app, limit=1, clock=clock, rand=lambda: 0.0)
+ await _measure(gate, app, clock, 0.5)
+ # One slot, opens taking 0.5 s: four get a slot within the 2 s wait.
+ line = [asyncio.create_task(_open(gate, _scope(c))) for c in "abcde"]
+ await _settle()
+ assert sum(task.done() for task in line) == 0
+
+ fifth_waiting = asyncio.create_task(_open(gate, _scope("f")))
+ await _settle()
+ assert fifth_waiting.done() # refused at once, not after the wait
+ # Behind the four waiting, one slot (0.5 s) each.
+ assert _told(fifth_waiting.result()) == pytest.approx(2.5)
+
+ await _answer_all(app, line)
+
+
+async def test_refused_opens_are_given_times_one_slot_apart_in_arrival_order() -> None:
+ clock = _Clock()
+ gate = OpenGate(_Held(), limit=4, clock=clock, rand=lambda: 0.0)
+ gate._open_s.append(0.8) # a slot frees every 0.8 s over 4
+ times = [gate._retry_after(f"caller-{n}", clock.now) for n in range(4)]
+ assert times == pytest.approx([0.2, 0.4, 0.6, 0.8])
+
+
+async def test_the_slots_follow_the_median_of_the_last_opens() -> None:
+ clock = _Clock()
+ gate = OpenGate(_Held(), limit=3, clock=clock, rand=lambda: 0.0)
+ # One open that hung does not move the line.
+ gate._open_s.extend([0.3, 30.0, 0.3])
+ assert gate._retry_after("a", clock.now) == pytest.approx(0.1)
+
+ # The last three opens took longer, so the slots are further apart.
+ clock.now += 10.0
+ gate._open_s.extend([0.6, 0.6, 0.6])
+ first = gate._retry_after("b", clock.now)
+ assert gate._retry_after("c", clock.now) - first == pytest.approx(0.2)
+
+
+async def test_a_caller_refused_again_before_its_time_keeps_its_time() -> None:
+ clock = _Clock()
+ gate = OpenGate(_Held(), limit=1, clock=clock, rand=lambda: 0.0)
+ gate._open_s.append(1.0)
+ for n in range(10):
+ gate._retry_after(f"caller-{n}", clock.now)
+ first = gate._retry_after("late", clock.now)
+ clock.now += 1.0
+ # Back too early, as a client that ignores the time is: the same moment,
+ # and the line behind it does not move.
+ assert gate._retry_after("late", clock.now) == pytest.approx(first - 1.0)
+ assert gate._retry_after("next", clock.now) == pytest.approx(first)
+
+
+async def test_callers_past_the_end_of_the_line_are_spread_within_it(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ monkeypatch.setattr(open_gate, "MAX_TRACKED", 4)
+ clock = _Clock()
+ draws = iter([0.0, 0.0, 0.0, 0.25, 0.75])
+ gate = OpenGate(_Held(), limit=1, clock=clock, rand=lambda: next(draws))
+ gate._open_s.append(1.0)
+ times = [gate._retry_after(f"caller-{n}", clock.now) for n in range(5)]
+ # Four callers' worth of one-second slots, and past that a random point
+ # within them, not all of them at the end.
+ assert times[:3] == pytest.approx([1.0, 2.0, 3.0])
+ assert times[3:] == pytest.approx([1.75, 3.25])
+ assert len(gate._given) == 4
+
+
+async def test_an_open_that_goes_away_while_waiting_leaves_no_slot_taken() -> None:
+ app = _Held()
+ gate = OpenGate(app, limit=1, clock=_Clock())
+ running = asyncio.create_task(_open(gate, _scope("a")))
+ await _settle()
+ leaving = asyncio.create_task(_open(gate, _scope("b")))
+ staying = asyncio.create_task(_open(gate, _scope("c")))
+ await _settle()
+
+ leaving.cancel()
+ app.answers[0].set()
+ await _settle()
+ assert app.calls == 2 # "c" has the slot "b" would have had
+
+ await _answer_all(app, [running, staying])
+ # And with every open done, the slot is free for the next.
+ nxt = asyncio.create_task(_open(gate, _scope("d")))
+ await _settle()
+ assert app.calls == 3
+ await _answer_all(app, [nxt])
+
+
+async def test_an_open_handed_a_slot_as_it_goes_away_passes_the_slot_on() -> None:
+ app = _Held()
+ gate = OpenGate(app, limit=1, clock=_Clock())
+ leaving: asyncio.Task[list[Message]] | None = None
+
+ async def send(message: Message) -> None:
+ # By its first output "a" has handed its slot to "b", which goes away
+ # before it runs.
+ if leaving is not None:
+ leaving.cancel()
+
+ scope = _scope("a")
+ running = asyncio.create_task(gate(scope, _receive(scope), send))
+ await _settle()
+ leaving = asyncio.create_task(_open(gate, _scope("b")))
+ staying = asyncio.create_task(_open(gate, _scope("c")))
+ await _settle()
+
+ app.answers[0].set()
+ await _settle()
+ assert leaving.cancelled()
+ assert app.calls == 2 # "c" has the slot
+
+ app.answers[1].set()
+ await asyncio.gather(running, staying)
+
+
+async def test_a_controller_open_is_refused_in_the_management_envelope(
+ short_wait: None,
+) -> None:
+ app = _Held()
+ gate = OpenGate(app, limit=1, clock=_Clock())
+ running = asyncio.create_task(_open(gate, _scope("a")))
+ await _settle()
+
+ sent = await _open(gate, _scope("ctl-token"))
+ assert sent[0]["status"] == 503
+ headers = dict(sent[0]["headers"])
+ error = json.loads(sent[1]["body"])["error"]
+ assert error["code"] == "busy"
+ assert error["retryable"] is True
+ assert int(headers[b"retry-after"]) >= error["retry_after_s"]
+
+ await _answer_all(app, [running])
+
+
+async def test_other_routes_and_a_limit_of_0_are_not_paced() -> None:
+ app = _Held()
+ gate = OpenGate(app, limit=1, clock=_Clock())
+ running = asyncio.create_task(_open(gate, _scope("a")))
+ await _settle()
+
+ # Every slot taken, yet a message post is not an open and goes straight in.
+ post = asyncio.create_task(_open(gate, _scope("a", "/agents/a1/message")))
+ await _settle()
+ assert app.calls == 2
+
+ off = OpenGate(app, limit=0, clock=_Clock())
+ opening = asyncio.create_task(_open(off, _scope("b")))
+ await _settle()
+ assert app.calls == 3
+
+ await _answer_all(app, [running, post, opening])
+
+
+async def test_one_line_is_logged_when_refusing_starts_and_one_when_it_is_over(
+ short_wait: None, caplog: pytest.LogCaptureFixture
+) -> None:
+ clock = _Clock()
+ app = _Held()
+ gate = OpenGate(app, limit=1, clock=clock, rand=lambda: 0.0)
+ await _measure(gate, app, clock, 0.5)
+ running = asyncio.create_task(_open(gate, _scope("a")))
+ await _settle()
+
+ with caplog.at_level(logging.WARNING, logger="switch_core.open_gate"):
+ for caller in ("c", "d", "e"):
+ assert _told(await _open(gate, _scope(caller))) is not None
+ assert len(caplog.records) == 1
+
+ clock.now += 10.0 # every time given has passed
+ await _answer_all(app, [running])
+ assert len(caplog.records) == 2
+ assert "3 were told to come back" in caplog.records[1].getMessage()
diff --git a/deploy/observability/dashboard.json b/deploy/observability/dashboard.json
index 5c0997923..f892d48e9 100644
--- a/deploy/observability/dashboard.json
+++ b/deploy/observability/dashboard.json
@@ -842,7 +842,29 @@
"layout": {
"x": 0,
"y": 71,
- "width": 12,
+ "width": 6,
+ "height": 4
+ }
+ },
+ {
+ "definition": {
+ "type": "timeseries",
+ "title": "Connection opens told to come back later per second, by route (a reconnect storm being paced)",
+ "show_legend": true,
+ "requests": [
+ {
+ "q": "sum:switch.agent.opens_deferred{$env,$service} by {route}.as_rate()",
+ "display_type": "bars",
+ "style": {
+ "palette": "warm"
+ }
+ }
+ ]
+ },
+ "layout": {
+ "x": 6,
+ "y": 71,
+ "width": 6,
"height": 4
}
},