Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

Run the load test every night on staging (CHOO-3667) (#1415)

Abel Dantasd35c861890ad
diff --git a/loadtest/.gitignore b/loadtest/.gitignore
new file mode 100644
index 000000000..a1e03960f
--- /dev/null
+++ b/loadtest/.gitignore
@@ -0,0 +1 @@
+runs/
diff --git a/loadtest/README.md b/loadtest/README.md
new file mode 100644
index 000000000..a0edc2d6a
--- /dev/null
+++ b/loadtest/README.md
@@ -0,0 +1,89 @@
+# switch-core load harness
+
+Measures how much load switch-core carries in one environment, so the same run can be repeated at one pod and at two and the numbers compared (CHOO-3667, Stage 1 of the scale-validation plan).
+
+A run signs in as the environment's admin, creates its own API key, agents and rooms, connects the agents over the agent WebSocket the way the agent runtime does, sends the planned traffic, and samples every switch-core pod and the database while it does. At the end it deletes what it created and writes a summary.
+
+It never restarts, scales or edits anything in Kubernetes. It only runs `kubectl get` (to find the pods, the deployed URLs and the admin login), `kubectl exec` (to run `pod_sampler.py` inside the pods) and, with `--via port-forward`, `kubectl port-forward`. That makes it safe on GitOps environments: ArgoCD has nothing to fight over. It refuses the pilot's namespace and hostnames, and anything with `prod` in the name.
+
+## Running it
+
+From the repo root, on a machine that can reach the environment's URLs, with `kubectl` pointed at the cluster (its current context, or name one with `--kube-context`):
+
+```sh
+uv run --project core python loadtest/run.py --namespace <namespace> \
+ --scenarios idle,steady,burst --steps 50,100,200 --rate 5 --duration 120 \
+ --rds-instance <db-instance>
+```
+
+The API and gateway URLs default to the ones the release was deployed with (`GATEWAY_PUBLIC_URL` and `FRONTEND_BASE_URL` on the switch-core deployment, which come from the release's values). Pass `--api-url` and `--gateway-url` to be explicit; a URL that belongs to a different environment is refused, so the pods sampled are always the pods loaded.
+
+At each step in `--steps`, the harness brings that many agents online and then runs each scenario for `--duration` seconds, with `--settle` quiet seconds after each:
+
+- `idle`: the agents stay connected and answer the server's pings, nothing else.
+- `steady`: messages at `--rate` per second, spread over the rooms, each posted by a member of the room. Receivers record the time from send to arrival, and a share of them (`--read-fraction`, 10% by default) follow a delivery with a `read_context` call, as a runtime catching up would.
+- `burst`: the steady traffic plus `--burst-size` messages packed into `--burst-window` seconds, every `--burst-every` seconds. `--hot-room` sends all the spikes to one room, to measure the per-room lock.
+
+### Keeping the agents between runs
+
+Seeding 10,000 agents and tearing them down take about 12 of a run's 22 minutes. With `--keep`, a run leaves its agents and rooms in place, revokes only its controllers, and writes `runs/<run id>/keep.json` (ids only: the agents have no key of their own, and a controller's credential is never written down). A later run with `--reuse <that run id>` makes fresh controllers and places the same agents on them instead of seeding; it needs the same `--seed`, `--steps`, `--rooms` and `--room-size`, which it checks. Add `--keep` again to pass them on; a run without it, or `--cleanup <the first run id>`, removes them. Both need `--controllers`.
+
+To compare one pod with two, run the same command, with the same `--seed`, once against each replica count. The replica count is changed by the deployment, not by this harness.
+
+`--rates 10,15,20,25` sweeps the message rate in one run: after the scenarios at each step, the same agents stay connected for one steady window of `--rate-duration` seconds (90 by default) at each rate, named `steady15@300` and so on. After each window the harness waits up to `--drain` seconds for that window's deliveries to arrive, so a backlog does not spill into the next rate, and it stops stepping after the first rate that breaks: a delivery lost, p99 over 20 s, or more than 20 errors. Agents are created, connected and measured idle once, so a five-rate sweep takes about 15 minutes instead of about 40 as separate runs:
+
+```sh
+uv run --project core python loadtest/run.py --namespace <namespace> \
+ --scenarios idle --steps 300 --rates 10,15,20,25,30 --rds-instance <db-instance>
+```
+
+`--plan` prints the workload (message counts and a fingerprint per scenario) without making any request. The same `--seed` and flags always produce the same plan, which `summary.md` records alongside the results.
+
+`--via port-forward` reaches each switch-core pod through `kubectl port-forward` instead of the deployed URLs, spreading the agents evenly over the pods. Use it for a smoke run from a machine that cannot reach those URLs, not for capacity numbers, because all the traffic then passes through the Kubernetes API server.
+
+## What it records
+
+Each run writes `loadtest/runs/<run_id>/` (gitignored):
+
+- `summary.md`, one row per scenario and step: agents connected, messages per second sent and delivered, delivery latency p50 and p99, errors, CPU and memory per pod, database connections, transactions per second, and RDS CPU when `--rds-instance` is given.
+- `summary.json`, the same numbers plus the run's settings.
+- `samples.jsonl`, the raw per-pod and database samples, one a second.
+- `messages.jsonl`, every message sent with each receiver's latency.
+- `events.jsonl`, connects, drops, refusals, evictions and gaps.
+
+Every request the harness makes carries `X-Request-Id: load-<run_id>-<n>`, so a run can be found in the server's logs.
+
+`harness lag max` in the summary is how far the harness fell behind its own schedule. If it climbs above a few hundred milliseconds, the client machine is the limit, not the server, and the run should be split across machines or slowed down.
+
+RDS numbers come from CloudWatch at one-minute resolution and need AWS credentials that can read it, such as an `AWS_PROFILE` (or `--aws-profile`) with an SSO session. CloudWatch lands its data a minute or two late, so the last minute of a run can read low.
+
+## Every night on staging
+
+`nightly.py` runs the harness from pods inside the cluster, as three CronJobs of the deployment's own load test chart, which sets the schedule and sizes:
+
+- `up` (01:45 UTC) takes staging's node group to its night size, starts the harness's node and scales switch-core to 12 pods.
+- `run` (02:00) starts ten workers of 1,000 agents each behind 50 controllers, all on the controller socket. Each worker reuses the batch of agents and rooms kept from earlier nights (`--keep`/`--reuse`, ids in S3), and the run makes them again when they are missing. The workers step their rates together: four realistic steps, one message per agent every 30, 10, 5 and 3 minutes for 10 minutes each, then a ramp from 150 messages a second, a minute a step, until a step breaks. It adds the workers up by window (`combine`), stores the night in S3, sends the numbers to Datadog (`switch.loadtest.*`) and posts the report to a webhook and, when one is set, a Switch room, as an agent in it, then puts staging back to its day size.
+- `down` (05:00) puts the day sizes back whatever happened, and reports a night that left no result.
+
+Inside the cluster the harness runs with `--kube-context ''` (kubectl uses the pod's service account) and reaches switch-core through its Service, which `--api-url` accepts as the namespace's own.
+
+Everything `nightly.py` needs comes from `LOADTEST_*` variables, which the chart sets; `Config.from_env` lists them. It is the only part of the harness that changes anything: switch-core's replica count and two node groups' sizes, all named in that configuration.
+
+## Cleaning up
+
+Teardown runs at the end of every run, and also after Ctrl-C or a failure: agents deleted, their API keys deleted, rooms archived, and the run's registration key deleted. A second Ctrl-C abandons the teardown. If that happens, or the process is killed, remove what is left by name:
+
+```sh
+uv run --project core python loadtest/run.py --namespace <namespace> --cleanup <run_id>
+uv run --project core python loadtest/run.py --namespace <namespace> --cleanup '*' # every load run
+```
+
+The cleanup prints what it removed and what is still there; a non-zero exit means something was left.
+
+## Checks
+
+```sh
+uv run --project core pytest loadtest/
+```
+
+These check that the same seed gives the same rooms and schedule, that a rate sweep plans one window per rate and knows when a window is broken, that the pilot and prod are refused, and how a night adds up, reports and cleans up after itself. They make no requests.
diff --git a/loadtest/agents.py b/loadtest/agents.py
new file mode 100644
index 000000000..c699b24b0
--- /dev/null
+++ b/loadtest/agents.py
@@ -0,0 +1,417 @@
+"""Fake agents on the agent WebSocket, and the traffic between them.
+
+Each agent opens `/agents/{id}/connection/ws` the way the agent runtime does
+(console/packages/switch-agent-runtime/src/event-stream.ts): agent-protocol
+9, its rooms declared, `start_at` its position, every `ping` answered with a
+`pong` carrying its cursor and position. It reads frames and does nothing
+else unless the traffic asks it to post or to read a room's context. A
+dropped socket is reopened from the last position with backoff, and counted.
+With `retry="runtime"` it is reopened the way the real runtime does it: every
+250 to 750 ms for a minute after the server closed it with 1012 (restarting),
+otherwise a doubling backoff with no jitter, and never again once a reattach
+is refused as taken over. With `honor_busy`, a busy server's `retry_after_s`
+is waited out first, as the runtime does from the open gate's release on.
+
+A message's text starts `lt:<token>:`; a receiver that sees one of its own
+run's tokens records the time from the send to its arrival.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import random
+import re
+import time
+import uuid
+from collections import defaultdict
+from dataclasses import dataclass, field
+from urllib.parse import urlencode
+
+import httpx
+import websockets
+from seed import RequestIds, SeededAgent
+from workload import Send, payload, rng_for
+
+OWN_EVENTS = "s"
+PROTOCOL = 9
+TOKEN = re.compile(r"lt:([A-Za-z0-9._-]+):")
+
+
+def furthest(a: dict, b: dict) -> dict:
+ out = dict(a)
+ for k, v in b.items():
+ if isinstance(v, int) and v > out.get(k, -1):
+ out[k] = v
+ return out
+
+
+@dataclass
+class Message:
+ token: str
+ phase: str
+ room: str
+ sender: str
+ size: int
+ sent: float # monotonic, when the POST went out
+ expected: int # connected members other than the sender, at send time
+ acked: float | None = None
+ status: int | None = None
+ latencies: list[float] = field(default_factory=list) # seconds, send to receive
+
+
+@dataclass
+class Stats:
+ phase: str = "setup"
+ messages: dict[str, Message] = field(default_factory=dict)
+ # The same messages and events by window, so reading one window's never
+ # goes through every window before it.
+ by_phase: dict[str, list[Message]] = field(
+ default_factory=lambda: defaultdict(list)
+ )
+ seen: set[tuple[str, str]] = field(default_factory=set)
+ send_errors: dict[str, int] = field(default_factory=lambda: defaultdict(int))
+ reads: dict[str, list[float]] = field(default_factory=lambda: defaultdict(list))
+ read_errors: dict[str, int] = field(default_factory=lambda: defaultdict(int))
+ events: list[dict] = field(default_factory=list) # connects, drops, refusals, gaps
+ events_by_phase: dict[str, list[dict]] = field(
+ default_factory=lambda: defaultdict(list)
+ )
+ lag_max: dict[str, float] = field(default_factory=lambda: defaultdict(float))
+
+ def add(self, msg: Message) -> None:
+ self.messages[msg.token] = msg
+ self.by_phase[msg.phase].append(msg)
+
+ def event(self, kind: str, agent: str, **kw) -> None:
+ e = {"t": time.time(), "phase": self.phase, "kind": kind, "agent": agent, **kw}
+ self.events.append(e)
+ self.events_by_phase[self.phase].append(e)
+
+
+class Agent:
+ # An agent reading its own socket with `scope=all` gets every message in
+ # its rooms. A controller-backed agent gets only the messages that address
+ # it, so its senders @-mention the receivers (controllers.ManagedAgent).
+ mentions_receivers = False
+
+ def __init__(
+ self,
+ seeded: SeededAgent,
+ rooms: list[str],
+ base_url: str,
+ ids: RequestIds,
+ stats: Stats,
+ http: httpx.AsyncClient,
+ read_fraction: float,
+ seed: int,
+ retry: str = "harness",
+ honor_busy: bool = False,
+ ) -> None:
+ self.s = seeded
+ self.retry = retry
+ self.honor_busy = honor_busy
+ self._restarting_until = 0.0
+ self._refusal: dict | None = None
+ self.rooms = rooms
+ self.ws_base = base_url.replace("https://", "wss://", 1).replace(
+ "http://", "ws://", 1
+ )
+ self.ids = ids
+ self.stats = stats
+ self.http = http
+ self.read_fraction = read_fraction
+ self.seed = seed
+ self.connection_id = str(uuid.uuid4())
+ self.position: dict[str, int] = {}
+ self.generation: int | None = None
+ self.connected = asyncio.Event()
+ self.task: asyncio.Task | None = None
+ self.auth = {"Authorization": f"Bearer {seeded.key}"}
+
+ @property
+ def cursor(self) -> int:
+ return self.position.get(OWN_EVENTS, 0)
+
+ def start(self) -> None:
+ self.task = asyncio.create_task(self._run())
+
+ async def stop(self) -> None:
+ if self.task:
+ self.task.cancel()
+ await asyncio.gather(self.task, return_exceptions=True)
+ self.connected.clear()
+
+ def _url(self) -> str:
+ params = {
+ "connection_id": self.connection_id,
+ "scope": "all",
+ "filter": "all",
+ "start_from": str(self.cursor) if self.cursor else "head",
+ "start_at": json.dumps(self.position) if self.position else "head",
+ "protocol": str(PROTOCOL),
+ "protocol_accepts": "1",
+ "client": "switch-loadtest",
+ "client_version": "1",
+ "rooms": ",".join(self.rooms),
+ }
+ if self.generation is not None:
+ params["expected_generation"] = str(self.generation)
+ return f"{self.ws_base}/agents/{self.s.id}/connection/ws?{urlencode(params)}"
+
+ async def _run(self) -> None:
+ backoff = 1.0
+ while True:
+ opened = time.monotonic()
+ why = "closed"
+ close_code: int | None = None
+ try:
+ async with websockets.connect(
+ self._url(),
+ additional_headers={**self.auth, "X-Request-Id": self.ids.next()},
+ open_timeout=30,
+ max_size=None,
+ ping_interval=None, # the server's own ping frames are the heartbeat
+ ) as ws:
+ self._restarting_until = 0.0 # the runtime ends it on an open
+ why = await self._read(ws, opened)
+ close_code = ws.close_code
+ except asyncio.CancelledError:
+ raise
+ except websockets.ConnectionClosed as exc:
+ why = type(exc).__name__
+ close_code = exc.rcvd.code if exc.rcvd is not None else None
+ except Exception as exc: # noqa: BLE001 - every ending is counted and retried
+ why = type(exc).__name__
+ if self.connected.is_set():
+ self.stats.event("drop", self.s.name, why=why, code=close_code)
+ elif why != "taken_over":
+ self.stats.event("connect_failed", self.s.name, why=why)
+ self.connected.clear()
+ if why == "taken_over":
+ return # reopening would itself be a takeover
+ refusal, self._refusal = self._refusal, None
+ detail = (refusal or {}).get("detail")
+ code = detail.get("code") if isinstance(detail, dict) else None
+ if self.retry == "runtime" and code == "taken_over":
+ # The runtime reads a reattach refused as taken over as another
+ # client holding the connection, and stands down for good.
+ self.stats.event("stood_down", self.s.name)
+ return
+ if (
+ self.honor_busy
+ and code == "busy"
+ and isinstance(detail.get("retry_after_s"), (int, float))
+ ):
+ await asyncio.sleep(detail["retry_after_s"])
+ continue
+ if self.retry == "runtime":
+ if close_code == 1012:
+ self._restarting_until = time.monotonic() + 60
+ if time.monotonic() - opened >= 30:
+ backoff = 1.0 # only a stream that lasted resets it
+ if time.monotonic() < self._restarting_until:
+ await asyncio.sleep(0.25 + random.random() * 0.5)
+ continue
+ await asyncio.sleep(backoff)
+ backoff = min(backoff * 2, 30)
+ continue
+ if time.monotonic() - opened > 10:
+ backoff = 1.0
+ await asyncio.sleep(backoff * (0.5 + random.random()))
+ backoff = min(backoff * 2, 30)
+
+ async def _read(self, ws, opened: float) -> str:
+ async for raw in ws:
+ msg = json.loads(raw)
+ ev = msg.get("event")
+ data = msg.get("data") or {}
+ if ev == "ping":
+ await ws.send(
+ json.dumps(
+ {
+ "type": "pong",
+ "cursor": self.cursor,
+ "position": self.position,
+ }
+ )
+ )
+ continue
+ if ev == "refused":
+ self._refusal = data
+ self.stats.event("refused", self.s.name, detail=str(data)[:200])
+ return f"refused {data.get('status')}"
+ if ev == "connection_state":
+ self.generation = data.get("generation", self.generation)
+ if isinstance(data.get("position"), dict):
+ self.position = furthest(self.position, data["position"])
+ if not self.connected.is_set():
+ self.stats.event(
+ "connected",
+ self.s.name,
+ open_s=round(time.monotonic() - opened, 3),
+ )
+ self.connected.set()
+ continue
+ if ev == "evicted":
+ code = str(data.get("code"))
+ self.stats.event("evicted", self.s.name, code=code)
+ return code
+ if ev == "gap":
+ self.stats.event(
+ "gap", self.s.name, reason=str(data.get("reason"))[:120]
+ )
+ if isinstance(data.get("resumed_position"), dict):
+ self.position = data["resumed_position"]
+ continue
+ self._on_event(data)
+ if isinstance(data.get("position"), dict):
+ self.position = furthest(self.position, data["position"])
+ elif msg.get("id") is not None:
+ self.position = furthest(self.position, {OWN_EVENTS: int(msg["id"])})
+ return "closed"
+
+ def _on_event(self, data: dict) -> None:
+ now = time.monotonic()
+ m = TOKEN.search(_text(data))
+ if not m:
+ return
+ token = m.group(1)
+ msg = self.stats.messages.get(token)
+ if (
+ msg is None
+ or msg.sender == self.s.id
+ or (token, self.s.id) in self.stats.seen
+ ):
+ return
+ self.stats.seen.add((token, self.s.id))
+ msg.latencies.append(now - msg.sent)
+ if (
+ self.read_fraction > 0
+ and rng_for(self.seed, "read", token, self.s.index).random()
+ < self.read_fraction
+ ):
+ asyncio.create_task(self.read_context(msg.room, msg.phase))
+
+ async def read_context(self, room: str, phase: str) -> None:
+ t = time.monotonic()
+ status, detail = await self.read(room)
+ if status == 200:
+ self.stats.reads[phase].append(time.monotonic() - t)
+ else:
+ self.stats.read_errors[phase] += 1
+ self.stats.event("read_error", self.s.name, status=status, detail=detail)
+
+ async def read(self, room: str) -> tuple[int, str]:
+ """One `read_context` of `room`: the status, -1 when nothing came back,
+ and what was said about a failure."""
+ try:
+ r = await self.http.post(
+ f"/agents/{self.s.id}/ops/read_context",
+ headers={**self.auth, "X-Request-Id": self.ids.next()},
+ json={"room_id": room, "limit": 20},
+ )
+ except httpx.HTTPError as exc:
+ return -1, type(exc).__name__
+ return r.status_code, "" if r.status_code == 200 else r.text[:200]
+
+ async def post(self, room: str, content: str) -> int:
+ """Post `content` to `room`: the status, -1 when nothing came back."""
+ try:
+ r = await self.http.post(
+ f"/agents/{self.s.id}/message",
+ headers={**self.auth, "X-Request-Id": self.ids.next()},
+ json={"room_id": room, "content": content},
+ )
+ except httpx.HTTPError as exc:
+ self.stats.event("send_error", self.s.name, why=type(exc).__name__)
+ return -1
+ return r.status_code
+
+
+def _text(data: dict) -> str:
+ """Where a room message's text is, across frame shapes: the body or
+ content field when there is one, the whole frame otherwise."""
+ for path in (("payload", "body"), ("body",), ("content",), ("message", "body")):
+ v: object = data
+ for k in path:
+ v = v.get(k) if isinstance(v, dict) else None
+ if isinstance(v, str):
+ return v
+ return json.dumps(data)
+
+
+async def play(
+ schedule: list[Send],
+ agents: list[Agent],
+ room_ids: list[str],
+ membership: list[list[int]],
+ stats: Stats,
+ run_id: str,
+ phase: str,
+ stop: asyncio.Event,
+) -> None:
+ """Post the schedule's messages at their times. Records how far behind
+ schedule the harness itself fell (`lag_max`): when that grows past a few
+ hundred milliseconds the client, not the server, is the limit."""
+ t0 = time.monotonic()
+ tasks: list[asyncio.Task] = []
+ for n, send in enumerate(schedule):
+ delay = t0 + send.at - time.monotonic()
+ if delay > 0:
+ try:
+ await asyncio.wait_for(stop.wait(), delay)
+ break
+ except TimeoutError:
+ pass
+ elif stop.is_set():
+ break
+ stats.lag_max[phase] = max(stats.lag_max[phase], -delay if delay < 0 else 0.0)
+ tasks.append(
+ asyncio.create_task(
+ _post(send, n, agents, room_ids, membership, stats, run_id, phase)
+ )
+ )
+ await asyncio.gather(*tasks, return_exceptions=True)
+
+
+async def _post(
+ send: Send,
+ n: int,
+ agents: list[Agent],
+ room_ids: list[str],
+ membership: list[list[int]],
+ stats: Stats,
+ run_id: str,
+ phase: str,
+) -> None:
+ sender = agents[send.sender]
+ token = f"{run_id}.{phase.replace('@', '.')}.{n}"
+ expected = sum(
+ 1
+ for a in membership[send.room]
+ if a != send.sender and a < len(agents) and agents[a].connected.is_set()
+ )
+ msg = Message(
+ token,
+ phase,
+ room_ids[send.room],
+ sender.s.id,
+ send.size,
+ time.monotonic(),
+ expected,
+ )
+ stats.add(msg)
+ content = payload(token, send.size)
+ if sender.mentions_receivers:
+ names = (
+ agents[a].s.name
+ for a in membership[send.room]
+ if a != send.sender and a < len(agents)
+ )
+ content = " ".join(f"@{n}" for n in names) + " " + content
+ msg.status = await sender.post(msg.room, content)
+ msg.acked = time.monotonic()
+ if msg.status != 200:
+ stats.send_errors[phase] += 1
+ msg.expected = 0
diff --git a/loadtest/controllers.py b/loadtest/controllers.py
new file mode 100644
index 000000000..7841adc5f
--- /dev/null
+++ b/loadtest/controllers.py
@@ -0,0 +1,582 @@
+"""Fake agent controllers: many agents behind one connection each, the way
+most agents reach switch-core (console/packages/agent-controller).
+
+A controller is made through the gateway as the admin, the way Switch Console
+makes one for its own machine (`POST /gateway/management/controllers`). From
+then on it speaks controller protocol 2 on the agent API, as the real one
+does: its credential for an access token, `POST /v1/controllers/{id}/connection`
+naming where each of its agents resumes, then the socket
+(`.../connection/ws`), where every server `ping` is answered with a `pong`
+carrying each agent's cursor. Its agents' events arrive as `agent.event`
+frames tagged with the agent and go to that agent's own receiving code, so a
+delivery is timed exactly as for an agent with a socket of its own.
+
+The rest of what the real controller sends is sent at the same pace: a
+status report on connecting, when its agents change (at most every 5 s) and
+every `report_within_s`, saying the provider is installed and signed in,
+which placing an agent on it needs; the assignment on connecting, on every
+`assignment.changed` and every 10 minutes, with the pending operations. A
+dropped socket is reopened from the last cursors with the real controller's
+backoff (1 s, doubling to 8 s), and counted, per agent, as a drop.
+
+A controller-backed agent has no key or socket of its own: it posts and reads
+with its controller's token (`ManagedAgent`). How, `calls` says: `message`
+posts on `/agents/{id}/message` and reads through the `read_context`
+operation over HTTP; `ops` does both as operations over HTTP, the way the
+real controller relays its agents' tool calls, naming the calling session's
+room in `X-Switch-Room-Id`; `socket` sends the same operations as calls on
+the controller's socket, once its first frame says the server takes them.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import itertools
+import json
+import random
+import time
+from datetime import UTC, datetime
+from urllib.parse import urlencode
+
+import httpx
+import websockets
+from agents import Agent, Stats
+from seed import Admin, RequestIds, SeededAgent
+
+PROTOCOL_HEADER = "Switch-Controller-Protocol"
+PROTOCOL = "2"
+PROVIDER = "claude"
+PLATFORM = {"os": "linux", "arch": "x64", "os_version": "loadtest"}
+# The real controller's pace (console/packages/agent-controller/src/controller.ts).
+STATUS_POLL_S = 5.0
+RESYNC_S = 600.0
+INITIAL_BACKOFF_S = 1.0
+MAX_BACKOFF_S = 8.0
+# Exchanged again this long before the access token (an hour) runs out.
+TOKEN_MARGIN_S = 300.0
+# A call on the socket the server has not answered by then is a failed send,
+# as an HTTP call is after its client's timeout.
+CALL_TIMEOUT_S = 60.0
+
+
+def _now() -> str:
+ return datetime.now(UTC).isoformat().replace("+00:00", "Z")
+
+
+class FakeController:
+ def __init__(
+ self,
+ index: int,
+ controller_id: str,
+ name: str,
+ credential: str,
+ api_base: str,
+ http: httpx.AsyncClient,
+ ids: RequestIds,
+ stats: Stats,
+ honor_busy: bool = False,
+ ) -> None:
+ self.index = index
+ self.id = controller_id
+ self.name = name
+ self._credential = credential # memory only
+ self.ws_base = api_base.replace("https://", "wss://", 1).replace(
+ "http://", "ws://", 1
+ )
+ self.http = http
+ self.ids = ids
+ self.stats = stats
+ self.token: str | None = None # memory only
+ self._expires = 0.0
+ self.agents: dict[str, ManagedAgent] = {}
+ self.cursors: dict[str, int] = {}
+ self.attached = asyncio.Event()
+ self.reported = asyncio.Event() # a status report accepted since connecting
+ self.report_within = 60.0
+ self.revision = 0
+ self._seq = itertools.count(int(time.time() * 1000))
+ self._changed = asyncio.Event() # its agents changed: report soon
+ self._sync = asyncio.Event() # pull the assignment soon
+ self._tasks: list[asyncio.Task] = []
+ self._refusal: dict | None = None
+ self.honor_busy = honor_busy
+ self._ws = None # the socket, while it is open
+ self.takes_calls = False # its first frame offered calls
+ self._call_ids = itertools.count(1)
+ self._answers: dict[int, asyncio.Future] = {}
+
+ def add(self, agent: ManagedAgent) -> None:
+ self.agents[agent.s.id] = agent
+
+ def headers(self) -> dict[str, str]:
+ return {
+ "Authorization": f"Bearer {self.token}",
+ PROTOCOL_HEADER: PROTOCOL,
+ "X-Request-Id": self.ids.next(),
+ }
+
+ def start(self) -> None:
+ self._tasks = [
+ asyncio.create_task(self._run()),
+ asyncio.create_task(self._report_loop()),
+ asyncio.create_task(self._sync_loop()),
+ ]
+
+ async def stop(self) -> None:
+ for t in self._tasks:
+ t.cancel()
+ await asyncio.gather(*self._tasks, return_exceptions=True)
+ self.attached.clear()
+ for a in self.agents.values():
+ a.connected.clear()
+
+ async def ensure_token(self) -> None:
+ if self.token and time.time() < self._expires - TOKEN_MARGIN_S:
+ return
+ r = await self.http.post(
+ f"/v1/management/controllers/{self.id}/token",
+ headers={PROTOCOL_HEADER: PROTOCOL, "X-Request-Id": self.ids.next()},
+ json={"credential": self._credential},
+ )
+ if r.status_code != 200:
+ raise RuntimeError(f"token exchange: HTTP {r.status_code}")
+ body = r.json()
+ self.token = body["access_token"]
+ expires = datetime.fromisoformat(body["expires_at"].replace("Z", "+00:00"))
+ self._expires = expires.timestamp()
+
+ # ── The connection ─────────────────────────────────────────────────────
+
+ async def _run(self) -> None:
+ backoff = INITIAL_BACKOFF_S
+ while True:
+ opened = time.monotonic()
+ why = "closed"
+ close_code: int | None = None
+ told: float | None = None
+ try:
+ await self.ensure_token()
+ r = await self.http.post(
+ f"/v1/controllers/{self.id}/connection",
+ headers=self.headers(),
+ json={
+ "client": "switch-loadtest",
+ "client_version": "1",
+ "cursors": dict(self.cursors),
+ },
+ )
+ if r.status_code != 201:
+ why = f"open HTTP {r.status_code}"
+ told = _busy_retry_after(r)
+ else:
+ body = r.json()
+ query = urlencode(
+ {
+ "connection_id": body["connection_id"],
+ "generation": str(body["generation"]),
+ }
+ )
+ async with websockets.connect(
+ f"{self.ws_base}/v1/controllers/{self.id}/connection/ws?{query}",
+ additional_headers=self.headers(),
+ open_timeout=30,
+ max_size=None,
+ ping_interval=None, # the server's ping frames are the heartbeat
+ ) as ws:
+ self._ws = ws
+ try:
+ why = await self._read(ws, opened)
+ finally:
+ self._ws = None
+ self.takes_calls = False
+ close_code = ws.close_code
+ except asyncio.CancelledError:
+ raise
+ except websockets.ConnectionClosed as exc:
+ why = type(exc).__name__
+ close_code = exc.rcvd.code if exc.rcvd is not None else None
+ except Exception as exc: # noqa: BLE001 - every ending is counted and retried
+ why = type(exc).__name__
+ attached = self.attached.is_set()
+ self._lost(why)
+ if why == "taken_over":
+ return # the real controller stops its agents and exits
+ refusal, self._refusal = self._refusal, None
+ if refusal is not None and refusal.get("code") == "busy":
+ told = refusal.get("retry_after_s")
+ if attached:
+ backoff = INITIAL_BACKOFF_S # a socket that attached starts over
+ # The real controller's waits (agent-controller/src/stream.ts): a
+ # busy server's time when it honours one, one quick retry after a
+ # restart, else half-jittered doubling up to 8 s.
+ if self.honor_busy and isinstance(told, (int, float)):
+ await asyncio.sleep(told)
+ continue
+ if close_code == 1012:
+ await asyncio.sleep(0.25 + random.random() * 0.5)
+ continue
+ await asyncio.sleep(backoff * (0.5 + random.random() * 0.5))
+ backoff = min(backoff * 2, MAX_BACKOFF_S)
+
+ def _lost(self, why: str) -> None:
+ for answer in self._answers.values():
+ if not answer.done():
+ answer.set_exception(ConnectionError(f"socket ended: {why}"))
+ self._answers.clear()
+ if self.attached.is_set():
+ self.stats.event("controller_drop", self.name, why=why)
+ else:
+ self.stats.event("controller_connect_failed", self.name, why=why)
+ self.attached.clear()
+ self.reported.clear()
+ now = time.monotonic()
+ for a in self.agents.values():
+ if a.connected.is_set():
+ # Not a "drop": the socket's own controller_drop is the error,
+ # counted once however many agents it carried.
+ self.stats.event("detached", a.s.name, why=why)
+ a.connected.clear()
+ a.waiting_since = now
+
+ async def _read(self, ws, opened: float) -> str:
+ async for raw in ws:
+ msg = json.loads(raw)
+ ev = msg.get("event")
+ data = msg.get("data") or {}
+ if ev == "ping":
+ await ws.send(json.dumps({"type": "pong", "cursors": self.cursors}))
+ continue
+ if ev == "agent.event":
+ agent_id = data.get("agent_id")
+ seq = data.get("seq")
+ if isinstance(seq, int) and seq > self.cursors.get(agent_id, -1):
+ self.cursors[agent_id] = seq
+ agent = self.agents.get(agent_id)
+ if agent is not None and isinstance(data.get("event"), dict):
+ agent._on_event(data["event"])
+ continue
+ if ev == "call.result":
+ answer = self._answers.pop(data.get("id"), None)
+ if answer is not None and not answer.done():
+ answer.set_result((data.get("status"), data.get("body")))
+ continue
+ if ev == "connection_state":
+ self.takes_calls = isinstance(data.get("calls"), dict)
+ self.report_within = float(data.get("report_within_s") or 60)
+ self.revision = int(data.get("assignment_revision") or 0)
+ if not self.attached.is_set():
+ self.stats.event(
+ "controller_connected",
+ self.name,
+ open_s=round(time.monotonic() - opened, 3),
+ )
+ self.attached.set()
+ self._changed.set()
+ self._sync.set()
+ continue
+ if ev == "agent.attached":
+ self._attach(data)
+ continue
+ if ev == "agent.gap":
+ agent = self.agents.get(data.get("agent_id"))
+ self.stats.event(
+ "gap",
+ agent.s.name if agent else self.name,
+ reason=str(data.get("reason"))[:120],
+ )
+ if isinstance(data.get("resumed_at"), int):
+ self.cursors[data["agent_id"]] = data["resumed_at"]
+ continue
+ if ev == "agent.detached":
+ agent = self.agents.get(data.get("agent_id"))
+ if agent is not None and agent.connected.is_set():
+ self.stats.event(
+ "detached", agent.s.name, reason=str(data.get("reason"))
+ )
+ agent.connected.clear()
+ self._changed.set()
+ continue
+ if ev == "assignment.changed":
+ self._sync.set()
+ continue
+ if ev == "evicted":
+ code = str(data.get("code"))
+ self.stats.event("controller_evicted", self.name, code=code)
+ return code
+ if ev == "refused":
+ detail = data.get("detail")
+ self._refusal = detail if isinstance(detail, dict) else None
+ self.stats.event("refused", self.name, detail=str(data)[:200])
+ return f"refused {data.get('status')}"
+ if ev == "credential.revoked":
+ return "credential_revoked"
+ # agent.rooms, agent.warm, agent.session_command,
+ # agent.approval_outcome, operation.pending: nothing to run here.
+ return "closed"
+
+ def _attach(self, data: dict) -> None:
+ agent_id = data.get("agent_id")
+ from_seq = data.get("from_seq")
+ if isinstance(from_seq, int) and from_seq > self.cursors.get(agent_id, -1):
+ self.cursors[agent_id] = from_seq
+ agent = self.agents.get(agent_id)
+ if agent is None or agent.connected.is_set():
+ return
+ waited = time.monotonic() - (agent.waiting_since or time.monotonic())
+ self.stats.event("connected", agent.s.name, open_s=round(waited, 3))
+ agent.connected.set()
+ self._changed.set()
+
+ async def call(
+ self, agent_id: str, operation: str, arguments: dict, room_id: str | None
+ ) -> tuple[int, object]:
+ """One of its agents' operations as a call on the socket: the status and
+ body it is answered with. Raises ConnectionError when the socket is
+ not open or takes no calls, ends before the answer, or gives none
+ within CALL_TIMEOUT_S."""
+ ws = self._ws
+ if ws is None or not self.takes_calls:
+ raise ConnectionError("no socket that takes calls")
+ id_ = next(self._call_ids)
+ answer = asyncio.get_running_loop().create_future()
+ self._answers[id_] = answer
+ try:
+ await ws.send(
+ json.dumps(
+ {
+ "type": "call",
+ "id": id_,
+ "agent_id": agent_id,
+ "operation": operation,
+ "arguments": arguments,
+ "room_id": room_id,
+ "request_id": self.ids.next(),
+ }
+ )
+ )
+ return await asyncio.wait_for(answer, CALL_TIMEOUT_S)
+ except websockets.ConnectionClosed as exc:
+ raise ConnectionError(type(exc).__name__) from exc
+ except TimeoutError as exc:
+ raise ConnectionError(f"no answer in {CALL_TIMEOUT_S:g}s") from exc
+ finally:
+ self._answers.pop(id_, None)
+
+ # ── What the controller sends besides its beat ─────────────────────────
+
+ async def report(self) -> None:
+ await self.ensure_token()
+ now = _now()
+ body = {
+ "seq": next(self._seq),
+ "observed_at": now,
+ "controller": {
+ "version": "loadtest",
+ "protocol": int(PROTOCOL),
+ "assignment_revision": self.revision,
+ "updated_by": "console",
+ },
+ "machine": {
+ "platform": PLATFORM,
+ "disk_free_bytes": 50 * 2**30,
+ "disk_total_bytes": 100 * 2**30,
+ "mem_free_bytes": 8 * 2**30,
+ "mem_total_bytes": 16 * 2**30,
+ "sessions_running": 0,
+ "sessions_max": 40,
+ "workspaces_dir": "/srv/loadtest",
+ },
+ "providers": [
+ {
+ "provider": PROVIDER,
+ "installed": True,
+ "version": "loadtest",
+ "auth": "ok",
+ "auth_source": "loadtest",
+ "checked_at": now,
+ }
+ ],
+ "tools": [],
+ "agents": [
+ {
+ "agent_id": a.s.id,
+ "applied_revision": None,
+ "process": "running",
+ "attached": True,
+ "sessions": {"active": 0, "ids": []},
+ "restarts_10m": 0,
+ "oom_kills": 0,
+ "since": now,
+ }
+ for a in self.agents.values()
+ if a.connected.is_set()
+ ],
+ }
+ r = await self.http.put(
+ f"/v1/management/controllers/{self.id}/status",
+ headers=self.headers(),
+ json=body,
+ )
+ if r.status_code != 200:
+ self.stats.event("status_failed", self.name, status=r.status_code)
+ return
+ answer = r.json()
+ self.report_within = float(answer.get("report_within_s") or self.report_within)
+ self.reported.set()
+
+ async def _report_loop(self) -> None:
+ last = 0.0
+ while True:
+ await self.attached.wait()
+ due = time.monotonic() - last >= self.report_within
+ if due or self._changed.is_set() or not self.reported.is_set():
+ self._changed.clear()
+ last = time.monotonic()
+ try:
+ await self.report()
+ except (httpx.HTTPError, RuntimeError) as exc:
+ self.stats.event("status_failed", self.name, why=type(exc).__name__)
+ await asyncio.sleep(STATUS_POLL_S)
+
+ async def _sync_loop(self) -> None:
+ last = time.monotonic()
+ while True:
+ try:
+ await asyncio.wait_for(
+ self._sync.wait(),
+ timeout=max(0.0, RESYNC_S - (time.monotonic() - last)),
+ )
+ except TimeoutError:
+ pass
+ self._sync.clear()
+ last = time.monotonic()
+ if not self.attached.is_set():
+ continue
+ try:
+ await self.ensure_token()
+ await self.http.get(
+ f"/v1/management/controllers/{self.id}/assignment",
+ headers=self.headers(),
+ )
+ await self.http.get(
+ f"/v1/management/controllers/{self.id}/operations",
+ headers=self.headers(),
+ params={"state": "pending"},
+ )
+ except (httpx.HTTPError, RuntimeError) as exc:
+ self.stats.event("sync_failed", self.name, why=type(exc).__name__)
+
+
+def _busy_retry_after(r: httpx.Response) -> float | None:
+ """The wait a busy server named in a refused open, or None."""
+ try:
+ error = r.json().get("error") or {}
+ except ValueError:
+ return None
+ after = error.get("retry_after_s")
+ if error.get("code") == "busy" and isinstance(after, (int, float)):
+ return float(after)
+ return None
+
+
+class ManagedAgent(Agent):
+ """An agent behind a fake controller, with no socket of its own.
+
+ Starting it places it on its controller, as moving an agent onto a machine
+ in the gateway does (`PATCH /gateway/management/agents/{id}`); the
+ controller's stream then attaches it, and it counts as connected from that
+ `agent.attached` frame. It posts and reads with the controller's token.
+ It receives only the messages that address it, so its messages @-mention
+ the room's other members, as a person would to bring their agents in.
+ """
+
+ mentions_receivers = True
+
+ def __init__(
+ self,
+ seeded: SeededAgent,
+ rooms: list[str],
+ ids: RequestIds,
+ stats: Stats,
+ http: httpx.AsyncClient,
+ read_fraction: float,
+ seed: int,
+ controller: FakeController,
+ admin: Admin,
+ calls: str = "message",
+ ) -> None:
+ self.controller = controller
+ super().__init__(
+ seeded, rooms, "http://unused", ids, stats, http, read_fraction, seed
+ )
+ self.admin = admin
+ self.calls = calls
+ self.waiting_since: float | None = None
+ controller.add(self)
+
+ @property
+ def auth(self) -> dict[str, str]:
+ return {"Authorization": f"Bearer {self.controller.token}"}
+
+ @auth.setter
+ def auth(self, value: dict[str, str]) -> None:
+ pass # the controller's token, never a key of its own
+
+ def start(self) -> None:
+ self.task = asyncio.create_task(self._place())
+
+ async def post(self, room: str, content: str) -> int:
+ if self.calls == "message":
+ return await super().post(room, content)
+ status, _ = await self._operation("post_message", {"body": content}, room)
+ return status
+
+ async def read(self, room: str) -> tuple[int, str]:
+ if self.calls == "message":
+ return await super().read(room)
+ status, body = await self._operation(
+ "read_context", {"room_id": room, "limit": 20}, room
+ )
+ return status, "" if status == 200 else str(body)[:200]
+
+ async def _operation(
+ self, operation: str, arguments: dict, room: str | None
+ ) -> tuple[int, object]:
+ """An operation as the real controller relays one: the calling
+ session's room named with it, over HTTP or on the socket."""
+ if self.calls == "socket":
+ try:
+ return await self.controller.call(self.s.id, operation, arguments, room)
+ except ConnectionError as exc:
+ self.stats.event("send_error", self.s.name, why=str(exc)[:80])
+ return -1, str(exc)
+ headers = {**self.auth, "X-Request-Id": self.ids.next()}
+ if room is not None:
+ headers["X-Switch-Room-Id"] = room
+ try:
+ r = await self.http.post(
+ f"/agents/{self.s.id}/ops/{operation}", headers=headers, json=arguments
+ )
+ except httpx.HTTPError as exc:
+ self.stats.event("send_error", self.s.name, why=type(exc).__name__)
+ return -1, type(exc).__name__
+ return r.status_code, r.text[:200]
+
+ async def _place(self) -> None:
+ backoff = 1.0
+ while True:
+ self.waiting_since = time.monotonic()
+ try:
+ r = await self.admin.call(
+ "PATCH",
+ f"/management/agents/{self.s.id}",
+ json={"controller_id": self.controller.id},
+ )
+ if r.status_code == 200:
+ return
+ why = f"place HTTP {r.status_code} {r.text[:120]}"
+ except httpx.HTTPError as exc:
+ why = f"place {type(exc).__name__}"
+ self.stats.event("connect_failed", self.s.name, why=why)
+ await asyncio.sleep(backoff * (0.5 + random.random()))
+ backoff = min(backoff * 2, 30)
diff --git a/loadtest/kube.py b/loadtest/kube.py
new file mode 100644
index 000000000..98aceb794
--- /dev/null
+++ b/loadtest/kube.py
@@ -0,0 +1,314 @@
+"""Everything the harness asks of Kubernetes, all of it read-only.
+
+`kubectl get` to find the switch-core pods, the Secret holding the admin
+login and the hostnames of the namespace's Ingresses and Services, `kubectl
+exec` to run the sampler inside the pods, and, only with `--via
+port-forward`, `kubectl port-forward` to reach the pods without the tailnet.
+Nothing here creates, changes, scales or restarts a Kubernetes object.
+
+With no context, kubectl uses the service account of the pod it runs in, for
+a run from inside the cluster.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import base64
+import json
+import re
+import subprocess
+import threading
+import time
+from collections import defaultdict
+from dataclasses import dataclass, field
+from pathlib import Path
+from urllib.parse import urlparse
+
+CORE_SELECTOR = "app.kubernetes.io/component=switch-core"
+CORE_CONTAINER = "switch-core"
+SAMPLER = Path(__file__).with_name("pod_sampler.py")
+
+# The pilot's namespace and its public hostnames. A load test never goes there.
+PILOT_NAMESPACE = "switch"
+PILOT_HOST_LABELS = {"switch-gateway", "switch-api"}
+
+
+class Refused(SystemExit):
+ """A target the harness will not run against."""
+
+
+def guard_target(namespace: str, *urls: str) -> None:
+ """Refuse the pilot and anything that looks like prod, before any request."""
+ if namespace == PILOT_NAMESPACE:
+ raise Refused(f"refusing namespace {namespace!r}: that is the pilot")
+ if "prod" in namespace.lower():
+ raise Refused(f"refusing namespace {namespace!r}: looks like prod")
+ for url in urls:
+ if not url:
+ continue
+ host = (urlparse(url).hostname or "").lower()
+ labels = host.split(".")
+ if labels[0] in PILOT_HOST_LABELS:
+ raise Refused(f"refusing {url}: that is the pilot's hostname")
+ # <service>.<namespace>[.svc...] inside the cluster.
+ if len(labels) > 1 and labels[1] == PILOT_NAMESPACE:
+ raise Refused(f"refusing {url}: that is a Service of the pilot")
+ if "prod" in host:
+ raise Refused(f"refusing {url}: looks like prod")
+
+
+class Kube:
+ def __init__(self, context: str, namespace: str) -> None:
+ self.context = context
+ self.namespace = namespace
+ context_args = ["--context", context] if context else []
+ self.base = ["kubectl", *context_args, "-n", namespace]
+
+ def get_json(self, *args: str) -> dict:
+ out = subprocess.run(
+ [*self.base, "get", *args, "-o", "json"],
+ capture_output=True,
+ text=True,
+ timeout=60,
+ )
+ if out.returncode != 0:
+ raise RuntimeError(
+ f"kubectl get {' '.join(args)}: {out.stderr.strip()[:300]}"
+ )
+ return json.loads(out.stdout)
+
+ def core_pods(self) -> list[dict]:
+ """The switch-core pods that are running, by name."""
+ items = self.get_json("pods", "-l", CORE_SELECTOR)["items"]
+ pods = []
+ for p in items:
+ if p["status"].get("phase") != "Running" or p["metadata"].get(
+ "deletionTimestamp"
+ ):
+ continue
+ statuses = p["status"].get("containerStatuses") or []
+ pods.append(
+ {
+ "name": p["metadata"]["name"],
+ "ready": all(s.get("ready") for s in statuses),
+ "restarts": sum(s.get("restartCount", 0) for s in statuses),
+ "node": p["spec"].get("nodeName"),
+ }
+ )
+ return sorted(pods, key=lambda p: p["name"])
+
+ def core_env(self) -> dict[str, dict]:
+ """The switch-core container's environment, as declared (no values
+ resolved): name to either {"value": ...} or {"secret": (name, key)}."""
+ deploys = self.get_json("deployments", "-l", CORE_SELECTOR)["items"]
+ if not deploys:
+ raise RuntimeError(
+ f"no switch-core deployment in namespace {self.namespace}"
+ )
+ containers = deploys[0]["spec"]["template"]["spec"]["containers"]
+ core = next(
+ (c for c in containers if c["name"] == CORE_CONTAINER), containers[0]
+ )
+ env: dict[str, dict] = {}
+ for e in core.get("env") or []:
+ ref = (e.get("valueFrom") or {}).get("secretKeyRef")
+ if ref:
+ env[e["name"]] = {"secret": (ref["name"], ref["key"])}
+ elif "value" in e:
+ env[e["name"]] = {"value": e["value"]}
+ return env
+
+ def admin_login(self) -> tuple[str, str]:
+ """The gateway admin's email and password, from the Secret the
+ switch-core pod reads them from. Kept in memory only."""
+ env = self.core_env()
+ values = []
+ for name in ("GATEWAY_ADMIN_EMAIL", "GATEWAY_ADMIN_PASSWORD"):
+ ref = env.get(name, {}).get("secret")
+ if ref is None:
+ raise RuntimeError(
+ f"switch-core does not read {name} from a Secret here"
+ )
+ secret, key = ref
+ data = self.get_json("secret", secret).get("data") or {}
+ if key not in data:
+ raise RuntimeError(f"Secret {secret} has no key {key}")
+ values.append(base64.b64decode(data[key]).decode())
+ return values[0], values[1]
+
+ def public_urls(self) -> tuple[str | None, str | None]:
+ """The gateway and API URLs this release was deployed with
+ (FRONTEND_BASE_URL and GATEWAY_PUBLIC_URL, from the release's values)."""
+ env = self.core_env()
+ gw = env.get("FRONTEND_BASE_URL", {}).get("value")
+ api = env.get("GATEWAY_PUBLIC_URL", {}).get("value")
+ return gw, api
+
+ def ingress_hosts(self) -> set[str]:
+ """The hostnames this namespace's Ingresses answer on, the tailnet
+ ones included, as their load balancers report them."""
+ return {
+ lb["hostname"]
+ for i in self.get_json("ingress")["items"]
+ for lb in (i.get("status") or {}).get("loadBalancer", {}).get("ingress")
+ or []
+ if lb.get("hostname")
+ }
+
+ def service_hosts(self) -> set[str]:
+ """The names this namespace's Services answer on inside the cluster,
+ short and qualified, for a run from a pod there."""
+ hosts: set[str] = set()
+ for s in self.get_json("services")["items"]:
+ name = s["metadata"]["name"]
+ ns = self.namespace
+ hosts |= {name, f"{name}.{ns}", f"{name}.{ns}.svc"}
+ hosts.add(f"{name}.{ns}.svc.cluster.local")
+ return hosts
+
+
+@dataclass
+class Sampler:
+ """Runs pod_sampler.py in every switch-core pod and collects its lines.
+
+ One `pod` sampler per pod (CPU and memory), and one `db` sampler in one of
+ them for the database-wide numbers. Pods are looked up again every
+ `rediscover` seconds: a pod that appears gets a sampler, one that goes
+ away loses it, and if the pod running the database sampler goes, another
+ one takes it over.
+ """
+
+ kube: Kube
+ interval: float = 1.0
+ rediscover: float = 10.0
+ phase: str = "setup"
+ samples: list[dict] = field(default_factory=list)
+ # The same samples by window, as each line is tagged with one.
+ by_phase: dict[str, list[dict]] = field(default_factory=lambda: defaultdict(list))
+ pods_seen: dict[str, dict] = field(default_factory=dict)
+ _procs: dict[tuple[str, str], asyncio.subprocess.Process] = field(
+ default_factory=dict
+ )
+ _tasks: list[asyncio.Task] = field(default_factory=list)
+
+ async def run(self) -> None:
+ src = SAMPLER.read_bytes()
+ while True:
+ try:
+ pods = await asyncio.to_thread(self.kube.core_pods)
+ except Exception as exc: # noqa: BLE001 - keep sampling through a blip
+ self.samples.append(
+ {"kind": "error", "ts": time.time(), "error": str(exc)}
+ )
+ pods = None
+ if pods is not None:
+ names = [p["name"] for p in pods]
+ for p in pods:
+ self.pods_seen[p["name"]] = p
+ for key in [k for k in self._procs if k[0] not in names]:
+ self._stop(key)
+ for name in names:
+ if (name, "pod") not in self._procs:
+ await self._start(name, "pod", src)
+ if names and not any(k[1] == "db" for k in self._procs):
+ await self._start(names[0], "db", src)
+ await asyncio.sleep(self.rediscover)
+
+ async def _start(self, pod: str, mode: str, src: bytes) -> None:
+ proc = await asyncio.create_subprocess_exec(
+ *self.kube.base,
+ "exec",
+ "-i",
+ pod,
+ "-c",
+ CORE_CONTAINER,
+ "--",
+ "python",
+ "-",
+ mode,
+ str(self.interval),
+ stdin=asyncio.subprocess.PIPE,
+ stdout=asyncio.subprocess.PIPE,
+ stderr=asyncio.subprocess.DEVNULL,
+ )
+ assert proc.stdin is not None
+ proc.stdin.write(src)
+ proc.stdin.close()
+ self._procs[(pod, mode)] = proc
+ self._tasks.append(asyncio.create_task(self._read(pod, mode, proc)))
+
+ async def _read(
+ self, pod: str, mode: str, proc: asyncio.subprocess.Process
+ ) -> None:
+ assert proc.stdout is not None
+ while True:
+ line = await proc.stdout.readline()
+ if not line:
+ break
+ try:
+ d = json.loads(line)
+ except ValueError:
+ continue
+ d["kind"] = mode
+ d["pod"] = pod
+ d["phase"] = self.phase
+ self.samples.append(d)
+ self.by_phase[self.phase].append(d)
+ await proc.wait()
+ # Ended on its own (pod gone, exec dropped): forget it, so the next
+ # discovery starts it again if the pod is still there.
+ if self._procs.get((pod, mode)) is proc:
+ del self._procs[(pod, mode)]
+
+ def _stop(self, key: tuple[str, str]) -> None:
+ proc = self._procs.pop(key, None)
+ if proc and proc.returncode is None:
+ proc.kill()
+
+ async def close(self) -> None:
+ for key in list(self._procs):
+ self._stop(key)
+ for t in self._tasks:
+ t.cancel()
+ await asyncio.gather(*self._tasks, return_exceptions=True)
+
+
+@dataclass
+class PortForward:
+ """`kubectl port-forward` to each switch-core pod, for a Mac that is not on
+ the tailnet. It goes through the Kubernetes API server, which adds latency
+ and a bottleneck of its own: fine for a smoke run, not for capacity numbers."""
+
+ kube: Kube
+ urls: dict[str, str] = field(default_factory=dict)
+ _procs: list[subprocess.Popen] = field(default_factory=list)
+
+ def open(self) -> list[str]:
+ for pod in self.kube.core_pods():
+ proc = subprocess.Popen(
+ [*self.kube.base, "port-forward", f"pod/{pod['name']}", ":8000"],
+ stdout=subprocess.PIPE,
+ stderr=subprocess.DEVNULL,
+ text=True,
+ )
+ self._procs.append(proc)
+ assert proc.stdout is not None
+ deadline = time.monotonic() + 20
+ while time.monotonic() < deadline:
+ line = proc.stdout.readline()
+ m = re.search(r"127\.0\.0\.1:(\d+)", line or "")
+ if m:
+ self.urls[pod["name"]] = f"http://127.0.0.1:{m.group(1)}"
+ # kubectl prints a line per connection it handles; keep
+ # reading them, or the pipe fills and kubectl stalls.
+ threading.Thread(
+ target=lambda out=proc.stdout: [None for _ in out], daemon=True
+ ).start()
+ break
+ else:
+ raise RuntimeError(f"port-forward to {pod['name']} did not come up")
+ return list(self.urls.values())
+
+ def close(self) -> None:
+ for p in self._procs:
+ p.terminate()
diff --git a/loadtest/nightly.py b/loadtest/nightly.py
new file mode 100644
index 000000000..cc0e75af1
--- /dev/null
+++ b/loadtest/nightly.py
@@ -0,0 +1,1428 @@
+"""The load test every night on staging, run from pods inside the cluster.
+
+Three commands, each one CronJob of the deployment's load test chart:
+
+ up Before the run, from a small pod on the shared nodes: staging's node
+ group to its night size and the harness node up, then switch-core to
+ its night pod count, waiting until every pod is ready.
+ run On the harness node: the agents kept from earlier nights (made again
+ when they are missing), an idle window, the realistic steps (one
+ message per agent every so many minutes), then the ramp until a step
+ breaks. Adds the workers' numbers up, stores the night in S3, sends
+ the numbers to Datadog and the report to the webhook and the Switch
+ room, then puts staging back to its day size.
+ down The safety net, after the run's deadline: the day sizes again,
+ whatever happened, and a report for a night that left no result.
+
+ python loadtest/nightly.py up|run|down
+
+Everything it needs comes from the environment, which the chart sets (see
+Config.from_env). This is the one part of the harness that changes anything in
+Kubernetes or AWS: switch-core's pod count and two node groups' sizes, all
+named in that environment. The harness itself stays read-only, and refuses
+the pilot's namespace and hostnames.
+
+A run is one batch of agents per worker process, because one process tops out
+around 60 messages a second: worker i always uses seed i and the agents of
+batch i, so every night loads the same agents and rooms, and its numbers
+compare with the night before.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import re
+import signal
+import statistics
+import subprocess
+import sys
+import time
+from dataclasses import dataclass, field
+from datetime import UTC, datetime, timedelta
+from pathlib import Path
+
+import httpx
+
+try:
+ import boto3
+except ImportError: # the pods add it (`uv run --with boto3`); the tests run without
+ boto3 = None
+
+HERE = Path(__file__).parent
+RUNS = HERE / "runs"
+sys.path.insert(0, str(HERE))
+
+from kube import guard_target # noqa: E402
+from run import broken as window_broken # noqa: E402
+
+# A worker's log lines that say where it is.
+WAITING = "waiting for"
+FAILED = ("run failed", "Traceback", "refused:")
+BROKE = " broke ("
+# Seconds between starting one worker and the next, so their sign-ins and
+# workspace switches do not overlap, nor their agents' first connects.
+STAGGER = 3.0
+CONNECTED = re.compile(r"step \d+: (\d+)/(\d+) agents connected")
+# Below this share of its agents connected, a worker's kept batch is taken as
+# gone (deleted, or the database replaced), and the night makes them again.
+CONNECTED_ENOUGH = 0.99
+
+
+def _list(value: str, kind=float) -> list:
+ return [kind(x) for x in value.split(",") if x.strip()]
+
+
+@dataclass(frozen=True)
+class Config:
+ """The night's settings, from the environment the chart gives the pods."""
+
+ namespace: str
+ deployment: str
+ api_url: str
+ workspace: str
+ cluster: str
+ region: str
+ # Node groups by name prefix: EKS adds a creation suffix to the names.
+ core_nodegroup: str
+ harness_nodegroup: str
+ night_nodes: int
+ day_nodes: int
+ night_pods: int
+ day_pods: int
+ workers: int
+ agents: int
+ controllers: int
+ room_size: int
+ idle_seconds: float
+ # One message per agent every this many minutes, one step each.
+ realistic_minutes: list[float]
+ realistic_seconds: float
+ # Messages a second over every agent, one step each, until one breaks.
+ ramp: list[float]
+ ramp_seconds: float
+ drain_seconds: float
+ # UTC, HH:MM: workers still running then are stopped.
+ deadline: str
+ bucket: str
+ prefix: str
+ rds_instance: str
+ environment: str
+ # Empty inside the cluster (the pod's service account); a context name for
+ # a rehearsal from a machine with a kubeconfig.
+ kube_context: str = ""
+ dd_api_key: str = field(repr=False, default="")
+ dd_site: str = "datadoghq.com"
+ webhook_url: str = field(repr=False, default="")
+ # A Switch room the report also goes to, posted by an agent in that room
+ # with its own token. Empty, no post.
+ switch_url: str = ""
+ switch_agent_id: str = ""
+ switch_room_id: str = ""
+ switch_token: str = field(repr=False, default="")
+ dashboard_url: str = ""
+
+ @classmethod
+ def from_env(cls, env: dict | None = None) -> Config:
+ e = os.environ if env is None else env
+
+ def get(name: str, default: str | None = None) -> str:
+ value = e.get(f"LOADTEST_{name}", default)
+ if value is None:
+ raise SystemExit(f"LOADTEST_{name} is not set")
+ return value
+
+ cfg = cls(
+ namespace=get("NAMESPACE"),
+ deployment=get("DEPLOYMENT"),
+ api_url=get("API_URL"),
+ workspace=get("WORKSPACE", "load-tests"),
+ cluster=get("CLUSTER"),
+ region=e.get("AWS_REGION") or e.get("AWS_DEFAULT_REGION", ""),
+ core_nodegroup=get("CORE_NODEGROUP"),
+ harness_nodegroup=get("HARNESS_NODEGROUP"),
+ night_nodes=int(get("NIGHT_NODES", "4")),
+ day_nodes=int(get("DAY_NODES", "1")),
+ night_pods=int(get("NIGHT_PODS", "12")),
+ day_pods=int(get("DAY_PODS", "1")),
+ workers=int(get("WORKERS", "10")),
+ agents=int(get("AGENTS", "10000")),
+ controllers=int(get("CONTROLLERS", "500")),
+ room_size=int(get("ROOM_SIZE", "4")),
+ idle_seconds=float(get("IDLE_SECONDS", "120")),
+ realistic_minutes=_list(get("REALISTIC_MINUTES", "30,10,5,3")),
+ realistic_seconds=float(get("REALISTIC_SECONDS", "600")),
+ ramp=_list(get("RAMP", "150,250,300,350,400,450,500")),
+ ramp_seconds=float(get("RAMP_SECONDS", "60")),
+ drain_seconds=float(get("DRAIN_SECONDS", "60")),
+ deadline=get("DEADLINE", "04:30"),
+ bucket=get("BUCKET"),
+ prefix=get("PREFIX", "staging/").rstrip("/") + "/",
+ rds_instance=get("RDS_INSTANCE", ""),
+ environment=get("ENVIRONMENT", "staging"),
+ kube_context=get("KUBE_CONTEXT", ""),
+ dd_api_key=e.get("DD_API_KEY", ""),
+ dd_site=e.get("DD_SITE", "datadoghq.com"),
+ webhook_url=get("WEBHOOK_URL", ""),
+ switch_url=get("SWITCH_URL", ""),
+ switch_agent_id=get("SWITCH_AGENT_ID", ""),
+ switch_room_id=get("SWITCH_ROOM_ID", ""),
+ switch_token=get("SWITCH_TOKEN", ""),
+ dashboard_url=get("DASHBOARD_URL", ""),
+ )
+ cfg.check()
+ return cfg
+
+ def check(self) -> None:
+ # The run's workers refuse the pilot themselves, but `up` and `down`
+ # scale whatever these name before any of them starts.
+ guard_target(self.namespace, self.api_url)
+ if not self.region:
+ raise SystemExit("AWS_REGION is not set")
+ if self.agents % self.workers or self.controllers % self.workers:
+ raise SystemExit(
+ "LOADTEST_AGENTS and _CONTROLLERS must split evenly over the workers"
+ )
+ rates = self.total_rates()
+ if rates != sorted(rates) or len(set(rates)) != len(rates):
+ raise SystemExit(
+ "every realistic step must send less than the first ramp step: "
+ f"{', '.join(f'{r:g}' for r in rates)} msg/s"
+ )
+ if self.day_nodes > self.night_nodes or self.day_pods > self.night_pods:
+ raise SystemExit("the day sizes must not be above the night sizes")
+
+ @property
+ def per_agents(self) -> int:
+ return self.agents // self.workers
+
+ @property
+ def per_controllers(self) -> int:
+ return self.controllers // self.workers
+
+ @property
+ def per_rooms(self) -> int:
+ # The harness's default: half the agents, so with rooms of 4 each
+ # agent is in two rooms.
+ return self.per_agents // 2
+
+ def realistic_rates(self) -> list[float]:
+ """Total messages a second for each realistic step."""
+ return [self.agents / (m * 60) for m in self.realistic_minutes]
+
+ def total_rates(self) -> list[float]:
+ return [*self.realistic_rates(), *self.ramp]
+
+ def worker_rates(self) -> list[float]:
+ """Each worker's share of every step, rounded as the harness names it."""
+ return [round(r / self.workers, 4) for r in self.total_rates()]
+
+ def durations(self) -> list[float]:
+ return [self.realistic_seconds] * len(self.realistic_minutes) + [
+ self.ramp_seconds
+ ] * len(self.ramp)
+
+ def step_labels(self) -> list[str]:
+ return [f"every {m:g} min" for m in self.realistic_minutes] + [
+ f"{r:g} msg/s" for r in self.ramp
+ ]
+
+ def population_shape(self) -> dict:
+ """What decides which agents and rooms the harness makes: a kept batch
+ is only reused by a run of the same shape."""
+ return {
+ "workers": self.workers,
+ "per_agents": self.per_agents,
+ "per_controllers": self.per_controllers,
+ "per_rooms": self.per_rooms,
+ "room_size": self.room_size,
+ "workspace": self.workspace,
+ }
+
+
+# ── Combining the workers ─────────────────────────────────────────────────────
+
+
+def _median(xs: list[float]) -> float | None:
+ return statistics.median(xs) if xs else None
+
+
+def combine(cfg: Config, worker_summaries: list[list[dict]]) -> list[dict]:
+ """One row per measured window, adding the workers up by position: each
+ worker names its windows by its own share of the rate, in the same order.
+ A row is broken when any worker's window is.
+
+ Rates, deliveries, errors and missing deliveries are summed. Delivery p99
+ is the worst worker's, an upper bound on the true combined p99, and p50
+ the median of the workers'. Pod and database figures are the first
+ worker's: every worker samples the same pods and database."""
+ if not worker_summaries:
+ return []
+ n = min(len(s) for s in worker_summaries)
+ labels = ["idle", *cfg.step_labels()]
+ kinds = (
+ ["idle"] + ["realistic"] * len(cfg.realistic_minutes) + ["ramp"] * len(cfg.ramp)
+ )
+ targets = [0.0, *cfg.total_rates()]
+ rows = []
+ for k in range(n):
+ ph = [s[k] for s in worker_summaries]
+ first = ph[0]
+ cpu = [
+ v["cpu_pct_avg"]
+ for v in first["pods"].values()
+ if v.get("cpu_pct_avg") is not None
+ ]
+ rds = first.get("rds") or {}
+ p99 = max((p["delivery_ms_p99"] or 0) for p in ph)
+ errors = sum(p["errors"] for p in ph)
+ missing = sum(p["deliveries_missing"] for p in ph)
+ # Broken when any worker's window broke by the harness's own rule
+ # (run.broken), which is also what stops that worker stepping.
+ why = next(
+ (
+ f"{w} (worker {i + 1})"
+ for i, p in enumerate(ph)
+ if (w := window_broken(p))
+ ),
+ None,
+ )
+ rows.append(
+ {
+ "label": labels[k] if k < len(labels) else first["phase"],
+ "kind": kinds[k] if k < len(kinds) else "extra",
+ "target_per_s": round(targets[k], 2) if k < len(targets) else None,
+ "workers": len(ph),
+ "agents": sum(p["agents_target"] for p in ph),
+ "agents_connected_min": sum(p["agents_connected_min"] or 0 for p in ph),
+ "sent_per_s": round(sum(p["messages_sent_per_s"] or 0 for p in ph), 2),
+ "delivery_ms_p50": _median(
+ [
+ p["delivery_ms_p50"]
+ for p in ph
+ if p["delivery_ms_p50"] is not None
+ ]
+ ),
+ "delivery_ms_p99": p99 or None,
+ "errors": errors,
+ "missing": missing,
+ "pods": len(first["pods"]),
+ "pod_cpu_avg_pct": round(sum(cpu) / len(cpu), 1) if cpu else None,
+ "pod_cpu_busiest_pct": round(max(cpu), 1) if cpu else None,
+ "db_cpu_avg_pct": rds.get("cpu_avg"),
+ "db_cpu_max_pct": rds.get("cpu_max"),
+ "db_xact_per_s": first.get("db_xact_per_s_avg"),
+ "harness_lag_ms_max": max((p["harness_lag_ms_max"] or 0) for p in ph),
+ "stopped_early": any("stopped early" in p["phase"] for p in ph),
+ "broken": why,
+ }
+ )
+ return rows
+
+
+def verdict(rows: list[dict], cfg: Config, failures: list[str]) -> dict:
+ """What the night says: whether realistic load ran clean, and the highest
+ ramp step every worker held. Any of `failures` fails the night."""
+ realistic = [r for r in rows if r["kind"] == "realistic"]
+ ramp = [r for r in rows if r["kind"] == "ramp"]
+ realistic_ok = len(realistic) == len(cfg.realistic_minutes) and all(
+ not r["broken"]
+ and not r["errors"]
+ and not r["missing"]
+ and not r["stopped_early"]
+ for r in realistic
+ )
+ held = [r for r in ramp if not r["broken"] and not r["stopped_early"]]
+ broke = next((r for r in ramp if r["broken"]), None)
+ complete = len(rows) >= 1 + len(cfg.realistic_minutes) and (
+ broke is not None or len(ramp) == len(cfg.ramp)
+ )
+ return {
+ "ok": realistic_ok and complete and not failures,
+ "realistic_ok": realistic_ok,
+ "complete": complete,
+ "held_per_s": held[-1]["target_per_s"] if held else None,
+ "held_sent_per_s": held[-1]["sent_per_s"] if held else None,
+ "broke_at_per_s": broke["target_per_s"] if broke else None,
+ "broke_why": broke["broken"] if broke else None,
+ "realistic_top_p99_ms": realistic[-1]["delivery_ms_p99"] if realistic else None,
+ }
+
+
+def compare(result: dict, history: list[dict]) -> list[str]:
+ """Lines comparing the night with the median of earlier nights, and a
+ warning when it is clearly worse. Needs three earlier nights to say so."""
+ past = [h for h in history if h.get("verdict", {}).get("complete")][-7:]
+ if len(past) < 3:
+ return [f"{len(past)} earlier night(s) on record; comparisons start at 3"]
+ lines = []
+ v = result["verdict"]
+ held = [h["verdict"]["held_per_s"] for h in past if h["verdict"].get("held_per_s")]
+ if held and v.get("held_per_s"):
+ med = statistics.median(held)
+ flag = " ⚠️ lower" if v["held_per_s"] < 0.85 * med else ""
+ lines.append(
+ f"held {v['held_per_s']:g} msg/s, median of the last {len(held)}: {med:g}{flag}"
+ )
+ p99s = [
+ h["verdict"]["realistic_top_p99_ms"]
+ for h in past
+ if h["verdict"].get("realistic_top_p99_ms")
+ ]
+ if p99s and v.get("realistic_top_p99_ms"):
+ med = statistics.median(p99s)
+ flag = " ⚠️ slower" if v["realistic_top_p99_ms"] > 1.5 * med else ""
+ lines.append(
+ f"p99 at the busiest realistic step {v['realistic_top_p99_ms'] / 1000:.1f} s, "
+ f"median {med / 1000:.1f} s{flag}"
+ )
+ return lines
+
+
+# ── The report ────────────────────────────────────────────────────────────────
+
+
+def _ms(v) -> str:
+ if v is None:
+ return "-"
+ return f"{v / 1000:.1f}s" if v >= 10000 else f"{v:.0f}"
+
+
+def _pct(v) -> str:
+ return "-" if v is None else f"{v:.0f}%"
+
+
+def report_text(result: dict, cfg: Config, comparison: list[str]) -> str:
+ """The message for the webhook and the room: under Discord's 2,000
+ characters."""
+ v = result["verdict"]
+ night = f"Switch nightly load test on {cfg.environment}, {cfg.agents:,} agents"
+ if not result["rows"]:
+ head = f"❌ {night}: no numbers tonight"
+ elif v["ok"]:
+ held = (
+ f"held {v['held_per_s']:g} msg/s"
+ if v["held_per_s"]
+ else "held no ramp step"
+ )
+ broke = (
+ f", broke at {v['broke_at_per_s']:g} ({v['broke_why']})"
+ if v["broke_at_per_s"]
+ else ", every ramp step held"
+ )
+ head = f"✅ {night}: {held}{broke}"
+ else:
+ failures = result.get("failures") or []
+ if failures:
+ why = failures[0]
+ elif not v["realistic_ok"]:
+ why = "realistic load did not run clean"
+ else:
+ why = "the run did not finish"
+ head = f"❌ {night}: {why}"
+ lines = [
+ head,
+ f"build {result['build']} · {result['pods']} pods on {result['nodes']} nodes · "
+ f"{cfg.controllers} controllers"
+ + (" · agents made tonight" if result.get("seeded") else ""),
+ ]
+ lines += [f"❌ {f}" for f in (result.get("failures") or [])[1:]]
+ fewest = min((r["agents_connected_min"] for r in result["rows"]), default=None)
+ if fewest is not None and fewest < cfg.agents:
+ lines.append(
+ f"⚠️ as few as {fewest:,} of the {cfg.agents:,} agents were connected "
+ "in a step"
+ )
+ lines += result.get("notes", [])
+ if result["rows"]:
+ table = [
+ f"{'step':<14}{'sent/s':>8}{'p50':>7}{'p99':>8}{'err':>5}{'lost':>6}{'pods':>6}{'db':>6}"
+ ]
+ for r in result["rows"]:
+ table.append(
+ f"{r['label']:<14}{r['sent_per_s']:>8.1f}{_ms(r['delivery_ms_p50']):>7}"
+ f"{_ms(r['delivery_ms_p99']):>8}{r['errors']:>5}{r['missing']:>6}"
+ f"{_pct(r['pod_cpu_avg_pct']):>6}{_pct(r['db_cpu_avg_pct']):>6}"
+ + (" broke" if r["broken"] else "")
+ )
+ lines.append("```\n" + "\n".join(table) + "\n```")
+ lines.append(
+ "p50 and p99 are delivery latency in ms (p99 is the worst worker's); "
+ "pods is switch-core's average CPU, db the database's"
+ )
+ lines += comparison
+ if cfg.dashboard_url:
+ lines.append(cfg.dashboard_url)
+ text = "\n".join(lines)
+ return text if len(text) <= 1990 else text[:1985] + "\n…"
+
+
+def post_webhook(url: str, text: str) -> None:
+ if not url:
+ print("no webhook set; report not sent", flush=True)
+ return
+ # Discord takes `content`; its Slack-compatible endpoint, Slack and
+ # Mattermost take `text`.
+ body = (
+ {"content": text}
+ if "discord" in url and not url.rstrip("/").endswith("/slack")
+ else {"text": text}
+ )
+ httpx.post(url, json=body, timeout=30).raise_for_status()
+
+
+def post_switch(cfg: Config, text: str) -> None:
+ """The report into the Switch room, as the agent the token belongs to."""
+ settings = {
+ "SWITCH_URL": cfg.switch_url,
+ "SWITCH_AGENT_ID": cfg.switch_agent_id,
+ "SWITCH_ROOM_ID": cfg.switch_room_id,
+ "SWITCH_TOKEN": cfg.switch_token,
+ }
+ if not any(settings.values()):
+ print("no Switch room set; report not posted there", flush=True)
+ return
+ missing = [f"LOADTEST_{k}" for k, v in settings.items() if not v]
+ if missing:
+ raise ValueError(f"{', '.join(missing)} not set")
+ httpx.post(
+ f"{cfg.switch_url.rstrip('/')}/agents/{cfg.switch_agent_id}/message",
+ json={"room_id": cfg.switch_room_id, "content": text},
+ headers={"Authorization": f"Bearer {cfg.switch_token}"},
+ timeout=30,
+ ).raise_for_status()
+
+
+def datadog_series(result: dict, cfg: Config, now: float) -> list[dict]:
+ """One point per night for each number, stamped with the report's time:
+ Datadog only takes points from the last hour, and a night is one value."""
+ base = [f"env:{cfg.environment}", "service:switch-loadtest"]
+ run_tags = [*base, f"build:{result['build']}"]
+ series = []
+
+ def point(metric: str, value, tags: list[str]) -> None:
+ if value is None:
+ return
+ series.append(
+ {
+ "metric": f"switch.loadtest.{metric}",
+ "type": 3, # gauge
+ "points": [{"timestamp": int(now), "value": float(value)}],
+ "tags": tags,
+ }
+ )
+
+ v = result["verdict"]
+ point("ok", 1 if v["ok"] else 0, run_tags)
+ point("held_msgs_per_s", v.get("held_per_s") or 0, run_tags)
+ point("pods", result["pods"], run_tags)
+ point("restarts", result.get("restarts"), run_tags)
+ point("duration_s", result.get("duration_s"), run_tags)
+ for i, r in enumerate(result["rows"]):
+ tags = [
+ *base,
+ f"phase:{r['kind']}",
+ f"step:{i:02d}",
+ f"target:{r['label'].replace(' ', '_')}",
+ ]
+ point("step.sent_per_s", r["sent_per_s"], tags)
+ point("step.delivery_p50_ms", r["delivery_ms_p50"], tags)
+ point("step.delivery_p99_ms", r["delivery_ms_p99"], tags)
+ point("step.errors", r["errors"], tags)
+ point("step.missing", r["missing"], tags)
+ point("step.pod_cpu_avg_pct", r["pod_cpu_avg_pct"], tags)
+ point("step.pod_cpu_busiest_pct", r["pod_cpu_busiest_pct"], tags)
+ point("step.db_cpu_avg_pct", r["db_cpu_avg_pct"], tags)
+ point("step.db_xact_per_s", r["db_xact_per_s"], tags)
+ point("step.broken", 1 if r["broken"] else 0, tags)
+ return series
+
+
+def send_datadog(result: dict, cfg: Config, text: str) -> None:
+ if not cfg.dd_api_key:
+ print("no DD_API_KEY; nothing sent to Datadog", flush=True)
+ return
+ headers = {"DD-API-KEY": cfg.dd_api_key}
+ api = f"https://api.{cfg.dd_site}"
+ series = datadog_series(result, cfg, time.time())
+ httpx.post(
+ f"{api}/api/v2/series", json={"series": series}, headers=headers, timeout=30
+ ).raise_for_status()
+ httpx.post(
+ f"{api}/api/v1/events",
+ json={
+ "title": text.splitlines()[0],
+ "text": "%%% \n" + text + "\n %%%",
+ "tags": [
+ f"env:{cfg.environment}",
+ "service:switch-loadtest",
+ f"build:{result['build']}",
+ ],
+ "alert_type": "success" if result["verdict"]["ok"] else "error",
+ "source_type_name": "switch-loadtest",
+ },
+ headers=headers,
+ timeout=30,
+ ).raise_for_status()
+ print(f"sent {len(series)} series and an event to Datadog", flush=True)
+
+
+# ── Kubernetes and AWS ────────────────────────────────────────────────────────
+
+
+def kubectl(cfg: Config, *args: str, timeout: float = 60) -> str:
+ context = ["--context", cfg.kube_context] if cfg.kube_context else []
+ out = subprocess.run(
+ ["kubectl", *context, "-n", cfg.namespace, *args],
+ capture_output=True,
+ text=True,
+ timeout=timeout,
+ )
+ if out.returncode != 0:
+ raise RuntimeError(f"kubectl {' '.join(args)}: {out.stderr.strip()[:300]}")
+ return out.stdout
+
+
+def build_of(labels: dict, image: str) -> str:
+ """The release's chart version (a dev build is 0.31.1-dev.271.g989225d),
+ else its app version, else the image's tag: images may be pinned by
+ digest, which says nothing to a reader."""
+ chart = labels.get("helm.sh/chart", "")
+ if chart.startswith("switch-"):
+ return chart.removeprefix("switch-")
+ return labels.get("app.kubernetes.io/version") or image.rsplit(":", 1)[-1][:24]
+
+
+def _core_image(pod_spec: dict) -> str:
+ containers = pod_spec["containers"]
+ return next(
+ (c["image"] for c in containers if c["name"] == "switch-core"),
+ containers[0]["image"],
+ )
+
+
+def core_state(cfg: Config) -> dict:
+ """switch-core's deployment: the build its ready pods run, the pods and
+ where they are. The build is the pods', not the deployment's: a rollout
+ that cannot finish (no room on a full node group) leaves them on the
+ build before."""
+ d = json.loads(kubectl(cfg, "get", "deployment", cfg.deployment, "-o", "json"))
+ selector = ",".join(
+ f"{k}={v}" for k, v in d["spec"]["selector"]["matchLabels"].items()
+ )
+ pods = json.loads(kubectl(cfg, "get", "pods", "-l", selector, "-o", "json"))[
+ "items"
+ ]
+ running = [p for p in pods if not p["metadata"].get("deletionTimestamp")]
+ ready = [
+ p
+ for p in running
+ if all(s.get("ready") for s in p["status"].get("containerStatuses") or [])
+ and p["status"].get("containerStatuses")
+ ]
+ by_node: dict[str, int] = {}
+ for p in ready:
+ node = p["spec"].get("nodeName") or "?"
+ by_node[node] = by_node.get(node, 0) + 1
+ template = d["spec"]["template"]
+ runs = {
+ (
+ _core_image(p["spec"]),
+ build_of(p["metadata"].get("labels") or {}, _core_image(p["spec"])),
+ )
+ for p in ready
+ } or {
+ (
+ _core_image(template["spec"]),
+ build_of(d["metadata"].get("labels") or {}, _core_image(template["spec"])),
+ )
+ }
+ return {
+ "image": ", ".join(sorted(i for i, _ in runs)),
+ "build": ", ".join(sorted(b for _, b in runs)),
+ "replicas": d["spec"].get("replicas", 0),
+ "ready": len(ready),
+ "updated": d["status"].get("updatedReplicas", 0),
+ "generation_seen": d["status"].get("observedGeneration")
+ == d["metadata"].get("generation"),
+ "by_node": by_node,
+ "restarts": sum(
+ s.get("restartCount", 0)
+ for p in running
+ for s in p["status"].get("containerStatuses") or []
+ ),
+ # Of the ready pods: one a rollout has made, still Pending, changes
+ # nothing the run measures.
+ "pod_template_hash": sorted(
+ {p["metadata"]["labels"].get("pod-template-hash", "") for p in ready}
+ ),
+ }
+
+
+def scale_core(cfg: Config, replicas: int) -> None:
+ kubectl(cfg, "scale", "deployment", cfg.deployment, f"--replicas={replicas}")
+ print(f"switch-core scaled to {replicas}", flush=True)
+
+
+def wait_core(cfg: Config, replicas: int, seconds: float) -> dict:
+ """Wait until switch-core runs `replicas` ready pods of one template.
+ Not for a newer build waiting to roll out: at the night's size there is
+ no room for the rollout's extra pod, so it would only wait out the time."""
+ t = time.monotonic()
+ while True:
+ s = core_state(cfg)
+ if (
+ s["ready"] == replicas
+ and len(s["pod_template_hash"]) == 1
+ and s["generation_seen"]
+ ):
+ return s
+ if time.monotonic() - t > seconds:
+ return s
+ time.sleep(10)
+
+
+def _eks(cfg: Config):
+ return boto3.client("eks", region_name=cfg.region)
+
+
+def nodegroup(cfg: Config, prefix: str) -> str:
+ names = _eks(cfg).list_nodegroups(clusterName=cfg.cluster)["nodegroups"]
+ match = [n for n in names if n.startswith(prefix)]
+ if len(match) != 1:
+ raise RuntimeError(f"expected one node group named {prefix}*, found {match}")
+ return match[0]
+
+
+def size_nodegroup(cfg: Config, prefix: str, desired: int) -> None:
+ """Set a node group's desired size. No prefix, no node group: for a
+ rehearsal in an environment whose nodes are not the test's to size."""
+ if not prefix:
+ return
+ eks = _eks(cfg)
+ name = nodegroup(cfg, prefix)
+ sc = eks.describe_nodegroup(clusterName=cfg.cluster, nodegroupName=name)[
+ "nodegroup"
+ ]["scalingConfig"]
+ if not sc["minSize"] <= desired <= sc["maxSize"]:
+ raise RuntimeError(
+ f"{name} takes {sc['minSize']} to {sc['maxSize']} nodes, not {desired}"
+ )
+ if sc["desiredSize"] != desired:
+ eks.update_nodegroup_config(
+ clusterName=cfg.cluster,
+ nodegroupName=name,
+ scalingConfig={
+ "minSize": sc["minSize"],
+ "maxSize": sc["maxSize"],
+ "desiredSize": desired,
+ },
+ )
+ print(f"node group {name}: {sc['desiredSize']} -> {desired}", flush=True)
+
+
+def _s3(cfg: Config):
+ return boto3.client("s3", region_name=cfg.region)
+
+
+def _local(cfg: Config) -> Path | None:
+ """A bucket named file:///some/dir is that directory, for a rehearsal
+ from a machine without the test's bucket."""
+ if cfg.bucket.startswith("file://"):
+ return Path(cfg.bucket.removeprefix("file://")) / cfg.prefix
+ return None
+
+
+def s3_get(cfg: Config, key: str) -> dict | None:
+ if (d := _local(cfg)) is not None:
+ return json.loads((d / key).read_text()) if (d / key).exists() else None
+ try:
+ body = (
+ _s3(cfg).get_object(Bucket=cfg.bucket, Key=cfg.prefix + key)["Body"].read()
+ )
+ except Exception as exc: # noqa: BLE001 - a missing key reads as nothing
+ if "NoSuchKey" in type(exc).__name__ or "NoSuchKey" in str(exc):
+ return None
+ raise
+ return json.loads(body)
+
+
+def s3_put(cfg: Config, key: str, data: dict | bytes) -> None:
+ body = (
+ data
+ if isinstance(data, bytes)
+ else json.dumps(data, indent=1, default=str).encode()
+ )
+ if (d := _local(cfg)) is not None:
+ (d / key).parent.mkdir(parents=True, exist_ok=True)
+ (d / key).write_bytes(body)
+ return
+ _s3(cfg).put_object(Bucket=cfg.bucket, Key=cfg.prefix + key, Body=body)
+
+
+def s3_delete(cfg: Config, key: str) -> None:
+ if (d := _local(cfg)) is not None:
+ (d / key).unlink(missing_ok=True)
+ return
+ _s3(cfg).delete_object(Bucket=cfg.bucket, Key=cfg.prefix + key)
+
+
+def history(cfg: Config, before: str, nights: int = 14) -> list[dict]:
+ """The results of earlier nights, oldest first."""
+ if (d := _local(cfg)) is not None:
+ names = sorted(f.name for f in (d / "history").glob("*.json"))
+ keys = [f"history/{n}" for n in names if Path(n).stem < before][-nights:]
+ return [h for h in (s3_get(cfg, k) for k in keys) if h]
+ pages = (
+ _s3(cfg)
+ .get_paginator("list_objects_v2")
+ .paginate(Bucket=cfg.bucket, Prefix=cfg.prefix + "history/")
+ )
+ keys = sorted(
+ o["Key"][len(cfg.prefix) :]
+ for page in pages
+ for o in page.get("Contents", [])
+ if o["Key"].endswith(".json")
+ )
+ keys = [k for k in keys if Path(k).stem < before][-nights:]
+ return [h for h in (s3_get(cfg, k) for k in keys) if h]
+
+
+# ── The harness's workers ─────────────────────────────────────────────────────
+
+
+def harness_args(cfg: Config, run_id: str, seed: int) -> list[str]:
+ return [
+ sys.executable, str(HERE / "run.py"),
+ "--namespace", cfg.namespace, "--kube-context", cfg.kube_context,
+ "--workspace", cfg.workspace,
+ "--api-url", cfg.api_url, "--gateway-url", cfg.api_url,
+ "--aws-profile", "", "--aws-region", cfg.region,
+ *(["--rds-instance", cfg.rds_instance] if cfg.rds_instance else []),
+ "--run-id", run_id, "--seed", str(seed),
+ "--steps", str(cfg.per_agents), "--controllers", str(cfg.per_controllers),
+ "--rooms", str(cfg.per_rooms), "--room-size", str(cfg.room_size),
+ "--calls", "socket", "--keep",
+ ] # fmt: skip
+
+
+def deadline_after(start: datetime, hhmm: str, longest: timedelta) -> datetime:
+ """The next HH:MM UTC after `start`, but no later than `start + longest`,
+ so a run started by hand in the day still ends."""
+ h, m = (int(x) for x in hhmm.split(":"))
+ at = start.replace(hour=h, minute=m, second=0, microsecond=0)
+ if at <= start:
+ at += timedelta(days=1)
+ return min(at, start + longest)
+
+
+@dataclass
+class Worker:
+ index: int
+ run_id: str
+ log: Path
+ proc: subprocess.Popen
+
+ def text(self) -> str:
+ try:
+ return self.log.read_text(errors="replace")
+ except FileNotFoundError:
+ return ""
+
+ def failed(self) -> str | None:
+ t = self.text()
+ for marker in FAILED:
+ if marker in t:
+ line = next(x for x in t.splitlines() if marker in x)
+ return line.strip()[:240]
+ if self.proc.poll() is not None and WAITING not in t:
+ return (
+ f"exited ({self.proc.returncode}) before its rates: {self.last_line()}"
+ )
+ return None
+
+ def crashed(self) -> str | None:
+ """Why a worker whose rates had started failed: its own `run failed`
+ line, or an exit that was not clean (a worker that could not revoke
+ its controllers exits 1)."""
+ t = self.text()
+ if "run failed" in t:
+ return next(x for x in t.splitlines() if "run failed" in x).strip()[:240]
+ if self.proc.poll() is not None and self.proc.returncode:
+ return f"exited {self.proc.returncode}: {self.last_line()}"
+ return None
+
+ def last_line(self) -> str:
+ return (self.text().strip().splitlines() or [""])[-1][:200]
+
+ def connected(self) -> tuple[int, int] | None:
+ """(connected, wanted) from the harness's `step N: x/N agents
+ connected` line, once it has printed it."""
+ m = CONNECTED.search(self.text())
+ return (int(m.group(1)), int(m.group(2))) if m else None
+
+ def interrupt(self) -> None:
+ if self.proc.poll() is None:
+ self.proc.send_signal(signal.SIGINT)
+
+
+def start(cfg: Config, run_id: str, seed: int, extra: list[str], out: Path) -> Worker:
+ log = out / f"{run_id}.log"
+ proc = subprocess.Popen(
+ [*harness_args(cfg, run_id, seed), *extra],
+ stdout=log.open("w"),
+ stderr=subprocess.STDOUT,
+ cwd=HERE.parent,
+ )
+ return Worker(seed, run_id, log, proc)
+
+
+def finish(workers: list[Worker], grace: float = 240) -> None:
+ """Interrupt every worker (each stops its traffic and revokes its
+ controllers), and kill what is still running after `grace` seconds."""
+ for w in workers:
+ w.interrupt()
+ t = time.monotonic()
+ while any(w.proc.poll() is None for w in workers) and time.monotonic() - t < grace:
+ time.sleep(2)
+ for w in workers:
+ if w.proc.poll() is None:
+ w.proc.kill()
+
+
+def seed_population(cfg: Config, base: str, out: Path) -> dict:
+ """Make and keep one batch of agents per worker, one batch at a time, so
+ making them never overlaps itself; returns the population record.
+
+ Each batch's ids go to S3 as soon as it is kept, under a population record
+ marked incomplete until the last one is, so a night that fails half way
+ leaves a record the next night removes rather than agents nobody knows of."""
+ pop = {
+ "base": base,
+ "shape": cfg.population_shape(),
+ "batches": 0,
+ "complete": False,
+ "made": datetime.now(UTC).isoformat(timespec="seconds"),
+ }
+ seeder: Worker | None = None
+ try:
+ for i in range(1, cfg.workers + 1):
+ rid = f"{base}s{i}"
+ t = time.monotonic()
+ # Counted before it starts: a batch stopped half way is not kept,
+ # but may have made agents, so its removal is tried too.
+ pop["batches"] = i
+ s3_put(cfg, "population.json", pop)
+ seeder = start(cfg, rid, i, ["--seed-only", "--concurrency", "8"], out)
+ seeder.proc.wait()
+ keep = RUNS / rid / "keep.json"
+ if seeder.proc.returncode != 0 or not keep.exists():
+ last = (seeder.text().strip().splitlines() or [""])[-1]
+ raise RuntimeError(f"batch {i} was not kept: {last[:200]}")
+ s3_put(cfg, f"population/w{i}.json", json.loads(keep.read_text()))
+ print(
+ f"batch {i}: {cfg.per_agents} agents made and kept in {time.monotonic() - t:.0f}s",
+ flush=True,
+ )
+ except BaseException as exc:
+ # A seeder still running would go on making agents after the cleanup
+ # has listed what exists.
+ if seeder is not None:
+ finish([seeder], grace=60)
+ s3_put(cfg, "population.json", pop)
+ if isinstance(exc, Interrupted):
+ # The pod is going, and removing can take longer than it has. The
+ # record, incomplete and counting every batch that may have
+ # agents, is the next night's to remove.
+ raise
+ try:
+ remove_population(cfg, pop, out)
+ except Exception as exc: # noqa: BLE001 - the first error is the one to report
+ print(f"removing the half-made batches failed too: {exc}", flush=True)
+ raise
+ pop["complete"] = True
+ s3_put(cfg, "population.json", pop)
+ return pop
+
+
+def remove_population(cfg: Config, pop: dict, out: Path) -> None:
+ """Delete a population's agents and rooms, one batch at a time, in the
+ workspace it was made in. Its record goes only once every batch is gone,
+ so a removal that fails is tried again the next night."""
+ batches = pop.get("batches", pop["shape"]["workers"])
+ workspace = pop["shape"].get("workspace", cfg.workspace)
+ failed = []
+ for i in range(1, batches + 1):
+ rid = f"{pop['base']}s{i}"
+ cmd = [
+ sys.executable, str(HERE / "run.py"),
+ "--namespace", cfg.namespace, "--kube-context", cfg.kube_context,
+ "--workspace", workspace,
+ "--api-url", cfg.api_url, "--gateway-url", cfg.api_url,
+ "--cleanup", rid,
+ ] # fmt: skip
+ with (out / f"cleanup-{rid}.log").open("w") as log:
+ r = subprocess.run(
+ cmd, stdout=log, stderr=subprocess.STDOUT, cwd=HERE.parent,
+ timeout=1800, check=False,
+ ) # fmt: skip
+ if r.returncode != 0:
+ failed.append(rid)
+ if failed:
+ raise RuntimeError(f"could not remove the kept batches {', '.join(failed)}")
+ s3_delete(cfg, "population.json")
+ print(f"removed population {pop['base']} ({batches} batches)", flush=True)
+
+
+def restore_population(cfg: Config, pop: dict) -> None:
+ """Put the kept batches where the harness's --reuse looks for them."""
+ for i in range(1, cfg.workers + 1):
+ record = s3_get(cfg, f"population/w{i}.json")
+ if record is None:
+ raise RuntimeError(f"population {pop['base']} has no batch {i}")
+ d = RUNS / f"{pop['base']}s{i}"
+ d.mkdir(parents=True, exist_ok=True)
+ (d / "keep.json").write_text(json.dumps(record))
+
+
+def run_workers(
+ cfg: Config, pop: dict, base: str, out: Path, until: datetime, workers: list[Worker]
+) -> tuple[str | None, bool]:
+ """Start every worker on its kept batch, a few seconds apart, wait until
+ all are connected and idle, then let them step their rates together; once
+ a worker breaks a step, stop them all. Each worker goes into `workers` as
+ it starts, so a run stopped at any point can still stop them. Returns why
+ the night stopped early if it did, and whether the rates had started."""
+ go = out / "rates.go"
+ rates = ",".join(f"{r:g}" for r in cfg.worker_rates())
+ durations = ",".join(f"{d:g}" for d in cfg.durations())
+ for i in range(1, cfg.workers + 1):
+ workers.append(
+ start(
+ cfg,
+ f"{base}w{i}",
+ i,
+ [
+ "--reuse",
+ f"{pop['base']}s{i}",
+ "--scenarios",
+ "idle",
+ "--duration",
+ f"{cfg.idle_seconds:g}",
+ "--rates",
+ rates,
+ "--rate-duration",
+ durations,
+ "--drain",
+ f"{cfg.drain_seconds:g}",
+ "--drain-full",
+ "--rates-after",
+ str(go),
+ ],
+ out,
+ ) # fmt: skip
+ )
+ time.sleep(STAGGER)
+ t = time.monotonic()
+ while True:
+ if bad := _failed(workers):
+ finish(workers)
+ return bad, False
+ if all(WAITING in w.text() for w in workers):
+ break
+ if time.monotonic() - t > 1200:
+ finish(workers)
+ return "the workers did not all connect within 20 minutes", False
+ time.sleep(5)
+ short = [
+ f"worker {w.index}: {c[0]}/{c[1]}"
+ for w in workers
+ if (c := w.connected()) and c[0] < CONNECTED_ENOUGH * c[1]
+ ]
+ if short:
+ finish(workers)
+ return "agents did not all connect (" + ", ".join(short)[:300] + ")", False
+ print(
+ f"all {len(workers)} workers connected in {time.monotonic() - t:.0f}s; rates start",
+ flush=True,
+ )
+ go.touch()
+ while any(w.proc.poll() is None for w in workers):
+ if any(BROKE in w.text() for w in workers):
+ # The step that broke is the night's answer; the other workers
+ # would only go on stepping with part of the load.
+ print("a worker broke a step; stopping the others", flush=True)
+ finish(workers)
+ break
+ if bad := _crashed(workers):
+ finish(workers)
+ return bad, True
+ if datetime.now(UTC) >= until:
+ finish(workers)
+ return f"stopped at the deadline, {until:%H:%M} UTC", True
+ time.sleep(5)
+ # Stopped or not, a worker that could not revoke its controllers exits 1.
+ return _crashed(workers), True
+
+
+def _failed(workers: list[Worker]) -> str | None:
+ """The workers that failed before their rates, and why, or None."""
+ bad = [f"worker {w.index}: {why}" for w in workers if (why := w.failed())]
+ return "; ".join(bad)[:400] if bad else None
+
+
+def _crashed(workers: list[Worker]) -> str | None:
+ """The workers that failed once their rates had started, and why, or None."""
+ bad = [f"worker {w.index}: {why}" for w in workers if (why := w.crashed())]
+ return "; ".join(bad)[:400] if bad else None
+
+
+# ── The three commands ────────────────────────────────────────────────────────
+
+
+def night_of(now: datetime) -> str:
+ """A night's date: the run after midnight belongs to that day."""
+ return now.strftime("%Y-%m-%d")
+
+
+def cmd_up(cfg: Config) -> int:
+ size_nodegroup(cfg, cfg.harness_nodegroup, 1)
+ size_nodegroup(cfg, cfg.core_nodegroup, cfg.night_nodes)
+ scale_core(cfg, cfg.night_pods)
+ t = time.monotonic()
+ s = wait_core(cfg, cfg.night_pods, 1500)
+ print(
+ f"switch-core: {s['ready']}/{cfg.night_pods} ready after {time.monotonic() - t:.0f}s, "
+ f"build {s['build']}, pods per node {sorted(s['by_node'].values())}",
+ flush=True,
+ )
+ return 0 if s["ready"] == cfg.night_pods else 1
+
+
+def day_sizes(cfg: Config) -> list[str]:
+ """switch-core first, so its pods leave the nodes before they go. Every
+ step is tried whatever the others do; returns the ones that failed."""
+ failed = []
+ for name, step in (
+ ("switch-core", lambda: scale_core(cfg, cfg.day_pods)),
+ (
+ "switch-core's nodes",
+ lambda: size_nodegroup(cfg, cfg.core_nodegroup, cfg.day_nodes),
+ ),
+ ("the harness's node", lambda: size_nodegroup(cfg, cfg.harness_nodegroup, 0)),
+ ):
+ try:
+ step()
+ except Exception as exc: # noqa: BLE001 - try the rest regardless
+ failed.append(f"{name}: {type(exc).__name__}: {exc}"[:200])
+ print(f"day sizes: {failed[-1]}", flush=True)
+ return failed
+
+
+class Interrupted(Exception):
+ """The pod was told to stop (SIGTERM): the night still reports."""
+
+
+class _Tee:
+ """Everything printed goes to the pod's log and to a file stored with
+ the night's records, so the run's own account outlives its pod."""
+
+ def __init__(self, stream, path: Path) -> None:
+ self.stream = stream
+ self.file = path.open("a")
+
+ def write(self, data: str) -> int:
+ self.stream.write(data)
+ self.file.write(data)
+ return len(data)
+
+ def flush(self) -> None:
+ self.stream.flush()
+ self.file.flush()
+
+
+@dataclass
+class Night:
+ """One night's run and what it found. `run` may stop anywhere; `result`
+ makes a report of whatever got done."""
+
+ cfg: Config
+ started: datetime
+ base: str
+ out: Path
+ # Failures make the night a ❌; notes (a pod restarting at the step that
+ # broke, say) are shown only.
+ failures: list[str] = field(default_factory=list)
+ notes: list[str] = field(default_factory=list)
+ workers: list[Worker] = field(default_factory=list)
+ pop: dict | None = None
+ seeded: bool = False
+ before: dict | None = None
+
+ def run(self) -> None:
+ cfg = self.cfg
+ if core_state(cfg)["replicas"] != cfg.night_pods:
+ # Started by hand without `up` first (this pod's own node is up,
+ # or it would not be running): size staging for the night here.
+ print("switch-core is not at its night size; scaling it up", flush=True)
+ size_nodegroup(cfg, cfg.core_nodegroup, cfg.night_nodes)
+ scale_core(cfg, cfg.night_pods)
+ s = self.before = wait_core(cfg, cfg.night_pods, 1500)
+ print(
+ f"{cfg.environment} runs {s['build']} on {s['ready']} pods, "
+ f"{sorted(s['by_node'].values())} per node",
+ flush=True,
+ )
+ if s["ready"] != cfg.night_pods:
+ self.failures.append(
+ f"{s['ready']} of {cfg.night_pods} switch-core pods ready: "
+ "not comparable with other nights"
+ )
+ elif s["updated"] != s["replicas"]:
+ self.notes.append(
+ f"⚠️ a newer switch-core build was waiting to roll out "
+ f"({s['updated']} of {s['replicas']} pods on it); the night ran "
+ f"{s['build']}"
+ )
+ most = -(-cfg.night_pods // max(1, cfg.night_nodes))
+ if max(s["by_node"].values(), default=0) > most:
+ self.failures.append(
+ f"switch-core's pods sit {sorted(s['by_node'].values())} per node, "
+ f"more than {most} on one: not comparable with other nights"
+ )
+ until = deadline_after(self.started, cfg.deadline, timedelta(hours=3))
+
+ self.pop = s3_get(cfg, "population.json")
+ if self.pop and (
+ self.pop.get("shape") != cfg.population_shape()
+ or not self.pop.get("complete", True)
+ ):
+ print(
+ "the kept agents are half made or of another shape; removing them",
+ flush=True,
+ )
+ remove_population(cfg, self.pop, self.out)
+ self.pop = None
+ if self.pop is None:
+ self.pop = seed_population(cfg, self.base, self.out)
+ self.seeded = True
+ restore_population(cfg, self.pop)
+ stopped, rates_started = run_workers(
+ cfg, self.pop, self.base, self.out, until, self.workers
+ )
+ if stopped and not rates_started and not self.seeded:
+ # Stopped before measuring anything: most likely the kept agents
+ # are gone (removed, or the database replaced). Make them again
+ # and try once more; a server that is down fails this one too.
+ print(
+ f"stopped before the rates ({stopped}); making the agents again",
+ flush=True,
+ )
+ self.notes.append(
+ f"the kept agents could not be used ({stopped}); made again"
+ )
+ remove_population(cfg, self.pop, self.out)
+ self.base += "b"
+ self.pop = seed_population(cfg, self.base, self.out)
+ self.seeded = True
+ restore_population(cfg, self.pop)
+ self.workers = []
+ stopped, rates_started = run_workers(
+ cfg, self.pop, self.base, self.out, until, self.workers
+ )
+ if stopped:
+ self.failures.append(f"stopped early: {stopped}")
+
+ def result(self) -> dict:
+ cfg = self.cfg
+ before = self.before or {}
+ after = None
+ try:
+ after = core_state(cfg)
+ except Exception as exc: # noqa: BLE001 - the night still reports
+ self.notes.append(
+ f"⚠️ could not read switch-core after the run: {exc}"[:200]
+ )
+ restarts = None
+ if after and before:
+ restarts = after["restarts"] - before["restarts"]
+ if restarts:
+ self.notes.append(
+ f"⚠️ switch-core restarted {restarts} time(s) during the run"
+ )
+ if after["pod_template_hash"] != before["pod_template_hash"]:
+ self.failures.append(
+ "switch-core's pods were replaced during the run (a deploy?)"
+ )
+ rows = combine(cfg, [p for w in self.workers if (p := _phases(w))])
+ ended = datetime.now(UTC)
+ result = {
+ "night": night_of(self.started),
+ "base": self.base,
+ "build": before.get("build", "unknown"),
+ "image": before.get("image"),
+ "pods": before.get("ready", 0),
+ "nodes": len(before.get("by_node") or {}),
+ "seeded": self.seeded,
+ "population": self.pop["base"] if self.pop else None,
+ "started": self.started.isoformat(timespec="seconds"),
+ "ended": ended.isoformat(timespec="seconds"),
+ "duration_s": round((ended - self.started).total_seconds()),
+ "restarts": restarts,
+ "rows": rows,
+ "failures": self.failures,
+ "notes": self.notes,
+ }
+ result["verdict"] = verdict(rows, cfg, self.failures)
+ return result
+
+
+def cmd_run(cfg: Config) -> int:
+ started = datetime.now(UTC)
+ base = "n" + started.strftime("%Y%m%dt%H%M")
+ out = Path(os.environ.get("LOADTEST_OUT", "/tmp/loadtest")) / base
+ out.mkdir(parents=True, exist_ok=True)
+ sys.stdout = _Tee(sys.stdout, out / "nightly.log") # type: ignore[assignment]
+
+ def on_term(signum, frame) -> None:
+ raise Interrupted("the pod was stopped (SIGTERM)")
+
+ signal.signal(signal.SIGTERM, on_term)
+ night = Night(cfg, started, base, out)
+ try:
+ night.run()
+ except Exception as exc: # noqa: BLE001 - whatever stopped it, report the night
+ night.failures.append(f"the run failed: {type(exc).__name__}: {exc}"[:400])
+ finish(night.workers)
+ result = night.result()
+ report(cfg, result, night.workers, out)
+ # A failure here fails the job too; `down` tries again and reports it.
+ left = day_sizes(cfg)
+ return 0 if result["verdict"]["ok"] and not left else 1
+
+
+def report(cfg: Config, result: dict, workers: list[Worker], out: Path) -> None:
+ """The night to S3, Datadog, the webhook and the Switch room; each is
+ tried whatever the others do."""
+ try:
+ comparison = compare(result, history(cfg, before=result["night"]))
+ except Exception as exc: # noqa: BLE001
+ comparison = [f"no comparison: {exc}"[:200]]
+ text = report_text(result, cfg, comparison)
+ print(text, flush=True)
+ for name, action in (
+ ("S3", lambda: _store(cfg, result, workers, out)),
+ ("Datadog", lambda: send_datadog(result, cfg, text)),
+ ("webhook", lambda: post_webhook(cfg.webhook_url, text)),
+ ("Switch", lambda: post_switch(cfg, text)),
+ ):
+ try:
+ action()
+ except Exception as exc: # noqa: BLE001 - the others still go
+ print(f"{name}: {type(exc).__name__}: {exc}"[:300], flush=True)
+
+
+def _phases(w: Worker) -> list[dict] | None:
+ """A worker's measured windows, once it has written its summary."""
+ p = RUNS / w.run_id / "summary.json"
+ if not p.exists():
+ return None
+ return json.loads(p.read_text()).get("phases") or None
+
+
+def _store(cfg: Config, result: dict, workers: list[Worker], out: Path) -> None:
+ """The night's result, each worker's own records and the logs, in S3.
+ Not the per-message file: it is large and the summaries hold what it says."""
+ s3_put(cfg, f"history/{result['night']}.json", result)
+ run = f"runs/{result['night']}/{result['base']}/"
+ s3_put(cfg, run + "result.json", result)
+ for w in workers:
+ for name in (
+ "summary.json",
+ "summary.md",
+ "phases.json",
+ "config.json",
+ "events.jsonl",
+ ):
+ f = RUNS / w.run_id / name
+ if f.exists():
+ s3_put(cfg, f"{run}{w.run_id}/{name}", f.read_bytes())
+ sys.stdout.flush()
+ for f in out.glob("*.log"):
+ s3_put(cfg, run + "logs/" + f.name, f.read_bytes())
+ print(f"stored in s3://{cfg.bucket}/{cfg.prefix}{run}", flush=True)
+
+
+def _keep_run_log(cfg: Config, night: str) -> str | None:
+ """The run pod's log, to S3: it lives on the harness's node, which the day
+ sizes remove. Where it went, or None when there was none to keep."""
+ try:
+ text = kubectl(
+ cfg, "logs", "-l", "app.kubernetes.io/component=loadtest-run",
+ "--tail=-1", "--prefix", "--max-log-requests=5", timeout=120,
+ ) # fmt: skip
+ except Exception as exc: # noqa: BLE001
+ print(f"no run pod log to keep: {exc}"[:300], flush=True)
+ return None
+ if not text.strip():
+ return None
+ key = f"runs/{night}/run-pod.log"
+ s3_put(cfg, key, text.encode())
+ return f"s3://{cfg.bucket}/{cfg.prefix}{key}"
+
+
+def cmd_down(cfg: Config) -> int:
+ night = night_of(datetime.now(UTC))
+ try:
+ found = s3_get(cfg, f"history/{night}.json")
+ except Exception as exc: # noqa: BLE001 - reported as a missing night
+ print(f"could not read tonight's result: {exc}", flush=True)
+ found = None
+ kept = _keep_run_log(cfg, night) if found is None else None
+ left = day_sizes(cfg)
+ if found is not None and not left:
+ return 0
+ lines = []
+ if found is None:
+ lines.append(
+ f"❌ Switch nightly load test on {cfg.environment}: no result for {night}; "
+ + (
+ f"the run pod's log is in {kept}"
+ if kept
+ else "the run pod left no log (it may never have started: is the harness's node up?)"
+ )
+ )
+ if left:
+ lines.append(
+ f"⚠️ {cfg.environment} may still be at its night size: putting it back "
+ "failed for " + "; ".join(left)
+ )
+ text = "\n".join(lines)
+ print(text, flush=True)
+ actions = [
+ lambda: post_webhook(cfg.webhook_url, text),
+ lambda: post_switch(cfg, text),
+ ]
+ # A night that left a result has sent Datadog its numbers already.
+ if found is None:
+ result = {"build": "unknown", "pods": 0, "rows": [], "verdict": {"ok": False}}
+ actions.append(lambda: send_datadog(result, cfg, text))
+ for action in actions:
+ try:
+ action()
+ except Exception as exc: # noqa: BLE001
+ print(f"{type(exc).__name__}: {exc}"[:300], flush=True)
+ return 1
+
+
+def main(argv: list[str] | None = None) -> int:
+ args = sys.argv[1:] if argv is None else argv
+ if len(args) != 1 or args[0] not in ("up", "run", "down"):
+ print(__doc__, file=sys.stderr)
+ return 2
+ cfg = Config.from_env()
+ return {"up": cmd_up, "run": cmd_run, "down": cmd_down}[args[0]](cfg)
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/loadtest/pod_sampler.py b/loadtest/pod_sampler.py
new file mode 100644
index 000000000..a2c42c464
--- /dev/null
+++ b/loadtest/pod_sampler.py
@@ -0,0 +1,103 @@
+# Runs inside a switch-core pod, fed to `python -` over `kubectl exec -i`, and
+# prints one JSON line per interval. It only reads: the container's own cgroup
+# files, and Postgres statistics views.
+#
+# python - pod [interval] this container's CPU (percent of one core, from
+# cpu.stat) and memory (memory.current), in MB.
+# python - db [interval] database-wide numbers, the same from any pod:
+# connections of the app's DB role by state (all
+# pods together), every connection to the database,
+# lock waits, and transactions per second. Connects
+# with the pod's own DB_* environment.
+#
+# asyncpg is for `db` mode; the switch-core image already has it.
+import asyncio
+import json
+import os
+import sys
+import time
+
+import asyncpg
+
+
+def _read(path):
+ try:
+ with open(path) as f:
+ return f.read()
+ except OSError:
+ return ""
+
+
+def _cpu_usec():
+ for line in _read("/sys/fs/cgroup/cpu.stat").splitlines():
+ if line.startswith("usage_usec"):
+ return int(line.split()[1])
+ return None
+
+
+def _mem_mb():
+ v = _read("/sys/fs/cgroup/memory.current").strip()
+ return round(int(v) / (1024 * 1024), 1) if v else None
+
+
+def pod(interval):
+ last = None
+ while True:
+ t, c = time.time(), _cpu_usec()
+ out = {"ts": t, "mem_mb": _mem_mb()}
+ if last and c is not None and last[1] is not None:
+ out["cpu_pct"] = round((c - last[1]) / ((t - last[0]) * 1e6) * 100, 1)
+ last = (t, c)
+ print(json.dumps(out), flush=True)
+ time.sleep(interval)
+
+
+async def db(interval):
+ conn = await asyncpg.connect(
+ host=os.environ["DB_HOST"],
+ port=int(os.environ.get("DB_PORT") or 5432),
+ user=os.environ["DB_USER"],
+ password=os.environ["DB_PASSWORD"],
+ database=os.environ["DB_NAME"],
+ ssl="prefer",
+ )
+ role, name = os.environ["DB_USER"], os.environ["DB_NAME"]
+ last = None
+ while True:
+ t = time.time()
+ rows = await conn.fetch(
+ "select usename = $1 as app, coalesce(state, '?') s, count(*) n, "
+ "count(*) filter (where wait_event_type = 'Lock') lw "
+ "from pg_stat_activity where datname = $2 and pid <> pg_backend_pid() "
+ "group by 1, 2",
+ role,
+ name,
+ )
+ x = await conn.fetchval(
+ "select xact_commit + xact_rollback from pg_stat_database where datname = $1",
+ name,
+ )
+ app = {r["s"]: r["n"] for r in rows if r["app"]}
+ out = {
+ "ts": t,
+ "app_active": app.get("active", 0),
+ "app_idle": app.get("idle", 0),
+ "app_idle_tx": app.get("idle in transaction", 0),
+ "app_total": sum(app.values()),
+ "db_total": sum(r["n"] for r in rows),
+ "lock_waits": sum(r["lw"] for r in rows),
+ }
+ if last:
+ out["xact_s"] = round((x - last[1]) / (t - last[0]), 1)
+ last = (t, x)
+ print(json.dumps(out), flush=True)
+ await asyncio.sleep(interval)
+
+
+if __name__ == "__main__":
+ mode = sys.argv[1] if len(sys.argv) > 1 else "pod"
+ every = float(sys.argv[2]) if len(sys.argv) > 2 else 1.0
+ if mode == "db":
+ asyncio.run(db(every))
+ else:
+ pod(every)
diff --git a/loadtest/report.py b/loadtest/report.py
new file mode 100644
index 000000000..e88bbde28
--- /dev/null
+++ b/loadtest/report.py
@@ -0,0 +1,205 @@
+"""Turning a run's raw records into numbers per phase, and into summary.md."""
+
+from __future__ import annotations
+
+import math
+import statistics
+from collections import defaultdict
+
+
+def pct(xs: list[float], p: float) -> float | None:
+ """The p-th quantile (nearest rank) in milliseconds, or None."""
+ if not xs:
+ return None
+ xs = sorted(xs)
+ # The ceil(p * n)-th value, rounded first: 0.07 * 100 is 7.000000000000001.
+ rank = math.ceil(round(p * len(xs), 9))
+ return round(xs[min(len(xs), max(rank, 1)) - 1] * 1000, 1)
+
+
+def _avg(xs: list[float]) -> float | None:
+ return round(statistics.mean(xs), 1) if xs else None
+
+
+def _max(xs: list[float]) -> float | None:
+ return round(max(xs), 1) if xs else None
+
+
+def phase_summary(
+ phase: dict, stats, samples: list[dict], agents_connected: list[int]
+) -> dict:
+ """The numbers for one measured window (`phase` has its name, scenario,
+ length in seconds and the target agent count)."""
+ name, secs = phase["name"], phase["seconds"]
+ msgs = stats.by_phase.get(name, [])
+ ok = [m for m in msgs if m.status == 200]
+ lat = [x for m in ok for x in m.latencies]
+ send = [m.acked - m.sent for m in ok if m.acked is not None]
+ expected = sum(m.expected for m in ok)
+ got = sum(min(len(m.latencies), m.expected) for m in ok)
+ ev = stats.events_by_phase.get(name, [])
+ count = defaultdict(int)
+ for e in ev:
+ count[e["kind"]] += 1
+
+ # Tagged with the phase as each line arrives, so pod clocks do not matter.
+ in_window = [s for s in samples if s.get("phase") == name]
+ pods: dict[str, dict] = {}
+ for s in in_window:
+ if s["kind"] != "pod":
+ continue
+ p = pods.setdefault(s["pod"], {"cpu": [], "mem": []})
+ if s.get("cpu_pct") is not None:
+ p["cpu"].append(s["cpu_pct"])
+ if s.get("mem_mb") is not None:
+ p["mem"].append(s["mem_mb"])
+ db = [s for s in in_window if s["kind"] == "db"]
+ xact = [s["xact_s"] for s in db if s.get("xact_s") is not None]
+
+ errors = (
+ stats.send_errors.get(name, 0)
+ + stats.read_errors.get(name, 0)
+ + count["refused"]
+ + count["drop"]
+ + count["controller_drop"]
+ + count["connect_failed"]
+ + count["evicted"]
+ )
+ return {
+ "phase": name + (" (stopped early)" if phase.get("stopped") else ""),
+ "scenario": phase["scenario"],
+ "agents_target": phase["agents"],
+ "agents_connected_min": min(agents_connected) if agents_connected else None,
+ "agents_connected_end": agents_connected[-1] if agents_connected else None,
+ "seconds": round(secs, 1),
+ "schedule_fingerprint": phase.get("fingerprint"),
+ "messages_planned": phase.get("planned", 0),
+ "messages_sent": len(ok),
+ "messages_sent_per_s": round(len(ok) / secs, 2) if secs else None,
+ "send_errors": stats.send_errors.get(name, 0),
+ "send_ms_p50": pct(send, 0.5),
+ "send_ms_p99": pct(send, 0.99),
+ "deliveries_expected": expected,
+ "deliveries_received": got,
+ "deliveries_missing": expected - got,
+ "deliveries_per_s": round(got / secs, 2) if secs else None,
+ "delivery_ms_p50": pct(lat, 0.5),
+ "delivery_ms_p99": pct(lat, 0.99),
+ "delivery_ms_max": pct(lat, 1.0),
+ "context_reads": len(stats.reads.get(name, [])),
+ "context_read_errors": stats.read_errors.get(name, 0),
+ "context_read_ms_p99": pct(stats.reads.get(name, []), 0.99),
+ "drops": count["drop"],
+ "refusals": count["refused"],
+ "connect_failures": count["connect_failed"],
+ "evictions": count["evicted"],
+ "gaps": count["gap"],
+ # Agents that gave up for good (`--retry runtime`, refused as taken
+ # over), and controller sockets that dropped or were refused.
+ "stood_down": count["stood_down"],
+ "controller_drops": count["controller_drop"],
+ # Agents a dropped controller socket took offline with it.
+ "detached": count["detached"],
+ "controller_connect_failures": count["controller_connect_failed"],
+ "errors": errors,
+ "harness_lag_ms_max": round(stats.lag_max.get(name, 0.0) * 1000, 1),
+ "pods": {
+ pod: {
+ "cpu_pct_avg": _avg(v["cpu"]),
+ "cpu_pct_max": _max(v["cpu"]),
+ "mem_mb_max": _max(v["mem"]),
+ }
+ for pod, v in sorted(pods.items())
+ },
+ "db_app_conns_max": max((s["app_total"] for s in db), default=None),
+ "db_app_active_max": max((s["app_active"] for s in db), default=None),
+ "db_app_idle_tx_max": max((s["app_idle_tx"] for s in db), default=None),
+ "db_conns_total_max": max((s["db_total"] for s in db), default=None),
+ "db_lock_waits_max": max((s["lock_waits"] for s in db), default=None),
+ "db_xact_per_s_avg": _avg(xact),
+ "db_xact_per_s_p50": round(statistics.median(xact), 1) if xact else None,
+ "db_xact_per_s_max": _max(xact),
+ "rds": phase.get("rds"),
+ }
+
+
+def _fmt(v) -> str:
+ return "-" if v is None else str(v)
+
+
+def render(meta: dict, phases: list[dict], teardown: dict | None) -> str:
+ lines = [
+ f"# Load run {meta['run_id']}",
+ "",
+ f"- Target: namespace `{meta['namespace']}` "
+ f"(context `{meta['kube_context'] or 'in-cluster'}`), "
+ f"API {meta['api_url']}, via {meta['via']}",
+ f"- switch-core pods seen: {', '.join(meta['pods']) or 'none'}",
+ f"- Seed {meta['seed']}, membership {meta['membership_fingerprint']}, "
+ f"{meta['rooms']} rooms of {meta['room_size']}",
+ f"- Started {meta['started']}, ended {meta.get('ended', '-')}",
+ "",
+ "Delivery latency is from the moment the sender's POST went out to the moment the "
+ "receiving agent read the frame, both on the harness clock. `errors` counts failed "
+ "sends, failed context reads, refused or failed connects, evictions and dropped sockets.",
+ "",
+ "## Traffic",
+ "",
+ "| phase | agents (target / min connected) | msgs/s sent | deliveries/s | delivered / expected "
+ "| delivery p50 / p99 ms | send p99 ms | errors | harness lag max ms |",
+ "|---|---|---|---|---|---|---|---|---|",
+ ]
+ for p in phases:
+ lines.append(
+ f"| {p['phase']} | {p['agents_target']} / {_fmt(p['agents_connected_min'])} "
+ f"| {_fmt(p['messages_sent_per_s'])} | {_fmt(p['deliveries_per_s'])} "
+ f"| {p['deliveries_received']} / {p['deliveries_expected']} "
+ f"| {_fmt(p['delivery_ms_p50'])} / {_fmt(p['delivery_ms_p99'])} "
+ f"| {_fmt(p['send_ms_p99'])} | {p['errors']} | {p['harness_lag_ms_max']} |"
+ )
+ lines += [
+ "",
+ "## Server",
+ "",
+ "| phase | pod | CPU avg / max % | memory max MB |",
+ "|---|---|---|---|",
+ ]
+ for p in phases:
+ for pod, v in p["pods"].items():
+ lines.append(
+ f"| {p['phase']} | {pod} | {_fmt(v['cpu_pct_avg'])} / {_fmt(v['cpu_pct_max'])} "
+ f"| {_fmt(v['mem_mb_max'])} |"
+ )
+ lines += [
+ "",
+ "| phase | DB conns app max (active / idle in tx) | DB conns total max | transactions/s avg / p50 / max "
+ "| lock waits max | RDS CPU avg / max % |",
+ "|---|---|---|---|---|---|",
+ ]
+ for p in phases:
+ rds = p.get("rds") or {}
+ lines.append(
+ f"| {p['phase']} | {_fmt(p['db_app_conns_max'])} ({_fmt(p['db_app_active_max'])} / "
+ f"{_fmt(p['db_app_idle_tx_max'])}) | {_fmt(p['db_conns_total_max'])} "
+ f"| {_fmt(p['db_xact_per_s_avg'])} / {_fmt(p['db_xact_per_s_p50'])} / {_fmt(p['db_xact_per_s_max'])} "
+ f"| {_fmt(p['db_lock_waits_max'])} | {_fmt(rds.get('cpu_avg'))} / {_fmt(rds.get('cpu_max'))} |"
+ )
+ if teardown is not None:
+ lines += [
+ "",
+ "## Teardown",
+ "",
+ ]
+ if meta.get("kept"):
+ lines.append(
+ f"Kept the agents and rooms for --reuse {meta['run_id']}; revoked "
+ f"{teardown.get('controllers_revoked')} controllers, errors {teardown.get('errors')}."
+ )
+ return "\n".join(lines) + "\n"
+ lines += [
+ f"Agents deleted {teardown.get('agents_deleted')}, rooms archived {teardown.get('rooms_archived')}, "
+ f"keys deleted {teardown.get('keys_deleted')}, errors {teardown.get('errors')}. "
+ f"Left behind: {teardown.get('left_agents')} agents, {teardown.get('left_rooms')} active rooms, "
+ f"{teardown.get('left_keys')} keys.",
+ ]
+ return "\n".join(lines) + "\n"
diff --git a/loadtest/run.py b/loadtest/run.py
new file mode 100644
index 000000000..690086f20
--- /dev/null
+++ b/loadtest/run.py
@@ -0,0 +1,940 @@
+"""Load test switch-core in a non-pilot environment, and publish the numbers.
+
+ uv run --project core python loadtest/run.py --namespace <namespace> \\
+ --scenarios idle,steady,burst --steps 50,100,200 --rate 5 --duration 120
+
+ # a message-rate sweep: one idle window, then 90 s at each rate until one breaks
+ uv run --project core python loadtest/run.py --namespace <namespace> \\
+ --scenarios idle --steps 300 --rates 10,15,20,25
+
+ # the same agents behind 25 fake controllers, as most real agents connect
+ uv run --project core python loadtest/run.py --namespace <namespace> \\
+ --controllers 25 --scenarios idle,steady --steps 500,1000 --rate 2
+
+Seeds its own agents and rooms, runs the scenarios at each step, samples
+every switch-core pod and the database while it does, then removes what it
+made, on Ctrl-C and on failure too. `--cleanup <run_id>` removes leftovers.
+It never restarts, scales or edits anything in Kubernetes: `kubectl get` and
+`kubectl exec` (and, with `--via port-forward`, `kubectl port-forward`) only.
+See loadtest/README.md.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import os
+import re
+import signal
+import subprocess
+import sys
+import time
+from dataclasses import asdict
+from datetime import UTC, datetime
+from pathlib import Path
+from urllib.parse import urlparse
+
+import httpx
+
+sys.path.insert(0, str(Path(__file__).parent))
+
+from agents import Agent, Stats, play # noqa: E402
+from controllers import FakeController, ManagedAgent # noqa: E402
+from kube import Kube, PortForward, Refused, Sampler, guard_target # noqa: E402
+from report import phase_summary, render # noqa: E402
+from seed import ( # noqa: E402
+ Admin,
+ NoWorkspace,
+ RequestIds,
+ SeededAgent,
+ create_controllers,
+ revoke_controllers,
+ seed,
+ seed_managed,
+ teardown,
+)
+from workload import ( # noqa: E402
+ SCENARIOS,
+ build_membership,
+ build_schedule,
+ fingerprint,
+)
+
+RUNS = Path(__file__).parent / "runs"
+# A run id names a directory under RUNS and prefixes what the run makes.
+RUN_ID = re.compile(r"[A-Za-z0-9][A-Za-z0-9_-]{0,63}")
+
+
+def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
+ ap = argparse.ArgumentParser(
+ description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
+ )
+ t = ap.add_argument_group("target")
+ t.add_argument(
+ "--namespace",
+ required=True,
+ help="the release's namespace; the pilot's and prod's are refused",
+ )
+ t.add_argument(
+ "--kube-context",
+ default="",
+ help="default: kubectl's current context, or from a pod inside the "
+ "cluster the pod's service account",
+ )
+ t.add_argument(
+ "--api-url", help="agent API URL; default: the release's GATEWAY_PUBLIC_URL"
+ )
+ t.add_argument(
+ "--gateway-url", help="gateway URL; default: the release's FRONTEND_BASE_URL"
+ )
+ t.add_argument(
+ "--via",
+ choices=["urls", "port-forward"],
+ default="urls",
+ help="port-forward: reach each core pod through kubectl, for a Mac off the tailnet (smoke runs only)",
+ )
+ t.add_argument(
+ "--workspace",
+ help="run in the admin's workspace of this name, made as an organization "
+ "of its own if missing, so the agents stay off the bridges of the "
+ "admin's own workspace",
+ )
+ w = ap.add_argument_group("workload")
+ w.add_argument(
+ "--scenarios",
+ default="idle,steady,burst",
+ help=f"comma list of {','.join(SCENARIOS)}",
+ )
+ w.add_argument(
+ "--agents", type=int, default=50, help="agents, when --steps is not given"
+ )
+ w.add_argument("--steps", help="agent counts to ramp through, e.g. 50,100,200")
+ w.add_argument("--rooms", type=int, help="rooms; default half the largest step")
+ w.add_argument(
+ "--controllers",
+ type=int,
+ default=0,
+ help="run the agents behind this many fake agent controllers, spread "
+ "evenly, instead of each on a socket of its own",
+ )
+ w.add_argument("--room-size", type=int, default=4)
+ w.add_argument(
+ "--rate",
+ type=float,
+ default=5.0,
+ help="messages per second (steady, and burst's background)",
+ )
+ w.add_argument(
+ "--duration",
+ type=float,
+ default=120.0,
+ help="seconds measured per scenario per step",
+ )
+ w.add_argument(
+ "--settle", type=float, default=10.0, help="quiet seconds after each scenario"
+ )
+ w.add_argument(
+ "--rates",
+ help="messages per second to step through after the scenarios at each step, "
+ "e.g. 10,15,20: one steady window per rate with the same agents connected, "
+ "stopping after the first rate that breaks",
+ )
+ w.add_argument(
+ "--rate-duration",
+ default="90",
+ help="seconds measured per rate: one number for every rate, or one per "
+ "rate in --rates, e.g. 600,600,60",
+ )
+ w.add_argument(
+ "--rates-after",
+ metavar="FILE",
+ help="before the first rate, wait with the agents connected until FILE "
+ "exists, so workers started one after another step their rates together",
+ )
+ w.add_argument(
+ "--drain",
+ type=float,
+ default=60.0,
+ help="most seconds to wait after each rate for its deliveries to arrive",
+ )
+ w.add_argument(
+ "--drain-full",
+ action="store_true",
+ help="always wait the whole --drain after each rate, so workers that "
+ "started their rates together start every rate together",
+ )
+ w.add_argument("--burst-every", type=float, default=30.0)
+ w.add_argument("--burst-size", type=int, default=100, help="messages per spike")
+ w.add_argument(
+ "--burst-window",
+ type=float,
+ default=2.0,
+ help="seconds each spike is spread over",
+ )
+ w.add_argument("--hot-room", action="store_true", help="spikes all go to one room")
+ w.add_argument("--payload-min", type=int, default=40)
+ w.add_argument("--payload-max", type=int, default=2000)
+ w.add_argument(
+ "--read-fraction",
+ type=float,
+ default=0.1,
+ help="share of deliveries the receiver follows with read_context",
+ )
+ w.add_argument(
+ "--connect-rate", type=float, default=20.0, help="agents opened per second"
+ )
+ w.add_argument(
+ "--retry",
+ choices=["harness", "runtime"],
+ default="harness",
+ help="how a dropped agent socket reopens: the harness's jittered backoff, "
+ "or as the real agent runtime does (fast retries for a minute after a "
+ "restart, unjittered doubling otherwise, standing down when refused as "
+ "taken over)",
+ )
+ w.add_argument(
+ "--honor-busy",
+ action="store_true",
+ help="wait out the retry_after_s of an open refused as busy, as clients "
+ "with the open gate's change do",
+ )
+ w.add_argument(
+ "--calls",
+ choices=["message", "ops", "socket"],
+ default="message",
+ help="how a controller's agents post and read: on the message route, as "
+ "operations over HTTP the way the real controller relays them, or as "
+ "operations on the controller's socket",
+ )
+ w.add_argument("--seed", type=int, default=1)
+ o = ap.add_argument_group("run")
+ o.add_argument("--run-id", help="default: UTC time, e.g. 20261026t153000")
+ o.add_argument("--sample-interval", type=float, default=1.0)
+ o.add_argument(
+ "--rds-instance",
+ help="RDS instance to read CPU from in CloudWatch",
+ )
+ o.add_argument(
+ "--aws-profile",
+ default=os.environ.get("AWS_PROFILE", ""),
+ help="default: the credentials already in the environment, such as a "
+ "pod's service account role",
+ )
+ o.add_argument(
+ "--aws-region",
+ default=os.environ.get("AWS_REGION")
+ or os.environ.get("AWS_DEFAULT_REGION", ""),
+ help="the RDS instance's region; default: AWS_REGION, else the AWS CLI's own",
+ )
+ o.add_argument(
+ "--plan", action="store_true", help="print the workload and exit; no requests"
+ )
+ o.add_argument(
+ "--cleanup",
+ metavar="RUN_ID",
+ help="remove a run's leftovers ('*' for every load run) and exit",
+ )
+ o.add_argument(
+ "--keep",
+ action="store_true",
+ help="leave the agents and rooms for a later run's --reuse; only the "
+ "controllers are revoked",
+ )
+ o.add_argument(
+ "--seed-only",
+ action="store_true",
+ help="with --keep: make the agents and rooms and keep them, connecting "
+ "nothing, for later runs' --reuse",
+ )
+ o.add_argument(
+ "--concurrency",
+ type=int,
+ default=8,
+ help="requests in flight while making and removing agents, rooms and "
+ "controllers; lower it on an environment with few pods",
+ )
+ o.add_argument(
+ "--reuse",
+ metavar="RUN_ID",
+ help="use the agents and rooms a --keep run left instead of seeding; "
+ "same --seed, --steps, --rooms and --room-size as that run",
+ )
+ a = ap.parse_args(argv)
+ a.scenarios = [s.strip() for s in a.scenarios.split(",") if s.strip()]
+ for s in a.scenarios:
+ if s not in SCENARIOS:
+ ap.error(f"unknown scenario {s}")
+ a.steps = [int(x) for x in a.steps.split(",")] if a.steps else [a.agents]
+ a.rates = [float(x) for x in a.rates.split(",")] if a.rates else []
+ if not _increasing(a.rates) or any(r <= 0 for r in a.rates):
+ ap.error("--rates must be increasing and above 0")
+ durations = [float(x) for x in str(a.rate_duration).split(",")]
+ if len(durations) == 1:
+ durations *= max(1, len(a.rates))
+ if len(durations) != max(1, len(a.rates)) or any(d <= 0 for d in durations):
+ ap.error("--rate-duration takes one number, or one per rate in --rates")
+ a.rate_durations = durations
+ if not _increasing(a.steps) or min(a.steps) < 2:
+ ap.error("--steps must be increasing and at least 2")
+ a.rooms = a.rooms or max(1, max(a.steps) // 2)
+ if a.controllers < 0 or a.controllers > max(a.steps):
+ ap.error("--controllers must be between 0 and the largest step")
+ if (a.keep or a.reuse) and not a.controllers:
+ # Agents of their own carry an API key each, which is not kept.
+ ap.error("--keep and --reuse need agents behind --controllers")
+ if a.seed_only and (not a.keep or a.reuse):
+ ap.error("--seed-only needs --keep, and makes agents rather than reusing them")
+ if a.concurrency < 1:
+ ap.error("--concurrency must be at least 1")
+ for flag, value in (("--run-id", a.run_id), ("--reuse", a.reuse)):
+ if value is not None and not RUN_ID.fullmatch(value):
+ ap.error(f"{flag} takes letters, digits, - and _ only")
+ if a.cleanup not in (None, "*") and not RUN_ID.fullmatch(a.cleanup):
+ ap.error("--cleanup takes a run id or '*'")
+ return a
+
+
+def _increasing(xs: list) -> bool:
+ """Each one above the last: a repeated rate or step would share a window."""
+ return all(x < y for x, y in zip(xs, xs[1:]))
+
+
+def kept_record(
+ run_id: str,
+ names_from: str,
+ seeded: list[SeededAgent],
+ room_ids: list[str],
+ membership: list[list[int]],
+ a: argparse.Namespace,
+) -> dict:
+ """What a later run needs to take over the agents and rooms: ids only.
+ No credential: the agents have none of their own, and the controllers,
+ which do, are revoked."""
+ return {
+ "kept_by": run_id,
+ "names_from": names_from,
+ "seed": a.seed,
+ "agents": [{"index": s.index, "id": s.id, "name": s.name} for s in seeded],
+ "rooms": room_ids,
+ "membership_fingerprint": fingerprint(membership),
+ "workspace": a.workspace,
+ }
+
+
+def take_over(
+ kept: dict, membership: list[list[int]], n_agents: int
+) -> list[SeededAgent]:
+ """The kept agents, once the plan is shown to put them in the same rooms."""
+ if kept["membership_fingerprint"] != fingerprint(membership):
+ raise RuntimeError(
+ f"the plan's rooms differ from {kept['kept_by']}'s: use its --seed, "
+ "--steps, --rooms and --room-size"
+ )
+ if len(kept["agents"]) < n_agents:
+ raise RuntimeError(f"{kept['kept_by']} kept {len(kept['agents'])} agents")
+ return [SeededAgent(x["index"], x["id"], x["name"], "") for x in kept["agents"]]
+
+
+def plan(a: argparse.Namespace) -> tuple[list[list[int]], dict[str, list]]:
+ membership = build_membership(a.seed, max(a.steps), a.rooms, a.room_size)
+ schedules = {}
+ for step in a.steps:
+ for sc in a.scenarios:
+ schedules[f"{sc}@{step}"] = build_schedule(
+ seed=a.seed,
+ scenario=sc,
+ step=step,
+ duration=a.duration,
+ rooms=membership,
+ active=step,
+ rate=a.rate,
+ burst_every=a.burst_every,
+ burst_size=a.burst_size,
+ burst_window=a.burst_window,
+ hot_room=a.hot_room,
+ payload_min=a.payload_min,
+ payload_max=a.payload_max,
+ )
+ for r, rate_duration in zip(a.rates, a.rate_durations, strict=False):
+ schedules[rate_phase(r, step)] = build_schedule(
+ seed=a.seed,
+ scenario="steady",
+ step=step,
+ duration=rate_duration,
+ rooms=membership,
+ active=step,
+ rate=r,
+ payload_min=a.payload_min,
+ payload_max=a.payload_max,
+ )
+ return membership, schedules
+
+
+def rate_phase(rate: float, step: int) -> str:
+ """The phase name of one rate in --rates: `steady15@300` is 15 messages a
+ second with 300 agents connected."""
+ return f"steady{rate:g}@{step}"
+
+
+def broken(summary: dict) -> str | None:
+ """Why a measured window counts as broken, or None: a delivery lost, p99
+ over 20 s, or more than 20 errors. A sweep does not step past it."""
+ if summary["deliveries_missing"]:
+ return f"{summary['deliveries_missing']} deliveries missing"
+ if (summary["delivery_ms_p99"] or 0) > 20000:
+ return f"p99 {summary['delivery_ms_p99'] / 1000:.1f} s"
+ if summary["errors"] > 20:
+ return f"{summary['errors']} errors"
+ return None
+
+
+def admin_login(kube: Kube) -> tuple[str, str]:
+ """The admin's email and password: LOADTEST_ADMIN_EMAIL and
+ LOADTEST_ADMIN_PASSWORD when both are set, for an environment whose admin
+ has changed the password it was deployed with, otherwise the release's
+ Secret. Kept in memory only."""
+ email = os.environ.get("LOADTEST_ADMIN_EMAIL")
+ password = os.environ.get("LOADTEST_ADMIN_PASSWORD")
+ if email and password:
+ return email, password
+ return kube.admin_login()
+
+
+def resolve_urls(a: argparse.Namespace, kube: Kube) -> None:
+ """Default the URLs to the release's own, and refuse URLs that belong to
+ another environment: the pods sampled must be the pods loaded. A URL on
+ one of the namespace's own Ingresses (its tailnet one, say) or Services
+ (from inside the cluster) is its too."""
+ gw, api = kube.public_urls()
+ hosts: set[str] | None = None
+ for flag, given, deployed in (
+ ("--gateway-url", a.gateway_url, gw),
+ ("--api-url", a.api_url, api),
+ ):
+ if not given or (deployed and given.rstrip("/") == deployed.rstrip("/")):
+ continue
+ if hosts is None:
+ hosts = kube.ingress_hosts() | kube.service_hosts()
+ if urlparse(given).hostname not in hosts:
+ raise Refused(
+ f"{flag} {given} is neither this namespace's "
+ f"({deployed or 'the release names none'}) nor on one of its "
+ "Ingresses or Services; refusing to load one env and sample another"
+ )
+ a.gateway_url = (a.gateway_url or gw or "").rstrip("/")
+ a.api_url = (a.api_url or api or "").rstrip("/")
+ if not a.gateway_url or not a.api_url:
+ raise Refused(
+ "no --gateway-url/--api-url given and the release does not name them"
+ )
+
+
+async def wait(stop: asyncio.Event, seconds: float) -> bool:
+ """Sleep, but wake on Ctrl-C. True when stopped."""
+ try:
+ await asyncio.wait_for(stop.wait(), max(0.0, seconds))
+ return True
+ except TimeoutError:
+ return False
+
+
+def rds_cpu(a: argparse.Namespace, start: float, end: float) -> dict | None:
+ """RDS CPU and connections over a window, from CloudWatch (one-minute
+ resolution, so short windows read coarsely; data lands a minute or two late)."""
+ if not a.rds_instance:
+ return None
+ env = dict(os.environ)
+ if a.aws_profile:
+ env["AWS_PROFILE"] = a.aws_profile
+ out: dict = {}
+ for metric, key in (("CPUUtilization", "cpu"), ("DatabaseConnections", "conns")):
+ r = subprocess.run(
+ [
+ "aws", "cloudwatch", "get-metric-statistics",
+ "--namespace", "AWS/RDS", "--metric-name", metric,
+ "--dimensions", f"Name=DBInstanceIdentifier,Value={a.rds_instance}",
+ "--start-time", datetime.fromtimestamp(start - 60, UTC).isoformat(),
+ "--end-time", datetime.fromtimestamp(end + 60, UTC).isoformat(),
+ "--period", "60", "--statistics", "Average", "Maximum",
+ *(["--region", a.aws_region] if a.aws_region else []),
+ "--output", "json",
+ ],
+ capture_output=True, text=True, timeout=60, env=env,
+ ) # fmt: skip
+ if r.returncode != 0:
+ out["error"] = r.stderr.strip()[:200]
+ continue
+ pts = json.loads(r.stdout).get("Datapoints", [])
+ if pts:
+ out[f"{key}_avg"] = round(sum(p["Average"] for p in pts) / len(pts), 1)
+ out[f"{key}_max"] = round(max(p["Maximum"] for p in pts), 1)
+ return out
+
+
+async def cleanup(a: argparse.Namespace, kube: Kube) -> int:
+ pf = None
+ if a.via == "port-forward":
+ pf = PortForward(kube)
+ a.gateway_url = pf.open()[0]
+ admin = Admin(a.gateway_url, RequestIds(f"cleanup-{a.cleanup}"))
+ try:
+ await admin.login(*admin_login(kube))
+ if a.workspace:
+ try:
+ await admin.use_workspace(a.workspace, create=False)
+ except NoWorkspace as exc:
+ print(f"{exc}: nothing to clean", flush=True)
+ return 0
+ done = await teardown(admin, a.cleanup, concurrency=a.concurrency)
+ finally:
+ await admin.close()
+ if pf:
+ pf.close()
+ print(json.dumps(done))
+ left = ("left_agents", "left_rooms", "left_keys", "left_controllers")
+ return 1 if any(done.get(k) for k in left) else 0
+
+
+async def run(a: argparse.Namespace, kube: Kube) -> int:
+ run_id = a.run_id or datetime.now(UTC).strftime("%Y%m%dt%H%M%S")
+ out = RUNS / run_id
+ out.mkdir(parents=True, exist_ok=False)
+ membership, schedules = plan(a)
+ ids = RequestIds(run_id)
+ stats = Stats()
+ stop = asyncio.Event()
+ loop = asyncio.get_running_loop()
+
+ def on_signal() -> None:
+ if stop.is_set():
+ print(
+ f"\nsecond interrupt: exiting without teardown; run --cleanup {run_id}",
+ flush=True,
+ )
+ os._exit(130)
+ print(
+ "\ninterrupted: stopping traffic and tearing down (Ctrl-C again to abandon)",
+ flush=True,
+ )
+ stop.set()
+
+ loop.add_signal_handler(signal.SIGINT, on_signal)
+ loop.add_signal_handler(signal.SIGTERM, on_signal)
+
+ pf = None
+ bases = [a.api_url]
+ if a.via == "port-forward":
+ pf = PortForward(kube)
+ bases = pf.open()
+ a.api_url = a.gateway_url = bases[0]
+
+ meta = {
+ "run_id": run_id,
+ "namespace": a.namespace,
+ "kube_context": a.kube_context,
+ "api_url": a.api_url if a.via == "urls" else "port-forward",
+ "gateway_url": a.gateway_url if a.via == "urls" else "port-forward",
+ "via": a.via,
+ "seed": a.seed,
+ "rooms": a.rooms,
+ "room_size": a.room_size,
+ "membership_fingerprint": fingerprint(membership),
+ "started": datetime.now(UTC).isoformat(timespec="seconds"),
+ "args": {k: v for k, v in vars(a).items() if k not in ("cleanup",)},
+ "reused": a.reuse,
+ # Set at the end: a run asked to keep that never finished seeding
+ # tears everything down instead.
+ "kept": False,
+ "pods": [],
+ }
+ (out / "config.json").write_text(json.dumps(meta, indent=1, default=str))
+ print(f"run {run_id} -> {out}", flush=True)
+
+ sampler = Sampler(kube, interval=a.sample_interval)
+ sampler_task = asyncio.create_task(sampler.run())
+ admin = Admin(a.gateway_url, ids)
+ state: dict = {"api_key_id": None, "agents": [], "rooms": []}
+ clients = {
+ b: httpx.AsyncClient(
+ base_url=b,
+ timeout=60,
+ limits=httpx.Limits(max_connections=512, max_keepalive_connections=256),
+ )
+ for b in bases
+ }
+ agents: list[Agent] = []
+ controllers: list[FakeController] = []
+ phases: list[dict] = []
+ connected_log: dict[str, list[int]] = {}
+ td: dict | None = None
+
+ async def count_connected() -> None:
+ while True:
+ connected_log.setdefault(stats.phase, []).append(
+ sum(x.connected.is_set() for x in agents)
+ )
+ await asyncio.sleep(1)
+
+ counter = asyncio.create_task(count_connected())
+
+ async def drained(name: str) -> bool:
+ """Wait, up to --drain seconds, until every message sent in `name` has
+ reached all its receivers, so a backlog does not run into the next
+ window. Errors in the meantime still count for `name`."""
+ t = time.monotonic()
+ while time.monotonic() - t < a.drain:
+ sent = stats.by_phase.get(name, [])
+ if all(m.status != 200 or len(m.latencies) >= m.expected for m in sent):
+ return True
+ if await wait(stop, 1.0):
+ break
+ return False
+
+ async def measure(
+ name: str, scenario: str, step: int, duration: float, drain: bool = False
+ ) -> None:
+ schedule = schedules[name]
+ stats.phase = sampler.phase = name
+ start, t0 = time.time(), time.monotonic()
+ print(f" {name}: {len(schedule)} messages over {duration:.0f}s", flush=True)
+ if schedule:
+ await play(
+ schedule, agents[:step], room_ids, membership, stats, run_id, name, stop
+ )
+ await wait(stop, duration - (time.monotonic() - t0))
+ end, secs = time.time(), time.monotonic() - t0
+ stopped = stop.is_set()
+ if drain:
+ sampler.phase = "drain"
+ t1 = time.monotonic()
+ done = await drained(name)
+ print(
+ f" {name}: deliveries {'drained' if done else 'still owed'} after {time.monotonic() - t1:.0f}s",
+ flush=True,
+ )
+ if a.drain_full:
+ await wait(stop, a.drain - (time.monotonic() - t1))
+ stats.phase = sampler.phase = "settle"
+ await wait(stop, a.settle)
+ phases.append(
+ {
+ "name": name,
+ "scenario": scenario,
+ "agents": step,
+ "start": start,
+ "end": end,
+ "seconds": secs,
+ "planned": len(schedule),
+ "fingerprint": fingerprint(schedule),
+ "stopped": stopped,
+ }
+ )
+
+ code = 0
+ # Whose name prefix the agents and rooms carry: this run's, or that of the
+ # run that seeded them, when they are reused.
+ names_from = run_id
+ seeded = []
+ room_ids = []
+ taken_over = False
+ try:
+ await admin.login(*admin_login(kube))
+ if a.workspace:
+ # Kept agents live in a workspace that exists: a reusing run never
+ # makes one, so several started at once cannot make one each.
+ meta["workspace"] = await admin.use_workspace(
+ a.workspace, create=not a.reuse
+ )
+ print(f"workspace: {json.dumps(meta['workspace'])}", flush=True)
+ await admin.refuse_bridges()
+ t = time.monotonic()
+ if a.reuse:
+ kept = json.loads((RUNS / a.reuse / "keep.json").read_text())
+ if kept.get("workspace") != a.workspace:
+ raise RuntimeError(
+ f"{a.reuse} kept its agents in workspace {kept.get('workspace')}: "
+ "reuse them with the same --workspace"
+ )
+ seeded = take_over(kept, membership, max(a.steps))
+ room_ids = kept["rooms"]
+ names_from = kept["names_from"]
+ meta["names_from"] = names_from
+ # From here on a run that fails still keeps them, under --keep.
+ taken_over = True
+ made = await create_controllers(
+ admin, run_id, a.controllers, state, asyncio.Semaphore(a.concurrency)
+ )
+ print(
+ f"reusing {len(seeded)} agents and {len(room_ids)} rooms kept by "
+ f"{a.reuse}; {len(made)} new controllers in "
+ f"{time.monotonic() - t:.1f}s",
+ flush=True,
+ )
+ elif a.controllers:
+ seeded, room_ids, made = await seed_managed(
+ admin,
+ run_id,
+ max(a.steps),
+ a.controllers,
+ membership,
+ state,
+ concurrency=a.concurrency,
+ )
+ taken_over = True
+ else:
+ seeded, room_ids = await seed(
+ admin,
+ a.api_url,
+ run_id,
+ max(a.steps),
+ membership,
+ state,
+ concurrency=a.concurrency,
+ )
+ made = []
+ for c in made:
+ base = bases[c.index % len(bases)]
+ controllers.append(
+ FakeController(
+ c.index,
+ c.id,
+ c.name,
+ c.credential,
+ base,
+ clients[base],
+ ids,
+ stats,
+ honor_busy=a.honor_busy,
+ )
+ )
+ if not a.reuse:
+ print(
+ f"seeded {len(seeded)} agents, {len(room_ids)} rooms"
+ + (f", {len(controllers)} controllers" if controllers else "")
+ + f" in {time.monotonic() - t:.1f}s",
+ flush=True,
+ )
+ if a.seed_only:
+ print("seed only: nothing connects; agents and rooms are kept", flush=True)
+ rooms_of: dict[int, list[str]] = {}
+ for j, members in enumerate(membership):
+ for m in members:
+ rooms_of.setdefault(m, []).append(room_ids[j])
+ for s in seeded:
+ base = bases[s.index % len(bases)]
+ if controllers:
+ owner = controllers[s.index % len(controllers)]
+ agents.append(
+ ManagedAgent(
+ s,
+ rooms_of.get(s.index, []),
+ ids,
+ stats,
+ owner.http,
+ a.read_fraction,
+ a.seed,
+ owner,
+ admin,
+ calls=a.calls,
+ )
+ )
+ continue
+ agents.append(
+ Agent(
+ s,
+ rooms_of.get(s.index, []),
+ base,
+ ids,
+ stats,
+ clients[base],
+ a.read_fraction,
+ a.seed,
+ retry=a.retry,
+ honor_busy=a.honor_busy,
+ )
+ )
+ if controllers and not a.seed_only:
+ # Placing an agent needs its controller online with a fresh status
+ # report, so every controller connects and reports first.
+ stats.phase = sampler.phase = "controllers"
+ t = time.monotonic()
+ for c in controllers:
+ c.start()
+ await asyncio.sleep(1 / a.connect_rate)
+ while (
+ sum(c.reported.is_set() for c in controllers) < len(controllers)
+ and time.monotonic() - t < 120
+ ):
+ if await wait(stop, 0.5):
+ break
+ up = sum(c.reported.is_set() for c in controllers)
+ print(
+ f"controllers: {up}/{len(controllers)} connected and reporting "
+ f"in {time.monotonic() - t:.1f}s",
+ flush=True,
+ )
+ if up < len(controllers):
+ raise RuntimeError("not every controller came online")
+
+ opened = 0
+ for step in [] if a.seed_only else a.steps:
+ if stop.is_set():
+ break
+ stats.phase = sampler.phase = f"connect@{step}"
+ t = time.monotonic()
+ while opened < step:
+ agents[opened].start()
+ opened += 1
+ await asyncio.sleep(1 / a.connect_rate)
+ while (
+ sum(x.connected.is_set() for x in agents[:step]) < step
+ and time.monotonic() - t < 180
+ ):
+ if await wait(stop, 0.5):
+ break
+ up = sum(x.connected.is_set() for x in agents[:step])
+ print(
+ f"step {step}: {up}/{step} agents connected in {time.monotonic() - t:.1f}s",
+ flush=True,
+ )
+
+ for sc in a.scenarios:
+ if stop.is_set():
+ break
+ await measure(f"{sc}@{step}", sc, step, a.duration)
+ if a.rates and a.rates_after and not stop.is_set():
+ stats.phase = sampler.phase = f"wait@{step}"
+ print(f" waiting for {a.rates_after} before the rates", flush=True)
+ while not Path(a.rates_after).exists():
+ if await wait(stop, 0.5):
+ break
+ now = datetime.now(UTC).isoformat(timespec="seconds")
+ print(f" rates start {now}", flush=True)
+ for r, rate_duration in zip(a.rates, a.rate_durations, strict=False):
+ if stop.is_set():
+ break
+ name = rate_phase(r, step)
+ await measure(name, "steady", step, rate_duration, drain=True)
+ phases[-1]["rate"] = r
+ why = broken(
+ phase_summary(
+ phases[-1],
+ stats,
+ sampler.by_phase.get(name, []),
+ connected_log.get(name, []),
+ )
+ )
+ if why:
+ print(f" {name} broke ({why}); not stepping higher", flush=True)
+ break
+ except Exception as exc: # noqa: BLE001 - reported, then torn down
+ print(f"run failed: {type(exc).__name__}: {exc}", flush=True)
+ code = 1
+ finally:
+ stats.phase = sampler.phase = "teardown"
+ counter.cancel()
+ await asyncio.gather(*(x.stop() for x in agents), return_exceptions=True)
+ await asyncio.gather(*(c.stop() for c in controllers), return_exceptions=True)
+ try:
+ if a.keep and taken_over:
+ meta["kept"] = True
+ td = await revoke_controllers(admin, run_id, concurrency=a.concurrency)
+ record = kept_record(
+ run_id, names_from, seeded, room_ids, membership, a
+ )
+ (out / "keep.json").write_text(json.dumps(record, indent=1))
+ print(
+ f"kept {len(seeded)} agents and {len(room_ids)} rooms for "
+ f"--reuse {run_id}; teardown: {json.dumps(td)}",
+ flush=True,
+ )
+ if td["left_controllers"] or td["errors"]:
+ code = code or 1
+ else:
+ td = await teardown(admin, run_id, concurrency=a.concurrency)
+ if names_from != run_id:
+ seeded_td = await teardown(
+ admin, names_from, concurrency=a.concurrency
+ )
+ td = {k: td.get(k, 0) + seeded_td.get(k, 0) for k in td | seeded_td}
+ print(f"teardown: {json.dumps(td)}", flush=True)
+ if (
+ td["left_agents"]
+ or td["left_rooms"]
+ or td["left_keys"]
+ or td.get("left_controllers")
+ or td["errors"]
+ ):
+ code = code or 1
+ except Exception as exc: # noqa: BLE001
+ also = f" and --cleanup {names_from}" if names_from != run_id else ""
+ print(f"teardown failed ({exc}); run: --cleanup {run_id}{also}", flush=True)
+ code = 1
+ await admin.close()
+ await asyncio.gather(
+ *(c.aclose() for c in clients.values()), return_exceptions=True
+ )
+ sampler_task.cancel()
+ await sampler.close()
+ if pf:
+ pf.close()
+
+ meta["pods"] = sorted(sampler.pods_seen)
+ meta["pod_info"] = sampler.pods_seen
+ meta["ended"] = datetime.now(UTC).isoformat(timespec="seconds")
+ for p in phases:
+ p["rds"] = await asyncio.to_thread(rds_cpu, a, p["start"], p["end"])
+ summaries = [
+ phase_summary(
+ p,
+ stats,
+ sampler.by_phase.get(p["name"], []),
+ connected_log.get(p["name"], []),
+ )
+ for p in phases
+ ]
+ _write_jsonl(out / "samples.jsonl", sampler.samples)
+ _write_jsonl(out / "events.jsonl", stats.events)
+ _write_jsonl(out / "messages.jsonl", (asdict(m) for m in stats.messages.values()))
+ (out / "phases.json").write_text(json.dumps(phases, indent=1))
+ (out / "summary.json").write_text(
+ json.dumps(
+ {"meta": meta, "phases": summaries, "teardown": td}, indent=1, default=str
+ )
+ )
+ (out / "summary.md").write_text(render(meta, summaries, td))
+ print((out / "summary.md").read_text())
+ return code
+
+
+def _write_jsonl(path: Path, rows) -> None:
+ with path.open("w") as f:
+ for r in rows:
+ f.write(json.dumps(r, default=str) + "\n")
+
+
+def main(argv: list[str] | None = None) -> int:
+ a = parse_args(argv)
+ if a.plan:
+ membership, schedules = plan(a)
+ print(f"membership {fingerprint(membership)}: {a.rooms} rooms of {a.room_size}")
+ for name, sch in schedules.items():
+ print(f"{name}: {len(sch)} messages, schedule {fingerprint(sch)}")
+ return 0
+ try:
+ guard_target(a.namespace, a.api_url or "", a.gateway_url or "")
+ kube = Kube(a.kube_context, a.namespace)
+ if a.via == "urls":
+ resolve_urls(a, kube)
+ guard_target(a.namespace, a.api_url, a.gateway_url)
+ except Refused as exc:
+ print(f"refused: {exc}", file=sys.stderr)
+ return 2
+ if a.cleanup:
+ return asyncio.run(cleanup(a, kube))
+ return asyncio.run(run(a, kube))
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/loadtest/seed.py b/loadtest/seed.py
new file mode 100644
index 000000000..d37d4af45
--- /dev/null
+++ b/loadtest/seed.py
@@ -0,0 +1,485 @@
+"""Creating the run's agents, rooms and key, and removing them again.
+
+Everything a run creates is named after it: the API key is labelled
+`load-<run_id>`, agents are `load-<run_id>-aNNNN`, rooms `load-<run_id>-rNNN`,
+and with `--controllers` the controllers are `load-<run_id>-cNNN`.
+Teardown and `--cleanup` find what to remove by those names, through the
+gateway as the environment's admin, so they work even when the process that
+created them died without saying what it made.
+
+No secret leaves this process: the admin password, the session cookie, the
+registration key, the agents' keys and the controllers' credentials are held
+in memory and never printed or written to the run folder.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import itertools
+from dataclasses import dataclass
+
+import httpx
+
+# What a load agent says it can do: always connected, exchanges messages,
+# nothing else. The same profile the throwaway agents of
+# runbooks/scripts/dev-load-agents.py register with.
+PROFILE = {
+ "connection_model": "always_on",
+ "message_exchange": True,
+ "pre_invocation_mediation": [],
+ "post_invocation_mediation": [],
+ "event_reporting": [],
+ # Retired on main, which drops unknown keys, but still required by builds
+ # from before that change, so a run against an older environment works.
+ "task_protocol": {"can_delegate": False, "can_accept": False},
+}
+
+
+class RequestIds:
+ """`load-<run_id>-<n>`, one per harness request, so a run can be picked
+ out of the server's logs and traces."""
+
+ def __init__(self, run_id: str) -> None:
+ self.run_id = run_id
+ self._n = itertools.count(1)
+
+ def next(self) -> str:
+ return f"load-{self.run_id}-{next(self._n)}"
+
+
+def prefix(run_id: str) -> str:
+ return f"load-{run_id}-"
+
+
+@dataclass
+class SeededAgent:
+ index: int
+ id: str
+ name: str
+ key: str # the agent's own API key; memory only
+
+
+class NoWorkspace(RuntimeError):
+ """The workspace a run was told to use is not there, and it may not make it."""
+
+
+class Admin:
+ """The gateway, signed in as the environment's admin."""
+
+ def __init__(self, gateway_url: str, ids: RequestIds) -> None:
+ self.ids = ids
+ self._cookie: str | None = None
+ self.http = httpx.AsyncClient(base_url=gateway_url.rstrip("/"), timeout=60)
+
+ async def call(self, method: str, path: str, **kw) -> httpx.Response:
+ headers = {"X-Request-Id": self.ids.next()}
+ if self._cookie:
+ # Carried by hand: the session cookie is Secure, and httpx will not
+ # send a Secure cookie over the plain http of a port-forward.
+ headers["Cookie"] = f"switch_auth={self._cookie}"
+ r = await self.http.request(method, f"/gateway{path}", headers=headers, **kw)
+ self.http.cookies.clear()
+ return r
+
+ async def login(self, email: str, password: str) -> None:
+ r = await self.call(
+ "POST", "/auth/login", json={"email": email, "password": password}
+ )
+ if r.status_code != 200:
+ raise RuntimeError(f"admin sign-in refused: HTTP {r.status_code}")
+ self._cookie = r.cookies.get("switch_auth")
+ if not self._cookie:
+ raise RuntimeError("admin sign-in returned no switch_auth cookie")
+
+ async def use_workspace(self, name: str, create: bool = True) -> dict:
+ """Bind this session to the admin's workspace called `name`, made as
+ an organization of its own when there is none (unless `create` is
+ false: then NoWorkspace), then put the admin's own selection back.
+
+ Signing in selects the workspace the admin last switched to, so
+ without that, whoever signs in as the admin while a run is going, a
+ person or a scheduled job, would land in the load test's workspace.
+ This session keeps the cookie bound to `name`; the one the second
+ switch mints is dropped."""
+ r = await self.call("GET", "/tenants/current")
+ before = r.json()["id"] if r.status_code == 200 else None
+ r = await self.call("GET", "/tenants")
+ if r.status_code != 200:
+ raise _fail("listing the admin's workspaces", r)
+ named = [t for t in r.json() if t.get("name") == name]
+ if len(named) > 1:
+ raise RuntimeError(f"the admin has {len(named)} workspaces named {name}")
+ if named:
+ workspace = named[0]
+ r = await self.call("POST", f"/tenants/{workspace['id']}/switch")
+ if r.status_code != 200:
+ raise _fail(f"switching to the workspace {name}", r)
+ elif not create:
+ raise NoWorkspace(f"the admin has no workspace named {name}")
+ else:
+ r = await self.call("POST", "/organizations", json={"name": name})
+ if r.status_code != 201:
+ raise _fail(f"creating the workspace {name}", r)
+ workspace = r.json()["workspace"]
+ cookie = r.cookies.get("switch_auth")
+ if not cookie:
+ raise RuntimeError(f"switching to {name} returned no switch_auth cookie")
+ self._cookie = cookie
+ if before is not None and before != workspace["id"]:
+ r = await self.call("POST", f"/tenants/{before}/switch")
+ if r.status_code != 200:
+ raise _fail("putting the admin's own workspace back", r)
+ return {"id": workspace["id"], "name": name, "made": not named}
+
+ async def refuse_bridges(self) -> None:
+ """Refuse a workspace with an active bridge: each bridge gives every
+ agent of its workspace an identity on its platform (a role on
+ Discord, a user group on Slack, a bot on Mattermost), so a run there
+ would make thousands of them."""
+ r = await self.call("GET", "/collaborations")
+ if r.status_code != 200:
+ raise _fail("listing the workspace's bridges", r)
+ active = [b for b in r.json() if b.get("status") == "active"]
+ if active:
+ names = ", ".join(
+ f"{b.get('bridge_type')} {b.get('display_name')!r}" for b in active
+ )
+ raise RuntimeError(
+ f"the run's workspace has active bridges ({names}); "
+ "run it in a workspace of its own with --workspace"
+ )
+
+ async def close(self) -> None:
+ await self.http.aclose()
+
+
+def _fail(what: str, r: httpx.Response) -> RuntimeError:
+ # Only for error responses, which carry a status and a reason, never a key.
+ return RuntimeError(f"{what}: HTTP {r.status_code} {r.text[:200]}")
+
+
+async def seed(
+ admin: Admin,
+ api_url: str,
+ run_id: str,
+ n_agents: int,
+ membership: list[list[int]],
+ state: dict,
+ *,
+ concurrency: int,
+) -> tuple[list[SeededAgent], list[str]]:
+ """Create the key, the agents and the rooms. Records each id in `state`
+ as it is made, so a failure part way still tears down what exists."""
+ r = await admin.call("POST", "/api-keys", json={"label": f"load-{run_id}"})
+ if r.status_code != 200:
+ raise _fail("creating the API key", r)
+ body = r.json()
+ state["api_key_id"], reg_key = body["id"], body["key"]
+
+ sem = asyncio.Semaphore(concurrency)
+ agents: list[SeededAgent | None] = [None] * n_agents
+ async with httpx.AsyncClient(base_url=api_url.rstrip("/"), timeout=60) as api:
+
+ async def register(i: int) -> None:
+ name = f"{prefix(run_id)}a{i:04d}"
+ async with sem:
+ r = await api.post(
+ "/agents",
+ headers={
+ "Authorization": f"Bearer {reg_key}",
+ "X-Request-Id": admin.ids.next(),
+ },
+ json={
+ "name": name,
+ "description": f"load test {run_id}, removed when the run ends",
+ "connector_type": "e2e",
+ "integration_profile": PROFILE,
+ },
+ )
+ if r.status_code != 200:
+ raise _fail(f"registering {name}", r)
+ b = r.json()
+ agents[i] = SeededAgent(i, b["id"], name, b["api_key"])
+ state["agents"].append(b["id"])
+
+ await asyncio.gather(*(register(i) for i in range(n_agents)))
+
+ seeded = [a for a in agents if a is not None]
+ return seeded, await create_rooms(admin, run_id, seeded, membership, state, sem)
+
+
+@dataclass
+class SeededController:
+ index: int
+ id: str
+ name: str
+ credential: str # memory only
+
+
+async def seed_managed(
+ admin: Admin,
+ run_id: str,
+ n_agents: int,
+ n_controllers: int,
+ membership: list[list[int]],
+ state: dict,
+ *,
+ concurrency: int,
+) -> tuple[list[SeededAgent], list[str], list[SeededController]]:
+ """Create the controllers, the agents they will run and the rooms, through
+ the gateway's management routes, as Switch Console does. The agents are
+ made unplaced: each is placed on its controller when its step connects it
+ (`controllers.ManagedAgent`), since placing needs the controller online.
+ Records each id in `state` as it is made."""
+ sem = asyncio.Semaphore(concurrency)
+ made = await create_controllers(admin, run_id, n_controllers, state, sem)
+ agents: list[SeededAgent | None] = [None] * n_agents
+
+ async def agent(i: int) -> None:
+ name = f"{prefix(run_id)}a{i:04d}"
+ async with sem:
+ r = await admin.call(
+ "POST",
+ "/management/agents",
+ json={
+ "name": name,
+ "description": f"load test {run_id}, removed when the run ends",
+ "controller_id": None,
+ "desired_state": "running",
+ "definition": {"provider": "claude"},
+ },
+ )
+ if r.status_code != 201:
+ raise _fail(f"creating managed agent {name}", r)
+ agents[i] = SeededAgent(i, r.json()["agent_id"], name, "")
+ state["agents"].append(agents[i].id)
+
+ await asyncio.gather(*(agent(i) for i in range(n_agents)))
+ seeded = [a for a in agents if a is not None]
+ rooms = await create_rooms(admin, run_id, seeded, membership, state, sem)
+ return seeded, rooms, made
+
+
+async def create_controllers(
+ admin: Admin,
+ run_id: str,
+ n_controllers: int,
+ state: dict,
+ sem: asyncio.Semaphore,
+) -> list[SeededController]:
+ """Create the run's controllers, named after it. A run that reuses another
+ run's agents makes its own, since a controller's credential is never kept."""
+ controllers: list[SeededController | None] = [None] * n_controllers
+
+ async def controller(k: int) -> None:
+ name = f"{prefix(run_id)}c{k:03d}"
+ async with sem:
+ r = await admin.call(
+ "POST",
+ "/management/controllers",
+ json={
+ "name": name,
+ "description": f"load test {run_id}, removed when the run ends",
+ "kind": "console",
+ "platform": {
+ "os": "linux",
+ "arch": "x64",
+ "os_version": "loadtest",
+ },
+ "version": "loadtest",
+ },
+ )
+ if r.status_code != 201:
+ raise _fail(f"creating controller {name}", r)
+ b = r.json()
+ controllers[k] = SeededController(k, b["controller_id"], name, b["credential"])
+ state.setdefault("controllers", []).append(b["controller_id"])
+
+ await asyncio.gather(*(controller(k) for k in range(n_controllers)))
+ return [c for c in controllers if c is not None]
+
+
+async def create_rooms(
+ admin: Admin,
+ run_id: str,
+ seeded: list[SeededAgent],
+ membership: list[list[int]],
+ state: dict,
+ sem: asyncio.Semaphore,
+) -> list[str]:
+ room_ids: list[str | None] = [None] * len(membership)
+
+ async def create_room(j: int, members: list[int]) -> None:
+ name = f"{prefix(run_id)}r{j:03d}"
+ async with sem:
+ r = await admin.call(
+ "POST",
+ "/rooms",
+ json={
+ "name": name,
+ "description": f"load test {run_id}",
+ "agent_ids": [seeded[m].id for m in members],
+ "internal_only": True,
+ },
+ )
+ if r.status_code not in (200, 201):
+ raise _fail(f"creating room {name}", r)
+ room_ids[j] = r.json()["id"]
+ state["rooms"].append(room_ids[j])
+
+ await asyncio.gather(*(create_room(j, m) for j, m in enumerate(membership)))
+ return [rid for rid in room_ids if rid]
+
+
+async def teardown(
+ admin: Admin, run_id: str, log=print, concurrency: int = 8
+) -> dict[str, int]:
+ """Remove everything named after the run: agents deleted, rooms archived,
+ the key deleted. `run_id` "*" removes every load run's leftovers.
+ Returns what it removed, and what is still there after (should be 0)."""
+ pre = "load-" if run_id == "*" else prefix(run_id)
+ label = None if run_id == "*" else f"load-{run_id}"
+ done = {
+ "agents_deleted": 0,
+ "rooms_archived": 0,
+ "keys_deleted": 0,
+ "controllers_revoked": 0,
+ "errors": 0,
+ }
+ sem = asyncio.Semaphore(concurrency)
+
+ async def one(method: str, path: str, counter: str) -> None:
+ async with sem:
+ try:
+ r = await admin.call(method, path)
+ except httpx.HTTPError as exc:
+ log(f" {method} {path}: {type(exc).__name__}")
+ done["errors"] += 1
+ return
+ if r.status_code in (200, 204, 404):
+ done[counter] += r.status_code != 404
+ else:
+ log(f" {method} {path}: HTTP {r.status_code} {r.text[:120]}")
+ done["errors"] += 1
+
+ # Controller-backed agents go through the management routes, which also
+ # drop their definitions; then their controllers are revoked.
+ managed = await _managed_agents(admin, pre)
+ await asyncio.gather(
+ *(
+ one("DELETE", f"/management/agents/{a['agent_id']}", "agents_deleted")
+ for a in managed
+ )
+ )
+ controllers = await _controllers(admin, pre)
+ await asyncio.gather(
+ *(
+ one("DELETE", f"/management/controllers/{c['id']}", "controllers_revoked")
+ for c in controllers
+ )
+ )
+ rooms = await _rooms(admin, pre)
+ await asyncio.gather(
+ *(one("POST", f"/rooms/{r['id']}/archive", "rooms_archived") for r in rooms)
+ )
+ agents = await _agents(admin, pre)
+ await asyncio.gather(
+ *(one("DELETE", f"/agents/{a['id']}", "agents_deleted") for a in agents)
+ )
+ keys = await _keys(admin, pre, label)
+ await asyncio.gather(
+ *(one("DELETE", f"/api-keys/{k['id']}", "keys_deleted") for k in keys)
+ )
+
+ done["left_agents"] = len(await _agents(admin, pre))
+ done["left_rooms"] = len(await _rooms(admin, pre))
+ done["left_keys"] = len(await _keys(admin, pre, label))
+ done["left_controllers"] = len(await _controllers(admin, pre))
+ return done
+
+
+async def revoke_controllers(
+ admin: Admin, run_id: str, log=print, concurrency: int = 8
+) -> dict[str, int]:
+ """Revoke the run's controllers and nothing else: what a kept run removes,
+ so the agents and rooms it leaves carry no credential with them."""
+ pre = prefix(run_id)
+ done = {"controllers_revoked": 0, "errors": 0}
+ sem = asyncio.Semaphore(concurrency)
+
+ async def one(controller_id: str) -> None:
+ path = f"/management/controllers/{controller_id}"
+ async with sem:
+ try:
+ r = await admin.call("DELETE", path)
+ except httpx.HTTPError as exc:
+ log(f" DELETE {path}: {type(exc).__name__}")
+ done["errors"] += 1
+ return
+ if r.status_code in (200, 204, 404):
+ done["controllers_revoked"] += r.status_code != 404
+ else:
+ log(f" DELETE {path}: HTTP {r.status_code} {r.text[:120]}")
+ done["errors"] += 1
+
+ await asyncio.gather(*(one(c["id"]) for c in await _controllers(admin, pre)))
+ done["left_controllers"] = len(await _controllers(admin, pre))
+ return done
+
+
+async def _managed_agents(admin: Admin, pre: str) -> list[dict]:
+ r = await admin.call("GET", "/management/agents")
+ if r.status_code == 404:
+ return [] # agent management is off here, so there are none
+ if r.status_code != 200:
+ raise _fail("listing managed agents", r)
+ return [a for a in r.json() if str(a.get("name", "")).startswith(pre)]
+
+
+async def _controllers(admin: Admin, pre: str) -> list[dict]:
+ # Unrevoked only: revoked is the end state, so it is not a leftover.
+ r = await admin.call("GET", "/management/controllers")
+ if r.status_code == 404:
+ return []
+ if r.status_code != 200:
+ raise _fail("listing controllers", r)
+ return [
+ c
+ for c in r.json()
+ if str(c.get("name", "")).startswith(pre) and not c.get("revoked_at")
+ ]
+
+
+async def _agents(admin: Admin, pre: str) -> list[dict]:
+ r = await admin.call("GET", "/agents")
+ if r.status_code != 200:
+ raise _fail("listing agents", r)
+ return [a for a in r.json() if str(a.get("name", "")).startswith(pre)]
+
+
+async def _rooms(admin: Admin, pre: str) -> list[dict]:
+ # Active rooms only: archived is the end state, so it is not a leftover.
+ r = await admin.call("GET", "/rooms", params={"search": pre})
+ if r.status_code != 200:
+ raise _fail("listing rooms", r)
+ return [
+ x
+ for x in r.json()
+ if str(x.get("name", "")).startswith(pre) and not x.get("archived")
+ ]
+
+
+async def _keys(admin: Admin, pre: str, label: str | None) -> list[dict]:
+ r = await admin.call("GET", "/api-keys")
+ if r.status_code != 200:
+ raise _fail("listing API keys", r)
+ # The registration key is labelled `load-<run_id>`. Each agent's own key is
+ # labelled with the agent's name, and deleting the agent does not delete
+ # that key's row (agents point at their key, not the other way round), so
+ # those are matched by the run's name prefix and removed too.
+ found = []
+ for k in r.json():
+ lab = str(k.get("label") or "")
+ if lab.startswith(pre) or (label is not None and lab == label):
+ found.append(k)
+ return found
diff --git a/loadtest/test_controllers.py b/loadtest/test_controllers.py
new file mode 100644
index 000000000..bea799676
--- /dev/null
+++ b/loadtest/test_controllers.py
@@ -0,0 +1,41 @@
+"""The fake controller's socket calls. No server."""
+
+from __future__ import annotations
+
+import asyncio
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).parent))
+
+import controllers # noqa: E402
+from agents import Stats # noqa: E402
+from controllers import FakeController # noqa: E402
+from seed import RequestIds # noqa: E402
+
+
+class _Socket:
+ """Takes every frame and never answers one."""
+
+ def __init__(self) -> None:
+ self.sent: list[str] = []
+
+ async def send(self, frame: str) -> None:
+ self.sent.append(frame)
+
+
+def test_a_call_the_server_never_answers_fails_after_the_timeout(monkeypatch):
+ monkeypatch.setattr(controllers, "CALL_TIMEOUT_S", 0.05)
+ c = FakeController(
+ 0, "c1", "load-c1", "cred", "http://core", None, RequestIds("t"), Stats()
+ )
+ c._ws, c.takes_calls = _Socket(), True
+
+ async def call():
+ return await c.call("a1", "send_message", {"content": "hi"}, "r1")
+
+ with pytest.raises(ConnectionError, match="no answer in 0.05s"):
+ asyncio.run(call())
+ assert len(c._ws.sent) == 1 and c._answers == {}
diff --git a/loadtest/test_nightly.py b/loadtest/test_nightly.py
new file mode 100644
index 000000000..f4718d5f5
--- /dev/null
+++ b/loadtest/test_nightly.py
@@ -0,0 +1,597 @@
+"""The nightly run's arithmetic and report. No requests, no cluster."""
+
+from __future__ import annotations
+
+import json
+import signal
+import sys
+from datetime import UTC, datetime, timedelta
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).parent))
+
+import nightly # noqa: E402
+from nightly import ( # noqa: E402
+ Config,
+ Night,
+ build_of,
+ combine,
+ compare,
+ datadog_series,
+ deadline_after,
+ history,
+ report_text,
+ s3_delete,
+ s3_get,
+ s3_put,
+ size_nodegroup,
+ verdict,
+)
+from run import parse_args # noqa: E402
+
+ENV = {
+ "LOADTEST_NAMESPACE": "switch-test",
+ "LOADTEST_DEPLOYMENT": "switch-test-switch-core",
+ "LOADTEST_API_URL": "http://switch-test-switch-core.switch-test.svc:8000",
+ "LOADTEST_CLUSTER": "test-cluster",
+ "LOADTEST_CORE_NODEGROUP": "core-",
+ "LOADTEST_HARNESS_NODEGROUP": "harness-",
+ "AWS_REGION": "us-east-1",
+ "LOADTEST_BUCKET": "bucket",
+}
+
+
+def _cfg(**env: str) -> Config:
+ return Config.from_env({**ENV, **{f"LOADTEST_{k}": v for k, v in env.items()}})
+
+
+def _phase(
+ name: str, sent: float, p99: float | None, errors=0, missing=0, cpu=40.0
+) -> dict:
+ return {
+ "phase": name,
+ "agents_target": 1000,
+ "agents_connected_min": 1000,
+ "messages_sent_per_s": sent,
+ "delivery_ms_p50": 90.0,
+ "delivery_ms_p99": p99,
+ "errors": errors,
+ "deliveries_missing": missing,
+ "harness_lag_ms_max": 2.0,
+ "pods": {"p1": {"cpu_pct_avg": cpu}, "p2": {"cpu_pct_avg": cpu + 10}},
+ "rds": {"cpu_avg": 20.0, "cpu_max": 30.0},
+ "db_xact_per_s_avg": 500.0,
+ }
+
+
+def _worker(cfg: Config, broke_at: int | None = None) -> list[dict]:
+ """One worker's windows: idle, every rate, stopping after `broke_at`."""
+ rates = cfg.worker_rates()
+ phases = [_phase("idle@1000", 0.0, None)]
+ for k, r in enumerate(rates):
+ bad = broke_at is not None and k == broke_at
+ phases.append(_phase(f"steady{r:g}@1000", r, 25000.0 if bad else 800.0))
+ if bad:
+ break
+ return phases
+
+
+def test_the_default_night_is_the_dev_abel_test_after_realistic_steps():
+ cfg = _cfg()
+ assert (cfg.per_agents, cfg.per_controllers, cfg.per_rooms) == (1000, 50, 500)
+ # One message per agent every 30, 10, 5 and 3 minutes, then the ramp.
+ assert [round(r, 1) for r in cfg.total_rates()] == [
+ 5.6, 16.7, 33.3, 55.6, 150, 250, 300, 350, 400, 450, 500,
+ ] # fmt: skip
+ assert cfg.worker_rates()[4:] == [15, 25, 30, 35, 40, 45, 50]
+ assert cfg.durations() == [600.0] * 4 + [60.0] * 7
+ assert (
+ cfg.step_labels()[0] == "every 30 min" and cfg.step_labels()[4] == "150 msg/s"
+ )
+
+
+def test_the_harness_takes_the_nightly_rates_and_durations():
+ cfg = _cfg()
+ a = parse_args(
+ [
+ "--namespace", "switch-test", "--steps", "1000", "--controllers", "50",
+ "--rates", ",".join(f"{r:g}" for r in cfg.worker_rates()),
+ "--rate-duration", ",".join(f"{d:g}" for d in cfg.durations()),
+ ]
+ ) # fmt: skip
+ assert a.rate_durations == cfg.durations()
+
+
+ ("env", "why"),
+ [
+ ({"AGENTS": "10001"}, "split evenly"),
+ ({"RAMP": "50,100"}, "less than the first ramp step"),
+ ({"DAY_PODS": "13"}, "day sizes"),
+ ({"NAMESPACE": "switch"}, "the pilot"),
+ ({"API_URL": "http://switch-switch-core.switch.svc:8000"}, "the pilot"),
+ ],
+)
+def test_settings_that_cannot_run_are_refused(env, why):
+ with pytest.raises(SystemExit, match=why):
+ _cfg(**env)
+
+
+def test_the_region_comes_from_the_environment_and_is_needed():
+ assert _cfg().region == "us-east-1"
+ env = {k: v for k, v in ENV.items() if k != "AWS_REGION"}
+ assert (
+ Config.from_env({**env, "AWS_DEFAULT_REGION": "eu-north-1"}).region
+ == "eu-north-1"
+ )
+ with pytest.raises(SystemExit, match="AWS_REGION is not set"):
+ Config.from_env(env)
+
+
+def test_workers_add_up_by_position_and_the_first_broken_step_is_named():
+ cfg = _cfg(WORKERS="2", AGENTS="2000", CONTROLLERS="100")
+ # The second worker breaks at the third ramp step (300 msg/s), the first
+ # one step later: rows stop where the earliest worker stopped.
+ rows = combine(cfg, [_worker(cfg, broke_at=7), _worker(cfg, broke_at=6)])
+ assert [r["label"] for r in rows][:2] == ["idle", "every 30 min"]
+ assert rows[-1]["label"] == "300 msg/s"
+ assert rows[-1]["broken"] == "p99 25.0 s (worker 2)"
+ assert rows[1]["sent_per_s"] == pytest.approx(2 * cfg.worker_rates()[0], abs=0.01)
+ assert rows[1]["pod_cpu_avg_pct"] == 45.0 and rows[1]["pod_cpu_busiest_pct"] == 50.0
+ v = verdict(rows, cfg, [])
+ assert v["ok"] and v["held_per_s"] == 250 and v["broke_at_per_s"] == 300
+
+
+def test_lost_deliveries_at_realistic_load_fail_the_night():
+ cfg = _cfg(WORKERS="1", AGENTS="1000", CONTROLLERS="50")
+ phases = _worker(cfg)
+ phases[2]["deliveries_missing"] = 3
+ v = verdict(combine(cfg, [phases]), cfg, [])
+ assert not v["ok"] and not v["realistic_ok"]
+
+
+def test_a_night_that_stopped_early_is_not_complete():
+ cfg = _cfg(WORKERS="1", AGENTS="1000", CONTROLLERS="50")
+ rows = combine(cfg, [_worker(cfg)[:3]])
+ v = verdict(rows, cfg, ["stopped early: deadline"])
+ assert not v["ok"] and not v["complete"]
+ result = {"build": "b", "pods": 12, "nodes": 4, "rows": rows, "verdict": v}
+ result["failures"] = ["stopped early: deadline"]
+ assert "staging, 1,000 agents: stopped early: deadline" in report_text(
+ result, cfg, []
+ )
+
+
+def test_the_report_fits_discord_and_has_no_em_dash():
+ cfg = _cfg(DASHBOARD_URL="https://app.datadoghq.com/dashboard/abc")
+ rows = combine(cfg, [_worker(cfg, broke_at=9)] * 10)
+ result = {
+ "build": "0.31.1-dev.280.gabc1234",
+ "pods": 12,
+ "nodes": 4,
+ "rows": rows,
+ "notes": [],
+ "restarts": 0,
+ "duration_s": 5000,
+ }
+ result["verdict"] = verdict(rows, cfg, [])
+ text = report_text(result, cfg, ["3 earlier night(s) on record"])
+ assert text.startswith(
+ "✅ Switch nightly load test on staging, 10,000 agents: held 400 msg/s, "
+ "broke at 450"
+ )
+ assert len(text) < 2000 and "—" not in text
+ assert "every 3 min" in text and cfg.dashboard_url in text
+ assert "were connected" not in text
+
+
+def test_the_report_says_when_agents_were_not_all_connected():
+ cfg = _cfg(WORKERS="2", AGENTS="2000", CONTROLLERS="100")
+ short = _worker(cfg)
+ short[2] = {**short[2], "agents_connected_min": 940}
+ rows = combine(cfg, [_worker(cfg), short])
+ result = {"build": "b", "pods": 12, "nodes": 4, "rows": rows}
+ result["verdict"] = verdict(rows, cfg, [])
+ text = report_text(result, cfg, [])
+ assert "⚠️ as few as 1,940 of the 2,000 agents were connected in a step" in text
+
+
+def test_the_report_goes_to_the_switch_room_as_its_agent(monkeypatch, capsys):
+ posts = []
+
+ class Response:
+ def raise_for_status(self) -> None:
+ pass
+
+ def post(url, json, headers, timeout):
+ posts.append((url, json, headers))
+ return Response()
+
+ monkeypatch.setattr(nightly.httpx, "post", post)
+ nightly.post_switch(_cfg(), "report")
+ assert posts == [] and "no Switch room set" in capsys.readouterr().out
+
+ with pytest.raises(ValueError, match="LOADTEST_SWITCH_TOKEN not set"):
+ nightly.post_switch(
+ _cfg(
+ SWITCH_URL="https://switch.example",
+ SWITCH_AGENT_ID="a1",
+ SWITCH_ROOM_ID="r1",
+ ),
+ "report",
+ )
+
+ cfg = _cfg(
+ SWITCH_URL="https://switch.example/",
+ SWITCH_AGENT_ID="a1",
+ SWITCH_ROOM_ID="r1",
+ SWITCH_TOKEN="t1",
+ )
+ nightly.post_switch(cfg, "report")
+ assert posts == [
+ (
+ "https://switch.example/agents/a1/message",
+ {"room_id": "r1", "content": "report"},
+ {"Authorization": "Bearer t1"},
+ )
+ ]
+ assert "t1" not in repr(cfg)
+
+
+def test_comparison_needs_three_nights_and_flags_a_drop():
+ def night(held: float, p99: float) -> dict:
+ return {
+ "verdict": {
+ "complete": True,
+ "held_per_s": held,
+ "realistic_top_p99_ms": p99,
+ }
+ }
+
+ tonight = night(300, 3000.0)
+ assert "comparisons start at 3" in compare(tonight, [night(400, 1000.0)] * 2)[0]
+ lines = compare(tonight, [night(400, 1000.0)] * 5)
+ assert "⚠️ lower" in lines[0] and "⚠️ slower" in lines[1]
+ assert "⚠️" not in "".join(compare(night(400, 1100.0), [night(400, 1000.0)] * 5))
+
+
+def test_datadog_gets_one_point_per_number_per_night():
+ cfg = _cfg()
+ rows = combine(cfg, [_worker(cfg, broke_at=9)])
+ result = {"build": "b1", "pods": 12, "rows": rows, "restarts": 0, "duration_s": 1}
+ result["verdict"] = verdict(rows, cfg, [])
+ series = datadog_series(result, cfg, 1_700_000_000.0)
+ names = {s["metric"] for s in series}
+ assert {"switch.loadtest.ok", "switch.loadtest.held_msgs_per_s"} <= names
+ assert all(len(s["points"]) == 1 for s in series)
+ p99 = {
+ next(t for t in s["tags"] if t.startswith("target:")): s
+ for s in series
+ if s["metric"] == "switch.loadtest.step.delivery_p99_ms"
+ }
+ # The idle window has no deliveries, so no p99 point.
+ assert "target:idle" not in p99
+ assert "env:staging" in p99["target:every_3_min"]["tags"]
+ assert "phase:ramp" in p99["target:150_msg/s"]["tags"]
+
+
+def test_the_deadline_is_tonights_or_three_hours_for_a_run_by_hand():
+ night = datetime(2026, 11, 4, 2, 0, tzinfo=UTC)
+ assert deadline_after(night, "04:30", timedelta(hours=3)) == night.replace(
+ hour=4, minute=30
+ )
+ day = datetime(2026, 11, 4, 15, 0, tzinfo=UTC)
+ assert deadline_after(day, "04:30", timedelta(hours=3)) == day + timedelta(hours=3)
+
+
+def test_the_build_is_the_chart_version_when_the_release_says_it():
+ assert build_of(
+ {"helm.sh/chart": "switch-0.31.1-dev.271.g989225d"}, "x@sha256:ab"
+ ) == ("0.31.1-dev.271.g989225d")
+ assert build_of({}, "ghcr.io/x/switch-core:0.31.1") == "0.31.1"
+
+
+def test_a_file_bucket_keeps_a_rehearsals_records_in_a_directory(tmp_path):
+ cfg = _cfg(BUCKET=f"file://{tmp_path}", PREFIX="rehearsal")
+ assert s3_get(cfg, "population.json") is None
+ s3_put(cfg, "population.json", {"base": "n1"})
+ for night in ("2026-11-01", "2026-11-02", "2026-11-03"):
+ s3_put(cfg, f"history/{night}.json", {"night": night})
+ assert s3_get(cfg, "population.json") == {"base": "n1"}
+ assert (tmp_path / "rehearsal" / "population.json").exists()
+ assert [h["night"] for h in history(cfg, before="2026-11-03")] == [
+ "2026-11-01",
+ "2026-11-02",
+ ]
+ s3_delete(cfg, "population.json")
+ assert s3_get(cfg, "population.json") is None
+
+
+def test_no_node_group_prefix_sizes_nothing():
+ # No AWS call: boto3 is not even installed where the tests run.
+ size_nodegroup(_cfg(CORE_NODEGROUP=""), "", 4)
+
+
+def test_one_worker_over_the_harness_error_limit_breaks_the_step():
+ # One worker past run.broken's 20 errors stops stepping; the combined row
+ # must say so, or the night reads as unfinished (the review's case).
+ cfg = _cfg(WORKERS="2", AGENTS="2000", CONTROLLERS="100")
+ a, b = _worker(cfg), _worker(cfg)[:7]
+ b[-1]["errors"] = 25
+ rows = combine(cfg, [a, b])
+ assert (
+ rows[-1]["label"] == "250 msg/s"
+ and rows[-1]["broken"] == "25 errors (worker 2)"
+ )
+ v = verdict(rows, cfg, [])
+ assert v["complete"] and v["held_per_s"] == 150 and v["broke_at_per_s"] == 250
+
+
+def test_a_night_that_could_not_read_the_cluster_still_reports(monkeypatch, tmp_path):
+ def unreachable(cfg):
+ raise RuntimeError("kubectl get deployment: the server is unreachable")
+
+ monkeypatch.setattr(nightly, "core_state", unreachable)
+ cfg = _cfg()
+ night = Night(cfg, datetime(2026, 11, 4, 2, 0, tzinfo=UTC), "n1", tmp_path)
+ night.failures.append("the run failed: RuntimeError: unreachable")
+ result = night.result()
+ assert result["build"] == "unknown" and result["rows"] == []
+ assert not result["verdict"]["ok"]
+ assert any("could not read switch-core" in n for n in result["notes"])
+ text = report_text(result, cfg, [])
+ assert text.startswith("❌") and "no numbers tonight" in text
+
+
+def test_day_sizes_try_every_step_and_say_which_failed(monkeypatch):
+ def unreachable(cfg, replicas):
+ raise RuntimeError("the server is unreachable")
+
+ sized = []
+ monkeypatch.setattr(nightly, "scale_core", unreachable)
+ monkeypatch.setattr(
+ nightly, "size_nodegroup", lambda cfg, prefix, n: sized.append((prefix, n))
+ )
+ left = nightly.day_sizes(_cfg())
+ assert sized == [("core-", 1), ("harness-", 0)]
+ assert left == ["switch-core: RuntimeError: the server is unreachable"]
+
+
+def test_down_says_so_when_staging_stays_at_its_night_size(monkeypatch):
+ sent = []
+ monkeypatch.setattr(nightly, "s3_get", lambda cfg, key: {"night": "2026-11-04"})
+ monkeypatch.setattr(nightly, "post_webhook", lambda url, text: sent.append(text))
+ monkeypatch.setattr(nightly, "send_datadog", lambda *a: sent.append("datadog"))
+ monkeypatch.setattr(nightly, "day_sizes", lambda cfg: [])
+ assert nightly.cmd_down(_cfg()) == 0 and sent == []
+
+ monkeypatch.setattr(
+ nightly, "day_sizes", lambda cfg: ["switch-core's nodes: AccessDenied"]
+ )
+ assert nightly.cmd_down(_cfg()) == 1
+ # Tonight's numbers are in Datadog already: only the warning goes out.
+ assert sent == [
+ "⚠️ staging may still be at its night size: putting it back failed for "
+ "switch-core's nodes: AccessDenied"
+ ]
+
+
+def _pod(hash_: str, build: str, ready: bool = True, node: str = "n1") -> dict:
+ return {
+ "metadata": {
+ "labels": {"pod-template-hash": hash_, "helm.sh/chart": f"switch-{build}"}
+ },
+ "spec": {
+ "nodeName": node if ready else None,
+ "containers": [{"name": "switch-core", "image": f"core:{build}"}],
+ },
+ "status": {"containerStatuses": [{"ready": True, "restartCount": 0}]}
+ if ready
+ else {},
+ }
+
+
+def test_the_build_is_what_the_ready_pods_run_not_the_template(monkeypatch):
+ # A rollout to b2 that cannot find room for its first pod: the pods still
+ # run b1, and the Pending one is no part of what the night measures.
+ deployment = {
+ "metadata": {"labels": {"helm.sh/chart": "switch-b2"}, "generation": 3},
+ "spec": {
+ "replicas": 2,
+ "selector": {"matchLabels": {"c": "core"}},
+ "template": {
+ "spec": {"containers": [{"name": "switch-core", "image": "core:b2"}]}
+ },
+ },
+ "status": {"updatedReplicas": 1, "observedGeneration": 3},
+ }
+ pods = {
+ "items": [
+ _pod("h1", "b1"),
+ _pod("h1", "b1", node="n2"),
+ _pod("h2", "b2", ready=False),
+ ]
+ }
+
+ def kubectl(cfg, *args):
+ return json.dumps(deployment if args[1] == "deployment" else pods)
+
+ monkeypatch.setattr(nightly, "kubectl", kubectl)
+ s = nightly.core_state(_cfg())
+ assert s["build"] == "b1" and s["image"] == "core:b1"
+ assert s["ready"] == 2 and s["pod_template_hash"] == ["h1"]
+ assert s["updated"] == 1 and s["by_node"] == {"n1": 1, "n2": 1}
+
+
+class _Proc:
+ """A worker process that has exited with `code`, or runs until signalled."""
+
+ def __init__(self, code: int | None = None) -> None:
+ self.returncode = code
+ self.signals: list[int] = []
+
+ def poll(self) -> int | None:
+ return self.returncode
+
+ def wait(self) -> int | None:
+ return self.returncode
+
+ def send_signal(self, sig: int) -> None:
+ self.signals.append(sig)
+ self.returncode = 0
+
+ def kill(self) -> None:
+ self.returncode = -9
+
+
+def _fake_workers(
+ monkeypatch, tmp_path, logs: dict[int, str], codes: dict[int, int | None]
+):
+ started = []
+
+ def start(cfg, run_id, seed, extra, out):
+ log = tmp_path / f"{run_id}.log"
+ log.write_text(logs.get(seed, ""))
+ w = nightly.Worker(seed, run_id, log, _Proc(codes.get(seed)))
+ started.append(w)
+ return w
+
+ monkeypatch.setattr(nightly, "start", start)
+ monkeypatch.setattr(nightly.time, "sleep", lambda s: None)
+ return started
+
+
+def test_a_worker_that_crashes_after_its_rates_stops_the_night_and_is_named(
+ monkeypatch, tmp_path
+):
+ cfg = _cfg(WORKERS="2", AGENTS="2000", CONTROLLERS="100")
+ ready = "step 1: 1000/1000 agents connected\nwaiting for the go\n"
+ started = _fake_workers(monkeypatch, tmp_path, {1: ready, 2: ready}, {})
+
+ def sleep(seconds):
+ # Worker 2 fails once the rates have started.
+ w = started[1] if len(started) > 1 else None
+ if w and (tmp_path / "rates.go").exists() and w.proc.returncode is None:
+ w.log.write_text(ready + "run failed: RuntimeError: socket gone\n")
+ w.proc.returncode = 1
+
+ monkeypatch.setattr(nightly.time, "sleep", sleep)
+ workers: list = []
+ until = datetime.now(UTC) + timedelta(hours=1)
+ stopped, rates_started = nightly.run_workers(
+ cfg, {"base": "p"}, "n1", tmp_path, until, workers
+ )
+ assert rates_started and workers == started and len(workers) == 2
+ assert stopped == "worker 2: run failed: RuntimeError: socket gone"
+ assert workers[0].proc.signals # the other one was stopped
+
+
+def test_a_worker_that_exits_unclean_after_a_break_fails_the_night(
+ monkeypatch, tmp_path
+):
+ cfg = _cfg(WORKERS="2", AGENTS="2000", CONTROLLERS="100")
+ ready = "step 1: 1000/1000 agents connected\nwaiting for the go\n"
+ _fake_workers(
+ monkeypatch,
+ tmp_path,
+ {
+ 1: ready + " steady150 broke (p99 25 s); not stepping higher\n",
+ 2: ready + "teardown failed\n",
+ },
+ {1: 0, 2: 1},
+ )
+ workers: list = []
+ until = datetime.now(UTC) + timedelta(hours=1)
+ stopped, _ = nightly.run_workers(cfg, {"base": "p"}, "n1", tmp_path, until, workers)
+ assert stopped == "worker 2: exited 1: teardown failed"
+
+
+def test_a_seeder_still_running_is_stopped_before_its_batch_is_removed(
+ monkeypatch, tmp_path
+):
+ cfg = _cfg(
+ WORKERS="2", AGENTS="2000", CONTROLLERS="100", BUCKET=f"file://{tmp_path}"
+ )
+ seeders = []
+
+ class Stopped(Exception):
+ pass
+
+ def start(cfg, run_id, seed, extra, out):
+ proc = _Proc()
+
+ def wait():
+ raise Stopped # the pod's SIGTERM, arriving while it makes agents
+
+ proc.wait = wait
+ w = nightly.Worker(seed, run_id, tmp_path / f"{run_id}.log", proc)
+ seeders.append(w)
+ return w
+
+ removed = []
+ monkeypatch.setattr(nightly, "start", start)
+ monkeypatch.setattr(nightly.time, "sleep", lambda s: None)
+ monkeypatch.setattr(
+ nightly,
+ "remove_population",
+ lambda cfg, pop, out: removed.append(
+ (dict(pop), list(seeders[0].proc.signals))
+ ),
+ )
+ with pytest.raises(Stopped):
+ nightly.seed_population(cfg, "n1", tmp_path)
+ # Stopped first, then removed, with the batch it was making counted.
+ assert removed == [({**removed[0][0], "batches": 1}, [signal.SIGINT])]
+ assert s3_get(cfg, "population.json")["batches"] == 1
+
+
+def test_a_pod_being_stopped_leaves_the_half_made_batches_to_the_next_night(
+ monkeypatch, tmp_path
+):
+ cfg = _cfg(
+ WORKERS="2", AGENTS="2000", CONTROLLERS="100", BUCKET=f"file://{tmp_path}"
+ )
+
+ def start(cfg, run_id, seed, extra, out):
+ proc = _Proc()
+
+ def wait():
+ raise nightly.Interrupted("the pod was stopped (SIGTERM)")
+
+ proc.wait = wait
+ return nightly.Worker(seed, run_id, tmp_path / f"{run_id}.log", proc)
+
+ removed = []
+ monkeypatch.setattr(nightly, "start", start)
+ monkeypatch.setattr(nightly.time, "sleep", lambda s: None)
+ monkeypatch.setattr(nightly, "remove_population", lambda *a: removed.append(a))
+ with pytest.raises(nightly.Interrupted):
+ nightly.seed_population(cfg, "n1", tmp_path)
+ pop = s3_get(cfg, "population.json")
+ assert removed == [] and pop["batches"] == 1 and not pop["complete"]
+
+
+def test_the_wait_ends_with_every_pod_ready_on_one_build_though_a_newer_one_waits(
+ monkeypatch,
+):
+ states = iter(
+ [
+ {
+ "ready": 11,
+ "updated": 0,
+ "pod_template_hash": ["h1"],
+ "generation_seen": True,
+ },
+ {
+ "ready": 12,
+ "updated": 0,
+ "pod_template_hash": ["h1"],
+ "generation_seen": True,
+ },
+ ]
+ )
+ monkeypatch.setattr(nightly, "core_state", lambda cfg: next(states))
+ monkeypatch.setattr(nightly.time, "sleep", lambda s: None)
+ assert nightly.wait_core(_cfg(), 12, 1500)["ready"] == 12
diff --git a/loadtest/test_report.py b/loadtest/test_report.py
new file mode 100644
index 000000000..84b63d105
--- /dev/null
+++ b/loadtest/test_report.py
@@ -0,0 +1,35 @@
+"""Numbers from a run's records. No requests."""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent))
+
+from agents import Stats # noqa: E402
+from report import pct, phase_summary # noqa: E402
+
+
+def test_p99_is_the_nearest_rank_not_the_maximum():
+ ms = [i / 1000 for i in range(1, 101)] # 1 to 100 ms
+ assert pct(ms, 0.99) == 99.0
+ assert pct(ms, 0.5) == 50.0 and pct(ms, 1.0) == 100.0
+ # 0.07 * 100 is 7.000000000000001 in floating point; still the 7th.
+ assert pct(ms, 0.07) == 7.0
+ assert pct([], 0.99) is None and pct([0.25], 0.99) == 250.0
+
+
+def test_a_dropped_controller_socket_is_one_error_however_many_agents_it_carried():
+ stats = Stats(phase="steady150@1000")
+ stats.event("controller_drop", "c1", why="1011")
+ for i in range(20):
+ stats.event("detached", f"a{i}", why="1011")
+ phase = {
+ "name": "steady150@1000",
+ "scenario": "steady",
+ "seconds": 60.0,
+ "agents": 1000,
+ }
+ s = phase_summary(phase, stats, [], [1000])
+ assert s["errors"] == 1 and s["controller_drops"] == 1 and s["detached"] == 20
diff --git a/loadtest/test_run.py b/loadtest/test_run.py
new file mode 100644
index 000000000..ffe513f0a
--- /dev/null
+++ b/loadtest/test_run.py
@@ -0,0 +1,285 @@
+"""The run's arguments and plan for a message-rate sweep (`--rates`). No requests."""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).parent))
+
+from kube import Refused # noqa: E402
+from run import ( # noqa: E402
+ broken,
+ kept_record,
+ parse_args,
+ plan,
+ rate_phase,
+ resolve_urls,
+ take_over,
+)
+from seed import SeededAgent # noqa: E402
+
+
+def _args(*extra: str):
+ return parse_args(["--namespace", "switch-test", "--steps", "300", *extra])
+
+
+def test_each_rate_gets_a_steady_window_of_its_own():
+ a = _args("--scenarios", "idle", "--rates", "10,15,20", "--rate-duration", "90")
+ _, schedules = plan(a)
+ assert list(schedules) == [
+ "idle@300",
+ "steady10@300",
+ "steady15@300",
+ "steady20@300",
+ ]
+ assert schedules["idle@300"] == []
+ for r in (10, 15, 20):
+ sent = len(schedules[rate_phase(r, 300)])
+ # Poisson arrivals: about rate x duration messages.
+ assert 0.8 * r * 90 < sent < 1.2 * r * 90
+
+
+def test_without_rates_the_plan_is_unchanged():
+ a = _args("--scenarios", "idle,steady")
+ _, schedules = plan(a)
+ assert list(schedules) == ["idle@300", "steady@300"]
+
+
+def test_each_rate_may_have_a_window_of_its_own_length():
+ a = _args(
+ "--scenarios", "idle", "--rates", "1,2,30", "--rate-duration", "600,600,60"
+ )
+ _, schedules = plan(a)
+ # Poisson arrivals: about rate x duration messages in each window.
+ for r, secs in ((1, 600), (2, 600), (30, 60)):
+ sent = len(schedules[rate_phase(r, 300)])
+ assert 0.8 * r * secs < sent < 1.2 * r * secs
+ assert _args("--rates", "1,2", "--rate-duration", "45").rate_durations == [45, 45]
+ with pytest.raises(SystemExit):
+ _args("--rates", "1,2,3", "--rate-duration", "600,60")
+
+
+def test_rates_must_go_up():
+ with pytest.raises(SystemExit):
+ _args("--rates", "20,10")
+ with pytest.raises(SystemExit):
+ _args("--rates", "0,10")
+
+
+def test_a_rate_name_reads_as_rate_at_agents():
+ assert rate_phase(15, 300) == "steady15@300"
+ assert rate_phase(7.5, 300) == "steady7.5@300"
+
+
+ ("summary", "why"),
+ [
+ ({"deliveries_missing": 0, "delivery_ms_p99": 900.0, "errors": 0}, None),
+ (
+ {"deliveries_missing": 2, "delivery_ms_p99": 900.0, "errors": 0},
+ "2 deliveries missing",
+ ),
+ (
+ {"deliveries_missing": 0, "delivery_ms_p99": 21179.1, "errors": 0},
+ "p99 21.2 s",
+ ),
+ (
+ {"deliveries_missing": 0, "delivery_ms_p99": 900.0, "errors": 21},
+ "21 errors",
+ ),
+ ({"deliveries_missing": 0, "delivery_ms_p99": None, "errors": 0}, None),
+ ],
+)
+def test_a_window_is_broken_when_deliveries_are_lost_slow_or_failing(summary, why):
+ assert broken(summary) == why
+
+
+def test_keeping_or_reusing_needs_controllers():
+ for flag in (["--keep"], ["--reuse", "20261102t104605w1"]):
+ with pytest.raises(SystemExit):
+ parse_args(["--namespace", "switch-test", "--steps", "10", *flag])
+ a = parse_args(
+ [
+ "--namespace",
+ "switch-test",
+ "--steps",
+ "10",
+ "--controllers",
+ "2",
+ "--keep",
+ ]
+ )
+ assert a.keep
+
+
+def _kept(a, membership, n=10):
+ seeded = [SeededAgent(i, f"agent-{i}", f"load-r1-a{i:04d}", "") for i in range(n)]
+ return kept_record("r1", "r1", seeded, ["room-0", "room-1"], membership, a)
+
+
+def test_a_kept_run_records_ids_and_no_credential():
+ a = parse_args(
+ [
+ "--namespace",
+ "switch-test",
+ "--steps",
+ "10",
+ "--controllers",
+ "2",
+ "--keep",
+ ]
+ )
+ membership, _ = plan(a)
+ record = _kept(a, membership)
+
+ assert record["names_from"] == "r1"
+ assert set(record["agents"][0]) == {"index", "id", "name"}
+ assert "credential" not in str(record) and "key" not in record
+
+
+def test_a_later_run_takes_the_agents_over_only_with_the_same_rooms():
+ a = parse_args(
+ [
+ "--namespace",
+ "switch-test",
+ "--steps",
+ "10",
+ "--controllers",
+ "2",
+ "--keep",
+ ]
+ )
+ membership, _ = plan(a)
+ record = _kept(a, membership)
+
+ taken = take_over(record, membership, 10)
+ assert [s.id for s in taken] == [f"agent-{i}" for i in range(10)]
+
+ other = parse_args(
+ [
+ "--namespace",
+ "switch-test",
+ "--steps",
+ "10",
+ "--controllers",
+ "2",
+ "--seed",
+ "2",
+ ]
+ )
+ with pytest.raises(RuntimeError, match="rooms differ"):
+ take_over(record, plan(other)[0], 10)
+ with pytest.raises(RuntimeError, match="kept 10 agents"):
+ take_over(record, membership, 12)
+
+
+class _Release:
+ """The release's own URLs and its Ingresses' hostnames, as `Kube` reads them."""
+
+ def public_urls(self) -> tuple[str, str]:
+ return "https://dev.example.com", "https://dev.example.com"
+
+ def ingress_hosts(self) -> set[str]:
+ return {"qa-gateway.tail.ts.net", "qa-api.tail.ts.net"}
+
+ def service_hosts(self) -> set[str]:
+ return {"switch-qa-switch-core.switch-qa.svc"}
+
+
+def test_urls_default_to_the_release_and_may_be_its_ingresses() -> None:
+ a = parse_args(["--namespace", "switch-qa"])
+ resolve_urls(a, _Release()) # type: ignore[arg-type]
+ assert (a.gateway_url, a.api_url) == ("https://dev.example.com",) * 2
+
+ a = parse_args(
+ [
+ "--namespace",
+ "switch-qa",
+ "--gateway-url",
+ "https://qa-gateway.tail.ts.net/",
+ "--api-url",
+ "https://qa-api.tail.ts.net",
+ ]
+ )
+ resolve_urls(a, _Release()) # type: ignore[arg-type]
+ assert a.gateway_url == "https://qa-gateway.tail.ts.net"
+ assert a.api_url == "https://qa-api.tail.ts.net"
+
+
+def test_from_inside_the_cluster_the_urls_may_be_its_services() -> None:
+ url = "http://switch-qa-switch-core.switch-qa.svc:8000"
+ a = parse_args(["--namespace", "switch-qa", "--api-url", url, "--gateway-url", url])
+ resolve_urls(a, _Release()) # type: ignore[arg-type]
+ assert (a.gateway_url, a.api_url) == (url, url)
+
+
+def test_urls_of_another_environment_are_refused() -> None:
+ a = parse_args(
+ ["--namespace", "switch-qa", "--api-url", "https://other-api.tail.ts.net"]
+ )
+ with pytest.raises(Refused, match="--api-url"):
+ resolve_urls(a, _Release()) # type: ignore[arg-type]
+
+
+class _ReleaseWithoutUrls(_Release):
+ """A release that takes its URLs from a Secret, so `Kube` reads none."""
+
+ def public_urls(self) -> tuple[None, None]: # type: ignore[override]
+ return None, None
+
+
+def test_without_the_release_s_urls_a_given_url_must_still_be_its_own() -> None:
+ gw = ["--gateway-url", "https://qa-gateway.tail.ts.net"]
+ a = parse_args(
+ ["--namespace", "switch-qa", *gw, "--api-url", "https://qa-api.tail.ts.net"]
+ )
+ resolve_urls(a, _ReleaseWithoutUrls()) # type: ignore[arg-type]
+ assert a.api_url == "https://qa-api.tail.ts.net"
+
+ a = parse_args(
+ [
+ "--namespace",
+ "switch-qa",
+ *gw,
+ "--api-url",
+ "https://other-api.tail.ts.net",
+ ]
+ )
+ with pytest.raises(Refused, match="the release names none"):
+ resolve_urls(a, _ReleaseWithoutUrls()) # type: ignore[arg-type]
+
+
+ ("flags", "why"),
+ [
+ (["--steps", "300", "--rates", "10,10,20"], "--rates must be increasing"),
+ (["--steps", "100,100"], "--steps must be increasing"),
+ (["--steps", "10", "--run-id", "../outside"], "--run-id takes"),
+ (["--steps", "10", "--run-id", ".hidden"], "--run-id takes"),
+ (["--steps", "10", "--controllers", "2", "--reuse", "a/b"], "--reuse takes"),
+ (["--cleanup", "../x"], "--cleanup takes"),
+ ],
+)
+def test_repeated_steps_and_run_ids_outside_runs_are_refused(flags, why, capsys):
+ with pytest.raises(SystemExit):
+ parse_args(["--namespace", "switch-qa", *flags])
+ assert why in capsys.readouterr().err
+
+
+def test_run_ids_as_the_nightly_makes_them_are_taken():
+ a = parse_args(["--namespace", "switch-qa", "--run-id", "n20261104t0942w1"])
+ assert a.run_id == "n20261104t0942w1"
+ assert parse_args(["--namespace", "switch-qa", "--cleanup", "*"]).cleanup == "*"
+
+
+def test_seed_only_keeps_what_it_makes_and_reuses_nothing() -> None:
+ base = ["--namespace", "switch-qa", "--controllers", "2"]
+ with pytest.raises(SystemExit):
+ parse_args([*base, "--seed-only"])
+ with pytest.raises(SystemExit):
+ parse_args([*base, "--keep", "--seed-only", "--reuse", "x"])
+ a = parse_args([*base, "--keep", "--seed-only", "--concurrency", "4"])
+ assert a.seed_only and a.concurrency == 4
diff --git a/loadtest/test_seed.py b/loadtest/test_seed.py
new file mode 100644
index 000000000..ca39ebf88
--- /dev/null
+++ b/loadtest/test_seed.py
@@ -0,0 +1,116 @@
+"""Which workspace a run's admin session is bound to, against a fake gateway."""
+
+from __future__ import annotations
+
+import asyncio
+import sys
+from pathlib import Path
+
+import httpx
+import pytest
+
+sys.path.insert(0, str(Path(__file__).parent))
+
+from seed import Admin, NoWorkspace, RequestIds # noqa: E402
+
+HOME = {"id": "home", "slug": "home", "name": "Main"}
+LOAD = {"id": "load", "slug": "load-tests", "name": "load-tests"}
+
+
+class Gateway:
+ """Answers the routes `use_workspace` and `refuse_bridges` call, and
+ records each request with the session cookie it carried."""
+
+ def __init__(self, workspaces: list[dict], bridges: list[dict] | None = None):
+ self.workspaces = workspaces
+ self.bridges = bridges or []
+ self.seen: list[tuple[str, str, str | None]] = []
+
+ def __call__(self, request: httpx.Request) -> httpx.Response:
+ cookie = request.headers.get("cookie", "").removeprefix("switch_auth=")
+ path = request.url.path.removeprefix("/gateway")
+ self.seen.append((request.method, path, cookie or None))
+ if (request.method, path) == ("GET", "/tenants/current"):
+ return httpx.Response(200, json=HOME)
+ if (request.method, path) == ("GET", "/tenants"):
+ return httpx.Response(200, json=self.workspaces)
+ if request.method == "POST" and path.startswith("/tenants/"):
+ tenant = path.split("/")[2]
+ return httpx.Response(200, json={}, headers=_cookie(tenant))
+ if (request.method, path) == ("POST", "/organizations"):
+ body = {"organization": {"id": "org"}, "workspace": LOAD}
+ return httpx.Response(201, json=body, headers=_cookie("load"))
+ if (request.method, path) == ("GET", "/collaborations"):
+ return httpx.Response(200, json=self.bridges)
+ return httpx.Response(404)
+
+
+def _cookie(tenant: str) -> dict[str, str]:
+ return {"set-cookie": f"switch_auth=cookie-{tenant}; Path=/; HttpOnly; Secure"}
+
+
+def _admin(gateway: Gateway) -> Admin:
+ admin = Admin("https://gateway.test", RequestIds("t"))
+ admin.http = httpx.AsyncClient(
+ base_url="https://gateway.test", transport=httpx.MockTransport(gateway)
+ )
+ admin._cookie = "cookie-home"
+ return admin
+
+
+def test_switches_into_the_workspace_and_puts_the_admins_selection_back() -> None:
+ gateway = Gateway([HOME, LOAD])
+ admin = _admin(gateway)
+
+ async def go() -> dict:
+ found = await admin.use_workspace("load-tests")
+ await admin.call("GET", "/agents")
+ return found
+
+ assert asyncio.run(go()) == {"id": "load", "name": "load-tests", "made": False}
+ assert gateway.seen[2:] == [
+ ("POST", "/tenants/load/switch", "cookie-home"),
+ ("POST", "/tenants/home/switch", "cookie-load"),
+ # The session stays in the workspace; the admin's selection went back.
+ ("GET", "/agents", "cookie-load"),
+ ]
+
+
+def test_makes_the_workspace_when_the_admin_has_none_of_that_name() -> None:
+ gateway = Gateway([HOME])
+ admin = _admin(gateway)
+
+ assert asyncio.run(admin.use_workspace("load-tests")) == {
+ "id": "load",
+ "name": "load-tests",
+ "made": True,
+ }
+ assert ("POST", "/organizations", "cookie-home") in gateway.seen
+ assert gateway.seen[-1] == ("POST", "/tenants/home/switch", "cookie-load")
+ assert admin._cookie == "cookie-load"
+
+
+def test_a_run_reusing_kept_agents_never_makes_the_workspace() -> None:
+ # Ten reusing workers started together would each make one otherwise.
+ gateway = Gateway([HOME])
+ admin = _admin(gateway)
+
+ with pytest.raises(NoWorkspace, match="no workspace named load-tests"):
+ asyncio.run(admin.use_workspace("load-tests", create=False))
+ assert ("POST", "/organizations", "cookie-home") not in gateway.seen
+
+
+def test_refuses_a_name_two_workspaces_share() -> None:
+ admin = _admin(Gateway([HOME, LOAD, {**LOAD, "id": "load-2"}]))
+
+ with pytest.raises(RuntimeError, match="2 workspaces named load-tests"):
+ asyncio.run(admin.use_workspace("load-tests"))
+
+
+def test_refuses_a_workspace_with_an_active_bridge() -> None:
+ slack = {"bridge_type": "slack", "display_name": "Team", "status": "active"}
+ paused = {"bridge_type": "mattermost", "display_name": "MM", "status": "inactive"}
+
+ with pytest.raises(RuntimeError, match="slack 'Team'"):
+ asyncio.run(_admin(Gateway([HOME], [slack, paused])).refuse_bridges())
+ asyncio.run(_admin(Gateway([HOME], [paused])).refuse_bridges())
diff --git a/loadtest/test_workload.py b/loadtest/test_workload.py
new file mode 100644
index 000000000..b6ce329cd
--- /dev/null
+++ b/loadtest/test_workload.py
@@ -0,0 +1,111 @@
+"""Determinism and shape of the planned workload. No server needed:
+
+uv run --project core pytest loadtest/test_workload.py
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).parent))
+
+from kube import Refused, guard_target # noqa: E402
+from workload import ( # noqa: E402
+ build_membership,
+ build_schedule,
+ fingerprint,
+ payload,
+)
+
+
+def _schedule(seed: int, scenario: str = "steady", **kw):
+ rooms = build_membership(seed, 40, 20, 4)
+ args = dict(
+ seed=seed,
+ scenario=scenario,
+ step=40,
+ duration=60,
+ rooms=rooms,
+ active=40,
+ rate=5,
+ )
+ args.update(kw)
+ return build_schedule(**args)
+
+
+def test_same_seed_same_workload():
+ assert build_membership(7, 40, 20, 4) == build_membership(7, 40, 20, 4)
+ for sc in ("steady", "burst"):
+ assert _schedule(7, sc) == _schedule(7, sc)
+ assert fingerprint(_schedule(7, sc)) == fingerprint(_schedule(7, sc))
+
+
+def test_different_seed_different_workload():
+ assert build_membership(7, 40, 20, 4) != build_membership(8, 40, 20, 4)
+ assert fingerprint(_schedule(7)) != fingerprint(_schedule(8))
+
+
+def test_every_agent_has_a_room_and_rooms_are_full():
+ rooms = build_membership(3, 40, 20, 4)
+ assert all(len(r) == 4 and len(set(r)) == 4 for r in rooms)
+ assert {a for r in rooms for a in r} == set(range(40))
+
+
+def test_messages_go_to_connected_members_only():
+ rooms = build_membership(3, 100, 50, 4)
+ sch = build_schedule(
+ seed=3, scenario="steady", step=30, duration=60, rooms=rooms, active=30, rate=10
+ )
+ assert sch
+ for s in sch:
+ assert s.sender < 30 and s.sender in rooms[s.room]
+ assert sum(1 for a in rooms[s.room] if a < 30) >= 2
+
+
+def test_burst_adds_spikes_and_idle_sends_nothing():
+ steady, burst = (
+ _schedule(1, "steady"),
+ _schedule(1, "burst", burst_every=20, burst_size=50),
+ )
+ assert len(burst) >= len(steady) + 3 * 50 - 5
+ assert _schedule(1, "idle") == []
+
+
+def test_hot_room_spikes_land_in_one_room():
+ sch = _schedule(1, "burst", rate=0, burst_every=20, burst_size=30, hot_room=True)
+ assert len({s.room for s in sch}) == 1
+
+
+def test_payload_sizes_and_token():
+ sch = _schedule(5, payload_min=40, payload_max=2000)
+ assert all(40 <= s.size <= 2000 for s in sch)
+ text = payload("run.steady.40.3", 300)
+ assert text.startswith("lt:run.steady.40.3: ") and len(text) == 300
+
+
+ "ns,url",
+ [
+ ("switch", ""),
+ ("switch-prod", ""),
+ ("switch-test", "https://switch-api.tail.ts.net"),
+ ("switch-test", "https://switch-gateway.tail.ts.net"),
+ ("switch-test", "https://switch-prod-api.example.com"),
+ ("switch-test", "http://switch-switch-core.switch.svc:8000"),
+ ],
+)
+def test_refuses_pilot_and_prod(ns, url):
+ with pytest.raises(Refused):
+ guard_target(ns, url)
+
+
+def test_allows_other_namespaces():
+ guard_target("switch-test", "https://switch-test-api.tail.ts.net")
+ guard_target("switch-qa", "https://switch-qa-gateway.tail.ts.net")
+ guard_target(
+ "switch-test",
+ "http://switch-test-switch-core.switch-test.svc.cluster.local:8000",
+ )
diff --git a/loadtest/workload.py b/loadtest/workload.py
new file mode 100644
index 000000000..1ca1f1074
--- /dev/null
+++ b/loadtest/workload.py
@@ -0,0 +1,171 @@
+"""The workload, worked out in advance from the seed and the flags.
+
+Nothing in here talks to a server or reads a clock. Room membership, the
+message schedule and the payload sizes all come from `random.Random` seeded
+from `--seed` plus the name of the thing being drawn, so the same seed and the
+same flags give the same workload on every run, and two different steps or
+scenarios do not share a stream of random numbers (adding a scenario does not
+shift the messages of another one).
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import random
+from dataclasses import asdict, dataclass
+
+SCENARIOS = ("idle", "steady", "burst")
+
+
+def rng_for(seed: int, *labels: object) -> random.Random:
+ """A random stream of its own for one purpose, derived from the seed."""
+ key = ":".join([str(seed), *(str(label) for label in labels)])
+ return random.Random(int.from_bytes(hashlib.sha256(key.encode()).digest()[:8]))
+
+
+def build_membership(
+ seed: int, n_agents: int, n_rooms: int, room_size: int
+) -> list[list[int]]:
+ """Which agents (by index) sit in each room.
+
+ Room j draws `room_size` members from the agents. Every agent is placed in
+ at least one room when there are enough seats, so no agent is connected
+ with nothing to read: the first pass deals agents out in a shuffled order,
+ the rest of the seats are drawn at random.
+ """
+ if n_agents < 2:
+ raise ValueError("a room needs at least two agents, so at least two agents")
+ size = max(2, min(room_size, n_agents))
+ rng = rng_for(seed, "membership", n_agents, n_rooms, size)
+ order = list(range(n_agents))
+ rng.shuffle(order)
+ rooms: list[list[int]] = [[] for _ in range(n_rooms)]
+ for i, agent in enumerate(order):
+ room = rooms[i % n_rooms]
+ if len(room) < size:
+ room.append(agent)
+ for room in rooms:
+ others = [a for a in range(n_agents) if a not in room]
+ room.extend(rng.sample(others, size - len(room)))
+ room.sort()
+ return rooms
+
+
+@dataclass(frozen=True, slots=True)
+class Send:
+ """One message the harness will post: when, where, by whom, how big."""
+
+ at: float # seconds from the start of the measured window
+ room: int # index into the membership
+ sender: int # agent index, a member of the room
+ size: int # payload length in characters
+
+
+def _payload_size(rng: random.Random, lo: int, hi: int) -> int:
+ # Log-uniform: mostly short chat lines, now and then a long one, which is
+ # what a room of agents posting results looks like.
+ if hi <= lo:
+ return lo
+ return int(round(lo * (hi / lo) ** rng.random()))
+
+
+def _pick(
+ rng: random.Random,
+ rooms: list[list[int]],
+ live: list[int],
+ active: int,
+ hot_room: int | None,
+) -> tuple[int, int]:
+ """A room with at least two connected members (`live`, the same for a
+ whole schedule), and a connected sender in it."""
+ room = hot_room if hot_room is not None else rng.choice(live)
+ senders = [a for a in rooms[room] if a < active]
+ return room, rng.choice(senders)
+
+
+def _live(members: list[int], active: int) -> int:
+ return sum(1 for a in members if a < active)
+
+
+def build_schedule(
+ *,
+ seed: int,
+ scenario: str,
+ step: int,
+ duration: float,
+ rooms: list[list[int]],
+ active: int,
+ rate: float,
+ burst_every: float = 30.0,
+ burst_size: int = 100,
+ burst_window: float = 2.0,
+ hot_room: bool = False,
+ payload_min: int = 40,
+ payload_max: int = 2000,
+) -> list[Send]:
+ """The messages of one scenario at one step, in time order.
+
+ `active` is how many agents are connected at this step (agents 0 to
+ active-1); a message only goes to a room with two or more of them, sent by
+ one of them.
+
+ - idle: nothing.
+ - steady: Poisson arrivals at `rate` per second for `duration` seconds.
+ - burst: the steady background at `rate`, plus `burst_size` messages
+ spread evenly at random over `burst_window` seconds, every `burst_every`
+ seconds (first one `burst_every / 2` in). With `hot_room`, the spikes all
+ land in the room with the most connected members.
+ """
+ if scenario not in SCENARIOS:
+ raise ValueError(f"unknown scenario {scenario!r}")
+ if scenario == "idle":
+ return []
+ rng = rng_for(seed, "schedule", scenario, step)
+ live = [j for j, members in enumerate(rooms) if _live(members, active) >= 2]
+ if not live:
+ raise ValueError(f"no room has two connected members at {active} agents")
+ sends: list[Send] = []
+ t = 0.0
+ while rate > 0:
+ t += rng.expovariate(rate)
+ if t >= duration:
+ break
+ room, sender = _pick(rng, rooms, live, active, None)
+ sends.append(
+ Send(t, room, sender, _payload_size(rng, payload_min, payload_max))
+ )
+ if scenario == "burst":
+ hot = (
+ max(range(len(rooms)), key=lambda j: (_live(rooms[j], active), -j))
+ if hot_room
+ else None
+ )
+ start = burst_every / 2
+ while start < duration:
+ for _ in range(burst_size):
+ at = start + rng.random() * burst_window
+ if at >= duration:
+ continue
+ room, sender = _pick(rng, rooms, live, active, hot)
+ sends.append(
+ Send(at, room, sender, _payload_size(rng, payload_min, payload_max))
+ )
+ start += burst_every
+ sends.sort(key=lambda s: (s.at, s.room, s.sender))
+ return sends
+
+
+def payload(token: str, size: int) -> str:
+ """The message text: the token the receivers match on, padded to `size`."""
+ head = f"lt:{token}: "
+ filler = "load test " * (size // 10 + 1)
+ return head + filler[: max(0, size - len(head))]
+
+
+def fingerprint(obj: object) -> str:
+ """A short hash of a membership or a schedule, to show two runs matched."""
+ if isinstance(obj, list) and obj and isinstance(obj[0], Send):
+ obj = [asdict(s) for s in obj]
+ blob = json.dumps(obj, sort_keys=True, separators=(",", ":"))
+ return hashlib.sha256(blob.encode()).hexdigest()[:12]