The fastest, litest AI Gateway. Rust core with Python SDK. Call 100+ LLM APIs in OpenAI (or native) format with cost tracking, guardrails, load balancing, and logging [Bedrock, Azure, OpenAI, Anthropic, OpenAI, VertexAI, vLLM, Nvidia NIM]
Python ai-gatewayanthropicazure-openaibedrockgatewaylangchainlitellmllmllm-gatewayllmopsmcp-gatewayopenaiopenai-proxyrustrust-aivertex-ai
test(perf): ramp the load cases until the pod stops keeping up and floor sustained_rps (#47758)
Kerry Lub99d2cc06e9f
diff --git a/tests/e2e/AGENTS.md b/tests/e2e/AGENTS.mdindex d976a2b5ee3..ef57135a4d2 100644--- a/tests/e2e/AGENTS.md+++ b/tests/e2e/AGENTS.md@@ -44,7 +44,7 @@ Each subdirectory under `tests/e2e/` is one suite, scoped to an endpoint family- `logging/` - logging-integration delivery (datadog and friends)- `security/` - secret handling and log-leak protection- `router/` - routing and reliability behavior (fallbacks, cooldowns) plus the memory tests (`test_reliability_memory_e2e.py`: every worker's RSS as read at collection time, before any test traffic, must sit under a fixed idle budget, the release-gate check for a DB-backed boot that idles near the pod limit the way v1.100.x did, and a few hundred failing requests with retries and fallbacks must not grow proxy RSS past a loose ceiling over the measured batches nor store a request snapshot past a fixed size, the release-gate check for the v1.100.0 retry-breadcrumb leak)-- `perf/` - performance-category tests, kept OUT of the main suite: throughput/load SLO tests are a different testing category from functional e2e (variance-driven, historically flaky) and live outside this suite until re-implemented as their own pipeline (LIT-5163). Do not add a live load test that runs in the default collection. What lives here: the weekly session-anomaly test (`test_weekly_session_anomaly_e2e.py`, Claude Code-shaped multi-turn sessions against real providers with ceilings on error rate, cache read/write, turn time, and spend, marked `weekly` and deselected unless `E2E_WEEKLY_ANOMALY` is set, driven by `.github/workflows/weekly_load_anomaly.yml`), the Redis chaos test (`test_redis_chaos_e2e.py`, locust load against mock deployments split round robin over `/chat/completions` and `/v1/messages`, one endpoint per simulated user, with `CLIENT PAUSE ALL` on the proxy's Redis mid-run to simulate it being down outright, asserting zero failed requests on every endpoint, budgeting RSS and CPU-per-request as ratios against the same run's healthy phase, and holding p50/p90/p99 latency and log-bytes-per-request to flat ceilings (a ratio cannot bound those two: an open breaker skips Redis instead of waiting on it, so the chaos phase can measure cheaper than baseline while still being far slower than a user should see), needs a proxy booted from `gateway/redis_chaos_ci_config.yml` on the same host with `E2E_PROXY_PID` and `E2E_PROXY_LOG` set, marked `redis_chaos`, deselected unless `E2E_PERF_TESTS` is set and excluded from the per-PR selector like the rest of `perf/`, driven by `.github/workflows/test-e2e-redis-chaos.yml` and by the Buildkite `e2e-redis-chaos` step in project-releaser, which runs the proxy, Postgres and Valkey co-located with pytest in one pod and sets the opt-in). New perf tests go here too, marked `perf`, behind the same `E2E_PERF_TESTS` opt-in. The perf cases (`test_perf_cases.py`, one `PerformanceTestCase` per module in `perf/cases/`, no `_e2e` suffix) are the request-performance runs. A case names what the gateway was deployed with (model groups and guardrails, written out in the gateway's config: the `proxy_config` block of project-releaser's perf helm values in the lane, `gateway/perf_ci_config.yml` locally, with each deployment's mock knobs in an `x-mock-knobs` extra header so the model ids are real ones like `gpt-6-luna` and `anthropic.claude-sonnet-5`) and lists what the test creates and sends: teams, Keycloak JWT users, virtual keys, `RequestShape`s (a coding-agent session from `perf/setup/corpus/` whose input-token size is drawn per request from the shape's range, with or without tools, and the output cap), a `Traffic` plan (seed, rps, request count, endpoint, shape and stream shares) and `Limit`s (one metric at one percentile under one ceiling). The infra side lives in `perf/setup/`: `provision.py` checks the deployed config matches the case and builds the callers, `plan.py` draws the request list from the seed, `run.py` replays it from a separate Locust process (`locustfile.py`) that builds each body right before sending and records status, header, first-byte, first-token and total time per request, and `observe.py` joins all of that with what the mocks received and the workers' RSS to score every `Limit`. Limits are about what the caller sees and what the gateway costs: every non-blocked request is a 200 (`unexpected_status_count`), the seconds the gateway added per request at p50 and p99 (`gateway_added_seconds`, to the first token on streams), the client's time to the first token of a stream at p50 and p99 (`time_to_first_token_seconds`, mock delay included), worker memory growth (`memory_growth_megabytes`), and the Postgres statements and Redis commands a request waits on before it is forwarded (`postgres_calls_per_request`, `redis_calls_per_request`: auth and routing reads, the rate-limiter script), with the work the gateway does after the response or on a timer (spend writes, daily aggregates, config reloads) reported separately as `postgres_background_calls_per_request` and `redis_background_calls_per_request`; the same overhead as the gateway itself reports it, read from the `x-litellm-overhead-duration-ms` header per request (`litellm_overhead_seconds`, gated next to `gateway_added_seconds`, the outside reading of the same thing, so the two can be charted together) and the gateway's own end-to-end time from `x-litellm-response-duration-ms` (`litellm_response_seconds`), the run mean of the gateway's `litellm_redis_latency` histogram from a `/metrics` scrape before and after the run (`telemetry.py`, `prometheus_redis_latency_mean_seconds`), and the database's own view of the run from the same `pg_stat_statements` rows and a once-a-second `pg_stat_activity` sample: execution seconds per wall second (`postgres_active_sessions`, the average-active-sessions number RDS reports as DB load), the slowest statement's mean (`postgres_slowest_statement_seconds`, named in the detail) and the peak client connection count (`postgres_peak_connections`), each under a ceiling set from the first local runs and due a retune once the lane has its ten-run baseline; `datastore.py` reads `pg_stat_statements` before and after the run and holds a Redis `MONITOR` connection during it, sorts each statement by verb and table and each command by key, and the run output lists the top statements and commands per bucket so a red run names what grew. Whether a body, a guardrail call or a spend row is right is printed as a diagnostic, never a limit, that is the functional suites' job. The providers and the guardrail are out-of-process servers in `perf/setup/mocks/` (`server.py openai|anthropic|bedrock|guardrail`, each answering with a reply generated for the request in that provider's wire format, sized from the prompt and output cap, a tool call when tools were offered, waiting the case's `mock_latency` first, request logs at `/mock/requests` and `/mock/calls`), reached through `E2E_PERF_MOCK_*_URL` by both the gateway config and the test. Adding a perf test is adding a module with a `CASE` to `perf/cases/`; a new deployment or guardrail is a config change; a new metric is a name in `Metric` plus a branch in `observe.observe`+- `perf/` - performance-category tests, kept OUT of the main suite: throughput/load SLO tests are a different testing category from functional e2e (variance-driven, historically flaky) and live outside this suite until re-implemented as their own pipeline (LIT-5163). Do not add a live load test that runs in the default collection. What lives here: the weekly session-anomaly test (`test_weekly_session_anomaly_e2e.py`, Claude Code-shaped multi-turn sessions against real providers with ceilings on error rate, cache read/write, turn time, and spend, marked `weekly` and deselected unless `E2E_WEEKLY_ANOMALY` is set, driven by `.github/workflows/weekly_load_anomaly.yml`), the Redis chaos test (`test_redis_chaos_e2e.py`, locust load against mock deployments split round robin over `/chat/completions` and `/v1/messages`, one endpoint per simulated user, with `CLIENT PAUSE ALL` on the proxy's Redis mid-run to simulate it being down outright, asserting zero failed requests on every endpoint, budgeting RSS and CPU-per-request as ratios against the same run's healthy phase, and holding p50/p90/p99 latency and log-bytes-per-request to flat ceilings (a ratio cannot bound those two: an open breaker skips Redis instead of waiting on it, so the chaos phase can measure cheaper than baseline while still being far slower than a user should see), needs a proxy booted from `gateway/redis_chaos_ci_config.yml` on the same host with `E2E_PROXY_PID` and `E2E_PROXY_LOG` set, marked `redis_chaos`, deselected unless `E2E_PERF_TESTS` is set and excluded from the per-PR selector like the rest of `perf/`, driven by `.github/workflows/test-e2e-redis-chaos.yml` and by the Buildkite `e2e-redis-chaos` step in project-releaser, which runs the proxy, Postgres and Valkey co-located with pytest in one pod and sets the opt-in). New perf tests go here too, marked `perf`, behind the same `E2E_PERF_TESTS` opt-in. The perf cases (`test_perf_cases.py`, one `PerformanceTestCase` per module in `perf/cases/`, no `_e2e` suffix) are the request-performance runs. A case names what the gateway was deployed with (model groups and guardrails, written out in the gateway's config: the `proxy_config` block of project-releaser's perf helm values in the lane, `gateway/perf_ci_config.yml` locally, with each deployment's mock knobs in an `x-mock-knobs` extra header so the model ids are real ones like `gpt-6-luna` and `anthropic.claude-sonnet-5`) and lists what the test creates and sends: teams, Keycloak JWT users, virtual keys, `RequestShape`s (a coding-agent session from `perf/setup/corpus/` whose input-token size is drawn per request from the shape's range, with or without tools, and the output cap), a `Traffic` plan (seed, rps, request count, endpoint, shape and stream shares, or a `Ramp` that steps the rate up through the run, built with `ramped()`, whose plan carries each request's send second and whose run reports `sustained_rps`, the last step the pod held, scored in `perf/setup/ramp.py`, with every step's latency, completed rate, non-200s, CPU per request, Redis latency and Postgres connections cut from the samplers' time-stamped `Series` (`perf/setup/series.py`) to the step's window and emitted as `perf_ramp_step_<rps>_<reading>` JUnit properties for the dashboard) and `Limit`s (one metric at one percentile under one ceiling, or over one floor for `sustained_rps`). The infra side lives in `perf/setup/`: `provision.py` checks the deployed config matches the case and builds the callers, `plan.py` draws the request list from the seed, `run.py` replays it from a separate Locust process (`locustfile.py`) that builds each body right before sending and records status, header, first-byte, first-token and total time per request, and `observe.py` joins all of that with what the mocks received and the workers' RSS to score every `Limit`. Limits are about what the caller sees and what the gateway costs: every non-blocked request is a 200 (`unexpected_status_count`), the seconds the gateway added per request at p50 and p99 (`gateway_added_seconds`, to the first token on streams), the client's time to the first token of a stream at p50 and p99 (`time_to_first_token_seconds`, mock delay included), worker memory growth (`memory_growth_megabytes`), and the Postgres statements and Redis commands a request waits on before it is forwarded (`postgres_calls_per_request`, `redis_calls_per_request`: auth and routing reads, the rate-limiter script), with the work the gateway does after the response or on a timer (spend writes, daily aggregates, config reloads) reported separately as `postgres_background_calls_per_request` and `redis_background_calls_per_request`; the same overhead as the gateway itself reports it, read from the `x-litellm-overhead-duration-ms` header per request (`litellm_overhead_seconds`, gated next to `gateway_added_seconds`, the outside reading of the same thing, so the two can be charted together) and the gateway's own end-to-end time from `x-litellm-response-duration-ms` (`litellm_response_seconds`), the run mean of the gateway's `litellm_redis_latency` histogram from a `/metrics` scrape before and after the run (`telemetry.py`, `prometheus_redis_latency_mean_seconds`), and the database's own view of the run from the same `pg_stat_statements` rows and a once-a-second `pg_stat_activity` sample: execution seconds per wall second (`postgres_active_sessions`, the average-active-sessions number RDS reports as DB load), the slowest statement's mean (`postgres_slowest_statement_seconds`, named in the detail) and the peak client connection count (`postgres_peak_connections`), each under a ceiling set from the first local runs and due a retune once the lane has its ten-run baseline; `datastore.py` reads `pg_stat_statements` before and after the run and holds a Redis `MONITOR` connection during it, sorts each statement by verb and table and each command by key, and the run output lists the top statements and commands per bucket so a red run names what grew. Whether a body, a guardrail call or a spend row is right is printed as a diagnostic, never a limit, that is the functional suites' job. The providers and the guardrail are out-of-process servers in `perf/setup/mocks/` (`server.py openai|anthropic|bedrock|guardrail`, each answering with a reply generated for the request in that provider's wire format, sized from the prompt and output cap, a tool call when tools were offered, waiting the case's `mock_latency` first, request logs at `/mock/requests` and `/mock/calls`), reached through `E2E_PERF_MOCK_*_URL` by both the gateway config and the test. Adding a perf test is adding a module with a `CASE` to `perf/cases/`; a new deployment or guardrail is a config change; a new metric is a name in `Metric` plus a branch in `observe.observe`- `other/` - the holding-pen suite for the `other.*` registry cluster with no home of its own yet: the master-key auth gate, JWT auth (access tokens issued by a real Keycloak realm, `idp.py` plus `idp_realm.json`, whose JWKS the proxy's `JWT_PUBLIC_KEY_URL` points at, see CONTRIBUTING.md for the start command and config block), and the process-lifecycle health probes (liveness, public readiness, authenticated readiness diagnostics). Promote a cluster out once it is large/stable enough for its own suite- `secret_manager/` - the gateway's `key_management_system` against a real secret manager: deployment keys resolved from it (`os.environ/<name>` where the name exists only in the manager) and virtual keys written to and deleted from it. The tests are backend-agnostic and each backend is its own lane, because the setting is global to the proxy: `E2E_SECRET_MANAGER=<system>` opts in and picks the backend from `secret_backends.BACKENDS`, the proxy is booted from `gateway/secret_manager_<system>_ci_config.yml` against the live manager, and the tests reach that manager through the backend's `SecretStore` (`secret_store_<system>.py`). A test needing something not every backend does carries `requires_capability(...)` and is deselected on lanes that lack it. `secret_manager/backend.sh up <system>` runs a backend in Docker and writes the proxy's and the tests' env. Marked `secret_manager`, deselected unless `E2E_SECRET_MANAGER` is set, and kept out of the per-PR selector. Backends today: `hashicorp_vault` and `cyberark` (CyberArk Conjur, which cannot delete, so the delete test is Vault-only)- `gateway/` - proxy configuration only (`litellm-config.yml`). No testsdiff --git a/tests/e2e/perf/cases/mixed_size_requests_ramp.py b/tests/e2e/perf/cases/mixed_size_requests_ramp.pynew file mode 100644index 00000000000..d7db545c031--- /dev/null+++ b/tests/e2e/perf/cases/mixed_size_requests_ramp.py@@ -0,0 +1,24 @@+"""The mixed-size traffic stepped up until the pod stops keeping up: the same callers, guardrails, mocks and body+mix (short chats, 300k to 1M-token contexts, tool calls, half streamed) as `mixed_size_requests_2rps`, 2 rps more+every 90 s from 2 to 14 on the same one 4-worker pod. Where the 2 rps case prices one big body on its own, this one+reports `sustained_rps`, the last step the pod held with big bodies overlapping in every worker, so a handler that+re-walks a 1M-token body on the event loop or a worker that holds several of them in memory at once moves the knee+down and fails the floor."""++from __future__ import annotations++from dataclasses import replace+from typing import Final++from setup.performance_test_case import Limit, Ramp, ramped++from cases.mixed_size_requests_2rps import CASE as MIXED++RAMP: Final = Ramp(first_rps=2.0, last_rps=14.0, step_rps=2.0, step_seconds=90.0, held_p99_seconds=5.0)++CASE: Final = replace(+ MIXED,+ name="mixed_size_requests_ramp",+ traffic=ramped(MIXED.traffic, RAMP, seed=9, concurrency=300),+ limits=(Limit("sustained_rps", min=8),),+)diff --git a/tests/e2e/perf/cases/small_bodies_ramp.py b/tests/e2e/perf/cases/small_bodies_ramp.pynew file mode 100644index 00000000000..64308168cee--- /dev/null+++ b/tests/e2e/perf/cases/small_bodies_ramp.py@@ -0,0 +1,24 @@+"""The small-bodies traffic stepped up until the pod stops keeping up: the same callers, guardrails, mocks and+2.5k to 4k-token bodies as `small_bodies_5rps`, 10 rps more every 90 s from 10 to 70 on the same one 4-worker pod.+The 5 rps case prices one request with the pod idle; this one reports `sustained_rps`, the last step the pod+held (gateway added p99 under 5 s, the queue still draining), so a change that moves the knee down, a lock held+across an await, a pool that fills, a flush that stalls the loop while dozens of requests are in flight, fails the+floor instead of hiding under a fixed rate the pod still manages."""++from __future__ import annotations++from dataclasses import replace+from typing import Final++from setup.performance_test_case import Limit, Ramp, ramped++from cases.small_bodies_5rps import CASE as SMALL++RAMP: Final = Ramp(first_rps=10.0, last_rps=70.0, step_rps=10.0, step_seconds=90.0, held_p99_seconds=5.0)++CASE: Final = replace(+ SMALL,+ name="small_bodies_ramp",+ traffic=ramped(SMALL.traffic, RAMP, seed=13, concurrency=400),+ limits=(Limit("sustained_rps", min=20),),+)diff --git a/tests/e2e/perf/setup/datastore.py b/tests/e2e/perf/setup/datastore.pyindex 89739b6d13e..6ce37da3554 100644--- a/tests/e2e/perf/setup/datastore.py+++ b/tests/e2e/perf/setup/datastore.py@@ -22,6 +22,8 @@ from typing import TYPE_CHECKING, Final, Literalfrom pydantic import TypeAdapter+from setup.series import Recorder, Series+if TYPE_CHECKING:import redis@@ -182,13 +184,22 @@ def slowest_statement(statements: tuple[StatementCalls, ...] | None) -> Statemenclass ConnectionsSampler:"""`pg_stat_activity` client backends on the gateway's database once a second on a background thread, minus- this sampler's own connection, so the peak is what the gateway (and the test's own reads) held open."""+ this sampler's own connection, so the peak is what the gateway (and the test's own reads) held open.+ `series` keeps every count with its time for a ramped run's per-step peaks."""def __init__(self, url: str) -> None:self._url: Final = urlself._stop: Final = threading.Event()self._thread: Final = threading.Thread(target=self._run, name="postgres-connections", daemon=True)- self.peak: int | None = None+ self._counts: Final[Recorder[int]] = Recorder()++ @property+ def peak(self) -> int | None:+ return max((sample.value for sample in self._counts.series.samples), default=None)++ @property+ def series(self) -> Series[int]:+ return self._counts.seriesdef start(self) -> None:self._thread.start()@@ -203,12 +214,13 @@ class ConnectionsSampler:with psycopg.connect(self._url, autocommit=True) as conn:while not self._stop.is_set():(count,) = _COMMITS_ROW.validate_python(conn.execute(_CONNECTIONS_QUERY).fetchone())- self.peak = count if self.peak is None else max(self.peak, count)+ self._counts.record(count)_ = self._stop.wait(_CONNECTIONS_SAMPLE_SECONDS)class NoConnectionsSampler:peak: None = None+ series: None = Nonedef start(self) -> None:return Nonediff --git a/tests/e2e/perf/setup/locustfile.py b/tests/e2e/perf/setup/locustfile.pyindex 1c38eae22a1..82a5749becf 100644--- a/tests/e2e/perf/setup/locustfile.py+++ b/tests/e2e/perf/setup/locustfile.py@@ -4,7 +4,8 @@ time to headers, time to first byte, time to first token (streams), total time,and whether it matched. Runs in its own process (`setup/run.py` starts it), never in the gateway's.Reads `PERF_PLAN_FILE` (JSON list of wire requests), writes `PERF_RECORDS_FILE` (one JSON object per-line), paces to `PERF_RPS` over `PERF_USERS` users, and quits once the last request is recorded.+line), paces to `PERF_RPS` over `PERF_USERS` users (or, when the plan carries `send_at_s`, sends each+request at its scheduled second), and quits once the last request is recorded."""from __future__ import annotations@@ -35,6 +36,8 @@ _JITTER: Final = 0.25_RNG: Final = random.Random(_PLAN[0].marker if _PLAN else "empty-plan")_RECORDS: Final = _RECORDS_PATH.open("a", encoding="utf-8")_PROGRESS: Final = PlanProgress(len(_PLAN))+_SCHEDULED: Final = bool(_PLAN) and _PLAN[0].send_at_s is not None+_RUN_STARTED: Final = time.monotonic()_STOP_FILE: Final = os.environ.get("PERF_STOP_FILE")BLOCK_REASON: Final = "perf-test-block"_TOKEN_MAX_AGE_S: Final = 240.0@@ -211,7 +214,8 @@ def _authorization(user: FastHttpUser, item: WireRequest) -> str:class PlanUser(FastHttpUser):"""One simulated client; all users share the plan and together hold the planned rate. Each user starts ata random point of its first interval and jitters every later interval by up to `_JITTER` either way, so the- users never fire in lockstep; the mean interval is unchanged and `rps` stays the total rate."""+ users never fire in lockstep; the mean interval is unchanged and `rps` stays the total rate. A scheduled+ plan (a ramp) is paced by the plan instead: a user takes the next request and sleeps until its second."""network_timeout = float(os.environ.get("PERF_REQUEST_TIMEOUT", "120"))connection_timeout = 30.0@@ -219,9 +223,12 @@ class PlanUser(FastHttpUser):def on_start(self) -> None:self._last_run = 0.0self._last_wait = 0.0- gevent.sleep(_RNG.uniform(0.0, _INTERVAL_S))+ if not _SCHEDULED:+ gevent.sleep(_RNG.uniform(0.0, _INTERVAL_S))def wait_time(self) -> float:+ if _SCHEDULED:+ return 0.1target: Final = _INTERVAL_S * _RNG.uniform(1.0 - _JITTER, 1.0 + _JITTER)now: Final = time.monotonic()run_s: Final = now - self._last_run - self._last_wait if self._last_run else 0.0@@ -237,6 +244,8 @@ class PlanUser(FastHttpUser):if _PROGRESS.drained:_quit(self)return+ if item.send_at_s is not None:+ gevent.sleep(max(0.0, _RUN_STARTED + item.send_at_s - time.monotonic()))started_at: Final = time.time()started: Final = time.perf_counter()first_byte_s: float | None = Nonediff --git a/tests/e2e/perf/setup/observe.py b/tests/e2e/perf/setup/observe.pyindex 1532a78606e..dccc6e272b3 100644--- a/tests/e2e/perf/setup/observe.py+++ b/tests/e2e/perf/setup/observe.py@@ -7,14 +7,16 @@ import hashlibimport jsonimport statisticsfrom collections import Counter, defaultdict-from collections.abc import Iterable+from collections.abc import Iterable, Mappingfrom dataclasses import asdict, dataclass-from itertools import chain+from itertools import chain, groupby+from types import MappingProxyTypefrom typing import Final, Literalfrom e2e_http import NoBody, get_external, post_external, unwrapfrom memory_readings import RssCapture, RssReading, WorkerKeyfrom pydantic import BaseModel+from typing_extensions import ReadOnly, TypedDictfrom setup.datastore import (DbCounters,@@ -27,7 +29,9 @@ from setup.datastore import (statement_deltas,)from setup.mocks.reply_common import Provider-from setup.performance_test_case import Limit, Metric, MockLatency, PerformanceTestCase+from setup.performance_test_case import Limit, Metric, MockLatency, PerformanceTestCase, Ramp+from setup.ramp import StepReading, render_steps, sustained_rps+from setup.series import Seriesfrom setup.telemetry import GatewayTelemetryfrom setup.wire import WireRecord@@ -122,6 +126,10 @@ class Observation:telemetry_before: GatewayTelemetry | None = Nonetelemetry_after: GatewayTelemetry | None = Nonepeak_connections: int | None = None+ ramp: Ramp | None = None+ cpu_series: Series[float] | None = None+ connection_series: Series[int] | None = None+ telemetry_series: Series[GatewayTelemetry] | None = None@propertydef telemetry(self) -> GatewayTelemetry | None:@@ -239,6 +247,67 @@ class Observation:post: Final = Counter(call.marker for call in self.guardrail_calls if call.input_type == "response")return sum(1 for record in self.served if pre[record.marker] != 1 or post[record.marker] != 1)+ def steps(self) -> tuple[StepReading, ...]:+ """One reading per step of a ramped run: the gateway's added latency over the step's requests, the rate+ at which requests completed inside the step's window, the step's unexpected statuses, and what the pod+ was doing in that window (CPU per completed request, Redis latency, Postgres connections)."""+ ramp: Final = self.ramp+ if ramp is None or not self.records:+ return ()+ started: Final = min(record.started_at for record in self.records)+ added: Final = _by_step(ramp, self.latency_pairs("gateway_added_seconds"))+ first_token: Final = _by_step(ramp, self.latency_pairs("time_to_first_token_seconds"))+ completed: Final = Counter(+ int((record.started_at + record.total_s - started) // ramp.step_seconds) for record in self.records+ )+ unexpected: Final = Counter(+ ramp.step_of(record.seq) for record in self.records if not record.blocked and record.status != 200+ )+ return tuple(+ StepReading(+ rps=rate,+ planned=count,+ completed_rps=completed[index] / ramp.step_seconds,+ gateway_added_p99=percentile(added.get(index, ()), 99),+ unexpected=unexpected[index],+ held_p99_seconds=ramp.held_p99_seconds,+ gateway_added_p50=percentile(added.get(index, ()), 50),+ time_to_first_token_p99=percentile(first_token.get(index, ()), 99),+ **self._window_readings(started + index * ramp.step_seconds, ramp.step_seconds, completed[index]),+ )+ for index, (rate, count) in enumerate(zip(ramp.rates, ramp.counts, strict=True))+ )++ def _window_readings(self, start: float, seconds: float, completed: int) -> WindowReadings:+ end: Final = start + seconds+ cpu: Final = None if self.cpu_series is None else sum(self.cpu_series.between(start, end))+ conns: Final = None if self.connection_series is None else self.connection_series.between(start, end)+ before: Final = None if self.telemetry_series is None else self.telemetry_series.last_before(start)+ after: Final = None if self.telemetry_series is None else self.telemetry_series.last_before(end)+ return WindowReadings(+ cpu_seconds_per_request=None if cpu is None or completed == 0 else cpu / completed,+ redis_latency_mean_seconds=None+ if before is None or after is None+ else after.minus(before).redis_latency.mean,+ postgres_peak_connections=max(conns, default=None) if conns else None,+ )+++class WindowReadings(TypedDict):+ cpu_seconds_per_request: ReadOnly[float | None]+ redis_latency_mean_seconds: ReadOnly[float | None]+ postgres_peak_connections: ReadOnly[int | None]+++def _by_step(ramp: Ramp, pairs: Iterable[tuple[WireRecord, float]]) -> Mapping[int, tuple[float, ...]]:+ ordered: Final = sorted(pairs, key=lambda pair: pair[0].seq)+ return MappingProxyType(+ {+ step: tuple(value for _, value in group)+ for step, group in groupby(ordered, key=lambda pair: ramp.step_of(pair[0].seq))+ }+ )+def _text_sha1(text: str, *, intervened: bool) -> str:original: Final = text.removesuffix(" [post]") if intervened else text@@ -415,6 +484,8 @@ def observe(if metric == "cpu_seconds_per_request":cpu: Final = observation.gateway_cpu_secondsreturn None if cpu is None or not observation.records else cpu / len(observation.records)+ if metric == "sustained_rps":+ return None if observation.ramp is None else sustained_rps(observation.steps())return worst_worker_growth_mb(observation.rss_before, observation.rss_after)@@ -424,6 +495,7 @@ class Row:percentile: int | Noneobserved: float | Nonelimit: float | None+ floor: float | None = Nonedetail: str = ""breakdown: tuple[Contributor, ...] = ()@@ -437,11 +509,22 @@ class Row:@propertydef over(self) -> bool:- return self.limit is not None and self.observed is not None and self.observed > self.limit+ """Past the limit: above a max, or below a floor."""+ if self.observed is None:+ return False+ return (self.limit is not None and self.observed > self.limit) or (+ self.floor is not None and self.observed < self.floor+ )@propertydef missing(self) -> bool:- return self.limit is not None and self.observed is None+ return (self.limit is not None or self.floor is not None) and self.observed is None++ @property+ def bound(self) -> str:+ if self.limit is not None:+ return f"{self.limit:.4f}"+ return "" if self.floor is None else f">= {self.floor:.4f}"REPORTED: Final[tuple[tuple[Metric, int | None], ...]] = (@@ -476,6 +559,8 @@ def evaluate(case: PerformanceTestCase, observation: Observation) -> tuple[list[(limit.metric, limit.percentile): limit for limit in case.limits}keys: list[tuple[Metric, int | None]] = list(REPORTED)+ if observation.ramp is not None:+ keys.append(("sustained_rps", None))keys.extend(key for key in limits if key not in keys)rows: list[Row] = []for metric, rank in keys:@@ -486,20 +571,23 @@ def evaluate(case: PerformanceTestCase, observation: Observation) -> tuple[list[percentile=rank,observed=observe(metric, rank, observation, case.background_cap_exempt_tables),limit=limit.max if limit else None,+ floor=limit.min if limit else None,detail=_detail(metric, observation, case.background_cap_exempt_tables),breakdown=breakdown(metric, rank, observation, case.background_cap_exempt_tables),))- exceeded: Final = [- f"{row.label}: observed {row.observed:.4f} > limit {row.limit:.4f}{f' ({row.detail})' if row.detail else ''}"- if row.over and row.observed is not None and row.limit is not None- else f"{row.label}: limit set but nothing observed"- for row in rows- if row.over or row.missing- ]+ exceeded: Final = [_breach(row) for row in rows if row.over or row.missing]return rows, exceeded+def _breach(row: Row) -> str:+ if row.observed is None:+ return f"{row.label}: limit set but nothing observed"+ if row.floor is not None:+ return f"{row.label}: observed {row.observed:.4f} < floor {row.floor:.4f}"+ return f"{row.label}: observed {row.observed:.4f} > limit {row.bound}{f' ({row.detail})' if row.detail else ''}"++def _detail(metric: Metric, observation: Observation, exempt_tables: frozenset[str]) -> str:if metric == "postgres_max_statement_per_request":busiest: Final = observation.busiest_awaited_statement@@ -516,12 +604,13 @@ def _detail(metric: Metric, observation: Observation, exempt_tables: frozenset[sdef _row_properties(row: Row) -> tuple[tuple[str, float | str], ...]:observed: Final = ((row.property_name, row.observed),) if row.observed is not None else ()limit: Final = ((f"{row.property_name}_limit", row.limit),) if row.limit is not None else ()+ floor: Final = ((f"{row.property_name}_floor", row.floor),) if row.floor is not None else ()contributors: Final = (((f"{row.property_name}_breakdown", json.dumps([asdict(item) for item in row.breakdown])),)if row.breakdownelse ())- return observed + limit + contributors+ return observed + limit + floor + contributorsdef junit_properties(rows: Iterable[Row]) -> tuple[tuple[str, float | str], ...]:@@ -546,9 +635,17 @@ def render(case: PerformanceTestCase, observation: Observation, rows: list[Row])]for row in rows:observed = "n/a" if row.observed is None else f"{row.observed:.4f}"- limit = "" if row.limit is None else f"{row.limit:.4f}"- flag = " OVER" if row.over else " MISSING" if row.missing else ""- lines.append(f" {row.label:32} {observed:>12} {limit:>12}{flag}")+ flag = (+ " UNDER"+ if row.over and row.floor is not None+ else " OVER"+ if row.over+ else " MISSING"+ if row.missing+ else ""+ )+ lines.append(f" {row.label:32} {observed:>12} {row.bound:>12}{flag}")+ lines.extend(render_steps(observation.steps()))lines.extend(_diagnostics(observation))lines.extend(_datastore_lines(observation))for record in failures:diff --git a/tests/e2e/perf/setup/performance_test_case.py b/tests/e2e/perf/setup/performance_test_case.pyindex 3f39b4c38c8..512ef8b911f 100644--- a/tests/e2e/perf/setup/performance_test_case.py+++ b/tests/e2e/perf/setup/performance_test_case.py@@ -9,8 +9,10 @@ so a case is a literal a reviewer can read top to bottom.from __future__ import annotations+import bisectfrom collections.abc import Mapping-from dataclasses import dataclass, field+from dataclasses import dataclass, field, replace+from itertools import accumulatefrom typing import Final, Literalfrom setup.event_loop import EventLoopStalls@@ -36,6 +38,7 @@ Metric = Literal["postgres_active_sessions","postgres_slowest_statement_seconds","postgres_peak_connections",+ "sustained_rps",]@@ -97,12 +100,50 @@ SHORT: Final = RequestShape("short", input_tokens=(5_000, 100_000), tools=True)LONG: Final = RequestShape("long", input_tokens=(300_000, 1_000_000), tools=True)+@dataclass(frozen=True, slots=True)+class Ramp:+ """Rate stepped up through the run: `first_rps`, then `step_rps` more every `step_seconds`, up to `last_rps`.+ Each step is scored on its own and the run reports `sustained_rps`, the last step the pod held before the+ first one it did not: a step holds when its `gateway_added_seconds` p99 stayed under `held_p99_seconds`, it+ completed at least `HELD_SHARE` of its rate and no more than `HELD_UNEXPECTED_SHARE` of its requests came back+ with an unexpected status (`setup/ramp.py`). Requests are sent on a schedule drawn from the plan's seed, not+ from per-user intervals, so the rate changes on the step boundary."""++ first_rps: float+ last_rps: float+ step_rps: float+ step_seconds: float+ held_p99_seconds: float++ @property+ def rates(self) -> tuple[float, ...]:+ count: Final = round((self.last_rps - self.first_rps) / self.step_rps) + 1+ return tuple(self.first_rps + index * self.step_rps for index in range(count))++ @property+ def counts(self) -> tuple[int, ...]:+ return tuple(round(rate * self.step_seconds) for rate in self.rates)++ @property+ def total_requests(self) -> int:+ return sum(self.counts)++ @property+ def seconds(self) -> float:+ return len(self.rates) * self.step_seconds++ def step_of(self, seq: int) -> int:+ """Zero-based step of the request with 1-based plan number `seq`."""+ return bisect.bisect_left(tuple(accumulate(self.counts)), seq)++@dataclass(frozen=True, slots=True)class Traffic:"""The plan: `total_requests` drawn from `seed`, sent at `rps` by `concurrency` simulated- users. Shares are fractions of the requests: `endpoints` and `shapes` must each sum to 1,- `intervene_share` picks non-streaming requests the guardrail rewrites, `block_share` picks- requests it blocks."""+ users, or stepped up through a `ramp` (then `rps` is its first step and `total_requests` its+ total, see `ramped`). Shares are fractions of the requests: `endpoints` and `shapes` must each+ sum to 1, `intervene_share` picks non-streaming requests the guardrail rewrites, `block_share`+ picks requests it blocks."""seed: intrps: float@@ -113,20 +154,33 @@ class Traffic:intervene_share: float = 0.0block_share: float = 0.0concurrency: int = 20+ ramp: Ramp | None = None@propertydef expected_seconds(self) -> float:- return self.total_requests / self.rps+ return self.total_requests / self.rps if self.ramp is None else self.ramp.seconds+++def ramped(base: Traffic, ramp: Ramp, *, seed: int, concurrency: int) -> Traffic:+ """`base`'s request mix sent along `ramp` instead of at one rate. `concurrency` caps the requests in+ flight, so size it past the deepest step the ramp is expected to reach."""+ return replace(+ base, seed=seed, rps=ramp.first_rps, total_requests=ramp.total_requests, concurrency=concurrency, ramp=ramp+ )@dataclass(frozen=True, slots=True)class Limit:- """One pass/fail rule: `metric` (at `percentile`, or the whole-run value) must not exceed `max`.- Metrics without a Limit are still reported."""+ """One pass/fail rule: `metric` (at `percentile`, or the whole-run value) must not exceed `max`, or must+ reach `min` for a floor such as `sustained_rps`. Metrics without a Limit are still reported."""metric: Metric- max: float+ max: float | None = Nonepercentile: int | None = None+ min: float | None = None++ def __post_init__(self) -> None:+ assert (self.max is None) != (self.min is None), f"{self.metric}: a Limit is either a max or a min"@propertydef label(self) -> str:@@ -201,6 +255,14 @@ class PerformanceTestCase:assert 0 <= self.traffic.stream_share <= 1, f"{self.name}: stream_share out of range"assert self.traffic.intervene_share + self.traffic.block_share <= 1, f"{self.name}: guardrail shares exceed 1"assert self.traffic.total_requests > 0 and self.traffic.rps > 0, f"{self.name}: empty traffic"+ ramp: Final = self.traffic.ramp+ assert ramp is None or (ramp.first_rps, ramp.total_requests) == (+ self.traffic.rps,+ self.traffic.total_requests,+ ), f"{self.name}: traffic rps and total_requests must be the ramp's, build it with ramped()"+ assert ramp is None or (0 < ramp.first_rps <= ramp.last_rps and ramp.step_rps > 0 and ramp.step_seconds > 0), (+ f"{self.name}: ramp {ramp} does not step up"+ )assert self.model_groups, f"{self.name}: no model groups"assert 0 <= self.mock_latency.jitter_seconds <= self.mock_latency.seconds or self.mock_latency.seconds == 0, (f"{self.name}: mock latency jitter {self.mock_latency.jitter_seconds} s exceeds {self.mock_latency.seconds} s"diff --git a/tests/e2e/perf/setup/plan.py b/tests/e2e/perf/setup/plan.pyindex 6b9a5a19662..4f28736a8e9 100644--- a/tests/e2e/perf/setup/plan.py+++ b/tests/e2e/perf/setup/plan.py@@ -12,6 +12,7 @@ from dataclasses import dataclassfrom typing import Final, Literal, TypeVarfrom setup.performance_test_case import Endpoint, PerformanceTestCase, RequestShape, Team+from setup.ramp import send_offsetsFlag = Literal["n", "i", "b"]CallerKind = Literal["jwt", "key"]@@ -39,6 +40,7 @@ class PlannedRequest:flag: Flagshape: RequestShapeinput_tokens: int+ send_at_s: float | None = None@propertydef blocked(self) -> bool:@@ -98,6 +100,7 @@ def build_plan(case: PerformanceTestCase) -> tuple[PlannedRequest, ...]:case.validate()rng: Final = random.Random(case.traffic.seed)callers: Final = _callers_by_group(case)+ offsets: Final = None if case.traffic.ramp is None else send_offsets(case.traffic.ramp, case.traffic.seed)planned: list[PlannedRequest] = []for seq in range(1, case.traffic.total_requests + 1):endpoint = _pick(rng, case.traffic.endpoints)@@ -117,6 +120,7 @@ def build_plan(case: PerformanceTestCase) -> tuple[PlannedRequest, ...]:flag=flag,shape=shape,input_tokens=rng.randint(low, high),+ send_at_s=None if offsets is None else offsets[seq - 1],))return tuple(planned)diff --git a/tests/e2e/perf/setup/podcpu.py b/tests/e2e/perf/setup/podcpu.pyindex 400e5b7d38b..865fa14efe0 100644--- a/tests/e2e/perf/setup/podcpu.py+++ b/tests/e2e/perf/setup/podcpu.py@@ -19,6 +19,8 @@ from urllib.parse import quotefrom e2e_http import AuthHeaders, Success, get_externalfrom pydantic import BaseModel+from setup.series import Recorder, Series+SELECTOR_ENV: Final = "E2E_PERF_GATEWAY_POD_SELECTOR"_SERVICE_ACCOUNT: Final = Path("/var/run/secrets/kubernetes.io/serviceaccount")_INTERVAL_S: Final = 5.0@@ -49,7 +51,8 @@ def cores(quantity: str) -> float:class GatewayCpuSampler:- """Integrates the matching pods' core use over the time it is running; `seconds` is CPU seconds."""+ """Integrates the matching pods' core use over the time it is running; `seconds` is CPU seconds and+ `series` the CPU seconds of each sampling interval, stamped when the interval ended."""def __init__(self, selector: str, namespace: str, token: str, ca_bundle: str) -> None:self._url: Final = (@@ -60,12 +63,15 @@ class GatewayCpuSampler:self._ca_bundle: Final = ca_bundleself._stop: Final = threading.Event()self._thread: Final = threading.Thread(target=self._run, name="gateway-cpu-sampler", daemon=True)- self._cpu_seconds = 0.0- self._samples = 0+ self._intervals: Final[Recorder[float]] = Recorder()@propertydef seconds(self) -> float | None:- return self._cpu_seconds if self._samples > 0 else None+ return sum(sample.value for sample in self._intervals.series.samples) if len(self._intervals) else None++ @property+ def series(self) -> Series[float]:+ return self._intervals.seriesdef _read_cores(self) -> float | None:result: Final = get_external(@@ -81,8 +87,7 @@ class GatewayCpuSampler:now: float = time.monotonic()current: float | None = self._read_cores()if current is not None:- self._cpu_seconds += current * (now - last)- self._samples += 1+ self._intervals.record(current * (now - last))last = now # rebind-ok: the sampling clock advances every tickdef start(self) -> None:@@ -95,6 +100,7 @@ class GatewayCpuSampler:class NoSampler:seconds: Final[None] = None+ series: Final[None] = Nonedef start(self) -> None:return Nonediff --git a/tests/e2e/perf/setup/ramp.py b/tests/e2e/perf/setup/ramp.pynew file mode 100644index 00000000000..217e140d470--- /dev/null+++ b/tests/e2e/perf/setup/ramp.py@@ -0,0 +1,116 @@+"""A ramped run step by step: the send schedule its plan carries (`send_offsets`), one `StepReading` per step+and `sustained_rps`, the last step the pod held before the first it did not. `Ramp` itself sits in+`performance_test_case.py` with the rest of the case schema."""++from __future__ import annotations++import random+from collections.abc import Iterable, Iterator+from dataclasses import dataclass+from itertools import chain, takewhile+from typing import Final++from setup.performance_test_case import Ramp++HELD_SHARE: Final = 0.9+"""A step that completed fewer than this share of its rate is a queue that stopped draining."""+HELD_UNEXPECTED_SHARE: Final = 0.01+"""A step with more than this share of unexpected statuses broke, whatever its latency."""+_JITTER: Final = 0.25+++def _step_offsets(ramp: Ramp, index: int, count: int, rng: random.Random) -> Iterator[float]:+ for slot in range(count):+ yield (index + (slot + 0.5 + rng.uniform(-_JITTER, _JITTER)) / count) * ramp.step_seconds+++def send_offsets(ramp: Ramp, seed: int) -> tuple[float, ...]:+ """Seconds after the run starts to send each request of the plan, in plan order: a step's requests spread+ evenly over its window, each moved by up to a quarter of a slot either way, drawn from `seed` so a rerun+ sends at the same seconds."""+ rng: Final = random.Random(f"ramp-{seed}")+ return tuple(chain.from_iterable(_step_offsets(ramp, index, count, rng) for index, count in enumerate(ramp.counts)))+++@dataclass(frozen=True, slots=True)+class StepReading:+ rps: float+ planned: int+ completed_rps: float+ gateway_added_p99: float | None+ unexpected: int+ held_p99_seconds: float+ gateway_added_p50: float | None = None+ time_to_first_token_p99: float | None = None+ cpu_seconds_per_request: float | None = None+ redis_latency_mean_seconds: float | None = None+ postgres_peak_connections: int | None = None+ """The hold verdict reads the first six. The rest say what the pod was doing on that step: CPU per request+ from the pod sampler, Redis latency from the gateway's `/metrics`, connections from `pg_stat_activity`,+ all cut to the step's window, so a latency jump can be told apart as CPU, waiting or neither."""++ @property+ def held(self) -> bool:+ return (+ self.gateway_added_p99 is not None+ and self.gateway_added_p99 <= self.held_p99_seconds+ and self.completed_rps >= HELD_SHARE * self.rps+ and self.unexpected <= HELD_UNEXPECTED_SHARE * self.planned+ )+++def sustained_rps(steps: Iterable[StepReading]) -> float:+ """The rate of the last step that held before the first that did not, 0 when the first step did not hold."""+ held: Final = tuple(takewhile(lambda step: step.held, steps))+ return held[-1].rps if held else 0.0+++def step_properties(steps: Iterable[StepReading]) -> tuple[tuple[str, float], ...]:+ """The per-step readings as JUnit properties (`perf_ramp_step_<rps>_<reading>`, every reading the step has)+ plus the hold bound, so the dashboard can draw the run's curves next to the sustained rate."""+ rows: Final = tuple(steps)+ hold: Final = (("perf_ramp_hold_p99_seconds", rows[0].held_p99_seconds),) if rows else ()+ return (*hold, *chain.from_iterable(_step_properties(step) for step in rows))+++def _step_properties(step: StepReading) -> Iterator[tuple[str, float]]:+ prefix: Final = f"perf_ramp_step_{step.rps:g}"+ readings: Final[tuple[tuple[str, float | None], ...]] = (+ ("gateway_added_p99", step.gateway_added_p99),+ ("completed_rps", step.completed_rps),+ ("unexpected", float(step.unexpected)),+ ("gateway_added_p50", step.gateway_added_p50),+ ("time_to_first_token_p99", step.time_to_first_token_p99),+ ("cpu_seconds_per_request", step.cpu_seconds_per_request),+ ("redis_latency_mean_seconds", step.redis_latency_mean_seconds),+ (+ "postgres_peak_connections",+ None if step.postgres_peak_connections is None else float(step.postgres_peak_connections),+ ),+ )+ return ((f"{prefix}_{name}", value) for name, value in readings if value is not None)+++def render_steps(steps: Iterable[StepReading]) -> tuple[str, ...]:+ rows: Final = tuple(steps)+ if not rows:+ return ()+ header: Final = (+ f" steps: {'rps':>6} {'planned':>8} {'completed rps':>14} {'gateway added p50':>18} {'p99':>8}"+ f" {'ttft p99':>9} {'unexpected':>11} {'cpu s/req':>10} {'redis mean':>11} {'pg conns':>9}"+ )+ return (header, *(_step_line(step) for step in rows))+++def _seconds(value: float | None) -> str:+ return "n/a" if value is None else f"{value:.4f}"+++def _step_line(step: StepReading) -> str:+ conns: Final = "n/a" if step.postgres_peak_connections is None else str(step.postgres_peak_connections)+ return (+ f" {step.rps:6.1f} {step.planned:8d} {step.completed_rps:14.2f} {_seconds(step.gateway_added_p50):>18}"+ f" {_seconds(step.gateway_added_p99):>8} {_seconds(step.time_to_first_token_p99):>9} {step.unexpected:11d}"+ f" {_seconds(step.cpu_seconds_per_request):>10} {_seconds(step.redis_latency_mean_seconds):>11} {conns:>9}"+ f" {'held' if step.held else 'broke'}"+ )diff --git a/tests/e2e/perf/setup/run.py b/tests/e2e/perf/setup/run.pyindex 272c64754c4..ec2927e90af 100644--- a/tests/e2e/perf/setup/run.py+++ b/tests/e2e/perf/setup/run.py@@ -63,6 +63,7 @@ def wire_request(item: PlannedRequest, setup: CaseSetup) -> WireRequest:token_source=setup.token_source(item.caller),stream=item.stream,blocked=item.blocked,+ send_at_s=item.send_at_s,)diff --git a/tests/e2e/perf/setup/series.py b/tests/e2e/perf/setup/series.pynew file mode 100644index 00000000000..78e5cf272e1--- /dev/null+++ b/tests/e2e/perf/setup/series.py@@ -0,0 +1,48 @@+"""Readings a background sampler takes while the traffic runs, each stamped with the wall clock+(`time.time()`, the clock the Locust records carry in `started_at`), so a ramped run can cut them into its+steps' windows after the fact."""++from __future__ import annotations++import time+from dataclasses import dataclass+from typing import Final, Generic, TypeVar++T = TypeVar("T")+++@dataclass(frozen=True, slots=True)+class Sample(Generic[T]):+ at: float+ value: T+++@dataclass(frozen=True, slots=True)+class Series(Generic[T]):+ samples: tuple[Sample[T], ...]++ def between(self, start: float, end: float) -> tuple[T, ...]:+ """The values stamped inside `[start, end)`, in time order."""+ return tuple(sample.value for sample in self.samples if start <= sample.at < end)++ def last_before(self, at: float) -> T | None:+ """The newest value stamped before `at`, None when the series starts later."""+ earlier: Final = tuple(sample.value for sample in self.samples if sample.at < at)+ return earlier[-1] if earlier else None+++class Recorder(Generic[T]):+ """What a sampler thread appends to; `series` is a frozen copy for the observation."""++ def __init__(self) -> None:+ self._samples: list[Sample[T]] = []++ def record(self, value: T) -> None:+ self._samples.append(Sample(at=time.time(), value=value))++ @property+ def series(self) -> Series[T]:+ return Series(samples=tuple(self._samples))++ def __len__(self) -> int:+ return len(self._samples)diff --git a/tests/e2e/perf/setup/telemetry.py b/tests/e2e/perf/setup/telemetry.pyindex e52bcabafda..185cc788601 100644--- a/tests/e2e/perf/setup/telemetry.py+++ b/tests/e2e/perf/setup/telemetry.py@@ -1,5 +1,5 @@-"""The gateway's own `/metrics`, read before and after a run for the numbers nothing outside the gateway-can see. Today that is `litellm_redis_latency`, the histogram customers' Grafana panels show when Redis+"""The gateway's own `/metrics`, read before and after a run (and every few seconds during a ramped run, see+`telemetry_sampler`) for the numbers nothing outside the gateway can see. Today that is `litellm_redis_latency`, the histogram customers' Grafana panels show when Redisslows the gateway down. The gateway's latency and request counts are read per request from its responseheaders instead (`setup/wire.py`), which gives percentiles where a histogram delta only gives a mean.@@ -10,7 +10,9 @@ on psutil (`proxy_usage.py`)."""from __future__ import annotationsimport math-from collections.abc import Iterable+import threading+from collections.abc import Generator, Iterable+from contextlib import contextmanagerfrom dataclasses import dataclassfrom itertools import chainfrom typing import Final@@ -20,6 +22,10 @@ from prometheus_client.parser import text_string_to_metric_familiesfrom prometheus_client.samples import Samplefrom proxy_client import ProxyClient+from setup.series import Recorder, Series++_SAMPLE_SECONDS: Final = 10.0+@dataclass(frozen=True, slots=True)class HistogramTotals:@@ -63,3 +69,42 @@ def read_gateway_telemetry(proxy: ProxyClient) -> GatewayTelemetry | None:case every limit on a gateway-reported metric fails as missing rather than reading a silent zero."""scrape: Final = proxy.probe("/metrics", params=NoBody())return parse_gateway_telemetry(scrape.body) if scrape.status_code == 200 else None+++class TelemetrySampler:+ """`/metrics` every `_SAMPLE_SECONDS` on a background thread while traffic runs, so a ramped run can read+ the Redis latency of each step's window as the difference between the scrapes around it."""++ def __init__(self, proxy: ProxyClient) -> None:+ self._proxy: Final = proxy+ self._stop: Final = threading.Event()+ self._thread: Final = threading.Thread(target=self._run, name="gateway-telemetry", daemon=True)+ self._scrapes: Final[Recorder[GatewayTelemetry]] = Recorder()++ @property+ def series(self) -> Series[GatewayTelemetry]:+ return self._scrapes.series++ def _run(self) -> None:+ while not self._stop.is_set():+ scrape: GatewayTelemetry | None = read_gateway_telemetry(self._proxy)+ if scrape is not None:+ self._scrapes.record(scrape)+ _ = self._stop.wait(_SAMPLE_SECONDS)++ def start(self) -> None:+ self._thread.start()++ def stop(self) -> None:+ self._stop.set()+ self._thread.join(timeout=30)+++@contextmanager+def telemetry_sampler(proxy: ProxyClient) -> Generator[TelemetrySampler]:+ sampler: Final = TelemetrySampler(proxy)+ sampler.start()+ try:+ yield sampler+ finally:+ sampler.stop()diff --git a/tests/e2e/perf/setup/upgrade.py b/tests/e2e/perf/setup/upgrade.pyindex 36efc16e1fe..c9114ccbff2 100644--- a/tests/e2e/perf/setup/upgrade.py+++ b/tests/e2e/perf/setup/upgrade.py@@ -145,7 +145,9 @@ class UpgradeMonitor:self.clock: Final = clockself.wall_clock: Final = wall_clockself.started: Final = clock()- self.status_limit: Final = next(limit.max for limit in CASE.limits if limit.metric == "unexpected_status_count")+ self.status_limit: Final = next(+ limit.max for limit in CASE.limits if limit.metric == "unexpected_status_count" and limit.max is not None+ )self.pending: tuple[WireRecord, ...] = ()self.processed = 0self.successes = 0diff --git a/tests/e2e/perf/setup/wire.py b/tests/e2e/perf/setup/wire.pyindex 18351c18c6b..a1833fbd44b 100644--- a/tests/e2e/perf/setup/wire.py+++ b/tests/e2e/perf/setup/wire.py@@ -51,6 +51,8 @@ class WireRequest(BaseModel):token_source: TokenSource | None = Nonestream: boolblocked: bool+ send_at_s: float | None = None+ """Seconds after the run starts to send this request, set when the case ramps; otherwise the users pace."""class WireRecord(BaseModel):diff --git a/tests/e2e/perf/test_perf_cases.py b/tests/e2e/perf/test_perf_cases.pyindex 3e5c3c522c8..db657a04b00 100644--- a/tests/e2e/perf/test_perf_cases.py+++ b/tests/e2e/perf/test_perf_cases.py@@ -44,8 +44,9 @@ from setup.performance_test_case import MockLatency, PerformanceTestCasefrom setup.plan import PlannedRequest, build_planfrom setup.podcpu import gateway_cpu_samplerfrom setup.provision import MOCK_PROVIDERS, build_setup, guardrail_url, mock_url+from setup.ramp import step_propertiesfrom setup.run import run_plan-from setup.telemetry import read_gateway_telemetry+from setup.telemetry import read_gateway_telemetry, telemetry_samplerpytestmark = [pytest.mark.perf, pytest.mark.quiet_stack]_SPEND_SAMPLE: Final = 40@@ -115,7 +116,12 @@ def test_perf_case(redis_before: Final = read_redis_counters()run_started: Final = time.monotonic()traffic_started_at: Final = datetime.now(tz=UTC)- with gateway_cpu_sampler() as cpu, redis_monitor() as monitor, connections_sampler() as connections:+ with (+ gateway_cpu_sampler() as cpu,+ redis_monitor() as monitor,+ connections_sampler() as connections,+ telemetry_sampler(proxy) as telemetry,+ ):outcome: Final = run_plan(case, plan, setup)run_finished: Final = time.monotonic()db_after: Final = read_db_counters()@@ -151,6 +157,10 @@ def test_perf_case(telemetry_before=telemetry_before,telemetry_after=telemetry_after,peak_connections=connections.peak,+ ramp=case.traffic.ramp,+ cpu_series=cpu.series,+ connection_series=connections.series,+ telemetry_series=telemetry.series,)rows, exceeded_limits = evaluate(case, observation)stall_rows: Final = _event_loop_rows(case, traffic_started_at) if case.event_loop is not None else ()@@ -162,6 +172,7 @@ def test_perf_case(item.user_properties.extend(stall_properties(case.event_loop, stall_rows) if case.event_loop is not None else junit_properties(rows))+ item.user_properties.extend(step_properties(observation.steps()))print(render(case, observation, rows))if case.event_loop is not None:print(render_stalls(case.event_loop, stall_rows))diff --git a/tests/e2e_harness/perf/setup/test_ramp.py b/tests/e2e_harness/perf/setup/test_ramp.pynew file mode 100644index 00000000000..6d6c1640b59--- /dev/null+++ b/tests/e2e_harness/perf/setup/test_ramp.py@@ -0,0 +1,168 @@+from __future__ import annotations++import math+from dataclasses import replace+from typing import Final++from cases.small_bodies_5rps import CASE+from memory_readings import RssCapture+from setup.observe import Observation, evaluate, junit_properties+from setup.performance_test_case import Limit, Ramp, ramped+from setup.plan import build_plan+from setup.ramp import StepReading, send_offsets, step_properties, sustained_rps+from setup.series import Sample, Series+from setup.telemetry import GatewayTelemetry, HistogramTotals+from setup.wire import WireRecord++NO_RSS: Final = RssCapture(readings=(), failures=())+++def test_send_offsets_put_each_step_s_requests_inside_its_window_in_plan_order() -> None:+ ramp: Final = Ramp(first_rps=10.0, last_rps=30.0, step_rps=10.0, step_seconds=10.0, held_p99_seconds=5.0)+ offsets: Final = send_offsets(ramp, seed=13)+ assert ramp.counts == (100, 200, 300)+ assert len(offsets) == ramp.total_requests+ assert offsets == tuple(sorted(offsets))+ for seq, offset in enumerate(offsets, start=1):+ step = ramp.step_of(seq)+ assert step * 10.0 <= offset < (step + 1) * 10.0, f"request {seq} at {offset:.2f}s is outside step {step}"+ assert send_offsets(ramp, seed=13) == offsets+ assert send_offsets(ramp, seed=14) != offsets+++def test_build_plan_carries_the_schedule_only_for_a_ramped_case() -> None:+ ramp: Final = Ramp(first_rps=2.0, last_rps=4.0, step_rps=2.0, step_seconds=5.0, held_p99_seconds=5.0)+ ramped_case: Final = replace(CASE, traffic=ramped(CASE.traffic, ramp, seed=3, concurrency=20))+ assert ramped_case.traffic.expected_seconds == 10.0+ assert [item.send_at_s for item in build_plan(ramped_case)] == list(send_offsets(ramp, seed=3))+ assert all(item.send_at_s is None for item in build_plan(CASE))+++def _step(rps: float, p99: float, completed_rps: float | None = None, unexpected: int = 0) -> StepReading:+ return StepReading(+ rps=rps,+ planned=int(rps * 90),+ completed_rps=rps if completed_rps is None else completed_rps,+ gateway_added_p99=p99,+ unexpected=unexpected,+ held_p99_seconds=5.0,+ )+++def test_sustained_rps_is_the_last_step_held_before_the_first_that_broke() -> None:+ assert sustained_rps((_step(10, 0.5), _step(20, 1.0), _step(30, 9.0), _step(40, 0.7))) == 20+ assert sustained_rps((_step(10, 0.5), _step(20, 1.0, completed_rps=17.0), _step(30, 1.0))) == 10+ assert sustained_rps((_step(10, 0.5), _step(20, 1.0, unexpected=19), _step(30, 1.0))) == 10+ assert sustained_rps((_step(10, 0.5), _step(20, 1.0, unexpected=18), _step(30, 1.0))) == 30+ assert sustained_rps((_step(10, 6.0), _step(20, 1.0))) == 0+++def _record(seq: int, started_at: float, total_s: float) -> WireRecord:+ return WireRecord(+ seq=seq,+ marker=f"perfreq-{seq:06d}-n",+ caller="key-a1",+ endpoint="chat_completions",+ stream=False,+ blocked=False,+ started_at=started_at,+ status=200,+ header_s=total_s,+ first_byte_s=total_s,+ first_token_s=None,+ total_s=total_s,+ text="",+ ok=True,+ mock_delay_s=1.0,+ call_id=None,+ error=None,+ )+++def test_a_floor_fails_when_the_second_step_s_latency_breaks_it_and_the_properties_carry_the_floor() -> None:+ ramp: Final = Ramp(first_rps=10.0, last_rps=20.0, step_rps=10.0, step_seconds=100.0, held_p99_seconds=2.0)+ case: Final = replace(+ CASE,+ traffic=ramped(CASE.traffic, ramp, seed=1, concurrency=50),+ limits=(Limit("sustained_rps", min=20),),+ )+ offsets: Final = send_offsets(ramp, seed=1)+ records: Final = tuple(+ _record(seq, 1000.0 + offset, 1.5 if ramp.step_of(seq) == 0 else 4.0)+ for seq, offset in enumerate(offsets, start=1)+ )+ observation: Final = Observation(+ records=records,+ mock_requests=(),+ guardrail_calls=(),+ rss_before=NO_RSS,+ rss_after=NO_RSS,+ wall_seconds=200.0,+ ramp=ramp,+ )+ first, second = observation.steps()+ assert first.held and first.gateway_added_p99 == 0.5 and first.completed_rps > 9.5+ assert not second.held and second.gateway_added_p99 == 3.0+ rows, exceeded = evaluate(case, observation)+ assert exceeded == ["sustained_rps: observed 10.0000 < floor 20.0000"]+ assert {("perf_sustained_rps", 10.0), ("perf_sustained_rps_floor", 20.0)} <= set(junit_properties(rows))+ assert dict(step_properties((first, second))) == {+ "perf_ramp_hold_p99_seconds": 2.0,+ "perf_ramp_step_10_gateway_added_p99": 0.5,+ "perf_ramp_step_10_completed_rps": first.completed_rps,+ "perf_ramp_step_10_unexpected": 0.0,+ "perf_ramp_step_10_gateway_added_p50": 0.5,+ "perf_ramp_step_20_gateway_added_p99": 3.0,+ "perf_ramp_step_20_completed_rps": second.completed_rps,+ "perf_ramp_step_20_unexpected": 0.0,+ "perf_ramp_step_20_gateway_added_p50": 3.0,+ }+ assert not evaluate(replace(case, limits=(Limit("sustained_rps", min=10),)), observation)[1]+++def test_each_step_reads_the_pod_s_cpu_redis_latency_and_connections_from_its_own_window() -> None:+ ramp: Final = Ramp(first_rps=10.0, last_rps=20.0, step_rps=10.0, step_seconds=100.0, held_p99_seconds=2.0)+ records: Final = tuple(+ _record(seq, 1000.0 + offset, 1.5) for seq, offset in enumerate(send_offsets(ramp, seed=1), start=1)+ )+ started: Final = min(record.started_at for record in records)+ redis: Final = tuple(+ Sample(at=started + at, value=GatewayTelemetry(redis_latency=HistogramTotals(total=total, count=count)))+ for at, total, count in ((-1.0, 0.0, 0.0), (50.0, 1.0, 100.0), (99.0, 2.0, 200.0), (199.0, 42.0, 300.0))+ )+ observation: Final = Observation(+ records=records,+ mock_requests=(),+ guardrail_calls=(),+ rss_before=NO_RSS,+ rss_after=NO_RSS,+ wall_seconds=200.0,+ ramp=ramp,+ cpu_series=Series(samples=tuple(Sample(at=started + at, value=5.0) for at in (10.0, 60.0, 110.0, 160.0))),+ connection_series=Series(+ samples=tuple(+ Sample(at=started + at, value=n) for at, n in ((1.0, 12), (50.0, 18), (150.0, 25), (199.0, 20))+ )+ ),+ telemetry_series=Series(samples=redis),+ )+ first, second = observation.steps()+ first_completed: Final = round(first.completed_rps * ramp.step_seconds)+ assert first.cpu_seconds_per_request is not None and math.isclose(+ first.cpu_seconds_per_request, 10.0 / first_completed+ )+ assert first.redis_latency_mean_seconds is not None and math.isclose(first.redis_latency_mean_seconds, 0.01)+ assert first.postgres_peak_connections == 18+ assert second.redis_latency_mean_seconds is not None and math.isclose(second.redis_latency_mean_seconds, 0.4)+ assert second.postgres_peak_connections == 25+ assert first.time_to_first_token_p99 is None+ names: Final = {name for name, _ in step_properties((first, second)) if name.startswith("perf_ramp_step_10_")}+ assert names == {+ "perf_ramp_step_10_gateway_added_p99",+ "perf_ramp_step_10_completed_rps",+ "perf_ramp_step_10_unexpected",+ "perf_ramp_step_10_gateway_added_p50",+ "perf_ramp_step_10_cpu_seconds_per_request",+ "perf_ramp_step_10_redis_latency_mean_seconds",+ "perf_ramp_step_10_postgres_peak_connections",+ }diff --git a/tests/e2e_harness/perf/test_upgrade.py b/tests/e2e_harness/perf/test_upgrade.pyindex 9a6c9d2af14..3414476ba4c 100644--- a/tests/e2e_harness/perf/test_upgrade.py+++ b/tests/e2e_harness/perf/test_upgrade.py@@ -304,7 +304,9 @@ def test_invalid_200_is_not_an_allowed_non_200(failed: WireRecord) -> None:def _ceiling(metric: str, percentile: int | None = None) -> float:return next(- limit.max for limit in upgrade_case().limits if limit.metric == metric and limit.percentile == percentile+ limit.max+ for limit in upgrade_case().limits+ if limit.metric == metric and limit.percentile == percentile and limit.max is not None)