The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
better output dirs and charts for benching
ddux68ad3a33a458
diff --git a/.github/workflows/benchmark-hardware.yml b/.github/workflows/benchmark-hardware.ymlindex 02d2ab6c..f03d29fb 100644--- a/.github/workflows/benchmark-hardware.yml+++ b/.github/workflows/benchmark-hardware.yml@@ -18,7 +18,7 @@ on:required: truedefault: 'spans,metrics,logs'scenario:- description: 'Benchmark scenario: throughput (two-phase ingest ramp, default) or read-probe (ingest to fill levels, probe a read, fail above 5s).'+ description: 'Benchmark scenario: throughput (three-phase ingest ramp, default; writes to benchmarks/results-throughput/) or read-probe (ingest to fill levels, probe a read, fail above 5s; writes to benchmarks/results-probe/).'required: truetype: choiceoptions:@@ -119,13 +119,13 @@ jobs:run: ./benchmarks/scripts/preflight.sh- name: Run matrix entry (${{ matrix.tier }} / ${{ matrix.mode }} / ${{ matrix.signal }})- run: ./benchmarks/scripts/run-matrix-entry.sh "${{ matrix.tier }}" "${{ matrix.mode }}" "${{ matrix.signal }}" "${{ github.event.inputs.duration }}" benchmarks/results/_artifact+ run: ./benchmarks/scripts/run-matrix-entry.sh "${{ matrix.tier }}" "${{ matrix.mode }}" "${{ matrix.signal }}" "${{ github.event.inputs.duration }}" benchmarks/_artifact- uses: actions/upload-artifact@v4if: always()with:name: result-${{ matrix.tier }}-${{ matrix.mode }}-${{ matrix.signal }}- path: benchmarks/results/_artifact/*.json+ path: benchmarks/_artifact/*.jsonif-no-files-found: warnaggregate:@@ -143,15 +143,19 @@ jobs:with:path: /tmp/results-raw- - name: Stage into benchmarks/results/latest/+ - name: Stage into scenario folderid: stagerun: |set -euo pipefail- # Always overwrite `latest/` so each dispatch reports only its own- # matrix entries. Cross-run comparison happens via `git log`, not- # via cumulative same-folder data — the old date-stamped folder- # approach silently mixed dispatches that landed on the same day.- dst="benchmarks/results/latest"+ # One canonical folder per scenario, wiped on each dispatch so the+ # committed tree reflects exactly one dispatch — but the two+ # scenarios live in sibling folders so a read-probe run never+ # clobbers throughput results and vice-versa.+ case "${{ github.event.inputs.scenario }}" in+ throughput) dst="benchmarks/results-throughput" ;;+ read-probe) dst="benchmarks/results-probe" ;;+ *) echo "unknown scenario '${{ github.event.inputs.scenario }}'" >&2; exit 1 ;;+ esacrm -rf "${dst}"mkdir -p "${dst}"find /tmp/results-raw -name '*.json' -exec cp {} "${dst}/" \;@@ -173,8 +177,8 @@ jobs:# Copy our staged data aside before touching git state. We always# need to restore it after `reset --hard` regardless of what main- # currently has at `latest/` — this dispatch's data is the new- # source of truth.+ # currently has at this scenario folder — this dispatch's data is+ # the new source of truth.tmp_staged="$(mktemp -d)/staged"cp -a "${staged}" "${tmp_staged}"@@ -182,12 +186,14 @@ jobs:git fetch origin maingit reset --hard origin/main- # Wipe whatever main had at `latest/` (probably a prior dispatch's- # results) and replace it with this dispatch's full output. Using- # rm + cp + `git add -A` so deletions are captured: if the prior- # commit had `ccx33-pgch-metrics.json` but this dispatch only ran- # sqlite, the pgch JSON gets staged for deletion. The committed- # `latest/` always reflects exactly one dispatch, never a union.+ # Wipe whatever main had at this scenario folder (probably a prior+ # dispatch of the SAME scenario) and replace it with this+ # dispatch's full output. Using rm + cp + `git add -A` so deletions+ # are captured: if the prior commit had `ccx33-pgch-metrics-throughput.json`+ # but this dispatch only ran sqlite, the pgch JSON gets staged for+ # deletion. The committed scenario folder always reflects exactly+ # one dispatch of that scenario, never a union. The OTHER scenario+ # folder is untouched by this step.rm -rf "${staged}"mkdir -p "${staged}"cp -a "${tmp_staged}/." "${staged}/"@@ -198,5 +204,5 @@ jobs:echo "nothing to commit"exit 0fi- git commit -m "bench: hardware benchmark $(date -u +%Y-%m-%d) (latest)"+ git commit -m "bench: ${{ github.event.inputs.scenario }} $(date -u +%Y-%m-%d)"git pushdiff --git a/.gitignore b/.gitignoreindex de49390f..38a97d79 100644--- a/.gitignore+++ b/.gitignore@@ -47,11 +47,11 @@ backend/static/dist//examples/nestjs-otel/node_modules/website/.wrangler-# Benchmark tooling: built binaries and laptop-only result dirs.-# Real CI-produced result dirs (e.g. benchmarks/results/2026-05-12/) are-# committed; the -local suffix marks dev runs we never want to push.+# Benchmark tooling: built binaries and runner-only scratch dirs. The+# committed result dirs are benchmarks/results-throughput/ and+# benchmarks/results-probe/; everything else is local or runner-scoped./benchmarks/loadgen/loadgen/benchmarks/loadgen/loadgen-linux-amd64/benchmarks/loadgen/loadgen-linux-arm64-/benchmarks/results/*-local/+/benchmarks/_artifact/benchmarks/loadgen/bench-loadgendiff --git a/benchmarks/LOCAL.md b/benchmarks/LOCAL.mdindex 04c29a99..909786a0 100644--- a/benchmarks/LOCAL.md+++ b/benchmarks/LOCAL.md@@ -124,8 +124,11 @@ echo "PROJECT_TOKEN=$PROJECT_TOKEN"`seed-project.sh` hits `/api/register`. On self-hosted backends only oneorganization is allowed, so this only works on a **fresh** DB — step 1's-`down -v` is what guarantees that. The OTLP-only loadgen no longer needs the-JWT or project ID, so we extract only the project token.+`down -v` is what guarantees that. The throughput scenario only needs the+project token (OTLP ingestion is bearer-authed); the read-probe scenario+also needs the JWT + project ID for the dashboard endpoints — see+[Running the read-probe scenario locally](#running-the-read-probe-scenario-locally)+below.### 4. Build and run the loadgen, one signal at a time@@ -134,9 +137,17 @@ JWT or project ID, so we extract only the project token.mkdir -p /tmp/bench-localhost && rm -f /tmp/bench-localhost/*.json```-Pick numbers appropriate for the mode you booted in step 1. The two-phase-ramp first grows batch size at a fixed low request rate, then grows request-rate at the winning batch size.+Pick numbers appropriate for the mode you booted in step 1. The throughput+scenario runs a three-phase ramp:+1. **Phase 1** grows batch size at a fixed low request rate (saturates one+ client; finds the per-request decode/insert ceiling).+2. **Phase 2** grows request rate at the winning batch size capped at+ `--phase2-batch-cap` (collector shape: fat batches, low req rate).+3. **Phase 3** grows request rate at batch=100 (SDK-fleet shape: small+ batches, high req rate — stresses HTTP/auth/decode rather than insert).++The examples below trim Phase 3 to keep laptop runs short; remove the+`--phase3-*` overrides to exercise it.**SQLite (laptop) — quick smoke per signal:**@@ -149,7 +160,8 @@ for sig in spans metrics logs; do--duration 5m --step-duration 30s \--phase1-batch-sizes 256,1024,4096 \--phase2-request-rates 1,5,20 \- --report-out "/tmp/bench-localhost/local-sqlite-${sig}.json" \+ --phase3-request-rates 10,100 \+ --report-out "/tmp/bench-localhost/local-sqlite-${sig}-throughput.json" \--tier local --mode sqlitedone```@@ -165,12 +177,17 @@ for sig in spans metrics logs; do--duration 8m --step-duration 45s \--phase1-batch-sizes 256,1024,4096,8192 \--phase2-request-rates 1,5,20,50 \- --report-out "/tmp/bench-localhost/local-pgch-${sig}.json" \+ --phase3-request-rates 10,100,500 \+ --report-out "/tmp/bench-localhost/local-pgch-${sig}-throughput.json" \--tier local --mode pgchdone```-On stderr you'll see two-phase progress like:+The `-throughput` suffix on the filename matches the scenario folder convention+(`benchmarks/results-throughput/`); use `-read-probe` when running+`--scenario read-probe` (see [Running the read-probe scenario locally](#running-the-read-probe-scenario-locally) below).++On stderr you'll see per-phase progress like:```phase1 step 1: batch=256 rate=5.0 items/s=1280 p99=12ms err=0.00% passed=true@@ -179,7 +196,9 @@ phase1 step 3: batch=4096 rate=5.0 items/s=20480 p99=42ms err=0.00% passed=truephase2 step 1: batch=4096 rate=1.0 items/s=4096 p99=15ms err=0.00% passed=truephase2 step 2: batch=4096 rate=5.0 items/s=20480 p99=21ms err=0.00% passed=truephase2 step 3: batch=4096 rate=20.0 items/s=81920 p99=180ms err=0.00% passed=true-wrote /tmp/bench-localhost/local-sqlite-spans.json: signal=spans max sustainable spans/sec = 81920+phase3 step 1: batch=100 rate=10.0 items/s=1000 p99=8ms err=0.00% passed=true+phase3 step 2: batch=100 rate=100.0 items/s=10000 p99=22ms err=0.00% passed=true+wrote /tmp/bench-localhost/local-sqlite-spans-throughput.json: signal=spans max sustainable spans/sec = 81920```A run finishes when either (a) a step fails the error threshold, (b) all@@ -189,12 +208,19 @@ configured steps pass, or (c) `--duration` expires.```bash.venv-bench/bin/python benchmarks/scripts/chart.py /tmp/bench-localhost/-open /tmp/bench-localhost/chart-spans.png-open /tmp/bench-localhost/chart-metrics.png-open /tmp/bench-localhost/chart-logs.png+open /tmp/bench-localhost/chart-spans.png # headline bars per tier × mode+open /tmp/bench-localhost/chart-phase2-rate-spans.png # collector-shape rate ramp+open /tmp/bench-localhost/chart-pareto-spans.png # latency–throughput trade-off+open /tmp/bench-localhost/chart-cliff-spans.png # step-by-step pass/fail gridcat /tmp/bench-localhost/summary.md```+The full chart suite (~14 PNG types per scenario folder, headline ++phase-by-phase ramps + Pareto + tier scaling + cliff grids + batch+efficiency + signal mix, plus read-probe equivalents) is described in+[charts.md](charts.md). `chart.py` auto-detects which scenarios are in+the folder and only renders the relevant charts.+A one-entry result renders fine; it just won't have multiple bars/lines tocompare against. Old pre-OTLP-split JSONs (without a `signal` field) areskipped with a log line.@@ -211,11 +237,12 @@ The `-v` deletes the named volume so the next run is guaranteed clean.## Reading the output-Each result JSON has two phases:+Each throughput result JSON has up to three phases plus a headline:```json{"tier": "local", "mode": "sqlite", "signal": "spans",+ "scenario": "throughput","startedAt": "2026-05-15T...", "endedAt": "...","phase1": {"kind": "batch-size-ramp",@@ -236,16 +263,31 @@ Each result JSON has two phases:"steps": [ ... ],"maxRequestRate": 100},+ "phase3": {+ "kind": "small-batch-rate-ramp",+ "fixedBatchSize": 100,+ "steps": [ ... ],+ "maxRequestRate": 1000+ },"maxSustainableItemsPerSec": 819200}```**A step passes** when the combined error rate (HTTP failures + OTLP-`PartialSuccess` rejected items) is ≤ `--ingest-err-threshold` (default 5%).-Read endpoints are not probed in this benchmark.+`PartialSuccess` rejected items) is ≤ `--ingest-err-threshold` (default 5%)+AND the achieved request rate is ≥ `--soft-cliff-ratio × target` (default+70%). The second check catches saturated-but-not-erroring soft cliffs.-`maxSustainableItemsPerSec = phase2.fixedBatchSize × phase2.maxRequestRate`,-where `maxRequestRate` is the rate of the last passing Phase 2 step.+`maxSustainableItemsPerSec` is the **highest `actualItemsPerSec` recorded+across any passing step from any phase** — measured from real OK responses,+not `batchSize × targetRate` (which over-reports when the workers saturate+before the limiter does). Different phases probe different shapes+(collector fat-batch vs SDK small-batch); the headline reports the best+shape's ceiling and per-phase numbers stay in the JSON for shape-specific+analysis.++Read-probe JSONs have a different shape — see [Running the read-probe+scenario locally](#running-the-read-probe-scenario-locally) below.---@@ -281,13 +323,19 @@ sustain on synthetic, evenly-distributed load."| Flag | Default | When to change ||------|---------|----------------|| `--signal` | (required) | One of `spans`, `metrics`, `logs`. |+| `--scenario` | `throughput` | `throughput` (three-phase ingest ramp) or `read-probe` (fill the table, probe a read at each level). || `--phase1-batch-sizes` | `256,1024,4096,8192,16384` | Drop the lower steps on beefy boxes to save time; raise the cap to find the per-request decode/insert ceiling. || `--phase2-request-rates` | `1,5,25,100,400` | Raise the cap to find the request-rate ceiling on big boxes; lower for cheap smoke tests. |+| `--phase3-request-rates` | `10,100,1000,5000,10000` | The SDK-fleet shape ramp at batch=100. Lower the cap for fast smoke tests; drop the flag entirely to use the default range. || `--phase1-fixed-rate` | 5 | Per-second request rate held constant during Phase 1. Small enough that batch size is the only variable. |-| `--phase2-batch-cap` | 8192 | Cap on Phase 2 batch size (matches OTel collector default `send_batch_size`). Phase 2 uses `min(this, Phase 1 winner)`. |+| `--phase2-batch-cap` | 16384 | Cap on Phase 2 batch size. Phase 2 uses `min(this, Phase 1 winner)`. Bumped past the OTel collector default of 8192 because pgch SUTs can usefully exceed it. |+| `--phase3-batch-size` | 100 | Fixed batch size for Phase 3 — matches typical language-SDK `BatchSpanProcessor` output rather than the collector's 8192. |+| `--soft-cliff-ratio` | 0.70 | A step also fails when achieved req-rate falls below this fraction of target — catches saturated-but-not-erroring cliffs. Set to 0 to disable. || `--step-duration` | 2m | Shorten for smoke checks (15–30s), keep at 2m for real measurements. || `--duration` | 30m | Hard cap on total wall time. || `--ingest-err-threshold` | 0.05 | Combined HTTP error + OTLP rejected rate that fails a step. |+| `--fill-levels` | `100000,1000000,10000000,100000000` | (read-probe only) Row counts to fill the table to before each probe. |+| `--read-threshold-ms` | 5000 | (read-probe only) A probe fails if the read exceeds this. |The ramp is geometric (256 → 1024 → 4096 → …), so cliff resolution is coarseby design — for a precise number, run a second pass with a narrower band of@@ -321,6 +369,87 @@ values around the known cliff.---+## Running the read-probe scenario locally++The throughput scenario above answers "how fast can the SUT swallow OTLP+data?" The read-probe scenario answers "how big can the table grow before+the dashboard read on this endpoint cliffs?" Each fill level is reached by+ingesting OTLP until the row count is hit, then the loadgen waits+`--settle-seconds` and issues one read against the signal's dashboard+endpoint (`/api/endpoints/grouped` for spans, `/api/metrics/application`+for metrics, `/api/logs` for logs). The step fails when the read takes+longer than `--read-threshold-ms` (default 5 s).++Read-probe needs a JWT + project ID, not just a project token, because the+dashboard endpoints are JWT-authenticated. `seed-project.sh` returns both:++```bash+seed_json=$(benchmarks/scripts/seed-project.sh http://localhost:8087)+export PROJECT_TOKEN=$(echo "${seed_json}" | jq -r .projectToken)+export PROJECT_ID=$(echo "${seed_json}" | jq -r .projectId)+export JWT=$(echo "${seed_json}" | jq -r .jwt)+```++Then run the loadgen with `--scenario read-probe`. Small fill levels keep+the laptop honest:++```bash+mkdir -p /tmp/bench-localhost && rm -f /tmp/bench-localhost/*.json+for sig in spans metrics logs; do+ benchmarks/loadgen/loadgen \+ --target http://localhost:8087 \+ --token "$PROJECT_TOKEN" --jwt "$JWT" --project-id "$PROJECT_ID" \+ --signal "$sig" --scenario read-probe \+ --fill-levels 10000,100000,1000000 \+ --settle-seconds 5s \+ --report-out "/tmp/bench-localhost/local-pgch-${sig}-read-probe.json" \+ --tier local --mode pgch+done+.venv-bench/bin/python benchmarks/scripts/chart.py /tmp/bench-localhost/+open /tmp/bench-localhost/chart-readprobe-headline-spans.png+open /tmp/bench-localhost/chart-readprobe-spans.png+open /tmp/bench-localhost/chart-readprobe-cliff-spans.png+```++`chart.py` against a folder mixing throughput and read-probe JSONs renders+both suites — the wrong-scenario renderers are no-ops. In production, the+CI workflow keeps them separate by writing to `benchmarks/results-throughput/`+vs `benchmarks/results-probe/` based on the `--scenario` flag.++Read-probe JSONs have a different shape from throughput:++```json+{+ "tier": "local", "mode": "pgch", "signal": "spans",+ "scenario": "read-probe",+ "readProbe": {+ "readPath": "/api/endpoints/grouped",+ "readThresholdMs": 5000,+ "settleSeconds": 5,+ "fillBatchSize": 8192,+ "fillRequestRate": 100,+ "steps": [+ {+ "fillLevelTarget": 10000,+ "rowsIngested": 10123,+ "ingestSecondsElapsed": 3.2,+ "readLatencyMs": 180,+ "readOk": true,+ "passed": true+ }+ ],+ "maxFillLevelPassed": 1000000+ },+ "maxFillLevelPassed": 1000000+}+```++The headline number is `maxFillLevelPassed` — the largest row count at+which the read came back under threshold. The matching read latency lives+in the last passing step's `readLatencyMs`.++---+## Going beyond localOnce the local run works end-to-end, move to the Hetzner ladder:diff --git a/benchmarks/README.md b/benchmarks/README.mdindex bf9ea3ef..876c4130 100644--- a/benchmarks/README.md+++ b/benchmarks/README.md@@ -31,13 +31,26 @@ pollute the latency signal:| CCX33 | 8 | 32 GB | 240 GB NVMe || CCX43 | 16 | 64 GB | 360 GB NVMe |-## Not measured here+## Two scenarios-This benchmark is **ingest-only** — it does not probe dashboard read latency.-A separate future benchmark will pre-load N events (100k, 1M, 10M, 100M) and-measure read endpoint P99 at each fill level. Don't read the numbers here as-"the dashboard stays usable at N items/sec" — they're "the SUT keeps swallowing-items at N items/sec without erroring or rejecting."+Two distinct questions; one matrix-entry script answers either depending on+the `--scenario` flag.++- **`throughput` (default).** A three-phase ingest ramp (Phase 1: batch+ size; Phase 2: collector-shape rate; Phase 3: SDK-fleet-shape rate).+ Results land in `benchmarks/results-throughput/`. Answers "how fast can+ the SUT swallow OTLP data without erroring or rejecting?"+- **`read-probe`.** Fills the table to a sequence of row counts+ (`100k, 1M, 10M, 100M`) and probes one dashboard read at each level,+ failing the step when the read exceeds `--read-threshold-ms` (default+ 5 s). Results land in `benchmarks/results-probe/`. Answers "how big+ can the table grow before the dashboard read on this endpoint cliffs?"++The two scenarios write to sibling folders so one never overwrites the+other. Each folder is wiped on each dispatch of its scenario.++Don't read the throughput number as "the dashboard stays usable at N+items/sec" — that's what read-probe is for.## How a run works@@ -51,7 +64,7 @@ Per matrix entry (one tier × one mode × one signal):4. `loadgen-bootstrap.sh` cross-compiles the loadgen, pushes it to theloadgen box, runs it with `--signal <spans|metrics|logs>` against theSUT's *private* IP.-5. The loadgen runs a two-phase ramp:+5. The loadgen runs a three-phase ramp:- **Phase 1 — batch-size ramp.** Single client at a fixed 5 req/sec.Batch sizes step through `256,1024,4096,8192,16384`. Each step holds for`--step-duration` (default 2 min). Stops at the first failing step.@@ -111,9 +124,11 @@ prevents a permanently-broken SUT from restart-looping forever; if thethird restart also crashes, the container stays down and the health-pollfires the clean-abort path.-After all matrix entries finish, `chart.py` renders three bar charts-(`chart-spans.png`, `chart-metrics.png`, `chart-logs.png`), three detail-charts (`chart-detail-<signal>.png`), and a combined `summary.md`.+After all matrix entries finish, `chart.py` renders the full chart suite+(headline bars, Phase 1 / 2 / 3 ramps, Pareto, tier scaling, cliff grid,+batch efficiency, signal mix — plus the read-probe equivalents in the+read-probe folder) and a combined `summary.md`. See+[charts.md](charts.md) for a guide to reading each chart.## Running from your laptop@@ -156,11 +171,12 @@ works, the full matrix works.```4 tiers × 2 modes × 3 signals, default `--duration 30m`. Output goes to-`benchmarks/results/latest-local/`, which is wiped at the start of every-run so each dispatch's output stands alone — no silent cross-run mixing-that the old date-folder layout produced. The `-local` suffix keeps dev-runs separate from the committed `latest/` so you don't accidentally-clobber CI's data.+`benchmarks/results-throughput/` (or `benchmarks/results-probe/` when+running `--scenario read-probe`), which is wiped at the start of every run+so each dispatch's output stands alone — no silent cross-run mixing that+the old date-folder layout produced. The two scenario folders are+independent: a read-probe run never touches `results-throughput/` and+vice-versa.### Other useful invocations@@ -180,9 +196,8 @@ clobber CI's data.# Override the per-entry runtime./benchmarks/scripts/run-local.sh --tier ccx13 --duration 10m-# Drop the "-local" suffix so the result dir matches the CI naming convention-# (use this only when you want to commit a one-off re-benchmark)-./benchmarks/scripts/run-local.sh --tier ccx23 --commit+# Switch to the read-probe scenario (writes to benchmarks/results-probe/)+./benchmarks/scripts/run-local.sh --scenario read-probe```## Running from GitHub Actions@@ -198,14 +213,18 @@ Required GitHub secrets:`benchmark-key`.After the matrix completes, an `aggregate` job downloads all artifacts, runs-`chart.py`, and commits `benchmarks/results/latest/` to `main` via a bot-commit (`git add -A`, so files from a prior dispatch that aren't in this one-get staged for deletion). No PR — it's a generated artifact. **The-committed `latest/` always reflects exactly one dispatch**; comparing to-previous runs is `git log -- benchmarks/results/latest/` or-`git show <sha>:benchmarks/results/latest/summary.md`. Dated historical-folders (`benchmarks/results/2026-05-15/` etc.) from before this layout-change are kept in git for reference but are no longer written to.+`chart.py`, and commits the matching scenario folder+(`benchmarks/results-throughput/` for throughput runs,+`benchmarks/results-probe/` for read-probe runs) to `main` via a bot commit+(`git add -A`, so files from a prior dispatch that aren't in this one get+staged for deletion). No PR — it's a generated artifact. **Each scenario+folder always reflects exactly one dispatch of that scenario**; a+throughput dispatch never modifies `results-probe/` and vice-versa.+Comparing to previous runs is `git log -- benchmarks/results-throughput/`+or `git show <sha>:benchmarks/results-throughput/summary.md`. Dated+historical folders (`benchmarks/results/2026-05-15/` etc.) from before+this layout change are kept in git for reference but are no longer written+to.## Running against managed ClickHouse@@ -278,9 +297,12 @@ benchmarks/reset-managed-ch.sh # Wipes the managed CH DB between matrix entriesloadgen-bootstrap.sh # Cross-compiles + runs loadgenseed-project.sh # /api/register -> JWT + project token JSON- chart.py # matplotlib renderer (per-signal charts)+ chart.py # matplotlib renderer (throughput + read-probe charts)_ssh.sh # Shared ssh/rsync helpers- results/ # Committed bench results live here+ results-throughput/ # Committed throughput results (wiped per dispatch)+ results-probe/ # Committed read-probe results (wiped per dispatch)+ results/ # Historical dated folders (not written to anymore)+ charts.md # Reading guide for every chart chart.py emits```## Failure modes & debuggingdiff --git a/benchmarks/charts.md b/benchmarks/charts.mdnew file mode 100644index 00000000..32312fba--- /dev/null+++ b/benchmarks/charts.md@@ -0,0 +1,309 @@+# benchmarks/charts.md — chart reading guide++Every PNG under `benchmarks/results-throughput/` and `benchmarks/results-probe/`+is rendered by [`benchmarks/scripts/chart.py`](scripts/chart.py) from the JSON+files in the same folder. This doc explains what each chart shows, what+question it answers, how to read it, and what to look for. Pair this with+[README.md](README.md) for the bench-mechanics side.++The two output folders are siblings — throughput runs never touch+`results-probe/` and vice-versa. Each folder is wiped on every dispatch of+its scenario, so the committed PNGs always reflect exactly one run.++**File naming.** Per-signal charts: `chart-<name>-<signal>.png` where+`<signal>` ∈ {`spans`, `metrics`, `logs`}. Cross-signal charts have no+signal suffix (`chart-signal-mix.png`, `chart-readprobe-signal-mix.png`).+The headline bar is the special case: `chart-<signal>.png` (no `-headline`+suffix; it's the most-cited artifact and gets the short name).++---++## Reading guide — throughput++Folder: `benchmarks/results-throughput/`++| Question | Chart |+|---|---|+| What's the headline number? | [`chart-<signal>.png`](#t1--chart-signalpng--headline-bars) |+| How does latency grow with batch size at low concurrency? | [`chart-phase1-batch-<signal>.png`](#t2--chart-phase1-batch-signalpng--phase-1-batch-size-scaling) |+| Where does the SUT cliff on collector-shape rate? | [`chart-phase2-rate-<signal>.png`](#t3--chart-phase2-rate-signalpng--phase-2-collector-shape-rate-ramp) |+| Where does the SUT cliff on SDK-fleet-shape rate? | [`chart-phase3-rate-<signal>.png`](#t4--chart-phase3-rate-signalpng--phase-3-sdk-fleet-rate-ramp-conditional) |+| What does each K items/sec cost in P99? | [`chart-pareto-<signal>.png`](#t5--chart-pareto-signalpng--latencythroughput-pareto) |+| Does doubling cores double throughput? | [`chart-tier-scaling-<signal>.png`](#t6--chart-tier-scaling-signalpng--tier-scaling) |+| What worked and what failed, at a glance? | [`chart-cliff-<signal>.png`](#t7--chart-cliff-signalpng--step-status-grid) |+| Is batching still paying off, or is the SUT dropping batches? | [`chart-batch-efficiency-<signal>.png`](#t8--chart-batch-efficiency-signalpng--batch-efficiency) |+| At the same hardware, how do spans vs metrics vs logs compare? | [`chart-signal-mix.png`](#t9--chart-signal-mixpng--cross-signal-comparison) |++## Reading guide — read-probe++Folder: `benchmarks/results-probe/`++| Question | Chart |+|---|---|+| How many rows can you have before the dashboard cliffs? | [`chart-readprobe-headline-<signal>.png`](#p1--chart-readprobe-headline-signalpng--max-fill-level-by-tier-mode) |+| How steeply does read latency grow with table size? | [`chart-readprobe-<signal>.png`](#p2--chart-readprobe-signalpng--read-latency-vs-rows-ingested) |+| How long does it take to fill each level? | [`chart-readprobe-ingest-time-<signal>.png`](#p3--chart-readprobe-ingest-time-signalpng--fill-time-vs-target-rows) |+| Does more hardware buy a bigger queryable dataset? | [`chart-readprobe-tier-scaling-<signal>.png`](#p4--chart-readprobe-tier-scaling-signalpng--read-probe-tier-scaling) |+| Which (tier × mode × fill) combos passed? | [`chart-readprobe-cliff-<signal>.png`](#p5--chart-readprobe-cliff-signalpng--read-probe-step-status-grid) |+| Which read endpoint scales worst on the same hardware? | [`chart-readprobe-signal-mix.png`](#p6--chart-readprobe-signal-mixpng--read-probe-cross-signal) |++---++## Failure-state glossary++The charts use a shared vocabulary for what "failed" means at each step. Most+status colors are reused across the cliff grids.++| Term | Meaning | Where it shows |+|---|---|---|+| **pass** (green) | Step is below the threshold and the loadgen marked it passing. | Cliff grids, `o` markers on lines/scatter. |+| **soft-cliff** (yellow) | Throughput steps: achieved ≤ 70% of attempted with no error spike — the SUT's workers couldn't keep up with the limiter. Read-probe: the read returned 200 but latency was above threshold (default 5000ms). | Cliff grid yellow cell; on line charts, look for the gap between `actual` (solid) and `attempted` (dashed) lines in T3/T8. |+| **hard fail** (red) | Throughput: combined HTTP error rate + OTLP `rejected` items exceeded 5% of attempted. Read-probe: the read returned a non-2xx response or errored out. | Cliff grid red cell; `x` marker on every line/scatter chart. |+| **0 requests completed** / **dead** (dark grey) | The SUT never returned a single OK response in the step's window — usually the container was being restarted by Docker's `restart: on-failure:3`. Shows up as `actualItemsPerSec = 0`. | Cliff grid dark-grey cell; Pareto chart bottom annotation; line charts drop the point (log scale). |+| **not run** (light grey) | The phase or step didn't execute in this matrix entry (e.g. Phase 2 was skipped because Phase 1 cliffed first, or this fill level was never reached). | Cliff grid light-grey cell. |+| **failed @ P1 step N** | Run never produced any sustainable throughput. Annotation on headline bars. | T1 hatched bar, T6 dropped point. |+| **Phase 1 floor** | The Phase 1 P99 latency at the matching batch size — what latency looks like with zero concurrency. Anything above the floor is concurrency tax. | Thin dotted horizontal lines on T3 / T4 middle subplot. |++Numeric thresholds match `benchmarks/loadgen/` defaults: `--ingest-err-threshold 0.05` (5% combined error rate) and `--soft-cliff-ratio 0.7` (70% achieved-vs-attempted).++---++## Throughput charts++### T1 — `chart-<signal>.png` — Headline bars++- **Filename:** `chart-spans.png`, `chart-metrics.png`, `chart-logs.png`+- **Folder:** `results-throughput/`+- **What it shows:** Grouped bars of `maxSustainableItemsPerSec` per (tier, mode).+- **Question answered:** On hardware tier X with DB mode Y, what's the largest sustainable ingest rate for signal Z?+- **How to read it:**+ - X axis: hardware tier. Y axis: max sustainable items/sec.+ - Each bar is annotated with the absolute value and the **phase that produced it** in parentheses (`283,187 (P2)`). P1 = batch-size ramp at 5 req/sec, P2 = collector-shape rate ramp, P3 = SDK-fleet-shape rate ramp.+ - **Hatched grey bars** signal "all phases failed" — the run never produced any sustainable throughput. The annotation tells you which phase + step was the first to fail (`failed @ P1 step 3`).+- **What to look for:**+ - Mode floor — sqlite vs pgch gap shows the cost of the simpler stack.+ - Phase tag — a P1-sourced headline means the SUT saturated on a single client; P2/P3 means it scaled with concurrency.+ - Hatched bars vs `0`-height bars: the old chart rendered failures as zero-height bars that looked identical to "not run." Now they're explicitly flagged.+- **Why it's good:** Headline answer in one image. Disambiguates "failed" from "not measured" — a UX gap the old chart had.++### T2 — `chart-phase1-batch-<signal>.png` — Phase 1 batch-size scaling++- **Folder:** `results-throughput/`+- **What it shows:** Latency P50/P95/P99 vs batch size at fixed 5 req/sec.+- **Question answered:** At fixed 5 req/sec, how does per-request latency grow as you fatten the batch?+- **How to read it:**+ - X axis: batch size, log scaled. Y axis: ingest latency in ms, log scaled.+ - Per (tier, mode) there are three lines: P50 (dotted, alpha 0.5), P95 (dash-dot, alpha 0.7), P99 (solid). Only the P99 line gets a legend entry; lower percentiles are colour-coded by the same colour.+ - `o` markers = step passed. `x` markers = step failed at this batch size.+- **What to look for:**+ - **Cliff knee** — the batch size where the line bends sharply upward. That's where the SUT stops scaling linearly with batch.+ - **Mode spread** — sqlite typically cliffs at ~1K, pgch cruises out to 16K, managed-ch pays a fixed network-RTT floor (visible as a high P50 even at the smallest batch).+- **Why it's good:** Phase 1 saturates one client; the latency curve here is the "no concurrency" baseline that Phase 2/3 charts overlay as the "P1 floor."++### T3 — `chart-phase2-rate-<signal>.png` — Phase 2 collector-shape rate ramp++- **Folder:** `results-throughput/`+- **What it shows:** Three stacked subplots over a shared X axis (request rate, log) for the collector-shape rate ramp (fixed fat batch from Phase 1's winner, default cap 16384).+- **Question answered:** Where does the SUT cliff when a single fat-batch client (like an OTel Collector) ramps its request rate?+- **How to read it:**+ - **Top subplot — throughput.** Solid line = `actualItemsPerSec`. Dashed line = `attemptedItemsPerSec` (= batch × target rate). When solid drops below dashed, the SUT can't keep up — that's the soft-cliff. Log Y.+ - **Middle subplot — latency percentiles.** P50/P95/P99 per run. Thin dotted horizontal lines = **Phase 1 P99 floor at the matching batch size** (the no-concurrency latency). Anything above the floor is concurrency tax. Log Y.+ - **Bottom subplot — failure signal.** Solid line = HTTP error rate (%). Twin Y axis dotted line (if any rejection happened) = OTLP `rejected` count. The horizontal red dashed line at 5% is the configured failure threshold.+ - `o` = passed step, `x` = failed step. Steps are sorted by `requestRate` before plotting because the bisection writer records them in execution order (e.g. 5 → 25 → 15 → 10 → 12).+- **What to look for:**+ - **Soft cliff vs hard fail** — does the SUT degrade gracefully (solid drifts below dashed in the top subplot) or fall off a cliff (errors spike in the bottom subplot at the same rate)?+ - **Concurrency tax** — distance between the latency line and the dotted P1 floor.+ - **Bisection precision** — the X markers should cluster tightly near the cliff. Wide spread means the bisection ran out of steps.+- **Why it's good:** Strictly supersets the legacy `chart-detail-<signal>.png` (which plotted only P99). Adds the achieved-vs-attempted gap, the error breakdown, and the P1 floor — the three signals you need to diagnose a cliff.++### T4 — `chart-phase3-rate-<signal>.png` — Phase 3 SDK-fleet rate ramp (conditional)++- **Folder:** `results-throughput/`+- **What it shows:** Same three-subplot layout as T3, but for the SDK-fleet shape: batch fixed at 100 (typical OTel SDK output), request rates from 10 to 10,000.+- **Question answered:** When each request is tiny and rates are huge, does the SUT cliff in a different place — HTTP/auth/decode path vs raw-insert path?+- **How to read it:** Identical to T3. The X-axis range is much wider (10–10,000 vs ~1–400 in T3) — Phase 3 stresses request handling rather than batch insertion.+- **What to look for:**+ - **Phase ceiling vs Phase 2 ceiling** — Phase 3 typically saturates HTTP/auth before raw insert, so the items/sec ceiling is lower despite higher request rate. If Phase 3's ceiling beats Phase 2's, the SUT is under-utilizing its insert path under fat batches.+ - **Failures** — at thousands of req/sec, decode and auth checks become measurable; spikes there manifest as bottom-subplot errors before the latency lines bend.+- **Why it's good:** Only chart that probes the "thousands of small batches" SDK fleet shape. Different bottleneck profile than the collector shape — surfacing it separately keeps the analysis honest.+- **Note:** This chart is only rendered when at least one run in the folder has a non-empty `phase3` (controlled by the `--phase3-batch-size`/`--phase3-request-rates` loadgen flags).++### T5 — `chart-pareto-<signal>.png` — Latency–throughput Pareto++- **Folder:** `results-throughput/`+- **What it shows:** Scatter of every Phase 2 and Phase 3 step at `(P99 latency, actualItemsPerSec)`. Both axes log. Color per (tier, mode), `o` = passed, `x` = failed.+- **Question answered:** What does each extra K items/sec cost you in P99 latency? What's each (tier, mode) Pareto frontier look like?+- **How to read it:**+ - X axis: P99 latency in ms, log. Y axis: achieved items/sec, log.+ - **Faint dotted polyline per run** connects passed steps sorted by P99 — that's the (tier, mode) frontier.+ - **Up-and-left is better** (more throughput, less latency).+ - **Zero-throughput failures** can't be plotted on a log Y axis — they're listed as a bottom-margin annotation in red (`Zero-throughput failures: ccx23 / sqlite@P2s1; ccx23 / pgch@P2s1`).+- **What to look for:**+ - **Frontier shape** — a flat-then-vertical curve means "lots of room to push more rate before latency rises" (the good case). A steep diagonal means "every K of throughput costs you proportionally in latency" (the bad case).+ - **Mode separation** — sqlite frontiers usually live in the lower-left (low throughput, low latency). pgch frontiers extend right. managed-ch's RTT floor pushes them rightward at the same throughput levels.+ - **Failure clustering** — `x` markers near the frontier top-right tell you what the SUT *attempted* before cliffing.+- **Why it's good:** The only chart that puts the latency-throughput trade-off on a single picture *and* keeps failures visible without crashing on log scale.++### T6 — `chart-tier-scaling-<signal>.png` — Tier scaling++- **Folder:** `results-throughput/`+- **What it shows:** Line: `maxSustainableItemsPerSec` vs hardware tier (ordinal X: ccx13 → ccx43), one line per mode. Tick labels include vCPU/RAM.+- **Question answered:** Does doubling cores actually double throughput? Where does extra hardware stop paying?+- **How to read it:**+ - X axis: tier in linear order (ccx13, ccx23, ccx33, ccx43 — Hetzner doubles vCPU at each step).+ - Y axis: max sustainable items/sec.+ - **Dashed reference line per mode** = perfect linear-vCPU scaling, anchored at the smallest tier present for that mode. If the solid line falls below the dashed, you're paying a scaling penalty.+- **What to look for:**+ - **Linear vs diminishing returns** — sqlite often plateaus quickly (single-process write); pgch usually scales better but with a knee at the largest tier (ClickHouse merges saturate disk I/O).+ - **Missing points** — if a mode has no point at a tier, it failed there. The reference line still projects, giving you a "what perfect scaling would have looked like" anchor.+- **Why it's good:** Distinguishes "you need a bigger box" from "the bottleneck is elsewhere." A flat-line mode says "more cores won't help."++### T7 — `chart-cliff-<signal>.png` — Step-status grid++- **Folder:** `results-throughput/`+- **What it shows:** Gantt-style heatmap. Rows = (tier, mode). Columns = step indices grouped by phase (P1 | P2 | P3 with vertical separators).+- **Question answered:** At a glance, what worked and what failed?+- **How to read it:**+ - Each cell is one step of one phase. Cell color: green=pass, yellow=soft-cliff, red=hard fail, dark grey=0 requests completed, light grey=not run.+ - Inside each cell: `bs=<batch size> r=<request rate>` (e.g. `bs=16K r=25`).+ - **Right margin:** the headline number for the row, or "failed" in red for all-failed runs.+- **What to look for:**+ - **Red rows** — runs that cliffed almost immediately. The old line charts hid these (zero-height bar or invisible single point); the cliff grid makes them legible.+ - **Yellow + red mix** — bisection territory. Several yellow cells around a red one means the loadgen narrowed the cliff via bisection.+ - **Missing-column patterns** — sqlite often has Phase 1 reds and no Phase 2 (because Phase 1 cliffed first, so Phase 2 was skipped). pgch's failures usually live in Phase 2 because Phase 1 cleared all batches.+- **Why it's good:** The most information-dense chart. Especially useful for runs that mostly fail — line charts make those invisible, but the grid shows every step and why.++### T8 — `chart-batch-efficiency-<signal>.png` — Batch efficiency++- **Folder:** `results-throughput/`+- **What it shows:** Phase 1 throughput vs batch size. Solid = `actualItemsPerSec`, dashed = `attemptedItemsPerSec` (= batch × 5 req/sec).+- **Question answered:** Is the batch size still paying off, or is the SUT rejecting fat batches?+- **How to read it:**+ - X axis: batch size, log. Y axis: items/sec, log.+ - Solid line = actually-delivered items/sec. Dashed line per run = attempted items/sec (the diagonal a perfect SUT would track).+ - **Solid below dashed** = the SUT is rejecting items (HTTP errors or OTLP `PartialSuccess.rejected_log_records` etc.).+ - `x` markers = step failed at that batch size.+- **What to look for:**+ - **The point where the solid line peels away from the dashed** — that's the batch size at which more items per request stop helping.+ - **Failed marker placement** — if `x` is exactly on the dashed line, the SUT accepted the work but errored out at the *next* step. If `x` is below the dashed, the SUT was already rejecting before failing.+- **Why it's good:** Disentangles "the load gen sent X" from "the SUT accepted X." Two systems with identical headlines can look very different here.++### T9 — `chart-signal-mix.png` — Cross-signal comparison++- **Folder:** `results-throughput/`+- **What it shows:** Single grouped-bars figure: one group per (tier, mode), three bars per group — spans / metrics / logs.+- **Question answered:** At the same hardware, how does the cost of each signal compare?+- **How to read it:**+ - X axis: (tier × mode) combinations. Y axis: max sustainable items/sec.+ - Bar colour: signal (spans blue, metrics green, logs red).+ - **The Y unit varies by signal** — spans/sec vs data points/sec vs log records/sec aren't strictly comparable item-to-item. Use relative shape rather than absolute height: which signal does this SUT do best on?+- **What to look for:**+ - **Signal ranking per row** — usually metrics > logs > spans (metrics is the cheapest schema, spans the heaviest), but the ratio shifts by mode.+ - **Missing bars** — a signal absent from a (tier, mode) row means that matrix entry failed.+- **Why it's good:** Reveals SUT specialization. A box that's great at spans but bad at logs probably has a string-store bottleneck.++---++## Read-probe charts++### P1 — `chart-readprobe-headline-<signal>.png` — Max fill level by (tier, mode)++- **Folder:** `results-probe/`+- **What it shows:** Grouped bars of `maxFillLevelPassed` (rows) per (tier, mode). Log Y axis.+- **Question answered:** How many rows can you have in the relevant table before the dashboard read on this endpoint cliffs?+- **How to read it:**+ - X axis: hardware tier. Y axis: max passing fill level (rows, log).+ - Each bar is annotated with the fill level (`1.0M`) and the read latency at that fill (`540ms`).+ - **Hatched grey bars** = no fill level passed (the smallest fill level was already over threshold).+ - The title names the read endpoint being probed (e.g. `/api/metrics/application`).+- **What to look for:**+ - **Order-of-magnitude steps** — fill levels usually 100K → 1M → 10M → 100M. A row that maxes at 100K is in serious trouble; 100M is comfortable.+ - **Mode spread** — sqlite usually maxes out an order of magnitude earlier than pgch.+- **Why it's good:** The read-probe equivalent of T1. One number per (tier, mode), with the latency-at-max so you know how close to the threshold the bar represents.++### P2 — `chart-readprobe-<signal>.png` — Read latency vs rows ingested++- **Folder:** `results-probe/`+- **What it shows:** Line: `readLatencyMs` (Y log) vs `rowsIngested` (X log), one line per (tier, mode). Horizontal threshold line (default 5000ms).+- **Question answered:** How steeply does read latency grow as the table fills up?+- **How to read it:**+ - X axis: rows ingested before the probe, log. Y axis: read latency in ms, log.+ - **Solid horizontal red dashed line** = the threshold from `--read-threshold-ms` (default 5000ms).+ - `o` = step passed (read OK, under threshold). `x` = failed (read errored or over threshold).+ - Title names the read path being probed.+- **What to look for:**+ - **Slope** — log-log slope ≈ 1 means linear scaling; > 1 means superlinear (degrading); < 1 means sublinear (the index is doing its job).+ - **Where the line crosses the threshold** — that's the cliff. Anything to the right of the crossing point is unreliable.+- **Why it's good:** The classic "how does the dashboard hold up?" picture. Slope information is the part you couldn't get from the headline.++### P3 — `chart-readprobe-ingest-time-<signal>.png` — Fill time vs target rows++- **Folder:** `results-probe/`+- **What it shows:** Line: time taken to reach the target fill level (Y log seconds) vs target fill level (X log rows), one line per (tier, mode).+- **Question answered:** How long does it take to FILL each level — i.e. how expensive are the writes at high cardinality?+- **How to read it:**+ - X axis: target fill level (rows, log). Y axis: ingest seconds elapsed (log).+ - Lines are read in execution order — they keep climbing as the test fills the table.+- **What to look for:**+ - **Curve sag** — if the curve bends upward at higher row counts, write cost is rising (compaction / index thrash). A perfectly linear log-log line means write speed is constant regardless of table size.+ - **Mode delta** — ClickHouse usually sags around 10–100M because background merges saturate the disk; SQLite usually fails before sagging at all.+- **Why it's good:** The read-probe view focuses on read latency, but the cost of *getting there* matters too. If filling 100M rows takes hours, that's a different operational story than seconds.++### P4 — `chart-readprobe-tier-scaling-<signal>.png` — Read-probe tier scaling++- **Folder:** `results-probe/`+- **What it shows:** Line: `maxFillLevelPassed` vs tier ordinal, one line per mode. Log Y. Dashed linear-vCPU reference per mode.+- **Question answered:** Does more hardware buy you a bigger queryable dataset on this endpoint?+- **How to read it:** Mirror of T6 but Y is rows instead of items/sec. Same linear-vCPU reference convention.+- **What to look for:**+ - **Plateau** — flat lines mean read scaling is bottlenecked elsewhere (network RTT for managed-ch, single-thread query plan for sqlite).+ - **Below-reference gap** — distance between solid and dashed = how much the read path leaves on the table.+- **Why it's good:** Throughput tier-scaling answers "can I serve more SDK traffic?". This one answers "can I keep the dashboard working at a bigger scale?". Different question, same chart shape.++### P5 — `chart-readprobe-cliff-<signal>.png` — Read-probe step-status grid++- **Folder:** `results-probe/`+- **What it shows:** Gantt-style heatmap. Rows = (tier, mode). Columns = fill levels (union across all runs).+- **Question answered:** Which (tier × mode × fill level) combos passed?+- **How to read it:**+ - Each cell is one probe of one fill level. Cell color: green=passed, yellow=read OK but over threshold, red=read errored, light-grey=not run.+ - Inside each cell: the read latency (`540ms`). Empty when not run.+ - Right margin: `max <fill>` if any fill passed, or "no fill passed" in red.+ - Column headers show the fill levels themselves (`100K`, `1.0M`, etc.).+- **What to look for:**+ - **Yellow vs red** — yellow means "the read came back but slowly," red means "the read errored out." Yellow is often actionable (tune the query plan); red usually means the SUT itself is unwell.+ - **Row patterns** — sqlite rows often have a single green cell and the rest red/yellow; pgch rows usually have a few green cells before yellow.+- **Why it's good:** The cliff equivalent of T7 but for read-probe. Lets you see the failure pattern across the full matrix on one page.++### P6 — `chart-readprobe-signal-mix.png` — Read-probe cross-signal++- **Folder:** `results-probe/`+- **What it shows:** Single grouped-bars figure: one group per (tier, mode), three bars per group — spans / metrics / logs. Y = `maxFillLevelPassed` (log).+- **Question answered:** Which read endpoint scales worst on the same hardware?+- **How to read it:**+ - Same shape as T9 but for read-probe.+ - Legend includes the read path for each signal (`spans (/api/endpoints/grouped)`), since each signal probes a different endpoint.+ - Log Y because fill levels span several orders of magnitude.+- **What to look for:**+ - **Signal worst-case** — usually `spans` (`/api/endpoints/grouped` joins on a high-cardinality table) is the first to cliff.+ - **Mode-invariant cliffs** — if every mode's bar for `spans` is short, the bottleneck is the query plan, not the storage engine.+- **Why it's good:** Three read paths × N (tier, mode) cells on one image. Shows you which API endpoint to optimize first.++---++## Generating these locally++```bash+python3 -m venv .venv-bench+.venv-bench/bin/pip install matplotlib++# Throughput results+.venv-bench/bin/python benchmarks/scripts/chart.py benchmarks/results-throughput++# Read-probe results+.venv-bench/bin/python benchmarks/scripts/chart.py benchmarks/results-probe+```++`chart.py` auto-detects scenarios per JSON file — running it against a folder+that mixes both scenarios will produce the union of charts. The wrong-scenario+render functions are no-ops, so you'll just see the charts that have data.diff --git a/benchmarks/scripts/chart.py b/benchmarks/scripts/chart.pyindex b141827a..aef98509 100755--- a/benchmarks/scripts/chart.py+++ b/benchmarks/scripts/chart.py@@ -1,18 +1,16 @@#!/usr/bin/env python3-"""Render per-signal benchmark charts from a directory of loadgen JSONs.+"""Render benchmark charts from a directory of loadgen JSONs.Inputs: a directory containing N JSON files (one per matrix entry), each inthe schema emitted by benchmarks/loadgen/. Files without a "signal" field(old pre-OTLP-split format) are skipped with a log line.-Outputs depend on which scenarios are present in the directory:- Throughput scenario:- - chart-spans.png / chart-metrics.png / chart-logs.png bar: max items/sec- - chart-detail-<sig>.png line: ingest p99 vs request rate- Read-probe scenario:- - chart-readprobe-<sig>.png line: read latency vs fill level- Always:- - summary.md tables per signal/scenario+The script auto-detects scenarios per file (each JSON has `scenario`). Run it+against `benchmarks/results-throughput/` for throughput charts, or+`benchmarks/results-probe/` for read-probe charts. Mixed folders also work —+each render function filters to its own scenario.++See benchmarks/charts.md for a reader's guide to each output."""import json@@ -23,11 +21,25 @@ import matplotlibmatplotlib.use("Agg")import matplotlib.pyplot as plt+from matplotlib.patches import Patch, Rectangle+TIER_ORDER = ["ccx13", "ccx23", "ccx33", "ccx43"]-MODE_ORDER = ["sqlite", "pgch"]+MODE_ORDER = ["sqlite", "pgch", "managed-ch"]SIGNALS = ["spans", "metrics", "logs"]-MODE_COLORS = {"sqlite": "#4f9fff", "pgch": "#ff9f4f"}++MODE_COLORS = {+ "sqlite": "#4f9fff",+ "pgch": "#ff9f4f",+ "managed-ch": "#7c4fff",+}+SIGNAL_COLORS = {+ "spans": "#1f77b4",+ "metrics": "#2ca02c",+ "logs": "#d62728",+}+PHASE_LABEL = {"phase1": "P1", "phase2": "P2", "phase3": "P3"}+ITEM_LABEL = {"spans": "spans/sec","metrics": "data points/sec",@@ -38,6 +50,28 @@ ROW_LABEL = {"metrics": "metric_points rows","logs": "log_records rows",}+TIER_META = {+ "ccx13": "2 vCPU / 8 GB",+ "ccx23": "4 vCPU / 16 GB",+ "ccx33": "8 vCPU / 32 GB",+ "ccx43": "16 vCPU / 64 GB",+}+TIER_VCPU = {"ccx13": 2, "ccx23": 4, "ccx33": 8, "ccx43": 16}++# Match loadgen defaults so chart thresholds line up with the JSON's `passed` bit.+SOFT_CLIFF_RATIO = 0.7+ERR_THRESHOLD_PCT = 5.0++STATUS_COLOR = {+ "pass": "#4caf50",+ "soft-cliff": "#ffc107",+ "hard-fail": "#f44336",+ "dead": "#616161",+ "missing": "#f5f5f5",+}+++# ---------- Loading + small helpers ----------def load_runs(results_dir: Path) -> list[dict]:@@ -68,71 +102,612 @@ def mode_rank(m: str) -> int:return MODE_ORDER.index(m) if m in MODE_ORDER else len(MODE_ORDER)-def render_throughput_bar(runs: list[dict], signal: str, out: Path) -> None:- runs = [r for r in runs if r["signal"] == signal and r["scenario"] == "throughput"]- if not runs:+def mode_color(m: str) -> str:+ return MODE_COLORS.get(m, "#777")+++def read_path_for_signal(signal: str) -> str:+ return {+ "spans": "/api/endpoints/grouped",+ "metrics": "/api/metrics/application",+ "logs": "/api/logs",+ }.get(signal, "")+++def fmt_count(n: float) -> str:+ n = int(n or 0)+ if n >= 1_000_000_000:+ return f"{n/1_000_000_000:.1f}B"+ if n >= 1_000_000:+ return f"{n/1_000_000:.1f}M"+ if n >= 1_000:+ return f"{n/1_000:.0f}K"+ return f"{n}"+++def run_label(run: dict) -> str:+ return f"{run['tier']} / {run['mode']}"+++def iter_runs(runs: list[dict], signal: str, scenario: str) -> list[dict]:+ out = [r for r in runs if r["signal"] == signal and r.get("scenario", "throughput") == scenario]+ return sorted(out, key=lambda r: (tier_rank(r["tier"]), mode_rank(r["mode"])))+++def phase_steps(run: dict, phase_key: str, sort_by: str = "step") -> list[dict]:+ """Return phase steps in a chosen sort order. Phase 2/3 record execution+ order, not rate order, so sorting before plotting keeps lines monotonic."""+ phase = run.get(phase_key) or {}+ steps = phase.get("steps") or []+ if not steps:+ return []+ if sort_by == "rate":+ return sorted(steps, key=lambda s: (s.get("requestRate", 0), s.get("step", 0)))+ if sort_by == "batch":+ return sorted(steps, key=lambda s: (s.get("batchSize", 0), s.get("step", 0)))+ return sorted(steps, key=lambda s: s.get("step", 0))+++def step_status(step: dict) -> str:+ """Categorize a step as pass / soft-cliff / hard-fail / dead.+ `pass` = loadgen marked it passed.+ `dead` = no requests completed (actual=0 or attempted=0).+ `soft-cliff` = achieved fell below SOFT_CLIFF_RATIO × attempted with no error spike.+ `hard-fail` = combined error rate above threshold (default 5%).+ """+ if step.get("passed"):+ return "pass"+ actual = step.get("actualItemsPerSec", 0) or 0+ attempted = step.get("attemptedItemsPerSec", 0) or 0+ err_rate_pct = ((step.get("ingest") or {}).get("errRate", 0) or 0) * 100+ if attempted == 0 or actual == 0:+ return "dead"+ if err_rate_pct > ERR_THRESHOLD_PCT:+ return "hard-fail"+ if attempted > 0 and actual / attempted < SOFT_CLIFF_RATIO:+ return "soft-cliff"+ return "hard-fail"+++def headline_source(run: dict) -> tuple[str, dict] | None:+ """Return (phase_key, step) whose actualItemsPerSec equals the headline.+ Returns None if the run has no passing step."""+ headline = run.get("maxSustainableItemsPerSec", 0) or 0+ if headline <= 0:+ return None+ best = None+ best_delta = float("inf")+ for phase_key in ("phase1", "phase2", "phase3"):+ for step in phase_steps(run, phase_key):+ if not step.get("passed"):+ continue+ actual = step.get("actualItemsPerSec", 0) or 0+ delta = abs(actual - headline)+ if delta < best_delta:+ best = (phase_key, step)+ best_delta = delta+ return best+++def first_failure(run: dict) -> tuple[str, dict] | None:+ for phase_key in ("phase1", "phase2", "phase3"):+ for step in phase_steps(run, phase_key):+ if not step.get("passed"):+ return (phase_key, step)+ return None+++def phase1_p99_at_batch(run: dict, batch_size: int) -> float | None:+ for step in phase_steps(run, "phase1"):+ if step.get("batchSize") == batch_size and step.get("passed"):+ return (step.get("ingest") or {}).get("p99")+ return None+++# ---------- Throughput charts ----------+++def render_headline_bar(runs: list[dict], signal: str, out: Path) -> None:+ sig_runs = iter_runs(runs, signal, "throughput")+ if not sig_runs:return- fig, ax = plt.subplots(figsize=(9, 5))- tiers_present = sorted({r["tier"] for r in runs}, key=tier_rank)- modes_present = sorted({r["mode"] for r in runs}, key=mode_rank)+ tiers_present = sorted({r["tier"] for r in sig_runs}, key=tier_rank)+ modes_present = sorted({r["mode"] for r in sig_runs}, key=mode_rank)+ if not tiers_present or not modes_present:+ return++ fig, ax = plt.subplots(figsize=(max(8, len(tiers_present) * 2.5), 6))++ values = [r.get("maxSustainableItemsPerSec", 0) or 0 for r in sig_runs]+ ymax = max(values) if values else 1+ ghost_height = max(ymax * 0.02, 1)width = 0.8 / max(len(modes_present), 1)- x = list(range(len(tiers_present)))+ x_centers = list(range(len(tiers_present)))for mi, mode in enumerate(modes_present):- ys = []- for tier in tiers_present:- match = [r for r in runs if r["tier"] == tier and r["mode"] == mode]- ys.append(match[0].get("maxSustainableItemsPerSec", 0) if match else 0)- offsets = [xi + (mi - (len(modes_present) - 1) / 2) * width for xi in x]- bars = ax.bar(offsets, ys, width=width, label=mode, color=MODE_COLORS.get(mode, "#777"))- for b, y in zip(bars, ys):- ax.annotate(f"{int(y):,}", xy=(b.get_x() + b.get_width() / 2, y),- xytext=(0, 3), textcoords="offset points",- ha="center", va="bottom", fontsize=9)+ for ti, tier in enumerate(tiers_present):+ match = [r for r in sig_runs if r["tier"] == tier and r["mode"] == mode]+ x = ti + (mi - (len(modes_present) - 1) / 2) * width+ if not match:+ continue+ run = match[0]+ value = run.get("maxSustainableItemsPerSec", 0) or 0+ if value > 0:+ ax.bar(x, value, width=width, color=mode_color(mode))+ src = headline_source(run)+ phase_tag = f" ({PHASE_LABEL[src[0]]})" if src else ""+ ax.annotate(+ f"{int(value):,}{phase_tag}",+ xy=(x, value), xytext=(0, 4), textcoords="offset points",+ ha="center", va="bottom", fontsize=8,+ )+ else:+ ax.bar(x, ghost_height, width=width, color="#dddddd",+ edgecolor="#999999", hatch="///", linewidth=0.7)+ fail = first_failure(run)+ if fail:+ pkey, step = fail+ label = f"failed @ {PHASE_LABEL[pkey]} step {step.get('step', '?')}"+ else:+ label = "failed"+ ax.annotate(+ label, xy=(x, ghost_height), xytext=(0, 4), textcoords="offset points",+ ha="center", va="bottom", fontsize=7, color="#aa3333",+ )- ax.set_xticks(x)+ ax.set_xticks(x_centers)ax.set_xticklabels(tiers_present)ax.set_ylabel(f"Max sustainable {ITEM_LABEL[signal]}")- ax.set_title(f"Traceway: max sustainable {signal} ingest by hardware tier\n"- "(failure = combined HTTP error + OTLP rejected > 5%)")- ax.legend(title="DB mode")+ ax.set_title(+ f"Traceway: max sustainable {signal} ingest by hardware tier\n"+ "Phase tag (P1/P2/P3) shows which workload shape produced the headline"+ )++ legend_handles = [Patch(facecolor=mode_color(m), label=m) for m in modes_present]+ legend_handles.append(Patch(facecolor="#dddddd", edgecolor="#999999", hatch="///", label="all phases failed"))+ ax.legend(handles=legend_handles, title="DB mode", loc="upper left")+ax.grid(axis="y", linestyle=":", alpha=0.4)+ ax.set_ylim(0, ymax * 1.3 if ymax > 0 else 1)+fig.tight_layout()fig.savefig(out, dpi=130)plt.close(fig)-def render_throughput_detail(runs: list[dict], signal: str, out: Path) -> None:- runs = [r for r in runs if r["signal"] == signal and r["scenario"] == "throughput"]- if not runs:+def render_phase1_latency(runs: list[dict], signal: str, out: Path) -> None:+ sig_runs = [r for r in iter_runs(runs, signal, "throughput") if phase_steps(r, "phase1")]+ if not sig_runs:+ return++ fig, ax = plt.subplots(figsize=(11, 6))+ plotted = False++ for run in sig_runs:+ color = mode_color(run["mode"])+ steps = phase_steps(run, "phase1", sort_by="batch")+ batches = [s["batchSize"] for s in steps]+ p50 = [(s.get("ingest") or {}).get("p50", 0) for s in steps]+ p95 = [(s.get("ingest") or {}).get("p95", 0) for s in steps]+ p99 = [(s.get("ingest") or {}).get("p99", 0) for s in steps]+ passed = [bool(s.get("passed")) for s in steps]++ def _safe(xs, ys):+ return [(x, y) for x, y in zip(xs, ys) if y and y > 0]++ for percs, ls, alpha in [(p50, ":", 0.5), (p95, "-.", 0.7), (p99, "-", 1.0)]:+ pts = _safe(batches, percs)+ if not pts:+ continue+ xs, ys = zip(*pts)+ label = run_label(run) if ls == "-" else None+ ax.plot(xs, ys, linestyle=ls, color=color, alpha=alpha, label=label)+ plotted = True++ for b, lat, p in zip(batches, p99, passed):+ if lat and lat > 0:+ ax.plot(b, lat,+ marker=("o" if p else "x"),+ color=color, markersize=8,+ markeredgewidth=2 if not p else 1, linestyle="")++ if not plotted:+ plt.close(fig)+ return++ ax.set_xlabel("Batch size (items/request, log)")+ ax.set_ylabel("Ingest latency (ms, log)\nP50 ·· P95 -· P99 —")+ ax.set_xscale("log")+ ax.set_yscale("log")+ ax.set_title(+ f"Phase 1 — batch-size scaling — {signal}\n"+ "Fixed 5 req/sec. Solid line = P99 per run, dot/dash = P50/P95. × = step failed."+ )+ ax.grid(True, which="both", linestyle=":", alpha=0.4)+ ax.legend(loc="upper left", fontsize=9, title="P99 line per run")+ fig.tight_layout()+ fig.savefig(out, dpi=130)+ plt.close(fig)+++def _render_rate_ramp(runs: list[dict], signal: str, phase_key: str, subtitle: str, out: Path) -> None:+ sig_runs = [r for r in iter_runs(runs, signal, "throughput") if phase_steps(r, phase_key)]+ if not sig_runs:+ return++ fig, (ax1, ax2, ax3) = plt.subplots(3, 1, figsize=(11, 12), sharex=True)+ plotted = False++ for run in sig_runs:+ color = mode_color(run["mode"])+ label = run_label(run)+ steps = phase_steps(run, phase_key, sort_by="rate")++ rates = [s["requestRate"] for s in steps]+ actual = [s["actualItemsPerSec"] for s in steps]+ attempted = [s["attemptedItemsPerSec"] for s in steps]+ p50 = [(s.get("ingest") or {}).get("p50", 0) for s in steps]+ p95 = [(s.get("ingest") or {}).get("p95", 0) for s in steps]+ p99 = [(s.get("ingest") or {}).get("p99", 0) for s in steps]+ errs_pct = [((s.get("ingest") or {}).get("errRate", 0) or 0) * 100 for s in steps]+ passed = [bool(s.get("passed")) for s in steps]++ # Top: throughput. Drop zeros for log scale.+ act_pts = [(r, a) for r, a in zip(rates, actual) if a and a > 0]+ att_pts = [(r, a) for r, a in zip(rates, attempted) if a and a > 0]+ if act_pts:+ xs, ys = zip(*act_pts)+ ax1.plot(xs, ys, linestyle="-", color=color, label=label)+ plotted = True+ if att_pts:+ xs, ys = zip(*att_pts)+ ax1.plot(xs, ys, linestyle="--", color=color, alpha=0.4)+ for r, a, p in zip(rates, actual, passed):+ if a and a > 0:+ ax1.plot(r, a, marker=("o" if p else "x"), color=color,+ markersize=8, markeredgewidth=2 if not p else 1, linestyle="")++ # Middle: latency percentiles.+ for percs, ls, alpha in [(p50, ":", 0.5), (p95, "-.", 0.7), (p99, "-", 1.0)]:+ pts = [(r, l) for r, l in zip(rates, percs) if l and l > 0]+ if pts:+ xs, ys = zip(*pts)+ ax2.plot(xs, ys, linestyle=ls, color=color, alpha=alpha)+ # P1 floor overlay+ fixed_batch = (run.get(phase_key) or {}).get("fixedBatchSize", 0)+ if fixed_batch:+ floor = phase1_p99_at_batch(run, fixed_batch)+ if floor:+ ax2.axhline(floor, linestyle=":", color=color, alpha=0.35, linewidth=1)+ for r, lat, p in zip(rates, p99, passed):+ if lat and lat > 0:+ ax2.plot(r, lat, marker=("o" if p else "x"), color=color,+ markersize=8, markeredgewidth=2 if not p else 1, linestyle="")++ # Bottom: error %+ ax3.plot(rates, errs_pct, linestyle="-", color=color, alpha=0.5)+ for r, e, p in zip(rates, errs_pct, passed):+ ax3.plot(r, e, marker=("o" if p else "x"), color=color,+ markersize=8, markeredgewidth=2 if not p else 1, linestyle="")++ # Bottom right twin: rejected+ rej = [s.get("rejected", 0) for s in steps]+ if any(rej):+ if not hasattr(ax3, "_rej_twin"):+ ax3._rej_twin = ax3.twinx()+ ax3._rej_twin.plot(rates, rej, linestyle=":", color=color, alpha=0.4)++ if not plotted:+ plt.close(fig)+ return++ ax1.set_ylabel(f"items/sec (log)\n— actual, -- attempted")+ ax1.set_xscale("log")+ ax1.set_yscale("log")+ ax1.grid(True, which="both", linestyle=":", alpha=0.4)+ ax1.legend(loc="lower right", fontsize=8)++ ax2.set_ylabel("Ingest latency (ms, log)\nP50 ·· P95 -· P99 —\n(thin dotted = P1 floor)")+ ax2.set_yscale("log")+ ax2.grid(True, which="both", linestyle=":", alpha=0.4)++ ax3.axhline(ERR_THRESHOLD_PCT, linestyle="--", color="#cc3333", alpha=0.5, linewidth=1)+ ax3.set_ylabel("Error rate (%)\n× = failed step")+ ax3.set_xlabel("Request rate (req/sec, log)")+ ax3.set_xscale("log")+ ax3.grid(True, which="both", linestyle=":", alpha=0.4)+ if hasattr(ax3, "_rej_twin"):+ ax3._rej_twin.set_ylabel("OTLP rejected items (dotted)", color="#666666")++ fig.suptitle(f"{subtitle} — {signal}\n× marker = failed step (err > 5% or soft-cliff). Dotted P1 floor = no-concurrency latency.")+ fig.tight_layout(rect=(0, 0, 1, 0.97))+ fig.savefig(out, dpi=130)+ plt.close(fig)+++def render_phase2_rate(runs: list[dict], signal: str, out: Path) -> None:+ _render_rate_ramp(runs, signal, "phase2",+ "Phase 2 — collector-shape rate ramp (fixed fat batch)", out)+++def render_phase3_rate(runs: list[dict], signal: str, out: Path) -> None:+ # Only render when at least one run has phase3 data.+ any_p3 = any(phase_steps(r, "phase3") for r in iter_runs(runs, signal, "throughput"))+ if not any_p3:+ return+ _render_rate_ramp(runs, signal, "phase3",+ "Phase 3 — SDK-fleet rate ramp (small batch, high rate)", out)+++def render_pareto(runs: list[dict], signal: str, out: Path) -> None:+ sig_runs = iter_runs(runs, signal, "throughput")+ if not sig_runs:+ return++ fig, ax = plt.subplots(figsize=(11, 7))+ plotted = False+ dead_runs = [] # (run_label, list_of_phase_keys+step_n)++ for run in sig_runs:+ color = mode_color(run["mode"])+ label = run_label(run)++ passed_pts = []+ failed_pts = []+ for phase_key in ("phase2", "phase3"):+ for step in phase_steps(run, phase_key):+ p99 = (step.get("ingest") or {}).get("p99", 0)+ actual = step.get("actualItemsPerSec", 0) or 0+ if p99 <= 0 or actual <= 0:+ if step.get("passed") is False:+ dead_runs.append((label, f"{PHASE_LABEL[phase_key]}s{step.get('step', '?')}"))+ continue+ if step.get("passed"):+ passed_pts.append((p99, actual))+ else:+ failed_pts.append((p99, actual))++ if passed_pts:+ sorted_pts = sorted(passed_pts, key=lambda p: p[0])+ xs, ys = zip(*sorted_pts)+ ax.plot(xs, ys, linestyle=":", color=color, alpha=0.4)+ ax.scatter(xs, ys, marker="o", color=color, s=70, edgecolor="black",+ linewidth=0.5, label=label, zorder=3)+ plotted = True+ if failed_pts:+ xs, ys = zip(*failed_pts)+ ax.scatter(xs, ys, marker="x", color=color, s=100, linewidth=2, zorder=4)+ plotted = True++ if not plotted and not dead_runs:+ plt.close(fig)+ return++ ax.set_xlabel("Ingest P99 latency (ms, log)")+ ax.set_ylabel(f"Achieved {ITEM_LABEL[signal]} (log)")+ if plotted:+ ax.set_xscale("log")+ ax.set_yscale("log")+ ax.set_title(+ f"Latency–throughput Pareto — {signal}\n"+ "Every Phase 2/3 step. ○ = passed, × = failed. Up-and-left is better."+ )+ ax.grid(True, which="both", linestyle=":", alpha=0.4)++ # Annotate runs that had only dead (0-actual) failures+ if dead_runs:+ dead_by_run: dict[str, list[str]] = {}+ for lab, key in dead_runs:+ dead_by_run.setdefault(lab, []).append(key)+ text = "Zero-throughput failures: " + "; ".join(+ f"{lab}@{','.join(keys)}" for lab, keys in dead_by_run.items()+ )+ ax.text(0.5, -0.15, text, transform=ax.transAxes, ha="center", va="top",+ fontsize=8, color="#aa3333")+ fig.subplots_adjust(bottom=0.18)++ if plotted:+ ax.legend(loc="upper left", fontsize=9)+ fig.tight_layout()+ fig.savefig(out, dpi=130)+ plt.close(fig)+++def render_tier_scaling(runs: list[dict], signal: str, out: Path) -> None:+ sig_runs = iter_runs(runs, signal, "throughput")+ if not sig_runs:+ return++ tiers_present = sorted({r["tier"] for r in sig_runs}, key=tier_rank)+ modes_present = sorted({r["mode"] for r in sig_runs}, key=mode_rank)+ if not tiers_present:returnfig, ax = plt.subplots(figsize=(10, 6))plotted = False+ x_positions = list(range(len(tiers_present)))- for run in sorted(runs, key=lambda r: (tier_rank(r["tier"]), mode_rank(r["mode"]))):- steps = (run.get("phase2") or {}).get("steps") or []- # Bisection appends steps in execution order, not rate order. Sort- # by requestRate so the line draws monotonically across the X axis.- steps = sorted([s for s in steps if s.get("ingest")], key=lambda s: s.get("requestRate", 0))- xs = [s["requestRate"] for s in steps]- ys = [s["ingest"]["p99"] for s in steps]+ for mode in modes_present:+ xs, ys = [], []+ for i, tier in enumerate(tiers_present):+ match = [r for r in sig_runs if r["tier"] == tier and r["mode"] == mode]+ if not match:+ continue+ value = match[0].get("maxSustainableItemsPerSec", 0) or 0+ if value <= 0:+ continue+ xs.append(i)+ ys.append(value)if not xs:continue- ax.plot(xs, ys, marker="o", label=f"{run['tier']} / {run['mode']}")+ ax.plot(xs, ys, marker="o", color=mode_color(mode), markersize=9,+ linewidth=2, label=mode)+ for x, y in zip(xs, ys):+ ax.annotate(fmt_count(y), xy=(x, y), xytext=(0, 8),+ textcoords="offset points", ha="center", va="bottom", fontsize=9)+ # Linear-vCPU reference, anchored at the smallest tier of THIS mode+ base_x = xs[0]+ base_y = ys[0]+ base_vcpu = TIER_VCPU.get(tiers_present[base_x], 1)+ ref_xs = list(range(len(tiers_present)))+ ref_ys = [base_y * (TIER_VCPU.get(tiers_present[i], 1) / base_vcpu) for i in ref_xs]+ ax.plot(ref_xs, ref_ys, linestyle="--", color=mode_color(mode), alpha=0.3, linewidth=1)plotted = Trueif not plotted:plt.close(fig)return- ax.set_xlabel("Phase 2 request rate (req/sec)")- ax.set_ylabel("Ingest P99 latency (ms)")- ax.set_title(f"Ingest latency vs request rate — {signal}")+ ax.set_xticks(x_positions)+ ax.set_xticklabels([f"{t}\n{TIER_META.get(t, '')}" for t in tiers_present])+ ax.set_ylabel(f"Max sustainable {ITEM_LABEL[signal]}")+ ax.set_title(+ f"Tier scaling — {signal}\n"+ "Dashed = linear-vCPU reference (anchored at smallest present tier). "+ "Distance below dashed = scaling penalty."+ )+ ax.legend(title="DB mode", loc="upper left")+ ax.grid(True, linestyle=":", alpha=0.4)+ fig.tight_layout()+ fig.savefig(out, dpi=130)+ plt.close(fig)+++def render_cliff_grid(runs: list[dict], signal: str, out: Path) -> None:+ sig_runs = iter_runs(runs, signal, "throughput")+ if not sig_runs:+ return++ n_p1 = max((len(phase_steps(r, "phase1")) for r in sig_runs), default=0)+ n_p2 = max((len(phase_steps(r, "phase2")) for r in sig_runs), default=0)+ n_p3 = max((len(phase_steps(r, "phase3")) for r in sig_runs), default=0)+ total_cols = n_p1 + n_p2 + n_p3+ if total_cols == 0:+ return++ n_rows = len(sig_runs)+ fig_w = max(11, total_cols * 1.55 + 4)+ fig_h = max(4.5, n_rows * 0.95 + 2.2)+ fig, ax = plt.subplots(figsize=(fig_w, fig_h))++ for ri, run in enumerate(sig_runs):+ y = n_rows - ri - 1+ ax.text(-0.5, y + 0.5, run_label(run), ha="right", va="center", fontsize=10, fontweight="bold")++ col_x = 0+ for pk, n_cols in [("phase1", n_p1), ("phase2", n_p2), ("phase3", n_p3)]:+ steps = phase_steps(run, pk, sort_by="step")+ for ci in range(n_cols):+ x = col_x + ci+ if ci < len(steps):+ step = steps[ci]+ color = STATUS_COLOR[step_status(step)]+ bs = step.get("batchSize", 0)+ r = step.get("requestRate", 0)+ rect = Rectangle((x, y), 1, 1, facecolor=color,+ edgecolor="white", linewidth=1.5)+ ax.add_patch(rect)+ ax.text(x + 0.5, y + 0.5, f"bs={fmt_count(bs)}\nr={r:g}",+ ha="center", va="center", fontsize=7.5)+ else:+ rect = Rectangle((x, y), 1, 1, facecolor=STATUS_COLOR["missing"],+ edgecolor="white", linewidth=1.5)+ ax.add_patch(rect)+ col_x += n_cols++ headline = run.get("maxSustainableItemsPerSec", 0) or 0+ margin = total_cols + 0.35+ if headline > 0:+ ax.text(margin, y + 0.5, f"{fmt_count(headline)}/s",+ ha="left", va="center", fontsize=10, fontweight="bold")+ else:+ ax.text(margin, y + 0.5, "failed",+ ha="left", va="center", fontsize=10, color="#cc3333", fontweight="bold")++ # Phase headers + separators+ headers = []+ if n_p1 > 0:+ headers.append((n_p1 / 2.0, "Phase 1 (batch ramp)"))+ if n_p2 > 0:+ headers.append((n_p1 + n_p2 / 2.0, "Phase 2 (collector rate ramp)"))+ if n_p3 > 0:+ headers.append((n_p1 + n_p2 + n_p3 / 2.0, "Phase 3 (SDK-fleet rate ramp)"))+ for cx, label in headers:+ ax.text(cx, n_rows + 0.25, label, ha="center", va="bottom",+ fontsize=10.5, fontweight="bold")++ for sep in [n_p1, n_p1 + n_p2]:+ if 0 < sep < total_cols:+ ax.plot([sep, sep], [0, n_rows], color="black", linewidth=2)++ ax.set_xlim(-4, total_cols + 5)+ ax.set_ylim(-0.6, n_rows + 0.9)+ ax.set_aspect("equal")+ ax.axis("off")++ legend = [+ Patch(facecolor=STATUS_COLOR["pass"], label="pass"),+ Patch(facecolor=STATUS_COLOR["soft-cliff"], label=f"soft-cliff (<{int(SOFT_CLIFF_RATIO*100)}% achieved)"),+ Patch(facecolor=STATUS_COLOR["hard-fail"], label=f"hard fail (err > {ERR_THRESHOLD_PCT:.0f}%)"),+ Patch(facecolor=STATUS_COLOR["dead"], label="0 requests completed"),+ Patch(facecolor=STATUS_COLOR["missing"], edgecolor="#cccccc", label="not run"),+ ]+ ax.legend(handles=legend, loc="lower center", bbox_to_anchor=(0.5, -0.08),+ ncol=5, fontsize=9, frameon=False)+ fig.suptitle(f"Step-status grid — {signal}\nbs = batch size, r = request rate (req/sec)",+ fontsize=12)+ fig.tight_layout(rect=(0, 0.04, 1, 0.95))+ fig.savefig(out, dpi=130, bbox_inches="tight")+ plt.close(fig)+++def render_batch_efficiency(runs: list[dict], signal: str, out: Path) -> None:+ sig_runs = [r for r in iter_runs(runs, signal, "throughput") if phase_steps(r, "phase1")]+ if not sig_runs:+ return++ fig, ax = plt.subplots(figsize=(11, 6))+ plotted = False++ for run in sig_runs:+ color = mode_color(run["mode"])+ steps = phase_steps(run, "phase1", sort_by="batch")+ batches = [s["batchSize"] for s in steps]+ actual = [s["actualItemsPerSec"] for s in steps]+ attempted = [s["attemptedItemsPerSec"] for s in steps]+ passed = [bool(s.get("passed")) for s in steps]++ # log scale; drop zeros+ act_pts = [(b, a) for b, a in zip(batches, actual) if a and a > 0]+ att_pts = [(b, a) for b, a in zip(batches, attempted) if a and a > 0]++ if act_pts:+ xs, ys = zip(*act_pts)+ ax.plot(xs, ys, linestyle="-", color=color, marker="o", label=run_label(run))+ plotted = True+ if att_pts:+ xs, ys = zip(*att_pts)+ ax.plot(xs, ys, linestyle="--", color=color, alpha=0.35)++ for b, a, p in zip(batches, actual, passed):+ if not p and a and a > 0:+ ax.plot(b, a, marker="x", color=color, markersize=11,+ markeredgewidth=2.2, linestyle="")++ if not plotted:+ plt.close(fig)+ return++ ax.set_xlabel("Batch size (items/request, log)")+ ax.set_ylabel(f"{ITEM_LABEL[signal]} (log)\n— actual, -- attempted (batch × 5 req/s)")ax.set_xscale("log")ax.set_yscale("log")+ ax.set_title(+ f"Batch efficiency — {signal}\n"+ "Phase 1 at fixed 5 req/sec. Solid below dashed = SUT dropping batches."+ )ax.grid(True, which="both", linestyle=":", alpha=0.4)ax.legend(loc="upper left", fontsize=9)fig.tight_layout()@@ -140,24 +715,151 @@ def render_throughput_detail(runs: list[dict], signal: str, out: Path) -> None:plt.close(fig)+def render_signal_mix(runs: list[dict], out: Path) -> None:+ """Cross-signal throughput comparison: bars per (tier, mode), one bar per signal."""+ thr_runs = [r for r in runs if r.get("scenario", "throughput") == "throughput"]+ if not thr_runs:+ return++ keys = sorted({(r["tier"], r["mode"]) for r in thr_runs},+ key=lambda k: (tier_rank(k[0]), mode_rank(k[1])))+ if not keys:+ return++ fig, ax = plt.subplots(figsize=(max(8, len(keys) * 1.9), 6))++ width = 0.25+ x = list(range(len(keys)))+ any_value = False++ for si, signal in enumerate(SIGNALS):+ ys = []+ for tier, mode in keys:+ match = [r for r in thr_runs if r["tier"] == tier and r["mode"] == mode and r["signal"] == signal]+ ys.append((match[0].get("maxSustainableItemsPerSec", 0) or 0) if match else 0)+ offsets = [xi + (si - 1) * width for xi in x]+ bars = ax.bar(offsets, ys, width=width, color=SIGNAL_COLORS[signal], label=signal)+ for b, y in zip(bars, ys):+ if y > 0:+ ax.annotate(fmt_count(y), xy=(b.get_x() + b.get_width() / 2, y),+ xytext=(0, 3), textcoords="offset points",+ ha="center", va="bottom", fontsize=7)+ any_value = True++ if not any_value:+ plt.close(fig)+ return++ ax.set_xticks(x)+ ax.set_xticklabels([f"{t}\n{m}" for t, m in keys], fontsize=9)+ ax.set_ylabel("Max sustainable items/sec (Y unit varies by signal)")+ ax.set_title("Cross-signal comparison — per (tier × mode), spans vs metrics vs logs")+ ax.legend(title="Signal")+ ax.grid(axis="y", linestyle=":", alpha=0.4)+ fig.tight_layout()+ fig.savefig(out, dpi=130)+ plt.close(fig)+++# ---------- Read-probe charts ----------+++def _readprobe_steps(run: dict) -> list[dict]:+ return ((run.get("readProbe") or {}).get("steps") or [])+++def render_readprobe_headline(runs: list[dict], signal: str, out: Path) -> None:+ sig_runs = iter_runs(runs, signal, "read-probe")+ if not sig_runs:+ return++ tiers_present = sorted({r["tier"] for r in sig_runs}, key=tier_rank)+ modes_present = sorted({r["mode"] for r in sig_runs}, key=mode_rank)+ if not tiers_present:+ return++ fig, ax = plt.subplots(figsize=(max(8, len(tiers_present) * 2.5), 6))++ values = [r.get("maxFillLevelPassed", 0) or 0 for r in sig_runs]+ ymax = max(values) if values else 1+ ghost_height = max(ymax * 0.02, 1)++ width = 0.8 / max(len(modes_present), 1)+ x_centers = list(range(len(tiers_present)))++ for mi, mode in enumerate(modes_present):+ for ti, tier in enumerate(tiers_present):+ match = [r for r in sig_runs if r["tier"] == tier and r["mode"] == mode]+ x = ti + (mi - (len(modes_present) - 1) / 2) * width+ if not match:+ continue+ run = match[0]+ value = run.get("maxFillLevelPassed", 0) or 0+ if value > 0:+ ax.bar(x, value, width=width, color=mode_color(mode))+ # latency at max+ last_pass = None+ for step in _readprobe_steps(run):+ if step.get("passed"):+ last_pass = step+ lat_str = f"\n{int(last_pass['readLatencyMs'])}ms" if last_pass else ""+ ax.annotate(+ f"{fmt_count(value)}{lat_str}",+ xy=(x, value), xytext=(0, 4), textcoords="offset points",+ ha="center", va="bottom", fontsize=8,+ )+ else:+ ax.bar(x, ghost_height, width=width, color="#dddddd",+ edgecolor="#999999", hatch="///", linewidth=0.7)+ ax.annotate(+ "no fill passed",+ xy=(x, ghost_height), xytext=(0, 4), textcoords="offset points",+ ha="center", va="bottom", fontsize=7, color="#aa3333",+ )++ ax.set_xticks(x_centers)+ ax.set_xticklabels(tiers_present)+ ax.set_yscale("log")+ ax.set_ylabel(f"Max passing fill level — {ROW_LABEL[signal]} (log)")+ ax.set_title(+ f"Read-probe headline — {signal} via {read_path_for_signal(signal)}\n"+ "Bar label: max passing fill + read latency at that fill"+ )++ legend = [Patch(facecolor=mode_color(m), label=m) for m in modes_present]+ legend.append(Patch(facecolor="#dddddd", edgecolor="#999999", hatch="///", label="no fill level passed"))+ ax.legend(handles=legend, title="DB mode", loc="upper left")+ ax.grid(axis="y", which="both", linestyle=":", alpha=0.4)++ fig.tight_layout()+ fig.savefig(out, dpi=130)+ plt.close(fig)++def render_readprobe(runs: list[dict], signal: str, out: Path) -> None:- runs = [r for r in runs if r["signal"] == signal and r["scenario"] == "read-probe"]- if not runs:+ """Read latency vs rows ingested — improved over legacy chart-readprobe."""+ sig_runs = [r for r in iter_runs(runs, signal, "read-probe") if _readprobe_steps(r)]+ if not sig_runs:return- fig, ax = plt.subplots(figsize=(10, 6))+ fig, ax = plt.subplots(figsize=(11, 6))plotted = Falsethreshold_ms = 5000- for run in sorted(runs, key=lambda r: (tier_rank(r["tier"]), mode_rank(r["mode"]))):+ for run in sig_runs:+ color = mode_color(run["mode"])rp = run.get("readProbe") or {}threshold_ms = rp.get("readThresholdMs", threshold_ms)- steps = rp.get("steps") or []+ steps = _readprobe_steps(run)xs = [s["rowsIngested"] for s in steps]ys = [s["readLatencyMs"] for s in steps]+ passed = [bool(s.get("passed")) for s in steps]if not xs:continue- ax.plot(xs, ys, marker="o", label=f"{run['tier']} / {run['mode']}")+ ax.plot(xs, ys, linestyle="-", color=color, label=run_label(run))+ for x, y, p in zip(xs, ys, passed):+ ax.plot(x, y, marker=("o" if p else "x"), color=color,+ markersize=9, markeredgewidth=2 if not p else 1, linestyle="")plotted = Trueif not plotted:@@ -166,11 +868,14 @@ def render_readprobe(runs: list[dict], signal: str, out: Path) -> None:ax.axhline(threshold_ms, linestyle="--", color="#cc3333", alpha=0.7,label=f"threshold ({threshold_ms}ms)")- ax.set_xlabel(f"{ROW_LABEL[signal]} (ingested before probe)")- ax.set_ylabel(f"Read latency on {readPathForSignal(signal)} (ms)")- ax.set_title(f"Read latency vs ingested rows — {signal}")+ ax.set_xlabel(f"{ROW_LABEL[signal]} ingested (log)")+ ax.set_ylabel("Read latency (ms, log)")ax.set_xscale("log")ax.set_yscale("log")+ ax.set_title(+ f"Read latency vs table size — {signal} via {read_path_for_signal(signal)}\n"+ "○ = passed (latency under threshold), × = failed step"+ )ax.grid(True, which="both", linestyle=":", alpha=0.4)ax.legend(loc="upper left", fontsize=9)fig.tight_layout()@@ -178,18 +883,247 @@ def render_readprobe(runs: list[dict], signal: str, out: Path) -> None:plt.close(fig)-def readPathForSignal(signal: str) -> str:- return {- "spans": "/api/endpoints/grouped",- "metrics": "/api/metrics/application",- "logs": "/api/logs",- }.get(signal, "")+def render_readprobe_ingest_time(runs: list[dict], signal: str, out: Path) -> None:+ sig_runs = [r for r in iter_runs(runs, signal, "read-probe") if _readprobe_steps(r)]+ if not sig_runs:+ return++ fig, ax = plt.subplots(figsize=(11, 6))+ plotted = False++ for run in sig_runs:+ color = mode_color(run["mode"])+ steps = _readprobe_steps(run)+ xs = [s["fillLevelTarget"] for s in steps]+ ys = [s.get("ingestSecondsElapsed", 0) for s in steps]+ pts = [(x, y) for x, y in zip(xs, ys) if y and y > 0]+ if not pts:+ continue+ xs2, ys2 = zip(*pts)+ ax.plot(xs2, ys2, linestyle="-", marker="o", color=color, label=run_label(run))+ plotted = True++ if not plotted:+ plt.close(fig)+ return++ ax.set_xlabel(f"Target fill level — {ROW_LABEL[signal]} (log)")+ ax.set_ylabel("Time to reach fill level (seconds, log)")+ ax.set_xscale("log")+ ax.set_yscale("log")+ ax.set_title(+ f"Fill time vs target rows — {signal}\n"+ "Curve sag = compaction/merge cost rising with table size."+ )+ ax.grid(True, which="both", linestyle=":", alpha=0.4)+ ax.legend(loc="upper left", fontsize=9)+ fig.tight_layout()+ fig.savefig(out, dpi=130)+ plt.close(fig)+++def render_readprobe_tier_scaling(runs: list[dict], signal: str, out: Path) -> None:+ sig_runs = iter_runs(runs, signal, "read-probe")+ if not sig_runs:+ return++ tiers_present = sorted({r["tier"] for r in sig_runs}, key=tier_rank)+ modes_present = sorted({r["mode"] for r in sig_runs}, key=mode_rank)+ if not tiers_present:+ return++ fig, ax = plt.subplots(figsize=(10, 6))+ plotted = False+ x_positions = list(range(len(tiers_present)))++ for mode in modes_present:+ xs, ys = [], []+ for i, tier in enumerate(tiers_present):+ match = [r for r in sig_runs if r["tier"] == tier and r["mode"] == mode]+ if not match:+ continue+ value = match[0].get("maxFillLevelPassed", 0) or 0+ if value <= 0:+ continue+ xs.append(i)+ ys.append(value)+ if not xs:+ continue+ ax.plot(xs, ys, marker="o", color=mode_color(mode), markersize=9,+ linewidth=2, label=mode)+ for x, y in zip(xs, ys):+ ax.annotate(fmt_count(y), xy=(x, y), xytext=(0, 8),+ textcoords="offset points", ha="center", va="bottom", fontsize=9)+ base_x = xs[0]+ base_y = ys[0]+ base_vcpu = TIER_VCPU.get(tiers_present[base_x], 1)+ ref_xs = list(range(len(tiers_present)))+ ref_ys = [base_y * (TIER_VCPU.get(tiers_present[i], 1) / base_vcpu) for i in ref_xs]+ ax.plot(ref_xs, ref_ys, linestyle="--", color=mode_color(mode), alpha=0.3, linewidth=1)+ plotted = True++ if not plotted:+ plt.close(fig)+ return++ ax.set_xticks(x_positions)+ ax.set_xticklabels([f"{t}\n{TIER_META.get(t, '')}" for t in tiers_present])+ ax.set_yscale("log")+ ax.set_ylabel(f"Max passing fill level — {ROW_LABEL[signal]} (log)")+ ax.set_title(+ f"Read-probe tier scaling — {signal} via {read_path_for_signal(signal)}\n"+ "Dashed = linear-vCPU reference anchored at smallest tier."+ )+ ax.legend(title="DB mode", loc="upper left")+ ax.grid(True, which="both", linestyle=":", alpha=0.4)+ fig.tight_layout()+ fig.savefig(out, dpi=130)+ plt.close(fig)+++def render_readprobe_cliff(runs: list[dict], signal: str, out: Path) -> None:+ sig_runs = iter_runs(runs, signal, "read-probe")+ if not sig_runs:+ return++ # Build the union of all fill levels seen across runs, sorted.+ all_targets = sorted({s["fillLevelTarget"] for r in sig_runs for s in _readprobe_steps(r)})+ if not all_targets:+ return++ n_cols = len(all_targets)+ n_rows = len(sig_runs)++ fig_w = max(10, n_cols * 1.8 + 3.5)+ fig_h = max(4, n_rows * 0.9 + 2)+ fig, ax = plt.subplots(figsize=(fig_w, fig_h))++ def _probe_status(step: dict) -> str:+ if step.get("passed"):+ return "pass"+ # over-threshold but read OK+ if step.get("readOk"):+ return "soft-cliff"+ return "hard-fail"++ for ri, run in enumerate(sig_runs):+ y = n_rows - ri - 1+ ax.text(-0.5, y + 0.5, run_label(run), ha="right", va="center",+ fontsize=10, fontweight="bold")++ by_target = {s["fillLevelTarget"]: s for s in _readprobe_steps(run)}+ for ci, target in enumerate(all_targets):+ x = ci+ step = by_target.get(target)+ if step is None:+ color = STATUS_COLOR["missing"]+ label_text = ""+ else:+ color = STATUS_COLOR[_probe_status(step)]+ lat = int(step.get("readLatencyMs", 0))+ label_text = f"{lat}ms"+ ax.add_patch(Rectangle((x, y), 1, 1, facecolor=color,+ edgecolor="white", linewidth=1.5))+ if label_text:+ ax.text(x + 0.5, y + 0.5, label_text, ha="center", va="center", fontsize=8)++ max_passed = run.get("maxFillLevelPassed", 0) or 0+ margin = n_cols + 0.35+ if max_passed > 0:+ ax.text(margin, y + 0.5, f"max {fmt_count(max_passed)}",+ ha="left", va="center", fontsize=10, fontweight="bold")+ else:+ ax.text(margin, y + 0.5, "no fill passed",+ ha="left", va="center", fontsize=10, color="#cc3333", fontweight="bold")++ # Column header (fill level)+ for ci, target in enumerate(all_targets):+ ax.text(ci + 0.5, n_rows + 0.25, fmt_count(target),+ ha="center", va="bottom", fontsize=10, fontweight="bold")+ ax.text(n_cols / 2.0, n_rows + 0.85, f"Fill level ({ROW_LABEL[signal]})",+ ha="center", va="bottom", fontsize=11, fontweight="bold")++ ax.set_xlim(-4, n_cols + 5)+ ax.set_ylim(-0.6, n_rows + 1.5)+ ax.set_aspect("equal")+ ax.axis("off")++ legend = [+ Patch(facecolor=STATUS_COLOR["pass"], label="passed (read OK, under threshold)"),+ Patch(facecolor=STATUS_COLOR["soft-cliff"], label="over threshold (read OK but slow)"),+ Patch(facecolor=STATUS_COLOR["hard-fail"], label="read errored"),+ Patch(facecolor=STATUS_COLOR["missing"], edgecolor="#cccccc", label="not run"),+ ]+ ax.legend(handles=legend, loc="lower center", bbox_to_anchor=(0.5, -0.05),+ ncol=4, fontsize=9, frameon=False)++ fig.suptitle(f"Read-probe step-status grid — {signal}\nread path: {read_path_for_signal(signal)}",+ fontsize=12)+ fig.tight_layout(rect=(0, 0.04, 1, 0.94))+ fig.savefig(out, dpi=130, bbox_inches="tight")+ plt.close(fig)+++def render_readprobe_signal_mix(runs: list[dict], out: Path) -> None:+ rp_runs = [r for r in runs if r.get("scenario") == "read-probe"]+ if not rp_runs:+ return++ keys = sorted({(r["tier"], r["mode"]) for r in rp_runs},+ key=lambda k: (tier_rank(k[0]), mode_rank(k[1])))+ if not keys:+ return++ fig, ax = plt.subplots(figsize=(max(8, len(keys) * 1.9), 6))++ width = 0.25+ x = list(range(len(keys)))+ any_value = False++ for si, signal in enumerate(SIGNALS):+ ys = []+ for tier, mode in keys:+ match = [r for r in rp_runs if r["tier"] == tier and r["mode"] == mode and r["signal"] == signal]+ ys.append((match[0].get("maxFillLevelPassed", 0) or 0) if match else 0)+ offsets = [xi + (si - 1) * width for xi in x]+ bars = ax.bar(offsets, ys, width=width,+ color=SIGNAL_COLORS[signal],+ label=f"{signal} ({read_path_for_signal(signal)})")+ for b, y in zip(bars, ys):+ if y > 0:+ ax.annotate(fmt_count(y), xy=(b.get_x() + b.get_width() / 2, y),+ xytext=(0, 3), textcoords="offset points",+ ha="center", va="bottom", fontsize=7)+ any_value = True++ if not any_value:+ plt.close(fig)+ return++ ax.set_xticks(x)+ ax.set_xticklabels([f"{t}\n{m}" for t, m in keys], fontsize=9)+ ax.set_yscale("log")+ ax.set_ylabel("Max passing fill level (rows, log)")+ ax.set_title("Read-probe cross-signal comparison — which read path scales worst?")+ ax.legend(title="Signal (read path)", loc="upper left", fontsize=8)+ ax.grid(axis="y", which="both", linestyle=":", alpha=0.4)+ fig.tight_layout()+ fig.savefig(out, dpi=130)+ plt.close(fig)+++# ---------- summary.md ----------def render_summary(runs: list[dict], out: Path) -> None:- lines = ["# Traceway hardware benchmark — summary", ""]- lines.append(f"Runs analyzed: {len(runs)}")- lines.append("")+ lines = [+ "# Traceway hardware benchmark — summary",+ "",+ "> See [../charts.md](../charts.md) for a guide to reading these charts.",+ "",+ f"Runs analyzed: {len(runs)}",+ "",+ ]throughput_runs = [r for r in runs if r["scenario"] == "throughput"]if throughput_runs:@@ -206,7 +1140,7 @@ def render_summary(runs: list[dict], out: Path) -> None:lines.append(f"| Tier | Mode | Max {ITEM_LABEL[signal]} | P1 max batch | P2 max req/sec @ batch | P3 max req/sec @ batch=100 |")lines.append("|------|------|---------|--------------|-------------------------|-------------------------------|")for run in sorted(sig_runs, key=lambda r: (tier_rank(r["tier"]), mode_rank(r["mode"]))):- max_items = int(run.get("maxSustainableItemsPerSec", 0))+ max_items = int(run.get("maxSustainableItemsPerSec", 0) or 0)max_batch = (run.get("phase1") or {}).get("maxBatchSize", 0)p2 = run.get("phase2") or {}p2_rate = p2.get("maxRequestRate", 0)@@ -214,8 +1148,7 @@ def render_summary(runs: list[dict], out: Path) -> None:p2_cell = f"{p2_rate:g} @ {p2_batch:,}" if p2_rate else "—"p3 = run.get("phase3") or {}p3_rate = p3.get("maxRequestRate", 0)- p3_batch = p3.get("fixedBatchSize", 0)- p3_cell = f"{p3_rate:g} @ {p3_batch:,}" if p3_rate else "—"+ p3_cell = f"{p3_rate:g}" if p3_rate else "—"lines.append(f"| {run['tier']} | {run['mode']} | {max_items:,} | {max_batch:,} | {p2_cell} | {p3_cell} |")lines.append("")@@ -230,14 +1163,13 @@ def render_summary(runs: list[dict], out: Path) -> None:sig_runs = [r for r in readprobe_runs if r["signal"] == signal]if not sig_runs:continue- lines.append(f"### {signal.capitalize()} — probing `{readPathForSignal(signal)}`")+ lines.append(f"### {signal.capitalize()} — probing `{read_path_for_signal(signal)}`")lines.append("")lines.append(f"| Tier | Mode | Max {ROW_LABEL[signal]} passed | Read latency @ max (ms) | First failing fill | Read latency @ failing (ms) |")lines.append("|------|------|----------------|-------------------|--------------------|------------------------|")for run in sorted(sig_runs, key=lambda r: (tier_rank(r["tier"]), mode_rank(r["mode"]))):- rp = run.get("readProbe") or {}- steps = rp.get("steps") or []- max_passed = run.get("maxFillLevelPassed", 0)+ steps = _readprobe_steps(run)+ max_passed = run.get("maxFillLevelPassed", 0) or 0latency_at_max = 0first_fail = ""latency_at_fail = ""@@ -254,6 +1186,9 @@ def render_summary(runs: list[dict], out: Path) -> None:out.write_text("\n".join(lines) + "\n")+# ---------- Entry point ----------++def main() -> int:if len(sys.argv) < 2:print("usage: chart.py <results-dir>", file=sys.stderr)@@ -269,9 +1204,25 @@ def main() -> int:return 1for signal in SIGNALS:- render_throughput_bar(runs, signal, results_dir / f"chart-{signal}.png")- render_throughput_detail(runs, signal, results_dir / f"chart-detail-{signal}.png")+ # Throughput charts (no-op if no throughput runs for this signal)+ render_headline_bar(runs, signal, results_dir / f"chart-{signal}.png")+ render_phase1_latency(runs, signal, results_dir / f"chart-phase1-batch-{signal}.png")+ render_phase2_rate(runs, signal, results_dir / f"chart-phase2-rate-{signal}.png")+ render_phase3_rate(runs, signal, results_dir / f"chart-phase3-rate-{signal}.png")+ render_pareto(runs, signal, results_dir / f"chart-pareto-{signal}.png")+ render_tier_scaling(runs, signal, results_dir / f"chart-tier-scaling-{signal}.png")+ render_cliff_grid(runs, signal, results_dir / f"chart-cliff-{signal}.png")+ render_batch_efficiency(runs, signal, results_dir / f"chart-batch-efficiency-{signal}.png")++ # Read-probe charts (no-op if no read-probe runs for this signal)+ render_readprobe_headline(runs, signal, results_dir / f"chart-readprobe-headline-{signal}.png")render_readprobe(runs, signal, results_dir / f"chart-readprobe-{signal}.png")+ render_readprobe_ingest_time(runs, signal, results_dir / f"chart-readprobe-ingest-time-{signal}.png")+ render_readprobe_tier_scaling(runs, signal, results_dir / f"chart-readprobe-tier-scaling-{signal}.png")+ render_readprobe_cliff(runs, signal, results_dir / f"chart-readprobe-cliff-{signal}.png")++ render_signal_mix(runs, results_dir / "chart-signal-mix.png")+ render_readprobe_signal_mix(runs, results_dir / "chart-readprobe-signal-mix.png")render_summary(runs, results_dir / "summary.md")print(f"wrote charts and summary.md into {results_dir}")return 0diff --git a/benchmarks/scripts/run-local.sh b/benchmarks/scripts/run-local.shindex ef1d9dae..02734a03 100755--- a/benchmarks/scripts/run-local.sh+++ b/benchmarks/scripts/run-local.sh@@ -14,7 +14,13 @@# run-local.sh --smoke # 1 tier, 1 mode, 1 signal, short steps (~5 min)# run-local.sh --tier ccx13 --mode sqlite --signal spans# run-local.sh --dry-run # validate env + print plan, no provisioning-# run-local.sh --commit # drop the "-local" suffix on the output dir+#+# Output:+# --scenario throughput -> benchmarks/results-throughput/+# --scenario read-probe -> benchmarks/results-probe/+# Each scenario folder is wiped at the start of its run so output reflects+# only the current dispatch. The two folders are never touched by each+# other's runs — running read-probe will not clobber throughput results.set -euo pipefailSCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"@@ -27,10 +33,9 @@ SCENARIO="throughput"DURATION="30m"SMOKE=0DRY_RUN=0-COMMIT=0usage() {- sed -n '1,17p' "$0"; exit 2+ sed -n '1,23p' "$0"; exit 2}while [[ $# -gt 0 ]]; do@@ -42,7 +47,6 @@ while [[ $# -gt 0 ]]; do--duration) DURATION="$2"; shift 2 ;;--smoke) SMOKE=1; shift ;;--dry-run) DRY_RUN=1; shift ;;- --commit) COMMIT=1; shift ;;-h|--help) usage ;;*) echo "unknown flag: $1" >&2; usage ;;esac@@ -67,17 +71,17 @@ fiexport BENCH_MODES_PLAN="${MODES}""${SCRIPT_DIR}/preflight.sh"-# Single, dispatch-scoped results folder. Wiped at the start so each run's-# committed output reflects only that run's matrix entries — no-# cross-dispatch mixing (which was the failure mode of the date-folder-# layout: two dispatches landing on the same day silently combined into one-# summary.md). Cross-run comparison happens via `git log`, not adjacent+# One canonical folder per scenario. Wiped at the start so each run's output+# reflects only that run's matrix entries — no cross-dispatch mixing (which+# was the failure mode of the date-folder layout: two dispatches landing on+# the same day silently combined into one summary.md). The throughput folder+# and the read-probe folder are siblings so each scenario stays out of the+# other's way. Cross-run comparison happens via `git log`, not adjacent# folders.-if [[ "${COMMIT}" -eq 1 ]]; then- OUT_DIR="${REPO_ROOT}/benchmarks/results/latest"-else- OUT_DIR="${REPO_ROOT}/benchmarks/results/latest-local"-fi+case "${SCENARIO}" in+ throughput) OUT_DIR="${REPO_ROOT}/benchmarks/results-throughput" ;;+ read-probe) OUT_DIR="${REPO_ROOT}/benchmarks/results-probe" ;;+esacrm -rf "${OUT_DIR}"mkdir -p "${OUT_DIR}"echo "results dir: ${OUT_DIR}" >&2