Skip to content

The only tool you need to know what is happening and how to fix it.

better output dirs and charts for benching

ddux68ad3a33a458
diff --git a/.github/workflows/benchmark-hardware.yml b/.github/workflows/benchmark-hardware.yml
index 02d2ab6c..f03d29fb 100644
--- a/.github/workflows/benchmark-hardware.yml
+++ b/.github/workflows/benchmark-hardware.yml
@@ -18,7 +18,7 @@ on:
required: true
default: 'spans,metrics,logs'
scenario:
- description: 'Benchmark scenario: throughput (two-phase ingest ramp, default) or read-probe (ingest to fill levels, probe a read, fail above 5s).'
+ description: 'Benchmark scenario: throughput (three-phase ingest ramp, default; writes to benchmarks/results-throughput/) or read-probe (ingest to fill levels, probe a read, fail above 5s; writes to benchmarks/results-probe/).'
required: true
type: choice
options:
@@ -119,13 +119,13 @@ jobs:
run: ./benchmarks/scripts/preflight.sh
- name: Run matrix entry (${{ matrix.tier }} / ${{ matrix.mode }} / ${{ matrix.signal }})
- run: ./benchmarks/scripts/run-matrix-entry.sh "${{ matrix.tier }}" "${{ matrix.mode }}" "${{ matrix.signal }}" "${{ github.event.inputs.duration }}" benchmarks/results/_artifact
+ run: ./benchmarks/scripts/run-matrix-entry.sh "${{ matrix.tier }}" "${{ matrix.mode }}" "${{ matrix.signal }}" "${{ github.event.inputs.duration }}" benchmarks/_artifact
- uses: actions/upload-artifact@v4
if: always()
with:
name: result-${{ matrix.tier }}-${{ matrix.mode }}-${{ matrix.signal }}
- path: benchmarks/results/_artifact/*.json
+ path: benchmarks/_artifact/*.json
if-no-files-found: warn
aggregate:
@@ -143,15 +143,19 @@ jobs:
with:
path: /tmp/results-raw
- - name: Stage into benchmarks/results/latest/
+ - name: Stage into scenario folder
id: stage
run: |
set -euo pipefail
- # Always overwrite `latest/` so each dispatch reports only its own
- # matrix entries. Cross-run comparison happens via `git log`, not
- # via cumulative same-folder data — the old date-stamped folder
- # approach silently mixed dispatches that landed on the same day.
- dst="benchmarks/results/latest"
+ # One canonical folder per scenario, wiped on each dispatch so the
+ # committed tree reflects exactly one dispatch — but the two
+ # scenarios live in sibling folders so a read-probe run never
+ # clobbers throughput results and vice-versa.
+ case "${{ github.event.inputs.scenario }}" in
+ throughput) dst="benchmarks/results-throughput" ;;
+ read-probe) dst="benchmarks/results-probe" ;;
+ *) echo "unknown scenario '${{ github.event.inputs.scenario }}'" >&2; exit 1 ;;
+ esac
rm -rf "${dst}"
mkdir -p "${dst}"
find /tmp/results-raw -name '*.json' -exec cp {} "${dst}/" \;
@@ -173,8 +177,8 @@ jobs:
# Copy our staged data aside before touching git state. We always
# need to restore it after `reset --hard` regardless of what main
- # currently has at `latest/` — this dispatch's data is the new
- # source of truth.
+ # currently has at this scenario folder — this dispatch's data is
+ # the new source of truth.
tmp_staged="$(mktemp -d)/staged"
cp -a "${staged}" "${tmp_staged}"
@@ -182,12 +186,14 @@ jobs:
git fetch origin main
git reset --hard origin/main
- # Wipe whatever main had at `latest/` (probably a prior dispatch's
- # results) and replace it with this dispatch's full output. Using
- # rm + cp + `git add -A` so deletions are captured: if the prior
- # commit had `ccx33-pgch-metrics.json` but this dispatch only ran
- # sqlite, the pgch JSON gets staged for deletion. The committed
- # `latest/` always reflects exactly one dispatch, never a union.
+ # Wipe whatever main had at this scenario folder (probably a prior
+ # dispatch of the SAME scenario) and replace it with this
+ # dispatch's full output. Using rm + cp + `git add -A` so deletions
+ # are captured: if the prior commit had `ccx33-pgch-metrics-throughput.json`
+ # but this dispatch only ran sqlite, the pgch JSON gets staged for
+ # deletion. The committed scenario folder always reflects exactly
+ # one dispatch of that scenario, never a union. The OTHER scenario
+ # folder is untouched by this step.
rm -rf "${staged}"
mkdir -p "${staged}"
cp -a "${tmp_staged}/." "${staged}/"
@@ -198,5 +204,5 @@ jobs:
echo "nothing to commit"
exit 0
fi
- git commit -m "bench: hardware benchmark $(date -u +%Y-%m-%d) (latest)"
+ git commit -m "bench: ${{ github.event.inputs.scenario }} $(date -u +%Y-%m-%d)"
git push
diff --git a/.gitignore b/.gitignore
index de49390f..38a97d79 100644
--- a/.gitignore
+++ b/.gitignore
@@ -47,11 +47,11 @@ backend/static/dist/
/examples/nestjs-otel/node_modules
/website/.wrangler
-# Benchmark tooling: built binaries and laptop-only result dirs.
-# Real CI-produced result dirs (e.g. benchmarks/results/2026-05-12/) are
-# committed; the -local suffix marks dev runs we never want to push.
+# Benchmark tooling: built binaries and runner-only scratch dirs. The
+# committed result dirs are benchmarks/results-throughput/ and
+# benchmarks/results-probe/; everything else is local or runner-scoped.
/benchmarks/loadgen/loadgen
/benchmarks/loadgen/loadgen-linux-amd64
/benchmarks/loadgen/loadgen-linux-arm64
-/benchmarks/results/*-local/
+/benchmarks/_artifact/
benchmarks/loadgen/bench-loadgen
diff --git a/benchmarks/LOCAL.md b/benchmarks/LOCAL.md
index 04c29a99..909786a0 100644
--- a/benchmarks/LOCAL.md
+++ b/benchmarks/LOCAL.md
@@ -124,8 +124,11 @@ echo "PROJECT_TOKEN=$PROJECT_TOKEN"
`seed-project.sh` hits `/api/register`. On self-hosted backends only one
organization is allowed, so this only works on a **fresh** DB — step 1's
-`down -v` is what guarantees that. The OTLP-only loadgen no longer needs the
-JWT or project ID, so we extract only the project token.
+`down -v` is what guarantees that. The throughput scenario only needs the
+project token (OTLP ingestion is bearer-authed); the read-probe scenario
+also needs the JWT + project ID for the dashboard endpoints — see
+[Running the read-probe scenario locally](#running-the-read-probe-scenario-locally)
+below.
### 4. Build and run the loadgen, one signal at a time
@@ -134,9 +137,17 @@ JWT or project ID, so we extract only the project token.
mkdir -p /tmp/bench-localhost && rm -f /tmp/bench-localhost/*.json
```
-Pick numbers appropriate for the mode you booted in step 1. The two-phase
-ramp first grows batch size at a fixed low request rate, then grows request
-rate at the winning batch size.
+Pick numbers appropriate for the mode you booted in step 1. The throughput
+scenario runs a three-phase ramp:
+1. **Phase 1** grows batch size at a fixed low request rate (saturates one
+ client; finds the per-request decode/insert ceiling).
+2. **Phase 2** grows request rate at the winning batch size capped at
+ `--phase2-batch-cap` (collector shape: fat batches, low req rate).
+3. **Phase 3** grows request rate at batch=100 (SDK-fleet shape: small
+ batches, high req rate — stresses HTTP/auth/decode rather than insert).
+
+The examples below trim Phase 3 to keep laptop runs short; remove the
+`--phase3-*` overrides to exercise it.
**SQLite (laptop) — quick smoke per signal:**
@@ -149,7 +160,8 @@ for sig in spans metrics logs; do
--duration 5m --step-duration 30s \
--phase1-batch-sizes 256,1024,4096 \
--phase2-request-rates 1,5,20 \
- --report-out "/tmp/bench-localhost/local-sqlite-${sig}.json" \
+ --phase3-request-rates 10,100 \
+ --report-out "/tmp/bench-localhost/local-sqlite-${sig}-throughput.json" \
--tier local --mode sqlite
done
```
@@ -165,12 +177,17 @@ for sig in spans metrics logs; do
--duration 8m --step-duration 45s \
--phase1-batch-sizes 256,1024,4096,8192 \
--phase2-request-rates 1,5,20,50 \
- --report-out "/tmp/bench-localhost/local-pgch-${sig}.json" \
+ --phase3-request-rates 10,100,500 \
+ --report-out "/tmp/bench-localhost/local-pgch-${sig}-throughput.json" \
--tier local --mode pgch
done
```
-On stderr you'll see two-phase progress like:
+The `-throughput` suffix on the filename matches the scenario folder convention
+(`benchmarks/results-throughput/`); use `-read-probe` when running
+`--scenario read-probe` (see [Running the read-probe scenario locally](#running-the-read-probe-scenario-locally) below).
+
+On stderr you'll see per-phase progress like:
```
phase1 step 1: batch=256 rate=5.0 items/s=1280 p99=12ms err=0.00% passed=true
@@ -179,7 +196,9 @@ phase1 step 3: batch=4096 rate=5.0 items/s=20480 p99=42ms err=0.00% passed=true
phase2 step 1: batch=4096 rate=1.0 items/s=4096 p99=15ms err=0.00% passed=true
phase2 step 2: batch=4096 rate=5.0 items/s=20480 p99=21ms err=0.00% passed=true
phase2 step 3: batch=4096 rate=20.0 items/s=81920 p99=180ms err=0.00% passed=true
-wrote /tmp/bench-localhost/local-sqlite-spans.json: signal=spans max sustainable spans/sec = 81920
+phase3 step 1: batch=100 rate=10.0 items/s=1000 p99=8ms err=0.00% passed=true
+phase3 step 2: batch=100 rate=100.0 items/s=10000 p99=22ms err=0.00% passed=true
+wrote /tmp/bench-localhost/local-sqlite-spans-throughput.json: signal=spans max sustainable spans/sec = 81920
```
A run finishes when either (a) a step fails the error threshold, (b) all
@@ -189,12 +208,19 @@ configured steps pass, or (c) `--duration` expires.
```bash
.venv-bench/bin/python benchmarks/scripts/chart.py /tmp/bench-localhost/
-open /tmp/bench-localhost/chart-spans.png
-open /tmp/bench-localhost/chart-metrics.png
-open /tmp/bench-localhost/chart-logs.png
+open /tmp/bench-localhost/chart-spans.png # headline bars per tier × mode
+open /tmp/bench-localhost/chart-phase2-rate-spans.png # collector-shape rate ramp
+open /tmp/bench-localhost/chart-pareto-spans.png # latency–throughput trade-off
+open /tmp/bench-localhost/chart-cliff-spans.png # step-by-step pass/fail grid
cat /tmp/bench-localhost/summary.md
```
+The full chart suite (~14 PNG types per scenario folder, headline +
+phase-by-phase ramps + Pareto + tier scaling + cliff grids + batch
+efficiency + signal mix, plus read-probe equivalents) is described in
+[charts.md](charts.md). `chart.py` auto-detects which scenarios are in
+the folder and only renders the relevant charts.
+
A one-entry result renders fine; it just won't have multiple bars/lines to
compare against. Old pre-OTLP-split JSONs (without a `signal` field) are
skipped with a log line.
@@ -211,11 +237,12 @@ The `-v` deletes the named volume so the next run is guaranteed clean.
## Reading the output
-Each result JSON has two phases:
+Each throughput result JSON has up to three phases plus a headline:
```json
{
"tier": "local", "mode": "sqlite", "signal": "spans",
+ "scenario": "throughput",
"startedAt": "2026-05-15T...", "endedAt": "...",
"phase1": {
"kind": "batch-size-ramp",
@@ -236,16 +263,31 @@ Each result JSON has two phases:
"steps": [ ... ],
"maxRequestRate": 100
},
+ "phase3": {
+ "kind": "small-batch-rate-ramp",
+ "fixedBatchSize": 100,
+ "steps": [ ... ],
+ "maxRequestRate": 1000
+ },
"maxSustainableItemsPerSec": 819200
}
```
**A step passes** when the combined error rate (HTTP failures + OTLP
-`PartialSuccess` rejected items) is ≤ `--ingest-err-threshold` (default 5%).
-Read endpoints are not probed in this benchmark.
+`PartialSuccess` rejected items) is ≤ `--ingest-err-threshold` (default 5%)
+AND the achieved request rate is ≥ `--soft-cliff-ratio × target` (default
+70%). The second check catches saturated-but-not-erroring soft cliffs.
-`maxSustainableItemsPerSec = phase2.fixedBatchSize × phase2.maxRequestRate`,
-where `maxRequestRate` is the rate of the last passing Phase 2 step.
+`maxSustainableItemsPerSec` is the **highest `actualItemsPerSec` recorded
+across any passing step from any phase** — measured from real OK responses,
+not `batchSize × targetRate` (which over-reports when the workers saturate
+before the limiter does). Different phases probe different shapes
+(collector fat-batch vs SDK small-batch); the headline reports the best
+shape's ceiling and per-phase numbers stay in the JSON for shape-specific
+analysis.
+
+Read-probe JSONs have a different shape — see [Running the read-probe
+scenario locally](#running-the-read-probe-scenario-locally) below.
---
@@ -281,13 +323,19 @@ sustain on synthetic, evenly-distributed load."
| Flag | Default | When to change |
|------|---------|----------------|
| `--signal` | (required) | One of `spans`, `metrics`, `logs`. |
+| `--scenario` | `throughput` | `throughput` (three-phase ingest ramp) or `read-probe` (fill the table, probe a read at each level). |
| `--phase1-batch-sizes` | `256,1024,4096,8192,16384` | Drop the lower steps on beefy boxes to save time; raise the cap to find the per-request decode/insert ceiling. |
| `--phase2-request-rates` | `1,5,25,100,400` | Raise the cap to find the request-rate ceiling on big boxes; lower for cheap smoke tests. |
+| `--phase3-request-rates` | `10,100,1000,5000,10000` | The SDK-fleet shape ramp at batch=100. Lower the cap for fast smoke tests; drop the flag entirely to use the default range. |
| `--phase1-fixed-rate` | 5 | Per-second request rate held constant during Phase 1. Small enough that batch size is the only variable. |
-| `--phase2-batch-cap` | 8192 | Cap on Phase 2 batch size (matches OTel collector default `send_batch_size`). Phase 2 uses `min(this, Phase 1 winner)`. |
+| `--phase2-batch-cap` | 16384 | Cap on Phase 2 batch size. Phase 2 uses `min(this, Phase 1 winner)`. Bumped past the OTel collector default of 8192 because pgch SUTs can usefully exceed it. |
+| `--phase3-batch-size` | 100 | Fixed batch size for Phase 3 — matches typical language-SDK `BatchSpanProcessor` output rather than the collector's 8192. |
+| `--soft-cliff-ratio` | 0.70 | A step also fails when achieved req-rate falls below this fraction of target — catches saturated-but-not-erroring cliffs. Set to 0 to disable. |
| `--step-duration` | 2m | Shorten for smoke checks (15–30s), keep at 2m for real measurements. |
| `--duration` | 30m | Hard cap on total wall time. |
| `--ingest-err-threshold` | 0.05 | Combined HTTP error + OTLP rejected rate that fails a step. |
+| `--fill-levels` | `100000,1000000,10000000,100000000` | (read-probe only) Row counts to fill the table to before each probe. |
+| `--read-threshold-ms` | 5000 | (read-probe only) A probe fails if the read exceeds this. |
The ramp is geometric (256 → 1024 → 4096 → …), so cliff resolution is coarse
by design — for a precise number, run a second pass with a narrower band of
@@ -321,6 +369,87 @@ values around the known cliff.
---
+## Running the read-probe scenario locally
+
+The throughput scenario above answers "how fast can the SUT swallow OTLP
+data?" The read-probe scenario answers "how big can the table grow before
+the dashboard read on this endpoint cliffs?" Each fill level is reached by
+ingesting OTLP until the row count is hit, then the loadgen waits
+`--settle-seconds` and issues one read against the signal's dashboard
+endpoint (`/api/endpoints/grouped` for spans, `/api/metrics/application`
+for metrics, `/api/logs` for logs). The step fails when the read takes
+longer than `--read-threshold-ms` (default 5 s).
+
+Read-probe needs a JWT + project ID, not just a project token, because the
+dashboard endpoints are JWT-authenticated. `seed-project.sh` returns both:
+
+```bash
+seed_json=$(benchmarks/scripts/seed-project.sh http://localhost:8087)
+export PROJECT_TOKEN=$(echo "${seed_json}" | jq -r .projectToken)
+export PROJECT_ID=$(echo "${seed_json}" | jq -r .projectId)
+export JWT=$(echo "${seed_json}" | jq -r .jwt)
+```
+
+Then run the loadgen with `--scenario read-probe`. Small fill levels keep
+the laptop honest:
+
+```bash
+mkdir -p /tmp/bench-localhost && rm -f /tmp/bench-localhost/*.json
+for sig in spans metrics logs; do
+ benchmarks/loadgen/loadgen \
+ --target http://localhost:8087 \
+ --token "$PROJECT_TOKEN" --jwt "$JWT" --project-id "$PROJECT_ID" \
+ --signal "$sig" --scenario read-probe \
+ --fill-levels 10000,100000,1000000 \
+ --settle-seconds 5s \
+ --report-out "/tmp/bench-localhost/local-pgch-${sig}-read-probe.json" \
+ --tier local --mode pgch
+done
+.venv-bench/bin/python benchmarks/scripts/chart.py /tmp/bench-localhost/
+open /tmp/bench-localhost/chart-readprobe-headline-spans.png
+open /tmp/bench-localhost/chart-readprobe-spans.png
+open /tmp/bench-localhost/chart-readprobe-cliff-spans.png
+```
+
+`chart.py` against a folder mixing throughput and read-probe JSONs renders
+both suites — the wrong-scenario renderers are no-ops. In production, the
+CI workflow keeps them separate by writing to `benchmarks/results-throughput/`
+vs `benchmarks/results-probe/` based on the `--scenario` flag.
+
+Read-probe JSONs have a different shape from throughput:
+
+```json
+{
+ "tier": "local", "mode": "pgch", "signal": "spans",
+ "scenario": "read-probe",
+ "readProbe": {
+ "readPath": "/api/endpoints/grouped",
+ "readThresholdMs": 5000,
+ "settleSeconds": 5,
+ "fillBatchSize": 8192,
+ "fillRequestRate": 100,
+ "steps": [
+ {
+ "fillLevelTarget": 10000,
+ "rowsIngested": 10123,
+ "ingestSecondsElapsed": 3.2,
+ "readLatencyMs": 180,
+ "readOk": true,
+ "passed": true
+ }
+ ],
+ "maxFillLevelPassed": 1000000
+ },
+ "maxFillLevelPassed": 1000000
+}
+```
+
+The headline number is `maxFillLevelPassed` — the largest row count at
+which the read came back under threshold. The matching read latency lives
+in the last passing step's `readLatencyMs`.
+
+---
+
## Going beyond local
Once the local run works end-to-end, move to the Hetzner ladder:
diff --git a/benchmarks/README.md b/benchmarks/README.md
index bf9ea3ef..876c4130 100644
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@@ -31,13 +31,26 @@ pollute the latency signal:
| CCX33 | 8 | 32 GB | 240 GB NVMe |
| CCX43 | 16 | 64 GB | 360 GB NVMe |
-## Not measured here
+## Two scenarios
-This benchmark is **ingest-only** — it does not probe dashboard read latency.
-A separate future benchmark will pre-load N events (100k, 1M, 10M, 100M) and
-measure read endpoint P99 at each fill level. Don't read the numbers here as
-"the dashboard stays usable at N items/sec" — they're "the SUT keeps swallowing
-items at N items/sec without erroring or rejecting."
+Two distinct questions; one matrix-entry script answers either depending on
+the `--scenario` flag.
+
+- **`throughput` (default).** A three-phase ingest ramp (Phase 1: batch
+ size; Phase 2: collector-shape rate; Phase 3: SDK-fleet-shape rate).
+ Results land in `benchmarks/results-throughput/`. Answers "how fast can
+ the SUT swallow OTLP data without erroring or rejecting?"
+- **`read-probe`.** Fills the table to a sequence of row counts
+ (`100k, 1M, 10M, 100M`) and probes one dashboard read at each level,
+ failing the step when the read exceeds `--read-threshold-ms` (default
+ 5 s). Results land in `benchmarks/results-probe/`. Answers "how big
+ can the table grow before the dashboard read on this endpoint cliffs?"
+
+The two scenarios write to sibling folders so one never overwrites the
+other. Each folder is wiped on each dispatch of its scenario.
+
+Don't read the throughput number as "the dashboard stays usable at N
+items/sec" — that's what read-probe is for.
## How a run works
@@ -51,7 +64,7 @@ Per matrix entry (one tier × one mode × one signal):
4. `loadgen-bootstrap.sh` cross-compiles the loadgen, pushes it to the
loadgen box, runs it with `--signal <spans|metrics|logs>` against the
SUT's *private* IP.
-5. The loadgen runs a two-phase ramp:
+5. The loadgen runs a three-phase ramp:
- **Phase 1 — batch-size ramp.** Single client at a fixed 5 req/sec.
Batch sizes step through `256,1024,4096,8192,16384`. Each step holds for
`--step-duration` (default 2 min). Stops at the first failing step.
@@ -111,9 +124,11 @@ prevents a permanently-broken SUT from restart-looping forever; if the
third restart also crashes, the container stays down and the health-poll
fires the clean-abort path.
-After all matrix entries finish, `chart.py` renders three bar charts
-(`chart-spans.png`, `chart-metrics.png`, `chart-logs.png`), three detail
-charts (`chart-detail-<signal>.png`), and a combined `summary.md`.
+After all matrix entries finish, `chart.py` renders the full chart suite
+(headline bars, Phase 1 / 2 / 3 ramps, Pareto, tier scaling, cliff grid,
+batch efficiency, signal mix — plus the read-probe equivalents in the
+read-probe folder) and a combined `summary.md`. See
+[charts.md](charts.md) for a guide to reading each chart.
## Running from your laptop
@@ -156,11 +171,12 @@ works, the full matrix works.
```
4 tiers × 2 modes × 3 signals, default `--duration 30m`. Output goes to
-`benchmarks/results/latest-local/`, which is wiped at the start of every
-run so each dispatch's output stands alone — no silent cross-run mixing
-that the old date-folder layout produced. The `-local` suffix keeps dev
-runs separate from the committed `latest/` so you don't accidentally
-clobber CI's data.
+`benchmarks/results-throughput/` (or `benchmarks/results-probe/` when
+running `--scenario read-probe`), which is wiped at the start of every run
+so each dispatch's output stands alone — no silent cross-run mixing that
+the old date-folder layout produced. The two scenario folders are
+independent: a read-probe run never touches `results-throughput/` and
+vice-versa.
### Other useful invocations
@@ -180,9 +196,8 @@ clobber CI's data.
# Override the per-entry runtime
./benchmarks/scripts/run-local.sh --tier ccx13 --duration 10m
-# Drop the "-local" suffix so the result dir matches the CI naming convention
-# (use this only when you want to commit a one-off re-benchmark)
-./benchmarks/scripts/run-local.sh --tier ccx23 --commit
+# Switch to the read-probe scenario (writes to benchmarks/results-probe/)
+./benchmarks/scripts/run-local.sh --scenario read-probe
```
## Running from GitHub Actions
@@ -198,14 +213,18 @@ Required GitHub secrets:
`benchmark-key`.
After the matrix completes, an `aggregate` job downloads all artifacts, runs
-`chart.py`, and commits `benchmarks/results/latest/` to `main` via a bot
-commit (`git add -A`, so files from a prior dispatch that aren't in this one
-get staged for deletion). No PR — it's a generated artifact. **The
-committed `latest/` always reflects exactly one dispatch**; comparing to
-previous runs is `git log -- benchmarks/results/latest/` or
-`git show <sha>:benchmarks/results/latest/summary.md`. Dated historical
-folders (`benchmarks/results/2026-05-15/` etc.) from before this layout
-change are kept in git for reference but are no longer written to.
+`chart.py`, and commits the matching scenario folder
+(`benchmarks/results-throughput/` for throughput runs,
+`benchmarks/results-probe/` for read-probe runs) to `main` via a bot commit
+(`git add -A`, so files from a prior dispatch that aren't in this one get
+staged for deletion). No PR — it's a generated artifact. **Each scenario
+folder always reflects exactly one dispatch of that scenario**; a
+throughput dispatch never modifies `results-probe/` and vice-versa.
+Comparing to previous runs is `git log -- benchmarks/results-throughput/`
+or `git show <sha>:benchmarks/results-throughput/summary.md`. Dated
+historical folders (`benchmarks/results/2026-05-15/` etc.) from before
+this layout change are kept in git for reference but are no longer written
+to.
## Running against managed ClickHouse
@@ -278,9 +297,12 @@ benchmarks/
reset-managed-ch.sh # Wipes the managed CH DB between matrix entries
loadgen-bootstrap.sh # Cross-compiles + runs loadgen
seed-project.sh # /api/register -> JWT + project token JSON
- chart.py # matplotlib renderer (per-signal charts)
+ chart.py # matplotlib renderer (throughput + read-probe charts)
_ssh.sh # Shared ssh/rsync helpers
- results/ # Committed bench results live here
+ results-throughput/ # Committed throughput results (wiped per dispatch)
+ results-probe/ # Committed read-probe results (wiped per dispatch)
+ results/ # Historical dated folders (not written to anymore)
+ charts.md # Reading guide for every chart chart.py emits
```
## Failure modes & debugging
diff --git a/benchmarks/charts.md b/benchmarks/charts.md
new file mode 100644
index 00000000..32312fba
--- /dev/null
+++ b/benchmarks/charts.md
@@ -0,0 +1,309 @@
+# benchmarks/charts.md — chart reading guide
+
+Every PNG under `benchmarks/results-throughput/` and `benchmarks/results-probe/`
+is rendered by [`benchmarks/scripts/chart.py`](scripts/chart.py) from the JSON
+files in the same folder. This doc explains what each chart shows, what
+question it answers, how to read it, and what to look for. Pair this with
+[README.md](README.md) for the bench-mechanics side.
+
+The two output folders are siblings — throughput runs never touch
+`results-probe/` and vice-versa. Each folder is wiped on every dispatch of
+its scenario, so the committed PNGs always reflect exactly one run.
+
+**File naming.** Per-signal charts: `chart-<name>-<signal>.png` where
+`<signal>` ∈ {`spans`, `metrics`, `logs`}. Cross-signal charts have no
+signal suffix (`chart-signal-mix.png`, `chart-readprobe-signal-mix.png`).
+The headline bar is the special case: `chart-<signal>.png` (no `-headline`
+suffix; it's the most-cited artifact and gets the short name).
+
+---
+
+## Reading guide — throughput
+
+Folder: `benchmarks/results-throughput/`
+
+| Question | Chart |
+|---|---|
+| What's the headline number? | [`chart-<signal>.png`](#t1--chart-signalpng--headline-bars) |
+| How does latency grow with batch size at low concurrency? | [`chart-phase1-batch-<signal>.png`](#t2--chart-phase1-batch-signalpng--phase-1-batch-size-scaling) |
+| Where does the SUT cliff on collector-shape rate? | [`chart-phase2-rate-<signal>.png`](#t3--chart-phase2-rate-signalpng--phase-2-collector-shape-rate-ramp) |
+| Where does the SUT cliff on SDK-fleet-shape rate? | [`chart-phase3-rate-<signal>.png`](#t4--chart-phase3-rate-signalpng--phase-3-sdk-fleet-rate-ramp-conditional) |
+| What does each K items/sec cost in P99? | [`chart-pareto-<signal>.png`](#t5--chart-pareto-signalpng--latencythroughput-pareto) |
+| Does doubling cores double throughput? | [`chart-tier-scaling-<signal>.png`](#t6--chart-tier-scaling-signalpng--tier-scaling) |
+| What worked and what failed, at a glance? | [`chart-cliff-<signal>.png`](#t7--chart-cliff-signalpng--step-status-grid) |
+| Is batching still paying off, or is the SUT dropping batches? | [`chart-batch-efficiency-<signal>.png`](#t8--chart-batch-efficiency-signalpng--batch-efficiency) |
+| At the same hardware, how do spans vs metrics vs logs compare? | [`chart-signal-mix.png`](#t9--chart-signal-mixpng--cross-signal-comparison) |
+
+## Reading guide — read-probe
+
+Folder: `benchmarks/results-probe/`
+
+| Question | Chart |
+|---|---|
+| How many rows can you have before the dashboard cliffs? | [`chart-readprobe-headline-<signal>.png`](#p1--chart-readprobe-headline-signalpng--max-fill-level-by-tier-mode) |
+| How steeply does read latency grow with table size? | [`chart-readprobe-<signal>.png`](#p2--chart-readprobe-signalpng--read-latency-vs-rows-ingested) |
+| How long does it take to fill each level? | [`chart-readprobe-ingest-time-<signal>.png`](#p3--chart-readprobe-ingest-time-signalpng--fill-time-vs-target-rows) |
+| Does more hardware buy a bigger queryable dataset? | [`chart-readprobe-tier-scaling-<signal>.png`](#p4--chart-readprobe-tier-scaling-signalpng--read-probe-tier-scaling) |
+| Which (tier × mode × fill) combos passed? | [`chart-readprobe-cliff-<signal>.png`](#p5--chart-readprobe-cliff-signalpng--read-probe-step-status-grid) |
+| Which read endpoint scales worst on the same hardware? | [`chart-readprobe-signal-mix.png`](#p6--chart-readprobe-signal-mixpng--read-probe-cross-signal) |
+
+---
+
+## Failure-state glossary
+
+The charts use a shared vocabulary for what "failed" means at each step. Most
+status colors are reused across the cliff grids.
+
+| Term | Meaning | Where it shows |
+|---|---|---|
+| **pass** (green) | Step is below the threshold and the loadgen marked it passing. | Cliff grids, `o` markers on lines/scatter. |
+| **soft-cliff** (yellow) | Throughput steps: achieved ≤ 70% of attempted with no error spike — the SUT's workers couldn't keep up with the limiter. Read-probe: the read returned 200 but latency was above threshold (default 5000ms). | Cliff grid yellow cell; on line charts, look for the gap between `actual` (solid) and `attempted` (dashed) lines in T3/T8. |
+| **hard fail** (red) | Throughput: combined HTTP error rate + OTLP `rejected` items exceeded 5% of attempted. Read-probe: the read returned a non-2xx response or errored out. | Cliff grid red cell; `x` marker on every line/scatter chart. |
+| **0 requests completed** / **dead** (dark grey) | The SUT never returned a single OK response in the step's window — usually the container was being restarted by Docker's `restart: on-failure:3`. Shows up as `actualItemsPerSec = 0`. | Cliff grid dark-grey cell; Pareto chart bottom annotation; line charts drop the point (log scale). |
+| **not run** (light grey) | The phase or step didn't execute in this matrix entry (e.g. Phase 2 was skipped because Phase 1 cliffed first, or this fill level was never reached). | Cliff grid light-grey cell. |
+| **failed @ P1 step N** | Run never produced any sustainable throughput. Annotation on headline bars. | T1 hatched bar, T6 dropped point. |
+| **Phase 1 floor** | The Phase 1 P99 latency at the matching batch size — what latency looks like with zero concurrency. Anything above the floor is concurrency tax. | Thin dotted horizontal lines on T3 / T4 middle subplot. |
+
+Numeric thresholds match `benchmarks/loadgen/` defaults: `--ingest-err-threshold 0.05` (5% combined error rate) and `--soft-cliff-ratio 0.7` (70% achieved-vs-attempted).
+
+---
+
+## Throughput charts
+
+### T1 — `chart-<signal>.png` — Headline bars
+
+- **Filename:** `chart-spans.png`, `chart-metrics.png`, `chart-logs.png`
+- **Folder:** `results-throughput/`
+- **What it shows:** Grouped bars of `maxSustainableItemsPerSec` per (tier, mode).
+- **Question answered:** On hardware tier X with DB mode Y, what's the largest sustainable ingest rate for signal Z?
+- **How to read it:**
+ - X axis: hardware tier. Y axis: max sustainable items/sec.
+ - Each bar is annotated with the absolute value and the **phase that produced it** in parentheses (`283,187 (P2)`). P1 = batch-size ramp at 5 req/sec, P2 = collector-shape rate ramp, P3 = SDK-fleet-shape rate ramp.
+ - **Hatched grey bars** signal "all phases failed" — the run never produced any sustainable throughput. The annotation tells you which phase + step was the first to fail (`failed @ P1 step 3`).
+- **What to look for:**
+ - Mode floor — sqlite vs pgch gap shows the cost of the simpler stack.
+ - Phase tag — a P1-sourced headline means the SUT saturated on a single client; P2/P3 means it scaled with concurrency.
+ - Hatched bars vs `0`-height bars: the old chart rendered failures as zero-height bars that looked identical to "not run." Now they're explicitly flagged.
+- **Why it's good:** Headline answer in one image. Disambiguates "failed" from "not measured" — a UX gap the old chart had.
+
+### T2 — `chart-phase1-batch-<signal>.png` — Phase 1 batch-size scaling
+
+- **Folder:** `results-throughput/`
+- **What it shows:** Latency P50/P95/P99 vs batch size at fixed 5 req/sec.
+- **Question answered:** At fixed 5 req/sec, how does per-request latency grow as you fatten the batch?
+- **How to read it:**
+ - X axis: batch size, log scaled. Y axis: ingest latency in ms, log scaled.
+ - Per (tier, mode) there are three lines: P50 (dotted, alpha 0.5), P95 (dash-dot, alpha 0.7), P99 (solid). Only the P99 line gets a legend entry; lower percentiles are colour-coded by the same colour.
+ - `o` markers = step passed. `x` markers = step failed at this batch size.
+- **What to look for:**
+ - **Cliff knee** — the batch size where the line bends sharply upward. That's where the SUT stops scaling linearly with batch.
+ - **Mode spread** — sqlite typically cliffs at ~1K, pgch cruises out to 16K, managed-ch pays a fixed network-RTT floor (visible as a high P50 even at the smallest batch).
+- **Why it's good:** Phase 1 saturates one client; the latency curve here is the "no concurrency" baseline that Phase 2/3 charts overlay as the "P1 floor."
+
+### T3 — `chart-phase2-rate-<signal>.png` — Phase 2 collector-shape rate ramp
+
+- **Folder:** `results-throughput/`
+- **What it shows:** Three stacked subplots over a shared X axis (request rate, log) for the collector-shape rate ramp (fixed fat batch from Phase 1's winner, default cap 16384).
+- **Question answered:** Where does the SUT cliff when a single fat-batch client (like an OTel Collector) ramps its request rate?
+- **How to read it:**
+ - **Top subplot — throughput.** Solid line = `actualItemsPerSec`. Dashed line = `attemptedItemsPerSec` (= batch × target rate). When solid drops below dashed, the SUT can't keep up — that's the soft-cliff. Log Y.
+ - **Middle subplot — latency percentiles.** P50/P95/P99 per run. Thin dotted horizontal lines = **Phase 1 P99 floor at the matching batch size** (the no-concurrency latency). Anything above the floor is concurrency tax. Log Y.
+ - **Bottom subplot — failure signal.** Solid line = HTTP error rate (%). Twin Y axis dotted line (if any rejection happened) = OTLP `rejected` count. The horizontal red dashed line at 5% is the configured failure threshold.
+ - `o` = passed step, `x` = failed step. Steps are sorted by `requestRate` before plotting because the bisection writer records them in execution order (e.g. 5 → 25 → 15 → 10 → 12).
+- **What to look for:**
+ - **Soft cliff vs hard fail** — does the SUT degrade gracefully (solid drifts below dashed in the top subplot) or fall off a cliff (errors spike in the bottom subplot at the same rate)?
+ - **Concurrency tax** — distance between the latency line and the dotted P1 floor.
+ - **Bisection precision** — the X markers should cluster tightly near the cliff. Wide spread means the bisection ran out of steps.
+- **Why it's good:** Strictly supersets the legacy `chart-detail-<signal>.png` (which plotted only P99). Adds the achieved-vs-attempted gap, the error breakdown, and the P1 floor — the three signals you need to diagnose a cliff.
+
+### T4 — `chart-phase3-rate-<signal>.png` — Phase 3 SDK-fleet rate ramp (conditional)
+
+- **Folder:** `results-throughput/`
+- **What it shows:** Same three-subplot layout as T3, but for the SDK-fleet shape: batch fixed at 100 (typical OTel SDK output), request rates from 10 to 10,000.
+- **Question answered:** When each request is tiny and rates are huge, does the SUT cliff in a different place — HTTP/auth/decode path vs raw-insert path?
+- **How to read it:** Identical to T3. The X-axis range is much wider (10–10,000 vs ~1–400 in T3) — Phase 3 stresses request handling rather than batch insertion.
+- **What to look for:**
+ - **Phase ceiling vs Phase 2 ceiling** — Phase 3 typically saturates HTTP/auth before raw insert, so the items/sec ceiling is lower despite higher request rate. If Phase 3's ceiling beats Phase 2's, the SUT is under-utilizing its insert path under fat batches.
+ - **Failures** — at thousands of req/sec, decode and auth checks become measurable; spikes there manifest as bottom-subplot errors before the latency lines bend.
+- **Why it's good:** Only chart that probes the "thousands of small batches" SDK fleet shape. Different bottleneck profile than the collector shape — surfacing it separately keeps the analysis honest.
+- **Note:** This chart is only rendered when at least one run in the folder has a non-empty `phase3` (controlled by the `--phase3-batch-size`/`--phase3-request-rates` loadgen flags).
+
+### T5 — `chart-pareto-<signal>.png` — Latency–throughput Pareto
+
+- **Folder:** `results-throughput/`
+- **What it shows:** Scatter of every Phase 2 and Phase 3 step at `(P99 latency, actualItemsPerSec)`. Both axes log. Color per (tier, mode), `o` = passed, `x` = failed.
+- **Question answered:** What does each extra K items/sec cost you in P99 latency? What's each (tier, mode) Pareto frontier look like?
+- **How to read it:**
+ - X axis: P99 latency in ms, log. Y axis: achieved items/sec, log.
+ - **Faint dotted polyline per run** connects passed steps sorted by P99 — that's the (tier, mode) frontier.
+ - **Up-and-left is better** (more throughput, less latency).
+ - **Zero-throughput failures** can't be plotted on a log Y axis — they're listed as a bottom-margin annotation in red (`Zero-throughput failures: ccx23 / sqlite@P2s1; ccx23 / pgch@P2s1`).
+- **What to look for:**
+ - **Frontier shape** — a flat-then-vertical curve means "lots of room to push more rate before latency rises" (the good case). A steep diagonal means "every K of throughput costs you proportionally in latency" (the bad case).
+ - **Mode separation** — sqlite frontiers usually live in the lower-left (low throughput, low latency). pgch frontiers extend right. managed-ch's RTT floor pushes them rightward at the same throughput levels.
+ - **Failure clustering** — `x` markers near the frontier top-right tell you what the SUT *attempted* before cliffing.
+- **Why it's good:** The only chart that puts the latency-throughput trade-off on a single picture *and* keeps failures visible without crashing on log scale.
+
+### T6 — `chart-tier-scaling-<signal>.png` — Tier scaling
+
+- **Folder:** `results-throughput/`
+- **What it shows:** Line: `maxSustainableItemsPerSec` vs hardware tier (ordinal X: ccx13 → ccx43), one line per mode. Tick labels include vCPU/RAM.
+- **Question answered:** Does doubling cores actually double throughput? Where does extra hardware stop paying?
+- **How to read it:**
+ - X axis: tier in linear order (ccx13, ccx23, ccx33, ccx43 — Hetzner doubles vCPU at each step).
+ - Y axis: max sustainable items/sec.
+ - **Dashed reference line per mode** = perfect linear-vCPU scaling, anchored at the smallest tier present for that mode. If the solid line falls below the dashed, you're paying a scaling penalty.
+- **What to look for:**
+ - **Linear vs diminishing returns** — sqlite often plateaus quickly (single-process write); pgch usually scales better but with a knee at the largest tier (ClickHouse merges saturate disk I/O).
+ - **Missing points** — if a mode has no point at a tier, it failed there. The reference line still projects, giving you a "what perfect scaling would have looked like" anchor.
+- **Why it's good:** Distinguishes "you need a bigger box" from "the bottleneck is elsewhere." A flat-line mode says "more cores won't help."
+
+### T7 — `chart-cliff-<signal>.png` — Step-status grid
+
+- **Folder:** `results-throughput/`
+- **What it shows:** Gantt-style heatmap. Rows = (tier, mode). Columns = step indices grouped by phase (P1 | P2 | P3 with vertical separators).
+- **Question answered:** At a glance, what worked and what failed?
+- **How to read it:**
+ - Each cell is one step of one phase. Cell color: green=pass, yellow=soft-cliff, red=hard fail, dark grey=0 requests completed, light grey=not run.
+ - Inside each cell: `bs=<batch size> r=<request rate>` (e.g. `bs=16K r=25`).
+ - **Right margin:** the headline number for the row, or "failed" in red for all-failed runs.
+- **What to look for:**
+ - **Red rows** — runs that cliffed almost immediately. The old line charts hid these (zero-height bar or invisible single point); the cliff grid makes them legible.
+ - **Yellow + red mix** — bisection territory. Several yellow cells around a red one means the loadgen narrowed the cliff via bisection.
+ - **Missing-column patterns** — sqlite often has Phase 1 reds and no Phase 2 (because Phase 1 cliffed first, so Phase 2 was skipped). pgch's failures usually live in Phase 2 because Phase 1 cleared all batches.
+- **Why it's good:** The most information-dense chart. Especially useful for runs that mostly fail — line charts make those invisible, but the grid shows every step and why.
+
+### T8 — `chart-batch-efficiency-<signal>.png` — Batch efficiency
+
+- **Folder:** `results-throughput/`
+- **What it shows:** Phase 1 throughput vs batch size. Solid = `actualItemsPerSec`, dashed = `attemptedItemsPerSec` (= batch × 5 req/sec).
+- **Question answered:** Is the batch size still paying off, or is the SUT rejecting fat batches?
+- **How to read it:**
+ - X axis: batch size, log. Y axis: items/sec, log.
+ - Solid line = actually-delivered items/sec. Dashed line per run = attempted items/sec (the diagonal a perfect SUT would track).
+ - **Solid below dashed** = the SUT is rejecting items (HTTP errors or OTLP `PartialSuccess.rejected_log_records` etc.).
+ - `x` markers = step failed at that batch size.
+- **What to look for:**
+ - **The point where the solid line peels away from the dashed** — that's the batch size at which more items per request stop helping.
+ - **Failed marker placement** — if `x` is exactly on the dashed line, the SUT accepted the work but errored out at the *next* step. If `x` is below the dashed, the SUT was already rejecting before failing.
+- **Why it's good:** Disentangles "the load gen sent X" from "the SUT accepted X." Two systems with identical headlines can look very different here.
+
+### T9 — `chart-signal-mix.png` — Cross-signal comparison
+
+- **Folder:** `results-throughput/`
+- **What it shows:** Single grouped-bars figure: one group per (tier, mode), three bars per group — spans / metrics / logs.
+- **Question answered:** At the same hardware, how does the cost of each signal compare?
+- **How to read it:**
+ - X axis: (tier × mode) combinations. Y axis: max sustainable items/sec.
+ - Bar colour: signal (spans blue, metrics green, logs red).
+ - **The Y unit varies by signal** — spans/sec vs data points/sec vs log records/sec aren't strictly comparable item-to-item. Use relative shape rather than absolute height: which signal does this SUT do best on?
+- **What to look for:**
+ - **Signal ranking per row** — usually metrics > logs > spans (metrics is the cheapest schema, spans the heaviest), but the ratio shifts by mode.
+ - **Missing bars** — a signal absent from a (tier, mode) row means that matrix entry failed.
+- **Why it's good:** Reveals SUT specialization. A box that's great at spans but bad at logs probably has a string-store bottleneck.
+
+---
+
+## Read-probe charts
+
+### P1 — `chart-readprobe-headline-<signal>.png` — Max fill level by (tier, mode)
+
+- **Folder:** `results-probe/`
+- **What it shows:** Grouped bars of `maxFillLevelPassed` (rows) per (tier, mode). Log Y axis.
+- **Question answered:** How many rows can you have in the relevant table before the dashboard read on this endpoint cliffs?
+- **How to read it:**
+ - X axis: hardware tier. Y axis: max passing fill level (rows, log).
+ - Each bar is annotated with the fill level (`1.0M`) and the read latency at that fill (`540ms`).
+ - **Hatched grey bars** = no fill level passed (the smallest fill level was already over threshold).
+ - The title names the read endpoint being probed (e.g. `/api/metrics/application`).
+- **What to look for:**
+ - **Order-of-magnitude steps** — fill levels usually 100K → 1M → 10M → 100M. A row that maxes at 100K is in serious trouble; 100M is comfortable.
+ - **Mode spread** — sqlite usually maxes out an order of magnitude earlier than pgch.
+- **Why it's good:** The read-probe equivalent of T1. One number per (tier, mode), with the latency-at-max so you know how close to the threshold the bar represents.
+
+### P2 — `chart-readprobe-<signal>.png` — Read latency vs rows ingested
+
+- **Folder:** `results-probe/`
+- **What it shows:** Line: `readLatencyMs` (Y log) vs `rowsIngested` (X log), one line per (tier, mode). Horizontal threshold line (default 5000ms).
+- **Question answered:** How steeply does read latency grow as the table fills up?
+- **How to read it:**
+ - X axis: rows ingested before the probe, log. Y axis: read latency in ms, log.
+ - **Solid horizontal red dashed line** = the threshold from `--read-threshold-ms` (default 5000ms).
+ - `o` = step passed (read OK, under threshold). `x` = failed (read errored or over threshold).
+ - Title names the read path being probed.
+- **What to look for:**
+ - **Slope** — log-log slope ≈ 1 means linear scaling; > 1 means superlinear (degrading); < 1 means sublinear (the index is doing its job).
+ - **Where the line crosses the threshold** — that's the cliff. Anything to the right of the crossing point is unreliable.
+- **Why it's good:** The classic "how does the dashboard hold up?" picture. Slope information is the part you couldn't get from the headline.
+
+### P3 — `chart-readprobe-ingest-time-<signal>.png` — Fill time vs target rows
+
+- **Folder:** `results-probe/`
+- **What it shows:** Line: time taken to reach the target fill level (Y log seconds) vs target fill level (X log rows), one line per (tier, mode).
+- **Question answered:** How long does it take to FILL each level — i.e. how expensive are the writes at high cardinality?
+- **How to read it:**
+ - X axis: target fill level (rows, log). Y axis: ingest seconds elapsed (log).
+ - Lines are read in execution order — they keep climbing as the test fills the table.
+- **What to look for:**
+ - **Curve sag** — if the curve bends upward at higher row counts, write cost is rising (compaction / index thrash). A perfectly linear log-log line means write speed is constant regardless of table size.
+ - **Mode delta** — ClickHouse usually sags around 10–100M because background merges saturate the disk; SQLite usually fails before sagging at all.
+- **Why it's good:** The read-probe view focuses on read latency, but the cost of *getting there* matters too. If filling 100M rows takes hours, that's a different operational story than seconds.
+
+### P4 — `chart-readprobe-tier-scaling-<signal>.png` — Read-probe tier scaling
+
+- **Folder:** `results-probe/`
+- **What it shows:** Line: `maxFillLevelPassed` vs tier ordinal, one line per mode. Log Y. Dashed linear-vCPU reference per mode.
+- **Question answered:** Does more hardware buy you a bigger queryable dataset on this endpoint?
+- **How to read it:** Mirror of T6 but Y is rows instead of items/sec. Same linear-vCPU reference convention.
+- **What to look for:**
+ - **Plateau** — flat lines mean read scaling is bottlenecked elsewhere (network RTT for managed-ch, single-thread query plan for sqlite).
+ - **Below-reference gap** — distance between solid and dashed = how much the read path leaves on the table.
+- **Why it's good:** Throughput tier-scaling answers "can I serve more SDK traffic?". This one answers "can I keep the dashboard working at a bigger scale?". Different question, same chart shape.
+
+### P5 — `chart-readprobe-cliff-<signal>.png` — Read-probe step-status grid
+
+- **Folder:** `results-probe/`
+- **What it shows:** Gantt-style heatmap. Rows = (tier, mode). Columns = fill levels (union across all runs).
+- **Question answered:** Which (tier × mode × fill level) combos passed?
+- **How to read it:**
+ - Each cell is one probe of one fill level. Cell color: green=passed, yellow=read OK but over threshold, red=read errored, light-grey=not run.
+ - Inside each cell: the read latency (`540ms`). Empty when not run.
+ - Right margin: `max <fill>` if any fill passed, or "no fill passed" in red.
+ - Column headers show the fill levels themselves (`100K`, `1.0M`, etc.).
+- **What to look for:**
+ - **Yellow vs red** — yellow means "the read came back but slowly," red means "the read errored out." Yellow is often actionable (tune the query plan); red usually means the SUT itself is unwell.
+ - **Row patterns** — sqlite rows often have a single green cell and the rest red/yellow; pgch rows usually have a few green cells before yellow.
+- **Why it's good:** The cliff equivalent of T7 but for read-probe. Lets you see the failure pattern across the full matrix on one page.
+
+### P6 — `chart-readprobe-signal-mix.png` — Read-probe cross-signal
+
+- **Folder:** `results-probe/`
+- **What it shows:** Single grouped-bars figure: one group per (tier, mode), three bars per group — spans / metrics / logs. Y = `maxFillLevelPassed` (log).
+- **Question answered:** Which read endpoint scales worst on the same hardware?
+- **How to read it:**
+ - Same shape as T9 but for read-probe.
+ - Legend includes the read path for each signal (`spans (/api/endpoints/grouped)`), since each signal probes a different endpoint.
+ - Log Y because fill levels span several orders of magnitude.
+- **What to look for:**
+ - **Signal worst-case** — usually `spans` (`/api/endpoints/grouped` joins on a high-cardinality table) is the first to cliff.
+ - **Mode-invariant cliffs** — if every mode's bar for `spans` is short, the bottleneck is the query plan, not the storage engine.
+- **Why it's good:** Three read paths × N (tier, mode) cells on one image. Shows you which API endpoint to optimize first.
+
+---
+
+## Generating these locally
+
+```bash
+python3 -m venv .venv-bench
+.venv-bench/bin/pip install matplotlib
+
+# Throughput results
+.venv-bench/bin/python benchmarks/scripts/chart.py benchmarks/results-throughput
+
+# Read-probe results
+.venv-bench/bin/python benchmarks/scripts/chart.py benchmarks/results-probe
+```
+
+`chart.py` auto-detects scenarios per JSON file — running it against a folder
+that mixes both scenarios will produce the union of charts. The wrong-scenario
+render functions are no-ops, so you'll just see the charts that have data.
diff --git a/benchmarks/scripts/chart.py b/benchmarks/scripts/chart.py
index b141827a..aef98509 100755
--- a/benchmarks/scripts/chart.py
+++ b/benchmarks/scripts/chart.py
@@ -1,18 +1,16 @@
#!/usr/bin/env python3
-"""Render per-signal benchmark charts from a directory of loadgen JSONs.
+"""Render benchmark charts from a directory of loadgen JSONs.
Inputs: a directory containing N JSON files (one per matrix entry), each in
the schema emitted by benchmarks/loadgen/. Files without a "signal" field
(old pre-OTLP-split format) are skipped with a log line.
-Outputs depend on which scenarios are present in the directory:
- Throughput scenario:
- - chart-spans.png / chart-metrics.png / chart-logs.png bar: max items/sec
- - chart-detail-<sig>.png line: ingest p99 vs request rate
- Read-probe scenario:
- - chart-readprobe-<sig>.png line: read latency vs fill level
- Always:
- - summary.md tables per signal/scenario
+The script auto-detects scenarios per file (each JSON has `scenario`). Run it
+against `benchmarks/results-throughput/` for throughput charts, or
+`benchmarks/results-probe/` for read-probe charts. Mixed folders also work —
+each render function filters to its own scenario.
+
+See benchmarks/charts.md for a reader's guide to each output.
"""
import json
@@ -23,11 +21,25 @@ import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
+from matplotlib.patches import Patch, Rectangle
+
TIER_ORDER = ["ccx13", "ccx23", "ccx33", "ccx43"]
-MODE_ORDER = ["sqlite", "pgch"]
+MODE_ORDER = ["sqlite", "pgch", "managed-ch"]
SIGNALS = ["spans", "metrics", "logs"]
-MODE_COLORS = {"sqlite": "#4f9fff", "pgch": "#ff9f4f"}
+
+MODE_COLORS = {
+ "sqlite": "#4f9fff",
+ "pgch": "#ff9f4f",
+ "managed-ch": "#7c4fff",
+}
+SIGNAL_COLORS = {
+ "spans": "#1f77b4",
+ "metrics": "#2ca02c",
+ "logs": "#d62728",
+}
+PHASE_LABEL = {"phase1": "P1", "phase2": "P2", "phase3": "P3"}
+
ITEM_LABEL = {
"spans": "spans/sec",
"metrics": "data points/sec",
@@ -38,6 +50,28 @@ ROW_LABEL = {
"metrics": "metric_points rows",
"logs": "log_records rows",
}
+TIER_META = {
+ "ccx13": "2 vCPU / 8 GB",
+ "ccx23": "4 vCPU / 16 GB",
+ "ccx33": "8 vCPU / 32 GB",
+ "ccx43": "16 vCPU / 64 GB",
+}
+TIER_VCPU = {"ccx13": 2, "ccx23": 4, "ccx33": 8, "ccx43": 16}
+
+# Match loadgen defaults so chart thresholds line up with the JSON's `passed` bit.
+SOFT_CLIFF_RATIO = 0.7
+ERR_THRESHOLD_PCT = 5.0
+
+STATUS_COLOR = {
+ "pass": "#4caf50",
+ "soft-cliff": "#ffc107",
+ "hard-fail": "#f44336",
+ "dead": "#616161",
+ "missing": "#f5f5f5",
+}
+
+
+# ---------- Loading + small helpers ----------
def load_runs(results_dir: Path) -> list[dict]:
@@ -68,71 +102,612 @@ def mode_rank(m: str) -> int:
return MODE_ORDER.index(m) if m in MODE_ORDER else len(MODE_ORDER)
-def render_throughput_bar(runs: list[dict], signal: str, out: Path) -> None:
- runs = [r for r in runs if r["signal"] == signal and r["scenario"] == "throughput"]
- if not runs:
+def mode_color(m: str) -> str:
+ return MODE_COLORS.get(m, "#777")
+
+
+def read_path_for_signal(signal: str) -> str:
+ return {
+ "spans": "/api/endpoints/grouped",
+ "metrics": "/api/metrics/application",
+ "logs": "/api/logs",
+ }.get(signal, "")
+
+
+def fmt_count(n: float) -> str:
+ n = int(n or 0)
+ if n >= 1_000_000_000:
+ return f"{n/1_000_000_000:.1f}B"
+ if n >= 1_000_000:
+ return f"{n/1_000_000:.1f}M"
+ if n >= 1_000:
+ return f"{n/1_000:.0f}K"
+ return f"{n}"
+
+
+def run_label(run: dict) -> str:
+ return f"{run['tier']} / {run['mode']}"
+
+
+def iter_runs(runs: list[dict], signal: str, scenario: str) -> list[dict]:
+ out = [r for r in runs if r["signal"] == signal and r.get("scenario", "throughput") == scenario]
+ return sorted(out, key=lambda r: (tier_rank(r["tier"]), mode_rank(r["mode"])))
+
+
+def phase_steps(run: dict, phase_key: str, sort_by: str = "step") -> list[dict]:
+ """Return phase steps in a chosen sort order. Phase 2/3 record execution
+ order, not rate order, so sorting before plotting keeps lines monotonic."""
+ phase = run.get(phase_key) or {}
+ steps = phase.get("steps") or []
+ if not steps:
+ return []
+ if sort_by == "rate":
+ return sorted(steps, key=lambda s: (s.get("requestRate", 0), s.get("step", 0)))
+ if sort_by == "batch":
+ return sorted(steps, key=lambda s: (s.get("batchSize", 0), s.get("step", 0)))
+ return sorted(steps, key=lambda s: s.get("step", 0))
+
+
+def step_status(step: dict) -> str:
+ """Categorize a step as pass / soft-cliff / hard-fail / dead.
+ `pass` = loadgen marked it passed.
+ `dead` = no requests completed (actual=0 or attempted=0).
+ `soft-cliff` = achieved fell below SOFT_CLIFF_RATIO × attempted with no error spike.
+ `hard-fail` = combined error rate above threshold (default 5%).
+ """
+ if step.get("passed"):
+ return "pass"
+ actual = step.get("actualItemsPerSec", 0) or 0
+ attempted = step.get("attemptedItemsPerSec", 0) or 0
+ err_rate_pct = ((step.get("ingest") or {}).get("errRate", 0) or 0) * 100
+ if attempted == 0 or actual == 0:
+ return "dead"
+ if err_rate_pct > ERR_THRESHOLD_PCT:
+ return "hard-fail"
+ if attempted > 0 and actual / attempted < SOFT_CLIFF_RATIO:
+ return "soft-cliff"
+ return "hard-fail"
+
+
+def headline_source(run: dict) -> tuple[str, dict] | None:
+ """Return (phase_key, step) whose actualItemsPerSec equals the headline.
+ Returns None if the run has no passing step."""
+ headline = run.get("maxSustainableItemsPerSec", 0) or 0
+ if headline <= 0:
+ return None
+ best = None
+ best_delta = float("inf")
+ for phase_key in ("phase1", "phase2", "phase3"):
+ for step in phase_steps(run, phase_key):
+ if not step.get("passed"):
+ continue
+ actual = step.get("actualItemsPerSec", 0) or 0
+ delta = abs(actual - headline)
+ if delta < best_delta:
+ best = (phase_key, step)
+ best_delta = delta
+ return best
+
+
+def first_failure(run: dict) -> tuple[str, dict] | None:
+ for phase_key in ("phase1", "phase2", "phase3"):
+ for step in phase_steps(run, phase_key):
+ if not step.get("passed"):
+ return (phase_key, step)
+ return None
+
+
+def phase1_p99_at_batch(run: dict, batch_size: int) -> float | None:
+ for step in phase_steps(run, "phase1"):
+ if step.get("batchSize") == batch_size and step.get("passed"):
+ return (step.get("ingest") or {}).get("p99")
+ return None
+
+
+# ---------- Throughput charts ----------
+
+
+def render_headline_bar(runs: list[dict], signal: str, out: Path) -> None:
+ sig_runs = iter_runs(runs, signal, "throughput")
+ if not sig_runs:
return
- fig, ax = plt.subplots(figsize=(9, 5))
- tiers_present = sorted({r["tier"] for r in runs}, key=tier_rank)
- modes_present = sorted({r["mode"] for r in runs}, key=mode_rank)
+ tiers_present = sorted({r["tier"] for r in sig_runs}, key=tier_rank)
+ modes_present = sorted({r["mode"] for r in sig_runs}, key=mode_rank)
+ if not tiers_present or not modes_present:
+ return
+
+ fig, ax = plt.subplots(figsize=(max(8, len(tiers_present) * 2.5), 6))
+
+ values = [r.get("maxSustainableItemsPerSec", 0) or 0 for r in sig_runs]
+ ymax = max(values) if values else 1
+ ghost_height = max(ymax * 0.02, 1)
width = 0.8 / max(len(modes_present), 1)
- x = list(range(len(tiers_present)))
+ x_centers = list(range(len(tiers_present)))
for mi, mode in enumerate(modes_present):
- ys = []
- for tier in tiers_present:
- match = [r for r in runs if r["tier"] == tier and r["mode"] == mode]
- ys.append(match[0].get("maxSustainableItemsPerSec", 0) if match else 0)
- offsets = [xi + (mi - (len(modes_present) - 1) / 2) * width for xi in x]
- bars = ax.bar(offsets, ys, width=width, label=mode, color=MODE_COLORS.get(mode, "#777"))
- for b, y in zip(bars, ys):
- ax.annotate(f"{int(y):,}", xy=(b.get_x() + b.get_width() / 2, y),
- xytext=(0, 3), textcoords="offset points",
- ha="center", va="bottom", fontsize=9)
+ for ti, tier in enumerate(tiers_present):
+ match = [r for r in sig_runs if r["tier"] == tier and r["mode"] == mode]
+ x = ti + (mi - (len(modes_present) - 1) / 2) * width
+ if not match:
+ continue
+ run = match[0]
+ value = run.get("maxSustainableItemsPerSec", 0) or 0
+ if value > 0:
+ ax.bar(x, value, width=width, color=mode_color(mode))
+ src = headline_source(run)
+ phase_tag = f" ({PHASE_LABEL[src[0]]})" if src else ""
+ ax.annotate(
+ f"{int(value):,}{phase_tag}",
+ xy=(x, value), xytext=(0, 4), textcoords="offset points",
+ ha="center", va="bottom", fontsize=8,
+ )
+ else:
+ ax.bar(x, ghost_height, width=width, color="#dddddd",
+ edgecolor="#999999", hatch="///", linewidth=0.7)
+ fail = first_failure(run)
+ if fail:
+ pkey, step = fail
+ label = f"failed @ {PHASE_LABEL[pkey]} step {step.get('step', '?')}"
+ else:
+ label = "failed"
+ ax.annotate(
+ label, xy=(x, ghost_height), xytext=(0, 4), textcoords="offset points",
+ ha="center", va="bottom", fontsize=7, color="#aa3333",
+ )
- ax.set_xticks(x)
+ ax.set_xticks(x_centers)
ax.set_xticklabels(tiers_present)
ax.set_ylabel(f"Max sustainable {ITEM_LABEL[signal]}")
- ax.set_title(f"Traceway: max sustainable {signal} ingest by hardware tier\n"
- "(failure = combined HTTP error + OTLP rejected > 5%)")
- ax.legend(title="DB mode")
+ ax.set_title(
+ f"Traceway: max sustainable {signal} ingest by hardware tier\n"
+ "Phase tag (P1/P2/P3) shows which workload shape produced the headline"
+ )
+
+ legend_handles = [Patch(facecolor=mode_color(m), label=m) for m in modes_present]
+ legend_handles.append(Patch(facecolor="#dddddd", edgecolor="#999999", hatch="///", label="all phases failed"))
+ ax.legend(handles=legend_handles, title="DB mode", loc="upper left")
+
ax.grid(axis="y", linestyle=":", alpha=0.4)
+ ax.set_ylim(0, ymax * 1.3 if ymax > 0 else 1)
+
fig.tight_layout()
fig.savefig(out, dpi=130)
plt.close(fig)
-def render_throughput_detail(runs: list[dict], signal: str, out: Path) -> None:
- runs = [r for r in runs if r["signal"] == signal and r["scenario"] == "throughput"]
- if not runs:
+def render_phase1_latency(runs: list[dict], signal: str, out: Path) -> None:
+ sig_runs = [r for r in iter_runs(runs, signal, "throughput") if phase_steps(r, "phase1")]
+ if not sig_runs:
+ return
+
+ fig, ax = plt.subplots(figsize=(11, 6))
+ plotted = False
+
+ for run in sig_runs:
+ color = mode_color(run["mode"])
+ steps = phase_steps(run, "phase1", sort_by="batch")
+ batches = [s["batchSize"] for s in steps]
+ p50 = [(s.get("ingest") or {}).get("p50", 0) for s in steps]
+ p95 = [(s.get("ingest") or {}).get("p95", 0) for s in steps]
+ p99 = [(s.get("ingest") or {}).get("p99", 0) for s in steps]
+ passed = [bool(s.get("passed")) for s in steps]
+
+ def _safe(xs, ys):
+ return [(x, y) for x, y in zip(xs, ys) if y and y > 0]
+
+ for percs, ls, alpha in [(p50, ":", 0.5), (p95, "-.", 0.7), (p99, "-", 1.0)]:
+ pts = _safe(batches, percs)
+ if not pts:
+ continue
+ xs, ys = zip(*pts)
+ label = run_label(run) if ls == "-" else None
+ ax.plot(xs, ys, linestyle=ls, color=color, alpha=alpha, label=label)
+ plotted = True
+
+ for b, lat, p in zip(batches, p99, passed):
+ if lat and lat > 0:
+ ax.plot(b, lat,
+ marker=("o" if p else "x"),
+ color=color, markersize=8,
+ markeredgewidth=2 if not p else 1, linestyle="")
+
+ if not plotted:
+ plt.close(fig)
+ return
+
+ ax.set_xlabel("Batch size (items/request, log)")
+ ax.set_ylabel("Ingest latency (ms, log)\nP50 ·· P95 -· P99 —")
+ ax.set_xscale("log")
+ ax.set_yscale("log")
+ ax.set_title(
+ f"Phase 1 — batch-size scaling — {signal}\n"
+ "Fixed 5 req/sec. Solid line = P99 per run, dot/dash = P50/P95. × = step failed."
+ )
+ ax.grid(True, which="both", linestyle=":", alpha=0.4)
+ ax.legend(loc="upper left", fontsize=9, title="P99 line per run")
+ fig.tight_layout()
+ fig.savefig(out, dpi=130)
+ plt.close(fig)
+
+
+def _render_rate_ramp(runs: list[dict], signal: str, phase_key: str, subtitle: str, out: Path) -> None:
+ sig_runs = [r for r in iter_runs(runs, signal, "throughput") if phase_steps(r, phase_key)]
+ if not sig_runs:
+ return
+
+ fig, (ax1, ax2, ax3) = plt.subplots(3, 1, figsize=(11, 12), sharex=True)
+ plotted = False
+
+ for run in sig_runs:
+ color = mode_color(run["mode"])
+ label = run_label(run)
+ steps = phase_steps(run, phase_key, sort_by="rate")
+
+ rates = [s["requestRate"] for s in steps]
+ actual = [s["actualItemsPerSec"] for s in steps]
+ attempted = [s["attemptedItemsPerSec"] for s in steps]
+ p50 = [(s.get("ingest") or {}).get("p50", 0) for s in steps]
+ p95 = [(s.get("ingest") or {}).get("p95", 0) for s in steps]
+ p99 = [(s.get("ingest") or {}).get("p99", 0) for s in steps]
+ errs_pct = [((s.get("ingest") or {}).get("errRate", 0) or 0) * 100 for s in steps]
+ passed = [bool(s.get("passed")) for s in steps]
+
+ # Top: throughput. Drop zeros for log scale.
+ act_pts = [(r, a) for r, a in zip(rates, actual) if a and a > 0]
+ att_pts = [(r, a) for r, a in zip(rates, attempted) if a and a > 0]
+ if act_pts:
+ xs, ys = zip(*act_pts)
+ ax1.plot(xs, ys, linestyle="-", color=color, label=label)
+ plotted = True
+ if att_pts:
+ xs, ys = zip(*att_pts)
+ ax1.plot(xs, ys, linestyle="--", color=color, alpha=0.4)
+ for r, a, p in zip(rates, actual, passed):
+ if a and a > 0:
+ ax1.plot(r, a, marker=("o" if p else "x"), color=color,
+ markersize=8, markeredgewidth=2 if not p else 1, linestyle="")
+
+ # Middle: latency percentiles.
+ for percs, ls, alpha in [(p50, ":", 0.5), (p95, "-.", 0.7), (p99, "-", 1.0)]:
+ pts = [(r, l) for r, l in zip(rates, percs) if l and l > 0]
+ if pts:
+ xs, ys = zip(*pts)
+ ax2.plot(xs, ys, linestyle=ls, color=color, alpha=alpha)
+ # P1 floor overlay
+ fixed_batch = (run.get(phase_key) or {}).get("fixedBatchSize", 0)
+ if fixed_batch:
+ floor = phase1_p99_at_batch(run, fixed_batch)
+ if floor:
+ ax2.axhline(floor, linestyle=":", color=color, alpha=0.35, linewidth=1)
+ for r, lat, p in zip(rates, p99, passed):
+ if lat and lat > 0:
+ ax2.plot(r, lat, marker=("o" if p else "x"), color=color,
+ markersize=8, markeredgewidth=2 if not p else 1, linestyle="")
+
+ # Bottom: error %
+ ax3.plot(rates, errs_pct, linestyle="-", color=color, alpha=0.5)
+ for r, e, p in zip(rates, errs_pct, passed):
+ ax3.plot(r, e, marker=("o" if p else "x"), color=color,
+ markersize=8, markeredgewidth=2 if not p else 1, linestyle="")
+
+ # Bottom right twin: rejected
+ rej = [s.get("rejected", 0) for s in steps]
+ if any(rej):
+ if not hasattr(ax3, "_rej_twin"):
+ ax3._rej_twin = ax3.twinx()
+ ax3._rej_twin.plot(rates, rej, linestyle=":", color=color, alpha=0.4)
+
+ if not plotted:
+ plt.close(fig)
+ return
+
+ ax1.set_ylabel(f"items/sec (log)\n— actual, -- attempted")
+ ax1.set_xscale("log")
+ ax1.set_yscale("log")
+ ax1.grid(True, which="both", linestyle=":", alpha=0.4)
+ ax1.legend(loc="lower right", fontsize=8)
+
+ ax2.set_ylabel("Ingest latency (ms, log)\nP50 ·· P95 -· P99 —\n(thin dotted = P1 floor)")
+ ax2.set_yscale("log")
+ ax2.grid(True, which="both", linestyle=":", alpha=0.4)
+
+ ax3.axhline(ERR_THRESHOLD_PCT, linestyle="--", color="#cc3333", alpha=0.5, linewidth=1)
+ ax3.set_ylabel("Error rate (%)\n× = failed step")
+ ax3.set_xlabel("Request rate (req/sec, log)")
+ ax3.set_xscale("log")
+ ax3.grid(True, which="both", linestyle=":", alpha=0.4)
+ if hasattr(ax3, "_rej_twin"):
+ ax3._rej_twin.set_ylabel("OTLP rejected items (dotted)", color="#666666")
+
+ fig.suptitle(f"{subtitle} — {signal}\n× marker = failed step (err > 5% or soft-cliff). Dotted P1 floor = no-concurrency latency.")
+ fig.tight_layout(rect=(0, 0, 1, 0.97))
+ fig.savefig(out, dpi=130)
+ plt.close(fig)
+
+
+def render_phase2_rate(runs: list[dict], signal: str, out: Path) -> None:
+ _render_rate_ramp(runs, signal, "phase2",
+ "Phase 2 — collector-shape rate ramp (fixed fat batch)", out)
+
+
+def render_phase3_rate(runs: list[dict], signal: str, out: Path) -> None:
+ # Only render when at least one run has phase3 data.
+ any_p3 = any(phase_steps(r, "phase3") for r in iter_runs(runs, signal, "throughput"))
+ if not any_p3:
+ return
+ _render_rate_ramp(runs, signal, "phase3",
+ "Phase 3 — SDK-fleet rate ramp (small batch, high rate)", out)
+
+
+def render_pareto(runs: list[dict], signal: str, out: Path) -> None:
+ sig_runs = iter_runs(runs, signal, "throughput")
+ if not sig_runs:
+ return
+
+ fig, ax = plt.subplots(figsize=(11, 7))
+ plotted = False
+ dead_runs = [] # (run_label, list_of_phase_keys+step_n)
+
+ for run in sig_runs:
+ color = mode_color(run["mode"])
+ label = run_label(run)
+
+ passed_pts = []
+ failed_pts = []
+ for phase_key in ("phase2", "phase3"):
+ for step in phase_steps(run, phase_key):
+ p99 = (step.get("ingest") or {}).get("p99", 0)
+ actual = step.get("actualItemsPerSec", 0) or 0
+ if p99 <= 0 or actual <= 0:
+ if step.get("passed") is False:
+ dead_runs.append((label, f"{PHASE_LABEL[phase_key]}s{step.get('step', '?')}"))
+ continue
+ if step.get("passed"):
+ passed_pts.append((p99, actual))
+ else:
+ failed_pts.append((p99, actual))
+
+ if passed_pts:
+ sorted_pts = sorted(passed_pts, key=lambda p: p[0])
+ xs, ys = zip(*sorted_pts)
+ ax.plot(xs, ys, linestyle=":", color=color, alpha=0.4)
+ ax.scatter(xs, ys, marker="o", color=color, s=70, edgecolor="black",
+ linewidth=0.5, label=label, zorder=3)
+ plotted = True
+ if failed_pts:
+ xs, ys = zip(*failed_pts)
+ ax.scatter(xs, ys, marker="x", color=color, s=100, linewidth=2, zorder=4)
+ plotted = True
+
+ if not plotted and not dead_runs:
+ plt.close(fig)
+ return
+
+ ax.set_xlabel("Ingest P99 latency (ms, log)")
+ ax.set_ylabel(f"Achieved {ITEM_LABEL[signal]} (log)")
+ if plotted:
+ ax.set_xscale("log")
+ ax.set_yscale("log")
+ ax.set_title(
+ f"Latency–throughput Pareto — {signal}\n"
+ "Every Phase 2/3 step. ○ = passed, × = failed. Up-and-left is better."
+ )
+ ax.grid(True, which="both", linestyle=":", alpha=0.4)
+
+ # Annotate runs that had only dead (0-actual) failures
+ if dead_runs:
+ dead_by_run: dict[str, list[str]] = {}
+ for lab, key in dead_runs:
+ dead_by_run.setdefault(lab, []).append(key)
+ text = "Zero-throughput failures: " + "; ".join(
+ f"{lab}@{','.join(keys)}" for lab, keys in dead_by_run.items()
+ )
+ ax.text(0.5, -0.15, text, transform=ax.transAxes, ha="center", va="top",
+ fontsize=8, color="#aa3333")
+ fig.subplots_adjust(bottom=0.18)
+
+ if plotted:
+ ax.legend(loc="upper left", fontsize=9)
+ fig.tight_layout()
+ fig.savefig(out, dpi=130)
+ plt.close(fig)
+
+
+def render_tier_scaling(runs: list[dict], signal: str, out: Path) -> None:
+ sig_runs = iter_runs(runs, signal, "throughput")
+ if not sig_runs:
+ return
+
+ tiers_present = sorted({r["tier"] for r in sig_runs}, key=tier_rank)
+ modes_present = sorted({r["mode"] for r in sig_runs}, key=mode_rank)
+ if not tiers_present:
return
fig, ax = plt.subplots(figsize=(10, 6))
plotted = False
+ x_positions = list(range(len(tiers_present)))
- for run in sorted(runs, key=lambda r: (tier_rank(r["tier"]), mode_rank(r["mode"]))):
- steps = (run.get("phase2") or {}).get("steps") or []
- # Bisection appends steps in execution order, not rate order. Sort
- # by requestRate so the line draws monotonically across the X axis.
- steps = sorted([s for s in steps if s.get("ingest")], key=lambda s: s.get("requestRate", 0))
- xs = [s["requestRate"] for s in steps]
- ys = [s["ingest"]["p99"] for s in steps]
+ for mode in modes_present:
+ xs, ys = [], []
+ for i, tier in enumerate(tiers_present):
+ match = [r for r in sig_runs if r["tier"] == tier and r["mode"] == mode]
+ if not match:
+ continue
+ value = match[0].get("maxSustainableItemsPerSec", 0) or 0
+ if value <= 0:
+ continue
+ xs.append(i)
+ ys.append(value)
if not xs:
continue
- ax.plot(xs, ys, marker="o", label=f"{run['tier']} / {run['mode']}")
+ ax.plot(xs, ys, marker="o", color=mode_color(mode), markersize=9,
+ linewidth=2, label=mode)
+ for x, y in zip(xs, ys):
+ ax.annotate(fmt_count(y), xy=(x, y), xytext=(0, 8),
+ textcoords="offset points", ha="center", va="bottom", fontsize=9)
+ # Linear-vCPU reference, anchored at the smallest tier of THIS mode
+ base_x = xs[0]
+ base_y = ys[0]
+ base_vcpu = TIER_VCPU.get(tiers_present[base_x], 1)
+ ref_xs = list(range(len(tiers_present)))
+ ref_ys = [base_y * (TIER_VCPU.get(tiers_present[i], 1) / base_vcpu) for i in ref_xs]
+ ax.plot(ref_xs, ref_ys, linestyle="--", color=mode_color(mode), alpha=0.3, linewidth=1)
plotted = True
if not plotted:
plt.close(fig)
return
- ax.set_xlabel("Phase 2 request rate (req/sec)")
- ax.set_ylabel("Ingest P99 latency (ms)")
- ax.set_title(f"Ingest latency vs request rate — {signal}")
+ ax.set_xticks(x_positions)
+ ax.set_xticklabels([f"{t}\n{TIER_META.get(t, '')}" for t in tiers_present])
+ ax.set_ylabel(f"Max sustainable {ITEM_LABEL[signal]}")
+ ax.set_title(
+ f"Tier scaling — {signal}\n"
+ "Dashed = linear-vCPU reference (anchored at smallest present tier). "
+ "Distance below dashed = scaling penalty."
+ )
+ ax.legend(title="DB mode", loc="upper left")
+ ax.grid(True, linestyle=":", alpha=0.4)
+ fig.tight_layout()
+ fig.savefig(out, dpi=130)
+ plt.close(fig)
+
+
+def render_cliff_grid(runs: list[dict], signal: str, out: Path) -> None:
+ sig_runs = iter_runs(runs, signal, "throughput")
+ if not sig_runs:
+ return
+
+ n_p1 = max((len(phase_steps(r, "phase1")) for r in sig_runs), default=0)
+ n_p2 = max((len(phase_steps(r, "phase2")) for r in sig_runs), default=0)
+ n_p3 = max((len(phase_steps(r, "phase3")) for r in sig_runs), default=0)
+ total_cols = n_p1 + n_p2 + n_p3
+ if total_cols == 0:
+ return
+
+ n_rows = len(sig_runs)
+ fig_w = max(11, total_cols * 1.55 + 4)
+ fig_h = max(4.5, n_rows * 0.95 + 2.2)
+ fig, ax = plt.subplots(figsize=(fig_w, fig_h))
+
+ for ri, run in enumerate(sig_runs):
+ y = n_rows - ri - 1
+ ax.text(-0.5, y + 0.5, run_label(run), ha="right", va="center", fontsize=10, fontweight="bold")
+
+ col_x = 0
+ for pk, n_cols in [("phase1", n_p1), ("phase2", n_p2), ("phase3", n_p3)]:
+ steps = phase_steps(run, pk, sort_by="step")
+ for ci in range(n_cols):
+ x = col_x + ci
+ if ci < len(steps):
+ step = steps[ci]
+ color = STATUS_COLOR[step_status(step)]
+ bs = step.get("batchSize", 0)
+ r = step.get("requestRate", 0)
+ rect = Rectangle((x, y), 1, 1, facecolor=color,
+ edgecolor="white", linewidth=1.5)
+ ax.add_patch(rect)
+ ax.text(x + 0.5, y + 0.5, f"bs={fmt_count(bs)}\nr={r:g}",
+ ha="center", va="center", fontsize=7.5)
+ else:
+ rect = Rectangle((x, y), 1, 1, facecolor=STATUS_COLOR["missing"],
+ edgecolor="white", linewidth=1.5)
+ ax.add_patch(rect)
+ col_x += n_cols
+
+ headline = run.get("maxSustainableItemsPerSec", 0) or 0
+ margin = total_cols + 0.35
+ if headline > 0:
+ ax.text(margin, y + 0.5, f"{fmt_count(headline)}/s",
+ ha="left", va="center", fontsize=10, fontweight="bold")
+ else:
+ ax.text(margin, y + 0.5, "failed",
+ ha="left", va="center", fontsize=10, color="#cc3333", fontweight="bold")
+
+ # Phase headers + separators
+ headers = []
+ if n_p1 > 0:
+ headers.append((n_p1 / 2.0, "Phase 1 (batch ramp)"))
+ if n_p2 > 0:
+ headers.append((n_p1 + n_p2 / 2.0, "Phase 2 (collector rate ramp)"))
+ if n_p3 > 0:
+ headers.append((n_p1 + n_p2 + n_p3 / 2.0, "Phase 3 (SDK-fleet rate ramp)"))
+ for cx, label in headers:
+ ax.text(cx, n_rows + 0.25, label, ha="center", va="bottom",
+ fontsize=10.5, fontweight="bold")
+
+ for sep in [n_p1, n_p1 + n_p2]:
+ if 0 < sep < total_cols:
+ ax.plot([sep, sep], [0, n_rows], color="black", linewidth=2)
+
+ ax.set_xlim(-4, total_cols + 5)
+ ax.set_ylim(-0.6, n_rows + 0.9)
+ ax.set_aspect("equal")
+ ax.axis("off")
+
+ legend = [
+ Patch(facecolor=STATUS_COLOR["pass"], label="pass"),
+ Patch(facecolor=STATUS_COLOR["soft-cliff"], label=f"soft-cliff (<{int(SOFT_CLIFF_RATIO*100)}% achieved)"),
+ Patch(facecolor=STATUS_COLOR["hard-fail"], label=f"hard fail (err > {ERR_THRESHOLD_PCT:.0f}%)"),
+ Patch(facecolor=STATUS_COLOR["dead"], label="0 requests completed"),
+ Patch(facecolor=STATUS_COLOR["missing"], edgecolor="#cccccc", label="not run"),
+ ]
+ ax.legend(handles=legend, loc="lower center", bbox_to_anchor=(0.5, -0.08),
+ ncol=5, fontsize=9, frameon=False)
+ fig.suptitle(f"Step-status grid — {signal}\nbs = batch size, r = request rate (req/sec)",
+ fontsize=12)
+ fig.tight_layout(rect=(0, 0.04, 1, 0.95))
+ fig.savefig(out, dpi=130, bbox_inches="tight")
+ plt.close(fig)
+
+
+def render_batch_efficiency(runs: list[dict], signal: str, out: Path) -> None:
+ sig_runs = [r for r in iter_runs(runs, signal, "throughput") if phase_steps(r, "phase1")]
+ if not sig_runs:
+ return
+
+ fig, ax = plt.subplots(figsize=(11, 6))
+ plotted = False
+
+ for run in sig_runs:
+ color = mode_color(run["mode"])
+ steps = phase_steps(run, "phase1", sort_by="batch")
+ batches = [s["batchSize"] for s in steps]
+ actual = [s["actualItemsPerSec"] for s in steps]
+ attempted = [s["attemptedItemsPerSec"] for s in steps]
+ passed = [bool(s.get("passed")) for s in steps]
+
+ # log scale; drop zeros
+ act_pts = [(b, a) for b, a in zip(batches, actual) if a and a > 0]
+ att_pts = [(b, a) for b, a in zip(batches, attempted) if a and a > 0]
+
+ if act_pts:
+ xs, ys = zip(*act_pts)
+ ax.plot(xs, ys, linestyle="-", color=color, marker="o", label=run_label(run))
+ plotted = True
+ if att_pts:
+ xs, ys = zip(*att_pts)
+ ax.plot(xs, ys, linestyle="--", color=color, alpha=0.35)
+
+ for b, a, p in zip(batches, actual, passed):
+ if not p and a and a > 0:
+ ax.plot(b, a, marker="x", color=color, markersize=11,
+ markeredgewidth=2.2, linestyle="")
+
+ if not plotted:
+ plt.close(fig)
+ return
+
+ ax.set_xlabel("Batch size (items/request, log)")
+ ax.set_ylabel(f"{ITEM_LABEL[signal]} (log)\n— actual, -- attempted (batch × 5 req/s)")
ax.set_xscale("log")
ax.set_yscale("log")
+ ax.set_title(
+ f"Batch efficiency — {signal}\n"
+ "Phase 1 at fixed 5 req/sec. Solid below dashed = SUT dropping batches."
+ )
ax.grid(True, which="both", linestyle=":", alpha=0.4)
ax.legend(loc="upper left", fontsize=9)
fig.tight_layout()
@@ -140,24 +715,151 @@ def render_throughput_detail(runs: list[dict], signal: str, out: Path) -> None:
plt.close(fig)
+def render_signal_mix(runs: list[dict], out: Path) -> None:
+ """Cross-signal throughput comparison: bars per (tier, mode), one bar per signal."""
+ thr_runs = [r for r in runs if r.get("scenario", "throughput") == "throughput"]
+ if not thr_runs:
+ return
+
+ keys = sorted({(r["tier"], r["mode"]) for r in thr_runs},
+ key=lambda k: (tier_rank(k[0]), mode_rank(k[1])))
+ if not keys:
+ return
+
+ fig, ax = plt.subplots(figsize=(max(8, len(keys) * 1.9), 6))
+
+ width = 0.25
+ x = list(range(len(keys)))
+ any_value = False
+
+ for si, signal in enumerate(SIGNALS):
+ ys = []
+ for tier, mode in keys:
+ match = [r for r in thr_runs if r["tier"] == tier and r["mode"] == mode and r["signal"] == signal]
+ ys.append((match[0].get("maxSustainableItemsPerSec", 0) or 0) if match else 0)
+ offsets = [xi + (si - 1) * width for xi in x]
+ bars = ax.bar(offsets, ys, width=width, color=SIGNAL_COLORS[signal], label=signal)
+ for b, y in zip(bars, ys):
+ if y > 0:
+ ax.annotate(fmt_count(y), xy=(b.get_x() + b.get_width() / 2, y),
+ xytext=(0, 3), textcoords="offset points",
+ ha="center", va="bottom", fontsize=7)
+ any_value = True
+
+ if not any_value:
+ plt.close(fig)
+ return
+
+ ax.set_xticks(x)
+ ax.set_xticklabels([f"{t}\n{m}" for t, m in keys], fontsize=9)
+ ax.set_ylabel("Max sustainable items/sec (Y unit varies by signal)")
+ ax.set_title("Cross-signal comparison — per (tier × mode), spans vs metrics vs logs")
+ ax.legend(title="Signal")
+ ax.grid(axis="y", linestyle=":", alpha=0.4)
+ fig.tight_layout()
+ fig.savefig(out, dpi=130)
+ plt.close(fig)
+
+
+# ---------- Read-probe charts ----------
+
+
+def _readprobe_steps(run: dict) -> list[dict]:
+ return ((run.get("readProbe") or {}).get("steps") or [])
+
+
+def render_readprobe_headline(runs: list[dict], signal: str, out: Path) -> None:
+ sig_runs = iter_runs(runs, signal, "read-probe")
+ if not sig_runs:
+ return
+
+ tiers_present = sorted({r["tier"] for r in sig_runs}, key=tier_rank)
+ modes_present = sorted({r["mode"] for r in sig_runs}, key=mode_rank)
+ if not tiers_present:
+ return
+
+ fig, ax = plt.subplots(figsize=(max(8, len(tiers_present) * 2.5), 6))
+
+ values = [r.get("maxFillLevelPassed", 0) or 0 for r in sig_runs]
+ ymax = max(values) if values else 1
+ ghost_height = max(ymax * 0.02, 1)
+
+ width = 0.8 / max(len(modes_present), 1)
+ x_centers = list(range(len(tiers_present)))
+
+ for mi, mode in enumerate(modes_present):
+ for ti, tier in enumerate(tiers_present):
+ match = [r for r in sig_runs if r["tier"] == tier and r["mode"] == mode]
+ x = ti + (mi - (len(modes_present) - 1) / 2) * width
+ if not match:
+ continue
+ run = match[0]
+ value = run.get("maxFillLevelPassed", 0) or 0
+ if value > 0:
+ ax.bar(x, value, width=width, color=mode_color(mode))
+ # latency at max
+ last_pass = None
+ for step in _readprobe_steps(run):
+ if step.get("passed"):
+ last_pass = step
+ lat_str = f"\n{int(last_pass['readLatencyMs'])}ms" if last_pass else ""
+ ax.annotate(
+ f"{fmt_count(value)}{lat_str}",
+ xy=(x, value), xytext=(0, 4), textcoords="offset points",
+ ha="center", va="bottom", fontsize=8,
+ )
+ else:
+ ax.bar(x, ghost_height, width=width, color="#dddddd",
+ edgecolor="#999999", hatch="///", linewidth=0.7)
+ ax.annotate(
+ "no fill passed",
+ xy=(x, ghost_height), xytext=(0, 4), textcoords="offset points",
+ ha="center", va="bottom", fontsize=7, color="#aa3333",
+ )
+
+ ax.set_xticks(x_centers)
+ ax.set_xticklabels(tiers_present)
+ ax.set_yscale("log")
+ ax.set_ylabel(f"Max passing fill level — {ROW_LABEL[signal]} (log)")
+ ax.set_title(
+ f"Read-probe headline — {signal} via {read_path_for_signal(signal)}\n"
+ "Bar label: max passing fill + read latency at that fill"
+ )
+
+ legend = [Patch(facecolor=mode_color(m), label=m) for m in modes_present]
+ legend.append(Patch(facecolor="#dddddd", edgecolor="#999999", hatch="///", label="no fill level passed"))
+ ax.legend(handles=legend, title="DB mode", loc="upper left")
+ ax.grid(axis="y", which="both", linestyle=":", alpha=0.4)
+
+ fig.tight_layout()
+ fig.savefig(out, dpi=130)
+ plt.close(fig)
+
+
def render_readprobe(runs: list[dict], signal: str, out: Path) -> None:
- runs = [r for r in runs if r["signal"] == signal and r["scenario"] == "read-probe"]
- if not runs:
+ """Read latency vs rows ingested — improved over legacy chart-readprobe."""
+ sig_runs = [r for r in iter_runs(runs, signal, "read-probe") if _readprobe_steps(r)]
+ if not sig_runs:
return
- fig, ax = plt.subplots(figsize=(10, 6))
+ fig, ax = plt.subplots(figsize=(11, 6))
plotted = False
threshold_ms = 5000
- for run in sorted(runs, key=lambda r: (tier_rank(r["tier"]), mode_rank(r["mode"]))):
+ for run in sig_runs:
+ color = mode_color(run["mode"])
rp = run.get("readProbe") or {}
threshold_ms = rp.get("readThresholdMs", threshold_ms)
- steps = rp.get("steps") or []
+ steps = _readprobe_steps(run)
xs = [s["rowsIngested"] for s in steps]
ys = [s["readLatencyMs"] for s in steps]
+ passed = [bool(s.get("passed")) for s in steps]
if not xs:
continue
- ax.plot(xs, ys, marker="o", label=f"{run['tier']} / {run['mode']}")
+ ax.plot(xs, ys, linestyle="-", color=color, label=run_label(run))
+ for x, y, p in zip(xs, ys, passed):
+ ax.plot(x, y, marker=("o" if p else "x"), color=color,
+ markersize=9, markeredgewidth=2 if not p else 1, linestyle="")
plotted = True
if not plotted:
@@ -166,11 +868,14 @@ def render_readprobe(runs: list[dict], signal: str, out: Path) -> None:
ax.axhline(threshold_ms, linestyle="--", color="#cc3333", alpha=0.7,
label=f"threshold ({threshold_ms}ms)")
- ax.set_xlabel(f"{ROW_LABEL[signal]} (ingested before probe)")
- ax.set_ylabel(f"Read latency on {readPathForSignal(signal)} (ms)")
- ax.set_title(f"Read latency vs ingested rows — {signal}")
+ ax.set_xlabel(f"{ROW_LABEL[signal]} ingested (log)")
+ ax.set_ylabel("Read latency (ms, log)")
ax.set_xscale("log")
ax.set_yscale("log")
+ ax.set_title(
+ f"Read latency vs table size — {signal} via {read_path_for_signal(signal)}\n"
+ "○ = passed (latency under threshold), × = failed step"
+ )
ax.grid(True, which="both", linestyle=":", alpha=0.4)
ax.legend(loc="upper left", fontsize=9)
fig.tight_layout()
@@ -178,18 +883,247 @@ def render_readprobe(runs: list[dict], signal: str, out: Path) -> None:
plt.close(fig)
-def readPathForSignal(signal: str) -> str:
- return {
- "spans": "/api/endpoints/grouped",
- "metrics": "/api/metrics/application",
- "logs": "/api/logs",
- }.get(signal, "")
+def render_readprobe_ingest_time(runs: list[dict], signal: str, out: Path) -> None:
+ sig_runs = [r for r in iter_runs(runs, signal, "read-probe") if _readprobe_steps(r)]
+ if not sig_runs:
+ return
+
+ fig, ax = plt.subplots(figsize=(11, 6))
+ plotted = False
+
+ for run in sig_runs:
+ color = mode_color(run["mode"])
+ steps = _readprobe_steps(run)
+ xs = [s["fillLevelTarget"] for s in steps]
+ ys = [s.get("ingestSecondsElapsed", 0) for s in steps]
+ pts = [(x, y) for x, y in zip(xs, ys) if y and y > 0]
+ if not pts:
+ continue
+ xs2, ys2 = zip(*pts)
+ ax.plot(xs2, ys2, linestyle="-", marker="o", color=color, label=run_label(run))
+ plotted = True
+
+ if not plotted:
+ plt.close(fig)
+ return
+
+ ax.set_xlabel(f"Target fill level — {ROW_LABEL[signal]} (log)")
+ ax.set_ylabel("Time to reach fill level (seconds, log)")
+ ax.set_xscale("log")
+ ax.set_yscale("log")
+ ax.set_title(
+ f"Fill time vs target rows — {signal}\n"
+ "Curve sag = compaction/merge cost rising with table size."
+ )
+ ax.grid(True, which="both", linestyle=":", alpha=0.4)
+ ax.legend(loc="upper left", fontsize=9)
+ fig.tight_layout()
+ fig.savefig(out, dpi=130)
+ plt.close(fig)
+
+
+def render_readprobe_tier_scaling(runs: list[dict], signal: str, out: Path) -> None:
+ sig_runs = iter_runs(runs, signal, "read-probe")
+ if not sig_runs:
+ return
+
+ tiers_present = sorted({r["tier"] for r in sig_runs}, key=tier_rank)
+ modes_present = sorted({r["mode"] for r in sig_runs}, key=mode_rank)
+ if not tiers_present:
+ return
+
+ fig, ax = plt.subplots(figsize=(10, 6))
+ plotted = False
+ x_positions = list(range(len(tiers_present)))
+
+ for mode in modes_present:
+ xs, ys = [], []
+ for i, tier in enumerate(tiers_present):
+ match = [r for r in sig_runs if r["tier"] == tier and r["mode"] == mode]
+ if not match:
+ continue
+ value = match[0].get("maxFillLevelPassed", 0) or 0
+ if value <= 0:
+ continue
+ xs.append(i)
+ ys.append(value)
+ if not xs:
+ continue
+ ax.plot(xs, ys, marker="o", color=mode_color(mode), markersize=9,
+ linewidth=2, label=mode)
+ for x, y in zip(xs, ys):
+ ax.annotate(fmt_count(y), xy=(x, y), xytext=(0, 8),
+ textcoords="offset points", ha="center", va="bottom", fontsize=9)
+ base_x = xs[0]
+ base_y = ys[0]
+ base_vcpu = TIER_VCPU.get(tiers_present[base_x], 1)
+ ref_xs = list(range(len(tiers_present)))
+ ref_ys = [base_y * (TIER_VCPU.get(tiers_present[i], 1) / base_vcpu) for i in ref_xs]
+ ax.plot(ref_xs, ref_ys, linestyle="--", color=mode_color(mode), alpha=0.3, linewidth=1)
+ plotted = True
+
+ if not plotted:
+ plt.close(fig)
+ return
+
+ ax.set_xticks(x_positions)
+ ax.set_xticklabels([f"{t}\n{TIER_META.get(t, '')}" for t in tiers_present])
+ ax.set_yscale("log")
+ ax.set_ylabel(f"Max passing fill level — {ROW_LABEL[signal]} (log)")
+ ax.set_title(
+ f"Read-probe tier scaling — {signal} via {read_path_for_signal(signal)}\n"
+ "Dashed = linear-vCPU reference anchored at smallest tier."
+ )
+ ax.legend(title="DB mode", loc="upper left")
+ ax.grid(True, which="both", linestyle=":", alpha=0.4)
+ fig.tight_layout()
+ fig.savefig(out, dpi=130)
+ plt.close(fig)
+
+
+def render_readprobe_cliff(runs: list[dict], signal: str, out: Path) -> None:
+ sig_runs = iter_runs(runs, signal, "read-probe")
+ if not sig_runs:
+ return
+
+ # Build the union of all fill levels seen across runs, sorted.
+ all_targets = sorted({s["fillLevelTarget"] for r in sig_runs for s in _readprobe_steps(r)})
+ if not all_targets:
+ return
+
+ n_cols = len(all_targets)
+ n_rows = len(sig_runs)
+
+ fig_w = max(10, n_cols * 1.8 + 3.5)
+ fig_h = max(4, n_rows * 0.9 + 2)
+ fig, ax = plt.subplots(figsize=(fig_w, fig_h))
+
+ def _probe_status(step: dict) -> str:
+ if step.get("passed"):
+ return "pass"
+ # over-threshold but read OK
+ if step.get("readOk"):
+ return "soft-cliff"
+ return "hard-fail"
+
+ for ri, run in enumerate(sig_runs):
+ y = n_rows - ri - 1
+ ax.text(-0.5, y + 0.5, run_label(run), ha="right", va="center",
+ fontsize=10, fontweight="bold")
+
+ by_target = {s["fillLevelTarget"]: s for s in _readprobe_steps(run)}
+ for ci, target in enumerate(all_targets):
+ x = ci
+ step = by_target.get(target)
+ if step is None:
+ color = STATUS_COLOR["missing"]
+ label_text = ""
+ else:
+ color = STATUS_COLOR[_probe_status(step)]
+ lat = int(step.get("readLatencyMs", 0))
+ label_text = f"{lat}ms"
+ ax.add_patch(Rectangle((x, y), 1, 1, facecolor=color,
+ edgecolor="white", linewidth=1.5))
+ if label_text:
+ ax.text(x + 0.5, y + 0.5, label_text, ha="center", va="center", fontsize=8)
+
+ max_passed = run.get("maxFillLevelPassed", 0) or 0
+ margin = n_cols + 0.35
+ if max_passed > 0:
+ ax.text(margin, y + 0.5, f"max {fmt_count(max_passed)}",
+ ha="left", va="center", fontsize=10, fontweight="bold")
+ else:
+ ax.text(margin, y + 0.5, "no fill passed",
+ ha="left", va="center", fontsize=10, color="#cc3333", fontweight="bold")
+
+ # Column header (fill level)
+ for ci, target in enumerate(all_targets):
+ ax.text(ci + 0.5, n_rows + 0.25, fmt_count(target),
+ ha="center", va="bottom", fontsize=10, fontweight="bold")
+ ax.text(n_cols / 2.0, n_rows + 0.85, f"Fill level ({ROW_LABEL[signal]})",
+ ha="center", va="bottom", fontsize=11, fontweight="bold")
+
+ ax.set_xlim(-4, n_cols + 5)
+ ax.set_ylim(-0.6, n_rows + 1.5)
+ ax.set_aspect("equal")
+ ax.axis("off")
+
+ legend = [
+ Patch(facecolor=STATUS_COLOR["pass"], label="passed (read OK, under threshold)"),
+ Patch(facecolor=STATUS_COLOR["soft-cliff"], label="over threshold (read OK but slow)"),
+ Patch(facecolor=STATUS_COLOR["hard-fail"], label="read errored"),
+ Patch(facecolor=STATUS_COLOR["missing"], edgecolor="#cccccc", label="not run"),
+ ]
+ ax.legend(handles=legend, loc="lower center", bbox_to_anchor=(0.5, -0.05),
+ ncol=4, fontsize=9, frameon=False)
+
+ fig.suptitle(f"Read-probe step-status grid — {signal}\nread path: {read_path_for_signal(signal)}",
+ fontsize=12)
+ fig.tight_layout(rect=(0, 0.04, 1, 0.94))
+ fig.savefig(out, dpi=130, bbox_inches="tight")
+ plt.close(fig)
+
+
+def render_readprobe_signal_mix(runs: list[dict], out: Path) -> None:
+ rp_runs = [r for r in runs if r.get("scenario") == "read-probe"]
+ if not rp_runs:
+ return
+
+ keys = sorted({(r["tier"], r["mode"]) for r in rp_runs},
+ key=lambda k: (tier_rank(k[0]), mode_rank(k[1])))
+ if not keys:
+ return
+
+ fig, ax = plt.subplots(figsize=(max(8, len(keys) * 1.9), 6))
+
+ width = 0.25
+ x = list(range(len(keys)))
+ any_value = False
+
+ for si, signal in enumerate(SIGNALS):
+ ys = []
+ for tier, mode in keys:
+ match = [r for r in rp_runs if r["tier"] == tier and r["mode"] == mode and r["signal"] == signal]
+ ys.append((match[0].get("maxFillLevelPassed", 0) or 0) if match else 0)
+ offsets = [xi + (si - 1) * width for xi in x]
+ bars = ax.bar(offsets, ys, width=width,
+ color=SIGNAL_COLORS[signal],
+ label=f"{signal} ({read_path_for_signal(signal)})")
+ for b, y in zip(bars, ys):
+ if y > 0:
+ ax.annotate(fmt_count(y), xy=(b.get_x() + b.get_width() / 2, y),
+ xytext=(0, 3), textcoords="offset points",
+ ha="center", va="bottom", fontsize=7)
+ any_value = True
+
+ if not any_value:
+ plt.close(fig)
+ return
+
+ ax.set_xticks(x)
+ ax.set_xticklabels([f"{t}\n{m}" for t, m in keys], fontsize=9)
+ ax.set_yscale("log")
+ ax.set_ylabel("Max passing fill level (rows, log)")
+ ax.set_title("Read-probe cross-signal comparison — which read path scales worst?")
+ ax.legend(title="Signal (read path)", loc="upper left", fontsize=8)
+ ax.grid(axis="y", which="both", linestyle=":", alpha=0.4)
+ fig.tight_layout()
+ fig.savefig(out, dpi=130)
+ plt.close(fig)
+
+
+# ---------- summary.md ----------
def render_summary(runs: list[dict], out: Path) -> None:
- lines = ["# Traceway hardware benchmark — summary", ""]
- lines.append(f"Runs analyzed: {len(runs)}")
- lines.append("")
+ lines = [
+ "# Traceway hardware benchmark — summary",
+ "",
+ "> See [../charts.md](../charts.md) for a guide to reading these charts.",
+ "",
+ f"Runs analyzed: {len(runs)}",
+ "",
+ ]
throughput_runs = [r for r in runs if r["scenario"] == "throughput"]
if throughput_runs:
@@ -206,7 +1140,7 @@ def render_summary(runs: list[dict], out: Path) -> None:
lines.append(f"| Tier | Mode | Max {ITEM_LABEL[signal]} | P1 max batch | P2 max req/sec @ batch | P3 max req/sec @ batch=100 |")
lines.append("|------|------|---------|--------------|-------------------------|-------------------------------|")
for run in sorted(sig_runs, key=lambda r: (tier_rank(r["tier"]), mode_rank(r["mode"]))):
- max_items = int(run.get("maxSustainableItemsPerSec", 0))
+ max_items = int(run.get("maxSustainableItemsPerSec", 0) or 0)
max_batch = (run.get("phase1") or {}).get("maxBatchSize", 0)
p2 = run.get("phase2") or {}
p2_rate = p2.get("maxRequestRate", 0)
@@ -214,8 +1148,7 @@ def render_summary(runs: list[dict], out: Path) -> None:
p2_cell = f"{p2_rate:g} @ {p2_batch:,}" if p2_rate else "—"
p3 = run.get("phase3") or {}
p3_rate = p3.get("maxRequestRate", 0)
- p3_batch = p3.get("fixedBatchSize", 0)
- p3_cell = f"{p3_rate:g} @ {p3_batch:,}" if p3_rate else "—"
+ p3_cell = f"{p3_rate:g}" if p3_rate else "—"
lines.append(f"| {run['tier']} | {run['mode']} | {max_items:,} | {max_batch:,} | {p2_cell} | {p3_cell} |")
lines.append("")
@@ -230,14 +1163,13 @@ def render_summary(runs: list[dict], out: Path) -> None:
sig_runs = [r for r in readprobe_runs if r["signal"] == signal]
if not sig_runs:
continue
- lines.append(f"### {signal.capitalize()} — probing `{readPathForSignal(signal)}`")
+ lines.append(f"### {signal.capitalize()} — probing `{read_path_for_signal(signal)}`")
lines.append("")
lines.append(f"| Tier | Mode | Max {ROW_LABEL[signal]} passed | Read latency @ max (ms) | First failing fill | Read latency @ failing (ms) |")
lines.append("|------|------|----------------|-------------------|--------------------|------------------------|")
for run in sorted(sig_runs, key=lambda r: (tier_rank(r["tier"]), mode_rank(r["mode"]))):
- rp = run.get("readProbe") or {}
- steps = rp.get("steps") or []
- max_passed = run.get("maxFillLevelPassed", 0)
+ steps = _readprobe_steps(run)
+ max_passed = run.get("maxFillLevelPassed", 0) or 0
latency_at_max = 0
first_fail = ""
latency_at_fail = ""
@@ -254,6 +1186,9 @@ def render_summary(runs: list[dict], out: Path) -> None:
out.write_text("\n".join(lines) + "\n")
+# ---------- Entry point ----------
+
+
def main() -> int:
if len(sys.argv) < 2:
print("usage: chart.py <results-dir>", file=sys.stderr)
@@ -269,9 +1204,25 @@ def main() -> int:
return 1
for signal in SIGNALS:
- render_throughput_bar(runs, signal, results_dir / f"chart-{signal}.png")
- render_throughput_detail(runs, signal, results_dir / f"chart-detail-{signal}.png")
+ # Throughput charts (no-op if no throughput runs for this signal)
+ render_headline_bar(runs, signal, results_dir / f"chart-{signal}.png")
+ render_phase1_latency(runs, signal, results_dir / f"chart-phase1-batch-{signal}.png")
+ render_phase2_rate(runs, signal, results_dir / f"chart-phase2-rate-{signal}.png")
+ render_phase3_rate(runs, signal, results_dir / f"chart-phase3-rate-{signal}.png")
+ render_pareto(runs, signal, results_dir / f"chart-pareto-{signal}.png")
+ render_tier_scaling(runs, signal, results_dir / f"chart-tier-scaling-{signal}.png")
+ render_cliff_grid(runs, signal, results_dir / f"chart-cliff-{signal}.png")
+ render_batch_efficiency(runs, signal, results_dir / f"chart-batch-efficiency-{signal}.png")
+
+ # Read-probe charts (no-op if no read-probe runs for this signal)
+ render_readprobe_headline(runs, signal, results_dir / f"chart-readprobe-headline-{signal}.png")
render_readprobe(runs, signal, results_dir / f"chart-readprobe-{signal}.png")
+ render_readprobe_ingest_time(runs, signal, results_dir / f"chart-readprobe-ingest-time-{signal}.png")
+ render_readprobe_tier_scaling(runs, signal, results_dir / f"chart-readprobe-tier-scaling-{signal}.png")
+ render_readprobe_cliff(runs, signal, results_dir / f"chart-readprobe-cliff-{signal}.png")
+
+ render_signal_mix(runs, results_dir / "chart-signal-mix.png")
+ render_readprobe_signal_mix(runs, results_dir / "chart-readprobe-signal-mix.png")
render_summary(runs, results_dir / "summary.md")
print(f"wrote charts and summary.md into {results_dir}")
return 0
diff --git a/benchmarks/scripts/run-local.sh b/benchmarks/scripts/run-local.sh
index ef1d9dae..02734a03 100755
--- a/benchmarks/scripts/run-local.sh
+++ b/benchmarks/scripts/run-local.sh
@@ -14,7 +14,13 @@
# run-local.sh --smoke # 1 tier, 1 mode, 1 signal, short steps (~5 min)
# run-local.sh --tier ccx13 --mode sqlite --signal spans
# run-local.sh --dry-run # validate env + print plan, no provisioning
-# run-local.sh --commit # drop the "-local" suffix on the output dir
+#
+# Output:
+# --scenario throughput -> benchmarks/results-throughput/
+# --scenario read-probe -> benchmarks/results-probe/
+# Each scenario folder is wiped at the start of its run so output reflects
+# only the current dispatch. The two folders are never touched by each
+# other's runs — running read-probe will not clobber throughput results.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
@@ -27,10 +33,9 @@ SCENARIO="throughput"
DURATION="30m"
SMOKE=0
DRY_RUN=0
-COMMIT=0
usage() {
- sed -n '1,17p' "$0"; exit 2
+ sed -n '1,23p' "$0"; exit 2
}
while [[ $# -gt 0 ]]; do
@@ -42,7 +47,6 @@ while [[ $# -gt 0 ]]; do
--duration) DURATION="$2"; shift 2 ;;
--smoke) SMOKE=1; shift ;;
--dry-run) DRY_RUN=1; shift ;;
- --commit) COMMIT=1; shift ;;
-h|--help) usage ;;
*) echo "unknown flag: $1" >&2; usage ;;
esac
@@ -67,17 +71,17 @@ fi
export BENCH_MODES_PLAN="${MODES}"
"${SCRIPT_DIR}/preflight.sh"
-# Single, dispatch-scoped results folder. Wiped at the start so each run's
-# committed output reflects only that run's matrix entries — no
-# cross-dispatch mixing (which was the failure mode of the date-folder
-# layout: two dispatches landing on the same day silently combined into one
-# summary.md). Cross-run comparison happens via `git log`, not adjacent
+# One canonical folder per scenario. Wiped at the start so each run's output
+# reflects only that run's matrix entries — no cross-dispatch mixing (which
+# was the failure mode of the date-folder layout: two dispatches landing on
+# the same day silently combined into one summary.md). The throughput folder
+# and the read-probe folder are siblings so each scenario stays out of the
+# other's way. Cross-run comparison happens via `git log`, not adjacent
# folders.
-if [[ "${COMMIT}" -eq 1 ]]; then
- OUT_DIR="${REPO_ROOT}/benchmarks/results/latest"
-else
- OUT_DIR="${REPO_ROOT}/benchmarks/results/latest-local"
-fi
+case "${SCENARIO}" in
+ throughput) OUT_DIR="${REPO_ROOT}/benchmarks/results-throughput" ;;
+ read-probe) OUT_DIR="${REPO_ROOT}/benchmarks/results-probe" ;;
+esac
rm -rf "${OUT_DIR}"
mkdir -p "${OUT_DIR}"
echo "results dir: ${OUT_DIR}" >&2