AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
perf: release performance gate for autoindex + search (budgets, ES floor, rc.99 regression harness) (#2432)
xerj-org255519b05111
diff --git a/docs/perf-release-gate.md b/docs/perf-release-gate.mdnew file mode 100644index 000000000..41a924630--- /dev/null+++ b/docs/perf-release-gate.md@@ -0,0 +1,160 @@+# Autoindex / search release performance gate++`scripts/perf/autoindex_release_gate.sh` blocks a build that repeats the+rc.99 field failures: #2421 (state dir 1.6x source), #2422 (454 indices,+13–23 s searches), #2423 (0.2 MB/s, OOM), #2424 (planning memory, long+names), #2427 (node over its own memory cap) and #2428 (client EAGAIN abort).+It runs the user-facing path end to end, `xerj autoindex <folder>` with+default flags, against a deterministic synthetic corpus on a throwaway node+with a fixed memory cap. It then measures uncached search on all three+surfaces and fails on any budget miss.++## Run it locally++```bash+(cd engine && cargo build --release --locked -p xerj-server)+scripts/perf/autoindex_release_gate.sh --bin engine/target/release/xerj # 1 GB tier+scripts/perf/autoindex_release_gate.sh --bin engine/target/release/xerj --tier smoke+scripts/perf/autoindex_release_gate.sh --bin engine/target/release/xerj \+ --baseline ~/xerj-prev/xerj # +regression checks+scripts/perf/autoindex_release_gate.sh --bin engine/target/release/xerj \+ --engine-baseline es-baseline.json # +ES/Lucene floor+```++| flag | meaning |+|---|---|+| `--tier smoke\|1gb\|5gb` | budget tier in `scripts/perf/budgets.toml` (sets corpus size, memory cap, timeout) |+| `--size-mb N` | override corpus size (budgets stay those of the tier) |+| `--seed N` | corpus seed (default 2421) |+| `--work DIR` | scratch: corpus + per-run data/state (default `$RUNNER_TEMP` or `/tmp`/xerj-perf-gate) |+| `--corpus-dir DIR` | reuse or put the corpus here (reused if its manifest matches size + seed) |+| `--out DIR` | results (`candidate.json`, `baseline.json`, `results.json`, `summary.md`, log tails) |+| `--baseline BIN` | also run the previous release binary on the same corpus; fail on >20% regression |+| `--engine-baseline FILE` | ES/Lucene floor JSON (from `es_baseline.py`); enables `[ratio.*]` budgets |+| `--autoindex-args "..."` | extra autoindex flags (default: none, the user-facing path) |+| `--keep` | keep run data/state dirs |+| `MIN_FREE_GB` env | disk reserve the guard keeps free (default 8) |++Exit 0 means every budget was met. Exit 1 means at least one was missed, and the markdown table is+printed and written to `summary.md` (and appended to `$GITHUB_STEP_SUMMARY`+in Actions). Exit 2 means a usage or environment error, e.g. not enough disk. Requires+Python ≥ 3.11 (stdlib only). On a 4-core machine a passing build finishes the+1 GB tier in under 20 minutes: corpus about 25 s, autoindex ≤ 15 min by budget, restart+plus 60 queries about 1 min.++## What a run does++1. `gen_corpus.py` writes a byte-for-byte deterministic corpus plus+ `<corpus>.queries.json` (20 queries per surface, 60 distinct strings+ drawn from phrases planted in the corpus, so each should hit) and+ `<corpus>.manifest.json`. The mix per GB is code (rs/go/py/ts, about 24%),+ markdown (12%), HTML with tables (8%), 3 large plus about 800 small+ distinct-schema CSVs, 24 JSONL, 16 logs, 4 big JSON flight-trace files with numeric+ arrays (15%), 1500 multi-array JSON/YAML, 8000 tiny files and deep Go-module+ style paths up to about 500 chars.+2. `gate.py run` starts `xerj --insecure --data-dir … --port … --disable-feedback`+ with `XERJ_MAX_PROCESS_MEMORY_MB=<tier cap>` and `XERJ_DISABLE_FEEDBACK=true`,+ then runs `xerj autoindex <corpus> --approve proceed --json` under the tier+ timeout. It samples node and client process-tree RSS and threads from `/proc`+ every 0.5 s.+3. It reads `_cat/indices/<prefix>-*`, stops the node, scans the node log for+ parent-breaker trips (`engaging the parent memory circuit breaker`), fatal+ patterns (EAGAIN / os error 11, OOM, SIGABRT/panic, EMFILE) and retry/429+ lines, and measures the state dir and data dir sizes.+4. It restarts the node (timed) so the page cache and query caches are cold,+ then runs 20 queries through `xerj search --json`, 20 through raw+ `POST /<prefix>-*/_search?request_cache=false` and 20 through `xerj mcp`+ (`tools/call xerj_search`). No query string repeats.+5. `gate.py eval` compares against `budgets.toml`.++## Budgets++See `scripts/perf/budgets.toml` for the numbers and the reasoning behind each one.++| check | smoke (200 MB) | 1gb | 5gb (release) | why |+|---|---|---|---|---|+| autoindex completes (exit 0/3, node alive, no OOM/abort/EAGAIN, within timeout) | 6 min | 15 min | 30 min | #2423, #2427, #2428 |+| throughput MB/s ≥ (absolute backstop) | 1.5 | 3.0 | 3.0 | about 0.5x the low end of ES on a 4-vCPU runner; rc.99 did 0.2–0.6 |+| node peak RSS ≤ memory cap (MB) | 2048 | 3072 | 3072 | #2427: rc.99 hit 3.9 GB (autoindex) / 5.6 GB (`_bulk`) under a 3 GB cap |+| client peak RSS (MB) ≤ | 768 | 1024 | 1536 | #2424, #2428 |+| state dir / source ≤ | 1.2 | 1.2 | 1.2 | #2421: 8.4 GB state for 5 GB |+| index + state on disk ≤ (MB) | 250 | 1300 | 6400 | about 1.2x source, stricter than ES force-merged (1.42x) |+| indices per prefix ≤ | 40 | 40 | 40 | #2422: 454; #2430 caps a run at 32 |+| uncached search p95 / max (ms) | 1000 / 2000 | 1000 / 2000 | 1000 / 2000 | #2422: 13–23 s |+| search errors | 0 | 0 | 0 | |+| queries with hits ≥ | 90% | 90% | 90% | catches "fast because it found nothing" |+| parent-breaker trips | 0 | 0 | 0 | #2427 |+| bulk retries / 429s ≤ | 20 | 20 | 50 | #2427 |+| node restart (s) ≤ | 60 | 60 | 120 | |+| regression vs `--baseline` binary | 20% | 20% | 20% | throughput, p95, node/client RSS, state ratio, index bytes |+| index bytes vs ES (force-merged) ≤ | 1.25x | 1.0x | 1.0x | ES floor (`[ratio.*]`) |+| ingest MB/s vs ES (default mapping) ≥ | 0.3x | 0.5x | 0.5x | ES floor; stretch goal 0.8x |+| search p95 vs ES ≤ | — | 1.5x | 1.5x | ES floor |++### Lucene / Elasticsearch floor++Absolute budgets drift with runner hardware, so index size and indexing speed are+also held against Elasticsearch on the same corpus and the same machine.+`scripts/perf/es_baseline.py` bulk-loads the corpus into a throwaway ES. It writes+one doc per line for jsonl/csv/log and one doc per file otherwise, using the default+dynamic mapping, 1 shard and 0 replicas. Ingest time runs until the index is refreshed+and flushed. It then **force-merges to one segment**, and `index_bytes` is the merged+size, because the unmerged size swings with the segment count (1.77x vs 1.42x source).+Finally it runs the same 60 queries uncached and writes:++```json+{"engine": "elasticsearch-9.x (lucene 10.x)", "corpus_mb": 1024, "seed": 2421,+ "index_bytes": 0, "ingest_mb_s": 0.0, "search_p95_ms": 0.0}+```++Pass that file as `--engine-baseline` and the `[ratio.<tier>]` keys apply:+`xerj_index_bytes ≤ k·es_index_bytes`, `xerj_ingest_MBps ≥ k·es_ingest_MBps`,+and `xerj_p95 ≤ k·es_p95`. In CI, ES runs on the same runner, so the floor scales with+the runner automatically. ES always runs **before** xerj and is stopped before xerj+starts, never at the same time: ES uses about 5.5 GB RSS with a 4 GB heap, and xerj has a 3 GB cap.+The last good baseline is cached and used as a fallback when the live ES step fails.++Reference floor, 1 GB corpus with seed 2421, 4 pinned cores on a heavily loaded shared box,+single noisy run (perf-floor `results.md`, 2026-10-25):++| engine | MiB/s | index bytes / source | peak RSS MB |+|---|---|---|---|+| ES 9.5.5 default dynamic mapping, 4 GB heap (refreshed, 28 segments) | 9.05 | 1.77 | 5,551 |+| same, after force-merge to 1 segment (+48.7 s) | | 1.42 | |+| Lucene 10.5.1 StandardAnalyzer, stored fields on, 4 threads | 25.4 | 0.78 | 3,036 |+| Lucene 10.5.1 StandardAnalyzer, stored fields off, 4 threads | 36.0 | 0.30 | 2,518 |+| xerj rc.99 `_bulk` text-only (cap 3072 MB; died at about 900 s) | 0.37 | n/a | 5,643 |+| xerj rc.99 autoindex (stopped at 38% after 661 s) | 0.59 | about 0.92 (est.) | about 2,960 |++So rc.99 runs at about 0.04–0.07x of ES and misses the 0.5x floor by an order of magnitude.++Runner sizing: public-repo `ubuntu-latest` has 4 vCPU, 16 GB and a 14 GB SSD, which is what the budgets+assume. Private repos get 2 vCPU and 8 GB there, so use the `smoke` tier with a 2 GB ES heap,+or a larger runner.++```bash+python3 scripts/perf/gen_corpus.py --out /tmp/c1g --size-mb 1024 --seed 2421+docker run -d --name es -p 9200:9200 -e discovery.type=single-node \+ -e xpack.security.enabled=false -e ES_JAVA_OPTS="-Xms4g -Xmx4g" \+ docker.elastic.co/elasticsearch/elasticsearch:9.5.5+python3 scripts/perf/es_baseline.py --es http://127.0.0.1:9200 --corpus /tmp/c1g \+ --queries /tmp/c1g.queries.json --out es-baseline.json+```++## CI++The workflow (`.github/workflows/perf-gate.yml`) is maintained by hand. See the+tracking issue for the ready-to-paste YAML. It runs:++* on PRs touching autoindex, engine or server paths: the 1 GB tier;+* nightly: the 1 GB tier with `--baseline` set to the latest release binary, plus the ES floor;+* on `v*` tags, as a job `release.yml` depends on (`needs: [build, perf-gate]`): the 5 GB tier.++Results are uploaded as the `perf-gate-*` artifact (`results.json` holds every+raw measurement, budget and verdict) and summarized in the job summary and as+a PR comment.++## Changing a budget++Edit `scripts/perf/budgets.toml` in the same PR as the change that needs it,+with a comment explaining why. Loosening a budget needs a linked issue.diff --git a/scripts/perf/autoindex_release_gate.sh b/scripts/perf/autoindex_release_gate.shnew file mode 100755index 000000000..9f60e32af--- /dev/null+++ b/scripts/perf/autoindex_release_gate.sh@@ -0,0 +1,105 @@+#!/usr/bin/env bash+# Autoindex/search release performance gate. See docs/perf-release-gate.md.+#+# scripts/perf/autoindex_release_gate.sh --bin target/release/xerj [--tier 1gb]+# [--size-mb N] [--seed 2421] [--corpus-dir DIR] [--work DIR] [--out DIR]+# [--baseline PREV_XERJ_BINARY] [--engine-baseline es-baseline.json]+# [--port 9400] [--autoindex-args "..."] [--keep]+#+# Generates (or reuses) the deterministic synthetic corpus, runs the candidate+# binary (and optionally the previous release binary) through gate.py on a+# throwaway node with a fixed memory cap, evaluates scripts/perf/budgets.toml+# and writes OUT/results.json + OUT/summary.md. Exit 0 = every budget met,+# 1 = at least one budget missed, 2 = usage / environment error.+set -euo pipefail++HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+BIN="" TIER="1gb" SIZE_MB="" SEED=2421 CORPUS="" WORK="" OUT="" BASE="" ENG="" PORT=9400 KEEP=0 AI_ARGS=""+BUDGETS="$HERE/budgets.toml"+MIN_FREE_GB="${MIN_FREE_GB:-8}"++die() { echo "perf-gate: $*" >&2; exit 2; }+while [[ $# -gt 0 ]]; do+ case "$1" in+ --bin) BIN="$2"; shift 2;;+ --tier) TIER="$2"; shift 2;;+ --size-mb) SIZE_MB="$2"; shift 2;;+ --seed) SEED="$2"; shift 2;;+ --corpus-dir) CORPUS="$2"; shift 2;;+ --work) WORK="$2"; shift 2;;+ --out) OUT="$2"; shift 2;;+ --baseline) BASE="$2"; shift 2;;+ --engine-baseline) ENG="$2"; shift 2;;+ --budgets) BUDGETS="$2"; shift 2;;+ --port) PORT="$2"; shift 2;;+ --autoindex-args) AI_ARGS="$2"; shift 2;;+ --keep) KEEP=1; shift;;+ -h|--help) sed -n '2,16p' "$0"; exit 0;;+ *) die "unknown argument: $1";;+ esac+done+[[ -n "$BIN" && -x "$BIN" ]] || die "--bin must point at an executable xerj binary"+[[ -z "$BASE" || -x "$BASE" ]] || die "--baseline must point at an executable xerj binary"+command -v python3 >/dev/null || die "python3 (3.11+) required"+python3 -c 'import tomllib' 2>/dev/null || die "python3 >= 3.11 required (tomllib)"++tier_val() { python3 -c 'import sys,tomllib;print(tomllib.load(open(sys.argv[1],"rb"))["tier"][sys.argv[2]][sys.argv[3]])' "$BUDGETS" "$TIER" "$1"; }+tier_val corpus_mb >/dev/null 2>&1 || die "tier '$TIER' not in $BUDGETS"+SIZE_MB="${SIZE_MB:-$(tier_val corpus_mb)}"+CAP_MB="$(tier_val node_memory_cap_mb)"+TIMEOUT_MIN="$(tier_val autoindex_timeout_min)"++WORK="${WORK:-${RUNNER_TEMP:-/tmp}/xerj-perf-gate}"+OUT="${OUT:-$WORK/out}"+CORPUS="${CORPUS:-$WORK/corpus-${SIZE_MB}mb-s${SEED}}"+mkdir -p "$WORK" "$OUT"+WORK="$(cd "$WORK" && pwd)"; OUT="$(cd "$OUT" && pwd)"++# Disk guard: corpus + per-run data/state can reach ~4x the corpus on a bad+# build (rc.99 wrote 1.6x state + 2x index). Refuse rather than fill the disk.+free_gb() { df -Pk "$1" | awk 'NR==2{printf "%d", $4/1048576}'; }+need_gb=$(( (SIZE_MB * 4 + 1023) / 1024 ))+[[ -d "$CORPUS" ]] && need_gb=$(( (SIZE_MB * 3 + 1023) / 1024 ))+have=$(free_gb "$WORK")+if (( have - need_gb < MIN_FREE_GB )); then+ die "need ~${need_gb} GB scratch + ${MIN_FREE_GB} GB reserve on $(df -P "$WORK" | awk 'NR==2{print $6}'), only ${have} GB free (set MIN_FREE_GB or --size-mb)"+fi++echo "perf-gate: tier=$TIER corpus=${SIZE_MB}MB seed=$SEED cap=${CAP_MB}MB timeout=${TIMEOUT_MIN}min bin=$BIN"+if [[ -f "$CORPUS.manifest.json" && -f "$CORPUS.queries.json" ]] && \+ python3 -c 'import json,sys;m=json.load(open(sys.argv[1]));sys.exit(0 if float(m.get("size_mb_requested",-1))==float(sys.argv[2]) and m.get("seed")==int(sys.argv[3]) else 1)' "$CORPUS.manifest.json" "$SIZE_MB" "$SEED"; then+ echo "perf-gate: reusing corpus $CORPUS"+else+ if [[ -e "$CORPUS" ]]; then die "$CORPUS exists but is not a seed=$SEED size=${SIZE_MB}MB corpus; remove it or pick another --corpus-dir"; fi+ t0=$(date +%s)+ python3 "$HERE/gen_corpus.py" --out "$CORPUS" --size-mb "$SIZE_MB" --seed "$SEED"+ echo "perf-gate: corpus generated in $(( $(date +%s) - t0 ))s"+fi++export XERJ_DISABLE_FEEDBACK=true+run_one() { # label bin port+ local label="$1" bin="$2" port="$3"+ rm -rf "$WORK/run-$label"+ python3 "$HERE/gate.py" run --bin "$bin" --label "$label" --corpus "$CORPUS" --queries "$CORPUS.queries.json" \+ --work "$WORK/run-$label" --out "$OUT/$label.json" --port "$port" --cap-mb "$CAP_MB" \+ --timeout-min "$TIMEOUT_MIN" --autoindex-args "$AI_ARGS"+ cp "$WORK/run-$label/autoindex.stderr" "$OUT/$label.autoindex.stderr" 2>/dev/null || true+ for f in node-ingest.log node-search.log; do+ [[ -f "$WORK/run-$label/$f" ]] && tail -c 2000000 "$WORK/run-$label/$f" > "$OUT/$label.$f" || true+ done+ (( KEEP )) || rm -rf "$WORK/run-$label"+}++if [[ -n "$BASE" ]]; then run_one baseline "$BASE" $((PORT + 10)); fi+run_one candidate "$BIN" "$PORT"++eval_args=(--budgets "$BUDGETS" --tier "$TIER" --candidate "$OUT/candidate.json" --md "$OUT/summary.md" --json "$OUT/results.json")+[[ -n "$BASE" ]] && eval_args+=(--baseline "$OUT/baseline.json")+[[ -n "$ENG" ]] && eval_args+=(--engine-baseline "$ENG")+set +e+python3 "$HERE/gate.py" eval "${eval_args[@]}"+rc=$?+set -e+[[ -n "${GITHUB_STEP_SUMMARY:-}" ]] && cat "$OUT/summary.md" >> "$GITHUB_STEP_SUMMARY"+echo "perf-gate: results in $OUT (results.json, summary.md); exit $rc"+exit $rcdiff --git a/scripts/perf/budgets.toml b/scripts/perf/budgets.tomlnew file mode 100644index 000000000..74060e246--- /dev/null+++ b/scripts/perf/budgets.toml@@ -0,0 +1,161 @@+# Release performance budgets for `scripts/perf/autoindex_release_gate.sh`.+#+# Every key is a hard ceiling (max_*) or floor (min_*). A run fails when ANY+# budget is broken; the gate prints which, as a markdown table, and exits 1.+# Tiers are picked by --tier (default: "1gb"). Values are proposals sized+# from the rc.99 regressions found on 2026-10-25 (#2421-#2428); each comment+# says why the number is where it is. Change a number here, in a reviewed PR,+# never by editing the workflow.++[tier.smoke]+# A ~200 MB corpus for a quick local check. Same shape, looser absolute+# time floors (fixed per-run overheads dominate at this size).+corpus_mb = 200+node_memory_cap_mb = 2048+autoindex_timeout_min = 6+min_throughput_mb_s = 1.5+max_node_peak_rss_mb = 2048+max_client_peak_rss_mb = 768+max_state_to_source_ratio = 1.2+max_index_plus_state_mb = 250+max_indices_per_prefix = 40+max_search_p95_ms = 1000+max_search_max_ms = 2000+max_breaker_trips = 0+max_bulk_retries = 20+max_restart_s = 60+min_queries_with_hits_pct = 90+max_regression_pct = 20++[tier.1gb]+# The PR / nightly tier. Runnable in < 20 min on 4 cores.+corpus_mb = 1024+# Node cap. 3 GB is the cap #2427 was measured with (node hit 3.9 GB). A+# 1 GB corpus must fit with room to spare; the gate sets+# XERJ_MAX_PROCESS_MEMORY_MB to this and the RSS ceiling below equals it:+# the cap is a promise, so exceeding it at all is a failure.+node_memory_cap_mb = 3072+# Hard wall-clock stop for `xerj autoindex` (a hang counts as a failure).+# 15 min > 1024 MB / 3 MB/s (5.7 min) * 2.5, so only a real stall hits it.+autoindex_timeout_min = 15+# Source MB / autoindex wall seconds. rc.99 on the 5 GB folder managed+# ~0.2 MB/s (748 MB in 68 min, #2423). The #2423 ask (5 GB in 20 min) is+# 4.3 MB/s; 3 MB/s at 1 GB leaves headroom for 4-core CI runners and the+# fixed per-run startup/finalize costs that weigh more on a smaller corpus.+min_throughput_mb_s = 3.0+max_node_peak_rss_mb = 3072+# Client RSS. #2424: phase A held several GB; after #2417 the client should+# be a streaming pipeline, so 1 GB for 1 GB of input is generous.+max_client_peak_rss_mb = 1024+# (state dir bytes) / (source bytes) after the run commits. rc.99: >= 1.6x+# (8.4 GB for 5 GB, #2421). Resume metadata + journal must stay well under+# a full copy; 1.2x is the ceiling the release issue asks for.+max_state_to_source_ratio = 1.2+# Absolute backstop on (node data dir + autoindex state dir) in MiB:+# 1.3 GB, about 1.2x the 1 GB source. ES 9.5.5 force-merged = 1.42x source on+# this corpus (perf-floor results.md), so this is stricter than ES. It holds+# even when no engine baseline is available.+max_index_plus_state_mb = 1300+# rc.99 made 454 indices for one folder (#2422). #2430 adds a run cap of+# 32 datasets plus docs/code indices; 40 leaves slack for those.+max_indices_per_prefix = 40+# Uncached search latency (wall-clock incl. client start for the CLI, HTTP+# round-trip for raw _search, tools/call round-trip for MCP). rc.99:+# 13-23 s on 5 GB (#2422). 1 s p95 / 2 s max is what an agent tolerates+# per call; #2422 targets p95 < 500 ms on 5 GB once fan-out is fixed.+max_search_p95_ms = 1000+max_search_max_ms = 2000+# Parent memory circuit breaker engagements in the node log (writes -> 429).+# #2427: rc.99 tripped it at 7.6 GB, then was OOM-killed. A release build on+# a 1 GB corpus under a 3 GB cap must never need it.+max_breaker_trips = 0+# Bulk back-pressure events reported by autoindex (bulk_congestion_events,+# i.e. 429/overload retries) + retry lines in its stderr. A few are normal+# adaptive-concurrency probing; dozens mean the node cannot keep up.+max_bulk_retries = 20+# Node restart before the search phase (replay/open of every index).+max_restart_s = 60+# Sanity floor: every query is built from phrases planted in the corpus, so+# a "fast" search that finds nothing is a failure, not a win.+min_queries_with_hits_pct = 90+# With --baseline: fail when the candidate is worse than the previous+# release binary by more than this on throughput, search p95, peak RSS+# (node and client), or state ratio, on the same corpus and machine.+max_regression_pct = 20++[tier.5gb]+# The pre-release tier (v* tags). Mirrors the 5 GB folder of #2421-#2428.+corpus_mb = 5120+# Same 3 GB cap as 1gb (the #2427 cap). #2423 asked "under 4 GB" for 5 GB;+# the floor review (2026-10-25) holds the release to 3 GB. rc.99 reached+# 5.6 GB RSS under a 3072 MB cap in bulk ingest (#2427).+node_memory_cap_mb = 3072+# #2423 ask: "finish in under 20 min"; 30 min is the hard stop.+autoindex_timeout_min = 30+# Absolute backstop. ES default mapping is estimated at 6-9 MiB/s on a 4-vCPU+# runner, so 3.0 sits at about 0.5x of its low end. The ES ratio below is the real floor.+min_throughput_mb_s = 3.0+max_node_peak_rss_mb = 3072+max_client_peak_rss_mb = 1536+max_state_to_source_ratio = 1.2+# 6.4 GB, about 1.2x of 5 GB source.+max_index_plus_state_mb = 6400+max_indices_per_prefix = 40+max_search_p95_ms = 1000+max_search_max_ms = 2000+max_breaker_trips = 0+max_bulk_retries = 50+max_restart_s = 120+min_queries_with_hits_pct = 90+max_regression_pct = 20++# ---------------------------------------------------------------------------+# Engine-relative floors (Ivan, 2026-10-25): index size and indexing time are+# floored against Lucene/Elasticsearch on the SAME corpus and the SAME runner,+# so a slower or smaller CI machine moves both sides together.+#+# Enabled when the gate gets `--engine-baseline <file.json>`, a JSON object+# measured on the same corpus (seed + size) and machine:+# { "engine": "elasticsearch-8.x", "corpus_mb": 1024, "seed": 2421,+# "index_bytes": <store.size after force-merge or final refresh>,+# "ingest_mb_s": <source MB / wall seconds to index + refresh>,+# "search_p95_ms": <optional, same 60 queries> }+# scripts/perf/es_baseline.py produces exactly that from a live ES (the+# workflow runs it against an ES service container), or point it at a+# results.json written by the floor-measurement harness.+#+# Each ratio is xerj / baseline. Values are per tier below; a key left out+# (or the whole table) disables that check.+# Measured floor (perf-floor results.md, 2026-10-25, 1 GB seed 2421, 4 cores):+# ES 9.5.5 default dynamic mapping, 4 GB heap: 9.05 MiB/s, 1.77x source+# refreshed, 1.42x source after force-merge to 1 segment.+# Lucene 10.5.1 stored fields on: 25.4 MiB/s, 0.78x source.+# xerj rc.99: 0.37-0.59 MiB/s, about 0.04-0.07x ES.+# index_bytes of the baseline MUST be measured after a force-merge to one+# segment (es_baseline.py does that). The pre-merge size swings with the+# segment count.+[ratio.default]+# xerj_index_bytes (node data dir) <= k * es_index_bytes (force-merged).+# 1.0: an index may not cost more disk than ES on the same data.+max_index_bytes_vs_baseline = 1.0+# xerj_ingest_MBps >= k * es_ingest_MBps (ES default mapping, to refreshed+# and flushed). autoindex does discovery and inference that a bulk load does not,+# so parity is not required. Stretch goal is 0.8.+min_ingest_mb_s_vs_baseline = 0.5+# xerj search p95 <= k * es search p95 (only if the baseline reports it).+max_search_p95_vs_baseline = 1.5++[ratio.smoke]+# Fixed per-index overhead weighs more at 200 MB.+max_index_bytes_vs_baseline = 1.25+min_ingest_mb_s_vs_baseline = 0.3++[ratio."1gb"]+max_index_bytes_vs_baseline = 1.0+min_ingest_mb_s_vs_baseline = 0.5+max_search_p95_vs_baseline = 1.5++[ratio."5gb"]+max_index_bytes_vs_baseline = 1.0+min_ingest_mb_s_vs_baseline = 0.5+max_search_p95_vs_baseline = 1.5diff --git a/scripts/perf/es_baseline.py b/scripts/perf/es_baseline.pynew file mode 100755index 000000000..baef7320c--- /dev/null+++ b/scripts/perf/es_baseline.py@@ -0,0 +1,136 @@+#!/usr/bin/env python3+"""Live Elasticsearch/Lucene floor for the autoindex release gate.++ es_baseline.py --es http://127.0.0.1:9200 --corpus DIR --queries DIR.queries.json \+ --out es-baseline.json [--workers 4] [--seed 2421]++Bulk-loads the SAME corpus produced by gen_corpus.py into one ES index with+default dynamic mappings (one doc per line for .jsonl/.csv/.log/.ndjson,+one doc per file otherwise, ES default dynamic mapping), refreshes + flushes+(ingest time stops here), force-merges to ONE segment (index_bytes is the+merged size), then runs the corpus' search queries uncached. Writes the JSON that+`gate.py eval --engine-baseline` consumes:++ {"engine", "corpus_mb", "seed", "source_bytes", "docs", "index_bytes",+ "ingest_mb_s", "search_p95_ms", "wall_s"}++ingest_mb_s is source bytes / wall time from first bulk to refreshed+flushed+index, the same definition gate.py uses for xerj (bytes / autoindex wall).+stdlib only; the ES host is assumed throwaway (the index is deleted first).+"""+import argparse, json, os, queue, threading, time, urllib.request++MiB = 1024 * 1024+LINE_EXT = {".jsonl", ".ndjson", ".csv", ".log"}+++def http(method, url, body=None, ctype="application/json", timeout=300):+ data = body if isinstance(body, (bytes, type(None))) else json.dumps(body).encode()+ req = urllib.request.Request(url, data=data, method=method, headers={"Content-Type": ctype})+ with urllib.request.urlopen(req, timeout=timeout) as r:+ return json.loads(r.read() or b"{}")+++def docs(corpus):+ for root, _, files in os.walk(corpus):+ for f in sorted(files):+ p = os.path.join(root, f)+ rel = os.path.relpath(p, corpus)+ ext = os.path.splitext(f)[1].lower()+ try:+ text = open(p, encoding="utf-8", errors="replace").read()+ except OSError:+ continue+ if ext in LINE_EXT:+ lines = text.splitlines()+ header = lines[0] if ext == ".csv" and lines else None+ for i, ln in enumerate(lines):+ if ln.strip():+ d = {"path": rel, "line": i, "body": ln}+ if header and i:+ d["header"] = header+ yield d+ else:+ yield {"path": rel, "body": text}+++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("--es", default=os.environ.get("ES_URL", "http://127.0.0.1:9200"))+ ap.add_argument("--corpus", required=True); ap.add_argument("--queries", required=True)+ ap.add_argument("--out", required=True); ap.add_argument("--index", default="perf-floor")+ ap.add_argument("--workers", type=int, default=4); ap.add_argument("--bulk-mb", type=float, default=5)+ ap.add_argument("--seed", type=int, default=2421)+ a = ap.parse_args()+ info = http("GET", a.es + "/")+ engine = "elasticsearch-%s (lucene %s)" % (info["version"]["number"], info["version"].get("lucene_version"))+ try:+ http("DELETE", "%s/%s" % (a.es, a.index))+ except Exception:+ pass+ http("PUT", "%s/%s" % (a.es, a.index), {"settings": {"number_of_shards": 1, "number_of_replicas": 0, "refresh_interval": "-1"}}) # default dynamic mapping, as the floor was measured+ src = sum(os.path.getsize(os.path.join(r, f)) for r, _, fs in os.walk(a.corpus) for f in fs)+ q = queue.Queue(maxsize=a.workers * 2)+ errs = []++ def worker():+ while True:+ b = q.get()+ if b is None:+ return+ last = None+ for attempt in range(5):+ try:+ r = http("POST", "%s/%s/_bulk" % (a.es, a.index), b, "application/x-ndjson")+ if r.get("errors"):+ errs.append(next(i for i in r["items"] if i["index"].get("error"))["index"]["error"])+ break+ except Exception as e: # 429 / transient+ last = e+ time.sleep(2 ** attempt)+ else:+ errs.append(str(last))++ th = [threading.Thread(target=worker, daemon=True) for _ in range(a.workers)]+ [t.start() for t in th]+ t0 = time.time(); buf = []; size = 0; n = 0+ for d in docs(a.corpus):+ line = b'{"index":{}}\n' + json.dumps(d).encode() + b"\n"+ buf.append(line); size += len(line); n += 1+ if size >= a.bulk_mb * MiB:+ q.put(b"".join(buf)); buf = []; size = 0+ if buf:+ q.put(b"".join(buf))+ for _ in th:+ q.put(None)+ [t.join() for t in th]+ http("POST", "%s/%s/_refresh" % (a.es, a.index)); http("POST", "%s/%s/_flush" % (a.es, a.index))+ wall = time.time() - t0+ unmerged = http("GET", "%s/%s/_stats/store" % (a.es, a.index))["_all"]["primaries"]["store"]["size_in_bytes"]+ # index_bytes is defined AFTER a force-merge to one segment: the refreshed+ # size swings with the segment count (1.77x vs 1.42x source on 1 GB).+ t1 = time.time()+ http("POST", "%s/%s/_forcemerge?max_num_segments=1&flush=true" % (a.es, a.index), timeout=3600)+ fm_s = time.time() - t1+ st = http("GET", "%s/%s/_stats/store,docs" % (a.es, a.index))["_all"]["primaries"]+ qs = json.load(open(a.queries))+ allq = [x for k in ("cli", "raw", "mcp") for x in qs.get(k, [])]+ lat = []+ for s in allq:+ t1 = time.time()+ http("POST", "%s/%s/_search?request_cache=false" % (a.es, a.index),+ {"size": 5, "query": {"simple_query_string": {"query": s, "fields": ["*"], "lenient": True}}})+ lat.append((time.time() - t1) * 1000)+ lat.sort()+ p95 = lat[min(len(lat) - 1, int(round(0.95 * (len(lat) - 1))))] if lat else None+ res = {"engine": engine, "corpus_mb": round(src / MiB), "seed": a.seed, "source_bytes": src,+ "docs": st["docs"]["count"], "index_bytes": st["store"]["size_in_bytes"],+ "index_bytes_refreshed_unmerged": unmerged, "forcemerge_s": round(fm_s, 1),+ "ingest_mb_s": round(src / MiB / wall, 2), "search_p95_ms": round(p95, 1) if p95 else None,+ "wall_s": round(wall, 1), "bulk_errors": len(errs), "first_error": errs[0] if errs else None}+ json.dump(res, open(a.out, "w"), indent=1)+ print(json.dumps(res))+++if __name__ == "__main__":+ main()diff --git a/scripts/perf/gate.py b/scripts/perf/gate.pynew file mode 100644index 000000000..d44aea36a--- /dev/null+++ b/scripts/perf/gate.py@@ -0,0 +1,500 @@+#!/usr/bin/env python3+"""Measurement + evaluation core of the autoindex release gate.++ gate.py run --bin X --corpus DIR --queries Q.json --work DIR [...] --out R.json+ gate.py eval --budgets budgets.toml --tier 1gb --candidate R.json+ [--baseline B.json] [--engine-baseline ES.json] --md OUT.md --json OUT.json++`run` measures ONE binary end to end on a private port with a throwaway data+dir and state dir; `eval` turns one or two result files into a verdict+table. Only the Python 3.11+ stdlib is used. Never submits xerj feedback+(XERJ_DISABLE_FEEDBACK=true everywhere).+"""+import argparse, json, os, re, signal, statistics, subprocess, sys, threading, time, tomllib, urllib.request++MiB = 1024 * 1024+FATAL_PATTERNS = {+ "eagain": re.compile(r"os error 11|Resource temporarily unavailable|EAGAIN", re.I),+ "oom": re.compile(r"out of memory|memory allocation of \d+ bytes failed|Cannot allocate memory|OOM", re.I),+ "abort": re.compile(r"SIGABRT|panicked at|FATAL:", re.I),+ "emfile": re.compile(r"Too many open files|os error 24", re.I),+}+BREAKER_RE = re.compile(r"engaging the parent memory circuit breaker", re.I)+RETRY_RE = re.compile(r"\bretry(ing)?\b|backing off|\b429\b|too many requests", re.I)+++def log(*a):+ print("[gate]", *a, file=sys.stderr, flush=True)+++def tree_bytes(path):+ total = 0+ for root, _, files in os.walk(path):+ for f in files:+ try:+ total += os.lstat(os.path.join(root, f)).st_size+ except OSError:+ pass+ return total+++def proc_status(pid):+ out = {}+ try:+ with open("/proc/%d/status" % pid) as f:+ for line in f:+ k, _, v = line.partition(":")+ if k in ("VmRSS", "VmHWM"):+ out[k] = int(v.split()[0]) // 1024 # MB+ elif k == "Threads":+ out[k] = int(v)+ except (OSError, ValueError):+ pass+ return out+++def children(pid):+ try:+ with open("/proc/%d/task/%d/children" % (pid, pid)) as f:+ return [int(x) for x in f.read().split()]+ except OSError:+ return []+++def tree_rss(pid):+ rss, stack = 0, [pid]+ while stack:+ p = stack.pop()+ rss += proc_status(p).get("VmRSS", 0)+ stack.extend(children(p))+ return rss+++class Sampler(threading.Thread):+ """Peak RSS / thread count of the node and the autoindex client tree."""++ def __init__(self):+ super().__init__(daemon=True)+ self.node_pid = None+ self.client_pid = None+ self.node_peak = 0+ self.client_peak = 0+ self.client_threads_peak = 0+ self.stop = False++ def run(self):+ while not self.stop:+ if self.node_pid:+ s = proc_status(self.node_pid)+ self.node_peak = max(self.node_peak, s.get("VmHWM", 0), s.get("VmRSS", 0))+ if self.client_pid:+ s = proc_status(self.client_pid)+ self.client_peak = max(self.client_peak, tree_rss(self.client_pid), s.get("VmHWM", 0))+ self.client_threads_peak = max(self.client_threads_peak, s.get("Threads", 0))+ time.sleep(0.25)+++def http(url, body=None, timeout=60):+ data = json.dumps(body).encode() if body is not None else None+ req = urllib.request.Request(url, data=data, headers={"Content-Type": "application/json"},+ method="POST" if data else "GET")+ with urllib.request.urlopen(req, timeout=timeout) as r:+ return json.loads(r.read() or b"null")+++class Node:+ def __init__(self, a, env, sampler):+ self.a, self.env, self.sampler = a, env, sampler+ self.url = "http://127.0.0.1:%d" % a.port+ self.p = None+ self.logs = []++ def start(self, tag):+ logp = os.path.join(self.a.work, "node-%s.log" % tag)+ self.logs.append(logp)+ t = time.time()+ self.p = subprocess.Popen([self.a.bin, "--insecure", "--data-dir", os.path.join(self.a.work, "data"),+ "--port", str(self.a.port), "--disable-feedback"],+ stdout=open(logp, "w"), stderr=subprocess.STDOUT, env=self.env)+ self.sampler.node_pid = self.p.pid+ while time.time() - t < 300:+ if self.p.poll() is not None:+ raise RuntimeError("node exited during boot (rc=%s), see %s" % (self.p.returncode, logp))+ try:+ http(self.url + "/_cluster/health", timeout=2)+ return time.time() - t+ except Exception:+ time.sleep(0.25)+ raise RuntimeError("node never became healthy")++ def alive(self):+ return self.p is not None and self.p.poll() is None++ def stop(self):+ if not self.alive():+ return+ self.p.send_signal(signal.SIGINT)+ try:+ self.p.wait(120)+ except subprocess.TimeoutExpired:+ self.p.kill(); self.p.wait()+++def run_autoindex(a, env, node, sampler, res):+ out = os.path.join(a.work, "autoindex.json")+ err = os.path.join(a.work, "autoindex.stderr")+ cmd = [a.bin, "autoindex", a.corpus, "--url", node.url, "--prefix", a.prefix,+ "--state-dir", os.path.join(a.work, "state"), "--approve", "proceed",+ "--json", "--progress", "plain", "--disable-feedback"] + a.autoindex_args+ log("autoindex:", " ".join(cmd))+ t = time.time()+ p = subprocess.Popen(cmd, stdout=open(out, "w"), stderr=open(err, "w"), env=env)+ sampler.client_pid = p.pid+ timed_out = False+ try:+ p.wait(a.timeout_min * 60)+ except subprocess.TimeoutExpired:+ timed_out = True+ p.send_signal(signal.SIGINT)+ try:+ p.wait(30)+ except subprocess.TimeoutExpired:+ p.kill(); p.wait()+ wall = time.time() - t+ sampler.client_pid = None+ res["autoindex"] = {"exit_code": p.returncode, "wall_s": round(wall, 1), "timed_out": timed_out, "cmd": cmd}+ try:+ txt = open(out).read().strip()+ j = json.loads(txt[txt.index("{"):]) if txt else {}+ except Exception:+ j = {}+ keep = ["files_total", "files_indexed", "records_total", "docs_per_sec", "workers", "bulk_congestion_events",+ "bulk_concurrency_final", "files_junk", "wall_seconds"]+ res["autoindex"]["summary"] = {k: j.get(k) for k in keep if k in j}+ stderr_txt = open(err, errors="replace").read()+ res["autoindex"]["stderr_tail"] = stderr_txt[-2000:]+ prog = re.findall(r"xerj-progress phase=index basis=bytes pct=[0-9.]+ items=\S+ bytes=(\d+)/(\d+)", stderr_txt)+ if prog:+ res["autoindex"]["progress_bytes_done"] = int(prog[-1][0])+ return stderr_txt+++def q_cli(a, env, url, q):+ t = time.time()+ try:+ r = subprocess.run([a.bin, "search", "--url", url, "--prefix", a.prefix, "-k", "5", "--json", q],+ env=env, capture_output=True, text=True, timeout=a.query_timeout)+ ms = (time.time() - t) * 1000+ j = json.loads(r.stdout)+ hits = len(j.get("hits", {}).get("hits", [])) if isinstance(j, dict) else 0+ return ms, hits, None if r.returncode == 0 else "rc=%d" % r.returncode+ except subprocess.TimeoutExpired:+ return a.query_timeout * 1000.0, 0, "timeout"+ except Exception as e:+ return (time.time() - t) * 1000, 0, "parse: %s" % str(e)[:80]+++def q_raw(a, env, url, q):+ body = {"size": 5, "query": {"simple_query_string": {"query": q, "fields": ["*"], "lenient": True}}}+ t = time.time()+ try:+ j = http("%s/%s-*/_search?request_cache=false" % (url, a.prefix), body, timeout=a.query_timeout)+ ms = (time.time() - t) * 1000+ return ms, len(j.get("hits", {}).get("hits", [])), None+ except Exception as e:+ return (time.time() - t) * 1000, 0, str(e)[:80]+++class MCP:+ def __init__(self, a, env, url):+ self.a = a+ self.p = subprocess.Popen([a.bin, "mcp", "--url", url, "--disable-feedback"], stdin=subprocess.PIPE,+ stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, env=env, bufsize=1)+ self.i = 0+ self.call("initialize", {"protocolVersion": "2024-11-05", "capabilities": {},+ "clientInfo": {"name": "perf-gate", "version": "1"}})+ self.p.stdin.write(json.dumps({"jsonrpc": "2.0", "method": "notifications/initialized"}) + "\n")+ self.p.stdin.flush()++ def call(self, m, params):+ self.i += 1+ self.p.stdin.write(json.dumps({"jsonrpc": "2.0", "id": self.i, "method": m, "params": params}) + "\n")+ self.p.stdin.flush()+ while True:+ line = self.p.stdout.readline()+ if not line:+ raise RuntimeError("mcp eof")+ try:+ j = json.loads(line)+ except ValueError:+ continue+ if j.get("id") == self.i:+ return j++ def query(self, q):+ t = time.time()+ try:+ j = self.call("tools/call", {"name": "xerj_search", "arguments": {"prefix": self.a.prefix, "query": q, "size": 5}})+ except Exception as e:+ return (time.time() - t) * 1000, 0, str(e)[:80]+ ms = (time.time() - t) * 1000+ r = j.get("result", {})+ txt = "".join(c.get("text", "") for c in r.get("content", []))+ hits = 0+ try:+ jj = json.loads(txt)+ hits = len(jj.get("hits", {}).get("hits", jj.get("results", []))) if isinstance(jj, dict) else len(jj)+ except Exception:+ hits = 1 if txt.strip() and "no results" not in txt.lower() else 0+ if r.get("isError") or "error" in j:+ msg = txt or json.dumps(j.get("error"))+ return ms, 0, ("mcp error: " + msg.replace("\n", " "))[:200]+ return ms, hits, None++ def close(self):+ try:+ self.p.stdin.close(); self.p.wait(10)+ except Exception:+ self.p.kill()+++def pct(v, p):+ v = sorted(v)+ if not v:+ return None+ k = (len(v) - 1) * p / 100.0+ lo = int(k); hi = min(lo + 1, len(v) - 1)+ return v[lo] + (v[hi] - v[lo]) * (k - lo)+++def cmd_run(a):+ os.makedirs(a.work, exist_ok=True)+ env = dict(os.environ, XERJ_DISABLE_FEEDBACK="true", XERJ_MAX_PROCESS_MEMORY_MB=str(a.cap_mb),+ XERJ_URL="http://127.0.0.1:%d" % a.port)+ env.pop("XERJ_API_KEY", None)+ queries = json.load(open(a.queries))+ res = {"label": a.label, "bin": a.bin, "corpus": a.corpus, "cap_mb": a.cap_mb, "started": time.strftime("%Y-%m-%dT%H:%M:%S%z")}+ try:+ res["version"] = subprocess.run([a.bin, "--version"], capture_output=True, text=True, env=env, timeout=30).stdout.strip()+ except Exception:+ res["version"] = "?"+ res["source_bytes"] = tree_bytes(a.corpus)+ sampler = Sampler(); sampler.start()+ node = Node(a, env, sampler)+ fatal = {}+ try:+ res["boot_s"] = round(node.start("ingest"), 2)+ stderr_txt = run_autoindex(a, env, node, sampler, res)+ res["node_alive_after_autoindex"] = node.alive()+ try:+ cat = http("%s/_cat/indices/%s-*?format=json&bytes=b&h=index,docs.count,store.size" % (node.url, a.prefix))+ except Exception:+ cat = []+ res["indices"] = len(cat)+ res["docs"] = sum(int(c.get("docs.count") or 0) for c in cat)+ res["index_store_bytes"] = sum(int(c.get("store.size") or 0) for c in cat)+ res["node_peak_rss_mb_ingest"] = sampler.node_peak+ node.stop()+ node_txt = "".join(open(p, errors="replace").read() for p in node.logs)+ res["breaker_trips"] = len(BREAKER_RE.findall(node_txt))+ res["bulk_retries"] = int(res["autoindex"]["summary"].get("bulk_congestion_events") or 0) + len(RETRY_RE.findall(stderr_txt))+ for k, rx in FATAL_PATTERNS.items():+ n = len(rx.findall(stderr_txt)) + len(rx.findall(node_txt))+ if n:+ fatal[k] = n+ res["fatal"] = fatal+ res["state_bytes"] = tree_bytes(os.path.join(a.work, "state"))+ res["data_dir_bytes"] = tree_bytes(os.path.join(a.work, "data"))+ res["client_peak_rss_mb"] = sampler.client_peak+ res["client_threads_peak"] = sampler.client_threads_peak+ if a.drop_caches:+ subprocess.run("sync; echo 3 | sudo -n tee /proc/sys/vm/drop_caches >/dev/null", shell=True)+ # Restart so every query below is served by a cold process (no+ # request/query/segment caches), the way an agent's first call is.+ res["restart_s"] = round(node.start("search"), 2)+ lat = {}+ for surface in ("cli", "raw", "mcp"):+ rows = []+ m = MCP(a, env, node.url) if surface == "mcp" else None+ for q in queries[surface]:+ if surface == "cli":+ ms, hits, e = q_cli(a, env, node.url, q)+ elif surface == "raw":+ ms, hits, e = q_raw(a, env, node.url, q)+ else:+ ms, hits, e = m.query(q)+ rows.append({"q": q, "ms": round(ms, 1), "hits": hits, "err": e})+ if m:+ m.close()+ lat[surface] = rows+ ms = [r["ms"] for r in rows]+ log("%s p50=%.0f p95=%.0f max=%.0f ms" % (surface, pct(ms, 50), pct(ms, 95), max(ms)))+ res["queries"] = lat+ allms = [r["ms"] for rows in lat.values() for r in rows]+ res["search"] = {"n": len(allms), "p50_ms": round(pct(allms, 50), 1), "p95_ms": round(pct(allms, 95), 1),+ "max_ms": round(max(allms), 1),+ "with_hits_pct": round(100.0 * sum(1 for rows in lat.values() for r in rows if r["hits"] and not r["err"]) / len(allms), 1),+ "errors": sum(1 for rows in lat.values() for r in rows if r["err"]),+ "first_error": next(("%s: %s" % (k, r["err"]) for k, rows in lat.items() for r in rows if r["err"]), None)}+ for s, rows in lat.items():+ ms = [r["ms"] for r in rows]+ res["search"][s] = {"p50_ms": round(pct(ms, 50), 1), "p95_ms": round(pct(ms, 95), 1), "max_ms": round(max(ms), 1)}+ except Exception as e:+ res["error"] = str(e)+ log("ERROR", e)+ finally:+ res["node_peak_rss_mb"] = sampler.node_peak+ node.stop()+ sampler.stop = True+ ai = res.get("autoindex", {})+ if ai.get("wall_s"):+ done = res["source_bytes"]+ if ai.get("exit_code") not in (0, 3) or ai.get("timed_out"):+ # incomplete run: count only bytes autoindex reported as indexed+ done = min(done, ai.get("progress_bytes_done") or 0)+ res["indexed_bytes"] = done+ res["throughput_mb_s"] = round(done / MiB / ai["wall_s"], 2)+ res["state_to_source_ratio"] = round(res.get("state_bytes", 0) / max(1, res["source_bytes"]), 3)+ res["index_bytes"] = res.get("data_dir_bytes", 0)+ json.dump(res, open(a.out, "w"), indent=1)+ log("wrote", a.out)+++# ---------------------------------------------------------------- eval ----+def cmd_eval(a):+ B = tomllib.load(open(a.budgets, "rb"))+ t = B["tier"][a.tier]+ ratio = B.get("ratio", {}).get(a.tier, B.get("ratio", {}).get("default", {}))+ c = json.load(open(a.candidate))+ base = json.load(open(a.baseline)) if a.baseline else None+ eng = json.load(open(a.engine_baseline)) if a.engine_baseline else None+ rows = [] # (check, value, budget, ok, note)++ def add(name, val, budget, ok, note=""):+ rows.append({"check": name, "value": val, "budget": budget, "pass": bool(ok), "note": note})++ ai = c.get("autoindex", {})+ fatal = c.get("fatal", {})+ completed = (not c.get("error") and not ai.get("timed_out") and ai.get("exit_code") in (0, 3)+ and c.get("node_alive_after_autoindex", False) and not fatal)+ note = []+ if ai.get("timed_out"): note.append("timed out after %s s" % ai.get("wall_s"))+ if ai.get("exit_code") not in (0, 3): note.append("exit %s" % ai.get("exit_code"))+ if not c.get("node_alive_after_autoindex", False): note.append("node died")+ if fatal: note.append(", ".join("%s x%d" % kv for kv in fatal.items()))+ if c.get("error"): note.append(c["error"][:80])+ add("autoindex completes (no OOM/abort/EAGAIN/timeout)", "yes" if completed else "NO", "exit 0/3, node alive", completed, "; ".join(note))+ tp = c.get("throughput_mb_s")+ add("autoindex throughput (MB/s)", tp, ">= %s" % t["min_throughput_mb_s"], completed and tp is not None and tp >= t["min_throughput_mb_s"],+ "%.0f of %.0f MB indexed in %s s" % (c.get("indexed_bytes", c.get("source_bytes", 0)) / MiB, c.get("source_bytes", 0) / MiB, ai.get("wall_s")))+ add("node peak RSS (MB), cap %d" % t["node_memory_cap_mb"], c.get("node_peak_rss_mb"), "<= %d" % t["max_node_peak_rss_mb"],+ (c.get("node_peak_rss_mb") or 0) <= t["max_node_peak_rss_mb"],+ "XERJ_MAX_PROCESS_MEMORY_MB=%s" % c.get("cap_mb"))+ add("client peak RSS (MB)", c.get("client_peak_rss_mb"), "<= %d" % t["max_client_peak_rss_mb"],+ (c.get("client_peak_rss_mb") or 0) <= t["max_client_peak_rss_mb"], "peak threads %s" % c.get("client_threads_peak"))+ sr = c.get("state_to_source_ratio", 0)+ add("state dir / source", sr, "<= %s" % t["max_state_to_source_ratio"], sr <= t["max_state_to_source_ratio"],+ "%.0f MB state" % (c.get("state_bytes", 0) / MiB))+ if "max_index_plus_state_mb" in t:+ tot = (c.get("index_bytes") or 0) + (c.get("state_bytes") or 0)+ add("index + state on disk (MB)", round(tot / MiB), "<= %d" % t["max_index_plus_state_mb"],+ tot / MiB <= t["max_index_plus_state_mb"], "%.2fx source" % (tot / max(1, c.get("source_bytes", 1))))+ add("indices per prefix", c.get("indices"), "<= %d" % t["max_indices_per_prefix"],+ 0 < (c.get("indices") or 0) <= t["max_indices_per_prefix"], "%s docs" % c.get("docs"))+ s = c.get("search", {})+ add("uncached search p95 (ms)", s.get("p95_ms"), "<= %d" % t["max_search_p95_ms"],+ s.get("p95_ms") is not None and s["p95_ms"] <= t["max_search_p95_ms"],+ " / ".join("%s %s" % (k, s[k]["p95_ms"]) for k in ("cli", "raw", "mcp") if k in s))+ add("uncached search max (ms)", s.get("max_ms"), "<= %d" % t["max_search_max_ms"],+ s.get("max_ms") is not None and s["max_ms"] <= t["max_search_max_ms"], "n=%s, errors=%s" % (s.get("n"), s.get("errors")))+ add("queries with hits (%)", s.get("with_hits_pct"), ">= %d" % t["min_queries_with_hits_pct"],+ (s.get("with_hits_pct") or 0) >= t["min_queries_with_hits_pct"])+ add("search errors", s.get("errors"), "0", s.get("errors") == 0, (s.get("first_error") or "")[:120])+ add("circuit-breaker trips", c.get("breaker_trips"), "<= %d" % t["max_breaker_trips"],+ c.get("breaker_trips") is not None and c["breaker_trips"] <= t["max_breaker_trips"])+ add("bulk retries / 429s", c.get("bulk_retries"), "<= %d" % t["max_bulk_retries"],+ c.get("bulk_retries") is not None and c["bulk_retries"] <= t["max_bulk_retries"])+ add("node restart (s)", c.get("restart_s"), "<= %d" % t["max_restart_s"],+ c.get("restart_s") is not None and c["restart_s"] <= t["max_restart_s"])++ lim = t["max_regression_pct"] / 100.0+ if base:+ def reg(name, key_fn, higher_better):+ cv, bv = key_fn(c), key_fn(base)+ if cv is None or bv is None or bv == 0:+ add("vs baseline: " + name, cv, "baseline %s" % bv, True, "skipped (missing)")+ return+ delta = (cv - bv) / bv+ worse = -delta if higher_better else delta+ add("vs baseline: " + name, cv, "%s %s (<= %d%% worse)" % ("baseline", bv, t["max_regression_pct"]),+ worse <= lim, "%+.0f%%" % (delta * 100))+ reg("throughput MB/s", lambda r: r.get("throughput_mb_s"), True)+ reg("search p95 ms", lambda r: r.get("search", {}).get("p95_ms"), False)+ reg("node peak RSS MB", lambda r: r.get("node_peak_rss_mb"), False)+ reg("client peak RSS MB", lambda r: r.get("client_peak_rss_mb"), False)+ reg("state/source", lambda r: r.get("state_to_source_ratio"), False)+ reg("index bytes", lambda r: r.get("index_bytes"), False)+ if eng and ratio:+ en = eng.get("engine", "baseline engine")+ eb = eng.get("source_bytes") or (eng.get("corpus_mb", 0) * MiB)+ if eb and abs(eb - c.get("source_bytes", 0)) > 0.05 * eb:+ add("engine baseline is for this corpus", "%.0f MB" % (eb / MiB), "%.0f MB" % (c.get("source_bytes", 0) / MiB), False,+ "re-measure the baseline on the same corpus")+ if "max_index_bytes_vs_baseline" in ratio and eng.get("index_bytes"):+ r_ = c.get("index_bytes", 0) / eng["index_bytes"]+ add("index bytes vs %s" % en, round(r_, 2), "<= %sx" % ratio["max_index_bytes_vs_baseline"],+ r_ <= ratio["max_index_bytes_vs_baseline"], "%.0f MB vs %.0f MB" % (c.get("index_bytes", 0) / MiB, eng["index_bytes"] / MiB))+ if "min_ingest_mb_s_vs_baseline" in ratio and eng.get("ingest_mb_s"):+ r_ = (c.get("throughput_mb_s") or 0) / eng["ingest_mb_s"]+ add("ingest MB/s vs %s" % en, round(r_, 2), ">= %sx" % ratio["min_ingest_mb_s_vs_baseline"],+ r_ >= ratio["min_ingest_mb_s_vs_baseline"], "%s vs %s MB/s" % (c.get("throughput_mb_s"), eng["ingest_mb_s"]))+ if "max_search_p95_vs_baseline" in ratio and eng.get("search_p95_ms") and s.get("p95_ms"):+ r_ = s["p95_ms"] / eng["search_p95_ms"]+ add("search p95 vs %s" % en, round(r_, 2), "<= %sx" % ratio["max_search_p95_vs_baseline"],+ r_ <= ratio["max_search_p95_vs_baseline"], "%s vs %s ms" % (s["p95_ms"], eng["search_p95_ms"]))++ ok = all(r["pass"] for r in rows)+ md = ["### autoindex release gate: %s (%s, tier %s)" % ("PASS" if ok else "FAIL", c.get("version"), a.tier), "",+ "| check | value | budget | result | note |", "|---|---|---|---|---|"]+ for r in rows:+ md.append("| %s | %s | %s | %s | %s |" % (r["check"], r["value"], r["budget"], "pass" if r["pass"] else "**FAIL**", r["note"].replace("|", "/")))+ md.append("")+ md.append("corpus %.0f MB, %s; %d indices, %s docs, index %.0f MB on disk" % (+ c.get("source_bytes", 0) / MiB, c.get("corpus"), c.get("indices") or 0, c.get("docs"), c.get("index_bytes", 0) / MiB))+ text = "\n".join(md)+ print(text)+ if a.md:+ open(a.md, "w").write(text + "\n")+ if a.json:+ json.dump({"pass": ok, "tier": a.tier, "budgets": t, "ratio_budgets": ratio if eng else None, "checks": rows,+ "candidate": c, "baseline": base, "engine_baseline": eng}, open(a.json, "w"), indent=1)+ sys.exit(0 if ok else 1)+++def main():+ ap = argparse.ArgumentParser()+ sp = ap.add_subparsers(dest="cmd", required=True)+ r = sp.add_parser("run")+ r.add_argument("--bin", required=True); r.add_argument("--corpus", required=True)+ r.add_argument("--queries", required=True); r.add_argument("--work", required=True)+ r.add_argument("--out", required=True); r.add_argument("--label", default="candidate")+ r.add_argument("--port", type=int, default=9400); r.add_argument("--prefix", default="pg")+ r.add_argument("--cap-mb", type=int, default=3072); r.add_argument("--timeout-min", type=float, default=15)+ r.add_argument("--query-timeout", type=float, default=60)+ r.add_argument("--drop-caches", action="store_true")+ r.add_argument("--autoindex-args", default="", help="extra args, space separated")+ e = sp.add_parser("eval")+ e.add_argument("--budgets", required=True); e.add_argument("--tier", default="1gb")+ e.add_argument("--candidate", required=True); e.add_argument("--baseline")+ e.add_argument("--engine-baseline"); e.add_argument("--md"); e.add_argument("--json")+ a = ap.parse_args()+ if a.cmd == "run":+ a.bin = os.path.abspath(a.bin); a.corpus = os.path.abspath(a.corpus); a.work = os.path.abspath(a.work)+ a.autoindex_args = a.autoindex_args.split()+ cmd_run(a)+ else:+ cmd_eval(a)+++if __name__ == "__main__":+ main()diff --git a/scripts/perf/gen_corpus.py b/scripts/perf/gen_corpus.pynew file mode 100755index 000000000..5064d0161--- /dev/null+++ b/scripts/perf/gen_corpus.py@@ -0,0 +1,299 @@+#!/usr/bin/env python3+"""Deterministic synthetic corpus for the autoindex release gate.++Same --seed + --size-mb => byte-identical tree on every machine, so a gate+number from CI and one from a laptop are about the same input. The mix is+modelled on the real 5 GB folder behind #2421-#2428 (Rust/Go dependency+sources, docs, HTML, JSONL audit data, logs, ADS-B flight traces):++ share family why it is here+ ----- ----------------------------- ------------------------------------+ 24% code (.rs .go .py .ts) the bulk of real repos; docs index+ 12% markdown docs index, headings, links+ 8% HTML pages with <table>s #2422: one index per page table+ 9% CSV (a few big + many small) tabular datasets; small distinct+ schemas tempt one-index-per-table+ 12% JSONL records record data+ 10% logs line records+ 15% big JSON, numeric arrays flight traces: few huge files+ 5% multi-array JSON + YAML #2422: grouped arrays skipped caps+ 5% tiny files (< 1 KiB) per-file overhead, many inodes+ deep/long paths #2424: 329-char index names++It also writes <out>/../queries.json (or --queries): 60 distinct queries, 20+per search surface, each built from planted phrases so every one has hits.+Only the stdlib is used.+"""+import argparse, json, os, random, sys++WORDS = ("alpha bravo charlie delta echo foxtrot golf hotel india juliet kilo lima mike "+ "november oscar papa quebec romeo sierra tango uniform victor whiskey xray yankee zulu "+ "buffer cache segment shard replica merge flush commit journal snapshot cursor token "+ "parser lexer schema mapping field record batch stream socket packet header payload "+ "retry backoff timeout deadline latency throughput budget quota limit window bucket "+ "kernel driver module plugin handler router service client server worker thread pool "+ "vector matrix tensor gradient kernel filter reducer mapper sorter index search query").split()++# Planted topics: every query is drawn from these; each phrase is written into+# several families so all three surfaces can find it.+TOPICS = [+ "fsync wal segment rotation", "tokio spawn blocking task", "http client retry exponential backoff",+ "parse toml config file", "tls certificate verification error", "use after free netfilter",+ "disk budget flood stage watermark", "json schema validation failure", "aircraft altitude ground speed",+ "goroutine leak context cancel", "memtable flush threshold", "circuit breaker memory pressure",+ "bloom filter false positive rate", "raft leader election timeout", "btree page split",+ "gzip decompression bomb", "oauth refresh token rotation", "dns resolver cache poisoning",+ "postgres vacuum autovacuum", "kafka consumer lag offset", "rate limiter token bucket",+ "mmap page fault storm", "unicode normalization nfc", "csv quoting escape delimiter",+ "sql injection prepared statement", "cron schedule daylight saving", "s3 multipart upload abort",+ "grpc deadline exceeded status", "websocket ping pong keepalive", "lru eviction policy",+ "simd utf8 validation", "heap profile allocation hotspot", "deadlock mutex ordering",+ "zstd dictionary compression level", "hnsw graph recall ef", "bm25 term frequency saturation",+ "semver caret range resolution", "crate feature unification", "go module proxy checksum",+ "flight squawk transponder code", "runway heading crosswind", "audit log retention policy",+ "invoice line item total", "sensor temperature humidity reading", "inventory warehouse sku",+ "payment refund chargeback", "kubernetes pod eviction", "terraform state lock",+ "docker layer cache miss", "prometheus histogram bucket", "opentelemetry span context",+ "redis pipeline transaction", "elastic shard allocation", "lucene segment merge policy",+ "regex catastrophic backtracking", "integer overflow checked arithmetic", "async cancellation safety",+ "file descriptor exhaustion", "thread spawn resource unavailable", "index name too long",+]++class Gen:+ def __init__(self, out, size_mb, seed):+ self.out = out+ self.target = int(size_mb * 1024 * 1024)+ self.r = random.Random(seed)+ self.written = 0+ self.files = 0+ self.by_family = {}+ # Pre-built pools make generation I/O-bound, not RNG-bound.+ r = self.r+ self.sentences = [" ".join(r.choice(WORDS) for _ in range(r.randint(6, 18))).capitalize() + "."+ for _ in range(4000)]+ self.idents = ["_".join(r.choice(WORDS) for _ in range(r.randint(1, 3))) for _ in range(3000)]++ # ---------- helpers ----------+ def topic(self):+ return self.r.choice(TOPICS)++ def para(self, n):+ s = self.r.choices(self.sentences, k=n)+ if self.r.random() < 0.5:+ s.insert(self.r.randrange(len(s) + 1), "Notes on %s." % self.topic())+ return " ".join(s)++ def write(self, fam, rel, data):+ p = os.path.join(self.out, rel)+ os.makedirs(os.path.dirname(p), exist_ok=True)+ b = data.encode() if isinstance(data, str) else data+ with open(p, "wb") as f:+ f.write(b)+ self.written += len(b)+ self.files += 1+ st = self.by_family.setdefault(fam, [0, 0])+ st[0] += 1+ st[1] += len(b)+ return len(b)++ # ---------- families ----------+ def code_file(self, i, nbytes=8192):+ r = self.r+ lang = r.choice(["rs", "go", "py", "ts"])+ repo = "repo-%02d" % (i % 24)+ mod = "/".join(r.choice(self.idents) for _ in range(r.randint(1, 4)))+ lines = []; size = 0; f = 0+ while size < nbytes:+ f += 1+ name = r.choice(self.idents) + "_%d" % f+ t = self.topic()+ if lang == "rs":+ l = "/// %s\npub fn %s(input: &[u8], limit: usize) -> Result<usize, Error> {\n // %s\n let mut n = 0usize;\n for b in input.iter().take(limit) { n = n.wrapping_add(*b as usize); }\n Ok(n)\n}\n" % (r.choice(self.sentences), name, t)+ elif lang == "go":+ l = "// %s handles %s.\nfunc %s(ctx context.Context, in []byte) (int, error) {\n\tif ctx.Err() != nil { return 0, ctx.Err() }\n\treturn len(in), nil\n}\n" % (name, t, name)+ elif lang == "py":+ l = 'def %s(data, limit=%d):\n """%s -- %s"""\n return sum(data[:limit])\n' % (name, r.randint(1, 999), r.choice(self.sentences), t)+ else:+ l = "// %s\nexport function %s(xs: number[]): number {\n return xs.reduce((a, b) => a + b, %d);\n}\n" % (t, name, r.randint(0, 9))+ lines.append(l); size += len(l) + 1+ self.write("code", "%s/src/%s/mod_%d.%s" % (repo, mod, i, lang), "\n".join(lines))++ def markdown(self, i, nbytes=4096):+ r = self.r+ body = ["# %s\n" % self.topic().title()]; size = 0+ while size < nbytes:+ l = "## %s\n\n%s\n" % (r.choice(self.idents).replace("_", " ").title(), self.para(r.randint(3, 10)))+ if r.random() < 0.3:+ l += "```\n%s\n```\n" % r.choice(self.sentences)+ body.append(l); size += len(l)+ self.write("markdown", "docs/%s/%s/note-%d.md" % (r.choice(WORDS), r.choice(WORDS), i), "\n".join(body))++ def html(self, i, nbytes=4096):+ r = self.r+ # A distinct column set per page: the #2422 trigger (202 of 454 b5+ # indices were HTML page tables, 197 of them under 16 rows).+ cols = r.sample(WORDS, r.randint(3, 6))+ rows = r.randint(3, 40)+ trs = "".join("<tr>%s</tr>" % "".join("<td>%s</td>" % (r.randint(0, 99999) if c in WORDS[:20] else r.choice(WORDS)) for c in cols) for _ in range(rows))+ paras = []; size = len(trs)+ while size < nbytes:+ p = "<h2>%s</h2><p>%s</p>" % (self.topic(), self.para(8)); paras.append(p); size += len(p)+ doc = ("<!doctype html><html><head><title>%s</title></head><body><h1>%s</h1><p>%s</p>"+ "<table><thead><tr>%s</tr></thead><tbody>%s</tbody></table>%s</body></html>") % (+ self.topic(), self.topic(), self.para(6), "".join("<th>%s_%d</th>" % (c, i % 97) for c in cols), trs, "".join(paras))+ self.write("html", "site/%s/page-%d.html" % (r.choice(WORDS), i), doc)++ def csv_small(self, i):+ r = self.r+ cols = ["%s_%d" % (c, i) for c in r.sample(WORDS, r.randint(3, 6))]+ lines = [",".join(cols)]+ for k in range(r.randint(4, 30)):+ lines.append(",".join(str(r.randint(0, 9999)) if j % 2 else r.choice(WORDS) for j in range(len(cols))))+ self.write("csv", "tables/t%03d/%s.csv" % (i % 400, r.choice(WORDS)), "\n".join(lines) + "\n")++ def csv_big(self, i, nbytes):+ r = self.r+ out = ["ts,sensor_id,site,temperature,humidity,status,note"]+ n = len(out[0])+ while n < nbytes:+ l = "%d,%d,%s,%.2f,%.1f,%s,%s" % (1700000000 + r.randint(0, 10**7), r.randint(1, 5000), r.choice(WORDS),+ r.uniform(-20, 45), r.uniform(0, 100), r.choice(["ok", "warn", "fail"]),+ r.choice(TOPICS))+ out.append(l); n += len(l) + 1+ self.write("csv", "warehouse/sensors/readings-%d.csv" % i, "\n".join(out) + "\n")++ def jsonl(self, i, nbytes):+ r = self.r+ out = []; n = 0+ while n < nbytes:+ rec = {"id": r.randint(1, 10**9), "ts": 1700000000 + r.randint(0, 10**7), "actor": r.choice(self.idents),+ "action": r.choice(["create", "update", "delete", "login", "export"]), "topic": r.choice(TOPICS),+ "message": r.choice(self.sentences), "amount": round(r.uniform(0, 5000), 2)}+ l = json.dumps(rec, separators=(",", ":")); out.append(l); n += len(l) + 1+ self.write("jsonl", "audit/%d/events-%d.jsonl" % (i % 12, i), "\n".join(out) + "\n")++ def log(self, i, nbytes):+ r = self.r+ out = []; n = 0+ lv = ["INFO", "WARN", "ERROR", "DEBUG"]+ while n < nbytes:+ l = "2026-10-%02dT%02d:%02d:%02d.%03dZ %s %s: %s (%s)" % (r.randint(1, 28), r.randint(0, 23), r.randint(0, 59), r.randint(0, 59),+ r.randint(0, 999), r.choice(lv), r.choice(self.idents), r.choice(self.sentences), r.choice(TOPICS))+ out.append(l); n += len(l) + 1+ self.write("logs", "var/log/svc-%d/app-%d.log" % (i % 8, i), "\n".join(out) + "\n")++ def flight_json(self, i, nbytes):+ # ADS-B style: one object with huge parallel numeric arrays.+ r = self.r+ n = max(1000, nbytes // 60)+ t0 = 1729000000+ obj = {"icao": "%06x" % r.randint(0, 0xFFFFFF), "callsign": "XRJ%04d" % i, "note": r.choice(TOPICS),+ "trace": [[t0 + k, round(r.uniform(-90, 90), 5), round(r.uniform(-180, 180), 5), r.randint(0, 45000),+ round(r.uniform(0, 600), 1), r.randint(0, 359)] for k in range(n)]}+ self.write("flight_json", "adsb/traces/%02d/trace_full_%06x.json" % (i % 16, i), json.dumps(obj, separators=(",", ":")))++ def multi_array_json(self, i):+ r = self.r+ obj = {}+ for a in range(r.randint(2, 6)):+ keys = r.sample(WORDS, 3)+ obj["%s_list" % r.choice(WORDS)] = [{k: r.randint(0, 999) if j else r.choice(WORDS) for j, k in enumerate(keys)} for _ in range(r.randint(2, 20))]+ if r.random() < 0.3:+ body = "\n".join("%s: %s" % (r.choice(self.idents), r.choice(WORDS)) for _ in range(r.randint(5, 40)))+ self.write("multi_json", "config/%s/settings-%d.yaml" % (r.choice(WORDS), i), body + "\n")+ else:+ self.write("multi_json", "grammars/%s/spec-%d.json" % (r.choice(WORDS), i), json.dumps(obj, indent=1))++ def tiny(self, i):+ r = self.r+ self.write("tiny", "small/%02d/%02d/f%d.txt" % (i % 50, (i // 50) % 50, i), r.choice(self.sentences) + " " + self.topic() + "\n")++ def deep(self, i):+ # Go-module-like deep paths: the slug that produced a 329-char index name (#2424).+ r = self.r+ segs = ["golang.org", "x", "github.com-%s-%s" % (r.choice(WORDS), r.choice(WORDS))]+ segs += ["%s-%s-%s-%s-v%d" % (r.choice(WORDS), r.choice(WORDS), r.choice(WORDS), r.choice(WORDS), r.randint(1, 9)) for _ in range(r.randint(8, 14))]+ p = "gomod/" + "/".join(segs)+ cols = ["%s_%d" % (c, i) for c in r.sample(WORDS, 3)]+ self.write("deep_paths", p + "/data-%d.csv" % i, ",".join(cols) + "\n" + "\n".join("%d,%s,%d" % (k, r.choice(WORDS), r.randint(0, 99)) for k in range(5)) + "\n")+ self.write("deep_paths", p + "/README-%d.md" % i, "# %s\n\n%s\n" % (self.topic(), self.para(4)))++ def run(self):+ T = self.target+ gb = T / float(1 << 30)+ shares = [("code", .24), ("markdown", .12), ("html", .08), ("csv", .09), ("jsonl", .12), ("logs", .10),+ ("flight_json", .15), ("multi_json", .05), ("tiny", .05)]+ budget = {k: T * v for k, v in shares}+ # File COUNTS per GB, tuned to the real folder's ~30k files/GB: the+ # count drives per-file and per-dataset overhead, the byte share+ # drives ingest volume. Sizes grow with --size-mb; counts scale too+ # but never drop below a floor, so a 50 MB smoke run keeps the shape.+ def cnt(per_gb, floor):+ return max(floor, int(per_gb * gb))+ for i in range(cnt(120, 20)):+ self.deep(i)+ # Big-file families: a fixed number of large files whose size scales.+ for i in range(4):+ self.flight_json(i, int(budget["flight_json"] / 4))+ for i in range(24):+ self.jsonl(i, int(budget["jsonl"] / 24))+ for i in range(16):+ self.log(i, int(budget["logs"] / 16))+ for i in range(3):+ self.csv_big(i, int(budget["csv"] * 0.8 / 3))+ n_small_csv = cnt(800, 60)+ for i in range(n_small_csv):+ self.csv_small(i)+ for i in range(cnt(1500, 80)):+ self.multi_array_json(i)+ for i in range(cnt(8000, 400)):+ self.tiny(i)+ n_html = cnt(3000, 150)+ for i in range(n_html):+ self.html(i, int(budget["html"] / n_html))+ n_md = cnt(6000, 300)+ for i in range(n_md):+ self.markdown(i, int(budget["markdown"] / n_md))+ n_code = cnt(16000, 800)+ for i in range(n_code):+ self.code_file(i, int(budget["code"] / n_code))+ # Top up with more code files so the total lands on --size-mb.+ i = n_code+ while self.written < T:+ self.code_file(i, int(budget["code"] / n_code)); i += 1+ return self+++def queries(seed):+ r = random.Random(seed + 1)+ t = TOPICS[:]+ r.shuffle(t)+ # 60 distinct strings, 20 per surface; no string repeats across surfaces,+ # so a cache warmed by one surface cannot serve another.+ return {"cli": t[0:20], "raw": t[20:40], "mcp": t[40:60]}+++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("--out", required=True)+ ap.add_argument("--size-mb", type=float, default=1024)+ ap.add_argument("--seed", type=int, default=2421)+ ap.add_argument("--queries", help="write the query set here (default <out>.queries.json)")+ a = ap.parse_args()+ if os.path.exists(a.out) and os.listdir(a.out):+ sys.exit("refusing to write into non-empty %s" % a.out)+ os.makedirs(a.out, exist_ok=True)+ g = Gen(a.out, a.size_mb, a.seed).run()+ qpath = a.queries or a.out.rstrip("/") + ".queries.json"+ with open(qpath, "w") as f:+ json.dump(queries(a.seed), f, indent=1)+ manifest = {"seed": a.seed, "size_mb_requested": a.size_mb, "bytes": g.written, "files": g.files,+ "by_family": {k: {"files": v[0], "bytes": v[1]} for k, v in sorted(g.by_family.items())},+ "queries": qpath}+ with open(a.out.rstrip("/") + ".manifest.json", "w") as f:+ json.dump(manifest, f, indent=1)+ print(json.dumps(manifest))+++if __name__ == "__main__":+ main()