Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

perf: release performance gate for autoindex + search (budgets, ES floor, rc.99 regression harness) (#2432)

xerj-org255519b05111
diff --git a/docs/perf-release-gate.md b/docs/perf-release-gate.md
new file mode 100644
index 000000000..41a924630
--- /dev/null
+++ b/docs/perf-release-gate.md
@@ -0,0 +1,160 @@
+# Autoindex / search release performance gate
+
+`scripts/perf/autoindex_release_gate.sh` blocks a build that repeats the
+rc.99 field failures: #2421 (state dir 1.6x source), #2422 (454 indices,
+13–23 s searches), #2423 (0.2 MB/s, OOM), #2424 (planning memory, long
+names), #2427 (node over its own memory cap) and #2428 (client EAGAIN abort).
+It runs the user-facing path end to end, `xerj autoindex <folder>` with
+default flags, against a deterministic synthetic corpus on a throwaway node
+with a fixed memory cap. It then measures uncached search on all three
+surfaces and fails on any budget miss.
+
+## Run it locally
+
+```bash
+(cd engine && cargo build --release --locked -p xerj-server)
+scripts/perf/autoindex_release_gate.sh --bin engine/target/release/xerj # 1 GB tier
+scripts/perf/autoindex_release_gate.sh --bin engine/target/release/xerj --tier smoke
+scripts/perf/autoindex_release_gate.sh --bin engine/target/release/xerj \
+ --baseline ~/xerj-prev/xerj # +regression checks
+scripts/perf/autoindex_release_gate.sh --bin engine/target/release/xerj \
+ --engine-baseline es-baseline.json # +ES/Lucene floor
+```
+
+| flag | meaning |
+|---|---|
+| `--tier smoke\|1gb\|5gb` | budget tier in `scripts/perf/budgets.toml` (sets corpus size, memory cap, timeout) |
+| `--size-mb N` | override corpus size (budgets stay those of the tier) |
+| `--seed N` | corpus seed (default 2421) |
+| `--work DIR` | scratch: corpus + per-run data/state (default `$RUNNER_TEMP` or `/tmp`/xerj-perf-gate) |
+| `--corpus-dir DIR` | reuse or put the corpus here (reused if its manifest matches size + seed) |
+| `--out DIR` | results (`candidate.json`, `baseline.json`, `results.json`, `summary.md`, log tails) |
+| `--baseline BIN` | also run the previous release binary on the same corpus; fail on >20% regression |
+| `--engine-baseline FILE` | ES/Lucene floor JSON (from `es_baseline.py`); enables `[ratio.*]` budgets |
+| `--autoindex-args "..."` | extra autoindex flags (default: none, the user-facing path) |
+| `--keep` | keep run data/state dirs |
+| `MIN_FREE_GB` env | disk reserve the guard keeps free (default 8) |
+
+Exit 0 means every budget was met. Exit 1 means at least one was missed, and the markdown table is
+printed and written to `summary.md` (and appended to `$GITHUB_STEP_SUMMARY`
+in Actions). Exit 2 means a usage or environment error, e.g. not enough disk. Requires
+Python ≥ 3.11 (stdlib only). On a 4-core machine a passing build finishes the
+1 GB tier in under 20 minutes: corpus about 25 s, autoindex ≤ 15 min by budget, restart
+plus 60 queries about 1 min.
+
+## What a run does
+
+1. `gen_corpus.py` writes a byte-for-byte deterministic corpus plus
+ `<corpus>.queries.json` (20 queries per surface, 60 distinct strings
+ drawn from phrases planted in the corpus, so each should hit) and
+ `<corpus>.manifest.json`. The mix per GB is code (rs/go/py/ts, about 24%),
+ markdown (12%), HTML with tables (8%), 3 large plus about 800 small
+ distinct-schema CSVs, 24 JSONL, 16 logs, 4 big JSON flight-trace files with numeric
+ arrays (15%), 1500 multi-array JSON/YAML, 8000 tiny files and deep Go-module
+ style paths up to about 500 chars.
+2. `gate.py run` starts `xerj --insecure --data-dir … --port … --disable-feedback`
+ with `XERJ_MAX_PROCESS_MEMORY_MB=<tier cap>` and `XERJ_DISABLE_FEEDBACK=true`,
+ then runs `xerj autoindex <corpus> --approve proceed --json` under the tier
+ timeout. It samples node and client process-tree RSS and threads from `/proc`
+ every 0.5 s.
+3. It reads `_cat/indices/<prefix>-*`, stops the node, scans the node log for
+ parent-breaker trips (`engaging the parent memory circuit breaker`), fatal
+ patterns (EAGAIN / os error 11, OOM, SIGABRT/panic, EMFILE) and retry/429
+ lines, and measures the state dir and data dir sizes.
+4. It restarts the node (timed) so the page cache and query caches are cold,
+ then runs 20 queries through `xerj search --json`, 20 through raw
+ `POST /<prefix>-*/_search?request_cache=false` and 20 through `xerj mcp`
+ (`tools/call xerj_search`). No query string repeats.
+5. `gate.py eval` compares against `budgets.toml`.
+
+## Budgets
+
+See `scripts/perf/budgets.toml` for the numbers and the reasoning behind each one.
+
+| check | smoke (200 MB) | 1gb | 5gb (release) | why |
+|---|---|---|---|---|
+| autoindex completes (exit 0/3, node alive, no OOM/abort/EAGAIN, within timeout) | 6 min | 15 min | 30 min | #2423, #2427, #2428 |
+| throughput MB/s ≥ (absolute backstop) | 1.5 | 3.0 | 3.0 | about 0.5x the low end of ES on a 4-vCPU runner; rc.99 did 0.2–0.6 |
+| node peak RSS ≤ memory cap (MB) | 2048 | 3072 | 3072 | #2427: rc.99 hit 3.9 GB (autoindex) / 5.6 GB (`_bulk`) under a 3 GB cap |
+| client peak RSS (MB) ≤ | 768 | 1024 | 1536 | #2424, #2428 |
+| state dir / source ≤ | 1.2 | 1.2 | 1.2 | #2421: 8.4 GB state for 5 GB |
+| index + state on disk ≤ (MB) | 250 | 1300 | 6400 | about 1.2x source, stricter than ES force-merged (1.42x) |
+| indices per prefix ≤ | 40 | 40 | 40 | #2422: 454; #2430 caps a run at 32 |
+| uncached search p95 / max (ms) | 1000 / 2000 | 1000 / 2000 | 1000 / 2000 | #2422: 13–23 s |
+| search errors | 0 | 0 | 0 | |
+| queries with hits ≥ | 90% | 90% | 90% | catches "fast because it found nothing" |
+| parent-breaker trips | 0 | 0 | 0 | #2427 |
+| bulk retries / 429s ≤ | 20 | 20 | 50 | #2427 |
+| node restart (s) ≤ | 60 | 60 | 120 | |
+| regression vs `--baseline` binary | 20% | 20% | 20% | throughput, p95, node/client RSS, state ratio, index bytes |
+| index bytes vs ES (force-merged) ≤ | 1.25x | 1.0x | 1.0x | ES floor (`[ratio.*]`) |
+| ingest MB/s vs ES (default mapping) ≥ | 0.3x | 0.5x | 0.5x | ES floor; stretch goal 0.8x |
+| search p95 vs ES ≤ | — | 1.5x | 1.5x | ES floor |
+
+### Lucene / Elasticsearch floor
+
+Absolute budgets drift with runner hardware, so index size and indexing speed are
+also held against Elasticsearch on the same corpus and the same machine.
+`scripts/perf/es_baseline.py` bulk-loads the corpus into a throwaway ES. It writes
+one doc per line for jsonl/csv/log and one doc per file otherwise, using the default
+dynamic mapping, 1 shard and 0 replicas. Ingest time runs until the index is refreshed
+and flushed. It then **force-merges to one segment**, and `index_bytes` is the merged
+size, because the unmerged size swings with the segment count (1.77x vs 1.42x source).
+Finally it runs the same 60 queries uncached and writes:
+
+```json
+{"engine": "elasticsearch-9.x (lucene 10.x)", "corpus_mb": 1024, "seed": 2421,
+ "index_bytes": 0, "ingest_mb_s": 0.0, "search_p95_ms": 0.0}
+```
+
+Pass that file as `--engine-baseline` and the `[ratio.<tier>]` keys apply:
+`xerj_index_bytes ≤ k·es_index_bytes`, `xerj_ingest_MBps ≥ k·es_ingest_MBps`,
+and `xerj_p95 ≤ k·es_p95`. In CI, ES runs on the same runner, so the floor scales with
+the runner automatically. ES always runs **before** xerj and is stopped before xerj
+starts, never at the same time: ES uses about 5.5 GB RSS with a 4 GB heap, and xerj has a 3 GB cap.
+The last good baseline is cached and used as a fallback when the live ES step fails.
+
+Reference floor, 1 GB corpus with seed 2421, 4 pinned cores on a heavily loaded shared box,
+single noisy run (perf-floor `results.md`, 2026-10-25):
+
+| engine | MiB/s | index bytes / source | peak RSS MB |
+|---|---|---|---|
+| ES 9.5.5 default dynamic mapping, 4 GB heap (refreshed, 28 segments) | 9.05 | 1.77 | 5,551 |
+| same, after force-merge to 1 segment (+48.7 s) | | 1.42 | |
+| Lucene 10.5.1 StandardAnalyzer, stored fields on, 4 threads | 25.4 | 0.78 | 3,036 |
+| Lucene 10.5.1 StandardAnalyzer, stored fields off, 4 threads | 36.0 | 0.30 | 2,518 |
+| xerj rc.99 `_bulk` text-only (cap 3072 MB; died at about 900 s) | 0.37 | n/a | 5,643 |
+| xerj rc.99 autoindex (stopped at 38% after 661 s) | 0.59 | about 0.92 (est.) | about 2,960 |
+
+So rc.99 runs at about 0.04–0.07x of ES and misses the 0.5x floor by an order of magnitude.
+
+Runner sizing: public-repo `ubuntu-latest` has 4 vCPU, 16 GB and a 14 GB SSD, which is what the budgets
+assume. Private repos get 2 vCPU and 8 GB there, so use the `smoke` tier with a 2 GB ES heap,
+or a larger runner.
+
+```bash
+python3 scripts/perf/gen_corpus.py --out /tmp/c1g --size-mb 1024 --seed 2421
+docker run -d --name es -p 9200:9200 -e discovery.type=single-node \
+ -e xpack.security.enabled=false -e ES_JAVA_OPTS="-Xms4g -Xmx4g" \
+ docker.elastic.co/elasticsearch/elasticsearch:9.5.5
+python3 scripts/perf/es_baseline.py --es http://127.0.0.1:9200 --corpus /tmp/c1g \
+ --queries /tmp/c1g.queries.json --out es-baseline.json
+```
+
+## CI
+
+The workflow (`.github/workflows/perf-gate.yml`) is maintained by hand. See the
+tracking issue for the ready-to-paste YAML. It runs:
+
+* on PRs touching autoindex, engine or server paths: the 1 GB tier;
+* nightly: the 1 GB tier with `--baseline` set to the latest release binary, plus the ES floor;
+* on `v*` tags, as a job `release.yml` depends on (`needs: [build, perf-gate]`): the 5 GB tier.
+
+Results are uploaded as the `perf-gate-*` artifact (`results.json` holds every
+raw measurement, budget and verdict) and summarized in the job summary and as
+a PR comment.
+
+## Changing a budget
+
+Edit `scripts/perf/budgets.toml` in the same PR as the change that needs it,
+with a comment explaining why. Loosening a budget needs a linked issue.
diff --git a/scripts/perf/autoindex_release_gate.sh b/scripts/perf/autoindex_release_gate.sh
new file mode 100755
index 000000000..9f60e32af
--- /dev/null
+++ b/scripts/perf/autoindex_release_gate.sh
@@ -0,0 +1,105 @@
+#!/usr/bin/env bash
+# Autoindex/search release performance gate. See docs/perf-release-gate.md.
+#
+# scripts/perf/autoindex_release_gate.sh --bin target/release/xerj [--tier 1gb]
+# [--size-mb N] [--seed 2421] [--corpus-dir DIR] [--work DIR] [--out DIR]
+# [--baseline PREV_XERJ_BINARY] [--engine-baseline es-baseline.json]
+# [--port 9400] [--autoindex-args "..."] [--keep]
+#
+# Generates (or reuses) the deterministic synthetic corpus, runs the candidate
+# binary (and optionally the previous release binary) through gate.py on a
+# throwaway node with a fixed memory cap, evaluates scripts/perf/budgets.toml
+# and writes OUT/results.json + OUT/summary.md. Exit 0 = every budget met,
+# 1 = at least one budget missed, 2 = usage / environment error.
+set -euo pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+BIN="" TIER="1gb" SIZE_MB="" SEED=2421 CORPUS="" WORK="" OUT="" BASE="" ENG="" PORT=9400 KEEP=0 AI_ARGS=""
+BUDGETS="$HERE/budgets.toml"
+MIN_FREE_GB="${MIN_FREE_GB:-8}"
+
+die() { echo "perf-gate: $*" >&2; exit 2; }
+while [[ $# -gt 0 ]]; do
+ case "$1" in
+ --bin) BIN="$2"; shift 2;;
+ --tier) TIER="$2"; shift 2;;
+ --size-mb) SIZE_MB="$2"; shift 2;;
+ --seed) SEED="$2"; shift 2;;
+ --corpus-dir) CORPUS="$2"; shift 2;;
+ --work) WORK="$2"; shift 2;;
+ --out) OUT="$2"; shift 2;;
+ --baseline) BASE="$2"; shift 2;;
+ --engine-baseline) ENG="$2"; shift 2;;
+ --budgets) BUDGETS="$2"; shift 2;;
+ --port) PORT="$2"; shift 2;;
+ --autoindex-args) AI_ARGS="$2"; shift 2;;
+ --keep) KEEP=1; shift;;
+ -h|--help) sed -n '2,16p' "$0"; exit 0;;
+ *) die "unknown argument: $1";;
+ esac
+done
+[[ -n "$BIN" && -x "$BIN" ]] || die "--bin must point at an executable xerj binary"
+[[ -z "$BASE" || -x "$BASE" ]] || die "--baseline must point at an executable xerj binary"
+command -v python3 >/dev/null || die "python3 (3.11+) required"
+python3 -c 'import tomllib' 2>/dev/null || die "python3 >= 3.11 required (tomllib)"
+
+tier_val() { python3 -c 'import sys,tomllib;print(tomllib.load(open(sys.argv[1],"rb"))["tier"][sys.argv[2]][sys.argv[3]])' "$BUDGETS" "$TIER" "$1"; }
+tier_val corpus_mb >/dev/null 2>&1 || die "tier '$TIER' not in $BUDGETS"
+SIZE_MB="${SIZE_MB:-$(tier_val corpus_mb)}"
+CAP_MB="$(tier_val node_memory_cap_mb)"
+TIMEOUT_MIN="$(tier_val autoindex_timeout_min)"
+
+WORK="${WORK:-${RUNNER_TEMP:-/tmp}/xerj-perf-gate}"
+OUT="${OUT:-$WORK/out}"
+CORPUS="${CORPUS:-$WORK/corpus-${SIZE_MB}mb-s${SEED}}"
+mkdir -p "$WORK" "$OUT"
+WORK="$(cd "$WORK" && pwd)"; OUT="$(cd "$OUT" && pwd)"
+
+# Disk guard: corpus + per-run data/state can reach ~4x the corpus on a bad
+# build (rc.99 wrote 1.6x state + 2x index). Refuse rather than fill the disk.
+free_gb() { df -Pk "$1" | awk 'NR==2{printf "%d", $4/1048576}'; }
+need_gb=$(( (SIZE_MB * 4 + 1023) / 1024 ))
+[[ -d "$CORPUS" ]] && need_gb=$(( (SIZE_MB * 3 + 1023) / 1024 ))
+have=$(free_gb "$WORK")
+if (( have - need_gb < MIN_FREE_GB )); then
+ die "need ~${need_gb} GB scratch + ${MIN_FREE_GB} GB reserve on $(df -P "$WORK" | awk 'NR==2{print $6}'), only ${have} GB free (set MIN_FREE_GB or --size-mb)"
+fi
+
+echo "perf-gate: tier=$TIER corpus=${SIZE_MB}MB seed=$SEED cap=${CAP_MB}MB timeout=${TIMEOUT_MIN}min bin=$BIN"
+if [[ -f "$CORPUS.manifest.json" && -f "$CORPUS.queries.json" ]] && \
+ python3 -c 'import json,sys;m=json.load(open(sys.argv[1]));sys.exit(0 if float(m.get("size_mb_requested",-1))==float(sys.argv[2]) and m.get("seed")==int(sys.argv[3]) else 1)' "$CORPUS.manifest.json" "$SIZE_MB" "$SEED"; then
+ echo "perf-gate: reusing corpus $CORPUS"
+else
+ if [[ -e "$CORPUS" ]]; then die "$CORPUS exists but is not a seed=$SEED size=${SIZE_MB}MB corpus; remove it or pick another --corpus-dir"; fi
+ t0=$(date +%s)
+ python3 "$HERE/gen_corpus.py" --out "$CORPUS" --size-mb "$SIZE_MB" --seed "$SEED"
+ echo "perf-gate: corpus generated in $(( $(date +%s) - t0 ))s"
+fi
+
+export XERJ_DISABLE_FEEDBACK=true
+run_one() { # label bin port
+ local label="$1" bin="$2" port="$3"
+ rm -rf "$WORK/run-$label"
+ python3 "$HERE/gate.py" run --bin "$bin" --label "$label" --corpus "$CORPUS" --queries "$CORPUS.queries.json" \
+ --work "$WORK/run-$label" --out "$OUT/$label.json" --port "$port" --cap-mb "$CAP_MB" \
+ --timeout-min "$TIMEOUT_MIN" --autoindex-args "$AI_ARGS"
+ cp "$WORK/run-$label/autoindex.stderr" "$OUT/$label.autoindex.stderr" 2>/dev/null || true
+ for f in node-ingest.log node-search.log; do
+ [[ -f "$WORK/run-$label/$f" ]] && tail -c 2000000 "$WORK/run-$label/$f" > "$OUT/$label.$f" || true
+ done
+ (( KEEP )) || rm -rf "$WORK/run-$label"
+}
+
+if [[ -n "$BASE" ]]; then run_one baseline "$BASE" $((PORT + 10)); fi
+run_one candidate "$BIN" "$PORT"
+
+eval_args=(--budgets "$BUDGETS" --tier "$TIER" --candidate "$OUT/candidate.json" --md "$OUT/summary.md" --json "$OUT/results.json")
+[[ -n "$BASE" ]] && eval_args+=(--baseline "$OUT/baseline.json")
+[[ -n "$ENG" ]] && eval_args+=(--engine-baseline "$ENG")
+set +e
+python3 "$HERE/gate.py" eval "${eval_args[@]}"
+rc=$?
+set -e
+[[ -n "${GITHUB_STEP_SUMMARY:-}" ]] && cat "$OUT/summary.md" >> "$GITHUB_STEP_SUMMARY"
+echo "perf-gate: results in $OUT (results.json, summary.md); exit $rc"
+exit $rc
diff --git a/scripts/perf/budgets.toml b/scripts/perf/budgets.toml
new file mode 100644
index 000000000..74060e246
--- /dev/null
+++ b/scripts/perf/budgets.toml
@@ -0,0 +1,161 @@
+# Release performance budgets for `scripts/perf/autoindex_release_gate.sh`.
+#
+# Every key is a hard ceiling (max_*) or floor (min_*). A run fails when ANY
+# budget is broken; the gate prints which, as a markdown table, and exits 1.
+# Tiers are picked by --tier (default: "1gb"). Values are proposals sized
+# from the rc.99 regressions found on 2026-10-25 (#2421-#2428); each comment
+# says why the number is where it is. Change a number here, in a reviewed PR,
+# never by editing the workflow.
+
+[tier.smoke]
+# A ~200 MB corpus for a quick local check. Same shape, looser absolute
+# time floors (fixed per-run overheads dominate at this size).
+corpus_mb = 200
+node_memory_cap_mb = 2048
+autoindex_timeout_min = 6
+min_throughput_mb_s = 1.5
+max_node_peak_rss_mb = 2048
+max_client_peak_rss_mb = 768
+max_state_to_source_ratio = 1.2
+max_index_plus_state_mb = 250
+max_indices_per_prefix = 40
+max_search_p95_ms = 1000
+max_search_max_ms = 2000
+max_breaker_trips = 0
+max_bulk_retries = 20
+max_restart_s = 60
+min_queries_with_hits_pct = 90
+max_regression_pct = 20
+
+[tier.1gb]
+# The PR / nightly tier. Runnable in < 20 min on 4 cores.
+corpus_mb = 1024
+# Node cap. 3 GB is the cap #2427 was measured with (node hit 3.9 GB). A
+# 1 GB corpus must fit with room to spare; the gate sets
+# XERJ_MAX_PROCESS_MEMORY_MB to this and the RSS ceiling below equals it:
+# the cap is a promise, so exceeding it at all is a failure.
+node_memory_cap_mb = 3072
+# Hard wall-clock stop for `xerj autoindex` (a hang counts as a failure).
+# 15 min > 1024 MB / 3 MB/s (5.7 min) * 2.5, so only a real stall hits it.
+autoindex_timeout_min = 15
+# Source MB / autoindex wall seconds. rc.99 on the 5 GB folder managed
+# ~0.2 MB/s (748 MB in 68 min, #2423). The #2423 ask (5 GB in 20 min) is
+# 4.3 MB/s; 3 MB/s at 1 GB leaves headroom for 4-core CI runners and the
+# fixed per-run startup/finalize costs that weigh more on a smaller corpus.
+min_throughput_mb_s = 3.0
+max_node_peak_rss_mb = 3072
+# Client RSS. #2424: phase A held several GB; after #2417 the client should
+# be a streaming pipeline, so 1 GB for 1 GB of input is generous.
+max_client_peak_rss_mb = 1024
+# (state dir bytes) / (source bytes) after the run commits. rc.99: >= 1.6x
+# (8.4 GB for 5 GB, #2421). Resume metadata + journal must stay well under
+# a full copy; 1.2x is the ceiling the release issue asks for.
+max_state_to_source_ratio = 1.2
+# Absolute backstop on (node data dir + autoindex state dir) in MiB:
+# 1.3 GB, about 1.2x the 1 GB source. ES 9.5.5 force-merged = 1.42x source on
+# this corpus (perf-floor results.md), so this is stricter than ES. It holds
+# even when no engine baseline is available.
+max_index_plus_state_mb = 1300
+# rc.99 made 454 indices for one folder (#2422). #2430 adds a run cap of
+# 32 datasets plus docs/code indices; 40 leaves slack for those.
+max_indices_per_prefix = 40
+# Uncached search latency (wall-clock incl. client start for the CLI, HTTP
+# round-trip for raw _search, tools/call round-trip for MCP). rc.99:
+# 13-23 s on 5 GB (#2422). 1 s p95 / 2 s max is what an agent tolerates
+# per call; #2422 targets p95 < 500 ms on 5 GB once fan-out is fixed.
+max_search_p95_ms = 1000
+max_search_max_ms = 2000
+# Parent memory circuit breaker engagements in the node log (writes -> 429).
+# #2427: rc.99 tripped it at 7.6 GB, then was OOM-killed. A release build on
+# a 1 GB corpus under a 3 GB cap must never need it.
+max_breaker_trips = 0
+# Bulk back-pressure events reported by autoindex (bulk_congestion_events,
+# i.e. 429/overload retries) + retry lines in its stderr. A few are normal
+# adaptive-concurrency probing; dozens mean the node cannot keep up.
+max_bulk_retries = 20
+# Node restart before the search phase (replay/open of every index).
+max_restart_s = 60
+# Sanity floor: every query is built from phrases planted in the corpus, so
+# a "fast" search that finds nothing is a failure, not a win.
+min_queries_with_hits_pct = 90
+# With --baseline: fail when the candidate is worse than the previous
+# release binary by more than this on throughput, search p95, peak RSS
+# (node and client), or state ratio, on the same corpus and machine.
+max_regression_pct = 20
+
+[tier.5gb]
+# The pre-release tier (v* tags). Mirrors the 5 GB folder of #2421-#2428.
+corpus_mb = 5120
+# Same 3 GB cap as 1gb (the #2427 cap). #2423 asked "under 4 GB" for 5 GB;
+# the floor review (2026-10-25) holds the release to 3 GB. rc.99 reached
+# 5.6 GB RSS under a 3072 MB cap in bulk ingest (#2427).
+node_memory_cap_mb = 3072
+# #2423 ask: "finish in under 20 min"; 30 min is the hard stop.
+autoindex_timeout_min = 30
+# Absolute backstop. ES default mapping is estimated at 6-9 MiB/s on a 4-vCPU
+# runner, so 3.0 sits at about 0.5x of its low end. The ES ratio below is the real floor.
+min_throughput_mb_s = 3.0
+max_node_peak_rss_mb = 3072
+max_client_peak_rss_mb = 1536
+max_state_to_source_ratio = 1.2
+# 6.4 GB, about 1.2x of 5 GB source.
+max_index_plus_state_mb = 6400
+max_indices_per_prefix = 40
+max_search_p95_ms = 1000
+max_search_max_ms = 2000
+max_breaker_trips = 0
+max_bulk_retries = 50
+max_restart_s = 120
+min_queries_with_hits_pct = 90
+max_regression_pct = 20
+
+# ---------------------------------------------------------------------------
+# Engine-relative floors (Ivan, 2026-10-25): index size and indexing time are
+# floored against Lucene/Elasticsearch on the SAME corpus and the SAME runner,
+# so a slower or smaller CI machine moves both sides together.
+#
+# Enabled when the gate gets `--engine-baseline <file.json>`, a JSON object
+# measured on the same corpus (seed + size) and machine:
+# { "engine": "elasticsearch-8.x", "corpus_mb": 1024, "seed": 2421,
+# "index_bytes": <store.size after force-merge or final refresh>,
+# "ingest_mb_s": <source MB / wall seconds to index + refresh>,
+# "search_p95_ms": <optional, same 60 queries> }
+# scripts/perf/es_baseline.py produces exactly that from a live ES (the
+# workflow runs it against an ES service container), or point it at a
+# results.json written by the floor-measurement harness.
+#
+# Each ratio is xerj / baseline. Values are per tier below; a key left out
+# (or the whole table) disables that check.
+# Measured floor (perf-floor results.md, 2026-10-25, 1 GB seed 2421, 4 cores):
+# ES 9.5.5 default dynamic mapping, 4 GB heap: 9.05 MiB/s, 1.77x source
+# refreshed, 1.42x source after force-merge to 1 segment.
+# Lucene 10.5.1 stored fields on: 25.4 MiB/s, 0.78x source.
+# xerj rc.99: 0.37-0.59 MiB/s, about 0.04-0.07x ES.
+# index_bytes of the baseline MUST be measured after a force-merge to one
+# segment (es_baseline.py does that). The pre-merge size swings with the
+# segment count.
+[ratio.default]
+# xerj_index_bytes (node data dir) <= k * es_index_bytes (force-merged).
+# 1.0: an index may not cost more disk than ES on the same data.
+max_index_bytes_vs_baseline = 1.0
+# xerj_ingest_MBps >= k * es_ingest_MBps (ES default mapping, to refreshed
+# and flushed). autoindex does discovery and inference that a bulk load does not,
+# so parity is not required. Stretch goal is 0.8.
+min_ingest_mb_s_vs_baseline = 0.5
+# xerj search p95 <= k * es search p95 (only if the baseline reports it).
+max_search_p95_vs_baseline = 1.5
+
+[ratio.smoke]
+# Fixed per-index overhead weighs more at 200 MB.
+max_index_bytes_vs_baseline = 1.25
+min_ingest_mb_s_vs_baseline = 0.3
+
+[ratio."1gb"]
+max_index_bytes_vs_baseline = 1.0
+min_ingest_mb_s_vs_baseline = 0.5
+max_search_p95_vs_baseline = 1.5
+
+[ratio."5gb"]
+max_index_bytes_vs_baseline = 1.0
+min_ingest_mb_s_vs_baseline = 0.5
+max_search_p95_vs_baseline = 1.5
diff --git a/scripts/perf/es_baseline.py b/scripts/perf/es_baseline.py
new file mode 100755
index 000000000..baef7320c
--- /dev/null
+++ b/scripts/perf/es_baseline.py
@@ -0,0 +1,136 @@
+#!/usr/bin/env python3
+"""Live Elasticsearch/Lucene floor for the autoindex release gate.
+
+ es_baseline.py --es http://127.0.0.1:9200 --corpus DIR --queries DIR.queries.json \
+ --out es-baseline.json [--workers 4] [--seed 2421]
+
+Bulk-loads the SAME corpus produced by gen_corpus.py into one ES index with
+default dynamic mappings (one doc per line for .jsonl/.csv/.log/.ndjson,
+one doc per file otherwise, ES default dynamic mapping), refreshes + flushes
+(ingest time stops here), force-merges to ONE segment (index_bytes is the
+merged size), then runs the corpus' search queries uncached. Writes the JSON that
+`gate.py eval --engine-baseline` consumes:
+
+ {"engine", "corpus_mb", "seed", "source_bytes", "docs", "index_bytes",
+ "ingest_mb_s", "search_p95_ms", "wall_s"}
+
+ingest_mb_s is source bytes / wall time from first bulk to refreshed+flushed
+index, the same definition gate.py uses for xerj (bytes / autoindex wall).
+stdlib only; the ES host is assumed throwaway (the index is deleted first).
+"""
+import argparse, json, os, queue, threading, time, urllib.request
+
+MiB = 1024 * 1024
+LINE_EXT = {".jsonl", ".ndjson", ".csv", ".log"}
+
+
+def http(method, url, body=None, ctype="application/json", timeout=300):
+ data = body if isinstance(body, (bytes, type(None))) else json.dumps(body).encode()
+ req = urllib.request.Request(url, data=data, method=method, headers={"Content-Type": ctype})
+ with urllib.request.urlopen(req, timeout=timeout) as r:
+ return json.loads(r.read() or b"{}")
+
+
+def docs(corpus):
+ for root, _, files in os.walk(corpus):
+ for f in sorted(files):
+ p = os.path.join(root, f)
+ rel = os.path.relpath(p, corpus)
+ ext = os.path.splitext(f)[1].lower()
+ try:
+ text = open(p, encoding="utf-8", errors="replace").read()
+ except OSError:
+ continue
+ if ext in LINE_EXT:
+ lines = text.splitlines()
+ header = lines[0] if ext == ".csv" and lines else None
+ for i, ln in enumerate(lines):
+ if ln.strip():
+ d = {"path": rel, "line": i, "body": ln}
+ if header and i:
+ d["header"] = header
+ yield d
+ else:
+ yield {"path": rel, "body": text}
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--es", default=os.environ.get("ES_URL", "http://127.0.0.1:9200"))
+ ap.add_argument("--corpus", required=True); ap.add_argument("--queries", required=True)
+ ap.add_argument("--out", required=True); ap.add_argument("--index", default="perf-floor")
+ ap.add_argument("--workers", type=int, default=4); ap.add_argument("--bulk-mb", type=float, default=5)
+ ap.add_argument("--seed", type=int, default=2421)
+ a = ap.parse_args()
+ info = http("GET", a.es + "/")
+ engine = "elasticsearch-%s (lucene %s)" % (info["version"]["number"], info["version"].get("lucene_version"))
+ try:
+ http("DELETE", "%s/%s" % (a.es, a.index))
+ except Exception:
+ pass
+ http("PUT", "%s/%s" % (a.es, a.index), {"settings": {"number_of_shards": 1, "number_of_replicas": 0, "refresh_interval": "-1"}}) # default dynamic mapping, as the floor was measured
+ src = sum(os.path.getsize(os.path.join(r, f)) for r, _, fs in os.walk(a.corpus) for f in fs)
+ q = queue.Queue(maxsize=a.workers * 2)
+ errs = []
+
+ def worker():
+ while True:
+ b = q.get()
+ if b is None:
+ return
+ last = None
+ for attempt in range(5):
+ try:
+ r = http("POST", "%s/%s/_bulk" % (a.es, a.index), b, "application/x-ndjson")
+ if r.get("errors"):
+ errs.append(next(i for i in r["items"] if i["index"].get("error"))["index"]["error"])
+ break
+ except Exception as e: # 429 / transient
+ last = e
+ time.sleep(2 ** attempt)
+ else:
+ errs.append(str(last))
+
+ th = [threading.Thread(target=worker, daemon=True) for _ in range(a.workers)]
+ [t.start() for t in th]
+ t0 = time.time(); buf = []; size = 0; n = 0
+ for d in docs(a.corpus):
+ line = b'{"index":{}}\n' + json.dumps(d).encode() + b"\n"
+ buf.append(line); size += len(line); n += 1
+ if size >= a.bulk_mb * MiB:
+ q.put(b"".join(buf)); buf = []; size = 0
+ if buf:
+ q.put(b"".join(buf))
+ for _ in th:
+ q.put(None)
+ [t.join() for t in th]
+ http("POST", "%s/%s/_refresh" % (a.es, a.index)); http("POST", "%s/%s/_flush" % (a.es, a.index))
+ wall = time.time() - t0
+ unmerged = http("GET", "%s/%s/_stats/store" % (a.es, a.index))["_all"]["primaries"]["store"]["size_in_bytes"]
+ # index_bytes is defined AFTER a force-merge to one segment: the refreshed
+ # size swings with the segment count (1.77x vs 1.42x source on 1 GB).
+ t1 = time.time()
+ http("POST", "%s/%s/_forcemerge?max_num_segments=1&flush=true" % (a.es, a.index), timeout=3600)
+ fm_s = time.time() - t1
+ st = http("GET", "%s/%s/_stats/store,docs" % (a.es, a.index))["_all"]["primaries"]
+ qs = json.load(open(a.queries))
+ allq = [x for k in ("cli", "raw", "mcp") for x in qs.get(k, [])]
+ lat = []
+ for s in allq:
+ t1 = time.time()
+ http("POST", "%s/%s/_search?request_cache=false" % (a.es, a.index),
+ {"size": 5, "query": {"simple_query_string": {"query": s, "fields": ["*"], "lenient": True}}})
+ lat.append((time.time() - t1) * 1000)
+ lat.sort()
+ p95 = lat[min(len(lat) - 1, int(round(0.95 * (len(lat) - 1))))] if lat else None
+ res = {"engine": engine, "corpus_mb": round(src / MiB), "seed": a.seed, "source_bytes": src,
+ "docs": st["docs"]["count"], "index_bytes": st["store"]["size_in_bytes"],
+ "index_bytes_refreshed_unmerged": unmerged, "forcemerge_s": round(fm_s, 1),
+ "ingest_mb_s": round(src / MiB / wall, 2), "search_p95_ms": round(p95, 1) if p95 else None,
+ "wall_s": round(wall, 1), "bulk_errors": len(errs), "first_error": errs[0] if errs else None}
+ json.dump(res, open(a.out, "w"), indent=1)
+ print(json.dumps(res))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/scripts/perf/gate.py b/scripts/perf/gate.py
new file mode 100644
index 000000000..d44aea36a
--- /dev/null
+++ b/scripts/perf/gate.py
@@ -0,0 +1,500 @@
+#!/usr/bin/env python3
+"""Measurement + evaluation core of the autoindex release gate.
+
+ gate.py run --bin X --corpus DIR --queries Q.json --work DIR [...] --out R.json
+ gate.py eval --budgets budgets.toml --tier 1gb --candidate R.json
+ [--baseline B.json] [--engine-baseline ES.json] --md OUT.md --json OUT.json
+
+`run` measures ONE binary end to end on a private port with a throwaway data
+dir and state dir; `eval` turns one or two result files into a verdict
+table. Only the Python 3.11+ stdlib is used. Never submits xerj feedback
+(XERJ_DISABLE_FEEDBACK=true everywhere).
+"""
+import argparse, json, os, re, signal, statistics, subprocess, sys, threading, time, tomllib, urllib.request
+
+MiB = 1024 * 1024
+FATAL_PATTERNS = {
+ "eagain": re.compile(r"os error 11|Resource temporarily unavailable|EAGAIN", re.I),
+ "oom": re.compile(r"out of memory|memory allocation of \d+ bytes failed|Cannot allocate memory|OOM", re.I),
+ "abort": re.compile(r"SIGABRT|panicked at|FATAL:", re.I),
+ "emfile": re.compile(r"Too many open files|os error 24", re.I),
+}
+BREAKER_RE = re.compile(r"engaging the parent memory circuit breaker", re.I)
+RETRY_RE = re.compile(r"\bretry(ing)?\b|backing off|\b429\b|too many requests", re.I)
+
+
+def log(*a):
+ print("[gate]", *a, file=sys.stderr, flush=True)
+
+
+def tree_bytes(path):
+ total = 0
+ for root, _, files in os.walk(path):
+ for f in files:
+ try:
+ total += os.lstat(os.path.join(root, f)).st_size
+ except OSError:
+ pass
+ return total
+
+
+def proc_status(pid):
+ out = {}
+ try:
+ with open("/proc/%d/status" % pid) as f:
+ for line in f:
+ k, _, v = line.partition(":")
+ if k in ("VmRSS", "VmHWM"):
+ out[k] = int(v.split()[0]) // 1024 # MB
+ elif k == "Threads":
+ out[k] = int(v)
+ except (OSError, ValueError):
+ pass
+ return out
+
+
+def children(pid):
+ try:
+ with open("/proc/%d/task/%d/children" % (pid, pid)) as f:
+ return [int(x) for x in f.read().split()]
+ except OSError:
+ return []
+
+
+def tree_rss(pid):
+ rss, stack = 0, [pid]
+ while stack:
+ p = stack.pop()
+ rss += proc_status(p).get("VmRSS", 0)
+ stack.extend(children(p))
+ return rss
+
+
+class Sampler(threading.Thread):
+ """Peak RSS / thread count of the node and the autoindex client tree."""
+
+ def __init__(self):
+ super().__init__(daemon=True)
+ self.node_pid = None
+ self.client_pid = None
+ self.node_peak = 0
+ self.client_peak = 0
+ self.client_threads_peak = 0
+ self.stop = False
+
+ def run(self):
+ while not self.stop:
+ if self.node_pid:
+ s = proc_status(self.node_pid)
+ self.node_peak = max(self.node_peak, s.get("VmHWM", 0), s.get("VmRSS", 0))
+ if self.client_pid:
+ s = proc_status(self.client_pid)
+ self.client_peak = max(self.client_peak, tree_rss(self.client_pid), s.get("VmHWM", 0))
+ self.client_threads_peak = max(self.client_threads_peak, s.get("Threads", 0))
+ time.sleep(0.25)
+
+
+def http(url, body=None, timeout=60):
+ data = json.dumps(body).encode() if body is not None else None
+ req = urllib.request.Request(url, data=data, headers={"Content-Type": "application/json"},
+ method="POST" if data else "GET")
+ with urllib.request.urlopen(req, timeout=timeout) as r:
+ return json.loads(r.read() or b"null")
+
+
+class Node:
+ def __init__(self, a, env, sampler):
+ self.a, self.env, self.sampler = a, env, sampler
+ self.url = "http://127.0.0.1:%d" % a.port
+ self.p = None
+ self.logs = []
+
+ def start(self, tag):
+ logp = os.path.join(self.a.work, "node-%s.log" % tag)
+ self.logs.append(logp)
+ t = time.time()
+ self.p = subprocess.Popen([self.a.bin, "--insecure", "--data-dir", os.path.join(self.a.work, "data"),
+ "--port", str(self.a.port), "--disable-feedback"],
+ stdout=open(logp, "w"), stderr=subprocess.STDOUT, env=self.env)
+ self.sampler.node_pid = self.p.pid
+ while time.time() - t < 300:
+ if self.p.poll() is not None:
+ raise RuntimeError("node exited during boot (rc=%s), see %s" % (self.p.returncode, logp))
+ try:
+ http(self.url + "/_cluster/health", timeout=2)
+ return time.time() - t
+ except Exception:
+ time.sleep(0.25)
+ raise RuntimeError("node never became healthy")
+
+ def alive(self):
+ return self.p is not None and self.p.poll() is None
+
+ def stop(self):
+ if not self.alive():
+ return
+ self.p.send_signal(signal.SIGINT)
+ try:
+ self.p.wait(120)
+ except subprocess.TimeoutExpired:
+ self.p.kill(); self.p.wait()
+
+
+def run_autoindex(a, env, node, sampler, res):
+ out = os.path.join(a.work, "autoindex.json")
+ err = os.path.join(a.work, "autoindex.stderr")
+ cmd = [a.bin, "autoindex", a.corpus, "--url", node.url, "--prefix", a.prefix,
+ "--state-dir", os.path.join(a.work, "state"), "--approve", "proceed",
+ "--json", "--progress", "plain", "--disable-feedback"] + a.autoindex_args
+ log("autoindex:", " ".join(cmd))
+ t = time.time()
+ p = subprocess.Popen(cmd, stdout=open(out, "w"), stderr=open(err, "w"), env=env)
+ sampler.client_pid = p.pid
+ timed_out = False
+ try:
+ p.wait(a.timeout_min * 60)
+ except subprocess.TimeoutExpired:
+ timed_out = True
+ p.send_signal(signal.SIGINT)
+ try:
+ p.wait(30)
+ except subprocess.TimeoutExpired:
+ p.kill(); p.wait()
+ wall = time.time() - t
+ sampler.client_pid = None
+ res["autoindex"] = {"exit_code": p.returncode, "wall_s": round(wall, 1), "timed_out": timed_out, "cmd": cmd}
+ try:
+ txt = open(out).read().strip()
+ j = json.loads(txt[txt.index("{"):]) if txt else {}
+ except Exception:
+ j = {}
+ keep = ["files_total", "files_indexed", "records_total", "docs_per_sec", "workers", "bulk_congestion_events",
+ "bulk_concurrency_final", "files_junk", "wall_seconds"]
+ res["autoindex"]["summary"] = {k: j.get(k) for k in keep if k in j}
+ stderr_txt = open(err, errors="replace").read()
+ res["autoindex"]["stderr_tail"] = stderr_txt[-2000:]
+ prog = re.findall(r"xerj-progress phase=index basis=bytes pct=[0-9.]+ items=\S+ bytes=(\d+)/(\d+)", stderr_txt)
+ if prog:
+ res["autoindex"]["progress_bytes_done"] = int(prog[-1][0])
+ return stderr_txt
+
+
+def q_cli(a, env, url, q):
+ t = time.time()
+ try:
+ r = subprocess.run([a.bin, "search", "--url", url, "--prefix", a.prefix, "-k", "5", "--json", q],
+ env=env, capture_output=True, text=True, timeout=a.query_timeout)
+ ms = (time.time() - t) * 1000
+ j = json.loads(r.stdout)
+ hits = len(j.get("hits", {}).get("hits", [])) if isinstance(j, dict) else 0
+ return ms, hits, None if r.returncode == 0 else "rc=%d" % r.returncode
+ except subprocess.TimeoutExpired:
+ return a.query_timeout * 1000.0, 0, "timeout"
+ except Exception as e:
+ return (time.time() - t) * 1000, 0, "parse: %s" % str(e)[:80]
+
+
+def q_raw(a, env, url, q):
+ body = {"size": 5, "query": {"simple_query_string": {"query": q, "fields": ["*"], "lenient": True}}}
+ t = time.time()
+ try:
+ j = http("%s/%s-*/_search?request_cache=false" % (url, a.prefix), body, timeout=a.query_timeout)
+ ms = (time.time() - t) * 1000
+ return ms, len(j.get("hits", {}).get("hits", [])), None
+ except Exception as e:
+ return (time.time() - t) * 1000, 0, str(e)[:80]
+
+
+class MCP:
+ def __init__(self, a, env, url):
+ self.a = a
+ self.p = subprocess.Popen([a.bin, "mcp", "--url", url, "--disable-feedback"], stdin=subprocess.PIPE,
+ stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, env=env, bufsize=1)
+ self.i = 0
+ self.call("initialize", {"protocolVersion": "2024-11-05", "capabilities": {},
+ "clientInfo": {"name": "perf-gate", "version": "1"}})
+ self.p.stdin.write(json.dumps({"jsonrpc": "2.0", "method": "notifications/initialized"}) + "\n")
+ self.p.stdin.flush()
+
+ def call(self, m, params):
+ self.i += 1
+ self.p.stdin.write(json.dumps({"jsonrpc": "2.0", "id": self.i, "method": m, "params": params}) + "\n")
+ self.p.stdin.flush()
+ while True:
+ line = self.p.stdout.readline()
+ if not line:
+ raise RuntimeError("mcp eof")
+ try:
+ j = json.loads(line)
+ except ValueError:
+ continue
+ if j.get("id") == self.i:
+ return j
+
+ def query(self, q):
+ t = time.time()
+ try:
+ j = self.call("tools/call", {"name": "xerj_search", "arguments": {"prefix": self.a.prefix, "query": q, "size": 5}})
+ except Exception as e:
+ return (time.time() - t) * 1000, 0, str(e)[:80]
+ ms = (time.time() - t) * 1000
+ r = j.get("result", {})
+ txt = "".join(c.get("text", "") for c in r.get("content", []))
+ hits = 0
+ try:
+ jj = json.loads(txt)
+ hits = len(jj.get("hits", {}).get("hits", jj.get("results", []))) if isinstance(jj, dict) else len(jj)
+ except Exception:
+ hits = 1 if txt.strip() and "no results" not in txt.lower() else 0
+ if r.get("isError") or "error" in j:
+ msg = txt or json.dumps(j.get("error"))
+ return ms, 0, ("mcp error: " + msg.replace("\n", " "))[:200]
+ return ms, hits, None
+
+ def close(self):
+ try:
+ self.p.stdin.close(); self.p.wait(10)
+ except Exception:
+ self.p.kill()
+
+
+def pct(v, p):
+ v = sorted(v)
+ if not v:
+ return None
+ k = (len(v) - 1) * p / 100.0
+ lo = int(k); hi = min(lo + 1, len(v) - 1)
+ return v[lo] + (v[hi] - v[lo]) * (k - lo)
+
+
+def cmd_run(a):
+ os.makedirs(a.work, exist_ok=True)
+ env = dict(os.environ, XERJ_DISABLE_FEEDBACK="true", XERJ_MAX_PROCESS_MEMORY_MB=str(a.cap_mb),
+ XERJ_URL="http://127.0.0.1:%d" % a.port)
+ env.pop("XERJ_API_KEY", None)
+ queries = json.load(open(a.queries))
+ res = {"label": a.label, "bin": a.bin, "corpus": a.corpus, "cap_mb": a.cap_mb, "started": time.strftime("%Y-%m-%dT%H:%M:%S%z")}
+ try:
+ res["version"] = subprocess.run([a.bin, "--version"], capture_output=True, text=True, env=env, timeout=30).stdout.strip()
+ except Exception:
+ res["version"] = "?"
+ res["source_bytes"] = tree_bytes(a.corpus)
+ sampler = Sampler(); sampler.start()
+ node = Node(a, env, sampler)
+ fatal = {}
+ try:
+ res["boot_s"] = round(node.start("ingest"), 2)
+ stderr_txt = run_autoindex(a, env, node, sampler, res)
+ res["node_alive_after_autoindex"] = node.alive()
+ try:
+ cat = http("%s/_cat/indices/%s-*?format=json&bytes=b&h=index,docs.count,store.size" % (node.url, a.prefix))
+ except Exception:
+ cat = []
+ res["indices"] = len(cat)
+ res["docs"] = sum(int(c.get("docs.count") or 0) for c in cat)
+ res["index_store_bytes"] = sum(int(c.get("store.size") or 0) for c in cat)
+ res["node_peak_rss_mb_ingest"] = sampler.node_peak
+ node.stop()
+ node_txt = "".join(open(p, errors="replace").read() for p in node.logs)
+ res["breaker_trips"] = len(BREAKER_RE.findall(node_txt))
+ res["bulk_retries"] = int(res["autoindex"]["summary"].get("bulk_congestion_events") or 0) + len(RETRY_RE.findall(stderr_txt))
+ for k, rx in FATAL_PATTERNS.items():
+ n = len(rx.findall(stderr_txt)) + len(rx.findall(node_txt))
+ if n:
+ fatal[k] = n
+ res["fatal"] = fatal
+ res["state_bytes"] = tree_bytes(os.path.join(a.work, "state"))
+ res["data_dir_bytes"] = tree_bytes(os.path.join(a.work, "data"))
+ res["client_peak_rss_mb"] = sampler.client_peak
+ res["client_threads_peak"] = sampler.client_threads_peak
+ if a.drop_caches:
+ subprocess.run("sync; echo 3 | sudo -n tee /proc/sys/vm/drop_caches >/dev/null", shell=True)
+ # Restart so every query below is served by a cold process (no
+ # request/query/segment caches), the way an agent's first call is.
+ res["restart_s"] = round(node.start("search"), 2)
+ lat = {}
+ for surface in ("cli", "raw", "mcp"):
+ rows = []
+ m = MCP(a, env, node.url) if surface == "mcp" else None
+ for q in queries[surface]:
+ if surface == "cli":
+ ms, hits, e = q_cli(a, env, node.url, q)
+ elif surface == "raw":
+ ms, hits, e = q_raw(a, env, node.url, q)
+ else:
+ ms, hits, e = m.query(q)
+ rows.append({"q": q, "ms": round(ms, 1), "hits": hits, "err": e})
+ if m:
+ m.close()
+ lat[surface] = rows
+ ms = [r["ms"] for r in rows]
+ log("%s p50=%.0f p95=%.0f max=%.0f ms" % (surface, pct(ms, 50), pct(ms, 95), max(ms)))
+ res["queries"] = lat
+ allms = [r["ms"] for rows in lat.values() for r in rows]
+ res["search"] = {"n": len(allms), "p50_ms": round(pct(allms, 50), 1), "p95_ms": round(pct(allms, 95), 1),
+ "max_ms": round(max(allms), 1),
+ "with_hits_pct": round(100.0 * sum(1 for rows in lat.values() for r in rows if r["hits"] and not r["err"]) / len(allms), 1),
+ "errors": sum(1 for rows in lat.values() for r in rows if r["err"]),
+ "first_error": next(("%s: %s" % (k, r["err"]) for k, rows in lat.items() for r in rows if r["err"]), None)}
+ for s, rows in lat.items():
+ ms = [r["ms"] for r in rows]
+ res["search"][s] = {"p50_ms": round(pct(ms, 50), 1), "p95_ms": round(pct(ms, 95), 1), "max_ms": round(max(ms), 1)}
+ except Exception as e:
+ res["error"] = str(e)
+ log("ERROR", e)
+ finally:
+ res["node_peak_rss_mb"] = sampler.node_peak
+ node.stop()
+ sampler.stop = True
+ ai = res.get("autoindex", {})
+ if ai.get("wall_s"):
+ done = res["source_bytes"]
+ if ai.get("exit_code") not in (0, 3) or ai.get("timed_out"):
+ # incomplete run: count only bytes autoindex reported as indexed
+ done = min(done, ai.get("progress_bytes_done") or 0)
+ res["indexed_bytes"] = done
+ res["throughput_mb_s"] = round(done / MiB / ai["wall_s"], 2)
+ res["state_to_source_ratio"] = round(res.get("state_bytes", 0) / max(1, res["source_bytes"]), 3)
+ res["index_bytes"] = res.get("data_dir_bytes", 0)
+ json.dump(res, open(a.out, "w"), indent=1)
+ log("wrote", a.out)
+
+
+# ---------------------------------------------------------------- eval ----
+def cmd_eval(a):
+ B = tomllib.load(open(a.budgets, "rb"))
+ t = B["tier"][a.tier]
+ ratio = B.get("ratio", {}).get(a.tier, B.get("ratio", {}).get("default", {}))
+ c = json.load(open(a.candidate))
+ base = json.load(open(a.baseline)) if a.baseline else None
+ eng = json.load(open(a.engine_baseline)) if a.engine_baseline else None
+ rows = [] # (check, value, budget, ok, note)
+
+ def add(name, val, budget, ok, note=""):
+ rows.append({"check": name, "value": val, "budget": budget, "pass": bool(ok), "note": note})
+
+ ai = c.get("autoindex", {})
+ fatal = c.get("fatal", {})
+ completed = (not c.get("error") and not ai.get("timed_out") and ai.get("exit_code") in (0, 3)
+ and c.get("node_alive_after_autoindex", False) and not fatal)
+ note = []
+ if ai.get("timed_out"): note.append("timed out after %s s" % ai.get("wall_s"))
+ if ai.get("exit_code") not in (0, 3): note.append("exit %s" % ai.get("exit_code"))
+ if not c.get("node_alive_after_autoindex", False): note.append("node died")
+ if fatal: note.append(", ".join("%s x%d" % kv for kv in fatal.items()))
+ if c.get("error"): note.append(c["error"][:80])
+ add("autoindex completes (no OOM/abort/EAGAIN/timeout)", "yes" if completed else "NO", "exit 0/3, node alive", completed, "; ".join(note))
+ tp = c.get("throughput_mb_s")
+ add("autoindex throughput (MB/s)", tp, ">= %s" % t["min_throughput_mb_s"], completed and tp is not None and tp >= t["min_throughput_mb_s"],
+ "%.0f of %.0f MB indexed in %s s" % (c.get("indexed_bytes", c.get("source_bytes", 0)) / MiB, c.get("source_bytes", 0) / MiB, ai.get("wall_s")))
+ add("node peak RSS (MB), cap %d" % t["node_memory_cap_mb"], c.get("node_peak_rss_mb"), "<= %d" % t["max_node_peak_rss_mb"],
+ (c.get("node_peak_rss_mb") or 0) <= t["max_node_peak_rss_mb"],
+ "XERJ_MAX_PROCESS_MEMORY_MB=%s" % c.get("cap_mb"))
+ add("client peak RSS (MB)", c.get("client_peak_rss_mb"), "<= %d" % t["max_client_peak_rss_mb"],
+ (c.get("client_peak_rss_mb") or 0) <= t["max_client_peak_rss_mb"], "peak threads %s" % c.get("client_threads_peak"))
+ sr = c.get("state_to_source_ratio", 0)
+ add("state dir / source", sr, "<= %s" % t["max_state_to_source_ratio"], sr <= t["max_state_to_source_ratio"],
+ "%.0f MB state" % (c.get("state_bytes", 0) / MiB))
+ if "max_index_plus_state_mb" in t:
+ tot = (c.get("index_bytes") or 0) + (c.get("state_bytes") or 0)
+ add("index + state on disk (MB)", round(tot / MiB), "<= %d" % t["max_index_plus_state_mb"],
+ tot / MiB <= t["max_index_plus_state_mb"], "%.2fx source" % (tot / max(1, c.get("source_bytes", 1))))
+ add("indices per prefix", c.get("indices"), "<= %d" % t["max_indices_per_prefix"],
+ 0 < (c.get("indices") or 0) <= t["max_indices_per_prefix"], "%s docs" % c.get("docs"))
+ s = c.get("search", {})
+ add("uncached search p95 (ms)", s.get("p95_ms"), "<= %d" % t["max_search_p95_ms"],
+ s.get("p95_ms") is not None and s["p95_ms"] <= t["max_search_p95_ms"],
+ " / ".join("%s %s" % (k, s[k]["p95_ms"]) for k in ("cli", "raw", "mcp") if k in s))
+ add("uncached search max (ms)", s.get("max_ms"), "<= %d" % t["max_search_max_ms"],
+ s.get("max_ms") is not None and s["max_ms"] <= t["max_search_max_ms"], "n=%s, errors=%s" % (s.get("n"), s.get("errors")))
+ add("queries with hits (%)", s.get("with_hits_pct"), ">= %d" % t["min_queries_with_hits_pct"],
+ (s.get("with_hits_pct") or 0) >= t["min_queries_with_hits_pct"])
+ add("search errors", s.get("errors"), "0", s.get("errors") == 0, (s.get("first_error") or "")[:120])
+ add("circuit-breaker trips", c.get("breaker_trips"), "<= %d" % t["max_breaker_trips"],
+ c.get("breaker_trips") is not None and c["breaker_trips"] <= t["max_breaker_trips"])
+ add("bulk retries / 429s", c.get("bulk_retries"), "<= %d" % t["max_bulk_retries"],
+ c.get("bulk_retries") is not None and c["bulk_retries"] <= t["max_bulk_retries"])
+ add("node restart (s)", c.get("restart_s"), "<= %d" % t["max_restart_s"],
+ c.get("restart_s") is not None and c["restart_s"] <= t["max_restart_s"])
+
+ lim = t["max_regression_pct"] / 100.0
+ if base:
+ def reg(name, key_fn, higher_better):
+ cv, bv = key_fn(c), key_fn(base)
+ if cv is None or bv is None or bv == 0:
+ add("vs baseline: " + name, cv, "baseline %s" % bv, True, "skipped (missing)")
+ return
+ delta = (cv - bv) / bv
+ worse = -delta if higher_better else delta
+ add("vs baseline: " + name, cv, "%s %s (<= %d%% worse)" % ("baseline", bv, t["max_regression_pct"]),
+ worse <= lim, "%+.0f%%" % (delta * 100))
+ reg("throughput MB/s", lambda r: r.get("throughput_mb_s"), True)
+ reg("search p95 ms", lambda r: r.get("search", {}).get("p95_ms"), False)
+ reg("node peak RSS MB", lambda r: r.get("node_peak_rss_mb"), False)
+ reg("client peak RSS MB", lambda r: r.get("client_peak_rss_mb"), False)
+ reg("state/source", lambda r: r.get("state_to_source_ratio"), False)
+ reg("index bytes", lambda r: r.get("index_bytes"), False)
+ if eng and ratio:
+ en = eng.get("engine", "baseline engine")
+ eb = eng.get("source_bytes") or (eng.get("corpus_mb", 0) * MiB)
+ if eb and abs(eb - c.get("source_bytes", 0)) > 0.05 * eb:
+ add("engine baseline is for this corpus", "%.0f MB" % (eb / MiB), "%.0f MB" % (c.get("source_bytes", 0) / MiB), False,
+ "re-measure the baseline on the same corpus")
+ if "max_index_bytes_vs_baseline" in ratio and eng.get("index_bytes"):
+ r_ = c.get("index_bytes", 0) / eng["index_bytes"]
+ add("index bytes vs %s" % en, round(r_, 2), "<= %sx" % ratio["max_index_bytes_vs_baseline"],
+ r_ <= ratio["max_index_bytes_vs_baseline"], "%.0f MB vs %.0f MB" % (c.get("index_bytes", 0) / MiB, eng["index_bytes"] / MiB))
+ if "min_ingest_mb_s_vs_baseline" in ratio and eng.get("ingest_mb_s"):
+ r_ = (c.get("throughput_mb_s") or 0) / eng["ingest_mb_s"]
+ add("ingest MB/s vs %s" % en, round(r_, 2), ">= %sx" % ratio["min_ingest_mb_s_vs_baseline"],
+ r_ >= ratio["min_ingest_mb_s_vs_baseline"], "%s vs %s MB/s" % (c.get("throughput_mb_s"), eng["ingest_mb_s"]))
+ if "max_search_p95_vs_baseline" in ratio and eng.get("search_p95_ms") and s.get("p95_ms"):
+ r_ = s["p95_ms"] / eng["search_p95_ms"]
+ add("search p95 vs %s" % en, round(r_, 2), "<= %sx" % ratio["max_search_p95_vs_baseline"],
+ r_ <= ratio["max_search_p95_vs_baseline"], "%s vs %s ms" % (s["p95_ms"], eng["search_p95_ms"]))
+
+ ok = all(r["pass"] for r in rows)
+ md = ["### autoindex release gate: %s (%s, tier %s)" % ("PASS" if ok else "FAIL", c.get("version"), a.tier), "",
+ "| check | value | budget | result | note |", "|---|---|---|---|---|"]
+ for r in rows:
+ md.append("| %s | %s | %s | %s | %s |" % (r["check"], r["value"], r["budget"], "pass" if r["pass"] else "**FAIL**", r["note"].replace("|", "/")))
+ md.append("")
+ md.append("corpus %.0f MB, %s; %d indices, %s docs, index %.0f MB on disk" % (
+ c.get("source_bytes", 0) / MiB, c.get("corpus"), c.get("indices") or 0, c.get("docs"), c.get("index_bytes", 0) / MiB))
+ text = "\n".join(md)
+ print(text)
+ if a.md:
+ open(a.md, "w").write(text + "\n")
+ if a.json:
+ json.dump({"pass": ok, "tier": a.tier, "budgets": t, "ratio_budgets": ratio if eng else None, "checks": rows,
+ "candidate": c, "baseline": base, "engine_baseline": eng}, open(a.json, "w"), indent=1)
+ sys.exit(0 if ok else 1)
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ sp = ap.add_subparsers(dest="cmd", required=True)
+ r = sp.add_parser("run")
+ r.add_argument("--bin", required=True); r.add_argument("--corpus", required=True)
+ r.add_argument("--queries", required=True); r.add_argument("--work", required=True)
+ r.add_argument("--out", required=True); r.add_argument("--label", default="candidate")
+ r.add_argument("--port", type=int, default=9400); r.add_argument("--prefix", default="pg")
+ r.add_argument("--cap-mb", type=int, default=3072); r.add_argument("--timeout-min", type=float, default=15)
+ r.add_argument("--query-timeout", type=float, default=60)
+ r.add_argument("--drop-caches", action="store_true")
+ r.add_argument("--autoindex-args", default="", help="extra args, space separated")
+ e = sp.add_parser("eval")
+ e.add_argument("--budgets", required=True); e.add_argument("--tier", default="1gb")
+ e.add_argument("--candidate", required=True); e.add_argument("--baseline")
+ e.add_argument("--engine-baseline"); e.add_argument("--md"); e.add_argument("--json")
+ a = ap.parse_args()
+ if a.cmd == "run":
+ a.bin = os.path.abspath(a.bin); a.corpus = os.path.abspath(a.corpus); a.work = os.path.abspath(a.work)
+ a.autoindex_args = a.autoindex_args.split()
+ cmd_run(a)
+ else:
+ cmd_eval(a)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/scripts/perf/gen_corpus.py b/scripts/perf/gen_corpus.py
new file mode 100755
index 000000000..5064d0161
--- /dev/null
+++ b/scripts/perf/gen_corpus.py
@@ -0,0 +1,299 @@
+#!/usr/bin/env python3
+"""Deterministic synthetic corpus for the autoindex release gate.
+
+Same --seed + --size-mb => byte-identical tree on every machine, so a gate
+number from CI and one from a laptop are about the same input. The mix is
+modelled on the real 5 GB folder behind #2421-#2428 (Rust/Go dependency
+sources, docs, HTML, JSONL audit data, logs, ADS-B flight traces):
+
+ share family why it is here
+ ----- ----------------------------- ------------------------------------
+ 24% code (.rs .go .py .ts) the bulk of real repos; docs index
+ 12% markdown docs index, headings, links
+ 8% HTML pages with <table>s #2422: one index per page table
+ 9% CSV (a few big + many small) tabular datasets; small distinct
+ schemas tempt one-index-per-table
+ 12% JSONL records record data
+ 10% logs line records
+ 15% big JSON, numeric arrays flight traces: few huge files
+ 5% multi-array JSON + YAML #2422: grouped arrays skipped caps
+ 5% tiny files (< 1 KiB) per-file overhead, many inodes
+ deep/long paths #2424: 329-char index names
+
+It also writes <out>/../queries.json (or --queries): 60 distinct queries, 20
+per search surface, each built from planted phrases so every one has hits.
+Only the stdlib is used.
+"""
+import argparse, json, os, random, sys
+
+WORDS = ("alpha bravo charlie delta echo foxtrot golf hotel india juliet kilo lima mike "
+ "november oscar papa quebec romeo sierra tango uniform victor whiskey xray yankee zulu "
+ "buffer cache segment shard replica merge flush commit journal snapshot cursor token "
+ "parser lexer schema mapping field record batch stream socket packet header payload "
+ "retry backoff timeout deadline latency throughput budget quota limit window bucket "
+ "kernel driver module plugin handler router service client server worker thread pool "
+ "vector matrix tensor gradient kernel filter reducer mapper sorter index search query").split()
+
+# Planted topics: every query is drawn from these; each phrase is written into
+# several families so all three surfaces can find it.
+TOPICS = [
+ "fsync wal segment rotation", "tokio spawn blocking task", "http client retry exponential backoff",
+ "parse toml config file", "tls certificate verification error", "use after free netfilter",
+ "disk budget flood stage watermark", "json schema validation failure", "aircraft altitude ground speed",
+ "goroutine leak context cancel", "memtable flush threshold", "circuit breaker memory pressure",
+ "bloom filter false positive rate", "raft leader election timeout", "btree page split",
+ "gzip decompression bomb", "oauth refresh token rotation", "dns resolver cache poisoning",
+ "postgres vacuum autovacuum", "kafka consumer lag offset", "rate limiter token bucket",
+ "mmap page fault storm", "unicode normalization nfc", "csv quoting escape delimiter",
+ "sql injection prepared statement", "cron schedule daylight saving", "s3 multipart upload abort",
+ "grpc deadline exceeded status", "websocket ping pong keepalive", "lru eviction policy",
+ "simd utf8 validation", "heap profile allocation hotspot", "deadlock mutex ordering",
+ "zstd dictionary compression level", "hnsw graph recall ef", "bm25 term frequency saturation",
+ "semver caret range resolution", "crate feature unification", "go module proxy checksum",
+ "flight squawk transponder code", "runway heading crosswind", "audit log retention policy",
+ "invoice line item total", "sensor temperature humidity reading", "inventory warehouse sku",
+ "payment refund chargeback", "kubernetes pod eviction", "terraform state lock",
+ "docker layer cache miss", "prometheus histogram bucket", "opentelemetry span context",
+ "redis pipeline transaction", "elastic shard allocation", "lucene segment merge policy",
+ "regex catastrophic backtracking", "integer overflow checked arithmetic", "async cancellation safety",
+ "file descriptor exhaustion", "thread spawn resource unavailable", "index name too long",
+]
+
+class Gen:
+ def __init__(self, out, size_mb, seed):
+ self.out = out
+ self.target = int(size_mb * 1024 * 1024)
+ self.r = random.Random(seed)
+ self.written = 0
+ self.files = 0
+ self.by_family = {}
+ # Pre-built pools make generation I/O-bound, not RNG-bound.
+ r = self.r
+ self.sentences = [" ".join(r.choice(WORDS) for _ in range(r.randint(6, 18))).capitalize() + "."
+ for _ in range(4000)]
+ self.idents = ["_".join(r.choice(WORDS) for _ in range(r.randint(1, 3))) for _ in range(3000)]
+
+ # ---------- helpers ----------
+ def topic(self):
+ return self.r.choice(TOPICS)
+
+ def para(self, n):
+ s = self.r.choices(self.sentences, k=n)
+ if self.r.random() < 0.5:
+ s.insert(self.r.randrange(len(s) + 1), "Notes on %s." % self.topic())
+ return " ".join(s)
+
+ def write(self, fam, rel, data):
+ p = os.path.join(self.out, rel)
+ os.makedirs(os.path.dirname(p), exist_ok=True)
+ b = data.encode() if isinstance(data, str) else data
+ with open(p, "wb") as f:
+ f.write(b)
+ self.written += len(b)
+ self.files += 1
+ st = self.by_family.setdefault(fam, [0, 0])
+ st[0] += 1
+ st[1] += len(b)
+ return len(b)
+
+ # ---------- families ----------
+ def code_file(self, i, nbytes=8192):
+ r = self.r
+ lang = r.choice(["rs", "go", "py", "ts"])
+ repo = "repo-%02d" % (i % 24)
+ mod = "/".join(r.choice(self.idents) for _ in range(r.randint(1, 4)))
+ lines = []; size = 0; f = 0
+ while size < nbytes:
+ f += 1
+ name = r.choice(self.idents) + "_%d" % f
+ t = self.topic()
+ if lang == "rs":
+ l = "/// %s\npub fn %s(input: &[u8], limit: usize) -> Result<usize, Error> {\n // %s\n let mut n = 0usize;\n for b in input.iter().take(limit) { n = n.wrapping_add(*b as usize); }\n Ok(n)\n}\n" % (r.choice(self.sentences), name, t)
+ elif lang == "go":
+ l = "// %s handles %s.\nfunc %s(ctx context.Context, in []byte) (int, error) {\n\tif ctx.Err() != nil { return 0, ctx.Err() }\n\treturn len(in), nil\n}\n" % (name, t, name)
+ elif lang == "py":
+ l = 'def %s(data, limit=%d):\n """%s -- %s"""\n return sum(data[:limit])\n' % (name, r.randint(1, 999), r.choice(self.sentences), t)
+ else:
+ l = "// %s\nexport function %s(xs: number[]): number {\n return xs.reduce((a, b) => a + b, %d);\n}\n" % (t, name, r.randint(0, 9))
+ lines.append(l); size += len(l) + 1
+ self.write("code", "%s/src/%s/mod_%d.%s" % (repo, mod, i, lang), "\n".join(lines))
+
+ def markdown(self, i, nbytes=4096):
+ r = self.r
+ body = ["# %s\n" % self.topic().title()]; size = 0
+ while size < nbytes:
+ l = "## %s\n\n%s\n" % (r.choice(self.idents).replace("_", " ").title(), self.para(r.randint(3, 10)))
+ if r.random() < 0.3:
+ l += "```\n%s\n```\n" % r.choice(self.sentences)
+ body.append(l); size += len(l)
+ self.write("markdown", "docs/%s/%s/note-%d.md" % (r.choice(WORDS), r.choice(WORDS), i), "\n".join(body))
+
+ def html(self, i, nbytes=4096):
+ r = self.r
+ # A distinct column set per page: the #2422 trigger (202 of 454 b5
+ # indices were HTML page tables, 197 of them under 16 rows).
+ cols = r.sample(WORDS, r.randint(3, 6))
+ rows = r.randint(3, 40)
+ trs = "".join("<tr>%s</tr>" % "".join("<td>%s</td>" % (r.randint(0, 99999) if c in WORDS[:20] else r.choice(WORDS)) for c in cols) for _ in range(rows))
+ paras = []; size = len(trs)
+ while size < nbytes:
+ p = "<h2>%s</h2><p>%s</p>" % (self.topic(), self.para(8)); paras.append(p); size += len(p)
+ doc = ("<!doctype html><html><head><title>%s</title></head><body><h1>%s</h1><p>%s</p>"
+ "<table><thead><tr>%s</tr></thead><tbody>%s</tbody></table>%s</body></html>") % (
+ self.topic(), self.topic(), self.para(6), "".join("<th>%s_%d</th>" % (c, i % 97) for c in cols), trs, "".join(paras))
+ self.write("html", "site/%s/page-%d.html" % (r.choice(WORDS), i), doc)
+
+ def csv_small(self, i):
+ r = self.r
+ cols = ["%s_%d" % (c, i) for c in r.sample(WORDS, r.randint(3, 6))]
+ lines = [",".join(cols)]
+ for k in range(r.randint(4, 30)):
+ lines.append(",".join(str(r.randint(0, 9999)) if j % 2 else r.choice(WORDS) for j in range(len(cols))))
+ self.write("csv", "tables/t%03d/%s.csv" % (i % 400, r.choice(WORDS)), "\n".join(lines) + "\n")
+
+ def csv_big(self, i, nbytes):
+ r = self.r
+ out = ["ts,sensor_id,site,temperature,humidity,status,note"]
+ n = len(out[0])
+ while n < nbytes:
+ l = "%d,%d,%s,%.2f,%.1f,%s,%s" % (1700000000 + r.randint(0, 10**7), r.randint(1, 5000), r.choice(WORDS),
+ r.uniform(-20, 45), r.uniform(0, 100), r.choice(["ok", "warn", "fail"]),
+ r.choice(TOPICS))
+ out.append(l); n += len(l) + 1
+ self.write("csv", "warehouse/sensors/readings-%d.csv" % i, "\n".join(out) + "\n")
+
+ def jsonl(self, i, nbytes):
+ r = self.r
+ out = []; n = 0
+ while n < nbytes:
+ rec = {"id": r.randint(1, 10**9), "ts": 1700000000 + r.randint(0, 10**7), "actor": r.choice(self.idents),
+ "action": r.choice(["create", "update", "delete", "login", "export"]), "topic": r.choice(TOPICS),
+ "message": r.choice(self.sentences), "amount": round(r.uniform(0, 5000), 2)}
+ l = json.dumps(rec, separators=(",", ":")); out.append(l); n += len(l) + 1
+ self.write("jsonl", "audit/%d/events-%d.jsonl" % (i % 12, i), "\n".join(out) + "\n")
+
+ def log(self, i, nbytes):
+ r = self.r
+ out = []; n = 0
+ lv = ["INFO", "WARN", "ERROR", "DEBUG"]
+ while n < nbytes:
+ l = "2026-10-%02dT%02d:%02d:%02d.%03dZ %s %s: %s (%s)" % (r.randint(1, 28), r.randint(0, 23), r.randint(0, 59), r.randint(0, 59),
+ r.randint(0, 999), r.choice(lv), r.choice(self.idents), r.choice(self.sentences), r.choice(TOPICS))
+ out.append(l); n += len(l) + 1
+ self.write("logs", "var/log/svc-%d/app-%d.log" % (i % 8, i), "\n".join(out) + "\n")
+
+ def flight_json(self, i, nbytes):
+ # ADS-B style: one object with huge parallel numeric arrays.
+ r = self.r
+ n = max(1000, nbytes // 60)
+ t0 = 1729000000
+ obj = {"icao": "%06x" % r.randint(0, 0xFFFFFF), "callsign": "XRJ%04d" % i, "note": r.choice(TOPICS),
+ "trace": [[t0 + k, round(r.uniform(-90, 90), 5), round(r.uniform(-180, 180), 5), r.randint(0, 45000),
+ round(r.uniform(0, 600), 1), r.randint(0, 359)] for k in range(n)]}
+ self.write("flight_json", "adsb/traces/%02d/trace_full_%06x.json" % (i % 16, i), json.dumps(obj, separators=(",", ":")))
+
+ def multi_array_json(self, i):
+ r = self.r
+ obj = {}
+ for a in range(r.randint(2, 6)):
+ keys = r.sample(WORDS, 3)
+ obj["%s_list" % r.choice(WORDS)] = [{k: r.randint(0, 999) if j else r.choice(WORDS) for j, k in enumerate(keys)} for _ in range(r.randint(2, 20))]
+ if r.random() < 0.3:
+ body = "\n".join("%s: %s" % (r.choice(self.idents), r.choice(WORDS)) for _ in range(r.randint(5, 40)))
+ self.write("multi_json", "config/%s/settings-%d.yaml" % (r.choice(WORDS), i), body + "\n")
+ else:
+ self.write("multi_json", "grammars/%s/spec-%d.json" % (r.choice(WORDS), i), json.dumps(obj, indent=1))
+
+ def tiny(self, i):
+ r = self.r
+ self.write("tiny", "small/%02d/%02d/f%d.txt" % (i % 50, (i // 50) % 50, i), r.choice(self.sentences) + " " + self.topic() + "\n")
+
+ def deep(self, i):
+ # Go-module-like deep paths: the slug that produced a 329-char index name (#2424).
+ r = self.r
+ segs = ["golang.org", "x", "github.com-%s-%s" % (r.choice(WORDS), r.choice(WORDS))]
+ segs += ["%s-%s-%s-%s-v%d" % (r.choice(WORDS), r.choice(WORDS), r.choice(WORDS), r.choice(WORDS), r.randint(1, 9)) for _ in range(r.randint(8, 14))]
+ p = "gomod/" + "/".join(segs)
+ cols = ["%s_%d" % (c, i) for c in r.sample(WORDS, 3)]
+ self.write("deep_paths", p + "/data-%d.csv" % i, ",".join(cols) + "\n" + "\n".join("%d,%s,%d" % (k, r.choice(WORDS), r.randint(0, 99)) for k in range(5)) + "\n")
+ self.write("deep_paths", p + "/README-%d.md" % i, "# %s\n\n%s\n" % (self.topic(), self.para(4)))
+
+ def run(self):
+ T = self.target
+ gb = T / float(1 << 30)
+ shares = [("code", .24), ("markdown", .12), ("html", .08), ("csv", .09), ("jsonl", .12), ("logs", .10),
+ ("flight_json", .15), ("multi_json", .05), ("tiny", .05)]
+ budget = {k: T * v for k, v in shares}
+ # File COUNTS per GB, tuned to the real folder's ~30k files/GB: the
+ # count drives per-file and per-dataset overhead, the byte share
+ # drives ingest volume. Sizes grow with --size-mb; counts scale too
+ # but never drop below a floor, so a 50 MB smoke run keeps the shape.
+ def cnt(per_gb, floor):
+ return max(floor, int(per_gb * gb))
+ for i in range(cnt(120, 20)):
+ self.deep(i)
+ # Big-file families: a fixed number of large files whose size scales.
+ for i in range(4):
+ self.flight_json(i, int(budget["flight_json"] / 4))
+ for i in range(24):
+ self.jsonl(i, int(budget["jsonl"] / 24))
+ for i in range(16):
+ self.log(i, int(budget["logs"] / 16))
+ for i in range(3):
+ self.csv_big(i, int(budget["csv"] * 0.8 / 3))
+ n_small_csv = cnt(800, 60)
+ for i in range(n_small_csv):
+ self.csv_small(i)
+ for i in range(cnt(1500, 80)):
+ self.multi_array_json(i)
+ for i in range(cnt(8000, 400)):
+ self.tiny(i)
+ n_html = cnt(3000, 150)
+ for i in range(n_html):
+ self.html(i, int(budget["html"] / n_html))
+ n_md = cnt(6000, 300)
+ for i in range(n_md):
+ self.markdown(i, int(budget["markdown"] / n_md))
+ n_code = cnt(16000, 800)
+ for i in range(n_code):
+ self.code_file(i, int(budget["code"] / n_code))
+ # Top up with more code files so the total lands on --size-mb.
+ i = n_code
+ while self.written < T:
+ self.code_file(i, int(budget["code"] / n_code)); i += 1
+ return self
+
+
+def queries(seed):
+ r = random.Random(seed + 1)
+ t = TOPICS[:]
+ r.shuffle(t)
+ # 60 distinct strings, 20 per surface; no string repeats across surfaces,
+ # so a cache warmed by one surface cannot serve another.
+ return {"cli": t[0:20], "raw": t[20:40], "mcp": t[40:60]}
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--out", required=True)
+ ap.add_argument("--size-mb", type=float, default=1024)
+ ap.add_argument("--seed", type=int, default=2421)
+ ap.add_argument("--queries", help="write the query set here (default <out>.queries.json)")
+ a = ap.parse_args()
+ if os.path.exists(a.out) and os.listdir(a.out):
+ sys.exit("refusing to write into non-empty %s" % a.out)
+ os.makedirs(a.out, exist_ok=True)
+ g = Gen(a.out, a.size_mb, a.seed).run()
+ qpath = a.queries or a.out.rstrip("/") + ".queries.json"
+ with open(qpath, "w") as f:
+ json.dump(queries(a.seed), f, indent=1)
+ manifest = {"seed": a.seed, "size_mb_requested": a.size_mb, "bytes": g.written, "files": g.files,
+ "by_family": {k: {"files": v[0], "bytes": v[1]} for k, v in sorted(g.by_family.items())},
+ "queries": qpath}
+ with open(a.out.rstrip("/") + ".manifest.json", "w") as f:
+ json.dump(manifest, f, indent=1)
+ print(json.dumps(manifest))
+
+
+if __name__ == "__main__":
+ main()