AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
bench(scripts): one-command repros for the four engine headline numbers (#806)
xerj-org955fc1642cf7
diff --git a/AGENTS.md b/AGENTS.mdindex fed49c320..66e766d4a 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -60,7 +60,7 @@ cannot push at all: [.github/AI_CONTRIBUTIONS.md](./.github/AI_CONTRIBUTIONS.md)## Ground rules for working in this repo-- **Honest claims only.** Every public number traces to a verified run. The **default** embedding mode is *lexical* feature-hashing — never claim the default mode does neural/semantic understanding. A built-in **neural** BERT embedder ships in the binary but is **off unless activated** (`--embed-mode neural`, model auto-downloads on first use); only describe output as neural when that mode is actually running. kNN: unfiltered queries are HNSW-served with exact rescoring — recall is *measured* (recall@10 1.00 on the official bench query, 100-probe mean 0.976), never claim "recall 1.00 by construction" for the ANN path; filtered/nested/SQ8/small-index shapes run the exact brute-force scan; benchmark headline is the audited scorecard in `demo/playbooks/SCORECARD.md` (currently 55W / 26T / 4L, 3 N/A vs live ES 8.13.4 — the 4 losses are all read-under-write p99 cells); do not claim TB-scale end-to-end (server heap ticket: `demo/usecases/autoindex/scale/TICKET_server-unbounded-ingest-heap.md`).+- **Honest claims only.** Every public number traces to a verified run. The **default** embedding mode is *lexical* feature-hashing — never claim the default mode does neural/semantic understanding. A built-in **neural** BERT embedder ships in the binary but is **off unless activated** (`--embed-mode neural`, model auto-downloads on first use); only describe output as neural when that mode is actually running. kNN: unfiltered queries are HNSW-served with exact rescoring — recall is *measured* (recall@10 1.00 on the official bench query, 100-probe mean 0.976), never claim "recall 1.00 by construction" for the ANN path; filtered/nested/SQ8/small-index shapes run the exact brute-force scan; benchmark headline is the audited scorecard in `demo/playbooks/SCORECARD.md` (currently 55W / 26T / 4L, 3 N/A vs live ES 8.13.4 — the 4 losses are all read-under-write p99 cells); do not claim TB-scale end-to-end (server heap ticket: `demo/usecases/autoindex/scale/TICKET_server-unbounded-ingest-heap.md`). Before quoting a headline number, check whether a one-command reproduction exists in [`scripts/bench/`](scripts/bench/) (ingest docs/s, size:0 latency, kNN recall, the rerank-stage contract — issue #806's rule is link-the-harness-or-drop-the-number; that directory's README maps every other published number to its harness too).- **Reference-code before you write.** For non-trivial engine work, retrieve how peer projects already solved the problem before writing your own version, and cite `file:line` for what you relied on. This repo dogfoods XERJ for it: reference projects (tantivy, meilisearch, quickwit, qdrant, usearch, hnswlib, sled, fjall, redb, ClickHouse) are indexed locally and queried with one command — `xerj search "<what you need>"` against a running node (it detects the node and ranks symbol definitions first), or `xerj def "<symbol>"` when you already know the name and want its file:line + signature in one call. For a shareable, pinned corpus definition across machines, the `xc-corpus.sh` / `xc.py` helpers in [`tools/xerj-code/`](tools/xerj-code/) wrap the same flow. Two rules bind regardless of tooling: **retrieved code is evidence, not authority** — adapt and attribute rather than paste; and **check the licence** — XERJ is Apache-2.0, so GPL-family sources (and Elasticsearch, which is AGPL/SSPL/Elastic-licensed) are approach-only, never copied. That second rule is also why XERJ can state it shares no code with Elasticsearch: ES is read for wire-protocol *semantics*, never for implementation. Honest scope: retrieval pays off on code a model has not memorised (internal, niche, post-cutoff) and is pure overhead on popular public libraries it already knows.- **Builds are scoped:** `cd engine && cargo build --release -j 32 -p <crate>` — never workspace-wide, never `cargo clean`.- **The hard gate:** the current CI ES-YAML conformance suite must stay at **0 failed**; pass and skip totals are reported by CI and grow as cases are added, so gate on failures, not on an exact total, before any engine change lands (see "Running the conformance tests" in the README).diff --git a/docs/RERANK.md b/docs/RERANK.mdindex 30ce905c9..6da2acbb9 100644--- a/docs/RERANK.md+++ b/docs/RERANK.md@@ -445,6 +445,14 @@ the lines.## Cost and concurrency facts+> Every measurable fact in this section — the 30-doc batch arithmetic+> (`window: 35` → two calls, `window: 300` → ten), the no-verdict-cache+> behaviour, the 8/16 in-flight ceilings, the degrade-on-deadline response and+> the no-key 503 — is reproduced end-to-end, against a local test double, by+> [`scripts/bench/rerank-stage.sh`](../scripts/bench/rerank-stage.sh) (9+> contract assertions; last run's output is in+> [`scripts/bench/README.md`](../scripts/bench/README.md)).+- **30 documents per provider call.** The `hev/jev-rerank` README reports a~32k-token request budget, "~30–50 typical passages per call", and uses 30;XERJ uses the same figure and has not measured the provider's limit itself. Adiff --git a/docs/WHY_XERJ.md b/docs/WHY_XERJ.mdindex 01d0c3cac..f14f9b3f8 100644--- a/docs/WHY_XERJ.md+++ b/docs/WHY_XERJ.md@@ -29,10 +29,10 @@ The autoindex pipeline was verified adversarially: 80/81 ground-truth checks on- **One static binary (61.6 MB, neural embedder included), no JVM.** Sub-second cold start. An agent can spawn its own search engine as a subprocess tool. Operationally: no heap tuning, no GC pauses. (A `--no-default-features` slim build without the neural backend is 58.2 MB — the neural backend accounts for 3.4 MB of the total, so dropping it is rarely worth the loss of capability.)- **Columnar aggregation fast paths + doc-value prefilters.** The analytical questions agents actually ask (counts, cardinalities, histograms, percentiles over millions of rows) are served from columnar data, not stored-document scans. This is where XERJ *beats* Elasticsearch in the head-to-head.-- **HNSW-served kNN with exact rescoring, measured honestly.** Unfiltered kNN is served by a persisted HNSW graph whose candidates are exact-rescored, so returned scores match the exact path bit-for-bit — measured recall@10 1.00 on the official bench query, 100-probe mean 0.976 (ES 8.13.4 same protocol: 0.937). Filtered, non-cosine, SQ8-quantized, and small-index shapes keep the exact brute-force scan (recall 1.00 by construction), and the trade-offs are documented rather than hidden behind approximate defaults.+- **HNSW-served kNN with exact rescoring, measured honestly.** Unfiltered kNN is served by a persisted HNSW graph whose candidates are exact-rescored, so returned scores match the exact path bit-for-bit — measured recall@10 1.00 on the official bench query, 100-probe mean 0.976 (ES 8.13.4 same protocol: 0.937). Filtered, non-cosine, SQ8-quantized, and small-index shapes keep the exact brute-force scan (recall 1.00 by construction), and the trade-offs are documented rather than hidden behind approximate defaults. One-command reproduction: [`scripts/bench/knn-recall.sh`](../scripts/bench/knn-recall.sh).- **Built-in zero-config embedder.** Lexical feature-hashing (384-dim) — documented honestly as *hybrid lexical+vector*, never claimed to be neural. Semantic-ish retrieval works offline with zero external dependencies; point `semantic_text` at an external inference endpoint when you need real embeddings.- **Durability verified adversarially.** Acked deletes survive SIGTERM/SIGKILL restarts (11/11 randomized crash-timing cells); merges are structurally unable to drop documents silently (loss firewall: a merge that cannot read an input aborts loudly and keeps inputs live). These properties are enforced by crash matrices and merge-torture harnesses, not by assertion.-- **Honest benchmarking as policy.** The published head-to-head vs live Elasticsearch 8.13.4 reports wins *and* losses with root causes (currently 55W / 26T / 4L, 3 N/A on the full matrix; 1.72× ingest and 1.61× disk wins; the 4 losses are all read-under-write p99 cells, documented). Numbers are only published after an independent adversarial re-measure — and when re-measurement moved the number *against* XERJ, the public number was corrected the same day.+- **Honest benchmarking as policy.** The published head-to-head vs live Elasticsearch 8.13.4 reports wins *and* losses with root causes (currently 55W / 26T / 4L, 3 N/A on the full matrix; 1.72× ingest and 1.61× disk wins; the 4 losses are all read-under-write p99 cells, documented). Numbers are only published after an independent adversarial re-measure — and when re-measurement moved the number *against* XERJ, the public number was corrected the same day. One-command reproductions for the XERJ side of the headline cells (ingest docs/s, size:0 latency, kNN recall, the rerank-stage contract) live in [`scripts/bench/`](../scripts/bench/).## What ES compatibility buys (and what it doesn't)diff --git a/docs/recipes/vector-search-knn.md b/docs/recipes/vector-search-knn.mdindex 30ec34d88..cdb13fb62 100644--- a/docs/recipes/vector-search-knn.md+++ b/docs/recipes/vector-search-knn.md@@ -232,7 +232,12 @@ kNN, non-cosine metrics, SQ8-quantized fields, and small corpora run theexact brute-force scan. This recipe's 8-row corpus is far below thatthreshold, so its queries ran the **exact** path — the script abovemeasures **recall@k = 1.000** against a brute-force cosine ground truth-on every query. The knobs above only start to matter at scale.+on every query. The knobs above only start to matter at scale. To measure+the HNSW path itself the way the published number was measured, run+[`scripts/bench/knn-recall.sh`](../../scripts/bench/knn-recall.sh): 50k+generated 128-d vectors, 100 probe queries scored against a client-side+brute-force ground truth, and a `GET /_cat/ann` check so a silent fallback+to the exact scan cannot pass for the graph.---diff --git a/scripts/bench/README.md b/scripts/bench/README.mdnew file mode 100644index 000000000..820c6011b--- /dev/null+++ b/scripts/bench/README.md@@ -0,0 +1,148 @@+# scripts/bench — one-command reproductions for the headline numbers++Issue #806's ask, in one line: *for every number that appears in public copy,+either link the exact harness that produces it, or remove the number.* These+four scripts are that harness for the ENGINE numbers a stranger is most likely+to want to check. Each one:++- builds nothing and argues nothing — it starts a **throwaway node on a private+ port (93xx, never `:9200`)** with a throwaway data dir,+- runs on **generated (or committed synthetic) data**, end-to-end, through the+ Elasticsearch wire API,+- prints the **measured value next to the claimed one**, with the claim's+ source pointer, and+- tears the node down (pass `KEEP=1` to keep the data dir and server log).++They are honest by construction: docs/s is computed from the verified+post-refresh `_count`, never the intended count; kNN recall is scored against+a client-side brute-force ground truth and `GET /_cat/ann` proves the HNSW+graph (not a silent exact-scan fallback) served the queries; the rerank script+counts the provider calls the node actually made, from the provider's side.++## The four repros++| Headline claim (where it is quoted) | Claimed | Reproduce with |+|---|---|---|+| Bulk ingest, 100k docs x 1 client — `demo/playbooks/SCORECARD.md:17`, landing `/benchmarks/elasticsearch` ("1.72x higher") | XERJ **191,286 docs/s** (ES 111,073) | `bash scripts/bench/ingest-docs-per-second.sh` |+| Server round-trip, `size:0` over 300k docs — scorecard methodology note, landing `/benchmarks` ("0.126 ms, 2.2x") | **0.126 ms** avg, keep-alive client (1.61 ms via fetch/undici — client overhead) | `bash scripts/bench/query-latency-size0.sh` |+| kNN recall, HNSW-served with exact rescoring — `docs/WHY_XERJ.md`, `ROADMAP.md`, landing `/docs/vectors` | recall@10 **1.00** bench query, **100-probe mean 0.976** / min 0.90 (ES 0.937) | `bash scripts/bench/knn-recall.sh` |+| Jev rerank stage, cost + failure contract — `docs/RERANK.md` ("30 documents per provider call", "no verdict cache", "8 in flight by default, 16 at most", "no key -> 503") | see each row in the script output | `bash scripts/bench/rerank-stage.sh` |++Requirements for all four: `bash`, `curl`, `python3` (the latency and kNN+repros also use `node >= 18`), and a built server binary —+`cd engine && cargo build --release -p xerj-server`, or point `XERJ_BIN` at an+existing one. Runtime on a 32-core dev box: ~30 s (rerank), ~1 min (ingest,+kNN), ~2 min (latency, which indexes 300k docs first).++## What each script deliberately does NOT claim++- **The Elasticsearch side of each comparison is not run here.** That needs an+ ES install; the head-to-head harness is `demo/playbooks/bench-matrix.mjs`+ (one file, Node builtins only). These scripts reproduce the XERJ side of the+ published cell, on the published protocol.+- **The Jev model is not run, ever.** `rerank-stage.sh` measures the stage's+ published cost/failure contract against `lib/jev_stub.py`, a local test+ double with the same word-overlap verdicts as the engine's own test double.+ The nDCG figures for Jev/Voyage/Cohere in `docs/RERANK.md` are+ `hev/jev-rerank`'s numbers and are marked "not run by us" — no harness in+ this repository can or should reproduce them.+- **Absolute milliseconds and docs/s are hardware-bound.** A repro on a slower+ or shared box will measure slower numbers; what reproduces everywhere is the+ ordering, the gap, and (for kNN recall) the value itself, which is a+ property of the index, not the disk.++## Last verified run (this repository's CI sandbox)++2026-09-20, xerj v1.0.0-rc.75 release binary, Linux x86_64, 32 cores — all+four scripts run back-to-back from this tree, commands verbatim as documented+above. This sandbox is **shared**, so the two wall-clock-bound numbers move+run to run (ranges from the same session noted under each); the two+index-property numbers do not.++### ingest-docs-per-second.sh++```+ CLAIMED 191,286 docs/s — ingest 100k x c1, scorecard run 2026-07 (demo/playbooks/SCORECARD.md:17)+ CLAIMED (ES 8.13.4 same corpus/box: 111,073 docs/s -> 1.72x; ES side needs an ES install, not run here)+ note corpus: demo/data/extras/chat-events.ndjson (4008 distinct events, cycled to 100000 docs — the published protocol)++ MEASURED 99602 docs/s (100000 docs indexed and verified by _count in 1004 ms, 1 client, 10000-doc batches)+ MEASURED bulk responses with errors: 0 (item errors: 0)+ ratio 0.52x of the claimed figure (same protocol, this machine — not the scorecard box)+```++Variance note: three runs the same hour measured 50,201 / 103,520 / 99,602+docs/s (the middle one on the script's fallback generator, the others on the+committed corpus) — a 2x spread from sandbox neighbours alone. That spread is+why the script prints its ratio next to the claim instead of asserting a pass,+and why the published cell names a dedicated benchmark box.++### query-latency-size0.sh++```+ CLAIMED 0.126 ms avg, size:0 over 300k, Node http keep-alive client (scorecard run 2026-07)+ CLAIMED 1.61 ms avg for the SAME request through Node's global fetch/undici — client overhead, not server time+ MEASURED keep-alive client : 0.491 ms avg (p50 0.464 / p99 0.870 / max 5.714)+ MEASURED fetch/undici : 0.735 ms avg (p50 0.711 / p99 1.377 / max 5.582)+ MEASURED client overhead : 0.245 ms (1.5x) — the published point: the client, not the server+ note 1200 timed iterations at 200 req/s open-loop, request_cache=false, query cache off, 1 merged segment+```++Same-session range: keep-alive avg 0.491–1.216 ms across two runs (the slower+one measured keep-alive 1.216 / fetch 2.631, a 2.16x gap). The reproduced fact+is the ordering and the client-overhead gap, not the absolute value.++### knn-recall.sh++```+ CLAIMED recall@10 = 1.00 on the official bench query; 100-probe mean 0.976 / min 0.90 (docs/WHY_XERJ.md:32)+ CLAIMED candidates exact-rescored: returned scores match the exact brute-force path+ MEASURED recall@10 : 0.977 mean over 100 probes (min 0.900, max 1.000, 77/100 probes at 1.00)+ MEASURED score parity : max |engine _score - client exact (1+cos)/2| = 5.61e-08+ MEASURED kNN latency : p50 5.85 ms per probe, k=10, num_candidates=100 over 50000 x 128-d vectors+ MEASURED ANN health : coverage=50000 stale=False nodes=50000 (GET /_cat/ann — proves the HNSW+ graph served this, not a silent exact-scan fallback)+```++The published figures reproduce almost exactly: 0.977 mean / 0.900 min here+against 0.976 / 0.90 published (an earlier run the same session read 0.974+mean / 0.800 min — the mean is stable, a single worst probe is a draw). Recall+is a property of the index and query distribution, not of the hardware, so+this is the one headline that should reproduce closely anywhere.++### rerank-stage.sh++```+ PASS defaults window/batch/max_concurrency claimed 30 / 30 / 8 measured 30 / 30 / 8+ PASS limits max_docs_per_call/max_window/... claimed 30 / 300 / 16 measured 30 / 300 / 16+ PASS window 35 -> provider calls / docs per call claimed 2 calls: 30 + 5 measured 2 calls: 5 + 30+ PASS window 300 -> provider calls / docs per call claimed 10 calls of 30 measured 10 calls of 30+ PASS 3 page requests -> calls / paid judgements claimed 3 calls, 90 measured 3 calls, 90+ PASS slow provider (2s) with timeout_ms=200 claimed applied:false + engine order+ measured HTTP 200, applied:False, score_kind:engine,+ order == engine, reason: deadline exceeded after 200ms+ PASS verdicts replace _score (min_score 0.5) claimed probability ordering + prune counting+ measured judged:40, pruned:21, order desc, min _score 0.800+ PASS max provider calls in flight, window 300 claimed 8 by default, 16 at most+ measured default: 8 in flight over 10 calls;+ max_concurrency=16: 10 in flight over 10 calls+ PASS no key configured -> refusal, not silence claimed HTTP 503, rerank_exception, no hits+ measured HTTP 503, hits 0, error type rerank_exception+ contract checks: 9/9 PASS (deterministic contract — these assertions are the reproduction)+```++(The per-call log order in the `window 35` row is completion order — the two+calls run concurrently; the assertion checks the multiset `{30, 5}`.)++## The other published numbers, and where their harnesses already live++So that every headline maps to a runnable thing from one place:++| Claim | Harness in this repo |+|---|---|+| 2.7x / 26x / up-to-278x fewer output tokens (reference coding) | `docs/case-studies/reference-coding/CASE_STUDY.md` + the 3-arm Harbor harness `tools/benchmarks/harbor-3arm/` (`run_matrix.sh`) |+| nDCG@10 rows (BM25 0.6572 / MiniLM 0.6764 / hybrid 0.6993 on SciFact) | `benchmarks/beir-hybrid/` (`load.py`, `eval.py`, raw output in `results/`) |+| Banking77 0.933 / SMS spam 0.983 decisions-from-history | `benchmarks/decisions-as-retrieval/` (`eval.py`, raw output in `results/`) |+| ES-YAML conformance badge (1366/1369) | the CI suite itself: `engine/tests/es-compat-yaml/` |+| 55 W / 26 T / 4 L vs Elasticsearch (all 88 cells) | `demo/playbooks/bench-matrix.mjs` -> `demo/playbooks/SCORECARD.md` |+| int8 quantization recall@10 ~ 0.998 | `docs/examples/vector-quantization/quant_demo.py` (docs in `docs/recipes/vector-quantization.md`) |diff --git a/scripts/bench/ingest-docs-per-second.sh b/scripts/bench/ingest-docs-per-second.shnew file mode 100755index 000000000..2655edbe1--- /dev/null+++ b/scripts/bench/ingest-docs-per-second.sh@@ -0,0 +1,164 @@+#!/usr/bin/env bash+# ─────────────────────────────────────────────────────────────────────────────+# ingest-docs-per-second.sh — one-command reproduction of the ingest headline.+#+# CLAIMED XERJ 191,286 docs/s on the "ingest 100k x c1" cell+# (ES 8.13.4 on the identical corpus and box: 111,073 docs/s,+# 1.72x) — demo/playbooks/SCORECARD.md:17, quoted on+# landing/benchmarks/elasticsearch.html ("Bulk ingest, identical+# corpus and box: 1.72x higher").+#+# What this script does: builds nothing, argues nothing — it starts a+# throwaway node on a PRIVATE port (93xx, never :9200), indexes 100,000+# synthetic LLM-telemetry docs through ES-wire _bulk exactly the way the+# scorecard harness does (1 client, 10k docs/batch, pre-generated batch files+# so file-write cost stays out of the timed window, docs/s computed from the+# VERIFIED post-refresh count, not the intended count), and prints the number+# next to the claim.+#+# The ES side of the comparison is NOT run here (it needs an ES install);+# demo/playbooks/bench-matrix.mjs --ingest-only is the head-to-head harness.+#+# Usage:+# bash scripts/bench/ingest-docs-per-second.sh+# Knobs: DOCS=100000 BATCH=10000 PORT=<free 93xx port> XERJ_BIN=<binary> KEEP=1+# Requires: curl, python3, a built server binary (engine/target/release/xerj).+# ─────────────────────────────────────────────────────────────────────────────+set -euo pipefail++HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+# shellcheck source=lib/repro.sh+source "$HERE/lib/repro.sh"++DOCS="${DOCS:-100000}"+BATCH="${BATCH:-10000}"+CORPUS="$REPO_ROOT/demo/data/extras/chat-events.ndjson"++repro_find_binary+repro_free_port 9310+PORT_NOTE="$REPRO_PORT"++repro_header "one-command repro: bulk ingest docs/s"+repro_claim "191,286 docs/s — ingest 100k x c1, scorecard run 2026-07 (demo/playbooks/SCORECARD.md:17)"+repro_claim " (ES 8.13.4 same corpus/box: 111,073 docs/s -> 1.72x; ES side needs an ES install, not run here)"++# ── corpus: the committed synthetic telemetry the published run cycled ──────+if [ ! -f "$CORPUS" ]; then+ echo "corpus $CORPUS missing — generating an equivalent-shaped fallback"+ CORPUS="$(mktemp /tmp/xerj-bench-corpus.XXXXXX.ndjson)"+ python3 - "$CORPUS" <<'PY'+import json, random, sys, time+random.seed(42)+out = open(sys.argv[1], "w")+models = ["claude-sonnet-4-5"] * 5 + ["claude-haiku-4-5"] * 3 + ["gpt-5.2", "gemini-3-pro"]+intents = ["refactor", "explain", "debug", "review", "test", "migrate", "optimise",+ "document", "design", "triage", "revert", "ship"]+tops = ["AGENTS.md", "engine/CLAUDE.md", "docs/RERANK.md", "xerj.default.toml",+ "landing/index.html", "scripts/bench", "crates/xerj-engine/src/index.rs",+ "Cargo.toml", "README.md", "es-compat", "wal", "hnsw"]+tenants = ["acme", "globex", "initech", "umbrella", "wayne"]+base = time.time()+for i in range(4008):+ pt = max(1, int(random.gauss(900, 300)))+ ct = max(0, int(random.gauss(250, 90)))+ comp = max(0, int(random.gauss(180, 70)))+ lat = max(1, int(random.gauss(random.choice([310, 520, 780, 1100, 1450]), 120)))+ out.write(json.dumps({+ "@timestamp": base + i, "model": random.choice(models), "intent": random.choice(intents),+ "top_doc": random.choice(tops), "tenant": random.choice(tenants),+ "status": "error" if random.random() < 0.0125 else "ok", "cache_hit": random.random() < 0.429,+ "prompt_tokens": pt, "context_tokens": ct, "completion_tokens": comp,+ "latency_ms": lat, "cost_usd": round((pt + ct + comp) * 3e-6, 6),+ }) + "\n")+PY+ repro_note "fallback corpus generated (the committed one is what the published run used)"+fi++RAW_LINES="$(wc -l <"$CORPUS")"+repro_note "corpus: $CORPUS ($RAW_LINES distinct events, cycled to $DOCS docs — the published protocol)"++repro_node_start -+trap repro_node_stop EXIT+repro_node_wait++BASE="http://127.0.0.1:$REPRO_PORT"+IDX="bench-ingest"++curl -sf -XDELETE "$BASE/$IDX" >/dev/null 2>&1 || true+curl -sf -XPUT "$BASE/$IDX" -H 'content-type: application/json' --data-binary @- >/dev/null <<'JSON'+{ "mappings": { "properties": {+ "@timestamp": { "type": "date" },+ "model": { "type": "keyword" }, "intent": { "type": "keyword" }, "status": { "type": "keyword" },+ "tenant": { "type": "keyword" }, "top_doc": { "type": "keyword" }, "cache_hit": { "type": "boolean" },+ "prompt_tokens": { "type": "integer" }, "context_tokens": { "type": "integer" }, "completion_tokens": { "type": "integer" },+ "latency_ms": { "type": "integer" }, "cost_usd": { "type": "double" }+} } }+JSON++# ── pre-generate every batch file, then time only the _bulk posts ───────────+SCRATCH="$(mktemp -d /tmp/xerj-bench-bulk.XXXXXX)"+trap 'rm -rf "$SCRATCH"; repro_node_stop' EXIT+python3 - "$CORPUS" "$SCRATCH" "$DOCS" "$BATCH" <<'PY'+import os, sys+corpus, scratch, total, batch = sys.argv[1], sys.argv[2], int(sys.argv[3]), int(sys.argv[4])+raw = open(corpus).read().splitlines()+n, written, b = len(raw), 0, 0+while written < total:+ lines = []+ for _ in range(min(batch, total - written)):+ lines.append('{"index":{}}')+ lines.append(raw[written % n])+ written += 1+ with open(os.path.join(scratch, f"batch_{b:04d}.ndjson"), "w") as f:+ f.write("\n".join(lines) + "\n")+ b += 1+print(f"batches: {b} x {batch} docs (pre-generated, out of the timed window)")+PY++bulk_errors=0+bulk_item_errors=0+start_ns=$(date +%s%N)+for f in "$SCRATCH"/batch_*.ndjson; do+ resp="$(curl -s -XPOST "$BASE/$IDX/_bulk" -H 'content-type: application/x-ndjson' --data-binary @"$f")"+ # a silent drop must not be able to inflate docs/s: read every response+ if ! errcount="$(printf '%s' "$resp" | python3 -c '+import json, sys+try:+ j = json.load(sys.stdin)+except Exception:+ print("PARSE"); sys.exit()+if j.get("errors"):+ bad = sum(1 for it in j.get("items", []) if (it and next(iter(it.values()), {}) or {}).get("status", 0) >= 400)+ print(bad or "ITEMS")+else:+ print(0)+')"; then+ bulk_errors=$((bulk_errors + 1))+ continue+ fi+ case "$errcount" in+ 0) ;;+ PARSE|ITEMS) bulk_errors=$((bulk_errors + 1)) ;;+ *) bulk_errors=$((bulk_errors + 1)); bulk_item_errors=$((bulk_item_errors + errcount)) ;;+ esac+done+end_ns=$(date +%s%N)++curl -sf -XPOST "$BASE/$IDX/_refresh" >/dev/null+count="$(curl -sf "$BASE/$IDX/_count" | python3 -c 'import json,sys; print(json.load(sys.stdin)["count"])')"+elapsed_ms=$(( (end_ns - start_ns) / 1000000 ))+dps="$(python3 -c "print(round($count / ($elapsed_ms / 1000)))")"++printf '\n'+repro_measured "$dps docs/s ($count docs indexed and verified by _count in ${elapsed_ms} ms, 1 client, $BATCH-doc batches)"+repro_measured "bulk responses with errors: $bulk_errors (item errors: $bulk_item_errors)"+python3 - "$dps" <<'PY'+import sys+dps, claimed = int(sys.argv[1]), 191286+r = dps / claimed+print(f" ratio {r:.2f}x of the claimed figure (same protocol, this machine — not the scorecard's box)")+print(f" vs the ES side of the published cell (111,073 docs/s): {dps / 111073:.2f}x")+PY+repro_note "docs/s uses the verified post-refresh count, never the intended count (scorecard rule)"+repro_env_box+repro_note "teardown: node stopped, throwaway data dir removed (KEEP=1 to keep)"diff --git a/scripts/bench/knn-recall.sh b/scripts/bench/knn-recall.shnew file mode 100755index 000000000..3de073db6--- /dev/null+++ b/scripts/bench/knn-recall.sh@@ -0,0 +1,198 @@+#!/usr/bin/env bash+# ─────────────────────────────────────────────────────────────────────────────+# knn-recall.sh — one-command reproduction of the kNN recall headline.+#+# CLAIMED unfiltered kNN on a full-precision cosine field (>=1,024 docs)+# is HNSW-served with exact rescoring: "measured recall@10 1.00 on+# the official bench query, 100-probe mean 0.976" — docs/WHY_XERJ.md:32,+# ROADMAP.md:23, landing/docs/vectors.html, quoted with ES 8.13.4+# at 0.937 on the same protocol.+#+# What this script does: starts a throwaway node on a PRIVATE port (93xx),+# indexes 50,000 generated 128-dim vectors (mulberry32 seed 0xC0FFEE, explicit+# ids — the exact generator and shape of demo/playbooks/bench-matrix.mjs+# knnBench), checks GET /_cat/ann so a silent fallback to the exact scan+# cannot pass for the HNSW path, then runs 100 probe queries of k=10,+# num_candidates=100 and scores each against a client-side brute-force cosine+# top-10 ground truth. Recall is MEASURED, never assumed — that is the whole+# point of the claim.+#+# It also verifies the second half of the claim — candidates are exact-+# rescored, so returned scores match the exact path — by comparing every+# engine _score against the client-side (1 + cos)/2 of the same document.+#+# Usage:+# bash scripts/bench/knn-recall.sh+# Knobs: VECS=50000 DIMS=128 PROBES=100 K=10 CANDIDATES=100 PORT=<free 93xx port>+# XERJ_BIN=<binary> KEEP=1+# Requires: node >= 18, curl, a built server binary.+# ─────────────────────────────────────────────────────────────────────────────+set -euo pipefail++HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+# shellcheck source=lib/repro.sh+source "$HERE/lib/repro.sh"++VECS="${VECS:-50000}"+DIMS="${DIMS:-128}"+PROBES="${PROBES:-100}"+K="${K:-10}"+CANDIDATES="${CANDIDATES:-100}"++repro_find_binary+repro_free_port 9334++repro_header "one-command repro: kNN recall@${K} (HNSW-served, exact-rescored)"+repro_claim "recall@10 = 1.00 on the official bench query; 100-probe mean 0.976 / min 0.90 (docs/WHY_XERJ.md:32)"+repro_claim "candidates exact-rescored: returned scores match the exact brute-force path"+repro_note "ES 8.13.4 scored 0.937 mean / 0.70 min on the same protocol — the ES side needs an ES install"++repro_node_start -+trap repro_node_stop EXIT+repro_node_wait++BASE="http://127.0.0.1:$REPRO_PORT"+IDX="bench-vec"+SCRATCH="$(mktemp -d /tmp/xerj-bench-knn.XXXXXX)"+trap 'rm -rf "$SCRATCH"; repro_node_stop' EXIT+RESULT_FILE="$SCRATCH/knn.json"++curl -sf -XDELETE "$BASE/$IDX" >/dev/null 2>&1 || true+curl -sf -XPUT "$BASE/$IDX" -H 'content-type: application/json' --data-binary @- >/dev/null <<JSON+{ "mappings": { "properties": {+ "v": { "type": "dense_vector", "dims": $DIMS, "index": true, "similarity": "cosine" }+} } }+JSON++node - "$BASE" "$IDX" "$VECS" "$DIMS" "$PROBES" "$K" "$CANDIDATES" "$RESULT_FILE" <<'NODE'+// Same generator, corpus shape and query shape as demo/playbooks/bench-matrix.mjs+// knnBench(): mulberry32(0xC0FFEE) uniforms mapped to [-1,1), explicit _ids so+// engine hits map back to generated vectors, cosine similarity, k=10,+// num_candidates=100. Extended from 1 probe to PROBES so the "100-probe mean"+// half of the claim is measured, not extrapolated from one draw.+const http = require('node:http');+const [base, idx, M, DIMS, PROBES, K, CAND, outFile] = [process.argv[2], process.argv[3], +process.argv[4], +process.argv[5], +process.argv[6], +process.argv[7], +process.argv[8], process.argv[9]];+const port = +base.split(':')[2];+const agent = new http.Agent({ keepAlive: true, maxSockets: 256 });++function mulberry32(a) {+ return function () {+ a |= 0; a = (a + 0x6D2B79F5) | 0;+ let t = Math.imul(a ^ (a >>> 15), 1 | a);+ t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;+ return ((t ^ (t >>> 14)) >>> 0) / 4294967296;+ };+}+const rng = mulberry32(0xC0FFEE);+const vecs = new Array(M);+for (let i = 0; i < M; i++) {+ const v = new Array(DIMS);+ for (let k = 0; k < DIMS; k++) v[k] = rng() * 2 - 1;+ vecs[i] = v;+}++function req(method, path, body, ndjson) {+ const payload = body === undefined ? undefined : (ndjson ? body : JSON.stringify(body));+ const headers = { 'content-type': ndjson ? 'application/x-ndjson' : 'application/json' };+ if (payload !== undefined) headers['content-length'] = Buffer.byteLength(payload);+ return new Promise((resolve, reject) => {+ const r = http.request({ hostname: '127.0.0.1', port, path, method, headers, agent }, (res) => {+ let txt = ''; res.setEncoding('utf8');+ res.on('data', (c) => { txt += c; });+ res.on('end', () => {+ if (res.statusCode >= 400) return reject(new Error(`${method} ${path} -> ${res.statusCode}: ${txt.slice(0, 300)}`));+ resolve(txt ? JSON.parse(txt) : null);+ });+ });+ r.on('error', reject);+ if (payload !== undefined) r.write(payload);+ r.end();+ });+}++function cosine(a, b) {+ let d = 0, na = 0, nb = 0;+ for (let i = 0; i < a.length; i++) { d += a[i] * b[i]; na += a[i] * a[i]; nb += b[i] * b[i]; }+ return d / (Math.sqrt(na) * Math.sqrt(nb) || 1);+}++(async () => {+ // bulk load in 5k-doc chunks through the same NDJSON wire the harness uses+ const CHUNK = 5000;+ for (let s = 0; s < M; s += CHUNK) {+ let nd = '';+ for (let i = s; i < Math.min(s + CHUNK, M); i++)+ nd += `{"index":{"_id":"${i}"}}\n` + JSON.stringify({ v: vecs[i] }) + '\n';+ const r = await req('POST', `/${idx}/_bulk`, nd, true);+ if (r && r.errors) throw new Error('bulk reported item errors');+ process.stderr.write(` indexed ${Math.min(s + CHUNK, M)}/${M}\n`);+ }+ await req('POST', `/${idx}/_refresh`);++ // wait for the HNSW graph to cover the corpus (a stale/under-covered graph+ // silently degrades kNN to the exact scan — this is the check)+ let ann = null;+ for (let i = 0; i < 120; i++) {+ const rows = await req('GET', `/_cat/ann/${idx}?format=json`);+ ann = rows[0] || {};+ if (Number(ann.coverage) >= 1 && !String(ann.stale).match(/true/i)) break;+ await new Promise((r) => setTimeout(r, 1000));+ }++ const recalls = []; let maxScoreDelta = 0; const lat = [];+ for (let q = 0; q < PROBES; q++) {+ const qv = new Array(DIMS);+ for (let k = 0; k < DIMS; k++) qv[k] = rng() * 2 - 1;+ // client-side exact brute-force top-K ground truth+ const scored = new Array(M);+ for (let i = 0; i < M; i++) scored[i] = [i, cosine(qv, vecs[i])];+ scored.sort((a, b) => b[1] - a[1]);+ const exact = new Map(scored.slice(0, K).map(([i, c]) => [i, (1 + c) / 2]));+ const body = { knn: { field: 'v', query_vector: qv, k: K, num_candidates: CAND }, size: K };+ const t0 = performance.now();+ const r = await req('POST', `/${idx}/_search?request_cache=false`, body);+ lat.push(performance.now() - t0);+ const hits = (r.hits && r.hits.hits) || [];+ const ids = hits.map((h) => parseInt(h._id, 10));+ let hit = 0;+ ids.forEach((id, n) => {+ if (exact.has(id)) hit++;+ if (hits[n] && typeof hits[n]._score === 'number' && exact.has(id)) {+ maxScoreDelta = Math.max(maxScoreDelta, Math.abs(hits[n]._score - exact.get(id)));+ }+ });+ recalls.push(ids.length ? hit / K : 0);+ }+ agent.destroy();+ const mean = recalls.reduce((a, b) => a + b, 0) / recalls.length;+ const out = {+ vectors: M, dims: DIMS, probes: PROBES, k: K, num_candidates: CAND,+ recall_mean: mean, recall_min: Math.min(...recalls), recall_max: Math.max(...recalls),+ perfect_probes: recalls.filter((r) => r === 1).length,+ max_abs_score_delta_vs_exact: maxScoreDelta,+ latency_p50_ms: lat.sort((a, b) => a - b)[Math.floor(lat.length / 2)],+ ann,+ };+ require('node:fs').writeFileSync(outFile, JSON.stringify(out));+ process.stderr.write('done\n');+})().catch((e) => { console.error(String(e && e.message || e)); process.exit(1); });+NODE++python3 - "$RESULT_FILE" <<'PY'+import json, sys+r = json.load(open(sys.argv[1]))+ann = r.get("ann") or {}+print()+print(f' MEASURED recall@{r["k"]} : {r["recall_mean"]:.3f} mean over {r["probes"]} probes '+ f'(min {r["recall_min"]:.3f}, max {r["recall_max"]:.3f}, {r["perfect_probes"]}/{r["probes"]} probes at 1.00)')+print(f' MEASURED score parity : max |engine _score - client exact (1+cos)/2| = {r["max_abs_score_delta_vs_exact"]:.2e} '+ f'(claim: exact rescoring, scores match the exact path)')+print(f' MEASURED kNN latency : p50 {r["latency_p50_ms"]:.2f} ms per probe, k={r["k"]}, num_candidates={r["num_candidates"]} '+ f'over {r["vectors"]} x {r["dims"]}-d vectors')+print(f' MEASURED ANN health : coverage={ann.get("coverage")} stale={ann.get("stale")} nodes={ann.get("nodes")} '+ f'(GET /_cat/ann — proves the HNSW graph served this, not a silent exact-scan fallback)')+print(' note 50k x 128-d uniform random vectors reproduce the harness generator; recall on a')+print(' different distribution can differ — the claim is measured on THIS shape, as published')+PY+repro_env_box+repro_note "teardown: node stopped, throwaway data dir removed (KEEP=1 to keep)"diff --git a/scripts/bench/lib/jev_stub.py b/scripts/bench/lib/jev_stub.pynew file mode 100644index 000000000..a54dd3981--- /dev/null+++ b/scripts/bench/lib/jev_stub.py@@ -0,0 +1,117 @@+#!/usr/bin/env python3+"""Local test double for the Jev rerank provider (TypeSafe AI System One).++XERJ's `rerank` search stage POSTs one request per judged document batch to+the provider's `/v1/systemone`. This stub speaks that wire format so the+stage's published cost/concurrency facts can be measured end-to-end against a+real node with no key, no network egress, and no non-determinism:++ request {"model": ..., "state": {"query": ..., "documents":+ {"d0": {"title": ..., "text": ...}, ...}}}+ response {"model": ..., "answers": {"d0": {"type": "noul", "noul": 0.42}},+ "usage": {"input_tokens": N, "output_tokens": N}}++Verdicts are the same deterministic word-overlap relevance the engine's own+test double uses (engine/crates/xerj-api/tests/rerank_stage_http.rs+`Verdicts::Overlap`): p = |query words found in title+text| / |query words|.++Control (a JSON file watched per request, so a running repro can flip modes):+ {"sleep_ms": 2000} stall every call, to exercise the degrade-on-deadline+ path and the in-flight ceiling++Every call is appended to a JSONL log: one line with start/end monotonic+timestamps, the document count, the batch keys, and the question. Call counts,+per-call document counts and max in-flight concurrency are computed from that+log by the caller. The Authorization header is NEVER logged.++Usage: jev_stub.py --port 9340 --log calls.jsonl --control control.json+"""+import argparse+import json+import time+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer+++def words(s):+ return {w for w in "".join(c if c.isalnum() else " " for c in s.lower()).split() if w}+++class Handler(BaseHTTPRequestHandler):+ protocol_version = "HTTP/1.1"++ def log_message(self, fmt, *args): # keep the repro's stdout clean+ pass++ def _json(self, code, obj):+ body = json.dumps(obj).encode()+ self.send_response(code)+ self.send_header("content-type", "application/json")+ self.send_header("content-length", str(len(body)))+ self.end_headers()+ self.wfile.write(body)++ def do_POST(self):+ if self.path != "/v1/systemone":+ return self._json(404, {"error": "not found"})+ length = int(self.headers.get("content-length") or 0)+ raw = self.rfile.read(length)+ try:+ parsed = json.loads(raw)+ except Exception:+ return self._json(400, {"error": "bad json"})+ start = time.monotonic()++ try:+ with open(self.server.control_path) as f:+ sleep_ms = float(json.load(f).get("sleep_ms", 0))+ except Exception:+ sleep_ms = 0.0+ if sleep_ms:+ time.sleep(sleep_ms / 1000.0)++ state = parsed.get("state") or {}+ qwords = words(str(state.get("query") or ""))+ docs = state.get("documents") or {}+ answers = {}+ for key, doc in docs.items():+ have = words(f"{doc.get('title', '')} {doc.get('text', '')}")+ p = (len(qwords & have) / len(qwords)) if qwords else 0.0+ answers[key] = {"type": "noul", "noul": round(p, 6)}+ self._json(200, {+ "model": parsed.get("model", "jev-latest"),+ "answers": answers,+ "usage": {"input_tokens": 100, "output_tokens": 10},+ })+ end = time.monotonic()+ with open(self.server.log_path, "a") as f:+ f.write(json.dumps({+ "seq": int(time.time() * 1000) % 10**9,+ "start": start,+ "end": end,+ "ndocs": len(docs),+ "keys": sorted(docs.keys()),+ "query": str(state.get("query") or "")[:200],+ }) + "\n")++ def do_GET(self):+ if self.path == "/health":+ return self._json(200, {"ok": True})+ return self._json(404, {"error": "not found"})+++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("--port", type=int, required=True)+ ap.add_argument("--log", required=True)+ ap.add_argument("--control", required=True)+ args = ap.parse_args()+ open(args.log, "w").close()+ json.dump({"sleep_ms": 0}, open(args.control, "w"))+ srv = ThreadingHTTPServer(("127.0.0.1", args.port), Handler)+ srv.log_path = args.log+ srv.control_path = args.control+ srv.serve_forever()+++if __name__ == "__main__":+ main()diff --git a/scripts/bench/lib/repro.sh b/scripts/bench/lib/repro.shnew file mode 100644index 000000000..07b286534--- /dev/null+++ b/scripts/bench/lib/repro.sh@@ -0,0 +1,181 @@+#!/usr/bin/env bash+# ─────────────────────────────────────────────────────────────────────────────+# repro.sh — shared plumbing for the scripts/bench one-command reproductions.+#+# Every scripts/bench/<name>.sh reproduces ONE headline number that public copy+# quotes (landing/benchmarks/*, docs/*, demo/playbooks/SCORECARD.md) against a+# throwaway local node on a PRIVATE port (93xx, never :9200 — the runner's+# cleanup_indices does DELETE /_all and would wipe anything living there).+#+# Sourced, never executed directly:+# source "$(dirname "${BASH_SOURCE[0]}")/lib/repro.sh"+#+# Provides:+# repro_find_binary -> sets $XERJ_BIN (existing release binary)+# repro_free_port MIN -> sets $REPRO_PORT (first free base in MIN..MIN+19,+# checking base/base+1/base+2 — the node claims all 3)+# repro_node_start [extra args...] -> starts $XERJ_BIN on $REPRO_PORT+# repro_node_wait -> poll GET / until it answers+# repro_node_stop -> kill + remove the throwaway data dir+# repro_claim / repro_measured -> the aligned CLAIM vs MEASURED lines+#+# Environment knobs (all optional):+# XERJ_BIN path to a built server binary (default: engine/target/release/xerj)+# PORT force a specific base port (still must be free)+# KEEP=1 keep the throwaway data dir + server log after the run+# ─────────────────────────────────────────────────────────────────────────────++# Repo root from THIS file's location (scripts/bench/lib/repro.sh -> repo root),+# never from the caller's — sourcing contexts vary.+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"+export REPO_ROOT++# ── binary ──────────────────────────────────────────────────────────────────+repro_find_binary() {+ if [ -n "${XERJ_BIN:-}" ]; then+ if [ ! -x "$XERJ_BIN" ]; then+ echo "repro: XERJ_BIN=$XERJ_BIN is not executable" >&2+ return 1+ fi+ export XERJ_BIN+ return 0+ fi+ local cand+ for cand in \+ "$REPO_ROOT/engine/target/release/xerj" \+ "$REPO_ROOT/target/release/xerj"; do+ if [ -x "$cand" ]; then+ export XERJ_BIN="$cand"+ return 0+ fi+ done+ echo "repro: no server binary found — build one first:" >&2+ echo " cd engine && cargo build --release -p xerj-server" >&2+ echo " or point XERJ_BIN at an existing binary." >&2+ return 1+}++# ── ports (private 93xx only) ───────────────────────────────────────────────+repro_port_free() {+ python3 - "$1" <<'PY'+import socket, sys+base = int(sys.argv[1])+for p in (base, base + 1, base + 2):+ s = socket.socket()+ try:+ s.bind(("127.0.0.1", p))+ except OSError:+ sys.exit(1)+ finally:+ s.close()+PY+}++repro_free_port() {+ local min="${1:-9310}" p+ if [ -n "${PORT:-}" ]; then+ if repro_port_free "$PORT"; then+ REPRO_PORT="$PORT"+ return 0+ fi+ echo "repro: PORT=$PORT (or its +1/+2 neighbours) is busy" >&2+ return 1+ fi+ for p in $(seq "$min" $((min + 19))); do+ if repro_port_free "$p"; then+ REPRO_PORT="$p"+ return 0+ fi+ done+ echo "repro: no free port in $min..$((min + 19))" >&2+ return 1+}++# ── node lifecycle ──────────────────────────────────────────────────────────+# repro_node_start <config.toml|-> [env VAR=VAL ...] — env pairs come as+# VAR=VAL words after the config path ("-" for none). The node is started+# detached with its own session, logs into $REPRO_LOG, data in $REPRO_DATA.+repro_node_start() {+ local config="$1"+ shift+ local env_pair+ REPRO_DATA="$(mktemp -d /tmp/xerj-bench.XXXXXX)"+ REPRO_LOG="$REPRO_DATA/server.log"+ export REPRO_DATA REPRO_LOG+ local args=("$XERJ_BIN" --insecure --port "$REPRO_PORT" --data-dir "$REPRO_DATA")+ [ "$config" != "-" ] && args+=(-c "$config")+ local envs=()+ for env_pair in "$@"; do envs+=("$env_pair"); done+ # Direct child of this shell (no setsid/pgrep — neither is guaranteed in a+ # minimal sandbox), log to file, pid remembered for the EXIT trap.+ env "${envs[@]}" "${args[@]}" >"$REPRO_LOG" 2>&1 &+ REPRO_PID=$!+ export REPRO_PID+ sleep 0.4+ if ! kill -0 "$REPRO_PID" 2>/dev/null; then+ echo "repro: node failed to start; log:" >&2+ tail -20 "$REPRO_LOG" >&2 || true+ return 1+ fi+}++repro_node_wait() {+ local i+ for i in $(seq 1 150); do+ if curl -sf -o /dev/null "http://127.0.0.1:$REPRO_PORT/"; then+ return 0+ fi+ if ! kill -0 "$REPRO_PID" 2>/dev/null; then+ echo "repro: node died while waiting; log:" >&2+ tail -20 "$REPRO_LOG" >&2 || true+ return 1+ fi+ sleep 0.2+ done+ echo "repro: node did not answer on :$REPRO_PORT within 30s" >&2+ return 1+}++repro_node_stop() {+ if [ -n "${REPRO_PID:-}" ]; then+ kill "$REPRO_PID" 2>/dev/null || true+ local i+ for i in $(seq 1 50); do+ kill -0 "$REPRO_PID" 2>/dev/null || break+ sleep 0.1+ done+ kill -9 "$REPRO_PID" 2>/dev/null || true+ fi+ if [ -n "${REPRO_DATA:-}" ] && [ "${KEEP:-0}" != "1" ]; then+ rm -rf "$REPRO_DATA"+ elif [ -n "${REPRO_DATA:-}" ]; then+ echo "repro: KEEP=1 — data dir and log kept at $REPRO_DATA" >&2+ fi+}++# ── output helpers ──────────────────────────────────────────────────────────+repro_claim() {+ printf ' CLAIMED %s\n' "$1"+}++repro_measured() {+ printf ' MEASURED %s\n' "$1"+}++repro_note() {+ printf ' note %s\n' "$1"+}++repro_header() {+ printf '\n== %s ==\n' "$1"+}++repro_env_box() {+ # what this run used — so a stranger can tell whether their numbers compare+ printf '\nrun environment\n'+ printf ' binary %s (%s)\n' "$XERJ_BIN" "$("$XERJ_BIN" --version | head -1)"+ printf ' port %s (private, throwaway data dir)\n' "$REPRO_PORT"+ printf ' host %s\n' "$(uname -srm 2>/dev/null || echo unknown)"+ printf ' nproc %s\n' "$(nproc 2>/dev/null || echo '?')"+ printf ' date %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"+}diff --git a/scripts/bench/query-latency-size0.sh b/scripts/bench/query-latency-size0.shnew file mode 100755index 000000000..50a01d93b--- /dev/null+++ b/scripts/bench/query-latency-size0.sh@@ -0,0 +1,172 @@+#!/usr/bin/env bash+# ─────────────────────────────────────────────────────────────────────────────+# query-latency-size0.sh — one-command reproduction of the query-latency+# headline ("server round-trip, size:0 over 300k docs, keep-alive client").+#+# CLAIMED 0.126 ms average round-trip for a size:0 search over 300k docs+# through Node's core http keep-alive client, against 1.61 ms for+# the SAME request through Node's global fetch (undici) — the+# ~1.5 ms is client overhead that swamps sub-millisecond server+# times. demo/playbooks/SCORECARD.md methodology note and+# demo/playbooks/bench-matrix.mjs:47-58; quoted on+# landing/benchmarks/elasticsearch.html ("Server round-trip,+# size:0 over 300k docs, keep-alive client: 0.126 ms, 2.2x").+#+# What this script does: starts a throwaway node on a PRIVATE port (93xx)+# with the query cache off (XERJ_DISABLE_QUERY_CACHE=1 — the scorecard's+# uncached-execution rule), indexes 300,000 synthetic telemetry docs, force-+# merges to one segment and settles 3 s (the scorecard's read-index-state+# rule), then drives the SAME size:0 body through BOTH clients with the+# harness's own open-loop pacer (fixed 200 req/s cadence, sleep+spin to the+# intended instant, hot event loop; service time + scheduling backlog, i.e.+# coordinated-omission corrected).+#+# Usage:+# bash scripts/bench/query-latency-size0.sh+# Knobs: DOCS=300000 ITERS=1200 RATE=200 PORT=<free 93xx port> XERJ_BIN=... KEEP=1+# Requires: node >= 18, curl, python3, a built server binary.+# ─────────────────────────────────────────────────────────────────────────────+set -euo pipefail++HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+# shellcheck source=lib/repro.sh+source "$HERE/lib/repro.sh"++DOCS="${DOCS:-300000}"+BATCH="${BATCH:-10000}"+ITERS="${ITERS:-1200}"+RATE="${RATE:-200}"+WARMUP="${WARMUP:-60}"+CORPUS="$REPO_ROOT/demo/data/extras/chat-events.ndjson"++repro_find_binary+repro_free_port 9322++repro_header "one-command repro: size:0 query latency over ${DOCS} docs"+repro_claim "0.126 ms avg, size:0 over 300k, Node http keep-alive client (scorecard run 2026-07)"+repro_claim "1.61 ms avg for the SAME request through Node's global fetch/undici — client overhead, not server time"+repro_note "this run measures the XERJ side only; the 2.2x column needs ES (demo/playbooks/bench-matrix.mjs)"++if [ ! -f "$CORPUS" ]; then+ echo "corpus $CORPUS missing — run scripts/bench/ingest-docs-per-second.sh's generator or restore demo/data/extras/" >&2+ exit 1+fi++repro_node_start - XERJ_DISABLE_QUERY_CACHE=1+trap repro_node_stop EXIT+repro_node_wait++BASE="http://127.0.0.1:$REPRO_PORT"+IDX="bench-lat"++curl -sf -XDELETE "$BASE/$IDX" >/dev/null 2>&1 || true+curl -sf -XPUT "$BASE/$IDX" -H 'content-type: application/json' --data-binary @- >/dev/null <<'JSON'+{ "mappings": { "properties": {+ "@timestamp": { "type": "date" },+ "model": { "type": "keyword" }, "intent": { "type": "keyword" }, "status": { "type": "keyword" },+ "tenant": { "type": "keyword" }, "top_doc": { "type": "keyword" }, "cache_hit": { "type": "boolean" },+ "prompt_tokens": { "type": "integer" }, "context_tokens": { "type": "integer" }, "completion_tokens": { "type": "integer" },+ "latency_ms": { "type": "integer" }, "cost_usd": { "type": "double" }+} } }+JSON++# ── load the corpus with curl _bulk (same client the harness uses for load) ─+SCRATCH="$(mktemp -d /tmp/xerj-bench-lat.XXXXXX)"+trap 'rm -rf "$SCRATCH"; repro_node_stop' EXIT+python3 - "$CORPUS" "$SCRATCH" "$DOCS" "$BATCH" <<'PY'+import os, sys+corpus, scratch, total, batch = sys.argv[1], sys.argv[2], int(sys.argv[3]), int(sys.argv[4])+raw = open(corpus).read().splitlines()+n, written, b = len(raw), 0, 0+while written < total:+ lines = []+ for _ in range(min(batch, total - written)):+ lines.append('{"index":{}}'); lines.append(raw[written % n]); written += 1+ open(os.path.join(scratch, f"batch_{b:04d}.ndjson"), "w").write("\n".join(lines) + "\n")+ b += 1+PY+t0=$(date +%s)+for f in "$SCRATCH"/batch_*.ndjson; do+ curl -s -XPOST "$BASE/$IDX/_bulk" -H 'content-type: application/x-ndjson' --data-binary @"$f" >/dev/null+done+curl -sf -XPOST "$BASE/$IDX/_refresh" >/dev/null+count="$(curl -sf "$BASE/$IDX/_count" | python3 -c 'import json,sys; print(json.load(sys.stdin)["count"])')"+echo "indexed $count docs in $(( $(date +%s) - t0 ))s"++# read-index state rule from the harness: force-merge to one segment, settle 3s+curl -sf -XPOST "$BASE/$IDX/_forcemerge?max_num_segments=1" >/dev/null+sleep 3++# ── the measurement: same body through BOTH clients, harness pacer ─────────+RESULT="$(node - "$BASE" "$IDX" "$ITERS" "$RATE" "$WARMUP" <<'NODE'+// Mirrors demo/playbooks/bench-matrix.mjs timed(): open-loop fixed cadence,+// sleep+spin pacer (SPIN_MS=1.5) to the intended instant, hot event loop so+// responses are processed the moment they arrive, latency = service time ++// scheduling backlog (end - min(intended, actualStart)).+const http = require('node:http');+const [base, idx, ITERS, RATE, WARMUP] = [process.argv[2], process.argv[3], +process.argv[4], +process.argv[5], +process.argv[6]];+const SPIN_MS = 1.5;+const PATH = `/${idx}/_search?request_cache=false`;+const BODY = JSON.stringify({ query: { match_all: {} }, size: 0, track_total_hits: true });++const agent = new http.Agent({ keepAlive: true, maxSockets: 256 });+const keepAliveReq = () => new Promise((resolve, reject) => {+ const r = http.request({ hostname: '127.0.0.1', port: +base.split(':')[2], path: PATH,+ method: 'POST', headers: { 'content-type': 'application/json', 'content-length': Buffer.byteLength(BODY) }, agent },+ (res) => { let n = 0; res.on('data', (c) => { n += c.length; }); res.on('end', () => resolve(res.statusCode)); });+ r.on('error', reject); r.end(BODY);+});+const fetchReq = async () => { const r = await fetch(base + PATH, { method: 'POST', headers: { 'content-type': 'application/json' }, body: BODY }); await r.text(); return r.status; };++function pct(a, p) { const s = [...a].sort((x, y) => x - y); return s[Math.min(s.length - 1, Math.floor((p / 100) * s.length))]; }++async function measure(send) {+ for (let i = 0; i < WARMUP; i++) await send();+ const lat = new Array(ITERS); const tasks = new Array(ITERS);+ let hot = true; (function loop() { if (hot) setImmediate(loop); })();+ const t0 = performance.now();+ for (let i = 0; i < ITERS; i++) {+ const intended = t0 + (i / RATE) * 1000;+ tasks[i] = (async () => {+ const wait = intended - performance.now();+ if (wait > SPIN_MS) await new Promise((r) => setTimeout(r, wait - SPIN_MS));+ while (performance.now() < intended) { /* spin */ }+ const start = performance.now();+ await send();+ lat[i] = performance.now() - Math.min(intended, start);+ })();+ }+ await Promise.all(tasks); hot = false;+ const mean = lat.reduce((a, b) => a + b, 0) / lat.length;+ return { mean, p50: pct(lat, 50), p99: pct(lat, 99), max: Math.max(...lat) };+}++(async () => {+ const ka = await measure(keepAliveReq);+ const fe = await measure(fetchReq);+ agent.destroy();+ const out = {+ keepalive: ka, fetch: fe, iters: ITERS, rate: RATE,+ ratio_fetch_over_keepalive: +(fe.mean / ka.mean).toFixed(2),+ };+ console.log(JSON.stringify(out));+})().catch((e) => { console.error(String(e)); process.exit(1); });+NODE+)"++RESULT_FILE="$SCRATCH/latency.json"+printf '%s\n' "$RESULT" >"$RESULT_FILE"+python3 - "$RESULT_FILE" <<'PY'+import json, sys+r = json.load(open(sys.argv[1]))+ka, fe = r["keepalive"], r["fetch"]+print()+print(f' MEASURED keep-alive client : {ka["mean"]:.3f} ms avg (p50 {ka["p50"]:.3f} / p99 {ka["p99"]:.3f} / max {ka["max"]:.3f})')+print(f' MEASURED fetch/undici : {fe["mean"]:.3f} ms avg (p50 {fe["p50"]:.3f} / p99 {fe["p99"]:.3f} / max {fe["max"]:.3f})')+print(f' MEASURED client overhead : {fe["mean"] - ka["mean"]:.3f} ms ({r["ratio_fetch_over_keepalive"]}x) — the published point: the client, not the server')+print(f' note {r["iters"]} timed iterations at {r["rate"]} req/s open-loop, request_cache=false, query cache off, 1 merged segment')+print(' note absolute ms is hardware-bound — THIS box is slower than the scorecard box; what')+print(' reproduces anywhere is the ordering and gap (keep-alive << fetch), not the exact value')+PY+repro_env_box+repro_note "teardown: node stopped, throwaway data dir removed (KEEP=1 to keep)"\ No newline at end of filediff --git a/scripts/bench/rerank-stage.sh b/scripts/bench/rerank-stage.shnew file mode 100755index 000000000..3df05d17d--- /dev/null+++ b/scripts/bench/rerank-stage.sh@@ -0,0 +1,343 @@+#!/usr/bin/env bash+# ─────────────────────────────────────────────────────────────────────────────+# rerank-stage.sh — one-command reproduction of the Jev rerank-stage numbers.+#+# The stage's RANKING QUALITY is not claimable — XERJ has never run the real+# Jev model (no provider key; docs/RERANK.md says so on the tin, and the BEIR+# figures for Jev/Voyage/Cohere are that other project's, marked "not run by+# us"). What public copy DOES claim about this stage is its cost and failure+# contract, and those are measurable against a local test double with zero+# network egress:+#+# CLAIMED "30 documents per provider call ... window: 35 is two calls+# (30 + 5), and window: 300, the maximum, is ten"+# "8 calls in flight by default, 16 at most"+# "There is no verdict cache: three page requests (from 0, 10, 20)+# over one 30-document window are three provider calls and 90 paid+# judgements, not 30"+# "a slow provider -> HTTP 200 with the engine's order and+# _rerank.applied:false"+# "no key -> 503 ... all rerank_exception with no hits"+# — docs/RERANK.md (Cost and concurrency facts / Failure policy),+# restated on landing/benchmarks/index.html and landing/llms.txt.+#+# What this script does: starts a Jev test double (lib/jev_stub.py — the same+# word-overlap verdicts as the engine's own double) on a PRIVATE port, boots a+# node pointed at it with a dummy key, indexes 320 generated docs, and measures+# each published fact end-to-end through the ES wire. Every provider call the+# node makes lands in a log this script reads back: call counts, per-call+# document counts, max in-flight concurrency, and judgements billed.+#+# Usage:+# bash scripts/bench/rerank-stage.sh+# Knobs: PORT=<free 93xx port> STUB_PORT=<free 93xx port> XERJ_BIN=... KEEP=1+# Requires: python3, curl, a built server binary. No network egress: the+# provider endpoint is 127.0.0.1.+# ─────────────────────────────────────────────────────────────────────────────+set -euo pipefail++HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+# shellcheck source=lib/repro.sh+source "$HERE/lib/repro.sh"++repro_find_binary+repro_free_port 9340+NODE_PORT="$REPRO_PORT"+repro_free_port 9350+STUB_PORT="$REPRO_PORT"++repro_header "one-command repro: Jev rerank-stage cost/failure contract"+repro_claim "30 docs per provider call — window 35 -> 2 calls (30+5), window 300 -> 10 (docs/RERANK.md)"+repro_claim "no verdict cache — three page requests over one window = 3 calls, 90 paid judgements"+repro_claim "8 calls in flight by default, 16 at most; slow provider -> HTTP 200, applied:false, engine order"+repro_claim "no key -> HTTP 503, rerank_exception, no hits; ranking quality with the real model: NOT claimed"+repro_note "verdicts come from lib/jev_stub.py (word overlap), not the real model — the quality rows stay unreproducible by design"++WORK="$(mktemp -d /tmp/xerj-bench-rerank.XXXXXX)"+STUB_LOG="$WORK/calls.jsonl"+STUB_CONTROL="$WORK/control.json"+CONFIG="$WORK/rerank.toml"+KEYLESS_PID=""+STUB_PID=""+cleanup() {+ for pid in "$STUB_PID" "$KEYLESS_PID"; do+ [ -n "$pid" ] && kill "$pid" 2>/dev/null || true+ done+ for pid in "$STUB_PID" "$KEYLESS_PID"; do+ [ -n "$pid" ] || continue+ for _ in $(seq 1 30); do kill -0 "$pid" 2>/dev/null || break; sleep 0.1; done+ done+ [ -n "${KEYLESS_DATA:-}" ] && rm -rf "$KEYLESS_DATA"+ repro_node_stop+ rm -rf "$WORK"+}+trap cleanup EXIT++# ── the provider test double ────────────────────────────────────────────────+python3 "$HERE/lib/jev_stub.py" --port "$STUB_PORT" --log "$STUB_LOG" --control "$STUB_CONTROL" &+STUB_PID=$!+for _ in $(seq 1 50); do+ curl -sf "http://127.0.0.1:$STUB_PORT/health" >/dev/null 2>&1 && break+ sleep 0.1+done++# ── the node, pointed at the double ─────────────────────────────────────────+# repro_node_start reads the REPRO_PORT global — restore the node's port after+# the stub's allocation changed it.+REPRO_PORT="$NODE_PORT"+cat >"$CONFIG" <<TOML+[rerank]+enabled = true+api_key = "stub-key-for-bench-only"+endpoint = "http://127.0.0.1:$STUB_PORT/v1/systemone"+TOML+repro_node_start "$CONFIG"+repro_node_wait+BASE="http://127.0.0.1:$NODE_PORT"+IDX="bench-rerank"++curl -sf -XDELETE "$BASE/$IDX" >/dev/null 2>&1 || true+curl -sf -XPUT "$BASE/$IDX" -H 'content-type: application/json' \+ --data-binary '{"mappings":{"properties":{"title":{"type":"text"},"body":{"type":"text"}}}}' >/dev/null++# ── measurement driver (writes $WORK/results.json, prints nothing) ──────────+python3 - "$BASE" "$IDX" "$STUB_LOG" "$STUB_CONTROL" "$WORK/results.json" <<'PY'+import http.client, json, subprocess, sys++base, idx, log_path, control_path, out_path = sys.argv[1:6]+host, port = "127.0.0.1", int(base.rsplit(":", 1)[1])+QUESTION = "vitamin d supplementation bone density"+WORDS = QUESTION.split()++def call(method, path, body=None):+ c = http.client.HTTPConnection(host, port, timeout=40)+ payload = json.dumps(body) if body is not None else None+ headers = {"content-type": "application/json"} if payload else {}+ c.request(method, path, body=payload, headers=headers)+ r = c.getresponse()+ raw = r.read()+ c.close()+ try:+ return r.status, json.loads(raw)+ except Exception:+ return r.status, raw.decode(errors="replace")[:400]++def set_sleep(ms):+ json.dump({"sleep_ms": ms}, open(control_path, "w"))++def log_lines():+ try:+ return [json.loads(l) for l in open(log_path) if l.strip()]+ except FileNotFoundError:+ return []++def calls_since(marker):+ return log_lines()[marker:]++def mark():+ return len(log_lines())++def max_in_flight(calls):+ events = []+ for c in calls:+ events += [(c["start"], 1), (c["end"], -1)]+ events.sort()+ cur = best = 0+ for _, d in events:+ cur += d+ best = max(best, cur)+ return best++# corpus: 320 docs whose word overlap with the question is a deterministic ramp+# (320 > the 300-doc max window so the window arithmetic is measured at its+# published maximum, not truncated by the corpus size)+nd = ""+for i in range(320):+ keep = (320 - i) % 6 # 0..5 of the 5 question words+ body = " ".join(WORDS[:keep]) + f" study report {i} filler text about trials"+ nd += '{"index":{}}\n' + json.dumps({"title": f"Trial report {i}", "body": body}) + "\n"+open(f"{out_path}.ndjson", "w").write(nd)+subprocess.run(["curl", "-s", "-XPOST", f"{base}/{idx}/_bulk",+ "-H", "content-type: application/x-ndjson",+ "--data-binary", f"@{out_path}.ndjson"], check=True,+ stdout=subprocess.DEVNULL)+call("POST", f"/{idx}/_refresh")+status, cnt = call("GET", f"/{idx}/_count")+indexed = (cnt or {}).get("count")+if indexed != 320:+ # a silent ingest failure would make every later check vacuously pass on+ # an empty index — refuse to measure instead+ json.dump([("corpus ingest", "320 docs", f"{indexed} docs indexed", False)], open(out_path, "w"))+ raise SystemExit(0)++def search(rerank, size=10, frm=0):+ # first stage is match_all on purpose: the question lives in rerank.query,+ # and a match on a term the analyzer does not return hits for would make+ # every check below vacuously pass on an empty hit list.+ body = {"query": {"match_all": {}}, "size": size, "from": frm}+ if rerank is not None:+ body["rerank"] = rerank+ return call("POST", f"/{idx}/_search", body)++results = [] # rows: (claim label, claimed value, measured value, ok|None)++# A — configuration facts as the live node reports them+status, cfg = call("GET", "/_xerj/rerank")+if status == 200:+ d, lim = cfg.get("defaults", {}), cfg.get("limits", {})+ results.append(("defaults window/batch/max_concurrency", "30 / 30 / 8",+ f"{d.get('window')} / {d.get('batch')} / {d.get('max_concurrency')}",+ (d.get("window"), d.get("batch"), d.get("max_concurrency")) == (30, 30, 8)))+ results.append(("limits max_docs_per_call/max_window/max_concurrency", "30 / 300 / 16",+ f"{lim.get('max_docs_per_call')} / {lim.get('max_window')} / {lim.get('max_concurrency')}",+ (lim.get("max_docs_per_call"), lim.get("max_window"), lim.get("max_concurrency")) == (30, 300, 16)))+else:+ results.append(("GET /_xerj/rerank", "200", f"HTTP {status}", False))++# B — window 35 -> 2 calls (30 + 5)+m = mark(); search({"window": 35, "query": QUESTION}, size=5)+c = calls_since(m)+results.append(("window 35 -> provider calls / docs per call", "2 calls: 30 + 5",+ f"{len(c)} calls: {' + '.join(str(x['ndocs']) for x in c)}",+ sorted((x["ndocs"] for x in c), reverse=True) == [30, 5]))++# C — window 300 (the maximum) -> 10 calls x 30+m = mark(); search({"window": 300, "query": QUESTION}, size=5)+c = calls_since(m)+results.append(("window 300 -> provider calls / docs per call", "10 calls of 30",+ f"{len(c)} calls: {' + '.join(str(x['ndocs']) for x in c)}",+ len(c) == 10 and all(x["ndocs"] == 30 for x in c)))++# D — no verdict cache: three page requests over one 30-doc window+m = mark()+for frm in (0, 10, 20):+ search({"window": 30, "query": QUESTION}, size=10, frm=frm)+c = calls_since(m)+judged = sum(x["ndocs"] for x in c)+results.append(("3 page requests (from 0/10/20) -> calls / paid judgements",+ "3 calls, 90 judgements — not 30",+ f"{len(c)} calls, {judged} judgements",+ len(c) == 3 and judged == 90))++# E — slow provider: degrade on deadline, HTTP 200, the engine's order back+_, plain = search(None, size=10)+engine_order = [h["_id"] for h in plain["hits"]["hits"]]+if len(engine_order) < 10:+ json.dump([("first-stage query", ">= 10 hits to judge",+ f"{len(engine_order)} hits — corpus or query wrong", False)], open(out_path, "w"))+ raise SystemExit(0)+set_sleep(2000)+status, deg = search({"window": 30, "query": QUESTION, "timeout_ms": 200}, size=10)+set_sleep(0)+rr = deg.get("_rerank", {}) if isinstance(deg, dict) else {}+deg_order = [h["_id"] for h in deg.get("hits", {}).get("hits", [])] if isinstance(deg, dict) else []+ok = (status == 200 and rr.get("applied") is False and rr.get("score_kind") == "engine"+ and deg_order == engine_order)+results.append(("slow provider (2s) with timeout_ms=200",+ "HTTP 200, applied:false, score_kind:engine, engine order, same totals",+ f"HTTP {status}, applied:{rr.get('applied')}, score_kind:{rr.get('score_kind')}, "+ f"order {'== engine' if deg_order == engine_order else 'DIFFERS'}, reason: {str(rr.get('reason'))[:48]}",+ ok))++# F — verdicts replace _score, order by probability, min_score prunes+status, applied = search({"window": 40, "query": QUESTION, "min_score": 0.5}, size=10)+rr = applied.get("_rerank", {})+hits = applied["hits"]["hits"]+scores = [h["_score"] for h in hits]+sorted_desc = scores == sorted(scores, reverse=True)+ok = (status == 200 and rr.get("score_kind") == "probability" and sorted_desc+ and scores and min(scores) >= 0.5 and (rr.get("pruned_below_min_score") or 0) > 0)+results.append(("verdicts replace _score (window 40, min_score 0.5)",+ "score_kind:probability, hits ordered by probability, all >= min_score, prunes counted",+ f"score_kind:{rr.get('score_kind')}, judged:{rr.get('judged')}, "+ f"pruned_below_min_score:{rr.get('pruned_below_min_score')}, "+ f"order {'desc' if sorted_desc else 'NOT SORTED'}, min _score {min(scores):.3f}" if scores else "no hits",+ ok))++# G — in-flight ceiling: 8 by default, a request can raise it to at most 16+set_sleep(400)+m = mark(); search({"window": 300, "query": QUESTION, "timeout_ms": 30000}, size=5)+c8 = calls_since(m)+m = mark(); search({"window": 300, "query": QUESTION, "timeout_ms": 30000, "max_concurrency": 16}, size=5)+c16 = calls_since(m)+set_sleep(0)+f8, f16 = max_in_flight(c8), max_in_flight(c16)+results.append(("max provider calls in flight, window 300 (10 calls)",+ "8 by default, 16 at most",+ f"default: {f8} in flight over {len(c8)} calls; max_concurrency=16: {f16} in flight over {len(c16)} calls",+ 1 <= f8 <= 8 and 8 < f16 <= 16))++json.dump(results, open(out_path, "w"))+PY++# ── the keyless node: no key configured -> 503, no hits ─────────────────────+KEYLESS_DATA="$(mktemp -d /tmp/xerj-bench-keyless.XXXXXX)"+repro_free_port 9360+KEYLESS_PORT="$REPRO_PORT"+REPRO_PORT="$NODE_PORT" # keep the env box honest: this repro's main node+env "$XERJ_BIN" --insecure --port "$KEYLESS_PORT" --data-dir "$KEYLESS_DATA" >"$KEYLESS_DATA/log" 2>&1 &+KEYLESS_PID=$!+for _ in $(seq 1 100); do+ curl -sf "http://127.0.0.1:$KEYLESS_PORT/" >/dev/null 2>&1 && break+ sleep 0.2+done+# the keyless node needs the index too — a 404 would test routing, not rerank+curl -sf -XPUT "http://127.0.0.1:$KEYLESS_PORT/$IDX" -H 'content-type: application/json' \+ --data-binary '{"mappings":{"properties":{"title":{"type":"text"},"body":{"type":"text"}}}}' >/dev/null+printf '{"index":{}}\n{"title":"Trial report 0","body":"trial filler text"}\n' \+ | curl -sf -XPOST "http://127.0.0.1:$KEYLESS_PORT/$IDX/_bulk" \+ -H 'content-type: application/x-ndjson' --data-binary @- >/dev/null+curl -sf -XPOST "http://127.0.0.1:$KEYLESS_PORT/$IDX/_refresh" >/dev/null++KEYLESS_OUT="$(python3 - "http://127.0.0.1:$KEYLESS_PORT" "$IDX" <<'PY'+import http.client, json, sys+base, idx = sys.argv[1], sys.argv[2]+host, port = "127.0.0.1", int(base.rsplit(":", 1)[1])+c = http.client.HTTPConnection(host, port, timeout=15)+body = {"query": {"match": {"body": "trial"}}, "size": 5,+ "rerank": {"query": "vitamin d supplementation bone density"}}+c.request("POST", f"/{idx}/_search", body=json.dumps(body), headers={"content-type": "application/json"})+r = c.getresponse()+raw = r.read().decode(errors="replace")+c.close()+try:+ j = json.loads(raw)+ hits = len((j.get("hits") or {}).get("hits") or [])+ err = j.get("error")+ if isinstance(err, dict):+ etype = err.get("type") or (err.get("caused_by") or {}).get("type") or ""+ else:+ etype = str(err)[:80]+ print(json.dumps({"status": r.status, "hits": hits, "error_type": etype, "raw_type": raw[:0]}))+except Exception:+ print(json.dumps({"status": r.status, "raw": raw[:200]}))+PY+)"++# ── verdicts ────────────────────────────────────────────────────────────────+python3 - "$WORK/results.json" "$KEYLESS_OUT" <<'PY'+import json, sys+rows = json.load(open(sys.argv[1]))+keyless = json.loads(sys.argv[2])+print()+fails = 0+for label, claimed, measured, ok in rows:+ verdict = "PASS" if ok else ("FAIL" if ok is False else "----")+ fails += 1 if ok is False else 0+ print(f" {verdict} {label}")+ print(f" claimed : {claimed}")+ print(f" measured : {measured}")+kl_ok = keyless.get("status") == 503 and keyless.get("hits") == 0 and "rerank_exception" in str(keyless.get("error_type", ""))+fails += 0 if kl_ok else 1+print(f" {'PASS' if kl_ok else 'FAIL'} no key configured -> refusal, not silence")+print(f" claimed : HTTP 503, rerank_exception, no hits")+print(f" measured : HTTP {keyless.get('status')}, hits {keyless.get('hits')}, error type {keyless.get('error_type') or keyless.get('raw')}")+print()+print(f" contract checks: {len(rows) + 1 - fails}/{len(rows) + 1} PASS "+ "(deterministic contract — these assertions are the reproduction)")+print(' note the quality rows (nDCG 0.768/0.358 for Jev, etc.) are hev/jev-rerank\'s numbers,')+print(' marked "not run by us" in docs/RERANK.md — no harness here can or should reproduce them')+PY++repro_env_box+repro_note "teardown: both nodes + the stub stopped, throwaway dirs removed (KEEP=1 to keep)"