Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

benchmarks: FiQA (BEIR) and AG News (HF) — two public datasets added with provenance, runs and validation

xerj-org96f90107552b
diff --git a/benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa/README.md b/benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa/README.md
new file mode 100644
index 000000000..a708e458e
--- /dev/null
+++ b/benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa/README.md
@@ -0,0 +1,69 @@
+# FiQA rerank full run — 2026-09-20 (648 queries, 3 repeats)
+
+Third BEIR dataset through the same harness as
+`../2026-09-20-rerank-full/` (SciFact + NFCorpus), added so the measured-Jev
+board covers a finance-domain QA corpus with sparse relevance judgements —
+the regime where a general-purpose judge is most often wrong-when-confident.
+
+## Dataset and provenance
+
+**FiQA-2018**, the BEIR `test` split: 57,638 corpus documents, 648 judged
+test queries, graded qrels — the same files every BEIR paper uses.
+
+- Task page: https://arxiv.org/abs/2104.08663 (BEIR, Thakur et al., 2021)
+- Canonical BEIR distribution: the `fiqa` corpus/queries/qrels from
+ https://github.com/beir-cellar/beir (MIT), mirrored at
+ `https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/fiqa.zip`
+- SHA of the zip we fetched is recorded in the local run log; nothing here
+ was re-tokenised or re-cut — corpus.jsonl / queries.jsonl / qrels/test.tsv
+ are the archive's files verbatim.
+
+**Harness validation before any model was called:** local BM25 nDCG@10 on the
+648 test queries came out **0.2382**, against BEIR's published BM25 figure of
+**0.236** for FiQA (BM25 from the BEIR paper's table, reproduced by many
+independent runs since). Within 0.002 — the harness scores the way the
+reference scores. The same BM25 figure reproduced identically (0.2382, three
+times, and again bit-for-bit after the WAND fast path landed in the engine,
+cf6a9989).
+
+## Headline numbers
+
+| Dataset | BM25 nDCG@10 | Jev median nDCG@10 | W / L / T | input tokens | cost | wall p50 |
+|---|---:|---:|---|---:|---:|---:|
+| FiQA (648 q) | 0.2382 | 0.3638 | 249 / 57 / 342 | 15,178,758 | $0.6375 | 1.23 s/q |
+
+Cost at $0.042/Mtok input-only. Zero failed requests; zero queries with an
+empty BM25 shortlist (FiQA's queries are vocabulary-rich, unlike NFCorpus's
+25 empties). Model pinned `jev-1.13.0`, one noul per document, medians over
+three shuffled repeats — the fixed request shape from the pilot's autopsy.
+
+FiQA is the judge's **biggest lift of the three datasets** (+0.126 over BM25;
+SciFact +0.084, NFCorpus +0.030) and the reason is visible in the shortlist:
+finance QA relevance is phrase-level ("business expense on a business trip"),
+and BM25's word-overlap ranking leaves real relevant documents far below
+rank 10 where the judge, reading the document, recovers them.
+
+## The finding: the confidence is worse than useless as a threshold here
+
+ECE **0.3109** over 19,440 (probability, relevance) pairs — three times
+SciFact's 0.1023. The curve is still monotone (the judge discriminates), but
+the top bin is the whole story: **mean confidence 0.93, actual relevance
+0.34**. Bin-by-bin curve in `fiqa-full-run.json`. On a sparse-judgement
+corpus a Jev probability near 1.0 still means "about a one-in-three chance
+the qrels call this relevant" — which is exactly the failure mode of
+thresholding a raw probability. The ordering is worth +0.126 nDCG; the number
+itself must never gate.
+
+## Reproduce
+
+```sh
+# corpus: fetch the BEIR fiqa.zip, extract corpus.jsonl / queries.jsonl / qrels/test.tsv
+python3 load.py fiqa/corpus.jsonl fiqa # this repo's loader, title+text fields
+XERJ_URL=http://localhost:<port> python3 raw_bench_full.py 648 fiqa
+# -> fiqa-full-run.json (aggregates, reliability curve, per-query numbers)
+```
+
+Run log: repeat 1 bm25=0.2382 jev=0.3648 · repeat 2 bm25=0.2382 jev=0.3622 ·
+repeat 3 bm25=0.2382 jev=0.3636 — the BM25 arm is deterministic across
+repeats (same engine, same index), the Jev arm wobbles ±0.0013, consistent
+with the provider non-determinism measured on the other two datasets.
diff --git a/benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa/fiqa-full-run.json b/benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa/fiqa-full-run.json
new file mode 100644
index 000000000..3da68d3aa
--- /dev/null
+++ b/benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa/fiqa-full-run.json
@@ -0,0 +1,2673 @@
+{
+ "dataset": "fiqa",
+ "queries": 648,
+ "empty_bm25": 0,
+ "fails": 0,
+ "bm25_ndcg10": 0.2382,
+ "jev_fixed_median_ndcg10": 0.3638,
+ "wins": 249,
+ "losses": 57,
+ "ties": 342,
+ "input_tokens": 15178758,
+ "output_tokens": 1038096,
+ "cost_usd": 0.6375,
+ "wall_s_p50": 1.23,
+ "wall_s_p95": 1.39,
+ "total_wall_min": 40.4,
+ "determinism": {
+ "pairs_compared": 900,
+ "max_abs_delta": 0.1,
+ "mean_abs_delta": 0.0148
+ },
+ "calibration": {
+ "n_pairs": 19440,
+ "bins": 10,
+ "ece": 0.3109,
+ "curve": {
+ "0": {
+ "mean_conf": 0.0455,
+ "acc": 0.0002,
+ "n": 5572
+ },
+ "1": {
+ "mean_conf": 0.1418,
+ "acc": 0.0003,
+ "n": 3005
+ },
+ "2": {
+ "mean_conf": 0.2434,
+ "acc": 0.0034,
+ "n": 2061
+ },
+ "3": {
+ "mean_conf": 0.3438,
+ "acc": 0.0033,
+ "n": 1529
+ },
+ "4": {
+ "mean_conf": 0.4445,
+ "acc": 0.0077,
+ "n": 1423
+ },
+ "5": {
+ "mean_conf": 0.5442,
+ "acc": 0.0071,
+ "n": 1271
+ },
+ "6": {
+ "mean_conf": 0.6451,
+ "acc": 0.0224,
+ "n": 1118
+ },
+ "7": {
+ "mean_conf": 0.7455,
+ "acc": 0.0408,
+ "n": 1224
+ },
+ "8": {
+ "mean_conf": 0.8452,
+ "acc": 0.1242,
+ "n": 1272
+ },
+ "9": {
+ "mean_conf": 0.9297,
+ "acc": 0.3378,
+ "n": 965
+ }
+ }
+ },
+ "per_query": {
+ "8": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "15": {
+ "bm25": 0.0,
+ "jev_median": 1.0
+ },
+ "18": {
+ "bm25": 0.0,
+ "jev_median": 0.6309
+ },
+ "26": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "34": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "42": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "56": {
+ "bm25": 0.301,
+ "jev_median": 0.6309
+ },
+ "68": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "89": {
+ "bm25": 0.2684,
+ "jev_median": 0.3422
+ },
+ "90": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "94": {
+ "bm25": 0.5,
+ "jev_median": 1.0
+ },
+ "98": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "104": {
+ "bm25": 0.3066,
+ "jev_median": 0.6131
+ },
+ "106": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "109": {
+ "bm25": 0.6309,
+ "jev_median": 1.0
+ },
+ "475": {
+ "bm25": 0.5,
+ "jev_median": 1.0
+ },
+ "503": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "504": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "515": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "529": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "547": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "549": {
+ "bm25": 0.0,
+ "jev_median": 1.0
+ },
+ "559": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "570": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "585": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "588": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "594": {
+ "bm25": 0.6131,
+ "jev_median": 0.2372
+ },
+ "603": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "604": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "620": {
+ "bm25": 0.1021,
+ "jev_median": 0.3392
+ },
+ "622": {
+ "bm25": 0.0,
+ "jev_median": 0.6131
+ },
+ "659": {
+ "bm25": 0.3785,
+ "jev_median": 0.3785
+ },
+ "672": {
+ "bm25": 0.1772,
+ "jev_median": 0.0
+ },
+ "684": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "687": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "689": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "691": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "699": {
+ "bm25": 0.0,
+ "jev_median": 1.0
+ },
+ "701": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "715": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "721": {
+ "bm25": 0.3869,
+ "jev_median": 1.0
+ },
+ "744": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "750": {
+ "bm25": 0.9197,
+ "jev_median": 0.8175
+ },
+ "753": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "766": {
+ "bm25": 0.0,
+ "jev_median": 0.4693
+ },
+ "776": {
+ "bm25": 0.2201,
+ "jev_median": 0.359
+ },
+ "810": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "813": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "849": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "852": {
+ "bm25": 0.8603,
+ "jev_median": 0.3836
+ },
+ "853": {
+ "bm25": 0.2044,
+ "jev_median": 0.3869
+ },
+ "858": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "859": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "864": {
+ "bm25": 0.3552,
+ "jev_median": 0.7751
+ },
+ "879": {
+ "bm25": 0.6309,
+ "jev_median": 1.0
+ },
+ "885": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "895": {
+ "bm25": 0.4307,
+ "jev_median": 1.0
+ },
+ "904": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "928": {
+ "bm25": 0.1772,
+ "jev_median": 0.6131
+ },
+ "929": {
+ "bm25": 0.0,
+ "jev_median": 1.0
+ },
+ "932": {
+ "bm25": 0.8316,
+ "jev_median": 0.6934
+ },
+ "939": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "945": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "957": {
+ "bm25": 0.4693,
+ "jev_median": 0.1672
+ },
+ "988": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "1074": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "1085": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "1090": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "1150": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "1157": {
+ "bm25": 0.6131,
+ "jev_median": 0.3869
+ },
+ "1159": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "1198": {
+ "bm25": 1.0,
+ "jev_median": 0.5
+ },
+ "1230": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "1281": {
+ "bm25": 0.0,
+ "jev_median": 1.0
+ },
+ "1284": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "1297": {
+ "bm25": 0.4307,
+ "jev_median": 1.0
+ },
+ "1306": {
+ "bm25": 0.0,
+ "jev_median": 0.3904
+ },
+ "1309": {
+ "bm25": 0.151,
+ "jev_median": 0.6367
+ },
+ "1310": {
+ "bm25": 0.4307,
+ "jev_median": 1.0
+ },
+ "1321": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "1322": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "1391": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "1393": {
+ "bm25": 0.0911,
+ "jev_median": 0.6448
+ },
+ "1415": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "1416": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "1441": {
+ "bm25": 0.5376,
+ "jev_median": 1.0
+ },
+ "1451": {
+ "bm25": 0.6309,
+ "jev_median": 0.6309
+ },
+ "1469": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "1530": {
+ "bm25": 0.1413,
+ "jev_median": 0.4693
+ },
+ "1670": {
+ "bm25": 0.7654,
+ "jev_median": 0.7039
+ },
+ "1676": {
+ "bm25": 0.5585,
+ "jev_median": 0.5654
+ },
+ "1736": {
+ "bm25": 0.1696,
+ "jev_median": 0.3392
+ },
+ "1748": {
+ "bm25": 0.4441,
+ "jev_median": 0.7654
+ },
+ "1783": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "1812": {
+ "bm25": 0.2044,
+ "jev_median": 0.2184
+ },
+ "1815": {
+ "bm25": 0.5,
+ "jev_median": 1.0
+ },
+ "1819": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "1824": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "1826": {
+ "bm25": 0.9829,
+ "jev_median": 0.9829
+ },
+ "1832": {
+ "bm25": 0.6309,
+ "jev_median": 0.4307
+ },
+ "1871": {
+ "bm25": 0.281,
+ "jev_median": 0.3633
+ },
+ "1877": {
+ "bm25": 1.0,
+ "jev_median": 0.5
+ },
+ "1889": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "1915": {
+ "bm25": 0.2641,
+ "jev_median": 0.6131
+ },
+ "1920": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "1933": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "1948": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "1994": {
+ "bm25": 0.107,
+ "jev_median": 0.5087
+ },
+ "2010": {
+ "bm25": 0.1815,
+ "jev_median": 0.5307
+ },
+ "2051": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2070": {
+ "bm25": 0.4693,
+ "jev_median": 0.4693
+ },
+ "2075": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2076": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2088": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2108": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2118": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2154": {
+ "bm25": 0.5,
+ "jev_median": 1.0
+ },
+ "2181": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2183": {
+ "bm25": 0.1909,
+ "jev_median": 0.1513
+ },
+ "2204": {
+ "bm25": 0.539,
+ "jev_median": 0.6479
+ },
+ "2264": {
+ "bm25": 0.3869,
+ "jev_median": 0.6131
+ },
+ "2296": {
+ "bm25": 0.1175,
+ "jev_median": 0.3833
+ },
+ "2306": {
+ "bm25": 0.1934,
+ "jev_median": 0.6131
+ },
+ "2316": {
+ "bm25": 0.3869,
+ "jev_median": 0.6131
+ },
+ "2318": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2330": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "2334": {
+ "bm25": 0.2346,
+ "jev_median": 0.7654
+ },
+ "2348": {
+ "bm25": 0.359,
+ "jev_median": 0.5638
+ },
+ "2376": {
+ "bm25": 0.4053,
+ "jev_median": 0.8359
+ },
+ "2383": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "2384": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2385": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "2388": {
+ "bm25": 0.3333,
+ "jev_median": 1.0
+ },
+ "2395": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2398": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2399": {
+ "bm25": 0.3333,
+ "jev_median": 1.0
+ },
+ "2400": {
+ "bm25": 0.2346,
+ "jev_median": 0.7039
+ },
+ "2407": {
+ "bm25": 0.3904,
+ "jev_median": 0.2463
+ },
+ "2416": {
+ "bm25": 0.3904,
+ "jev_median": 0.8319
+ },
+ "2423": {
+ "bm25": 0.4123,
+ "jev_median": 0.2918
+ },
+ "2443": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2445": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "2460": {
+ "bm25": 0.3378,
+ "jev_median": 0.7654
+ },
+ "2465": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2472": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2486": {
+ "bm25": 1.0,
+ "jev_median": 0.6309
+ },
+ "2498": {
+ "bm25": 0.0,
+ "jev_median": 0.5
+ },
+ "2513": {
+ "bm25": 0.0,
+ "jev_median": 0.6309
+ },
+ "2516": {
+ "bm25": 0.3904,
+ "jev_median": 0.5856
+ },
+ "2549": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2551": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2568": {
+ "bm25": 0.0867,
+ "jev_median": 0.2749
+ },
+ "2579": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2580": {
+ "bm25": 0.7654,
+ "jev_median": 0.7654
+ },
+ "2587": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "2589": {
+ "bm25": 0.8194,
+ "jev_median": 0.8711
+ },
+ "2590": {
+ "bm25": 0.1564,
+ "jev_median": 0.4693
+ },
+ "2593": {
+ "bm25": 0.4693,
+ "jev_median": 0.4693
+ },
+ "2598": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2648": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "2676": {
+ "bm25": 0.0,
+ "jev_median": 1.0
+ },
+ "2685": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2695": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "2713": {
+ "bm25": 0.2372,
+ "jev_median": 0.3066
+ },
+ "2724": {
+ "bm25": 0.2641,
+ "jev_median": 0.6131
+ },
+ "2737": {
+ "bm25": 0.3869,
+ "jev_median": 1.0
+ },
+ "2747": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2749": {
+ "bm25": 0.6131,
+ "jev_median": 1.0
+ },
+ "2790": {
+ "bm25": 0.1374,
+ "jev_median": 0.3543
+ },
+ "2801": {
+ "bm25": 0.8175,
+ "jev_median": 1.0
+ },
+ "2856": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2857": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2880": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "2885": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2891": {
+ "bm25": 0.3333,
+ "jev_median": 1.0
+ },
+ "2895": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2903": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2923": {
+ "bm25": 0.3392,
+ "jev_median": 0.3392
+ },
+ "2964": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "2968": {
+ "bm25": 0.5803,
+ "jev_median": 0.6509
+ },
+ "2994": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3006": {
+ "bm25": 0.0,
+ "jev_median": 0.0875
+ },
+ "3008": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3014": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3033": {
+ "bm25": 0.2044,
+ "jev_median": 0.6131
+ },
+ "3039": {
+ "bm25": 0.3155,
+ "jev_median": 0.3333
+ },
+ "3049": {
+ "bm25": 0.0,
+ "jev_median": 0.3904
+ },
+ "3051": {
+ "bm25": 0.8316,
+ "jev_median": 1.0
+ },
+ "3067": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3085": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3091": {
+ "bm25": 0.0867,
+ "jev_median": 0.5857
+ },
+ "3103": {
+ "bm25": 0.5,
+ "jev_median": 1.0
+ },
+ "3115": {
+ "bm25": 0.4935,
+ "jev_median": 0.6448
+ },
+ "3125": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3148": {
+ "bm25": 0.5087,
+ "jev_median": 0.5531
+ },
+ "3149": {
+ "bm25": 0.1846,
+ "jev_median": 0.2641
+ },
+ "3177": {
+ "bm25": 0.148,
+ "jev_median": 0.148
+ },
+ "3179": {
+ "bm25": 0.3066,
+ "jev_median": 0.8772
+ },
+ "3186": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "3189": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3254": {
+ "bm25": 0.4693,
+ "jev_median": 0.7654
+ },
+ "3264": {
+ "bm25": 0.0,
+ "jev_median": 0.3904
+ },
+ "3357": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "3369": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3394": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3404": {
+ "bm25": 0.1063,
+ "jev_median": 0.2749
+ },
+ "3405": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3446": {
+ "bm25": 0.2339,
+ "jev_median": 0.5873
+ },
+ "3451": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3453": {
+ "bm25": 0.4911,
+ "jev_median": 1.0
+ },
+ "3480": {
+ "bm25": 0.7877,
+ "jev_median": 0.8319
+ },
+ "3490": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3500": {
+ "bm25": 0.1989,
+ "jev_median": 0.4539
+ },
+ "3503": {
+ "bm25": 0.2749,
+ "jev_median": 0.5857
+ },
+ "3512": {
+ "bm25": 0.3066,
+ "jev_median": 0.6131
+ },
+ "3528": {
+ "bm25": 0.4307,
+ "jev_median": 1.0
+ },
+ "3530": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3534": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3566": {
+ "bm25": 0.2961,
+ "jev_median": 0.4693
+ },
+ "3569": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3594": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3612": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3615": {
+ "bm25": 0.6131,
+ "jev_median": 0.3869
+ },
+ "3625": {
+ "bm25": 0.0,
+ "jev_median": 0.4693
+ },
+ "3682": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3683": {
+ "bm25": 0.1303,
+ "jev_median": 0.6106
+ },
+ "3694": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "3724": {
+ "bm25": 0.0,
+ "jev_median": 0.2201
+ },
+ "3735": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3759": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3767": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3771": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3781": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "3789": {
+ "bm25": 0.7757,
+ "jev_median": 0.8048
+ },
+ "3791": {
+ "bm25": 0.4693,
+ "jev_median": 0.4693
+ },
+ "3801": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "3822": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3829": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3830": {
+ "bm25": 0.301,
+ "jev_median": 1.0
+ },
+ "3837": {
+ "bm25": 0.6309,
+ "jev_median": 0.5
+ },
+ "3859": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3875": {
+ "bm25": 0.6241,
+ "jev_median": 1.0
+ },
+ "3888": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3909": {
+ "bm25": 0.0734,
+ "jev_median": 0.3301
+ },
+ "3932": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "3934": {
+ "bm25": 0.1391,
+ "jev_median": 0.3904
+ },
+ "3995": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4007": {
+ "bm25": 0.4693,
+ "jev_median": 0.7654
+ },
+ "4011": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4019": {
+ "bm25": 0.3392,
+ "jev_median": 0.1208
+ },
+ "4031": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4037": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4047": {
+ "bm25": 0.9447,
+ "jev_median": 0.955
+ },
+ "4071": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4084": {
+ "bm25": 0.1846,
+ "jev_median": 0.3869
+ },
+ "4102": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4103": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4105": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4116": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "4125": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4142": {
+ "bm25": 0.1301,
+ "jev_median": 0.4382
+ },
+ "4153": {
+ "bm25": 0.0,
+ "jev_median": 0.6309
+ },
+ "4179": {
+ "bm25": 0.0,
+ "jev_median": 0.4307
+ },
+ "4188": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4205": {
+ "bm25": 0.0,
+ "jev_median": 0.4693
+ },
+ "4233": {
+ "bm25": 0.674,
+ "jev_median": 0.7227
+ },
+ "4265": {
+ "bm25": 0.1909,
+ "jev_median": 0.6448
+ },
+ "4286": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "4289": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4306": {
+ "bm25": 0.5294,
+ "jev_median": 0.5414
+ },
+ "4312": {
+ "bm25": 0.6725,
+ "jev_median": 0.7709
+ },
+ "4335": {
+ "bm25": 0.6367,
+ "jev_median": 0.6367
+ },
+ "4339": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "4394": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4409": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4411": {
+ "bm25": 0.0,
+ "jev_median": 0.4693
+ },
+ "4414": {
+ "bm25": 0.6131,
+ "jev_median": 0.3869
+ },
+ "4415": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4433": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4447": {
+ "bm25": 1.0,
+ "jev_median": 0.5
+ },
+ "4464": {
+ "bm25": 0.0,
+ "jev_median": 1.0
+ },
+ "4465": {
+ "bm25": 0.0,
+ "jev_median": 0.6364
+ },
+ "4484": {
+ "bm25": 0.0,
+ "jev_median": 1.0
+ },
+ "4499": {
+ "bm25": 0.148,
+ "jev_median": 0.7039
+ },
+ "4500": {
+ "bm25": 0.0,
+ "jev_median": 0.3869
+ },
+ "4504": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "4514": {
+ "bm25": 0.1513,
+ "jev_median": 0.2388
+ },
+ "4523": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4539": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4571": {
+ "bm25": 0.4693,
+ "jev_median": 0.7654
+ },
+ "4600": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4605": {
+ "bm25": 0.1063,
+ "jev_median": 0.3665
+ },
+ "4615": {
+ "bm25": 0.1009,
+ "jev_median": 0.545
+ },
+ "4640": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4641": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4678": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4681": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "4700": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "4714": {
+ "bm25": 0.6984,
+ "jev_median": 0.6096
+ },
+ "4756": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4767": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4775": {
+ "bm25": 0.0,
+ "jev_median": 0.6309
+ },
+ "4777": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4785": {
+ "bm25": 0.0,
+ "jev_median": 1.0
+ },
+ "4804": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4813": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4823": {
+ "bm25": 0.4693,
+ "jev_median": 0.4693
+ },
+ "4827": {
+ "bm25": 0.0,
+ "jev_median": 1.0
+ },
+ "4837": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4844": {
+ "bm25": 0.5714,
+ "jev_median": 1.0
+ },
+ "4845": {
+ "bm25": 0.3045,
+ "jev_median": 0.9469
+ },
+ "4846": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4863": {
+ "bm25": 0.3501,
+ "jev_median": 0.9675
+ },
+ "4865": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4920": {
+ "bm25": 0.6364,
+ "jev_median": 1.0
+ },
+ "4942": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4946": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "4955": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4962": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "4968": {
+ "bm25": 0.5,
+ "jev_median": 0.3333
+ },
+ "4981": {
+ "bm25": 0.6508,
+ "jev_median": 0.7654
+ },
+ "4999": {
+ "bm25": 0.3904,
+ "jev_median": 0.5294
+ },
+ "5021": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5030": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5045": {
+ "bm25": 1.0,
+ "jev_median": 0.5
+ },
+ "5054": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5061": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5064": {
+ "bm25": 0.3066,
+ "jev_median": 1.0
+ },
+ "5067": {
+ "bm25": 0.5,
+ "jev_median": 0.6309
+ },
+ "5080": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5083": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5085": {
+ "bm25": 0.301,
+ "jev_median": 1.0
+ },
+ "5086": {
+ "bm25": 1.0,
+ "jev_median": 0.4307
+ },
+ "5090": {
+ "bm25": 0.0,
+ "jev_median": 0.2641
+ },
+ "5125": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "5134": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5150": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5155": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "5172": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5178": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5185": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5196": {
+ "bm25": 0.1564,
+ "jev_median": 0.4693
+ },
+ "5206": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5228": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5231": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5241": {
+ "bm25": 0.3392,
+ "jev_median": 0.1312
+ },
+ "5254": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5255": {
+ "bm25": 0.9325,
+ "jev_median": 0.9325
+ },
+ "5264": {
+ "bm25": 0.5087,
+ "jev_median": 0.5296
+ },
+ "5271": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5331": {
+ "bm25": 0.3869,
+ "jev_median": 1.0
+ },
+ "5343": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5347": {
+ "bm25": 0.4307,
+ "jev_median": 1.0
+ },
+ "5356": {
+ "bm25": 0.0,
+ "jev_median": 0.3392
+ },
+ "5369": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5374": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5380": {
+ "bm25": 0.6309,
+ "jev_median": 1.0
+ },
+ "5402": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5410": {
+ "bm25": 0.1564,
+ "jev_median": 0.2961
+ },
+ "5422": {
+ "bm25": 0.0,
+ "jev_median": 0.3904
+ },
+ "5427": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5460": {
+ "bm25": 0.3392,
+ "jev_median": 0.3392
+ },
+ "5464": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5503": {
+ "bm25": 0.2044,
+ "jev_median": 0.6131
+ },
+ "5505": {
+ "bm25": 0.6309,
+ "jev_median": 1.0
+ },
+ "5511": {
+ "bm25": 0.0,
+ "jev_median": 0.2201
+ },
+ "5534": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5549": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5585": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5592": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "5616": {
+ "bm25": 0.0,
+ "jev_median": 0.5
+ },
+ "5620": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5646": {
+ "bm25": 0.7039,
+ "jev_median": 1.0
+ },
+ "5653": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "5683": {
+ "bm25": 0.8503,
+ "jev_median": 0.9197
+ },
+ "5710": {
+ "bm25": 0.0,
+ "jev_median": 0.6309
+ },
+ "5741": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5763": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5782": {
+ "bm25": 0.3392,
+ "jev_median": 0.3008
+ },
+ "5790": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5808": {
+ "bm25": 0.0,
+ "jev_median": 1.0
+ },
+ "5853": {
+ "bm25": 0.0731,
+ "jev_median": 0.3619
+ },
+ "5862": {
+ "bm25": 0.1128,
+ "jev_median": 0.0
+ },
+ "5888": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "5903": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5906": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5940": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "5951": {
+ "bm25": 0.2495,
+ "jev_median": 0.42
+ },
+ "5970": {
+ "bm25": 0.0,
+ "jev_median": 0.2961
+ },
+ "5981": {
+ "bm25": 0.7227,
+ "jev_median": 0.8688
+ },
+ "5993": {
+ "bm25": 0.224,
+ "jev_median": 0.2985
+ },
+ "6002": {
+ "bm25": 0.4807,
+ "jev_median": 0.7538
+ },
+ "6004": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6005": {
+ "bm25": 0.4252,
+ "jev_median": 0.3679
+ },
+ "6009": {
+ "bm25": 0.2891,
+ "jev_median": 0.6309
+ },
+ "6041": {
+ "bm25": 0.7495,
+ "jev_median": 0.1301
+ },
+ "6080": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "6110": {
+ "bm25": 0.4773,
+ "jev_median": 0.57
+ },
+ "6121": {
+ "bm25": 0.2044,
+ "jev_median": 0.6131
+ },
+ "6122": {
+ "bm25": 0.4693,
+ "jev_median": 0.4693
+ },
+ "6131": {
+ "bm25": 0.3786,
+ "jev_median": 0.5077
+ },
+ "6133": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "6142": {
+ "bm25": 0.0,
+ "jev_median": 0.6131
+ },
+ "6146": {
+ "bm25": 0.0,
+ "jev_median": 0.6131
+ },
+ "6199": {
+ "bm25": 0.0,
+ "jev_median": 0.3904
+ },
+ "6219": {
+ "bm25": 1.0,
+ "jev_median": 0.9197
+ },
+ "6221": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6252": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6262": {
+ "bm25": 0.1564,
+ "jev_median": 0.4693
+ },
+ "6278": {
+ "bm25": 0.6053,
+ "jev_median": 0.0
+ },
+ "6395": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6410": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6420": {
+ "bm25": 0.6131,
+ "jev_median": 0.3869
+ },
+ "6441": {
+ "bm25": 0.3869,
+ "jev_median": 0.0
+ },
+ "6467": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6468": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "6479": {
+ "bm25": 0.7977,
+ "jev_median": 0.8772
+ },
+ "6525": {
+ "bm25": 0.5307,
+ "jev_median": 0.4982
+ },
+ "6554": {
+ "bm25": 0.2184,
+ "jev_median": 0.6131
+ },
+ "6562": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6611": {
+ "bm25": 0.2372,
+ "jev_median": 0.8772
+ },
+ "6612": {
+ "bm25": 0.1413,
+ "jev_median": 0.4693
+ },
+ "6625": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6629": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6635": {
+ "bm25": 0.1952,
+ "jev_median": 0.3904
+ },
+ "6644": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6647": {
+ "bm25": 0.0,
+ "jev_median": 0.4693
+ },
+ "6668": {
+ "bm25": 0.3869,
+ "jev_median": 0.2184
+ },
+ "6679": {
+ "bm25": 0.1815,
+ "jev_median": 0.7654
+ },
+ "6683": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6713": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "6715": {
+ "bm25": 0.0,
+ "jev_median": 1.0
+ },
+ "6746": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6787": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6792": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6800": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6803": {
+ "bm25": 0.2685,
+ "jev_median": 0.7087
+ },
+ "6807": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "6814": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6832": {
+ "bm25": 0.0,
+ "jev_median": 0.2641
+ },
+ "6835": {
+ "bm25": 0.0,
+ "jev_median": 0.6131
+ },
+ "6849": {
+ "bm25": 1.0,
+ "jev_median": 0.4307
+ },
+ "6862": {
+ "bm25": 0.301,
+ "jev_median": 1.0
+ },
+ "6867": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6875": {
+ "bm25": 0.4307,
+ "jev_median": 0.0
+ },
+ "6890": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6891": {
+ "bm25": 0.4693,
+ "jev_median": 0.2961
+ },
+ "6896": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6901": {
+ "bm25": 0.3066,
+ "jev_median": 0.3869
+ },
+ "6907": {
+ "bm25": 0.0,
+ "jev_median": 0.5
+ },
+ "6909": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6959": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "6985": {
+ "bm25": 0.6309,
+ "jev_median": 1.0
+ },
+ "7017": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7068": {
+ "bm25": 0.8066,
+ "jev_median": 0.3066
+ },
+ "7071": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7080": {
+ "bm25": 0.3066,
+ "jev_median": 1.0
+ },
+ "7096": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7098": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7105": {
+ "bm25": 0.3333,
+ "jev_median": 1.0
+ },
+ "7109": {
+ "bm25": 0.6049,
+ "jev_median": 0.4982
+ },
+ "7124": {
+ "bm25": 0.2463,
+ "jev_median": 0.1952
+ },
+ "7141": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7145": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7178": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7188": {
+ "bm25": 0.0,
+ "jev_median": 0.6309
+ },
+ "7205": {
+ "bm25": 0.5,
+ "jev_median": 1.0
+ },
+ "7206": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7218": {
+ "bm25": 0.3333,
+ "jev_median": 1.0
+ },
+ "7221": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7269": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7279": {
+ "bm25": 0.0,
+ "jev_median": 0.3904
+ },
+ "7295": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7311": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7326": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7329": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7344": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "7345": {
+ "bm25": 0.0,
+ "jev_median": 0.2961
+ },
+ "7377": {
+ "bm25": 0.3869,
+ "jev_median": 1.0
+ },
+ "7431": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7441": {
+ "bm25": 0.2021,
+ "jev_median": 0.1564
+ },
+ "7445": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7448": {
+ "bm25": 0.6309,
+ "jev_median": 1.0
+ },
+ "7456": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7463": {
+ "bm25": 0.0,
+ "jev_median": 0.3392
+ },
+ "7467": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "7484": {
+ "bm25": 0.5,
+ "jev_median": 1.0
+ },
+ "7509": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7512": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7513": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7529": {
+ "bm25": 0.1934,
+ "jev_median": 0.6131
+ },
+ "7533": {
+ "bm25": 0.1934,
+ "jev_median": 0.2372
+ },
+ "7534": {
+ "bm25": 0.4693,
+ "jev_median": 0.4693
+ },
+ "7590": {
+ "bm25": 0.3066,
+ "jev_median": 1.0
+ },
+ "7592": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7594": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7622": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7633": {
+ "bm25": 0.5079,
+ "jev_median": 0.5856
+ },
+ "7674": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7700": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7702": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7705": {
+ "bm25": 0.2346,
+ "jev_median": 0.6714
+ },
+ "7734": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "7747": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7754": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7758": {
+ "bm25": 0.0,
+ "jev_median": 1.0
+ },
+ "7801": {
+ "bm25": 0.7904,
+ "jev_median": 0.8503
+ },
+ "7803": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7823": {
+ "bm25": 0.6364,
+ "jev_median": 0.6714
+ },
+ "7876": {
+ "bm25": 0.2891,
+ "jev_median": 0.6309
+ },
+ "7879": {
+ "bm25": 0.0,
+ "jev_median": 0.4693
+ },
+ "7880": {
+ "bm25": 0.0,
+ "jev_median": 0.3869
+ },
+ "7911": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "7925": {
+ "bm25": 0.0,
+ "jev_median": 0.1909
+ },
+ "7928": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "7936": {
+ "bm25": 0.2837,
+ "jev_median": 0.6183
+ },
+ "7992": {
+ "bm25": 0.3066,
+ "jev_median": 0.6131
+ },
+ "8002": {
+ "bm25": 0.1564,
+ "jev_median": 0.4693
+ },
+ "8005": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8013": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8017": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8034": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8040": {
+ "bm25": 1.0,
+ "jev_median": 0.5
+ },
+ "8072": {
+ "bm25": 0.5307,
+ "jev_median": 0.3028
+ },
+ "8079": {
+ "bm25": 0.2961,
+ "jev_median": 0.4693
+ },
+ "8102": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8116": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8121": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8202": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8230": {
+ "bm25": 0.0,
+ "jev_median": 0.6131
+ },
+ "8247": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8271": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "8275": {
+ "bm25": 0.0,
+ "jev_median": 0.3869
+ },
+ "8296": {
+ "bm25": 0.5803,
+ "jev_median": 0.9197
+ },
+ "8332": {
+ "bm25": 0.0,
+ "jev_median": 0.7654
+ },
+ "8351": {
+ "bm25": 0.3066,
+ "jev_median": 0.6131
+ },
+ "8378": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8456": {
+ "bm25": 0.4693,
+ "jev_median": 0.4693
+ },
+ "8475": {
+ "bm25": 0.4483,
+ "jev_median": 0.5857
+ },
+ "8507": {
+ "bm25": 0.1357,
+ "jev_median": 0.4693
+ },
+ "8512": {
+ "bm25": 0.2891,
+ "jev_median": 0.6309
+ },
+ "8513": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8532": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8537": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8539": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8544": {
+ "bm25": 0.2372,
+ "jev_median": 0.6131
+ },
+ "8592": {
+ "bm25": 0.2346,
+ "jev_median": 0.4693
+ },
+ "8632": {
+ "bm25": 0.2372,
+ "jev_median": 0.3066
+ },
+ "8635": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8702": {
+ "bm25": 0.7654,
+ "jev_median": 0.6364
+ },
+ "8779": {
+ "bm25": 0.1846,
+ "jev_median": 0.1846
+ },
+ "8789": {
+ "bm25": 0.7654,
+ "jev_median": 0.7654
+ },
+ "8795": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "8832": {
+ "bm25": 0.6309,
+ "jev_median": 1.0
+ },
+ "8834": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8855": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8874": {
+ "bm25": 0.3155,
+ "jev_median": 0.3562
+ },
+ "8934": {
+ "bm25": 0.3066,
+ "jev_median": 0.1846
+ },
+ "8937": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8947": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "8959": {
+ "bm25": 0.0,
+ "jev_median": 0.5
+ },
+ "8970": {
+ "bm25": 0.2372,
+ "jev_median": 0.6131
+ },
+ "8974": {
+ "bm25": 0.0798,
+ "jev_median": 0.2529
+ },
+ "8982": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9060": {
+ "bm25": 0.2044,
+ "jev_median": 0.6131
+ },
+ "9088": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9108": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9115": {
+ "bm25": 0.1672,
+ "jev_median": 0.4693
+ },
+ "9126": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9164": {
+ "bm25": 0.0,
+ "jev_median": 0.1815
+ },
+ "9174": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9188": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "9245": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9275": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9291": {
+ "bm25": 0.2044,
+ "jev_median": 0.6131
+ },
+ "9296": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9329": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "9332": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9381": {
+ "bm25": 0.1846,
+ "jev_median": 0.6131
+ },
+ "9385": {
+ "bm25": 0.5,
+ "jev_median": 0.3562
+ },
+ "9391": {
+ "bm25": 0.0,
+ "jev_median": 0.3904
+ },
+ "9403": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9481": {
+ "bm25": 0.6241,
+ "jev_median": 1.0
+ },
+ "9487": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9548": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9556": {
+ "bm25": 0.6309,
+ "jev_median": 1.0
+ },
+ "9565": {
+ "bm25": 0.6131,
+ "jev_median": 0.6131
+ },
+ "9598": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9617": {
+ "bm25": 0.0,
+ "jev_median": 0.7654
+ },
+ "9633": {
+ "bm25": 0.2184,
+ "jev_median": 0.3066
+ },
+ "9643": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "9644": {
+ "bm25": 0.3155,
+ "jev_median": 0.6309
+ },
+ "9646": {
+ "bm25": 0.6131,
+ "jev_median": 0.0
+ },
+ "9668": {
+ "bm25": 0.6714,
+ "jev_median": 0.7654
+ },
+ "9701": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9733": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9735": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9737": {
+ "bm25": 1.0,
+ "jev_median": 0.852
+ },
+ "9771": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9808": {
+ "bm25": 0.1672,
+ "jev_median": 0.1357
+ },
+ "9824": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9871": {
+ "bm25": 0.3904,
+ "jev_median": 0.3904
+ },
+ "9882": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9925": {
+ "bm25": 0.6131,
+ "jev_median": 0.6241
+ },
+ "9929": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "9961": {
+ "bm25": 0.6309,
+ "jev_median": 0.5
+ },
+ "9979": {
+ "bm25": 0.3904,
+ "jev_median": 0.1681
+ },
+ "10034": {
+ "bm25": 0.2641,
+ "jev_median": 0.3869
+ },
+ "10039": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "10109": {
+ "bm25": 0.3933,
+ "jev_median": 0.2651
+ },
+ "10122": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "10136": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "10137": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "10152": {
+ "bm25": 0.5,
+ "jev_median": 1.0
+ },
+ "10183": {
+ "bm25": 0.1846,
+ "jev_median": 0.2641
+ },
+ "10213": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "10246": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "10267": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "10414": {
+ "bm25": 0.8316,
+ "jev_median": 1.0
+ },
+ "10447": {
+ "bm25": 0.148,
+ "jev_median": 0.4693
+ },
+ "10462": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "10482": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "10497": {
+ "bm25": 0.4252,
+ "jev_median": 0.5321
+ },
+ "10526": {
+ "bm25": 0.1846,
+ "jev_median": 0.0
+ },
+ "10547": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "10558": {
+ "bm25": 0.2372,
+ "jev_median": 0.3066
+ },
+ "10596": {
+ "bm25": 0.8539,
+ "jev_median": 0.786
+ },
+ "10601": {
+ "bm25": 1.0,
+ "jev_median": 0.5
+ },
+ "10628": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "10639": {
+ "bm25": 0.1461,
+ "jev_median": 0.3392
+ },
+ "10645": {
+ "bm25": 0.3066,
+ "jev_median": 0.2372
+ },
+ "10674": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "10710": {
+ "bm25": 0.0,
+ "jev_median": 1.0
+ },
+ "10734": {
+ "bm25": 0.6106,
+ "jev_median": 0.6106
+ },
+ "10792": {
+ "bm25": 0.1846,
+ "jev_median": 0.5912
+ },
+ "10808": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "10809": {
+ "bm25": 0.2961,
+ "jev_median": 0.4693
+ },
+ "10812": {
+ "bm25": 0.5294,
+ "jev_median": 0.7366
+ },
+ "10827": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "10845": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "10912": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "10932": {
+ "bm25": 0.6309,
+ "jev_median": 1.0
+ },
+ "10975": {
+ "bm25": 0.0,
+ "jev_median": 0.0
+ },
+ "10979": {
+ "bm25": 0.1952,
+ "jev_median": 0.5585
+ },
+ "10994": {
+ "bm25": 0.1772,
+ "jev_median": 0.6509
+ },
+ "11039": {
+ "bm25": 0.0843,
+ "jev_median": 0.1776
+ },
+ "11054": {
+ "bm25": 1.0,
+ "jev_median": 1.0
+ },
+ "11088": {
+ "bm25": 0.0,
+ "jev_median": 1.0
+ }
+ }
+}
\ No newline at end of file
diff --git a/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/README.md b/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/README.md
new file mode 100644
index 000000000..65d988042
--- /dev/null
+++ b/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/README.md
@@ -0,0 +1,70 @@
+# AG News — 4-way topic classification, 120k-deep history (2026-09-20)
+
+The scale end of the vote's range: **120,000 labelled train items** indexed,
+the full **7,600-item test set** classified, k = 10, rank-weighted 1/(i+1),
+BM25 neighbours only — no model of any kind, no embedder, the default lexical
+node. This is the workload the System One wire answers (`/v1/systemone`,
+`/_decide`), and it is the workload that motivated the engine's WAND fast
+path (engine cf6a9989): a full news item is a ~40-clause disjunction whose
+match set is most of the index.
+
+## Dataset and provenance
+
+**AG News** via HuggingFace `fancyzhx/ag_news` — the standard 4-class topic
+classification corpus (World / Sports / Business / SciTech), 120,000 train /
+7,600 test, originally by Zhang et al., *Character-level Convolutional
+Networks for Text Classification*, NeurIPS 2015
+(https://arxiv.org/abs/1509.01626). Downloaded as parquet from
+https://huggingface.co/datasets/fancyzhx/ag_news and converted to TSV
+(`label\ttext`) with pandas+pyarrow; the conversion script is not checked in
+because it is two lines — the dataset card's own `train`/`test` splits used
+verbatim, no re-cutting, no shuffling.
+
+## Headline numbers (all three runs, identical index, identical items)
+
+| Binary | ms/item | Accuracy | ECE | ≥0.8 conf | …and right | no-hit |
+|---|---:|---:|---:|---:|---:|---:|
+| pre-WAND (284d1f8f) | 68.7 | 0.9182 | 0.019 | 83.1% | 0.9669 | 0 |
+| WAND first cut | 51.2 | 0.9182 | 0.019 | 83.1% | 0.9669 | 0 |
+| WAND + alloc-free loop (cf6a9989) | **34.2** | 0.9182 | 0.019 | 83.1% | 0.9669 | 0 |
+
+The point of the three rows: **2.0× end-to-end latency, bit-identical
+decisions** — same 7,600 labels, same confidence bands, same ECE. Keep-alive
+HTTP, one connection, distinct items (no cache reuse — every item is new,
+which is the real workload). Run logs: `run-2026-09-20.txt`.
+
+Same-box wire comparison for the same workload class: the hosted Jev API
+measured p50 1,230 ms per item on SciFact-scale shortlists (see
+`../../../beir-hybrid/results/2026-09-20-rerank-full/`); the local vote on
+120k history answers `/v1/systemone` at p50 42.7 ms (300 distinct items,
+final binary). Different tools — the vote needs this labelled history, Jev
+needs nothing — so this is a cost/latency/egress contrast, not a quality
+race (Jev was not run on AG News; no Jev row belongs on this dataset).
+
+## What the numbers mean
+
+- **92% of 4-way routing decisions made locally, free, with no egress** — and
+ 83% of them at ≥0.8 confidence, where the vote is right 96.7% of the time.
+ The ECE of 0.019 means the confidence can gate: send the uncertain 17% to
+ a judge or a person.
+- **Why AG News is easy for retrieval**: the four topics have disjoint
+ vocabularies (scoreboard, ticker, spacecraft). The vote is not
+ understanding news; it is finding the ten most lexically similar
+ headlines in 120k history and reading their labels. Banking77 (77 near-
+ duplicate intents) is the hard in-domain case; this is the high-volume
+ cheap one.
+- **Latency scales with the match set, not the corpus**: 120k docs, but the
+ cost driver is that a 40-token item matches nearly everything, so nearly
+ every doc is scored. Shorter texts against the same index are far under
+ this (SMS: 0.9 ms/item at 4k history). No TB-scale claim is being made
+ here; this is one shape at one scale, measured.
+
+## Reproduce
+
+```sh
+# 1. dataset: HF fancyzhx/ag_news -> TSV (label<TAB>text), train + test
+xerj --insecure --port 9460 --data-dir ./agnews-data &
+python3 agnews_vote.py # indexes 120k train, runs 7,600 test items
+# A/B against an older binary: boot both, then
+python3 vote_eval.py <port> # keep-alive eval, prints acc/ECE/ms-item
+```
diff --git a/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/agnews_vote.py b/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/agnews_vote.py
new file mode 100644
index 000000000..e459bef85
--- /dev/null
+++ b/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/agnews_vote.py
@@ -0,0 +1,45 @@
+"""AG News (HF fancyzhx/ag_news): classify the 7,600 test items by a
+rank-weighted k=10 BM25 vote over the 120,000 labelled train items.
+Same methodology as benchmarks/decisions-as-retrieval/eval.py."""
+import json, time, urllib.request, collections
+U = "http://localhost:9460"; K = 10
+def req(m, p, b=None, ct="application/json"):
+ d = b if isinstance(b, (bytes, type(None))) else json.dumps(b).encode()
+ r = urllib.request.Request(U + p, data=d, method=m, headers={"content-type": ct})
+ try: return json.loads(urllib.request.urlopen(r, timeout=300).read())
+ except urllib.error.HTTPError as e: return {"_err": e.code}
+
+req("DELETE", "/agnews")
+req("PUT", "/agnews", {"mappings": {"properties": {"text": {"type": "text"}, "label": {"type": "keyword"}}}})
+lines = []
+for i, l in enumerate(open("agnews-train.tsv").readlines()[1:]):
+ lab, txt = l.rstrip("\n").split("\t", 1)
+ lines.append(json.dumps({"index": {"_index": "agnews", "_id": str(i)}}))
+ lines.append(json.dumps({"text": txt, "label": lab}))
+ if len(lines) >= 4000:
+ req("POST", "/_bulk", ("\n".join(lines) + "\n").encode(), "application/x-ndjson"); lines = []
+ if i % 20000 == 0: print("indexed", i, flush=True)
+if lines: req("POST", "/_bulk", ("\n".join(lines) + "\n").encode(), "application/x-ndjson")
+print(req("POST", "/agnews/_refresh")); print("count:", req("GET", "/agnews/_count")["count"])
+
+test = [l.rstrip("\n").split("\t", 1) for l in open("agnews-test.tsv").readlines()[1:]]
+print("test items:", len(test))
+cc = []; t0 = time.time(); none = 0
+for gold, txt in test:
+ r = req("POST", "/agnews/_search", {"size": K, "_source": ["label"], "query": {"match": {"text": txt}}})
+ hs = r.get("hits", {}).get("hits", [])
+ if not hs: none += 1; cc.append((0.0, 0)); continue
+ v = collections.Counter()
+ for i, h in enumerate(hs): v[h["_source"]["label"]] += 1.0 / (i + 1)
+ lab, w = v.most_common(1)[0]; conf = w / sum(v.values())
+ cc.append((conf, 1 if lab == gold else 0))
+acc = sum(k for _, k in cc) / len(cc); ms = (time.time() - t0) / len(cc) * 1000
+def ece(cc, bins=10):
+ n = len(cc); tot = 0.0
+ for b in range(bins):
+ xs = [(c, k) for c, k in cc if b / bins < c <= (b + 1) / bins or (b == 0 and c == 0)]
+ if xs: tot += len(xs) / n * abs(sum(k for _, k in xs) / len(xs) - sum(c for c, _ in xs) / len(xs))
+ return tot
+hi = [k for c, k in cc if c >= 0.8]
+print(f"AG News 4-way — BM25 vote, k={K}")
+print(f" acc={acc:.4f} ECE={ece(cc):.3f} conf>=0.8: {len(hi)/len(cc):.1%} of items at {sum(hi)/max(len(hi),1):.4f} acc no-hit={none} {ms:.1f} ms/item")
diff --git a/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/run-2026-09-20.txt b/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/run-2026-09-20.txt
new file mode 100644
index 000000000..487f35a08
--- /dev/null
+++ b/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/run-2026-09-20.txt
@@ -0,0 +1,48 @@
+ 500/7600 70.04 ms/item
+ 1000/7600 69.69 ms/item
+ 1500/7600 69.74 ms/item
+ 2000/7600 69.73 ms/item
+ 2500/7600 69.57 ms/item
+ 3000/7600 69.56 ms/item
+ 3500/7600 69.81 ms/item
+ 4000/7600 69.70 ms/item
+ 4500/7600 69.34 ms/item
+ 5000/7600 69.18 ms/item
+ 5500/7600 69.10 ms/item
+ 6000/7600 68.91 ms/item
+ 6500/7600 68.85 ms/item
+ 7000/7600 68.86 ms/item
+ 7500/7600 68.75 ms/item
+port 9460: acc=0.9182 no-hit=0 conf>=0.8: 83.1% at 0.9669 68.74 ms/item
+ 500/7600 54.37 ms/item
+ 1000/7600 53.25 ms/item
+ 1500/7600 52.93 ms/item
+ 2000/7600 51.99 ms/item
+ 2500/7600 51.97 ms/item
+ 3000/7600 51.95 ms/item
+ 3500/7600 52.18 ms/item
+ 4000/7600 52.04 ms/item
+ 4500/7600 51.96 ms/item
+ 5000/7600 51.94 ms/item
+ 5500/7600 51.97 ms/item
+ 6000/7600 51.67 ms/item
+ 6500/7600 51.50 ms/item
+ 7000/7600 51.45 ms/item
+ 7500/7600 51.23 ms/item
+port 9480: acc=0.9182 no-hit=0 conf>=0.8: 83.1% at 0.9669 51.19 ms/item
+ 500/7600 37.43 ms/item
+ 1000/7600 36.34 ms/item
+ 1500/7600 35.92 ms/item
+ 2000/7600 35.17 ms/item
+ 2500/7600 34.86 ms/item
+ 3000/7600 34.80 ms/item
+ 3500/7600 34.91 ms/item
+ 4000/7600 34.79 ms/item
+ 4500/7600 34.64 ms/item
+ 5000/7600 34.55 ms/item
+ 5500/7600 34.50 ms/item
+ 6000/7600 34.33 ms/item
+ 6500/7600 34.29 ms/item
+ 7000/7600 34.30 ms/item
+ 7500/7600 34.20 ms/item
+port 9480: acc=0.9182 no-hit=0 conf>=0.8: 83.1% at 0.9669 34.19 ms/item
diff --git a/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/vote_eval.py b/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/vote_eval.py
new file mode 100644
index 000000000..5c54710a1
--- /dev/null
+++ b/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/vote_eval.py
@@ -0,0 +1,28 @@
+"""AG News vote eval against one node, keep-alive, with progress.
+Usage: vote_eval.py <port> — prints acc / no-hit / conf>=0.8 / ms-item."""
+import http.client, json, time, collections, sys
+
+port = int(sys.argv[1])
+test = [l.rstrip("\n").split("\t", 1) for l in open("/root/xerj-bench/agnews-test.tsv").readlines()[1:]]
+c = http.client.HTTPConnection("localhost", port, timeout=300)
+cc = []; none = 0
+t0 = time.time()
+for n, (gold, txt) in enumerate(test):
+ body = json.dumps({"size": 10, "_source": ["label"], "query": {"match": {"text": txt}}}).encode()
+ c.request("POST", "/agnews/_search", body, {"content-type": "application/json"})
+ r = json.loads(c.getresponse().read())
+ hs = r.get("hits", {}).get("hits", [])
+ if not hs:
+ none += 1; cc.append((0.0, 0)); continue
+ v = collections.Counter()
+ for i, h in enumerate(hs): v[h["_source"]["label"]] += 1.0 / (i + 1)
+ lab, w = v.most_common(1)[0]; conf = w / sum(v.values())
+ cc.append((conf, 1 if lab == gold else 0))
+ if (n + 1) % 500 == 0:
+ el = time.time() - t0
+ print(f" {n+1}/{len(test)} {el/(n+1)*1000:.2f} ms/item", flush=True)
+c.close()
+ms = (time.time() - t0) / len(cc) * 1000
+acc = sum(k for _, k in cc) / len(cc)
+hi = [k for c, k in cc if c >= 0.8]
+print(f"port {port}: acc={acc:.4f} no-hit={none} conf>=0.8: {len(hi)/len(cc):.1%} at {sum(hi)/max(len(hi),1):.4f} {ms:.2f} ms/item")