AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
benchmarks: FiQA (BEIR) and AG News (HF) — two public datasets added with provenance, runs and validation
xerj-org96f90107552b
diff --git a/benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa/README.md b/benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa/README.mdnew file mode 100644index 000000000..a708e458e--- /dev/null+++ b/benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa/README.md@@ -0,0 +1,69 @@+# FiQA rerank full run — 2026-09-20 (648 queries, 3 repeats)++Third BEIR dataset through the same harness as+`../2026-09-20-rerank-full/` (SciFact + NFCorpus), added so the measured-Jev+board covers a finance-domain QA corpus with sparse relevance judgements —+the regime where a general-purpose judge is most often wrong-when-confident.++## Dataset and provenance++**FiQA-2018**, the BEIR `test` split: 57,638 corpus documents, 648 judged+test queries, graded qrels — the same files every BEIR paper uses.++- Task page: https://arxiv.org/abs/2104.08663 (BEIR, Thakur et al., 2021)+- Canonical BEIR distribution: the `fiqa` corpus/queries/qrels from+ https://github.com/beir-cellar/beir (MIT), mirrored at+ `https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/fiqa.zip`+- SHA of the zip we fetched is recorded in the local run log; nothing here+ was re-tokenised or re-cut — corpus.jsonl / queries.jsonl / qrels/test.tsv+ are the archive's files verbatim.++**Harness validation before any model was called:** local BM25 nDCG@10 on the+648 test queries came out **0.2382**, against BEIR's published BM25 figure of+**0.236** for FiQA (BM25 from the BEIR paper's table, reproduced by many+independent runs since). Within 0.002 — the harness scores the way the+reference scores. The same BM25 figure reproduced identically (0.2382, three+times, and again bit-for-bit after the WAND fast path landed in the engine,+cf6a9989).++## Headline numbers++| Dataset | BM25 nDCG@10 | Jev median nDCG@10 | W / L / T | input tokens | cost | wall p50 |+|---|---:|---:|---|---:|---:|---:|+| FiQA (648 q) | 0.2382 | 0.3638 | 249 / 57 / 342 | 15,178,758 | $0.6375 | 1.23 s/q |++Cost at $0.042/Mtok input-only. Zero failed requests; zero queries with an+empty BM25 shortlist (FiQA's queries are vocabulary-rich, unlike NFCorpus's+25 empties). Model pinned `jev-1.13.0`, one noul per document, medians over+three shuffled repeats — the fixed request shape from the pilot's autopsy.++FiQA is the judge's **biggest lift of the three datasets** (+0.126 over BM25;+SciFact +0.084, NFCorpus +0.030) and the reason is visible in the shortlist:+finance QA relevance is phrase-level ("business expense on a business trip"),+and BM25's word-overlap ranking leaves real relevant documents far below+rank 10 where the judge, reading the document, recovers them.++## The finding: the confidence is worse than useless as a threshold here++ECE **0.3109** over 19,440 (probability, relevance) pairs — three times+SciFact's 0.1023. The curve is still monotone (the judge discriminates), but+the top bin is the whole story: **mean confidence 0.93, actual relevance+0.34**. Bin-by-bin curve in `fiqa-full-run.json`. On a sparse-judgement+corpus a Jev probability near 1.0 still means "about a one-in-three chance+the qrels call this relevant" — which is exactly the failure mode of+thresholding a raw probability. The ordering is worth +0.126 nDCG; the number+itself must never gate.++## Reproduce++```sh+# corpus: fetch the BEIR fiqa.zip, extract corpus.jsonl / queries.jsonl / qrels/test.tsv+python3 load.py fiqa/corpus.jsonl fiqa # this repo's loader, title+text fields+XERJ_URL=http://localhost:<port> python3 raw_bench_full.py 648 fiqa+# -> fiqa-full-run.json (aggregates, reliability curve, per-query numbers)+```++Run log: repeat 1 bm25=0.2382 jev=0.3648 · repeat 2 bm25=0.2382 jev=0.3622 ·+repeat 3 bm25=0.2382 jev=0.3636 — the BM25 arm is deterministic across+repeats (same engine, same index), the Jev arm wobbles ±0.0013, consistent+with the provider non-determinism measured on the other two datasets.diff --git a/benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa/fiqa-full-run.json b/benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa/fiqa-full-run.jsonnew file mode 100644index 000000000..3da68d3aa--- /dev/null+++ b/benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa/fiqa-full-run.json@@ -0,0 +1,2673 @@+{+ "dataset": "fiqa",+ "queries": 648,+ "empty_bm25": 0,+ "fails": 0,+ "bm25_ndcg10": 0.2382,+ "jev_fixed_median_ndcg10": 0.3638,+ "wins": 249,+ "losses": 57,+ "ties": 342,+ "input_tokens": 15178758,+ "output_tokens": 1038096,+ "cost_usd": 0.6375,+ "wall_s_p50": 1.23,+ "wall_s_p95": 1.39,+ "total_wall_min": 40.4,+ "determinism": {+ "pairs_compared": 900,+ "max_abs_delta": 0.1,+ "mean_abs_delta": 0.0148+ },+ "calibration": {+ "n_pairs": 19440,+ "bins": 10,+ "ece": 0.3109,+ "curve": {+ "0": {+ "mean_conf": 0.0455,+ "acc": 0.0002,+ "n": 5572+ },+ "1": {+ "mean_conf": 0.1418,+ "acc": 0.0003,+ "n": 3005+ },+ "2": {+ "mean_conf": 0.2434,+ "acc": 0.0034,+ "n": 2061+ },+ "3": {+ "mean_conf": 0.3438,+ "acc": 0.0033,+ "n": 1529+ },+ "4": {+ "mean_conf": 0.4445,+ "acc": 0.0077,+ "n": 1423+ },+ "5": {+ "mean_conf": 0.5442,+ "acc": 0.0071,+ "n": 1271+ },+ "6": {+ "mean_conf": 0.6451,+ "acc": 0.0224,+ "n": 1118+ },+ "7": {+ "mean_conf": 0.7455,+ "acc": 0.0408,+ "n": 1224+ },+ "8": {+ "mean_conf": 0.8452,+ "acc": 0.1242,+ "n": 1272+ },+ "9": {+ "mean_conf": 0.9297,+ "acc": 0.3378,+ "n": 965+ }+ }+ },+ "per_query": {+ "8": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "15": {+ "bm25": 0.0,+ "jev_median": 1.0+ },+ "18": {+ "bm25": 0.0,+ "jev_median": 0.6309+ },+ "26": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "34": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "42": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "56": {+ "bm25": 0.301,+ "jev_median": 0.6309+ },+ "68": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "89": {+ "bm25": 0.2684,+ "jev_median": 0.3422+ },+ "90": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "94": {+ "bm25": 0.5,+ "jev_median": 1.0+ },+ "98": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "104": {+ "bm25": 0.3066,+ "jev_median": 0.6131+ },+ "106": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "109": {+ "bm25": 0.6309,+ "jev_median": 1.0+ },+ "475": {+ "bm25": 0.5,+ "jev_median": 1.0+ },+ "503": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "504": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "515": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "529": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "547": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "549": {+ "bm25": 0.0,+ "jev_median": 1.0+ },+ "559": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "570": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "585": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "588": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "594": {+ "bm25": 0.6131,+ "jev_median": 0.2372+ },+ "603": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "604": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "620": {+ "bm25": 0.1021,+ "jev_median": 0.3392+ },+ "622": {+ "bm25": 0.0,+ "jev_median": 0.6131+ },+ "659": {+ "bm25": 0.3785,+ "jev_median": 0.3785+ },+ "672": {+ "bm25": 0.1772,+ "jev_median": 0.0+ },+ "684": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "687": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "689": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "691": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "699": {+ "bm25": 0.0,+ "jev_median": 1.0+ },+ "701": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "715": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "721": {+ "bm25": 0.3869,+ "jev_median": 1.0+ },+ "744": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "750": {+ "bm25": 0.9197,+ "jev_median": 0.8175+ },+ "753": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "766": {+ "bm25": 0.0,+ "jev_median": 0.4693+ },+ "776": {+ "bm25": 0.2201,+ "jev_median": 0.359+ },+ "810": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "813": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "849": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "852": {+ "bm25": 0.8603,+ "jev_median": 0.3836+ },+ "853": {+ "bm25": 0.2044,+ "jev_median": 0.3869+ },+ "858": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "859": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "864": {+ "bm25": 0.3552,+ "jev_median": 0.7751+ },+ "879": {+ "bm25": 0.6309,+ "jev_median": 1.0+ },+ "885": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "895": {+ "bm25": 0.4307,+ "jev_median": 1.0+ },+ "904": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "928": {+ "bm25": 0.1772,+ "jev_median": 0.6131+ },+ "929": {+ "bm25": 0.0,+ "jev_median": 1.0+ },+ "932": {+ "bm25": 0.8316,+ "jev_median": 0.6934+ },+ "939": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "945": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "957": {+ "bm25": 0.4693,+ "jev_median": 0.1672+ },+ "988": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "1074": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "1085": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "1090": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "1150": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "1157": {+ "bm25": 0.6131,+ "jev_median": 0.3869+ },+ "1159": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "1198": {+ "bm25": 1.0,+ "jev_median": 0.5+ },+ "1230": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "1281": {+ "bm25": 0.0,+ "jev_median": 1.0+ },+ "1284": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "1297": {+ "bm25": 0.4307,+ "jev_median": 1.0+ },+ "1306": {+ "bm25": 0.0,+ "jev_median": 0.3904+ },+ "1309": {+ "bm25": 0.151,+ "jev_median": 0.6367+ },+ "1310": {+ "bm25": 0.4307,+ "jev_median": 1.0+ },+ "1321": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "1322": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "1391": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "1393": {+ "bm25": 0.0911,+ "jev_median": 0.6448+ },+ "1415": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "1416": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "1441": {+ "bm25": 0.5376,+ "jev_median": 1.0+ },+ "1451": {+ "bm25": 0.6309,+ "jev_median": 0.6309+ },+ "1469": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "1530": {+ "bm25": 0.1413,+ "jev_median": 0.4693+ },+ "1670": {+ "bm25": 0.7654,+ "jev_median": 0.7039+ },+ "1676": {+ "bm25": 0.5585,+ "jev_median": 0.5654+ },+ "1736": {+ "bm25": 0.1696,+ "jev_median": 0.3392+ },+ "1748": {+ "bm25": 0.4441,+ "jev_median": 0.7654+ },+ "1783": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "1812": {+ "bm25": 0.2044,+ "jev_median": 0.2184+ },+ "1815": {+ "bm25": 0.5,+ "jev_median": 1.0+ },+ "1819": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "1824": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "1826": {+ "bm25": 0.9829,+ "jev_median": 0.9829+ },+ "1832": {+ "bm25": 0.6309,+ "jev_median": 0.4307+ },+ "1871": {+ "bm25": 0.281,+ "jev_median": 0.3633+ },+ "1877": {+ "bm25": 1.0,+ "jev_median": 0.5+ },+ "1889": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "1915": {+ "bm25": 0.2641,+ "jev_median": 0.6131+ },+ "1920": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "1933": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "1948": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "1994": {+ "bm25": 0.107,+ "jev_median": 0.5087+ },+ "2010": {+ "bm25": 0.1815,+ "jev_median": 0.5307+ },+ "2051": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2070": {+ "bm25": 0.4693,+ "jev_median": 0.4693+ },+ "2075": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2076": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2088": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2108": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2118": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2154": {+ "bm25": 0.5,+ "jev_median": 1.0+ },+ "2181": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2183": {+ "bm25": 0.1909,+ "jev_median": 0.1513+ },+ "2204": {+ "bm25": 0.539,+ "jev_median": 0.6479+ },+ "2264": {+ "bm25": 0.3869,+ "jev_median": 0.6131+ },+ "2296": {+ "bm25": 0.1175,+ "jev_median": 0.3833+ },+ "2306": {+ "bm25": 0.1934,+ "jev_median": 0.6131+ },+ "2316": {+ "bm25": 0.3869,+ "jev_median": 0.6131+ },+ "2318": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2330": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "2334": {+ "bm25": 0.2346,+ "jev_median": 0.7654+ },+ "2348": {+ "bm25": 0.359,+ "jev_median": 0.5638+ },+ "2376": {+ "bm25": 0.4053,+ "jev_median": 0.8359+ },+ "2383": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "2384": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2385": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "2388": {+ "bm25": 0.3333,+ "jev_median": 1.0+ },+ "2395": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2398": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2399": {+ "bm25": 0.3333,+ "jev_median": 1.0+ },+ "2400": {+ "bm25": 0.2346,+ "jev_median": 0.7039+ },+ "2407": {+ "bm25": 0.3904,+ "jev_median": 0.2463+ },+ "2416": {+ "bm25": 0.3904,+ "jev_median": 0.8319+ },+ "2423": {+ "bm25": 0.4123,+ "jev_median": 0.2918+ },+ "2443": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2445": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "2460": {+ "bm25": 0.3378,+ "jev_median": 0.7654+ },+ "2465": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2472": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2486": {+ "bm25": 1.0,+ "jev_median": 0.6309+ },+ "2498": {+ "bm25": 0.0,+ "jev_median": 0.5+ },+ "2513": {+ "bm25": 0.0,+ "jev_median": 0.6309+ },+ "2516": {+ "bm25": 0.3904,+ "jev_median": 0.5856+ },+ "2549": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2551": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2568": {+ "bm25": 0.0867,+ "jev_median": 0.2749+ },+ "2579": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2580": {+ "bm25": 0.7654,+ "jev_median": 0.7654+ },+ "2587": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "2589": {+ "bm25": 0.8194,+ "jev_median": 0.8711+ },+ "2590": {+ "bm25": 0.1564,+ "jev_median": 0.4693+ },+ "2593": {+ "bm25": 0.4693,+ "jev_median": 0.4693+ },+ "2598": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2648": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "2676": {+ "bm25": 0.0,+ "jev_median": 1.0+ },+ "2685": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2695": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "2713": {+ "bm25": 0.2372,+ "jev_median": 0.3066+ },+ "2724": {+ "bm25": 0.2641,+ "jev_median": 0.6131+ },+ "2737": {+ "bm25": 0.3869,+ "jev_median": 1.0+ },+ "2747": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2749": {+ "bm25": 0.6131,+ "jev_median": 1.0+ },+ "2790": {+ "bm25": 0.1374,+ "jev_median": 0.3543+ },+ "2801": {+ "bm25": 0.8175,+ "jev_median": 1.0+ },+ "2856": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2857": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2880": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "2885": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2891": {+ "bm25": 0.3333,+ "jev_median": 1.0+ },+ "2895": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2903": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2923": {+ "bm25": 0.3392,+ "jev_median": 0.3392+ },+ "2964": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "2968": {+ "bm25": 0.5803,+ "jev_median": 0.6509+ },+ "2994": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3006": {+ "bm25": 0.0,+ "jev_median": 0.0875+ },+ "3008": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3014": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3033": {+ "bm25": 0.2044,+ "jev_median": 0.6131+ },+ "3039": {+ "bm25": 0.3155,+ "jev_median": 0.3333+ },+ "3049": {+ "bm25": 0.0,+ "jev_median": 0.3904+ },+ "3051": {+ "bm25": 0.8316,+ "jev_median": 1.0+ },+ "3067": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3085": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3091": {+ "bm25": 0.0867,+ "jev_median": 0.5857+ },+ "3103": {+ "bm25": 0.5,+ "jev_median": 1.0+ },+ "3115": {+ "bm25": 0.4935,+ "jev_median": 0.6448+ },+ "3125": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3148": {+ "bm25": 0.5087,+ "jev_median": 0.5531+ },+ "3149": {+ "bm25": 0.1846,+ "jev_median": 0.2641+ },+ "3177": {+ "bm25": 0.148,+ "jev_median": 0.148+ },+ "3179": {+ "bm25": 0.3066,+ "jev_median": 0.8772+ },+ "3186": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "3189": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3254": {+ "bm25": 0.4693,+ "jev_median": 0.7654+ },+ "3264": {+ "bm25": 0.0,+ "jev_median": 0.3904+ },+ "3357": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "3369": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3394": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3404": {+ "bm25": 0.1063,+ "jev_median": 0.2749+ },+ "3405": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3446": {+ "bm25": 0.2339,+ "jev_median": 0.5873+ },+ "3451": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3453": {+ "bm25": 0.4911,+ "jev_median": 1.0+ },+ "3480": {+ "bm25": 0.7877,+ "jev_median": 0.8319+ },+ "3490": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3500": {+ "bm25": 0.1989,+ "jev_median": 0.4539+ },+ "3503": {+ "bm25": 0.2749,+ "jev_median": 0.5857+ },+ "3512": {+ "bm25": 0.3066,+ "jev_median": 0.6131+ },+ "3528": {+ "bm25": 0.4307,+ "jev_median": 1.0+ },+ "3530": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3534": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3566": {+ "bm25": 0.2961,+ "jev_median": 0.4693+ },+ "3569": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3594": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3612": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3615": {+ "bm25": 0.6131,+ "jev_median": 0.3869+ },+ "3625": {+ "bm25": 0.0,+ "jev_median": 0.4693+ },+ "3682": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3683": {+ "bm25": 0.1303,+ "jev_median": 0.6106+ },+ "3694": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "3724": {+ "bm25": 0.0,+ "jev_median": 0.2201+ },+ "3735": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3759": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3767": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3771": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3781": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "3789": {+ "bm25": 0.7757,+ "jev_median": 0.8048+ },+ "3791": {+ "bm25": 0.4693,+ "jev_median": 0.4693+ },+ "3801": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "3822": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3829": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3830": {+ "bm25": 0.301,+ "jev_median": 1.0+ },+ "3837": {+ "bm25": 0.6309,+ "jev_median": 0.5+ },+ "3859": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3875": {+ "bm25": 0.6241,+ "jev_median": 1.0+ },+ "3888": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3909": {+ "bm25": 0.0734,+ "jev_median": 0.3301+ },+ "3932": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "3934": {+ "bm25": 0.1391,+ "jev_median": 0.3904+ },+ "3995": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4007": {+ "bm25": 0.4693,+ "jev_median": 0.7654+ },+ "4011": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4019": {+ "bm25": 0.3392,+ "jev_median": 0.1208+ },+ "4031": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4037": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4047": {+ "bm25": 0.9447,+ "jev_median": 0.955+ },+ "4071": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4084": {+ "bm25": 0.1846,+ "jev_median": 0.3869+ },+ "4102": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4103": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4105": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4116": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "4125": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4142": {+ "bm25": 0.1301,+ "jev_median": 0.4382+ },+ "4153": {+ "bm25": 0.0,+ "jev_median": 0.6309+ },+ "4179": {+ "bm25": 0.0,+ "jev_median": 0.4307+ },+ "4188": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4205": {+ "bm25": 0.0,+ "jev_median": 0.4693+ },+ "4233": {+ "bm25": 0.674,+ "jev_median": 0.7227+ },+ "4265": {+ "bm25": 0.1909,+ "jev_median": 0.6448+ },+ "4286": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "4289": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4306": {+ "bm25": 0.5294,+ "jev_median": 0.5414+ },+ "4312": {+ "bm25": 0.6725,+ "jev_median": 0.7709+ },+ "4335": {+ "bm25": 0.6367,+ "jev_median": 0.6367+ },+ "4339": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "4394": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4409": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4411": {+ "bm25": 0.0,+ "jev_median": 0.4693+ },+ "4414": {+ "bm25": 0.6131,+ "jev_median": 0.3869+ },+ "4415": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4433": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4447": {+ "bm25": 1.0,+ "jev_median": 0.5+ },+ "4464": {+ "bm25": 0.0,+ "jev_median": 1.0+ },+ "4465": {+ "bm25": 0.0,+ "jev_median": 0.6364+ },+ "4484": {+ "bm25": 0.0,+ "jev_median": 1.0+ },+ "4499": {+ "bm25": 0.148,+ "jev_median": 0.7039+ },+ "4500": {+ "bm25": 0.0,+ "jev_median": 0.3869+ },+ "4504": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "4514": {+ "bm25": 0.1513,+ "jev_median": 0.2388+ },+ "4523": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4539": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4571": {+ "bm25": 0.4693,+ "jev_median": 0.7654+ },+ "4600": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4605": {+ "bm25": 0.1063,+ "jev_median": 0.3665+ },+ "4615": {+ "bm25": 0.1009,+ "jev_median": 0.545+ },+ "4640": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4641": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4678": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4681": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "4700": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "4714": {+ "bm25": 0.6984,+ "jev_median": 0.6096+ },+ "4756": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4767": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4775": {+ "bm25": 0.0,+ "jev_median": 0.6309+ },+ "4777": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4785": {+ "bm25": 0.0,+ "jev_median": 1.0+ },+ "4804": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4813": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4823": {+ "bm25": 0.4693,+ "jev_median": 0.4693+ },+ "4827": {+ "bm25": 0.0,+ "jev_median": 1.0+ },+ "4837": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4844": {+ "bm25": 0.5714,+ "jev_median": 1.0+ },+ "4845": {+ "bm25": 0.3045,+ "jev_median": 0.9469+ },+ "4846": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4863": {+ "bm25": 0.3501,+ "jev_median": 0.9675+ },+ "4865": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4920": {+ "bm25": 0.6364,+ "jev_median": 1.0+ },+ "4942": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4946": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "4955": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4962": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "4968": {+ "bm25": 0.5,+ "jev_median": 0.3333+ },+ "4981": {+ "bm25": 0.6508,+ "jev_median": 0.7654+ },+ "4999": {+ "bm25": 0.3904,+ "jev_median": 0.5294+ },+ "5021": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5030": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5045": {+ "bm25": 1.0,+ "jev_median": 0.5+ },+ "5054": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5061": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5064": {+ "bm25": 0.3066,+ "jev_median": 1.0+ },+ "5067": {+ "bm25": 0.5,+ "jev_median": 0.6309+ },+ "5080": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5083": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5085": {+ "bm25": 0.301,+ "jev_median": 1.0+ },+ "5086": {+ "bm25": 1.0,+ "jev_median": 0.4307+ },+ "5090": {+ "bm25": 0.0,+ "jev_median": 0.2641+ },+ "5125": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "5134": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5150": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5155": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "5172": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5178": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5185": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5196": {+ "bm25": 0.1564,+ "jev_median": 0.4693+ },+ "5206": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5228": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5231": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5241": {+ "bm25": 0.3392,+ "jev_median": 0.1312+ },+ "5254": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5255": {+ "bm25": 0.9325,+ "jev_median": 0.9325+ },+ "5264": {+ "bm25": 0.5087,+ "jev_median": 0.5296+ },+ "5271": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5331": {+ "bm25": 0.3869,+ "jev_median": 1.0+ },+ "5343": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5347": {+ "bm25": 0.4307,+ "jev_median": 1.0+ },+ "5356": {+ "bm25": 0.0,+ "jev_median": 0.3392+ },+ "5369": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5374": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5380": {+ "bm25": 0.6309,+ "jev_median": 1.0+ },+ "5402": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5410": {+ "bm25": 0.1564,+ "jev_median": 0.2961+ },+ "5422": {+ "bm25": 0.0,+ "jev_median": 0.3904+ },+ "5427": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5460": {+ "bm25": 0.3392,+ "jev_median": 0.3392+ },+ "5464": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5503": {+ "bm25": 0.2044,+ "jev_median": 0.6131+ },+ "5505": {+ "bm25": 0.6309,+ "jev_median": 1.0+ },+ "5511": {+ "bm25": 0.0,+ "jev_median": 0.2201+ },+ "5534": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5549": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5585": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5592": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "5616": {+ "bm25": 0.0,+ "jev_median": 0.5+ },+ "5620": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5646": {+ "bm25": 0.7039,+ "jev_median": 1.0+ },+ "5653": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "5683": {+ "bm25": 0.8503,+ "jev_median": 0.9197+ },+ "5710": {+ "bm25": 0.0,+ "jev_median": 0.6309+ },+ "5741": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5763": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5782": {+ "bm25": 0.3392,+ "jev_median": 0.3008+ },+ "5790": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5808": {+ "bm25": 0.0,+ "jev_median": 1.0+ },+ "5853": {+ "bm25": 0.0731,+ "jev_median": 0.3619+ },+ "5862": {+ "bm25": 0.1128,+ "jev_median": 0.0+ },+ "5888": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "5903": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5906": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5940": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "5951": {+ "bm25": 0.2495,+ "jev_median": 0.42+ },+ "5970": {+ "bm25": 0.0,+ "jev_median": 0.2961+ },+ "5981": {+ "bm25": 0.7227,+ "jev_median": 0.8688+ },+ "5993": {+ "bm25": 0.224,+ "jev_median": 0.2985+ },+ "6002": {+ "bm25": 0.4807,+ "jev_median": 0.7538+ },+ "6004": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6005": {+ "bm25": 0.4252,+ "jev_median": 0.3679+ },+ "6009": {+ "bm25": 0.2891,+ "jev_median": 0.6309+ },+ "6041": {+ "bm25": 0.7495,+ "jev_median": 0.1301+ },+ "6080": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "6110": {+ "bm25": 0.4773,+ "jev_median": 0.57+ },+ "6121": {+ "bm25": 0.2044,+ "jev_median": 0.6131+ },+ "6122": {+ "bm25": 0.4693,+ "jev_median": 0.4693+ },+ "6131": {+ "bm25": 0.3786,+ "jev_median": 0.5077+ },+ "6133": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "6142": {+ "bm25": 0.0,+ "jev_median": 0.6131+ },+ "6146": {+ "bm25": 0.0,+ "jev_median": 0.6131+ },+ "6199": {+ "bm25": 0.0,+ "jev_median": 0.3904+ },+ "6219": {+ "bm25": 1.0,+ "jev_median": 0.9197+ },+ "6221": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6252": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6262": {+ "bm25": 0.1564,+ "jev_median": 0.4693+ },+ "6278": {+ "bm25": 0.6053,+ "jev_median": 0.0+ },+ "6395": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6410": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6420": {+ "bm25": 0.6131,+ "jev_median": 0.3869+ },+ "6441": {+ "bm25": 0.3869,+ "jev_median": 0.0+ },+ "6467": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6468": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "6479": {+ "bm25": 0.7977,+ "jev_median": 0.8772+ },+ "6525": {+ "bm25": 0.5307,+ "jev_median": 0.4982+ },+ "6554": {+ "bm25": 0.2184,+ "jev_median": 0.6131+ },+ "6562": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6611": {+ "bm25": 0.2372,+ "jev_median": 0.8772+ },+ "6612": {+ "bm25": 0.1413,+ "jev_median": 0.4693+ },+ "6625": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6629": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6635": {+ "bm25": 0.1952,+ "jev_median": 0.3904+ },+ "6644": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6647": {+ "bm25": 0.0,+ "jev_median": 0.4693+ },+ "6668": {+ "bm25": 0.3869,+ "jev_median": 0.2184+ },+ "6679": {+ "bm25": 0.1815,+ "jev_median": 0.7654+ },+ "6683": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6713": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "6715": {+ "bm25": 0.0,+ "jev_median": 1.0+ },+ "6746": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6787": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6792": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6800": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6803": {+ "bm25": 0.2685,+ "jev_median": 0.7087+ },+ "6807": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "6814": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6832": {+ "bm25": 0.0,+ "jev_median": 0.2641+ },+ "6835": {+ "bm25": 0.0,+ "jev_median": 0.6131+ },+ "6849": {+ "bm25": 1.0,+ "jev_median": 0.4307+ },+ "6862": {+ "bm25": 0.301,+ "jev_median": 1.0+ },+ "6867": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6875": {+ "bm25": 0.4307,+ "jev_median": 0.0+ },+ "6890": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6891": {+ "bm25": 0.4693,+ "jev_median": 0.2961+ },+ "6896": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6901": {+ "bm25": 0.3066,+ "jev_median": 0.3869+ },+ "6907": {+ "bm25": 0.0,+ "jev_median": 0.5+ },+ "6909": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6959": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "6985": {+ "bm25": 0.6309,+ "jev_median": 1.0+ },+ "7017": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7068": {+ "bm25": 0.8066,+ "jev_median": 0.3066+ },+ "7071": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7080": {+ "bm25": 0.3066,+ "jev_median": 1.0+ },+ "7096": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7098": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7105": {+ "bm25": 0.3333,+ "jev_median": 1.0+ },+ "7109": {+ "bm25": 0.6049,+ "jev_median": 0.4982+ },+ "7124": {+ "bm25": 0.2463,+ "jev_median": 0.1952+ },+ "7141": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7145": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7178": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7188": {+ "bm25": 0.0,+ "jev_median": 0.6309+ },+ "7205": {+ "bm25": 0.5,+ "jev_median": 1.0+ },+ "7206": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7218": {+ "bm25": 0.3333,+ "jev_median": 1.0+ },+ "7221": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7269": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7279": {+ "bm25": 0.0,+ "jev_median": 0.3904+ },+ "7295": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7311": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7326": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7329": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7344": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "7345": {+ "bm25": 0.0,+ "jev_median": 0.2961+ },+ "7377": {+ "bm25": 0.3869,+ "jev_median": 1.0+ },+ "7431": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7441": {+ "bm25": 0.2021,+ "jev_median": 0.1564+ },+ "7445": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7448": {+ "bm25": 0.6309,+ "jev_median": 1.0+ },+ "7456": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7463": {+ "bm25": 0.0,+ "jev_median": 0.3392+ },+ "7467": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "7484": {+ "bm25": 0.5,+ "jev_median": 1.0+ },+ "7509": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7512": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7513": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7529": {+ "bm25": 0.1934,+ "jev_median": 0.6131+ },+ "7533": {+ "bm25": 0.1934,+ "jev_median": 0.2372+ },+ "7534": {+ "bm25": 0.4693,+ "jev_median": 0.4693+ },+ "7590": {+ "bm25": 0.3066,+ "jev_median": 1.0+ },+ "7592": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7594": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7622": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7633": {+ "bm25": 0.5079,+ "jev_median": 0.5856+ },+ "7674": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7700": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7702": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7705": {+ "bm25": 0.2346,+ "jev_median": 0.6714+ },+ "7734": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "7747": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7754": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7758": {+ "bm25": 0.0,+ "jev_median": 1.0+ },+ "7801": {+ "bm25": 0.7904,+ "jev_median": 0.8503+ },+ "7803": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7823": {+ "bm25": 0.6364,+ "jev_median": 0.6714+ },+ "7876": {+ "bm25": 0.2891,+ "jev_median": 0.6309+ },+ "7879": {+ "bm25": 0.0,+ "jev_median": 0.4693+ },+ "7880": {+ "bm25": 0.0,+ "jev_median": 0.3869+ },+ "7911": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "7925": {+ "bm25": 0.0,+ "jev_median": 0.1909+ },+ "7928": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "7936": {+ "bm25": 0.2837,+ "jev_median": 0.6183+ },+ "7992": {+ "bm25": 0.3066,+ "jev_median": 0.6131+ },+ "8002": {+ "bm25": 0.1564,+ "jev_median": 0.4693+ },+ "8005": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8013": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8017": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8034": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8040": {+ "bm25": 1.0,+ "jev_median": 0.5+ },+ "8072": {+ "bm25": 0.5307,+ "jev_median": 0.3028+ },+ "8079": {+ "bm25": 0.2961,+ "jev_median": 0.4693+ },+ "8102": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8116": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8121": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8202": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8230": {+ "bm25": 0.0,+ "jev_median": 0.6131+ },+ "8247": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8271": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "8275": {+ "bm25": 0.0,+ "jev_median": 0.3869+ },+ "8296": {+ "bm25": 0.5803,+ "jev_median": 0.9197+ },+ "8332": {+ "bm25": 0.0,+ "jev_median": 0.7654+ },+ "8351": {+ "bm25": 0.3066,+ "jev_median": 0.6131+ },+ "8378": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8456": {+ "bm25": 0.4693,+ "jev_median": 0.4693+ },+ "8475": {+ "bm25": 0.4483,+ "jev_median": 0.5857+ },+ "8507": {+ "bm25": 0.1357,+ "jev_median": 0.4693+ },+ "8512": {+ "bm25": 0.2891,+ "jev_median": 0.6309+ },+ "8513": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8532": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8537": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8539": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8544": {+ "bm25": 0.2372,+ "jev_median": 0.6131+ },+ "8592": {+ "bm25": 0.2346,+ "jev_median": 0.4693+ },+ "8632": {+ "bm25": 0.2372,+ "jev_median": 0.3066+ },+ "8635": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8702": {+ "bm25": 0.7654,+ "jev_median": 0.6364+ },+ "8779": {+ "bm25": 0.1846,+ "jev_median": 0.1846+ },+ "8789": {+ "bm25": 0.7654,+ "jev_median": 0.7654+ },+ "8795": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "8832": {+ "bm25": 0.6309,+ "jev_median": 1.0+ },+ "8834": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8855": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8874": {+ "bm25": 0.3155,+ "jev_median": 0.3562+ },+ "8934": {+ "bm25": 0.3066,+ "jev_median": 0.1846+ },+ "8937": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8947": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "8959": {+ "bm25": 0.0,+ "jev_median": 0.5+ },+ "8970": {+ "bm25": 0.2372,+ "jev_median": 0.6131+ },+ "8974": {+ "bm25": 0.0798,+ "jev_median": 0.2529+ },+ "8982": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9060": {+ "bm25": 0.2044,+ "jev_median": 0.6131+ },+ "9088": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9108": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9115": {+ "bm25": 0.1672,+ "jev_median": 0.4693+ },+ "9126": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9164": {+ "bm25": 0.0,+ "jev_median": 0.1815+ },+ "9174": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9188": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "9245": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9275": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9291": {+ "bm25": 0.2044,+ "jev_median": 0.6131+ },+ "9296": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9329": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "9332": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9381": {+ "bm25": 0.1846,+ "jev_median": 0.6131+ },+ "9385": {+ "bm25": 0.5,+ "jev_median": 0.3562+ },+ "9391": {+ "bm25": 0.0,+ "jev_median": 0.3904+ },+ "9403": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9481": {+ "bm25": 0.6241,+ "jev_median": 1.0+ },+ "9487": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9548": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9556": {+ "bm25": 0.6309,+ "jev_median": 1.0+ },+ "9565": {+ "bm25": 0.6131,+ "jev_median": 0.6131+ },+ "9598": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9617": {+ "bm25": 0.0,+ "jev_median": 0.7654+ },+ "9633": {+ "bm25": 0.2184,+ "jev_median": 0.3066+ },+ "9643": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "9644": {+ "bm25": 0.3155,+ "jev_median": 0.6309+ },+ "9646": {+ "bm25": 0.6131,+ "jev_median": 0.0+ },+ "9668": {+ "bm25": 0.6714,+ "jev_median": 0.7654+ },+ "9701": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9733": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9735": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9737": {+ "bm25": 1.0,+ "jev_median": 0.852+ },+ "9771": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9808": {+ "bm25": 0.1672,+ "jev_median": 0.1357+ },+ "9824": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9871": {+ "bm25": 0.3904,+ "jev_median": 0.3904+ },+ "9882": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9925": {+ "bm25": 0.6131,+ "jev_median": 0.6241+ },+ "9929": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "9961": {+ "bm25": 0.6309,+ "jev_median": 0.5+ },+ "9979": {+ "bm25": 0.3904,+ "jev_median": 0.1681+ },+ "10034": {+ "bm25": 0.2641,+ "jev_median": 0.3869+ },+ "10039": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "10109": {+ "bm25": 0.3933,+ "jev_median": 0.2651+ },+ "10122": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "10136": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "10137": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "10152": {+ "bm25": 0.5,+ "jev_median": 1.0+ },+ "10183": {+ "bm25": 0.1846,+ "jev_median": 0.2641+ },+ "10213": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "10246": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "10267": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "10414": {+ "bm25": 0.8316,+ "jev_median": 1.0+ },+ "10447": {+ "bm25": 0.148,+ "jev_median": 0.4693+ },+ "10462": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "10482": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "10497": {+ "bm25": 0.4252,+ "jev_median": 0.5321+ },+ "10526": {+ "bm25": 0.1846,+ "jev_median": 0.0+ },+ "10547": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "10558": {+ "bm25": 0.2372,+ "jev_median": 0.3066+ },+ "10596": {+ "bm25": 0.8539,+ "jev_median": 0.786+ },+ "10601": {+ "bm25": 1.0,+ "jev_median": 0.5+ },+ "10628": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "10639": {+ "bm25": 0.1461,+ "jev_median": 0.3392+ },+ "10645": {+ "bm25": 0.3066,+ "jev_median": 0.2372+ },+ "10674": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "10710": {+ "bm25": 0.0,+ "jev_median": 1.0+ },+ "10734": {+ "bm25": 0.6106,+ "jev_median": 0.6106+ },+ "10792": {+ "bm25": 0.1846,+ "jev_median": 0.5912+ },+ "10808": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "10809": {+ "bm25": 0.2961,+ "jev_median": 0.4693+ },+ "10812": {+ "bm25": 0.5294,+ "jev_median": 0.7366+ },+ "10827": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "10845": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "10912": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "10932": {+ "bm25": 0.6309,+ "jev_median": 1.0+ },+ "10975": {+ "bm25": 0.0,+ "jev_median": 0.0+ },+ "10979": {+ "bm25": 0.1952,+ "jev_median": 0.5585+ },+ "10994": {+ "bm25": 0.1772,+ "jev_median": 0.6509+ },+ "11039": {+ "bm25": 0.0843,+ "jev_median": 0.1776+ },+ "11054": {+ "bm25": 1.0,+ "jev_median": 1.0+ },+ "11088": {+ "bm25": 0.0,+ "jev_median": 1.0+ }+ }+}\ No newline at end of filediff --git a/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/README.md b/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/README.mdnew file mode 100644index 000000000..65d988042--- /dev/null+++ b/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/README.md@@ -0,0 +1,70 @@+# AG News — 4-way topic classification, 120k-deep history (2026-09-20)++The scale end of the vote's range: **120,000 labelled train items** indexed,+the full **7,600-item test set** classified, k = 10, rank-weighted 1/(i+1),+BM25 neighbours only — no model of any kind, no embedder, the default lexical+node. This is the workload the System One wire answers (`/v1/systemone`,+`/_decide`), and it is the workload that motivated the engine's WAND fast+path (engine cf6a9989): a full news item is a ~40-clause disjunction whose+match set is most of the index.++## Dataset and provenance++**AG News** via HuggingFace `fancyzhx/ag_news` — the standard 4-class topic+classification corpus (World / Sports / Business / SciTech), 120,000 train /+7,600 test, originally by Zhang et al., *Character-level Convolutional+Networks for Text Classification*, NeurIPS 2015+(https://arxiv.org/abs/1509.01626). Downloaded as parquet from+https://huggingface.co/datasets/fancyzhx/ag_news and converted to TSV+(`label\ttext`) with pandas+pyarrow; the conversion script is not checked in+because it is two lines — the dataset card's own `train`/`test` splits used+verbatim, no re-cutting, no shuffling.++## Headline numbers (all three runs, identical index, identical items)++| Binary | ms/item | Accuracy | ECE | ≥0.8 conf | …and right | no-hit |+|---|---:|---:|---:|---:|---:|---:|+| pre-WAND (284d1f8f) | 68.7 | 0.9182 | 0.019 | 83.1% | 0.9669 | 0 |+| WAND first cut | 51.2 | 0.9182 | 0.019 | 83.1% | 0.9669 | 0 |+| WAND + alloc-free loop (cf6a9989) | **34.2** | 0.9182 | 0.019 | 83.1% | 0.9669 | 0 |++The point of the three rows: **2.0× end-to-end latency, bit-identical+decisions** — same 7,600 labels, same confidence bands, same ECE. Keep-alive+HTTP, one connection, distinct items (no cache reuse — every item is new,+which is the real workload). Run logs: `run-2026-09-20.txt`.++Same-box wire comparison for the same workload class: the hosted Jev API+measured p50 1,230 ms per item on SciFact-scale shortlists (see+`../../../beir-hybrid/results/2026-09-20-rerank-full/`); the local vote on+120k history answers `/v1/systemone` at p50 42.7 ms (300 distinct items,+final binary). Different tools — the vote needs this labelled history, Jev+needs nothing — so this is a cost/latency/egress contrast, not a quality+race (Jev was not run on AG News; no Jev row belongs on this dataset).++## What the numbers mean++- **92% of 4-way routing decisions made locally, free, with no egress** — and+ 83% of them at ≥0.8 confidence, where the vote is right 96.7% of the time.+ The ECE of 0.019 means the confidence can gate: send the uncertain 17% to+ a judge or a person.+- **Why AG News is easy for retrieval**: the four topics have disjoint+ vocabularies (scoreboard, ticker, spacecraft). The vote is not+ understanding news; it is finding the ten most lexically similar+ headlines in 120k history and reading their labels. Banking77 (77 near-+ duplicate intents) is the hard in-domain case; this is the high-volume+ cheap one.+- **Latency scales with the match set, not the corpus**: 120k docs, but the+ cost driver is that a 40-token item matches nearly everything, so nearly+ every doc is scored. Shorter texts against the same index are far under+ this (SMS: 0.9 ms/item at 4k history). No TB-scale claim is being made+ here; this is one shape at one scale, measured.++## Reproduce++```sh+# 1. dataset: HF fancyzhx/ag_news -> TSV (label<TAB>text), train + test+xerj --insecure --port 9460 --data-dir ./agnews-data &+python3 agnews_vote.py # indexes 120k train, runs 7,600 test items+# A/B against an older binary: boot both, then+python3 vote_eval.py <port> # keep-alive eval, prints acc/ECE/ms-item+```diff --git a/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/agnews_vote.py b/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/agnews_vote.pynew file mode 100644index 000000000..e459bef85--- /dev/null+++ b/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/agnews_vote.py@@ -0,0 +1,45 @@+"""AG News (HF fancyzhx/ag_news): classify the 7,600 test items by a+rank-weighted k=10 BM25 vote over the 120,000 labelled train items.+Same methodology as benchmarks/decisions-as-retrieval/eval.py."""+import json, time, urllib.request, collections+U = "http://localhost:9460"; K = 10+def req(m, p, b=None, ct="application/json"):+ d = b if isinstance(b, (bytes, type(None))) else json.dumps(b).encode()+ r = urllib.request.Request(U + p, data=d, method=m, headers={"content-type": ct})+ try: return json.loads(urllib.request.urlopen(r, timeout=300).read())+ except urllib.error.HTTPError as e: return {"_err": e.code}++req("DELETE", "/agnews")+req("PUT", "/agnews", {"mappings": {"properties": {"text": {"type": "text"}, "label": {"type": "keyword"}}}})+lines = []+for i, l in enumerate(open("agnews-train.tsv").readlines()[1:]):+ lab, txt = l.rstrip("\n").split("\t", 1)+ lines.append(json.dumps({"index": {"_index": "agnews", "_id": str(i)}}))+ lines.append(json.dumps({"text": txt, "label": lab}))+ if len(lines) >= 4000:+ req("POST", "/_bulk", ("\n".join(lines) + "\n").encode(), "application/x-ndjson"); lines = []+ if i % 20000 == 0: print("indexed", i, flush=True)+if lines: req("POST", "/_bulk", ("\n".join(lines) + "\n").encode(), "application/x-ndjson")+print(req("POST", "/agnews/_refresh")); print("count:", req("GET", "/agnews/_count")["count"])++test = [l.rstrip("\n").split("\t", 1) for l in open("agnews-test.tsv").readlines()[1:]]+print("test items:", len(test))+cc = []; t0 = time.time(); none = 0+for gold, txt in test:+ r = req("POST", "/agnews/_search", {"size": K, "_source": ["label"], "query": {"match": {"text": txt}}})+ hs = r.get("hits", {}).get("hits", [])+ if not hs: none += 1; cc.append((0.0, 0)); continue+ v = collections.Counter()+ for i, h in enumerate(hs): v[h["_source"]["label"]] += 1.0 / (i + 1)+ lab, w = v.most_common(1)[0]; conf = w / sum(v.values())+ cc.append((conf, 1 if lab == gold else 0))+acc = sum(k for _, k in cc) / len(cc); ms = (time.time() - t0) / len(cc) * 1000+def ece(cc, bins=10):+ n = len(cc); tot = 0.0+ for b in range(bins):+ xs = [(c, k) for c, k in cc if b / bins < c <= (b + 1) / bins or (b == 0 and c == 0)]+ if xs: tot += len(xs) / n * abs(sum(k for _, k in xs) / len(xs) - sum(c for c, _ in xs) / len(xs))+ return tot+hi = [k for c, k in cc if c >= 0.8]+print(f"AG News 4-way — BM25 vote, k={K}")+print(f" acc={acc:.4f} ECE={ece(cc):.3f} conf>=0.8: {len(hi)/len(cc):.1%} of items at {sum(hi)/max(len(hi),1):.4f} acc no-hit={none} {ms:.1f} ms/item")diff --git a/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/run-2026-09-20.txt b/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/run-2026-09-20.txtnew file mode 100644index 000000000..487f35a08--- /dev/null+++ b/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/run-2026-09-20.txt@@ -0,0 +1,48 @@+ 500/7600 70.04 ms/item+ 1000/7600 69.69 ms/item+ 1500/7600 69.74 ms/item+ 2000/7600 69.73 ms/item+ 2500/7600 69.57 ms/item+ 3000/7600 69.56 ms/item+ 3500/7600 69.81 ms/item+ 4000/7600 69.70 ms/item+ 4500/7600 69.34 ms/item+ 5000/7600 69.18 ms/item+ 5500/7600 69.10 ms/item+ 6000/7600 68.91 ms/item+ 6500/7600 68.85 ms/item+ 7000/7600 68.86 ms/item+ 7500/7600 68.75 ms/item+port 9460: acc=0.9182 no-hit=0 conf>=0.8: 83.1% at 0.9669 68.74 ms/item+ 500/7600 54.37 ms/item+ 1000/7600 53.25 ms/item+ 1500/7600 52.93 ms/item+ 2000/7600 51.99 ms/item+ 2500/7600 51.97 ms/item+ 3000/7600 51.95 ms/item+ 3500/7600 52.18 ms/item+ 4000/7600 52.04 ms/item+ 4500/7600 51.96 ms/item+ 5000/7600 51.94 ms/item+ 5500/7600 51.97 ms/item+ 6000/7600 51.67 ms/item+ 6500/7600 51.50 ms/item+ 7000/7600 51.45 ms/item+ 7500/7600 51.23 ms/item+port 9480: acc=0.9182 no-hit=0 conf>=0.8: 83.1% at 0.9669 51.19 ms/item+ 500/7600 37.43 ms/item+ 1000/7600 36.34 ms/item+ 1500/7600 35.92 ms/item+ 2000/7600 35.17 ms/item+ 2500/7600 34.86 ms/item+ 3000/7600 34.80 ms/item+ 3500/7600 34.91 ms/item+ 4000/7600 34.79 ms/item+ 4500/7600 34.64 ms/item+ 5000/7600 34.55 ms/item+ 5500/7600 34.50 ms/item+ 6000/7600 34.33 ms/item+ 6500/7600 34.29 ms/item+ 7000/7600 34.30 ms/item+ 7500/7600 34.20 ms/item+port 9480: acc=0.9182 no-hit=0 conf>=0.8: 83.1% at 0.9669 34.19 ms/itemdiff --git a/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/vote_eval.py b/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/vote_eval.pynew file mode 100644index 000000000..5c54710a1--- /dev/null+++ b/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/vote_eval.py@@ -0,0 +1,28 @@+"""AG News vote eval against one node, keep-alive, with progress.+Usage: vote_eval.py <port> — prints acc / no-hit / conf>=0.8 / ms-item."""+import http.client, json, time, collections, sys++port = int(sys.argv[1])+test = [l.rstrip("\n").split("\t", 1) for l in open("/root/xerj-bench/agnews-test.tsv").readlines()[1:]]+c = http.client.HTTPConnection("localhost", port, timeout=300)+cc = []; none = 0+t0 = time.time()+for n, (gold, txt) in enumerate(test):+ body = json.dumps({"size": 10, "_source": ["label"], "query": {"match": {"text": txt}}}).encode()+ c.request("POST", "/agnews/_search", body, {"content-type": "application/json"})+ r = json.loads(c.getresponse().read())+ hs = r.get("hits", {}).get("hits", [])+ if not hs:+ none += 1; cc.append((0.0, 0)); continue+ v = collections.Counter()+ for i, h in enumerate(hs): v[h["_source"]["label"]] += 1.0 / (i + 1)+ lab, w = v.most_common(1)[0]; conf = w / sum(v.values())+ cc.append((conf, 1 if lab == gold else 0))+ if (n + 1) % 500 == 0:+ el = time.time() - t0+ print(f" {n+1}/{len(test)} {el/(n+1)*1000:.2f} ms/item", flush=True)+c.close()+ms = (time.time() - t0) / len(cc) * 1000+acc = sum(k for _, k in cc) / len(cc)+hi = [k for c, k in cc if c >= 0.8]+print(f"port {port}: acc={acc:.4f} no-hit={none} conf>=0.8: {len(hi)/len(cc):.1%} at {sum(hi)/max(len(hi),1):.4f} {ms:.2f} ms/item")