Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

bench: raw log for the FiQA bm30+judge arm — re-run clean, 0.1650 x3 (spread 0.0000)

xerj-org526a1da32bbb
diff --git a/benchmarks/beir-hybrid/results/2026-09-30-judge-gate/fiqa-quality-bm30arms.log b/benchmarks/beir-hybrid/results/2026-09-30-judge-gate/fiqa-quality-bm30arms.log
new file mode 100644
index 000000000..1c364a98a
--- /dev/null
+++ b/benchmarks/beir-hybrid/results/2026-09-30-judge-gate/fiqa-quality-bm30arms.log
@@ -0,0 +1,14 @@
+index=fiqa queries=648 mode=quality size=30 url=http://127.0.0.1:9646
+docs=57638
+run 1 bm30 nDCG@10=0.2380 empty= 0
+run 1 bm30+judge nDCG@10=0.1650 empty= 0 judge_took_ms_p50=0 kept_avg=30.0 dropped_avg=0.0
+run 2 bm30 nDCG@10=0.2382 empty= 0
+run 2 bm30+judge nDCG@10=0.1650 empty= 0 judge_took_ms_p50=0 kept_avg=30.0 dropped_avg=0.0
+run 3 bm30 nDCG@10=0.2382 empty= 0
+run 3 bm30+judge nDCG@10=0.1650 empty= 0 judge_took_ms_p50=0 kept_avg=30.0 dropped_avg=0.0
+
+== per-arm means over runs, spread, and head-to-head vs hy30 ==
+bm30 runs=0.2380 0.2382 0.2382 mean=0.2381 spread=0.0002
+bm30+judge runs=0.1650 0.1650 0.1650 mean=0.1650 spread=0.0000
+hy30+judge vs hy30: skipped (an arm was not run — ARMS filter?)
+bm30+judge vs bm30: W/L/T = 86/209/353 mean per-query delta=-0.0731
diff --git a/benchmarks/beir-hybrid/results/2026-09-30-judge-gate/manifest.txt b/benchmarks/beir-hybrid/results/2026-09-30-judge-gate/manifest.txt
index 537d9d2a4..162c6041d 100644
--- a/benchmarks/beir-hybrid/results/2026-09-30-judge-gate/manifest.txt
+++ b/benchmarks/beir-hybrid/results/2026-09-30-judge-gate/manifest.txt
@@ -76,3 +76,16 @@ Incidents during the run (recorded, none change a gate number)
by SIGKILL of the old process + clean restart before ANY measurement;
SciFact's index replayed from WAL and counted 5,183 both before and
after. No measurement ran against the contended state.
+
+FiQA bm30-arms re-run (2026-09-30, verifier follow-up):
+ The first FiQA quality attempt aborted mid-bm30+judge (node died under the
+ runaway hybrid client — fiqa-quality-aborted.log), leaving the cited
+ 0.1650 without a raw log. Re-run clean: same fiqa.zip (sha256 above),
+ fresh throwaway node --embed-mode lexical :9646, load.py mapping identical,
+ ARMS=bm30,bm30+judge (hybrid arms excluded: 13-26 s/query, issue #1091).
+ bm30 0.2380/0.2382/0.2382 (run-1 warm-up wobble 0.0002; the aborted run's
+ single bm30 pass was 0.2382); bm30+judge 0.1650/0.1650/0.1650, spread
+ 0.0000, W/L/T 86/209/353, mean per-query delta -0.0731. Raw log:
+ fiqa-quality-bm30arms.log. The judge arm is embedder-independent (the
+ stage scores BM25 shortlists), which is why the lexical boot is valid for
+ these two arms; the hybrid arms were never measured on FiQA.