Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs: CRITICAL — XERJ read-perf wins were a query-cache mirage

xerj-org2ff3e9a0d3a9
diff --git a/demo/playbooks/CRITICAL_FINDING_read_perf_cache_mirage.md b/demo/playbooks/CRITICAL_FINDING_read_perf_cache_mirage.md
new file mode 100644
index 000000000..20b69548c
--- /dev/null
+++ b/demo/playbooks/CRITICAL_FINDING_read_perf_cache_mirage.md
@@ -0,0 +1,78 @@
+# CRITICAL FINDING (2026-07-01): XERJ read-perf wins were a query-cache mirage
+
+## What we believed
+Prior head-to-head benchmarks reported XERJ winning read latency vs
+Elasticsearch by 1.3–2.2× on p50 and even winning steady-state p99. The
+"beat ES on reads" story rested entirely on these numbers.
+
+## What is actually true
+Those benchmarks repeated the **same query** hundreds/thousands of times
+against a **static** index. XERJ's result cache (`query_cache`, keyed by
+`(query_hash, dataset_version)`) served every call after the first. We were
+measuring **cache hits**, not query execution.
+
+Measured directly (single client, NO write load, 500k docs, 16 flushed
+segments, `scratchpad/` diagnostics):
+
+| query | call #1 (uncached) | call #2/#3 (cached) |
+|---|--:|--:|
+| match_all size:10 | **2.28 s** | 0.00 s |
+| term(status:ok) size:10 | **2.42 s** | 0.00 s |
+| range size:10 (novel) | **2.2–2.4 s** | — |
+| terms agg (size:0) | **8.48 s** | 0.01 s |
+
+Every **distinct** query pays 2–8 s. ES does the same uncached in single-digit
+ms. So XERJ's true uncached read latency is ~1000× ES; it only looked fast
+because benchmarks hammered one cached query on a static dataset.
+
+This is also the real explanation for "reads collapse under write": a write
+bumps `dataset_version`, invalidating the query_cache, so every read reverts
+to the 2–8 s uncached path — under sustained write, reads hit 5–137 s.
+
+## Root cause (characterized, code-located)
+Two distinct O(N) paths, both masked by caches:
+
+1. **Hit materialization is O(total matches), not O(from+size).** Proof:
+ `match_all size:0` = **0.01 s** (matching + counting are fast) but
+ `match_all size:10` = **2.28 s**. Fetching 10 docs scans/decodes ALL
+ matches' stored `_source` then truncates. The code even documents that
+ `size:0` was special-cased away from the "full stored-doc scan" but
+ `size>0` still does it (`crates/xerj-engine/src/index.rs` ~4002–4055,
+ segment stored-scan; memtable side `all_docs_with_sources`/`all_doc_ids`
+ ~3826–3894).
+2. **Cold doc-values decode per segment for aggregations.** First agg over a
+ segment's column decodes the whole column (8 s for 16 segments), cached
+ after (`dv_cache`). Under write, new segments are cold → aggs slow again.
+
+CPU is idle (2–11 of 32 cores) during the stalls → not CPU-bound; it's the
+serial O(N) scan/decode + lock handoff, not parallel compute.
+
+## Why the harness now catches this
+`bench-matrix.mjs` (Phase 0) already added `track_total_hits=true` and a
+mixed read-under-write mode; the mixed mode is what exposed the collapse.
+To measure UNCACHED steady-state too, the harness should vary query params
+per iteration (novel term/range values) so the query_cache can't serve
+repeats — TODO before the next scorecard so read cells reflect reality.
+
+## The fix (new top priority — supersedes tail/ingest/disk)
+**F1. Bounded hit materialization** — collect only top-(from+size) doc refs
+(heap by score/sort), then fetch `_source` for those alone. Target: term/
+match_all/range size:10 from ~2.4 s → single-digit ms uncached. Biggest lever.
+
+**F2. Faster / precomputable agg doc-values** — keep decoded columns warm
+across flushes (additive, like P3.2) and/or decode lazily per-agg-field only;
+consider persisting decoded/bit-packed columns so cold-agg isn't O(column).
+
+**F3. Re-baseline honestly** — after F1/F2, re-run `bench-matrix.mjs` with
+per-iteration novel query params; report UNCACHED p50/p99. Only then are the
+read cells trustworthy.
+
+Guardrail unchanged: ES-YAML conformance 1326/0 at every step; this is core
+search-path surgery → worktree agent + hard gate + the `scratchpad/`
+uncached diagnostics as the win metric.
+
+## Honest status correction
+The project is NOT "reads already beat ES." It is: ingest near-parity (100k
+wins, 1M/8-client behind), disk 2× larger, and **reads are ~1000× slower
+uncached** — the read engine needs fundamental work before any "beats ES on
+reads" claim holds. Everything else is secondary to F1.