AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs: CRITICAL — XERJ read-perf wins were a query-cache mirage
xerj-org2ff3e9a0d3a9
diff --git a/demo/playbooks/CRITICAL_FINDING_read_perf_cache_mirage.md b/demo/playbooks/CRITICAL_FINDING_read_perf_cache_mirage.mdnew file mode 100644index 000000000..20b69548c--- /dev/null+++ b/demo/playbooks/CRITICAL_FINDING_read_perf_cache_mirage.md@@ -0,0 +1,78 @@+# CRITICAL FINDING (2026-07-01): XERJ read-perf wins were a query-cache mirage++## What we believed+Prior head-to-head benchmarks reported XERJ winning read latency vs+Elasticsearch by 1.3–2.2× on p50 and even winning steady-state p99. The+"beat ES on reads" story rested entirely on these numbers.++## What is actually true+Those benchmarks repeated the **same query** hundreds/thousands of times+against a **static** index. XERJ's result cache (`query_cache`, keyed by+`(query_hash, dataset_version)`) served every call after the first. We were+measuring **cache hits**, not query execution.++Measured directly (single client, NO write load, 500k docs, 16 flushed+segments, `scratchpad/` diagnostics):++| query | call #1 (uncached) | call #2/#3 (cached) |+|---|--:|--:|+| match_all size:10 | **2.28 s** | 0.00 s |+| term(status:ok) size:10 | **2.42 s** | 0.00 s |+| range size:10 (novel) | **2.2–2.4 s** | — |+| terms agg (size:0) | **8.48 s** | 0.01 s |++Every **distinct** query pays 2–8 s. ES does the same uncached in single-digit+ms. So XERJ's true uncached read latency is ~1000× ES; it only looked fast+because benchmarks hammered one cached query on a static dataset.++This is also the real explanation for "reads collapse under write": a write+bumps `dataset_version`, invalidating the query_cache, so every read reverts+to the 2–8 s uncached path — under sustained write, reads hit 5–137 s.++## Root cause (characterized, code-located)+Two distinct O(N) paths, both masked by caches:++1. **Hit materialization is O(total matches), not O(from+size).** Proof:+ `match_all size:0` = **0.01 s** (matching + counting are fast) but+ `match_all size:10` = **2.28 s**. Fetching 10 docs scans/decodes ALL+ matches' stored `_source` then truncates. The code even documents that+ `size:0` was special-cased away from the "full stored-doc scan" but+ `size>0` still does it (`crates/xerj-engine/src/index.rs` ~4002–4055,+ segment stored-scan; memtable side `all_docs_with_sources`/`all_doc_ids`+ ~3826–3894).+2. **Cold doc-values decode per segment for aggregations.** First agg over a+ segment's column decodes the whole column (8 s for 16 segments), cached+ after (`dv_cache`). Under write, new segments are cold → aggs slow again.++CPU is idle (2–11 of 32 cores) during the stalls → not CPU-bound; it's the+serial O(N) scan/decode + lock handoff, not parallel compute.++## Why the harness now catches this+`bench-matrix.mjs` (Phase 0) already added `track_total_hits=true` and a+mixed read-under-write mode; the mixed mode is what exposed the collapse.+To measure UNCACHED steady-state too, the harness should vary query params+per iteration (novel term/range values) so the query_cache can't serve+repeats — TODO before the next scorecard so read cells reflect reality.++## The fix (new top priority — supersedes tail/ingest/disk)+**F1. Bounded hit materialization** — collect only top-(from+size) doc refs+(heap by score/sort), then fetch `_source` for those alone. Target: term/+match_all/range size:10 from ~2.4 s → single-digit ms uncached. Biggest lever.++**F2. Faster / precomputable agg doc-values** — keep decoded columns warm+across flushes (additive, like P3.2) and/or decode lazily per-agg-field only;+consider persisting decoded/bit-packed columns so cold-agg isn't O(column).++**F3. Re-baseline honestly** — after F1/F2, re-run `bench-matrix.mjs` with+per-iteration novel query params; report UNCACHED p50/p99. Only then are the+read cells trustworthy.++Guardrail unchanged: ES-YAML conformance 1326/0 at every step; this is core+search-path surgery → worktree agent + hard gate + the `scratchpad/`+uncached diagnostics as the win metric.++## Honest status correction+The project is NOT "reads already beat ES." It is: ingest near-parity (100k+wins, 1M/8-client behind), disk 2× larger, and **reads are ~1000× slower+uncached** — the read engine needs fundamental work before any "beats ES on+reads" claim holds. Everything else is secondary to F1.