Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

site+engine: mixed-p99 root cause documented; ship memtable hardening

xerj-org583505d756a6
diff --git a/landing/benchmarks/index.html b/landing/benchmarks/index.html
index 73d0875c6..5054f88c6 100644
--- a/landing/benchmarks/index.html
+++ b/landing/benchmarks/index.html
@@ -11,7 +11,7 @@
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Big+Shoulders+Display:wght@400;700;800;900&family=Inter:wght@400;500;600;700&family=IBM+Plex+Sans:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;700&display=swap" rel="stylesheet">
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><rect width='100' height='100' fill='%230b0b0d'/><text x='50' y='68' font-size='60' font-family='Big Shoulders Display,sans-serif' font-weight='900' fill='%23ffc400' text-anchor='middle'>X</text></svg>">
-<link rel="stylesheet" href="/style.css?v=brand-ux2">
+<link rel="stylesheet" href="/style.css?v=rootcause1">
<style>
/* page-local: 5-column variant of the product .bench grid + verdict tones */
.bench-scroll { overflow-x: auto; max-width: 100%; }
@@ -946,7 +946,7 @@
<div class="kicker"><span class="accent">05</span><span class="dash">·</span><span>KNOWN ISSUES &amp; WHAT'S NEXT</span></div>
<h2 class="scene">THE RED CELLS<br><span class="accent">ARE THE ROADMAP.</span></h2>
- <div class="row"><div class="k">Mixed read-under-write · 5 rows</div><div class="v">The real, constant weakness. With a concurrent bulk writer running, read p99 degrades under sustained flush/merge pressure. A merged runtime-scheduler fix <strong>roughly halves the tail</strong> — typical p99 now ~45–80&nbsp;ms, down from 62–152 — but it still trails ES's quiesced <strong>2–19&nbsp;ms</strong>. XERJ also ingests ~5× faster than ES, so it hits far more finalize windows; shrinking this tail to ES's steady-state is the top active work item.</div></div>
+ <div class="row"><div class="k">Mixed read-under-write · 5 rows</div><div class="v">The real, constant weakness — now precisely root-caused. With a concurrent bulk writer, read p99 spikes to <strong>~50–120&nbsp;ms</strong> vs ES's <strong>2–19&nbsp;ms</strong>. It is <em>not</em> a per-query cost: <span class="mono">match_all</span> — an O(1) counter that does no work per doc — stalls identically to <span class="mono">range</span>/<span class="mono">terms</span>, which proves the tail is a <strong>global flush stall</strong>, not an algorithmic loop. XERJ ingests ~5× faster than ES, so it flushes ~90× per run; each flush briefly freezes queries. The flush path is already deprioritized, storm-gated and warm-before-publish, and we've bounded the per-match allocation (range 152→~117&nbsp;ms) and made counts incremental — but beating ES's steady-state here needs a rolling/incremental segment build with no discrete finalize stall. That flush-path re-architecture is the top open item; it is the honest price of 5×-faster ingest.</div></div>
<div class="row"><div class="k">Sub-millisecond tie-band</div><div class="v">A handful of aggregation rows where XERJ and ES land within tenths of a millisecond — which side "wins" flips run to run, so the total drifts inside <strong>77–84</strong>. Run 10 caught agg <span class="mono">filter</span>, pipe <span class="mono">sum_bucket</span> / <span class="mono">derivative</span>; other runs catch <span class="mono">percentiles</span> or <span class="mono">composite</span> instead. Per-request overhead, not algorithmic gaps — under profiling, not counted as a real loss.</div></div>
<div class="row"><div class="k">Hardening along the way</div><div class="v">Durable mappings across restarts, a merge-race segment lease (601k queries under merge churn, zero mismatches), WAL/segment garbage collection, a merge data-loss fix, a dedicated deprioritised ingest pool, and cooperative query timeouts.</div></div>
<div class="row"><div class="k">Not measured here</div><div class="v">Skipped, not hidden — each needs a purpose-built index the flat corpus lacks: <span class="mono">geo_*</span>, <span class="mono">ip_range / ip_prefix</span>, <span class="mono">nested / has_child / has_parent</span>, <span class="mono">span_*</span>, <span class="mono">significant_text</span>, <span class="mono">semantic / hybrid</span> retrieval, and <span class="mono">percolate</span>. Purpose-built corpora are planned follow-ups.</div></div>
@@ -992,6 +992,7 @@
<p class="sb-note">Wins out of 91 per published run (0–91 scale). The gold line is the honest trajectory; run 6 (dashed) was retracted — a 77-win high that vanished once a stale result cache was fixed.</p>
</div>
+ <div class="row"><div class="k">2026-07-06 (mixed-p99 root-cause)</div><div class="v"><strong>We finally caught what the 5 mixed losses actually are.</strong> The tempting theory was an O(N) per-query loop that grows with data. We disproved it with one measurement: <span class="mono">match_all</span> — which counts hits with an O(1) counter and does zero per-doc work — stalls to ~55&nbsp;ms <em>identically</em> to <span class="mono">range</span> and <span class="mono">terms</span>. If it were per-query cost, match_all would stay fast. So the tail is a <strong>global flush stall</strong>: XERJ ingests ~5× faster than ES and flushes ~90× per run, and each flush briefly freezes every in-flight query. We shipped two honest wins along the way — bounding a per-match allocation storm (range 152→~117&nbsp;ms) and making memtable counts incremental — and confirmed the flush path is already deprioritized, storm-gated and warm-before-publish. Two things we <em>refused</em>: a thread-pool trick that quietly slowed the sub-millisecond aggregations, and inflating the flush buffer just to dodge flushes inside the 15&nbsp;s test window (that's measuring the gap you left, not closing it). Real fix = a rolling segment build with no discrete finalize stall. Filed under: name the wall before you climb it.</div></div>
<div class="row"><div class="k">2026-07-06 (runtime-worker fix)</div><div class="v"><strong>Post-run-10 engine change.</strong> Shipped a tokio runtime over-provision that un-starves request-accept during flush/merge windows — roughly <strong>halving the mixed read-under-write p99 tail</strong> (match_all 65→~45&nbsp;ms, range 152→~50&nbsp;ms typical). Still short of ES's quiesced 2–19&nbsp;ms, but no longer a cliff. Re-running the fixed binary a handful of times, the total lands <strong>77–84</strong> (median ~80) as the sub-millisecond aggregation tie-band flips run to run; the 5 mixed cells stay the only constant losses. We keep run 10's <strong>81/8/2</strong> as the published board — squarely mid-band — rather than cherry-pick the 84-win draw.</div></div>
<div class="row"><div class="k">2026-07-06 (run 10)</div><div class="v"><strong>81 WIN.</strong> Eight cells flipped LOSE→WIN over run 9 as the engine work landed — ingest 1m×c8 (now <span class="mono">1.02×</span>), aggs <span class="mono">global / composite / terms+avg / range</span>, pipe <span class="mono">serial_diff</span>, <span class="mono">simple_query_string</span>, and deep <span class="mono">from+size</span> — while two former wins regressed to sub-millisecond near-ties (<span class="mono">sum_bucket</span>, <span class="mono">derivative</span>). Net <strong>81 W / 8 L / 2 N/A</strong>: the 8 losses are the 5 mixed read-under-write p99 rows plus 3 sub-millisecond near-ties (agg <span class="mono">filter</span>, pipe <span class="mono">sum_bucket / derivative</span>). Supersedes run 9.</div></div>
<div class="row"><div class="k">2026-07-06 (run 9)</div><div class="v">75 WIN. Corrected the read-latency transport: earlier runs timed reads through Node's global <span class="mono">fetch</span>/undici (~1.5&nbsp;ms of client overhead per request), which swamped both engines' sub-millisecond server times and pushed ~40 rows into a noise band. Re-ran on a lean keep-alive HTTP client, applied identically to both engines and cross-checked with <span class="mono">curl</span> (XERJ ≤ ES on every op). True server round-trips now show full-text / term / range / wildcard reads winning outright, kNN 5.28× at 100% recall parity, index 1.26× smaller, ingest 3 of 4 cells. 14 honest losses remain · 75 W / 14 L / 2 N/A. Supersedes run 8.</div></div>