Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

landing: benchmarks page — run 4 (35W/47L/9NA), honest regression framing

xerj-orgc500ab5a2cf5
diff --git a/landing/benchmarks/index.html b/landing/benchmarks/index.html
index 0600eb5f4..bae5cbe85 100644
--- a/landing/benchmarks/index.html
+++ b/landing/benchmarks/index.html
@@ -6,7 +6,7 @@
<title>XERJ vs Elasticsearch — Reproducible Benchmarks</title>
<meta name="description" content="XERJ vs Elasticsearch 8.13.4: 91 measured dimensions, every WIN and every LOSE published. Reproducible with two scripts, versioned in the repo, re-run on every change.">
<meta property="og:title" content="XERJ vs Elasticsearch — Reproducible Benchmarks">
-<meta property="og:description" content="43 WIN · 33 LOSE · 15 N/A. Measured, reproducible, versioned. The losses are published next to the wins.">
+<meta property="og:description" content="35 WIN · 47 LOSE · 9 N/A. Measured, reproducible, versioned. The losses are published next to the wins.">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Big+Shoulders+Display:wght@400;700;800;900&family=Inter:wght@400;500;600;700&family=IBM+Plex+Sans:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;700&display=swap" rel="stylesheet">
@@ -50,7 +50,7 @@
<div class="kicker">
<span class="accent">BENCHMARKS</span><span class="dash">·</span>
<span>XERJ V1.0.0-RC.1 VS ELASTICSEARCH 8.13.4</span><span class="dash">·</span>
- <span>RUN 2026-07-04</span>
+ <span>RUN 4 · 2026-07-04</span>
</div>
<h1 class="hero">EVERY CELL.<br><span class="accent">BOTH</span> DIRECTIONS.</h1>
@@ -58,29 +58,38 @@
<p class="lead">
One harness, two engines, identical workload: 91 measured dimensions
across ingest, every query / aggregation / pipeline family, mixed
- read-under-write, kNN, and disk. As of
- <span class="mono">2026-07-04</span> the score is
- <strong>43 WIN · 33 LOSE · 15 N/A</strong> for XERJ. The losses are
- published next to the wins — this run was truncated by a
- <span class="mono">search_after</span> defect that the benchmark
- itself uncovered, and every casualty row is shown below, scored
- against us.
+ read-under-write, kNN, and disk. As of the
+ <span class="mono">2026-07-04</span> evening rerun (run 4) the score
+ is <strong>35 WIN · 47 LOSE · 9 N/A</strong> for XERJ — a worse
+ headline than the previous run, because the benchmark got fairer,
+ not because the engine got slower. The round-1 fixes landed and
+ verified: single-client ingest is now a <strong>1.26× WIN at both
+ corpus sizes</strong>, <span class="mono">search_after</span> went
+ from crashing the engine to a <strong>3.40× WIN</strong>
+ (5.95&nbsp;ms vs 20.2&nbsp;ms), sort-heavy reads win 9.63×, and
+ <span class="mono">regexp</span> latency fell from collapsed to
+ 1.9&nbsp;ms against ES's 11.1&nbsp;ms. But this cleaner, warmer-ES
+ run also exposed a persistent cluster of ~27 rows where XERJ
+ requests hang, a <span class="mono">regexp</span> hits-count
+ regression introduced by a merge, and a tail of marginal read
+ losses. Every one is published below, scored against us, with a
+ named fix in flight.
</p>
<div class="metrics">
<div class="metric">
<div class="eyebrow">XERJ WINS</div>
- <div class="value">43</div>
+ <div class="value">35</div>
<div class="hint">Ratio &gt; 1× · lower latency or higher throughput</div>
</div>
<div class="metric">
<div class="eyebrow">XERJ LOSSES</div>
- <div class="value">33</div>
+ <div class="value">47</div>
<div class="hint">Published, tracked, and each one fails CI</div>
</div>
<div class="metric">
<div class="eyebrow">NOT SCORED</div>
- <div class="value">15</div>
+ <div class="value">9</div>
<div class="hint">Unsupported on one side or result mismatch</div>
</div>
<div class="metric">
@@ -155,16 +164,20 @@
<p class="wide">
All 91 rows from <span class="mono">demo/playbooks/SCORECARD.md</span>,
unedited. Latency rows are p50 in milliseconds; ingest rows are docs/s.
- <strong>Read this run's caveat first:</strong> partway through, the
- <span class="mono">search_after</span> family (its 9,924&nbsp;ms row
- below) triggered a defect that drove XERJ to an out-of-memory kill.
- Rows recorded as <span class="mono">collapsed</span> or
- <span class="mono">unsupported</span> on the XERJ side after that point
- are casualties of that crash — the engine was down or dying when those
- families ran, not measurably slower. They are scored LOSE anyway,
- because an engine that dies mid-benchmark loses those rows. A fix is
- in flight (see What's Next); the matrix will be re-run and this page
- updated when it lands.
+ <strong>Read this run's caveat first:</strong> this is the clean rerun
+ after the round-1 fixes — no crash, no OOM, and a warmer, fairer
+ Elasticsearch baseline (ES p50s here are broadly lower than in the
+ previous run). That fairness is exactly why the headline score got
+ worse: with ES no longer handicapped, a persistent cluster stands
+ exposed — <span class="mono">terms</span>,
+ <span class="mono">top_hits</span>, <span class="mono">range</span>,
+ <span class="mono">filter</span>/<span class="mono">filters</span>,
+ <span class="mono">composite</span>, every pipeline aggregation, and
+ every mixed read-under-write row (~27 rows in all) where XERJ accepts
+ the request and hangs past the 15&nbsp;s bound, recorded as
+ <span class="mono">collapsed</span>. Those rows are scored LOSE, as
+ they should be. The engine did not get slower between runs; the
+ benchmark got harder to flatter.
</p>
<div class="bench-scroll">
@@ -175,448 +188,448 @@
<div class="head num">RATIO</div>
<div class="head num">VERDICT</div>
<div class="cell test">ingest 100k × c1 (docs/s)</div>
- <div class="cell num xy">64,451</div>
- <div class="cell num es">48,979</div>
- <div class="cell num ratio win">1.32×</div>
+ <div class="cell num xy">89,638</div>
+ <div class="cell num es">71,316</div>
+ <div class="cell num ratio win">1.26×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">ingest 100k × c8 (docs/s)</div>
- <div class="cell num xy">233,602</div>
- <div class="cell num es">199,589</div>
- <div class="cell num ratio win">1.17×</div>
- <div class="cell num verdict win">WIN</div>
- <div class="cell test">ingest 1m × c1 (docs/s)</div>
- <div class="cell num xy">65,225</div>
- <div class="cell num es">69,328</div>
- <div class="cell num ratio lose">0.94×</div>
+ <div class="cell num xy">245,298</div>
+ <div class="cell num es">265,243</div>
+ <div class="cell num ratio lose">0.92×</div>
<div class="cell num verdict lose">LOSE</div>
+ <div class="cell test">ingest 1m × c1 (docs/s)</div>
+ <div class="cell num xy">88,722</div>
+ <div class="cell num es">70,271</div>
+ <div class="cell num ratio win">1.26×</div>
+ <div class="cell num verdict win">WIN</div>
<div class="cell test">ingest 1m × c8 (docs/s)</div>
- <div class="cell num xy">205,273</div>
- <div class="cell num es">374,490</div>
- <div class="cell num ratio lose">0.55×</div>
+ <div class="cell num xy">266,553</div>
+ <div class="cell num es">375,057</div>
+ <div class="cell num ratio lose">0.71×</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read q: match_all (p50 ms)</div>
- <div class="cell num xy">2.14</div>
- <div class="cell num es">5.07</div>
- <div class="cell num ratio win">2.37×</div>
+ <div class="cell num xy">2.15</div>
+ <div class="cell num es">2.72</div>
+ <div class="cell num ratio win">1.27×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read q: match_none (p50 ms)</div>
- <div class="cell num xy">2.71</div>
- <div class="cell num es">3.10</div>
- <div class="cell num ratio win">1.14×</div>
+ <div class="cell num xy">1.54</div>
+ <div class="cell num es">2.15</div>
+ <div class="cell num ratio win">1.39×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read q: match(model) (p50 ms)</div>
- <div class="cell num xy">1.93</div>
- <div class="cell num es">4.58</div>
- <div class="cell num ratio win">2.37×</div>
+ <div class="cell num xy">2.60</div>
+ <div class="cell num es">3.20</div>
+ <div class="cell num ratio win">1.23×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read q: match_phrase(top_doc) (p50 ms)</div>
- <div class="cell num xy">1.87</div>
- <div class="cell num es">4.39</div>
- <div class="cell num ratio win">2.35×</div>
+ <div class="cell num xy">2.59</div>
+ <div class="cell num es">2.81</div>
+ <div class="cell num ratio win">1.08×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read q: match_phrase_prefix (p50 ms)</div>
- <div class="cell num xy">2.80</div>
+ <div class="cell num xy">2.58</div>
<div class="cell num es">unsupported (400)</div>
<div class="cell num ratio na">—</div>
<div class="cell num verdict na">N/A</div>
<div class="cell test">read q: match_bool_prefix (p50 ms)</div>
- <div class="cell num xy">1.93</div>
- <div class="cell num es">6.34</div>
- <div class="cell num ratio win">3.29×</div>
+ <div class="cell num xy">2.22</div>
+ <div class="cell num es">5.90</div>
+ <div class="cell num ratio win">2.66×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read q: multi_match (p50 ms) <span class="note">result mismatch: hits 0 vs 277449</span></div>
- <div class="cell num xy">2.04</div>
- <div class="cell num es">7.32</div>
+ <div class="cell num xy">2.02</div>
+ <div class="cell num es">6.83</div>
<div class="cell num ratio na">mismatch</div>
<div class="cell num verdict na">N/A</div>
<div class="cell test">read q: combined_fields (p50 ms)</div>
- <div class="cell num xy">2.05</div>
+ <div class="cell num xy">1.79</div>
<div class="cell num es">unsupported (400)</div>
<div class="cell num ratio na">—</div>
<div class="cell num verdict na">N/A</div>
<div class="cell test">read q: query_string (p50 ms) <span class="note">result mismatch: hits 0 vs 273706</span></div>
- <div class="cell num xy">2.30</div>
- <div class="cell num es">11.89</div>
+ <div class="cell num xy">2.03</div>
+ <div class="cell num es">12.76</div>
<div class="cell num ratio na">mismatch</div>
<div class="cell num verdict na">N/A</div>
<div class="cell test">read q: simple_query_string (p50 ms) <span class="note">result mismatch: hits 0 vs 987529</span></div>
- <div class="cell num xy">2.11</div>
- <div class="cell num es">3.15</div>
+ <div class="cell num xy">2.71</div>
+ <div class="cell num es">2.50</div>
<div class="cell num ratio na">mismatch</div>
<div class="cell num verdict na">N/A</div>
<div class="cell test">read q: more_like_this (p50 ms)</div>
- <div class="cell num xy">1.78</div>
- <div class="cell num es">3.17</div>
- <div class="cell num ratio win">1.78×</div>
- <div class="cell num verdict win">WIN</div>
+ <div class="cell num xy">2.50</div>
+ <div class="cell num es">2.27</div>
+ <div class="cell num ratio lose">0.91×</div>
+ <div class="cell num verdict lose">LOSE</div>
<div class="cell test">read q: term(status) (p50 ms)</div>
- <div class="cell num xy">2.51</div>
- <div class="cell num es">2.65</div>
- <div class="cell num ratio win">1.06×</div>
- <div class="cell num verdict win">WIN</div>
+ <div class="cell num xy">2.43</div>
+ <div class="cell num es">1.75</div>
+ <div class="cell num ratio lose">0.72×</div>
+ <div class="cell num verdict lose">LOSE</div>
<div class="cell test">read q: terms(model) (p50 ms)</div>
- <div class="cell num xy">2.23</div>
- <div class="cell num es">6.42</div>
- <div class="cell num ratio win">2.88×</div>
+ <div class="cell num xy">1.99</div>
+ <div class="cell num es">5.41</div>
+ <div class="cell num ratio win">2.71×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read q: range(latency_ms) (p50 ms)</div>
- <div class="cell num xy">1.95</div>
- <div class="cell num es">6.12</div>
- <div class="cell num ratio win">3.14×</div>
+ <div class="cell num xy">2.10</div>
+ <div class="cell num es">6.03</div>
+ <div class="cell num ratio win">2.87×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read q: range(@timestamp) (p50 ms)</div>
- <div class="cell num xy">2.24</div>
- <div class="cell num es">2.63</div>
- <div class="cell num ratio win">1.17×</div>
+ <div class="cell num xy">1.78</div>
+ <div class="cell num es">1.81</div>
+ <div class="cell num ratio win">1.02×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read q: range(cost_usd) (p50 ms)</div>
- <div class="cell num xy">1.14</div>
- <div class="cell num es">6.54</div>
- <div class="cell num ratio win">5.75×</div>
+ <div class="cell num xy">2.37</div>
+ <div class="cell num es">7.55</div>
+ <div class="cell num ratio win">3.18×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read q: prefix(model) (p50 ms)</div>
- <div class="cell num xy">1.71</div>
- <div class="cell num es">11.00</div>
- <div class="cell num ratio win">6.43×</div>
+ <div class="cell num xy">2.12</div>
+ <div class="cell num es">10.27</div>
+ <div class="cell num ratio win">4.85×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read q: wildcard(model) (p50 ms)</div>
- <div class="cell num xy">1.40</div>
- <div class="cell num es">10.42</div>
- <div class="cell num ratio win">7.43×</div>
+ <div class="cell num xy">2.05</div>
+ <div class="cell num es">10.45</div>
+ <div class="cell num ratio win">5.11×</div>
<div class="cell num verdict win">WIN</div>
- <div class="cell test">read q: regexp(model) (p50 ms)</div>
- <div class="cell num xy">collapsed</div>
- <div class="cell num es">11.38</div>
- <div class="cell num ratio lose">—</div>
- <div class="cell num verdict lose">LOSE</div>
- <div class="cell test">read q: fuzzy(model) (p50 ms) <span class="note">result mismatch: hits 0 vs 277449</span></div>
- <div class="cell num xy">1.62</div>
- <div class="cell num es">3.11</div>
+ <div class="cell test">read q: regexp(model) (p50 ms) <span class="note">result mismatch: hits 512 vs 847306</span></div>
+ <div class="cell num xy">1.89</div>
+ <div class="cell num es">11.12</div>
<div class="cell num ratio na">mismatch</div>
<div class="cell num verdict na">N/A</div>
+ <div class="cell test">read q: fuzzy(model) (p50 ms)</div>
+ <div class="cell num xy">2.50</div>
+ <div class="cell num es">1.94</div>
+ <div class="cell num ratio lose">0.78×</div>
+ <div class="cell num verdict lose">LOSE</div>
<div class="cell test">read q: exists(cost_usd) (p50 ms)</div>
- <div class="cell num xy">1.41</div>
- <div class="cell num es">2.50</div>
- <div class="cell num ratio win">1.77×</div>
- <div class="cell num verdict win">WIN</div>
+ <div class="cell num xy">2.52</div>
+ <div class="cell num es">1.66</div>
+ <div class="cell num ratio lose">0.66×</div>
+ <div class="cell num verdict lose">LOSE</div>
<div class="cell test">read q: ids (p50 ms)</div>
- <div class="cell num xy">1.10</div>
- <div class="cell num es">2.24</div>
- <div class="cell num ratio win">2.03×</div>
- <div class="cell num verdict win">WIN</div>
+ <div class="cell num xy">2.24</div>
+ <div class="cell num es">1.56</div>
+ <div class="cell num ratio lose">0.70×</div>
+ <div class="cell num verdict lose">LOSE</div>
<div class="cell test">read q: term(cache_hit) (p50 ms)</div>
- <div class="cell num xy">1.25</div>
- <div class="cell num es">2.37</div>
- <div class="cell num ratio win">1.90×</div>
- <div class="cell num verdict win">WIN</div>
+ <div class="cell num xy">1.98</div>
+ <div class="cell num es">1.95</div>
+ <div class="cell num ratio lose">0.98×</div>
+ <div class="cell num verdict lose">LOSE</div>
<div class="cell test">read q: bool must+filter+should+must_not (p50 ms)</div>
- <div class="cell num xy">0.82</div>
- <div class="cell num es">20.54</div>
- <div class="cell num ratio win">25.10×</div>
+ <div class="cell num xy">2.14</div>
+ <div class="cell num es">18.50</div>
+ <div class="cell num ratio win">8.64×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read q: constant_score (p50 ms)</div>
- <div class="cell num xy">1.36</div>
- <div class="cell num es">2.27</div>
- <div class="cell num ratio win">1.67×</div>
- <div class="cell num verdict win">WIN</div>
+ <div class="cell num xy">2.62</div>
+ <div class="cell num es">1.59</div>
+ <div class="cell num ratio lose">0.61×</div>
+ <div class="cell num verdict lose">LOSE</div>
<div class="cell test">read q: boosting (p50 ms)</div>
- <div class="cell num xy">0.41</div>
- <div class="cell num es">35.77</div>
- <div class="cell num ratio win">87.78×</div>
+ <div class="cell num xy">1.63</div>
+ <div class="cell num es">33.03</div>
+ <div class="cell num ratio win">20.20×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read q: dis_max (p50 ms)</div>
- <div class="cell num xy">1.49</div>
- <div class="cell num es">6.27</div>
- <div class="cell num ratio win">4.20×</div>
+ <div class="cell num xy">2.23</div>
+ <div class="cell num es">6.26</div>
+ <div class="cell num ratio win">2.81×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read q: function_score (p50 ms)</div>
- <div class="cell num xy">1.15</div>
- <div class="cell num es">35.40</div>
- <div class="cell num ratio win">30.84×</div>
+ <div class="cell num xy">2.03</div>
+ <div class="cell num es">40.23</div>
+ <div class="cell num ratio win">19.80×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read q: pinned (p50 ms)</div>
- <div class="cell num xy">0.98</div>
- <div class="cell num es">15.67</div>
- <div class="cell num ratio win">15.99×</div>
+ <div class="cell num xy">2.00</div>
+ <div class="cell num es">16.03</div>
+ <div class="cell num ratio win">8.03×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read agg: avg (p50 ms)</div>
- <div class="cell num xy">0.89</div>
- <div class="cell num es">2.66</div>
- <div class="cell num ratio win">2.97×</div>
- <div class="cell num verdict win">WIN</div>
+ <div class="cell num xy">2.60</div>
+ <div class="cell num es">2.39</div>
+ <div class="cell num ratio lose">0.92×</div>
+ <div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: sum (p50 ms)</div>
- <div class="cell num xy">1.21</div>
- <div class="cell num es">2.57</div>
- <div class="cell num ratio win">2.12×</div>
+ <div class="cell num xy">1.82</div>
+ <div class="cell num es">2.49</div>
+ <div class="cell num ratio win">1.37×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read agg: min (p50 ms)</div>
- <div class="cell num xy">1.11</div>
- <div class="cell num es">2.24</div>
- <div class="cell num ratio win">2.01×</div>
- <div class="cell num verdict win">WIN</div>
+ <div class="cell num xy">1.64</div>
+ <div class="cell num es">1.52</div>
+ <div class="cell num ratio lose">0.92×</div>
+ <div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: max (p50 ms)</div>
- <div class="cell num xy">0.62</div>
- <div class="cell num es">2.75</div>
- <div class="cell num ratio win">4.47×</div>
- <div class="cell num verdict win">WIN</div>
+ <div class="cell num xy">2.46</div>
+ <div class="cell num es">2.42</div>
+ <div class="cell num ratio lose">0.98×</div>
+ <div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: stats (p50 ms)</div>
- <div class="cell num xy">1.01</div>
- <div class="cell num es">2.53</div>
- <div class="cell num ratio win">2.51×</div>
- <div class="cell num verdict win">WIN</div>
+ <div class="cell num xy">2.35</div>
+ <div class="cell num es">2.06</div>
+ <div class="cell num ratio lose">0.88×</div>
+ <div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: extended_stats (p50 ms)</div>
- <div class="cell num xy">0.75</div>
- <div class="cell num es">2.49</div>
- <div class="cell num ratio win">3.33×</div>
+ <div class="cell num xy">1.66</div>
+ <div class="cell num es">1.80</div>
+ <div class="cell num ratio win">1.08×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read agg: value_count (p50 ms)</div>
- <div class="cell num xy">1.21</div>
- <div class="cell num es">2.18</div>
- <div class="cell num ratio win">1.81×</div>
+ <div class="cell num xy">2.06</div>
+ <div class="cell num es">2.15</div>
+ <div class="cell num ratio win">1.04×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read agg: cardinality (p50 ms)</div>
- <div class="cell num xy">0.98</div>
- <div class="cell num es">2.52</div>
- <div class="cell num ratio win">2.58×</div>
+ <div class="cell num xy">1.24</div>
+ <div class="cell num es">1.79</div>
+ <div class="cell num ratio win">1.44×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read agg: percentiles (p50 ms)</div>
- <div class="cell num xy">1.16</div>
- <div class="cell num es">2.05</div>
- <div class="cell num ratio win">1.77×</div>
- <div class="cell num verdict win">WIN</div>
+ <div class="cell num xy">2.17</div>
+ <div class="cell num es">1.50</div>
+ <div class="cell num ratio lose">0.69×</div>
+ <div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: percentile_ranks (p50 ms)</div>
- <div class="cell num xy">1.19</div>
- <div class="cell num es">2.33</div>
- <div class="cell num ratio win">1.97×</div>
- <div class="cell num verdict win">WIN</div>
+ <div class="cell num xy">2.47</div>
+ <div class="cell num es">1.45</div>
+ <div class="cell num ratio lose">0.59×</div>
+ <div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: median_absolute_deviation (p50 ms)</div>
- <div class="cell num xy">2.46</div>
- <div class="cell num es">2.79</div>
- <div class="cell num ratio win">1.13×</div>
- <div class="cell num verdict win">WIN</div>
+ <div class="cell num xy">2.34</div>
+ <div class="cell num es">1.85</div>
+ <div class="cell num ratio lose">0.79×</div>
+ <div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: matrix_stats (p50 ms)</div>
- <div class="cell num xy">2.09</div>
- <div class="cell num es">2.61</div>
- <div class="cell num ratio win">1.25×</div>
- <div class="cell num verdict win">WIN</div>
+ <div class="cell num xy">1.75</div>
+ <div class="cell num es">1.72</div>
+ <div class="cell num ratio lose">0.98×</div>
+ <div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: scripted_metric (p50 ms)</div>
- <div class="cell num xy">2.05</div>
- <div class="cell num es">2.13</div>
- <div class="cell num ratio win">1.04×</div>
- <div class="cell num verdict win">WIN</div>
+ <div class="cell num xy">2.22</div>
+ <div class="cell num es">1.79</div>
+ <div class="cell num ratio lose">0.80×</div>
+ <div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: top_hits (sub) (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">2.96</div>
+ <div class="cell num es">1.81</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: terms (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">2.12</div>
+ <div class="cell num es">1.42</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: rare_terms (p50 ms)</div>
- <div class="cell num xy">2.01</div>
- <div class="cell num es">1.96</div>
- <div class="cell num ratio lose">0.97×</div>
- <div class="cell num verdict lose">LOSE</div>
+ <div class="cell num xy">1.96</div>
+ <div class="cell num es">2.22</div>
+ <div class="cell num ratio win">1.13×</div>
+ <div class="cell num verdict win">WIN</div>
<div class="cell test">read agg: significant_terms (p50 ms)</div>
- <div class="cell num xy">1.55</div>
- <div class="cell num es">2.03</div>
- <div class="cell num ratio win">1.31×</div>
+ <div class="cell num xy">0.92</div>
+ <div class="cell num es">1.29</div>
+ <div class="cell num ratio win">1.41×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read agg: histogram (p50 ms)</div>
- <div class="cell num xy">2.66</div>
- <div class="cell num es">2.71</div>
- <div class="cell num ratio win">1.02×</div>
+ <div class="cell num xy">0.94</div>
+ <div class="cell num es">1.31</div>
+ <div class="cell num ratio win">1.39×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read agg: date_histogram (p50 ms)</div>
- <div class="cell num xy">2.31</div>
- <div class="cell num es">2.53</div>
- <div class="cell num ratio win">1.10×</div>
- <div class="cell num verdict win">WIN</div>
+ <div class="cell num xy">2.63</div>
+ <div class="cell num es">1.91</div>
+ <div class="cell num ratio lose">0.72×</div>
+ <div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: auto_date_histogram (p50 ms)</div>
- <div class="cell num xy">2.16</div>
+ <div class="cell num xy">1.89</div>
<div class="cell num es">2.10</div>
- <div class="cell num ratio lose">0.97×</div>
- <div class="cell num verdict lose">LOSE</div>
+ <div class="cell num ratio win">1.11×</div>
+ <div class="cell num verdict win">WIN</div>
<div class="cell test">read agg: variable_width_histogram (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">3.56</div>
+ <div class="cell num es">3.25</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: range (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">2.30</div>
+ <div class="cell num es">2.65</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: date_range (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">3.18</div>
+ <div class="cell num es">2.19</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: filter (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">2.59</div>
+ <div class="cell num es">2.22</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: filters (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">3.25</div>
+ <div class="cell num es">2.69</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: missing (p50 ms)</div>
- <div class="cell num xy">1.21</div>
- <div class="cell num es">2.89</div>
- <div class="cell num ratio win">2.39×</div>
+ <div class="cell num xy">1.24</div>
+ <div class="cell num es">2.29</div>
+ <div class="cell num ratio win">1.85×</div>
+ <div class="cell num verdict win">WIN</div>
+ <div class="cell test">read agg: global (p50 ms)</div>
+ <div class="cell num xy">1.61</div>
+ <div class="cell num es">2.30</div>
+ <div class="cell num ratio win">1.42×</div>
<div class="cell num verdict win">WIN</div>
- <div class="cell test">read agg: global (p50 ms) <span class="note">result mismatch: hits 257 vs 12471</span></div>
- <div class="cell num xy">1.20</div>
- <div class="cell num es">2.77</div>
- <div class="cell num ratio na">mismatch</div>
- <div class="cell num verdict na">N/A</div>
<div class="cell test">read agg: adjacency_matrix (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">3.10</div>
+ <div class="cell num es">2.17</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: composite (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">2.94</div>
+ <div class="cell num es">2.71</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: random_sampler (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">6.48</div>
+ <div class="cell num es">6.85</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read agg: terms+avg(cost) (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">2.62</div>
+ <div class="cell num es">2.72</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read pipe: sum_bucket (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">3.08</div>
+ <div class="cell num es">3.50</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read pipe: avg_bucket (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">3.00</div>
+ <div class="cell num es">2.53</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read pipe: max_bucket (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">3.34</div>
+ <div class="cell num es">3.02</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read pipe: stats_bucket (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">2.68</div>
+ <div class="cell num es">2.48</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read pipe: percentiles_bucket (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">2.87</div>
+ <div class="cell num es">2.98</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read pipe: derivative (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">3.56</div>
+ <div class="cell num es">3.24</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read pipe: cumulative_sum (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">2.53</div>
+ <div class="cell num es">2.37</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read pipe: moving_fn (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">3.84</div>
+ <div class="cell num es">2.51</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read pipe: serial_diff (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">3.25</div>
+ <div class="cell num es">2.80</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read pipe: bucket_script (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">3.26</div>
+ <div class="cell num es">2.94</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read pipe: bucket_selector (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">3.35</div>
+ <div class="cell num es">2.91</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read pipe: bucket_sort (p50 ms)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">3.59</div>
+ <div class="cell num es">2.73</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
<div class="cell test">read feat: sort-heavy (p50 ms)</div>
- <div class="cell num xy">2.19</div>
- <div class="cell num es">20.04</div>
- <div class="cell num ratio win">9.13×</div>
+ <div class="cell num xy">2.02</div>
+ <div class="cell num es">19.46</div>
+ <div class="cell num ratio win">9.63×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read feat: deep from+size (from 500) (p50 ms)</div>
- <div class="cell num xy">2.36</div>
- <div class="cell num es">3.43</div>
- <div class="cell num ratio win">1.45×</div>
+ <div class="cell num xy">1.84</div>
+ <div class="cell num es">3.80</div>
+ <div class="cell num ratio win">2.06×</div>
<div class="cell num verdict win">WIN</div>
<div class="cell test">read feat: search_after (p50 ms)</div>
- <div class="cell num xy">9924.66</div>
- <div class="cell num es">18.76</div>
+ <div class="cell num xy">5.95</div>
+ <div class="cell num es">20.20</div>
+ <div class="cell num ratio win">3.40×</div>
+ <div class="cell num verdict win">WIN</div>
+ <div class="cell test">read feat: highlight (p50 ms)</div>
+ <div class="cell num xy">1.95</div>
+ <div class="cell num es">3.29</div>
+ <div class="cell num ratio win">1.69×</div>
+ <div class="cell num verdict win">WIN</div>
+ <div class="cell test">read feat: _count (p50 ms)</div>
+ <div class="cell num xy">1.37</div>
+ <div class="cell num es">2.90</div>
+ <div class="cell num ratio win">2.11×</div>
+ <div class="cell num verdict win">WIN</div>
+ <div class="cell test">read feat: _msearch (p50 ms)</div>
+ <div class="cell num xy">1.36</div>
+ <div class="cell num es">3.39</div>
+ <div class="cell num ratio win">2.50×</div>
+ <div class="cell num verdict win">WIN</div>
+ <div class="cell test">read feat: _mget (p50 ms)</div>
+ <div class="cell num xy">0.85</div>
+ <div class="cell num es">2.77</div>
+ <div class="cell num ratio win">3.26×</div>
+ <div class="cell num verdict win">WIN</div>
+ <div class="cell test">mixed match_all (p99 ms, under write)</div>
+ <div class="cell num xy">15086.42</div>
+ <div class="cell num es">5.35</div>
<div class="cell num ratio lose">0.00×</div>
<div class="cell num verdict lose">LOSE</div>
- <div class="cell test">read feat: highlight (p50 ms)</div>
+ <div class="cell test">mixed bool (p99 ms, under write)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">2.94</div>
+ <div class="cell num es">195.61</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
- <div class="cell test">read feat: _count (p50 ms)</div>
+ <div class="cell test">mixed range (p99 ms, under write)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">1.74</div>
+ <div class="cell num es">8.95</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
- <div class="cell test">read feat: _msearch (p50 ms)</div>
+ <div class="cell test">mixed terms (p99 ms, under write)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">2.34</div>
+ <div class="cell num es">4.52</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
- <div class="cell test">read feat: _mget (p50 ms)</div>
+ <div class="cell test">mixed cardinality (p99 ms, under write)</div>
<div class="cell num xy">collapsed</div>
- <div class="cell num es">2.27</div>
+ <div class="cell num es">12.32</div>
<div class="cell num ratio lose">—</div>
<div class="cell num verdict lose">LOSE</div>
- <div class="cell test">mixed match_all (p99 ms, under write)</div>
- <div class="cell num xy">unsupported</div>
- <div class="cell num es">20.22</div>
- <div class="cell num ratio na">—</div>
- <div class="cell num verdict na">N/A</div>
- <div class="cell test">mixed bool (p99 ms, under write)</div>
- <div class="cell num xy">unsupported</div>
- <div class="cell num es">17.36</div>
- <div class="cell num ratio na">—</div>
- <div class="cell num verdict na">N/A</div>
- <div class="cell test">mixed range (p99 ms, under write)</div>
- <div class="cell num xy">unsupported</div>
- <div class="cell num es">37.85</div>
- <div class="cell num ratio na">—</div>
- <div class="cell num verdict na">N/A</div>
- <div class="cell test">mixed terms (p99 ms, under write)</div>
- <div class="cell num xy">unsupported</div>
- <div class="cell num es">6.15</div>
- <div class="cell num ratio na">—</div>
- <div class="cell num verdict na">N/A</div>
- <div class="cell test">mixed cardinality (p99 ms, under write)</div>
- <div class="cell num xy">unsupported</div>
- <div class="cell num es">21.44</div>
- <div class="cell num ratio na">—</div>
- <div class="cell num verdict na">N/A</div>
<div class="cell test">kNN k=10 (p50 ms)</div>
<div class="cell num xy">error</div>
- <div class="cell num es">4.84</div>
+ <div class="cell num es">4.64</div>
<div class="cell num ratio na">—</div>
<div class="cell num verdict na">N/A</div>
<div class="cell test">kNN recall@10</div>
@@ -626,21 +639,22 @@
<div class="cell num verdict na">N/A</div>
<div class="cell test">index on-disk size</div>
<div class="cell num xy">unsupported</div>
- <div class="cell num es">819.4 MB</div>
+ <div class="cell num es">835.6 MB</div>
<div class="cell num ratio na">—</div>
<div class="cell num verdict na">N/A</div>
</div>
</div>
<p class="wide" style="margin-top: var(--sp-6);">
- Beyond the crash casualties, the honest latency losses in this run:
- ingest 1M×1&nbsp;client (<span class="mono">0.94×</span>), ingest
- 1M×8&nbsp;clients (<span class="mono">0.55×</span>),
- <span class="mono">rare_terms</span> (<span class="mono">0.97×</span>),
- <span class="mono">auto_date_histogram</span>
- (<span class="mono">0.97×</span>), and
- <span class="mono">search_after</span> itself. All five are tracked
- work items.
+ Beyond the hang cluster, the honest losses in this run: 8-client
+ ingest at both sizes (<span class="mono">0.92×</span> at 100k,
+ <span class="mono">0.71×</span> at 1M — up from 0.55×) and seventeen
+ marginal read losses between <span class="mono">0.59×</span> and
+ <span class="mono">0.98×</span> (<span class="mono">term</span>,
+ <span class="mono">exists</span>, <span class="mono">ids</span>,
+ <span class="mono">fuzzy</span>,
+ <span class="mono">constant_score</span>, and most single-metric
+ aggregations). All are tracked work items.
</p>
</section>
@@ -676,22 +690,49 @@
<div class="kicker"><span class="accent">05</span><span class="dash">·</span><span>KNOWN ISSUES &amp; WHAT'S NEXT</span></div>
<h2 class="scene">THE RED CELLS<br><span class="accent">ARE THE ROADMAP.</span></h2>
- <h3 class="section">search_after OOM — found by this benchmark</h3>
+ <h3 class="section">The agg-hang cluster — the biggest lever</h3>
<p class="wide">
- The deep-pagination <span class="mono">search_after</span> family
- exposed a defect that ballooned memory until the kernel killed XERJ
- mid-run — the single largest distortion in this scorecard (28 of the
- 33 LOSE rows are its casualties). The fix is in flight on a dedicated
- branch; the matrix re-runs, and this page is republished, when it
- lands. Finding this class of bug is what the benchmark exists to do.
+ The single largest block of red: ~27 rows where XERJ accepts the
+ request and never returns within the 15&nbsp;s bound —
+ <span class="mono">terms</span>, <span class="mono">top_hits</span>,
+ <span class="mono">range</span>, <span class="mono">filter</span> /
+ <span class="mono">filters</span>, <span class="mono">composite</span>
+ and friends, every pipeline aggregation (they all sit on a bucketing
+ parent), and every mixed read-under-write row. One hang in the shared
+ bucketing path explains all of them, which also makes it the single
+ biggest scorecard lever: fixing it flips more rows than everything
+ else on this page combined. The fix is in flight; the matrix re-runs,
+ and this page is republished, when it lands.
</p>
- <h3 class="section">Ingest at 1M docs — the real performance gap</h3>
+ <h3 class="section">regexp hits-count regression</h3>
<p class="wide">
- XERJ wins ingest at 100k docs (1.32× single-client, 1.17× at 8
- clients) but loses at 1M: <span class="mono">0.94×</span>
- single-client and <span class="mono">0.55×</span> at 8 clients. The
- phased plan (<span class="mono">demo/playbooks/BEAT_ES_MASTER_PLAN.md</span>):
+ <span class="mono">regexp(model)</span> latency is now excellent —
+ 1.9&nbsp;ms against ES's 11.1&nbsp;ms, where the previous run
+ collapsed outright — but a merge introduced a correctness regression:
+ XERJ returns 512 hits where ES returns 847,306. The mismatch rule
+ does its job and voids the row to N/A; XERJ does not get to win on
+ latency with the wrong answer. Bisection is in progress.
+ </p>
+
+ <h3 class="section">Keyword-field full-text mismatches</h3>
+ <p class="wide">
+ <span class="mono">multi_match</span>,
+ <span class="mono">query_string</span>, and
+ <span class="mono">simple_query_string</span> return 0 hits on this
+ corpus's keyword fields where ES returns hundreds of thousands. All
+ three would be latency wins for XERJ; none is allowed to score until
+ the results match. Under investigation.
+ </p>
+
+ <h3 class="section">Ingest at 8 clients — the remaining throughput gap</h3>
+ <p class="wide">
+ Single-client ingest is now a <strong>1.26× WIN at both corpus
+ sizes</strong>. The remaining gap is concurrency: at 8 clients XERJ
+ does 266k&nbsp;docs/s to ES's 375k at 1M
+ (<span class="mono">0.71×</span>, improved from 205k and 0.55× last
+ run) and <span class="mono">0.92×</span> at 100k. The phased plan
+ (<span class="mono">demo/playbooks/BEAT_ES_MASTER_PLAN.md</span>):
a <strong>no-reparse flush</strong> that threads already-parsed
documents into segment building instead of re-parsing them twice;
<strong>search-pool isolation</strong> so background flush and merge
@@ -701,6 +742,15 @@
scorecard green, enforced by CI — any new LOSE fails the build.
</p>
+ <h3 class="section">Unmeasured this run: kNN and disk</h3>
+ <p class="wide">
+ Both kNN rows recorded an error in this run and stay N/A — they will
+ be re-verified on the next pass — and the on-disk-size row is
+ unmeasured because XERJ does not yet expose the stats endpoint the
+ harness reads (ES weighed in at 835.6&nbsp;MB). Neither row is scored
+ in XERJ's favor.
+ </p>
+
<h3 class="section">Families this corpus can't measure</h3>
<p class="wide">
Skipped, not hidden — each needs a purpose-built index the flat
@@ -729,7 +779,8 @@
<!-- 06 — CHANGELOG -------------------------------------------->
<section class="scene-section">
<div class="kicker"><span class="accent">06</span><span class="dash">·</span><span>CHANGELOG</span></div>
- <div class="row"><div class="k">2026-07-04</div><div class="v">First public full-matrix publication. XERJ v1.0.0-rc.1 vs Elasticsearch 8.13.4 · 91 dimensions · 43 WIN / 33 LOSE / 15 N/A · run truncated by the search_after defect documented above.</div></div>
+ <div class="row"><div class="k">2026-07-04 (late)</div><div class="v">Run 4: clean rerun after the round-1 fixes. c1 ingest and search_after flip to WIN; agg-hang cluster isolated (~27 rows, fix in flight) · 35 WIN / 47 LOSE / 9 N/A.</div></div>
+ <div class="row"><div class="k">2026-07-04</div><div class="v">First public full-matrix publication. XERJ v1.0.0-rc.1 vs Elasticsearch 8.13.4 · 91 dimensions · 43 WIN / 33 LOSE / 15 N/A · run truncated by a search_after OOM defect, since fixed and verified in run 4 above.</div></div>
</section>
<footer class="footer">