Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

bench: measure true server latency via lean keep-alive HTTP client (fair to both engines)

xerj-org539ab60aecfb
diff --git a/demo/playbooks/BENCHMARK_VS_ES.md b/demo/playbooks/BENCHMARK_VS_ES.md
index b51772059..840974b00 100644
--- a/demo/playbooks/BENCHMARK_VS_ES.md
+++ b/demo/playbooks/BENCHMARK_VS_ES.md
@@ -4,99 +4,101 @@ Generated by `demo/playbooks/bench-matrix.mjs`. Identical workload, same machine
Verdict is from XERJ's POV: WIN = XERJ better (lower latency / higher docs·s / smaller disk / higher recall). Ratio is normalized so **>1× means XERJ is better**. Any LOSE fails CI.
+> **Methodology — read latency transport.** Read/mixed/kNN latencies are measured with a lean **keep-alive HTTP client** (Node core `http` + a shared per-host `http.Agent`, `maxSockets: 256`), applied **identically to both engines** for every operation and keyed only by `host:port` (both localhost). We do **not** use Node's global `fetch`/undici: it adds ~1.5ms of pure client overhead per request, which swamps the sub-millisecond server times of *both* engines and compresses every read into a noise band. Measured with the identical client hitting both engines (`size:0` avg over 300k): undici fetch → XERJ 1.61ms, ES 2.77ms; http keep-alive → XERJ 0.126ms, ES 0.283ms. The keep-alive client reveals each engine's **true server round-trip** (the ~2.2× gap the client overhead had hidden), not the Node client. Ingest/mixed/kNN bulk load uses `curl` identically for both engines.
+
| dimension | XERJ | ES | ratio | verdict |
|---|--:|--:|--:|:--:|
-| ingest 100k × c1 (docs/s) | 106,968 | 71,024 | 1.51× | WIN |
-| ingest 100k × c8 (docs/s) | 287,606 | 283,000 | 1.02× | WIN |
-| ingest 1m × c1 (docs/s) | 108,619 | 70,055 | 1.55× | WIN |
-| ingest 1m × c8 (docs/s) | 333,148 | 378,298 | 0.88× | LOSE |
-| read q: match_all (p50 ms) | 1.88 | 2.86 | 1.52× | WIN |
-| read q: match_none (p50 ms) | 0.84 | 2.16 | 2.58× | WIN |
-| read q: match(model) (p50 ms) | 1.29 | 2.63 | 2.03× | WIN |
-| read q: match_phrase(top_doc) (p50 ms) | 1.49 | 2.27 | 1.53× | WIN |
-| read q: match_phrase_prefix (p50 ms) | 1.75 | unsupported (400) | — | N/A |
-| read q: match_bool_prefix (p50 ms) | 1.49 | 6.16 | 4.13× | WIN |
-| read q: multi_match (p50 ms) | 2.40 | 6.28 | 2.62× | WIN |
-| read q: combined_fields (p50 ms) | 2.04 | unsupported (400) | — | N/A |
-| read q: query_string (p50 ms) | 0.86 | 10.06 | 11.65× | WIN |
-| read q: simple_query_string (p50 ms) | 2.14 | 2.35 | 1.10× | WIN |
-| read q: more_like_this (p50 ms) | 1.79 | 2.09 | 1.17× | WIN |
-| read q: term(status) (p50 ms) | 1.78 | 2.14 | 1.20× | WIN |
-| read q: terms(model) (p50 ms) | 2.16 | 4.87 | 2.26× | WIN |
-| read q: range(latency_ms) (p50 ms) | 2.05 | 5.96 | 2.91× | WIN |
-| read q: range(@timestamp) (p50 ms) | 2.36 | 1.64 | 0.69× | LOSE |
-| read q: range(cost_usd) (p50 ms) | 2.77 | 7.24 | 2.61× | WIN |
-| read q: prefix(model) (p50 ms) | 1.85 | 9.97 | 5.38× | WIN |
-| read q: wildcard(model) (p50 ms) | 1.86 | 9.86 | 5.31× | WIN |
-| read q: regexp(model) (p50 ms) | 2.46 | 9.49 | 3.86× | WIN |
-| read q: fuzzy(model) (p50 ms) | 2.48 | 2.24 | 0.91× | LOSE |
-| read q: exists(cost_usd) (p50 ms) | 2.21 | 1.23 | 0.56× | LOSE |
-| read q: ids (p50 ms) | 1.75 | 1.43 | 0.82× | LOSE |
-| read q: term(cache_hit) (p50 ms) | 1.96 | 1.62 | 0.83× | LOSE |
-| read q: bool must+filter+should+must_not (p50 ms) | 2.19 | 16.07 | 7.34× | WIN |
-| read q: constant_score (p50 ms) | 1.78 | 1.75 | 0.98× | LOSE |
-| read q: boosting (p50 ms) | 2.23 | 34.08 | 15.26× | WIN |
-| read q: dis_max (p50 ms) | 2.27 | 6.23 | 2.75× | WIN |
-| read q: function_score (p50 ms) | 1.74 | 39.85 | 22.85× | WIN |
-| read q: pinned (p50 ms) | 1.79 | 15.38 | 8.61× | WIN |
-| read agg: avg (p50 ms) | 2.57 | 1.88 | 0.73× | LOSE |
-| read agg: sum (p50 ms) | 2.42 | 2.20 | 0.91× | LOSE |
-| read agg: min (p50 ms) | 2.13 | 2.11 | 0.99× | LOSE |
-| read agg: max (p50 ms) | 1.96 | 2.13 | 1.09× | WIN |
-| read agg: stats (p50 ms) | 2.50 | 1.50 | 0.60× | LOSE |
-| read agg: extended_stats (p50 ms) | 1.90 | 1.86 | 0.98× | LOSE |
-| read agg: value_count (p50 ms) | 1.80 | 1.87 | 1.04× | WIN |
-| read agg: cardinality (p50 ms) | 2.52 | 1.90 | 0.75× | LOSE |
-| read agg: percentiles (p50 ms) | 1.66 | 1.94 | 1.16× | WIN |
-| read agg: percentile_ranks (p50 ms) | 2.60 | 2.36 | 0.91× | LOSE |
-| read agg: median_absolute_deviation (p50 ms) | 1.98 | 2.05 | 1.04× | WIN |
-| read agg: matrix_stats (p50 ms) | 2.08 | 1.38 | 0.66× | LOSE |
-| read agg: scripted_metric (p50 ms) | 1.24 | 1.65 | 1.33× | WIN |
-| read agg: top_hits (sub) (p50 ms) | 1.68 | 2.20 | 1.31× | WIN |
-| read agg: terms (p50 ms) | 1.38 | 1.72 | 1.25× | WIN |
-| read agg: rare_terms (p50 ms) | 2.38 | 2.01 | 0.85× | LOSE |
-| read agg: significant_terms (p50 ms) | 1.57 | 1.32 | 0.84× | LOSE |
-| read agg: histogram (p50 ms) | 2.61 | 2.04 | 0.78× | LOSE |
-| read agg: date_histogram (p50 ms) | 2.21 | 1.95 | 0.88× | LOSE |
-| read agg: auto_date_histogram (p50 ms) | 1.98 | 1.66 | 0.84× | LOSE |
-| read agg: variable_width_histogram (p50 ms) | 1.87 | 2.48 | 1.32× | WIN |
-| read agg: range (p50 ms) | 2.27 | 1.52 | 0.67× | LOSE |
-| read agg: date_range (p50 ms) | 2.57 | 1.68 | 0.65× | LOSE |
-| read agg: filter (p50 ms) | 2.33 | 1.92 | 0.82× | LOSE |
-| read agg: filters (p50 ms) | 2.17 | 1.37 | 0.63× | LOSE |
-| read agg: missing (p50 ms) | 2.25 | 1.18 | 0.53× | LOSE |
-| read agg: global (p50 ms) | 2.30 | 1.56 | 0.68× | LOSE |
-| read agg: adjacency_matrix (p50 ms) | 2.17 | 1.77 | 0.82× | LOSE |
-| read agg: composite (p50 ms) | 1.78 | 1.95 | 1.09× | WIN |
-| read agg: random_sampler (p50 ms) | 2.63 | 5.36 | 2.04× | WIN |
-| read agg: terms+avg(cost) (p50 ms) | 1.67 | 1.72 | 1.03× | WIN |
-| read pipe: sum_bucket (p50 ms) | 2.55 | 2.23 | 0.87× | LOSE |
-| read pipe: avg_bucket (p50 ms) | 1.89 | 1.49 | 0.79× | LOSE |
-| read pipe: max_bucket (p50 ms) | 2.42 | 2.20 | 0.91× | LOSE |
-| read pipe: stats_bucket (p50 ms) | 2.13 | 2.11 | 0.99× | LOSE |
-| read pipe: percentiles_bucket (p50 ms) | 2.57 | 2.45 | 0.95× | LOSE |
-| read pipe: derivative (p50 ms) | 2.46 | 2.01 | 0.82× | LOSE |
-| read pipe: cumulative_sum (p50 ms) | 1.86 | 1.65 | 0.89× | LOSE |
-| read pipe: moving_fn (p50 ms) | 2.13 | 1.45 | 0.68× | LOSE |
-| read pipe: serial_diff (p50 ms) | 2.46 | 1.48 | 0.60× | LOSE |
-| read pipe: bucket_script (p50 ms) | 2.20 | 1.78 | 0.81× | LOSE |
-| read pipe: bucket_selector (p50 ms) | 2.42 | 2.65 | 1.10× | WIN |
-| read pipe: bucket_sort (p50 ms) | 2.68 | 2.33 | 0.87× | LOSE |
-| read feat: sort-heavy (p50 ms) | 2.11 | 17.08 | 8.09× | WIN |
-| read feat: deep from+size (from 500) (p50 ms) | 1.72 | 1.84 | 1.07× | WIN |
-| read feat: search_after (p50 ms) | 5.75 | 19.85 | 3.45× | WIN |
-| read feat: highlight (p50 ms) | 2.16 | 2.18 | 1.01× | WIN |
-| read feat: _count (p50 ms) | 1.77 | 1.96 | 1.11× | WIN |
-| read feat: _msearch (p50 ms) | 2.03 | 1.47 | 0.72× | LOSE |
-| read feat: _mget (p50 ms) | 2.18 | 1.87 | 0.86× | LOSE |
-| mixed match_all (p99 ms, under write) | 79.24 | 4.74 | 0.06× | LOSE |
-| mixed bool (p99 ms, under write) | 77.27 | 12.13 | 0.16× | LOSE |
-| mixed range (p99 ms, under write) | 149.09 | 7.71 | 0.05× | LOSE |
-| mixed terms (p99 ms, under write) | 73.15 | 4.32 | 0.06× | LOSE |
-| mixed cardinality (p99 ms, under write) | 80.86 | 22.14 | 0.27× | LOSE |
-| kNN k=10 (p50 ms) | 1.82 | 4.67 | 2.57× | WIN |
+| ingest 100k × c1 (docs/s) | 103,126 | 71,432 | 1.44× | WIN |
+| ingest 100k × c8 (docs/s) | 355,363 | 264,935 | 1.34× | WIN |
+| ingest 1m × c1 (docs/s) | 109,543 | 70,197 | 1.56× | WIN |
+| ingest 1m × c8 (docs/s) | 338,555 | 365,530 | 0.93× | LOSE |
+| read q: match_all (p50 ms) | 0.78 | 1.08 | 1.38× | WIN |
+| read q: match_none (p50 ms) | 0.57 | 0.77 | 1.35× | WIN |
+| read q: match(model) (p50 ms) | 0.54 | 0.96 | 1.77× | WIN |
+| read q: match_phrase(top_doc) (p50 ms) | 0.53 | 0.93 | 1.75× | WIN |
+| read q: match_phrase_prefix (p50 ms) | 0.46 | unsupported (400) | — | N/A |
+| read q: match_bool_prefix (p50 ms) | 0.56 | 3.91 | 6.94× | WIN |
+| read q: multi_match (p50 ms) | 0.74 | 5.17 | 7.03× | WIN |
+| read q: combined_fields (p50 ms) | 0.45 | unsupported (400) | — | N/A |
+| read q: query_string (p50 ms) | 0.61 | 8.46 | 13.97× | WIN |
+| read q: simple_query_string (p50 ms) | 1.33 | 1.18 | 0.89× | LOSE |
+| read q: more_like_this (p50 ms) | 0.46 | 1.12 | 2.42× | WIN |
+| read q: term(status) (p50 ms) | 0.69 | 1.29 | 1.86× | WIN |
+| read q: terms(model) (p50 ms) | 0.50 | 4.33 | 8.57× | WIN |
+| read q: range(latency_ms) (p50 ms) | 0.51 | 4.69 | 9.20× | WIN |
+| read q: range(@timestamp) (p50 ms) | 0.48 | 0.75 | 1.56× | WIN |
+| read q: range(cost_usd) (p50 ms) | 0.57 | 5.81 | 10.19× | WIN |
+| read q: prefix(model) (p50 ms) | 0.69 | 8.50 | 12.24× | WIN |
+| read q: wildcard(model) (p50 ms) | 0.58 | 8.65 | 14.96× | WIN |
+| read q: regexp(model) (p50 ms) | 0.55 | 8.73 | 15.84× | WIN |
+| read q: fuzzy(model) (p50 ms) | 0.61 | 1.53 | 2.49× | WIN |
+| read q: exists(cost_usd) (p50 ms) | 0.83 | 1.44 | 1.74× | WIN |
+| read q: ids (p50 ms) | 0.46 | 0.53 | 1.15× | WIN |
+| read q: term(cache_hit) (p50 ms) | 0.49 | 0.92 | 1.88× | WIN |
+| read q: bool must+filter+should+must_not (p50 ms) | 1.33 | 15.49 | 11.61× | WIN |
+| read q: constant_score (p50 ms) | 0.55 | 0.90 | 1.64× | WIN |
+| read q: boosting (p50 ms) | 0.68 | 33.95 | 49.64× | WIN |
+| read q: dis_max (p50 ms) | 0.61 | 4.10 | 6.76× | WIN |
+| read q: function_score (p50 ms) | 0.72 | 38.87 | 54.10× | WIN |
+| read q: pinned (p50 ms) | 0.41 | 14.53 | 35.21× | WIN |
+| read agg: avg (p50 ms) | 0.65 | 1.18 | 1.80× | WIN |
+| read agg: sum (p50 ms) | 0.30 | 1.27 | 4.25× | WIN |
+| read agg: min (p50 ms) | 0.50 | 1.19 | 2.37× | WIN |
+| read agg: max (p50 ms) | 0.44 | 1.02 | 2.33× | WIN |
+| read agg: stats (p50 ms) | 0.34 | 1.30 | 3.79× | WIN |
+| read agg: extended_stats (p50 ms) | 0.35 | 1.39 | 4.00× | WIN |
+| read agg: value_count (p50 ms) | 1.11 | 1.17 | 1.05× | WIN |
+| read agg: cardinality (p50 ms) | 0.32 | 1.12 | 3.53× | WIN |
+| read agg: percentiles (p50 ms) | 1.11 | 1.31 | 1.18× | WIN |
+| read agg: percentile_ranks (p50 ms) | 1.14 | 1.28 | 1.12× | WIN |
+| read agg: median_absolute_deviation (p50 ms) | 0.55 | 1.51 | 2.77× | WIN |
+| read agg: matrix_stats (p50 ms) | 1.22 | 2.21 | 1.82× | WIN |
+| read agg: scripted_metric (p50 ms) | 1.22 | 2.14 | 1.75× | WIN |
+| read agg: top_hits (sub) (p50 ms) | 0.54 | 1.46 | 2.69× | WIN |
+| read agg: terms (p50 ms) | 1.14 | 1.48 | 1.29× | WIN |
+| read agg: rare_terms (p50 ms) | 0.78 | 2.02 | 2.59× | WIN |
+| read agg: significant_terms (p50 ms) | 0.47 | 2.07 | 4.40× | WIN |
+| read agg: histogram (p50 ms) | 0.37 | 0.69 | 1.87× | WIN |
+| read agg: date_histogram (p50 ms) | 0.46 | 2.00 | 4.33× | WIN |
+| read agg: auto_date_histogram (p50 ms) | 0.47 | 0.94 | 2.00× | WIN |
+| read agg: variable_width_histogram (p50 ms) | 0.51 | 2.17 | 4.22× | WIN |
+| read agg: range (p50 ms) | 1.16 | 0.67 | 0.58× | LOSE |
+| read agg: date_range (p50 ms) | 0.35 | 1.53 | 4.39× | WIN |
+| read agg: filter (p50 ms) | 1.26 | 1.22 | 0.97× | LOSE |
+| read agg: filters (p50 ms) | 0.37 | 1.08 | 2.89× | WIN |
+| read agg: missing (p50 ms) | 0.43 | 0.62 | 1.43× | WIN |
+| read agg: global (p50 ms) | 1.15 | 0.62 | 0.54× | LOSE |
+| read agg: adjacency_matrix (p50 ms) | 0.38 | 2.02 | 5.36× | WIN |
+| read agg: composite (p50 ms) | 1.21 | 0.72 | 0.60× | LOSE |
+| read agg: random_sampler (p50 ms) | 0.36 | 4.82 | 13.24× | WIN |
+| read agg: terms+avg(cost) (p50 ms) | 1.22 | 0.76 | 0.63× | LOSE |
+| read pipe: sum_bucket (p50 ms) | 0.39 | 0.74 | 1.91× | WIN |
+| read pipe: avg_bucket (p50 ms) | 1.20 | 2.15 | 1.80× | WIN |
+| read pipe: max_bucket (p50 ms) | 0.37 | 0.71 | 1.93× | WIN |
+| read pipe: stats_bucket (p50 ms) | 0.59 | 2.15 | 3.66× | WIN |
+| read pipe: percentiles_bucket (p50 ms) | 1.19 | 1.39 | 1.17× | WIN |
+| read pipe: derivative (p50 ms) | 0.34 | 2.11 | 6.31× | WIN |
+| read pipe: cumulative_sum (p50 ms) | 1.19 | 2.07 | 1.74× | WIN |
+| read pipe: moving_fn (p50 ms) | 0.36 | 0.75 | 2.08× | WIN |
+| read pipe: serial_diff (p50 ms) | 1.19 | 0.89 | 0.75× | LOSE |
+| read pipe: bucket_script (p50 ms) | 0.33 | 0.65 | 1.97× | WIN |
+| read pipe: bucket_selector (p50 ms) | 1.17 | 2.17 | 1.86× | WIN |
+| read pipe: bucket_sort (p50 ms) | 0.43 | 0.70 | 1.62× | WIN |
+| read feat: sort-heavy (p50 ms) | 1.04 | 16.95 | 16.24× | WIN |
+| read feat: deep from+size (from 500) (p50 ms) | 1.41 | 1.21 | 0.86× | LOSE |
+| read feat: search_after (p50 ms) | 4.34 | 17.97 | 4.14× | WIN |
+| read feat: highlight (p50 ms) | 1.35 | 1.94 | 1.44× | WIN |
+| read feat: _count (p50 ms) | 0.24 | 1.73 | 7.30× | WIN |
+| read feat: _msearch (p50 ms) | 0.98 | 1.29 | 1.32× | WIN |
+| read feat: _mget (p50 ms) | 0.21 | 0.47 | 2.23× | WIN |
+| mixed match_all (p99 ms, under write) | 76.55 | 2.27 | 0.03× | LOSE |
+| mixed bool (p99 ms, under write) | 82.14 | 9.49 | 0.12× | LOSE |
+| mixed range (p99 ms, under write) | 171.28 | 4.71 | 0.03× | LOSE |
+| mixed terms (p99 ms, under write) | 75.60 | 2.76 | 0.04× | LOSE |
+| mixed cardinality (p99 ms, under write) | 81.84 | 17.33 | 0.21× | LOSE |
+| kNN k=10 (p50 ms) | 0.57 | 3.02 | 5.28× | WIN |
| kNN recall@10 | 100.0% | 100.0% | 1.00× | WIN |
-| index on-disk size | 669.3 MB | 818.0 MB | 1.22× | WIN |
+| index on-disk size | 644.4 MB | 809.7 MB | 1.26× | WIN |
## Skipped families (need a purpose-built index the flat corpus lacks)
@@ -110,4 +112,4 @@ Verdict is from XERJ's POV: WIN = XERJ better (lower latency / higher docs·s /
- `knn (as query/top-level)` — covered separately by --knn on a purpose-built dense_vector index
- `percolate` — parses but no-ops to match_none — not benchmarkable for correctness
-_Summary: 44 WIN, 45 LOSE, 2 N/A._
+_Summary: 75 WIN, 14 LOSE, 2 N/A._
diff --git a/demo/playbooks/SCORECARD.md b/demo/playbooks/SCORECARD.md
index b51772059..840974b00 100644
--- a/demo/playbooks/SCORECARD.md
+++ b/demo/playbooks/SCORECARD.md
@@ -4,99 +4,101 @@ Generated by `demo/playbooks/bench-matrix.mjs`. Identical workload, same machine
Verdict is from XERJ's POV: WIN = XERJ better (lower latency / higher docs·s / smaller disk / higher recall). Ratio is normalized so **>1× means XERJ is better**. Any LOSE fails CI.
+> **Methodology — read latency transport.** Read/mixed/kNN latencies are measured with a lean **keep-alive HTTP client** (Node core `http` + a shared per-host `http.Agent`, `maxSockets: 256`), applied **identically to both engines** for every operation and keyed only by `host:port` (both localhost). We do **not** use Node's global `fetch`/undici: it adds ~1.5ms of pure client overhead per request, which swamps the sub-millisecond server times of *both* engines and compresses every read into a noise band. Measured with the identical client hitting both engines (`size:0` avg over 300k): undici fetch → XERJ 1.61ms, ES 2.77ms; http keep-alive → XERJ 0.126ms, ES 0.283ms. The keep-alive client reveals each engine's **true server round-trip** (the ~2.2× gap the client overhead had hidden), not the Node client. Ingest/mixed/kNN bulk load uses `curl` identically for both engines.
+
| dimension | XERJ | ES | ratio | verdict |
|---|--:|--:|--:|:--:|
-| ingest 100k × c1 (docs/s) | 106,968 | 71,024 | 1.51× | WIN |
-| ingest 100k × c8 (docs/s) | 287,606 | 283,000 | 1.02× | WIN |
-| ingest 1m × c1 (docs/s) | 108,619 | 70,055 | 1.55× | WIN |
-| ingest 1m × c8 (docs/s) | 333,148 | 378,298 | 0.88× | LOSE |
-| read q: match_all (p50 ms) | 1.88 | 2.86 | 1.52× | WIN |
-| read q: match_none (p50 ms) | 0.84 | 2.16 | 2.58× | WIN |
-| read q: match(model) (p50 ms) | 1.29 | 2.63 | 2.03× | WIN |
-| read q: match_phrase(top_doc) (p50 ms) | 1.49 | 2.27 | 1.53× | WIN |
-| read q: match_phrase_prefix (p50 ms) | 1.75 | unsupported (400) | — | N/A |
-| read q: match_bool_prefix (p50 ms) | 1.49 | 6.16 | 4.13× | WIN |
-| read q: multi_match (p50 ms) | 2.40 | 6.28 | 2.62× | WIN |
-| read q: combined_fields (p50 ms) | 2.04 | unsupported (400) | — | N/A |
-| read q: query_string (p50 ms) | 0.86 | 10.06 | 11.65× | WIN |
-| read q: simple_query_string (p50 ms) | 2.14 | 2.35 | 1.10× | WIN |
-| read q: more_like_this (p50 ms) | 1.79 | 2.09 | 1.17× | WIN |
-| read q: term(status) (p50 ms) | 1.78 | 2.14 | 1.20× | WIN |
-| read q: terms(model) (p50 ms) | 2.16 | 4.87 | 2.26× | WIN |
-| read q: range(latency_ms) (p50 ms) | 2.05 | 5.96 | 2.91× | WIN |
-| read q: range(@timestamp) (p50 ms) | 2.36 | 1.64 | 0.69× | LOSE |
-| read q: range(cost_usd) (p50 ms) | 2.77 | 7.24 | 2.61× | WIN |
-| read q: prefix(model) (p50 ms) | 1.85 | 9.97 | 5.38× | WIN |
-| read q: wildcard(model) (p50 ms) | 1.86 | 9.86 | 5.31× | WIN |
-| read q: regexp(model) (p50 ms) | 2.46 | 9.49 | 3.86× | WIN |
-| read q: fuzzy(model) (p50 ms) | 2.48 | 2.24 | 0.91× | LOSE |
-| read q: exists(cost_usd) (p50 ms) | 2.21 | 1.23 | 0.56× | LOSE |
-| read q: ids (p50 ms) | 1.75 | 1.43 | 0.82× | LOSE |
-| read q: term(cache_hit) (p50 ms) | 1.96 | 1.62 | 0.83× | LOSE |
-| read q: bool must+filter+should+must_not (p50 ms) | 2.19 | 16.07 | 7.34× | WIN |
-| read q: constant_score (p50 ms) | 1.78 | 1.75 | 0.98× | LOSE |
-| read q: boosting (p50 ms) | 2.23 | 34.08 | 15.26× | WIN |
-| read q: dis_max (p50 ms) | 2.27 | 6.23 | 2.75× | WIN |
-| read q: function_score (p50 ms) | 1.74 | 39.85 | 22.85× | WIN |
-| read q: pinned (p50 ms) | 1.79 | 15.38 | 8.61× | WIN |
-| read agg: avg (p50 ms) | 2.57 | 1.88 | 0.73× | LOSE |
-| read agg: sum (p50 ms) | 2.42 | 2.20 | 0.91× | LOSE |
-| read agg: min (p50 ms) | 2.13 | 2.11 | 0.99× | LOSE |
-| read agg: max (p50 ms) | 1.96 | 2.13 | 1.09× | WIN |
-| read agg: stats (p50 ms) | 2.50 | 1.50 | 0.60× | LOSE |
-| read agg: extended_stats (p50 ms) | 1.90 | 1.86 | 0.98× | LOSE |
-| read agg: value_count (p50 ms) | 1.80 | 1.87 | 1.04× | WIN |
-| read agg: cardinality (p50 ms) | 2.52 | 1.90 | 0.75× | LOSE |
-| read agg: percentiles (p50 ms) | 1.66 | 1.94 | 1.16× | WIN |
-| read agg: percentile_ranks (p50 ms) | 2.60 | 2.36 | 0.91× | LOSE |
-| read agg: median_absolute_deviation (p50 ms) | 1.98 | 2.05 | 1.04× | WIN |
-| read agg: matrix_stats (p50 ms) | 2.08 | 1.38 | 0.66× | LOSE |
-| read agg: scripted_metric (p50 ms) | 1.24 | 1.65 | 1.33× | WIN |
-| read agg: top_hits (sub) (p50 ms) | 1.68 | 2.20 | 1.31× | WIN |
-| read agg: terms (p50 ms) | 1.38 | 1.72 | 1.25× | WIN |
-| read agg: rare_terms (p50 ms) | 2.38 | 2.01 | 0.85× | LOSE |
-| read agg: significant_terms (p50 ms) | 1.57 | 1.32 | 0.84× | LOSE |
-| read agg: histogram (p50 ms) | 2.61 | 2.04 | 0.78× | LOSE |
-| read agg: date_histogram (p50 ms) | 2.21 | 1.95 | 0.88× | LOSE |
-| read agg: auto_date_histogram (p50 ms) | 1.98 | 1.66 | 0.84× | LOSE |
-| read agg: variable_width_histogram (p50 ms) | 1.87 | 2.48 | 1.32× | WIN |
-| read agg: range (p50 ms) | 2.27 | 1.52 | 0.67× | LOSE |
-| read agg: date_range (p50 ms) | 2.57 | 1.68 | 0.65× | LOSE |
-| read agg: filter (p50 ms) | 2.33 | 1.92 | 0.82× | LOSE |
-| read agg: filters (p50 ms) | 2.17 | 1.37 | 0.63× | LOSE |
-| read agg: missing (p50 ms) | 2.25 | 1.18 | 0.53× | LOSE |
-| read agg: global (p50 ms) | 2.30 | 1.56 | 0.68× | LOSE |
-| read agg: adjacency_matrix (p50 ms) | 2.17 | 1.77 | 0.82× | LOSE |
-| read agg: composite (p50 ms) | 1.78 | 1.95 | 1.09× | WIN |
-| read agg: random_sampler (p50 ms) | 2.63 | 5.36 | 2.04× | WIN |
-| read agg: terms+avg(cost) (p50 ms) | 1.67 | 1.72 | 1.03× | WIN |
-| read pipe: sum_bucket (p50 ms) | 2.55 | 2.23 | 0.87× | LOSE |
-| read pipe: avg_bucket (p50 ms) | 1.89 | 1.49 | 0.79× | LOSE |
-| read pipe: max_bucket (p50 ms) | 2.42 | 2.20 | 0.91× | LOSE |
-| read pipe: stats_bucket (p50 ms) | 2.13 | 2.11 | 0.99× | LOSE |
-| read pipe: percentiles_bucket (p50 ms) | 2.57 | 2.45 | 0.95× | LOSE |
-| read pipe: derivative (p50 ms) | 2.46 | 2.01 | 0.82× | LOSE |
-| read pipe: cumulative_sum (p50 ms) | 1.86 | 1.65 | 0.89× | LOSE |
-| read pipe: moving_fn (p50 ms) | 2.13 | 1.45 | 0.68× | LOSE |
-| read pipe: serial_diff (p50 ms) | 2.46 | 1.48 | 0.60× | LOSE |
-| read pipe: bucket_script (p50 ms) | 2.20 | 1.78 | 0.81× | LOSE |
-| read pipe: bucket_selector (p50 ms) | 2.42 | 2.65 | 1.10× | WIN |
-| read pipe: bucket_sort (p50 ms) | 2.68 | 2.33 | 0.87× | LOSE |
-| read feat: sort-heavy (p50 ms) | 2.11 | 17.08 | 8.09× | WIN |
-| read feat: deep from+size (from 500) (p50 ms) | 1.72 | 1.84 | 1.07× | WIN |
-| read feat: search_after (p50 ms) | 5.75 | 19.85 | 3.45× | WIN |
-| read feat: highlight (p50 ms) | 2.16 | 2.18 | 1.01× | WIN |
-| read feat: _count (p50 ms) | 1.77 | 1.96 | 1.11× | WIN |
-| read feat: _msearch (p50 ms) | 2.03 | 1.47 | 0.72× | LOSE |
-| read feat: _mget (p50 ms) | 2.18 | 1.87 | 0.86× | LOSE |
-| mixed match_all (p99 ms, under write) | 79.24 | 4.74 | 0.06× | LOSE |
-| mixed bool (p99 ms, under write) | 77.27 | 12.13 | 0.16× | LOSE |
-| mixed range (p99 ms, under write) | 149.09 | 7.71 | 0.05× | LOSE |
-| mixed terms (p99 ms, under write) | 73.15 | 4.32 | 0.06× | LOSE |
-| mixed cardinality (p99 ms, under write) | 80.86 | 22.14 | 0.27× | LOSE |
-| kNN k=10 (p50 ms) | 1.82 | 4.67 | 2.57× | WIN |
+| ingest 100k × c1 (docs/s) | 103,126 | 71,432 | 1.44× | WIN |
+| ingest 100k × c8 (docs/s) | 355,363 | 264,935 | 1.34× | WIN |
+| ingest 1m × c1 (docs/s) | 109,543 | 70,197 | 1.56× | WIN |
+| ingest 1m × c8 (docs/s) | 338,555 | 365,530 | 0.93× | LOSE |
+| read q: match_all (p50 ms) | 0.78 | 1.08 | 1.38× | WIN |
+| read q: match_none (p50 ms) | 0.57 | 0.77 | 1.35× | WIN |
+| read q: match(model) (p50 ms) | 0.54 | 0.96 | 1.77× | WIN |
+| read q: match_phrase(top_doc) (p50 ms) | 0.53 | 0.93 | 1.75× | WIN |
+| read q: match_phrase_prefix (p50 ms) | 0.46 | unsupported (400) | — | N/A |
+| read q: match_bool_prefix (p50 ms) | 0.56 | 3.91 | 6.94× | WIN |
+| read q: multi_match (p50 ms) | 0.74 | 5.17 | 7.03× | WIN |
+| read q: combined_fields (p50 ms) | 0.45 | unsupported (400) | — | N/A |
+| read q: query_string (p50 ms) | 0.61 | 8.46 | 13.97× | WIN |
+| read q: simple_query_string (p50 ms) | 1.33 | 1.18 | 0.89× | LOSE |
+| read q: more_like_this (p50 ms) | 0.46 | 1.12 | 2.42× | WIN |
+| read q: term(status) (p50 ms) | 0.69 | 1.29 | 1.86× | WIN |
+| read q: terms(model) (p50 ms) | 0.50 | 4.33 | 8.57× | WIN |
+| read q: range(latency_ms) (p50 ms) | 0.51 | 4.69 | 9.20× | WIN |
+| read q: range(@timestamp) (p50 ms) | 0.48 | 0.75 | 1.56× | WIN |
+| read q: range(cost_usd) (p50 ms) | 0.57 | 5.81 | 10.19× | WIN |
+| read q: prefix(model) (p50 ms) | 0.69 | 8.50 | 12.24× | WIN |
+| read q: wildcard(model) (p50 ms) | 0.58 | 8.65 | 14.96× | WIN |
+| read q: regexp(model) (p50 ms) | 0.55 | 8.73 | 15.84× | WIN |
+| read q: fuzzy(model) (p50 ms) | 0.61 | 1.53 | 2.49× | WIN |
+| read q: exists(cost_usd) (p50 ms) | 0.83 | 1.44 | 1.74× | WIN |
+| read q: ids (p50 ms) | 0.46 | 0.53 | 1.15× | WIN |
+| read q: term(cache_hit) (p50 ms) | 0.49 | 0.92 | 1.88× | WIN |
+| read q: bool must+filter+should+must_not (p50 ms) | 1.33 | 15.49 | 11.61× | WIN |
+| read q: constant_score (p50 ms) | 0.55 | 0.90 | 1.64× | WIN |
+| read q: boosting (p50 ms) | 0.68 | 33.95 | 49.64× | WIN |
+| read q: dis_max (p50 ms) | 0.61 | 4.10 | 6.76× | WIN |
+| read q: function_score (p50 ms) | 0.72 | 38.87 | 54.10× | WIN |
+| read q: pinned (p50 ms) | 0.41 | 14.53 | 35.21× | WIN |
+| read agg: avg (p50 ms) | 0.65 | 1.18 | 1.80× | WIN |
+| read agg: sum (p50 ms) | 0.30 | 1.27 | 4.25× | WIN |
+| read agg: min (p50 ms) | 0.50 | 1.19 | 2.37× | WIN |
+| read agg: max (p50 ms) | 0.44 | 1.02 | 2.33× | WIN |
+| read agg: stats (p50 ms) | 0.34 | 1.30 | 3.79× | WIN |
+| read agg: extended_stats (p50 ms) | 0.35 | 1.39 | 4.00× | WIN |
+| read agg: value_count (p50 ms) | 1.11 | 1.17 | 1.05× | WIN |
+| read agg: cardinality (p50 ms) | 0.32 | 1.12 | 3.53× | WIN |
+| read agg: percentiles (p50 ms) | 1.11 | 1.31 | 1.18× | WIN |
+| read agg: percentile_ranks (p50 ms) | 1.14 | 1.28 | 1.12× | WIN |
+| read agg: median_absolute_deviation (p50 ms) | 0.55 | 1.51 | 2.77× | WIN |
+| read agg: matrix_stats (p50 ms) | 1.22 | 2.21 | 1.82× | WIN |
+| read agg: scripted_metric (p50 ms) | 1.22 | 2.14 | 1.75× | WIN |
+| read agg: top_hits (sub) (p50 ms) | 0.54 | 1.46 | 2.69× | WIN |
+| read agg: terms (p50 ms) | 1.14 | 1.48 | 1.29× | WIN |
+| read agg: rare_terms (p50 ms) | 0.78 | 2.02 | 2.59× | WIN |
+| read agg: significant_terms (p50 ms) | 0.47 | 2.07 | 4.40× | WIN |
+| read agg: histogram (p50 ms) | 0.37 | 0.69 | 1.87× | WIN |
+| read agg: date_histogram (p50 ms) | 0.46 | 2.00 | 4.33× | WIN |
+| read agg: auto_date_histogram (p50 ms) | 0.47 | 0.94 | 2.00× | WIN |
+| read agg: variable_width_histogram (p50 ms) | 0.51 | 2.17 | 4.22× | WIN |
+| read agg: range (p50 ms) | 1.16 | 0.67 | 0.58× | LOSE |
+| read agg: date_range (p50 ms) | 0.35 | 1.53 | 4.39× | WIN |
+| read agg: filter (p50 ms) | 1.26 | 1.22 | 0.97× | LOSE |
+| read agg: filters (p50 ms) | 0.37 | 1.08 | 2.89× | WIN |
+| read agg: missing (p50 ms) | 0.43 | 0.62 | 1.43× | WIN |
+| read agg: global (p50 ms) | 1.15 | 0.62 | 0.54× | LOSE |
+| read agg: adjacency_matrix (p50 ms) | 0.38 | 2.02 | 5.36× | WIN |
+| read agg: composite (p50 ms) | 1.21 | 0.72 | 0.60× | LOSE |
+| read agg: random_sampler (p50 ms) | 0.36 | 4.82 | 13.24× | WIN |
+| read agg: terms+avg(cost) (p50 ms) | 1.22 | 0.76 | 0.63× | LOSE |
+| read pipe: sum_bucket (p50 ms) | 0.39 | 0.74 | 1.91× | WIN |
+| read pipe: avg_bucket (p50 ms) | 1.20 | 2.15 | 1.80× | WIN |
+| read pipe: max_bucket (p50 ms) | 0.37 | 0.71 | 1.93× | WIN |
+| read pipe: stats_bucket (p50 ms) | 0.59 | 2.15 | 3.66× | WIN |
+| read pipe: percentiles_bucket (p50 ms) | 1.19 | 1.39 | 1.17× | WIN |
+| read pipe: derivative (p50 ms) | 0.34 | 2.11 | 6.31× | WIN |
+| read pipe: cumulative_sum (p50 ms) | 1.19 | 2.07 | 1.74× | WIN |
+| read pipe: moving_fn (p50 ms) | 0.36 | 0.75 | 2.08× | WIN |
+| read pipe: serial_diff (p50 ms) | 1.19 | 0.89 | 0.75× | LOSE |
+| read pipe: bucket_script (p50 ms) | 0.33 | 0.65 | 1.97× | WIN |
+| read pipe: bucket_selector (p50 ms) | 1.17 | 2.17 | 1.86× | WIN |
+| read pipe: bucket_sort (p50 ms) | 0.43 | 0.70 | 1.62× | WIN |
+| read feat: sort-heavy (p50 ms) | 1.04 | 16.95 | 16.24× | WIN |
+| read feat: deep from+size (from 500) (p50 ms) | 1.41 | 1.21 | 0.86× | LOSE |
+| read feat: search_after (p50 ms) | 4.34 | 17.97 | 4.14× | WIN |
+| read feat: highlight (p50 ms) | 1.35 | 1.94 | 1.44× | WIN |
+| read feat: _count (p50 ms) | 0.24 | 1.73 | 7.30× | WIN |
+| read feat: _msearch (p50 ms) | 0.98 | 1.29 | 1.32× | WIN |
+| read feat: _mget (p50 ms) | 0.21 | 0.47 | 2.23× | WIN |
+| mixed match_all (p99 ms, under write) | 76.55 | 2.27 | 0.03× | LOSE |
+| mixed bool (p99 ms, under write) | 82.14 | 9.49 | 0.12× | LOSE |
+| mixed range (p99 ms, under write) | 171.28 | 4.71 | 0.03× | LOSE |
+| mixed terms (p99 ms, under write) | 75.60 | 2.76 | 0.04× | LOSE |
+| mixed cardinality (p99 ms, under write) | 81.84 | 17.33 | 0.21× | LOSE |
+| kNN k=10 (p50 ms) | 0.57 | 3.02 | 5.28× | WIN |
| kNN recall@10 | 100.0% | 100.0% | 1.00× | WIN |
-| index on-disk size | 669.3 MB | 818.0 MB | 1.22× | WIN |
+| index on-disk size | 644.4 MB | 809.7 MB | 1.26× | WIN |
## Skipped families (need a purpose-built index the flat corpus lacks)
@@ -110,4 +112,4 @@ Verdict is from XERJ's POV: WIN = XERJ better (lower latency / higher docs·s /
- `knn (as query/top-level)` — covered separately by --knn on a purpose-built dense_vector index
- `percolate` — parses but no-ops to match_none — not benchmarkable for correctness
-_Summary: 44 WIN, 45 LOSE, 2 N/A._
+_Summary: 75 WIN, 14 LOSE, 2 N/A._
diff --git a/demo/playbooks/bench-matrix.mjs b/demo/playbooks/bench-matrix.mjs
index bf95fd88e..8c301833a 100644
--- a/demo/playbooks/bench-matrix.mjs
+++ b/demo/playbooks/bench-matrix.mjs
@@ -22,8 +22,12 @@
// node demo/playbooks/bench-matrix.mjs --reads-only
// node demo/playbooks/bench-matrix.mjs --ingest-only --docs 1m --clients 1,4,8
//
-// Requires: Node 24, global fetch, curl on PATH. No deps beyond node builtins.
+// Requires: Node 24, curl on PATH. Reads go through Node-core http keep-alive
+// (NOT the global fetch/undici — see the transport note above req()). No deps
+// beyond node builtins.
import fs from 'node:fs';
+import http from 'node:http';
+import path from 'node:path';
import { exec, spawn } from 'node:child_process';
import { promisify } from 'node:util';
@@ -34,12 +38,56 @@ const BATCH = 10000;
const CORPUS = '/home/claude/ai/xerj/demo/data/extras/chat-events.ndjson';
const SCRATCH = '/tmp/xerj';
const DEFAULT_OUT = '/home/claude/ai/xerj/demo/playbooks/SCORECARD.md';
-const HEADTOHEAD_OUT = '/home/claude/ai/xerj/demo/playbooks/BENCHMARK_VS_ES.md';
+// BENCHMARK_VS_ES.md (the head-to-head slice kept for continuity with
+// bench-vs-es.mjs) is written next to whatever --out resolves to, so a
+// redirected run (e.g. --out into a worktree) keeps both files co-located
+// rather than scattering the head-to-head copy back into the default dir.
+const HEADTOHEAD_NAME = 'BENCHMARK_VS_ES.md';
+
+// ────────────────────────────── HTTP transport ──────────────────────────────
+// TRANSPORT NOTE — why NOT global fetch (undici):
+// Read latency was previously measured through Node's global `fetch` (undici),
+// which adds ~1.5ms of pure CLIENT overhead per request. That overhead swamps
+// the sub-millisecond SERVER times of BOTH engines and compresses every read
+// into a single noise band, so the scorecard measured Node's client, not the
+// engines. Measured with the IDENTICAL client hitting both engines (size:0 avg
+// over a 300k corpus):
+// undici fetch: XERJ 1.61ms, ES 2.77ms
+// http keep-alive: XERJ 0.126ms, ES 0.283ms (~2.2× true server gap revealed)
+// A raw TCP socket is even leaner (XERJ 0.054ms) but can't parse chunked
+// responses; Node's core http parser handles BOTH content-length and chunked,
+// so it is fair to ES too. We therefore drive every request through one shared,
+// per-host keep-alive http.Agent.
+//
+// FAIRNESS INVARIANT: the SAME Agent config, SAME code path, and SAME headers
+// are applied to BOTH engines for ALL operations (reads, ingest setup, mixed,
+// kNN) — the agent is keyed ONLY by host:port, both localhost. Neither engine
+// gets any client advantage; this measures each engine's true server round-trip.
+//
+// maxSockets 256 >= the max concurrency the matrix ever offers (ingest fans out
+// up to `clients` concurrent workers via curl; mixed runs a background writer +
+// a reader burst; open-loop timed() may hold several requests in flight at once)
+// so the client never becomes the bottleneck for either engine.
+const AGENTS = new Map();
+function agentFor(hostname, port) {
+ const key = `${hostname}:${port}`;
+ let a = AGENTS.get(key);
+ if (!a) { a = new http.Agent({ keepAlive: true, maxSockets: 256 }); AGENTS.set(key, a); }
+ return a;
+}
+// Destroy all pooled keep-alive sockets so the event loop can drain and Node can
+// exit cleanly at the end of a run (keep-alive sockets would otherwise linger).
+function destroyAgents() { for (const a of AGENTS.values()) { try { a.destroy(); } catch {} } }
// ────────────────────────────── helpers (lifted from bench-vs-es.mjs) ──────────────────────────────
-async function req(base, method, p, body, ndjson = false) {
+// Lean keep-alive HTTP request. Preserves the exact prior interface/semantics of
+// the fetch-based req(): returns {status, j, txt} on success and
+// {status:0, j:null, txt:'', failed:true} on collapse/timeout; injects
+// track_total_hits:true for JSON _search bodies; honours the ndjson content-type;
+// routes GET/POST/PUT/DELETE; and bounds each request at 15s (was an
+// AbortController, now a socket timeout that destroys the request → failed).
+function req(base, method, p, body, ndjson = false) {
const headers = { 'content-type': ndjson ? 'application/x-ndjson' : 'application/json' };
- const opts = { method, headers };
// Apples-to-apples total-hits: ES caps hits.total.value at 10,000 by default
// (track_total_hits=10000, relation "gte") while XERJ returns the exact total.
// Force BOTH engines to compute the true total so (a) hit counts are
@@ -54,22 +102,37 @@ async function req(base, method, p, body, ndjson = false) {
&& /\/_search(\?|$)/.test(p) && body.track_total_hits === undefined) {
sendBody = { ...body, track_total_hits: true };
}
- if (sendBody !== undefined) opts.body = ndjson ? sendBody : JSON.stringify(sendBody);
- // Bound each request so an engine that collapses under load surfaces as a
- // timeout (recorded, scored) instead of hanging the whole matrix.
- const ac = new AbortController();
- const to = setTimeout(() => ac.abort(), 15000);
- opts.signal = ac.signal;
- let r, txt = '', j = null, failed = false;
- try {
- r = await fetch(base + p, opts);
- txt = await r.text();
- try { j = JSON.parse(txt); } catch {}
- } catch (e) {
- failed = true; // timeout / connection reset — engine unresponsive under load
- } finally { clearTimeout(to); }
- if (failed) return { status: 0, j: null, txt: '', failed: true };
- return { status: r.status, j, txt };
+ let payload;
+ if (sendBody !== undefined) payload = ndjson ? sendBody : JSON.stringify(sendBody);
+ if (payload !== undefined) headers['content-length'] = Buffer.byteLength(payload);
+ const u = new URL(base + p);
+ return new Promise((resolve) => {
+ let settled = false;
+ const done = (v) => { if (!settled) { settled = true; resolve(v); } };
+ const fail = () => done({ status: 0, j: null, txt: '', failed: true }); // collapse/timeout — engine unresponsive
+ const r = http.request({
+ protocol: u.protocol, hostname: u.hostname, port: u.port,
+ path: u.pathname + u.search, method, headers,
+ agent: agentFor(u.hostname, u.port),
+ }, (res) => {
+ let txt = '';
+ res.setEncoding('utf8');
+ res.on('data', (c) => { txt += c; });
+ res.on('end', () => {
+ let j = null;
+ try { j = JSON.parse(txt); } catch {}
+ done({ status: res.statusCode, j, txt });
+ });
+ res.on('error', fail);
+ });
+ // Bound each request so an engine that collapses under load surfaces as a
+ // timeout (recorded, scored) instead of hanging the whole matrix. Equivalent
+ // to the old 15s AbortController: destroy the request → 'error' → failed.
+ r.setTimeout(15000, () => r.destroy(new Error('timeout')));
+ r.on('error', fail);
+ if (payload !== undefined) r.write(payload);
+ r.end();
+ });
}
// percentile: p in [0,100], nearest-rank on a sorted copy.
@@ -135,10 +198,20 @@ function signalMismatch(xs, es) {
//
// Coordinated-omission correction: request start times are scheduled on a FIXED
// cadence (t0 + i/rate) independent of when prior responses return; requests may
-// be concurrently in flight; latency is measured from the INTENDED start time, so
-// a server stall (GC/flush/merge) inflates the tail of the backlog instead of
-// silently shifting the schedule and omitting the would-be-slow samples. Also
-// reports offered vs achieved rate so the load level is visible.
+// be concurrently in flight. Latency uses the CANONICAL CO form — service time
+// PLUS any scheduling backlog: lat = (end - actualStart) + max(0, actualStart -
+// intended) = end - min(intended, actualStart). When a request starts on-or-after
+// its slot (server stall / GC / flush / merge pushes actualStart >= intended)
+// this is exactly end - intended, so the stall still inflates the backlog tail
+// instead of being silently omitted — identical to the prior formula. But it is
+// also numerically robust at SUB-MILLISECOND service times: Node's ms-resolution
+// setTimeout releases ~half of the slots a few hundred µs EARLY (measured: 70/120
+// fires up to 1.3ms before `intended`), so the old `end - intended` produced
+// NEGATIVE latencies once the lean keep-alive transport exposed <0.2ms server
+// round-trips. Measuring from min(intended, actualStart) removes that artifact
+// (a request can never be "faster than it started") while preserving CO exactly
+// in the backlog regime. Applied identically to both engines. Also reports
+// offered vs achieved rate so the load level is visible.
async function timed(base, p, body, opts = {}) {
const { iters = 120, warmup = 15, method = 'POST', ndjson = false, rate = 200 } = opts;
const chk = await req(base, method, p, body, ndjson);
@@ -157,8 +230,10 @@ async function timed(base, p, body, opts = {}) {
tasks[i] = (async () => {
const wait = intended - performance.now();
if (wait > 0) await new Promise((r) => setTimeout(r, wait));
+ const start = performance.now(); // when the request actually left the client
await req(base, method, p, body, ndjson);
- lat[i] = performance.now() - intended; // CO-corrected: measured from intended start
+ // service time + scheduling backlog; == end - intended when start >= intended.
+ lat[i] = performance.now() - Math.min(intended, start);
})();
}
await Promise.all(tasks);
@@ -746,6 +821,13 @@ async function main() {
md += `docs = {${A.docsList.map(humanDocs).join(', ')}}, clients = {${A.clientsList.join(', ')}}.\n\n`;
md += `Verdict is from XERJ's POV: WIN = XERJ better (lower latency / higher docs·s / smaller disk / higher recall). `;
md += `Ratio is normalized so **>1× means XERJ is better**. Any LOSE fails CI.\n\n`;
+ md += `> **Methodology — read latency transport.** Read/mixed/kNN latencies are measured with a lean **keep-alive HTTP client** `;
+ md += `(Node core \`http\` + a shared per-host \`http.Agent\`, \`maxSockets: 256\`), applied **identically to both engines** for every `;
+ md += `operation and keyed only by \`host:port\` (both localhost). We do **not** use Node's global \`fetch\`/undici: it adds ~1.5ms of pure `;
+ md += `client overhead per request, which swamps the sub-millisecond server times of *both* engines and compresses every read into a noise band. `;
+ md += `Measured with the identical client hitting both engines (\`size:0\` avg over 300k): undici fetch → XERJ 1.61ms, ES 2.77ms; `;
+ md += `http keep-alive → XERJ 0.126ms, ES 0.283ms. The keep-alive client reveals each engine's **true server round-trip** (the ~2.2× gap the `;
+ md += `client overhead had hidden), not the Node client. Ingest/mixed/kNN bulk load uses \`curl\` identically for both engines.\n\n`;
md += `| dimension | XERJ | ES | ratio | verdict |\n|---|--:|--:|--:|:--:|\n`;
for (const r of rows) md += `| ${r.dim} | ${r.xs} | ${r.es} | ${r.ratio} | ${r.verdict} |\n`;
md += `\n`;
@@ -757,11 +839,14 @@ async function main() {
md += `_Summary: ${rows.filter((r) => r.verdict === 'WIN').length} WIN, ${loses.length} LOSE, ${rows.filter((r) => r.verdict === 'N/A').length} N/A._\n`;
fs.writeFileSync(A.out, md);
- // keep the head-to-head slice around for continuity with bench-vs-es.mjs
- try { fs.writeFileSync(HEADTOHEAD_OUT, md); } catch {}
+ // keep the head-to-head slice around for continuity with bench-vs-es.mjs,
+ // co-located with the scorecard (dirname of A.out) so a redirected run keeps
+ // both files together.
+ try { fs.writeFileSync(path.join(path.dirname(A.out), HEADTOHEAD_NAME), md); } catch {}
log(`\nwrote ${A.out}`);
console.log(md);
+ destroyAgents(); // release keep-alive sockets so the loop can drain
if (loses.length) { log(`\nFAIL: ${loses.length} LOSE cell(s) — ${loses.map((r) => r.dim).join('; ')}`); process.exit(1); }
}