AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
bench: XERJ vs real Elasticsearch 8.13.4 head-to-head
xerj-org17b641778619
diff --git a/demo/playbooks/BENCHMARK_VS_ES.md b/demo/playbooks/BENCHMARK_VS_ES.mdnew file mode 100644index 000000000..58366b533--- /dev/null+++ b/demo/playbooks/BENCHMARK_VS_ES.md@@ -0,0 +1,78 @@+# XERJ vs Elasticsearch — head-to-head benchmark++Real Elasticsearch 8.13.4 (Lucene 9.10.0) downloaded and run alongside XERJ+8.13.0, **identical workloads, same machine, same corpus**. The only variable is+the engine. Reproduce with `node demo/playbooks/bench-vs-es.mjs <N> <xerj_url> <es_url>`.++## Setup (fair-comparison notes)++| | XERJ | Elasticsearch 8.13.4 |+|---|---|---|+| Process | single Rust binary, native heap | JVM, `-Xmx4g -Xms4g` |+| Security/TLS | `--insecure` (off) | `xpack.security.enabled: false` |+| Topology | single node, 1 shard | single node, 1 shard (replica unassigned) |+| Mapping | identical (keyword/date/int/double/dense_vector) | identical |+| Client | same Node.js harness, localhost, sequential bulk (10k/batch), `_refresh` after ingest | same |+| Latency metric | **end-to-end client wall-clock** (not server `took`) — 80 iters after 15 warmup | same |++Machine: AMD Ryzen AI MAX+ 395, 32 cores, 119 GB RAM. Corpus: real LLM-telemetry+events. Both engines were given the same per-batch file-write + curl overhead, so+ingest deltas are real engine time.++## Ingest throughput (bulk, sequential, 10k/batch)++| Engine | 100k docs | 1,000,000 docs |+|---|--:|--:|+| **Elasticsearch** | 68,464 docs/s | **110,012 docs/s** |+| **XERJ** | 31,429 docs/s | 22,417 docs/s |+| ES advantage | 2.2× | **4.9×** |++**Elasticsearch wins bulk ingest decisively, and the gap widens with scale** —+Lucene's segment-based bulk indexing is extremely mature, and ES throughput+*rose* from 100k→1M (68k→110k docs/s) while XERJ's *fell* (31k→22k), reflecting+segment-flush / merge pressure that XERJ's single-binary ingest path does not yet+amortize as well. **If your workload is write-heavy at scale, ES indexes faster.**++## Read-path latency — 1,000,000 docs (ms, end-to-end, 80 iters)++| operation | XERJ p50 | ES p50 | XERJ p95 | ES p95 | XERJ p99 | ES p99 | p50: XERJ faster by |+|---|--:|--:|--:|--:|--:|--:|--:|+| match_all (size 10) | **1.52** | 2.51 | 1.88 | 3.47 | 6.16 | 3.72 | 1.65× |+| term filter | **1.51** | 2.34 | 1.82 | 3.06 | 2.01 | 3.29 | 1.55× |+| bool must+filter | **1.48** | 2.73 | 1.92 | 4.26 | 3.00 | 4.84 | 1.84× |+| range | **1.46** | 2.41 | 1.55 | 3.21 | 1.99 | 3.44 | 1.65× |+| agg: terms(model) | **1.39** | 2.09 | 1.59 | 2.52 | 1.97 | 3.18 | 1.51× |+| agg: stats(latency_ms) | **1.38** | 2.04 | 1.98 | 3.07 | 2.39 | 3.40 | 1.48× |+| agg: date_histogram(day) | **1.28** | 1.94 | 1.81 | 2.29 | 70.06¹ | 2.47 | 1.51× |+| agg: terms+avg(cost) | **1.28** | 1.94 | 1.47 | 2.45 | 2.31 | 3.11 | 1.51× |+| agg: cardinality(top_doc) | **1.24** | 1.84 | 1.43 | 2.90 | 1.99 | 3.22 | 1.49× |+| _count match_all | **1.19** | 2.10 | 1.25 | 2.94 | 4.98 | 3.29 | 1.76× |+| kNN k=10 (20k×16d) | **1.36** | 2.52 | 1.49 | 4.17 | 1.70 | 4.95 | 1.85× |++**XERJ wins read latency on every operation — ~1.5–1.85× lower p50** for queries,+aggregations, and vector kNN. The lean Rust request path has materially less+per-request overhead than the JVM stack: XERJ p50 sits at 1.2–1.5 ms where ES is+1.8–2.7 ms. XERJ's p95 is also lower across the board.++¹ XERJ p99 outliers (e.g. one 70 ms date_histogram, occasional 5–6 ms match_all)+are tail events from a background segment flush/merge coinciding with a sampled+request — ES's mature merge scheduler produces a tighter tail (p99 ≤ ~5 ms). This+is the same ingest-side machinery that costs XERJ on write throughput showing up+on the read tail under concurrent background work.++## Bottom line++| Dimension | Winner | Margin |+|---|---|---|+| **Read latency** (query / agg / kNN p50, p95) | **XERJ** | 1.5–1.85× lower |+| **Bulk ingest throughput** | **Elasticsearch** | 2.2× (100k) → 4.9× (1M) |+| **Tail latency under background merges** (p99) | **Elasticsearch** | tighter |+| Footprint / startup | **XERJ** | single ~22 MB binary, native heap, sub-second start vs JVM + 4 GB |++**XERJ is the faster engine for the read/serve path** — lower, steadier query and+vector latency from a tiny single binary — and is **drop-in compatible** (100% of+the 1,326-case ES YAML wire-conformance suite passes, version 8.13). **Elasticsearch+is the faster bulk indexer**, especially at scale, with a tighter latency tail under+heavy background indexing. The honest read: pick XERJ for low-latency search/serving+and a minimal operational footprint; pick ES (today) for write-heavy ingest at scale.+Closing XERJ's ingest-throughput and p99-tail gap is the clear next perf target.diff --git a/demo/playbooks/bench-vs-es.mjs b/demo/playbooks/bench-vs-es.mjsnew file mode 100644index 000000000..47ec7b87e--- /dev/null+++ b/demo/playbooks/bench-vs-es.mjs@@ -0,0 +1,147 @@+#!/usr/bin/env node+// XERJ vs Elasticsearch head-to-head benchmark.+//+// Runs IDENTICAL workloads against two engines on localhost and reports+// end-to-end (client wall-clock) latency side by side, plus bulk ingest+// throughput. Both engines run --insecure / security-off, single node, same+// machine, same corpus — so the only variable is the engine.+//+// Usage:+// node bench-vs-es.mjs [N_DOCS] [XERJ_URL] [ES_URL]+// node bench-vs-es.mjs 100000 http://localhost:9200 http://localhost:9201+import fs from 'node:fs';+import { execSync } from 'node:child_process';++const N = parseInt(process.argv[2] || '100000', 10);+const ENGINES = [+ { name: 'XERJ', url: process.argv[3] || 'http://localhost:9200' },+ { name: 'Elasticsearch', url: process.argv[4] || 'http://localhost:9201' },+];+const BATCH = 10000;+const CORPUS = '/home/claude/ai/xerj/demo/data/extras/chat-events.ndjson';+const SCRATCH = '/tmp/xerj';++const sh = (c) => execSync(c, { stdio: ['ignore', 'pipe', 'ignore'] }).toString();+async function req(base, method, p, body, ndjson = false) {+ const headers = { 'content-type': ndjson ? 'application/x-ndjson' : 'application/json' };+ const opts = { method, headers };+ if (body !== undefined) opts.body = ndjson ? body : JSON.stringify(body);+ const r = await fetch(base + p, opts);+ const txt = await r.text();+ let j = null; try { j = JSON.parse(txt); } catch {}+ return { status: r.status, j, txt };+}+function pct(a, p) { const s = [...a].sort((x, y) => x - y); return s[Math.min(s.length - 1, Math.floor(p / 100 * s.length))]; }+function f3(x) { return x === undefined || x === null ? '—' : x.toFixed(2); }++// Time `iters` end-to-end requests (wall clock) after `warmup`.+async function timed(base, p, body, iters = 80, warmup = 15) {+ for (let i = 0; i < warmup; i++) await req(base, 'POST', p, body);+ const lat = [];+ for (let i = 0; i < iters; i++) {+ const t0 = performance.now();+ await req(base, 'POST', p, body);+ lat.push(performance.now() - t0);+ }+ return { mean: lat.reduce((a, b) => a + b, 0) / lat.length, p50: pct(lat, 50), p95: pct(lat, 95), p99: pct(lat, 99), n: iters };+}++const READS = [+ ['match_all (size 10)', { query: { match_all: {} }, size: 10 }],+ ['term filter', { query: { term: { model: 'claude-haiku-4-5' } }, size: 10 }],+ ['bool must+filter', { query: { bool: { must: [{ match: { status: 'ok' } }], filter: [{ range: { latency_ms: { gte: 200 } } }] } }, size: 10 }],+ ['range', { query: { range: { cost_usd: { gte: 0.01 } } }, size: 10 }],+ ['agg: terms(model)', { size: 0, aggs: { m: { terms: { field: 'model', size: 10 } } } }],+ ['agg: stats(latency_ms)', { size: 0, aggs: { s: { stats: { field: 'latency_ms' } } } }],+ ['agg: date_histogram(day)', { size: 0, aggs: { t: { date_histogram: { field: '@timestamp', calendar_interval: 'day' } } } }],+ ['agg: terms+avg(cost)', { size: 0, aggs: { m: { terms: { field: 'model', size: 10 }, aggs: { c: { avg: { field: 'cost_usd' } } } } } }],+ ['agg: cardinality(top_doc)', { size: 0, aggs: { u: { cardinality: { field: 'top_doc' } } } }],+ ['_count match_all', null], // handled specially via _count+];++const MAPPING = {+ mappings: { properties: {+ '@timestamp': { type: 'date' },+ model: { type: 'keyword' }, intent: { type: 'keyword' }, status: { type: 'keyword' },+ tenant: { type: 'keyword' }, top_doc: { type: 'keyword' }, cache_hit: { type: 'boolean' },+ prompt_tokens: { type: 'integer' }, context_tokens: { type: 'integer' }, completion_tokens: { type: 'integer' },+ latency_ms: { type: 'integer' }, cost_usd: { type: 'double' },+ } },+};++async function ingest(base) {+ await req(base, 'DELETE', '/perf');+ await req(base, 'PUT', '/perf', MAPPING);+ const raw = fs.readFileSync(CORPUS, 'utf8').trim().split('\n');+ let written = 0; const t0 = performance.now();+ while (written < N) {+ let nd = '';+ for (let i = 0; i < BATCH && written < N; i++, written++) nd += '{"index":{}}\n' + raw[written % raw.length] + '\n';+ const f = `${SCRATCH}/_perf_batch.ndjson`; fs.writeFileSync(f, nd);+ sh(`curl -s -XPOST ${base}/perf/_bulk -H 'content-type: application/x-ndjson' --data-binary @${f} >/dev/null`);+ }+ const ms = performance.now() - t0;+ await req(base, 'POST', '/perf/_refresh');+ const c = await req(base, 'GET', '/perf/_count');+ return { ms, docs: c.j?.count ?? written, dps: Math.round(written / (ms / 1000)) };+}++async function ingestVec(base) {+ await req(base, 'DELETE', '/perfvec');+ await req(base, 'PUT', '/perfvec', { mappings: { properties: { v: { type: 'dense_vector', dims: 16, index: true, similarity: 'cosine' } } } });+ const D = 16, M = 20000; let nd = '';+ for (let i = 0; i < M; i++) {+ const v = Array.from({ length: D }, (_, k) => Math.sin(i * 0.13 + k));+ nd += '{"index":{}}\n' + JSON.stringify({ v }) + '\n';+ if (nd.length > 4e6) { const f = `${SCRATCH}/_vec.ndjson`; fs.writeFileSync(f, nd); sh(`curl -s -XPOST ${base}/perfvec/_bulk -H 'content-type: application/x-ndjson' --data-binary @${f} >/dev/null`); nd = ''; }+ }+ if (nd) { const f = `${SCRATCH}/_vec.ndjson`; fs.writeFileSync(f, nd); sh(`curl -s -XPOST ${base}/perfvec/_bulk -H 'content-type: application/x-ndjson' --data-binary @${f} >/dev/null`); }+ await req(base, 'POST', '/perfvec/_refresh');+ return Array.from({ length: D }, (_, k) => Math.sin(0.07 + k));+}++async function main() {+ const out = {};+ for (const e of ENGINES) {+ const v = await req(e.url, 'GET', '/');+ process.stderr.write(`\n== ${e.name} (${e.url}) — version ${v.j?.version?.number || '?'} ==\n`);+ process.stderr.write(` ingesting ${N.toLocaleString()} docs...\n`);+ const ing = await ingest(e.url);+ process.stderr.write(` ingest: ${ing.docs.toLocaleString()} docs in ${(ing.ms / 1000).toFixed(1)}s = ${ing.dps.toLocaleString()} docs/s\n`);+ const reads = {};+ for (const [name, body] of READS) {+ if (body === null) { reads[name] = await timed(e.url, '/perf/_count', { query: { match_all: {} } }); }+ else { reads[name] = await timed(e.url, '/perf/_search', body); }+ process.stderr.write(` ${name}: p50 ${f3(reads[name].p50)}ms\n`);+ }+ const qv = await ingestVec(e.url);+ const knn = await timed(e.url, '/perfvec/_search', { knn: { field: 'v', query_vector: qv, k: 10, num_candidates: 100 }, size: 10 });+ process.stderr.write(` kNN k=10: p50 ${f3(knn.p50)}ms\n`);+ out[e.name] = { ing, reads, knn };+ }++ // ── Report ──+ const [A, B] = ENGINES.map((e) => e.name);+ let md = `# XERJ vs Elasticsearch — head-to-head benchmark\n\n`;+ md += `Identical workload, same machine (localhost), single node, security off. `;+ md += `End-to-end client wall-clock latency. Corpus: real LLM-telemetry, **${out[A].ing.docs.toLocaleString()} docs**.\n\n`;+ md += `## Ingest (bulk, ${BATCH.toLocaleString()}/batch)\n\n`;+ md += `| Engine | docs | wall | throughput |\n|---|--:|--:|--:|\n`;+ for (const e of ENGINES) md += `| ${e.name} | ${out[e.name].ing.docs.toLocaleString()} | ${(out[e.name].ing.ms / 1000).toFixed(1)}s | **${out[e.name].ing.dps.toLocaleString()} docs/s** |\n`;+ const speedup = (out[A].ing.dps / out[B].ing.dps);+ md += `\n_${A} ingest is **${speedup.toFixed(2)}×** ${speedup >= 1 ? 'faster' : 'slower'} than ${B}._\n\n`;+ md += `## Read-path latency (ms, end-to-end, ${out[A].reads['range'].n} iters after warmup)\n\n`;+ md += `| operation | ${A} p50 | ${B} p50 | ${A} p95 | ${B} p95 | ${A} p99 | ${B} p99 | p50 ratio |\n|---|--:|--:|--:|--:|--:|--:|--:|\n`;+ const rows = [...READS.map(([n]) => n), 'kNN k=10 (20k×16d)'];+ for (const name of rows) {+ const ra = name.startsWith('kNN') ? out[A].knn : out[A].reads[name];+ const rb = name.startsWith('kNN') ? out[B].knn : out[B].reads[name];+ const ratio = ra && rb && rb.p50 ? (rb.p50 / ra.p50) : null;+ md += `| ${name} | ${f3(ra?.p50)} | ${f3(rb?.p50)} | ${f3(ra?.p95)} | ${f3(rb?.p95)} | ${f3(ra?.p99)} | ${f3(rb?.p99)} | ${ratio ? ratio.toFixed(2) + '×' : '—'} |\n`;+ }+ md += `\n_p50 ratio = ${B} p50 ÷ ${A} p50; >1 means ${A} is faster._\n`;+ fs.writeFileSync('/home/claude/ai/xerj/demo/playbooks/BENCHMARK_VS_ES.md', md);+ process.stderr.write('\nwrote demo/playbooks/BENCHMARK_VS_ES.md\n');+ console.log(md);+}+main().catch((e) => { console.error(e); process.exit(1); });