AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
benchmarks/index-size: measure the force-merged on-disk footprint per component
xerj-org9a5570091fa4
diff --git a/benchmarks/index-size/README.md b/benchmarks/index-size/README.mdnew file mode 100644index 000000000..e6bac97ca--- /dev/null+++ b/benchmarks/index-size/README.md@@ -0,0 +1,88 @@+# index-size — on-disk footprint of a force-merged XERJ index++Measures where the durable bytes are, so encoding changes (bitpacking,+deltas, dictionary reuse, zstd dictionaries) land with before/after+evidence instead of estimates. Design and staged plan:+[DESIGN.md](./DESIGN.md). Epic: issue #1038.++## Run it++```sh+cargo build --release -p xerj-server # scoped build, per repo rules+benchmarks/index-size/run.sh engine/target/release/xerj /data/sizebench 9530 100000+```++`<work-dir>` must be on a real disk (the script refuses tmpfs — du figures+would be RAM figures). It boots a throwaway node on the given port block+with its own data dir, the default lexical embedder and auth on, indexes+100k docs, force-merges to one segment, waits for du-stable-30s, and+writes `result-<LABEL>-<LEVEL>.json` plus a printed table:++```text+== index-size base (level=balanced, 100000 docs, forcemerge 1) ==+durable (excl .wal): 10,415,477 B (.wal 552,154 B, excluded)+ .post 5,570,604 B 53.5 %+ .seg 2,072,355 B 19.9 %+ .dv 1,819,138 B 17.5 %+ ...+```++(the numbers above are the 2026-07-09 measured split, shown as shape —+see the comparability caveat).++`LEVEL=best` re-runs with `[compression] level = "best"` in the node+config — the merge-path encoder knob, honoured at merge only, never at+flush (the flush path is pinned to zstd 3 after the ingest regression;+see the test `compression_level_reaches_the_merge_encoder_but_never_the_+flush_path`). That is the level ceiling any format-level change must beat+or coexist with.++## Corpus++`demo/data/extras/chat-events.ndjson` (4,008 events, seed-42+regenerable), cycled to `DOCS`, each doc enriched with a deterministic+multi-sentence `body` (in-script pool, see run.sh) and a unique `doc_id`.+Mapping is explicit: 6 keyword (incl. `doc_id`), 1 text, 4 long++ 1 double, 1 date, 1 boolean — the same shape as the 2026-07-09+diagnosis corpus.++## Baseline (measured, cited — do not re-derive)++From [demo/playbooks/DISK_SIZE_2026-07-09.md](../../demo/playbooks/DISK_SIZE_2026-07-09.md),+100k docs force-merged to one segment:++| zstd level | total | .seg | .dv | .post | .meta |+|---|---:|---:|---:|---:|---:|+| 3 (flush default) | 10,415,477 | 2,072,355 | 1,819,138 | 5,570,604 | 229,560 |+| 9 | 9,789,136 | 1,726,778 | 1,733,544 | 5,406,891 | 198,112 |+| 12 | 9,737,700 | 1,725,519 | 1,698,279 | 5,394,763 | 195,328 |+| 19 | 9,022,248 | 1,561,830 | 1,436,158 | 5,119,175 | 181,281 |++`body.post` alone was 3,928,612 B (37.7 % of the total at level 3).++**Comparability caveat:** that run's body-text generator was not+committed, so this harness's absolute totals are NOT comparable with the+table above; the component structure is. For a before/after on any+change, run THIS harness on the base commit and on the change, same+`DOCS`, same `LEVEL`, and commit both `result-*.json` files here under+`results/` — the mbox-ingest before/after convention.++## CI++`demo/playbooks/ci-check.sh` runs a 20k-doc pass informationally on every+PR (same job as the smoke suite — no new CI job) and prints the+per-extension table into the log. CI numbers use `[profile.ci-test]`+builds, which must never produce a published number+(`engine/Cargo.toml` profile comment); they gate regressions and back PR+evidence only. The public disk comparison vs Elasticsearch stays on the+manual `bench-matrix.mjs` → `SCORECARD.md` path, which needs a real ES+node and a release-profile binary.++## What is deliberately NOT measured here++- **WAL bytes** — reclamation is async and dominates whole-dir variance;+ every figure here is durable-excluding-`.wal` (same convention as the+ in-tree integration harness).+- **The public 1.61× vs ES cell** — its basis is an un-force-merged+ ~2.38M-doc index after the mixed-writer benchmark; a force-merged 100k+ CI number must never drift into that cell's context.diff --git a/benchmarks/index-size/run.sh b/benchmarks/index-size/run.shnew file mode 100755index 000000000..88368aea3--- /dev/null+++ b/benchmarks/index-size/run.sh@@ -0,0 +1,204 @@+#!/usr/bin/env bash+# Measure the on-disk size of a force-merged XERJ index, per component.+#+# benchmarks/index-size/run.sh <xerj-binary> <work-dir> <es-port> [docs]+# e.g. run.sh engine/target/release/xerj /data/sizebench 9530 100000+#+# LABEL=base suffix of the result file (default "base")+# LEVEL=best [compression] level in the node config: fast |+# balanced | best. The knob is honoured at MERGE only+# (xerj-engine/src/index.rs compression_level_reaches_+# the_merge_encoder_but_never_the_flush_path), so this+# is how you measure the merge-level ceiling.+# DOCS=100000 docs to index (cycles the corpus, default 100000)+#+# Protocol (clones demo/playbooks/DISK_SIZE_2026-07-09.md):+# 1. cycle demo/data/extras/chat-events.ndjson, enriching every doc with a+# deterministic analyzed `body` text and a unique `doc_id` keyword —+# the same corpus shape (6 keyword incl. doc_id, 1 text, 5 numeric,+# 1 date, 1 boolean) as the 2026-07-09 run, generated in-script so the+# harness is self-contained and reproducible;+# 2. explicit mapping, bulk, refresh;+# 3. _forcemerge?max_num_segments=1 + _flush (steady state, one segment);+# 4. wait for du-stable-30s;+# 5. sum bytes per file extension over the data dir EXCLUDING .wal (WAL+# reclamation is async and would swamp the signal), plus a per-field+# split of .post/.meta/.fst/.norms and the totals;+# 6. write <work-dir>/result-<LABEL>.json and print the table.+#+# The body text here is NOT the same text as the 2026-07-09 run (that run's+# generator was not committed), so absolute totals are not comparable with+# that table; the per-component structure is. Compare like with like: run+# this harness on the base commit and on the change, same DOCS.+#+# It boots a THROWAWAY node on <es-port> (+1 rest, +2 grpc) with its own data+# directory, the default LEXICAL embedder and AUTH ON. Nothing here talks to+# any other node: before the first write it checks that the listener on+# <es-port> is the process it just started.+set -euo pipefail+XERJ=$(readlink -f "$1"); WORK=$(readlink -f "$2"); PORT=$3+DOCS=${4:-${DOCS:-100000}}+HERE=$(cd "$(dirname "$0")" && pwd); REPO=$(cd "$HERE/../.." && pwd)+LABEL=${LABEL:-base}; LEVEL=${LEVEL:-balanced}+CORPUS="$REPO/demo/data/extras/chat-events.ndjson"+DATA="$WORK/node-$LABEL-$LEVEL"; OUT="$WORK/result-$LABEL-$LEVEL.json"+URL="http://127.0.0.1:$PORT"+IDX="disksize"++case "$(findmnt -no FSTYPE -T "$WORK")" in tmpfs|ramfs) echo "refusing: $WORK is RAM-backed"; exit 2;; esac+if ss -ltn | grep -qE ":($PORT|$((PORT+1))|$((PORT+2)))\b"; then echo "refusing: port block $PORT.. is in use"; exit 2; fi+[ -x "$XERJ" ] || { echo "refusing: $XERJ is not an executable xerj binary"; exit 2; }++rm -rf "$DATA"; mkdir -p "$DATA"+cat > "$WORK/node-$LABEL-$LEVEL.toml" <<TOML+[server]+es_compat_port = $PORT+rest_port = $((PORT+1))+grpc_port = $((PORT+2))+data_dir = "$DATA"+[tls]+enabled = false+[embedding]+mode = "lexical"+[compression]+level = "$LEVEL"+TOML+nohup "$XERJ" -c "$WORK/node-$LABEL-$LEVEL.toml" --embed-mode lexical > "$WORK/server-$LABEL-$LEVEL.log" 2>&1 &+SERVER=$!+trap 'kill $SERVER 2>/dev/null || true' EXIT+KEY=""+for _ in $(seq 1 120); do+ [ -s "$DATA/admin.key" ] && KEY=$(cat "$DATA/admin.key")+ [ -n "$KEY" ] && curl -fsS -H "Authorization: ApiKey $KEY" "$URL/_cluster/health" >/dev/null 2>&1 && break+ sleep 0.5+done+curl -fsS -H "Authorization: ApiKey $KEY" "$URL/_cluster/health" >/dev/null || { echo "node did not come up"; tail -5 "$WORK/server-$LABEL-$LEVEL.log"; exit 1; }+# Never write into somebody else's node.+LISTENER=$(ss -ltnp 2>/dev/null | grep -E ":$PORT\b" | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)+[ "$LISTENER" = "$SERVER" ] || { echo "refusing: :$PORT is held by pid ${LISTENER:-?}, not the node this script started ($SERVER)"; exit 2; }+AUTH="Authorization: ApiKey $KEY"++# ── 1. corpus: cycle chat-events, enrich body + doc_id ──────────────────+# Deterministic: doc i takes event (i % events); body is a multi-sentence+# text built from the event's own strings plus a 12-sentence pool indexed+# by (i // events) and (i * 5) so repeated cycles differ; doc_id is+# chat-events-<i>.+python3 - "$CORPUS" "$WORK/bulk-$LABEL.ndjson" "$DOCS" "$IDX" <<'PY'+import json, sys+src, out, want, idx = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4]+events = [json.loads(l) for l in open(src) if l.strip()]+POOL = [+ "the assistant reviewed the change and flagged the missing rollback step",+ "a tenant reported slow queries on the dashboard after the deploy",+ "the nightly job retried three times before the cache warmed up",+ "an operator paused ingestion to drain the queue during maintenance",+ "the model cached the prefix and skipped the recomputation entirely",+ "review asked for a test covering the empty pagination edge case",+ "the index force-merged overnight and search latency dropped",+ "a watcher fired on the error budget but nobody was on call",+ "the runbook walkthrough found two stale commands in section three",+ "traces showed the fan-out waiting on one cold shard every time",+ "the schema migration backfilled dates in batches of ten thousand",+ "an alert about disk growth turned out to be an unforced merge",+]+with open(out, "w") as f:+ for i in range(want):+ e = events[i % len(events)]+ turn = (i // len(events)) % len(POOL)+ body = (+ f"{POOL[turn]}. the {e['intent']} request on {e['top_doc']} used "+ f"{e['model']} with {e['context_tokens']} tokens of context and "+ f"{POOL[(i + 7) % len(POOL)]}. "+ f"{POOL[(i * 5 + turn) % len(POOL)]}"+ )+ doc = dict(e)+ doc["body"] = body+ doc["doc_id"] = f"chat-events-{i}"+ f.write(json.dumps({"index": {"_index": idx, "_id": doc["doc_id"]}}) + "\n")+ f.write(json.dumps(doc) + "\n")+PY++# ── 2. explicit mapping, bulk, refresh ───────────────────────────────────+curl -fsS -XPUT "$URL/$IDX" -H "$AUTH" -H 'content-type: application/json' -d '{+ "settings": { "index": { "number_of_shards": 1 } },+ "mappings": { "properties": {+ "@timestamp": { "type": "date" },+ "model": { "type": "keyword" },+ "intent": { "type": "keyword" },+ "top_doc": { "type": "keyword" },+ "tenant": { "type": "keyword" },+ "status": { "type": "keyword" },+ "doc_id": { "type": "keyword" },+ "body": { "type": "text" },+ "prompt_tokens": { "type": "long" },+ "context_tokens": { "type": "long" },+ "completion_tokens": { "type": "long" },+ "latency_ms": { "type": "long" },+ "cost_usd": { "type": "double" },+ "cache_hit": { "type": "boolean" }+ } } }' >/dev/null+BULK="$WORK/bulk-$LABEL.ndjson"+# ~25 MB at 100k docs; chunked to keep peak memory bounded on small runners.+split -l 20000 -d -a 3 "$BULK" "$BULK.part-"+for part in "$BULK".part-*; do+ curl -fsS -XPOST "$URL/_bulk" -H "$AUTH" -H 'content-type: application/x-ndjson' \+ --data-binary @"$part" >/dev/null+done+rm -f "$BULK".part-*+curl -fsS -XPOST "$URL/$IDX/_refresh" -H "$AUTH" >/dev/null+N=$(curl -fsS "$URL/$IDX/_count" -H "$AUTH" | python3 -c 'import json,sys; print(json.load(sys.stdin)["count"])')+[ "$N" = "$DOCS" ] || { echo "refusing: indexed $N of $DOCS docs"; exit 1; }++# ── 3. forcemerge 1 + flush (steady state) ───────────────────────────────+curl -fsS -XPOST "$URL/$IDX/_forcemerge?max_num_segments=1" -H "$AUTH" >/dev/null+curl -fsS -XPOST "$URL/$IDX/_flush" -H "$AUTH" >/dev/null || true++# ── 4. settle: du stable for 30 s ────────────────────────────────────────+prev=-1; stable=0+for _ in $(seq 1 120); do+ cur=$(du -sb "$DATA" | cut -f1)+ if [ "$cur" = "$prev" ]; then stable=$((stable+1)); else stable=0; fi+ [ "$stable" -ge 6 ] && break; prev=$cur; sleep 5+done++# ── 5. breakdown: bytes per extension, .post split per field ─────────────+python3 - "$DATA" "$OUT" "$N" "$LABEL" "$LEVEL" <<'PY'+import json, os, sys+data, out, ndocs, label, level = sys.argv[1:6]+by_ext, by_field, wal, other = {}, {}, 0, {}+for root, _, files in os.walk(data):+ for f in files:+ p = os.path.join(root, f)+ n = os.path.getsize(p)+ if f.endswith(".wal"):+ wal += n+ continue+ # <seg>.<field>.post / .meta / .fst / .norms carry a field name;+ # attribute those to "<field>.<ext>" as well as the extension total.+ stem_ext = ""+ parts = f.split(".")+ if len(parts) >= 3 and parts[-1] in ("post", "meta", "fst", "norms"):+ stem_ext = parts[-2] + "." + parts[-1]+ ext = parts[-1] if len(parts) > 1 else "(none)"+ by_ext[ext] = by_ext.get(ext, 0) + n+ if stem_ext:+ by_field[stem_ext] = by_field.get(stem_ext, 0) + n+durable = sum(by_ext.values())+res = {+ "label": label, "compression_level": level, "docs": int(ndocs),+ "durable_bytes_excluding_wal": durable, "wal_bytes": wal,+ "by_extension": dict(sorted(by_ext.items(), key=lambda kv: -kv[1])),+ "by_field_post_meta": dict(sorted(by_field.items(), key=lambda kv: -kv[1])),+}+json.dump(res, open(out, "w"), indent=1)+print(f"== index-size {label} (level={level}, {ndocs} docs, forcemerge 1) ==")+print(f"durable (excl .wal): {durable:>12,} B (.wal {wal:,} B, excluded)")+for ext, n in res["by_extension"].items():+ print(f" .{ext:<12} {n:>12,} B {100.0 * n / durable:5.1f} %")+top = list(res["by_field_post_meta"].items())[:8]+if top:+ print(" per-field families (top 8):")+ for name, n in top:+ print(f" {name:<22} {n:>12,} B")+print(f"result: {out}")+PYdiff --git a/demo/playbooks/ci-check.sh b/demo/playbooks/ci-check.shindex 1b4698663..6cce952a4 100755--- a/demo/playbooks/ci-check.sh+++ b/demo/playbooks/ci-check.sh@@ -3,6 +3,7 @@# 1. smoke suite (must be all-green / exit 0)# 2. API liveness (no 5xx across the read surface)# 3. benchmark (informational — prints throughput + latency)+# 4. index size (informational — force-merged per-extension footprint)## Expects a built binary. Set XERJ_BIN to override (default: engine release build).# No model / LLM needed — safe to run in GitHub Actions.@@ -44,4 +45,15 @@ node "$HERE/liveness.mjs"echo "== 3. benchmark (informational) =="node "$HERE/bench.mjs" "${BENCH_DOCS:-100000}" || echo "(benchmark non-fatal)"+echo "== 4. index size (informational) =="+# Force-merged 20k-doc pass with the DISK_SIZE_2026-07-09 corpus shape and a+# per-extension breakdown, so encoding changes land with before/after evidence+# in the PR logs. ci-test-profile numbers gate regressions only — they are+# never published cells (see benchmarks/index-size/README.md). Non-fatal for+# the same reason the benchmark is; a tmpfs WORK is refused by run.sh itself.+SIZE_WORK="$(mktemp -d)"+bash "$REPO/benchmarks/index-size/run.sh" \+ "$XERJ_BIN" "$SIZE_WORK" 9530 "${SIZE_DOCS:-20000}" || echo "(index-size non-fatal)"+rm -rf "$SIZE_WORK" 2>/dev/null || true+echo "== CI checks passed =="