AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
test(perf): doc-values skip measured — 1.553x smaller disk, 2.33x faster merge
xerj-orgae71140f3863
diff --git a/demo/playbooks/rc12/RESULT-doc-values-skip.md b/demo/playbooks/rc12/RESULT-doc-values-skip.mdnew file mode 100644index 000000000..123d4ca35--- /dev/null+++ b/demo/playbooks/rc12/RESULT-doc-values-skip.md@@ -0,0 +1,96 @@+# Measured: honouring `doc_values` on the mapping++A/B on the same seeded 500k-doc corpus, force-merged to one segment,+`XERJ_DISABLE_QUERY_CACHE=1`.++- before: `results-baseline-nocache.json` (`main` @ 8a6fa38)+- after: `results-dvskip.json` (branch `perf/fts-reader-cache`)++The mapping declares one analyzed `text` field (`body`) and no explicit+`doc_values`, so after the change `body` gets no doc-values column — matching+Elasticsearch's default.++---++## Disk — 1.553x smaller, measured end-to-end++| | baseline | dvskip | change |+|---|---:|---:|---:|+| index bytes | 154,413,167 | **99,424,772** | **−35.6% = 1.553x smaller** |+| index / raw | 0.433x | **0.279x** | |++The prediction from parsing the `DV01` envelope was **99,424,789 B**. The+measured result is **99,424,772 B** — within 17 bytes. The attribution method+in `BASELINE.md` is sound and can be trusted for sizing future levers.++### The change is surgical++| artifact | baseline | dvskip |+|---|---:|---:|+| **`.dv`** | 57,357,593 | **2,369,197** (−95.9%) |+| `.seg` | 51,449,817 | 51,449,817 |+| `.post` | 40,951,589 | 40,951,589 |+| `.ids` | 3,057,388 | 3,057,388 |+| `.meta` | 1,464,327 | 1,464,327 |+| `.fst` | 64,485 | 64,485 |+| `.norms` | 28,305 | 28,305 |++Every artifact other than `.dv` is **byte-identical**. The residual 2.37 MB of+doc-values is the keyword and numeric fields, which still get columns because+their mapping still asks for them — exactly as intended.++---++## Write path — an unpredicted 2.33x on force-merge++| | baseline | dvskip | change |+|---|---:|---:|---:|+| **force-merge to 1 segment** | 139.13 s | **59.69 s** | **2.33x faster** |+| ingest | 31.86 s | 28.22 s | 1.13x faster |++This was not predicted and is the largest write-path win of the campaign. The+cause is direct: merge re-encodes the doc-values side-car, and there are now+55 MB fewer bytes to build, sort and zstd. Flush gets a smaller version of the+same benefit.++It is worth noting *why* this matters beyond the number. The corrected baseline+identified force-merge (139 s) as one of the few large, genuinely measurable+performance costs in the engine. A change made purely for footprint reasons+turned out to attack it directly — and it does so by **not doing work**, which is+strictly better than doing the same work faster.++It also removes the tension that made merge-time zstd tiering risky: that lever+raises merge CPU, and merge just got 2.33x cheaper, so the two are far more+compatible than the audits assumed.++---++## Correctness++ES-YAML conformance with this change **and** the FTS reader cache in:++```+1365 passed · 0 failed · 3 skipped · 1368 total+```++See `GATE-conformance.md`. The cases that could have caught a wrong+implementation — `aggregations/terms_text_docvalues.yml`, which requires an+explicit `"doc_values": true` on a text field to keep working — pass.++---++## Scope and honesty++**This corpus sits at the high end and the number will not transfer unchanged.**+`body` is ~400 B of a ~713 B document, so doc-values are 37.1% of the index here.+Measured elsewhere, `.dv` is only **6.97%** across 4,245 MiB of `xc-*` code+corpora (1.71%–11.88% per index), because `_source` dominates there at 64.21%.++The defensible public statement is therefore:++> Removes 90–99% of the doc-values side-car. That side-car is 2–37% of index+> size depending on how large your text bodies are relative to `_source` —+> 35.6% on our benchmark corpus.++**Never quote 1.553x as universal**, and never add it to the merge-tier zstd+saving: with `.dv` gone, the zstd lever has a much smaller denominator to work on.diff --git a/demo/playbooks/rc12/results-dvskip.json b/demo/playbooks/rc12/results-dvskip.jsonnew file mode 100644index 000000000..e576df9dc--- /dev/null+++ b/demo/playbooks/rc12/results-dvskip.json@@ -0,0 +1,24 @@+{+ "label": "dvskip",+ "docs": 500000,+ "raw_source_bytes": 356593195,+ "index_total_bytes": 99424772,+ "index_over_raw": 0.2788,+ "ingest_seconds": 28.22,+ "forcemerge_seconds": 59.69,+ "by_artifact": {+ "seg": 51449817,+ "post": 40951589,+ "ids": 3057388,+ "dv": 2369197,+ "meta": 1464327,+ "fst": 64485,+ "json": 35339,+ "norms": 28305,+ "wal": 4117,+ "toml": 104,+ "xerj_master_key": 64,+ "sidx": 32,+ "lock": 8+ }+}\ No newline at end of file