AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Merge main (ZBS4 #1042) into perf/meta-slim-2026-09-26
xerj-org870fe72a0b57
diff --cc CHANGELOG.mdindex 629d36c0b,6ef4e31ce..a323fa9ca--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -88,31 -88,52 +88,77 @@@ and this project adheres to [Semantic V### Performance+- **`.meta` term records go columnar-varint (ZFM5)** ([#1038](https://github.com/xerj-org/xerj/issues/1038)):+ the per-term `{df u32, ttf u64, offset u64, len u32}` records under the+ ZFM4 zstd envelope kept a fixed 24-byte stride, and zstd cannot match+ across the stride's alternating field widths — on the 100 k-doc size+ harness `doc_id.meta` (100 k unique terms, every one `ttf == df`, every+ postings gap zero) still cost 138,727 B on disk for ~200 B of+ information. ZFM5 re-shapes the section before compression into four+ columnar varint streams — `df`, `ttf − df` (elided entirely when every+ term in the field has `ttf == df`, the keyword-field shape), `offset+ gap`, and `postings_length` — so zstd sees long runs of one-symbol+ lanes. The reader expands the streams once at open time into the same+ in-memory flat-record array ZFM3/ZFM4 build, leaving the FST-offset+ lookup hot path untouched. The encoder fails closed on non-monotone+ postings offsets (gaps reconstruct by accumulation, so a negative gap is+ unrepresentable) and the decoder rejects truncated streams, trailing+ stream bytes, and length mismatches. Peer pattern: tantivy's+ `TermInfoValueWriter::serialize_block` (`src/termdict/sstable_termdict/+ mod.rs:92-105`, MIT) writes block-leading absolute range starts once+ and then per-term length varints only. Measured on the 100 k-doc+ harness (LEVEL=balanced, forcemerge 1, same-day control): the `.meta`+ family drops 200,826 → 13,774 B (−93.1 %), total durable −3.5 %, with+ every other extension byte-identical; the win scales+ with term cardinality, which is exactly the shape of unique-id fields+ on large deployments. ZFM4-and-older segments decode unchanged.++ - **Stored sections get typed-int columns, duplicate-column references and+ a merge-path zstd effort chooser (`ZBS4`)** — stage 1 of the index-size+ epic [#1038](https://github.com/xerj-org/xerj/issues/1038), redesigned+ by measurement: dissecting a harness `.seg` showed the planned+ dict-stream work targeted ~5 % of the section while the `body` RAW_JSON+ fallback alone was 84.2 %, `__seq_no` was stored as textual consecutive+ integers, and `doc_id` was a byte-identical duplicate of `__id`. Three+ codecs instead (`engine/crates/xerj-storage/src/stored_codec.rs`):+ **TYPED_INT** stores an all-integer column as a presence bitmap plus+ 128-value frame-of-reference bit-packed blocks (`[width][zigzag-varint+ min]`, width 0 = constant block — the ZPS2 `.post` shape) with a+ zigzag-varint delta residual; blocks whose spread exceeds u32 lanes keep+ the JSON fallback, floats/strings never qualify. A dict-eligible+ integer column races the two codecs and keeps the smaller (the Parquet+ pick-the-smallest rule the `.dv` ZNV2 chooser uses). **COPY_OF** stores+ a later column byte-identical to an earlier pass-1 column as a 4-byte+ reference (digest + full-equality verify before writing), so an id echo+ in `_source` costs 4 B instead of a second full payload. **The+ long-effort chooser** — merge path only, flush stays pinned to zstd 3+ after the ingest regression — additionally compresses each JSON fallback+ column at level 19 with long-distance matching and an explicit ≤ 16 MB+ window and keeps whichever payload is smaller. Two measured reasons it+ is a chooser and not a level bump: the window knee is sharp (body 29.8 MB+ column: 679 kB at the default window, 330 kB at 16 MB — the sentence+ pool repeats just past a small window), and high levels REGRESS+ small-alphabet streams (the id column: 54 kB at zstd-3, 143 kB at 19) —+ a global 19 would grow the ids while shrinking body. Offline A/B and+ the container-walk dissection are recorded in+ `benchmarks/index-size/DESIGN.md` (new — it fixes the dangling link the+ #1039 README and this epic's anchor list pointed at). Compatibility:+ the `ZBS4` magic is written only when a column actually uses codec 5/6+ or a form-1 RAW payload; anything expressible as ZBS2 stays+ byte-identical ZBS2 (flush parity and downgrade blast radius unchanged,+ `XERJ_FLUSH_PARITY=1` verified on a full 100k harness run), ZBS3+ present-null wrapping composes, the estimator counts COPY_OF as its+ source's bound and fails closed on crafted forward references, and the+ full decoder's fixpoint resolves reference chains while hard-erroring on+ cycles. Measured on the 100k `benchmarks/index-size` harness at+ LEVEL=balanced against a same-day main control (`results/result-zbs4*-balanced.json`):+ `.seg` **−73.1 %** (1,671,271 → 449,094 B), total durable **−22.7 %**+ (5,398,478 → 4,176,300 B); every non-`.seg` extension byte-identical+ between the runs. ES-YAML conformance 1380 passed / 0 failed; 10 new+ storage unit tests cover typed-int round-trips (nulls, negatives,+ i64-extreme clusters), copy-of decode across full/projection/hydration+ paths, form-1 decode on all four read paths, and the malformed-payload+ rejections.- **`.post` block framing slimmed (ZPS2)** ([#1038](https://github.com/xerj-org/xerj/issues/1038)):the packed doc-id-delta and term-freq streams in every 128-doc postingblock switch to frame-of-reference coding — `[width][vbyte min]` with the