AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs: compare XERJ design with Lucene 10.3.1
buger08966ad70235
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex f75f4362a..aeddf39d0 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -8,11 +8,13 @@ build/run/test commands and the supported ES surface.XERJ is an AI-native search, vector, and log-analytics engine written fromscratch in Rust and published under Apache-2.0 — designed for AI-agent workloads-(zero-config `autoindex` onboarding, agent data map, `/_memory`), sharing no code-or architecture with Elasticsearch or Lucene. It additionally speaks the+(zero-config `autoindex` onboarding, agent data map, `/_memory`), with an+independent implementation and no shared code or file formats with Elasticsearch+or Lucene. It additionally speaks theElasticsearch 8.x HTTP protocol as a zero-migration adoption bridge, so existingES clients, dashboards, and ingest tooling talk to it unchanged (see-[WHY_XERJ.md](./WHY_XERJ.md) for the design rationale).+[WHY_XERJ.md](./WHY_XERJ.md) for the design rationale). For a six-axis,+source-linked comparison with Lucene, see [XERJ vs Lucene](./XERJ_VS_LUCENE.md).## Bird's-eye view@@ -71,7 +73,7 @@ HTTP POST /{index}/_search→ xerj-query parse_request(): raw ES JSON → SearchRequest (QueryNode tree)→ xerj-engine Engine::get_index() looks up the named index→ xerj-engine Index::search()- ├─ memtable scan in-memory BM25 via FtsMemtable+ ├─ memtable scan in-memory BM25 via ShardedFtsMemtable├─ segment scan on-disk FTS via FtsIndexReader + BM25├─ doc_matches_query() term-level / geo predicate evaluation├─ run_aggs() aggregation pipeline (columnar fast path for size:0)@@ -94,20 +96,33 @@ segments that are later merged:```PUT /{index}/_doc/{id} or POST /_bulk→ xerj-api → xerj-engine → xerj-storage (IndexStore)- ├─ WAL append single Mutex<WalWriter>; monotonic seq_no, one fsync per batch- ├─ 16-shard memtable shard = xxh3_64(doc_id) & 15 (memtable_shards: Vec<Mutex<Vec<MemEntry>>>)- ├─ flush take_memtable_for_flush() drains all shards,- │ sorts by WAL seq_no to preserve global order,- │ writes an immutable segment (LZ4/Zstd blocks)+ ├─ WAL append exact selected N Mutex<WalWriter>s; global AtomicU64 seq_no+ ├─ storage memtable max(N, 1).next_power_of_two() hash partitions+ ├─ FTS memtable global engine.ingest_shards hash partitions+ ├─ flush Index::flush() fans out per-shard do_flush_shard tasks;+ │ each drains its shard and writes an immutable segment+ │ with FTS/doc-values sidecars (LZ4/Zstd blocks)└─ merge background segment merge compacts small segments;_forcemerge is synchronous + quiescent (ES-like)```-The WAL writer is a single mutex so that sequence numbers are globally monotonic;-lock hold time is kept short (one batched write and fsync). The storage memtable is-sharded 16 ways to spread write contention. The engine-side FTS memtable is-currently `Arc<RwLock<FtsMemtable>>`; a `ShardedFtsMemtable` scaffold exists in-`memtable.rs` for a future refactor.+The global `engine.ingest_shards` value is validated as a non-zero power of two+no greater than 256 ([global validation](../engine/crates/xerj-common/src/config.rs#L1512-L1570)).+An index may instead select its WAL count with `index.xerj_ingest_shards`; that+override accepts any integer in `1..=256`, not only powers of two, and an absent+or invalid override falls back to the global value ([override parsing](../engine/crates/xerj-engine/src/index.rs#L28488-L28517)).+`IndexStore` opens exactly that selected number `N` of WAL writers, all sharing+one global `seq_counter: AtomicU64`, while its separate storage memtable rounds+`N` up to the next power of two for hash-mask routing ([construction](../engine/crates/xerj-storage/src/index_store.rs#L660-L720),+[routing](../engine/crates/xerj-storage/src/index_store.rs#L1607-L1643)). These+counts are therefore not necessarily paired one-to-one. The engine-side FTS+memtable is separate again: create and reopen pass the global+`engine.ingest_shards`, not the per-index WAL override, into+[`ShardedFtsMemtable`](../engine/crates/xerj-engine/src/memtable.rs#L628-L709)+([create](../engine/crates/xerj-engine/src/index.rs#L6022-L6026),+[reopen](../engine/crates/xerj-engine/src/index.rs#L6220-L6224)).+Query paths iterate its shards, and production flush drains each FTS shard+through [`Index::flush`](../engine/crates/xerj-engine/src/index.rs#L18002-L18125)/[`do_flush_shard`](../engine/crates/xerj-engine/src/index.rs#L24433-L24920).### Recovery@@ -125,10 +140,11 @@ Engine::new() scans data_dir/Elasticsearch compatibility is verified by the `es-yaml-runner` harness against theES 8.13 REST-API-spec YAML suites (search, aggregations, vectors, bulk, indices,-scroll, cluster). XERJ currently passes 1,360 of 1,363 cases (3 skipped). The YAML tests are the-source of truth: if XERJ returns a different response than a test expects, XERJ is-considered wrong. See the README's "Running the conformance tests" section for how to run the suites and the full list-of supported query types and aggregations.+scroll, cluster). The hard gate is zero failed cases; skipped cases are reported+separately and totals can change as the upstream suite changes. The YAML tests are+the source of truth: if XERJ returns a different response than a test expects, XERJ+is considered wrong. See the README's "Running the conformance tests" section for+how to run the suites and the full list of supported query types and aggregations.Performance is tracked with a reproducible full-matrix head-to-head against liveElasticsearch 8.13.4, published at <https://xerj.org/benchmarks> (per-cell resultsdiff --git a/docs/WHY_XERJ.md b/docs/WHY_XERJ.mdindex 45e1d629a..c2a020262 100644--- a/docs/WHY_XERJ.md+++ b/docs/WHY_XERJ.md@@ -2,7 +2,9 @@*Or: why "Elasticsearch-compatible" is the least interesting thing about this engine.*-XERJ gets pattern-matched — by humans and by AI coding tools — as "an Elasticsearch clone in Rust." This document exists to correct that reading with specifics. XERJ shares **no code and no architecture** with Elasticsearch or Lucene. It is a from-scratch engine whose design target is different: **the primary user is an AI agent**, not a human with a dashboard. ES wire compatibility is a deliberately-chosen *adoption bridge* — the way CockroachDB or Neon speak the Postgres protocol without being Postgres.+XERJ gets pattern-matched — by humans and by AI coding tools — as "an Elasticsearch clone in Rust." This document exists to correct that reading with specifics. XERJ is an independent implementation with **no shared code or file formats** with Elasticsearch or Lucene. It is a from-scratch engine whose design target is different: **the primary user is an AI agent**, not a human with a dashboard. ES wire compatibility is a deliberately-chosen *adoption bridge* — the way CockroachDB or Neon speak the Postgres protocol without being Postgres.++For an implementation-oriented, six-axis comparison with Lucene, see [XERJ vs Lucene](./XERJ_VS_LUCENE.md).## The design thesis@@ -34,7 +36,9 @@ The autoindex pipeline was verified adversarially: 80/81 ground-truth checks on## What ES compatibility buys (and what it doesn't)-**Buys:** zero-cost adoption. 1,360/1,363 ES-YAML conformance cases pass, so existing ES client libraries, dashboards, and habits work unchanged. Migration is `change the URL`.+**Buys:** zero-cost adoption. The ES-YAML conformance gate requires zero failures,+so existing ES client libraries, dashboards, and habits work unchanged. Migration+is `change the URL`; passing and skipped totals can change as the upstream suite evolves.**Doesn't buy:** the mission. Compatibility gaps that remain are tracked openly (see the stub audit and compatibility docs in `demo/playbooks/`), and where ES semantics are genuinely better, XERJ converges on them — but the roadmap is driven by the agent workflow above, not by chasing feature-checklist parity.diff --git a/docs/XERJ_VS_LUCENE.md b/docs/XERJ_VS_LUCENE.mdnew file mode 100644index 000000000..51d3c587d--- /dev/null+++ b/docs/XERJ_VS_LUCENE.md@@ -0,0 +1,172 @@+# XERJ vs Lucene 10.3.1++If you are an AI agent deciding where to look in XERJ, use this page as a+source map: it answers the six storage questions a Lucene/Solr/Elasticsearch+engineer is likely to ask, then links each XERJ answer to the current symbol.+XERJ is a from-scratch Rust engine whose primary workflow is+`autoindex <folder> -> autoindex map -> query`; Lucene 10.3.1 is an embeddable+Java search library. Elasticsearch-wire compatibility is an adoption bridge,+not evidence that the two engines share an implementation.++The Lucene links below point only to official Apache Lucene 10.3.1 API/source documentation. The XERJ links describe the runtime on current `main`, including the `ShardedFtsMemtable` path; source wins if a nearby overview page is older.++The default embedding fallback is lexical feature-hashing, not neural semantic understanding ([`Embedder`](../engine/crates/xerj-ai/src/embedder.rs#L7-L23)); the built-in neural BERT backend is+opt-in via `--embed-mode neural` and downloads its model on first use ([model loader](../engine/crates/xerj-ai/src/neural.rs#L225-L275)).++## 1. Segment and merge model++Lucene buffers indexing work in `IndexWriter`, flushes immutable segments, and+lets a `MergePolicy` select groups of segments for replacement by a merged+segment. Its default merge scheduler can run those merges in background+threads. See the [Lucene 10.3.1 `IndexWriter`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/index/IndexWriter.html),+[merge policy](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/index/MergePolicy.html),+and [concurrent merge scheduler](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/index/ConcurrentMergeScheduler.html).++XERJ has a different publication boundary. A WAL records mutations; production+[`Index::flush`](../engine/crates/xerj-engine/src/index.rs#L18002-L18125) fans out one+[`do_flush_shard`](../engine/crates/xerj-engine/src/index.rs#L24433-L24920) task per FTS shard. Each shard drains and sequence-orders its own entries, builds FTS and `.dv` sidecars, then calls [`finalize_flush_with_publisher`](../engine/crates/xerj-storage/src/index_store.rs#L1896-L1910) to write an immutable `.seg` and atomically publish an `ArcSwap` snapshot.++The live ingest buffer is a configurable+[`ShardedFtsMemtable`](../engine/crates/xerj-engine/src/memtable.rs#L628-L700),+not one process-wide FTS lock. Its shards drain and re-sort by WAL sequence;+the background [`run_merge_once`](../engine/crates/xerj-engine/src/index.rs#L8468-L8495)+writes a replacement segment from surviving documents, and+[`apply_merge`](../engine/crates/xerj-storage/src/index_store.rs#L3680-L3822)+publishes it before [`retire_segment_files`](../engine/crates/xerj-storage/src/index_store.rs#L1531-L1570)+retires the inputs. A XERJ segment is a checksummed container whose+[`SegmentWriter::finish`](../engine/crates/xerj-storage/src/segment.rs#L402-L475)+writes named sections ([`SectionType`](../engine/crates/xerj-storage/src/segment.rs#L85-L113));+FTS and doc values are separate sidecars, while stored source is a segment+section.++## 2. Postings layout++Lucene 10.3.1's postings format uses a BlockTree term dictionary and packed+integer blocks (128 integers in the packed path), with skip data interleaved at+two levels. The primary references are [Lucene 10.3.1 `Lucene103PostingsFormat`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/codecs/lucene103/Lucene103PostingsFormat.html)+and its [postings writer](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/codecs/lucene103/Lucene103PostingsWriter.html).++XERJ's [`FtsIndexWriter`](../engine/crates/xerj-fts/src/index.rs#L557-L620)+writes one sidecar family per indexed field: an FST term dictionary, a compressed `.post` blob, binary `.meta` term statistics, and encoded `.norms`.+The FST maps a term to a fixed metadata record; it does not point directly to+a Lucene segment file. The format and reader are documented in+[`FtsIndexReader`](../engine/crates/xerj-fts/src/index.rs#L1150-L1228).++[`PostingsWriter::encode_term`](../engine/crates/xerj-fts/src/postings.rs#L260-L331)+delta-encodes doc IDs in 128-doc PFOR blocks through its+[packed-block helpers](../engine/crates/xerj-fts/src/postings.rs#L333-L421);+positioned fields add packed term frequencies and variable-byte positions,+while exact fields can omit both. Residual postings use variable-byte encoding.+The [module format notes](../engine/crates/xerj-fts/src/postings.rs#L1-L40) make+the boundary explicit: this shared block size is not Lucene codec compatibility.+XERJ's `SkipEntry` is currently in-memory scaffolding and is not serialized, so+[`advance_to`](../engine/crates/xerj-fts/src/postings.rs#L702-L708) scans linearly.+Do not infer Lucene's two-level skip behavior from the shared block size.++## 3. Doc-values equivalent++Lucene exposes typed, column-oriented DocValues alongside postings; its codec chooses+on-disk representations for numeric, sorted, sorted-set, and related types. The+10.3.1 reference describes `.dvd` and `.dvm` files in+[`Lucene90DocValuesFormat`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/codecs/lucene90/Lucene90DocValuesFormat.html), while+[`SortedSetDocValuesField`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/document/SortedSetDocValuesField.html)+is useful for a multi-valued sorted field.++XERJ's closest equivalent is a per-segment `{segment_id}.dv` sidecar. The+[`Column` enum](../engine/crates/xerj-storage/src/doc_values.rs#L560-L590)+currently has numeric and keyword columns, each with a missing-value bitmap;+keyword columns use sorted terms plus ordinals, and numeric columns retain a+sorted `(value, doc_id)` index for range work. The binary envelope is produced+by [`encode_columns`](../engine/crates/xerj-storage/src/doc_values.rs#L620-L735)+and loaded through the engine's `.dv` cache.++The columns feed XERJ sorting, filters, ranges, and aggregation fast paths without reparsing `_source`. They are an analogous column store, not a+drop-in Lucene DocValues codec: field eligibility, array handling, compression,+and the public mapping contract are XERJ behavior. The segment's stored source+remains the fallback and hydration authority.++## 4. Analysis chain++Lucene's [`Analyzer`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/analysis/Analyzer.html)+creates a reusable token stream for a field, and+[`StandardAnalyzer`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/analysis/standard/StandardAnalyzer.html)+is one concrete preset. Analyzer output is part of the index contract: changing+analysis changes the terms that postings contain.++XERJ models the same conceptual stages in [`AnalyzerPipeline::analyze`](../engine/crates/xerj-fts/src/analyzer.rs#L69-L126): char filters, one tokenizer, then ordered token filters.+[`AnalyzerRegistry`](../engine/crates/xerj-fts/src/analyzer.rs#L937-L1100) starts with built-ins; supported custom settings are validated and installed at index creation ([create-time gate](../engine/crates/xerj-engine/src/index.rs#L5922-L5940)).+The FTS mapping builder chooses `standard` for `text` and `keyword` for other mapped field types ([`build_fts_field_configs`](../engine/crates/xerj-engine/src/index.rs#L19383-L19407)); its `standard` preset is Unicode word splitting plus lowercase, with no stop-word removal or stemming. Source wins over older architecture prose.++Query analysis is shape-specific, not general index/query custom-analyzer symmetry:+`match` uses the requested built-in analyzer or `standard` when omitted; unknown/custom+names decline to stored evaluation in this default registry ([`match` projection](../engine/crates/xerj-engine/src/index.rs#L37926-L37975)). Keyword matches use whole-value terms.+On keyword/exact fields, `match_phrase` uses the whole-value term path. On analyzed text fields, its FTS phrase path requires `standard` and zero slop; other analyzed shapes fall back to stored evaluation ([`match_phrase` implementation](../engine/crates/xerj-engine/src/index.rs#L38447-L38507)). Index-creation validation may accept supported custom settings,+but this projection does not promise general index/query custom-analyzer symmetry.++## 5. Update and delete semantics++Lucene documents are immutable once in a segment. In Lucene's own API,+`updateDocument` is expressed as delete-then-add; buffered deletes and live+document state are reconciled as segments are flushed and merged. The core+lifecycle is stable, but concrete segment, live-docs, and DocValues files are+codec/package details that can vary by Lucene commit; see the official+[10.3.1 index package](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/index/package-summary.html)+and [`IndexWriter`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/index/IndexWriter.html),+not universal filenames. Live-doc encoding is defined by the [codec live-docs+contract](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/codecs/LiveDocsFormat.html).+Lucene core accepts `IndexableField` objects, not JSON. Its+`updateNumericDocValue`/`updateBinaryDocValue` APIs only update existing fields+that were indexed with the corresponding DocValues type and DocValues only;+they cannot add arbitrary fields and are not a general JSON partial-update API.++XERJ also does not edit an already-published segment in place. A document+update appends a new WAL/index version under the same `_id`; the+[`VersionMap`](../engine/crates/xerj-storage/src/version_map.rs#L45-L118)+tracks the newest sequence number and whether the current version is deleted.+Old physical copies are ghosts until merge filtering removes them. Deletes+are WAL tombstones and can be carried in a segment's sequence-aware tombstone+section; the latest-version checks in merge and recovery keep an older copy+from reappearing.++The ES-shaped partial-update surface is real: [`update_document`](../engine/crates/xerj-engine/src/index.rs#L13144-L13210)+reads the current source, overlays top-level fields, and reindexes the merged+document; the API exposes `/_update/{id}` in [`update_doc`](../engine/crates/xerj-api/src/es_compat.rs#L17362-L17480)+and `/_update_by_query` in [`update_by_query`](../engine/crates/xerj-api/src/es_compat.rs#L24244-L24281).+That query-by-query handler hard-caps its fetch at 10,000 hits (`size: 10000`),+so one request is not a full-corpus traversal.+Updates are therefore logical replacement plus versioning, not byte-level+mutation. A delete also tombstones the old HNSW node when one exists, so a+vector result cannot outlive the document ([`delete_document_versioned`](../engine/crates/xerj-engine/src/index.rs#L12929-L13140)).++## 6. Vector index placement++Lucene treats a vector field as a first-class per-segment vector value and+delegates nearest-neighbor search to a `KnnVectorsFormat` reader. A+[`KnnFloatVectorField`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/document/KnnFloatVectorField.html)+creates that field, while [`KnnFloatVectorQuery`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/search/KnnFloatVectorQuery.html)+executes it; the codec boundary is [`KnnVectorsFormat`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/codecs/KnnVectorsFormat.html).+The vector index is separate from the inverted term postings even though both+belong to the same Lucene segment lifecycle.++XERJ keeps vectors out of the lexical FTS term dictionary. A mapped+`dense_vector` is stored in `_source` for hydration and maintained by+[`index_vectors`](../engine/crates/xerj-engine/src/index.rs#L9939-L10070) in an+index-level persisted HNSW artifact (`hnsw/`; [`save_hnsw_to_disk`](../engine/crates/xerj-engine/src/index.rs#L12274-L12335)).+Graph serialization is [`HnswIndex::save_to`](../engine/crates/xerj-vector/src/hnsw.rs#L963-L1008);+the doc-ID maps are persisted alongside it.+That graph is not a postings list and is not an active FTS field sidecar.++Unfiltered top-level cosine kNN may use HNSW only after field-identity,+coverage, freshness, and size checks; candidates are fetched and exact+rescored. Exact rescoring does not make candidate selection exact: HNSW remains+approximate, so candidate-set recall can differ from brute force. Filters,+nested/passage shapes, unsupported metrics, small or stale graphs, and any+failed eligibility check use the exact brute-force path. The dispatch and+fallback rules are explicit in [`search_at_generation`](../engine/crates/xerj-engine/src/index.rs#L13844-L13930)+and [`run_knn_hnsw`](../engine/crates/xerj-engine/src/index.rs#L10470-L10530).++Non-claims: this page does not claim byte-compatible indexes, identical merge+policies, Lucene codec compatibility, complete analyzer parity, identical+vector recall or latency, or that an Elasticsearch API name implies Lucene+internals. It is a current-source orientation guide; re-check the linked+symbols when changing the engine.