Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs: compare XERJ design with Lucene 10.3.1

buger08966ad70235
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index f75f4362a..aeddf39d0 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -8,11 +8,13 @@ build/run/test commands and the supported ES surface.
XERJ is an AI-native search, vector, and log-analytics engine written from
scratch in Rust and published under Apache-2.0 — designed for AI-agent workloads
-(zero-config `autoindex` onboarding, agent data map, `/_memory`), sharing no code
-or architecture with Elasticsearch or Lucene. It additionally speaks the
+(zero-config `autoindex` onboarding, agent data map, `/_memory`), with an
+independent implementation and no shared code or file formats with Elasticsearch
+or Lucene. It additionally speaks the
Elasticsearch 8.x HTTP protocol as a zero-migration adoption bridge, so existing
ES clients, dashboards, and ingest tooling talk to it unchanged (see
-[WHY_XERJ.md](./WHY_XERJ.md) for the design rationale).
+[WHY_XERJ.md](./WHY_XERJ.md) for the design rationale). For a six-axis,
+source-linked comparison with Lucene, see [XERJ vs Lucene](./XERJ_VS_LUCENE.md).
## Bird's-eye view
@@ -71,7 +73,7 @@ HTTP POST /{index}/_search
→ xerj-query parse_request(): raw ES JSON → SearchRequest (QueryNode tree)
→ xerj-engine Engine::get_index() looks up the named index
→ xerj-engine Index::search()
- ├─ memtable scan in-memory BM25 via FtsMemtable
+ ├─ memtable scan in-memory BM25 via ShardedFtsMemtable
├─ segment scan on-disk FTS via FtsIndexReader + BM25
├─ doc_matches_query() term-level / geo predicate evaluation
├─ run_aggs() aggregation pipeline (columnar fast path for size:0)
@@ -94,20 +96,33 @@ segments that are later merged:
```
PUT /{index}/_doc/{id} or POST /_bulk
→ xerj-api → xerj-engine → xerj-storage (IndexStore)
- ├─ WAL append single Mutex<WalWriter>; monotonic seq_no, one fsync per batch
- ├─ 16-shard memtable shard = xxh3_64(doc_id) & 15 (memtable_shards: Vec<Mutex<Vec<MemEntry>>>)
- ├─ flush take_memtable_for_flush() drains all shards,
- │ sorts by WAL seq_no to preserve global order,
- │ writes an immutable segment (LZ4/Zstd blocks)
+ ├─ WAL append exact selected N Mutex<WalWriter>s; global AtomicU64 seq_no
+ ├─ storage memtable max(N, 1).next_power_of_two() hash partitions
+ ├─ FTS memtable global engine.ingest_shards hash partitions
+ ├─ flush Index::flush() fans out per-shard do_flush_shard tasks;
+ │ each drains its shard and writes an immutable segment
+ │ with FTS/doc-values sidecars (LZ4/Zstd blocks)
└─ merge background segment merge compacts small segments;
_forcemerge is synchronous + quiescent (ES-like)
```
-The WAL writer is a single mutex so that sequence numbers are globally monotonic;
-lock hold time is kept short (one batched write and fsync). The storage memtable is
-sharded 16 ways to spread write contention. The engine-side FTS memtable is
-currently `Arc<RwLock<FtsMemtable>>`; a `ShardedFtsMemtable` scaffold exists in
-`memtable.rs` for a future refactor.
+The global `engine.ingest_shards` value is validated as a non-zero power of two
+no greater than 256 ([global validation](../engine/crates/xerj-common/src/config.rs#L1512-L1570)).
+An index may instead select its WAL count with `index.xerj_ingest_shards`; that
+override accepts any integer in `1..=256`, not only powers of two, and an absent
+or invalid override falls back to the global value ([override parsing](../engine/crates/xerj-engine/src/index.rs#L28488-L28517)).
+`IndexStore` opens exactly that selected number `N` of WAL writers, all sharing
+one global `seq_counter: AtomicU64`, while its separate storage memtable rounds
+`N` up to the next power of two for hash-mask routing ([construction](../engine/crates/xerj-storage/src/index_store.rs#L660-L720),
+[routing](../engine/crates/xerj-storage/src/index_store.rs#L1607-L1643)). These
+counts are therefore not necessarily paired one-to-one. The engine-side FTS
+memtable is separate again: create and reopen pass the global
+`engine.ingest_shards`, not the per-index WAL override, into
+[`ShardedFtsMemtable`](../engine/crates/xerj-engine/src/memtable.rs#L628-L709)
+([create](../engine/crates/xerj-engine/src/index.rs#L6022-L6026),
+[reopen](../engine/crates/xerj-engine/src/index.rs#L6220-L6224)).
+Query paths iterate its shards, and production flush drains each FTS shard
+through [`Index::flush`](../engine/crates/xerj-engine/src/index.rs#L18002-L18125)/[`do_flush_shard`](../engine/crates/xerj-engine/src/index.rs#L24433-L24920).
### Recovery
@@ -125,10 +140,11 @@ Engine::new() scans data_dir/
Elasticsearch compatibility is verified by the `es-yaml-runner` harness against the
ES 8.13 REST-API-spec YAML suites (search, aggregations, vectors, bulk, indices,
-scroll, cluster). XERJ currently passes 1,360 of 1,363 cases (3 skipped). The YAML tests are the
-source of truth: if XERJ returns a different response than a test expects, XERJ is
-considered wrong. See the README's "Running the conformance tests" section for how to run the suites and the full list
-of supported query types and aggregations.
+scroll, cluster). The hard gate is zero failed cases; skipped cases are reported
+separately and totals can change as the upstream suite changes. The YAML tests are
+the source of truth: if XERJ returns a different response than a test expects, XERJ
+is considered wrong. See the README's "Running the conformance tests" section for
+how to run the suites and the full list of supported query types and aggregations.
Performance is tracked with a reproducible full-matrix head-to-head against live
Elasticsearch 8.13.4, published at <https://xerj.org/benchmarks> (per-cell results
diff --git a/docs/WHY_XERJ.md b/docs/WHY_XERJ.md
index 45e1d629a..c2a020262 100644
--- a/docs/WHY_XERJ.md
+++ b/docs/WHY_XERJ.md
@@ -2,7 +2,9 @@
*Or: why "Elasticsearch-compatible" is the least interesting thing about this engine.*
-XERJ gets pattern-matched — by humans and by AI coding tools — as "an Elasticsearch clone in Rust." This document exists to correct that reading with specifics. XERJ shares **no code and no architecture** with Elasticsearch or Lucene. It is a from-scratch engine whose design target is different: **the primary user is an AI agent**, not a human with a dashboard. ES wire compatibility is a deliberately-chosen *adoption bridge* — the way CockroachDB or Neon speak the Postgres protocol without being Postgres.
+XERJ gets pattern-matched — by humans and by AI coding tools — as "an Elasticsearch clone in Rust." This document exists to correct that reading with specifics. XERJ is an independent implementation with **no shared code or file formats** with Elasticsearch or Lucene. It is a from-scratch engine whose design target is different: **the primary user is an AI agent**, not a human with a dashboard. ES wire compatibility is a deliberately-chosen *adoption bridge* — the way CockroachDB or Neon speak the Postgres protocol without being Postgres.
+
+For an implementation-oriented, six-axis comparison with Lucene, see [XERJ vs Lucene](./XERJ_VS_LUCENE.md).
## The design thesis
@@ -34,7 +36,9 @@ The autoindex pipeline was verified adversarially: 80/81 ground-truth checks on
## What ES compatibility buys (and what it doesn't)
-**Buys:** zero-cost adoption. 1,360/1,363 ES-YAML conformance cases pass, so existing ES client libraries, dashboards, and habits work unchanged. Migration is `change the URL`.
+**Buys:** zero-cost adoption. The ES-YAML conformance gate requires zero failures,
+so existing ES client libraries, dashboards, and habits work unchanged. Migration
+is `change the URL`; passing and skipped totals can change as the upstream suite evolves.
**Doesn't buy:** the mission. Compatibility gaps that remain are tracked openly (see the stub audit and compatibility docs in `demo/playbooks/`), and where ES semantics are genuinely better, XERJ converges on them — but the roadmap is driven by the agent workflow above, not by chasing feature-checklist parity.
diff --git a/docs/XERJ_VS_LUCENE.md b/docs/XERJ_VS_LUCENE.md
new file mode 100644
index 000000000..51d3c587d
--- /dev/null
+++ b/docs/XERJ_VS_LUCENE.md
@@ -0,0 +1,172 @@
+# XERJ vs Lucene 10.3.1
+
+If you are an AI agent deciding where to look in XERJ, use this page as a
+source map: it answers the six storage questions a Lucene/Solr/Elasticsearch
+engineer is likely to ask, then links each XERJ answer to the current symbol.
+XERJ is a from-scratch Rust engine whose primary workflow is
+`autoindex <folder> -> autoindex map -> query`; Lucene 10.3.1 is an embeddable
+Java search library. Elasticsearch-wire compatibility is an adoption bridge,
+not evidence that the two engines share an implementation.
+
+The Lucene links below point only to official Apache Lucene 10.3.1 API/source documentation. The XERJ links describe the runtime on current `main`, including the `ShardedFtsMemtable` path; source wins if a nearby overview page is older.
+
+The default embedding fallback is lexical feature-hashing, not neural semantic understanding ([`Embedder`](../engine/crates/xerj-ai/src/embedder.rs#L7-L23)); the built-in neural BERT backend is
+opt-in via `--embed-mode neural` and downloads its model on first use ([model loader](../engine/crates/xerj-ai/src/neural.rs#L225-L275)).
+
+## 1. Segment and merge model
+
+Lucene buffers indexing work in `IndexWriter`, flushes immutable segments, and
+lets a `MergePolicy` select groups of segments for replacement by a merged
+segment. Its default merge scheduler can run those merges in background
+threads. See the [Lucene 10.3.1 `IndexWriter`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/index/IndexWriter.html),
+[merge policy](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/index/MergePolicy.html),
+and [concurrent merge scheduler](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/index/ConcurrentMergeScheduler.html).
+
+XERJ has a different publication boundary. A WAL records mutations; production
+[`Index::flush`](../engine/crates/xerj-engine/src/index.rs#L18002-L18125) fans out one
+[`do_flush_shard`](../engine/crates/xerj-engine/src/index.rs#L24433-L24920) task per FTS shard. Each shard drains and sequence-orders its own entries, builds FTS and `.dv` sidecars, then calls [`finalize_flush_with_publisher`](../engine/crates/xerj-storage/src/index_store.rs#L1896-L1910) to write an immutable `.seg` and atomically publish an `ArcSwap` snapshot.
+
+The live ingest buffer is a configurable
+[`ShardedFtsMemtable`](../engine/crates/xerj-engine/src/memtable.rs#L628-L700),
+not one process-wide FTS lock. Its shards drain and re-sort by WAL sequence;
+the background [`run_merge_once`](../engine/crates/xerj-engine/src/index.rs#L8468-L8495)
+writes a replacement segment from surviving documents, and
+[`apply_merge`](../engine/crates/xerj-storage/src/index_store.rs#L3680-L3822)
+publishes it before [`retire_segment_files`](../engine/crates/xerj-storage/src/index_store.rs#L1531-L1570)
+retires the inputs. A XERJ segment is a checksummed container whose
+[`SegmentWriter::finish`](../engine/crates/xerj-storage/src/segment.rs#L402-L475)
+writes named sections ([`SectionType`](../engine/crates/xerj-storage/src/segment.rs#L85-L113));
+FTS and doc values are separate sidecars, while stored source is a segment
+section.
+
+## 2. Postings layout
+
+Lucene 10.3.1's postings format uses a BlockTree term dictionary and packed
+integer blocks (128 integers in the packed path), with skip data interleaved at
+two levels. The primary references are [Lucene 10.3.1 `Lucene103PostingsFormat`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/codecs/lucene103/Lucene103PostingsFormat.html)
+and its [postings writer](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/codecs/lucene103/Lucene103PostingsWriter.html).
+
+XERJ's [`FtsIndexWriter`](../engine/crates/xerj-fts/src/index.rs#L557-L620)
+writes one sidecar family per indexed field: an FST term dictionary, a compressed `.post` blob, binary `.meta` term statistics, and encoded `.norms`.
+The FST maps a term to a fixed metadata record; it does not point directly to
+a Lucene segment file. The format and reader are documented in
+[`FtsIndexReader`](../engine/crates/xerj-fts/src/index.rs#L1150-L1228).
+
+[`PostingsWriter::encode_term`](../engine/crates/xerj-fts/src/postings.rs#L260-L331)
+delta-encodes doc IDs in 128-doc PFOR blocks through its
+[packed-block helpers](../engine/crates/xerj-fts/src/postings.rs#L333-L421);
+positioned fields add packed term frequencies and variable-byte positions,
+while exact fields can omit both. Residual postings use variable-byte encoding.
+The [module format notes](../engine/crates/xerj-fts/src/postings.rs#L1-L40) make
+the boundary explicit: this shared block size is not Lucene codec compatibility.
+XERJ's `SkipEntry` is currently in-memory scaffolding and is not serialized, so
+[`advance_to`](../engine/crates/xerj-fts/src/postings.rs#L702-L708) scans linearly.
+Do not infer Lucene's two-level skip behavior from the shared block size.
+
+## 3. Doc-values equivalent
+
+Lucene exposes typed, column-oriented DocValues alongside postings; its codec chooses
+on-disk representations for numeric, sorted, sorted-set, and related types. The
+10.3.1 reference describes `.dvd` and `.dvm` files in
+[`Lucene90DocValuesFormat`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/codecs/lucene90/Lucene90DocValuesFormat.html), while
+[`SortedSetDocValuesField`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/document/SortedSetDocValuesField.html)
+is useful for a multi-valued sorted field.
+
+XERJ's closest equivalent is a per-segment `{segment_id}.dv` sidecar. The
+[`Column` enum](../engine/crates/xerj-storage/src/doc_values.rs#L560-L590)
+currently has numeric and keyword columns, each with a missing-value bitmap;
+keyword columns use sorted terms plus ordinals, and numeric columns retain a
+sorted `(value, doc_id)` index for range work. The binary envelope is produced
+by [`encode_columns`](../engine/crates/xerj-storage/src/doc_values.rs#L620-L735)
+and loaded through the engine's `.dv` cache.
+
+The columns feed XERJ sorting, filters, ranges, and aggregation fast paths without reparsing `_source`. They are an analogous column store, not a
+drop-in Lucene DocValues codec: field eligibility, array handling, compression,
+and the public mapping contract are XERJ behavior. The segment's stored source
+remains the fallback and hydration authority.
+
+## 4. Analysis chain
+
+Lucene's [`Analyzer`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/analysis/Analyzer.html)
+creates a reusable token stream for a field, and
+[`StandardAnalyzer`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/analysis/standard/StandardAnalyzer.html)
+is one concrete preset. Analyzer output is part of the index contract: changing
+analysis changes the terms that postings contain.
+
+XERJ models the same conceptual stages in [`AnalyzerPipeline::analyze`](../engine/crates/xerj-fts/src/analyzer.rs#L69-L126): char filters, one tokenizer, then ordered token filters.
+[`AnalyzerRegistry`](../engine/crates/xerj-fts/src/analyzer.rs#L937-L1100) starts with built-ins; supported custom settings are validated and installed at index creation ([create-time gate](../engine/crates/xerj-engine/src/index.rs#L5922-L5940)).
+The FTS mapping builder chooses `standard` for `text` and `keyword` for other mapped field types ([`build_fts_field_configs`](../engine/crates/xerj-engine/src/index.rs#L19383-L19407)); its `standard` preset is Unicode word splitting plus lowercase, with no stop-word removal or stemming. Source wins over older architecture prose.
+
+Query analysis is shape-specific, not general index/query custom-analyzer symmetry:
+`match` uses the requested built-in analyzer or `standard` when omitted; unknown/custom
+names decline to stored evaluation in this default registry ([`match` projection](../engine/crates/xerj-engine/src/index.rs#L37926-L37975)). Keyword matches use whole-value terms.
+On keyword/exact fields, `match_phrase` uses the whole-value term path. On analyzed text fields, its FTS phrase path requires `standard` and zero slop; other analyzed shapes fall back to stored evaluation ([`match_phrase` implementation](../engine/crates/xerj-engine/src/index.rs#L38447-L38507)). Index-creation validation may accept supported custom settings,
+but this projection does not promise general index/query custom-analyzer symmetry.
+
+## 5. Update and delete semantics
+
+Lucene documents are immutable once in a segment. In Lucene's own API,
+`updateDocument` is expressed as delete-then-add; buffered deletes and live
+document state are reconciled as segments are flushed and merged. The core
+lifecycle is stable, but concrete segment, live-docs, and DocValues files are
+codec/package details that can vary by Lucene commit; see the official
+[10.3.1 index package](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/index/package-summary.html)
+and [`IndexWriter`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/index/IndexWriter.html),
+not universal filenames. Live-doc encoding is defined by the [codec live-docs
+contract](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/codecs/LiveDocsFormat.html).
+Lucene core accepts `IndexableField` objects, not JSON. Its
+`updateNumericDocValue`/`updateBinaryDocValue` APIs only update existing fields
+that were indexed with the corresponding DocValues type and DocValues only;
+they cannot add arbitrary fields and are not a general JSON partial-update API.
+
+XERJ also does not edit an already-published segment in place. A document
+update appends a new WAL/index version under the same `_id`; the
+[`VersionMap`](../engine/crates/xerj-storage/src/version_map.rs#L45-L118)
+tracks the newest sequence number and whether the current version is deleted.
+Old physical copies are ghosts until merge filtering removes them. Deletes
+are WAL tombstones and can be carried in a segment's sequence-aware tombstone
+section; the latest-version checks in merge and recovery keep an older copy
+from reappearing.
+
+The ES-shaped partial-update surface is real: [`update_document`](../engine/crates/xerj-engine/src/index.rs#L13144-L13210)
+reads the current source, overlays top-level fields, and reindexes the merged
+document; the API exposes `/_update/{id}` in [`update_doc`](../engine/crates/xerj-api/src/es_compat.rs#L17362-L17480)
+and `/_update_by_query` in [`update_by_query`](../engine/crates/xerj-api/src/es_compat.rs#L24244-L24281).
+That query-by-query handler hard-caps its fetch at 10,000 hits (`size: 10000`),
+so one request is not a full-corpus traversal.
+Updates are therefore logical replacement plus versioning, not byte-level
+mutation. A delete also tombstones the old HNSW node when one exists, so a
+vector result cannot outlive the document ([`delete_document_versioned`](../engine/crates/xerj-engine/src/index.rs#L12929-L13140)).
+
+## 6. Vector index placement
+
+Lucene treats a vector field as a first-class per-segment vector value and
+delegates nearest-neighbor search to a `KnnVectorsFormat` reader. A
+[`KnnFloatVectorField`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/document/KnnFloatVectorField.html)
+creates that field, while [`KnnFloatVectorQuery`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/search/KnnFloatVectorQuery.html)
+executes it; the codec boundary is [`KnnVectorsFormat`](https://lucene.apache.org/core/10_3_1/core/org/apache/lucene/codecs/KnnVectorsFormat.html).
+The vector index is separate from the inverted term postings even though both
+belong to the same Lucene segment lifecycle.
+
+XERJ keeps vectors out of the lexical FTS term dictionary. A mapped
+`dense_vector` is stored in `_source` for hydration and maintained by
+[`index_vectors`](../engine/crates/xerj-engine/src/index.rs#L9939-L10070) in an
+index-level persisted HNSW artifact (`hnsw/`; [`save_hnsw_to_disk`](../engine/crates/xerj-engine/src/index.rs#L12274-L12335)).
+Graph serialization is [`HnswIndex::save_to`](../engine/crates/xerj-vector/src/hnsw.rs#L963-L1008);
+the doc-ID maps are persisted alongside it.
+That graph is not a postings list and is not an active FTS field sidecar.
+
+Unfiltered top-level cosine kNN may use HNSW only after field-identity,
+coverage, freshness, and size checks; candidates are fetched and exact
+rescored. Exact rescoring does not make candidate selection exact: HNSW remains
+approximate, so candidate-set recall can differ from brute force. Filters,
+nested/passage shapes, unsupported metrics, small or stale graphs, and any
+failed eligibility check use the exact brute-force path. The dispatch and
+fallback rules are explicit in [`search_at_generation`](../engine/crates/xerj-engine/src/index.rs#L13844-L13930)
+and [`run_knn_hnsw`](../engine/crates/xerj-engine/src/index.rs#L10470-L10530).
+
+Non-claims: this page does not claim byte-compatible indexes, identical merge
+policies, Lucene codec compatibility, complete analyzer parity, identical
+vector recall or latency, or that an Elasticsearch API name implies Lucene
+internals. It is a current-source orientation guide; re-check the linked
+symbols when changing the engine.