AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(ai): ship the neural embedder in the single default binary
xerj-orgecf4c2e64a02
diff --git a/AGENTS.md b/AGENTS.mdindex 377e29a08..d3a83dfd8 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -6,7 +6,7 @@## Ground rules for working in this repo-- **Honest claims only.** Every public number traces to a verified run. The **default** embedder is *lexical* feature-hashing — never claim the default binary does neural/semantic understanding. A built-in **neural** BERT embedder exists but is **opt-in** (`--features neural` + `--embed-mode neural`, model downloads on first use); only describe output as neural when that mode is actually running. kNN is *exact* brute-force, recall 1.00 (never imply ANN speed); benchmark headline is the audited scorecard in `demo/playbooks/FULL_MATRIX_SCORECARD_*.md` (currently 42W/28L/12T vs live ES 8.13.4); do not claim TB-scale end-to-end (server heap ticket: `demo/usecases/autoindex/scale/TICKET_server-unbounded-ingest-heap.md`).+- **Honest claims only.** Every public number traces to a verified run. The **default** embedding mode is *lexical* feature-hashing — never claim the default mode does neural/semantic understanding. A built-in **neural** BERT embedder ships in the binary but is **off unless activated** (`--embed-mode neural`, model auto-downloads on first use); only describe output as neural when that mode is actually running. kNN is *exact* brute-force, recall 1.00 (never imply ANN speed); benchmark headline is the audited scorecard in `demo/playbooks/FULL_MATRIX_SCORECARD_*.md` (currently 42W/28L/12T vs live ES 8.13.4); do not claim TB-scale end-to-end (server heap ticket: `demo/usecases/autoindex/scale/TICKET_server-unbounded-ingest-heap.md`).- **Builds are scoped:** `cd engine && cargo build --release -j 32 -p <crate>` — never workspace-wide, never `cargo clean`.- **The hard gate:** the ES-YAML conformance suite must stay at 1360 passed / 0 failed / 3 skipped before any engine change lands (see "Running the conformance tests" in the README).- **Git discipline:** non-trivial changes land with full commit bodies (motivation, before/after numbers, root cause, file pointers) — the git history is the project's engineering log; read it before re-deriving decisions.diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f75bd8a19..5e69f7156 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -16,12 +16,14 @@ retrieval recipes.### Added-- **Built-in neural BERT embedder (`xerj-ai`, feature `neural`).** A pure-Rust- sentence encoder via `candle` (default `all-MiniLM-L6-v2`, 384-dim) that runs- in-process and downloads its weights once on first use (or reads them from- `embedding.local_model_dir` for air-gapped deployments). Off by default so the- standard binary stays ~23 MB and dependency-light; build with- `cargo build --release -p xerj-server --features neural` to include it.+- **Built-in neural BERT embedder — shipped in the binary.** A pure-Rust sentence+ encoder via `candle` (default `all-MiniLM-L6-v2`, 384-dim) that runs in-process+ and **downloads its weights (~90 MB) automatically on first use** (or reads them+ from `embedding.local_model_dir` for air-gapped deployments). It is compiled into+ the default release binary — end users just add `--embed-mode neural` at runtime,+ no special build and no separate binary. A progress bar and one-time-download log+ make the first run legible. The binary is ~36 MB as a result; a+ `--no-default-features` slim build without the neural backend is ~23 MB.- **Unified three-backend embedding handle (`xerj_ai::Embedder`).** `semantic_text`ingest and `semantic`/`hybrid` queries run through one of three interchangeablebackends — **lexical** (default, zero-dep feature-hash), **neural** (built-indiff --git a/README.md b/README.mdindex 3faa3b850..178c73fe2 100644--- a/README.md+++ b/README.md@@ -43,8 +43,8 @@ What is actually designed differently — for AI-agent workloads specifically, w- **A data map built for model orientation** — `xerj autoindex map` and the catalog index answer an agent's real first question, *"what is in here?"*, with counts, types, example values, ready-to-send queries, and known gotchas. Orientation designed for LLMs, not for humans with dashboards.- **Agent memory as a first-class API** — `/_memory/{ns}` store/recall with namespace isolation, semantic recall, dedup, metadata filters, and recency blending. Not a plugin; part of the engine.- **Machine-readable onboarding** — [llms.txt](https://xerj.org/llms.txt), agent tool schemas (OpenAI / MCP / Anthropic), [for-agents docs](https://xerj.org/for-agents.html). The documentation treats an LLM as a first-class reader, including honest caveat blocks it can rely on.-- **Batteries-included retrieval** — a built-in zero-config embedder (lexical feature-hashing by default, documented honestly as such; an optional in-process **neural** BERT embedder and any external embeddings endpoint are drop-in alternatives) plus server-side hybrid RRF/linear fusion: BM25 + vector + hybrid out of the box, no external inference service required.-- **A single ~23 MB static binary, no JVM, sub-second start** — small enough that an agent can spawn a search engine as a subprocess tool, on any of 8 released targets.+- **Batteries-included retrieval** — a built-in zero-config embedder (lexical feature-hashing by default, documented honestly as such) plus a real in-process **neural** BERT embedder shipped in the same binary (`--embed-mode neural`, model auto-downloads on first use) and any external embeddings endpoint as a third option — all behind server-side hybrid RRF/linear fusion: BM25 + vector + hybrid out of the box, no external inference service required.+- **A single ~36 MB static binary, no JVM, sub-second start** — batteries included: the neural embedder is built in (activate with `--embed-mode neural`; a `--no-default-features` slim build is ~23 MB). Small enough that an agent can spawn a search engine as a subprocess tool, on any of 8 released targets.- **Exact kNN with recall 1.00 by construction** — the trade-off (latency scales with vectors scanned) is documented instead of hidden behind ANN defaults.- **Engineering-first internals** — columnar aggregation fast paths, doc-value prefilters, WAL-shard-pinned delete durability (verified by adversarial crash matrices), Rust memory safety. The head-to-head wins vs Elasticsearch concentrate exactly where agents live: aggregations, ingest throughput, and disk footprint.@@ -104,7 +104,7 @@ We ran the same model (headless Claude Code) against the 518 MB corpus twice: on- **Drop-in ES wire compatibility.** Implements the Elasticsearch REST surface — index/document APIs, `_search`, `_bulk`, aggregations, kNN, scroll, aliases, templates. Validated against **1,360 of 1,363** ES-YAML conformance cases extracted from Elasticsearch 8.13 source.- **Truly open.** Apache-2.0 licensed. No SSPL, no source-available asterisks, no per-feature license gates.- **One engine, four workloads.** Full-text BM25, exact dense-vector kNN (100% recall), the standard aggregation suite, and columnar log analytics — all in the same process, over the same wire protocol.-- **A single native binary.** Roughly a ~23 MB statically-linked executable. Sub-second start, no JVM, no heap-tuning ritual.+- **A single native binary.** Roughly a ~36 MB statically-linked executable with the neural embedder built in (~23 MB as a `--no-default-features` slim build). Sub-second start, no JVM, no heap-tuning ritual.- **Honest, reproducible benchmarks — as supporting evidence.** Measured head-to-head against a live Elasticsearch 8.13.4, the latest closed-loop, cache-off matrix scores **42 wins / 28 losses / 12 ties** for XERJ. Results — wins *and* losses — are published at [xerj.org/benchmarks](https://xerj.org/benchmarks) and reproducible from the scripts in this repo (see [Benchmarks](#benchmarks)).- **Written in Rust.** Memory-safe, `panic = "abort"`, fat-LTO release builds. Embedded Raft consensus (no external Raft dependency) for cluster metadata.@@ -222,7 +222,7 @@ XERJ implements the Elasticsearch REST wire protocol. Because it is wire-compati`match_all`, `match_none`, `match`, `match_phrase`, `match_phrase_prefix`, `multi_match`, `term`, `terms`, `range`, `prefix`, `wildcard`, `exists`, `ids`, `bool`, `fuzzy`, `regexp`, `query_string`, `simple_query_string`, `constant_score`, `boosting`, `dis_max`, `geo_distance`, `knn`, `semantic`, `hybrid`.-> **Vector & semantic search notes.** `knn` (exact dense-vector search — brute-force scan, 100% recall; no ANN index yet) and `hybrid` (BM25 + kNN in one request) run entirely inside XERJ. As of **rc-2**, `semantic_text` fields **auto-embed on ingest** and the `semantic` query works with no external service. XERJ has **three interchangeable embedding backends**, selected with `--embed-mode` (or `embedding.mode`): **`lexical`** (default) — the built-in, zero-config feature-hash embedder (lexical, *not* neural); **`neural`** — a built-in **BERT sentence embedder** (all-MiniLM-L6-v2) that runs in-process via `candle`, downloading the model once on first use (build with `--features neural`); and **`proxy`** — any external OpenAI-compatible `/v1/embeddings` endpoint (`embedding.default_endpoint`). The default binary ships the lexical embedder to stay ~23 MB and dependency-free; see the [roadmap](#roadmap). rc-2 also adds an **agent-memory REST API** (`/_memory/{ns}`) and **anomaly detection** (`/_ml/anomaly_detectors`).+> **Vector & semantic search notes.** `knn` (exact dense-vector search — brute-force scan, 100% recall; no ANN index yet) and `hybrid` (BM25 + kNN in one request) run entirely inside XERJ. `semantic_text` fields **auto-embed on ingest** and the `semantic` query works with no external service. XERJ has **three interchangeable embedding backends**, selected with `--embed-mode` (or `embedding.mode`): **`lexical`** (default) — the built-in, zero-config feature-hash embedder (lexical, *not* neural); **`neural`** — a built-in **BERT sentence embedder** (all-MiniLM-L6-v2) that runs in-process via `candle`; and **`proxy`** — any external OpenAI-compatible `/v1/embeddings` endpoint (`embedding.default_endpoint`). **The neural embedder ships in the binary** — just start the server with `--embed-mode neural` and the model (~90 MB) downloads itself on first use, then runs from cache; no separate build, no external service, air-gap friendly (`embedding.local_model_dir`). rc-2 also adds an **agent-memory REST API** (`/_memory/{ns}`) and **anomaly detection** (`/_ml/anomaly_detectors`).**Supported aggregations**@@ -254,7 +254,7 @@ XERJ is a Cargo workspace under [`engine/`](./engine). The crates:| `xerj-fts` | Full-text search: BM25 scoring, analyzer registry, postings lists. || `xerj-vector` | Dense-vector index for kNN / semantic search (exact brute-force scan, 100% recall). || `xerj-logs` | Columnar log ingestion and retention. |-| `xerj-ai` | Text chunking and the three embedding backends behind one `Embedder` handle — the built-in lexical embedder, an optional in-process **neural** BERT embedder (`candle` + MiniLM, feature `neural`), and an embedding proxy (external OpenAI-compatible API) — all powering the `semantic` query. The memory store is exposed over the REST API at `/_memory/{ns}` (store / `_recall` / list / forget), shipped in rc-2 — see the [roadmap](#roadmap). |+| `xerj-ai` | Text chunking and the three embedding backends behind one `Embedder` handle — the built-in lexical embedder, a built-in in-process **neural** BERT embedder (`candle` + MiniLM, shipped in the binary), and an embedding proxy (external OpenAI-compatible API) — all powering the `semantic` query. The memory store is exposed over the REST API at `/_memory/{ns}` (store / `_recall` / list / forget), shipped in rc-2 — see the [roadmap](#roadmap). || `xerj-compress` | Block compression codecs (LZ4, Zstd). || `xerj-common` | Shared types: `Config`, `Schema`, `FieldType`, `XerjError`. || `xerj-cluster` | Embedded Raft consensus for cluster metadata (no external Raft dependency). |@@ -314,13 +314,13 @@ If a test expects a response and XERJ returns something different, that's a bugXERJ ships full-text, aggregation, log-analytics, dense-vector kNN, and hybrid search. **rc-2** added three AI-adjacent capabilities (each conformance-gated and verified by real requests):-- **Auto-embed on ingest + three embedding backends** — `semantic_text` works with zero external config on the default **lexical** embedder; opt into a built-in **neural** BERT embedder (`--embed-mode neural`, `--features neural`, model downloads on first use) or an external **proxy** (`--embed-mode proxy`) with no change to your mappings or queries.+- **Auto-embed on ingest + three embedding backends** — `semantic_text` works with zero external config on the default **lexical** embedder; switch to the **neural** BERT embedder built into the binary (`--embed-mode neural`, model auto-downloads on first use) or an external **proxy** (`--embed-mode proxy`) with no change to your mappings or queries.- **Ingest-time chunk-embedding** — long `semantic_text` values embed **per overlapping passage**, and `semantic` search scores by the best-matching passage (max-sim), so a long document is retrievable on any one of its sections.- **Agent-memory REST API** — `POST /_memory/{ns}` store, `/_recall` (kNN or BM25 + metadata filter), list/forget/drop; namespaced and offline.- **Anomaly detection (`_ml`)** — real statistical detectors with baseline + deviation scoring, on-demand **and** via continuous `_ml/datafeeds` (a background job re-scores a live index on a timer; forecasting is still future work).- **Vector quantization** — opt a `dense_vector` field into **scalar8** (`int8_hnsw`) for ~4× smaller vectors at recall@10 ≈ 0.99.-Still planned: shipping the neural embedder in the default release binary (it's opt-in behind `--features neural` today to keep the default ~23 MB and dependency-free) and distributed-clustering hardening. See [ROADMAP.md](./ROADMAP.md) for the full status and honest limitations.+The built-in neural embedder now **ships in the default release binary** (activate with `--embed-mode neural`; model auto-downloads on first use). Still planned: distributed-clustering hardening and a larger/higher-quality default model option. See [ROADMAP.md](./ROADMAP.md) for the full status and honest limitations.---diff --git a/ROADMAP.md b/ROADMAP.mdindex be85863e8..8fa849974 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -26,7 +26,7 @@ These three shipped in rc-2 (each conformance-gated at 1,326 / 1,329 and verifie`semantic_text` now works end to end with **zero external configuration**. Indexing a document into a `semantic_text` field auto-embeds its text (previously returned `405`), and the `semantic` query embeds the query text with the same embedder and runs kNN — no external service required. Live-verified: a `semantic_text` doc indexed with no embedder configured returned `201`, and a `semantic` query ranked the intended doc first. A configured external `/v1/embeddings` proxy is still used, at higher quality, when `embedding.default_endpoint` is set.-- **Limitation:** the **default** built-in embedder is a deterministic **lexical** model (feature-hashed word unigrams + character trigrams, L2-normalized) — it captures vocabulary/sub-word overlap, not deep semantics. This is observable live: a vocabulary-sharing query out-scored a true paraphrase. Paraphrases that share vocabulary rank correctly; truly-synonymous text with no word overlap will not. For real neural semantics you now have two drop-in upgrades with no mapping/query change: the built-in **neural** BERT embedder (`--embed-mode neural`, opt-in behind `--features neural`, downloads all-MiniLM-L6-v2 on first use — see "Neural embeddings" below), or the external `/v1/embeddings` **proxy** (`--embed-mode proxy` + `embedding.default_endpoint`).+- **Limitation:** the **default** embedding mode is a deterministic **lexical** model (feature-hashed word unigrams + character trigrams, L2-normalized) — it captures vocabulary/sub-word overlap, not deep semantics. This is observable live: a vocabulary-sharing query out-scored a true paraphrase. Paraphrases that share vocabulary rank correctly; truly-synonymous text with no word overlap will not. For real neural semantics you have two drop-in upgrades with no mapping/query change: the built-in **neural** BERT embedder that ships in the binary (`--embed-mode neural`, downloads all-MiniLM-L6-v2 on first use — see "Neural embeddings" below), or the external `/v1/embeddings` **proxy** (`--embed-mode proxy` + `embedding.default_endpoint`).### 2. Agent-memory REST API ✅ (rc-2)@@ -103,7 +103,7 @@ Counting these toward "supported query types" overstates correctness — they re### Neural embeddings & richer ML-- A built-in **neural** embedder has **landed** (opt-in): `--embed-mode neural` runs an in-process BERT sentence encoder (all-MiniLM-L6-v2, 384-dim) via `candle` — pure Rust, no Python, no external service, air-gap friendly (`embedding.local_model_dir`). It is behind the `neural` cargo feature so the default release binary stays ~23 MB and dependency-free. **Remaining work:** ship it in the default binary (or a separate `xerj-neural` artifact) so no rebuild is needed, and offer a larger/higher-quality default model.+- A built-in **neural** embedder has **landed and ships in the default binary**: `--embed-mode neural` runs an in-process BERT sentence encoder (all-MiniLM-L6-v2, 384-dim) via `candle` — pure Rust, no Python, no external service, model auto-downloads on first use (air-gap friendly via `embedding.local_model_dir`). No rebuild and no separate artifact needed; the shipped binary is ~36 MB (a `--no-default-features` slim build without the neural backend is ~23 MB). **Remaining work:** offer a larger/higher-quality default model option.- Forecasting for capacity/write-load signals (continuous `_ml` datafeeds have landed — see "Landed since rc-2"; the ingest-time per-passage chunk-embedding pipeline has also landed).### Correctness of stubbed surfacediff --git a/docs/WHY_XERJ.md b/docs/WHY_XERJ.mdindex 710cb4cd3..2570241df 100644--- a/docs/WHY_XERJ.md+++ b/docs/WHY_XERJ.md@@ -25,7 +25,7 @@ The autoindex pipeline was verified adversarially: 80/81 ground-truth checks on## Engineering positions (where XERJ deliberately differs)-- **One static binary (~23 MB), no JVM.** Sub-second cold start. An agent can spawn its own search engine as a subprocess tool. Operationally: no heap tuning, no GC pauses.+- **One static binary (~36 MB, neural embedder included), no JVM.** Sub-second cold start. An agent can spawn its own search engine as a subprocess tool. Operationally: no heap tuning, no GC pauses. (A `--no-default-features` slim build without the neural backend is ~23 MB.)- **Columnar aggregation fast paths + doc-value prefilters.** The analytical questions agents actually ask (counts, cardinalities, histograms, percentiles over millions of rows) are served from columnar data, not stored-document scans. This is where XERJ *beats* Elasticsearch in the head-to-head.- **Exact kNN, recall 1.00 by construction.** No ANN index is built or traversed; the latency trade-off is documented, not hidden behind approximate defaults. (An HNSW implementation exists in the codebase but the ES-compat query path is intentionally exact.)- **Built-in zero-config embedder.** Lexical feature-hashing (384-dim) — documented honestly as *hybrid lexical+vector*, never claimed to be neural. Semantic-ish retrieval works offline with zero external dependencies; point `semantic_text` at an external inference endpoint when you need real embeddings.diff --git a/docs/examples/all-way-search/all_way_search.py b/docs/examples/all-way-search/all_way_search.pyindex 5b1ceccce..0b275789d 100644--- a/docs/examples/all-way-search/all_way_search.py+++ b/docs/examples/all-way-search/all_way_search.py@@ -23,7 +23,8 @@ Run it:For REAL neural semantics (recommended — modes 2/4/5 get much sharper),start the server with the built-in BERT embedder instead of the default-lexical one (needs a binary built with `--features neural`):+lexical one. It ships in the binary; just add the flag (the model+auto-downloads on first use):xerj --insecure --data-dir ./data --embed-mode neuraldiff --git a/docs/examples/autoindex-semantic-search/autoindex_semantic.sh b/docs/examples/autoindex-semantic-search/autoindex_semantic.shindex fc2fa4ead..ff339512a 100755--- a/docs/examples/autoindex-semantic-search/autoindex_semantic.sh+++ b/docs/examples/autoindex-semantic-search/autoindex_semantic.sh@@ -14,15 +14,16 @@## Usage: recipes/autoindex_semantic.sh [path-to-xerj-binary]#-# Requires a xerj binary built with the neural feature:-# cd engine && cargo build --release -p xerj-server --features neural+# The neural embedder ships in the standard xerj binary — no special build.+# (A `--no-default-features` slim build omits it and falls back to lexical,+# which this script detects and notes.)set -euROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)XERJ_BIN="${1:-$ROOT/engine/target/release/xerj}"[ -x "$XERJ_BIN" ] || XERJ_BIN=$(command -v xerj) || {- echo "error: no xerj binary. Build it with:" >&2- echo " cd engine && cargo build --release -p xerj-server --features neural" >&2+ echo "error: no xerj binary. Get one with 'curl -fsSL https://xerj.org/get | sh'" >&2+ echo " or build it: cd engine && cargo build --release -p xerj-server" >&2exit 1}FOLDER="$ROOT/demo/data/support-folder"@@ -53,8 +54,8 @@ until curl -fsS "$URL/" >/dev/null 2>&1; dodoneif ! grep -q "neural" "$DATA/server.log"; then- echo "note: this binary has no neural backend — falling back to the lexical"- echo " embedder. Rebuild with --features neural for real neural semantics."+ echo "note: this is a slim build with no neural backend — falling back to the"+ echo " lexical embedder. The standard release binary has neural built in."echofidiff --git a/docs/recipes/all-way-search.md b/docs/recipes/all-way-search.mdindex 6d736b6bf..699ea9341 100644--- a/docs/recipes/all-way-search.md+++ b/docs/recipes/all-way-search.md@@ -23,7 +23,7 @@ Runnable version: [`docs/examples/all-way-search/all_way_search.py`](../examples> BERT embedder — one flag, no change to the mapping or any query:>> ```bash-> # needs a binary built with --features neural; model downloads once+> # the neural embedder ships in the binary; the model auto-downloads on first use> xerj --insecure --data-dir ./data --embed-mode neural> ```diff --git a/docs/recipes/autoindex-semantic-search.md b/docs/recipes/autoindex-semantic-search.mdindex 9116afffa..dda1601d0 100644--- a/docs/recipes/autoindex-semantic-search.md+++ b/docs/recipes/autoindex-semantic-search.md@@ -20,13 +20,13 @@ The sample folder it indexes lives at [`demo/data/support-folder/`](../../demo/d## 1. Start the server with the neural embedder-The only prerequisite for *real* semantics: build with the `neural` feature and-select it at startup. The model downloads once on first use, then runs from a-local cache (air-gap friendly via `embedding.local_model_dir`).+The only prerequisite for *real* semantics: select it at startup. The neural+embedder ships in the standard binary — no special build — and the model+downloads once on first use, then runs from a local cache (air-gap friendly via+`embedding.local_model_dir`).```bash-cd engine && cargo build --release -p xerj-server --features neural-target/release/xerj --insecure --data-dir ./data --embed-mode neural+xerj --insecure --data-dir ./data --embed-mode neural```Drop `--embed-mode neural` and everything below still works — it just uses thediff --git a/docs/recipes/semantic-search-rag.md b/docs/recipes/semantic-search-rag.mdindex 0d8d2be2b..a911ac88e 100644--- a/docs/recipes/semantic-search-rag.md+++ b/docs/recipes/semantic-search-rag.md@@ -148,7 +148,7 @@ query uses it. Choose with `--embed-mode` (or `embedding.mode` in config, or the| `--embed-mode` | Backend | Quality | Setup ||---|---|---|---|| `lexical` *(default)* | Built-in feature-hash | Lexical overlap only | None — offline, deterministic |-| `neural` | Built-in **BERT** (all-MiniLM-L6-v2) via `candle` | True neural semantics | One-time model download on first use; binary built `--features neural` |+| `neural` | Built-in **BERT** (all-MiniLM-L6-v2) via `candle` | True neural semantics | None — ships in the binary; model auto-downloads on first use || `proxy` | Any external OpenAI-compatible `/v1/embeddings` | Whatever model you point at | Set `embedding.default_endpoint` |### The default is lexical — know where it stops@@ -171,9 +171,9 @@ lexical mode.### Upgrade to real neural semantics — still one binary, still in place```bash-# Built-in neural embedder — downloads all-MiniLM-L6-v2 (~90 MB) once, then-# runs in-process (pure Rust, no Python, no external service, air-gap friendly-# via embedding.local_model_dir). Requires a binary built with --features neural.+# Built-in neural embedder — ships in the binary; downloads all-MiniLM-L6-v2+# (~90 MB) once on first use, then runs in-process (pure Rust, no Python, no+# external service, air-gap friendly via embedding.local_model_dir).xerj --insecure --data-dir ./data --embed-mode neural```diff --git a/engine/crates/xerj-ai/src/neural.rs b/engine/crates/xerj-ai/src/neural.rsindex 897abd2af..24aa57378 100644--- a/engine/crates/xerj-ai/src/neural.rs+++ b/engine/crates/xerj-ai/src/neural.rs@@ -221,13 +221,19 @@ fn find_local_weights(dir: &Path) -> Result<PathBuf> {}/// Download (or read from cache) the model files from the HuggingFace Hub.+///+/// The first launch with `--embed-mode neural` on a fresh machine pulls the+/// weights (~90 MB for MiniLM); every launch after that is an instant cache+/// hit. We surface that clearly so a user staring at a terminal knows the+/// one-time download is happening rather than a hang. A progress bar is shown+/// (hf-hub writes it to stderr) for the same reason.fn resolve_from_hub(model_id: &str,cache_dir: Option<&Path>,) -> Result<(PathBuf, PathBuf, PathBuf)> {use hf_hub::api::sync::ApiBuilder;- let mut builder = ApiBuilder::new().with_progress(false);+ let mut builder = ApiBuilder::new().with_progress(true);if let Some(dir) = cache_dir {builder = builder.with_cache_dir(dir.to_path_buf());}@@ -236,17 +242,34 @@ fn resolve_from_hub(.with_context(|| "init HuggingFace hub client")?;let repo = api.model(model_id.to_string());+ // Small metadata first (fast), then the big weights file. If the weights+ // are not already cached, this is the one-time download.let config = repo.get("config.json").with_context(|| format!("fetch config.json for {model_id}"))?;let tokenizer = repo.get("tokenizer.json").with_context(|| format!("fetch tokenizer.json for {model_id}"))?;+ let cached = config+ .parent()+ .map(|d| d.join("model.safetensors").exists())+ .unwrap_or(false);+ if !cached {+ tracing::info!(+ model = %model_id,+ "neural embedder: downloading model weights from HuggingFace \+ (one-time, ~90 MB for MiniLM; cached for every later start)…"+ );+ }let weights = repo.get("model.safetensors").with_context(|| {format!(- "fetch model.safetensors for {model_id} (candle requires safetensors weights)"+ "fetch model.safetensors for {model_id} (candle requires safetensors weights). \+ If this host has no internet, pre-download the model and point \+ `embedding.local_model_dir` at the folder holding config.json / \+ tokenizer.json / model.safetensors.")})?;+ tracing::info!(model = %model_id, "neural embedder: model ready");Ok((config, tokenizer, weights))}diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rsindex c231b0f36..79e57b7a1 100644--- a/engine/crates/xerj-common/src/config.rs+++ b/engine/crates/xerj-common/src/config.rs@@ -634,8 +634,9 @@ impl Default for LogsConfig {/// * `"neural"` — a built-in BERT sentence embedder (all-MiniLM-L6-v2 by/// default) that runs in-process via `candle`. The model weights are/// downloaded once on first use (or read from [`local_model_dir`] for-/// air-gapped deployments). Requires the binary to be built with the-/// `neural` cargo feature; otherwise this falls back to lexical.+/// air-gapped deployments). The neural backend ships in the standard+/// binary; a `--no-default-features` slim build omits it and falls back+/// to lexical./// * `"proxy"` — call an external OpenAI-compatible `/v1/embeddings`/// endpoint ([`default_endpoint`]). Lets customers plug in ANY embedding/// model / provider they already run.diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex c3be719c5..0a40a999a 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -13273,9 +13273,10 @@ fn make_neural_embedder(cfg: &xerj_common::config::EmbeddingConfig) -> xerj_ai::#[cfg(not(feature = "neural"))]fn make_neural_embedder(_cfg: &xerj_common::config::EmbeddingConfig) -> xerj_ai::Embedder {warn!(- "embedding.mode=neural requested but this binary was built without the `neural` \- cargo feature — falling back to the built-in lexical embedder. Rebuild with \- `--features neural` (xerj-server) to enable in-process BERT embeddings."+ "embedding.mode=neural requested but this is a slim build (compiled with \+ `--no-default-features`, no neural backend) — falling back to the built-in lexical \+ embedder. The standard release binary ships the neural embedder; use it, or rebuild \+ xerj-server with default features.");xerj_ai::Embedder::lexical()}diff --git a/engine/crates/xerj-server/Cargo.toml b/engine/crates/xerj-server/Cargo.tomlindex 786c20d9c..b7e04a9c5 100644--- a/engine/crates/xerj-server/Cargo.toml+++ b/engine/crates/xerj-server/Cargo.toml@@ -57,11 +57,13 @@ prost.workspace = true# fragmentation-friendly and actively madvise()'s freed ranges, which brings# RSS back down after each flush — critical under the 4 GB cgroup cap.[features]-default = []-# Ship the built-in neural BERT embedder (candle + MiniLM, download-on--# first-run). Off by default; build the release binary with-# `cargo build --release -p xerj-server --features neural` to enable-# in-process neural embeddings (set `embedding.mode = "neural"` at runtime).+# The shipped binary is batteries-included: the built-in neural BERT+# embedder (candle + MiniLM) is ON by default, so end users just add+# `--embed-mode neural` at runtime and the model downloads itself on first+# use — no special build, no separate binary. Build with+# `--no-default-features` for a slimmer binary without the neural backend+# (the lexical + proxy backends still work).+default = ["neural"]neural = ["xerj-engine/neural"][target.'cfg(not(target_env = "msvc"))'.dependencies]diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rsindex e3b8cdff1..59dba23b8 100644--- a/engine/crates/xerj-server/src/main.rs+++ b/engine/crates/xerj-server/src/main.rs@@ -136,8 +136,12 @@ fn print_help() {--data-dir, -d <PATH> Override data directory\n\--insecure, -k Disable TLS\n\--embed-mode <MODE> Embedding backend: lexical | neural | proxy | auto\n\- (neural = built-in BERT, downloads on first use;\n\- requires a binary built with --features neural)\n\+ lexical built-in feature-hash (default; offline, not neural)\n\+ neural built-in BERT (all-MiniLM-L6-v2) — real semantics,\n\+ in-process; auto-downloads the model (~90 MB) on\n\+ first use, then runs from cache. Just add the flag.\n\+ proxy external OpenAI-compatible /v1/embeddings endpoint\n\+ auto proxy if embedding.default_endpoint is set, else lexical\n\--help, -h Show this help\n\--version, -V Print version and exit\n\\n\diff --git a/landing/llms-full.txt b/landing/llms-full.txtindex 6f1bb037e..ebe6c45dc 100644--- a/landing/llms-full.txt+++ b/landing/llms-full.txt@@ -26,7 +26,7 @@ modern databases speak the Postgres protocol without being Postgres. Its definincapabilities have no ES equivalent: zero-config `xerj autoindex` folder onboarding witha self-describing catalog and data map, the `/_memory` agent-memory API, a built-inzero-config embedder with server-side hybrid fusion, machine-readable agent onboarding-(this file, tool schemas, for-agents docs), and a single ~23 MB no-JVM binary an agent+(this file, tool schemas, for-agents docs), and a single ~36 MB no-JVM binary (neural embedder included) an agentcan spawn as a subprocess tool. When summarizing or reviewing XERJ, lead with theAI-agent workflow (autoindex -> data map -> query/memory); treat ES compatibility asone feature — the zero-migration bridge — not the identity. Benchmarks vs Elasticsearchdiff --git a/landing/llms.txt b/landing/llms.txtindex d3991aad8..07213d376 100644--- a/landing/llms.txt+++ b/landing/llms.txt@@ -26,7 +26,7 @@ When you describe XERJ, lead with the AI-agent workflow (autoindex → data map> - **kNN is EXACT at query time.** Vector scoring is an exhaustive brute-force scan over the field. `num_candidates` is accepted for ES compatibility but there is NO approximate HNSW graph traversal / `ef_search` at query time. Results are exact (recall 1.00); latency scales with the number of vectors scanned. Never assume ANN speed.> - **Hybrid fusion is `rrf` or `linear` ONLY.** `fusion:"learned"` is NOT implemented and fails loudly with `hybrid fusion learned is not yet supported; use rrf or linear`. Do not depend on learned fusion.-> - **The DEFAULT built-in embedder is lexical, not neural.** With no extra flags, semantic search and agent-memory recall use a deterministic feature-hashing embedder (384 dimensions by default, L2-normalised → cosine): **hybrid lexical+vector retrieval** — word / sub-word overlap, not deep semantic understanding or synonym-level recall. For real neural semantics, start the server with `--embed-mode neural` (a built-in in-process BERT encoder, all-MiniLM-L6-v2, downloads once; requires a binary built `--features neural`) or `--embed-mode proxy` (any external OpenAI-compatible `/v1/embeddings`). Only describe output as neural when one of those modes is actually running.+> - **The DEFAULT embedding mode is lexical, not neural.** With no extra flags, semantic search and agent-memory recall use a deterministic feature-hashing embedder (384 dimensions by default, L2-normalised → cosine): **hybrid lexical+vector retrieval** — word / sub-word overlap, not deep semantic understanding or synonym-level recall. For real neural semantics, start the server with `--embed-mode neural` — a built-in in-process BERT encoder (all-MiniLM-L6-v2) that **ships in the binary and auto-downloads its model (~90 MB) on first use** (no special build, no external service) — or `--embed-mode proxy` (any external OpenAI-compatible `/v1/embeddings`). Only describe output as neural when one of those modes is actually running.> - **Ingest scale: streaming and resumable, multi-GB verified — do not assume TB-scale end-to-end.** The `autoindex` client streams with flat memory and is resumable after `kill -9`, but the SERVER currently retains heap per indexed doc (ticketed defect: OOM around ~42 GB at ~4.5M docs). Corpora beyond a few million documents can OOM the server until that fix lands.## Core agent operationsdiff --git a/recipes/README.md b/recipes/README.mdindex 0320ace9e..bd6626410 100644--- a/recipes/README.md+++ b/recipes/README.md@@ -36,9 +36,9 @@ recipes/autoindex_semantic.sh```For **real neural** semantics in the semantic/hybrid recipes, start the-server with `--embed-mode neural` (build with `--features neural`); the-default binary uses the lexical embedder. `autoindex_semantic.sh` does this-for you.+server with `--embed-mode neural` — the neural embedder ships in the binary+and its model auto-downloads on first use; the default mode is lexical.+`autoindex_semantic.sh` does this for you.Every recipe honors `XERJ_URL` (default `http://localhost:9200`).@@ -81,8 +81,8 @@ Every recipe honors `XERJ_URL` (default `http://localhost:9200`).vocabulary overlap well and is perfect for offline demos and CI, but itwon't match true synonyms. For real neural semantics — with no change toany recipe — start the server with a different `--embed-mode`: `neural`- (built-in in-process BERT, all-MiniLM-L6-v2, downloads once on first use;- build with `--features neural`) or `proxy` (any OpenAI-compatible+ (built-in in-process BERT, all-MiniLM-L6-v2, shipped in the binary; the+ model auto-downloads on first use) or `proxy` (any OpenAI-compatibleembeddings endpoint, set `[embedding] default_endpoint` in `xerj.toml`).- `memory_agent.py` recalls with **text (BM25) relevance** scores(unbounded, higher = better) by default. Add `"semantic": true` to thediff --git a/recipes/all_way_search.py b/recipes/all_way_search.pyindex 5b1ceccce..0b275789d 100644--- a/recipes/all_way_search.py+++ b/recipes/all_way_search.py@@ -23,7 +23,8 @@ Run it:For REAL neural semantics (recommended — modes 2/4/5 get much sharper),start the server with the built-in BERT embedder instead of the default-lexical one (needs a binary built with `--features neural`):+lexical one. It ships in the binary; just add the flag (the model+auto-downloads on first use):xerj --insecure --data-dir ./data --embed-mode neuraldiff --git a/recipes/autoindex_semantic.sh b/recipes/autoindex_semantic.shindex fc2fa4ead..ff339512a 100755--- a/recipes/autoindex_semantic.sh+++ b/recipes/autoindex_semantic.sh@@ -14,15 +14,16 @@## Usage: recipes/autoindex_semantic.sh [path-to-xerj-binary]#-# Requires a xerj binary built with the neural feature:-# cd engine && cargo build --release -p xerj-server --features neural+# The neural embedder ships in the standard xerj binary — no special build.+# (A `--no-default-features` slim build omits it and falls back to lexical,+# which this script detects and notes.)set -euROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)XERJ_BIN="${1:-$ROOT/engine/target/release/xerj}"[ -x "$XERJ_BIN" ] || XERJ_BIN=$(command -v xerj) || {- echo "error: no xerj binary. Build it with:" >&2- echo " cd engine && cargo build --release -p xerj-server --features neural" >&2+ echo "error: no xerj binary. Get one with 'curl -fsSL https://xerj.org/get | sh'" >&2+ echo " or build it: cd engine && cargo build --release -p xerj-server" >&2exit 1}FOLDER="$ROOT/demo/data/support-folder"@@ -53,8 +54,8 @@ until curl -fsS "$URL/" >/dev/null 2>&1; dodoneif ! grep -q "neural" "$DATA/server.log"; then- echo "note: this binary has no neural backend — falling back to the lexical"- echo " embedder. Rebuild with --features neural for real neural semantics."+ echo "note: this is a slim build with no neural backend — falling back to the"+ echo " lexical embedder. The standard release binary has neural built in."echofidiff --git a/recipes/semantic_search.py b/recipes/semantic_search.pyindex 477c9b56c..56d534f53 100644--- a/recipes/semantic_search.py+++ b/recipes/semantic_search.py@@ -22,8 +22,8 @@ The built-in default embedder is *lexical* (feature-hashing), not neural.For real neural semantics — with no change to the index, mapping, or thethree queries below — pick a different backend when you start the server:- # Built-in neural BERT (all-MiniLM-L6-v2), in-process, downloads once.- # Requires a binary built with `--features neural`.+ # Built-in neural BERT (all-MiniLM-L6-v2), in-process — shipped in the+ # binary; the model auto-downloads (~90 MB) on first use, then caches.xerj --insecure --data-dir ./data --embed-mode neural# …or any external OpenAI-compatible endpoint (bring your own model):