Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(ai): ship the neural embedder in the single default binary

xerj-orgecf4c2e64a02
diff --git a/AGENTS.md b/AGENTS.md
index 377e29a08..d3a83dfd8 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -6,7 +6,7 @@
## Ground rules for working in this repo
-- **Honest claims only.** Every public number traces to a verified run. The **default** embedder is *lexical* feature-hashing — never claim the default binary does neural/semantic understanding. A built-in **neural** BERT embedder exists but is **opt-in** (`--features neural` + `--embed-mode neural`, model downloads on first use); only describe output as neural when that mode is actually running. kNN is *exact* brute-force, recall 1.00 (never imply ANN speed); benchmark headline is the audited scorecard in `demo/playbooks/FULL_MATRIX_SCORECARD_*.md` (currently 42W/28L/12T vs live ES 8.13.4); do not claim TB-scale end-to-end (server heap ticket: `demo/usecases/autoindex/scale/TICKET_server-unbounded-ingest-heap.md`).
+- **Honest claims only.** Every public number traces to a verified run. The **default** embedding mode is *lexical* feature-hashing — never claim the default mode does neural/semantic understanding. A built-in **neural** BERT embedder ships in the binary but is **off unless activated** (`--embed-mode neural`, model auto-downloads on first use); only describe output as neural when that mode is actually running. kNN is *exact* brute-force, recall 1.00 (never imply ANN speed); benchmark headline is the audited scorecard in `demo/playbooks/FULL_MATRIX_SCORECARD_*.md` (currently 42W/28L/12T vs live ES 8.13.4); do not claim TB-scale end-to-end (server heap ticket: `demo/usecases/autoindex/scale/TICKET_server-unbounded-ingest-heap.md`).
- **Builds are scoped:** `cd engine && cargo build --release -j 32 -p <crate>` — never workspace-wide, never `cargo clean`.
- **The hard gate:** the ES-YAML conformance suite must stay at 1360 passed / 0 failed / 3 skipped before any engine change lands (see "Running the conformance tests" in the README).
- **Git discipline:** non-trivial changes land with full commit bodies (motivation, before/after numbers, root cause, file pointers) — the git history is the project's engineering log; read it before re-deriving decisions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f75bd8a19..5e69f7156 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -16,12 +16,14 @@ retrieval recipes.
### Added
-- **Built-in neural BERT embedder (`xerj-ai`, feature `neural`).** A pure-Rust
- sentence encoder via `candle` (default `all-MiniLM-L6-v2`, 384-dim) that runs
- in-process and downloads its weights once on first use (or reads them from
- `embedding.local_model_dir` for air-gapped deployments). Off by default so the
- standard binary stays ~23 MB and dependency-light; build with
- `cargo build --release -p xerj-server --features neural` to include it.
+- **Built-in neural BERT embedder — shipped in the binary.** A pure-Rust sentence
+ encoder via `candle` (default `all-MiniLM-L6-v2`, 384-dim) that runs in-process
+ and **downloads its weights (~90 MB) automatically on first use** (or reads them
+ from `embedding.local_model_dir` for air-gapped deployments). It is compiled into
+ the default release binary — end users just add `--embed-mode neural` at runtime,
+ no special build and no separate binary. A progress bar and one-time-download log
+ make the first run legible. The binary is ~36 MB as a result; a
+ `--no-default-features` slim build without the neural backend is ~23 MB.
- **Unified three-backend embedding handle (`xerj_ai::Embedder`).** `semantic_text`
ingest and `semantic`/`hybrid` queries run through one of three interchangeable
backends — **lexical** (default, zero-dep feature-hash), **neural** (built-in
diff --git a/README.md b/README.md
index 3faa3b850..178c73fe2 100644
--- a/README.md
+++ b/README.md
@@ -43,8 +43,8 @@ What is actually designed differently — for AI-agent workloads specifically, w
- **A data map built for model orientation** — `xerj autoindex map` and the catalog index answer an agent's real first question, *"what is in here?"*, with counts, types, example values, ready-to-send queries, and known gotchas. Orientation designed for LLMs, not for humans with dashboards.
- **Agent memory as a first-class API** — `/_memory/{ns}` store/recall with namespace isolation, semantic recall, dedup, metadata filters, and recency blending. Not a plugin; part of the engine.
- **Machine-readable onboarding** — [llms.txt](https://xerj.org/llms.txt), agent tool schemas (OpenAI / MCP / Anthropic), [for-agents docs](https://xerj.org/for-agents.html). The documentation treats an LLM as a first-class reader, including honest caveat blocks it can rely on.
-- **Batteries-included retrieval** — a built-in zero-config embedder (lexical feature-hashing by default, documented honestly as such; an optional in-process **neural** BERT embedder and any external embeddings endpoint are drop-in alternatives) plus server-side hybrid RRF/linear fusion: BM25 + vector + hybrid out of the box, no external inference service required.
-- **A single ~23 MB static binary, no JVM, sub-second start** — small enough that an agent can spawn a search engine as a subprocess tool, on any of 8 released targets.
+- **Batteries-included retrieval** — a built-in zero-config embedder (lexical feature-hashing by default, documented honestly as such) plus a real in-process **neural** BERT embedder shipped in the same binary (`--embed-mode neural`, model auto-downloads on first use) and any external embeddings endpoint as a third option — all behind server-side hybrid RRF/linear fusion: BM25 + vector + hybrid out of the box, no external inference service required.
+- **A single ~36 MB static binary, no JVM, sub-second start** — batteries included: the neural embedder is built in (activate with `--embed-mode neural`; a `--no-default-features` slim build is ~23 MB). Small enough that an agent can spawn a search engine as a subprocess tool, on any of 8 released targets.
- **Exact kNN with recall 1.00 by construction** — the trade-off (latency scales with vectors scanned) is documented instead of hidden behind ANN defaults.
- **Engineering-first internals** — columnar aggregation fast paths, doc-value prefilters, WAL-shard-pinned delete durability (verified by adversarial crash matrices), Rust memory safety. The head-to-head wins vs Elasticsearch concentrate exactly where agents live: aggregations, ingest throughput, and disk footprint.
@@ -104,7 +104,7 @@ We ran the same model (headless Claude Code) against the 518 MB corpus twice: on
- **Drop-in ES wire compatibility.** Implements the Elasticsearch REST surface — index/document APIs, `_search`, `_bulk`, aggregations, kNN, scroll, aliases, templates. Validated against **1,360 of 1,363** ES-YAML conformance cases extracted from Elasticsearch 8.13 source.
- **Truly open.** Apache-2.0 licensed. No SSPL, no source-available asterisks, no per-feature license gates.
- **One engine, four workloads.** Full-text BM25, exact dense-vector kNN (100% recall), the standard aggregation suite, and columnar log analytics — all in the same process, over the same wire protocol.
-- **A single native binary.** Roughly a ~23 MB statically-linked executable. Sub-second start, no JVM, no heap-tuning ritual.
+- **A single native binary.** Roughly a ~36 MB statically-linked executable with the neural embedder built in (~23 MB as a `--no-default-features` slim build). Sub-second start, no JVM, no heap-tuning ritual.
- **Honest, reproducible benchmarks — as supporting evidence.** Measured head-to-head against a live Elasticsearch 8.13.4, the latest closed-loop, cache-off matrix scores **42 wins / 28 losses / 12 ties** for XERJ. Results — wins *and* losses — are published at [xerj.org/benchmarks](https://xerj.org/benchmarks) and reproducible from the scripts in this repo (see [Benchmarks](#benchmarks)).
- **Written in Rust.** Memory-safe, `panic = "abort"`, fat-LTO release builds. Embedded Raft consensus (no external Raft dependency) for cluster metadata.
@@ -222,7 +222,7 @@ XERJ implements the Elasticsearch REST wire protocol. Because it is wire-compati
`match_all`, `match_none`, `match`, `match_phrase`, `match_phrase_prefix`, `multi_match`, `term`, `terms`, `range`, `prefix`, `wildcard`, `exists`, `ids`, `bool`, `fuzzy`, `regexp`, `query_string`, `simple_query_string`, `constant_score`, `boosting`, `dis_max`, `geo_distance`, `knn`, `semantic`, `hybrid`.
-> **Vector & semantic search notes.** `knn` (exact dense-vector search — brute-force scan, 100% recall; no ANN index yet) and `hybrid` (BM25 + kNN in one request) run entirely inside XERJ. As of **rc-2**, `semantic_text` fields **auto-embed on ingest** and the `semantic` query works with no external service. XERJ has **three interchangeable embedding backends**, selected with `--embed-mode` (or `embedding.mode`): **`lexical`** (default) — the built-in, zero-config feature-hash embedder (lexical, *not* neural); **`neural`** — a built-in **BERT sentence embedder** (all-MiniLM-L6-v2) that runs in-process via `candle`, downloading the model once on first use (build with `--features neural`); and **`proxy`** — any external OpenAI-compatible `/v1/embeddings` endpoint (`embedding.default_endpoint`). The default binary ships the lexical embedder to stay ~23 MB and dependency-free; see the [roadmap](#roadmap). rc-2 also adds an **agent-memory REST API** (`/_memory/{ns}`) and **anomaly detection** (`/_ml/anomaly_detectors`).
+> **Vector & semantic search notes.** `knn` (exact dense-vector search — brute-force scan, 100% recall; no ANN index yet) and `hybrid` (BM25 + kNN in one request) run entirely inside XERJ. `semantic_text` fields **auto-embed on ingest** and the `semantic` query works with no external service. XERJ has **three interchangeable embedding backends**, selected with `--embed-mode` (or `embedding.mode`): **`lexical`** (default) — the built-in, zero-config feature-hash embedder (lexical, *not* neural); **`neural`** — a built-in **BERT sentence embedder** (all-MiniLM-L6-v2) that runs in-process via `candle`; and **`proxy`** — any external OpenAI-compatible `/v1/embeddings` endpoint (`embedding.default_endpoint`). **The neural embedder ships in the binary** — just start the server with `--embed-mode neural` and the model (~90 MB) downloads itself on first use, then runs from cache; no separate build, no external service, air-gap friendly (`embedding.local_model_dir`). rc-2 also adds an **agent-memory REST API** (`/_memory/{ns}`) and **anomaly detection** (`/_ml/anomaly_detectors`).
**Supported aggregations**
@@ -254,7 +254,7 @@ XERJ is a Cargo workspace under [`engine/`](./engine). The crates:
| `xerj-fts` | Full-text search: BM25 scoring, analyzer registry, postings lists. |
| `xerj-vector` | Dense-vector index for kNN / semantic search (exact brute-force scan, 100% recall). |
| `xerj-logs` | Columnar log ingestion and retention. |
-| `xerj-ai` | Text chunking and the three embedding backends behind one `Embedder` handle — the built-in lexical embedder, an optional in-process **neural** BERT embedder (`candle` + MiniLM, feature `neural`), and an embedding proxy (external OpenAI-compatible API) — all powering the `semantic` query. The memory store is exposed over the REST API at `/_memory/{ns}` (store / `_recall` / list / forget), shipped in rc-2 — see the [roadmap](#roadmap). |
+| `xerj-ai` | Text chunking and the three embedding backends behind one `Embedder` handle — the built-in lexical embedder, a built-in in-process **neural** BERT embedder (`candle` + MiniLM, shipped in the binary), and an embedding proxy (external OpenAI-compatible API) — all powering the `semantic` query. The memory store is exposed over the REST API at `/_memory/{ns}` (store / `_recall` / list / forget), shipped in rc-2 — see the [roadmap](#roadmap). |
| `xerj-compress` | Block compression codecs (LZ4, Zstd). |
| `xerj-common` | Shared types: `Config`, `Schema`, `FieldType`, `XerjError`. |
| `xerj-cluster` | Embedded Raft consensus for cluster metadata (no external Raft dependency). |
@@ -314,13 +314,13 @@ If a test expects a response and XERJ returns something different, that's a bug
XERJ ships full-text, aggregation, log-analytics, dense-vector kNN, and hybrid search. **rc-2** added three AI-adjacent capabilities (each conformance-gated and verified by real requests):
-- **Auto-embed on ingest + three embedding backends** — `semantic_text` works with zero external config on the default **lexical** embedder; opt into a built-in **neural** BERT embedder (`--embed-mode neural`, `--features neural`, model downloads on first use) or an external **proxy** (`--embed-mode proxy`) with no change to your mappings or queries.
+- **Auto-embed on ingest + three embedding backends** — `semantic_text` works with zero external config on the default **lexical** embedder; switch to the **neural** BERT embedder built into the binary (`--embed-mode neural`, model auto-downloads on first use) or an external **proxy** (`--embed-mode proxy`) with no change to your mappings or queries.
- **Ingest-time chunk-embedding** — long `semantic_text` values embed **per overlapping passage**, and `semantic` search scores by the best-matching passage (max-sim), so a long document is retrievable on any one of its sections.
- **Agent-memory REST API** — `POST /_memory/{ns}` store, `/_recall` (kNN or BM25 + metadata filter), list/forget/drop; namespaced and offline.
- **Anomaly detection (`_ml`)** — real statistical detectors with baseline + deviation scoring, on-demand **and** via continuous `_ml/datafeeds` (a background job re-scores a live index on a timer; forecasting is still future work).
- **Vector quantization** — opt a `dense_vector` field into **scalar8** (`int8_hnsw`) for ~4× smaller vectors at recall@10 ≈ 0.99.
-Still planned: shipping the neural embedder in the default release binary (it's opt-in behind `--features neural` today to keep the default ~23 MB and dependency-free) and distributed-clustering hardening. See [ROADMAP.md](./ROADMAP.md) for the full status and honest limitations.
+The built-in neural embedder now **ships in the default release binary** (activate with `--embed-mode neural`; model auto-downloads on first use). Still planned: distributed-clustering hardening and a larger/higher-quality default model option. See [ROADMAP.md](./ROADMAP.md) for the full status and honest limitations.
---
diff --git a/ROADMAP.md b/ROADMAP.md
index be85863e8..8fa849974 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -26,7 +26,7 @@ These three shipped in rc-2 (each conformance-gated at 1,326 / 1,329 and verifie
`semantic_text` now works end to end with **zero external configuration**. Indexing a document into a `semantic_text` field auto-embeds its text (previously returned `405`), and the `semantic` query embeds the query text with the same embedder and runs kNN — no external service required. Live-verified: a `semantic_text` doc indexed with no embedder configured returned `201`, and a `semantic` query ranked the intended doc first. A configured external `/v1/embeddings` proxy is still used, at higher quality, when `embedding.default_endpoint` is set.
-- **Limitation:** the **default** built-in embedder is a deterministic **lexical** model (feature-hashed word unigrams + character trigrams, L2-normalized) — it captures vocabulary/sub-word overlap, not deep semantics. This is observable live: a vocabulary-sharing query out-scored a true paraphrase. Paraphrases that share vocabulary rank correctly; truly-synonymous text with no word overlap will not. For real neural semantics you now have two drop-in upgrades with no mapping/query change: the built-in **neural** BERT embedder (`--embed-mode neural`, opt-in behind `--features neural`, downloads all-MiniLM-L6-v2 on first use — see "Neural embeddings" below), or the external `/v1/embeddings` **proxy** (`--embed-mode proxy` + `embedding.default_endpoint`).
+- **Limitation:** the **default** embedding mode is a deterministic **lexical** model (feature-hashed word unigrams + character trigrams, L2-normalized) — it captures vocabulary/sub-word overlap, not deep semantics. This is observable live: a vocabulary-sharing query out-scored a true paraphrase. Paraphrases that share vocabulary rank correctly; truly-synonymous text with no word overlap will not. For real neural semantics you have two drop-in upgrades with no mapping/query change: the built-in **neural** BERT embedder that ships in the binary (`--embed-mode neural`, downloads all-MiniLM-L6-v2 on first use — see "Neural embeddings" below), or the external `/v1/embeddings` **proxy** (`--embed-mode proxy` + `embedding.default_endpoint`).
### 2. Agent-memory REST API ✅ (rc-2)
@@ -103,7 +103,7 @@ Counting these toward "supported query types" overstates correctness — they re
### Neural embeddings & richer ML
-- A built-in **neural** embedder has **landed** (opt-in): `--embed-mode neural` runs an in-process BERT sentence encoder (all-MiniLM-L6-v2, 384-dim) via `candle` — pure Rust, no Python, no external service, air-gap friendly (`embedding.local_model_dir`). It is behind the `neural` cargo feature so the default release binary stays ~23 MB and dependency-free. **Remaining work:** ship it in the default binary (or a separate `xerj-neural` artifact) so no rebuild is needed, and offer a larger/higher-quality default model.
+- A built-in **neural** embedder has **landed and ships in the default binary**: `--embed-mode neural` runs an in-process BERT sentence encoder (all-MiniLM-L6-v2, 384-dim) via `candle` — pure Rust, no Python, no external service, model auto-downloads on first use (air-gap friendly via `embedding.local_model_dir`). No rebuild and no separate artifact needed; the shipped binary is ~36 MB (a `--no-default-features` slim build without the neural backend is ~23 MB). **Remaining work:** offer a larger/higher-quality default model option.
- Forecasting for capacity/write-load signals (continuous `_ml` datafeeds have landed — see "Landed since rc-2"; the ingest-time per-passage chunk-embedding pipeline has also landed).
### Correctness of stubbed surface
diff --git a/docs/WHY_XERJ.md b/docs/WHY_XERJ.md
index 710cb4cd3..2570241df 100644
--- a/docs/WHY_XERJ.md
+++ b/docs/WHY_XERJ.md
@@ -25,7 +25,7 @@ The autoindex pipeline was verified adversarially: 80/81 ground-truth checks on
## Engineering positions (where XERJ deliberately differs)
-- **One static binary (~23 MB), no JVM.** Sub-second cold start. An agent can spawn its own search engine as a subprocess tool. Operationally: no heap tuning, no GC pauses.
+- **One static binary (~36 MB, neural embedder included), no JVM.** Sub-second cold start. An agent can spawn its own search engine as a subprocess tool. Operationally: no heap tuning, no GC pauses. (A `--no-default-features` slim build without the neural backend is ~23 MB.)
- **Columnar aggregation fast paths + doc-value prefilters.** The analytical questions agents actually ask (counts, cardinalities, histograms, percentiles over millions of rows) are served from columnar data, not stored-document scans. This is where XERJ *beats* Elasticsearch in the head-to-head.
- **Exact kNN, recall 1.00 by construction.** No ANN index is built or traversed; the latency trade-off is documented, not hidden behind approximate defaults. (An HNSW implementation exists in the codebase but the ES-compat query path is intentionally exact.)
- **Built-in zero-config embedder.** Lexical feature-hashing (384-dim) — documented honestly as *hybrid lexical+vector*, never claimed to be neural. Semantic-ish retrieval works offline with zero external dependencies; point `semantic_text` at an external inference endpoint when you need real embeddings.
diff --git a/docs/examples/all-way-search/all_way_search.py b/docs/examples/all-way-search/all_way_search.py
index 5b1ceccce..0b275789d 100644
--- a/docs/examples/all-way-search/all_way_search.py
+++ b/docs/examples/all-way-search/all_way_search.py
@@ -23,7 +23,8 @@ Run it:
For REAL neural semantics (recommended — modes 2/4/5 get much sharper),
start the server with the built-in BERT embedder instead of the default
-lexical one (needs a binary built with `--features neural`):
+lexical one. It ships in the binary; just add the flag (the model
+auto-downloads on first use):
xerj --insecure --data-dir ./data --embed-mode neural
diff --git a/docs/examples/autoindex-semantic-search/autoindex_semantic.sh b/docs/examples/autoindex-semantic-search/autoindex_semantic.sh
index fc2fa4ead..ff339512a 100755
--- a/docs/examples/autoindex-semantic-search/autoindex_semantic.sh
+++ b/docs/examples/autoindex-semantic-search/autoindex_semantic.sh
@@ -14,15 +14,16 @@
#
# Usage: recipes/autoindex_semantic.sh [path-to-xerj-binary]
#
-# Requires a xerj binary built with the neural feature:
-# cd engine && cargo build --release -p xerj-server --features neural
+# The neural embedder ships in the standard xerj binary — no special build.
+# (A `--no-default-features` slim build omits it and falls back to lexical,
+# which this script detects and notes.)
set -eu
ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
XERJ_BIN="${1:-$ROOT/engine/target/release/xerj}"
[ -x "$XERJ_BIN" ] || XERJ_BIN=$(command -v xerj) || {
- echo "error: no xerj binary. Build it with:" >&2
- echo " cd engine && cargo build --release -p xerj-server --features neural" >&2
+ echo "error: no xerj binary. Get one with 'curl -fsSL https://xerj.org/get | sh'" >&2
+ echo " or build it: cd engine && cargo build --release -p xerj-server" >&2
exit 1
}
FOLDER="$ROOT/demo/data/support-folder"
@@ -53,8 +54,8 @@ until curl -fsS "$URL/" >/dev/null 2>&1; do
done
if ! grep -q "neural" "$DATA/server.log"; then
- echo "note: this binary has no neural backend — falling back to the lexical"
- echo " embedder. Rebuild with --features neural for real neural semantics."
+ echo "note: this is a slim build with no neural backend — falling back to the"
+ echo " lexical embedder. The standard release binary has neural built in."
echo
fi
diff --git a/docs/recipes/all-way-search.md b/docs/recipes/all-way-search.md
index 6d736b6bf..699ea9341 100644
--- a/docs/recipes/all-way-search.md
+++ b/docs/recipes/all-way-search.md
@@ -23,7 +23,7 @@ Runnable version: [`docs/examples/all-way-search/all_way_search.py`](../examples
> BERT embedder — one flag, no change to the mapping or any query:
>
> ```bash
-> # needs a binary built with --features neural; model downloads once
+> # the neural embedder ships in the binary; the model auto-downloads on first use
> xerj --insecure --data-dir ./data --embed-mode neural
> ```
diff --git a/docs/recipes/autoindex-semantic-search.md b/docs/recipes/autoindex-semantic-search.md
index 9116afffa..dda1601d0 100644
--- a/docs/recipes/autoindex-semantic-search.md
+++ b/docs/recipes/autoindex-semantic-search.md
@@ -20,13 +20,13 @@ The sample folder it indexes lives at [`demo/data/support-folder/`](../../demo/d
## 1. Start the server with the neural embedder
-The only prerequisite for *real* semantics: build with the `neural` feature and
-select it at startup. The model downloads once on first use, then runs from a
-local cache (air-gap friendly via `embedding.local_model_dir`).
+The only prerequisite for *real* semantics: select it at startup. The neural
+embedder ships in the standard binary — no special build — and the model
+downloads once on first use, then runs from a local cache (air-gap friendly via
+`embedding.local_model_dir`).
```bash
-cd engine && cargo build --release -p xerj-server --features neural
-target/release/xerj --insecure --data-dir ./data --embed-mode neural
+xerj --insecure --data-dir ./data --embed-mode neural
```
Drop `--embed-mode neural` and everything below still works — it just uses the
diff --git a/docs/recipes/semantic-search-rag.md b/docs/recipes/semantic-search-rag.md
index 0d8d2be2b..a911ac88e 100644
--- a/docs/recipes/semantic-search-rag.md
+++ b/docs/recipes/semantic-search-rag.md
@@ -148,7 +148,7 @@ query uses it. Choose with `--embed-mode` (or `embedding.mode` in config, or the
| `--embed-mode` | Backend | Quality | Setup |
|---|---|---|---|
| `lexical` *(default)* | Built-in feature-hash | Lexical overlap only | None — offline, deterministic |
-| `neural` | Built-in **BERT** (all-MiniLM-L6-v2) via `candle` | True neural semantics | One-time model download on first use; binary built `--features neural` |
+| `neural` | Built-in **BERT** (all-MiniLM-L6-v2) via `candle` | True neural semantics | None — ships in the binary; model auto-downloads on first use |
| `proxy` | Any external OpenAI-compatible `/v1/embeddings` | Whatever model you point at | Set `embedding.default_endpoint` |
### The default is lexical — know where it stops
@@ -171,9 +171,9 @@ lexical mode.
### Upgrade to real neural semantics — still one binary, still in place
```bash
-# Built-in neural embedder — downloads all-MiniLM-L6-v2 (~90 MB) once, then
-# runs in-process (pure Rust, no Python, no external service, air-gap friendly
-# via embedding.local_model_dir). Requires a binary built with --features neural.
+# Built-in neural embedder — ships in the binary; downloads all-MiniLM-L6-v2
+# (~90 MB) once on first use, then runs in-process (pure Rust, no Python, no
+# external service, air-gap friendly via embedding.local_model_dir).
xerj --insecure --data-dir ./data --embed-mode neural
```
diff --git a/engine/crates/xerj-ai/src/neural.rs b/engine/crates/xerj-ai/src/neural.rs
index 897abd2af..24aa57378 100644
--- a/engine/crates/xerj-ai/src/neural.rs
+++ b/engine/crates/xerj-ai/src/neural.rs
@@ -221,13 +221,19 @@ fn find_local_weights(dir: &Path) -> Result<PathBuf> {
}
/// Download (or read from cache) the model files from the HuggingFace Hub.
+///
+/// The first launch with `--embed-mode neural` on a fresh machine pulls the
+/// weights (~90 MB for MiniLM); every launch after that is an instant cache
+/// hit. We surface that clearly so a user staring at a terminal knows the
+/// one-time download is happening rather than a hang. A progress bar is shown
+/// (hf-hub writes it to stderr) for the same reason.
fn resolve_from_hub(
model_id: &str,
cache_dir: Option<&Path>,
) -> Result<(PathBuf, PathBuf, PathBuf)> {
use hf_hub::api::sync::ApiBuilder;
- let mut builder = ApiBuilder::new().with_progress(false);
+ let mut builder = ApiBuilder::new().with_progress(true);
if let Some(dir) = cache_dir {
builder = builder.with_cache_dir(dir.to_path_buf());
}
@@ -236,17 +242,34 @@ fn resolve_from_hub(
.with_context(|| "init HuggingFace hub client")?;
let repo = api.model(model_id.to_string());
+ // Small metadata first (fast), then the big weights file. If the weights
+ // are not already cached, this is the one-time download.
let config = repo
.get("config.json")
.with_context(|| format!("fetch config.json for {model_id}"))?;
let tokenizer = repo
.get("tokenizer.json")
.with_context(|| format!("fetch tokenizer.json for {model_id}"))?;
+ let cached = config
+ .parent()
+ .map(|d| d.join("model.safetensors").exists())
+ .unwrap_or(false);
+ if !cached {
+ tracing::info!(
+ model = %model_id,
+ "neural embedder: downloading model weights from HuggingFace \
+ (one-time, ~90 MB for MiniLM; cached for every later start)…"
+ );
+ }
let weights = repo.get("model.safetensors").with_context(|| {
format!(
- "fetch model.safetensors for {model_id} (candle requires safetensors weights)"
+ "fetch model.safetensors for {model_id} (candle requires safetensors weights). \
+ If this host has no internet, pre-download the model and point \
+ `embedding.local_model_dir` at the folder holding config.json / \
+ tokenizer.json / model.safetensors."
)
})?;
+ tracing::info!(model = %model_id, "neural embedder: model ready");
Ok((config, tokenizer, weights))
}
diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rs
index c231b0f36..79e57b7a1 100644
--- a/engine/crates/xerj-common/src/config.rs
+++ b/engine/crates/xerj-common/src/config.rs
@@ -634,8 +634,9 @@ impl Default for LogsConfig {
/// * `"neural"` — a built-in BERT sentence embedder (all-MiniLM-L6-v2 by
/// default) that runs in-process via `candle`. The model weights are
/// downloaded once on first use (or read from [`local_model_dir`] for
-/// air-gapped deployments). Requires the binary to be built with the
-/// `neural` cargo feature; otherwise this falls back to lexical.
+/// air-gapped deployments). The neural backend ships in the standard
+/// binary; a `--no-default-features` slim build omits it and falls back
+/// to lexical.
/// * `"proxy"` — call an external OpenAI-compatible `/v1/embeddings`
/// endpoint ([`default_endpoint`]). Lets customers plug in ANY embedding
/// model / provider they already run.
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index c3be719c5..0a40a999a 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -13273,9 +13273,10 @@ fn make_neural_embedder(cfg: &xerj_common::config::EmbeddingConfig) -> xerj_ai::
#[cfg(not(feature = "neural"))]
fn make_neural_embedder(_cfg: &xerj_common::config::EmbeddingConfig) -> xerj_ai::Embedder {
warn!(
- "embedding.mode=neural requested but this binary was built without the `neural` \
- cargo feature — falling back to the built-in lexical embedder. Rebuild with \
- `--features neural` (xerj-server) to enable in-process BERT embeddings."
+ "embedding.mode=neural requested but this is a slim build (compiled with \
+ `--no-default-features`, no neural backend) — falling back to the built-in lexical \
+ embedder. The standard release binary ships the neural embedder; use it, or rebuild \
+ xerj-server with default features."
);
xerj_ai::Embedder::lexical()
}
diff --git a/engine/crates/xerj-server/Cargo.toml b/engine/crates/xerj-server/Cargo.toml
index 786c20d9c..b7e04a9c5 100644
--- a/engine/crates/xerj-server/Cargo.toml
+++ b/engine/crates/xerj-server/Cargo.toml
@@ -57,11 +57,13 @@ prost.workspace = true
# fragmentation-friendly and actively madvise()'s freed ranges, which brings
# RSS back down after each flush — critical under the 4 GB cgroup cap.
[features]
-default = []
-# Ship the built-in neural BERT embedder (candle + MiniLM, download-on-
-# first-run). Off by default; build the release binary with
-# `cargo build --release -p xerj-server --features neural` to enable
-# in-process neural embeddings (set `embedding.mode = "neural"` at runtime).
+# The shipped binary is batteries-included: the built-in neural BERT
+# embedder (candle + MiniLM) is ON by default, so end users just add
+# `--embed-mode neural` at runtime and the model downloads itself on first
+# use — no special build, no separate binary. Build with
+# `--no-default-features` for a slimmer binary without the neural backend
+# (the lexical + proxy backends still work).
+default = ["neural"]
neural = ["xerj-engine/neural"]
[target.'cfg(not(target_env = "msvc"))'.dependencies]
diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rs
index e3b8cdff1..59dba23b8 100644
--- a/engine/crates/xerj-server/src/main.rs
+++ b/engine/crates/xerj-server/src/main.rs
@@ -136,8 +136,12 @@ fn print_help() {
--data-dir, -d <PATH> Override data directory\n\
--insecure, -k Disable TLS\n\
--embed-mode <MODE> Embedding backend: lexical | neural | proxy | auto\n\
- (neural = built-in BERT, downloads on first use;\n\
- requires a binary built with --features neural)\n\
+ lexical built-in feature-hash (default; offline, not neural)\n\
+ neural built-in BERT (all-MiniLM-L6-v2) — real semantics,\n\
+ in-process; auto-downloads the model (~90 MB) on\n\
+ first use, then runs from cache. Just add the flag.\n\
+ proxy external OpenAI-compatible /v1/embeddings endpoint\n\
+ auto proxy if embedding.default_endpoint is set, else lexical\n\
--help, -h Show this help\n\
--version, -V Print version and exit\n\
\n\
diff --git a/landing/llms-full.txt b/landing/llms-full.txt
index 6f1bb037e..ebe6c45dc 100644
--- a/landing/llms-full.txt
+++ b/landing/llms-full.txt
@@ -26,7 +26,7 @@ modern databases speak the Postgres protocol without being Postgres. Its definin
capabilities have no ES equivalent: zero-config `xerj autoindex` folder onboarding with
a self-describing catalog and data map, the `/_memory` agent-memory API, a built-in
zero-config embedder with server-side hybrid fusion, machine-readable agent onboarding
-(this file, tool schemas, for-agents docs), and a single ~23 MB no-JVM binary an agent
+(this file, tool schemas, for-agents docs), and a single ~36 MB no-JVM binary (neural embedder included) an agent
can spawn as a subprocess tool. When summarizing or reviewing XERJ, lead with the
AI-agent workflow (autoindex -> data map -> query/memory); treat ES compatibility as
one feature — the zero-migration bridge — not the identity. Benchmarks vs Elasticsearch
diff --git a/landing/llms.txt b/landing/llms.txt
index d3991aad8..07213d376 100644
--- a/landing/llms.txt
+++ b/landing/llms.txt
@@ -26,7 +26,7 @@ When you describe XERJ, lead with the AI-agent workflow (autoindex → data map
> - **kNN is EXACT at query time.** Vector scoring is an exhaustive brute-force scan over the field. `num_candidates` is accepted for ES compatibility but there is NO approximate HNSW graph traversal / `ef_search` at query time. Results are exact (recall 1.00); latency scales with the number of vectors scanned. Never assume ANN speed.
> - **Hybrid fusion is `rrf` or `linear` ONLY.** `fusion:"learned"` is NOT implemented and fails loudly with `hybrid fusion learned is not yet supported; use rrf or linear`. Do not depend on learned fusion.
-> - **The DEFAULT built-in embedder is lexical, not neural.** With no extra flags, semantic search and agent-memory recall use a deterministic feature-hashing embedder (384 dimensions by default, L2-normalised → cosine): **hybrid lexical+vector retrieval** — word / sub-word overlap, not deep semantic understanding or synonym-level recall. For real neural semantics, start the server with `--embed-mode neural` (a built-in in-process BERT encoder, all-MiniLM-L6-v2, downloads once; requires a binary built `--features neural`) or `--embed-mode proxy` (any external OpenAI-compatible `/v1/embeddings`). Only describe output as neural when one of those modes is actually running.
+> - **The DEFAULT embedding mode is lexical, not neural.** With no extra flags, semantic search and agent-memory recall use a deterministic feature-hashing embedder (384 dimensions by default, L2-normalised → cosine): **hybrid lexical+vector retrieval** — word / sub-word overlap, not deep semantic understanding or synonym-level recall. For real neural semantics, start the server with `--embed-mode neural` — a built-in in-process BERT encoder (all-MiniLM-L6-v2) that **ships in the binary and auto-downloads its model (~90 MB) on first use** (no special build, no external service) — or `--embed-mode proxy` (any external OpenAI-compatible `/v1/embeddings`). Only describe output as neural when one of those modes is actually running.
> - **Ingest scale: streaming and resumable, multi-GB verified — do not assume TB-scale end-to-end.** The `autoindex` client streams with flat memory and is resumable after `kill -9`, but the SERVER currently retains heap per indexed doc (ticketed defect: OOM around ~42 GB at ~4.5M docs). Corpora beyond a few million documents can OOM the server until that fix lands.
## Core agent operations
diff --git a/recipes/README.md b/recipes/README.md
index 0320ace9e..bd6626410 100644
--- a/recipes/README.md
+++ b/recipes/README.md
@@ -36,9 +36,9 @@ recipes/autoindex_semantic.sh
```
For **real neural** semantics in the semantic/hybrid recipes, start the
-server with `--embed-mode neural` (build with `--features neural`); the
-default binary uses the lexical embedder. `autoindex_semantic.sh` does this
-for you.
+server with `--embed-mode neural` — the neural embedder ships in the binary
+and its model auto-downloads on first use; the default mode is lexical.
+`autoindex_semantic.sh` does this for you.
Every recipe honors `XERJ_URL` (default `http://localhost:9200`).
@@ -81,8 +81,8 @@ Every recipe honors `XERJ_URL` (default `http://localhost:9200`).
vocabulary overlap well and is perfect for offline demos and CI, but it
won't match true synonyms. For real neural semantics — with no change to
any recipe — start the server with a different `--embed-mode`: `neural`
- (built-in in-process BERT, all-MiniLM-L6-v2, downloads once on first use;
- build with `--features neural`) or `proxy` (any OpenAI-compatible
+ (built-in in-process BERT, all-MiniLM-L6-v2, shipped in the binary; the
+ model auto-downloads on first use) or `proxy` (any OpenAI-compatible
embeddings endpoint, set `[embedding] default_endpoint` in `xerj.toml`).
- `memory_agent.py` recalls with **text (BM25) relevance** scores
(unbounded, higher = better) by default. Add `"semantic": true` to the
diff --git a/recipes/all_way_search.py b/recipes/all_way_search.py
index 5b1ceccce..0b275789d 100644
--- a/recipes/all_way_search.py
+++ b/recipes/all_way_search.py
@@ -23,7 +23,8 @@ Run it:
For REAL neural semantics (recommended — modes 2/4/5 get much sharper),
start the server with the built-in BERT embedder instead of the default
-lexical one (needs a binary built with `--features neural`):
+lexical one. It ships in the binary; just add the flag (the model
+auto-downloads on first use):
xerj --insecure --data-dir ./data --embed-mode neural
diff --git a/recipes/autoindex_semantic.sh b/recipes/autoindex_semantic.sh
index fc2fa4ead..ff339512a 100755
--- a/recipes/autoindex_semantic.sh
+++ b/recipes/autoindex_semantic.sh
@@ -14,15 +14,16 @@
#
# Usage: recipes/autoindex_semantic.sh [path-to-xerj-binary]
#
-# Requires a xerj binary built with the neural feature:
-# cd engine && cargo build --release -p xerj-server --features neural
+# The neural embedder ships in the standard xerj binary — no special build.
+# (A `--no-default-features` slim build omits it and falls back to lexical,
+# which this script detects and notes.)
set -eu
ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
XERJ_BIN="${1:-$ROOT/engine/target/release/xerj}"
[ -x "$XERJ_BIN" ] || XERJ_BIN=$(command -v xerj) || {
- echo "error: no xerj binary. Build it with:" >&2
- echo " cd engine && cargo build --release -p xerj-server --features neural" >&2
+ echo "error: no xerj binary. Get one with 'curl -fsSL https://xerj.org/get | sh'" >&2
+ echo " or build it: cd engine && cargo build --release -p xerj-server" >&2
exit 1
}
FOLDER="$ROOT/demo/data/support-folder"
@@ -53,8 +54,8 @@ until curl -fsS "$URL/" >/dev/null 2>&1; do
done
if ! grep -q "neural" "$DATA/server.log"; then
- echo "note: this binary has no neural backend — falling back to the lexical"
- echo " embedder. Rebuild with --features neural for real neural semantics."
+ echo "note: this is a slim build with no neural backend — falling back to the"
+ echo " lexical embedder. The standard release binary has neural built in."
echo
fi
diff --git a/recipes/semantic_search.py b/recipes/semantic_search.py
index 477c9b56c..56d534f53 100644
--- a/recipes/semantic_search.py
+++ b/recipes/semantic_search.py
@@ -22,8 +22,8 @@ The built-in default embedder is *lexical* (feature-hashing), not neural.
For real neural semantics — with no change to the index, mapping, or the
three queries below — pick a different backend when you start the server:
- # Built-in neural BERT (all-MiniLM-L6-v2), in-process, downloads once.
- # Requires a binary built with `--features neural`.
+ # Built-in neural BERT (all-MiniLM-L6-v2), in-process — shipped in the
+ # binary; the model auto-downloads (~90 MB) on first use, then caches.
xerj --insecure --data-dir ./data --embed-mode neural
# …or any external OpenAI-compatible endpoint (bring your own model):