Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(ai): built-in neural BERT embedder + unified 3-backend Embedder

xerj-org8af50ccda3e2
diff --git a/AGENTS.md b/AGENTS.md
index 53e57006e..377e29a08 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -6,7 +6,7 @@
## Ground rules for working in this repo
-- **Honest claims only.** Every public number traces to a verified run. The embedder is *lexical* feature-hashing (never claim neural/semantic understanding); kNN is *exact* brute-force, recall 1.00 (never imply ANN speed); benchmark headline is the audited scorecard in `demo/playbooks/FULL_MATRIX_SCORECARD_*.md` (currently 42W/28L/12T vs live ES 8.13.4); do not claim TB-scale end-to-end (server heap ticket: `demo/usecases/autoindex/scale/TICKET_server-unbounded-ingest-heap.md`).
+- **Honest claims only.** Every public number traces to a verified run. The **default** embedder is *lexical* feature-hashing — never claim the default binary does neural/semantic understanding. A built-in **neural** BERT embedder exists but is **opt-in** (`--features neural` + `--embed-mode neural`, model downloads on first use); only describe output as neural when that mode is actually running. kNN is *exact* brute-force, recall 1.00 (never imply ANN speed); benchmark headline is the audited scorecard in `demo/playbooks/FULL_MATRIX_SCORECARD_*.md` (currently 42W/28L/12T vs live ES 8.13.4); do not claim TB-scale end-to-end (server heap ticket: `demo/usecases/autoindex/scale/TICKET_server-unbounded-ingest-heap.md`).
- **Builds are scoped:** `cd engine && cargo build --release -j 32 -p <crate>` — never workspace-wide, never `cargo clean`.
- **The hard gate:** the ES-YAML conformance suite must stay at 1360 passed / 0 failed / 3 skipped before any engine change lands (see "Running the conformance tests" in the README).
- **Git discipline:** non-trivial changes land with full commit bodies (motivation, before/after numbers, root cause, file pointers) — the git history is the project's engineering log; read it before re-deriving decisions.
diff --git a/README.md b/README.md
index e5422beec..27bd2c24f 100644
--- a/README.md
+++ b/README.md
@@ -43,7 +43,7 @@ What is actually designed differently — for AI-agent workloads specifically, w
- **A data map built for model orientation** — `xerj autoindex map` and the catalog index answer an agent's real first question, *"what is in here?"*, with counts, types, example values, ready-to-send queries, and known gotchas. Orientation designed for LLMs, not for humans with dashboards.
- **Agent memory as a first-class API** — `/_memory/{ns}` store/recall with namespace isolation, semantic recall, dedup, metadata filters, and recency blending. Not a plugin; part of the engine.
- **Machine-readable onboarding** — [llms.txt](https://xerj.org/llms.txt), agent tool schemas (OpenAI / MCP / Anthropic), [for-agents docs](https://xerj.org/for-agents.html). The documentation treats an LLM as a first-class reader, including honest caveat blocks it can rely on.
-- **Batteries-included retrieval** — a built-in zero-config embedder (lexical feature-hashing, documented honestly as such) plus server-side hybrid RRF/linear fusion: BM25 + vector + hybrid out of the box, no external inference service required.
+- **Batteries-included retrieval** — a built-in zero-config embedder (lexical feature-hashing by default, documented honestly as such; an optional in-process **neural** BERT embedder and any external embeddings endpoint are drop-in alternatives) plus server-side hybrid RRF/linear fusion: BM25 + vector + hybrid out of the box, no external inference service required.
- **A single ~23 MB static binary, no JVM, sub-second start** — small enough that an agent can spawn a search engine as a subprocess tool, on any of 8 released targets.
- **Exact kNN with recall 1.00 by construction** — the trade-off (latency scales with vectors scanned) is documented instead of hidden behind ANN defaults.
- **Engineering-first internals** — columnar aggregation fast paths, doc-value prefilters, WAL-shard-pinned delete durability (verified by adversarial crash matrices), Rust memory safety. The head-to-head wins vs Elasticsearch concentrate exactly where agents live: aggregations, ingest throughput, and disk footprint.
@@ -222,7 +222,7 @@ XERJ implements the Elasticsearch REST wire protocol. Because it is wire-compati
`match_all`, `match_none`, `match`, `match_phrase`, `match_phrase_prefix`, `multi_match`, `term`, `terms`, `range`, `prefix`, `wildcard`, `exists`, `ids`, `bool`, `fuzzy`, `regexp`, `query_string`, `simple_query_string`, `constant_score`, `boosting`, `dis_max`, `geo_distance`, `knn`, `semantic`, `hybrid`.
-> **Vector & semantic search notes.** `knn` (exact dense-vector search — brute-force scan, 100% recall; no ANN index yet) and `hybrid` (BM25 + kNN in one request) run entirely inside XERJ. As of **rc-2**, `semantic_text` fields **auto-embed on ingest** using a built-in, zero-config embedder, and the `semantic` query works with no external service; a configured external OpenAI-compatible `/v1/embeddings` endpoint (`embedding.default_endpoint`) is still used at higher quality when set. The built-in embedder is lexical (not neural) — see the [roadmap](#roadmap) and [ROADMAP.md](./ROADMAP.md) for the honest limitation and what's next. rc-2 also adds an **agent-memory REST API** (`/_memory/{ns}`) and **anomaly detection** (`/_ml/anomaly_detectors`).
+> **Vector & semantic search notes.** `knn` (exact dense-vector search — brute-force scan, 100% recall; no ANN index yet) and `hybrid` (BM25 + kNN in one request) run entirely inside XERJ. As of **rc-2**, `semantic_text` fields **auto-embed on ingest** and the `semantic` query works with no external service. XERJ has **three interchangeable embedding backends**, selected with `--embed-mode` (or `embedding.mode`): **`lexical`** (default) — the built-in, zero-config feature-hash embedder (lexical, *not* neural); **`neural`** — a built-in **BERT sentence embedder** (all-MiniLM-L6-v2) that runs in-process via `candle`, downloading the model once on first use (build with `--features neural`); and **`proxy`** — any external OpenAI-compatible `/v1/embeddings` endpoint (`embedding.default_endpoint`). The default binary ships the lexical embedder to stay ~23 MB and dependency-free; see the [roadmap](#roadmap). rc-2 also adds an **agent-memory REST API** (`/_memory/{ns}`) and **anomaly detection** (`/_ml/anomaly_detectors`).
**Supported aggregations**
@@ -254,7 +254,7 @@ XERJ is a Cargo workspace under [`engine/`](./engine). The crates:
| `xerj-fts` | Full-text search: BM25 scoring, analyzer registry, postings lists. |
| `xerj-vector` | Dense-vector index for kNN / semantic search (exact brute-force scan, 100% recall). |
| `xerj-logs` | Columnar log ingestion and retention. |
-| `xerj-ai` | Text chunking, the built-in lexical embedder, and an embedding proxy (external OpenAI-compatible API) that powers the `semantic` query. The memory store is now exposed over the REST API at `/_memory/{ns}` (store / `_recall` / list / forget), shipped in rc-2 — see the [roadmap](#roadmap). |
+| `xerj-ai` | Text chunking and the three embedding backends behind one `Embedder` handle — the built-in lexical embedder, an optional in-process **neural** BERT embedder (`candle` + MiniLM, feature `neural`), and an embedding proxy (external OpenAI-compatible API) — all powering the `semantic` query. The memory store is exposed over the REST API at `/_memory/{ns}` (store / `_recall` / list / forget), shipped in rc-2 — see the [roadmap](#roadmap). |
| `xerj-compress` | Block compression codecs (LZ4, Zstd). |
| `xerj-common` | Shared types: `Config`, `Schema`, `FieldType`, `XerjError`. |
| `xerj-cluster` | Embedded Raft consensus for cluster metadata (no external Raft dependency). |
@@ -314,13 +314,13 @@ If a test expects a response and XERJ returns something different, that's a bug
XERJ ships full-text, aggregation, log-analytics, dense-vector kNN, and hybrid search. **rc-2** added three AI-adjacent capabilities (each conformance-gated and verified by real requests):
-- **Auto-embed on ingest + a built-in embedder** — `semantic_text` works with zero external config; the built-in embedder is lexical (a bundled *neural* model is still future work).
+- **Auto-embed on ingest + three embedding backends** — `semantic_text` works with zero external config on the default **lexical** embedder; opt into a built-in **neural** BERT embedder (`--embed-mode neural`, `--features neural`, model downloads on first use) or an external **proxy** (`--embed-mode proxy`) with no change to your mappings or queries.
- **Ingest-time chunk-embedding** — long `semantic_text` values embed **per overlapping passage**, and `semantic` search scores by the best-matching passage (max-sim), so a long document is retrievable on any one of its sections.
- **Agent-memory REST API** — `POST /_memory/{ns}` store, `/_recall` (kNN or BM25 + metadata filter), list/forget/drop; namespaced and offline.
- **Anomaly detection (`_ml`)** — real statistical detectors with baseline + deviation scoring, on-demand **and** via continuous `_ml/datafeeds` (a background job re-scores a live index on a timer; forecasting is still future work).
- **Vector quantization** — opt a `dense_vector` field into **scalar8** (`int8_hnsw`) for ~4× smaller vectors at recall@10 ≈ 0.99.
-Still planned: a bundled neural embedding model and distributed-clustering hardening. See [ROADMAP.md](./ROADMAP.md) for the full status and honest limitations.
+Still planned: shipping the neural embedder in the default release binary (it's opt-in behind `--features neural` today to keep the default ~23 MB and dependency-free) and distributed-clustering hardening. See [ROADMAP.md](./ROADMAP.md) for the full status and honest limitations.
---
diff --git a/ROADMAP.md b/ROADMAP.md
index 50ee296ba..be85863e8 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -26,7 +26,7 @@ These three shipped in rc-2 (each conformance-gated at 1,326 / 1,329 and verifie
`semantic_text` now works end to end with **zero external configuration**. Indexing a document into a `semantic_text` field auto-embeds its text (previously returned `405`), and the `semantic` query embeds the query text with the same embedder and runs kNN — no external service required. Live-verified: a `semantic_text` doc indexed with no embedder configured returned `201`, and a `semantic` query ranked the intended doc first. A configured external `/v1/embeddings` proxy is still used, at higher quality, when `embedding.default_endpoint` is set.
-- **Limitation:** the built-in embedder is a deterministic **lexical** model (feature-hashed word unigrams + character trigrams, L2-normalized) — it captures vocabulary/sub-word overlap, not deep semantics. This is observable live: a vocabulary-sharing query out-scored a true paraphrase. Paraphrases that share vocabulary rank correctly; truly-synonymous text with no word overlap will not. For production-grade semantics, configure the external proxy. A bundled neural model remains future work.
+- **Limitation:** the **default** built-in embedder is a deterministic **lexical** model (feature-hashed word unigrams + character trigrams, L2-normalized) — it captures vocabulary/sub-word overlap, not deep semantics. This is observable live: a vocabulary-sharing query out-scored a true paraphrase. Paraphrases that share vocabulary rank correctly; truly-synonymous text with no word overlap will not. For real neural semantics you now have two drop-in upgrades with no mapping/query change: the built-in **neural** BERT embedder (`--embed-mode neural`, opt-in behind `--features neural`, downloads all-MiniLM-L6-v2 on first use — see "Neural embeddings" below), or the external `/v1/embeddings` **proxy** (`--embed-mode proxy` + `embedding.default_endpoint`).
### 2. Agent-memory REST API ✅ (rc-2)
@@ -62,7 +62,7 @@ The datafeed scheduler that rc-2 lacked: `PUT/GET/DELETE /_ml/datafeeds/{id}` +
Long `semantic_text` values are split into overlapping passages, embedded **per passage**, and the per-passage vectors persisted (in `<field>_vector_chunks`, only when a value spans >1 passage). A `semantic` query scores each document by its **best-matching passage** (max-sim) instead of a single pooled vector, so a long document competes on any one of its sections. Live-verified: on 40 articles + a compendium of all 40, the compendium reached top-3 for **98%** of single-topic queries with per-passage scoring vs **32%** pooled. Short single-passage values are byte-identical to before. Recipe: `recipes/passage_search.py`; guide: `docs/recipes/passage-retrieval.md`.
-- **Limitation:** the built-in embedder is still lexical (per-passage vectors are only as good as the embedder); a bundled neural model remains future work. A field that is *also* scalar8-quantized scores against the pooled vector (per-passage max-sim is exact-f32 only).
+- **Limitation:** per-passage vectors are only as good as the active embedder. The default is lexical; switch to `--embed-mode neural` (built-in BERT) or `--embed-mode proxy` for neural-quality passage vectors — the chunk-embedding pipeline is backend-agnostic. A field that is *also* scalar8-quantized scores against the pooled vector (per-passage max-sim is exact-f32 only).
## Partial / in progress
@@ -103,7 +103,7 @@ Counting these toward "supported query types" overstates correctness — they re
### Neural embeddings & richer ML
-- A bundled/local **neural** embedding model (the current built-in embedder is lexical).
+- A built-in **neural** embedder has **landed** (opt-in): `--embed-mode neural` runs an in-process BERT sentence encoder (all-MiniLM-L6-v2, 384-dim) via `candle` — pure Rust, no Python, no external service, air-gap friendly (`embedding.local_model_dir`). It is behind the `neural` cargo feature so the default release binary stays ~23 MB and dependency-free. **Remaining work:** ship it in the default binary (or a separate `xerj-neural` artifact) so no rebuild is needed, and offer a larger/higher-quality default model.
- Forecasting for capacity/write-load signals (continuous `_ml` datafeeds have landed — see "Landed since rc-2"; the ingest-time per-passage chunk-embedding pipeline has also landed).
### Correctness of stubbed surface
diff --git a/docs/recipes/semantic-search-rag.md b/docs/recipes/semantic-search-rag.md
index 65cad454c..0d8d2be2b 100644
--- a/docs/recipes/semantic-search-rag.md
+++ b/docs/recipes/semantic-search-rag.md
@@ -138,38 +138,62 @@ Q: 'get my money back after buying'
semantic top -> kb-2 Refund policy
```
-## Honest limitation: the built-in embedder is *lexical*, not neural
+## Three embedding backends: lexical (default), built-in neural, or bring-your-own
+
+The `semantic_text` mapping and the `semantic` query are **identical** across all
+three backends — you pick the backend once, at the server, and every ingest and
+query uses it. Choose with `--embed-mode` (or `embedding.mode` in config, or the
+`XERJ_EMBED_MODE` env var):
+
+| `--embed-mode` | Backend | Quality | Setup |
+|---|---|---|---|
+| `lexical` *(default)* | Built-in feature-hash | Lexical overlap only | None — offline, deterministic |
+| `neural` | Built-in **BERT** (all-MiniLM-L6-v2) via `candle` | True neural semantics | One-time model download on first use; binary built `--features neural` |
+| `proxy` | Any external OpenAI-compatible `/v1/embeddings` | Whatever model you point at | Set `embedding.default_endpoint` |
+
+### The default is lexical — know where it stops
The zero-config embedder is a **lexical** model — it maps overlapping and related
vocabulary into vector space. It shines when the question and the answer share
*some* terms (a synonym here, a rephrase there), which covers a lot of real
support/FAQ traffic. It is **not** a neural sentence embedder and won't capture
-deep semantics with no lexical bridge.
-
-Concretely, from this same KB:
+deep semantics with no lexical bridge. Concretely, from this same KB:
```
Q: 'how do I rotate my API key'
- semantic top -> kb-5 Rate limits (expected kb-1)
+ semantic top -> kb-5 Rate limits (expected kb-1) # lexical mode
```
The doc is *titled* "Rotating an API key," but only its **body** is embedded, and
-the body never uses the word "rotate" — so a pure "rotate" paraphrase misses. Two
-honest takeaways:
+the body never uses the word "rotate" — so a pure "rotate" paraphrase misses in
+lexical mode.
+
+### Upgrade to real neural semantics — still one binary, still in place
+
+```bash
+# Built-in neural embedder — downloads all-MiniLM-L6-v2 (~90 MB) once, then
+# runs in-process (pure Rust, no Python, no external service, air-gap friendly
+# via embedding.local_model_dir). Requires a binary built with --features neural.
+xerj --insecure --data-dir ./data --embed-mode neural
+```
+
+With `--embed-mode neural`, the "rotate" paraphrase above lands on `kb-1` — the
+BERT encoder captures that *rotate ≈ regenerate ≈ replace* even with no shared
+term. Measured on the standard sentence-similarity pair, the built-in neural
+embedder scores a paraphrase at cosine **0.72** vs an unrelated sentence at
+**−0.04**; the lexical embedder cannot separate them that way.
+
+Two honest takeaways that still apply in every mode:
- **Embed the text that carries the meaning.** If titles matter, index them into
the `semantic_text` field too (or add a second `semantic_text` field and combine
with `hybrid`), so the vocabulary the user might use is actually in the vector.
-- **For production-grade semantics, wire an external embeddings endpoint.** XERJ
- will call any OpenAI-compatible `/v1/embeddings` service. Set
- `embedding.default_endpoint` in config, or point a field at it in the mapping
- via `inference_endpoint` / `inference_id`. The `semantic` query and the ingest
- path are identical — you just get neural-quality vectors instead of the built-in
- lexical ones. (Anomaly-style continuous re-embedding isn't a thing here;
- embedding happens at ingest time.)
-
-Everything above is `semantic_text` + the built-in embedder — no external service
-was configured for this recipe.
+- **Bring your own model when you need a specific one.** `--embed-mode proxy` with
+ `embedding.default_endpoint` calls any OpenAI-compatible `/v1/embeddings`
+ service (OpenAI, Cohere, a local vLLM, etc.). Embedding happens at ingest time.
+
+Everything in the numbered walk-through above uses the default lexical embedder —
+no external service was configured for this recipe.
---
diff --git a/engine/Cargo.lock b/engine/Cargo.lock
index b72b8e013..d181f4788 100644
--- a/engine/Cargo.lock
+++ b/engine/Cargo.lock
@@ -15,7 +15,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75"
dependencies = [
"cfg-if",
+ "getrandom 0.3.4",
"once_cell",
+ "serde",
"version_check",
"zerocopy",
]
@@ -321,6 +323,12 @@ dependencies = [
"tower-service",
]
+[[package]]
+name = "base64"
+version = "0.13.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9e1b586273c5702936fe7b7d6896644d8be71e6314cfe09d3167c95f712589e8"
+
[[package]]
name = "base64"
version = "0.21.7"
@@ -350,6 +358,21 @@ version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3a8241f3ebb85c056b509d4327ad0358fbbba6ffb340bf388f26350aeda225b1"
+[[package]]
+name = "bit-set"
+version = "0.8.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "08807e080ed7f9d5433fa9b275196cfc35414f66a0c79d864dc51a0d825231a3"
+dependencies = [
+ "bit-vec",
+]
+
+[[package]]
+name = "bit-vec"
+version = "0.8.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5e764a1d40d510daf35e07be9eb06e75770908c27d411ee6c92109c9840eaaf7"
+
[[package]]
name = "bitflags"
version = "2.11.1"
@@ -385,6 +408,20 @@ name = "bytemuck"
version = "1.25.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c8efb64bd706a16a1bdde310ae86b351e4d21550d98d056f22f8a7f7a2183fec"
+dependencies = [
+ "bytemuck_derive",
+]
+
+[[package]]
+name = "bytemuck_derive"
+version = "1.10.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f9abbd1bc6865053c427f7198e6af43bfdedc55ab791faed4fbd361d789575ff"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn",
+]
[[package]]
name = "byteorder"
@@ -398,12 +435,79 @@ version = "1.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33"
+[[package]]
+name = "candle-core"
+version = "0.9.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c15b675b80d994b2eadb20a4bbe434eabeb454eac3ee5e2b4cf6f147ee9be091"
+dependencies = [
+ "byteorder",
+ "float8",
+ "gemm",
+ "half",
+ "libm",
+ "memmap2",
+ "num-traits",
+ "num_cpus",
+ "rand 0.9.4",
+ "rand_distr",
+ "rayon",
+ "safetensors",
+ "thiserror 2.0.18",
+ "yoke",
+ "zip 7.2.0",
+]
+
+[[package]]
+name = "candle-nn"
+version = "0.9.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3045fa9e7aef8567d209a27d56b692f60b96f4d0569f4c3011f8ca6715c65e03"
+dependencies = [
+ "candle-core",
+ "half",
+ "libc",
+ "num-traits",
+ "rayon",
+ "safetensors",
+ "serde",
+ "thiserror 2.0.18",
+]
+
+[[package]]
+name = "candle-transformers"
+version = "0.9.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b538ec4aa807c416a2ddd3621044888f188827862e2a6fcacba4738e89795d01"
+dependencies = [
+ "byteorder",
+ "candle-core",
+ "candle-nn",
+ "fancy-regex 0.17.0",
+ "num-traits",
+ "rand 0.9.4",
+ "rayon",
+ "serde",
+ "serde_json",
+ "serde_plain",
+ "tracing",
+]
+
[[package]]
name = "cast"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "37b2a672a2cb129a2e41c10b1224bb368f9f37a2b16b612598138befd7b37eb5"
+[[package]]
+name = "castaway"
+version = "0.2.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "dec551ab6e7578819132c713a93c022a05d60159dc86e7a7050223577484c55a"
+dependencies = [
+ "rustversion",
+]
+
[[package]]
name = "cc"
version = "1.2.60"
@@ -547,6 +651,34 @@ dependencies = [
"windows-sys 0.52.0",
]
+[[package]]
+name = "compact_str"
+version = "0.9.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9dfdd1c2274d9aa354115b09dc9a901d6c5576818cdf70d14cae2bdb47df00ab"
+dependencies = [
+ "castaway",
+ "cfg-if",
+ "itoa",
+ "rustversion",
+ "ryu",
+ "serde",
+ "static_assertions",
+]
+
+[[package]]
+name = "console"
+version = "0.15.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "054ccb5b10f9f2cbf51eb355ca1d05c2d279ce1804688d0db74b4733a5aeafd8"
+dependencies = [
+ "encode_unicode",
+ "libc",
+ "once_cell",
+ "unicode-width 0.2.2",
+ "windows-sys 0.59.0",
+]
+
[[package]]
name = "cookie"
version = "0.18.1"
@@ -614,7 +746,7 @@ dependencies = [
"clap",
"criterion-plot",
"is-terminal",
- "itertools",
+ "itertools 0.10.5",
"num-traits",
"once_cell",
"oorandom",
@@ -635,7 +767,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6b50826342786a51a89e2da3a28f1c32b06e387201bc2d19791f622c673706b1"
dependencies = [
"cast",
- "itertools",
+ "itertools 0.10.5",
]
[[package]]
@@ -731,6 +863,50 @@ dependencies = [
"memchr",
]
+[[package]]
+name = "darling"
+version = "0.20.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "fc7f46116c46ff9ab3eb1597a45688b6715c6e628b5c133e288e709a29bcb4ee"
+dependencies = [
+ "darling_core",
+ "darling_macro",
+]
+
+[[package]]
+name = "darling_core"
+version = "0.20.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0d00b9596d185e565c2207a0b01f8bd1a135483d02d9b7b0a54b11da8d53412e"
+dependencies = [
+ "fnv",
+ "ident_case",
+ "proc-macro2",
+ "quote",
+ "strsim",
+ "syn",
+]
+
+[[package]]
+name = "darling_macro"
+version = "0.20.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead"
+dependencies = [
+ "darling_core",
+ "quote",
+ "syn",
+]
+
+[[package]]
+name = "dary_heap"
+version = "0.3.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8b1e3a325bc115f096c8b77bbf027a7c2592230e70be2d985be950d3d5e60ebe"
+dependencies = [
+ "serde",
+]
+
[[package]]
name = "dashmap"
version = "6.1.0"
@@ -786,6 +962,37 @@ dependencies = [
"syn",
]
+[[package]]
+name = "derive_builder"
+version = "0.20.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "507dfb09ea8b7fa618fcf76e953f4f5e192547945816d5358edffe39f6f94947"
+dependencies = [
+ "derive_builder_macro",
+]
+
+[[package]]
+name = "derive_builder_core"
+version = "0.20.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2d5bcf7b024d6835cfb3d473887cd966994907effbe9227e8c8219824d06c4e8"
+dependencies = [
+ "darling",
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
+[[package]]
+name = "derive_builder_macro"
+version = "0.20.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c"
+dependencies = [
+ "derive_builder_core",
+ "syn",
+]
+
[[package]]
name = "digest"
version = "0.10.7"
@@ -797,6 +1004,27 @@ dependencies = [
"subtle",
]
+[[package]]
+name = "dirs"
+version = "6.0.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c3e8aa94d75141228480295a7d0e7feb620b1a5ad9f12bc40be62411e38cce4e"
+dependencies = [
+ "dirs-sys",
+]
+
+[[package]]
+name = "dirs-sys"
+version = "0.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e01a3366d27ee9890022452ee61b2b63a67e6f13f58900b651ff5665f0bb1fab"
+dependencies = [
+ "libc",
+ "option-ext",
+ "redox_users",
+ "windows-sys 0.61.2",
+]
+
[[package]]
name = "displaydoc"
version = "0.2.5"
@@ -808,12 +1036,34 @@ dependencies = [
"syn",
]
+[[package]]
+name = "dyn-stack"
+version = "0.13.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1c4713e43e2886ba72b8271aa66c93d722116acf7a75555cce11dcde84388fe8"
+dependencies = [
+ "bytemuck",
+ "dyn-stack-macros",
+]
+
+[[package]]
+name = "dyn-stack-macros"
+version = "0.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e1d926b4d407d372f141f93bb444696142c29d32962ccbd3531117cf3aa0bfa9"
+
[[package]]
name = "either"
version = "1.15.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719"
+[[package]]
+name = "encode_unicode"
+version = "1.0.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "34aa73646ffb006b8f5147f3dc182bd4bcb190227ce861fc4a4844bf8e3cb2c0"
+
[[package]]
name = "encoding_rs"
version = "0.8.35"
@@ -823,6 +1073,18 @@ dependencies = [
"cfg-if",
]
+[[package]]
+name = "enum-as-inner"
+version = "0.6.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a1e6a265c649f3f5979b601d26f1d05ada116434c87741c9493cb56218f76cbc"
+dependencies = [
+ "heck",
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
[[package]]
name = "equivalent"
version = "1.0.2"
@@ -853,6 +1115,12 @@ dependencies = [
"serde_yaml",
]
+[[package]]
+name = "esaxx-rs"
+version = "0.1.10"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d817e038c30374a4bcb22f94d0a8a0e216958d4c3dcde369b1439fec4bdda6e6"
+
[[package]]
name = "fallible-iterator"
version = "0.3.0"
@@ -865,6 +1133,28 @@ version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a"
+[[package]]
+name = "fancy-regex"
+version = "0.14.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6e24cb5a94bcae1e5408b0effca5cd7172ea3c5755049c5f3af4cd283a165298"
+dependencies = [
+ "bit-set",
+ "regex-automata",
+ "regex-syntax",
+]
+
+[[package]]
+name = "fancy-regex"
+version = "0.17.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "72cf461f865c862bb7dc573f643dd6a2b6842f7c30b07882b56bd148cc2761b8"
+dependencies = [
+ "bit-set",
+ "regex-automata",
+ "regex-syntax",
+]
+
[[package]]
name = "fastrand"
version = "2.4.1"
@@ -902,6 +1192,18 @@ dependencies = [
"num-traits",
]
+[[package]]
+name = "float8"
+version = "0.6.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "719a903cc23e4a89e87962c2a80fdb45cdaad0983a89bd150bb57b4c8571a7d5"
+dependencies = [
+ "half",
+ "num-traits",
+ "rand 0.9.4",
+ "rand_distr",
+]
+
[[package]]
name = "fnv"
version = "1.0.7"
@@ -914,6 +1216,12 @@ version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2"
+[[package]]
+name = "foldhash"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb"
+
[[package]]
name = "foreign-types"
version = "0.3.2"
@@ -1042,6 +1350,125 @@ dependencies = [
"slab",
]
+[[package]]
+name = "gemm"
+version = "0.19.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "aa0673db364b12263d103b68337a68fbecc541d6f6b61ba72fe438654709eacb"
+dependencies = [
+ "dyn-stack",
+ "gemm-c32",
+ "gemm-c64",
+ "gemm-common",
+ "gemm-f16",
+ "gemm-f32",
+ "gemm-f64",
+ "num-complex",
+ "num-traits",
+ "paste",
+ "raw-cpuid",
+ "seq-macro",
+]
+
+[[package]]
+name = "gemm-c32"
+version = "0.19.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "086936dbdcb99e37aad81d320f98f670e53c1e55a98bee70573e83f95beb128c"
+dependencies = [
+ "dyn-stack",
+ "gemm-common",
+ "num-complex",
+ "num-traits",
+ "paste",
+ "raw-cpuid",
+ "seq-macro",
+]
+
+[[package]]
+name = "gemm-c64"
+version = "0.19.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "20c8aeeeec425959bda4d9827664029ba1501a90a0d1e6228e48bef741db3a3f"
+dependencies = [
+ "dyn-stack",
+ "gemm-common",
+ "num-complex",
+ "num-traits",
+ "paste",
+ "raw-cpuid",
+ "seq-macro",
+]
+
+[[package]]
+name = "gemm-common"
+version = "0.19.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "88027625910cc9b1085aaaa1c4bc46bb3a36aad323452b33c25b5e4e7c8e2a3e"
+dependencies = [
+ "bytemuck",
+ "dyn-stack",
+ "half",
+ "libm",
+ "num-complex",
+ "num-traits",
+ "once_cell",
+ "paste",
+ "pulp",
+ "raw-cpuid",
+ "rayon",
+ "seq-macro",
+ "sysctl",
+]
+
+[[package]]
+name = "gemm-f16"
+version = "0.19.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e3df7a55202e6cd6739d82ae3399c8e0c7e1402859b30e4cb780e61525d9486e"
+dependencies = [
+ "dyn-stack",
+ "gemm-common",
+ "gemm-f32",
+ "half",
+ "num-complex",
+ "num-traits",
+ "paste",
+ "raw-cpuid",
+ "rayon",
+ "seq-macro",
+]
+
+[[package]]
+name = "gemm-f32"
+version = "0.19.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "02e0b8c9da1fbec6e3e3ab2ce6bc259ef18eb5f6f0d3e4edf54b75f9fd41a81c"
+dependencies = [
+ "dyn-stack",
+ "gemm-common",
+ "num-complex",
+ "num-traits",
+ "paste",
+ "raw-cpuid",
+ "seq-macro",
+]
+
+[[package]]
+name = "gemm-f64"
+version = "0.19.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "056131e8f2a521bfab322f804ccd652520c79700d81209e9d9275bbdecaadc6a"
+dependencies = [
+ "dyn-stack",
+ "gemm-common",
+ "num-complex",
+ "num-traits",
+ "paste",
+ "raw-cpuid",
+ "seq-macro",
+]
+
[[package]]
name = "generic-array"
version = "0.14.7"
@@ -1123,8 +1550,12 @@ version = "2.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b"
dependencies = [
+ "bytemuck",
"cfg-if",
"crunchy",
+ "num-traits",
+ "rand 0.9.4",
+ "rand_distr",
"zerocopy",
]
@@ -1160,7 +1591,20 @@ version = "0.15.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1"
dependencies = [
- "foldhash",
+ "foldhash 0.1.5",
+]
+
+[[package]]
+name = "hashbrown"
+version = "0.16.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100"
+dependencies = [
+ "allocator-api2",
+ "equivalent",
+ "foldhash 0.2.0",
+ "serde",
+ "serde_core",
]
[[package]]
@@ -1196,6 +1640,25 @@ version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70"
+[[package]]
+name = "hf-hub"
+version = "0.4.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "629d8f3bbeda9d148036d6b0de0a3ab947abd08ce90626327fc3547a49d59d97"
+dependencies = [
+ "dirs",
+ "http",
+ "indicatif",
+ "libc",
+ "log",
+ "rand 0.9.4",
+ "serde",
+ "serde_json",
+ "thiserror 2.0.18",
+ "ureq",
+ "windows-sys 0.60.2",
+]
+
[[package]]
name = "hmac"
version = "0.12.1"
@@ -1285,7 +1748,7 @@ dependencies = [
"tokio",
"tokio-rustls",
"tower-service",
- "webpki-roots",
+ "webpki-roots 1.0.7",
]
[[package]]
@@ -1454,6 +1917,12 @@ version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954"
+[[package]]
+name = "ident_case"
+version = "1.0.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39"
+
[[package]]
name = "idna"
version = "1.1.0"
@@ -1497,6 +1966,19 @@ dependencies = [
"serde_core",
]
+[[package]]
+name = "indicatif"
+version = "0.17.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "183b3088984b400f4cfac3620d5e076c84da5364016b4f49473de574b2586235"
+dependencies = [
+ "console",
+ "number_prefix",
+ "portable-atomic",
+ "unicode-width 0.2.2",
+ "web-time",
+]
+
[[package]]
name = "ipnet"
version = "2.12.0"
@@ -1539,6 +2021,15 @@ dependencies = [
"either",
]
+[[package]]
+name = "itertools"
+version = "0.14.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285"
+dependencies = [
+ "either",
+]
+
[[package]]
name = "itoa"
version = "1.0.18"
@@ -1585,6 +2076,21 @@ version = "0.2.185"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "52ff2c0fe9bc6cb6b14a0592c2ff4fa9ceb83eea9db979b0487cd054946a2b8f"
+[[package]]
+name = "libm"
+version = "0.2.16"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981"
+
+[[package]]
+name = "libredox"
+version = "0.1.18"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c943259e342f1e06ff2da7a83eabdfe7f92ce10262688dbf1895ff0b3e6e4652"
+dependencies = [
+ "libc",
+]
+
[[package]]
name = "libsqlite3-sys"
version = "0.30.1"
@@ -1671,6 +2177,22 @@ dependencies = [
"twox-hash",
]
+[[package]]
+name = "macro_rules_attribute"
+version = "0.2.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "65049d7923698040cd0b1ddcced9b0eb14dd22c5f86ae59c3740eab64a676520"
+dependencies = [
+ "macro_rules_attribute-proc_macro",
+ "paste",
+]
+
+[[package]]
+name = "macro_rules_attribute-proc_macro"
+version = "0.2.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "670fdfda89751bc4a84ac13eaa63e205cf0fd22b4c9a5fbfa085b63c1f1d3a30"
+
[[package]]
name = "matchers"
version = "0.2.0"
@@ -1699,6 +2221,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "714098028fe011992e1c3962653c96b2d578c4b4bce9036e15ff220319b1e0e3"
dependencies = [
"libc",
+ "stable_deref_trait",
]
[[package]]
@@ -1709,7 +2232,7 @@ checksum = "5f98efec8807c63c752b5bd61f862c165c115b0a35685bdcfd9238c7aeb592b7"
dependencies = [
"cfg-if",
"miette-derive",
- "unicode-width",
+ "unicode-width 0.1.14",
]
[[package]]
@@ -1756,6 +2279,28 @@ dependencies = [
"windows-sys 0.61.2",
]
+[[package]]
+name = "monostate"
+version = "0.1.18"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3341a273f6c9d5bef1908f17b7267bbab0e95c9bf69a0d4dcf8e9e1b2c76ef67"
+dependencies = [
+ "monostate-impl",
+ "serde",
+ "serde_core",
+]
+
+[[package]]
+name = "monostate-impl"
+version = "0.1.18"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e4db6d5580af57bf992f59068d4ea26fd518574ff48d7639b255a36f9de6e7e9"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
[[package]]
name = "multer"
version = "3.1.0"
@@ -1825,6 +2370,16 @@ dependencies = [
"num-traits",
]
+[[package]]
+name = "num-complex"
+version = "0.4.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495"
+dependencies = [
+ "bytemuck",
+ "num-traits",
+]
+
[[package]]
name = "num-conv"
version = "0.2.1"
@@ -1847,8 +2402,25 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841"
dependencies = [
"autocfg",
+ "libm",
+]
+
+[[package]]
+name = "num_cpus"
+version = "1.17.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "91df4bbde75afed763b708b7eee1e8e7651e02d97f6d5dd763e89367e957b23b"
+dependencies = [
+ "hermit-abi",
+ "libc",
]
+[[package]]
+name = "number_prefix"
+version = "0.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "830b246a0e5f20af87141b25c173cd1b609bd7779a4617d6ec582abaf90870f3"
+
[[package]]
name = "oid-registry"
version = "0.7.1"
@@ -1930,6 +2502,12 @@ dependencies = [
"vcpkg",
]
+[[package]]
+name = "option-ext"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d"
+
[[package]]
name = "parking_lot"
version = "0.12.5"
@@ -1962,6 +2540,12 @@ dependencies = [
"regex",
]
+[[package]]
+name = "paste"
+version = "1.0.15"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a"
+
[[package]]
name = "pastey"
version = "0.1.1"
@@ -2092,6 +2676,12 @@ dependencies = [
"plotters-backend",
]
+[[package]]
+name = "portable-atomic"
+version = "1.13.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49"
+
[[package]]
name = "potential_utf"
version = "0.1.5"
@@ -2167,7 +2757,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"
dependencies = [
"heck",
- "itertools",
+ "itertools 0.10.5",
"log",
"multimap",
"once_cell",
@@ -2187,7 +2777,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d"
dependencies = [
"anyhow",
- "itertools",
+ "itertools 0.10.5",
"proc-macro2",
"quote",
"syn",
@@ -2248,6 +2838,29 @@ dependencies = [
"thiserror 1.0.69",
]
+[[package]]
+name = "pulp"
+version = "0.22.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "046aa45b989642ec2e4717c8e72d677b13edd831a4d3b6cf37d9a3e54912496a"
+dependencies = [
+ "bytemuck",
+ "cfg-if",
+ "libm",
+ "num-complex",
+ "paste",
+ "pulp-wasm-simd-flag",
+ "raw-cpuid",
+ "reborrow",
+ "version_check",
+]
+
+[[package]]
+name = "pulp-wasm-simd-flag"
+version = "0.1.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1d8f70e07b9c3962945a74e59ca1c511bba65b6419468acc217c457d93f3c740"
+
[[package]]
name = "quick-xml"
version = "0.36.2"
@@ -2392,6 +3005,25 @@ dependencies = [
"getrandom 0.3.4",
]
+[[package]]
+name = "rand_distr"
+version = "0.5.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6a8615d50dcf34fa31f7ab52692afec947c4dd0ab803cc87cb3b0b4570ff7463"
+dependencies = [
+ "num-traits",
+ "rand 0.9.4",
+]
+
+[[package]]
+name = "raw-cpuid"
+version = "11.6.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "498cd0dc59d73224351ee52a95fee0f1a617a2eae0e7d9d720cc622c73a54186"
+dependencies = [
+ "bitflags",
+]
+
[[package]]
name = "rayon"
version = "1.12.0"
@@ -2402,6 +3034,17 @@ dependencies = [
"rayon-core",
]
+[[package]]
+name = "rayon-cond"
+version = "0.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2964d0cf57a3e7a06e8183d14a8b527195c706b7983549cd5462d5aa3747438f"
+dependencies = [
+ "either",
+ "itertools 0.14.0",
+ "rayon",
+]
+
[[package]]
name = "rayon-core"
version = "1.13.0"
@@ -2425,6 +3068,12 @@ dependencies = [
"yasna",
]
+[[package]]
+name = "reborrow"
+version = "0.5.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "03251193000f4bd3b042892be858ee50e8b3719f2b08e5833ac4353724632430"
+
[[package]]
name = "redox_syscall"
version = "0.5.18"
@@ -2434,6 +3083,17 @@ dependencies = [
"bitflags",
]
+[[package]]
+name = "redox_users"
+version = "0.5.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a4e608c6638b9c18977b00b475ac1f28d14e84b27d8d42f70e0bf1e3dec127ac"
+dependencies = [
+ "getrandom 0.2.17",
+ "libredox",
+ "thiserror 2.0.18",
+]
+
[[package]]
name = "ref-cast"
version = "1.0.25"
@@ -2526,7 +3186,7 @@ dependencies = [
"wasm-bindgen",
"wasm-bindgen-futures",
"web-sys",
- "webpki-roots",
+ "webpki-roots 1.0.7",
]
[[package]]
@@ -2611,6 +3271,7 @@ version = "0.23.38"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "69f9466fb2c14ea04357e91413efb882e2a6d4a406e625449bc0a5d360d53a21"
dependencies = [
+ "log",
"once_cell",
"ring",
"rustls-pki-types",
@@ -2661,6 +3322,17 @@ version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
+[[package]]
+name = "safetensors"
+version = "0.7.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "675656c1eabb620b921efea4f9199f97fc86e36dd6ffd1fbbe48d0f59a4987f5"
+dependencies = [
+ "hashbrown 0.16.1",
+ "serde",
+ "serde_json",
+]
+
[[package]]
name = "same-file"
version = "1.0.6"
@@ -2714,6 +3386,12 @@ version = "1.0.28"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd"
+[[package]]
+name = "seq-macro"
+version = "0.3.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc"
+
[[package]]
name = "serde"
version = "1.0.228"
@@ -2779,6 +3457,15 @@ dependencies = [
"serde_core",
]
+[[package]]
+name = "serde_plain"
+version = "1.0.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9ce1fc6db65a611022b23a0dec6975d63fb80a302cb3388835ff02c097258d50"
+dependencies = [
+ "serde",
+]
+
[[package]]
name = "serde_spanned"
version = "0.6.9"
@@ -2914,18 +3601,47 @@ dependencies = [
"windows-sys 0.61.2",
]
+[[package]]
+name = "socks"
+version = "0.3.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f0c3dbbd9ae980613c6dd8e28a9407b50509d3803b57624d5dfe8315218cd58b"
+dependencies = [
+ "byteorder",
+ "libc",
+ "winapi",
+]
+
[[package]]
name = "spin"
version = "0.9.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67"
+[[package]]
+name = "spm_precompiled"
+version = "0.1.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5851699c4033c63636f7ea4cf7b7c1f1bf06d0cc03cfb42e711de5a5c46cf326"
+dependencies = [
+ "base64 0.13.1",
+ "nom",
+ "serde",
+ "unicode-segmentation",
+]
+
[[package]]
name = "stable_deref_trait"
version = "1.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596"
+[[package]]
+name = "static_assertions"
+version = "1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f"
+
[[package]]
name = "strsim"
version = "0.11.1"
@@ -2969,6 +3685,20 @@ dependencies = [
"syn",
]
+[[package]]
+name = "sysctl"
+version = "0.6.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "01198a2debb237c62b6826ec7081082d951f46dbb64b0e8c7649a452230d1dfc"
+dependencies = [
+ "bitflags",
+ "byteorder",
+ "enum-as-inner",
+ "libc",
+ "thiserror 1.0.69",
+ "walkdir",
+]
+
[[package]]
name = "system-configuration"
version = "0.7.0"
@@ -3138,6 +3868,39 @@ version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20"
+[[package]]
+name = "tokenizers"
+version = "0.21.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a620b996116a59e184c2fa2dfd8251ea34a36d0a514758c6f966386bd2e03476"
+dependencies = [
+ "ahash",
+ "aho-corasick",
+ "compact_str",
+ "dary_heap",
+ "derive_builder",
+ "esaxx-rs",
+ "fancy-regex 0.14.0",
+ "getrandom 0.3.4",
+ "itertools 0.14.0",
+ "log",
+ "macro_rules_attribute",
+ "monostate",
+ "paste",
+ "rand 0.9.4",
+ "rayon",
+ "rayon-cond",
+ "regex",
+ "regex-syntax",
+ "serde",
+ "serde_json",
+ "spm_precompiled",
+ "thiserror 2.0.18",
+ "unicode-normalization-alignments",
+ "unicode-segmentation",
+ "unicode_categories",
+]
+
[[package]]
name = "tokio"
version = "1.52.0"
@@ -3465,6 +4228,12 @@ version = "2.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ea3136b675547379c4bd395ca6b938e5ad3c3d20fad76e7fe85f9e0d011419c"
+[[package]]
+name = "typed-path"
+version = "0.12.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8e28f89b80c87b8fb0cf04ab448d5dd0dd0ade2f8891bae878de66a75a28600e"
+
[[package]]
name = "typenum"
version = "1.20.0"
@@ -3486,6 +4255,15 @@ dependencies = [
"tinyvec",
]
+[[package]]
+name = "unicode-normalization-alignments"
+version = "0.1.12"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "43f613e4fa046e69818dd287fdc4bc78175ff20331479dab6e1b0f98d57062de"
+dependencies = [
+ "smallvec",
+]
+
[[package]]
name = "unicode-segmentation"
version = "1.13.2"
@@ -3498,12 +4276,24 @@ version = "0.1.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7dd6e30e90baa6f72411720665d41d89b9a3d039dc45b8faea1ddd07f617f6af"
+[[package]]
+name = "unicode-width"
+version = "0.2.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254"
+
[[package]]
name = "unicode-xid"
version = "0.2.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853"
+[[package]]
+name = "unicode_categories"
+version = "0.1.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "39ec24b3121d976906ece63c9daad25b85969647682eee313cb5779fdd69e14e"
+
[[package]]
name = "unsafe-libyaml"
version = "0.2.11"
@@ -3516,6 +4306,25 @@ version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"
+[[package]]
+name = "ureq"
+version = "2.12.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "02d1a66277ed75f640d608235660df48c8e3c19f3b4edb6a263315626cc3c01d"
+dependencies = [
+ "base64 0.22.1",
+ "flate2",
+ "log",
+ "once_cell",
+ "rustls",
+ "rustls-pki-types",
+ "serde",
+ "serde_json",
+ "socks",
+ "url",
+ "webpki-roots 0.26.11",
+]
+
[[package]]
name = "url"
version = "2.5.8"
@@ -3803,6 +4612,15 @@ dependencies = [
"url",
]
+[[package]]
+name = "webpki-roots"
+version = "0.26.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "521bc38abb08001b01866da9f51eb7c5d647a19260e00054a8c7fd5f9e57f7a9"
+dependencies = [
+ "webpki-roots 1.0.7",
+]
+
[[package]]
name = "webpki-roots"
version = "1.0.7"
@@ -3812,6 +4630,22 @@ dependencies = [
"rustls-pki-types",
]
+[[package]]
+name = "winapi"
+version = "0.3.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419"
+dependencies = [
+ "winapi-i686-pc-windows-gnu",
+ "winapi-x86_64-pc-windows-gnu",
+]
+
+[[package]]
+name = "winapi-i686-pc-windows-gnu"
+version = "0.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"
+
[[package]]
name = "winapi-util"
version = "0.1.11"
@@ -3821,6 +4655,12 @@ dependencies = [
"windows-sys 0.61.2",
]
+[[package]]
+name = "winapi-x86_64-pc-windows-gnu"
+version = "0.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
+
[[package]]
name = "windows-core"
version = "0.62.2"
@@ -3897,7 +4737,25 @@ version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
dependencies = [
- "windows-targets",
+ "windows-targets 0.52.6",
+]
+
+[[package]]
+name = "windows-sys"
+version = "0.59.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b"
+dependencies = [
+ "windows-targets 0.52.6",
+]
+
+[[package]]
+name = "windows-sys"
+version = "0.60.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb"
+dependencies = [
+ "windows-targets 0.53.5",
]
[[package]]
@@ -3915,14 +4773,31 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"
dependencies = [
- "windows_aarch64_gnullvm",
- "windows_aarch64_msvc",
- "windows_i686_gnu",
- "windows_i686_gnullvm",
- "windows_i686_msvc",
- "windows_x86_64_gnu",
- "windows_x86_64_gnullvm",
- "windows_x86_64_msvc",
+ "windows_aarch64_gnullvm 0.52.6",
+ "windows_aarch64_msvc 0.52.6",
+ "windows_i686_gnu 0.52.6",
+ "windows_i686_gnullvm 0.52.6",
+ "windows_i686_msvc 0.52.6",
+ "windows_x86_64_gnu 0.52.6",
+ "windows_x86_64_gnullvm 0.52.6",
+ "windows_x86_64_msvc 0.52.6",
+]
+
+[[package]]
+name = "windows-targets"
+version = "0.53.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3"
+dependencies = [
+ "windows-link",
+ "windows_aarch64_gnullvm 0.53.1",
+ "windows_aarch64_msvc 0.53.1",
+ "windows_i686_gnu 0.53.1",
+ "windows_i686_gnullvm 0.53.1",
+ "windows_i686_msvc 0.53.1",
+ "windows_x86_64_gnu 0.53.1",
+ "windows_x86_64_gnullvm 0.53.1",
+ "windows_x86_64_msvc 0.53.1",
]
[[package]]
@@ -3931,48 +4806,96 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
+[[package]]
+name = "windows_aarch64_gnullvm"
+version = "0.53.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53"
+
[[package]]
name = "windows_aarch64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
+[[package]]
+name = "windows_aarch64_msvc"
+version = "0.53.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006"
+
[[package]]
name = "windows_i686_gnu"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"
+[[package]]
+name = "windows_i686_gnu"
+version = "0.53.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3"
+
[[package]]
name = "windows_i686_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
+[[package]]
+name = "windows_i686_gnullvm"
+version = "0.53.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c"
+
[[package]]
name = "windows_i686_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
+[[package]]
+name = "windows_i686_msvc"
+version = "0.53.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2"
+
[[package]]
name = "windows_x86_64_gnu"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
+[[package]]
+name = "windows_x86_64_gnu"
+version = "0.53.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499"
+
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
+[[package]]
+name = "windows_x86_64_gnullvm"
+version = "0.53.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1"
+
[[package]]
name = "windows_x86_64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
+[[package]]
+name = "windows_x86_64_msvc"
+version = "0.53.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650"
+
[[package]]
name = "winnow"
version = "0.7.15"
@@ -4098,10 +5021,15 @@ name = "xerj-ai"
version = "1.0.0-rc.2"
dependencies = [
"anyhow",
+ "candle-core",
+ "candle-nn",
+ "candle-transformers",
+ "hf-hub",
"reqwest",
"serde",
"serde_json",
"thiserror 2.0.18",
+ "tokenizers",
"tokio",
"tracing",
"xerj-common",
@@ -4157,7 +5085,7 @@ dependencies = [
"tempfile",
"walkdir",
"xxhash-rust",
- "zip",
+ "zip 2.4.2",
]
[[package]]
@@ -4596,6 +5524,18 @@ dependencies = [
"zopfli",
]
+[[package]]
+name = "zip"
+version = "7.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c42e33efc22a0650c311c2ef19115ce232583abbe80850bc8b66509ebef02de0"
+dependencies = [
+ "crc32fast",
+ "indexmap 2.14.0",
+ "memchr",
+ "typed-path",
+]
+
[[package]]
name = "zmij"
version = "1.0.21"
diff --git a/engine/crates/xerj-ai/Cargo.toml b/engine/crates/xerj-ai/Cargo.toml
index 7cf7aed53..0e7084b0a 100644
--- a/engine/crates/xerj-ai/Cargo.toml
+++ b/engine/crates/xerj-ai/Cargo.toml
@@ -16,5 +16,22 @@ tracing.workspace = true
thiserror.workspace = true
anyhow.workspace = true
+# Built-in neural embedder (candle + MiniLM), download-on-first-run.
+# Feature-gated so the default build stays dependency-light and the
+# lexical embedder remains the zero-dep fallback. When `neural` is off,
+# none of this compiles.
+candle-core = { version = "0.9", optional = true }
+candle-nn = { version = "0.9", optional = true }
+candle-transformers = { version = "0.9", optional = true }
+tokenizers = { version = "0.21", optional = true, default-features = false, features = ["fancy-regex"] }
+# `ureq` (sync, blocking) pulls rustls by default — pure-Rust TLS, no OpenSSL,
+# so the neural feature cross-compiles cleanly to musl/aarch64/windows.
+hf-hub = { version = "0.4", optional = true, default-features = false, features = ["ureq"] }
+
+[features]
+default = []
+# Bundles the neural BERT embedder. Adds ~candle + tokenizers to the build.
+neural = ["dep:candle-core", "dep:candle-nn", "dep:candle-transformers", "dep:tokenizers", "dep:hf-hub"]
+
[dev-dependencies]
tokio = { version = "1", features = ["full"] }
diff --git a/engine/crates/xerj-ai/src/embedder.rs b/engine/crates/xerj-ai/src/embedder.rs
new file mode 100644
index 000000000..d4c1dd823
--- /dev/null
+++ b/engine/crates/xerj-ai/src/embedder.rs
@@ -0,0 +1,131 @@
+//! Unified embedding backend.
+//!
+//! XERJ embeds `semantic_text` fields through one of three interchangeable
+//! backends, all behind a single [`Embedder`] handle so the engine's ingest
+//! and query paths never branch on the backend:
+//!
+//! * [`Embedder::Lexical`] — the zero-dependency built-in feature-hash
+//! embedder ([`crate::local::local_embed`]). Deterministic, offline,
+//! fast — but lexical, *not* neural semantic understanding. This is the
+//! honest default when nothing else is configured.
+//! * [`Embedder::Proxy`] — an external OpenAI-compatible `/v1/embeddings`
+//! service ([`crate::embed::EmbeddingProxy`]). Bring any model/provider.
+//! * [`Embedder::Neural`] — the built-in BERT sentence encoder
+//! ([`crate::neural`]), running in-process via `candle`. Compiled only
+//! under the `neural` cargo feature; the model is loaded lazily on first
+//! use (download-on-first-run) so startup stays instant.
+//!
+//! [`Embedder::is_active`] distinguishes a *real* embedder (proxy or neural)
+//! from the lexical fallback — the query path only auto-embeds against the
+//! lexical backend for `semantic_text` fields, matching how they were
+//! embedded at ingest.
+
+use anyhow::{anyhow, Result};
+
+use crate::embed::EmbeddingProxy;
+use crate::local::{local_embed, DEFAULT_DIMS};
+
+/// A backend-agnostic text embedder shared across the engine.
+pub enum Embedder {
+ /// Built-in lexical feature-hash embedder (no model, no network).
+ Lexical,
+ /// External OpenAI-compatible embedding service.
+ Proxy(EmbeddingProxy),
+ /// Built-in neural BERT embedder (candle). Lazily loaded on first use.
+ #[cfg(feature = "neural")]
+ Neural(NeuralHandle),
+}
+
+impl Embedder {
+ /// The zero-config lexical fallback.
+ pub fn lexical() -> Self {
+ Embedder::Lexical
+ }
+
+ /// Wrap an already-constructed external embedding proxy.
+ pub fn proxy(proxy: EmbeddingProxy) -> Self {
+ Embedder::Proxy(proxy)
+ }
+
+ /// A lazily-loaded built-in neural embedder.
+ #[cfg(feature = "neural")]
+ pub fn neural(cfg: crate::neural::NeuralConfig) -> Self {
+ Embedder::Neural(NeuralHandle::new(cfg))
+ }
+
+ /// `true` when a *real* embedder (neural or external proxy) is configured;
+ /// `false` for the lexical fallback. The query path uses this to decide
+ /// whether to embed arbitrary query text (active) or restrict to
+ /// `semantic_text` fields embedded the same lexical way at ingest.
+ pub fn is_active(&self) -> bool {
+ !matches!(self, Embedder::Lexical)
+ }
+
+ /// A short human-readable label for logs / honesty reporting.
+ pub fn describe(&self) -> &'static str {
+ match self {
+ Embedder::Lexical => "lexical feature-hash (built-in, 384-dim, non-neural)",
+ Embedder::Proxy(_) => "external proxy (OpenAI-compatible /v1/embeddings)",
+ #[cfg(feature = "neural")]
+ Embedder::Neural(_) => "neural BERT (built-in, candle)",
+ }
+ }
+
+ /// Embed a batch of texts into vectors. Order matches the input.
+ pub async fn embed_batch(&self, texts: Vec<String>) -> Result<Vec<Vec<f32>>> {
+ match self {
+ Embedder::Lexical => Ok(texts
+ .iter()
+ .map(|t| local_embed(t, DEFAULT_DIMS))
+ .collect()),
+ Embedder::Proxy(proxy) => proxy
+ .embed_batch(texts)
+ .await
+ .map_err(|e| anyhow!("embedding proxy failed: {e}")),
+ #[cfg(feature = "neural")]
+ Embedder::Neural(handle) => handle.embed_batch(texts).await,
+ }
+ }
+}
+
+/// Lazily-loaded neural backend. The heavy model is loaded once, on the first
+/// `embed_batch`, off the async executor via `spawn_blocking`; every later
+/// call reuses the shared `Arc`.
+#[cfg(feature = "neural")]
+pub struct NeuralHandle {
+ cfg: crate::neural::NeuralConfig,
+ cell: tokio::sync::OnceCell<std::sync::Arc<crate::neural::NeuralEmbedder>>,
+}
+
+#[cfg(feature = "neural")]
+impl NeuralHandle {
+ pub fn new(cfg: crate::neural::NeuralConfig) -> Self {
+ Self {
+ cfg,
+ cell: tokio::sync::OnceCell::new(),
+ }
+ }
+
+ /// Get-or-load the model. First caller pays the (blocking) load / download;
+ /// concurrent callers await the same init.
+ async fn get(&self) -> Result<std::sync::Arc<crate::neural::NeuralEmbedder>> {
+ self.cell
+ .get_or_try_init(|| async {
+ let cfg = self.cfg.clone();
+ let model =
+ tokio::task::spawn_blocking(move || crate::neural::NeuralEmbedder::load(&cfg))
+ .await
+ .map_err(|e| anyhow!("neural model load task panicked: {e}"))??;
+ Ok::<_, anyhow::Error>(std::sync::Arc::new(model))
+ })
+ .await
+ .cloned()
+ }
+
+ async fn embed_batch(&self, texts: Vec<String>) -> Result<Vec<Vec<f32>>> {
+ let model = self.get().await?;
+ tokio::task::spawn_blocking(move || model.embed_blocking(&texts))
+ .await
+ .map_err(|e| anyhow!("neural embed task panicked: {e}"))?
+ }
+}
diff --git a/engine/crates/xerj-ai/src/lib.rs b/engine/crates/xerj-ai/src/lib.rs
index fed59df45..415755944 100644
--- a/engine/crates/xerj-ai/src/lib.rs
+++ b/engine/crates/xerj-ai/src/lib.rs
@@ -3,18 +3,24 @@
//! AI-native features for the xerj search engine.
//!
//! Provides:
+//! - [`embedder`] — Unified backend handle (lexical / proxy / neural) used by the engine
//! - [`embed`] — Embedding proxy: async HTTP client for OpenAI-compatible embedding APIs
//! - [`local`] — Built-in zero-config deterministic text embedder (feature hashing)
+//! - [`neural`] — Built-in neural BERT sentence embedder via candle (feature `neural`)
//! - [`chunker`] — Text chunking with sentence-aware splitting and overlap
//! - [`memory`] — Agent memory: semantic dedup + recency-blended recall
pub mod chunker;
pub mod embed;
+pub mod embedder;
pub mod local;
pub mod memory;
+#[cfg(feature = "neural")]
+pub mod neural;
pub use chunker::{Chunk, TextChunker};
pub use embed::{EmbeddingProxy, EmbeddingProxyConfig};
+pub use embedder::Embedder;
pub use local::{local_embed, DEFAULT_DIMS};
pub use memory::{AgentMemory, MemoryEntry};
diff --git a/engine/crates/xerj-ai/src/neural.rs b/engine/crates/xerj-ai/src/neural.rs
new file mode 100644
index 000000000..897abd2af
--- /dev/null
+++ b/engine/crates/xerj-ai/src/neural.rs
@@ -0,0 +1,294 @@
+//! Built-in neural sentence embedder — pure-Rust BERT inference via `candle`.
+//!
+//! This is XERJ's optional *real* semantic embedder. It loads a
+//! sentence-transformers BERT model (default `all-MiniLM-L6-v2`, 384-dim)
+//! and produces genuine neural embeddings, in-process, with no Python and
+//! no external service. It complements the two existing backends:
+//!
+//! * [`crate::local::local_embed`] — zero-dependency lexical feature-hash
+//! (the honest default; fast, deterministic, but *not* semantic).
+//! * [`crate::embed::EmbeddingProxy`] — any external OpenAI-compatible
+//! `/v1/embeddings` provider (bring-your-own model).
+//!
+//! Model files are fetched once on first use from the HuggingFace Hub and
+//! cached on disk; air-gapped deployments point [`NeuralConfig::local_dir`]
+//! at a directory holding `config.json`, `tokenizer.json`, and the
+//! safetensors weights instead.
+//!
+//! Compiled only under the `neural` cargo feature.
+
+use anyhow::{anyhow, Context, Result};
+use candle_core::{DType, Device, Tensor};
+use candle_nn::VarBuilder;
+use candle_transformers::models::bert::{BertModel, Config, DTYPE};
+use std::path::{Path, PathBuf};
+use tokenizers::{PaddingParams, PaddingStrategy, Tokenizer, TruncationParams};
+
+/// Default sentence encoder: 6-layer MiniLM, 384-dim, ~90 MB.
+pub const DEFAULT_MODEL_ID: &str = "sentence-transformers/all-MiniLM-L6-v2";
+
+/// Cap on tokens per passage. MiniLM's positional table is 512; passages are
+/// already chunked upstream, so this is a safety clamp, not the usual path.
+const MAX_TOKENS: usize = 512;
+
+/// How to obtain the model weights.
+#[derive(Debug, Clone)]
+pub struct NeuralConfig {
+ /// HuggingFace model id (e.g. `sentence-transformers/all-MiniLM-L6-v2`).
+ pub model_id: String,
+ /// Override the HuggingFace cache directory. `None` uses the default
+ /// (`~/.cache/huggingface`).
+ pub cache_dir: Option<PathBuf>,
+ /// Air-gapped: load `config.json` / `tokenizer.json` / weights from this
+ /// directory instead of downloading. Takes precedence over the hub.
+ pub local_dir: Option<PathBuf>,
+}
+
+impl Default for NeuralConfig {
+ fn default() -> Self {
+ Self {
+ model_id: DEFAULT_MODEL_ID.to_string(),
+ cache_dir: None,
+ local_dir: None,
+ }
+ }
+}
+
+/// A loaded BERT sentence embedder. Cheap to share behind an `Arc`; `embed`
+/// takes `&self` and is safe to call from many threads.
+pub struct NeuralEmbedder {
+ model: BertModel,
+ tokenizer: Tokenizer,
+ device: Device,
+ dims: usize,
+}
+
+impl NeuralEmbedder {
+ /// Output dimensionality (hidden size of the loaded model, e.g. 384).
+ pub fn dims(&self) -> usize {
+ self.dims
+ }
+
+ /// Load the model, downloading from the HuggingFace Hub on first use
+ /// (unless [`NeuralConfig::local_dir`] is set). **Blocking** — the caller
+ /// runs this off the async executor (see [`crate::embedder`]).
+ pub fn load(cfg: &NeuralConfig) -> Result<Self> {
+ let (config_path, tokenizer_path, weights_path) = match &cfg.local_dir {
+ Some(dir) => resolve_local(dir)?,
+ None => resolve_from_hub(&cfg.model_id, cfg.cache_dir.as_deref())?,
+ };
+
+ let config_json = std::fs::read_to_string(&config_path)
+ .with_context(|| format!("read model config {}", config_path.display()))?;
+ let config: Config = serde_json::from_str(&config_json)
+ .with_context(|| format!("parse model config {}", config_path.display()))?;
+ let dims = config.hidden_size;
+
+ let mut tokenizer = Tokenizer::from_file(&tokenizer_path)
+ .map_err(|e| anyhow!("load tokenizer {}: {e}", tokenizer_path.display()))?;
+ // Pad each batch to its longest member so we can stack into one tensor,
+ // and clamp over-long passages to the model's positional limit.
+ tokenizer.with_padding(Some(PaddingParams {
+ strategy: PaddingStrategy::BatchLongest,
+ ..Default::default()
+ }));
+ tokenizer
+ .with_truncation(Some(TruncationParams {
+ max_length: MAX_TOKENS,
+ ..Default::default()
+ }))
+ .map_err(|e| anyhow!("configure tokenizer truncation: {e}"))?;
+
+ let device = Device::Cpu;
+ // Safetensors is memory-mapped; the file must outlive the model, which
+ // it does (candle copies tensors into the VarBuilder-backed model).
+ let vb = unsafe {
+ VarBuilder::from_mmaped_safetensors(&[weights_path.clone()], DTYPE, &device)
+ .with_context(|| format!("map weights {}", weights_path.display()))?
+ };
+ let model = BertModel::load(vb, &config).map_err(|e| anyhow!("load BERT model: {e}"))?;
+
+ Ok(Self {
+ model,
+ tokenizer,
+ device,
+ dims,
+ })
+ }
+
+ /// Embed a batch of passages into L2-normalized sentence vectors using
+ /// attention-masked mean pooling (the sentence-transformers convention).
+ /// **Blocking / CPU-bound** — call via `spawn_blocking`.
+ pub fn embed_blocking(&self, texts: &[String]) -> Result<Vec<Vec<f32>>> {
+ if texts.is_empty() {
+ return Ok(vec![]);
+ }
+
+ let encodings = self
+ .tokenizer
+ .encode_batch(texts.to_vec(), true)
+ .map_err(|e| anyhow!("tokenize batch: {e}"))?;
+
+ let batch = encodings.len();
+ let seq_len = encodings.first().map(|e| e.get_ids().len()).unwrap_or(0);
+ if seq_len == 0 {
+ // All-empty input — return zero vectors of the right width.
+ return Ok(vec![vec![0.0; self.dims]; batch]);
+ }
+
+ let mut ids: Vec<u32> = Vec::with_capacity(batch * seq_len);
+ let mut mask: Vec<u32> = Vec::with_capacity(batch * seq_len);
+ for enc in &encodings {
+ ids.extend_from_slice(enc.get_ids());
+ mask.extend_from_slice(enc.get_attention_mask());
+ }
+
+ let input_ids = Tensor::from_vec(ids, (batch, seq_len), &self.device)
+ .map_err(|e| anyhow!("build input_ids tensor: {e}"))?;
+ let attention_mask = Tensor::from_vec(mask, (batch, seq_len), &self.device)
+ .map_err(|e| anyhow!("build attention_mask tensor: {e}"))?;
+ let token_type_ids = input_ids
+ .zeros_like()
+ .map_err(|e| anyhow!("build token_type_ids: {e}"))?;
+
+ // (batch, seq_len, hidden)
+ let hidden = self
+ .model
+ .forward(&input_ids, &token_type_ids, Some(&attention_mask))
+ .map_err(|e| anyhow!("bert forward: {e}"))?;
+
+ // Attention-masked mean pooling: sum(token * mask) / sum(mask).
+ let mask_f = attention_mask
+ .to_dtype(DType::F32)
+ .and_then(|m| m.unsqueeze(2)) // (batch, seq_len, 1)
+ .map_err(|e| anyhow!("mask to f32: {e}"))?;
+ let summed = hidden
+ .broadcast_mul(&mask_f)
+ .and_then(|h| h.sum(1)) // (batch, hidden)
+ .map_err(|e| anyhow!("masked sum: {e}"))?;
+ let counts = mask_f
+ .sum(1) // (batch, 1)
+ .and_then(|c| c.clamp(1e-9, f32::INFINITY))
+ .map_err(|e| anyhow!("mask counts: {e}"))?;
+ let mean = summed
+ .broadcast_div(&counts)
+ .map_err(|e| anyhow!("mean pool: {e}"))?;
+
+ // L2 normalize each row.
+ let norm = mean
+ .sqr()
+ .and_then(|s| s.sum_keepdim(1))
+ .and_then(|s| s.sqrt())
+ .and_then(|n| n.clamp(1e-12, f32::INFINITY))
+ .map_err(|e| anyhow!("l2 norm: {e}"))?;
+ let normed = mean
+ .broadcast_div(&norm)
+ .map_err(|e| anyhow!("normalize: {e}"))?;
+
+ normed
+ .to_vec2::<f32>()
+ .map_err(|e| anyhow!("read embeddings: {e}"))
+ }
+}
+
+/// Resolve the three model files from a local directory (air-gapped).
+fn resolve_local(dir: &Path) -> Result<(PathBuf, PathBuf, PathBuf)> {
+ let config = dir.join("config.json");
+ let tokenizer = dir.join("tokenizer.json");
+ let weights = find_local_weights(dir)?;
+ for (label, p) in [("config.json", &config), ("tokenizer.json", &tokenizer)] {
+ if !p.exists() {
+ return Err(anyhow!(
+ "local model dir {} is missing {label}",
+ dir.display()
+ ));
+ }
+ }
+ Ok((config, tokenizer, weights))
+}
+
+/// Prefer `model.safetensors`; candle cannot read PyTorch `.bin` weights.
+fn find_local_weights(dir: &Path) -> Result<PathBuf> {
+ let st = dir.join("model.safetensors");
+ if st.exists() {
+ return Ok(st);
+ }
+ Err(anyhow!(
+ "local model dir {} has no model.safetensors (candle requires safetensors, \
+ not pytorch_model.bin)",
+ dir.display()
+ ))
+}
+
+/// Download (or read from cache) the model files from the HuggingFace Hub.
+fn resolve_from_hub(
+ model_id: &str,
+ cache_dir: Option<&Path>,
+) -> Result<(PathBuf, PathBuf, PathBuf)> {
+ use hf_hub::api::sync::ApiBuilder;
+
+ let mut builder = ApiBuilder::new().with_progress(false);
+ if let Some(dir) = cache_dir {
+ builder = builder.with_cache_dir(dir.to_path_buf());
+ }
+ let api = builder
+ .build()
+ .with_context(|| "init HuggingFace hub client")?;
+ let repo = api.model(model_id.to_string());
+
+ let config = repo
+ .get("config.json")
+ .with_context(|| format!("fetch config.json for {model_id}"))?;
+ let tokenizer = repo
+ .get("tokenizer.json")
+ .with_context(|| format!("fetch tokenizer.json for {model_id}"))?;
+ let weights = repo.get("model.safetensors").with_context(|| {
+ format!(
+ "fetch model.safetensors for {model_id} (candle requires safetensors weights)"
+ )
+ })?;
+ Ok((config, tokenizer, weights))
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn cosine(a: &[f32], b: &[f32]) -> f32 {
+ a.iter().zip(b).map(|(x, y)| x * y).sum::<f32>()
+ }
+
+ /// Live test: downloads MiniLM on first run and checks that neural
+ /// embeddings capture semantic similarity (paraphrase > unrelated),
+ /// which the lexical feature-hash embedder cannot do. Network + ~90 MB
+ /// download, so it is `#[ignore]`d — run explicitly:
+ /// cargo test -p xerj-ai --features neural -- --ignored --nocapture
+ #[test]
+ #[ignore]
+ fn neural_captures_semantic_similarity() {
+ let emb = NeuralEmbedder::load(&NeuralConfig::default()).expect("load MiniLM");
+ assert_eq!(emb.dims(), 384, "MiniLM is 384-dim");
+
+ let texts = vec![
+ "A man is playing a guitar on stage.".to_string(),
+ "A musician performs with his guitar at a concert.".to_string(),
+ "The quarterly financial report shows rising interest rates.".to_string(),
+ ];
+ let vecs = emb.embed_blocking(&texts).expect("embed");
+ assert_eq!(vecs.len(), 3);
+ assert_eq!(vecs[0].len(), 384);
+
+ // Each vector is L2-normalized, so cosine == dot product.
+ let norm0 = cosine(&vecs[0], &vecs[0]).sqrt();
+ assert!((norm0 - 1.0).abs() < 1e-3, "vectors should be L2-normalized");
+
+ let sim_paraphrase = cosine(&vecs[0], &vecs[1]);
+ let sim_unrelated = cosine(&vecs[0], &vecs[2]);
+ eprintln!("paraphrase cos = {sim_paraphrase:.4}, unrelated cos = {sim_unrelated:.4}");
+ assert!(
+ sim_paraphrase > sim_unrelated + 0.15,
+ "neural embedder must rank the paraphrase far above the unrelated \
+ sentence (got paraphrase={sim_paraphrase:.3}, unrelated={sim_unrelated:.3})"
+ );
+ }
+}
diff --git a/engine/crates/xerj-autoindex/src/catalog.rs b/engine/crates/xerj-autoindex/src/catalog.rs
index 5a068480a..aade95dfa 100644
--- a/engine/crates/xerj-autoindex/src/catalog.rs
+++ b/engine/crates/xerj-autoindex/src/catalog.rs
@@ -53,7 +53,7 @@ pub fn catalog_mapping() -> Value {
pub const GOTCHAS: &[&str] = &[
"hybrid search: use {\"query\":{\"hybrid\":{\"queries\":[…]}}} ONLY — retriever.rrf is a silent stub and rank.rrf is ignored on this engine",
- "the semantic embedder is hash-bucket LEXICAL (384-dim): this is hybrid lexical+vector retrieval, NOT neural semantic understanding",
+ "semantic_text fields are embedded server-side: the DEFAULT is the built-in LEXICAL feature-hash embedder (384-dim hybrid lexical+vector, NOT neural) — start the server with `--embed-mode neural` (built-in BERT) or `--embed-mode proxy` for true neural semantics",
"semantic queries ignore _source filtering and return the ~8KB *_vector field in _source — strip client-side",
"exact filters use TOP-LEVEL keyword fields (term on .keyword subfields returns 0 hits on this engine)",
"all dates are normalized to RFC3339 UTC millis; mappings use strict_date_optional_time||epoch_millis",
@@ -180,7 +180,7 @@ pub fn build_sample_queries(pd: &PlanDataset, correlations: &[KeyCorr]) -> Vec<V
if let Some(sf) = &pd.semantic_field {
out.push(json!({
"class": "hybrid_lexical_vector",
- "title": format!("Hybrid lexical+vector (RRF) on {sf} — hash-bucket embedder, lexical not neural"),
+ "title": format!("Hybrid lexical+vector (RRF) on {sf} — embedder set server-side (lexical by default; neural/proxy if configured)"),
"request": format!("POST /{}/_search", pd.index),
"body": {"query": {"hybrid": {"queries": [
{"query": {"match": {(sf.clone()): word.clone()}}, "weight": 1},
@@ -296,7 +296,7 @@ pub fn render_map(
s.push_str(&format!("### `{}`\n\n", g("index_name")));
if let Some(sem) = d.get("semantic_field").filter(|v| v.is_string()) {
s.push_str(&format!(
- "semantic body field: `{}` (hybrid lexical+vector — hash-bucket embedder, lexical, not neural)\n\n",
+ "semantic body field: `{}` (hybrid lexical+vector; embedder set server-side — lexical by default, neural/proxy if configured)\n\n",
pretty_val(sem)
));
}
diff --git a/engine/crates/xerj-autoindex/src/infer/mod.rs b/engine/crates/xerj-autoindex/src/infer/mod.rs
index a0eeabcf6..0c4225e47 100644
--- a/engine/crates/xerj-autoindex/src/infer/mod.rs
+++ b/engine/crates/xerj-autoindex/src/infer/mod.rs
@@ -410,7 +410,7 @@ pub fn infer_fields(
if let Some((i, _)) = best {
specs[i].es_type = "semantic_text".into();
specs[i].notes.push(
- "hybrid lexical+vector body (hash-bucket embedder — lexical, not neural)".into(),
+ "hybrid lexical+vector body (embedded server-side: lexical by default, neural/proxy if configured)".into(),
);
}
}
diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rs
index 5124eab33..c231b0f36 100644
--- a/engine/crates/xerj-common/src/config.rs
+++ b/engine/crates/xerj-common/src/config.rs
@@ -625,12 +625,30 @@ impl Default for LogsConfig {
// ─────────────────────────────────────────────────────────────────────────────
-/// External embedding service settings.
+/// Embedding backend settings.
///
-/// **4 settings.**
+/// XERJ can embed `semantic_text` fields three ways, chosen by [`mode`]:
+/// * `"lexical"` — the zero-dependency built-in feature-hash embedder
+/// (deterministic, 384-dim, no model, no network). This is the honest
+/// default: lexical, *not* neural semantic understanding.
+/// * `"neural"` — a built-in BERT sentence embedder (all-MiniLM-L6-v2 by
+/// default) that runs in-process via `candle`. The model weights are
+/// downloaded once on first use (or read from [`local_model_dir`] for
+/// air-gapped deployments). Requires the binary to be built with the
+/// `neural` cargo feature; otherwise this falls back to lexical.
+/// * `"proxy"` — call an external OpenAI-compatible `/v1/embeddings`
+/// endpoint ([`default_endpoint`]). Lets customers plug in ANY embedding
+/// model / provider they already run.
+/// * `"auto"` (default) — use the proxy when [`default_endpoint`] is set,
+/// otherwise lexical. This preserves the historical behavior exactly.
+///
+/// **8 settings.**
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct EmbeddingConfig {
+ /// Backend selector: `"auto"` (default), `"lexical"`, `"neural"`, or
+ /// `"proxy"`. Unknown values are treated as `"auto"`.
+ pub mode: String,
/// OpenAI-compatible endpoint URL (default: `""` — disabled).
pub default_endpoint: String,
/// Model name to request from the endpoint (default: `""`).
@@ -639,15 +657,29 @@ pub struct EmbeddingConfig {
pub batch_size: usize,
/// HTTP timeout for embedding requests in ms (default: `5000`).
pub timeout_ms: u64,
+ /// Neural backend: HuggingFace model id to load (default
+ /// `sentence-transformers/all-MiniLM-L6-v2`, a 384-dim sentence encoder).
+ pub neural_model: String,
+ /// Neural backend: directory to cache downloaded model weights. Empty
+ /// (default) uses the standard HuggingFace cache (`~/.cache/huggingface`).
+ pub model_cache_dir: String,
+ /// Neural backend: if set, load `config.json`, `tokenizer.json`, and the
+ /// safetensors weights from this local directory instead of downloading
+ /// — for air-gapped / offline deployments. Empty (default) = download.
+ pub local_model_dir: String,
}
impl Default for EmbeddingConfig {
fn default() -> Self {
Self {
+ mode: "auto".to_string(),
default_endpoint: String::new(),
default_model: String::new(),
batch_size: 64,
timeout_ms: 5000,
+ neural_model: "sentence-transformers/all-MiniLM-L6-v2".to_string(),
+ model_cache_dir: String::new(),
+ local_model_dir: String::new(),
}
}
}
diff --git a/engine/crates/xerj-engine/Cargo.toml b/engine/crates/xerj-engine/Cargo.toml
index d93ee1b06..f39e8589c 100644
--- a/engine/crates/xerj-engine/Cargo.toml
+++ b/engine/crates/xerj-engine/Cargo.toml
@@ -45,6 +45,13 @@ xxhash-rust = { workspace = true }
# — SipHash was >21% of on-CPU time under the shard write locks at c8.
rustc-hash = { workspace = true }
+[features]
+default = []
+# Built-in neural BERT embedder (candle). Propagates to xerj-ai. Off by
+# default so the standard build stays lean and cross-compiles without the
+# candle/tokenizer toolchain; enable via xerj-server's `neural` feature.
+neural = ["xerj-ai/neural"]
+
[dev-dependencies]
tempfile = { workspace = true }
tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index 2185f9bfe..c3be719c5 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -463,12 +463,15 @@ pub struct Index {
/// runs hold the snapshot for the duration of one batch.
merge_config: xerj_common::config::MergeConfig,
- /// Optional embedding proxy used by the `semantic` query type
- /// (v0.7-P2). None when `Config.embedding.default_endpoint` is
- /// empty — semantic queries then return a clear 400 telling the
- /// operator to configure an endpoint. Reused across queries
- /// (semaphore + retry budget are per-proxy).
- embedding_proxy: Arc<Option<xerj_ai::embed::EmbeddingProxy>>,
+ /// Embedding backend for `semantic` / `semantic_text` (v0.7-P2).
+ /// One of lexical (built-in, default), an external proxy, or the
+ /// built-in neural BERT embedder — selected by `Config.embedding.mode`.
+ /// [`xerj_ai::Embedder::is_active`] is false only for the lexical
+ /// fallback; an active backend embeds arbitrary query text, while the
+ /// lexical fallback embeds only `semantic_text` fields (embedded the
+ /// same way at ingest). Reused across queries (proxy semaphore / neural
+ /// model are shared behind the `Arc`).
+ embedder: Arc<xerj_ai::Embedder>,
// ── Per-index metrics ─────────────────────────────────────────────────────
/// Total search queries executed.
@@ -836,7 +839,7 @@ impl Index {
)),
merge_in_progress: Arc::new(std::sync::atomic::AtomicBool::new(false)),
merge_config: config.merge.clone(),
- embedding_proxy: Arc::new(make_embedding_proxy(&config.embedding)),
+ embedder: Arc::new(make_embedder(&config.embedding)),
dv_cache: Arc::new(dashmap::DashMap::new()),
sort_shadow_cache: Arc::new(dashmap::DashMap::new()),
sort_shadow_fields: Arc::new(dashmap::DashMap::new()),
@@ -1037,7 +1040,7 @@ impl Index {
)),
merge_in_progress: Arc::new(std::sync::atomic::AtomicBool::new(false)),
merge_config: config.merge.clone(),
- embedding_proxy: Arc::new(make_embedding_proxy(&config.embedding)),
+ embedder: Arc::new(make_embedder(&config.embedding)),
dv_cache: Arc::new(dashmap::DashMap::new()),
sort_shadow_cache: Arc::new(dashmap::DashMap::new()),
sort_shadow_fields: Arc::new(dashmap::DashMap::new()),
@@ -3854,16 +3857,18 @@ impl Index {
};
// Embed the query text with the effective embedder:
- // * EmbeddingProxy when configured (high-quality, same as ingest),
- // * else the built-in deterministic embedder — but ONLY for
+ // * An ACTIVE backend (neural BERT or external proxy) embeds the
+ // query text directly — the same backend used at ingest.
+ // * Otherwise the built-in lexical embedder — but ONLY for
// `semantic_text` fields, which were embedded that same way at
// ingest. A `semantic` query against a plain dense_vector field
- // with no proxy configured still returns the original 400 (there
+ // with no active backend still returns the original 400 (there
// is no comparable stored vector to match against).
- let query_vec = if let Some(proxy) = &*self.embedding_proxy {
- // `embed_batch` takes Vec<String>; we only have one text but the
- // proxy's batching keeps the wire format stable for callers.
- let mut vectors = proxy
+ let query_vec = if self.embedder.is_active() {
+ // `embed_batch` takes Vec<String>; we only have one text but
+ // batching keeps the wire format stable for callers.
+ let mut vectors = self
+ .embedder
.embed_batch(vec![text.to_string()])
.await
.map_err(|e| {
@@ -3875,7 +3880,7 @@ impl Index {
Some(v) if !v.is_empty() => v,
_ => {
return Err(EngineError::Common(xerj_common::XerjError::invalid_query(
- "embedding proxy returned no vector for semantic query",
+ "embedder returned no vector for semantic query",
)));
}
}
@@ -3884,8 +3889,9 @@ impl Index {
} else {
return Err(EngineError::Common(xerj_common::XerjError::invalid_query(
"semantic query requires either a `semantic_text` field (auto-embedded \
- with the built-in embedder) or `embedding.default_endpoint` configured \
- to an OpenAI-compatible /v1/embeddings endpoint.",
+ with the built-in lexical embedder) or an active embedding backend \
+ (`embedding.mode = \"neural\"`, or `\"proxy\"` with \
+ `embedding.default_endpoint` set to an OpenAI-compatible endpoint).",
)));
};
@@ -3944,20 +3950,23 @@ impl Index {
// Chunk once and embed EACH overlapping chunk. `chunk_vecs` is one
// embedding per passage (>= 1). Short text (<= chunk_size) yields a
// single chunk == the exact pre-chunking behavior.
- let chunk_vecs: Vec<Vec<f32>> = if let Some(proxy) = &*self.embedding_proxy {
+ let chunk_vecs: Vec<Vec<f32>> = if self.embedder.is_active() {
+ // Active backend (neural or proxy): chunk, then embed each
+ // passage. The same backend + chunking is used at query time.
let chunks = semantic_chunker().chunk(&text, None);
let chunk_texts: Vec<String> = if chunks.len() <= 1 {
vec![text.clone()]
} else {
chunks.iter().map(|c| c.text.clone()).collect()
};
- let vs = proxy.embed_batch(chunk_texts).await.map_err(|e| {
+ let vs = self.embedder.embed_batch(chunk_texts).await.map_err(|e| {
EngineError::Common(xerj_common::XerjError::invalid_query(format!(
"semantic_text embed failed for field [{field}]: {e}"
)))
})?;
vs.into_iter().filter(|v| !v.is_empty()).collect()
} else {
+ // Lexical fallback: deterministic per-chunk feature-hash.
local_chunk_vectors(&text, dims)
};
// Pooled vector for `target`: single chunk is stored as-is (no
@@ -13193,31 +13202,82 @@ fn build_highlight_fragments(
fragments
}
-/// Build an `EmbeddingProxy` from `Config.embedding`. Returns `None`
-/// when no endpoint is configured — semantic queries then return a
-/// helpful 400 ("configure embedding.default_endpoint") instead of
-/// a runtime crash.
-fn make_embedding_proxy(
- cfg: &xerj_common::config::EmbeddingConfig,
-) -> Option<xerj_ai::embed::EmbeddingProxy> {
- if cfg.default_endpoint.is_empty() {
- return None;
+/// Build the effective [`xerj_ai::Embedder`] from `Config.embedding`.
+///
+/// Backend selection follows `embedding.mode`:
+/// * `"proxy"` — external OpenAI-compatible endpoint (requires
+/// `default_endpoint`).
+/// * `"neural"` — the built-in BERT embedder (requires the `neural`
+/// cargo feature; falls back to lexical with a warning otherwise).
+/// * `"lexical"` — the built-in feature-hash embedder.
+/// * `"auto"` (default / unknown) — proxy when `default_endpoint` is set,
+/// else lexical. Preserves the historical behavior exactly.
+///
+/// Any misconfiguration degrades to the lexical fallback (never a crash);
+/// a `semantic` query with no active backend against a plain vector field
+/// still returns a helpful 400.
+fn make_embedder(cfg: &xerj_common::config::EmbeddingConfig) -> xerj_ai::Embedder {
+ let mode = cfg.mode.trim().to_ascii_lowercase();
+ let want_proxy = mode == "proxy" || (mode != "neural" && mode != "lexical" && !cfg.default_endpoint.is_empty());
+
+ if mode == "neural" {
+ return make_neural_embedder(cfg);
+ }
+
+ if want_proxy {
+ if cfg.default_endpoint.is_empty() {
+ warn!("embedding.mode=proxy but embedding.default_endpoint is empty — falling back to lexical");
+ return xerj_ai::Embedder::lexical();
+ }
+ let proxy_cfg = xerj_ai::embed::EmbeddingProxyConfig {
+ endpoint: cfg.default_endpoint.clone(),
+ api_key: std::env::var("XERJ_EMBEDDING_API_KEY").ok(),
+ model: cfg.default_model.clone(),
+ timeout_secs: cfg.timeout_ms / 1000,
+ max_concurrent: 4,
+ max_retries: 3,
+ };
+ return match xerj_ai::embed::EmbeddingProxy::new(proxy_cfg) {
+ Ok(p) => {
+ info!(endpoint = %cfg.default_endpoint, "embedding backend: external proxy");
+ xerj_ai::Embedder::proxy(p)
+ }
+ Err(e) => {
+ warn!(error = %e, "embedding proxy init failed — falling back to lexical");
+ xerj_ai::Embedder::lexical()
+ }
+ };
}
- let proxy_cfg = xerj_ai::embed::EmbeddingProxyConfig {
- endpoint: cfg.default_endpoint.clone(),
- api_key: std::env::var("XERJ_EMBEDDING_API_KEY").ok(),
- model: cfg.default_model.clone(),
- timeout_secs: cfg.timeout_ms / 1000,
- max_concurrent: 4,
- max_retries: 3,
+
+ xerj_ai::Embedder::lexical()
+}
+
+/// Construct the built-in neural embedder when compiled in; otherwise warn
+/// and fall back to lexical so a `mode=neural` config never bricks startup.
+#[cfg(feature = "neural")]
+fn make_neural_embedder(cfg: &xerj_common::config::EmbeddingConfig) -> xerj_ai::Embedder {
+ let model_id = if cfg.neural_model.trim().is_empty() {
+ xerj_ai::neural::DEFAULT_MODEL_ID.to_string()
+ } else {
+ cfg.neural_model.clone()
};
- match xerj_ai::embed::EmbeddingProxy::new(proxy_cfg) {
- Ok(p) => Some(p),
- Err(e) => {
- warn!(error = %e, "embedding proxy init failed — semantic queries will 400");
- None
- }
- }
+ let neural_cfg = xerj_ai::neural::NeuralConfig {
+ model_id,
+ cache_dir: (!cfg.model_cache_dir.is_empty()).then(|| cfg.model_cache_dir.clone().into()),
+ local_dir: (!cfg.local_model_dir.is_empty()).then(|| cfg.local_model_dir.clone().into()),
+ };
+ info!(model = %neural_cfg.model_id, "embedding backend: built-in neural BERT (loads on first use)");
+ xerj_ai::Embedder::neural(neural_cfg)
+}
+
+#[cfg(not(feature = "neural"))]
+fn make_neural_embedder(_cfg: &xerj_common::config::EmbeddingConfig) -> xerj_ai::Embedder {
+ warn!(
+ "embedding.mode=neural requested but this binary was built without the `neural` \
+ cargo feature — falling back to the built-in lexical embedder. Rebuild with \
+ `--features neural` (xerj-server) to enable in-process BERT embeddings."
+ );
+ xerj_ai::Embedder::lexical()
}
// ── HNSW persistence helpers (module-level, called from sync `Index::open`)
diff --git a/engine/crates/xerj-server/Cargo.toml b/engine/crates/xerj-server/Cargo.toml
index bf4a2cdfc..786c20d9c 100644
--- a/engine/crates/xerj-server/Cargo.toml
+++ b/engine/crates/xerj-server/Cargo.toml
@@ -56,6 +56,14 @@ prost.workspace = true
# because it never returns freed heap pages to the OS. jemalloc is
# fragmentation-friendly and actively madvise()'s freed ranges, which brings
# RSS back down after each flush — critical under the 4 GB cgroup cap.
+[features]
+default = []
+# Ship the built-in neural BERT embedder (candle + MiniLM, download-on-
+# first-run). Off by default; build the release binary with
+# `cargo build --release -p xerj-server --features neural` to enable
+# in-process neural embeddings (set `embedding.mode = "neural"` at runtime).
+neural = ["xerj-engine/neural"]
+
[target.'cfg(not(target_env = "msvc"))'.dependencies]
tikv-jemallocator = "0.6"
diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rs
index f042f581a..e3b8cdff1 100644
--- a/engine/crates/xerj-server/src/main.rs
+++ b/engine/crates/xerj-server/src/main.rs
@@ -77,6 +77,7 @@ struct CliArgs {
config: Option<PathBuf>,
data_dir: Option<String>,
insecure: bool,
+ embed_mode: Option<String>,
}
fn parse_args() -> CliArgs {
@@ -84,6 +85,7 @@ fn parse_args() -> CliArgs {
let mut config = None;
let mut data_dir = None;
let mut insecure = false;
+ let mut embed_mode = None;
while let Some(arg) = args.next() {
match arg.as_str() {
@@ -96,6 +98,9 @@ fn parse_args() -> CliArgs {
"--insecure" | "-k" => {
insecure = true;
}
+ "--embed-mode" => {
+ embed_mode = args.next();
+ }
"--help" | "-h" => {
print_help();
std::process::exit(0);
@@ -115,6 +120,7 @@ fn parse_args() -> CliArgs {
config,
data_dir,
insecure,
+ embed_mode,
}
}
@@ -129,6 +135,9 @@ fn print_help() {
--config, -c <PATH> Path to TOML config file\n\
--data-dir, -d <PATH> Override data directory\n\
--insecure, -k Disable TLS\n\
+ --embed-mode <MODE> Embedding backend: lexical | neural | proxy | auto\n\
+ (neural = built-in BERT, downloads on first use;\n\
+ requires a binary built with --features neural)\n\
--help, -h Show this help\n\
--version, -V Print version and exit\n\
\n\
@@ -138,8 +147,9 @@ fn print_help() {
xerj autoindex map print the discovered data map\n\
\n\
ENVIRONMENT:\n\
- XERJ_LOG Log level filter (default: info)\n\
- XERJ_CONFIG Config file path\n",
+ XERJ_LOG Log level filter (default: info)\n\
+ XERJ_CONFIG Config file path\n\
+ XERJ_EMBED_MODE Embedding backend (lexical|neural|proxy|auto)\n",
env!("CARGO_PKG_VERSION")
);
}
@@ -231,6 +241,25 @@ fn load_config(args: &CliArgs) -> Result<Config> {
cfg.auth.enabled = false;
}
+ // Embedding backend override: `--embed-mode` flag or `XERJ_EMBED_MODE`
+ // env (flag wins). Accepts `lexical` | `neural` | `proxy` | `auto`.
+ if let Some(mode) = args
+ .embed_mode
+ .clone()
+ .or_else(|| std::env::var("XERJ_EMBED_MODE").ok())
+ {
+ let mode = mode.trim().to_ascii_lowercase();
+ match mode.as_str() {
+ "lexical" | "neural" | "proxy" | "auto" => {
+ info!("embedding.mode = {mode} (from CLI/env)");
+ cfg.embedding.mode = mode;
+ }
+ other => {
+ warn!("ignoring unknown --embed-mode '{other}' (use lexical|neural|proxy|auto)");
+ }
+ }
+ }
+
Ok(cfg)
}
@@ -570,6 +599,7 @@ async fn run_cli_index(cmd: IndexCmdArgs) -> Result<()> {
config: cmd.config.clone(),
data_dir: cmd.data_dir.clone(),
insecure: true,
+ embed_mode: None,
};
let mut cfg = load_config(&fake_cli)?;
cfg.tls.enabled = false;
diff --git a/recipes/README.md b/recipes/README.md
index 0358ff085..1fb82fe36 100644
--- a/recipes/README.md
+++ b/recipes/README.md
@@ -65,12 +65,14 @@ Every recipe honors `XERJ_URL` (default `http://localhost:9200`).
## Honest notes
-- The built-in embedder is **lexical** (feature-hashing over word +
- character n-grams, L2-normalized), not neural. It captures vocabulary
- overlap well and is perfect for offline demos and CI, but it won't
- match true synonyms. For production semantics, point XERJ at an
- OpenAI-compatible embeddings endpoint (`[embedding]` in `xerj.toml`) —
- the recipes need no change.
+- The **default** built-in embedder is **lexical** (feature-hashing over
+ word + character n-grams, L2-normalized), not neural. It captures
+ vocabulary overlap well and is perfect for offline demos and CI, but it
+ won't match true synonyms. For real neural semantics — with no change to
+ any recipe — start the server with a different `--embed-mode`: `neural`
+ (built-in in-process BERT, all-MiniLM-L6-v2, downloads once on first use;
+ build with `--features neural`) or `proxy` (any OpenAI-compatible
+ embeddings endpoint, set `[embedding] default_endpoint` in `xerj.toml`).
- `memory_agent.py` recalls with **text (BM25) relevance** scores
(unbounded, higher = better) by default. Add `"semantic": true` to the
`_recall` body to have XERJ embed the query server-side and recall by
diff --git a/recipes/semantic_search.py b/recipes/semantic_search.py
index c815f5761..477c9b56c 100644
--- a/recipes/semantic_search.py
+++ b/recipes/semantic_search.py
@@ -18,11 +18,20 @@ Run it:
Optional env:
XERJ_URL (default http://localhost:9200)
-To use a real neural embedding model instead of the built-in lexical
-embedder, point XERJ at any OpenAI-compatible endpoint in xerj.toml:
- [embedding]
- default_endpoint = "https://api.openai.com/v1/embeddings"
- default_model = "text-embedding-3-small"
+The built-in default embedder is *lexical* (feature-hashing), not neural.
+For real neural semantics — with no change to the index, mapping, or the
+three queries below — pick a different backend when you start the server:
+
+ # Built-in neural BERT (all-MiniLM-L6-v2), in-process, downloads once.
+ # Requires a binary built with `--features neural`.
+ xerj --insecure --data-dir ./data --embed-mode neural
+
+ # …or any external OpenAI-compatible endpoint (bring your own model):
+ xerj --insecure --data-dir ./data --embed-mode proxy
+ # with, in xerj.toml:
+ # [embedding]
+ # default_endpoint = "https://api.openai.com/v1/embeddings"
+ # default_model = "text-embedding-3-small"
"""
import json