AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(ai): built-in neural BERT embedder + unified 3-backend Embedder
xerj-org8af50ccda3e2
diff --git a/AGENTS.md b/AGENTS.mdindex 53e57006e..377e29a08 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -6,7 +6,7 @@## Ground rules for working in this repo-- **Honest claims only.** Every public number traces to a verified run. The embedder is *lexical* feature-hashing (never claim neural/semantic understanding); kNN is *exact* brute-force, recall 1.00 (never imply ANN speed); benchmark headline is the audited scorecard in `demo/playbooks/FULL_MATRIX_SCORECARD_*.md` (currently 42W/28L/12T vs live ES 8.13.4); do not claim TB-scale end-to-end (server heap ticket: `demo/usecases/autoindex/scale/TICKET_server-unbounded-ingest-heap.md`).+- **Honest claims only.** Every public number traces to a verified run. The **default** embedder is *lexical* feature-hashing — never claim the default binary does neural/semantic understanding. A built-in **neural** BERT embedder exists but is **opt-in** (`--features neural` + `--embed-mode neural`, model downloads on first use); only describe output as neural when that mode is actually running. kNN is *exact* brute-force, recall 1.00 (never imply ANN speed); benchmark headline is the audited scorecard in `demo/playbooks/FULL_MATRIX_SCORECARD_*.md` (currently 42W/28L/12T vs live ES 8.13.4); do not claim TB-scale end-to-end (server heap ticket: `demo/usecases/autoindex/scale/TICKET_server-unbounded-ingest-heap.md`).- **Builds are scoped:** `cd engine && cargo build --release -j 32 -p <crate>` — never workspace-wide, never `cargo clean`.- **The hard gate:** the ES-YAML conformance suite must stay at 1360 passed / 0 failed / 3 skipped before any engine change lands (see "Running the conformance tests" in the README).- **Git discipline:** non-trivial changes land with full commit bodies (motivation, before/after numbers, root cause, file pointers) — the git history is the project's engineering log; read it before re-deriving decisions.diff --git a/README.md b/README.mdindex e5422beec..27bd2c24f 100644--- a/README.md+++ b/README.md@@ -43,7 +43,7 @@ What is actually designed differently — for AI-agent workloads specifically, w- **A data map built for model orientation** — `xerj autoindex map` and the catalog index answer an agent's real first question, *"what is in here?"*, with counts, types, example values, ready-to-send queries, and known gotchas. Orientation designed for LLMs, not for humans with dashboards.- **Agent memory as a first-class API** — `/_memory/{ns}` store/recall with namespace isolation, semantic recall, dedup, metadata filters, and recency blending. Not a plugin; part of the engine.- **Machine-readable onboarding** — [llms.txt](https://xerj.org/llms.txt), agent tool schemas (OpenAI / MCP / Anthropic), [for-agents docs](https://xerj.org/for-agents.html). The documentation treats an LLM as a first-class reader, including honest caveat blocks it can rely on.-- **Batteries-included retrieval** — a built-in zero-config embedder (lexical feature-hashing, documented honestly as such) plus server-side hybrid RRF/linear fusion: BM25 + vector + hybrid out of the box, no external inference service required.+- **Batteries-included retrieval** — a built-in zero-config embedder (lexical feature-hashing by default, documented honestly as such; an optional in-process **neural** BERT embedder and any external embeddings endpoint are drop-in alternatives) plus server-side hybrid RRF/linear fusion: BM25 + vector + hybrid out of the box, no external inference service required.- **A single ~23 MB static binary, no JVM, sub-second start** — small enough that an agent can spawn a search engine as a subprocess tool, on any of 8 released targets.- **Exact kNN with recall 1.00 by construction** — the trade-off (latency scales with vectors scanned) is documented instead of hidden behind ANN defaults.- **Engineering-first internals** — columnar aggregation fast paths, doc-value prefilters, WAL-shard-pinned delete durability (verified by adversarial crash matrices), Rust memory safety. The head-to-head wins vs Elasticsearch concentrate exactly where agents live: aggregations, ingest throughput, and disk footprint.@@ -222,7 +222,7 @@ XERJ implements the Elasticsearch REST wire protocol. Because it is wire-compati`match_all`, `match_none`, `match`, `match_phrase`, `match_phrase_prefix`, `multi_match`, `term`, `terms`, `range`, `prefix`, `wildcard`, `exists`, `ids`, `bool`, `fuzzy`, `regexp`, `query_string`, `simple_query_string`, `constant_score`, `boosting`, `dis_max`, `geo_distance`, `knn`, `semantic`, `hybrid`.-> **Vector & semantic search notes.** `knn` (exact dense-vector search — brute-force scan, 100% recall; no ANN index yet) and `hybrid` (BM25 + kNN in one request) run entirely inside XERJ. As of **rc-2**, `semantic_text` fields **auto-embed on ingest** using a built-in, zero-config embedder, and the `semantic` query works with no external service; a configured external OpenAI-compatible `/v1/embeddings` endpoint (`embedding.default_endpoint`) is still used at higher quality when set. The built-in embedder is lexical (not neural) — see the [roadmap](#roadmap) and [ROADMAP.md](./ROADMAP.md) for the honest limitation and what's next. rc-2 also adds an **agent-memory REST API** (`/_memory/{ns}`) and **anomaly detection** (`/_ml/anomaly_detectors`).+> **Vector & semantic search notes.** `knn` (exact dense-vector search — brute-force scan, 100% recall; no ANN index yet) and `hybrid` (BM25 + kNN in one request) run entirely inside XERJ. As of **rc-2**, `semantic_text` fields **auto-embed on ingest** and the `semantic` query works with no external service. XERJ has **three interchangeable embedding backends**, selected with `--embed-mode` (or `embedding.mode`): **`lexical`** (default) — the built-in, zero-config feature-hash embedder (lexical, *not* neural); **`neural`** — a built-in **BERT sentence embedder** (all-MiniLM-L6-v2) that runs in-process via `candle`, downloading the model once on first use (build with `--features neural`); and **`proxy`** — any external OpenAI-compatible `/v1/embeddings` endpoint (`embedding.default_endpoint`). The default binary ships the lexical embedder to stay ~23 MB and dependency-free; see the [roadmap](#roadmap). rc-2 also adds an **agent-memory REST API** (`/_memory/{ns}`) and **anomaly detection** (`/_ml/anomaly_detectors`).**Supported aggregations**@@ -254,7 +254,7 @@ XERJ is a Cargo workspace under [`engine/`](./engine). The crates:| `xerj-fts` | Full-text search: BM25 scoring, analyzer registry, postings lists. || `xerj-vector` | Dense-vector index for kNN / semantic search (exact brute-force scan, 100% recall). || `xerj-logs` | Columnar log ingestion and retention. |-| `xerj-ai` | Text chunking, the built-in lexical embedder, and an embedding proxy (external OpenAI-compatible API) that powers the `semantic` query. The memory store is now exposed over the REST API at `/_memory/{ns}` (store / `_recall` / list / forget), shipped in rc-2 — see the [roadmap](#roadmap). |+| `xerj-ai` | Text chunking and the three embedding backends behind one `Embedder` handle — the built-in lexical embedder, an optional in-process **neural** BERT embedder (`candle` + MiniLM, feature `neural`), and an embedding proxy (external OpenAI-compatible API) — all powering the `semantic` query. The memory store is exposed over the REST API at `/_memory/{ns}` (store / `_recall` / list / forget), shipped in rc-2 — see the [roadmap](#roadmap). || `xerj-compress` | Block compression codecs (LZ4, Zstd). || `xerj-common` | Shared types: `Config`, `Schema`, `FieldType`, `XerjError`. || `xerj-cluster` | Embedded Raft consensus for cluster metadata (no external Raft dependency). |@@ -314,13 +314,13 @@ If a test expects a response and XERJ returns something different, that's a bugXERJ ships full-text, aggregation, log-analytics, dense-vector kNN, and hybrid search. **rc-2** added three AI-adjacent capabilities (each conformance-gated and verified by real requests):-- **Auto-embed on ingest + a built-in embedder** — `semantic_text` works with zero external config; the built-in embedder is lexical (a bundled *neural* model is still future work).+- **Auto-embed on ingest + three embedding backends** — `semantic_text` works with zero external config on the default **lexical** embedder; opt into a built-in **neural** BERT embedder (`--embed-mode neural`, `--features neural`, model downloads on first use) or an external **proxy** (`--embed-mode proxy`) with no change to your mappings or queries.- **Ingest-time chunk-embedding** — long `semantic_text` values embed **per overlapping passage**, and `semantic` search scores by the best-matching passage (max-sim), so a long document is retrievable on any one of its sections.- **Agent-memory REST API** — `POST /_memory/{ns}` store, `/_recall` (kNN or BM25 + metadata filter), list/forget/drop; namespaced and offline.- **Anomaly detection (`_ml`)** — real statistical detectors with baseline + deviation scoring, on-demand **and** via continuous `_ml/datafeeds` (a background job re-scores a live index on a timer; forecasting is still future work).- **Vector quantization** — opt a `dense_vector` field into **scalar8** (`int8_hnsw`) for ~4× smaller vectors at recall@10 ≈ 0.99.-Still planned: a bundled neural embedding model and distributed-clustering hardening. See [ROADMAP.md](./ROADMAP.md) for the full status and honest limitations.+Still planned: shipping the neural embedder in the default release binary (it's opt-in behind `--features neural` today to keep the default ~23 MB and dependency-free) and distributed-clustering hardening. See [ROADMAP.md](./ROADMAP.md) for the full status and honest limitations.---diff --git a/ROADMAP.md b/ROADMAP.mdindex 50ee296ba..be85863e8 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -26,7 +26,7 @@ These three shipped in rc-2 (each conformance-gated at 1,326 / 1,329 and verifie`semantic_text` now works end to end with **zero external configuration**. Indexing a document into a `semantic_text` field auto-embeds its text (previously returned `405`), and the `semantic` query embeds the query text with the same embedder and runs kNN — no external service required. Live-verified: a `semantic_text` doc indexed with no embedder configured returned `201`, and a `semantic` query ranked the intended doc first. A configured external `/v1/embeddings` proxy is still used, at higher quality, when `embedding.default_endpoint` is set.-- **Limitation:** the built-in embedder is a deterministic **lexical** model (feature-hashed word unigrams + character trigrams, L2-normalized) — it captures vocabulary/sub-word overlap, not deep semantics. This is observable live: a vocabulary-sharing query out-scored a true paraphrase. Paraphrases that share vocabulary rank correctly; truly-synonymous text with no word overlap will not. For production-grade semantics, configure the external proxy. A bundled neural model remains future work.+- **Limitation:** the **default** built-in embedder is a deterministic **lexical** model (feature-hashed word unigrams + character trigrams, L2-normalized) — it captures vocabulary/sub-word overlap, not deep semantics. This is observable live: a vocabulary-sharing query out-scored a true paraphrase. Paraphrases that share vocabulary rank correctly; truly-synonymous text with no word overlap will not. For real neural semantics you now have two drop-in upgrades with no mapping/query change: the built-in **neural** BERT embedder (`--embed-mode neural`, opt-in behind `--features neural`, downloads all-MiniLM-L6-v2 on first use — see "Neural embeddings" below), or the external `/v1/embeddings` **proxy** (`--embed-mode proxy` + `embedding.default_endpoint`).### 2. Agent-memory REST API ✅ (rc-2)@@ -62,7 +62,7 @@ The datafeed scheduler that rc-2 lacked: `PUT/GET/DELETE /_ml/datafeeds/{id}` +Long `semantic_text` values are split into overlapping passages, embedded **per passage**, and the per-passage vectors persisted (in `<field>_vector_chunks`, only when a value spans >1 passage). A `semantic` query scores each document by its **best-matching passage** (max-sim) instead of a single pooled vector, so a long document competes on any one of its sections. Live-verified: on 40 articles + a compendium of all 40, the compendium reached top-3 for **98%** of single-topic queries with per-passage scoring vs **32%** pooled. Short single-passage values are byte-identical to before. Recipe: `recipes/passage_search.py`; guide: `docs/recipes/passage-retrieval.md`.-- **Limitation:** the built-in embedder is still lexical (per-passage vectors are only as good as the embedder); a bundled neural model remains future work. A field that is *also* scalar8-quantized scores against the pooled vector (per-passage max-sim is exact-f32 only).+- **Limitation:** per-passage vectors are only as good as the active embedder. The default is lexical; switch to `--embed-mode neural` (built-in BERT) or `--embed-mode proxy` for neural-quality passage vectors — the chunk-embedding pipeline is backend-agnostic. A field that is *also* scalar8-quantized scores against the pooled vector (per-passage max-sim is exact-f32 only).## Partial / in progress@@ -103,7 +103,7 @@ Counting these toward "supported query types" overstates correctness — they re### Neural embeddings & richer ML-- A bundled/local **neural** embedding model (the current built-in embedder is lexical).+- A built-in **neural** embedder has **landed** (opt-in): `--embed-mode neural` runs an in-process BERT sentence encoder (all-MiniLM-L6-v2, 384-dim) via `candle` — pure Rust, no Python, no external service, air-gap friendly (`embedding.local_model_dir`). It is behind the `neural` cargo feature so the default release binary stays ~23 MB and dependency-free. **Remaining work:** ship it in the default binary (or a separate `xerj-neural` artifact) so no rebuild is needed, and offer a larger/higher-quality default model.- Forecasting for capacity/write-load signals (continuous `_ml` datafeeds have landed — see "Landed since rc-2"; the ingest-time per-passage chunk-embedding pipeline has also landed).### Correctness of stubbed surfacediff --git a/docs/recipes/semantic-search-rag.md b/docs/recipes/semantic-search-rag.mdindex 65cad454c..0d8d2be2b 100644--- a/docs/recipes/semantic-search-rag.md+++ b/docs/recipes/semantic-search-rag.md@@ -138,38 +138,62 @@ Q: 'get my money back after buying'semantic top -> kb-2 Refund policy```-## Honest limitation: the built-in embedder is *lexical*, not neural+## Three embedding backends: lexical (default), built-in neural, or bring-your-own++The `semantic_text` mapping and the `semantic` query are **identical** across all+three backends — you pick the backend once, at the server, and every ingest and+query uses it. Choose with `--embed-mode` (or `embedding.mode` in config, or the+`XERJ_EMBED_MODE` env var):++| `--embed-mode` | Backend | Quality | Setup |+|---|---|---|---|+| `lexical` *(default)* | Built-in feature-hash | Lexical overlap only | None — offline, deterministic |+| `neural` | Built-in **BERT** (all-MiniLM-L6-v2) via `candle` | True neural semantics | One-time model download on first use; binary built `--features neural` |+| `proxy` | Any external OpenAI-compatible `/v1/embeddings` | Whatever model you point at | Set `embedding.default_endpoint` |++### The default is lexical — know where it stopsThe zero-config embedder is a **lexical** model — it maps overlapping and relatedvocabulary into vector space. It shines when the question and the answer share*some* terms (a synonym here, a rephrase there), which covers a lot of realsupport/FAQ traffic. It is **not** a neural sentence embedder and won't capture-deep semantics with no lexical bridge.--Concretely, from this same KB:+deep semantics with no lexical bridge. Concretely, from this same KB:```Q: 'how do I rotate my API key'- semantic top -> kb-5 Rate limits (expected kb-1)+ semantic top -> kb-5 Rate limits (expected kb-1) # lexical mode```The doc is *titled* "Rotating an API key," but only its **body** is embedded, and-the body never uses the word "rotate" — so a pure "rotate" paraphrase misses. Two-honest takeaways:+the body never uses the word "rotate" — so a pure "rotate" paraphrase misses in+lexical mode.++### Upgrade to real neural semantics — still one binary, still in place++```bash+# Built-in neural embedder — downloads all-MiniLM-L6-v2 (~90 MB) once, then+# runs in-process (pure Rust, no Python, no external service, air-gap friendly+# via embedding.local_model_dir). Requires a binary built with --features neural.+xerj --insecure --data-dir ./data --embed-mode neural+```++With `--embed-mode neural`, the "rotate" paraphrase above lands on `kb-1` — the+BERT encoder captures that *rotate ≈ regenerate ≈ replace* even with no shared+term. Measured on the standard sentence-similarity pair, the built-in neural+embedder scores a paraphrase at cosine **0.72** vs an unrelated sentence at+**−0.04**; the lexical embedder cannot separate them that way.++Two honest takeaways that still apply in every mode:- **Embed the text that carries the meaning.** If titles matter, index them intothe `semantic_text` field too (or add a second `semantic_text` field and combinewith `hybrid`), so the vocabulary the user might use is actually in the vector.-- **For production-grade semantics, wire an external embeddings endpoint.** XERJ- will call any OpenAI-compatible `/v1/embeddings` service. Set- `embedding.default_endpoint` in config, or point a field at it in the mapping- via `inference_endpoint` / `inference_id`. The `semantic` query and the ingest- path are identical — you just get neural-quality vectors instead of the built-in- lexical ones. (Anomaly-style continuous re-embedding isn't a thing here;- embedding happens at ingest time.)--Everything above is `semantic_text` + the built-in embedder — no external service-was configured for this recipe.+- **Bring your own model when you need a specific one.** `--embed-mode proxy` with+ `embedding.default_endpoint` calls any OpenAI-compatible `/v1/embeddings`+ service (OpenAI, Cohere, a local vLLM, etc.). Embedding happens at ingest time.++Everything in the numbered walk-through above uses the default lexical embedder —+no external service was configured for this recipe.---diff --git a/engine/Cargo.lock b/engine/Cargo.lockindex b72b8e013..d181f4788 100644--- a/engine/Cargo.lock+++ b/engine/Cargo.lock@@ -15,7 +15,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75"dependencies = ["cfg-if",+ "getrandom 0.3.4","once_cell",+ "serde","version_check","zerocopy",]@@ -321,6 +323,12 @@ dependencies = ["tower-service",]+[[package]]+name = "base64"+version = "0.13.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "9e1b586273c5702936fe7b7d6896644d8be71e6314cfe09d3167c95f712589e8"+[[package]]name = "base64"version = "0.21.7"@@ -350,6 +358,21 @@ version = "0.5.2"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "3a8241f3ebb85c056b509d4327ad0358fbbba6ffb340bf388f26350aeda225b1"+[[package]]+name = "bit-set"+version = "0.8.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "08807e080ed7f9d5433fa9b275196cfc35414f66a0c79d864dc51a0d825231a3"+dependencies = [+ "bit-vec",+]++[[package]]+name = "bit-vec"+version = "0.8.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "5e764a1d40d510daf35e07be9eb06e75770908c27d411ee6c92109c9840eaaf7"+[[package]]name = "bitflags"version = "2.11.1"@@ -385,6 +408,20 @@ name = "bytemuck"version = "1.25.0"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "c8efb64bd706a16a1bdde310ae86b351e4d21550d98d056f22f8a7f7a2183fec"+dependencies = [+ "bytemuck_derive",+]++[[package]]+name = "bytemuck_derive"+version = "1.10.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "f9abbd1bc6865053c427f7198e6af43bfdedc55ab791faed4fbd361d789575ff"+dependencies = [+ "proc-macro2",+ "quote",+ "syn",+][[package]]name = "byteorder"@@ -398,12 +435,79 @@ version = "1.11.1"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33"+[[package]]+name = "candle-core"+version = "0.9.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "c15b675b80d994b2eadb20a4bbe434eabeb454eac3ee5e2b4cf6f147ee9be091"+dependencies = [+ "byteorder",+ "float8",+ "gemm",+ "half",+ "libm",+ "memmap2",+ "num-traits",+ "num_cpus",+ "rand 0.9.4",+ "rand_distr",+ "rayon",+ "safetensors",+ "thiserror 2.0.18",+ "yoke",+ "zip 7.2.0",+]++[[package]]+name = "candle-nn"+version = "0.9.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "3045fa9e7aef8567d209a27d56b692f60b96f4d0569f4c3011f8ca6715c65e03"+dependencies = [+ "candle-core",+ "half",+ "libc",+ "num-traits",+ "rayon",+ "safetensors",+ "serde",+ "thiserror 2.0.18",+]++[[package]]+name = "candle-transformers"+version = "0.9.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "b538ec4aa807c416a2ddd3621044888f188827862e2a6fcacba4738e89795d01"+dependencies = [+ "byteorder",+ "candle-core",+ "candle-nn",+ "fancy-regex 0.17.0",+ "num-traits",+ "rand 0.9.4",+ "rayon",+ "serde",+ "serde_json",+ "serde_plain",+ "tracing",+]+[[package]]name = "cast"version = "0.3.0"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "37b2a672a2cb129a2e41c10b1224bb368f9f37a2b16b612598138befd7b37eb5"+[[package]]+name = "castaway"+version = "0.2.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "dec551ab6e7578819132c713a93c022a05d60159dc86e7a7050223577484c55a"+dependencies = [+ "rustversion",+]+[[package]]name = "cc"version = "1.2.60"@@ -547,6 +651,34 @@ dependencies = ["windows-sys 0.52.0",]+[[package]]+name = "compact_str"+version = "0.9.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "9dfdd1c2274d9aa354115b09dc9a901d6c5576818cdf70d14cae2bdb47df00ab"+dependencies = [+ "castaway",+ "cfg-if",+ "itoa",+ "rustversion",+ "ryu",+ "serde",+ "static_assertions",+]++[[package]]+name = "console"+version = "0.15.11"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "054ccb5b10f9f2cbf51eb355ca1d05c2d279ce1804688d0db74b4733a5aeafd8"+dependencies = [+ "encode_unicode",+ "libc",+ "once_cell",+ "unicode-width 0.2.2",+ "windows-sys 0.59.0",+]+[[package]]name = "cookie"version = "0.18.1"@@ -614,7 +746,7 @@ dependencies = ["clap","criterion-plot","is-terminal",- "itertools",+ "itertools 0.10.5","num-traits","once_cell","oorandom",@@ -635,7 +767,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "6b50826342786a51a89e2da3a28f1c32b06e387201bc2d19791f622c673706b1"dependencies = ["cast",- "itertools",+ "itertools 0.10.5",][[package]]@@ -731,6 +863,50 @@ dependencies = ["memchr",]+[[package]]+name = "darling"+version = "0.20.11"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "fc7f46116c46ff9ab3eb1597a45688b6715c6e628b5c133e288e709a29bcb4ee"+dependencies = [+ "darling_core",+ "darling_macro",+]++[[package]]+name = "darling_core"+version = "0.20.11"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "0d00b9596d185e565c2207a0b01f8bd1a135483d02d9b7b0a54b11da8d53412e"+dependencies = [+ "fnv",+ "ident_case",+ "proc-macro2",+ "quote",+ "strsim",+ "syn",+]++[[package]]+name = "darling_macro"+version = "0.20.11"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead"+dependencies = [+ "darling_core",+ "quote",+ "syn",+]++[[package]]+name = "dary_heap"+version = "0.3.9"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "8b1e3a325bc115f096c8b77bbf027a7c2592230e70be2d985be950d3d5e60ebe"+dependencies = [+ "serde",+]+[[package]]name = "dashmap"version = "6.1.0"@@ -786,6 +962,37 @@ dependencies = ["syn",]+[[package]]+name = "derive_builder"+version = "0.20.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "507dfb09ea8b7fa618fcf76e953f4f5e192547945816d5358edffe39f6f94947"+dependencies = [+ "derive_builder_macro",+]++[[package]]+name = "derive_builder_core"+version = "0.20.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "2d5bcf7b024d6835cfb3d473887cd966994907effbe9227e8c8219824d06c4e8"+dependencies = [+ "darling",+ "proc-macro2",+ "quote",+ "syn",+]++[[package]]+name = "derive_builder_macro"+version = "0.20.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c"+dependencies = [+ "derive_builder_core",+ "syn",+]+[[package]]name = "digest"version = "0.10.7"@@ -797,6 +1004,27 @@ dependencies = ["subtle",]+[[package]]+name = "dirs"+version = "6.0.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "c3e8aa94d75141228480295a7d0e7feb620b1a5ad9f12bc40be62411e38cce4e"+dependencies = [+ "dirs-sys",+]++[[package]]+name = "dirs-sys"+version = "0.5.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "e01a3366d27ee9890022452ee61b2b63a67e6f13f58900b651ff5665f0bb1fab"+dependencies = [+ "libc",+ "option-ext",+ "redox_users",+ "windows-sys 0.61.2",+]+[[package]]name = "displaydoc"version = "0.2.5"@@ -808,12 +1036,34 @@ dependencies = ["syn",]+[[package]]+name = "dyn-stack"+version = "0.13.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "1c4713e43e2886ba72b8271aa66c93d722116acf7a75555cce11dcde84388fe8"+dependencies = [+ "bytemuck",+ "dyn-stack-macros",+]++[[package]]+name = "dyn-stack-macros"+version = "0.1.3"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "e1d926b4d407d372f141f93bb444696142c29d32962ccbd3531117cf3aa0bfa9"+[[package]]name = "either"version = "1.15.0"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719"+[[package]]+name = "encode_unicode"+version = "1.0.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "34aa73646ffb006b8f5147f3dc182bd4bcb190227ce861fc4a4844bf8e3cb2c0"+[[package]]name = "encoding_rs"version = "0.8.35"@@ -823,6 +1073,18 @@ dependencies = ["cfg-if",]+[[package]]+name = "enum-as-inner"+version = "0.6.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "a1e6a265c649f3f5979b601d26f1d05ada116434c87741c9493cb56218f76cbc"+dependencies = [+ "heck",+ "proc-macro2",+ "quote",+ "syn",+]+[[package]]name = "equivalent"version = "1.0.2"@@ -853,6 +1115,12 @@ dependencies = ["serde_yaml",]+[[package]]+name = "esaxx-rs"+version = "0.1.10"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "d817e038c30374a4bcb22f94d0a8a0e216958d4c3dcde369b1439fec4bdda6e6"+[[package]]name = "fallible-iterator"version = "0.3.0"@@ -865,6 +1133,28 @@ version = "0.1.9"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a"+[[package]]+name = "fancy-regex"+version = "0.14.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "6e24cb5a94bcae1e5408b0effca5cd7172ea3c5755049c5f3af4cd283a165298"+dependencies = [+ "bit-set",+ "regex-automata",+ "regex-syntax",+]++[[package]]+name = "fancy-regex"+version = "0.17.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "72cf461f865c862bb7dc573f643dd6a2b6842f7c30b07882b56bd148cc2761b8"+dependencies = [+ "bit-set",+ "regex-automata",+ "regex-syntax",+]+[[package]]name = "fastrand"version = "2.4.1"@@ -902,6 +1192,18 @@ dependencies = ["num-traits",]+[[package]]+name = "float8"+version = "0.6.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "719a903cc23e4a89e87962c2a80fdb45cdaad0983a89bd150bb57b4c8571a7d5"+dependencies = [+ "half",+ "num-traits",+ "rand 0.9.4",+ "rand_distr",+]+[[package]]name = "fnv"version = "1.0.7"@@ -914,6 +1216,12 @@ version = "0.1.5"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2"+[[package]]+name = "foldhash"+version = "0.2.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb"+[[package]]name = "foreign-types"version = "0.3.2"@@ -1042,6 +1350,125 @@ dependencies = ["slab",]+[[package]]+name = "gemm"+version = "0.19.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "aa0673db364b12263d103b68337a68fbecc541d6f6b61ba72fe438654709eacb"+dependencies = [+ "dyn-stack",+ "gemm-c32",+ "gemm-c64",+ "gemm-common",+ "gemm-f16",+ "gemm-f32",+ "gemm-f64",+ "num-complex",+ "num-traits",+ "paste",+ "raw-cpuid",+ "seq-macro",+]++[[package]]+name = "gemm-c32"+version = "0.19.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "086936dbdcb99e37aad81d320f98f670e53c1e55a98bee70573e83f95beb128c"+dependencies = [+ "dyn-stack",+ "gemm-common",+ "num-complex",+ "num-traits",+ "paste",+ "raw-cpuid",+ "seq-macro",+]++[[package]]+name = "gemm-c64"+version = "0.19.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "20c8aeeeec425959bda4d9827664029ba1501a90a0d1e6228e48bef741db3a3f"+dependencies = [+ "dyn-stack",+ "gemm-common",+ "num-complex",+ "num-traits",+ "paste",+ "raw-cpuid",+ "seq-macro",+]++[[package]]+name = "gemm-common"+version = "0.19.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "88027625910cc9b1085aaaa1c4bc46bb3a36aad323452b33c25b5e4e7c8e2a3e"+dependencies = [+ "bytemuck",+ "dyn-stack",+ "half",+ "libm",+ "num-complex",+ "num-traits",+ "once_cell",+ "paste",+ "pulp",+ "raw-cpuid",+ "rayon",+ "seq-macro",+ "sysctl",+]++[[package]]+name = "gemm-f16"+version = "0.19.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "e3df7a55202e6cd6739d82ae3399c8e0c7e1402859b30e4cb780e61525d9486e"+dependencies = [+ "dyn-stack",+ "gemm-common",+ "gemm-f32",+ "half",+ "num-complex",+ "num-traits",+ "paste",+ "raw-cpuid",+ "rayon",+ "seq-macro",+]++[[package]]+name = "gemm-f32"+version = "0.19.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "02e0b8c9da1fbec6e3e3ab2ce6bc259ef18eb5f6f0d3e4edf54b75f9fd41a81c"+dependencies = [+ "dyn-stack",+ "gemm-common",+ "num-complex",+ "num-traits",+ "paste",+ "raw-cpuid",+ "seq-macro",+]++[[package]]+name = "gemm-f64"+version = "0.19.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "056131e8f2a521bfab322f804ccd652520c79700d81209e9d9275bbdecaadc6a"+dependencies = [+ "dyn-stack",+ "gemm-common",+ "num-complex",+ "num-traits",+ "paste",+ "raw-cpuid",+ "seq-macro",+]+[[package]]name = "generic-array"version = "0.14.7"@@ -1123,8 +1550,12 @@ version = "2.7.1"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b"dependencies = [+ "bytemuck","cfg-if","crunchy",+ "num-traits",+ "rand 0.9.4",+ "rand_distr","zerocopy",]@@ -1160,7 +1591,20 @@ version = "0.15.5"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1"dependencies = [- "foldhash",+ "foldhash 0.1.5",+]++[[package]]+name = "hashbrown"+version = "0.16.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100"+dependencies = [+ "allocator-api2",+ "equivalent",+ "foldhash 0.2.0",+ "serde",+ "serde_core",][[package]]@@ -1196,6 +1640,25 @@ version = "0.4.3"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70"+[[package]]+name = "hf-hub"+version = "0.4.3"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "629d8f3bbeda9d148036d6b0de0a3ab947abd08ce90626327fc3547a49d59d97"+dependencies = [+ "dirs",+ "http",+ "indicatif",+ "libc",+ "log",+ "rand 0.9.4",+ "serde",+ "serde_json",+ "thiserror 2.0.18",+ "ureq",+ "windows-sys 0.60.2",+]+[[package]]name = "hmac"version = "0.12.1"@@ -1285,7 +1748,7 @@ dependencies = ["tokio","tokio-rustls","tower-service",- "webpki-roots",+ "webpki-roots 1.0.7",][[package]]@@ -1454,6 +1917,12 @@ version = "2.3.0"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954"+[[package]]+name = "ident_case"+version = "1.0.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39"+[[package]]name = "idna"version = "1.1.0"@@ -1497,6 +1966,19 @@ dependencies = ["serde_core",]+[[package]]+name = "indicatif"+version = "0.17.11"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "183b3088984b400f4cfac3620d5e076c84da5364016b4f49473de574b2586235"+dependencies = [+ "console",+ "number_prefix",+ "portable-atomic",+ "unicode-width 0.2.2",+ "web-time",+]+[[package]]name = "ipnet"version = "2.12.0"@@ -1539,6 +2021,15 @@ dependencies = ["either",]+[[package]]+name = "itertools"+version = "0.14.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285"+dependencies = [+ "either",+]+[[package]]name = "itoa"version = "1.0.18"@@ -1585,6 +2076,21 @@ version = "0.2.185"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "52ff2c0fe9bc6cb6b14a0592c2ff4fa9ceb83eea9db979b0487cd054946a2b8f"+[[package]]+name = "libm"+version = "0.2.16"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981"++[[package]]+name = "libredox"+version = "0.1.18"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "c943259e342f1e06ff2da7a83eabdfe7f92ce10262688dbf1895ff0b3e6e4652"+dependencies = [+ "libc",+]+[[package]]name = "libsqlite3-sys"version = "0.30.1"@@ -1671,6 +2177,22 @@ dependencies = ["twox-hash",]+[[package]]+name = "macro_rules_attribute"+version = "0.2.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "65049d7923698040cd0b1ddcced9b0eb14dd22c5f86ae59c3740eab64a676520"+dependencies = [+ "macro_rules_attribute-proc_macro",+ "paste",+]++[[package]]+name = "macro_rules_attribute-proc_macro"+version = "0.2.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "670fdfda89751bc4a84ac13eaa63e205cf0fd22b4c9a5fbfa085b63c1f1d3a30"+[[package]]name = "matchers"version = "0.2.0"@@ -1699,6 +2221,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "714098028fe011992e1c3962653c96b2d578c4b4bce9036e15ff220319b1e0e3"dependencies = ["libc",+ "stable_deref_trait",][[package]]@@ -1709,7 +2232,7 @@ checksum = "5f98efec8807c63c752b5bd61f862c165c115b0a35685bdcfd9238c7aeb592b7"dependencies = ["cfg-if","miette-derive",- "unicode-width",+ "unicode-width 0.1.14",][[package]]@@ -1756,6 +2279,28 @@ dependencies = ["windows-sys 0.61.2",]+[[package]]+name = "monostate"+version = "0.1.18"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "3341a273f6c9d5bef1908f17b7267bbab0e95c9bf69a0d4dcf8e9e1b2c76ef67"+dependencies = [+ "monostate-impl",+ "serde",+ "serde_core",+]++[[package]]+name = "monostate-impl"+version = "0.1.18"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "e4db6d5580af57bf992f59068d4ea26fd518574ff48d7639b255a36f9de6e7e9"+dependencies = [+ "proc-macro2",+ "quote",+ "syn",+]+[[package]]name = "multer"version = "3.1.0"@@ -1825,6 +2370,16 @@ dependencies = ["num-traits",]+[[package]]+name = "num-complex"+version = "0.4.6"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495"+dependencies = [+ "bytemuck",+ "num-traits",+]+[[package]]name = "num-conv"version = "0.2.1"@@ -1847,8 +2402,25 @@ source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841"dependencies = ["autocfg",+ "libm",+]++[[package]]+name = "num_cpus"+version = "1.17.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "91df4bbde75afed763b708b7eee1e8e7651e02d97f6d5dd763e89367e957b23b"+dependencies = [+ "hermit-abi",+ "libc",]+[[package]]+name = "number_prefix"+version = "0.4.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "830b246a0e5f20af87141b25c173cd1b609bd7779a4617d6ec582abaf90870f3"+[[package]]name = "oid-registry"version = "0.7.1"@@ -1930,6 +2502,12 @@ dependencies = ["vcpkg",]+[[package]]+name = "option-ext"+version = "0.2.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d"+[[package]]name = "parking_lot"version = "0.12.5"@@ -1962,6 +2540,12 @@ dependencies = ["regex",]+[[package]]+name = "paste"+version = "1.0.15"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a"+[[package]]name = "pastey"version = "0.1.1"@@ -2092,6 +2676,12 @@ dependencies = ["plotters-backend",]+[[package]]+name = "portable-atomic"+version = "1.13.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49"+[[package]]name = "potential_utf"version = "0.1.5"@@ -2167,7 +2757,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"dependencies = ["heck",- "itertools",+ "itertools 0.10.5","log","multimap","once_cell",@@ -2187,7 +2777,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d"dependencies = ["anyhow",- "itertools",+ "itertools 0.10.5","proc-macro2","quote","syn",@@ -2248,6 +2838,29 @@ dependencies = ["thiserror 1.0.69",]+[[package]]+name = "pulp"+version = "0.22.3"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "046aa45b989642ec2e4717c8e72d677b13edd831a4d3b6cf37d9a3e54912496a"+dependencies = [+ "bytemuck",+ "cfg-if",+ "libm",+ "num-complex",+ "paste",+ "pulp-wasm-simd-flag",+ "raw-cpuid",+ "reborrow",+ "version_check",+]++[[package]]+name = "pulp-wasm-simd-flag"+version = "0.1.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "1d8f70e07b9c3962945a74e59ca1c511bba65b6419468acc217c457d93f3c740"+[[package]]name = "quick-xml"version = "0.36.2"@@ -2392,6 +3005,25 @@ dependencies = ["getrandom 0.3.4",]+[[package]]+name = "rand_distr"+version = "0.5.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "6a8615d50dcf34fa31f7ab52692afec947c4dd0ab803cc87cb3b0b4570ff7463"+dependencies = [+ "num-traits",+ "rand 0.9.4",+]++[[package]]+name = "raw-cpuid"+version = "11.6.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "498cd0dc59d73224351ee52a95fee0f1a617a2eae0e7d9d720cc622c73a54186"+dependencies = [+ "bitflags",+]+[[package]]name = "rayon"version = "1.12.0"@@ -2402,6 +3034,17 @@ dependencies = ["rayon-core",]+[[package]]+name = "rayon-cond"+version = "0.4.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "2964d0cf57a3e7a06e8183d14a8b527195c706b7983549cd5462d5aa3747438f"+dependencies = [+ "either",+ "itertools 0.14.0",+ "rayon",+]+[[package]]name = "rayon-core"version = "1.13.0"@@ -2425,6 +3068,12 @@ dependencies = ["yasna",]+[[package]]+name = "reborrow"+version = "0.5.5"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "03251193000f4bd3b042892be858ee50e8b3719f2b08e5833ac4353724632430"+[[package]]name = "redox_syscall"version = "0.5.18"@@ -2434,6 +3083,17 @@ dependencies = ["bitflags",]+[[package]]+name = "redox_users"+version = "0.5.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "a4e608c6638b9c18977b00b475ac1f28d14e84b27d8d42f70e0bf1e3dec127ac"+dependencies = [+ "getrandom 0.2.17",+ "libredox",+ "thiserror 2.0.18",+]+[[package]]name = "ref-cast"version = "1.0.25"@@ -2526,7 +3186,7 @@ dependencies = ["wasm-bindgen","wasm-bindgen-futures","web-sys",- "webpki-roots",+ "webpki-roots 1.0.7",][[package]]@@ -2611,6 +3271,7 @@ version = "0.23.38"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "69f9466fb2c14ea04357e91413efb882e2a6d4a406e625449bc0a5d360d53a21"dependencies = [+ "log","once_cell","ring","rustls-pki-types",@@ -2661,6 +3322,17 @@ version = "1.0.23"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"+[[package]]+name = "safetensors"+version = "0.7.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "675656c1eabb620b921efea4f9199f97fc86e36dd6ffd1fbbe48d0f59a4987f5"+dependencies = [+ "hashbrown 0.16.1",+ "serde",+ "serde_json",+]+[[package]]name = "same-file"version = "1.0.6"@@ -2714,6 +3386,12 @@ version = "1.0.28"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd"+[[package]]+name = "seq-macro"+version = "0.3.6"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc"+[[package]]name = "serde"version = "1.0.228"@@ -2779,6 +3457,15 @@ dependencies = ["serde_core",]+[[package]]+name = "serde_plain"+version = "1.0.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "9ce1fc6db65a611022b23a0dec6975d63fb80a302cb3388835ff02c097258d50"+dependencies = [+ "serde",+]+[[package]]name = "serde_spanned"version = "0.6.9"@@ -2914,18 +3601,47 @@ dependencies = ["windows-sys 0.61.2",]+[[package]]+name = "socks"+version = "0.3.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "f0c3dbbd9ae980613c6dd8e28a9407b50509d3803b57624d5dfe8315218cd58b"+dependencies = [+ "byteorder",+ "libc",+ "winapi",+]+[[package]]name = "spin"version = "0.9.8"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67"+[[package]]+name = "spm_precompiled"+version = "0.1.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "5851699c4033c63636f7ea4cf7b7c1f1bf06d0cc03cfb42e711de5a5c46cf326"+dependencies = [+ "base64 0.13.1",+ "nom",+ "serde",+ "unicode-segmentation",+]+[[package]]name = "stable_deref_trait"version = "1.2.1"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596"+[[package]]+name = "static_assertions"+version = "1.1.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f"+[[package]]name = "strsim"version = "0.11.1"@@ -2969,6 +3685,20 @@ dependencies = ["syn",]+[[package]]+name = "sysctl"+version = "0.6.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "01198a2debb237c62b6826ec7081082d951f46dbb64b0e8c7649a452230d1dfc"+dependencies = [+ "bitflags",+ "byteorder",+ "enum-as-inner",+ "libc",+ "thiserror 1.0.69",+ "walkdir",+]+[[package]]name = "system-configuration"version = "0.7.0"@@ -3138,6 +3868,39 @@ version = "0.1.1"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20"+[[package]]+name = "tokenizers"+version = "0.21.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "a620b996116a59e184c2fa2dfd8251ea34a36d0a514758c6f966386bd2e03476"+dependencies = [+ "ahash",+ "aho-corasick",+ "compact_str",+ "dary_heap",+ "derive_builder",+ "esaxx-rs",+ "fancy-regex 0.14.0",+ "getrandom 0.3.4",+ "itertools 0.14.0",+ "log",+ "macro_rules_attribute",+ "monostate",+ "paste",+ "rand 0.9.4",+ "rayon",+ "rayon-cond",+ "regex",+ "regex-syntax",+ "serde",+ "serde_json",+ "spm_precompiled",+ "thiserror 2.0.18",+ "unicode-normalization-alignments",+ "unicode-segmentation",+ "unicode_categories",+]+[[package]]name = "tokio"version = "1.52.0"@@ -3465,6 +4228,12 @@ version = "2.1.2"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "9ea3136b675547379c4bd395ca6b938e5ad3c3d20fad76e7fe85f9e0d011419c"+[[package]]+name = "typed-path"+version = "0.12.3"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "8e28f89b80c87b8fb0cf04ab448d5dd0dd0ade2f8891bae878de66a75a28600e"+[[package]]name = "typenum"version = "1.20.0"@@ -3486,6 +4255,15 @@ dependencies = ["tinyvec",]+[[package]]+name = "unicode-normalization-alignments"+version = "0.1.12"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "43f613e4fa046e69818dd287fdc4bc78175ff20331479dab6e1b0f98d57062de"+dependencies = [+ "smallvec",+]+[[package]]name = "unicode-segmentation"version = "1.13.2"@@ -3498,12 +4276,24 @@ version = "0.1.14"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "7dd6e30e90baa6f72411720665d41d89b9a3d039dc45b8faea1ddd07f617f6af"+[[package]]+name = "unicode-width"+version = "0.2.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254"+[[package]]name = "unicode-xid"version = "0.2.6"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853"+[[package]]+name = "unicode_categories"+version = "0.1.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "39ec24b3121d976906ece63c9daad25b85969647682eee313cb5779fdd69e14e"+[[package]]name = "unsafe-libyaml"version = "0.2.11"@@ -3516,6 +4306,25 @@ version = "0.9.0"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"+[[package]]+name = "ureq"+version = "2.12.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "02d1a66277ed75f640d608235660df48c8e3c19f3b4edb6a263315626cc3c01d"+dependencies = [+ "base64 0.22.1",+ "flate2",+ "log",+ "once_cell",+ "rustls",+ "rustls-pki-types",+ "serde",+ "serde_json",+ "socks",+ "url",+ "webpki-roots 0.26.11",+]+[[package]]name = "url"version = "2.5.8"@@ -3803,6 +4612,15 @@ dependencies = ["url",]+[[package]]+name = "webpki-roots"+version = "0.26.11"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "521bc38abb08001b01866da9f51eb7c5d647a19260e00054a8c7fd5f9e57f7a9"+dependencies = [+ "webpki-roots 1.0.7",+]+[[package]]name = "webpki-roots"version = "1.0.7"@@ -3812,6 +4630,22 @@ dependencies = ["rustls-pki-types",]+[[package]]+name = "winapi"+version = "0.3.9"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419"+dependencies = [+ "winapi-i686-pc-windows-gnu",+ "winapi-x86_64-pc-windows-gnu",+]++[[package]]+name = "winapi-i686-pc-windows-gnu"+version = "0.4.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"+[[package]]name = "winapi-util"version = "0.1.11"@@ -3821,6 +4655,12 @@ dependencies = ["windows-sys 0.61.2",]+[[package]]+name = "winapi-x86_64-pc-windows-gnu"+version = "0.4.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"+[[package]]name = "windows-core"version = "0.62.2"@@ -3897,7 +4737,25 @@ version = "0.52.0"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"dependencies = [- "windows-targets",+ "windows-targets 0.52.6",+]++[[package]]+name = "windows-sys"+version = "0.59.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b"+dependencies = [+ "windows-targets 0.52.6",+]++[[package]]+name = "windows-sys"+version = "0.60.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb"+dependencies = [+ "windows-targets 0.53.5",][[package]]@@ -3915,14 +4773,31 @@ version = "0.52.6"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"dependencies = [- "windows_aarch64_gnullvm",- "windows_aarch64_msvc",- "windows_i686_gnu",- "windows_i686_gnullvm",- "windows_i686_msvc",- "windows_x86_64_gnu",- "windows_x86_64_gnullvm",- "windows_x86_64_msvc",+ "windows_aarch64_gnullvm 0.52.6",+ "windows_aarch64_msvc 0.52.6",+ "windows_i686_gnu 0.52.6",+ "windows_i686_gnullvm 0.52.6",+ "windows_i686_msvc 0.52.6",+ "windows_x86_64_gnu 0.52.6",+ "windows_x86_64_gnullvm 0.52.6",+ "windows_x86_64_msvc 0.52.6",+]++[[package]]+name = "windows-targets"+version = "0.53.5"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3"+dependencies = [+ "windows-link",+ "windows_aarch64_gnullvm 0.53.1",+ "windows_aarch64_msvc 0.53.1",+ "windows_i686_gnu 0.53.1",+ "windows_i686_gnullvm 0.53.1",+ "windows_i686_msvc 0.53.1",+ "windows_x86_64_gnu 0.53.1",+ "windows_x86_64_gnullvm 0.53.1",+ "windows_x86_64_msvc 0.53.1",][[package]]@@ -3931,48 +4806,96 @@ version = "0.52.6"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"+[[package]]+name = "windows_aarch64_gnullvm"+version = "0.53.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53"+[[package]]name = "windows_aarch64_msvc"version = "0.52.6"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"+[[package]]+name = "windows_aarch64_msvc"+version = "0.53.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006"+[[package]]name = "windows_i686_gnu"version = "0.52.6"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"+[[package]]+name = "windows_i686_gnu"+version = "0.53.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3"+[[package]]name = "windows_i686_gnullvm"version = "0.52.6"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"+[[package]]+name = "windows_i686_gnullvm"+version = "0.53.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c"+[[package]]name = "windows_i686_msvc"version = "0.52.6"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"+[[package]]+name = "windows_i686_msvc"+version = "0.53.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2"+[[package]]name = "windows_x86_64_gnu"version = "0.52.6"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"+[[package]]+name = "windows_x86_64_gnu"+version = "0.53.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499"+[[package]]name = "windows_x86_64_gnullvm"version = "0.52.6"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"+[[package]]+name = "windows_x86_64_gnullvm"+version = "0.53.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1"+[[package]]name = "windows_x86_64_msvc"version = "0.52.6"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"+[[package]]+name = "windows_x86_64_msvc"+version = "0.53.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650"+[[package]]name = "winnow"version = "0.7.15"@@ -4098,10 +5021,15 @@ name = "xerj-ai"version = "1.0.0-rc.2"dependencies = ["anyhow",+ "candle-core",+ "candle-nn",+ "candle-transformers",+ "hf-hub","reqwest","serde","serde_json","thiserror 2.0.18",+ "tokenizers","tokio","tracing","xerj-common",@@ -4157,7 +5085,7 @@ dependencies = ["tempfile","walkdir","xxhash-rust",- "zip",+ "zip 2.4.2",][[package]]@@ -4596,6 +5524,18 @@ dependencies = ["zopfli",]+[[package]]+name = "zip"+version = "7.2.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "c42e33efc22a0650c311c2ef19115ce232583abbe80850bc8b66509ebef02de0"+dependencies = [+ "crc32fast",+ "indexmap 2.14.0",+ "memchr",+ "typed-path",+]+[[package]]name = "zmij"version = "1.0.21"diff --git a/engine/crates/xerj-ai/Cargo.toml b/engine/crates/xerj-ai/Cargo.tomlindex 7cf7aed53..0e7084b0a 100644--- a/engine/crates/xerj-ai/Cargo.toml+++ b/engine/crates/xerj-ai/Cargo.toml@@ -16,5 +16,22 @@ tracing.workspace = truethiserror.workspace = trueanyhow.workspace = true+# Built-in neural embedder (candle + MiniLM), download-on-first-run.+# Feature-gated so the default build stays dependency-light and the+# lexical embedder remains the zero-dep fallback. When `neural` is off,+# none of this compiles.+candle-core = { version = "0.9", optional = true }+candle-nn = { version = "0.9", optional = true }+candle-transformers = { version = "0.9", optional = true }+tokenizers = { version = "0.21", optional = true, default-features = false, features = ["fancy-regex"] }+# `ureq` (sync, blocking) pulls rustls by default — pure-Rust TLS, no OpenSSL,+# so the neural feature cross-compiles cleanly to musl/aarch64/windows.+hf-hub = { version = "0.4", optional = true, default-features = false, features = ["ureq"] }++[features]+default = []+# Bundles the neural BERT embedder. Adds ~candle + tokenizers to the build.+neural = ["dep:candle-core", "dep:candle-nn", "dep:candle-transformers", "dep:tokenizers", "dep:hf-hub"]+[dev-dependencies]tokio = { version = "1", features = ["full"] }diff --git a/engine/crates/xerj-ai/src/embedder.rs b/engine/crates/xerj-ai/src/embedder.rsnew file mode 100644index 000000000..d4c1dd823--- /dev/null+++ b/engine/crates/xerj-ai/src/embedder.rs@@ -0,0 +1,131 @@+//! Unified embedding backend.+//!+//! XERJ embeds `semantic_text` fields through one of three interchangeable+//! backends, all behind a single [`Embedder`] handle so the engine's ingest+//! and query paths never branch on the backend:+//!+//! * [`Embedder::Lexical`] — the zero-dependency built-in feature-hash+//! embedder ([`crate::local::local_embed`]). Deterministic, offline,+//! fast — but lexical, *not* neural semantic understanding. This is the+//! honest default when nothing else is configured.+//! * [`Embedder::Proxy`] — an external OpenAI-compatible `/v1/embeddings`+//! service ([`crate::embed::EmbeddingProxy`]). Bring any model/provider.+//! * [`Embedder::Neural`] — the built-in BERT sentence encoder+//! ([`crate::neural`]), running in-process via `candle`. Compiled only+//! under the `neural` cargo feature; the model is loaded lazily on first+//! use (download-on-first-run) so startup stays instant.+//!+//! [`Embedder::is_active`] distinguishes a *real* embedder (proxy or neural)+//! from the lexical fallback — the query path only auto-embeds against the+//! lexical backend for `semantic_text` fields, matching how they were+//! embedded at ingest.++use anyhow::{anyhow, Result};++use crate::embed::EmbeddingProxy;+use crate::local::{local_embed, DEFAULT_DIMS};++/// A backend-agnostic text embedder shared across the engine.+pub enum Embedder {+ /// Built-in lexical feature-hash embedder (no model, no network).+ Lexical,+ /// External OpenAI-compatible embedding service.+ Proxy(EmbeddingProxy),+ /// Built-in neural BERT embedder (candle). Lazily loaded on first use.+ #[cfg(feature = "neural")]+ Neural(NeuralHandle),+}++impl Embedder {+ /// The zero-config lexical fallback.+ pub fn lexical() -> Self {+ Embedder::Lexical+ }++ /// Wrap an already-constructed external embedding proxy.+ pub fn proxy(proxy: EmbeddingProxy) -> Self {+ Embedder::Proxy(proxy)+ }++ /// A lazily-loaded built-in neural embedder.+ #[cfg(feature = "neural")]+ pub fn neural(cfg: crate::neural::NeuralConfig) -> Self {+ Embedder::Neural(NeuralHandle::new(cfg))+ }++ /// `true` when a *real* embedder (neural or external proxy) is configured;+ /// `false` for the lexical fallback. The query path uses this to decide+ /// whether to embed arbitrary query text (active) or restrict to+ /// `semantic_text` fields embedded the same lexical way at ingest.+ pub fn is_active(&self) -> bool {+ !matches!(self, Embedder::Lexical)+ }++ /// A short human-readable label for logs / honesty reporting.+ pub fn describe(&self) -> &'static str {+ match self {+ Embedder::Lexical => "lexical feature-hash (built-in, 384-dim, non-neural)",+ Embedder::Proxy(_) => "external proxy (OpenAI-compatible /v1/embeddings)",+ #[cfg(feature = "neural")]+ Embedder::Neural(_) => "neural BERT (built-in, candle)",+ }+ }++ /// Embed a batch of texts into vectors. Order matches the input.+ pub async fn embed_batch(&self, texts: Vec<String>) -> Result<Vec<Vec<f32>>> {+ match self {+ Embedder::Lexical => Ok(texts+ .iter()+ .map(|t| local_embed(t, DEFAULT_DIMS))+ .collect()),+ Embedder::Proxy(proxy) => proxy+ .embed_batch(texts)+ .await+ .map_err(|e| anyhow!("embedding proxy failed: {e}")),+ #[cfg(feature = "neural")]+ Embedder::Neural(handle) => handle.embed_batch(texts).await,+ }+ }+}++/// Lazily-loaded neural backend. The heavy model is loaded once, on the first+/// `embed_batch`, off the async executor via `spawn_blocking`; every later+/// call reuses the shared `Arc`.+#[cfg(feature = "neural")]+pub struct NeuralHandle {+ cfg: crate::neural::NeuralConfig,+ cell: tokio::sync::OnceCell<std::sync::Arc<crate::neural::NeuralEmbedder>>,+}++#[cfg(feature = "neural")]+impl NeuralHandle {+ pub fn new(cfg: crate::neural::NeuralConfig) -> Self {+ Self {+ cfg,+ cell: tokio::sync::OnceCell::new(),+ }+ }++ /// Get-or-load the model. First caller pays the (blocking) load / download;+ /// concurrent callers await the same init.+ async fn get(&self) -> Result<std::sync::Arc<crate::neural::NeuralEmbedder>> {+ self.cell+ .get_or_try_init(|| async {+ let cfg = self.cfg.clone();+ let model =+ tokio::task::spawn_blocking(move || crate::neural::NeuralEmbedder::load(&cfg))+ .await+ .map_err(|e| anyhow!("neural model load task panicked: {e}"))??;+ Ok::<_, anyhow::Error>(std::sync::Arc::new(model))+ })+ .await+ .cloned()+ }++ async fn embed_batch(&self, texts: Vec<String>) -> Result<Vec<Vec<f32>>> {+ let model = self.get().await?;+ tokio::task::spawn_blocking(move || model.embed_blocking(&texts))+ .await+ .map_err(|e| anyhow!("neural embed task panicked: {e}"))?+ }+}diff --git a/engine/crates/xerj-ai/src/lib.rs b/engine/crates/xerj-ai/src/lib.rsindex fed59df45..415755944 100644--- a/engine/crates/xerj-ai/src/lib.rs+++ b/engine/crates/xerj-ai/src/lib.rs@@ -3,18 +3,24 @@//! AI-native features for the xerj search engine.//!//! Provides:+//! - [`embedder`] — Unified backend handle (lexical / proxy / neural) used by the engine//! - [`embed`] — Embedding proxy: async HTTP client for OpenAI-compatible embedding APIs//! - [`local`] — Built-in zero-config deterministic text embedder (feature hashing)+//! - [`neural`] — Built-in neural BERT sentence embedder via candle (feature `neural`)//! - [`chunker`] — Text chunking with sentence-aware splitting and overlap//! - [`memory`] — Agent memory: semantic dedup + recency-blended recallpub mod chunker;pub mod embed;+pub mod embedder;pub mod local;pub mod memory;+#[cfg(feature = "neural")]+pub mod neural;pub use chunker::{Chunk, TextChunker};pub use embed::{EmbeddingProxy, EmbeddingProxyConfig};+pub use embedder::Embedder;pub use local::{local_embed, DEFAULT_DIMS};pub use memory::{AgentMemory, MemoryEntry};diff --git a/engine/crates/xerj-ai/src/neural.rs b/engine/crates/xerj-ai/src/neural.rsnew file mode 100644index 000000000..897abd2af--- /dev/null+++ b/engine/crates/xerj-ai/src/neural.rs@@ -0,0 +1,294 @@+//! Built-in neural sentence embedder — pure-Rust BERT inference via `candle`.+//!+//! This is XERJ's optional *real* semantic embedder. It loads a+//! sentence-transformers BERT model (default `all-MiniLM-L6-v2`, 384-dim)+//! and produces genuine neural embeddings, in-process, with no Python and+//! no external service. It complements the two existing backends:+//!+//! * [`crate::local::local_embed`] — zero-dependency lexical feature-hash+//! (the honest default; fast, deterministic, but *not* semantic).+//! * [`crate::embed::EmbeddingProxy`] — any external OpenAI-compatible+//! `/v1/embeddings` provider (bring-your-own model).+//!+//! Model files are fetched once on first use from the HuggingFace Hub and+//! cached on disk; air-gapped deployments point [`NeuralConfig::local_dir`]+//! at a directory holding `config.json`, `tokenizer.json`, and the+//! safetensors weights instead.+//!+//! Compiled only under the `neural` cargo feature.++use anyhow::{anyhow, Context, Result};+use candle_core::{DType, Device, Tensor};+use candle_nn::VarBuilder;+use candle_transformers::models::bert::{BertModel, Config, DTYPE};+use std::path::{Path, PathBuf};+use tokenizers::{PaddingParams, PaddingStrategy, Tokenizer, TruncationParams};++/// Default sentence encoder: 6-layer MiniLM, 384-dim, ~90 MB.+pub const DEFAULT_MODEL_ID: &str = "sentence-transformers/all-MiniLM-L6-v2";++/// Cap on tokens per passage. MiniLM's positional table is 512; passages are+/// already chunked upstream, so this is a safety clamp, not the usual path.+const MAX_TOKENS: usize = 512;++/// How to obtain the model weights.+#[derive(Debug, Clone)]+pub struct NeuralConfig {+ /// HuggingFace model id (e.g. `sentence-transformers/all-MiniLM-L6-v2`).+ pub model_id: String,+ /// Override the HuggingFace cache directory. `None` uses the default+ /// (`~/.cache/huggingface`).+ pub cache_dir: Option<PathBuf>,+ /// Air-gapped: load `config.json` / `tokenizer.json` / weights from this+ /// directory instead of downloading. Takes precedence over the hub.+ pub local_dir: Option<PathBuf>,+}++impl Default for NeuralConfig {+ fn default() -> Self {+ Self {+ model_id: DEFAULT_MODEL_ID.to_string(),+ cache_dir: None,+ local_dir: None,+ }+ }+}++/// A loaded BERT sentence embedder. Cheap to share behind an `Arc`; `embed`+/// takes `&self` and is safe to call from many threads.+pub struct NeuralEmbedder {+ model: BertModel,+ tokenizer: Tokenizer,+ device: Device,+ dims: usize,+}++impl NeuralEmbedder {+ /// Output dimensionality (hidden size of the loaded model, e.g. 384).+ pub fn dims(&self) -> usize {+ self.dims+ }++ /// Load the model, downloading from the HuggingFace Hub on first use+ /// (unless [`NeuralConfig::local_dir`] is set). **Blocking** — the caller+ /// runs this off the async executor (see [`crate::embedder`]).+ pub fn load(cfg: &NeuralConfig) -> Result<Self> {+ let (config_path, tokenizer_path, weights_path) = match &cfg.local_dir {+ Some(dir) => resolve_local(dir)?,+ None => resolve_from_hub(&cfg.model_id, cfg.cache_dir.as_deref())?,+ };++ let config_json = std::fs::read_to_string(&config_path)+ .with_context(|| format!("read model config {}", config_path.display()))?;+ let config: Config = serde_json::from_str(&config_json)+ .with_context(|| format!("parse model config {}", config_path.display()))?;+ let dims = config.hidden_size;++ let mut tokenizer = Tokenizer::from_file(&tokenizer_path)+ .map_err(|e| anyhow!("load tokenizer {}: {e}", tokenizer_path.display()))?;+ // Pad each batch to its longest member so we can stack into one tensor,+ // and clamp over-long passages to the model's positional limit.+ tokenizer.with_padding(Some(PaddingParams {+ strategy: PaddingStrategy::BatchLongest,+ ..Default::default()+ }));+ tokenizer+ .with_truncation(Some(TruncationParams {+ max_length: MAX_TOKENS,+ ..Default::default()+ }))+ .map_err(|e| anyhow!("configure tokenizer truncation: {e}"))?;++ let device = Device::Cpu;+ // Safetensors is memory-mapped; the file must outlive the model, which+ // it does (candle copies tensors into the VarBuilder-backed model).+ let vb = unsafe {+ VarBuilder::from_mmaped_safetensors(&[weights_path.clone()], DTYPE, &device)+ .with_context(|| format!("map weights {}", weights_path.display()))?+ };+ let model = BertModel::load(vb, &config).map_err(|e| anyhow!("load BERT model: {e}"))?;++ Ok(Self {+ model,+ tokenizer,+ device,+ dims,+ })+ }++ /// Embed a batch of passages into L2-normalized sentence vectors using+ /// attention-masked mean pooling (the sentence-transformers convention).+ /// **Blocking / CPU-bound** — call via `spawn_blocking`.+ pub fn embed_blocking(&self, texts: &[String]) -> Result<Vec<Vec<f32>>> {+ if texts.is_empty() {+ return Ok(vec![]);+ }++ let encodings = self+ .tokenizer+ .encode_batch(texts.to_vec(), true)+ .map_err(|e| anyhow!("tokenize batch: {e}"))?;++ let batch = encodings.len();+ let seq_len = encodings.first().map(|e| e.get_ids().len()).unwrap_or(0);+ if seq_len == 0 {+ // All-empty input — return zero vectors of the right width.+ return Ok(vec![vec![0.0; self.dims]; batch]);+ }++ let mut ids: Vec<u32> = Vec::with_capacity(batch * seq_len);+ let mut mask: Vec<u32> = Vec::with_capacity(batch * seq_len);+ for enc in &encodings {+ ids.extend_from_slice(enc.get_ids());+ mask.extend_from_slice(enc.get_attention_mask());+ }++ let input_ids = Tensor::from_vec(ids, (batch, seq_len), &self.device)+ .map_err(|e| anyhow!("build input_ids tensor: {e}"))?;+ let attention_mask = Tensor::from_vec(mask, (batch, seq_len), &self.device)+ .map_err(|e| anyhow!("build attention_mask tensor: {e}"))?;+ let token_type_ids = input_ids+ .zeros_like()+ .map_err(|e| anyhow!("build token_type_ids: {e}"))?;++ // (batch, seq_len, hidden)+ let hidden = self+ .model+ .forward(&input_ids, &token_type_ids, Some(&attention_mask))+ .map_err(|e| anyhow!("bert forward: {e}"))?;++ // Attention-masked mean pooling: sum(token * mask) / sum(mask).+ let mask_f = attention_mask+ .to_dtype(DType::F32)+ .and_then(|m| m.unsqueeze(2)) // (batch, seq_len, 1)+ .map_err(|e| anyhow!("mask to f32: {e}"))?;+ let summed = hidden+ .broadcast_mul(&mask_f)+ .and_then(|h| h.sum(1)) // (batch, hidden)+ .map_err(|e| anyhow!("masked sum: {e}"))?;+ let counts = mask_f+ .sum(1) // (batch, 1)+ .and_then(|c| c.clamp(1e-9, f32::INFINITY))+ .map_err(|e| anyhow!("mask counts: {e}"))?;+ let mean = summed+ .broadcast_div(&counts)+ .map_err(|e| anyhow!("mean pool: {e}"))?;++ // L2 normalize each row.+ let norm = mean+ .sqr()+ .and_then(|s| s.sum_keepdim(1))+ .and_then(|s| s.sqrt())+ .and_then(|n| n.clamp(1e-12, f32::INFINITY))+ .map_err(|e| anyhow!("l2 norm: {e}"))?;+ let normed = mean+ .broadcast_div(&norm)+ .map_err(|e| anyhow!("normalize: {e}"))?;++ normed+ .to_vec2::<f32>()+ .map_err(|e| anyhow!("read embeddings: {e}"))+ }+}++/// Resolve the three model files from a local directory (air-gapped).+fn resolve_local(dir: &Path) -> Result<(PathBuf, PathBuf, PathBuf)> {+ let config = dir.join("config.json");+ let tokenizer = dir.join("tokenizer.json");+ let weights = find_local_weights(dir)?;+ for (label, p) in [("config.json", &config), ("tokenizer.json", &tokenizer)] {+ if !p.exists() {+ return Err(anyhow!(+ "local model dir {} is missing {label}",+ dir.display()+ ));+ }+ }+ Ok((config, tokenizer, weights))+}++/// Prefer `model.safetensors`; candle cannot read PyTorch `.bin` weights.+fn find_local_weights(dir: &Path) -> Result<PathBuf> {+ let st = dir.join("model.safetensors");+ if st.exists() {+ return Ok(st);+ }+ Err(anyhow!(+ "local model dir {} has no model.safetensors (candle requires safetensors, \+ not pytorch_model.bin)",+ dir.display()+ ))+}++/// Download (or read from cache) the model files from the HuggingFace Hub.+fn resolve_from_hub(+ model_id: &str,+ cache_dir: Option<&Path>,+) -> Result<(PathBuf, PathBuf, PathBuf)> {+ use hf_hub::api::sync::ApiBuilder;++ let mut builder = ApiBuilder::new().with_progress(false);+ if let Some(dir) = cache_dir {+ builder = builder.with_cache_dir(dir.to_path_buf());+ }+ let api = builder+ .build()+ .with_context(|| "init HuggingFace hub client")?;+ let repo = api.model(model_id.to_string());++ let config = repo+ .get("config.json")+ .with_context(|| format!("fetch config.json for {model_id}"))?;+ let tokenizer = repo+ .get("tokenizer.json")+ .with_context(|| format!("fetch tokenizer.json for {model_id}"))?;+ let weights = repo.get("model.safetensors").with_context(|| {+ format!(+ "fetch model.safetensors for {model_id} (candle requires safetensors weights)"+ )+ })?;+ Ok((config, tokenizer, weights))+}++#[cfg(test)]+mod tests {+ use super::*;++ fn cosine(a: &[f32], b: &[f32]) -> f32 {+ a.iter().zip(b).map(|(x, y)| x * y).sum::<f32>()+ }++ /// Live test: downloads MiniLM on first run and checks that neural+ /// embeddings capture semantic similarity (paraphrase > unrelated),+ /// which the lexical feature-hash embedder cannot do. Network + ~90 MB+ /// download, so it is `#[ignore]`d — run explicitly:+ /// cargo test -p xerj-ai --features neural -- --ignored --nocapture+ #[test]+ #[ignore]+ fn neural_captures_semantic_similarity() {+ let emb = NeuralEmbedder::load(&NeuralConfig::default()).expect("load MiniLM");+ assert_eq!(emb.dims(), 384, "MiniLM is 384-dim");++ let texts = vec![+ "A man is playing a guitar on stage.".to_string(),+ "A musician performs with his guitar at a concert.".to_string(),+ "The quarterly financial report shows rising interest rates.".to_string(),+ ];+ let vecs = emb.embed_blocking(&texts).expect("embed");+ assert_eq!(vecs.len(), 3);+ assert_eq!(vecs[0].len(), 384);++ // Each vector is L2-normalized, so cosine == dot product.+ let norm0 = cosine(&vecs[0], &vecs[0]).sqrt();+ assert!((norm0 - 1.0).abs() < 1e-3, "vectors should be L2-normalized");++ let sim_paraphrase = cosine(&vecs[0], &vecs[1]);+ let sim_unrelated = cosine(&vecs[0], &vecs[2]);+ eprintln!("paraphrase cos = {sim_paraphrase:.4}, unrelated cos = {sim_unrelated:.4}");+ assert!(+ sim_paraphrase > sim_unrelated + 0.15,+ "neural embedder must rank the paraphrase far above the unrelated \+ sentence (got paraphrase={sim_paraphrase:.3}, unrelated={sim_unrelated:.3})"+ );+ }+}diff --git a/engine/crates/xerj-autoindex/src/catalog.rs b/engine/crates/xerj-autoindex/src/catalog.rsindex 5a068480a..aade95dfa 100644--- a/engine/crates/xerj-autoindex/src/catalog.rs+++ b/engine/crates/xerj-autoindex/src/catalog.rs@@ -53,7 +53,7 @@ pub fn catalog_mapping() -> Value {pub const GOTCHAS: &[&str] = &["hybrid search: use {\"query\":{\"hybrid\":{\"queries\":[…]}}} ONLY — retriever.rrf is a silent stub and rank.rrf is ignored on this engine",- "the semantic embedder is hash-bucket LEXICAL (384-dim): this is hybrid lexical+vector retrieval, NOT neural semantic understanding",+ "semantic_text fields are embedded server-side: the DEFAULT is the built-in LEXICAL feature-hash embedder (384-dim hybrid lexical+vector, NOT neural) — start the server with `--embed-mode neural` (built-in BERT) or `--embed-mode proxy` for true neural semantics","semantic queries ignore _source filtering and return the ~8KB *_vector field in _source — strip client-side","exact filters use TOP-LEVEL keyword fields (term on .keyword subfields returns 0 hits on this engine)","all dates are normalized to RFC3339 UTC millis; mappings use strict_date_optional_time||epoch_millis",@@ -180,7 +180,7 @@ pub fn build_sample_queries(pd: &PlanDataset, correlations: &[KeyCorr]) -> Vec<Vif let Some(sf) = &pd.semantic_field {out.push(json!({"class": "hybrid_lexical_vector",- "title": format!("Hybrid lexical+vector (RRF) on {sf} — hash-bucket embedder, lexical not neural"),+ "title": format!("Hybrid lexical+vector (RRF) on {sf} — embedder set server-side (lexical by default; neural/proxy if configured)"),"request": format!("POST /{}/_search", pd.index),"body": {"query": {"hybrid": {"queries": [{"query": {"match": {(sf.clone()): word.clone()}}, "weight": 1},@@ -296,7 +296,7 @@ pub fn render_map(s.push_str(&format!("### `{}`\n\n", g("index_name")));if let Some(sem) = d.get("semantic_field").filter(|v| v.is_string()) {s.push_str(&format!(- "semantic body field: `{}` (hybrid lexical+vector — hash-bucket embedder, lexical, not neural)\n\n",+ "semantic body field: `{}` (hybrid lexical+vector; embedder set server-side — lexical by default, neural/proxy if configured)\n\n",pretty_val(sem)));}diff --git a/engine/crates/xerj-autoindex/src/infer/mod.rs b/engine/crates/xerj-autoindex/src/infer/mod.rsindex a0eeabcf6..0c4225e47 100644--- a/engine/crates/xerj-autoindex/src/infer/mod.rs+++ b/engine/crates/xerj-autoindex/src/infer/mod.rs@@ -410,7 +410,7 @@ pub fn infer_fields(if let Some((i, _)) = best {specs[i].es_type = "semantic_text".into();specs[i].notes.push(- "hybrid lexical+vector body (hash-bucket embedder — lexical, not neural)".into(),+ "hybrid lexical+vector body (embedded server-side: lexical by default, neural/proxy if configured)".into(),);}}diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rsindex 5124eab33..c231b0f36 100644--- a/engine/crates/xerj-common/src/config.rs+++ b/engine/crates/xerj-common/src/config.rs@@ -625,12 +625,30 @@ impl Default for LogsConfig {// ─────────────────────────────────────────────────────────────────────────────-/// External embedding service settings.+/// Embedding backend settings.///-/// **4 settings.**+/// XERJ can embed `semantic_text` fields three ways, chosen by [`mode`]:+/// * `"lexical"` — the zero-dependency built-in feature-hash embedder+/// (deterministic, 384-dim, no model, no network). This is the honest+/// default: lexical, *not* neural semantic understanding.+/// * `"neural"` — a built-in BERT sentence embedder (all-MiniLM-L6-v2 by+/// default) that runs in-process via `candle`. The model weights are+/// downloaded once on first use (or read from [`local_model_dir`] for+/// air-gapped deployments). Requires the binary to be built with the+/// `neural` cargo feature; otherwise this falls back to lexical.+/// * `"proxy"` — call an external OpenAI-compatible `/v1/embeddings`+/// endpoint ([`default_endpoint`]). Lets customers plug in ANY embedding+/// model / provider they already run.+/// * `"auto"` (default) — use the proxy when [`default_endpoint`] is set,+/// otherwise lexical. This preserves the historical behavior exactly.+///+/// **8 settings.**#[derive(Debug, Clone, Serialize, Deserialize)]#[serde(default, deny_unknown_fields)]pub struct EmbeddingConfig {+ /// Backend selector: `"auto"` (default), `"lexical"`, `"neural"`, or+ /// `"proxy"`. Unknown values are treated as `"auto"`.+ pub mode: String,/// OpenAI-compatible endpoint URL (default: `""` — disabled).pub default_endpoint: String,/// Model name to request from the endpoint (default: `""`).@@ -639,15 +657,29 @@ pub struct EmbeddingConfig {pub batch_size: usize,/// HTTP timeout for embedding requests in ms (default: `5000`).pub timeout_ms: u64,+ /// Neural backend: HuggingFace model id to load (default+ /// `sentence-transformers/all-MiniLM-L6-v2`, a 384-dim sentence encoder).+ pub neural_model: String,+ /// Neural backend: directory to cache downloaded model weights. Empty+ /// (default) uses the standard HuggingFace cache (`~/.cache/huggingface`).+ pub model_cache_dir: String,+ /// Neural backend: if set, load `config.json`, `tokenizer.json`, and the+ /// safetensors weights from this local directory instead of downloading+ /// — for air-gapped / offline deployments. Empty (default) = download.+ pub local_model_dir: String,}impl Default for EmbeddingConfig {fn default() -> Self {Self {+ mode: "auto".to_string(),default_endpoint: String::new(),default_model: String::new(),batch_size: 64,timeout_ms: 5000,+ neural_model: "sentence-transformers/all-MiniLM-L6-v2".to_string(),+ model_cache_dir: String::new(),+ local_model_dir: String::new(),}}}diff --git a/engine/crates/xerj-engine/Cargo.toml b/engine/crates/xerj-engine/Cargo.tomlindex d93ee1b06..f39e8589c 100644--- a/engine/crates/xerj-engine/Cargo.toml+++ b/engine/crates/xerj-engine/Cargo.toml@@ -45,6 +45,13 @@ xxhash-rust = { workspace = true }# — SipHash was >21% of on-CPU time under the shard write locks at c8.rustc-hash = { workspace = true }+[features]+default = []+# Built-in neural BERT embedder (candle). Propagates to xerj-ai. Off by+# default so the standard build stays lean and cross-compiles without the+# candle/tokenizer toolchain; enable via xerj-server's `neural` feature.+neural = ["xerj-ai/neural"]+[dev-dependencies]tempfile = { workspace = true }tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex 2185f9bfe..c3be719c5 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -463,12 +463,15 @@ pub struct Index {/// runs hold the snapshot for the duration of one batch.merge_config: xerj_common::config::MergeConfig,- /// Optional embedding proxy used by the `semantic` query type- /// (v0.7-P2). None when `Config.embedding.default_endpoint` is- /// empty — semantic queries then return a clear 400 telling the- /// operator to configure an endpoint. Reused across queries- /// (semaphore + retry budget are per-proxy).- embedding_proxy: Arc<Option<xerj_ai::embed::EmbeddingProxy>>,+ /// Embedding backend for `semantic` / `semantic_text` (v0.7-P2).+ /// One of lexical (built-in, default), an external proxy, or the+ /// built-in neural BERT embedder — selected by `Config.embedding.mode`.+ /// [`xerj_ai::Embedder::is_active`] is false only for the lexical+ /// fallback; an active backend embeds arbitrary query text, while the+ /// lexical fallback embeds only `semantic_text` fields (embedded the+ /// same way at ingest). Reused across queries (proxy semaphore / neural+ /// model are shared behind the `Arc`).+ embedder: Arc<xerj_ai::Embedder>,// ── Per-index metrics ─────────────────────────────────────────────────────/// Total search queries executed.@@ -836,7 +839,7 @@ impl Index {)),merge_in_progress: Arc::new(std::sync::atomic::AtomicBool::new(false)),merge_config: config.merge.clone(),- embedding_proxy: Arc::new(make_embedding_proxy(&config.embedding)),+ embedder: Arc::new(make_embedder(&config.embedding)),dv_cache: Arc::new(dashmap::DashMap::new()),sort_shadow_cache: Arc::new(dashmap::DashMap::new()),sort_shadow_fields: Arc::new(dashmap::DashMap::new()),@@ -1037,7 +1040,7 @@ impl Index {)),merge_in_progress: Arc::new(std::sync::atomic::AtomicBool::new(false)),merge_config: config.merge.clone(),- embedding_proxy: Arc::new(make_embedding_proxy(&config.embedding)),+ embedder: Arc::new(make_embedder(&config.embedding)),dv_cache: Arc::new(dashmap::DashMap::new()),sort_shadow_cache: Arc::new(dashmap::DashMap::new()),sort_shadow_fields: Arc::new(dashmap::DashMap::new()),@@ -3854,16 +3857,18 @@ impl Index {};// Embed the query text with the effective embedder:- // * EmbeddingProxy when configured (high-quality, same as ingest),- // * else the built-in deterministic embedder — but ONLY for+ // * An ACTIVE backend (neural BERT or external proxy) embeds the+ // query text directly — the same backend used at ingest.+ // * Otherwise the built-in lexical embedder — but ONLY for// `semantic_text` fields, which were embedded that same way at// ingest. A `semantic` query against a plain dense_vector field- // with no proxy configured still returns the original 400 (there+ // with no active backend still returns the original 400 (there// is no comparable stored vector to match against).- let query_vec = if let Some(proxy) = &*self.embedding_proxy {- // `embed_batch` takes Vec<String>; we only have one text but the- // proxy's batching keeps the wire format stable for callers.- let mut vectors = proxy+ let query_vec = if self.embedder.is_active() {+ // `embed_batch` takes Vec<String>; we only have one text but+ // batching keeps the wire format stable for callers.+ let mut vectors = self+ .embedder.embed_batch(vec![text.to_string()]).await.map_err(|e| {@@ -3875,7 +3880,7 @@ impl Index {Some(v) if !v.is_empty() => v,_ => {return Err(EngineError::Common(xerj_common::XerjError::invalid_query(- "embedding proxy returned no vector for semantic query",+ "embedder returned no vector for semantic query",)));}}@@ -3884,8 +3889,9 @@ impl Index {} else {return Err(EngineError::Common(xerj_common::XerjError::invalid_query("semantic query requires either a `semantic_text` field (auto-embedded \- with the built-in embedder) or `embedding.default_endpoint` configured \- to an OpenAI-compatible /v1/embeddings endpoint.",+ with the built-in lexical embedder) or an active embedding backend \+ (`embedding.mode = \"neural\"`, or `\"proxy\"` with \+ `embedding.default_endpoint` set to an OpenAI-compatible endpoint).",)));};@@ -3944,20 +3950,23 @@ impl Index {// Chunk once and embed EACH overlapping chunk. `chunk_vecs` is one// embedding per passage (>= 1). Short text (<= chunk_size) yields a// single chunk == the exact pre-chunking behavior.- let chunk_vecs: Vec<Vec<f32>> = if let Some(proxy) = &*self.embedding_proxy {+ let chunk_vecs: Vec<Vec<f32>> = if self.embedder.is_active() {+ // Active backend (neural or proxy): chunk, then embed each+ // passage. The same backend + chunking is used at query time.let chunks = semantic_chunker().chunk(&text, None);let chunk_texts: Vec<String> = if chunks.len() <= 1 {vec![text.clone()]} else {chunks.iter().map(|c| c.text.clone()).collect()};- let vs = proxy.embed_batch(chunk_texts).await.map_err(|e| {+ let vs = self.embedder.embed_batch(chunk_texts).await.map_err(|e| {EngineError::Common(xerj_common::XerjError::invalid_query(format!("semantic_text embed failed for field [{field}]: {e}")))})?;vs.into_iter().filter(|v| !v.is_empty()).collect()} else {+ // Lexical fallback: deterministic per-chunk feature-hash.local_chunk_vectors(&text, dims)};// Pooled vector for `target`: single chunk is stored as-is (no@@ -13193,31 +13202,82 @@ fn build_highlight_fragments(fragments}-/// Build an `EmbeddingProxy` from `Config.embedding`. Returns `None`-/// when no endpoint is configured — semantic queries then return a-/// helpful 400 ("configure embedding.default_endpoint") instead of-/// a runtime crash.-fn make_embedding_proxy(- cfg: &xerj_common::config::EmbeddingConfig,-) -> Option<xerj_ai::embed::EmbeddingProxy> {- if cfg.default_endpoint.is_empty() {- return None;+/// Build the effective [`xerj_ai::Embedder`] from `Config.embedding`.+///+/// Backend selection follows `embedding.mode`:+/// * `"proxy"` — external OpenAI-compatible endpoint (requires+/// `default_endpoint`).+/// * `"neural"` — the built-in BERT embedder (requires the `neural`+/// cargo feature; falls back to lexical with a warning otherwise).+/// * `"lexical"` — the built-in feature-hash embedder.+/// * `"auto"` (default / unknown) — proxy when `default_endpoint` is set,+/// else lexical. Preserves the historical behavior exactly.+///+/// Any misconfiguration degrades to the lexical fallback (never a crash);+/// a `semantic` query with no active backend against a plain vector field+/// still returns a helpful 400.+fn make_embedder(cfg: &xerj_common::config::EmbeddingConfig) -> xerj_ai::Embedder {+ let mode = cfg.mode.trim().to_ascii_lowercase();+ let want_proxy = mode == "proxy" || (mode != "neural" && mode != "lexical" && !cfg.default_endpoint.is_empty());++ if mode == "neural" {+ return make_neural_embedder(cfg);+ }++ if want_proxy {+ if cfg.default_endpoint.is_empty() {+ warn!("embedding.mode=proxy but embedding.default_endpoint is empty — falling back to lexical");+ return xerj_ai::Embedder::lexical();+ }+ let proxy_cfg = xerj_ai::embed::EmbeddingProxyConfig {+ endpoint: cfg.default_endpoint.clone(),+ api_key: std::env::var("XERJ_EMBEDDING_API_KEY").ok(),+ model: cfg.default_model.clone(),+ timeout_secs: cfg.timeout_ms / 1000,+ max_concurrent: 4,+ max_retries: 3,+ };+ return match xerj_ai::embed::EmbeddingProxy::new(proxy_cfg) {+ Ok(p) => {+ info!(endpoint = %cfg.default_endpoint, "embedding backend: external proxy");+ xerj_ai::Embedder::proxy(p)+ }+ Err(e) => {+ warn!(error = %e, "embedding proxy init failed — falling back to lexical");+ xerj_ai::Embedder::lexical()+ }+ };}- let proxy_cfg = xerj_ai::embed::EmbeddingProxyConfig {- endpoint: cfg.default_endpoint.clone(),- api_key: std::env::var("XERJ_EMBEDDING_API_KEY").ok(),- model: cfg.default_model.clone(),- timeout_secs: cfg.timeout_ms / 1000,- max_concurrent: 4,- max_retries: 3,++ xerj_ai::Embedder::lexical()+}++/// Construct the built-in neural embedder when compiled in; otherwise warn+/// and fall back to lexical so a `mode=neural` config never bricks startup.+#[cfg(feature = "neural")]+fn make_neural_embedder(cfg: &xerj_common::config::EmbeddingConfig) -> xerj_ai::Embedder {+ let model_id = if cfg.neural_model.trim().is_empty() {+ xerj_ai::neural::DEFAULT_MODEL_ID.to_string()+ } else {+ cfg.neural_model.clone()};- match xerj_ai::embed::EmbeddingProxy::new(proxy_cfg) {- Ok(p) => Some(p),- Err(e) => {- warn!(error = %e, "embedding proxy init failed — semantic queries will 400");- None- }- }+ let neural_cfg = xerj_ai::neural::NeuralConfig {+ model_id,+ cache_dir: (!cfg.model_cache_dir.is_empty()).then(|| cfg.model_cache_dir.clone().into()),+ local_dir: (!cfg.local_model_dir.is_empty()).then(|| cfg.local_model_dir.clone().into()),+ };+ info!(model = %neural_cfg.model_id, "embedding backend: built-in neural BERT (loads on first use)");+ xerj_ai::Embedder::neural(neural_cfg)+}++#[cfg(not(feature = "neural"))]+fn make_neural_embedder(_cfg: &xerj_common::config::EmbeddingConfig) -> xerj_ai::Embedder {+ warn!(+ "embedding.mode=neural requested but this binary was built without the `neural` \+ cargo feature — falling back to the built-in lexical embedder. Rebuild with \+ `--features neural` (xerj-server) to enable in-process BERT embeddings."+ );+ xerj_ai::Embedder::lexical()}// ── HNSW persistence helpers (module-level, called from sync `Index::open`)diff --git a/engine/crates/xerj-server/Cargo.toml b/engine/crates/xerj-server/Cargo.tomlindex bf4a2cdfc..786c20d9c 100644--- a/engine/crates/xerj-server/Cargo.toml+++ b/engine/crates/xerj-server/Cargo.toml@@ -56,6 +56,14 @@ prost.workspace = true# because it never returns freed heap pages to the OS. jemalloc is# fragmentation-friendly and actively madvise()'s freed ranges, which brings# RSS back down after each flush — critical under the 4 GB cgroup cap.+[features]+default = []+# Ship the built-in neural BERT embedder (candle + MiniLM, download-on-+# first-run). Off by default; build the release binary with+# `cargo build --release -p xerj-server --features neural` to enable+# in-process neural embeddings (set `embedding.mode = "neural"` at runtime).+neural = ["xerj-engine/neural"]+[target.'cfg(not(target_env = "msvc"))'.dependencies]tikv-jemallocator = "0.6"diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rsindex f042f581a..e3b8cdff1 100644--- a/engine/crates/xerj-server/src/main.rs+++ b/engine/crates/xerj-server/src/main.rs@@ -77,6 +77,7 @@ struct CliArgs {config: Option<PathBuf>,data_dir: Option<String>,insecure: bool,+ embed_mode: Option<String>,}fn parse_args() -> CliArgs {@@ -84,6 +85,7 @@ fn parse_args() -> CliArgs {let mut config = None;let mut data_dir = None;let mut insecure = false;+ let mut embed_mode = None;while let Some(arg) = args.next() {match arg.as_str() {@@ -96,6 +98,9 @@ fn parse_args() -> CliArgs {"--insecure" | "-k" => {insecure = true;}+ "--embed-mode" => {+ embed_mode = args.next();+ }"--help" | "-h" => {print_help();std::process::exit(0);@@ -115,6 +120,7 @@ fn parse_args() -> CliArgs {config,data_dir,insecure,+ embed_mode,}}@@ -129,6 +135,9 @@ fn print_help() {--config, -c <PATH> Path to TOML config file\n\--data-dir, -d <PATH> Override data directory\n\--insecure, -k Disable TLS\n\+ --embed-mode <MODE> Embedding backend: lexical | neural | proxy | auto\n\+ (neural = built-in BERT, downloads on first use;\n\+ requires a binary built with --features neural)\n\--help, -h Show this help\n\--version, -V Print version and exit\n\\n\@@ -138,8 +147,9 @@ fn print_help() {xerj autoindex map print the discovered data map\n\\n\ENVIRONMENT:\n\- XERJ_LOG Log level filter (default: info)\n\- XERJ_CONFIG Config file path\n",+ XERJ_LOG Log level filter (default: info)\n\+ XERJ_CONFIG Config file path\n\+ XERJ_EMBED_MODE Embedding backend (lexical|neural|proxy|auto)\n",env!("CARGO_PKG_VERSION"));}@@ -231,6 +241,25 @@ fn load_config(args: &CliArgs) -> Result<Config> {cfg.auth.enabled = false;}+ // Embedding backend override: `--embed-mode` flag or `XERJ_EMBED_MODE`+ // env (flag wins). Accepts `lexical` | `neural` | `proxy` | `auto`.+ if let Some(mode) = args+ .embed_mode+ .clone()+ .or_else(|| std::env::var("XERJ_EMBED_MODE").ok())+ {+ let mode = mode.trim().to_ascii_lowercase();+ match mode.as_str() {+ "lexical" | "neural" | "proxy" | "auto" => {+ info!("embedding.mode = {mode} (from CLI/env)");+ cfg.embedding.mode = mode;+ }+ other => {+ warn!("ignoring unknown --embed-mode '{other}' (use lexical|neural|proxy|auto)");+ }+ }+ }+Ok(cfg)}@@ -570,6 +599,7 @@ async fn run_cli_index(cmd: IndexCmdArgs) -> Result<()> {config: cmd.config.clone(),data_dir: cmd.data_dir.clone(),insecure: true,+ embed_mode: None,};let mut cfg = load_config(&fake_cli)?;cfg.tls.enabled = false;diff --git a/recipes/README.md b/recipes/README.mdindex 0358ff085..1fb82fe36 100644--- a/recipes/README.md+++ b/recipes/README.md@@ -65,12 +65,14 @@ Every recipe honors `XERJ_URL` (default `http://localhost:9200`).## Honest notes-- The built-in embedder is **lexical** (feature-hashing over word +- character n-grams, L2-normalized), not neural. It captures vocabulary- overlap well and is perfect for offline demos and CI, but it won't- match true synonyms. For production semantics, point XERJ at an- OpenAI-compatible embeddings endpoint (`[embedding]` in `xerj.toml`) —- the recipes need no change.+- The **default** built-in embedder is **lexical** (feature-hashing over+ word + character n-grams, L2-normalized), not neural. It captures+ vocabulary overlap well and is perfect for offline demos and CI, but it+ won't match true synonyms. For real neural semantics — with no change to+ any recipe — start the server with a different `--embed-mode`: `neural`+ (built-in in-process BERT, all-MiniLM-L6-v2, downloads once on first use;+ build with `--features neural`) or `proxy` (any OpenAI-compatible+ embeddings endpoint, set `[embedding] default_endpoint` in `xerj.toml`).- `memory_agent.py` recalls with **text (BM25) relevance** scores(unbounded, higher = better) by default. Add `"semantic": true` to the`_recall` body to have XERJ embed the query server-side and recall bydiff --git a/recipes/semantic_search.py b/recipes/semantic_search.pyindex c815f5761..477c9b56c 100644--- a/recipes/semantic_search.py+++ b/recipes/semantic_search.py@@ -18,11 +18,20 @@ Run it:Optional env:XERJ_URL (default http://localhost:9200)-To use a real neural embedding model instead of the built-in lexical-embedder, point XERJ at any OpenAI-compatible endpoint in xerj.toml:- [embedding]- default_endpoint = "https://api.openai.com/v1/embeddings"- default_model = "text-embedding-3-small"+The built-in default embedder is *lexical* (feature-hashing), not neural.+For real neural semantics — with no change to the index, mapping, or the+three queries below — pick a different backend when you start the server:++ # Built-in neural BERT (all-MiniLM-L6-v2), in-process, downloads once.+ # Requires a binary built with `--features neural`.+ xerj --insecure --data-dir ./data --embed-mode neural++ # …or any external OpenAI-compatible endpoint (bring your own model):+ xerj --insecure --data-dir ./data --embed-mode proxy+ # with, in xerj.toml:+ # [embedding]+ # default_endpoint = "https://api.openai.com/v1/embeddings"+ # default_model = "text-embedding-3-small""""import json