AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(embed): refuse startup when the neural model cannot load (#1877)
xerj-org23b5f1216dd4
diff --git a/docs/recipes/air-gapped-deployment.md b/docs/recipes/air-gapped-deployment.mdindex 75c5410a4..d1adabd61 100644--- a/docs/recipes/air-gapped-deployment.md+++ b/docs/recipes/air-gapped-deployment.md@@ -9,10 +9,14 @@ an optional, separately staged path.- The default embedding mode is **lexical feature hashing**. It needs no model,no embedding service, and no network connection.-- Neural mode is optional. Without `embedding.local_model_dir`, the first- operation that needs an embedding downloads `config.json`, `tokenizer.json`,- and `model.safetensors` for `sentence-transformers/all-MiniLM-L6-v2` from the- Hugging Face Hub. With that directory set, XERJ reads those files locally.+- Neural mode is optional. The model is loaded when the process starts, not+ on the first embed. Without `embedding.local_model_dir`, startup downloads+ `config.json`, `tokenizer.json`, and `model.safetensors` for+ `sentence-transformers/all-MiniLM-L6-v2` from the Hugging Face Hub+ (`HF_ENDPOINT` selects a mirror). If that download fails, the process+ refuses to start rather than indexing with the lexical embedder. With+ `local_model_dir` set, XERJ reads those files locally and does not contact+ the Hub.- `proxy` calls only the configured OpenAI-compatible`embedding.default_endpoint`. The experimental `onnx-experimental` backendneeds an ONNX-enabled build and explicit local assets; neither is the basediff --git a/docs/recipes/production-deployment.md b/docs/recipes/production-deployment.mdindex 9bfa1c8b4..4a1d16d81 100644--- a/docs/recipes/production-deployment.md+++ b/docs/recipes/production-deployment.md@@ -467,9 +467,14 @@ in-process BERT sentence encoder (default `all-MiniLM-L6-v2`, 384-dim, ~90 MB),no Python, no external service. Select it per node with `[embedding] mode ="neural"` (or `--embed-mode neural`, or `XERJ_EMBED_MODE=neural`).-On a connected host the model downloads itself **once, on first use**, from the-HuggingFace Hub and is cached for every later start. On an **air-gapped** host-there is no Hub to reach, so stage the model files yourself.+On a connected host the model is downloaded from the HuggingFace Hub and loaded+**at startup**, then cached for every later start. If that load fails the+process **refuses to start** with one error that names the cause, the command+`xerj --embed-mode lexical`, `embedding.local_model_dir`, and `HF_ENDPOINT`+(a Hub mirror). It does not stay up and fail every ingest, and it does not+substitute lexical vectors — those are not comparable with neural vectors.+Explicit `proxy` already fails closed the same way. On an **air-gapped** host+there is no Hub to reach, so stage the model files yourself before starting.### Step 1 — download the three files on a connected machine@@ -505,26 +510,21 @@ local_model_dir = "/opt/xerj/models/all-MiniLM-L6-v2" # holds the 3 files abov```When `local_model_dir` is set it takes precedence over the Hub — XERJ makes-**no outbound network call** for the model. Selecting the neural backend logs a-line when the semantic index is built:+**no outbound network call** for the model. The weights load at startup, before+the data directory is opened. A correct staging just works and makes no network+call. A **missing or wrong file refuses startup** with the offending path:```-INFO embedding backend: built-in neural BERT (loads on first use)+embedding.mode=neural failed to load model sentence-transformers/all-MiniLM-L6-v2: local model dir /opt/xerj/models/... is missing tokenizer.json+… has no model.safetensors (candle requires safetensors, not pytorch_model.bin).+Refusing to start. …+To index without a model, restart with: xerj --embed-mode lexical```-The weights themselves load lazily, from your staged directory, on the first-`semantic_text` ingest (not at boot). A correct staging just works — the local-path is silent on success and makes no network call. A **missing or wrong file-fails loud** with the offending path, which is your signal to fix the staging:--```-local model dir /opt/xerj/models/... is missing tokenizer.json-… has no model.safetensors (candle requires safetensors, not pytorch_model.bin)-```--(The `neural embedder: model ready` log line you may have seen elsewhere is the-*Hub-download* path's completion message; the offline `local_model_dir` path-does not print it.)+(The `neural embedder: model ready` log line is the *Hub-download* path's+completion message; the offline `local_model_dir` path does not print it.+After a successful load, opening a semantic index logs+`embedding backend: built-in neural BERT (loaded)`.)> **Slim builds.** A `--no-default-features` build omits the neural backend;> `mode = "neural"` on such a binary logs a warning and falls back to thediff --git a/engine/crates/xerj-ai/src/embedder.rs b/engine/crates/xerj-ai/src/embedder.rsindex 812909c17..29c51c0e9 100644--- a/engine/crates/xerj-ai/src/embedder.rs+++ b/engine/crates/xerj-ai/src/embedder.rs@@ -12,8 +12,10 @@//! service ([`crate::embed::EmbeddingProxy`]). Bring any model/provider.//! * [`Embedder::Neural`] — the built-in BERT sentence encoder//! ([`crate::neural`]), running in-process via `candle`. Compiled only-//! under the `neural` cargo feature; the model is loaded lazily on first-//! use (download-on-first-run) so startup stays instant.+//! under the `neural` cargo feature. The server loads it at startup+//! (`Embedder::preload_neural`); a blocked download refuses the process+//! rather than failing every ingest. The handle still shares one+//! process-wide model.//! * [`Embedder::Onnx`] — the experimental in-process ONNX Runtime backend.//! It is compiled only under `onnx-experimental`, requires explicit model//! and tokenizer paths, and loads the model lazily on first use.@@ -215,7 +217,8 @@ pub enum Embedder {Lexical,/// External OpenAI-compatible embedding service.Proxy(EmbeddingProxy),- /// Built-in neural BERT embedder (candle). Lazily loaded on first use.+ /// Built-in neural BERT embedder (candle). The server preloads it at+ /// startup; a handle built without that preload still loads on first embed.#[cfg(feature = "neural")]Neural(NeuralHandle),/// Experimental local ONNX Runtime sentence encoder.@@ -234,12 +237,36 @@ impl Embedder {Embedder::Proxy(proxy)}- /// A lazily-loaded built-in neural embedder.+ /// A built-in neural embedder. Call [`preload_neural`] first when a+ /// failed download must surface before the first embed.#[cfg(feature = "neural")]pub fn neural(cfg: crate::neural::NeuralConfig) -> Self {Embedder::Neural(NeuralHandle::new(cfg))}+ /// Load the neural model into the process-wide cache.+ ///+ /// The server calls this at startup (#1840). On success, every later+ /// [`Self::neural`] handle with the same config reuses the loaded model+ /// instead of downloading again. On failure the cell stays empty — this+ /// never installs the lexical embedder.+ #[cfg(feature = "neural")]+ pub fn preload_neural(+ cfg: &crate::neural::NeuralConfig,+ ) -> Result<std::sync::Arc<crate::neural::NeuralEmbedder>> {+ let cell = shared_neural_cell(cfg);+ if let Some(ready) = cell.get() {+ return Ok(std::sync::Arc::clone(ready));+ }+ let loaded = std::sync::Arc::new(crate::neural::NeuralEmbedder::load(cfg)?);+ // A concurrent preload may win `set`. Keep whichever model landed in+ // the cell so every handle shares one copy.+ match cell.set(std::sync::Arc::clone(&loaded)) {+ Ok(()) => Ok(loaded),+ Err(_) => Ok(cell.get().cloned().unwrap_or(loaded)),+ }+ }+#[cfg(feature = "onnx-experimental")]pub fn onnx(cfg: OnnxConfig) -> Self {Embedder::Onnx(OnnxHandle::new(cfg))@@ -870,6 +897,7 @@ mod neural_handle_tests {model_id: "test/model-shared".into(),cache_dir: Some(PathBuf::from("/tmp/xerj-neural-shared-cache")),local_dir: None,+ hub_endpoint: None,};let first = NeuralHandle::new(cfg.clone());let second = NeuralHandle::new(cfg);@@ -884,11 +912,13 @@ mod neural_handle_tests {model_id: "test/model-a".into(),cache_dir: None,local_dir: None,+ hub_endpoint: None,});let second = NeuralHandle::new(NeuralConfig {model_id: "test/model-b".into(),cache_dir: None,local_dir: None,+ hub_endpoint: None,});assert!(!Arc::ptr_eq(&first.cell, &second.cell));@@ -900,6 +930,7 @@ mod neural_handle_tests {model_id: "test/model-reclaimable".into(),cache_dir: None,local_dir: None,+ hub_endpoint: None,};let weak = {let handle = NeuralHandle::new(cfg.clone());diff --git a/engine/crates/xerj-ai/src/neural.rs b/engine/crates/xerj-ai/src/neural.rsindex c2dd5a31d..c3c3d45af 100644--- a/engine/crates/xerj-ai/src/neural.rs+++ b/engine/crates/xerj-ai/src/neural.rs@@ -10,10 +10,13 @@//! * [`crate::embed::EmbeddingProxy`] — any external OpenAI-compatible//! `/v1/embeddings` provider (bring-your-own model).//!-//! Model files are fetched once on first use from the HuggingFace Hub and-//! cached on disk; air-gapped deployments point [`NeuralConfig::local_dir`]-//! at a directory holding `config.json`, `tokenizer.json`, and the-//! safetensors weights instead.+//! Model files are resolved when the embedder is loaded. The server does+//! that at startup when neural is requested (#1840): a blocked Hub or a+//! missing local directory refuses the process once, instead of failing+//! every later ingest. Air-gapped deployments point+//! [`NeuralConfig::local_dir`] at a directory holding `config.json`,+//! `tokenizer.json`, and the safetensors weights. A HuggingFace mirror is+//! `HF_ENDPOINT` (or [`NeuralConfig::hub_endpoint`]).//!//! Compiled only under the `neural` cargo feature.@@ -87,6 +90,12 @@ pub struct NeuralConfig {/// Air-gapped: load `config.json` / `tokenizer.json` / weights from this/// directory instead of downloading. Takes precedence over the hub.pub local_dir: Option<PathBuf>,+ /// Override the HuggingFace Hub base URL. `None` uses `HF_ENDPOINT` when+ /// that variable is set, otherwise `https://huggingface.co`. Tests point+ /// this at a closed local port so a blocked download never touches the+ /// network. `ApiBuilder::new()` does not read `HF_ENDPOINT` itself+ /// (hf-hub 0.4 only does that in `from_env`).+ pub hub_endpoint: Option<String>,}impl Default for NeuralConfig {@@ -95,6 +104,7 @@ impl Default for NeuralConfig {model_id: DEFAULT_MODEL_ID.to_string(),cache_dir: None,local_dir: None,+ hub_endpoint: None,}}}@@ -114,13 +124,14 @@ impl NeuralEmbedder {self.dims}- /// Load the model, downloading from the HuggingFace Hub on first use- /// (unless [`NeuralConfig::local_dir`] is set). **Blocking** — the caller- /// runs this off the async executor (see [`crate::embedder`]).+ /// Load the model, downloading from the HuggingFace Hub when it is not+ /// already cached (unless [`NeuralConfig::local_dir`] is set).+ /// **Blocking** — the server runs this at startup, off the async+ /// executor (see [`crate::Embedder::preload_neural`]).pub fn load(cfg: &NeuralConfig) -> Result<Self> {let (config_path, tokenizer_path, weights_path) = match &cfg.local_dir {Some(dir) => resolve_local(dir)?,- None => resolve_from_hub(&cfg.model_id, cfg.cache_dir.as_deref())?,+ None => resolve_from_hub(cfg)?,};let config_json = std::fs::read_to_string(&config_path)@@ -443,7 +454,7 @@ fn compute_asset_identity() -> std::result::Result<AssetIdentity, AssetIdentityError> {let (config_path, tokenizer_path, weights_path) = match &cfg.local_dir {Some(dir) => resolve_local_assets(dir)?,- None => resolve_from_hub(&cfg.model_id, cfg.cache_dir.as_deref())+ None => resolve_from_hub(cfg)// The hub error names the model id and the cache path; neither may// reach a client, so only the generic label travels..map_err(|e| AssetIdentityError::new("the model download", format!("{e:#}")))?,@@ -537,22 +548,46 @@ fn find_local_weights(dir: &Path) -> std::result::Result<PathBuf, AssetIdentityE))}+/// The single startup error for a neural model that cannot be loaded.+///+/// Callers that refuse to start (#1840) use this text so the operator sees+/// the cause once, plus the two ways out: lexical, or an offline / mirrored+/// model. Lexical is named as a different command, never as a silent swap —+/// those vectors are not comparable with neural vectors (#1189).+pub fn startup_failure_message(model_id: &str, error: &anyhow::Error) -> String {+ format!(+ "embedding.mode=neural failed to load model {model_id}: {error:#}. \+ Refusing to start. Lexical vectors are not comparable with neural vectors, \+ so this process will not switch to the lexical embedder or keep running and \+ fail every ingest. To index without a model, restart with: \+ xerj --embed-mode lexical. To keep neural offline, set \+ embedding.local_model_dir to a directory containing config.json, \+ tokenizer.json and model.safetensors. To use a HuggingFace mirror, set \+ HF_ENDPOINT to the mirror base URL and start again."+ )+}+/// Download (or read from cache) the model files from the HuggingFace Hub.///-/// The first launch with `--embed-mode neural` on a fresh machine pulls the-/// weights (~90 MB for MiniLM); every launch after that is an instant cache-/// hit. We surface that clearly so a user staring at a terminal knows the-/// one-time download is happening rather than a hang. A progress bar is shown-/// (hf-hub writes it to stderr) for the same reason.-fn resolve_from_hub(- model_id: &str,- cache_dir: Option<&Path>,-) -> Result<(PathBuf, PathBuf, PathBuf)> {+/// The server calls this at startup when `--embed-mode neural` is requested,+/// before it listens. The first start on a fresh machine pulls the weights+/// (~90 MB for MiniLM); every start after that is a cache hit. A progress bar+/// is shown (hf-hub writes it to stderr) so a download is not mistaken for a+/// hang.+fn resolve_from_hub(cfg: &NeuralConfig) -> Result<(PathBuf, PathBuf, PathBuf)> {use hf_hub::api::sync::ApiBuilder;- let mut builder = ApiBuilder::new().with_progress(true);- if let Some(dir) = cache_dir {- builder = builder.with_cache_dir(dir.to_path_buf());+ let model_id = cfg.model_id.as_str();+ // `ApiBuilder::new()` does not read `HF_ENDPOINT` (hf-hub 0.4 reserved+ // that for `from_env`, which also relocates the cache via `HF_HOME`).+ // Keep the historical cache directory and apply only the mirror URL,+ // unless the caller set `model_cache_dir` / `hub_endpoint`.+ let mut builder = ApiBuilder::new().with_progress(true).with_retries(0);+ if let Some(dir) = &cfg.cache_dir {+ builder = builder.with_cache_dir(dir.clone());+ }+ if let Some(endpoint) = hub_endpoint_override(cfg) {+ builder = builder.with_endpoint(endpoint);}let api = builder.build()@@ -590,6 +625,22 @@ fn resolve_from_hub(Ok((config, tokenizer, weights))}+/// Explicit [`NeuralConfig::hub_endpoint`], otherwise a non-empty `HF_ENDPOINT`.+fn hub_endpoint_override(cfg: &NeuralConfig) -> Option<String> {+ if let Some(endpoint) = cfg+ .hub_endpoint+ .as_deref()+ .map(str::trim)+ .filter(|endpoint| !endpoint.is_empty())+ {+ return Some(endpoint.to_string());+ }+ std::env::var("HF_ENDPOINT")+ .ok()+ .map(|endpoint| endpoint.trim().to_string())+ .filter(|endpoint| !endpoint.is_empty())+}+#[cfg(test)]mod tests {use super::*;diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rsindex 30d36b6b6..81d18e3b6 100644--- a/engine/crates/xerj-common/src/config.rs+++ b/engine/crates/xerj-common/src/config.rs@@ -1556,11 +1556,15 @@ impl LogsConfig {/// default: lexical, *not* neural semantic understanding./// * `"neural"` — a built-in BERT sentence embedder (all-MiniLM-L6-v2 by/// default) that runs in-process via `candle`. The model weights are-/// downloaded once on first use (or read from+/// downloaded and loaded at startup (or read from/// [`local_model_dir`](Self::local_model_dir) for-/// air-gapped deployments). The neural backend ships in the standard-/// binary; a `--no-default-features` slim build omits it and falls back-/// to lexical.+/// air-gapped deployments; `HF_ENDPOINT` selects a Hub mirror). If that+/// load fails, the process refuses to start (#1840) instead of accepting+/// traffic and failing every ingest. It does not substitute the lexical+/// embedder: those vectors are not comparable with neural vectors, the+/// same fail-closed rule as explicit `"proxy"` (#1189). The neural+/// backend ships in the standard binary; a `--no-default-features` slim+/// build omits it and falls back to lexical, and identity reports lexical./// * `"proxy"` — call an external OpenAI-compatible `/v1/embeddings`/// endpoint ([`default_endpoint`](Self::default_endpoint)). Lets/// customers plug in ANY embedding model / provider they already run. An explicit `mode = "proxy"` with adiff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex cb080febb..775defc4c 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -37185,6 +37185,9 @@ fn configured_neural_config(model_id,cache_dir: (!cfg.model_cache_dir.is_empty()).then(|| cfg.model_cache_dir.clone().into()),local_dir: (!cfg.local_model_dir.is_empty()).then(|| cfg.local_model_dir.clone().into()),+ // The mirror URL is `HF_ENDPOINT`, read inside the hub client. There+ // is no config field for it; tests inject one on `NeuralConfig` directly.+ hub_endpoint: None,}}@@ -38599,7 +38602,7 @@ fn make_embedder(cfg: &xerj_common::config::EmbeddingConfig) -> Result<xerj_ai::|| (mode != "neural" && mode != "lexical" && !cfg.default_endpoint.is_empty());if mode == "neural" {- return Ok(make_neural_embedder(cfg));+ return make_neural_embedder(cfg);}if mode == "onnx-experimental" {@@ -38735,27 +38738,66 @@ fn make_onnx_embedder(_cfg: &xerj_common::config::EmbeddingConfig) -> Result<xer)))}-/// Construct the built-in neural embedder when compiled in; otherwise warn-/// and fall back to lexical so a `mode=neural` config never bricks startup.+/// Load the neural model when this binary can, before any ingest.+///+/// Explicit `mode=neural` fails closed (#1840), the same shape as explicit+/// `mode=proxy` (#1189): a download or file error is returned, and the lexical+/// embedder is not installed in its place. Those vectors are not comparable.+/// The server calls this at startup so the failure happens once, before it+/// listens. A slim build without the `neural` feature still returns `Ok` —+/// `make_neural_embedder` warns and substitutes lexical, and identity reports+/// lexical, because there is no model to load.+pub fn ensure_neural_ready(cfg: &xerj_common::config::EmbeddingConfig) -> Result<()> {+ if !cfg.mode.trim().eq_ignore_ascii_case("neural") {+ return Ok(());+ }+ #[cfg(feature = "neural")]+ {+ let neural_cfg = configured_neural_config(cfg);+ if let Err(error) = xerj_ai::Embedder::preload_neural(&neural_cfg) {+ return Err(EngineError::Common(xerj_common::XerjError::embedding(+ xerj_ai::neural::startup_failure_message(&neural_cfg.model_id, &error),+ )));+ }+ Ok(())+ }+ #[cfg(not(feature = "neural"))]+ {+ Ok(())+ }+}++/// Construct the built-in neural embedder when compiled in.+///+/// The model is loaded here (and at server startup via [`ensure_neural_ready`]).+/// A failure is an error, not a lexical embedder. A slim build without the+/// feature warns and returns lexical so a `mode=neural` config on that binary+/// still starts; identity reports lexical for that case.#[cfg(feature = "neural")]-fn make_neural_embedder(cfg: &xerj_common::config::EmbeddingConfig) -> xerj_ai::Embedder {+fn make_neural_embedder(cfg: &xerj_common::config::EmbeddingConfig) -> Result<xerj_ai::Embedder> {// Exactly the resolution the execution identity content-addresses — see// `configured_neural_config`. Duplicating it here once let the reported// identity describe a different model than the one that loads.+ ensure_neural_ready(cfg)?;let neural_cfg = configured_neural_config(cfg);- info!(model = %neural_cfg.model_id, "embedding backend: built-in neural BERT (loads on first use)");- xerj_ai::Embedder::neural(neural_cfg)+ info!(+ model = %neural_cfg.model_id,+ "embedding backend: built-in neural BERT (loaded)"+ );+ Ok(xerj_ai::Embedder::neural(neural_cfg))}#[cfg(not(feature = "neural"))]-fn make_neural_embedder(_cfg: &xerj_common::config::EmbeddingConfig) -> xerj_ai::Embedder {+fn make_neural_embedder(_cfg: &xerj_common::config::EmbeddingConfig) -> Result<xerj_ai::Embedder> {+ // Identity reports lexical for this build. The warning is the whole+ // signal: there is no model download that can fail later.warn!("embedding.mode=neural requested but this is a slim build (compiled with \`--no-default-features`, no neural backend) — falling back to the built-in lexical \embedder. The standard release binary ships the neural embedder; use it, or rebuild \xerj-server with default features.");- xerj_ai::Embedder::lexical()+ Ok(xerj_ai::Embedder::lexical())}// ── HNSW persistence helpers (module-level, called from sync `Index::open`)diff --git a/engine/crates/xerj-server/Cargo.toml b/engine/crates/xerj-server/Cargo.tomlindex 9c935402f..eb9c2c5c3 100644--- a/engine/crates/xerj-server/Cargo.toml+++ b/engine/crates/xerj-server/Cargo.toml@@ -64,8 +64,9 @@ prost.workspace = true[features]# The shipped binary is batteries-included: the built-in neural BERT# embedder (candle + MiniLM) is ON by default, so end users just add-# `--embed-mode neural` at runtime and the model downloads itself on first-# use — no special build, no separate binary. Build with+# `--embed-mode neural` at runtime and the model downloads and loads at+# startup — a blocked Hub refuses to start, it does not fail every ingest.+# No special build, no separate binary. Build with# `--no-default-features` for a slimmer binary without the neural backend# (the lexical + proxy backends still work).default = ["neural", "decide-local"]diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rsindex 3a971938a..135c7157f 100644--- a/engine/crates/xerj-server/src/main.rs+++ b/engine/crates/xerj-server/src/main.rs@@ -330,8 +330,13 @@ fn help_text(feedback: bool) -> String {\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20onnx-experimental\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20lexical built-in feature-hash (default; offline, not neural)\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20neural built-in BERT (all-MiniLM-L6-v2) — real semantics,\n\- \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20in-process; auto-downloads the model (~90 MB) on\n\- \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20first use, then runs from cache. Just add the flag.\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20in-process. The model (~90 MB) loads at startup:\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20downloaded from the HuggingFace Hub, or read from\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20embedding.local_model_dir, then cached. If that load\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20fails, the process refuses to start. It does not keep\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20running and fail every ingest, and it does not write\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20lexical vectors. Restart with: xerj --embed-mode lexical\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20Or set HF_ENDPOINT to a Hub mirror and start again.\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20proxy external OpenAI-compatible /v1/embeddings endpoint\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20auto proxy if embedding.default_endpoint is set, else lexical\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20onnx-experimental local all-MiniLM-L6-v2-compatible FP32\n\@@ -2297,10 +2302,141 @@ fn build_runtime(worker_threads: usize) -> Result<tokio::runtime::Runtime> {.context("build tokio runtime")}+/// #1840: a neural startup whose Hub cannot be reached fails once, here,+/// before any listener exists. This lives in `xerj-server` because that+/// crate's default features include `neural`, which is what+/// `cargo test --workspace` actually runs. `xerj-ai`'s own tests do not+/// enable the feature unless asked.+#[cfg(all(test, feature = "neural"))]+mod neural_startup_tests {+ use xerj_ai::neural::{startup_failure_message, NeuralConfig};+ use xerj_ai::Embedder;++ /// Closed port on loopback. Connection refused is local and immediate;+ /// nothing is sent off the machine.+ const UNREACHABLE_HUB: &str = "http://127.0.0.1:1";++ #[test]+ fn unreachable_hub_refuses_once_and_does_not_install_lexical() {+ let cache = tempfile::tempdir().unwrap();+ let cfg = NeuralConfig {+ model_id: "sentence-transformers/all-MiniLM-L6-v2".into(),+ cache_dir: Some(cache.path().to_path_buf()),+ local_dir: None,+ hub_endpoint: Some(UNREACHABLE_HUB.into()),+ };+ // `NeuralEmbedder` is not `Debug`, so `expect_err` cannot name the Ok side.+ let error = match Embedder::preload_neural(&cfg) {+ Ok(_) => panic!("a closed hub must not load a model"),+ Err(error) => error,+ };+ let message = startup_failure_message(&cfg.model_id, &error);+ assert!(+ message.contains("127.0.0.1") || message.contains("Connection refused"),+ "the cause must be the closed local hub, not a real download: {message}"+ );+ assert!(+ message.contains("xerj --embed-mode lexical"),+ "the error must give the lexical command: {message}"+ );+ assert!(+ message.contains("embedding.local_model_dir"),+ "the error must name the offline model directory: {message}"+ );+ assert!(+ message.contains("HF_ENDPOINT"),+ "the error must name the mirror variable: {message}"+ );+ assert!(+ message.contains("Refusing to start"),+ "the error must say the process will not start: {message}"+ );+ assert!(+ !message+ .to_ascii_lowercase()+ .contains("falling back to lexical"),+ "must not describe a lexical swap: {message}"+ );+ // The shared cell stays empty, so a later embed cannot pretend the+ // model loaded. Preload's return type is the neural model itself:+ // failure cannot hand back a lexical embedder.+ assert!(+ Embedder::preload_neural(&cfg).is_err(),+ "a second attempt must still fail rather than reuse a lexical stand-in"+ );+ }++ #[test]+ fn missing_local_model_refuses_with_the_same_startup_error() {+ let dir = tempfile::tempdir().unwrap();+ let cfg = xerj_common::config::EmbeddingConfig {+ mode: "neural".into(),+ local_model_dir: dir.path().to_string_lossy().into_owned(),+ ..Default::default()+ };+ let error = xerj_engine::index::ensure_neural_ready(&cfg)+ .expect_err("an empty local model directory must refuse startup");+ let message = error.to_string();+ assert!(+ message.contains("embedding.mode=neural"),+ "must name the setting: {message}"+ );+ assert!(+ message.contains("xerj --embed-mode lexical"),+ "must give the lexical command: {message}"+ );+ assert!(+ message.contains("model.safetensors") || message.contains("missing"),+ "must name the missing local asset: {message}"+ );+ assert!(+ !message.contains("127.0.0.1"),+ "a local directory must not contact the hub: {message}"+ );+ }+}+#[cfg(test)]mod help_text_tests {use super::{auth_disabled_line, help_text};+ /// #1840: `--embed-mode neural` must say the model loads at startup and+ /// that a failed download refuses to start, with the lexical command.+ #[test]+ fn neural_help_says_a_failed_download_refuses_to_start() {+ let help = help_text(true);+ let neural = help+ .lines()+ .position(|line| line.contains("neural built-in BERT"))+ .expect("neural mode must be documented");+ let window: String = help+ .lines()+ .skip(neural)+ .take(12)+ .collect::<Vec<_>>()+ .join("\n");+ assert!(+ window.contains("loads at startup"),+ "neural help must say the model loads at startup:\n{window}"+ );+ assert!(+ window.contains("refuses to start"),+ "neural help must say a failed load refuses to start:\n{window}"+ );+ assert!(+ window.contains("xerj --embed-mode lexical"),+ "neural help must give the lexical command:\n{window}"+ );+ assert!(+ window.contains("embedding.local_model_dir"),+ "neural help must name the offline model directory:\n{window}"+ );+ assert!(+ window.contains("HF_ENDPOINT"),+ "neural help must name the Hub mirror variable:\n{window}"+ );+ }+/// A loopback `--insecure` boot says nothing off this host can connect,/// then used to say anyone on the network can write. Those two lines/// cannot both be true.@@ -3039,6 +3175,35 @@ async fn async_main() -> Result<()> {);}+ // 3d. Explicit mode=neural loads the model before the data directory, the+ // admin key, or any listener (#1840). A blocked Hub used to boot a+ // node that then failed every ingest. Explicit proxy already refuses+ // to start rather than substitute lexical vectors (#1189); neural does+ // the same, because those vectors are not comparable. A slim build+ // without the neural feature keeps its documented lexical fallback+ // (identity reports lexical) — there is no model to load.+ if cfg.embedding.mode.trim().eq_ignore_ascii_case("neural") {+ #[cfg(feature = "neural")]+ {+ let embedding = cfg.embedding.clone();+ info!(+ model = %embedding.neural_model,+ "embedding.mode=neural: loading the model before opening the data directory"+ );+ tokio::task::spawn_blocking(move || {+ xerj_engine::index::ensure_neural_ready(&embedding)+ })+ .await+ .context("neural embedder load task panicked")??;+ info!("embedding.mode=neural: model loaded");+ }+ #[cfg(not(feature = "neural"))]+ {+ // Slim binary. `make_neural_embedder` warns and uses lexical when+ // an index actually embeds. Do not refuse to start.+ }+ }+// 4. Data directorystd::fs::create_dir_all(&cfg.server.data_dir).with_context(|| format!("create data dir {}", cfg.server.data_dir))?;