Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(ai): wire experimental ONNX through autoindex

bugerc14aa5cb1c3d
diff --git a/docs/EXPERIMENTAL_ONNX.md b/docs/EXPERIMENTAL_ONNX.md
index cd0203894..67ef128d7 100644
--- a/docs/EXPERIMENTAL_ONNX.md
+++ b/docs/EXPERIMENTAL_ONNX.md
@@ -1,63 +1,155 @@
# Experimental ONNX embedding backend
-Status: embedding-library prototype, **not a server or CLI mode**.
+Status: end-to-end experimental server mode for GNU/Linux glibc builds. It is
+off by default. XERJ's default remains the non-neural lexical feature hash;
+`--embed-mode neural` remains the built-in Candle backend.
-XERJ's supported built-in neural backend remains Candle and
-`--embed-mode neural` continues to mean Candle. Default and musl builds do not
-compile ONNX Runtime.
+## Exact model contract
-The opt-in `xerj-ai` feature provides the exact FP32 MiniLM inference path and
-a bounded length-aware microbatch API:
+This is not a generic ONNX-model interface. The supplied graph must be an FP32
+`sentence-transformers/all-MiniLM-L6-v2`-compatible feature-extraction export:
+
+- int64 inputs named `input_ids`, `attention_mask`, and `token_type_ids`;
+- rank-3 token output named `last_hidden_state` or `token_embeddings`;
+- output width exactly 384;
+- a `tokenizer.json` from the same model/export.
+
+XERJ applies attention-mask mean pooling and L2 normalization. It truncates at
+512 tokens. A mapping that declares another vector width is rejected.
+
+## Copy-paste workflow
+
+Export a compatible model and inspect its interface:
+
+```bash
+python3 -m venv /tmp/xerj-onnx-export
+/tmp/xerj-onnx-export/bin/pip install 'optimum[onnxruntime]' onnx
+/tmp/xerj-onnx-export/bin/optimum-cli export onnx \
+ --model sentence-transformers/all-MiniLM-L6-v2 \
+ --task feature-extraction \
+ /tmp/xerj-minilm-onnx
+
+/tmp/xerj-onnx-export/bin/python - <<'PY'
+import onnx
+p = "/tmp/xerj-minilm-onnx/model.onnx"
+m = onnx.load(p)
+print("inputs:", [(v.name, [d.dim_value for d in v.type.tensor_type.shape.dim])
+ for v in m.graph.input])
+print("outputs:", [(v.name, [d.dim_value for d in v.type.tensor_type.shape.dim])
+ for v in m.graph.output])
+PY
+
+sha256sum \
+ /tmp/xerj-minilm-onnx/model.onnx \
+ /tmp/xerj-minilm-onnx/tokenizer.json
+```
+
+Confirm the three required input names, a supported output name, and width 384.
+Export tools can change their file layout or output names; XERJ deliberately
+fails instead of guessing.
+
+Build the opt-in server:
```bash
cd engine
-cargo run --release -p xerj-ai --features onnx-experimental \
- --example onnx_experimental -- /path/model.onnx /path/tokenizer.json
+cargo build --release -j 32 -p xerj-server --features onnx-experimental
+```
+
+Start it with explicit local assets:
+
+```bash
+target/release/xerj \
+ --insecure \
+ --data-dir /tmp/xerj-onnx-data \
+ --embed-mode onnx-experimental \
+ --onnx-model /tmp/xerj-minilm-onnx/model.onnx \
+ --onnx-tokenizer /tmp/xerj-minilm-onnx/tokenizer.json
```
-The caller supplies an ONNX model with `input_ids`, `attention_mask`, and
-`token_type_ids` inputs and a `[batch, sequence, 384]` `last_hidden_state`
-output. The current verified artifact is the official FP32
-`sentence-transformers/all-MiniLM-L6-v2` ONNX export.
+No model is downloaded at server startup. Incorrect paths, an incompatible
+build, and invalid admission limits fail startup with a corrective error. The
+first real semantic inference prints one concise activation message containing
+the verified model and tokenizer hashes. ONNX Runtime messages below warning
+are hidden by default; use `XERJ_ONNX_LOG=info` or `verbose` for diagnosis.
+
+In another terminal, index a corpus:
+
+```bash
+target/release/xerj autoindex /path/to/corpus \
+ --url http://localhost:9200 \
+ --prefix finance-onnx \
+ --fresh
+
+target/release/xerj autoindex map \
+ --url http://localhost:9200 \
+ --prefix finance-onnx
+```
+
+ONNX runs only for fields inferred as `semantic_text`, normally a sufficiently
+long body field. Short or structured datasets may infer none. Check
+`autoindex --dry-run`, the data map's `semantic_field`, and the server's
+activation log before claiming an ONNX result.
+
+Query the mapped semantic field:
+
+```bash
+curl -s http://localhost:9200/finance-onnx-*/_search \
+ -H 'content-type: application/json' \
+ -d '{"query":{"semantic":{"field":"body","query":"Which quarter had the largest operating-margin decline?","k":10}},"size":5}'
+```
+
+## Restart and vector-space safety
+
+Every ONNX semantic index stores `embedding_identity.json` with SHA-256 model
+and tokenizer fingerprints plus dimensions, pooling, and token limit. Restart
+with the same assets. XERJ refuses:
+
+- another model or tokenizer at restart;
+- switching an ONNX-pinned index to another backend;
+- enabling ONNX in place on a populated marker-less semantic index;
+- caller-supplied derived vectors whose identity cannot be verified.
+
+The error tells the operator to restore the original assets or re-run
+autoindex with `--fresh` and a new prefix. XERJ never silently mixes vector
+spaces.
+
+## Admission and errors
-Safety properties:
+One safe ONNX Runtime session is shared per complete model configuration.
+Length-aware microbatches are serialized through it. Before model loading or
+tokenization, shared admission enforces:
-- safe `ort` API only;
-- one session behind a mutex;
-- bounded pending inputs, documents per batch, and padded token slots;
-- length-aware grouping;
-- exact restoration to caller input order;
-- clear backpressure and invalid-budget errors.
+- `onnx_max_inflight_calls` (default 8);
+- `onnx_max_input_bytes_per_call` (default 8 MiB);
+- `onnx_max_inflight_input_bytes` (default 32 MiB);
+- `onnx_max_pending`, microbatch size, and padded-token limits.
-## Measured prototype results
+Overload returns retryable HTTP 429. Autoindex does not call that junk and does
+not journal the affected source file complete; correct the pressure/config and
+rerun the same command to resume.
-On the controlled 128-document mixed-length benchmark, exact FP32 ONNX plus
-the retained scheduler processed 116.671 documents/s versus 9.045 documents/s
-for Candle using the identical precomputed length-aware batch plan: **12.90x
-at the `xerj-ai` embedding layer**. The median minimum same-document
-Candle/ONNX cosine was `0.9999991655`.
+These settings are available under `[embedding]` in the TOML configuration.
+The server validates impossible or zero admission limits before serving.
-This is not a claim of 12.90x end-to-end XERJ indexing. The benchmark excludes
-document extraction, HTTP handling, persistence, and HNSW construction. ONNX
-also used 16 intra-op threads; the measured CPU-efficiency improvement was
-2.92x, smaller than the wall-time result. A server-level comparison is required
-before publishing an indexing-throughput claim.
+## Measured performance and limits
-Measured stripped full-server binary sizes were:
+On the controlled 128-document mixed-length benchmark, optimized Candle
+processed 9.045 documents/s and FP32 ONNX with the retained scheduler processed
+116.671 documents/s: **12.90x at the embedding layer**. The median minimum
+same-document Candle/ONNX cosine was `0.9999991655`; output ordering was
+preserved. CPU-efficiency improved 2.92x.
-- current Candle build: 36.06 MiB;
-- Candle plus ONNX: 54.81 MiB (**+18.75 MiB / +52.0%**);
-- ONNX-only experimental build: 52.49 MiB (**+16.43 MiB / +45.6%**).
+This is not a 12.90x end-to-end indexing claim. It excludes extraction, HTTP,
+lexical indexing, persistence, HNSW construction, and contention. Full
+FinanceBench autoindex time must be measured, not projected.
-The roughly 90 MiB ONNX model is a separate runtime asset comparable to the
-existing Candle safetensors asset. The much larger static ONNX Runtime archive
-downloaded while compiling is not added wholesale to the final binary.
+Measured stripped server binaries:
-The bundled `ort` binaries do not cover XERJ's musl release targets. Default
-and musl builds therefore remain Candle-only; production adoption requires a
-reproducible musl packaging strategy or an explicitly narrower target matrix.
+- Candle: 36.06 MiB;
+- Candle plus ONNX: 54.81 MiB (+18.75 MiB / +52.0%);
+- ONNX-only experimental build: 52.49 MiB (+16.43 MiB / +45.6%).
-This is deliberately not exposed in `xerj --help`. Server integration still
-needs model acquisition, backend fingerprinting/reindex compatibility,
-platform packaging, lifecycle metrics, cancellation, and end-to-end retrieval
-quality gates. Do not describe XERJ as supporting `--embed-mode onnx`.
+The approximately 90 MiB model is a separate runtime asset. Bundled `ort`
+binaries do not cover XERJ's musl release targets, so standard musl releases
+remain Candle-only. ONNX production adoption still needs a supported target
+matrix, end-to-end quality gates, and full-corpus throughput/resource results.
diff --git a/engine/Cargo.lock b/engine/Cargo.lock
index 7bab24fa1..c50e18b30 100644
--- a/engine/Cargo.lock
+++ b/engine/Cargo.lock
@@ -5140,6 +5140,7 @@ dependencies = [
"reqwest",
"serde",
"serde_json",
+ "sha2",
"thiserror 2.0.18",
"tokenizers",
"tokio",
diff --git a/engine/crates/xerj-ai/Cargo.toml b/engine/crates/xerj-ai/Cargo.toml
index 63db37e5b..e46c00452 100644
--- a/engine/crates/xerj-ai/Cargo.toml
+++ b/engine/crates/xerj-ai/Cargo.toml
@@ -33,13 +33,15 @@ ort = { version = "2.0.0-rc.12", optional = true, default-features = false, feat
"std", "ndarray", "tracing", "download-binaries", "tls-rustls",
"copy-dylibs", "api-24",
] }
+sha2 = { version = "0.10", optional = true }
[features]
default = []
# Bundles the neural BERT embedder. Adds ~candle + tokenizers to the build.
neural = ["dep:candle-core", "dep:candle-nn", "dep:candle-transformers", "dep:tokenizers", "dep:hf-hub"]
-# Embedding-layer prototype only; this does not add a server `--embed-mode`.
-onnx-experimental = ["dep:ort", "dep:tokenizers"]
+# Experimental local ONNX backend; xerj-server exposes it only when the
+# feature is propagated and explicitly selected at runtime.
+onnx-experimental = ["dep:ort", "dep:tokenizers", "dep:sha2"]
[dev-dependencies]
tokio = { version = "1", features = ["full"] }
diff --git a/engine/crates/xerj-ai/src/embedder.rs b/engine/crates/xerj-ai/src/embedder.rs
index 4f18d1e93..8838eada1 100644
--- a/engine/crates/xerj-ai/src/embedder.rs
+++ b/engine/crates/xerj-ai/src/embedder.rs
@@ -1,6 +1,6 @@
//! Unified embedding backend.
//!
-//! XERJ embeds `semantic_text` fields through one of three interchangeable
+//! XERJ embeds `semantic_text` fields through one of four interchangeable
//! backends, all behind a single [`Embedder`] handle so the engine's ingest
//! and query paths never branch on the backend:
//!
@@ -14,17 +14,30 @@
//! ([`crate::neural`]), running in-process via `candle`. Compiled only
//! under the `neural` cargo feature; the model is loaded lazily on first
//! use (download-on-first-run) so startup stays instant.
+//! * [`Embedder::Onnx`] — the experimental in-process ONNX Runtime backend.
+//! It is compiled only under `onnx-experimental`, requires explicit model
+//! and tokenizer paths, and loads the model lazily on first use.
//!
-//! [`Embedder::is_active`] distinguishes a *real* embedder (proxy or neural)
-//! from the lexical fallback — the query path only auto-embeds against the
-//! lexical backend for `semantic_text` fields, matching how they were
-//! embedded at ingest.
+//! [`Embedder::is_active`] distinguishes a neural/proxy/ONNX backend from the
+//! lexical fallback. The query path nevertheless auto-embeds `semantic_text`
+//! fields with whichever backend indexed them, including lexical, so ingest
+//! and query vectors always use the same embedding identity.
use anyhow::{anyhow, Result};
use crate::embed::EmbeddingProxy;
use crate::local::{local_embed, DEFAULT_DIMS};
+#[cfg(feature = "onnx-experimental")]
+type OnnxCell = tokio::sync::OnceCell<std::sync::Arc<crate::onnx::OnnxEmbedder>>;
+
+#[cfg(feature = "onnx-experimental")]
+struct OnnxShared {
+ cell: OnnxCell,
+ calls: std::sync::Arc<tokio::sync::Semaphore>,
+ bytes: std::sync::Arc<tokio::sync::Semaphore>,
+}
+
#[cfg(feature = "neural")]
type NeuralCell = tokio::sync::OnceCell<std::sync::Arc<crate::neural::NeuralEmbedder>>;
@@ -66,6 +79,9 @@ pub enum Embedder {
/// Built-in neural BERT embedder (candle). Lazily loaded on first use.
#[cfg(feature = "neural")]
Neural(NeuralHandle),
+ /// Experimental local ONNX Runtime sentence encoder.
+ #[cfg(feature = "onnx-experimental")]
+ Onnx(OnnxHandle),
}
impl Embedder {
@@ -85,10 +101,16 @@ impl Embedder {
Embedder::Neural(NeuralHandle::new(cfg))
}
- /// `true` when a *real* embedder (neural or external proxy) is configured;
- /// `false` for the lexical fallback. The query path uses this to decide
- /// whether to embed arbitrary query text (active) or restrict to
- /// `semantic_text` fields embedded the same lexical way at ingest.
+ #[cfg(feature = "onnx-experimental")]
+ pub fn onnx(cfg: OnnxConfig) -> Self {
+ Embedder::Onnx(OnnxHandle::new(cfg))
+ }
+
+ /// `true` when a Candle neural, experimental ONNX, or external proxy
+ /// embedder is configured; `false` for the lexical fallback. The query
+ /// path uses this to decide whether to embed arbitrary query text (active)
+ /// or restrict to `semantic_text` fields embedded the same lexical way at
+ /// ingest.
pub fn is_active(&self) -> bool {
!matches!(self, Embedder::Lexical)
}
@@ -100,6 +122,8 @@ impl Embedder {
Embedder::Proxy(_) => "external proxy (OpenAI-compatible /v1/embeddings)",
#[cfg(feature = "neural")]
Embedder::Neural(_) => "neural BERT (built-in, candle)",
+ #[cfg(feature = "onnx-experimental")]
+ Embedder::Onnx(_) => "neural BERT (experimental ONNX Runtime)",
}
}
@@ -113,8 +137,305 @@ impl Embedder {
.map_err(|e| anyhow!("embedding proxy failed: {e}")),
#[cfg(feature = "neural")]
Embedder::Neural(handle) => handle.embed_batch(texts).await,
+ #[cfg(feature = "onnx-experimental")]
+ Embedder::Onnx(handle) => handle.embed_batch(texts).await,
+ }
+ }
+}
+
+#[cfg(feature = "onnx-experimental")]
+#[derive(Clone, Debug, Hash, PartialEq, Eq)]
+pub struct OnnxConfig {
+ pub model_path: std::path::PathBuf,
+ pub tokenizer_path: std::path::PathBuf,
+ /// Content fingerprints are part of the shared-session cache key and are
+ /// rechecked immediately before load, preventing same-path asset swaps.
+ pub model_sha256: String,
+ pub tokenizer_sha256: String,
+ pub intra_threads: usize,
+ pub microbatch: crate::onnx::MicrobatchConfig,
+ pub max_inflight_calls: usize,
+ pub max_input_bytes_per_call: usize,
+ pub max_inflight_input_bytes: usize,
+}
+
+#[cfg(feature = "onnx-experimental")]
+#[derive(Debug, thiserror::Error)]
+#[error("{reason}")]
+pub struct OnnxAdmissionError {
+ pub reason: String,
+}
+
+#[cfg(feature = "onnx-experimental")]
+pub struct OnnxHandle {
+ cfg: OnnxConfig,
+ shared: std::sync::Arc<OnnxShared>,
+}
+
+#[cfg(feature = "onnx-experimental")]
+impl OnnxHandle {
+ fn new(cfg: OnnxConfig) -> Self {
+ use std::collections::HashMap;
+ use std::sync::{Mutex, OnceLock, Weak};
+ static CELLS: OnceLock<Mutex<HashMap<OnnxConfig, Weak<OnnxShared>>>> = OnceLock::new();
+ let mut cells = CELLS
+ .get_or_init(|| Mutex::new(HashMap::new()))
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner());
+ if let Some(shared) = cells.get(&cfg).and_then(Weak::upgrade) {
+ return Self { cfg, shared };
+ }
+ cells.retain(|_, shared| shared.strong_count() > 0);
+ let shared = std::sync::Arc::new(OnnxShared {
+ cell: OnnxCell::new(),
+ calls: std::sync::Arc::new(tokio::sync::Semaphore::new(cfg.max_inflight_calls.max(1))),
+ bytes: std::sync::Arc::new(tokio::sync::Semaphore::new(
+ cfg.max_inflight_input_bytes.max(1),
+ )),
+ });
+ cells.insert(cfg.clone(), std::sync::Arc::downgrade(&shared));
+ Self { cfg, shared }
+ }
+
+ async fn get(&self) -> Result<std::sync::Arc<crate::onnx::OnnxEmbedder>> {
+ self.shared
+ .cell
+ .get_or_try_init(|| async {
+ let cfg = self.cfg.clone();
+ let model = tokio::task::spawn_blocking(move || {
+ let model_bytes = std::fs::read(&cfg.model_path).map_err(|e| {
+ anyhow!("read ONNX model {}: {e}", cfg.model_path.display())
+ })?;
+ let tokenizer_bytes = std::fs::read(&cfg.tokenizer_path).map_err(|e| {
+ anyhow!("read ONNX tokenizer {}: {e}", cfg.tokenizer_path.display())
+ })?;
+ let actual_model = sha256_bytes(&model_bytes);
+ let actual_tokenizer = sha256_bytes(&tokenizer_bytes);
+ if actual_model != cfg.model_sha256 || actual_tokenizer != cfg.tokenizer_sha256
+ {
+ return Err(anyhow!(
+ "ONNX assets changed after configuration; refusing to load a \
+ different vector space from the same path (model expected {}, \
+ actual {}; tokenizer expected {}, actual {}). Restart with the \
+ intended assets or reindex under a new prefix",
+ cfg.model_sha256,
+ actual_model,
+ cfg.tokenizer_sha256,
+ actual_tokenizer
+ ));
+ }
+ let embedder = crate::onnx::OnnxEmbedder::load_bytes(
+ &model_bytes,
+ &tokenizer_bytes,
+ cfg.intra_threads,
+ )?;
+ tracing::info!(
+ model_sha256 = %cfg.model_sha256,
+ tokenizer_sha256 = %cfg.tokenizer_sha256,
+ dimensions = crate::onnx::DIMS,
+ "experimental ONNX embedding backend active; first semantic inference loaded the verified model"
+ );
+ Ok::<_, anyhow::Error>(embedder)
+ })
+ .await
+ .map_err(|e| anyhow!("ONNX model load task panicked: {e}"))??;
+ Ok::<_, anyhow::Error>(std::sync::Arc::new(model))
+ })
+ .await
+ .cloned()
+ }
+
+ async fn embed_batch(&self, texts: Vec<String>) -> Result<Vec<Vec<f32>>> {
+ if texts.is_empty() {
+ return Ok(Vec::new());
+ }
+ if texts.len() > self.cfg.microbatch.max_pending {
+ return Err(anyhow::Error::new(OnnxAdmissionError {
+ reason: format!(
+ "ONNX request rejected before tokenization: {} texts exceed max_pending={}; \
+ split the request and retry",
+ texts.len(),
+ self.cfg.microbatch.max_pending
+ ),
+ }));
+ }
+ let input_bytes = texts
+ .iter()
+ .try_fold(0usize, |total, text| total.checked_add(text.len()))
+ .ok_or_else(|| {
+ anyhow::Error::new(OnnxAdmissionError {
+ reason: "ONNX input byte count overflowed; split the request".into(),
+ })
+ })?;
+ let (_call_permit, _byte_permits) = self.try_admit(input_bytes)?;
+ let model = self.get().await?;
+ let limits = self.cfg.microbatch;
+ tokio::task::spawn_blocking(move || model.embed_scheduled_blocking(&texts, limits))
+ .await
+ .map_err(|e| anyhow!("ONNX embed task panicked: {e}"))?
+ }
+
+ fn try_admit(
+ &self,
+ input_bytes: usize,
+ ) -> Result<(
+ tokio::sync::OwnedSemaphorePermit,
+ tokio::sync::OwnedSemaphorePermit,
+ )> {
+ if input_bytes > self.cfg.max_input_bytes_per_call {
+ return Err(anyhow::Error::new(OnnxAdmissionError {
+ reason: format!(
+ "ONNX request rejected before tokenization: input is {input_bytes} bytes, \
+ per-call limit is {} bytes; split the request and retry",
+ self.cfg.max_input_bytes_per_call
+ ),
+ }));
+ }
+ let byte_permits = u32::try_from(input_bytes.max(1)).map_err(|_| {
+ anyhow::Error::new(OnnxAdmissionError {
+ reason: format!(
+ "ONNX request rejected before tokenization: {input_bytes} input bytes \
+ exceed the semaphore permit range; split the request"
+ ),
+ })
+ })?;
+ let call = self
+ .shared
+ .calls
+ .clone()
+ .try_acquire_owned()
+ .map_err(|_| {
+ anyhow::Error::new(OnnxAdmissionError {
+ reason: format!(
+ "ONNX embedding admission full: {} calls are already admitted \
+ (loading, running, or awaiting the serialized session); retry with backoff",
+ self.cfg.max_inflight_calls
+ ),
+ })
+ })?;
+ let bytes = self
+ .shared
+ .bytes
+ .clone()
+ .try_acquire_many_owned(byte_permits)
+ .map_err(|_| {
+ anyhow::Error::new(OnnxAdmissionError {
+ reason: format!(
+ "ONNX embedding byte budget full: request needs {input_bytes} bytes, \
+ global in-flight limit is {} bytes; retry with backoff",
+ self.cfg.max_inflight_input_bytes
+ ),
+ })
+ })?;
+ Ok((call, bytes))
+ }
+}
+
+#[cfg(feature = "onnx-experimental")]
+fn sha256_bytes(bytes: &[u8]) -> String {
+ use sha2::{Digest, Sha256};
+ let mut hasher = Sha256::new();
+ hasher.update(bytes);
+ format!("{:x}", hasher.finalize())
+}
+
+#[cfg(all(test, feature = "onnx-experimental"))]
+mod onnx_handle_tests {
+ use super::{OnnxConfig, OnnxHandle};
+ use crate::onnx::MicrobatchConfig;
+ use std::path::PathBuf;
+ use std::sync::Arc;
+
+ fn cfg(model_sha256: &str) -> OnnxConfig {
+ OnnxConfig {
+ model_path: PathBuf::from("/tmp/model.onnx"),
+ tokenizer_path: PathBuf::from("/tmp/tokenizer.json"),
+ model_sha256: model_sha256.into(),
+ tokenizer_sha256: "tokenizer-hash".into(),
+ intra_threads: 4,
+ microbatch: MicrobatchConfig::default(),
+ max_inflight_calls: 2,
+ max_input_bytes_per_call: 10,
+ max_inflight_input_bytes: 12,
}
}
+
+ #[test]
+ fn same_paths_with_different_content_hashes_never_share_session_cell() {
+ let first = OnnxHandle::new(cfg("model-a"));
+ let second = OnnxHandle::new(cfg("model-b"));
+ assert!(!Arc::ptr_eq(&first.shared, &second.shared));
+ }
+
+ #[test]
+ fn global_admission_caps_calls_and_releases_permits() {
+ let handle = OnnxHandle::new(cfg("admission-calls"));
+ let first = handle.try_admit(1).unwrap();
+ let second = handle.try_admit(1).unwrap();
+ let error = handle.try_admit(1).unwrap_err().to_string();
+ assert!(error.contains("admission full"), "{error}");
+ drop(first);
+ assert!(handle.try_admit(1).is_ok());
+ drop(second);
+ }
+
+ #[test]
+ fn concurrent_handles_share_global_call_cap() {
+ let config = cfg("admission-concurrent");
+ let first = OnnxHandle::new(config.clone());
+ let second = OnnxHandle::new(config.clone());
+ let rejected = OnnxHandle::new(config);
+ let (ready_tx, ready_rx) = std::sync::mpsc::channel();
+ let release = Arc::new(std::sync::Barrier::new(3));
+ std::thread::scope(|scope| {
+ for handle in [first, second] {
+ let ready_tx = ready_tx.clone();
+ let release = Arc::clone(&release);
+ scope.spawn(move || {
+ let _permit = handle.try_admit(1).unwrap();
+ ready_tx.send(()).unwrap();
+ release.wait();
+ });
+ }
+ ready_rx.recv().unwrap();
+ ready_rx.recv().unwrap();
+ let error = rejected.try_admit(1).unwrap_err().to_string();
+ assert!(error.contains("admission full"), "{error}");
+ release.wait();
+ });
+ assert!(rejected.try_admit(1).is_ok(), "permits must release");
+ }
+
+ #[test]
+ fn global_byte_budget_and_per_call_limit_are_enforced_before_work() {
+ let handle = OnnxHandle::new(cfg("admission-bytes"));
+ let held = handle.try_admit(8).unwrap();
+ let error = handle.try_admit(5).unwrap_err().to_string();
+ assert!(error.contains("byte budget full"), "{error}");
+ drop(held);
+ assert!(handle.try_admit(5).is_ok());
+
+ let error = handle.try_admit(11).unwrap_err().to_string();
+ assert!(error.contains("per-call limit"), "{error}");
+ }
+
+ #[tokio::test]
+ async fn document_cap_rejects_before_model_load_or_tokenization() {
+ let mut config = cfg("admission-docs");
+ config.microbatch.max_pending = 1;
+ let handle = OnnxHandle::new(config);
+ let error = handle
+ .embed_batch(vec!["one".into(), "two".into()])
+ .await
+ .unwrap_err()
+ .to_string();
+ assert!(error.contains("before tokenization"), "{error}");
+ assert!(error.contains("max_pending=1"), "{error}");
+ assert!(
+ handle.shared.cell.get().is_none(),
+ "model must remain unloaded"
+ );
+ }
}
/// Lazily-loaded neural backend. The heavy model is loaded once, on the first
diff --git a/engine/crates/xerj-ai/src/lib.rs b/engine/crates/xerj-ai/src/lib.rs
index 49150bd57..9e3e3c134 100644
--- a/engine/crates/xerj-ai/src/lib.rs
+++ b/engine/crates/xerj-ai/src/lib.rs
@@ -3,12 +3,13 @@
//! AI-native features for the xerj search engine.
//!
//! Provides:
-//! - [`embedder`] — Unified backend handle (lexical / proxy / neural) used by the engine
+//! - [`embedder`] — Unified backend handle (lexical / proxy / Candle neural /
+//! experimental ONNX) used by the engine
//! - [`embed`] — Embedding proxy: async HTTP client for OpenAI-compatible embedding APIs
//! - [`local`] — Built-in zero-config deterministic text embedder (feature hashing)
//! - [`neural`] — Built-in neural BERT sentence embedder via candle (feature `neural`)
-//! - `onnx` — Experimental FP32 ONNX backend (feature `onnx-experimental`);
-//! not wired into the server
+//! - `onnx` — Experimental MiniLM-compatible FP32 ONNX backend
+//! (feature `onnx-experimental`; server feature + explicit runtime selection required)
//! - [`chunker`] — Text chunking with sentence-aware splitting and overlap
//! - [`memory`] — Agent memory: semantic dedup + recency-blended recall
diff --git a/engine/crates/xerj-ai/src/onnx.rs b/engine/crates/xerj-ai/src/onnx.rs
index c721f3dc3..65af3fbeb 100644
--- a/engine/crates/xerj-ai/src/onnx.rs
+++ b/engine/crates/xerj-ai/src/onnx.rs
@@ -1,7 +1,15 @@
//! Experimental FP32 ONNX Runtime MiniLM backend.
//!
-//! This module is an embedding-layer prototype. It is not wired to XERJ's
-//! server or CLI. The safe `ort` API requires mutable session access, so one
+//! This backend is available end-to-end only in builds with the
+//! `onnx-experimental` feature and must be explicitly selected with local
+//! model/tokenizer paths. It is deliberately not a generic ONNX interface: the
+//! graph must be an FP32 all-MiniLM-L6-v2-compatible encoder with int64 inputs
+//! named `input_ids`, `attention_mask`, and `token_type_ids`, and a rank-3,
+//! width-384 output named `last_hidden_state` or `token_embeddings`. The
+//! tokenizer.json must come from the same model/export. XERJ attention-mask
+//! mean-pools and L2-normalizes the token output.
+//!
+//! The safe `ort` API requires mutable session access, so one
//! session is serialized behind a mutex and aggregate throughput comes from
//! bounded, length-aware microbatches rather than unsafe concurrent `Run`.
@@ -16,7 +24,7 @@ pub const MAX_TOKENS: usize = 512;
pub const DIMS: usize = 384;
/// Bounds for one offline scheduling window.
-#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq)]
pub struct MicrobatchConfig {
/// Maximum inputs accepted in one call. Larger calls receive a clear
/// backpressure error instead of allocating an unbounded queue.
@@ -58,8 +66,20 @@ pub struct OnnxEmbedder {
impl OnnxEmbedder {
pub fn load(model_path: &Path, tokenizer_path: &Path, intra_threads: usize) -> Result<Self> {
- let mut tokenizer = Tokenizer::from_file(tokenizer_path)
- .map_err(|e| anyhow!("load tokenizer {}: {e}", tokenizer_path.display()))?;
+ let model = std::fs::read(model_path)
+ .with_context(|| format!("read ONNX model {}", model_path.display()))?;
+ let tokenizer = std::fs::read(tokenizer_path)
+ .with_context(|| format!("read tokenizer {}", tokenizer_path.display()))?;
+ Self::load_bytes(&model, &tokenizer, intra_threads)
+ }
+
+ pub(crate) fn load_bytes(
+ model_bytes: &[u8],
+ tokenizer_bytes: &[u8],
+ intra_threads: usize,
+ ) -> Result<Self> {
+ let mut tokenizer = Tokenizer::from_bytes(tokenizer_bytes)
+ .map_err(|e| anyhow!("load tokenizer.json bytes: {e}"))?;
tokenizer.with_padding(Some(PaddingParams {
strategy: PaddingStrategy::BatchLongest,
..Default::default()
@@ -71,16 +91,29 @@ impl OnnxEmbedder {
}))
.map_err(|e| anyhow!("configure tokenizer truncation: {e}"))?;
+ let ort_log_level = match std::env::var("XERJ_ONNX_LOG")
+ .unwrap_or_default()
+ .to_ascii_lowercase()
+ .as_str()
+ {
+ "verbose" | "trace" => ort::logging::LogLevel::Verbose,
+ "info" | "debug" => ort::logging::LogLevel::Info,
+ "error" => ort::logging::LogLevel::Error,
+ "fatal" => ort::logging::LogLevel::Fatal,
+ _ => ort::logging::LogLevel::Warning,
+ };
let session = Session::builder()
.map_err(|e| anyhow!("create ONNX Runtime session builder: {e}"))?
+ .with_log_level(ort_log_level)
+ .map_err(|e| anyhow!("configure ONNX Runtime log level: {e}"))?
.with_optimization_level(GraphOptimizationLevel::Level3)
.map_err(|e| anyhow!("configure ONNX graph optimizations: {e}"))?
.with_config_entry("session.intra_op.allow_spinning", "0")
.map_err(|e| anyhow!("disable ONNX intra-op spinning: {e}"))?
.with_intra_threads(intra_threads.max(1))
.map_err(|e| anyhow!("configure ONNX intra-op threads: {e}"))?
- .commit_from_file(model_path)
- .with_context(|| format!("load ONNX model {}", model_path.display()))?;
+ .commit_from_memory(model_bytes)
+ .context("load ONNX model bytes")?;
Ok(Self {
session: Mutex::new(session),
tokenizer,
diff --git a/engine/crates/xerj-autoindex/src/catalog.rs b/engine/crates/xerj-autoindex/src/catalog.rs
index aade95dfa..e64c046d2 100644
--- a/engine/crates/xerj-autoindex/src/catalog.rs
+++ b/engine/crates/xerj-autoindex/src/catalog.rs
@@ -53,7 +53,7 @@ pub fn catalog_mapping() -> Value {
pub const GOTCHAS: &[&str] = &[
"hybrid search: use {\"query\":{\"hybrid\":{\"queries\":[…]}}} ONLY — retriever.rrf is a silent stub and rank.rrf is ignored on this engine",
- "semantic_text fields are embedded server-side: the DEFAULT is the built-in LEXICAL feature-hash embedder (384-dim hybrid lexical+vector, NOT neural) — start the server with `--embed-mode neural` (built-in BERT) or `--embed-mode proxy` for true neural semantics",
+ "semantic_text fields are embedded server-side: the DEFAULT is the built-in LEXICAL feature-hash embedder (384-dim hybrid lexical+vector, NOT neural) — start the server with `--embed-mode neural` (built-in Candle BERT), `--embed-mode proxy`, or an ONNX-enabled build with `--embed-mode onnx-experimental --onnx-model … --onnx-tokenizer …` for neural semantics; ONNX runs only when this map shows a semantic_field, and its first real inference is confirmed by the server activation log",
"semantic queries ignore _source filtering and return the ~8KB *_vector field in _source — strip client-side",
"exact filters use TOP-LEVEL keyword fields (term on .keyword subfields returns 0 hits on this engine)",
"all dates are normalized to RFC3339 UTC millis; mappings use strict_date_optional_time||epoch_millis",
@@ -180,7 +180,7 @@ pub fn build_sample_queries(pd: &PlanDataset, correlations: &[KeyCorr]) -> Vec<V
if let Some(sf) = &pd.semantic_field {
out.push(json!({
"class": "hybrid_lexical_vector",
- "title": format!("Hybrid lexical+vector (RRF) on {sf} — embedder set server-side (lexical by default; neural/proxy if configured)"),
+ "title": format!("Hybrid lexical+vector (RRF) on {sf} — embedder set server-side (lexical by default; Candle neural, proxy, or experimental ONNX if configured)"),
"request": format!("POST /{}/_search", pd.index),
"body": {"query": {"hybrid": {"queries": [
{"query": {"match": {(sf.clone()): word.clone()}}, "weight": 1},
@@ -296,7 +296,7 @@ pub fn render_map(
s.push_str(&format!("### `{}`\n\n", g("index_name")));
if let Some(sem) = d.get("semantic_field").filter(|v| v.is_string()) {
s.push_str(&format!(
- "semantic body field: `{}` (hybrid lexical+vector; embedder set server-side — lexical by default, neural/proxy if configured)\n\n",
+ "semantic body field: `{}` (hybrid lexical+vector; embedder set server-side — lexical by default, Candle neural, proxy, or experimental ONNX if configured)\n\n",
pretty_val(sem)
));
}
diff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rs
index 87263d5be..c046d6fe6 100644
--- a/engine/crates/xerj-autoindex/src/cli.rs
+++ b/engine/crates/xerj-autoindex/src/cli.rs
@@ -73,6 +73,15 @@ pub fn print_help() {
--dataset <SLUG> (map) show a single dataset\n\
--help, -h this help\n\
\n\
+ EMBEDDINGS:\n\
+ autoindex sends semantic_text to the running server; it does not choose the\n\
+ server's embedding backend. The default is lexical (not neural). For the\n\
+ experimental ONNX backend, start xerj with `--embed-mode onnx-experimental\n\
+ --onnx-model MODEL.onnx --onnx-tokenizer tokenizer.json`, then run autoindex.\n\
+ ONNX runs only for fields inferred as semantic_text (normally long body text;\n\
+ short/structured datasets may infer none). Use --dry-run or `autoindex map` to\n\
+ confirm a semantic field before attributing an indexing result to embeddings.\n\
+ \n\
EXIT CODES: 0 complete; 3 completed-with-junk (junk recorded, never fatal);\n\
2 usage; 1 endpoint unreachable / journal-config mismatch\n"
);
diff --git a/engine/crates/xerj-autoindex/src/esclient.rs b/engine/crates/xerj-autoindex/src/esclient.rs
index 1eb2ff1a6..3b4273345 100644
--- a/engine/crates/xerj-autoindex/src/esclient.rs
+++ b/engine/crates/xerj-autoindex/src/esclient.rs
@@ -14,7 +14,11 @@ pub struct Es {
pub struct BulkOutcome {
pub item_errors: u64,
+ /// Per-item 5xx/429 failures are backend/admission failures, not bad source
+ /// records. Callers must not journal the source file complete.
+ pub server_errors: u64,
pub first_error: Option<String>,
+ pub first_server_error: Option<String>,
}
impl Es {
@@ -112,7 +116,9 @@ impl Es {
}
let v: Value = resp.json().context("parse bulk response")?;
let mut item_errors = 0u64;
+ let mut server_errors = 0u64;
let mut first_error = None;
+ let mut first_server_error = None;
if v.get("errors").and_then(|e| e.as_bool()).unwrap_or(false) {
if let Some(items) = v.get("items").and_then(|i| i.as_array()) {
for it in items {
@@ -123,6 +129,16 @@ impl Es {
if let Some(op) = op {
if op.get("error").is_some() {
item_errors += 1;
+ let item_status =
+ op.get("status").and_then(Value::as_u64).unwrap_or(500);
+ if item_status == 429 || item_status >= 500 {
+ server_errors += 1;
+ if first_server_error.is_none() {
+ first_server_error = Some(
+ op["error"].to_string().chars().take(500).collect(),
+ );
+ }
+ }
if first_error.is_none() {
first_error = Some(
op["error"].to_string().chars().take(300).collect(),
@@ -135,7 +151,9 @@ impl Es {
}
Ok(BulkOutcome {
item_errors,
+ server_errors,
first_error,
+ first_server_error,
})
},
)
diff --git a/engine/crates/xerj-autoindex/src/infer/mod.rs b/engine/crates/xerj-autoindex/src/infer/mod.rs
index e187dcf86..4f22ca52d 100644
--- a/engine/crates/xerj-autoindex/src/infer/mod.rs
+++ b/engine/crates/xerj-autoindex/src/infer/mod.rs
@@ -515,7 +515,7 @@ pub fn infer_fields(
let best = specs
.iter()
.enumerate()
- .filter(|(i, s)| {
+ .filter(|(_i, s)| {
s.es_type == "text"
&& s.avg_len >= 80.0
&& fields
@@ -530,7 +530,7 @@ pub fn infer_fields(
specs[i].notes.push(format!(
"hybrid lexical+vector body — elected because it looks like natural language \
(word_ratio {:.2}, {:.1} tokens/value); embedded server-side (lexical by \
- default, neural/proxy if configured)",
+ default; Candle neural, proxy, or experimental ONNX if configured)",
a.map(|x| x.word_ratio()).unwrap_or(0.0),
a.map(|x| x.mean_tokens()).unwrap_or(0.0),
));
diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rs
index fb86a96b6..643c6f6d7 100644
--- a/engine/crates/xerj-autoindex/src/lib.rs
+++ b/engine/crates/xerj-autoindex/src/lib.rs
@@ -491,6 +491,19 @@ fn run_index(cfg: IndexCfg) -> Result<i32> {
if buf.len() >= bulk_cut || buf_docs >= 5000 {
match es.bulk(std::mem::take(&mut buf)) {
Ok(o) => {
+ if o.server_errors > 0 {
+ send_err = Some(format!(
+ "bulk backend failed for {} item(s): {}. \
+ Source file was not journaled complete; fix \
+ the server/embedding configuration and rerun \
+ autoindex to resume",
+ o.server_errors,
+ o.first_server_error
+ .as_deref()
+ .unwrap_or("unknown server error")
+ ));
+ return false;
+ }
if o.item_errors > 0 {
junk_records
.fetch_add(o.item_errors, Ordering::Relaxed);
@@ -533,8 +546,22 @@ fn run_index(cfg: IndexCfg) -> Result<i32> {
if !buf.is_empty() && send_err.is_none() {
match es.bulk(std::mem::take(&mut buf)) {
Ok(o) => {
+ if o.server_errors > 0 {
+ send_err = Some(format!(
+ "bulk backend failed for {} item(s): {}. Source file \
+ was not journaled complete; fix the server/embedding \
+ configuration and rerun autoindex to resume",
+ o.server_errors,
+ o.first_server_error
+ .as_deref()
+ .unwrap_or("unknown server error")
+ ));
+ }
if o.item_errors > 0 {
- junk_records.fetch_add(o.item_errors, Ordering::Relaxed);
+ junk_records.fetch_add(
+ o.item_errors.saturating_sub(o.server_errors),
+ Ordering::Relaxed,
+ );
}
}
Err(e) => send_err = Some(format!("{e:#}")),
@@ -582,7 +609,12 @@ fn run_index(cfg: IndexCfg) -> Result<i32> {
let bulk_errs = bulk_errors.into_inner().unwrap();
if !bulk_errs.is_empty() {
- eprintln!("bulk errors encountered (first few): {bulk_errs:?}");
+ anyhow::bail!(
+ "autoindex stopped with bulk/backend failures: {}. Failed source files were not \
+ journaled complete; fix the reported server or embedding configuration and rerun \
+ the same command to resume safely",
+ bulk_errs.join(" | ")
+ );
}
// ── finalize: refresh, verify, correlate, catalog ────────────────────
diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rs
index 833bc8e00..0c058e55c 100644
--- a/engine/crates/xerj-common/src/config.rs
+++ b/engine/crates/xerj-common/src/config.rs
@@ -786,6 +786,26 @@ pub struct EmbeddingConfig {
/// safetensors weights from this local directory instead of downloading
/// — for air-gapped / offline deployments. Empty (default) = download.
pub local_model_dir: String,
+ /// Experimental ONNX backend: local FP32 all-MiniLM-L6-v2-compatible
+ /// model with int64 BERT inputs and a width-384 token-embedding output.
+ /// Required when `mode = "onnx-experimental"`; never auto-downloaded.
+ pub onnx_model_path: String,
+ /// Experimental ONNX backend: tokenizer.json from the same model/export.
+ pub onnx_tokenizer_path: String,
+ /// Experimental ONNX Runtime intra-op threads (default: available CPUs).
+ pub onnx_intra_threads: usize,
+ /// Maximum texts accepted by one ONNX scheduling window.
+ pub onnx_max_pending: usize,
+ /// Maximum documents in one ONNX inference microbatch.
+ pub onnx_max_batch: usize,
+ /// Maximum batch × padded-token slots per ONNX inference microbatch.
+ pub onnx_padded_token_budget: usize,
+ /// Maximum ONNX calls admitted globally per shared model/session.
+ pub onnx_max_inflight_calls: usize,
+ /// Reject one ONNX call above this many UTF-8 input bytes before tokenizing.
+ pub onnx_max_input_bytes_per_call: usize,
+ /// Aggregate UTF-8 bytes admitted globally per shared ONNX model/session.
+ pub onnx_max_inflight_input_bytes: usize,
}
impl Default for EmbeddingConfig {
@@ -799,6 +819,17 @@ impl Default for EmbeddingConfig {
neural_model: "sentence-transformers/all-MiniLM-L6-v2".to_string(),
model_cache_dir: String::new(),
local_model_dir: String::new(),
+ onnx_model_path: String::new(),
+ onnx_tokenizer_path: String::new(),
+ onnx_intra_threads: std::thread::available_parallelism()
+ .map(|n| n.get())
+ .unwrap_or(1),
+ onnx_max_pending: 4096,
+ onnx_max_batch: 64,
+ onnx_padded_token_budget: 4096,
+ onnx_max_inflight_calls: 8,
+ onnx_max_input_bytes_per_call: 8 * 1024 * 1024,
+ onnx_max_inflight_input_bytes: 32 * 1024 * 1024,
}
}
}
diff --git a/engine/crates/xerj-engine/Cargo.toml b/engine/crates/xerj-engine/Cargo.toml
index f39e8589c..afc704bab 100644
--- a/engine/crates/xerj-engine/Cargo.toml
+++ b/engine/crates/xerj-engine/Cargo.toml
@@ -51,6 +51,7 @@ default = []
# default so the standard build stays lean and cross-compiles without the
# candle/tokenizer toolchain; enable via xerj-server's `neural` feature.
neural = ["xerj-ai/neural"]
+onnx-experimental = ["xerj-ai/onnx-experimental"]
[dev-dependencies]
tempfile = { workspace = true }
diff --git a/engine/crates/xerj-engine/src/bulk.rs b/engine/crates/xerj-engine/src/bulk.rs
index 644fcc3a2..5b1e9d229 100644
--- a/engine/crates/xerj-engine/src/bulk.rs
+++ b/engine/crates/xerj-engine/src/bulk.rs
@@ -40,6 +40,18 @@ pub struct BulkItemResult {
pub get_source: Option<Value>,
}
+fn engine_error_http_status(error: &EngineError) -> u16 {
+ match error {
+ EngineError::Common(xerj_common::XerjError::IndexBlocked { block_type, .. })
+ if block_type.contains("read_only_allow_delete") =>
+ {
+ 429
+ }
+ EngineError::Common(error) => error.http_status(),
+ _ => 500,
+ }
+}
+
// ── Bulk processor ────────────────────────────────────────────────────────────
/// A parsed bulk action before execution.
@@ -820,6 +832,10 @@ pub async fn process_bulk_with_opts(
// created-vs-updated / 201-vs-200 semantics turbo can't express.
let mut auto_id_batches: HashMap<String, Vec<(usize, String, std::sync::Arc<[u8]>)>> =
HashMap::new();
+ let mut semantic_index_batches: HashMap<
+ String,
+ Vec<(usize, Option<String>, std::sync::Arc<[u8]>)>,
+ > = HashMap::new();
let mut non_index_actions: Vec<ParsedAction> = Vec::new();
// Precompute which target indices declare any strict-format date
@@ -919,15 +935,20 @@ pub async fn process_bulk_with_opts(
// them through the non-index path. Also divert when the index
// has date fields with a strict `format`, so the per-item
// validation loop can reject malformed values like ES does.
- let is_plain_index = action.action_type == "index"
+ let has_strict_date =
+ index_has_strict_date(&action.target_index, &mut index_needs_date_validation);
+ let has_dynamic_copy =
+ index_has_dynamic_copy(&action.target_index, &mut index_needs_dynamic_copy);
+ let has_embedding = index_has_embedding(&action.target_index, &mut index_needs_embedding);
+ let has_plain_metadata = action.action_type == "index"
&& action.if_seq_no.is_none()
&& action.if_primary_term.is_none()
&& action.routing.is_none()
&& action.dynamic_templates.is_none()
&& action.pipeline.is_none()
- && !index_has_strict_date(&action.target_index, &mut index_needs_date_validation)
- && !index_has_dynamic_copy(&action.target_index, &mut index_needs_dynamic_copy)
- && !index_has_embedding(&action.target_index, &mut index_needs_embedding);
+ && !has_strict_date
+ && !has_dynamic_copy;
+ let is_plain_index = has_plain_metadata && !has_embedding;
if is_plain_index {
match action.doc_id {
None => {
@@ -949,6 +970,11 @@ pub async fn process_bulk_with_opts(
));
}
}
+ } else if has_plain_metadata && has_embedding {
+ semantic_index_batches
+ .entry(action.target_index)
+ .or_default()
+ .push((action.item_index, action.doc_id, action.doc_bytes));
} else {
non_index_actions.push(action);
}
@@ -958,6 +984,106 @@ pub async fn process_bulk_with_opts(
let group_ms = t_group.elapsed().as_millis() as u64;
let t_exec = std::time::Instant::now();
+ // ── semantic_text index actions: cross-document embedding windows ────
+ //
+ // Parse and prepare a target-index group together so the embedder sees
+ // useful multi-document windows. The normal per-document index method is
+ // still used after preparation; it observes the supplied companion vector
+ // and therefore preserves WAL, overwrite status, response ordering, and
+ // item-level errors.
+ for (index_name, batch) in semantic_index_batches {
+ let idx = match engine.get_or_create_index(&index_name) {
+ Ok(i) => i,
+ Err(e) => {
+ for (item_idx, id, _) in batch {
+ items[item_idx] = Some(BulkItemResult {
+ action: "index".into(),
+ index: index_name.clone(),
+ id: id.unwrap_or_default(),
+ status: 500,
+ result: None,
+ error: Some(e.to_string()),
+ get_source: None,
+ });
+ errors = true;
+ }
+ continue;
+ }
+ };
+
+ let mut valid_meta = Vec::with_capacity(batch.len());
+ let mut valid_sources = Vec::with_capacity(batch.len());
+ for (item_idx, id, bytes) in batch {
+ match serde_json::from_slice::<Value>(&bytes) {
+ Ok(source) => {
+ valid_meta.push((item_idx, id));
+ valid_sources.push(source);
+ }
+ Err(e) => {
+ items[item_idx] = Some(BulkItemResult {
+ action: "index".into(),
+ index: index_name.clone(),
+ id: id.unwrap_or_default(),
+ status: 400,
+ result: None,
+ error: Some(format!("failed to parse: invalid document JSON: {e}")),
+ get_source: None,
+ });
+ errors = true;
+ }
+ }
+ }
+
+ let prepared = idx.apply_semantic_embeddings_batch(valid_sources).await;
+ for ((item_idx, id), prepared_source) in valid_meta.into_iter().zip(prepared) {
+ let source = match prepared_source {
+ Ok(source) => source,
+ Err(e) => {
+ let status = engine_error_http_status(&e);
+ items[item_idx] = Some(BulkItemResult {
+ action: "index".into(),
+ index: index_name.clone(),
+ id: id.unwrap_or_default(),
+ status,
+ result: None,
+ error: Some(e.to_string()),
+ get_source: None,
+ });
+ errors = true;
+ continue;
+ }
+ };
+ let error_id = id.clone().unwrap_or_default();
+ match idx.index_document_prepared(id, source).await {
+ Ok(resp) => {
+ let status = if resp.result == "updated" { 200 } else { 201 };
+ items[item_idx] = Some(BulkItemResult {
+ action: "index".into(),
+ index: index_name.clone(),
+ id: resp.id,
+ status,
+ result: Some(resp.result),
+ error: None,
+ get_source: None,
+ });
+ }
+ Err(e) => {
+ let status = engine_error_http_status(&e);
+ items[item_idx] = Some(BulkItemResult {
+ action: "index".into(),
+ index: index_name.clone(),
+ id: error_id,
+ status,
+ result: None,
+ error: Some(e.to_string()),
+ get_source: None,
+ });
+ errors = true;
+ }
+ }
+ }
+ }
+
// ── Auto-id index actions: ONE turbo batch per target index ──────────
//
// `index_batch_turbo_raw` appends the whole group with a single WAL
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index 12e6ccb0c..ddb5ada11 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -556,16 +556,20 @@ pub struct Index {
/// `Config.merge` at index construction; reads are cheap and merge
/// runs hold the snapshot for the duration of one batch.
merge_config: xerj_common::config::MergeConfig,
+ /// Retained so a semantic field added after index creation can pin the
+ /// same embedding identity before its first document is accepted.
+ embedding_config: xerj_common::config::EmbeddingConfig,
/// Embedding backend for `semantic` / `semantic_text` (v0.7-P2).
- /// One of lexical (built-in, default), an external proxy, or the
- /// built-in neural BERT embedder — selected by `Config.embedding.mode`.
+ /// One of lexical (built-in, default), an external proxy, the built-in
+ /// Candle neural BERT embedder, or the experimental ONNX Runtime backend
+ /// — selected by `Config.embedding.mode`.
/// [`xerj_ai::Embedder::is_active`] is false only for the lexical
/// fallback; an active backend embeds arbitrary query text, while the
/// lexical fallback embeds only `semantic_text` fields (embedded the
- /// same way at ingest). Reused across queries (proxy semaphore / neural
- /// model are shared behind the `Arc`).
- embedder: Arc<xerj_ai::Embedder>,
+ /// same way at ingest). Reused across queries (proxy admission state and
+ /// neural/ONNX models are shared behind the `Arc`).
+ embedder: Arc<RwLock<xerj_ai::Embedder>>,
// ── Per-index metrics ─────────────────────────────────────────────────────
/// Total search queries executed.
@@ -844,6 +848,8 @@ impl Index {
schema,
dynamic: xerj_common::schema::DynamicMapping::Dynamic,
};
+ validate_embedding_identity(&index_dir, &managed.schema, &config.embedding, true, false)?;
+ let effective_embedder = make_embedder_for_schema(&managed.schema, &config.embedding)?;
// Doc count is a sanity cap only — the byte threshold is the primary
// driver. Historically this was 10 000, which forced flushes every ~8 MB
@@ -940,7 +946,8 @@ impl Index {
)),
merge_in_progress: Arc::new(std::sync::atomic::AtomicBool::new(false)),
merge_config: config.merge.clone(),
- embedder: Arc::new(make_embedder(&config.embedding)),
+ embedding_config: config.embedding.clone(),
+ embedder: Arc::new(RwLock::new(effective_embedder)),
dv_cache: Arc::new(dashmap::DashMap::new()),
sort_shadow_cache: Arc::new(dashmap::DashMap::new()),
sort_shadow_fields: Arc::new(dashmap::DashMap::new()),
@@ -993,6 +1000,14 @@ impl Index {
// Load schema from disk if it exists.
let schema = load_schema(&index_dir).unwrap_or_else(|_| ManagedSchema::dynamic());
+ validate_embedding_identity(
+ &index_dir,
+ &schema.schema,
+ &config.embedding,
+ false,
+ segment_doc_count > 0 || store.version_map.live_count() > 0,
+ )?;
+ let effective_embedder = make_embedder_for_schema(&schema.schema, &config.embedding)?;
// Load persisted settings early so we can build the registry before WAL replay.
let settings = load_settings(&index_dir).unwrap_or(Value::Null);
@@ -1178,7 +1193,8 @@ impl Index {
)),
merge_in_progress: Arc::new(std::sync::atomic::AtomicBool::new(false)),
merge_config: config.merge.clone(),
- embedder: Arc::new(make_embedder(&config.embedding)),
+ embedding_config: config.embedding.clone(),
+ embedder: Arc::new(RwLock::new(effective_embedder)),
dv_cache: Arc::new(dashmap::DashMap::new()),
sort_shadow_cache: Arc::new(dashmap::DashMap::new()),
sort_shadow_fields: Arc::new(dashmap::DashMap::new()),
@@ -1266,6 +1282,27 @@ impl Index {
source: Value,
if_seq_no: Option<u64>,
if_primary_term: Option<u64>,
+ ) -> Result<IndexResponse> {
+ self.index_document_with_version_inner(id, source, if_seq_no, if_primary_term, false)
+ .await
+ }
+
+ pub(crate) async fn index_document_prepared(
+ &self,
+ id: Option<String>,
+ source: Value,
+ ) -> Result<IndexResponse> {
+ self.index_document_with_version_inner(id, source, None, None, true)
+ .await
+ }
+
+ async fn index_document_with_version_inner(
+ &self,
+ id: Option<String>,
+ source: Value,
+ if_seq_no: Option<u64>,
+ if_primary_term: Option<u64>,
+ semantic_embeddings_prepared: bool,
) -> Result<IndexResponse> {
// Check write block.
if self.is_write_blocked().await {
@@ -1321,7 +1358,11 @@ impl Index {
// its companion vector field (`<field>_vector`) so it becomes
// kNN-searchable. Runs before copy_to / storage so the derived vector
// is part of `_source` and gets picked up by HNSW indexing below.
- let source = self.apply_semantic_embeddings(source).await?;
+ let source = if semantic_embeddings_prepared {
+ source
+ } else {
+ self.apply_semantic_embeddings(source).await?
+ };
// Apply copy_to: expand the source by copying field values to their target fields.
let source = {
@@ -4939,18 +4980,19 @@ impl Index {
};
// Embed the query text with the effective embedder:
- // * An ACTIVE backend (neural BERT or external proxy) embeds the
- // query text directly — the same backend used at ingest.
+ // * An ACTIVE backend (Candle neural BERT, experimental ONNX, or an
+ // external proxy) embeds the query text directly — the same
+ // backend used at ingest.
// * Otherwise the built-in lexical embedder — but ONLY for
// `semantic_text` fields, which were embedded that same way at
// ingest. A `semantic` query against a plain dense_vector field
// with no active backend still returns the original 400 (there
// is no comparable stored vector to match against).
- let query_vec = if self.embedder.is_active() {
+ let embedder = self.embedder.read().await;
+ let query_vec = if embedder.is_active() {
// `embed_batch` takes Vec<String>; we only have one text but
// batching keeps the wire format stable for callers.
- let mut vectors = self
- .embedder
+ let mut vectors = embedder
.embed_batch(vec![text.to_string()])
.await
.map_err(|e| {
@@ -4974,8 +5016,10 @@ impl Index {
return Err(EngineError::Common(xerj_common::XerjError::invalid_query(
"semantic query requires either a `semantic_text` field (auto-embedded \
with the built-in lexical embedder) or an active embedding backend \
- (`embedding.mode = \"neural\"`, or `\"proxy\"` with \
- `embedding.default_endpoint` set to an OpenAI-compatible endpoint).",
+ (`embedding.mode = \"neural\"`; `\"proxy\"` with \
+ `embedding.default_endpoint` set to an OpenAI-compatible endpoint; \
+ or `\"onnx-experimental\"` in an ONNX-enabled build with explicit \
+ model and tokenizer paths).",
)));
};
@@ -5001,7 +5045,22 @@ impl Index {
/// deterministic embedder — the *same* choice made at query time so the
/// vectors are comparable. Fields where the target already holds a value
/// (caller pre-embedded) or whose value is not a string are left untouched.
- async fn apply_semantic_embeddings(&self, mut source: Value) -> Result<Value> {
+ async fn apply_semantic_embeddings(&self, source: Value) -> Result<Value> {
+ self.apply_semantic_embeddings_batch(vec![source])
+ .await
+ .pop()
+ .expect("one input produces one output")
+ }
+
+ /// Prepare embeddings for several documents in shared inference windows.
+ ///
+ /// The returned vector is position-aligned with `sources`. A failed
+ /// backend window is retried per document-field job so one bad item does
+ /// not turn otherwise valid bulk items into failures.
+ pub(crate) async fn apply_semantic_embeddings_batch(
+ &self,
+ mut sources: Vec<Value>,
+ ) -> Vec<Result<Value>> {
// Collect (field, target_field, dims) specs without holding the schema
// lock across the (possibly async) embedding calls.
let specs: Vec<(String, String, usize)> = {
@@ -5026,44 +5085,183 @@ impl Index {
.collect()
};
if specs.is_empty() {
- return Ok(source);
- }
- let Some(obj) = source.as_object_mut() else {
- return Ok(source);
- };
- for (field, target, dims) in specs {
- // Respect a caller-supplied vector (e.g. pre-computed offline).
- if obj.get(&target).map(|v| !v.is_null()).unwrap_or(false) {
- continue;
- }
- let Some(text) = obj.get(&field).and_then(Value::as_str) else {
+ return sources.into_iter().map(Ok).collect();
+ }
+
+ struct EmbedJob {
+ source_idx: usize,
+ field: String,
+ target: String,
+ dims: usize,
+ original_text: String,
+ texts: Vec<String>,
+ }
+
+ let mut jobs = Vec::<EmbedJob>::new();
+ let mut pre_failures: Vec<Option<String>> = (0..sources.len()).map(|_| None).collect();
+ let onnx_pinned = self
+ .embedding_config
+ .mode
+ .eq_ignore_ascii_case("onnx-experimental");
+ for (source_idx, source) in sources.iter().enumerate() {
+ let Some(obj) = source.as_object() else {
continue;
};
- let text = text.to_string();
- // Chunk once and embed EACH overlapping chunk. `chunk_vecs` is one
- // embedding per passage (>= 1). Short text (<= chunk_size) yields a
- // single chunk == the exact pre-chunking behavior.
- let chunk_vecs: Vec<Vec<f32>> = if self.embedder.is_active() {
- // Active backend (neural or proxy): chunk, then embed each
- // passage. The same backend + chunking is used at query time.
- let chunks = semantic_chunker().chunk(&text, None);
- let chunk_texts: Vec<String> = if chunks.len() <= 1 {
- vec![text.clone()]
+ for (field, target, dims) in &specs {
+ let supplied_vector = obj.get(target).map(|v| !v.is_null()).unwrap_or(false);
+ let supplied_chunks = obj
+ .get(&format!("{target}_chunks"))
+ .map(|v| !v.is_null())
+ .unwrap_or(false);
+ if onnx_pinned && (supplied_vector || supplied_chunks) {
+ pre_failures[source_idx] = Some(format!(
+ "field [{field}]: caller-supplied derived vectors are not accepted for \
+ an ONNX-pinned semantic index because their model identity cannot be \
+ verified; submit text only or use a separate explicitly pre-embedded index"
+ ));
+ continue;
+ }
+ if supplied_vector {
+ continue;
+ }
+ let Some(text) = obj.get(field).and_then(Value::as_str) else {
+ continue;
+ };
+ let chunks = semantic_chunker().chunk(text, None);
+ let texts = if chunks.len() <= 1 {
+ vec![text.to_string()]
} else {
chunks.iter().map(|c| c.text.clone()).collect()
};
- let vs = self.embedder.embed_batch(chunk_texts).await.map_err(|e| {
- // Item 9: preserve the embedder's 5xx class on ingest too —
- // a proxy outage must not masquerade as a 400 bad document.
- EngineError::Common(embed_backend_error(
- e,
- &format!("semantic_text embed failed for field [{field}]"),
- ))
- })?;
- vs.into_iter().filter(|v| !v.is_empty()).collect()
- } else {
- // Lexical fallback: deterministic per-chunk feature-hash.
- local_chunk_vectors(&text, dims)
+ jobs.push(EmbedJob {
+ source_idx,
+ field: field.clone(),
+ target: target.clone(),
+ dims: *dims,
+ original_text: text.to_string(),
+ texts,
+ });
+ }
+ }
+
+ let mut outputs: Vec<Option<Vec<Vec<f32>>>> = vec![None; jobs.len()];
+ let mut failures = pre_failures;
+ let mut admission_rejections = vec![false; sources.len()];
+
+ let embedder = self.embedder.read().await;
+ if embedder.is_active() {
+ // Bound caller-side windows as well as backend-internal batches.
+ // This avoids an arbitrarily large proxy payload and gives ONNX a
+ // useful cross-document scheduling window without giant padding.
+ const MAX_PASSAGES_PER_WINDOW: usize = 64;
+ let mut start = 0;
+ while start < jobs.len() {
+ let mut end = start;
+ let mut passages = 0;
+ while end < jobs.len()
+ && (end == start || passages + jobs[end].texts.len() <= MAX_PASSAGES_PER_WINDOW)
+ {
+ passages += jobs[end].texts.len();
+ end += 1;
+ }
+ let texts: Vec<String> = jobs[start..end]
+ .iter()
+ .flat_map(|job| job.texts.iter().cloned())
+ .collect();
+ match embedder.embed_batch(texts).await {
+ Ok(vectors) if vectors.len() == passages => {
+ let mut offset = 0;
+ for job_idx in start..end {
+ let count = jobs[job_idx].texts.len();
+ let job_vectors = &vectors[offset..offset + count];
+ if let Some(vector) = job_vectors
+ .iter()
+ .find(|vector| vector.len() != jobs[job_idx].dims)
+ {
+ failures[jobs[job_idx].source_idx] = Some(format!(
+ "field [{}]: embedding backend returned width {}, mapping requires {}",
+ jobs[job_idx].field,
+ vector.len(),
+ jobs[job_idx].dims
+ ));
+ } else {
+ outputs[job_idx] = Some(job_vectors.to_vec());
+ }
+ offset += count;
+ }
+ }
+ batch_result => {
+ // Preserve per-item bulk behavior on a backend error:
+ // retry each document-field job separately and only
+ // fail the sources whose own retry fails.
+ let batch_error = match batch_result {
+ Ok(v) => format!(
+ "embedding backend returned {} vectors for {passages} texts",
+ v.len()
+ ),
+ Err(e) => e.to_string(),
+ };
+ tracing::warn!(
+ error = %batch_error,
+ jobs = end - start,
+ passages,
+ "semantic embedding window failed; retrying per item"
+ );
+ for job_idx in start..end {
+ match embedder.embed_batch(jobs[job_idx].texts.clone()).await {
+ Ok(v)
+ if v.len() == jobs[job_idx].texts.len()
+ && v.iter()
+ .all(|vector| vector.len() == jobs[job_idx].dims) =>
+ {
+ outputs[job_idx] = Some(v);
+ }
+ Ok(v) => {
+ let widths = v
+ .iter()
+ .map(Vec::len)
+ .collect::<std::collections::BTreeSet<_>>();
+ failures[jobs[job_idx].source_idx] = Some(format!(
+ "field [{}]: embedding backend returned {} vectors with widths {:?} for {} texts; mapping requires width {}",
+ jobs[job_idx].field,
+ v.len(),
+ widths,
+ jobs[job_idx].texts.len(),
+ jobs[job_idx].dims,
+ ));
+ }
+ Err(e) => {
+ admission_rejections[jobs[job_idx].source_idx] =
+ is_embedding_admission_rejection(&e);
+ failures[jobs[job_idx].source_idx] =
+ Some(format!("field [{}]: {e}", jobs[job_idx].field));
+ }
+ }
+ }
+ }
+ }
+ start = end;
+ }
+ } else {
+ for (job_idx, job) in jobs.iter().enumerate() {
+ // The lexical fallback intentionally keeps its deterministic
+ // per-document chunking behavior.
+ outputs[job_idx] = Some(local_chunk_vectors(&job.original_text, job.dims));
+ }
+ }
+
+ for (job_idx, job) in jobs.into_iter().enumerate() {
+ if failures[job.source_idx].is_some() {
+ continue;
+ }
+ let chunk_vecs: Vec<Vec<f32>> = outputs[job_idx]
+ .take()
+ .unwrap_or_default()
+ .into_iter()
+ .filter(|v| !v.is_empty())
+ .collect();
+ let Some(obj) = sources[job.source_idx].as_object_mut() else {
+ continue;
};
// Pooled vector for `target`: single chunk is stored as-is (no
// re-normalization, matching pre-chunking behavior); multiple
@@ -5076,7 +5274,7 @@ impl Index {
_ => mean_pool_normalize(&chunk_vecs),
};
let arr: Vec<Value> = pooled.into_iter().map(|f| Value::from(f as f64)).collect();
- obj.insert(target.clone(), Value::Array(arr));
+ obj.insert(job.target.clone(), Value::Array(arr));
// Per-chunk passage vectors — persisted ONLY when the document
// actually spans more than one chunk, under `<target>_chunks` as an
// array of vectors. Semantic search prefers these and scores by the
@@ -5088,10 +5286,26 @@ impl Index {
.iter()
.map(|cv| Value::Array(cv.iter().map(|f| Value::from(*f as f64)).collect()))
.collect();
- obj.insert(format!("{target}_chunks"), Value::Array(chunks_json));
+ obj.insert(format!("{}_chunks", job.target), Value::Array(chunks_json));
}
}
- Ok(source)
+
+ sources
+ .into_iter()
+ .enumerate()
+ .map(|(idx, source)| match failures[idx].take() {
+ Some(error) if admission_rejections[idx] => Err(EngineError::Common(
+ xerj_common::XerjError::resource_exhausted(format!(
+ "semantic_text batch embed rejected before tokenization: {error}"
+ )),
+ )),
+ Some(error) => Err(EngineError::Common(embed_backend_error(
+ anyhow::anyhow!(error),
+ "semantic_text batch embed failed",
+ ))),
+ None => Ok(source),
+ })
+ .collect()
}
/// Hybrid (multi-query + fusion) executor. Recursively runs each
@@ -10158,6 +10372,26 @@ impl Index {
/// Add a field to the schema.
pub async fn add_field(&self, field: FieldConfig) -> Result<()> {
+ let activates_embedder = schema_needs_embedder(std::slice::from_ref(&field))
+ && !schema_needs_embedder(&self.schema.read().await.schema.fields);
+ let activated = if activates_embedder {
+ Some(make_embedder(&self.embedding_config)?)
+ } else {
+ None
+ };
+ if schema_has_embeddings(std::slice::from_ref(&field)) {
+ if self
+ .embedding_config
+ .mode
+ .eq_ignore_ascii_case("onnx-experimental")
+ {
+ validate_onnx_dimensions(std::slice::from_ref(&field))?;
+ }
+ ensure_embedding_identity_for_new_field(&self.data_dir, &self.embedding_config)?;
+ }
+ if let Some(embedder) = activated {
+ *self.embedder.write().await = embedder;
+ }
let mut schema = self.schema.write().await;
schema.add_field(field)?;
self.save_schema(&schema).await?;
@@ -15990,13 +16224,18 @@ fn build_highlight_fragments(
/// `default_endpoint`).
/// * `"neural"` — the built-in BERT embedder (requires the `neural`
/// cargo feature; falls back to lexical with a warning otherwise).
+/// * `"onnx-experimental"` — an explicit FP32 ONNX model and tokenizer
+/// loaded through ONNX Runtime (requires the `onnx-experimental` feature).
/// * `"lexical"` — the built-in feature-hash embedder.
/// * `"auto"` (default / unknown) — proxy when `default_endpoint` is set,
/// else lexical. Preserves the historical behavior exactly.
///
-/// Any misconfiguration degrades to the lexical fallback (never a crash);
-/// a `semantic` query with no active backend against a plain vector field
-/// still returns a helpful 400.
+/// Proxy initialization and an unavailable Candle feature degrade to the
+/// lexical fallback with a warning. ONNX is deliberately fail-closed: missing
+/// assets, a feature-disabled build, or an incompatible model returns an
+/// actionable configuration error instead of silently changing vector
+/// identity. A `semantic` query with no active backend against a plain vector
+/// field still returns a helpful 400.
/// Map an embedder failure to the correct ES class (item 9). An embedding
/// backend outage or misconfiguration is a server-side (5xx) failure, never a
/// 400 `invalid_query`: downgrading it hid proxy failures behind a client
@@ -16004,23 +16243,389 @@ fn build_highlight_fragments(
/// `EmbeddingError` (500, `circuit_breaking_exception`) with `ctx` prepended.
/// Takes `impl Display` because the `Embedder` surfaces `anyhow::Error` (the
/// classification of transient-vs-permanent already happened inside the proxy).
-fn embed_backend_error(e: impl std::fmt::Display, ctx: &str) -> xerj_common::XerjError {
+fn is_embedding_admission_rejection(e: &anyhow::Error) -> bool {
+ #[cfg(feature = "onnx-experimental")]
+ if e.downcast_ref::<xerj_ai::embedder::OnnxAdmissionError>()
+ .is_some()
+ {
+ return true;
+ }
+ false
+}
+
+fn embed_backend_error(e: anyhow::Error, ctx: &str) -> xerj_common::XerjError {
+ if is_embedding_admission_rejection(&e) {
+ return xerj_common::XerjError::resource_exhausted(format!("{ctx}: {e}"));
+ }
xerj_common::XerjError::embedding(format!("{ctx}: {e}"))
}
-fn make_embedder(cfg: &xerj_common::config::EmbeddingConfig) -> xerj_ai::Embedder {
+const EMBEDDING_IDENTITY_FILE: &str = "embedding_identity.json";
+
+#[derive(Debug, PartialEq, Serialize, Deserialize)]
+struct PersistedEmbeddingIdentity {
+ version: u32,
+ backend: String,
+ model_sha256: String,
+ tokenizer_sha256: String,
+ dimensions: usize,
+ pooling: String,
+ max_tokens: usize,
+}
+
+fn schema_has_embeddings(fields: &[FieldConfig]) -> bool {
+ fields
+ .iter()
+ .any(|field| field.embedding.is_some() || schema_has_embeddings(&field.fields))
+}
+
+fn schema_needs_embedder(fields: &[FieldConfig]) -> bool {
+ fields.iter().any(|field| {
+ field.embedding.is_some()
+ || matches!(field.field_type, FieldType::Vector | FieldType::Chunk)
+ || schema_needs_embedder(&field.fields)
+ })
+}
+
+fn validate_onnx_dimensions(fields: &[FieldConfig]) -> Result<()> {
+ for field in fields {
+ if field.embedding.is_some() {
+ let dims = field
+ .options
+ .dimensions
+ .unwrap_or(xerj_ai::local::DEFAULT_DIMS);
+ if dims != 384 {
+ return Err(EngineError::Common(xerj_common::XerjError::embedding(
+ format!(
+ "semantic field [{}] declares {dims} dimensions, but the experimental \
+ MiniLM ONNX backend returns exactly 384. Change the mapping to 384 or \
+ use an embedding backend matching the declared dimensions.",
+ field.name
+ ),
+ )));
+ }
+ }
+ validate_onnx_dimensions(&field.fields)?;
+ }
+ Ok(())
+}
+
+fn sha256_file(path: &Path) -> Result<String> {
+ use sha2::{Digest, Sha256};
+ use std::collections::HashMap;
+ use std::io::Read;
+ use std::sync::{Mutex, OnceLock};
+ use std::time::UNIX_EPOCH;
+
+ static CACHE: OnceLock<Mutex<HashMap<(PathBuf, u64, u128), String>>> = OnceLock::new();
+ let canonical = path.canonicalize().map_err(|e| {
+ EngineError::Common(xerj_common::XerjError::embedding(format!(
+ "cannot resolve embedding asset {}: {e}",
+ path.display()
+ )))
+ })?;
+ let metadata = std::fs::metadata(&canonical)?;
+ let modified = metadata
+ .modified()
+ .ok()
+ .and_then(|time| time.duration_since(UNIX_EPOCH).ok())
+ .map(|duration| duration.as_nanos())
+ .unwrap_or(0);
+ let key = (canonical.clone(), metadata.len(), modified);
+ let mut cache = CACHE
+ .get_or_init(|| Mutex::new(HashMap::new()))
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner());
+ if let Some(hash) = cache.get(&key).cloned() {
+ return Ok(hash);
+ }
+ let mut file = std::fs::File::open(&canonical).map_err(|e| {
+ EngineError::Common(xerj_common::XerjError::embedding(format!(
+ "cannot fingerprint embedding asset {}: {e}",
+ path.display()
+ )))
+ })?;
+ let mut hasher = Sha256::new();
+ let mut buf = [0_u8; 1024 * 1024];
+ loop {
+ let n = file.read(&mut buf)?;
+ if n == 0 {
+ break;
+ }
+ hasher.update(&buf[..n]);
+ }
+ let hash = format!("{:x}", hasher.finalize());
+ cache.insert(key, hash.clone());
+ Ok(hash)
+}
+
+fn configured_onnx_identity(
+ cfg: &xerj_common::config::EmbeddingConfig,
+) -> Result<Option<PersistedEmbeddingIdentity>> {
+ if cfg.mode.trim().to_ascii_lowercase() != "onnx-experimental" {
+ return Ok(None);
+ }
+ #[cfg(not(feature = "onnx-experimental"))]
+ return Err(EngineError::Common(xerj_common::XerjError::embedding(
+ "onnx-experimental was requested but this engine build does not include the \
+ onnx-experimental feature; no embedding identity marker was written",
+ )));
+ #[cfg(feature = "onnx-experimental")]
+ Ok(Some(PersistedEmbeddingIdentity {
+ version: 1,
+ backend: "onnx-experimental".into(),
+ model_sha256: sha256_file(Path::new(&cfg.onnx_model_path))?,
+ tokenizer_sha256: sha256_file(Path::new(&cfg.onnx_tokenizer_path))?,
+ dimensions: 384,
+ pooling: "attention-mask-mean+l2-normalize".into(),
+ max_tokens: 512,
+ }))
+}
+
+/// Fail closed rather than mixing vectors produced by different model,
+/// tokenizer, pooling, or backend configurations after a restart/resume.
+fn validate_embedding_identity(
+ index_dir: &Path,
+ schema: &Schema,
+ cfg: &xerj_common::config::EmbeddingConfig,
+ new_index: bool,
+ has_documents: bool,
+) -> Result<()> {
+ if !schema_has_embeddings(&schema.fields) {
+ return Ok(());
+ }
+ let marker_path = index_dir.join(EMBEDDING_IDENTITY_FILE);
+ let configured = configured_onnx_identity(cfg)?;
+ if configured.is_some() {
+ validate_onnx_dimensions(&schema.fields)?;
+ }
+ if marker_path.exists() {
+ let persisted: PersistedEmbeddingIdentity =
+ serde_json::from_slice(&std::fs::read(&marker_path)?).map_err(|e| {
+ EngineError::Common(xerj_common::XerjError::embedding(format!(
+ "cannot read {}: {e}; restore the marker or reindex this index",
+ marker_path.display()
+ )))
+ })?;
+ return match configured {
+ Some(current) if current == persisted => Ok(()),
+ Some(current) => Err(EngineError::Common(xerj_common::XerjError::embedding(
+ format!(
+ "embedding identity mismatch for index {}: persisted backend={} \
+ model={} tokenizer={}, configured backend={} model={} tokenizer={}. \
+ Refusing to mix incompatible vector spaces. Restore the original assets \
+ or re-run autoindex with --fresh and a new index prefix.",
+ index_dir.display(),
+ persisted.backend,
+ persisted.model_sha256,
+ persisted.tokenizer_sha256,
+ current.backend,
+ current.model_sha256,
+ current.tokenizer_sha256,
+ ),
+ ))),
+ None => Err(EngineError::Common(xerj_common::XerjError::embedding(
+ format!(
+ "index {} contains ONNX vectors but the server is configured for \
+ embedding.mode={:?}. Restart with onnx-experimental and the original \
+ assets, or reindex under a new prefix.",
+ index_dir.display(),
+ cfg.mode
+ ),
+ ))),
+ };
+ }
+ let Some(identity) = configured else {
+ return Ok(());
+ };
+ if !new_index && has_documents {
+ return Err(EngineError::Common(xerj_common::XerjError::embedding(
+ format!(
+ "existing semantic index {} has no embedding identity marker. It may contain \
+ vectors from another backend, so ONNX cannot be enabled safely in place. \
+ Re-run autoindex with --fresh and a new --prefix, or rebuild the index from \
+ source.",
+ index_dir.display()
+ ),
+ )));
+ }
+ write_file_atomic(&marker_path, &serde_json::to_vec_pretty(&identity)?)?;
+ Ok(())
+}
+
+fn ensure_embedding_identity_for_new_field(
+ index_dir: &Path,
+ cfg: &xerj_common::config::EmbeddingConfig,
+) -> Result<()> {
+ let Some(configured) = configured_onnx_identity(cfg)? else {
+ return Ok(());
+ };
+ let marker_path = index_dir.join(EMBEDDING_IDENTITY_FILE);
+ if marker_path.exists() {
+ let persisted: PersistedEmbeddingIdentity =
+ serde_json::from_slice(&std::fs::read(&marker_path)?).map_err(|e| {
+ EngineError::Common(xerj_common::XerjError::embedding(format!(
+ "cannot read {}: {e}; restore the marker or reindex this index",
+ marker_path.display()
+ )))
+ })?;
+ if persisted != configured {
+ return Err(EngineError::Common(xerj_common::XerjError::embedding(
+ format!(
+ "embedding identity mismatch for index {}; refusing to add a semantic \
+ field in a different vector space. Restore the original ONNX assets or \
+ create a new index.",
+ index_dir.display()
+ ),
+ )));
+ }
+ return Ok(());
+ }
+ // This field does not exist yet, so existing documents cannot contain its
+ // derived vectors. Pin the identity before publishing the mapping; all
+ // later writes and restarts are then checked against it.
+ write_file_atomic(&marker_path, &serde_json::to_vec_pretty(&configured)?)?;
+ Ok(())
+}
+
+#[cfg(test)]
+mod embedding_identity_tests {
+ use super::*;
+
+ fn semantic_schema() -> Schema {
+ let mut schema = Schema::empty();
+ schema
+ .add_field(FieldConfig::new("body", FieldType::Text).with_embedding(
+ xerj_common::types::EmbeddingConfig {
+ endpoint: None,
+ model: None,
+ target_field: None,
+ },
+ ))
+ .unwrap();
+ schema
+ }
+
+ fn onnx_config(dir: &Path, suffix: &str) -> xerj_common::config::EmbeddingConfig {
+ let model = dir.join(format!("model-{suffix}.onnx"));
+ let tokenizer = dir.join(format!("tokenizer-{suffix}.json"));
+ std::fs::write(&model, format!("model-{suffix}")).unwrap();
+ std::fs::write(&tokenizer, format!("tokenizer-{suffix}")).unwrap();
+ xerj_common::config::EmbeddingConfig {
+ mode: "onnx-experimental".into(),
+ onnx_model_path: model.to_string_lossy().into_owned(),
+ onnx_tokenizer_path: tokenizer.to_string_lossy().into_owned(),
+ ..Default::default()
+ }
+ }
+
+ #[test]
+ fn onnx_identity_is_persisted_and_matching_restart_is_allowed() {
+ let dir = tempfile::tempdir().unwrap();
+ let cfg = onnx_config(dir.path(), "a");
+ validate_embedding_identity(dir.path(), &semantic_schema(), &cfg, true, false).unwrap();
+ assert!(dir.path().join(EMBEDDING_IDENTITY_FILE).is_file());
+ validate_embedding_identity(dir.path(), &semantic_schema(), &cfg, false, true).unwrap();
+ }
+
+ #[test]
+ fn changed_onnx_assets_are_rejected() {
+ let dir = tempfile::tempdir().unwrap();
+ let first = onnx_config(dir.path(), "a");
+ validate_embedding_identity(dir.path(), &semantic_schema(), &first, true, false).unwrap();
+ let changed = onnx_config(dir.path(), "b");
+ let error =
+ validate_embedding_identity(dir.path(), &semantic_schema(), &changed, false, true)
+ .unwrap_err()
+ .to_string();
+ assert!(error.contains("identity mismatch"), "{error}");
+ assert!(error.contains("new index prefix"), "{error}");
+ }
+
+ #[test]
+ fn markerless_existing_index_only_fails_closed_for_onnx() {
+ let dir = tempfile::tempdir().unwrap();
+ let lexical = xerj_common::config::EmbeddingConfig::default();
+ validate_embedding_identity(dir.path(), &semantic_schema(), &lexical, false, true).unwrap();
+ let onnx = onnx_config(dir.path(), "a");
+ let error = validate_embedding_identity(dir.path(), &semantic_schema(), &onnx, false, true)
+ .unwrap_err()
+ .to_string();
+ assert!(error.contains("no embedding identity marker"), "{error}");
+ }
+
+ #[test]
+ fn later_semantic_field_pins_identity_before_first_write() {
+ let dir = tempfile::tempdir().unwrap();
+ let onnx = onnx_config(dir.path(), "a");
+ ensure_embedding_identity_for_new_field(dir.path(), &onnx).unwrap();
+ assert!(dir.path().join(EMBEDDING_IDENTITY_FILE).is_file());
+ ensure_embedding_identity_for_new_field(dir.path(), &onnx).unwrap();
+ }
+
+ #[test]
+ fn onnx_rejects_non_384_semantic_mapping() {
+ let dir = tempfile::tempdir().unwrap();
+ let onnx = onnx_config(dir.path(), "a");
+ let mut schema = semantic_schema();
+ schema.fields[0].options.dimensions = Some(768);
+ let error = validate_embedding_identity(dir.path(), &schema, &onnx, true, false)
+ .unwrap_err()
+ .to_string();
+ assert!(error.contains("declares 768 dimensions"), "{error}");
+ assert!(error.contains("exactly 384"), "{error}");
+ }
+
+ #[cfg(feature = "onnx-experimental")]
+ #[test]
+ fn onnx_admission_rejection_maps_to_retryable_429() {
+ let error = anyhow::Error::new(xerj_ai::embedder::OnnxAdmissionError {
+ reason: "global admission full; retry".into(),
+ });
+ let mapped = embed_backend_error(error, "semantic query embed failed");
+ assert_eq!(mapped.http_status(), 429);
+ assert!(mapped.to_string().contains("retry"));
+ }
+
+ #[cfg(feature = "onnx-experimental")]
+ #[test]
+ fn non_vector_schema_does_not_touch_onnx_assets() {
+ let config = xerj_common::config::EmbeddingConfig {
+ mode: "onnx-experimental".into(),
+ onnx_model_path: "/definitely/missing/model.onnx".into(),
+ onnx_tokenizer_path: "/definitely/missing/tokenizer.json".into(),
+ ..Default::default()
+ };
+ let embedder = make_embedder_for_schema(&Schema::empty(), &config).unwrap();
+ assert!(embedder.describe().contains("lexical"));
+
+ let mut schema = semantic_schema();
+ schema.fields[0].options.dimensions = Some(384);
+ let error = make_embedder_for_schema(&schema, &config)
+ .err()
+ .expect("semantic schema must activate and validate ONNX")
+ .to_string();
+ assert!(error.contains("readable FP32 model"), "{error}");
+ }
+}
+
+fn make_embedder(cfg: &xerj_common::config::EmbeddingConfig) -> Result<xerj_ai::Embedder> {
let mode = cfg.mode.trim().to_ascii_lowercase();
let want_proxy = mode == "proxy"
|| (mode != "neural" && mode != "lexical" && !cfg.default_endpoint.is_empty());
if mode == "neural" {
- return make_neural_embedder(cfg);
+ return Ok(make_neural_embedder(cfg));
+ }
+
+ if mode == "onnx-experimental" {
+ return make_onnx_embedder(cfg);
}
if want_proxy {
if cfg.default_endpoint.is_empty() {
warn!("embedding.mode=proxy but embedding.default_endpoint is empty — falling back to lexical");
- return xerj_ai::Embedder::lexical();
+ return Ok(xerj_ai::Embedder::lexical());
}
let proxy_cfg = xerj_ai::embed::EmbeddingProxyConfig {
endpoint: cfg.default_endpoint.clone(),
@@ -16030,7 +16635,7 @@ fn make_embedder(cfg: &xerj_common::config::EmbeddingConfig) -> xerj_ai::Embedde
max_concurrent: 4,
max_retries: 3,
};
- return match xerj_ai::embed::EmbeddingProxy::new(proxy_cfg) {
+ return Ok(match xerj_ai::embed::EmbeddingProxy::new(proxy_cfg) {
Ok(p) => {
info!(endpoint = %cfg.default_endpoint, "embedding backend: external proxy");
xerj_ai::Embedder::proxy(p)
@@ -16039,10 +16644,100 @@ fn make_embedder(cfg: &xerj_common::config::EmbeddingConfig) -> xerj_ai::Embedde
warn!(error = %e, "embedding proxy init failed — falling back to lexical");
xerj_ai::Embedder::lexical()
}
- };
+ });
}
- xerj_ai::Embedder::lexical()
+ Ok(xerj_ai::Embedder::lexical())
+}
+
+fn make_embedder_for_schema(
+ schema: &Schema,
+ cfg: &xerj_common::config::EmbeddingConfig,
+) -> Result<xerj_ai::Embedder> {
+ if schema_needs_embedder(&schema.fields) {
+ make_embedder(cfg)
+ } else {
+ Ok(xerj_ai::Embedder::lexical())
+ }
+}
+
+#[cfg(feature = "onnx-experimental")]
+fn make_onnx_embedder(cfg: &xerj_common::config::EmbeddingConfig) -> Result<xerj_ai::Embedder> {
+ use std::path::PathBuf;
+ let model_path = PathBuf::from(cfg.onnx_model_path.trim());
+ let tokenizer_path = PathBuf::from(cfg.onnx_tokenizer_path.trim());
+ if cfg.onnx_model_path.trim().is_empty() || !model_path.is_file() {
+ return Err(EngineError::Common(xerj_common::XerjError::embedding(
+ format!(
+ "embedding.mode=onnx-experimental requires a readable FP32 model file; \
+ set --onnx-model <PATH> or embedding.onnx_model_path (got {:?})",
+ cfg.onnx_model_path
+ ),
+ )));
+ }
+ if cfg.onnx_tokenizer_path.trim().is_empty() || !tokenizer_path.is_file() {
+ return Err(EngineError::Common(xerj_common::XerjError::embedding(
+ format!(
+ "embedding.mode=onnx-experimental requires the matching tokenizer.json; \
+ set --onnx-tokenizer <PATH> or embedding.onnx_tokenizer_path (got {:?})",
+ cfg.onnx_tokenizer_path
+ ),
+ )));
+ }
+ if cfg.onnx_max_pending == 0
+ || cfg.onnx_max_batch == 0
+ || cfg.onnx_padded_token_budget == 0
+ || cfg.onnx_max_inflight_calls == 0
+ || cfg.onnx_max_input_bytes_per_call == 0
+ || cfg.onnx_max_inflight_input_bytes == 0
+ || cfg.onnx_max_input_bytes_per_call > cfg.onnx_max_inflight_input_bytes
+ || cfg.onnx_max_inflight_input_bytes > u32::MAX as usize
+ {
+ return Err(EngineError::Common(xerj_common::XerjError::embedding(
+ format!(
+ "invalid ONNX limits: pending={}, batch={}, padded_tokens={}, inflight_calls={}, \
+ bytes_per_call={}, inflight_bytes={}; all must be non-zero, bytes_per_call \
+ must not exceed inflight_bytes, and inflight_bytes must be <= {}",
+ cfg.onnx_max_pending,
+ cfg.onnx_max_batch,
+ cfg.onnx_padded_token_budget,
+ cfg.onnx_max_inflight_calls,
+ cfg.onnx_max_input_bytes_per_call,
+ cfg.onnx_max_inflight_input_bytes,
+ u32::MAX
+ ),
+ )));
+ }
+ let onnx_cfg = xerj_ai::embedder::OnnxConfig {
+ model_sha256: sha256_file(&model_path)?,
+ tokenizer_sha256: sha256_file(&tokenizer_path)?,
+ model_path,
+ tokenizer_path,
+ intra_threads: cfg.onnx_intra_threads.max(1),
+ microbatch: xerj_ai::onnx::MicrobatchConfig {
+ max_pending: cfg.onnx_max_pending,
+ max_batch: cfg.onnx_max_batch,
+ padded_token_budget: cfg.onnx_padded_token_budget,
+ },
+ max_inflight_calls: cfg.onnx_max_inflight_calls,
+ max_input_bytes_per_call: cfg.onnx_max_input_bytes_per_call,
+ max_inflight_input_bytes: cfg.onnx_max_inflight_input_bytes,
+ };
+ info!(
+ model = %onnx_cfg.model_path.display(),
+ tokenizer = %onnx_cfg.tokenizer_path.display(),
+ "embedding backend: experimental ONNX Runtime (loads on first use)"
+ );
+ Ok(xerj_ai::Embedder::onnx(onnx_cfg))
+}
+
+#[cfg(not(feature = "onnx-experimental"))]
+fn make_onnx_embedder(_cfg: &xerj_common::config::EmbeddingConfig) -> Result<xerj_ai::Embedder> {
+ Err(EngineError::Common(xerj_common::XerjError::embedding(
+ "embedding.mode=onnx-experimental requested, but this binary was built without it; \
+ rebuild xerj-server with --features onnx-experimental. XERJ will not silently \
+ substitute lexical vectors because that would mislabel search quality.",
+ )))
}
/// Construct the built-in neural embedder when compiled in; otherwise warn
diff --git a/engine/crates/xerj-engine/tests/integration.rs b/engine/crates/xerj-engine/tests/integration.rs
index ed8042035..da84261f2 100644
--- a/engine/crates/xerj-engine/tests/integration.rs
+++ b/engine/crates/xerj-engine/tests/integration.rs
@@ -4905,6 +4905,107 @@ async fn test_semantic_text_match_survives_flush() {
);
}
+#[tokio::test]
+async fn test_semantic_text_bulk_preserves_item_order_status_and_vectors() {
+ let dir = TempDir::new().unwrap();
+ let engine = make_engine(&dir);
+
+ let mut schema = Schema::empty();
+ let mut fc = FieldConfig::new("content", FieldType::Text);
+ fc.options.dimensions = Some(16);
+ fc.options.similarity = Some("cosine".to_string());
+ fc.embedding = Some(xerj_common::types::EmbeddingConfig {
+ endpoint: None,
+ model: None,
+ target_field: Some("content_vector".to_string()),
+ });
+ schema.fields.push(fc);
+ engine.create_index("sem-bulk", schema).unwrap();
+
+ let body = concat!(
+ "{\"index\":{\"_index\":\"sem-bulk\",\"_id\":\"a\"}}\n",
+ "{\"content\":\"alpha financial report\"}\n",
+ "{\"index\":{\"_index\":\"sem-bulk\",\"_id\":\"b\"}}\n",
+ "{\"content\":\"beta quarterly earnings\"}\n",
+ "{\"index\":{\"_index\":\"sem-bulk\",\"_id\":\"a\"}}\n",
+ "{\"content\":\"alpha annual report updated\"}\n",
+ );
+ let result = xerj_engine::bulk::process_bulk(&engine, None, body).await;
+ assert!(!result.errors, "{:?}", result.items);
+ assert_eq!(result.items.len(), 3);
+ assert_eq!(result.items[0].id, "a");
+ assert_eq!(result.items[0].status, 201);
+ assert_eq!(result.items[1].id, "b");
+ assert_eq!(result.items[1].status, 201);
+ assert_eq!(result.items[2].id, "a");
+ assert_eq!(result.items[2].status, 200);
+
+ let idx = engine.get_index("sem-bulk").unwrap();
+ let a = idx.get_document("a").await.unwrap().unwrap();
+ let b = idx.get_document("b").await.unwrap().unwrap();
+ assert_eq!(a["content"], "alpha annual report updated");
+ assert_eq!(a["content_vector"].as_array().unwrap().len(), 16);
+ assert_eq!(b["content_vector"].as_array().unwrap().len(), 16);
+}
+
+#[cfg(feature = "onnx-experimental")]
+#[tokio::test]
+async fn test_semantic_bulk_preserves_onnx_admission_429() {
+ let dir = TempDir::new().unwrap();
+ let model = dir.path().join("model.onnx");
+ let tokenizer = dir.path().join("tokenizer.json");
+ std::fs::write(&model, b"not loaded: admission rejects first").unwrap();
+ std::fs::write(&tokenizer, b"not loaded: admission rejects first").unwrap();
+
+ let mut config = Config::default();
+ config.server.data_dir = dir.path().join("data").to_string_lossy().into_owned();
+ config.embedding.mode = "onnx-experimental".into();
+ config.embedding.onnx_model_path = model.to_string_lossy().into_owned();
+ config.embedding.onnx_tokenizer_path = tokenizer.to_string_lossy().into_owned();
+ config.embedding.onnx_max_input_bytes_per_call = 1;
+ config.embedding.onnx_max_inflight_input_bytes = 1;
+ let engine = Engine::new(config).unwrap();
+
+ let mut schema = Schema::empty();
+ let mut field = FieldConfig::new("content", FieldType::Text);
+ field.options.dimensions = Some(384);
+ field.embedding = Some(xerj_common::types::EmbeddingConfig {
+ endpoint: None,
+ model: None,
+ target_field: Some("content_vector".into()),
+ });
+ schema.fields.push(field);
+ engine.create_index("sem-onnx-overload", schema).unwrap();
+
+ let body = concat!(
+ "{\"index\":{\"_index\":\"sem-onnx-overload\",\"_id\":\"a\"}}\n",
+ "{\"content\":\"too large\"}\n",
+ );
+ let result = xerj_engine::bulk::process_bulk(&engine, None, body).await;
+ assert!(result.errors);
+ assert_eq!(result.items.len(), 1);
+ assert_eq!(result.items[0].status, 429, "{:?}", result.items[0]);
+ assert!(
+ result.items[0]
+ .error
+ .as_deref()
+ .unwrap_or_default()
+ .contains("rejected before tokenization"),
+ "{:?}",
+ result.items[0]
+ );
+ assert!(
+ engine
+ .get_index("sem-onnx-overload")
+ .unwrap()
+ .get_document("a")
+ .await
+ .unwrap()
+ .is_none(),
+ "rejected bulk item must not be persisted"
+ );
+}
+
/// Blocker 5: snapshot RESTORE ignored the request `indices` filter and
/// rewrote EVERY index in the snapshot with snapshot-time state, silently
/// destroying all writes made since. The filter must select exactly the
diff --git a/engine/crates/xerj-server/Cargo.toml b/engine/crates/xerj-server/Cargo.toml
index b7e04a9c5..189e3aeb4 100644
--- a/engine/crates/xerj-server/Cargo.toml
+++ b/engine/crates/xerj-server/Cargo.toml
@@ -65,6 +65,7 @@ prost.workspace = true
# (the lexical + proxy backends still work).
default = ["neural"]
neural = ["xerj-engine/neural"]
+onnx-experimental = ["xerj-engine/onnx-experimental"]
[target.'cfg(not(target_env = "msvc"))'.dependencies]
tikv-jemallocator = "0.6"
diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rs
index 6f3c34cd7..d68995fd8 100644
--- a/engine/crates/xerj-server/src/main.rs
+++ b/engine/crates/xerj-server/src/main.rs
@@ -78,6 +78,8 @@ struct CliArgs {
data_dir: Option<String>,
insecure: bool,
embed_mode: Option<String>,
+ onnx_model: Option<String>,
+ onnx_tokenizer: Option<String>,
}
fn parse_args() -> CliArgs {
@@ -86,6 +88,8 @@ fn parse_args() -> CliArgs {
let mut data_dir = None;
let mut insecure = false;
let mut embed_mode = None;
+ let mut onnx_model = None;
+ let mut onnx_tokenizer = None;
while let Some(arg) = args.next() {
match arg.as_str() {
@@ -101,6 +105,8 @@ fn parse_args() -> CliArgs {
"--embed-mode" => {
embed_mode = args.next();
}
+ "--onnx-model" => onnx_model = args.next(),
+ "--onnx-tokenizer" => onnx_tokenizer = args.next(),
"--help" | "-h" => {
print_help();
std::process::exit(0);
@@ -121,6 +127,8 @@ fn parse_args() -> CliArgs {
data_dir,
insecure,
embed_mode,
+ onnx_model,
+ onnx_tokenizer,
}
}
@@ -135,13 +143,20 @@ fn print_help() {
--config, -c <PATH> Path to TOML config file\n\
--data-dir, -d <PATH> Override data directory\n\
--insecure, -k Disable TLS\n\
- --embed-mode <MODE> Embedding backend: lexical | neural | proxy | auto\n\
+ --embed-mode <MODE> Embedding backend: lexical | neural | proxy | auto |\n\
+ onnx-experimental\n\
lexical built-in feature-hash (default; offline, not neural)\n\
neural built-in BERT (all-MiniLM-L6-v2) — real semantics,\n\
in-process; auto-downloads the model (~90 MB) on\n\
first use, then runs from cache. Just add the flag.\n\
proxy external OpenAI-compatible /v1/embeddings endpoint\n\
auto proxy if embedding.default_endpoint is set, else lexical\n\
+ onnx-experimental local all-MiniLM-L6-v2-compatible FP32\n\
+ ONNX prototype; width 384, named BERT inputs and\n\
+ token output; requires matching tokenizer.json;\n\
+ no auto-download\n\
+ --onnx-model <PATH> compatible FP32 MiniLM ONNX model\n\
+ --onnx-tokenizer <PATH> tokenizer.json from the same model/export\n\
--help, -h Show this help\n\
--version, -V Print version and exit\n\
\n\
@@ -153,7 +168,9 @@ fn print_help() {
ENVIRONMENT:\n\
XERJ_LOG Log level filter (default: info)\n\
XERJ_CONFIG Config file path\n\
- XERJ_EMBED_MODE Embedding backend (lexical|neural|proxy|auto)\n",
+ XERJ_EMBED_MODE Embedding backend (lexical|neural|proxy|auto|onnx-experimental)\n\
+ XERJ_ONNX_MODEL FP32 ONNX model path\n\
+ XERJ_ONNX_TOKENIZER matching tokenizer.json path\n",
env!("CARGO_PKG_VERSION")
);
}
@@ -246,7 +263,8 @@ fn load_config(args: &CliArgs) -> Result<Config> {
}
// Embedding backend override: `--embed-mode` flag or `XERJ_EMBED_MODE`
- // env (flag wins). Accepts `lexical` | `neural` | `proxy` | `auto`.
+ // env (flag wins). Also accepts `onnx-experimental` in a feature-enabled
+ // build with explicit model and tokenizer paths.
if let Some(mode) = args
.embed_mode
.clone()
@@ -254,12 +272,69 @@ fn load_config(args: &CliArgs) -> Result<Config> {
{
let mode = mode.trim().to_ascii_lowercase();
match mode.as_str() {
- "lexical" | "neural" | "proxy" | "auto" => {
+ "lexical" | "neural" | "proxy" | "auto" | "onnx-experimental" => {
info!("embedding.mode = {mode} (from CLI/env)");
cfg.embedding.mode = mode;
}
other => {
- warn!("ignoring unknown --embed-mode '{other}' (use lexical|neural|proxy|auto)");
+ anyhow::bail!("unknown --embed-mode '{other}'; use lexical|neural|proxy|auto|onnx-experimental");
+ }
+ }
+ }
+ if let Some(path) = args
+ .onnx_model
+ .clone()
+ .or_else(|| std::env::var("XERJ_ONNX_MODEL").ok())
+ {
+ cfg.embedding.onnx_model_path = path;
+ }
+ if let Some(path) = args
+ .onnx_tokenizer
+ .clone()
+ .or_else(|| std::env::var("XERJ_ONNX_TOKENIZER").ok())
+ {
+ cfg.embedding.onnx_tokenizer_path = path;
+ }
+ if cfg.embedding.mode == "onnx-experimental" {
+ #[cfg(not(feature = "onnx-experimental"))]
+ anyhow::bail!(
+ "onnx-experimental was requested, but this binary was built without it; \
+ rebuild with `cargo build --release -p xerj-server --features onnx-experimental`"
+ );
+ #[cfg(feature = "onnx-experimental")]
+ {
+ let model = Path::new(&cfg.embedding.onnx_model_path);
+ let tokenizer = Path::new(&cfg.embedding.onnx_tokenizer_path);
+ if !model.is_file() {
+ anyhow::bail!(
+ "onnx-experimental needs a readable FP32 model file at {:?}; \
+ pass --onnx-model <PATH> (no model is downloaded automatically)",
+ cfg.embedding.onnx_model_path
+ );
+ }
+ if !tokenizer.is_file() {
+ anyhow::bail!(
+ "onnx-experimental needs the matching tokenizer.json at {:?}; \
+ pass --onnx-tokenizer <PATH>",
+ cfg.embedding.onnx_tokenizer_path
+ );
+ }
+ if cfg.embedding.onnx_max_inflight_calls == 0
+ || cfg.embedding.onnx_max_input_bytes_per_call == 0
+ || cfg.embedding.onnx_max_inflight_input_bytes == 0
+ || cfg.embedding.onnx_max_input_bytes_per_call
+ > cfg.embedding.onnx_max_inflight_input_bytes
+ || cfg.embedding.onnx_max_inflight_input_bytes > u32::MAX as usize
+ {
+ anyhow::bail!(
+ "invalid ONNX admission limits: inflight_calls={}, bytes_per_call={}, \
+ inflight_bytes={}; use non-zero values, keep bytes_per_call <= \
+ inflight_bytes, and inflight_bytes <= {}",
+ cfg.embedding.onnx_max_inflight_calls,
+ cfg.embedding.onnx_max_input_bytes_per_call,
+ cfg.embedding.onnx_max_inflight_input_bytes,
+ u32::MAX
+ );
}
}
}
@@ -461,9 +536,18 @@ async fn build_tls_config(cfg: &Config) -> Result<Option<RustlsConfig>> {
// ─────────────────────────────────────────────────────────────────────────────
fn init_tracing(logging: &xerj_common::config::LoggingConfig) {
- let filter = EnvFilter::try_from_env("XERJ_LOG")
+ let mut filter = EnvFilter::try_from_env("XERJ_LOG")
.or_else(|_| EnvFilter::try_from_env("RUST_LOG"))
.unwrap_or_else(|_| EnvFilter::new("info"));
+ let onnx_log = std::env::var("XERJ_ONNX_LOG")
+ .unwrap_or_default()
+ .to_ascii_lowercase();
+ if !matches!(onnx_log.as_str(), "info" | "debug" | "verbose" | "trace") {
+ // `ort`'s tracing bridge emits allocator and graph-optimizer internals
+ // at INFO independently of the ONNX Runtime session severity. Keep
+ // normal XERJ logs concise; XERJ_ONNX_LOG explicitly opts back in.
+ filter = filter.add_directive("ort=warn".parse().expect("valid ort log directive"));
+ }
let builder = tracing_subscriber::fmt()
.with_env_filter(filter)
@@ -727,6 +811,8 @@ async fn run_cli_index(cmd: IndexCmdArgs) -> Result<()> {
data_dir: cmd.data_dir.clone(),
insecure: true,
embed_mode: None,
+ onnx_model: None,
+ onnx_tokenizer: None,
};
let mut cfg = load_config(&fake_cli)?;
// Tracing after config so the [logging] format applies (RC4-W4 item 6).