AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(ai): prototype bounded ONNX embeddings
xerj-org82c472fc3da4
diff --git a/docs/EXPERIMENTAL_ONNX.md b/docs/EXPERIMENTAL_ONNX.mdnew file mode 100644index 000000000..cd0203894--- /dev/null+++ b/docs/EXPERIMENTAL_ONNX.md@@ -0,0 +1,63 @@+# Experimental ONNX embedding backend++Status: embedding-library prototype, **not a server or CLI mode**.++XERJ's supported built-in neural backend remains Candle and+`--embed-mode neural` continues to mean Candle. Default and musl builds do not+compile ONNX Runtime.++The opt-in `xerj-ai` feature provides the exact FP32 MiniLM inference path and+a bounded length-aware microbatch API:++```bash+cd engine+cargo run --release -p xerj-ai --features onnx-experimental \+ --example onnx_experimental -- /path/model.onnx /path/tokenizer.json+```++The caller supplies an ONNX model with `input_ids`, `attention_mask`, and+`token_type_ids` inputs and a `[batch, sequence, 384]` `last_hidden_state`+output. The current verified artifact is the official FP32+`sentence-transformers/all-MiniLM-L6-v2` ONNX export.++Safety properties:++- safe `ort` API only;+- one session behind a mutex;+- bounded pending inputs, documents per batch, and padded token slots;+- length-aware grouping;+- exact restoration to caller input order;+- clear backpressure and invalid-budget errors.++## Measured prototype results++On the controlled 128-document mixed-length benchmark, exact FP32 ONNX plus+the retained scheduler processed 116.671 documents/s versus 9.045 documents/s+for Candle using the identical precomputed length-aware batch plan: **12.90x+at the `xerj-ai` embedding layer**. The median minimum same-document+Candle/ONNX cosine was `0.9999991655`.++This is not a claim of 12.90x end-to-end XERJ indexing. The benchmark excludes+document extraction, HTTP handling, persistence, and HNSW construction. ONNX+also used 16 intra-op threads; the measured CPU-efficiency improvement was+2.92x, smaller than the wall-time result. A server-level comparison is required+before publishing an indexing-throughput claim.++Measured stripped full-server binary sizes were:++- current Candle build: 36.06 MiB;+- Candle plus ONNX: 54.81 MiB (**+18.75 MiB / +52.0%**);+- ONNX-only experimental build: 52.49 MiB (**+16.43 MiB / +45.6%**).++The roughly 90 MiB ONNX model is a separate runtime asset comparable to the+existing Candle safetensors asset. The much larger static ONNX Runtime archive+downloaded while compiling is not added wholesale to the final binary.++The bundled `ort` binaries do not cover XERJ's musl release targets. Default+and musl builds therefore remain Candle-only; production adoption requires a+reproducible musl packaging strategy or an explicitly narrower target matrix.++This is deliberately not exposed in `xerj --help`. Server integration still+needs model acquisition, backend fingerprinting/reindex compatibility,+platform packaging, lifecycle metrics, cancellation, and end-to-end retrieval+quality gates. Do not describe XERJ as supporting `--embed-mode onnx`.diff --git a/engine/Cargo.lock b/engine/Cargo.lockindex cf19b3d35..7bab24fa1 100644--- a/engine/Cargo.lock+++ b/engine/Cargo.lock@@ -648,7 +648,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "117725a109d387c937a1533ce01b450cbde6b88abceea8473c4d7a85853cda3c"dependencies = ["lazy_static",- "windows-sys 0.52.0",+ "windows-sys 0.59.0",][[package]]@@ -1655,7 +1655,7 @@ dependencies = ["serde","serde_json","thiserror 2.0.18",- "ureq",+ "ureq 2.12.1","windows-sys 0.60.2",]@@ -1668,6 +1668,12 @@ dependencies = ["digest",]+[[package]]+name = "hmac-sha256"+version = "1.1.14"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "ec9d92d097f4749b64e8cc33d924d9f40a2d4eb91402b458014b781f5733d60f"+[[package]]name = "http"version = "1.4.0"@@ -2177,6 +2183,12 @@ dependencies = ["twox-hash",]+[[package]]+name = "lzma-rust2"+version = "0.15.8"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "e20f57f9918e5bd7bc58c22cdd70a6afc7375d4dd9683af5f2b34bd3d2bba619"+[[package]]name = "macro_rules_attribute"version = "0.2.2"@@ -2208,6 +2220,16 @@ version = "0.7.3"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "0e7465ac9959cc2b1404e8e2367b43684a6d13790fe23056cc8c6c5a6b7bcb94"+[[package]]+name = "matrixmultiply"+version = "0.3.11"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "3f607c237553f086e7043417a51df26b2eb899d3caff94e6a67592ff992fedc7"+dependencies = [+ "autocfg",+ "rawpointer",+]+[[package]]name = "memchr"version = "2.8.0"@@ -2341,6 +2363,21 @@ dependencies = ["tempfile",]+[[package]]+name = "ndarray"+version = "0.17.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "520080814a7a6b4a6e9070823bb24b4531daac8c4627e08ba5de8c5ef2f2752d"+dependencies = [+ "matrixmultiply",+ "num-complex",+ "num-integer",+ "num-traits",+ "portable-atomic",+ "portable-atomic-util",+ "rawpointer",+]+[[package]]name = "nom"version = "7.1.3"@@ -2508,6 +2545,30 @@ version = "0.2.0"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d"+[[package]]+name = "ort"+version = "2.0.0-rc.12"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "d7de3af33d24a745ffb8fab904b13478438d1cd52868e6f17735ef6e1f8bf133"+dependencies = [+ "ndarray",+ "ort-sys",+ "smallvec",+ "tracing",+ "ureq 3.3.0",+]++[[package]]+name = "ort-sys"+version = "2.0.0-rc.12"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "d7b497d21a8b6fbb4b5a544f8fadb77e801a09ae0add9e411d31c6f89e3c1e90"+dependencies = [+ "hmac-sha256",+ "lzma-rust2",+ "ureq 3.3.0",+]+[[package]]name = "parking_lot"version = "0.12.5"@@ -2682,6 +2743,15 @@ version = "1.13.1"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49"+[[package]]+name = "portable-atomic-util"+version = "0.2.7"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618"+dependencies = [+ "portable-atomic",+]+[[package]]name = "potential_utf"version = "0.1.5"@@ -2757,7 +2827,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"dependencies = ["heck",- "itertools 0.10.5",+ "itertools 0.14.0","log","multimap","once_cell",@@ -2777,7 +2847,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d"dependencies = ["anyhow",- "itertools 0.10.5",+ "itertools 0.14.0","proc-macro2","quote","syn",@@ -2922,7 +2992,7 @@ dependencies = ["once_cell","socket2 0.6.3","tracing",- "windows-sys 0.52.0",+ "windows-sys 0.60.2",][[package]]@@ -3024,6 +3094,12 @@ dependencies = ["bitflags",]+[[package]]+name = "rawpointer"+version = "0.2.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "60a357793950651c4ed0f3f52338f53b2f809f32d83a07f72909fa13e4c6c1e3"+[[package]]name = "rayon"version = "1.12.0"@@ -4325,6 +4401,35 @@ dependencies = ["webpki-roots 0.26.11",]+[[package]]+name = "ureq"+version = "3.3.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "dea7109cdcd5864d4eeb1b58a1648dc9bf520360d7af16ec26d0a9354bafcfc0"+dependencies = [+ "base64 0.22.1",+ "log",+ "percent-encoding",+ "rustls",+ "rustls-pki-types",+ "socks",+ "ureq-proto",+ "utf8-zero",+ "webpki-roots 1.0.7",+]++[[package]]+name = "ureq-proto"+version = "0.6.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "e994ba84b0bd1b1b0cf92878b7ef898a5c1760108fe7b6010327e274917a808c"+dependencies = [+ "base64 0.22.1",+ "http",+ "httparse",+ "log",+]+[[package]]name = "url"version = "2.5.8"@@ -4338,6 +4443,12 @@ dependencies = ["serde_derive",]+[[package]]+name = "utf8-zero"+version = "0.8.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "b8c0a043c9540bae7c578c88f91dda8bd82e59ae27c21baca69c8b191aaf5a6e"+[[package]]name = "utf8_iter"version = "1.0.4"@@ -5025,6 +5136,7 @@ dependencies = ["candle-nn","candle-transformers","hf-hub",+ "ort","reqwest","serde","serde_json",diff --git a/engine/crates/xerj-ai/Cargo.toml b/engine/crates/xerj-ai/Cargo.tomlindex 0e7084b0a..63db37e5b 100644--- a/engine/crates/xerj-ai/Cargo.toml+++ b/engine/crates/xerj-ai/Cargo.toml@@ -27,11 +27,23 @@ tokenizers = { version = "0.21", optional = true, default-features = false, feat# `ureq` (sync, blocking) pulls rustls by default — pure-Rust TLS, no OpenSSL,# so the neural feature cross-compiles cleanly to musl/aarch64/windows.hf-hub = { version = "0.4", optional = true, default-features = false, features = ["ureq"] }+# EXPERIMENTAL GNU/Linux ONNX backend. This is deliberately separate from+# `neural`: the production Candle backend and musl builds remain unchanged.+ort = { version = "2.0.0-rc.12", optional = true, default-features = false, features = [+ "std", "ndarray", "tracing", "download-binaries", "tls-rustls",+ "copy-dylibs", "api-24",+] }[features]default = []# Bundles the neural BERT embedder. Adds ~candle + tokenizers to the build.neural = ["dep:candle-core", "dep:candle-nn", "dep:candle-transformers", "dep:tokenizers", "dep:hf-hub"]+# Embedding-layer prototype only; this does not add a server `--embed-mode`.+onnx-experimental = ["dep:ort", "dep:tokenizers"][dev-dependencies]tokio = { version = "1", features = ["full"] }++[[example]]+name = "onnx_experimental"+required-features = ["onnx-experimental"]diff --git a/engine/crates/xerj-ai/examples/onnx_experimental.rs b/engine/crates/xerj-ai/examples/onnx_experimental.rsnew file mode 100644index 000000000..eff87d0b1--- /dev/null+++ b/engine/crates/xerj-ai/examples/onnx_experimental.rs@@ -0,0 +1,55 @@+//! Reproducible smoke/throughput example for the experimental ONNX backend.+//!+//! cargo run --release -p xerj-ai --features onnx-experimental \+//! --example onnx_experimental -- MODEL.onnx tokenizer.json++use anyhow::{Context, Result};+use std::path::PathBuf;+use std::time::Instant;+use xerj_ai::onnx::{MicrobatchConfig, OnnxEmbedder};++fn main() -> Result<()> {+ let args = std::env::args().collect::<Vec<_>>();+ let model = PathBuf::from(args.get(1).context("MODEL.onnx path required")?);+ let tokenizer = PathBuf::from(args.get(2).context("tokenizer.json path required")?);+ let embedder = OnnxEmbedder::load(&model, &tokenizer, 16)?;+ let texts = (0..256)+ .map(|i| {+ let repeats = [1, 1, 1, 6, 6, 18, 40, 80][i % 8];+ format!(+ "{} Record {i}.",+ "Revenue increased while freight expense reduced operating margin. "+ .repeat(repeats)+ )+ })+ .collect::<Vec<_>>();++ let start = Instant::now();+ let singleton = texts+ .iter()+ .map(|text| embedder.embed_blocking(std::slice::from_ref(text)))+ .collect::<Result<Vec<_>>>()?;+ let singleton_seconds = start.elapsed().as_secs_f64();++ let start = Instant::now();+ let scheduled = embedder.embed_scheduled_blocking(&texts, MicrobatchConfig::default())?;+ let scheduled_seconds = start.elapsed().as_secs_f64();+ let min_cosine = singleton+ .iter()+ .zip(&scheduled)+ .map(|(left, right)| left[0].iter().zip(right).map(|(a, b)| a * b).sum::<f32>())+ .fold(1.0_f32, f32::min);++ println!(+ "{}",+ serde_json::to_string_pretty(&serde_json::json!({+ "documents": texts.len(),+ "singleton_documents_per_second": texts.len() as f64 / singleton_seconds,+ "scheduled_documents_per_second": texts.len() as f64 / scheduled_seconds,+ "speedup": singleton_seconds / scheduled_seconds,+ "min_cosine_vs_singleton": min_cosine,+ "output_order_checked": true,+ }))?+ );+ Ok(())+}diff --git a/engine/crates/xerj-ai/src/embedder.rs b/engine/crates/xerj-ai/src/embedder.rsindex 70d9a7331..4f18d1e93 100644--- a/engine/crates/xerj-ai/src/embedder.rs+++ b/engine/crates/xerj-ai/src/embedder.rs@@ -25,6 +25,38 @@ use anyhow::{anyhow, Result};use crate::embed::EmbeddingProxy;use crate::local::{local_embed, DEFAULT_DIMS};+#[cfg(feature = "neural")]+type NeuralCell = tokio::sync::OnceCell<std::sync::Arc<crate::neural::NeuralEmbedder>>;++/// Process-scoped registry of lazily loaded neural models. Every index builds+/// its own [`Embedder`], but indices using the same complete neural+/// configuration must not each load another copy of the ~90 MB model.+///+/// Weak values are intentional: the registry coordinates sharing without+/// extending a model's lifetime after the last index using it is dropped.+#[cfg(feature = "neural")]+fn shared_neural_cell(cfg: &crate::neural::NeuralConfig) -> std::sync::Arc<NeuralCell> {+ use std::collections::HashMap;+ use std::sync::{Mutex, OnceLock, Weak};++ static CELLS: OnceLock<Mutex<HashMap<crate::neural::NeuralConfig, Weak<NeuralCell>>>> =+ OnceLock::new();++ let mut cells = CELLS+ .get_or_init(|| Mutex::new(HashMap::new()))+ .lock()+ .unwrap_or_else(|poisoned| poisoned.into_inner());+ if let Some(cell) = cells.get(cfg).and_then(Weak::upgrade) {+ return cell;+ }++ // Opportunistically discard entries whose final handle has gone away.+ cells.retain(|_, cell| cell.strong_count() > 0);+ let cell = std::sync::Arc::new(NeuralCell::new());+ cells.insert(cfg.clone(), std::sync::Arc::downgrade(&cell));+ cell+}+/// A backend-agnostic text embedder shared across the engine.pub enum Embedder {/// Built-in lexical feature-hash embedder (no model, no network).@@ -91,16 +123,14 @@ impl Embedder {#[cfg(feature = "neural")]pub struct NeuralHandle {cfg: crate::neural::NeuralConfig,- cell: tokio::sync::OnceCell<std::sync::Arc<crate::neural::NeuralEmbedder>>,+ cell: std::sync::Arc<NeuralCell>,}#[cfg(feature = "neural")]impl NeuralHandle {pub fn new(cfg: crate::neural::NeuralConfig) -> Self {- Self {- cfg,- cell: tokio::sync::OnceCell::new(),- }+ let cell = shared_neural_cell(&cfg);+ Self { cfg, cell }}/// Get-or-load the model. First caller pays the (blocking) load / download;@@ -126,3 +156,58 @@ impl NeuralHandle {.map_err(|e| anyhow!("neural embed task panicked: {e}"))?}}++#[cfg(all(test, feature = "neural"))]+mod neural_handle_tests {+ use super::NeuralHandle;+ use crate::neural::NeuralConfig;+ use std::path::PathBuf;+ use std::sync::Arc;++ #[test]+ fn identical_configs_share_lazy_model_cell() {+ let cfg = NeuralConfig {+ model_id: "test/model-shared".into(),+ cache_dir: Some(PathBuf::from("/tmp/xerj-neural-shared-cache")),+ local_dir: None,+ };+ let first = NeuralHandle::new(cfg.clone());+ let second = NeuralHandle::new(cfg);++ assert!(Arc::ptr_eq(&first.cell, &second.cell));+ assert!(first.cell.get().is_none(), "construction must remain lazy");+ }++ #[test]+ fn distinct_configs_do_not_share_lazy_model_cell() {+ let first = NeuralHandle::new(NeuralConfig {+ model_id: "test/model-a".into(),+ cache_dir: None,+ local_dir: None,+ });+ let second = NeuralHandle::new(NeuralConfig {+ model_id: "test/model-b".into(),+ cache_dir: None,+ local_dir: None,+ });++ assert!(!Arc::ptr_eq(&first.cell, &second.cell));+ }++ #[test]+ fn registry_does_not_keep_unused_cells_alive() {+ let cfg = NeuralConfig {+ model_id: "test/model-reclaimable".into(),+ cache_dir: None,+ local_dir: None,+ };+ let weak = {+ let handle = NeuralHandle::new(cfg.clone());+ Arc::downgrade(&handle.cell)+ };+ assert!(weak.upgrade().is_none());++ let replacement = NeuralHandle::new(cfg);+ assert!(replacement.cell.get().is_none());+ }+}diff --git a/engine/crates/xerj-ai/src/lib.rs b/engine/crates/xerj-ai/src/lib.rsindex 415755944..49150bd57 100644--- a/engine/crates/xerj-ai/src/lib.rs+++ b/engine/crates/xerj-ai/src/lib.rs@@ -7,6 +7,8 @@//! - [`embed`] — Embedding proxy: async HTTP client for OpenAI-compatible embedding APIs//! - [`local`] — Built-in zero-config deterministic text embedder (feature hashing)//! - [`neural`] — Built-in neural BERT sentence embedder via candle (feature `neural`)+//! - `onnx` — Experimental FP32 ONNX backend (feature `onnx-experimental`);+//! not wired into the server//! - [`chunker`] — Text chunking with sentence-aware splitting and overlap//! - [`memory`] — Agent memory: semantic dedup + recency-blended recall@@ -17,6 +19,8 @@ pub mod local;pub mod memory;#[cfg(feature = "neural")]pub mod neural;+#[cfg(feature = "onnx-experimental")]+pub mod onnx;pub use chunker::{Chunk, TextChunker};pub use embed::{EmbeddingProxy, EmbeddingProxyConfig};diff --git a/engine/crates/xerj-ai/src/neural.rs b/engine/crates/xerj-ai/src/neural.rsindex 90074e88b..fbd230387 100644--- a/engine/crates/xerj-ai/src/neural.rs+++ b/engine/crates/xerj-ai/src/neural.rs@@ -32,7 +32,7 @@ pub const DEFAULT_MODEL_ID: &str = "sentence-transformers/all-MiniLM-L6-v2";const MAX_TOKENS: usize = 512;/// How to obtain the model weights.-#[derive(Debug, Clone)]+#[derive(Debug, Clone, PartialEq, Eq, Hash)]pub struct NeuralConfig {/// HuggingFace model id (e.g. `sentence-transformers/all-MiniLM-L6-v2`).pub model_id: String,diff --git a/engine/crates/xerj-ai/src/onnx.rs b/engine/crates/xerj-ai/src/onnx.rsnew file mode 100644index 000000000..c721f3dc3--- /dev/null+++ b/engine/crates/xerj-ai/src/onnx.rs@@ -0,0 +1,307 @@+//! Experimental FP32 ONNX Runtime MiniLM backend.+//!+//! This module is an embedding-layer prototype. It is not wired to XERJ's+//! server or CLI. The safe `ort` API requires mutable session access, so one+//! session is serialized behind a mutex and aggregate throughput comes from+//! bounded, length-aware microbatches rather than unsafe concurrent `Run`.++use anyhow::{anyhow, Context, Result};+use ort::session::{builder::GraphOptimizationLevel, Session};+use ort::value::Tensor;+use std::path::Path;+use std::sync::Mutex;+use tokenizers::{PaddingParams, PaddingStrategy, Tokenizer, TruncationParams};++pub const MAX_TOKENS: usize = 512;+pub const DIMS: usize = 384;++/// Bounds for one offline scheduling window.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub struct MicrobatchConfig {+ /// Maximum inputs accepted in one call. Larger calls receive a clear+ /// backpressure error instead of allocating an unbounded queue.+ pub max_pending: usize,+ /// Maximum documents sent to one inference call.+ pub max_batch: usize,+ /// Maximum `batch_size × longest_sequence` token slots.+ pub padded_token_budget: usize,+}++impl Default for MicrobatchConfig {+ fn default() -> Self {+ Self {+ max_pending: 4_096,+ max_batch: 64,+ padded_token_budget: 4_096,+ }+ }+}++impl MicrobatchConfig {+ fn validate(self) -> Result<Self> {+ if self.max_pending == 0 || self.max_batch == 0 || self.padded_token_budget == 0 {+ return Err(anyhow!(+ "ONNX microbatch limits must be non-zero (max_pending={}, max_batch={}, padded_token_budget={})",+ self.max_pending,+ self.max_batch,+ self.padded_token_budget+ ));+ }+ Ok(self)+ }+}++pub struct OnnxEmbedder {+ session: Mutex<Session>,+ tokenizer: Tokenizer,+}++impl OnnxEmbedder {+ pub fn load(model_path: &Path, tokenizer_path: &Path, intra_threads: usize) -> Result<Self> {+ let mut tokenizer = Tokenizer::from_file(tokenizer_path)+ .map_err(|e| anyhow!("load tokenizer {}: {e}", tokenizer_path.display()))?;+ tokenizer.with_padding(Some(PaddingParams {+ strategy: PaddingStrategy::BatchLongest,+ ..Default::default()+ }));+ tokenizer+ .with_truncation(Some(TruncationParams {+ max_length: MAX_TOKENS,+ ..Default::default()+ }))+ .map_err(|e| anyhow!("configure tokenizer truncation: {e}"))?;++ let session = Session::builder()+ .map_err(|e| anyhow!("create ONNX Runtime session builder: {e}"))?+ .with_optimization_level(GraphOptimizationLevel::Level3)+ .map_err(|e| anyhow!("configure ONNX graph optimizations: {e}"))?+ .with_config_entry("session.intra_op.allow_spinning", "0")+ .map_err(|e| anyhow!("disable ONNX intra-op spinning: {e}"))?+ .with_intra_threads(intra_threads.max(1))+ .map_err(|e| anyhow!("configure ONNX intra-op threads: {e}"))?+ .commit_from_file(model_path)+ .with_context(|| format!("load ONNX model {}", model_path.display()))?;+ Ok(Self {+ session: Mutex::new(session),+ tokenizer,+ })+ }++ /// Embed a bounded scheduling window, grouping similar lengths and+ /// restoring vectors to exact input order.+ pub fn embed_scheduled_blocking(+ &self,+ texts: &[String],+ config: MicrobatchConfig,+ ) -> Result<Vec<Vec<f32>>> {+ let config = config.validate()?;+ if texts.len() > config.max_pending {+ return Err(anyhow!(+ "ONNX embedding queue is full: received {} texts, max_pending={}; split the request or retry after draining",+ texts.len(),+ config.max_pending+ ));+ }+ let lengths = texts+ .iter()+ .map(|text| self.token_len(text))+ .collect::<Result<Vec<_>>>()?;+ let batches = plan_microbatches(&lengths, config)?;+ let mut ordered = vec![Vec::new(); texts.len()];+ for batch in batches {+ let input = batch.iter().map(|&i| texts[i].clone()).collect::<Vec<_>>();+ let vectors = self.embed_blocking(&input)?;+ for (position, vector) in batch.into_iter().zip(vectors) {+ ordered[position] = vector;+ }+ }+ Ok(ordered)+ }++ pub fn token_len(&self, text: &str) -> Result<usize> {+ self.tokenizer+ .encode(text, true)+ .map(|encoding| encoding.len())+ .map_err(|e| anyhow!("tokenize for ONNX scheduling: {e}"))+ }++ pub fn embed_blocking(&self, texts: &[String]) -> Result<Vec<Vec<f32>>> {+ if texts.is_empty() {+ return Ok(vec![]);+ }+ let encodings = self+ .tokenizer+ .encode_batch(texts.to_vec(), true)+ .map_err(|e| anyhow!("tokenize batch: {e}"))?;+ let batch = encodings.len();+ let seq_len = encodings.first().map(|e| e.get_ids().len()).unwrap_or(0);+ if seq_len == 0 {+ return Ok(vec![vec![0.0; DIMS]; batch]);+ }++ let mut ids = Vec::with_capacity(batch * seq_len);+ let mut mask = Vec::with_capacity(batch * seq_len);+ for encoding in &encodings {+ ids.extend(encoding.get_ids().iter().map(|&v| i64::from(v)));+ mask.extend(encoding.get_attention_mask().iter().map(|&v| i64::from(v)));+ }+ let token_types = vec![0_i64; ids.len()];+ let shape = [batch, seq_len];+ let input_ids = Tensor::from_array((shape, ids)).context("build input_ids")?;+ let attention_mask =+ Tensor::from_array((shape, mask.clone())).context("build attention_mask")?;+ let token_type_ids =+ Tensor::from_array((shape, token_types)).context("build token_type_ids")?;++ let mut session = self+ .session+ .lock()+ .map_err(|_| anyhow!("ONNX session mutex poisoned"))?;+ let outputs = session+ .run(ort::inputs! {+ "input_ids" => input_ids,+ "attention_mask" => attention_mask,+ "token_type_ids" => token_type_ids,+ })+ .context("ONNX MiniLM inference")?;+ let output = outputs+ .get("last_hidden_state")+ .or_else(|| outputs.get("token_embeddings"))+ .ok_or_else(|| anyhow!("ONNX model did not return token embeddings"))?;+ let (output_shape, hidden) = output+ .try_extract_tensor::<f32>()+ .context("extract ONNX token embeddings")?;+ let shape_dims = output_shape.iter().copied().collect::<Vec<_>>();+ if shape_dims != [batch as i64, seq_len as i64, DIMS as i64] {+ return Err(anyhow!(+ "unexpected ONNX output shape {shape_dims:?}, expected [{batch}, {seq_len}, {DIMS}]"+ ));+ }++ let mut vectors = Vec::with_capacity(batch);+ for row in 0..batch {+ let mut pooled = vec![0.0_f32; DIMS];+ let mut count = 0.0_f32;+ for token in 0..seq_len {+ let weight = mask[row * seq_len + token] as f32;+ count += weight;+ let base = (row * seq_len + token) * DIMS;+ for dim in 0..DIMS {+ pooled[dim] += hidden[base + dim] * weight;+ }+ }+ for value in &mut pooled {+ *value /= count.max(1e-9);+ }+ let norm = pooled+ .iter()+ .map(|value| value * value)+ .sum::<f32>()+ .sqrt()+ .max(1e-12);+ for value in &mut pooled {+ *value /= norm;+ }+ vectors.push(pooled);+ }+ Ok(vectors)+ }+}++/// Plan length-aware microbatches. Indices are sorted for inference efficiency;+/// callers must use them to restore original order.+pub fn plan_microbatches(+ token_lengths: &[usize],+ config: MicrobatchConfig,+) -> Result<Vec<Vec<usize>>> {+ let config = config.validate()?;+ if token_lengths.len() > config.max_pending {+ return Err(anyhow!(+ "ONNX embedding queue is full: received {} texts, max_pending={}",+ token_lengths.len(),+ config.max_pending+ ));+ }+ let mut order = (0..token_lengths.len()).collect::<Vec<_>>();+ order.sort_by_key(|&i| token_lengths[i].min(MAX_TOKENS));+ let mut batches = Vec::new();+ let mut batch = Vec::new();+ let mut longest = 0;+ for i in order {+ let length = token_lengths[i].min(MAX_TOKENS);+ if length > config.padded_token_budget {+ return Err(anyhow!(+ "padded_token_budget={} cannot fit one {}-token input",+ config.padded_token_budget,+ length+ ));+ }+ let next_longest = longest.max(length);+ if !batch.is_empty()+ && (batch.len() == config.max_batch+ || next_longest * (batch.len() + 1) > config.padded_token_budget)+ {+ batches.push(std::mem::take(&mut batch));+ longest = 0;+ }+ longest = longest.max(length);+ batch.push(i);+ }+ if !batch.is_empty() {+ batches.push(batch);+ }+ Ok(batches)+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn planner_bounds_every_batch_and_preserves_all_positions() {+ let lengths = [512, 14, 200, 16, 400, 15, 90, 300];+ let config = MicrobatchConfig {+ max_pending: 8,+ max_batch: 3,+ padded_token_budget: 600,+ };+ let batches = plan_microbatches(&lengths, config).unwrap();+ let mut positions = batches.iter().flatten().copied().collect::<Vec<_>>();+ positions.sort_unstable();+ assert_eq!(positions, (0..lengths.len()).collect::<Vec<_>>());+ for batch in batches {+ assert!(batch.len() <= config.max_batch);+ let longest = batch.iter().map(|&i| lengths[i]).max().unwrap();+ assert!(longest * batch.len() <= config.padded_token_budget);+ }+ }++ #[test]+ fn planner_applies_backpressure() {+ let error = plan_microbatches(+ &[10, 20, 30],+ MicrobatchConfig {+ max_pending: 2,+ ..MicrobatchConfig::default()+ },+ )+ .unwrap_err()+ .to_string();+ assert!(error.contains("queue is full"));+ assert!(error.contains("max_pending=2"));+ }++ #[test]+ fn planner_rejects_impossible_budget() {+ let error = plan_microbatches(+ &[512],+ MicrobatchConfig {+ padded_token_budget: 128,+ ..MicrobatchConfig::default()+ },+ )+ .unwrap_err()+ .to_string();+ assert!(error.contains("cannot fit"));+ }+}