Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(ai): prototype bounded ONNX embeddings

xerj-org82c472fc3da4
diff --git a/docs/EXPERIMENTAL_ONNX.md b/docs/EXPERIMENTAL_ONNX.md
new file mode 100644
index 000000000..cd0203894
--- /dev/null
+++ b/docs/EXPERIMENTAL_ONNX.md
@@ -0,0 +1,63 @@
+# Experimental ONNX embedding backend
+
+Status: embedding-library prototype, **not a server or CLI mode**.
+
+XERJ's supported built-in neural backend remains Candle and
+`--embed-mode neural` continues to mean Candle. Default and musl builds do not
+compile ONNX Runtime.
+
+The opt-in `xerj-ai` feature provides the exact FP32 MiniLM inference path and
+a bounded length-aware microbatch API:
+
+```bash
+cd engine
+cargo run --release -p xerj-ai --features onnx-experimental \
+ --example onnx_experimental -- /path/model.onnx /path/tokenizer.json
+```
+
+The caller supplies an ONNX model with `input_ids`, `attention_mask`, and
+`token_type_ids` inputs and a `[batch, sequence, 384]` `last_hidden_state`
+output. The current verified artifact is the official FP32
+`sentence-transformers/all-MiniLM-L6-v2` ONNX export.
+
+Safety properties:
+
+- safe `ort` API only;
+- one session behind a mutex;
+- bounded pending inputs, documents per batch, and padded token slots;
+- length-aware grouping;
+- exact restoration to caller input order;
+- clear backpressure and invalid-budget errors.
+
+## Measured prototype results
+
+On the controlled 128-document mixed-length benchmark, exact FP32 ONNX plus
+the retained scheduler processed 116.671 documents/s versus 9.045 documents/s
+for Candle using the identical precomputed length-aware batch plan: **12.90x
+at the `xerj-ai` embedding layer**. The median minimum same-document
+Candle/ONNX cosine was `0.9999991655`.
+
+This is not a claim of 12.90x end-to-end XERJ indexing. The benchmark excludes
+document extraction, HTTP handling, persistence, and HNSW construction. ONNX
+also used 16 intra-op threads; the measured CPU-efficiency improvement was
+2.92x, smaller than the wall-time result. A server-level comparison is required
+before publishing an indexing-throughput claim.
+
+Measured stripped full-server binary sizes were:
+
+- current Candle build: 36.06 MiB;
+- Candle plus ONNX: 54.81 MiB (**+18.75 MiB / +52.0%**);
+- ONNX-only experimental build: 52.49 MiB (**+16.43 MiB / +45.6%**).
+
+The roughly 90 MiB ONNX model is a separate runtime asset comparable to the
+existing Candle safetensors asset. The much larger static ONNX Runtime archive
+downloaded while compiling is not added wholesale to the final binary.
+
+The bundled `ort` binaries do not cover XERJ's musl release targets. Default
+and musl builds therefore remain Candle-only; production adoption requires a
+reproducible musl packaging strategy or an explicitly narrower target matrix.
+
+This is deliberately not exposed in `xerj --help`. Server integration still
+needs model acquisition, backend fingerprinting/reindex compatibility,
+platform packaging, lifecycle metrics, cancellation, and end-to-end retrieval
+quality gates. Do not describe XERJ as supporting `--embed-mode onnx`.
diff --git a/engine/Cargo.lock b/engine/Cargo.lock
index cf19b3d35..7bab24fa1 100644
--- a/engine/Cargo.lock
+++ b/engine/Cargo.lock
@@ -648,7 +648,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "117725a109d387c937a1533ce01b450cbde6b88abceea8473c4d7a85853cda3c"
dependencies = [
"lazy_static",
- "windows-sys 0.52.0",
+ "windows-sys 0.59.0",
]
[[package]]
@@ -1655,7 +1655,7 @@ dependencies = [
"serde",
"serde_json",
"thiserror 2.0.18",
- "ureq",
+ "ureq 2.12.1",
"windows-sys 0.60.2",
]
@@ -1668,6 +1668,12 @@ dependencies = [
"digest",
]
+[[package]]
+name = "hmac-sha256"
+version = "1.1.14"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ec9d92d097f4749b64e8cc33d924d9f40a2d4eb91402b458014b781f5733d60f"
+
[[package]]
name = "http"
version = "1.4.0"
@@ -2177,6 +2183,12 @@ dependencies = [
"twox-hash",
]
+[[package]]
+name = "lzma-rust2"
+version = "0.15.8"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e20f57f9918e5bd7bc58c22cdd70a6afc7375d4dd9683af5f2b34bd3d2bba619"
+
[[package]]
name = "macro_rules_attribute"
version = "0.2.2"
@@ -2208,6 +2220,16 @@ version = "0.7.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0e7465ac9959cc2b1404e8e2367b43684a6d13790fe23056cc8c6c5a6b7bcb94"
+[[package]]
+name = "matrixmultiply"
+version = "0.3.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3f607c237553f086e7043417a51df26b2eb899d3caff94e6a67592ff992fedc7"
+dependencies = [
+ "autocfg",
+ "rawpointer",
+]
+
[[package]]
name = "memchr"
version = "2.8.0"
@@ -2341,6 +2363,21 @@ dependencies = [
"tempfile",
]
+[[package]]
+name = "ndarray"
+version = "0.17.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "520080814a7a6b4a6e9070823bb24b4531daac8c4627e08ba5de8c5ef2f2752d"
+dependencies = [
+ "matrixmultiply",
+ "num-complex",
+ "num-integer",
+ "num-traits",
+ "portable-atomic",
+ "portable-atomic-util",
+ "rawpointer",
+]
+
[[package]]
name = "nom"
version = "7.1.3"
@@ -2508,6 +2545,30 @@ version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d"
+[[package]]
+name = "ort"
+version = "2.0.0-rc.12"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d7de3af33d24a745ffb8fab904b13478438d1cd52868e6f17735ef6e1f8bf133"
+dependencies = [
+ "ndarray",
+ "ort-sys",
+ "smallvec",
+ "tracing",
+ "ureq 3.3.0",
+]
+
+[[package]]
+name = "ort-sys"
+version = "2.0.0-rc.12"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d7b497d21a8b6fbb4b5a544f8fadb77e801a09ae0add9e411d31c6f89e3c1e90"
+dependencies = [
+ "hmac-sha256",
+ "lzma-rust2",
+ "ureq 3.3.0",
+]
+
[[package]]
name = "parking_lot"
version = "0.12.5"
@@ -2682,6 +2743,15 @@ version = "1.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49"
+[[package]]
+name = "portable-atomic-util"
+version = "0.2.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618"
+dependencies = [
+ "portable-atomic",
+]
+
[[package]]
name = "potential_utf"
version = "0.1.5"
@@ -2757,7 +2827,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"
dependencies = [
"heck",
- "itertools 0.10.5",
+ "itertools 0.14.0",
"log",
"multimap",
"once_cell",
@@ -2777,7 +2847,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d"
dependencies = [
"anyhow",
- "itertools 0.10.5",
+ "itertools 0.14.0",
"proc-macro2",
"quote",
"syn",
@@ -2922,7 +2992,7 @@ dependencies = [
"once_cell",
"socket2 0.6.3",
"tracing",
- "windows-sys 0.52.0",
+ "windows-sys 0.60.2",
]
[[package]]
@@ -3024,6 +3094,12 @@ dependencies = [
"bitflags",
]
+[[package]]
+name = "rawpointer"
+version = "0.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "60a357793950651c4ed0f3f52338f53b2f809f32d83a07f72909fa13e4c6c1e3"
+
[[package]]
name = "rayon"
version = "1.12.0"
@@ -4325,6 +4401,35 @@ dependencies = [
"webpki-roots 0.26.11",
]
+[[package]]
+name = "ureq"
+version = "3.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "dea7109cdcd5864d4eeb1b58a1648dc9bf520360d7af16ec26d0a9354bafcfc0"
+dependencies = [
+ "base64 0.22.1",
+ "log",
+ "percent-encoding",
+ "rustls",
+ "rustls-pki-types",
+ "socks",
+ "ureq-proto",
+ "utf8-zero",
+ "webpki-roots 1.0.7",
+]
+
+[[package]]
+name = "ureq-proto"
+version = "0.6.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e994ba84b0bd1b1b0cf92878b7ef898a5c1760108fe7b6010327e274917a808c"
+dependencies = [
+ "base64 0.22.1",
+ "http",
+ "httparse",
+ "log",
+]
+
[[package]]
name = "url"
version = "2.5.8"
@@ -4338,6 +4443,12 @@ dependencies = [
"serde_derive",
]
+[[package]]
+name = "utf8-zero"
+version = "0.8.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b8c0a043c9540bae7c578c88f91dda8bd82e59ae27c21baca69c8b191aaf5a6e"
+
[[package]]
name = "utf8_iter"
version = "1.0.4"
@@ -5025,6 +5136,7 @@ dependencies = [
"candle-nn",
"candle-transformers",
"hf-hub",
+ "ort",
"reqwest",
"serde",
"serde_json",
diff --git a/engine/crates/xerj-ai/Cargo.toml b/engine/crates/xerj-ai/Cargo.toml
index 0e7084b0a..63db37e5b 100644
--- a/engine/crates/xerj-ai/Cargo.toml
+++ b/engine/crates/xerj-ai/Cargo.toml
@@ -27,11 +27,23 @@ tokenizers = { version = "0.21", optional = true, default-features = false, feat
# `ureq` (sync, blocking) pulls rustls by default — pure-Rust TLS, no OpenSSL,
# so the neural feature cross-compiles cleanly to musl/aarch64/windows.
hf-hub = { version = "0.4", optional = true, default-features = false, features = ["ureq"] }
+# EXPERIMENTAL GNU/Linux ONNX backend. This is deliberately separate from
+# `neural`: the production Candle backend and musl builds remain unchanged.
+ort = { version = "2.0.0-rc.12", optional = true, default-features = false, features = [
+ "std", "ndarray", "tracing", "download-binaries", "tls-rustls",
+ "copy-dylibs", "api-24",
+] }
[features]
default = []
# Bundles the neural BERT embedder. Adds ~candle + tokenizers to the build.
neural = ["dep:candle-core", "dep:candle-nn", "dep:candle-transformers", "dep:tokenizers", "dep:hf-hub"]
+# Embedding-layer prototype only; this does not add a server `--embed-mode`.
+onnx-experimental = ["dep:ort", "dep:tokenizers"]
[dev-dependencies]
tokio = { version = "1", features = ["full"] }
+
+[[example]]
+name = "onnx_experimental"
+required-features = ["onnx-experimental"]
diff --git a/engine/crates/xerj-ai/examples/onnx_experimental.rs b/engine/crates/xerj-ai/examples/onnx_experimental.rs
new file mode 100644
index 000000000..eff87d0b1
--- /dev/null
+++ b/engine/crates/xerj-ai/examples/onnx_experimental.rs
@@ -0,0 +1,55 @@
+//! Reproducible smoke/throughput example for the experimental ONNX backend.
+//!
+//! cargo run --release -p xerj-ai --features onnx-experimental \
+//! --example onnx_experimental -- MODEL.onnx tokenizer.json
+
+use anyhow::{Context, Result};
+use std::path::PathBuf;
+use std::time::Instant;
+use xerj_ai::onnx::{MicrobatchConfig, OnnxEmbedder};
+
+fn main() -> Result<()> {
+ let args = std::env::args().collect::<Vec<_>>();
+ let model = PathBuf::from(args.get(1).context("MODEL.onnx path required")?);
+ let tokenizer = PathBuf::from(args.get(2).context("tokenizer.json path required")?);
+ let embedder = OnnxEmbedder::load(&model, &tokenizer, 16)?;
+ let texts = (0..256)
+ .map(|i| {
+ let repeats = [1, 1, 1, 6, 6, 18, 40, 80][i % 8];
+ format!(
+ "{} Record {i}.",
+ "Revenue increased while freight expense reduced operating margin. "
+ .repeat(repeats)
+ )
+ })
+ .collect::<Vec<_>>();
+
+ let start = Instant::now();
+ let singleton = texts
+ .iter()
+ .map(|text| embedder.embed_blocking(std::slice::from_ref(text)))
+ .collect::<Result<Vec<_>>>()?;
+ let singleton_seconds = start.elapsed().as_secs_f64();
+
+ let start = Instant::now();
+ let scheduled = embedder.embed_scheduled_blocking(&texts, MicrobatchConfig::default())?;
+ let scheduled_seconds = start.elapsed().as_secs_f64();
+ let min_cosine = singleton
+ .iter()
+ .zip(&scheduled)
+ .map(|(left, right)| left[0].iter().zip(right).map(|(a, b)| a * b).sum::<f32>())
+ .fold(1.0_f32, f32::min);
+
+ println!(
+ "{}",
+ serde_json::to_string_pretty(&serde_json::json!({
+ "documents": texts.len(),
+ "singleton_documents_per_second": texts.len() as f64 / singleton_seconds,
+ "scheduled_documents_per_second": texts.len() as f64 / scheduled_seconds,
+ "speedup": singleton_seconds / scheduled_seconds,
+ "min_cosine_vs_singleton": min_cosine,
+ "output_order_checked": true,
+ }))?
+ );
+ Ok(())
+}
diff --git a/engine/crates/xerj-ai/src/embedder.rs b/engine/crates/xerj-ai/src/embedder.rs
index 70d9a7331..4f18d1e93 100644
--- a/engine/crates/xerj-ai/src/embedder.rs
+++ b/engine/crates/xerj-ai/src/embedder.rs
@@ -25,6 +25,38 @@ use anyhow::{anyhow, Result};
use crate::embed::EmbeddingProxy;
use crate::local::{local_embed, DEFAULT_DIMS};
+#[cfg(feature = "neural")]
+type NeuralCell = tokio::sync::OnceCell<std::sync::Arc<crate::neural::NeuralEmbedder>>;
+
+/// Process-scoped registry of lazily loaded neural models. Every index builds
+/// its own [`Embedder`], but indices using the same complete neural
+/// configuration must not each load another copy of the ~90 MB model.
+///
+/// Weak values are intentional: the registry coordinates sharing without
+/// extending a model's lifetime after the last index using it is dropped.
+#[cfg(feature = "neural")]
+fn shared_neural_cell(cfg: &crate::neural::NeuralConfig) -> std::sync::Arc<NeuralCell> {
+ use std::collections::HashMap;
+ use std::sync::{Mutex, OnceLock, Weak};
+
+ static CELLS: OnceLock<Mutex<HashMap<crate::neural::NeuralConfig, Weak<NeuralCell>>>> =
+ OnceLock::new();
+
+ let mut cells = CELLS
+ .get_or_init(|| Mutex::new(HashMap::new()))
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner());
+ if let Some(cell) = cells.get(cfg).and_then(Weak::upgrade) {
+ return cell;
+ }
+
+ // Opportunistically discard entries whose final handle has gone away.
+ cells.retain(|_, cell| cell.strong_count() > 0);
+ let cell = std::sync::Arc::new(NeuralCell::new());
+ cells.insert(cfg.clone(), std::sync::Arc::downgrade(&cell));
+ cell
+}
+
/// A backend-agnostic text embedder shared across the engine.
pub enum Embedder {
/// Built-in lexical feature-hash embedder (no model, no network).
@@ -91,16 +123,14 @@ impl Embedder {
#[cfg(feature = "neural")]
pub struct NeuralHandle {
cfg: crate::neural::NeuralConfig,
- cell: tokio::sync::OnceCell<std::sync::Arc<crate::neural::NeuralEmbedder>>,
+ cell: std::sync::Arc<NeuralCell>,
}
#[cfg(feature = "neural")]
impl NeuralHandle {
pub fn new(cfg: crate::neural::NeuralConfig) -> Self {
- Self {
- cfg,
- cell: tokio::sync::OnceCell::new(),
- }
+ let cell = shared_neural_cell(&cfg);
+ Self { cfg, cell }
}
/// Get-or-load the model. First caller pays the (blocking) load / download;
@@ -126,3 +156,58 @@ impl NeuralHandle {
.map_err(|e| anyhow!("neural embed task panicked: {e}"))?
}
}
+
+#[cfg(all(test, feature = "neural"))]
+mod neural_handle_tests {
+ use super::NeuralHandle;
+ use crate::neural::NeuralConfig;
+ use std::path::PathBuf;
+ use std::sync::Arc;
+
+ #[test]
+ fn identical_configs_share_lazy_model_cell() {
+ let cfg = NeuralConfig {
+ model_id: "test/model-shared".into(),
+ cache_dir: Some(PathBuf::from("/tmp/xerj-neural-shared-cache")),
+ local_dir: None,
+ };
+ let first = NeuralHandle::new(cfg.clone());
+ let second = NeuralHandle::new(cfg);
+
+ assert!(Arc::ptr_eq(&first.cell, &second.cell));
+ assert!(first.cell.get().is_none(), "construction must remain lazy");
+ }
+
+ #[test]
+ fn distinct_configs_do_not_share_lazy_model_cell() {
+ let first = NeuralHandle::new(NeuralConfig {
+ model_id: "test/model-a".into(),
+ cache_dir: None,
+ local_dir: None,
+ });
+ let second = NeuralHandle::new(NeuralConfig {
+ model_id: "test/model-b".into(),
+ cache_dir: None,
+ local_dir: None,
+ });
+
+ assert!(!Arc::ptr_eq(&first.cell, &second.cell));
+ }
+
+ #[test]
+ fn registry_does_not_keep_unused_cells_alive() {
+ let cfg = NeuralConfig {
+ model_id: "test/model-reclaimable".into(),
+ cache_dir: None,
+ local_dir: None,
+ };
+ let weak = {
+ let handle = NeuralHandle::new(cfg.clone());
+ Arc::downgrade(&handle.cell)
+ };
+ assert!(weak.upgrade().is_none());
+
+ let replacement = NeuralHandle::new(cfg);
+ assert!(replacement.cell.get().is_none());
+ }
+}
diff --git a/engine/crates/xerj-ai/src/lib.rs b/engine/crates/xerj-ai/src/lib.rs
index 415755944..49150bd57 100644
--- a/engine/crates/xerj-ai/src/lib.rs
+++ b/engine/crates/xerj-ai/src/lib.rs
@@ -7,6 +7,8 @@
//! - [`embed`] — Embedding proxy: async HTTP client for OpenAI-compatible embedding APIs
//! - [`local`] — Built-in zero-config deterministic text embedder (feature hashing)
//! - [`neural`] — Built-in neural BERT sentence embedder via candle (feature `neural`)
+//! - `onnx` — Experimental FP32 ONNX backend (feature `onnx-experimental`);
+//! not wired into the server
//! - [`chunker`] — Text chunking with sentence-aware splitting and overlap
//! - [`memory`] — Agent memory: semantic dedup + recency-blended recall
@@ -17,6 +19,8 @@ pub mod local;
pub mod memory;
#[cfg(feature = "neural")]
pub mod neural;
+#[cfg(feature = "onnx-experimental")]
+pub mod onnx;
pub use chunker::{Chunk, TextChunker};
pub use embed::{EmbeddingProxy, EmbeddingProxyConfig};
diff --git a/engine/crates/xerj-ai/src/neural.rs b/engine/crates/xerj-ai/src/neural.rs
index 90074e88b..fbd230387 100644
--- a/engine/crates/xerj-ai/src/neural.rs
+++ b/engine/crates/xerj-ai/src/neural.rs
@@ -32,7 +32,7 @@ pub const DEFAULT_MODEL_ID: &str = "sentence-transformers/all-MiniLM-L6-v2";
const MAX_TOKENS: usize = 512;
/// How to obtain the model weights.
-#[derive(Debug, Clone)]
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct NeuralConfig {
/// HuggingFace model id (e.g. `sentence-transformers/all-MiniLM-L6-v2`).
pub model_id: String,
diff --git a/engine/crates/xerj-ai/src/onnx.rs b/engine/crates/xerj-ai/src/onnx.rs
new file mode 100644
index 000000000..c721f3dc3
--- /dev/null
+++ b/engine/crates/xerj-ai/src/onnx.rs
@@ -0,0 +1,307 @@
+//! Experimental FP32 ONNX Runtime MiniLM backend.
+//!
+//! This module is an embedding-layer prototype. It is not wired to XERJ's
+//! server or CLI. The safe `ort` API requires mutable session access, so one
+//! session is serialized behind a mutex and aggregate throughput comes from
+//! bounded, length-aware microbatches rather than unsafe concurrent `Run`.
+
+use anyhow::{anyhow, Context, Result};
+use ort::session::{builder::GraphOptimizationLevel, Session};
+use ort::value::Tensor;
+use std::path::Path;
+use std::sync::Mutex;
+use tokenizers::{PaddingParams, PaddingStrategy, Tokenizer, TruncationParams};
+
+pub const MAX_TOKENS: usize = 512;
+pub const DIMS: usize = 384;
+
+/// Bounds for one offline scheduling window.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub struct MicrobatchConfig {
+ /// Maximum inputs accepted in one call. Larger calls receive a clear
+ /// backpressure error instead of allocating an unbounded queue.
+ pub max_pending: usize,
+ /// Maximum documents sent to one inference call.
+ pub max_batch: usize,
+ /// Maximum `batch_size × longest_sequence` token slots.
+ pub padded_token_budget: usize,
+}
+
+impl Default for MicrobatchConfig {
+ fn default() -> Self {
+ Self {
+ max_pending: 4_096,
+ max_batch: 64,
+ padded_token_budget: 4_096,
+ }
+ }
+}
+
+impl MicrobatchConfig {
+ fn validate(self) -> Result<Self> {
+ if self.max_pending == 0 || self.max_batch == 0 || self.padded_token_budget == 0 {
+ return Err(anyhow!(
+ "ONNX microbatch limits must be non-zero (max_pending={}, max_batch={}, padded_token_budget={})",
+ self.max_pending,
+ self.max_batch,
+ self.padded_token_budget
+ ));
+ }
+ Ok(self)
+ }
+}
+
+pub struct OnnxEmbedder {
+ session: Mutex<Session>,
+ tokenizer: Tokenizer,
+}
+
+impl OnnxEmbedder {
+ pub fn load(model_path: &Path, tokenizer_path: &Path, intra_threads: usize) -> Result<Self> {
+ let mut tokenizer = Tokenizer::from_file(tokenizer_path)
+ .map_err(|e| anyhow!("load tokenizer {}: {e}", tokenizer_path.display()))?;
+ tokenizer.with_padding(Some(PaddingParams {
+ strategy: PaddingStrategy::BatchLongest,
+ ..Default::default()
+ }));
+ tokenizer
+ .with_truncation(Some(TruncationParams {
+ max_length: MAX_TOKENS,
+ ..Default::default()
+ }))
+ .map_err(|e| anyhow!("configure tokenizer truncation: {e}"))?;
+
+ let session = Session::builder()
+ .map_err(|e| anyhow!("create ONNX Runtime session builder: {e}"))?
+ .with_optimization_level(GraphOptimizationLevel::Level3)
+ .map_err(|e| anyhow!("configure ONNX graph optimizations: {e}"))?
+ .with_config_entry("session.intra_op.allow_spinning", "0")
+ .map_err(|e| anyhow!("disable ONNX intra-op spinning: {e}"))?
+ .with_intra_threads(intra_threads.max(1))
+ .map_err(|e| anyhow!("configure ONNX intra-op threads: {e}"))?
+ .commit_from_file(model_path)
+ .with_context(|| format!("load ONNX model {}", model_path.display()))?;
+ Ok(Self {
+ session: Mutex::new(session),
+ tokenizer,
+ })
+ }
+
+ /// Embed a bounded scheduling window, grouping similar lengths and
+ /// restoring vectors to exact input order.
+ pub fn embed_scheduled_blocking(
+ &self,
+ texts: &[String],
+ config: MicrobatchConfig,
+ ) -> Result<Vec<Vec<f32>>> {
+ let config = config.validate()?;
+ if texts.len() > config.max_pending {
+ return Err(anyhow!(
+ "ONNX embedding queue is full: received {} texts, max_pending={}; split the request or retry after draining",
+ texts.len(),
+ config.max_pending
+ ));
+ }
+ let lengths = texts
+ .iter()
+ .map(|text| self.token_len(text))
+ .collect::<Result<Vec<_>>>()?;
+ let batches = plan_microbatches(&lengths, config)?;
+ let mut ordered = vec![Vec::new(); texts.len()];
+ for batch in batches {
+ let input = batch.iter().map(|&i| texts[i].clone()).collect::<Vec<_>>();
+ let vectors = self.embed_blocking(&input)?;
+ for (position, vector) in batch.into_iter().zip(vectors) {
+ ordered[position] = vector;
+ }
+ }
+ Ok(ordered)
+ }
+
+ pub fn token_len(&self, text: &str) -> Result<usize> {
+ self.tokenizer
+ .encode(text, true)
+ .map(|encoding| encoding.len())
+ .map_err(|e| anyhow!("tokenize for ONNX scheduling: {e}"))
+ }
+
+ pub fn embed_blocking(&self, texts: &[String]) -> Result<Vec<Vec<f32>>> {
+ if texts.is_empty() {
+ return Ok(vec![]);
+ }
+ let encodings = self
+ .tokenizer
+ .encode_batch(texts.to_vec(), true)
+ .map_err(|e| anyhow!("tokenize batch: {e}"))?;
+ let batch = encodings.len();
+ let seq_len = encodings.first().map(|e| e.get_ids().len()).unwrap_or(0);
+ if seq_len == 0 {
+ return Ok(vec![vec![0.0; DIMS]; batch]);
+ }
+
+ let mut ids = Vec::with_capacity(batch * seq_len);
+ let mut mask = Vec::with_capacity(batch * seq_len);
+ for encoding in &encodings {
+ ids.extend(encoding.get_ids().iter().map(|&v| i64::from(v)));
+ mask.extend(encoding.get_attention_mask().iter().map(|&v| i64::from(v)));
+ }
+ let token_types = vec![0_i64; ids.len()];
+ let shape = [batch, seq_len];
+ let input_ids = Tensor::from_array((shape, ids)).context("build input_ids")?;
+ let attention_mask =
+ Tensor::from_array((shape, mask.clone())).context("build attention_mask")?;
+ let token_type_ids =
+ Tensor::from_array((shape, token_types)).context("build token_type_ids")?;
+
+ let mut session = self
+ .session
+ .lock()
+ .map_err(|_| anyhow!("ONNX session mutex poisoned"))?;
+ let outputs = session
+ .run(ort::inputs! {
+ "input_ids" => input_ids,
+ "attention_mask" => attention_mask,
+ "token_type_ids" => token_type_ids,
+ })
+ .context("ONNX MiniLM inference")?;
+ let output = outputs
+ .get("last_hidden_state")
+ .or_else(|| outputs.get("token_embeddings"))
+ .ok_or_else(|| anyhow!("ONNX model did not return token embeddings"))?;
+ let (output_shape, hidden) = output
+ .try_extract_tensor::<f32>()
+ .context("extract ONNX token embeddings")?;
+ let shape_dims = output_shape.iter().copied().collect::<Vec<_>>();
+ if shape_dims != [batch as i64, seq_len as i64, DIMS as i64] {
+ return Err(anyhow!(
+ "unexpected ONNX output shape {shape_dims:?}, expected [{batch}, {seq_len}, {DIMS}]"
+ ));
+ }
+
+ let mut vectors = Vec::with_capacity(batch);
+ for row in 0..batch {
+ let mut pooled = vec![0.0_f32; DIMS];
+ let mut count = 0.0_f32;
+ for token in 0..seq_len {
+ let weight = mask[row * seq_len + token] as f32;
+ count += weight;
+ let base = (row * seq_len + token) * DIMS;
+ for dim in 0..DIMS {
+ pooled[dim] += hidden[base + dim] * weight;
+ }
+ }
+ for value in &mut pooled {
+ *value /= count.max(1e-9);
+ }
+ let norm = pooled
+ .iter()
+ .map(|value| value * value)
+ .sum::<f32>()
+ .sqrt()
+ .max(1e-12);
+ for value in &mut pooled {
+ *value /= norm;
+ }
+ vectors.push(pooled);
+ }
+ Ok(vectors)
+ }
+}
+
+/// Plan length-aware microbatches. Indices are sorted for inference efficiency;
+/// callers must use them to restore original order.
+pub fn plan_microbatches(
+ token_lengths: &[usize],
+ config: MicrobatchConfig,
+) -> Result<Vec<Vec<usize>>> {
+ let config = config.validate()?;
+ if token_lengths.len() > config.max_pending {
+ return Err(anyhow!(
+ "ONNX embedding queue is full: received {} texts, max_pending={}",
+ token_lengths.len(),
+ config.max_pending
+ ));
+ }
+ let mut order = (0..token_lengths.len()).collect::<Vec<_>>();
+ order.sort_by_key(|&i| token_lengths[i].min(MAX_TOKENS));
+ let mut batches = Vec::new();
+ let mut batch = Vec::new();
+ let mut longest = 0;
+ for i in order {
+ let length = token_lengths[i].min(MAX_TOKENS);
+ if length > config.padded_token_budget {
+ return Err(anyhow!(
+ "padded_token_budget={} cannot fit one {}-token input",
+ config.padded_token_budget,
+ length
+ ));
+ }
+ let next_longest = longest.max(length);
+ if !batch.is_empty()
+ && (batch.len() == config.max_batch
+ || next_longest * (batch.len() + 1) > config.padded_token_budget)
+ {
+ batches.push(std::mem::take(&mut batch));
+ longest = 0;
+ }
+ longest = longest.max(length);
+ batch.push(i);
+ }
+ if !batch.is_empty() {
+ batches.push(batch);
+ }
+ Ok(batches)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn planner_bounds_every_batch_and_preserves_all_positions() {
+ let lengths = [512, 14, 200, 16, 400, 15, 90, 300];
+ let config = MicrobatchConfig {
+ max_pending: 8,
+ max_batch: 3,
+ padded_token_budget: 600,
+ };
+ let batches = plan_microbatches(&lengths, config).unwrap();
+ let mut positions = batches.iter().flatten().copied().collect::<Vec<_>>();
+ positions.sort_unstable();
+ assert_eq!(positions, (0..lengths.len()).collect::<Vec<_>>());
+ for batch in batches {
+ assert!(batch.len() <= config.max_batch);
+ let longest = batch.iter().map(|&i| lengths[i]).max().unwrap();
+ assert!(longest * batch.len() <= config.padded_token_budget);
+ }
+ }
+
+ #[test]
+ fn planner_applies_backpressure() {
+ let error = plan_microbatches(
+ &[10, 20, 30],
+ MicrobatchConfig {
+ max_pending: 2,
+ ..MicrobatchConfig::default()
+ },
+ )
+ .unwrap_err()
+ .to_string();
+ assert!(error.contains("queue is full"));
+ assert!(error.contains("max_pending=2"));
+ }
+
+ #[test]
+ fn planner_rejects_impossible_budget() {
+ let error = plan_microbatches(
+ &[512],
+ MicrobatchConfig {
+ padded_token_budget: 128,
+ ..MicrobatchConfig::default()
+ },
+ )
+ .unwrap_err()
+ .to_string();
+ assert!(error.contains("cannot fit"));
+ }
+}