Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
laya (Rust)
Rust port of the Laya Python SDK (the laya package on PyPI, from this
repository): typed-decision inference over a local ONNX model. It reproduces the Python inference
path (the same tokenization, sequence building, calibration and answers), verified against golden
outputs recorded from the Python code (tests/golden), but runs on
ONNX Runtime instead of PyTorch, so no Python process runs at runtime.
You ask typed questions (choice, score, noul) about a piece of state, and they are answered
in one non-autoregressive forward pass with calibrated probabilities. Nothing is generated, so
there is nothing to parse and nothing to hallucinate.
Mirrors the Laya Python SDK module for module — see the "Module map" below for the correspondence.
| Crate | laya-onnx 0.1.0 (imported as laya), versioned separately from the Python package |
| Checkpoints | multilingual (mmBERT-base, 322M, 1024-token, 100+ languages, default), english (ModernBERT-large, 421M, 512-token, English-optimized), typed-decisions (ModernBERT-large, 421M, 1024-token, fine-tuned on four typed-decisions workflows) |
| Model format | ONNX (opset 18, weights in an external-data sidecar), exported from the official PyTorch checkpoints with tools/export_onnx.py. See MODELS.md. |
| Runtime | ONNX Runtime through ort 2.x (CPU by default; cuda / directml cargo features). No Python process runs at runtime. |
| Tokenizer | tokenizers — the same HuggingFace Rust tokenizers library the Python SDK uses |
| Download | opt-in, via ureq (blocking, rustls, feature download, on by default) |
Contents
- Installation
- Getting the model
- Quickstart
- Choosing a checkpoint
- Writing questions
- Built-in presets
- Reading results
- Long prediction
- Batch prediction
- Per-language temperatures and per-call overrides
- Abstention (
min_confidence) - Thread safety
- Execution providers and cargo features
- Async usage
- Configuration reference
- Routing, language detection, email and shortlist
- Samples
- Testing
- Module map
- Differences from the Python SDK
- Known gaps
Installation
The crate is published on crates.io as laya-onnx:
cargo add laya-onnx
or in Cargo.toml:
[dependencies]
laya-onnx = "0.1.0"
# or, straight from the repository (cargo finds the `laya-onnx` package in laya-rust/):
laya-onnx = { git = "https://github.com/NandhaKishorM/laya" }
The package is named laya-onnx, but the library is imported as laya: everything lives at
the crate root (use laya::...), and tokenizer types are under laya::tokenization.
Getting the model
This crate runs ONNX models (opset 18) and auto-detects which of two export layouts a checkpoint directory holds:
- Fused — a single
model.onnxgraph with its weights in amodel.onnx.datasidecar beside it, plusrl_agent_config.jsonand a nestedtokenizer/directory (tokenizer/tokenizer.json+tokenizer/tokenizer_config.json). This is whattools/export_onnx.pyproduces. - Split —
encoder.onnx(text in, hidden states out) andhead.onnx(hidden states + marker metadata in, logits out) as two separate graphs, plusrl_agent_config.jsonand a root-leveltokenizer.json(no siblingtokenizer_config.json). This is the layout laya-ts's exporter produces.
No Python is needed at runtime either way; Python (or, for the split layout, laya-ts) is used once,
to export the model from the published PyTorch checkpoint. Point the engine at a checkpoint
directory with LayaEngine::from_directory(...), LayaOptions::model_directory or
LAYA_ONNX_ROOT, as in the Quickstart — the layout present on disk is detected
automatically (fused wins if, unusually, a directory has both); ModelArtifacts::layout() reports
which one was found.
See MODELS.md for exporting the ONNX model step by step, where the engine looks for it (including the split layout), and downloading.
Quickstart
use laya::{LayaEngine, LayaOptions, LayaResult, Question, QuestionSet};
use serde_json::json;
let engine = LayaEngine::from_directory("C:/models/laya-multilingual")?;
// 1. State: a string, or anything serde_json::Value can represent. Insertion order is preserved
// for objects (the crate enables serde_json's `preserve_order` feature).
let state = json!({
"from": "[email protected]",
"subject": "Duplicate charge on invoice #4411",
"body": "Hi, we were billed twice for March. Please refund the duplicate today \
or we will cancel our plan.",
});
// 2. Typed questions, keyed by an id you choose. All three question types in one set:
let questions = QuestionSet::new()
.with("department", Question::choice(
"Which department should handle this request?",
[
("billing", "invoices, payments, refunds"),
("technical", "bugs, outages, system errors"),
("sales", "pricing, new contracts"),
("other", "everything else"),
],
)?)?
.with("urgency", Question::score(
"How urgent is this request?",
["not urgent", "soon", "critical deadline or blocking issue"],
)?)?
.with("churn_risk", Question::noul(
"Does the user threaten to cancel or leave?", serde_json::Value::Null, serde_json::Value::Null,
))?;
// 3. One batch, one forward pass, every question answered.
let result: LayaResult = engine.predict(state, &questions)?;
// (`predict` also accepts a state that only implements `serde::Serialize` via
// `engine.predict_serialize(&value, &questions)`.)
let department = result["department"].as_choice().unwrap();
println!("{}", department.choice); // billing
let urgency = result["urgency"].as_score().unwrap();
println!("{}", urgency.score); // 1.8974
let churn = result["churn_risk"].as_noul().unwrap();
println!("{}", churn.value()); // false
// 4. The same questions work in any language, with no router.
let hindi = engine.predict(
json!({ "body": "मार्च का बिल दो बार लिया गया, कृपया रिफंड करें।" }),
&questions,
)?;
println!("{:?}", hindi["department"].as_choice().unwrap().choice); // billing
# Ok::<(), laya::LayaError>(())
result["department"] panics on an unknown id (Index<&str>); use LayaResult::get for a
fallible lookup. Question::choice / Question::score return Result, so the ? above surfaces
LayaError::InvalidQuestion for a duplicate label, an empty option list, or an empty level list.
Choosing a checkpoint
use laya::{LayaCheckpoint, LayaEngine, LayaOptions};
let engine = LayaEngine::create(LayaOptions {
checkpoint: LayaCheckpoint::English,
..Default::default()
})?;
# Ok::<(), laya::LayaError>(())
| Checkpoint | Encoder | Parameters | max_len |
head_max_len |
Best for |
|---|---|---|---|---|---|
Multilingual (default) |
jhu-clsp/mmBERT-base |
322M | 1024 | 256 | Mixed-language or unknown-language traffic |
English |
answerdotai/ModernBERT-large |
421M | 512 | 192 | English-only workloads (accuracy collapses on non-Latin scripts) |
TypedDecisions |
answerdotai/ModernBERT-large |
421M | 1024 | 256 | Four specific typed-decisions workflows only |
max_len and head_max_len come from each checkpoint's rl_agent_config.json
(LayaConfig::max_len / LayaConfig::head_max_len); the table above lists the shipped defaults.
Writing questions
Three primitives, each built with an associated function on Question:
| Primitive | Constructor | Answer type | Output |
|---|---|---|---|
| choice | Question::choice(instructions, options)? / Question::choice_labels(instructions, labels)? |
ChoiceAnswer |
top label, probability per option, confidence |
| score | Question::score(instructions, levels)? |
ScoreAnswer |
expected level on an ordinal scale, distribution, confidence |
| noul | Question::noul(instructions, if_false, if_true) |
NoulAnswer |
calibrated P(true) in [0, 1] |
use laya::Question;
use serde_json::Value;
// Labels with descriptions: an ordered sequence of (label, description) pairs.
Question::choice(
"Which team should handle this?",
[("billing", "invoices, payments, refunds"), ("technical", "bugs, outages"), ("other", "")],
)?;
// Bare labels, no descriptions.
Question::choice_labels("What is `prompt` about?", ["coding", "writing", "other"])?;
// Score: levels from lowest to highest. The answer is a fractional expected index.
Question::score(
"How frustrated does the customer sound?",
["calm and neutral", "concerned but civil", "clearly annoyed", "very angry"],
)?;
// Noul (yes/no), with optional custom wording for each side.
Question::noul("Does the user explicitly request a refund?", Value::Null, Value::Null);
Question::noul("Is this email a phishing or scam attempt?", "a legitimate email", "phishing, scam, or fraud");
# Ok::<(), laya::LayaError>(())
- Labels are positional. The model scores one marker token per option, in the order you
declared them — every collection here (
QuestionSet,ChoiceQuestion::options) preserves insertion order for this reason. Question::choicerejects a duplicate label or an empty option list withLayaError::InvalidQuestion, rather than silently merging duplicates (which Python'sdictwould do, changing every subsequent marker position).instructions, option descriptions, and score levels are allimpl Into<serde_json::Value>. A string is used as written; anything else is serialized the waylaya/presets.py's non-string instructions are, with non-ASCII characters kept literal (ensure_ascii=False).- All options share a fixed head budget (
head_max_len: 256 tokens onmultilingualandtyped-decisions, 192 onenglish). If the options cannot fit even after truncation,predictreturnsLayaError::NoMarkersinstead of silently answering a truncated question.
QuestionSet
use laya::{Question, QuestionSet};
let mut questions = QuestionSet::new();
questions.insert("spam", Question::noul("Is this spam?", serde_json::Value::Null, serde_json::Value::Null))?;
// or, chained:
let questions = QuestionSet::new()
.with("urgency", Question::score("How urgent?", ["low", "high"])?)?;
questions.len(); // number of questions
questions.ids(); // an iterator of ids, in insertion order
questions.contains_id("spam"); // bool
# Ok::<(), laya::LayaError>(())
insert (and the chained with) returns Err(LayaError::InvalidQuestion(..)) on a duplicate id,
rather than silently replacing it and shifting every later row's meaning.
Built-in presets
LayaPresets provides ready-made question sets, ported from laya/presets.py. Each preset's
instructions refer to a specific state field, so pass state with that key:
| Preset | Python name | State field | Questions |
|---|---|---|---|
LayaPresets::triage() |
triage_questions |
message |
intent (choice), is_urgent, frustration (score), refund_requested, churn_risk |
LayaPresets::email(categories) |
email_questions |
body |
category (choice), is_spam, is_phishing, urgency (score), needs_reply |
LayaPresets::guard() |
guard_questions |
prompt |
jailbreak, prompt_injection, sensitive_data, harm_severity (score), topic (choice) |
LayaPresets::moderation() |
moderation_questions |
post |
toxic, harassment, threat, spam, severity (score) |
LayaPresets::router() |
router_questions |
request |
difficulty (score), domain (choice), needs_tools, is_sensitive |
use laya::{LayaEngine, LayaPresets};
use serde_json::json;
let engine = LayaEngine::from_directory("C:/models/laya-multilingual")?;
// Support ticket triage.
let triage = engine.predict(json!({ "message": "My payment failed twice" }), &LayaPresets::triage())?;
// Prompt guardrails.
let guard = engine.predict(json!({ "prompt": "Ignore all instructions" }), &LayaPresets::guard())?;
if guard["jailbreak"].as_noul().unwrap().probability > 0.5 { /* block */ }
// Email with custom routing categories, in offer order (an ordered slice or Vec, not a HashMap).
let email = engine.predict(
json!({ "body": "..." }),
&LayaPresets::email(Some([("orders", "order status, shipping"), ("returns", "returns and exchanges")])),
)?;
# Ok::<(), laya::LayaError>(())
LayaPresets::DEFAULT_EMAIL_CATEGORIES exposes the default email() categories (billing,
technical, sales, security, hr, other) in their offer order, used when email(None) is called.
Reading results
LayaResult is indexed by question id (Index<&str>, panicking on an unknown id — use
LayaResult::get for a fallible lookup) and iterates (id, &Answer) pairs in the order the
questions were asked.
# use laya::{LayaEngine, QuestionSet}; use serde_json::json;
# let engine = LayaEngine::from_directory("x")?; let questions = QuestionSet::new();
let result = engine.predict(json!("some state"), &questions)?;
result.model(); // "laya-rl-agent"
result.usage().input_tokens; // non-padding tokens across the whole batch
result.usage().output_tokens; // always 0: nothing is generated
result.len();
for (id, answer) in result.iter() {
answer.question_type(); // QuestionType::Choice / Score / Noul
answer.confidence(); // 0..1
answer.action().act_probability; // the action head: P(act rather than escalate)
match answer {
laya::Answer::Choice(c) => { let _ = &c.choice; let _ = c.probability("billing"); }
laya::Answer::Score(s) => { let _ = s.score; let _ = s.most_likely_level(); }
laya::Answer::Noul(n) => { let _ = n.probability; let _ = n.value(); }
}
}
// Narrow without a match: `as_choice()` / `as_score()` / `as_noul()` return `Option<&_>`.
let department = result["department"].as_choice().unwrap();
if let Some(refund) = result.get("refund_requested") { /* optional id */ }
# Ok::<(), laya::LayaError>(())
Confidence. For choice and score it is normalized Shannon entropy, 1 - H(p) / log k. A
single-option question gets 1.0. For noul it is max(p, 1 - p). Every probability, score and
confidence is rounded to 4 decimal places (Calibration::round4), as in Python.
Thread safety
LayaEngine holds a multi-hundred-megabyte ONNX Runtime session. Create one and share it,
typically behind an Arc:
use laya::{LayaEngine, LayaOptions};
use std::sync::Arc;
let engine = Arc::new(LayaEngine::create(LayaOptions::default())?);
let handles: Vec<_> = (0..4).map(|_| {
let engine = Arc::clone(&engine);
std::thread::spawn(move || {
// engine.predict(..) from multiple threads concurrently.
})
}).collect();
for h in handles { h.join().unwrap(); }
# Ok::<(), laya::LayaError>(())
LayaEngine is Send + Sync (asserted at compile time in src/laya_engine.rs), so Arc<LayaEngine>
works across threads. Internally: LayaTokenizer::encode takes &self (the tokenizers crate's
Tokenizer needs no lock), but ort 2.x's Session::run takes &mut self — a Session is not
Sync on its own, even though ONNX Runtime's underlying InferenceSession::Run is itself
documented as thread-safe. LayaEngine therefore holds the ONNX session in an internal Mutex, so predict calls from
multiple threads serialize on the forward pass itself but never block on tokenization or
calibration.
Execution providers and cargo features
| Feature | Default | Effect |
|---|---|---|
download |
on | Enables ModelArtifacts::resolve's opt-in HTTP download via ureq (TLS through ureq's default rustls backend — no system OpenSSL needed). Without it, allow_download = true returns a descriptive LayaError::Download instead of failing to compile a download path. |
cuda |
off | Forwards to ort's cuda feature. Requires LayaOptions::execution_provider = LayaExecutionProvider::Cuda and a working CUDA/cuDNN install at runtime. |
directml |
off | Forwards to ort's directml feature (Windows only). Requires LayaOptions::execution_provider = LayaExecutionProvider::DirectMl. |
laya = { path = "...", features = ["cuda"] }
ort's default download-binaries feature fetches a prebuilt CPU ONNX Runtime at build time, so
CPU inference needs no system install. LayaOptions::execution_provider defaults to
LayaExecutionProvider::Cpu; requesting Cuda or DirectMl without the matching cargo feature
returns LayaError::Onnx naming the missing feature, rather than a Session::builder failure with
no context.
Async usage
predict is CPU-bound and blocking, like the underlying ONNX Runtime call and (when
allow_download triggers a fetch) like ureq. There is no async predict (see
Differences from the Python SDK). From an async runtime, offload
it:
# use laya::{LayaEngine, LayaOptions, QuestionSet}; use std::sync::Arc; use serde_json::json;
# async fn example() -> laya::Result<()> {
let engine = Arc::new(LayaEngine::create(LayaOptions::default())?);
let questions = QuestionSet::new();
let engine2 = Arc::clone(&engine);
let result = tokio::task::spawn_blocking(move || {
engine2.predict(json!("state"), &questions)
})
.await
.expect("predict panicked")?;
# let _ = result;
# Ok(())
# }
The same pattern applies to ModelArtifacts::resolve / LayaEngine::create with
allow_download = true: run it inside spawn_blocking if it must not stall an async executor.
Configuration reference (LayaOptions)
| Field | Type | Default | Notes |
|---|---|---|---|
model_directory |
Option<PathBuf> |
None |
Highest-priority artifact location. When set, all other resolution steps are skipped. |
checkpoint |
LayaCheckpoint |
Multilingual |
Derived subdirectory, HF subfolder and cache path all come from this unless overridden by setting hugging_face_subfolder directly, or by model_directory. |
execution_provider |
LayaExecutionProvider |
Cpu |
Cuda / DirectMl need the matching cargo feature (see above). |
intra_op_threads |
Option<u32> |
ORT default | Threads within one operator. |
inter_op_threads |
Option<u32> |
ORT default | Threads across independent operators. |
allow_download |
bool |
false |
Network access is never implicit. |
hugging_face_repo |
String |
"convaiinnovations/laya" |
Repository to download from. |
hugging_face_subfolder |
Option<String> |
None (derived from checkpoint) |
Subfolder override in the repository and local cache, for a custom export that doesn't follow the standard bundle layout. Prefer setting checkpoint instead. The computed getters hugging_face_subfolder() and hugging_face_download_subfolder() fall back to checkpoint's subdirectory when this is None; English's repository subfolder differs from its local cache directory name, which is why the two getters exist separately. |
hugging_face_token |
Option<String> |
None → HF_TOKEN |
Bearer token for gated or private repos. Some(String::new()) disables auth. |
cache_directory |
Option<PathBuf> |
per-user cache | Root that downloads (and cache lookups) land under. |
download_progress |
Option<Arc<dyn Fn(ArtifactDownloadProgress) + Send + Sync>> |
None |
Per-file progress sink. |
Every field is pub, including hugging_face_subfolder, so LayaOptions can be built with
ordinary struct-update syntax as well as LayaOptions::new() plus field assignment:
use laya::LayaOptions;
let options = LayaOptions {
allow_download: true,
..Default::default()
};
Long prediction
predict reads one sequence: a state longer than the room the questions leave is truncated (the
start is kept, or the end for a conversation-turn list). LayaEngine::predict_long scans the whole
state instead, in overlapping token windows, and keeps one answer per question. It ports
ONNXAgent.predict_long with aggregate="auto":
# use laya::{LayaEngine, LongOptions, QuestionSet};
# let engine = LayaEngine::from_directory("x")?; let questions = QuestionSet::new();
# let long_transcript = "";
let options = LongOptions::new().window(96).stride(48).batch_size(8);
let result = engine.predict_long(long_transcript, &questions, &options)?;
result.scan_usage().unwrap().windows; // windows scored (1 = no scan was needed)
let answer = &result["refund_requested"];
if let Some(window) = answer.window() { // None for a single-pass state
println!("decided by window {}/{} (tokens {}..{})",
window.index + 1, window.count, window.token_start, window.token_end);
}
# Ok::<(), laya::LayaError>(())
- Aggregation.
noultakes the window with the greatest P(true).choiceandscoretake the window with the greatestanswer_confidence. A tie goes to the first window. The probabilities are that window's, not a calibrated number for the whole document. - One pass. A state that fits the room the questions leave (with an explicit
LongOptions::window, a state that fits that window) is scored by the ordinarypredictpass: one window, noAnswer::window(). - Options.
window,strideandbatch_sizeare optional and0counts as unset. The default window ismax(64, max_len - head_max_len - 8), capped at the smallest state room any question leaves; the default stride is half the effective window. A stride that would skip tokens between windows isLayaError::InvalidWindow, as is a question set whose options leave no room for the state.aggregateaccepts only"auto"(LayaError::UnsupportedAggregateotherwise). - Usage.
result.usage().input_tokensis summed over windows.result.state_usage()sumsstate_tokensandstate_tokens_droppedover windows and lists the last window's truncated questions;result.scan_usage()gives the window count and how many windows lost state tokens. - Re-encoding. Each window is decoded back to text with special tokens kept (Python's default
decode) and scored as an ordinary state, so its tokens can differ slightly from the original after re-encoding, exactly as in Python. A tokenizer that does not overrideLayaTokenizer::decodemakespredict_longfail withLayaError::Tokenizer. - Router.
LayaRouter::predict_long(state, &questions, &options, model, task, lang)routes once, on the original state, then scans on the chosen checkpoint. A predictor without a scan capability (seeLayaPredictor::as_long_predictor) givesLayaError::NotSupportedrather than truncating. - Warnings. The two
warnings.warnnotices Python emits when it caps the window or reduces the stride are sent, with Python's exact text, to thelogfacade atwarnlevel with targetlaya. Install anylogbackend (for exampleenv_logger) to see them; with none installed they are dropped. - Not ported. Per-call hooks are not part of
predict_longhere. As in Python, it takes nomin_confidence,max_lenorhead_max_len;LongOptions::langselects the per-language temperatures, see Per-language temperatures.
Batch prediction
LayaEngine::predict_batch answers the same questions over many states and shares forward passes
between them: N states cost one pass, or ceil(N / batch_size) of them, instead of N. It ports
ONNXAgent.predict_batch and gives each state the result predict would, in input order:
# use laya::{BatchOptions, LayaEngine, QuestionSet};
# let engine = LayaEngine::from_directory("x")?; let questions = QuestionSet::new();
# let states: Vec<serde_json::Value> = vec![];
let options = BatchOptions::new().batch_size(8).sort_by_length(true);
let results = engine.predict_batch(&states, &questions, &options)?; // results[i] answers states[i]
# Ok::<(), laya::LayaError>(())
batch_size. The most states per forward pass.Noneor0sends them all in one pass, as Python does forNone,0and negative values.sort_by_length. Groups similarly sized states inside windows of eight batches, so each batch pads to a shorter maximum. It takes effect only with abatch_sizeabove one and below the number of states, and results keep input order. Different batch shapes can change floating-point predictions slightly near a decision threshold.- Edges. No states give an empty list. No questions give one empty result per state, with no
state usage. A
nullstate isLayaError::InvalidState(Python'sTypeError); any other error fails the whole call. - Router.
LayaRouter::predict_batch(&requests, &options)takesBatchRequests (state, questions and the optionalmodel,task,langandlang_guessoverrides), routes each, groups them by checkpoint and then by question schema (order-sensitive, so option order is positional), runs each group throughpredict_batchand returns the results in request order, each with itsrouting.LayaRouter::route_batch(&requests)returns only the decisions and loads nothing. A predictor without a batch capability (seeLayaPredictor::as_batch_predictor) givesLayaError::NotSupported. min_confidence.BatchOptions::min_confidencegates every result of the call; see Abstention.- Per-call
lang,max_len,head_max_len. OnBatchOptionsfor the engine, and per request onBatchRequestfor the router; see Per-language temperatures. - Not ported. Per-call hooks and the router's
diagnosticsreport.BatchOptionsandBatchRequestare#[non_exhaustive]so those can be added without breaking callers.
Per-language temperatures and per-call overrides
Ports lang_temperatures, lang, max_len and head_max_len from Python's ONNXAgent and
Router.
LayaOptions::lang_temperatures. A table of language code toLangTemperature(an optional list of three temperatures, which inherits the checkpoint's own when absent, and optionaltemperature_by_optionsbuckets), built once when the engine is. A router passes it to every engine it builds. Keys are matched ascode.split('-')[0].lower()(de-ATandDEboth namede), the last duplicate wins, and every value is clamped to[0.5, 5]exactly as the checkpoint's own are. A value the clamp replaced is named in one warning per language, sent with Python's text to thelogfacade (warn, targetlaya). A temperature list that is not three long isLayaError::InvalidConfigwith Python'sValueErrortext. The same parser isresolve_lang_temperatures.- Per-call
lang.PredictOptions::lang,BatchOptions::langandLongOptions::langselect an entry for that call. A language in the table replaces both the checkpoint's bucket map and its per-type temperature (there is no fallback to the checkpoint's bucket); an unknown or empty code changes nothing. This port has no histogram-binning calibration, which Python skips under an override. OnLayaRouter, the explicitlangargument is the language (it also routes), else the language the routing detection found; the fields on the options are not read there (batch-wide ones are refused bypredict_batch). Inpredict_batcha request'slang(else the detected one) joins the grouping key, raw and unnormalised, only when the engine has a table, so a table-less group still shares one pass.LayaRouter::predict_batchhas no batch-widelang,max_lenorhead_max_len, as in Python (TypeError: Router.predict_batch() got an unexpected keyword argument 'lang'): the same fields on itsBatchOptionsare refused withLayaError::NotSupportedand that text. - Per-call
max_lenandhead_max_len.PredictOptions,BatchOptionsandBatchRequest(settersmax_len/head_max_len) override the checkpoint's budget for that call;Nonekeeps it. They arei64because Python does no validation:0and negative values flow into the same arithmetic and slices, somax_lenof0or below leaves no option marker and the question isLayaError::NoMarkersnaming the value as written, while ahead_max_lenof0or below gives each option four tokens.predict_longtakes neither, as in Python. InLayaRouter::predict_batchrequests that ask for different budgets are split into separate forward passes. (A config's ownmax_len/head_max_lenare still refused at load when negative; see the differences below.) - Attached predictors.
LayaPredictor::predict_with_optionsandhas_lang_temperatureshave defaults, so existing implementors compile. The default dropslangand refuses a token-budget override withLayaError::NotSupported; Python passes the budget on and the agent raises aTypeError.
Abstention (min_confidence)
An opt-in gate on answer_confidence (max(p), the quantity the calibration figures describe). It
ports min_confidence from Python's system_one, predict_batch and Router.predict. A threshold
is a MinConfidence: one number for every answer, or a map from option-count bucket to threshold,
because one level does not transfer across option counts.
# use laya::{LayaEngine, MinConfidence, PredictOptions, QuestionSet};
# let engine = LayaEngine::from_directory("x")?; let questions = QuestionSet::new();
let gate = MinConfidence::map([("choice:2", 0.8), ("choice:3-5", 0.6), ("default", 0.5)])?;
let options = PredictOptions::new().min_confidence(gate);
let result = engine.predict_with_options("I want a refund", &questions, &options)?;
for (_id, answer) in result.iter() {
// "passed", "abstained" or "unevaluated"; None when no threshold was set.
println!("{:?} below {:?}: {}", answer.abstention(), answer.abstention_threshold(), answer.low_confidence());
}
# Ok::<(), laya::LayaError>(())
- Buckets. Keys are
choice,scoreornoulplus:2,:3-5,:6-10or:11+(laya::bucket_keys()), and"default"for any bucket the map does not name. A bucket the map does not name, with no"default", is gated at0.0, which gates nothing. - Verdict. With a threshold set every answer reports
abstention(laya::GATE_STATES) andabstention_threshold;low_confidence()is true only for"abstained". An answer with no usable confidence is"unevaluated", not a pass. A threshold of0.0was set, so it still reports states. With no threshold nothing is reported (abstention()isNone). - Validation.
MinConfidence::scalarandMinConfidence::maprefuse what Python'scheck_min_confidencerefuses (out of[0.0, 1.0], NaN, an empty map, an unknown bucket key) with Python'sValueErrortext asLayaError::InvalidMinConfidence. The variants are public, so the engine and the router validate again where Python does: the router before it routes or loads anything, the engine before it looks at the states. - Router.
LayaRouter::predict_with_optionsandBatchOptions::min_confidenceonLayaRouter::predict_batch(one threshold for the whole batch, not part of the grouping). The router applies the gate itself, so an attached predictor needs no support for it. predict_long. Takes no threshold, as in Python.Answer::jev_confidence(). TheconfidenceJev would report for the answer's probabilities (choice and score only;Nonefor noul or malformed probabilities). It is a different formula from Laya's normalized entropy, so a threshold carried over from Jev selects about twice as much on Laya.- Python 3.12 arithmetic.
jev_confidencesums with Python's compensatedsum()(CPython 3.12 and later), so its last digits match a 3.12+ Python, not 3.11 or earlier.
Routing, language detection, email and shortlist
Runnable end-to-end example: samples/laya-routing.
Language detection — no model, no network:
use laya::language_detection::LanguageDetection;
use serde_json::json;
let analysis = LanguageDetection::analyse(&json!("Mein Konto wurde zweimal belastet"));
assert_eq!(analysis.script, "latin");
assert_eq!(analysis.language.as_deref(), Some("de"));
assert!(!analysis.is_english);
LayaRouter picks a checkpoint per request from script/language detection (or an explicit
override), loading checkpoints lazily under an LRU cache (Arc<dyn LayaPredictor> per slot, so an
evicted checkpoint stays alive until every in-flight predict on it finishes):
use laya::laya_router::{LayaPredictor, LayaRouter, LayaRouterOptions};
use serde_json::json;
# use laya::QuestionSet;
# let questions = QuestionSet::new();
let router = LayaRouter::new(LayaRouterOptions { max_loaded: 2, ..Default::default() })?;
let result = router.predict(json!({ "message": "I was charged twice" }), &questions, None, None, None)?; // -> english
let de = router.predict(json!({ "message": "Mein Konto wurde zweimal belastet" }), &questions, None, None, None)?; // -> multilingual
println!("{}", result.routing().unwrap().reason); // "English Latin text"
# Ok::<(), laya::LayaError>(())
router.route(state, Some(&questions), None, None, None) decides without loading or running
anything. router.preload(None) builds every checkpoint up front for a server or demo.
A state that reads English as a whole is still checked line by line and field by field, so a
Portuguese message under an English stack trace goes to the multilingual checkpoint
(LanguageAnalysis::mixed_segment names the line). An explicit lang that names no language
(blank, C, POSIX, C.UTF-8, und, zxx, mul) falls through to detection instead of pinning a
checkpoint. If you already run a language-identification model, hand its answer to the router as a
LangGuess -- a code or a callback, checked after an explicit lang and before detection, and
abstaining (None, blank, C, ...) falls through:
use laya::laya_router::{LangGuess, LayaRouter, LayaRouterOptions};
use serde_json::json;
// one hint for every request ...
let router = LayaRouter::new(LayaRouterOptions::default())?.with_lang_guess("ro");
// ... or per call, as a function of the state
let hint = LangGuess::callback(|state| my_language_id_model(state));
let decision = router.route_with_lang_guess(&json!("Care este ora in Tokyo?"), None, None, None, None, Some(&hint))?;
# fn my_language_id_model(_: laya::laya_router::GuessInput<'_>) -> Option<String> { None }
# Ok::<(), laya::LayaError>(())
Email cleaning strips quoted history, signatures and disclaimers before you ask any questions:
use laya::laya_email::LayaEmail;
let cleaned = LayaEmail::clean_body(raw_body, laya::laya_email::DEFAULT_MAX_CHARS);
let state = LayaEmail::state(subject, raw_body, Some("[email protected]"), true, std::iter::empty());
It reads English, Portuguese, Spanish and French mail (Gmail's Em ... escreveu: / Le ... a écrit :,
Outlook's -----Mensagem original-----, Atenciosamente / Cordialement, Envoyé depuis mon iPhone, confidentiality footers). max_chars is a slice budget: raise it for a long message
(LayaEmail::state_with_max_chars, clean_body_signed). 0 gives an empty body and a negative value
trims from the end, exactly as in Python, so "no limit" means a budget larger than the body.
Embedding shortlist narrows a high-cardinality choice question before the model sees it, using a caller-supplied embedding function (bring your own bi-encoder — this SDK does not include one):
use laya::laya_shortlist::LayaShortlist;
# use laya::{laya_router::LayaPredictor, QuestionSet}; use serde_json::json;
# fn embed(_texts: &[String]) -> laya::Result<Vec<Vec<f64>>> { Ok(vec![]) }
# let router: &dyn LayaPredictor = todo!(); let state = json!(""); let questions = QuestionSet::new();
let result = LayaShortlist::predict(router, state, &questions, embed, 20)?;
let kept = &result.shortlist().unwrap()["banking_intent"]; // kept.labels, kept.scores, kept.k, kept.n
# Ok::<(), laya::LayaError>(())
Samples
laya-rust is a Cargo workspace ([workspace] members in its Cargo.toml lists the three
samples/ crates, with the laya-onnx library itself as the workspace's root package) so the sample crates, the library, and their tests all share one target/ directory and Cargo.lock. Plain
cargo build / cargo test at the workspace root still only touch the laya-onnx library crate (the
default member); build or test everything, including the samples, with --workspace:
cargo build --workspace
cargo test --workspace
| Sample | What it shows |
|---|---|
samples/laya-sample |
The quickstart above as a runnable console app: one engine, four typed questions, answered in English then Hindi. Also a parity fixture — see its README's "What it is testing". |
samples/laya-benchmark |
Cold/warm latency, p95, peak memory and an answer snapshot for one checkpoint, as one JSON object — pairs with the Python benchmark. |
samples/laya-routing |
Language detection, LayaRouter (routing decisions and multi-checkpoint predict), email cleaning and the embedding shortlist, in five sections. |
# from laya-rust/, with the exports in the repository-root onnx/ folder
cargo run --release -p laya-sample -- --model-dir ../onnx/multilingual
cargo run --release -p laya-benchmark -- --checkpoint multilingual --model-dir ../onnx/multilingual
cargo run --release -p laya-routing -- --model-root ../onnx
Testing
cargo build
cargo build --no-default-features # download feature off
cargo test
cargo clippy --all-targets -- -D warnings
cargo fmt --check
The suite has tiers, so it stays useful without the ~1.3 GB artifact:
| Tier | Needs | Covers |
|---|---|---|
| 1 | nothing | PythonJson dialects, calibration math, sequence building against a stub tokenizer, collation, presets, ModelArtifacts resolution/validation with temp directories |
| tokenizer | tokenizer/tokenizer.json per checkpoint |
token ids and marker positions against recorded Python output |
| 2 | the full artifact per checkpoint | predict against recorded Python answers, plus a shape sweep and a concurrency test |
Golden vectors are read from tests/golden/<checkpoint>/ (recorded from Python by
tools/dump_golden.py), overridable with LAYA_GOLDEN_DIR. ONNX artifacts are found via LAYA_ONNX_DIR (multilingual only, legacy) or
LAYA_ONNX_ROOT (parent of all three checkpoint subdirectories), or by walking up from the crate
root looking for onnx/<checkpoint>/. Tests whose artifacts are missing print a skip: line and
return, rather than failing — a machine with no exported checkpoints still gets a green Tier-1 run.
The advisory Rust parity lane runs on every push and PR,
with no path filter. It is sharded per checkpoint (english, multilingual, typed-decisions, plus
a router cell for the tests that load english and multilingual together). Each cell regenerates
its own checkpoint's goldens, and the routing goldens, from the Python laya/ at that commit
using tools/regen_golden.py, then runs the Cargo tests with
LAYA_TEST_CHECKPOINTS set to its checkpoints and its own --min-passed floor.
Goldens are never cached; the exports are, per checkpoint. A skip: diagnostic or ignored test makes the lane red, even when
Cargo counts a missing-model early return as a pass. Job-level continue-on-error keeps Python
improvements unblocked; Rust drift needs a follow-up port. This workflow never publishes crates.
The skip guard runs only after successful Cargo tests; regeneration/build failures retain their
own diagnostics instead of producing a secondary missing-log error.
The regeneration self-check compares option logits within 1e-3 and every unrounded action
softmax probability within 1e-4, matching the signal Python and Rust predict consume.
Action classes are not padded option slots and none are excluded. Raw action logits remain
recorded (and their max difference reported) for diagnostics, but large saturated logits can
vary across CPU kernels without changing action probabilities. Nonfinite or mismatched-shaped
outputs are rejected. The recorded-graph Rust test uses the same action probability bound.
Engine results expose Python 0.4.0's state-budget diagnostics via LayaResult::state_usage():
original state tokens, worst-case dropped tokens, whether truncation occurred, and the affected
question ids. The existing Usage totals and result constructor remain source-compatible.
state_usage().options additionally identifies questions whose clipped option spans collide.
To reproduce it, install CPU torch and tools/requirements-regen.txt,
then run from the repository root:
python laya-rust/tools/test_ci_tools.py
python laya-rust/tools/regen_golden.py
cd laya-rust
cargo test --workspace --locked -- --nocapture --test-threads=1 2>&1 | tee cargo-test.log
python tools/check_test_skips.py cargo-test.log --min-passed 200
Float-formatting probes can also be regenerated without models:
python laya-rust/tools/regen_golden.py --checkpoint repr. Their IEEE-754 bit patterns cover
half-even ties, subnormals, notation boundaries and 2,000 deterministic random finite values.
Use --artifacts-root for exports outside the checkout, with LAYA_ONNX_ROOT=<root>/onnx
and LAYA_ONNX_SPLIT_ROOT=<root>/onnx-split. Regeneration reuses only complete exports whose
stamp matches the pinned checkpoint, exporters, toolchain and Python model definition.
The fused exporter is shared with laya-dotnet/tools/export_onnx.py; split exports use
laya-ts/scripts/export_onnx.py.
LAYA_TEST_CHECKPOINTS (space- or comma-separated subdirectory names, e.g. english or
"english multilingual") limits the model-backed and golden-backed tests to those checkpoints; a
deselected checkpoint is skipped silently, a selected one with missing artifacts or goldens prints a
skip: line, and an unknown name panics. Unset runs every checkpoint.
Model loads are ~1.3–1.7 GB each; run model-backed tests serially (optionally with --release):
LAYA_ONNX_ROOT=/path/to/onnx cargo test --release -- --test-threads=1
ModelArtifacts's own download-path unit tests (404 / 401 / 403 handling, the .part-then-rename
write) run against a local TcpListener, not the real network, so they run in every cargo test
invocation that has the download feature enabled (the default) with no external dependency.
Module map
| Rust module | Purpose | Rust type(s) |
|---|---|---|
src/python_json.rs |
Byte-for-byte json.dumps dialects |
PythonJson |
src/questions.rs |
Typed question definitions | QuestionType, Question, ChoiceQuestion, ScoreQuestion, NoulQuestion, QuestionSet |
src/answers.rs |
Typed answer results | Answer, ChoiceAnswer, ScoreAnswer, NoulAnswer, ActionInfo, LayaResult, Usage, ScanUsage, StateUsage, WindowInfo |
src/calibration.rs |
Temperature scaling and confidence math | Calibration |
src/laya_config.rs |
Parsed rl_agent_config.json |
LayaConfig |
src/lang_temperatures.rs |
Per-language temperature table and its parser | LangTemperature, TemperatureValue, LangTemperatures, resolve_lang_temperatures |
src/laya_options.rs |
Engine construction options | LayaOptions, LayaCheckpoint, LayaExecutionProvider, ArtifactDownloadProgress |
src/model_artifacts.rs |
Locating/downloading model files | ModelArtifacts, ModelLayout, ModelPaths (the fused/split layout enum and its resolved paths — see Getting the model) |
src/laya_presets.rs |
Built-in question presets | LayaPresets |
src/sequence_builder.rs |
Token sequence construction | SequenceBuilder |
src/collator.rs |
Batches sequences for one forward pass | Collator, CollatedBatch, SequenceItem |
src/tokenization/laya_tokenizer.rs |
Tokenizer abstraction | LayaTokenizer (trait) |
src/tokenization/hf_tokenizer.rs |
HuggingFace tokenizer implementation | HfTokenizer |
src/laya_engine.rs |
Loads the ONNX model and answers questions | LayaEngine |
src/long_prediction.rs |
Window plan and aggregation for predict_long |
LongOptions |
src/batch_prediction.rs |
Chunking, sort windows and order restoration for predict_batch |
BatchOptions |
src/confidence.rs |
The abstention gate and Jev's confidence |
MinConfidence, GATE_STATES, bucket_keys |
src/predict_options.rs |
Per-call options for predict_with_options |
PredictOptions |
src/error.rs |
Error type | LayaError, Result<T> — every fallible call in this crate returns Result<T, LayaError> |
Where the equivalent Python code is a module of related functions (common.py, agent.py), the
Rust port groups the corresponding functions into a unit struct with associated functions
(PythonJson::state(...), Calibration::round4(...), ...) so call sites read the same way.
Differences from the Python SDK
LayaErrorinstead of exceptions. Onethiserrorenum covers every failure mode Python expresses by raising (missing files, bad config, invalid questions, no markers left after truncation). Library code never panics on bad input.- No async
predict.ModelArtifacts::resolve's download step andLayaEngine::predictare both synchronous, blocking calls (ureqfor HTTP, blockingort::Session::runfor inference). An async caller wraps the call inspawn_blocking(see Async usage). serde_json::Valueinstead of a dynamically-typed object. State, instructions, option descriptions and score levels areserde_json::Value(with thepreserve_orderfeature, so objects keep insertion order), the same JSON model Python'sdict/list/scalar values map onto. Public constructors takeimpl Into<Value>, so&str,String, numbers andserde_json::json!{..}all work directly; a type that implementsserde::Serializebut isn't already aValuegoes throughserde_json::to_valuefirst.PythonJson::criterion'sdefault=strfallback is not reachable. Python'srender_criterionfalls back tostr(value)for valuesjson.dumpscannot serialize on its own; the three dialect functions here take&serde_json::Value, which can only ever hold JSON-representable data, so that fallback path has nothing to reach. The blank-criterion rule and the float/escaping rules are otherwise byte-for-byte identical to Python.- Warnings use the
logfacade. The clamped-temperature warning (LayaConfig::parse) and the twopredict_longwindow warnings are sent, with Python's text, tologatwarnlevel with targetlaya.LayaConfig::clamped_temperaturesstill records the clamped entries. QuestionTypecarries no methods. The Python type name lives onSequenceBuilder::type_namerather than on the enum, and theqtypetensor's integer encoding is a plainqtype as i64cast (the enum's discriminants already matchQTYPES). The reverse conversion usesTryFrom<i64>, since Rust has no implicit enum-from-integer cast the way Python's dynamic typing allows.Agent::new_with_defaults(), reused across all five files in one download. Matches Python's single reused HTTP session, but there's no async cancellation token — a download that must be cancellable should run on its own thread and be dropped/joined by the caller.- Config values Python reads loosely are refused at load.
max_lenandhead_max_lenaccept a non-negative JSON integer or a bool (trueis 1,falseis 0), as Python does. A float, string,null, list or object is refused when the engine is created, with theTypeErrortext Python raises at its first prediction (head_max_lenis met first); a negative value is refused withmax_len=-5: must be a non-negative integer, since Python slices from the end for it. Python also tolerates a small fractionalhead_max_len(roughly up to 8 plus 4 per option); that is refused here too. - Lone surrogate escapes in config JSON are refused.
"\ud800"loads in Python and is an error here, because a RustStringcannot hold a lone surrogate. - JSON nesting is cut off at 1000 levels. Python's limit depends on the interpreter's recursion limit.
- Config files are decoded as strict UTF-8. Python opens them with the locale's encoding (cp1252
on a default Windows install), which only differs for non-ASCII bytes; a BOM is rejected, as
json.loadsdoes.
Known gaps
Language detection, routing and email cleaning follow Python 0.4.14 and are checked against routing probes and a seeded differential corpus regenerated at that version (
tools/regen_golden.py --checkpoint routing). A later Python change needs them regenerated.Opt-in features are not ported yet: histogram-binning calibration and per-call hooks.
predict_long,predict_batch, themin_confidencegate,lang_temperaturesand the per-calllang,max_lenandhead_max_lenare ported, without hooks. They are off by default in Python, so default predictions are unaffected. They are planned as follow-ups.