Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Changelog
All notable changes to the laya-onnx crate are documented here. The format follows
Keep a Changelog, and the project uses
Semantic Versioning. Releases are tagged laya-rust-v<version>.
[0.1.0] - unreleased
First release: a port of the Laya Python SDK, checked against goldens and routing probes regenerated from laya (Python) 0.4.14 for the supported paths (see the known gaps below).
Added
- Per-language temperatures and per-call overrides, ported from
laya/common.pyresolve_lang_temperaturesandONNXAgent:LayaOptions::lang_temperatures(a table ofLangTemperatureentries, resolved with Python's key normalisationlang.split('-')[0].lower(), last-duplicate-wins, clamp and one warning per language through thelogfacade; a temperature list that is not three long isLayaError::InvalidConfigwith Python'sValueErrortext), the publicresolve_lang_temperatures, and the router passing the table to every engine it builds.PredictOptions::lang,BatchOptions::langandLongOptions::langselect an entry per call (it replaces both checkpoint maps, with no fallback to the checkpoint's bucket); the router uses its explicitlangargument, else the detected language, andpredict_batchadds a request'slangto the grouping key, raw, only for an engine with a table.PredictOptions,BatchOptionsandBatchRequestalso takemax_lenandhead_max_len(i64: Python does not validate them, so0and negative values follow its arithmetic, and a question that loses an option marker isLayaError::NoMarkersnaming the value as written); the router splits a batch on them.predict_longtakes neither, as in Python.LayaRouter::predict_batchrefuses a batch-wideBatchOptions::{lang, max_len, head_max_len}with Python'sTypeErrortext, as the request is where they belong. API notes: breaking: themax_lenandhead_max_lenfields ofLayaError::NoMarkerschange fromusizetoi64, so code that reads or builds them must adapt;LayaOptionshas a new public field, andLayaPredictorgained the provided methodspredict_with_optionsandhas_lang_temperatures. - The abstention gate, ported from
laya/confidence.py:PredictOptions::min_confidencewithLayaEngine::predict_with_optionsandLayaRouter::predict_with_options, andBatchOptions::min_confidenceforpredict_batchon the engine and the router (one threshold for the whole batch, applied to every result and no part of the router's grouping).MinConfidenceis a scalar or a per-bucket map, built withMinConfidence::scalarandMinConfidence::map, which refuse whatcheck_min_confidencerefuses with Python'sValueErrortext (LayaError::InvalidMinConfidence); the router validates before it routes or loads, and the engine before it looks at the states. With a threshold set,Answer::{low_confidence, abstention, abstention_threshold}report the gate (GATE_STATES: passed, abstained, unevaluated); without one they report nothing.Answer::jev_confidenceis Python'sjev_confidenceandbucket_keys()itsbucket_keys.predict_longtakes no threshold, as in Python. Checked againstrouting/confidence_probe.json, which Python's ownconfidence.pywrites. Two API notes:BatchOptionsno longer derivesEq(it now holds floats), andLayaErroris not#[non_exhaustive], so an exhaustivematchon it outside the crate needs an arm forInvalidMinConfidence. LangGuessandLayaRouter::{route_with_lang_guess, predict_with_lang_guess, predict_long_with_lang_guess, with_lang_guess}: Python'slang_guesshint (a language code or a callback taking the state), per call or installed on the router, checked after an explicitlangand before detection. A hint that names no language falls through to detection.LanguageAnalysis::mixed_segment, the line or field the segment scan singled out in a state that reads English as a whole, and the matching routing reason ("Latin script, mostly English, but a line or field reads as ...").LanguageAnalysisis now#[non_exhaustive], so a field added later does not break callers again; read it, do not construct it.LayaEmail::{clean_body_signed, state_with_max_chars}:max_charsas Python takes it, including themax_chars * 4pre-cut and the specified results for0and negative budgets.clean_bodyandstateare unchanged and keep the 3000-character default.LayaEngine::predict_longandLayaRouter::predict_long(aggregate="auto"): scan a state longer than one sequence in overlapping token windows and keep the most confident window per question, withAnswer::window()attribution,LayaResult::scan_usage()and summedstate_usage(). Matches Python'sONNXAgent.predict_longand the .NET SDK; hooks are not part of it, and, as in Python, it takes nomin_confidence. NewLongOptions,LayaLongPredictor,LayaError::{UnsupportedAggregate, InvalidWindow, NotSupported}and a defaultLayaTokenizer::decode.LayaErroris not#[non_exhaustive], so an exhaustivematchon it outside the crate needs arms for the three new variants.LayaEngine::predict_batch(andpredict_batch_serialize) andLayaRouter::{predict_batch, route_batch}: Python'sONNXAgent.predict_batchandRouter.predict_batch/route_batch. One result per state in input order,batch_sizestates per forward pass (None,0and, in Python, negative values send them all in one), andsort_by_lengthwindows of eight batches that apply only with abatch_sizeabove one and below the state count. The router groups requests by checkpoint and then by question schema, restores request order, attaches each request'srouting, and honours each request'smodel,task,langandlang_guess;route_batchloads nothing. NewBatchOptions,BatchRequest(both#[non_exhaustive], with chained setters, so hooks can be added later),LayaBatchPredictorwith the provided methodLayaPredictor::as_batch_predictor(a default ofNone, so existing implementors keep compiling) andLayaError::InvalidState, raised for anullstate as Python raisesTypeError. Hooks and the router'sdiagnosticsreport are not part of it.LayaErroris not#[non_exhaustive], so an exhaustivematchon it outside the crate needs an arm for the new variant.- Behaviour change:
LayaEngine::predictandpredict_serializenow act as Python'ssystem_one, a one-statepredict_batch. Anullstate returnsLayaError::InvalidState(Python'sTypeError) instead of being scored as the text"null", and an emptyQuestionSetreturns an empty result with zero usage instead ofLayaError::InvalidQuestion.predict_longrefuses anullstate the same way, before it checks the aggregate mode, as Python does. predictnow runs through a shared multi-state core that scores several states in one forward pass; its results are unchanged.- Advisory parity CI on every push and PR: regenerate fixtures from the current Python checkout, test all checkpoints, and reject skipped tests. No golden cache or crate publication.
- Python float-repr parity probes, including exact half-even decimal ties and random bit patterns.
LayaResult::state_usage()reports state-token truncation across question budgets, matching Python 0.4.0 without changing the existingUsagetotals or result constructor.- Engine score legends render structured criteria as strings, matching Python 0.4.0.
NoMarkerserrors include actual/requested marker counts and the total sequence budget, matching Python's actionable truncation diagnostic.- State usage also reports option-span collisions after clipping, including the applied cap.
LayaEngine:choice,scoreandnoulpredictions over the exported ONNX checkpoints (english, multilingual, typed-decisions); all questions for a state run in one batched forward pass.- Loads both ONNX layouts: the fused
model.onnxand the splitencoder.onnx+head.onnx. - Answers report
confidenceandanswer_confidence(added in Python 0.3.21). - Python-compatible sequence building, JSON serialization and calibration, checked against goldens recorded from the Python SDK.
- Checkpoint router, language detection, email state and option shortlisting.
- Opt-in model download from Hugging Face (Cargo feature
download). - Samples: quickstart, routing and benchmark.
Fixed
Calibration::round4is now exactly Python'sround(x, 4): it rounds the exact binary value half to even in integer arithmetic (it used to scale by 10 000 in floating point, which differed on about 2% of probabilities, e.g.0.00625gave0.0062where Python gives0.0063) and keeps the sign of a value that rounds to zero.predict_longemits Python's two warnings (window capped, stride reduced), and loading a checkpoint whose temperatures were clamped emits Python's warning, each with Python's text through thelogfacade (warn, targetlaya);logis a new dependency.A string in a checkpoint's
temperatureortemperature_by_optionsis read as Python'sfloat(str)reads it (surrounding whitespace, any-script digits, single underscores,inf/nan) before clamping, instead of being replaced by 1.0.A checkpoint whose temperature is a JSON integer too large for a float loads with the neutral 1.0 for it, and the load warning names the integer digit for digit, as Python 0.4.20's
clamp_temperatureandtemperature_rejectedanswer it; a float literal such as1e400is read as infinity and replaced by 1.0, as in Python.The bare JSON constants
NaN,Infinityand-Infinity, which Python'sjson.loadsreads as floats, are accepted inrl_agent_config.jsonandtokenizer_config.json(a temperature ofNaNclamps to 1.0 with Python's warning).Checkpoint JSON (
rl_agent_config.json,tokenizer_config.json) is read by a reader that follows Python'sjson.loads: integers keep their literal (the temperature warning prints0for-0and big integers as Python does),NaN/Infinity/-Infinityare numbers only in value position, and nothing is encoded in-band.serde_jsonnow usesfloat_roundtripso JSON floats equal Python'sfloat()bit for bit. An identical warning is emitted once per process, as Python's default filter does.Config loading matches Python's
ONNXAgent.__init__andjson.loadsfurther:option_layoutother thansequentialis refused with Python's message; an integer literal of more than 4300 digits is refused with Python'sValueErrortext;max_len/head_max_lenreadtrue/falseas 1/0 and-0as 0, and a float, string,null, list or object is an error (Python fails at the first prediction; this port refuses the config when it is loaded, with the sameTypeErrortext,head_max_lenfirst). A negative value is refused too, since Python slices from the end for it. The JSON reader no longer re-validates the rest of the document for every character of a string.Config JSON errors read like Python's
JSONDecodeError(Expecting ',' delimiter: line 1 column 7 (char 6), checked on 1,500 generated malformed texts), a file is read in text mode (\r\nbecomes\nfirst), andtemperature/temperature_by_optionsof the wrong JSON type fail while loading with Python'sTypeError/AttributeErrortext (a string iterates its characters, a dict its keys).A
temperaturelist shorter than three loads, andpredictfails withIndexError: list index out of rangefor a question whose type index it lacks (choice 0, score 1, noul 2), as Python does when it evaluatesself.temperature[qtype](even when atemperature_by_optionsbucket matches); entries past the third are ignored.LangGuesscallbacks receive the state as Python passes it (GuessInput::Value, or the original bytes asGuessInput::BytesfromLayaRouter::route_bytes).Email cleaning matches Python's
rewhere theregexcrate differs:\bfollows Python's\w(superscripts and otherNodigits are word characters; combining marks, joiners and connector punctuation other than_are not) andre.Ifolds U+0130 and U+0131 toi.Added
LanguageDetection::{state_text_bytes, analyse_bytes, is_english_bytes}andLayaRouter::route_bytes: Python'slang/routeraccept abytesstate, decode it as strict UTF-8 and read invalid UTF-8 as no text.LayaRouterOptions::max_loadeddefaults to 2, as Python'sRouter(max_loaded=2)does, so a router that alternates two languages no longer reloads a checkpoint on every request.Language detection follows Python 0.4.11: the line-by-line and field-by-field scan of a state that reads English as a whole, the Swedish and Nordic rules (Swedish, Banglish and Azerbaijani word lists, short Swedish requests, the Swedish-Danish overlap marker), the English rescue of a single accented loanword, acronym-shaped and all-caps lines no longer named foreign, identifier and code lines skipped, fullwidth and IPA letters counted as Latin (cutoff U+02B0) and letters no script range claims counted as "other". Python's
\w,\s,isupperandunicodedata.combiningare reproduced character for character rather than through theregexcrate.A
langofC,POSIX,C.UTF-8,und,zxx,mulor a blank string no longer pins a checkpoint: it falls through tolang_guessand detection.en_USanden_US.UTF-8now read as English instead of routing to the multilingual checkpoint.Routing reasons quote caller text like Python's
%r, escaping non-printable characters.Email cleaning follows Python 0.4.11: nine quote-header patterns (Portuguese, Spanish and French attributions that carry a date, Outlook headers in four languages, bare
From:/De:headers confirmed by theSent:/Enviado:/Date:line after them), the wrapped-attribution tail, Portuguese/Spanish/French sign-offs, the English sign-off name rule, device and mail-app footers, and Portuguese/Spanish/French disclaimers. A bareFrom:line that opens a sentence is kept.Language detection follows Python 0.4.14: the English stop list counts question words, auxiliaries, pronouns and connectives (116 words, up from 34), so a short request such as "Why hasn't my card been delivered?" is read as English instead of staying undecided and taking the multilingual checkpoint. The opt-in
explaindiagnostics Python added in 0.4.14 are not ported.Dependency:
unicode-normalization(canonical combining classes forunicodedata.combining).
Known gaps
- Language detection, routing and email cleaning follow the regenerated Python 0.4.14 probes; a later Python change needs the probes regenerated.
- Not yet ported: histogram-binning calibration and per-call hooks.