Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Laya for .NET
Laya's System 1 decision engine for .NET. This is a .NET port of the
Laya Python SDK (the laya package on PyPI, from this repository). It reproduces
the Python inference path (the same tokenization, sequence building, calibration and answers),
verified against golden outputs recorded from the Python code (tests/Laya.Tests/golden), but
runs on ONNX Runtime instead of PyTorch, so no Python process runs
at runtime. What is not ported yet is listed under Not in v1.
You ask typed questions (choice, score, noul) about any state (text, an email, a ticket or a
JSON-shaped object), and they are answered in one non-autoregressive forward pass with
calibrated probabilities. It generates no text, so there is nothing to parse and nothing to
hallucinate.
| Package | Laya.Onnx 0.1.0 (namespace Laya), versioned separately from the Python package |
| Target | .NET 10 (net10.0) |
| Checkpoints | multilingual (mmBERT-base, 322M, 1024-token, 100+ languages), english (ModernBERT-large, 421M, 512-token, English-optimized), typed-decisions (ModernBERT-large, 421M, 1024-token, fine-tuned on four typed-decisions workflows). Default is multilingual. |
| Model format | ONNX (opset 18, weights in an external-data sidecar), exported from the official PyTorch checkpoints with tools/export_onnx.py. See MODELS.md. |
| Runtime | Microsoft.ML.OnnxRuntime 1.30 (CPU by default; CUDA and DirectML execution providers with the matching ORT package) |
| Tokenizer | Tokenizers.DotNet, a binding to the same Rust HuggingFace tokenizers library Python uses. Its token streams match Python's because it runs the same code, not a reimplementation. |
| Native RIDs | win-x64, linux-x64, linux-arm64, osx-x64, osx-arm64 |
Contents
- Installation
- Getting the model
- Quickstart
- Writing questions
- Supplying state
- Long prediction
- Batch prediction
- Reading results
- Confidence gating
- Built-in presets
- Hosting: lifetime, threading, DI
- Routing, language detection, email and shortlist
- API reference
- Errors
- Performance
- Tests
- Not in v1
- Known gaps
Installation
The package is published on NuGet as Laya.Onnx:
dotnet add package Laya.Onnx --version 0.1.0
Or reference the project from a clone of this repository:
dotnet add reference path/to/laya-dotnet/src/Laya/Laya.csproj
The package and assembly are named Laya.Onnx, but everything lives in the Laya namespace. The tokenizer abstraction is in Laya.Tokenization.
using Laya;
Getting the model
This SDK runs ONNX models (opset 18), in either of two artifact layouts. The
fused layout — this repo's own export_onnx.py — is a model.onnx graph with its weights in
a model.onnx.data sidecar beside it, plus rl_agent_config.json and tokenizer/. The
split layout — produced by laya-ts's exporter — is an encoder.onnx and head.onnx
pair, each with its own .data sidecar when that graph's weights exceed the protobuf limit,
plus the same rl_agent_config.json and a root-level tokenizer.json. No Python is needed at
runtime for either; Python (or, for the split layout, laya-ts) is used once, to export the
model from the published PyTorch checkpoint.
The engine auto-detects which layout a directory holds — nothing in the API asks you to choose
one. Point it at a checkpoint directory with LayaEngine.FromDirectory(...),
LayaOptions.ModelDirectory or LAYA_ONNX_ROOT, as in the Quickstart.
See MODELS.md for exporting the ONNX model step by step (both layouts), where the engine looks for it, and downloading.
Quickstart
using Laya;
using var engine = LayaEngine.FromDirectory(@"C:\models\laya-multilingual");
// 1. State: a string, or any dictionary/list shape. Insertion order is preserved.
var state = new Dictionary<string, object?>
{
["from"] = "[email protected]",
["subject"] = "Duplicate charge on invoice #4411",
["body"] = "Hi, we were billed twice for March. Please refund the duplicate today "
+ "or we will cancel our plan.",
};
// 2. Typed questions, keyed by an id you choose.
var questions = new QuestionSet
{
["department"] = Question.Choice(
"Which department should handle this request?",
("billing", "invoices, payments, refunds"),
("technical", "bugs, outages, system errors"),
("sales", "pricing, new contracts"),
("other", "everything else")),
["urgency"] = Question.Score(
"How urgent is this request?",
"not urgent", "soon", "critical deadline or blocking issue"),
["churn_risk"] = Question.Noul("Does the user threaten to cancel or leave?"),
["refund_requested"] = Question.Noul("Does the user explicitly request a refund?"),
};
// 3. One batch, one forward pass, every question answered.
LayaResult result = engine.Predict(state, questions);
Console.WriteLine(result["department"].AsChoice().Choice); // billing
Console.WriteLine(result["urgency"].AsScore().Score); // 1.8974
Console.WriteLine(result["churn_risk"].AsNoul().Value); // False
Console.WriteLine(result["refund_requested"].AsNoul().Value); // True
// 4. The same questions work in any language, with no router.
var hindi = engine.Predict(
new Dictionary<string, object?> { ["body"] = "मार्च का बिल दो बार लिया गया, कृपया रिफंड करें।" },
questions);
Console.WriteLine(hindi["department"].AsChoice()); // billing (0.995 confidence)
The runnable version of this code is in samples/Laya.Sample:
dotnet run --project samples/Laya.Sample -- --model-dir "<artifact dir>"
Writing questions
There are three primitives. Each one is built with a static factory on Question.
| Primitive | Factory | Answer type | Output | Typical uses |
|---|---|---|---|---|
choice |
Question.Choice(...) |
ChoiceAnswer |
top label, probability per option, confidence | department routing, intent, topic |
score |
Question.Score(...) |
ScoreAnswer |
expected level on an ordinal scale, distribution, confidence | urgency, frustration, harm severity |
noul |
Question.Noul(...) |
NoulAnswer |
calibrated P(true) from 0.0 to 1.0 | phishing, spam, jailbreak, churn risk |
Choice
// Labels with descriptions: (label, description) tuples.
Question.Choice("Which team should handle this?",
("billing", "invoices, payments, refunds"),
("technical", "bugs, outages"),
("other", null)); // null or "" means no description
// Bare labels, with no descriptions.
Question.Choice("What is `prompt` about?", "coding", "writing", "other");
// Any ordered sequence of label/description pairs, for example built at runtime.
IEnumerable<KeyValuePair<string, object?>> options = LoadLabelsFromDb();
Question.Choice("Which intent is this?", options);
- Labels are positional. The model scores one marker token per option, in order, so the order
you declare options in is the order they are scored in. Every collection in the SDK preserves
insertion order for this reason. Don't pass options through a plain
Dictionary<,>, because its enumeration order is not guaranteed. - A duplicate label throws
ArgumentException. Python would silently merge duplicates into one option. An empty option list also throws. - Descriptions are
object?. Strings are used as written. Any other value (a number, a list, a dictionary) is rendered as JSON.0andfalseare real descriptions and are rendered. - All options share a fixed head budget (
Config.HeadMaxLen: 256 tokens onmultilingualandtyped-decisions, 192 onenglish). Each option is capped at 48 tokens, and with many options each one is squeezed further. If the options can't fit at all,Predictthrows instead of answering a truncated question. Past about 20 options, labels start to blur together. Split the question into a coarse one and a fine one.
Score
// Levels from lowest to highest. The answer is an expected level index: 0 .. levels-1.
Question.Score("How frustrated does the customer sound?",
"calm and neutral",
"concerned but civil",
"clearly annoyed",
"very angry or using strong language");
// Or from any sequence.
Question.Score("How severe is this?", severityLevels);
Levels are object?. They are rendered as level 0: …, level 1: … and so on. At least one level
is required.
Noul (yes/no)
Question.Noul("Does the user explicitly request a refund?");
// Optionally replace the default "no, the statement does not hold" /
// "yes, the statement holds" wording for each side.
Question.Noul(
"Is this email a phishing or scam attempt?",
ifFalse: "a legitimate email",
ifTrue: "phishing, scam, or fraud");
Instructions
instructions is object, not string. A string is used exactly as written, which is what you
normally want. Any other value is serialized as JSON with non-ASCII characters kept as written,
matching the Python SDK's behavior with non-string instructions. Refer to state fields in backticks
(`body`), as the presets do.
QuestionSet
QuestionSet is an insertion-ordered map from question id to Question. Answers come back in the
same order.
var questions = new QuestionSet
{
["spam"] = Question.Noul("Is this spam?"), // indexer: add, or replace in place
};
questions.Add("urgency", Question.Score("How urgent?", "low", "high")); // throws on a duplicate id
questions["spam"] = Question.Noul("Is this bulk marketing?"); // keeps its position
questions.Count; // 2
questions.Ids; // ["spam", "urgency"]
questions.ContainsId("spam"); // true
// Build one from any ordered sequence, for example to combine presets.
var combined = new QuestionSet(LayaPresets.Guard().Concat(LayaPresets.Moderation()));
Ids must be non-empty. Re-assigning an existing id with the indexer replaces the question and keeps
its position. Add throws instead.
Supplying state
Predict(object? state, QuestionSet questions) accepts:
| State | How it is rendered |
|---|---|
string |
used as written |
IDictionary, IEnumerable<KeyValuePair<string, object?>> |
a JSON object, in the dictionary's enumeration order |
IList (arrays, List<T>, conversation turns) and tuples (ValueTuple, Tuple) |
a JSON array; only an IList is left-truncated when over budget, as Python truncates a list (a tuple keeps its start) |
a set, a Queue<T>, a LINQ iterator or any other non-list enumerable |
not accepted: serializing it throws JsonException, as Python's json.dumps rejects a set or a generator. Pass ToList() / ToArray() |
null, bool, integer types, BigInteger, double, float, decimal |
the JSON scalar |
Serialization matches Python's json.dumps byte for byte (", " / ": " separators, float
formatting like Python's repr, non-ASCII kept as written). This makes the prompt identical to the
one the Python SDK builds. Nesting is allowed.
engine.Predict("My payment failed twice", questions); // plain text
engine.Predict(new Dictionary<string, object?> // an object
{
["subject"] = "Refund",
["amount"] = 42.5,
["tags"] = new[] { "billing", "vip" },
}, questions);
engine.Predict(new[] // a conversation
{
new Dictionary<string, object?> { ["role"] = "user", ["content"] = "Cancel my plan" },
new Dictionary<string, object?> { ["role"] = "agent", ["content"] = "May I ask why?" },
}, questions);
- POCOs and anonymous types are not supported. A value with no JSON equivalent throws
System.Text.Json.JsonException, as Python raisesTypeError. Convert records to a dictionary first. Circular references also throw. - Use an insertion-ordered dictionary (
Dictionary<,>built by adding keys, or aListofKeyValuePair) when key order matters. The serialized order is part of the prompt. - A long state is truncated from the end. The state gets whatever the question head leaves of
Config.MaxLen(1024 tokens onmultilingualandtyped-decisions, 512 onenglish). Put the most important fields first. - A literal
<mask>anywhere in state, instructions or options is replaced with a space, so user input can't forge a marker token.
Long prediction
Predict keeps only the state tokens that fit one sequence. To scan an entire long document,
use the port of Python ONNXAgent.predict_long (there is no long-prediction API in laya-ts):
LayaResult result = engine.PredictLong(document, questions, batchSize: 2);
Console.WriteLine(result.Usage.Windows);
if (result["churn_risk"].Window is { } deciding)
Console.WriteLine($"window {deciding.Index}: tokens {deciding.TokenStart}..{deciding.TokenEnd}");
// The router routes the original document once, then scans it on that checkpoint.
using var router = new LayaRouter();
LayaResult routed = router.PredictLong(document, questions, model: "english", batchSize: 2);
windowis a state-token count, defaulting tomax(64, max_len - head_max_len - 8), capped at the smallest state budget left by the questions.stridedefaults to half the effective window (at least 1). Nonpositive values select the defaults, as in Python.- Tokens are decoded back to text and scored as ordinary states, with the same overlapping
windows and re-tokenization as Python.
batchSizecaps windows per forward pass; without a positive cap they share a run, except for Python's automatic cap when a small window multiplies the batch by more than 2x. noultakes the greatest rounded P(true).choiceandscoretake the greatest roundedAnswerConfidence. Ties use the first window. The complete deciding answer (including probabilities, score legend and action) is preserved; it is not a calibrated probability for the whole document.Answer.Windowreports zero-based index, start/end offsets into the original tokenized state (end exclusive), and total window count. A state that fits the room the questions leave (or, with an explicitwindow, that window) is scored unchanged by onePredictpass, withUsage.Windows == 1and no answer attribution.- Scan usage sums
InputTokens,StateTokens,StateTokensDroppedandTruncated(a window count, including 0/1 for a single window).TruncatedQuestionsis the last window's list;Optionsmerges collapsed-option records by question id. These additional diagnostics are populated byPredict,PredictBatchandPredictLong. aggregateaccepts only"auto". Null state, unsupported aggregation, no state room, or a gap-causing stride throwsArgumentException(ArgumentNullExceptionfor null state). A stride valid for a requested window that gets clamped is reduced to half its new size. Window/stride clamps emitSystem.Diagnostics.Tracewarnings; install a listener to see them.- The router's usual
model,taskandlanghints select a checkpoint. Attached predictors must implementILayaLongPredictor; otherwise the router throwsNotSupportedException. Language-temperature and hook arguments remain unported.
The .NET adaptations are typed records instead of Python dictionaries, trace warnings instead of
RuntimeWarning, and .NET argument/capability exceptions. The existing "laya-rl-agent" model
identifier is kept, rather than Python ONNXAgent's "laya-rl-agent-onnx" suffix.
Batch prediction
PredictBatch answers the same questions about many states, sharing forward passes. It ports Python
ONNXAgent.predict_batch (batch_size, sort_by_length):
IReadOnlyList<LayaResult> results = engine.PredictBatch(
[ticket1, ticket2, ticket3], questions,
new LayaBatchOptions { BatchSize = 2, SortByLength = true }); // one result per state, input order
- One session run holds every state's question rows.
BatchSizecaps states per run;null,0or a negative value sends them all in one run, exactly as Python (which raises nothing for such values). SortByLengthgroups similarly sized encoded states inside windows of eight batches to reduce padding. It has an effect only with1 < BatchSize < states.Count; otherwise it is ignored, as in Python. Results keep input order. A different batch shape can move a prediction by a rounding step near a threshold.- An empty
stateslist returns an empty list and never looks at the questions. An emptyQuestionSetgives each state an empty answer set with zero usage. A null state throwsArgumentNullException(state must not be None; pass a string, dict, or list, the text Python'sTypeErrorcarries, asPredictandPredictLongnow do too); a question whose options cannot fit the head budget throwsArgumentException, asPredictdoes. States are encoded and run a window at a time, so an unanswerable state fails after the earlier windows have run, as in Python. - Each result carries the usage diagnostics Python's ONNX path reports (
StateTokens,StateTokensDropped,Truncated,TruncatedQuestions,Options), the same asPredictreports for a single state. LayaBatchOptionsis a class of init-only properties, so further per-call options extend it without changing a signature.ILayaBatchPredictoris a separate interface;ILayaPredictoris unchanged.
Router. LayaRouter.PredictBatch(requests, options) takes LayaBatchRequests (State, Questions,
and the optional routing overrides Model, Task, Lang, LangGuess, and the token budgets MaxLen,
HeadMaxLen), routes all of them first, groups them by routed checkpoint (first-seen order, so each checkpoint
loads at most once), splits each group by question schema (order-sensitive), token budgets and, for a checkpoint
that has per-language temperatures, language, so only requests
that agree on all of those share a pass, and returns results in request order, each with Routing. LayaRouter.RouteBatch(requests) returns just the RouteDecisions and loads nothing. A null
state or questions, or an unknown model, fails the whole call before anything loads. Attached predictors must
implement ILayaBatchPredictor, else NotSupportedException. Hooks and the diagnostics dictionary are not
ported yet.
Per-language temperatures and per-call options
Python's opt-in lang_temperatures, and the per-call lang, max_len and head_max_len. All are off by default,
so default predictions are unchanged.
using var engine = LayaEngine.Create(new LayaOptions
{
LangTemperatures = new Dictionary<string, LayaLangTemperature?>
{
["de"] = new() { Temperature = [1.4, 1.8, 2.0] }, // choice, score, noul
["fr"] = null, // the checkpoint's own temperature
["es"] = new() { TemperatureByOptions = new Dictionary<string, double> { ["choice:3-5"] = 2.5 } },
},
});
var result = engine.Predict(state, questions, new LayaPredictOptions { Lang = "de-AT", MaxLen = 256, HeadMaxLen = 96 });
engine.PredictBatch(states, questions, new LayaBatchOptions { Lang = "de" });
engine.PredictLong(document, questions, new LayaLongOptions { Lang = "de", BatchSize = 2 });
- The table (
LayaOptions.LangTemperatures) maps a language code toLayaLangTemperature(Temperature: exactly three values inchoice,score,noulorder, ornullto inherit the checkpoint's;TemperatureByOptions: bucket to value). It is resolved when the engine is created, as Python'sresolve_lang_temperaturesdoes: a key is matched on its part before the first-, lower-cased ("de-AT","DE"and"de"are one language,"de_DE"is not), the last of several keys for one language wins, every value is clamped to 0.5..5, and a value the clamp moved is reported once per language throughSystem.Diagnostics.Trace(the text Python warns with). ATemperaturethat is not three values throwsArgumentExceptioncarrying Python'sValueErrortext. A router hands the table to every engine it builds (LayaRouterOptions.EngineOptions). - A call that names a language the table holds (
LangonLayaPredictOptions/LayaBatchOptions/LayaLongOptions) replaces the checkpoint's per-type temperature and its per-bucket map with that language's; there is no fallback to the checkpoint's bucket map. A missing, empty or unknown language changes nothing. - On a router the language is the
langargument ofPredict/PredictLong(it also routes), else the language the router's own detection found (alang_guesshint or an explicitmodelleaves none), and it is handed to the engine. InPredictBatchthe request'sLang(or the detected one) joins the grouping key, as written ("de"and"de-AT"are separate passes), only for a checkpoint that has a table. MaxLen/HeadMaxLenreplace the checkpoint'smax_len/head_max_lenfor one call (LayaPredictOptions,LayaBatchOptions, and per request onLayaBatchRequest; not onPredictLong, which takes none in Python). Python checks nothing, and neither does this:0, a negative value or one too small for a question's option markers makes the call throw the sameArgumentExceptiona checkpoint with such a budget would (only N of its M option markers fit in max_len=...), while a small or negativeHeadMaxLensqueezes each option to 4 tokens and the instruction to 8. Booleans are not a .NETint:True/Falseare1/0.
Calibration files and histogram binning
Python's opt-in calibration= (load_calibration), off by default. A calibration file carries fitted temperatures and,
optionally, a histogram-binning map; this SDK reads and applies one, it does not fit or write them.
using var engine = LayaEngine.Create(new LayaOptions { Calibration = "calibration.json" });
{
"version": 1,
"temperature": [1.8, 2.2, 1.3],
"temperature_by_options": { "choice:3-5": 2.4, "noul:2": 0.8 },
"binning_map": {
"choice:3-5": { "bins": 4, "values": [0.47, 0.31, 0.92, 0.68] },
"noul:2": { "bins": 2, "values": [0.55, 0.85] }
}
}
- When it is read. Once, when the engine is created, after the checkpoint's own temperatures and the
LangTemperaturestable (as Python'sONNXAgent.__init__does); there is no per-call form. A null or empty path reads nothing. A router hands the path to every engine it builds. The file is read as strict UTF-8 and parsed likejson.load(NaNandInfinityare accepted), as the checkpoint config is. - Temperatures.
temperature(exactly three) andtemperature_by_optionsreplace the checkpoint's, both of them: a bucket the file does not name falls back to the file's per-type temperature, not the checkpoint's. Values are clamped to 0.5..5 and the ones the clamp moved are reported once throughSystem.Diagnostics.Trace, with Python's text.temperature_by_optionsmay be absent ornull. - Binning.
binning_mapmaps an option-count bucket (MinConfidence.BucketKeys) tobinsequal-width bins over [0, 1] and one recalibrated value per bin. It recalibratesAnswerConfidence, and only that: it is applied to the unroundedmax(p)after the temperature softmax, the bin isint(p * bins)clipped to0..bins-1, and the value found is rounded to four places.Confidence, the probabilities, a noul answer'sProbabilityand a score'sScoreare not binned. A bucket the map does not name keeps the temperature-scaled value. A call that names a language ofLangTemperaturesis not binned (that language's temperatures replace the file's). A file with nobinning_map, or anullone, installs none. - The gate and
PredictLongsee the binned value.MinConfidencecomparesAnswerConfidence, and a long scan keeps for a choice or score question the window with the greatestAnswerConfidence(a noul question the greatestProbability, which is not binned). - Refusals. A file Python's
apply_calibration_payloadrefuses makesLayaEngine.CreatethrowInvalidDataException, whose message is Python's exception type and text (ValueError: calibration JSON must contain temperature: [3 floats]): a payload that is not an object, aversionthat is not an integer (a missing ornullone is 1;2.0and"2"are 2,1.9is refused), atemperaturethat is not a list of three, atemperature_by_optionsorbinning_mapthat is not an object, and abinning_mapentry that is not an object with an integerbins >= 1andvaluesof exactly that length, each a finite number in [0, 1] (a bool, a string or anullis refused). A bad temperature is not refused but clamped; a bad binning value is, because there is nothing to fall back to.
Abstention: MinConfidence
Python's opt-in min_confidence, off by default. Pass a MinConfidence per call and every answer says
whether it cleared the gate; leave it unset and nothing is written (Abstention, AbstentionThreshold
are null, LowConfidence is false).
var result = engine.Predict(state, questions,
new LayaPredictOptions { MinConfidence = MinConfidence.Of(0.85) });
var answer = result["department"];
answer.LowConfidence; // true only when AnswerConfidence fell below the threshold
answer.Abstention; // "passed" | "abstained" | "unevaluated" (GateStates.*)
answer.AbstentionThreshold; // the threshold this answer was gated at
// One cut per option-count bucket (MinConfidence.BucketKeys, plus "default"):
var perBucket = MinConfidence.ForBuckets(new Dictionary<string, double>
{
["choice:2"] = 0.9, ["choice:3-5"] = 0.8, ["default"] = 0.6,
});
engine.PredictBatch(states, questions, new LayaBatchOptions { MinConfidence = perBucket });
router.Predict(state, questions, null, null, null, null, new LayaPredictOptions { MinConfidence = perBucket });
The gate compares AnswerConfidence, falling back to Confidence; an answer with neither usable (NaN, infinite)
is unevaluated, never passed. A map is read per answer bucket (choice:3-5, score:11+, noul:2, ...): a
bucket the map omits falls to its "default", then to 0.0 (ungated). A threshold outside [0, 1], NaN or infinite,
an empty map or an unknown key throws ArgumentException with Python's ValueError text, when the call receives it
(the engine checks it before anything else, the router after the state and questions and before routing or
loading). Answers are immutable; the gate returns new answers, so a reused result never carries a stale flag.
PredictLong takes no threshold, as Python's predict_long takes none. The router and LayaBatchOptions.MinConfidence
apply one batch-wide threshold; it is not part of the request grouping.
Answer.JevConfidence() is Python's jev_confidence: the confidence Jev would report for the answer's
(rounded) probabilities, null for noul answers and malformed probabilities.
It sums the probabilities as CPython 3.12 and later do (sum() of floats is compensated from 3.12 on), so its
last digits match laya on Python 3.12+, not on 3.10 or 3.11; the goldens are recorded with 3.12.
Reading results
LayaResult is indexed by question id. Ids returns the ids in the order they were asked.
LayaResult result = engine.Predict(state, questions);
result.Model; // "laya-rl-agent"
result.Usage.InputTokens; // non-padding tokens across the whole batch
result.Usage.OutputTokens; // always 0: nothing is generated
result.Count; // number of answers
foreach (var id in result.Ids)
{
Answer answer = result[id]; // throws KeyNotFoundException for an unknown id
answer.Type; // QuestionType.Choice / Score / Noul
answer.Confidence; // 0..1 (see below)
answer.AnswerConfidence; // 0..1, max(p): the same quantity on every question type
answer.Action.ActProbability; // the action head: P(act rather than escalate)
switch (answer)
{
case ChoiceAnswer choice:
choice.Choice; // the argmax label
choice["billing"]; // the probability of one label (KeyNotFoundException if absent)
choice.Probabilities; // IReadOnlyList<KeyValuePair<string, double>>, in option order
break;
case ScoreAnswer score:
score.Score; // expected level: sum of i * p[i], fractional, 0..Legend.Count-1
score.MostLikelyLevel; // argmax level index
score.Probabilities; // IReadOnlyList<double>, one per level
score.Legend; // rendered level descriptions (structured criteria become JSON text)
break;
case NoulAnswer noul:
noul.Probability; // P(true)
noul.Value; // Probability >= 0.5
break;
}
}
// Narrow without a switch. These throw InvalidOperationException if the type is wrong.
ChoiceAnswer dept = result["department"].AsChoice();
ScoreAnswer urgency = result["urgency"].AsScore();
NoulAnswer churn = result["churn_risk"].AsNoul();
// For ids that may be absent:
if (result.TryGetAnswer("refund_requested", out var refund)) { /* ... */ }
// Or enumerate (id, answer) pairs in asked order:
foreach (var (id, a) in result.Answers) Console.WriteLine($"{id}: {a}");
Confidence. For choice and score it is normalized Shannon entropy, 1 - H(p) / log k. A
single-option question gets 1.0. For noul it is max(p, 1 - p). Every probability, score and
confidence is rounded to 4 decimal places, as in Python.
AnswerConfidence. max(p) (clipped to [0, 1]), the same quantity on every question type —
unlike Confidence, whose formula differs between noul and the other two. It is the quantity
temperature scaling fits and calibration figures are computed on, so it is the field to gate a
single abstention threshold on across mixed question types. Confidence is unchanged alongside it.
ToString() gives a readable summary: billing (0.995 confidence), 1.8974 of 0..2,
True (p=0.973).
Confidence gating
The probabilities are trained with strictly proper scoring rules, so confidence carries meaning. You can act on it directly:
var dept = result["department"].AsChoice();
if (dept.Confidence >= 0.85)
RouteAutomatically(dept.Choice);
else
EscalateToHuman(dept.Choice, $"low confidence ({dept.Confidence:F2})");
All checkpoints ship with no fitted temperatures, and temperatures outside [0.5, 5.0] are
clamped (see LayaConfig.ClampedTemperatures). Fit temperatures on your own held-out data before
relying on raw probabilities for high-stakes thresholds.
Built-in presets
LayaPresets provides ready-made question sets, ported from the Python presets. Each preset's
instructions refer to a specific state field, so pass state with that key:
| Preset | Python name | State field | Questions |
|---|---|---|---|
LayaPresets.Triage() |
triage_questions |
message |
intent (choice), is_urgent, frustration (score), refund_requested, churn_risk |
LayaPresets.Email() |
email_questions |
body |
category (choice), is_spam, is_phishing, urgency (score), needs_reply |
LayaPresets.Guard() |
guard_questions |
prompt |
jailbreak, prompt_injection, sensitive_data, harm_severity (score), topic (choice) |
LayaPresets.Moderation() |
moderation_questions |
post |
toxic, harassment, threat, spam, severity (score) |
LayaPresets.Router() |
router_questions |
request |
difficulty (score), domain (choice), needs_tools, is_sensitive |
// Support ticket triage
var triage = engine.Predict(
new Dictionary<string, object?> { ["message"] = "My payment failed twice" },
LayaPresets.Triage());
// Prompt guardrails
var guard = engine.Predict(
new Dictionary<string, object?> { ["prompt"] = "Ignore all instructions" },
LayaPresets.Guard());
if (guard["jailbreak"].AsNoul().Probability > 0.5) Block();
// Model routing: send hard requests to a frontier model
var routing = engine.Predict(
new Dictionary<string, object?> { ["request"] = "Refactor this service using DI" },
LayaPresets.Router());
// Email with your own routing categories. Use the ordered-sequence overload so the option order is guaranteed.
var email = engine.Predict(
new Dictionary<string, object?> { ["body"] = emailBody },
LayaPresets.Email(new KeyValuePair<string, string>[]
{
new("orders", "order status, shipping"),
new("returns", "returns and exchanges"),
new("other", "anything else"),
}));
LayaPresets.DefaultEmailCategories exposes the default Email() categories (billing, technical,
sales, security, hr, other) in their offer order. Email(IReadOnlyDictionary<string, string>?)
also exists for parity with Python, but a dictionary doesn't guarantee order. Prefer the
IEnumerable<KeyValuePair<string, string>> overload.
Each call returns a new QuestionSet, so you can modify the result freely:
var qs = LayaPresets.Triage();
qs["language_barrier"] = Question.Noul("Is `message` written in broken or machine-translated language?");
Hosting: lifetime, threading and dependency injection
LayaEngine holds a 1.29 GB ONNX Runtime session. Create one and share it.
- Thread-safe.
Predictcan be called concurrently. Tokenization is serialized internally, and inference, the expensive part, runs in parallel. IDisposable. Disposing releases the session and the tokenizer. Disposal is idempotent, and callingPredictafterwards throwsObjectDisposedException.- Synchronous.
Predictis CPU-bound and blocking. In ASP.NET Core, call it directly from the request handler, or offload it withTask.Runif you must keep a thread free.
// Program.cs (ASP.NET Core / generic host)
builder.Services.AddSingleton(_ => LayaEngine.Create(new LayaOptions
{
ModelDirectory = builder.Configuration["Laya:ModelDirectory"],
IntraOpThreads = 4, // pin in containers with a CPU limit
}));
app.MapPost("/triage", (LayaEngine engine, TicketDto ticket) =>
{
var r = engine.Predict(new Dictionary<string, object?> { ["message"] = ticket.Text },
LayaPresets.Triage());
var intent = r["intent"].AsChoice();
return Results.Ok(new { intent = intent.Choice, intent.Confidence });
});
The container disposes the singleton when the host shuts down.
Routing, language detection, email and shortlist
Runnable end-to-end example: samples/Laya.Routing.
Language detection — no model, no network:
var analysis = LanguageDetection.Analyse("Mein Konto wurde zweimal belastet");
// analysis.Script == "latin", analysis.Language == "de", analysis.IsEnglish == false
LayaRouter picks a checkpoint per request from script/language detection (or an explicit
override), loading checkpoints lazily under an LRU cache:
using var router = new LayaRouter(new LayaRouterOptions { MaxLoaded = 2 });
var result = router.Predict(
new Dictionary<string, object?> { ["message"] = "I was charged twice" }, questions); // -> english
var deDe = router.Predict(
new Dictionary<string, object?> { ["message"] = "Mein Konto wurde zweimal belastet" }, questions); // -> multilingual
Console.WriteLine(result.Routing!.Reason); // "English Latin text"
router.PredictBatch(requests, options) and router.RouteBatch(requests) do the same for a mixed request list; see
Batch prediction. router.Route(state, questions) decides without loading or running anything — useful for logging
or testing the routing logic in isolation. Preload() builds every checkpoint up front for a
server or demo, so no request pays a cold load.
Language hints. Precedence is explicit model > task > detected typed-decisions workflow (opt-in) >
explicit lang > LangGuess > detection > default. An explicit lang decides only when its code names a
language (en, EN, en-US, en_US, en_US.UTF-8, pt-BR); blank, C, C.UTF-8, POSIX, und, zxx
and mul fall through to detection. A LangGuess is Python's lang_guess, a code or a callback of the state
(a callback returning null abstains, so a language-identification model can decline); pass one per call
to the Route/Predict/PredictLong overloads that end in LangGuess? langGuess, or install one for every
request with new LayaRouterOptions { LangGuess = LangGuess.FromCallback(state => Identify(state)) }.
Route reasons quote caller text with Python's repr, escaping non-printable characters.
Email cleaning strips quoted history, signatures and disclaimers before you ask any questions:
var cleaned = LayaEmail.CleanBody(rawBody);
var state = LayaEmail.State(subject, rawBody, sender: "[email protected]");
var answers = router.Predict(state, LayaPresets.Email());
LayaEmail.State(subject, body, sender, clean, extra, maxChars) takes Python's max_chars budget (default 3000).
It is a slice budget, applied as Python slices: 0 gives an empty body and a negative value cuts from the end
(5 * |maxChars| characters in all), so pass a number larger than the body to mean "no limit". It is ignored
when clean is false and is never added to the state as a field.
Embedding shortlist narrows a high-cardinality choice question before the model sees it, using a caller-supplied embedding function (bring your own bi-encoder — this SDK does not include one):
LayaEmbedFunction embed = texts => myBiEncoder.Embed(texts);
var result = LayaShortlist.Predict(router, state, questions, embed, k: 20);
var kept = result.Shortlist!["banking_intent"]; // kept.Labels, kept.Scores, kept.K, kept.N
API reference
LayaEngine : IDisposable
Answers typed questions about a piece of state in one forward pass. It is the .NET counterpart of
Python's laya.Agent.
| Member | Description |
|---|---|
static LayaEngine Create(LayaOptions? options = null) |
Finds the artifacts (see resolution order), loads the config, and builds the ONNX session and tokenizer. May download if AllowDownload is set. |
static LayaEngine FromDirectory(string directory) |
Shorthand for Create(new LayaOptions { ModelDirectory = directory }). |
LayaResult Predict(object? state, QuestionSet questions) |
Answers every question about state in a single batched inference call. See Supplying state. |
LayaResult PredictLong(object state, QuestionSet questions, int? window = null, int? stride = null, string aggregate = "auto", int? batchSize = null) |
Scans overlapping token windows. See Long prediction. |
IReadOnlyList<LayaResult> PredictBatch(IReadOnlyList<object?> states, QuestionSet questions, LayaBatchOptions? options = null) |
Answers the same questions over many states in shared passes. See Batch prediction. |
LayaConfig Config { get; } |
The loaded checkpoint's rl_agent_config.json. |
ModelArtifacts Artifacts { get; } |
The paths the artifacts were loaded from. |
void Dispose() |
Releases the session and tokenizer. Safe to call more than once. |
LayaCheckpoint (enum)
Selects which checkpoint to load.
| Value | Encoder | Parameters | max_len | head_max_len | Best for |
|---|---|---|---|---|---|
Multilingual (default) |
jhu-clsp/mmBERT-base |
322M | 1024 | 256 | Mixed-language or unknown-language traffic |
English |
answerdotai/ModernBERT-large |
421M | 512 | 192 | English-only workloads (accuracy collapses on non-Latin scripts) |
TypedDecisions |
answerdotai/ModernBERT-large |
421M | 1024 | 256 | Four specific typed-decisions workflows only |
LayaOptions
Controls where artifacts come from and how ONNX Runtime is configured. All properties are
settable ({ get; set; }).
| Property | Type | Default | Notes |
|---|---|---|---|
ModelDirectory |
string? |
null |
Highest-priority artifact location. When set, all other resolution steps are skipped. |
Checkpoint |
LayaCheckpoint |
Multilingual |
Which checkpoint to load. Derived subdirectory, HF subfolder and cache path all come from this value unless overridden by HuggingFaceSubfolder or ModelDirectory. |
ExecutionProvider |
LayaExecutionProvider |
Cpu |
Cuda and DirectMl need the matching ORT native package (Microsoft.ML.OnnxRuntime.Gpu / .DirectML) in place of the CPU one. |
IntraOpThreads |
int? |
ORT default | Threads within one operator. Worth pinning in a container with a CPU limit. |
InterOpThreads |
int? |
ORT default | Threads across independent operators. |
AllowDownload |
bool |
false |
Network access is never implicit. Needs the ONNX export to be published (see above). |
HuggingFaceRepo |
string |
convaiinnovations/laya |
Repository to download from. |
HuggingFaceSubfolder |
string |
derived from Checkpoint |
Subfolder in the repository. Prefer setting Checkpoint instead. |
HuggingFaceToken |
string? |
null → HF_TOKEN |
Bearer token for gated or private repos. "" disables auth. |
CacheDirectory |
string? |
per-user cache | Root that downloads land under: %LOCALAPPDATA%\laya\onnx or ~/.cache/laya/onnx. |
DownloadProgress |
IProgress<ArtifactDownloadProgress>? |
null |
Per-file progress sink. |
LayaExecutionProvider (enum)
Cpu (default), Cuda, DirectMl.
ArtifactDownloadProgress (record struct)
(string FileName, long BytesReceived, long? TotalBytes). TotalBytes is null when the server
omits Content-Length. Files are downloaded to a .part file and renamed when complete, so an
interrupted download never leaves a file that looks complete.
ModelArtifacts
Validated paths to every file the engine loads. You normally don't need this type directly.
LayaEngine.Create calls it for you. Use it to check or pre-fetch artifacts, for example in a
startup health check or a build step.
| Member | Description |
|---|---|
static ModelArtifacts Resolve(LayaOptions options) |
Resolves, validates and (if allowed) downloads synchronously. |
static Task<ModelArtifacts> ResolveAsync(LayaOptions options, CancellationToken ct = default) |
Async version. Supports cancelling the download. |
string Directory |
Absolute artifact directory. |
LayaArtifactLayout Layout |
Fused or Split — whichever layout was found in Directory. |
string? ModelPath |
…/model.onnx, or null when Layout is Split. |
string? EncoderPath |
…/encoder.onnx, or null when Layout is Fused. |
string? HeadPath |
…/head.onnx, or null when Layout is Fused. |
string ConfigPath |
…/rl_agent_config.json |
string TokenizerPath |
…/tokenizer/tokenizer.json (fused) or …/tokenizer.json (split). |
// Pre-fetch at deploy time, then start the engine from the resolved directory.
var artifacts = await ModelArtifacts.ResolveAsync(new LayaOptions { AllowDownload = true }, ct);
using var engine = LayaEngine.FromDirectory(artifacts.Directory);
LayaConfig
The contents of rl_agent_config.json. Missing keys fall back to the same defaults as Python's
cfg.get(...) (max_len 512, head_max_len 192, temperatures 1.0).
| Member | Description |
|---|---|
static LayaConfig Load(string path) |
Reads a config file. |
static LayaConfig Parse(string json) |
Parses config JSON. |
string? Encoder |
The encoder the checkpoint was trained on. For information only. |
int MaxLen |
Total sequence budget, in tokens (1024 on multilingual). |
int HeadMaxLen |
Budget for instructions plus all option fragments (256 on multilingual). |
IReadOnlyList<double> Temperature |
Per-question-type temperature [choice, score, noul], clamped; the entries of the config as written (a list shorter than three fails at the prediction that needs the missing one, as Python does). |
IReadOnlyDictionary<string, double> TemperatureByOptions |
Per-bucket temperature, clamped. Keys look like choice:3-5 (buckets 2, 3-5, 6-10, 11+). |
IReadOnlyList<double> TemperatureRaw / TemperatureByOptionsRaw |
What the checkpoint shipped, before clamping, as numbers (a value that is not a number reads NaN). |
IReadOnlyList<object?> TemperatureRawValues / IReadOnlyDictionary<string, object?> TemperatureByOptionsRawValues |
The same entries as the JSON held them (long, double, string, bool, null, list). |
IReadOnlyList<string> ClampedTemperatures |
Every temperature that was clamped into [0.5, 5.0]. A non-empty list means confidence from those buckets is uncalibrated. Entries read name=<Python repr of the value> -> <applied>, as Python's warning lists them. One Trace.TraceWarning listing every entry is raised at load time. |
foreach (var warning in engine.Config.ClampedTemperatures)
logger.LogWarning("Laya temperature clamped: {Bucket}", warning);
Question (abstract) and subclasses
| Member | Description |
|---|---|
static ChoiceQuestion Choice(object instructions, params (string Label, object? Description)[] options) |
Choice with described options. |
static ChoiceQuestion Choice(object instructions, params string[] labels) |
Choice with bare labels. |
static ChoiceQuestion Choice(object instructions, IEnumerable<KeyValuePair<string, object?>> options) |
Choice from any ordered sequence. |
static ScoreQuestion Score(object instructions, params object?[] levels) |
Score over levels, lowest first. |
static ScoreQuestion Score(object instructions, IEnumerable<object?> levels) |
Same, from a sequence. |
static NoulQuestion Noul(object instructions, object? ifFalse = null, object? ifTrue = null) |
Yes/no, with optional wording for each side. |
QuestionType Type |
Choice, Score or Noul. |
object Instructions |
The prompt, as supplied. |
ChoiceQuestion:IReadOnlyList<KeyValuePair<string, object?>> Options(index N is the label for logit N),IReadOnlyList<string> Labels.ScoreQuestion:IReadOnlyList<object?> Levels, lowest first.NoulQuestion:object? IfFalse,object? IfTrue(nullmeans the default wording).
Questions are immutable. You can reuse the same instance across calls, sets and threads.
QuestionType (enum)
Choice = 0, Score = 1, Noul = 2. The values match Python's QTYPES and the model's type
embedding.
QuestionSet : IEnumerable<KeyValuePair<string, Question>>
| Member | Description |
|---|---|
QuestionSet() |
An empty set. Supports collection-initializer syntax (["id"] = …). |
QuestionSet(IEnumerable<KeyValuePair<string, Question>> questions) |
Seeded from an ordered sequence. Later duplicates replace earlier ones in place. |
Question this[string id] { get; set; } |
Gets a question, or adds/replaces one. Replacing keeps the original position. |
void Add(string id, Question question) |
Appends a question. Throws ArgumentException on a duplicate id. |
bool ContainsId(string id) |
Whether the id is present. |
int Count |
Number of questions. |
IReadOnlyList<string> Ids |
Ids in insertion order. |
LayaResult
| Member | Description |
|---|---|
Answer this[string id] |
The answer for one question. Throws KeyNotFoundException for an unknown id. |
bool TryGetAnswer(string id, out Answer? answer) |
Non-throwing lookup. |
IEnumerable<KeyValuePair<string, Answer>> Answers |
Every answer, in the order asked. |
IReadOnlyList<string> Ids |
Question ids, in the order asked. |
int Count |
Number of answers. |
string Model |
"laya-rl-agent", the same value Python reports. |
Usage Usage |
Token accounting for the call. |
Answer (abstract) and subclasses
| Member | Description |
|---|---|
QuestionType Type |
Which question type produced this answer. |
double Confidence |
0..1. Entropy-based for choice and score, max(p, 1-p) for noul. |
double AnswerConfidence |
0..1. max(p), the same quantity on every question type. |
ActionInfo Action |
The action head's output. |
WindowInfo? Window |
Deciding window for a multi-window scan; otherwise null. Token end is exclusive. |
ChoiceAnswer AsChoice() / ScoreAnswer AsScore() / NoulAnswer AsNoul() |
Narrows to the typed answer. Throws InvalidOperationException on a mismatch. |
ChoiceAnswer:string Choice,IReadOnlyList<KeyValuePair<string, double>> Probabilities(in option order),double this[string label].ScoreAnswer:double Score(expected level),int MostLikelyLevel,IReadOnlyList<double> Probabilities,IReadOnlyList<object?> Legend.NoulAnswer:double Probability(P(true)),bool Value(Probability >= 0.5).
ActionInfo and Usage (record structs)
ActionInfo(double ActProbability): the model's own estimate of whether the answer can be acted on directly rather than escalated. It corresponds to Python'saction.act_probability.Usage(int InputTokens, int OutputTokens): non-padding input tokens across the batch.OutputTokensis always 0. Optional scan properties areWindows,StateTokens,StateTokensDropped,Truncated(allint?),TruncatedQuestions(IReadOnlyList<string>?) andOptions(IReadOnlyDictionary<string, OptionUsage>?).WindowInfo(int Index, int TokenStart, int TokenEnd, int Count)andOptionUsage(int Total, int Distinct, int? TokensPerOption)are immutable record structs. See Long prediction for their semantics.
LayaPresets (static)
Triage(), Email(IReadOnlyDictionary<string, string>? categories = null),
Email(IEnumerable<KeyValuePair<string, string>> categories), Guard(), Moderation(), Router(),
and DefaultEmailCategories. Each method returns a new QuestionSet. See
Built-in presets.
Lower-level building blocks
These types are public so that you can inspect the prompt the model sees, test calibration, or
plug in a different tokenizer. Predict uses all of them internally. Most applications never need
them.
| Type | Members | Purpose |
|---|---|---|
SequenceBuilder (static) |
TypeName(QuestionType), RenderInstructions(Question), RenderOptions(Question), Build(ILayaTokenizer, object? state, Question, int maxLen, int headMaxLen, bool truncateLeft = false) |
Builds [CLS] <type> question: … [SEP] [MASK] opt0 [MASK] opt1 … [SEP] state [SEP] and returns the marker positions. Useful for seeing exactly how an option is rendered. |
Calibration (static) |
TempMin (0.5), TempMax (5.0), ClampTemperature, TempBucket, ResolveTemperature, Probabilities, Softmax, ConfidenceFromProbs, AnswerConfidence, ExpectedScore, ArgMax, Round4 |
Logits to calibrated probabilities, confidence and scores. |
Collator (static) |
MinMarkers (2), Collate(IReadOnlyList<SequenceItem>, int padId) |
Pads a batch into the tensors the ONNX graph takes. |
CollatedBatch, SequenceItem |
tensor arrays and shapes | Collator input and output. |
PythonJson (static) |
State, Criterion, Instructions, Dumps(value, escapeNonAscii, fallbackToStr), Repr(double) |
Serialization that matches Python's json.dumps byte for byte, in the three dialects the prompt uses. |
Laya.Tokenization.ILayaTokenizer |
Encode(string), PadId, ClsId, SepId, MaskId, UnkId, MaskToken |
Tokenizer abstraction. Encode must add no special tokens. |
Laya.Tokenization.HfTokenizer |
HfTokenizer(string tokenizerJsonPath), IDisposable |
The shipped implementation. It removes the post_processor so no special tokens are added (caching the result next to the file, or in %TEMP% if read-only). Special-token ids are resolved from tokenizer_config.json + tokenizer.json at load time, so both mmBERT (<pad>=0, etc.) and ModernBERT ([PAD]=50283, etc.) tokenizers work. Throws if a token string is present in the config but absent from the id maps. |
// See the exact option text the model scores for a question.
foreach (var line in SequenceBuilder.RenderOptions(Question.Noul("Is this spam?")))
Console.WriteLine(line);
// false: no, the statement does not hold
// true: yes, the statement holds
Errors
| Exception | Thrown by | When |
|---|---|---|
DirectoryNotFoundException |
Create, ModelArtifacts.Resolve* |
No artifact in any location and AllowDownload is false. Also thrown when the given directory doesn't exist, or a download returns 404 / 401 / 403. The message lists what was tried and what to do. |
FileNotFoundException |
Create, ModelArtifacts.Resolve* |
The directory exists but is missing a file. Most often model.onnx.data was left behind when copying. |
InvalidOperationException |
Create |
tokenizer.json belongs to a different checkpoint (special-token id mismatch). |
OnnxRuntimeException |
Create |
ORT couldn't load the graph, or the requested execution provider isn't installed. |
ArgumentException |
Predict |
Empty QuestionSet, or a question whose options can't fit in HeadMaxLen. |
ArgumentException |
Question.*, QuestionSet |
Duplicate choice label or question id, no options or levels, empty id. |
System.Text.Json.JsonException |
Predict |
State contains a value with no JSON equivalent (a POCO, for example), or a circular reference. |
ObjectDisposedException |
Predict |
The engine has been disposed. |
KeyNotFoundException |
LayaResult[id], ChoiceAnswer[label] |
Unknown id or label. Use TryGetAnswer for optional ids. |
InvalidOperationException |
AsChoice / AsScore / AsNoul |
Answer type mismatch. |
Performance
Expect a few hundred milliseconds per call on CPU for a 322M-parameter encoder. That's around
190–230 ms warm on a recent desktop x64 core. The first call takes roughly an extra second while
the session warms up, so run one throwaway Predict at startup if first-request latency matters.
The 33 ms figure in the Python README was measured on a T4 GPU and doesn't apply to CPU inference.
Latency is per call, not per question. The architecture is built around batching every question
into one forward pass, so asking four questions together costs about the same as asking one. Put
related questions in one QuestionSet rather than calling Predict once per question.
Tests
The suite has three tiers, so it stays useful without the 1.3 GB artifact:
| Tier | Needs | Covers |
|---|---|---|
| 1 | nothing | PythonJson dialects, collation, calibration math, option-budget truncation, presets, artifact-layout detection (ModelArtifactsTests), tokenizer special-token fallback against synthetic fixtures (HfTokenizerFallbackTests) |
| tokenizer | tokenizer/tokenizer.json |
token ids and marker positions against recorded Python output |
| 2 | the full artifact | Predict against recorded Python answers, and a per-checkpoint shape sweep (72 configurations for multilingual and typed-decisions; 66 for english whose max_len=512 drops the 1024-token length) — run against both artifact layouts when both are present (onnx/<checkpoint> fused, onnx-split/<checkpoint> split); a checkpoint missing one layout only skips that layout's rows |
# Run with all three checkpoints, fused layout (recommended)
LAYA_ONNX_ROOT=/path/to/onnx dotnet test --solution Laya.slnx
# Backward compat: multilingual only
LAYA_ONNX_DIR=/path/to/onnx/multilingual dotnet test --solution Laya.slnx
The split-layout theories (...Split test names) look for <repo>/onnx-split/<checkpoint>
via a fixed walk-up from the test binary — there is no split-layout environment variable yet.
They skip per-checkpoint like the fused ones do when that directory is absent.
Checkpoints and layouts whose artifacts are missing are skipped, not failed. How many tests
run is not a fixed number — most of Tier 2 is parameterized per checkpoint, per shape, and now
per layout, so the count grows every time golden-answer coverage or shape-sweep coverage grows.
As a concrete snapshot from this repository: the 646 tests that need no ONNX artifact at all
(Tier 1 plus the tokenizer-lookup fixtures) always run with 0 failed and 0 skipped; a filtered
run excluding the five model-backed classes (PredictParityTests, ShapeSweepTests,
TokenizerParityTests, RouterEndToEndTests, ShortlistEndToEndTests) confirms this:
dotnet bin/Release/net10.0/Laya.Tests.dll \
-class- "Laya.Tests.PredictParityTests" \
-class- "Laya.Tests.ShapeSweepTests" \
-class- "Laya.Tests.TokenizerParityTests" \
-class- "Laya.Tests.RouterEndToEndTests" \
-class- "Laya.Tests.ShortlistEndToEndTests"
With every checkpoint present in both layouts, the full suite (including the five
model-backed classes above) is 1358 tests — up from a pre-split-layout snapshot of 1088;
the "621" this replaced was measured before several rounds of per-checkpoint parameterization and
had already gone stale independent of the split layout. Re-run the count yourself with
dotnet test --solution laya-dotnet/Laya.slnx -c Release once artifacts are in place, since this
number is expected to keep moving as coverage grows.
The parity vectors in tests/Laya.Tests/golden/<checkpoint>/ are recorded by
tools/dump_golden.py --checkpoint all, which runs the real laya.Agent.system_one over the
same ONNX graph. The vectors come from the shipping Python code, not from a second implementation.
The same golden answers back both the fused and the split parity theories: the split graph is the
fused graph cut into two ONNX files, not a different model, so one recording serves both.
Regenerating golden fixtures / CI
The goldens are recorded from the Python package, so they have to be re-recorded whenever a change
to laya/ moves something they capture (sequence building, option rendering, calibration,
language detection, ...). One command does the whole chain: it downloads each checkpoint at the
Hugging Face revision pinned in tools/regen_golden.py (HF_REVISION), exports the fused layout
(tools/export_onnx.py) and the split layout (laya-ts/scripts/export_onnx.py), then runs
tools/dump_golden.py and tools/dump_routing_golden.py into tests/Laya.Tests/golden/.
# once: CPU-only torch plus the toolchain the committed goldens were recorded with
pip install "torch==2.14.0" --index-url https://download.pytorch.org/whl/cpu
pip install -r laya-dotnet/tools/requirements-regen.txt
# from the repository root; all three checkpoints and the routing goldens
python laya-dotnet/tools/regen_golden.py
# or one piece, with the exported artifacts somewhere that has room (1-3 GB per checkpoint)
python laya-dotnet/tools/regen_golden.py --checkpoint english --artifacts-root /path/to/artifacts
python laya-dotnet/tools/regen_golden.py --checkpoint routing # no model needed
cd laya-dotnet # global.json selects the SDK and Microsoft.Testing.Platform runner
LAYA_ONNX_ROOT=/path/to/artifacts/onnx dotnet test --solution Laya.slnx -c Release
Exports are reused when they are already present and were made from the same inputs, so only the
first run is slow. The tests find the split layout only at <some ancestor of the test binary>/onnx-split,
so with a custom --artifacts-root use a directory junction/symlink from the repository's
onnx-split to the exported onnx-split directory, or export at the default repository root.
A diff in git diff -- laya-dotnet/tests/Laya.Tests/golden after
regenerating is expected whenever the Python behavior changed on purpose; review it, and commit it
together with the C# change that follows it.
.github/workflows/dotnet.yml runs the same chain on every push and pull request (there is no path
filter, so a change under laya/ triggers it): the build job re-records the routing goldens,
builds the whole solution with warnings as errors and runs the tests that need no model; one
parity job per checkpoint and layout (fused, split), and one for the router end-to-end tests
(which need english and multilingual together), exports that checkpoint, re-records its goldens
from the Python code at that commit and runs its model-backed classes against them.
LAYA_TEST_CHECKPOINTS selects the checkpoint theory rows and LAYA_TEST_LAYOUTS (fused,
split, or both when unset) the layout rows of the theories that have one per layout; the
split-only tests carry [Trait("Layout", "split-only")] and the layout-parameterised ones
Layout=any, which the split cell selects with --filter-trait (ordinary local runs still cover
everything). Every job then runs
tools/check_test_skips.py, because dotnet test reports skipped tests as success: it fails the
run when any test that should have run was skipped, and tolerates only the skips for checkpoints
the job does not export. The job logs print git diff --stat of the regenerated goldens against
the committed ones; that is informational (the C# tests and their tolerances are the gate).
Exports are cached on the pinned revision, the exporters, requirements-regen.txt and
laya/common.py; pinned Hugging Face downloads are cached separately.
The test project is an xUnit v3 application running on Microsoft.Testing.Platform. The .NET 10
SDK no longer runs these projects through VSTest. That's why laya-dotnet has a
global.json that selects the Microsoft.Testing.Platform runner, and why the solution is passed
with --solution rather than as a positional argument.
Not in v1
- Quantized artifacts.
- Async
Predict. embed_fn_from_agent(mean-pooling the checkpoint's own encoder into a shortlist embedder) —LayaShortlisttakes only a caller-supplied embedding function; see Routing, language detection, email and shortlist.
Known gaps
Default and long predictions are verified against goldens regenerated from Python
0.4.11, and routing against goldens from Python 0.4.12. Language detection uses current script and mixed-language rules; MixedSegment
identifies the foreign line or field that overrides a mostly-English state. Undetermined
Latin text uses the router's configured default. Email cleaning follows current request,
disclaimer, signature and reply-header rules, including Portuguese, Spanish and French markers.
- Opt-in features added in Python 0.3.21: abstention (
min_confidence) and per-language temperatures (lang_temperatures, with per-calllang) are ported (see "Abstention" and "Per-language temperatures" above). Calibration files and histogram binning (calibration=/load_calibration) are ported too (see "Calibration files and histogram binning" above); fitting and saving one (fit_binning,save_calibration) is not. - Other Python-only APIs are not ported: hooks, question option ordering and digest controls.
Batching is ported (
PredictBatch,LayaRouter.PredictBatch/RouteBatch) without hooks and the routerdiagnosticsdictionary. - Usage diagnostics: Python's
state_tokens,truncated,state_tokens_dropped,truncated_questionsandoptionsare reported byPredict,PredictBatchandPredictLong, as Python'ssystem_oneispredict_batch([state]).
Differences from Python
Where this SDK deliberately does not match the Python reference, so nobody has to find out from a failing load:
max_lenandhead_max_lenare checked when the config is loaded. An integer>= 0or a boolean (trueis 1,falseis 0, as in Python) is accepted. Any other value is refused byLayaConfig.Parse/LayaEngine.Createwith anInvalidDataException: a float, string,null, list or object gets theTypeErrortext Python raises at the first prediction (head first, thenmax_len), and a negative integer gets a message naming the field and value, because Python then slices from the end, which this port does not support. Python loads such a checkpoint and fails later. The related corner: a small fractionalhead_max_lenthat Python happens to survive (roughly up to 8 plus 4 per option) is refused here too.LangTemperaturesis a typed table. Python also refuses a key that is not a string, an entry ortemperature_by_optionsthat is not a mapping, and atemperaturethat is not a list; a typed table cannot hold those, so only the three-value check (and the inherited-list check) remains. A warning prints a float the way Python prints a float (99.0), where Python prints the99an integer was written as.- A router takes the language, and the token budgets, in the places Python does.
LayaPredictOptions.LangonLayaRouter.PredictandLayaBatchOptions.Lang/MaxLen/HeadMaxLenonLayaRouter.PredictBatchthrowArgumentException: Python'spredicthas the onelangargument, andpredict_batchhas no such arguments (it is aTypeErrorthere), only the per-request keys. - A version 2 calibration file's checkpoint-identity warning is not emitted. Python warns when the
model_id_or_path,subfolderorconfiga file records differs from the agent's, but itsmodel_id_or_pathis the string the caller passed (a PyTorch checkpoint path or Hub id), which this SDK has no equivalent of. Temperatures, binning and errors are unaffected, and the file loads as Python's does. - A calibration file's refusals are
InvalidDataException. Python raisesValueError(and anOverflowErrorfor an integer too large for a float); the message here is the exception's name and text, as for a malformed checkpoint config. The clamp warning is aRuntimeWarningin Python; it reachesTracehere as text, with no category, shown once per call site. - JSON nesting. Config and tokenizer JSON is read like
json.loads, with the nesting cut-off set to 2,997 levels, what CPython 3.12 measured from a shallow call. Python's real limit depends on the call depth at the point of loading. - Config and calibration files are decoded as strict UTF-8. Python opens them with the locale's encoding (cp1252 on a default Windows
install), which only differs for non-ASCII bytes; a BOM is rejected, as
json.loadsdoes.
License
Apache 2.0. Developed by Convai Innovations.