Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
docs(dotnet): add SDK README
kevin-gatimu6a2e022a2178
diff --git a/laya-dotnet/CHANGELOG.md b/laya-dotnet/CHANGELOG.mdnew file mode 100644index 00000000..3eb0df09--- /dev/null+++ b/laya-dotnet/CHANGELOG.md@@ -0,0 +1,27 @@+# Changelog++All notable changes to the `Laya.Onnx` NuGet package are documented here. The format follows+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses+[Semantic Versioning](https://semver.org/). Releases are tagged `laya-dotnet-v<version>`.++## [0.1.0] - unreleased++First release: a port of the Laya Python SDK, tested against laya (Python) 0.3.21.++### Added++- `LayaEngine`: `choice`, `score` and `noul` predictions over the exported ONNX checkpoints+ (english, multilingual, typed-decisions); all questions for a state run in one batched forward pass.+- Loads both ONNX layouts: the fused `model.onnx` and the split `encoder.onnx` + `head.onnx`.+- Answers report `confidence` and `answer_confidence` (added in Python 0.3.21).+- Python-compatible sequence building, JSON serialization and calibration, checked against goldens+ recorded from the Python SDK.+- Checkpoint router, language detection, email state and option shortlisting.+- Opt-in model download from Hugging Face.+- Samples: quickstart, routing and benchmark.++### Known gaps++- Language detection and email handling follow Python 0.3.6 (`lang.py`, `email.py`).+- Not yet ported from Python 0.3.21: abstention (`min_confidence`), `predict_long` and+ `lang_temperatures`.diff --git a/laya-dotnet/README.md b/laya-dotnet/README.mdnew file mode 100644index 00000000..e3c4585a--- /dev/null+++ b/laya-dotnet/README.md@@ -0,0 +1,835 @@+# Laya for .NET++**Laya's System 1 decision engine for .NET.** This is a .NET port of the+[Laya Python SDK](https://github.com/NandhaKishorM/laya/blob/main/README.md) (the `laya` package on PyPI, from this repository). It reproduces+the Python inference path (the same tokenization, sequence building, calibration and answers),+verified against golden outputs recorded from the Python code (`tests/Laya.Tests/golden`), but+runs on [ONNX Runtime](https://onnxruntime.ai/) instead of PyTorch, so **no Python process runs+at runtime**. What is not ported yet is listed under [Not in v1](#not-in-v1).++You ask typed questions (`choice`, `score`, `noul`) about any state (text, an email, a ticket or a+JSON-shaped object), and they are answered in **one non-autoregressive forward pass** with+calibrated probabilities. It generates no text, so there is nothing to parse and nothing to+hallucinate.++| | |+|---|---|+| Package | **[`Laya.Onnx`](https://www.nuget.org/packages/Laya.Onnx) 0.1.0** (namespace `Laya`), versioned separately from the Python package |+| Target | **.NET 10** (`net10.0`) |+| Checkpoints | **multilingual** (`mmBERT-base`, 322M, 1024-token, 100+ languages), **english** (`ModernBERT-large`, 421M, 512-token, English-optimized), **typed-decisions** (`ModernBERT-large`, 421M, 1024-token, fine-tuned on four typed-decisions workflows). Default is `multilingual`. |+| Model format | **ONNX** (opset 18, weights in an external-data sidecar), exported from the official PyTorch checkpoints with [`tools/export_onnx.py`](https://github.com/NandhaKishorM/laya/blob/main/laya-dotnet/tools/export_onnx.py). See [MODELS.md](https://github.com/NandhaKishorM/laya/blob/main/laya-dotnet/MODELS.md). |+| Runtime | `Microsoft.ML.OnnxRuntime` 1.30 (CPU by default; CUDA and DirectML execution providers with the matching ORT package) |+| Tokenizer | [`Tokenizers.DotNet`](https://www.nuget.org/packages/Tokenizers.DotNet), a binding to the same Rust HuggingFace `tokenizers` library Python uses. Its token streams match Python's because it runs the same code, not a reimplementation. |+| Native RIDs | `win-x64`, `linux-x64`, `linux-arm64`, `osx-x64`, `osx-arm64` |++---++## Contents++- [Installation](#installation)+- [Getting the model](#getting-the-model)+- [Quickstart](#quickstart)+- [Writing questions](#writing-questions)+- [Supplying state](#supplying-state)+- [Reading results](#reading-results)+- [Confidence gating](#confidence-gating)+- [Built-in presets](#built-in-presets)+- [Hosting: lifetime, threading, DI](#hosting-lifetime-threading-and-dependency-injection)+- [Routing, language detection, email and shortlist](#routing-language-detection-email-and-shortlist)+- [API reference](#api-reference)+- [Errors](#errors)+- [Performance](#performance)+- [Tests](#tests)+- [Not in v1](#not-in-v1)+- [Known gaps](#known-gaps)++---++## Installation++The package is published on NuGet as **`Laya.Onnx`**:++```bash+dotnet add package Laya.Onnx --version 0.1.0+```++Or reference the project from a clone of this repository:++```bash+dotnet add reference path/to/laya-dotnet/src/Laya/Laya.csproj+```++The package and assembly are named `Laya.Onnx`, but everything lives in the `Laya` namespace. The tokenizer abstraction is in `Laya.Tokenization`.++```csharp+using Laya;+```++---++## Getting the model++This SDK runs **ONNX** models (opset 18), in either of two artifact layouts. The+**fused** layout — this repo's own `export_onnx.py` — is a `model.onnx` graph with its weights in+a `model.onnx.data` sidecar beside it, plus `rl_agent_config.json` and `tokenizer/`. The+**split** layout — produced by `laya-ts`'s exporter — is a self-contained `encoder.onnx` and+`head.onnx` pair with no external-data sidecar, plus the same `rl_agent_config.json` and a+root-level `tokenizer.json`. No Python is needed at runtime for either; Python (or, for the split+layout, `laya-ts`) is used once, to export the model from the published PyTorch checkpoint.++The engine auto-detects which layout a directory holds — nothing in the API asks you to choose+one. Point it at a checkpoint directory with `LayaEngine.FromDirectory(...)`,+`LayaOptions.ModelDirectory` or `LAYA_ONNX_ROOT`, as in the [Quickstart](#quickstart).++See [MODELS.md](https://github.com/NandhaKishorM/laya/blob/main/laya-dotnet/MODELS.md) for exporting the ONNX model step by step (both layouts), where the+engine looks for it, and downloading.++---++## Quickstart++```csharp+using Laya;++using var engine = LayaEngine.FromDirectory(@"C:\models\laya-multilingual");++// 1. State: a string, or any dictionary/list shape. Insertion order is preserved.+var state = new Dictionary<string, object?>+{+ ["from"] = "[email protected]",+ ["subject"] = "Duplicate charge on invoice #4411",+ ["body"] = "Hi, we were billed twice for March. Please refund the duplicate today "+ + "or we will cancel our plan.",+};++// 2. Typed questions, keyed by an id you choose.+var questions = new QuestionSet+{+ ["department"] = Question.Choice(+ "Which department should handle this request?",+ ("billing", "invoices, payments, refunds"),+ ("technical", "bugs, outages, system errors"),+ ("sales", "pricing, new contracts"),+ ("other", "everything else")),+ ["urgency"] = Question.Score(+ "How urgent is this request?",+ "not urgent", "soon", "critical deadline or blocking issue"),+ ["churn_risk"] = Question.Noul("Does the user threaten to cancel or leave?"),+ ["refund_requested"] = Question.Noul("Does the user explicitly request a refund?"),+};++// 3. One batch, one forward pass, every question answered.+LayaResult result = engine.Predict(state, questions);++Console.WriteLine(result["department"].AsChoice().Choice); // billing+Console.WriteLine(result["urgency"].AsScore().Score); // 1.8974+Console.WriteLine(result["churn_risk"].AsNoul().Value); // False+Console.WriteLine(result["refund_requested"].AsNoul().Value); // True++// 4. The same questions work in any language, with no router.+var hindi = engine.Predict(+ new Dictionary<string, object?> { ["body"] = "मार्च का बिल दो बार लिया गया, कृपया रिफंड करें।" },+ questions);+Console.WriteLine(hindi["department"].AsChoice()); // billing (0.995 confidence)+```++The runnable version of this code is in [`samples/Laya.Sample`](https://github.com/NandhaKishorM/laya/blob/main/laya-dotnet/samples/Laya.Sample/README.md):++```bash+dotnet run --project samples/Laya.Sample -- --model-dir "<artifact dir>"+```++---++## Writing questions++There are three primitives. Each one is built with a static factory on `Question`.++| Primitive | Factory | Answer type | Output | Typical uses |+|---|---|---|---|---|+| **`choice`** | `Question.Choice(...)` | `ChoiceAnswer` | top label, probability per option, confidence | department routing, intent, topic |+| **`score`** | `Question.Score(...)` | `ScoreAnswer` | expected level on an ordinal scale, distribution, confidence | urgency, frustration, harm severity |+| **`noul`** | `Question.Noul(...)` | `NoulAnswer` | calibrated P(true) from 0.0 to 1.0 | phishing, spam, jailbreak, churn risk |++### Choice++```csharp+// Labels with descriptions: (label, description) tuples.+Question.Choice("Which team should handle this?",+ ("billing", "invoices, payments, refunds"),+ ("technical", "bugs, outages"),+ ("other", null)); // null or "" means no description++// Bare labels, with no descriptions.+Question.Choice("What is `prompt` about?", "coding", "writing", "other");++// Any ordered sequence of label/description pairs, for example built at runtime.+IEnumerable<KeyValuePair<string, object?>> options = LoadLabelsFromDb();+Question.Choice("Which intent is this?", options);+```++- **Labels are positional.** The model scores one marker token per option, in order, so the order+ you declare options in is the order they are scored in. Every collection in the SDK preserves+ insertion order for this reason. Don't pass options through a plain `Dictionary<,>`, because its+ enumeration order is not guaranteed.+- A duplicate label throws `ArgumentException`. Python would silently merge duplicates into one+ option. An empty option list also throws.+- Descriptions are `object?`. Strings are used as written. Any other value (a number, a list, a+ dictionary) is rendered as JSON. `0` and `false` are real descriptions and are rendered.+- All options share a fixed head budget (`Config.HeadMaxLen`: 256 tokens on `multilingual` and+ `typed-decisions`, 192 on `english`). Each option is capped at 48 tokens, and with many options+ each one is squeezed further. If the options can't fit at all, `Predict` throws instead of+ answering a truncated question. Past about 20 options, labels start to blur together. Split the+ question into a coarse one and a fine one.++### Score++```csharp+// Levels from lowest to highest. The answer is an expected level index: 0 .. levels-1.+Question.Score("How frustrated does the customer sound?",+ "calm and neutral",+ "concerned but civil",+ "clearly annoyed",+ "very angry or using strong language");++// Or from any sequence.+Question.Score("How severe is this?", severityLevels);+```++Levels are `object?`. They are rendered as `level 0: …`, `level 1: …` and so on. At least one level+is required.++### Noul (yes/no)++```csharp+Question.Noul("Does the user explicitly request a refund?");++// Optionally replace the default "no, the statement does not hold" /+// "yes, the statement holds" wording for each side.+Question.Noul(+ "Is this email a phishing or scam attempt?",+ ifFalse: "a legitimate email",+ ifTrue: "phishing, scam, or fraud");+```++### Instructions++`instructions` is `object`, not `string`. A string is used exactly as written, which is what you+normally want. Any other value is serialized as JSON with non-ASCII characters escaped, matching+the Python SDK's behavior with non-string instructions. Refer to state fields in backticks+(`` `body` ``), as the presets do.++### QuestionSet++`QuestionSet` is an insertion-ordered map from question id to `Question`. Answers come back in the+same order.++```csharp+var questions = new QuestionSet+{+ ["spam"] = Question.Noul("Is this spam?"), // indexer: add, or replace in place+};+questions.Add("urgency", Question.Score("How urgent?", "low", "high")); // throws on a duplicate id+questions["spam"] = Question.Noul("Is this bulk marketing?"); // keeps its position++questions.Count; // 2+questions.Ids; // ["spam", "urgency"]+questions.ContainsId("spam"); // true++// Build one from any ordered sequence, for example to combine presets.+var combined = new QuestionSet(LayaPresets.Guard().Concat(LayaPresets.Moderation()));+```++Ids must be non-empty. Re-assigning an existing id with the indexer replaces the question and keeps+its position. `Add` throws instead.++---++## Supplying state++`Predict(object? state, QuestionSet questions)` accepts:++| State | How it is rendered |+|---|---|+| `string` | used as written |+| `IDictionary`, `IEnumerable<KeyValuePair<string, object?>>` | a JSON object, in the dictionary's enumeration order |+| any other `IEnumerable` (arrays, lists, conversation turns) | a JSON array |+| `null`, `bool`, integer types, `BigInteger`, `double`, `float`, `decimal` | the JSON scalar |++Serialization matches Python's `json.dumps` byte for byte (`", "` / `": "` separators, float+formatting like Python's `repr`, non-ASCII kept as written). This makes the prompt identical to the+one the Python SDK builds. Nesting is allowed.++```csharp+engine.Predict("My payment failed twice", questions); // plain text++engine.Predict(new Dictionary<string, object?> // an object+{+ ["subject"] = "Refund",+ ["amount"] = 42.5,+ ["tags"] = new[] { "billing", "vip" },+}, questions);++engine.Predict(new[] // a conversation+{+ new Dictionary<string, object?> { ["role"] = "user", ["content"] = "Cancel my plan" },+ new Dictionary<string, object?> { ["role"] = "agent", ["content"] = "May I ask why?" },+}, questions);+```++- **POCOs and anonymous types are not supported.** A value with no JSON equivalent throws+ `System.Text.Json.JsonException`, as Python raises `TypeError`. Convert records to a dictionary+ first. Circular references also throw.+- Use an insertion-ordered dictionary (`Dictionary<,>` built by adding keys, or a `List` of+ `KeyValuePair`) when key order matters. The serialized order is part of the prompt.+- A long state is **truncated from the end**. The state gets whatever the question head leaves of+ `Config.MaxLen` (1024 tokens on `multilingual` and `typed-decisions`, 512 on `english`). Put the+ most important fields first.+- A literal `<mask>` anywhere in state, instructions or options is replaced with a space, so user+ input can't forge a marker token.++---++## Reading results++`LayaResult` is indexed by question id. `Ids` returns the ids in the order they were asked.++```csharp+LayaResult result = engine.Predict(state, questions);++result.Model; // "laya-rl-agent"+result.Usage.InputTokens; // non-padding tokens across the whole batch+result.Usage.OutputTokens; // always 0: nothing is generated+result.Count; // number of answers++foreach (var id in result.Ids)+{+ Answer answer = result[id]; // throws KeyNotFoundException for an unknown id+ answer.Type; // QuestionType.Choice / Score / Noul+ answer.Confidence; // 0..1 (see below)+ answer.AnswerConfidence; // 0..1, max(p): the same quantity on every question type+ answer.Action.ActProbability; // the action head: P(act rather than escalate)++ switch (answer)+ {+ case ChoiceAnswer choice:+ choice.Choice; // the argmax label+ choice["billing"]; // the probability of one label (KeyNotFoundException if absent)+ choice.Probabilities; // IReadOnlyList<KeyValuePair<string, double>>, in option order+ break;++ case ScoreAnswer score:+ score.Score; // expected level: sum of i * p[i], fractional, 0..Legend.Count-1+ score.MostLikelyLevel; // argmax level index+ score.Probabilities; // IReadOnlyList<double>, one per level+ score.Legend; // the level descriptions you supplied+ break;++ case NoulAnswer noul:+ noul.Probability; // P(true)+ noul.Value; // Probability >= 0.5+ break;+ }+}++// Narrow without a switch. These throw InvalidOperationException if the type is wrong.+ChoiceAnswer dept = result["department"].AsChoice();+ScoreAnswer urgency = result["urgency"].AsScore();+NoulAnswer churn = result["churn_risk"].AsNoul();++// For ids that may be absent:+if (result.TryGetAnswer("refund_requested", out var refund)) { /* ... */ }++// Or enumerate (id, answer) pairs in asked order:+foreach (var (id, a) in result.Answers) Console.WriteLine($"{id}: {a}");+```++**Confidence.** For `choice` and `score` it is normalized Shannon entropy, `1 - H(p) / log k`. A+single-option question gets 1.0. For `noul` it is `max(p, 1 - p)`. Every probability, score and+confidence is rounded to 4 decimal places, as in Python.++**`AnswerConfidence`.** `max(p)` (clipped to [0, 1]), the same quantity on every question type —+unlike `Confidence`, whose formula differs between noul and the other two. It is the quantity+temperature scaling fits and calibration figures are computed on, so it is the field to gate a+single abstention threshold on across mixed question types. `Confidence` is unchanged alongside it.++**`ToString()`** gives a readable summary: `billing (0.995 confidence)`, `1.8974 of 0..2`,+`True (p=0.973)`.++---++## Confidence gating++The probabilities are trained with strictly proper scoring rules, so confidence carries meaning.+You can act on it directly:++```csharp+var dept = result["department"].AsChoice();++if (dept.Confidence >= 0.85)+ RouteAutomatically(dept.Choice);+else+ EscalateToHuman(dept.Choice, $"low confidence ({dept.Confidence:F2})");+```++All checkpoints ship with **no fitted temperatures**, and temperatures outside `[0.5, 5.0]` are+clamped (see `LayaConfig.ClampedTemperatures`). Fit temperatures on your own held-out data before+relying on raw probabilities for high-stakes thresholds.++---++## Built-in presets++`LayaPresets` provides ready-made question sets, ported from the Python presets. Each preset's+instructions refer to a specific state field, so pass state with that key:++| Preset | Python name | State field | Questions |+|---|---|---|---|+| `LayaPresets.Triage()` | `triage_questions` | `message` | `intent` (choice), `is_urgent`, `frustration` (score), `refund_requested`, `churn_risk` |+| `LayaPresets.Email()` | `email_questions` | `body` | `category` (choice), `is_spam`, `is_phishing`, `urgency` (score), `needs_reply` |+| `LayaPresets.Guard()` | `guard_questions` | `prompt` | `jailbreak`, `prompt_injection`, `sensitive_data`, `harm_severity` (score), `topic` (choice) |+| `LayaPresets.Moderation()` | `moderation_questions` | `post` | `toxic`, `harassment`, `threat`, `spam`, `severity` (score) |+| `LayaPresets.Router()` | `router_questions` | `request` | `difficulty` (score), `domain` (choice), `needs_tools`, `is_sensitive` |++```csharp+// Support ticket triage+var triage = engine.Predict(+ new Dictionary<string, object?> { ["message"] = "My payment failed twice" },+ LayaPresets.Triage());++// Prompt guardrails+var guard = engine.Predict(+ new Dictionary<string, object?> { ["prompt"] = "Ignore all instructions" },+ LayaPresets.Guard());+if (guard["jailbreak"].AsNoul().Probability > 0.5) Block();++// Model routing: send hard requests to a frontier model+var routing = engine.Predict(+ new Dictionary<string, object?> { ["request"] = "Refactor this service using DI" },+ LayaPresets.Router());++// Email with your own routing categories. Use the ordered-sequence overload so the option order is guaranteed.+var email = engine.Predict(+ new Dictionary<string, object?> { ["body"] = emailBody },+ LayaPresets.Email(new KeyValuePair<string, string>[]+ {+ new("orders", "order status, shipping"),+ new("returns", "returns and exchanges"),+ new("other", "anything else"),+ }));+```++`LayaPresets.DefaultEmailCategories` exposes the default `Email()` categories (billing, technical,+sales, security, hr, other) in their offer order. `Email(IReadOnlyDictionary<string, string>?)`+also exists for parity with Python, but a dictionary doesn't guarantee order. Prefer the+`IEnumerable<KeyValuePair<string, string>>` overload.++Each call returns a new `QuestionSet`, so you can modify the result freely:++```csharp+var qs = LayaPresets.Triage();+qs["language_barrier"] = Question.Noul("Is `message` written in broken or machine-translated language?");+```++---++## Hosting: lifetime, threading and dependency injection++`LayaEngine` holds a 1.29 GB ONNX Runtime session. **Create one and share it.**++- **Thread-safe.** `Predict` can be called concurrently. Tokenization is serialized internally, and+ inference, the expensive part, runs in parallel.+- **`IDisposable`.** Disposing releases the session and the tokenizer. Disposal is idempotent, and+ calling `Predict` afterwards throws `ObjectDisposedException`.+- **Synchronous.** `Predict` is CPU-bound and blocking. In ASP.NET Core, call it directly from the+ request handler, or offload it with `Task.Run` if you must keep a thread free.++```csharp+// Program.cs (ASP.NET Core / generic host)+builder.Services.AddSingleton(_ => LayaEngine.Create(new LayaOptions+{+ ModelDirectory = builder.Configuration["Laya:ModelDirectory"],+ IntraOpThreads = 4, // pin in containers with a CPU limit+}));++app.MapPost("/triage", (LayaEngine engine, TicketDto ticket) =>+{+ var r = engine.Predict(new Dictionary<string, object?> { ["message"] = ticket.Text },+ LayaPresets.Triage());+ var intent = r["intent"].AsChoice();+ return Results.Ok(new { intent = intent.Choice, intent.Confidence });+});+```++The container disposes the singleton when the host shuts down.++---++## Routing, language detection, email and shortlist++Runnable end-to-end example: [`samples/Laya.Routing`](https://github.com/NandhaKishorM/laya/tree/main/laya-dotnet/samples/Laya.Routing).++**Language detection** — no model, no network:++```csharp+var analysis = LanguageDetection.Analyse("Mein Konto wurde zweimal belastet");+// analysis.Script == "latin", analysis.Language == "de", analysis.IsEnglish == false+```++**`LayaRouter`** picks a checkpoint per request from script/language detection (or an explicit+override), loading checkpoints lazily under an LRU cache:++```csharp+using var router = new LayaRouter(new LayaRouterOptions { MaxLoaded = 2 });+var result = router.Predict(new { message = "I was charged twice" }, questions); // -> english+var deDe = router.Predict(new { message = "Mein Konto wurde zweimal belastet" }, questions); // -> multilingual+Console.WriteLine(result.Routing!.Reason); // "English Latin text"+```++`router.Route(state, questions)` decides without loading or running anything — useful for logging+or testing the routing logic in isolation. `Preload()` builds every checkpoint up front for a+server or demo, so no request pays a cold load.++**Email cleaning** strips quoted history, signatures and disclaimers before you ask any questions:++```csharp+var cleaned = LayaEmail.CleanBody(rawBody);+var state = LayaEmail.State(subject, rawBody, sender: "[email protected]");+var answers = router.Predict(state, LayaPresets.Email());+```++**Embedding shortlist** narrows a high-cardinality choice question before the model sees it, using+a caller-supplied embedding function (bring your own bi-encoder — this SDK does not include one):++```csharp+LayaEmbedFunction embed = texts => myBiEncoder.Embed(texts);+var result = LayaShortlist.Predict(router, state, questions, embed, k: 20);+var kept = result.Shortlist!["banking_intent"]; // kept.Labels, kept.Scores, kept.K, kept.N+```++---++## API reference++### `LayaEngine` : `IDisposable`++Answers typed questions about a piece of state in one forward pass. It is the .NET counterpart of+Python's `laya.Agent`.++| Member | Description |+|---|---|+| `static LayaEngine Create(LayaOptions? options = null)` | Finds the artifacts (see [resolution order](https://github.com/NandhaKishorM/laya/blob/main/laya-dotnet/MODELS.md#5-where-the-engine-looks)), loads the config, and builds the ONNX session and tokenizer. May download if `AllowDownload` is set. |+| `static LayaEngine FromDirectory(string directory)` | Shorthand for `Create(new LayaOptions { ModelDirectory = directory })`. |+| `LayaResult Predict(object? state, QuestionSet questions)` | Answers every question about `state` in a single batched inference call. See [Supplying state](#supplying-state). |+| `LayaConfig Config { get; }` | The loaded checkpoint's `rl_agent_config.json`. |+| `ModelArtifacts Artifacts { get; }` | The paths the artifacts were loaded from. |+| `void Dispose()` | Releases the session and tokenizer. Safe to call more than once. |++### `LayaCheckpoint` (enum)++Selects which checkpoint to load.++| Value | Encoder | Parameters | max\_len | head\_max\_len | Best for |+|---|---|---|---|---|---|+| `Multilingual` (default) | `jhu-clsp/mmBERT-base` | 322M | 1024 | 256 | Mixed-language or unknown-language traffic |+| `English` | `answerdotai/ModernBERT-large` | 421M | 512 | 192 | English-only workloads (accuracy collapses on non-Latin scripts) |+| `TypedDecisions` | `answerdotai/ModernBERT-large` | 421M | 1024 | 256 | Four specific typed-decisions workflows only |++### `LayaOptions`++Controls where artifacts come from and how ONNX Runtime is configured. All properties are+settable (`{ get; set; }`).++| Property | Type | Default | Notes |+|---|---|---|---|+| `ModelDirectory` | `string?` | `null` | Highest-priority artifact location. When set, all other resolution steps are skipped. |+| `Checkpoint` | `LayaCheckpoint` | `Multilingual` | Which checkpoint to load. Derived subdirectory, HF subfolder and cache path all come from this value unless overridden by `HuggingFaceSubfolder` or `ModelDirectory`. |+| `ExecutionProvider` | `LayaExecutionProvider` | `Cpu` | `Cuda` and `DirectMl` need the matching ORT native package (`Microsoft.ML.OnnxRuntime.Gpu` / `.DirectML`) in place of the CPU one. |+| `IntraOpThreads` | `int?` | ORT default | Threads within one operator. Worth pinning in a container with a CPU limit. |+| `InterOpThreads` | `int?` | ORT default | Threads across independent operators. |+| `AllowDownload` | `bool` | `false` | Network access is never implicit. Needs the ONNX export to be published (see above). |+| `HuggingFaceRepo` | `string` | `convaiinnovations/laya` | Repository to download from. |+| `HuggingFaceSubfolder` | `string` | derived from `Checkpoint` | Subfolder in the repository. Prefer setting `Checkpoint` instead. |+| `HuggingFaceToken` | `string?` | `null` → `HF_TOKEN` | Bearer token for gated or private repos. `""` disables auth. |+| `CacheDirectory` | `string?` | per-user cache | Root that downloads land under: `%LOCALAPPDATA%\laya\onnx` or `~/.cache/laya/onnx`. |+| `DownloadProgress` | `IProgress<ArtifactDownloadProgress>?` | `null` | Per-file progress sink. |++### `LayaExecutionProvider` (enum)++`Cpu` (default), `Cuda`, `DirectMl`.++### `ArtifactDownloadProgress` (record struct)++`(string FileName, long BytesReceived, long? TotalBytes)`. `TotalBytes` is `null` when the server+omits `Content-Length`. Files are downloaded to a `.part` file and renamed when complete, so an+interrupted download never leaves a file that looks complete.++### `ModelArtifacts`++Validated paths to every file the engine loads. You normally don't need this type directly.+`LayaEngine.Create` calls it for you. Use it to check or pre-fetch artifacts, for example in a+startup health check or a build step.++| Member | Description |+|---|---|+| `static ModelArtifacts Resolve(LayaOptions options)` | Resolves, validates and (if allowed) downloads synchronously. |+| `static Task<ModelArtifacts> ResolveAsync(LayaOptions options, CancellationToken ct = default)` | Async version. Supports cancelling the download. |+| `string Directory` | Absolute artifact directory. |+| `LayaArtifactLayout Layout` | `Fused` or `Split` — whichever layout was found in `Directory`. |+| `string? ModelPath` | `…/model.onnx`, or `null` when `Layout` is `Split`. |+| `string? EncoderPath` | `…/encoder.onnx`, or `null` when `Layout` is `Fused`. |+| `string? HeadPath` | `…/head.onnx`, or `null` when `Layout` is `Fused`. |+| `string ConfigPath` | `…/rl_agent_config.json` |+| `string TokenizerPath` | `…/tokenizer/tokenizer.json` (fused) or `…/tokenizer.json` (split). |++```csharp+// Pre-fetch at deploy time, then start the engine from the resolved directory.+var artifacts = await ModelArtifacts.ResolveAsync(new LayaOptions { AllowDownload = true }, ct);+using var engine = LayaEngine.FromDirectory(artifacts.Directory);+```++### `LayaConfig`++The contents of `rl_agent_config.json`. Missing keys fall back to the same defaults as Python's+`cfg.get(...)` (`max_len` 512, `head_max_len` 192, temperatures 1.0).++| Member | Description |+|---|---|+| `static LayaConfig Load(string path)` | Reads a config file. |+| `static LayaConfig Parse(string json)` | Parses config JSON. |+| `string? Encoder` | The encoder the checkpoint was trained on. For information only. |+| `int MaxLen` | Total sequence budget, in tokens (1024 on multilingual). |+| `int HeadMaxLen` | Budget for instructions plus all option fragments (256 on multilingual). |+| `IReadOnlyList<double> Temperature` | Per-question-type temperature `[choice, score, noul]`, clamped. Always 3 entries. |+| `IReadOnlyDictionary<string, double> TemperatureByOptions` | Per-bucket temperature, clamped. Keys look like `choice:3-5` (buckets `2`, `3-5`, `6-10`, `11+`). |+| `IReadOnlyList<double> TemperatureRaw` / `TemperatureByOptionsRaw` | What the checkpoint shipped, before clamping. |+| `IReadOnlyList<string> ClampedTemperatures` | Every temperature that was clamped into `[0.5, 5.0]`. A non-empty list means confidence from those buckets is uncalibrated. Each entry is also logged at load time via `Trace.TraceWarning`. |++```csharp+foreach (var warning in engine.Config.ClampedTemperatures)+ logger.LogWarning("Laya temperature clamped: {Bucket}", warning);+```++### `Question` (abstract) and subclasses++| Member | Description |+|---|---|+| `static ChoiceQuestion Choice(object instructions, params (string Label, object? Description)[] options)` | Choice with described options. |+| `static ChoiceQuestion Choice(object instructions, params string[] labels)` | Choice with bare labels. |+| `static ChoiceQuestion Choice(object instructions, IEnumerable<KeyValuePair<string, object?>> options)` | Choice from any ordered sequence. |+| `static ScoreQuestion Score(object instructions, params object?[] levels)` | Score over levels, lowest first. |+| `static ScoreQuestion Score(object instructions, IEnumerable<object?> levels)` | Same, from a sequence. |+| `static NoulQuestion Noul(object instructions, object? ifFalse = null, object? ifTrue = null)` | Yes/no, with optional wording for each side. |+| `QuestionType Type` | `Choice`, `Score` or `Noul`. |+| `object Instructions` | The prompt, as supplied. |++- **`ChoiceQuestion`**: `IReadOnlyList<KeyValuePair<string, object?>> Options` (index N is the+ label for logit N), `IReadOnlyList<string> Labels`.+- **`ScoreQuestion`**: `IReadOnlyList<object?> Levels`, lowest first.+- **`NoulQuestion`**: `object? IfFalse`, `object? IfTrue` (`null` means the default wording).++Questions are immutable. You can reuse the same instance across calls, sets and threads.++### `QuestionType` (enum)++`Choice = 0`, `Score = 1`, `Noul = 2`. The values match Python's `QTYPES` and the model's type+embedding.++### `QuestionSet` : `IEnumerable<KeyValuePair<string, Question>>`++| Member | Description |+|---|---|+| `QuestionSet()` | An empty set. Supports collection-initializer syntax (`["id"] = …`). |+| `QuestionSet(IEnumerable<KeyValuePair<string, Question>> questions)` | Seeded from an ordered sequence. Later duplicates replace earlier ones in place. |+| `Question this[string id] { get; set; }` | Gets a question, or adds/replaces one. Replacing keeps the original position. |+| `void Add(string id, Question question)` | Appends a question. Throws `ArgumentException` on a duplicate id. |+| `bool ContainsId(string id)` | Whether the id is present. |+| `int Count` | Number of questions. |+| `IReadOnlyList<string> Ids` | Ids in insertion order. |++### `LayaResult`++| Member | Description |+|---|---|+| `Answer this[string id]` | The answer for one question. Throws `KeyNotFoundException` for an unknown id. |+| `bool TryGetAnswer(string id, out Answer? answer)` | Non-throwing lookup. |+| `IEnumerable<KeyValuePair<string, Answer>> Answers` | Every answer, in the order asked. |+| `IReadOnlyList<string> Ids` | Question ids, in the order asked. |+| `int Count` | Number of answers. |+| `string Model` | `"laya-rl-agent"`, the same value Python reports. |+| `Usage Usage` | Token accounting for the call. |++### `Answer` (abstract) and subclasses++| Member | Description |+|---|---|+| `QuestionType Type` | Which question type produced this answer. |+| `double Confidence` | 0..1. Entropy-based for choice and score, `max(p, 1-p)` for noul. |+| `double AnswerConfidence` | 0..1. `max(p)`, the same quantity on every question type. |+| `ActionInfo Action` | The action head's output. |+| `ChoiceAnswer AsChoice()` / `ScoreAnswer AsScore()` / `NoulAnswer AsNoul()` | Narrows to the typed answer. Throws `InvalidOperationException` on a mismatch. |++- **`ChoiceAnswer`**: `string Choice`, `IReadOnlyList<KeyValuePair<string, double>> Probabilities`+ (in option order), `double this[string label]`.+- **`ScoreAnswer`**: `double Score` (expected level), `int MostLikelyLevel`,+ `IReadOnlyList<double> Probabilities`, `IReadOnlyList<object?> Legend`.+- **`NoulAnswer`**: `double Probability` (P(true)), `bool Value` (`Probability >= 0.5`).++### `ActionInfo` and `Usage` (record structs)++- `ActionInfo(double ActProbability)`: the model's own estimate of whether the answer can be+ acted on directly rather than escalated. It corresponds to Python's `action.act_probability`.+- `Usage(int InputTokens, int OutputTokens)`: non-padding input tokens across the batch.+ `OutputTokens` is always 0.++### `LayaPresets` (static)++`Triage()`, `Email(IReadOnlyDictionary<string, string>? categories = null)`,+`Email(IEnumerable<KeyValuePair<string, string>> categories)`, `Guard()`, `Moderation()`, `Router()`,+and `DefaultEmailCategories`. Each method returns a new `QuestionSet`. See+[Built-in presets](#built-in-presets).++### Lower-level building blocks++These types are public so that you can inspect the prompt the model sees, test calibration, or+plug in a different tokenizer. `Predict` uses all of them internally. Most applications never need+them.++| Type | Members | Purpose |+|---|---|---|+| `SequenceBuilder` (static) | `TypeName(QuestionType)`, `RenderInstructions(Question)`, `RenderOptions(Question)`, `Build(ILayaTokenizer, object? state, Question, int maxLen, int headMaxLen, bool truncateLeft = false)` | Builds `[CLS] <type> question: … [SEP] [MASK] opt0 [MASK] opt1 … [SEP] state [SEP]` and returns the marker positions. Useful for seeing exactly how an option is rendered. |+| `Calibration` (static) | `TempMin` (0.5), `TempMax` (5.0), `ClampTemperature`, `TempBucket`, `ResolveTemperature`, `Probabilities`, `Softmax`, `ConfidenceFromProbs`, `AnswerConfidence`, `ExpectedScore`, `ArgMax`, `Round4` | Logits to calibrated probabilities, confidence and scores. |+| `Collator` (static) | `MinMarkers` (2), `Collate(IReadOnlyList<SequenceItem>, int padId)` | Pads a batch into the tensors the ONNX graph takes. |+| `CollatedBatch`, `SequenceItem` | tensor arrays and shapes | Collator input and output. |+| `PythonJson` (static) | `State`, `Criterion`, `Instructions`, `Dumps(value, escapeNonAscii, fallbackToStr)`, `Repr(double)` | Serialization that matches Python's `json.dumps` byte for byte, in the three dialects the prompt uses. |+| `Laya.Tokenization.ILayaTokenizer` | `Encode(string)`, `PadId`, `ClsId`, `SepId`, `MaskId`, `UnkId`, `MaskToken` | Tokenizer abstraction. `Encode` must add no special tokens. |+| `Laya.Tokenization.HfTokenizer` | `HfTokenizer(string tokenizerJsonPath)`, `IDisposable` | The shipped implementation. It removes the `post_processor` so no special tokens are added (caching the result next to the file, or in `%TEMP%` if read-only). Special-token ids are resolved from `tokenizer_config.json` + `tokenizer.json` at load time, so both mmBERT (`<pad>=0`, etc.) and ModernBERT (`[PAD]=50283`, etc.) tokenizers work. Throws if a token string is present in the config but absent from the id maps. |++```csharp+// See the exact option text the model scores for a question.+foreach (var line in SequenceBuilder.RenderOptions(Question.Noul("Is this spam?")))+ Console.WriteLine(line);+// false: no, the statement does not hold+// true: yes, the statement holds+```++---++## Errors++| Exception | Thrown by | When |+|---|---|---|+| `DirectoryNotFoundException` | `Create`, `ModelArtifacts.Resolve*` | No artifact in any location and `AllowDownload` is false. Also thrown when the given directory doesn't exist, or a download returns 404 / 401 / 403. The message lists what was tried and what to do. |+| `FileNotFoundException` | `Create`, `ModelArtifacts.Resolve*` | The directory exists but is missing a file. Most often `model.onnx.data` was left behind when copying. |+| `InvalidOperationException` | `Create` | `tokenizer.json` belongs to a different checkpoint (special-token id mismatch). |+| `OnnxRuntimeException` | `Create` | ORT couldn't load the graph, or the requested execution provider isn't installed. |+| `ArgumentException` | `Predict` | Empty `QuestionSet`, or a question whose options can't fit in `HeadMaxLen`. |+| `ArgumentException` | `Question.*`, `QuestionSet` | Duplicate choice label or question id, no options or levels, empty id. |+| `System.Text.Json.JsonException` | `Predict` | State contains a value with no JSON equivalent (a POCO, for example), or a circular reference. |+| `ObjectDisposedException` | `Predict` | The engine has been disposed. |+| `KeyNotFoundException` | `LayaResult[id]`, `ChoiceAnswer[label]` | Unknown id or label. Use `TryGetAnswer` for optional ids. |+| `InvalidOperationException` | `AsChoice` / `AsScore` / `AsNoul` | Answer type mismatch. |++---++## Performance++Expect **a few hundred milliseconds per call on CPU** for a 322M-parameter encoder. That's around+190–230 ms warm on a recent desktop x64 core. The first call takes roughly an extra second while+the session warms up, so run one throwaway `Predict` at startup if first-request latency matters.+The 33 ms figure in the Python README was measured on a T4 GPU and doesn't apply to CPU inference.++Latency is per *call*, not per question. The architecture is built around batching every question+into one forward pass, so asking four questions together costs about the same as asking one. Put+related questions in one `QuestionSet` rather than calling `Predict` once per question.++---++## Tests++The suite has three tiers, so it stays useful without the 1.3 GB artifact:++| Tier | Needs | Covers |+|---|---|---|+| 1 | nothing | `PythonJson` dialects, collation, calibration math, option-budget truncation, presets, artifact-layout detection (`ModelArtifactsTests`), tokenizer special-token fallback against synthetic fixtures (`HfTokenizerFallbackTests`) |+| tokenizer | `tokenizer/tokenizer.json` | token ids and marker positions against recorded Python output |+| 2 | the full artifact | `Predict` against recorded Python answers, and a per-checkpoint shape sweep (72 configurations for multilingual and typed-decisions; 66 for english whose `max_len=512` drops the 1024-token length) — run against **both** artifact layouts when both are present (`onnx/<checkpoint>` fused, `onnx-split/<checkpoint>` split); a checkpoint missing one layout only skips that layout's rows |++```bash+# Run with all three checkpoints, fused layout (recommended)+LAYA_ONNX_ROOT=/path/to/onnx dotnet test --solution Laya.slnx++# Backward compat: multilingual only+LAYA_ONNX_DIR=/path/to/onnx/multilingual dotnet test --solution Laya.slnx+```++The split-layout theories (`...Split` test names) look for `<repo>/onnx-split/<checkpoint>`+via a fixed walk-up from the test binary — there is no split-layout environment variable yet.+They skip per-checkpoint like the fused ones do when that directory is absent.++Checkpoints and layouts whose artifacts are missing are skipped, not failed. **How many tests+run is not a fixed number** — most of Tier 2 is parameterized per checkpoint, per shape, and now+per layout, so the count grows every time golden-answer coverage or shape-sweep coverage grows.+As a concrete snapshot from this repository: the 646 tests that need no ONNX artifact at all+(Tier 1 plus the tokenizer-lookup fixtures) always run with `0` failed and `0` skipped; a filtered+run excluding the five model-backed classes (`PredictParityTests`, `ShapeSweepTests`,+`TokenizerParityTests`, `RouterEndToEndTests`, `ShortlistEndToEndTests`) confirms this:++```bash+dotnet bin/Release/net10.0/Laya.Tests.dll \+ -class- "Laya.Tests.PredictParityTests" \+ -class- "Laya.Tests.ShapeSweepTests" \+ -class- "Laya.Tests.TokenizerParityTests" \+ -class- "Laya.Tests.RouterEndToEndTests" \+ -class- "Laya.Tests.ShortlistEndToEndTests"+```++With every checkpoint present in **both** layouts, the full suite (including the five+model-backed classes above) is 1358 tests — up from a pre-split-layout snapshot of 1088;+the "621" this replaced was measured before several rounds of per-checkpoint parameterization and+had already gone stale independent of the split layout. Re-run the count yourself with+`dotnet test --solution laya-dotnet/Laya.slnx -c Release` once artifacts are in place, since this+number is expected to keep moving as coverage grows.++The parity vectors in `tests/Laya.Tests/golden/<checkpoint>/` are recorded by+[`tools/dump_golden.py`](https://github.com/NandhaKishorM/laya/blob/main/laya-dotnet/tools/dump_golden.py) `--checkpoint all`, which runs the **real** `laya.Agent.system_one` over the+same ONNX graph. The vectors come from the shipping Python code, not from a second implementation.+The same golden answers back both the fused and the split parity theories: the split graph is the+fused graph cut into two ONNX files, not a different model, so one recording serves both.++The test project is an xUnit **v3** application running on Microsoft.Testing.Platform. The .NET 10+SDK no longer runs these projects through VSTest. That's why `laya-dotnet` has a+`global.json` that selects the `Microsoft.Testing.Platform` runner, and why the solution is passed+with `--solution` rather than as a positional argument.++---++## Not in v1++- Quantized artifacts.+- Async `Predict`, and runtime overrides of `max_len` / `head_max_len`.+- `embed_fn_from_agent` (mean-pooling the checkpoint's own encoder into a shortlist embedder) —+ `LayaShortlist` takes only a caller-supplied embedding function; see [Routing, language+ detection, email and shortlist](#routing-language-detection-email-and-shortlist).++## Known gaps++This SDK tracks the Python SDK at **0.3.21** for everything above, but two modules are pinned to+an older Python release and have not been ported forward yet:++- **Language detection** (`LanguageDetection`) follows Python's `lang.py` at **0.3.6**.+- **Email handling** (`LayaEmail`) follows Python's `email.py` at **0.3.6**.++Bringing both up to 0.3.21 is planned as a follow-up; until then, treat their behavior as frozen+at 0.3.6 even though the rest of the SDK (sequence building, JSON dialects, calibration and+answers) has moved to 0.3.21.++- **Opt-in features added in Python 0.3.21 are not ported yet:** abstention (`min_confidence`),+ `predict_long`, and per-language temperatures (`lang_temperatures`). They are off by default in+ Python, so default predictions are unaffected. They are planned as follow-ups.++---++## License++Apache 2.0. Developed by Convai Innovations.