Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Laya Models: Obtaining the ONNX Artifacts
The Laya .NET SDK (README), a .NET port of the Laya Python SDK, runs ONNX models through ONNX Runtime, not PyTorch weights, so no Python process is needed at runtime. This guide explains how to get those ONNX artifacts: either convert them yourself from the published PyTorch checkpoints (works today), or let the SDK download them once they are published.
Quick path: create a venv, pip install -e . plus the exporter's
dependencies, run python laya-dotnet/tools/export_onnx.py --model all, and set LAYA_ONNX_ROOT to the
resulting onnx/ folder. Section 1
walks through each step.
The ONNX export is not on Hugging Face yet, so exporting it yourself is the only way to get the artifacts today. Downloading (section 6) will work without code changes once they are published.
1. Export the ONNX model (step by step)
You need Python 3.9+ and about 2.5 GB free per checkpoint (6.7 GB for all three, counting the Hugging Face download cache; see step 3). Run everything from the repository root unless a step says otherwise.
Step 1: Create a virtual environment and install
The dynamo-based ONNX exporter requires torch >= 2.9.
Tested versions (from the repo's .venv):
| Package | Tested version |
|---|---|
torch |
2.14.0 |
onnx |
1.23.0 |
onnxscript |
0.7.2 |
onnxruntime |
1.30.0 |
transformers |
5.17.0 |
safetensors |
0.8.0 |
huggingface_hub |
1.32.0 |
numpy |
2.5.3 |
The laya Python package itself must be importable, so the commands below
also install the repository with pip install -e ..
# PowerShell
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install "torch>=2.9" onnx onnxscript onnxruntime transformers safetensors huggingface_hub numpy
pip install -e .
# Bash / WSL
python -m venv .venv && source .venv/bin/activate
pip install "torch>=2.9" onnx onnxscript onnxruntime transformers safetensors huggingface_hub numpy
pip install -e .
Step 2 (optional): Hugging Face authentication
The convaiinnovations/laya checkpoints are public and need no token. For
gated or private repos, supply a token:
$env:HF_TOKEN = "hf_..." # PowerShell — persists for the session
export HF_TOKEN="hf_..." # Bash
Or pass --token hf_... directly to the script.
Step 3 (optional): Disk space and HF_HOME
The exporter calls huggingface_hub.snapshot_download to fetch the PyTorch
weights. If the default HuggingFace cache (%LOCALAPPDATA%\HuggingFace on
Windows, ~/.cache/huggingface elsewhere) is on a volume without enough
space, redirect it with HF_HOME.
| Checkpoint | PyTorch weights (HF cache) | ONNX output | Total |
|---|---|---|---|
multilingual |
~644 MB | ~1.2 GB | ~1.9 GB |
english |
~842 MB | ~1.6 GB | ~2.5 GB |
typed-decisions |
~842 MB | ~1.6 GB | ~2.5 GB |
| all three | ~2.3 GB | ~4.4 GB | ~6.7 GB |
$env:HF_HOME = "C:\hf_cache" # PowerShell
export HF_HOME=/d/hf_cache # Bash
Step 4: Export
The script downloads the PyTorch checkpoint from convaiinnovations/laya
(public, no token needed) and writes onnx/<checkpoint>/ under the repository
root. The multilingual checkpoint takes about 9 minutes on CPU.
# Export the multilingual checkpoint to onnx/multilingual/ (~9 min on CPU)
python laya-dotnet/tools/export_onnx.py --model multilingual
# Export and verify ONNX parity against PyTorch (~9 min + ~5 min verify)
python laya-dotnet/tools/export_onnx.py --model multilingual --verify
# Export typed-decisions with parity check (~30 min total)
python laya-dotnet/tools/export_onnx.py --model typed-decisions --verify
# Export all three checkpoints
python laya-dotnet/tools/export_onnx.py --model all
# Export to a custom directory (no --force needed; leaves onnx/ untouched)
python laya-dotnet/tools/export_onnx.py --model multilingual --out C:\tmp\laya-test --verify
# Re-export, overwriting an existing artifact
python laya-dotnet/tools/export_onnx.py --model multilingual --force
# Export a local checkpoint directory (must contain rl_agent_config.json)
python laya-dotnet/tools/export_onnx.py C:\path\to\checkpoint --out onnx\my-model
The script refuses to overwrite <out>/<name>/model.onnx if it already exists.
Pass --force to overwrite. The check happens before any download, so a
refusal wastes no time or bandwidth.
The --verify flag runs a self-check against the fixtures.npz reference
outputs and a 36-combination shape sweep (6 seq_lens × 2 batch_sizes × 3 marker_counts). Skip it if you just need the files quickly; run it before
publishing.
Step 5: Point the SDK at the export
The quickest way — no code changes.
Single checkpoint (legacy, multilingual only):
# PowerShell
$env:LAYA_ONNX_DIR = "C:\models\laya\multilingual"
# Bash
export LAYA_ONNX_DIR=/c/models/laya/multilingual
All three checkpoints at once:
If you export all three under a common parent directory (e.g. onnx/multilingual,
onnx/english, onnx/typed-decisions), point LAYA_ONNX_ROOT at the parent:
# PowerShell
$env:LAYA_ONNX_ROOT = "C:\models\laya"
# Bash
export LAYA_ONNX_ROOT=/c/models/laya
The SDK appends the checkpoint subdirectory name automatically
(multilingual, english, or typed-decisions).
Use absolute paths. Relative paths depend on the working directory, which is not reliable across test runners and hosts.
Or pass the checkpoint folder in code with LayaOptions.ModelDirectory /
LayaEngine.FromDirectory(...) (see section 7).
Step 6: Check it works
Run the quickstart sample against the export, from the repository root:
# Multilingual (default)
dotnet run --project laya-dotnet/samples/Laya.Sample -- --model-dir onnx/multilingual
# English checkpoint
dotnet run --project laya-dotnet/samples/Laya.Sample -- --checkpoint english --model-dir onnx/english
# Typed-decisions checkpoint
dotnet run --project laya-dotnet/samples/Laya.Sample -- --checkpoint typed-decisions --model-dir onnx/typed-decisions
2. The three checkpoints
All three are published under the bundle repo convaiinnovations/laya; the
exporter downloads from there by default.
| Name | Bundle subfolder | Encoder | Params | max_len | head_max_len | model.onnx | model.onnx.data |
|---|---|---|---|---|---|---|---|
english |
(root) | answerdotai/ModernBERT-large |
421 M | 512 | 192 | 3.2 MB | 1.57 GB |
multilingual |
multilingual |
jhu-clsp/mmBERT-base |
322 M | 1024 | 256 | 2.5 MB | 1.20 GB |
typed-decisions |
typed-decisions |
answerdotai/ModernBERT-large |
421 M | 1024 | 256 | 2.6 MB | 1.57 GB |
Each checkpoint also has a standalone repo (convaiinnovations/laya-multilingual,
etc.) that mirrors the same weights.
3. SDK support status
All three checkpoints are fully supported by the .NET SDK.
HfTokenizer resolves special-token ids from tokenizer_config.json +
tokenizer.json at load time, so it works for both the mmBERT-style tokens
(<pad>=0, <bos>=2, <eos>=1, <unk>=3, <mask>=4) and the ModernBERT
ByteLevel-BPE tokens ([PAD]=50283, [CLS]=50281, [SEP]=50282,
[UNK]=50280, [MASK]=50284). Select a checkpoint with LayaOptions.Checkpoint
or point LayaOptions.ModelDirectory / LAYA_ONNX_DIR at the artifact
directory directly.
// explicit checkpoint selection
using var engine = LayaEngine.Create(new LayaOptions
{
Checkpoint = LayaCheckpoint.English,
AllowDownload = true,
});
// or point directly at the directory
using var engine = LayaEngine.FromDirectory(@"C:\models\laya\english");
4. Output layout
export_onnx.py (this SDK's exporter) writes the fused layout below to <out>/<name>/:
<out>/<name>/
model.onnx # ONNX graph (~2.5–3.2 MB, opset 18)
model.onnx.data # External weight sidecar (~1.2–1.6 GB)
rl_agent_config.json # Sequence lengths and temperature calibration
tokenizer/
tokenizer.json # HuggingFace tokenizer (post_processor kept intact)
tokenizer_config.json
fixtures.npz # PyTorch reference outputs (used by --verify; not needed by SDK)
model.onnx and model.onnx.data must stay in the same directory. ONNX
Runtime resolves the external-data file relative to the graph file, which is
also why the API takes a directory rather than a file. Moving model.onnx
alone produces a baffling "missing initializer" error.
Tokenizer post_processor: tokenizer.json retains its TemplateProcessing
post_processor. The SDK strips it at load time via
HfTokenizer.EnsureStrippedTokenizer and caches the result as
tokenizer.nopost.json alongside the original. That cache file is written by
the SDK, not by the exporter.
Split layout (laya-ts exporter)
The laya-ts exporter produces a second, split layout instead of one fused graph — the
encoder and the RL head as two separate ONNX files, each with its own external-data sidecar:
<out>/<name>/
encoder.onnx # Encoder graph: (input_ids, attention_mask) -> last_hidden_state
encoder.onnx.data # Encoder weights (~1.2-1.6 GB)
head.onnx # RL head graph: (hidden_states, marker_pos, marker_mask,
# qtype [B, 1], attention_mask) -> logits, act_logits
head.onnx.data # Head weights (~60-110 MB)
rl_agent_config.json # Same shape/config file as the fused layout
tokenizer.json # Sits directly in the directory, not under tokenizer/
Each .data file must stay next to its graph, for the same reason as model.onnx.data.
Note the split layout's tokenizer.json has no sibling tokenizer_config.json and no
tokenizer/ subfolder. When tokenizer_config.json is absent, HfTokenizer takes CLS and SEP
from tokenizer.json's own post_processor template ([CLS] $A [SEP], or <bos> $A <eos> for
multilingual) and resolves PAD, MASK and UNK from added_tokens' conventional spellings
([PAD]/[MASK]/[UNK] or <pad>/<mask>/<unk>). A spelling guess alone is not safe for
CLS/SEP: the multilingual vocabulary also contains <s>/</s>, which are not its CLS/SEP.
The .NET SDK auto-detects which layout a directory holds (see section 5);
callers do not choose a layout explicitly. If a directory somehow has both model.onnx and
encoder.onnx+head.onnx, the fused layout wins, since that is what this SDK's own exporter has
always produced.
In both layouts, rl_agent_config.json is where MaxLen (max_len) and HeadMaxLen
(head_max_len) come from — LayaConfig.Load reads the same file regardless of which ONNX
layout sits beside it, so a split export needs the same config file the fused one does.
5. Where the engine looks
The engine looks for the directory in this order:
LayaOptions.ModelDirectory- the
LAYA_ONNX_DIRenvironment variable — maps to the multilingual checkpoint only (legacy) LAYA_ONNX_ROOT/<checkpoint-name>— the recommended multi-checkpoint path; the SDK appends the subfolder name (multilingual,english, ortyped-decisions)- the local cache,
<CacheDirectory>/<checkpoint-name>.CacheDirectorydefaults to%LOCALAPPDATA%\laya\onnxon Windows and~/.cache/laya/onnxelsewhere. - a download into that cache, only if
LayaOptions.AllowDownload = true
If none of these has the artifact, resolution fails with a message that lists every location it tried.
Once a directory is chosen, ModelArtifacts.Resolve decides fused vs. split by checking which
files are actually present, not by any option or environment variable:
model.onnx+model.onnx.data→ fused (LayaArtifactLayout.Fused)encoder.onnx+head.onnx→ split (LayaArtifactLayout.Split)- both present → fused wins
- neither present, or only one half of a layout (e.g.
encoder.onnxwith nohead.onnx) →FileNotFoundExceptionlisting all four filenames, so a partial/interrupted export fails with the same message as an empty directory rather than a confusing null-session error later.
The tokenizer path is resolved the same way in both layouts: tokenizer/tokenizer.json (fused)
is tried before the root-level tokenizer.json (split); if a directory happens to have both, the
nested one wins.
6. Downloading (future, once published)
The ONNX artifacts have not been uploaded to HuggingFace yet. The
convaiinnovations/laya repository ships PyTorch weights
(model.safetensors), which the SDK cannot load. Until the ONNX files are
uploaded, AllowDownload = true fails immediately with a 404 and an explanatory
message.
Downloading is opt-in. Nothing touches the network unless you ask for it. Once the files are published (see section 8), enabling the download requires only:
using var engine = LayaEngine.Create(new LayaOptions
{
AllowDownload = true, // fetch from Hugging Face into the cache, once
HuggingFaceToken = null, // null falls back to HF_TOKEN; "" disables auth
DownloadProgress = new Progress<ArtifactDownloadProgress>(p =>
Console.Error.WriteLine($"{p.FileName}: {p.BytesReceived}/{p.TotalBytes}")),
});
The SDK downloads five files into the per-user cache and skips the download on subsequent runs if the directory is already complete:
- Windows:
%LOCALAPPDATA%\laya\onnx\multilingual - Linux / macOS:
~/.cache/laya/onnx/multilingual
Override the cache root with LayaOptions.CacheDirectory.
7. Using the artifact in code and tests
Create the engine in code
// From a known directory (all checkpoints work)
using var engine = LayaEngine.FromDirectory(@"C:\models\laya\multilingual");
// English checkpoint via LayaOptions.Checkpoint
using var engine = LayaEngine.Create(new LayaOptions
{
Checkpoint = LayaCheckpoint.English,
ModelDirectory = @"C:\models\laya\english",
IntraOpThreads = 4, // pin in containers with a CPU limit
});
// Multilingual with thread config, no ModelDirectory (LAYA_ONNX_DIR / LAYA_ONNX_ROOT / cache)
using var engine = LayaEngine.Create(new LayaOptions
{
IntraOpThreads = 4,
});
Run the tests
The test suite is parameterized over all three checkpoints. Set LAYA_ONNX_ROOT
to the parent of the three checkpoint directories:
# PowerShell — all three checkpoints
$env:LAYA_ONNX_ROOT = "C:\models\laya"
dotnet test --solution laya-dotnet/Laya.slnx
# Bash — all three checkpoints
LAYA_ONNX_ROOT=/c/models/laya dotnet test --solution laya-dotnet/Laya.slnx
The legacy LAYA_ONNX_DIR still works and maps to the multilingual checkpoint
only; the other two are found via walk-up or skipped gracefully.
Checkpoints whose artifacts are missing are skipped, not failed. The 646 tests that need no
ONNX artifact at all (Tier 1, plus the layout-detection and tokenizer-fallback fixtures) always
run; a much larger number of theory rows additionally run once onnx/<checkpoint> (fused) and/or
onnx-split/<checkpoint> (split) are present, since most of Tier 2 is parameterized per checkpoint,
per shape, and now per layout. That theory-row count is not a fixed constant — it grows every time
a checkpoint's golden-answer coverage or shape sweep grows, and again once split-layout artifacts
exist for a checkpoint — so treat any specific figure quoted here as a snapshot, not a target.
The 0.1.0 snapshot of the full count (all three checkpoints, both layouts, everything present)
is kept in one place, README.md § Tests, rather than restated here. Re-derive it
yourself with dotnet test --solution laya-dotnet/Laya.slnx -c Release (all artifacts present),
or with the -class- exclusions that section lists for the model-backed classes if you need the
true count without loading any weights.
8. Publishing to HuggingFace (maintainers only)
Requires write access to convaiinnovations/laya.
The SDK downloader fetches five files per checkpoint from convaiinnovations/laya.
A maintainer must upload the following paths for each checkpoint:
| Checkpoint | HF path in repo |
|---|---|
multilingual |
multilingual/<files> |
english |
<files> (bundle root — no subfolder) |
typed-decisions |
typed-decisions/<files> |
The five files per checkpoint are:
model.onnx, model.onnx.data, rl_agent_config.json,
tokenizer/tokenizer.json, tokenizer/tokenizer_config.json.
onnx/ is gitignored (the weights exceed GitHub's 100 MB hard limit), so
upload must go through the HuggingFace Hub CLI or Python API directly.
Run from the repo root after completing --verify:
# huggingface-cli (pip install huggingface_hub)
huggingface-cli upload convaiinnovations/laya onnx/multilingual multilingual \
--repo-type model --token hf_...
huggingface-cli upload convaiinnovations/laya onnx/english . \
--repo-type model --token hf_...
huggingface-cli upload convaiinnovations/laya onnx/typed-decisions typed-decisions \
--repo-type model --token hf_...
Once these files are live, AllowDownload = true with the matching checkpoint
works without any code changes.
The downloader handles either layout. It first sends a HEAD request for model.onnx; if that
exists it fetches the fused files above. Otherwise, if encoder.onnx and head.onnx both exist,
it fetches the split files (encoder.onnx, head.onnx, their .data sidecars when present,
rl_agent_config.json and tokenizer.json; see section 4). So whichever
layout gets uploaded to Hugging Face, AllowDownload = true works without code changes.
9. Troubleshooting
404 on download (AllowDownload)
The ONNX artifacts have not been published yet (see section 8). Export locally with
laya-dotnet/tools/export_onnx.py and point the SDK at the result via LAYA_ONNX_DIR or
LayaOptions.ModelDirectory.
Low disk space during export
Set HF_HOME to a volume with enough free space before running the exporter
(see the disk-space table in section 1, step 3). --out controls where the ONNX output lands.
model.onnx.data not found / "missing initializer" in ONNX Runtime
The sidecar must sit beside model.onnx. Copy both files together; the SDK
error message already reminds you of this: "both model.onnx and model.onnx.data
must reside in the same directory."
"already exists, pass --force"
<out>/<name>/model.onnx is present from a previous run. Use --force to
overwrite in place, or use --out to write to a separate directory.
Wrong-checkpoint error (tokenizer/config mismatch)
tokenizer.json and tokenizer_config.json are from different checkpoints
(e.g., the tokenizer from english with the config from multilingual). The
error message names the missing token. Keep the two files together as exported.
"neither model.onnx nor encoder.onnx/head.onnx found" (or the message names only three of the
four filenames it tried)
The directory has neither a complete fused export nor a complete split export — most often a
split export that only partly copied (encoder.onnx present, head.onnx missing, or vice
versa). ModelArtifacts.Resolve treats a half-split directory the same as an empty one rather
than guessing; copy the missing file, or re-run whichever exporter produced the directory.
Split-layout tokenizer error naming a token role (e.g. "cls_token") instead of a filename
The split layout has no tokenizer_config.json, so HfTokenizer reads CLS/SEP from the
post_processor template and the other roles from added_tokens' spelling instead (see
section 4). This error means the vocabulary uses a spelling the fallback does
not recognise; either add a tokenizer_config.json beside tokenizer.json (the fused layout's
approach), or update the fallback if this is a spelling worth supporting generally.
Windows console encoding during export
The dynamo exporter logs Unicode emoji (✅, etc.) that crash a non-UTF-8 Windows
console. The script reconfigures stdout/stderr to UTF-8 with replacement
automatically. Any unrecognised emoji appears as ?; the export continues
normally.