Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Laya Models: Obtaining the ONNX Artifacts
The Laya Rust SDK (README), a Rust port of the Laya Python SDK, runs ONNX models through ONNX Runtime, not PyTorch weights, so no Python process is needed at runtime. This guide explains how to get those ONNX artifacts: either convert them yourself from the published PyTorch checkpoints (works today), or let the SDK download them once they are published.
Quick path: create a venv, pip install -e . plus the exporter's
dependencies, run python laya-rust/tools/export_onnx.py --model all, and set LAYA_ONNX_ROOT to the
resulting onnx/ folder. Section 1
walks through each step.
The ONNX export is not on Hugging Face yet, so exporting it yourself is the only way to get the artifacts today. Downloading (section 6) will work without code changes once they are published.
1. Export the ONNX model (step by step)
You need Python 3.9+ and about 2.5 GB free per checkpoint (6.7 GB for all three, counting the Hugging Face download cache; see step 3). Run everything from the repository root unless a step says otherwise.
Step 1: Create a virtual environment and install
The dynamo-based ONNX exporter requires torch >= 2.9.
Tested versions (from the repo's .venv):
| Package | Tested version |
|---|---|
torch |
2.14.0 |
onnx |
1.23.0 |
onnxscript |
0.7.2 |
onnxruntime |
1.30.0 |
transformers |
5.17.0 |
safetensors |
0.8.0 |
huggingface_hub |
1.32.0 |
numpy |
2.5.3 |
The laya Python package itself must be importable, so the commands below
also install the repository with pip install -e ..
# PowerShell
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install "torch>=2.9" onnx onnxscript onnxruntime transformers safetensors huggingface_hub numpy
pip install -e .
# Bash / WSL
python -m venv .venv && source .venv/bin/activate
pip install "torch>=2.9" onnx onnxscript onnxruntime transformers safetensors huggingface_hub numpy
pip install -e .
Step 2 (optional): Hugging Face authentication
The convaiinnovations/laya checkpoints are public and need no token. For
gated or private repos, supply a token:
$env:HF_TOKEN = "hf_..." # PowerShell — persists for the session
export HF_TOKEN="hf_..." # Bash
Or pass --token hf_... directly to the script.
Step 3 (optional): Disk space and HF_HOME
The exporter calls huggingface_hub.snapshot_download to fetch the PyTorch
weights. If the default HuggingFace cache (%LOCALAPPDATA%\HuggingFace on
Windows, ~/.cache/huggingface elsewhere) is on a volume without enough
space, redirect it with HF_HOME.
| Checkpoint | PyTorch weights (HF cache) | ONNX output | Total |
|---|---|---|---|
multilingual |
~644 MB | ~1.2 GB | ~1.9 GB |
english |
~842 MB | ~1.6 GB | ~2.5 GB |
typed-decisions |
~842 MB | ~1.6 GB | ~2.5 GB |
| all three | ~2.3 GB | ~4.4 GB | ~6.7 GB |
$env:HF_HOME = "C:\hf_cache" # PowerShell
export HF_HOME=/d/hf_cache # Bash
Step 4: Export
The script downloads the PyTorch checkpoint from convaiinnovations/laya
(public, no token needed) and writes onnx/<checkpoint>/ under the repository
root. The multilingual checkpoint takes about 9 minutes on CPU.
# Export the multilingual checkpoint to onnx/multilingual/ (~9 min on CPU)
python laya-rust/tools/export_onnx.py --model multilingual
# Export and verify ONNX parity against PyTorch (~9 min + ~5 min verify)
python laya-rust/tools/export_onnx.py --model multilingual --verify
# Export typed-decisions with parity check (~30 min total)
python laya-rust/tools/export_onnx.py --model typed-decisions --verify
# Export all three checkpoints
python laya-rust/tools/export_onnx.py --model all
# Export to a custom directory (no --force needed; leaves onnx/ untouched)
python laya-rust/tools/export_onnx.py --model multilingual --out C:\tmp\laya-test --verify
# Re-export, overwriting an existing artifact
python laya-rust/tools/export_onnx.py --model multilingual --force
# Export a local checkpoint directory (must contain rl_agent_config.json)
python laya-rust/tools/export_onnx.py C:\path\to\checkpoint --out onnx\my-model
The script refuses to overwrite <out>/<name>/model.onnx if it already exists.
Pass --force to overwrite. The check happens before any download, so a
refusal wastes no time or bandwidth.
The --verify flag runs a self-check against the fixtures.npz reference
outputs and a 36-combination shape sweep (6 seq_lens × 2 batch_sizes × 3 marker_counts). Skip it if you just need the files quickly; run it before
publishing.
Step 5: Point the SDK at the export
The quickest way — no code changes.
Single checkpoint (legacy, multilingual only):
# PowerShell
$env:LAYA_ONNX_DIR = "C:\models\laya\multilingual"
# Bash
export LAYA_ONNX_DIR=/c/models/laya/multilingual
All three checkpoints at once:
If you export all three under a common parent directory (e.g. onnx/multilingual,
onnx/english, onnx/typed-decisions), point LAYA_ONNX_ROOT at the parent:
# PowerShell
$env:LAYA_ONNX_ROOT = "C:\models\laya"
# Bash
export LAYA_ONNX_ROOT=/c/models/laya
The SDK appends the checkpoint subdirectory name automatically
(multilingual, english, or typed-decisions).
Use absolute paths. Relative paths depend on the working directory, which is not reliable across test runners and hosts.
Or pass the checkpoint folder in code with LayaOptions::model_directory /
LayaEngine::from_directory(...) (see section 7).
Step 6: Check it works
Run the quickstart sample against the export. From laya-rust/, the default
export location is ../onnx/<checkpoint>:
cargo run --release -p laya-sample -- --model-dir ../onnx/multilingual
cargo run --release -p laya-sample -- --checkpoint english --model-dir ../onnx/english
cargo run --release -p laya-sample -- --checkpoint typed-decisions --model-dir ../onnx/typed-decisions
2. The three checkpoints
All three are published under the bundle repo convaiinnovations/laya; the
exporter downloads from there by default.
| Name | Bundle subfolder | Encoder | Params | max_len | head_max_len | model.onnx | model.onnx.data |
|---|---|---|---|---|---|---|---|
english |
(root) | answerdotai/ModernBERT-large |
421 M | 512 | 192 | 3.2 MB | 1.57 GB |
multilingual |
multilingual |
jhu-clsp/mmBERT-base |
322 M | 1024 | 256 | 2.5 MB | 1.20 GB |
typed-decisions |
typed-decisions |
answerdotai/ModernBERT-large |
421 M | 1024 | 256 | 2.6 MB | 1.57 GB |
Each checkpoint also has a standalone repo (convaiinnovations/laya-multilingual,
etc.) that mirrors the same weights.
3. SDK support status
All three checkpoints are fully supported by the Rust SDK.
HfTokenizer resolves special-token ids from tokenizer_config.json +
tokenizer.json at load time, so it works for both the mmBERT-style tokens
(<pad>=0, <bos>=2, <eos>=1, <unk>=3, <mask>=4) and the ModernBERT
ByteLevel-BPE tokens ([PAD]=50283, [CLS]=50281, [SEP]=50282,
[UNK]=50280, [MASK]=50284). Select a checkpoint with LayaOptions::checkpoint
or point LayaOptions::model_directory / LAYA_ONNX_DIR at the artifact
directory directly.
use laya::{LayaCheckpoint, LayaEngine, LayaOptions};
// explicit checkpoint selection
let engine = LayaEngine::create(LayaOptions {
checkpoint: LayaCheckpoint::English,
..Default::default()
})?;
// or point directly at the directory
let engine = LayaEngine::from_directory("/models/laya/english")?;
4. Output layout
The Rust SDK auto-detects two different export layouts for a checkpoint directory. Whichever one
is present, only that directory needs to be pointed at — ModelArtifacts::layout() reports which
one it found.
Fused layout (export_onnx.py, this section's exporter)
Each model is written to <out>/<name>/:
<out>/<name>/
model.onnx # ONNX graph (~2.5–3.2 MB, opset 18)
model.onnx.data # External weight sidecar (~1.2–1.6 GB)
rl_agent_config.json # Sequence lengths and temperature calibration
tokenizer/
tokenizer.json # HuggingFace tokenizer (post_processor kept intact)
tokenizer_config.json # optional but recommended; fetched by download
fixtures.npz # PyTorch reference outputs (used by --verify; not needed by SDK)
Inputs: input_ids, attention_mask, marker_pos, marker_mask, qtype (shape [B]).
Outputs: logits, act_logits.
model.onnx and model.onnx.data must stay in the same directory. ONNX
Runtime resolves the external-data file relative to the graph file, which is
also why the API takes a directory rather than a file. Moving model.onnx
alone produces a baffling "missing initializer" error.
Tokenizer post_processor: tokenizer.json retains its TemplateProcessing
post_processor. The Rust SDK needs no stripped copy: the tokenizers crate
encodes without special tokens directly.
Split layout (laya-ts's exporter)
This is the layout produced by laya-ts's own ONNX exporter — a graph split into a text encoder and a small prediction head instead of one fused graph:
<out>/<name>/
encoder.onnx # text -> hidden states: inputs input_ids, attention_mask;
# output last_hidden_state
encoder.onnx.data # encoder weights (~1.2-1.6 GB)
head.onnx # hidden states + marker metadata -> logits: inputs hidden_states,
# marker_pos, marker_mask, qtype (shape [B, 1]), attention_mask;
# outputs logits, act_logits
head.onnx.data # head weights (~60-110 MB)
rl_agent_config.json # same schema as the fused layout
tokenizer.json # HuggingFace tokenizer, at the directory root (no tokenizer/ nesting)
Each .data file must stay next to its graph, for the same reason as model.onnx.data.
There is no sibling tokenizer_config.json in this layout. The SDK instead takes CLS and SEP
from tokenizer.json's own post_processor template ([CLS] $A [SEP], or <bos> $A <eos> for
multilingual), and resolves pad/mask/unk from added_tokens by matching the usual content
spellings ([PAD]/[MASK]/[UNK] or <pad>/<mask>/<unk>); see
src/tokenization/hf_tokenizer.rs. A spelling guess alone is not safe for CLS/SEP: the
multilingual vocabulary also contains <s>/</s>, which are not its CLS/SEP. qtype's declared rank ([B] vs. [B, 1]) is read from each
graph's own input metadata rather than assumed from the layout, so the same binding code drives
both head.onnx here and model.onnx in the fused layout.
The two ort sessions (encoder, head) are run in sequence inside one LayaEngine::run call: the
encoder's last_hidden_state output is fed into the head's hidden_states input, with no
intermediate copy back into caller-visible state.
If a directory somehow has both a complete fused export and a complete split export, the fused layout wins.
5. Where the engine looks
ModelArtifacts::resolve (called for you by LayaEngine::create) looks for the directory in this
order:
LayaOptions::model_directory- the
LAYA_ONNX_DIRenvironment variable — maps to the multilingual checkpoint only (legacy) LAYA_ONNX_ROOT/<checkpoint-subdir>— the recommended multi-checkpoint path (multilingual,english, ortyped-decisions)- the local cache,
<cache_directory>/<checkpoint-subdir>.cache_directorydefaults to%LOCALAPPDATA%\laya\onnxon Windows and~/.cache/laya/onnxelsewhere. - a download into that cache, only if
LayaOptions::allow_download = true
Once a directory is chosen, the SDK detects which of the two layouts from
section 4 it holds: model.onnx present means fused; otherwise both
encoder.onnx and head.onnx must be present for the split layout. If a directory has neither —
or half of a split export (only one of encoder.onnx / head.onnx) — resolution fails with a
LayaError::MissingFile naming what was expected for both layouts, rather than a generic "model
not found". Downloading (step 5 above, and section 6)
handles either layout: it sends a HEAD request for model.onnx and fetches the fused files if it
exists, otherwise the split files if both encoder.onnx and head.onnx exist.
If none of these has the artifact, resolution fails with LayaError::ArtifactsNotFound, whose
message lists every location it tried and what to do next.
6. Downloading (future, once published)
The ONNX artifacts have not been uploaded to HuggingFace yet. The
convaiinnovations/laya repository ships PyTorch weights
(model.safetensors), which the SDK cannot load. Until the ONNX files are
uploaded, allow_download: true fails immediately with a 404 and an explanatory
message.
Downloading is opt-in. Nothing touches the network unless you ask for it. Once the files are published (see section 8), enabling the download requires only:
use laya::{ArtifactDownloadProgress, LayaEngine, LayaOptions};
use std::sync::Arc;
let engine = LayaEngine::create(LayaOptions {
allow_download: true, // fetch from Hugging Face into the cache, once
// hugging_face_token: None is the default (falls back to HF_TOKEN); Some(String::new()) disables auth
download_progress: Some(Arc::new(|p: ArtifactDownloadProgress| {
eprintln!("{}: {}/{:?}", p.file_name, p.bytes_received, p.total_bytes);
})),
..Default::default()
})?;
The SDK downloads the files of whichever layout the repository publishes (five for fused; for
split, the two graphs, their .data sidecars when present, rl_agent_config.json and
tokenizer.json) into the per-user cache, and skips the download on subsequent runs if the
directory is already complete:
- Windows:
%LOCALAPPDATA%\laya\onnx\multilingual - Linux / macOS:
~/.cache/laya/onnx/multilingual
Override the cache root with LayaOptions::cache_directory.
Without the download cargo feature (--no-default-features), setting allow_download = true
returns a clear LayaError::Download explaining that the crate was built without it, rather than
silently failing to compile a network call.
7. Using the artifact in code and tests
Create the engine in code
use laya::{LayaCheckpoint, LayaEngine, LayaOptions};
// From a known directory (all checkpoints work)
let engine = LayaEngine::from_directory("/models/laya/multilingual")?;
// English checkpoint via LayaOptions::checkpoint
let engine = LayaEngine::create(LayaOptions {
checkpoint: LayaCheckpoint::English,
model_directory: Some("/models/laya/english".into()),
intra_op_threads: Some(4), // pin in containers with a CPU limit
..Default::default()
})?;
Run the tests
Run from laya-rust/. The tests find the repository-root onnx/<checkpoint>
folders by walking up from the crate, or use LAYA_ONNX_ROOT / LAYA_ONNX_DIR.
Checkpoints whose artifacts are missing are skipped with a skip: line;
model-backed tests should run in --release:
LAYA_ONNX_ROOT=/c/models/laya cargo test --release --workspace
8. Publishing to HuggingFace (maintainers only)
Requires write access to convaiinnovations/laya.
The SDK downloader fetches five files per checkpoint from convaiinnovations/laya.
A maintainer must upload the following paths for each checkpoint:
| Checkpoint | HF path in repo |
|---|---|
multilingual |
multilingual/<files> |
english |
<files> (bundle root — no subfolder) |
typed-decisions |
typed-decisions/<files> |
The five files per checkpoint are:
model.onnx, model.onnx.data, rl_agent_config.json,
tokenizer/tokenizer.json, tokenizer/tokenizer_config.json.
onnx/ is gitignored (the weights exceed GitHub's 100 MB hard limit), so
upload must go through the HuggingFace Hub CLI or Python API directly.
Run from the repo root after completing --verify:
# huggingface-cli (pip install huggingface_hub)
huggingface-cli upload convaiinnovations/laya onnx/multilingual multilingual \
--repo-type model --token hf_...
huggingface-cli upload convaiinnovations/laya onnx/english . \
--repo-type model --token hf_...
huggingface-cli upload convaiinnovations/laya onnx/typed-decisions typed-decisions \
--repo-type model --token hf_...
Once these files are live, allow_download: true with the matching checkpoint
works without any code changes.
9. Troubleshooting
404 on download (allow_download)
The ONNX artifacts have not been published yet (see section 8). Export locally with
laya-rust/tools/export_onnx.py and point the SDK at the result via LAYA_ONNX_DIR or
LayaOptions::model_directory.
Low disk space during export
Set HF_HOME to a volume with enough free space before running the exporter
(see the disk-space table in section 1, step 3). --out controls where the ONNX output lands.
model.onnx.data not found / "missing initializer" in ONNX Runtime
The sidecar must sit beside model.onnx. Copy both files together; the SDK
error message already reminds you of this: "both model.onnx and model.onnx.data
must reside in the same directory."
"already exists, pass --force"
<out>/<name>/model.onnx is present from a previous run. Use --force to
overwrite in place, or use --out to write to a separate directory.
Wrong-checkpoint error (tokenizer/config mismatch)
tokenizer.json and tokenizer_config.json are from different checkpoints
(e.g., the tokenizer from english with the config from multilingual). The
error message names the missing token. Keep the two files together as exported.
Windows console encoding during export
The dynamo exporter logs Unicode emoji (✅, etc.) that crash a non-UTF-8 Windows
console. The script reconfigures stdout/stderr to UTF-8 with replacement
automatically. Any unrecognised emoji appears as ?; the export continues
normally.