Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(onnx): export a batch-dynamic graph, and check that it is one

Adeshina Lasisi2f1adaddf914
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index c97f733d..8919c4f7 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -209,9 +209,9 @@ jobs:
-q
# The `test` job above deliberately stays free of `onnx`: this is the lane that
- # installs it, so the exporter's quantization suite runs for real instead of
- # skip-guarding into a trivial pass. Weight-free -- it builds tiny synthetic
- # graphs, no checkpoint download.
+ # installs it, so the exporter's suites run for real instead of skip-guarding into a
+ # trivial pass. Weight-free -- it builds tiny synthetic graphs and a tiny stand-in
+ # model, no checkpoint download.
onnx-export:
name: onnx export (quantization, weight-free)
runs-on: ubuntu-latest
@@ -227,12 +227,19 @@ jobs:
run: |
python -m pip install --upgrade pip
pip install torch --index-url https://download.pytorch.org/whl/cpu
- pip install -e .
- pip install onnx onnxruntime
+ # The declared extra rather than a hand-listed pair: it also brings onnxscript, which
+ # torch.onnx.export needs for the torch.export path it takes by default.
+ pip install -e ".[onnx]"
- name: Quantized-export tests
run: python tests/test_onnx_quantize.py
+ # The only step that exports a graph and then runs it. tests/test_onnx_batch.py and
+ # tests/test_onnx_long.py drive ONNXAgent with a stub session, so neither can see an
+ # exporter that bakes the traced batch size into the graph -- which is what it did.
+ - name: Batch-dynamic export tests
+ run: python tests/test_onnx_export_batch.py
+
# The eval harness needs numpy and laya only, so it stays a separate, weight-free job: metric
# math, dataset parsing, the CLI and the API guard. The real-checkpoint gate is the scheduled
# `evals` workflow, which needs weights and network.
diff --git a/docs/evals.md b/docs/evals.md
index 7a5ada2c..87b76b5f 100644
--- a/docs/evals.md
+++ b/docs/evals.md
@@ -84,6 +84,14 @@ python scripts/export_onnx.py --model convaiinnovations/laya --output laya.onnx
laya-evals run data.jsonl --onnx laya.int8.onnx --max-ece 0.05
```
+The export **verifies itself before it exits**: it runs the graph it just wrote in ONNX Runtime at
+batch 1, 2 and 3 and compares against PyTorch, and with `--quantize` it does the same for the INT8
+copy at a looser tolerance. A graph that cannot serve more than one question fails the export instead
+of being written and deployed. That needs `onnxruntime` (`pip install laya[onnx]`); pass
+`--no-verify` to skip it if you only have the exporter installed. Marker width is not dynamic in the
+exported graph — it is fixed at the traced value — so a one-criterion `score` question is not
+servable from ONNX; use the torch path for those.
+
`--model` names the checkpoint the export came from — a Hub id or local path, not a Router
short name like `english`, since there is no Router on this path (default
`convaiinnovations/laya`). Its config and tokenizer are loaded from there. The agent serves one checkpoint, so a dataset row
diff --git a/scripts/export_onnx.py b/scripts/export_onnx.py
index 4b5ce64c..4528a20e 100644
--- a/scripts/export_onnx.py
+++ b/scripts/export_onnx.py
@@ -1,8 +1,12 @@
import argparse
import os
+from typing import Optional
+
+import numpy as np
import torch
from laya.agent import Agent
+from laya.common import QTYPES
def quantize_model(model_path: str, output_path: str, per_channel: bool = False) -> str:
@@ -54,79 +58,234 @@ def int8_output_path(output_path: str) -> str:
return "%s.int8%s" % (root, ext or ".onnx")
-def export_to_onnx(model_id_or_path: str, output_path: str):
- print(f"Loading PyTorch Agent from: {model_id_or_path}")
- agent = Agent(model_id_or_path, compile=False, device="cpu")
-
- print("Creating dummy input tensors...")
- # 1. Dummy tensors for tracing. torch.export specialises any dimension that is 1 (or equal
- # to another) at trace time, so a batch-1 dummy baked batch=1 into the decision head's
- # attention: the export ran at batch 1 and failed at batch >= 2 (#695). Batch, sequence and
- # marker counts are therefore all > 1 and pairwise different. laya-ts's exporter does the same.
- batch, seq_len, num_markers = 2, 17, 3
- dummy_input_ids = torch.randint(0, 100, (batch, seq_len), dtype=torch.long)
- dummy_attention_mask = torch.ones((batch, seq_len), dtype=torch.long)
- dummy_marker_pos = torch.tensor([[1, 5, 9]] * batch, dtype=torch.long)
- dummy_marker_mask = torch.ones((batch, num_markers), dtype=torch.bool)
- dummy_qtype = torch.zeros(batch, dtype=torch.long)
-
- inputs = (
- dummy_input_ids,
- dummy_attention_mask,
- dummy_marker_pos,
- dummy_marker_mask,
- dummy_qtype,
- )
+# Tracing shapes. `torch.export` specializes every example dimension whose extent is 1, so a
+# batch-of-one example bakes batch=1 into the decision head's attention reshapes no matter how
+# the batch axis is declared. Nothing about that failure is visible at export time: the graph
+# still *reports* a symbolic batch axis, ONNX Runtime still accepts a wider feed, and the run
+# then dies inside the graph ("Attempting to broadcast an axis by a dimension other than 1") --
+# which is every `ONNXAgent` request carrying more than one question or state.
+# Two rows keep the axis symbolic. laya-ts/scripts/export_onnx.py carries the same note for the
+# encoder/head pair it writes.
+TRACE_BATCH = 2
- # 2. Dynamic dimensions. `dynamic_axes` rather than torch.export Dims: it is the declaration
- # every torch this package supports accepts (`dynamic_shapes` raises under the TorchScript
- # exporter, the default through torch 2.8, and does not exist before 2.5), and the dynamo
- # exporter converts it to Dims. With the batch-2 dummies above, both give the same graph;
- # the dummies are what fix #695, not the declaration.
- dynamic_axes = {
- "input_ids": {0: "batch_size", 1: "seq_len"},
- "attention_mask": {0: "batch_size", 1: "seq_len"},
- "marker_pos": {0: "batch_size", 1: "num_markers"},
- "marker_mask": {0: "batch_size", 1: "num_markers"},
- "qtype": {0: "batch_size"},
- "logits": {0: "batch_size", 1: "num_markers"},
- "act_logits": {0: "batch_size"},
- }
-
- input_names = [
- "input_ids",
- "attention_mask",
- "marker_pos",
- "marker_mask",
- "qtype",
- ]
-
- output_names = ["logits", "act_logits"]
+# Tolerance for the INT8 graph, which is looser than the fp32 one on purpose: weight-only
+# quantization is *meant* to change the numbers. Measured on the tiny synthetic model in
+# tests/test_onnx_export_batch.py, the worst probability drift across the swept shapes is 9.95e-04 --
+# already at the fp32 atol of 1e-3, so reusing that would flake. What this check is really for is
+# that the quantized graph still RUNS at batch > 1: `quantize_model` deletes every `value_info` and
+# rewrites every `MatMul`, which is exactly the kind of rewrite that can re-specialize an axis.
+INT8_ATOL = 2e-2
- print(f"Exporting to {output_path} (this may take a minute)...")
-
+INPUT_NAMES = [
+ "input_ids",
+ "attention_mask",
+ "marker_pos",
+ "marker_mask",
+ "qtype",
+]
+
+OUTPUT_NAMES = ["logits", "act_logits"]
+
+# `dynamic_axes` rather than `torch.export.Dim`s, per #726: it is the declaration every torch this
+# package supports accepts (`dynamic_shapes` raises under the TorchScript exporter -- the default
+# through torch 2.8 -- and does not exist before 2.5), and the dynamo exporter converts it to Dims.
+# With the batch-2 example inputs above, both produce the same graph; `TRACE_BATCH` is what fixes
+# #695, not the declaration. An earlier revision of this branch switched to `dynamic_shapes` before
+# that reasoning existed upstream; it is reverted here rather than carried.
+#
+# One `batch_size` symbol shared by every input and output, so a feed is only valid when its rows
+# agree -- which is what `ONNXAgent` always sends.
+DYNAMIC_AXES = {
+ "input_ids": {0: "batch_size", 1: "seq_len"},
+ "attention_mask": {0: "batch_size", 1: "seq_len"},
+ "marker_pos": {0: "batch_size", 1: "num_markers"},
+ "marker_mask": {0: "batch_size", 1: "num_markers"},
+ "qtype": {0: "batch_size"},
+ "logits": {0: "batch_size", 1: "num_markers"},
+ "act_logits": {0: "batch_size"},
+}
+
+
+def example_inputs(batch: Optional[int] = None, seq_len: int = 16, num_markers: int = 2):
+ """Positional inputs for `DecisionModel.forward`, shaped `(batch, ...)`.
+
+ Ragged past the first row on purpose -- padding in `attention_mask` and a masked-off
+ trailing marker -- so tracing and verification both run through `src_key_padding_mask` and
+ the `masked_fill` on the option logits instead of the degenerate all-ones case. Values are
+ deterministic so a verification failure is reproducible.
+
+ `batch` defaults to `TRACE_BATCH` at call time rather than in the signature, so that setting
+ `export_onnx.TRACE_BATCH` reaches this function -- which is the one experiment anybody
+ revisiting the batch bug will want to run.
+ """
+ if batch is None:
+ batch = TRACE_BATCH
+ # `marker_pos` indexes the sequence axis, so it has to stay inside it: the largest position
+ # written below is `num_markers`, and a position at or past `seq_len` makes the head's marker
+ # gather read out of bounds -- `IndexError` in torch, and in ONNX Runtime the much less
+ # obvious "GatherElements op: Out of range value in index tensor".
+ if num_markers >= seq_len:
+ raise ValueError(
+ "num_markers=%d needs seq_len > %d: marker positions index the sequence axis, and "
+ "position %d does not exist in a length-%d sequence"
+ % (num_markers, num_markers, num_markers, seq_len))
+ generator = torch.Generator().manual_seed(0)
+ input_ids = torch.randint(0, 100, (batch, seq_len), generator=generator, dtype=torch.long)
+ attention_mask = torch.ones((batch, seq_len), dtype=torch.long)
+ marker_pos = torch.arange(1, num_markers + 1, dtype=torch.long).repeat(batch, 1)
+ marker_mask = torch.ones((batch, num_markers), dtype=torch.bool)
+ for row in range(1, batch):
+ # One more padded token per row, but never the whole row: `max(1, ...)` keeps at least
+ # one real token. Unclamped, `seq_len - row` reaches 0 at `row == seq_len` and turns
+ # negative after it, so `batch > seq_len` would hand the model an all-padding row -- on
+ # which torch and ONNX Runtime legitimately disagree: measured at batch=32, seq_len=16,
+ # where 16 of the 32 rows come out entirely padding, 2.76e-01 on `act_logits` and
+ # 1.12e-01 on `logits`, against 2.38e-07 and 1.19e-07 once clamped. The caller would read
+ # that as an export failure.
+ attention_mask[row, max(1, seq_len - row):] = 0
+ marker_mask[row, -1] = False
+ qtype = torch.arange(batch, dtype=torch.long) % len(QTYPES)
+ return input_ids, attention_mask, marker_pos, marker_mask, qtype
+
+
+def export_module(model, output_path: str, opset_version: int = 18) -> str:
+ """Write `model` to `output_path` as an ONNX graph with a symbolic batch, sequence and
+ marker axis.
+
+ Split out from `export_to_onnx` so the batch-dynamic contract can be checked on a small
+ hand-built model, with no checkpoint to download; see `tests/test_onnx_export_batch.py`.
+ """
out_dir = os.path.dirname(os.path.abspath(output_path))
if out_dir:
os.makedirs(out_dir, exist_ok=True)
-
- # We must detach the encoder because ONNX export runs the model in trace mode.
- # The `detach_encoder` flag in forward() just detaches the hidden state gradient,
- # but we don't even need to pass it since kwargs are ignored by tracing.
-
+
+ # The encoder is not detached here: ONNX export runs the model under `torch.no_grad`
+ # semantics anyway, and `detach_encoder` only affects the hidden-state gradient, so the
+ # keyword would change nothing in the exported graph.
+ # `dynamic_shapes`, not `dynamic_axes`. Both work from a two-row example, and neither fixes
+ # the bug on its own -- `TRACE_BATCH` does, because `torch.export` specializes any example
+ # dimension of extent 1 whatever the axis declaration says. What decides between them is which
+ # one keeps working: torch calls `dynamic_axes` deprecated under `dynamo=True` ("Prefer
+ # specifying ``dynamic_shapes``"), and on the release that drops the conversion an export
+ # declaring only `dynamic_axes` would go *silently* static -- a graph that serves one question,
+ # no error, exactly the failure this file exists to close. `verify_batch_dynamic` is the only
+ # thing that would catch that day, which is why it runs by default and compares numbers rather
+ # than shapes.
torch.onnx.export(
- agent.model,
- inputs,
+ model,
+ example_inputs(),
output_path,
export_params=True,
- opset_version=18,
+ opset_version=opset_version,
do_constant_folding=True,
- input_names=input_names,
- output_names=output_names,
- dynamic_axes=dynamic_axes,
+ input_names=INPUT_NAMES,
+ output_names=OUTPUT_NAMES,
+ dynamic_axes=DYNAMIC_AXES,
)
-
+ return output_path
+
+
+def probabilities(logits):
+ """Softmax over the last axis -- what `ONNXAgent` turns both outputs into."""
+ shifted = logits - np.max(logits, axis=-1, keepdims=True)
+ exponentiated = np.exp(shifted)
+ return exponentiated / np.sum(exponentiated, axis=-1, keepdims=True)
+
+
+def verify_batch_dynamic(model, output_path: str, batches=(1, 2, 3), atol: float = 1e-3) -> None:
+ """Run the graph at `output_path` in ONNX Runtime at each batch size and compare to `model`.
+
+ This is the check the exporter did not have. Nothing short of running the graph at batch > 1
+ distinguishes a usable export from a batch-1-only one: the declared input shapes read
+ `batch_size` in both cases, and torch reports the problem as a `UserWarning` the script
+ printed straight past. Raises `SystemExit`, because a graph that cannot serve two questions
+ is not a successful export, and writing one without a word is how a user ends up debugging
+ ONNX Runtime instead of reading an error here.
+
+ Compared as probabilities rather than raw logits, because probabilities are what `ONNXAgent`
+ returns and all a decision depends on. The `torch.export` decompositions move a raw logit far
+ more than they move the answer: measured on the 322M multilingual checkpoint, raw `act_logits`
+ differ by up to 1.2e-02 and raw `logits` by 4.3e-04, while the probabilities both produce
+ agree to 6.1e-05. laya-ts/scripts/export_onnx.py compares its act head the same way, for the
+ same reason.
+
+ **Marker width is not dynamic, and this does not claim it is.** `DecisionModel.forward` branches
+ on `p.size(-1) >= 2` in Python, so the traced width bakes that branch into the graph: a graph
+ traced at 2 markers raises an ONNX Runtime `TopK` error at width 1, which a one-criterion `score`
+ question produces (`criteria=["only"]` is accepted and yields exactly one marker). The sweep below
+ therefore covers widths 2-4 and says so, rather than printing a success that would cover it. That
+ limitation predates this function; making the width dynamic means removing the Python branch from
+ the model, which is a separate change.
+
+ Probability space has one blind spot, and it is deliberate: softmax is shift-invariant, so a
+ constant added to every logit in a row is invisible here -- measured on the exported graph,
+ `logits + 100.0` passes while `logits * 3.0` is caught. That is safe only because every
+ consumer softmaxes. `ONNXAgent._decode_answers` is the only reader of `logits`, and it divides
+ by `t_scale` first, so for the `t_scale > 1` part of the clamped [0.5, 5.0] range this check is
+ strictly more sensitive than production; at the bottom of that range production is at most
+ twice as sensitive, still far inside `atol` at the 6.1e-05 the checkpoint produces. A consumer
+ that ever read a raw logit would need a raw-logit check added here.
+ """
+ try:
+ import onnxruntime as ort
+ except ModuleNotFoundError:
+ # The export itself never needed onnxruntime -- `torch.onnx.export` does not touch it -- so
+ # verifying by default makes it a new requirement of a command that used to work without it.
+ # A bare traceback after "Successfully exported" reads as a failed export, which it was not.
+ raise SystemExit(
+ "the export succeeded, but verifying it needs onnxruntime, which is not installed.\n"
+ "Install it with `pip install laya[onnx]`, or skip the check with `--no-verify`.")
+
+ try:
+ session = ort.InferenceSession(output_path, providers=["CPUExecutionProvider"])
+ except Exception as error:
+ # A graph ONNX Runtime will not even load is a failed export too, and the caller should
+ # read that here rather than in a raw ORT traceback.
+ raise SystemExit(
+ "verification failed: ONNX Runtime cannot load the graph just written to %s, so "
+ "ONNXAgent cannot use it. ONNX Runtime said: %s" % (output_path, error))
+ for batch in batches:
+ # Sequence and marker counts move with the batch so no run can pass by accidentally
+ # matching the traced shape.
+ inputs = example_inputs(batch=batch, seq_len=16 + 8 * batch, num_markers=1 + batch)
+ with torch.no_grad():
+ expected = model(*inputs)
+ feed = {name: tensor.numpy() for name, tensor in zip(INPUT_NAMES, inputs)}
+ try:
+ got = session.run(OUTPUT_NAMES, feed)
+ except Exception as error:
+ raise SystemExit(
+ "verification failed: the exported graph does not run at batch %d, so ONNXAgent "
+ "cannot serve a request with %d questions or states. ONNX Runtime said: %s"
+ % (batch, batch, error)
+ )
+ for name, actual, want in zip(OUTPUT_NAMES, got, expected):
+ want = want.numpy()
+ diff = float(np.max(np.abs(probabilities(actual) - probabilities(want))))
+ if not diff <= atol:
+ raise SystemExit("verification failed: %s probabilities differ from PyTorch by "
+ "%.2e (> %.0e) at batch %d" % (name, diff, atol, batch))
+ print(" batch %-2d %-10s max abs prob diff vs PyTorch %.2e (raw logits %.2e)"
+ % (batch, name, diff, float(np.max(np.abs(actual - want)))))
+
+
+def export_to_onnx(model_id_or_path: str, output_path: str, verify: bool = True):
+ print(f"Loading PyTorch Agent from: {model_id_or_path}")
+ agent = Agent(model_id_or_path, compile=False, device="cpu")
+
+ print(f"Exporting to {output_path} (this may take a minute)...")
+ export_module(agent.model, output_path)
print(f"Successfully exported ONNX model to: {output_path}")
+ if verify:
+ print("Verifying the export against PyTorch at batch 1, 2 and 3...")
+ verify_batch_dynamic(agent.model, output_path)
+ print("Verification passed: the graph runs at batch 1, 2 and 3 with 2-4 markers and\n matches PyTorch. Marker width is NOT dynamic -- see verify_batch_dynamic.")
+ # Handed back so the caller can verify a quantized copy against the same weights without
+ # loading the checkpoint a second time.
+ return agent.model
+
+
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Export a Laya model to ONNX format")
parser.add_argument("--model", type=str, default="convaiinnovations/laya", help="HuggingFace Hub ID or local path")
@@ -139,10 +298,21 @@ if __name__ == "__main__":
help="Quantize weights per output channel instead of per tensor. Off by "
"default: on the dynamic path per-channel collapses the model (see "
"issue #790). Only meaningful with --quantize.")
+ parser.add_argument("--no-verify", action="store_true",
+ help="Skip the post-export check that runs the graph in ONNX Runtime at "
+ "batch 1, 2 and 3 and compares it with PyTorch")
args = parser.parse_args()
- export_to_onnx(args.model, args.output)
+ model = export_to_onnx(args.model, args.output, verify=not args.no_verify)
if args.quantize:
int8_path = quantize_model(args.output, int8_output_path(args.output),
per_channel=args.per_channel)
print(f"Successfully wrote INT8 quantized model to: {int8_path}")
+ # The quantized graph is the one docs/evals.md tells people to deploy, and quantization is
+ # precisely the "runs but returns different numbers" case this verifier exists for -- so it
+ # gets checked too, at a tolerance that expects quantization to move the numbers.
+ if not args.no_verify:
+ print("Verifying the INT8 graph at batch 1, 2 and 3...")
+ verify_batch_dynamic(model, int8_path, atol=INT8_ATOL)
+ print("INT8 verification passed: it runs at batch > 1 and stays within "
+ f"{INT8_ATOL:.0e} of PyTorch in probability space.")
diff --git a/tests/test_onnx_export_batch.py b/tests/test_onnx_export_batch.py
new file mode 100644
index 00000000..1af83aba
--- /dev/null
+++ b/tests/test_onnx_export_batch.py
@@ -0,0 +1,358 @@
+"""The ONNX export must stay batch-dynamic: `ONNXAgent` sends one row per question.
+
+`scripts/export_onnx.py` declared `dynamic_axes` for a batch axis but traced the model with a
+single row, and `torch.export` specializes any example dimension of extent 1. The graph it wrote
+ran at batch 1 and failed inside ONNX Runtime at batch 2 ("Attempting to broadcast an axis by a
+dimension other than 1"), so every `predict`/`system_one` call with two questions, and
+`predict_batch`/`predict_long` at any size, was unservable -- while the export printed
+"Successfully exported" and the graph's declared input shape still read `batch_size`. That last
+part is why this suite runs the graph instead of reading its shapes.
+
+No checkpoint and no network: the 322M-parameter encoder is replaced by a tiny embedding, and
+the real `DecisionModel` head (`_DynamicMultiheadAttention`, the marker gather, the typed
+scorer, the act head) is what the exported graph is made of -- the same head that bakes the
+traced batch size in.
+
+Run: python tests/test_onnx_export_batch.py
+"""
+import importlib.util
+import inspect
+import os
+import shutil
+import subprocess
+import sys
+import types
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import numpy as np # noqa: E402
+import onnx # noqa: E402
+import onnxruntime as ort # noqa: E402
+import torch # noqa: E402
+from torch import nn # noqa: E402
+
+from laya.common import DecisionModel # noqa: E402
+
+_here = os.path.dirname(os.path.abspath(__file__))
+_spec = importlib.util.spec_from_file_location("export_onnx",
+ os.path.join(_here, os.pardir, "scripts", "export_onnx.py"))
+export_onnx = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(export_onnx)
+
+PASS, FAIL = [], []
+
+
+def check(name, got, want):
+ if got == want:
+ PASS.append(name)
+ else:
+ FAIL.append("%s:\n got %r\n want %r" % (name, got, want))
+
+
+def check_true(name, cond, detail=""):
+ if cond:
+ PASS.append(name)
+ else:
+ FAIL.append("%s %s" % (name, detail))
+
+
+def attempt(name, thunk):
+ """Run `thunk`, recording any exception as a failure and returning None.
+
+ Every failure mode has to arrive as a line in the `N passed, M failed` summary, which is how
+ this repo's script suites are read. Some exporter breakages are hard rather than numerical --
+ drop `dynamic_axes` and ONNX Runtime refuses the graph outright -- and an uncaught error there
+ would end the script on a traceback with no summary at all.
+ """
+ try:
+ return thunk()
+ except (Exception, SystemExit) as error:
+ FAIL.append("%s raised %s: %s" % (name, type(error).__name__,
+ str(error).replace("\n", " ")[:200]))
+ return None
+
+
+# ---------------------------------------------------------------- a tiny stand-in encoder
+HIDDEN, VOCAB = 32, 100
+
+
+class _TinyEncoderConfig:
+ hidden_size = HIDDEN
+
+
+class _TinyEncoder(nn.Module):
+ """Shape-compatible stand-in for the transformers encoder `DecisionModel` wraps.
+
+ It consumes `attention_mask` so padding reaches the hidden states, as the real encoder does;
+ what matters for this suite is that `DecisionModel.forward` and its head run unchanged.
+ """
+
+ def __init__(self):
+ super().__init__()
+ self.config = _TinyEncoderConfig()
+ self.embed = nn.Embedding(VOCAB, HIDDEN)
+ self.proj = nn.Linear(HIDDEN, HIDDEN)
+
+ def forward(self, input_ids, attention_mask=None):
+ hidden = self.proj(torch.tanh(self.embed(input_ids)))
+ if attention_mask is not None:
+ hidden = hidden * attention_mask[:, :, None].to(hidden.dtype)
+ return types.SimpleNamespace(last_hidden_state=hidden)
+
+
+def _build_model():
+ torch.manual_seed(0)
+ # head_layers=2 is the shipped default, and the head is where the traced batch got baked in.
+ model = DecisionModel(_TinyEncoder(), head_layers=2, n_act=2, dropout=0.0)
+ return model.eval()
+
+
+tmp = os.path.join(_here, "_tmp_onnx_export_batch")
+shutil.rmtree(tmp, ignore_errors=True)
+os.makedirs(tmp, exist_ok=True)
+onnx_path = os.path.join(tmp, "tiny.onnx")
+
+model = _build_model()
+check("export/returns the output path",
+ attempt("export/export_module", lambda: export_onnx.export_module(model, onnx_path)),
+ onnx_path)
+check_true("export/writes the file", os.path.exists(onnx_path))
+
+# ---------------------------------------------------------------- the example inputs are >1 row
+check("tracing batch is greater than one (a single row bakes batch=1 into the head)",
+ export_onnx.TRACE_BATCH > 1, True)
+traced = export_onnx.example_inputs()
+check("example_inputs/one row per batch entry", [tuple(t.shape)[0] for t in traced],
+ [export_onnx.TRACE_BATCH] * 5)
+check_true("example_inputs/carries padding, so the trace sees src_key_padding_mask",
+ int(traced[1].min()) == 0, traced[1])
+check_true("example_inputs/carries a masked-off marker, so the trace sees the masked_fill",
+ bool((~traced[3]).any()), traced[3])
+# `TRACE_BATCH` is read at call time, not bound into the signature default -- otherwise the one
+# experiment a reader of this file will run (set it to 1 and watch the suite fail) silently does
+# nothing and looks like evidence the bug is not real.
+export_onnx.TRACE_BATCH = 1
+check("example_inputs/TRACE_BATCH is read at call time, so it can be overridden",
+ [tuple(t.shape)[0] for t in export_onnx.example_inputs()], [1] * 5)
+export_onnx.TRACE_BATCH = 2
+# Both of these are latent traps for whoever adds the next shape case rather than bugs today:
+# an all-padding row makes torch and ONNX Runtime legitimately disagree (measured 2.76e-01 on
+# `act_logits` and 1.12e-01 on `logits`, against 2.38e-07 / 1.19e-07 clamped),
+# and a marker position outside the sequence makes ONNX Runtime raise from `GatherElements`.
+wide = export_onnx.example_inputs(batch=32, seq_len=16, num_markers=2)
+check_true("example_inputs/no row is entirely padding, even at batch > seq_len",
+ int(wide[1].sum(dim=1).min()) >= 1, wide[1].sum(dim=1))
+try:
+ export_onnx.example_inputs(batch=2, seq_len=8, num_markers=8)
+ FAIL.append("example_inputs/rejects marker positions outside the sequence: it was accepted")
+except ValueError as error:
+ check_true("example_inputs/rejects marker positions outside the sequence",
+ "seq_len" in str(error), str(error)[:200])
+
+# ---------------------------------------------------------------- the graph declares a batch axis
+loaded = attempt("graph/loads as ONNX", lambda: onnx.load(onnx_path))
+
+
+def _dims(value_info):
+ return [d.dim_param if d.HasField("dim_param") else d.dim_value
+ for d in value_info.type.tensor_type.shape.dim]
+
+
+if loaded is not None:
+ graph = loaded.graph
+ declared = {v.name: _dims(v) for v in list(graph.input) + list(graph.output)}
+ check("graph/I/O names are the ones ONNXAgent feeds and reads",
+ ([i.name for i in graph.input], [o.name for o in graph.output]),
+ (export_onnx.INPUT_NAMES, export_onnx.OUTPUT_NAMES))
+ check_true("graph/every input and output declares a symbolic leading axis",
+ all(isinstance(d[0], str) for d in declared.values()), declared)
+
+# ---------------------------------------------------------------- it actually runs at batch > 1
+# A declared axis is not a working one: the batch-1-only graph this suite guards against
+# declared "batch_size" too. Running it is the only check that tells them apart.
+session = attempt("graph/opens in ONNX Runtime",
+ lambda: ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"]))
+worst = 0.0
+_biggest_output = 0.0 # the graph must actually produce something
+for batch, seq_len, num_markers in [(1, 16, 2), (2, 16, 2), (2, 24, 3), (3, 40, 5), (5, 64, 7)]:
+ label = "batch=%d seq_len=%d num_markers=%d" % (batch, seq_len, num_markers)
+ inputs = export_onnx.example_inputs(batch=batch, seq_len=seq_len, num_markers=num_markers)
+ feed = {name: tensor.numpy() for name, tensor in zip(export_onnx.INPUT_NAMES, inputs)}
+ try:
+ got = session.run(export_onnx.OUTPUT_NAMES, feed)
+ except Exception as error:
+ FAIL.append("run/%s: %s: %s" % (label, type(error).__name__, str(error).replace("\n", " ")[:200]))
+ continue
+ PASS.append("run/%s" % label)
+ with torch.no_grad():
+ want = model(*inputs)
+ check(("run/%s: logits shape" % label), tuple(got[0].shape), (batch, num_markers))
+ check(("run/%s: act_logits shape" % label), tuple(got[1].shape), (batch, 2))
+ diff = max(float(np.max(np.abs(a - w.numpy()))) for a, w in zip(got, want))
+ worst = max(worst, diff)
+ _biggest_output = max(_biggest_output, max(float(np.max(np.abs(a))) for a in got))
+ check_true("parity/%s matches PyTorch within 1e-4" % label, diff <= 1e-4, "max abs diff %.2e" % diff)
+
+# Non-vacuity belongs on the outputs, not on the disagreement: if a future ORT or opset happens to
+# reproduce this tiny fp32 graph bit-exactly, `worst == 0.0` is the best possible result, not a
+# failure. What must not be true is that the graph returns nothing.
+check_true("parity/the comparison is real, not an all-zero graph", _biggest_output > 0.0,
+ "max abs diff across every shape was exactly 0.0")
+
+# ---------------------------------------------------------------- verify_batch_dynamic accepts it
+try:
+ export_onnx.verify_batch_dynamic(model, onnx_path)
+ PASS.append("verify/accepts a batch-dynamic export")
+except SystemExit as error:
+ FAIL.append("verify/accepts a batch-dynamic export: raised %r" % (str(error),))
+
+# ---------------------------------------------------------------- ...and rejects a batch-1 graph
+# Pin the leading axis of the good graph to 1 and nothing else, which is what an export with a
+# baked batch produces. Without this case the guard above could be vacuous.
+pinned_path = os.path.join(tmp, "tiny.batch1.onnx")
+pinned = attempt("graph/reloads for the pinned-axis check", lambda: onnx.load(onnx_path))
+for value_info in list(pinned.graph.input) + list(pinned.graph.output):
+ leading = value_info.type.tensor_type.shape.dim[0]
+ leading.ClearField("dim_param")
+ leading.dim_value = 1
+onnx.save(pinned, pinned_path)
+try:
+ export_onnx.verify_batch_dynamic(model, pinned_path)
+ FAIL.append("verify/rejects a batch-1-only graph: it was accepted, so the guard is vacuous")
+except SystemExit as error:
+ check_true("verify/rejects a batch-1-only graph", "batch 2" in str(error), str(error)[:200])
+
+# ------------------------------------------------- the verifier compares NUMBERS, not just shapes
+# Rejecting a batch-1-only graph only proves it notices a graph that fails to *run*. The reason this
+# verifier compares probabilities at all is the case where the graph runs fine at every batch and
+# quietly returns different values -- a decomposition or quantization change, say. Exporting one
+# model and verifying a *perturbed* copy against that graph is that case, exactly.
+import copy # noqa: E402
+
+_perturbed = copy.deepcopy(model)
+with torch.no_grad():
+ # Non-uniform, and on a weight rather than a final bias. A uniform shift of the last bias moves
+ # no probability at all -- softmax is shift-invariant, which is the blind spot
+ # `verify_batch_dynamic`'s own docstring records -- so perturbing that way would make this check
+ # pass for the wrong reason.
+ _w = [q for q in _perturbed.parameters() if q.ndim >= 2][-1]
+ _w.add_(torch.linspace(0.0, 5.0, _w.numel()).reshape(_w.shape))
+try:
+ export_onnx.verify_batch_dynamic(_perturbed, onnx_path)
+ FAIL.append("verify/rejects a graph whose numbers drifted: it was accepted, so the guard is vacuous")
+except SystemExit as error:
+ check_true("verify/rejects a graph whose numbers drifted",
+ "differ" in str(error) or "probabilit" in str(error), str(error)[:160])
+
+# ------------------------------------------------------- export_to_onnx verifies by default
+# `verify_batch_dynamic` is the safety net for the day `dynamic_axes` is removed and this export
+# goes silently static. A net that defaults to off is not a net, and flipping that default is a
+# one-character change no other check here would notice: the suite calls `verify_batch_dynamic`
+# directly, and asserting `--no-verify` is *documented* says nothing about what happens without it.
+_sig = inspect.signature(export_onnx.export_to_onnx)
+check("export_to_onnx/verify defaults to True", _sig.parameters["verify"].default, True)
+
+_calls = []
+_real_agent = getattr(export_onnx, "Agent", None)
+_real_export = export_onnx.export_module
+_real_verify = export_onnx.verify_batch_dynamic
+
+
+class _StubAgent:
+ def __init__(self, *a, **k):
+ self.model = model
+
+
+export_onnx.Agent = _StubAgent
+export_onnx.export_module = lambda *a, **k: _calls.append("export")
+export_onnx.verify_batch_dynamic = lambda *a, **k: _calls.append("verify")
+try:
+ export_onnx.export_to_onnx("stub", os.path.join(tmp, "unused.onnx"))
+ check("export_to_onnx/verifies when not asked otherwise", _calls, ["export", "verify"])
+ _calls.clear()
+ export_onnx.export_to_onnx("stub", os.path.join(tmp, "unused.onnx"), verify=False)
+ check("export_to_onnx/--no-verify skips it", _calls, ["export"])
+finally:
+ export_onnx.export_module = _real_export
+ export_onnx.verify_batch_dynamic = _real_verify
+ if _real_agent is not None:
+ export_onnx.Agent = _real_agent
+
+# ------------------------------------------- marker width is baked in, and the message says so
+# `DecisionModel.forward` branches on `p.size(-1) >= 2` in Python, so the traced marker width bakes
+# that branch into the graph. A one-criterion `score` question produces exactly one marker and is
+# accepted by `Agent._to_internal`, so this is reachable -- and the graph raises an ONNX Runtime
+# `TopK` error there. That predates this PR; what this check pins is that the export no longer
+# *claims* marker width is dynamic, so nobody reads "Verification passed" as covering it.
+_w1 = export_onnx.example_inputs(batch=2, seq_len=16, num_markers=1)
+try:
+ session.run(export_onnx.OUTPUT_NAMES,
+ {n: t.numpy() for n, t in zip(export_onnx.INPUT_NAMES, _w1)})
+ check_true("limitation/num_markers=1 is known to fail", False,
+ "it succeeded -- the limitation is gone, so update the message and this check")
+except Exception as error:
+ check_true("limitation/num_markers=1 fails on a graph traced at width 2",
+ "TopK" in str(error) or "topk" in str(error), str(error).replace("\n", " ")[:120])
+_verify_doc = export_onnx.verify_batch_dynamic.__doc__ or ""
+check_true("limitation/the verifier documents that marker width is not dynamic",
+ "not dynamic" in _verify_doc.lower(), _verify_doc[:80])
+
+# --------------------------------------------- onnxruntime missing is a message, not a traceback
+# Verifying by default made onnxruntime a requirement of a command that never needed it: the export
+# itself does not touch ORT. A bare ModuleNotFoundError after "Successfully exported" reads as a
+# failed export, which it was not.
+_hidden = {"onnxruntime": None}
+_saved = {k: sys.modules.get(k) for k in _hidden}
+sys.modules.update(_hidden)
+try:
+ export_onnx.verify_batch_dynamic(model, onnx_path)
+ check_true("cli/missing onnxruntime is explained", False, "no SystemExit was raised")
+except SystemExit as error:
+ msg = str(error)
+ check_true("cli/missing onnxruntime is explained, not a traceback",
+ "onnxruntime" in msg and "--no-verify" in msg and "laya[onnx]" in msg, msg[:160])
+except ModuleNotFoundError as error:
+ check_true("cli/missing onnxruntime is explained, not a traceback", False,
+ "raised a bare %s" % type(error).__name__)
+finally:
+ for k, v in _saved.items():
+ if v is None:
+ sys.modules.pop(k, None)
+ else:
+ sys.modules[k] = v
+
+# ------------------------------------------------------ the quantized graph is verified too
+check_true("quantize/export_to_onnx hands the model back so the INT8 graph can be checked",
+ "return agent.model" in inspect.getsource(export_onnx.export_to_onnx),
+ inspect.getsource(export_onnx.export_to_onnx)[-120:])
+_main = inspect.getsource(export_onnx).split('if __name__ == "__main__":')[-1]
+check_true("quantize/the INT8 graph is verified when --quantize is used",
+ "verify_batch_dynamic(model, int8_path" in _main, _main[-300:])
+check_true("quantize/INT8 uses its own looser tolerance",
+ "INT8_ATOL" in _main and export_onnx.INT8_ATOL > 1e-3,
+ "INT8_ATOL=%r" % (export_onnx.INT8_ATOL,))
+
+# ------------------------------------------------------------------ the docs say it too
+# The export gained a verification pass, a hard onnxruntime requirement and a --no-verify flag.
+# docs/evals.md is the page that documents the invocation, and nothing else enforces that it keeps up.
+_evals = os.path.join(_here, os.pardir, "docs", "evals.md")
+with open(_evals, encoding="utf-8") as fh:
+ _evals_text = fh.read()
+for _needle in ("--no-verify", "laya[onnx]", "verifies itself"):
+ check_true("docs/evals.md mentions %r" % _needle, _needle in _evals_text,
+ "add it next to the export invocation")
+
+# ---------------------------------------------------------------- CLI surface
+help_text = subprocess.run(
+ [sys.executable, os.path.join(_here, os.pardir, "scripts", "export_onnx.py"), "--help"],
+ capture_output=True, text=True, timeout=300,
+).stdout
+check_true("cli/--no-verify is documented in --help", "--no-verify" in help_text,
+ [ln for ln in help_text.splitlines() if "verify" in ln])
+
+# ---------------------------------------------------------------- report
+shutil.rmtree(tmp, ignore_errors=True)
+print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
+for f in FAIL:
+ print(" FAIL " + f)
+sys.exit(1 if FAIL else 0)