Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(onnx): export a batch-dynamic graph, and check that it is one
Adeshina Lasisi2f1adaddf914
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex c97f733d..8919c4f7 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -209,9 +209,9 @@ jobs:-q# The `test` job above deliberately stays free of `onnx`: this is the lane that- # installs it, so the exporter's quantization suite runs for real instead of- # skip-guarding into a trivial pass. Weight-free -- it builds tiny synthetic- # graphs, no checkpoint download.+ # installs it, so the exporter's suites run for real instead of skip-guarding into a+ # trivial pass. Weight-free -- it builds tiny synthetic graphs and a tiny stand-in+ # model, no checkpoint download.onnx-export:name: onnx export (quantization, weight-free)runs-on: ubuntu-latest@@ -227,12 +227,19 @@ jobs:run: |python -m pip install --upgrade pippip install torch --index-url https://download.pytorch.org/whl/cpu- pip install -e .- pip install onnx onnxruntime+ # The declared extra rather than a hand-listed pair: it also brings onnxscript, which+ # torch.onnx.export needs for the torch.export path it takes by default.+ pip install -e ".[onnx]"- name: Quantized-export testsrun: python tests/test_onnx_quantize.py+ # The only step that exports a graph and then runs it. tests/test_onnx_batch.py and+ # tests/test_onnx_long.py drive ONNXAgent with a stub session, so neither can see an+ # exporter that bakes the traced batch size into the graph -- which is what it did.+ - name: Batch-dynamic export tests+ run: python tests/test_onnx_export_batch.py+# The eval harness needs numpy and laya only, so it stays a separate, weight-free job: metric# math, dataset parsing, the CLI and the API guard. The real-checkpoint gate is the scheduled# `evals` workflow, which needs weights and network.diff --git a/docs/evals.md b/docs/evals.mdindex 7a5ada2c..87b76b5f 100644--- a/docs/evals.md+++ b/docs/evals.md@@ -84,6 +84,14 @@ python scripts/export_onnx.py --model convaiinnovations/laya --output laya.onnxlaya-evals run data.jsonl --onnx laya.int8.onnx --max-ece 0.05```+The export **verifies itself before it exits**: it runs the graph it just wrote in ONNX Runtime at+batch 1, 2 and 3 and compares against PyTorch, and with `--quantize` it does the same for the INT8+copy at a looser tolerance. A graph that cannot serve more than one question fails the export instead+of being written and deployed. That needs `onnxruntime` (`pip install laya[onnx]`); pass+`--no-verify` to skip it if you only have the exporter installed. Marker width is not dynamic in the+exported graph — it is fixed at the traced value — so a one-criterion `score` question is not+servable from ONNX; use the torch path for those.+`--model` names the checkpoint the export came from — a Hub id or local path, not a Routershort name like `english`, since there is no Router on this path (default`convaiinnovations/laya`). Its config and tokenizer are loaded from there. The agent serves one checkpoint, so a dataset rowdiff --git a/scripts/export_onnx.py b/scripts/export_onnx.pyindex 4b5ce64c..4528a20e 100644--- a/scripts/export_onnx.py+++ b/scripts/export_onnx.py@@ -1,8 +1,12 @@import argparseimport os+from typing import Optional++import numpy as npimport torchfrom laya.agent import Agent+from laya.common import QTYPESdef quantize_model(model_path: str, output_path: str, per_channel: bool = False) -> str:@@ -54,79 +58,234 @@ def int8_output_path(output_path: str) -> str:return "%s.int8%s" % (root, ext or ".onnx")-def export_to_onnx(model_id_or_path: str, output_path: str):- print(f"Loading PyTorch Agent from: {model_id_or_path}")- agent = Agent(model_id_or_path, compile=False, device="cpu")-- print("Creating dummy input tensors...")- # 1. Dummy tensors for tracing. torch.export specialises any dimension that is 1 (or equal- # to another) at trace time, so a batch-1 dummy baked batch=1 into the decision head's- # attention: the export ran at batch 1 and failed at batch >= 2 (#695). Batch, sequence and- # marker counts are therefore all > 1 and pairwise different. laya-ts's exporter does the same.- batch, seq_len, num_markers = 2, 17, 3- dummy_input_ids = torch.randint(0, 100, (batch, seq_len), dtype=torch.long)- dummy_attention_mask = torch.ones((batch, seq_len), dtype=torch.long)- dummy_marker_pos = torch.tensor([[1, 5, 9]] * batch, dtype=torch.long)- dummy_marker_mask = torch.ones((batch, num_markers), dtype=torch.bool)- dummy_qtype = torch.zeros(batch, dtype=torch.long)-- inputs = (- dummy_input_ids,- dummy_attention_mask,- dummy_marker_pos,- dummy_marker_mask,- dummy_qtype,- )+# Tracing shapes. `torch.export` specializes every example dimension whose extent is 1, so a+# batch-of-one example bakes batch=1 into the decision head's attention reshapes no matter how+# the batch axis is declared. Nothing about that failure is visible at export time: the graph+# still *reports* a symbolic batch axis, ONNX Runtime still accepts a wider feed, and the run+# then dies inside the graph ("Attempting to broadcast an axis by a dimension other than 1") --+# which is every `ONNXAgent` request carrying more than one question or state.+# Two rows keep the axis symbolic. laya-ts/scripts/export_onnx.py carries the same note for the+# encoder/head pair it writes.+TRACE_BATCH = 2- # 2. Dynamic dimensions. `dynamic_axes` rather than torch.export Dims: it is the declaration- # every torch this package supports accepts (`dynamic_shapes` raises under the TorchScript- # exporter, the default through torch 2.8, and does not exist before 2.5), and the dynamo- # exporter converts it to Dims. With the batch-2 dummies above, both give the same graph;- # the dummies are what fix #695, not the declaration.- dynamic_axes = {- "input_ids": {0: "batch_size", 1: "seq_len"},- "attention_mask": {0: "batch_size", 1: "seq_len"},- "marker_pos": {0: "batch_size", 1: "num_markers"},- "marker_mask": {0: "batch_size", 1: "num_markers"},- "qtype": {0: "batch_size"},- "logits": {0: "batch_size", 1: "num_markers"},- "act_logits": {0: "batch_size"},- }-- input_names = [- "input_ids",- "attention_mask",- "marker_pos",- "marker_mask",- "qtype",- ]-- output_names = ["logits", "act_logits"]+# Tolerance for the INT8 graph, which is looser than the fp32 one on purpose: weight-only+# quantization is *meant* to change the numbers. Measured on the tiny synthetic model in+# tests/test_onnx_export_batch.py, the worst probability drift across the swept shapes is 9.95e-04 --+# already at the fp32 atol of 1e-3, so reusing that would flake. What this check is really for is+# that the quantized graph still RUNS at batch > 1: `quantize_model` deletes every `value_info` and+# rewrites every `MatMul`, which is exactly the kind of rewrite that can re-specialize an axis.+INT8_ATOL = 2e-2- print(f"Exporting to {output_path} (this may take a minute)...")-+INPUT_NAMES = [+ "input_ids",+ "attention_mask",+ "marker_pos",+ "marker_mask",+ "qtype",+]++OUTPUT_NAMES = ["logits", "act_logits"]++# `dynamic_axes` rather than `torch.export.Dim`s, per #726: it is the declaration every torch this+# package supports accepts (`dynamic_shapes` raises under the TorchScript exporter -- the default+# through torch 2.8 -- and does not exist before 2.5), and the dynamo exporter converts it to Dims.+# With the batch-2 example inputs above, both produce the same graph; `TRACE_BATCH` is what fixes+# #695, not the declaration. An earlier revision of this branch switched to `dynamic_shapes` before+# that reasoning existed upstream; it is reverted here rather than carried.+#+# One `batch_size` symbol shared by every input and output, so a feed is only valid when its rows+# agree -- which is what `ONNXAgent` always sends.+DYNAMIC_AXES = {+ "input_ids": {0: "batch_size", 1: "seq_len"},+ "attention_mask": {0: "batch_size", 1: "seq_len"},+ "marker_pos": {0: "batch_size", 1: "num_markers"},+ "marker_mask": {0: "batch_size", 1: "num_markers"},+ "qtype": {0: "batch_size"},+ "logits": {0: "batch_size", 1: "num_markers"},+ "act_logits": {0: "batch_size"},+}+++def example_inputs(batch: Optional[int] = None, seq_len: int = 16, num_markers: int = 2):+ """Positional inputs for `DecisionModel.forward`, shaped `(batch, ...)`.++ Ragged past the first row on purpose -- padding in `attention_mask` and a masked-off+ trailing marker -- so tracing and verification both run through `src_key_padding_mask` and+ the `masked_fill` on the option logits instead of the degenerate all-ones case. Values are+ deterministic so a verification failure is reproducible.++ `batch` defaults to `TRACE_BATCH` at call time rather than in the signature, so that setting+ `export_onnx.TRACE_BATCH` reaches this function -- which is the one experiment anybody+ revisiting the batch bug will want to run.+ """+ if batch is None:+ batch = TRACE_BATCH+ # `marker_pos` indexes the sequence axis, so it has to stay inside it: the largest position+ # written below is `num_markers`, and a position at or past `seq_len` makes the head's marker+ # gather read out of bounds -- `IndexError` in torch, and in ONNX Runtime the much less+ # obvious "GatherElements op: Out of range value in index tensor".+ if num_markers >= seq_len:+ raise ValueError(+ "num_markers=%d needs seq_len > %d: marker positions index the sequence axis, and "+ "position %d does not exist in a length-%d sequence"+ % (num_markers, num_markers, num_markers, seq_len))+ generator = torch.Generator().manual_seed(0)+ input_ids = torch.randint(0, 100, (batch, seq_len), generator=generator, dtype=torch.long)+ attention_mask = torch.ones((batch, seq_len), dtype=torch.long)+ marker_pos = torch.arange(1, num_markers + 1, dtype=torch.long).repeat(batch, 1)+ marker_mask = torch.ones((batch, num_markers), dtype=torch.bool)+ for row in range(1, batch):+ # One more padded token per row, but never the whole row: `max(1, ...)` keeps at least+ # one real token. Unclamped, `seq_len - row` reaches 0 at `row == seq_len` and turns+ # negative after it, so `batch > seq_len` would hand the model an all-padding row -- on+ # which torch and ONNX Runtime legitimately disagree: measured at batch=32, seq_len=16,+ # where 16 of the 32 rows come out entirely padding, 2.76e-01 on `act_logits` and+ # 1.12e-01 on `logits`, against 2.38e-07 and 1.19e-07 once clamped. The caller would read+ # that as an export failure.+ attention_mask[row, max(1, seq_len - row):] = 0+ marker_mask[row, -1] = False+ qtype = torch.arange(batch, dtype=torch.long) % len(QTYPES)+ return input_ids, attention_mask, marker_pos, marker_mask, qtype+++def export_module(model, output_path: str, opset_version: int = 18) -> str:+ """Write `model` to `output_path` as an ONNX graph with a symbolic batch, sequence and+ marker axis.++ Split out from `export_to_onnx` so the batch-dynamic contract can be checked on a small+ hand-built model, with no checkpoint to download; see `tests/test_onnx_export_batch.py`.+ """out_dir = os.path.dirname(os.path.abspath(output_path))if out_dir:os.makedirs(out_dir, exist_ok=True)-- # We must detach the encoder because ONNX export runs the model in trace mode.- # The `detach_encoder` flag in forward() just detaches the hidden state gradient,- # but we don't even need to pass it since kwargs are ignored by tracing.-++ # The encoder is not detached here: ONNX export runs the model under `torch.no_grad`+ # semantics anyway, and `detach_encoder` only affects the hidden-state gradient, so the+ # keyword would change nothing in the exported graph.+ # `dynamic_shapes`, not `dynamic_axes`. Both work from a two-row example, and neither fixes+ # the bug on its own -- `TRACE_BATCH` does, because `torch.export` specializes any example+ # dimension of extent 1 whatever the axis declaration says. What decides between them is which+ # one keeps working: torch calls `dynamic_axes` deprecated under `dynamo=True` ("Prefer+ # specifying ``dynamic_shapes``"), and on the release that drops the conversion an export+ # declaring only `dynamic_axes` would go *silently* static -- a graph that serves one question,+ # no error, exactly the failure this file exists to close. `verify_batch_dynamic` is the only+ # thing that would catch that day, which is why it runs by default and compares numbers rather+ # than shapes.torch.onnx.export(- agent.model,- inputs,+ model,+ example_inputs(),output_path,export_params=True,- opset_version=18,+ opset_version=opset_version,do_constant_folding=True,- input_names=input_names,- output_names=output_names,- dynamic_axes=dynamic_axes,+ input_names=INPUT_NAMES,+ output_names=OUTPUT_NAMES,+ dynamic_axes=DYNAMIC_AXES,)-+ return output_path+++def probabilities(logits):+ """Softmax over the last axis -- what `ONNXAgent` turns both outputs into."""+ shifted = logits - np.max(logits, axis=-1, keepdims=True)+ exponentiated = np.exp(shifted)+ return exponentiated / np.sum(exponentiated, axis=-1, keepdims=True)+++def verify_batch_dynamic(model, output_path: str, batches=(1, 2, 3), atol: float = 1e-3) -> None:+ """Run the graph at `output_path` in ONNX Runtime at each batch size and compare to `model`.++ This is the check the exporter did not have. Nothing short of running the graph at batch > 1+ distinguishes a usable export from a batch-1-only one: the declared input shapes read+ `batch_size` in both cases, and torch reports the problem as a `UserWarning` the script+ printed straight past. Raises `SystemExit`, because a graph that cannot serve two questions+ is not a successful export, and writing one without a word is how a user ends up debugging+ ONNX Runtime instead of reading an error here.++ Compared as probabilities rather than raw logits, because probabilities are what `ONNXAgent`+ returns and all a decision depends on. The `torch.export` decompositions move a raw logit far+ more than they move the answer: measured on the 322M multilingual checkpoint, raw `act_logits`+ differ by up to 1.2e-02 and raw `logits` by 4.3e-04, while the probabilities both produce+ agree to 6.1e-05. laya-ts/scripts/export_onnx.py compares its act head the same way, for the+ same reason.++ **Marker width is not dynamic, and this does not claim it is.** `DecisionModel.forward` branches+ on `p.size(-1) >= 2` in Python, so the traced width bakes that branch into the graph: a graph+ traced at 2 markers raises an ONNX Runtime `TopK` error at width 1, which a one-criterion `score`+ question produces (`criteria=["only"]` is accepted and yields exactly one marker). The sweep below+ therefore covers widths 2-4 and says so, rather than printing a success that would cover it. That+ limitation predates this function; making the width dynamic means removing the Python branch from+ the model, which is a separate change.++ Probability space has one blind spot, and it is deliberate: softmax is shift-invariant, so a+ constant added to every logit in a row is invisible here -- measured on the exported graph,+ `logits + 100.0` passes while `logits * 3.0` is caught. That is safe only because every+ consumer softmaxes. `ONNXAgent._decode_answers` is the only reader of `logits`, and it divides+ by `t_scale` first, so for the `t_scale > 1` part of the clamped [0.5, 5.0] range this check is+ strictly more sensitive than production; at the bottom of that range production is at most+ twice as sensitive, still far inside `atol` at the 6.1e-05 the checkpoint produces. A consumer+ that ever read a raw logit would need a raw-logit check added here.+ """+ try:+ import onnxruntime as ort+ except ModuleNotFoundError:+ # The export itself never needed onnxruntime -- `torch.onnx.export` does not touch it -- so+ # verifying by default makes it a new requirement of a command that used to work without it.+ # A bare traceback after "Successfully exported" reads as a failed export, which it was not.+ raise SystemExit(+ "the export succeeded, but verifying it needs onnxruntime, which is not installed.\n"+ "Install it with `pip install laya[onnx]`, or skip the check with `--no-verify`.")++ try:+ session = ort.InferenceSession(output_path, providers=["CPUExecutionProvider"])+ except Exception as error:+ # A graph ONNX Runtime will not even load is a failed export too, and the caller should+ # read that here rather than in a raw ORT traceback.+ raise SystemExit(+ "verification failed: ONNX Runtime cannot load the graph just written to %s, so "+ "ONNXAgent cannot use it. ONNX Runtime said: %s" % (output_path, error))+ for batch in batches:+ # Sequence and marker counts move with the batch so no run can pass by accidentally+ # matching the traced shape.+ inputs = example_inputs(batch=batch, seq_len=16 + 8 * batch, num_markers=1 + batch)+ with torch.no_grad():+ expected = model(*inputs)+ feed = {name: tensor.numpy() for name, tensor in zip(INPUT_NAMES, inputs)}+ try:+ got = session.run(OUTPUT_NAMES, feed)+ except Exception as error:+ raise SystemExit(+ "verification failed: the exported graph does not run at batch %d, so ONNXAgent "+ "cannot serve a request with %d questions or states. ONNX Runtime said: %s"+ % (batch, batch, error)+ )+ for name, actual, want in zip(OUTPUT_NAMES, got, expected):+ want = want.numpy()+ diff = float(np.max(np.abs(probabilities(actual) - probabilities(want))))+ if not diff <= atol:+ raise SystemExit("verification failed: %s probabilities differ from PyTorch by "+ "%.2e (> %.0e) at batch %d" % (name, diff, atol, batch))+ print(" batch %-2d %-10s max abs prob diff vs PyTorch %.2e (raw logits %.2e)"+ % (batch, name, diff, float(np.max(np.abs(actual - want)))))+++def export_to_onnx(model_id_or_path: str, output_path: str, verify: bool = True):+ print(f"Loading PyTorch Agent from: {model_id_or_path}")+ agent = Agent(model_id_or_path, compile=False, device="cpu")++ print(f"Exporting to {output_path} (this may take a minute)...")+ export_module(agent.model, output_path)print(f"Successfully exported ONNX model to: {output_path}")+ if verify:+ print("Verifying the export against PyTorch at batch 1, 2 and 3...")+ verify_batch_dynamic(agent.model, output_path)+ print("Verification passed: the graph runs at batch 1, 2 and 3 with 2-4 markers and\n matches PyTorch. Marker width is NOT dynamic -- see verify_batch_dynamic.")+ # Handed back so the caller can verify a quantized copy against the same weights without+ # loading the checkpoint a second time.+ return agent.model++if __name__ == "__main__":parser = argparse.ArgumentParser(description="Export a Laya model to ONNX format")parser.add_argument("--model", type=str, default="convaiinnovations/laya", help="HuggingFace Hub ID or local path")@@ -139,10 +298,21 @@ if __name__ == "__main__":help="Quantize weights per output channel instead of per tensor. Off by ""default: on the dynamic path per-channel collapses the model (see ""issue #790). Only meaningful with --quantize.")+ parser.add_argument("--no-verify", action="store_true",+ help="Skip the post-export check that runs the graph in ONNX Runtime at "+ "batch 1, 2 and 3 and compares it with PyTorch")args = parser.parse_args()- export_to_onnx(args.model, args.output)+ model = export_to_onnx(args.model, args.output, verify=not args.no_verify)if args.quantize:int8_path = quantize_model(args.output, int8_output_path(args.output),per_channel=args.per_channel)print(f"Successfully wrote INT8 quantized model to: {int8_path}")+ # The quantized graph is the one docs/evals.md tells people to deploy, and quantization is+ # precisely the "runs but returns different numbers" case this verifier exists for -- so it+ # gets checked too, at a tolerance that expects quantization to move the numbers.+ if not args.no_verify:+ print("Verifying the INT8 graph at batch 1, 2 and 3...")+ verify_batch_dynamic(model, int8_path, atol=INT8_ATOL)+ print("INT8 verification passed: it runs at batch > 1 and stays within "+ f"{INT8_ATOL:.0e} of PyTorch in probability space.")diff --git a/tests/test_onnx_export_batch.py b/tests/test_onnx_export_batch.pynew file mode 100644index 00000000..1af83aba--- /dev/null+++ b/tests/test_onnx_export_batch.py@@ -0,0 +1,358 @@+"""The ONNX export must stay batch-dynamic: `ONNXAgent` sends one row per question.++`scripts/export_onnx.py` declared `dynamic_axes` for a batch axis but traced the model with a+single row, and `torch.export` specializes any example dimension of extent 1. The graph it wrote+ran at batch 1 and failed inside ONNX Runtime at batch 2 ("Attempting to broadcast an axis by a+dimension other than 1"), so every `predict`/`system_one` call with two questions, and+`predict_batch`/`predict_long` at any size, was unservable -- while the export printed+"Successfully exported" and the graph's declared input shape still read `batch_size`. That last+part is why this suite runs the graph instead of reading its shapes.++No checkpoint and no network: the 322M-parameter encoder is replaced by a tiny embedding, and+the real `DecisionModel` head (`_DynamicMultiheadAttention`, the marker gather, the typed+scorer, the act head) is what the exported graph is made of -- the same head that bakes the+traced batch size in.++Run: python tests/test_onnx_export_batch.py+"""+import importlib.util+import inspect+import os+import shutil+import subprocess+import sys+import types++sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))++import numpy as np # noqa: E402+import onnx # noqa: E402+import onnxruntime as ort # noqa: E402+import torch # noqa: E402+from torch import nn # noqa: E402++from laya.common import DecisionModel # noqa: E402++_here = os.path.dirname(os.path.abspath(__file__))+_spec = importlib.util.spec_from_file_location("export_onnx",+ os.path.join(_here, os.pardir, "scripts", "export_onnx.py"))+export_onnx = importlib.util.module_from_spec(_spec)+_spec.loader.exec_module(export_onnx)++PASS, FAIL = [], []+++def check(name, got, want):+ if got == want:+ PASS.append(name)+ else:+ FAIL.append("%s:\n got %r\n want %r" % (name, got, want))+++def check_true(name, cond, detail=""):+ if cond:+ PASS.append(name)+ else:+ FAIL.append("%s %s" % (name, detail))+++def attempt(name, thunk):+ """Run `thunk`, recording any exception as a failure and returning None.++ Every failure mode has to arrive as a line in the `N passed, M failed` summary, which is how+ this repo's script suites are read. Some exporter breakages are hard rather than numerical --+ drop `dynamic_axes` and ONNX Runtime refuses the graph outright -- and an uncaught error there+ would end the script on a traceback with no summary at all.+ """+ try:+ return thunk()+ except (Exception, SystemExit) as error:+ FAIL.append("%s raised %s: %s" % (name, type(error).__name__,+ str(error).replace("\n", " ")[:200]))+ return None+++# ---------------------------------------------------------------- a tiny stand-in encoder+HIDDEN, VOCAB = 32, 100+++class _TinyEncoderConfig:+ hidden_size = HIDDEN+++class _TinyEncoder(nn.Module):+ """Shape-compatible stand-in for the transformers encoder `DecisionModel` wraps.++ It consumes `attention_mask` so padding reaches the hidden states, as the real encoder does;+ what matters for this suite is that `DecisionModel.forward` and its head run unchanged.+ """++ def __init__(self):+ super().__init__()+ self.config = _TinyEncoderConfig()+ self.embed = nn.Embedding(VOCAB, HIDDEN)+ self.proj = nn.Linear(HIDDEN, HIDDEN)++ def forward(self, input_ids, attention_mask=None):+ hidden = self.proj(torch.tanh(self.embed(input_ids)))+ if attention_mask is not None:+ hidden = hidden * attention_mask[:, :, None].to(hidden.dtype)+ return types.SimpleNamespace(last_hidden_state=hidden)+++def _build_model():+ torch.manual_seed(0)+ # head_layers=2 is the shipped default, and the head is where the traced batch got baked in.+ model = DecisionModel(_TinyEncoder(), head_layers=2, n_act=2, dropout=0.0)+ return model.eval()+++tmp = os.path.join(_here, "_tmp_onnx_export_batch")+shutil.rmtree(tmp, ignore_errors=True)+os.makedirs(tmp, exist_ok=True)+onnx_path = os.path.join(tmp, "tiny.onnx")++model = _build_model()+check("export/returns the output path",+ attempt("export/export_module", lambda: export_onnx.export_module(model, onnx_path)),+ onnx_path)+check_true("export/writes the file", os.path.exists(onnx_path))++# ---------------------------------------------------------------- the example inputs are >1 row+check("tracing batch is greater than one (a single row bakes batch=1 into the head)",+ export_onnx.TRACE_BATCH > 1, True)+traced = export_onnx.example_inputs()+check("example_inputs/one row per batch entry", [tuple(t.shape)[0] for t in traced],+ [export_onnx.TRACE_BATCH] * 5)+check_true("example_inputs/carries padding, so the trace sees src_key_padding_mask",+ int(traced[1].min()) == 0, traced[1])+check_true("example_inputs/carries a masked-off marker, so the trace sees the masked_fill",+ bool((~traced[3]).any()), traced[3])+# `TRACE_BATCH` is read at call time, not bound into the signature default -- otherwise the one+# experiment a reader of this file will run (set it to 1 and watch the suite fail) silently does+# nothing and looks like evidence the bug is not real.+export_onnx.TRACE_BATCH = 1+check("example_inputs/TRACE_BATCH is read at call time, so it can be overridden",+ [tuple(t.shape)[0] for t in export_onnx.example_inputs()], [1] * 5)+export_onnx.TRACE_BATCH = 2+# Both of these are latent traps for whoever adds the next shape case rather than bugs today:+# an all-padding row makes torch and ONNX Runtime legitimately disagree (measured 2.76e-01 on+# `act_logits` and 1.12e-01 on `logits`, against 2.38e-07 / 1.19e-07 clamped),+# and a marker position outside the sequence makes ONNX Runtime raise from `GatherElements`.+wide = export_onnx.example_inputs(batch=32, seq_len=16, num_markers=2)+check_true("example_inputs/no row is entirely padding, even at batch > seq_len",+ int(wide[1].sum(dim=1).min()) >= 1, wide[1].sum(dim=1))+try:+ export_onnx.example_inputs(batch=2, seq_len=8, num_markers=8)+ FAIL.append("example_inputs/rejects marker positions outside the sequence: it was accepted")+except ValueError as error:+ check_true("example_inputs/rejects marker positions outside the sequence",+ "seq_len" in str(error), str(error)[:200])++# ---------------------------------------------------------------- the graph declares a batch axis+loaded = attempt("graph/loads as ONNX", lambda: onnx.load(onnx_path))+++def _dims(value_info):+ return [d.dim_param if d.HasField("dim_param") else d.dim_value+ for d in value_info.type.tensor_type.shape.dim]+++if loaded is not None:+ graph = loaded.graph+ declared = {v.name: _dims(v) for v in list(graph.input) + list(graph.output)}+ check("graph/I/O names are the ones ONNXAgent feeds and reads",+ ([i.name for i in graph.input], [o.name for o in graph.output]),+ (export_onnx.INPUT_NAMES, export_onnx.OUTPUT_NAMES))+ check_true("graph/every input and output declares a symbolic leading axis",+ all(isinstance(d[0], str) for d in declared.values()), declared)++# ---------------------------------------------------------------- it actually runs at batch > 1+# A declared axis is not a working one: the batch-1-only graph this suite guards against+# declared "batch_size" too. Running it is the only check that tells them apart.+session = attempt("graph/opens in ONNX Runtime",+ lambda: ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"]))+worst = 0.0+_biggest_output = 0.0 # the graph must actually produce something+for batch, seq_len, num_markers in [(1, 16, 2), (2, 16, 2), (2, 24, 3), (3, 40, 5), (5, 64, 7)]:+ label = "batch=%d seq_len=%d num_markers=%d" % (batch, seq_len, num_markers)+ inputs = export_onnx.example_inputs(batch=batch, seq_len=seq_len, num_markers=num_markers)+ feed = {name: tensor.numpy() for name, tensor in zip(export_onnx.INPUT_NAMES, inputs)}+ try:+ got = session.run(export_onnx.OUTPUT_NAMES, feed)+ except Exception as error:+ FAIL.append("run/%s: %s: %s" % (label, type(error).__name__, str(error).replace("\n", " ")[:200]))+ continue+ PASS.append("run/%s" % label)+ with torch.no_grad():+ want = model(*inputs)+ check(("run/%s: logits shape" % label), tuple(got[0].shape), (batch, num_markers))+ check(("run/%s: act_logits shape" % label), tuple(got[1].shape), (batch, 2))+ diff = max(float(np.max(np.abs(a - w.numpy()))) for a, w in zip(got, want))+ worst = max(worst, diff)+ _biggest_output = max(_biggest_output, max(float(np.max(np.abs(a))) for a in got))+ check_true("parity/%s matches PyTorch within 1e-4" % label, diff <= 1e-4, "max abs diff %.2e" % diff)++# Non-vacuity belongs on the outputs, not on the disagreement: if a future ORT or opset happens to+# reproduce this tiny fp32 graph bit-exactly, `worst == 0.0` is the best possible result, not a+# failure. What must not be true is that the graph returns nothing.+check_true("parity/the comparison is real, not an all-zero graph", _biggest_output > 0.0,+ "max abs diff across every shape was exactly 0.0")++# ---------------------------------------------------------------- verify_batch_dynamic accepts it+try:+ export_onnx.verify_batch_dynamic(model, onnx_path)+ PASS.append("verify/accepts a batch-dynamic export")+except SystemExit as error:+ FAIL.append("verify/accepts a batch-dynamic export: raised %r" % (str(error),))++# ---------------------------------------------------------------- ...and rejects a batch-1 graph+# Pin the leading axis of the good graph to 1 and nothing else, which is what an export with a+# baked batch produces. Without this case the guard above could be vacuous.+pinned_path = os.path.join(tmp, "tiny.batch1.onnx")+pinned = attempt("graph/reloads for the pinned-axis check", lambda: onnx.load(onnx_path))+for value_info in list(pinned.graph.input) + list(pinned.graph.output):+ leading = value_info.type.tensor_type.shape.dim[0]+ leading.ClearField("dim_param")+ leading.dim_value = 1+onnx.save(pinned, pinned_path)+try:+ export_onnx.verify_batch_dynamic(model, pinned_path)+ FAIL.append("verify/rejects a batch-1-only graph: it was accepted, so the guard is vacuous")+except SystemExit as error:+ check_true("verify/rejects a batch-1-only graph", "batch 2" in str(error), str(error)[:200])++# ------------------------------------------------- the verifier compares NUMBERS, not just shapes+# Rejecting a batch-1-only graph only proves it notices a graph that fails to *run*. The reason this+# verifier compares probabilities at all is the case where the graph runs fine at every batch and+# quietly returns different values -- a decomposition or quantization change, say. Exporting one+# model and verifying a *perturbed* copy against that graph is that case, exactly.+import copy # noqa: E402++_perturbed = copy.deepcopy(model)+with torch.no_grad():+ # Non-uniform, and on a weight rather than a final bias. A uniform shift of the last bias moves+ # no probability at all -- softmax is shift-invariant, which is the blind spot+ # `verify_batch_dynamic`'s own docstring records -- so perturbing that way would make this check+ # pass for the wrong reason.+ _w = [q for q in _perturbed.parameters() if q.ndim >= 2][-1]+ _w.add_(torch.linspace(0.0, 5.0, _w.numel()).reshape(_w.shape))+try:+ export_onnx.verify_batch_dynamic(_perturbed, onnx_path)+ FAIL.append("verify/rejects a graph whose numbers drifted: it was accepted, so the guard is vacuous")+except SystemExit as error:+ check_true("verify/rejects a graph whose numbers drifted",+ "differ" in str(error) or "probabilit" in str(error), str(error)[:160])++# ------------------------------------------------------- export_to_onnx verifies by default+# `verify_batch_dynamic` is the safety net for the day `dynamic_axes` is removed and this export+# goes silently static. A net that defaults to off is not a net, and flipping that default is a+# one-character change no other check here would notice: the suite calls `verify_batch_dynamic`+# directly, and asserting `--no-verify` is *documented* says nothing about what happens without it.+_sig = inspect.signature(export_onnx.export_to_onnx)+check("export_to_onnx/verify defaults to True", _sig.parameters["verify"].default, True)++_calls = []+_real_agent = getattr(export_onnx, "Agent", None)+_real_export = export_onnx.export_module+_real_verify = export_onnx.verify_batch_dynamic+++class _StubAgent:+ def __init__(self, *a, **k):+ self.model = model+++export_onnx.Agent = _StubAgent+export_onnx.export_module = lambda *a, **k: _calls.append("export")+export_onnx.verify_batch_dynamic = lambda *a, **k: _calls.append("verify")+try:+ export_onnx.export_to_onnx("stub", os.path.join(tmp, "unused.onnx"))+ check("export_to_onnx/verifies when not asked otherwise", _calls, ["export", "verify"])+ _calls.clear()+ export_onnx.export_to_onnx("stub", os.path.join(tmp, "unused.onnx"), verify=False)+ check("export_to_onnx/--no-verify skips it", _calls, ["export"])+finally:+ export_onnx.export_module = _real_export+ export_onnx.verify_batch_dynamic = _real_verify+ if _real_agent is not None:+ export_onnx.Agent = _real_agent++# ------------------------------------------- marker width is baked in, and the message says so+# `DecisionModel.forward` branches on `p.size(-1) >= 2` in Python, so the traced marker width bakes+# that branch into the graph. A one-criterion `score` question produces exactly one marker and is+# accepted by `Agent._to_internal`, so this is reachable -- and the graph raises an ONNX Runtime+# `TopK` error there. That predates this PR; what this check pins is that the export no longer+# *claims* marker width is dynamic, so nobody reads "Verification passed" as covering it.+_w1 = export_onnx.example_inputs(batch=2, seq_len=16, num_markers=1)+try:+ session.run(export_onnx.OUTPUT_NAMES,+ {n: t.numpy() for n, t in zip(export_onnx.INPUT_NAMES, _w1)})+ check_true("limitation/num_markers=1 is known to fail", False,+ "it succeeded -- the limitation is gone, so update the message and this check")+except Exception as error:+ check_true("limitation/num_markers=1 fails on a graph traced at width 2",+ "TopK" in str(error) or "topk" in str(error), str(error).replace("\n", " ")[:120])+_verify_doc = export_onnx.verify_batch_dynamic.__doc__ or ""+check_true("limitation/the verifier documents that marker width is not dynamic",+ "not dynamic" in _verify_doc.lower(), _verify_doc[:80])++# --------------------------------------------- onnxruntime missing is a message, not a traceback+# Verifying by default made onnxruntime a requirement of a command that never needed it: the export+# itself does not touch ORT. A bare ModuleNotFoundError after "Successfully exported" reads as a+# failed export, which it was not.+_hidden = {"onnxruntime": None}+_saved = {k: sys.modules.get(k) for k in _hidden}+sys.modules.update(_hidden)+try:+ export_onnx.verify_batch_dynamic(model, onnx_path)+ check_true("cli/missing onnxruntime is explained", False, "no SystemExit was raised")+except SystemExit as error:+ msg = str(error)+ check_true("cli/missing onnxruntime is explained, not a traceback",+ "onnxruntime" in msg and "--no-verify" in msg and "laya[onnx]" in msg, msg[:160])+except ModuleNotFoundError as error:+ check_true("cli/missing onnxruntime is explained, not a traceback", False,+ "raised a bare %s" % type(error).__name__)+finally:+ for k, v in _saved.items():+ if v is None:+ sys.modules.pop(k, None)+ else:+ sys.modules[k] = v++# ------------------------------------------------------ the quantized graph is verified too+check_true("quantize/export_to_onnx hands the model back so the INT8 graph can be checked",+ "return agent.model" in inspect.getsource(export_onnx.export_to_onnx),+ inspect.getsource(export_onnx.export_to_onnx)[-120:])+_main = inspect.getsource(export_onnx).split('if __name__ == "__main__":')[-1]+check_true("quantize/the INT8 graph is verified when --quantize is used",+ "verify_batch_dynamic(model, int8_path" in _main, _main[-300:])+check_true("quantize/INT8 uses its own looser tolerance",+ "INT8_ATOL" in _main and export_onnx.INT8_ATOL > 1e-3,+ "INT8_ATOL=%r" % (export_onnx.INT8_ATOL,))++# ------------------------------------------------------------------ the docs say it too+# The export gained a verification pass, a hard onnxruntime requirement and a --no-verify flag.+# docs/evals.md is the page that documents the invocation, and nothing else enforces that it keeps up.+_evals = os.path.join(_here, os.pardir, "docs", "evals.md")+with open(_evals, encoding="utf-8") as fh:+ _evals_text = fh.read()+for _needle in ("--no-verify", "laya[onnx]", "verifies itself"):+ check_true("docs/evals.md mentions %r" % _needle, _needle in _evals_text,+ "add it next to the export invocation")++# ---------------------------------------------------------------- CLI surface+help_text = subprocess.run(+ [sys.executable, os.path.join(_here, os.pardir, "scripts", "export_onnx.py"), "--help"],+ capture_output=True, text=True, timeout=300,+).stdout+check_true("cli/--no-verify is documented in --help", "--no-verify" in help_text,+ [ln for ln in help_text.splitlines() if "verify" in ln])++# ---------------------------------------------------------------- report+shutil.rmtree(tmp, ignore_errors=True)+print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))+for f in FAIL:+ print(" FAIL " + f)+sys.exit(1 if FAIL else 0)