Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
ci(rust): regenerate Python goldens and reject skipped parity tests
kevin-gatimu073503ac8dab
diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.ymlnew file mode 100644index 00000000..a06b7bf7--- /dev/null+++ b/.github/workflows/rust.yml@@ -0,0 +1,77 @@+name: Rust++# Python is the product; Rust follows it. Report parity drift without blocking+# Python improvements. Always regenerate fixtures from this checkout, never cache+# goldens or quietly pass model-backed tests with missing artifacts.+on:+ push:+ pull_request:+ workflow_dispatch:++permissions:+ contents: read++concurrency:+ group: rust-${{ github.event_name == 'pull_request' && github.ref || github.sha }}+ cancel-in-progress: ${{ github.event_name == 'pull_request' }}++env:+ USE_TF: "0"+ USE_TORCH: "1"+ TOKENIZERS_PARALLELISM: "false"+ CARGO_BUILD_JOBS: "4"+ OMP_NUM_THREADS: "4"+ MKL_NUM_THREADS: "4"++jobs:+ parity:+ name: build + Python parity (all checkpoints)+ runs-on: ubuntu-latest+ continue-on-error: true+ timeout-minutes: 120+ steps:+ - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0+ - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0+ with:+ python-version: "3.12"+ cache: pip+ cache-dependency-path: laya-rust/tools/requirements-regen.txt+ - name: Install regeneration dependencies+ run: |+ python -m pip install --upgrade pip+ pip install "torch==2.14.0" --index-url https://download.pytorch.org/whl/cpu+ pip install -r laya-rust/tools/requirements-regen.txt+ - name: Test regeneration cache and skip guard+ run: python laya-rust/tools/test_ci_tools.py+ - name: Regenerate all goldens from Python at this commit+ run: python laya-rust/tools/regen_golden.py+ - name: Golden drift vs. committed (informational)+ run: |+ git --no-pager diff --stat -- laya-rust/tests/golden+ git status --short -- laya-rust/tests/golden+ - name: Install stable Rust+ run: rustup update stable --no-self-update && rustup default stable+ - name: Cache Cargo dependencies and build output (not goldens)+ uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0+ with:+ path: |+ ~/.cargo/registry+ ~/.cargo/git+ laya-rust/target+ key: rust-${{ runner.os }}-${{ hashFiles('laya-rust/Cargo.lock') }}+ - name: Cargo tests (serial model loads)+ working-directory: laya-rust+ shell: bash+ run: |+ set -o pipefail+ cargo test --workspace --locked -- --nocapture --test-threads=1 2>&1 | tee cargo-test.log+ - name: No test was skipped+ if: ${{ !cancelled() }}+ run: python laya-rust/tools/check_test_skips.py laya-rust/cargo-test.log --min-passed 200+ - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2+ if: ${{ failure() }}+ with:+ name: rust-parity-diagnostics+ path: |+ laya-rust/cargo-test.log+ laya-rust/tests/goldendiff --git a/laya-rust/CHANGELOG.md b/laya-rust/CHANGELOG.mdindex ccdef381..a76e6c61 100644--- a/laya-rust/CHANGELOG.md+++ b/laya-rust/CHANGELOG.md@@ -10,6 +10,8 @@ First release: a port of the Laya Python SDK, tested against laya (Python) 0.3.2### Added+- Advisory parity CI on every push and PR: regenerate fixtures from the current Python checkout,+ test all checkpoints, and reject skipped tests. No golden cache or crate publication.- `LayaEngine`: `choice`, `score` and `noul` predictions over the exported ONNX checkpoints(english, multilingual, typed-decisions); all questions for a state run in one batched forward pass.- Loads both ONNX layouts: the fused `model.onnx` and the split `encoder.onnx` + `head.onnx`.diff --git a/laya-rust/README.md b/laya-rust/README.mdindex bd0dac88..f53ebb11 100644--- a/laya-rust/README.md+++ b/laya-rust/README.md@@ -551,10 +551,34 @@ Golden vectors are read from `tests/golden/<checkpoint>/` (recorded from Pythonroot looking for `onnx/<checkpoint>/`. Tests whose artifacts are missing print a `skip:` line andreturn, rather than failing — a machine with no exported checkpoints still gets a green Tier-1 run.-Model loads are ~1.3–1.7 GB each; the heavier tiers should be run with `--release`:+The advisory [Rust parity lane](../.github/workflows/rust.yml) runs on **every push and PR**,+with no path filter. It regenerates fixtures from the Python `laya/` at that commit using+[`tools/regen_golden.py`](tools/regen_golden.py), then runs the whole Cargo workspace.+Goldens are never cached. A `skip:` diagnostic or ignored test makes the lane red, even when+Cargo counts a missing-model early return as a pass. Job-level `continue-on-error` keeps Python+improvements unblocked; Rust drift needs a follow-up port. This workflow never publishes crates.++To reproduce it, install CPU torch and [`tools/requirements-regen.txt`](tools/requirements-regen.txt),+then run from the repository root:++```bash+python laya-rust/tools/test_ci_tools.py+python laya-rust/tools/regen_golden.py+cd laya-rust+cargo test --workspace --locked -- --nocapture --test-threads=1 2>&1 | tee cargo-test.log+python tools/check_test_skips.py cargo-test.log --min-passed 200+```++Use `--artifacts-root` for exports outside the checkout, with `LAYA_ONNX_ROOT=<root>/onnx`+and `LAYA_ONNX_SPLIT_ROOT=<root>/onnx-split`. Regeneration reuses only complete exports whose+stamp matches the pinned checkpoint, exporters, toolchain and Python model definition.+The fused exporter is shared with `laya-dotnet/tools/export_onnx.py`; split exports use+`laya-ts/scripts/export_onnx.py`.++Model loads are ~1.3–1.7 GB each; run model-backed tests serially (optionally with `--release`):```bash-LAYA_ONNX_ROOT=/path/to/onnx cargo test --release+LAYA_ONNX_ROOT=/path/to/onnx cargo test --release -- --test-threads=1````ModelArtifacts`'s own download-path unit tests (404 / 401 / 403 handling, the `.part`-then-renamediff --git a/laya-rust/tests/common/test_artifacts.rs b/laya-rust/tests/common/test_artifacts.rsindex d97191e1..c1a051b8 100644--- a/laya-rust/tests/common/test_artifacts.rs+++ b/laya-rust/tests/common/test_artifacts.rs@@ -58,13 +58,17 @@ pub fn has_model(checkpoint: LayaCheckpoint) -> bool {/// Resolve the split-layout artifact directory for `checkpoint`, or `None` if it cannot be/// found.///-/// Unlike [`locate`], the split layout (laya-ts's exporter output) has no env-var override — per-/// the task brief, tests for it look only in a fixed location relative to the crate: walking up-/// from `CARGO_MANIFEST_DIR` looking for `onnx-split/<checkpoint>/`. Absence is expected on most-/// machines and every split-layout test skips rather than fails when this returns `None`, exactly-/// like the fused ones do via [`locate`].+/// `LAYA_ONNX_SPLIT_ROOT` selects the parent of the checkpoint directories when exports live+/// outside the checkout. Otherwise walk up from `CARGO_MANIFEST_DIR` for `onnx-split/<checkpoint>`.+/// Missing exports skip locally; the CI output guard rejects every such skip.pub fn locate_split(checkpoint: LayaCheckpoint) -> Option<PathBuf> {let subfolder = checkpoint.subdir();+ if let Ok(root) = std::env::var("LAYA_ONNX_SPLIT_ROOT")+ && !root.trim().is_empty()+ {+ let candidate = PathBuf::from(root).join(subfolder);+ return candidate.join("rl_agent_config.json").is_file().then_some(candidate);+ }let mut dir = Some(Path::new(env!("CARGO_MANIFEST_DIR")).to_path_buf());while let Some(d) = dir {let candidate = d.join("onnx-split").join(subfolder);diff --git a/laya-rust/tools/check_test_skips.py b/laya-rust/tools/check_test_skips.pynew file mode 100644index 00000000..ae734337--- /dev/null+++ b/laya-rust/tools/check_test_skips.py@@ -0,0 +1,59 @@+"""Fail closed on cargo test skips, failures, empty or incomplete output.++Capture BOTH stdout and stderr with `cargo test --workspace --locked --+--nocapture --test-threads=1`. Rust's fixture helpers return early (libtest counts+that as a pass), so their `skip:` diagnostics must also be checked.++ python laya-rust/tools/check_test_skips.py cargo-test.log --min-passed 1+"""++import argparse+import re+from pathlib import Path++SUMMARY = re.compile(+ r"^test result: (ok|FAILED)\. (\d+) passed; (\d+) failed; (\d+) ignored; "+ r"(\d+) measured; (\d+) filtered out;", re.MULTILINE,+)+SKIP = re.compile(r"\bskip(?:ped)?\s*:", re.IGNORECASE)+ERROR = re.compile(r"^(?:error:|error\[|error: test failed)", re.MULTILINE)+RUN = re.compile(r"^\s*(?:Running .+ \(.+\)|Doc-tests .+)$", re.MULTILINE)+++def check(text, minimum):+ summaries = SUMMARY.findall(text)+ passed = sum(int(s[1]) for s in summaries)+ failed = sum(int(s[2]) for s in summaries)+ ignored = sum(int(s[3]) for s in summaries)+ measured = sum(int(s[4]) for s in summaries)+ filtered = sum(int(s[5]) for s in summaries)+ skips = [line for line in text.splitlines() if SKIP.search(line)]+ print("%d passed; %d failed; %d ignored; %d skip diagnostics; %d filtered" %+ (passed, failed, ignored, len(skips), filtered))+ for line in skips:+ print(line)+ # Filtered or measured tests are not an all-tests parity run either.+ unknown = len(re.findall(r"^test result:", text, re.MULTILINE)) != len(summaries)+ incomplete = len(RUN.findall(text)) > len(summaries)+ bad = (not summaries or unknown or incomplete or passed < minimum or failed or ignored or measured or filtered+ or skips or ERROR.search(text) or any(s[0] != "ok" for s in summaries))+ return 1 if bad else 0+++def main():+ ap = argparse.ArgumentParser(description=__doc__)+ ap.add_argument("log")+ ap.add_argument("--min-passed", type=int, default=1)+ args = ap.parse_args()+ if args.min_passed < 1:+ ap.error("--min-passed must be positive")+ try:+ text = Path(args.log).read_text(encoding="utf-8")+ except (OSError, UnicodeError) as e:+ print("Cannot read cargo test output: %s" % e)+ return 1+ return check(text, args.min_passed)+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/laya-rust/tools/regen_golden.py b/laya-rust/tools/regen_golden.pynew file mode 100644index 00000000..9788c1b9--- /dev/null+++ b/laya-rust/tools/regen_golden.py@@ -0,0 +1,162 @@+"""Regenerate Rust parity fixtures from the Python laya package at this checkout.++Uses the same pinned checkpoint and exporters as the .NET parity lane. Exports may+be reused only when complete and stamped with the current exporter/model inputs;+goldens are ALWAYS rewritten, never read as an input or cached.++ python laya-rust/tools/regen_golden.py+ python laya-rust/tools/regen_golden.py --checkpoint routing+ python laya-rust/tools/regen_golden.py --checkpoint english --artifacts-root D:/laya-artifacts++Install requirements-regen.txt and CPU torch first. The tests discover fused+exports via LAYA_ONNX_ROOT and split exports via LAYA_ONNX_SPLIT_ROOT.+"""++import argparse+import hashlib+import json+import os+import shutil+import subprocess+import sys++from google.protobuf.message import DecodeError++TOOLS = os.path.dirname(os.path.abspath(__file__))+REPO = os.path.dirname(os.path.dirname(TOOLS))+sys.path.insert(0, REPO)++HF_REPO = "convaiinnovations/laya"+HF_REVISION = "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851"+CHECKPOINTS = ["english", "multilingual", "typed-decisions"]+EXPORT_FUSED = os.path.join(REPO, "laya-dotnet", "tools", "export_onnx.py")+EXPORT_SPLIT = os.path.join(REPO, "laya-ts", "scripts", "export_onnx.py")+EXPORT_INPUTS = [+ EXPORT_FUSED, EXPORT_SPLIT, os.path.join(TOOLS, "requirements-regen.txt"),+ os.path.join(REPO, "laya", "common.py"),+]+STAMP = ".regen-stamp.json"+FUSED_FILES = [+ "model.onnx", "model.onnx.data", "rl_agent_config.json",+ os.path.join("tokenizer", "tokenizer.json"), "fixtures.npz",+]+SPLIT_FILES = ["encoder.onnx", "head.onnx", "rl_agent_config.json", "tokenizer.json"]+++def inputs_hash(revision):+ h = hashlib.sha256(revision.encode("utf-8"))+ for path in EXPORT_INPUTS:+ with open(path, "rb") as f:+ h.update(f.read().replace(b"\r\n", b"\n"))+ return h.hexdigest()+++def is_valid(path, files, stamp):+ """Reject stale exports, missing graphs and truncated external-data sidecars."""+ try:+ if not all(os.path.getsize(os.path.join(path, f)) > 0 for f in files):+ return False+ with open(os.path.join(path, STAMP), encoding="utf-8") as f:+ if json.load(f).get("inputs_sha256") != stamp:+ return False+ import onnx++ for file in files:+ if not file.endswith(".onnx"):+ continue+ model = onnx.load(os.path.join(path, file), load_external_data=False)+ for tensor in model.graph.initializer:+ if tensor.data_location != onnx.TensorProto.EXTERNAL:+ continue+ data = {entry.key: entry.value for entry in tensor.external_data}+ size = os.path.getsize(os.path.join(path, data["location"]))+ if size <= 0 or size < int(data.get("offset", 0)) + int(data.get("length", 0)):+ return False+ return True+ except (OSError, ValueError, KeyError, DecodeError):+ return False+++def run(cmd, env=None):+ print("\n$ " + " ".join(cmd), flush=True)+ subprocess.run(cmd, check=True, cwd=REPO, env=env)+++def snapshot(name, revision):+ from huggingface_hub import snapshot_download+ from laya.router import DEFAULT_MODELS++ repo, subfolder = DEFAULT_MODELS[name]+ if repo != HF_REPO:+ raise SystemExit("checkpoint repository changed; update HF_REPO/HF_REVISION")+ prefix = subfolder + "/" if subfolder else ""+ root = snapshot_download(repo, revision=revision, allow_patterns=[+ prefix + n for n in ("rl_agent_config.json", "model.safetensors", "config.json",+ "tokenizer.json", "tokenizer/*", "encoder/*")+ ])+ return os.path.join(root, subfolder) if subfolder else root+++def publish(partial, final, stamp):+ with open(os.path.join(partial, STAMP), "w", encoding="utf-8") as f:+ json.dump({"inputs_sha256": stamp}, f)+ if os.path.exists(final):+ shutil.rmtree(final)+ os.makedirs(os.path.dirname(final), exist_ok=True)+ os.replace(partial, final)+++def regen_checkpoint(name, root, revision, force):+ stamp = inputs_hash(revision)+ fused = os.path.join(root, "onnx", name)+ split = os.path.join(root, "onnx-split", name)+ need_fused = force or not is_valid(fused, FUSED_FILES, stamp)+ need_split = force or not is_valid(split, SPLIT_FILES, stamp)+ if need_fused or need_split:+ model_dir = snapshot(name, revision)+ env = dict(os.environ, HF_HUB_OFFLINE="1",+ PYTHONPATH=REPO + os.pathsep + os.environ.get("PYTHONPATH", ""))+ if need_fused:+ staging = os.path.join(root, ".partial-rust", "onnx")+ shutil.rmtree(staging, ignore_errors=True)+ run([sys.executable, EXPORT_FUSED, "--model", name, "--out", staging,+ "--revision", revision], env)+ publish(os.path.join(staging, name), fused, stamp)+ if need_split:+ staging = os.path.join(root, ".partial-rust", "onnx-split", name)+ shutil.rmtree(staging, ignore_errors=True)+ run([sys.executable, EXPORT_SPLIT, "--model-dir", model_dir,+ "--out-dir", staging], env)+ publish(staging, split, stamp)+ run([sys.executable, os.path.join(TOOLS, "dump_golden.py"),+ "--checkpoint", name, "--onnx", fused])+++def main():+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ ap.add_argument("--checkpoint", action="append", choices=CHECKPOINTS + ["routing", "all"])+ ap.add_argument("--artifacts-root", default=REPO)+ ap.add_argument("--revision", default=HF_REVISION)+ ap.add_argument("--force-export", action="store_true")+ args = ap.parse_args()+ selected = args.checkpoint or ["all"]+ if "all" in selected:+ selected = CHECKPOINTS + ["routing"]+ root = os.path.abspath(args.artifacts_root)+ os.environ.setdefault("PYTHONUTF8", "1")+ try:+ for name in CHECKPOINTS:+ if name in selected:+ regen_checkpoint(name, root, args.revision, args.force_export)+ if "routing" in selected:+ run([sys.executable, os.path.join(TOOLS, "dump_routing_golden.py"), "--force"])+ except subprocess.CalledProcessError as e:+ print("FAILED: %s" % " ".join(e.cmd), file=sys.stderr)+ return e.returncode or 1+ finally:+ shutil.rmtree(os.path.join(root, ".partial-rust"), ignore_errors=True)+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/laya-rust/tools/requirements-regen.txt b/laya-rust/tools/requirements-regen.txtnew file mode 100644index 00000000..0ee1241c--- /dev/null+++ b/laya-rust/tools/requirements-regen.txt@@ -0,0 +1,9 @@+# Same pinned regeneration toolchain as the upstream .NET lane.+# Install CPU torch separately: pip install "torch==2.14.0" --index-url https://download.pytorch.org/whl/cpu+transformers==5.17.0+safetensors==0.8.0+huggingface_hub==1.33.0+numpy==2.5.3+onnx==1.23.0+onnxruntime==1.30.0+onnxscript==0.7.2diff --git a/laya-rust/tools/test_ci_tools.py b/laya-rust/tools/test_ci_tools.pynew file mode 100644index 00000000..be3bc738--- /dev/null+++ b/laya-rust/tools/test_ci_tools.py@@ -0,0 +1,122 @@+"""Model-free regressions: python laya-rust/tools/test_ci_tools.py."""++import contextlib+import io+import types+import unittest+from unittest.mock import mock_open, patch++from google.protobuf.message import DecodeError++import check_test_skips as guard+import regen_golden as regen+++def summary(passed=10, failed=0, ignored=0, filtered=0):+ outcome = "FAILED" if failed else "ok"+ return ("test result: %s. %d passed; %d failed; %d ignored; 0 measured; %d filtered out;\n"+ % (outcome, passed, failed, ignored, filtered))+++class SkipGuardTests(unittest.TestCase):+ def check(self, text, minimum=1):+ with contextlib.redirect_stdout(io.StringIO()):+ return guard.check(text, minimum)++ def test_passed_run_including_empty_sample_and_doc_binaries(self):+ self.assertEqual(0, self.check(summary() + summary(0)))++ def test_early_returns_counted_as_passes_are_rejected(self):+ for reason in [+ "skip: no ONNX artifacts for 'english'",+ "skip: no split-layout ONNX artifacts for 'multilingual'",+ "skip: no golden data for 'typed-decisions'",+ "test parity ... skip: unexpected reason",+ ]:+ with self.subTest(reason=reason):+ self.assertEqual(1, self.check(reason + "\n" + summary()))++ def test_ignored_and_failed_tests_are_rejected(self):+ for text in [summary(ignored=1), summary(failed=1), summary(filtered=1)]:+ with self.subTest(text=text):+ self.assertEqual(1, self.check(text))++ def test_empty_wrong_filter_and_build_errors_fail_closed(self):+ for text in ["", summary(0), summary(1), summary() + "error: compiler failed\n"]:+ with self.subTest(text=text):+ self.assertEqual(1, self.check(text, minimum=2))++ def test_unknown_or_incomplete_runs_fail_closed(self):+ for text in [+ summary() + "test result: Unknown. 1 passed;\n",+ " Running tests/one.rs (target/one)\n" + summary()+ + " Running tests/two.rs (target/two)\n",+ ]:+ with self.subTest(text=text):+ self.assertEqual(1, self.check(text))++ def test_missing_output_fails(self):+ with patch("sys.argv", ["guard", "missing.log"]), patch.object(+ guard.Path, "read_text", side_effect=FileNotFoundError("missing")+ ), contextlib.redirect_stdout(io.StringIO()):+ self.assertEqual(1, guard.main())+++class ExportCacheTests(unittest.TestCase):+ def validate(self, stamp="expected", sidecar_size=100, length=100, broken_graph=False):+ import onnx++ tensor = onnx.TensorProto(data_location=onnx.TensorProto.EXTERNAL)+ for key, value in [("location", "encoder.onnx.data"), ("length", str(length))]:+ entry = tensor.external_data.add()+ entry.key, entry.value = key, value+ model = types.SimpleNamespace(graph=types.SimpleNamespace(initializer=[tensor]))++ def size(path):+ if path.endswith(".data"):+ if sidecar_size is None:+ raise FileNotFoundError(path)+ return sidecar_size+ return 100++ with patch("builtins.open", mock_open(read_data='{"inputs_sha256": "%s"}' % stamp)):+ with patch.object(regen.os.path, "getsize", side_effect=size):+ with patch.object(onnx, "load", return_value=model,+ side_effect=DecodeError("broken") if broken_graph else None):+ return regen.is_valid("artifacts", ["encoder.onnx"], "expected")++ def test_complete_cache_is_reused(self):+ self.assertTrue(self.validate())++ def test_stale_stamp_is_rejected(self):+ self.assertFalse(self.validate(stamp="old"))++ def test_missing_empty_or_truncated_sidecar_is_rejected(self):+ for size in [None, 0, 99]:+ with self.subTest(size=size):+ self.assertFalse(self.validate(sidecar_size=size))++ def test_malformed_graph_is_rejected(self):+ self.assertFalse(self.validate(broken_graph=True))++ def test_cached_exports_still_regenerate_goldens(self):+ with patch.object(regen, "is_valid", return_value=True), patch.object(+ regen, "inputs_hash", return_value="stamp"+ ), patch.object(regen, "run") as run, patch.object(regen, "snapshot") as snapshot:+ regen.regen_checkpoint("english", "artifacts", "revision", False)+ snapshot.assert_not_called()+ run.assert_called_once()+ self.assertIn("dump_golden.py", run.call_args.args[0][1])+ self.assertIn("--checkpoint", run.call_args.args[0])++ def test_routing_regeneration_is_unconditional(self):+ with patch("sys.argv", ["regen", "--checkpoint", "routing"]), patch.object(+ regen, "run"+ ) as run, patch.object(regen.shutil, "rmtree"):+ self.assertEqual(0, regen.main())+ self.assertIn("dump_routing_golden.py", run.call_args.args[0][1])+ self.assertIn("--force", run.call_args.args[0])+++if __name__ == "__main__":+ unittest.main()