Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
ci(dotnet): split the parity cells by layout and size each cell's timeout
kevin-gatimu1bea314b463d
diff --git a/.github/workflows/dotnet.yml b/.github/workflows/dotnet.ymlindex 1aaf4c35..efc3add1 100644--- a/.github/workflows/dotnet.yml+++ b/.github/workflows/dotnet.yml@@ -108,45 +108,119 @@ jobs:name: parity (${{ matrix.name }})runs-on: ubuntu-latestcontinue-on-error: true # advisory: see the header- timeout-minutes: 60+ # Per cell, from the matrix (see the derivation below).+ timeout-minutes: ${{ matrix.timeout }}strategy:fail-fast: false- # One cell at a time. Every cell that needs weights fetches them from the same Hugging+ # FOUR at a time (was two). Every cell that needs weights fetches them from the same Hugging# Face repo, and four in parallel -- alongside `java.yml`'s parity matrix against the same- # repo in the same run -- is what provoked the throttle. `regen_golden.py` now refuses an- # incomplete snapshot at the fetch, so this is no longer the only thing between a- # truncated download and a confusing failure; not provoking the throttle is simply- # cheaper than diagnosing it, and this lane is advisory.+ # repo in the same run -- is what once provoked the throttle. Two was the answer then; it is+ # now the wrong trade, because the matrix is seven cells (a checkpoint x layout pair for each+ # checkpoint, plus the router) and at two the critical path would be half of the summed cell+ # time, about 105 minutes on a slow host, no better than the one-cell-per-checkpoint shape+ # this replaces. What changed since: the exports and the pinned Hub downloads are cached+ # (keys below), so a warm run fetches nothing; a cold run fetches only for the cells that+ # miss, `HF_TOKEN` authenticates those fetches, and `regen_golden.py` refuses an incomplete+ # snapshot at the fetch, naming the file. Four is below the seven concurrent fetches that+ # provoked the original throttle and gives, on the slow-host estimates below, a critical path+ # of about 60 minutes (about 45 with seven slots, about 105 with two). The cells are listed+ # longest first so the long ones start together and the short ones fill the slots as they+ # free up. If a cold run is throttled, lower this to 2: the cost is wall time only, and this+ # lane is advisory.+ max-parallel: 4+ # One cell per checkpoint x layout. Both cells of a checkpoint regenerate that checkpoint's+ # goldens from Python at this commit (the recording is layout independent but each cell owns+ # its copy; nothing is shared between cells and goldens are never cached).#- # TWO, not one. One was the first attempt and it cost too much: measured from this PR's own- # cells, serialising takes the parity critical path from 10:56 to 30:08. Two keeps the peak- # well below the seven concurrent fetches that provoked the throttle while costing roughly- # half of that. And serialising is no longer the load-bearing half of the fix -- the- # download sites now refuse an incomplete snapshot AT THE FETCH, naming the file, so a- # throttled fetch fails immediately and legibly instead of three minutes later inside a- # tokenizer constructor. This only reduces how often that path is taken.- max-parallel: 2+ # `layouts` is LAYA_TEST_LAYOUTS: it narrows the theory rows that exist for both layouts+ # (long and batch predictions). The test methods that exercise one layout only carry+ # `[Trait("Layout", ...)]`: `split-only` for the split-layout theories, `any` for the+ # layout-parameterised ones. The fused cell therefore runs everything except `split-only`,+ # including the tokenizer, shortlist, error-case and multilingual-only tests, which do not+ # depend on the layout and so run once; the split cell runs `split-only` and `any`. A+ # deselected test is never discovered, so it cannot show up as a skip.+ #+ # `min-passed` and `timeout`, from the committed goldens and the lane logs in+ # researcher-agent/ci-runtime.md (slow-host mode, 4 vCPU, the worse of the two modes seen):+ # * Tests per cell, counted from the theory data with LAYA_TEST_CHECKPOINTS and+ # LAYA_TEST_LAYOUTS set (all layouts together the english count is 484, the same as the+ # a478221b log):+ # english fused 222 pass (216 rows + 6 layout independent), split 130+ # multilingual fused 323 pass, split 136+ # typed-decisions fused 228 pass, split 136+ # `min-passed` is about 80% of that, rounded down: a floor against a wrong filter or a+ # dropped group, with room for the case count to shrink. A layout whose artifacts are+ # missing is caught by the skip guard, not by this floor.+ # * Time. Tests measured on a slow host with both layouts in one cell: english 33.8 min,+ # typed-decisions about 41 (its cell hit the old 60 minute cap), multilingual 8-16; regen+ # 17.1 / 19.4 / 4.5 min; the router cell 15-22 in total. A layout cell does about half of+ # the forward passes (each success case runs once per layout); take 0.55 for the fused+ # cell, which also carries the one-off tests. Volume is expected to grow by about 47% when+ # the pending option-order and lang-temperature cases land. Projected cell: install + regen+ # + 0.55 x tests x 1.47 + 1 => english about 46, typed-decisions about 54, multilingual+ # about 17, router 22. `timeout` is that with 1.5x to 2x headroom for a slower host or a+ # cold export cache (about 3 minutes per checkpoint): 75 / 90 / 40 / 45.matrix:include:- - name: english- checkpoints: english- filter: --filter-class Laya.Tests.PredictParityTests Laya.Tests.TokenizerParityTests Laya.Tests.ShapeSweepTests Laya.Tests.ShortlistEndToEndTests- min-passed: 50- - name: multilingual- checkpoints: multilingual- filter: --filter-class Laya.Tests.PredictParityTests Laya.Tests.TokenizerParityTests Laya.Tests.ShapeSweepTests Laya.Tests.ShortlistEndToEndTests- min-passed: 50- - name: typed-decisions+ - name: typed-decisions-fused+ cache: typed-decisionscheckpoints: typed-decisions- filter: --filter-class Laya.Tests.PredictParityTests Laya.Tests.TokenizerParityTests Laya.Tests.ShapeSweepTests Laya.Tests.ShortlistEndToEndTests- min-passed: 50- # The router end-to-end tests load english and multilingual side by side.+ layouts: fused+ filter: --filter-class Laya.Tests.PredictParityTests Laya.Tests.TokenizerParityTests Laya.Tests.ShapeSweepTests Laya.Tests.ShortlistEndToEndTests --filter-not-trait Layout=split-only+ min-passed: 180+ timeout: 90+ - name: typed-decisions-split+ cache: typed-decisions+ checkpoints: typed-decisions+ layouts: split+ filter: --filter-class Laya.Tests.PredictParityTests Laya.Tests.TokenizerParityTests Laya.Tests.ShapeSweepTests Laya.Tests.ShortlistEndToEndTests --filter-trait Layout=split-only --filter-trait Layout=any+ min-passed: 105+ timeout: 90+ - name: english-fused+ cache: english+ checkpoints: english+ layouts: fused+ filter: --filter-class Laya.Tests.PredictParityTests Laya.Tests.TokenizerParityTests Laya.Tests.ShapeSweepTests Laya.Tests.ShortlistEndToEndTests --filter-not-trait Layout=split-only+ min-passed: 175+ timeout: 75+ - name: english-split+ cache: english+ checkpoints: english+ layouts: split+ filter: --filter-class Laya.Tests.PredictParityTests Laya.Tests.TokenizerParityTests Laya.Tests.ShapeSweepTests Laya.Tests.ShortlistEndToEndTests --filter-trait Layout=split-only --filter-trait Layout=any+ min-passed: 100+ timeout: 75+ # The router end-to-end tests load english and multilingual side by side; they have+ # no layout parameter, so LAYA_TEST_LAYOUTS stays unset.- name: router+ cache: routercheckpoints: english multilingual+ layouts: ""filter: --filter-class Laya.Tests.RouterEndToEndTestsmin-passed: 1+ timeout: 45+ - name: multilingual-fused+ cache: multilingual+ checkpoints: multilingual+ layouts: fused+ filter: --filter-class Laya.Tests.PredictParityTests Laya.Tests.TokenizerParityTests Laya.Tests.ShapeSweepTests Laya.Tests.ShortlistEndToEndTests --filter-not-trait Layout=split-only+ min-passed: 255+ timeout: 40+ - name: multilingual-split+ cache: multilingual+ checkpoints: multilingual+ layouts: split+ filter: --filter-class Laya.Tests.PredictParityTests Laya.Tests.TokenizerParityTests Laya.Tests.ShapeSweepTests Laya.Tests.ShortlistEndToEndTests --filter-trait Layout=split-only --filter-trait Layout=any+ min-passed: 105+ timeout: 40steps:- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1+ # So the runner's speed can be read from the log: the same job has run at twice the+ # per-token cost on some hosts, and the CPU model and core count are what tell them apart.+ - name: Runner CPU+ run: |+ nproc+ lscpu | grep -E '^(Model name|CPU\(s\)|Thread\(s\) per core|Core\(s\) per socket|Socket\(s\)|CPU max MHz)'- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0with:python-version: "3.12"@@ -167,7 +241,7 @@ jobs:uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0with:path: ~/.cache/huggingface/hub- key: laya-hf-${{ runner.os }}-${{ matrix.name }}-${{ hashFiles('laya-dotnet/tools/regen_golden.py', 'laya-dotnet/tools/export_onnx.py', 'laya-ts/scripts/export_onnx.py') }}+ key: laya-hf-${{ runner.os }}-${{ matrix.cache }}-${{ hashFiles('laya-dotnet/tools/regen_golden.py', 'laya-dotnet/tools/export_onnx.py', 'laya-ts/scripts/export_onnx.py') }}- name: Cache exported english artifactsif: ${{ contains(matrix.checkpoints, 'english') }}uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0@@ -225,6 +299,7 @@ jobs:env:LAYA_ONNX_ROOT: ${{ github.workspace }}/onnxLAYA_TEST_CHECKPOINTS: ${{ matrix.checkpoints }}+ LAYA_TEST_LAYOUTS: ${{ matrix.layouts }}run: >dotnet test --solution Laya.slnx -c Release--results-directory "$RUNNER_TEMP/trx" --report-xunit-trx --report-xunit-trx-filename parity.trxdiff --git a/laya-dotnet/README.md b/laya-dotnet/README.mdindex 02e49306..33320c5d 100644--- a/laya-dotnet/README.md+++ b/laya-dotnet/README.md@@ -988,10 +988,14 @@ together with the C# change that follows it.`.github/workflows/dotnet.yml` runs the same chain on every push and pull request (there is no pathfilter, so a change under `laya/` triggers it): the `build` job re-records the routing goldens,builds the whole solution with warnings as errors and runs the tests that need no model; one-`parity` job per checkpoint (and one for the router end-to-end tests, which need english and-multilingual together) exports that checkpoint, re-records its goldens from the Python code at-that commit and runs its model-backed classes against them. `LAYA_TEST_CHECKPOINTS` selects the-checkpoint theory rows (ordinary local runs still cover all checkpoints). Every job then runs+`parity` job per checkpoint and layout (fused, split), and one for the router end-to-end tests+(which need english and multilingual together), exports that checkpoint, re-records its goldens+from the Python code at that commit and runs its model-backed classes against them.+`LAYA_TEST_CHECKPOINTS` selects the checkpoint theory rows and `LAYA_TEST_LAYOUTS` (`fused`,+`split`, or both when unset) the layout rows of the theories that have one per layout; the+split-only tests carry `[Trait("Layout", "split-only")]` and the layout-parameterised ones+`Layout=any`, which the split cell selects with `--filter-trait` (ordinary local runs still cover+everything). Every job then runs`tools/check_test_skips.py`, because `dotnet test` reports skipped tests as success: it fails therun when any test that should have run was skipped, and tolerates only the skips for checkpointsthe job does not export. The job logs print `git diff --stat` of the regenerated goldens againstdiff --git a/laya-dotnet/tests/Laya.Tests/PredictParityTests.Batch.cs b/laya-dotnet/tests/Laya.Tests/PredictParityTests.Batch.csindex 9b798394..b53a0679 100644--- a/laya-dotnet/tests/Laya.Tests/PredictParityTests.Batch.cs+++ b/laya-dotnet/tests/Laya.Tests/PredictParityTests.Batch.cs@@ -9,7 +9,7 @@ public sealed partial class PredictParityTestsfrom checkpoint in TestArtifacts.ParityCheckpointswhere GoldenData.For(checkpoint).Availablefrom info in GoldenData.For(checkpoint).Index.Where(i => i.Kind == "batch" && !i.ExpectsError)- from split in new[] { false, true }+ from split in TestArtifacts.ParitySplitFlagsselect new object[] { checkpoint, info, split };public static IEnumerable<object[]> BatchErrorCases() =>@@ -29,6 +29,7 @@ public sealed partial class PredictParityTests[.. root.GetProperty("states").EnumerateArray().Select(GoldenData.ToClr)];[Theory]+ [Trait(TestArtifacts.LayoutTrait, TestArtifacts.AnyLayout)][MemberData(nameof(BatchSuccessCases))]public void ReproducesBatchPredictions(LayaCheckpoint checkpoint, GoldenCaseInfo info, bool split){diff --git a/laya-dotnet/tests/Laya.Tests/PredictParityTests.Long.cs b/laya-dotnet/tests/Laya.Tests/PredictParityTests.Long.csindex 099fc783..8b6f43c6 100644--- a/laya-dotnet/tests/Laya.Tests/PredictParityTests.Long.cs+++ b/laya-dotnet/tests/Laya.Tests/PredictParityTests.Long.cs@@ -9,7 +9,7 @@ public sealed partial class PredictParityTestsfrom checkpoint in TestArtifacts.ParityCheckpointswhere GoldenData.For(checkpoint).Availablefrom info in GoldenData.For(checkpoint).Index.Where(i => i.Kind == "long" && !i.ExpectsError)- from split in new[] { false, true }+ from split in TestArtifacts.ParitySplitFlagsselect new object[] { checkpoint, info, split };public static IEnumerable<object[]> LongErrorCases() =>@@ -19,6 +19,7 @@ public sealed partial class PredictParityTestsselect new object[] { checkpoint, info };[Theory]+ [Trait(TestArtifacts.LayoutTrait, TestArtifacts.AnyLayout)][MemberData(nameof(LongSuccessCases))]public void ReproducesLongPredictions(LayaCheckpoint checkpoint, GoldenCaseInfo info, bool split){diff --git a/laya-dotnet/tests/Laya.Tests/PredictParityTests.cs b/laya-dotnet/tests/Laya.Tests/PredictParityTests.csindex 8e8bc7be..81df6328 100644--- a/laya-dotnet/tests/Laya.Tests/PredictParityTests.cs+++ b/laya-dotnet/tests/Laya.Tests/PredictParityTests.cs@@ -131,6 +131,7 @@ public sealed partial class PredictParityTests(AllEnginesFixture fixture)/// Not run in CI or locally until an <c>onnx-split/</c> export exists — see the class remarks./// </summary>[Theory]+ [Trait(TestArtifacts.LayoutTrait, TestArtifacts.SplitOnly)][MemberData(nameof(AllSuccessCasesSplit))]public void ReproducesTheRecordedAnswersSplit(LayaCheckpoint checkpoint, GoldenCaseInfo info){diff --git a/laya-dotnet/tests/Laya.Tests/ShapeSweepTests.cs b/laya-dotnet/tests/Laya.Tests/ShapeSweepTests.csindex 73dc6dbc..1074c79f 100644--- a/laya-dotnet/tests/Laya.Tests/ShapeSweepTests.cs+++ b/laya-dotnet/tests/Laya.Tests/ShapeSweepTests.cs@@ -210,6 +210,7 @@ public sealed class ShapeSweepTests(AllEnginesFixture fixture)/// different artifact directory rather than a separate code path being exercised here./// </summary>[Theory]+ [Trait(TestArtifacts.LayoutTrait, TestArtifacts.SplitOnly)][MemberData(nameof(CheckpointShapesSplit))]public void RunsAtEveryShapeForAllCheckpointsSplit(LayaCheckpoint checkpoint, int seq, int batch, int markers){diff --git a/laya-dotnet/tests/Laya.Tests/TestArtifacts.cs b/laya-dotnet/tests/Laya.Tests/TestArtifacts.csindex 54b63ba1..5063fdf8 100644--- a/laya-dotnet/tests/Laya.Tests/TestArtifacts.cs+++ b/laya-dotnet/tests/Laya.Tests/TestArtifacts.cs@@ -169,6 +169,41 @@ public static class TestArtifacts}}+ /// <summary>+ /// The trait that places a layout-specific test in a CI cell: <c>Layout=split-only</c> runs+ /// against the split export only, <c>Layout=any</c> is parameterised by layout (its rows are+ /// narrowed by <see cref="ParitySplitFlags"/>), and an untagged test is fused or layout+ /// independent. A split cell runs <c>split-only</c> and <c>any</c>; a fused cell runs everything+ /// except <c>split-only</c>. Filtering by trait (rather than skipping at run time) means a+ /// deselected layout produces no skipped test.+ /// </summary>+ public const string LayoutTrait = "Layout";++ /// <summary>Value of <see cref="LayoutTrait"/> for a test that only exercises the split layout.</summary>+ public const string SplitOnly = "split-only";++ /// <summary>Value of <see cref="LayoutTrait"/> for a theory with a fused and a split row per case.</summary>+ public const string AnyLayout = "any";++ /// <summary>+ /// The <c>split</c> flag values a layout-parameterised theory yields (<c>false</c> is the fused+ /// layout, <c>true</c> the split one). <c>LAYA_TEST_LAYOUTS</c> is a space-separated list of+ /// <c>fused</c> and <c>split</c>; unset selects both, so ordinary local runs are unchanged.+ /// </summary>+ public static IEnumerable<bool> ParitySplitFlags+ {+ get+ {+ var selected = Environment.GetEnvironmentVariable("LAYA_TEST_LAYOUTS");+ if (string.IsNullOrWhiteSpace(selected))+ return [false, true];+ var names = selected.Split(' ', StringSplitOptions.RemoveEmptyEntries);+ if (names.Any(name => name != "fused" && name != "split"))+ throw new ArgumentException($"unknown LAYA_TEST_LAYOUTS: {selected}");+ return new[] { false, true }.Where(split => names.Contains(split ? "split" : "fused"));+ }+ }+// One CheckpointTestArtifacts per enum value; indexed by the enum's integer value.private static readonly CheckpointTestArtifacts[] ByCheckpoint =Enum.GetValues<LayaCheckpoint>().Select(c => new CheckpointTestArtifacts(c)).ToArray();