Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

ci(dotnet): split the parity cells by layout and size each cell's timeout

kevin-gatimu1bea314b463d
diff --git a/.github/workflows/dotnet.yml b/.github/workflows/dotnet.yml
index 1aaf4c35..efc3add1 100644
--- a/.github/workflows/dotnet.yml
+++ b/.github/workflows/dotnet.yml
@@ -108,45 +108,119 @@ jobs:
name: parity (${{ matrix.name }})
runs-on: ubuntu-latest
continue-on-error: true # advisory: see the header
- timeout-minutes: 60
+ # Per cell, from the matrix (see the derivation below).
+ timeout-minutes: ${{ matrix.timeout }}
strategy:
fail-fast: false
- # One cell at a time. Every cell that needs weights fetches them from the same Hugging
+ # FOUR at a time (was two). Every cell that needs weights fetches them from the same Hugging
# Face repo, and four in parallel -- alongside `java.yml`'s parity matrix against the same
- # repo in the same run -- is what provoked the throttle. `regen_golden.py` now refuses an
- # incomplete snapshot at the fetch, so this is no longer the only thing between a
- # truncated download and a confusing failure; not provoking the throttle is simply
- # cheaper than diagnosing it, and this lane is advisory.
+ # repo in the same run -- is what once provoked the throttle. Two was the answer then; it is
+ # now the wrong trade, because the matrix is seven cells (a checkpoint x layout pair for each
+ # checkpoint, plus the router) and at two the critical path would be half of the summed cell
+ # time, about 105 minutes on a slow host, no better than the one-cell-per-checkpoint shape
+ # this replaces. What changed since: the exports and the pinned Hub downloads are cached
+ # (keys below), so a warm run fetches nothing; a cold run fetches only for the cells that
+ # miss, `HF_TOKEN` authenticates those fetches, and `regen_golden.py` refuses an incomplete
+ # snapshot at the fetch, naming the file. Four is below the seven concurrent fetches that
+ # provoked the original throttle and gives, on the slow-host estimates below, a critical path
+ # of about 60 minutes (about 45 with seven slots, about 105 with two). The cells are listed
+ # longest first so the long ones start together and the short ones fill the slots as they
+ # free up. If a cold run is throttled, lower this to 2: the cost is wall time only, and this
+ # lane is advisory.
+ max-parallel: 4
+ # One cell per checkpoint x layout. Both cells of a checkpoint regenerate that checkpoint's
+ # goldens from Python at this commit (the recording is layout independent but each cell owns
+ # its copy; nothing is shared between cells and goldens are never cached).
#
- # TWO, not one. One was the first attempt and it cost too much: measured from this PR's own
- # cells, serialising takes the parity critical path from 10:56 to 30:08. Two keeps the peak
- # well below the seven concurrent fetches that provoked the throttle while costing roughly
- # half of that. And serialising is no longer the load-bearing half of the fix -- the
- # download sites now refuse an incomplete snapshot AT THE FETCH, naming the file, so a
- # throttled fetch fails immediately and legibly instead of three minutes later inside a
- # tokenizer constructor. This only reduces how often that path is taken.
- max-parallel: 2
+ # `layouts` is LAYA_TEST_LAYOUTS: it narrows the theory rows that exist for both layouts
+ # (long and batch predictions). The test methods that exercise one layout only carry
+ # `[Trait("Layout", ...)]`: `split-only` for the split-layout theories, `any` for the
+ # layout-parameterised ones. The fused cell therefore runs everything except `split-only`,
+ # including the tokenizer, shortlist, error-case and multilingual-only tests, which do not
+ # depend on the layout and so run once; the split cell runs `split-only` and `any`. A
+ # deselected test is never discovered, so it cannot show up as a skip.
+ #
+ # `min-passed` and `timeout`, from the committed goldens and the lane logs in
+ # researcher-agent/ci-runtime.md (slow-host mode, 4 vCPU, the worse of the two modes seen):
+ # * Tests per cell, counted from the theory data with LAYA_TEST_CHECKPOINTS and
+ # LAYA_TEST_LAYOUTS set (all layouts together the english count is 484, the same as the
+ # a478221b log):
+ # english fused 222 pass (216 rows + 6 layout independent), split 130
+ # multilingual fused 323 pass, split 136
+ # typed-decisions fused 228 pass, split 136
+ # `min-passed` is about 80% of that, rounded down: a floor against a wrong filter or a
+ # dropped group, with room for the case count to shrink. A layout whose artifacts are
+ # missing is caught by the skip guard, not by this floor.
+ # * Time. Tests measured on a slow host with both layouts in one cell: english 33.8 min,
+ # typed-decisions about 41 (its cell hit the old 60 minute cap), multilingual 8-16; regen
+ # 17.1 / 19.4 / 4.5 min; the router cell 15-22 in total. A layout cell does about half of
+ # the forward passes (each success case runs once per layout); take 0.55 for the fused
+ # cell, which also carries the one-off tests. Volume is expected to grow by about 47% when
+ # the pending option-order and lang-temperature cases land. Projected cell: install + regen
+ # + 0.55 x tests x 1.47 + 1 => english about 46, typed-decisions about 54, multilingual
+ # about 17, router 22. `timeout` is that with 1.5x to 2x headroom for a slower host or a
+ # cold export cache (about 3 minutes per checkpoint): 75 / 90 / 40 / 45.
matrix:
include:
- - name: english
- checkpoints: english
- filter: --filter-class Laya.Tests.PredictParityTests Laya.Tests.TokenizerParityTests Laya.Tests.ShapeSweepTests Laya.Tests.ShortlistEndToEndTests
- min-passed: 50
- - name: multilingual
- checkpoints: multilingual
- filter: --filter-class Laya.Tests.PredictParityTests Laya.Tests.TokenizerParityTests Laya.Tests.ShapeSweepTests Laya.Tests.ShortlistEndToEndTests
- min-passed: 50
- - name: typed-decisions
+ - name: typed-decisions-fused
+ cache: typed-decisions
checkpoints: typed-decisions
- filter: --filter-class Laya.Tests.PredictParityTests Laya.Tests.TokenizerParityTests Laya.Tests.ShapeSweepTests Laya.Tests.ShortlistEndToEndTests
- min-passed: 50
- # The router end-to-end tests load english and multilingual side by side.
+ layouts: fused
+ filter: --filter-class Laya.Tests.PredictParityTests Laya.Tests.TokenizerParityTests Laya.Tests.ShapeSweepTests Laya.Tests.ShortlistEndToEndTests --filter-not-trait Layout=split-only
+ min-passed: 180
+ timeout: 90
+ - name: typed-decisions-split
+ cache: typed-decisions
+ checkpoints: typed-decisions
+ layouts: split
+ filter: --filter-class Laya.Tests.PredictParityTests Laya.Tests.TokenizerParityTests Laya.Tests.ShapeSweepTests Laya.Tests.ShortlistEndToEndTests --filter-trait Layout=split-only --filter-trait Layout=any
+ min-passed: 105
+ timeout: 90
+ - name: english-fused
+ cache: english
+ checkpoints: english
+ layouts: fused
+ filter: --filter-class Laya.Tests.PredictParityTests Laya.Tests.TokenizerParityTests Laya.Tests.ShapeSweepTests Laya.Tests.ShortlistEndToEndTests --filter-not-trait Layout=split-only
+ min-passed: 175
+ timeout: 75
+ - name: english-split
+ cache: english
+ checkpoints: english
+ layouts: split
+ filter: --filter-class Laya.Tests.PredictParityTests Laya.Tests.TokenizerParityTests Laya.Tests.ShapeSweepTests Laya.Tests.ShortlistEndToEndTests --filter-trait Layout=split-only --filter-trait Layout=any
+ min-passed: 100
+ timeout: 75
+ # The router end-to-end tests load english and multilingual side by side; they have
+ # no layout parameter, so LAYA_TEST_LAYOUTS stays unset.
- name: router
+ cache: router
checkpoints: english multilingual
+ layouts: ""
filter: --filter-class Laya.Tests.RouterEndToEndTests
min-passed: 1
+ timeout: 45
+ - name: multilingual-fused
+ cache: multilingual
+ checkpoints: multilingual
+ layouts: fused
+ filter: --filter-class Laya.Tests.PredictParityTests Laya.Tests.TokenizerParityTests Laya.Tests.ShapeSweepTests Laya.Tests.ShortlistEndToEndTests --filter-not-trait Layout=split-only
+ min-passed: 255
+ timeout: 40
+ - name: multilingual-split
+ cache: multilingual
+ checkpoints: multilingual
+ layouts: split
+ filter: --filter-class Laya.Tests.PredictParityTests Laya.Tests.TokenizerParityTests Laya.Tests.ShapeSweepTests Laya.Tests.ShortlistEndToEndTests --filter-trait Layout=split-only --filter-trait Layout=any
+ min-passed: 105
+ timeout: 40
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+ # So the runner's speed can be read from the log: the same job has run at twice the
+ # per-token cost on some hosts, and the CPU model and core count are what tell them apart.
+ - name: Runner CPU
+ run: |
+ nproc
+ lscpu | grep -E '^(Model name|CPU\(s\)|Thread\(s\) per core|Core\(s\) per socket|Socket\(s\)|CPU max MHz)'
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
@@ -167,7 +241,7 @@ jobs:
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/.cache/huggingface/hub
- key: laya-hf-${{ runner.os }}-${{ matrix.name }}-${{ hashFiles('laya-dotnet/tools/regen_golden.py', 'laya-dotnet/tools/export_onnx.py', 'laya-ts/scripts/export_onnx.py') }}
+ key: laya-hf-${{ runner.os }}-${{ matrix.cache }}-${{ hashFiles('laya-dotnet/tools/regen_golden.py', 'laya-dotnet/tools/export_onnx.py', 'laya-ts/scripts/export_onnx.py') }}
- name: Cache exported english artifacts
if: ${{ contains(matrix.checkpoints, 'english') }}
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
@@ -225,6 +299,7 @@ jobs:
env:
LAYA_ONNX_ROOT: ${{ github.workspace }}/onnx
LAYA_TEST_CHECKPOINTS: ${{ matrix.checkpoints }}
+ LAYA_TEST_LAYOUTS: ${{ matrix.layouts }}
run: >
dotnet test --solution Laya.slnx -c Release
--results-directory "$RUNNER_TEMP/trx" --report-xunit-trx --report-xunit-trx-filename parity.trx
diff --git a/laya-dotnet/README.md b/laya-dotnet/README.md
index 02e49306..33320c5d 100644
--- a/laya-dotnet/README.md
+++ b/laya-dotnet/README.md
@@ -988,10 +988,14 @@ together with the C# change that follows it.
`.github/workflows/dotnet.yml` runs the same chain on every push and pull request (there is no path
filter, so a change under `laya/` triggers it): the `build` job re-records the routing goldens,
builds the whole solution with warnings as errors and runs the tests that need no model; one
-`parity` job per checkpoint (and one for the router end-to-end tests, which need english and
-multilingual together) exports that checkpoint, re-records its goldens from the Python code at
-that commit and runs its model-backed classes against them. `LAYA_TEST_CHECKPOINTS` selects the
-checkpoint theory rows (ordinary local runs still cover all checkpoints). Every job then runs
+`parity` job per checkpoint and layout (fused, split), and one for the router end-to-end tests
+(which need english and multilingual together), exports that checkpoint, re-records its goldens
+from the Python code at that commit and runs its model-backed classes against them.
+`LAYA_TEST_CHECKPOINTS` selects the checkpoint theory rows and `LAYA_TEST_LAYOUTS` (`fused`,
+`split`, or both when unset) the layout rows of the theories that have one per layout; the
+split-only tests carry `[Trait("Layout", "split-only")]` and the layout-parameterised ones
+`Layout=any`, which the split cell selects with `--filter-trait` (ordinary local runs still cover
+everything). Every job then runs
`tools/check_test_skips.py`, because `dotnet test` reports skipped tests as success: it fails the
run when any test that should have run was skipped, and tolerates only the skips for checkpoints
the job does not export. The job logs print `git diff --stat` of the regenerated goldens against
diff --git a/laya-dotnet/tests/Laya.Tests/PredictParityTests.Batch.cs b/laya-dotnet/tests/Laya.Tests/PredictParityTests.Batch.cs
index 9b798394..b53a0679 100644
--- a/laya-dotnet/tests/Laya.Tests/PredictParityTests.Batch.cs
+++ b/laya-dotnet/tests/Laya.Tests/PredictParityTests.Batch.cs
@@ -9,7 +9,7 @@ public sealed partial class PredictParityTests
from checkpoint in TestArtifacts.ParityCheckpoints
where GoldenData.For(checkpoint).Available
from info in GoldenData.For(checkpoint).Index.Where(i => i.Kind == "batch" && !i.ExpectsError)
- from split in new[] { false, true }
+ from split in TestArtifacts.ParitySplitFlags
select new object[] { checkpoint, info, split };
public static IEnumerable<object[]> BatchErrorCases() =>
@@ -29,6 +29,7 @@ public sealed partial class PredictParityTests
[.. root.GetProperty("states").EnumerateArray().Select(GoldenData.ToClr)];
[Theory]
+ [Trait(TestArtifacts.LayoutTrait, TestArtifacts.AnyLayout)]
[MemberData(nameof(BatchSuccessCases))]
public void ReproducesBatchPredictions(LayaCheckpoint checkpoint, GoldenCaseInfo info, bool split)
{
diff --git a/laya-dotnet/tests/Laya.Tests/PredictParityTests.Long.cs b/laya-dotnet/tests/Laya.Tests/PredictParityTests.Long.cs
index 099fc783..8b6f43c6 100644
--- a/laya-dotnet/tests/Laya.Tests/PredictParityTests.Long.cs
+++ b/laya-dotnet/tests/Laya.Tests/PredictParityTests.Long.cs
@@ -9,7 +9,7 @@ public sealed partial class PredictParityTests
from checkpoint in TestArtifacts.ParityCheckpoints
where GoldenData.For(checkpoint).Available
from info in GoldenData.For(checkpoint).Index.Where(i => i.Kind == "long" && !i.ExpectsError)
- from split in new[] { false, true }
+ from split in TestArtifacts.ParitySplitFlags
select new object[] { checkpoint, info, split };
public static IEnumerable<object[]> LongErrorCases() =>
@@ -19,6 +19,7 @@ public sealed partial class PredictParityTests
select new object[] { checkpoint, info };
[Theory]
+ [Trait(TestArtifacts.LayoutTrait, TestArtifacts.AnyLayout)]
[MemberData(nameof(LongSuccessCases))]
public void ReproducesLongPredictions(LayaCheckpoint checkpoint, GoldenCaseInfo info, bool split)
{
diff --git a/laya-dotnet/tests/Laya.Tests/PredictParityTests.cs b/laya-dotnet/tests/Laya.Tests/PredictParityTests.cs
index 8e8bc7be..81df6328 100644
--- a/laya-dotnet/tests/Laya.Tests/PredictParityTests.cs
+++ b/laya-dotnet/tests/Laya.Tests/PredictParityTests.cs
@@ -131,6 +131,7 @@ public sealed partial class PredictParityTests(AllEnginesFixture fixture)
/// Not run in CI or locally until an <c>onnx-split/</c> export exists — see the class remarks.
/// </summary>
[Theory]
+ [Trait(TestArtifacts.LayoutTrait, TestArtifacts.SplitOnly)]
[MemberData(nameof(AllSuccessCasesSplit))]
public void ReproducesTheRecordedAnswersSplit(LayaCheckpoint checkpoint, GoldenCaseInfo info)
{
diff --git a/laya-dotnet/tests/Laya.Tests/ShapeSweepTests.cs b/laya-dotnet/tests/Laya.Tests/ShapeSweepTests.cs
index 73dc6dbc..1074c79f 100644
--- a/laya-dotnet/tests/Laya.Tests/ShapeSweepTests.cs
+++ b/laya-dotnet/tests/Laya.Tests/ShapeSweepTests.cs
@@ -210,6 +210,7 @@ public sealed class ShapeSweepTests(AllEnginesFixture fixture)
/// different artifact directory rather than a separate code path being exercised here.
/// </summary>
[Theory]
+ [Trait(TestArtifacts.LayoutTrait, TestArtifacts.SplitOnly)]
[MemberData(nameof(CheckpointShapesSplit))]
public void RunsAtEveryShapeForAllCheckpointsSplit(LayaCheckpoint checkpoint, int seq, int batch, int markers)
{
diff --git a/laya-dotnet/tests/Laya.Tests/TestArtifacts.cs b/laya-dotnet/tests/Laya.Tests/TestArtifacts.cs
index 54b63ba1..5063fdf8 100644
--- a/laya-dotnet/tests/Laya.Tests/TestArtifacts.cs
+++ b/laya-dotnet/tests/Laya.Tests/TestArtifacts.cs
@@ -169,6 +169,41 @@ public static class TestArtifacts
}
}
+ /// <summary>
+ /// The trait that places a layout-specific test in a CI cell: <c>Layout=split-only</c> runs
+ /// against the split export only, <c>Layout=any</c> is parameterised by layout (its rows are
+ /// narrowed by <see cref="ParitySplitFlags"/>), and an untagged test is fused or layout
+ /// independent. A split cell runs <c>split-only</c> and <c>any</c>; a fused cell runs everything
+ /// except <c>split-only</c>. Filtering by trait (rather than skipping at run time) means a
+ /// deselected layout produces no skipped test.
+ /// </summary>
+ public const string LayoutTrait = "Layout";
+
+ /// <summary>Value of <see cref="LayoutTrait"/> for a test that only exercises the split layout.</summary>
+ public const string SplitOnly = "split-only";
+
+ /// <summary>Value of <see cref="LayoutTrait"/> for a theory with a fused and a split row per case.</summary>
+ public const string AnyLayout = "any";
+
+ /// <summary>
+ /// The <c>split</c> flag values a layout-parameterised theory yields (<c>false</c> is the fused
+ /// layout, <c>true</c> the split one). <c>LAYA_TEST_LAYOUTS</c> is a space-separated list of
+ /// <c>fused</c> and <c>split</c>; unset selects both, so ordinary local runs are unchanged.
+ /// </summary>
+ public static IEnumerable<bool> ParitySplitFlags
+ {
+ get
+ {
+ var selected = Environment.GetEnvironmentVariable("LAYA_TEST_LAYOUTS");
+ if (string.IsNullOrWhiteSpace(selected))
+ return [false, true];
+ var names = selected.Split(' ', StringSplitOptions.RemoveEmptyEntries);
+ if (names.Any(name => name != "fused" && name != "split"))
+ throw new ArgumentException($"unknown LAYA_TEST_LAYOUTS: {selected}");
+ return new[] { false, true }.Where(split => names.Contains(split ? "split" : "fused"));
+ }
+ }
+
// One CheckpointTestArtifacts per enum value; indexed by the enum's integer value.
private static readonly CheckpointTestArtifacts[] ByCheckpoint =
Enum.GetValues<LayaCheckpoint>().Select(c => new CheckpointTestArtifacts(c)).ToArray();