SolidCraft: an open-source parametric 3D CAD application built in pure Rust. Desktop, web, and automation through MCP.
Wave 351/509: Kernel rig: every area and the ladder as a parallel gate on pull requests that touch the kernel (and 3 more)
jyviko524b25b8ba44
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 740ab4c943..cac0919e7b 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -12,8 +12,9 @@# checks (no archive to upload and download). The GPU path tracer's tests run one at a time# (.config/nextest.toml), so they have a job of their own in four shards (GPU path tests). A few# heavy tests (see .config/nextest.toml) are skipped on pull requests and run on pushes to main and-# nightly. The reference tests and the doc tests run in the merge queue, on main and nightly, not-# on pull requests (the fast lane, in Select checks); a break there is fixed forward on main. The+# nightly. The reference tests run on a pull request that selects solidcraft-cli (they replay+# recipes through it). The doc tests run in the merge queue, on main and nightly, not on pull+# requests (the fast lane, in Select checks); a break there is fixed forward on main. The# Windows jobs (D3D12 shaders and app startup) run nightly, and in the merge queue only for a group# that changes what they exercise. Only main saves the build caches; pull requests restore them.# sccache is on a Blacksmith sticky disk that only main commits.@@ -21,8 +22,9 @@# commit, or when nothing a build reads changed since the last push to main whose run passed.# Otherwise it runs everything.# Heavy checks wait for review: a pull request without the `approved-for-ci` label runs only-# Select checks, the format and repository checks and a quick compile. Adding the label (after-# review) runs everything above, and every later push to the PR does too, except a push that only+# Select checks, the format and repository checks and a quick compile, unless its author is an+# owner, member or collaborator of the repository (they could push the code to a branch here+# anyway). Adding the label (after review) runs everything above, and every later push to the PR does too, except a push that only# merges main into a change that already passed (scripts/ci/main_merge.py): it carries that# result and runs the repository checks. Merge groups, the nightly run and Run workflow are not# pull requests and always run the full suite.@@ -31,6 +33,8 @@# `ci` job still runs and reports the commit's real result (see the `ci` job): the merge queue# reads `ci` from the newest run of a commit, so a run without it would block a PR that passed.# Cross-compilation is opt-in through Run workflow; release builds remain in release.yml.+# The release gate (release-gate.yml) calls this workflow (workflow_call) as a Run workflow with+# cross-compilation on.# Every Linux job runs on Blacksmith's Ubuntu 24.04 runners (blacksmith-<N>vcpu-ubuntu-2404), sized# per job: 32 vCPU for the tests, 16 for Clippy, the GPU path tests, the archive build (nightly) and the merge queue's, 8 for compiles, doc and reference tests, 4 for the light checks, 2 for Select checks and ci.# The Windows shader job runs on Blacksmith's Windows Server 2025 runners (blacksmith-8vcpu-windows-2025);@@ -64,6 +68,14 @@ on:description: "Also check WebAssembly and RISC-V (before a release)"type: booleandefault: false+ # The release gate (release-gate.yml): the event stays the caller's (workflow_dispatch), so a+ # call runs exactly what Run workflow runs.+ workflow_call:+ inputs:+ cross_compile:+ description: "Also check WebAssembly and RISC-V (before a release)"+ type: boolean+ default: falsepermissions:contents: read@@ -137,6 +149,9 @@ jobs:HEAD_REF: ${{ github.event.pull_request.head.ref }}REPO: ${{ github.repository }}APPROVED: ${{ contains(github.event.pull_request.labels.*.name, 'approved-for-ci') }}+ # An owner, member or collaborator could push the change to a branch here: their pull+ # requests need no label.+ TRUSTED: ${{ contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.pull_request.author_association) }}GH_TOKEN: ${{ github.token }}run: |native=false@@ -230,16 +245,18 @@ jobs:if [ "$EVENT_NAME" = pull_request ] && [ "$full" != true ]; thencase " $packages " in *" solidcraft-cli "*) ;; *) oracle=false ;; esacfi- # The pull request fast lane: Windows, the reference tests and the doc tests run on main- # (and the nightly), not on every pull request push; a break there is fixed forward on- # main. Over three days of pull requests the doc tests were the only failing job 3 times- # in about 7,900 runner-minutes, and the reference tests never were.+ # The pull request fast lane: Windows and the doc tests run on main (and the nightly),+ # not on every pull request push; a break there is fixed forward on main. Over three+ # days of pull requests the doc tests were the only failing job 3 times in about 7,900+ # runner-minutes. The reference tests run when solidcraft-cli is selected (above): a+ # kernel change that broke them went unseen until a full run on develop.doctests=$native- if [ "$EVENT_NAME" = pull_request ]; then windows=false; oracle=false; doctests=false; fi- # Until a maintainer adds approved-for-ci, a pull request gets a quick compile instead of- # Clippy, the tests and the doc tests.+ if [ "$EVENT_NAME" = pull_request ]; then windows=false; doctests=false; fi+ # Until a maintainer adds approved-for-ci, a pull request from outside the repository's+ # owners, members and collaborators gets a quick compile instead of Clippy, the tests,+ # the reference tests and the doc tests.quick=false- if [ "$EVENT_NAME" = pull_request ] && [ "$APPROVED" != true ]; then+ if [ "$EVENT_NAME" = pull_request ] && [ "$APPROVED" != true ] && [ "$TRUSTED" != true ]; thenquick=$nativenative=falseoracle=falsediff --git a/.github/workflows/kernel-rig.yml b/.github/workflows/kernel-rig.ymlnew file mode 100644index 0000000000..18aede4d5b--- /dev/null+++ b/.github/workflows/kernel-rig.yml@@ -0,0 +1,175 @@+# The kernel rig and the complexity ladder as a regression gate. A pull request that touches the+# kernel surface (the paths below) builds solidcraft-cli once (release, SMLib prebuilt), hands it+# to a matrix of shards as an artifact, and each shard runs one area of tests/kernel-cases/ (or+# the ladder) in parallel, so the wall time is the build plus the slowest shard. See+# tests/kernel-cases/README.md.+#+# - An area fails on a regression (a case that fails and its tests/kernel-cases/<area>/KNOWN_FAILING+# does not list) or on a listed case that passes (it is fixed: remove its line in the same pull+# request, so the list only shrinks).+# - The ladder fails on any case whose status or counts differ from+# tests/kernel-cases/ladder/golden.json; a right change is re-recorded (`--record "why"`).+#+# GitHub filters a workflow by whole files, so any change to crates/doc/src/eval*.rs or+# crates/engine/src/cmd/features*.rs selects it. Pushes to main and develop that touch the same+# paths run it too and commit the compiler cache (a Blacksmith sticky disk) that every other run+# mounts read-only. A merge group (once the queue is on) always runs it: it is the last check+# before the change lands. The release gate (release-gate.yml) calls it (workflow_call).+name: Kernel rig++on:+ pull_request:+ types: [opened, synchronize, reopened, ready_for_review]+ paths:+ - "crates/kernel/**"+ - "crates/geom/**"+ - "crates/smlib/**"+ - "crates/step/**"+ - "crates/doc/src/eval*.rs"+ - "crates/engine/src/cmd/features*.rs"+ - "tests/kernel-cases/**"+ # The rig itself: the grading, the runner and this workflow.+ - "apps/solidcraft-cli/src/kernel_cases.rs"+ - "xtask/src/kernel_cases.rs"+ - "xtask/src/kernel_ladder.rs"+ - "scripts/fetch-smlib.sh"+ - ".github/workflows/kernel-rig.yml"+ push:+ branches: [main, develop]+ # The same paths as pull_request.+ paths:+ - "crates/kernel/**"+ - "crates/geom/**"+ - "crates/smlib/**"+ - "crates/step/**"+ - "crates/doc/src/eval*.rs"+ - "crates/engine/src/cmd/features*.rs"+ - "tests/kernel-cases/**"+ # The rig itself: the grading, the runner and this workflow.+ - "apps/solidcraft-cli/src/kernel_cases.rs"+ - "xtask/src/kernel_cases.rs"+ - "xtask/src/kernel_ladder.rs"+ - "scripts/fetch-smlib.sh"+ - ".github/workflows/kernel-rig.yml"+ merge_group:+ workflow_call:+ workflow_dispatch:++permissions:+ contents: read++concurrency:+ # A new push to a pull request replaces its old run.+ group: kernel-rig-${{ github.event_name }}-${{ github.ref }}+ cancel-in-progress: ${{ github.event_name == 'pull_request' }}++env:+ CARGO_TERM_COLOR: always++jobs:+ build:+ name: Build the CLI+ # Draft pull requests run no CI, as in ci.yml.+ if: github.event_name != 'pull_request' || !github.event.pull_request.draft+ runs-on: blacksmith-8vcpu-ubuntu-2404+ timeout-minutes: 60+ env:+ CARGO_INCREMENTAL: "0"+ RUSTC_WRAPPER: sccache+ SCCACHE_DIR: /home/runner/.cache/sccache+ SCCACHE_CACHE_SIZE: 10G+ steps:+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1+ - name: Size parallel compiles to the job's memory+ run: scripts/ci/build-jobs.sh+ - uses: dtolnay/rust-toolchain@stable+ # Only pushes to main and develop commit the cache; pull requests, merge groups and forks+ # mount a clone and never write back.+ - name: Mount the compiler cache+ uses: useblacksmith/stickydisk@3f6be1451e3cad893b778bc93495c80452a5da26 # v1.7.2+ continue-on-error: true+ with:+ key: ${{ github.repository }}-sccache-kernel-rig+ path: /home/runner/.cache/sccache+ commit: ${{ github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/develop') && 'on-change' || 'false' }}+ - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2+ with:+ key: kernel-rig+ save-if: ${{ github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/develop') }}+ - name: Fetch SMLib (crates/smlib/SMLIB_REV)+ shell: bash+ run: scripts/fetch-smlib.sh+ - name: Set up compiler cache+ uses: mozilla-actions/sccache-action@fc920bf0ec8de6ee65d409111f7ec508035751ba # v0.0.11+ with:+ version: v0.16.0+ - name: Build the CLI (release, SMLib) and the rig's runner+ run: |+ cargo build --release --locked -p solidcraft-cli --features smlib+ cargo build --locked -p xtask+ # A tar keeps the executable bits, which an artifact drops. xtask finds the repository from+ # the path it was built in (CARGO_MANIFEST_DIR), which is the same checkout path in every+ # job of this workflow.+ - name: Pack the binaries+ run: tar -cf "$RUNNER_TEMP/kernel-rig-bin.tar" -C target release/solidcraft-cli debug/xtask+ - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2+ with:+ name: kernel-rig-bin+ path: ${{ runner.temp }}/kernel-rig-bin.tar+ retention-days: 1+ - name: Compiler cache statistics+ if: always()+ run: sccache --show-stats++ shard:+ name: Kernel rig (${{ matrix.shard }})+ needs: build+ runs-on: blacksmith-4vcpu-ubuntu-2404+ timeout-minutes: 30+ strategy:+ fail-fast: false+ matrix:+ # Every area of tests/kernel-cases/ and the ladder. On the build box (aarch64, 4 cases at+ # a time) the slowest areas took about 1 minute (booleans, 212 s of cases with one of+ # 84 s; threads, 161 s) and the ladder, one case at a time, 92 s: no area needs splitting.+ shard: [booleans, continuity, crashes, fillets, import, ladder, sheet-metal, shell, sweeps, tessellation, threads]+ steps:+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1+ - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2+ with:+ name: kernel-rig-bin+ path: ${{ runner.temp }}+ - name: Unpack the binaries+ run: mkdir -p target && tar -xf "$RUNNER_TEMP/kernel-rig-bin.tar" -C target+ - name: Run the cases against the known-failing list (the ladder against its golden)+ env:+ SHARD: ${{ matrix.shard }}+ run: |+ if [ "$SHARD" = ladder ]; then+ set -- --ladder --gate+ else+ set -- --area "$SHARD" --known-failing "tests/kernel-cases/$SHARD/KNOWN_FAILING"+ fi+ target/debug/xtask kernel-cases "$@" --cli target/release/solidcraft-cli --out "$RUNNER_TEMP/kernel-rig"+ - name: Keep the results+ if: always()+ uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2+ with:+ name: kernel-rig-results-${{ matrix.shard }}+ path: ${{ runner.temp }}/kernel-rig/results.json+ if-no-files-found: ignore++ # One check for the whole gate, for branch protection and the merge queue.+ kernel-rig:+ name: kernel-rig+ needs: [build, shard]+ if: always() && (github.event_name != 'pull_request' || !github.event.pull_request.draft)+ runs-on: blacksmith-2vcpu-ubuntu-2404+ steps:+ - name: Every shard passed+ env:+ BUILD: ${{ needs.build.result }}+ SHARDS: ${{ needs.shard.result }}+ run: |+ test "$BUILD" = success+ test "$SHARDS" = successdiff --git a/.github/workflows/release-gate.yml b/.github/workflows/release-gate.ymlnew file mode 100644index 0000000000..de7ad30dc7--- /dev/null+++ b/.github/workflows/release-gate.yml@@ -0,0 +1,70 @@+# The release gate: everything, in full and in parallel, on one branch's head, before that branch+# replaces main as a release (develop, rebased onto main). Run it by hand:+# gh workflow run release-gate.yml --ref develop+# It calls the workflows that own each check instead of copying their jobs:+# - ci.yml as Run workflow with cross-compilation: every native suite in full (not scoped), the+# reference tests, the doc tests, the GPU path tests, the action matrix (all 16 shards) and its+# determinism check, the property tests, the repository checks (web-offline among them),+# WebAssembly and RISC-V, Windows D3D12 FXC and Windows app startup;+# - kernel-rig.yml: every rig area against its KNOWN_FAILING and the ladder against its golden;+# - release.yml as a dry run (draft false): the Linux, Flatpak, Snap, FreeBSD, macOS, Windows and+# web packages with their smoke tests and package checks.+# The gate passes only when all three pass. Before main is replaced, tag the old main+# (main-before-develop-<date>) and push the tag.+name: Release gate++on:+ workflow_dispatch:++permissions:+ contents: read++concurrency:+ group: release-gate-${{ github.ref }}+ cancel-in-progress: false++jobs:+ ci:+ name: CI+ uses: ./.github/workflows/ci.yml+ permissions:+ contents: read+ # Select checks reads this commit's other runs.+ actions: read+ with:+ cross_compile: true++ kernel-rig:+ name: Kernel rig+ uses: ./.github/workflows/kernel-rig.yml+ permissions:+ contents: read++ packages:+ name: Packages+ uses: ./.github/workflows/release.yml+ # release.yml's last job asks for contents: write (it creates releases); a dry run only keeps+ # the files as an artifact.+ permissions:+ contents: write+ actions: read+ secrets: inherit+ with:+ draft: false++ release-gate:+ name: release-gate+ needs: [ci, kernel-rig, packages]+ if: always()+ runs-on: blacksmith-2vcpu-ubuntu-2404+ steps:+ - name: Every part passed+ env:+ CI: ${{ needs.ci.result }}+ KERNEL_RIG: ${{ needs.kernel-rig.result }}+ PACKAGES: ${{ needs.packages.result }}+ run: |+ echo "ci=$CI kernel-rig=$KERNEL_RIG packages=$PACKAGES"+ test "$CI" = success+ test "$KERNEL_RIG" = success+ test "$PACKAGES" = successdiff --git a/.github/workflows/release.yml b/.github/workflows/release.ymlindex 0329f134f1..07bcbe839b 100644--- a/.github/workflows/release.yml+++ b/.github/workflows/release.yml@@ -20,6 +20,7 @@# releases/latest/download/<name> links in the README keep working across releases.## A dry run: dispatch with draft=false (the default). Only the release branch may create drafts.+# The release gate (release-gate.yml) calls it as such a dry run.# A dry run from any branch also test-signs the Windows files when SIGNPATH_API_TOKEN exists:# gh workflow run release.yml --ref <branch> -f draft=false [-f signpath_signing_policy=test-signing]name: Release@@ -40,6 +41,18 @@ on:description: "SignPath signing policy for the Windows files (empty: the repository variable, else test-signing)"type: stringdefault: ""+ # The release gate (release-gate.yml): a dry run (draft false) of every package and its checks.+ # The event stays the caller's (workflow_dispatch), so a call runs as a dispatched dry run does.+ workflow_call:+ inputs:+ draft:+ description: "Create/update a draft (release branch only; off: build artifacts only)"+ type: boolean+ default: false+ signpath_signing_policy:+ description: "SignPath signing policy for the Windows files (empty: the repository variable, else test-signing)"+ type: string+ default: ""permissions:contents: readdiff --git a/tests/kernel-cases/booleans/KNOWN_FAILING b/tests/kernel-cases/booleans/KNOWN_FAILINGnew file mode 100644index 0000000000..01f33ea2a2--- /dev/null+++ b/tests/kernel-cases/booleans/KNOWN_FAILING@@ -0,0 +1,4 @@+# The booleans cases that fail today, one a line: `<case> <why it fails today>`. Sorted. The kernel+# rig gate (.github/workflows/kernel-rig.yml) fails on a case that fails and is not listed here,+# and on a listed case that passes: a fix removes its line in the same pull request.+# Baseline: not seeded yet; seeded from a develop run after the SMLib pin bump.diff --git a/tests/kernel-cases/continuity/KNOWN_FAILING b/tests/kernel-cases/continuity/KNOWN_FAILINGnew file mode 100644index 0000000000..6220878c6e--- /dev/null+++ b/tests/kernel-cases/continuity/KNOWN_FAILING@@ -0,0 +1,4 @@+# The continuity cases that fail today, one a line: `<case> <why it fails today>`. Sorted. The kernel+# rig gate (.github/workflows/kernel-rig.yml) fails on a case that fails and is not listed here,+# and on a listed case that passes: a fix removes its line in the same pull request.+# Baseline: not seeded yet; seeded from a develop run after the SMLib pin bump.diff --git a/tests/kernel-cases/crashes/KNOWN_FAILING b/tests/kernel-cases/crashes/KNOWN_FAILINGnew file mode 100644index 0000000000..b322b1f3b2--- /dev/null+++ b/tests/kernel-cases/crashes/KNOWN_FAILING@@ -0,0 +1,4 @@+# The crashes cases that fail today, one a line: `<case> <why it fails today>`. Sorted. The kernel+# rig gate (.github/workflows/kernel-rig.yml) fails on a case that fails and is not listed here,+# and on a listed case that passes: a fix removes its line in the same pull request.+# Baseline: not seeded yet; seeded from a develop run after the SMLib pin bump.diff --git a/tests/kernel-cases/import/KNOWN_FAILING b/tests/kernel-cases/import/KNOWN_FAILINGnew file mode 100644index 0000000000..636c9e35e5--- /dev/null+++ b/tests/kernel-cases/import/KNOWN_FAILING@@ -0,0 +1,4 @@+# The import cases that fail today, one a line: `<case> <why it fails today>`. Sorted. The kernel+# rig gate (.github/workflows/kernel-rig.yml) fails on a case that fails and is not listed here,+# and on a listed case that passes: a fix removes its line in the same pull request.+# Baseline: not seeded yet; seeded from a develop run after the SMLib pin bump.diff --git a/tests/kernel-cases/sheet-metal/KNOWN_FAILING b/tests/kernel-cases/sheet-metal/KNOWN_FAILINGnew file mode 100644index 0000000000..2ac87890e7--- /dev/null+++ b/tests/kernel-cases/sheet-metal/KNOWN_FAILING@@ -0,0 +1,4 @@+# The sheet-metal cases that fail today, one a line: `<case> <why it fails today>`. Sorted. The kernel+# rig gate (.github/workflows/kernel-rig.yml) fails on a case that fails and is not listed here,+# and on a listed case that passes: a fix removes its line in the same pull request.+# Baseline: not seeded yet; seeded from a develop run after the SMLib pin bump.diff --git a/tests/kernel-cases/shell/KNOWN_FAILING b/tests/kernel-cases/shell/KNOWN_FAILINGnew file mode 100644index 0000000000..07f36eb3c3--- /dev/null+++ b/tests/kernel-cases/shell/KNOWN_FAILING@@ -0,0 +1,4 @@+# The shell cases that fail today, one a line: `<case> <why it fails today>`. Sorted. The kernel+# rig gate (.github/workflows/kernel-rig.yml) fails on a case that fails and is not listed here,+# and on a listed case that passes: a fix removes its line in the same pull request.+# Baseline: not seeded yet; seeded from a develop run after the SMLib pin bump.diff --git a/tests/kernel-cases/sweeps/KNOWN_FAILING b/tests/kernel-cases/sweeps/KNOWN_FAILINGnew file mode 100644index 0000000000..82aa00854d--- /dev/null+++ b/tests/kernel-cases/sweeps/KNOWN_FAILING@@ -0,0 +1,4 @@+# The sweeps cases that fail today, one a line: `<case> <why it fails today>`. Sorted. The kernel+# rig gate (.github/workflows/kernel-rig.yml) fails on a case that fails and is not listed here,+# and on a listed case that passes: a fix removes its line in the same pull request.+# Baseline: not seeded yet; seeded from a develop run after the SMLib pin bump.diff --git a/tests/kernel-cases/tessellation/KNOWN_FAILING b/tests/kernel-cases/tessellation/KNOWN_FAILINGnew file mode 100644index 0000000000..1f7f372c37--- /dev/null+++ b/tests/kernel-cases/tessellation/KNOWN_FAILING@@ -0,0 +1,4 @@+# The tessellation cases that fail today, one a line: `<case> <why it fails today>`. Sorted. The kernel+# rig gate (.github/workflows/kernel-rig.yml) fails on a case that fails and is not listed here,+# and on a listed case that passes: a fix removes its line in the same pull request.+# Baseline: not seeded yet; seeded from a develop run after the SMLib pin bump.diff --git a/tests/kernel-cases/threads/KNOWN_FAILING b/tests/kernel-cases/threads/KNOWN_FAILINGnew file mode 100644index 0000000000..07e2659a73--- /dev/null+++ b/tests/kernel-cases/threads/KNOWN_FAILING@@ -0,0 +1,4 @@+# The threads cases that fail today, one a line: `<case> <why it fails today>`. Sorted. The kernel+# rig gate (.github/workflows/kernel-rig.yml) fails on a case that fails and is not listed here,+# and on a listed case that passes: a fix removes its line in the same pull request.+# Baseline: not seeded yet; seeded from a develop run after the SMLib pin bump.diff --git a/xtask/src/kernel_cases.rs b/xtask/src/kernel_cases.rsindex ff9499f1f1..7121ea2edd 100644--- a/xtask/src/kernel_cases.rs+++ b/xtask/src/kernel_cases.rs@@ -276,7 +276,7 @@ fn write_report(dir: &Path, list: &[PathBuf], results: &[Value]) -> Result<Strin);}let mut md = String::from(- "# Kernel regression rig\n\nGenerated by `cargo xtask kernel-cases`. Every geometry case the old kernel (truck) got wrong, as an issue, a PR or a crash report recorded it, is a use case in [`tests/kernel-cases/`](../tests/kernel-cases/) with its own right answer: numbers from analysis or a hand calculation (**analytic**) where the geometry has them, and otherwise the invariants (**invariants**): the operation succeeds, or refuses cleanly where that is the right answer; each body is one closed shell that passes the record's structure and geometry checks and SMLib's own check; the expected face, edge and vertex counts. Never a kernel's output, and never a commercial kernel's.\n\nThe cases run on SMLib, SolidCraft's kernel. A failing case stays failing until the kernel is fixed; there is no workaround on SolidCraft's side. A failed SMLib call leaves its inputs for replay in the run's output folder (`target/kernel-cases/dumps/<area>-<case>/`). See [`tests/kernel-cases/README.md`](../tests/kernel-cases/README.md) for the format. The fillet cases are also a regression gate: CI's Fillet rig workflow runs them on every pull request that touches fillet code and fails on a case that fails and [`fillets/KNOWN_FAILING`](../tests/kernel-cases/fillets/KNOWN_FAILING) does not list, or a listed case that passes.\n\n",+ "# Kernel regression rig\n\nGenerated by `cargo xtask kernel-cases`. Every geometry case the old kernel (truck) got wrong, as an issue, a PR or a crash report recorded it, is a use case in [`tests/kernel-cases/`](../tests/kernel-cases/) with its own right answer: numbers from analysis or a hand calculation (**analytic**) where the geometry has them, and otherwise the invariants (**invariants**): the operation succeeds, or refuses cleanly where that is the right answer; each body is one closed shell that passes the record's structure and geometry checks and SMLib's own check; the expected face, edge and vertex counts. Never a kernel's output, and never a commercial kernel's.\n\nThe cases run on SMLib, SolidCraft's kernel. A failing case stays failing until the kernel is fixed; there is no workaround on SolidCraft's side. A failed SMLib call leaves its inputs for replay in the run's output folder (`target/kernel-cases/dumps/<area>-<case>/`). See [`tests/kernel-cases/README.md`](../tests/kernel-cases/README.md) for the format. Every area is also a regression gate: CI's Kernel rig workflow runs them on every pull request that touches the kernel and fails on a case that fails and its area's `KNOWN_FAILING` (such as [`fillets/KNOWN_FAILING`](../tests/kernel-cases/fillets/KNOWN_FAILING)) does not list, or a listed case that passes.\n\n",);md += "## Coverage\n\n| Area | Cases | Results |\n|---|---|---|\n";for (area, (n, statuses)) in &areas {diff --git a/xtask/src/kernel_ladder.rs b/xtask/src/kernel_ladder.rsindex 93efd826b9..5e384c7a54 100644--- a/xtask/src/kernel_ladder.rs+++ b/xtask/src/kernel_ladder.rs@@ -1,4 +1,4 @@-//! `cargo xtask kernel-cases --ladder [--record REASON]`: the kernel complexity ladder.+//! `cargo xtask kernel-cases --ladder [--record REASON | --gate]`: the kernel complexity ladder.//!//! `tests/kernel-cases/ladder/ladder.json` lists tiers in order of complexity and, per tier, its//! cases in order (paths under `tests/kernel-cases/`, so a tier may also list rig cases from other@@ -9,6 +9,8 @@//! not shared with other cases. `--record REASON` re-measures the golden and re-sorts each tier//! by time, fastest first; both files are committed. The run's output folder (`--out`, default//! `target/kernel-ladder`) holds `results.json` and the SMLib dumps, as the rig's does.+//! `--gate` makes the run a regression gate instead of the report: it fails on any case whose+//! status or counts differ from the golden (times aside, which are the machine's).//!//! The report, `docs/kernel-ladder.md`, gives per tier the counts per status and the robustness//! frontier: the first case, in ladder order, that fails.@@ -103,17 +105,26 @@ fn golden_entry(r: &Value) -> Value {/// What changed against the golden: a slowdown, a status change, other counts.fn flags(r: &Value, g: Option<&Value>) -> Vec<String> {- let Some(g) = g else { return vec!["no golden".into()] };- let mut out = Vec::new();- if g["status"] != r["status"] {- out.push(format!("status was {}", g["status"].as_str().unwrap_or("?")));- }- if let (Some(now), Some(then)) = (r["seconds"].as_f64(), g["seconds"].as_f64())+ let mut out = changes(r, g);+ if let Some(g) = g+ && let (Some(now), Some(then)) = (r["seconds"].as_f64(), g["seconds"].as_f64())&& now > then * SLOWDOWN_FACTOR&& now - then > SLOWDOWN_MIN_S{out.push(format!("slower: {now:.2} s (golden {then:.2} s)"));}+ out+}++/// What changed against the golden that the machine does not explain: no golden, a status+/// change, other counts. Times are left out (they depend on the machine the golden was recorded+/// on), so these are what the gate fails on.+fn changes(r: &Value, g: Option<&Value>) -> Vec<String> {+ let Some(g) = g else { return vec!["no golden".into()] };+ let mut out = Vec::new();+ if g["status"] != r["status"] {+ out.push(format!("status was {}", g["status"].as_str().unwrap_or("?")));+ }for k in ["body_count", "faces", "edges", "vertices"] {let (now, then) = (&r["measured"][k], &g["measured"][k]);if !now.is_null() && !then.is_null() && now != then {@@ -123,6 +134,35 @@ fn flags(r: &Value, g: Option<&Value>) -> Vec<String> {out}+/// The ladder's regression gate (`--gate`): every case's status and counts are the golden's. A+/// case that changed fails it, better or worse, as does a case with no golden: the change is+/// re-recorded with its reason (`--record`) in the same pull request.+fn gate(tiers: &[Tier], results: &BTreeMap<String, Value>, golden: &Value) -> Result<String, String> {+ let mut problems = Vec::new();+ let mut pass = 0;+ for c in tiers.iter().flat_map(|t| &t.cases) {+ let Some(r) = results.get(c) else {+ problems.push(format!("{c}: no result"));+ continue;+ };+ pass += usize::from(r["status"] == "pass");+ let changed = changes(r, golden["cases"].get(c));+ if !changed.is_empty() {+ problems.push(format!("CHANGED ({}): {c}: {}", r["status"].as_str().unwrap_or("?"), changed.join("; ")));+ }+ }+ let n: usize = tiers.iter().map(|t| t.cases.len()).sum();+ if problems.is_empty() {+ Ok(format!("kernel ladder gate: {pass}/{n} pass, every case as the golden records it"))+ } else {+ Err(format!(+ "kernel ladder gate: {} case(s) differ from tests/kernel-cases/ladder/golden.json (re-record with --record \"why\" if the change is right):\n {}",+ problems.len(),+ problems.join("\n ")+ ))+ }+}+fn secs(v: &Value) -> String {v.as_f64().map(|s| format!("{s:.3}")).unwrap_or_else(|| "–".into())}@@ -218,6 +258,10 @@ pub fn run(rest: &[&str]) -> Result<(), String> {Some(None) => return Err("--record needs a reason: --record \"why the golden changes\"".into()),Some(Some(r)) => Some(r.to_string()),};+ let gate_run = rest.contains(&"--gate");+ if gate_run && reason.is_some() {+ return Err("--gate and --record: a gate run compares with the golden, a record run rewrites it".into());+ }let mut o = options(rest, "target/kernel-ladder")?;// Times are compared with the golden: cases run one at a time unless --jobs says otherwise,// so they don't share the machine with each other.@@ -236,7 +280,10 @@ pub fn run(rest: &[&str]) -> Result<(), String> {save_results(&o, &all)?;let results: BTreeMap<String, Value> = all.into_iter().map(|r| (rel_of(&r), r)).collect();let golden = read(&golden_path());- if let Some(reason) = &reason {+ if gate_run {+ // A gate run grades against the golden; the report stays the last full run's.+ println!("{}", gate(&tiers, &results, &golden)?);+ } else if let Some(reason) = &reason {record(reason, &manifest, &tiers, &results, &golden)?;let tiers = self::tiers(&read(&manifest_path()))?;println!("{}", write_report(&tiers, &results, &read(&golden_path()))?);@@ -264,6 +311,24 @@ mod tests {assert_eq!(flags(&same, None), ["no golden"]);}+ #[test]+ fn the_gate_fails_on_any_change_but_time() {+ let t = vec![Tier { tier: "01-a".into(), what: String::new(), cases: vec!["ladder/01-a/001-x".into(), "ladder/01-a/002-y".into()] }];+ let golden = json!({"cases": {+ "ladder/01-a/001-x": {"status": "pass", "seconds": 1.0, "measured": {"faces": 6}},+ "ladder/01-a/002-y": {"status": "fail", "seconds": 1.0}+ }});+ let run = |x: Value, y: Value| gate(&t, &BTreeMap::from([("ladder/01-a/001-x".to_string(), x), ("ladder/01-a/002-y".to_string(), y)]), &golden);+ // Ten times slower is the machine, not a change.+ assert!(run(json!({"status": "pass", "seconds": 10.0, "measured": {"faces": 6}}), json!({"status": "fail"})).is_ok());+ let fixed = run(json!({"status": "pass", "measured": {"faces": 6}}), json!({"status": "pass"})).unwrap_err();+ assert!(fixed.contains("CHANGED (pass): ladder/01-a/002-y: status was fail"), "{fixed}");+ let counts = run(json!({"status": "pass", "measured": {"faces": 7}}), json!({"status": "fail"})).unwrap_err();+ assert!(counts.contains("faces 7 (golden 6)"), "{counts}");+ let unrecorded = gate(&t, &BTreeMap::new(), &golden).unwrap_err();+ assert!(unrecorded.contains("ladder/01-a/001-x: no result"), "{unrecorded}");+ }+#[test]fn tiers_need_paths() {let m = json!({"tiers": [{"tier": "01-a", "what": "w", "cases": ["ladder/01-a/001-x"]}]});diff --git a/xtask/src/main.rs b/xtask/src/main.rsindex 6916254c33..13be4f8d77 100644--- a/xtask/src/main.rs+++ b/xtask/src/main.rs@@ -55,12 +55,13 @@ commands:each failing SMLib call (--cli: a solidcraft-cli built with --features smlib)--known-failing FILE: a regression gate instead of the report: fails on a casethat fails and FILE does not list, and on a listed case that passes- (tests/kernel-cases/fillets/KNOWN_FAILING, CI's fillet rig job)- --ladder [--record REASON]: the complexity ladder instead+ (tests/kernel-cases/<area>/KNOWN_FAILING, CI's Kernel rig workflow)+ --ladder [--record REASON | --gate]: the complexity ladder instead(tests/kernel-cases/ladder/ladder.json, tiers in order), graded and comparedwith its timing golden (one case at a time unless --jobs), intodocs/kernel-ladder.md (DIR default target/kernel-ladder); --record re-measures- the golden and re-sorts each tier by time+ the golden and re-sorts each tier by time; --gate fails on any case whose status+ or counts differ from the golden, instead of the reportaction-matrix [--bless] [--results FILE] [--from DIR]try every ordinary action on every face, edge, vertex and body of the designcorpus and compare with tests/action_matrix/baseline.json (--bless rewrites it;