Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Partition complete frontend unit proof across isolated CI shards
pulse-triage[bot]bd20f7ce54c0
diff --git a/.github/workflows/build-and-test.yml b/.github/workflows/build-and-test.ymlindex 15e0a69096..1c4b866366 100644--- a/.github/workflows/build-and-test.yml+++ b/.github/workflows/build-and-test.yml@@ -246,16 +246,16 @@ jobs:grep -q '^--- PASS: TestIntegrationProviderPairNetworkIsolation ' \"$RUNNER_TEMP/provider-pair-network-test.log"- frontend:- name: Frontend+ frontend-checks:+ name: Frontend build and runtimeneeds: changesif: needs.changes.outputs.code == 'true'runs-on: ubuntu-24.04- # Everything except the audit runs in ~11m. The audit is now bounded to a- # 4 minute wall-clock budget, so 30 leaves real headroom while staying a- # meaningful ceiling: an unbounded 40 would have absorbed the 2026-09-04- # stall instead of reporting it.+ # Keep the original bounded audit and build/runtime deadline. Unit proof+ # follows in isolated file shards instead of one 650-second serial step.timeout-minutes: 30+ outputs:+ unit_admitted: ${{ steps.unit-admission.outputs.admitted }}env:FRONTEND_DIR: frontend-modern@@ -338,10 +338,6 @@ jobs:working-directory: frontend-modernrun: npm run check:bundlesize- - name: Frontend unit tests- working-directory: frontend-modern- run: npm run test-# Use this checkout's production assets, not another job's embed stub.# Keep the existing job deadline, audit and full frontend tests gating.- name: Set up Go for real-embed verification@@ -412,6 +408,13 @@ jobs:if-no-files-found: warnretention-days: 7+ # A failed audit must still permit unit proof, as in the serial job.+ # Only the unchanged build/size/real-embed/native/runtime gates admit+ # these shards. A missing output never grants admission.+ - name: Admit frontend unit shards+ id: unit-admission+ run: echo 'admitted=true' >> "$GITHUB_OUTPUT"+# An audit service failure or real advisory stays gating, but it must not# suppress formatting, tests, type-checking, or production-build proof.- name: Require frontend dependency audit@@ -424,6 +427,67 @@ jobs:exit 1fi+ frontend-unit:+ name: Frontend unit tests (${{ matrix.shard }}/4)+ needs: [changes, frontend-checks]+ if: ${{ !cancelled() && needs.changes.outputs.code == 'true' && needs.frontend-checks.outputs.unit_admitted == 'true' }}+ runs-on: ubuntu-24.04+ timeout-minutes: 30+ strategy:+ fail-fast: false+ matrix:+ shard: [1, 2, 3, 4]+ steps:+ - name: Checkout repository+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1+ with:+ persist-credentials: false++ - name: Set up Node.js+ uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0+ with:+ node-version: '24'+ cache: npm+ cache-dependency-path: frontend-modern/package-lock.json++ - name: Install frontend dependencies+ working-directory: frontend-modern+ # The unchanged complete audit in frontend-checks owns its verdict.+ run: npm ci --no-audit++ # Vitest partitions its complete file inventory, preserving the existing+ # config, isolation and assertions. No test allowlist or changed-file+ # filter can silently leave part of the suite behind.+ - name: Frontend unit tests+ working-directory: frontend-modern+ run: npm run test -- --shard=${{ matrix.shard }}/4++ # Preserve the required check name; a failed/cancelled/skipped shard or a+ # failed audit/build/runtime check must never produce a green Frontend.+ frontend:+ name: Frontend+ needs: [changes, frontend-checks, frontend-unit]+ if: always()+ runs-on: ubuntu-24.04+ timeout-minutes: 5+ steps:+ - name: Require complete frontend proof+ env:+ CHANGE_RESULT: ${{ needs.changes.result }}+ CODE_CHANGED: ${{ needs.changes.outputs.code }}+ CHECKS_RESULT: ${{ needs.frontend-checks.result }}+ UNIT_RESULT: ${{ needs.frontend-unit.result }}+ run: |+ case "$CHANGE_RESULT:$CODE_CHANGED:$CHECKS_RESULT:$UNIT_RESULT" in+ success:true:success:success)+ echo "Frontend build, complete audit, runtime and every unit shard passed." ;;+ success:false:skipped:skipped)+ echo "Application runtime checks are not applicable to this classified change; no frontend tests ran." ;;+ *)+ echo "::error::Frontend proof is incomplete or failed. Inspect changed-area classification, build/audit/runtime and all four unit shards."+ exit 1 ;;+ esac+# Reject cross-package test/API incompatibilities before spending seven# runners on race tests. This links test binaries but never executes them;# an exclusively test-file diff compiles its packages and imports, otherwisediff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.mdindex c8d6673bb5..cf0ee7723a 100644--- a/docs/release-control/v6/internal/subsystems/deployment-installability.md+++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md@@ -625,19 +625,35 @@ candidate commit; an unowned or unreachable route is not proof coverage.### Ordinary CI compiles the backend with the same checkout's real frontend-The existing Frontend job retains its production bundle, size, whole-tree tests,-audit and 30-minute ceiling. After those tests it selects Go from `go.mod`,+The required `Frontend` check aggregates the complete build/audit/runtime job+and every one of four isolated Vitest file shards. Missing, failed, cancelled+or skipped proof fails that check; only an explicit, consistent non-runtime+classification admits inapplicable checks. Each execution job retains the+30-minute ceiling; the aggregate has a five-minute orchestration deadline.++Before admitting unit shards, `frontend-checks` builds and size-checks the+production frontend, selects Go from `go.mod`,requires a nonempty built index/assets tree and compares the complete generatedfrontend and embed copy before backend compilation and server/API vet. Missing,extra or changed embedded files stop the phase; it never creates a placeholderor imports assets from another job/source. Build and vet use the read-only Gograph and two workers; the binary stays in the runner's temporary directory.The log records the checkout and binary digest without publishing an artifact.+The same job retains native updater and ordinary notification/restart proof.+Only successful build/size/runtime phases emit the explicit unit-admission+output. The complete dependency audit still reports its own terminal verdict+after that output: an audit failure does not suppress unit proof, but always+fails the required aggregate. Shards use the same lock, configuration and+assertions with only Vitest's built-in file partition; no maintained file list,+changed-file filter, shared mutable runner or extra in-runner concurrency is+introduced. Type/build/size failure still stops the expensive suite early.`TestFrontendCIRequiresSameCheckoutRealEmbedBackend` executes the exact phasewith private assets and Git/compiler doubles, covering mismatch refusal,declared command/ordering checks and terminal compiler/vet failures. These-controls are not real compilation. Actual containing CI checkout/phase logs+controls are not real compilation. `TestFrontendCIRequiresCompleteIsolatedUnitProof`+executes the workflow-order, complete-shard and actual fail-closed audit/verdict+controls in `scripts/tests/test_frontend_workflow_order.py`. Actual containing CI checkout/phase logsremain necessary before using the result as real-embed source proof; unrelatedstub builds, workflow success, releases and installed recovery remain distinct.diff --git a/scripts/installtests/build_release_assets_test.go b/scripts/installtests/build_release_assets_test.goindex 4335523b58..2d858b1123 100644--- a/scripts/installtests/build_release_assets_test.go+++ b/scripts/installtests/build_release_assets_test.go@@ -4520,6 +4520,17 @@ func TestFrontendCIRequiresSameCheckoutRealEmbedBackend(t *testing.T) {}}+// Keep the registered frontend security-proof route bound to the actual+// workflow's complete inventory, runtime admission and audit/shard verdicts.+func TestFrontendCIRequiresCompleteIsolatedUnitProof(t *testing.T) {+ ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)+ defer cancel()+ cmd := exec.CommandContext(ctx, "python3", repoFile("scripts", "tests", "test_frontend_workflow_order.py"), "-v")+ if out, err := cmd.CombinedOutput(); err != nil {+ t.Fatalf("complete isolated frontend proof and required verdict controls: %v\n%s", err, out)+ }+}+// The issue-event check runs after publication. Its real parser tests and// silent-mutation policy also need credential-free PR/main admission, without// relaxing product runtime, dependency audit or receipt-only boundaries.@@ -4619,7 +4630,7 @@ func TestFrontendDependencySecurityAuditsAreRequired(t *testing.T) {if err != nil {t.Fatalf("read %s: %v", workflowPath, err)}- frontendJob := workflowJobBlock(t, string(workflow), "frontend")+ frontendJob := workflowJobBlock(t, string(workflow), "frontend-checks")if !strings.Contains(frontendJob, "timeout-minutes: 30") {t.Fatal("frontend job must keep a bounded timeout above the audit budget")}diff --git a/scripts/tests/test_auto_update_systemd.py b/scripts/tests/test_auto_update_systemd.pyindex 9a4ddda335..b62e43b856 100644--- a/scripts/tests/test_auto_update_systemd.py+++ b/scripts/tests/test_auto_update_systemd.py@@ -36,7 +36,7 @@ def selected_fixture_commands():def recipe(source):- job = re.search(r"(?ms)^ frontend:\n(.*?)(?=^ [a-z][a-z0-9-]*:|\Z)", source)[1]+ job = re.search(r"(?ms)^ frontend-checks:\n(.*?)(?=^ [a-z][a-z0-9-]*:|\Z)", source)[1]steps = dict(re.findall(r"(?ms)^ - name: ([^\n]+)\n(.*?)(?=^ - name:|\Z)", job))order = list(steps)assert order.index("Verify backend with real frontend embed") < order.index(STEP) < order.index("Require frontend dependency audit")diff --git a/scripts/tests/test_dependency_advisory_watch.py b/scripts/tests/test_dependency_advisory_watch.pyindex 7fad177dc0..370ba57836 100644--- a/scripts/tests/test_dependency_advisory_watch.py+++ b/scripts/tests/test_dependency_advisory_watch.py@@ -202,7 +202,7 @@ class WorkflowContractTest(unittest.TestCase):def test_audit_matches_the_required_build_and_test_audit(self) -> None:build = yaml.safe_load(BUILD_AND_TEST.read_text(encoding="utf-8"))- frontend = build["jobs"]["frontend"]["steps"]+ frontend = build["jobs"]["frontend-checks"]["steps"]audit = self.workflow["jobs"]["audit"]["steps"]def step(steps: list[dict], name: str) -> dict:diff --git a/scripts/tests/test_frontend_real_embed.py b/scripts/tests/test_frontend_real_embed.pyindex 370cfbd7fb..abd7a47a7e 100644--- a/scripts/tests/test_frontend_real_embed.py+++ b/scripts/tests/test_frontend_real_embed.py@@ -21,7 +21,7 @@ VERIFY = "Verify backend with real frontend embed"def recipe(source):- job = re.search(r"(?ms)^ frontend:\n(.*?)(?=^ [a-z][a-z0-9-]*:|\Z)", source)+ job = re.search(r"(?ms)^ frontend-checks:\n(.*?)(?=^ [a-z][a-z0-9-]*:|\Z)", source)if not job:raise AssertionError("frontend job missing")job = job[1]@@ -33,10 +33,10 @@ def recipe(source):raise AssertionError("duplicate named frontend step")order = list(steps)required = ["Checkout repository", "Build frontend bundle (with embed copy)",- "Check frontend bundle size budget", "Frontend unit tests", GO_SETUP, VERIFY,+ "Check frontend bundle size budget", GO_SETUP, VERIFY,"Require frontend dependency audit"]if any(name not in order for name in required) or [order.index(name) for name in required] != sorted(order.index(name) for name in required):- raise AssertionError("same-checkout real build, tests, compiler or audit ordering missing")+ raise AssertionError("same-checkout real build, compiler or audit ordering missing")if sum("uses: actions/checkout@" in step for step in steps.values()) != 1 or "run: make frontend" not in steps[required[1]]:raise AssertionError("must use this checkout's production frontend build")if re.search(r"^ (ref|repository):", steps[required[0]], re.M):diff --git a/scripts/tests/test_frontend_workflow_order.py b/scripts/tests/test_frontend_workflow_order.pyindex 4911febd28..41add8544e 100644--- a/scripts/tests/test_frontend_workflow_order.py+++ b/scripts/tests/test_frontend_workflow_order.py@@ -1,7 +1,9 @@#!/usr/bin/env python3-"""An unshippable frontend must fail before the expensive full unit suite."""+"""Full frontend proof stays required across isolated file-level unit shards."""+import osfrom pathlib import Path+import subprocessimport unittestimport yaml@@ -13,7 +15,9 @@ WORKFLOW = ROOT / ".github/workflows/build-and-test.yml"class FrontendWorkflowOrderTest(unittest.TestCase):def setUp(self):- self.job = yaml.safe_load(WORKFLOW.read_text())["jobs"]["frontend"]+ self.workflow = yaml.safe_load(WORKFLOW.read_text())+ self.jobs = self.workflow["jobs"]+ self.job = self.jobs["frontend-checks"]self.steps = self.job["steps"]def step(self, name):@@ -28,7 +32,10 @@ class FrontendWorkflowOrderTest(unittest.TestCase):"Type-check frontend","Build frontend bundle (with embed copy)","Check frontend bundle size budget",- "Frontend unit tests",+ "Verify backend with real frontend embed",+ "Check current-source unattended recovery on native systemd",+ "Check ordinary notification policy and restart",+ "Admit frontend unit shards","Require frontend dependency audit",]for name in ordered:@@ -41,7 +48,6 @@ class FrontendWorkflowOrderTest(unittest.TestCase):"Type-check frontend": "npm run type-check","Build frontend bundle (with embed copy)": "make frontend","Check frontend bundle size budget": "npm run check:bundlesize",- "Frontend unit tests": "npm run test",}for name, command in commands.items():with self.subTest(step=name):@@ -68,6 +74,115 @@ class FrontendWorkflowOrderTest(unittest.TestCase):self.assertIn('if [ "${COMPLETE_AUDIT_RESULT}" != success ]; then', required["run"])self.assertIn("exit 1", required["run"])+ def test_unit_admission_is_after_runtime_but_before_the_audit_verdict(self):+ admission = self.step("Admit frontend unit shards")+ self.assertEqual(admission, {+ "name": "Admit frontend unit shards", "id": "unit-admission",+ "run": "echo 'admitted=true' >> \"$GITHUB_OUTPUT\"",+ })+ self.assertEqual(self.job["outputs"], {+ "unit_admitted": "${{ steps.unit-admission.outputs.admitted }}",+ })+ self.assertEqual(self.job["needs"], "changes")+ self.assertEqual(self.job["if"], "needs.changes.outputs.code == 'true'")+ self.assertEqual(self.job["timeout-minutes"], 30)+ # GitHub's default success() is essential: a failed/hidden runtime+ # phase must never produce the output admitting expensive unit shards.+ for step in self.steps:+ if step["name"].startswith("Retain bounded "):+ self.assertEqual(step["if"], "always()")+ elif step["name"] != "Require frontend dependency audit":+ self.assertNotIn("if", step)+ if step["name"] != "Audit complete frontend dependency graph":+ self.assertNotIn("continue-on-error", step)+ self.assertEqual(self.steps[-1]["name"], "Require frontend dependency audit")++ def test_four_isolated_shards_use_complete_vitest_inventory(self):+ unit = self.jobs["frontend-unit"]+ self.assertEqual(unit["needs"], ["changes", "frontend-checks"])+ self.assertEqual(unit["if"], "${{ !cancelled() && needs.changes.outputs.code == 'true' && needs.frontend-checks.outputs.unit_admitted == 'true' }}")+ self.assertEqual(unit["strategy"], {+ "fail-fast": False, "matrix": {"shard": [1, 2, 3, 4]},+ })+ self.assertEqual(unit["runs-on"], "ubuntu-24.04")+ self.assertEqual(unit["timeout-minutes"], 30)+ self.assertNotIn("continue-on-error", unit)+ self.assertNotIn("env", unit)+ steps = unit["steps"]+ self.assertEqual([step["name"] for step in steps], [+ "Checkout repository", "Set up Node.js", "Install frontend dependencies", "Frontend unit tests",+ ])+ for step in steps:+ self.assertNotIn("if", step)+ self.assertNotIn("continue-on-error", step)+ if step["name"] != "Frontend unit tests":+ self.assertEqual(step, self.step(step["name"]))+ self.assertEqual(steps[-1], {+ "name": "Frontend unit tests", "working-directory": "frontend-modern",+ "run": "npm run test -- --shard=${{ matrix.shard }}/4",+ })+ self.assertEqual(self.workflow["permissions"], {"contents": "read"})++ def verdict(self, **values):+ step = self.jobs["frontend"]["steps"][0]+ env = dict(os.environ, CHANGE_RESULT="success", CODE_CHANGED="true",+ CHECKS_RESULT="success", UNIT_RESULT="success")+ env.update(values)+ return subprocess.run(["bash", "-e", "-u", "-o", "pipefail", "-c", step["run"]],+ env=env, text=True, capture_output=True, timeout=5)++ def test_required_name_depends_on_classification_checks_and_every_shard(self):+ verdict = self.jobs["frontend"]+ self.assertEqual(verdict["name"], "Frontend")+ self.assertEqual(sum(job.get("name") == "Frontend" for job in self.jobs.values()), 1)+ self.assertEqual(verdict["needs"], ["changes", "frontend-checks", "frontend-unit"])+ self.assertEqual(verdict["if"], "always()")+ self.assertNotIn("continue-on-error", verdict)+ self.assertEqual(len(verdict["steps"]), 1)+ step = verdict["steps"][0]+ self.assertNotIn("if", step)+ self.assertNotIn("continue-on-error", step)+ self.assertEqual(step["env"], {+ "CHANGE_RESULT": "${{ needs.changes.result }}",+ "CODE_CHANGED": "${{ needs.changes.outputs.code }}",+ "CHECKS_RESULT": "${{ needs.frontend-checks.result }}",+ "UNIT_RESULT": "${{ needs.frontend-unit.result }}",+ })+ result = self.verdict()+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)+ self.assertIn("every unit shard passed", result.stdout)+ for name in ("CHANGE_RESULT", "CHECKS_RESULT", "UNIT_RESULT"):+ for state in ("failure", "cancelled", "skipped", "", "unknown", "pending", "SUCCESS"):+ with self.subTest(name=name, state=state):+ result = self.verdict(**{name: state})+ self.assertNotEqual(result.returncode, 0)+ self.assertIn("::error::Frontend proof", result.stdout)+ for state in ("false", "", "unknown", "TRUE"):+ with self.subTest(classification=state):+ self.assertNotEqual(self.verdict(CODE_CHANGED=state).returncode, 0)++ def test_only_consistent_inapplicable_proof_can_skip_the_unit_inventory(self):+ values = dict(CODE_CHANGED="false", CHECKS_RESULT="skipped", UNIT_RESULT="skipped")+ result = self.verdict(**values)+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)+ self.assertIn("no frontend tests ran", result.stdout)+ for name in ("CHANGE_RESULT", "CHECKS_RESULT", "UNIT_RESULT", "CODE_CHANGED"):+ states = ("failure", "cancelled", "", "unknown")+ if name in ("CHECKS_RESULT", "UNIT_RESULT"):+ states += ("success",)+ for state in states:+ with self.subTest(name=name, state=state):+ self.assertNotEqual(self.verdict(**{**values, name: state}).returncode, 0)++ def test_complete_audit_failure_still_fails_after_runtime_admits_units(self):+ required = self.step("Require frontend dependency audit")+ for state in ("success", "failure", "cancelled", "skipped", "", "unknown"):+ with self.subTest(state=state):+ result = subprocess.run(["bash", "-e", "-u", "-o", "pipefail", "-c", required["run"]],+ env=dict(os.environ, COMPLETE_AUDIT_RESULT=state),+ text=True, capture_output=True, timeout=5)+ self.assertEqual(result.returncode == 0, state == "success")+if __name__ == "__main__":unittest.main()diff --git a/scripts/tests/test_go_test_compilation.py b/scripts/tests/test_go_test_compilation.pyindex 92840d6074..81242982bc 100644--- a/scripts/tests/test_go_test_compilation.py+++ b/scripts/tests/test_go_test_compilation.py@@ -139,7 +139,9 @@ func TestStaleFixture(t *testing.T) { _ = api.Resource{Alerts: []api.ResourceAleself.assertIn("if: needs.changes.outputs.go_compile == 'true'", compile_job)self.assertIn("go test -race -vet=off -count=1 -exec /bin/true ./...", COMPILER.read_text())# Frontend work does not wait for the Go-only gate.- self.assertIn("needs: changes", job(source, "frontend"))+ self.assertIn("needs: changes", job(source, "frontend-checks"))+ self.assertIn("needs: [changes, frontend-checks]", job(source, "frontend-unit"))+ self.assertIn("needs: [changes, frontend-checks, frontend-unit]", job(source, "frontend"))def test_required_checks_admit_only_explicit_consistent_results(self):source = WORKFLOW.read_text()diff --git a/scripts/tests/test_npm_audit_retry.py b/scripts/tests/test_npm_audit_retry.pyindex 1d3a76b554..a71eb430e4 100755--- a/scripts/tests/test_npm_audit_retry.py+++ b/scripts/tests/test_npm_audit_retry.py@@ -637,7 +637,7 @@ class NpmAuditRetryTest(unittest.TestCase):def test_all_workflow_audits_use_the_retry_boundary(self) -> None:expected = {".github/workflows/build-and-test.yml": {- "job": "frontend",+ "job": "frontend-checks","runs": ['bash "$GITHUB_WORKSPACE/scripts/npm-audit-retry.sh" all',],