Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Retain producer artifact identity across partial CI retries
pulse-triage[bot]52c69c222815
diff --git a/.github/workflows/build-and-test.yml b/.github/workflows/build-and-test.ymlindex dcb6208a9a..0ce86c2653 100644--- a/.github/workflows/build-and-test.yml+++ b/.github/workflows/build-and-test.yml@@ -266,6 +266,7 @@ jobs:outputs:unit_admitted: ${{ steps.unit-admission.outputs.admitted }}embed_sha256: ${{ steps.embed-packet.outputs.sha256 }}+ embed_artifact_id: ${{ steps.embed-upload.outputs.artifact-id }}env:FRONTEND_DIR: frontend-modern@@ -366,6 +367,7 @@ jobs:printf 'CI_FRONTEND_EMBED_SOURCE=%s SHA256=%s\n' "$EXPECTED_SOURCE_SHA" "${digest%% *}"- name: Transfer the validated frontend embed+ id: embed-uploaduses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2with:name: ci-frontend-embed-${{ github.sha }}-${{ github.run_attempt }}@@ -380,7 +382,12 @@ jobs:# A missing output never grants admission.- name: Admit frontend unit shardsid: unit-admission- run: echo 'admitted=true' >> "$GITHUB_OUTPUT"+ env:+ EMBED_ARTIFACT_ID: ${{ steps.embed-upload.outputs.artifact-id }}+ run: |+ set -euo pipefail+ [[ "$EMBED_ARTIFACT_ID" =~ ^[1-9][0-9]*$ ]]+ echo 'admitted=true' >> "$GITHUB_OUTPUT"# An audit service failure or real advisory stays gating, but it must not# suppress formatting, tests, type-checking, or production-build proof.@@ -406,11 +413,25 @@ jobs:with:persist-credentials: false+ # A failed-jobs rerun keeps a successful producer's original outputs.+ # Validate its immutable ID before download: an empty selector must not+ # fall back to downloading every artifact. Never derive it from the+ # consumer's run_attempt, which can differ from the producer's attempt.+ - name: Check validated frontend embed identity+ env:+ EMBED_ARTIFACT_ID: ${{ needs.frontend-build.outputs.embed_artifact_id }}+ EXPECTED_EMBED_SHA256: ${{ needs.frontend-build.outputs.embed_sha256 }}+ run: |+ set -euo pipefail+ [[ "$EMBED_ARTIFACT_ID" =~ ^[1-9][0-9]*$ ]]+ [[ "$EXPECTED_EMBED_SHA256" =~ ^[a-f0-9]{64}$ ]]+- name: Download this run's validated frontend embeduses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2with:- name: ci-frontend-embed-${{ github.sha }}-${{ github.run_attempt }}+ artifact-ids: ${{ needs.frontend-build.outputs.embed_artifact_id }}path: ${{ runner.temp }}/frontend-embed+ merge-multiple: true- name: Verify and restore the validated frontend embedenv:diff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.mdindex 68fad9f206..4b6501997a 100644--- a/docs/release-control/v6/internal/subsystems/deployment-installability.md+++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md@@ -833,15 +833,22 @@ lint and type checks, then builds and size-checks the production frontend. Itrequires a nonempty index/assets tree and compares the complete generatedfrontend and embed copy. Only that successful checkout-bound build is packedand transferred as an immutable, one-day artifact named for this run's source-and attempt, with its SHA256 separately exported by the producer. Successful-build/size/transfer phases emit the explicit admission output. Unit shards and-`frontend-checks` then run independently, rather than making every unit shard+and producer attempt. The producer exports the upload's immutable artifact ID+and the packet's SHA256 separately; missing or malformed upload identity cannot+emit admission. Successful build/size/transfer phases emit the explicit admission+output. Unit shards and `frontend-checks` then run independently, rather than making every unit shardwait for backend compilation and runtime checks. An audit failure still permitsthat diagnostic proof but always fails the required aggregate.-The runtime job downloads only the same run/source/attempt artifact. It verifies-the checkout and producer digest before opening the archive, rejects entries-outside the two fixed dist trees, traversal and non-file/non-directory entries,+The runtime job requires a single positive artifact ID and a valid producer+digest before downloading only that ID from the current run, with no alternate+run, repository or token selector. It consumes the successful producer's outputs,+never a name recomputed from its own attempt: a partial failed-jobs retry on+attempt two still consumes the untouched attempt-one producer, while a full+rerun consumes its new producer's ID and digest even when both artifacts exist.+Missing identity fails before download rather than selecting all artifacts.+It verifies the checkout and producer digest before opening the archive, rejects+entries outside the two fixed dist trees, traversal and non-file/non-directory entries,and safely restores the complete built bytes. Missing, extra or changed embeddedfiles stop the phase; it never creates a placeholder or rebuilds a differentbundle. Go comes from `go.mod`; build and server/API vet retain the read-only@@ -856,8 +863,11 @@ introduced. Type/build/size failure still stops the expensive suite early.`TestFrontendCIRequiresSameCheckoutRealEmbedBackend` executes the exact phasewith private assets and Git/compiler doubles, covering mismatch refusal,declared command/ordering checks and terminal compiler/vet failures. Its transfer-controls also execute the actual pack/restore shells, verify byte-for-byte-round trips and reject source, digest, missing-packet and unsafe-member controls.+controls also execute the actual pack/restore and identity-admission shells,+verify byte-for-byte round trips and reject source, digest, missing-packet and+unsafe-member controls. Confined artifact/output fixtures cover producer-one /+consumer-two retries, full attempt-two reruns and the old consumer-attempt name+selection failure; they do not reproduce hosted GitHub rerun/transport behaviour.`TestFrontendCIBuildTransferIsSourceBound` runs those transfer controls explicitlyin the Go install-test suite, separately from compiler/asset admission. Thesecontrols are not real compilation. `TestFrontendCIRequiresCompleteIsolatedUnitProof`diff --git a/scripts/installtests/build_release_assets_test.go b/scripts/installtests/build_release_assets_test.goindex ff89ac279c..a365cfd9c9 100644--- a/scripts/installtests/build_release_assets_test.go+++ b/scripts/installtests/build_release_assets_test.go@@ -4523,15 +4523,19 @@ func TestFrontendCIRequiresSameCheckoutRealEmbedBackend(t *testing.T) {// Execute the exact producer/consumer shells, including every-byte round trips// and source, digest and unsafe-member refusals before any asset extraction.+// Retained producer-output fixtures also cover partial/full retry selection+// and missing identity refusing admission before any artifact download.// Artifact transport and hosted overlap still need containing CI observations.func TestFrontendCIBuildTransferIsSourceBound(t *testing.T) {ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)defer cancel()cmd := exec.CommandContext(ctx, "python3", repoFile("scripts", "tests", "test_frontend_real_embed.py"), "FrontendEmbedTransfer", "-v")cmd.Env = append(os.Environ(), "PULSE_BUILD_WORKFLOW_UNDER_TEST="+repoFile(".github", "workflows", "build-and-test.yml"))- if out, err := cmd.CombinedOutput(); err != nil {+ out, err := cmd.CombinedOutput()+ if err != nil {t.Fatalf("source-bound frontend build transfer controls: %v\n%s", err, out)}+ t.Logf("source-bound frontend transfer and retry controls:\n%s", out)}// Keep the registered frontend security-proof route bound to the actualdiff --git a/scripts/tests/test_frontend_real_embed.py b/scripts/tests/test_frontend_real_embed.pyindex c9c3a64e4c..7272806e3b 100644--- a/scripts/tests/test_frontend_real_embed.py+++ b/scripts/tests/test_frontend_real_embed.py@@ -37,7 +37,8 @@ def recipe(source):build, producer = workflow_steps(source, "frontend-build")job, steps = workflow_steps(source, "frontend-checks")order = list(steps)- required = ["Checkout repository", "Download this run's validated frontend embed",+ required = ["Checkout repository", "Check validated frontend embed identity",+ "Download this run's validated frontend embed","Verify and restore the validated frontend embed", GO_SETUP, VERIFY]if any(name not in order for name in required) or [order.index(name) for name in required] != sorted(order.index(name) for name in required):raise AssertionError("same-run build transfer or real-embed ordering missing")@@ -160,13 +161,16 @@ class FrontendEmbedTransfer(unittest.TestCase):def setUp(self):self.source = WORKFLOW.read_text()- _, self.producer = workflow_steps(self.source, "frontend-build")+ self.build, self.producer = workflow_steps(self.source, "frontend-build")_, self.consumer = workflow_steps(self.source, "frontend-checks")- def test_transfer_is_fixed_to_this_run_source_and_attempt(self):+ def test_transfer_selects_producer_immutable_identity_in_this_run(self):upload = self.producer["Transfer the validated frontend embed"]download = self.consumer["Download this run's validated frontend embed"]self.assertEqual(upload["uses"], "actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9")+ self.assertEqual(upload["id"], "embed-upload")+ self.assertEqual(self.build["outputs"]["embed_artifact_id"], "${{ steps.embed-upload.outputs.artifact-id }}")+ self.assertEqual(self.build["outputs"]["embed_sha256"], "${{ steps.embed-packet.outputs.sha256 }}")self.assertEqual(upload["with"], {"name": "ci-frontend-embed-${{ github.sha }}-${{ github.run_attempt }}","path": "${{ runner.temp }}/frontend-embed.tar", "if-no-files-found": "error",@@ -174,7 +178,15 @@ class FrontendEmbedTransfer(unittest.TestCase):})self.assertEqual(download["uses"], "actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333")self.assertEqual(download["with"], {- "name": upload["with"]["name"], "path": "${{ runner.temp }}/frontend-embed",+ "artifact-ids": "${{ needs.frontend-build.outputs.embed_artifact_id }}",+ "path": "${{ runner.temp }}/frontend-embed", "merge-multiple": True,+ })+ self.assertEqual(self.producer["Admit frontend unit shards"]["env"], {+ "EMBED_ARTIFACT_ID": "${{ steps.embed-upload.outputs.artifact-id }}",+ })+ self.assertEqual(self.consumer["Check validated frontend embed identity"]["env"], {+ "EMBED_ARTIFACT_ID": "${{ needs.frontend-build.outputs.embed_artifact_id }}",+ "EXPECTED_EMBED_SHA256": "${{ needs.frontend-build.outputs.embed_sha256 }}",})self.assertEqual(self.producer["Bind the validated frontend embed"]["env"], {"EXPECTED_SOURCE_SHA": "${{ github.sha }}"})self.assertEqual(self.consumer["Verify and restore the validated frontend embed"]["env"], {@@ -182,6 +194,105 @@ class FrontendEmbedTransfer(unittest.TestCase):"EXPECTED_SOURCE_SHA": "${{ github.sha }}",})+ @staticmethod+ def expand(value, context):+ """Resolve only the dotted output references used by this recipe.++ This is not a workflow engine: the caller supplies retained successful+ producer outputs, as failed-jobs reruns do, or new full-rerun outputs.+ """+ def substitute(match):+ result = context+ for key in match.group(1).strip().split("."):+ result = result[key]+ return str(result)+ return re.sub(r"\$\{\{\s*([\w. -]+)\s*\}\}", substitute, value)++ def check_identity(self, step, env):+ with tempfile.TemporaryDirectory(prefix="pulse-embed-identity-") as directory:+ output = Path(directory) / "outputs"+ result = subprocess.run(["bash", "-c", step["run"]],+ env=dict(os.environ, **env, GITHUB_OUTPUT=str(output)),+ text=True, capture_output=True, timeout=5)+ return result, output.read_text() if output.exists() else ""++ def select_artifact(self, *, producer_attempt, consumer_attempt, full_rerun=False,+ legacy_consumer_name=False):+ """Private immutable-artifact fixtures exercise the actual selectors.++ No GitHub upload, download or hosted rerun is performed. Both runs and+ both attempts deliberately coexist to expose wrong/latest selection.+ """+ upload = self.producer["Transfer the validated frontend embed"]+ download = self.consumer["Download this run's validated frontend embed"]+ # The current-run store is the download action's default scope; the+ # static contract above refuses repository/run-id/token overrides.+ artifacts = {"current": {}, "other-run": {}}+ for attempt in ((1, 2) if full_rerun else (1,)):+ producer_context = {"github": {"sha": self.SHA, "run_attempt": attempt}}+ artifacts["current"][str(1000 + attempt)] = {+ "name": self.expand(upload["with"]["name"], producer_context),+ "digest": hashlib.sha256(f"fixture build {attempt}".encode()).hexdigest(),+ "producer_attempt": attempt,+ }+ # Same name in another run must not become an alternate source.+ artifacts["other-run"]["9001"] = dict(artifacts["current"]["1001"])+ producer_id = str(1000 + producer_attempt)+ producer_digest = artifacts["current"][producer_id]["digest"]+ output_context = {"steps": {+ upload["id"]: {"outputs": {"artifact-id": producer_id}},+ self.producer["Bind the validated frontend embed"]["id"]: {"outputs": {"sha256": producer_digest}},+ }}+ outputs = {key: self.expand(value, output_context) for key, value in self.build["outputs"].items()+ if key in ("embed_artifact_id", "embed_sha256")}+ context = {"github": {"sha": self.SHA, "run_attempt": consumer_attempt},+ "needs": {"frontend-build": {"outputs": outputs}}}+ identity = self.consumer["Check validated frontend embed identity"]+ result, _ = self.check_identity(identity, {key: self.expand(value, context) for key, value in identity["env"].items()})+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)+ if legacy_consumer_name:+ # The rejected recipe independently resolved the upload name in+ # the consumer. Retain that exact fault as a negative control.+ name = self.expand(upload["with"]["name"], context)+ selected = [artifact for artifact in artifacts["current"].values() if artifact["name"] == name]+ if len(selected) != 1:+ raise ValueError("consumer-attempt artifact does not exist")+ selected = selected[0]+ else:+ selected_id = self.expand(download["with"]["artifact-ids"], context)+ selected = artifacts["current"][selected_id]+ self.assertEqual(selected_id, producer_id)+ restore = self.consumer["Verify and restore the validated frontend embed"]+ self.assertEqual(self.expand(restore["env"]["EXPECTED_EMBED_SHA256"], context), selected["digest"])+ return selected["producer_attempt"]++ def test_partial_retry_consumes_untouched_attempt_one_producer(self):+ self.assertEqual(self.select_artifact(producer_attempt=1, consumer_attempt=2), 1)++ def test_full_retry_selects_new_attempt_two_producer_not_old_artifact(self):+ self.assertEqual(self.select_artifact(producer_attempt=2, consumer_attempt=2, full_rerun=True), 2)++ def test_retry_fixture_exposes_the_rejected_consumer_attempt_name(self):+ with self.assertRaisesRegex(ValueError, "consumer-attempt artifact does not exist"):+ self.select_artifact(producer_attempt=1, consumer_attempt=2, legacy_consumer_name=True)++ def test_missing_or_malformed_identity_never_admits_or_downloads(self):+ for artifact_id in ("", "0", "-1", "1,2", "1.5", " 1001", "1001\n1002", "latest", "${{ missing }}"):+ for name, step in (("producer", self.producer["Admit frontend unit shards"]),+ ("consumer", self.consumer["Check validated frontend embed identity"])):+ with self.subTest(phase=name, artifact_id=artifact_id):+ result, output = self.check_identity(step, {"EMBED_ARTIFACT_ID": artifact_id, "EXPECTED_EMBED_SHA256": "a" * 64})+ self.assertNotEqual(result.returncode, 0)+ self.assertEqual(output, "")+ for digest in ("", "a" * 63, "A" * 64, "not-a-digest"):+ with self.subTest(digest=digest):+ result, _ = self.check_identity(self.consumer["Check validated frontend embed identity"],+ {"EMBED_ARTIFACT_ID": "1001", "EXPECTED_EMBED_SHA256": digest})+ self.assertNotEqual(result.returncode, 0)+ result, output = self.check_identity(self.producer["Admit frontend unit shards"], {"EMBED_ARTIFACT_ID": "1001"})+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)+ self.assertEqual(output, "admitted=true\n")+def transfer(self, mutation=""):with tempfile.TemporaryDirectory(prefix="pulse-embed-transfer-") as directory:base = Path(directory)diff --git a/scripts/tests/test_frontend_workflow_order.py b/scripts/tests/test_frontend_workflow_order.pyindex a7e8b1cb5c..118e730606 100644--- a/scripts/tests/test_frontend_workflow_order.py+++ b/scripts/tests/test_frontend_workflow_order.py@@ -75,13 +75,15 @@ class FrontendWorkflowOrderTest(unittest.TestCase):def test_unit_admission_is_after_bundle_transfer_but_before_the_audit_verdict(self):admission = self.step("Admit frontend unit shards")- self.assertEqual(admission, {- "name": "Admit frontend unit shards", "id": "unit-admission",- "run": "echo 'admitted=true' >> \"$GITHUB_OUTPUT\"",+ self.assertEqual(admission["id"], "unit-admission")+ self.assertEqual(admission["env"], {+ "EMBED_ARTIFACT_ID": "${{ steps.embed-upload.outputs.artifact-id }}",})+ self.assertIn('echo \'admitted=true\' >> "$GITHUB_OUTPUT"', admission["run"])self.assertEqual(self.job["outputs"], {"unit_admitted": "${{ steps.unit-admission.outputs.admitted }}","embed_sha256": "${{ steps.embed-packet.outputs.sha256 }}",+ "embed_artifact_id": "${{ steps.embed-upload.outputs.artifact-id }}",})self.assertEqual(self.job["needs"], "changes")self.assertEqual(self.job["if"], "needs.changes.outputs.code == 'true'")@@ -132,7 +134,8 @@ class FrontendWorkflowOrderTest(unittest.TestCase):self.assertEqual(runtime["timeout-minutes"], 30)self.assertNotIn("continue-on-error", runtime)names = [step["name"] for step in runtime["steps"]]- ordered = ["Download this run's validated frontend embed",+ ordered = ["Check validated frontend embed identity",+ "Download this run's validated frontend embed","Verify and restore the validated frontend embed","Set up Go for real-embed verification", "Verify backend with real frontend embed","Check current-source unattended recovery on native systemd",