Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Check unattended updater transactions on native source CI
pulse-triage[bot]ba2a8f99034d
diff --git a/.github/workflows/build-and-test.yml b/.github/workflows/build-and-test.ymlindex 822ca3d6d8..35dc925f35 100644--- a/.github/workflows/build-and-test.yml+++ b/.github/workflows/build-and-test.yml@@ -290,6 +290,34 @@ jobs:go vet -mod=readonly -p=2 ./cmd/pulse ./internal/apisha256sum "$RUNNER_TEMP/pulse-real-embed"+ # #2785: confined manager doubles cannot establish native recovery.+ # Reuse this job's real embed/cache on a fresh hosted VM, not a release,+ # customer LXC, privileged worker or new maintainer dispatcher.+ - name: Check current-source unattended recovery on native systemd+ env:+ EXPECTED_SOURCE_SHA: ${{ github.sha }}+ run: |+ set -euo pipefail+ test "$(git rev-parse HEAD)" = "$EXPECTED_SOURCE_SHA"+ diff -qr --no-dereference frontend-modern/dist internal/api/frontend-modern/dist+ go build -mod=readonly -p=2 -ldflags='-X main.Version=v0.0.1' -o "$RUNNER_TEMP/pulse-native-before" ./cmd/pulse+ go build -mod=readonly -p=2 -ldflags='-X main.Version=v0.0.2' -o "$RUNNER_TEMP/pulse-native-after" ./cmd/pulse+ sudo env -i PATH=/usr/sbin:/usr/bin:/sbin:/bin LANG=C.UTF-8 \+ GITHUB_ACTIONS=true RUNNER_ENVIRONMENT=github-hosted GITHUB_REPOSITORY=rcourtman/Pulse \+ EXPECTED_SOURCE_SHA="$EXPECTED_SOURCE_SHA" RUNNER_TEMP="$RUNNER_TEMP" \+ python3 scripts/installtests/auto_update_systemd.py \+ --before "$RUNNER_TEMP/pulse-native-before" --after "$RUNNER_TEMP/pulse-native-after" \+ --receipt "$RUNNER_TEMP/pulse-native-updater.json"++ - name: Retain bounded native updater receipt+ if: always()+ uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1+ with:+ name: native-updater-${{ github.run_id }}-${{ github.run_attempt }}+ path: ${{ runner.temp }}/pulse-native-updater.json+ if-no-files-found: warn+ retention-days: 7+# An audit service failure or real advisory stays gating, but it must not# suppress formatting, tests, type-checking, or production-build proof.- name: Require frontend dependency auditdiff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.mdindex cccb5d6a0e..e71c6e2ea3 100644--- a/docs/release-control/v6/internal/subsystems/deployment-installability.md+++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md@@ -99,6 +99,40 @@ restoration safeguards. These factscannot reconstruct exits or signals from an earlier interrupted attempt, oridentify who stopped a host.+### Current-source unattended transactions have a native CI check++After the same checkout's production frontend, backend build and vet, ordinary+Frontend CI links two real server executables with **synthetic** v0.0.1/v0.0.2+version ldflags. On that fresh GitHub-hosted VM, `auto_update_systemd.py` uses+real installer archive staging, unit/helper writers, the unattended transaction+and generated manual recovery helper. It observes actual systemd load/activity,+mask/enablement, installed and running executable hashes, healthy served version,+persistent-file hashes, restoration mode, backup disposition and owned cleanup.+Seven phases cover forward/manual reverse, failed active/inactive/masked installs,+invalid signatures and an incomplete restore that retains its backup without start.++The test uses signed local archives/installers and an ephemeral test signer;+production source pins are unchanged. Only fixed fixture download transport and+private temporary-file placement are adapted. No network fallback, manager/sleep+double, production credential, release publication or container-detection bypass+is admitted by the native runner. The real unit keeps ProtectHome, PrivateTmp and+ProtectSystem; its fixed owned `/opt/pulse-source-native` lab avoids hidden home+and private temporary paths. Existing units/scratch refuse before mutation, and+cleanup removes only objects created by this attempt. Compiler and root-command+failures stay gating under the existing job's deadline; a seven-day closed receipt+contains source/binary/output hashes, all phases and cleanup, never signer keys,+persistent contents or raw application/installer logs.++`TestAutoUpdateNativeSystemdHarnessContracts` checks the exact recipe, admission,+closed transport, real local signatures and phase refusals, and runs the connected+source/archive/recovery orchestration with an explicitly confined manager/account/+unit-path double and tiny ELF fixtures. Those controls are not native acceptance.+The first actually containing CI receipt/logs must establish the native result.+Even a complete native synthetic transaction does **not** establish a published+signed historical release pair, full installer main, timer-service sandbox,+authenticated browser journey, Proxmox LXC stop cause or reporter recovery.+The existing published-release routes and field owners retain those outcomes.+### Ordinary CI compiles the backend with the same checkout's real frontendThe existing Frontend job retains its production bundle, size, whole-tree tests,diff --git a/docs/release-control/v6/internal/subsystems/registry.json b/docs/release-control/v6/internal/subsystems/registry.jsonindex a36932d2d1..11253e1b37 100644--- a/docs/release-control/v6/internal/subsystems/registry.json+++ b/docs/release-control/v6/internal/subsystems/registry.json@@ -4957,6 +4957,20 @@"scripts/release_control/ssh_host_key_policy_test.py"]},+ {+ "id": "current-source-unattended-systemd",+ "label": "current-source native updater harness proof",+ "match_prefixes": [],+ "match_files": [+ "scripts/installtests/auto_update_systemd.py"+ ],+ "allow_same_subsystem_tests": false,+ "test_prefixes": [],+ "exact_files": [+ "scripts/tests/test_auto_update_systemd.py",+ "scripts/installtests/pulse_auto_update_test.go"+ ]+ },{"id": "unattended-update-runtime","label": "unattended updater runtime proof",diff --git a/scripts/installtests/auto_update_systemd.py b/scripts/installtests/auto_update_systemd.pynew file mode 100644index 0000000000..8134a4e414--- /dev/null+++ b/scripts/installtests/auto_update_systemd.py@@ -0,0 +1,405 @@+#!/usr/bin/env python3+"""Current-source updater transactions on a disposable GitHub-hosted systemd VM.++The two real, real-embed binaries differ only in synthetic version ldflags.+Installers and archives use an ephemeral test signer, never a release key.+Only download transport and /tmp allocation are redirected to private fixtures;+production signature, replacement, service observation and recovery functions run.+This is not a published-release/LXC, timer-sandbox or browser acceptance test.+"""++import argparse+import hashlib+import json+import os+from pathlib import Path+import re+import shutil+import signal+import subprocess+import tarfile+import time+import urllib.error+import urllib.request++UNIT = "pulse-source-updater"+TAGS = ("v0.0.1", "v0.0.2")+PORT = 17655+PARSE_MARKER = "# Parse command line arguments\n"+++def digest(path):+ with Path(path).open("rb") as stream:+ return hashlib.file_digest(stream, "sha256").hexdigest()+++def installer_definitions(source):+ if source.count(PARSE_MARKER) != 1:+ raise ValueError("installer definition boundary changed")+ definitions, entry = source.split(PARSE_MARKER)+ if not entry.startswith("while [[ $# -gt 0 ]]; do\n"):+ raise ValueError("installer entry boundary changed")+ return definitions+++def hosted_admission(env, uid, source):+ if (uid != 0 or env.get("GITHUB_ACTIONS") != "true"+ or env.get("RUNNER_ENVIRONMENT") != "github-hosted"+ or env.get("GITHUB_REPOSITORY") != "rcourtman/Pulse"+ or not re.fullmatch(r"[0-9a-f]{40}", source)+ or any(Path(p).exists() for p in ("/.dockerenv", "/opt/pulse", "/etc/pulse"))):+ raise ValueError("native updater proof requires a fresh public hosted VM")+++# A closed adapter: missing fixtures fail, never fall back to online curl.+TRANSPORT = r'''#!/usr/bin/python3+import os, pathlib, shutil, subprocess, sys+root = pathlib.Path(os.environ["PULSE_NATIVE_ROOT"])+args = sys.argv[1:]+if pathlib.Path(sys.argv[0]).name == "mktemp":+ if args and args[-1].startswith("/tmp/pulse-"):+ args[-1] = str(root / "tmp" / pathlib.Path(args[-1]).name)+ os.execv("/usr/bin/mktemp", ["mktemp", *args])+out = None+url = None+while args:+ arg = args.pop(0)+ if arg == "-o" and args:+ out = pathlib.Path(args.pop(0))+ elif arg == "-fsSL":+ pass+ elif arg.startswith("https://") and url is None:+ url = arg+ else:+ sys.exit(97)+allowed = {"https://github.com/rcourtman/Pulse/releases/latest/download/install.sh": "installer"}+for tag in ("v0.0.1", "v0.0.2"):+ allowed["https://github.com/rcourtman/Pulse/releases/download/" + tag + "/install.sh"] = "installer"+allowed.update({url + ".sshsig": name + ".sshsig" for url, name in list(allowed.items())})+name = allowed.get(url)+if name is None:+ sys.exit(97)+file = root / name+if out is None:+ sys.stdout.buffer.write(file.read_bytes())+else:+ shutil.copyfile(file, out)+'''++INSTALLER = r'''#!/bin/bash+set -euo pipefail+[[ $# == 2 && "$1" == --version ]]+case "$2" in v0.0.1|v0.0.2) ;; *) exit 97 ;; esac+source "$PULSE_NATIVE_ROOT/installer-definitions"+PINNED_RELEASE_SSH_PUBLIC_KEY=$(cat "$PULSE_NATIVE_ROOT/signer/key.pub")+printf 'attempted\n' >> "$PULSE_NATIVE_ROOT/installer-calls"+install_pulse_archive "$PULSE_NATIVE_ROOT/archives/pulse-$2-linux-amd64.tar.gz" "$2"+# Fail after real signed replacement, not before the interesting transaction.+case "${PULSE_NATIVE_FAULT:-}" in installer|restore) exit 23 ;; esac+if [[ "$PULSE_WAS_ACTIVE" == true ]]; then+ systemctl start "$SERVICE_NAME"+fi+'''++UPDATER = r'''+set -euo pipefail+source "$PULSE_NATIVE_UPDATER"+PINNED_RELEASE_SSH_PUBLIC_KEY=$(cat "$PULSE_NATIVE_ROOT/signer/key.pub")+# One explicit restore-copy failure tests retained backup/no unsafe restart.+if [[ "${PULSE_NATIVE_FAULT:-}" == restore ]]; then+ cp() {+ if [[ "${*: -1}" == *.rollback.* ]]; then return 71; fi+ command cp "$@"+ }+fi+status=0+perform_update "$PULSE_NATIVE_TARGET" || status=$?+# Exercise a subsequent function return: the transaction's trap must be gone.+is_prerelease_tag v0.0.3-rc.1+exit "$status"+'''+++class NativeLab:+ def __init__(self, root, repo, before, after, source):+ self.root, self.repo = Path(root), Path(repo)+ self.binaries = dict(zip(TAGS, (Path(before), Path(after))))+ self.receipt = {"schema_version": 1, "source_sha": source,+ "scope": "current-source-synthetic-signed-systemd-transaction",+ "phases": [], "cleanup": {"complete": False}}+ self.env = {"PATH": str(self.root / "tools") + ":/usr/sbin:/usr/bin:/sbin:/bin",+ "LANG": "C.UTF-8", "LC_ALL": "C", "HOME": str(self.root),+ "PULSE_NATIVE_ROOT": str(self.root),+ "PULSE_NATIVE_UPDATER": str(self.repo / "scripts/pulse-auto-update.sh"),+ "PULSE_SERVICE_NAME": UNIT, "PULSE_INSTALL_DIR": str(self.root / "install"),+ "PULSE_CONFIG_DIR": str(self.root / "data"),+ "PULSE_BINARY_LINK_PATH": str(self.root / "pulse"),+ "PULSE_UPDATE_HELPER_PATH": str(self.root / "update"),+ "PULSE_PROFILE_PATH": str(self.root / "profile"),+ "PULSE_BASHRC_PATH": str(self.root / "bashrc"), "FRONTEND_PORT": str(PORT)}+ self.unit = Path("/etc/systemd/system") / (UNIT + ".service")+ self.owned_unit = False+ self.owned_root = False+ self.owned_user = False++ def run(self, *args, timeout=120, ok=(0,)):+ process = subprocess.Popen(args, env=self.env, stdin=subprocess.DEVNULL,+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT,+ start_new_session=True)+ try:+ output, _ = process.communicate(timeout=timeout)+ except subprocess.TimeoutExpired:+ os.killpg(process.pid, signal.SIGTERM)+ try:+ process.communicate(timeout=2)+ except subprocess.TimeoutExpired:+ os.killpg(process.pid, signal.SIGKILL)+ process.communicate(timeout=2)+ self.receipt["command_failure"] = {"command": Path(args[0]).name,+ "deadline_exceeded": True,+ "observed_exit": process.returncode}+ raise RuntimeError("native command deadline exceeded") from None+ result = subprocess.CompletedProcess(args, process.returncode, output)+ if result.returncode not in ok:+ self.receipt["command_failure"] = {"command": Path(args[0]).name,+ "observed_exit": result.returncode,+ "output_bytes": len(output),+ "output_sha256": hashlib.sha256(output).hexdigest()}+ raise RuntimeError(f"native command failed ({result.returncode}): {Path(args[0]).name}")+ return result++ def shell(self, script, **env):+ script = "set -euo pipefail\n" + script+ old = self.env.copy()+ try:+ self.env.update(env)+ return self.run("/bin/bash", "-c", script)+ finally:+ self.env = old++ def state(self):+ result = self.run("systemctl", "show", UNIT + ".service", "--no-pager",+ "--property=LoadState,ActiveState,UnitFileState,MainPID")+ pairs = dict(line.split("=", 1) for line in result.stdout.decode().splitlines())+ if set(pairs) != {"LoadState", "ActiveState", "UnitFileState", "MainPID"}:+ raise RuntimeError("incomplete native manager observation")+ return pairs++ def prepare(self):+ for path in (self.unit, Path("/run/systemd/system") / self.unit.name):+ if path.exists() or path.is_symlink():+ raise ValueError("native fixture unit already exists")+ if self.state()["LoadState"] != "not-found":+ raise ValueError("native fixture unit is already known to systemd")+ self.root.mkdir(mode=0o755) # A pulse-owned service must traverse it.+ self.owned_root = True+ for name in ("tools", "tmp", "archives", "signer"):+ (self.root / name).mkdir(mode=0o700 if name == "signer" else 0o755)+ source = (self.repo / "install.sh").read_text()+ (self.root / "installer-definitions").write_text(installer_definitions(source))+ self.receipt["sources"] = {p: digest(self.repo / p) for p in+ ("install.sh", "scripts/pulse-auto-update.sh",+ "scripts/installtests/auto_update_systemd.py")}+ self.receipt["binaries"] = {tag: digest(binary) for tag, binary in self.binaries.items()}+ if len(set(self.receipt["binaries"].values())) != 2:+ raise ValueError("same executable bytes prove no replacement")+ for tag, binary in self.binaries.items():+ if self.run(str(binary), "--version").stdout.decode().splitlines()[0] != "Pulse " + tag:+ raise ValueError("synthetic binary identity mismatch")+ self.run("ssh-keygen", "-q", "-t", "ed25519", "-N", "", "-f", str(self.root / "signer/key"))+ for tag, binary in self.binaries.items():+ version = self.root / "VERSION"+ version.write_text(tag + "\n")+ archive = self.root / "archives" / f"pulse-{tag}-linux-amd64.tar.gz"+ with tarfile.open(archive, "w:gz") as tar:+ tar.add(binary, arcname="bin/pulse")+ tar.add(version, arcname="VERSION")+ self.sign(archive)+ (self.root / "installer").write_text(INSTALLER)+ self.sign(self.root / "installer")+ for name in ("curl", "mktemp"):+ path = self.root / "tools" / name+ path.write_text(TRANSPORT)+ path.chmod(0o755)+ # Establish the baseline using real source staging and unit/helper writers.+ self.owned_user = self.run("id", "-u", "pulse", ok=(0, 1)).returncode == 1+ self.owned_unit = True # Any partial writer output is ours to clean up.+ self.shell('source "$PULSE_NATIVE_ROOT/installer-definitions"; '+ 'PINNED_RELEASE_SSH_PUBLIC_KEY=$(cat "$PULSE_NATIVE_ROOT/signer/key.pub"); '+ 'create_user; setup_directories; '+ 'install_pulse_archive "$PULSE_NATIVE_ROOT/archives/pulse-v0.0.1-linux-amd64.tar.gz" v0.0.1; '+ 'install_systemd_service; setup_update_command')+ data = self.root / "data"+ (data / ".env").write_text("PULSE_TELEMETRY=false\n")+ (data / "system.json").write_text('{"autoUpdateEnabled":false,"updateChannel":"stable"}\n')+ (data / "native-persistence-sentinel").write_text("original persistent data\n")+ self.run("systemctl", "start", self.unit.name)+ self.assert_runtime(TAGS[0], "active")+ time.sleep(2) # Let asynchronous initial persistence settle once.+ files = (".env", "system.json", ".encryption.key", "native-persistence-sentinel")+ self.baseline_data = {p: digest(data / p) for p in files}+ self.baseline_mode = (self.root / "install/bin/pulse").stat().st_mode & 0o777++ def sign(self, path):+ self.run("ssh-keygen", "-Y", "sign", "-f", str(self.root / "signer/key"),+ "-n", "pulse-install", str(path))+ Path(str(path) + ".sig").rename(str(path) + ".sshsig")++ def assert_runtime(self, tag, active):+ binary = self.root / "install/bin/pulse"+ if digest(binary) != self.receipt["binaries"][tag]:+ raise RuntimeError("installed executable differs from authenticated fixture")+ if (self.root / "install/VERSION").read_text() != tag + "\n":+ raise RuntimeError("version sidecar differs")+ state = self.state()+ if state["ActiveState"] != active:+ raise RuntimeError("prior activity was not restored")+ if active == "active":+ if digest(Path("/proc") / state["MainPID"] / "exe") != digest(binary):+ raise RuntimeError("running executable differs from installed executable")+ deadline = time.monotonic() + 30+ while True:+ try:+ observations = {}+ for endpoint in ("health", "version"):+ with urllib.request.urlopen(f"http://127.0.0.1:{PORT}/api/{endpoint}", timeout=2) as response:+ body = response.read(16385)+ if len(body) > 16384:+ raise RuntimeError("native API observation too large")+ observations[endpoint] = json.loads(body)+ if observations["version"].get("version", "").lstrip("v") != tag.lstrip("v"):+ raise RuntimeError("served version differs")+ if observations["health"].get("status") != "healthy":+ raise RuntimeError("native application is not healthy")+ break+ except urllib.error.HTTPError:+ # Do not reinterpret refusal as a restart gap or retry it.+ raise RuntimeError("native API request refused") from None+ except OSError:+ if time.monotonic() >= deadline:+ raise RuntimeError("native application readiness deadline") from None+ time.sleep(0.25)+ return state++ def phase(self, name, tag, expected_exit=0, active="active", fault="", retain=False, manual=False):+ before_state = self.state()+ self.env.update(PULSE_NATIVE_TARGET=tag, PULSE_NATIVE_FAULT=fault)+ result = self.run("/bin/bash", "-c", '"$PULSE_UPDATE_HELPER_PATH" --version "$PULSE_NATIVE_TARGET"'+ if manual else UPDATER, ok=(expected_exit,))+ (self.root / (name + ".log")).write_bytes(result.stdout)+ expected_tag = TAGS[1] if retain else (TAGS[0] if expected_exit else tag)+ state = self.assert_runtime(expected_tag, active)+ if state["LoadState"] != before_state["LoadState"] or state["UnitFileState"] != before_state["UnitFileState"]:+ raise RuntimeError("unit mask/enablement changed")+ if {p: digest(self.root / "data" / p) for p in self.baseline_data} != self.baseline_data:+ raise RuntimeError("persistent state changed")+ backups = list((self.root / "tmp").glob("pulse-backup.*"))+ if len(backups) != (1 if retain else 0):+ raise RuntimeError("rollback backup disposition differs")+ if retain and digest(backups[0] / "pulse-bin") != self.receipt["binaries"][TAGS[0]]:+ raise RuntimeError("retained rollback backup differs from baseline")+ if not retain and list((self.root / "install/bin").glob("*.rollback.*")):+ raise RuntimeError("rollback staging leaked")+ if digest(self.root / "install/bin/pulse") == self.receipt["binaries"][TAGS[0]]:+ if (self.root / "install/bin/pulse").stat().st_mode & 0o777 != self.baseline_mode:+ raise RuntimeError("baseline executable mode not restored")+ self.receipt["phases"].append({"name": name, "exit": result.returncode, "passed": True,+ "installed_tag": expected_tag, "state": state,+ "persistent_hashes_unchanged": True, "backup_retained": retain,+ "output_sha256": hashlib.sha256(result.stdout).hexdigest()})+ print("NATIVE_UPDATER_PHASE=" + name, flush=True)++ def exercise(self):+ self.phase("forward", TAGS[1])+ self.phase("manual-reverse", TAGS[0], manual=True)+ self.phase("failed-active", TAGS[1], 1, fault="installer")+ self.run("systemctl", "stop", self.unit.name)+ self.phase("failed-inactive", TAGS[1], 1, active="inactive", fault="installer")+ # A real /dev/null mask, not a manager double returning 'masked'.+ saved_unit = self.unit.read_bytes()+ self.unit.unlink()+ self.run("systemctl", "mask", self.unit.name)+ self.run("systemctl", "daemon-reload")+ self.phase("failed-masked", TAGS[1], 1, active="inactive", fault="installer")+ self.run("systemctl", "unmask", self.unit.name)+ self.unit.write_bytes(saved_unit)+ self.run("systemctl", "daemon-reload")+ self.run("systemctl", "start", self.unit.name)+ installer = self.root / "installer"+ original = installer.read_bytes()+ calls = (self.root / "installer-calls").read_bytes()+ installer.write_bytes(original + b"# tampered after signing\n")+ try:+ self.phase("bad-signature", TAGS[1], 1)+ if (self.root / "installer-calls").read_bytes() != calls:+ raise RuntimeError("invalid signature reached installer")+ finally:+ installer.write_bytes(original)+ # Native stop/byte replacement precede a deliberate failed restore copy.+ self.phase("incomplete-restore", TAGS[1], 1, active="inactive", fault="restore", retain=True)++ def cleanup(self):+ if self.owned_unit:+ state = self.state()+ if state["LoadState"] != "not-found":+ self.run("systemctl", "stop", self.unit.name)+ state = self.state()+ if state["ActiveState"] != "inactive" or state["MainPID"] != "0":+ raise RuntimeError("fixture service did not stop")+ # A failed writer may have left a file not yet known to the manager.+ self.unit.unlink(missing_ok=True)+ self.run("systemctl", "daemon-reload")+ self.run("systemctl", "reset-failed", self.unit.name, ok=(0, 1))+ if self.state()["LoadState"] != "not-found" or self.state()["MainPID"] != "0":+ raise RuntimeError("fixture unit/process remained")+ if self.owned_user:+ if self.run("id", "-u", "pulse", ok=(0, 1)).returncode == 0:+ self.run("userdel", "pulse")+ if self.run("id", "-u", "pulse", ok=(0, 1)).returncode != 1:+ raise RuntimeError("owned fixture account remained")+ if self.owned_root:+ # Delete only this invocation's owned signer and scratch directories.+ shutil.rmtree(self.root)+ self.receipt["cleanup"] = {"complete": True, "owned_unit_removed": self.owned_unit,+ "owned_account_removed": self.owned_user,+ "owned_fixture_removed": self.owned_root,+ "fixture_files_absent": not self.root.exists()}+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--before", required=True)+ parser.add_argument("--after", required=True)+ parser.add_argument("--receipt", required=True)+ args = parser.parse_args()+ source = os.environ.get("EXPECTED_SOURCE_SHA", "")+ hosted_admission(os.environ, os.geteuid(), source)+ runner_temp = Path(os.environ["RUNNER_TEMP"]).resolve()+ receipt = Path(args.receipt).resolve()+ if receipt.parent != runner_temp or receipt.exists():+ raise ValueError("receipt must be a new direct runner-temp file")+ # /home is hidden by the real unit's ProtectHome; /tmp and /var/tmp are+ # private under PrivateTmp. Preserve both protections, use a fixed /opt lab.+ root = Path("/opt/pulse-source-native")+ lab = NativeLab(root, Path(__file__).resolve().parents[2], args.before, args.after, source)+ failed = False+ try:+ lab.prepare()+ lab.exercise()+ except (OSError, ValueError, RuntimeError, subprocess.SubprocessError) as error:+ failed = True+ lab.receipt["failure"] = {"type": type(error).__name__, "message": str(error)}+ finally:+ try:+ lab.cleanup()+ except (OSError, RuntimeError, subprocess.SubprocessError) as error:+ failed = True+ lab.receipt["cleanup"] = {"complete": False, "error_type": type(error).__name__}+ receipt.write_text(json.dumps(lab.receipt, indent=2) + "\n")+ if failed or len(lab.receipt["phases"]) != 7:+ raise SystemExit("Native updater proof incomplete; inspect the bounded receipt.")+ print("NATIVE_UPDATER_COMPLETE=" + source)+++if __name__ == "__main__":+ main()diff --git a/scripts/installtests/build_release_assets_test.go b/scripts/installtests/build_release_assets_test.goindex cf99311c0b..6a3c248530 100644--- a/scripts/installtests/build_release_assets_test.go+++ b/scripts/installtests/build_release_assets_test.go@@ -4322,6 +4322,21 @@ func TestFrontendCIRequiresSameCheckoutRealEmbedBackend(t *testing.T) {}}+// The post-build root transaction receives only an exact-source identity and+// synthetic binaries; native failure cannot hide or skip the existing audit.+func TestFrontendCINativeUpdaterKeepsSecretFreeRequiredVerification(t *testing.T) {+ ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)+ defer cancel()+ cmd := exec.CommandContext(ctx, "python3", repoFile("scripts", "tests", "test_auto_update_systemd.py"), "-v",+ "NativeContracts.test_exact_workflow_recipe_is_scrubbed_and_propagates_all_failures",+ "NativeContracts.test_workflow_cannot_hide_or_rebind_native_step",+ "NativeContracts.test_host_admission_requires_fresh_public_vm_not_worker_or_container")+ out, err := cmd.CombinedOutput()+ if err != nil {+ t.Fatalf("secret-free native CI admission controls: %v\n%s", err, out)+ }+}+func TestFrontendDependencySecurityAuditsAreRequired(t *testing.T) {workflowPath := repoFile(".github", "workflows", "build-and-test.yml")assertFileContainsAll(t, workflowPath,diff --git a/scripts/installtests/pulse_auto_update_test.go b/scripts/installtests/pulse_auto_update_test.goindex 3369810ae9..b68e54cb47 100644--- a/scripts/installtests/pulse_auto_update_test.go+++ b/scripts/installtests/pulse_auto_update_test.go@@ -10,6 +10,19 @@ import ("time")+// The native CI runner is checked without host-manager access. These controls+// exercise its exact recipe, admission, signature/archive and recovery wiring;+// the hosted runner still supplies the actual systemd/real-Pulse verdict.+func TestAutoUpdateNativeSystemdHarnessContracts(t *testing.T) {+ ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)+ defer cancel()+ cmd := exec.CommandContext(ctx, "python3", repoFile("scripts", "tests", "test_auto_update_systemd.py"), "-v")+ out, err := cmd.CombinedOutput()+ if err != nil {+ t.Fatalf("native updater harness controls: %v\n%s", err, out)+ }+}+// The standalone helper passes an explicit name to the installer. Exercise// that producer and its real entry/handoff, not just installer discovery.func TestAutoUpdateServiceDiscoveryAndInstallerHandoff(t *testing.T) {diff --git a/scripts/tests/test_auto_update_systemd.py b/scripts/tests/test_auto_update_systemd.pynew file mode 100644index 0000000000..7d38aa9d19--- /dev/null+++ b/scripts/tests/test_auto_update_systemd.py@@ -0,0 +1,291 @@+#!/usr/bin/env python3+"""Discriminating controls for the native CI recipe; no host manager/root use."""+import importlib.util+import json+import os+from pathlib import Path+import re+import subprocess+import tempfile+import unittest+from unittest.mock import patch++ROOT = Path(__file__).resolve().parents[2]+SPEC = importlib.util.spec_from_file_location("native_updater", ROOT / "scripts/installtests/auto_update_systemd.py")+native = importlib.util.module_from_spec(SPEC)+SPEC.loader.exec_module(native)+WORKFLOW = ROOT / ".github/workflows/build-and-test.yml"+STEP = "Check current-source unattended recovery on native systemd"+++def recipe(source):+ job = re.search(r"(?ms)^ frontend:\n(.*?)(?=^ [a-z][a-z0-9-]*:|\Z)", source)[1]+ steps = dict(re.findall(r"(?ms)^ - name: ([^\n]+)\n(.*?)(?=^ - name:|\Z)", job))+ order = list(steps)+ assert order.index("Verify backend with real frontend embed") < order.index(STEP) < order.index("Require frontend dependency audit")+ step = steps[STEP]+ assert not re.search(r"^ (if|continue-on-error|working-directory):", step, re.M)+ assert "runs-on: ubuntu-24.04" in job and "timeout-minutes: 30" in job+ assert "persist-credentials: false" in steps["Checkout repository"]+ assert "EXPECTED_SOURCE_SHA: ${{ github.sha }}" in step+ shell = re.search(r"(?ms)^ run: \|\n((?: [^\n]*\n)+)", step)[1]+ shell = "\n".join(line[10:] for line in shell.splitlines()) + "\n"+ for command in ('test "$(git rev-parse HEAD)" = "$EXPECTED_SOURCE_SHA"',+ "diff -qr --no-dereference frontend-modern/dist internal/api/frontend-modern/dist",+ "-ldflags='-X main.Version=v0.0.1'", "-ldflags='-X main.Version=v0.0.2'",+ "go build -mod=readonly -p=2", "sudo env -i",+ "GITHUB_ACTIONS=true RUNNER_ENVIRONMENT=github-hosted GITHUB_REPOSITORY=rcourtman/Pulse",+ "python3 scripts/installtests/auto_update_systemd.py",+ '--receipt "$RUNNER_TEMP/pulse-native-updater.json"'):+ assert command in shell, command+ assert "GH_TOKEN" not in shell and "secrets." not in step+ retention = steps["Retain bounded native updater receipt"]+ assert "if: always()" in retention and "retention-days: 7" in retention+ assert "path: ${{ runner.temp }}/pulse-native-updater.json" in retention+ assert "*.log" not in retention and "pulse-native-before" not in retention+ return shell+++class NativeContracts(unittest.TestCase):+ def test_definition_boundary_is_exact_and_rejects_drift(self):+ source = (ROOT / "install.sh").read_text()+ definitions = native.installer_definitions(source)+ self.assertEqual(definitions, source[:source.index(native.PARSE_MARKER)])+ for changed in (source.replace(native.PARSE_MARKER, "# moved\n"),+ source.replace(native.PARSE_MARKER, native.PARSE_MARKER * 2),+ source.replace(native.PARSE_MARKER, native.PARSE_MARKER + "echo different entry\n")):+ with self.assertRaises(ValueError): native.installer_definitions(changed)++ def test_host_admission_requires_fresh_public_vm_not_worker_or_container(self):+ env = dict(GITHUB_ACTIONS="true", RUNNER_ENVIRONMENT="github-hosted", GITHUB_REPOSITORY="rcourtman/Pulse")+ with patch.object(Path, "exists", return_value=False):+ native.hosted_admission(env, 0, "a" * 40)+ for changed, uid, sha in ((env, 1000, "a" * 40), ({}, 0, "a" * 40),+ ({**env, "RUNNER_ENVIRONMENT": "self-hosted"}, 0, "a" * 40),+ ({**env, "GITHUB_REPOSITORY": "other/repo"}, 0, "a" * 40),+ (env, 0, "main")):+ with self.assertRaises(ValueError): native.hosted_admission(changed, uid, sha)+ with patch.object(Path, "exists", return_value=True):+ with self.assertRaises(ValueError): native.hosted_admission(env, 0, "a" * 40)++ def test_transport_and_private_allocator_have_no_online_fallback(self):+ with tempfile.TemporaryDirectory() as directory:+ root = Path(directory); (root / "tmp").mkdir()+ script = root / "curl"; script.write_text(native.TRANSPORT); script.chmod(0o755)+ (root / "installer").write_text("signed fixture\n")+ (root / "installer.sshsig").write_text("signature sidecar\n")+ env = {**os.environ, "PULSE_NATIVE_ROOT": str(root)}+ for tag in ("latest/download", "download/v0.0.1", "download/v0.0.2"):+ for suffix in ("", ".sshsig"):+ result = subprocess.run([str(script), "-fsSL", f"https://github.com/rcourtman/Pulse/releases/{tag}/install.sh{suffix}"], env=env, capture_output=True)+ self.assertEqual(result.returncode, 0)+ self.assertEqual(result.stdout, (root / ("installer" + suffix)).read_bytes())+ for url in ("https://example.com/install.sh", "https://github.com/rcourtman/Pulse/releases/download/v6.5.0/install.sh"):+ result = subprocess.run([str(script), "-fsSL", url, "-o", str(root / "unexpected")], env=env, capture_output=True)+ self.assertEqual(result.returncode, 97)+ self.assertFalse((root / "unexpected").exists())+ allocator = root / "mktemp"; allocator.write_text(native.TRANSPORT); allocator.chmod(0o755)+ result = subprocess.run([str(allocator), "-d", "/tmp/pulse-backup.XXXXXX"], env=env, capture_output=True, check=True)+ self.assertEqual(Path(result.stdout.decode().strip()).parent, root / "tmp")++ def test_real_verifier_accepts_fixture_and_refuses_tampering(self):+ with tempfile.TemporaryDirectory() as directory:+ root = Path(directory)+ lab = native.NativeLab(root, ROOT, root / "a", root / "b", "a" * 40)+ (root / "signer").mkdir()+ lab.run("ssh-keygen", "-q", "-t", "ed25519", "-N", "", "-f", str(root / "signer/key"))+ installer = root / "installer"; installer.write_text(native.INSTALLER)+ lab.sign(installer)+ shell = ('source "$PULSE_NATIVE_UPDATER"; '+ 'PINNED_RELEASE_SSH_PUBLIC_KEY=$(cat "$PULSE_NATIVE_ROOT/signer/key.pub"); '+ 'verify_release_signature "$PULSE_NATIVE_ROOT/installer" "$PULSE_NATIVE_ROOT/installer.sshsig" fixture')+ self.assertEqual(lab.shell(shell).returncode, 0)+ installer.write_text(native.INSTALLER + "# tampered\n")+ result = subprocess.run(["/bin/bash", "-c", shell], env=lab.env, capture_output=True)+ self.assertEqual(result.returncode, 1)+ self.assertIn(b"Cryptographic signature verification failed", result.stdout)++ def make_phase(self, directory, mutation=""):+ root = Path(directory)+ for name in ("install/bin", "data", "tmp"):+ (root / name).mkdir(parents=True)+ before, after = root / "a", root / "b"+ before.write_bytes(b"baseline executable\n"); after.write_bytes(b"changed executable\n")+ (root / "install/bin/pulse").write_bytes(after.read_bytes())+ (root / "install/VERSION").write_text("v0.0.2\n")+ (root / "data/sentinel").write_text("original\n")+ lab = native.NativeLab(root, ROOT, before, after, "a" * 40)+ lab.baseline_data = {"sentinel": native.digest(root / "data/sentinel")}+ lab.baseline_mode = 0o644+ lab.receipt["binaries"] = {"v0.0.1": native.digest(before), "v0.0.2": native.digest(after)}+ states = [{"LoadState": "loaded", "ActiveState": "inactive", "UnitFileState": "disabled", "MainPID": "0"}] * 2+ if mutation == "mask": states[1] = {**states[1], "LoadState": "masked", "UnitFileState": "masked"}+ if mutation == "activity": states[1] = {**states[1], "ActiveState": "active", "MainPID": "123"}+ lab.state = unittest.mock.Mock(side_effect=states)+ def run(*args, **kwargs):+ if mutation == "persistent": (root / "data/sentinel").write_text("changed\n")+ if mutation == "binary": (root / "install/bin/pulse").write_text("different\n")+ if mutation == "version": (root / "install/VERSION").write_text("v0.0.1\n")+ if mutation == "backup": (root / "tmp/pulse-backup.extra").mkdir()+ if mutation == "staging": (root / "install/bin/pulse.rollback.leak").write_text("leaked\n")+ return subprocess.CompletedProcess(args, 0, b"fixture command, not native execution\n")+ lab.run = run+ return lab++ def test_phase_cannot_accept_wrong_bytes_state_mask_persistence_or_backup(self):+ for mutation in ("", "binary", "version", "mask", "activity", "persistent", "backup", "staging"):+ with self.subTest(mutation=mutation), tempfile.TemporaryDirectory() as directory:+ lab = self.make_phase(directory, mutation)+ if mutation:+ with self.assertRaises(RuntimeError): lab.phase("forward", "v0.0.2", active="inactive")+ self.assertEqual(lab.receipt["phases"], [])+ else:+ lab.phase("forward", "v0.0.2", active="inactive")+ self.assertTrue(lab.receipt["phases"][0]["passed"])++ def test_cleanup_never_deletes_preexisting_unowned_scratch(self):+ with tempfile.TemporaryDirectory() as directory:+ root = Path(directory); marker = root / "prior"; marker.write_text("not ours\n")+ lab = native.NativeLab(root, ROOT, root / "a", root / "b", "a" * 40)+ with patch.object(lab, "run", side_effect=AssertionError("unowned manager operation")):+ lab.cleanup()+ self.assertTrue(marker.exists())+ self.assertFalse(lab.receipt["cleanup"]["owned_fixture_removed"])++ def run_recipe(self, failure="", mismatch=False):+ with tempfile.TemporaryDirectory() as directory:+ root = Path(directory); tools = root / "tools"; tools.mkdir()+ for name in ("frontend-modern/dist", "internal/api/frontend-modern/dist"):+ (root / name).mkdir(parents=True); (root / name / "index.html").write_text("real-embed fixture\n")+ trace = root / "trace"+ fake = '''#!/usr/bin/python3+import json, os, pathlib, sys+args = sys.argv[1:]; name = pathlib.Path(sys.argv[0]).name+with pathlib.Path(os.environ["TRACE"]).open("a") as out: out.write(json.dumps([name, *args])+"\\n")+if name == "git": print("b"*40 if os.environ["MISMATCH"] == "true" else "a"*40)+if name == "go":+ if args[args.index("-o")+1].endswith(os.environ["FAIL_BUILD"]) and os.environ["FAIL_BUILD"]: sys.exit(43)+ pathlib.Path(args[args.index("-o")+1]).write_text("compiler double, not a product build\\n")+if name == "sudo":+ if os.environ["FAIL_BUILD"] == "native": sys.exit(45)+ pathlib.Path(os.environ["RUNNER_TEMP"]+"/pulse-native-updater.json").write_text("{}\\n")+'''+ for name in ("git", "go", "sudo"):+ path = tools / name; path.write_text(fake); path.chmod(0o755)+ env = {**os.environ, "PATH": str(tools) + ":" + os.environ["PATH"], "TRACE": str(trace),+ "RUNNER_TEMP": str(root), "EXPECTED_SOURCE_SHA": "a" * 40,+ "FAIL_BUILD": failure, "MISMATCH": "true" if mismatch else "false"}+ result = subprocess.run(["/bin/bash", "-c", recipe(WORKFLOW.read_text())], env=env, cwd=root, capture_output=True)+ return result, [json.loads(line) for line in trace.read_text().splitlines()], (root / "pulse-native-updater.json").exists()++ def test_exact_workflow_recipe_is_scrubbed_and_propagates_all_failures(self):+ result, calls, receipt = self.run_recipe()+ self.assertEqual(result.returncode, 0, result.stderr)+ self.assertEqual([call[0] for call in calls], ["git", "go", "go", "sudo"])+ self.assertTrue(receipt)+ self.assertEqual(calls[-1][1:3], ["env", "-i"])+ for failure, code, count in (("before", 43, 2), ("after", 43, 3), ("native", 45, 4)):+ result, calls, receipt = self.run_recipe(failure)+ self.assertEqual(result.returncode, code)+ self.assertEqual(len(calls), count)+ self.assertFalse(receipt)+ result, calls, receipt = self.run_recipe(mismatch=True)+ self.assertEqual(result.returncode, 1)+ self.assertEqual([call[0] for call in calls], ["git"])+ self.assertFalse(receipt)++ def test_connected_signed_archive_and_recovery_controls_without_host_manager(self):+ # Same orchestration and production shell functions, real signatures and+ # ELF staging. Only this test replaces the manager, account/chown and+ # unit output path. It is not native or real-Pulse runtime acceptance.+ manager = r'''#!/usr/bin/python3+import os, pathlib, sys+root = pathlib.Path(os.environ["PULSE_NATIVE_ROOT"])+unit = root / "pulse-source-updater.service"+args = sys.argv[1:]+if any(".service" in a and a != "pulse-source-updater.service" for a in args): sys.exit(97)+load = "masked" if unit.is_symlink() else ("loaded" if unit.exists() else "not-found")+statefile = root / "manager-state"+active = statefile.read_text().strip() if statefile.exists() and load != "not-found" else "inactive"+values = dict(LoadState=load, ActiveState=active, UnitFileState="masked" if load == "masked" else "disabled", MainPID="123" if active == "active" else "0")+if args[0] == "show":+ properties = next(a.split("=", 1)[1] for a in args if a.startswith("--property=")).split(",")+ for p in properties: print(values[p] if "--value" in args else p+"="+values[p])+elif args[0] in ("start", "stop"):+ if args[0] == "start" and load != "loaded": sys.exit(1)+ statefile.write_text("active" if args[0] == "start" else "inactive")+ if args[0] == "start" and not (root / "data/.encryption.key").exists():+ (root / "data/.encryption.key").write_text("synthetic persistence marker, not a key\n")+elif args[0] == "mask": unit.symlink_to("/dev/null")+elif args[0] == "unmask": unit.unlink()+elif args[0] not in ("daemon-reload", "reset-failed"): sys.exit(97)+'''+ class FixtureLab(native.NativeLab):+ def assert_runtime(self, tag, active):+ if native.digest(self.root / "install/bin/pulse") != self.receipt["binaries"][tag]:+ raise RuntimeError("fixture executable mismatch")+ if (self.root / "install/VERSION").read_text() != tag + "\n":+ raise RuntimeError("fixture version mismatch")+ state = self.state()+ if state["ActiveState"] != active: raise RuntimeError("fixture activity mismatch")+ return state+ with tempfile.TemporaryDirectory() as directory:+ base = Path(directory); tools = base / "tools"; tools.mkdir()+ path = tools / "systemctl"; path.write_text(manager); path.chmod(0o755)+ account = r'''#!/usr/bin/python3+import os, pathlib, sys+marker = pathlib.Path(os.environ["PULSE_NATIVE_ROOT"]) / "fixture-user"+if pathlib.Path(sys.argv[0]).name == "id":+ if sys.argv[1:] != ["-u", "pulse"]: sys.exit(97)+ if not marker.exists(): sys.exit(1)+ print("1234")+else:+ if sys.argv[1:] != ["pulse"]: sys.exit(97)+ marker.unlink()+'''+ for name in ("id", "userdel"):+ tool = tools / name; tool.write_text(account); tool.chmod(0o755)+ program = base / "fixture.go"+ program.write_text('package main\nimport "fmt"\nvar Version string\nfunc main(){fmt.Println("Pulse " + Version)}\n')+ binaries = []+ for tag in native.TAGS:+ binary = base / tag+ subprocess.run(["go", "build", "-o", str(binary), "-ldflags=-X main.Version=" + tag, str(program)],+ check=True, capture_output=True, timeout=60)+ binaries.append(binary)+ lab = FixtureLab(base / "lab", ROOT, *binaries, "a" * 40)+ lab.unit = lab.root / "pulse-source-updater.service"+ lab.env["PATH"] = str(lab.root / "tools") + ":" + str(tools) + ":/usr/sbin:/usr/bin:/sbin:/bin"+ original_write = Path.write_text+ def relocate(path, data, *args, **kwargs):+ if path.name == "installer-definitions":+ before = "cat > /etc/systemd/system/$SERVICE_NAME.service << EOF"+ self.assertEqual(data.count(before), 1)+ data = data.replace(before, 'cat > "$PULSE_NATIVE_ROOT/pulse-source-updater.service" << EOF')+ data += '\ncreate_user() { touch "$PULSE_NATIVE_ROOT/fixture-user"; }\nchown() { :; }\n'+ return original_write(path, data, *args, **kwargs)+ try:+ with patch.object(Path, "write_text", new=relocate), patch.object(native.time, "sleep"):+ lab.prepare()+ lab.exercise()+ self.assertEqual([p["name"] for p in lab.receipt["phases"]],+ ["forward", "manual-reverse", "failed-active", "failed-inactive",+ "failed-masked", "bad-signature", "incomplete-restore"])+ self.assertTrue(lab.receipt["phases"][-1]["backup_retained"])+ finally:+ lab.cleanup()+ self.assertTrue(lab.receipt["cleanup"]["complete"])+ self.assertFalse(lab.root.exists())++ def test_workflow_cannot_hide_or_rebind_native_step(self):+ source = WORKFLOW.read_text()+ for changed in (source.replace("- name: " + STEP + "\n", "- name: " + STEP + "\n continue-on-error: true\n"),+ source.replace("sudo env -i", "sudo env"),+ source.replace("main.Version=v0.0.2", "main.Version=v0.0.1"),+ source.replace("EXPECTED_SOURCE_SHA: ${{ github.sha }}", "EXPECTED_SOURCE_SHA: main"),+ source.replace("runs-on: ubuntu-24.04", "runs-on: self-hosted")):+ with self.assertRaises(AssertionError): recipe(changed)+++if __name__ == "__main__":+ unittest.main()