Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Check agent removal against native hosted systemd
pulse-triage[bot]27f9c1de5e48
diff --git a/.github/workflows/unified-agent-native.yml b/.github/workflows/unified-agent-native.ymlindex 4c039436c1..ee3ce15c3e 100644--- a/.github/workflows/unified-agent-native.yml+++ b/.github/workflows/unified-agent-native.yml@@ -31,6 +31,7 @@ on:- 'scripts/pulse-auto-update.sh'- 'scripts/tests/test_auto_update_service_state.py'- 'scripts/tests/test_auto_update_systemd.py'+ - 'scripts/tests/test_agent_removal_systemd.py'- 'scripts/tests/test_server_service_discovery.py'- 'scripts/installtests/**'- 'go.mod'@@ -61,6 +62,7 @@ on:- 'scripts/pulse-auto-update.sh'- 'scripts/tests/test_auto_update_service_state.py'- 'scripts/tests/test_auto_update_systemd.py'+ - 'scripts/tests/test_agent_removal_systemd.py'- 'scripts/tests/test_server_service_discovery.py'- 'scripts/installtests/**'- 'go.mod'@@ -104,6 +106,35 @@ jobs:with:persist-credentials: false+ # These are real manager/deletion checks, not the installer doubles. Run+ # before compiler/frontend acquisition so unrelated preparation cannot+ # hide Linux removal safety. Only the new public hosted runner is mutated.+ - name: Check agent removal on native systemd+ if: runner.os == 'Linux'+ timeout-minutes: 5+ env:+ EXPECTED_SOURCE_SHA: ${{ github.sha }}+ shell: bash+ run: |+ set -euo pipefail+ test "$(git rev-parse HEAD)" = "$EXPECTED_SOURCE_SHA"+ git diff --exit-code HEAD+ python3 -m unittest discover -s scripts/tests -p test_agent_removal_systemd.py -v+ sudo env -i PATH=/usr/sbin:/usr/bin:/sbin:/bin LC_ALL=C \+ GITHUB_ACTIONS=true RUNNER_ENVIRONMENT=github-hosted GITHUB_REPOSITORY=rcourtman/Pulse \+ EXPECTED_SOURCE_SHA="$EXPECTED_SOURCE_SHA" RUNNER_TEMP="$RUNNER_TEMP" \+ python3 scripts/installtests/agent_removal_systemd.py \+ --receipt "$RUNNER_TEMP/pulse-agent-removal.json"++ - name: Retain bounded native agent removal receipt+ if: always() && runner.os == 'Linux'+ uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1+ with:+ name: native-agent-removal-${{ matrix.runner }}-${{ github.run_id }}-${{ github.run_attempt }}+ path: ${{ runner.temp }}/pulse-agent-removal.json+ if-no-files-found: warn+ retention-days: 7+- name: Set up Gouses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0with:diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 341147242a..cabef9e7e3 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -44,6 +44,27 @@ saved-state removal controls also require refusal before remote or local mutatioThese are exact-source controls, not native systemd installation, TLS, publishedavailability or resolution of Linux/PowerShell #1099.+`unified-agent-native.yml` additionally requires `agent_removal_systemd.py` on+both new public hosted Linux runners, before compiler/frontend preparation. The+ordinary runner binds a clean exact checkout; the root lab receives no ambient+credentials and refuses workers, self-hosted/container runs, existing Pulse+state and pre-existing fixture units. It extracts the six production removal+functions unchanged and binds their bytes in a bounded terminal receipt.++Its fixed synthetic units exercise active collector teardown, real helper socket+activation, runner deletion only after fixture revocation, real refused stops,+unavailable-user-manager read refusal, stopped/retained failed-revocation state,+an earlier runner reactivated by a later unit's real ExecStop, and absent/repeated+removal. Independent systemd/property and prior-PID observations must agree with+the intended phase; recovery file hashes, no repeated synthetic revocation and+owned unit/process/fixture cleanup are retained. Cleanup failure fails the job.+The absent-user-manager adapter is an injected read-failure condition, not a+production system-bus diagnosis. Unit names, credential files and services are+synthetic: this does not establish signed agent installation, real TLS/exact-token+revocation, published availability, Windows acceptance or reporter recovery.+`TestNativeAgentRemovalSystemdRecipeRuntime` exercises the lab and mandatory CI+recipe with closed manager models; those portable controls are not a native pass.+### Unix removal retains credentials when connection recovery is incompleteAfter existing installed-state recovery, the unified Unix installer checks fordiff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.mdindex 0f5cf3466c..0877bcf585 100644--- a/docs/release-control/v6/internal/subsystems/deployment-installability.md+++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md@@ -1,5 +1,6 @@# Deployment Installability Contract+## Contract Metadata```json@@ -15,6 +16,30 @@## Purpose+### Linux agent removal native boundary++The existing Unified Agent Native Verification job now runs the unchanged Unix+removal functions against actual systemd on its fresh public hosted Linux x64+and ARM64 runners, before acquiring compiler/frontend dependencies. Its fixed+private fixtures cover refused stop/read/revocation, socket/batch activation,+observed quiescence, recovery-file retention, confirmed deletion and repeated+absence. The exact source, extracted-function digest, per-phase independent unit+and prior-process observations, file-retention verdicts and terminal cleanup are+retained in a seven-day bounded JSON artifact even when the native command fails.+No default-Pulse unit, existing installation, worker/self-hosted host, service+credential or production endpoint is admitted; root execution belongs to that+new hosted VM only. No missing or green-only receipt establishes acceptance.++`TestNativeAgentRemovalSystemdRecipeRuntime` requires discriminating portable+lab/recipe controls, including rejected worker admission, preserved pre-existing+units, missing/duplicate observations, successful commands without actual active+identity or quiescence, omitted final batch readback and failed cleanup. The+native result still requires the containing job's terminal artifact and logs.+Runner revocation is deliberately a closed fixture, not authenticated transport+proof; signed installed-agent/Windows/service-manager/customer recovery remain+separate. See the agent-lifecycle contract for the exact fixture boundaries.++### Agent systemd teardown refuses unobserved stops and activation sources`scripts/install.sh` must not discard unit files, helper credentials or runnerdiff --git a/scripts/installtests/agent_removal_systemd.py b/scripts/installtests/agent_removal_systemd.pynew file mode 100644index 0000000000..091d001de1--- /dev/null+++ b/scripts/installtests/agent_removal_systemd.py@@ -0,0 +1,378 @@+#!/usr/bin/env python3+"""Agent removal boundaries on a fresh public hosted systemd runner.++Use the committed installer functions unchanged, with fixed private unit names+and synthetic recovery files. systemctl, socket activation and deletion are real;+runner credential revocation is a closed fixture, not TLS/server acceptance.+This is not installation of a signed release or Windows/other-manager proof.+Never run this lab on a worker, self-hosted runner or an existing Pulse host.+"""++import argparse+import hashlib+import json+import os+from pathlib import Path+import re+import shlex+import shutil+import signal+import socket+import subprocess+import time++ROOT = Path(__file__).resolve().parents[2]+LAB = Path("/opt/pulse-agent-removal-native")+UNIT_DIR = Path("/etc/systemd/system")+AGENT = "pulse-removal-native-agent"+HELPER = "pulse-removal-native-helper"+RUNNER = "pulse-removal-native-runner"+UNITS = (RUNNER + ".service", HELPER + ".socket", HELPER + ".service", AGENT + ".service")+FUNCTIONS = ("read_agent_removal_unit_state", "confirm_agent_unit_quiescent",+ "quiesce_agent_units_for_removal", "teardown_systemd_agent_service",+ "teardown_privileged_helper_service", "teardown_action_runner_service")+PHASES = ("active-agent", "activated-helper", "confirmed-runner",+ "refused-agent-stop", "refused-helper-stop", "unavailable-manager",+ "refused-runner-revocation", "batch-reactivation", "absent-and-repeated")+++def definitions(source):+ result = []+ for name in FUNCTIONS:+ marker = name + "() {"+ if source.count(marker) != 1:+ raise ValueError("installer function boundary changed")+ body = source.split(marker, 1)[1]+ if "\n}\n" not in body:+ raise ValueError("installer function end missing")+ result.append(marker + body.split("\n}\n", 1)[0] + "\n}\n")+ return "\n".join(result)+++def admission(env, uid, source):+ if (uid != 0 or env.get("GITHUB_ACTIONS") != "true"+ or env.get("RUNNER_ENVIRONMENT") != "github-hosted"+ or env.get("GITHUB_REPOSITORY") != "rcourtman/Pulse"+ or not re.fullmatch(r"[0-9a-f]{40}", source)+ or any(Path(p).exists() for p in+ ("/.dockerenv", "/opt/pulse", "/etc/pulse", "/etc/pulse-agent",+ "/etc/pulse-agent-runner", "/var/lib/pulse-agent", LAB))):+ raise ValueError("agent removal proof requires a fresh public hosted VM")+++def state(output):+ values = {}+ for line in output.decode("ascii").splitlines():+ key, separator, value = line.partition("=")+ if not separator or key not in {"LoadState", "ActiveState", "UnitFileState", "MainPID"} or key in values:+ raise ValueError("independent unit observation incomplete")+ values[key] = value+ if not {"LoadState", "ActiveState", "UnitFileState"} <= values.keys():+ raise ValueError("independent unit observation missing")+ return values+++def pid_alive(pid):+ if not re.fullmatch(r"[1-9][0-9]*", pid):+ raise ValueError("native process identity unavailable")+ try:+ os.kill(int(pid), 0)+ except ProcessLookupError:+ return False+ # Permission/read errors stay failures; never manufacture process absence.+ return True+++class Lab:+ def __init__(self, source, installer):+ self.source = source+ self.definitions = definitions(installer)+ self.owned_units = set()+ self.owned_pids = set()+ self.owned_root = False+ self.env = {"PATH": "/usr/sbin:/usr/bin:/sbin:/bin", "LC_ALL": "C", "HOME": str(LAB)}+ self.receipt = {"schema_version": 1, "source_sha": source,+ "scope": "current-source-native-systemd-removal-boundary",+ "installer_sha256": hashlib.sha256(installer.encode()).hexdigest(),+ "definitions_sha256": hashlib.sha256(self.definitions.encode()).hexdigest(),+ "credential_transport": "closed-synthetic-revocation-only",+ "phases": [], "cleanup": {"complete": False}}++ def run(self, *args, ok=(0,), timeout=20):+ process = subprocess.Popen(args, env=self.env, stdin=subprocess.DEVNULL,+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT,+ start_new_session=True)+ try:+ output, _ = process.communicate(timeout=timeout)+ except subprocess.TimeoutExpired:+ os.killpg(process.pid, signal.SIGTERM)+ try:+ process.communicate(timeout=2)+ except subprocess.TimeoutExpired:+ os.killpg(process.pid, signal.SIGKILL)+ process.communicate(timeout=2)+ self.receipt["command_failure"] = {"deadline_exceeded": True,+ "observed_exit": process.returncode}+ raise RuntimeError("owned command deadline exceeded") from None+ result = subprocess.CompletedProcess(args, process.returncode, output)+ if result.returncode not in ok:+ self.receipt["command_failure"] = {"command": Path(args[0]).name,+ "observed_exit": result.returncode,+ "output_bytes": len(output),+ "output_sha256": hashlib.sha256(output).hexdigest()}+ raise RuntimeError("native command failed")+ return result++ def observe(self):+ observed = {unit: state(self.run("systemctl", "show", unit,+ "--property=LoadState,ActiveState,UnitFileState,MainPID").stdout)+ for unit in UNITS}+ for unit in self.owned_units:+ pid = observed[unit].get("MainPID", "0")+ if pid != "0":+ if not re.fullmatch(r"[1-9][0-9]*", pid):+ raise ValueError("native process identity unavailable")+ self.owned_pids.add(pid)+ return observed++ def prepare(self):+ # Reject loaded or vendor-provided units as well as on-disk collisions.+ if any(Path(UNIT_DIR / unit).exists() or Path(UNIT_DIR / unit).is_symlink() for unit in UNITS):+ raise ValueError("native unit name already owned")+ if any(v["LoadState"] != "not-found" for v in self.observe().values()):+ raise ValueError("native unit already loaded")+ LAB.mkdir(mode=0o700)+ self.owned_root = True++ def install_unit(self, unit, text):+ if unit not in UNITS:+ raise ValueError("unit outside fixed lab")+ with (UNIT_DIR / unit).open("x") as stream:+ self.owned_units.add(unit)+ stream.write(text)+ self.run("systemctl", "daemon-reload")+ self.run("systemctl", "enable", "--now", unit)++ def service(self, unit, extra=""):+ self.install_unit(unit, "[Unit]\nDescription=Pulse owned removal fixture\n" ++ ("RefuseManualStop=yes\n" if extra == "refuse" else "") ++ "[Service]\nType=simple\nExecStart=/usr/bin/sleep infinity\n" ++ ("ExecStop=/usr/bin/systemctl start " + RUNNER + ".service\n"+ if extra == "reactivate" else "") ++ "[Install]\nWantedBy=multi-user.target\n")++ def helper(self, directory, refuse=False):+ # The pending connection genuinely starts the paired service. The+ # synthetic process does not implement the privileged-helper protocol.+ with (UNIT_DIR / (HELPER + ".service")).open("x") as stream:+ self.owned_units.add(HELPER + ".service")+ stream.write("[Service]\nExecStart=/usr/bin/sleep infinity\n")+ self.install_unit(HELPER + ".socket",+ "[Unit]\nDescription=Pulse owned socket fixture\n" ++ ("RefuseManualStop=yes\n" if refuse else "") ++ "[Socket]\nListenStream=" + str(directory / "helper.sock") ++ "\n[Install]\nWantedBy=sockets.target\n")+ with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as connection:+ connection.settimeout(2)+ connection.connect(str(directory / "helper.sock"))+ deadline = time.monotonic() + 5+ while self.observe()[HELPER + ".service"]["ActiveState"] != "active":+ if time.monotonic() >= deadline:+ raise RuntimeError("real helper socket activation was not observed")+ time.sleep(0.05)++ def shell(self, directory, command, *, revoke=True, unavailable=False):+ values = {"AGENT_NAME": AGENT, "PRIVILEGED_HELPER_NAME": HELPER, "ACTION_RUNNER_NAME": RUNNER,+ "PRIVILEGED_HELPER_SOCKET_UNIT": UNIT_DIR / (HELPER + ".socket"),+ "PRIVILEGED_HELPER_SERVICE_UNIT": UNIT_DIR / (HELPER + ".service"),+ "PRIVILEGED_HELPER_SOCKET_PATH": directory / "helper.sock",+ "PRIVILEGED_HELPER_CREDENTIAL_DIR": directory / "helper-credentials",+ "ACTION_RUNNER_SERVICE_UNIT": UNIT_DIR / (RUNNER + ".service"),+ "ACTION_RUNNER_BINARY_PATH": directory / "runner",+ "ACTION_RUNNER_CONFIG_DIR": directory / "runner-config",+ "ACTION_RUNNER_STATE_DIR": directory / "runner-state"}+ prefix = "set -euo pipefail\nEXIT_GENERAL=1\n" + "\n".join(+ key + "=" + shlex.quote(str(value)) for key, value in values.items()) + "\n"+ prefix += 'fail() { exit "${2:-1}"; }\nlog_info() { :; }\nlog_error() { :; }\n'+ prefix += "revoke_action_runner_credential() { printf 'attempted\\n' >> " + shlex.quote(str(directory / "revoke")) + "; " + ("return 0" if revoke else "return 1") + "; }\n"+ if unavailable:+ # This adapter directs the real systemctl to a deliberately absent+ # user manager. It proves read-failure retention, not a system-bus+ # incident. No failed system-manager command is replayed.+ # timeout execs its target; a shell function would not intercept it.+ tools = directory / "unavailable-manager-tools"+ tools.mkdir()+ manager = shutil.which("systemctl", path=self.env["PATH"])+ if manager is None:+ raise ValueError("native manager executable unavailable")+ adapter = tools / "systemctl"+ adapter.write_text("#!/bin/sh\nexec env XDG_RUNTIME_DIR=" ++ shlex.quote(str(directory / "no-manager")) ++ " DBUS_SESSION_BUS_ADDRESS=" + shlex.quote("unix:path=" + str(directory / "no-bus")) ++ " " + shlex.quote(manager) + ' --user "$@"\n')+ adapter.chmod(0o700)+ prefix += "PATH=" + shlex.quote(str(tools)) + ':"$PATH"\n'+ return self.run("/bin/bash", "-c", prefix + self.definitions + command, ok=(0, 1), timeout=40)++ def snapshot(self, directory):+ paths = [directory / "runner"] + [UNIT_DIR / unit for unit in self.owned_units]+ for name in ("helper-credentials", "runner-config", "runner-state"):+ paths.extend((directory / name).rglob("*"))+ return {str(path): hashlib.sha256(path.read_bytes()).hexdigest()+ for path in paths if path.is_file() and path.name != "revoke"}++ def exercise(self, phase):+ directory = LAB / phase+ directory.mkdir(mode=0o700)+ for name in ("helper-credentials", "runner-config", "runner-state"):+ (directory / name).mkdir(mode=0o700)+ (directory / name / "token").write_text("synthetic-recovery-material\n")+ (directory / "runner").write_text("synthetic-runner-not-executable\n")+ if phase == "batch-reactivation":+ self.service(RUNNER + ".service")+ self.helper(directory)+ self.service(AGENT + ".service", "reactivate")+ elif "helper" in phase:+ self.helper(directory, phase == "refused-helper-stop")+ elif "runner" in phase:+ self.service(RUNNER + ".service")+ elif phase != "absent-and-repeated":+ self.service(AGENT + ".service", "refuse" if phase == "refused-agent-stop" else "")+ before, files_before = self.observe(), self.snapshot(directory)+ for unit, observed in before.items():+ if unit in self.owned_units:+ if (observed["LoadState"] != "loaded" or observed["ActiveState"] != "active"+ or observed["UnitFileState"] not in {"enabled", "static"}+ or (unit.endswith(".service") and not pid_alive(observed.get("MainPID", "0")))):+ raise RuntimeError("intended active fixture was not observed")+ elif observed["LoadState"] != "not-found" or observed["ActiveState"] != "inactive":+ raise RuntimeError("unowned native unit appeared")+ agent_command = "teardown_systemd_agent_service " + shlex.quote(str(UNIT_DIR / (AGENT + ".service")))+ if phase == "batch-reactivation":+ command = "quiesce_agent_units_for_removal " + " ".join(UNITS) + " || fail refused 1\nprintf mutation > " + shlex.quote(str(directory / "mutation"))+ elif phase == "absent-and-repeated":+ command = (agent_command + "\nteardown_privileged_helper_service\nteardown_action_runner_service\n") * 2+ elif "helper" in phase:+ command = "teardown_privileged_helper_service"+ elif "runner" in phase:+ command = "teardown_action_runner_service"+ else:+ command = agent_command+ result = self.shell(directory, command, revoke=phase != "refused-runner-revocation",+ unavailable=phase == "unavailable-manager")+ after = self.observe()+ refused = phase.startswith("refused-") or phase in {"unavailable-manager", "batch-reactivation"}+ record = {"name": phase, "command_exit": result.returncode, "before": before, "after": after,+ "recovery_files_unchanged": self.snapshot(directory) == files_before,+ "revocation_attempted": (directory / "revoke").exists(),+ "revocation_attempts": len((directory / "revoke").read_text().splitlines()) if (directory / "revoke").exists() else 0,+ "prior_process_alive": {unit: pid_alive(v["MainPID"]) for unit, v in before.items()+ if v.get("MainPID", "0") != "0"}, "passed": False}+ self.receipt["phases"].append(record)+ if result.returncode != (1 if refused else 0):+ raise RuntimeError("native removal verdict differs")+ if refused:+ if not record["recovery_files_unchanged"] or (directory / "mutation").exists():+ raise RuntimeError("unconfirmed removal erased recovery material")+ if phase in {"refused-agent-stop", "unavailable-manager"} and (+ after[AGENT + ".service"]["ActiveState"] != "active" or+ before[AGENT + ".service"]["MainPID"] != after[AGENT + ".service"]["MainPID"] or+ not record["prior_process_alive"][AGENT + ".service"]):+ raise RuntimeError("refused collector changed process identity")+ if phase == "batch-reactivation" and after[RUNNER + ".service"]["ActiveState"] != "active":+ raise RuntimeError("batch reactivation adverse control did not occur")+ if phase == "refused-runner-revocation" and (+ not record["revocation_attempted"] or after[RUNNER + ".service"]["ActiveState"] != "inactive" or+ after[RUNNER + ".service"]["UnitFileState"] != "disabled"):+ raise RuntimeError("revocation refusal lacks stopped retained runner")+ else:+ targets = (UNITS if phase == "absent-and-repeated" else+ (HELPER + ".socket", HELPER + ".service") if "helper" in phase else+ (RUNNER + ".service",) if "runner" in phase else (AGENT + ".service",))+ for unit in targets:+ if (after[unit]["LoadState"] != "not-found" or after[unit]["ActiveState"] != "inactive"+ or after[unit].get("MainPID", "0") != "0" or (UNIT_DIR / unit).exists()):+ raise RuntimeError("native successful removal left unit or process")+ if record["prior_process_alive"].get(unit, False):+ raise RuntimeError("native successful removal left its prior process alive")+ if "helper" in phase and any((directory / n).exists() for n in ("helper.sock", "helper-credentials")):+ raise RuntimeError("helper recovery material not removed after confirmation")+ if "runner" in phase and (not record["revocation_attempted"] or+ any((directory / n).exists() for n in ("runner", "runner-config", "runner-state"))):+ raise RuntimeError("confirmed runner removal incomplete")+ if phase == "absent-and-repeated" and record["revocation_attempts"] != 1:+ raise RuntimeError("repeated absent removal repeated synthetic revocation")+ record["passed"] = True++ def cleanup_units(self):+ # Withdraw only our own refusal/ExecStop fixtures before stopping them.+ # Never repair, stop or unlink a pre-existing unit.+ for unit in self.owned_units:+ path = UNIT_DIR / unit+ if path.exists():+ text = path.read_text()+ text = re.sub(r"(?m)^(RefuseManualStop|ExecStop)=.*\n", "", text)+ path.write_text(text)+ if self.owned_units:+ self.run("systemctl", "daemon-reload")+ for unit in UNITS:+ if unit in self.owned_units:+ self.run("systemctl", "disable", unit, ok=(0, 1))+ self.run("systemctl", "stop", unit, ok=(0, 5))+ (UNIT_DIR / unit).unlink(missing_ok=True)+ if self.owned_units:+ self.run("systemctl", "daemon-reload")+ self.run("systemctl", "reset-failed", *sorted(self.owned_units), ok=(0, 1))+ if any(v["LoadState"] != "not-found" or v["ActiveState"] != "inactive" or v.get("MainPID", "0") != "0"+ for v in self.observe().values()):+ raise RuntimeError("owned native unit cleanup incomplete")+ if any(pid_alive(pid) for pid in self.owned_pids):+ raise RuntimeError("owned native process cleanup incomplete")+ self.owned_units.clear()+ self.owned_pids.clear()++ def cleanup(self):+ self.cleanup_units()+ if self.owned_root:+ shutil.rmtree(LAB)+ self.owned_root = False+ self.receipt["cleanup"] = {"complete": True, "owned_units_absent": True,+ "owned_fixture_absent": not LAB.exists()}+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--receipt", required=True)+ args = parser.parse_args()+ source = os.environ.get("EXPECTED_SOURCE_SHA", "")+ admission(os.environ, os.geteuid(), source)+ receipt = Path(args.receipt).resolve()+ if receipt.parent != Path(os.environ["RUNNER_TEMP"]).resolve() or receipt.exists():+ raise ValueError("receipt must be a new direct runner-temp file")+ # The ordinary runner checks the exact clean checkout before sudo. Do not+ # change root's Git ownership/configuration to read a runner-owned checkout.+ # Bind the actual installer and extracted bytes in the terminal receipt too.+ lab = Lab(source, (ROOT / "scripts/install.sh").read_text())+ failed = False+ try:+ lab.prepare()+ for phase in PHASES:+ lab.exercise(phase)+ lab.cleanup_units()+ except (OSError, ValueError, RuntimeError, subprocess.SubprocessError) as error:+ failed = True+ lab.receipt["failure"] = {"type": type(error).__name__}+ finally:+ try:+ lab.cleanup()+ except (OSError, ValueError, RuntimeError, subprocess.SubprocessError) as error:+ failed = True+ lab.receipt["cleanup"] = {"complete": False, "error_type": type(error).__name__}+ receipt.write_text(json.dumps(lab.receipt, indent=2) + "\n")+ if failed or len(lab.receipt["phases"]) != len(PHASES):+ raise SystemExit("Native agent removal proof incomplete; inspect the bounded receipt.")+ print("NATIVE_AGENT_REMOVAL_COMPLETE=" + source)+++if __name__ == "__main__":+ main()diff --git a/scripts/installtests/install_ps1_test.go b/scripts/installtests/install_ps1_test.goindex c39bb3490f..da35216005 100644--- a/scripts/installtests/install_ps1_test.go+++ b/scripts/installtests/install_ps1_test.go@@ -611,6 +611,47 @@ func TestWindowsAgentLifecycleWorkflowChecksEveryNativeExit(t *testing.T) {}}+func TestWindowsAgentLifecycleKeepsLinuxRemovalProofIsolated(t *testing.T) {+ steps := nativeWindowsExitSteps(t)+ validate := func(steps []nativeWindowsExitStep) error {+ wanted := map[string]string{+ "Check agent removal on native systemd": "runner.os == 'Linux'",+ "Retain bounded native agent removal receipt": "always() && runner.os == 'Linux'",+ }+ for _, step := range steps {+ condition, found := wanted[step.Name]+ if !found {+ continue+ }+ if step.If != condition || step.ContinueOnError {+ return fmt.Errorf("Linux removal must not run on or replace Windows lifecycle: %s", step.Name)+ }+ delete(wanted, step.Name)+ }+ if len(wanted) != 0 {+ return fmt.Errorf("missing isolated Linux removal proof: %v", wanted)+ }+ _, err := nativeWindowsExitGuards(steps)+ return err+ }+ if err := validate(steps); err != nil {+ t.Fatal(err)+ }+ for _, name := range []string{"Check agent removal on native systemd", "Retain bounded native agent removal receipt"} {+ for _, condition := range []string{"", "true", "${{ !matrix.unix }}"} {+ changed := append([]nativeWindowsExitStep(nil), steps...)+ for i := range changed {+ if changed[i].Name == name {+ changed[i].If = condition+ }+ }+ if err := validate(changed); err == nil {+ t.Fatalf("admitted unbounded Linux proof on Windows: %s / %q", name, condition)+ }+ }+ }+}+func TestWindowsAgentLifecycleHarnessChecksEveryNativeExit(t *testing.T) {content, err := os.ReadFile(repoFile("scripts", "installtests", "windows_agent_lifecycle.ps1"))if err != nil {diff --git a/scripts/installtests/native_agent_removal_test.go b/scripts/installtests/native_agent_removal_test.gonew file mode 100644index 0000000000..0bd15aa8da--- /dev/null+++ b/scripts/installtests/native_agent_removal_test.go@@ -0,0 +1,39 @@+package installtests++import (+ "context"+ "os/exec"+ "runtime"+ "strings"+ "testing"+ "time"+)++func TestNativeAgentRemovalSystemdRecipeRuntime(t *testing.T) {+ if runtime.GOOS != "linux" {+ t.Skip("Linux-only native systemd lab; Windows lifecycle remains independently required")+ }+ ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)+ defer cancel()+ command := exec.CommandContext(ctx, "python3", repoFile("scripts", "tests", "test_agent_removal_systemd.py"), "-v")+ output, err := command.CombinedOutput()+ if err != nil {+ t.Fatalf("native systemd lab controls: %v\n%s", err, output)+ }+ for _, name := range []string{+ "test_definition_selection_preserves_each_actual_function_and_refuses_drift",+ "test_admission_excludes_worker_self_hosted_nonroot_existing_hosts_and_unbound_source",+ "test_recipe_owns_exact_clean_source_mandatory_linux_execution_and_bounded_retention",+ "test_all_nine_phase_verdicts_and_owned_cleanup_with_closed_manager_model",+ "test_lab_rejects_removal_that_discards_final_batch_readback",+ "test_preexisting_unit_is_never_stopped_or_removed",+ "test_failed_cleanup_stays_failed_and_keeps_ownership_for_reconciliation",+ } {+ if !strings.Contains(string(output), name+" (") {+ t.Fatalf("native systemd lab control %s did not execute:\n%s", name, output)+ }+ }+ if !strings.Contains(string(output), "\nOK\n") || strings.Contains(string(output), "skipped") {+ t.Fatalf("native systemd lab controls incomplete:\n%s", output)+ }+}diff --git a/scripts/tests/test_agent_removal_systemd.py b/scripts/tests/test_agent_removal_systemd.pynew file mode 100644index 0000000000..1c86dfd2e5--- /dev/null+++ b/scripts/tests/test_agent_removal_systemd.py@@ -0,0 +1,246 @@+#!/usr/bin/env python3+"""Portable lab/recipe controls. Manager doubles are NOT native acceptance."""++import importlib.util+import json+import os+from pathlib import Path+import re+import subprocess+import tempfile+import unittest+from unittest.mock import patch++ROOT = Path(__file__).resolve().parents[2]+SPEC = importlib.util.spec_from_file_location("agent_removal_native", ROOT / "scripts/installtests/agent_removal_systemd.py")+native = importlib.util.module_from_spec(SPEC)+SPEC.loader.exec_module(native)+STEP = "Check agent removal on native systemd"+++def recipe(source):+ job = re.search(r"(?ms)^ native-agent:\n(.*?)(?=^ [a-z][a-z0-9-]*:|\Z)", source)[1]+ steps = dict(re.findall(r"(?ms)^ - name: ([^\n]+)\n(.*?)(?=^ - name:|\Z)", job))+ step = steps[STEP]+ assert "if: runner.os == 'Linux'" in step+ assert "timeout-minutes: 5" in step+ assert "EXPECTED_SOURCE_SHA: ${{ github.sha }}" in step+ assert not re.search(r"^ (continue-on-error|working-directory):", step, re.M)+ assert list(steps).index(STEP) < list(steps).index("Set up Go")+ assert "ubuntu-24.04\n" in job and "ubuntu-24.04-arm\n" in job+ assert "persist-credentials: false" in steps["Checkout repository"]+ shell = re.search(r"(?ms)^ run: \|\n((?: [^\n]*\n)+)", step)[1]+ shell = "\n".join(line[10:] for line in shell.splitlines()) + "\n"+ for required in ('set -euo pipefail', 'test "$(git rev-parse HEAD)" = "$EXPECTED_SOURCE_SHA"',+ "git diff --exit-code HEAD", "python3 -m unittest discover -s scripts/tests -p test_agent_removal_systemd.py -v",+ "sudo env -i", "GITHUB_ACTIONS=true RUNNER_ENVIRONMENT=github-hosted GITHUB_REPOSITORY=rcourtman/Pulse",+ 'EXPECTED_SOURCE_SHA="$EXPECTED_SOURCE_SHA" RUNNER_TEMP="$RUNNER_TEMP"',+ "python3 scripts/installtests/agent_removal_systemd.py", '--receipt "$RUNNER_TEMP/pulse-agent-removal.json"'):+ assert required in shell, required+ assert shell.index("git diff") < shell.index("sudo env -i")+ assert "secrets." not in step and "GH_TOKEN" not in shell+ retained = steps["Retain bounded native agent removal receipt"]+ assert "if: always() && runner.os == 'Linux'" in retained+ assert "path: ${{ runner.temp }}/pulse-agent-removal.json\n" in retained+ assert "retention-days: 7" in retained+ assert len(re.findall(r"(?m)^ path:", retained)) == 1 and "*.log" not in retained+ for event in ("push", "pull_request"):+ section = re.search(r"(?ms)^ " + event + r":\n(.*?)(?=^ [a-z_]+:|\Z)", source)[1]+ assert "'scripts/tests/test_agent_removal_systemd.py'" in section+ assert "'scripts/installtests/**'" in section and "'scripts/install.sh'" in section+ return shell+++# An explicit file-backed manager model for the lab's verdict and cleanup+# controls only. The native CI command uses /usr/bin/systemctl, not this model.+MANAGER = r'''#!/usr/bin/python3+import json, os, pathlib, sys+units = pathlib.Path(os.environ['MODEL_UNITS'])+database = pathlib.Path(os.environ['MODEL_STATES'])+values = json.loads(database.read_text()) if database.exists() else {}+args = sys.argv[1:]+if args[0] == '--user':+ print('closed fixture: no user manager', file=sys.stderr)+ sys.exit(1)+verb = args.pop(0)+unit = args[-1] if args else ''+def observed(name):+ file = units / name+ value = values.get(name, {'ActiveState':'inactive', 'UnitFileState':'disabled', 'MainPID':'0'})+ return dict(value, LoadState='loaded' if file.exists() else 'not-found',+ UnitFileState=value['UnitFileState'] if file.exists() else '')+if verb == 'show':+ unit, properties = args+ if os.environ.get('MODEL_EMPTY_READ') == 'true': sys.exit(0)+ for key in reversed(properties.split('=', 1)[1].split(',')):+ if key != 'MainPID' or unit.endswith('.service'): print(key + '=' + observed(unit)[key])+elif verb == 'enable' or verb == 'start':+ v = observed(unit)+ text = (units / unit).read_text()+ v.update(ActiveState='active', MainPID=str(int(v.get('MainPID', '0')) + 100))+ if os.environ.get('MODEL_START_INACTIVE') == 'true': v.update(ActiveState='inactive', MainPID='0')+ if verb == 'enable': v['UnitFileState'] = 'enabled' if '[Install]' in text else 'static'+ values[unit] = v+elif verb == 'stop':+ file = units / unit+ text = file.read_text() if file.exists() else ''+ if 'RefuseManualStop=yes' in text: sys.exit(1)+ if os.environ.get('MODEL_IGNORE_STOP') != unit:+ target = 'pulse-removal-native-runner.service'+ if 'ExecStop=/usr/bin/systemctl start ' + target in text:+ values[target] = dict(observed(target), ActiveState='active', MainPID='200')+ values[unit] = dict(observed(unit), ActiveState='inactive', MainPID='0')+elif verb == 'disable':+ v = observed(unit)+ if v['UnitFileState'] != 'static': v['UnitFileState'] = 'disabled' if (units / unit).exists() else ''+ values[unit] = v+elif verb not in ('daemon-reload', 'reset-failed'): sys.exit(97)+database.write_text(json.dumps(values))+'''+++class ModelLab(native.Lab):+ def helper(self, directory, refuse=False):+ # Model activation directly; never claim this is a native socket.+ self.service(native.HELPER + ".service")+ self.install_unit(native.HELPER + ".socket", "[Unit]\n" ++ ("RefuseManualStop=yes\n" if refuse else "") + "[Install]\n")+++class NativeRemovalControls(unittest.TestCase):+ installer = (ROOT / "scripts/install.sh").read_text()++ def test_definition_selection_preserves_each_actual_function_and_refuses_drift(self):+ selected = native.definitions(self.installer)+ for name in native.FUNCTIONS:+ marker = name + "() {"+ body = marker + self.installer.split(marker, 1)[1].split("\n}\n", 1)[0] + "\n}\n"+ self.assertIn(body, selected)+ for change in (self.installer.replace(marker, name + "_moved() {"), self.installer + body):+ with self.assertRaises(ValueError): native.definitions(change)++ def test_admission_excludes_worker_self_hosted_nonroot_existing_hosts_and_unbound_source(self):+ env = dict(GITHUB_ACTIONS="true", RUNNER_ENVIRONMENT="github-hosted", GITHUB_REPOSITORY="rcourtman/Pulse")+ with patch.object(Path, "exists", return_value=False):+ native.admission(env, 0, "a" * 40)+ for changed, uid, sha in ((env, 1000, "a" * 40), ({}, 0, "a" * 40),+ ({**env, "RUNNER_ENVIRONMENT": "self-hosted"}, 0, "a" * 40),+ ({**env, "GITHUB_REPOSITORY": "other/repo"}, 0, "a" * 40),+ (env, 0, "main")):+ with self.assertRaises(ValueError): native.admission(changed, uid, sha)+ with patch.object(Path, "exists", return_value=True):+ with self.assertRaises(ValueError): native.admission(env, 0, "a" * 40)++ def test_independent_observation_refuses_missing_duplicate_and_diagnostic_lines(self):+ complete = b"LoadState=loaded\nActiveState=active\nUnitFileState=enabled\nMainPID=123\n"+ self.assertEqual(native.state(complete)["MainPID"], "123")+ for changed in (b"", complete.replace(b"LoadState=loaded\n", b""), complete + b"\n",+ complete + b"LoadState=loaded\n", complete + b"diagnostic\n"):+ with self.assertRaises(ValueError): native.state(changed)++ def test_recipe_owns_exact_clean_source_mandatory_linux_execution_and_bounded_retention(self):+ workflow = (ROOT / ".github/workflows/unified-agent-native.yml").read_text()+ recipe(workflow)+ for old, new in (("if: runner.os == 'Linux'\n timeout-minutes: 5", "if: false\n timeout-minutes: 5"),+ ("sudo env -i", "sudo env"), ("git diff --exit-code HEAD", "true"),+ ("EXPECTED_SOURCE_SHA: ${{ github.sha }}", "EXPECTED_SOURCE_SHA: main"),+ ("'scripts/tests/test_agent_removal_systemd.py'", "'other.py'"),+ ("path: ${{ runner.temp }}/pulse-agent-removal.json", "path: '**/*'")):+ self.assertIn(old, workflow)+ with self.assertRaises(AssertionError): recipe(workflow.replace(old, new))++ def model(self, callback):+ with tempfile.TemporaryDirectory() as temporary:+ root = Path(temporary)+ units, tools = root / "units", root / "tools"+ units.mkdir(); tools.mkdir()+ executable = tools / "systemctl"+ executable.write_text(MANAGER); executable.chmod(0o755)+ def model_pid(pid):+ database = root / "states.json"+ values = json.loads(database.read_text()) if database.exists() else {}+ return any(v["ActiveState"] == "active" and v["MainPID"] == pid for v in values.values())+ with patch.object(native, "LAB", root / "lab"), patch.object(native, "UNIT_DIR", units), patch.object(native, "pid_alive", side_effect=model_pid):+ lab = ModelLab("a" * 40, self.installer)+ lab.env.update(PATH=str(tools) + ":" + lab.env["PATH"], MODEL_UNITS=str(units), MODEL_STATES=str(root / "states.json"))+ try:+ callback(lab, root)+ finally:+ lab.cleanup()++ def test_all_nine_phase_verdicts_and_owned_cleanup_with_closed_manager_model(self):+ def run(lab, root):+ lab.prepare()+ for phase in native.PHASES:+ lab.exercise(phase)+ lab.cleanup_units()+ self.assertEqual([p["name"] for p in lab.receipt["phases"]], list(native.PHASES))+ self.assertTrue(all(p["passed"] for p in lab.receipt["phases"]))+ self.assertEqual(lab.receipt["credential_transport"], "closed-synthetic-revocation-only")+ lab.cleanup()+ self.assertTrue(lab.receipt["cleanup"]["complete"])+ self.assertFalse((root / "lab").exists())+ self.assertEqual(list((root / "units").iterdir()), [])+ self.model(run)++ def test_passing_stop_command_without_quiescence_is_not_a_native_pass(self):+ def run(lab, root):+ lab.prepare()+ lab.env["MODEL_IGNORE_STOP"] = native.AGENT + ".service"+ with self.assertRaisesRegex(RuntimeError, "verdict differs"): lab.exercise("active-agent")+ self.assertFalse(lab.receipt["phases"][0]["passed"])+ self.assertTrue(lab.receipt["phases"][0]["recovery_files_unchanged"])+ lab.env.pop("MODEL_IGNORE_STOP")+ self.model(run)++ def test_empty_successful_manager_read_is_not_an_absence_pass(self):+ def run(lab, root):+ lab.prepare()+ lab.env["MODEL_EMPTY_READ"] = "true"+ with self.assertRaisesRegex(ValueError, "observation missing"): lab.exercise("active-agent")+ self.assertEqual(lab.receipt["phases"], [])+ self.assertTrue((root / "units" / (native.AGENT + ".service")).exists())+ lab.env.pop("MODEL_EMPTY_READ")+ self.model(run)++ def test_successful_start_without_real_active_identity_cannot_satisfy_a_phase(self):+ def run(lab, root):+ lab.prepare()+ lab.env["MODEL_START_INACTIVE"] = "true"+ with self.assertRaisesRegex(RuntimeError, "intended active fixture"): lab.exercise("active-agent")+ self.assertEqual(lab.receipt["phases"], [])+ self.model(run)++ def test_lab_rejects_removal_that_discards_final_batch_readback(self):+ def run(lab, root):+ lab.prepare()+ last_read = ' for unit in "$@"; do\n confirm_agent_unit_quiescent "$unit" || return 1\n done\n'+ self.assertEqual(lab.definitions.count(last_read), 1)+ lab.definitions = lab.definitions.replace(last_read, " :\n")+ with self.assertRaisesRegex(RuntimeError, "verdict differs"): lab.exercise("batch-reactivation")+ record = lab.receipt["phases"][0]+ self.assertEqual(record["after"][native.RUNNER + ".service"]["ActiveState"], "active")+ self.assertFalse(record["passed"])+ self.model(run)++ def test_preexisting_unit_is_never_stopped_or_removed(self):+ def run(lab, root):+ sentinel = root / "units" / (native.AGENT + ".service")+ sentinel.write_text("preexisting unrelated service\n")+ with self.assertRaisesRegex(ValueError, "already owned"): lab.prepare()+ lab.cleanup()+ self.assertEqual(sentinel.read_text(), "preexisting unrelated service\n")+ self.assertFalse(lab.owned_root)+ self.model(run)++ def test_failed_cleanup_stays_failed_and_keeps_ownership_for_reconciliation(self):+ lab = native.Lab("a" * 40, self.installer)+ lab.owned_units.add(native.AGENT + ".service")+ with patch.object(lab, "cleanup_units", side_effect=RuntimeError("stop refused")):+ with self.assertRaises(RuntimeError): lab.cleanup()+ self.assertFalse(lab.receipt["cleanup"]["complete"])+ self.assertEqual(lab.owned_units, {native.AGENT + ".service"})+++if __name__ == "__main__":+ unittest.main()