Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Check agent removal against native hosted systemd

pulse-triage[bot]27f9c1de5e48
diff --git a/.github/workflows/unified-agent-native.yml b/.github/workflows/unified-agent-native.yml
index 4c039436c1..ee3ce15c3e 100644
--- a/.github/workflows/unified-agent-native.yml
+++ b/.github/workflows/unified-agent-native.yml
@@ -31,6 +31,7 @@ on:
- 'scripts/pulse-auto-update.sh'
- 'scripts/tests/test_auto_update_service_state.py'
- 'scripts/tests/test_auto_update_systemd.py'
+ - 'scripts/tests/test_agent_removal_systemd.py'
- 'scripts/tests/test_server_service_discovery.py'
- 'scripts/installtests/**'
- 'go.mod'
@@ -61,6 +62,7 @@ on:
- 'scripts/pulse-auto-update.sh'
- 'scripts/tests/test_auto_update_service_state.py'
- 'scripts/tests/test_auto_update_systemd.py'
+ - 'scripts/tests/test_agent_removal_systemd.py'
- 'scripts/tests/test_server_service_discovery.py'
- 'scripts/installtests/**'
- 'go.mod'
@@ -104,6 +106,35 @@ jobs:
with:
persist-credentials: false
+ # These are real manager/deletion checks, not the installer doubles. Run
+ # before compiler/frontend acquisition so unrelated preparation cannot
+ # hide Linux removal safety. Only the new public hosted runner is mutated.
+ - name: Check agent removal on native systemd
+ if: runner.os == 'Linux'
+ timeout-minutes: 5
+ env:
+ EXPECTED_SOURCE_SHA: ${{ github.sha }}
+ shell: bash
+ run: |
+ set -euo pipefail
+ test "$(git rev-parse HEAD)" = "$EXPECTED_SOURCE_SHA"
+ git diff --exit-code HEAD
+ python3 -m unittest discover -s scripts/tests -p test_agent_removal_systemd.py -v
+ sudo env -i PATH=/usr/sbin:/usr/bin:/sbin:/bin LC_ALL=C \
+ GITHUB_ACTIONS=true RUNNER_ENVIRONMENT=github-hosted GITHUB_REPOSITORY=rcourtman/Pulse \
+ EXPECTED_SOURCE_SHA="$EXPECTED_SOURCE_SHA" RUNNER_TEMP="$RUNNER_TEMP" \
+ python3 scripts/installtests/agent_removal_systemd.py \
+ --receipt "$RUNNER_TEMP/pulse-agent-removal.json"
+
+ - name: Retain bounded native agent removal receipt
+ if: always() && runner.os == 'Linux'
+ uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+ with:
+ name: native-agent-removal-${{ matrix.runner }}-${{ github.run_id }}-${{ github.run_attempt }}
+ path: ${{ runner.temp }}/pulse-agent-removal.json
+ if-no-files-found: warn
+ retention-days: 7
+
- name: Set up Go
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 341147242a..cabef9e7e3 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -44,6 +44,27 @@ saved-state removal controls also require refusal before remote or local mutatio
These are exact-source controls, not native systemd installation, TLS, published
availability or resolution of Linux/PowerShell #1099.
+`unified-agent-native.yml` additionally requires `agent_removal_systemd.py` on
+both new public hosted Linux runners, before compiler/frontend preparation. The
+ordinary runner binds a clean exact checkout; the root lab receives no ambient
+credentials and refuses workers, self-hosted/container runs, existing Pulse
+state and pre-existing fixture units. It extracts the six production removal
+functions unchanged and binds their bytes in a bounded terminal receipt.
+
+Its fixed synthetic units exercise active collector teardown, real helper socket
+activation, runner deletion only after fixture revocation, real refused stops,
+unavailable-user-manager read refusal, stopped/retained failed-revocation state,
+an earlier runner reactivated by a later unit's real ExecStop, and absent/repeated
+removal. Independent systemd/property and prior-PID observations must agree with
+the intended phase; recovery file hashes, no repeated synthetic revocation and
+owned unit/process/fixture cleanup are retained. Cleanup failure fails the job.
+The absent-user-manager adapter is an injected read-failure condition, not a
+production system-bus diagnosis. Unit names, credential files and services are
+synthetic: this does not establish signed agent installation, real TLS/exact-token
+revocation, published availability, Windows acceptance or reporter recovery.
+`TestNativeAgentRemovalSystemdRecipeRuntime` exercises the lab and mandatory CI
+recipe with closed manager models; those portable controls are not a native pass.
+
### Unix removal retains credentials when connection recovery is incomplete
After existing installed-state recovery, the unified Unix installer checks for
diff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.md
index 0f5cf3466c..0877bcf585 100644
--- a/docs/release-control/v6/internal/subsystems/deployment-installability.md
+++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md
@@ -1,5 +1,6 @@
# Deployment Installability Contract
+
## Contract Metadata
```json
@@ -15,6 +16,30 @@
## Purpose
+### Linux agent removal native boundary
+
+The existing Unified Agent Native Verification job now runs the unchanged Unix
+removal functions against actual systemd on its fresh public hosted Linux x64
+and ARM64 runners, before acquiring compiler/frontend dependencies. Its fixed
+private fixtures cover refused stop/read/revocation, socket/batch activation,
+observed quiescence, recovery-file retention, confirmed deletion and repeated
+absence. The exact source, extracted-function digest, per-phase independent unit
+and prior-process observations, file-retention verdicts and terminal cleanup are
+retained in a seven-day bounded JSON artifact even when the native command fails.
+No default-Pulse unit, existing installation, worker/self-hosted host, service
+credential or production endpoint is admitted; root execution belongs to that
+new hosted VM only. No missing or green-only receipt establishes acceptance.
+
+`TestNativeAgentRemovalSystemdRecipeRuntime` requires discriminating portable
+lab/recipe controls, including rejected worker admission, preserved pre-existing
+units, missing/duplicate observations, successful commands without actual active
+identity or quiescence, omitted final batch readback and failed cleanup. The
+native result still requires the containing job's terminal artifact and logs.
+Runner revocation is deliberately a closed fixture, not authenticated transport
+proof; signed installed-agent/Windows/service-manager/customer recovery remain
+separate. See the agent-lifecycle contract for the exact fixture boundaries.
+
+
### Agent systemd teardown refuses unobserved stops and activation sources
`scripts/install.sh` must not discard unit files, helper credentials or runner
diff --git a/scripts/installtests/agent_removal_systemd.py b/scripts/installtests/agent_removal_systemd.py
new file mode 100644
index 0000000000..091d001de1
--- /dev/null
+++ b/scripts/installtests/agent_removal_systemd.py
@@ -0,0 +1,378 @@
+#!/usr/bin/env python3
+"""Agent removal boundaries on a fresh public hosted systemd runner.
+
+Use the committed installer functions unchanged, with fixed private unit names
+and synthetic recovery files. systemctl, socket activation and deletion are real;
+runner credential revocation is a closed fixture, not TLS/server acceptance.
+This is not installation of a signed release or Windows/other-manager proof.
+Never run this lab on a worker, self-hosted runner or an existing Pulse host.
+"""
+
+import argparse
+import hashlib
+import json
+import os
+from pathlib import Path
+import re
+import shlex
+import shutil
+import signal
+import socket
+import subprocess
+import time
+
+ROOT = Path(__file__).resolve().parents[2]
+LAB = Path("/opt/pulse-agent-removal-native")
+UNIT_DIR = Path("/etc/systemd/system")
+AGENT = "pulse-removal-native-agent"
+HELPER = "pulse-removal-native-helper"
+RUNNER = "pulse-removal-native-runner"
+UNITS = (RUNNER + ".service", HELPER + ".socket", HELPER + ".service", AGENT + ".service")
+FUNCTIONS = ("read_agent_removal_unit_state", "confirm_agent_unit_quiescent",
+ "quiesce_agent_units_for_removal", "teardown_systemd_agent_service",
+ "teardown_privileged_helper_service", "teardown_action_runner_service")
+PHASES = ("active-agent", "activated-helper", "confirmed-runner",
+ "refused-agent-stop", "refused-helper-stop", "unavailable-manager",
+ "refused-runner-revocation", "batch-reactivation", "absent-and-repeated")
+
+
+def definitions(source):
+ result = []
+ for name in FUNCTIONS:
+ marker = name + "() {"
+ if source.count(marker) != 1:
+ raise ValueError("installer function boundary changed")
+ body = source.split(marker, 1)[1]
+ if "\n}\n" not in body:
+ raise ValueError("installer function end missing")
+ result.append(marker + body.split("\n}\n", 1)[0] + "\n}\n")
+ return "\n".join(result)
+
+
+def admission(env, uid, source):
+ if (uid != 0 or env.get("GITHUB_ACTIONS") != "true"
+ or env.get("RUNNER_ENVIRONMENT") != "github-hosted"
+ or env.get("GITHUB_REPOSITORY") != "rcourtman/Pulse"
+ or not re.fullmatch(r"[0-9a-f]{40}", source)
+ or any(Path(p).exists() for p in
+ ("/.dockerenv", "/opt/pulse", "/etc/pulse", "/etc/pulse-agent",
+ "/etc/pulse-agent-runner", "/var/lib/pulse-agent", LAB))):
+ raise ValueError("agent removal proof requires a fresh public hosted VM")
+
+
+def state(output):
+ values = {}
+ for line in output.decode("ascii").splitlines():
+ key, separator, value = line.partition("=")
+ if not separator or key not in {"LoadState", "ActiveState", "UnitFileState", "MainPID"} or key in values:
+ raise ValueError("independent unit observation incomplete")
+ values[key] = value
+ if not {"LoadState", "ActiveState", "UnitFileState"} <= values.keys():
+ raise ValueError("independent unit observation missing")
+ return values
+
+
+def pid_alive(pid):
+ if not re.fullmatch(r"[1-9][0-9]*", pid):
+ raise ValueError("native process identity unavailable")
+ try:
+ os.kill(int(pid), 0)
+ except ProcessLookupError:
+ return False
+ # Permission/read errors stay failures; never manufacture process absence.
+ return True
+
+
+class Lab:
+ def __init__(self, source, installer):
+ self.source = source
+ self.definitions = definitions(installer)
+ self.owned_units = set()
+ self.owned_pids = set()
+ self.owned_root = False
+ self.env = {"PATH": "/usr/sbin:/usr/bin:/sbin:/bin", "LC_ALL": "C", "HOME": str(LAB)}
+ self.receipt = {"schema_version": 1, "source_sha": source,
+ "scope": "current-source-native-systemd-removal-boundary",
+ "installer_sha256": hashlib.sha256(installer.encode()).hexdigest(),
+ "definitions_sha256": hashlib.sha256(self.definitions.encode()).hexdigest(),
+ "credential_transport": "closed-synthetic-revocation-only",
+ "phases": [], "cleanup": {"complete": False}}
+
+ def run(self, *args, ok=(0,), timeout=20):
+ process = subprocess.Popen(args, env=self.env, stdin=subprocess.DEVNULL,
+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+ start_new_session=True)
+ try:
+ output, _ = process.communicate(timeout=timeout)
+ except subprocess.TimeoutExpired:
+ os.killpg(process.pid, signal.SIGTERM)
+ try:
+ process.communicate(timeout=2)
+ except subprocess.TimeoutExpired:
+ os.killpg(process.pid, signal.SIGKILL)
+ process.communicate(timeout=2)
+ self.receipt["command_failure"] = {"deadline_exceeded": True,
+ "observed_exit": process.returncode}
+ raise RuntimeError("owned command deadline exceeded") from None
+ result = subprocess.CompletedProcess(args, process.returncode, output)
+ if result.returncode not in ok:
+ self.receipt["command_failure"] = {"command": Path(args[0]).name,
+ "observed_exit": result.returncode,
+ "output_bytes": len(output),
+ "output_sha256": hashlib.sha256(output).hexdigest()}
+ raise RuntimeError("native command failed")
+ return result
+
+ def observe(self):
+ observed = {unit: state(self.run("systemctl", "show", unit,
+ "--property=LoadState,ActiveState,UnitFileState,MainPID").stdout)
+ for unit in UNITS}
+ for unit in self.owned_units:
+ pid = observed[unit].get("MainPID", "0")
+ if pid != "0":
+ if not re.fullmatch(r"[1-9][0-9]*", pid):
+ raise ValueError("native process identity unavailable")
+ self.owned_pids.add(pid)
+ return observed
+
+ def prepare(self):
+ # Reject loaded or vendor-provided units as well as on-disk collisions.
+ if any(Path(UNIT_DIR / unit).exists() or Path(UNIT_DIR / unit).is_symlink() for unit in UNITS):
+ raise ValueError("native unit name already owned")
+ if any(v["LoadState"] != "not-found" for v in self.observe().values()):
+ raise ValueError("native unit already loaded")
+ LAB.mkdir(mode=0o700)
+ self.owned_root = True
+
+ def install_unit(self, unit, text):
+ if unit not in UNITS:
+ raise ValueError("unit outside fixed lab")
+ with (UNIT_DIR / unit).open("x") as stream:
+ self.owned_units.add(unit)
+ stream.write(text)
+ self.run("systemctl", "daemon-reload")
+ self.run("systemctl", "enable", "--now", unit)
+
+ def service(self, unit, extra=""):
+ self.install_unit(unit, "[Unit]\nDescription=Pulse owned removal fixture\n" +
+ ("RefuseManualStop=yes\n" if extra == "refuse" else "") +
+ "[Service]\nType=simple\nExecStart=/usr/bin/sleep infinity\n" +
+ ("ExecStop=/usr/bin/systemctl start " + RUNNER + ".service\n"
+ if extra == "reactivate" else "") +
+ "[Install]\nWantedBy=multi-user.target\n")
+
+ def helper(self, directory, refuse=False):
+ # The pending connection genuinely starts the paired service. The
+ # synthetic process does not implement the privileged-helper protocol.
+ with (UNIT_DIR / (HELPER + ".service")).open("x") as stream:
+ self.owned_units.add(HELPER + ".service")
+ stream.write("[Service]\nExecStart=/usr/bin/sleep infinity\n")
+ self.install_unit(HELPER + ".socket",
+ "[Unit]\nDescription=Pulse owned socket fixture\n" +
+ ("RefuseManualStop=yes\n" if refuse else "") +
+ "[Socket]\nListenStream=" + str(directory / "helper.sock") +
+ "\n[Install]\nWantedBy=sockets.target\n")
+ with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as connection:
+ connection.settimeout(2)
+ connection.connect(str(directory / "helper.sock"))
+ deadline = time.monotonic() + 5
+ while self.observe()[HELPER + ".service"]["ActiveState"] != "active":
+ if time.monotonic() >= deadline:
+ raise RuntimeError("real helper socket activation was not observed")
+ time.sleep(0.05)
+
+ def shell(self, directory, command, *, revoke=True, unavailable=False):
+ values = {"AGENT_NAME": AGENT, "PRIVILEGED_HELPER_NAME": HELPER, "ACTION_RUNNER_NAME": RUNNER,
+ "PRIVILEGED_HELPER_SOCKET_UNIT": UNIT_DIR / (HELPER + ".socket"),
+ "PRIVILEGED_HELPER_SERVICE_UNIT": UNIT_DIR / (HELPER + ".service"),
+ "PRIVILEGED_HELPER_SOCKET_PATH": directory / "helper.sock",
+ "PRIVILEGED_HELPER_CREDENTIAL_DIR": directory / "helper-credentials",
+ "ACTION_RUNNER_SERVICE_UNIT": UNIT_DIR / (RUNNER + ".service"),
+ "ACTION_RUNNER_BINARY_PATH": directory / "runner",
+ "ACTION_RUNNER_CONFIG_DIR": directory / "runner-config",
+ "ACTION_RUNNER_STATE_DIR": directory / "runner-state"}
+ prefix = "set -euo pipefail\nEXIT_GENERAL=1\n" + "\n".join(
+ key + "=" + shlex.quote(str(value)) for key, value in values.items()) + "\n"
+ prefix += 'fail() { exit "${2:-1}"; }\nlog_info() { :; }\nlog_error() { :; }\n'
+ prefix += "revoke_action_runner_credential() { printf 'attempted\\n' >> " + shlex.quote(str(directory / "revoke")) + "; " + ("return 0" if revoke else "return 1") + "; }\n"
+ if unavailable:
+ # This adapter directs the real systemctl to a deliberately absent
+ # user manager. It proves read-failure retention, not a system-bus
+ # incident. No failed system-manager command is replayed.
+ # timeout execs its target; a shell function would not intercept it.
+ tools = directory / "unavailable-manager-tools"
+ tools.mkdir()
+ manager = shutil.which("systemctl", path=self.env["PATH"])
+ if manager is None:
+ raise ValueError("native manager executable unavailable")
+ adapter = tools / "systemctl"
+ adapter.write_text("#!/bin/sh\nexec env XDG_RUNTIME_DIR=" +
+ shlex.quote(str(directory / "no-manager")) +
+ " DBUS_SESSION_BUS_ADDRESS=" + shlex.quote("unix:path=" + str(directory / "no-bus")) +
+ " " + shlex.quote(manager) + ' --user "$@"\n')
+ adapter.chmod(0o700)
+ prefix += "PATH=" + shlex.quote(str(tools)) + ':"$PATH"\n'
+ return self.run("/bin/bash", "-c", prefix + self.definitions + command, ok=(0, 1), timeout=40)
+
+ def snapshot(self, directory):
+ paths = [directory / "runner"] + [UNIT_DIR / unit for unit in self.owned_units]
+ for name in ("helper-credentials", "runner-config", "runner-state"):
+ paths.extend((directory / name).rglob("*"))
+ return {str(path): hashlib.sha256(path.read_bytes()).hexdigest()
+ for path in paths if path.is_file() and path.name != "revoke"}
+
+ def exercise(self, phase):
+ directory = LAB / phase
+ directory.mkdir(mode=0o700)
+ for name in ("helper-credentials", "runner-config", "runner-state"):
+ (directory / name).mkdir(mode=0o700)
+ (directory / name / "token").write_text("synthetic-recovery-material\n")
+ (directory / "runner").write_text("synthetic-runner-not-executable\n")
+ if phase == "batch-reactivation":
+ self.service(RUNNER + ".service")
+ self.helper(directory)
+ self.service(AGENT + ".service", "reactivate")
+ elif "helper" in phase:
+ self.helper(directory, phase == "refused-helper-stop")
+ elif "runner" in phase:
+ self.service(RUNNER + ".service")
+ elif phase != "absent-and-repeated":
+ self.service(AGENT + ".service", "refuse" if phase == "refused-agent-stop" else "")
+ before, files_before = self.observe(), self.snapshot(directory)
+ for unit, observed in before.items():
+ if unit in self.owned_units:
+ if (observed["LoadState"] != "loaded" or observed["ActiveState"] != "active"
+ or observed["UnitFileState"] not in {"enabled", "static"}
+ or (unit.endswith(".service") and not pid_alive(observed.get("MainPID", "0")))):
+ raise RuntimeError("intended active fixture was not observed")
+ elif observed["LoadState"] != "not-found" or observed["ActiveState"] != "inactive":
+ raise RuntimeError("unowned native unit appeared")
+ agent_command = "teardown_systemd_agent_service " + shlex.quote(str(UNIT_DIR / (AGENT + ".service")))
+ if phase == "batch-reactivation":
+ command = "quiesce_agent_units_for_removal " + " ".join(UNITS) + " || fail refused 1\nprintf mutation > " + shlex.quote(str(directory / "mutation"))
+ elif phase == "absent-and-repeated":
+ command = (agent_command + "\nteardown_privileged_helper_service\nteardown_action_runner_service\n") * 2
+ elif "helper" in phase:
+ command = "teardown_privileged_helper_service"
+ elif "runner" in phase:
+ command = "teardown_action_runner_service"
+ else:
+ command = agent_command
+ result = self.shell(directory, command, revoke=phase != "refused-runner-revocation",
+ unavailable=phase == "unavailable-manager")
+ after = self.observe()
+ refused = phase.startswith("refused-") or phase in {"unavailable-manager", "batch-reactivation"}
+ record = {"name": phase, "command_exit": result.returncode, "before": before, "after": after,
+ "recovery_files_unchanged": self.snapshot(directory) == files_before,
+ "revocation_attempted": (directory / "revoke").exists(),
+ "revocation_attempts": len((directory / "revoke").read_text().splitlines()) if (directory / "revoke").exists() else 0,
+ "prior_process_alive": {unit: pid_alive(v["MainPID"]) for unit, v in before.items()
+ if v.get("MainPID", "0") != "0"}, "passed": False}
+ self.receipt["phases"].append(record)
+ if result.returncode != (1 if refused else 0):
+ raise RuntimeError("native removal verdict differs")
+ if refused:
+ if not record["recovery_files_unchanged"] or (directory / "mutation").exists():
+ raise RuntimeError("unconfirmed removal erased recovery material")
+ if phase in {"refused-agent-stop", "unavailable-manager"} and (
+ after[AGENT + ".service"]["ActiveState"] != "active" or
+ before[AGENT + ".service"]["MainPID"] != after[AGENT + ".service"]["MainPID"] or
+ not record["prior_process_alive"][AGENT + ".service"]):
+ raise RuntimeError("refused collector changed process identity")
+ if phase == "batch-reactivation" and after[RUNNER + ".service"]["ActiveState"] != "active":
+ raise RuntimeError("batch reactivation adverse control did not occur")
+ if phase == "refused-runner-revocation" and (
+ not record["revocation_attempted"] or after[RUNNER + ".service"]["ActiveState"] != "inactive" or
+ after[RUNNER + ".service"]["UnitFileState"] != "disabled"):
+ raise RuntimeError("revocation refusal lacks stopped retained runner")
+ else:
+ targets = (UNITS if phase == "absent-and-repeated" else
+ (HELPER + ".socket", HELPER + ".service") if "helper" in phase else
+ (RUNNER + ".service",) if "runner" in phase else (AGENT + ".service",))
+ for unit in targets:
+ if (after[unit]["LoadState"] != "not-found" or after[unit]["ActiveState"] != "inactive"
+ or after[unit].get("MainPID", "0") != "0" or (UNIT_DIR / unit).exists()):
+ raise RuntimeError("native successful removal left unit or process")
+ if record["prior_process_alive"].get(unit, False):
+ raise RuntimeError("native successful removal left its prior process alive")
+ if "helper" in phase and any((directory / n).exists() for n in ("helper.sock", "helper-credentials")):
+ raise RuntimeError("helper recovery material not removed after confirmation")
+ if "runner" in phase and (not record["revocation_attempted"] or
+ any((directory / n).exists() for n in ("runner", "runner-config", "runner-state"))):
+ raise RuntimeError("confirmed runner removal incomplete")
+ if phase == "absent-and-repeated" and record["revocation_attempts"] != 1:
+ raise RuntimeError("repeated absent removal repeated synthetic revocation")
+ record["passed"] = True
+
+ def cleanup_units(self):
+ # Withdraw only our own refusal/ExecStop fixtures before stopping them.
+ # Never repair, stop or unlink a pre-existing unit.
+ for unit in self.owned_units:
+ path = UNIT_DIR / unit
+ if path.exists():
+ text = path.read_text()
+ text = re.sub(r"(?m)^(RefuseManualStop|ExecStop)=.*\n", "", text)
+ path.write_text(text)
+ if self.owned_units:
+ self.run("systemctl", "daemon-reload")
+ for unit in UNITS:
+ if unit in self.owned_units:
+ self.run("systemctl", "disable", unit, ok=(0, 1))
+ self.run("systemctl", "stop", unit, ok=(0, 5))
+ (UNIT_DIR / unit).unlink(missing_ok=True)
+ if self.owned_units:
+ self.run("systemctl", "daemon-reload")
+ self.run("systemctl", "reset-failed", *sorted(self.owned_units), ok=(0, 1))
+ if any(v["LoadState"] != "not-found" or v["ActiveState"] != "inactive" or v.get("MainPID", "0") != "0"
+ for v in self.observe().values()):
+ raise RuntimeError("owned native unit cleanup incomplete")
+ if any(pid_alive(pid) for pid in self.owned_pids):
+ raise RuntimeError("owned native process cleanup incomplete")
+ self.owned_units.clear()
+ self.owned_pids.clear()
+
+ def cleanup(self):
+ self.cleanup_units()
+ if self.owned_root:
+ shutil.rmtree(LAB)
+ self.owned_root = False
+ self.receipt["cleanup"] = {"complete": True, "owned_units_absent": True,
+ "owned_fixture_absent": not LAB.exists()}
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--receipt", required=True)
+ args = parser.parse_args()
+ source = os.environ.get("EXPECTED_SOURCE_SHA", "")
+ admission(os.environ, os.geteuid(), source)
+ receipt = Path(args.receipt).resolve()
+ if receipt.parent != Path(os.environ["RUNNER_TEMP"]).resolve() or receipt.exists():
+ raise ValueError("receipt must be a new direct runner-temp file")
+ # The ordinary runner checks the exact clean checkout before sudo. Do not
+ # change root's Git ownership/configuration to read a runner-owned checkout.
+ # Bind the actual installer and extracted bytes in the terminal receipt too.
+ lab = Lab(source, (ROOT / "scripts/install.sh").read_text())
+ failed = False
+ try:
+ lab.prepare()
+ for phase in PHASES:
+ lab.exercise(phase)
+ lab.cleanup_units()
+ except (OSError, ValueError, RuntimeError, subprocess.SubprocessError) as error:
+ failed = True
+ lab.receipt["failure"] = {"type": type(error).__name__}
+ finally:
+ try:
+ lab.cleanup()
+ except (OSError, ValueError, RuntimeError, subprocess.SubprocessError) as error:
+ failed = True
+ lab.receipt["cleanup"] = {"complete": False, "error_type": type(error).__name__}
+ receipt.write_text(json.dumps(lab.receipt, indent=2) + "\n")
+ if failed or len(lab.receipt["phases"]) != len(PHASES):
+ raise SystemExit("Native agent removal proof incomplete; inspect the bounded receipt.")
+ print("NATIVE_AGENT_REMOVAL_COMPLETE=" + source)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/scripts/installtests/install_ps1_test.go b/scripts/installtests/install_ps1_test.go
index c39bb3490f..da35216005 100644
--- a/scripts/installtests/install_ps1_test.go
+++ b/scripts/installtests/install_ps1_test.go
@@ -611,6 +611,47 @@ func TestWindowsAgentLifecycleWorkflowChecksEveryNativeExit(t *testing.T) {
}
}
+func TestWindowsAgentLifecycleKeepsLinuxRemovalProofIsolated(t *testing.T) {
+ steps := nativeWindowsExitSteps(t)
+ validate := func(steps []nativeWindowsExitStep) error {
+ wanted := map[string]string{
+ "Check agent removal on native systemd": "runner.os == 'Linux'",
+ "Retain bounded native agent removal receipt": "always() && runner.os == 'Linux'",
+ }
+ for _, step := range steps {
+ condition, found := wanted[step.Name]
+ if !found {
+ continue
+ }
+ if step.If != condition || step.ContinueOnError {
+ return fmt.Errorf("Linux removal must not run on or replace Windows lifecycle: %s", step.Name)
+ }
+ delete(wanted, step.Name)
+ }
+ if len(wanted) != 0 {
+ return fmt.Errorf("missing isolated Linux removal proof: %v", wanted)
+ }
+ _, err := nativeWindowsExitGuards(steps)
+ return err
+ }
+ if err := validate(steps); err != nil {
+ t.Fatal(err)
+ }
+ for _, name := range []string{"Check agent removal on native systemd", "Retain bounded native agent removal receipt"} {
+ for _, condition := range []string{"", "true", "${{ !matrix.unix }}"} {
+ changed := append([]nativeWindowsExitStep(nil), steps...)
+ for i := range changed {
+ if changed[i].Name == name {
+ changed[i].If = condition
+ }
+ }
+ if err := validate(changed); err == nil {
+ t.Fatalf("admitted unbounded Linux proof on Windows: %s / %q", name, condition)
+ }
+ }
+ }
+}
+
func TestWindowsAgentLifecycleHarnessChecksEveryNativeExit(t *testing.T) {
content, err := os.ReadFile(repoFile("scripts", "installtests", "windows_agent_lifecycle.ps1"))
if err != nil {
diff --git a/scripts/installtests/native_agent_removal_test.go b/scripts/installtests/native_agent_removal_test.go
new file mode 100644
index 0000000000..0bd15aa8da
--- /dev/null
+++ b/scripts/installtests/native_agent_removal_test.go
@@ -0,0 +1,39 @@
+package installtests
+
+import (
+ "context"
+ "os/exec"
+ "runtime"
+ "strings"
+ "testing"
+ "time"
+)
+
+func TestNativeAgentRemovalSystemdRecipeRuntime(t *testing.T) {
+ if runtime.GOOS != "linux" {
+ t.Skip("Linux-only native systemd lab; Windows lifecycle remains independently required")
+ }
+ ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
+ defer cancel()
+ command := exec.CommandContext(ctx, "python3", repoFile("scripts", "tests", "test_agent_removal_systemd.py"), "-v")
+ output, err := command.CombinedOutput()
+ if err != nil {
+ t.Fatalf("native systemd lab controls: %v\n%s", err, output)
+ }
+ for _, name := range []string{
+ "test_definition_selection_preserves_each_actual_function_and_refuses_drift",
+ "test_admission_excludes_worker_self_hosted_nonroot_existing_hosts_and_unbound_source",
+ "test_recipe_owns_exact_clean_source_mandatory_linux_execution_and_bounded_retention",
+ "test_all_nine_phase_verdicts_and_owned_cleanup_with_closed_manager_model",
+ "test_lab_rejects_removal_that_discards_final_batch_readback",
+ "test_preexisting_unit_is_never_stopped_or_removed",
+ "test_failed_cleanup_stays_failed_and_keeps_ownership_for_reconciliation",
+ } {
+ if !strings.Contains(string(output), name+" (") {
+ t.Fatalf("native systemd lab control %s did not execute:\n%s", name, output)
+ }
+ }
+ if !strings.Contains(string(output), "\nOK\n") || strings.Contains(string(output), "skipped") {
+ t.Fatalf("native systemd lab controls incomplete:\n%s", output)
+ }
+}
diff --git a/scripts/tests/test_agent_removal_systemd.py b/scripts/tests/test_agent_removal_systemd.py
new file mode 100644
index 0000000000..1c86dfd2e5
--- /dev/null
+++ b/scripts/tests/test_agent_removal_systemd.py
@@ -0,0 +1,246 @@
+#!/usr/bin/env python3
+"""Portable lab/recipe controls. Manager doubles are NOT native acceptance."""
+
+import importlib.util
+import json
+import os
+from pathlib import Path
+import re
+import subprocess
+import tempfile
+import unittest
+from unittest.mock import patch
+
+ROOT = Path(__file__).resolve().parents[2]
+SPEC = importlib.util.spec_from_file_location("agent_removal_native", ROOT / "scripts/installtests/agent_removal_systemd.py")
+native = importlib.util.module_from_spec(SPEC)
+SPEC.loader.exec_module(native)
+STEP = "Check agent removal on native systemd"
+
+
+def recipe(source):
+ job = re.search(r"(?ms)^ native-agent:\n(.*?)(?=^ [a-z][a-z0-9-]*:|\Z)", source)[1]
+ steps = dict(re.findall(r"(?ms)^ - name: ([^\n]+)\n(.*?)(?=^ - name:|\Z)", job))
+ step = steps[STEP]
+ assert "if: runner.os == 'Linux'" in step
+ assert "timeout-minutes: 5" in step
+ assert "EXPECTED_SOURCE_SHA: ${{ github.sha }}" in step
+ assert not re.search(r"^ (continue-on-error|working-directory):", step, re.M)
+ assert list(steps).index(STEP) < list(steps).index("Set up Go")
+ assert "ubuntu-24.04\n" in job and "ubuntu-24.04-arm\n" in job
+ assert "persist-credentials: false" in steps["Checkout repository"]
+ shell = re.search(r"(?ms)^ run: \|\n((?: [^\n]*\n)+)", step)[1]
+ shell = "\n".join(line[10:] for line in shell.splitlines()) + "\n"
+ for required in ('set -euo pipefail', 'test "$(git rev-parse HEAD)" = "$EXPECTED_SOURCE_SHA"',
+ "git diff --exit-code HEAD", "python3 -m unittest discover -s scripts/tests -p test_agent_removal_systemd.py -v",
+ "sudo env -i", "GITHUB_ACTIONS=true RUNNER_ENVIRONMENT=github-hosted GITHUB_REPOSITORY=rcourtman/Pulse",
+ 'EXPECTED_SOURCE_SHA="$EXPECTED_SOURCE_SHA" RUNNER_TEMP="$RUNNER_TEMP"',
+ "python3 scripts/installtests/agent_removal_systemd.py", '--receipt "$RUNNER_TEMP/pulse-agent-removal.json"'):
+ assert required in shell, required
+ assert shell.index("git diff") < shell.index("sudo env -i")
+ assert "secrets." not in step and "GH_TOKEN" not in shell
+ retained = steps["Retain bounded native agent removal receipt"]
+ assert "if: always() && runner.os == 'Linux'" in retained
+ assert "path: ${{ runner.temp }}/pulse-agent-removal.json\n" in retained
+ assert "retention-days: 7" in retained
+ assert len(re.findall(r"(?m)^ path:", retained)) == 1 and "*.log" not in retained
+ for event in ("push", "pull_request"):
+ section = re.search(r"(?ms)^ " + event + r":\n(.*?)(?=^ [a-z_]+:|\Z)", source)[1]
+ assert "'scripts/tests/test_agent_removal_systemd.py'" in section
+ assert "'scripts/installtests/**'" in section and "'scripts/install.sh'" in section
+ return shell
+
+
+# An explicit file-backed manager model for the lab's verdict and cleanup
+# controls only. The native CI command uses /usr/bin/systemctl, not this model.
+MANAGER = r'''#!/usr/bin/python3
+import json, os, pathlib, sys
+units = pathlib.Path(os.environ['MODEL_UNITS'])
+database = pathlib.Path(os.environ['MODEL_STATES'])
+values = json.loads(database.read_text()) if database.exists() else {}
+args = sys.argv[1:]
+if args[0] == '--user':
+ print('closed fixture: no user manager', file=sys.stderr)
+ sys.exit(1)
+verb = args.pop(0)
+unit = args[-1] if args else ''
+def observed(name):
+ file = units / name
+ value = values.get(name, {'ActiveState':'inactive', 'UnitFileState':'disabled', 'MainPID':'0'})
+ return dict(value, LoadState='loaded' if file.exists() else 'not-found',
+ UnitFileState=value['UnitFileState'] if file.exists() else '')
+if verb == 'show':
+ unit, properties = args
+ if os.environ.get('MODEL_EMPTY_READ') == 'true': sys.exit(0)
+ for key in reversed(properties.split('=', 1)[1].split(',')):
+ if key != 'MainPID' or unit.endswith('.service'): print(key + '=' + observed(unit)[key])
+elif verb == 'enable' or verb == 'start':
+ v = observed(unit)
+ text = (units / unit).read_text()
+ v.update(ActiveState='active', MainPID=str(int(v.get('MainPID', '0')) + 100))
+ if os.environ.get('MODEL_START_INACTIVE') == 'true': v.update(ActiveState='inactive', MainPID='0')
+ if verb == 'enable': v['UnitFileState'] = 'enabled' if '[Install]' in text else 'static'
+ values[unit] = v
+elif verb == 'stop':
+ file = units / unit
+ text = file.read_text() if file.exists() else ''
+ if 'RefuseManualStop=yes' in text: sys.exit(1)
+ if os.environ.get('MODEL_IGNORE_STOP') != unit:
+ target = 'pulse-removal-native-runner.service'
+ if 'ExecStop=/usr/bin/systemctl start ' + target in text:
+ values[target] = dict(observed(target), ActiveState='active', MainPID='200')
+ values[unit] = dict(observed(unit), ActiveState='inactive', MainPID='0')
+elif verb == 'disable':
+ v = observed(unit)
+ if v['UnitFileState'] != 'static': v['UnitFileState'] = 'disabled' if (units / unit).exists() else ''
+ values[unit] = v
+elif verb not in ('daemon-reload', 'reset-failed'): sys.exit(97)
+database.write_text(json.dumps(values))
+'''
+
+
+class ModelLab(native.Lab):
+ def helper(self, directory, refuse=False):
+ # Model activation directly; never claim this is a native socket.
+ self.service(native.HELPER + ".service")
+ self.install_unit(native.HELPER + ".socket", "[Unit]\n" +
+ ("RefuseManualStop=yes\n" if refuse else "") + "[Install]\n")
+
+
+class NativeRemovalControls(unittest.TestCase):
+ installer = (ROOT / "scripts/install.sh").read_text()
+
+ def test_definition_selection_preserves_each_actual_function_and_refuses_drift(self):
+ selected = native.definitions(self.installer)
+ for name in native.FUNCTIONS:
+ marker = name + "() {"
+ body = marker + self.installer.split(marker, 1)[1].split("\n}\n", 1)[0] + "\n}\n"
+ self.assertIn(body, selected)
+ for change in (self.installer.replace(marker, name + "_moved() {"), self.installer + body):
+ with self.assertRaises(ValueError): native.definitions(change)
+
+ def test_admission_excludes_worker_self_hosted_nonroot_existing_hosts_and_unbound_source(self):
+ env = dict(GITHUB_ACTIONS="true", RUNNER_ENVIRONMENT="github-hosted", GITHUB_REPOSITORY="rcourtman/Pulse")
+ with patch.object(Path, "exists", return_value=False):
+ native.admission(env, 0, "a" * 40)
+ for changed, uid, sha in ((env, 1000, "a" * 40), ({}, 0, "a" * 40),
+ ({**env, "RUNNER_ENVIRONMENT": "self-hosted"}, 0, "a" * 40),
+ ({**env, "GITHUB_REPOSITORY": "other/repo"}, 0, "a" * 40),
+ (env, 0, "main")):
+ with self.assertRaises(ValueError): native.admission(changed, uid, sha)
+ with patch.object(Path, "exists", return_value=True):
+ with self.assertRaises(ValueError): native.admission(env, 0, "a" * 40)
+
+ def test_independent_observation_refuses_missing_duplicate_and_diagnostic_lines(self):
+ complete = b"LoadState=loaded\nActiveState=active\nUnitFileState=enabled\nMainPID=123\n"
+ self.assertEqual(native.state(complete)["MainPID"], "123")
+ for changed in (b"", complete.replace(b"LoadState=loaded\n", b""), complete + b"\n",
+ complete + b"LoadState=loaded\n", complete + b"diagnostic\n"):
+ with self.assertRaises(ValueError): native.state(changed)
+
+ def test_recipe_owns_exact_clean_source_mandatory_linux_execution_and_bounded_retention(self):
+ workflow = (ROOT / ".github/workflows/unified-agent-native.yml").read_text()
+ recipe(workflow)
+ for old, new in (("if: runner.os == 'Linux'\n timeout-minutes: 5", "if: false\n timeout-minutes: 5"),
+ ("sudo env -i", "sudo env"), ("git diff --exit-code HEAD", "true"),
+ ("EXPECTED_SOURCE_SHA: ${{ github.sha }}", "EXPECTED_SOURCE_SHA: main"),
+ ("'scripts/tests/test_agent_removal_systemd.py'", "'other.py'"),
+ ("path: ${{ runner.temp }}/pulse-agent-removal.json", "path: '**/*'")):
+ self.assertIn(old, workflow)
+ with self.assertRaises(AssertionError): recipe(workflow.replace(old, new))
+
+ def model(self, callback):
+ with tempfile.TemporaryDirectory() as temporary:
+ root = Path(temporary)
+ units, tools = root / "units", root / "tools"
+ units.mkdir(); tools.mkdir()
+ executable = tools / "systemctl"
+ executable.write_text(MANAGER); executable.chmod(0o755)
+ def model_pid(pid):
+ database = root / "states.json"
+ values = json.loads(database.read_text()) if database.exists() else {}
+ return any(v["ActiveState"] == "active" and v["MainPID"] == pid for v in values.values())
+ with patch.object(native, "LAB", root / "lab"), patch.object(native, "UNIT_DIR", units), patch.object(native, "pid_alive", side_effect=model_pid):
+ lab = ModelLab("a" * 40, self.installer)
+ lab.env.update(PATH=str(tools) + ":" + lab.env["PATH"], MODEL_UNITS=str(units), MODEL_STATES=str(root / "states.json"))
+ try:
+ callback(lab, root)
+ finally:
+ lab.cleanup()
+
+ def test_all_nine_phase_verdicts_and_owned_cleanup_with_closed_manager_model(self):
+ def run(lab, root):
+ lab.prepare()
+ for phase in native.PHASES:
+ lab.exercise(phase)
+ lab.cleanup_units()
+ self.assertEqual([p["name"] for p in lab.receipt["phases"]], list(native.PHASES))
+ self.assertTrue(all(p["passed"] for p in lab.receipt["phases"]))
+ self.assertEqual(lab.receipt["credential_transport"], "closed-synthetic-revocation-only")
+ lab.cleanup()
+ self.assertTrue(lab.receipt["cleanup"]["complete"])
+ self.assertFalse((root / "lab").exists())
+ self.assertEqual(list((root / "units").iterdir()), [])
+ self.model(run)
+
+ def test_passing_stop_command_without_quiescence_is_not_a_native_pass(self):
+ def run(lab, root):
+ lab.prepare()
+ lab.env["MODEL_IGNORE_STOP"] = native.AGENT + ".service"
+ with self.assertRaisesRegex(RuntimeError, "verdict differs"): lab.exercise("active-agent")
+ self.assertFalse(lab.receipt["phases"][0]["passed"])
+ self.assertTrue(lab.receipt["phases"][0]["recovery_files_unchanged"])
+ lab.env.pop("MODEL_IGNORE_STOP")
+ self.model(run)
+
+ def test_empty_successful_manager_read_is_not_an_absence_pass(self):
+ def run(lab, root):
+ lab.prepare()
+ lab.env["MODEL_EMPTY_READ"] = "true"
+ with self.assertRaisesRegex(ValueError, "observation missing"): lab.exercise("active-agent")
+ self.assertEqual(lab.receipt["phases"], [])
+ self.assertTrue((root / "units" / (native.AGENT + ".service")).exists())
+ lab.env.pop("MODEL_EMPTY_READ")
+ self.model(run)
+
+ def test_successful_start_without_real_active_identity_cannot_satisfy_a_phase(self):
+ def run(lab, root):
+ lab.prepare()
+ lab.env["MODEL_START_INACTIVE"] = "true"
+ with self.assertRaisesRegex(RuntimeError, "intended active fixture"): lab.exercise("active-agent")
+ self.assertEqual(lab.receipt["phases"], [])
+ self.model(run)
+
+ def test_lab_rejects_removal_that_discards_final_batch_readback(self):
+ def run(lab, root):
+ lab.prepare()
+ last_read = ' for unit in "$@"; do\n confirm_agent_unit_quiescent "$unit" || return 1\n done\n'
+ self.assertEqual(lab.definitions.count(last_read), 1)
+ lab.definitions = lab.definitions.replace(last_read, " :\n")
+ with self.assertRaisesRegex(RuntimeError, "verdict differs"): lab.exercise("batch-reactivation")
+ record = lab.receipt["phases"][0]
+ self.assertEqual(record["after"][native.RUNNER + ".service"]["ActiveState"], "active")
+ self.assertFalse(record["passed"])
+ self.model(run)
+
+ def test_preexisting_unit_is_never_stopped_or_removed(self):
+ def run(lab, root):
+ sentinel = root / "units" / (native.AGENT + ".service")
+ sentinel.write_text("preexisting unrelated service\n")
+ with self.assertRaisesRegex(ValueError, "already owned"): lab.prepare()
+ lab.cleanup()
+ self.assertEqual(sentinel.read_text(), "preexisting unrelated service\n")
+ self.assertFalse(lab.owned_root)
+ self.model(run)
+
+ def test_failed_cleanup_stays_failed_and_keeps_ownership_for_reconciliation(self):
+ lab = native.Lab("a" * 40, self.installer)
+ lab.owned_units.add(native.AGENT + ".service")
+ with patch.object(lab, "cleanup_units", side_effect=RuntimeError("stop refused")):
+ with self.assertRaises(RuntimeError): lab.cleanup()
+ self.assertFalse(lab.receipt["cleanup"]["complete"])
+ self.assertEqual(lab.owned_units, {native.AGENT + ".service"})
+
+
+if __name__ == "__main__":
+ unittest.main()