Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Retain agent recovery files until systemd teardown is confirmed
pulse-triage[bot]4fce9ba024af
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 127bf0ac6e..341147242a 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -15,6 +15,35 @@## Purpose+### Systemd agent removal retains recovery material until quiescence is observed++Unix agent removal must confirm the collector, action runner and privileged+helper socket/service are inactive and not enabled before server deregistration,+process termination or local erasure. Missing saved connection information still+refuses before stopping a working installation. Each systemd read is one bounded,+combined stdout/stderr observation with exactly LoadState, ActiveState and+UnitFileState, including a completion sentinel so trailing blank diagnostics+cannot be discarded. Failed, incomplete, diagnostic-bearing, duplicate, unknown+or transitional observations refuse removal. Known absent/inactive units are+no-ops; stop and disable success require independent state readback.++The complete affected set is read again after the stop sequence. Shared collector,+helper and runner teardown functions enforce the same boundary for their other+callers. Helper service and socket are reconciled together; runner credential+revocation follows confirmed quiescence and still precedes local deletion.+An unavailable manager cannot authorise deleting an existing runner. No partial+stop is automatically restarted, no new mask is installed, and no trust, credential+scope, non-systemd teardown or Windows behaviour changes.++`TestInstallSHSystemdTeardownRequiresObservedQuiescence` executes the actual shell+functions with closed systemd doubles and only owned temporary deletion targets.+It covers failed/ignored stop and disable, malformed/refused reads, socket and+batch reactivation, authorised teardown and retained failed-revocation material;+the real timeout control confirms its owned child terminates. The existing+saved-state removal controls also require refusal before remote or local mutation.+These are exact-source controls, not native systemd installation, TLS, published+availability or resolution of Linux/PowerShell #1099.+### Unix removal retains credentials when connection recovery is incompleteAfter existing installed-state recovery, the unified Unix installer checks fordiff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.mdindex 90b59db6ed..0f5cf3466c 100644--- a/docs/release-control/v6/internal/subsystems/deployment-installability.md+++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md@@ -15,6 +15,31 @@## Purpose+### Agent systemd teardown refuses unobserved stops and activation sources++`scripts/install.sh` must not discard unit files, helper credentials or runner+recovery state merely because stop/disable was attempted. It uses a five-second+read/stop/disable deadline with a one-second kill bound, requires complete+diagnostic-free combined property reads, and independently confirms inactive,+non-enabled units. Genuine absent/inactive units require no stop or disable.+Known masks remain masks; unknown/transitional/refused or still-enabled state+blocks deletion. The completion sentinel preserves trailing blank diagnostics.++Full uninstall reconciles collector, runner and both helper activation units+before server removal or process mutation, then each shared teardown function+checks its own complete affected set before erasure. Runner revocation still+requires the exact saved credential and a successful durable server result.+Failure retains local recovery files and reports the service-state reconciliation+needed next, without claiming a partial stop was restored or automatically+starting anything. Missing-URL refusal, caller-owned token-file retention,+signature/TLS policy and non-systemd platforms are unchanged.++The registered `scripts/installtests/install_sh_test.go` removal-state artifact+executes `agent_systemd_teardown.py` production-function refusal/ordering controls+and `agent_uninstall_state.py` caller controls. A hanging private systemctl double+also exercises the actual timeout and verifies its child is gone. Native service+and installed/removal acceptance remain separate from these confined controls.+### Unix removal retains credentials when connection recovery is incompleteAfter existing installed-state recovery, the unified Unix installer checks fordiff --git a/scripts/install.sh b/scripts/install.shindex 429bb241f1..601f7be565 100755--- a/scripts/install.sh+++ b/scripts/install.sh@@ -1045,10 +1045,74 @@ restart_sysv_agent_service() {"$initscript" start}+# Destructive systemd teardown needs one complete, diagnostic-free observation,+# not is-active's conflation of absence and a refused manager connection.+read_agent_removal_unit_state() {+ local unit="$1" observation="" line="" load="" active="" enabled=""+ local load_count=0 active_count=0 enabled_count=0 read_completed="false"+ # The sentinel prevents command substitution from discarding trailing blank+ # diagnostics. Success requires the complete three-property response only.+ observation=$(timeout -k 1 5 systemctl show "$unit" --property=LoadState,ActiveState,UnitFileState 2>&1 &&+ printf '%s' '__pulse_state_read_completed__') || return 1+ [[ -n "$observation" ]] || return 1+ while IFS= read -r line; do+ [[ "$read_completed" == "false" ]] || return 1+ case "$line" in+ LoadState=*) load="${line#LoadState=}"; load_count=$((load_count + 1)) ;;+ ActiveState=*) active="${line#ActiveState=}"; active_count=$((active_count + 1)) ;;+ UnitFileState=*) enabled="${line#UnitFileState=}"; enabled_count=$((enabled_count + 1)) ;;+ __pulse_state_read_completed__) read_completed="true" ;;+ *) return 1 ;;+ esac+ done <<< "$observation"+ [[ $load_count -eq 1 && $active_count -eq 1 && $enabled_count -eq 1 && "$read_completed" == "true" ]] || return 1+ if [[ "$load" == not-found ]]; then+ [[ "$active" == inactive && ( -z "$enabled" || "$enabled" == not-found ) ]] || return 1+ printf 'not-found inactive -\n'+ return 0+ fi+ case "$load" in loaded|masked) ;; *) return 1 ;; esac+ case "$active" in active|inactive|failed) ;; *) return 1 ;; esac+ case "$enabled" in+ enabled|enabled-runtime|disabled|static|indirect|linked|linked-runtime|alias|generated|transient|masked|masked-runtime) ;;+ *) return 1 ;;+ esac+ printf '%s %s %s\n' "$load" "$active" "$enabled"+}++confirm_agent_unit_quiescent() {+ local unit="$1" state="" load="" active="" enabled=""+ state=$(read_agent_removal_unit_state "$unit") || return 1+ read -r load active enabled <<< "$state"+ [[ "$active" == inactive ]] || return 1+ [[ "$load" == not-found ]] && return 0+ case "$enabled" in disabled|static|indirect|masked|masked-runtime) return 0 ;; esac+ return 1+}++quiesce_agent_units_for_removal() {+ local unit="" state="" load="" active="" enabled=""+ for unit in "$@"; do+ state=$(read_agent_removal_unit_state "$unit") || return 1+ read -r load active enabled <<< "$state"+ [[ "$load" == not-found ]] && continue+ if [[ "$active" != inactive ]]; then+ timeout -k 1 5 systemctl stop "$unit" >/dev/null 2>&1 || return 1+ fi+ timeout -k 1 5 systemctl disable "$unit" >/dev/null 2>&1 || return 1+ confirm_agent_unit_quiescent "$unit" || return 1+ done+ # Socket activation or another actor can change an earlier target while a+ # later stop is in progress. Re-read the entire affected set before erasure.+ for unit in "$@"; do+ confirm_agent_unit_quiescent "$unit" || return 1+ done+}+teardown_systemd_agent_service() {local unit_path="${1:-/etc/systemd/system/${AGENT_NAME}.service}"- systemctl stop "${AGENT_NAME}" 2>/dev/null || true- systemctl disable "${AGENT_NAME}" 2>/dev/null || true+ quiesce_agent_units_for_removal "${AGENT_NAME}.service" ||+ fail "Agent service removal could not confirm a stopped, disabled unit; files were retained. Reconcile service state before retrying; no automatic restart was attempted." "$EXIT_GENERAL"rm -f "$unit_path"systemctl daemon-reload 2>/dev/null || true}@@ -1057,9 +1121,8 @@ teardown_privileged_helper_service() {if ! command -v systemctl >/dev/null 2>&1; thenreturn 0fi- systemctl stop "${PRIVILEGED_HELPER_NAME}.socket" 2>/dev/null || true- systemctl stop "${PRIVILEGED_HELPER_NAME}.service" 2>/dev/null || true- systemctl disable "${PRIVILEGED_HELPER_NAME}.socket" 2>/dev/null || true+ quiesce_agent_units_for_removal "${PRIVILEGED_HELPER_NAME}.socket" "${PRIVILEGED_HELPER_NAME}.service" ||+ fail "Privileged helper removal could not confirm both socket and service quiescent; credentials and files were retained. Reconcile service state before retrying; no automatic restart was attempted." "$EXIT_GENERAL"rm -f "$PRIVILEGED_HELPER_SOCKET_UNIT" "$PRIVILEGED_HELPER_SERVICE_UNIT"rm -f "$PRIVILEGED_HELPER_SOCKET_PATH"rm -rf "$PRIVILEGED_HELPER_CREDENTIAL_DIR"@@ -1152,8 +1215,10 @@ teardown_action_runner_service() {had_runner_artifact="true"fiif command -v systemctl >/dev/null 2>&1; then- systemctl stop "${ACTION_RUNNER_NAME}.service" 2>/dev/null || true- systemctl disable "${ACTION_RUNNER_NAME}.service" 2>/dev/null || true+ quiesce_agent_units_for_removal "${ACTION_RUNNER_NAME}.service" ||+ fail "Action runner removal could not confirm a stopped, disabled unit; credentials and files were retained. Reconcile service state before retrying; no automatic restart was attempted." "$EXIT_GENERAL"+ elif [[ "$had_runner_artifact" == "true" ]]; then+ fail "Action runner removal requires observable systemd state; credentials and files were retained." "$EXIT_GENERAL"fiif [[ "$had_runner_artifact" == "true" ]]; thenif revoke_action_runner_credential; then@@ -5528,14 +5593,27 @@ finalize_plist_env_blockif [[ "$UNINSTALL" == "true" ]]; thenlog_info "Uninstalling ${AGENT_NAME} and cleaning up legacy agents..."local qnap_state_dir=""+ local uninstall_has_credential="false"- # A credential-bearing install must durably remove its exact server record- # through the same CA/fingerprint/no-proxy transport used by lifecycle- # migration before any local credential or service state is deleted.+ # Validate saved connection availability before stopping a working install.+ # Stop every systemd activation source before remote removal, process kills+ # or any local teardown. Unknown/partial stops retain all recovery material.if collector_credential_state_present; then+ uninstall_has_credential="true"if [[ -z "$PULSE_URL" ]]; thenfail "The saved Pulse URL is missing; local credentials and services were retained. Restore saved connection state or provide --url for the same trusted Pulse instance and retry uninstall." "$EXIT_MISSING_ARGS"fi+ fi+ if command -v systemctl >/dev/null 2>&1; then+ quiesce_agent_units_for_removal "${ACTION_RUNNER_NAME}.service" \+ "${PRIVILEGED_HELPER_NAME}.socket" "${PRIVILEGED_HELPER_NAME}.service" "${AGENT_NAME}.service" ||+ fail "Agent removal could not confirm every systemd unit stopped and disabled; local credentials and files were retained. Reconcile service state before retrying; no automatic restart was attempted." "$EXIT_GENERAL"+ fi++ # A credential-bearing install must durably remove its exact server record+ # through the same CA/fingerprint/no-proxy transport used by lifecycle+ # migration before any local credential or service state is deleted.+ if [[ "$uninstall_has_credential" == "true" ]]; thenlog_info "Authenticating Pulse server removal before local teardown..."if ! uninstall_collector_registration; thenfail "Pulse did not durably confirm collector removal; local credentials and services were retained. Restore trusted server connectivity and retry uninstall." "$EXIT_GENERAL"diff --git a/scripts/installtests/agent_systemd_teardown.py b/scripts/installtests/agent_systemd_teardown.pynew file mode 100644index 0000000000..8e0c80a8c0--- /dev/null+++ b/scripts/installtests/agent_systemd_teardown.py@@ -0,0 +1,261 @@+#!/usr/bin/env python3+"""Production agent teardown against private systemd doubles, never host units.++The real removal functions delete only owned temporary fixture files. All+systemd/process/credential transport operations are closed doubles. These+controls establish refusal and ordering, not native systemd or installed use.+"""++import os+from pathlib import Path+import shlex+import subprocess+import tempfile+import time+import unittest++ROOT = Path(__file__).resolve().parents[2]+SOURCE = Path(os.environ.get("PULSE_AGENT_TEARDOWN_SOURCE", ROOT / "scripts/install.sh"))+++def function(source, name):+ marker = name + "() {"+ return marker + source.split(marker, 1)[1].split("\n}\n", 1)[0] + "\n}\n"+++DOUBLE = r'''+set -euo pipefail+EXIT_GENERAL=1 AGENT_NAME=pulse-agent+PRIVILEGED_HELPER_NAME=pulse-agent-helper ACTION_RUNNER_NAME=pulse-agent-runner+log_info() { :; }+log_error() { printf '%s\n' "$*"; }+revoke_action_runner_credential() { printf 'revoke\n' >> "$EVENTS"; [[ "$FAULT" != revoke-refused ]]; }+# timeout is exercised separately by a hanging child. All other controls call+# this closed systemctl double, and never invoke a real service manager.+timeout() {+ [[ "$1 $2 $3 $4" == '-k 1 5 systemctl' ]] || return 95+ shift 4+ systemctl "$@"+}+systemctl() {+ local verb="$1" unit="${2:-}" load active enabled+ case "$unit" in pulse-agent.service|pulse-agent-helper.socket|pulse-agent-helper.service|pulse-agent-runner.service) ;;+ *) [[ "$verb" == daemon-reload ]] || return 96 ;;+ esac+ printf '%s:%s\n' "$verb" "$unit" >> "$EVENTS"+ case "$verb" in+ show)+ [[ $# == 3 && "$3" == --property=LoadState,ActiveState,UnitFileState ]] || return 97+ if [[ "$unit" == "$TARGET" ]]; then+ case "$FAULT" in+ read-refused) printf 'private diagnostic\n' >&2; return 1 ;;+ read-refused-after-stop) [[ ! -f "$ROOT/stopped" ]] || return 1 ;;+ read-empty) return 0 ;;+ read-blank-diagnostic) printf '\n' >&2 ;;+ read-diagnostic) printf 'private diagnostic\n' >&2 ;;+ read-duplicate) printf 'ActiveState=inactive\n' ;;+ read-missing) printf 'LoadState=loaded\nActiveState=inactive\n'; return 0 ;;+ esac+ fi+ read -r load active enabled < "$ROOT/$unit"+ [[ "$enabled" != - ]] || enabled=""+ printf 'UnitFileState=%s\nActiveState=%s\nLoadState=%s\n' "$enabled" "$active" "$load"+ [[ "$FAULT" != read-trailing-blank-diagnostic || "$unit" != "$TARGET" ]] || printf '\n' >&2+ ;;+ stop)+ if [[ "$unit" == "$TARGET" ]]; then+ [[ "$FAULT" != stop-refused ]] || return 1+ [[ "$FAULT" != stop-timeout ]] || return 124+ [[ "$FAULT" != ignored-stop ]] || return 0+ fi+ read -r load active enabled < "$ROOT/$unit"+ printf '%s inactive %s\n' "$load" "$enabled" > "$ROOT/$unit"+ touch "$ROOT/stopped"+ # Earlier activation sources changing during a later stop are not+ # safe just because each individual stop once succeeded.+ if [[ "$unit" == pulse-agent-helper.service ]]; then+ case "$FAULT" in+ socket-reactivated) printf 'loaded active disabled\n' > "$ROOT/pulse-agent-helper.socket" ;;+ socket-reenabled) printf 'loaded inactive enabled\n' > "$ROOT/pulse-agent-helper.socket" ;;+ esac+ fi+ if [[ "$unit" == pulse-agent.service && "$FAULT" == runner-reactivated ]]; then+ printf 'loaded active disabled\n' > "$ROOT/pulse-agent-runner.service"+ fi+ ;;+ disable)+ if [[ "$unit" == "$TARGET" ]]; then+ [[ "$FAULT" != disable-refused ]] || return 1+ [[ "$FAULT" != ignored-disable ]] || return 0+ fi+ read -r load active enabled < "$ROOT/$unit"+ case "$enabled" in masked|masked-runtime|static|indirect) ;;+ *) enabled=disabled ;;+ esac+ printf '%s %s %s\n' "$load" "$active" "$enabled" > "$ROOT/$unit"+ ;;+ daemon-reload|reset-failed) return 0 ;;+ *) return 98 ;;+ esac+}+rm() {+ local path+ [[ "$1" == -f || "$1" == -rf ]] || return 99+ for path in "${@:2}"; do+ case "$path" in "$ROOT/artifacts/"*) ;;+ *) printf 'outside fixture deletion refused\n' >&2; return 100 ;;+ esac+ done+ printf 'delete\n' >> "$EVENTS"+ command rm "$@"+}+'''+++class AgentSystemdTeardown(unittest.TestCase):+ source = SOURCE.read_text()++ def exercise(self, family, fault="", state="loaded active enabled"):+ with tempfile.TemporaryDirectory() as temporary:+ root = Path(temporary)+ artifacts = root / "artifacts"+ artifacts.mkdir()+ paths = {+ "PRIVILEGED_HELPER_SOCKET_UNIT": artifacts / "helper.socket",+ "PRIVILEGED_HELPER_SERVICE_UNIT": artifacts / "helper.service",+ "PRIVILEGED_HELPER_SOCKET_PATH": artifacts / "helper.sock",+ "PRIVILEGED_HELPER_CREDENTIAL_DIR": artifacts / "helper-credentials",+ "ACTION_RUNNER_BINARY_PATH": artifacts / "runner",+ "ACTION_RUNNER_SERVICE_UNIT": artifacts / "runner.service",+ "ACTION_RUNNER_CONFIG_DIR": artifacts / "runner-config",+ "ACTION_RUNNER_STATE_DIR": artifacts / "runner-state",+ }+ for name, path in paths.items():+ if name.endswith("DIR"):+ path.mkdir()+ (path / "token").write_text("synthetic-recovery-credential\n")+ else:+ path.write_text("owned recovery material\n")+ agent_unit = artifacts / "agent.service"+ agent_unit.write_text("owned agent unit\n")+ unrelated = root / "unrelated"+ unrelated.write_text("keep\n")+ before = {str(p.relative_to(artifacts)): p.read_bytes()+ for p in artifacts.rglob("*") if p.is_file()}+ target = {"agent": "pulse-agent.service", "helper": "pulse-agent-helper.socket",+ "runner": "pulse-agent-runner.service", "batch": "pulse-agent.service"}[family]+ for unit in ("pulse-agent.service", "pulse-agent-helper.socket",+ "pulse-agent-helper.service", "pulse-agent-runner.service"):+ (root / unit).write_text(state + "\n")+ values = dict(paths, ROOT=root, EVENTS=root / "events", FAULT=fault, TARGET=target)+ setup = "\n".join(f"{key}={shlex.quote(str(value))}" for key, value in values.items())+ names = ("read_agent_removal_unit_state", "confirm_agent_unit_quiescent",+ "quiesce_agent_units_for_removal", "teardown_systemd_agent_service",+ "teardown_privileged_helper_service", "teardown_action_runner_service")+ # The same controls can reproduce the older source, which has no+ # observation helper: never insert a substitute implementation.+ functions = "\n".join(function(self.source, name) for name in names+ if name + "() {" in self.source)+ calls = {"agent": "teardown_systemd_agent_service " + shlex.quote(str(agent_unit)),+ "helper": "teardown_privileged_helper_service",+ "runner": "teardown_action_runner_service",+ "batch": "quiesce_agent_units_for_removal pulse-agent-runner.service pulse-agent-helper.socket pulse-agent-helper.service pulse-agent.service"}+ # Production fail() exits even if the caller's errexit is disabled.+ functions += "\n" + function(self.source, "fail")+ shell = DOUBLE + "\nOUTPUT_FORMAT=text\n" + setup + "\n" + functions + "\n" + calls[family]+ result = subprocess.run(["bash", "-c", shell], capture_output=True,+ text=True, timeout=8)+ events = (root / "events").read_text().splitlines() if (root / "events").exists() else []+ after = {str(p.relative_to(artifacts)): p.read_bytes()+ for p in artifacts.rglob("*") if p.is_file()}+ self.assertEqual(unrelated.read_text(), "keep\n")+ return result, events, before, after++ def test_unknown_read_stop_and_disable_retain_every_artifact(self):+ faults = ("read-refused", "read-empty", "read-blank-diagnostic", "read-diagnostic",+ "read-trailing-blank-diagnostic", "read-duplicate", "read-missing", "stop-refused", "stop-timeout",+ "ignored-stop", "disable-refused", "ignored-disable", "read-refused-after-stop")+ for family in ("agent", "helper", "runner"):+ for fault in faults:+ with self.subTest(family=family, fault=fault):+ result, events, before, after = self.exercise(family, fault)+ self.assertNotEqual(result.returncode, 0, result.stdout + result.stderr)+ self.assertEqual(after, before)+ self.assertNotIn("delete", events)+ self.assertNotIn("revoke", events)+ self.assertNotIn("private diagnostic", result.stdout + result.stderr)++ def test_unsettled_or_unknown_states_never_authorise_deletion(self):+ for state in ("error active enabled", "loaded activating enabled", "loaded deactivating enabled",+ "loaded reloading enabled", "loaded unknown enabled", "loaded inactive unknown",+ "not-found active -", "not-found inactive enabled"):+ for family in ("agent", "helper", "runner"):+ with self.subTest(family=family, state=state):+ result, events, before, after = self.exercise(family, state=state)+ self.assertNotEqual(result.returncode, 0, result.stdout + result.stderr)+ self.assertEqual(after, before)+ self.assertNotIn("delete", events)++ def test_helper_socket_and_service_are_reconciled_together(self):+ for fault in ("socket-reactivated", "socket-reenabled"):+ with self.subTest(fault=fault):+ result, events, before, after = self.exercise("helper", fault)+ self.assertNotEqual(result.returncode, 0, result.stdout + result.stderr)+ self.assertEqual(after, before)+ self.assertNotIn("delete", events)++ def test_known_quiescence_allows_only_owned_teardown(self):+ for family in ("agent", "helper", "runner"):+ for state in ("loaded active enabled", "loaded failed enabled", "loaded inactive disabled",+ "loaded inactive static", "loaded inactive indirect", "masked inactive masked",+ "masked inactive masked-runtime", "not-found inactive -"):+ with self.subTest(family=family, state=state):+ result, events, before, after = self.exercise(family, state=state)+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)+ self.assertLess(len(after), len(before))+ first_delete = events.index("delete")+ self.assertTrue(any(event.startswith("show:") for event in events[:first_delete]))+ if family == "runner":+ self.assertLess(events.index("revoke"), first_delete)+ self.assertTrue(any(event.startswith("show:") for event in events[:events.index("revoke")]))+ if state.startswith("not-found"):+ self.assertFalse(any(event.startswith(("stop:", "disable:")) for event in events))++ def test_failed_revocation_retains_stopped_runner_recovery_material(self):+ result, events, before, after = self.exercise("runner", "revoke-refused")+ self.assertNotEqual(result.returncode, 0)+ self.assertIn("revoke", events)+ self.assertNotIn("delete", events)+ self.assertEqual(after, before)++ def test_full_batch_reconciles_earlier_activation_sources(self):+ for fault in ("", "runner-reactivated", "stop-refused"):+ with self.subTest(fault=fault):+ result, events, before, after = self.exercise("batch", fault)+ self.assertEqual(result.returncode == 0, fault == "", result.stdout + result.stderr)+ self.assertEqual(after, before)+ self.assertNotIn("delete", events)+ self.assertNotIn("revoke", events)++ def test_real_read_timeout_stops_its_owned_child(self):+ with tempfile.TemporaryDirectory() as temporary:+ root = Path(temporary)+ executable = root / "systemctl"+ child_id = root / "child-id"+ executable.write_text("#!/bin/bash\nprintf '%s\\n' \"$$\" > \"$CHILD_ID\"\nexec sleep 60\n")+ executable.chmod(0o700)+ env = dict(os.environ, PATH=str(root) + ":/usr/bin:/bin", CHILD_ID=str(child_id))+ started = time.monotonic()+ result = subprocess.run(["bash", "-c", function(self.source, "read_agent_removal_unit_state") ++ "\nread_agent_removal_unit_state pulse-agent.service\n"],+ env=env, capture_output=True, text=True, timeout=8)+ self.assertNotEqual(result.returncode, 0, result.stdout + result.stderr)+ self.assertLess(time.monotonic() - started, 7)+ self.assertEqual(result.stdout, "")+ self.assertTrue(child_id.exists(), "the bounded read never started")+ with self.assertRaises(ProcessLookupError):+ os.kill(int(child_id.read_text()), 0)+++if __name__ == "__main__":+ unittest.main(verbosity=2)diff --git a/scripts/installtests/agent_uninstall_state.py b/scripts/installtests/agent_uninstall_state.pyindex 0f2b20e35a..029e6a8fe0 100644--- a/scripts/installtests/agent_uninstall_state.py+++ b/scripts/installtests/agent_uninstall_state.py@@ -42,7 +42,7 @@ class UnixUninstallState(unittest.TestCase):def exercise(self, credential="runtime", url="", reply="confirmed", authority=True,input_token=False, interactive=False, root_refusal=False, action="uninstall",- ingress_only=False):+ ingress_only=False, quiescence_refused=False):source = self.source# Keep the real entry and all initial credential/service/state teardown.# Stop before unrelated platform boot/log cleanup; never run it on a host.@@ -104,7 +104,8 @@ class UnixUninstallState(unittest.TestCase):"SAFE_PROFILE_STATE_DIR_REMOVAL_AUTHORITY": profile,"PULSE_TOKEN": token, "RUNTIME_TOKEN_FILE": selected,"PULSE_URL": url, "AGENT_ID": "owned-agent", "HOSTNAME_OVERRIDE": "owned-host",- "EVENTS": root / "events", "SERVICE": service, "REPLY": reply}+ "EVENTS": root / "events", "SERVICE": service, "REPLY": reply,+ "QUIESCENCE_REFUSED": str(quiescence_refused).lower()}setup = "\n".join(f"{key}={shlex.quote(str(value))}" for key, value in values.items())functions = "\n".join(function(source, name) for name in names)if input_token:@@ -113,6 +114,7 @@ class UnixUninstallState(unittest.TestCase):shell = r'''set -euo pipefailUNINSTALL=true AGENT_NAME=pulse-agent EXIT_GENERAL=1 EXIT_MISSING_ARGS=16+ACTION_RUNNER_NAME=pulse-agent-runner PRIVILEGED_HELPER_NAME=pulse-agent-helperOUTPUT_FORMAT=text NON_INTERACTIVE=trueTOKEN_FILE_PATH="" UNINSTALL_ACTION_RUNNER=false UPDATE_ONLY=falseRETARGET_ONLY=false PREFLIGHT_ONLY=false SAFE_PROFILE_ACTION=""@@ -143,6 +145,12 @@ run_collector_lifecycle_command() {pkill() { printf 'process-stop\n' >> "$EVENTS"; }sleep() { :; }systemctl() { return 91; }+# This fixture checks the caller's refusal gate, not systemd observations.+# agent_systemd_teardown.py executes that real boundary independently.+quiesce_agent_units_for_removal() {+ [[ "$*" == 'pulse-agent-runner.service pulse-agent-helper.socket pulse-agent-helper.service pulse-agent.service' ]] || return 90+ [[ "$QUIESCENCE_REFUSED" == false ]]+}teardown_action_runner_service() { printf 'runner-stop\n' >> "$EVENTS"; }teardown_privileged_helper_service() { printf 'helper-stop\n' >> "$EVENTS"; }teardown_systemd_agent_service() { printf 'service-stop\n' >> "$EVENTS"; printf 'stopped\n' > "$SERVICE"; }@@ -190,6 +198,16 @@ rm() {self.assertEqual(events, "request:collector-uninstall --agent-id owned-agent --hostname owned-host\n")self.assertEqual(after, before)+ def test_systemd_refusal_precedes_remote_removal_and_every_local_mutation(self):+ for credential in ("runtime", "inline", "none"):+ with self.subTest(credential=credential):+ result, events, before, after = self.exercise(+ credential, url="https://pulse.invalid", quiescence_refused=True)+ self.assertEqual(result.returncode, 1, result.stdout + result.stderr)+ self.assertEqual(events, "")+ self.assertEqual(after, before)+ self.assertIn("every systemd unit stopped and disabled", result.stdout)+def test_exact_confirmation_precedes_process_service_and_state_teardown(self):result, events, before, after = self.exercise(url="https://pulse.invalid")self.assertEqual(result.returncode, 0, result.stderr)diff --git a/scripts/installtests/install_sh_test.go b/scripts/installtests/install_sh_test.goindex e2b05197ef..5310f23c08 100644--- a/scripts/installtests/install_sh_test.go+++ b/scripts/installtests/install_sh_test.go@@ -8286,6 +8286,7 @@ func TestInstallSHUninstallRetainsCredentialsWhenSavedURLIsMissing(t *testing.T)for _, name := range []string{"test_missing_url_retains_every_credential_representation_without_a_request","test_server_refusal_unknown_or_other_identity_retains_service_and_state",+ "test_systemd_refusal_precedes_remote_removal_and_every_local_mutation","test_exact_confirmation_precedes_process_service_and_state_teardown","test_credential_free_local_only_removal_needs_no_url_or_request","test_server_confirmation_does_not_grant_unknown_path_removal",@@ -8300,8 +8301,45 @@ func TestInstallSHUninstallRetainsCredentialsWhenSavedURLIsMissing(t *testing.T)t.Fatalf("Unix removal control %s did not pass:\n%s", name, output)}}- if !strings.Contains(string(output), "\nRan 11 tests in ") || !strings.Contains(string(output), "\nOK\n") {+ if !strings.Contains(string(output), "\nRan 12 tests in ") || !strings.Contains(string(output), "\nOK\n") {t.Fatalf("incomplete Unix removal state verdict:\n%s", output)}t.Logf("%s", output)}++func TestInstallSHSystemdTeardownRequiresObservedQuiescence(t *testing.T) {+ ctx, cancel := context.WithTimeout(context.Background(), time.Minute)+ defer cancel()+ cmd := exec.CommandContext(ctx, "python3", repoFile("scripts", "installtests", "agent_systemd_teardown.py"))+ output, err := cmd.CombinedOutput()+ if err != nil {+ t.Fatalf("systemd teardown refusal controls: %v\n%s", err, output)+ }+ for _, name := range []string{+ "test_unknown_read_stop_and_disable_retain_every_artifact",+ "test_unsettled_or_unknown_states_never_authorise_deletion",+ "test_helper_socket_and_service_are_reconciled_together",+ "test_known_quiescence_allows_only_owned_teardown",+ "test_failed_revocation_retains_stopped_runner_recovery_material",+ "test_full_batch_reconciles_earlier_activation_sources",+ "test_real_read_timeout_stops_its_owned_child",+ } {+ if !strings.Contains(string(output), name+" (__main__.AgentSystemdTeardown."+name+") ... ok") {+ t.Fatalf("systemd teardown control %s did not pass:\n%s", name, output)+ }+ }+ if !strings.Contains(string(output), "\nRan 7 tests in ") || !strings.Contains(string(output), "\nOK\n") {+ t.Fatalf("incomplete systemd teardown verdict:\n%s", output)+ }+ entry, err := os.ReadFile(repoFile("scripts", "install.sh"))+ if err != nil {+ t.Fatal(err)+ }+ uninstall := strings.SplitN(string(entry), "# --- Uninstall Logic ---", 2)[1]+ quiesce := strings.Index(uninstall, "quiesce_agent_units_for_removal")+ if quiesce < 0 || quiesce >= strings.Index(uninstall, "if ! uninstall_collector_registration;") ||+ quiesce >= strings.Index(uninstall, "pkill -f") {+ t.Fatal("full removal must quiesce every affected systemd unit before remote removal or process mutation")+ }+ t.Logf("%s", output)+}