Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Retain agent recovery files until systemd teardown is confirmed

pulse-triage[bot]4fce9ba024af
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 127bf0ac6e..341147242a 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -15,6 +15,35 @@
## Purpose
+### Systemd agent removal retains recovery material until quiescence is observed
+
+Unix agent removal must confirm the collector, action runner and privileged
+helper socket/service are inactive and not enabled before server deregistration,
+process termination or local erasure. Missing saved connection information still
+refuses before stopping a working installation. Each systemd read is one bounded,
+combined stdout/stderr observation with exactly LoadState, ActiveState and
+UnitFileState, including a completion sentinel so trailing blank diagnostics
+cannot be discarded. Failed, incomplete, diagnostic-bearing, duplicate, unknown
+or transitional observations refuse removal. Known absent/inactive units are
+no-ops; stop and disable success require independent state readback.
+
+The complete affected set is read again after the stop sequence. Shared collector,
+helper and runner teardown functions enforce the same boundary for their other
+callers. Helper service and socket are reconciled together; runner credential
+revocation follows confirmed quiescence and still precedes local deletion.
+An unavailable manager cannot authorise deleting an existing runner. No partial
+stop is automatically restarted, no new mask is installed, and no trust, credential
+scope, non-systemd teardown or Windows behaviour changes.
+
+`TestInstallSHSystemdTeardownRequiresObservedQuiescence` executes the actual shell
+functions with closed systemd doubles and only owned temporary deletion targets.
+It covers failed/ignored stop and disable, malformed/refused reads, socket and
+batch reactivation, authorised teardown and retained failed-revocation material;
+the real timeout control confirms its owned child terminates. The existing
+saved-state removal controls also require refusal before remote or local mutation.
+These are exact-source controls, not native systemd installation, TLS, published
+availability or resolution of Linux/PowerShell #1099.
+
### Unix removal retains credentials when connection recovery is incomplete
After existing installed-state recovery, the unified Unix installer checks for
diff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.md
index 90b59db6ed..0f5cf3466c 100644
--- a/docs/release-control/v6/internal/subsystems/deployment-installability.md
+++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md
@@ -15,6 +15,31 @@
## Purpose
+### Agent systemd teardown refuses unobserved stops and activation sources
+
+`scripts/install.sh` must not discard unit files, helper credentials or runner
+recovery state merely because stop/disable was attempted. It uses a five-second
+read/stop/disable deadline with a one-second kill bound, requires complete
+diagnostic-free combined property reads, and independently confirms inactive,
+non-enabled units. Genuine absent/inactive units require no stop or disable.
+Known masks remain masks; unknown/transitional/refused or still-enabled state
+blocks deletion. The completion sentinel preserves trailing blank diagnostics.
+
+Full uninstall reconciles collector, runner and both helper activation units
+before server removal or process mutation, then each shared teardown function
+checks its own complete affected set before erasure. Runner revocation still
+requires the exact saved credential and a successful durable server result.
+Failure retains local recovery files and reports the service-state reconciliation
+needed next, without claiming a partial stop was restored or automatically
+starting anything. Missing-URL refusal, caller-owned token-file retention,
+signature/TLS policy and non-systemd platforms are unchanged.
+
+The registered `scripts/installtests/install_sh_test.go` removal-state artifact
+executes `agent_systemd_teardown.py` production-function refusal/ordering controls
+and `agent_uninstall_state.py` caller controls. A hanging private systemctl double
+also exercises the actual timeout and verifies its child is gone. Native service
+and installed/removal acceptance remain separate from these confined controls.
+
### Unix removal retains credentials when connection recovery is incomplete
After existing installed-state recovery, the unified Unix installer checks for
diff --git a/scripts/install.sh b/scripts/install.sh
index 429bb241f1..601f7be565 100755
--- a/scripts/install.sh
+++ b/scripts/install.sh
@@ -1045,10 +1045,74 @@ restart_sysv_agent_service() {
"$initscript" start
}
+# Destructive systemd teardown needs one complete, diagnostic-free observation,
+# not is-active's conflation of absence and a refused manager connection.
+read_agent_removal_unit_state() {
+ local unit="$1" observation="" line="" load="" active="" enabled=""
+ local load_count=0 active_count=0 enabled_count=0 read_completed="false"
+ # The sentinel prevents command substitution from discarding trailing blank
+ # diagnostics. Success requires the complete three-property response only.
+ observation=$(timeout -k 1 5 systemctl show "$unit" --property=LoadState,ActiveState,UnitFileState 2>&1 &&
+ printf '%s' '__pulse_state_read_completed__') || return 1
+ [[ -n "$observation" ]] || return 1
+ while IFS= read -r line; do
+ [[ "$read_completed" == "false" ]] || return 1
+ case "$line" in
+ LoadState=*) load="${line#LoadState=}"; load_count=$((load_count + 1)) ;;
+ ActiveState=*) active="${line#ActiveState=}"; active_count=$((active_count + 1)) ;;
+ UnitFileState=*) enabled="${line#UnitFileState=}"; enabled_count=$((enabled_count + 1)) ;;
+ __pulse_state_read_completed__) read_completed="true" ;;
+ *) return 1 ;;
+ esac
+ done <<< "$observation"
+ [[ $load_count -eq 1 && $active_count -eq 1 && $enabled_count -eq 1 && "$read_completed" == "true" ]] || return 1
+ if [[ "$load" == not-found ]]; then
+ [[ "$active" == inactive && ( -z "$enabled" || "$enabled" == not-found ) ]] || return 1
+ printf 'not-found inactive -\n'
+ return 0
+ fi
+ case "$load" in loaded|masked) ;; *) return 1 ;; esac
+ case "$active" in active|inactive|failed) ;; *) return 1 ;; esac
+ case "$enabled" in
+ enabled|enabled-runtime|disabled|static|indirect|linked|linked-runtime|alias|generated|transient|masked|masked-runtime) ;;
+ *) return 1 ;;
+ esac
+ printf '%s %s %s\n' "$load" "$active" "$enabled"
+}
+
+confirm_agent_unit_quiescent() {
+ local unit="$1" state="" load="" active="" enabled=""
+ state=$(read_agent_removal_unit_state "$unit") || return 1
+ read -r load active enabled <<< "$state"
+ [[ "$active" == inactive ]] || return 1
+ [[ "$load" == not-found ]] && return 0
+ case "$enabled" in disabled|static|indirect|masked|masked-runtime) return 0 ;; esac
+ return 1
+}
+
+quiesce_agent_units_for_removal() {
+ local unit="" state="" load="" active="" enabled=""
+ for unit in "$@"; do
+ state=$(read_agent_removal_unit_state "$unit") || return 1
+ read -r load active enabled <<< "$state"
+ [[ "$load" == not-found ]] && continue
+ if [[ "$active" != inactive ]]; then
+ timeout -k 1 5 systemctl stop "$unit" >/dev/null 2>&1 || return 1
+ fi
+ timeout -k 1 5 systemctl disable "$unit" >/dev/null 2>&1 || return 1
+ confirm_agent_unit_quiescent "$unit" || return 1
+ done
+ # Socket activation or another actor can change an earlier target while a
+ # later stop is in progress. Re-read the entire affected set before erasure.
+ for unit in "$@"; do
+ confirm_agent_unit_quiescent "$unit" || return 1
+ done
+}
+
teardown_systemd_agent_service() {
local unit_path="${1:-/etc/systemd/system/${AGENT_NAME}.service}"
- systemctl stop "${AGENT_NAME}" 2>/dev/null || true
- systemctl disable "${AGENT_NAME}" 2>/dev/null || true
+ quiesce_agent_units_for_removal "${AGENT_NAME}.service" ||
+ fail "Agent service removal could not confirm a stopped, disabled unit; files were retained. Reconcile service state before retrying; no automatic restart was attempted." "$EXIT_GENERAL"
rm -f "$unit_path"
systemctl daemon-reload 2>/dev/null || true
}
@@ -1057,9 +1121,8 @@ teardown_privileged_helper_service() {
if ! command -v systemctl >/dev/null 2>&1; then
return 0
fi
- systemctl stop "${PRIVILEGED_HELPER_NAME}.socket" 2>/dev/null || true
- systemctl stop "${PRIVILEGED_HELPER_NAME}.service" 2>/dev/null || true
- systemctl disable "${PRIVILEGED_HELPER_NAME}.socket" 2>/dev/null || true
+ quiesce_agent_units_for_removal "${PRIVILEGED_HELPER_NAME}.socket" "${PRIVILEGED_HELPER_NAME}.service" ||
+ fail "Privileged helper removal could not confirm both socket and service quiescent; credentials and files were retained. Reconcile service state before retrying; no automatic restart was attempted." "$EXIT_GENERAL"
rm -f "$PRIVILEGED_HELPER_SOCKET_UNIT" "$PRIVILEGED_HELPER_SERVICE_UNIT"
rm -f "$PRIVILEGED_HELPER_SOCKET_PATH"
rm -rf "$PRIVILEGED_HELPER_CREDENTIAL_DIR"
@@ -1152,8 +1215,10 @@ teardown_action_runner_service() {
had_runner_artifact="true"
fi
if command -v systemctl >/dev/null 2>&1; then
- systemctl stop "${ACTION_RUNNER_NAME}.service" 2>/dev/null || true
- systemctl disable "${ACTION_RUNNER_NAME}.service" 2>/dev/null || true
+ quiesce_agent_units_for_removal "${ACTION_RUNNER_NAME}.service" ||
+ fail "Action runner removal could not confirm a stopped, disabled unit; credentials and files were retained. Reconcile service state before retrying; no automatic restart was attempted." "$EXIT_GENERAL"
+ elif [[ "$had_runner_artifact" == "true" ]]; then
+ fail "Action runner removal requires observable systemd state; credentials and files were retained." "$EXIT_GENERAL"
fi
if [[ "$had_runner_artifact" == "true" ]]; then
if revoke_action_runner_credential; then
@@ -5528,14 +5593,27 @@ finalize_plist_env_block
if [[ "$UNINSTALL" == "true" ]]; then
log_info "Uninstalling ${AGENT_NAME} and cleaning up legacy agents..."
local qnap_state_dir=""
+ local uninstall_has_credential="false"
- # A credential-bearing install must durably remove its exact server record
- # through the same CA/fingerprint/no-proxy transport used by lifecycle
- # migration before any local credential or service state is deleted.
+ # Validate saved connection availability before stopping a working install.
+ # Stop every systemd activation source before remote removal, process kills
+ # or any local teardown. Unknown/partial stops retain all recovery material.
if collector_credential_state_present; then
+ uninstall_has_credential="true"
if [[ -z "$PULSE_URL" ]]; then
fail "The saved Pulse URL is missing; local credentials and services were retained. Restore saved connection state or provide --url for the same trusted Pulse instance and retry uninstall." "$EXIT_MISSING_ARGS"
fi
+ fi
+ if command -v systemctl >/dev/null 2>&1; then
+ quiesce_agent_units_for_removal "${ACTION_RUNNER_NAME}.service" \
+ "${PRIVILEGED_HELPER_NAME}.socket" "${PRIVILEGED_HELPER_NAME}.service" "${AGENT_NAME}.service" ||
+ fail "Agent removal could not confirm every systemd unit stopped and disabled; local credentials and files were retained. Reconcile service state before retrying; no automatic restart was attempted." "$EXIT_GENERAL"
+ fi
+
+ # A credential-bearing install must durably remove its exact server record
+ # through the same CA/fingerprint/no-proxy transport used by lifecycle
+ # migration before any local credential or service state is deleted.
+ if [[ "$uninstall_has_credential" == "true" ]]; then
log_info "Authenticating Pulse server removal before local teardown..."
if ! uninstall_collector_registration; then
fail "Pulse did not durably confirm collector removal; local credentials and services were retained. Restore trusted server connectivity and retry uninstall." "$EXIT_GENERAL"
diff --git a/scripts/installtests/agent_systemd_teardown.py b/scripts/installtests/agent_systemd_teardown.py
new file mode 100644
index 0000000000..8e0c80a8c0
--- /dev/null
+++ b/scripts/installtests/agent_systemd_teardown.py
@@ -0,0 +1,261 @@
+#!/usr/bin/env python3
+"""Production agent teardown against private systemd doubles, never host units.
+
+The real removal functions delete only owned temporary fixture files. All
+systemd/process/credential transport operations are closed doubles. These
+controls establish refusal and ordering, not native systemd or installed use.
+"""
+
+import os
+from pathlib import Path
+import shlex
+import subprocess
+import tempfile
+import time
+import unittest
+
+ROOT = Path(__file__).resolve().parents[2]
+SOURCE = Path(os.environ.get("PULSE_AGENT_TEARDOWN_SOURCE", ROOT / "scripts/install.sh"))
+
+
+def function(source, name):
+ marker = name + "() {"
+ return marker + source.split(marker, 1)[1].split("\n}\n", 1)[0] + "\n}\n"
+
+
+DOUBLE = r'''
+set -euo pipefail
+EXIT_GENERAL=1 AGENT_NAME=pulse-agent
+PRIVILEGED_HELPER_NAME=pulse-agent-helper ACTION_RUNNER_NAME=pulse-agent-runner
+log_info() { :; }
+log_error() { printf '%s\n' "$*"; }
+revoke_action_runner_credential() { printf 'revoke\n' >> "$EVENTS"; [[ "$FAULT" != revoke-refused ]]; }
+# timeout is exercised separately by a hanging child. All other controls call
+# this closed systemctl double, and never invoke a real service manager.
+timeout() {
+ [[ "$1 $2 $3 $4" == '-k 1 5 systemctl' ]] || return 95
+ shift 4
+ systemctl "$@"
+}
+systemctl() {
+ local verb="$1" unit="${2:-}" load active enabled
+ case "$unit" in pulse-agent.service|pulse-agent-helper.socket|pulse-agent-helper.service|pulse-agent-runner.service) ;;
+ *) [[ "$verb" == daemon-reload ]] || return 96 ;;
+ esac
+ printf '%s:%s\n' "$verb" "$unit" >> "$EVENTS"
+ case "$verb" in
+ show)
+ [[ $# == 3 && "$3" == --property=LoadState,ActiveState,UnitFileState ]] || return 97
+ if [[ "$unit" == "$TARGET" ]]; then
+ case "$FAULT" in
+ read-refused) printf 'private diagnostic\n' >&2; return 1 ;;
+ read-refused-after-stop) [[ ! -f "$ROOT/stopped" ]] || return 1 ;;
+ read-empty) return 0 ;;
+ read-blank-diagnostic) printf '\n' >&2 ;;
+ read-diagnostic) printf 'private diagnostic\n' >&2 ;;
+ read-duplicate) printf 'ActiveState=inactive\n' ;;
+ read-missing) printf 'LoadState=loaded\nActiveState=inactive\n'; return 0 ;;
+ esac
+ fi
+ read -r load active enabled < "$ROOT/$unit"
+ [[ "$enabled" != - ]] || enabled=""
+ printf 'UnitFileState=%s\nActiveState=%s\nLoadState=%s\n' "$enabled" "$active" "$load"
+ [[ "$FAULT" != read-trailing-blank-diagnostic || "$unit" != "$TARGET" ]] || printf '\n' >&2
+ ;;
+ stop)
+ if [[ "$unit" == "$TARGET" ]]; then
+ [[ "$FAULT" != stop-refused ]] || return 1
+ [[ "$FAULT" != stop-timeout ]] || return 124
+ [[ "$FAULT" != ignored-stop ]] || return 0
+ fi
+ read -r load active enabled < "$ROOT/$unit"
+ printf '%s inactive %s\n' "$load" "$enabled" > "$ROOT/$unit"
+ touch "$ROOT/stopped"
+ # Earlier activation sources changing during a later stop are not
+ # safe just because each individual stop once succeeded.
+ if [[ "$unit" == pulse-agent-helper.service ]]; then
+ case "$FAULT" in
+ socket-reactivated) printf 'loaded active disabled\n' > "$ROOT/pulse-agent-helper.socket" ;;
+ socket-reenabled) printf 'loaded inactive enabled\n' > "$ROOT/pulse-agent-helper.socket" ;;
+ esac
+ fi
+ if [[ "$unit" == pulse-agent.service && "$FAULT" == runner-reactivated ]]; then
+ printf 'loaded active disabled\n' > "$ROOT/pulse-agent-runner.service"
+ fi
+ ;;
+ disable)
+ if [[ "$unit" == "$TARGET" ]]; then
+ [[ "$FAULT" != disable-refused ]] || return 1
+ [[ "$FAULT" != ignored-disable ]] || return 0
+ fi
+ read -r load active enabled < "$ROOT/$unit"
+ case "$enabled" in masked|masked-runtime|static|indirect) ;;
+ *) enabled=disabled ;;
+ esac
+ printf '%s %s %s\n' "$load" "$active" "$enabled" > "$ROOT/$unit"
+ ;;
+ daemon-reload|reset-failed) return 0 ;;
+ *) return 98 ;;
+ esac
+}
+rm() {
+ local path
+ [[ "$1" == -f || "$1" == -rf ]] || return 99
+ for path in "${@:2}"; do
+ case "$path" in "$ROOT/artifacts/"*) ;;
+ *) printf 'outside fixture deletion refused\n' >&2; return 100 ;;
+ esac
+ done
+ printf 'delete\n' >> "$EVENTS"
+ command rm "$@"
+}
+'''
+
+
+class AgentSystemdTeardown(unittest.TestCase):
+ source = SOURCE.read_text()
+
+ def exercise(self, family, fault="", state="loaded active enabled"):
+ with tempfile.TemporaryDirectory() as temporary:
+ root = Path(temporary)
+ artifacts = root / "artifacts"
+ artifacts.mkdir()
+ paths = {
+ "PRIVILEGED_HELPER_SOCKET_UNIT": artifacts / "helper.socket",
+ "PRIVILEGED_HELPER_SERVICE_UNIT": artifacts / "helper.service",
+ "PRIVILEGED_HELPER_SOCKET_PATH": artifacts / "helper.sock",
+ "PRIVILEGED_HELPER_CREDENTIAL_DIR": artifacts / "helper-credentials",
+ "ACTION_RUNNER_BINARY_PATH": artifacts / "runner",
+ "ACTION_RUNNER_SERVICE_UNIT": artifacts / "runner.service",
+ "ACTION_RUNNER_CONFIG_DIR": artifacts / "runner-config",
+ "ACTION_RUNNER_STATE_DIR": artifacts / "runner-state",
+ }
+ for name, path in paths.items():
+ if name.endswith("DIR"):
+ path.mkdir()
+ (path / "token").write_text("synthetic-recovery-credential\n")
+ else:
+ path.write_text("owned recovery material\n")
+ agent_unit = artifacts / "agent.service"
+ agent_unit.write_text("owned agent unit\n")
+ unrelated = root / "unrelated"
+ unrelated.write_text("keep\n")
+ before = {str(p.relative_to(artifacts)): p.read_bytes()
+ for p in artifacts.rglob("*") if p.is_file()}
+ target = {"agent": "pulse-agent.service", "helper": "pulse-agent-helper.socket",
+ "runner": "pulse-agent-runner.service", "batch": "pulse-agent.service"}[family]
+ for unit in ("pulse-agent.service", "pulse-agent-helper.socket",
+ "pulse-agent-helper.service", "pulse-agent-runner.service"):
+ (root / unit).write_text(state + "\n")
+ values = dict(paths, ROOT=root, EVENTS=root / "events", FAULT=fault, TARGET=target)
+ setup = "\n".join(f"{key}={shlex.quote(str(value))}" for key, value in values.items())
+ names = ("read_agent_removal_unit_state", "confirm_agent_unit_quiescent",
+ "quiesce_agent_units_for_removal", "teardown_systemd_agent_service",
+ "teardown_privileged_helper_service", "teardown_action_runner_service")
+ # The same controls can reproduce the older source, which has no
+ # observation helper: never insert a substitute implementation.
+ functions = "\n".join(function(self.source, name) for name in names
+ if name + "() {" in self.source)
+ calls = {"agent": "teardown_systemd_agent_service " + shlex.quote(str(agent_unit)),
+ "helper": "teardown_privileged_helper_service",
+ "runner": "teardown_action_runner_service",
+ "batch": "quiesce_agent_units_for_removal pulse-agent-runner.service pulse-agent-helper.socket pulse-agent-helper.service pulse-agent.service"}
+ # Production fail() exits even if the caller's errexit is disabled.
+ functions += "\n" + function(self.source, "fail")
+ shell = DOUBLE + "\nOUTPUT_FORMAT=text\n" + setup + "\n" + functions + "\n" + calls[family]
+ result = subprocess.run(["bash", "-c", shell], capture_output=True,
+ text=True, timeout=8)
+ events = (root / "events").read_text().splitlines() if (root / "events").exists() else []
+ after = {str(p.relative_to(artifacts)): p.read_bytes()
+ for p in artifacts.rglob("*") if p.is_file()}
+ self.assertEqual(unrelated.read_text(), "keep\n")
+ return result, events, before, after
+
+ def test_unknown_read_stop_and_disable_retain_every_artifact(self):
+ faults = ("read-refused", "read-empty", "read-blank-diagnostic", "read-diagnostic",
+ "read-trailing-blank-diagnostic", "read-duplicate", "read-missing", "stop-refused", "stop-timeout",
+ "ignored-stop", "disable-refused", "ignored-disable", "read-refused-after-stop")
+ for family in ("agent", "helper", "runner"):
+ for fault in faults:
+ with self.subTest(family=family, fault=fault):
+ result, events, before, after = self.exercise(family, fault)
+ self.assertNotEqual(result.returncode, 0, result.stdout + result.stderr)
+ self.assertEqual(after, before)
+ self.assertNotIn("delete", events)
+ self.assertNotIn("revoke", events)
+ self.assertNotIn("private diagnostic", result.stdout + result.stderr)
+
+ def test_unsettled_or_unknown_states_never_authorise_deletion(self):
+ for state in ("error active enabled", "loaded activating enabled", "loaded deactivating enabled",
+ "loaded reloading enabled", "loaded unknown enabled", "loaded inactive unknown",
+ "not-found active -", "not-found inactive enabled"):
+ for family in ("agent", "helper", "runner"):
+ with self.subTest(family=family, state=state):
+ result, events, before, after = self.exercise(family, state=state)
+ self.assertNotEqual(result.returncode, 0, result.stdout + result.stderr)
+ self.assertEqual(after, before)
+ self.assertNotIn("delete", events)
+
+ def test_helper_socket_and_service_are_reconciled_together(self):
+ for fault in ("socket-reactivated", "socket-reenabled"):
+ with self.subTest(fault=fault):
+ result, events, before, after = self.exercise("helper", fault)
+ self.assertNotEqual(result.returncode, 0, result.stdout + result.stderr)
+ self.assertEqual(after, before)
+ self.assertNotIn("delete", events)
+
+ def test_known_quiescence_allows_only_owned_teardown(self):
+ for family in ("agent", "helper", "runner"):
+ for state in ("loaded active enabled", "loaded failed enabled", "loaded inactive disabled",
+ "loaded inactive static", "loaded inactive indirect", "masked inactive masked",
+ "masked inactive masked-runtime", "not-found inactive -"):
+ with self.subTest(family=family, state=state):
+ result, events, before, after = self.exercise(family, state=state)
+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+ self.assertLess(len(after), len(before))
+ first_delete = events.index("delete")
+ self.assertTrue(any(event.startswith("show:") for event in events[:first_delete]))
+ if family == "runner":
+ self.assertLess(events.index("revoke"), first_delete)
+ self.assertTrue(any(event.startswith("show:") for event in events[:events.index("revoke")]))
+ if state.startswith("not-found"):
+ self.assertFalse(any(event.startswith(("stop:", "disable:")) for event in events))
+
+ def test_failed_revocation_retains_stopped_runner_recovery_material(self):
+ result, events, before, after = self.exercise("runner", "revoke-refused")
+ self.assertNotEqual(result.returncode, 0)
+ self.assertIn("revoke", events)
+ self.assertNotIn("delete", events)
+ self.assertEqual(after, before)
+
+ def test_full_batch_reconciles_earlier_activation_sources(self):
+ for fault in ("", "runner-reactivated", "stop-refused"):
+ with self.subTest(fault=fault):
+ result, events, before, after = self.exercise("batch", fault)
+ self.assertEqual(result.returncode == 0, fault == "", result.stdout + result.stderr)
+ self.assertEqual(after, before)
+ self.assertNotIn("delete", events)
+ self.assertNotIn("revoke", events)
+
+ def test_real_read_timeout_stops_its_owned_child(self):
+ with tempfile.TemporaryDirectory() as temporary:
+ root = Path(temporary)
+ executable = root / "systemctl"
+ child_id = root / "child-id"
+ executable.write_text("#!/bin/bash\nprintf '%s\\n' \"$$\" > \"$CHILD_ID\"\nexec sleep 60\n")
+ executable.chmod(0o700)
+ env = dict(os.environ, PATH=str(root) + ":/usr/bin:/bin", CHILD_ID=str(child_id))
+ started = time.monotonic()
+ result = subprocess.run(["bash", "-c", function(self.source, "read_agent_removal_unit_state") +
+ "\nread_agent_removal_unit_state pulse-agent.service\n"],
+ env=env, capture_output=True, text=True, timeout=8)
+ self.assertNotEqual(result.returncode, 0, result.stdout + result.stderr)
+ self.assertLess(time.monotonic() - started, 7)
+ self.assertEqual(result.stdout, "")
+ self.assertTrue(child_id.exists(), "the bounded read never started")
+ with self.assertRaises(ProcessLookupError):
+ os.kill(int(child_id.read_text()), 0)
+
+
+if __name__ == "__main__":
+ unittest.main(verbosity=2)
diff --git a/scripts/installtests/agent_uninstall_state.py b/scripts/installtests/agent_uninstall_state.py
index 0f2b20e35a..029e6a8fe0 100644
--- a/scripts/installtests/agent_uninstall_state.py
+++ b/scripts/installtests/agent_uninstall_state.py
@@ -42,7 +42,7 @@ class UnixUninstallState(unittest.TestCase):
def exercise(self, credential="runtime", url="", reply="confirmed", authority=True,
input_token=False, interactive=False, root_refusal=False, action="uninstall",
- ingress_only=False):
+ ingress_only=False, quiescence_refused=False):
source = self.source
# Keep the real entry and all initial credential/service/state teardown.
# Stop before unrelated platform boot/log cleanup; never run it on a host.
@@ -104,7 +104,8 @@ class UnixUninstallState(unittest.TestCase):
"SAFE_PROFILE_STATE_DIR_REMOVAL_AUTHORITY": profile,
"PULSE_TOKEN": token, "RUNTIME_TOKEN_FILE": selected,
"PULSE_URL": url, "AGENT_ID": "owned-agent", "HOSTNAME_OVERRIDE": "owned-host",
- "EVENTS": root / "events", "SERVICE": service, "REPLY": reply}
+ "EVENTS": root / "events", "SERVICE": service, "REPLY": reply,
+ "QUIESCENCE_REFUSED": str(quiescence_refused).lower()}
setup = "\n".join(f"{key}={shlex.quote(str(value))}" for key, value in values.items())
functions = "\n".join(function(source, name) for name in names)
if input_token:
@@ -113,6 +114,7 @@ class UnixUninstallState(unittest.TestCase):
shell = r'''
set -euo pipefail
UNINSTALL=true AGENT_NAME=pulse-agent EXIT_GENERAL=1 EXIT_MISSING_ARGS=16
+ACTION_RUNNER_NAME=pulse-agent-runner PRIVILEGED_HELPER_NAME=pulse-agent-helper
OUTPUT_FORMAT=text NON_INTERACTIVE=true
TOKEN_FILE_PATH="" UNINSTALL_ACTION_RUNNER=false UPDATE_ONLY=false
RETARGET_ONLY=false PREFLIGHT_ONLY=false SAFE_PROFILE_ACTION=""
@@ -143,6 +145,12 @@ run_collector_lifecycle_command() {
pkill() { printf 'process-stop\n' >> "$EVENTS"; }
sleep() { :; }
systemctl() { return 91; }
+# This fixture checks the caller's refusal gate, not systemd observations.
+# agent_systemd_teardown.py executes that real boundary independently.
+quiesce_agent_units_for_removal() {
+ [[ "$*" == 'pulse-agent-runner.service pulse-agent-helper.socket pulse-agent-helper.service pulse-agent.service' ]] || return 90
+ [[ "$QUIESCENCE_REFUSED" == false ]]
+}
teardown_action_runner_service() { printf 'runner-stop\n' >> "$EVENTS"; }
teardown_privileged_helper_service() { printf 'helper-stop\n' >> "$EVENTS"; }
teardown_systemd_agent_service() { printf 'service-stop\n' >> "$EVENTS"; printf 'stopped\n' > "$SERVICE"; }
@@ -190,6 +198,16 @@ rm() {
self.assertEqual(events, "request:collector-uninstall --agent-id owned-agent --hostname owned-host\n")
self.assertEqual(after, before)
+ def test_systemd_refusal_precedes_remote_removal_and_every_local_mutation(self):
+ for credential in ("runtime", "inline", "none"):
+ with self.subTest(credential=credential):
+ result, events, before, after = self.exercise(
+ credential, url="https://pulse.invalid", quiescence_refused=True)
+ self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
+ self.assertEqual(events, "")
+ self.assertEqual(after, before)
+ self.assertIn("every systemd unit stopped and disabled", result.stdout)
+
def test_exact_confirmation_precedes_process_service_and_state_teardown(self):
result, events, before, after = self.exercise(url="https://pulse.invalid")
self.assertEqual(result.returncode, 0, result.stderr)
diff --git a/scripts/installtests/install_sh_test.go b/scripts/installtests/install_sh_test.go
index e2b05197ef..5310f23c08 100644
--- a/scripts/installtests/install_sh_test.go
+++ b/scripts/installtests/install_sh_test.go
@@ -8286,6 +8286,7 @@ func TestInstallSHUninstallRetainsCredentialsWhenSavedURLIsMissing(t *testing.T)
for _, name := range []string{
"test_missing_url_retains_every_credential_representation_without_a_request",
"test_server_refusal_unknown_or_other_identity_retains_service_and_state",
+ "test_systemd_refusal_precedes_remote_removal_and_every_local_mutation",
"test_exact_confirmation_precedes_process_service_and_state_teardown",
"test_credential_free_local_only_removal_needs_no_url_or_request",
"test_server_confirmation_does_not_grant_unknown_path_removal",
@@ -8300,8 +8301,45 @@ func TestInstallSHUninstallRetainsCredentialsWhenSavedURLIsMissing(t *testing.T)
t.Fatalf("Unix removal control %s did not pass:\n%s", name, output)
}
}
- if !strings.Contains(string(output), "\nRan 11 tests in ") || !strings.Contains(string(output), "\nOK\n") {
+ if !strings.Contains(string(output), "\nRan 12 tests in ") || !strings.Contains(string(output), "\nOK\n") {
t.Fatalf("incomplete Unix removal state verdict:\n%s", output)
}
t.Logf("%s", output)
}
+
+func TestInstallSHSystemdTeardownRequiresObservedQuiescence(t *testing.T) {
+ ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
+ defer cancel()
+ cmd := exec.CommandContext(ctx, "python3", repoFile("scripts", "installtests", "agent_systemd_teardown.py"))
+ output, err := cmd.CombinedOutput()
+ if err != nil {
+ t.Fatalf("systemd teardown refusal controls: %v\n%s", err, output)
+ }
+ for _, name := range []string{
+ "test_unknown_read_stop_and_disable_retain_every_artifact",
+ "test_unsettled_or_unknown_states_never_authorise_deletion",
+ "test_helper_socket_and_service_are_reconciled_together",
+ "test_known_quiescence_allows_only_owned_teardown",
+ "test_failed_revocation_retains_stopped_runner_recovery_material",
+ "test_full_batch_reconciles_earlier_activation_sources",
+ "test_real_read_timeout_stops_its_owned_child",
+ } {
+ if !strings.Contains(string(output), name+" (__main__.AgentSystemdTeardown."+name+") ... ok") {
+ t.Fatalf("systemd teardown control %s did not pass:\n%s", name, output)
+ }
+ }
+ if !strings.Contains(string(output), "\nRan 7 tests in ") || !strings.Contains(string(output), "\nOK\n") {
+ t.Fatalf("incomplete systemd teardown verdict:\n%s", output)
+ }
+ entry, err := os.ReadFile(repoFile("scripts", "install.sh"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ uninstall := strings.SplitN(string(entry), "# --- Uninstall Logic ---", 2)[1]
+ quiesce := strings.Index(uninstall, "quiesce_agent_units_for_removal")
+ if quiesce < 0 || quiesce >= strings.Index(uninstall, "if ! uninstall_collector_registration;") ||
+ quiesce >= strings.Index(uninstall, "pkill -f") {
+ t.Fatal("full removal must quiesce every affected systemd unit before remote removal or process mutation")
+ }
+ t.Logf("%s", output)
+}