Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Confirm DSM Upstart retirement before erasing agent recovery
pulse-triage[bot]a6527d9ae321
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 479602c0d5..eda91ef8a3 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -15,6 +15,39 @@## Purpose+### DSM Upstart removal retains recovery until automatic activation and the job are absent++DSM 6 Upstart jobs (including leftovers on DSM 7) are not classified as SysV.+Before remote removal, the installer validates protected regular Pulse job and+manual-override files, observes one complete combined status and independent+process inventory, installs a non-clobbering `manual` override, reloads and+checks the effective configuration has no start condition, then confirms stopped+status and process absence. A missing manager, transitional/unknown state,+diagnostic, duplicate, empty or failed observation refuses without another+sample. A successful stop/override may remain after failure; do not restart or+restore automatic activation. Foreign override/configuration and any previous+`.removal.*` recovery copy refuse without deletion.++After remote confirmation, a fresh gate precedes retiring the job. Its original+configuration inode is staged outside Upstart's `.conf` namespace until checked+reload, exact unknown-job status and independent process absence establish+retirement. On unconfirmed retirement it is restored only if the original path+is still absent, leaving the manual override and all local recovery state;+a concurrently created config is never overwritten and the staged recovery copy+is retained. Only confirmed retirement removes the staged config and manual+before credential/state erasure. No credentials, arguments or config contents+are returned by these observations. Existing same-instance URL, exact record,+TLS, systemd/OpenRC/rc.d/SysV and state-directory ownership remain unchanged.++The registered `scripts/installtests/install_sh_test.go` requires all thirteen+`agent_upstart_teardown.py` production-function/caller controls, including exact+file restoration, effective activation, late refusal, stopped/absent/repeated+removal and real timeout-child termination. These are confined ordinary-user+files and closed manager/process/transport models, not native DSM/Upstart,+current-compiler or installed Linux/PowerShell acceptance. Unraid/QNAP/TrueNAS+boot/watchdog ownership, Windows and the original native/TLS/cleanup journey+remain separate; this does not broaden #1099 into macOS acceptance.+### FreeBSD rc.d and SysV removal retains recovery on unknown or live stateBefore server deregistration or local erasure, Unix uninstall selects itsdiff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.mdindex 120f3d48a3..9ee766de95 100644--- a/docs/release-control/v6/internal/subsystems/deployment-installability.md+++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md@@ -16,6 +16,39 @@## Purpose+### DSM Upstart removal retains recovery until automatic activation and the job are absent++DSM 6 Upstart jobs (including leftovers on DSM 7) are not classified as SysV.+Before remote removal, the installer validates protected regular Pulse job and+manual-override files, observes one complete combined status and independent+process inventory, installs a non-clobbering `manual` override, reloads and+checks the effective configuration has no start condition, then confirms stopped+status and process absence. A missing manager, transitional/unknown state,+diagnostic, duplicate, empty or failed observation refuses without another+sample. A successful stop/override may remain after failure; do not restart or+restore automatic activation. Foreign override/configuration and any previous+`.removal.*` recovery copy refuse without deletion.++After remote confirmation, a fresh gate precedes retiring the job. Its original+configuration inode is staged outside Upstart's `.conf` namespace until checked+reload, exact unknown-job status and independent process absence establish+retirement. On unconfirmed retirement it is restored only if the original path+is still absent, leaving the manual override and all local recovery state;+a concurrently created config is never overwritten and the staged recovery copy+is retained. Only confirmed retirement removes the staged config and manual+before credential/state erasure. No credentials, arguments or config contents+are returned by these observations. Existing same-instance URL, exact record,+TLS, systemd/OpenRC/rc.d/SysV and state-directory ownership remain unchanged.++The registered `scripts/installtests/install_sh_test.go` requires all thirteen+`agent_upstart_teardown.py` production-function/caller controls, including exact+file restoration, effective activation, late refusal, stopped/absent/repeated+removal and real timeout-child termination. These are confined ordinary-user+files and closed manager/process/transport models, not native DSM/Upstart,+current-compiler or installed Linux/PowerShell acceptance. Unraid/QNAP/TrueNAS+boot/watchdog ownership, Windows and the original native/TLS/cleanup journey+remain separate; this does not broaden #1099 into macOS acceptance.+### FreeBSD rc.d and SysV removal retains recovery on unknown or live stateBefore server deregistration or local erasure, Unix uninstall selects itsdiff --git a/scripts/install.sh b/scripts/install.shindex 10cc1e28e5..a697c51c2c 100755--- a/scripts/install.sh+++ b/scripts/install.sh@@ -1552,6 +1552,118 @@ teardown_sysv_agent_service() {rm -f "/var/run/${AGENT_NAME}.pid"}+# DSM 6 uses Upstart, not SysV. A successful stop alone does not disable the+# syno.network.ready event, and respawn/status errors must not lose recovery.+read_upstart_agent_removal_status() {+ local observation=""+ observation=$(LC_ALL=C timeout -k 1 5 initctl status "$AGENT_NAME" 2>&1 &&+ printf '%s' '__pulse_upstart_exit__=0' || printf '%s' "__pulse_upstart_exit__=$?") || return 1+ case "$observation" in+ "$AGENT_NAME stop/waiting"$'\n__pulse_upstart_exit__=0') printf 'stopped\n' ;;+ "initctl: Unknown job: $AGENT_NAME"$'\n__pulse_upstart_exit__=1') printf 'absent\n' ;;+ *)+ [[ "$observation" =~ ^pulse-agent\ start/running,\ process\ [1-9][0-9]*$'\n'__pulse_upstart_exit__=0$ ]] || return 1+ printf 'started\n' ;;+ esac+}++upstart_agent_removal_paths_safe() {+ local conf="/etc/init/${AGENT_NAME}.conf" override="/etc/init/${AGENT_NAME}.override"+ local path="" mode="" size="" content="" line="" exec_count=0+ # An interrupted config retirement belongs to its previous operation. Do+ # not delete or overwrite its recovery copy on a later uninstall.+ for path in "$conf".removal.*; do [[ ! -e "$path" && ! -L "$path" ]] || return 1; done+ for path in "$conf" "$override"; do+ [[ -e "$path" || -L "$path" ]] || continue+ mode=$(portable_path_mode "$path") || return 1+ [[ "$mode" == 600 || "$mode" == 644 ]] || return 1+ trusted_lifecycle_regular_file "$path" "$mode" || return 1+ size=$(wc -c < "$path") || return 1+ [[ "$size" =~ ^[[:space:]]*([0-9]+)[[:space:]]*$ && ${BASH_REMATCH[1]} -le 1048576 ]] || return 1+ content=$(cat "$path" && printf '%s' '__pulse_upstart_file_complete__') || return 1+ if [[ "$path" == "$override" ]]; then+ [[ "$content" == $'manual\n__pulse_upstart_file_complete__' ]] || return 1+ else+ [[ "$content" == *$'description "Pulse Unified Agent"\n'* && "$content" == *$'author "Pulse"\n'* ]] || return 1+ while IFS= read -r line; do+ if [[ "$line" == exec\ * ]]; then+ [[ "$line" == "exec ${INSTALL_DIR}/${BINARY_NAME} "* ]] || return 1+ exec_count=$((exec_count + 1))+ fi+ done <<< "$content"+ [[ "$exec_count" == 1 ]] || return 1+ fi+ done+}++read_upstart_agent_removal_enablement() {+ local observation=""+ observation=$(LC_ALL=C timeout -k 1 5 initctl show-config "$AGENT_NAME" 2>&1 &&+ printf '%s' '__pulse_upstart_config_complete__') || return 1+ # The installer renders this single-instance job with a stop condition.+ # manual removes its start condition from Upstart's effective config.+ [[ "$observation" == "$AGENT_NAME"$'\n stop on runlevel [06]\n__pulse_upstart_config_complete__' ]]+}++quiesce_upstart_agent_for_removal() {+ local conf="/etc/init/${AGENT_NAME}.conf" override="/etc/init/${AGENT_NAME}.override"+ local state="" inventory="" pending=""+ upstart_agent_removal_paths_safe || return 1+ state=$(read_upstart_agent_removal_status) || return 1+ inventory=$(read_legacy_agent_removal_processes) || return 1+ if [[ ! -e "$conf" ]]; then+ [[ "$state" == absent ]] && legacy_agent_processes_absent "$inventory"+ return+ fi+ [[ "$state" != absent ]] || return 1+ if [[ "$state" == stopped ]]; then legacy_agent_processes_absent "$inventory" || return 1+ else legacy_agent_processes_absent "$inventory" && return 1; fi+ if [[ ! -e "$override" ]]; then+ pending=$(mktemp "${override}.XXXXXX") || return 1+ # Link without replacement: an override created concurrently is not+ # ours to overwrite. The temporary file contains no credential.+ if ! { printf 'manual\n' > "$pending" && chmod 644 "$pending" && ln -- "$pending" "$override"; }; then+ rm -f -- "$pending"+ return 1+ fi+ rm -f -- "$pending" || return 1+ fi+ LC_ALL=C timeout -k 1 5 initctl reload-configuration >/dev/null 2>&1 || return 1+ read_upstart_agent_removal_enablement || return 1+ if [[ "$state" == started ]]; then+ LC_ALL=C timeout -k 1 15 initctl stop "$AGENT_NAME" >/dev/null 2>&1 || return 1+ fi+ state=$(read_upstart_agent_removal_status) || return 1+ [[ "$state" == stopped ]] || return 1+ inventory=$(read_legacy_agent_removal_processes) || return 1+ legacy_agent_processes_absent "$inventory"+}++teardown_upstart_agent_service() {+ local conf="/etc/init/${AGENT_NAME}.conf" override="/etc/init/${AGENT_NAME}.override"+ local retired="" state="" inventory="" confirmed=false+ quiesce_upstart_agent_for_removal ||+ fail "DSM Upstart removal could not confirm stopped processes and disabled automatic activation; local recovery files were retained. A preceding stop or manual override may have succeeded; no automatic restart was attempted." "$EXIT_GENERAL"+ if [[ -e "$conf" ]]; then+ # Keep the original inode/contents until the manager confirms the job+ # absent. A failed reload/read restores it without removing manual or+ # restarting; never clobber a configuration created in the meantime.+ retired=$(mktemp "${conf}.removal.XXXXXX") || return 1+ if ! mv -- "$conf" "$retired"; then rm -f -- "$retired"; return 1; fi+ if LC_ALL=C timeout -k 1 5 initctl reload-configuration >/dev/null 2>&1 &&+ state=$(read_upstart_agent_removal_status) && [[ "$state" == absent ]] &&+ inventory=$(read_legacy_agent_removal_processes) && legacy_agent_processes_absent "$inventory"; then+ confirmed=true+ fi+ if [[ "$confirmed" != true ]]; then+ if [[ ! -e "$conf" && ! -L "$conf" ]]; then mv -- "$retired" "$conf" || return 1; fi+ return 1+ fi+ rm -f -- "$retired" || return 1+ fi+ rm -f -- "$override" || return 1+}+enable_sysv_agent_service() {local init_path="${1:-/etc/init.d/${AGENT_NAME}}"if command -v update-rc.d >/dev/null 2>&1; then@@ -5870,6 +5982,7 @@ if [[ "$UNINSTALL" == "true" ]]; thenlocal qnap_state_dir=""local uninstall_has_credential="false"local uninstall_legacy_manager=""+ local uninstall_upstart="false"# Validate saved connection availability before stopping a working install.# Stop every systemd activation source before remote removal, process kills@@ -5880,6 +5993,15 @@ if [[ "$UNINSTALL" == "true" ]]; thenfail "The saved Pulse URL is missing; local credentials and services were retained. Restore saved connection state or provide --url for the same trusted Pulse instance and retry uninstall." "$EXIT_MISSING_ARGS"fifi+ # Detect stale DSM 6 jobs even on a DSM 7 host. Another init manager cannot+ # establish their absence; missing initctl is an explicit safe refusal.+ if [[ -d /usr/syno ]] && { command -v initctl >/dev/null 2>&1 ||+ [[ -e "/etc/init/${AGENT_NAME}.conf" || -L "/etc/init/${AGENT_NAME}.conf" ||+ -e "/etc/init/${AGENT_NAME}.override" || -L "/etc/init/${AGENT_NAME}.override" ]]; }; then+ uninstall_upstart="true"+ quiesce_upstart_agent_for_removal ||+ fail "DSM Upstart removal could not confirm stopped processes and disabled automatic activation; credentials and service files were retained. Reconcile the job and any manual override before retrying; no automatic restart was attempted." "$EXIT_GENERAL"+ fiif command -v systemctl >/dev/null 2>&1; thenquiesce_agent_units_for_removal "${ACTION_RUNNER_NAME}.service" \"${PRIVILEGED_HELPER_NAME}.socket" "${PRIVILEGED_HELPER_NAME}.service" "${AGENT_NAME}.service" ||@@ -5894,7 +6016,7 @@ if [[ "$UNINSTALL" == "true" ]]; thenuninstall_legacy_manager=FreeBSDquiesce_freebsd_agent_for_removal ||fail "FreeBSD removal could not confirm stopped rc.d, disabled boot activation and absent supervisor/child; local recovery files were retained. Reconcile linked/appliance boot tasks first when present; no automatic restart was attempted." "$EXIT_GENERAL"- elif ! command -v systemctl >/dev/null 2>&1 && ! command -v rc-service >/dev/null 2>&1 && [[ -d /etc/init.d ]]; then+ elif [[ "$uninstall_upstart" != true ]] && ! command -v systemctl >/dev/null 2>&1 && ! command -v rc-service >/dev/null 2>&1 && [[ -d /etc/init.d ]]; thenuninstall_legacy_manager=SysVquiesce_sysv_agent_for_removal ||fail "SysV removal could not confirm stopped process and disabled runlevels; local recovery files were retained. A preceding stop or disable may have succeeded; no automatic restart was attempted." "$EXIT_GENERAL"@@ -5926,6 +6048,10 @@ if [[ "$UNINSTALL" == "true" ]]; thenelif [[ "$uninstall_legacy_manager" == SysV ]]; thenteardown_sysv_agent_service "/etc/init.d/${AGENT_NAME}"fi+ if [[ "$uninstall_upstart" == true ]]; then+ teardown_upstart_agent_service ||+ fail "DSM Upstart job retirement was not confirmed; local recovery state was retained. The manual override remains after a partial stop. Reconcile the original config or its .removal recovery copy before retrying; no automatic restart was attempted." "$EXIT_GENERAL"+ fi# Kill wrapper scripts first: they are watchdogs, so stopping the agent# while its wrapper still loops only races the respawn. No leading path@@ -5984,11 +6110,7 @@ if [[ "$UNINSTALL" == "true" ]]; thenif [[ -f "/etc/systemd/system/${AGENT_NAME}.service" ]]; thenteardown_systemd_agent_servicefi- # DSM 6.x uses upstart- if [[ -f "/etc/init/${AGENT_NAME}.conf" ]]; then- initctl stop "${AGENT_NAME}" 2>/dev/null || true- rm -f "/etc/init/${AGENT_NAME}.conf"- fi+ # DSM 6.x Upstart retirement was confirmed before recovery erasure.fi# Unraiddiff --git a/scripts/installtests/agent_upstart_teardown.py b/scripts/installtests/agent_upstart_teardown.pynew file mode 100644index 0000000000..9dc684cb8b--- /dev/null+++ b/scripts/installtests/agent_upstart_teardown.py@@ -0,0 +1,374 @@+#!/usr/bin/env python3+"""Run DSM Upstart production admission/retirement with closed manager models.++No native DSM, host init manager, privileged installer, real credential or+network is used. Production functions and the caller execute against private+regular files, including real config moves/restoration and bounded child stop.+"""+import json+import os+from pathlib import Path+import shlex+import stat+import subprocess+import tempfile+import unittest++ROOT = Path(__file__).resolve().parents[2]+SOURCE = Path(os.environ.get("PULSE_AGENT_UPSTART_SOURCE", ROOT / "scripts/install.sh"))+FUNCTIONS = ("portable_path_uid", "portable_path_mode", "trusted_lifecycle_regular_file",+ "read_legacy_agent_removal_processes", "legacy_agent_processes_absent",+ "read_upstart_agent_removal_status", "upstart_agent_removal_paths_safe",+ "read_upstart_agent_removal_enablement", "quiesce_upstart_agent_for_removal",+ "teardown_upstart_agent_service")+++def definitions(source):+ selected = []+ for name in FUNCTIONS:+ marker = name + "() {"+ if marker not in source: # Exact parent adverse controls.+ continue+ if source.count(marker) != 1:+ raise ValueError("ambiguous production function")+ selected.append(marker + source.split(marker, 1)[1].split("\n}\n", 1)[0] + "\n}\n")+ return "\n".join(selected)+++MODEL = r'''#!/usr/bin/python3+import json, os, pathlib, sys+root = pathlib.Path(os.environ['FIXTURE'])+path = root / 'model.json'+v = json.loads(path.read_text())+role, args = pathlib.Path(sys.argv[0]).name, sys.argv[1:]+fault = v['fault']+remote = (root / 'events').exists() and 'remote:remove' in (root / 'events').read_text()+conf = root / 'init/pulse-agent.conf'+override = root / 'init/pulse-agent.override'+retired = not conf.exists() and bool(list((root / 'init').glob('pulse-agent.conf.removal.*')))+def save():+ pending = root / 'model.pending'+ pending.write_text(json.dumps(v)); pending.replace(path)+def event(s):+ with (root / 'events').open('a') as f: f.write(s + '\n')+if role == 'ps':+ assert args == ['-ax', '-o', 'pid=', '-o', 'ppid=', '-o', 'comm=', '-o', 'args='], args+ event('process:read'); v['process_reads'] += 1; save()+ if fault == 'failed-process' or (fault == 'failed-retirement-process' and retired): sys.exit(2)+ if fault == 'empty-process': sys.exit(0)+ print('1 0 init /sbin/init')+ if fault == 'duplicate-process': print('1 0 init /sbin/init')+ if v['state'] == 'started' or fault == 'orphan-process' or (fault == 'retirement-orphan' and retired):+ print('100 1 pulse-agent /usr/local/bin/pulse-agent --token-file /synthetic/recovery-only')+ if fault == 'diagnostic-process': print('read refused', file=sys.stderr)+ sys.exit(0)+assert role == 'initctl', role+if args == ['reload-configuration']:+ event('manager:reload'); v['reloads'] += 1+ if fault == 'refused-reload' or (fault == 'refused-retirement-reload' and retired):+ save(); sys.exit(1)+ v['manual'] = override.exists() and override.read_text() == 'manual\n' and fault != 'ignored-manual'+ if not conf.exists(): v['state'] = 'absent'+ save(); sys.exit(0)+assert len(args) == 2 and args[1] == 'pulse-agent', args+verb = args[0]+if verb == 'status':+ event('manager:status'); v['status_reads'] += 1; save()+ if fault == 'timeout-status':+ (root / 'timed-pid').write_text(str(os.getpid()))+ os.execl('/bin/sleep', 'sleep', '30')+ if fault == 'failed-status' or (fault == 'failed-after-remote' and remote) or (fault == 'failed-retirement-status' and retired):+ print('initctl: manager connection refused', file=sys.stderr); sys.exit(1)+ state = v['state']+ code = 1 if state == 'absent' else 0+ if fault == 'wrong-status-exit': code = 2+ if fault == 'empty-status': sys.exit(code)+ if fault == 'unknown-status': print('pulse-agent start/post-start, process 100')+ elif state == 'started': print('pulse-agent start/running, process 100')+ elif state == 'stopped': print('pulse-agent stop/waiting')+ else: print('initctl: Unknown job: pulse-agent', file=sys.stderr)+ if fault == 'duplicate-status': print('pulse-agent stop/waiting')+ if fault == 'blank-status': print()+ if fault == 'diagnostic-status': print('read refused', file=sys.stderr)+ sys.exit(code)+if verb == 'show-config':+ event('manager:config')+ if fault == 'failed-config': sys.exit(1)+ if fault == 'empty-config': sys.exit(0)+ print('other-agent' if fault == 'foreign-config' else 'pulse-agent')+ if not v['manual']: print(' start on syno.network.ready')+ print(' stop on runlevel [06]')+ if fault == 'diagnostic-config': print('read refused', file=sys.stderr)+ if fault == 'duplicate-config': print(' stop on runlevel [06]')+ if fault == 'blank-config': print()+ sys.exit(0)+assert verb == 'stop', verb+event('manager:stop')+if fault == 'refused-stop': sys.exit(1)+if fault != 'ignored-stop': v['state'] = 'stopped'+save(); sys.exit(0)+'''+++class AgentUpstartTeardown(unittest.TestCase):+ source = SOURCE.read_text()++ def exercise(self, *, fault="", flow="caller", state="started", conf="regular",+ override="missing", reply="confirmed", repeat=False):+ with tempfile.TemporaryDirectory() as temporary:+ root = Path(temporary)+ tools = root / "tools"; tools.mkdir()+ for name in ("ps", "initctl"):+ p = tools / name; p.write_text(MODEL); p.chmod(0o700)+ init = root / "init"; init.mkdir()+ config = init / "pulse-agent.conf"+ unrelated = root / "unrelated"; unrelated.write_text("retain unrelated recovery\n")+ if conf == "linked": config.symlink_to(unrelated)+ elif conf == "directory": config.mkdir()+ elif conf != "missing":+ text = ('description "Pulse Unified Agent"\nauthor "Pulse"\n'+ 'start on syno.network.ready\nstop on runlevel [06]\n'+ 'respawn\nrespawn limit 5 10\n\n'+ 'exec /usr/local/bin/pulse-agent --token-file /synthetic/recovery >> /private/log 2>&1\n')+ if conf == "foreign": text = text.replace("/usr/local/bin/pulse-agent", "/usr/local/bin/other-agent")+ if conf == "oversized": text += "#" * 1048577+ config.write_text(text); config.chmod(0o666 if conf == "writable" else 0o644)+ manual = init / "pulse-agent.override"+ if override == "linked": manual.symlink_to(unrelated)+ elif override != "missing":+ manual.write_text("manual\n" if override == "manual" else "start on started other\n")+ manual.chmod(0o644)+ if fault == "old-retirement":+ (init / "pulse-agent.conf.removal.prior").write_text("retained previous recovery\n")+ for name in ("state", "helper", "profile"):+ p = root / name; p.mkdir()+ for file in ("runtime.token", "agent-id", "connection.env", "install.sh"):+ (p / file).write_text("synthetic recovery only\n")+ (root / "dsm").mkdir()+ (root / "model.json").write_text(json.dumps(dict(fault=fault, state=state, manual=False,+ status_reads=0, process_reads=0, reloads=0)))+ before = self.snapshot(root)+ def redirect(source):+ source = source.replace("/etc/init/" + "$" + "{AGENT_NAME}", str(init / "pulse-agent"))+ source = source.replace("[[ -d /usr/syno ]]", "[[ -d " + shlex.quote(str(root / "dsm")) + " ]]")+ source = source.replace("[[ -d /etc/init.d ]]", "[[ -d " + shlex.quote(str(root / "no-sysv")) + " ]]")+ return source.replace("/var/log/pulse-agent.log", str(root / "absent-log"))+ values = dict(FIXTURE=root, EVENTS=root / "events", STATE_DIR=root / "state",+ PRIVILEGED_HELPER_STATE_DIR=root / "helper", SAFE_PROFILE_STATE_DIR=root / "profile",+ PRIVILEGE_SUDOERS_FILE=root / "absent-sudoers", PRIVILEGE_HELPER_DIR=root / "absent-helper",+ REPLY=reply, FAULT=fault)+ setup = "\n".join(k + "=" + shlex.quote(str(v)) for k, v in values.items()) + "\n"+ shell = r'''+set -euo pipefail+UNINSTALL=true AGENT_NAME=pulse-agent BINARY_NAME=pulse-agent INSTALL_DIR=/usr/local/bin+EXIT_GENERAL=1 EXIT_MISSING_ARGS=16 PULSE_URL=https://same-instance.example.invalid AGENT_ID=synthetic+ACTION_RUNNER_NAME=pulse-agent-runner PRIVILEGED_HELPER_NAME=pulse-agent-helper+log_info() { :; }; log_warn() { :; }+fail() { printf '%s\n' "$1" >&2; exit "$2"; }+uname() { printf 'Linux\n'; }+command() {+ case "$*" in+ '-v systemctl'|'-v rc-service') return 1 ;;+ '-v initctl') [[ "$FAULT" != missing-initctl ]] || return 1 ;;+ esac+ builtin command "$@"+}+if [[ "$FAULT" == missing-initctl ]]; then initctl() { return 127; }; fi+if [[ "$FAULT" == missing-timeout ]]; then timeout() { return 127; }; fi+collector_credential_state_present() { return 0; }+uninstall_collector_registration() { printf 'remote:remove\n' >> "$EVENTS"; [[ "$REPLY" == confirmed ]]; }+pkill() { printf 'unchecked:process-stop\n' >> "$EVENTS"; }+sleep() { :; }+rm() {+ local arg+ for arg in "$@"; do [[ "$arg" == -* || "$arg" == "$FIXTURE/"* ]] || return 97; done+ printf 'file:remove\n' >> "$EVENTS"+ builtin command rm "$@"+}+mv() {+ local arg+ for arg in "$@"; do [[ "$arg" == -- || "$arg" == "$FIXTURE/"* ]] || return 97; done+ [[ "$FAULT" != refused-retirement-move || "$1" != -- || "$2" != "$FIXTURE/init/pulse-agent.conf" ]] || return 1+ printf 'file:move\n' >> "$EVENTS"+ builtin command mv "$@"+}+remove_agent_state_dir() { printf 'state:remove\n' >> "$EVENTS"; builtin command rm -r -- "$STATE_DIR"; }+remove_privileged_helper_state_dir() { builtin command rm -r -- "$PRIVILEGED_HELPER_STATE_DIR"; }+remove_safe_profile_state_dir() { builtin command rm -r -- "$SAFE_PROFILE_STATE_DIR"; }+'''+ if flow == "caller":+ entry = self.source.split("# --- Uninstall Logic ---\n", 1)[1].split(" # Unraid\n", 1)[0] + "fi\n"+ invoke = "main() {\n" + redirect(entry) + "}\nmain\n"+ elif flow == "status":+ invoke = "read_upstart_agent_removal_status\n"+ else:+ invoke = ("teardown_upstart_agent_service\n") * (2 if repeat else 1)+ env = dict(os.environ, PATH=str(tools) + ":" + os.environ["PATH"], FIXTURE=str(root))+ result = subprocess.run(["bash", "-c", setup + shell + redirect(definitions(self.source)) + invoke],+ cwd=root, env=env, text=True, capture_output=True, timeout=15)+ events = (root / "events").read_text().splitlines() if (root / "events").exists() else []+ model = json.loads((root / "model.json").read_text())+ if (root / "timed-pid").exists():+ try: os.kill(int((root / "timed-pid").read_text()), 0); model["timed_child_absent"] = False+ except ProcessLookupError: model["timed_child_absent"] = True+ return result, events, before, self.snapshot(root), model++ @staticmethod+ def snapshot(root):+ result = {}+ for path in root.rglob("*"):+ if path.name in ("model.json", "model.pending", "events", "timed-pid") or "tools" in path.parts:+ continue+ mode = path.lstat().st_mode+ payload = os.readlink(path) if stat.S_ISLNK(mode) else (path.read_bytes() if stat.S_ISREG(mode) else None)+ result[str(path.relative_to(root))] = (mode, path.lstat().st_ino, payload)+ return result++ def assert_recovery(self, before, after):+ for name, value in before.items():+ self.assertIn(name, after, name)+ self.assertEqual(after[name], value, name)+ added = set(after) - set(before)+ self.assertTrue(added <= {"init/pulse-agent.override"}, added)+ if added: self.assertEqual(after["init/pulse-agent.override"][2], b"manual\n")++ def test_refused_manager_precedes_remote_and_state_erasure(self):+ for fault in ("refused-stop", "refused-reload", "ignored-stop", "ignored-manual"):+ with self.subTest(fault=fault):+ r, events, before, after, _ = self.exercise(fault=fault)+ self.assertNotEqual(r.returncode, 0, r.stderr)+ self.assertNotIn("remote:remove", events)+ self.assertNotIn("state:remove", events)+ self.assert_recovery(before, after)++ def test_failed_empty_partial_diagnostic_status_cannot_authorise_removal(self):+ for fault in ("failed-status", "empty-status", "unknown-status", "duplicate-status",+ "diagnostic-status", "blank-status", "wrong-status-exit"):+ with self.subTest(fault=fault):+ r, events, before, after, _ = self.exercise(fault=fault)+ self.assertNotEqual(r.returncode, 0, r.stderr)+ self.assertEqual(after, before)+ self.assertNotIn("remote:remove", events)+ self.assertEqual(events.count("manager:status"), 1)++ def test_effective_disabled_config_is_required_after_override_reload(self):+ for fault in ("failed-config", "empty-config", "foreign-config", "diagnostic-config",+ "duplicate-config", "blank-config"):+ with self.subTest(fault=fault):+ r, events, before, after, _ = self.exercise(fault=fault)+ self.assertNotEqual(r.returncode, 0, r.stderr)+ self.assertNotIn("remote:remove", events)+ self.assertNotIn("manager:stop", events)+ self.assert_recovery(before, after)++ def test_independent_process_inventory_rejects_orphan_or_unavailable_read(self):+ for fault in ("failed-process", "empty-process", "duplicate-process", "diagnostic-process", "orphan-process"):+ with self.subTest(fault=fault):+ r, events, before, after, _ = self.exercise(fault=fault, state="stopped")+ self.assertNotEqual(r.returncode, 0, r.stderr)+ self.assertEqual(after, before)+ self.assertNotIn("remote:remove", events)++ def test_untrusted_config_override_and_prior_retirement_refuse_without_changes(self):+ cases = [dict(conf=c) for c in ("linked", "directory", "foreign", "writable", "oversized")]+ cases += [dict(override=o) for o in ("linked", "foreign")]+ cases += [dict(fault="old-retirement")]+ for case in cases:+ with self.subTest(case=case):+ r, events, before, after, _ = self.exercise(**case)+ self.assertNotEqual(r.returncode, 0, r.stderr)+ self.assertEqual(after, before)+ self.assertEqual(events, [])++ def test_missing_tools_refuse_without_remote_or_erasure(self):+ for fault in ("missing-initctl", "missing-timeout"):+ with self.subTest(fault=fault):+ r, events, before, after, _ = self.exercise(fault=fault)+ self.assertNotEqual(r.returncode, 0, r.stderr)+ self.assertEqual(after, before)+ self.assertNotIn("remote:remove", events)++ def test_remote_refusal_keeps_stopped_disabled_recovery_without_restart(self):+ for reply in ("refused", "unknown"):+ with self.subTest(reply=reply):+ r, events, before, after, model = self.exercise(reply=reply)+ self.assertNotEqual(r.returncode, 0, r.stderr)+ self.assertEqual(model["state"], "stopped")+ self.assertTrue(model["manual"])+ self.assert_recovery(before, after)+ self.assertNotIn("state:remove", events)+ self.assertNotIn("file:move", events)++ def test_late_manager_refusal_retains_recovery_after_remote_confirmation(self):+ r, events, before, after, _ = self.exercise(fault="failed-after-remote")+ self.assertNotEqual(r.returncode, 0, r.stderr)+ self.assertIn("remote:remove", events)+ self.assertNotIn("state:remove", events)+ self.assert_recovery(before, after)++ def test_unconfirmed_job_retirement_restores_exact_config_and_retains_manual(self):+ for fault in ("refused-retirement-reload", "failed-retirement-status",+ "failed-retirement-process", "retirement-orphan", "refused-retirement-move"):+ with self.subTest(fault=fault):+ r, events, before, after, _ = self.exercise(fault=fault)+ self.assertNotEqual(r.returncode, 0, r.stderr)+ self.assertIn("remote:remove", events)+ self.assertNotIn("state:remove", events)+ self.assert_recovery(before, after)++ def test_confirmed_job_absence_precedes_selected_local_erasure(self):+ r, events, before, after, model = self.exercise()+ self.assertEqual(r.returncode, 0, r.stderr)+ self.assertEqual(model["state"], "absent")+ self.assertNotIn("init/pulse-agent.conf", after)+ self.assertNotIn("init/pulse-agent.override", after)+ self.assertNotIn("state", after)+ self.assertEqual(after["unrelated"], before["unrelated"])+ self.assertLess(events.index("manager:stop"), events.index("remote:remove"))+ self.assertLess(events.index("remote:remove"), events.index("file:move"))+ self.assertLess(events.index("file:move"), events.index("state:remove"))+ last_inventory = max(i for i, e in enumerate(events) if e == "process:read")+ self.assertLess(last_inventory, events.index("state:remove"))+ self.assertNotIn("syno", r.stdout + r.stderr) # No configuration/argument dump.++ def test_absent_stopped_and_repeated_removal_require_observed_manager_absence(self):+ for case in (dict(conf="missing", state="absent"), dict(state="stopped"), dict(override="manual")):+ with self.subTest(case=case):+ r, events, before, after, _ = self.exercise(flow="teardown", repeat=True, **case)+ self.assertEqual(r.returncode, 0, r.stderr)+ self.assertNotIn("init/pulse-agent.conf", after)+ self.assertNotIn("init/pulse-agent.override", after)+ for name in before:+ if name.startswith(("state", "helper", "profile")):+ self.assertEqual(after[name], before[name])+ for case in (dict(conf="missing", state="started"), dict(conf="regular", state="absent")):+ with self.subTest(case=case):+ r, _, before, after, _ = self.exercise(flow="teardown", **case)+ self.assertNotEqual(r.returncode, 0)+ self.assertEqual(after, before)++ def test_source_selection_keeps_upstart_out_of_sysv_and_before_erasure(self):+ entry = self.source.split("# --- Uninstall Logic ---\n", 1)[1]+ self.assertEqual(entry.count("quiesce_upstart_agent_for_removal"), 1)+ self.assertEqual(entry.count("teardown_upstart_agent_service"), 1)+ admission = entry.index("quiesce_upstart_agent_for_removal")+ remote = entry.index("if ! uninstall_collector_registration;")+ retirement = entry.index("teardown_upstart_agent_service")+ erase = entry.index('if ! remove_agent_state_dir "$STATE_DIR";')+ self.assertLess(admission, remote)+ self.assertLess(remote, retirement)+ self.assertLess(retirement, erase)+ self.assertIn('[[ "$uninstall_upstart" != true ]]', entry)+ self.assertNotIn('initctl stop', entry)+ self.assertNotIn('initctl start', definitions(self.source))++ def test_real_status_timeout_stops_owned_child_without_erasure(self):+ r, events, before, after, model = self.exercise(fault="timeout-status")+ self.assertNotEqual(r.returncode, 0, r.stderr)+ self.assertEqual(after, before)+ self.assertNotIn("remote:remove", events)+ self.assertTrue(model["timed_child_absent"])+++if __name__ == "__main__":+ unittest.main(verbosity=2)diff --git a/scripts/installtests/install_sh_test.go b/scripts/installtests/install_sh_test.goindex 18e7fd13d9..913d1b04d9 100644--- a/scripts/installtests/install_sh_test.go+++ b/scripts/installtests/install_sh_test.go@@ -8463,3 +8463,39 @@ func TestInstallSHLegacyTeardownRequiresObservedQuiescence(t *testing.T) {}t.Logf("%s", output)}++func TestInstallSHUpstartTeardownRequiresObservedQuiescence(t *testing.T) {+ if runtime.GOOS == "windows" {+ t.Skip("DSM Upstart shell controls; Windows has its own native SCM gate")+ }+ ctx, cancel := context.WithTimeout(context.Background(), time.Minute)+ defer cancel()+ cmd := exec.CommandContext(ctx, "python3", repoFile("scripts", "installtests", "agent_upstart_teardown.py"))+ output, err := cmd.CombinedOutput()+ if err != nil {+ t.Fatalf("DSM Upstart removal controls: %v\n%s", err, output)+ }+ for _, name := range []string{+ "test_refused_manager_precedes_remote_and_state_erasure",+ "test_failed_empty_partial_diagnostic_status_cannot_authorise_removal",+ "test_effective_disabled_config_is_required_after_override_reload",+ "test_independent_process_inventory_rejects_orphan_or_unavailable_read",+ "test_untrusted_config_override_and_prior_retirement_refuse_without_changes",+ "test_missing_tools_refuse_without_remote_or_erasure",+ "test_remote_refusal_keeps_stopped_disabled_recovery_without_restart",+ "test_late_manager_refusal_retains_recovery_after_remote_confirmation",+ "test_unconfirmed_job_retirement_restores_exact_config_and_retains_manual",+ "test_confirmed_job_absence_precedes_selected_local_erasure",+ "test_absent_stopped_and_repeated_removal_require_observed_manager_absence",+ "test_source_selection_keeps_upstart_out_of_sysv_and_before_erasure",+ "test_real_status_timeout_stops_owned_child_without_erasure",+ } {+ if !strings.Contains(string(output), name+" (__main__.AgentUpstartTeardown."+name+") ... ok") {+ t.Fatalf("DSM Upstart control %s did not pass:\n%s", name, output)+ }+ }+ if !strings.Contains(string(output), "\nRan 13 tests in ") || !strings.Contains(string(output), "\nOK\n") {+ t.Fatalf("incomplete DSM Upstart verdict:\n%s", output)+ }+ t.Logf("%s", output)+}