Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Retain NAS agent recovery until boot and watchdog owners are quiescent
pulse-triage[bot]edb4b922f3d8
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 811bbb6cec..36ab893d03 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -15,6 +15,54 @@## Purpose+### Unraid/QNAP removal retires exact boot entries and confirms watchdog quiescence++The Unix uninstall caller selects these platforms from their vendor/installed+footprints, not from an ordinary recovered Linux state directory or QNAP state+finder alone. Existing exact state-removal authority must select the wrapper's+state directory. Regular root-owned files and parents are required; FAT boot+volumes cannot be assumed to implement Unix chmod semantics. Unknown wrapper+ownership and missing observation tools refuse before mutation.++One complete bounded boot-file parse admits only the exact generated Unraid+launch or canonical QNAP deferred block/legacy direct launch for that wrapper.+All duplicates retire; unrelated entries (including co-installed names and+legacy Telegraf) remain. Foreign/indirect targets, partial or changed blocks,+linked/non-regular files, binary/incomplete content and over-1-MiB files refuse+without evaluating or disclosing their text. The original remains in a private+`.pulse-removal.*` recovery directory outside the active boot namespace through+remote confirmation. A changed source during staging or a concurrently created+file, directory or symlink refuses without clobbering it; interrupted staging+retains the actual original and never restores automatic activation. New boot+content uses no-clobber file creation, compatible with FAT without hard links;+this is not an atomic whole-content swap or a lock against external root writers.++Complete bounded kernel evidence returns only numeric IDs and roles. Only exact+selected watchdog/collector processes receive checked TERM, watchdog first,+with independent absence observations. Editor sessions, backup wrappers and+co-installed names are not targets. Relative/foreign same-name processes or+unattributable QNAP deferred autorun shells refuse instead of signalling another+boot owner. Failed/diagnostic/duplicate/empty reads and ignored/refused signals+retain identity, credentials, trust and recoverable files. QNAP mount and+unmount each have checked bounded results, including cleanup after partial mount.++Whole boot/process and applicable init-manager quiescence closes again before+same-instance server removal and before local erasure; late activation refuses,+even after remote removal succeeded. NAS paths do not subsequently run the+unchecked global pkill or delete arbitrary discovered QNAP state/whole legacy+Unraid directories. Successful/absent/repeated removal still requires complete+fresh observations. No automatic restart, credential churn or transport-policy+change follows a partial operation.++`TestInstallSHNASBootOwnersQuiesceBeforeRecoveryErasure` requires all seventeen+actual-function/full-uninstall caller controls in `agent_nas_teardown.py`. The+boot filesystem transformations are real private user operations; flash/kernel+faults are closed models. Two generated watchdog/child controls use real private+ordinary-user processes, not appliance mounts or root services. Native signed+Unraid/QNAP/Linux/PowerShell, TLS/identity/repetition/cleanup and containing+publication remain separate obligations; this source proof does not resolve+#1099 or claim installed restoration.+### TrueNAS SCALE removal retires every owned middleware boot task before recovery erasureTrueNAS SCALE has an independent activation owner: its Init/Shutdown bootstrapdiff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.mdindex 1f61cabe18..c083060b4a 100644--- a/docs/release-control/v6/internal/subsystems/deployment-installability.md+++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md@@ -16,6 +16,54 @@## Purpose+### Unraid/QNAP removal retires exact boot entries and confirms watchdog quiescence++The Unix uninstall caller selects these platforms from their vendor/installed+footprints, not from an ordinary recovered Linux state directory or QNAP state+finder alone. Existing exact state-removal authority must select the wrapper's+state directory. Regular root-owned files and parents are required; FAT boot+volumes cannot be assumed to implement Unix chmod semantics. Unknown wrapper+ownership and missing observation tools refuse before mutation.++One complete bounded boot-file parse admits only the exact generated Unraid+launch or canonical QNAP deferred block/legacy direct launch for that wrapper.+All duplicates retire; unrelated entries (including co-installed names and+legacy Telegraf) remain. Foreign/indirect targets, partial or changed blocks,+linked/non-regular files, binary/incomplete content and over-1-MiB files refuse+without evaluating or disclosing their text. The original remains in a private+`.pulse-removal.*` recovery directory outside the active boot namespace through+remote confirmation. A changed source during staging or a concurrently created+file, directory or symlink refuses without clobbering it; interrupted staging+retains the actual original and never restores automatic activation. New boot+content uses no-clobber file creation, compatible with FAT without hard links;+this is not an atomic whole-content swap or a lock against external root writers.++Complete bounded kernel evidence returns only numeric IDs and roles. Only exact+selected watchdog/collector processes receive checked TERM, watchdog first,+with independent absence observations. Editor sessions, backup wrappers and+co-installed names are not targets. Relative/foreign same-name processes or+unattributable QNAP deferred autorun shells refuse instead of signalling another+boot owner. Failed/diagnostic/duplicate/empty reads and ignored/refused signals+retain identity, credentials, trust and recoverable files. QNAP mount and+unmount each have checked bounded results, including cleanup after partial mount.++Whole boot/process and applicable init-manager quiescence closes again before+same-instance server removal and before local erasure; late activation refuses,+even after remote removal succeeded. NAS paths do not subsequently run the+unchecked global pkill or delete arbitrary discovered QNAP state/whole legacy+Unraid directories. Successful/absent/repeated removal still requires complete+fresh observations. No automatic restart, credential churn or transport-policy+change follows a partial operation.++`TestInstallSHNASBootOwnersQuiesceBeforeRecoveryErasure` requires all seventeen+actual-function/full-uninstall caller controls in `agent_nas_teardown.py`. The+boot filesystem transformations are real private user operations; flash/kernel+faults are closed models. Two generated watchdog/child controls use real private+ordinary-user processes, not appliance mounts or root services. Native signed+Unraid/QNAP/Linux/PowerShell, TLS/identity/repetition/cleanup and containing+publication remain separate obligations; this source proof does not resolve+#1099 or claim installed restoration.+### TrueNAS SCALE removal retires every owned middleware boot task before recovery erasureTrueNAS SCALE has an independent activation owner: its Init/Shutdown bootstrapdiff --git a/scripts/install.sh b/scripts/install.shindex 91bf5cb06e..1103b644c3 100755--- a/scripts/install.sh+++ b/scripts/install.sh@@ -3872,6 +3872,262 @@ find_qnap_state_dir() {return 1}+# Unraid/QNAP have boot-file and watchdog owners independent of init managers.+# A recovered ordinary Linux state directory is not a QNAP installation.+nas_agent_removal_platform() {+ [[ "$(uname -s)" == Linux ]] || return 1+ if [[ -f /etc/unraid-version || -d /boot/config/plugins/pulse-agent ]]; then+ printf 'Unraid\n'+ elif [[ -f /sbin/getcfg || -f /etc/config/qpkg.conf ]]; then+ printf 'QNAP\n'+ else+ return 1+ fi+}++nas_agent_removal_file_safe() {+ local path="$1" parent="" uid=""+ [[ "$path" == /* && -f "$path" && ! -L "$path" && -r "$path" ]] || return 1+ parent=$(dirname "$path") || return 1+ [[ -d "$parent" && ! -L "$parent" ]] || return 1+ uid=$(id -u) || return 1+ [[ "$(portable_path_uid "$path")" == "$uid" && "$(portable_path_uid "$parent")" == "$uid" ]]+ # FAT boot volumes do not implement Unix chmod semantics. Exact ownership,+ # regular-file and complete-content checks apply without inventing mode bits.+}++nas_agent_removal_paths_safe() {+ local platform="$1" wrapper="$STATE_DIR/start-pulse-agent.sh" size=""+ [[ "$STATE_DIR" == /* && "$STATE_DIR" != / &&+ "$STATE_DIR" != *$'\n'* && "$STATE_DIR" != *$'\r'* &&+ "${STATE_DIR_REMOVAL_AUTHORITY:-}" == "$STATE_DIR" && ! -L "$STATE_DIR" ]] || return 1+ if [[ -e "$wrapper" || -L "$wrapper" ]]; then+ nas_agent_removal_file_safe "$wrapper" || return 1+ size=$(wc -c < "$wrapper") || return 1+ [[ "$size" =~ ^[[:space:]]*([0-9]+)[[:space:]]*$ && ${BASH_REMATCH[1]} -le 1048576 ]] || return 1+ grep -Fxq "# Pulse Agent startup script for $platform" "$wrapper" || return 1+ fi+}++# Filter only exact generated entries. Never evaluate boot text or remove every+# line containing pulse-agent; that also deletes unrelated tools and installs.+# status returns one fixed word, never the source (which may contain secrets).+filter_nas_agent_boot_file() {+ local platform="$1" path="$2" mode="$3" size="" expected="" last=""+ if [[ ! -e "$path" && ! -L "$path" ]]; then+ [[ "$mode" != status ]] || printf 'absent\n'+ return 0+ fi+ nas_agent_removal_file_safe "$path" || return 1+ size=$(wc -c < "$path") || return 1+ [[ "$size" =~ ^[[:space:]]*([0-9]+)[[:space:]]*$ && ${BASH_REMATCH[1]} -le 1048576 ]] || return 1+ if (( ${BASH_REMATCH[1]} > 0 )); then+ last=$(tail -c 1 "$path" && printf x) || return 1+ [[ "$last" == $'\nx' ]] || return 1+ fi+ if [[ "$platform" == QNAP ]]; then+ expected=$(render_qnap_autorun_block "$STATE_DIR/start-pulse-agent.sh") || return 1+ fi+ PULSE_NAS_BOOT_EXPECTED="$expected" LC_ALL=C awk -v platform="$platform" -v mode="$mode" \+ -v wrapper="$STATE_DIR/start-pulse-agent.sh" -v runtime="$INSTALL_DIR/$BINARY_NAME" '+ function launch(s) {+ if (platform == "Unraid") return s == "bash " wrapper || s == "bash \"" wrapper "\""+ return s == wrapper " &" || s == "\"" wrapper "\" &" ||+ s == "sh " wrapper " &" || s == "sh \"" wrapper "\" &"+ }+ function target(s, word, i,c) {+ i = index(s, word); if (!i) return 0+ c = substr(s, i + length(word), 1)+ return c == "" || c ~ /[[:space:]"\047;&|)]/+ }+ BEGIN { expected = ENVIRON["PULSE_NAS_BOOT_EXPECTED"]; owned = 0; block = ""; marker = 0; bad = 0 }+ {+ if ($0 ~ /[\000-\010\013-\037\177]/) { bad = 1; exit 1 }+ if (block != "") {+ block = block $0 "\n"+ if ($0 == "# Pulse Agent bootstrap end") {+ if (block != expected "\n") { bad = 1; exit 1 }+ block = ""; owned++; next+ }+ next+ }+ if ($0 == "# Pulse Agent bootstrap begin") {+ if (platform != "QNAP" || marker) { bad = 1; exit 1 }+ block = $0 "\n"; next+ }+ if (marker) {+ if (!launch($0)) { bad = 1; exit 1 }+ marker = 0; owned++; next+ }+ if ($0 == "# Pulse Agent") { marker = 1; next }+ if ($0 == "# Pulse Agent bootstrap end") { bad = 1; exit 1 }+ if (launch($0)) { owned++; next }+ if ($0 !~ /^[[:space:]]*#/ && (target($0, "start-pulse-agent.sh") || target($0, runtime))) {+ bad = 1; exit 1+ }+ if (mode == "filter") print+ }+ END {+ if (bad || marker || block != "") exit 1+ if (mode == "status") print (owned ? "present" : "absent")+ }+ ' "$path"+}++# Keep a private original outside the active boot namespace through remote+# confirmation. A partial edit remains disabled; never restore boot activation+# automatically. No-clobber creation refuses concurrent files/directories even+# on FAT, where hard links are unavailable. Failed staging stays for recovery.+disable_nas_agent_boot_file() {+ local platform="$1" path="$2" directory="" mode="" observed=""+ for directory in "$path".pulse-removal.*; do+ [[ ! -e "$directory" && ! -L "$directory" ]] || return 1+ done+ observed=$(filter_nas_agent_boot_file "$platform" "$path" status) || return 1+ [[ "$observed" == present || "$observed" == absent ]] || return 1+ [[ "$observed" != absent ]] || return 0+ directory=$(mktemp -d "$path.pulse-removal.XXXXXX") || return 1+ chmod 700 "$directory" || return 1+ cp "$path" "$directory/original" || return 1+ filter_nas_agent_boot_file "$platform" "$directory/original" filter > "$directory/disabled" || return 1+ mode=$(portable_path_mode "$path") || return 1+ nas_agent_removal_file_safe "$path" && cmp -s "$path" "$directory/original" || return 1+ mv "$path" "$directory/retired" || return 1+ # Catch a changed source during staging, retaining the actual retired bytes.+ nas_agent_removal_file_safe "$directory/retired" && cmp -s "$directory/retired" "$directory/original" || return 1+ (set -o noclobber; cat "$directory/disabled" > "$path") || return 1+ chmod "$mode" "$path" || return 1+ nas_agent_removal_file_safe "$path" && cmp -s "$path" "$directory/disabled" || return 1+ observed=$(filter_nas_agent_boot_file "$platform" "$path" status) || return 1+ [[ "$observed" == absent ]] || return 1+ NAS_REMOVAL_BOOT_RECOVERY="$directory"+}++nas_agent_boot_operation() {+ local platform="$1" operation="$2" observed="" code=0+ local path=/boot/config/go+ [[ -d /boot/config && ! -L /boot/config ]] || [[ "$platform" == QNAP ]] || return 1+ if [[ "$platform" == QNAP ]]; then+ [[ -x /etc/init.d/init_disk.sh ]] && command -v timeout >/dev/null 2>&1 || return 1+ if ! timeout -k 1 10 /etc/init.d/init_disk.sh mount_flash_config >/dev/null 2>&1; then+ # The mount may have partially succeeded; an owned unmount is still+ # attempted and checked. Neither failure permits remote/local removal.+ timeout -k 1 10 /etc/init.d/init_disk.sh umount_flash_config >/dev/null 2>&1 || return 1+ return 1+ fi+ path=/tmp/nasconfig_tmp/autorun.sh+ [[ -d /tmp/nasconfig_tmp && ! -L /tmp/nasconfig_tmp ]] || code=1+ fi+ if [[ "$code" == 0 ]]; then+ if [[ "$operation" == disable ]]; then+ disable_nas_agent_boot_file "$platform" "$path" || code=1+ else+ observed=$(filter_nas_agent_boot_file "$platform" "$path" status) || code=1+ [[ "$observed" == absent ]] || code=1+ if [[ "$code" == 0 && "$operation" == retire && -n "${NAS_REMOVAL_BOOT_RECOVERY:-}" ]]; then+ [[ "$NAS_REMOVAL_BOOT_RECOVERY" == "$path".pulse-removal.* &&+ -d "$NAS_REMOVAL_BOOT_RECOVERY" && ! -L "$NAS_REMOVAL_BOOT_RECOVERY" ]] &&+ rm -r -- "$NAS_REMOVAL_BOOT_RECOVERY" || code=1+ fi+ fi+ fi+ if [[ "$platform" == QNAP ]]; then+ timeout -k 1 10 /etc/init.d/init_disk.sh umount_flash_config >/dev/null 2>&1 || code=1+ fi+ return "$code"+}++# Read complete kernel evidence before signalling; expose only numeric IDs and+# roles. Relative/foreign wrappers and QNAP deferred autorun shells have unknown+# ownership, so retain recovery instead of signalling an unrelated boot owner.+read_nas_agent_removal_processes() {+ local platform="$1" observation="" line="" pid="" parent="" comm="" args="" role=""+ local wrapper="$STATE_DIR/start-pulse-agent.sh" runtime="$INSTALL_DIR/$BINARY_NAME"+ local complete=false count=0 seen=" " shell=""+ local -a rows=()+ observation=$(LC_ALL=C timeout -k 1 5 ps -ax -o pid= -o ppid= -o comm= -o args= 2>&1 &&+ printf '%s' '__pulse_nas_processes_complete__') || return 1+ while IFS= read -r line; do+ [[ "$complete" == false ]] || return 1+ if [[ "$line" == __pulse_nas_processes_complete__ ]]; then complete=true; continue; fi+ [[ "$line" =~ ^[[:space:]]*([1-9][0-9]*)[[:space:]]+([0-9]+)[[:space:]]+([^[:space:]]+)[[:space:]]+(.+)$ ]] || return 1+ pid="${BASH_REMATCH[1]}" parent="${BASH_REMATCH[2]}" comm="${BASH_REMATCH[3]}" args="${BASH_REMATCH[4]}"+ [[ "$seen" != *" $pid "* ]] || return 1+ seen+="$pid "; role=other+ if [[ "$args" == "$runtime" || "$args" == "$runtime "* ]]; then+ [[ "${comm##*/}" == "$BINARY_NAME" ]] || return 1+ role=collector+ elif [[ "${comm##*/}" == "$BINARY_NAME" ]]; then+ role=unknown+ fi+ case "${comm##*/}" in+ sh|bash|dash|ash|start-pulse-age|start-pulse-agent.sh)+ for shell in sh bash dash ash /bin/sh /bin/bash /bin/dash /bin/ash /usr/bin/sh /usr/bin/bash; do+ if [[ "$args" == "$shell $wrapper" || "$args" == "$shell $wrapper "* ]]; then role=watchdog; fi+ done+ if [[ "$args" == "$wrapper" || "$args" == "$wrapper "* ]]; then role=watchdog; fi+ if [[ "$role" != watchdog && "$args" =~ start-pulse-agent\.sh([[:space:]]|$) ]]; then role=unknown; fi+ if [[ "$platform" == QNAP && "$args" == *'/nasconfig_tmp/autorun.sh'* ]]; then role=unknown; fi+ ;;+ esac+ rows+=("$pid $parent $role"); count=$((count + 1))+ [[ $count -le 65536 ]] || return 1+ done <<< "$observation"+ [[ "$complete" == true && $count -gt 0 ]] || return 1+ printf '%s\n' "${rows[@]}"+}++nas_agent_processes_absent() {+ [[ "$1" != *' watchdog'* && "$1" != *' collector'* && "$1" != *' unknown'* ]]+}++quiesce_nas_agent_for_removal() {+ local platform="$1" inventory="" pid="" parent="" role=""+ command -v timeout >/dev/null 2>&1 && command -v ps >/dev/null 2>&1 || return 1+ nas_agent_removal_paths_safe "$platform" || return 1+ inventory=$(read_nas_agent_removal_processes "$platform") || return 1+ [[ "$inventory" != *' unknown'* ]] || return 1+ nas_agent_boot_operation "$platform" disable || return 1+ # Observe again after boot mutation; never signal PIDs from the old read.+ inventory=$(read_nas_agent_removal_processes "$platform") || return 1+ [[ "$inventory" != *' unknown'* ]] || return 1+ while read -r pid parent role; do+ [[ "$role" != watchdog ]] || kill -TERM "$pid" || return 1+ done <<< "$inventory"+ sleep 1+ inventory=$(read_nas_agent_removal_processes "$platform") || return 1+ [[ "$inventory" != *' watchdog'* && "$inventory" != *' unknown'* ]] || return 1+ while read -r pid parent role; do+ [[ "$role" != collector ]] || kill -TERM "$pid" || return 1+ done <<< "$inventory"+ sleep 1+ nas_agent_boot_operation "$platform" observe || return 1+ inventory=$(read_nas_agent_removal_processes "$platform") || return 1+ nas_agent_processes_absent "$inventory"+}++confirm_nas_agent_removal_quiescent() {+ local platform="$1" inventory="" unit=""+ nas_agent_boot_operation "$platform" observe || return 1+ inventory=$(read_nas_agent_removal_processes "$platform") || return 1+ nas_agent_processes_absent "$inventory" || return 1+ if command -v systemctl >/dev/null 2>&1; then+ for unit in "${ACTION_RUNNER_NAME}.service" "${PRIVILEGED_HELPER_NAME}.socket" \+ "${PRIVILEGED_HELPER_NAME}.service" "${AGENT_NAME}.service"; do+ confirm_agent_unit_quiescent "$unit" || return 1+ done+ fi+}++teardown_nas_agent_boot_owner() {+ local platform="$1"+ confirm_nas_agent_removal_quiescent "$platform" || return 1+ nas_agent_boot_operation "$platform" retire || return 1+ # Close state once more after flash unmount/recovery retirement, while the+ # credentials and wrapper still exist. No unchecked global pkill follows.+ confirm_nas_agent_removal_quiescent "$platform"+}+remove_qnap_autorun_block() {local autorun_path="$1"@@ -4024,18 +4280,9 @@ EOFchmod +x "$wrapper_script"}-append_qnap_autorun_block() {- local autorun_path="$1"- local wrapper_script="$2"- local state_dir="$3"-- remove_qnap_autorun_block "$autorun_path"- if [[ ! -f "$autorun_path" ]]; then- echo "#!/bin/sh" > "$autorun_path"- fi-- cat >> "$autorun_path" <<EOF-+render_qnap_autorun_block() {+ local wrapper_script="$1"+ cat <<EOF# Pulse Agent bootstrap begin(_pulse_wrapper="${wrapper_script}"@@ -4054,8 +4301,22 @@ append_qnap_autorun_block() {fi) >> /var/log/${AGENT_NAME}.log 2>&1 &# Pulse Agent bootstrap end-EOF+}++append_qnap_autorun_block() {+ local autorun_path="$1"+ local wrapper_script="$2"+ local state_dir="$3"++ remove_qnap_autorun_block "$autorun_path"+ if [[ ! -f "$autorun_path" ]]; then+ echo "#!/bin/sh" > "$autorun_path"+ fi++ printf '\n' >> "$autorun_path"+ render_qnap_autorun_block "$wrapper_script" >> "$autorun_path"+ printf '\n' >> "$autorun_path"chmod +x "$autorun_path"}@@ -6129,6 +6390,8 @@ if [[ "$UNINSTALL" == "true" ]]; thenlocal uninstall_upstart="false"local uninstall_truenas_scale="false"local uninstall_truenas_tasks=""+ local uninstall_nas_platform=""+ NAS_REMOVAL_BOOT_RECOVERY=""# Validate saved connection availability before stopping a working install.# Stop every systemd activation source before remote removal, process kills@@ -6139,6 +6402,11 @@ if [[ "$UNINSTALL" == "true" ]]; thenfail "The saved Pulse URL is missing; local credentials and services were retained. Restore saved connection state or provide --url for the same trusted Pulse instance and retry uninstall." "$EXIT_MISSING_ARGS"fifi+ uninstall_nas_platform=$(nas_agent_removal_platform || true)+ if [[ -n "$uninstall_nas_platform" ]]; then+ quiesce_nas_agent_for_removal "$uninstall_nas_platform" ||+ fail "NAS removal could not confirm the owned boot entry disabled and watchdog/collector stopped; credentials, wrapper and recovery files were retained. Reconcile boot state and any .pulse-removal recovery directory before retrying; no automatic restart was attempted." "$EXIT_GENERAL"+ fi# Middleware can recreate and enable the systemd unit at boot. Disable all# exact owned tasks before stopping services; a partial/refused observation# retains every recovery file and never restores automatic activation.@@ -6170,7 +6438,7 @@ if [[ "$UNINSTALL" == "true" ]]; thenuninstall_legacy_manager=FreeBSDquiesce_freebsd_agent_for_removal ||fail "FreeBSD removal could not confirm stopped rc.d, disabled boot activation and absent supervisor/child; local recovery files were retained. Reconcile linked/appliance boot tasks first when present; no automatic restart was attempted." "$EXIT_GENERAL"- elif [[ "$uninstall_upstart" != true ]] && ! command -v systemctl >/dev/null 2>&1 && ! command -v rc-service >/dev/null 2>&1 && [[ -d /etc/init.d ]]; then+ elif [[ "$uninstall_upstart" != true && -z "$uninstall_nas_platform" ]] && ! command -v systemctl >/dev/null 2>&1 && ! command -v rc-service >/dev/null 2>&1 && [[ -d /etc/init.d ]]; thenuninstall_legacy_manager=SysVquiesce_sysv_agent_for_removal ||fail "SysV removal could not confirm stopped process and disabled runlevels; local recovery files were retained. A preceding stop or disable may have succeeded; no automatic restart was attempted." "$EXIT_GENERAL"@@ -6180,6 +6448,11 @@ if [[ "$UNINSTALL" == "true" ]]; thenfail "TrueNAS SCALE removal could not confirm stopped units, collector and bootstrap processes; local recovery was retained. Reconcile stopped/disabled state before retrying; no automatic restart was attempted." "$EXIT_GENERAL"fi+ if [[ -n "$uninstall_nas_platform" ]]; then+ confirm_nas_agent_removal_quiescent "$uninstall_nas_platform" ||+ fail "NAS boot or process state changed before server removal; local recovery was retained. Reconcile stopped and disabled state before retrying; no automatic restart was attempted." "$EXIT_GENERAL"+ fi+# A credential-bearing install must durably remove its exact server record# through the same CA/fingerprint/no-proxy transport used by lifecycle# migration before any local credential or service state is deleted.@@ -6215,18 +6488,23 @@ if [[ "$UNINSTALL" == "true" ]]; thenfail "TrueNAS SCALE boot-task retirement was not confirmed; local recovery was retained. A preceding task disable/delete or server removal may have succeeded. Reconcile middleware and stopped service state before retrying; no automatic restart was attempted." "$EXIT_GENERAL"fi- # Kill wrapper scripts first: they are watchdogs, so stopping the agent- # while its wrapper still loops only races the respawn. No leading path- # separator here, unlike the install paths, so a wrapper invoked by a- # relative path or left at an older location is still caught. The dot is- # escaped and the far end bounded so an editor session or a .bak copy of- # the wrapper is not swept up with it.- pkill -f "start-pulse-agent\.sh([[:space:]]|$)" 2>/dev/null || true- # Then the agent itself.- # Use -x (exact process name match) to avoid killing THIS uninstall script,- # whose command line path contains "pulse-agent" (e.g. /boot/config/plugins/pulse-agent/install.sh).- pkill -x "pulse-agent" 2>/dev/null || true- sleep 1+ if [[ -n "$uninstall_nas_platform" ]]; then+ teardown_nas_agent_boot_owner "$uninstall_nas_platform" ||+ fail "NAS boot retirement or final quiescence was not confirmed; local recovery was retained even if server removal succeeded. Reconcile boot/process state and any .pulse-removal directory before retrying; no automatic restart was attempted." "$EXIT_GENERAL"+ else+ # Kill wrapper scripts first: they are watchdogs, so stopping the agent+ # while its wrapper still loops only races the respawn. No leading path+ # separator here, unlike the install paths, so a wrapper invoked by a+ # relative path or left at an older location is still caught. The dot is+ # escaped and the far end bounded so an editor session or a .bak copy of+ # the wrapper is not swept up with it.+ pkill -f "start-pulse-agent\.sh([[:space:]]|$)" 2>/dev/null || true+ # Then the agent itself.+ # Use -x (exact process name match) to avoid killing THIS uninstall script,+ # whose command line path contains "pulse-agent" (e.g. /boot/config/plugins/pulse-agent/install.sh).+ pkill -x "pulse-agent" 2>/dev/null || true+ sleep 1+ fi# Systemd - unified agentif command -v systemctl >/dev/null 2>&1; then@@ -6275,57 +6553,10 @@ if [[ "$UNINSTALL" == "true" ]]; then# DSM 6.x Upstart retirement was confirmed before recovery erasure.fi- # Unraid- if [[ -f /etc/unraid-version ]] || [[ -d /boot/config/plugins/pulse-agent ]]; then- log_info "Removing Unraid installation..."- # Stop the wrapper watchdogs before the agents they supervise, and keep- # the match bounded so a .bak copy or an editor session on the wrapper- # is not swept up.- pkill -f "start-pulse-agent\.sh([[:space:]]|$)" 2>/dev/null || true- pkill -x "pulse-agent" 2>/dev/null || true- sleep 1-- # Remove from /boot/config/go - all pulse-related entries- GO_SCRIPT="/boot/config/go"- if [[ -f "$GO_SCRIPT" ]]; then- # Remove unified agent entries (line-by-line, not range-based,- # to avoid consuming adjacent non-pulse entries when no trailing- # blank line separates them).- sed -i '/^# Pulse Agent$/d' "$GO_SCRIPT" 2>/dev/null || true- sed -i '/pulse-agent/d' "$GO_SCRIPT" 2>/dev/null || true- fi-- # Remove installation directories- rm -rf /boot/config/plugins/pulse-agent- rm -rf /boot/config/pulse # Legacy pulse directory-- # Remove binaries from RAM disk- rm -f "${INSTALL_DIR}/${BINARY_NAME}"-- # Remove log directory- rm -rf /var/log/pulse- fi-- # QNAP QTS/QuTS hero- qnap_state_dir=$(find_qnap_state_dir || true)- if [[ -n "$qnap_state_dir" ]] || [[ -f /sbin/getcfg ]] || [[ -f /etc/config/qpkg.conf ]]; then- log_info "Removing QNAP installation..."- if [[ -x /etc/init.d/init_disk.sh ]]; then- if /etc/init.d/init_disk.sh mount_flash_config 2>/dev/null && [[ -d /tmp/nasconfig_tmp ]]; then- AUTORUN_PATH="/tmp/nasconfig_tmp/autorun.sh"- if [[ -f "$AUTORUN_PATH" ]]; then- remove_qnap_autorun_block "$AUTORUN_PATH"- fi- /etc/init.d/init_disk.sh umount_flash_config 2>/dev/null || true- else- /etc/init.d/init_disk.sh umount_flash_config 2>/dev/null || true- log_warn "Could not mount QNAP flash config to remove autorun.sh entry."- fi- fi- if [[ -n "$qnap_state_dir" ]]; then- rm -rf "$qnap_state_dir"- fi- fi+ # NAS boot/watchdog retirement completed above. Erase only the selected+ # authorised state directory through remove_agent_state_dir, never an entire+ # legacy /boot/config/pulse tree (which can also contain unrelated tools).+ # QNAP state discovery is not independent platform or deletion authority.# SCALE's task retirement was confirmed above, before authorised state# erasure. A recovered ordinary Linux directory is not a TrueNAS install.diff --git a/scripts/installtests/agent_nas_teardown.py b/scripts/installtests/agent_nas_teardown.pynew file mode 100644index 0000000000..ebaba1af85--- /dev/null+++ b/scripts/installtests/agent_nas_teardown.py@@ -0,0 +1,442 @@+#!/usr/bin/env python3+"""Actual NAS uninstall caller and boot transformations in private user fixtures.++Flash and kernel/manager refusal controls are models. No host signal, mount,+service, credential, boot path, network or native appliance is exercised.+"""+import json+import os+from pathlib import Path+import shlex+import stat+import subprocess+import tempfile+import unittest++ROOT = Path(__file__).resolve().parents[2]+SOURCE = Path(os.environ.get("PULSE_AGENT_NAS_SOURCE", ROOT / "scripts/install.sh"))+FUNCTIONS = (+ "portable_path_uid", "portable_path_mode", "portable_sed_in_place",+ "nas_agent_removal_platform", "nas_agent_removal_file_safe", "nas_agent_removal_paths_safe",+ "filter_nas_agent_boot_file", "disable_nas_agent_boot_file", "nas_agent_boot_operation",+ "read_nas_agent_removal_processes", "nas_agent_processes_absent", "quiesce_nas_agent_for_removal",+ "confirm_nas_agent_removal_quiescent", "teardown_nas_agent_boot_owner",+ "remove_qnap_autorun_block", "render_qnap_autorun_block", "append_qnap_autorun_block",+)+++def definitions(source, names=FUNCTIONS):+ result = []+ for name in names:+ marker = name + "() {"+ if marker not in source: # Exact-parent adverse controls, not substitute code.+ continue+ if source.count(marker) != 1:+ raise ValueError("ambiguous production function " + name)+ result.append(marker + source.split(marker, 1)[1].split("\n}\n", 1)[0] + "\n}\n")+ return "\n".join(result)+++MODEL = r'''#!/usr/bin/python3+import json, os, pathlib, sys+root = pathlib.Path(os.environ['FIXTURE'])+path = root / 'model.json'+v = json.loads(path.read_text()); role, args = pathlib.Path(sys.argv[0]).name, sys.argv[1:]+fault = v['fault']+def event(s):+ with (root / 'events').open('a') as f: f.write(s + '\n')+def save():+ pending = root / 'model.pending'; pending.write_text(json.dumps(v)); pending.replace(path)+if role == 'ps':+ assert args == ['-ax', '-o', 'pid=', '-o', 'ppid=', '-o', 'comm=', '-o', 'args='], args+ event('process:read'); v['reads'] += 1+ if fault == 'reactivated-before-remote' and v['reads'] == 5: v['watchdog'] = True+ save()+ if fault == 'failed-process': sys.exit(2)+ if fault == 'empty-process': sys.exit(0)+ if fault == 'timeout-process':+ (root / 'timed-pid').write_text(str(os.getpid())); os.execl('/bin/sleep', 'sleep', '30')+ print('1 0 init /sbin/init')+ if fault == 'duplicate-process': print('1 0 init /sbin/init')+ if fault == 'diagnostic-process': print('read refused', file=sys.stderr)+ wrapper = str(root / 'state/start-pulse-agent.sh')+ if v['watchdog']:+ if fault == 'script-comm': print('100 1 start-pulse-age /bin/sh ' + wrapper)+ else: print('100 1 sh /bin/sh ' + wrapper if v['platform'] == 'QNAP' else '100 1 bash /bin/bash ' + wrapper)+ if v['collector']: print('101 100 pulse-agent ' + str(root / 'bin/pulse-agent') + ' --token-file /synthetic/private-only')+ if fault == 'foreign-process': print('102 1 pulse-agent /other/instance/pulse-agent --token-file /synthetic/private-only')+ if fault == 'relative-wrapper': print('103 1 sh sh start-pulse-agent.sh')+ if fault == 'deferred-bootstrap': print('104 1 sh /bin/sh /tmp/nasconfig_tmp/autorun.sh')+ print('200 1 vi vi ' + wrapper) # Editor is not a watchdog or signal target.+ print('201 1 sh /bin/sh ' + wrapper + '.bak')+ print('202 1 pulse-agent-prod /other/pulse-agent-prod')+ sys.exit(0)+if role == 'kill':+ assert len(args) == 2 and args[0] == '-TERM', args+ pid = int(args[1]); assert pid in (100, 101), 'unrelated process signalled'+ event('signal:' + str(pid))+ if fault == 'refused-watchdog-stop' and pid == 100: sys.exit(1)+ if fault == 'refused-collector-stop' and pid == 101: sys.exit(1)+ if pid == 100 and fault != 'ignored-watchdog-stop': v['watchdog'] = False+ if pid == 101 and fault != 'ignored-collector-stop': v['collector'] = False+ save(); sys.exit(0)+assert role == 'flash' and args[0] in ('mount_flash_config', 'umount_flash_config'), (role, args)+verb = args[0]; event('flash:' + verb)+if fault == 'mount-refusal' and verb == 'mount_flash_config': sys.exit(1)+if fault == 'unmount-refusal' and verb == 'umount_flash_config': sys.exit(1)+sys.exit(0)+'''+++class AgentNASTeardown(unittest.TestCase):+ source = SOURCE.read_text()++ def exercise(self, platform, fault="", *, reply="confirmed", boot="canonical", wrapper="regular",+ state="selected", repeat=False, flow="caller", appliance=True):+ with tempfile.TemporaryDirectory() as directory:+ root = Path(directory)+ tools = root / "tools"; tools.mkdir()+ for name in ("ps", "kill", "flash"):+ p = tools / name; p.write_text(MODEL); p.chmod(0o700)+ for name in ("state", "helper", "profile", "bin", "flash", "legacy", "foreign"):+ d = root / name; d.mkdir(mode=0o700)+ if name in ("state", "helper", "profile", "foreign"):+ for f in ("runtime.token", "agent-id", "connection.env", "install.sh"):+ (d / f).write_text("synthetic recovery only\n")+ (root / "legacy/telegraf").write_text("unrelated legacy tool\n")+ target = root / "state/start-pulse-agent.sh"+ other = root / "unrelated"; other.write_text("unchanged unrelated file\n")+ if wrapper == "linked": target.symlink_to(other)+ elif wrapper == "directory": target.mkdir()+ elif wrapper != "missing":+ target.write_text("#!/bin/sh\n# Pulse Agent startup script for " + ("foreign" if wrapper == "foreign" else platform) + "\n")+ target.chmod(0o700)+ vendor = root / "vendor"+ if appliance: vendor.write_text(platform + "\n")+ bootfile = root / "flash/autorun.sh" if platform == "QNAP" else root / "go"+ unrelated = "#!/bin/sh\necho before\nbash /other/start-pulse-agent-prod.sh\necho pulse-agent-prod\n/boot/config/pulse/telegraf/start_telegraf.sh\n"+ bootfile.write_text(unrelated)+ producer = definitions(self.source, ("portable_sed_in_place", "remove_qnap_autorun_block", "render_qnap_autorun_block", "append_qnap_autorun_block"))+ if platform == "QNAP":+ # Consume the real production bootstrap emitter, not a second literal.+ command = "AGENT_NAME=pulse-agent\nSTATE_DIR=" + shlex.quote(str(root / "state")) + "\n" + producer + "\nappend_qnap_autorun_block " + shlex.quote(str(bootfile)) + " " + shlex.quote(str(target)) + " " + shlex.quote(str(root / "state"))+ subprocess.run(["bash", "-c", command], check=True, capture_output=True, text=True)+ else:+ bootfile.write_text(unrelated + "# Pulse Agent\nbash " + str(target) + "\n")+ bootfile.write_text(bootfile.read_text() + "echo after\n")+ if boot == "duplicate":+ if platform == "QNAP":+ block = bootfile.read_text().split("# Pulse Agent bootstrap begin", 1)[1].split("# Pulse Agent bootstrap end", 1)[0]+ bootfile.write_text(bootfile.read_text() + "# Pulse Agent bootstrap begin" + block + "# Pulse Agent bootstrap end\n")+ else: bootfile.write_text(bootfile.read_text() + "# Pulse Agent\nbash " + str(target) + "\n")+ elif boot == "foreign": bootfile.write_text(bootfile.read_text().replace(str(target), "/other/start-pulse-agent.sh"))+ elif boot == "indirect": bootfile.write_text(bootfile.read_text() + "bash -c 'bash " + str(target) + "'\n")+ elif boot == "unterminated": bootfile.write_text(bootfile.read_text().replace("# Pulse Agent bootstrap end", "# unknown") if platform == "QNAP" else bootfile.read_text() + "# Pulse Agent\n")+ elif boot == "changed-block": bootfile.write_text(bootfile.read_text().replace("_pulse_wait_max=1800", "_pulse_wait_max=999") if platform == "QNAP" else bootfile.read_text().replace("bash " + str(target), "bash " + str(target) + " --unknown"))+ elif boot == "no-newline": bootfile.write_text(bootfile.read_text().rstrip("\n"))+ elif boot == "binary": bootfile.write_bytes(bootfile.read_bytes() + b'\0\n')+ elif boot == "oversized": bootfile.write_text(bootfile.read_text() + "#" * 1048577 + "\n")+ elif boot == "absent": bootfile.write_text(unrelated + "echo after\n")+ elif boot == "missing": bootfile.unlink()+ elif boot in ("linked", "directory"):+ bootfile.unlink()+ if boot == "linked": bootfile.symlink_to(other)+ else: bootfile.mkdir()+ original_boot = bootfile.read_bytes() if bootfile.is_file() and not bootfile.is_symlink() else None+ if fault == "prior-recovery": (Path(str(bootfile) + ".pulse-removal.retained")).mkdir()+ if fault == "missing-flash": (tools / "flash").unlink()+ model = dict(platform=platform, fault=fault, reads=0, watchdog=wrapper != "missing", collector=wrapper != "missing")+ if boot in ("absent", "missing") and wrapper == "missing": model.update(watchdog=False, collector=False)+ (root / "model.json").write_text(json.dumps(model))+ before = self.snapshot(root)++ replacements = {"/etc/unraid-version": str(vendor if platform == "Unraid" else root / "absent-vendor"),+ "/boot/config/plugins/pulse-agent": str(root / "absent-plugin"),+ "/etc/config/qpkg.conf": str(vendor if platform == "QNAP" else root / "absent-vendor"),+ "/sbin/getcfg": str(root / "absent-getcfg"),+ "/etc/init.d/init_disk.sh": str(tools / "flash"),+ "/tmp/nasconfig_tmp": str(root / "flash"),+ "/boot/config/go": str(root / "go"),+ "[[ -d /boot/config && ! -L /boot/config ]]": "[[ -d " + str(root) + " && ! -L " + str(root) + " ]]",+ "/boot/config/pulse": str(root / "legacy"),+ "[[ -d /etc/init.d ]]": "[[ -d " + str(root / "no-init") + " ]]",+ "/usr/syno": str(root / "no-dsm"),+ "/var/log/pulse-agent.log": str(root / "absent-log"),+ "/var/log/pulse": str(root / "absent-log-dir"),+ "/etc/truenas-version": str(root / "no-truenas"),+ "/etc/version": str(root / "no-version")}+ def redirect(text):+ for a, b in replacements.items(): text = text.replace(a, b)+ return text+ values = dict(FIXTURE=root, EVENTS=root / "events", PLATFORM=platform, FAULT=fault,+ STATE_DIR=root / "state", STATE_DIR_REMOVAL_AUTHORITY=(root / "state" if state == "selected" else root / "foreign"),+ INSTALL_DIR=root / "bin", PRIVILEGED_HELPER_STATE_DIR=root / "helper",+ SAFE_PROFILE_STATE_DIR=root / "profile", PRIVILEGE_SUDOERS_FILE=root / "absent-sudoers",+ PRIVILEGE_HELPER_DIR=root / "absent-helper", TRUENAS_STATE_DIR=root / "no-truenas-state",+ TRUENAS_BOOTSTRAP_SCRIPT=root / "no-bootstrap", BOOT_PATH=bootfile, REPLY=reply)+ setup = "\n".join(k + "=" + shlex.quote(str(v)) for k, v in values.items())+ shell = r'''+set -euo pipefail+UNINSTALL=true AGENT_NAME=pulse-agent BINARY_NAME=pulse-agent+EXIT_GENERAL=1 EXIT_MISSING_ARGS=16 PULSE_URL=https://same-instance.example.invalid AGENT_ID=synthetic+ACTION_RUNNER_NAME=pulse-agent-runner PRIVILEGED_HELPER_NAME=pulse-agent-helper+log_info() { :; }; log_warn() { :; }+fail() { printf '%s\n' "$1" >&2; exit "$2"; }+uname() { printf 'Linux\n'; }+command() {+ case "$*" in+ '-v systemctl'|'-v rc-service'|'-v initctl') return 1 ;;+ '-v timeout') [[ "$FAULT" != missing-timeout ]] || return 1 ;;+ '-v ps') [[ "$FAULT" != missing-ps ]] || return 1 ;;+ esac+ builtin command "$@"+}+truenas_scale_agent_removal_present() { return 1; }+collector_credential_state_present() { [[ -e "$STATE_DIR/runtime.token" ]]; }+uninstall_collector_registration() {+ printf 'remote:remove\n' >> "$EVENTS"+ if [[ "$FAULT" == late-boot ]]; then+ cp "$NAS_REMOVAL_BOOT_RECOVERY/original" "$BOOT_PATH"+ elif [[ "$FAULT" == late-process ]]; then+ python3 -c 'import json,os,pathlib; p=pathlib.Path(os.environ["FIXTURE"])/"model.json"; v=json.loads(p.read_text()); v["watchdog"]=True; p.write_text(json.dumps(v))'+ fi+ [[ "$REPLY" == confirmed ]]+}+find_qnap_state_dir() { [[ "$PLATFORM" != QNAP ]] || printf '%s\n' "$STATE_DIR"; }+pkill() { printf 'unchecked:process-stop\n' >> "$EVENTS"; }+kill() { builtin command "$FIXTURE/tools/kill" "$@"; }+sleep() { :; }+rm() {+ local arg+ for arg in "$@"; do [[ "$arg" == -* || "$arg" == "$FIXTURE/"* ]] || return 97; done+ printf 'file:remove\n' >> "$EVENTS"; builtin command rm "$@"+}+cp() { [[ "$FAULT" != refused-copy ]] || return 1; builtin command cp "$@"; }+chmod() { [[ "$FAULT" != refused-chmod || "${@: -1}" != "$BOOT_PATH" ]] || return 1; builtin command chmod "$@"; }+mv() {+ [[ "$FAULT" != refused-edit ]] || return 1+ if [[ "$FAULT" == source-changed ]]; then printf 'echo concurrent-owner\n' >> "$1"; fi+ builtin command mv "$@" || return 1+ case "$FAULT" in+ concurrent-file) printf 'concurrent boot owner\n' > "$BOOT_PATH" ;;+ concurrent-directory) mkdir "$BOOT_PATH" ;;+ concurrent-symlink) ln -s "$FIXTURE/unrelated" "$BOOT_PATH" ;;+ esac+}+remove_agent_state_dir() { printf 'state:remove\n' >> "$EVENTS"; rm -rf -- "$STATE_DIR"; }+remove_privileged_helper_state_dir() { rm -rf -- "$PRIVILEGED_HELPER_STATE_DIR"; }+remove_safe_profile_state_dir() { rm -rf -- "$SAFE_PROFILE_STATE_DIR"; }+'''+ entry = self.source.split("# --- Uninstall Logic ---\n", 1)[1].split("# --- Validation ---\n", 1)[0]+ if flow == "process": entry = 'read_nas_agent_removal_processes "$PLATFORM"\n'+ executable = shell + "\n" + setup + "\n" + redirect(definitions(self.source)) + "\nmain() {\n" + redirect(entry) + "}\nmain\n"+ env = dict(os.environ, FIXTURE=str(root), PATH=str(tools) + ":" + os.environ["PATH"])+ result = subprocess.run(["bash", "-c", executable], env=env, capture_output=True, text=True, timeout=14)+ second = subprocess.run(["bash", "-c", executable], env=env, capture_output=True, text=True, timeout=14) if repeat else None+ events = (root / "events").read_text().splitlines() if (root / "events").exists() else []+ after = self.snapshot(root)+ content = bootfile.read_bytes() if bootfile.is_file() and not bootfile.is_symlink() else None+ recovered = {str(p.relative_to(root)): p.read_bytes() for p in root.rglob("retired")}+ timed_stopped = not (root / "timed-pid").exists() or not Path('/proc/' + (root / "timed-pid").read_text()).exists()+ return result, second, events, before, after, json.loads((root / "model.json").read_text()), content, original_boot, recovered, timed_stopped++ @staticmethod+ def snapshot(root):+ result = {}+ for p in root.rglob("*"):+ if p.parts[len(root.parts)] not in ("state", "helper", "profile", "unrelated", "legacy", "foreign"):+ continue+ mode = p.lstat().st_mode+ data = os.readlink(p) if stat.S_ISLNK(mode) else p.read_bytes() if stat.S_ISREG(mode) else None+ result[str(p.relative_to(root))] = (mode, data, p.lstat().st_ino)+ return result++ def retained(self, result, *, remote=False):+ execution, _, events, before, after, _, _, _, _, _ = result+ self.assertEqual(execution.returncode, 1, execution.stdout + execution.stderr)+ self.assertNotIn('state:remove', events)+ self.assertEqual('remote:remove' in events, remote)+ for path in before: self.assertEqual(after.get(path), before[path], path)+ self.assertNotIn('synthetic recovery only', execution.stdout + execution.stderr)+ self.assertNotIn('/synthetic/private-only', execution.stdout + execution.stderr)+ return events++ def test_complete_process_evidence_is_required_before_boot_or_remote_mutation(self):+ for platform in ('Unraid', 'QNAP'):+ for fault in ('failed-process', 'empty-process', 'duplicate-process', 'diagnostic-process', 'missing-timeout', 'missing-ps'):+ with self.subTest(platform=platform, fault=fault):+ result = self.exercise(platform, fault); events = self.retained(result)+ self.assertEqual(result[6], result[7]); self.assertFalse(any(e.startswith('signal:') for e in events))++ def test_unknown_relative_foreign_or_deferred_boot_owners_are_not_signalled(self):+ for platform, fault in (('Unraid', 'foreign-process'), ('QNAP', 'foreign-process'), ('Unraid', 'relative-wrapper'), ('QNAP', 'deferred-bootstrap')):+ with self.subTest(platform=platform, fault=fault):+ result = self.exercise(platform, fault); events = self.retained(result)+ self.assertEqual(result[6], result[7]); self.assertFalse(any(e.startswith('signal:') for e in events))++ def test_non_regular_foreign_wrapper_and_unselected_state_preserve_recovery(self):+ for platform in ('Unraid', 'QNAP'):+ for wrapper in ('linked', 'directory', 'foreign'):+ with self.subTest(platform=platform, wrapper=wrapper):+ result = self.exercise(platform, wrapper=wrapper); self.retained(result); self.assertEqual(result[6], result[7])+ self.retained(self.exercise(platform, state='foreign'))++ def test_only_complete_owned_boot_entries_are_admitted(self):+ for platform in ('Unraid', 'QNAP'):+ for boot in ('foreign', 'indirect', 'unterminated', 'changed-block', 'no-newline', 'binary', 'oversized', 'linked', 'directory'):+ with self.subTest(platform=platform, boot=boot):+ result = self.exercise(platform, boot=boot); self.retained(result); self.assertEqual(result[6], result[7])++ def test_refused_edit_copy_or_prior_recovery_cannot_erase_credentials(self):+ for platform in ('Unraid', 'QNAP'):+ for fault in ('refused-edit', 'refused-copy', 'refused-chmod', 'prior-recovery'):+ with self.subTest(platform=platform, fault=fault): self.retained(self.exercise(platform, fault))++ def test_concurrent_boot_file_directory_and_symlink_are_not_clobbered(self):+ for platform in ('Unraid', 'QNAP'):+ for fault in ('concurrent-file', 'concurrent-directory', 'concurrent-symlink', 'source-changed'):+ with self.subTest(platform=platform, fault=fault):+ result = self.exercise(platform, fault); self.retained(result)+ self.assertTrue(result[8], 'actual retired boot recovery must survive')+ if fault == 'concurrent-file': self.assertEqual(result[6], b'concurrent boot owner\n')+ if fault == 'source-changed': self.assertTrue(any(b'echo concurrent-owner\n' in data for data in result[8].values()))++ def test_flash_mount_and_unmount_failures_are_not_local_only_permission(self):+ for fault in ('mount-refusal', 'unmount-refusal', 'missing-flash'):+ with self.subTest(fault=fault):+ events = self.retained(self.exercise('QNAP', fault)); self.assertFalse(any(e.startswith('signal:') for e in events))+ if fault != 'missing-flash': self.assertIn('flash:umount_flash_config', events)++ def test_watchdog_stops_before_collector_and_ignored_or_refused_signals_retain_state(self):+ for platform in ('Unraid', 'QNAP'):+ for fault in ('ignored-watchdog-stop', 'refused-watchdog-stop', 'ignored-collector-stop', 'refused-collector-stop'):+ with self.subTest(platform=platform, fault=fault):+ events = self.retained(self.exercise(platform, fault))+ self.assertNotIn('unchecked:process-stop', events)+ if 'watchdog' in fault: self.assertNotIn('signal:101', events)++ def test_all_owned_entries_disable_before_signalling_and_remote_erasure(self):+ for platform, fault in (('Unraid', ''), ('QNAP', ''), ('QNAP', 'script-comm')):+ with self.subTest(platform=platform, fault=fault):+ r = self.exercise(platform, fault, boot='duplicate')+ self.assertEqual(r[0].returncode, 0, r[0].stderr)+ self.assertLess(r[2].index('signal:100'), r[2].index('signal:101'))+ self.assertLess(r[2].index('signal:101'), r[2].index('remote:remove'))+ self.assertLess(r[2].index('remote:remove'), r[2].index('state:remove'))+ self.assertNotIn('unchecked:process-stop', r[2]); self.assertFalse(r[5]['watchdog'] or r[5]['collector'])+ for key in ('unrelated', 'legacy', 'legacy/telegraf'): self.assertEqual(r[3][key], r[4][key])+ self.assertIn(b'bash /other/start-pulse-agent-prod.sh\n', r[6]); self.assertIn(b'echo after\n', r[6])+ self.assertNotIn(b'# Pulse Agent\n', r[6]); self.assertNotIn(b'# Pulse Agent bootstrap', r[6]); self.assertEqual(r[8], {})++ def test_remote_refusal_keeps_disabled_boot_and_exact_local_recovery(self):+ for platform in ('Unraid', 'QNAP'):+ with self.subTest(platform=platform):+ r = self.exercise(platform, reply='refused'); self.retained(r, remote=True)+ self.assertFalse(r[5]['watchdog'] or r[5]['collector']); self.assertNotIn(b'# Pulse Agent', r[6]); self.assertTrue(r[8])++ def test_reactivation_before_remote_refuses_without_erasure(self):+ for platform in ('Unraid', 'QNAP'):+ with self.subTest(platform=platform): self.retained(self.exercise(platform, 'reactivated-before-remote'))++ def test_late_boot_or_process_activation_retains_recovery_after_remote(self):+ for platform in ('Unraid', 'QNAP'):+ for fault in ('late-boot', 'late-process'):+ with self.subTest(platform=platform, fault=fault): self.retained(self.exercise(platform, fault), remote=True)++ def test_absent_missing_and_repeated_removal_require_fresh_quiescence(self):+ for platform in ('Unraid', 'QNAP'):+ for boot in ('absent', 'missing'):+ with self.subTest(platform=platform, boot=boot):+ r = self.exercise(platform, boot=boot, wrapper='missing', repeat=True)+ self.assertEqual(r[0].returncode, 0, r[0].stderr); self.assertEqual(r[1].returncode, 0, r[1].stderr)+ self.assertEqual(r[2].count('remote:remove'), 1); self.assertNotIn('unchecked:process-stop', r[2])+ self.assertGreater(r[2].count('process:read'), 4)++ def test_process_observation_returns_roles_not_arguments_and_stops_timeout_child(self):+ r = self.exercise('Unraid', flow='process')+ self.assertEqual(r[0].returncode, 0, r[0].stderr)+ self.assertIn('100 1 watchdog\n', r[0].stdout); self.assertIn('101 100 collector\n', r[0].stdout)+ self.assertNotIn('/synthetic/private-only', r[0].stdout); self.assertIn('200 1 other\n', r[0].stdout)+ r = self.exercise('QNAP', 'timeout-process'); self.retained(r); self.assertTrue(r[9])++ def test_recovered_ordinary_linux_state_is_not_qnap_platform_authority(self):+ r = self.exercise('QNAP', appliance=False)+ self.assertEqual(r[0].returncode, 0, r[0].stderr)+ self.assertFalse(any(e.startswith('flash:') or e.startswith('signal:') for e in r[2]))+ for key in ('foreign', 'foreign/runtime.token', 'legacy/telegraf'):+ self.assertEqual(r[3][key], r[4][key])++ def test_actual_generated_watchdogs_and_children_become_quiescent(self):+ # Private ordinary-user processes only. The full Go suite runs this in+ # the standard offline source VM; it is not native NAS/root acceptance.+ import signal+ import time+ for platform in ('Unraid', 'QNAP'):+ with self.subTest(platform=platform), tempfile.TemporaryDirectory() as directory:+ root = Path(directory)+ for name in ('state', 'bin', 'tools', 'flash', 'logs'): (root / name).mkdir(mode=0o700)+ runtime = root / 'bin/pulse-agent'; stored = root / 'state/stored-agent'+ stored.write_bytes(Path('/bin/sleep').read_bytes()); stored.chmod(0o700)+ watchdog = root / 'state/start-pulse-agent.sh'+ (root / 'tools/pkill').write_text('#!/bin/sh\nexit 0\n'); (root / 'tools/pkill').chmod(0o700)+ (root / 'tools/flash').write_text('#!/bin/sh\nexit 0\n'); (root / 'tools/flash').chmod(0o700)+ env = dict(os.environ, PATH=str(root / 'tools') + ':' + os.environ['PATH'])+ setup = '\n'.join(k + '=' + shlex.quote(str(v)) for k, v in dict(+ STATE_DIR=root / 'state', STATE_DIR_REMOVAL_AUTHORITY=root / 'state', INSTALL_DIR=root / 'bin',+ WRAPPER_SCRIPT=watchdog, UNRAID_STORED_BINARY=stored, RUNTIME_BINARY=runtime,+ AGENT_LOG_FILE=root / 'logs/agent.log').items())+ setup += '\nAGENT_NAME=pulse-agent BINARY_NAME=pulse-agent EXEC_ARGS=30\nEXPORT_SERVICE_ENV="" SHELL_EXPORT_LINES=""\n'+ if platform == 'QNAP':+ renderer = definitions(self.source, ('write_qnap_wrapper_script',))+ render = setup + renderer + '\nwrite_qnap_wrapper_script "$WRAPPER_SCRIPT" "$RUNTIME_BINARY" "$UNRAID_STORED_BINARY" ' + shlex.quote(str(root / 'logs')) + ' "$STATE_DIR"\n'+ else:+ start = self.source.index(' cat > "$WRAPPER_SCRIPT" <<EOF', self.source.index('# 3. Unraid'))+ end = self.source.index('\nEOF\n', start) + 5+ render = setup + self.source[start:end].replace('/var/log/${AGENT_NAME}-watchdog.log', str(root / 'logs/watchdog.log'))+ render_file = root / 'render.sh'; render_file.write_text(render)+ subprocess.run(['bash', str(render_file)], env=env, check=True, capture_output=True, text=True)+ bootfile = root / 'flash/autorun.sh' if platform == 'QNAP' else root / 'go'+ if platform == 'QNAP':+ bootfile.write_text('#!/bin/sh\n')+ emitter = definitions(self.source, ('portable_sed_in_place', 'remove_qnap_autorun_block', 'render_qnap_autorun_block', 'append_qnap_autorun_block'))+ render_file.write_text(setup + emitter + '\nappend_qnap_autorun_block ' + shlex.quote(str(bootfile)) + ' "$WRAPPER_SCRIPT" "$STATE_DIR"\n')+ subprocess.run(['bash', str(render_file)], env=env, check=True, capture_output=True, text=True)+ else: bootfile.write_text('#!/bin/bash\n# Pulse Agent\nbash ' + str(watchdog) + '\n')+ functions = definitions(self.source)+ functions = functions.replace('/boot/config/go', str(bootfile))+ functions = functions.replace('[[ -d /boot/config && ! -L /boot/config ]]', '[[ -d ' + str(root) + ' && ! -L ' + str(root) + ' ]]')+ functions = functions.replace('/etc/init.d/init_disk.sh', str(root / 'tools/flash'))+ functions = functions.replace('/tmp/nasconfig_tmp', str(root / 'flash'))+ control = root / 'control.sh'+ control.write_text('set -euo pipefail\n' + setup + '\n' + functions + '\ncommand() { [[ "$*" != "-v systemctl" ]] || return 1; builtin command "$@"; }\nquiesce_nas_agent_for_removal ' + platform + '\n')+ output = root / 'watchdog.log'+ with output.open('wb') as stream:+ process = subprocess.Popen([str(watchdog)] if platform == 'QNAP' else ['bash', str(watchdog)], env=env,+ stdout=stream, stderr=stream, start_new_session=True)+ try:+ deadline = time.monotonic() + 8+ while time.monotonic() < deadline:+ listing = subprocess.run(['ps', '-ax', '-o', 'comm=', '-o', 'args='], capture_output=True, text=True, check=True).stdout+ if any(line.strip().startswith('pulse-agent ') and str(runtime) in line for line in listing.splitlines()): break+ self.assertIsNone(process.poll(), output.read_text()); time.sleep(.05)+ else: self.fail('private generated collector did not start')+ result = subprocess.run(['bash', str(control)], env=env, capture_output=True, text=True, timeout=12)+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr + output.read_text())+ process.wait(timeout=3)+ with self.assertRaises(ProcessLookupError): os.killpg(process.pid, 0)+ self.assertNotIn('# Pulse Agent', bootfile.read_text())+ finally:+ try: os.killpg(process.pid, signal.SIGKILL)+ except ProcessLookupError: pass+ process.wait(timeout=3)++ def test_caller_has_no_blanket_nas_or_legacy_tree_deletion(self):+ caller = self.source.split('# --- Uninstall Logic ---\n', 1)[1].split('# --- Validation ---\n', 1)[0]+ self.assertNotIn("sed -i '/pulse-agent/d'", caller)+ self.assertNotIn('rm -rf /boot/config/pulse', caller)+ self.assertNotIn('rm -rf "$qnap_state_dir"', caller)+ self.assertIn('teardown_nas_agent_boot_owner "$uninstall_nas_platform"', caller)+ self.assertIn('[[ "$uninstall_upstart" != true && -z "$uninstall_nas_platform" ]]', caller)+++if __name__ == '__main__': unittest.main(verbosity=2)diff --git a/scripts/installtests/install_sh_test.go b/scripts/installtests/install_sh_test.goindex 635f494f41..42297d5b3d 100644--- a/scripts/installtests/install_sh_test.go+++ b/scripts/installtests/install_sh_test.go@@ -8540,3 +8540,43 @@ func TestInstallSHTrueNASScaleBootTasksRetireBeforeRecoveryErasure(t *testing.T)}t.Logf("%s", output)}++func TestInstallSHNASBootOwnersQuiesceBeforeRecoveryErasure(t *testing.T) {+ if runtime.GOOS == "windows" {+ t.Skip("Unix NAS shell controls; Windows retains its separate native SCM gate")+ }+ ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)+ defer cancel()+ cmd := exec.CommandContext(ctx, "python3", repoFile("scripts", "installtests", "agent_nas_teardown.py"))+ output, err := cmd.CombinedOutput()+ if err != nil {+ t.Fatalf("NAS boot-owner/watchdog production caller controls: %v\n%s", err, output)+ }+ for _, name := range []string{+ "test_complete_process_evidence_is_required_before_boot_or_remote_mutation",+ "test_unknown_relative_foreign_or_deferred_boot_owners_are_not_signalled",+ "test_non_regular_foreign_wrapper_and_unselected_state_preserve_recovery",+ "test_only_complete_owned_boot_entries_are_admitted",+ "test_refused_edit_copy_or_prior_recovery_cannot_erase_credentials",+ "test_concurrent_boot_file_directory_and_symlink_are_not_clobbered",+ "test_flash_mount_and_unmount_failures_are_not_local_only_permission",+ "test_watchdog_stops_before_collector_and_ignored_or_refused_signals_retain_state",+ "test_all_owned_entries_disable_before_signalling_and_remote_erasure",+ "test_remote_refusal_keeps_disabled_boot_and_exact_local_recovery",+ "test_reactivation_before_remote_refuses_without_erasure",+ "test_late_boot_or_process_activation_retains_recovery_after_remote",+ "test_absent_missing_and_repeated_removal_require_fresh_quiescence",+ "test_process_observation_returns_roles_not_arguments_and_stops_timeout_child",+ "test_recovered_ordinary_linux_state_is_not_qnap_platform_authority",+ "test_actual_generated_watchdogs_and_children_become_quiescent",+ "test_caller_has_no_blanket_nas_or_legacy_tree_deletion",+ } {+ if !strings.Contains(string(output), name+" (__main__.AgentNASTeardown."+name+") ... ok") {+ t.Fatalf("NAS removal control %s did not pass:\n%s", name, output)+ }+ }+ if !strings.Contains(string(output), "\nRan 17 tests in ") || !strings.Contains(string(output), "\nOK\n") {+ t.Fatalf("incomplete NAS removal verdict:\n%s", output)+ }+ t.Logf("%s", output)+}