Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Retain SCALE agent recovery until every boot task is retired
pulse-triage[bot]59ff0dabcf84
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 7193c99f46..d71de6fb80 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -15,6 +15,48 @@## Purpose+### TrueNAS SCALE removal retires every owned middleware boot task before recovery erasure++TrueNAS SCALE has an independent activation owner: its Init/Shutdown bootstrap+can recreate and enable the systemd unit after a normal disable. A recovered+state directory or an unrelated `/etc/version` alone is not appliance identity.+The uninstall caller selects SCALE only from a bootstrap footprint or vendor+marker. The existing protected/explicit state-removal authority must select the+same state directory and exact bootstrap path; linked, writable or non-regular+bootstrap files refuse before task/service mutation.++One complete, bounded middleware inventory must identify every exact generated+SCRIPT/POSTINIT task, including duplicates. Unknown ownership, indirect command+activation or another-install bootstrap refuses without deleting unrelated+tasks. The JSON reader rejects duplicate keys/IDs, invalid types, failed/empty/+diagnostic/incomplete reads, over 512 tasks or over 1 MiB; it returns only owned+numeric IDs and enablement, never task commands, names or credentials. All owned+tasks are disabled with checked calls and independent exact-set readback before+systemd stop. A partial stop/disable stays stopped/disabled: do not re-enable or+automatically restart after a refusal.++The complete systemd target set and independent kernel collector/bootstrap+process inventory and a final exact disabled-task readback must be quiescent+before same-instance server removal, including task changes during systemd stop.+After remote confirmation, the exact disabled task set is re-read before the first+task deletion. Every owned deletion is checked; final whole-unit/process+quiescence and complete task absence precede credential, stored binary,+bootstrap and state erasure. A changed set, refusal, ignored deletion, new task+or reactivation retains local recovery, even if earlier task/server operations+succeeded. No favourable replacement observation, task recreation or credential+churn is performed. Unrelated tasks survive unchanged.++`TestInstallSHTrueNASScaleBootTasksRetireBeforeRecoveryErasure` requires all+fifteen `agent_truenas_teardown.py` actual-function/full-caller controls,+including multiple tasks, partial/refused/unknown operations, post-remote+changes, ignored/recreated deletion, process privacy and a real timeout child.+These run against closed middleware/systemd/process models and private ordinary-+user files, not a native appliance. Existing systemd/OpenRC/DSM/rc.d/SysV gates,+transport trust and identity remain. TrueNAS CORE retains its linked rc.d/+appliance refusal; Unraid/QNAP boot ownership, Windows and signed installed+Linux/PowerShell/native/TLS/cleanup acceptance remain separate. This does not+expand the original removal job into macOS or introduce another dispatcher.+### DSM Upstart removal retains recovery until automatic activation and the job are absentDSM 6 Upstart jobs (including leftovers on DSM 7) are not classified as SysV.diff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.mdindex e71c8a5d55..b6d4667eec 100644--- a/docs/release-control/v6/internal/subsystems/deployment-installability.md+++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md@@ -16,6 +16,48 @@## Purpose+### TrueNAS SCALE removal retires every owned middleware boot task before recovery erasure++TrueNAS SCALE has an independent activation owner: its Init/Shutdown bootstrap+can recreate and enable the systemd unit after a normal disable. A recovered+state directory or an unrelated `/etc/version` alone is not appliance identity.+The uninstall caller selects SCALE only from a bootstrap footprint or vendor+marker. The existing protected/explicit state-removal authority must select the+same state directory and exact bootstrap path; linked, writable or non-regular+bootstrap files refuse before task/service mutation.++One complete, bounded middleware inventory must identify every exact generated+SCRIPT/POSTINIT task, including duplicates. Unknown ownership, indirect command+activation or another-install bootstrap refuses without deleting unrelated+tasks. The JSON reader rejects duplicate keys/IDs, invalid types, failed/empty/+diagnostic/incomplete reads, over 512 tasks or over 1 MiB; it returns only owned+numeric IDs and enablement, never task commands, names or credentials. All owned+tasks are disabled with checked calls and independent exact-set readback before+systemd stop. A partial stop/disable stays stopped/disabled: do not re-enable or+automatically restart after a refusal.++The complete systemd target set and independent kernel collector/bootstrap+process inventory and a final exact disabled-task readback must be quiescent+before same-instance server removal, including task changes during systemd stop.+After remote confirmation, the exact disabled task set is re-read before the first+task deletion. Every owned deletion is checked; final whole-unit/process+quiescence and complete task absence precede credential, stored binary,+bootstrap and state erasure. A changed set, refusal, ignored deletion, new task+or reactivation retains local recovery, even if earlier task/server operations+succeeded. No favourable replacement observation, task recreation or credential+churn is performed. Unrelated tasks survive unchanged.++`TestInstallSHTrueNASScaleBootTasksRetireBeforeRecoveryErasure` requires all+fifteen `agent_truenas_teardown.py` actual-function/full-caller controls,+including multiple tasks, partial/refused/unknown operations, post-remote+changes, ignored/recreated deletion, process privacy and a real timeout child.+These run against closed middleware/systemd/process models and private ordinary-+user files, not a native appliance. Existing systemd/OpenRC/DSM/rc.d/SysV gates,+transport trust and identity remain. TrueNAS CORE retains its linked rc.d/+appliance refusal; Unraid/QNAP boot ownership, Windows and signed installed+Linux/PowerShell/native/TLS/cleanup acceptance remain separate. This does not+expand the original removal job into macOS or introduce another dispatcher.+### DSM Upstart removal retains recovery until automatic activation and the job are absentDSM 6 Upstart jobs (including leftovers on DSM 7) are not classified as SysV.diff --git a/scripts/install.sh b/scripts/install.shindex 543578ece3..82134ddf26 100755--- a/scripts/install.sh+++ b/scripts/install.sh@@ -1344,6 +1344,7 @@ teardown_openrc_agent_service() {# cannot establish that after an ignored signal or lost PID file.read_legacy_agent_removal_processes() {local observation="" line="" pid="" parent="" comm="" args="" role=""+ local bootstrap="${1:-}"local complete="false" count=0 parents=" " seen=" "local -a rows=()observation=$(LC_ALL=C timeout -k 1 5 ps -ax -o pid= -o ppid= -o comm= -o args= 2>&1 &&@@ -1365,6 +1366,7 @@ read_legacy_agent_removal_processes() {elif [[ "${comm##*/}" == daemon && "$args" =~ (^|[[:space:]/])pulse-agent([[:space:]\[]|$) ]]; thenrole="supervisor"fi+ if [[ -n "$bootstrap" && "$args" == *"$bootstrap"* ]]; then role="bootstrap"; firows+=("$pid $parent $role ${comm##*/}")count=$((count + 1))[[ $count -le 65536 ]] || return 1@@ -1706,6 +1708,130 @@ enable_sysv_agent_service() {log_info "Created rc.d symlinks manually."}+# TrueNAS SCALE has a second activation owner: middleware's boot task can+# recreate and enable the systemd unit even after systemctl disable. Recovered+# state directories alone are not appliance identity (ordinary Linux has them).+truenas_scale_agent_removal_present() {+ [[ "$(uname -s)" == Linux ]] || return 1+ [[ -e "${TRUENAS_BOOTSTRAP_SCRIPT:-}" || -L "${TRUENAS_BOOTSTRAP_SCRIPT:-}" ||+ -f /etc/truenas-version || -d /data/ix-applications || -d /etc/ix-apps.d ]] && return 0+ [[ -f /etc/version ]] && grep -qi truenas /etc/version+}++read_truenas_agent_removal_tasks() {+ local observation=""+ observation=$(LC_ALL=C timeout -k 1 5 midclt call initshutdownscript.query 2>&1 &&+ printf '\n%s' '__pulse_truenas_tasks_complete__') || return 1+ # Parse the complete inventory, including indirect/other-install bootstrap+ # references. No task command, name, argument or credential is returned.+ python3 -c '+import json, pathlib, sys+def unique_object(pairs):+ result = {}+ for key, value in pairs:+ if key in result: raise ValueError("duplicate field")+ result[key] = value+ return result+def reject_constant(value):+ raise ValueError("non-JSON constant")+try:+ raw = sys.stdin.buffer.read(1048577)+ suffix = b"\n__pulse_truenas_tasks_complete__\n"+ if len(raw) > 1048576 or not raw.endswith(suffix): raise ValueError("incomplete inventory")+ tasks = json.loads(raw[:-len(suffix)].decode("utf-8"), object_pairs_hook=unique_object,+ parse_constant=reject_constant)+ if not isinstance(tasks, list) or len(tasks) > 512: raise ValueError("unbounded inventory")+ expected, seen, owned = sys.argv[1], set(), []+ for task in tasks:+ if not isinstance(task, dict): raise ValueError("invalid task")+ ident, kind, enabled = task.get("id"), task.get("type"), task.get("enabled")+ if type(ident) is not int or not 0 < ident <= 2147483647 or ident in seen:+ raise ValueError("invalid identity")+ seen.add(ident)+ if kind not in ("SCRIPT", "COMMAND") or type(enabled) is not bool:+ raise ValueError("unknown activation")+ script, command = task.get("script", ""), task.get("command", "")+ if script is None: script = ""+ if command is None: command = ""+ if not isinstance(script, str) or not isinstance(command, str): raise ValueError("invalid activation")+ if script == expected:+ if (kind != "SCRIPT" or command or task.get("when") != "POSTINIT" or+ task.get("comment") != "Pulse Agent Bootstrap" or type(task.get("timeout")) is not int or+ task["timeout"] != 30): raise ValueError("unowned bootstrap")+ owned.append((ident, enabled))+ elif (pathlib.PurePosixPath(script).name == "bootstrap-pulse-agent.sh" or+ "bootstrap-pulse-agent.sh" in command or "pulse-agent" in command):+ raise ValueError("indirect or other-install activation")+ for ident, enabled in sorted(owned): print(ident, "true" if enabled else "false")+ print("complete")+except (ValueError, TypeError, KeyError, UnicodeError):+ print("TrueNAS boot-task inventory is incomplete or not owned; recovery retained.", file=sys.stderr)+ sys.exit(1)+' "$TRUENAS_BOOTSTRAP_SCRIPT" <<< "$observation"+}++quiesce_truenas_agent_boot_tasks_for_removal() {+ local inventory="" expected="" observed="" ident="" enabled="" mode=""+ command -v midclt >/dev/null 2>&1 && command -v python3 >/dev/null 2>&1 &&+ command -v timeout >/dev/null 2>&1 && command -v systemctl >/dev/null 2>&1 || return 1+ # Do not let a platform marker or an arbitrary saved state path authorise+ # erasure of a different appliance install. Keep the existing state owner.+ [[ "$TRUENAS_STATE_DIR" == "$STATE_DIR" &&+ "${STATE_DIR_REMOVAL_AUTHORITY:-}" == "$STATE_DIR" &&+ "$TRUENAS_BOOTSTRAP_SCRIPT" == "$STATE_DIR/bootstrap-pulse-agent.sh" ]] || return 1+ if [[ -e "$TRUENAS_BOOTSTRAP_SCRIPT" || -L "$TRUENAS_BOOTSTRAP_SCRIPT" ]]; then+ mode=$(portable_path_mode "$TRUENAS_BOOTSTRAP_SCRIPT") || return 1+ [[ "$mode" =~ ^[0-7]{3,4}$ ]] && (( (8#$mode & 0022) == 0 )) || return 1+ trusted_lifecycle_regular_file "$TRUENAS_BOOTSTRAP_SCRIPT" "$mode" || return 1+ fi+ inventory=$(read_truenas_agent_removal_tasks) || return 1+ while read -r ident enabled; do+ [[ "$ident" != complete ]] || continue+ if [[ "$enabled" == true ]]; then+ LC_ALL=C timeout -k 1 5 midclt call initshutdownscript.update "$ident" '{"enabled":false}' >/dev/null 2>&1 || return 1+ fi+ expected+="$ident false"$'\n'+ done <<< "$inventory"+ expected+=complete+ observed=$(read_truenas_agent_removal_tasks) || return 1+ [[ "$observed" == "$expected" ]] || return 1+ printf '%s\n' "$expected"+}++confirm_truenas_agent_removal_quiescent() {+ local unit="" inventory="" observed=""+ for unit in "${ACTION_RUNNER_NAME}.service" "${PRIVILEGED_HELPER_NAME}.socket" \+ "${PRIVILEGED_HELPER_NAME}.service" "${AGENT_NAME}.service"; do+ confirm_agent_unit_quiescent "$unit" || return 1+ done+ inventory=$(read_legacy_agent_removal_processes "$TRUENAS_BOOTSTRAP_SCRIPT") || return 1+ legacy_agent_processes_absent "$inventory" && [[ "$inventory" != *' bootstrap'* ]] || return 1+ # The middleware set can change during the preceding systemd operations.+ # Close it again before remote removal/first deletion, not only afterwards.+ if [[ $# -gt 0 ]]; then+ observed=$(read_truenas_agent_removal_tasks) || return 1+ [[ "$observed" == "$1" ]] || return 1+ fi+}++teardown_truenas_agent_boot_tasks() {+ local expected="$1" observed="" ident="" enabled=""+ # No newly discovered task is disabled/deleted after remote removal. A+ # changed set, refused read or concurrent activation retains local recovery.+ observed=$(read_truenas_agent_removal_tasks) || return 1+ [[ "$observed" == "$expected" ]] || return 1+ confirm_truenas_agent_removal_quiescent "$expected" || return 1+ while read -r ident enabled; do+ [[ "$ident" != complete ]] || continue+ LC_ALL=C timeout -k 1 5 midclt call initshutdownscript.delete "$ident" >/dev/null 2>&1 || return 1+ done <<< "$expected"+ # Close the whole manager/process/task set after the last deletion, while+ # every credential, bootstrap, stored binary and service file still exists.+ confirm_truenas_agent_removal_quiescent || return 1+ observed=$(read_truenas_agent_removal_tasks) || return 1+ [[ "$observed" == complete ]]+}+write_truenas_bootstrap_script() {local platform="$1"local service_link=""@@ -5999,6 +6125,8 @@ if [[ "$UNINSTALL" == "true" ]]; thenlocal uninstall_has_credential="false"local uninstall_legacy_manager=""local uninstall_upstart="false"+ local uninstall_truenas_scale="false"+ local uninstall_truenas_tasks=""# Validate saved connection availability before stopping a working install.# Stop every systemd activation source before remote removal, process kills@@ -6009,6 +6137,14 @@ if [[ "$UNINSTALL" == "true" ]]; thenfail "The saved Pulse URL is missing; local credentials and services were retained. Restore saved connection state or provide --url for the same trusted Pulse instance and retry uninstall." "$EXIT_MISSING_ARGS"fifi+ # Middleware can recreate and enable the systemd unit at boot. Disable all+ # exact owned tasks before stopping services; a partial/refused observation+ # retains every recovery file and never restores automatic activation.+ if truenas_scale_agent_removal_present; then+ uninstall_truenas_scale="true"+ uninstall_truenas_tasks=$(quiesce_truenas_agent_boot_tasks_for_removal) ||+ fail "TrueNAS SCALE removal could not confirm every owned boot task disabled; credentials, bootstrap and service files were retained. Reconcile middleware task ownership/state before retrying; no automatic restart was attempted." "$EXIT_GENERAL"+ fi# Detect stale DSM 6 jobs even on a DSM 7 host. Another init manager cannot# establish their absence; missing initctl is an explicit safe refusal.if [[ -d /usr/syno ]] && { command -v initctl >/dev/null 2>&1 ||@@ -6037,6 +6173,10 @@ if [[ "$UNINSTALL" == "true" ]]; thenquiesce_sysv_agent_for_removal ||fail "SysV removal could not confirm stopped process and disabled runlevels; local recovery files were retained. A preceding stop or disable may have succeeded; no automatic restart was attempted." "$EXIT_GENERAL"fi+ if [[ "$uninstall_truenas_scale" == true ]]; then+ confirm_truenas_agent_removal_quiescent "$uninstall_truenas_tasks" ||+ fail "TrueNAS SCALE removal could not confirm stopped units, collector and bootstrap processes; local recovery was retained. Reconcile stopped/disabled state before retrying; no automatic restart was attempted." "$EXIT_GENERAL"+ fi# A credential-bearing install must durably remove its exact server record# through the same CA/fingerprint/no-proxy transport used by lifecycle@@ -6068,6 +6208,10 @@ if [[ "$UNINSTALL" == "true" ]]; thenteardown_upstart_agent_service ||fail "DSM Upstart job retirement was not confirmed; local recovery state was retained. The manual override remains after a partial stop. Reconcile the original config or its .removal recovery copy before retrying; no automatic restart was attempted." "$EXIT_GENERAL"fi+ if [[ "$uninstall_truenas_scale" == true ]]; then+ teardown_truenas_agent_boot_tasks "$uninstall_truenas_tasks" ||+ fail "TrueNAS SCALE boot-task retirement was not confirmed; local recovery was retained. A preceding task disable/delete or server removal may have succeeded. Reconcile middleware and stopped service state before retrying; no automatic restart was attempted." "$EXIT_GENERAL"+ fi# Kill wrapper scripts first: they are watchdogs, so stopping the agent# while its wrapper still loops only races the respawn. No leading path@@ -6181,15 +6325,12 @@ if [[ "$UNINSTALL" == "true" ]]; thenfifi- # TrueNAS SCALE/CORE- if [[ -d "$TRUENAS_STATE_DIR" ]] || [[ -f /etc/truenas-version ]] || [[ -f /etc/version ]]; then- if [[ "$(uname -s)" == "Linux" ]]; then- log_info "Removing TrueNAS SCALE installation..."- teardown_systemd_agent_service- elif [[ "$(uname -s)" == "FreeBSD" ]]; then- log_info "Removing TrueNAS CORE installation..."- # rc.d teardown was confirmed before remote/local recovery erasure.- fi+ # SCALE's task retirement was confirmed above, before authorised state+ # erasure. A recovered ordinary Linux directory is not a TrueNAS install.+ # CORE remains behind its existing rc.d/appliance activation refusal; this+ # SCALE step does not claim CORE boot-task or native platform acceptance.+ if [[ "$(uname -s)" == "FreeBSD" ]] && { [[ -d "$TRUENAS_STATE_DIR" ]] || [[ -f /etc/truenas-version ]] || [[ -f /etc/version ]]; }; then+ log_info "Removing TrueNAS CORE installation..."# Remove Init/Shutdown taskif command -v midclt >/dev/null 2>&1 && command -v python3 >/dev/null 2>&1; thenTASK_ID=$(midclt call initshutdownscript.query '[["script","=","'"$TRUENAS_BOOTSTRAP_SCRIPT"'"]]' 2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); print(d[0]['id'] if d else '')" 2>/dev/null || echo "")diff --git a/scripts/installtests/agent_legacy_teardown.py b/scripts/installtests/agent_legacy_teardown.pyindex 54afa1b99c..51dc2a0587 100644--- a/scripts/installtests/agent_legacy_teardown.py+++ b/scripts/installtests/agent_legacy_teardown.py@@ -17,7 +17,7 @@ import unittestROOT = Path(__file__).resolve().parents[2]SOURCE = Path(os.environ.get("PULSE_AGENT_LEGACY_SOURCE", ROOT / "scripts/install.sh"))-FUNCTIONS = ("read_legacy_agent_removal_processes", "legacy_agent_processes_absent",+FUNCTIONS = ("truenas_scale_agent_removal_present", "read_legacy_agent_removal_processes", "legacy_agent_processes_absent","validate_legacy_agent_removal_pidfiles", "read_legacy_agent_removal_status","read_freebsd_agent_removal_enablement", "quiesce_freebsd_agent_for_removal","teardown_freebsd_agent_service", "read_sysv_agent_removal_links",diff --git a/scripts/installtests/agent_openrc_teardown.py b/scripts/installtests/agent_openrc_teardown.pyindex a2cc349ea5..fbc2102ebd 100644--- a/scripts/installtests/agent_openrc_teardown.py+++ b/scripts/installtests/agent_openrc_teardown.py@@ -21,7 +21,7 @@ import unittestROOT = Path(__file__).resolve().parents[2]SOURCE = Path(os.environ.get("PULSE_AGENT_OPENRC_SOURCE", ROOT / "scripts/install.sh"))-FUNCTIONS = ("read_openrc_agent_removal_status", "read_openrc_agent_removal_runlevels",+FUNCTIONS = ("truenas_scale_agent_removal_present", "read_openrc_agent_removal_status", "read_openrc_agent_removal_runlevels","quiesce_openrc_agent_for_removal", "teardown_openrc_agent_service")diff --git a/scripts/installtests/agent_truenas_teardown.py b/scripts/installtests/agent_truenas_teardown.pynew file mode 100644index 0000000000..b5542df5b8--- /dev/null+++ b/scripts/installtests/agent_truenas_teardown.py@@ -0,0 +1,377 @@+#!/usr/bin/env python3+"""TrueNAS SCALE uninstall, using production functions and private manager models.++Only fixed vendor/service paths are redirected into owned temporary files.+No appliance, root installer, network, credential or native acceptance is used.+The complete uninstall caller runs; middleware/process/systemd are closed models.+"""+import json+import os+from pathlib import Path+import shlex+import stat+import subprocess+import tempfile+import unittest++ROOT = Path(__file__).resolve().parents[2]+SOURCE = Path(os.environ.get("PULSE_AGENT_TRUENAS_SOURCE", ROOT / "scripts/install.sh"))+FUNCTIONS = (+ "portable_path_uid", "portable_path_mode", "trusted_lifecycle_regular_file",+ "read_agent_removal_unit_state", "confirm_agent_unit_quiescent",+ "quiesce_agent_units_for_removal", "teardown_systemd_agent_service",+ "read_legacy_agent_removal_processes", "legacy_agent_processes_absent",+ "truenas_scale_agent_removal_present", "read_truenas_agent_removal_tasks",+ "quiesce_truenas_agent_boot_tasks_for_removal", "confirm_truenas_agent_removal_quiescent",+ "teardown_truenas_agent_boot_tasks",+)+++def definitions(source):+ selected = []+ for name in FUNCTIONS:+ marker = name + "() {"+ if marker not in source: # The exact-parent adverse comparison.+ continue+ if source.count(marker) != 1:+ raise ValueError("ambiguous production function " + name)+ selected.append(marker + source.split(marker, 1)[1].split("\n}\n", 1)[0] + "\n}\n")+ return "\n".join(selected)+++MODEL = r'''#!/usr/bin/python3+import json, os, pathlib, sys+root = pathlib.Path(os.environ['FIXTURE'])+path = root / 'model.json'+v = json.loads(path.read_text())+role, args = pathlib.Path(sys.argv[0]).name, sys.argv[1:]+fault = v['fault']+def save():+ pending = root / 'model.pending'+ pending.write_text(json.dumps(v)); pending.replace(path)+def event(s):+ with (root / 'events').open('a') as f: f.write(s + '\n')+remote = (root / 'events').exists() and 'remote:remove' in (root / 'events').read_text()+if role == 'ps':+ assert args == ['-ax', '-o', 'pid=', '-o', 'ppid=', '-o', 'comm=', '-o', 'args='], args+ event('process:read')+ if fault == 'failed-process': sys.exit(1)+ if fault == 'empty-process': sys.exit(0)+ print('1 0 init /sbin/init')+ if fault == 'duplicate-process': print('1 0 init /sbin/init')+ if v['active'] == 'active' or fault == 'orphan-agent':+ print('100 1 pulse-agent /usr/local/bin/pulse-agent --token-file /synthetic/recovery-only')+ if fault == 'live-bootstrap': print('101 1 bash /bin/bash ' + str(root / 'state/bootstrap-pulse-agent.sh'))+ if fault == 'diagnostic-process': print('read unavailable', file=sys.stderr)+ sys.exit(0)+if role == 'systemctl':+ verb = args[0]+ if verb == 'daemon-reload': event('unit:reload'); sys.exit(0)+ assert len(args) >= 2, args+ unit = args[1]+ assert unit in ('pulse-agent.service', 'pulse-agent-helper.service', 'pulse-agent-helper.socket', 'pulse-agent-runner.service'), args+ if verb == 'show':+ assert args[2:] == ['--property=LoadState,ActiveState,UnitFileState'], args+ event('unit:read:' + unit)+ if fault == 'failed-unit-read': sys.exit(1)+ present = unit == 'pulse-agent.service' and (root / 'unit/pulse-agent.service').exists()+ if present:+ print('LoadState=loaded\nActiveState=' + v['active'] + '\nUnitFileState=' + v['enabled'])+ else: print('LoadState=not-found\nActiveState=inactive\nUnitFileState=')+ sys.exit(0)+ assert verb in ('stop', 'disable'), args+ event('unit:' + verb)+ if fault == 'refused-stop' and verb == 'stop': sys.exit(1)+ if verb == 'stop' and fault != 'ignored-stop': v['active'] = 'inactive'+ if verb == 'stop' and fault == 'reactivated-task-on-stop': v['tasks'][0]['enabled'] = True+ if verb == 'disable': v['enabled'] = 'disabled'+ save(); sys.exit(0)+assert role == 'midclt' and args[0] == 'call', (role, args)+verb = args[1]+if verb == 'initshutdownscript.query':+ assert len(args) in (2, 3), args+ event('task:query'); v['queries'] += 1+ if fault == 'timeout-query':+ save(); (root / 'timed-pid').write_text(str(os.getpid()))+ os.execl('/bin/sleep', 'sleep', '30')+ if fault == 'failed-query' or (fault == 'failed-after-remote-query' and remote): save(); sys.exit(1)+ if fault == 'new-task-after-remote' and remote and not any(t['id'] == 3 for t in v['tasks']):+ v['tasks'].append(dict(v['tasks'][0], id=3, enabled=True))+ if fault == 'reactivated-task' and v['updates'] and v['tasks']: v['tasks'][0]['enabled'] = True+ if fault == 'recreated-task' and v['deletes'] and not any(t['script'] == str(root / 'state/bootstrap-pulse-agent.sh') for t in v['tasks']):+ v['tasks'].append(dict(id=3, type='SCRIPT', script=str(root / 'state/bootstrap-pulse-agent.sh'), command='', when='POSTINIT', enabled=True, timeout=30, comment='Pulse Agent Bootstrap'))+ save()+ tasks = v['tasks']+ if len(args) == 3: # Preserve the parent's exact filter, not a new graph.+ filters = json.loads(args[2]); assert filters[0][:2] == ['script', '='], filters+ tasks = [t for t in tasks if t['script'] == filters[0][2]]+ if fault == 'empty-query': sys.exit(0)+ if fault == 'malformed-query': print('not-json'); sys.exit(0)+ if fault == 'object-query': print('{}'); sys.exit(0)+ if fault == 'duplicate-field': print('[{"id":1,"id":2}]'); sys.exit(0)+ if fault == 'oversized-query': print(' ' * 1048577 + '[]'); sys.exit(0)+ if fault == 'oversized-inventory': tasks = [dict(v['tasks'][0], id=i + 1) for i in range(513)]+ if fault == 'duplicate-id': tasks = tasks + [tasks[0]]+ if fault == 'bool-id': tasks[0]['id'] = True+ if fault == 'string-enabled': tasks[0]['enabled'] = 'false'+ if fault == 'foreign-owner': tasks[0]['comment'] = 'not owned by Pulse'+ if fault == 'wrong-kind': tasks[0]['type'] = 'UNKNOWN'+ if fault == 'command-bootstrap': tasks[0].update(type='COMMAND', script='', command='/bin/bash ' + str(root / 'state/bootstrap-pulse-agent.sh'))+ if fault == 'other-bootstrap': tasks[0]['script'] = '/other-install/bootstrap-pulse-agent.sh'+ if fault == 'diagnostic-query': print('read unavailable', file=sys.stderr)+ print(json.dumps(tasks)); sys.exit(0)+if verb == 'initshutdownscript.update':+ assert len(args) == 4 and json.loads(args[3]) == {'enabled': False}, args+ ident = int(args[2]); event('task:disable:' + str(ident)); v['updates'] += 1+ if fault == 'refused-disable' or (fault == 'partial-disable' and ident == 2): save(); sys.exit(1)+ task = next(t for t in v['tasks'] if t['id'] == ident)+ if fault != 'ignored-disable': task['enabled'] = False+ save(); print(json.dumps(task)); sys.exit(0)+assert verb == 'initshutdownscript.delete' and len(args) == 3, args+ident = int(args[2]); event('task:delete:' + str(ident)); v['deletes'] += 1+if fault == 'refused-delete' or (fault == 'partial-delete' and ident == 2): save(); sys.exit(1)+if fault != 'ignored-delete': v['tasks'] = [t for t in v['tasks'] if t['id'] != ident]+if fault == 'unit-reactivation-on-delete': v['active'] = 'active'+save(); print('true')+'''+++class AgentTrueNASTeardown(unittest.TestCase):+ source = SOURCE.read_text()++ def exercise(self, fault="", reply="confirmed", bootstrap="regular", tasks="duplicate",+ appliance=True, repeat=False, flow="caller"):+ with tempfile.TemporaryDirectory() as temporary:+ root = Path(temporary)+ tools = root / "tools"; tools.mkdir()+ for name in ("midclt", "ps", "systemctl"):+ file = tools / name; file.write_text(MODEL); file.chmod(0o700)+ if fault == "missing-midclt": (tools / "midclt").unlink()+ for name in ("state", "helper", "profile", "unit"):+ directory = root / name; directory.mkdir(mode=0o700)+ if name != "unit":+ for file in ("runtime.token", "agent-id", "connection.env", "install.sh"):+ (directory / file).write_text("synthetic recovery only\n")+ script = root / "state/bootstrap-pulse-agent.sh"+ unrelated = root / "unrelated"; unrelated.write_text("do not alter\n")+ if bootstrap == "linked": script.symlink_to(unrelated)+ elif bootstrap == "directory": script.mkdir()+ elif bootstrap != "missing":+ script.write_text("#!/bin/bash\n# Pulse Agent Bootstrap for TrueNAS\n")+ script.chmod(0o777 if bootstrap == "writable" else 0o755)+ vendor = root / "truenas-version"+ if appliance: vendor.write_text("TrueNAS-SCALE\n")+ (root / "version").write_text("unrelated platform\n")+ (root / "unit/pulse-agent.service").write_text("owned service\n")+ owned = dict(id=1, type="SCRIPT", script=str(script), command="", when="POSTINIT",+ enabled=True, timeout=30, comment="Pulse Agent Bootstrap")+ foreign = dict(owned, id=99, script="/unrelated/boot.sh", comment="unrelated task")+ selected = [] if tasks == "absent" else [owned]+ if tasks == "duplicate": selected.append(dict(owned, id=2))+ if tasks == "disabled": selected[0]["enabled"] = False+ selected.append(foreign)+ model = dict(fault=fault, tasks=selected, updates=0, deletes=0, queries=0,+ active="active", enabled="enabled")+ (root / "model.json").write_text(json.dumps(model))+ before = self.snapshot(root)++ def redirect(source):+ for path, target in (+ ("/etc/truenas-version", vendor), ("/etc/version", root / "version"),+ ("/data/ix-applications", root / "no-ix"), ("/etc/ix-apps.d", root / "no-apps"),+ ("/etc/ix.rc.d", root / "no-rc"), ("/usr/syno", root / "no-dsm"),+ ("/etc/unraid-version", root / "no-unraid"),+ ("/boot/config/plugins/pulse-agent", root / "no-plugin"),+ ("/etc/config/qpkg.conf", root / "no-qnap"),+ ("/sbin/getcfg", root / "no-getcfg"),+ ("/var/log/pulse-agent.log", root / "absent-log"),+ ("/etc/systemd/system/${AGENT_NAME}.service", root / "unit/pulse-agent.service"),+ ): source = source.replace(path, str(target))+ return source++ values = dict(FIXTURE=root, EVENTS=root / "events", STATE_DIR=root / "state",+ INSTALL_DIR=root / "bin",+ STATE_DIR_REMOVAL_AUTHORITY=root / "state",+ TRUENAS_STATE_DIR=root / "state", TRUENAS_BOOTSTRAP_SCRIPT=script,+ PRIVILEGED_HELPER_STATE_DIR=root / "helper", SAFE_PROFILE_STATE_DIR=root / "profile",+ PRIVILEGE_SUDOERS_FILE=root / "absent-sudoers", PRIVILEGE_HELPER_DIR=root / "absent-helper",+ REPLY=reply, FAULT=fault)+ setup = "\n".join(k + "=" + shlex.quote(str(v)) for k, v in values.items())+ shell = r'''+set -euo pipefail+UNINSTALL=true AGENT_NAME=pulse-agent BINARY_NAME=pulse-agent INSTALL_DIR=/usr/local/bin+EXIT_GENERAL=1 EXIT_MISSING_ARGS=16 PULSE_URL=https://same-instance.example.invalid AGENT_ID=synthetic+ACTION_RUNNER_NAME=pulse-agent-runner PRIVILEGED_HELPER_NAME=pulse-agent-helper+log_info() { :; }; log_warn() { :; }+fail() { printf '%s\n' "$1" >&2; exit "$2"; }+uname() { printf 'Linux\n'; }+command() {+ case "$*" in+ '-v rc-service') return 1 ;;+ '-v midclt') [[ "$FAULT" != missing-midclt ]] || return 1 ;;+ '-v python3') [[ "$FAULT" != missing-python ]] || return 1 ;;+ '-v timeout') [[ "$FAULT" != missing-timeout ]] || return 1 ;;+ esac+ builtin command "$@"+}+collector_credential_state_present() { [[ -e "$STATE_DIR/runtime.token" ]]; }+uninstall_collector_registration() { printf 'remote:remove\n' >> "$EVENTS"; [[ "$REPLY" == confirmed ]]; }+teardown_action_runner_service() { :; }+teardown_privileged_helper_service() { :; }+find_qnap_state_dir() { :; }+pkill() { printf 'unchecked:process-stop\n' >> "$EVENTS"; }+sleep() { :; }+rm() {+ local arg+ for arg in "$@"; do [[ "$arg" == -* || "$arg" == "$FIXTURE/"* ]] || return 97; done+ printf 'file:remove\n' >> "$EVENTS"+ builtin command rm "$@"+}+remove_agent_state_dir() { printf 'state:remove\n' >> "$EVENTS"; rm -rf -- "$STATE_DIR"; }+remove_privileged_helper_state_dir() { rm -rf -- "$PRIVILEGED_HELPER_STATE_DIR"; }+remove_safe_profile_state_dir() { rm -rf -- "$SAFE_PROFILE_STATE_DIR"; }+'''+ entry = self.source.split("# --- Uninstall Logic ---\n", 1)[1].split("# --- Validation ---\n", 1)[0]+ selected = definitions(self.source)+ if flow == "process": entry = 'read_legacy_agent_removal_processes "$TRUENAS_BOOTSTRAP_SCRIPT"\n'+ executable = shell + "\n" + setup + "\n" + redirect(selected) + "\nmain() {\n" + redirect(entry) + "}\nmain\n"+ env = dict(os.environ, FIXTURE=str(root), PATH=str(tools) + ":" + os.environ["PATH"])+ result = subprocess.run(["bash", "-c", executable], env=env, capture_output=True, text=True, timeout=12)+ second = subprocess.run(["bash", "-c", executable], env=env, capture_output=True, text=True, timeout=12) if repeat else None+ events = (root / "events").read_text() if (root / "events").exists() else ""+ after = self.snapshot(root)+ state = json.loads((root / "model.json").read_text())+ timed_stopped = not (root / "timed-pid").exists() or not Path('/proc/' + (root / "timed-pid").read_text()).exists()+ return result, second, events.splitlines(), before, after, state, timed_stopped++ @staticmethod+ def snapshot(root):+ result = {}+ for path in root.rglob("*"):+ if path.parts[len(root.parts)] not in ("state", "helper", "profile", "unit", "unrelated"):+ continue+ mode = path.lstat().st_mode+ contents = os.readlink(path) if stat.S_ISLNK(mode) else path.read_bytes() if stat.S_ISREG(mode) else None+ result[str(path.relative_to(root))] = (mode, contents, path.lstat().st_ino)+ return result++ def retained(self, result):+ execution, _, events, before, after, _, _ = result+ self.assertEqual(execution.returncode, 1, execution.stdout + execution.stderr)+ self.assertNotIn('state:remove', events)+ for path in before:+ if path.startswith(('state', 'helper', 'profile')) or path == 'unrelated': self.assertEqual(after.get(path), before[path], path)+ self.assertNotIn('synthetic recovery only', execution.stdout + execution.stderr)+ return events++ def test_unavailable_or_untrusted_task_inventory_cannot_remove_recovery(self):+ for fault in ('failed-query', 'empty-query', 'malformed-query', 'object-query', 'duplicate-field',+ 'diagnostic-query', 'duplicate-id', 'bool-id', 'string-enabled', 'wrong-kind',+ 'oversized-query', 'oversized-inventory'):+ with self.subTest(fault=fault):+ events = self.retained(self.exercise(fault))+ self.assertNotIn('remote:remove', events)+ self.assertFalse(any(e.startswith('task:disable:') or e.startswith('task:delete:') for e in events))++ def test_foreign_or_indirect_bootstrap_is_not_mutated(self):+ for fault in ('foreign-owner', 'command-bootstrap', 'other-bootstrap'):+ with self.subTest(fault=fault):+ events = self.retained(self.exercise(fault))+ self.assertNotIn('remote:remove', events)+ self.assertFalse(any(e.startswith('task:disable:') or e.startswith('task:delete:') for e in events))++ def test_bootstrap_symlink_directory_or_writable_file_refuses_before_mutation(self):+ for bootstrap in ('linked', 'directory', 'writable'):+ with self.subTest(bootstrap=bootstrap):+ events = self.retained(self.exercise(bootstrap=bootstrap))+ self.assertEqual(events, [])++ def test_missing_tools_are_not_local_only_removal_permission(self):+ for fault in ('missing-midclt', 'missing-python', 'missing-timeout'):+ with self.subTest(fault=fault):+ events = self.retained(self.exercise(fault))+ self.assertEqual(events, [])++ def test_every_matching_task_is_disabled_and_independently_observed_before_stop(self):+ result = self.exercise()+ self.assertEqual(result[0].returncode, 0, result[0].stderr)+ events = result[2]+ self.assertLess(events.index('task:disable:2'), events.index('unit:stop'))+ self.assertEqual(events[:4], ['task:query', 'task:disable:1', 'task:disable:2', 'task:query'])+ self.assertLess(events.index('process:read'), events.index('remote:remove'))++ def test_partial_refused_ignored_or_reactivated_disable_retains_recovery(self):+ for fault in ('refused-disable', 'partial-disable', 'ignored-disable', 'reactivated-task'):+ with self.subTest(fault=fault):+ events = self.retained(self.exercise(fault))+ self.assertNotIn('remote:remove', events)+ self.assertNotIn('unit:stop', events)++ def test_unit_or_process_uncertainty_keeps_stopped_task_recovery(self):+ for fault in ('failed-unit-read', 'refused-stop', 'ignored-stop', 'failed-process', 'empty-process',+ 'duplicate-process', 'diagnostic-process', 'orphan-agent', 'live-bootstrap'):+ with self.subTest(fault=fault):+ events = self.retained(self.exercise(fault))+ self.assertNotIn('remote:remove', events)++ def test_task_activation_during_systemd_stop_refuses_before_remote(self):+ result = self.exercise('reactivated-task-on-stop')+ events = self.retained(result)+ self.assertIn('unit:stop', events)+ self.assertNotIn('remote:remove', events)+ self.assertFalse(any(e.startswith('task:delete:') for e in events))++ def test_server_refusal_retains_disabled_tasks_and_all_recovery_without_restart(self):+ result = self.exercise(reply='refused')+ events = self.retained(result)+ self.assertIn('remote:remove', events)+ self.assertFalse(any(e.startswith('task:delete:') for e in events))+ self.assertTrue(all(not t['enabled'] for t in result[5]['tasks'] if t['id'] != 99))++ def test_changed_or_refused_post_remote_inventory_precedes_first_deletion(self):+ for fault in ('failed-after-remote-query', 'new-task-after-remote'):+ with self.subTest(fault=fault):+ events = self.retained(self.exercise(fault))+ self.assertIn('remote:remove', events)+ self.assertFalse(any(e.startswith('task:delete:') for e in events))++ def test_partial_refused_ignored_or_recreated_task_deletion_keeps_recovery(self):+ for fault in ('refused-delete', 'partial-delete', 'ignored-delete', 'recreated-task', 'unit-reactivation-on-delete'):+ with self.subTest(fault=fault): self.retained(self.exercise(fault))++ def test_all_owned_tasks_retire_before_state_erasure_and_unrelated_task_survives(self):+ result = self.exercise()+ self.assertEqual(result[0].returncode, 0, result[0].stderr)+ events = result[2]+ self.assertLess(events.index('task:delete:2'), events.index('state:remove'))+ self.assertIn('task:query', events[events.index('task:delete:2') + 1:events.index('state:remove')])+ self.assertEqual([t['id'] for t in result[5]['tasks']], [99])+ self.assertTrue(result[5]['tasks'][0]['enabled'])+ self.assertNotIn('state/runtime.token', result[4])+ self.assertEqual(result[4]['unrelated'], result[3]['unrelated'])++ def test_disabled_absent_missing_bootstrap_and_repeated_removal_are_observed(self):+ for tasks, bootstrap in (('disabled', 'regular'), ('absent', 'missing'), ('single', 'missing')):+ with self.subTest(tasks=tasks, bootstrap=bootstrap):+ result = self.exercise(tasks=tasks, bootstrap=bootstrap, repeat=True)+ self.assertEqual(result[0].returncode, 0, result[0].stderr)+ self.assertEqual(result[1].returncode, 0, result[1].stderr)+ self.assertEqual([t['id'] for t in result[5]['tasks']], [99])++ def test_ordinary_linux_state_is_not_appliance_evidence(self):+ result = self.exercise(bootstrap='missing', tasks='absent', appliance=False)+ self.assertEqual(result[0].returncode, 0, result[0].stderr)+ self.assertFalse(any(e.startswith('task:') for e in result[2]))++ def test_process_read_exposes_only_roles_and_real_timeout_child_is_stopped(self):+ result = self.exercise(fault='live-bootstrap', flow='process')+ self.assertEqual(result[0].returncode, 0, result[0].stderr)+ self.assertIn('101 1 bootstrap', result[0].stdout)+ self.assertNotIn('recovery-only', result[0].stdout + result[0].stderr)+ result = self.exercise(fault='timeout-query')+ self.retained(result)+ self.assertTrue(result[6], 'timed middleware child still exists')+++if __name__ == '__main__':+ unittest.main(verbosity=2)diff --git a/scripts/installtests/agent_uninstall_state.py b/scripts/installtests/agent_uninstall_state.pyindex 029e6a8fe0..725ac2b4c4 100644--- a/scripts/installtests/agent_uninstall_state.py+++ b/scripts/installtests/agent_uninstall_state.py@@ -48,7 +48,7 @@ class UnixUninstallState(unittest.TestCase):# Stop before unrelated platform boot/log cleanup; never run it on a host.entry = source.split("# --- Uninstall Logic ---\n", 1)[1].split(" # Remove least-privilege helper artifacts.", 1)[0] + "fi\n"- names = ("collector_credential_state_present", "uninstall_collector_registration",+ names = ("truenas_scale_agent_removal_present", "collector_credential_state_present", "uninstall_collector_registration","remove_authorized_runtime_dir", "remove_agent_state_dir","remove_privileged_helper_state_dir", "remove_safe_profile_state_dir", "fail")with tempfile.TemporaryDirectory() as temporary:diff --git a/scripts/installtests/agent_upstart_teardown.py b/scripts/installtests/agent_upstart_teardown.pyindex 376a405435..3b4a6a5d4b 100644--- a/scripts/installtests/agent_upstart_teardown.py+++ b/scripts/installtests/agent_upstart_teardown.py@@ -16,7 +16,7 @@ import unittestROOT = Path(__file__).resolve().parents[2]SOURCE = Path(os.environ.get("PULSE_AGENT_UPSTART_SOURCE", ROOT / "scripts/install.sh"))-FUNCTIONS = ("cleanup", "portable_path_uid", "portable_path_mode", "trusted_lifecycle_regular_file",+FUNCTIONS = ("truenas_scale_agent_removal_present", "cleanup", "portable_path_uid", "portable_path_mode", "trusted_lifecycle_regular_file","read_legacy_agent_removal_processes", "legacy_agent_processes_absent","read_upstart_agent_removal_status", "upstart_agent_removal_paths_safe","read_upstart_agent_removal_enablement", "quiesce_upstart_agent_for_removal",diff --git a/scripts/installtests/install_sh_test.go b/scripts/installtests/install_sh_test.goindex 490b04be73..928e005506 100644--- a/scripts/installtests/install_sh_test.go+++ b/scripts/installtests/install_sh_test.go@@ -8501,3 +8501,41 @@ func TestInstallSHUpstartTeardownRequiresObservedQuiescence(t *testing.T) {}t.Logf("%s", output)}++func TestInstallSHTrueNASScaleBootTasksRetireBeforeRecoveryErasure(t *testing.T) {+ if runtime.GOOS == "windows" {+ t.Skip("TrueNAS SCALE shell controls; Windows has its own native SCM gate")+ }+ ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)+ defer cancel()+ cmd := exec.CommandContext(ctx, "python3", repoFile("scripts", "installtests", "agent_truenas_teardown.py"))+ output, err := cmd.CombinedOutput()+ if err != nil {+ t.Fatalf("TrueNAS SCALE boot-task ownership/retirement controls: %v\n%s", err, output)+ }+ for _, name := range []string{+ "test_unavailable_or_untrusted_task_inventory_cannot_remove_recovery",+ "test_foreign_or_indirect_bootstrap_is_not_mutated",+ "test_bootstrap_symlink_directory_or_writable_file_refuses_before_mutation",+ "test_missing_tools_are_not_local_only_removal_permission",+ "test_every_matching_task_is_disabled_and_independently_observed_before_stop",+ "test_partial_refused_ignored_or_reactivated_disable_retains_recovery",+ "test_unit_or_process_uncertainty_keeps_stopped_task_recovery",+ "test_task_activation_during_systemd_stop_refuses_before_remote",+ "test_server_refusal_retains_disabled_tasks_and_all_recovery_without_restart",+ "test_changed_or_refused_post_remote_inventory_precedes_first_deletion",+ "test_partial_refused_ignored_or_recreated_task_deletion_keeps_recovery",+ "test_all_owned_tasks_retire_before_state_erasure_and_unrelated_task_survives",+ "test_disabled_absent_missing_bootstrap_and_repeated_removal_are_observed",+ "test_ordinary_linux_state_is_not_appliance_evidence",+ "test_process_read_exposes_only_roles_and_real_timeout_child_is_stopped",+ } {+ if !strings.Contains(string(output), name+" (__main__.AgentTrueNASTeardown."+name+") ... ok") {+ t.Fatalf("TrueNAS SCALE control %s did not pass:\n%s", name, output)+ }+ }+ if !strings.Contains(string(output), "\nRan 15 tests in ") || !strings.Contains(string(output), "\nOK\n") {+ t.Fatalf("incomplete TrueNAS SCALE verdict:\n%s", output)+ }+ t.Logf("%s", output)+}