Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Retain SCALE agent recovery until every boot task is retired

pulse-triage[bot]59ff0dabcf84
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 7193c99f46..d71de6fb80 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -15,6 +15,48 @@
## Purpose
+### TrueNAS SCALE removal retires every owned middleware boot task before recovery erasure
+
+TrueNAS SCALE has an independent activation owner: its Init/Shutdown bootstrap
+can recreate and enable the systemd unit after a normal disable. A recovered
+state directory or an unrelated `/etc/version` alone is not appliance identity.
+The uninstall caller selects SCALE only from a bootstrap footprint or vendor
+marker. The existing protected/explicit state-removal authority must select the
+same state directory and exact bootstrap path; linked, writable or non-regular
+bootstrap files refuse before task/service mutation.
+
+One complete, bounded middleware inventory must identify every exact generated
+SCRIPT/POSTINIT task, including duplicates. Unknown ownership, indirect command
+activation or another-install bootstrap refuses without deleting unrelated
+tasks. The JSON reader rejects duplicate keys/IDs, invalid types, failed/empty/
+diagnostic/incomplete reads, over 512 tasks or over 1 MiB; it returns only owned
+numeric IDs and enablement, never task commands, names or credentials. All owned
+tasks are disabled with checked calls and independent exact-set readback before
+systemd stop. A partial stop/disable stays stopped/disabled: do not re-enable or
+automatically restart after a refusal.
+
+The complete systemd target set and independent kernel collector/bootstrap
+process inventory and a final exact disabled-task readback must be quiescent
+before same-instance server removal, including task changes during systemd stop.
+After remote confirmation, the exact disabled task set is re-read before the first
+task deletion. Every owned deletion is checked; final whole-unit/process
+quiescence and complete task absence precede credential, stored binary,
+bootstrap and state erasure. A changed set, refusal, ignored deletion, new task
+or reactivation retains local recovery, even if earlier task/server operations
+succeeded. No favourable replacement observation, task recreation or credential
+churn is performed. Unrelated tasks survive unchanged.
+
+`TestInstallSHTrueNASScaleBootTasksRetireBeforeRecoveryErasure` requires all
+fifteen `agent_truenas_teardown.py` actual-function/full-caller controls,
+including multiple tasks, partial/refused/unknown operations, post-remote
+changes, ignored/recreated deletion, process privacy and a real timeout child.
+These run against closed middleware/systemd/process models and private ordinary-
+user files, not a native appliance. Existing systemd/OpenRC/DSM/rc.d/SysV gates,
+transport trust and identity remain. TrueNAS CORE retains its linked rc.d/
+appliance refusal; Unraid/QNAP boot ownership, Windows and signed installed
+Linux/PowerShell/native/TLS/cleanup acceptance remain separate. This does not
+expand the original removal job into macOS or introduce another dispatcher.
+
### DSM Upstart removal retains recovery until automatic activation and the job are absent
DSM 6 Upstart jobs (including leftovers on DSM 7) are not classified as SysV.
diff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.md
index e71c8a5d55..b6d4667eec 100644
--- a/docs/release-control/v6/internal/subsystems/deployment-installability.md
+++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md
@@ -16,6 +16,48 @@
## Purpose
+### TrueNAS SCALE removal retires every owned middleware boot task before recovery erasure
+
+TrueNAS SCALE has an independent activation owner: its Init/Shutdown bootstrap
+can recreate and enable the systemd unit after a normal disable. A recovered
+state directory or an unrelated `/etc/version` alone is not appliance identity.
+The uninstall caller selects SCALE only from a bootstrap footprint or vendor
+marker. The existing protected/explicit state-removal authority must select the
+same state directory and exact bootstrap path; linked, writable or non-regular
+bootstrap files refuse before task/service mutation.
+
+One complete, bounded middleware inventory must identify every exact generated
+SCRIPT/POSTINIT task, including duplicates. Unknown ownership, indirect command
+activation or another-install bootstrap refuses without deleting unrelated
+tasks. The JSON reader rejects duplicate keys/IDs, invalid types, failed/empty/
+diagnostic/incomplete reads, over 512 tasks or over 1 MiB; it returns only owned
+numeric IDs and enablement, never task commands, names or credentials. All owned
+tasks are disabled with checked calls and independent exact-set readback before
+systemd stop. A partial stop/disable stays stopped/disabled: do not re-enable or
+automatically restart after a refusal.
+
+The complete systemd target set and independent kernel collector/bootstrap
+process inventory and a final exact disabled-task readback must be quiescent
+before same-instance server removal, including task changes during systemd stop.
+After remote confirmation, the exact disabled task set is re-read before the first
+task deletion. Every owned deletion is checked; final whole-unit/process
+quiescence and complete task absence precede credential, stored binary,
+bootstrap and state erasure. A changed set, refusal, ignored deletion, new task
+or reactivation retains local recovery, even if earlier task/server operations
+succeeded. No favourable replacement observation, task recreation or credential
+churn is performed. Unrelated tasks survive unchanged.
+
+`TestInstallSHTrueNASScaleBootTasksRetireBeforeRecoveryErasure` requires all
+fifteen `agent_truenas_teardown.py` actual-function/full-caller controls,
+including multiple tasks, partial/refused/unknown operations, post-remote
+changes, ignored/recreated deletion, process privacy and a real timeout child.
+These run against closed middleware/systemd/process models and private ordinary-
+user files, not a native appliance. Existing systemd/OpenRC/DSM/rc.d/SysV gates,
+transport trust and identity remain. TrueNAS CORE retains its linked rc.d/
+appliance refusal; Unraid/QNAP boot ownership, Windows and signed installed
+Linux/PowerShell/native/TLS/cleanup acceptance remain separate. This does not
+expand the original removal job into macOS or introduce another dispatcher.
+
### DSM Upstart removal retains recovery until automatic activation and the job are absent
DSM 6 Upstart jobs (including leftovers on DSM 7) are not classified as SysV.
diff --git a/scripts/install.sh b/scripts/install.sh
index 543578ece3..82134ddf26 100755
--- a/scripts/install.sh
+++ b/scripts/install.sh
@@ -1344,6 +1344,7 @@ teardown_openrc_agent_service() {
# cannot establish that after an ignored signal or lost PID file.
read_legacy_agent_removal_processes() {
local observation="" line="" pid="" parent="" comm="" args="" role=""
+ local bootstrap="${1:-}"
local complete="false" count=0 parents=" " seen=" "
local -a rows=()
observation=$(LC_ALL=C timeout -k 1 5 ps -ax -o pid= -o ppid= -o comm= -o args= 2>&1 &&
@@ -1365,6 +1366,7 @@ read_legacy_agent_removal_processes() {
elif [[ "${comm##*/}" == daemon && "$args" =~ (^|[[:space:]/])pulse-agent([[:space:]\[]|$) ]]; then
role="supervisor"
fi
+ if [[ -n "$bootstrap" && "$args" == *"$bootstrap"* ]]; then role="bootstrap"; fi
rows+=("$pid $parent $role ${comm##*/}")
count=$((count + 1))
[[ $count -le 65536 ]] || return 1
@@ -1706,6 +1708,130 @@ enable_sysv_agent_service() {
log_info "Created rc.d symlinks manually."
}
+# TrueNAS SCALE has a second activation owner: middleware's boot task can
+# recreate and enable the systemd unit even after systemctl disable. Recovered
+# state directories alone are not appliance identity (ordinary Linux has them).
+truenas_scale_agent_removal_present() {
+ [[ "$(uname -s)" == Linux ]] || return 1
+ [[ -e "${TRUENAS_BOOTSTRAP_SCRIPT:-}" || -L "${TRUENAS_BOOTSTRAP_SCRIPT:-}" ||
+ -f /etc/truenas-version || -d /data/ix-applications || -d /etc/ix-apps.d ]] && return 0
+ [[ -f /etc/version ]] && grep -qi truenas /etc/version
+}
+
+read_truenas_agent_removal_tasks() {
+ local observation=""
+ observation=$(LC_ALL=C timeout -k 1 5 midclt call initshutdownscript.query 2>&1 &&
+ printf '\n%s' '__pulse_truenas_tasks_complete__') || return 1
+ # Parse the complete inventory, including indirect/other-install bootstrap
+ # references. No task command, name, argument or credential is returned.
+ python3 -c '
+import json, pathlib, sys
+def unique_object(pairs):
+ result = {}
+ for key, value in pairs:
+ if key in result: raise ValueError("duplicate field")
+ result[key] = value
+ return result
+def reject_constant(value):
+ raise ValueError("non-JSON constant")
+try:
+ raw = sys.stdin.buffer.read(1048577)
+ suffix = b"\n__pulse_truenas_tasks_complete__\n"
+ if len(raw) > 1048576 or not raw.endswith(suffix): raise ValueError("incomplete inventory")
+ tasks = json.loads(raw[:-len(suffix)].decode("utf-8"), object_pairs_hook=unique_object,
+ parse_constant=reject_constant)
+ if not isinstance(tasks, list) or len(tasks) > 512: raise ValueError("unbounded inventory")
+ expected, seen, owned = sys.argv[1], set(), []
+ for task in tasks:
+ if not isinstance(task, dict): raise ValueError("invalid task")
+ ident, kind, enabled = task.get("id"), task.get("type"), task.get("enabled")
+ if type(ident) is not int or not 0 < ident <= 2147483647 or ident in seen:
+ raise ValueError("invalid identity")
+ seen.add(ident)
+ if kind not in ("SCRIPT", "COMMAND") or type(enabled) is not bool:
+ raise ValueError("unknown activation")
+ script, command = task.get("script", ""), task.get("command", "")
+ if script is None: script = ""
+ if command is None: command = ""
+ if not isinstance(script, str) or not isinstance(command, str): raise ValueError("invalid activation")
+ if script == expected:
+ if (kind != "SCRIPT" or command or task.get("when") != "POSTINIT" or
+ task.get("comment") != "Pulse Agent Bootstrap" or type(task.get("timeout")) is not int or
+ task["timeout"] != 30): raise ValueError("unowned bootstrap")
+ owned.append((ident, enabled))
+ elif (pathlib.PurePosixPath(script).name == "bootstrap-pulse-agent.sh" or
+ "bootstrap-pulse-agent.sh" in command or "pulse-agent" in command):
+ raise ValueError("indirect or other-install activation")
+ for ident, enabled in sorted(owned): print(ident, "true" if enabled else "false")
+ print("complete")
+except (ValueError, TypeError, KeyError, UnicodeError):
+ print("TrueNAS boot-task inventory is incomplete or not owned; recovery retained.", file=sys.stderr)
+ sys.exit(1)
+' "$TRUENAS_BOOTSTRAP_SCRIPT" <<< "$observation"
+}
+
+quiesce_truenas_agent_boot_tasks_for_removal() {
+ local inventory="" expected="" observed="" ident="" enabled="" mode=""
+ command -v midclt >/dev/null 2>&1 && command -v python3 >/dev/null 2>&1 &&
+ command -v timeout >/dev/null 2>&1 && command -v systemctl >/dev/null 2>&1 || return 1
+ # Do not let a platform marker or an arbitrary saved state path authorise
+ # erasure of a different appliance install. Keep the existing state owner.
+ [[ "$TRUENAS_STATE_DIR" == "$STATE_DIR" &&
+ "${STATE_DIR_REMOVAL_AUTHORITY:-}" == "$STATE_DIR" &&
+ "$TRUENAS_BOOTSTRAP_SCRIPT" == "$STATE_DIR/bootstrap-pulse-agent.sh" ]] || return 1
+ if [[ -e "$TRUENAS_BOOTSTRAP_SCRIPT" || -L "$TRUENAS_BOOTSTRAP_SCRIPT" ]]; then
+ mode=$(portable_path_mode "$TRUENAS_BOOTSTRAP_SCRIPT") || return 1
+ [[ "$mode" =~ ^[0-7]{3,4}$ ]] && (( (8#$mode & 0022) == 0 )) || return 1
+ trusted_lifecycle_regular_file "$TRUENAS_BOOTSTRAP_SCRIPT" "$mode" || return 1
+ fi
+ inventory=$(read_truenas_agent_removal_tasks) || return 1
+ while read -r ident enabled; do
+ [[ "$ident" != complete ]] || continue
+ if [[ "$enabled" == true ]]; then
+ LC_ALL=C timeout -k 1 5 midclt call initshutdownscript.update "$ident" '{"enabled":false}' >/dev/null 2>&1 || return 1
+ fi
+ expected+="$ident false"$'\n'
+ done <<< "$inventory"
+ expected+=complete
+ observed=$(read_truenas_agent_removal_tasks) || return 1
+ [[ "$observed" == "$expected" ]] || return 1
+ printf '%s\n' "$expected"
+}
+
+confirm_truenas_agent_removal_quiescent() {
+ local unit="" inventory="" observed=""
+ for unit in "${ACTION_RUNNER_NAME}.service" "${PRIVILEGED_HELPER_NAME}.socket" \
+ "${PRIVILEGED_HELPER_NAME}.service" "${AGENT_NAME}.service"; do
+ confirm_agent_unit_quiescent "$unit" || return 1
+ done
+ inventory=$(read_legacy_agent_removal_processes "$TRUENAS_BOOTSTRAP_SCRIPT") || return 1
+ legacy_agent_processes_absent "$inventory" && [[ "$inventory" != *' bootstrap'* ]] || return 1
+ # The middleware set can change during the preceding systemd operations.
+ # Close it again before remote removal/first deletion, not only afterwards.
+ if [[ $# -gt 0 ]]; then
+ observed=$(read_truenas_agent_removal_tasks) || return 1
+ [[ "$observed" == "$1" ]] || return 1
+ fi
+}
+
+teardown_truenas_agent_boot_tasks() {
+ local expected="$1" observed="" ident="" enabled=""
+ # No newly discovered task is disabled/deleted after remote removal. A
+ # changed set, refused read or concurrent activation retains local recovery.
+ observed=$(read_truenas_agent_removal_tasks) || return 1
+ [[ "$observed" == "$expected" ]] || return 1
+ confirm_truenas_agent_removal_quiescent "$expected" || return 1
+ while read -r ident enabled; do
+ [[ "$ident" != complete ]] || continue
+ LC_ALL=C timeout -k 1 5 midclt call initshutdownscript.delete "$ident" >/dev/null 2>&1 || return 1
+ done <<< "$expected"
+ # Close the whole manager/process/task set after the last deletion, while
+ # every credential, bootstrap, stored binary and service file still exists.
+ confirm_truenas_agent_removal_quiescent || return 1
+ observed=$(read_truenas_agent_removal_tasks) || return 1
+ [[ "$observed" == complete ]]
+}
+
write_truenas_bootstrap_script() {
local platform="$1"
local service_link=""
@@ -5999,6 +6125,8 @@ if [[ "$UNINSTALL" == "true" ]]; then
local uninstall_has_credential="false"
local uninstall_legacy_manager=""
local uninstall_upstart="false"
+ local uninstall_truenas_scale="false"
+ local uninstall_truenas_tasks=""
# Validate saved connection availability before stopping a working install.
# Stop every systemd activation source before remote removal, process kills
@@ -6009,6 +6137,14 @@ if [[ "$UNINSTALL" == "true" ]]; then
fail "The saved Pulse URL is missing; local credentials and services were retained. Restore saved connection state or provide --url for the same trusted Pulse instance and retry uninstall." "$EXIT_MISSING_ARGS"
fi
fi
+ # Middleware can recreate and enable the systemd unit at boot. Disable all
+ # exact owned tasks before stopping services; a partial/refused observation
+ # retains every recovery file and never restores automatic activation.
+ if truenas_scale_agent_removal_present; then
+ uninstall_truenas_scale="true"
+ uninstall_truenas_tasks=$(quiesce_truenas_agent_boot_tasks_for_removal) ||
+ fail "TrueNAS SCALE removal could not confirm every owned boot task disabled; credentials, bootstrap and service files were retained. Reconcile middleware task ownership/state before retrying; no automatic restart was attempted." "$EXIT_GENERAL"
+ fi
# Detect stale DSM 6 jobs even on a DSM 7 host. Another init manager cannot
# establish their absence; missing initctl is an explicit safe refusal.
if [[ -d /usr/syno ]] && { command -v initctl >/dev/null 2>&1 ||
@@ -6037,6 +6173,10 @@ if [[ "$UNINSTALL" == "true" ]]; then
quiesce_sysv_agent_for_removal ||
fail "SysV removal could not confirm stopped process and disabled runlevels; local recovery files were retained. A preceding stop or disable may have succeeded; no automatic restart was attempted." "$EXIT_GENERAL"
fi
+ if [[ "$uninstall_truenas_scale" == true ]]; then
+ confirm_truenas_agent_removal_quiescent "$uninstall_truenas_tasks" ||
+ fail "TrueNAS SCALE removal could not confirm stopped units, collector and bootstrap processes; local recovery was retained. Reconcile stopped/disabled state before retrying; no automatic restart was attempted." "$EXIT_GENERAL"
+ fi
# A credential-bearing install must durably remove its exact server record
# through the same CA/fingerprint/no-proxy transport used by lifecycle
@@ -6068,6 +6208,10 @@ if [[ "$UNINSTALL" == "true" ]]; then
teardown_upstart_agent_service ||
fail "DSM Upstart job retirement was not confirmed; local recovery state was retained. The manual override remains after a partial stop. Reconcile the original config or its .removal recovery copy before retrying; no automatic restart was attempted." "$EXIT_GENERAL"
fi
+ if [[ "$uninstall_truenas_scale" == true ]]; then
+ teardown_truenas_agent_boot_tasks "$uninstall_truenas_tasks" ||
+ fail "TrueNAS SCALE boot-task retirement was not confirmed; local recovery was retained. A preceding task disable/delete or server removal may have succeeded. Reconcile middleware and stopped service state before retrying; no automatic restart was attempted." "$EXIT_GENERAL"
+ fi
# Kill wrapper scripts first: they are watchdogs, so stopping the agent
# while its wrapper still loops only races the respawn. No leading path
@@ -6181,15 +6325,12 @@ if [[ "$UNINSTALL" == "true" ]]; then
fi
fi
- # TrueNAS SCALE/CORE
- if [[ -d "$TRUENAS_STATE_DIR" ]] || [[ -f /etc/truenas-version ]] || [[ -f /etc/version ]]; then
- if [[ "$(uname -s)" == "Linux" ]]; then
- log_info "Removing TrueNAS SCALE installation..."
- teardown_systemd_agent_service
- elif [[ "$(uname -s)" == "FreeBSD" ]]; then
- log_info "Removing TrueNAS CORE installation..."
- # rc.d teardown was confirmed before remote/local recovery erasure.
- fi
+ # SCALE's task retirement was confirmed above, before authorised state
+ # erasure. A recovered ordinary Linux directory is not a TrueNAS install.
+ # CORE remains behind its existing rc.d/appliance activation refusal; this
+ # SCALE step does not claim CORE boot-task or native platform acceptance.
+ if [[ "$(uname -s)" == "FreeBSD" ]] && { [[ -d "$TRUENAS_STATE_DIR" ]] || [[ -f /etc/truenas-version ]] || [[ -f /etc/version ]]; }; then
+ log_info "Removing TrueNAS CORE installation..."
# Remove Init/Shutdown task
if command -v midclt >/dev/null 2>&1 && command -v python3 >/dev/null 2>&1; then
TASK_ID=$(midclt call initshutdownscript.query '[["script","=","'"$TRUENAS_BOOTSTRAP_SCRIPT"'"]]' 2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); print(d[0]['id'] if d else '')" 2>/dev/null || echo "")
diff --git a/scripts/installtests/agent_legacy_teardown.py b/scripts/installtests/agent_legacy_teardown.py
index 54afa1b99c..51dc2a0587 100644
--- a/scripts/installtests/agent_legacy_teardown.py
+++ b/scripts/installtests/agent_legacy_teardown.py
@@ -17,7 +17,7 @@ import unittest
ROOT = Path(__file__).resolve().parents[2]
SOURCE = Path(os.environ.get("PULSE_AGENT_LEGACY_SOURCE", ROOT / "scripts/install.sh"))
-FUNCTIONS = ("read_legacy_agent_removal_processes", "legacy_agent_processes_absent",
+FUNCTIONS = ("truenas_scale_agent_removal_present", "read_legacy_agent_removal_processes", "legacy_agent_processes_absent",
"validate_legacy_agent_removal_pidfiles", "read_legacy_agent_removal_status",
"read_freebsd_agent_removal_enablement", "quiesce_freebsd_agent_for_removal",
"teardown_freebsd_agent_service", "read_sysv_agent_removal_links",
diff --git a/scripts/installtests/agent_openrc_teardown.py b/scripts/installtests/agent_openrc_teardown.py
index a2cc349ea5..fbc2102ebd 100644
--- a/scripts/installtests/agent_openrc_teardown.py
+++ b/scripts/installtests/agent_openrc_teardown.py
@@ -21,7 +21,7 @@ import unittest
ROOT = Path(__file__).resolve().parents[2]
SOURCE = Path(os.environ.get("PULSE_AGENT_OPENRC_SOURCE", ROOT / "scripts/install.sh"))
-FUNCTIONS = ("read_openrc_agent_removal_status", "read_openrc_agent_removal_runlevels",
+FUNCTIONS = ("truenas_scale_agent_removal_present", "read_openrc_agent_removal_status", "read_openrc_agent_removal_runlevels",
"quiesce_openrc_agent_for_removal", "teardown_openrc_agent_service")
diff --git a/scripts/installtests/agent_truenas_teardown.py b/scripts/installtests/agent_truenas_teardown.py
new file mode 100644
index 0000000000..b5542df5b8
--- /dev/null
+++ b/scripts/installtests/agent_truenas_teardown.py
@@ -0,0 +1,377 @@
+#!/usr/bin/env python3
+"""TrueNAS SCALE uninstall, using production functions and private manager models.
+
+Only fixed vendor/service paths are redirected into owned temporary files.
+No appliance, root installer, network, credential or native acceptance is used.
+The complete uninstall caller runs; middleware/process/systemd are closed models.
+"""
+import json
+import os
+from pathlib import Path
+import shlex
+import stat
+import subprocess
+import tempfile
+import unittest
+
+ROOT = Path(__file__).resolve().parents[2]
+SOURCE = Path(os.environ.get("PULSE_AGENT_TRUENAS_SOURCE", ROOT / "scripts/install.sh"))
+FUNCTIONS = (
+ "portable_path_uid", "portable_path_mode", "trusted_lifecycle_regular_file",
+ "read_agent_removal_unit_state", "confirm_agent_unit_quiescent",
+ "quiesce_agent_units_for_removal", "teardown_systemd_agent_service",
+ "read_legacy_agent_removal_processes", "legacy_agent_processes_absent",
+ "truenas_scale_agent_removal_present", "read_truenas_agent_removal_tasks",
+ "quiesce_truenas_agent_boot_tasks_for_removal", "confirm_truenas_agent_removal_quiescent",
+ "teardown_truenas_agent_boot_tasks",
+)
+
+
+def definitions(source):
+ selected = []
+ for name in FUNCTIONS:
+ marker = name + "() {"
+ if marker not in source: # The exact-parent adverse comparison.
+ continue
+ if source.count(marker) != 1:
+ raise ValueError("ambiguous production function " + name)
+ selected.append(marker + source.split(marker, 1)[1].split("\n}\n", 1)[0] + "\n}\n")
+ return "\n".join(selected)
+
+
+MODEL = r'''#!/usr/bin/python3
+import json, os, pathlib, sys
+root = pathlib.Path(os.environ['FIXTURE'])
+path = root / 'model.json'
+v = json.loads(path.read_text())
+role, args = pathlib.Path(sys.argv[0]).name, sys.argv[1:]
+fault = v['fault']
+def save():
+ pending = root / 'model.pending'
+ pending.write_text(json.dumps(v)); pending.replace(path)
+def event(s):
+ with (root / 'events').open('a') as f: f.write(s + '\n')
+remote = (root / 'events').exists() and 'remote:remove' in (root / 'events').read_text()
+if role == 'ps':
+ assert args == ['-ax', '-o', 'pid=', '-o', 'ppid=', '-o', 'comm=', '-o', 'args='], args
+ event('process:read')
+ if fault == 'failed-process': sys.exit(1)
+ if fault == 'empty-process': sys.exit(0)
+ print('1 0 init /sbin/init')
+ if fault == 'duplicate-process': print('1 0 init /sbin/init')
+ if v['active'] == 'active' or fault == 'orphan-agent':
+ print('100 1 pulse-agent /usr/local/bin/pulse-agent --token-file /synthetic/recovery-only')
+ if fault == 'live-bootstrap': print('101 1 bash /bin/bash ' + str(root / 'state/bootstrap-pulse-agent.sh'))
+ if fault == 'diagnostic-process': print('read unavailable', file=sys.stderr)
+ sys.exit(0)
+if role == 'systemctl':
+ verb = args[0]
+ if verb == 'daemon-reload': event('unit:reload'); sys.exit(0)
+ assert len(args) >= 2, args
+ unit = args[1]
+ assert unit in ('pulse-agent.service', 'pulse-agent-helper.service', 'pulse-agent-helper.socket', 'pulse-agent-runner.service'), args
+ if verb == 'show':
+ assert args[2:] == ['--property=LoadState,ActiveState,UnitFileState'], args
+ event('unit:read:' + unit)
+ if fault == 'failed-unit-read': sys.exit(1)
+ present = unit == 'pulse-agent.service' and (root / 'unit/pulse-agent.service').exists()
+ if present:
+ print('LoadState=loaded\nActiveState=' + v['active'] + '\nUnitFileState=' + v['enabled'])
+ else: print('LoadState=not-found\nActiveState=inactive\nUnitFileState=')
+ sys.exit(0)
+ assert verb in ('stop', 'disable'), args
+ event('unit:' + verb)
+ if fault == 'refused-stop' and verb == 'stop': sys.exit(1)
+ if verb == 'stop' and fault != 'ignored-stop': v['active'] = 'inactive'
+ if verb == 'stop' and fault == 'reactivated-task-on-stop': v['tasks'][0]['enabled'] = True
+ if verb == 'disable': v['enabled'] = 'disabled'
+ save(); sys.exit(0)
+assert role == 'midclt' and args[0] == 'call', (role, args)
+verb = args[1]
+if verb == 'initshutdownscript.query':
+ assert len(args) in (2, 3), args
+ event('task:query'); v['queries'] += 1
+ if fault == 'timeout-query':
+ save(); (root / 'timed-pid').write_text(str(os.getpid()))
+ os.execl('/bin/sleep', 'sleep', '30')
+ if fault == 'failed-query' or (fault == 'failed-after-remote-query' and remote): save(); sys.exit(1)
+ if fault == 'new-task-after-remote' and remote and not any(t['id'] == 3 for t in v['tasks']):
+ v['tasks'].append(dict(v['tasks'][0], id=3, enabled=True))
+ if fault == 'reactivated-task' and v['updates'] and v['tasks']: v['tasks'][0]['enabled'] = True
+ if fault == 'recreated-task' and v['deletes'] and not any(t['script'] == str(root / 'state/bootstrap-pulse-agent.sh') for t in v['tasks']):
+ v['tasks'].append(dict(id=3, type='SCRIPT', script=str(root / 'state/bootstrap-pulse-agent.sh'), command='', when='POSTINIT', enabled=True, timeout=30, comment='Pulse Agent Bootstrap'))
+ save()
+ tasks = v['tasks']
+ if len(args) == 3: # Preserve the parent's exact filter, not a new graph.
+ filters = json.loads(args[2]); assert filters[0][:2] == ['script', '='], filters
+ tasks = [t for t in tasks if t['script'] == filters[0][2]]
+ if fault == 'empty-query': sys.exit(0)
+ if fault == 'malformed-query': print('not-json'); sys.exit(0)
+ if fault == 'object-query': print('{}'); sys.exit(0)
+ if fault == 'duplicate-field': print('[{"id":1,"id":2}]'); sys.exit(0)
+ if fault == 'oversized-query': print(' ' * 1048577 + '[]'); sys.exit(0)
+ if fault == 'oversized-inventory': tasks = [dict(v['tasks'][0], id=i + 1) for i in range(513)]
+ if fault == 'duplicate-id': tasks = tasks + [tasks[0]]
+ if fault == 'bool-id': tasks[0]['id'] = True
+ if fault == 'string-enabled': tasks[0]['enabled'] = 'false'
+ if fault == 'foreign-owner': tasks[0]['comment'] = 'not owned by Pulse'
+ if fault == 'wrong-kind': tasks[0]['type'] = 'UNKNOWN'
+ if fault == 'command-bootstrap': tasks[0].update(type='COMMAND', script='', command='/bin/bash ' + str(root / 'state/bootstrap-pulse-agent.sh'))
+ if fault == 'other-bootstrap': tasks[0]['script'] = '/other-install/bootstrap-pulse-agent.sh'
+ if fault == 'diagnostic-query': print('read unavailable', file=sys.stderr)
+ print(json.dumps(tasks)); sys.exit(0)
+if verb == 'initshutdownscript.update':
+ assert len(args) == 4 and json.loads(args[3]) == {'enabled': False}, args
+ ident = int(args[2]); event('task:disable:' + str(ident)); v['updates'] += 1
+ if fault == 'refused-disable' or (fault == 'partial-disable' and ident == 2): save(); sys.exit(1)
+ task = next(t for t in v['tasks'] if t['id'] == ident)
+ if fault != 'ignored-disable': task['enabled'] = False
+ save(); print(json.dumps(task)); sys.exit(0)
+assert verb == 'initshutdownscript.delete' and len(args) == 3, args
+ident = int(args[2]); event('task:delete:' + str(ident)); v['deletes'] += 1
+if fault == 'refused-delete' or (fault == 'partial-delete' and ident == 2): save(); sys.exit(1)
+if fault != 'ignored-delete': v['tasks'] = [t for t in v['tasks'] if t['id'] != ident]
+if fault == 'unit-reactivation-on-delete': v['active'] = 'active'
+save(); print('true')
+'''
+
+
+class AgentTrueNASTeardown(unittest.TestCase):
+ source = SOURCE.read_text()
+
+ def exercise(self, fault="", reply="confirmed", bootstrap="regular", tasks="duplicate",
+ appliance=True, repeat=False, flow="caller"):
+ with tempfile.TemporaryDirectory() as temporary:
+ root = Path(temporary)
+ tools = root / "tools"; tools.mkdir()
+ for name in ("midclt", "ps", "systemctl"):
+ file = tools / name; file.write_text(MODEL); file.chmod(0o700)
+ if fault == "missing-midclt": (tools / "midclt").unlink()
+ for name in ("state", "helper", "profile", "unit"):
+ directory = root / name; directory.mkdir(mode=0o700)
+ if name != "unit":
+ for file in ("runtime.token", "agent-id", "connection.env", "install.sh"):
+ (directory / file).write_text("synthetic recovery only\n")
+ script = root / "state/bootstrap-pulse-agent.sh"
+ unrelated = root / "unrelated"; unrelated.write_text("do not alter\n")
+ if bootstrap == "linked": script.symlink_to(unrelated)
+ elif bootstrap == "directory": script.mkdir()
+ elif bootstrap != "missing":
+ script.write_text("#!/bin/bash\n# Pulse Agent Bootstrap for TrueNAS\n")
+ script.chmod(0o777 if bootstrap == "writable" else 0o755)
+ vendor = root / "truenas-version"
+ if appliance: vendor.write_text("TrueNAS-SCALE\n")
+ (root / "version").write_text("unrelated platform\n")
+ (root / "unit/pulse-agent.service").write_text("owned service\n")
+ owned = dict(id=1, type="SCRIPT", script=str(script), command="", when="POSTINIT",
+ enabled=True, timeout=30, comment="Pulse Agent Bootstrap")
+ foreign = dict(owned, id=99, script="/unrelated/boot.sh", comment="unrelated task")
+ selected = [] if tasks == "absent" else [owned]
+ if tasks == "duplicate": selected.append(dict(owned, id=2))
+ if tasks == "disabled": selected[0]["enabled"] = False
+ selected.append(foreign)
+ model = dict(fault=fault, tasks=selected, updates=0, deletes=0, queries=0,
+ active="active", enabled="enabled")
+ (root / "model.json").write_text(json.dumps(model))
+ before = self.snapshot(root)
+
+ def redirect(source):
+ for path, target in (
+ ("/etc/truenas-version", vendor), ("/etc/version", root / "version"),
+ ("/data/ix-applications", root / "no-ix"), ("/etc/ix-apps.d", root / "no-apps"),
+ ("/etc/ix.rc.d", root / "no-rc"), ("/usr/syno", root / "no-dsm"),
+ ("/etc/unraid-version", root / "no-unraid"),
+ ("/boot/config/plugins/pulse-agent", root / "no-plugin"),
+ ("/etc/config/qpkg.conf", root / "no-qnap"),
+ ("/sbin/getcfg", root / "no-getcfg"),
+ ("/var/log/pulse-agent.log", root / "absent-log"),
+ ("/etc/systemd/system/${AGENT_NAME}.service", root / "unit/pulse-agent.service"),
+ ): source = source.replace(path, str(target))
+ return source
+
+ values = dict(FIXTURE=root, EVENTS=root / "events", STATE_DIR=root / "state",
+ INSTALL_DIR=root / "bin",
+ STATE_DIR_REMOVAL_AUTHORITY=root / "state",
+ TRUENAS_STATE_DIR=root / "state", TRUENAS_BOOTSTRAP_SCRIPT=script,
+ PRIVILEGED_HELPER_STATE_DIR=root / "helper", SAFE_PROFILE_STATE_DIR=root / "profile",
+ PRIVILEGE_SUDOERS_FILE=root / "absent-sudoers", PRIVILEGE_HELPER_DIR=root / "absent-helper",
+ REPLY=reply, FAULT=fault)
+ setup = "\n".join(k + "=" + shlex.quote(str(v)) for k, v in values.items())
+ shell = r'''
+set -euo pipefail
+UNINSTALL=true AGENT_NAME=pulse-agent BINARY_NAME=pulse-agent INSTALL_DIR=/usr/local/bin
+EXIT_GENERAL=1 EXIT_MISSING_ARGS=16 PULSE_URL=https://same-instance.example.invalid AGENT_ID=synthetic
+ACTION_RUNNER_NAME=pulse-agent-runner PRIVILEGED_HELPER_NAME=pulse-agent-helper
+log_info() { :; }; log_warn() { :; }
+fail() { printf '%s\n' "$1" >&2; exit "$2"; }
+uname() { printf 'Linux\n'; }
+command() {
+ case "$*" in
+ '-v rc-service') return 1 ;;
+ '-v midclt') [[ "$FAULT" != missing-midclt ]] || return 1 ;;
+ '-v python3') [[ "$FAULT" != missing-python ]] || return 1 ;;
+ '-v timeout') [[ "$FAULT" != missing-timeout ]] || return 1 ;;
+ esac
+ builtin command "$@"
+}
+collector_credential_state_present() { [[ -e "$STATE_DIR/runtime.token" ]]; }
+uninstall_collector_registration() { printf 'remote:remove\n' >> "$EVENTS"; [[ "$REPLY" == confirmed ]]; }
+teardown_action_runner_service() { :; }
+teardown_privileged_helper_service() { :; }
+find_qnap_state_dir() { :; }
+pkill() { printf 'unchecked:process-stop\n' >> "$EVENTS"; }
+sleep() { :; }
+rm() {
+ local arg
+ for arg in "$@"; do [[ "$arg" == -* || "$arg" == "$FIXTURE/"* ]] || return 97; done
+ printf 'file:remove\n' >> "$EVENTS"
+ builtin command rm "$@"
+}
+remove_agent_state_dir() { printf 'state:remove\n' >> "$EVENTS"; rm -rf -- "$STATE_DIR"; }
+remove_privileged_helper_state_dir() { rm -rf -- "$PRIVILEGED_HELPER_STATE_DIR"; }
+remove_safe_profile_state_dir() { rm -rf -- "$SAFE_PROFILE_STATE_DIR"; }
+'''
+ entry = self.source.split("# --- Uninstall Logic ---\n", 1)[1].split("# --- Validation ---\n", 1)[0]
+ selected = definitions(self.source)
+ if flow == "process": entry = 'read_legacy_agent_removal_processes "$TRUENAS_BOOTSTRAP_SCRIPT"\n'
+ executable = shell + "\n" + setup + "\n" + redirect(selected) + "\nmain() {\n" + redirect(entry) + "}\nmain\n"
+ env = dict(os.environ, FIXTURE=str(root), PATH=str(tools) + ":" + os.environ["PATH"])
+ result = subprocess.run(["bash", "-c", executable], env=env, capture_output=True, text=True, timeout=12)
+ second = subprocess.run(["bash", "-c", executable], env=env, capture_output=True, text=True, timeout=12) if repeat else None
+ events = (root / "events").read_text() if (root / "events").exists() else ""
+ after = self.snapshot(root)
+ state = json.loads((root / "model.json").read_text())
+ timed_stopped = not (root / "timed-pid").exists() or not Path('/proc/' + (root / "timed-pid").read_text()).exists()
+ return result, second, events.splitlines(), before, after, state, timed_stopped
+
+ @staticmethod
+ def snapshot(root):
+ result = {}
+ for path in root.rglob("*"):
+ if path.parts[len(root.parts)] not in ("state", "helper", "profile", "unit", "unrelated"):
+ continue
+ mode = path.lstat().st_mode
+ contents = os.readlink(path) if stat.S_ISLNK(mode) else path.read_bytes() if stat.S_ISREG(mode) else None
+ result[str(path.relative_to(root))] = (mode, contents, path.lstat().st_ino)
+ return result
+
+ def retained(self, result):
+ execution, _, events, before, after, _, _ = result
+ self.assertEqual(execution.returncode, 1, execution.stdout + execution.stderr)
+ self.assertNotIn('state:remove', events)
+ for path in before:
+ if path.startswith(('state', 'helper', 'profile')) or path == 'unrelated': self.assertEqual(after.get(path), before[path], path)
+ self.assertNotIn('synthetic recovery only', execution.stdout + execution.stderr)
+ return events
+
+ def test_unavailable_or_untrusted_task_inventory_cannot_remove_recovery(self):
+ for fault in ('failed-query', 'empty-query', 'malformed-query', 'object-query', 'duplicate-field',
+ 'diagnostic-query', 'duplicate-id', 'bool-id', 'string-enabled', 'wrong-kind',
+ 'oversized-query', 'oversized-inventory'):
+ with self.subTest(fault=fault):
+ events = self.retained(self.exercise(fault))
+ self.assertNotIn('remote:remove', events)
+ self.assertFalse(any(e.startswith('task:disable:') or e.startswith('task:delete:') for e in events))
+
+ def test_foreign_or_indirect_bootstrap_is_not_mutated(self):
+ for fault in ('foreign-owner', 'command-bootstrap', 'other-bootstrap'):
+ with self.subTest(fault=fault):
+ events = self.retained(self.exercise(fault))
+ self.assertNotIn('remote:remove', events)
+ self.assertFalse(any(e.startswith('task:disable:') or e.startswith('task:delete:') for e in events))
+
+ def test_bootstrap_symlink_directory_or_writable_file_refuses_before_mutation(self):
+ for bootstrap in ('linked', 'directory', 'writable'):
+ with self.subTest(bootstrap=bootstrap):
+ events = self.retained(self.exercise(bootstrap=bootstrap))
+ self.assertEqual(events, [])
+
+ def test_missing_tools_are_not_local_only_removal_permission(self):
+ for fault in ('missing-midclt', 'missing-python', 'missing-timeout'):
+ with self.subTest(fault=fault):
+ events = self.retained(self.exercise(fault))
+ self.assertEqual(events, [])
+
+ def test_every_matching_task_is_disabled_and_independently_observed_before_stop(self):
+ result = self.exercise()
+ self.assertEqual(result[0].returncode, 0, result[0].stderr)
+ events = result[2]
+ self.assertLess(events.index('task:disable:2'), events.index('unit:stop'))
+ self.assertEqual(events[:4], ['task:query', 'task:disable:1', 'task:disable:2', 'task:query'])
+ self.assertLess(events.index('process:read'), events.index('remote:remove'))
+
+ def test_partial_refused_ignored_or_reactivated_disable_retains_recovery(self):
+ for fault in ('refused-disable', 'partial-disable', 'ignored-disable', 'reactivated-task'):
+ with self.subTest(fault=fault):
+ events = self.retained(self.exercise(fault))
+ self.assertNotIn('remote:remove', events)
+ self.assertNotIn('unit:stop', events)
+
+ def test_unit_or_process_uncertainty_keeps_stopped_task_recovery(self):
+ for fault in ('failed-unit-read', 'refused-stop', 'ignored-stop', 'failed-process', 'empty-process',
+ 'duplicate-process', 'diagnostic-process', 'orphan-agent', 'live-bootstrap'):
+ with self.subTest(fault=fault):
+ events = self.retained(self.exercise(fault))
+ self.assertNotIn('remote:remove', events)
+
+ def test_task_activation_during_systemd_stop_refuses_before_remote(self):
+ result = self.exercise('reactivated-task-on-stop')
+ events = self.retained(result)
+ self.assertIn('unit:stop', events)
+ self.assertNotIn('remote:remove', events)
+ self.assertFalse(any(e.startswith('task:delete:') for e in events))
+
+ def test_server_refusal_retains_disabled_tasks_and_all_recovery_without_restart(self):
+ result = self.exercise(reply='refused')
+ events = self.retained(result)
+ self.assertIn('remote:remove', events)
+ self.assertFalse(any(e.startswith('task:delete:') for e in events))
+ self.assertTrue(all(not t['enabled'] for t in result[5]['tasks'] if t['id'] != 99))
+
+ def test_changed_or_refused_post_remote_inventory_precedes_first_deletion(self):
+ for fault in ('failed-after-remote-query', 'new-task-after-remote'):
+ with self.subTest(fault=fault):
+ events = self.retained(self.exercise(fault))
+ self.assertIn('remote:remove', events)
+ self.assertFalse(any(e.startswith('task:delete:') for e in events))
+
+ def test_partial_refused_ignored_or_recreated_task_deletion_keeps_recovery(self):
+ for fault in ('refused-delete', 'partial-delete', 'ignored-delete', 'recreated-task', 'unit-reactivation-on-delete'):
+ with self.subTest(fault=fault): self.retained(self.exercise(fault))
+
+ def test_all_owned_tasks_retire_before_state_erasure_and_unrelated_task_survives(self):
+ result = self.exercise()
+ self.assertEqual(result[0].returncode, 0, result[0].stderr)
+ events = result[2]
+ self.assertLess(events.index('task:delete:2'), events.index('state:remove'))
+ self.assertIn('task:query', events[events.index('task:delete:2') + 1:events.index('state:remove')])
+ self.assertEqual([t['id'] for t in result[5]['tasks']], [99])
+ self.assertTrue(result[5]['tasks'][0]['enabled'])
+ self.assertNotIn('state/runtime.token', result[4])
+ self.assertEqual(result[4]['unrelated'], result[3]['unrelated'])
+
+ def test_disabled_absent_missing_bootstrap_and_repeated_removal_are_observed(self):
+ for tasks, bootstrap in (('disabled', 'regular'), ('absent', 'missing'), ('single', 'missing')):
+ with self.subTest(tasks=tasks, bootstrap=bootstrap):
+ result = self.exercise(tasks=tasks, bootstrap=bootstrap, repeat=True)
+ self.assertEqual(result[0].returncode, 0, result[0].stderr)
+ self.assertEqual(result[1].returncode, 0, result[1].stderr)
+ self.assertEqual([t['id'] for t in result[5]['tasks']], [99])
+
+ def test_ordinary_linux_state_is_not_appliance_evidence(self):
+ result = self.exercise(bootstrap='missing', tasks='absent', appliance=False)
+ self.assertEqual(result[0].returncode, 0, result[0].stderr)
+ self.assertFalse(any(e.startswith('task:') for e in result[2]))
+
+ def test_process_read_exposes_only_roles_and_real_timeout_child_is_stopped(self):
+ result = self.exercise(fault='live-bootstrap', flow='process')
+ self.assertEqual(result[0].returncode, 0, result[0].stderr)
+ self.assertIn('101 1 bootstrap', result[0].stdout)
+ self.assertNotIn('recovery-only', result[0].stdout + result[0].stderr)
+ result = self.exercise(fault='timeout-query')
+ self.retained(result)
+ self.assertTrue(result[6], 'timed middleware child still exists')
+
+
+if __name__ == '__main__':
+ unittest.main(verbosity=2)
diff --git a/scripts/installtests/agent_uninstall_state.py b/scripts/installtests/agent_uninstall_state.py
index 029e6a8fe0..725ac2b4c4 100644
--- a/scripts/installtests/agent_uninstall_state.py
+++ b/scripts/installtests/agent_uninstall_state.py
@@ -48,7 +48,7 @@ class UnixUninstallState(unittest.TestCase):
# Stop before unrelated platform boot/log cleanup; never run it on a host.
entry = source.split("# --- Uninstall Logic ---\n", 1)[1].split(
" # Remove least-privilege helper artifacts.", 1)[0] + "fi\n"
- names = ("collector_credential_state_present", "uninstall_collector_registration",
+ names = ("truenas_scale_agent_removal_present", "collector_credential_state_present", "uninstall_collector_registration",
"remove_authorized_runtime_dir", "remove_agent_state_dir",
"remove_privileged_helper_state_dir", "remove_safe_profile_state_dir", "fail")
with tempfile.TemporaryDirectory() as temporary:
diff --git a/scripts/installtests/agent_upstart_teardown.py b/scripts/installtests/agent_upstart_teardown.py
index 376a405435..3b4a6a5d4b 100644
--- a/scripts/installtests/agent_upstart_teardown.py
+++ b/scripts/installtests/agent_upstart_teardown.py
@@ -16,7 +16,7 @@ import unittest
ROOT = Path(__file__).resolve().parents[2]
SOURCE = Path(os.environ.get("PULSE_AGENT_UPSTART_SOURCE", ROOT / "scripts/install.sh"))
-FUNCTIONS = ("cleanup", "portable_path_uid", "portable_path_mode", "trusted_lifecycle_regular_file",
+FUNCTIONS = ("truenas_scale_agent_removal_present", "cleanup", "portable_path_uid", "portable_path_mode", "trusted_lifecycle_regular_file",
"read_legacy_agent_removal_processes", "legacy_agent_processes_absent",
"read_upstart_agent_removal_status", "upstart_agent_removal_paths_safe",
"read_upstart_agent_removal_enablement", "quiesce_upstart_agent_for_removal",
diff --git a/scripts/installtests/install_sh_test.go b/scripts/installtests/install_sh_test.go
index 490b04be73..928e005506 100644
--- a/scripts/installtests/install_sh_test.go
+++ b/scripts/installtests/install_sh_test.go
@@ -8501,3 +8501,41 @@ func TestInstallSHUpstartTeardownRequiresObservedQuiescence(t *testing.T) {
}
t.Logf("%s", output)
}
+
+func TestInstallSHTrueNASScaleBootTasksRetireBeforeRecoveryErasure(t *testing.T) {
+ if runtime.GOOS == "windows" {
+ t.Skip("TrueNAS SCALE shell controls; Windows has its own native SCM gate")
+ }
+ ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
+ defer cancel()
+ cmd := exec.CommandContext(ctx, "python3", repoFile("scripts", "installtests", "agent_truenas_teardown.py"))
+ output, err := cmd.CombinedOutput()
+ if err != nil {
+ t.Fatalf("TrueNAS SCALE boot-task ownership/retirement controls: %v\n%s", err, output)
+ }
+ for _, name := range []string{
+ "test_unavailable_or_untrusted_task_inventory_cannot_remove_recovery",
+ "test_foreign_or_indirect_bootstrap_is_not_mutated",
+ "test_bootstrap_symlink_directory_or_writable_file_refuses_before_mutation",
+ "test_missing_tools_are_not_local_only_removal_permission",
+ "test_every_matching_task_is_disabled_and_independently_observed_before_stop",
+ "test_partial_refused_ignored_or_reactivated_disable_retains_recovery",
+ "test_unit_or_process_uncertainty_keeps_stopped_task_recovery",
+ "test_task_activation_during_systemd_stop_refuses_before_remote",
+ "test_server_refusal_retains_disabled_tasks_and_all_recovery_without_restart",
+ "test_changed_or_refused_post_remote_inventory_precedes_first_deletion",
+ "test_partial_refused_ignored_or_recreated_task_deletion_keeps_recovery",
+ "test_all_owned_tasks_retire_before_state_erasure_and_unrelated_task_survives",
+ "test_disabled_absent_missing_bootstrap_and_repeated_removal_are_observed",
+ "test_ordinary_linux_state_is_not_appliance_evidence",
+ "test_process_read_exposes_only_roles_and_real_timeout_child_is_stopped",
+ } {
+ if !strings.Contains(string(output), name+" (__main__.AgentTrueNASTeardown."+name+") ... ok") {
+ t.Fatalf("TrueNAS SCALE control %s did not pass:\n%s", name, output)
+ }
+ }
+ if !strings.Contains(string(output), "\nRan 15 tests in ") || !strings.Contains(string(output), "\nOK\n") {
+ t.Fatalf("incomplete TrueNAS SCALE verdict:\n%s", output)
+ }
+ t.Logf("%s", output)
+}