Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep legacy sensor cleanup local and preserve recovery evidence
pulse-triage[bot]953e0c6decf7
diff --git a/docs/TEMPERATURE_MONITORING.md b/docs/TEMPERATURE_MONITORING.mdindex 624fe83fad..f491ba5621 100644--- a/docs/TEMPERATURE_MONITORING.md+++ b/docs/TEMPERATURE_MONITORING.md@@ -183,9 +183,36 @@ the exit, relevant redacted error and affected sensor if help is needed.## Legacy Cleanup (If Upgrading)-If you still have the old sensor proxy installed from prior releases, remove it-from each **Proxmox host** (not the Pulse container) with the supported cleanup-helper. In that host's trusted administrative shell, download it privately.+The retired `pulse-sensor-proxy` is separate from the unified agent and the+Pulse server. A server upgrade does not remove it from a Proxmox host. Do not+run cleanup merely because a temperature is missing, or reinstall/re-add a+working connection to retire it.++**This helper is disruptive maintenance, not a diagnostic or a dry run.** It+stops legacy proxy units, removes their installed files and marked legacy SSH+keys, and edits LXC configurations containing `pulse-sensor-proxy`. It can+**stop and start running LXCs**, including the Pulse container. `--local-only`+prevents cluster SSH; it does not restrict cleanup to one container. Removing+a marked SSH key can also interrupt a legacy temperature-collection path.++Before executing anything, an authorised Proxmox administrator must:++1. Inspect the saved helper and the existing host configuration privately.+ Identify every affected LXC, the proxy units and marked SSH keys, and record+ which services and containers were active. Do not assume a familiar CT name+ or an upgraded Pulse server establishes the cleanup scope.+2. Keep private backups of the affected LXC configurations, authorised-key file,+ proxy installation, configuration and logs. These are recovery material, not+ an issue attachment; do not share tokens, SSH keys or complete host files.+3. Schedule an outage for all affected workloads, not just Pulse. Finish any+ backup, restore or migration first. If an operation's state or the affected+ container set is unknown, stop and resolve it before cleanup. Deliberately+ shut down the affected LXCs through their normal maintenance procedure before+ running the helper; do not rely on its forced stop/start path. Arrange+ independent monitoring while Pulse or its old collection path is unavailable.++Download preparation alone makes no host change. In the Proxmox host's trusted+administrative shell (not inside the Pulse container), save the helper privately.This refuses an existing target, symlinked configuration directory orfailed/partial download; it does not overwrite an earlier helper or run aresponse as a command:@@ -223,26 +250,40 @@ Curl's local configuration is ignored, redirects are not followed and TLSverification stays enabled. This is an HTTPS download from the repository'scurrent main, not a signed release asset. Inspect the complete saved scriptand choose the cleanup scope before running it.-Do not pipe a web response into a privileged shell:+Do not pipe a web response into a privileged shell. Only after completing the+maintenance steps above, run the inspected helper in the authorised host shell:```bash-bash "$HOME/.config/pulse/sensor-proxy-uninstall.sh" \- --uninstall --purge --local-only+bash "$HOME/.config/pulse/sensor-proxy-uninstall.sh" --uninstall --local-only```-`--local-only` avoids cluster SSH entirely; run the command once on every-Proxmox node that carried the proxy. For one cluster-wide pass instead, omit-`--local-only`. Remote nodes must already have trusted host keys in root's-normal OpenSSH user/system known_hosts files, or supply a separately-provisioned file with `--ssh-known-hosts /path/to/known_hosts`. The helper never-accepts or enrolls an unknown host key, and a missing or changed key makes the-remote portion fail after local cleanup completes.--If you also want to remove the old `pulse-monitor@pam` API user and tokens before re-adding the node, include `--remove-proxmox-access`:--```bash-bash "$HOME/.config/pulse/sensor-proxy-uninstall.sh" \- --uninstall --purge --remove-proxmox-access --local-only-```--Reinstalling or upgrading the Pulse container does **not** remove the sensor proxy from the host — they are separate installations. If you skip this cleanup, the selfheal timer will keep running and may generate recurring `TASK ERROR` entries in the Proxmox task log.+This deliberately omits `--purge` and `--remove-proxmox-access`. The default+preserves persisted proxy state, configuration, logs and its service account;+it still removes installed proxy files and marked SSH keys. Purging would+remove recovery evidence, and removing Proxmox access would delete the+`pulse-monitor@pam` API user and its tokens, which a current connection may+still use. Neither is required to stop the retired proxy. Preserve current+monitoring credentials, connections and history; do not re-add nodes or rerun+setup as part of cleanup.++Use `--local-only` separately on each host confirmed to carry the proxy, during+that host's planned maintenance. Omitting it can edit SSH keys on other cluster+nodes; that is **not a cluster-wide uninstall** of their proxy services or+files. Do not broaden the operation or enrol SSH host keys to make cleanup work.++**Verify the actual result before resuming workloads.** The helper suppresses+some service and container-operation errors; exit zero or its completion message+is not proof that cleanup succeeded. Check locally that the legacy proxy and+selfheal units are inactive, the affected live LXC configurations no longer+reference the proxy, and unrelated configuration and SSH access remain intact.+Keep the original output private and stop on a warning, failed check or unknown+state rather than repeating cleanup or adding destructive flags.++Restore only the containers and services recorded active before maintenance,+using their normal deployment procedure, then verify workload liveness and+ordinary Pulse/agent monitoring. A restored temperature alone does not prove+all workloads or monitoring recovered. The helper has no automatic rollback:+if restoration fails, retain the backups and use the administrator's targeted+recovery procedure; do not blindly overwrite cluster configuration or today's+data. Keep preserved proxy evidence until recovery and its normal retention+policy permit disposal.diff --git a/frontend-modern/public/docs/TEMPERATURE_MONITORING.md b/frontend-modern/public/docs/TEMPERATURE_MONITORING.mdindex 624fe83fad..f491ba5621 100644--- a/frontend-modern/public/docs/TEMPERATURE_MONITORING.md+++ b/frontend-modern/public/docs/TEMPERATURE_MONITORING.md@@ -183,9 +183,36 @@ the exit, relevant redacted error and affected sensor if help is needed.## Legacy Cleanup (If Upgrading)-If you still have the old sensor proxy installed from prior releases, remove it-from each **Proxmox host** (not the Pulse container) with the supported cleanup-helper. In that host's trusted administrative shell, download it privately.+The retired `pulse-sensor-proxy` is separate from the unified agent and the+Pulse server. A server upgrade does not remove it from a Proxmox host. Do not+run cleanup merely because a temperature is missing, or reinstall/re-add a+working connection to retire it.++**This helper is disruptive maintenance, not a diagnostic or a dry run.** It+stops legacy proxy units, removes their installed files and marked legacy SSH+keys, and edits LXC configurations containing `pulse-sensor-proxy`. It can+**stop and start running LXCs**, including the Pulse container. `--local-only`+prevents cluster SSH; it does not restrict cleanup to one container. Removing+a marked SSH key can also interrupt a legacy temperature-collection path.++Before executing anything, an authorised Proxmox administrator must:++1. Inspect the saved helper and the existing host configuration privately.+ Identify every affected LXC, the proxy units and marked SSH keys, and record+ which services and containers were active. Do not assume a familiar CT name+ or an upgraded Pulse server establishes the cleanup scope.+2. Keep private backups of the affected LXC configurations, authorised-key file,+ proxy installation, configuration and logs. These are recovery material, not+ an issue attachment; do not share tokens, SSH keys or complete host files.+3. Schedule an outage for all affected workloads, not just Pulse. Finish any+ backup, restore or migration first. If an operation's state or the affected+ container set is unknown, stop and resolve it before cleanup. Deliberately+ shut down the affected LXCs through their normal maintenance procedure before+ running the helper; do not rely on its forced stop/start path. Arrange+ independent monitoring while Pulse or its old collection path is unavailable.++Download preparation alone makes no host change. In the Proxmox host's trusted+administrative shell (not inside the Pulse container), save the helper privately.This refuses an existing target, symlinked configuration directory orfailed/partial download; it does not overwrite an earlier helper or run aresponse as a command:@@ -223,26 +250,40 @@ Curl's local configuration is ignored, redirects are not followed and TLSverification stays enabled. This is an HTTPS download from the repository'scurrent main, not a signed release asset. Inspect the complete saved scriptand choose the cleanup scope before running it.-Do not pipe a web response into a privileged shell:+Do not pipe a web response into a privileged shell. Only after completing the+maintenance steps above, run the inspected helper in the authorised host shell:```bash-bash "$HOME/.config/pulse/sensor-proxy-uninstall.sh" \- --uninstall --purge --local-only+bash "$HOME/.config/pulse/sensor-proxy-uninstall.sh" --uninstall --local-only```-`--local-only` avoids cluster SSH entirely; run the command once on every-Proxmox node that carried the proxy. For one cluster-wide pass instead, omit-`--local-only`. Remote nodes must already have trusted host keys in root's-normal OpenSSH user/system known_hosts files, or supply a separately-provisioned file with `--ssh-known-hosts /path/to/known_hosts`. The helper never-accepts or enrolls an unknown host key, and a missing or changed key makes the-remote portion fail after local cleanup completes.--If you also want to remove the old `pulse-monitor@pam` API user and tokens before re-adding the node, include `--remove-proxmox-access`:--```bash-bash "$HOME/.config/pulse/sensor-proxy-uninstall.sh" \- --uninstall --purge --remove-proxmox-access --local-only-```--Reinstalling or upgrading the Pulse container does **not** remove the sensor proxy from the host — they are separate installations. If you skip this cleanup, the selfheal timer will keep running and may generate recurring `TASK ERROR` entries in the Proxmox task log.+This deliberately omits `--purge` and `--remove-proxmox-access`. The default+preserves persisted proxy state, configuration, logs and its service account;+it still removes installed proxy files and marked SSH keys. Purging would+remove recovery evidence, and removing Proxmox access would delete the+`pulse-monitor@pam` API user and its tokens, which a current connection may+still use. Neither is required to stop the retired proxy. Preserve current+monitoring credentials, connections and history; do not re-add nodes or rerun+setup as part of cleanup.++Use `--local-only` separately on each host confirmed to carry the proxy, during+that host's planned maintenance. Omitting it can edit SSH keys on other cluster+nodes; that is **not a cluster-wide uninstall** of their proxy services or+files. Do not broaden the operation or enrol SSH host keys to make cleanup work.++**Verify the actual result before resuming workloads.** The helper suppresses+some service and container-operation errors; exit zero or its completion message+is not proof that cleanup succeeded. Check locally that the legacy proxy and+selfheal units are inactive, the affected live LXC configurations no longer+reference the proxy, and unrelated configuration and SSH access remain intact.+Keep the original output private and stop on a warning, failed check or unknown+state rather than repeating cleanup or adding destructive flags.++Restore only the containers and services recorded active before maintenance,+using their normal deployment procedure, then verify workload liveness and+ordinary Pulse/agent monitoring. A restored temperature alone does not prove+all workloads or monitoring recovered. The helper has no automatic rollback:+if restoration fails, retain the backups and use the administrator's targeted+recovery procedure; do not blindly overwrite cluster configuration or today's+data. Keep preserved proxy evidence until recovery and its normal retention+policy permit disposal.diff --git a/scripts/tests/test_agent_docs.py b/scripts/tests/test_agent_docs.pyindex 0ad67e2d41..db95a746f1 100644--- a/scripts/tests/test_agent_docs.py+++ b/scripts/tests/test_agent_docs.py@@ -81,7 +81,7 @@ class AgentDocsTest(unittest.TestCase):with self.subTest(name=name, command=block):subprocess.run(["bash", "-n", "-c", block], check=True, capture_output=True)count += 1- self.assertEqual(count, 18)+ self.assertEqual(count, 17)def test_private_file_preparation_preserves_existing_token(self):command = recipe(NAMES[0], 'credential_file="$config_dir/agent-token"')diff --git a/scripts/tests/test_pve_backup_troubleshooting_docs.py b/scripts/tests/test_pve_backup_troubleshooting_docs.pyindex 7359a6f7e3..e14bfa233a 100644--- a/scripts/tests/test_pve_backup_troubleshooting_docs.py+++ b/scripts/tests/test_pve_backup_troubleshooting_docs.py@@ -5,7 +5,11 @@ No provider calls, setup commands, backups or guest-agent probes are executed."""from pathlib import Path+import json+import osimport re+import subprocess+import tempfileimport unittest@@ -140,6 +144,83 @@ class PVEBackupTroubleshootingDocsTest(unittest.TestCase):self.assertIn("Pulse monitoring and alerts are unavailable", safety)+class LegacySensorCleanupDocsTest(unittest.TestCase):+ """Exercise the copied invocation only, never the real host cleanup helper."""++ def section(self):+ return (ROOT / "docs/TEMPERATURE_MONITORING.md").read_text().split(+ "## Legacy Cleanup (If Upgrading)\n", 1+ )[1]++ def invocation(self):+ commands = re.findall(r"```bash\n(.*?)```", self.section(), re.DOTALL)+ invocations = [c for c in commands if c.lstrip().startswith('bash ')]+ self.assertEqual(len(invocations), 1, "one deliberate local cleanup only")+ return invocations[0]++ def test_copied_cleanup_is_one_local_preserving_action(self):+ command = self.invocation()+ self.assertNotRegex(command, r"--purge|--remove-proxmox-access|--ssh-known-hosts|\|\||;")+ with tempfile.TemporaryDirectory() as temporary:+ home = Path(temporary)+ private = home / ".config/pulse"+ private.mkdir(parents=True)+ # Only this synthetic helper executes. No pct/systemctl/SSH or+ # credential operation is available through it.+ helper = private / "sensor-proxy-uninstall.sh"+ helper.write_text('''python3 - "$@" <<'PY'+import json, os, sys+from pathlib import Path+Path(os.environ['CALLS']).write_text(json.dumps(sys.argv[1:]))+sys.exit(int(os.environ['HELPER_EXIT']))+PY+''')+ witness = home / "preserved-config-and-log"+ witness.write_text("unchanged synthetic recovery evidence\n")+ calls = home / "calls.json"+ for status in (0, 29):+ with self.subTest(status=status):+ calls.unlink(missing_ok=True)+ result = subprocess.run(+ ["bash", "-c", command], text=True, capture_output=True, timeout=10,+ env=dict(os.environ, HOME=str(home), CALLS=str(calls), HELPER_EXIT=str(status)),+ )+ self.assertEqual(result.returncode, status, result.stdout + result.stderr)+ self.assertEqual(json.loads(calls.read_text()), ["--uninstall", "--local-only"])+ self.assertEqual(witness.read_text(), "unchanged synthetic recovery evidence\n")++ def test_disruption_and_failed_operations_are_not_hidden_by_local_only(self):+ text = " ".join(self.section().split())+ for phrase in ("disruptive maintenance, not a diagnostic or a dry run",+ "stop and start running LXCs", "including the Pulse container",+ "does not restrict cleanup to one container", "every affected LXC",+ "backup, restore or migration", "shut down the affected LXCs",+ "independent monitoring", "private backups", "marked SSH keys",+ "exit zero or its completion message is not proof",+ "no automatic rollback", "recorded active before maintenance"):+ self.assertIn(phrase, text)+ helper = (ROOT / "scripts/uninstall-sensor-proxy.sh").read_text()+ for fact in ('timeout 30 pct stop "$ctid"', 'timeout 30 pct start "$ctid"',+ 'cleanup_sensor_proxy_lines_in_conf "$conf" "$snapshot_line"',+ 'systemctl "$@" >/dev/null 2>&1 || true'):+ self.assertIn(fact, helper)++ def test_preserves_credentials_and_explains_remote_cleanup_scope(self):+ text = " ".join(self.section().split())+ for phrase in ("preserves persisted proxy state, configuration, logs and its service account",+ "current connection may still use", "Neither is required to stop",+ "not re-add nodes or rerun setup", "not a cluster-wide uninstall",+ "stop on a warning, failed check or unknown state",+ "unrelated configuration and SSH access remain intact"):+ self.assertIn(phrase, text)+ helper = (ROOT / "scripts/uninstall-sensor-proxy.sh").read_text()+ for fact in ("PURGE=false", "REMOVE_PROXMOX_ACCESS=false", 'if [[ "$PURGE" == "true" ]]',+ 'if [[ "$REMOVE_PROXMOX_ACCESS" != "true" ]]'):+ self.assertIn(fact, helper)+ self.assertEqual((ROOT / "docs/TEMPERATURE_MONITORING.md").read_bytes(),+ (ROOT / "frontend-modern/public/docs/TEMPERATURE_MONITORING.md").read_bytes())++ZFS_DOC = ROOT / "docs/ZFS_MONITORING.md"