Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Explain unavailable backup-age evaluation without weakening protection
pulse-triage[bot]c8e0f0deb2cf
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.mdindex e5daeb1be2..4c777f9b3a 100644--- a/docs/TROUBLESHOOTING.md+++ b/docs/TROUBLESHOOTING.md@@ -381,6 +381,48 @@ planned maintenance window, preserving the deployment's settings and data.pre-pause states are not permission to start them. **Pulse monitoring andalerts are unavailable while stopped**; arrange independent outage coverage.+#### Backup-age alerts were not evaluated++The **Pulse** warning “Backup-age alerts were not evaluated because recovery+data could not be read” means that Pulse could not complete that backup-age+check. It is a monitoring failure, **not evidence that a backup failed**.+Existing backup-age alerts are kept during the failed read; they are not a+fresh assessment, and the absence of a new age alert does not establish that+backups are current. A working connection or a visible backup row does not+prove that age evaluation completed.++Use the owning Proxmox/PBS installation's existing backup records and your+established backup checks while evaluation is unavailable. Compare the same+installation, guest type/ID, datastore, namespace and backup time; a matching+VMID in another installation is not the same guest. For a separate Coverage+disagreement, use the [backup health checks](PBS.md#backups-are-visible-but-coverage-says-unprotected).++Keep the warning's original time and message. On the actual Pulse server,+use the existing [bounded log reader](#inspect-notification-logs) for the+incident window, with the matching systemd or Docker deployment. The log+message **Failed to list recovery rollups for backup alerts** can identify a+failed evaluation read; a timeout does not by itself establish its cause.+A failed log read is unavailable, not an empty successful search. Share only+the running Pulse version, affected time window, warning and relevant manually+redacted error. Keep credentials, private paths/hostnames, full logs, API+responses and database files out of public reports.++**A disappearing warning alone is not recovery.** Disabling alerts or backup-age+checks, or setting both age thresholds to zero, can clear it without a+successful evaluation. Keep the intended policy unchanged and observe ordinary+polling; do not disable protection just to clear the warning. A successful age+evaluation still does not establish backup completeness, verification, restore+readiness or guest thaw.++Do not restart or reinstall Pulse, clear History, delete/prune backup records,+change database schema or increase timeouts just to diagnose this warning. Do+not run a new backup, verification, restore, live diagnostics or guest-agent+probe as a test. An OK backup task does not prove guest thaw; a frozen or+unresponsive guest needs the separate [backup safety procedure](VM_DISK_MONITORING.md#backup-safety),+including independent post-backup thaw, successful writes to every covered+filesystem and workload liveness before restoring only previously active+monitoring.+#### Backup health disagrees with PBSA visible or Verified PBS backup is not the same reading as a workload'sdiff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.mdindex e5daeb1be2..4c777f9b3a 100644--- a/frontend-modern/public/docs/TROUBLESHOOTING.md+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md@@ -381,6 +381,48 @@ planned maintenance window, preserving the deployment's settings and data.pre-pause states are not permission to start them. **Pulse monitoring andalerts are unavailable while stopped**; arrange independent outage coverage.+#### Backup-age alerts were not evaluated++The **Pulse** warning “Backup-age alerts were not evaluated because recovery+data could not be read” means that Pulse could not complete that backup-age+check. It is a monitoring failure, **not evidence that a backup failed**.+Existing backup-age alerts are kept during the failed read; they are not a+fresh assessment, and the absence of a new age alert does not establish that+backups are current. A working connection or a visible backup row does not+prove that age evaluation completed.++Use the owning Proxmox/PBS installation's existing backup records and your+established backup checks while evaluation is unavailable. Compare the same+installation, guest type/ID, datastore, namespace and backup time; a matching+VMID in another installation is not the same guest. For a separate Coverage+disagreement, use the [backup health checks](PBS.md#backups-are-visible-but-coverage-says-unprotected).++Keep the warning's original time and message. On the actual Pulse server,+use the existing [bounded log reader](#inspect-notification-logs) for the+incident window, with the matching systemd or Docker deployment. The log+message **Failed to list recovery rollups for backup alerts** can identify a+failed evaluation read; a timeout does not by itself establish its cause.+A failed log read is unavailable, not an empty successful search. Share only+the running Pulse version, affected time window, warning and relevant manually+redacted error. Keep credentials, private paths/hostnames, full logs, API+responses and database files out of public reports.++**A disappearing warning alone is not recovery.** Disabling alerts or backup-age+checks, or setting both age thresholds to zero, can clear it without a+successful evaluation. Keep the intended policy unchanged and observe ordinary+polling; do not disable protection just to clear the warning. A successful age+evaluation still does not establish backup completeness, verification, restore+readiness or guest thaw.++Do not restart or reinstall Pulse, clear History, delete/prune backup records,+change database schema or increase timeouts just to diagnose this warning. Do+not run a new backup, verification, restore, live diagnostics or guest-agent+probe as a test. An OK backup task does not prove guest thaw; a frozen or+unresponsive guest needs the separate [backup safety procedure](VM_DISK_MONITORING.md#backup-safety),+including independent post-backup thaw, successful writes to every covered+filesystem and workload liveness before restoring only previously active+monitoring.+#### Backup health disagrees with PBSA visible or Verified PBS backup is not the same reading as a workload'sdiff --git a/scripts/tests/test_pve_backup_troubleshooting_docs.py b/scripts/tests/test_pve_backup_troubleshooting_docs.pyindex e14bfa233a..05b3090552 100644--- a/scripts/tests/test_pve_backup_troubleshooting_docs.py+++ b/scripts/tests/test_pve_backup_troubleshooting_docs.py@@ -23,6 +23,63 @@ def backup_section():return text.split(HEADING, 1)[1].split("\n### ", 1)[0]+class BackupAgeEvaluationDocsTest(unittest.TestCase):+ """Check existing warning guidance, not database or native recovery."""++ guide = ROOT / "docs/TROUBLESHOOTING.md"+ mirror = ROOT / "frontend-modern/public/docs/TROUBLESHOOTING.md"+ heading = "#### Backup-age alerts were not evaluated\n"++ def section(self):+ text = self.guide.read_text()+ self.assertEqual(text.count(self.heading), 1)+ return " ".join(text.split(self.heading, 1)[1].split("\n#### ", 1)[0].split())++ def test_warning_distinguishes_unavailable_evaluation_from_backup_failure(self):+ text = self.section()+ source = (ROOT / "internal/monitoring/recovery_rollups.go").read_text()+ for phrase in ("Backup-age alerts were not evaluated because recovery data could not be read",+ "Failed to list recovery rollups for backup alerts"):+ self.assertIn(phrase, text)+ self.assertIn(phrase, source)+ for distinction in ("not evidence that a backup failed", "not a fresh assessment",+ "absence of a new age alert does not establish",+ "visible backup row does not prove that age evaluation completed",+ "installation, guest type/ID, datastore, namespace and backup time"):+ self.assertIn(distinction, text)++ def test_policy_changes_and_warning_clearance_do_not_prove_recovery(self):+ text = self.section()+ for distinction in ("A disappearing warning alone is not recovery",+ "Disabling alerts or backup-age checks",+ "setting both age thresholds to zero",+ "without a successful evaluation", "intended policy unchanged",+ "ordinary polling", "restore readiness or guest thaw"):+ self.assertIn(distinction, text)+ source = (ROOT / "internal/monitoring/recovery_rollups.go").read_text()+ for branch in ("!cfg.Enabled || !cfg.BackupDefaults.Enabled",+ "cfg.BackupDefaults.WarningDays <= 0 && cfg.BackupDefaults.CriticalDays <= 0",+ "ClearSystemAlert(alerts.BackupEvaluationAlertType)"):+ self.assertIn(branch, source)++ def test_help_reuses_safe_logs_and_preserves_workloads_and_evidence(self):+ text = self.section()+ for boundary in ("[bounded log reader](#inspect-notification-logs)",+ "matching systemd or Docker deployment", "original time and message",+ "timeout does not by itself establish its cause",+ "failed log read is unavailable", "manually redacted error",+ "database files out of public reports", "Do not restart or reinstall Pulse",+ "change database schema", "Do not run a new backup",+ "live diagnostics or guest-agent probe", "VM_DISK_MONITORING.md#backup-safety",+ "successful writes to every covered filesystem", "workload liveness",+ "only previously active monitoring"):+ self.assertIn(boundary, text)+ self.assertNotIn("```", text, "reuse the contained reader, not a new copied command")++ def test_warning_help_is_in_the_shipped_mirror(self):+ self.assertEqual(self.guide.read_bytes(), self.mirror.read_bytes())++class PVEBackupTroubleshootingDocsTest(unittest.TestCase):def test_replication_help_uses_existing_row_details_not_backup_commands(self):trouble = (ROOT / "docs/TROUBLESHOOTING.md").read_text()@@ -200,10 +257,16 @@ PY"no automatic rollback", "recorded active before maintenance"):self.assertIn(phrase, text)helper = (ROOT / "scripts/uninstall-sensor-proxy.sh").read_text()- for fact in ('timeout 30 pct stop "$ctid"', 'timeout 30 pct start "$ctid"',- 'cleanup_sensor_proxy_lines_in_conf "$conf" "$snapshot_line"',- 'systemctl "$@" >/dev/null 2>&1 || true'):+ for fact in ('timeout --kill-after=1s 30 pct stop "$ctid"',+ 'timeout --kill-after=1s 30 pct start "$ctid"',+ 'if ! cleanup_sensor_proxy_lines_in_conf "$conf"; then',+ 'if ! disable_legacy_units; then',+ 'if (( container_cleanup_status != 0 )); then'):self.assertIn(fact, helper)+ # The current helper now returns failures rather than hiding them.+ # Keep the guide's stop-on-unknown boundary tied to that repair, not+ # the superseded permissive systemctl adapter.+ self.assertNotIn('systemctl "$@" >/dev/null 2>&1 || true', helper)def test_preserves_credentials_and_explains_remote_cleanup_scope(self):text = " ".join(self.section().split())