Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Explain unavailable backup-age evaluation without weakening protection

pulse-triage[bot]c8e0f0deb2cf
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md
index e5daeb1be2..4c777f9b3a 100644
--- a/docs/TROUBLESHOOTING.md
+++ b/docs/TROUBLESHOOTING.md
@@ -381,6 +381,48 @@ planned maintenance window, preserving the deployment's settings and data.
pre-pause states are not permission to start them. **Pulse monitoring and
alerts are unavailable while stopped**; arrange independent outage coverage.
+#### Backup-age alerts were not evaluated
+
+The **Pulse** warning “Backup-age alerts were not evaluated because recovery
+data could not be read” means that Pulse could not complete that backup-age
+check. It is a monitoring failure, **not evidence that a backup failed**.
+Existing backup-age alerts are kept during the failed read; they are not a
+fresh assessment, and the absence of a new age alert does not establish that
+backups are current. A working connection or a visible backup row does not
+prove that age evaluation completed.
+
+Use the owning Proxmox/PBS installation's existing backup records and your
+established backup checks while evaluation is unavailable. Compare the same
+installation, guest type/ID, datastore, namespace and backup time; a matching
+VMID in another installation is not the same guest. For a separate Coverage
+disagreement, use the [backup health checks](PBS.md#backups-are-visible-but-coverage-says-unprotected).
+
+Keep the warning's original time and message. On the actual Pulse server,
+use the existing [bounded log reader](#inspect-notification-logs) for the
+incident window, with the matching systemd or Docker deployment. The log
+message **Failed to list recovery rollups for backup alerts** can identify a
+failed evaluation read; a timeout does not by itself establish its cause.
+A failed log read is unavailable, not an empty successful search. Share only
+the running Pulse version, affected time window, warning and relevant manually
+redacted error. Keep credentials, private paths/hostnames, full logs, API
+responses and database files out of public reports.
+
+**A disappearing warning alone is not recovery.** Disabling alerts or backup-age
+checks, or setting both age thresholds to zero, can clear it without a
+successful evaluation. Keep the intended policy unchanged and observe ordinary
+polling; do not disable protection just to clear the warning. A successful age
+evaluation still does not establish backup completeness, verification, restore
+readiness or guest thaw.
+
+Do not restart or reinstall Pulse, clear History, delete/prune backup records,
+change database schema or increase timeouts just to diagnose this warning. Do
+not run a new backup, verification, restore, live diagnostics or guest-agent
+probe as a test. An OK backup task does not prove guest thaw; a frozen or
+unresponsive guest needs the separate [backup safety procedure](VM_DISK_MONITORING.md#backup-safety),
+including independent post-backup thaw, successful writes to every covered
+filesystem and workload liveness before restoring only previously active
+monitoring.
+
#### Backup health disagrees with PBS
A visible or Verified PBS backup is not the same reading as a workload's
diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.md
index e5daeb1be2..4c777f9b3a 100644
--- a/frontend-modern/public/docs/TROUBLESHOOTING.md
+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md
@@ -381,6 +381,48 @@ planned maintenance window, preserving the deployment's settings and data.
pre-pause states are not permission to start them. **Pulse monitoring and
alerts are unavailable while stopped**; arrange independent outage coverage.
+#### Backup-age alerts were not evaluated
+
+The **Pulse** warning “Backup-age alerts were not evaluated because recovery
+data could not be read” means that Pulse could not complete that backup-age
+check. It is a monitoring failure, **not evidence that a backup failed**.
+Existing backup-age alerts are kept during the failed read; they are not a
+fresh assessment, and the absence of a new age alert does not establish that
+backups are current. A working connection or a visible backup row does not
+prove that age evaluation completed.
+
+Use the owning Proxmox/PBS installation's existing backup records and your
+established backup checks while evaluation is unavailable. Compare the same
+installation, guest type/ID, datastore, namespace and backup time; a matching
+VMID in another installation is not the same guest. For a separate Coverage
+disagreement, use the [backup health checks](PBS.md#backups-are-visible-but-coverage-says-unprotected).
+
+Keep the warning's original time and message. On the actual Pulse server,
+use the existing [bounded log reader](#inspect-notification-logs) for the
+incident window, with the matching systemd or Docker deployment. The log
+message **Failed to list recovery rollups for backup alerts** can identify a
+failed evaluation read; a timeout does not by itself establish its cause.
+A failed log read is unavailable, not an empty successful search. Share only
+the running Pulse version, affected time window, warning and relevant manually
+redacted error. Keep credentials, private paths/hostnames, full logs, API
+responses and database files out of public reports.
+
+**A disappearing warning alone is not recovery.** Disabling alerts or backup-age
+checks, or setting both age thresholds to zero, can clear it without a
+successful evaluation. Keep the intended policy unchanged and observe ordinary
+polling; do not disable protection just to clear the warning. A successful age
+evaluation still does not establish backup completeness, verification, restore
+readiness or guest thaw.
+
+Do not restart or reinstall Pulse, clear History, delete/prune backup records,
+change database schema or increase timeouts just to diagnose this warning. Do
+not run a new backup, verification, restore, live diagnostics or guest-agent
+probe as a test. An OK backup task does not prove guest thaw; a frozen or
+unresponsive guest needs the separate [backup safety procedure](VM_DISK_MONITORING.md#backup-safety),
+including independent post-backup thaw, successful writes to every covered
+filesystem and workload liveness before restoring only previously active
+monitoring.
+
#### Backup health disagrees with PBS
A visible or Verified PBS backup is not the same reading as a workload's
diff --git a/scripts/tests/test_pve_backup_troubleshooting_docs.py b/scripts/tests/test_pve_backup_troubleshooting_docs.py
index e14bfa233a..05b3090552 100644
--- a/scripts/tests/test_pve_backup_troubleshooting_docs.py
+++ b/scripts/tests/test_pve_backup_troubleshooting_docs.py
@@ -23,6 +23,63 @@ def backup_section():
return text.split(HEADING, 1)[1].split("\n### ", 1)[0]
+class BackupAgeEvaluationDocsTest(unittest.TestCase):
+ """Check existing warning guidance, not database or native recovery."""
+
+ guide = ROOT / "docs/TROUBLESHOOTING.md"
+ mirror = ROOT / "frontend-modern/public/docs/TROUBLESHOOTING.md"
+ heading = "#### Backup-age alerts were not evaluated\n"
+
+ def section(self):
+ text = self.guide.read_text()
+ self.assertEqual(text.count(self.heading), 1)
+ return " ".join(text.split(self.heading, 1)[1].split("\n#### ", 1)[0].split())
+
+ def test_warning_distinguishes_unavailable_evaluation_from_backup_failure(self):
+ text = self.section()
+ source = (ROOT / "internal/monitoring/recovery_rollups.go").read_text()
+ for phrase in ("Backup-age alerts were not evaluated because recovery data could not be read",
+ "Failed to list recovery rollups for backup alerts"):
+ self.assertIn(phrase, text)
+ self.assertIn(phrase, source)
+ for distinction in ("not evidence that a backup failed", "not a fresh assessment",
+ "absence of a new age alert does not establish",
+ "visible backup row does not prove that age evaluation completed",
+ "installation, guest type/ID, datastore, namespace and backup time"):
+ self.assertIn(distinction, text)
+
+ def test_policy_changes_and_warning_clearance_do_not_prove_recovery(self):
+ text = self.section()
+ for distinction in ("A disappearing warning alone is not recovery",
+ "Disabling alerts or backup-age checks",
+ "setting both age thresholds to zero",
+ "without a successful evaluation", "intended policy unchanged",
+ "ordinary polling", "restore readiness or guest thaw"):
+ self.assertIn(distinction, text)
+ source = (ROOT / "internal/monitoring/recovery_rollups.go").read_text()
+ for branch in ("!cfg.Enabled || !cfg.BackupDefaults.Enabled",
+ "cfg.BackupDefaults.WarningDays <= 0 && cfg.BackupDefaults.CriticalDays <= 0",
+ "ClearSystemAlert(alerts.BackupEvaluationAlertType)"):
+ self.assertIn(branch, source)
+
+ def test_help_reuses_safe_logs_and_preserves_workloads_and_evidence(self):
+ text = self.section()
+ for boundary in ("[bounded log reader](#inspect-notification-logs)",
+ "matching systemd or Docker deployment", "original time and message",
+ "timeout does not by itself establish its cause",
+ "failed log read is unavailable", "manually redacted error",
+ "database files out of public reports", "Do not restart or reinstall Pulse",
+ "change database schema", "Do not run a new backup",
+ "live diagnostics or guest-agent probe", "VM_DISK_MONITORING.md#backup-safety",
+ "successful writes to every covered filesystem", "workload liveness",
+ "only previously active monitoring"):
+ self.assertIn(boundary, text)
+ self.assertNotIn("```", text, "reuse the contained reader, not a new copied command")
+
+ def test_warning_help_is_in_the_shipped_mirror(self):
+ self.assertEqual(self.guide.read_bytes(), self.mirror.read_bytes())
+
+
class PVEBackupTroubleshootingDocsTest(unittest.TestCase):
def test_replication_help_uses_existing_row_details_not_backup_commands(self):
trouble = (ROOT / "docs/TROUBLESHOOTING.md").read_text()
@@ -200,10 +257,16 @@ PY
"no automatic rollback", "recorded active before maintenance"):
self.assertIn(phrase, text)
helper = (ROOT / "scripts/uninstall-sensor-proxy.sh").read_text()
- for fact in ('timeout 30 pct stop "$ctid"', 'timeout 30 pct start "$ctid"',
- 'cleanup_sensor_proxy_lines_in_conf "$conf" "$snapshot_line"',
- 'systemctl "$@" >/dev/null 2>&1 || true'):
+ for fact in ('timeout --kill-after=1s 30 pct stop "$ctid"',
+ 'timeout --kill-after=1s 30 pct start "$ctid"',
+ 'if ! cleanup_sensor_proxy_lines_in_conf "$conf"; then',
+ 'if ! disable_legacy_units; then',
+ 'if (( container_cleanup_status != 0 )); then'):
self.assertIn(fact, helper)
+ # The current helper now returns failures rather than hiding them.
+ # Keep the guide's stop-on-unknown boundary tied to that repair, not
+ # the superseded permissive systemctl adapter.
+ self.assertNotIn('systemctl "$@" >/dev/null 2>&1 || true', helper)
def test_preserves_credentials_and_explains_remote_cleanup_scope(self):
text = " ".join(self.section().split())