Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep missing PVE backup diagnostics non-destructive
pulse-triage[bot]138901e422c1
diff --git a/docs/UNIFIED_AGENT.md b/docs/UNIFIED_AGENT.mdindex 5bd325603e..0b891d5ead 100644--- a/docs/UNIFIED_AGENT.md+++ b/docs/UNIFIED_AGENT.md@@ -1165,20 +1165,47 @@ If your Docker Swarm cluster isn't being detected:### PVE Backups Not Showing (Recovery)-If local PVE backups aren't appearing in Pulse after setting up via `--enable-proxmox`:--1. **Check permissions**: The API token needs `PVEDatastoreAdmin` on `/storage`:- ```bash- pveum aclmod /storage -user pulse-monitor@pve -role PVEDatastoreAdmin- pveum aclmod /storage -token 'pulse-monitor@pve!<token-name>' -role PVEDatastoreAdmin- ```- Replace `pulse-monitor@pve!<token-name>` with the full token ID shown in Pulse.- Privilege-separated PVE tokens need the storage ACL on the token as well as the service user.--2. **Re-run setup**: Delete the node in Pulse Settings and re-run the agent with `--enable-proxmox`. Recent versions grant this permission automatically.--3. **Check state file**: If re-running doesn't trigger setup, remove the state file:- ```bash- rm /var/lib/pulse-agent/proxmox-pve-registered- ```- Then restart the agent.+Pulse's **server collects PVE backup inventory through the saved Proxmox API+connection**, including storage and storage-content requests. A working host+agent or successful `--enable-proxmox` setup does not prove that this separate+API collection is current. Guest-agent disk readings are not backup inventory.++1. **Check existing evidence in Proxmox first.** In the native PVE interface,+ identify the existing archive's node, storage, guest type/ID and time. Do not+ run another backup or restore to create evidence. A missing Pulse row is not+ proof that the archive is absent, and an OK task does not prove guest thaw.+2. **Compare the same source in Pulse.** Open **Proxmox → Backups → By date**+ and check the selected connection and filters against that archive. Coverage+ posture is a different reading: expand its explanation rather than treating+ an unprotected badge as a missing backup. Direct PBS collection and backups+ seen through PVE are distinct sources; see [PBS data sources](PBS.md#data-source-indicator)+ and [Recovery](RECOVERY.md#missing-or-inconsistent-evidence).+3. **Inspect ordinary collection, not a new probe.** In **Settings →+ Infrastructure**, check the affected PVE connection's existing status and+ last successful collection, together with the original redacted error and+ time. The server checks online nodes and queryable storage configured for+ backup content. A failed, unavailable or partial read is not an empty+ inventory; a successful connection test is not proof of sustained collection.+4. **Repair access only when the evidence supports it.** Ask the Proxmox+ administrator to compare the actual rejected endpoint and installed PVE+ version with the configured service user and token's effective storage+ permissions. With privilege separation, both user and token ACLs must allow+ the request, including the relevant scope and inheritance. Audit-only+ inventory access does not establish access to every backup-content request.+ Do not apply blanket storage-administrator ACLs merely because a table is+ empty, disable privilege separation or use an administrator token as a test.+ See [Proxmox permission checks](TROUBLESHOOTING.md#check-permissions-proxmox).++**Do not delete the monitored node, remove registration state or rerun setup as+a diagnostic.** Setup can rotate an existing API token and change permissions;+it is not a harmless refresh. Preserve the connection, credentials, registration+state and backup/history data. Make any evidenced configuration repair through+normal maintenance outside backups, then observe ordinary polling without+manual guest-agent probes, restarts or repeated **Test Connection** requests.++For freeze-enabled backups, follow the [backup safety precaution](VM_DISK_MONITORING.md#backup-safety).+It is not an incident recovery procedure or proof of a repaired thaw failure.+If reporting a continuing mismatch, retain the running server version, source,+original time, relevant missing privilege or redacted error and the differing+native/Pulse observations. Keep token secrets, full ACL listings and private+infrastructure details out of public reports; follow [Getting Help](TROUBLESHOOTING.md#-getting-help).diff --git a/frontend-modern/public/docs/UNIFIED_AGENT.md b/frontend-modern/public/docs/UNIFIED_AGENT.mdindex 5bd325603e..0b891d5ead 100644--- a/frontend-modern/public/docs/UNIFIED_AGENT.md+++ b/frontend-modern/public/docs/UNIFIED_AGENT.md@@ -1165,20 +1165,47 @@ If your Docker Swarm cluster isn't being detected:### PVE Backups Not Showing (Recovery)-If local PVE backups aren't appearing in Pulse after setting up via `--enable-proxmox`:--1. **Check permissions**: The API token needs `PVEDatastoreAdmin` on `/storage`:- ```bash- pveum aclmod /storage -user pulse-monitor@pve -role PVEDatastoreAdmin- pveum aclmod /storage -token 'pulse-monitor@pve!<token-name>' -role PVEDatastoreAdmin- ```- Replace `pulse-monitor@pve!<token-name>` with the full token ID shown in Pulse.- Privilege-separated PVE tokens need the storage ACL on the token as well as the service user.--2. **Re-run setup**: Delete the node in Pulse Settings and re-run the agent with `--enable-proxmox`. Recent versions grant this permission automatically.--3. **Check state file**: If re-running doesn't trigger setup, remove the state file:- ```bash- rm /var/lib/pulse-agent/proxmox-pve-registered- ```- Then restart the agent.+Pulse's **server collects PVE backup inventory through the saved Proxmox API+connection**, including storage and storage-content requests. A working host+agent or successful `--enable-proxmox` setup does not prove that this separate+API collection is current. Guest-agent disk readings are not backup inventory.++1. **Check existing evidence in Proxmox first.** In the native PVE interface,+ identify the existing archive's node, storage, guest type/ID and time. Do not+ run another backup or restore to create evidence. A missing Pulse row is not+ proof that the archive is absent, and an OK task does not prove guest thaw.+2. **Compare the same source in Pulse.** Open **Proxmox → Backups → By date**+ and check the selected connection and filters against that archive. Coverage+ posture is a different reading: expand its explanation rather than treating+ an unprotected badge as a missing backup. Direct PBS collection and backups+ seen through PVE are distinct sources; see [PBS data sources](PBS.md#data-source-indicator)+ and [Recovery](RECOVERY.md#missing-or-inconsistent-evidence).+3. **Inspect ordinary collection, not a new probe.** In **Settings →+ Infrastructure**, check the affected PVE connection's existing status and+ last successful collection, together with the original redacted error and+ time. The server checks online nodes and queryable storage configured for+ backup content. A failed, unavailable or partial read is not an empty+ inventory; a successful connection test is not proof of sustained collection.+4. **Repair access only when the evidence supports it.** Ask the Proxmox+ administrator to compare the actual rejected endpoint and installed PVE+ version with the configured service user and token's effective storage+ permissions. With privilege separation, both user and token ACLs must allow+ the request, including the relevant scope and inheritance. Audit-only+ inventory access does not establish access to every backup-content request.+ Do not apply blanket storage-administrator ACLs merely because a table is+ empty, disable privilege separation or use an administrator token as a test.+ See [Proxmox permission checks](TROUBLESHOOTING.md#check-permissions-proxmox).++**Do not delete the monitored node, remove registration state or rerun setup as+a diagnostic.** Setup can rotate an existing API token and change permissions;+it is not a harmless refresh. Preserve the connection, credentials, registration+state and backup/history data. Make any evidenced configuration repair through+normal maintenance outside backups, then observe ordinary polling without+manual guest-agent probes, restarts or repeated **Test Connection** requests.++For freeze-enabled backups, follow the [backup safety precaution](VM_DISK_MONITORING.md#backup-safety).+It is not an incident recovery procedure or proof of a repaired thaw failure.+If reporting a continuing mismatch, retain the running server version, source,+original time, relevant missing privilege or redacted error and the differing+native/Pulse observations. Keep token secrets, full ACL listings and private+infrastructure details out of public reports; follow [Getting Help](TROUBLESHOOTING.md#-getting-help).diff --git a/scripts/tests/test_pve_backup_troubleshooting_docs.py b/scripts/tests/test_pve_backup_troubleshooting_docs.pynew file mode 100644index 0000000000..44215d7f0a--- /dev/null+++ b/scripts/tests/test_pve_backup_troubleshooting_docs.py@@ -0,0 +1,82 @@+#!/usr/bin/env python3+"""Keep missing-backup guidance on the real collector and non-destructive path.++No provider calls, setup commands, backups or guest-agent probes are executed.+"""++from pathlib import Path+import re+import unittest+++ROOT = Path(__file__).resolve().parents[2]+GUIDE = "UNIFIED_AGENT.md"+HEADING = "### PVE Backups Not Showing (Recovery)"+++def backup_section():+ text = (ROOT / "docs" / GUIDE).read_text(encoding="utf-8")+ return text.split(HEADING, 1)[1].split("\n### ", 1)[0]+++class PVEBackupTroubleshootingDocsTest(unittest.TestCase):+ def test_shipped_help_matches_source(self):+ self.assertEqual((ROOT / "docs" / GUIDE).read_bytes(),+ (ROOT / "frontend-modern/public/docs" / GUIDE).read_bytes())++ def test_explains_actual_server_collection_not_guest_agent_readings(self):+ section = backup_section()+ for phrase in ("server collects PVE backup inventory", "saved Proxmox API",+ "storage-content requests", "Guest-agent disk readings are not backup inventory",+ "online nodes", "queryable storage", "backup content"):+ self.assertIn(phrase, section)+ collector = (ROOT / "internal/monitoring/monitor_backups.go").read_text()+ for operation in ("client.GetStorage(ctx, node.Node)",+ "client.GetStorageContent(ctx, node.Node, storage.Storage)",+ 'nodeEffectiveStatus[node.Node] != "online"',+ 'strings.Contains(storage.Content, "backup")',+ "storageContentQueryable(storage)"):+ self.assertIn(operation, collector)++ def test_no_destructive_reenrolment_or_blanket_acl_recipe(self):+ section = backup_section()+ shell = "\n".join(re.findall(r"```(?:bash|sh)\n(.*?)```", section, re.DOTALL))+ self.assertNotRegex(shell, r"\brm\b|aclmod|--enable-proxmox|systemctl\s+restart")+ normalised = " ".join(section.split())+ self.assertIn("Do not delete the monitored node, remove registration state or rerun setup", normalised)+ self.assertIn("Setup can rotate an existing API token", normalised)+ for item in ("connection", "credentials", "registration state", "backup/history data"):+ self.assertIn(item, normalised)+ setup = (ROOT / "internal/hostagent/proxmox_setup.go").read_text()+ self.assertIn("rotating token in place", setup)+ self.assertIn('"token", "remove", proxmoxUserPVE, tokenName', setup)++ def test_permissions_are_endpoint_specific_and_keep_privilege_separation(self):+ section = " ".join(backup_section().split())+ for phrase in ("actual rejected endpoint", "installed PVE version",+ "both user and token ACLs", "scope and inheritance",+ "Audit-only inventory access does not establish access",+ "Do not apply blanket storage-administrator ACLs",+ "disable privilege separation", "administrator token as a test"):+ self.assertIn(phrase, section)+ self.assertIn("TROUBLESHOOTING.md#check-permissions-proxmox", section)++ def test_preserves_source_freshness_and_backup_safety_distinctions(self):+ section = " ".join(backup_section().split())+ for phrase in ("node, storage, guest type/ID and time", "Proxmox → Backups → By date",+ "Coverage posture is a different reading", "distinct sources",+ "failed, unavailable or partial read is not an empty inventory",+ "OK task does not prove guest thaw", "Do not run another backup or restore",+ "outside backups", "without manual guest-agent probes",+ "Keep token secrets, full ACL listings and private infrastructure details"):+ self.assertIn(phrase, section)+ for target in ("PBS.md#data-source-indicator", "RECOVERY.md#missing-or-inconsistent-evidence",+ "VM_DISK_MONITORING.md#backup-safety", "TROUBLESHOOTING.md#-getting-help"):+ self.assertIn(target, section)+ safety = (ROOT / "docs/VM_DISK_MONITORING.md").read_text()+ self.assertIn("every filesystem covered by the backup", safety)+ self.assertIn("Pulse monitoring and alerts are unavailable", safety)+++if __name__ == "__main__":+ unittest.main()