Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Explain safe Proxmox replication disagreement checks
pulse-triage[bot]cb74547e150a
diff --git a/docs/FAQ.md b/docs/FAQ.mdindex d057a40391..b63fe1a319 100644--- a/docs/FAQ.md+++ b/docs/FAQ.md@@ -221,7 +221,8 @@ First distinguish what is missing; Pulse monitors more than Proxmox.A green connection badge, recent **Last seen** or successful **Test Connection**does not prove that every reading is fresh. Missing or unavailable is not zero.Use the relevant checks for [stale TrueNAS data](TRUENAS.md#stale-truenas-data),-[empty PBS History](PBS.md#pbs-is-connected-but-history-stays-empty) or+[empty PBS History](PBS.md#pbs-is-connected-but-history-stays-empty),+[missing or stale PVE replication jobs](TROUBLESHOOTING.md#replication-jobs-are-pending-stale-or-missing) or[missing VM disk usage](VM_DISK_MONITORING.md#a-missing-reading-is-not-an-installation-diagnosis). Check Proxmox[effective permissions](TROUBLESHOOTING.md#check-permissions-proxmox) only whenthe affected operation reports an access error, not on a missing chart alone.diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.mdindex d2c7def3e8..e5daeb1be2 100644--- a/docs/TROUBLESHOOTING.md+++ b/docs/TROUBLESHOOTING.md@@ -389,6 +389,52 @@ to compare one affected row's explanation, Job, History and Access with thematching native PBS record. Do not run a new backup, restart or clear historyjust to diagnose the disagreement.+#### Replication jobs are Pending, stale or missing++**Proxmox → Replication** monitors storage replication between PVE nodes,+not PBS backups. A replication result does not establish backup coverage,+restore readiness or guest thaw. Use the owning Proxmox installation's+existing replication status while Pulse's reading is uncertain.++Start with **All** and clear the table's search. **No replication jobs match+current filters** is different from an empty inventory, a missing tab or+**Could not load replication jobs**. Record the actual message; a missing+row is not proof that Proxmox deleted the job.++| Observation in Pulse | What to compare | What it does not establish |+| --- | --- | --- |+| **Pending**, with current sync times | The same job's outcome, failures and error in Proxmox. Pulse can retain sync times without a usable outcome for its Healthy/Pending classification. | Pending alone does not prove a job is waiting or failed. A recent time, zero displayed failures or no displayed error alone does not prove success. |+| Old sync times or an overdue next sync | The same job's existing native last/next sync and outcome. Record whether actual sync times advance during ordinary polling. | A changing “ago” or “overdue” label is the browser clock advancing, not evidence of a new poll. A Healthy pill does not establish current inventory or a running scheduler. |+| Empty inventory, missing tab or load error | Whether the expected jobs still exist in Proxmox, which Pulse connection owns them, and any displayed read error. | Missing or failed collection is unknown, not an empty healthy result. Correcting a status pill does not establish lifecycle or polling recovery. |++For one affected job, expand its row using the control beside **Guest**;+the details retain **Job**, **Route**, **Last sync**, **Next sync**, **Duration**+and **Failures** even when narrow layouts hide table columns. Compare the+owning installation, job/guest ID and source → target route with the existing+native Proxmox replication view. Similar names or VMIDs in separate+installations do not identify the same job. An unavailable native observation+can be reported as unavailable; do not force a sync to obtain one.++Treat **classification** and **stale or absent inventory after a lifecycle+change** as separate symptoms. If a previous restart restored data, keep that+observation and the now-working setup; do not repeat a restart, reboot or+update to reproduce it. One current row does not prove every job recovered.++For help, give the running Pulse/PVE versions, container image tag if relevant,+the original sequence and time, which symptoms remain, and one matching job's+Pulse versus native status, sync times, failures and redacted error. Include+any already-observed recovery, without recreating missing original evidence.+Use consistent placeholders for private installations, nodes and guest/job+identities so the route remains comparable. Do not share tokens, full API+responses, configuration, HAR exports or unredacted screenshots.++Do not restart or upgrade Pulse, reboot nodes, change schedules, disable/re-enable+jobs, recreate connections, widen token permissions or disable privilege+separation just to diagnose the table. Do not run a new sync, backup, restore,+live diagnostics or guest-agent probe as a test. A frozen or unresponsive+guest needs the separate [backup safety procedure](VM_DISK_MONITORING.md#backup-safety),+not these display checks.+#### Monitoring is mixed between Proxmox installationsPulse is intended to monitor [multiple Proxmox installations](CONFIGURATION.md#multiple-proxmox-installations).diff --git a/frontend-modern/public/docs/FAQ.md b/frontend-modern/public/docs/FAQ.mdindex d057a40391..b63fe1a319 100644--- a/frontend-modern/public/docs/FAQ.md+++ b/frontend-modern/public/docs/FAQ.md@@ -221,7 +221,8 @@ First distinguish what is missing; Pulse monitors more than Proxmox.A green connection badge, recent **Last seen** or successful **Test Connection**does not prove that every reading is fresh. Missing or unavailable is not zero.Use the relevant checks for [stale TrueNAS data](TRUENAS.md#stale-truenas-data),-[empty PBS History](PBS.md#pbs-is-connected-but-history-stays-empty) or+[empty PBS History](PBS.md#pbs-is-connected-but-history-stays-empty),+[missing or stale PVE replication jobs](TROUBLESHOOTING.md#replication-jobs-are-pending-stale-or-missing) or[missing VM disk usage](VM_DISK_MONITORING.md#a-missing-reading-is-not-an-installation-diagnosis). Check Proxmox[effective permissions](TROUBLESHOOTING.md#check-permissions-proxmox) only whenthe affected operation reports an access error, not on a missing chart alone.diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.mdindex d2c7def3e8..e5daeb1be2 100644--- a/frontend-modern/public/docs/TROUBLESHOOTING.md+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md@@ -389,6 +389,52 @@ to compare one affected row's explanation, Job, History and Access with thematching native PBS record. Do not run a new backup, restart or clear historyjust to diagnose the disagreement.+#### Replication jobs are Pending, stale or missing++**Proxmox → Replication** monitors storage replication between PVE nodes,+not PBS backups. A replication result does not establish backup coverage,+restore readiness or guest thaw. Use the owning Proxmox installation's+existing replication status while Pulse's reading is uncertain.++Start with **All** and clear the table's search. **No replication jobs match+current filters** is different from an empty inventory, a missing tab or+**Could not load replication jobs**. Record the actual message; a missing+row is not proof that Proxmox deleted the job.++| Observation in Pulse | What to compare | What it does not establish |+| --- | --- | --- |+| **Pending**, with current sync times | The same job's outcome, failures and error in Proxmox. Pulse can retain sync times without a usable outcome for its Healthy/Pending classification. | Pending alone does not prove a job is waiting or failed. A recent time, zero displayed failures or no displayed error alone does not prove success. |+| Old sync times or an overdue next sync | The same job's existing native last/next sync and outcome. Record whether actual sync times advance during ordinary polling. | A changing “ago” or “overdue” label is the browser clock advancing, not evidence of a new poll. A Healthy pill does not establish current inventory or a running scheduler. |+| Empty inventory, missing tab or load error | Whether the expected jobs still exist in Proxmox, which Pulse connection owns them, and any displayed read error. | Missing or failed collection is unknown, not an empty healthy result. Correcting a status pill does not establish lifecycle or polling recovery. |++For one affected job, expand its row using the control beside **Guest**;+the details retain **Job**, **Route**, **Last sync**, **Next sync**, **Duration**+and **Failures** even when narrow layouts hide table columns. Compare the+owning installation, job/guest ID and source → target route with the existing+native Proxmox replication view. Similar names or VMIDs in separate+installations do not identify the same job. An unavailable native observation+can be reported as unavailable; do not force a sync to obtain one.++Treat **classification** and **stale or absent inventory after a lifecycle+change** as separate symptoms. If a previous restart restored data, keep that+observation and the now-working setup; do not repeat a restart, reboot or+update to reproduce it. One current row does not prove every job recovered.++For help, give the running Pulse/PVE versions, container image tag if relevant,+the original sequence and time, which symptoms remain, and one matching job's+Pulse versus native status, sync times, failures and redacted error. Include+any already-observed recovery, without recreating missing original evidence.+Use consistent placeholders for private installations, nodes and guest/job+identities so the route remains comparable. Do not share tokens, full API+responses, configuration, HAR exports or unredacted screenshots.++Do not restart or upgrade Pulse, reboot nodes, change schedules, disable/re-enable+jobs, recreate connections, widen token permissions or disable privilege+separation just to diagnose the table. Do not run a new sync, backup, restore,+live diagnostics or guest-agent probe as a test. A frozen or unresponsive+guest needs the separate [backup safety procedure](VM_DISK_MONITORING.md#backup-safety),+not these display checks.+#### Monitoring is mixed between Proxmox installationsPulse is intended to monitor [multiple Proxmox installations](CONFIGURATION.md#multiple-proxmox-installations).diff --git a/frontend-modern/src/features/docs/__tests__/proxmoxReplicationGuide.test.ts b/frontend-modern/src/features/docs/__tests__/proxmoxReplicationGuide.test.tsnew file mode 100644index 0000000000..450c9f6644--- /dev/null+++ b/frontend-modern/src/features/docs/__tests__/proxmoxReplicationGuide.test.ts@@ -0,0 +1,115 @@+import { readFileSync } from 'node:fs';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { describe, expect, it } from 'vitest';+import { renderDocMarkdown } from '../docMarkdown';++const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../../..');+const fragment = '#replication-jobs-are-pending-stale-or-missing';++function renderGuide(name: string): HTMLElement {+ // Literal assigned-base inputs exercise missing guidance without editing+ // the candidate or implying acceptance of its runtime replication path.+ const directory =+ process.env.PULSE_REPLICATION_GUIDE_INPUT_DIRECTORY ??+ path.join(repoRoot, 'frontend-modern/public/docs');+ const article = document.createElement('article');+ article.innerHTML = renderDocMarkdown(readFileSync(path.join(directory, `${name}.md`), 'utf8'), name);+ return article;+}++function replicationSection(): HTMLElement {+ const heading = renderGuide('TROUBLESHOOTING').querySelector(fragment);+ expect(heading?.textContent).toBe('Replication jobs are Pending, stale or missing');+ const section = document.createElement('section');+ for (let next = heading!.nextElementSibling; next; next = next.nextElementSibling) {+ if (/^H[1-4]$/.test(next.tagName)) break;+ section.appendChild(next.cloneNode(true));+ }+ return section;+}++const prose = (section: HTMLElement): string =>+ (section.textContent ?? '').replace(/\s+/g, ' ').trim();++describe('shipped Proxmox replication disagreement guidance', () => {+ it('takes existing missing-data help to the real replication heading', () => {+ const link = renderGuide('FAQ').querySelector<HTMLAnchorElement>(+ `a[href="/docs/TROUBLESHOOTING${fragment}"]`,+ );+ expect(link?.textContent).toBe('missing or stale PVE replication jobs');+ expect(renderGuide('TROUBLESHOOTING').querySelector(new URL(link!.href).hash)).not.toBeNull();+ });++ it('distinguishes filtering, outcome classification and collection failures', () => {+ const section = replicationSection();+ const text = prose(section);+ expect(text).toContain("Start with All and clear the table's search");+ expect(text).toContain('No replication jobs match current filters');+ expect(text).toContain('Could not load replication jobs');+ const rows = [...section.querySelectorAll('tbody tr')];+ expect(rows).toHaveLength(3);+ expect(rows.map((row) => row.querySelector('td')?.textContent)).toEqual([+ 'Pending, with current sync times',+ 'Old sync times or an overdue next sync',+ 'Empty inventory, missing tab or load error',+ ]);+ for (const row of rows) expect(row.querySelectorAll('td')).toHaveLength(3);+ expect(prose(rows[0])).toContain('Pending alone does not prove a job is waiting or failed');+ expect(prose(rows[0])).toContain('alone does not prove success');+ expect(prose(rows[1])).toContain('browser clock advancing, not evidence of a new poll');+ expect(prose(rows[2])).toContain('unknown, not an empty healthy result');+ expect(prose(rows[2])).toContain('does not establish lifecycle or polling recovery');+ });++ it('makes the matching job and narrow-layout evidence reachable without new probes', () => {+ const text = prose(replicationSection());+ expect(text).toContain('expand its row using the control beside Guest');+ expect(text).toContain('Job, Route, Last sync, Next sync, Duration and Failures');+ expect(text).toContain('even when narrow layouts hide table columns');+ expect(text).toContain('owning installation, job/guest ID and source → target route');+ expect(text).toContain('Similar names or VMIDs in separate installations');+ expect(text).toContain('existing native Proxmox replication view');+ expect(text).toContain('do not force a sync to obtain one');+ });++ it('preserves separate symptoms, original recovery and a minimal private report', () => {+ const text = prose(replicationSection());+ expect(text).toContain('classification and stale or absent inventory after a lifecycle change');+ expect(text).toContain('keep that observation and the now-working setup');+ expect(text).toContain('One current row does not prove every job recovered');+ expect(text).toContain('which symptoms remain');+ expect(text).toContain('Pulse versus native status, sync times, failures and redacted error');+ expect(text).toContain('without recreating missing original evidence');+ expect(text).toContain('consistent placeholders');+ for (const withheld of ['tokens', 'full API responses', 'configuration', 'HAR exports']) {+ expect(text).toContain(withheld);+ }+ });++ it('keeps replication separate from backups, restore readiness and guest safety', () => {+ const section = replicationSection();+ const text = prose(section);+ expect(text).toContain('storage replication between PVE nodes, not PBS backups');+ expect(text).toContain('does not establish backup coverage, restore readiness or guest thaw');+ const link = section.querySelector<HTMLAnchorElement>(+ 'a[href="/docs/VM_DISK_MONITORING#backup-safety"]',+ );+ expect(link?.textContent).toBe('backup safety procedure');+ expect(renderGuide('VM_DISK_MONITORING').querySelector(new URL(link!.href).hash)).not.toBeNull();+ });++ it('rejects disruptive diagnostics, broad permissions and live reenactment', () => {+ const section = replicationSection();+ expect(section.querySelector('pre')).toBeNull();+ const text = prose(section);+ for (const boundary of [+ 'do not repeat a restart, reboot or update to reproduce it',+ 'Do not restart or upgrade Pulse, reboot nodes, change schedules, disable/re-enable jobs',+ 'recreate connections, widen token permissions or disable privilege separation',+ 'Do not run a new sync, backup, restore, live diagnostics or guest-agent probe as a test',+ ]) {+ expect(text).toContain(boundary);+ }+ });+});diff --git a/scripts/tests/test_pve_backup_troubleshooting_docs.py b/scripts/tests/test_pve_backup_troubleshooting_docs.pyindex 1e8074245a..9691d9fa5f 100644--- a/scripts/tests/test_pve_backup_troubleshooting_docs.py+++ b/scripts/tests/test_pve_backup_troubleshooting_docs.py@@ -20,6 +20,30 @@ def backup_section():class PVEBackupTroubleshootingDocsTest(unittest.TestCase):+ def test_replication_help_uses_existing_row_details_not_backup_commands(self):+ trouble = (ROOT / "docs/TROUBLESHOOTING.md").read_text()+ section = trouble.split("#### Replication jobs are Pending, stale or missing\n", 1)[1].split("\n#### ", 1)[0]+ text = " ".join(section.split())+ self.assertNotIn("```", section)+ table = (ROOT / "frontend-modern/src/features/proxmox/ProxmoxReplicationTable.tsx").read_text()+ # Tie the help's expansion and privacy-safe observation fields to the+ # actual surface, not a speculative job API or native recovery proof.+ self.assertIn("PlatformResourceDetailToggleButton", table)+ for label in ("Guest", "Job", "Route", "Last sync", "Next sync", "Duration", "Failures"):+ with self.subTest(label=label):+ self.assertIn(f"**{label}**", section)+ self.assertRegex(table, rf">{label}</(?:dt|TableHead)>")+ for phrase in ("not PBS backups", "unknown, not an empty healthy result",+ "does not establish lifecycle or polling recovery",+ "do not repeat a restart, reboot or update", "widen token permissions",+ "consistent placeholders", "VM_DISK_MONITORING.md#backup-safety"):+ self.assertIn(phrase, text)+ faq = (ROOT / "docs/FAQ.md").read_text()+ self.assertIn("TROUBLESHOOTING.md#replication-jobs-are-pending-stale-or-missing", faq)+ for name in ("FAQ.md", "TROUBLESHOOTING.md"):+ self.assertEqual((ROOT / "docs" / name).read_bytes(),+ (ROOT / "frontend-modern/public/docs" / name).read_bytes())+def test_multi_installation_help_is_mirrored_and_keeps_cosmetic_names_separate(self):for name in ("CONFIGURATION.md", "TROUBLESHOOTING.md", "PBS.md"):with self.subTest(guide=name):