Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Explain safe Proxmox replication disagreement checks

pulse-triage[bot]cb74547e150a
diff --git a/docs/FAQ.md b/docs/FAQ.md
index d057a40391..b63fe1a319 100644
--- a/docs/FAQ.md
+++ b/docs/FAQ.md
@@ -221,7 +221,8 @@ First distinguish what is missing; Pulse monitors more than Proxmox.
A green connection badge, recent **Last seen** or successful **Test Connection**
does not prove that every reading is fresh. Missing or unavailable is not zero.
Use the relevant checks for [stale TrueNAS data](TRUENAS.md#stale-truenas-data),
-[empty PBS History](PBS.md#pbs-is-connected-but-history-stays-empty) or
+[empty PBS History](PBS.md#pbs-is-connected-but-history-stays-empty),
+[missing or stale PVE replication jobs](TROUBLESHOOTING.md#replication-jobs-are-pending-stale-or-missing) or
[missing VM disk usage](VM_DISK_MONITORING.md#a-missing-reading-is-not-an-installation-diagnosis). Check Proxmox
[effective permissions](TROUBLESHOOTING.md#check-permissions-proxmox) only when
the affected operation reports an access error, not on a missing chart alone.
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md
index d2c7def3e8..e5daeb1be2 100644
--- a/docs/TROUBLESHOOTING.md
+++ b/docs/TROUBLESHOOTING.md
@@ -389,6 +389,52 @@ to compare one affected row's explanation, Job, History and Access with the
matching native PBS record. Do not run a new backup, restart or clear history
just to diagnose the disagreement.
+#### Replication jobs are Pending, stale or missing
+
+**Proxmox → Replication** monitors storage replication between PVE nodes,
+not PBS backups. A replication result does not establish backup coverage,
+restore readiness or guest thaw. Use the owning Proxmox installation's
+existing replication status while Pulse's reading is uncertain.
+
+Start with **All** and clear the table's search. **No replication jobs match
+current filters** is different from an empty inventory, a missing tab or
+**Could not load replication jobs**. Record the actual message; a missing
+row is not proof that Proxmox deleted the job.
+
+| Observation in Pulse | What to compare | What it does not establish |
+| --- | --- | --- |
+| **Pending**, with current sync times | The same job's outcome, failures and error in Proxmox. Pulse can retain sync times without a usable outcome for its Healthy/Pending classification. | Pending alone does not prove a job is waiting or failed. A recent time, zero displayed failures or no displayed error alone does not prove success. |
+| Old sync times or an overdue next sync | The same job's existing native last/next sync and outcome. Record whether actual sync times advance during ordinary polling. | A changing “ago” or “overdue” label is the browser clock advancing, not evidence of a new poll. A Healthy pill does not establish current inventory or a running scheduler. |
+| Empty inventory, missing tab or load error | Whether the expected jobs still exist in Proxmox, which Pulse connection owns them, and any displayed read error. | Missing or failed collection is unknown, not an empty healthy result. Correcting a status pill does not establish lifecycle or polling recovery. |
+
+For one affected job, expand its row using the control beside **Guest**;
+the details retain **Job**, **Route**, **Last sync**, **Next sync**, **Duration**
+and **Failures** even when narrow layouts hide table columns. Compare the
+owning installation, job/guest ID and source → target route with the existing
+native Proxmox replication view. Similar names or VMIDs in separate
+installations do not identify the same job. An unavailable native observation
+can be reported as unavailable; do not force a sync to obtain one.
+
+Treat **classification** and **stale or absent inventory after a lifecycle
+change** as separate symptoms. If a previous restart restored data, keep that
+observation and the now-working setup; do not repeat a restart, reboot or
+update to reproduce it. One current row does not prove every job recovered.
+
+For help, give the running Pulse/PVE versions, container image tag if relevant,
+the original sequence and time, which symptoms remain, and one matching job's
+Pulse versus native status, sync times, failures and redacted error. Include
+any already-observed recovery, without recreating missing original evidence.
+Use consistent placeholders for private installations, nodes and guest/job
+identities so the route remains comparable. Do not share tokens, full API
+responses, configuration, HAR exports or unredacted screenshots.
+
+Do not restart or upgrade Pulse, reboot nodes, change schedules, disable/re-enable
+jobs, recreate connections, widen token permissions or disable privilege
+separation just to diagnose the table. Do not run a new sync, backup, restore,
+live diagnostics or guest-agent probe as a test. A frozen or unresponsive
+guest needs the separate [backup safety procedure](VM_DISK_MONITORING.md#backup-safety),
+not these display checks.
+
#### Monitoring is mixed between Proxmox installations
Pulse is intended to monitor [multiple Proxmox installations](CONFIGURATION.md#multiple-proxmox-installations).
diff --git a/frontend-modern/public/docs/FAQ.md b/frontend-modern/public/docs/FAQ.md
index d057a40391..b63fe1a319 100644
--- a/frontend-modern/public/docs/FAQ.md
+++ b/frontend-modern/public/docs/FAQ.md
@@ -221,7 +221,8 @@ First distinguish what is missing; Pulse monitors more than Proxmox.
A green connection badge, recent **Last seen** or successful **Test Connection**
does not prove that every reading is fresh. Missing or unavailable is not zero.
Use the relevant checks for [stale TrueNAS data](TRUENAS.md#stale-truenas-data),
-[empty PBS History](PBS.md#pbs-is-connected-but-history-stays-empty) or
+[empty PBS History](PBS.md#pbs-is-connected-but-history-stays-empty),
+[missing or stale PVE replication jobs](TROUBLESHOOTING.md#replication-jobs-are-pending-stale-or-missing) or
[missing VM disk usage](VM_DISK_MONITORING.md#a-missing-reading-is-not-an-installation-diagnosis). Check Proxmox
[effective permissions](TROUBLESHOOTING.md#check-permissions-proxmox) only when
the affected operation reports an access error, not on a missing chart alone.
diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.md
index d2c7def3e8..e5daeb1be2 100644
--- a/frontend-modern/public/docs/TROUBLESHOOTING.md
+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md
@@ -389,6 +389,52 @@ to compare one affected row's explanation, Job, History and Access with the
matching native PBS record. Do not run a new backup, restart or clear history
just to diagnose the disagreement.
+#### Replication jobs are Pending, stale or missing
+
+**Proxmox → Replication** monitors storage replication between PVE nodes,
+not PBS backups. A replication result does not establish backup coverage,
+restore readiness or guest thaw. Use the owning Proxmox installation's
+existing replication status while Pulse's reading is uncertain.
+
+Start with **All** and clear the table's search. **No replication jobs match
+current filters** is different from an empty inventory, a missing tab or
+**Could not load replication jobs**. Record the actual message; a missing
+row is not proof that Proxmox deleted the job.
+
+| Observation in Pulse | What to compare | What it does not establish |
+| --- | --- | --- |
+| **Pending**, with current sync times | The same job's outcome, failures and error in Proxmox. Pulse can retain sync times without a usable outcome for its Healthy/Pending classification. | Pending alone does not prove a job is waiting or failed. A recent time, zero displayed failures or no displayed error alone does not prove success. |
+| Old sync times or an overdue next sync | The same job's existing native last/next sync and outcome. Record whether actual sync times advance during ordinary polling. | A changing “ago” or “overdue” label is the browser clock advancing, not evidence of a new poll. A Healthy pill does not establish current inventory or a running scheduler. |
+| Empty inventory, missing tab or load error | Whether the expected jobs still exist in Proxmox, which Pulse connection owns them, and any displayed read error. | Missing or failed collection is unknown, not an empty healthy result. Correcting a status pill does not establish lifecycle or polling recovery. |
+
+For one affected job, expand its row using the control beside **Guest**;
+the details retain **Job**, **Route**, **Last sync**, **Next sync**, **Duration**
+and **Failures** even when narrow layouts hide table columns. Compare the
+owning installation, job/guest ID and source → target route with the existing
+native Proxmox replication view. Similar names or VMIDs in separate
+installations do not identify the same job. An unavailable native observation
+can be reported as unavailable; do not force a sync to obtain one.
+
+Treat **classification** and **stale or absent inventory after a lifecycle
+change** as separate symptoms. If a previous restart restored data, keep that
+observation and the now-working setup; do not repeat a restart, reboot or
+update to reproduce it. One current row does not prove every job recovered.
+
+For help, give the running Pulse/PVE versions, container image tag if relevant,
+the original sequence and time, which symptoms remain, and one matching job's
+Pulse versus native status, sync times, failures and redacted error. Include
+any already-observed recovery, without recreating missing original evidence.
+Use consistent placeholders for private installations, nodes and guest/job
+identities so the route remains comparable. Do not share tokens, full API
+responses, configuration, HAR exports or unredacted screenshots.
+
+Do not restart or upgrade Pulse, reboot nodes, change schedules, disable/re-enable
+jobs, recreate connections, widen token permissions or disable privilege
+separation just to diagnose the table. Do not run a new sync, backup, restore,
+live diagnostics or guest-agent probe as a test. A frozen or unresponsive
+guest needs the separate [backup safety procedure](VM_DISK_MONITORING.md#backup-safety),
+not these display checks.
+
#### Monitoring is mixed between Proxmox installations
Pulse is intended to monitor [multiple Proxmox installations](CONFIGURATION.md#multiple-proxmox-installations).
diff --git a/frontend-modern/src/features/docs/__tests__/proxmoxReplicationGuide.test.ts b/frontend-modern/src/features/docs/__tests__/proxmoxReplicationGuide.test.ts
new file mode 100644
index 0000000000..450c9f6644
--- /dev/null
+++ b/frontend-modern/src/features/docs/__tests__/proxmoxReplicationGuide.test.ts
@@ -0,0 +1,115 @@
+import { readFileSync } from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { describe, expect, it } from 'vitest';
+import { renderDocMarkdown } from '../docMarkdown';
+
+const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../../..');
+const fragment = '#replication-jobs-are-pending-stale-or-missing';
+
+function renderGuide(name: string): HTMLElement {
+ // Literal assigned-base inputs exercise missing guidance without editing
+ // the candidate or implying acceptance of its runtime replication path.
+ const directory =
+ process.env.PULSE_REPLICATION_GUIDE_INPUT_DIRECTORY ??
+ path.join(repoRoot, 'frontend-modern/public/docs');
+ const article = document.createElement('article');
+ article.innerHTML = renderDocMarkdown(readFileSync(path.join(directory, `${name}.md`), 'utf8'), name);
+ return article;
+}
+
+function replicationSection(): HTMLElement {
+ const heading = renderGuide('TROUBLESHOOTING').querySelector(fragment);
+ expect(heading?.textContent).toBe('Replication jobs are Pending, stale or missing');
+ const section = document.createElement('section');
+ for (let next = heading!.nextElementSibling; next; next = next.nextElementSibling) {
+ if (/^H[1-4]$/.test(next.tagName)) break;
+ section.appendChild(next.cloneNode(true));
+ }
+ return section;
+}
+
+const prose = (section: HTMLElement): string =>
+ (section.textContent ?? '').replace(/\s+/g, ' ').trim();
+
+describe('shipped Proxmox replication disagreement guidance', () => {
+ it('takes existing missing-data help to the real replication heading', () => {
+ const link = renderGuide('FAQ').querySelector<HTMLAnchorElement>(
+ `a[href="/docs/TROUBLESHOOTING${fragment}"]`,
+ );
+ expect(link?.textContent).toBe('missing or stale PVE replication jobs');
+ expect(renderGuide('TROUBLESHOOTING').querySelector(new URL(link!.href).hash)).not.toBeNull();
+ });
+
+ it('distinguishes filtering, outcome classification and collection failures', () => {
+ const section = replicationSection();
+ const text = prose(section);
+ expect(text).toContain("Start with All and clear the table's search");
+ expect(text).toContain('No replication jobs match current filters');
+ expect(text).toContain('Could not load replication jobs');
+ const rows = [...section.querySelectorAll('tbody tr')];
+ expect(rows).toHaveLength(3);
+ expect(rows.map((row) => row.querySelector('td')?.textContent)).toEqual([
+ 'Pending, with current sync times',
+ 'Old sync times or an overdue next sync',
+ 'Empty inventory, missing tab or load error',
+ ]);
+ for (const row of rows) expect(row.querySelectorAll('td')).toHaveLength(3);
+ expect(prose(rows[0])).toContain('Pending alone does not prove a job is waiting or failed');
+ expect(prose(rows[0])).toContain('alone does not prove success');
+ expect(prose(rows[1])).toContain('browser clock advancing, not evidence of a new poll');
+ expect(prose(rows[2])).toContain('unknown, not an empty healthy result');
+ expect(prose(rows[2])).toContain('does not establish lifecycle or polling recovery');
+ });
+
+ it('makes the matching job and narrow-layout evidence reachable without new probes', () => {
+ const text = prose(replicationSection());
+ expect(text).toContain('expand its row using the control beside Guest');
+ expect(text).toContain('Job, Route, Last sync, Next sync, Duration and Failures');
+ expect(text).toContain('even when narrow layouts hide table columns');
+ expect(text).toContain('owning installation, job/guest ID and source → target route');
+ expect(text).toContain('Similar names or VMIDs in separate installations');
+ expect(text).toContain('existing native Proxmox replication view');
+ expect(text).toContain('do not force a sync to obtain one');
+ });
+
+ it('preserves separate symptoms, original recovery and a minimal private report', () => {
+ const text = prose(replicationSection());
+ expect(text).toContain('classification and stale or absent inventory after a lifecycle change');
+ expect(text).toContain('keep that observation and the now-working setup');
+ expect(text).toContain('One current row does not prove every job recovered');
+ expect(text).toContain('which symptoms remain');
+ expect(text).toContain('Pulse versus native status, sync times, failures and redacted error');
+ expect(text).toContain('without recreating missing original evidence');
+ expect(text).toContain('consistent placeholders');
+ for (const withheld of ['tokens', 'full API responses', 'configuration', 'HAR exports']) {
+ expect(text).toContain(withheld);
+ }
+ });
+
+ it('keeps replication separate from backups, restore readiness and guest safety', () => {
+ const section = replicationSection();
+ const text = prose(section);
+ expect(text).toContain('storage replication between PVE nodes, not PBS backups');
+ expect(text).toContain('does not establish backup coverage, restore readiness or guest thaw');
+ const link = section.querySelector<HTMLAnchorElement>(
+ 'a[href="/docs/VM_DISK_MONITORING#backup-safety"]',
+ );
+ expect(link?.textContent).toBe('backup safety procedure');
+ expect(renderGuide('VM_DISK_MONITORING').querySelector(new URL(link!.href).hash)).not.toBeNull();
+ });
+
+ it('rejects disruptive diagnostics, broad permissions and live reenactment', () => {
+ const section = replicationSection();
+ expect(section.querySelector('pre')).toBeNull();
+ const text = prose(section);
+ for (const boundary of [
+ 'do not repeat a restart, reboot or update to reproduce it',
+ 'Do not restart or upgrade Pulse, reboot nodes, change schedules, disable/re-enable jobs',
+ 'recreate connections, widen token permissions or disable privilege separation',
+ 'Do not run a new sync, backup, restore, live diagnostics or guest-agent probe as a test',
+ ]) {
+ expect(text).toContain(boundary);
+ }
+ });
+});
diff --git a/scripts/tests/test_pve_backup_troubleshooting_docs.py b/scripts/tests/test_pve_backup_troubleshooting_docs.py
index 1e8074245a..9691d9fa5f 100644
--- a/scripts/tests/test_pve_backup_troubleshooting_docs.py
+++ b/scripts/tests/test_pve_backup_troubleshooting_docs.py
@@ -20,6 +20,30 @@ def backup_section():
class PVEBackupTroubleshootingDocsTest(unittest.TestCase):
+ def test_replication_help_uses_existing_row_details_not_backup_commands(self):
+ trouble = (ROOT / "docs/TROUBLESHOOTING.md").read_text()
+ section = trouble.split("#### Replication jobs are Pending, stale or missing\n", 1)[1].split("\n#### ", 1)[0]
+ text = " ".join(section.split())
+ self.assertNotIn("```", section)
+ table = (ROOT / "frontend-modern/src/features/proxmox/ProxmoxReplicationTable.tsx").read_text()
+ # Tie the help's expansion and privacy-safe observation fields to the
+ # actual surface, not a speculative job API or native recovery proof.
+ self.assertIn("PlatformResourceDetailToggleButton", table)
+ for label in ("Guest", "Job", "Route", "Last sync", "Next sync", "Duration", "Failures"):
+ with self.subTest(label=label):
+ self.assertIn(f"**{label}**", section)
+ self.assertRegex(table, rf">{label}</(?:dt|TableHead)>")
+ for phrase in ("not PBS backups", "unknown, not an empty healthy result",
+ "does not establish lifecycle or polling recovery",
+ "do not repeat a restart, reboot or update", "widen token permissions",
+ "consistent placeholders", "VM_DISK_MONITORING.md#backup-safety"):
+ self.assertIn(phrase, text)
+ faq = (ROOT / "docs/FAQ.md").read_text()
+ self.assertIn("TROUBLESHOOTING.md#replication-jobs-are-pending-stale-or-missing", faq)
+ for name in ("FAQ.md", "TROUBLESHOOTING.md"):
+ self.assertEqual((ROOT / "docs" / name).read_bytes(),
+ (ROOT / "frontend-modern/public/docs" / name).read_bytes())
+
def test_multi_installation_help_is_mirrored_and_keeps_cosmetic_names_separate(self):
for name in ("CONFIGURATION.md", "TROUBLESHOOTING.md", "PBS.md"):
with self.subTest(guide=name):