Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Explain safe cross-installation Proxmox identity comparisons
pulse-triage[bot]453509905aff
diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.mdindex 9fca27bca8..0653fae3d3 100644--- a/docs/CONFIGURATION.md+++ b/docs/CONFIGURATION.md@@ -336,6 +336,20 @@ PULSE_AGENT_CONNECT_URL=https://agents.example.com:7656Agents then post telemetry to `https://agents.example.com:7656/api/agents/agent/report` and establish their command channel at `wss://agents.example.com:7656/api/agent/ws`, while the web UI and management API remain reachable only on the private `FRONTEND_PORT` listener. If command execution is enabled, both routes must traverse the same proxy/firewall path; a successful report does not prove that the WebSocket is admitted.+### Multiple Proxmox installations++Pulse is intended to monitor multiple Proxmox clusters and standalone nodes+through separate saved Proxmox connections. Native node names and VMIDs can+repeat across independent installations; a matching name or VMID alone does+not identify the same resource.++A distinct Pulse connection name helps you recognise the saved connection.+Neither that label nor a [cluster member display name](#proxmox-cluster-node-display-names)+renames the native Proxmox node or VMID, or repairs incorrect attribution.+If node errors, backup status or agent-backed Docker monitoring become mixed+after adding a connection, use the [cross-installation identity checks](TROUBLESHOOTING.md#monitoring-is-mixed-between-proxmox-installations).+Do not rename production nodes or repeat the addition as a diagnostic.+### Proxmox Cluster Node Display NamesNodes discovered through one Proxmox VE cluster connection can have andiff --git a/docs/PBS.md b/docs/PBS.mdindex 43aabdbba6..d188c13f5f 100644--- a/docs/PBS.md+++ b/docs/PBS.md@@ -424,6 +424,11 @@ If the same backup remains listed twice, report those redacted distinctions andwhether each entry came from direct PBS or PVE passthrough. Do not delete backupsor change retention to hide a display problem.+If overlapping VMIDs also coincide with wrong-node errors or lost agent-backed+Docker monitoring after adding another Proxmox connection, use the+[cross-installation identity checks](TROUBLESHOOTING.md#monitoring-is-mixed-between-proxmox-installations).+A duplicate-backup check alone does not resolve those other symptoms.+---## Data Source Indicatordiff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.mdindex cf3a061068..d2c7def3e8 100644--- a/docs/TROUBLESHOOTING.md+++ b/docs/TROUBLESHOOTING.md@@ -281,6 +281,9 @@ restore, not by overwriting today's history. Follow#### Agent fleet update or identity issue+- If readings became mixed after adding another Proxmox installation, start+ with the [cross-installation identity checks](#monitoring-is-mixed-between-proxmox-installations),+ not an agent update or re-enrolment command.- Open an outdated-agent notice or`/settings/infrastructure?agentDoctor=1` to open **Agent Doctor** andcopy the platform-specific command for each reported host. This is a manual@@ -386,6 +389,47 @@ to compare one affected row's explanation, Job, History and Access with thematching native PBS record. Do not run a new backup, restart or clear historyjust to diagnose the disagreement.+#### Monitoring is mixed between Proxmox installations++Pulse is intended to monitor [multiple Proxmox installations](CONFIGURATION.md#multiple-proxmox-installations).+Separate installations can reuse native node names and VMIDs. A distinct+Pulse connection label or display name is not a repair for wrong attribution.++If adding a connection led to node errors on the wrong node, unavailable+backup status for overlapping VMIDs, or lost agent-backed Docker monitoring,+treat each as an unresolved symptom. One restored view does not establish+that the others recovered. Use **existing observations only**; if you already+repaired the setup, keep that working setup and the original sequence rather+than undoing the repair or repeating the addition.++| Affected reading | Safe comparison | What it does not prove |+| --- | --- | --- |+| Node errors | Compare the affected node and observation time with that installation's own Proxmox view. | A similar node name or a different Pulse label does not establish which installation owns the error. |+| Guest backup status | Compare one affected guest with the matching native Proxmox/PBS record: owning installation, guest type/VMID, datastore, namespace and backup time. Follow the [PBS evidence checks](PBS.md#backups-are-visible-but-coverage-says-unprotected). | The same VMID or a visible backup from another installation does not establish protection for this guest. |+| Agent-backed Docker monitoring | In **Settings → Infrastructure → Agent Doctor**, inspect the affected host's existing **Last seen** and **Identity evidence**, including **Connection** and **Hostname** where available. Compare with the known Docker host, not a similar guest name. | Recent agent contact or a Healthy badge does not prove that the correct host's containers are visible or correctly attributed. Missing evidence is unknown, not proof that the agent stopped. |++While attribution is uncertain, use each installation's own Proxmox/PBS+views for node errors and backup status, and the Docker host's existing runtime+view for container status. Do not make backup, restore or workload changes+from a potentially misattributed Pulse reading.++For a report, give the running server/agent versions, the original sequence,+which of the three symptoms remain, and one affected pair's expected versus+displayed origin and observation times where available. Use **consistent+placeholders** such as `site-A`, `site-B`, `node-X` and `guest-100` across every+view; say explicitly when the native names or VMIDs are equal. Keep actual+addresses, hostnames, connection IDs and machine identities private. Do not+share a full Agent Doctor report, configuration, API response, HAR export,+token or unredacted screenshot. Unavailable original evidence can be reported+as unavailable; do not recreate the incident to obtain it.++Do not rename production nodes, change VMIDs or machine IDs, delete/re-add+connections, re-enrol agents, rotate tokens, restart services or clear History+just to diagnose this disagreement. Do not run live diagnostics, guest-agent+probes, a new backup or a restore as an identity test. An OK backup does not+prove thaw; a frozen or unresponsive guest needs the separate+[backup safety procedure](VM_DISK_MONITORING.md#backup-safety), not these display checks.+#### Temperature data missing- Compare the affected host's active agent version, last report, sensor andobservation time. A current server or another sensor's value is not evidencediff --git a/frontend-modern/public/docs/CONFIGURATION.md b/frontend-modern/public/docs/CONFIGURATION.mdindex 9fca27bca8..0653fae3d3 100644--- a/frontend-modern/public/docs/CONFIGURATION.md+++ b/frontend-modern/public/docs/CONFIGURATION.md@@ -336,6 +336,20 @@ PULSE_AGENT_CONNECT_URL=https://agents.example.com:7656Agents then post telemetry to `https://agents.example.com:7656/api/agents/agent/report` and establish their command channel at `wss://agents.example.com:7656/api/agent/ws`, while the web UI and management API remain reachable only on the private `FRONTEND_PORT` listener. If command execution is enabled, both routes must traverse the same proxy/firewall path; a successful report does not prove that the WebSocket is admitted.+### Multiple Proxmox installations++Pulse is intended to monitor multiple Proxmox clusters and standalone nodes+through separate saved Proxmox connections. Native node names and VMIDs can+repeat across independent installations; a matching name or VMID alone does+not identify the same resource.++A distinct Pulse connection name helps you recognise the saved connection.+Neither that label nor a [cluster member display name](#proxmox-cluster-node-display-names)+renames the native Proxmox node or VMID, or repairs incorrect attribution.+If node errors, backup status or agent-backed Docker monitoring become mixed+after adding a connection, use the [cross-installation identity checks](TROUBLESHOOTING.md#monitoring-is-mixed-between-proxmox-installations).+Do not rename production nodes or repeat the addition as a diagnostic.+### Proxmox Cluster Node Display NamesNodes discovered through one Proxmox VE cluster connection can have andiff --git a/frontend-modern/public/docs/PBS.md b/frontend-modern/public/docs/PBS.mdindex 43aabdbba6..d188c13f5f 100644--- a/frontend-modern/public/docs/PBS.md+++ b/frontend-modern/public/docs/PBS.md@@ -424,6 +424,11 @@ If the same backup remains listed twice, report those redacted distinctions andwhether each entry came from direct PBS or PVE passthrough. Do not delete backupsor change retention to hide a display problem.+If overlapping VMIDs also coincide with wrong-node errors or lost agent-backed+Docker monitoring after adding another Proxmox connection, use the+[cross-installation identity checks](TROUBLESHOOTING.md#monitoring-is-mixed-between-proxmox-installations).+A duplicate-backup check alone does not resolve those other symptoms.+---## Data Source Indicatordiff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.mdindex cf3a061068..d2c7def3e8 100644--- a/frontend-modern/public/docs/TROUBLESHOOTING.md+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md@@ -281,6 +281,9 @@ restore, not by overwriting today's history. Follow#### Agent fleet update or identity issue+- If readings became mixed after adding another Proxmox installation, start+ with the [cross-installation identity checks](#monitoring-is-mixed-between-proxmox-installations),+ not an agent update or re-enrolment command.- Open an outdated-agent notice or`/settings/infrastructure?agentDoctor=1` to open **Agent Doctor** andcopy the platform-specific command for each reported host. This is a manual@@ -386,6 +389,47 @@ to compare one affected row's explanation, Job, History and Access with thematching native PBS record. Do not run a new backup, restart or clear historyjust to diagnose the disagreement.+#### Monitoring is mixed between Proxmox installations++Pulse is intended to monitor [multiple Proxmox installations](CONFIGURATION.md#multiple-proxmox-installations).+Separate installations can reuse native node names and VMIDs. A distinct+Pulse connection label or display name is not a repair for wrong attribution.++If adding a connection led to node errors on the wrong node, unavailable+backup status for overlapping VMIDs, or lost agent-backed Docker monitoring,+treat each as an unresolved symptom. One restored view does not establish+that the others recovered. Use **existing observations only**; if you already+repaired the setup, keep that working setup and the original sequence rather+than undoing the repair or repeating the addition.++| Affected reading | Safe comparison | What it does not prove |+| --- | --- | --- |+| Node errors | Compare the affected node and observation time with that installation's own Proxmox view. | A similar node name or a different Pulse label does not establish which installation owns the error. |+| Guest backup status | Compare one affected guest with the matching native Proxmox/PBS record: owning installation, guest type/VMID, datastore, namespace and backup time. Follow the [PBS evidence checks](PBS.md#backups-are-visible-but-coverage-says-unprotected). | The same VMID or a visible backup from another installation does not establish protection for this guest. |+| Agent-backed Docker monitoring | In **Settings → Infrastructure → Agent Doctor**, inspect the affected host's existing **Last seen** and **Identity evidence**, including **Connection** and **Hostname** where available. Compare with the known Docker host, not a similar guest name. | Recent agent contact or a Healthy badge does not prove that the correct host's containers are visible or correctly attributed. Missing evidence is unknown, not proof that the agent stopped. |++While attribution is uncertain, use each installation's own Proxmox/PBS+views for node errors and backup status, and the Docker host's existing runtime+view for container status. Do not make backup, restore or workload changes+from a potentially misattributed Pulse reading.++For a report, give the running server/agent versions, the original sequence,+which of the three symptoms remain, and one affected pair's expected versus+displayed origin and observation times where available. Use **consistent+placeholders** such as `site-A`, `site-B`, `node-X` and `guest-100` across every+view; say explicitly when the native names or VMIDs are equal. Keep actual+addresses, hostnames, connection IDs and machine identities private. Do not+share a full Agent Doctor report, configuration, API response, HAR export,+token or unredacted screenshot. Unavailable original evidence can be reported+as unavailable; do not recreate the incident to obtain it.++Do not rename production nodes, change VMIDs or machine IDs, delete/re-add+connections, re-enrol agents, rotate tokens, restart services or clear History+just to diagnose this disagreement. Do not run live diagnostics, guest-agent+probes, a new backup or a restore as an identity test. An OK backup does not+prove thaw; a frozen or unresponsive guest needs the separate+[backup safety procedure](VM_DISK_MONITORING.md#backup-safety), not these display checks.+#### Temperature data missing- Compare the affected host's active agent version, last report, sensor andobservation time. A current server or another sensor's value is not evidencediff --git a/frontend-modern/src/features/docs/__tests__/proxmoxIdentityGuide.test.ts b/frontend-modern/src/features/docs/__tests__/proxmoxIdentityGuide.test.tsnew file mode 100644index 0000000000..077d1eec46--- /dev/null+++ b/frontend-modern/src/features/docs/__tests__/proxmoxIdentityGuide.test.ts@@ -0,0 +1,137 @@+import { readFileSync } from 'node:fs';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { describe, expect, it } from 'vitest';+import { renderDocMarkdown } from '../docMarkdown';++const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../../..');+const identityFragment = '#monitoring-is-mixed-between-proxmox-installations';++function renderGuide(name: string): HTMLElement {+ // Optional exact-base inputs exercise the former shipped advice without+ // changing the committed candidate being checked in the proof VM.+ const directory =+ process.env.PULSE_PROXMOX_IDENTITY_GUIDE_INPUT_DIRECTORY ??+ path.join(repoRoot, 'frontend-modern/public/docs');+ const article = document.createElement('article');+ article.innerHTML = renderDocMarkdown(readFileSync(path.join(directory, `${name}.md`), 'utf8'), name);+ return article;+}++function sectionAt(article: HTMLElement, fragment: string): HTMLElement {+ const heading = article.querySelector(fragment);+ expect(heading).not.toBeNull();+ const section = document.createElement('section');+ for (let next = heading!.nextElementSibling; next; next = next.nextElementSibling) {+ if (/^H[1-4]$/.test(next.tagName)) break;+ section.appendChild(next.cloneNode(true));+ }+ return section;+}++const identitySection = (): HTMLElement => sectionAt(renderGuide('TROUBLESHOOTING'), identityFragment);+const prose = (section: HTMLElement): string => (section.textContent ?? '').replace(/\s+/g, ' ').trim();++describe('shipped cross-installation Proxmox identity guidance', () => {+ it('distinguishes intended multi-installation support from cosmetic names and a verified repair', () => {+ const article = renderGuide('CONFIGURATION');+ const section = sectionAt(article, '#multiple-proxmox-installations');+ const text = prose(section);+ expect(text).toContain('intended to monitor multiple Proxmox clusters and standalone nodes');+ expect(text).toContain('separate saved Proxmox connections');+ expect(text).toContain('matching name or VMID alone does not identify the same resource');+ expect(text).toContain('Neither that label nor a cluster member display name');+ expect(text).toContain('repairs incorrect attribution');+ expect(section.querySelector('a[href="#proxmox-cluster-node-display-names"]')).not.toBeNull();+ expect(article.querySelector('#proxmox-cluster-node-display-names')).not.toBeNull();+ });++ it('keeps all three symptoms separate with a safe comparison and a limitation for each', () => {+ const section = identitySection();+ const rows = [...section.querySelectorAll('tbody tr')];+ expect(rows).toHaveLength(3);+ expect(rows.map((row) => row.querySelector('td')?.textContent)).toEqual([+ 'Node errors',+ 'Guest backup status',+ 'Agent-backed Docker monitoring',+ ]);+ for (const row of rows) expect(row.querySelectorAll('td')).toHaveLength(3);+ expect(prose(section)).toContain('One restored view does not establish that the others recovered');+ expect(prose(rows[0])).toContain("that installation's own Proxmox view");+ expect(prose(rows[1])).toContain('owning installation, guest type/VMID, datastore, namespace and backup time');+ expect(prose(rows[2])).toContain('Last seen and Identity evidence');+ expect(prose(rows[2])).toContain('Recent agent contact or a Healthy badge does not prove');+ });++ it('takes configuration, backup and generic agent-help entry points to the same real heading', () => {+ for (const name of ['CONFIGURATION', 'PBS']) {+ const link = renderGuide(name).querySelector<HTMLAnchorElement>(+ `a[href="/docs/TROUBLESHOOTING${identityFragment}"]`,+ );+ expect(link?.textContent).toBe('cross-installation identity checks');+ expect(renderGuide('TROUBLESHOOTING').querySelector(new URL(link!.href).hash)?.textContent).toBe(+ 'Monitoring is mixed between Proxmox installations',+ );+ }+ const agent = sectionAt(renderGuide('TROUBLESHOOTING'), '#agent-fleet-update-or-identity-issue');+ expect(agent.querySelector(`a[href="${identityFragment}"]`)).not.toBeNull();+ expect(prose(agent)).toContain('not an agent update or re-enrolment command');+ });++ it('reaches existing configuration, PBS and guest safety guidance rather than dead fragments', () => {+ const section = identitySection();+ for (const [name, fragment] of [+ ['CONFIGURATION', '#multiple-proxmox-installations'],+ ['PBS', '#backups-are-visible-but-coverage-says-unprotected'],+ ['VM_DISK_MONITORING', '#backup-safety'],+ ]) {+ expect(section.querySelector(`a[href="/docs/${name}${fragment}"]`)).not.toBeNull();+ expect(renderGuide(name).querySelector(fragment)).not.toBeNull();+ }+ expect(prose(section)).toContain('An OK backup does not prove thaw');+ expect(prose(section)).toContain('a frozen or unresponsive guest needs the separate backup safety procedure');+ });++ it('offers independent native readings before asking for a minimal report', () => {+ const section = identitySection();+ const paragraphs = [...section.querySelectorAll('p')];+ const workaround = paragraphs.findIndex((p) => prose(p).startsWith('While attribution is uncertain'));+ const report = paragraphs.findIndex((p) => prose(p).startsWith('For a report'));+ expect(workaround).toBeGreaterThanOrEqual(0);+ expect(report).toBeGreaterThan(workaround);+ expect(prose(paragraphs[workaround])).toContain("Docker host's existing runtime view");+ expect(prose(paragraphs[workaround])).toContain('Do not make backup, restore or workload changes');+ expect(prose(paragraphs[report])).toContain('which of the three symptoms remain');+ expect(prose(paragraphs[report])).toContain('expected versus displayed origin');+ });++ it('preserves privacy while keeping equal native identities comparable across views', () => {+ const text = prose(identitySection());+ expect(text).toContain('consistent placeholders such as site-A, site-B, node-X and guest-100');+ expect(text).toContain('say explicitly when the native names or VMIDs are equal');+ expect(text).toContain('Keep actual addresses, hostnames, connection IDs and machine identities private');+ expect(text).toContain('Do not share a full Agent Doctor report');+ for (const withheld of ['API response', 'HAR export', 'token', 'unredacted screenshot']) {+ expect(text).toContain(withheld);+ }+ });++ it('uses existing evidence without destructive recovery, live probes or incident recreation', () => {+ const section = identitySection();+ expect(section.querySelector('pre')).toBeNull();+ const text = prose(section);+ for (const boundary of [+ 'existing observations only',+ 'keep that working setup',+ 'rather than undoing the repair or repeating the addition',+ 'Missing evidence is unknown',+ 'Unavailable original evidence can be reported as unavailable',+ 'do not recreate the incident',+ 'Do not rename production nodes, change VMIDs or machine IDs',+ 'delete/re-add connections, re-enrol agents, rotate tokens, restart services or clear History',+ 'Do not run live diagnostics, guest-agent probes, a new backup or a restore as an identity test',+ ]) {+ expect(text).toContain(boundary);+ }+ });+});diff --git a/scripts/tests/test_pve_backup_troubleshooting_docs.py b/scripts/tests/test_pve_backup_troubleshooting_docs.pyindex b871737f8f..1e8074245a 100644--- a/scripts/tests/test_pve_backup_troubleshooting_docs.py+++ b/scripts/tests/test_pve_backup_troubleshooting_docs.py@@ -20,6 +20,42 @@ def backup_section():class PVEBackupTroubleshootingDocsTest(unittest.TestCase):+ def test_multi_installation_help_is_mirrored_and_keeps_cosmetic_names_separate(self):+ for name in ("CONFIGURATION.md", "TROUBLESHOOTING.md", "PBS.md"):+ with self.subTest(guide=name):+ self.assertEqual((ROOT / "docs" / name).read_bytes(),+ (ROOT / "frontend-modern/public/docs" / name).read_bytes())+ config = (ROOT / "docs/CONFIGURATION.md").read_text()+ section = config.split("### Multiple Proxmox installations\n", 1)[1].split("\n### ", 1)[0]+ text = " ".join(section.split())+ self.assertIn("intended to monitor multiple Proxmox clusters and standalone nodes", text)+ self.assertIn("matching name or VMID alone does not identify the same resource", text)+ self.assertIn("repairs incorrect attribution", text)+ self.assertIn("TROUBLESHOOTING.md#monitoring-is-mixed-between-proxmox-installations", text)+ # Static evidence of existing multi-instance intent, not native recovery.+ helpers = (ROOT / "internal/monitoring/monitor_helpers.go").read_text()+ self.assertIn('fmt.Sprintf("%s:%s:%d", instanceName, node, vmid)', helpers)+ collector = (ROOT / "internal/monitoring/monitor_backups.go").read_text()+ self.assertIn("vm.Instance() == instanceName", collector)+ self.assertIn("vm.Instance == instanceName", collector)++ def test_cross_installation_comparison_uses_existing_visible_agent_evidence(self):+ trouble = (ROOT / "docs/TROUBLESHOOTING.md").read_text()+ section = trouble.split("#### Monitoring is mixed between Proxmox installations\n", 1)[1].split("\n#### ", 1)[0]+ text = " ".join(section.split())+ self.assertNotIn("```", section)+ for boundary in ("existing observations only", "One restored view does not establish",+ "Keep actual addresses, hostnames, connection IDs and machine identities private",+ "Do not share a full Agent Doctor report", "Do not rename production nodes",+ "re-enrol agents", "Do not run live diagnostics, guest-agent probes"):+ self.assertIn(boundary, text)+ model = (ROOT / "frontend-modern/src/components/Settings/infrastructureAgentUpdateCommandsModel.ts").read_text()+ for field in ("Connection: ${connection.id}", "Hostname: ${connection.agentIdentity.hostname}"):+ self.assertIn(field, model)+ page = (ROOT / "frontend-modern/src/components/Settings/InfrastructureAgentDoctorPage.tsx").read_text()+ for label in ("Identity evidence", "Last seen", "Healthy"):+ self.assertIn(label, page)+def test_shipped_help_matches_source(self):self.assertEqual((ROOT / "docs" / GUIDE).read_bytes(),(ROOT / "frontend-modern/public/docs" / GUIDE).read_bytes())