Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep Operational Trust upgrade checks read-only
pulse-triage[bot]e4a96c9302a4
diff --git a/docs/OPERATIONAL_TRUST.md b/docs/OPERATIONAL_TRUST.mdindex 6fc9e08d58..a80a775d20 100644--- a/docs/OPERATIONAL_TRUST.md+++ b/docs/OPERATIONAL_TRUST.md@@ -181,25 +181,67 @@ shape is now a local display adapter, not a writable source of truth.Before upgrading:-1. back up the Pulse data directory;-2. confirm that the v6 process can write its alert, notification, recovery, and- action database directories;-3. confirm supported clients can read additive JSON fields;-4. expose the metrics listener to a protected scraper if rollout telemetry is- required;-5. verify Pulse Pro entitlement connectivity before relying on action offers.--After upgrading:--1. confirm the Patrol navigation count matches the active queue;-2. inspect an active item through its deepest evidence and protection detail;-3. acknowledge and unacknowledge a test item;-4. verify a bounded suppression returns to active attention;-5. confirm stale collection remains visible rather than resolving;-6. exercise notification retry/dead-letter monitoring;-7. if using Pulse Pro actions, complete a review/approve/run/verify journey.+1. Record the running version, edition and deployment configuration. Keep a+ consistent private backup of the actual data directory, matching encryption+ and audit signing keys, and any external stores; use the+ [full-state recovery guidance](MIGRATION.md#full-state-recovery), not a copy+ of a live database file or a configuration-only export.+2. Check the existing service account and persistent mounts through the normal+ deployment configuration. Do not broaden permissions or replace keys to+ make an upgrade check pass.+3. Keep independent monitoring and alert coverage available during the update.+ For affected freeze-enabled Proxmox backups, preserve the separate+ [guest-safety precaution](VM_DISK_MONITORING.md#backup-safety).+4. Keep the previous binary or image and its matching data backup until+ recovery is checked. Reverting a binary does not reverse data migrations;+ follow the deployment's [update and rollback guidance](AUTO_UPDATE.md#rollback).++### Read-only checks after upgrading++Use an existing authenticated browser session and ordinary collection. These+checks inspect existing evidence; they do not require a new incident or a+change to a workload, notification queue or alert lifecycle.++1. Check the running server version and edition, then the normal connection+ health and observation times. Compare the same connection, resource and time+ range before and after the upgrade; reused names or VMIDs are not identity.+2. Compare the Patrol navigation count with the active queue using the same+ organisation, access scope and filters. Open an existing item, if present,+ and inspect its resource, lifecycle, evidence and protection detail. An+ empty queue does not establish complete or healthy collection.+3. Confirm that retained evidence still shows its observation time and any+ stale, denied, partial or unknown state. Do not disconnect a collector or+ induce a fault to obtain a test case. A successful task or action is not+ fresh detector evidence of recovery, guest thaw or a tested restore.+4. Inspect **Recent delivery activity** in Alert History and existing failed+ delivery details, if any. Compare the original alert, queued attempt and+ recipient's actual message separately; Test success does not establish+ ordinary, grouped or resolved delivery. Use the+ [notification checks](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing)+ for the actual symptom.+5. If using Pro actions, inspect existing execution and verification records+ separately. A successful execution does not establish verified recovery;+ an unavailable action offer is not by itself a monitoring failure.++**Do not acknowledge, suppress, retry, dismiss or clear records, send a Test,+or approve or run an action merely to validate an upgrade.** Retry can send a+real notification, suppression changes active attention, and an offered Docker+restart changes a workload. These controls remain available for intentional+operator work, not passive diagnostics. If a check needs an event that has not+occurred, record it as unverified rather than manufacture one. Exercise those+paths only in an isolated non-production fixture with synthetic destinations+and disposable workloads, never against live resources.++Keep any existing metrics listener private; reading through an already+protected scraper is optional, not a reason to expose a new listener or widen+network access. Keep authentication, TLS verification and least-privilege+collection unchanged. Preserve errors and existing delivery/action records;+share only the affected view, version, time and locally redacted explanation,+not credentials, full inventories, notification payloads or Debug exports. See+[safe issue reporting](TROUBLESHOOTING.md#-getting-help).If a migration fails, stop the upgraded process, preserve the data directory-and logs, and restore the prior release with the pre-upgrade data backup. Do-not delete lifecycle, evidence, notification, recovery, or action records to-force startup.+and private logs, and use the deployment's recovery procedure with the prior+release and its matching consistent pre-upgrade data backup. A failed read or+an empty queue alone is not a reason to restore data. Do not delete lifecycle,+evidence, notification, recovery or action records to force startup.diff --git a/frontend-modern/public/docs/OPERATIONAL_TRUST.md b/frontend-modern/public/docs/OPERATIONAL_TRUST.mdindex 6fc9e08d58..a80a775d20 100644--- a/frontend-modern/public/docs/OPERATIONAL_TRUST.md+++ b/frontend-modern/public/docs/OPERATIONAL_TRUST.md@@ -181,25 +181,67 @@ shape is now a local display adapter, not a writable source of truth.Before upgrading:-1. back up the Pulse data directory;-2. confirm that the v6 process can write its alert, notification, recovery, and- action database directories;-3. confirm supported clients can read additive JSON fields;-4. expose the metrics listener to a protected scraper if rollout telemetry is- required;-5. verify Pulse Pro entitlement connectivity before relying on action offers.--After upgrading:--1. confirm the Patrol navigation count matches the active queue;-2. inspect an active item through its deepest evidence and protection detail;-3. acknowledge and unacknowledge a test item;-4. verify a bounded suppression returns to active attention;-5. confirm stale collection remains visible rather than resolving;-6. exercise notification retry/dead-letter monitoring;-7. if using Pulse Pro actions, complete a review/approve/run/verify journey.+1. Record the running version, edition and deployment configuration. Keep a+ consistent private backup of the actual data directory, matching encryption+ and audit signing keys, and any external stores; use the+ [full-state recovery guidance](MIGRATION.md#full-state-recovery), not a copy+ of a live database file or a configuration-only export.+2. Check the existing service account and persistent mounts through the normal+ deployment configuration. Do not broaden permissions or replace keys to+ make an upgrade check pass.+3. Keep independent monitoring and alert coverage available during the update.+ For affected freeze-enabled Proxmox backups, preserve the separate+ [guest-safety precaution](VM_DISK_MONITORING.md#backup-safety).+4. Keep the previous binary or image and its matching data backup until+ recovery is checked. Reverting a binary does not reverse data migrations;+ follow the deployment's [update and rollback guidance](AUTO_UPDATE.md#rollback).++### Read-only checks after upgrading++Use an existing authenticated browser session and ordinary collection. These+checks inspect existing evidence; they do not require a new incident or a+change to a workload, notification queue or alert lifecycle.++1. Check the running server version and edition, then the normal connection+ health and observation times. Compare the same connection, resource and time+ range before and after the upgrade; reused names or VMIDs are not identity.+2. Compare the Patrol navigation count with the active queue using the same+ organisation, access scope and filters. Open an existing item, if present,+ and inspect its resource, lifecycle, evidence and protection detail. An+ empty queue does not establish complete or healthy collection.+3. Confirm that retained evidence still shows its observation time and any+ stale, denied, partial or unknown state. Do not disconnect a collector or+ induce a fault to obtain a test case. A successful task or action is not+ fresh detector evidence of recovery, guest thaw or a tested restore.+4. Inspect **Recent delivery activity** in Alert History and existing failed+ delivery details, if any. Compare the original alert, queued attempt and+ recipient's actual message separately; Test success does not establish+ ordinary, grouped or resolved delivery. Use the+ [notification checks](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing)+ for the actual symptom.+5. If using Pro actions, inspect existing execution and verification records+ separately. A successful execution does not establish verified recovery;+ an unavailable action offer is not by itself a monitoring failure.++**Do not acknowledge, suppress, retry, dismiss or clear records, send a Test,+or approve or run an action merely to validate an upgrade.** Retry can send a+real notification, suppression changes active attention, and an offered Docker+restart changes a workload. These controls remain available for intentional+operator work, not passive diagnostics. If a check needs an event that has not+occurred, record it as unverified rather than manufacture one. Exercise those+paths only in an isolated non-production fixture with synthetic destinations+and disposable workloads, never against live resources.++Keep any existing metrics listener private; reading through an already+protected scraper is optional, not a reason to expose a new listener or widen+network access. Keep authentication, TLS verification and least-privilege+collection unchanged. Preserve errors and existing delivery/action records;+share only the affected view, version, time and locally redacted explanation,+not credentials, full inventories, notification payloads or Debug exports. See+[safe issue reporting](TROUBLESHOOTING.md#-getting-help).If a migration fails, stop the upgraded process, preserve the data directory-and logs, and restore the prior release with the pre-upgrade data backup. Do-not delete lifecycle, evidence, notification, recovery, or action records to-force startup.+and private logs, and use the deployment's recovery procedure with the prior+release and its matching consistent pre-upgrade data backup. A failed read or+an empty queue alone is not a reason to restore data. Do not delete lifecycle,+evidence, notification, recovery or action records to force startup.diff --git a/frontend-modern/src/utils/__tests__/operationalUpgradeHelp.test.ts b/frontend-modern/src/utils/__tests__/operationalUpgradeHelp.test.tsnew file mode 100644index 0000000000..288c4663f0--- /dev/null+++ b/frontend-modern/src/utils/__tests__/operationalUpgradeHelp.test.ts@@ -0,0 +1,58 @@+import { readFileSync } from 'node:fs';+import { resolve } from 'node:path';+import { describe, expect, it } from 'vitest';+import { renderDocMarkdown } from '@/features/docs/docMarkdown';++const read = (path: string) => readFileSync(resolve(process.cwd(), '..', path), 'utf8');++const article = (name: string) => {+ const element = document.createElement('article');+ element.innerHTML = renderDocMarkdown(read(`docs/${name}.md`), name);+ return element;+};++describe('Operational Trust upgrade help', () => {+ it('ships the same recovery and passive-check guidance in the in-app document', () => {+ expect(read('frontend-modern/public/docs/OPERATIONAL_TRUST.md')).toBe(+ read('docs/OPERATIONAL_TRUST.md'),+ );+ const guide = article('OPERATIONAL_TRUST');+ expect(guide.querySelector('#read-only-checks-after-upgrading')?.textContent).toBe(+ 'Read-only checks after upgrading',+ );+ const warning = [...guide.querySelectorAll('strong')].find((element) =>+ element.textContent?.includes('merely to validate an upgrade'),+ );+ expect(warning?.textContent?.replace(/\s+/g, ' ')).toBe(+ 'Do not acknowledge, suppress, retry, dismiss or clear records, send a Test, or approve or run an action merely to validate an upgrade.',+ );+ expect(guide.textContent?.replace(/\s+/g, ' ')).toContain(+ 'Retry can send a real notification, suppression changes active attention, and an offered Docker restart changes a workload.',+ );+ });++ it('keeps every recovery, safety and reporting link on an existing in-app guide and anchor', () => {+ const guide = article('OPERATIONAL_TRUST');+ for (const [name, fragment, label] of [+ ['MIGRATION', 'full-state-recovery', 'full-state recovery guidance'],+ ['VM_DISK_MONITORING', 'backup-safety', 'guest-safety precaution'],+ ['AUTO_UPDATE', 'rollback', 'update and rollback guidance'],+ ['TROUBLESHOOTING', 'test-succeeds-but-real-alerts-are-missing', 'notification checks'],+ ['TROUBLESHOOTING', '-getting-help', 'safe issue reporting'],+ ]) {+ const link = guide.querySelector(`a[href="/docs/${name}#${fragment}"]`);+ expect(link?.textContent).toBe(label);+ expect(link?.hasAttribute('data-doc-link')).toBe(true);+ expect(article(name).querySelector(`[id="${fragment}"]`)).not.toBeNull();+ }+ });++ it('does not turn an unobserved event or a failed read into an upgrade-success claim', () => {+ const text = article('OPERATIONAL_TRUST').textContent!.replace(/\s+/g, ' ');+ expect(text).toContain('An empty queue does not establish complete or healthy collection');+ expect(text).toContain('record it as unverified rather than manufacture one');+ expect(text).toContain('A failed read or an empty queue alone is not a reason to restore data');+ expect(text).not.toContain('exercise notification retry/dead-letter monitoring');+ expect(text).not.toContain('complete a review/approve/run/verify journey');+ });+});diff --git a/scripts/tests/test_production_rollout_docs.py b/scripts/tests/test_production_rollout_docs.pyindex 367f807e7a..dcb77c546c 100644--- a/scripts/tests/test_production_rollout_docs.py+++ b/scripts/tests/test_production_rollout_docs.py@@ -11,6 +11,7 @@ import unittestROOT = Path(__file__).resolve().parents[2]DOC = ROOT / "docs/PRODUCTION_SECURITY.md"+TRUST_DOC = ROOT / "docs/OPERATIONAL_TRUST.md"def section(heading):@@ -112,6 +113,55 @@ class ProductionRolloutDocsTest(unittest.TestCase):self.assertIn(boundary, text)self.assertNotRegex(text, r"(?:--insecure|curl[^\n]*\||--token\s)")+ def test_operational_upgrade_checks_are_observational_not_action_probes(self):+ text = " ".join(TRUST_DOC.read_text().split("### Read-only checks after upgrading\n", 1)[1].split())+ for boundary in ("existing authenticated browser session", "ordinary collection",+ "same organisation, access scope and filters",+ "An empty queue does not establish complete or healthy collection",+ "Do not disconnect a collector or induce a fault",+ "Recent delivery activity", "recipient's actual message separately",+ "Test success does not establish ordinary, grouped or resolved delivery",+ "execution and verification records separately",+ "not by itself a monitoring failure", "Retry can send a real notification",+ "suppression changes active attention", "restart changes a workload",+ "Do not acknowledge, suppress, retry, dismiss or clear records, send a Test",+ "or approve or run an action merely to validate an upgrade",+ "unverified rather than manufacture one",+ "isolated non-production fixture", "synthetic destinations",+ "disposable workloads", "never against live resources"):+ with self.subTest(boundary=boundary):+ self.assertIn(boundary, text)+ for old_instruction in ("acknowledge and unacknowledge a test item",+ "verify a bounded suppression returns to active attention",+ "exercise notification retry/dead-letter monitoring",+ "complete a review/approve/run/verify journey"):+ self.assertNotIn(old_instruction, text)+ # The original advice crossed real mutation boundaries, not just UI+ # demonstrations. Keep that reason grounded in the current owners.+ queue = (ROOT / "internal/api/alerting/notification_queue.go").read_text()+ self.assertIn("queue.ScheduleRetry(request.ID, 0)", queue)+ actions = (ROOT / "internal/api/actions.go").read_text()+ self.assertIn("h.ActionLifecycle().Execute(", actions)++ def test_operational_upgrade_preserves_recovery_and_access_boundaries(self):+ text = " ".join(TRUST_DOC.read_text().split("## Upgrade and compatibility\n", 1)[1].split())+ for boundary in ("version, edition and deployment configuration", "consistent private backup",+ "matching encryption and audit signing keys", "external stores",+ "not a copy of a live database file or a configuration-only export",+ "Do not broaden permissions or replace keys", "independent monitoring and alert coverage",+ "MIGRATION.md#full-state-recovery", "VM_DISK_MONITORING.md#backup-safety",+ "Reverting a binary does not reverse data migrations", "AUTO_UPDATE.md#rollback",+ "metrics listener private", "not a reason to expose a new listener",+ "authentication, TLS verification and least-privilege collection unchanged",+ "Preserve errors and existing delivery/action records", "locally redacted explanation",+ "not credentials, full inventories, notification payloads or Debug exports",+ "A failed read or an empty queue alone is not a reason to restore data",+ "Do not delete lifecycle, evidence, notification, recovery or action records"):+ with self.subTest(boundary=boundary):+ self.assertIn(boundary, text)+ self.assertEqual(TRUST_DOC.read_bytes(),+ (ROOT / "frontend-modern/public/docs/OPERATIONAL_TRUST.md").read_bytes())+if __name__ == "__main__":unittest.main()