Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep migration trials isolated until monitoring handover
pulse-triage[bot]16e4b6161534
diff --git a/docs/MIGRATION.md b/docs/MIGRATION.mdindex fac9166194..03c13fdaa6 100644--- a/docs/MIGRATION.md+++ b/docs/MIGRATION.md@@ -44,9 +44,15 @@ an import error or skipped import does not confirm a successful transfer.### Full-state recoveryIf you need the same installation's history and excluded state, retain a-consistent filesystem/volume backup as well. Stop Pulse before taking that-backup; monitoring and alert delivery pause while it is stopped. Preserve every-effective data path, the matching `.encryption.key`, audit data and signing key,+consistent filesystem/volume backup as well. Before stopping Pulse, confirm+that all state survives the stop: a container's writable layer or Kubernetes+`emptyDir` is not retained storage when its container or pod is replaced. If+persistence is unknown, preserve the state through your deployment's recovery+procedure first; do not delete or scale down the server to find out. See the+[Kubernetes storage and controller checks](KUBERNETES.md#proxmox-backup-safety).++Stop Pulse before taking the consistent backup; monitoring and alert delivery+pause while it is stopped. Preserve every effective data path, the matching `.encryption.key`, audit data and signing key,environment files, external certificates, binaries and service/containerconfiguration together. Custom paths and organisation directories matter.Copying a live database file alone can omit its sidecar files.@@ -104,8 +110,12 @@ configuration import as proof of a complete multi-organisation migration.is complete.2. Install Pulse at the destination using the [installation guide](INSTALL.md).Configure destination-local administrator access before importing. Keep the- destination isolated from agents and notification destinations during the- trial so two servers do not send duplicate alerts or remote actions.+ destination isolated from **monitored provider APIs, agents, notification+ destinations and remote-action targets before its first start or import**.+ Imported credentials can start collection immediately, including QEMU Guest+ Agent requests through Proxmox. API-only is not a passive trial. Keep this+ isolation for a full-state recovery trial too; give it separate restored+ storage, never the source's live data volume.3. If the destination already has useful settings, export them first. Open**Settings → System → Recovery → Restore Configuration**, select the fileand enter its passphrase. Import replaces the included configuration and@@ -121,6 +131,37 @@ a password, API token or session cookie.## Cut over and verify+### Hand over monitoring once++Validate destination-local sign-in and the imported settings while the trial+is isolated. **Fresh provider readings belong after the handover below**, not+while both servers can poll the same guests. Hiding rows, disabling alerts or+turning off backup polling does not stop guest-agent collection.++1. Plan an outage with independent monitoring and alert coverage. Do not cut+ over during a backup, freeze/thaw or an unresponsive-guest incident; use+ [Backup safety](VM_DISK_MONITORING.md#backup-safety) if those states are+ uncertain. An OK backup alone does not establish guest recovery.+2. Using the deployment's supported controls, stop **every source Pulse server**+ that monitors the transferred resources, not just its agents or browser.+ First confirm retained storage as above, record prior service/replica and+ automatic-update states, and prevent an updater or controller from restarting+ it. Require a confirmed normal shutdown; a deleted pod, paused log display,+ failed stop or unknown state is not a completed handover. Do not force-kill+ or delete state to satisfy this check.+3. A stopped server does not cancel a guest-agent request already issued.+ Confirm existing guest/backup operations have ended normally through+ established platform/workload checks, not new guest-agent probes or an+ arbitrary delay. If this is unknown, keep the destination isolated and+ resolve it before enabling collection.+4. Only then allow the destination to reach the intended providers and retarget+ agents. Keep the source stopped and its automation paused during the checks+ below. For full-state recovery, reconcile retained pending notifications and+ actions before allowing delivery or execution; a configuration-only import+ does not contain those queues.++### Check the destination+- **Login and permissions:** verify destination-local administrator access andany imported SSO provider, group mappings and callback address. Re-createcustom roles and supply external CA/certificate files if needed. Environment@@ -146,9 +187,15 @@ a password, API token or session cookie.Relay is no longer sold; do not enable it as a new migration route.Retire the old installation only after these checks succeed. Keep the private,-consistent pre-move backup for recovery. If validation fails, stop the trial-destination before returning traffic to the intact source; check agent URLs-and avoid two active writers. There is no guaranteed five-minute recovery time.+consistent pre-move backup for recovery. If validation fails, keep the source+intact and isolate the destination from providers, agents, notifications and+action targets. Confirm the destination is+stopped and cannot be automatically restarted before returning traffic or+restoring the source's recorded prior-active controls. If shutdown or in-flight+operations are uncertain, do not enable a second poller. Preserve the failed+destination for diagnosis; do not copy its changed data over the source or+re-import merely to retry. Check agent URLs and avoid two active writers.+There is no guaranteed five-minute recovery time.## Troubleshootingdiff --git a/frontend-modern/public/docs/MIGRATION.md b/frontend-modern/public/docs/MIGRATION.mdindex fac9166194..03c13fdaa6 100644--- a/frontend-modern/public/docs/MIGRATION.md+++ b/frontend-modern/public/docs/MIGRATION.md@@ -44,9 +44,15 @@ an import error or skipped import does not confirm a successful transfer.### Full-state recoveryIf you need the same installation's history and excluded state, retain a-consistent filesystem/volume backup as well. Stop Pulse before taking that-backup; monitoring and alert delivery pause while it is stopped. Preserve every-effective data path, the matching `.encryption.key`, audit data and signing key,+consistent filesystem/volume backup as well. Before stopping Pulse, confirm+that all state survives the stop: a container's writable layer or Kubernetes+`emptyDir` is not retained storage when its container or pod is replaced. If+persistence is unknown, preserve the state through your deployment's recovery+procedure first; do not delete or scale down the server to find out. See the+[Kubernetes storage and controller checks](KUBERNETES.md#proxmox-backup-safety).++Stop Pulse before taking the consistent backup; monitoring and alert delivery+pause while it is stopped. Preserve every effective data path, the matching `.encryption.key`, audit data and signing key,environment files, external certificates, binaries and service/containerconfiguration together. Custom paths and organisation directories matter.Copying a live database file alone can omit its sidecar files.@@ -104,8 +110,12 @@ configuration import as proof of a complete multi-organisation migration.is complete.2. Install Pulse at the destination using the [installation guide](INSTALL.md).Configure destination-local administrator access before importing. Keep the- destination isolated from agents and notification destinations during the- trial so two servers do not send duplicate alerts or remote actions.+ destination isolated from **monitored provider APIs, agents, notification+ destinations and remote-action targets before its first start or import**.+ Imported credentials can start collection immediately, including QEMU Guest+ Agent requests through Proxmox. API-only is not a passive trial. Keep this+ isolation for a full-state recovery trial too; give it separate restored+ storage, never the source's live data volume.3. If the destination already has useful settings, export them first. Open**Settings → System → Recovery → Restore Configuration**, select the fileand enter its passphrase. Import replaces the included configuration and@@ -121,6 +131,37 @@ a password, API token or session cookie.## Cut over and verify+### Hand over monitoring once++Validate destination-local sign-in and the imported settings while the trial+is isolated. **Fresh provider readings belong after the handover below**, not+while both servers can poll the same guests. Hiding rows, disabling alerts or+turning off backup polling does not stop guest-agent collection.++1. Plan an outage with independent monitoring and alert coverage. Do not cut+ over during a backup, freeze/thaw or an unresponsive-guest incident; use+ [Backup safety](VM_DISK_MONITORING.md#backup-safety) if those states are+ uncertain. An OK backup alone does not establish guest recovery.+2. Using the deployment's supported controls, stop **every source Pulse server**+ that monitors the transferred resources, not just its agents or browser.+ First confirm retained storage as above, record prior service/replica and+ automatic-update states, and prevent an updater or controller from restarting+ it. Require a confirmed normal shutdown; a deleted pod, paused log display,+ failed stop or unknown state is not a completed handover. Do not force-kill+ or delete state to satisfy this check.+3. A stopped server does not cancel a guest-agent request already issued.+ Confirm existing guest/backup operations have ended normally through+ established platform/workload checks, not new guest-agent probes or an+ arbitrary delay. If this is unknown, keep the destination isolated and+ resolve it before enabling collection.+4. Only then allow the destination to reach the intended providers and retarget+ agents. Keep the source stopped and its automation paused during the checks+ below. For full-state recovery, reconcile retained pending notifications and+ actions before allowing delivery or execution; a configuration-only import+ does not contain those queues.++### Check the destination+- **Login and permissions:** verify destination-local administrator access andany imported SSO provider, group mappings and callback address. Re-createcustom roles and supply external CA/certificate files if needed. Environment@@ -146,9 +187,15 @@ a password, API token or session cookie.Relay is no longer sold; do not enable it as a new migration route.Retire the old installation only after these checks succeed. Keep the private,-consistent pre-move backup for recovery. If validation fails, stop the trial-destination before returning traffic to the intact source; check agent URLs-and avoid two active writers. There is no guaranteed five-minute recovery time.+consistent pre-move backup for recovery. If validation fails, keep the source+intact and isolate the destination from providers, agents, notifications and+action targets. Confirm the destination is+stopped and cannot be automatically restarted before returning traffic or+restoring the source's recorded prior-active controls. If shutdown or in-flight+operations are uncertain, do not enable a second poller. Preserve the failed+destination for diagnosis; do not copy its changed data over the source or+re-import merely to retry. Check agent URLs and avoid two active writers.+There is no guaranteed five-minute recovery time.## Troubleshootingdiff --git a/frontend-modern/src/features/docs/__tests__/migrationHandoverSafetyDocs.test.ts b/frontend-modern/src/features/docs/__tests__/migrationHandoverSafetyDocs.test.tsnew file mode 100644index 0000000000..b6ee74aae3--- /dev/null+++ b/frontend-modern/src/features/docs/__tests__/migrationHandoverSafetyDocs.test.ts@@ -0,0 +1,110 @@+import { readFileSync } from 'node:fs';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { describe, expect, it } from 'vitest';+import { renderDocMarkdown } from '../docMarkdown';++const frontendRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');+const repoRoot = path.resolve(frontendRoot, '..');+const readDoc = (name: string) => readFileSync(path.join(repoRoot, 'docs', `${name}.md`), 'utf8');++function articleFor(name: string) {+ const article = document.createElement('article');+ article.innerHTML = renderDocMarkdown(readDoc(name), name);+ return article;+}++const textOf = (element: Element) => element.textContent!.replace(/\s+/g, ' ');++function handover() {+ const article = articleFor('MIGRATION');+ const heading = article.querySelector('#hand-over-monitoring-once');+ expect(heading).not.toBeNull();+ const section = document.createElement('section');+ for (+ let node = heading!.nextElementSibling;+ node && !/^H[23]$/.test(node.tagName);+ node = node.nextElementSibling+ ) {+ section.append(node.cloneNode(true));+ }+ return section;+}++describe('Migration monitoring handover safety', () => {+ it('isolates provider access before starting either kind of recovery trial', () => {+ const article = articleFor('MIGRATION');+ const step = article.querySelector(+ '#transfer-the-included-configuration + ol > li:nth-child(2)',+ );+ expect(step).not.toBeNull();+ const text = textOf(step!);+ expect(text).toContain('monitored provider APIs, agents, notification destinations');+ expect(text).toContain('remote-action targets before its first start or import');+ expect(text).toContain('QEMU Guest Agent requests through Proxmox');+ expect(text).toContain('API-only is not a passive trial');+ expect(text).toContain('full-state recovery trial too');+ expect(text).toContain("never the source's live data volume");+ });++ it('requires retained storage before a full-state backup stop', () => {+ const article = articleFor('MIGRATION');+ const text = textOf(article);+ expect(text).toContain('Before stopping Pulse, confirm that all state survives the stop');+ expect(text).toContain('emptyDir');+ expect(text).toContain('If persistence is unknown, preserve the state');+ expect(text).toContain('do not delete or scale down the server to find out');+ expect(text.indexOf('Before stopping Pulse')).toBeLessThan(+ text.indexOf('Stop Pulse before taking the consistent backup'),+ );+ });++ it('places the four-step handover before fresh destination checks', () => {+ const article = articleFor('MIGRATION');+ const headings = [...article.querySelectorAll('h3')].map((heading) => heading.id);+ expect(headings.indexOf('hand-over-monitoring-once')).toBeLessThan(+ headings.indexOf('check-the-destination'),+ );+ const section = handover();+ expect(textOf(section)).toContain('Fresh provider readings belong after the handover');+ expect(textOf(section)).toContain('turning off backup polling does not stop');+ const steps = [...section.querySelectorAll('ol > li')].map(textOf);+ expect(steps).toHaveLength(4);+ expect(steps[0]).toContain('Do not cut over during a backup, freeze/thaw');+ expect(steps[1]).toContain('every source Pulse server');+ expect(steps[1]).toContain('prevent an updater or controller from restarting it');+ expect(steps[1]).toContain('Require a confirmed normal shutdown');+ expect(steps[2]).toContain('does not cancel a guest-agent request already issued');+ expect(steps[2]).toContain('not new guest-agent probes or an arbitrary delay');+ expect(steps[2]).toContain('keep the destination isolated');+ expect(steps[3]).toContain('Only then allow the destination');+ expect(steps[3]).toContain('reconcile retained pending notifications and actions');+ expect(section.querySelector('pre')).toBeNull();+ });++ it('keeps rollback single-owner and preserves both installations', () => {+ const text = textOf(articleFor('MIGRATION'));+ expect(text).toContain(+ 'Confirm the destination is stopped and cannot be automatically restarted',+ );+ expect(text).toContain("restoring the source's recorded prior-active controls");+ expect(text).toContain('do not enable a second poller');+ expect(text).toContain('do not copy its changed data over the source');+ expect(text).toContain('re-import merely to retry');+ });++ it('routes storage and incident safeguards to shipped, existing headings', () => {+ const article = articleFor('MIGRATION');+ for (const [name, fragment] of [+ ['KUBERNETES', 'proxmox-backup-safety'],+ ['VM_DISK_MONITORING', 'backup-safety'],+ ]) {+ const link = article.querySelector(`a[href="/docs/${name}#${fragment}"]`);+ expect(link?.hasAttribute('data-doc-link')).toBe(true);+ expect(articleFor(name).querySelector(`#${fragment}`)).not.toBeNull();+ }+ expect(readFileSync(path.join(frontendRoot, 'public/docs/MIGRATION.md'), 'utf8')).toBe(+ readDoc('MIGRATION'),+ );+ });+});