Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep recovery help on the full guest safety precaution
pulse-triage[bot]4b08b353b885
diff --git a/docs/RECOVERY.md b/docs/RECOVERY.mdindex 2fc26450f2..dd9d0cb144 100644--- a/docs/RECOVERY.md+++ b/docs/RECOVERY.md@@ -37,13 +37,36 @@ Keep these observations separate:| Protected posture | Current, linked evidence meets Pulse's protection policy | A recovery guarantee or permission to discard other backups |**A backup task's OK status is not confirmation that the guest has thawed.**-For freeze-enabled Proxmox backups, stop Pulse before the backup and restart it-only after independently confirming guest thaw. Pulse monitoring and alert-delivery are unavailable while it is stopped. Do not send additional-guest-agent commands during freeze/thaw to diagnose the problem, or disable-filesystem freezing merely to make a task look successful. Follow the-[backup safety precaution](VM_DISK_MONITORING.md#backup-safety); it is not a-claim that Pulse has repaired or reproduced a native thaw failure.+For an affected installation, use the+[backup safety precaution](VM_DISK_MONITORING.md#backup-safety) **before a planned+freeze-enabled Proxmox backup**. It covers the actual server deployment and+automatic updaters, not just a guest agent. A planned pause is not an incident+recovery procedure, and stopping Pulse does not cancel a guest-agent request+already issued. If an existing operation's state is unknown, do not start a+backup on the strength of a stopped service or an elapsed waiting period.++Keep Pulse stopped until the backup has ended and **independent post-backup+checks confirm all three**:++1. Guest thaw.+2. Fresh successful workload writes to **every filesystem covered by the backup**.+3. Workload liveness.++Use the workload's established safe checks, independent of Pulse and the QEMU+Guest Agent. An OK task, an absent lock, a console connection, a successful read+or a write to only the OS disk is not enough. If any check fails or is+unavailable, leave Pulse and its automatic updater paused and use the+guest/platform's recovery procedure. Do not force writes, repeat a backup or+send guest-agent probes to fill the gap.++After all checks pass, restore **only services and timers that were active+before the pause**. Unknown pre-pause states are not permission to start them.+An updater must not restart a previously inactive server; follow the+deployment-specific restoration steps in the precaution. **Pulse monitoring+and alert delivery are unavailable while it is stopped**; arrange independent+outage coverage. Do not disable filesystem freezing to make a task look+successful. This precaution is not proof of a repaired or reproduced native+thaw failure.Before relying on an artifact, confirm its provider, repository, workloadidentity and time in the provider's own tools. For recovery assurance, test a@@ -117,8 +140,11 @@ point separately rather than treating current activity as protection.If the guest stopped responding during backup, preserve the task's warning orthaw error and the observations already available. Do not repeat the backup,reset the guest, or run active guest-agent probes just to reproduce it. Use-the provider's recovery procedure and a separate trusted guest console to-assess guest liveness and thaw, not Pulse's backup badge.+the provider's recovery procedure; a separate trusted guest console can help+assess the guest, but a console connection alone does not clear the precaution.+Keep monitoring paused until the independent post-backup checks above confirm+thaw, writes to every covered filesystem and workload liveness, not just until+Pulse's backup badge looks healthy.When reporting missing records, share the failing view, running Pulse version,affected provider, time range, expected reading and relevant redacted error.diff --git a/frontend-modern/public/docs/RECOVERY.md b/frontend-modern/public/docs/RECOVERY.mdindex 2fc26450f2..dd9d0cb144 100644--- a/frontend-modern/public/docs/RECOVERY.md+++ b/frontend-modern/public/docs/RECOVERY.md@@ -37,13 +37,36 @@ Keep these observations separate:| Protected posture | Current, linked evidence meets Pulse's protection policy | A recovery guarantee or permission to discard other backups |**A backup task's OK status is not confirmation that the guest has thawed.**-For freeze-enabled Proxmox backups, stop Pulse before the backup and restart it-only after independently confirming guest thaw. Pulse monitoring and alert-delivery are unavailable while it is stopped. Do not send additional-guest-agent commands during freeze/thaw to diagnose the problem, or disable-filesystem freezing merely to make a task look successful. Follow the-[backup safety precaution](VM_DISK_MONITORING.md#backup-safety); it is not a-claim that Pulse has repaired or reproduced a native thaw failure.+For an affected installation, use the+[backup safety precaution](VM_DISK_MONITORING.md#backup-safety) **before a planned+freeze-enabled Proxmox backup**. It covers the actual server deployment and+automatic updaters, not just a guest agent. A planned pause is not an incident+recovery procedure, and stopping Pulse does not cancel a guest-agent request+already issued. If an existing operation's state is unknown, do not start a+backup on the strength of a stopped service or an elapsed waiting period.++Keep Pulse stopped until the backup has ended and **independent post-backup+checks confirm all three**:++1. Guest thaw.+2. Fresh successful workload writes to **every filesystem covered by the backup**.+3. Workload liveness.++Use the workload's established safe checks, independent of Pulse and the QEMU+Guest Agent. An OK task, an absent lock, a console connection, a successful read+or a write to only the OS disk is not enough. If any check fails or is+unavailable, leave Pulse and its automatic updater paused and use the+guest/platform's recovery procedure. Do not force writes, repeat a backup or+send guest-agent probes to fill the gap.++After all checks pass, restore **only services and timers that were active+before the pause**. Unknown pre-pause states are not permission to start them.+An updater must not restart a previously inactive server; follow the+deployment-specific restoration steps in the precaution. **Pulse monitoring+and alert delivery are unavailable while it is stopped**; arrange independent+outage coverage. Do not disable filesystem freezing to make a task look+successful. This precaution is not proof of a repaired or reproduced native+thaw failure.Before relying on an artifact, confirm its provider, repository, workloadidentity and time in the provider's own tools. For recovery assurance, test a@@ -117,8 +140,11 @@ point separately rather than treating current activity as protection.If the guest stopped responding during backup, preserve the task's warning orthaw error and the observations already available. Do not repeat the backup,reset the guest, or run active guest-agent probes just to reproduce it. Use-the provider's recovery procedure and a separate trusted guest console to-assess guest liveness and thaw, not Pulse's backup badge.+the provider's recovery procedure; a separate trusted guest console can help+assess the guest, but a console connection alone does not clear the precaution.+Keep monitoring paused until the independent post-backup checks above confirm+thaw, writes to every covered filesystem and workload liveness, not just until+Pulse's backup badge looks healthy.When reporting missing records, share the failing view, running Pulse version,affected provider, time range, expected reading and relevant redacted error.diff --git a/frontend-modern/src/utils/__tests__/recoveryHelp.test.ts b/frontend-modern/src/utils/__tests__/recoveryHelp.test.tsindex 577dba9042..52c95f653f 100644--- a/frontend-modern/src/utils/__tests__/recoveryHelp.test.ts+++ b/frontend-modern/src/utils/__tests__/recoveryHelp.test.ts@@ -8,14 +8,21 @@ import { renderDocMarkdown } from '@/features/docs/docMarkdown';const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');const read = (relative: string): string => readFileSync(path.join(repoRoot, relative), 'utf8');const guide = read('docs/RECOVERY.md');+const safetyText = () => {+ const article = document.createElement('article');+ const section = guide+ .split("**A backup task's OK status")[1]+ .split('Before relying on an artifact')[0];+ article.innerHTML = renderDocMarkdown(`**A backup task's OK status${section}`, 'RECOVERY');+ return article.textContent?.replace(/\s+/g, ' ');+};describe('recovery help', () => {it('separates backup evidence from thaw and tested recovery in the shipped guide', () => {expect(read('frontend-modern/public/docs/RECOVERY.md')).toBe(guide);expect(guide).toContain('Pulse does not restore workloads from these views.');expect(guide).toContain('OK status is not confirmation that the guest has thawed.');- expect(guide).toContain('only after independently confirming guest thaw');- expect(guide).toContain('Pulse monitoring and alert\ndelivery are unavailable');+ expect(safetyText()).toContain('Pulse monitoring and alert delivery are unavailable');expect(guide).toContain('restore to an isolated destination');expect(guide).toContain('Do not overwrite the live\nworkload');expect(guide).toContain('an omitted value supplies no verification result');@@ -23,6 +30,75 @@ describe('recovery help', () => {expect(guide).not.toContain('What can I actually recover?');});+ it('keeps the planned pause distinct from incident recovery and in-flight completion', () => {+ expect(safetyText()).toContain('For an affected installation');+ expect(safetyText()).toContain('before a planned freeze-enabled Proxmox backup');+ expect(safetyText()).toContain(+ 'actual server deployment and automatic updaters, not just a guest agent',+ );+ expect(safetyText()).toContain('A planned pause is not an incident recovery procedure');+ expect(safetyText()).toContain(+ 'stopping Pulse does not cancel a guest-agent request already issued',+ );+ expect(safetyText()).toContain(+ 'do not start a backup on the strength of a stopped service or an elapsed waiting period',+ );+ });++ it('requires all three independent post-backup checks, including writes to every covered filesystem', () => {+ const article = document.createElement('article');+ article.innerHTML = renderDocMarkdown(guide, 'RECOVERY');+ const checks = [...article.querySelector('ol')!.querySelectorAll('li')].map(+ (item) => item.textContent,+ );+ expect(checks).toEqual([+ 'Guest thaw.',+ 'Fresh successful workload writes to every filesystem covered by the backup.',+ 'Workload liveness.',+ ]);+ expect(safetyText()).toContain(+ 'until the backup has ended and independent post-backup checks confirm all three',+ );+ expect(safetyText()).toContain(+ 'established safe checks, independent of Pulse and the QEMU Guest Agent',+ );+ expect(safetyText()).toContain(+ 'a console connection, a successful read or a write to only the OS disk is not enough',+ );+ });++ it('leaves failed or unavailable recovery checks paused without inducing unsafe proof', () => {+ expect(safetyText()).toContain(+ 'If any check fails or is unavailable, leave Pulse and its automatic updater paused',+ );+ expect(safetyText()).toContain("use the guest/platform's recovery procedure");+ expect(safetyText()).toContain(+ 'Do not force writes, repeat a backup or send guest-agent probes to fill the gap',+ );+ expect(safetyText()).toContain('Do not disable filesystem freezing');+ const article = document.createElement('article');+ article.innerHTML = renderDocMarkdown(+ guide+ .split('If the guest stopped responding during backup')[1]+ .split('When reporting missing records')[0],+ 'RECOVERY',+ );+ expect(article.textContent?.replace(/\s+/g, ' ')).toContain(+ 'a console connection alone does not clear the precaution',+ );+ });++ it('restores only prior-active services and timers and preserves outage coverage', () => {+ expect(safetyText()).toContain(+ 'After all checks pass, restore only services and timers that were active before the pause',+ );+ expect(safetyText()).toContain('Unknown pre-pause states are not permission to start them');+ expect(safetyText()).toContain('An updater must not restart a previously inactive server');+ expect(safetyText()).toContain('deployment-specific restoration steps');+ expect(safetyText()).toContain('arrange independent outage coverage');+ expect(safetyText()).toContain('not proof of a repaired or reproduced native thaw failure');+ });+it('points to existing platform views and a usable backup-safety heading', () => {expect(guide).toContain('There is no top-level Recovery page.');expect(guide).toContain('**Proxmox → Backups** (`/proxmox/backups`)');