Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Make backup-pause restoration conditional on complete workload checks
pulse-triage[bot]e35df4b7779f
diff --git a/docs/VM_DISK_MONITORING.md b/docs/VM_DISK_MONITORING.mdindex f331ccba0c..abf24f75d5 100644--- a/docs/VM_DISK_MONITORING.md+++ b/docs/VM_DISK_MONITORING.md@@ -13,9 +13,12 @@ There is a [report of a VM remaining frozen despite an OK backup task](https://gThe reported command-ID collision is not established as a reproduced cause.If your installation is affected, stop the Pulse service before the backup-and restart it only after independently confirming the guest has thawed, using-your existing guest console or workload checks. **Pulse monitoring and alerts-are unavailable while it is stopped.** An OK backup task or an absent VM lock+and keep it stopped until the backup has ended and independent checks confirm+**thaw, fresh successful writes to every filesystem covered by the backup, and+workload liveness**. Use your established guest-console or workload checks, not+Pulse readings or guest-agent probes. Restore only services and timers that were+active before the pause. **Pulse monitoring and alerts are unavailable while it+is stopped.** An OK backup task or an absent VM lockdoes not prove thaw succeeded. This is a temporary precaution, not a claimthat the monitoring defect is fixed.@@ -81,13 +84,29 @@ in progress or an unresponsive guest, and **not an automatic backup hook**.4. **Verify the guest independently, then restore only what you paused.**Keep Pulse stopped until the backup has ended **and** your established- guest-console or workload checks confirm thaw, including **fresh successful- workload writes** to the filesystems covered by the backup. A console- connection or a successful read alone is not enough. If thaw cannot be- confirmed, leave Pulse stopped and use the guest/platform's recovery- procedure; do not repeat the backup or use guest-agent probes to test it.-- Only after that confirmation, if Pulse was active beforehand:+ guest-console or workload checks confirm all three: **thaw**, **fresh successful+ workload writes to every filesystem covered by the backup**, and **workload+ liveness**. Use evidence from **after the backup ended**, independent of Pulse+ and the QEMU Guest Agent. A console connection or a successful read alone is+ not enough; neither is a write to only the OS disk when the backup covers+ other filesystems. Use the workload's normal safe checks, not forced writes,+ test-file commands or a new freeze/thaw cycle. If any check is unavailable or+ fails, leave Pulse stopped and use the guest/platform's recovery procedure;+ do not repeat the backup or use guest-agent probes to test it.++ Apply the recorded pre-pause states separately:++ | Pulse server before pause | Update timer before pause | Restore after all safety checks pass |+ | --- | --- | --- |+ | Active | Active | Start Pulse and confirm it is active; then restore and check the timer. |+ | Active | Inactive | Start Pulse and confirm it is active; leave the timer inactive. |+ | Inactive | Active | Leave Pulse inactive. Restore the timer only if its installed updater is confirmed not to start an inactive Pulse server; otherwise keep it paused until that behaviour is resolved through normal maintenance. |+ | Inactive | Inactive | Leave both inactive. |++ If either pre-pause state is unknown, do not guess or start either unit;+ establish the original states before restoring.++ Only after all safety checks pass, if Pulse was active beforehand:```bashsudo systemctl start pulse.service@@ -96,13 +115,22 @@ in progress or an unresponsive guest, and **not an automatic backup hook**.`active` confirms service startup, not proof that polling or alerts haverecovered. Check normal observation times in Pulse without Run Diagnostics- or manual guest-agent probes. Restore the update timer **only if it was- active beforehand**, after Pulse has started:+ or manual guest-agent probes. If startup fails or its state is unknown, keep+ the timer paused and resolve the startup failure through normal maintenance.+ Restore the update timer **only if it was active beforehand**, using the+ matching row above:```bashsudo systemctl start pulse-update.timer+ systemctl is-active pulse-update.timer```+ Require `active` for a timer you restored. A persistent timer may run a missed+ update immediately, so do not restore it before the safety checks. The current+ Pulse installer makes its updater skip an inactive Pulse server; an older or+ customised installed unit may differ. A timer's `active` state does not prove+ that an update or monitoring succeeded.+Leave previously inactive services/timers inactive. Arrange independentoutage coverage and repeat the precaution for each affected backup window;this manual sequence does not schedule future pauses or prove a fix.diff --git a/frontend-modern/public/docs/VM_DISK_MONITORING.md b/frontend-modern/public/docs/VM_DISK_MONITORING.mdindex f331ccba0c..abf24f75d5 100644--- a/frontend-modern/public/docs/VM_DISK_MONITORING.md+++ b/frontend-modern/public/docs/VM_DISK_MONITORING.md@@ -13,9 +13,12 @@ There is a [report of a VM remaining frozen despite an OK backup task](https://gThe reported command-ID collision is not established as a reproduced cause.If your installation is affected, stop the Pulse service before the backup-and restart it only after independently confirming the guest has thawed, using-your existing guest console or workload checks. **Pulse monitoring and alerts-are unavailable while it is stopped.** An OK backup task or an absent VM lock+and keep it stopped until the backup has ended and independent checks confirm+**thaw, fresh successful writes to every filesystem covered by the backup, and+workload liveness**. Use your established guest-console or workload checks, not+Pulse readings or guest-agent probes. Restore only services and timers that were+active before the pause. **Pulse monitoring and alerts are unavailable while it+is stopped.** An OK backup task or an absent VM lockdoes not prove thaw succeeded. This is a temporary precaution, not a claimthat the monitoring defect is fixed.@@ -81,13 +84,29 @@ in progress or an unresponsive guest, and **not an automatic backup hook**.4. **Verify the guest independently, then restore only what you paused.**Keep Pulse stopped until the backup has ended **and** your established- guest-console or workload checks confirm thaw, including **fresh successful- workload writes** to the filesystems covered by the backup. A console- connection or a successful read alone is not enough. If thaw cannot be- confirmed, leave Pulse stopped and use the guest/platform's recovery- procedure; do not repeat the backup or use guest-agent probes to test it.-- Only after that confirmation, if Pulse was active beforehand:+ guest-console or workload checks confirm all three: **thaw**, **fresh successful+ workload writes to every filesystem covered by the backup**, and **workload+ liveness**. Use evidence from **after the backup ended**, independent of Pulse+ and the QEMU Guest Agent. A console connection or a successful read alone is+ not enough; neither is a write to only the OS disk when the backup covers+ other filesystems. Use the workload's normal safe checks, not forced writes,+ test-file commands or a new freeze/thaw cycle. If any check is unavailable or+ fails, leave Pulse stopped and use the guest/platform's recovery procedure;+ do not repeat the backup or use guest-agent probes to test it.++ Apply the recorded pre-pause states separately:++ | Pulse server before pause | Update timer before pause | Restore after all safety checks pass |+ | --- | --- | --- |+ | Active | Active | Start Pulse and confirm it is active; then restore and check the timer. |+ | Active | Inactive | Start Pulse and confirm it is active; leave the timer inactive. |+ | Inactive | Active | Leave Pulse inactive. Restore the timer only if its installed updater is confirmed not to start an inactive Pulse server; otherwise keep it paused until that behaviour is resolved through normal maintenance. |+ | Inactive | Inactive | Leave both inactive. |++ If either pre-pause state is unknown, do not guess or start either unit;+ establish the original states before restoring.++ Only after all safety checks pass, if Pulse was active beforehand:```bashsudo systemctl start pulse.service@@ -96,13 +115,22 @@ in progress or an unresponsive guest, and **not an automatic backup hook**.`active` confirms service startup, not proof that polling or alerts haverecovered. Check normal observation times in Pulse without Run Diagnostics- or manual guest-agent probes. Restore the update timer **only if it was- active beforehand**, after Pulse has started:+ or manual guest-agent probes. If startup fails or its state is unknown, keep+ the timer paused and resolve the startup failure through normal maintenance.+ Restore the update timer **only if it was active beforehand**, using the+ matching row above:```bashsudo systemctl start pulse-update.timer+ systemctl is-active pulse-update.timer```+ Require `active` for a timer you restored. A persistent timer may run a missed+ update immediately, so do not restore it before the safety checks. The current+ Pulse installer makes its updater skip an inactive Pulse server; an older or+ customised installed unit may differ. A timer's `active` state does not prove+ that an update or monitoring succeeded.+Leave previously inactive services/timers inactive. Arrange independentoutage coverage and repeat the precaution for each affected backup window;this manual sequence does not schedule future pauses or prove a fix.diff --git a/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts b/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.tsindex 3d007253ce..03df331959 100644--- a/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts+++ b/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts@@ -19,6 +19,53 @@ function render(markdown: string, name: string): HTMLElement {}describe('manual backup safety help', () => {+ it('puts the complete restart boundary in the opening backup warning', () => {+ const opening = guide.split('### Pause Pulse for a planned freeze-enabled backup')[0];+ const text = render(opening, 'VM_DISK_MONITORING').textContent?.replace(/\s+/g, ' ');+ expect(text).toContain('until the backup has ended');+ expect(text).toContain(+ 'thaw, fresh successful writes to every filesystem covered by the backup, and workload liveness',+ );+ expect(text).toContain('not Pulse readings or guest-agent probes');+ expect(text).toContain('Restore only services and timers that were active before the pause');+ expect(text).toContain('Pulse monitoring and alerts are unavailable');+ });++ it('requires post-backup evidence for every covered filesystem without manufacturing writes', () => {+ const text = render(section, 'VM_DISK_MONITORING').textContent?.replace(/\s+/g, ' ');+ expect(text).toContain('after the backup ended');+ expect(text).toContain('independent of Pulse and the QEMU Guest Agent');+ expect(text).toContain('a write to only the OS disk');+ expect(text).toContain('not forced writes, test-file commands or a new freeze/thaw cycle');+ expect(text).toContain('If any check is unavailable or fails, leave Pulse stopped');+ });++ it('renders each prior-state restoration choice without starting an originally inactive server', () => {+ const rows = [...render(section, 'VM_DISK_MONITORING').querySelectorAll('tbody tr')].map(+ (row) => [...row.querySelectorAll('td')].map((cell) => cell.textContent?.trim()),+ );+ expect(rows.map((row) => row.slice(0, 2))).toEqual([+ ['Active', 'Active'],+ ['Active', 'Inactive'],+ ['Inactive', 'Active'],+ ['Inactive', 'Inactive'],+ ]);+ expect(rows[0][2]).toContain('then restore and check the timer');+ expect(rows[1][2]).toContain('leave the timer inactive');+ expect(rows[2][2]).toContain('Leave Pulse inactive');+ expect(rows[2][2]).toContain('confirmed not to start an inactive Pulse server');+ expect(rows[2][2]).toContain('otherwise keep it paused');+ expect(rows[3][2]).toBe('Leave both inactive.');+ expect(words).toContain(+ 'If either pre-pause state is unknown, do not guess or start either unit',+ );+ expect(words).toContain('If startup fails or its state is unknown, keep the timer paused');+ expect(words).toContain('A persistent timer may run a missed update immediately');+ expect(words).toContain('an older or customised installed unit may differ');+ expect(words).toContain('does not prove that an update or monitoring succeeded');+ expect(read('install.sh')).toContain('Persistent=true');+ });+it('does not turn a FAQ disk dash into mandatory installation or unsafe recovery', () => {const faq = read('docs/FAQ.md');expect(read('frontend-modern/public/docs/FAQ.md')).toBe(faq);@@ -122,7 +169,7 @@ describe('manual backup safety help', () => {'sudo systemctl stop pulse-update.timer\nsystemctl show pulse-update.timer pulse-update.service \\\n --property=Id,LoadState,ActiveState,MainPID','sudo systemctl stop pulse.service\nsystemctl show pulse.service --property=LoadState,ActiveState,MainPID','sudo systemctl start pulse.service\nsystemctl is-active pulse.service',- 'sudo systemctl start pulse-update.timer',+ 'sudo systemctl start pulse-update.timer\nsystemctl is-active pulse-update.timer',]);// These defaults and the stopped-service condition are from the existing// installer, not a new service or a promise about a custom deployment.@@ -158,7 +205,7 @@ describe('manual backup safety help', () => {it('does not turn a planned precaution into an incident recovery or an automatic restart hook', () => {expect(words).toContain('not a recovery procedure');expect(words).toContain('not an automatic backup hook');- expect(words).toContain('If thaw cannot be confirmed, leave Pulse stopped');+ expect(words).toContain('If any check is unavailable or fails, leave Pulse stopped');expect(guide).toContain('Do not test freeze/thaw commands, clear backup locks, disable backup freezing',);diff --git a/scripts/tests/test_vm_disk_diagnostics.py b/scripts/tests/test_vm_disk_diagnostics.pyindex dd58ee5641..956b29e48a 100644--- a/scripts/tests/test_vm_disk_diagnostics.py+++ b/scripts/tests/test_vm_disk_diagnostics.py@@ -466,7 +466,7 @@ class VMDiskHelpTest(unittest.TestCase):"sudo systemctl stop pulse.service\n""systemctl show pulse.service --property=LoadState,ActiveState,MainPID","sudo systemctl start pulse.service\nsystemctl is-active pulse.service",- "sudo systemctl start pulse-update.timer",+ "sudo systemctl start pulse-update.timer\nsystemctl is-active pulse-update.timer",])for block in bash_examples(guide.replace(precaution, "", 1)):self.assertNotRegex(block, r"\b(curl|wget|qm agent|pveum|systemctl)\b")