Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Make backup-pause restoration conditional on complete workload checks

pulse-triage[bot]e35df4b7779f
diff --git a/docs/VM_DISK_MONITORING.md b/docs/VM_DISK_MONITORING.md
index f331ccba0c..abf24f75d5 100644
--- a/docs/VM_DISK_MONITORING.md
+++ b/docs/VM_DISK_MONITORING.md
@@ -13,9 +13,12 @@ There is a [report of a VM remaining frozen despite an OK backup task](https://g
The reported command-ID collision is not established as a reproduced cause.
If your installation is affected, stop the Pulse service before the backup
-and restart it only after independently confirming the guest has thawed, using
-your existing guest console or workload checks. **Pulse monitoring and alerts
-are unavailable while it is stopped.** An OK backup task or an absent VM lock
+and keep it stopped until the backup has ended and independent checks confirm
+**thaw, fresh successful writes to every filesystem covered by the backup, and
+workload liveness**. Use your established guest-console or workload checks, not
+Pulse readings or guest-agent probes. Restore only services and timers that were
+active before the pause. **Pulse monitoring and alerts are unavailable while it
+is stopped.** An OK backup task or an absent VM lock
does not prove thaw succeeded. This is a temporary precaution, not a claim
that the monitoring defect is fixed.
@@ -81,13 +84,29 @@ in progress or an unresponsive guest, and **not an automatic backup hook**.
4. **Verify the guest independently, then restore only what you paused.**
Keep Pulse stopped until the backup has ended **and** your established
- guest-console or workload checks confirm thaw, including **fresh successful
- workload writes** to the filesystems covered by the backup. A console
- connection or a successful read alone is not enough. If thaw cannot be
- confirmed, leave Pulse stopped and use the guest/platform's recovery
- procedure; do not repeat the backup or use guest-agent probes to test it.
-
- Only after that confirmation, if Pulse was active beforehand:
+ guest-console or workload checks confirm all three: **thaw**, **fresh successful
+ workload writes to every filesystem covered by the backup**, and **workload
+ liveness**. Use evidence from **after the backup ended**, independent of Pulse
+ and the QEMU Guest Agent. A console connection or a successful read alone is
+ not enough; neither is a write to only the OS disk when the backup covers
+ other filesystems. Use the workload's normal safe checks, not forced writes,
+ test-file commands or a new freeze/thaw cycle. If any check is unavailable or
+ fails, leave Pulse stopped and use the guest/platform's recovery procedure;
+ do not repeat the backup or use guest-agent probes to test it.
+
+ Apply the recorded pre-pause states separately:
+
+ | Pulse server before pause | Update timer before pause | Restore after all safety checks pass |
+ | --- | --- | --- |
+ | Active | Active | Start Pulse and confirm it is active; then restore and check the timer. |
+ | Active | Inactive | Start Pulse and confirm it is active; leave the timer inactive. |
+ | Inactive | Active | Leave Pulse inactive. Restore the timer only if its installed updater is confirmed not to start an inactive Pulse server; otherwise keep it paused until that behaviour is resolved through normal maintenance. |
+ | Inactive | Inactive | Leave both inactive. |
+
+ If either pre-pause state is unknown, do not guess or start either unit;
+ establish the original states before restoring.
+
+ Only after all safety checks pass, if Pulse was active beforehand:
```bash
sudo systemctl start pulse.service
@@ -96,13 +115,22 @@ in progress or an unresponsive guest, and **not an automatic backup hook**.
`active` confirms service startup, not proof that polling or alerts have
recovered. Check normal observation times in Pulse without Run Diagnostics
- or manual guest-agent probes. Restore the update timer **only if it was
- active beforehand**, after Pulse has started:
+ or manual guest-agent probes. If startup fails or its state is unknown, keep
+ the timer paused and resolve the startup failure through normal maintenance.
+ Restore the update timer **only if it was active beforehand**, using the
+ matching row above:
```bash
sudo systemctl start pulse-update.timer
+ systemctl is-active pulse-update.timer
```
+ Require `active` for a timer you restored. A persistent timer may run a missed
+ update immediately, so do not restore it before the safety checks. The current
+ Pulse installer makes its updater skip an inactive Pulse server; an older or
+ customised installed unit may differ. A timer's `active` state does not prove
+ that an update or monitoring succeeded.
+
Leave previously inactive services/timers inactive. Arrange independent
outage coverage and repeat the precaution for each affected backup window;
this manual sequence does not schedule future pauses or prove a fix.
diff --git a/frontend-modern/public/docs/VM_DISK_MONITORING.md b/frontend-modern/public/docs/VM_DISK_MONITORING.md
index f331ccba0c..abf24f75d5 100644
--- a/frontend-modern/public/docs/VM_DISK_MONITORING.md
+++ b/frontend-modern/public/docs/VM_DISK_MONITORING.md
@@ -13,9 +13,12 @@ There is a [report of a VM remaining frozen despite an OK backup task](https://g
The reported command-ID collision is not established as a reproduced cause.
If your installation is affected, stop the Pulse service before the backup
-and restart it only after independently confirming the guest has thawed, using
-your existing guest console or workload checks. **Pulse monitoring and alerts
-are unavailable while it is stopped.** An OK backup task or an absent VM lock
+and keep it stopped until the backup has ended and independent checks confirm
+**thaw, fresh successful writes to every filesystem covered by the backup, and
+workload liveness**. Use your established guest-console or workload checks, not
+Pulse readings or guest-agent probes. Restore only services and timers that were
+active before the pause. **Pulse monitoring and alerts are unavailable while it
+is stopped.** An OK backup task or an absent VM lock
does not prove thaw succeeded. This is a temporary precaution, not a claim
that the monitoring defect is fixed.
@@ -81,13 +84,29 @@ in progress or an unresponsive guest, and **not an automatic backup hook**.
4. **Verify the guest independently, then restore only what you paused.**
Keep Pulse stopped until the backup has ended **and** your established
- guest-console or workload checks confirm thaw, including **fresh successful
- workload writes** to the filesystems covered by the backup. A console
- connection or a successful read alone is not enough. If thaw cannot be
- confirmed, leave Pulse stopped and use the guest/platform's recovery
- procedure; do not repeat the backup or use guest-agent probes to test it.
-
- Only after that confirmation, if Pulse was active beforehand:
+ guest-console or workload checks confirm all three: **thaw**, **fresh successful
+ workload writes to every filesystem covered by the backup**, and **workload
+ liveness**. Use evidence from **after the backup ended**, independent of Pulse
+ and the QEMU Guest Agent. A console connection or a successful read alone is
+ not enough; neither is a write to only the OS disk when the backup covers
+ other filesystems. Use the workload's normal safe checks, not forced writes,
+ test-file commands or a new freeze/thaw cycle. If any check is unavailable or
+ fails, leave Pulse stopped and use the guest/platform's recovery procedure;
+ do not repeat the backup or use guest-agent probes to test it.
+
+ Apply the recorded pre-pause states separately:
+
+ | Pulse server before pause | Update timer before pause | Restore after all safety checks pass |
+ | --- | --- | --- |
+ | Active | Active | Start Pulse and confirm it is active; then restore and check the timer. |
+ | Active | Inactive | Start Pulse and confirm it is active; leave the timer inactive. |
+ | Inactive | Active | Leave Pulse inactive. Restore the timer only if its installed updater is confirmed not to start an inactive Pulse server; otherwise keep it paused until that behaviour is resolved through normal maintenance. |
+ | Inactive | Inactive | Leave both inactive. |
+
+ If either pre-pause state is unknown, do not guess or start either unit;
+ establish the original states before restoring.
+
+ Only after all safety checks pass, if Pulse was active beforehand:
```bash
sudo systemctl start pulse.service
@@ -96,13 +115,22 @@ in progress or an unresponsive guest, and **not an automatic backup hook**.
`active` confirms service startup, not proof that polling or alerts have
recovered. Check normal observation times in Pulse without Run Diagnostics
- or manual guest-agent probes. Restore the update timer **only if it was
- active beforehand**, after Pulse has started:
+ or manual guest-agent probes. If startup fails or its state is unknown, keep
+ the timer paused and resolve the startup failure through normal maintenance.
+ Restore the update timer **only if it was active beforehand**, using the
+ matching row above:
```bash
sudo systemctl start pulse-update.timer
+ systemctl is-active pulse-update.timer
```
+ Require `active` for a timer you restored. A persistent timer may run a missed
+ update immediately, so do not restore it before the safety checks. The current
+ Pulse installer makes its updater skip an inactive Pulse server; an older or
+ customised installed unit may differ. A timer's `active` state does not prove
+ that an update or monitoring succeeded.
+
Leave previously inactive services/timers inactive. Arrange independent
outage coverage and repeat the precaution for each affected backup window;
this manual sequence does not schedule future pauses or prove a fix.
diff --git a/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts b/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts
index 3d007253ce..03df331959 100644
--- a/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts
+++ b/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts
@@ -19,6 +19,53 @@ function render(markdown: string, name: string): HTMLElement {
}
describe('manual backup safety help', () => {
+ it('puts the complete restart boundary in the opening backup warning', () => {
+ const opening = guide.split('### Pause Pulse for a planned freeze-enabled backup')[0];
+ const text = render(opening, 'VM_DISK_MONITORING').textContent?.replace(/\s+/g, ' ');
+ expect(text).toContain('until the backup has ended');
+ expect(text).toContain(
+ 'thaw, fresh successful writes to every filesystem covered by the backup, and workload liveness',
+ );
+ expect(text).toContain('not Pulse readings or guest-agent probes');
+ expect(text).toContain('Restore only services and timers that were active before the pause');
+ expect(text).toContain('Pulse monitoring and alerts are unavailable');
+ });
+
+ it('requires post-backup evidence for every covered filesystem without manufacturing writes', () => {
+ const text = render(section, 'VM_DISK_MONITORING').textContent?.replace(/\s+/g, ' ');
+ expect(text).toContain('after the backup ended');
+ expect(text).toContain('independent of Pulse and the QEMU Guest Agent');
+ expect(text).toContain('a write to only the OS disk');
+ expect(text).toContain('not forced writes, test-file commands or a new freeze/thaw cycle');
+ expect(text).toContain('If any check is unavailable or fails, leave Pulse stopped');
+ });
+
+ it('renders each prior-state restoration choice without starting an originally inactive server', () => {
+ const rows = [...render(section, 'VM_DISK_MONITORING').querySelectorAll('tbody tr')].map(
+ (row) => [...row.querySelectorAll('td')].map((cell) => cell.textContent?.trim()),
+ );
+ expect(rows.map((row) => row.slice(0, 2))).toEqual([
+ ['Active', 'Active'],
+ ['Active', 'Inactive'],
+ ['Inactive', 'Active'],
+ ['Inactive', 'Inactive'],
+ ]);
+ expect(rows[0][2]).toContain('then restore and check the timer');
+ expect(rows[1][2]).toContain('leave the timer inactive');
+ expect(rows[2][2]).toContain('Leave Pulse inactive');
+ expect(rows[2][2]).toContain('confirmed not to start an inactive Pulse server');
+ expect(rows[2][2]).toContain('otherwise keep it paused');
+ expect(rows[3][2]).toBe('Leave both inactive.');
+ expect(words).toContain(
+ 'If either pre-pause state is unknown, do not guess or start either unit',
+ );
+ expect(words).toContain('If startup fails or its state is unknown, keep the timer paused');
+ expect(words).toContain('A persistent timer may run a missed update immediately');
+ expect(words).toContain('an older or customised installed unit may differ');
+ expect(words).toContain('does not prove that an update or monitoring succeeded');
+ expect(read('install.sh')).toContain('Persistent=true');
+ });
+
it('does not turn a FAQ disk dash into mandatory installation or unsafe recovery', () => {
const faq = read('docs/FAQ.md');
expect(read('frontend-modern/public/docs/FAQ.md')).toBe(faq);
@@ -122,7 +169,7 @@ describe('manual backup safety help', () => {
'sudo systemctl stop pulse-update.timer\nsystemctl show pulse-update.timer pulse-update.service \\\n --property=Id,LoadState,ActiveState,MainPID',
'sudo systemctl stop pulse.service\nsystemctl show pulse.service --property=LoadState,ActiveState,MainPID',
'sudo systemctl start pulse.service\nsystemctl is-active pulse.service',
- 'sudo systemctl start pulse-update.timer',
+ 'sudo systemctl start pulse-update.timer\nsystemctl is-active pulse-update.timer',
]);
// These defaults and the stopped-service condition are from the existing
// installer, not a new service or a promise about a custom deployment.
@@ -158,7 +205,7 @@ describe('manual backup safety help', () => {
it('does not turn a planned precaution into an incident recovery or an automatic restart hook', () => {
expect(words).toContain('not a recovery procedure');
expect(words).toContain('not an automatic backup hook');
- expect(words).toContain('If thaw cannot be confirmed, leave Pulse stopped');
+ expect(words).toContain('If any check is unavailable or fails, leave Pulse stopped');
expect(guide).toContain(
'Do not test freeze/thaw commands, clear backup locks, disable backup freezing',
);
diff --git a/scripts/tests/test_vm_disk_diagnostics.py b/scripts/tests/test_vm_disk_diagnostics.py
index dd58ee5641..956b29e48a 100644
--- a/scripts/tests/test_vm_disk_diagnostics.py
+++ b/scripts/tests/test_vm_disk_diagnostics.py
@@ -466,7 +466,7 @@ class VMDiskHelpTest(unittest.TestCase):
"sudo systemctl stop pulse.service\n"
"systemctl show pulse.service --property=LoadState,ActiveState,MainPID",
"sudo systemctl start pulse.service\nsystemctl is-active pulse.service",
- "sudo systemctl start pulse-update.timer",
+ "sudo systemctl start pulse-update.timer\nsystemctl is-active pulse-update.timer",
])
for block in bash_examples(guide.replace(precaution, "", 1)):
self.assertNotRegex(block, r"\b(curl|wget|qm agent|pveum|systemctl)\b")