Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Make the backup monitoring pause usable and verifiable

pulse-triage[bot]3fd44b832567
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md
index 547b336a02..f05100da0a 100644
--- a/docs/TROUBLESHOOTING.md
+++ b/docs/TROUBLESHOOTING.md
@@ -233,8 +233,11 @@ repair an older generated unit rather than adding a JSON-parsing wrapper.
token's read permissions. Schedule any setup change or restart outside backups.
- Do not run guest-agent probes during backup freeze/thaw. An OK backup task
or an absent lock does not confirm thaw.
-- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight
- and backup safety precaution; it does not verify a fresh disk poll.
+- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight;
+ it does not verify a fresh disk poll. If an affected installation needs to
+ pause monitoring, follow the [manual backup precaution](VM_DISK_MONITORING.md#pause-pulse-for-a-planned-freeze-enabled-backup).
+ It checks the server service and update timer before the backup, then requires
+ independent guest thaw confirmation before starting Pulse again.
#### Temperature data missing
- Install `lm-sensors` on the host.
diff --git a/docs/VM_DISK_MONITORING.md b/docs/VM_DISK_MONITORING.md
index 9c0b762933..5b3272aba2 100644
--- a/docs/VM_DISK_MONITORING.md
+++ b/docs/VM_DISK_MONITORING.md
@@ -24,6 +24,89 @@ or force-reset a guest as a disk-monitoring diagnostic. Those operations can
affect workloads or backup consistency. Preserve the existing backup task and
redacted error, rather than repeat a potentially harmful backup for a report.
+### Pause Pulse for a planned freeze-enabled backup
+
+This manual precaution is for an affected **systemd installation**, before a
+scheduled backup starts. It is **not a recovery procedure** for a backup already
+in progress or an unresponsive guest, and **not an automatic backup hook**.
+
+1. **Identify the Pulse server service.** Run these commands on the machine
+ running the Pulse server: for Proxmox LXC, **inside the Pulse LXC**,
+ **not the Proxmox host or the backed-up VM**. The examples use
+ `pulse.service`; substitute your actual server unit throughout
+ (`pulse-backend.service` on some older installs), **not pulse-agent.service**.
+
+ ```bash
+ systemctl show pulse.service --property=LoadState,ActiveState,MainPID
+ ```
+
+ Require `LoadState=loaded` and record whether Pulse was active. If the unit
+ is missing, do not assume monitoring has stopped: identify your deployment's
+ actual stop control. Docker and other supervisors need their own controls.
+
+2. **Prevent an update from starting Pulse again.** Do not install or update
+ Pulse during this window. If you use unattended updates, inspect the actual
+ matching timer (the default is `pulse-update.timer`):
+
+ ```bash
+ systemctl show pulse-update.timer --property=LoadState,ActiveState
+ ```
+
+ Record whether the timer was active. For a loaded timer, pause it and check
+ both it and its update service, substituting your actual unit names:
+
+ ```bash
+ sudo systemctl stop pulse-update.timer
+ systemctl show pulse-update.timer pulse-update.service \
+ --property=Id,LoadState,ActiveState,MainPID
+ ```
+
+ The timer must be inactive. If the update service is active, activating or
+ deactivating, let it finish normally before continuing. Do not interrupt an
+ installation. A missing update unit is not proof that a custom updater is
+ idle; check your deployment's update controls too.
+
+3. **Stop Pulse and confirm it is stopped, before the backup starts.**
+
+ ```bash
+ sudo systemctl stop pulse.service
+ systemctl show pulse.service --property=LoadState,ActiveState,MainPID
+ ```
+
+ Require `LoadState=loaded`, `ActiveState=inactive` and `MainPID=0` after a
+ successful stop. Do not start the backup if the stop fails, the state is
+ unknown or another Pulse server is still polling the affected guests.
+ **Pulse monitoring and alerts are unavailable while stopped.** Do not run
+ an update, start another Pulse instance or reboot its host during the pause.
+
+4. **Verify the guest independently, then restore only what you paused.**
+ Keep Pulse stopped until the backup has ended **and** your established
+ guest-console or workload checks confirm thaw, including **fresh successful
+ workload writes** to the filesystems covered by the backup. A console
+ connection or a successful read alone is not enough. If thaw cannot be
+ confirmed, leave Pulse stopped and use the guest/platform's recovery
+ procedure; do not repeat the backup or use guest-agent probes to test it.
+
+ Only after that confirmation, if Pulse was active beforehand:
+
+ ```bash
+ sudo systemctl start pulse.service
+ systemctl is-active pulse.service
+ ```
+
+ `active` confirms service startup, not proof that polling or alerts have
+ recovered. Check normal observation times in Pulse without Run Diagnostics
+ or manual guest-agent probes. Restore the update timer **only if it was
+ active beforehand**, after Pulse has started:
+
+ ```bash
+ sudo systemctl start pulse-update.timer
+ ```
+
+ Leave previously inactive services/timers inactive. Arrange independent
+ outage coverage and repeat the precaution for each affected backup window;
+ this manual sequence does not schedule future pauses or prove a fix.
+
## 🚀 Setup
Plan changes outside backup windows and follow the guest's normal maintenance
diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.md
index 547b336a02..f05100da0a 100644
--- a/frontend-modern/public/docs/TROUBLESHOOTING.md
+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md
@@ -233,8 +233,11 @@ repair an older generated unit rather than adding a JSON-parsing wrapper.
token's read permissions. Schedule any setup change or restart outside backups.
- Do not run guest-agent probes during backup freeze/thaw. An OK backup task
or an absent lock does not confirm thaw.
-- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight
- and backup safety precaution; it does not verify a fresh disk poll.
+- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight;
+ it does not verify a fresh disk poll. If an affected installation needs to
+ pause monitoring, follow the [manual backup precaution](VM_DISK_MONITORING.md#pause-pulse-for-a-planned-freeze-enabled-backup).
+ It checks the server service and update timer before the backup, then requires
+ independent guest thaw confirmation before starting Pulse again.
#### Temperature data missing
- Install `lm-sensors` on the host.
diff --git a/frontend-modern/public/docs/VM_DISK_MONITORING.md b/frontend-modern/public/docs/VM_DISK_MONITORING.md
index 9c0b762933..5b3272aba2 100644
--- a/frontend-modern/public/docs/VM_DISK_MONITORING.md
+++ b/frontend-modern/public/docs/VM_DISK_MONITORING.md
@@ -24,6 +24,89 @@ or force-reset a guest as a disk-monitoring diagnostic. Those operations can
affect workloads or backup consistency. Preserve the existing backup task and
redacted error, rather than repeat a potentially harmful backup for a report.
+### Pause Pulse for a planned freeze-enabled backup
+
+This manual precaution is for an affected **systemd installation**, before a
+scheduled backup starts. It is **not a recovery procedure** for a backup already
+in progress or an unresponsive guest, and **not an automatic backup hook**.
+
+1. **Identify the Pulse server service.** Run these commands on the machine
+ running the Pulse server: for Proxmox LXC, **inside the Pulse LXC**,
+ **not the Proxmox host or the backed-up VM**. The examples use
+ `pulse.service`; substitute your actual server unit throughout
+ (`pulse-backend.service` on some older installs), **not pulse-agent.service**.
+
+ ```bash
+ systemctl show pulse.service --property=LoadState,ActiveState,MainPID
+ ```
+
+ Require `LoadState=loaded` and record whether Pulse was active. If the unit
+ is missing, do not assume monitoring has stopped: identify your deployment's
+ actual stop control. Docker and other supervisors need their own controls.
+
+2. **Prevent an update from starting Pulse again.** Do not install or update
+ Pulse during this window. If you use unattended updates, inspect the actual
+ matching timer (the default is `pulse-update.timer`):
+
+ ```bash
+ systemctl show pulse-update.timer --property=LoadState,ActiveState
+ ```
+
+ Record whether the timer was active. For a loaded timer, pause it and check
+ both it and its update service, substituting your actual unit names:
+
+ ```bash
+ sudo systemctl stop pulse-update.timer
+ systemctl show pulse-update.timer pulse-update.service \
+ --property=Id,LoadState,ActiveState,MainPID
+ ```
+
+ The timer must be inactive. If the update service is active, activating or
+ deactivating, let it finish normally before continuing. Do not interrupt an
+ installation. A missing update unit is not proof that a custom updater is
+ idle; check your deployment's update controls too.
+
+3. **Stop Pulse and confirm it is stopped, before the backup starts.**
+
+ ```bash
+ sudo systemctl stop pulse.service
+ systemctl show pulse.service --property=LoadState,ActiveState,MainPID
+ ```
+
+ Require `LoadState=loaded`, `ActiveState=inactive` and `MainPID=0` after a
+ successful stop. Do not start the backup if the stop fails, the state is
+ unknown or another Pulse server is still polling the affected guests.
+ **Pulse monitoring and alerts are unavailable while stopped.** Do not run
+ an update, start another Pulse instance or reboot its host during the pause.
+
+4. **Verify the guest independently, then restore only what you paused.**
+ Keep Pulse stopped until the backup has ended **and** your established
+ guest-console or workload checks confirm thaw, including **fresh successful
+ workload writes** to the filesystems covered by the backup. A console
+ connection or a successful read alone is not enough. If thaw cannot be
+ confirmed, leave Pulse stopped and use the guest/platform's recovery
+ procedure; do not repeat the backup or use guest-agent probes to test it.
+
+ Only after that confirmation, if Pulse was active beforehand:
+
+ ```bash
+ sudo systemctl start pulse.service
+ systemctl is-active pulse.service
+ ```
+
+ `active` confirms service startup, not proof that polling or alerts have
+ recovered. Check normal observation times in Pulse without Run Diagnostics
+ or manual guest-agent probes. Restore the update timer **only if it was
+ active beforehand**, after Pulse has started:
+
+ ```bash
+ sudo systemctl start pulse-update.timer
+ ```
+
+ Leave previously inactive services/timers inactive. Arrange independent
+ outage coverage and repeat the precaution for each affected backup window;
+ this manual sequence does not schedule future pauses or prove a fix.
+
## 🚀 Setup
Plan changes outside backup windows and follow the guest's normal maintenance
diff --git a/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts b/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts
new file mode 100644
index 0000000000..fa22a89310
--- /dev/null
+++ b/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts
@@ -0,0 +1,107 @@
+import { readFileSync } from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { describe, expect, it } from 'vitest';
+import { renderDocMarkdown } from '@/features/docs/docMarkdown';
+
+const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');
+const read = (name: string) => readFileSync(path.join(root, name), 'utf8');
+const guide = read('docs/VM_DISK_MONITORING.md');
+const section =
+ guide.split('### Pause Pulse for a planned freeze-enabled backup')[1]?.split('## 🚀 Setup')[0] ??
+ '';
+const words = section.replace(/\s+/g, ' ');
+
+function render(markdown: string, name: string): HTMLElement {
+ const article = document.createElement('article');
+ article.innerHTML = renderDocMarkdown(markdown, name);
+ return article;
+}
+
+describe('manual backup safety help', () => {
+ it('ships the complete precaution and makes it reachable from troubleshooting', () => {
+ expect(read('frontend-modern/public/docs/VM_DISK_MONITORING.md')).toBe(guide);
+ const troubleshooting = read('docs/TROUBLESHOOTING.md');
+ expect(read('frontend-modern/public/docs/TROUBLESHOOTING.md')).toBe(troubleshooting);
+ const link = render(troubleshooting, 'TROUBLESHOOTING').querySelector(
+ 'a[href="/docs/VM_DISK_MONITORING#pause-pulse-for-a-planned-freeze-enabled-backup"]',
+ );
+ expect(link?.textContent).toBe('manual backup precaution');
+ expect(
+ render(guide, 'VM_DISK_MONITORING').querySelector(
+ '#pause-pulse-for-a-planned-freeze-enabled-backup',
+ )?.textContent,
+ ).toBe('Pause Pulse for a planned freeze-enabled backup');
+ });
+
+ it('keeps service control in the actual Pulse server container, not the guest or host agent', () => {
+ expect(words).toContain('inside the Pulse LXC');
+ expect(words).toContain('not the Proxmox host or the backed-up VM');
+ expect(words).toContain('pulse-backend.service');
+ expect(words).toContain('not pulse-agent.service');
+ expect(words).toContain('LoadState=loaded');
+ expect(words).toContain('If the unit is missing');
+ });
+
+ it('checks updater state before server stop and preserves the original timer state', () => {
+ expect(words).toContain('Record whether the timer was active');
+ expect(words).toContain('let it finish normally before continuing');
+ expect(words).toContain('Do not interrupt an installation');
+ expect(words).toContain('only if it was active beforehand');
+ const blocks = [...render(section, 'VM_DISK_MONITORING').querySelectorAll('pre code')].map(
+ (code) => code.textContent?.trim(),
+ );
+ expect(blocks).toEqual([
+ 'systemctl show pulse.service --property=LoadState,ActiveState,MainPID',
+ 'systemctl show pulse-update.timer --property=LoadState,ActiveState',
+ 'sudo systemctl stop pulse-update.timer\nsystemctl show pulse-update.timer pulse-update.service \\\n --property=Id,LoadState,ActiveState,MainPID',
+ 'sudo systemctl stop pulse.service\nsystemctl show pulse.service --property=LoadState,ActiveState,MainPID',
+ 'sudo systemctl start pulse.service\nsystemctl is-active pulse.service',
+ 'sudo systemctl start pulse-update.timer',
+ ]);
+ // These defaults and the stopped-service condition are from the existing
+ // installer, not a new service or a promise about a custom deployment.
+ const installer = read('install.sh');
+ expect(installer).toContain('DEFAULT_SERVICE_NAME="pulse"');
+ expect(installer).toContain('SERVICE_NAME="pulse-backend"');
+ expect(installer).toContain(
+ "ExecCondition=/bin/sh -c 'systemctl is-active --quiet ${service_name}'",
+ );
+ });
+
+ it('requires stopped-state readback and discloses the whole monitoring outage', () => {
+ expect(words).toContain('ActiveState=inactive');
+ expect(words).toContain('MainPID=0');
+ expect(words).toContain('Do not start the backup if');
+ expect(words).toContain('Pulse monitoring and alerts are unavailable');
+ expect(words).toContain('Do not run an update, start another Pulse instance');
+ expect(words).toContain('repeat the precaution for each affected backup window');
+ });
+
+ it('requires independent workload writes before restart, never merely an OK task or an open console', () => {
+ expect(words).toContain('Keep Pulse stopped until');
+ expect(words).toContain('fresh successful workload writes');
+ expect(words).toContain('console connection or a successful read alone is not enough');
+ expect(words).toContain('not proof that polling or alerts have recovered');
+ expect(words).toContain('normal observation times');
+ expect(words).toContain('without Run Diagnostics or manual guest-agent probes');
+ expect(guide).toContain('An OK backup task or an absent VM lock');
+ expect(guide).toContain('does not prove thaw succeeded');
+ expect(guide).toContain('not a claim\nthat the monitoring defect is fixed');
+ });
+
+ it('does not turn a planned precaution into an incident recovery or an automatic restart hook', () => {
+ expect(words).toContain('not a recovery procedure');
+ expect(words).toContain('not an automatic backup hook');
+ expect(words).toContain('If thaw cannot be confirmed, leave Pulse stopped');
+ expect(guide).toContain(
+ 'Do not test freeze/thaw commands, clear backup locks, disable backup freezing',
+ );
+ const commands = [...render(section, 'VM_DISK_MONITORING').querySelectorAll('pre code')]
+ .map((code) => code.textContent)
+ .join('\n');
+ expect(commands).not.toMatch(
+ /qm |pct |pvesh |guest-fsfreeze|restart|disable|enable|mask|reset|rm |kill|curl|wget|Environment/,
+ );
+ });
+});