Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Make the backup monitoring pause usable and verifiable
pulse-triage[bot]3fd44b832567
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.mdindex 547b336a02..f05100da0a 100644--- a/docs/TROUBLESHOOTING.md+++ b/docs/TROUBLESHOOTING.md@@ -233,8 +233,11 @@ repair an older generated unit rather than adding a JSON-parsing wrapper.token's read permissions. Schedule any setup change or restart outside backups.- Do not run guest-agent probes during backup freeze/thaw. An OK backup taskor an absent lock does not confirm thaw.-- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight- and backup safety precaution; it does not verify a fresh disk poll.+- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight;+ it does not verify a fresh disk poll. If an affected installation needs to+ pause monitoring, follow the [manual backup precaution](VM_DISK_MONITORING.md#pause-pulse-for-a-planned-freeze-enabled-backup).+ It checks the server service and update timer before the backup, then requires+ independent guest thaw confirmation before starting Pulse again.#### Temperature data missing- Install `lm-sensors` on the host.diff --git a/docs/VM_DISK_MONITORING.md b/docs/VM_DISK_MONITORING.mdindex 9c0b762933..5b3272aba2 100644--- a/docs/VM_DISK_MONITORING.md+++ b/docs/VM_DISK_MONITORING.md@@ -24,6 +24,89 @@ or force-reset a guest as a disk-monitoring diagnostic. Those operations canaffect workloads or backup consistency. Preserve the existing backup task andredacted error, rather than repeat a potentially harmful backup for a report.+### Pause Pulse for a planned freeze-enabled backup++This manual precaution is for an affected **systemd installation**, before a+scheduled backup starts. It is **not a recovery procedure** for a backup already+in progress or an unresponsive guest, and **not an automatic backup hook**.++1. **Identify the Pulse server service.** Run these commands on the machine+ running the Pulse server: for Proxmox LXC, **inside the Pulse LXC**,+ **not the Proxmox host or the backed-up VM**. The examples use+ `pulse.service`; substitute your actual server unit throughout+ (`pulse-backend.service` on some older installs), **not pulse-agent.service**.++ ```bash+ systemctl show pulse.service --property=LoadState,ActiveState,MainPID+ ```++ Require `LoadState=loaded` and record whether Pulse was active. If the unit+ is missing, do not assume monitoring has stopped: identify your deployment's+ actual stop control. Docker and other supervisors need their own controls.++2. **Prevent an update from starting Pulse again.** Do not install or update+ Pulse during this window. If you use unattended updates, inspect the actual+ matching timer (the default is `pulse-update.timer`):++ ```bash+ systemctl show pulse-update.timer --property=LoadState,ActiveState+ ```++ Record whether the timer was active. For a loaded timer, pause it and check+ both it and its update service, substituting your actual unit names:++ ```bash+ sudo systemctl stop pulse-update.timer+ systemctl show pulse-update.timer pulse-update.service \+ --property=Id,LoadState,ActiveState,MainPID+ ```++ The timer must be inactive. If the update service is active, activating or+ deactivating, let it finish normally before continuing. Do not interrupt an+ installation. A missing update unit is not proof that a custom updater is+ idle; check your deployment's update controls too.++3. **Stop Pulse and confirm it is stopped, before the backup starts.**++ ```bash+ sudo systemctl stop pulse.service+ systemctl show pulse.service --property=LoadState,ActiveState,MainPID+ ```++ Require `LoadState=loaded`, `ActiveState=inactive` and `MainPID=0` after a+ successful stop. Do not start the backup if the stop fails, the state is+ unknown or another Pulse server is still polling the affected guests.+ **Pulse monitoring and alerts are unavailable while stopped.** Do not run+ an update, start another Pulse instance or reboot its host during the pause.++4. **Verify the guest independently, then restore only what you paused.**+ Keep Pulse stopped until the backup has ended **and** your established+ guest-console or workload checks confirm thaw, including **fresh successful+ workload writes** to the filesystems covered by the backup. A console+ connection or a successful read alone is not enough. If thaw cannot be+ confirmed, leave Pulse stopped and use the guest/platform's recovery+ procedure; do not repeat the backup or use guest-agent probes to test it.++ Only after that confirmation, if Pulse was active beforehand:++ ```bash+ sudo systemctl start pulse.service+ systemctl is-active pulse.service+ ```++ `active` confirms service startup, not proof that polling or alerts have+ recovered. Check normal observation times in Pulse without Run Diagnostics+ or manual guest-agent probes. Restore the update timer **only if it was+ active beforehand**, after Pulse has started:++ ```bash+ sudo systemctl start pulse-update.timer+ ```++ Leave previously inactive services/timers inactive. Arrange independent+ outage coverage and repeat the precaution for each affected backup window;+ this manual sequence does not schedule future pauses or prove a fix.+## 🚀 SetupPlan changes outside backup windows and follow the guest's normal maintenancediff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.mdindex 547b336a02..f05100da0a 100644--- a/frontend-modern/public/docs/TROUBLESHOOTING.md+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md@@ -233,8 +233,11 @@ repair an older generated unit rather than adding a JSON-parsing wrapper.token's read permissions. Schedule any setup change or restart outside backups.- Do not run guest-agent probes during backup freeze/thaw. An OK backup taskor an absent lock does not confirm thaw.-- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight- and backup safety precaution; it does not verify a fresh disk poll.+- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight;+ it does not verify a fresh disk poll. If an affected installation needs to+ pause monitoring, follow the [manual backup precaution](VM_DISK_MONITORING.md#pause-pulse-for-a-planned-freeze-enabled-backup).+ It checks the server service and update timer before the backup, then requires+ independent guest thaw confirmation before starting Pulse again.#### Temperature data missing- Install `lm-sensors` on the host.diff --git a/frontend-modern/public/docs/VM_DISK_MONITORING.md b/frontend-modern/public/docs/VM_DISK_MONITORING.mdindex 9c0b762933..5b3272aba2 100644--- a/frontend-modern/public/docs/VM_DISK_MONITORING.md+++ b/frontend-modern/public/docs/VM_DISK_MONITORING.md@@ -24,6 +24,89 @@ or force-reset a guest as a disk-monitoring diagnostic. Those operations canaffect workloads or backup consistency. Preserve the existing backup task andredacted error, rather than repeat a potentially harmful backup for a report.+### Pause Pulse for a planned freeze-enabled backup++This manual precaution is for an affected **systemd installation**, before a+scheduled backup starts. It is **not a recovery procedure** for a backup already+in progress or an unresponsive guest, and **not an automatic backup hook**.++1. **Identify the Pulse server service.** Run these commands on the machine+ running the Pulse server: for Proxmox LXC, **inside the Pulse LXC**,+ **not the Proxmox host or the backed-up VM**. The examples use+ `pulse.service`; substitute your actual server unit throughout+ (`pulse-backend.service` on some older installs), **not pulse-agent.service**.++ ```bash+ systemctl show pulse.service --property=LoadState,ActiveState,MainPID+ ```++ Require `LoadState=loaded` and record whether Pulse was active. If the unit+ is missing, do not assume monitoring has stopped: identify your deployment's+ actual stop control. Docker and other supervisors need their own controls.++2. **Prevent an update from starting Pulse again.** Do not install or update+ Pulse during this window. If you use unattended updates, inspect the actual+ matching timer (the default is `pulse-update.timer`):++ ```bash+ systemctl show pulse-update.timer --property=LoadState,ActiveState+ ```++ Record whether the timer was active. For a loaded timer, pause it and check+ both it and its update service, substituting your actual unit names:++ ```bash+ sudo systemctl stop pulse-update.timer+ systemctl show pulse-update.timer pulse-update.service \+ --property=Id,LoadState,ActiveState,MainPID+ ```++ The timer must be inactive. If the update service is active, activating or+ deactivating, let it finish normally before continuing. Do not interrupt an+ installation. A missing update unit is not proof that a custom updater is+ idle; check your deployment's update controls too.++3. **Stop Pulse and confirm it is stopped, before the backup starts.**++ ```bash+ sudo systemctl stop pulse.service+ systemctl show pulse.service --property=LoadState,ActiveState,MainPID+ ```++ Require `LoadState=loaded`, `ActiveState=inactive` and `MainPID=0` after a+ successful stop. Do not start the backup if the stop fails, the state is+ unknown or another Pulse server is still polling the affected guests.+ **Pulse monitoring and alerts are unavailable while stopped.** Do not run+ an update, start another Pulse instance or reboot its host during the pause.++4. **Verify the guest independently, then restore only what you paused.**+ Keep Pulse stopped until the backup has ended **and** your established+ guest-console or workload checks confirm thaw, including **fresh successful+ workload writes** to the filesystems covered by the backup. A console+ connection or a successful read alone is not enough. If thaw cannot be+ confirmed, leave Pulse stopped and use the guest/platform's recovery+ procedure; do not repeat the backup or use guest-agent probes to test it.++ Only after that confirmation, if Pulse was active beforehand:++ ```bash+ sudo systemctl start pulse.service+ systemctl is-active pulse.service+ ```++ `active` confirms service startup, not proof that polling or alerts have+ recovered. Check normal observation times in Pulse without Run Diagnostics+ or manual guest-agent probes. Restore the update timer **only if it was+ active beforehand**, after Pulse has started:++ ```bash+ sudo systemctl start pulse-update.timer+ ```++ Leave previously inactive services/timers inactive. Arrange independent+ outage coverage and repeat the precaution for each affected backup window;+ this manual sequence does not schedule future pauses or prove a fix.+## 🚀 SetupPlan changes outside backup windows and follow the guest's normal maintenancediff --git a/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts b/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.tsnew file mode 100644index 0000000000..fa22a89310--- /dev/null+++ b/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts@@ -0,0 +1,107 @@+import { readFileSync } from 'node:fs';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { describe, expect, it } from 'vitest';+import { renderDocMarkdown } from '@/features/docs/docMarkdown';++const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');+const read = (name: string) => readFileSync(path.join(root, name), 'utf8');+const guide = read('docs/VM_DISK_MONITORING.md');+const section =+ guide.split('### Pause Pulse for a planned freeze-enabled backup')[1]?.split('## 🚀 Setup')[0] ??+ '';+const words = section.replace(/\s+/g, ' ');++function render(markdown: string, name: string): HTMLElement {+ const article = document.createElement('article');+ article.innerHTML = renderDocMarkdown(markdown, name);+ return article;+}++describe('manual backup safety help', () => {+ it('ships the complete precaution and makes it reachable from troubleshooting', () => {+ expect(read('frontend-modern/public/docs/VM_DISK_MONITORING.md')).toBe(guide);+ const troubleshooting = read('docs/TROUBLESHOOTING.md');+ expect(read('frontend-modern/public/docs/TROUBLESHOOTING.md')).toBe(troubleshooting);+ const link = render(troubleshooting, 'TROUBLESHOOTING').querySelector(+ 'a[href="/docs/VM_DISK_MONITORING#pause-pulse-for-a-planned-freeze-enabled-backup"]',+ );+ expect(link?.textContent).toBe('manual backup precaution');+ expect(+ render(guide, 'VM_DISK_MONITORING').querySelector(+ '#pause-pulse-for-a-planned-freeze-enabled-backup',+ )?.textContent,+ ).toBe('Pause Pulse for a planned freeze-enabled backup');+ });++ it('keeps service control in the actual Pulse server container, not the guest or host agent', () => {+ expect(words).toContain('inside the Pulse LXC');+ expect(words).toContain('not the Proxmox host or the backed-up VM');+ expect(words).toContain('pulse-backend.service');+ expect(words).toContain('not pulse-agent.service');+ expect(words).toContain('LoadState=loaded');+ expect(words).toContain('If the unit is missing');+ });++ it('checks updater state before server stop and preserves the original timer state', () => {+ expect(words).toContain('Record whether the timer was active');+ expect(words).toContain('let it finish normally before continuing');+ expect(words).toContain('Do not interrupt an installation');+ expect(words).toContain('only if it was active beforehand');+ const blocks = [...render(section, 'VM_DISK_MONITORING').querySelectorAll('pre code')].map(+ (code) => code.textContent?.trim(),+ );+ expect(blocks).toEqual([+ 'systemctl show pulse.service --property=LoadState,ActiveState,MainPID',+ 'systemctl show pulse-update.timer --property=LoadState,ActiveState',+ 'sudo systemctl stop pulse-update.timer\nsystemctl show pulse-update.timer pulse-update.service \\\n --property=Id,LoadState,ActiveState,MainPID',+ 'sudo systemctl stop pulse.service\nsystemctl show pulse.service --property=LoadState,ActiveState,MainPID',+ 'sudo systemctl start pulse.service\nsystemctl is-active pulse.service',+ 'sudo systemctl start pulse-update.timer',+ ]);+ // These defaults and the stopped-service condition are from the existing+ // installer, not a new service or a promise about a custom deployment.+ const installer = read('install.sh');+ expect(installer).toContain('DEFAULT_SERVICE_NAME="pulse"');+ expect(installer).toContain('SERVICE_NAME="pulse-backend"');+ expect(installer).toContain(+ "ExecCondition=/bin/sh -c 'systemctl is-active --quiet ${service_name}'",+ );+ });++ it('requires stopped-state readback and discloses the whole monitoring outage', () => {+ expect(words).toContain('ActiveState=inactive');+ expect(words).toContain('MainPID=0');+ expect(words).toContain('Do not start the backup if');+ expect(words).toContain('Pulse monitoring and alerts are unavailable');+ expect(words).toContain('Do not run an update, start another Pulse instance');+ expect(words).toContain('repeat the precaution for each affected backup window');+ });++ it('requires independent workload writes before restart, never merely an OK task or an open console', () => {+ expect(words).toContain('Keep Pulse stopped until');+ expect(words).toContain('fresh successful workload writes');+ expect(words).toContain('console connection or a successful read alone is not enough');+ expect(words).toContain('not proof that polling or alerts have recovered');+ expect(words).toContain('normal observation times');+ expect(words).toContain('without Run Diagnostics or manual guest-agent probes');+ expect(guide).toContain('An OK backup task or an absent VM lock');+ expect(guide).toContain('does not prove thaw succeeded');+ expect(guide).toContain('not a claim\nthat the monitoring defect is fixed');+ });++ it('does not turn a planned precaution into an incident recovery or an automatic restart hook', () => {+ expect(words).toContain('not a recovery procedure');+ expect(words).toContain('not an automatic backup hook');+ expect(words).toContain('If thaw cannot be confirmed, leave Pulse stopped');+ expect(guide).toContain(+ 'Do not test freeze/thaw commands, clear backup locks, disable backup freezing',+ );+ const commands = [...render(section, 'VM_DISK_MONITORING').querySelectorAll('pre code')]+ .map((code) => code.textContent)+ .join('\n');+ expect(commands).not.toMatch(+ /qm |pct |pvesh |guest-fsfreeze|restart|disable|enable|mask|reset|rm |kill|curl|wget|Environment/,+ );+ });+});