Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Make the planned backup pause usable for Docker servers
pulse-triage[bot]bcd8710ac8f7
diff --git a/docs/DOCKER.md b/docs/DOCKER.mdindex a08442f8c2..313d0f229e 100644--- a/docs/DOCKER.md+++ b/docs/DOCKER.md@@ -52,6 +52,12 @@ volumes:Run with: `docker compose up -d`+**Freeze-enabled VM backups:** API-only monitoring can still send QEMU Guest+Agent requests. For an affected installation, follow the+[planned server pause and checked restoration](VM_DISK_MONITORING.md#pause-a-docker-or-compose-server-for-a-planned-backup),+not container recreation or guest-agent probes. Monitoring and alerts are+unavailable while Pulse is stopped; an OK backup alone does not prove recovery.+Leave authentication overrides unset for a new install and complete[bootstrap-token setup](INSTALL.md#step-1-get-the-token) in your browser. Do not add ashared example password to the Compose file. If automation must skip setup,diff --git a/docs/VM_DISK_MONITORING.md b/docs/VM_DISK_MONITORING.mdindex abf24f75d5..09480663a1 100644--- a/docs/VM_DISK_MONITORING.md+++ b/docs/VM_DISK_MONITORING.md@@ -32,6 +32,8 @@ redacted error, rather than repeat a potentially harmful backup for a report.This manual precaution is for an affected **systemd installation**, before ascheduled backup starts. It is **not a recovery procedure** for a backup alreadyin progress or an unresponsive guest, and **not an automatic backup hook**.+For a Docker or Compose server, use the [container pause procedure](#pause-a-docker-or-compose-server-for-a-planned-backup) instead;+the systemd commands below do not stop a containerised Pulse server.1. **Identify the Pulse server service.** Run these commands on the machinerunning the Pulse server: for Proxmox LXC, **inside the Pulse LXC**,@@ -135,6 +137,112 @@ in progress or an unresponsive guest, and **not an automatic backup hook**.outage coverage and repeat the precaution for each affected backup window;this manual sequence does not schedule future pauses or prove a fix.+### Pause a Docker or Compose server for a planned backup++This is a manual precaution for an affected **standalone Docker or plain Compose+Pulse server**, before a scheduled freeze-enabled backup. It is **not incident+recovery or an automatic backup hook**. Swarm, Kubernetes and other controllers+need their own supported pause controls: stopping a container alone does not+stop a controller from replacing it. Do not use this procedure if you cannot+prevent another server from polling the affected guests.++**Pulse monitoring and alerts are unavailable while stopped.** Arrange+independent outage coverage. A container stop does not cancel a guest-agent+request already issued; let any existing guest/backup operation finish normally+before the planned backup. If that state is unknown, do not start the backup on+the strength of a stopped container.++1. **Identify the existing Pulse server container on its Docker host**, not a+ monitored workload, the backed-up VM or a Pulse Agent container. For Compose,+ use the original project directory and the same project/file options used to+ deploy it. The example service is `pulse`; substitute your actual service:++ ```bash+ docker compose ps --all --quiet pulse+ ```++ Record every returned server container ID privately. An empty list or a+ failed command is not proof that monitoring has stopped. For a `docker run`+ installation, identify the existing server through its saved deployment.+ Set the ID explicitly, then read only its identity and state:++ ```bash+ PULSE_CONTAINER_ID='paste-the-recorded-server-container-id'+ docker inspect --format 'Id={{.Id}} Running={{.State.Running}} Paused={{.State.Paused}} Restarting={{.State.Restarting}} Status={{.State.Status}} Pid={{.State.Pid}} ExitCode={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}' "$PULSE_CONTAINER_ID"+ ```++ Record whether this exact container was running before the pause. Require a+ stable state: either running and not paused/restarting, or already exited+ with PID 0. If it is paused, restarting, being replaced or its state cannot+ be read, resolve that through the deployment's normal controls first. Repeat+ the state check for every server; do not infer another container's state.+ Do not share full `docker inspect`, container environments or resolved+ Compose configuration: they can contain credentials.++2. **Prevent a deployment or updater from restarting/replacing Pulse.** Do not+ install, update, recreate the server, run `docker compose up` or reboot the+ Docker host during this window. Pause any automatic updater or deployment+ job using its supported controls, recording whether it was active first.+ Let an in-progress update finish normally; do not interrupt an installation.+ The supplied Compose example uses `restart: unless-stopped`, but that does+ not prevent an external updater or another operator from starting Pulse.+ Do not change the restart policy, image, mounts or saved deployment as a+ substitute for these checks.++3. **Stop each previously running server and verify the same ID is stopped.**+ Docker commands on the recorded existing container also work for a plain+ Compose deployment; they do not recreate its configuration or data volume:++ ```bash+ docker stop --timeout 60 "$PULSE_CONTAINER_ID"+ docker inspect --format 'Id={{.Id}} Running={{.State.Running}} Paused={{.State.Paused}} Restarting={{.State.Restarting}} Status={{.State.Status}} Pid={{.State.Pid}} ExitCode={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}' "$PULSE_CONTAINER_ID"+ ```++ Require a successful stop and readback of the **same recorded ID** with+ `Running=false`, `Paused=false`, `Restarting=false`, `Status=exited` and+ `Pid=0`. For a server stopped in this step, also require `ExitCode=0` and+ `OOMKilled=false`; a non-zero exit (including 137 after forced termination)+ or unknown shutdown result needs normal maintenance before the backup.+ Leave an already stopped server stopped. If stopping times out,+ fails, the identity changes or any state is unknown, do not start the backup.+ Do not delete/recreate a container, remove volumes or force-kill it to clear+ this check. Docker can terminate a process after its stop timeout; stopped+ state alone does not prove graceful shutdown or completion of earlier guest+ requests. Preserve a shutdown failure for normal maintenance, not a forced+ backup test.++4. **Keep every server stopped until independent recovery checks pass.** The+ backup must have ended, and evidence from **after it ended**, independent of+ Pulse and the QEMU Guest Agent, must confirm **thaw, fresh successful workload+ writes to every filesystem covered by the backup, and workload liveness**.+ Use the workload's established safe checks, not forced writes, test files or+ another freeze/thaw cycle. An OK backup, an absent lock, a console connection+ or a successful read alone is not enough. If any check fails or is unavailable,+ leave Pulse and its automatic updater paused and use the platform's recovery+ procedure, without guest-agent probes or repeating the backup.++ Only for a container recorded as running before the pause, start that+ **same existing ID**, then repeat the bounded state readback from step 1:++ ```bash+ docker start "$PULSE_CONTAINER_ID"+ ```++ Require the original ID, `Running=true`, `Paused=false`, `Restarting=false`+ and `Status=running`. A previously stopped container stays stopped; if its+ original state is unknown, do not guess. Do not use a blanket Compose start+ or `up` to restore a mixture of previously running and stopped servers. If+ the original container is missing or startup fails, keep automation paused+ and resolve that through normal maintenance rather than creating another+ server against the same data.++ Running state is not proof of healthy collection or alert delivery. Check+ normal observation times in Pulse without Run Diagnostics or guest-agent+ probes. Restore only automatic jobs that were active before the pause, after+ verifying they will not start a previously inactive server. Unknown original+ job state or restart behaviour means leave that job paused. Repeat the+ precaution for each affected backup window; this is not proof of a fix.+## π SetupPlan changes outside backup windows and follow the guest's normal maintenancediff --git a/docs/release-control/v6/internal/subsystems/frontend-primitives.md b/docs/release-control/v6/internal/subsystems/frontend-primitives.mdindex e5e4e0b25b..77dc13349a 100644--- a/docs/release-control/v6/internal/subsystems/frontend-primitives.md+++ b/docs/release-control/v6/internal/subsystems/frontend-primitives.md@@ -20,6 +20,26 @@## Purpose+### Container backup-pause help preserves identity and prior state++The existing VM Disk Monitoring guide also covers a standalone Docker or plain+Compose Pulse server, linked from the Docker guide and the systemd precaution.+It identifies the original server container(s), discloses the monitoring/alert+outage, requires suspended external restart controls and same-ID stopped-state+readback before a planned backup. Stopped state never proves an earlier guest+request completed. Incident recovery and controller-managed replacement are+outside this manual sequence.++Restoration requires independent post-backup thaw, fresh successful workload+writes on every covered filesystem and liveness. Only previously running,+existing container IDs and previously active automatic jobs may be restored;+unknown state, missing IDs, failed startup or unknown restart behaviour keep+automation paused. No image, restart-policy, mounted data, guest operation or+diagnostic permission changes. Bounded state templates omit environment/config+secrets. `dockerBackupSafetyHelp.test.ts` exercises rendered commands, gates,+cross-links and shipped mirrors; this is help validation, not a Docker daemon,+installed guest recovery, native acceptance or release proof.+### Narrow Workloads identity during guest locksIn narrow and phone layouts, the guest name and status occupy the first line;diff --git a/frontend-modern/public/docs/DOCKER.md b/frontend-modern/public/docs/DOCKER.mdindex a08442f8c2..313d0f229e 100644--- a/frontend-modern/public/docs/DOCKER.md+++ b/frontend-modern/public/docs/DOCKER.md@@ -52,6 +52,12 @@ volumes:Run with: `docker compose up -d`+**Freeze-enabled VM backups:** API-only monitoring can still send QEMU Guest+Agent requests. For an affected installation, follow the+[planned server pause and checked restoration](VM_DISK_MONITORING.md#pause-a-docker-or-compose-server-for-a-planned-backup),+not container recreation or guest-agent probes. Monitoring and alerts are+unavailable while Pulse is stopped; an OK backup alone does not prove recovery.+Leave authentication overrides unset for a new install and complete[bootstrap-token setup](INSTALL.md#step-1-get-the-token) in your browser. Do not add ashared example password to the Compose file. If automation must skip setup,diff --git a/frontend-modern/public/docs/VM_DISK_MONITORING.md b/frontend-modern/public/docs/VM_DISK_MONITORING.mdindex abf24f75d5..09480663a1 100644--- a/frontend-modern/public/docs/VM_DISK_MONITORING.md+++ b/frontend-modern/public/docs/VM_DISK_MONITORING.md@@ -32,6 +32,8 @@ redacted error, rather than repeat a potentially harmful backup for a report.This manual precaution is for an affected **systemd installation**, before ascheduled backup starts. It is **not a recovery procedure** for a backup alreadyin progress or an unresponsive guest, and **not an automatic backup hook**.+For a Docker or Compose server, use the [container pause procedure](#pause-a-docker-or-compose-server-for-a-planned-backup) instead;+the systemd commands below do not stop a containerised Pulse server.1. **Identify the Pulse server service.** Run these commands on the machinerunning the Pulse server: for Proxmox LXC, **inside the Pulse LXC**,@@ -135,6 +137,112 @@ in progress or an unresponsive guest, and **not an automatic backup hook**.outage coverage and repeat the precaution for each affected backup window;this manual sequence does not schedule future pauses or prove a fix.+### Pause a Docker or Compose server for a planned backup++This is a manual precaution for an affected **standalone Docker or plain Compose+Pulse server**, before a scheduled freeze-enabled backup. It is **not incident+recovery or an automatic backup hook**. Swarm, Kubernetes and other controllers+need their own supported pause controls: stopping a container alone does not+stop a controller from replacing it. Do not use this procedure if you cannot+prevent another server from polling the affected guests.++**Pulse monitoring and alerts are unavailable while stopped.** Arrange+independent outage coverage. A container stop does not cancel a guest-agent+request already issued; let any existing guest/backup operation finish normally+before the planned backup. If that state is unknown, do not start the backup on+the strength of a stopped container.++1. **Identify the existing Pulse server container on its Docker host**, not a+ monitored workload, the backed-up VM or a Pulse Agent container. For Compose,+ use the original project directory and the same project/file options used to+ deploy it. The example service is `pulse`; substitute your actual service:++ ```bash+ docker compose ps --all --quiet pulse+ ```++ Record every returned server container ID privately. An empty list or a+ failed command is not proof that monitoring has stopped. For a `docker run`+ installation, identify the existing server through its saved deployment.+ Set the ID explicitly, then read only its identity and state:++ ```bash+ PULSE_CONTAINER_ID='paste-the-recorded-server-container-id'+ docker inspect --format 'Id={{.Id}} Running={{.State.Running}} Paused={{.State.Paused}} Restarting={{.State.Restarting}} Status={{.State.Status}} Pid={{.State.Pid}} ExitCode={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}' "$PULSE_CONTAINER_ID"+ ```++ Record whether this exact container was running before the pause. Require a+ stable state: either running and not paused/restarting, or already exited+ with PID 0. If it is paused, restarting, being replaced or its state cannot+ be read, resolve that through the deployment's normal controls first. Repeat+ the state check for every server; do not infer another container's state.+ Do not share full `docker inspect`, container environments or resolved+ Compose configuration: they can contain credentials.++2. **Prevent a deployment or updater from restarting/replacing Pulse.** Do not+ install, update, recreate the server, run `docker compose up` or reboot the+ Docker host during this window. Pause any automatic updater or deployment+ job using its supported controls, recording whether it was active first.+ Let an in-progress update finish normally; do not interrupt an installation.+ The supplied Compose example uses `restart: unless-stopped`, but that does+ not prevent an external updater or another operator from starting Pulse.+ Do not change the restart policy, image, mounts or saved deployment as a+ substitute for these checks.++3. **Stop each previously running server and verify the same ID is stopped.**+ Docker commands on the recorded existing container also work for a plain+ Compose deployment; they do not recreate its configuration or data volume:++ ```bash+ docker stop --timeout 60 "$PULSE_CONTAINER_ID"+ docker inspect --format 'Id={{.Id}} Running={{.State.Running}} Paused={{.State.Paused}} Restarting={{.State.Restarting}} Status={{.State.Status}} Pid={{.State.Pid}} ExitCode={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}' "$PULSE_CONTAINER_ID"+ ```++ Require a successful stop and readback of the **same recorded ID** with+ `Running=false`, `Paused=false`, `Restarting=false`, `Status=exited` and+ `Pid=0`. For a server stopped in this step, also require `ExitCode=0` and+ `OOMKilled=false`; a non-zero exit (including 137 after forced termination)+ or unknown shutdown result needs normal maintenance before the backup.+ Leave an already stopped server stopped. If stopping times out,+ fails, the identity changes or any state is unknown, do not start the backup.+ Do not delete/recreate a container, remove volumes or force-kill it to clear+ this check. Docker can terminate a process after its stop timeout; stopped+ state alone does not prove graceful shutdown or completion of earlier guest+ requests. Preserve a shutdown failure for normal maintenance, not a forced+ backup test.++4. **Keep every server stopped until independent recovery checks pass.** The+ backup must have ended, and evidence from **after it ended**, independent of+ Pulse and the QEMU Guest Agent, must confirm **thaw, fresh successful workload+ writes to every filesystem covered by the backup, and workload liveness**.+ Use the workload's established safe checks, not forced writes, test files or+ another freeze/thaw cycle. An OK backup, an absent lock, a console connection+ or a successful read alone is not enough. If any check fails or is unavailable,+ leave Pulse and its automatic updater paused and use the platform's recovery+ procedure, without guest-agent probes or repeating the backup.++ Only for a container recorded as running before the pause, start that+ **same existing ID**, then repeat the bounded state readback from step 1:++ ```bash+ docker start "$PULSE_CONTAINER_ID"+ ```++ Require the original ID, `Running=true`, `Paused=false`, `Restarting=false`+ and `Status=running`. A previously stopped container stays stopped; if its+ original state is unknown, do not guess. Do not use a blanket Compose start+ or `up` to restore a mixture of previously running and stopped servers. If+ the original container is missing or startup fails, keep automation paused+ and resolve that through normal maintenance rather than creating another+ server against the same data.++ Running state is not proof of healthy collection or alert delivery. Check+ normal observation times in Pulse without Run Diagnostics or guest-agent+ probes. Restore only automatic jobs that were active before the pause, after+ verifying they will not start a previously inactive server. Unknown original+ job state or restart behaviour means leave that job paused. Repeat the+ precaution for each affected backup window; this is not proof of a fix.+## π SetupPlan changes outside backup windows and follow the guest's normal maintenancediff --git a/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts b/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.tsindex 03df331959..e903009c48 100644--- a/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts+++ b/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts@@ -8,8 +8,9 @@ const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../.const read = (name: string) => readFileSync(path.join(root, name), 'utf8');const guide = read('docs/VM_DISK_MONITORING.md');const section =- guide.split('### Pause Pulse for a planned freeze-enabled backup')[1]?.split('## π Setup')[0] ??- '';+ guide+ .split('### Pause Pulse for a planned freeze-enabled backup')[1]+ ?.split('### Pause a Docker or Compose server for a planned backup')[0] ?? '';const words = section.replace(/\s+/g, ' ');function render(markdown: string, name: string): HTMLElement {diff --git a/frontend-modern/src/utils/__tests__/dockerBackupSafetyHelp.test.ts b/frontend-modern/src/utils/__tests__/dockerBackupSafetyHelp.test.tsnew file mode 100644index 0000000000..aa6bd0d9b7--- /dev/null+++ b/frontend-modern/src/utils/__tests__/dockerBackupSafetyHelp.test.ts@@ -0,0 +1,130 @@+import { readFileSync } from 'node:fs';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { describe, expect, it } from 'vitest';+import { renderDocMarkdown } from '@/features/docs/docMarkdown';++const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');+const read = (name: string) => readFileSync(path.join(root, name), 'utf8');+const guide = read('docs/VM_DISK_MONITORING.md');+const heading = '### Pause a Docker or Compose server for a planned backup';+const fragment = 'pause-a-docker-or-compose-server-for-a-planned-backup';+const section = guide.split(heading)[1]?.split('## π Setup')[0] ?? '';++function render(markdown: string, name = 'VM_DISK_MONITORING'): HTMLElement {+ const article = document.createElement('article');+ article.innerHTML = renderDocMarkdown(markdown, name);+ return article;+}++const text = () => render(section).textContent?.replace(/\s+/g, ' ');+const stateCommand =+ 'docker inspect --format \'Id={{.Id}} Running={{.State.Running}} Paused={{.State.Paused}} Restarting={{.State.Restarting}} Status={{.State.Status}} Pid={{.State.Pid}} ExitCode={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}\' "$PULSE_CONTAINER_ID"';++describe('Docker backup-pause help', () => {+ it('ships the procedure and routes both Docker and systemd readers to its real heading', () => {+ expect(section).not.toBe('');+ expect(read('frontend-modern/public/docs/VM_DISK_MONITORING.md')).toBe(guide);+ const docker = read('docs/DOCKER.md');+ expect(read('frontend-modern/public/docs/DOCKER.md')).toBe(docker);+ expect(render(guide).querySelector(`#${fragment}`)?.textContent).toBe(+ heading.replace('### ', ''),+ );+ expect(+ render(docker, 'DOCKER').querySelector(`a[href="/docs/VM_DISK_MONITORING#${fragment}"]`)+ ?.textContent,+ ).toBe('planned server pause and checked restoration');+ expect(render(guide).querySelector(`a[href="#${fragment}"]`)?.textContent).toBe(+ 'container pause procedure',+ );+ expect(render(docker, 'DOCKER').textContent).toContain(+ 'an OK backup alone does not prove recovery',+ );+ });++ it('identifies the server and every existing ID without exposing environment or configuration', () => {+ expect(text()).toContain('on its Docker host');+ expect(text()).toContain(+ 'not a monitored workload, the backed-up VM or a Pulse Agent container',+ );+ expect(text()).toContain('original project directory');+ expect(text()).toContain('same project/file options');+ expect(text()).toContain('Record every returned server container ID privately');+ expect(text()).toContain('An empty list or a failed command is not proof');+ expect(text()).toContain('Repeat the state check for every server');+ expect(text()).toContain('Do not share full docker inspect');+ const commands = [...render(section).querySelectorAll('pre code')].map((e) => e.textContent);+ expect(commands).toEqual([+ 'docker compose ps --all --quiet pulse\n',+ `PULSE_CONTAINER_ID='paste-the-recorded-server-container-id'\n${stateCommand}\n`,+ `docker stop --timeout 60 "$PULSE_CONTAINER_ID"\n${stateCommand}\n`,+ 'docker start "$PULSE_CONTAINER_ID"\n',+ ]);+ expect(commands.join('\n')).not.toMatch(+ /\.Config|\.Mounts|\.Env|compose config|\brm\b|\bkill\b|\bexec\b|\brun\b|\bup\b|\bdown\b|\bpull\b|\bprune\b|qm |pct |pvesh /,+ );+ });++ it('requires stopped identity, shutdown evidence and restart controls rather than equating a stop with safety', () => {+ for (const state of [+ 'same recorded ID',+ 'Running=false',+ 'Paused=false',+ 'Restarting=false',+ 'Status=exited',+ 'Pid=0',+ 'ExitCode=0',+ 'OOMKilled=false',+ ])+ expect(text()).toContain(state);+ expect(text()).toContain('including 137 after forced termination');+ expect(text()).toContain('identity changes or any state is unknown, do not start the backup');+ expect(text()).toContain(+ 'A container stop does not cancel a guest-agent request already issued',+ );+ expect(text()).toContain('do not start the backup on the strength of a stopped container');+ expect(text()).toContain('Swarm, Kubernetes and other controllers');+ expect(text()).toContain('does not stop a controller from replacing it');+ expect(text()).toContain('does not prevent an external updater or another operator');+ expect(text()).toContain('Do not change the restart policy, image, mounts or saved deployment');+ expect(text()).toContain('Let an in-progress update finish normally');+ expect(read('docker-compose.yml')).toContain('restart: unless-stopped');+ expect(read('docker-compose.yml')).toContain('container_name: pulse');+ });++ it('gates restoration on independent post-backup thaw, every covered write and workload liveness', () => {+ expect(text()).toContain(+ 'evidence from after it ended, independent of Pulse and the QEMU Guest Agent',+ );+ expect(text()).toContain(+ 'thaw, fresh successful workload writes to every filesystem covered by the backup, and workload liveness',+ );+ expect(text()).toContain('not forced writes, test files or another freeze/thaw cycle');+ expect(text()).toContain('a console connection or a successful read alone is not enough');+ expect(text()).toContain(+ 'If any check fails or is unavailable, leave Pulse and its automatic updater paused',+ );+ expect(text()).toContain('without guest-agent probes or repeating the backup');+ expect(text()).toContain('not incident recovery or an automatic backup hook');+ expect(text()).toContain('Pulse monitoring and alerts are unavailable');+ expect(text()).toContain('independent outage coverage');+ });++ it('restores only prior-running identities and prior-active jobs, never a replacement or inactive peer', () => {+ expect(text()).toContain('Only for a container recorded as running before the pause');+ expect(text()).toContain('same existing ID');+ expect(text()).toContain('A previously stopped container stays stopped');+ expect(text()).toContain('original state is unknown, do not guess');+ expect(text()).toContain('Do not use a blanket Compose start or up');+ expect(text()).toContain(+ 'If the original container is missing or startup fails, keep automation paused',+ );+ expect(text()).toContain('rather than creating another server against the same data');+ expect(text()).toContain('Running state is not proof of healthy collection or alert delivery');+ expect(text()).toContain('Restore only automatic jobs that were active before the pause');+ expect(text()).toContain('they will not start a previously inactive server');+ expect(text()).toContain(+ 'Unknown original job state or restart behaviour means leave that job paused',+ );+ });+});