Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Explain controller-aware backup safety for Kubernetes servers
pulse-triage[bot]0b1ce1b2529b
diff --git a/docs/KUBERNETES.md b/docs/KUBERNETES.mdindex d768315f00..caafb47cb4 100644--- a/docs/KUBERNETES.md+++ b/docs/KUBERNETES.md@@ -10,6 +10,62 @@ This guide explains how to deploy the Pulse Server (Hub) and Pulse Agents on Kub- `helm` (v3+) installed locally- `kubectl` configured to talk to your cluster+## Proxmox backup safety++**A Kubernetes-hosted Pulse server can still send QEMU Guest Agent requests to+Proxmox VMs, including with API-only monitoring.** The Kubernetes-only collector+below is a separate component: stopping its DaemonSet does not stop the server's+Proxmox polling. Read [Backup safety](VM_DISK_MONITORING.md#backup-safety) before+changing an affected installation.++For an affected server, prepare a controller-aware pause **before a planned+freeze-enabled backup**, using your cluster's normal maintenance procedure.+This is not incident recovery or an automatic backup hook. If a backup is+already in progress, a guest is unresponsive or an operation's state is unknown,+do not restart pods or send guest-agent probes to diagnose it. Use the+guest/platform's recovery procedure and preserve the original evidence.++1. **Identify every Pulse server and its owner.** Confirm the cluster,+ namespace, controller and existing pods for each server polling the affected+ guests, including servers outside this cluster. Record the actual desired+ replica count and whether each deployment controller, autoscaler, GitOps+ reconciler and automatic updater was active. Keep that record private.+ The chart's default is not evidence of your installed state. An unknown+ owner or another polling server prevents a confirmed pause.+2. **Prevent replacement, then stop the server through its owning controller.**+ Use the controller's supported maintenance controls, including suspending+ reconciliation or autoscaling where needed, before deliberately scaling the+ server to zero. Deleting a pod only invites its controller to replace it;+ pausing a Deployment rollout does not stop existing pods. Confirm the+ desired count remains zero, every previous server process has terminated+ normally and no replacement is polling. A force-deleted, terminating or+ unreachable pod, a failed shutdown or missing termination evidence is not+ a confirmed stop. Do not force deletion, uninstall Helm, delete PVCs or+ recreate Secrets to satisfy this check. **Pulse monitoring and alerts are+ unavailable while stopped**; arrange independent outage coverage.+3. **Keep the pause until independent recovery checks pass.** Stopping the+ server does not cancel a guest-agent request already issued. Let existing+ operations finish normally; if their state is unknown, do not start the+ planned backup on the strength of zero replicas or an elapsed wait. After+ the backup ends, established checks independent of Pulse and the QEMU Guest+ Agent must confirm **thaw, fresh successful workload writes to every+ filesystem covered by the backup, and workload liveness**. An OK task, an+ absent lock, a console connection or a write to only the OS disk is not+ enough. If any check fails or is unavailable, keep the pause and use the+ guest/platform's recovery procedure, without a new backup or probe.+4. **Restore only the recorded prior state after all checks pass.** Restore+ each server's original desired count and only the reconciliation,+ autoscaling and updater controls that were active beforehand. A previously+ zero-replica server stays at zero; an unknown prior state is not permission+ to start it. Preserve the existing image, PVCs, configuration, identities+ and credentials. Confirm the intended server is ready and monitoring is+ fresh before relying on alerts again. Do not use an install command from+ this page as a restart or recovery shortcut.++If your cluster cannot establish these conditions, do not treat a deleted pod+or a successful scale request as permission to start the planned backup. This+guidance does not establish that a native freeze/thaw defect has been fixed.+## 1. Deploying the Pulse ServerThe Pulse Server is the central hub that collects metrics and manages agents.@@ -438,6 +494,10 @@ Talos Linux is immutable, so you cannot install the agent via the shell script.## 4. Troubleshooting+For a Proxmox backup, freeze/thaw or unresponsive-guest incident, the+[backup safety checks](#proxmox-backup-safety) take precedence over setup or+pod restarts. A Kubernetes pod's Ready state does not prove guest thaw.+Start with the affected pod's existing status and a bounded, locally reviewedlog excerpt around the original failure. Missing usage, missing inventory anda rejected Pulse report are different failures:diff --git a/docs/VM_DISK_MONITORING.md b/docs/VM_DISK_MONITORING.mdindex 506758fb18..0ea02ac9db 100644--- a/docs/VM_DISK_MONITORING.md+++ b/docs/VM_DISK_MONITORING.md@@ -165,7 +165,9 @@ This is a manual precaution for an affected **standalone Docker or plain ComposePulse server**, before a scheduled freeze-enabled backup. It is **not incidentrecovery or an automatic backup hook**. Swarm, Kubernetes and other controllersneed their own supported pause controls: stopping a container alone does not-stop a controller from replacing it. Do not use this procedure if you cannot+stop a controller from replacing it. For Kubernetes, start with the+[controller-aware backup safety checks](KUBERNETES.md#proxmox-backup-safety),+not the Docker commands below. Do not use this procedure if you cannotprevent another server from polling the affected guests.**Pulse monitoring and alerts are unavailable while stopped.** Arrangediff --git a/frontend-modern/public/docs/KUBERNETES.md b/frontend-modern/public/docs/KUBERNETES.mdindex d768315f00..caafb47cb4 100644--- a/frontend-modern/public/docs/KUBERNETES.md+++ b/frontend-modern/public/docs/KUBERNETES.md@@ -10,6 +10,62 @@ This guide explains how to deploy the Pulse Server (Hub) and Pulse Agents on Kub- `helm` (v3+) installed locally- `kubectl` configured to talk to your cluster+## Proxmox backup safety++**A Kubernetes-hosted Pulse server can still send QEMU Guest Agent requests to+Proxmox VMs, including with API-only monitoring.** The Kubernetes-only collector+below is a separate component: stopping its DaemonSet does not stop the server's+Proxmox polling. Read [Backup safety](VM_DISK_MONITORING.md#backup-safety) before+changing an affected installation.++For an affected server, prepare a controller-aware pause **before a planned+freeze-enabled backup**, using your cluster's normal maintenance procedure.+This is not incident recovery or an automatic backup hook. If a backup is+already in progress, a guest is unresponsive or an operation's state is unknown,+do not restart pods or send guest-agent probes to diagnose it. Use the+guest/platform's recovery procedure and preserve the original evidence.++1. **Identify every Pulse server and its owner.** Confirm the cluster,+ namespace, controller and existing pods for each server polling the affected+ guests, including servers outside this cluster. Record the actual desired+ replica count and whether each deployment controller, autoscaler, GitOps+ reconciler and automatic updater was active. Keep that record private.+ The chart's default is not evidence of your installed state. An unknown+ owner or another polling server prevents a confirmed pause.+2. **Prevent replacement, then stop the server through its owning controller.**+ Use the controller's supported maintenance controls, including suspending+ reconciliation or autoscaling where needed, before deliberately scaling the+ server to zero. Deleting a pod only invites its controller to replace it;+ pausing a Deployment rollout does not stop existing pods. Confirm the+ desired count remains zero, every previous server process has terminated+ normally and no replacement is polling. A force-deleted, terminating or+ unreachable pod, a failed shutdown or missing termination evidence is not+ a confirmed stop. Do not force deletion, uninstall Helm, delete PVCs or+ recreate Secrets to satisfy this check. **Pulse monitoring and alerts are+ unavailable while stopped**; arrange independent outage coverage.+3. **Keep the pause until independent recovery checks pass.** Stopping the+ server does not cancel a guest-agent request already issued. Let existing+ operations finish normally; if their state is unknown, do not start the+ planned backup on the strength of zero replicas or an elapsed wait. After+ the backup ends, established checks independent of Pulse and the QEMU Guest+ Agent must confirm **thaw, fresh successful workload writes to every+ filesystem covered by the backup, and workload liveness**. An OK task, an+ absent lock, a console connection or a write to only the OS disk is not+ enough. If any check fails or is unavailable, keep the pause and use the+ guest/platform's recovery procedure, without a new backup or probe.+4. **Restore only the recorded prior state after all checks pass.** Restore+ each server's original desired count and only the reconciliation,+ autoscaling and updater controls that were active beforehand. A previously+ zero-replica server stays at zero; an unknown prior state is not permission+ to start it. Preserve the existing image, PVCs, configuration, identities+ and credentials. Confirm the intended server is ready and monitoring is+ fresh before relying on alerts again. Do not use an install command from+ this page as a restart or recovery shortcut.++If your cluster cannot establish these conditions, do not treat a deleted pod+or a successful scale request as permission to start the planned backup. This+guidance does not establish that a native freeze/thaw defect has been fixed.+## 1. Deploying the Pulse ServerThe Pulse Server is the central hub that collects metrics and manages agents.@@ -438,6 +494,10 @@ Talos Linux is immutable, so you cannot install the agent via the shell script.## 4. Troubleshooting+For a Proxmox backup, freeze/thaw or unresponsive-guest incident, the+[backup safety checks](#proxmox-backup-safety) take precedence over setup or+pod restarts. A Kubernetes pod's Ready state does not prove guest thaw.+Start with the affected pod's existing status and a bounded, locally reviewedlog excerpt around the original failure. Missing usage, missing inventory anda rejected Pulse report are different failures:diff --git a/frontend-modern/public/docs/VM_DISK_MONITORING.md b/frontend-modern/public/docs/VM_DISK_MONITORING.mdindex 506758fb18..0ea02ac9db 100644--- a/frontend-modern/public/docs/VM_DISK_MONITORING.md+++ b/frontend-modern/public/docs/VM_DISK_MONITORING.md@@ -165,7 +165,9 @@ This is a manual precaution for an affected **standalone Docker or plain ComposePulse server**, before a scheduled freeze-enabled backup. It is **not incidentrecovery or an automatic backup hook**. Swarm, Kubernetes and other controllersneed their own supported pause controls: stopping a container alone does not-stop a controller from replacing it. Do not use this procedure if you cannot+stop a controller from replacing it. For Kubernetes, start with the+[controller-aware backup safety checks](KUBERNETES.md#proxmox-backup-safety),+not the Docker commands below. Do not use this procedure if you cannotprevent another server from polling the affected guests.**Pulse monitoring and alerts are unavailable while stopped.** Arrangediff --git a/frontend-modern/src/features/docs/__tests__/kubernetesBackupSafetyDocs.test.ts b/frontend-modern/src/features/docs/__tests__/kubernetesBackupSafetyDocs.test.tsnew file mode 100644index 0000000000..661ef860fb--- /dev/null+++ b/frontend-modern/src/features/docs/__tests__/kubernetesBackupSafetyDocs.test.ts@@ -0,0 +1,123 @@+import { readFileSync } from 'node:fs';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { describe, expect, it } from 'vitest';+import { renderDocMarkdown } from '../docMarkdown';++const frontendRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');+const repoRoot = path.resolve(frontendRoot, '..');+const readDoc = (name: string) => readFileSync(path.join(repoRoot, 'docs', `${name}.md`), 'utf8');+const articleFor = (name: string) => {+ const article = document.createElement('article');+ article.innerHTML = renderDocMarkdown(readDoc(name), name);+ return article;+};++function safetySection() {+ const article = articleFor('KUBERNETES');+ const heading = article.querySelector('#proxmox-backup-safety');+ expect(heading).not.toBeNull();+ const section = document.createElement('section');+ for (let node = heading!.nextElementSibling; node && node.tagName !== 'H2';) {+ const next = node.nextElementSibling;+ section.appendChild(node);+ node = next;+ }+ return section;+}++const text = (element: Element) => element.textContent!.replace(/\s+/g, ' ');++describe('Kubernetes server backup safety in shipped documentation', () => {+ it('puts server safety before installation, separately from the Kubernetes-only collector', () => {+ const article = articleFor('KUBERNETES');+ const headings = [...article.querySelectorAll('h2')].map((heading) => heading.id);+ expect(headings.indexOf('proxmox-backup-safety')).toBeGreaterThanOrEqual(0);+ expect(headings.indexOf('proxmox-backup-safety')).toBeLessThan(+ headings.indexOf('1-deploying-the-pulse-server'),+ );+ const safety = safetySection();+ expect(text(safety)).toContain('including with API-only monitoring');+ expect(text(safety)).toContain("stopping its DaemonSet does not stop the server's Proxmox polling");+ expect(safety.querySelectorAll('ol > li')).toHaveLength(4);+ // No copied scale/delete/Helm command can discard the required identity,+ // reconciliation and prior-state checks when used on a different cluster.+ expect(safety.querySelector('pre, code')).toBeNull();+ });++ it('requires a controller-aware stop, not pod deletion or a paused rollout', () => {+ const steps = [...safetySection().querySelectorAll('ol > li')].map(text);+ for (const phrase of [+ 'every Pulse server',+ 'servers outside this cluster',+ 'actual desired replica count',+ 'autoscaler, GitOps reconciler and automatic updater',+ 'unknown owner or another polling server',+ ])+ expect(steps[0]).toContain(phrase);+ for (const phrase of [+ 'before deliberately scaling the server to zero',+ 'Deleting a pod only invites its controller to replace it',+ 'pausing a Deployment rollout does not stop existing pods',+ 'every previous server process has terminated normally',+ 'no replacement is polling',+ 'missing termination evidence is not a confirmed stop',+ 'Do not force deletion, uninstall Helm, delete PVCs or recreate Secrets',+ 'Pulse monitoring and alerts are unavailable while stopped',+ ])+ expect(steps[1]).toContain(phrase);+ });++ it('keeps unknown operations stopped and requires independent every-filesystem recovery', () => {+ const section = safetySection();+ const step = text(section.querySelectorAll('ol > li')[2]);+ for (const phrase of [+ 'does not cancel a guest-agent request already issued',+ 'if their state is unknown, do not start the planned backup',+ 'After the backup ends',+ 'independent of Pulse and the QEMU Guest Agent',+ 'thaw, fresh successful workload writes to every filesystem covered by the backup, and workload liveness',+ 'a write to only the OS disk is not enough',+ 'If any check fails or is unavailable, keep the pause',+ 'without a new backup or probe',+ ])+ expect(step).toContain(phrase);+ expect(text(section)).toContain('not incident recovery or an automatic backup hook');+ expect(text(section)).toContain('do not restart pods or send guest-agent probes');+ expect(text(section)).toContain(+ 'does not establish that a native freeze/thaw defect has been fixed',+ );+ });++ it('restores actual prior state without replaying installation or losing data and credentials', () => {+ const step = text(safetySection().querySelectorAll('ol > li')[3]);+ for (const phrase of [+ 'after all checks pass',+ "each server's original desired count",+ 'controls that were active beforehand',+ 'previously zero-replica server stays at zero',+ 'unknown prior state is not permission to start it',+ 'Preserve the existing image, PVCs, configuration, identities and credentials',+ 'monitoring is fresh before relying on alerts again',+ 'Do not use an install command from this page as a restart or recovery shortcut',+ ])+ expect(step).toContain(phrase);+ });++ it('connects both deployment guides and troubleshooting to real shipped safety headings', () => {+ const kubernetes = articleFor('KUBERNETES');+ const vm = articleFor('VM_DISK_MONITORING');+ expect(kubernetes.querySelector('a[href="/docs/VM_DISK_MONITORING#backup-safety"]')).not.toBeNull();+ expect(vm.querySelector('#backup-safety')).not.toBeNull();+ expect(vm.querySelector('a[href="/docs/KUBERNETES#proxmox-backup-safety"]')).not.toBeNull();+ const troubleshooting = kubernetes.querySelector('[id="4-troubleshooting"]')!;+ const warning = troubleshooting.nextElementSibling!;+ expect(warning.querySelector('a[href="#proxmox-backup-safety"]')).not.toBeNull();+ expect(text(warning)).toContain("A Kubernetes pod's Ready state does not prove guest thaw");+ for (const name of ['KUBERNETES', 'VM_DISK_MONITORING']) {+ expect(readFileSync(path.join(frontendRoot, 'public/docs', `${name}.md`), 'utf8')).toBe(+ readDoc(name),+ );+ }+ });+});