Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Explain controller-aware backup safety for Kubernetes servers

pulse-triage[bot]0b1ce1b2529b
diff --git a/docs/KUBERNETES.md b/docs/KUBERNETES.md
index d768315f00..caafb47cb4 100644
--- a/docs/KUBERNETES.md
+++ b/docs/KUBERNETES.md
@@ -10,6 +10,62 @@ This guide explains how to deploy the Pulse Server (Hub) and Pulse Agents on Kub
- `helm` (v3+) installed locally
- `kubectl` configured to talk to your cluster
+## Proxmox backup safety
+
+**A Kubernetes-hosted Pulse server can still send QEMU Guest Agent requests to
+Proxmox VMs, including with API-only monitoring.** The Kubernetes-only collector
+below is a separate component: stopping its DaemonSet does not stop the server's
+Proxmox polling. Read [Backup safety](VM_DISK_MONITORING.md#backup-safety) before
+changing an affected installation.
+
+For an affected server, prepare a controller-aware pause **before a planned
+freeze-enabled backup**, using your cluster's normal maintenance procedure.
+This is not incident recovery or an automatic backup hook. If a backup is
+already in progress, a guest is unresponsive or an operation's state is unknown,
+do not restart pods or send guest-agent probes to diagnose it. Use the
+guest/platform's recovery procedure and preserve the original evidence.
+
+1. **Identify every Pulse server and its owner.** Confirm the cluster,
+ namespace, controller and existing pods for each server polling the affected
+ guests, including servers outside this cluster. Record the actual desired
+ replica count and whether each deployment controller, autoscaler, GitOps
+ reconciler and automatic updater was active. Keep that record private.
+ The chart's default is not evidence of your installed state. An unknown
+ owner or another polling server prevents a confirmed pause.
+2. **Prevent replacement, then stop the server through its owning controller.**
+ Use the controller's supported maintenance controls, including suspending
+ reconciliation or autoscaling where needed, before deliberately scaling the
+ server to zero. Deleting a pod only invites its controller to replace it;
+ pausing a Deployment rollout does not stop existing pods. Confirm the
+ desired count remains zero, every previous server process has terminated
+ normally and no replacement is polling. A force-deleted, terminating or
+ unreachable pod, a failed shutdown or missing termination evidence is not
+ a confirmed stop. Do not force deletion, uninstall Helm, delete PVCs or
+ recreate Secrets to satisfy this check. **Pulse monitoring and alerts are
+ unavailable while stopped**; arrange independent outage coverage.
+3. **Keep the pause until independent recovery checks pass.** Stopping the
+ server does not cancel a guest-agent request already issued. Let existing
+ operations finish normally; if their state is unknown, do not start the
+ planned backup on the strength of zero replicas or an elapsed wait. After
+ the backup ends, established checks independent of Pulse and the QEMU Guest
+ Agent must confirm **thaw, fresh successful workload writes to every
+ filesystem covered by the backup, and workload liveness**. An OK task, an
+ absent lock, a console connection or a write to only the OS disk is not
+ enough. If any check fails or is unavailable, keep the pause and use the
+ guest/platform's recovery procedure, without a new backup or probe.
+4. **Restore only the recorded prior state after all checks pass.** Restore
+ each server's original desired count and only the reconciliation,
+ autoscaling and updater controls that were active beforehand. A previously
+ zero-replica server stays at zero; an unknown prior state is not permission
+ to start it. Preserve the existing image, PVCs, configuration, identities
+ and credentials. Confirm the intended server is ready and monitoring is
+ fresh before relying on alerts again. Do not use an install command from
+ this page as a restart or recovery shortcut.
+
+If your cluster cannot establish these conditions, do not treat a deleted pod
+or a successful scale request as permission to start the planned backup. This
+guidance does not establish that a native freeze/thaw defect has been fixed.
+
## 1. Deploying the Pulse Server
The Pulse Server is the central hub that collects metrics and manages agents.
@@ -438,6 +494,10 @@ Talos Linux is immutable, so you cannot install the agent via the shell script.
## 4. Troubleshooting
+For a Proxmox backup, freeze/thaw or unresponsive-guest incident, the
+[backup safety checks](#proxmox-backup-safety) take precedence over setup or
+pod restarts. A Kubernetes pod's Ready state does not prove guest thaw.
+
Start with the affected pod's existing status and a bounded, locally reviewed
log excerpt around the original failure. Missing usage, missing inventory and
a rejected Pulse report are different failures:
diff --git a/docs/VM_DISK_MONITORING.md b/docs/VM_DISK_MONITORING.md
index 506758fb18..0ea02ac9db 100644
--- a/docs/VM_DISK_MONITORING.md
+++ b/docs/VM_DISK_MONITORING.md
@@ -165,7 +165,9 @@ This is a manual precaution for an affected **standalone Docker or plain Compose
Pulse server**, before a scheduled freeze-enabled backup. It is **not incident
recovery or an automatic backup hook**. Swarm, Kubernetes and other controllers
need their own supported pause controls: stopping a container alone does not
-stop a controller from replacing it. Do not use this procedure if you cannot
+stop a controller from replacing it. For Kubernetes, start with the
+[controller-aware backup safety checks](KUBERNETES.md#proxmox-backup-safety),
+not the Docker commands below. Do not use this procedure if you cannot
prevent another server from polling the affected guests.
**Pulse monitoring and alerts are unavailable while stopped.** Arrange
diff --git a/frontend-modern/public/docs/KUBERNETES.md b/frontend-modern/public/docs/KUBERNETES.md
index d768315f00..caafb47cb4 100644
--- a/frontend-modern/public/docs/KUBERNETES.md
+++ b/frontend-modern/public/docs/KUBERNETES.md
@@ -10,6 +10,62 @@ This guide explains how to deploy the Pulse Server (Hub) and Pulse Agents on Kub
- `helm` (v3+) installed locally
- `kubectl` configured to talk to your cluster
+## Proxmox backup safety
+
+**A Kubernetes-hosted Pulse server can still send QEMU Guest Agent requests to
+Proxmox VMs, including with API-only monitoring.** The Kubernetes-only collector
+below is a separate component: stopping its DaemonSet does not stop the server's
+Proxmox polling. Read [Backup safety](VM_DISK_MONITORING.md#backup-safety) before
+changing an affected installation.
+
+For an affected server, prepare a controller-aware pause **before a planned
+freeze-enabled backup**, using your cluster's normal maintenance procedure.
+This is not incident recovery or an automatic backup hook. If a backup is
+already in progress, a guest is unresponsive or an operation's state is unknown,
+do not restart pods or send guest-agent probes to diagnose it. Use the
+guest/platform's recovery procedure and preserve the original evidence.
+
+1. **Identify every Pulse server and its owner.** Confirm the cluster,
+ namespace, controller and existing pods for each server polling the affected
+ guests, including servers outside this cluster. Record the actual desired
+ replica count and whether each deployment controller, autoscaler, GitOps
+ reconciler and automatic updater was active. Keep that record private.
+ The chart's default is not evidence of your installed state. An unknown
+ owner or another polling server prevents a confirmed pause.
+2. **Prevent replacement, then stop the server through its owning controller.**
+ Use the controller's supported maintenance controls, including suspending
+ reconciliation or autoscaling where needed, before deliberately scaling the
+ server to zero. Deleting a pod only invites its controller to replace it;
+ pausing a Deployment rollout does not stop existing pods. Confirm the
+ desired count remains zero, every previous server process has terminated
+ normally and no replacement is polling. A force-deleted, terminating or
+ unreachable pod, a failed shutdown or missing termination evidence is not
+ a confirmed stop. Do not force deletion, uninstall Helm, delete PVCs or
+ recreate Secrets to satisfy this check. **Pulse monitoring and alerts are
+ unavailable while stopped**; arrange independent outage coverage.
+3. **Keep the pause until independent recovery checks pass.** Stopping the
+ server does not cancel a guest-agent request already issued. Let existing
+ operations finish normally; if their state is unknown, do not start the
+ planned backup on the strength of zero replicas or an elapsed wait. After
+ the backup ends, established checks independent of Pulse and the QEMU Guest
+ Agent must confirm **thaw, fresh successful workload writes to every
+ filesystem covered by the backup, and workload liveness**. An OK task, an
+ absent lock, a console connection or a write to only the OS disk is not
+ enough. If any check fails or is unavailable, keep the pause and use the
+ guest/platform's recovery procedure, without a new backup or probe.
+4. **Restore only the recorded prior state after all checks pass.** Restore
+ each server's original desired count and only the reconciliation,
+ autoscaling and updater controls that were active beforehand. A previously
+ zero-replica server stays at zero; an unknown prior state is not permission
+ to start it. Preserve the existing image, PVCs, configuration, identities
+ and credentials. Confirm the intended server is ready and monitoring is
+ fresh before relying on alerts again. Do not use an install command from
+ this page as a restart or recovery shortcut.
+
+If your cluster cannot establish these conditions, do not treat a deleted pod
+or a successful scale request as permission to start the planned backup. This
+guidance does not establish that a native freeze/thaw defect has been fixed.
+
## 1. Deploying the Pulse Server
The Pulse Server is the central hub that collects metrics and manages agents.
@@ -438,6 +494,10 @@ Talos Linux is immutable, so you cannot install the agent via the shell script.
## 4. Troubleshooting
+For a Proxmox backup, freeze/thaw or unresponsive-guest incident, the
+[backup safety checks](#proxmox-backup-safety) take precedence over setup or
+pod restarts. A Kubernetes pod's Ready state does not prove guest thaw.
+
Start with the affected pod's existing status and a bounded, locally reviewed
log excerpt around the original failure. Missing usage, missing inventory and
a rejected Pulse report are different failures:
diff --git a/frontend-modern/public/docs/VM_DISK_MONITORING.md b/frontend-modern/public/docs/VM_DISK_MONITORING.md
index 506758fb18..0ea02ac9db 100644
--- a/frontend-modern/public/docs/VM_DISK_MONITORING.md
+++ b/frontend-modern/public/docs/VM_DISK_MONITORING.md
@@ -165,7 +165,9 @@ This is a manual precaution for an affected **standalone Docker or plain Compose
Pulse server**, before a scheduled freeze-enabled backup. It is **not incident
recovery or an automatic backup hook**. Swarm, Kubernetes and other controllers
need their own supported pause controls: stopping a container alone does not
-stop a controller from replacing it. Do not use this procedure if you cannot
+stop a controller from replacing it. For Kubernetes, start with the
+[controller-aware backup safety checks](KUBERNETES.md#proxmox-backup-safety),
+not the Docker commands below. Do not use this procedure if you cannot
prevent another server from polling the affected guests.
**Pulse monitoring and alerts are unavailable while stopped.** Arrange
diff --git a/frontend-modern/src/features/docs/__tests__/kubernetesBackupSafetyDocs.test.ts b/frontend-modern/src/features/docs/__tests__/kubernetesBackupSafetyDocs.test.ts
new file mode 100644
index 0000000000..661ef860fb
--- /dev/null
+++ b/frontend-modern/src/features/docs/__tests__/kubernetesBackupSafetyDocs.test.ts
@@ -0,0 +1,123 @@
+import { readFileSync } from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { describe, expect, it } from 'vitest';
+import { renderDocMarkdown } from '../docMarkdown';
+
+const frontendRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');
+const repoRoot = path.resolve(frontendRoot, '..');
+const readDoc = (name: string) => readFileSync(path.join(repoRoot, 'docs', `${name}.md`), 'utf8');
+const articleFor = (name: string) => {
+ const article = document.createElement('article');
+ article.innerHTML = renderDocMarkdown(readDoc(name), name);
+ return article;
+};
+
+function safetySection() {
+ const article = articleFor('KUBERNETES');
+ const heading = article.querySelector('#proxmox-backup-safety');
+ expect(heading).not.toBeNull();
+ const section = document.createElement('section');
+ for (let node = heading!.nextElementSibling; node && node.tagName !== 'H2';) {
+ const next = node.nextElementSibling;
+ section.appendChild(node);
+ node = next;
+ }
+ return section;
+}
+
+const text = (element: Element) => element.textContent!.replace(/\s+/g, ' ');
+
+describe('Kubernetes server backup safety in shipped documentation', () => {
+ it('puts server safety before installation, separately from the Kubernetes-only collector', () => {
+ const article = articleFor('KUBERNETES');
+ const headings = [...article.querySelectorAll('h2')].map((heading) => heading.id);
+ expect(headings.indexOf('proxmox-backup-safety')).toBeGreaterThanOrEqual(0);
+ expect(headings.indexOf('proxmox-backup-safety')).toBeLessThan(
+ headings.indexOf('1-deploying-the-pulse-server'),
+ );
+ const safety = safetySection();
+ expect(text(safety)).toContain('including with API-only monitoring');
+ expect(text(safety)).toContain("stopping its DaemonSet does not stop the server's Proxmox polling");
+ expect(safety.querySelectorAll('ol > li')).toHaveLength(4);
+ // No copied scale/delete/Helm command can discard the required identity,
+ // reconciliation and prior-state checks when used on a different cluster.
+ expect(safety.querySelector('pre, code')).toBeNull();
+ });
+
+ it('requires a controller-aware stop, not pod deletion or a paused rollout', () => {
+ const steps = [...safetySection().querySelectorAll('ol > li')].map(text);
+ for (const phrase of [
+ 'every Pulse server',
+ 'servers outside this cluster',
+ 'actual desired replica count',
+ 'autoscaler, GitOps reconciler and automatic updater',
+ 'unknown owner or another polling server',
+ ])
+ expect(steps[0]).toContain(phrase);
+ for (const phrase of [
+ 'before deliberately scaling the server to zero',
+ 'Deleting a pod only invites its controller to replace it',
+ 'pausing a Deployment rollout does not stop existing pods',
+ 'every previous server process has terminated normally',
+ 'no replacement is polling',
+ 'missing termination evidence is not a confirmed stop',
+ 'Do not force deletion, uninstall Helm, delete PVCs or recreate Secrets',
+ 'Pulse monitoring and alerts are unavailable while stopped',
+ ])
+ expect(steps[1]).toContain(phrase);
+ });
+
+ it('keeps unknown operations stopped and requires independent every-filesystem recovery', () => {
+ const section = safetySection();
+ const step = text(section.querySelectorAll('ol > li')[2]);
+ for (const phrase of [
+ 'does not cancel a guest-agent request already issued',
+ 'if their state is unknown, do not start the planned backup',
+ 'After the backup ends',
+ 'independent of Pulse and the QEMU Guest Agent',
+ 'thaw, fresh successful workload writes to every filesystem covered by the backup, and workload liveness',
+ 'a write to only the OS disk is not enough',
+ 'If any check fails or is unavailable, keep the pause',
+ 'without a new backup or probe',
+ ])
+ expect(step).toContain(phrase);
+ expect(text(section)).toContain('not incident recovery or an automatic backup hook');
+ expect(text(section)).toContain('do not restart pods or send guest-agent probes');
+ expect(text(section)).toContain(
+ 'does not establish that a native freeze/thaw defect has been fixed',
+ );
+ });
+
+ it('restores actual prior state without replaying installation or losing data and credentials', () => {
+ const step = text(safetySection().querySelectorAll('ol > li')[3]);
+ for (const phrase of [
+ 'after all checks pass',
+ "each server's original desired count",
+ 'controls that were active beforehand',
+ 'previously zero-replica server stays at zero',
+ 'unknown prior state is not permission to start it',
+ 'Preserve the existing image, PVCs, configuration, identities and credentials',
+ 'monitoring is fresh before relying on alerts again',
+ 'Do not use an install command from this page as a restart or recovery shortcut',
+ ])
+ expect(step).toContain(phrase);
+ });
+
+ it('connects both deployment guides and troubleshooting to real shipped safety headings', () => {
+ const kubernetes = articleFor('KUBERNETES');
+ const vm = articleFor('VM_DISK_MONITORING');
+ expect(kubernetes.querySelector('a[href="/docs/VM_DISK_MONITORING#backup-safety"]')).not.toBeNull();
+ expect(vm.querySelector('#backup-safety')).not.toBeNull();
+ expect(vm.querySelector('a[href="/docs/KUBERNETES#proxmox-backup-safety"]')).not.toBeNull();
+ const troubleshooting = kubernetes.querySelector('[id="4-troubleshooting"]')!;
+ const warning = troubleshooting.nextElementSibling!;
+ expect(warning.querySelector('a[href="#proxmox-backup-safety"]')).not.toBeNull();
+ expect(text(warning)).toContain("A Kubernetes pod's Ready state does not prove guest thaw");
+ for (const name of ['KUBERNETES', 'VM_DISK_MONITORING']) {
+ expect(readFileSync(path.join(frontendRoot, 'public/docs', `${name}.md`), 'utf8')).toBe(
+ readDoc(name),
+ );
+ }
+ });
+});