Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep ICMP permission recovery scoped to the observation host
pulse-triage[bot]f8a33376cec0
diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.mdindex b7de212224..5c8e8f591c 100644--- a/docs/CONFIGURATION.md+++ b/docs/CONFIGURATION.md@@ -936,32 +936,111 @@ an agent assignment remain available in every edition.### ICMP probe privileges-ICMP probes run the system `ping` binary, which needs the `CAP_NET_RAW`-capability. The systemd unit written by the installer hardens the service-with `NoNewPrivileges=true`, which strips ping's setuid bit and file-capabilities, so the unit also grants the capability directly with-`AmbientCapabilities=CAP_NET_RAW`. Units written by older versions of the-installer lack that line, and ICMP probes fail with-`icmp probe failed: ping: socktype: SOCK_RAW ... missing cap_net_raw+p capability`.+ICMP checks execute the system `ping` binary **on the selected observation+host**. A local permission error (`Operation not permitted`, `cap_net_raw` or+`icmp probe blocked`) is not evidence that the target is down, nor proof that+one particular service setting caused it. First distinguish **This Pulse+server** from an external agent observation. Changing the server's unit cannot+repair an agent's permissions on another host.++For Linux systemd servers, the current installer grants both+`AmbientCapabilities=CAP_NET_RAW` and `CapabilityBoundingSet=CAP_NET_RAW`, while+keeping `NoNewPrivileges=true`. This allows the child `ping` to inherit the+capability without gaining it through setuid or file capabilities. Older units+can lack the ambient grant. **Updates preserve an existing service unit**;+rerunning an installer is not a reliable unit repair, even if an older error+message recommends it. Do not reinstall over a working data directory.++#### Inspect the actual systemd server first++On the machine running the Pulse server, read only these effective properties.+For a Pulse LXC, run this **inside the Pulse LXC**, not on the Proxmox host or a+monitored guest. The example uses `pulse.service`; substitute your actual server+unit throughout, such as `pulse-backend.service` on some older installs,+**not pulse-agent.service**:-To fix an existing install, either re-run the install script (it rewrites-the unit) or add the capability as an override:+```bash+timeout --signal=TERM --kill-after=1s 8s systemctl show pulse.service \+ --property=LoadState,ActiveState,User,NoNewPrivileges,AmbientCapabilities,CapabilityBoundingSet+```++Interpret the result only after a successful read with `LoadState=loaded`.+A missing unit, failed or timed-out read is **unknown**, not an empty capability+set or a stopped server. This command does not print environment values,+credentials or full unit contents; keep those private.++| Existing observation | What follows |+| --- | --- |+| `NoNewPrivileges=yes`, ceiling includes `cap_net_raw`, ambient grant missing | An older non-root server unit may need the narrow override below. |+| Both capability properties already include `cap_net_raw` | Do not add the grant again. Check the selected observation host, installed ping and its OS/container restrictions through normal administration. |+| Ceiling excludes `cap_net_raw`, or the deployment deliberately forbids it | An ambient line alone cannot overcome the ceiling. Preserve that restriction; do not replace the capability set with a permissive one. |+| `User=` is empty/root, or the service is custom | This is not the standard non-root server recipe. Review its existing privilege and data-access requirements using [manual-service guidance](INSTALL.md#2-bare-metal--systemd). Do not change the user or grant broader privileges just to clear the error. |++#### Narrow override for an older non-root server++Use this only for an **active** non-root server whose capability ceiling already+permits `cap_net_raw`, whose ambient grant is missing, and where granting it is+consistent with the deployment's intended policy. Keep deliberately inactive+services inactive. Record the existing drop-in locally before editing; it may+contain private configuration. Preserve every unrelated setting:```bash-systemctl edit pulse # pulse-backend on ProxmoxVE community-script installs+sudo systemctl edit pulse.service```+Add just this service setting to the drop-in:+```ini[Service]AmbientCapabilities=CAP_NET_RAW```-Then `systemctl daemon-reload && systemctl restart pulse`.+Keep `NoNewPrivileges` enabled; do not run Pulse as root, set file capabilities+on the system-wide ping binary or remove sandbox settings. Do not reset or widen+`CapabilityBoundingSet` to make this example apply to a different profile.++```bash+sudo systemctl daemon-reload+```++If editing or reloading fails, stop. Repeat the read-only property check above+to confirm the intended ambient grant and preserved ceiling/hardening. Reload+changes systemd's configuration, **not the running process's capabilities**.++Schedule the restart as maintenance, outside backups or other guest operations.+**Pulse monitoring and alerts are unavailable during the restart**; arrange+independent outage coverage. Only after the checks pass, for the server that+was active beforehand:++```bash+sudo systemctl restart pulse.service && \+ timeout --signal=TERM --kill-after=1s 8s systemctl is-active pulse.service+```-Docker installs are unaffected: Docker's default capability set includes-`NET_RAW`. If you run the container with `--cap-drop=ALL`, add-`--cap-add=NET_RAW` to keep ICMP probes working. TCP and HTTP/HTTPS probes-need no special privileges.+`active` proves service startup, not ICMP recovery or notification delivery.+Check the next normally scheduled observation from the **same location**;+do not induce a workload failure or another notification to prove recovery.+A failed/unknown restart needs normal recovery, not repeated restarts. To undo+this change, edit only the added line, preserve other drop-in content and+reload; do not use `systemctl revert`, which can remove unrelated overrides.+Apply any further restart only under the same maintenance and prior-active+conditions. Preserve the data directory and encryption key throughout.++#### Containers and external agents++Docker's default capability set includes `NET_RAW`, but that does not prove ICMP+works in every rootless, non-root or restricted container. For an intentionally+restricted deployment, review only the selected observation host's ping/socket+permissions and saved deployment. Do not use `--privileged`, `--cap-add=ALL`,+disable user namespaces or recreate a container as a diagnostic. A deliberate+capability change must use the deployment's normal maintenance route while+preserving data and configuration; the systemd override above does not apply.++External agents have their own OS/service requirements; see+[Agent Security](AGENT_SECURITY.md). Do not grant command execution or widen+agent privileges merely because the server's ICMP check failed. TCP and+HTTP/HTTPS checks need no raw-socket capability, but measure different behaviour;+switching protocol or observation location is not proof that ICMP recovered.---diff --git a/frontend-modern/public/docs/CONFIGURATION.md b/frontend-modern/public/docs/CONFIGURATION.mdindex b7de212224..5c8e8f591c 100644--- a/frontend-modern/public/docs/CONFIGURATION.md+++ b/frontend-modern/public/docs/CONFIGURATION.md@@ -936,32 +936,111 @@ an agent assignment remain available in every edition.### ICMP probe privileges-ICMP probes run the system `ping` binary, which needs the `CAP_NET_RAW`-capability. The systemd unit written by the installer hardens the service-with `NoNewPrivileges=true`, which strips ping's setuid bit and file-capabilities, so the unit also grants the capability directly with-`AmbientCapabilities=CAP_NET_RAW`. Units written by older versions of the-installer lack that line, and ICMP probes fail with-`icmp probe failed: ping: socktype: SOCK_RAW ... missing cap_net_raw+p capability`.+ICMP checks execute the system `ping` binary **on the selected observation+host**. A local permission error (`Operation not permitted`, `cap_net_raw` or+`icmp probe blocked`) is not evidence that the target is down, nor proof that+one particular service setting caused it. First distinguish **This Pulse+server** from an external agent observation. Changing the server's unit cannot+repair an agent's permissions on another host.++For Linux systemd servers, the current installer grants both+`AmbientCapabilities=CAP_NET_RAW` and `CapabilityBoundingSet=CAP_NET_RAW`, while+keeping `NoNewPrivileges=true`. This allows the child `ping` to inherit the+capability without gaining it through setuid or file capabilities. Older units+can lack the ambient grant. **Updates preserve an existing service unit**;+rerunning an installer is not a reliable unit repair, even if an older error+message recommends it. Do not reinstall over a working data directory.++#### Inspect the actual systemd server first++On the machine running the Pulse server, read only these effective properties.+For a Pulse LXC, run this **inside the Pulse LXC**, not on the Proxmox host or a+monitored guest. The example uses `pulse.service`; substitute your actual server+unit throughout, such as `pulse-backend.service` on some older installs,+**not pulse-agent.service**:-To fix an existing install, either re-run the install script (it rewrites-the unit) or add the capability as an override:+```bash+timeout --signal=TERM --kill-after=1s 8s systemctl show pulse.service \+ --property=LoadState,ActiveState,User,NoNewPrivileges,AmbientCapabilities,CapabilityBoundingSet+```++Interpret the result only after a successful read with `LoadState=loaded`.+A missing unit, failed or timed-out read is **unknown**, not an empty capability+set or a stopped server. This command does not print environment values,+credentials or full unit contents; keep those private.++| Existing observation | What follows |+| --- | --- |+| `NoNewPrivileges=yes`, ceiling includes `cap_net_raw`, ambient grant missing | An older non-root server unit may need the narrow override below. |+| Both capability properties already include `cap_net_raw` | Do not add the grant again. Check the selected observation host, installed ping and its OS/container restrictions through normal administration. |+| Ceiling excludes `cap_net_raw`, or the deployment deliberately forbids it | An ambient line alone cannot overcome the ceiling. Preserve that restriction; do not replace the capability set with a permissive one. |+| `User=` is empty/root, or the service is custom | This is not the standard non-root server recipe. Review its existing privilege and data-access requirements using [manual-service guidance](INSTALL.md#2-bare-metal--systemd). Do not change the user or grant broader privileges just to clear the error. |++#### Narrow override for an older non-root server++Use this only for an **active** non-root server whose capability ceiling already+permits `cap_net_raw`, whose ambient grant is missing, and where granting it is+consistent with the deployment's intended policy. Keep deliberately inactive+services inactive. Record the existing drop-in locally before editing; it may+contain private configuration. Preserve every unrelated setting:```bash-systemctl edit pulse # pulse-backend on ProxmoxVE community-script installs+sudo systemctl edit pulse.service```+Add just this service setting to the drop-in:+```ini[Service]AmbientCapabilities=CAP_NET_RAW```-Then `systemctl daemon-reload && systemctl restart pulse`.+Keep `NoNewPrivileges` enabled; do not run Pulse as root, set file capabilities+on the system-wide ping binary or remove sandbox settings. Do not reset or widen+`CapabilityBoundingSet` to make this example apply to a different profile.++```bash+sudo systemctl daemon-reload+```++If editing or reloading fails, stop. Repeat the read-only property check above+to confirm the intended ambient grant and preserved ceiling/hardening. Reload+changes systemd's configuration, **not the running process's capabilities**.++Schedule the restart as maintenance, outside backups or other guest operations.+**Pulse monitoring and alerts are unavailable during the restart**; arrange+independent outage coverage. Only after the checks pass, for the server that+was active beforehand:++```bash+sudo systemctl restart pulse.service && \+ timeout --signal=TERM --kill-after=1s 8s systemctl is-active pulse.service+```-Docker installs are unaffected: Docker's default capability set includes-`NET_RAW`. If you run the container with `--cap-drop=ALL`, add-`--cap-add=NET_RAW` to keep ICMP probes working. TCP and HTTP/HTTPS probes-need no special privileges.+`active` proves service startup, not ICMP recovery or notification delivery.+Check the next normally scheduled observation from the **same location**;+do not induce a workload failure or another notification to prove recovery.+A failed/unknown restart needs normal recovery, not repeated restarts. To undo+this change, edit only the added line, preserve other drop-in content and+reload; do not use `systemctl revert`, which can remove unrelated overrides.+Apply any further restart only under the same maintenance and prior-active+conditions. Preserve the data directory and encryption key throughout.++#### Containers and external agents++Docker's default capability set includes `NET_RAW`, but that does not prove ICMP+works in every rootless, non-root or restricted container. For an intentionally+restricted deployment, review only the selected observation host's ping/socket+permissions and saved deployment. Do not use `--privileged`, `--cap-add=ALL`,+disable user namespaces or recreate a container as a diagnostic. A deliberate+capability change must use the deployment's normal maintenance route while+preserving data and configuration; the systemd override above does not apply.++External agents have their own OS/service requirements; see+[Agent Security](AGENT_SECURITY.md). Do not grant command execution or widen+agent privileges merely because the server's ICMP check failed. TCP and+HTTP/HTTPS checks need no raw-socket capability, but measure different behaviour;+switching protocol or observation location is not proof that ICMP recovered.---diff --git a/scripts/tests/test_icmp_privilege_docs.py b/scripts/tests/test_icmp_privilege_docs.pynew file mode 100644index 0000000000..f575f821f1--- /dev/null+++ b/scripts/tests/test_icmp_privilege_docs.py@@ -0,0 +1,170 @@+#!/usr/bin/env python3+"""Exercise copied ICMP-help commands without systemd or privilege changes.++Synthetic command recorders check disclosure, deadlines and failure sequencing.+They do not establish native ICMP recovery, capability grants or notification+delivery. The guide deliberately requires an operator's policy/state judgment.+"""++import json+import os+from pathlib import Path+import re+import subprocess+import tempfile+import unittest+++ROOT = Path(__file__).resolve().parents[2]+HEADING = "### ICMP probe privileges\n"+PROPERTIES = "LoadState,ActiveState,User,NoNewPrivileges,AmbientCapabilities,CapabilityBoundingSet"+PRIVATE = "synthetic-private-environment-value"+++def section():+ text = (ROOT / "docs/CONFIGURATION.md").read_text(encoding="utf-8")+ return text.split(HEADING, 1)[1].split("\n---\n", 1)[0]+++def recipe(needle):+ blocks = [b for b in re.findall(r"```bash\n(.*?)```", section(), re.S)+ if needle in b]+ if len(blocks) != 1:+ raise AssertionError(f"expected one ICMP-help recipe for {needle!r}")+ return blocks[0]+++SYSTEMCTL = '''#!/usr/bin/env python3+import json, os, sys, time+from pathlib import Path+args = sys.argv[1:]+root = Path(os.environ['RECORD_DIR'])+with (root / 'calls.jsonl').open('a') as f:+ f.write(json.dumps(args) + '\\n')+mode = os.environ.get('READ_MODE', 'loaded')+if args[0] == 'show':+ if mode == 'denied':+ print('Read denied', file=sys.stderr)+ sys.exit(1)+ if mode == 'hang':+ time.sleep(30)+ requested = args[-1].removeprefix('--property=').split(',')+ values = {+ 'LoadState': 'not-found' if mode == 'missing' else 'loaded',+ 'ActiveState': 'active', 'User': 'pulse', 'NoNewPrivileges': 'yes',+ 'AmbientCapabilities': '', 'CapabilityBoundingSet': 'cap_net_raw',+ }+ if args[-1] == '--all' or not args[-1].startswith('--property='):+ print('Environment=' + os.environ['PRIVATE_VALUE'])+ for key in requested:+ print(key + '=' + values.get(key, os.environ['PRIVATE_VALUE']))+elif args[0] == 'restart':+ sys.exit(int(os.environ.get('RESTART_EXIT', '0')))+elif args[0] == 'is-active':+ print('active')+else:+ sys.exit(65)+'''+++def execute(command, **overrides):+ with tempfile.TemporaryDirectory(prefix="icmp docs ") as directory:+ root = Path(directory)+ tools = root / "tools"+ tools.mkdir()+ for name, body in (("systemctl", SYSTEMCTL), ("sudo", '#!/bin/sh\nexec "$@"\n')):+ path = tools / name+ path.write_text(body, encoding="utf-8")+ path.chmod(0o700)+ env = {**os.environ, "RECORD_DIR": directory, "PRIVATE_VALUE": PRIVATE,+ "PATH": str(tools) + os.pathsep + os.environ["PATH"], **overrides}+ result = subprocess.run(["bash", "-c", command], env=env, text=True,+ capture_output=True, timeout=12)+ record = root / "calls.jsonl"+ calls = [json.loads(line) for line in record.read_text().splitlines()] if record.exists() else []+ return result, calls+++class ICMPPrivilegeDocsTest(unittest.TestCase):+ def test_install_and_privilege_claims_match_existing_source(self):+ text = " ".join(section().split())+ installer = (ROOT / "install.sh").read_text()+ for setting in ("AmbientCapabilities=CAP_NET_RAW", "CapabilityBoundingSet=CAP_NET_RAW",+ "NoNewPrivileges=true"):+ self.assertIn(setting, text)+ self.assertIn(setting, installer)+ self.assertIn("Updates keep an existing unit file untouched", installer)+ for boundary in ("Updates preserve an existing service unit", "not a reliable unit repair",+ "An ambient line alone cannot overcome the ceiling",+ "Preserve that restriction", "Do not reset or widen",+ "set file capabilities on the system-wide ping binary",+ "do not run Pulse as root"):+ self.assertIn(boundary, text)+ self.assertNotIn("Docker installs are unaffected", text)+ self.assertNotIn("it rewrites the unit", text)++ def test_location_state_and_recovery_are_not_inferred(self):+ text = " ".join(section().split())+ for boundary in ("selected observation host", "not evidence that the target is down",+ "cannot repair an agent's permissions", "inside the Pulse LXC",+ "not pulse-agent.service", "failed or timed-out read is **unknown**",+ "Keep deliberately inactive services inactive", "not the running process",+ "monitoring and alerts are unavailable", "same location",+ "not ICMP recovery or notification delivery", "not repeated restarts",+ "do not use `systemctl revert`", "preserve other drop-in content",+ "preserving data and configuration", "Do not use `--privileged`",+ "--cap-add=ALL", "not proof that ICMP recovered"):+ self.assertIn(boundary, text)+ # The installer-independent probe executes on either server or agent.+ probe = (ROOT / "internal/availabilityprobe/probe.go").read_text()+ self.assertIn('exec.CommandContext(ctx, "ping", args...)', probe)++ def test_copied_read_is_bounded_allowlisted_and_read_only(self):+ command = recipe("systemctl show")+ self.assertIn("timeout --signal=TERM --kill-after=1s 8s", command)+ self.assertIn("--property=" + PROPERTIES, command)+ self.assertNotIn("sudo", command)+ for mode in ("loaded", "missing"):+ with self.subTest(mode=mode):+ result, calls = execute(command, READ_MODE=mode)+ self.assertEqual(result.returncode, 0, result.stderr)+ self.assertEqual(calls, [["show", "pulse.service", "--property=" + PROPERTIES]])+ self.assertNotIn(PRIVATE, result.stdout + result.stderr)+ self.assertEqual(set(line.split("=", 1)[0] for line in result.stdout.splitlines()),+ set(PROPERTIES.split(",")))+ self.assertIn("LoadState=" + ("not-found" if mode == "missing" else "loaded"),+ result.stdout)++ def test_failed_or_hung_read_has_no_mutation_or_retry(self):+ for mode, code in (("denied", 1), ("hang", 124)):+ with self.subTest(mode=mode):+ result, calls = execute(recipe("systemctl show"), READ_MODE=mode)+ self.assertEqual(result.returncode, code, result.stderr)+ self.assertEqual(calls, [["show", "pulse.service", "--property=" + PROPERTIES]])+ self.assertNotIn(PRIVATE, result.stdout + result.stderr)++ def test_copied_restart_failure_stops_before_readback(self):+ command = recipe("systemctl restart")+ for code in (0, 5):+ with self.subTest(restart_exit=code):+ result, calls = execute(command, RESTART_EXIT=str(code))+ self.assertEqual(result.returncode, code, result.stderr)+ expected = [["restart", "pulse.service"]]+ if code == 0:+ expected.append(["is-active", "pulse.service"])+ self.assertEqual(calls, expected)+ self.assertNotIn(PRIVATE, result.stdout + result.stderr)+ self.assertEqual(result.stdout, "active\n" if code == 0 else "")++ def test_copied_commands_parse_and_mirror_matches(self):+ blocks = re.findall(r"```bash\n(.*?)```", section(), re.S)+ self.assertEqual(len(blocks), 4)+ for command in blocks:+ subprocess.run(["bash", "-n", "-c", command], check=True, capture_output=True)+ self.assertIn("INSTALL.md#2-bare-metal--systemd", section())+ self.assertEqual((ROOT / "docs/CONFIGURATION.md").read_bytes(),+ (ROOT / "frontend-modern/public/docs/CONFIGURATION.md").read_bytes())+++if __name__ == "__main__":+ unittest.main()