Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Keep ICMP permission recovery scoped to the observation host

pulse-triage[bot]f8a33376cec0
diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md
index b7de212224..5c8e8f591c 100644
--- a/docs/CONFIGURATION.md
+++ b/docs/CONFIGURATION.md
@@ -936,32 +936,111 @@ an agent assignment remain available in every edition.
### ICMP probe privileges
-ICMP probes run the system `ping` binary, which needs the `CAP_NET_RAW`
-capability. The systemd unit written by the installer hardens the service
-with `NoNewPrivileges=true`, which strips ping's setuid bit and file
-capabilities, so the unit also grants the capability directly with
-`AmbientCapabilities=CAP_NET_RAW`. Units written by older versions of the
-installer lack that line, and ICMP probes fail with
-`icmp probe failed: ping: socktype: SOCK_RAW ... missing cap_net_raw+p capability`.
+ICMP checks execute the system `ping` binary **on the selected observation
+host**. A local permission error (`Operation not permitted`, `cap_net_raw` or
+`icmp probe blocked`) is not evidence that the target is down, nor proof that
+one particular service setting caused it. First distinguish **This Pulse
+server** from an external agent observation. Changing the server's unit cannot
+repair an agent's permissions on another host.
+
+For Linux systemd servers, the current installer grants both
+`AmbientCapabilities=CAP_NET_RAW` and `CapabilityBoundingSet=CAP_NET_RAW`, while
+keeping `NoNewPrivileges=true`. This allows the child `ping` to inherit the
+capability without gaining it through setuid or file capabilities. Older units
+can lack the ambient grant. **Updates preserve an existing service unit**;
+rerunning an installer is not a reliable unit repair, even if an older error
+message recommends it. Do not reinstall over a working data directory.
+
+#### Inspect the actual systemd server first
+
+On the machine running the Pulse server, read only these effective properties.
+For a Pulse LXC, run this **inside the Pulse LXC**, not on the Proxmox host or a
+monitored guest. The example uses `pulse.service`; substitute your actual server
+unit throughout, such as `pulse-backend.service` on some older installs,
+**not pulse-agent.service**:
-To fix an existing install, either re-run the install script (it rewrites
-the unit) or add the capability as an override:
+```bash
+timeout --signal=TERM --kill-after=1s 8s systemctl show pulse.service \
+ --property=LoadState,ActiveState,User,NoNewPrivileges,AmbientCapabilities,CapabilityBoundingSet
+```
+
+Interpret the result only after a successful read with `LoadState=loaded`.
+A missing unit, failed or timed-out read is **unknown**, not an empty capability
+set or a stopped server. This command does not print environment values,
+credentials or full unit contents; keep those private.
+
+| Existing observation | What follows |
+| --- | --- |
+| `NoNewPrivileges=yes`, ceiling includes `cap_net_raw`, ambient grant missing | An older non-root server unit may need the narrow override below. |
+| Both capability properties already include `cap_net_raw` | Do not add the grant again. Check the selected observation host, installed ping and its OS/container restrictions through normal administration. |
+| Ceiling excludes `cap_net_raw`, or the deployment deliberately forbids it | An ambient line alone cannot overcome the ceiling. Preserve that restriction; do not replace the capability set with a permissive one. |
+| `User=` is empty/root, or the service is custom | This is not the standard non-root server recipe. Review its existing privilege and data-access requirements using [manual-service guidance](INSTALL.md#2-bare-metal--systemd). Do not change the user or grant broader privileges just to clear the error. |
+
+#### Narrow override for an older non-root server
+
+Use this only for an **active** non-root server whose capability ceiling already
+permits `cap_net_raw`, whose ambient grant is missing, and where granting it is
+consistent with the deployment's intended policy. Keep deliberately inactive
+services inactive. Record the existing drop-in locally before editing; it may
+contain private configuration. Preserve every unrelated setting:
```bash
-systemctl edit pulse # pulse-backend on ProxmoxVE community-script installs
+sudo systemctl edit pulse.service
```
+Add just this service setting to the drop-in:
+
```ini
[Service]
AmbientCapabilities=CAP_NET_RAW
```
-Then `systemctl daemon-reload && systemctl restart pulse`.
+Keep `NoNewPrivileges` enabled; do not run Pulse as root, set file capabilities
+on the system-wide ping binary or remove sandbox settings. Do not reset or widen
+`CapabilityBoundingSet` to make this example apply to a different profile.
+
+```bash
+sudo systemctl daemon-reload
+```
+
+If editing or reloading fails, stop. Repeat the read-only property check above
+to confirm the intended ambient grant and preserved ceiling/hardening. Reload
+changes systemd's configuration, **not the running process's capabilities**.
+
+Schedule the restart as maintenance, outside backups or other guest operations.
+**Pulse monitoring and alerts are unavailable during the restart**; arrange
+independent outage coverage. Only after the checks pass, for the server that
+was active beforehand:
+
+```bash
+sudo systemctl restart pulse.service && \
+ timeout --signal=TERM --kill-after=1s 8s systemctl is-active pulse.service
+```
-Docker installs are unaffected: Docker's default capability set includes
-`NET_RAW`. If you run the container with `--cap-drop=ALL`, add
-`--cap-add=NET_RAW` to keep ICMP probes working. TCP and HTTP/HTTPS probes
-need no special privileges.
+`active` proves service startup, not ICMP recovery or notification delivery.
+Check the next normally scheduled observation from the **same location**;
+do not induce a workload failure or another notification to prove recovery.
+A failed/unknown restart needs normal recovery, not repeated restarts. To undo
+this change, edit only the added line, preserve other drop-in content and
+reload; do not use `systemctl revert`, which can remove unrelated overrides.
+Apply any further restart only under the same maintenance and prior-active
+conditions. Preserve the data directory and encryption key throughout.
+
+#### Containers and external agents
+
+Docker's default capability set includes `NET_RAW`, but that does not prove ICMP
+works in every rootless, non-root or restricted container. For an intentionally
+restricted deployment, review only the selected observation host's ping/socket
+permissions and saved deployment. Do not use `--privileged`, `--cap-add=ALL`,
+disable user namespaces or recreate a container as a diagnostic. A deliberate
+capability change must use the deployment's normal maintenance route while
+preserving data and configuration; the systemd override above does not apply.
+
+External agents have their own OS/service requirements; see
+[Agent Security](AGENT_SECURITY.md). Do not grant command execution or widen
+agent privileges merely because the server's ICMP check failed. TCP and
+HTTP/HTTPS checks need no raw-socket capability, but measure different behaviour;
+switching protocol or observation location is not proof that ICMP recovered.
---
diff --git a/frontend-modern/public/docs/CONFIGURATION.md b/frontend-modern/public/docs/CONFIGURATION.md
index b7de212224..5c8e8f591c 100644
--- a/frontend-modern/public/docs/CONFIGURATION.md
+++ b/frontend-modern/public/docs/CONFIGURATION.md
@@ -936,32 +936,111 @@ an agent assignment remain available in every edition.
### ICMP probe privileges
-ICMP probes run the system `ping` binary, which needs the `CAP_NET_RAW`
-capability. The systemd unit written by the installer hardens the service
-with `NoNewPrivileges=true`, which strips ping's setuid bit and file
-capabilities, so the unit also grants the capability directly with
-`AmbientCapabilities=CAP_NET_RAW`. Units written by older versions of the
-installer lack that line, and ICMP probes fail with
-`icmp probe failed: ping: socktype: SOCK_RAW ... missing cap_net_raw+p capability`.
+ICMP checks execute the system `ping` binary **on the selected observation
+host**. A local permission error (`Operation not permitted`, `cap_net_raw` or
+`icmp probe blocked`) is not evidence that the target is down, nor proof that
+one particular service setting caused it. First distinguish **This Pulse
+server** from an external agent observation. Changing the server's unit cannot
+repair an agent's permissions on another host.
+
+For Linux systemd servers, the current installer grants both
+`AmbientCapabilities=CAP_NET_RAW` and `CapabilityBoundingSet=CAP_NET_RAW`, while
+keeping `NoNewPrivileges=true`. This allows the child `ping` to inherit the
+capability without gaining it through setuid or file capabilities. Older units
+can lack the ambient grant. **Updates preserve an existing service unit**;
+rerunning an installer is not a reliable unit repair, even if an older error
+message recommends it. Do not reinstall over a working data directory.
+
+#### Inspect the actual systemd server first
+
+On the machine running the Pulse server, read only these effective properties.
+For a Pulse LXC, run this **inside the Pulse LXC**, not on the Proxmox host or a
+monitored guest. The example uses `pulse.service`; substitute your actual server
+unit throughout, such as `pulse-backend.service` on some older installs,
+**not pulse-agent.service**:
-To fix an existing install, either re-run the install script (it rewrites
-the unit) or add the capability as an override:
+```bash
+timeout --signal=TERM --kill-after=1s 8s systemctl show pulse.service \
+ --property=LoadState,ActiveState,User,NoNewPrivileges,AmbientCapabilities,CapabilityBoundingSet
+```
+
+Interpret the result only after a successful read with `LoadState=loaded`.
+A missing unit, failed or timed-out read is **unknown**, not an empty capability
+set or a stopped server. This command does not print environment values,
+credentials or full unit contents; keep those private.
+
+| Existing observation | What follows |
+| --- | --- |
+| `NoNewPrivileges=yes`, ceiling includes `cap_net_raw`, ambient grant missing | An older non-root server unit may need the narrow override below. |
+| Both capability properties already include `cap_net_raw` | Do not add the grant again. Check the selected observation host, installed ping and its OS/container restrictions through normal administration. |
+| Ceiling excludes `cap_net_raw`, or the deployment deliberately forbids it | An ambient line alone cannot overcome the ceiling. Preserve that restriction; do not replace the capability set with a permissive one. |
+| `User=` is empty/root, or the service is custom | This is not the standard non-root server recipe. Review its existing privilege and data-access requirements using [manual-service guidance](INSTALL.md#2-bare-metal--systemd). Do not change the user or grant broader privileges just to clear the error. |
+
+#### Narrow override for an older non-root server
+
+Use this only for an **active** non-root server whose capability ceiling already
+permits `cap_net_raw`, whose ambient grant is missing, and where granting it is
+consistent with the deployment's intended policy. Keep deliberately inactive
+services inactive. Record the existing drop-in locally before editing; it may
+contain private configuration. Preserve every unrelated setting:
```bash
-systemctl edit pulse # pulse-backend on ProxmoxVE community-script installs
+sudo systemctl edit pulse.service
```
+Add just this service setting to the drop-in:
+
```ini
[Service]
AmbientCapabilities=CAP_NET_RAW
```
-Then `systemctl daemon-reload && systemctl restart pulse`.
+Keep `NoNewPrivileges` enabled; do not run Pulse as root, set file capabilities
+on the system-wide ping binary or remove sandbox settings. Do not reset or widen
+`CapabilityBoundingSet` to make this example apply to a different profile.
+
+```bash
+sudo systemctl daemon-reload
+```
+
+If editing or reloading fails, stop. Repeat the read-only property check above
+to confirm the intended ambient grant and preserved ceiling/hardening. Reload
+changes systemd's configuration, **not the running process's capabilities**.
+
+Schedule the restart as maintenance, outside backups or other guest operations.
+**Pulse monitoring and alerts are unavailable during the restart**; arrange
+independent outage coverage. Only after the checks pass, for the server that
+was active beforehand:
+
+```bash
+sudo systemctl restart pulse.service && \
+ timeout --signal=TERM --kill-after=1s 8s systemctl is-active pulse.service
+```
-Docker installs are unaffected: Docker's default capability set includes
-`NET_RAW`. If you run the container with `--cap-drop=ALL`, add
-`--cap-add=NET_RAW` to keep ICMP probes working. TCP and HTTP/HTTPS probes
-need no special privileges.
+`active` proves service startup, not ICMP recovery or notification delivery.
+Check the next normally scheduled observation from the **same location**;
+do not induce a workload failure or another notification to prove recovery.
+A failed/unknown restart needs normal recovery, not repeated restarts. To undo
+this change, edit only the added line, preserve other drop-in content and
+reload; do not use `systemctl revert`, which can remove unrelated overrides.
+Apply any further restart only under the same maintenance and prior-active
+conditions. Preserve the data directory and encryption key throughout.
+
+#### Containers and external agents
+
+Docker's default capability set includes `NET_RAW`, but that does not prove ICMP
+works in every rootless, non-root or restricted container. For an intentionally
+restricted deployment, review only the selected observation host's ping/socket
+permissions and saved deployment. Do not use `--privileged`, `--cap-add=ALL`,
+disable user namespaces or recreate a container as a diagnostic. A deliberate
+capability change must use the deployment's normal maintenance route while
+preserving data and configuration; the systemd override above does not apply.
+
+External agents have their own OS/service requirements; see
+[Agent Security](AGENT_SECURITY.md). Do not grant command execution or widen
+agent privileges merely because the server's ICMP check failed. TCP and
+HTTP/HTTPS checks need no raw-socket capability, but measure different behaviour;
+switching protocol or observation location is not proof that ICMP recovered.
---
diff --git a/scripts/tests/test_icmp_privilege_docs.py b/scripts/tests/test_icmp_privilege_docs.py
new file mode 100644
index 0000000000..f575f821f1
--- /dev/null
+++ b/scripts/tests/test_icmp_privilege_docs.py
@@ -0,0 +1,170 @@
+#!/usr/bin/env python3
+"""Exercise copied ICMP-help commands without systemd or privilege changes.
+
+Synthetic command recorders check disclosure, deadlines and failure sequencing.
+They do not establish native ICMP recovery, capability grants or notification
+delivery. The guide deliberately requires an operator's policy/state judgment.
+"""
+
+import json
+import os
+from pathlib import Path
+import re
+import subprocess
+import tempfile
+import unittest
+
+
+ROOT = Path(__file__).resolve().parents[2]
+HEADING = "### ICMP probe privileges\n"
+PROPERTIES = "LoadState,ActiveState,User,NoNewPrivileges,AmbientCapabilities,CapabilityBoundingSet"
+PRIVATE = "synthetic-private-environment-value"
+
+
+def section():
+ text = (ROOT / "docs/CONFIGURATION.md").read_text(encoding="utf-8")
+ return text.split(HEADING, 1)[1].split("\n---\n", 1)[0]
+
+
+def recipe(needle):
+ blocks = [b for b in re.findall(r"```bash\n(.*?)```", section(), re.S)
+ if needle in b]
+ if len(blocks) != 1:
+ raise AssertionError(f"expected one ICMP-help recipe for {needle!r}")
+ return blocks[0]
+
+
+SYSTEMCTL = '''#!/usr/bin/env python3
+import json, os, sys, time
+from pathlib import Path
+args = sys.argv[1:]
+root = Path(os.environ['RECORD_DIR'])
+with (root / 'calls.jsonl').open('a') as f:
+ f.write(json.dumps(args) + '\\n')
+mode = os.environ.get('READ_MODE', 'loaded')
+if args[0] == 'show':
+ if mode == 'denied':
+ print('Read denied', file=sys.stderr)
+ sys.exit(1)
+ if mode == 'hang':
+ time.sleep(30)
+ requested = args[-1].removeprefix('--property=').split(',')
+ values = {
+ 'LoadState': 'not-found' if mode == 'missing' else 'loaded',
+ 'ActiveState': 'active', 'User': 'pulse', 'NoNewPrivileges': 'yes',
+ 'AmbientCapabilities': '', 'CapabilityBoundingSet': 'cap_net_raw',
+ }
+ if args[-1] == '--all' or not args[-1].startswith('--property='):
+ print('Environment=' + os.environ['PRIVATE_VALUE'])
+ for key in requested:
+ print(key + '=' + values.get(key, os.environ['PRIVATE_VALUE']))
+elif args[0] == 'restart':
+ sys.exit(int(os.environ.get('RESTART_EXIT', '0')))
+elif args[0] == 'is-active':
+ print('active')
+else:
+ sys.exit(65)
+'''
+
+
+def execute(command, **overrides):
+ with tempfile.TemporaryDirectory(prefix="icmp docs ") as directory:
+ root = Path(directory)
+ tools = root / "tools"
+ tools.mkdir()
+ for name, body in (("systemctl", SYSTEMCTL), ("sudo", '#!/bin/sh\nexec "$@"\n')):
+ path = tools / name
+ path.write_text(body, encoding="utf-8")
+ path.chmod(0o700)
+ env = {**os.environ, "RECORD_DIR": directory, "PRIVATE_VALUE": PRIVATE,
+ "PATH": str(tools) + os.pathsep + os.environ["PATH"], **overrides}
+ result = subprocess.run(["bash", "-c", command], env=env, text=True,
+ capture_output=True, timeout=12)
+ record = root / "calls.jsonl"
+ calls = [json.loads(line) for line in record.read_text().splitlines()] if record.exists() else []
+ return result, calls
+
+
+class ICMPPrivilegeDocsTest(unittest.TestCase):
+ def test_install_and_privilege_claims_match_existing_source(self):
+ text = " ".join(section().split())
+ installer = (ROOT / "install.sh").read_text()
+ for setting in ("AmbientCapabilities=CAP_NET_RAW", "CapabilityBoundingSet=CAP_NET_RAW",
+ "NoNewPrivileges=true"):
+ self.assertIn(setting, text)
+ self.assertIn(setting, installer)
+ self.assertIn("Updates keep an existing unit file untouched", installer)
+ for boundary in ("Updates preserve an existing service unit", "not a reliable unit repair",
+ "An ambient line alone cannot overcome the ceiling",
+ "Preserve that restriction", "Do not reset or widen",
+ "set file capabilities on the system-wide ping binary",
+ "do not run Pulse as root"):
+ self.assertIn(boundary, text)
+ self.assertNotIn("Docker installs are unaffected", text)
+ self.assertNotIn("it rewrites the unit", text)
+
+ def test_location_state_and_recovery_are_not_inferred(self):
+ text = " ".join(section().split())
+ for boundary in ("selected observation host", "not evidence that the target is down",
+ "cannot repair an agent's permissions", "inside the Pulse LXC",
+ "not pulse-agent.service", "failed or timed-out read is **unknown**",
+ "Keep deliberately inactive services inactive", "not the running process",
+ "monitoring and alerts are unavailable", "same location",
+ "not ICMP recovery or notification delivery", "not repeated restarts",
+ "do not use `systemctl revert`", "preserve other drop-in content",
+ "preserving data and configuration", "Do not use `--privileged`",
+ "--cap-add=ALL", "not proof that ICMP recovered"):
+ self.assertIn(boundary, text)
+ # The installer-independent probe executes on either server or agent.
+ probe = (ROOT / "internal/availabilityprobe/probe.go").read_text()
+ self.assertIn('exec.CommandContext(ctx, "ping", args...)', probe)
+
+ def test_copied_read_is_bounded_allowlisted_and_read_only(self):
+ command = recipe("systemctl show")
+ self.assertIn("timeout --signal=TERM --kill-after=1s 8s", command)
+ self.assertIn("--property=" + PROPERTIES, command)
+ self.assertNotIn("sudo", command)
+ for mode in ("loaded", "missing"):
+ with self.subTest(mode=mode):
+ result, calls = execute(command, READ_MODE=mode)
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertEqual(calls, [["show", "pulse.service", "--property=" + PROPERTIES]])
+ self.assertNotIn(PRIVATE, result.stdout + result.stderr)
+ self.assertEqual(set(line.split("=", 1)[0] for line in result.stdout.splitlines()),
+ set(PROPERTIES.split(",")))
+ self.assertIn("LoadState=" + ("not-found" if mode == "missing" else "loaded"),
+ result.stdout)
+
+ def test_failed_or_hung_read_has_no_mutation_or_retry(self):
+ for mode, code in (("denied", 1), ("hang", 124)):
+ with self.subTest(mode=mode):
+ result, calls = execute(recipe("systemctl show"), READ_MODE=mode)
+ self.assertEqual(result.returncode, code, result.stderr)
+ self.assertEqual(calls, [["show", "pulse.service", "--property=" + PROPERTIES]])
+ self.assertNotIn(PRIVATE, result.stdout + result.stderr)
+
+ def test_copied_restart_failure_stops_before_readback(self):
+ command = recipe("systemctl restart")
+ for code in (0, 5):
+ with self.subTest(restart_exit=code):
+ result, calls = execute(command, RESTART_EXIT=str(code))
+ self.assertEqual(result.returncode, code, result.stderr)
+ expected = [["restart", "pulse.service"]]
+ if code == 0:
+ expected.append(["is-active", "pulse.service"])
+ self.assertEqual(calls, expected)
+ self.assertNotIn(PRIVATE, result.stdout + result.stderr)
+ self.assertEqual(result.stdout, "active\n" if code == 0 else "")
+
+ def test_copied_commands_parse_and_mirror_matches(self):
+ blocks = re.findall(r"```bash\n(.*?)```", section(), re.S)
+ self.assertEqual(len(blocks), 4)
+ for command in blocks:
+ subprocess.run(["bash", "-n", "-c", command], check=True, capture_output=True)
+ self.assertIn("INSTALL.md#2-bare-metal--systemd", section())
+ self.assertEqual((ROOT / "docs/CONFIGURATION.md").read_bytes(),
+ (ROOT / "frontend-modern/public/docs/CONFIGURATION.md").read_bytes())
+
+
+if __name__ == "__main__":
+ unittest.main()