Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Bound passive agent log collection across help entry points
pulse-triage[bot]4733e816e2db
diff --git a/docs/UNIFIED_AGENT.md b/docs/UNIFIED_AGENT.mdindex 7d053904a7..ac61a2ea77 100644--- a/docs/UNIFIED_AGENT.md+++ b/docs/UNIFIED_AGENT.md@@ -582,11 +582,15 @@ Apply and verify the change:```bashsudo systemctl daemon-reloadsudo systemctl restart pulse-agent-sudo journalctl -u pulse-agent --since "5 minutes ago"```-Set the value to `debug` temporarily when collecting diagnostics, then restore-`info` or `warn`. For a container agent, set `LOG_LEVEL=warn` in the container+These are configuration changes, not evidence-collection steps: apply them in+a suitable maintenance window, not just to file a report. Inspect the existing+journal with the [bounded agent log reader](#collect-agent-logs-safely).+Start with the current log level; changing it to `debug` can expose more private+details and a restart changes the run being investigated. Do not enable debug+logging or restart the agent merely to manufacture a missing diagnostic entry.+For a container agent, set `LOG_LEVEL=warn` in the containerenvironment and recreate the container. Pro installations can also manage`log_level` with an [agent configuration profile](CENTRALIZED_MANAGEMENT.md).@@ -921,6 +925,54 @@ Set `--health-addr=""` or `PULSE_HEALTH_ADDR=off` to disable the health/metrics## Troubleshooting+### Collect agent logs safely++Prefer existing observations from the original failure. On a **Linux systemd+agent host**, use this bounded Bash reader with an account already authorised+to read that journal. Run it on the affected agent host, not the Pulse server+or a different Proxmox node; use the actual service name and original time+window if they differ from the example. This reads logs only: it does not+contact Pulse, a container runtime or a guest agent.++```bash+(+command -v timeout >/dev/null 2>&1 || {+ printf 'Agent log read unavailable: GNU timeout is required; no unbounded fallback.\n' >&2+ exit 1+}+if agent_logs=$(timeout --signal=TERM --kill-after=1s 8s journalctl -u pulse-agent.service --since '15 minutes ago' --lines 200 --no-pager --output=cat 2>&1); then+ if [ -n "$agent_logs" ]; then+ printf '%s\n' "$agent_logs"+ fi+else+ agent_log_status=$?+ printf 'Agent log read unavailable (exit %s); no partial excerpt shown.\n' "$agent_log_status" >&2+ exit "$agent_log_status"+fi+)+```++This requires **GNU `timeout`**, reads at most 200 records from the default+15-minute window, and allows eight seconds plus a one-second termination grace.+A failed or timed-out read is unavailable, not an empty search; partial output+is withheld. Stop if the utility, service journal or existing access is+unavailable; do not follow the journal indefinitely, remove the deadline or+broaden permissions to collect it. A successful empty read is inconclusive:+the event may be outside the window or absent at the configured log level.+Do not restart, re-enrol or enable debug logging just to obtain an entry.++The excerpt is **not sanitised**. Review it locally before searching for the+relevant error or sharing a manually redacted timestamp and failure stage.+Tokens, cookies, URLs, private names or addresses can appear in errors; do not+post the full excerpt, service environment, connection files or saved identity.+A log message does not prove fresh readings, successful updates or delivery.++For a container agent or a non-systemd host, use the existing deployment's log+source instead; this journal recipe does not apply. Keep its read bounded and+private, and report evidence as unavailable if safe collection is not possible.+QNAP and Unraid log locations are described in the+[space troubleshooting section](#installer-fails-with-not-enough-free-disk-space).+### pfSense service disabled after a major upgradePulse installs two service files on pfSense: the FreeBSD rc.d service at@@ -1002,7 +1054,8 @@ space problem is resolved; verify fresh reporting, free space and log growthafter any planned repair. Keep log contents, connection files and tokens private.### Agent Not Updating-- Check logs: `journalctl -u pulse-agent -f`+- Inspect existing errors with the [bounded agent log reader](#collect-agent-logs-safely);+ a missing update entry is not proof that an update succeeded.- Verify network connectivity to Pulse server- Ensure auto-update is not disabled- Confirm the agent can authenticate and that its saved connection state still@@ -1147,11 +1200,8 @@ monitoring report or **Automatic updates ready** also does not prove that thisseparate WebSocket channel is working. Changing Proxmox API permissions cannotrepair a Pulse command-channel connection.-For a systemd agent, inspect the recent journal **locally on the affected node**:--```bash-sudo journalctl -u pulse-agent.service --since '15 minutes ago' -n 200 --no-pager --output=cat-```+For a systemd agent, inspect the recent journal **locally on the affected node**+with the [bounded agent log reader](#collect-agent-logs-safely).Look for **Connected and registered with Pulse command server**, or**WebSocket connection failed repeatedly, reconnecting** and its error. A@@ -1182,10 +1232,10 @@ launchctl list | grep pulseIf your Docker Swarm cluster isn't being detected:-1. **Check runtime detection**: Pulse disables Swarm for Podman. Look for "Podman runtime detected" in logs:- ```bash- journalctl -u pulse-agent | grep -i podman- ```+1. **Check runtime detection**: Pulse disables Swarm for Podman. Look locally+ for "Podman runtime detected" in the existing excerpt from the+ [bounded agent log reader](#collect-agent-logs-safely). An empty search does+ not establish that Docker was detected or that Swarm collection succeeded.2. **Force Docker runtime**: If auto-detection is incorrect:```bash@@ -1205,12 +1255,11 @@ If your Docker Swarm cluster isn't being detected:typed-helper summary does not include Swarm inventory; granting broadersocket access is a security decision, not a routine permission repair.-5. **Enable debug logging**: For more detail:- ```bash- # Set the service to debug as described under "Agent log level", restart it,- # then follow the service journal.- journalctl -u pulse-agent -f- ```+5. **Keep the original failure evidence**: retain the failure stage and a+ reviewed, redacted error, or say that no relevant entry is available. Do+ not enable debug logging, restart the agent or change its runtime merely+ to obtain a log entry. Use the existing observations before deciding that+ any configuration repair is needed.### PVE Backups Not Showing (Recovery)diff --git a/frontend-modern/public/docs/UNIFIED_AGENT.md b/frontend-modern/public/docs/UNIFIED_AGENT.mdindex 7d053904a7..ac61a2ea77 100644--- a/frontend-modern/public/docs/UNIFIED_AGENT.md+++ b/frontend-modern/public/docs/UNIFIED_AGENT.md@@ -582,11 +582,15 @@ Apply and verify the change:```bashsudo systemctl daemon-reloadsudo systemctl restart pulse-agent-sudo journalctl -u pulse-agent --since "5 minutes ago"```-Set the value to `debug` temporarily when collecting diagnostics, then restore-`info` or `warn`. For a container agent, set `LOG_LEVEL=warn` in the container+These are configuration changes, not evidence-collection steps: apply them in+a suitable maintenance window, not just to file a report. Inspect the existing+journal with the [bounded agent log reader](#collect-agent-logs-safely).+Start with the current log level; changing it to `debug` can expose more private+details and a restart changes the run being investigated. Do not enable debug+logging or restart the agent merely to manufacture a missing diagnostic entry.+For a container agent, set `LOG_LEVEL=warn` in the containerenvironment and recreate the container. Pro installations can also manage`log_level` with an [agent configuration profile](CENTRALIZED_MANAGEMENT.md).@@ -921,6 +925,54 @@ Set `--health-addr=""` or `PULSE_HEALTH_ADDR=off` to disable the health/metrics## Troubleshooting+### Collect agent logs safely++Prefer existing observations from the original failure. On a **Linux systemd+agent host**, use this bounded Bash reader with an account already authorised+to read that journal. Run it on the affected agent host, not the Pulse server+or a different Proxmox node; use the actual service name and original time+window if they differ from the example. This reads logs only: it does not+contact Pulse, a container runtime or a guest agent.++```bash+(+command -v timeout >/dev/null 2>&1 || {+ printf 'Agent log read unavailable: GNU timeout is required; no unbounded fallback.\n' >&2+ exit 1+}+if agent_logs=$(timeout --signal=TERM --kill-after=1s 8s journalctl -u pulse-agent.service --since '15 minutes ago' --lines 200 --no-pager --output=cat 2>&1); then+ if [ -n "$agent_logs" ]; then+ printf '%s\n' "$agent_logs"+ fi+else+ agent_log_status=$?+ printf 'Agent log read unavailable (exit %s); no partial excerpt shown.\n' "$agent_log_status" >&2+ exit "$agent_log_status"+fi+)+```++This requires **GNU `timeout`**, reads at most 200 records from the default+15-minute window, and allows eight seconds plus a one-second termination grace.+A failed or timed-out read is unavailable, not an empty search; partial output+is withheld. Stop if the utility, service journal or existing access is+unavailable; do not follow the journal indefinitely, remove the deadline or+broaden permissions to collect it. A successful empty read is inconclusive:+the event may be outside the window or absent at the configured log level.+Do not restart, re-enrol or enable debug logging just to obtain an entry.++The excerpt is **not sanitised**. Review it locally before searching for the+relevant error or sharing a manually redacted timestamp and failure stage.+Tokens, cookies, URLs, private names or addresses can appear in errors; do not+post the full excerpt, service environment, connection files or saved identity.+A log message does not prove fresh readings, successful updates or delivery.++For a container agent or a non-systemd host, use the existing deployment's log+source instead; this journal recipe does not apply. Keep its read bounded and+private, and report evidence as unavailable if safe collection is not possible.+QNAP and Unraid log locations are described in the+[space troubleshooting section](#installer-fails-with-not-enough-free-disk-space).+### pfSense service disabled after a major upgradePulse installs two service files on pfSense: the FreeBSD rc.d service at@@ -1002,7 +1054,8 @@ space problem is resolved; verify fresh reporting, free space and log growthafter any planned repair. Keep log contents, connection files and tokens private.### Agent Not Updating-- Check logs: `journalctl -u pulse-agent -f`+- Inspect existing errors with the [bounded agent log reader](#collect-agent-logs-safely);+ a missing update entry is not proof that an update succeeded.- Verify network connectivity to Pulse server- Ensure auto-update is not disabled- Confirm the agent can authenticate and that its saved connection state still@@ -1147,11 +1200,8 @@ monitoring report or **Automatic updates ready** also does not prove that thisseparate WebSocket channel is working. Changing Proxmox API permissions cannotrepair a Pulse command-channel connection.-For a systemd agent, inspect the recent journal **locally on the affected node**:--```bash-sudo journalctl -u pulse-agent.service --since '15 minutes ago' -n 200 --no-pager --output=cat-```+For a systemd agent, inspect the recent journal **locally on the affected node**+with the [bounded agent log reader](#collect-agent-logs-safely).Look for **Connected and registered with Pulse command server**, or**WebSocket connection failed repeatedly, reconnecting** and its error. A@@ -1182,10 +1232,10 @@ launchctl list | grep pulseIf your Docker Swarm cluster isn't being detected:-1. **Check runtime detection**: Pulse disables Swarm for Podman. Look for "Podman runtime detected" in logs:- ```bash- journalctl -u pulse-agent | grep -i podman- ```+1. **Check runtime detection**: Pulse disables Swarm for Podman. Look locally+ for "Podman runtime detected" in the existing excerpt from the+ [bounded agent log reader](#collect-agent-logs-safely). An empty search does+ not establish that Docker was detected or that Swarm collection succeeded.2. **Force Docker runtime**: If auto-detection is incorrect:```bash@@ -1205,12 +1255,11 @@ If your Docker Swarm cluster isn't being detected:typed-helper summary does not include Swarm inventory; granting broadersocket access is a security decision, not a routine permission repair.-5. **Enable debug logging**: For more detail:- ```bash- # Set the service to debug as described under "Agent log level", restart it,- # then follow the service journal.- journalctl -u pulse-agent -f- ```+5. **Keep the original failure evidence**: retain the failure stage and a+ reviewed, redacted error, or say that no relevant entry is available. Do+ not enable debug logging, restart the agent or change its runtime merely+ to obtain a log entry. Use the existing observations before deciding that+ any configuration repair is needed.### PVE Backups Not Showing (Recovery)diff --git a/scripts/tests/test_agent_log_docs.py b/scripts/tests/test_agent_log_docs.pynew file mode 100644index 0000000000..f477524fd8--- /dev/null+++ b/scripts/tests/test_agent_log_docs.py@@ -0,0 +1,129 @@+#!/usr/bin/env python3+"""Exercise the copied agent log reader against synthetic journal processes.++No real journal, service, credential, agent or runtime is accessed. These+controls establish recipe bounds and failure handling, not native recovery.+"""++from pathlib import Path+import re+import subprocess+import unittest++from test_troubleshooting_logs import exercise_log_recipe+++ROOT = Path(__file__).resolve().parents[2]+DOC = ROOT / "docs/UNIFIED_AGENT.md"+LOG = "Podman runtime detected\n"+PRIVATE_ERROR = "synthetic-private-endpoint-and-token\n"+++def section(heading):+ text = DOC.read_text(encoding="utf-8")+ marker = "### " + heading + "\n"+ if marker not in text:+ raise AssertionError(f"missing section: {heading}")+ return re.split(r"\n#{2,3} ", text.split(marker, 1)[1], 1)[0]+++def recipe():+ blocks = re.findall(r"```bash\n(.*?)```", section("Collect agent logs safely"), re.DOTALL)+ if len(blocks) != 1:+ raise AssertionError("expected one shared systemd journal reader")+ return blocks[0]+++class AgentLogDocsTest(unittest.TestCase):+ def exercise(self, **settings):+ return exercise_log_recipe("journalctl", recipe(), **settings)++ def test_shipped_guide_matches_source(self):+ self.assertEqual(DOC.read_bytes(),+ (ROOT / "frontend-modern/public/docs/UNIFIED_AGENT.md").read_bytes())++ def test_copied_reader_is_bounded_and_read_only(self):+ copied = recipe()+ subprocess.run(["bash", "-n", "-c", copied], check=True, capture_output=True)+ result, argv = self.exercise(stdout=LOG)+ self.assertEqual(result.returncode, 0, result.stderr)+ self.assertEqual(result.stdout, LOG)+ self.assertEqual(result.stderr, "")+ self.assertEqual(argv, ["-u", "pulse-agent.service", "--since", "15 minutes ago",+ "--lines", "200", "--no-pager", "--output=cat"])+ self.assertIn("timeout --signal=TERM --kill-after=1s 8s", copied)+ self.assertIn("2>&1", copied)+ self.assertNotRegex(copied, r"\b(?:grep|curl|docker|systemctl|printenv|sudo)\b|--follow|--token")++ def test_both_streams_survive_success_without_claiming_redaction(self):+ result, _ = self.exercise(stdout=LOG, stderr=PRIVATE_ERROR)+ self.assertEqual(result.returncode, 0, result.stderr)+ # Independently buffered streams do not establish a total order.+ self.assertCountEqual(result.stdout.splitlines(), (LOG + PRIVATE_ERROR).splitlines())+ self.assertEqual(result.stderr, "")++ def test_reader_failure_withholds_partial_logs_and_keeps_exit(self):+ for output in ("", LOG):+ with self.subTest(partial=bool(output)):+ result, _ = self.exercise(stdout=output, stderr=PRIVATE_ERROR, exit_code=2)+ self.assertEqual(result.returncode, 2)+ self.assertEqual(result.stdout, "")+ self.assertIn("Agent log read unavailable (exit 2)", result.stderr)+ self.assertNotIn(PRIVATE_ERROR.strip(), result.stderr)++ def test_hung_reader_is_stopped_by_the_documented_deadline(self):+ result, _ = self.exercise(stdout=LOG, stderr=PRIVATE_ERROR, hang="term")+ self.assertEqual(result.returncode, 124, result.stderr)+ self.assertEqual(result.stdout, "")+ self.assertIn("Agent log read unavailable (exit 124)", result.stderr)+ self.assertNotIn(PRIVATE_ERROR.strip(), result.stderr)++ def test_reader_ignoring_term_is_killed_after_the_documented_grace(self):+ result, _ = self.exercise(stdout=LOG, hang="ignore-term")+ self.assertEqual(result.returncode, 137, result.stderr)+ self.assertEqual(result.stdout, "")+ self.assertIn("Agent log read unavailable (exit 137)", result.stderr)++ def test_missing_timeout_does_not_start_the_reader(self):+ result, argv = self.exercise(missing_timeout=True)+ self.assertIsNone(argv)+ self.assertEqual(result.returncode, 1)+ self.assertEqual(result.stdout, "")+ self.assertIn("no unbounded fallback", result.stderr)++ def test_successful_empty_read_is_inconclusive(self):+ result, _ = self.exercise()+ self.assertEqual(result.returncode, 0, result.stderr)+ self.assertEqual(result.stdout + result.stderr, "")+ self.assertIn("successful empty read is inconclusive", section("Collect agent logs safely"))++ def test_shared_reader_preserves_scope_and_private_output_boundaries(self):+ text = " ".join(section("Collect agent logs safely").split())+ for boundary in ("Linux systemd agent host", "already authorised", "affected agent host",+ "not the Pulse server", "actual service name", "original time window",+ "does not contact Pulse", "partial output is withheld", "not sanitised",+ "manually redacted", "Tokens, cookies, URLs", "do not post the full excerpt",+ "service environment, connection files or saved identity",+ "broaden permissions", "Do not restart, re-enrol or enable debug logging",+ "container agent or a non-systemd host", "evidence as unavailable"):+ with self.subTest(boundary=boundary):+ self.assertIn(boundary, text)++ def test_diagnostic_entry_points_reuse_the_reader_without_live_followers(self):+ text = DOC.read_text(encoding="utf-8")+ log_level = text.split("### Agent log level\n", 1)[1].split("\n## ", 1)[0]+ for entry in (log_level, section("Agent Not Updating"),+ section("Commands enabled but remote control blocked"),+ section("Docker Swarm Not Detected")):+ self.assertIn("[bounded agent log reader](#collect-agent-logs-safely)", entry)+ self.assertNotIn("journalctl", entry)+ self.assertIn("not evidence-collection steps", log_level)+ self.assertIn("restart changes the run", log_level)+ self.assertNotRegex(text, r"journalctl[^\n]*(?:\s-f\b|--follow|\|\s*grep)")+ swarm = " ".join(section("Docker Swarm Not Detected").split())+ self.assertIn("An empty search does not establish", swarm)+ self.assertIn("not enable debug logging, restart the agent or change its runtime", swarm)+++if __name__ == "__main__":+ unittest.main()diff --git a/scripts/tests/test_docker_troubleshooting_docs.py b/scripts/tests/test_docker_troubleshooting_docs.pyindex de42c48233..914e03cf3d 100644--- a/scripts/tests/test_docker_troubleshooting_docs.py+++ b/scripts/tests/test_docker_troubleshooting_docs.py@@ -133,27 +133,11 @@ class DockerTroubleshootingDocsTest(unittest.TestCase):self.assertNotRegex(text, r"--property=Environment|systemctl (?:cat|restart)|--insecure")def test_command_channel_journal_check_is_bounded_and_read_only(self):- recipe = command("Commands enabled but remote control blocked")- with tempfile.TemporaryDirectory() as temporary:- root = Path(temporary)- for name, body in (- ("sudo", '#!/bin/sh\nexec "$@"\n'),- ("journalctl", '#!/usr/bin/python3\nimport json, sys\nfrom pathlib import Path\n'- 'Path("' + str(root / "argv.json") + '").write_text(json.dumps(sys.argv[1:]))\n'- 'print("WebSocket connection failed repeatedly, reconnecting")\n'),- ):- path = root / name- path.write_text(body)- path.chmod(0o700)- env = dict(os.environ, PATH=f"{root}:{os.environ['PATH']}", SYNTHETIC_SECRET=SECRET)- result = subprocess.run(["bash", "-eu", "-c", recipe], env=env,- capture_output=True, text=True, timeout=5)- self.assertEqual(result.returncode, 0, result.stderr)- self.assertEqual(json.loads((root / "argv.json").read_text()), [- "-u", "pulse-agent.service", "--since", "15 minutes ago", "-n", "200",- "--no-pager", "--output=cat",- ])- self.assertNotIn(SECRET, result.stdout + result.stderr)+ # The shared reader has executable deadline, partial-failure and stream+ # controls in test_agent_log_docs; keep this entry point bound to it.+ text = section("Commands enabled but remote control blocked")+ self.assertIn("[bounded agent log reader](#collect-agent-logs-safely)", text)+ self.assertNotIn("journalctl", text)def run_guest(self, *, socket="present", docker="ok", pct="ok", cli=True):with tempfile.TemporaryDirectory() as temporary: