Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Bound passive agent log collection across help entry points

pulse-triage[bot]4733e816e2db
diff --git a/docs/UNIFIED_AGENT.md b/docs/UNIFIED_AGENT.md
index 7d053904a7..ac61a2ea77 100644
--- a/docs/UNIFIED_AGENT.md
+++ b/docs/UNIFIED_AGENT.md
@@ -582,11 +582,15 @@ Apply and verify the change:
```bash
sudo systemctl daemon-reload
sudo systemctl restart pulse-agent
-sudo journalctl -u pulse-agent --since "5 minutes ago"
```
-Set the value to `debug` temporarily when collecting diagnostics, then restore
-`info` or `warn`. For a container agent, set `LOG_LEVEL=warn` in the container
+These are configuration changes, not evidence-collection steps: apply them in
+a suitable maintenance window, not just to file a report. Inspect the existing
+journal with the [bounded agent log reader](#collect-agent-logs-safely).
+Start with the current log level; changing it to `debug` can expose more private
+details and a restart changes the run being investigated. Do not enable debug
+logging or restart the agent merely to manufacture a missing diagnostic entry.
+For a container agent, set `LOG_LEVEL=warn` in the container
environment and recreate the container. Pro installations can also manage
`log_level` with an [agent configuration profile](CENTRALIZED_MANAGEMENT.md).
@@ -921,6 +925,54 @@ Set `--health-addr=""` or `PULSE_HEALTH_ADDR=off` to disable the health/metrics
## Troubleshooting
+### Collect agent logs safely
+
+Prefer existing observations from the original failure. On a **Linux systemd
+agent host**, use this bounded Bash reader with an account already authorised
+to read that journal. Run it on the affected agent host, not the Pulse server
+or a different Proxmox node; use the actual service name and original time
+window if they differ from the example. This reads logs only: it does not
+contact Pulse, a container runtime or a guest agent.
+
+```bash
+(
+command -v timeout >/dev/null 2>&1 || {
+ printf 'Agent log read unavailable: GNU timeout is required; no unbounded fallback.\n' >&2
+ exit 1
+}
+if agent_logs=$(timeout --signal=TERM --kill-after=1s 8s journalctl -u pulse-agent.service --since '15 minutes ago' --lines 200 --no-pager --output=cat 2>&1); then
+ if [ -n "$agent_logs" ]; then
+ printf '%s\n' "$agent_logs"
+ fi
+else
+ agent_log_status=$?
+ printf 'Agent log read unavailable (exit %s); no partial excerpt shown.\n' "$agent_log_status" >&2
+ exit "$agent_log_status"
+fi
+)
+```
+
+This requires **GNU `timeout`**, reads at most 200 records from the default
+15-minute window, and allows eight seconds plus a one-second termination grace.
+A failed or timed-out read is unavailable, not an empty search; partial output
+is withheld. Stop if the utility, service journal or existing access is
+unavailable; do not follow the journal indefinitely, remove the deadline or
+broaden permissions to collect it. A successful empty read is inconclusive:
+the event may be outside the window or absent at the configured log level.
+Do not restart, re-enrol or enable debug logging just to obtain an entry.
+
+The excerpt is **not sanitised**. Review it locally before searching for the
+relevant error or sharing a manually redacted timestamp and failure stage.
+Tokens, cookies, URLs, private names or addresses can appear in errors; do not
+post the full excerpt, service environment, connection files or saved identity.
+A log message does not prove fresh readings, successful updates or delivery.
+
+For a container agent or a non-systemd host, use the existing deployment's log
+source instead; this journal recipe does not apply. Keep its read bounded and
+private, and report evidence as unavailable if safe collection is not possible.
+QNAP and Unraid log locations are described in the
+[space troubleshooting section](#installer-fails-with-not-enough-free-disk-space).
+
### pfSense service disabled after a major upgrade
Pulse installs two service files on pfSense: the FreeBSD rc.d service at
@@ -1002,7 +1054,8 @@ space problem is resolved; verify fresh reporting, free space and log growth
after any planned repair. Keep log contents, connection files and tokens private.
### Agent Not Updating
-- Check logs: `journalctl -u pulse-agent -f`
+- Inspect existing errors with the [bounded agent log reader](#collect-agent-logs-safely);
+ a missing update entry is not proof that an update succeeded.
- Verify network connectivity to Pulse server
- Ensure auto-update is not disabled
- Confirm the agent can authenticate and that its saved connection state still
@@ -1147,11 +1200,8 @@ monitoring report or **Automatic updates ready** also does not prove that this
separate WebSocket channel is working. Changing Proxmox API permissions cannot
repair a Pulse command-channel connection.
-For a systemd agent, inspect the recent journal **locally on the affected node**:
-
-```bash
-sudo journalctl -u pulse-agent.service --since '15 minutes ago' -n 200 --no-pager --output=cat
-```
+For a systemd agent, inspect the recent journal **locally on the affected node**
+with the [bounded agent log reader](#collect-agent-logs-safely).
Look for **Connected and registered with Pulse command server**, or
**WebSocket connection failed repeatedly, reconnecting** and its error. A
@@ -1182,10 +1232,10 @@ launchctl list | grep pulse
If your Docker Swarm cluster isn't being detected:
-1. **Check runtime detection**: Pulse disables Swarm for Podman. Look for "Podman runtime detected" in logs:
- ```bash
- journalctl -u pulse-agent | grep -i podman
- ```
+1. **Check runtime detection**: Pulse disables Swarm for Podman. Look locally
+ for "Podman runtime detected" in the existing excerpt from the
+ [bounded agent log reader](#collect-agent-logs-safely). An empty search does
+ not establish that Docker was detected or that Swarm collection succeeded.
2. **Force Docker runtime**: If auto-detection is incorrect:
```bash
@@ -1205,12 +1255,11 @@ If your Docker Swarm cluster isn't being detected:
typed-helper summary does not include Swarm inventory; granting broader
socket access is a security decision, not a routine permission repair.
-5. **Enable debug logging**: For more detail:
- ```bash
- # Set the service to debug as described under "Agent log level", restart it,
- # then follow the service journal.
- journalctl -u pulse-agent -f
- ```
+5. **Keep the original failure evidence**: retain the failure stage and a
+ reviewed, redacted error, or say that no relevant entry is available. Do
+ not enable debug logging, restart the agent or change its runtime merely
+ to obtain a log entry. Use the existing observations before deciding that
+ any configuration repair is needed.
### PVE Backups Not Showing (Recovery)
diff --git a/frontend-modern/public/docs/UNIFIED_AGENT.md b/frontend-modern/public/docs/UNIFIED_AGENT.md
index 7d053904a7..ac61a2ea77 100644
--- a/frontend-modern/public/docs/UNIFIED_AGENT.md
+++ b/frontend-modern/public/docs/UNIFIED_AGENT.md
@@ -582,11 +582,15 @@ Apply and verify the change:
```bash
sudo systemctl daemon-reload
sudo systemctl restart pulse-agent
-sudo journalctl -u pulse-agent --since "5 minutes ago"
```
-Set the value to `debug` temporarily when collecting diagnostics, then restore
-`info` or `warn`. For a container agent, set `LOG_LEVEL=warn` in the container
+These are configuration changes, not evidence-collection steps: apply them in
+a suitable maintenance window, not just to file a report. Inspect the existing
+journal with the [bounded agent log reader](#collect-agent-logs-safely).
+Start with the current log level; changing it to `debug` can expose more private
+details and a restart changes the run being investigated. Do not enable debug
+logging or restart the agent merely to manufacture a missing diagnostic entry.
+For a container agent, set `LOG_LEVEL=warn` in the container
environment and recreate the container. Pro installations can also manage
`log_level` with an [agent configuration profile](CENTRALIZED_MANAGEMENT.md).
@@ -921,6 +925,54 @@ Set `--health-addr=""` or `PULSE_HEALTH_ADDR=off` to disable the health/metrics
## Troubleshooting
+### Collect agent logs safely
+
+Prefer existing observations from the original failure. On a **Linux systemd
+agent host**, use this bounded Bash reader with an account already authorised
+to read that journal. Run it on the affected agent host, not the Pulse server
+or a different Proxmox node; use the actual service name and original time
+window if they differ from the example. This reads logs only: it does not
+contact Pulse, a container runtime or a guest agent.
+
+```bash
+(
+command -v timeout >/dev/null 2>&1 || {
+ printf 'Agent log read unavailable: GNU timeout is required; no unbounded fallback.\n' >&2
+ exit 1
+}
+if agent_logs=$(timeout --signal=TERM --kill-after=1s 8s journalctl -u pulse-agent.service --since '15 minutes ago' --lines 200 --no-pager --output=cat 2>&1); then
+ if [ -n "$agent_logs" ]; then
+ printf '%s\n' "$agent_logs"
+ fi
+else
+ agent_log_status=$?
+ printf 'Agent log read unavailable (exit %s); no partial excerpt shown.\n' "$agent_log_status" >&2
+ exit "$agent_log_status"
+fi
+)
+```
+
+This requires **GNU `timeout`**, reads at most 200 records from the default
+15-minute window, and allows eight seconds plus a one-second termination grace.
+A failed or timed-out read is unavailable, not an empty search; partial output
+is withheld. Stop if the utility, service journal or existing access is
+unavailable; do not follow the journal indefinitely, remove the deadline or
+broaden permissions to collect it. A successful empty read is inconclusive:
+the event may be outside the window or absent at the configured log level.
+Do not restart, re-enrol or enable debug logging just to obtain an entry.
+
+The excerpt is **not sanitised**. Review it locally before searching for the
+relevant error or sharing a manually redacted timestamp and failure stage.
+Tokens, cookies, URLs, private names or addresses can appear in errors; do not
+post the full excerpt, service environment, connection files or saved identity.
+A log message does not prove fresh readings, successful updates or delivery.
+
+For a container agent or a non-systemd host, use the existing deployment's log
+source instead; this journal recipe does not apply. Keep its read bounded and
+private, and report evidence as unavailable if safe collection is not possible.
+QNAP and Unraid log locations are described in the
+[space troubleshooting section](#installer-fails-with-not-enough-free-disk-space).
+
### pfSense service disabled after a major upgrade
Pulse installs two service files on pfSense: the FreeBSD rc.d service at
@@ -1002,7 +1054,8 @@ space problem is resolved; verify fresh reporting, free space and log growth
after any planned repair. Keep log contents, connection files and tokens private.
### Agent Not Updating
-- Check logs: `journalctl -u pulse-agent -f`
+- Inspect existing errors with the [bounded agent log reader](#collect-agent-logs-safely);
+ a missing update entry is not proof that an update succeeded.
- Verify network connectivity to Pulse server
- Ensure auto-update is not disabled
- Confirm the agent can authenticate and that its saved connection state still
@@ -1147,11 +1200,8 @@ monitoring report or **Automatic updates ready** also does not prove that this
separate WebSocket channel is working. Changing Proxmox API permissions cannot
repair a Pulse command-channel connection.
-For a systemd agent, inspect the recent journal **locally on the affected node**:
-
-```bash
-sudo journalctl -u pulse-agent.service --since '15 minutes ago' -n 200 --no-pager --output=cat
-```
+For a systemd agent, inspect the recent journal **locally on the affected node**
+with the [bounded agent log reader](#collect-agent-logs-safely).
Look for **Connected and registered with Pulse command server**, or
**WebSocket connection failed repeatedly, reconnecting** and its error. A
@@ -1182,10 +1232,10 @@ launchctl list | grep pulse
If your Docker Swarm cluster isn't being detected:
-1. **Check runtime detection**: Pulse disables Swarm for Podman. Look for "Podman runtime detected" in logs:
- ```bash
- journalctl -u pulse-agent | grep -i podman
- ```
+1. **Check runtime detection**: Pulse disables Swarm for Podman. Look locally
+ for "Podman runtime detected" in the existing excerpt from the
+ [bounded agent log reader](#collect-agent-logs-safely). An empty search does
+ not establish that Docker was detected or that Swarm collection succeeded.
2. **Force Docker runtime**: If auto-detection is incorrect:
```bash
@@ -1205,12 +1255,11 @@ If your Docker Swarm cluster isn't being detected:
typed-helper summary does not include Swarm inventory; granting broader
socket access is a security decision, not a routine permission repair.
-5. **Enable debug logging**: For more detail:
- ```bash
- # Set the service to debug as described under "Agent log level", restart it,
- # then follow the service journal.
- journalctl -u pulse-agent -f
- ```
+5. **Keep the original failure evidence**: retain the failure stage and a
+ reviewed, redacted error, or say that no relevant entry is available. Do
+ not enable debug logging, restart the agent or change its runtime merely
+ to obtain a log entry. Use the existing observations before deciding that
+ any configuration repair is needed.
### PVE Backups Not Showing (Recovery)
diff --git a/scripts/tests/test_agent_log_docs.py b/scripts/tests/test_agent_log_docs.py
new file mode 100644
index 0000000000..f477524fd8
--- /dev/null
+++ b/scripts/tests/test_agent_log_docs.py
@@ -0,0 +1,129 @@
+#!/usr/bin/env python3
+"""Exercise the copied agent log reader against synthetic journal processes.
+
+No real journal, service, credential, agent or runtime is accessed. These
+controls establish recipe bounds and failure handling, not native recovery.
+"""
+
+from pathlib import Path
+import re
+import subprocess
+import unittest
+
+from test_troubleshooting_logs import exercise_log_recipe
+
+
+ROOT = Path(__file__).resolve().parents[2]
+DOC = ROOT / "docs/UNIFIED_AGENT.md"
+LOG = "Podman runtime detected\n"
+PRIVATE_ERROR = "synthetic-private-endpoint-and-token\n"
+
+
+def section(heading):
+ text = DOC.read_text(encoding="utf-8")
+ marker = "### " + heading + "\n"
+ if marker not in text:
+ raise AssertionError(f"missing section: {heading}")
+ return re.split(r"\n#{2,3} ", text.split(marker, 1)[1], 1)[0]
+
+
+def recipe():
+ blocks = re.findall(r"```bash\n(.*?)```", section("Collect agent logs safely"), re.DOTALL)
+ if len(blocks) != 1:
+ raise AssertionError("expected one shared systemd journal reader")
+ return blocks[0]
+
+
+class AgentLogDocsTest(unittest.TestCase):
+ def exercise(self, **settings):
+ return exercise_log_recipe("journalctl", recipe(), **settings)
+
+ def test_shipped_guide_matches_source(self):
+ self.assertEqual(DOC.read_bytes(),
+ (ROOT / "frontend-modern/public/docs/UNIFIED_AGENT.md").read_bytes())
+
+ def test_copied_reader_is_bounded_and_read_only(self):
+ copied = recipe()
+ subprocess.run(["bash", "-n", "-c", copied], check=True, capture_output=True)
+ result, argv = self.exercise(stdout=LOG)
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertEqual(result.stdout, LOG)
+ self.assertEqual(result.stderr, "")
+ self.assertEqual(argv, ["-u", "pulse-agent.service", "--since", "15 minutes ago",
+ "--lines", "200", "--no-pager", "--output=cat"])
+ self.assertIn("timeout --signal=TERM --kill-after=1s 8s", copied)
+ self.assertIn("2>&1", copied)
+ self.assertNotRegex(copied, r"\b(?:grep|curl|docker|systemctl|printenv|sudo)\b|--follow|--token")
+
+ def test_both_streams_survive_success_without_claiming_redaction(self):
+ result, _ = self.exercise(stdout=LOG, stderr=PRIVATE_ERROR)
+ self.assertEqual(result.returncode, 0, result.stderr)
+ # Independently buffered streams do not establish a total order.
+ self.assertCountEqual(result.stdout.splitlines(), (LOG + PRIVATE_ERROR).splitlines())
+ self.assertEqual(result.stderr, "")
+
+ def test_reader_failure_withholds_partial_logs_and_keeps_exit(self):
+ for output in ("", LOG):
+ with self.subTest(partial=bool(output)):
+ result, _ = self.exercise(stdout=output, stderr=PRIVATE_ERROR, exit_code=2)
+ self.assertEqual(result.returncode, 2)
+ self.assertEqual(result.stdout, "")
+ self.assertIn("Agent log read unavailable (exit 2)", result.stderr)
+ self.assertNotIn(PRIVATE_ERROR.strip(), result.stderr)
+
+ def test_hung_reader_is_stopped_by_the_documented_deadline(self):
+ result, _ = self.exercise(stdout=LOG, stderr=PRIVATE_ERROR, hang="term")
+ self.assertEqual(result.returncode, 124, result.stderr)
+ self.assertEqual(result.stdout, "")
+ self.assertIn("Agent log read unavailable (exit 124)", result.stderr)
+ self.assertNotIn(PRIVATE_ERROR.strip(), result.stderr)
+
+ def test_reader_ignoring_term_is_killed_after_the_documented_grace(self):
+ result, _ = self.exercise(stdout=LOG, hang="ignore-term")
+ self.assertEqual(result.returncode, 137, result.stderr)
+ self.assertEqual(result.stdout, "")
+ self.assertIn("Agent log read unavailable (exit 137)", result.stderr)
+
+ def test_missing_timeout_does_not_start_the_reader(self):
+ result, argv = self.exercise(missing_timeout=True)
+ self.assertIsNone(argv)
+ self.assertEqual(result.returncode, 1)
+ self.assertEqual(result.stdout, "")
+ self.assertIn("no unbounded fallback", result.stderr)
+
+ def test_successful_empty_read_is_inconclusive(self):
+ result, _ = self.exercise()
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertEqual(result.stdout + result.stderr, "")
+ self.assertIn("successful empty read is inconclusive", section("Collect agent logs safely"))
+
+ def test_shared_reader_preserves_scope_and_private_output_boundaries(self):
+ text = " ".join(section("Collect agent logs safely").split())
+ for boundary in ("Linux systemd agent host", "already authorised", "affected agent host",
+ "not the Pulse server", "actual service name", "original time window",
+ "does not contact Pulse", "partial output is withheld", "not sanitised",
+ "manually redacted", "Tokens, cookies, URLs", "do not post the full excerpt",
+ "service environment, connection files or saved identity",
+ "broaden permissions", "Do not restart, re-enrol or enable debug logging",
+ "container agent or a non-systemd host", "evidence as unavailable"):
+ with self.subTest(boundary=boundary):
+ self.assertIn(boundary, text)
+
+ def test_diagnostic_entry_points_reuse_the_reader_without_live_followers(self):
+ text = DOC.read_text(encoding="utf-8")
+ log_level = text.split("### Agent log level\n", 1)[1].split("\n## ", 1)[0]
+ for entry in (log_level, section("Agent Not Updating"),
+ section("Commands enabled but remote control blocked"),
+ section("Docker Swarm Not Detected")):
+ self.assertIn("[bounded agent log reader](#collect-agent-logs-safely)", entry)
+ self.assertNotIn("journalctl", entry)
+ self.assertIn("not evidence-collection steps", log_level)
+ self.assertIn("restart changes the run", log_level)
+ self.assertNotRegex(text, r"journalctl[^\n]*(?:\s-f\b|--follow|\|\s*grep)")
+ swarm = " ".join(section("Docker Swarm Not Detected").split())
+ self.assertIn("An empty search does not establish", swarm)
+ self.assertIn("not enable debug logging, restart the agent or change its runtime", swarm)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/scripts/tests/test_docker_troubleshooting_docs.py b/scripts/tests/test_docker_troubleshooting_docs.py
index de42c48233..914e03cf3d 100644
--- a/scripts/tests/test_docker_troubleshooting_docs.py
+++ b/scripts/tests/test_docker_troubleshooting_docs.py
@@ -133,27 +133,11 @@ class DockerTroubleshootingDocsTest(unittest.TestCase):
self.assertNotRegex(text, r"--property=Environment|systemctl (?:cat|restart)|--insecure")
def test_command_channel_journal_check_is_bounded_and_read_only(self):
- recipe = command("Commands enabled but remote control blocked")
- with tempfile.TemporaryDirectory() as temporary:
- root = Path(temporary)
- for name, body in (
- ("sudo", '#!/bin/sh\nexec "$@"\n'),
- ("journalctl", '#!/usr/bin/python3\nimport json, sys\nfrom pathlib import Path\n'
- 'Path("' + str(root / "argv.json") + '").write_text(json.dumps(sys.argv[1:]))\n'
- 'print("WebSocket connection failed repeatedly, reconnecting")\n'),
- ):
- path = root / name
- path.write_text(body)
- path.chmod(0o700)
- env = dict(os.environ, PATH=f"{root}:{os.environ['PATH']}", SYNTHETIC_SECRET=SECRET)
- result = subprocess.run(["bash", "-eu", "-c", recipe], env=env,
- capture_output=True, text=True, timeout=5)
- self.assertEqual(result.returncode, 0, result.stderr)
- self.assertEqual(json.loads((root / "argv.json").read_text()), [
- "-u", "pulse-agent.service", "--since", "15 minutes ago", "-n", "200",
- "--no-pager", "--output=cat",
- ])
- self.assertNotIn(SECRET, result.stdout + result.stderr)
+ # The shared reader has executable deadline, partial-failure and stream
+ # controls in test_agent_log_docs; keep this entry point bound to it.
+ text = section("Commands enabled but remote control blocked")
+ self.assertIn("[bounded agent log reader](#collect-agent-logs-safely)", text)
+ self.assertNotIn("journalctl", text)
def run_guest(self, *, socket="present", docker="ok", pct="ok", cli=True):
with tempfile.TemporaryDirectory() as temporary: