Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Bound severity-filtered incident logs without reinstalling Pulse
pulse-triage[bot]0fbc00609f54
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.mdindex a10dd2ad66..307d70e167 100644--- a/docs/TROUBLESHOOTING.md+++ b/docs/TROUBLESHOOTING.md@@ -309,18 +309,48 @@ first step before platform-local cleanup.#### Filter Pulse's systemd journal by severityCurrent systemd installs preserve Pulse's structured log level as the journal-priority. For example, show warnings and more important records with:+priority. For warnings and more important records, use this bounded Bash reader+on the Pulse host, with an account authorised to read its journal. In Proxmox+LXC, run it inside the Pulse container, not on the Proxmox host. Substitute the+actual service name (`pulse-backend` on some older installs) and original+incident's time window; no new action or request ID is needed.```bash-sudo journalctl -u pulse -p warning+# systemd / Proxmox LXC — warning and higher priority+(+command -v timeout >/dev/null 2>&1 || {+ printf 'Log read unavailable: GNU timeout is required; no unbounded fallback.\n' >&2+ exit 1+}+if pulse_logs=$(timeout --signal=TERM --kill-after=1s 8s journalctl -u pulse -p warning --since '15 minutes ago' --lines 200 --no-pager 2>&1); then+ if [ -n "$pulse_logs" ]; then+ printf '%s\n' "$pulse_logs"+ fi+else+ pulse_log_status=$?+ printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2+ exit "$pulse_log_status"+fi+)```+This requires **GNU `timeout`**, reads at most 200 records from the last 15+minutes and allows eight seconds plus a one-second termination grace. A failed+or timed-out read is unavailable, not an empty warning search; no partial+excerpt is shown. Stop if the utility or journal access is unavailable; do not+use an unbounded substitute. These local excerpts are **not sanitised**; follow+[log-sharing precautions](#inspect-notification-logs) before sharing any line.+The stored message remains JSON (`"level":"warn"`, for example), while-`PRIORITY` is available to `journalctl` and syslog forwarding. If a current-Pulse warning appears only with `-p info`, inspect `systemctl cat pulse`; the-installed service must contain `SyslogLevelPrefix=true` and-`PULSE_LOG_JOURNAL_LEVEL_PREFIX=true`. Re-run the current signed installer to-repair an older generated unit rather than adding a JSON-parsing wrapper.+`PRIORITY` is available to `journalctl` and syslog forwarding. An empty filtered+read does not prove that no warning occurred: an older generated unit may store+all records at info priority. Use the [bounded unfiltered systemd reader](#inspect-notification-logs)+to retain the original incident evidence instead. An administrator can check+locally whether the service configuration contains `SyslogLevelPrefix=true` and+`PULSE_LOG_JOURNAL_LEVEL_PREFIX=true`; do not post the full unit or environment,+which can contain credentials. Do not reinstall or restart Pulse just to obtain+logs. Repair an older generated unit through the current signed installer at a+planned maintenance window, preserving the deployment's settings and data.#### VMs show "-" for disk usage- Read the disk value's explanation and observation time first; a dash is notdiff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.mdindex a10dd2ad66..307d70e167 100644--- a/frontend-modern/public/docs/TROUBLESHOOTING.md+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md@@ -309,18 +309,48 @@ first step before platform-local cleanup.#### Filter Pulse's systemd journal by severityCurrent systemd installs preserve Pulse's structured log level as the journal-priority. For example, show warnings and more important records with:+priority. For warnings and more important records, use this bounded Bash reader+on the Pulse host, with an account authorised to read its journal. In Proxmox+LXC, run it inside the Pulse container, not on the Proxmox host. Substitute the+actual service name (`pulse-backend` on some older installs) and original+incident's time window; no new action or request ID is needed.```bash-sudo journalctl -u pulse -p warning+# systemd / Proxmox LXC — warning and higher priority+(+command -v timeout >/dev/null 2>&1 || {+ printf 'Log read unavailable: GNU timeout is required; no unbounded fallback.\n' >&2+ exit 1+}+if pulse_logs=$(timeout --signal=TERM --kill-after=1s 8s journalctl -u pulse -p warning --since '15 minutes ago' --lines 200 --no-pager 2>&1); then+ if [ -n "$pulse_logs" ]; then+ printf '%s\n' "$pulse_logs"+ fi+else+ pulse_log_status=$?+ printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2+ exit "$pulse_log_status"+fi+)```+This requires **GNU `timeout`**, reads at most 200 records from the last 15+minutes and allows eight seconds plus a one-second termination grace. A failed+or timed-out read is unavailable, not an empty warning search; no partial+excerpt is shown. Stop if the utility or journal access is unavailable; do not+use an unbounded substitute. These local excerpts are **not sanitised**; follow+[log-sharing precautions](#inspect-notification-logs) before sharing any line.+The stored message remains JSON (`"level":"warn"`, for example), while-`PRIORITY` is available to `journalctl` and syslog forwarding. If a current-Pulse warning appears only with `-p info`, inspect `systemctl cat pulse`; the-installed service must contain `SyslogLevelPrefix=true` and-`PULSE_LOG_JOURNAL_LEVEL_PREFIX=true`. Re-run the current signed installer to-repair an older generated unit rather than adding a JSON-parsing wrapper.+`PRIORITY` is available to `journalctl` and syslog forwarding. An empty filtered+read does not prove that no warning occurred: an older generated unit may store+all records at info priority. Use the [bounded unfiltered systemd reader](#inspect-notification-logs)+to retain the original incident evidence instead. An administrator can check+locally whether the service configuration contains `SyslogLevelPrefix=true` and+`PULSE_LOG_JOURNAL_LEVEL_PREFIX=true`; do not post the full unit or environment,+which can contain credentials. Do not reinstall or restart Pulse just to obtain+logs. Repair an older generated unit through the current signed installer at a+planned maintenance window, preserving the deployment's settings and data.#### VMs show "-" for disk usage- Read the disk value's explanation and observation time first; a dash is notdiff --git a/scripts/tests/test_troubleshooting_logs.py b/scripts/tests/test_troubleshooting_logs.pyindex 5c2025e417..47b96f4ea1 100644--- a/scripts/tests/test_troubleshooting_logs.py+++ b/scripts/tests/test_troubleshooting_logs.py@@ -235,5 +235,76 @@ class GeneralLogEvidenceTest(unittest.TestCase):self.assertIn("no unbounded fallback", result.stderr)+class SeverityLogEvidenceTest(unittest.TestCase):+ """Priority filtering must not bypass the incident reader's safety bounds."""++ heading = "#### Filter Pulse's systemd journal by severity"++ def copied(self):+ # Stop at the next H4; the generic section helper stops at H3.+ text = section(self.heading).split("\n#### ", 1)[0]+ blocks = re.findall(r"```bash\n(.*?)```", text, re.DOTALL)+ self.assertEqual(len(blocks), 1)+ return blocks[0]++ def exercise(self, **settings):+ result, argv = exercise_log_recipe("journalctl", self.copied(), **settings)+ expected = ["-u", "pulse", "-p", "warning", "--since", "15 minutes ago",+ "--lines", "200", "--no-pager"]+ self.assertEqual(argv, None if settings.get("missing_timeout") else expected)+ return result++ def test_priority_reader_keeps_incident_bounds_without_privilege_or_mutation(self):+ copied = self.copied()+ for bound in ("--signal=TERM --kill-after=1s 8s", "--since '15 minutes ago'",+ "--lines 200", "--no-pager"):+ self.assertIn(bound, copied)+ self.assertNotRegex(copied, r"\b(?:sudo|systemctl|curl|grep)\b|--follow")+ self.assertEqual(DOC.read_bytes(),+ (ROOT / "frontend-modern/public/docs/TROUBLESHOOTING.md").read_bytes())++ def test_success_keeps_both_complete_streams(self):+ result = self.exercise(stdout=JSON_LOG, stderr=CONSOLE_LOG)+ self.assertEqual(result.returncode, 0, result.stderr)+ self.assertCountEqual(result.stdout.splitlines(),+ (JSON_LOG + CONSOLE_LOG).splitlines())+ self.assertEqual(result.stderr, "")++ def test_failed_reader_withholds_partial_warning_and_private_error(self):+ result = self.exercise(stdout=JSON_LOG, stderr="synthetic private error\n", exit_code=2)+ self.assertEqual(result.returncode, 2)+ self.assertEqual(result.stdout, "")+ self.assertIn("Log read unavailable (exit 2)", result.stderr)+ self.assertNotIn("synthetic private error", result.stderr)++ def test_real_deadline_withholds_partial_warning(self):+ result = self.exercise(stdout=JSON_LOG, hang="term")+ self.assertEqual(result.returncode, 124, result.stderr)+ self.assertEqual(result.stdout, "")+ self.assertIn("Log read unavailable (exit 124)", result.stderr)++ def test_missing_timeout_stops_before_reading(self):+ result = self.exercise(missing_timeout=True)+ self.assertEqual(result.returncode, 1)+ self.assertEqual(result.stdout, "")+ self.assertIn("no unbounded fallback", result.stderr)++ def test_empty_success_does_not_invent_a_warning_or_reader_error(self):+ result = self.exercise()+ self.assertEqual((result.returncode, result.stdout, result.stderr), (0, "", ""))++ def test_old_unit_and_sharing_guidance_does_not_require_incident_reinstallation(self):+ guide = " ".join(section(self.heading).split("\n#### ", 1)[0].split())+ for boundary in ("inside the Pulse container, not on the Proxmox host",+ "not sanitised", "#inspect-notification-logs",+ "An empty filtered read does not prove", "info priority",+ "bounded unfiltered systemd reader", "check locally",+ "do not post the full unit or environment",+ "Do not reinstall or restart Pulse just to obtain logs",+ "planned maintenance window", "settings and data"):+ self.assertIn(boundary, guide)+ self.assertNotIn("systemctl cat", guide)++if __name__ == "__main__":unittest.main()