Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Route TrueNAS diagnostics through bounded fail-closed log readers
pulse-triage[bot]26db2344de47
diff --git a/docs/TRUENAS.md b/docs/TRUENAS.mdindex a1b7b1bf67..238a333687 100644--- a/docs/TRUENAS.md+++ b/docs/TRUENAS.md@@ -255,31 +255,30 @@ Distinguish these other results before choosing an action:[polling checks](#stale-truenas-data) to distinguish a completed failure frommissing or stale observations; a connection test is not a substitute.-If a local log excerpt is needed, run only the reader for your deployment on-the machine running **Pulse**, using an account authorised to read its logs.-For Proxmox LXC, run it **inside the Pulse container**, not on the Proxmox host-or the TrueNAS appliance. Substitute the actual service or container name-(`pulse-backend` on some older systemd installs), and adjust the time window to-the original incident. These commands read at most 100 records from the last-15 minutes; they do not follow logs, make API requests or change logging:--```bash-# systemd / Proxmox LXC-journalctl -u pulse --since '15 minutes ago' --lines 100 --no-pager-```--```bash-# Docker-docker logs --since 15m --tail 100 pulse-```--Inspect both output streams; Docker can write application logs to stderr.-Do not pipe the reader into `grep truenas`: it can hide a failed read behind a-matching partial line and omit relevant startup or storage errors. A nonzero-reader exit, denied read or missing service/container is a failed read, not-"no TrueNAS errors". Even a successful empty read is inconclusive: check the-window and selected instance, not the API key. Do not enable Debug or repeat-the failing action just to collect more logs.+If a local log excerpt is needed, use the+[bounded Pulse log readers](TROUBLESHOOTING.md#inspect-notification-logs).+Despite that section's notification heading, the unfiltered readers also+cover TrueNAS polling and startup errors; no request ID is required. Choose+only the reader for the deployment running **Pulse**, using an account+authorised to read its logs. For Proxmox LXC, run it **inside the Pulse+container**, not on the Proxmox host or the TrueNAS appliance. Substitute the+actual service or container name (`pulse-backend` on some older systemd+installs), and adjust the time window to the original incident.++Those Bash recipes read at most 200 records from the last 15 minutes and+require **GNU `timeout`**, with an eight-second deadline and one-second+termination grace. A record limit alone does not bound a hung reader. If+`timeout` is unavailable, stop; do not use an unbounded substitute. They do+not follow logs, make API requests or change logging.++Both output streams are captured because Docker can write application logs+to stderr. An excerpt is shown only after a successful read; failure or+timeout withholds partial output and reports the exit code. A nonzero reader+exit, denied read or missing service/container is a failed read, not+"no TrueNAS errors". Do not filter with `grep truenas`: relevant startup or+storage errors may not contain that word. Even a successful empty read is+inconclusive: check the window and selected instance, not the API key. Do not+enable Debug or repeat the failing action just to collect more logs.Local logs are **not sanitised**. Share only the relevant timestamp, method,HTTP status or error category and a manually redacted error, not the wholediff --git a/frontend-modern/public/docs/TRUENAS.md b/frontend-modern/public/docs/TRUENAS.mdindex a1b7b1bf67..238a333687 100644--- a/frontend-modern/public/docs/TRUENAS.md+++ b/frontend-modern/public/docs/TRUENAS.md@@ -255,31 +255,30 @@ Distinguish these other results before choosing an action:[polling checks](#stale-truenas-data) to distinguish a completed failure frommissing or stale observations; a connection test is not a substitute.-If a local log excerpt is needed, run only the reader for your deployment on-the machine running **Pulse**, using an account authorised to read its logs.-For Proxmox LXC, run it **inside the Pulse container**, not on the Proxmox host-or the TrueNAS appliance. Substitute the actual service or container name-(`pulse-backend` on some older systemd installs), and adjust the time window to-the original incident. These commands read at most 100 records from the last-15 minutes; they do not follow logs, make API requests or change logging:--```bash-# systemd / Proxmox LXC-journalctl -u pulse --since '15 minutes ago' --lines 100 --no-pager-```--```bash-# Docker-docker logs --since 15m --tail 100 pulse-```--Inspect both output streams; Docker can write application logs to stderr.-Do not pipe the reader into `grep truenas`: it can hide a failed read behind a-matching partial line and omit relevant startup or storage errors. A nonzero-reader exit, denied read or missing service/container is a failed read, not-"no TrueNAS errors". Even a successful empty read is inconclusive: check the-window and selected instance, not the API key. Do not enable Debug or repeat-the failing action just to collect more logs.+If a local log excerpt is needed, use the+[bounded Pulse log readers](TROUBLESHOOTING.md#inspect-notification-logs).+Despite that section's notification heading, the unfiltered readers also+cover TrueNAS polling and startup errors; no request ID is required. Choose+only the reader for the deployment running **Pulse**, using an account+authorised to read its logs. For Proxmox LXC, run it **inside the Pulse+container**, not on the Proxmox host or the TrueNAS appliance. Substitute the+actual service or container name (`pulse-backend` on some older systemd+installs), and adjust the time window to the original incident.++Those Bash recipes read at most 200 records from the last 15 minutes and+require **GNU `timeout`**, with an eight-second deadline and one-second+termination grace. A record limit alone does not bound a hung reader. If+`timeout` is unavailable, stop; do not use an unbounded substitute. They do+not follow logs, make API requests or change logging.++Both output streams are captured because Docker can write application logs+to stderr. An excerpt is shown only after a successful read; failure or+timeout withholds partial output and reports the exit code. A nonzero reader+exit, denied read or missing service/container is a failed read, not+"no TrueNAS errors". Do not filter with `grep truenas`: relevant startup or+storage errors may not contain that word. Even a successful empty read is+inconclusive: check the window and selected instance, not the API key. Do not+enable Debug or repeat the failing action just to collect more logs.Local logs are **not sanitised**. Share only the relevant timestamp, method,HTTP status or error category and a manually redacted error, not the wholediff --git a/frontend-modern/src/features/docs/__tests__/truenasErrorGuide.test.ts b/frontend-modern/src/features/docs/__tests__/truenasErrorGuide.test.tsindex be7938449d..18676be13e 100644--- a/frontend-modern/src/features/docs/__tests__/truenasErrorGuide.test.ts+++ b/frontend-modern/src/features/docs/__tests__/truenasErrorGuide.test.ts@@ -19,8 +19,8 @@ function renderGuide(name: string): HTMLElement {return article;}-function unavailableSection(article: HTMLElement): HTMLElement {- const heading = article.querySelector('#truenas-service-unavailable');+function sectionAt(article: HTMLElement, id: string): HTMLElement {+ const heading = article.querySelector(id);expect(heading).not.toBeNull();const section = document.createElement('section');for (let next = heading!.nextElementSibling; next; next = next.nextElementSibling) {@@ -30,6 +30,9 @@ function unavailableSection(article: HTMLElement): HTMLElement {return section;}+const unavailableSection = (article: HTMLElement): HTMLElement =>+ sectionAt(article, '#truenas-service-unavailable');+const prose = (section: HTMLElement): string =>(section.textContent ?? '').replace(/\s+/g, ' ').trim();@@ -92,4 +95,45 @@ describe('shipped TrueNAS error diagnosis', () => {expect(pollingLink?.textContent).toBe('polling checks');expect(guide.querySelector('#stale-truenas-data')?.textContent).toBe('Stale TrueNAS data');});++ it('takes NAS diagnosis to the shared bounded readers, without an unsafe copied alternative', () => {+ const nas = sectionAt(renderGuide('TRUENAS'), '#no-data-appearing-after-adding-connection');+ const link = nas.querySelector<HTMLAnchorElement>(+ 'a[href="/docs/TROUBLESHOOTING#inspect-notification-logs"]',+ );+ expect(link?.textContent).toBe('bounded Pulse log readers');+ expect(nas.querySelector('pre')).toBeNull();+ const target = sectionAt(renderGuide('TROUBLESHOOTING'), new URL(link!.href).hash);+ const commands = [...target.querySelectorAll('pre code')].map((node) => node.textContent!);+ expect(commands).toHaveLength(2);+ for (const command of commands) {+ expect(command).toContain('--signal=TERM --kill-after=1s 8s');+ expect(command).toContain('no partial excerpt shown');+ expect(command).toContain('no unbounded fallback');+ expect(command).not.toMatch(/grep|--follow|restart|curl/);+ }+ expect(commands[0]).toContain('journalctl -u pulse');+ expect(commands[1]).toContain('docker logs');+ });++ it('keeps NAS-specific host, deadline and disclosure boundaries beside the reader link', () => {+ const text = prose(sectionAt(renderGuide('TRUENAS'), '#no-data-appearing-after-adding-connection'));+ for (const boundary of [+ 'no request ID is required',+ 'inside the Pulse container',+ 'not on the Proxmox host or the TrueNAS appliance',+ 'eight-second deadline and one-second termination grace',+ 'record limit alone does not bound a hung reader',+ 'do not use an unbounded substitute',+ 'only after a successful read',+ 'withholds partial output',+ 'successful empty read is inconclusive',+ 'Do not enable Debug or repeat the failing action',+ 'not sanitised',+ 'not the whole excerpt',+ 'makes live API and guest-agent requests',+ ]) {+ expect(text).toContain(boundary);+ }+ });});diff --git a/scripts/tests/test_truenas_docs.py b/scripts/tests/test_truenas_docs.pyindex 2ed3fbebf9..c486695221 100644--- a/scripts/tests/test_truenas_docs.py+++ b/scripts/tests/test_truenas_docs.py@@ -19,6 +19,8 @@ import tempfileimport threadingimport unittest+from test_troubleshooting_logs import exercise_log_recipe+ROOT = Path(__file__).resolve().parents[2]DOC = ROOT / "docs/TRUENAS.md"@@ -69,8 +71,6 @@ class TrueNASDocsTest(unittest.TestCase):text = DOC.read_text()bash = "\n".join(re.findall(r"```bash\n(.*?)```", text, re.DOTALL))self.assertNotRegex(bash, r"Authorization:|Bearer\s|\$TOKEN|apiKey|--insecure")- self.assertIn("journalctl -u pulse --since '15 minutes ago' --lines 100 --no-pager", bash)- self.assertIn("docker logs --since 15m --tail 100 pulse", bash)self.assertIn("does not establish a live reading", text)self.assertIn("Do not upload a full browser network capture", text)@@ -79,35 +79,27 @@ class TrueNASDocsTest(unittest.TestCase):"### Inventory works but CPU, memory or History is missing", 1)[0]def log_recipes(self):- blocks = re.findall(r"```bash\n(.*?)```", self.log_section(), re.DOTALL)+ # Follow the actual local help link, not an unrelated test fixture or+ # a second copy of the shell commands in this guide.+ links = re.findall(r"\[bounded Pulse log readers\]\(([^)]+)\)", self.log_section())+ self.assertEqual(links, ["TROUBLESHOOTING.md#inspect-notification-logs"])+ self.assertNotIn("```", self.log_section(), "do not leave an unbounded alternate reader")+ target = ROOT / "docs" / links[0].split("#", 1)[0]+ self.assertEqual(target.read_bytes(), (ROOT / "frontend-modern/public/docs" / target.name).read_bytes())+ section = target.read_text().split("### Inspect Notification Logs\n", 1)[1].split("\n### ", 1)[0]+ blocks = re.findall(r"```bash\n(.*?)```", section, re.DOTALL)self.assertEqual(len(blocks), 2, "one bounded reader per deployment")return {"docker" if "# Docker" in block else "journalctl": block for block in blocks}- def exercise_log(self, reader, recipe, *, stdout="", stderr="", exitcode=0):- with tempfile.TemporaryDirectory() as temporary:- directory = Path(temporary)- command = directory / reader- command.write_text(- "#!/usr/bin/env python3\nimport json, os, pathlib, sys\n"- "pathlib.Path(os.environ['LOG_ARGV']).write_text(json.dumps(sys.argv[1:]))\n"- "sys.stdout.write(os.environ['LOG_STDOUT'])\n"- "sys.stderr.write(os.environ['LOG_STDERR'])\n"- "sys.exit(int(os.environ['LOG_EXIT']))\n"- )- command.chmod(0o700)- argv_path = directory / "argv.json"- env = dict(os.environ, PATH=f"{directory}:{os.environ['PATH']}",- LOG_ARGV=str(argv_path), LOG_STDOUT=stdout, LOG_STDERR=stderr,- LOG_EXIT=str(exitcode))- result = subprocess.run(["bash", "-c", recipe], env=env, capture_output=True,- text=True, timeout=5)- self.assertTrue(argv_path.exists(), "copied recipe must invoke the log reader")- return result, json.loads(argv_path.read_text())+ def exercise_log(self, reader, recipe, *, exitcode=0, **settings):+ result, argv = exercise_log_recipe(reader, recipe, exit_code=exitcode, **settings)+ self.assertEqual(argv is None, bool(settings.get("missing_timeout")))+ return result, argvdef test_log_recipes_bound_the_read_and_keep_unfiltered_output(self):expected = {- "journalctl": ["-u", "pulse", "--since", "15 minutes ago", "--lines", "100", "--no-pager"],- "docker": ["logs", "--since", "15m", "--tail", "100", "pulse"],+ "journalctl": ["-u", "pulse", "--since", "15 minutes ago", "--lines", "200", "--no-pager"],+ "docker": ["logs", "--since", "15m", "--tail", "200", "pulse"],}self.assertEqual(set(self.log_recipes()), set(expected))for reader, recipe in self.log_recipes().items():@@ -116,9 +108,10 @@ class TrueNASDocsTest(unittest.TestCase):stderr="TrueNAS collection failed\n")self.assertEqual(argv, expected[reader])self.assertEqual(result.returncode, 0, result.stderr)- self.assertEqual(result.stdout, "startup failed\n")- self.assertEqual(result.stderr, "TrueNAS collection failed\n")- self.assertNotRegex(recipe, r"[|<>]|\b(?:grep|curl|inspect|restart|printenv)\b|--follow")+ self.assertCountEqual(result.stdout.splitlines(), ["startup failed", "TrueNAS collection failed"])+ self.assertEqual(result.stderr, "")+ self.assertNotRegex(recipe, r"\b(?:grep|curl|inspect|restart|printenv)\b|--follow")+ self.assertIn("--signal=TERM --kill-after=1s 8s", recipe)def test_log_reader_failure_is_preserved_even_with_a_matching_partial_line(self):for reader, recipe in self.log_recipes().items():@@ -127,8 +120,33 @@ class TrueNASDocsTest(unittest.TestCase):result, _ = self.exercise_log(reader, recipe, stdout=output,stderr="synthetic access failure\n", exitcode=2)self.assertEqual(result.returncode, 2)- self.assertEqual(result.stdout, output)- self.assertEqual(result.stderr, "synthetic access failure\n")+ self.assertEqual(result.stdout, "")+ self.assertIn("Log read unavailable (exit 2)", result.stderr)+ self.assertNotIn("synthetic access failure", result.stderr)++ def test_hung_readers_withhold_partial_nas_output_at_the_real_deadline(self):+ for reader, recipe in self.log_recipes().items():+ with self.subTest(reader=reader):+ result, _ = self.exercise_log(reader, recipe, stdout="TrueNAS polling\n", hang="term")+ self.assertEqual(result.returncode, 124, result.stderr)+ self.assertEqual(result.stdout, "")+ self.assertIn("Log read unavailable (exit 124)", result.stderr)++ def test_reader_ignoring_term_is_killed_after_the_grace_period(self):+ result, _ = self.exercise_log("docker", self.log_recipes()["docker"],+ stdout="TrueNAS polling\n", hang="ignore-term")+ self.assertEqual(result.returncode, 137, result.stderr)+ self.assertEqual(result.stdout, "")+ self.assertIn("Log read unavailable (exit 137)", result.stderr)++ def test_missing_timeout_stops_before_reading_without_an_unbounded_fallback(self):+ for reader, recipe in self.log_recipes().items():+ with self.subTest(reader=reader):+ result, argv = self.exercise_log(reader, recipe, missing_timeout=True)+ self.assertIsNone(argv)+ self.assertEqual(result.returncode, 1)+ self.assertEqual(result.stdout, "")+ self.assertIn("no unbounded fallback", result.stderr)def test_legacy_pipelines_demonstrate_the_masked_reader_failure(self):legacy = {@@ -151,12 +169,16 @@ class TrueNASDocsTest(unittest.TestCase):self.assertEqual(result.returncode, 0)self.assertEqual(result.stdout + result.stderr, "")result, _ = self.exercise_log(reader, recipe, stderr="synthetic-secret-echo\n")- self.assertIn("synthetic-secret-echo", result.stderr)+ self.assertIn("synthetic-secret-echo", result.stdout)prose = " ".join(self.log_section().split())for boundary in ("inside the Pulse container", "actual service or container name","failed read, not", "successful empty read is inconclusive","not sanitised", "not the whole excerpt", "manually redacted",- "anything echoed in an error", "Do not enable Debug"):+ "anything echoed in an error", "enable Debug",+ "record limit alone does not bound a hung reader", "GNU `timeout`",+ "eight-second deadline and one-second termination grace",+ "no request ID is required", "only after a successful read",+ "withholds partial output", "do not use an unbounded substitute"):self.assertIn(boundary, prose)def test_diagnostics_export_guidance_uses_current_copy_and_preserves_live_check_boundary(self):