Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Route TrueNAS diagnostics through bounded fail-closed log readers

pulse-triage[bot]26db2344de47
diff --git a/docs/TRUENAS.md b/docs/TRUENAS.md
index a1b7b1bf67..238a333687 100644
--- a/docs/TRUENAS.md
+++ b/docs/TRUENAS.md
@@ -255,31 +255,30 @@ Distinguish these other results before choosing an action:
[polling checks](#stale-truenas-data) to distinguish a completed failure from
missing or stale observations; a connection test is not a substitute.
-If a local log excerpt is needed, run only the reader for your deployment on
-the machine running **Pulse**, using an account authorised to read its logs.
-For Proxmox LXC, run it **inside the Pulse container**, not on the Proxmox host
-or the TrueNAS appliance. Substitute the actual service or container name
-(`pulse-backend` on some older systemd installs), and adjust the time window to
-the original incident. These commands read at most 100 records from the last
-15 minutes; they do not follow logs, make API requests or change logging:
-
-```bash
-# systemd / Proxmox LXC
-journalctl -u pulse --since '15 minutes ago' --lines 100 --no-pager
-```
-
-```bash
-# Docker
-docker logs --since 15m --tail 100 pulse
-```
-
-Inspect both output streams; Docker can write application logs to stderr.
-Do not pipe the reader into `grep truenas`: it can hide a failed read behind a
-matching partial line and omit relevant startup or storage errors. A nonzero
-reader exit, denied read or missing service/container is a failed read, not
-"no TrueNAS errors". Even a successful empty read is inconclusive: check the
-window and selected instance, not the API key. Do not enable Debug or repeat
-the failing action just to collect more logs.
+If a local log excerpt is needed, use the
+[bounded Pulse log readers](TROUBLESHOOTING.md#inspect-notification-logs).
+Despite that section's notification heading, the unfiltered readers also
+cover TrueNAS polling and startup errors; no request ID is required. Choose
+only the reader for the deployment running **Pulse**, using an account
+authorised to read its logs. For Proxmox LXC, run it **inside the Pulse
+container**, not on the Proxmox host or the TrueNAS appliance. Substitute the
+actual service or container name (`pulse-backend` on some older systemd
+installs), and adjust the time window to the original incident.
+
+Those Bash recipes read at most 200 records from the last 15 minutes and
+require **GNU `timeout`**, with an eight-second deadline and one-second
+termination grace. A record limit alone does not bound a hung reader. If
+`timeout` is unavailable, stop; do not use an unbounded substitute. They do
+not follow logs, make API requests or change logging.
+
+Both output streams are captured because Docker can write application logs
+to stderr. An excerpt is shown only after a successful read; failure or
+timeout withholds partial output and reports the exit code. A nonzero reader
+exit, denied read or missing service/container is a failed read, not
+"no TrueNAS errors". Do not filter with `grep truenas`: relevant startup or
+storage errors may not contain that word. Even a successful empty read is
+inconclusive: check the window and selected instance, not the API key. Do not
+enable Debug or repeat the failing action just to collect more logs.
Local logs are **not sanitised**. Share only the relevant timestamp, method,
HTTP status or error category and a manually redacted error, not the whole
diff --git a/frontend-modern/public/docs/TRUENAS.md b/frontend-modern/public/docs/TRUENAS.md
index a1b7b1bf67..238a333687 100644
--- a/frontend-modern/public/docs/TRUENAS.md
+++ b/frontend-modern/public/docs/TRUENAS.md
@@ -255,31 +255,30 @@ Distinguish these other results before choosing an action:
[polling checks](#stale-truenas-data) to distinguish a completed failure from
missing or stale observations; a connection test is not a substitute.
-If a local log excerpt is needed, run only the reader for your deployment on
-the machine running **Pulse**, using an account authorised to read its logs.
-For Proxmox LXC, run it **inside the Pulse container**, not on the Proxmox host
-or the TrueNAS appliance. Substitute the actual service or container name
-(`pulse-backend` on some older systemd installs), and adjust the time window to
-the original incident. These commands read at most 100 records from the last
-15 minutes; they do not follow logs, make API requests or change logging:
-
-```bash
-# systemd / Proxmox LXC
-journalctl -u pulse --since '15 minutes ago' --lines 100 --no-pager
-```
-
-```bash
-# Docker
-docker logs --since 15m --tail 100 pulse
-```
-
-Inspect both output streams; Docker can write application logs to stderr.
-Do not pipe the reader into `grep truenas`: it can hide a failed read behind a
-matching partial line and omit relevant startup or storage errors. A nonzero
-reader exit, denied read or missing service/container is a failed read, not
-"no TrueNAS errors". Even a successful empty read is inconclusive: check the
-window and selected instance, not the API key. Do not enable Debug or repeat
-the failing action just to collect more logs.
+If a local log excerpt is needed, use the
+[bounded Pulse log readers](TROUBLESHOOTING.md#inspect-notification-logs).
+Despite that section's notification heading, the unfiltered readers also
+cover TrueNAS polling and startup errors; no request ID is required. Choose
+only the reader for the deployment running **Pulse**, using an account
+authorised to read its logs. For Proxmox LXC, run it **inside the Pulse
+container**, not on the Proxmox host or the TrueNAS appliance. Substitute the
+actual service or container name (`pulse-backend` on some older systemd
+installs), and adjust the time window to the original incident.
+
+Those Bash recipes read at most 200 records from the last 15 minutes and
+require **GNU `timeout`**, with an eight-second deadline and one-second
+termination grace. A record limit alone does not bound a hung reader. If
+`timeout` is unavailable, stop; do not use an unbounded substitute. They do
+not follow logs, make API requests or change logging.
+
+Both output streams are captured because Docker can write application logs
+to stderr. An excerpt is shown only after a successful read; failure or
+timeout withholds partial output and reports the exit code. A nonzero reader
+exit, denied read or missing service/container is a failed read, not
+"no TrueNAS errors". Do not filter with `grep truenas`: relevant startup or
+storage errors may not contain that word. Even a successful empty read is
+inconclusive: check the window and selected instance, not the API key. Do not
+enable Debug or repeat the failing action just to collect more logs.
Local logs are **not sanitised**. Share only the relevant timestamp, method,
HTTP status or error category and a manually redacted error, not the whole
diff --git a/frontend-modern/src/features/docs/__tests__/truenasErrorGuide.test.ts b/frontend-modern/src/features/docs/__tests__/truenasErrorGuide.test.ts
index be7938449d..18676be13e 100644
--- a/frontend-modern/src/features/docs/__tests__/truenasErrorGuide.test.ts
+++ b/frontend-modern/src/features/docs/__tests__/truenasErrorGuide.test.ts
@@ -19,8 +19,8 @@ function renderGuide(name: string): HTMLElement {
return article;
}
-function unavailableSection(article: HTMLElement): HTMLElement {
- const heading = article.querySelector('#truenas-service-unavailable');
+function sectionAt(article: HTMLElement, id: string): HTMLElement {
+ const heading = article.querySelector(id);
expect(heading).not.toBeNull();
const section = document.createElement('section');
for (let next = heading!.nextElementSibling; next; next = next.nextElementSibling) {
@@ -30,6 +30,9 @@ function unavailableSection(article: HTMLElement): HTMLElement {
return section;
}
+const unavailableSection = (article: HTMLElement): HTMLElement =>
+ sectionAt(article, '#truenas-service-unavailable');
+
const prose = (section: HTMLElement): string =>
(section.textContent ?? '').replace(/\s+/g, ' ').trim();
@@ -92,4 +95,45 @@ describe('shipped TrueNAS error diagnosis', () => {
expect(pollingLink?.textContent).toBe('polling checks');
expect(guide.querySelector('#stale-truenas-data')?.textContent).toBe('Stale TrueNAS data');
});
+
+ it('takes NAS diagnosis to the shared bounded readers, without an unsafe copied alternative', () => {
+ const nas = sectionAt(renderGuide('TRUENAS'), '#no-data-appearing-after-adding-connection');
+ const link = nas.querySelector<HTMLAnchorElement>(
+ 'a[href="/docs/TROUBLESHOOTING#inspect-notification-logs"]',
+ );
+ expect(link?.textContent).toBe('bounded Pulse log readers');
+ expect(nas.querySelector('pre')).toBeNull();
+ const target = sectionAt(renderGuide('TROUBLESHOOTING'), new URL(link!.href).hash);
+ const commands = [...target.querySelectorAll('pre code')].map((node) => node.textContent!);
+ expect(commands).toHaveLength(2);
+ for (const command of commands) {
+ expect(command).toContain('--signal=TERM --kill-after=1s 8s');
+ expect(command).toContain('no partial excerpt shown');
+ expect(command).toContain('no unbounded fallback');
+ expect(command).not.toMatch(/grep|--follow|restart|curl/);
+ }
+ expect(commands[0]).toContain('journalctl -u pulse');
+ expect(commands[1]).toContain('docker logs');
+ });
+
+ it('keeps NAS-specific host, deadline and disclosure boundaries beside the reader link', () => {
+ const text = prose(sectionAt(renderGuide('TRUENAS'), '#no-data-appearing-after-adding-connection'));
+ for (const boundary of [
+ 'no request ID is required',
+ 'inside the Pulse container',
+ 'not on the Proxmox host or the TrueNAS appliance',
+ 'eight-second deadline and one-second termination grace',
+ 'record limit alone does not bound a hung reader',
+ 'do not use an unbounded substitute',
+ 'only after a successful read',
+ 'withholds partial output',
+ 'successful empty read is inconclusive',
+ 'Do not enable Debug or repeat the failing action',
+ 'not sanitised',
+ 'not the whole excerpt',
+ 'makes live API and guest-agent requests',
+ ]) {
+ expect(text).toContain(boundary);
+ }
+ });
});
diff --git a/scripts/tests/test_truenas_docs.py b/scripts/tests/test_truenas_docs.py
index 2ed3fbebf9..c486695221 100644
--- a/scripts/tests/test_truenas_docs.py
+++ b/scripts/tests/test_truenas_docs.py
@@ -19,6 +19,8 @@ import tempfile
import threading
import unittest
+from test_troubleshooting_logs import exercise_log_recipe
+
ROOT = Path(__file__).resolve().parents[2]
DOC = ROOT / "docs/TRUENAS.md"
@@ -69,8 +71,6 @@ class TrueNASDocsTest(unittest.TestCase):
text = DOC.read_text()
bash = "\n".join(re.findall(r"```bash\n(.*?)```", text, re.DOTALL))
self.assertNotRegex(bash, r"Authorization:|Bearer\s|\$TOKEN|apiKey|--insecure")
- self.assertIn("journalctl -u pulse --since '15 minutes ago' --lines 100 --no-pager", bash)
- self.assertIn("docker logs --since 15m --tail 100 pulse", bash)
self.assertIn("does not establish a live reading", text)
self.assertIn("Do not upload a full browser network capture", text)
@@ -79,35 +79,27 @@ class TrueNASDocsTest(unittest.TestCase):
"### Inventory works but CPU, memory or History is missing", 1)[0]
def log_recipes(self):
- blocks = re.findall(r"```bash\n(.*?)```", self.log_section(), re.DOTALL)
+ # Follow the actual local help link, not an unrelated test fixture or
+ # a second copy of the shell commands in this guide.
+ links = re.findall(r"\[bounded Pulse log readers\]\(([^)]+)\)", self.log_section())
+ self.assertEqual(links, ["TROUBLESHOOTING.md#inspect-notification-logs"])
+ self.assertNotIn("```", self.log_section(), "do not leave an unbounded alternate reader")
+ target = ROOT / "docs" / links[0].split("#", 1)[0]
+ self.assertEqual(target.read_bytes(), (ROOT / "frontend-modern/public/docs" / target.name).read_bytes())
+ section = target.read_text().split("### Inspect Notification Logs\n", 1)[1].split("\n### ", 1)[0]
+ blocks = re.findall(r"```bash\n(.*?)```", section, re.DOTALL)
self.assertEqual(len(blocks), 2, "one bounded reader per deployment")
return {"docker" if "# Docker" in block else "journalctl": block for block in blocks}
- def exercise_log(self, reader, recipe, *, stdout="", stderr="", exitcode=0):
- with tempfile.TemporaryDirectory() as temporary:
- directory = Path(temporary)
- command = directory / reader
- command.write_text(
- "#!/usr/bin/env python3\nimport json, os, pathlib, sys\n"
- "pathlib.Path(os.environ['LOG_ARGV']).write_text(json.dumps(sys.argv[1:]))\n"
- "sys.stdout.write(os.environ['LOG_STDOUT'])\n"
- "sys.stderr.write(os.environ['LOG_STDERR'])\n"
- "sys.exit(int(os.environ['LOG_EXIT']))\n"
- )
- command.chmod(0o700)
- argv_path = directory / "argv.json"
- env = dict(os.environ, PATH=f"{directory}:{os.environ['PATH']}",
- LOG_ARGV=str(argv_path), LOG_STDOUT=stdout, LOG_STDERR=stderr,
- LOG_EXIT=str(exitcode))
- result = subprocess.run(["bash", "-c", recipe], env=env, capture_output=True,
- text=True, timeout=5)
- self.assertTrue(argv_path.exists(), "copied recipe must invoke the log reader")
- return result, json.loads(argv_path.read_text())
+ def exercise_log(self, reader, recipe, *, exitcode=0, **settings):
+ result, argv = exercise_log_recipe(reader, recipe, exit_code=exitcode, **settings)
+ self.assertEqual(argv is None, bool(settings.get("missing_timeout")))
+ return result, argv
def test_log_recipes_bound_the_read_and_keep_unfiltered_output(self):
expected = {
- "journalctl": ["-u", "pulse", "--since", "15 minutes ago", "--lines", "100", "--no-pager"],
- "docker": ["logs", "--since", "15m", "--tail", "100", "pulse"],
+ "journalctl": ["-u", "pulse", "--since", "15 minutes ago", "--lines", "200", "--no-pager"],
+ "docker": ["logs", "--since", "15m", "--tail", "200", "pulse"],
}
self.assertEqual(set(self.log_recipes()), set(expected))
for reader, recipe in self.log_recipes().items():
@@ -116,9 +108,10 @@ class TrueNASDocsTest(unittest.TestCase):
stderr="TrueNAS collection failed\n")
self.assertEqual(argv, expected[reader])
self.assertEqual(result.returncode, 0, result.stderr)
- self.assertEqual(result.stdout, "startup failed\n")
- self.assertEqual(result.stderr, "TrueNAS collection failed\n")
- self.assertNotRegex(recipe, r"[|<>]|\b(?:grep|curl|inspect|restart|printenv)\b|--follow")
+ self.assertCountEqual(result.stdout.splitlines(), ["startup failed", "TrueNAS collection failed"])
+ self.assertEqual(result.stderr, "")
+ self.assertNotRegex(recipe, r"\b(?:grep|curl|inspect|restart|printenv)\b|--follow")
+ self.assertIn("--signal=TERM --kill-after=1s 8s", recipe)
def test_log_reader_failure_is_preserved_even_with_a_matching_partial_line(self):
for reader, recipe in self.log_recipes().items():
@@ -127,8 +120,33 @@ class TrueNASDocsTest(unittest.TestCase):
result, _ = self.exercise_log(reader, recipe, stdout=output,
stderr="synthetic access failure\n", exitcode=2)
self.assertEqual(result.returncode, 2)
- self.assertEqual(result.stdout, output)
- self.assertEqual(result.stderr, "synthetic access failure\n")
+ self.assertEqual(result.stdout, "")
+ self.assertIn("Log read unavailable (exit 2)", result.stderr)
+ self.assertNotIn("synthetic access failure", result.stderr)
+
+ def test_hung_readers_withhold_partial_nas_output_at_the_real_deadline(self):
+ for reader, recipe in self.log_recipes().items():
+ with self.subTest(reader=reader):
+ result, _ = self.exercise_log(reader, recipe, stdout="TrueNAS polling\n", hang="term")
+ self.assertEqual(result.returncode, 124, result.stderr)
+ self.assertEqual(result.stdout, "")
+ self.assertIn("Log read unavailable (exit 124)", result.stderr)
+
+ def test_reader_ignoring_term_is_killed_after_the_grace_period(self):
+ result, _ = self.exercise_log("docker", self.log_recipes()["docker"],
+ stdout="TrueNAS polling\n", hang="ignore-term")
+ self.assertEqual(result.returncode, 137, result.stderr)
+ self.assertEqual(result.stdout, "")
+ self.assertIn("Log read unavailable (exit 137)", result.stderr)
+
+ def test_missing_timeout_stops_before_reading_without_an_unbounded_fallback(self):
+ for reader, recipe in self.log_recipes().items():
+ with self.subTest(reader=reader):
+ result, argv = self.exercise_log(reader, recipe, missing_timeout=True)
+ self.assertIsNone(argv)
+ self.assertEqual(result.returncode, 1)
+ self.assertEqual(result.stdout, "")
+ self.assertIn("no unbounded fallback", result.stderr)
def test_legacy_pipelines_demonstrate_the_masked_reader_failure(self):
legacy = {
@@ -151,12 +169,16 @@ class TrueNASDocsTest(unittest.TestCase):
self.assertEqual(result.returncode, 0)
self.assertEqual(result.stdout + result.stderr, "")
result, _ = self.exercise_log(reader, recipe, stderr="synthetic-secret-echo\n")
- self.assertIn("synthetic-secret-echo", result.stderr)
+ self.assertIn("synthetic-secret-echo", result.stdout)
prose = " ".join(self.log_section().split())
for boundary in ("inside the Pulse container", "actual service or container name",
"failed read, not", "successful empty read is inconclusive",
"not sanitised", "not the whole excerpt", "manually redacted",
- "anything echoed in an error", "Do not enable Debug"):
+ "anything echoed in an error", "enable Debug",
+ "record limit alone does not bound a hung reader", "GNU `timeout`",
+ "eight-second deadline and one-second termination grace",
+ "no request ID is required", "only after a successful read",
+ "withholds partial output", "do not use an unbounded substitute"):
self.assertIn(boundary, prose)
def test_diagnostics_export_guidance_uses_current_copy_and_preserves_live_check_boundary(self):