Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep TrueNAS diagnostic reads bounded and truthful
pulse-triage[bot]a6c40e693b35
diff --git a/docs/TRUENAS.md b/docs/TRUENAS.mdindex 8578f16dd9..095acb8bfb 100644--- a/docs/TRUENAS.md+++ b/docs/TRUENAS.md@@ -231,16 +231,51 @@ HTTP status and a manually redacted error, never the full response or headers.the test succeeds. A successful test also does not establish live CPU, memoryor History readings. The legacy REST diagnostic is expected for recognizedCORE 13 systems; it is not itself a connection error.-- Inspect a bounded local log excerpt for TrueNAS-related errors:- ```bash- journalctl -u pulse -n 100 --no-pager | grep -i truenas- # or- docker logs --tail 100 pulse 2>&1 | grep -i truenas- ```-- If collection is safe, **Settings → Diagnostics → Export for GitHub- (sanitized)** can provide connection evidence. Review the export or log- excerpt before sharing: remove credentials, cookies, private hostnames,- addresses and webhook URLs. Do not upload configuration or credential files.+- Preserve the existing error before testing or restarting. Follow the+ [polling checks](#stale-truenas-data) to distinguish a completed failure from+ missing or stale observations; a connection test is not a substitute.++If a local log excerpt is needed, run only the reader for your deployment on+the machine running **Pulse**, using an account authorised to read its logs.+For Proxmox LXC, run it **inside the Pulse container**, not on the Proxmox host+or the TrueNAS appliance. Substitute the actual service or container name+(`pulse-backend` on some older systemd installs), and adjust the time window to+the original incident. These commands read at most 100 records from the last+15 minutes; they do not follow logs, make API requests or change logging:++```bash+# systemd / Proxmox LXC+journalctl -u pulse --since '15 minutes ago' --lines 100 --no-pager+```++```bash+# Docker+docker logs --since 15m --tail 100 pulse+```++Inspect both output streams; Docker can write application logs to stderr.+Do not pipe the reader into `grep truenas`: it can hide a failed read behind a+matching partial line and omit relevant startup or storage errors. A nonzero+reader exit, denied read or missing service/container is a failed read, not+"no TrueNAS errors". Even a successful empty read is inconclusive: check the+window and selected instance, not the API key. Do not enable Debug or repeat+the failing action just to collect more logs.++Local logs are **not sanitised**. Share only the relevant timestamp, method,+HTTP status or error category and a manually redacted error, not the whole+excerpt. Remove credentials, cookies, secret URLs, private hostnames,+addresses and personal information, including anything echoed in an error.+Do not upload configuration or credential files.++In **Settings → Diagnostics**, an existing result's download buttons reuse it+without running the checks again; nothing is uploaded. Use **GitHub (review first)**+for an export intended for sharing, but sanitisation does not guarantee that+every error is safe to share. Open the downloaded file locally and review it+before sharing; keep **Full (private)** private. **Run Diagnostics** makes live+API and guest-agent requests:+do not run it during a backup, freeze/thaw or an unresponsive-host incident+merely to obtain an export. Keep the existing observations instead. See+[safe diagnostics collection](TROUBLESHOOTING.md#collect-diagnostics-safely).### Inventory works but CPU, memory or History is missingdiff --git a/frontend-modern/public/docs/TRUENAS.md b/frontend-modern/public/docs/TRUENAS.mdindex 8578f16dd9..095acb8bfb 100644--- a/frontend-modern/public/docs/TRUENAS.md+++ b/frontend-modern/public/docs/TRUENAS.md@@ -231,16 +231,51 @@ HTTP status and a manually redacted error, never the full response or headers.the test succeeds. A successful test also does not establish live CPU, memoryor History readings. The legacy REST diagnostic is expected for recognizedCORE 13 systems; it is not itself a connection error.-- Inspect a bounded local log excerpt for TrueNAS-related errors:- ```bash- journalctl -u pulse -n 100 --no-pager | grep -i truenas- # or- docker logs --tail 100 pulse 2>&1 | grep -i truenas- ```-- If collection is safe, **Settings → Diagnostics → Export for GitHub- (sanitized)** can provide connection evidence. Review the export or log- excerpt before sharing: remove credentials, cookies, private hostnames,- addresses and webhook URLs. Do not upload configuration or credential files.+- Preserve the existing error before testing or restarting. Follow the+ [polling checks](#stale-truenas-data) to distinguish a completed failure from+ missing or stale observations; a connection test is not a substitute.++If a local log excerpt is needed, run only the reader for your deployment on+the machine running **Pulse**, using an account authorised to read its logs.+For Proxmox LXC, run it **inside the Pulse container**, not on the Proxmox host+or the TrueNAS appliance. Substitute the actual service or container name+(`pulse-backend` on some older systemd installs), and adjust the time window to+the original incident. These commands read at most 100 records from the last+15 minutes; they do not follow logs, make API requests or change logging:++```bash+# systemd / Proxmox LXC+journalctl -u pulse --since '15 minutes ago' --lines 100 --no-pager+```++```bash+# Docker+docker logs --since 15m --tail 100 pulse+```++Inspect both output streams; Docker can write application logs to stderr.+Do not pipe the reader into `grep truenas`: it can hide a failed read behind a+matching partial line and omit relevant startup or storage errors. A nonzero+reader exit, denied read or missing service/container is a failed read, not+"no TrueNAS errors". Even a successful empty read is inconclusive: check the+window and selected instance, not the API key. Do not enable Debug or repeat+the failing action just to collect more logs.++Local logs are **not sanitised**. Share only the relevant timestamp, method,+HTTP status or error category and a manually redacted error, not the whole+excerpt. Remove credentials, cookies, secret URLs, private hostnames,+addresses and personal information, including anything echoed in an error.+Do not upload configuration or credential files.++In **Settings → Diagnostics**, an existing result's download buttons reuse it+without running the checks again; nothing is uploaded. Use **GitHub (review first)**+for an export intended for sharing, but sanitisation does not guarantee that+every error is safe to share. Open the downloaded file locally and review it+before sharing; keep **Full (private)** private. **Run Diagnostics** makes live+API and guest-agent requests:+do not run it during a backup, freeze/thaw or an unresponsive-host incident+merely to obtain an export. Keep the existing observations instead. See+[safe diagnostics collection](TROUBLESHOOTING.md#collect-diagnostics-safely).### Inventory works but CPU, memory or History is missingdiff --git a/scripts/tests/test_truenas_docs.py b/scripts/tests/test_truenas_docs.pyindex 330f60806f..2ed3fbebf9 100644--- a/scripts/tests/test_truenas_docs.py+++ b/scripts/tests/test_truenas_docs.py@@ -1,7 +1,9 @@#!/usr/bin/env python3-"""Exercise TrueNAS connection recipes without putting credentials in argv.+"""Exercise TrueNAS connection and diagnostic recipes safely.All credentials are synthetic. HTTP checks require the offline source-proof VM.+Log-reader controls establish bounds and exit handling, not native NAS recovery+or automatic redaction."""from contextlib import contextmanager@@ -67,11 +69,110 @@ class TrueNASDocsTest(unittest.TestCase):text = DOC.read_text()bash = "\n".join(re.findall(r"```bash\n(.*?)```", text, re.DOTALL))self.assertNotRegex(bash, r"Authorization:|Bearer\s|\$TOKEN|apiKey|--insecure")- self.assertIn("journalctl -u pulse -n 100 --no-pager", bash)- self.assertIn("docker logs --tail 100 pulse 2>&1", bash)+ self.assertIn("journalctl -u pulse --since '15 minutes ago' --lines 100 --no-pager", bash)+ self.assertIn("docker logs --since 15m --tail 100 pulse", bash)self.assertIn("does not establish a live reading", text)self.assertIn("Do not upload a full browser network capture", text)+ def log_section(self):+ return DOC.read_text().split("### No data appearing after adding connection\n", 1)[1].split(+ "### Inventory works but CPU, memory or History is missing", 1)[0]++ def log_recipes(self):+ blocks = re.findall(r"```bash\n(.*?)```", self.log_section(), re.DOTALL)+ self.assertEqual(len(blocks), 2, "one bounded reader per deployment")+ return {"docker" if "# Docker" in block else "journalctl": block for block in blocks}++ def exercise_log(self, reader, recipe, *, stdout="", stderr="", exitcode=0):+ with tempfile.TemporaryDirectory() as temporary:+ directory = Path(temporary)+ command = directory / reader+ command.write_text(+ "#!/usr/bin/env python3\nimport json, os, pathlib, sys\n"+ "pathlib.Path(os.environ['LOG_ARGV']).write_text(json.dumps(sys.argv[1:]))\n"+ "sys.stdout.write(os.environ['LOG_STDOUT'])\n"+ "sys.stderr.write(os.environ['LOG_STDERR'])\n"+ "sys.exit(int(os.environ['LOG_EXIT']))\n"+ )+ command.chmod(0o700)+ argv_path = directory / "argv.json"+ env = dict(os.environ, PATH=f"{directory}:{os.environ['PATH']}",+ LOG_ARGV=str(argv_path), LOG_STDOUT=stdout, LOG_STDERR=stderr,+ LOG_EXIT=str(exitcode))+ result = subprocess.run(["bash", "-c", recipe], env=env, capture_output=True,+ text=True, timeout=5)+ self.assertTrue(argv_path.exists(), "copied recipe must invoke the log reader")+ return result, json.loads(argv_path.read_text())++ def test_log_recipes_bound_the_read_and_keep_unfiltered_output(self):+ expected = {+ "journalctl": ["-u", "pulse", "--since", "15 minutes ago", "--lines", "100", "--no-pager"],+ "docker": ["logs", "--since", "15m", "--tail", "100", "pulse"],+ }+ self.assertEqual(set(self.log_recipes()), set(expected))+ for reader, recipe in self.log_recipes().items():+ with self.subTest(reader=reader):+ result, argv = self.exercise_log(reader, recipe, stdout="startup failed\n",+ stderr="TrueNAS collection failed\n")+ self.assertEqual(argv, expected[reader])+ self.assertEqual(result.returncode, 0, result.stderr)+ self.assertEqual(result.stdout, "startup failed\n")+ self.assertEqual(result.stderr, "TrueNAS collection failed\n")+ self.assertNotRegex(recipe, r"[|<>]|\b(?:grep|curl|inspect|restart|printenv)\b|--follow")++ def test_log_reader_failure_is_preserved_even_with_a_matching_partial_line(self):+ for reader, recipe in self.log_recipes().items():+ for output in ("", "TrueNAS polling\n"):+ with self.subTest(reader=reader, output=output):+ result, _ = self.exercise_log(reader, recipe, stdout=output,+ stderr="synthetic access failure\n", exitcode=2)+ self.assertEqual(result.returncode, 2)+ self.assertEqual(result.stdout, output)+ self.assertEqual(result.stderr, "synthetic access failure\n")++ def test_legacy_pipelines_demonstrate_the_masked_reader_failure(self):+ legacy = {+ "journalctl": "journalctl -u pulse -n 100 --no-pager | grep -i truenas",+ "docker": "docker logs --tail 100 pulse 2>&1 | grep -i truenas",+ }+ for reader, recipe in legacy.items():+ with self.subTest(reader=reader):+ result, _ = self.exercise_log(reader, recipe, stdout="TrueNAS polling\n",+ stderr="synthetic access failure\n", exitcode=2)+ self.assertEqual(result.returncode, 0, "old recipe must expose the masked failure")+ self.assertIn("TrueNAS polling", result.stdout)+ result, _ = self.exercise_log("docker", legacy["docker"], stdout="storage init failed\n")+ self.assertEqual(result.returncode, 1)+ self.assertEqual(result.stdout, "", "old filter discards related non-TrueNAS errors")++ def test_empty_or_sensitive_logs_are_not_claimed_to_be_safe_or_healthy(self):+ for reader, recipe in self.log_recipes().items():+ result, _ = self.exercise_log(reader, recipe)+ self.assertEqual(result.returncode, 0)+ self.assertEqual(result.stdout + result.stderr, "")+ result, _ = self.exercise_log(reader, recipe, stderr="synthetic-secret-echo\n")+ self.assertIn("synthetic-secret-echo", result.stderr)+ prose = " ".join(self.log_section().split())+ for boundary in ("inside the Pulse container", "actual service or container name",+ "failed read, not", "successful empty read is inconclusive",+ "not sanitised", "not the whole excerpt", "manually redacted",+ "anything echoed in an error", "Do not enable Debug"):+ self.assertIn(boundary, prose)++ def test_diagnostics_export_guidance_uses_current_copy_and_preserves_live_check_boundary(self):+ text = " ".join(self.log_section().split())+ copy = (ROOT / "frontend-modern/src/utils/diagnosticsPresentation.ts").read_text()+ for field in ("exportFullLabel", "exportGithubLabel", "runActionLabel"):+ label = re.search(rf"{field}: '([^']+)'", copy).group(1)+ self.assertIn(label, text)+ for boundary in ("without running the checks again", "nothing is uploaded",+ "review it before sharing", "keep **Full (private)** private",+ "makes live API and guest-agent requests", "during a backup, freeze/thaw",+ "unresponsive-host incident", "Keep the existing observations",+ "TROUBLESHOOTING.md#collect-diagnostics-safely"):+ self.assertIn(boundary, text)+ self.assertNotIn("Export for GitHub (sanitized)", text)+def test_core_graph_guidance_distinguishes_request_from_reply(self):text = DOC.read_text().split("### Inventory works but CPU, memory or History is missing", 1)[1]text = text.split("### Stale TrueNAS data", 1)[0]