Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Make memory troubleshooting passive and process-scoped
pulse-triage[bot]e70e2c64305e
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.mdindex a263607230..28802c951b 100644--- a/docs/TROUBLESHOOTING.md+++ b/docs/TROUBLESHOOTING.md@@ -280,6 +280,76 @@ leave the original host unchanged. Follow[Clone identity recovery](UNIFIED_AGENT.md#duplicate-agents) for the configurationprecedence, systemd example and checks after restart.+#### Memory use keeps growing++A Proxmox LXC memory chart covers the container, not just Pulse. It can include+other processes and filesystem cache; Docker's displayed memory also uses its+own cache accounting. Neither is interchangeable with Pulse's resident memory+(RSS), a Go heap size, or virtual address space (`VSZ` / `VmSize`). A large+container reading alone does not establish a Pulse memory leak.++For a responsive **Linux systemd / Proxmox LXC** install, take this small,+read-only sample at a normally occurring high point. Run it **inside the Pulse+container** for LXC, not on the Proxmox host. Substitute the actual service name+(`pulse-backend` on older installs) in both `systemctl` calls. Use an account+authorised to read these counters; a failed read is unavailable, not zero.++```bash+# systemd / Proxmox LXC: Pulse resident-memory sample+(+ set -eu+ date -u +'%Y-%m-%dT%H:%M:%SZ'+ pid=$(systemctl show pulse --property=MainPID --value)+ case "$pid" in+ ''|0|*[!0-9]*) printf 'No running Pulse PID; sample unavailable.\n' >&2; exit 1 ;;+ esac+ identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)+ if [ -z "$identity" ]; then+ printf 'Process identity unavailable.\n' >&2; exit 1+ fi+ counters=$(awk '+ $1 == "VmRSS:" || $1 == "RssAnon:" || $1 == "RssFile:" ||+ $1 == "RssShmem:" || $1 == "VmSwap:" {+ if (NF != 3 || $2 !~ /^[0-9]+$/ || $3 != "kB" || seen[$1]++) exit 1+ print; fields+++ }+ END { if (fields != 5) exit 1 }+ ' "/proc/$pid/status")+ current_pid=$(systemctl show pulse --property=MainPID --value)+ current_identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)+ if [ "$pid" != "$current_pid" ] || [ "$identity" != "$current_identity" ]; then+ printf 'Pulse changed during collection; discard this sample.\n' >&2; exit 1+ fi+ printf 'Pulse process (PID and UTC start): %s\n%s\n' "$identity" "$counters"+)+```++Linux labels these values `kB`, meaning 1,024 bytes; divide by 1,024 for MiB.+`VmRSS` is resident process memory, split into anonymous (`RssAnon`),+file-backed (`RssFile`) and shared-memory (`RssShmem`) pages. `VmSwap` is swapped+private anonymous memory, not additional resident memory or all container swap.+These counters are approximate and not an atomic snapshot; a nonzero+`RssFile` is not proof of a leak, and `RssAnon` is not specifically the Go heap.+Missing fields on an older kernel make this recipe fail rather than invent+values. If it fails or Pulse stops, retain that fact; do not loosen access+controls or keep retrying against an unresponsive installation.++Keep the sample time, PID/start time, running version, container memory limit,+fleet size, polling interval and whether dashboards were open. If safe, compare+another naturally occurring point with the **same PID and process start time**;+record any restart or upgrade as a different run, not evidence that the cause+was fixed. For Docker, the [bounded container statistics below](#excessive-cpu-writes-or-database-growth)+retain the container start time and memory usage but do not measure Pulse RSS;+do not assume its PID 1 is Pulse.++Do not restart Pulse, drop caches, force garbage collection, change memory limits+or delete history just to obtain a lower reading. If the container is near its+limit or the host is unresponsive, stop sampling and prioritise safe recovery.+Do not post a heap dump, profile, full `/proc` status or process command line:+they can contain private details. Share only these counters and the relevant+context after local review; existing screenshots remain useful if collection+is unsafe.+#### Excessive CPU, writes or database growthFirst distinguish **Pulse server activity**, agent activity and total host ordiff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.mdindex a263607230..28802c951b 100644--- a/frontend-modern/public/docs/TROUBLESHOOTING.md+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md@@ -280,6 +280,76 @@ leave the original host unchanged. Follow[Clone identity recovery](UNIFIED_AGENT.md#duplicate-agents) for the configurationprecedence, systemd example and checks after restart.+#### Memory use keeps growing++A Proxmox LXC memory chart covers the container, not just Pulse. It can include+other processes and filesystem cache; Docker's displayed memory also uses its+own cache accounting. Neither is interchangeable with Pulse's resident memory+(RSS), a Go heap size, or virtual address space (`VSZ` / `VmSize`). A large+container reading alone does not establish a Pulse memory leak.++For a responsive **Linux systemd / Proxmox LXC** install, take this small,+read-only sample at a normally occurring high point. Run it **inside the Pulse+container** for LXC, not on the Proxmox host. Substitute the actual service name+(`pulse-backend` on older installs) in both `systemctl` calls. Use an account+authorised to read these counters; a failed read is unavailable, not zero.++```bash+# systemd / Proxmox LXC: Pulse resident-memory sample+(+ set -eu+ date -u +'%Y-%m-%dT%H:%M:%SZ'+ pid=$(systemctl show pulse --property=MainPID --value)+ case "$pid" in+ ''|0|*[!0-9]*) printf 'No running Pulse PID; sample unavailable.\n' >&2; exit 1 ;;+ esac+ identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)+ if [ -z "$identity" ]; then+ printf 'Process identity unavailable.\n' >&2; exit 1+ fi+ counters=$(awk '+ $1 == "VmRSS:" || $1 == "RssAnon:" || $1 == "RssFile:" ||+ $1 == "RssShmem:" || $1 == "VmSwap:" {+ if (NF != 3 || $2 !~ /^[0-9]+$/ || $3 != "kB" || seen[$1]++) exit 1+ print; fields+++ }+ END { if (fields != 5) exit 1 }+ ' "/proc/$pid/status")+ current_pid=$(systemctl show pulse --property=MainPID --value)+ current_identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)+ if [ "$pid" != "$current_pid" ] || [ "$identity" != "$current_identity" ]; then+ printf 'Pulse changed during collection; discard this sample.\n' >&2; exit 1+ fi+ printf 'Pulse process (PID and UTC start): %s\n%s\n' "$identity" "$counters"+)+```++Linux labels these values `kB`, meaning 1,024 bytes; divide by 1,024 for MiB.+`VmRSS` is resident process memory, split into anonymous (`RssAnon`),+file-backed (`RssFile`) and shared-memory (`RssShmem`) pages. `VmSwap` is swapped+private anonymous memory, not additional resident memory or all container swap.+These counters are approximate and not an atomic snapshot; a nonzero+`RssFile` is not proof of a leak, and `RssAnon` is not specifically the Go heap.+Missing fields on an older kernel make this recipe fail rather than invent+values. If it fails or Pulse stops, retain that fact; do not loosen access+controls or keep retrying against an unresponsive installation.++Keep the sample time, PID/start time, running version, container memory limit,+fleet size, polling interval and whether dashboards were open. If safe, compare+another naturally occurring point with the **same PID and process start time**;+record any restart or upgrade as a different run, not evidence that the cause+was fixed. For Docker, the [bounded container statistics below](#excessive-cpu-writes-or-database-growth)+retain the container start time and memory usage but do not measure Pulse RSS;+do not assume its PID 1 is Pulse.++Do not restart Pulse, drop caches, force garbage collection, change memory limits+or delete history just to obtain a lower reading. If the container is near its+limit or the host is unresponsive, stop sampling and prioritise safe recovery.+Do not post a heap dump, profile, full `/proc` status or process command line:+they can contain private details. Share only these counters and the relevant+context after local review; existing screenshots remain useful if collection+is unsafe.+#### Excessive CPU, writes or database growthFirst distinguish **Pulse server activity**, agent activity and total host ordiff --git a/scripts/tests/test_memory_troubleshooting_docs.py b/scripts/tests/test_memory_troubleshooting_docs.pynew file mode 100644index 0000000000..fd9e76c4ae--- /dev/null+++ b/scripts/tests/test_memory_troubleshooting_docs.py@@ -0,0 +1,209 @@+#!/usr/bin/env python3+"""Exercise the copied resident-memory recipe, not a Pulse memory repair.++The fixture adapter runs real awk over synthetic status files. A separate+check reads only a disposable child process's live Linux counters; it does not+inspect a host service, database, heap or container daemon.+"""++from __future__ import annotations++import json+import os+from pathlib import Path+import re+import shutil+import subprocess+import sys+import tempfile+import unittest+++ROOT = Path(__file__).resolve().parents[2]+DOC = ROOT / "docs/TROUBLESHOOTING.md"+COUNTERS = "VmRSS: 8192 kB\nRssAnon: 6144 kB\nRssFile: 1536 kB\nRssShmem: 512 kB\nVmSwap: 1024 kB\n"+IDENTITY = "4321 Tue Oct 6 00:00:00 2026"+++def section() -> str:+ return DOC.read_text(encoding="utf-8").split(+ "#### Memory use keeps growing\n", 1+ )[1].split("\n#### ", 1)[0]+++def recipe() -> str:+ blocks = re.findall(r"```bash\n(.*?)```", section(), re.DOTALL)+ if len(blocks) != 1:+ raise AssertionError("expected one bounded memory recipe")+ return blocks[0]+++READER = '''#!/usr/bin/env python3+import json, os, pathlib, subprocess, sys+name = pathlib.Path(sys.argv[0]).name+args = sys.argv[1:]+root = pathlib.Path(os.environ['RECIPE_FIXTURE'])+log = root / 'calls.jsonl'+previous = [json.loads(line) for line in log.read_text().splitlines()] if log.exists() else []+count = sum(call['name'] == name for call in previous)+with log.open('a') as stream:+ stream.write(json.dumps({'name': name, 'args': args}) + '\\n')+if name == 'date':+ assert args == ['-u', '+%Y-%m-%dT%H:%M:%SZ']+ print('2026-10-06T00:00:00Z')+elif name == 'systemctl':+ assert args == ['show', 'pulse', '--property=MainPID', '--value']+ print(os.environ.get(f'PID_{count}', '4321'))+ sys.exit(int(os.environ.get(f'SERVICE_EXIT_{count}', '0')))+elif name == 'ps':+ assert args == ['-p', '4321', '-o', 'pid=,lstart=']+ assert os.environ['TZ'] == 'UTC'+ print(os.environ.get(f'IDENTITY_{count}', '4321 Tue Oct 6 00:00:00 2026'))+ sys.exit(int(os.environ.get(f'PROCESS_EXIT_{count}', '0')))+elif name == 'awk':+ assert args[-1] == '/proc/4321/status'+ target = root / ('missing' if os.environ.get('STATUS_MISSING') else 'status')+ sys.exit(subprocess.run([os.environ['REAL_AWK'], *args[:-1], str(target)]).returncode)+else:+ raise AssertionError('unexpected command')+'''+++class MemoryTroubleshootingDocsTest(unittest.TestCase):+ def exercise(self, *, status: str = COUNTERS, copied: str | None = None,+ **settings: str):+ with tempfile.TemporaryDirectory() as temporary:+ directory = Path(temporary)+ for name in ("date", "systemctl", "ps", "awk"):+ command = directory / name+ command.write_text(READER, encoding="utf-8")+ command.chmod(0o700)+ (directory / "status").write_text(status, encoding="utf-8")+ env = dict(os.environ, PATH=f"{directory}:{os.environ['PATH']}",+ RECIPE_FIXTURE=str(directory), REAL_AWK=shutil.which("awk"), **settings)+ result = subprocess.run(["bash", "-c", copied if copied is not None else recipe()],+ env=env, text=True, capture_output=True, timeout=10)+ calls = [json.loads(line) for line in (directory / "calls.jsonl").read_text().splitlines()]+ return result, calls++ def assert_no_memory_claim(self, result):+ self.assertNotEqual(result.returncode, 0)+ self.assertNotIn("Pulse process (PID and UTC start)", result.stdout)+ self.assertNotRegex(result.stdout, r"VmRSS:|RssAnon:|RssFile:|RssShmem:|VmSwap:")++ def test_shipped_guide_is_the_exercised_source_and_shell_is_valid(self):+ self.assertEqual(DOC.read_bytes(),+ (ROOT / "frontend-modern/public/docs/TROUBLESHOOTING.md").read_bytes())+ result = subprocess.run(["bash", "-n"], input=recipe(), text=True, capture_output=True)+ self.assertEqual(result.returncode, 0, result.stderr)++ def test_only_five_counters_and_stable_process_identity_leave_the_reader(self):+ result, calls = self.exercise(status="Name: private-workload\nUid: 1001 1001\n"+ + COUNTERS + "VmSize: 1234567 kB\nThreads: 19\n")+ self.assertEqual(result.returncode, 0, result.stderr)+ self.assertIn("2026-10-06T00:00:00Z", result.stdout)+ self.assertIn(IDENTITY, result.stdout)+ self.assertIn(COUNTERS, result.stdout)+ for private in ("private-workload", "Uid:", "VmSize:", "Threads:"):+ self.assertNotIn(private, result.stdout)+ self.assertEqual([call["name"] for call in calls],+ ["date", "systemctl", "ps", "awk", "systemctl", "ps"])++ def test_missing_or_invalid_pid_stops_before_a_process_read(self):+ for pid in ("", "0", "not-a-pid", "4321 extra"):+ with self.subTest(pid=pid):+ result, calls = self.exercise(PID_0=pid)+ self.assert_no_memory_claim(result)+ self.assertIn("sample unavailable", result.stderr)+ self.assertEqual([call["name"] for call in calls], ["date", "systemctl"])++ def test_service_and_process_failures_remain_failures(self):+ for key, names in (+ ("SERVICE_EXIT_0", ["date", "systemctl"]),+ ("PROCESS_EXIT_0", ["date", "systemctl", "ps"]),+ ("SERVICE_EXIT_1", ["date", "systemctl", "ps", "awk", "systemctl"]),+ ("PROCESS_EXIT_1", ["date", "systemctl", "ps", "awk", "systemctl", "ps"]),+ ):+ with self.subTest(key=key):+ result, calls = self.exercise(**{key: "3"})+ self.assert_no_memory_claim(result)+ self.assertEqual(result.returncode, 3)+ self.assertEqual([call["name"] for call in calls], names)++ def test_empty_initial_identity_is_not_a_complete_sample(self):+ result, calls = self.exercise(IDENTITY_0="")+ self.assert_no_memory_claim(result)+ self.assertIn("identity unavailable", result.stderr)+ self.assertNotIn("awk", [call["name"] for call in calls])++ def test_missing_unreadable_or_incomplete_status_does_not_print_partial_counters(self):+ for status, settings in ((COUNTERS, {"STATUS_MISSING": "1"}),+ ("", {}), (COUNTERS.replace("RssAnon: 6144 kB\n", ""), {})):+ with self.subTest(status=status, settings=settings):+ result, calls = self.exercise(status=status, **settings)+ self.assert_no_memory_claim(result)+ self.assertEqual([call["name"] for call in calls],+ ["date", "systemctl", "ps", "awk"])++ def test_malformed_units_numbers_duplicates_and_extra_fields_fail_closed(self):+ for value in ("-1 kB", "unknown kB", "1 MB", "1", "1 kB extra"):+ with self.subTest(value=value):+ result, _ = self.exercise(status=COUNTERS.replace("8192 kB", value))+ self.assert_no_memory_claim(result)+ result, _ = self.exercise(status=COUNTERS + "VmRSS: 8192 kB\n")+ self.assert_no_memory_claim(result)++ def test_service_restart_or_pid_reuse_discards_the_captured_counters(self):+ for settings in ({"PID_1": "9999"}, {"PID_1": "0"},+ {"IDENTITY_1": "4321 Tue Oct 6 00:00:01 2026"},+ {"IDENTITY_1": ""}):+ with self.subTest(settings=settings):+ result, _ = self.exercise(**settings)+ self.assert_no_memory_claim(result)+ self.assertIn("discard this sample", result.stderr)++ def test_measured_zero_is_retained_without_interpreting_missing_as_zero(self):+ result, _ = self.exercise(status=re.sub(r"\d+ kB", "0 kB", COUNTERS))+ self.assertEqual(result.returncode, 0, result.stderr)+ self.assertEqual(result.stdout.count("0 kB"), 5)++ @unittest.skipUnless(sys.platform.startswith("linux"), "Linux /proc recipe")+ def test_real_linux_status_of_an_owned_disposable_process(self):+ with tempfile.TemporaryDirectory() as temporary:+ directory = Path(temporary)+ child = subprocess.Popen([sys.executable, "-c", "import time; time.sleep(30)"])+ try:+ command = directory / "systemctl"+ command.write_text("#!/bin/sh\nprintf '%s\\n' \"$RECIPE_CHILD_PID\"\n", encoding="utf-8")+ command.chmod(0o700)+ env = dict(os.environ, PATH=f"{directory}:{os.environ['PATH']}",+ RECIPE_CHILD_PID=str(child.pid))+ result = subprocess.run(["bash", "-c", recipe()], env=env,+ text=True, capture_output=True, timeout=10)+ self.assertEqual(result.returncode, 0, result.stderr)+ self.assertIn("Pulse process (PID and UTC start)", result.stdout)+ for field in ("VmRSS", "RssAnon", "RssFile", "RssShmem", "VmSwap"):+ self.assertRegex(result.stdout, rf"(?m)^{field}:\s+[0-9]+ kB$")+ self.assertNotIn("Name:", result.stdout)+ self.assertNotIn("Uid:", result.stdout)+ finally:+ child.terminate()+ child.wait(timeout=5)++ def test_guidance_distinguishes_scope_units_and_unsafe_shortcuts(self):+ guide = " ".join(section().split())+ for phrase in ("container, not just Pulse", "own cache accounting", "Go heap size",+ "inside the Pulse container", "in both `systemctl` calls",+ "1,024 bytes", "approximate and not an atomic snapshot",+ "not specifically the Go heap", "different run",+ "do not measure Pulse RSS", "do not assume its PID 1 is Pulse",+ "Do not restart Pulse, drop caches, force garbage collection",+ "Do not post a heap dump", "existing screenshots remain useful"):+ self.assertIn(phrase, guide)+ self.assertNotRegex(recipe(),+ r"\b(?:sudo|restart|kill|rm|truncate|sqlite3|curl|printenv|cat)\b"+ r"|--follow|--token|cmdline|environ|smaps")+++if __name__ == "__main__":+ unittest.main()