Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Make memory troubleshooting passive and process-scoped

pulse-triage[bot]e70e2c64305e
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md
index a263607230..28802c951b 100644
--- a/docs/TROUBLESHOOTING.md
+++ b/docs/TROUBLESHOOTING.md
@@ -280,6 +280,76 @@ leave the original host unchanged. Follow
[Clone identity recovery](UNIFIED_AGENT.md#duplicate-agents) for the configuration
precedence, systemd example and checks after restart.
+#### Memory use keeps growing
+
+A Proxmox LXC memory chart covers the container, not just Pulse. It can include
+other processes and filesystem cache; Docker's displayed memory also uses its
+own cache accounting. Neither is interchangeable with Pulse's resident memory
+(RSS), a Go heap size, or virtual address space (`VSZ` / `VmSize`). A large
+container reading alone does not establish a Pulse memory leak.
+
+For a responsive **Linux systemd / Proxmox LXC** install, take this small,
+read-only sample at a normally occurring high point. Run it **inside the Pulse
+container** for LXC, not on the Proxmox host. Substitute the actual service name
+(`pulse-backend` on older installs) in both `systemctl` calls. Use an account
+authorised to read these counters; a failed read is unavailable, not zero.
+
+```bash
+# systemd / Proxmox LXC: Pulse resident-memory sample
+(
+ set -eu
+ date -u +'%Y-%m-%dT%H:%M:%SZ'
+ pid=$(systemctl show pulse --property=MainPID --value)
+ case "$pid" in
+ ''|0|*[!0-9]*) printf 'No running Pulse PID; sample unavailable.\n' >&2; exit 1 ;;
+ esac
+ identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)
+ if [ -z "$identity" ]; then
+ printf 'Process identity unavailable.\n' >&2; exit 1
+ fi
+ counters=$(awk '
+ $1 == "VmRSS:" || $1 == "RssAnon:" || $1 == "RssFile:" ||
+ $1 == "RssShmem:" || $1 == "VmSwap:" {
+ if (NF != 3 || $2 !~ /^[0-9]+$/ || $3 != "kB" || seen[$1]++) exit 1
+ print; fields++
+ }
+ END { if (fields != 5) exit 1 }
+ ' "/proc/$pid/status")
+ current_pid=$(systemctl show pulse --property=MainPID --value)
+ current_identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)
+ if [ "$pid" != "$current_pid" ] || [ "$identity" != "$current_identity" ]; then
+ printf 'Pulse changed during collection; discard this sample.\n' >&2; exit 1
+ fi
+ printf 'Pulse process (PID and UTC start): %s\n%s\n' "$identity" "$counters"
+)
+```
+
+Linux labels these values `kB`, meaning 1,024 bytes; divide by 1,024 for MiB.
+`VmRSS` is resident process memory, split into anonymous (`RssAnon`),
+file-backed (`RssFile`) and shared-memory (`RssShmem`) pages. `VmSwap` is swapped
+private anonymous memory, not additional resident memory or all container swap.
+These counters are approximate and not an atomic snapshot; a nonzero
+`RssFile` is not proof of a leak, and `RssAnon` is not specifically the Go heap.
+Missing fields on an older kernel make this recipe fail rather than invent
+values. If it fails or Pulse stops, retain that fact; do not loosen access
+controls or keep retrying against an unresponsive installation.
+
+Keep the sample time, PID/start time, running version, container memory limit,
+fleet size, polling interval and whether dashboards were open. If safe, compare
+another naturally occurring point with the **same PID and process start time**;
+record any restart or upgrade as a different run, not evidence that the cause
+was fixed. For Docker, the [bounded container statistics below](#excessive-cpu-writes-or-database-growth)
+retain the container start time and memory usage but do not measure Pulse RSS;
+do not assume its PID 1 is Pulse.
+
+Do not restart Pulse, drop caches, force garbage collection, change memory limits
+or delete history just to obtain a lower reading. If the container is near its
+limit or the host is unresponsive, stop sampling and prioritise safe recovery.
+Do not post a heap dump, profile, full `/proc` status or process command line:
+they can contain private details. Share only these counters and the relevant
+context after local review; existing screenshots remain useful if collection
+is unsafe.
+
#### Excessive CPU, writes or database growth
First distinguish **Pulse server activity**, agent activity and total host or
diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.md
index a263607230..28802c951b 100644
--- a/frontend-modern/public/docs/TROUBLESHOOTING.md
+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md
@@ -280,6 +280,76 @@ leave the original host unchanged. Follow
[Clone identity recovery](UNIFIED_AGENT.md#duplicate-agents) for the configuration
precedence, systemd example and checks after restart.
+#### Memory use keeps growing
+
+A Proxmox LXC memory chart covers the container, not just Pulse. It can include
+other processes and filesystem cache; Docker's displayed memory also uses its
+own cache accounting. Neither is interchangeable with Pulse's resident memory
+(RSS), a Go heap size, or virtual address space (`VSZ` / `VmSize`). A large
+container reading alone does not establish a Pulse memory leak.
+
+For a responsive **Linux systemd / Proxmox LXC** install, take this small,
+read-only sample at a normally occurring high point. Run it **inside the Pulse
+container** for LXC, not on the Proxmox host. Substitute the actual service name
+(`pulse-backend` on older installs) in both `systemctl` calls. Use an account
+authorised to read these counters; a failed read is unavailable, not zero.
+
+```bash
+# systemd / Proxmox LXC: Pulse resident-memory sample
+(
+ set -eu
+ date -u +'%Y-%m-%dT%H:%M:%SZ'
+ pid=$(systemctl show pulse --property=MainPID --value)
+ case "$pid" in
+ ''|0|*[!0-9]*) printf 'No running Pulse PID; sample unavailable.\n' >&2; exit 1 ;;
+ esac
+ identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)
+ if [ -z "$identity" ]; then
+ printf 'Process identity unavailable.\n' >&2; exit 1
+ fi
+ counters=$(awk '
+ $1 == "VmRSS:" || $1 == "RssAnon:" || $1 == "RssFile:" ||
+ $1 == "RssShmem:" || $1 == "VmSwap:" {
+ if (NF != 3 || $2 !~ /^[0-9]+$/ || $3 != "kB" || seen[$1]++) exit 1
+ print; fields++
+ }
+ END { if (fields != 5) exit 1 }
+ ' "/proc/$pid/status")
+ current_pid=$(systemctl show pulse --property=MainPID --value)
+ current_identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)
+ if [ "$pid" != "$current_pid" ] || [ "$identity" != "$current_identity" ]; then
+ printf 'Pulse changed during collection; discard this sample.\n' >&2; exit 1
+ fi
+ printf 'Pulse process (PID and UTC start): %s\n%s\n' "$identity" "$counters"
+)
+```
+
+Linux labels these values `kB`, meaning 1,024 bytes; divide by 1,024 for MiB.
+`VmRSS` is resident process memory, split into anonymous (`RssAnon`),
+file-backed (`RssFile`) and shared-memory (`RssShmem`) pages. `VmSwap` is swapped
+private anonymous memory, not additional resident memory or all container swap.
+These counters are approximate and not an atomic snapshot; a nonzero
+`RssFile` is not proof of a leak, and `RssAnon` is not specifically the Go heap.
+Missing fields on an older kernel make this recipe fail rather than invent
+values. If it fails or Pulse stops, retain that fact; do not loosen access
+controls or keep retrying against an unresponsive installation.
+
+Keep the sample time, PID/start time, running version, container memory limit,
+fleet size, polling interval and whether dashboards were open. If safe, compare
+another naturally occurring point with the **same PID and process start time**;
+record any restart or upgrade as a different run, not evidence that the cause
+was fixed. For Docker, the [bounded container statistics below](#excessive-cpu-writes-or-database-growth)
+retain the container start time and memory usage but do not measure Pulse RSS;
+do not assume its PID 1 is Pulse.
+
+Do not restart Pulse, drop caches, force garbage collection, change memory limits
+or delete history just to obtain a lower reading. If the container is near its
+limit or the host is unresponsive, stop sampling and prioritise safe recovery.
+Do not post a heap dump, profile, full `/proc` status or process command line:
+they can contain private details. Share only these counters and the relevant
+context after local review; existing screenshots remain useful if collection
+is unsafe.
+
#### Excessive CPU, writes or database growth
First distinguish **Pulse server activity**, agent activity and total host or
diff --git a/scripts/tests/test_memory_troubleshooting_docs.py b/scripts/tests/test_memory_troubleshooting_docs.py
new file mode 100644
index 0000000000..fd9e76c4ae
--- /dev/null
+++ b/scripts/tests/test_memory_troubleshooting_docs.py
@@ -0,0 +1,209 @@
+#!/usr/bin/env python3
+"""Exercise the copied resident-memory recipe, not a Pulse memory repair.
+
+The fixture adapter runs real awk over synthetic status files. A separate
+check reads only a disposable child process's live Linux counters; it does not
+inspect a host service, database, heap or container daemon.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+from pathlib import Path
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+import unittest
+
+
+ROOT = Path(__file__).resolve().parents[2]
+DOC = ROOT / "docs/TROUBLESHOOTING.md"
+COUNTERS = "VmRSS: 8192 kB\nRssAnon: 6144 kB\nRssFile: 1536 kB\nRssShmem: 512 kB\nVmSwap: 1024 kB\n"
+IDENTITY = "4321 Tue Oct 6 00:00:00 2026"
+
+
+def section() -> str:
+ return DOC.read_text(encoding="utf-8").split(
+ "#### Memory use keeps growing\n", 1
+ )[1].split("\n#### ", 1)[0]
+
+
+def recipe() -> str:
+ blocks = re.findall(r"```bash\n(.*?)```", section(), re.DOTALL)
+ if len(blocks) != 1:
+ raise AssertionError("expected one bounded memory recipe")
+ return blocks[0]
+
+
+READER = '''#!/usr/bin/env python3
+import json, os, pathlib, subprocess, sys
+name = pathlib.Path(sys.argv[0]).name
+args = sys.argv[1:]
+root = pathlib.Path(os.environ['RECIPE_FIXTURE'])
+log = root / 'calls.jsonl'
+previous = [json.loads(line) for line in log.read_text().splitlines()] if log.exists() else []
+count = sum(call['name'] == name for call in previous)
+with log.open('a') as stream:
+ stream.write(json.dumps({'name': name, 'args': args}) + '\\n')
+if name == 'date':
+ assert args == ['-u', '+%Y-%m-%dT%H:%M:%SZ']
+ print('2026-10-06T00:00:00Z')
+elif name == 'systemctl':
+ assert args == ['show', 'pulse', '--property=MainPID', '--value']
+ print(os.environ.get(f'PID_{count}', '4321'))
+ sys.exit(int(os.environ.get(f'SERVICE_EXIT_{count}', '0')))
+elif name == 'ps':
+ assert args == ['-p', '4321', '-o', 'pid=,lstart=']
+ assert os.environ['TZ'] == 'UTC'
+ print(os.environ.get(f'IDENTITY_{count}', '4321 Tue Oct 6 00:00:00 2026'))
+ sys.exit(int(os.environ.get(f'PROCESS_EXIT_{count}', '0')))
+elif name == 'awk':
+ assert args[-1] == '/proc/4321/status'
+ target = root / ('missing' if os.environ.get('STATUS_MISSING') else 'status')
+ sys.exit(subprocess.run([os.environ['REAL_AWK'], *args[:-1], str(target)]).returncode)
+else:
+ raise AssertionError('unexpected command')
+'''
+
+
+class MemoryTroubleshootingDocsTest(unittest.TestCase):
+ def exercise(self, *, status: str = COUNTERS, copied: str | None = None,
+ **settings: str):
+ with tempfile.TemporaryDirectory() as temporary:
+ directory = Path(temporary)
+ for name in ("date", "systemctl", "ps", "awk"):
+ command = directory / name
+ command.write_text(READER, encoding="utf-8")
+ command.chmod(0o700)
+ (directory / "status").write_text(status, encoding="utf-8")
+ env = dict(os.environ, PATH=f"{directory}:{os.environ['PATH']}",
+ RECIPE_FIXTURE=str(directory), REAL_AWK=shutil.which("awk"), **settings)
+ result = subprocess.run(["bash", "-c", copied if copied is not None else recipe()],
+ env=env, text=True, capture_output=True, timeout=10)
+ calls = [json.loads(line) for line in (directory / "calls.jsonl").read_text().splitlines()]
+ return result, calls
+
+ def assert_no_memory_claim(self, result):
+ self.assertNotEqual(result.returncode, 0)
+ self.assertNotIn("Pulse process (PID and UTC start)", result.stdout)
+ self.assertNotRegex(result.stdout, r"VmRSS:|RssAnon:|RssFile:|RssShmem:|VmSwap:")
+
+ def test_shipped_guide_is_the_exercised_source_and_shell_is_valid(self):
+ self.assertEqual(DOC.read_bytes(),
+ (ROOT / "frontend-modern/public/docs/TROUBLESHOOTING.md").read_bytes())
+ result = subprocess.run(["bash", "-n"], input=recipe(), text=True, capture_output=True)
+ self.assertEqual(result.returncode, 0, result.stderr)
+
+ def test_only_five_counters_and_stable_process_identity_leave_the_reader(self):
+ result, calls = self.exercise(status="Name: private-workload\nUid: 1001 1001\n"
+ + COUNTERS + "VmSize: 1234567 kB\nThreads: 19\n")
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertIn("2026-10-06T00:00:00Z", result.stdout)
+ self.assertIn(IDENTITY, result.stdout)
+ self.assertIn(COUNTERS, result.stdout)
+ for private in ("private-workload", "Uid:", "VmSize:", "Threads:"):
+ self.assertNotIn(private, result.stdout)
+ self.assertEqual([call["name"] for call in calls],
+ ["date", "systemctl", "ps", "awk", "systemctl", "ps"])
+
+ def test_missing_or_invalid_pid_stops_before_a_process_read(self):
+ for pid in ("", "0", "not-a-pid", "4321 extra"):
+ with self.subTest(pid=pid):
+ result, calls = self.exercise(PID_0=pid)
+ self.assert_no_memory_claim(result)
+ self.assertIn("sample unavailable", result.stderr)
+ self.assertEqual([call["name"] for call in calls], ["date", "systemctl"])
+
+ def test_service_and_process_failures_remain_failures(self):
+ for key, names in (
+ ("SERVICE_EXIT_0", ["date", "systemctl"]),
+ ("PROCESS_EXIT_0", ["date", "systemctl", "ps"]),
+ ("SERVICE_EXIT_1", ["date", "systemctl", "ps", "awk", "systemctl"]),
+ ("PROCESS_EXIT_1", ["date", "systemctl", "ps", "awk", "systemctl", "ps"]),
+ ):
+ with self.subTest(key=key):
+ result, calls = self.exercise(**{key: "3"})
+ self.assert_no_memory_claim(result)
+ self.assertEqual(result.returncode, 3)
+ self.assertEqual([call["name"] for call in calls], names)
+
+ def test_empty_initial_identity_is_not_a_complete_sample(self):
+ result, calls = self.exercise(IDENTITY_0="")
+ self.assert_no_memory_claim(result)
+ self.assertIn("identity unavailable", result.stderr)
+ self.assertNotIn("awk", [call["name"] for call in calls])
+
+ def test_missing_unreadable_or_incomplete_status_does_not_print_partial_counters(self):
+ for status, settings in ((COUNTERS, {"STATUS_MISSING": "1"}),
+ ("", {}), (COUNTERS.replace("RssAnon: 6144 kB\n", ""), {})):
+ with self.subTest(status=status, settings=settings):
+ result, calls = self.exercise(status=status, **settings)
+ self.assert_no_memory_claim(result)
+ self.assertEqual([call["name"] for call in calls],
+ ["date", "systemctl", "ps", "awk"])
+
+ def test_malformed_units_numbers_duplicates_and_extra_fields_fail_closed(self):
+ for value in ("-1 kB", "unknown kB", "1 MB", "1", "1 kB extra"):
+ with self.subTest(value=value):
+ result, _ = self.exercise(status=COUNTERS.replace("8192 kB", value))
+ self.assert_no_memory_claim(result)
+ result, _ = self.exercise(status=COUNTERS + "VmRSS: 8192 kB\n")
+ self.assert_no_memory_claim(result)
+
+ def test_service_restart_or_pid_reuse_discards_the_captured_counters(self):
+ for settings in ({"PID_1": "9999"}, {"PID_1": "0"},
+ {"IDENTITY_1": "4321 Tue Oct 6 00:00:01 2026"},
+ {"IDENTITY_1": ""}):
+ with self.subTest(settings=settings):
+ result, _ = self.exercise(**settings)
+ self.assert_no_memory_claim(result)
+ self.assertIn("discard this sample", result.stderr)
+
+ def test_measured_zero_is_retained_without_interpreting_missing_as_zero(self):
+ result, _ = self.exercise(status=re.sub(r"\d+ kB", "0 kB", COUNTERS))
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertEqual(result.stdout.count("0 kB"), 5)
+
+ @unittest.skipUnless(sys.platform.startswith("linux"), "Linux /proc recipe")
+ def test_real_linux_status_of_an_owned_disposable_process(self):
+ with tempfile.TemporaryDirectory() as temporary:
+ directory = Path(temporary)
+ child = subprocess.Popen([sys.executable, "-c", "import time; time.sleep(30)"])
+ try:
+ command = directory / "systemctl"
+ command.write_text("#!/bin/sh\nprintf '%s\\n' \"$RECIPE_CHILD_PID\"\n", encoding="utf-8")
+ command.chmod(0o700)
+ env = dict(os.environ, PATH=f"{directory}:{os.environ['PATH']}",
+ RECIPE_CHILD_PID=str(child.pid))
+ result = subprocess.run(["bash", "-c", recipe()], env=env,
+ text=True, capture_output=True, timeout=10)
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertIn("Pulse process (PID and UTC start)", result.stdout)
+ for field in ("VmRSS", "RssAnon", "RssFile", "RssShmem", "VmSwap"):
+ self.assertRegex(result.stdout, rf"(?m)^{field}:\s+[0-9]+ kB$")
+ self.assertNotIn("Name:", result.stdout)
+ self.assertNotIn("Uid:", result.stdout)
+ finally:
+ child.terminate()
+ child.wait(timeout=5)
+
+ def test_guidance_distinguishes_scope_units_and_unsafe_shortcuts(self):
+ guide = " ".join(section().split())
+ for phrase in ("container, not just Pulse", "own cache accounting", "Go heap size",
+ "inside the Pulse container", "in both `systemctl` calls",
+ "1,024 bytes", "approximate and not an atomic snapshot",
+ "not specifically the Go heap", "different run",
+ "do not measure Pulse RSS", "do not assume its PID 1 is Pulse",
+ "Do not restart Pulse, drop caches, force garbage collection",
+ "Do not post a heap dump", "existing screenshots remain useful"):
+ self.assertIn(phrase, guide)
+ self.assertNotRegex(recipe(),
+ r"\b(?:sudo|restart|kill|rm|truncate|sqlite3|curl|printenv|cat)\b"
+ r"|--follow|--token|cmdline|environ|smaps")
+
+
+if __name__ == "__main__":
+ unittest.main()