Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Make performance sample pairs bounded and identity-safe
pulse-triage[bot]b621549e699a
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.mdindex 6151fbb6c0..f19185c985 100644--- a/docs/TROUBLESHOOTING.md+++ b/docs/TROUBLESHOOTING.md@@ -509,31 +509,79 @@ back to an older version does not make that data a clean older-version baseline.For a responsive **Linux systemd / Proxmox LXC** install, the following readstwo process-I/O samples, waiting 60 seconds between them. Run it inside thePulse container for LXC, not on the Proxmox host. Substitute the actual service-name (`pulse-backend` on some older installs). Use an account authorised to-read the process counters; no service restart or database access is needed.+name (`pulse-backend` on some older installs) in both `systemctl` calls. Use an+account authorised to read the process counters; `sudo -n` must already allow+this read without prompting. Do not change privileges to make it pass; no+service restart or database access is needed.```bash# systemd / Proxmox LXC: bounded process-write samples(- set -e- for sample in 1 2; do- date -u +'%Y-%m-%dT%H:%M:%SZ'- pid=$(systemctl show pulse --property=MainPID --value)- case "$pid" in- ''|0|*[!0-9]*) printf 'No running Pulse PID; sample unavailable.\n' >&2; exit 1 ;;- esac- TZ=UTC ps -p "$pid" -o pid=,lstart=- sudo awk '- $1 == "write_bytes:" || $1 == "cancelled_write_bytes:" { print; fields++ }- END { if (fields != 2) exit 1 }- ' "/proc/$pid/io"- if [ "$sample" -eq 1 ]; then sleep 60; fi- done+ set -eu+ command -v timeout >/dev/null 2>&1 || {+ printf 'Write samples unavailable: GNU timeout is required.\n' >&2; exit 1+ }+ if samples=$(timeout --signal=TERM --kill-after=1s 80s bash <<'PULSE_WRITE_SAMPLES' 2>/dev/null+set -eu+read_sample() {+ set -eu+ pid=$(systemctl show pulse --property=MainPID --value)+ case "$pid" in+ ''|0|*[!0-9]*) exit 1 ;;+ esac+ identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)+ [ -n "$identity" ] || exit 1+ start_ticks=$(awk '+ { sub(/^.*\) /, ""); if ($20 !~ /^[0-9]+$/) exit 1; print $20; found++ }+ END { if (found != 1) exit 1 }+ ' "/proc/$pid/stat")+ timestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')+ counters=$(sudo -n awk '+ $1 == "write_bytes:" || $1 == "cancelled_write_bytes:" {+ if (NF != 2 || $2 !~ /^[0-9]+$/ || seen[$1]++) exit 1+ print; fields+++ }+ END { if (fields != 2) exit 1 }+ ' "/proc/$pid/io")+ current_pid=$(systemctl show pulse --property=MainPID --value)+ current_identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)+ current_ticks=$(awk '+ { sub(/^.*\) /, ""); if ($20 !~ /^[0-9]+$/) exit 1; print $20; found++ }+ END { if (found != 1) exit 1 }+ ' "/proc/$pid/stat")+ [ "$pid" = "$current_pid" ] && [ "$identity" = "$current_identity" ] &&+ [ "$start_ticks" = "$current_ticks" ] || exit 1+ printf 'Identity: %s; start ticks=%s\nSample UTC: %s\n%s\n' \+ "$identity" "$start_ticks" "$timestamp" "$counters"+}+first=$(read_sample)+sleep 60+second=$(read_sample)+[ "${first%%$'\n'*}" = "${second%%$'\n'*}" ] || exit 1+printf '%s\n\n%s\n' "$first" "$second"+PULSE_WRITE_SAMPLES+ ); then+ printf '%s\n' "$samples"+ else+ status=$?+ printf 'Write samples unavailable (exit %s); no complete pair. Stop sampling.\n' "$status" >&2+ exit "$status"+ fi)```-Compare `write_bytes` only when both samples have the same PID and process-start time, no restart occurred, and the counter did not decrease. Divide the+Each recipe requires **GNU `timeout`** and limits the complete two-sample+collection, including the 60-second wait, to 80 seconds plus a one-second+termination grace. It signals only its own collection commands, not Pulse or+Docker. A missing utility, permission failure, timeout or changed identity+makes the pair unavailable; no partial identity, counter or raw reader error is+printed, and there is no unbounded fallback. Stop sampling on failure rather+than removing the deadline or retrying against an unresponsive installation.++The process recipe checks the same PID and process start time before and after+each read and across both samples. Linux start ticks also distinguish PID reuse+within one displayed second. These checks do not make the counters atomic.+Compare `write_bytes` only when no restart occurred and the counter did not decrease. Divide thebyte difference by the **actual elapsed seconds**. This is storage-accountedprocess I/O, not filesystem growth or physical SSD wear; cancelled writes andbackground writeback can differ from device measurements. Keep@@ -543,24 +591,59 @@ day's total. Preserve the window and units with the result.For **Docker / Compose**, run this on the Docker host, replacing `pulse` withthe running container name. It reads only the start time and selected statistics,-not the container environment or configuration.+not the container environment or configuration. Each stats call targets the+full container ID, not a reusable name. The name must still resolve to the same+running ID, start time and restart count after the read and across both samples;+a replacement, restart or stopped container makes the pair unavailable.```bash# Docker: bounded container statistics(- set -e- for sample in 1 2; do- date -u +'%Y-%m-%dT%H:%M:%SZ'- docker inspect --format 'Started={{.State.StartedAt}}' pulse- stats=$(docker stats --no-stream --format \- 'CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}' pulse)- if [ -z "$stats" ]; then- printf 'Container statistics unavailable; no zero inferred.\n' >&2- exit 1- fi- printf '%s\n' "$stats"- if [ "$sample" -eq 1 ]; then sleep 60; fi- done+ set -eu+ command -v timeout >/dev/null 2>&1 || {+ printf 'Container samples unavailable: GNU timeout is required.\n' >&2; exit 1+ }+ if samples=$(timeout --signal=TERM --kill-after=1s 80s bash <<'PULSE_CONTAINER_SAMPLES' 2>/dev/null+set -eu+read_sample() {+ set -eu+ identity=$(docker inspect --type container --format \+ '{{.Id}} {{.State.StartedAt}} {{.State.Running}} {{.RestartCount}}' pulse)+ case "$identity" in *$'\n'*) exit 1 ;; esac+ read -r id started running restarts extra <<< "$identity"+ [ "${#id}" -eq 64 ] && [ "$running" = true ] && [ -z "$extra" ] || exit 1+ case "$id" in *[!0-9a-f]*) exit 1 ;; esac+ case "$started" in 0001-*) exit 1 ;; ????-??-??T??:??:??*Z) ;; *) exit 1 ;; esac+ case "$restarts" in ''|*[!0-9]*) exit 1 ;; esac+ timestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')+ stats=$(docker stats --no-stream --format \+ 'CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}' "$id")+ case "$stats" in+ *$'\n'*) exit 1 ;;+ CPU=*%\ Memory=*\ /\ *\ BlockIO=*\ /\ *) ;;+ *) exit 1 ;;+ esac+ cpu=${stats%% *}+ [[ "${cpu#CPU=}" =~ ^[0-9]+([.][0-9]+)?%$ ]] || exit 1+ case "$stats" in *--*) exit 1 ;; esac+ current_identity=$(docker inspect --type container --format \+ '{{.Id}} {{.State.StartedAt}} {{.State.Running}} {{.RestartCount}}' pulse)+ [ "$identity" = "$current_identity" ] || exit 1+ printf 'Identity: %s\nSample UTC: %s\n%s\n' "$identity" "$timestamp" "$stats"+}+first=$(read_sample)+sleep 60+second=$(read_sample)+[ "${first%%$'\n'*}" = "${second%%$'\n'*}" ] || exit 1+printf '%s\n\n%s\n' "$first" "$second"+PULSE_CONTAINER_SAMPLES+ ); then+ printf '%s\n' "$samples"+ else+ status=$?+ printf 'Container samples unavailable (exit %s); no complete pair. Stop sampling.\n' "$status" >&2+ exit "$status"+ fi)```diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.mdindex 6151fbb6c0..f19185c985 100644--- a/frontend-modern/public/docs/TROUBLESHOOTING.md+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md@@ -509,31 +509,79 @@ back to an older version does not make that data a clean older-version baseline.For a responsive **Linux systemd / Proxmox LXC** install, the following readstwo process-I/O samples, waiting 60 seconds between them. Run it inside thePulse container for LXC, not on the Proxmox host. Substitute the actual service-name (`pulse-backend` on some older installs). Use an account authorised to-read the process counters; no service restart or database access is needed.+name (`pulse-backend` on some older installs) in both `systemctl` calls. Use an+account authorised to read the process counters; `sudo -n` must already allow+this read without prompting. Do not change privileges to make it pass; no+service restart or database access is needed.```bash# systemd / Proxmox LXC: bounded process-write samples(- set -e- for sample in 1 2; do- date -u +'%Y-%m-%dT%H:%M:%SZ'- pid=$(systemctl show pulse --property=MainPID --value)- case "$pid" in- ''|0|*[!0-9]*) printf 'No running Pulse PID; sample unavailable.\n' >&2; exit 1 ;;- esac- TZ=UTC ps -p "$pid" -o pid=,lstart=- sudo awk '- $1 == "write_bytes:" || $1 == "cancelled_write_bytes:" { print; fields++ }- END { if (fields != 2) exit 1 }- ' "/proc/$pid/io"- if [ "$sample" -eq 1 ]; then sleep 60; fi- done+ set -eu+ command -v timeout >/dev/null 2>&1 || {+ printf 'Write samples unavailable: GNU timeout is required.\n' >&2; exit 1+ }+ if samples=$(timeout --signal=TERM --kill-after=1s 80s bash <<'PULSE_WRITE_SAMPLES' 2>/dev/null+set -eu+read_sample() {+ set -eu+ pid=$(systemctl show pulse --property=MainPID --value)+ case "$pid" in+ ''|0|*[!0-9]*) exit 1 ;;+ esac+ identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)+ [ -n "$identity" ] || exit 1+ start_ticks=$(awk '+ { sub(/^.*\) /, ""); if ($20 !~ /^[0-9]+$/) exit 1; print $20; found++ }+ END { if (found != 1) exit 1 }+ ' "/proc/$pid/stat")+ timestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')+ counters=$(sudo -n awk '+ $1 == "write_bytes:" || $1 == "cancelled_write_bytes:" {+ if (NF != 2 || $2 !~ /^[0-9]+$/ || seen[$1]++) exit 1+ print; fields+++ }+ END { if (fields != 2) exit 1 }+ ' "/proc/$pid/io")+ current_pid=$(systemctl show pulse --property=MainPID --value)+ current_identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)+ current_ticks=$(awk '+ { sub(/^.*\) /, ""); if ($20 !~ /^[0-9]+$/) exit 1; print $20; found++ }+ END { if (found != 1) exit 1 }+ ' "/proc/$pid/stat")+ [ "$pid" = "$current_pid" ] && [ "$identity" = "$current_identity" ] &&+ [ "$start_ticks" = "$current_ticks" ] || exit 1+ printf 'Identity: %s; start ticks=%s\nSample UTC: %s\n%s\n' \+ "$identity" "$start_ticks" "$timestamp" "$counters"+}+first=$(read_sample)+sleep 60+second=$(read_sample)+[ "${first%%$'\n'*}" = "${second%%$'\n'*}" ] || exit 1+printf '%s\n\n%s\n' "$first" "$second"+PULSE_WRITE_SAMPLES+ ); then+ printf '%s\n' "$samples"+ else+ status=$?+ printf 'Write samples unavailable (exit %s); no complete pair. Stop sampling.\n' "$status" >&2+ exit "$status"+ fi)```-Compare `write_bytes` only when both samples have the same PID and process-start time, no restart occurred, and the counter did not decrease. Divide the+Each recipe requires **GNU `timeout`** and limits the complete two-sample+collection, including the 60-second wait, to 80 seconds plus a one-second+termination grace. It signals only its own collection commands, not Pulse or+Docker. A missing utility, permission failure, timeout or changed identity+makes the pair unavailable; no partial identity, counter or raw reader error is+printed, and there is no unbounded fallback. Stop sampling on failure rather+than removing the deadline or retrying against an unresponsive installation.++The process recipe checks the same PID and process start time before and after+each read and across both samples. Linux start ticks also distinguish PID reuse+within one displayed second. These checks do not make the counters atomic.+Compare `write_bytes` only when no restart occurred and the counter did not decrease. Divide thebyte difference by the **actual elapsed seconds**. This is storage-accountedprocess I/O, not filesystem growth or physical SSD wear; cancelled writes andbackground writeback can differ from device measurements. Keep@@ -543,24 +591,59 @@ day's total. Preserve the window and units with the result.For **Docker / Compose**, run this on the Docker host, replacing `pulse` withthe running container name. It reads only the start time and selected statistics,-not the container environment or configuration.+not the container environment or configuration. Each stats call targets the+full container ID, not a reusable name. The name must still resolve to the same+running ID, start time and restart count after the read and across both samples;+a replacement, restart or stopped container makes the pair unavailable.```bash# Docker: bounded container statistics(- set -e- for sample in 1 2; do- date -u +'%Y-%m-%dT%H:%M:%SZ'- docker inspect --format 'Started={{.State.StartedAt}}' pulse- stats=$(docker stats --no-stream --format \- 'CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}' pulse)- if [ -z "$stats" ]; then- printf 'Container statistics unavailable; no zero inferred.\n' >&2- exit 1- fi- printf '%s\n' "$stats"- if [ "$sample" -eq 1 ]; then sleep 60; fi- done+ set -eu+ command -v timeout >/dev/null 2>&1 || {+ printf 'Container samples unavailable: GNU timeout is required.\n' >&2; exit 1+ }+ if samples=$(timeout --signal=TERM --kill-after=1s 80s bash <<'PULSE_CONTAINER_SAMPLES' 2>/dev/null+set -eu+read_sample() {+ set -eu+ identity=$(docker inspect --type container --format \+ '{{.Id}} {{.State.StartedAt}} {{.State.Running}} {{.RestartCount}}' pulse)+ case "$identity" in *$'\n'*) exit 1 ;; esac+ read -r id started running restarts extra <<< "$identity"+ [ "${#id}" -eq 64 ] && [ "$running" = true ] && [ -z "$extra" ] || exit 1+ case "$id" in *[!0-9a-f]*) exit 1 ;; esac+ case "$started" in 0001-*) exit 1 ;; ????-??-??T??:??:??*Z) ;; *) exit 1 ;; esac+ case "$restarts" in ''|*[!0-9]*) exit 1 ;; esac+ timestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')+ stats=$(docker stats --no-stream --format \+ 'CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}' "$id")+ case "$stats" in+ *$'\n'*) exit 1 ;;+ CPU=*%\ Memory=*\ /\ *\ BlockIO=*\ /\ *) ;;+ *) exit 1 ;;+ esac+ cpu=${stats%% *}+ [[ "${cpu#CPU=}" =~ ^[0-9]+([.][0-9]+)?%$ ]] || exit 1+ case "$stats" in *--*) exit 1 ;; esac+ current_identity=$(docker inspect --type container --format \+ '{{.Id}} {{.State.StartedAt}} {{.State.Running}} {{.RestartCount}}' pulse)+ [ "$identity" = "$current_identity" ] || exit 1+ printf 'Identity: %s\nSample UTC: %s\n%s\n' "$identity" "$timestamp" "$stats"+}+first=$(read_sample)+sleep 60+second=$(read_sample)+[ "${first%%$'\n'*}" = "${second%%$'\n'*}" ] || exit 1+printf '%s\n\n%s\n' "$first" "$second"+PULSE_CONTAINER_SAMPLES+ ); then+ printf '%s\n' "$samples"+ else+ status=$?+ printf 'Container samples unavailable (exit %s); no complete pair. Stop sampling.\n' "$status" >&2+ exit "$status"+ fi)```diff --git a/scripts/tests/test_performance_troubleshooting_docs.py b/scripts/tests/test_performance_troubleshooting_docs.pyindex 329b5a2c77..897bebbfb0 100644--- a/scripts/tests/test_performance_troubleshooting_docs.py+++ b/scripts/tests/test_performance_troubleshooting_docs.py@@ -1,24 +1,30 @@#!/usr/bin/env python3-"""Exercise the guide's bounded I/O recipes with synthetic OS/container readers.+"""Execute both copied performance recipes with synthetic readers.-The real awk filters run against fixtures. No Pulse process, database, Docker-daemon or host service is inspected. These checks establish recipe behaviour,-not native write rates, CPU relief or an operational repair.+Real awk validates fixtures and real GNU timeout enforces the deadline. Only+this test's shell group can be killed by the test cleanup; no Pulse service,+database, Docker socket, workload or destination is used."""-from __future__ import annotationsimport jsonimport osfrom pathlib import Pathimport re+import shutil+import signalimport subprocessimport tempfile+import timeimport unittest-ROOT = Path(__file__).resolve().parents[2]DOC = ROOT / "docs/TROUBLESHOOTING.md"+CONTAINER_ID = "a" * 64+DOCKER_IDENTITY = f"{CONTAINER_ID} 2026-10-02T20:00:00Z true 0"+PROCESS_IDENTITY = "4321 Sat Oct 3 00:00:00 2026"+DEFAULT_IO = ("rchar: 987654\nwchar: 999999\nwrite_bytes: 1000\ncancelled_write_bytes: 20\n",+ "rchar: 999999\nwchar: 999999\nwrite_bytes: 13000\ncancelled_write_bytes: 30\n")def section() -> str:@@ -30,42 +36,63 @@ def section() -> str:def recipes() -> dict[str, str]:blocks = re.findall(r"```bash\n(.*?)```", section(), re.DOTALL)if len(blocks) != 2:- raise AssertionError("expected two bounded deployment recipes")+ raise AssertionError("expected two deployment recipes")return {"docker" if "# Docker:" in block else "systemd": block for block in blocks}-# All external commands record exact arguments. sudo is a test-only adapter:-# it replaces only the documented /proc/PID/io input and executes real awk.READER = '''#!/usr/bin/env python3-import json, os, pathlib, subprocess, sys+import json, os, pathlib, signal, subprocess, sys, timename = pathlib.Path(sys.argv[0]).nameargs = sys.argv[1:]root = pathlib.Path(os.environ['RECIPE_FIXTURE'])log = root / 'calls.jsonl'previous = [json.loads(line) for line in log.read_text().splitlines()] if log.exists() else []count = sum(call['name'] == name for call in previous)+if name == 'docker':+ count = sum(call['name'] == name and call['args'][0] == args[0] for call in previous)with log.open('a') as stream:stream.write(json.dumps({'name': name, 'args': args}) + '\\n')+if os.environ.get('HANG_TOOL') == name:+ if os.environ.get('IGNORE_TERM'):+ signal.signal(signal.SIGTERM, signal.SIG_IGN)+ time.sleep(100)if name == 'date':+ assert args == ['-u', '+%Y-%m-%dT%H:%M:%SZ']print('2026-10-03T00:00:00Z' if count == 0 else '2026-10-03T00:01:00Z')elif name == 'sleep':assert args == ['60']elif name == 'systemctl':- print(os.environ.get('FIXTURE_PID', '4321'))+ assert args == ['show', 'pulse', '--property=MainPID', '--value']+ print(os.environ.get(f'PID_{count}', os.environ.get('FIXTURE_PID', '4321')))sys.exit(int(os.environ.get('SYSTEMCTL_EXIT', '0')))elif name == 'ps':+ assert args == ['-p', '4321', '-o', 'pid=,lstart=']assert os.environ['TZ'] == 'UTC'- print('4321 Sat Oct 3 00:00:00 2026')+ print(os.environ.get(f'PROCESS_{count}', '4321 Sat Oct 3 00:00:00 2026'))sys.exit(int(os.environ.get('PS_EXIT', '0')))+elif name == 'awk':+ assert args[-1] == '/proc/4321/stat'+ # comm may contain spaces and right parentheses. Starttime is field 22.+ ticks = os.environ.get(f'TICKS_{count}', '10000')+ fixture = root / f'stat-{count}'+ fixture.write_text('4321 (private ) worker) S ' + ' '.join(['0'] * 18 + [ticks]) + '\\n')+ sys.exit(subprocess.run([os.environ['REAL_AWK'], *args[:-1], str(fixture)]).returncode)elif name == 'sudo':- assert args[0] == 'awk'- assert args[-1] == '/proc/4321/io'+ assert args[0] in ('awk', '-n')+ offset = 2 if args[0] == '-n' else 1+ assert args[offset-1] == 'awk' and args[-1] == '/proc/4321/io'target = root / ('missing' if os.environ.get('IO_MISSING') else f'io-{count}')- result = subprocess.run([os.environ['REAL_AWK'], *args[1:-1], str(target)])- sys.exit(result.returncode)+ if os.environ.get('SUDO_EXIT'):+ print('private permission error', file=sys.stderr)+ sys.exit(int(os.environ['SUDO_EXIT']))+ sys.exit(subprocess.run([os.environ['REAL_AWK'], *args[offset:-1], str(target)]).returncode)elif name == 'docker':if args[0] == 'inspect':- print('Started=2026-10-02T20:00:00Z')+ identity = os.environ.get(f'INSPECT_{count}', os.environ['DOCKER_IDENTITY'])+ if args[1:] == ['--format', 'Started={{.State.StartedAt}}', 'pulse']:+ print('Started=' + (identity.split()[1] if identity else ''))+ else:+ print(identity)elif args[0] == 'stats':text = os.environ.get('DOCKER_STATS', 'CPU=12.50% Memory=100MiB / 1GiB BlockIO=1MB / 12MB')if text:@@ -73,136 +100,206 @@ elif name == 'docker':else:raise AssertionError('unsupported docker command')setting = 'DOCKER_INSPECT_EXIT' if args[0] == 'inspect' else 'DOCKER_STATS_EXIT'+ if os.environ.get(setting):+ print('private daemon error', file=sys.stderr)sys.exit(int(os.environ.get(setting, '0')))else:raise AssertionError('unexpected command')'''-class PerformanceTroubleshootingDocsTest(unittest.TestCase):- def exercise(self, deployment: str, *, counters: tuple[str, str] | None = None,- **settings: str):- import shutil+def run_owned(copied: str, env: dict[str, str]):+ with subprocess.Popen(["bash", "-c", copied], env=env, text=True,+ stdout=subprocess.PIPE, stderr=subprocess.PIPE,+ start_new_session=True) as process:+ try:+ stdout, stderr = process.communicate(timeout=83)+ except subprocess.TimeoutExpired:+ # This is an adverse test error, never an empty/passing sample.+ try:+ os.killpg(process.pid, signal.SIGKILL)+ except ProcessLookupError:+ pass+ process.communicate(timeout=5)+ raise+ return subprocess.CompletedProcess(process.args, process.returncode, stdout, stderr)++class PerformanceTroubleshootingDocsTest(unittest.TestCase):+ def exercise(self, deployment: str, *, counters: tuple[str, str] = DEFAULT_IO,+ copied: str | None = None, **settings: str):with tempfile.TemporaryDirectory() as temporary:directory = Path(temporary)- for name in ("date", "sleep", "systemctl", "ps", "sudo", "docker"):+ for name in ("date", "sleep", "systemctl", "ps", "awk", "sudo", "docker"):command = directory / namecommand.write_text(READER, encoding="utf-8")command.chmod(0o700)- counters = counters or (- "rchar: 987654\nwchar: 999999\nwrite_bytes: 1000\ncancelled_write_bytes: 20\n",- "rchar: 999999\nwchar: 999999\nwrite_bytes: 13000\ncancelled_write_bytes: 30\n",- )for index, text in enumerate(counters):(directory / f"io-{index}").write_text(text, encoding="utf-8")env = dict(os.environ, PATH=f"{directory}:{os.environ['PATH']}",- RECIPE_FIXTURE=str(directory), REAL_AWK=shutil.which("awk"), **settings)- result = subprocess.run(["bash", "-c", recipes()[deployment]], env=env,- text=True, capture_output=True, timeout=10)- calls = [json.loads(line) for line in (directory / "calls.jsonl").read_text().splitlines()]+ RECIPE_FIXTURE=str(directory), REAL_AWK=shutil.which("awk"),+ DOCKER_IDENTITY=DOCKER_IDENTITY, **settings)+ result = run_owned(copied if copied is not None else recipes()[deployment], env)+ log = directory / "calls.jsonl"+ calls = [json.loads(line) for line in log.read_text().splitlines()] if log.exists() else []return result, calls- def test_shipped_document_matches_the_exercised_source(self):+ def assert_unavailable(self, result):+ self.assertNotEqual(result.returncode, 0)+ self.assertEqual(result.stdout, "", "an incomplete pair must not be published")+ self.assertIn("unavailable", result.stderr)+ self.assertNotIn("private", result.stderr)++ def test_shipped_source_and_shell_syntax(self):self.assertEqual(DOC.read_bytes(),(ROOT / "frontend-modern/public/docs/TROUBLESHOOTING.md").read_bytes())+ for copied in recipes().values():+ result = subprocess.run(["bash", "-n"], input=copied, text=True, capture_output=True)+ self.assertEqual(result.returncode, 0, result.stderr)+ self.assertIn("timeout --signal=TERM --kill-after=1s 80s bash", copied)- def test_systemd_samples_only_write_counters_and_process_identity(self):+ def test_systemd_samples_only_valid_write_counters_and_stable_identity(self):result, calls = self.exercise("systemd")self.assertEqual(result.returncode, 0, result.stderr)- self.assertEqual([call["name"] for call in calls],- ["date", "systemctl", "ps", "sudo", "sleep",- "date", "systemctl", "ps", "sudo"])- for call in calls:- if call["name"] == "systemctl":- self.assertEqual(call["args"], ["show", "pulse", "--property=MainPID", "--value"])- elif call["name"] == "ps":- self.assertEqual(call["args"], ["-p", "4321", "-o", "pid=,lstart="])- self.assertIn("write_bytes: 1000", result.stdout)- self.assertIn("write_bytes: 13000", result.stdout)- self.assertIn("cancelled_write_bytes: 20", result.stdout)- self.assertNotIn("rchar", result.stdout)- self.assertNotIn("wchar", result.stdout)+ self.assertEqual([c["name"] for c in calls],+ ["systemctl", "ps", "awk", "date", "sudo", "systemctl", "ps", "awk",+ "sleep", "systemctl", "ps", "awk", "date", "sudo", "systemctl", "ps", "awk"])+ self.assertEqual([c["args"][0:2] for c in calls if c["name"] == "sudo"], [["-n", "awk"]] * 2)+ for value in (PROCESS_IDENTITY, "start ticks=10000", "write_bytes: 1000",+ "write_bytes: 13000", "cancelled_write_bytes: 20", "cancelled_write_bytes: 30"):+ self.assertIn(value, result.stdout)+ for forbidden in ("rchar", "wchar", "private ) worker"):+ self.assertNotIn(forbidden, result.stdout)- def test_unavailable_pid_never_reads_a_process_or_sleeps(self):+ def test_invalid_pid_stops_before_process_reads(self):for pid in ("", "0", "not-a-pid", "4321 extra"):with self.subTest(pid=pid):result, calls = self.exercise("systemd", FIXTURE_PID=pid)- self.assertNotEqual(result.returncode, 0)- self.assertIn("sample unavailable", result.stderr)- self.assertEqual([c["name"] for c in calls], ["date", "systemctl"])+ self.assert_unavailable(result)+ self.assertEqual([c["name"] for c in calls], ["systemctl"])- def test_service_and_process_read_failures_stop_the_collection(self):- for settings, expected in (({"SYSTEMCTL_EXIT": "3"}, ["date", "systemctl"]),- ({"PS_EXIT": "4"}, ["date", "systemctl", "ps"])):+ def test_failed_or_empty_identity_does_not_publish_counters(self):+ for settings in ({"SYSTEMCTL_EXIT": "3"}, {"PS_EXIT": "4"}, {"PROCESS_0": ""},+ {"TICKS_0": ""}, {"TICKS_0": "bad"}, {"IO_MISSING": "1"}, {"SUDO_EXIT": "5"}):with self.subTest(settings=settings):result, calls = self.exercise("systemd", **settings)- self.assertNotEqual(result.returncode, 0)- self.assertEqual([c["name"] for c in calls], expected)-- def test_missing_process_io_is_failure_not_zero_writes(self):- result, calls = self.exercise("systemd", IO_MISSING="1")- self.assertNotEqual(result.returncode, 0)- self.assertNotIn("write_bytes:", result.stdout)- self.assertNotIn("sleep", [c["name"] for c in calls])+ self.assert_unavailable(result)+ self.assertNotIn("sleep", [c["name"] for c in calls])- def test_incomplete_process_counters_do_not_become_a_success(self):- for text in ("", "rchar: 999\n", "write_bytes: 1000\n"):+ def test_partial_malformed_or_duplicate_counters_are_withheld(self):+ for text in ("", "rchar: 999\n", "write_bytes: 1000\n", "write_bytes: nope\ncancelled_write_bytes: 2\n",+ "write_bytes: 1000 bytes\ncancelled_write_bytes: 2\n",+ "write_bytes: 1000\nwrite_bytes: 1001\n",+ "write_bytes: 1000\ncancelled_write_bytes: 2\nwrite_bytes: 1001\n"):with self.subTest(text=text):- result, calls = self.exercise("systemd", counters=(text, text))- self.assertNotEqual(result.returncode, 0)- self.assertNotIn("sleep", [c["name"] for c in calls])+ result, _ = self.exercise("systemd", counters=(text, text))+ self.assert_unavailable(result)- def test_decreasing_counters_are_retained_not_turned_into_a_rate(self):- result, _ = self.exercise("systemd", counters=(- "write_bytes: 13000\ncancelled_write_bytes: 30\n",- "write_bytes: 1000\ncancelled_write_bytes: 20\n",- ))+ def test_pid_start_or_same_second_reuse_during_or_between_reads_is_withheld(self):+ for settings in ({"PID_1": "5432"}, {"PID_2": "5432"},+ {"PROCESS_1": "4321 Sat Oct 3 00:00:01 2026"},+ {"PROCESS_2": "4321 Sat Oct 3 00:00:01 2026"},+ {"TICKS_1": "10001"}, {"TICKS_2": "10001", "TICKS_3": "10001"}):+ with self.subTest(settings=settings):+ result, _ = self.exercise("systemd", **settings)+ self.assert_unavailable(result)++ def test_second_sample_failure_withholds_the_completed_first_sample(self):+ result, _ = self.exercise("systemd", counters=(DEFAULT_IO[0], "write_bytes: 99\n"))+ self.assert_unavailable(result)++ def test_decreasing_counters_remain_raw_not_a_rate(self):+ result, _ = self.exercise("systemd", counters=(DEFAULT_IO[1], DEFAULT_IO[0]))self.assertEqual(result.returncode, 0, result.stderr)self.assertIn("write_bytes: 13000", result.stdout)self.assertIn("write_bytes: 1000", result.stdout)self.assertNotRegex(result.stdout, r"GB/day|bytes/s|bytes: -[0-9]+")- def test_docker_reads_only_start_time_and_selected_statistics_twice(self):+ def test_docker_targets_the_full_id_and_checks_its_running_instance(self):result, calls = self.exercise("docker")self.assertEqual(result.returncode, 0, result.stderr)- expected = [- ["inspect", "--format", "Started={{.State.StartedAt}}", "pulse"],- ["stats", "--no-stream", "--format",- "CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}", "pulse"],- ]- self.assertEqual([c["args"] for c in calls if c["name"] == "docker"], expected * 2)- self.assertEqual([c["name"] for c in calls],- ["date", "docker", "docker", "sleep", "date", "docker", "docker"])+ expected = ["inspect", "--type", "container", "--format",+ "{{.Id}} {{.State.StartedAt}} {{.State.Running}} {{.RestartCount}}", "pulse"]+ self.assertEqual([c["args"] for c in calls if c["name"] == "docker" and c["args"][0] == "inspect"],+ [expected] * 4)+ self.assertEqual([c["args"] for c in calls if c["name"] == "docker" and c["args"][0] == "stats"],+ [["stats", "--no-stream", "--format",+ "CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}", CONTAINER_ID]] * 2)+ self.assertEqual(result.stdout.count(DOCKER_IDENTITY), 2)self.assertEqual(result.stdout.count("BlockIO=1MB / 12MB"), 2)- def test_empty_docker_statistics_are_unavailable_not_zero(self):- result, calls = self.exercise("docker", DOCKER_STATS="")- self.assertNotEqual(result.returncode, 0)- self.assertIn("no zero inferred", result.stderr)- self.assertNotIn("sleep", [c["name"] for c in calls])-- def test_docker_reader_failure_stops_before_another_sample(self):- for settings, expected in (- ({"DOCKER_INSPECT_EXIT": "2"}, ["date", "docker"]),- ({"DOCKER_STATS_EXIT": "3"}, ["date", "docker", "docker"]),- ):- with self.subTest(settings=settings):- result, calls = self.exercise("docker", **settings)- self.assertEqual(result.returncode, int(next(iter(settings.values()))))- self.assertEqual([c["name"] for c in calls], expected)+ def test_invalid_stopped_or_unstarted_docker_identity_is_unavailable(self):+ for identity in ("", DOCKER_IDENTITY.replace(CONTAINER_ID, "short"),+ DOCKER_IDENTITY.replace("true", "false"),+ DOCKER_IDENTITY.replace("2026-10-02T20:00:00Z", "0001-01-01T00:00:00Z"),+ DOCKER_IDENTITY.replace("2026-10-02T20:00:00Z", "bad"),+ DOCKER_IDENTITY + " extra", DOCKER_IDENTITY + "\n" + DOCKER_IDENTITY):+ with self.subTest(identity=identity):+ result, calls = self.exercise("docker", INSPECT_0=identity)+ self.assert_unavailable(result)+ self.assertFalse(any(c["name"] == "docker" and c["args"][0] == "stats" for c in calls))++ def test_empty_malformed_or_multiple_docker_stats_are_unavailable(self):+ for text in ("", "--", "CPU=-- Memory=-- / -- BlockIO=-- / --", "CPU=bad% Memory=1 / 2 BlockIO=1 / 2",+ "CPU=1% Memory=1 / 2", "CPU=1% Memory=1 / 2 BlockIO=1 / 2\nprivate extra"):+ with self.subTest(text=text):+ result, _ = self.exercise("docker", DOCKER_STATS=text)+ self.assert_unavailable(result)++ def test_docker_errors_preserve_exit_without_partial_or_raw_error_output(self):+ for setting, value in (("DOCKER_INSPECT_EXIT", "2"), ("DOCKER_STATS_EXIT", "3")):+ with self.subTest(setting=setting):+ result, calls = self.exercise("docker", **{setting: value})+ self.assert_unavailable(result)+ self.assertEqual(result.returncode, int(value))+ self.assertNotIn("sleep", [c["name"] for c in calls])++ def test_docker_replacement_restart_or_stop_during_and_between_reads_is_unavailable(self):+ for identity in (DOCKER_IDENTITY.replace(CONTAINER_ID, "b" * 64),+ DOCKER_IDENTITY.replace("20:00:00", "20:01:00"),+ DOCKER_IDENTITY[:-1] + "1", DOCKER_IDENTITY.replace("true", "false")):+ for index in (1, 2, 3):+ with self.subTest(identity=identity, index=index):+ result, _ = self.exercise("docker", **{f"INSPECT_{index}": identity})+ self.assert_unavailable(result)++ def test_missing_timeout_stops_before_any_reader(self):+ # No timeout in this private PATH. bash is invoked by absolute path.+ with tempfile.TemporaryDirectory() as temporary:+ for deployment, copied in recipes().items():+ with self.subTest(deployment=deployment):+ result = subprocess.run([shutil.which("bash"), "-c", copied],+ env=dict(os.environ, PATH=temporary),+ text=True, capture_output=True, timeout=3)+ self.assert_unavailable(result)+ self.assertIn("GNU timeout is required", result.stderr)++ def test_real_deadline_stops_a_hung_reader_in_both_recipes(self):+ for deployment, name in (("systemd", "systemctl"), ("docker", "docker")):+ with self.subTest(deployment=deployment):+ started = time.monotonic()+ result, _ = self.exercise(deployment, HANG_TOOL=name)+ self.assert_unavailable(result)+ self.assertEqual(result.returncode, 124)+ self.assertLess(time.monotonic() - started, 83)++ def test_real_kill_grace_stops_a_reader_ignoring_term(self):+ started = time.monotonic()+ result, _ = self.exercise("docker", HANG_TOOL="docker", IGNORE_TERM="1")+ self.assert_unavailable(result)+ self.assertEqual(result.returncode, 137)+ self.assertLess(time.monotonic() - started, 83)- def test_guidance_keeps_scope_units_baselines_and_data_safety(self):+ def test_guidance_preserves_scope_units_baselines_and_data_safety(self):guide = " ".join(section().split())- for phrase in ("same PID and process start time", "actual elapsed seconds",- "not a measured day's total", "not bytes per second",- "not a recent sampling window", "different scopes",- "inside the Pulse container", "does not make that data a clean",- "Do not enable Debug", "remove database indexes", "tmpfs",- "Do not post databases", "manually redacted error"):+ for phrase in ("same PID and process start time", "actual elapsed seconds", "not a measured day's total",+ "not bytes per second", "not a recent sampling window", "different scopes",+ "inside the Pulse container", "does not make that data a clean", "Do not enable Debug",+ "remove database indexes", "tmpfs", "Do not post databases", "manually redacted error",+ "no partial identity", "no unbounded fallback", "full container ID", "sudo -n"):self.assertIn(phrase, guide)- for recipe in recipes().values():- self.assertNotRegex(recipe, r"\b(?:restart|kill|rm|truncate|sqlite3|curl|printenv)\b|--follow|--token")+ for copied in recipes().values():+ self.assertNotRegex(copied, r"\b(?:restart|rm|truncate|sqlite3|curl|printenv)\b|--follow|--token|(?:^|[;\n])\s*kill\b")if __name__ == "__main__":