Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Make performance sample pairs bounded and identity-safe

pulse-triage[bot]b621549e699a
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md
index 6151fbb6c0..f19185c985 100644
--- a/docs/TROUBLESHOOTING.md
+++ b/docs/TROUBLESHOOTING.md
@@ -509,31 +509,79 @@ back to an older version does not make that data a clean older-version baseline.
For a responsive **Linux systemd / Proxmox LXC** install, the following reads
two process-I/O samples, waiting 60 seconds between them. Run it inside the
Pulse container for LXC, not on the Proxmox host. Substitute the actual service
-name (`pulse-backend` on some older installs). Use an account authorised to
-read the process counters; no service restart or database access is needed.
+name (`pulse-backend` on some older installs) in both `systemctl` calls. Use an
+account authorised to read the process counters; `sudo -n` must already allow
+this read without prompting. Do not change privileges to make it pass; no
+service restart or database access is needed.
```bash
# systemd / Proxmox LXC: bounded process-write samples
(
- set -e
- for sample in 1 2; do
- date -u +'%Y-%m-%dT%H:%M:%SZ'
- pid=$(systemctl show pulse --property=MainPID --value)
- case "$pid" in
- ''|0|*[!0-9]*) printf 'No running Pulse PID; sample unavailable.\n' >&2; exit 1 ;;
- esac
- TZ=UTC ps -p "$pid" -o pid=,lstart=
- sudo awk '
- $1 == "write_bytes:" || $1 == "cancelled_write_bytes:" { print; fields++ }
- END { if (fields != 2) exit 1 }
- ' "/proc/$pid/io"
- if [ "$sample" -eq 1 ]; then sleep 60; fi
- done
+ set -eu
+ command -v timeout >/dev/null 2>&1 || {
+ printf 'Write samples unavailable: GNU timeout is required.\n' >&2; exit 1
+ }
+ if samples=$(timeout --signal=TERM --kill-after=1s 80s bash <<'PULSE_WRITE_SAMPLES' 2>/dev/null
+set -eu
+read_sample() {
+ set -eu
+ pid=$(systemctl show pulse --property=MainPID --value)
+ case "$pid" in
+ ''|0|*[!0-9]*) exit 1 ;;
+ esac
+ identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)
+ [ -n "$identity" ] || exit 1
+ start_ticks=$(awk '
+ { sub(/^.*\) /, ""); if ($20 !~ /^[0-9]+$/) exit 1; print $20; found++ }
+ END { if (found != 1) exit 1 }
+ ' "/proc/$pid/stat")
+ timestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
+ counters=$(sudo -n awk '
+ $1 == "write_bytes:" || $1 == "cancelled_write_bytes:" {
+ if (NF != 2 || $2 !~ /^[0-9]+$/ || seen[$1]++) exit 1
+ print; fields++
+ }
+ END { if (fields != 2) exit 1 }
+ ' "/proc/$pid/io")
+ current_pid=$(systemctl show pulse --property=MainPID --value)
+ current_identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)
+ current_ticks=$(awk '
+ { sub(/^.*\) /, ""); if ($20 !~ /^[0-9]+$/) exit 1; print $20; found++ }
+ END { if (found != 1) exit 1 }
+ ' "/proc/$pid/stat")
+ [ "$pid" = "$current_pid" ] && [ "$identity" = "$current_identity" ] &&
+ [ "$start_ticks" = "$current_ticks" ] || exit 1
+ printf 'Identity: %s; start ticks=%s\nSample UTC: %s\n%s\n' \
+ "$identity" "$start_ticks" "$timestamp" "$counters"
+}
+first=$(read_sample)
+sleep 60
+second=$(read_sample)
+[ "${first%%$'\n'*}" = "${second%%$'\n'*}" ] || exit 1
+printf '%s\n\n%s\n' "$first" "$second"
+PULSE_WRITE_SAMPLES
+ ); then
+ printf '%s\n' "$samples"
+ else
+ status=$?
+ printf 'Write samples unavailable (exit %s); no complete pair. Stop sampling.\n' "$status" >&2
+ exit "$status"
+ fi
)
```
-Compare `write_bytes` only when both samples have the same PID and process
-start time, no restart occurred, and the counter did not decrease. Divide the
+Each recipe requires **GNU `timeout`** and limits the complete two-sample
+collection, including the 60-second wait, to 80 seconds plus a one-second
+termination grace. It signals only its own collection commands, not Pulse or
+Docker. A missing utility, permission failure, timeout or changed identity
+makes the pair unavailable; no partial identity, counter or raw reader error is
+printed, and there is no unbounded fallback. Stop sampling on failure rather
+than removing the deadline or retrying against an unresponsive installation.
+
+The process recipe checks the same PID and process start time before and after
+each read and across both samples. Linux start ticks also distinguish PID reuse
+within one displayed second. These checks do not make the counters atomic.
+Compare `write_bytes` only when no restart occurred and the counter did not decrease. Divide the
byte difference by the **actual elapsed seconds**. This is storage-accounted
process I/O, not filesystem growth or physical SSD wear; cancelled writes and
background writeback can differ from device measurements. Keep
@@ -543,24 +591,59 @@ day's total. Preserve the window and units with the result.
For **Docker / Compose**, run this on the Docker host, replacing `pulse` with
the running container name. It reads only the start time and selected statistics,
-not the container environment or configuration.
+not the container environment or configuration. Each stats call targets the
+full container ID, not a reusable name. The name must still resolve to the same
+running ID, start time and restart count after the read and across both samples;
+a replacement, restart or stopped container makes the pair unavailable.
```bash
# Docker: bounded container statistics
(
- set -e
- for sample in 1 2; do
- date -u +'%Y-%m-%dT%H:%M:%SZ'
- docker inspect --format 'Started={{.State.StartedAt}}' pulse
- stats=$(docker stats --no-stream --format \
- 'CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}' pulse)
- if [ -z "$stats" ]; then
- printf 'Container statistics unavailable; no zero inferred.\n' >&2
- exit 1
- fi
- printf '%s\n' "$stats"
- if [ "$sample" -eq 1 ]; then sleep 60; fi
- done
+ set -eu
+ command -v timeout >/dev/null 2>&1 || {
+ printf 'Container samples unavailable: GNU timeout is required.\n' >&2; exit 1
+ }
+ if samples=$(timeout --signal=TERM --kill-after=1s 80s bash <<'PULSE_CONTAINER_SAMPLES' 2>/dev/null
+set -eu
+read_sample() {
+ set -eu
+ identity=$(docker inspect --type container --format \
+ '{{.Id}} {{.State.StartedAt}} {{.State.Running}} {{.RestartCount}}' pulse)
+ case "$identity" in *$'\n'*) exit 1 ;; esac
+ read -r id started running restarts extra <<< "$identity"
+ [ "${#id}" -eq 64 ] && [ "$running" = true ] && [ -z "$extra" ] || exit 1
+ case "$id" in *[!0-9a-f]*) exit 1 ;; esac
+ case "$started" in 0001-*) exit 1 ;; ????-??-??T??:??:??*Z) ;; *) exit 1 ;; esac
+ case "$restarts" in ''|*[!0-9]*) exit 1 ;; esac
+ timestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
+ stats=$(docker stats --no-stream --format \
+ 'CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}' "$id")
+ case "$stats" in
+ *$'\n'*) exit 1 ;;
+ CPU=*%\ Memory=*\ /\ *\ BlockIO=*\ /\ *) ;;
+ *) exit 1 ;;
+ esac
+ cpu=${stats%% *}
+ [[ "${cpu#CPU=}" =~ ^[0-9]+([.][0-9]+)?%$ ]] || exit 1
+ case "$stats" in *--*) exit 1 ;; esac
+ current_identity=$(docker inspect --type container --format \
+ '{{.Id}} {{.State.StartedAt}} {{.State.Running}} {{.RestartCount}}' pulse)
+ [ "$identity" = "$current_identity" ] || exit 1
+ printf 'Identity: %s\nSample UTC: %s\n%s\n' "$identity" "$timestamp" "$stats"
+}
+first=$(read_sample)
+sleep 60
+second=$(read_sample)
+[ "${first%%$'\n'*}" = "${second%%$'\n'*}" ] || exit 1
+printf '%s\n\n%s\n' "$first" "$second"
+PULSE_CONTAINER_SAMPLES
+ ); then
+ printf '%s\n' "$samples"
+ else
+ status=$?
+ printf 'Container samples unavailable (exit %s); no complete pair. Stop sampling.\n' "$status" >&2
+ exit "$status"
+ fi
)
```
diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.md
index 6151fbb6c0..f19185c985 100644
--- a/frontend-modern/public/docs/TROUBLESHOOTING.md
+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md
@@ -509,31 +509,79 @@ back to an older version does not make that data a clean older-version baseline.
For a responsive **Linux systemd / Proxmox LXC** install, the following reads
two process-I/O samples, waiting 60 seconds between them. Run it inside the
Pulse container for LXC, not on the Proxmox host. Substitute the actual service
-name (`pulse-backend` on some older installs). Use an account authorised to
-read the process counters; no service restart or database access is needed.
+name (`pulse-backend` on some older installs) in both `systemctl` calls. Use an
+account authorised to read the process counters; `sudo -n` must already allow
+this read without prompting. Do not change privileges to make it pass; no
+service restart or database access is needed.
```bash
# systemd / Proxmox LXC: bounded process-write samples
(
- set -e
- for sample in 1 2; do
- date -u +'%Y-%m-%dT%H:%M:%SZ'
- pid=$(systemctl show pulse --property=MainPID --value)
- case "$pid" in
- ''|0|*[!0-9]*) printf 'No running Pulse PID; sample unavailable.\n' >&2; exit 1 ;;
- esac
- TZ=UTC ps -p "$pid" -o pid=,lstart=
- sudo awk '
- $1 == "write_bytes:" || $1 == "cancelled_write_bytes:" { print; fields++ }
- END { if (fields != 2) exit 1 }
- ' "/proc/$pid/io"
- if [ "$sample" -eq 1 ]; then sleep 60; fi
- done
+ set -eu
+ command -v timeout >/dev/null 2>&1 || {
+ printf 'Write samples unavailable: GNU timeout is required.\n' >&2; exit 1
+ }
+ if samples=$(timeout --signal=TERM --kill-after=1s 80s bash <<'PULSE_WRITE_SAMPLES' 2>/dev/null
+set -eu
+read_sample() {
+ set -eu
+ pid=$(systemctl show pulse --property=MainPID --value)
+ case "$pid" in
+ ''|0|*[!0-9]*) exit 1 ;;
+ esac
+ identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)
+ [ -n "$identity" ] || exit 1
+ start_ticks=$(awk '
+ { sub(/^.*\) /, ""); if ($20 !~ /^[0-9]+$/) exit 1; print $20; found++ }
+ END { if (found != 1) exit 1 }
+ ' "/proc/$pid/stat")
+ timestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
+ counters=$(sudo -n awk '
+ $1 == "write_bytes:" || $1 == "cancelled_write_bytes:" {
+ if (NF != 2 || $2 !~ /^[0-9]+$/ || seen[$1]++) exit 1
+ print; fields++
+ }
+ END { if (fields != 2) exit 1 }
+ ' "/proc/$pid/io")
+ current_pid=$(systemctl show pulse --property=MainPID --value)
+ current_identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)
+ current_ticks=$(awk '
+ { sub(/^.*\) /, ""); if ($20 !~ /^[0-9]+$/) exit 1; print $20; found++ }
+ END { if (found != 1) exit 1 }
+ ' "/proc/$pid/stat")
+ [ "$pid" = "$current_pid" ] && [ "$identity" = "$current_identity" ] &&
+ [ "$start_ticks" = "$current_ticks" ] || exit 1
+ printf 'Identity: %s; start ticks=%s\nSample UTC: %s\n%s\n' \
+ "$identity" "$start_ticks" "$timestamp" "$counters"
+}
+first=$(read_sample)
+sleep 60
+second=$(read_sample)
+[ "${first%%$'\n'*}" = "${second%%$'\n'*}" ] || exit 1
+printf '%s\n\n%s\n' "$first" "$second"
+PULSE_WRITE_SAMPLES
+ ); then
+ printf '%s\n' "$samples"
+ else
+ status=$?
+ printf 'Write samples unavailable (exit %s); no complete pair. Stop sampling.\n' "$status" >&2
+ exit "$status"
+ fi
)
```
-Compare `write_bytes` only when both samples have the same PID and process
-start time, no restart occurred, and the counter did not decrease. Divide the
+Each recipe requires **GNU `timeout`** and limits the complete two-sample
+collection, including the 60-second wait, to 80 seconds plus a one-second
+termination grace. It signals only its own collection commands, not Pulse or
+Docker. A missing utility, permission failure, timeout or changed identity
+makes the pair unavailable; no partial identity, counter or raw reader error is
+printed, and there is no unbounded fallback. Stop sampling on failure rather
+than removing the deadline or retrying against an unresponsive installation.
+
+The process recipe checks the same PID and process start time before and after
+each read and across both samples. Linux start ticks also distinguish PID reuse
+within one displayed second. These checks do not make the counters atomic.
+Compare `write_bytes` only when no restart occurred and the counter did not decrease. Divide the
byte difference by the **actual elapsed seconds**. This is storage-accounted
process I/O, not filesystem growth or physical SSD wear; cancelled writes and
background writeback can differ from device measurements. Keep
@@ -543,24 +591,59 @@ day's total. Preserve the window and units with the result.
For **Docker / Compose**, run this on the Docker host, replacing `pulse` with
the running container name. It reads only the start time and selected statistics,
-not the container environment or configuration.
+not the container environment or configuration. Each stats call targets the
+full container ID, not a reusable name. The name must still resolve to the same
+running ID, start time and restart count after the read and across both samples;
+a replacement, restart or stopped container makes the pair unavailable.
```bash
# Docker: bounded container statistics
(
- set -e
- for sample in 1 2; do
- date -u +'%Y-%m-%dT%H:%M:%SZ'
- docker inspect --format 'Started={{.State.StartedAt}}' pulse
- stats=$(docker stats --no-stream --format \
- 'CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}' pulse)
- if [ -z "$stats" ]; then
- printf 'Container statistics unavailable; no zero inferred.\n' >&2
- exit 1
- fi
- printf '%s\n' "$stats"
- if [ "$sample" -eq 1 ]; then sleep 60; fi
- done
+ set -eu
+ command -v timeout >/dev/null 2>&1 || {
+ printf 'Container samples unavailable: GNU timeout is required.\n' >&2; exit 1
+ }
+ if samples=$(timeout --signal=TERM --kill-after=1s 80s bash <<'PULSE_CONTAINER_SAMPLES' 2>/dev/null
+set -eu
+read_sample() {
+ set -eu
+ identity=$(docker inspect --type container --format \
+ '{{.Id}} {{.State.StartedAt}} {{.State.Running}} {{.RestartCount}}' pulse)
+ case "$identity" in *$'\n'*) exit 1 ;; esac
+ read -r id started running restarts extra <<< "$identity"
+ [ "${#id}" -eq 64 ] && [ "$running" = true ] && [ -z "$extra" ] || exit 1
+ case "$id" in *[!0-9a-f]*) exit 1 ;; esac
+ case "$started" in 0001-*) exit 1 ;; ????-??-??T??:??:??*Z) ;; *) exit 1 ;; esac
+ case "$restarts" in ''|*[!0-9]*) exit 1 ;; esac
+ timestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
+ stats=$(docker stats --no-stream --format \
+ 'CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}' "$id")
+ case "$stats" in
+ *$'\n'*) exit 1 ;;
+ CPU=*%\ Memory=*\ /\ *\ BlockIO=*\ /\ *) ;;
+ *) exit 1 ;;
+ esac
+ cpu=${stats%% *}
+ [[ "${cpu#CPU=}" =~ ^[0-9]+([.][0-9]+)?%$ ]] || exit 1
+ case "$stats" in *--*) exit 1 ;; esac
+ current_identity=$(docker inspect --type container --format \
+ '{{.Id}} {{.State.StartedAt}} {{.State.Running}} {{.RestartCount}}' pulse)
+ [ "$identity" = "$current_identity" ] || exit 1
+ printf 'Identity: %s\nSample UTC: %s\n%s\n' "$identity" "$timestamp" "$stats"
+}
+first=$(read_sample)
+sleep 60
+second=$(read_sample)
+[ "${first%%$'\n'*}" = "${second%%$'\n'*}" ] || exit 1
+printf '%s\n\n%s\n' "$first" "$second"
+PULSE_CONTAINER_SAMPLES
+ ); then
+ printf '%s\n' "$samples"
+ else
+ status=$?
+ printf 'Container samples unavailable (exit %s); no complete pair. Stop sampling.\n' "$status" >&2
+ exit "$status"
+ fi
)
```
diff --git a/scripts/tests/test_performance_troubleshooting_docs.py b/scripts/tests/test_performance_troubleshooting_docs.py
index 329b5a2c77..897bebbfb0 100644
--- a/scripts/tests/test_performance_troubleshooting_docs.py
+++ b/scripts/tests/test_performance_troubleshooting_docs.py
@@ -1,24 +1,30 @@
#!/usr/bin/env python3
-"""Exercise the guide's bounded I/O recipes with synthetic OS/container readers.
+"""Execute both copied performance recipes with synthetic readers.
-The real awk filters run against fixtures. No Pulse process, database, Docker
-daemon or host service is inspected. These checks establish recipe behaviour,
-not native write rates, CPU relief or an operational repair.
+Real awk validates fixtures and real GNU timeout enforces the deadline. Only
+this test's shell group can be killed by the test cleanup; no Pulse service,
+database, Docker socket, workload or destination is used.
"""
-
from __future__ import annotations
import json
import os
from pathlib import Path
import re
+import shutil
+import signal
import subprocess
import tempfile
+import time
import unittest
-
ROOT = Path(__file__).resolve().parents[2]
DOC = ROOT / "docs/TROUBLESHOOTING.md"
+CONTAINER_ID = "a" * 64
+DOCKER_IDENTITY = f"{CONTAINER_ID} 2026-10-02T20:00:00Z true 0"
+PROCESS_IDENTITY = "4321 Sat Oct 3 00:00:00 2026"
+DEFAULT_IO = ("rchar: 987654\nwchar: 999999\nwrite_bytes: 1000\ncancelled_write_bytes: 20\n",
+ "rchar: 999999\nwchar: 999999\nwrite_bytes: 13000\ncancelled_write_bytes: 30\n")
def section() -> str:
@@ -30,42 +36,63 @@ def section() -> str:
def recipes() -> dict[str, str]:
blocks = re.findall(r"```bash\n(.*?)```", section(), re.DOTALL)
if len(blocks) != 2:
- raise AssertionError("expected two bounded deployment recipes")
+ raise AssertionError("expected two deployment recipes")
return {"docker" if "# Docker:" in block else "systemd": block for block in blocks}
-# All external commands record exact arguments. sudo is a test-only adapter:
-# it replaces only the documented /proc/PID/io input and executes real awk.
READER = '''#!/usr/bin/env python3
-import json, os, pathlib, subprocess, sys
+import json, os, pathlib, signal, subprocess, sys, time
name = pathlib.Path(sys.argv[0]).name
args = sys.argv[1:]
root = pathlib.Path(os.environ['RECIPE_FIXTURE'])
log = root / 'calls.jsonl'
previous = [json.loads(line) for line in log.read_text().splitlines()] if log.exists() else []
count = sum(call['name'] == name for call in previous)
+if name == 'docker':
+ count = sum(call['name'] == name and call['args'][0] == args[0] for call in previous)
with log.open('a') as stream:
stream.write(json.dumps({'name': name, 'args': args}) + '\\n')
+if os.environ.get('HANG_TOOL') == name:
+ if os.environ.get('IGNORE_TERM'):
+ signal.signal(signal.SIGTERM, signal.SIG_IGN)
+ time.sleep(100)
if name == 'date':
+ assert args == ['-u', '+%Y-%m-%dT%H:%M:%SZ']
print('2026-10-03T00:00:00Z' if count == 0 else '2026-10-03T00:01:00Z')
elif name == 'sleep':
assert args == ['60']
elif name == 'systemctl':
- print(os.environ.get('FIXTURE_PID', '4321'))
+ assert args == ['show', 'pulse', '--property=MainPID', '--value']
+ print(os.environ.get(f'PID_{count}', os.environ.get('FIXTURE_PID', '4321')))
sys.exit(int(os.environ.get('SYSTEMCTL_EXIT', '0')))
elif name == 'ps':
+ assert args == ['-p', '4321', '-o', 'pid=,lstart=']
assert os.environ['TZ'] == 'UTC'
- print('4321 Sat Oct 3 00:00:00 2026')
+ print(os.environ.get(f'PROCESS_{count}', '4321 Sat Oct 3 00:00:00 2026'))
sys.exit(int(os.environ.get('PS_EXIT', '0')))
+elif name == 'awk':
+ assert args[-1] == '/proc/4321/stat'
+ # comm may contain spaces and right parentheses. Starttime is field 22.
+ ticks = os.environ.get(f'TICKS_{count}', '10000')
+ fixture = root / f'stat-{count}'
+ fixture.write_text('4321 (private ) worker) S ' + ' '.join(['0'] * 18 + [ticks]) + '\\n')
+ sys.exit(subprocess.run([os.environ['REAL_AWK'], *args[:-1], str(fixture)]).returncode)
elif name == 'sudo':
- assert args[0] == 'awk'
- assert args[-1] == '/proc/4321/io'
+ assert args[0] in ('awk', '-n')
+ offset = 2 if args[0] == '-n' else 1
+ assert args[offset-1] == 'awk' and args[-1] == '/proc/4321/io'
target = root / ('missing' if os.environ.get('IO_MISSING') else f'io-{count}')
- result = subprocess.run([os.environ['REAL_AWK'], *args[1:-1], str(target)])
- sys.exit(result.returncode)
+ if os.environ.get('SUDO_EXIT'):
+ print('private permission error', file=sys.stderr)
+ sys.exit(int(os.environ['SUDO_EXIT']))
+ sys.exit(subprocess.run([os.environ['REAL_AWK'], *args[offset:-1], str(target)]).returncode)
elif name == 'docker':
if args[0] == 'inspect':
- print('Started=2026-10-02T20:00:00Z')
+ identity = os.environ.get(f'INSPECT_{count}', os.environ['DOCKER_IDENTITY'])
+ if args[1:] == ['--format', 'Started={{.State.StartedAt}}', 'pulse']:
+ print('Started=' + (identity.split()[1] if identity else ''))
+ else:
+ print(identity)
elif args[0] == 'stats':
text = os.environ.get('DOCKER_STATS', 'CPU=12.50% Memory=100MiB / 1GiB BlockIO=1MB / 12MB')
if text:
@@ -73,136 +100,206 @@ elif name == 'docker':
else:
raise AssertionError('unsupported docker command')
setting = 'DOCKER_INSPECT_EXIT' if args[0] == 'inspect' else 'DOCKER_STATS_EXIT'
+ if os.environ.get(setting):
+ print('private daemon error', file=sys.stderr)
sys.exit(int(os.environ.get(setting, '0')))
else:
raise AssertionError('unexpected command')
'''
-class PerformanceTroubleshootingDocsTest(unittest.TestCase):
- def exercise(self, deployment: str, *, counters: tuple[str, str] | None = None,
- **settings: str):
- import shutil
+def run_owned(copied: str, env: dict[str, str]):
+ with subprocess.Popen(["bash", "-c", copied], env=env, text=True,
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ start_new_session=True) as process:
+ try:
+ stdout, stderr = process.communicate(timeout=83)
+ except subprocess.TimeoutExpired:
+ # This is an adverse test error, never an empty/passing sample.
+ try:
+ os.killpg(process.pid, signal.SIGKILL)
+ except ProcessLookupError:
+ pass
+ process.communicate(timeout=5)
+ raise
+ return subprocess.CompletedProcess(process.args, process.returncode, stdout, stderr)
+
+class PerformanceTroubleshootingDocsTest(unittest.TestCase):
+ def exercise(self, deployment: str, *, counters: tuple[str, str] = DEFAULT_IO,
+ copied: str | None = None, **settings: str):
with tempfile.TemporaryDirectory() as temporary:
directory = Path(temporary)
- for name in ("date", "sleep", "systemctl", "ps", "sudo", "docker"):
+ for name in ("date", "sleep", "systemctl", "ps", "awk", "sudo", "docker"):
command = directory / name
command.write_text(READER, encoding="utf-8")
command.chmod(0o700)
- counters = counters or (
- "rchar: 987654\nwchar: 999999\nwrite_bytes: 1000\ncancelled_write_bytes: 20\n",
- "rchar: 999999\nwchar: 999999\nwrite_bytes: 13000\ncancelled_write_bytes: 30\n",
- )
for index, text in enumerate(counters):
(directory / f"io-{index}").write_text(text, encoding="utf-8")
env = dict(os.environ, PATH=f"{directory}:{os.environ['PATH']}",
- RECIPE_FIXTURE=str(directory), REAL_AWK=shutil.which("awk"), **settings)
- result = subprocess.run(["bash", "-c", recipes()[deployment]], env=env,
- text=True, capture_output=True, timeout=10)
- calls = [json.loads(line) for line in (directory / "calls.jsonl").read_text().splitlines()]
+ RECIPE_FIXTURE=str(directory), REAL_AWK=shutil.which("awk"),
+ DOCKER_IDENTITY=DOCKER_IDENTITY, **settings)
+ result = run_owned(copied if copied is not None else recipes()[deployment], env)
+ log = directory / "calls.jsonl"
+ calls = [json.loads(line) for line in log.read_text().splitlines()] if log.exists() else []
return result, calls
- def test_shipped_document_matches_the_exercised_source(self):
+ def assert_unavailable(self, result):
+ self.assertNotEqual(result.returncode, 0)
+ self.assertEqual(result.stdout, "", "an incomplete pair must not be published")
+ self.assertIn("unavailable", result.stderr)
+ self.assertNotIn("private", result.stderr)
+
+ def test_shipped_source_and_shell_syntax(self):
self.assertEqual(DOC.read_bytes(),
(ROOT / "frontend-modern/public/docs/TROUBLESHOOTING.md").read_bytes())
+ for copied in recipes().values():
+ result = subprocess.run(["bash", "-n"], input=copied, text=True, capture_output=True)
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertIn("timeout --signal=TERM --kill-after=1s 80s bash", copied)
- def test_systemd_samples_only_write_counters_and_process_identity(self):
+ def test_systemd_samples_only_valid_write_counters_and_stable_identity(self):
result, calls = self.exercise("systemd")
self.assertEqual(result.returncode, 0, result.stderr)
- self.assertEqual([call["name"] for call in calls],
- ["date", "systemctl", "ps", "sudo", "sleep",
- "date", "systemctl", "ps", "sudo"])
- for call in calls:
- if call["name"] == "systemctl":
- self.assertEqual(call["args"], ["show", "pulse", "--property=MainPID", "--value"])
- elif call["name"] == "ps":
- self.assertEqual(call["args"], ["-p", "4321", "-o", "pid=,lstart="])
- self.assertIn("write_bytes: 1000", result.stdout)
- self.assertIn("write_bytes: 13000", result.stdout)
- self.assertIn("cancelled_write_bytes: 20", result.stdout)
- self.assertNotIn("rchar", result.stdout)
- self.assertNotIn("wchar", result.stdout)
+ self.assertEqual([c["name"] for c in calls],
+ ["systemctl", "ps", "awk", "date", "sudo", "systemctl", "ps", "awk",
+ "sleep", "systemctl", "ps", "awk", "date", "sudo", "systemctl", "ps", "awk"])
+ self.assertEqual([c["args"][0:2] for c in calls if c["name"] == "sudo"], [["-n", "awk"]] * 2)
+ for value in (PROCESS_IDENTITY, "start ticks=10000", "write_bytes: 1000",
+ "write_bytes: 13000", "cancelled_write_bytes: 20", "cancelled_write_bytes: 30"):
+ self.assertIn(value, result.stdout)
+ for forbidden in ("rchar", "wchar", "private ) worker"):
+ self.assertNotIn(forbidden, result.stdout)
- def test_unavailable_pid_never_reads_a_process_or_sleeps(self):
+ def test_invalid_pid_stops_before_process_reads(self):
for pid in ("", "0", "not-a-pid", "4321 extra"):
with self.subTest(pid=pid):
result, calls = self.exercise("systemd", FIXTURE_PID=pid)
- self.assertNotEqual(result.returncode, 0)
- self.assertIn("sample unavailable", result.stderr)
- self.assertEqual([c["name"] for c in calls], ["date", "systemctl"])
+ self.assert_unavailable(result)
+ self.assertEqual([c["name"] for c in calls], ["systemctl"])
- def test_service_and_process_read_failures_stop_the_collection(self):
- for settings, expected in (({"SYSTEMCTL_EXIT": "3"}, ["date", "systemctl"]),
- ({"PS_EXIT": "4"}, ["date", "systemctl", "ps"])):
+ def test_failed_or_empty_identity_does_not_publish_counters(self):
+ for settings in ({"SYSTEMCTL_EXIT": "3"}, {"PS_EXIT": "4"}, {"PROCESS_0": ""},
+ {"TICKS_0": ""}, {"TICKS_0": "bad"}, {"IO_MISSING": "1"}, {"SUDO_EXIT": "5"}):
with self.subTest(settings=settings):
result, calls = self.exercise("systemd", **settings)
- self.assertNotEqual(result.returncode, 0)
- self.assertEqual([c["name"] for c in calls], expected)
-
- def test_missing_process_io_is_failure_not_zero_writes(self):
- result, calls = self.exercise("systemd", IO_MISSING="1")
- self.assertNotEqual(result.returncode, 0)
- self.assertNotIn("write_bytes:", result.stdout)
- self.assertNotIn("sleep", [c["name"] for c in calls])
+ self.assert_unavailable(result)
+ self.assertNotIn("sleep", [c["name"] for c in calls])
- def test_incomplete_process_counters_do_not_become_a_success(self):
- for text in ("", "rchar: 999\n", "write_bytes: 1000\n"):
+ def test_partial_malformed_or_duplicate_counters_are_withheld(self):
+ for text in ("", "rchar: 999\n", "write_bytes: 1000\n", "write_bytes: nope\ncancelled_write_bytes: 2\n",
+ "write_bytes: 1000 bytes\ncancelled_write_bytes: 2\n",
+ "write_bytes: 1000\nwrite_bytes: 1001\n",
+ "write_bytes: 1000\ncancelled_write_bytes: 2\nwrite_bytes: 1001\n"):
with self.subTest(text=text):
- result, calls = self.exercise("systemd", counters=(text, text))
- self.assertNotEqual(result.returncode, 0)
- self.assertNotIn("sleep", [c["name"] for c in calls])
+ result, _ = self.exercise("systemd", counters=(text, text))
+ self.assert_unavailable(result)
- def test_decreasing_counters_are_retained_not_turned_into_a_rate(self):
- result, _ = self.exercise("systemd", counters=(
- "write_bytes: 13000\ncancelled_write_bytes: 30\n",
- "write_bytes: 1000\ncancelled_write_bytes: 20\n",
- ))
+ def test_pid_start_or_same_second_reuse_during_or_between_reads_is_withheld(self):
+ for settings in ({"PID_1": "5432"}, {"PID_2": "5432"},
+ {"PROCESS_1": "4321 Sat Oct 3 00:00:01 2026"},
+ {"PROCESS_2": "4321 Sat Oct 3 00:00:01 2026"},
+ {"TICKS_1": "10001"}, {"TICKS_2": "10001", "TICKS_3": "10001"}):
+ with self.subTest(settings=settings):
+ result, _ = self.exercise("systemd", **settings)
+ self.assert_unavailable(result)
+
+ def test_second_sample_failure_withholds_the_completed_first_sample(self):
+ result, _ = self.exercise("systemd", counters=(DEFAULT_IO[0], "write_bytes: 99\n"))
+ self.assert_unavailable(result)
+
+ def test_decreasing_counters_remain_raw_not_a_rate(self):
+ result, _ = self.exercise("systemd", counters=(DEFAULT_IO[1], DEFAULT_IO[0]))
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("write_bytes: 13000", result.stdout)
self.assertIn("write_bytes: 1000", result.stdout)
self.assertNotRegex(result.stdout, r"GB/day|bytes/s|bytes: -[0-9]+")
- def test_docker_reads_only_start_time_and_selected_statistics_twice(self):
+ def test_docker_targets_the_full_id_and_checks_its_running_instance(self):
result, calls = self.exercise("docker")
self.assertEqual(result.returncode, 0, result.stderr)
- expected = [
- ["inspect", "--format", "Started={{.State.StartedAt}}", "pulse"],
- ["stats", "--no-stream", "--format",
- "CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}", "pulse"],
- ]
- self.assertEqual([c["args"] for c in calls if c["name"] == "docker"], expected * 2)
- self.assertEqual([c["name"] for c in calls],
- ["date", "docker", "docker", "sleep", "date", "docker", "docker"])
+ expected = ["inspect", "--type", "container", "--format",
+ "{{.Id}} {{.State.StartedAt}} {{.State.Running}} {{.RestartCount}}", "pulse"]
+ self.assertEqual([c["args"] for c in calls if c["name"] == "docker" and c["args"][0] == "inspect"],
+ [expected] * 4)
+ self.assertEqual([c["args"] for c in calls if c["name"] == "docker" and c["args"][0] == "stats"],
+ [["stats", "--no-stream", "--format",
+ "CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}", CONTAINER_ID]] * 2)
+ self.assertEqual(result.stdout.count(DOCKER_IDENTITY), 2)
self.assertEqual(result.stdout.count("BlockIO=1MB / 12MB"), 2)
- def test_empty_docker_statistics_are_unavailable_not_zero(self):
- result, calls = self.exercise("docker", DOCKER_STATS="")
- self.assertNotEqual(result.returncode, 0)
- self.assertIn("no zero inferred", result.stderr)
- self.assertNotIn("sleep", [c["name"] for c in calls])
-
- def test_docker_reader_failure_stops_before_another_sample(self):
- for settings, expected in (
- ({"DOCKER_INSPECT_EXIT": "2"}, ["date", "docker"]),
- ({"DOCKER_STATS_EXIT": "3"}, ["date", "docker", "docker"]),
- ):
- with self.subTest(settings=settings):
- result, calls = self.exercise("docker", **settings)
- self.assertEqual(result.returncode, int(next(iter(settings.values()))))
- self.assertEqual([c["name"] for c in calls], expected)
+ def test_invalid_stopped_or_unstarted_docker_identity_is_unavailable(self):
+ for identity in ("", DOCKER_IDENTITY.replace(CONTAINER_ID, "short"),
+ DOCKER_IDENTITY.replace("true", "false"),
+ DOCKER_IDENTITY.replace("2026-10-02T20:00:00Z", "0001-01-01T00:00:00Z"),
+ DOCKER_IDENTITY.replace("2026-10-02T20:00:00Z", "bad"),
+ DOCKER_IDENTITY + " extra", DOCKER_IDENTITY + "\n" + DOCKER_IDENTITY):
+ with self.subTest(identity=identity):
+ result, calls = self.exercise("docker", INSPECT_0=identity)
+ self.assert_unavailable(result)
+ self.assertFalse(any(c["name"] == "docker" and c["args"][0] == "stats" for c in calls))
+
+ def test_empty_malformed_or_multiple_docker_stats_are_unavailable(self):
+ for text in ("", "--", "CPU=-- Memory=-- / -- BlockIO=-- / --", "CPU=bad% Memory=1 / 2 BlockIO=1 / 2",
+ "CPU=1% Memory=1 / 2", "CPU=1% Memory=1 / 2 BlockIO=1 / 2\nprivate extra"):
+ with self.subTest(text=text):
+ result, _ = self.exercise("docker", DOCKER_STATS=text)
+ self.assert_unavailable(result)
+
+ def test_docker_errors_preserve_exit_without_partial_or_raw_error_output(self):
+ for setting, value in (("DOCKER_INSPECT_EXIT", "2"), ("DOCKER_STATS_EXIT", "3")):
+ with self.subTest(setting=setting):
+ result, calls = self.exercise("docker", **{setting: value})
+ self.assert_unavailable(result)
+ self.assertEqual(result.returncode, int(value))
+ self.assertNotIn("sleep", [c["name"] for c in calls])
+
+ def test_docker_replacement_restart_or_stop_during_and_between_reads_is_unavailable(self):
+ for identity in (DOCKER_IDENTITY.replace(CONTAINER_ID, "b" * 64),
+ DOCKER_IDENTITY.replace("20:00:00", "20:01:00"),
+ DOCKER_IDENTITY[:-1] + "1", DOCKER_IDENTITY.replace("true", "false")):
+ for index in (1, 2, 3):
+ with self.subTest(identity=identity, index=index):
+ result, _ = self.exercise("docker", **{f"INSPECT_{index}": identity})
+ self.assert_unavailable(result)
+
+ def test_missing_timeout_stops_before_any_reader(self):
+ # No timeout in this private PATH. bash is invoked by absolute path.
+ with tempfile.TemporaryDirectory() as temporary:
+ for deployment, copied in recipes().items():
+ with self.subTest(deployment=deployment):
+ result = subprocess.run([shutil.which("bash"), "-c", copied],
+ env=dict(os.environ, PATH=temporary),
+ text=True, capture_output=True, timeout=3)
+ self.assert_unavailable(result)
+ self.assertIn("GNU timeout is required", result.stderr)
+
+ def test_real_deadline_stops_a_hung_reader_in_both_recipes(self):
+ for deployment, name in (("systemd", "systemctl"), ("docker", "docker")):
+ with self.subTest(deployment=deployment):
+ started = time.monotonic()
+ result, _ = self.exercise(deployment, HANG_TOOL=name)
+ self.assert_unavailable(result)
+ self.assertEqual(result.returncode, 124)
+ self.assertLess(time.monotonic() - started, 83)
+
+ def test_real_kill_grace_stops_a_reader_ignoring_term(self):
+ started = time.monotonic()
+ result, _ = self.exercise("docker", HANG_TOOL="docker", IGNORE_TERM="1")
+ self.assert_unavailable(result)
+ self.assertEqual(result.returncode, 137)
+ self.assertLess(time.monotonic() - started, 83)
- def test_guidance_keeps_scope_units_baselines_and_data_safety(self):
+ def test_guidance_preserves_scope_units_baselines_and_data_safety(self):
guide = " ".join(section().split())
- for phrase in ("same PID and process start time", "actual elapsed seconds",
- "not a measured day's total", "not bytes per second",
- "not a recent sampling window", "different scopes",
- "inside the Pulse container", "does not make that data a clean",
- "Do not enable Debug", "remove database indexes", "tmpfs",
- "Do not post databases", "manually redacted error"):
+ for phrase in ("same PID and process start time", "actual elapsed seconds", "not a measured day's total",
+ "not bytes per second", "not a recent sampling window", "different scopes",
+ "inside the Pulse container", "does not make that data a clean", "Do not enable Debug",
+ "remove database indexes", "tmpfs", "Do not post databases", "manually redacted error",
+ "no partial identity", "no unbounded fallback", "full container ID", "sudo -n"):
self.assertIn(phrase, guide)
- for recipe in recipes().values():
- self.assertNotRegex(recipe, r"\b(?:restart|kill|rm|truncate|sqlite3|curl|printenv)\b|--follow|--token")
+ for copied in recipes().values():
+ self.assertNotRegex(copied, r"\b(?:restart|rm|truncate|sqlite3|curl|printenv)\b|--follow|--token|(?:^|[;\n])\s*kill\b")
if __name__ == "__main__":