Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
ci: compare memory with the latest stable release every day
rcourtman44af30787847
diff --git a/.github/workflows/README.md b/.github/workflows/README.mdindex 8587eb278f..9700483974 100644--- a/.github/workflows/README.md+++ b/.github/workflows/README.md@@ -173,6 +173,23 @@ access) reports one outcome under its own title, an error annotation or a`passed` notice, so a check that was never reached is distinguishable fromone that passed. It uploads nothing and gates nothing.+`memory-regression.yml` compares Pulse's memory with the latest stable release,+daily at 04:41 UTC and on manual dispatch. It builds that release and `main` from+source with a placeholder frontend, runs both for 15 minutes at once against the+same small mock estate (2 nodes, 24 guests, a Docker host, telemetry off, no+licence server), and splits steady-state resident memory into the Go heap, memory+outside Go (where the SQLite driver allocates) and file pages+(`scripts/memory/compare_release_memory.sh`). It fails when `main` uses more than+15% and 16 MiB more resident memory than the release. The step summary has the+table, and the `memory-regression` artifact holds the samples, both heap profiles+and `heap-growth.txt`, which lists what grew by function. Mock fixtures live in+the heap, so a growth list led by `internal/mock` is not production memory. A+failed scheduled run opens an issue titled "Pulse memory grew against the last+stable release", or comments on the open one, and the next passing scheduled run+closes it. SQLite page caches only fill on large databases, which this short run+never builds; `pkg/db/pagecache_test.go` guards those on every pull request.+It gates nothing.+`dependency-advisory-watch.yml` runs the required build-and-test frontend auditdaily against `main` and every active `release/v<major>.<minor>` line (thelatest stable's line and newer), because a new npm advisory against andiff --git a/.github/workflows/memory-regression.yml b/.github/workflows/memory-regression.ymlnew file mode 100644index 0000000000..266c5b4ec1--- /dev/null+++ b/.github/workflows/memory-regression.yml@@ -0,0 +1,136 @@+name: Memory Regression Watch++# Pulse's resident memory rose release after release without anything noticing+# (#1803): a 256 MB LXC that ran v5 for a year needed 1 GB by v6.3. Nothing+# compared memory across versions, and part of the growth sat outside the Go+# heap, where heap-only tests cannot see it. This watch builds the latest stable+# release and main from source, runs both against the same small mock estate,+# and compares resident memory split into the Go heap, memory outside Go and+# file pages. Both heap profiles and their difference are uploaded, so a failure+# names what grew. SQLite page caches only fill on large databases, which a+# short mock run never builds; pkg/db's page cache tests guard those instead.+on:+ schedule:+ - cron: '41 4 * * *'+ workflow_dispatch:++permissions:+ contents: read++concurrency:+ group: memory-regression+ cancel-in-progress: false++jobs:+ compare:+ name: Compare memory with the latest stable release+ runs-on: ubuntu-24.04+ timeout-minutes: 45+ steps:+ - name: Checkout candidate+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1+ with:+ path: candidate+ persist-credentials: false++ - name: Resolve the latest stable release+ id: stable+ env:+ GH_TOKEN: ${{ github.token }}+ GH_REPO: ${{ github.repository }}+ run: |+ set -euo pipefail+ tag="$(gh release view --json tagName,isPrerelease --jq 'select(.isPrerelease | not) | .tagName')"+ [[ "${tag}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]] || { echo "No stable release tag found: '${tag}'" >&2; exit 1; }+ echo "tag=${tag}" >> "$GITHUB_OUTPUT"+ echo "Baseline: ${tag}"++ - name: Checkout baseline+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1+ with:+ ref: ${{ steps.stable.outputs.tag }}+ path: baseline+ persist-credentials: false++ - name: Set up Go+ uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0+ with:+ go-version-file: candidate/go.mod+ cache: false++ - name: Measure both builds+ env:+ # Each tree builds with the toolchain its own go.mod names.+ GOTOOLCHAIN: auto+ PULSE_MEMORY_DURATION: '900'+ run: bash candidate/scripts/memory/compare_release_memory.sh baseline candidate "$RUNNER_TEMP/memory"++ - name: Publish the comparison+ if: ${{ !cancelled() }}+ run: |+ if [[ -f "$RUNNER_TEMP/memory/summary.md" ]]; then+ cat "$RUNNER_TEMP/memory/summary.md" >> "$GITHUB_STEP_SUMMARY"+ else+ echo "The measurement did not finish; see the run logs." >> "$GITHUB_STEP_SUMMARY"+ fi++ - name: Upload memory evidence+ if: ${{ !cancelled() }}+ uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2+ with:+ name: memory-regression+ path: |+ ${{ runner.temp }}/memory/*.tsv+ ${{ runner.temp }}/memory/*.pb.gz+ ${{ runner.temp }}/memory/heap-growth.txt+ ${{ runner.temp }}/memory/summary.md+ ${{ runner.temp }}/memory/result.json+ ${{ runner.temp }}/memory/run-*.log+ if-no-files-found: ignore+ retention-days: 30++ # Scheduled runs have no caller watching them. Keep one open issue while the+ # watch fails and close it when memory is back within tolerance.+ report-scheduled-result:+ name: Report scheduled result+ needs: compare+ if: ${{ always() && github.event_name == 'schedule' }}+ runs-on: ubuntu-24.04+ timeout-minutes: 5+ permissions:+ issues: write+ steps:+ - name: Open, update or close the memory regression issue+ shell: bash+ env:+ GH_TOKEN: ${{ github.token }}+ GH_REPO: ${{ github.repository }}+ RESULT: ${{ needs.compare.result }}+ RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}+ run: |+ set -euo pipefail+ title="Pulse memory grew against the last stable release"+ number="$(gh issue list --state open --search "\"${title}\" in:title" \+ --limit 100 --json number,title \+ --jq "map(select(.title == \"${title}\")) | first | .number // empty")"+ case "${RESULT}" in+ success)+ if [[ -n "${number}" ]]; then+ gh issue close "${number}" --comment "Memory is back within tolerance: ${RUN_URL}"+ fi+ ;;+ failure)+ if [[ -n "${number}" ]]; then+ gh issue comment "${number}" --body "It failed again: ${RUN_URL}"+ else+ gh issue create --title "${title}" --body "The daily memory comparison failed: ${RUN_URL}++ The run summary compares main with the latest stable release on the same small mock estate, split into the Go heap, memory outside Go and file pages. The memory-regression artifact holds both heap profiles and heap-growth.txt, which lists what grew by function. Check that list for internal/mock first: mock fixtures live in the heap and are not production memory. A failure with no samples means a build or startup failed, not a memory change.++ This issue closes itself when the comparison passes again."+ fi+ ;;+ *)+ echo "The comparison ended as ${RESULT}; nothing to report."+ ;;+ esacdiff --git a/scripts/memory/compare_release_memory.sh b/scripts/memory/compare_release_memory.shnew file mode 100644index 0000000000..664523db7c--- /dev/null+++ b/scripts/memory/compare_release_memory.sh@@ -0,0 +1,113 @@+#!/usr/bin/env bash+# compare_release_memory.sh — run two Pulse source trees against the same fixed+# small mock estate and compare their steady-state memory (#1803).+#+# Usage: bash scripts/memory/compare_release_memory.sh <baseline-tree> <candidate-tree> <out-dir>+#+# Each tree is built from source with a placeholder frontend (release builds+# refuse mock fixtures), started with telemetry off and no licence server, and+# sampled every PULSE_MEMORY_INTERVAL seconds for PULSE_MEMORY_DURATION seconds.+# Samples split resident memory into the Go heap mapping, unnamed anonymous+# memory (where the modernc SQLite driver allocates) and file pages. At the end+# each run's heap profile is saved, and summarize_release_memory.py writes the+# comparison and exits 1 when the candidate grew past the tolerance.+#+# Mock mode keeps its own fixtures in the heap, so a fixture change shows up+# here too. Read the heap profile diff before blaming production code.++set -euo pipefail++BASELINE_DIR="${1:?baseline tree}"+CANDIDATE_DIR="${2:?candidate tree}"+OUT_DIR="${3:?output directory}"+DURATION="${PULSE_MEMORY_DURATION:-900}"+INTERVAL="${PULSE_MEMORY_INTERVAL:-15}"+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"++mkdir -p "${OUT_DIR}"+OUT_DIR="$(cd "${OUT_DIR}" && pwd)"++build_tree() {+ local label=$1 tree=$2+ local dist="${tree}/internal/api/frontend-modern/dist"+ mkdir -p "${dist}"+ [[ -e "${dist}/index.html" ]] || printf '<!doctype html><title>pulse</title>\n' > "${dist}/index.html"+ (cd "${tree}" && CGO_ENABLED=0 go build -o "${OUT_DIR}/pulse-${label}" ./cmd/pulse)+}++# Resident KiB by mapping class: Go heap, other Go runtime, unnamed anonymous,+# file-backed. Go 1.25+ names its own anonymous mappings.+classify_smaps() {+ awk '+ /^[0-9a-f]+-[0-9a-f]+ / {+ name = ""+ for (i = 6; i <= NF; i++) name = name (i > 6 ? " " : "") $i+ if (name == "[anon: Go: heap]") cls = "go_heap"+ else if (name ~ /^\[anon: Go:/) cls = "go_other"+ else if (name == "") cls = "anon_other"+ else if (name ~ /^\//) cls = "file"+ else cls = "special"+ next+ }+ /^Rss:/ { kib[cls] += $2 }+ END { printf "%d\t%d\t%d\t%d", kib["go_heap"], kib["go_other"], kib["anon_other"], kib["file"] }+ ' "$1"+}++run_tree() {+ local label=$1 port=$2+ local data="${OUT_DIR}/data-${label}" log="${OUT_DIR}/run-${label}.log"+ local samples="${OUT_DIR}/samples-${label}.tsv"+ rm -rf "${data}" && mkdir -p "${data}"+ env -i HOME="${HOME}" PATH="${PATH}" \+ PULSE_MOCK_MODE=true PULSE_MOCK_NODES=2 PULSE_MOCK_VMS_PER_NODE=6 PULSE_MOCK_LXCS_PER_NODE=6 \+ PULSE_MOCK_DOCKER_HOSTS=1 PULSE_MOCK_DOCKER_CONTAINERS=10 PULSE_MOCK_GENERIC_HOSTS=1 \+ PULSE_MOCK_K8S_CLUSTERS=0 \+ PULSE_DATA_DIR="${data}" PORT="${port}" FRONTEND_PORT="${port}" \+ PULSE_DEV=true ALLOW_ADMIN_BYPASS=1 PULSE_DEV_DISABLE_BACKGROUND_AI=true \+ PULSE_TELEMETRY=false PULSE_LICENSE_SERVER_URL=http://127.0.0.1:9 \+ GOMAXPROCS=1 \+ "${OUT_DIR}/pulse-${label}" > "${log}" 2>&1 &+ local pid=$!+ printf 'elapsed_s\trss_kib\tgo_heap_kib\tgo_other_kib\tanon_other_kib\tfile_kib\n' > "${samples}"+ local start now+ start=$(date +%s)+ while kill -0 "${pid}" 2>/dev/null; do+ now=$(( $(date +%s) - start ))+ (( now >= DURATION )) && break+ local rss+ rss=$(awk '/^VmRSS:/ {print $2}' "/proc/${pid}/status" 2>/dev/null || true)+ if [[ -n "${rss}" ]]; then+ printf '%s\t%s\t%s\n' "${now}" "${rss}" "$(classify_smaps "/proc/${pid}/smaps")" >> "${samples}"+ fi+ sleep "${INTERVAL}"+ done+ if kill -0 "${pid}" 2>/dev/null; then+ curl -fsS -o "${OUT_DIR}/heap-${label}.pb.gz" "http://127.0.0.1:${port}/debug/pprof/heap" || true+ kill "${pid}" 2>/dev/null || true+ wait "${pid}" 2>/dev/null || true+ else+ echo "${label} exited before the measurement ended; see ${log}" >&2+ return 1+ fi+}++build_tree baseline "${BASELINE_DIR}"+build_tree candidate "${CANDIDATE_DIR}"++# Run both at once so they share host conditions. Only one can bind the fixed+# Prometheus port; the other logs a bind error, which costs no measurable memory.+run_tree baseline 7801 & baseline_pid=$!+run_tree candidate 7802 & candidate_pid=$!+status=0+wait "${baseline_pid}" || status=1+wait "${candidate_pid}" || status=1+(( status == 0 )) || exit 1++if [[ -s "${OUT_DIR}/heap-baseline.pb.gz" && -s "${OUT_DIR}/heap-candidate.pb.gz" ]]; then+ (cd "${CANDIDATE_DIR}" && go tool pprof -sample_index=inuse_space -top -nodecount=25 \+ -base "${OUT_DIR}/heap-baseline.pb.gz" "${OUT_DIR}/heap-candidate.pb.gz") \+ > "${OUT_DIR}/heap-growth.txt" 2>/dev/null || true+fi++python3 "${SCRIPT_DIR}/summarize_release_memory.py" "${OUT_DIR}"diff --git a/scripts/memory/summarize_release_memory.py b/scripts/memory/summarize_release_memory.pynew file mode 100644index 0000000000..13ed82b9b6--- /dev/null+++ b/scripts/memory/summarize_release_memory.py@@ -0,0 +1,97 @@+#!/usr/bin/env python3+"""Compare steady-state memory from compare_release_memory.sh samples (#1803).++Reads samples-baseline.tsv and samples-candidate.tsv from the output+directory, takes the median of each column over the final third of each run,+writes summary.md and result.json, and exits 1 when the candidate's resident+memory exceeds the baseline by more than both the percentage tolerance and the+absolute slack. Requiring both keeps small installs' noise from failing it.+"""++from __future__ import annotations++import csv+import json+import os+import statistics+import sys+from pathlib import Path++COLUMNS = ("rss_kib", "go_heap_kib", "go_other_kib", "anon_other_kib", "file_kib")+LABELS = {+ "rss_kib": "Resident",+ "go_heap_kib": "Go heap",+ "go_other_kib": "Go runtime",+ "anon_other_kib": "Outside Go (SQLite and C allocations)",+ "file_kib": "File pages",+}+++def steady_state(path: Path) -> dict[str, float]:+ with path.open(newline="") as handle:+ rows = [row for row in csv.DictReader(handle, delimiter="\t") if row.get("rss_kib")]+ if len(rows) < 3:+ raise SystemExit(f"{path.name}: only {len(rows)} samples; the run ended early")+ last = int(rows[-1]["elapsed_s"])+ tail = [row for row in rows if int(row["elapsed_s"]) >= last * 2 // 3] or rows[-3:]+ return {column: statistics.median(float(row[column]) for row in tail) for column in COLUMNS}+++def main(argv: list[str]) -> int:+ if len(argv) != 2:+ print(__doc__, file=sys.stderr)+ return 2+ out = Path(argv[1])+ tolerance = float(os.environ.get("PULSE_MEMORY_TOLERANCE_PCT", "15"))+ slack_kib = float(os.environ.get("PULSE_MEMORY_SLACK_MIB", "16")) * 1024+ baseline = steady_state(out / "samples-baseline.tsv")+ candidate = steady_state(out / "samples-candidate.tsv")++ growth = candidate["rss_kib"] - baseline["rss_kib"]+ growth_pct = 100 * growth / baseline["rss_kib"] if baseline["rss_kib"] else 0.0+ regressed = growth > slack_kib and growth_pct > tolerance++ lines = [+ "## Pulse memory: candidate against baseline",+ "",+ "Steady state is the median over the final third of each run, on the fixed small mock estate.",+ "",+ "| | Baseline MiB | Candidate MiB | Change |",+ "|---|---:|---:|---:|",+ ]+ for column in COLUMNS:+ base, cand = baseline[column] / 1024, candidate[column] / 1024+ lines.append(f"| {LABELS[column]} | {base:.1f} | {cand:.1f} | {cand - base:+.1f} |")+ verdict = (+ f"**Regression:** resident memory grew {growth / 1024:+.1f} MiB ({growth_pct:+.1f}%), "+ f"past both {tolerance:g}% and {slack_kib / 1024:g} MiB."+ if regressed+ else f"No regression: resident memory changed {growth / 1024:+.1f} MiB ({growth_pct:+.1f}%)."+ )+ lines += ["", verdict]+ growth_file = out / "heap-growth.txt"+ if growth_file.exists() and growth_file.stat().st_size:+ top = growth_file.read_text().splitlines()[:30]+ lines += ["", "Heap growth by function (candidate minus baseline, in-use space):", "", "```", *top, "```"]+ lines += [+ "",+ "Mock mode keeps its own fixtures in the heap, so check the growth list for `internal/mock` before",+ "attributing a change to production code.",+ ]+ summary = "\n".join(lines) + "\n"+ (out / "summary.md").write_text(summary)+ (out / "result.json").write_text(json.dumps({+ "baseline": baseline,+ "candidate": candidate,+ "rss_growth_kib": growth,+ "rss_growth_pct": growth_pct,+ "tolerance_pct": tolerance,+ "slack_kib": slack_kib,+ "regressed": regressed,+ }, indent=2) + "\n")+ print(summary)+ return 1 if regressed else 0+++if __name__ == "__main__":+ sys.exit(main(sys.argv))