Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

ci: compare memory with the latest stable release every day

rcourtman44af30787847
diff --git a/.github/workflows/README.md b/.github/workflows/README.md
index 8587eb278f..9700483974 100644
--- a/.github/workflows/README.md
+++ b/.github/workflows/README.md
@@ -173,6 +173,23 @@ access) reports one outcome under its own title, an error annotation or a
`passed` notice, so a check that was never reached is distinguishable from
one that passed. It uploads nothing and gates nothing.
+`memory-regression.yml` compares Pulse's memory with the latest stable release,
+daily at 04:41 UTC and on manual dispatch. It builds that release and `main` from
+source with a placeholder frontend, runs both for 15 minutes at once against the
+same small mock estate (2 nodes, 24 guests, a Docker host, telemetry off, no
+licence server), and splits steady-state resident memory into the Go heap, memory
+outside Go (where the SQLite driver allocates) and file pages
+(`scripts/memory/compare_release_memory.sh`). It fails when `main` uses more than
+15% and 16 MiB more resident memory than the release. The step summary has the
+table, and the `memory-regression` artifact holds the samples, both heap profiles
+and `heap-growth.txt`, which lists what grew by function. Mock fixtures live in
+the heap, so a growth list led by `internal/mock` is not production memory. A
+failed scheduled run opens an issue titled "Pulse memory grew against the last
+stable release", or comments on the open one, and the next passing scheduled run
+closes it. SQLite page caches only fill on large databases, which this short run
+never builds; `pkg/db/pagecache_test.go` guards those on every pull request.
+It gates nothing.
+
`dependency-advisory-watch.yml` runs the required build-and-test frontend audit
daily against `main` and every active `release/v<major>.<minor>` line (the
latest stable's line and newer), because a new npm advisory against an
diff --git a/.github/workflows/memory-regression.yml b/.github/workflows/memory-regression.yml
new file mode 100644
index 0000000000..266c5b4ec1
--- /dev/null
+++ b/.github/workflows/memory-regression.yml
@@ -0,0 +1,136 @@
+name: Memory Regression Watch
+
+# Pulse's resident memory rose release after release without anything noticing
+# (#1803): a 256 MB LXC that ran v5 for a year needed 1 GB by v6.3. Nothing
+# compared memory across versions, and part of the growth sat outside the Go
+# heap, where heap-only tests cannot see it. This watch builds the latest stable
+# release and main from source, runs both against the same small mock estate,
+# and compares resident memory split into the Go heap, memory outside Go and
+# file pages. Both heap profiles and their difference are uploaded, so a failure
+# names what grew. SQLite page caches only fill on large databases, which a
+# short mock run never builds; pkg/db's page cache tests guard those instead.
+on:
+ schedule:
+ - cron: '41 4 * * *'
+ workflow_dispatch:
+
+permissions:
+ contents: read
+
+concurrency:
+ group: memory-regression
+ cancel-in-progress: false
+
+jobs:
+ compare:
+ name: Compare memory with the latest stable release
+ runs-on: ubuntu-24.04
+ timeout-minutes: 45
+ steps:
+ - name: Checkout candidate
+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+ with:
+ path: candidate
+ persist-credentials: false
+
+ - name: Resolve the latest stable release
+ id: stable
+ env:
+ GH_TOKEN: ${{ github.token }}
+ GH_REPO: ${{ github.repository }}
+ run: |
+ set -euo pipefail
+ tag="$(gh release view --json tagName,isPrerelease --jq 'select(.isPrerelease | not) | .tagName')"
+ [[ "${tag}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]] || { echo "No stable release tag found: '${tag}'" >&2; exit 1; }
+ echo "tag=${tag}" >> "$GITHUB_OUTPUT"
+ echo "Baseline: ${tag}"
+
+ - name: Checkout baseline
+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+ with:
+ ref: ${{ steps.stable.outputs.tag }}
+ path: baseline
+ persist-credentials: false
+
+ - name: Set up Go
+ uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
+ with:
+ go-version-file: candidate/go.mod
+ cache: false
+
+ - name: Measure both builds
+ env:
+ # Each tree builds with the toolchain its own go.mod names.
+ GOTOOLCHAIN: auto
+ PULSE_MEMORY_DURATION: '900'
+ run: bash candidate/scripts/memory/compare_release_memory.sh baseline candidate "$RUNNER_TEMP/memory"
+
+ - name: Publish the comparison
+ if: ${{ !cancelled() }}
+ run: |
+ if [[ -f "$RUNNER_TEMP/memory/summary.md" ]]; then
+ cat "$RUNNER_TEMP/memory/summary.md" >> "$GITHUB_STEP_SUMMARY"
+ else
+ echo "The measurement did not finish; see the run logs." >> "$GITHUB_STEP_SUMMARY"
+ fi
+
+ - name: Upload memory evidence
+ if: ${{ !cancelled() }}
+ uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
+ with:
+ name: memory-regression
+ path: |
+ ${{ runner.temp }}/memory/*.tsv
+ ${{ runner.temp }}/memory/*.pb.gz
+ ${{ runner.temp }}/memory/heap-growth.txt
+ ${{ runner.temp }}/memory/summary.md
+ ${{ runner.temp }}/memory/result.json
+ ${{ runner.temp }}/memory/run-*.log
+ if-no-files-found: ignore
+ retention-days: 30
+
+ # Scheduled runs have no caller watching them. Keep one open issue while the
+ # watch fails and close it when memory is back within tolerance.
+ report-scheduled-result:
+ name: Report scheduled result
+ needs: compare
+ if: ${{ always() && github.event_name == 'schedule' }}
+ runs-on: ubuntu-24.04
+ timeout-minutes: 5
+ permissions:
+ issues: write
+ steps:
+ - name: Open, update or close the memory regression issue
+ shell: bash
+ env:
+ GH_TOKEN: ${{ github.token }}
+ GH_REPO: ${{ github.repository }}
+ RESULT: ${{ needs.compare.result }}
+ RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+ run: |
+ set -euo pipefail
+ title="Pulse memory grew against the last stable release"
+ number="$(gh issue list --state open --search "\"${title}\" in:title" \
+ --limit 100 --json number,title \
+ --jq "map(select(.title == \"${title}\")) | first | .number // empty")"
+ case "${RESULT}" in
+ success)
+ if [[ -n "${number}" ]]; then
+ gh issue close "${number}" --comment "Memory is back within tolerance: ${RUN_URL}"
+ fi
+ ;;
+ failure)
+ if [[ -n "${number}" ]]; then
+ gh issue comment "${number}" --body "It failed again: ${RUN_URL}"
+ else
+ gh issue create --title "${title}" --body "The daily memory comparison failed: ${RUN_URL}
+
+ The run summary compares main with the latest stable release on the same small mock estate, split into the Go heap, memory outside Go and file pages. The memory-regression artifact holds both heap profiles and heap-growth.txt, which lists what grew by function. Check that list for internal/mock first: mock fixtures live in the heap and are not production memory. A failure with no samples means a build or startup failed, not a memory change.
+
+ This issue closes itself when the comparison passes again."
+ fi
+ ;;
+ *)
+ echo "The comparison ended as ${RESULT}; nothing to report."
+ ;;
+ esac
diff --git a/scripts/memory/compare_release_memory.sh b/scripts/memory/compare_release_memory.sh
new file mode 100644
index 0000000000..664523db7c
--- /dev/null
+++ b/scripts/memory/compare_release_memory.sh
@@ -0,0 +1,113 @@
+#!/usr/bin/env bash
+# compare_release_memory.sh — run two Pulse source trees against the same fixed
+# small mock estate and compare their steady-state memory (#1803).
+#
+# Usage: bash scripts/memory/compare_release_memory.sh <baseline-tree> <candidate-tree> <out-dir>
+#
+# Each tree is built from source with a placeholder frontend (release builds
+# refuse mock fixtures), started with telemetry off and no licence server, and
+# sampled every PULSE_MEMORY_INTERVAL seconds for PULSE_MEMORY_DURATION seconds.
+# Samples split resident memory into the Go heap mapping, unnamed anonymous
+# memory (where the modernc SQLite driver allocates) and file pages. At the end
+# each run's heap profile is saved, and summarize_release_memory.py writes the
+# comparison and exits 1 when the candidate grew past the tolerance.
+#
+# Mock mode keeps its own fixtures in the heap, so a fixture change shows up
+# here too. Read the heap profile diff before blaming production code.
+
+set -euo pipefail
+
+BASELINE_DIR="${1:?baseline tree}"
+CANDIDATE_DIR="${2:?candidate tree}"
+OUT_DIR="${3:?output directory}"
+DURATION="${PULSE_MEMORY_DURATION:-900}"
+INTERVAL="${PULSE_MEMORY_INTERVAL:-15}"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+mkdir -p "${OUT_DIR}"
+OUT_DIR="$(cd "${OUT_DIR}" && pwd)"
+
+build_tree() {
+ local label=$1 tree=$2
+ local dist="${tree}/internal/api/frontend-modern/dist"
+ mkdir -p "${dist}"
+ [[ -e "${dist}/index.html" ]] || printf '<!doctype html><title>pulse</title>\n' > "${dist}/index.html"
+ (cd "${tree}" && CGO_ENABLED=0 go build -o "${OUT_DIR}/pulse-${label}" ./cmd/pulse)
+}
+
+# Resident KiB by mapping class: Go heap, other Go runtime, unnamed anonymous,
+# file-backed. Go 1.25+ names its own anonymous mappings.
+classify_smaps() {
+ awk '
+ /^[0-9a-f]+-[0-9a-f]+ / {
+ name = ""
+ for (i = 6; i <= NF; i++) name = name (i > 6 ? " " : "") $i
+ if (name == "[anon: Go: heap]") cls = "go_heap"
+ else if (name ~ /^\[anon: Go:/) cls = "go_other"
+ else if (name == "") cls = "anon_other"
+ else if (name ~ /^\//) cls = "file"
+ else cls = "special"
+ next
+ }
+ /^Rss:/ { kib[cls] += $2 }
+ END { printf "%d\t%d\t%d\t%d", kib["go_heap"], kib["go_other"], kib["anon_other"], kib["file"] }
+ ' "$1"
+}
+
+run_tree() {
+ local label=$1 port=$2
+ local data="${OUT_DIR}/data-${label}" log="${OUT_DIR}/run-${label}.log"
+ local samples="${OUT_DIR}/samples-${label}.tsv"
+ rm -rf "${data}" && mkdir -p "${data}"
+ env -i HOME="${HOME}" PATH="${PATH}" \
+ PULSE_MOCK_MODE=true PULSE_MOCK_NODES=2 PULSE_MOCK_VMS_PER_NODE=6 PULSE_MOCK_LXCS_PER_NODE=6 \
+ PULSE_MOCK_DOCKER_HOSTS=1 PULSE_MOCK_DOCKER_CONTAINERS=10 PULSE_MOCK_GENERIC_HOSTS=1 \
+ PULSE_MOCK_K8S_CLUSTERS=0 \
+ PULSE_DATA_DIR="${data}" PORT="${port}" FRONTEND_PORT="${port}" \
+ PULSE_DEV=true ALLOW_ADMIN_BYPASS=1 PULSE_DEV_DISABLE_BACKGROUND_AI=true \
+ PULSE_TELEMETRY=false PULSE_LICENSE_SERVER_URL=http://127.0.0.1:9 \
+ GOMAXPROCS=1 \
+ "${OUT_DIR}/pulse-${label}" > "${log}" 2>&1 &
+ local pid=$!
+ printf 'elapsed_s\trss_kib\tgo_heap_kib\tgo_other_kib\tanon_other_kib\tfile_kib\n' > "${samples}"
+ local start now
+ start=$(date +%s)
+ while kill -0 "${pid}" 2>/dev/null; do
+ now=$(( $(date +%s) - start ))
+ (( now >= DURATION )) && break
+ local rss
+ rss=$(awk '/^VmRSS:/ {print $2}' "/proc/${pid}/status" 2>/dev/null || true)
+ if [[ -n "${rss}" ]]; then
+ printf '%s\t%s\t%s\n' "${now}" "${rss}" "$(classify_smaps "/proc/${pid}/smaps")" >> "${samples}"
+ fi
+ sleep "${INTERVAL}"
+ done
+ if kill -0 "${pid}" 2>/dev/null; then
+ curl -fsS -o "${OUT_DIR}/heap-${label}.pb.gz" "http://127.0.0.1:${port}/debug/pprof/heap" || true
+ kill "${pid}" 2>/dev/null || true
+ wait "${pid}" 2>/dev/null || true
+ else
+ echo "${label} exited before the measurement ended; see ${log}" >&2
+ return 1
+ fi
+}
+
+build_tree baseline "${BASELINE_DIR}"
+build_tree candidate "${CANDIDATE_DIR}"
+
+# Run both at once so they share host conditions. Only one can bind the fixed
+# Prometheus port; the other logs a bind error, which costs no measurable memory.
+run_tree baseline 7801 & baseline_pid=$!
+run_tree candidate 7802 & candidate_pid=$!
+status=0
+wait "${baseline_pid}" || status=1
+wait "${candidate_pid}" || status=1
+(( status == 0 )) || exit 1
+
+if [[ -s "${OUT_DIR}/heap-baseline.pb.gz" && -s "${OUT_DIR}/heap-candidate.pb.gz" ]]; then
+ (cd "${CANDIDATE_DIR}" && go tool pprof -sample_index=inuse_space -top -nodecount=25 \
+ -base "${OUT_DIR}/heap-baseline.pb.gz" "${OUT_DIR}/heap-candidate.pb.gz") \
+ > "${OUT_DIR}/heap-growth.txt" 2>/dev/null || true
+fi
+
+python3 "${SCRIPT_DIR}/summarize_release_memory.py" "${OUT_DIR}"
diff --git a/scripts/memory/summarize_release_memory.py b/scripts/memory/summarize_release_memory.py
new file mode 100644
index 0000000000..13ed82b9b6
--- /dev/null
+++ b/scripts/memory/summarize_release_memory.py
@@ -0,0 +1,97 @@
+#!/usr/bin/env python3
+"""Compare steady-state memory from compare_release_memory.sh samples (#1803).
+
+Reads samples-baseline.tsv and samples-candidate.tsv from the output
+directory, takes the median of each column over the final third of each run,
+writes summary.md and result.json, and exits 1 when the candidate's resident
+memory exceeds the baseline by more than both the percentage tolerance and the
+absolute slack. Requiring both keeps small installs' noise from failing it.
+"""
+
+from __future__ import annotations
+
+import csv
+import json
+import os
+import statistics
+import sys
+from pathlib import Path
+
+COLUMNS = ("rss_kib", "go_heap_kib", "go_other_kib", "anon_other_kib", "file_kib")
+LABELS = {
+ "rss_kib": "Resident",
+ "go_heap_kib": "Go heap",
+ "go_other_kib": "Go runtime",
+ "anon_other_kib": "Outside Go (SQLite and C allocations)",
+ "file_kib": "File pages",
+}
+
+
+def steady_state(path: Path) -> dict[str, float]:
+ with path.open(newline="") as handle:
+ rows = [row for row in csv.DictReader(handle, delimiter="\t") if row.get("rss_kib")]
+ if len(rows) < 3:
+ raise SystemExit(f"{path.name}: only {len(rows)} samples; the run ended early")
+ last = int(rows[-1]["elapsed_s"])
+ tail = [row for row in rows if int(row["elapsed_s"]) >= last * 2 // 3] or rows[-3:]
+ return {column: statistics.median(float(row[column]) for row in tail) for column in COLUMNS}
+
+
+def main(argv: list[str]) -> int:
+ if len(argv) != 2:
+ print(__doc__, file=sys.stderr)
+ return 2
+ out = Path(argv[1])
+ tolerance = float(os.environ.get("PULSE_MEMORY_TOLERANCE_PCT", "15"))
+ slack_kib = float(os.environ.get("PULSE_MEMORY_SLACK_MIB", "16")) * 1024
+ baseline = steady_state(out / "samples-baseline.tsv")
+ candidate = steady_state(out / "samples-candidate.tsv")
+
+ growth = candidate["rss_kib"] - baseline["rss_kib"]
+ growth_pct = 100 * growth / baseline["rss_kib"] if baseline["rss_kib"] else 0.0
+ regressed = growth > slack_kib and growth_pct > tolerance
+
+ lines = [
+ "## Pulse memory: candidate against baseline",
+ "",
+ "Steady state is the median over the final third of each run, on the fixed small mock estate.",
+ "",
+ "| | Baseline MiB | Candidate MiB | Change |",
+ "|---|---:|---:|---:|",
+ ]
+ for column in COLUMNS:
+ base, cand = baseline[column] / 1024, candidate[column] / 1024
+ lines.append(f"| {LABELS[column]} | {base:.1f} | {cand:.1f} | {cand - base:+.1f} |")
+ verdict = (
+ f"**Regression:** resident memory grew {growth / 1024:+.1f} MiB ({growth_pct:+.1f}%), "
+ f"past both {tolerance:g}% and {slack_kib / 1024:g} MiB."
+ if regressed
+ else f"No regression: resident memory changed {growth / 1024:+.1f} MiB ({growth_pct:+.1f}%)."
+ )
+ lines += ["", verdict]
+ growth_file = out / "heap-growth.txt"
+ if growth_file.exists() and growth_file.stat().st_size:
+ top = growth_file.read_text().splitlines()[:30]
+ lines += ["", "Heap growth by function (candidate minus baseline, in-use space):", "", "```", *top, "```"]
+ lines += [
+ "",
+ "Mock mode keeps its own fixtures in the heap, so check the growth list for `internal/mock` before",
+ "attributing a change to production code.",
+ ]
+ summary = "\n".join(lines) + "\n"
+ (out / "summary.md").write_text(summary)
+ (out / "result.json").write_text(json.dumps({
+ "baseline": baseline,
+ "candidate": candidate,
+ "rss_growth_kib": growth,
+ "rss_growth_pct": growth_pct,
+ "tolerance_pct": tolerance,
+ "slack_kib": slack_kib,
+ "regressed": regressed,
+ }, indent=2) + "\n")
+ print(summary)
+ return 1 if regressed else 0
+
+
+if __name__ == "__main__":
+ sys.exit(main(sys.argv))