Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Run the absolute chart SLO independently of baseline context
rcourtman0a6eb9b541f1
diff --git a/.github/workflows/build-and-test.yml b/.github/workflows/build-and-test.ymlindex 0707265940..cbd6b1a9d8 100644--- a/.github/workflows/build-and-test.yml+++ b/.github/workflows/build-and-test.yml@@ -772,13 +772,6 @@ jobs:with:persist-credentials: false- - name: Checkout pull request base- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1- with:- persist-credentials: false- ref: ${{ github.event.pull_request.base.sha }}- path: chart-baseline-- name: Set up Gouses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0with:@@ -788,17 +781,15 @@ jobs:- name: Check idle proof guardsrun: python3 -m unittest discover -s internal/api/testdata -p test_chart_latency_proof.py -v- - name: Compile fixed baseline and candidate executables+ - name: Compile fixed candidate executablerun: |set -euo pipefailproof_dir="$RUNNER_TEMP/chart-latency"mkdir -p "$proof_dir"- for source in chart-baseline .; do- mkdir -p "$source/internal/api/frontend-modern/dist"- [ -f "$source/internal/api/frontend-modern/dist/index.html" ] || \- printf '<!doctype html><title>ci embed stub</title>\n' \- > "$source/internal/api/frontend-modern/dist/index.html"- done+ mkdir -p internal/api/frontend-modern/dist+ [ -f internal/api/frontend-modern/dist/index.html ] || \+ printf '<!doctype html><title>ci embed stub</title>\n' \+ > internal/api/frontend-modern/dist/index.html{date -u +%FT%TZgo version@@ -806,14 +797,11 @@ jobs:lscpuprintf 'GOMAXPROCS=%s\nGITHUB_RUN_ID=%s\nGITHUB_RUN_ATTEMPT=%s\n' \"$GOMAXPROCS" "$GITHUB_RUN_ID" "$GITHUB_RUN_ATTEMPT"- printf 'baseline HEAD/tree:\n'- git -C chart-baseline rev-parse HEAD HEAD^{tree}printf 'candidate HEAD/tree:\n'git rev-parse HEAD HEAD^{tree}printf 'candidate source changes:\n'git diff --exit-code} > "$proof_dir/metadata.txt"- (cd chart-baseline && go test -p 1 ./internal/api -run '^$' -c -o "$proof_dir/baseline-api.test")go test -p 1 ./internal/api -run '^$' -c -o "$proof_dir/candidate-api.test"sha256sum "$proof_dir/"*.test >> "$proof_dir/metadata.txt"@@ -823,7 +811,6 @@ jobs:- name: Verify chart latency on observed idle capacityrun: |python3 internal/api/testdata/chart_latency_proof.py \- --baseline "$RUNNER_TEMP/chart-latency/baseline-api.test" \--candidate "$RUNNER_TEMP/chart-latency/candidate-api.test" \--output "$RUNNER_TEMP/chart-latency/samples"diff --git a/docs/release-control/v6/internal/records/workload-chart-latency-2026-10-18.md b/docs/release-control/v6/internal/records/workload-chart-latency-2026-10-18.mdindex 1f30fcb715..63de58fddf 100644--- a/docs/release-control/v6/internal/records/workload-chart-latency-2026-10-18.md+++ b/docs/release-control/v6/internal/records/workload-chart-latency-2026-10-18.md@@ -50,9 +50,9 @@ Native raw evidence is retained under## Absolute SLO verificationThe optional `Workload chart 90ms SLO` job in Build and Test is the repeatable-proof surface. A fresh hosted VM compiles base and candidate API test-executables before timing, records their committed HEAD/tree and binary-hashes, and observes idle capacity before each process. CPU, guest, swap and+proof surface. A fresh hosted VM compiles the candidate API test executable+before timing, records its committed HEAD/tree and binary hash, and observes+idle capacity before the process. CPU, guest, swap andpressure samples remain in the `chart-latency` artifact alongside raw outputand post-measurement samples. The candidate must execute exactly one fresh200-sample workload-chart test and report the existing 90 ms budget; skipped@@ -65,3 +65,26 @@ diagnostic benchmark mean for those percentiles. The earlier historical12.72% comparison remains preserved; changed harness boundaries cannotretroactively resolve it. Broader performance headroom remains governed bythe existing L10 follow-up.++## Retained adverse evidence++The initial hosted attempt at head `2a77054035513eabc884135d3a8c19fc76e0956b`+passed its pre-run idle window, then observed 25.5% and 24.6% competing CPU+during the baseline process. Its 138.45634 ms p95 used the ordinary 550 ms+envelope and is not a valid idle-worker result. The candidate produced zero+samples because the initial proof stopped on that context failure. See+[job 117757206920](https://github.com/rcourtman/Pulse/actions/runs/39153766278/job/117757206920)+and [raw artifact 12147832177](https://github.com/rcourtman/Pulse/actions/runs/39153766278/artifacts/12147832177).++Absolute verification now runs the candidate directly; the existing paired+benchmark job retains relative context. The sampler also retains per-process+CPU names/counters and the measurement's I/O counters. It requires quiet I/O+before admission and keeps the test's own I/O in its measured latency rather+than classifying every I/O wait as an unrelated workload.++A separate snapshot-safe fleet-tier-pruning experiment passed the full metrics+suite. Ten alternating current-source comparisons measured 196.1 ms versus+179.8 ms (p=0.063), with 0.20% more bytes and 0.07% more allocations. This+shared-worker result is inconclusive; the experiment was reverted and its raw+samples remain under the native proof directory. It establishes no speedup or+SLO result.diff --git a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md b/docs/release-control/v6/internal/subsystems/performance-and-scalability.mdindex 80a587dce0..9e48fe961a 100644--- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md+++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md@@ -1873,11 +1873,17 @@ dashboard-specific overview, trend, or summary transport.enforce the existing 90 ms local budget, including on hosted runners,without changing their actual runner identity or accepting a skippedtest. Ordinary hosted execution retains its 550 ms envelope. The optional- Build and Test chart-latency job compiles fixed base/candidate executables+ Build and Test chart-latency job compiles a fixed candidate executablebefore measuring, requires observed idle capacity, and retains executablehashes, raw percentiles, and CPU/guest/swap evidence before, during and- after measurement. Its base timing is context under the ordinary envelope;- only the candidate's explicit 90 ms report establishes the local SLO.+ after measurement. It executes the candidate directly; relative baseline+ comparisons belong to the separate paired benchmark job. A baseline+ failure cannot suppress the candidate's absolute check. The proof requires+ quiet I/O before starting; the measured process's own I/O remains part of+ its latency, with I/O wait and process I/O counters retained for diagnosis.+ Competing CPU, guest/steal or swap activity invalidates the measurement,+ and process CPU counters identify the largest observed background users.+ Only the explicit 90 ms report establishes the local SLO.This evidence does not replace required source checks or release gates.31. Keep the retired dashboard overview route absent from the protected hotpath. `frontend-modern/src/pages/Dashboard.tsx`,diff --git a/internal/api/testdata/chart_latency_proof.py b/internal/api/testdata/chart_latency_proof.pyindex 7741f2811a..1d414dc5fb 100644--- a/internal/api/testdata/chart_latency_proof.py+++ b/internal/api/testdata/chart_latency_proof.py@@ -69,52 +69,103 @@ def ticks(pid):return None+def process_counters(root=Path('/proc')):+ counters = {}+ for path in root.iterdir():+ if not path.name.isdigit():+ continue+ try:+ name, tail = (path / 'stat').read_text().rsplit(')', 1)+ fields = tail.split()+ counters[int(path.name)] = dict(name=name.split('(', 1)[1],+ ticks=int(fields[11]) + int(fields[12]),+ start=fields[19])+ except (OSError, ValueError, IndexError):+ continue+ return counters+++def background_processes(before, after, measurement_pid, total_ticks):+ rows = []+ if total_ticks <= 0:+ return rows+ for pid, current in after.items():+ if pid == measurement_pid:+ continue+ previous = before.get(pid, {})+ previous_ticks = previous.get('ticks', 0) if previous.get('start') == current['start'] else 0+ delta = current['ticks'] - previous_ticks+ if delta > 0:+ rows.append(dict(pid=pid, name=current['name'], cpu_pct=100 * delta / total_ticks))+ return sorted(rows, key=lambda row: row['cpu_pct'], reverse=True)[:5]+++def process_io(pid):+ try:+ fields = dict(line.split(': ') for line in Path(f'/proc/{pid}/io').read_text().splitlines())+ return {key: int(fields[key]) for key in ('read_bytes', 'write_bytes')}+ except (OSError, ValueError, KeyError):+ return None+++def competing_capacity(row):+ # The measured SQLite process performs I/O. Keep I/O wait in the raw+ # receipt and latency, rather than mistaking its own reads for contention.+ # Admission still requires quiet I/O before the process starts.+ return (row['background_busy_pct'] > 10 or row['guest_pct'] > 2+ or row['steal_pct'] > 0.1 or row['swap_pages'] != 0)++def digest(path):with path.open('rb') as source:return hashlib.file_digest(source, 'sha256').hexdigest()-def validate_output(text, returncode, strict):+def validate_output(text, returncode):if returncode != 0:raise RuntimeError(f'SLO process exited {returncode}')passes = re.findall(r'^--- PASS: TestSLO_WorkloadCharts \(', text, re.MULTILINE)if len(passes) != 1 or re.search(r'^--- SKIP: TestSLO_WorkloadCharts ', text, re.MULTILINE):raise RuntimeError('SLO did not pass, including a skipped or missing test')reports = re.findall(r'charts/workloads p50=\S+ p95=\S+ p99=\S+ SLO=(\S+)', text)- if len(reports) != 1 or (strict and reports[0] != '90ms'):+ if len(reports) != 1 or reports[0] != '90ms':raise RuntimeError(f'Expected one explicit SLO report at the selected budget, got {reports}')-def measure(binary, prefix, strict):+def measure(binary, prefix):identity = digest(binary)wait_idle(prefix)# Preserve the real Actions identity, without forwarding runner credentials.env = {key: os.environ[key] for key in ('PATH', 'HOME', 'TMPDIR', 'TZ', 'GOMAXPROCS', 'CI','GITHUB_ACTIONS', 'GITHUB_RUN_ID', 'GITHUB_RUN_ATTEMPT') if key in os.environ}- if strict:- env['PULSE_STRICT_LOCAL_API_SLOS'] = 'true'+ env['PULSE_STRICT_LOCAL_API_SLOS'] = 'true'command = [str(binary), '-test.run=^TestSLO_WorkloadCharts$', '-test.v','-test.count=1', '-test.cpu=2', '-test.timeout=5m']capacity_errors = []with prefix.with_suffix('.log').open('w') as log, prefix.with_suffix('.cpu.jsonl').open('w') as receipt:before = snapshot()+ processes_before = process_counters()process = subprocess.Popen(command, env=env, stdout=log, stderr=subprocess.STDOUT)own_before = 0while process.poll() is None:time.sleep(1)after = snapshot()+ processes_after = process_counters()own_after = ticks(process.pid)row = interval(before, after)+ row['measurement_io_bytes'] = process_io(process.pid)+ row['background_processes'] = background_processes(+ processes_before, processes_after, process.pid, row['total_ticks'])if own_before is not None and own_after is not None and row['total_ticks'] > 0:row['measurement_cpu_pct'] = 100 * (own_after - own_before) / row['total_ticks']row['background_busy_pct'] = max(0, row['busy_pct'] - row['measurement_cpu_pct'])- if (row['background_busy_pct'] > 10 or row['guest_pct'] > 2- or row['steal_pct'] > 0.1 or row['iowait_pct'] > 1 or row['swap_pages']):+ if competing_capacity(row):capacity_errors.append(row)receipt.write(json.dumps(row) + '\n')receipt.flush()before, own_before = after, own_after+ processes_before = processes_afterreturncode = process.wait()with prefix.with_suffix('.after.jsonl').open('w') as receipt:before = snapshot()@@ -130,22 +181,20 @@ def measure(binary, prefix, strict):print(text, flush=True)if digest(binary) != identity:raise RuntimeError('Measured executable changed')- validate_output(text, returncode, strict)+ validate_output(text, returncode)if capacity_errors:raise RuntimeError('Background CPU or guest/steal activity invalidated the idle measurement')def main():parser = argparse.ArgumentParser(description=__doc__)- parser.add_argument('--baseline', type=Path, required=True)parser.add_argument('--candidate', type=Path, required=True)parser.add_argument('--output', type=Path, required=True)args = parser.parse_args()args.output.mkdir(parents=True, exist_ok=False)- # Baseline uses its ordinary hosted envelope only as timing context. The- # candidate must pass the existing local 90 ms budget without any skip.- measure(args.baseline.resolve(), args.output / 'baseline', strict=False)- measure(args.candidate.resolve(), args.output / 'candidate', strict=True)+ # Absolute qualification executes the candidate directly. The separate+ # paired benchmark job owns relative comparisons and baseline context.+ measure(args.candidate.resolve(), args.output / 'candidate')if __name__ == '__main__':diff --git a/internal/api/testdata/test_chart_latency_proof.py b/internal/api/testdata/test_chart_latency_proof.pyindex 677e82c56e..5998c0cec5 100644--- a/internal/api/testdata/test_chart_latency_proof.py+++ b/internal/api/testdata/test_chart_latency_proof.py@@ -25,14 +25,36 @@ class ChartLatencyProofTests(unittest.TestCase):def test_skip_or_hosted_pass_cannot_satisfy_local_slo(self):report = 'charts/workloads p50=40ms p95=50ms p99=60ms SLO=90ms\n'passed = report + '--- PASS: TestSLO_WorkloadCharts (1.0s)\n'- proof.validate_output(passed, 0, strict=True)+ proof.validate_output(passed, 0)for text, status in [(report + '--- SKIP: TestSLO_WorkloadCharts (1.0s)\n', 0),(passed.replace('90ms', '550ms'), 0), (passed, 1), ('PASS\n', 0),(passed + report, 0), (passed + passed, 0),(passed + '--- SKIP: TestSLO_WorkloadCharts (1.0s)\n', 0)]:with self.subTest(text=text, status=status):with self.assertRaises(RuntimeError):- proof.validate_output(text, status, strict=True)+ proof.validate_output(text, status)++ def test_measurement_io_stays_in_latency_and_background_cpu_invalidates(self):+ row = dict(background_busy_pct=0, guest_pct=0, steal_pct=0, swap_pages=0,+ iowait_pct=20)+ self.assertFalse(proof.competing_capacity(row))+ for field, value in [('background_busy_pct', 25), ('guest_pct', 3),+ ('steal_pct', 1), ('swap_pages', 1)]:+ with self.subTest(field=field):+ self.assertTrue(proof.competing_capacity(dict(row, **{field: value})))++ def test_process_receipt_excludes_measurement_and_handles_new_or_reused_pids(self):+ before = {10: dict(name='test', ticks=20, start='1'),+ 20: dict(name='other', ticks=30, start='2'),+ 30: dict(name='old', ticks=100, start='3')}+ after = {10: dict(name='test', ticks=200, start='1'),+ 20: dict(name='other', ticks=55, start='2'),+ 30: dict(name='new', ticks=10, start='4'),+ 40: dict(name='new process', ticks=5, start='5')}+ self.assertEqual(proof.background_processes(before, after, 10, 100), [+ dict(pid=20, name='other', cpu_pct=25),+ dict(pid=30, name='new', cpu_pct=10),+ dict(pid=40, name='new process', cpu_pct=5)])if __name__ == '__main__':