Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Avoid duplicate extrema decoding in retained metric reads
rcourtmanea3d71a79907
diff --git a/docs/release-control/v6/internal/records/workload-chart-latency-2026-10-18.md b/docs/release-control/v6/internal/records/workload-chart-latency-2026-10-18.mdindex 63de58fddf..c328ce911f 100644--- a/docs/release-control/v6/internal/records/workload-chart-latency-2026-10-18.md+++ b/docs/release-control/v6/internal/records/workload-chart-latency-2026-10-18.md@@ -88,3 +88,45 @@ suite. Ten alternating current-source comparisons measured 196.1 ms versusshared-worker result is inconclusive; the experiment was reverted and its rawsamples remain under the native proof directory. It establishes no speedup orSLO result.+++## Follow-up decoding cost and candidate evidence++The candidate-only proof at head `0a6eb9b541f12a9dd6cb64caa7c231355426077e`+executed the strict 90 ms test: p50 106.672837 ms, p95 130.023547 ms and p99+144.836677 ms. It failed the budget. The retained capacity receipt also+invalidates idle qualification: `provjobd4515890` consumed 18.7–25.1% of the+four-logical-CPU worker during three intervals, with competing busy CPU+18.7–25.4%. See [job 117769795027](https://github.com/rcourtman/Pulse/actions/runs/39159417029/job/117769795027)+and [raw artifact 12148113764](https://github.com/rcourtman/Pulse/actions/runs/39159417029/artifacts/12148113764).+Neither the failed budget report nor idle admission alone establishes an+idle-worker SLO result. Native capacity remained unsuitable as well (load+above 15 and sustained I/O wait at 19:33 UTC). No provider or native workload+was stopped and no repeated measurement was selected for a passing result.++The original paired hosted comparison measured cold-service cost from+148.8 to 133.6 ms/op (-10.20%, p<0.001, n=10); warm-handler and HTTP-transfer+changes were not significant. [Job 117757206872](https://github.com/rcourtman/Pulse/actions/runs/39153766278/job/117757206872)+failed on two unchanged test-only legacy history-filter cases, which exercise+neither SQL nor collectors. These relative results do not establish 90 ms.++A second retained-read improvement avoids decoding duplicate floats when+stored extrema are NULL. Non-aggregated SQL now returns nullable bounds;+Go applies the same per-observation value fallback. SQL aggregation and+streamed buckets retain their original extrema semantics, including explicit+zero and independently missing bounds. The full metrics suite passed in+95.727 seconds. The final strengthened mixed-tier nullable guard passed all+twelve read-shape cases in 0.024 seconds. The first new test assertion incorrectly compared time-zone representations; it was+corrected to compare timestamp instants, and its failed log is preserved.++Ten alternating base/candidate diagnostic samples on the shared native+worker measured 142.3 versus 136.9 ms/op (p=0.579), so no latency gain is+established. Allocated bytes fell 4.24% (29.85 to 28.58 MiB/op) and allocations+fell 24.59% (674.6k to 508.7k/op), both p<0.001. This change is retained for+its measured allocation reduction. Executable hashes are+`737f1383deecd2d34e20f49731e1a634a690e8de789bb0221bd63a44b3441b70`+(base) and `843ea5877f711966fdbdf590854b5f3dca96319c5a9d4cd1f5f2fa3b03f98856`+(candidate); raw samples and metadata are retained as `nullable-*` under the+native proof directory. This is not an idle-worker result. The strict proof+for the resulting source remains the source-check run's retained artifact;+the 90 ms verification and broader L10 headroom follow-up remain open.diff --git a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md b/docs/release-control/v6/internal/subsystems/performance-and-scalability.mdindex 9e48fe961a..ef19fe5b1b 100644--- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md+++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md@@ -1883,7 +1883,7 @@ dashboard-specific overview, trend, or summary transport.its latency, with I/O wait and process I/O counters retained for diagnosis.Competing CPU, guest/steal or swap activity invalidates the measurement,and process CPU counters identify the largest observed background users.- Only the explicit 90 ms report establishes the local SLO.+ Only a passing explicit 90 ms report on valid capacity establishes the local SLO.This evidence does not replace required source checks or release gates.31. Keep the retired dashboard overview route absent from the protected hotpath. `frontend-modern/src/pages/Dashboard.tsx`,@@ -2706,7 +2706,14 @@ series' timestamp order, tier precedence, values and extrema. Streamingdisplay-bucket aggregation retains its required series grouping. The queryplan guard covers filtered and unfiltered reads across retained windows andsingle/fleet scopes; tier-coverage tests compare filtered results with the-unfiltered reconciliation contract. Legacy databases keep their+unfiltered reconciliation contract. Non-aggregated rows preserve nullable+extrema across the SQLite driver boundary; decoding applies each missing+bound's value fallback per observation while retaining explicit bounds,+including zero. SQL display aggregation applies the same fallback before+MIN/MAX, and streamed buckets receive normalized bounds. Avoid materializing+the same float three times for observations without stored extrema. Mixed+nullable-bound tests cover single/fleet reads, metric filters and display+buckets. Legacy databases keep theirexisting unique index authoritative while the replacement index is built, butthat O(rows) migration must run on deferred startup maintenance and swap indexesin one SQLite transaction so constructor latency, concurrent reads, uniqueness,diff --git a/internal/api/testdata/chart_latency_proof.py b/internal/api/testdata/chart_latency_proof.pyindex 1d414dc5fb..9b12bac4c9 100644--- a/internal/api/testdata/chart_latency_proof.py+++ b/internal/api/testdata/chart_latency_proof.py@@ -181,9 +181,10 @@ def measure(binary, prefix):print(text, flush=True)if digest(binary) != identity:raise RuntimeError('Measured executable changed')- validate_output(text, returncode)if capacity_errors:- raise RuntimeError('Background CPU or guest/steal activity invalidated the idle measurement')+ raise RuntimeError('Background CPU or guest/steal activity invalidated the idle measurement '+ f'(SLO process exit {returncode}; see raw log)')+ validate_output(text, returncode)def main():diff --git a/pkg/metrics/store.go b/pkg/metrics/store.goindex 77e00a927f..9a9dc479c8 100644--- a/pkg/metrics/store.go+++ b/pkg/metrics/store.go@@ -1657,8 +1657,10 @@ func retainedQuerySQL(resourceType string, resourceIDs, metricTypes []string, st}return clause + fmt.Sprintf(" AND %s.tier = :p%d AND %s.timestamp >= :p%d AND %s.timestamp <= :p%d", alias, tierParam+tierIndex, alias, startParam, alias, endParam)}- projection := identityColumns + `m.timestamp, m.value,- COALESCE(m.min_value, m.value) AS min_value, COALESCE(m.max_value, m.value) AS max_value`+ // Preserve nullable extrema across the row boundary. Materializing the+ // value again for each missing bound makes SQLite and its driver decode+ // three floats for an observation that stores only one.+ projection := identityColumns + `m.timestamp, m.value, m.min_value, m.max_value`directAggregate := stepSecs > 1 && len(tiers) == 1bucketExpression := fmt.Sprintf("(timestamp / :p%d) * :p%d + (:p%d / 2)", stepParam, stepParam, stepParam)if directAggregate {@@ -1692,7 +1694,7 @@ func retainedQuerySQL(resourceType string, resourceIDs, metricTypes []string, stquery += " GROUP BY " + identityColumns + "bucket_ts ORDER BY " + identityColumns + "bucket_ts ASC"} else if stepSecs > 1 {query = "SELECT " + identityColumns + bucketExpression + ` AS bucket_ts,- AVG(value), MIN(min_value), MAX(max_value)+ AVG(value), MIN(COALESCE(min_value, value)), MAX(COALESCE(max_value, value))FROM (` + query + ") GROUP BY " + identityColumns + "bucket_ts ORDER BY " + identityColumns + "bucket_ts ASC"} else {orderColumns := identityColumns@@ -1907,6 +1909,7 @@ func (s *Store) queryRetainedChunk(resourceType string, resourceIDs []string, mevar resourceID, metricType stringvar ts int64var p MetricPoint+ var low, high sql.NullFloat64destinations := make([]interface{}, 0, 6)if len(resourceIDs) == 1 {resourceID = resourceIDs[0]@@ -1918,12 +1921,19 @@ func (s *Store) queryRetainedChunk(resourceType string, resourceIDs []string, me} else {destinations = append(destinations, &metricType)}- destinations = append(destinations, &ts, &p.Value, &p.Min, &p.Max)+ destinations = append(destinations, &ts, &p.Value, &low, &high)for rows.Next() {if err := rows.Scan(destinations...); err != nil {log.Warn().Err(err).Msg("Failed to scan batch metric row")continue}+ p.Min, p.Max = p.Value, p.Value+ if low.Valid {+ p.Min = low.Float64+ }+ if high.Valid {+ p.Max = high.Float64+ }if !streamBuckets {p.Timestamp = time.Unix(ts, 0)appendPoint(resourceID, metricType, p)diff --git a/pkg/metrics/store_tier_coverage_test.go b/pkg/metrics/store_tier_coverage_test.goindex fb6e688a10..b054b40d4d 100644--- a/pkg/metrics/store_tier_coverage_test.go+++ b/pkg/metrics/store_tier_coverage_test.go@@ -277,3 +277,84 @@ func TestRetainedQueryTemplatesBindCurrentWindowAndStep(t *testing.T) {}}}++// Extrema can be missing independently, including after a row with explicit+// zero bounds. Every retained read shape must apply the fallback per point.+func TestStoreRetainedNullableExtrema(t *testing.T) {+ db := newPlanTestDB(t)+ store := &Store{db: pdb.Wrap(db, "nullable-extrema-test")}+ base := time.Date(2026, 9, 5, 12, 0, 0, 0, time.UTC)+ type observation struct {+ value float64+ low, high interface{}+ wantLow, wantHigh float64+ }+ observations := []observation{+ {10, -2.0, 99.0, -2, 99},+ {20, nil, nil, 20, 20},+ {30, 0.0, 0.0, 0, 0},+ {-10, nil, -3.0, -10, -3},+ {50, 2.0, nil, 2, 50},+ {60, nil, nil, 60, 60},+ }+ for _, id := range []string{"a", "b"} {+ for _, metric := range []string{"cpu", "memory"} {+ for i, row := range observations {+ // Alternate non-overlapping tiers so single-resource buckets must+ // also normalize NULL bounds across the reconciliation UNION.+ tier := TierMinute+ if i%2 != 0 {+ tier = TierRaw+ }+ if _, err := db.Exec(`INSERT INTO metrics(resource_type,resource_id,metric_type,tier,timestamp,value,min_value,max_value) VALUES ('node',?,?,?,?,?,?,?)`, id, metric, tier, base.Add(time.Duration(i)*time.Minute).Unix(), row.value, row.low, row.high); err != nil {+ t.Fatal(err)+ }+ }+ }+ }+ for _, step := range []int64{0, 120} {+ for _, ids := range [][]string{{"a"}, {"a", "b"}} {+ for _, filter := range [][]string{nil, {"cpu"}, {"cpu", "memory"}} {+ t.Run(fmt.Sprintf("step=%d/ids=%d/filter=%d", step, len(ids), len(filter)), func(t *testing.T) {+ got, err := store.QueryMetricTypesBatch("node", ids, filter, base.Add(-2*time.Hour), base.Add(6*time.Minute), step)+ if err != nil {+ t.Fatal(err)+ }+ for _, id := range ids {+ metrics := filter+ if len(metrics) == 0 {+ metrics = []string{"cpu", "memory"}+ }+ for _, metric := range metrics {+ want := make([]MetricPoint, 0, len(observations))+ for i := 0; i < len(observations); {+ row := observations[i]+ point := MetricPoint{Timestamp: base.Add(time.Duration(i) * time.Minute), Value: row.value, Min: row.wantLow, Max: row.wantHigh}+ i+++ if step > 1 {+ next := observations[i]+ point.Timestamp = point.Timestamp.Add(time.Minute)+ point.Value = (point.Value + next.value) / 2+ point.Min = min(point.Min, next.wantLow)+ point.Max = max(point.Max, next.wantHigh)+ i+++ }+ want = append(want, point)+ }+ points := got[id][metric]+ if len(points) != len(want) {+ t.Fatalf("%s/%s: got %+v, want %+v", id, metric, points, want)+ }+ for i, point := range points {+ expected := want[i]+ if !point.Timestamp.Equal(expected.Timestamp) || point.Value != expected.Value || point.Min != expected.Min || point.Max != expected.Max {+ t.Fatalf("%s/%s point %d: got %+v, want %+v", id, metric, i, point, expected)+ }+ }+ }+ }+ })+ }+ }+ }+}