Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Avoid duplicate extrema decoding in retained metric reads

rcourtmanea3d71a79907
diff --git a/docs/release-control/v6/internal/records/workload-chart-latency-2026-10-18.md b/docs/release-control/v6/internal/records/workload-chart-latency-2026-10-18.md
index 63de58fddf..c328ce911f 100644
--- a/docs/release-control/v6/internal/records/workload-chart-latency-2026-10-18.md
+++ b/docs/release-control/v6/internal/records/workload-chart-latency-2026-10-18.md
@@ -88,3 +88,45 @@ suite. Ten alternating current-source comparisons measured 196.1 ms versus
shared-worker result is inconclusive; the experiment was reverted and its raw
samples remain under the native proof directory. It establishes no speedup or
SLO result.
+
+
+## Follow-up decoding cost and candidate evidence
+
+The candidate-only proof at head `0a6eb9b541f12a9dd6cb64caa7c231355426077e`
+executed the strict 90 ms test: p50 106.672837 ms, p95 130.023547 ms and p99
+144.836677 ms. It failed the budget. The retained capacity receipt also
+invalidates idle qualification: `provjobd4515890` consumed 18.7–25.1% of the
+four-logical-CPU worker during three intervals, with competing busy CPU
+18.7–25.4%. See [job 117769795027](https://github.com/rcourtman/Pulse/actions/runs/39159417029/job/117769795027)
+and [raw artifact 12148113764](https://github.com/rcourtman/Pulse/actions/runs/39159417029/artifacts/12148113764).
+Neither the failed budget report nor idle admission alone establishes an
+idle-worker SLO result. Native capacity remained unsuitable as well (load
+above 15 and sustained I/O wait at 19:33 UTC). No provider or native workload
+was stopped and no repeated measurement was selected for a passing result.
+
+The original paired hosted comparison measured cold-service cost from
+148.8 to 133.6 ms/op (-10.20%, p<0.001, n=10); warm-handler and HTTP-transfer
+changes were not significant. [Job 117757206872](https://github.com/rcourtman/Pulse/actions/runs/39153766278/job/117757206872)
+failed on two unchanged test-only legacy history-filter cases, which exercise
+neither SQL nor collectors. These relative results do not establish 90 ms.
+
+A second retained-read improvement avoids decoding duplicate floats when
+stored extrema are NULL. Non-aggregated SQL now returns nullable bounds;
+Go applies the same per-observation value fallback. SQL aggregation and
+streamed buckets retain their original extrema semantics, including explicit
+zero and independently missing bounds. The full metrics suite passed in
+95.727 seconds. The final strengthened mixed-tier nullable guard passed all
+twelve read-shape cases in 0.024 seconds. The first new test assertion incorrectly compared time-zone representations; it was
+corrected to compare timestamp instants, and its failed log is preserved.
+
+Ten alternating base/candidate diagnostic samples on the shared native
+worker measured 142.3 versus 136.9 ms/op (p=0.579), so no latency gain is
+established. Allocated bytes fell 4.24% (29.85 to 28.58 MiB/op) and allocations
+fell 24.59% (674.6k to 508.7k/op), both p<0.001. This change is retained for
+its measured allocation reduction. Executable hashes are
+`737f1383deecd2d34e20f49731e1a634a690e8de789bb0221bd63a44b3441b70`
+(base) and `843ea5877f711966fdbdf590854b5f3dca96319c5a9d4cd1f5f2fa3b03f98856`
+(candidate); raw samples and metadata are retained as `nullable-*` under the
+native proof directory. This is not an idle-worker result. The strict proof
+for the resulting source remains the source-check run's retained artifact;
+the 90 ms verification and broader L10 headroom follow-up remain open.
diff --git a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md
index 9e48fe961a..ef19fe5b1b 100644
--- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md
+++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md
@@ -1883,7 +1883,7 @@ dashboard-specific overview, trend, or summary transport.
its latency, with I/O wait and process I/O counters retained for diagnosis.
Competing CPU, guest/steal or swap activity invalidates the measurement,
and process CPU counters identify the largest observed background users.
- Only the explicit 90 ms report establishes the local SLO.
+ Only a passing explicit 90 ms report on valid capacity establishes the local SLO.
This evidence does not replace required source checks or release gates.
31. Keep the retired dashboard overview route absent from the protected hot
path. `frontend-modern/src/pages/Dashboard.tsx`,
@@ -2706,7 +2706,14 @@ series' timestamp order, tier precedence, values and extrema. Streaming
display-bucket aggregation retains its required series grouping. The query
plan guard covers filtered and unfiltered reads across retained windows and
single/fleet scopes; tier-coverage tests compare filtered results with the
-unfiltered reconciliation contract. Legacy databases keep their
+unfiltered reconciliation contract. Non-aggregated rows preserve nullable
+extrema across the SQLite driver boundary; decoding applies each missing
+bound's value fallback per observation while retaining explicit bounds,
+including zero. SQL display aggregation applies the same fallback before
+MIN/MAX, and streamed buckets receive normalized bounds. Avoid materializing
+the same float three times for observations without stored extrema. Mixed
+nullable-bound tests cover single/fleet reads, metric filters and display
+buckets. Legacy databases keep their
existing unique index authoritative while the replacement index is built, but
that O(rows) migration must run on deferred startup maintenance and swap indexes
in one SQLite transaction so constructor latency, concurrent reads, uniqueness,
diff --git a/internal/api/testdata/chart_latency_proof.py b/internal/api/testdata/chart_latency_proof.py
index 1d414dc5fb..9b12bac4c9 100644
--- a/internal/api/testdata/chart_latency_proof.py
+++ b/internal/api/testdata/chart_latency_proof.py
@@ -181,9 +181,10 @@ def measure(binary, prefix):
print(text, flush=True)
if digest(binary) != identity:
raise RuntimeError('Measured executable changed')
- validate_output(text, returncode)
if capacity_errors:
- raise RuntimeError('Background CPU or guest/steal activity invalidated the idle measurement')
+ raise RuntimeError('Background CPU or guest/steal activity invalidated the idle measurement '
+ f'(SLO process exit {returncode}; see raw log)')
+ validate_output(text, returncode)
def main():
diff --git a/pkg/metrics/store.go b/pkg/metrics/store.go
index 77e00a927f..9a9dc479c8 100644
--- a/pkg/metrics/store.go
+++ b/pkg/metrics/store.go
@@ -1657,8 +1657,10 @@ func retainedQuerySQL(resourceType string, resourceIDs, metricTypes []string, st
}
return clause + fmt.Sprintf(" AND %s.tier = :p%d AND %s.timestamp >= :p%d AND %s.timestamp <= :p%d", alias, tierParam+tierIndex, alias, startParam, alias, endParam)
}
- projection := identityColumns + `m.timestamp, m.value,
- COALESCE(m.min_value, m.value) AS min_value, COALESCE(m.max_value, m.value) AS max_value`
+ // Preserve nullable extrema across the row boundary. Materializing the
+ // value again for each missing bound makes SQLite and its driver decode
+ // three floats for an observation that stores only one.
+ projection := identityColumns + `m.timestamp, m.value, m.min_value, m.max_value`
directAggregate := stepSecs > 1 && len(tiers) == 1
bucketExpression := fmt.Sprintf("(timestamp / :p%d) * :p%d + (:p%d / 2)", stepParam, stepParam, stepParam)
if directAggregate {
@@ -1692,7 +1694,7 @@ func retainedQuerySQL(resourceType string, resourceIDs, metricTypes []string, st
query += " GROUP BY " + identityColumns + "bucket_ts ORDER BY " + identityColumns + "bucket_ts ASC"
} else if stepSecs > 1 {
query = "SELECT " + identityColumns + bucketExpression + ` AS bucket_ts,
- AVG(value), MIN(min_value), MAX(max_value)
+ AVG(value), MIN(COALESCE(min_value, value)), MAX(COALESCE(max_value, value))
FROM (` + query + ") GROUP BY " + identityColumns + "bucket_ts ORDER BY " + identityColumns + "bucket_ts ASC"
} else {
orderColumns := identityColumns
@@ -1907,6 +1909,7 @@ func (s *Store) queryRetainedChunk(resourceType string, resourceIDs []string, me
var resourceID, metricType string
var ts int64
var p MetricPoint
+ var low, high sql.NullFloat64
destinations := make([]interface{}, 0, 6)
if len(resourceIDs) == 1 {
resourceID = resourceIDs[0]
@@ -1918,12 +1921,19 @@ func (s *Store) queryRetainedChunk(resourceType string, resourceIDs []string, me
} else {
destinations = append(destinations, &metricType)
}
- destinations = append(destinations, &ts, &p.Value, &p.Min, &p.Max)
+ destinations = append(destinations, &ts, &p.Value, &low, &high)
for rows.Next() {
if err := rows.Scan(destinations...); err != nil {
log.Warn().Err(err).Msg("Failed to scan batch metric row")
continue
}
+ p.Min, p.Max = p.Value, p.Value
+ if low.Valid {
+ p.Min = low.Float64
+ }
+ if high.Valid {
+ p.Max = high.Float64
+ }
if !streamBuckets {
p.Timestamp = time.Unix(ts, 0)
appendPoint(resourceID, metricType, p)
diff --git a/pkg/metrics/store_tier_coverage_test.go b/pkg/metrics/store_tier_coverage_test.go
index fb6e688a10..b054b40d4d 100644
--- a/pkg/metrics/store_tier_coverage_test.go
+++ b/pkg/metrics/store_tier_coverage_test.go
@@ -277,3 +277,84 @@ func TestRetainedQueryTemplatesBindCurrentWindowAndStep(t *testing.T) {
}
}
}
+
+// Extrema can be missing independently, including after a row with explicit
+// zero bounds. Every retained read shape must apply the fallback per point.
+func TestStoreRetainedNullableExtrema(t *testing.T) {
+ db := newPlanTestDB(t)
+ store := &Store{db: pdb.Wrap(db, "nullable-extrema-test")}
+ base := time.Date(2026, 9, 5, 12, 0, 0, 0, time.UTC)
+ type observation struct {
+ value float64
+ low, high interface{}
+ wantLow, wantHigh float64
+ }
+ observations := []observation{
+ {10, -2.0, 99.0, -2, 99},
+ {20, nil, nil, 20, 20},
+ {30, 0.0, 0.0, 0, 0},
+ {-10, nil, -3.0, -10, -3},
+ {50, 2.0, nil, 2, 50},
+ {60, nil, nil, 60, 60},
+ }
+ for _, id := range []string{"a", "b"} {
+ for _, metric := range []string{"cpu", "memory"} {
+ for i, row := range observations {
+ // Alternate non-overlapping tiers so single-resource buckets must
+ // also normalize NULL bounds across the reconciliation UNION.
+ tier := TierMinute
+ if i%2 != 0 {
+ tier = TierRaw
+ }
+ if _, err := db.Exec(`INSERT INTO metrics(resource_type,resource_id,metric_type,tier,timestamp,value,min_value,max_value) VALUES ('node',?,?,?,?,?,?,?)`, id, metric, tier, base.Add(time.Duration(i)*time.Minute).Unix(), row.value, row.low, row.high); err != nil {
+ t.Fatal(err)
+ }
+ }
+ }
+ }
+ for _, step := range []int64{0, 120} {
+ for _, ids := range [][]string{{"a"}, {"a", "b"}} {
+ for _, filter := range [][]string{nil, {"cpu"}, {"cpu", "memory"}} {
+ t.Run(fmt.Sprintf("step=%d/ids=%d/filter=%d", step, len(ids), len(filter)), func(t *testing.T) {
+ got, err := store.QueryMetricTypesBatch("node", ids, filter, base.Add(-2*time.Hour), base.Add(6*time.Minute), step)
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, id := range ids {
+ metrics := filter
+ if len(metrics) == 0 {
+ metrics = []string{"cpu", "memory"}
+ }
+ for _, metric := range metrics {
+ want := make([]MetricPoint, 0, len(observations))
+ for i := 0; i < len(observations); {
+ row := observations[i]
+ point := MetricPoint{Timestamp: base.Add(time.Duration(i) * time.Minute), Value: row.value, Min: row.wantLow, Max: row.wantHigh}
+ i++
+ if step > 1 {
+ next := observations[i]
+ point.Timestamp = point.Timestamp.Add(time.Minute)
+ point.Value = (point.Value + next.value) / 2
+ point.Min = min(point.Min, next.wantLow)
+ point.Max = max(point.Max, next.wantHigh)
+ i++
+ }
+ want = append(want, point)
+ }
+ points := got[id][metric]
+ if len(points) != len(want) {
+ t.Fatalf("%s/%s: got %+v, want %+v", id, metric, points, want)
+ }
+ for i, point := range points {
+ expected := want[i]
+ if !point.Timestamp.Equal(expected.Timestamp) || point.Value != expected.Value || point.Min != expected.Min || point.Max != expected.Max {
+ t.Fatalf("%s/%s point %d: got %+v, want %+v", id, metric, i, point, expected)
+ }
+ }
+ }
+ }
+ })
+ }
+ }
+ }
+}