Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Read clustered rollups through bounded primary ranges
rcourtmanfc58c597caf0
diff --git a/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md b/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.mdindex 9b938810c5..250df28938 100644--- a/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md+++ b/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md@@ -117,3 +117,79 @@ benchmark reported 111.005964 ms/op, 22,262,176 B/op and 508,181 allocs/op on thshared AMD Ryzen worker. That timing is retained without treating it as a90 ms SLO proof. Status and contract audits passed; the new explicitly routedtest must be staged before the registry audit can recognize its tracked path.++## First paired CI result and rollup correction++[Run 39174382045](https://github.com/rcourtman/Pulse/actions/runs/39174382045)+compared baseline `ed4a2c6b54ee1d139d5f198950c53b403596c17c` with PR head+`5c37e620b45b50426c159a8a68af8e7bea04896e` through synthetic merge+`1b5a54e8a050217d84ae5ea0ddb64058840a5f63` (tree+`c7ecd028d86d4fafc8854f05182048e1f2bc2163`). All nine required source checks+passed. Source landing was paused after the optional paired benchmark exposed+a related production regression.++Ten paired 100 ms samples measured the four-hour cold chart at 124.37 ms+versus 83.75 ms, a 32.66% improvement (p < 0.001). Allocation volume remained+21.47 MiB and about 508,200 allocations per request. The warm handler and HTTP+transfer comparisons were not statistically significant. The same comparison+flagged `RollupTierBatched` at 3.919 versus 4.899 ms (+25.03%) and its 500-node+counterpart at 115.8 versus 145.7 ms (+25.78%), both p < 0.001. These are real+same-source production paths, not the earlier unchanged history-filter+benchmark signal. Artifact+[12154581287](https://github.com/rcourtman/Pulse/actions/runs/39174382045/artifacts/12154581287)+has GitHub digest+`5f9c9eeb9cfe3d3e67d07e5d50245f15dc1df19a937639e4d58b90227ae2c56c`.++The separate hosted strict job measured p50 84.827550 ms, p95 119.864148 ms and+p99 124.118998 ms. It failed both the explicit 90 ms assertion and capacity+validation: four proof intervals included a process named `provjobd4100855`+using 12.69% to 25.31% of total CPU. No matching process source was found in+the chart proof, its tests or reviewed runner scripts; its origin is not+established. The runner exposed four AMD EPYC 7763 CPUs, with GOMAXPROCS=2 and+real Actions identity. Admission and after samples were quiet; the interfering+intervals remain retained. Fixed executable SHA-256:+`f2d2dc6f86bd9e0a064d52d61df377bb4d3d71d0ac148c3e8ec18c48585a1085`.+Artifact+[12153956818](https://github.com/rcourtman/Pulse/actions/runs/39174382045/artifacts/12153956818)+has GitHub digest+`7ae238c3982119d122f1eb1a5b3c67c7059894bd61b2e1ecd602b43b2ad82b1a`.+Neither the faster cold benchmark nor this invalid-capacity SLO run establishes+the idle-worker 90 ms budget.++A fixed-executable native rollup profile attributed 34.09% cumulative CPU to+`BtreeIndexMoveto` and 17.18% to text comparisons in the aggregate sorter.+The old rollup scanned the tier/time secondary index, then sought the wider+composite primary tree once for every value. Clustered rollups now discover+only the resource identities in the source window through that covering+index, then read each resource's contiguous primary range. A query-local+integer resource number reduces repeated identity-string group comparisons.+It never changes persisted resource identity. The legacy aggregate remains+in use while the physical table is rowid-based. No extra index, cached data,+changed bucket limit or wider write budget is introduced.++Two retained native SQL probes compared the original aggregate, primary-order+grouping, resource ranges and integer resource grouping. Every variant+returned the same 4,000 rows from 40,000 source points on both schemas, digest+`eb61ee4816ff653ad69d7415d28fd234f04496a1c257bc4aad3ad31be8c1e288`.+In eight alternating rounds of the second probe, the clustered original and+integer-grouped medians were 161.180 and 103.572 ms. The legacy original median+was 123.433 ms. Shared-worker timing is diagnostic only; the retained paired+CI comparison must establish whether the rollup regression is removed.+Both temporary probes were hash-checked and moved outside the package before+compiling the revised production test executable.++The revised fixed executable passed targeted rollup tests, the full metrics+suite and the rollup race suite (3.866 s). Its SHA-256 is+`b439f1c36464ad14f6ea1392b31dbb83ced8f66948af3c06a504bd24856e2867`;+five-file source hashes are retained in `clustered-rollup/source.sha256`.+Tests compare legacy and clustered aggregates across three resource types,+reused resource names, three metrics, all rollup tiers, nullable/zero bounds,+window edges and authoritative pre-existing outputs. The query-plan guard+requires covering time-window discovery and a primary-key resource/time range.++Ten native 100 ms samples per benchmark gave revised/previous clustered+medians of 4.735637/5.7315405 ms for the small rollup and+118.3057575/165.052439 ms for the 500-node rollup. These sequential diagnostics+are retained without substituting for the revised paired CI result or the+still-pending strict idle-worker chart proof. The previous executable and+initial adverse CI results are preserved.diff --git a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md b/docs/release-control/v6/internal/subsystems/performance-and-scalability.mdindex 8699e9e6a8..87e05ed6e2 100644--- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md+++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md@@ -2774,7 +2774,17 @@ nullable bounds and all tiers, metadata, concurrent old snapshots and queuedwrites, cross-Store cached reads, cancellation after swap, crash before commit,conflicting replacement tables and primary-table range plans. Historical writeprofiles reconstruct their original rowid tables; `clustered` profiles exercise-the current production schema. The+the current production schema. Clustered rollups first discover resource+identities within the bounded source window through the covering tier/time+index, then read each resource through its primary-key time range. They must+not seek the primary table once per source observation through the secondary+index. A statement-local resource number gives the aggregate an integer group+key without changing stored resource identity, bucket boundaries, nullable+extrema fallback or `INSERT OR IGNORE` idempotence. It is neither persisted nor+cached. Legacy rowid databases retain their existing aggregate query until+physical migration. The clustered rollup plan must preserve covering discovery+and bounded primary ranges; full-table scans and extra write indexes are not+accepted substitutes. Thedeterministic issue-1124 profile must continue to attribute writes by logicalpayload, table/index `dbstat` pages, page-cache writes/spills, WAL frames,checkpoint frames, process write bytes where the host exposes them, and traceddiff --git a/pkg/metrics/clustered_layout.go b/pkg/metrics/clustered_layout.goindex 68f55d9112..ee2623e8e6 100644--- a/pkg/metrics/clustered_layout.go+++ b/pkg/metrics/clustered_layout.go@@ -16,6 +16,32 @@ import (// Keep the time-major key shared by ingestion, reconciliation and range reads.const clusteredMetricsIndex = "sqlite_autoindex_metrics_1"+// The tier/time tree covers resource identity, but fetching every value through+// it would seek the composite primary tree once per source point. Discover+// only the resources in this bounded window, then read their contiguous primary+// ranges. Group by a statement-local resource number to avoid comparing both+// identity strings for every input row; the number is unique within this query+// and never becomes a persisted or externally visible resource identity.+const clusteredMetricsRollupSQL = `+ WITH selected_resources AS MATERIALIZED (+ SELECT DISTINCT resource_type, resource_id FROM metrics+ WHERE tier = ? AND timestamp >= ? AND timestamp < ?+ ), resources AS MATERIALIZED (+ SELECT ROW_NUMBER() OVER (ORDER BY resource_type, resource_id) AS resource_number,+ resource_type, resource_id FROM selected_resources+ )+ INSERT OR IGNORE INTO metrics+ (resource_type, resource_id, metric_type, value, min_value, max_value, timestamp, tier)+ SELECT r.resource_type, r.resource_id, m.metric_type, AVG(m.value),+ MIN(COALESCE(m.min_value, m.value)), MAX(COALESCE(m.max_value, m.value)),+ (m.timestamp / ?) * ? AS bucket_ts, ?+ FROM resources AS r CROSS JOIN metrics AS m+ WHERE m.resource_type = r.resource_type AND m.resource_id = r.resource_id+ AND m.tier = ? AND m.timestamp >= ? AND m.timestamp < ?+ GROUP BY r.resource_number, bucket_ts, m.metric_type+ ORDER BY r.resource_number, bucket_ts, m.metric_type+`+const metricsTableColumns = `resource_type TEXT NOT NULL,resource_id TEXT NOT NULL,diff --git a/pkg/metrics/clustered_layout_test.go b/pkg/metrics/clustered_layout_test.goindex a1a6c1c14e..d61082f284 100644--- a/pkg/metrics/clustered_layout_test.go+++ b/pkg/metrics/clustered_layout_test.go@@ -60,6 +60,135 @@ func assertClusteredStore(t *testing.T, store *Store) {}}+func TestClusteredMetricsRollupPlanUsesBoundedPrimaryRanges(t *testing.T) {+ store, err := NewStore(DefaultConfig(t.TempDir()))+ if err != nil {+ t.Fatal(err)+ }+ t.Cleanup(func() { _ = store.Close() })+ rows, err := store.db.Query("EXPLAIN QUERY PLAN "+clusteredMetricsRollupSQL,+ "raw", int64(1_700_000_000), int64(1_700_000_120),+ int64(60), int64(60), "minute", "raw", int64(1_700_000_000), int64(1_700_000_120))+ if err != nil {+ t.Fatal(err)+ }+ defer rows.Close()+ var plan []string+ for rows.Next() {+ var id, parent, unused int+ var detail string+ if err := rows.Scan(&id, &parent, &unused, &detail); err != nil {+ t.Fatal(err)+ }+ plan = append(plan, detail)+ }+ if err := rows.Err(); err != nil {+ t.Fatal(err)+ }+ joined := strings.Join(plan, "\n")+ for _, want := range []string{+ "SEARCH metrics USING COVERING INDEX idx_metrics_tier_time (tier=? AND timestamp>? AND timestamp<?)",+ "SEARCH m USING PRIMARY KEY (resource_type=? AND resource_id=? AND tier=? AND timestamp>? AND timestamp<?)",+ } {+ if !strings.Contains(joined, want) {+ t.Fatalf("rollup lost its bounded covering discovery/primary range: missing %q in\n%s", want, joined)+ }+ }+ if strings.Contains(joined, "SCAN metrics") || strings.Contains(joined, "SCAN m ") {+ t.Fatalf("rollup scans observations outside its indexed window:\n%s", joined)+ }+}++func TestClusteredMetricsRollupsMatchLegacyAggregation(t *testing.T) {+ suppressLogs(t)+ stores := make([]*Store, 2)+ for i := range stores {+ cfg := DefaultConfig(t.TempDir())+ cfg.FlushInterval, cfg.RollupInterval = time.Hour, time.Hour+ store, err := NewStore(cfg)+ if err != nil {+ t.Fatal(err)+ }+ t.Cleanup(func() { _ = store.Close() })+ if err := store.WaitForMaintenance(30 * time.Second); err != nil {+ t.Fatal(err)+ }+ if i == 0 {+ issue1124InstallRowidSchema(t, store)+ }+ stores[i] = store+ }+ base := int64(1_700_006_400) // Aligned to a day, hour and minute.+ for _, store := range stores {+ for _, resourceType := range []string{"vm", "node", "dockercontainer"} {+ for _, resourceID := range []string{"same-id", "same-id:child", "z"} {+ for _, metric := range []string{"cpu", "memory", "bytes_total"} {+ for i, offset := range []int64{-1, 0, 5, 59, 60, 65, 119, 120} {+ var low, high any+ if i%2 == 0 {+ low = 0.0 // Explicit zero must remain a bound.+ }+ if i%3 == 0 {+ high = 60.0+ }+ if _, err := store.db.Exec(`INSERT INTO metrics+ (resource_type,resource_id,metric_type,value,min_value,max_value,timestamp,tier)+ VALUES (?,?,?,?,?,?,?,'raw')`, resourceType, resourceID, metric,+ float64(i*6), low, high, base+offset); err != nil {+ t.Fatal(err)+ }+ }+ }+ }+ }+ // A completed output is authoritative. Re-running a window must retain it.+ if _, err := store.db.Exec(`INSERT INTO metrics+ (resource_type,resource_id,metric_type,value,min_value,max_value,timestamp,tier)+ VALUES ('vm','same-id','cpu',900,0,1000,?,'minute')`, base); err != nil {+ t.Fatal(err)+ }+ }+ read := func(store *Store) []string {+ t.Helper()+ rows, err := store.db.Query(`SELECT resource_type,resource_id,metric_type,+ quote(value),quote(min_value),quote(max_value),timestamp,tier+ FROM metrics ORDER BY resource_type,resource_id,tier,timestamp,metric_type`)+ if err != nil {+ t.Fatal(err)+ }+ defer rows.Close()+ var result []string+ for rows.Next() {+ var rt, id, metric, value, low, high, tier string+ var ts int64+ if err := rows.Scan(&rt, &id, &metric, &value, &low, &high, &ts, &tier); err != nil {+ t.Fatal(err)+ }+ result = append(result, fmt.Sprint(rt, id, metric, value, low, high, ts, tier))+ }+ if err := rows.Err(); err != nil {+ t.Fatal(err)+ }+ return result+ }+ for _, stage := range []struct {+ from, to Tier+ bucket int64+ end int64+ }{{TierRaw, TierMinute, 60, base + 120}, {TierMinute, TierHourly, 3600, base + 3600}, {TierHourly, TierDaily, 86400, base + 86400}} {+ for repeat := 0; repeat < 2; repeat++ {+ for _, store := range stores {+ if !store.rollupTierWindow(stage.from, stage.to, stage.bucket, base, stage.end) {+ t.Fatalf("%s -> %s rollup failed", stage.from, stage.to)+ }+ }+ if old, current := read(stores[0]), read(stores[1]); !reflect.DeepEqual(old, current) {+ t.Fatalf("%s -> %s changed nullable bounds, identity, bucket edges or idempotence\nlegacy=%v\nclustered=%v", stage.from, stage.to, old, current)+ }+ }+ }+}+func TestClusteredMetricsFreshAndRestart(t *testing.T) {cfg := DefaultConfig(t.TempDir())for i := 0; i < 2; i++ {diff --git a/pkg/metrics/store.go b/pkg/metrics/store.goindex 87c7fb0d78..dacc21032f 100644--- a/pkg/metrics/store.go+++ b/pkg/metrics/store.go@@ -2359,7 +2359,7 @@ func (s *Store) rollupTierWindow(fromTier, toTier Tier, bucketSecs, startTs, end}defer func() { _ = tx.Rollback() }()- _, err = tx.Exec(`+ query := `INSERT OR IGNORE INTO metrics (resource_type, resource_id, metric_type, value, min_value, max_value, timestamp, tier)SELECTresource_type,@@ -2373,7 +2373,13 @@ func (s *Store) rollupTierWindow(fromTier, toTier Tier, bucketSecs, startTs, endFROM metricsWHERE tier = ? AND timestamp >= ? AND timestamp < ?GROUP BY resource_type, resource_id, metric_type, bucket_ts- `, bucketSecs, bucketSecs, string(toTier), string(fromTier), startTs, endTs)+ `+ params := []interface{}{bucketSecs, bucketSecs, string(toTier), string(fromTier), startTs, endTs}+ if s.clusteredMetrics.Load() {+ query = clusteredMetricsRollupSQL+ params = append([]interface{}{string(fromTier), startTs, endTs}, params...)+ }+ _, err = tx.Exec(query, params...)if err != nil {log.Warn().Err(err).Str("from", string(fromTier)).