Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Read clustered rollups through bounded primary ranges

rcourtmanfc58c597caf0
diff --git a/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md b/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md
index 9b938810c5..250df28938 100644
--- a/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md
+++ b/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md
@@ -117,3 +117,79 @@ benchmark reported 111.005964 ms/op, 22,262,176 B/op and 508,181 allocs/op on th
shared AMD Ryzen worker. That timing is retained without treating it as a
90 ms SLO proof. Status and contract audits passed; the new explicitly routed
test must be staged before the registry audit can recognize its tracked path.
+
+## First paired CI result and rollup correction
+
+[Run 39174382045](https://github.com/rcourtman/Pulse/actions/runs/39174382045)
+compared baseline `ed4a2c6b54ee1d139d5f198950c53b403596c17c` with PR head
+`5c37e620b45b50426c159a8a68af8e7bea04896e` through synthetic merge
+`1b5a54e8a050217d84ae5ea0ddb64058840a5f63` (tree
+`c7ecd028d86d4fafc8854f05182048e1f2bc2163`). All nine required source checks
+passed. Source landing was paused after the optional paired benchmark exposed
+a related production regression.
+
+Ten paired 100 ms samples measured the four-hour cold chart at 124.37 ms
+versus 83.75 ms, a 32.66% improvement (p < 0.001). Allocation volume remained
+21.47 MiB and about 508,200 allocations per request. The warm handler and HTTP
+transfer comparisons were not statistically significant. The same comparison
+flagged `RollupTierBatched` at 3.919 versus 4.899 ms (+25.03%) and its 500-node
+counterpart at 115.8 versus 145.7 ms (+25.78%), both p < 0.001. These are real
+same-source production paths, not the earlier unchanged history-filter
+benchmark signal. Artifact
+[12154581287](https://github.com/rcourtman/Pulse/actions/runs/39174382045/artifacts/12154581287)
+has GitHub digest
+`5f9c9eeb9cfe3d3e67d07e5d50245f15dc1df19a937639e4d58b90227ae2c56c`.
+
+The separate hosted strict job measured p50 84.827550 ms, p95 119.864148 ms and
+p99 124.118998 ms. It failed both the explicit 90 ms assertion and capacity
+validation: four proof intervals included a process named `provjobd4100855`
+using 12.69% to 25.31% of total CPU. No matching process source was found in
+the chart proof, its tests or reviewed runner scripts; its origin is not
+established. The runner exposed four AMD EPYC 7763 CPUs, with GOMAXPROCS=2 and
+real Actions identity. Admission and after samples were quiet; the interfering
+intervals remain retained. Fixed executable SHA-256:
+`f2d2dc6f86bd9e0a064d52d61df377bb4d3d71d0ac148c3e8ec18c48585a1085`.
+Artifact
+[12153956818](https://github.com/rcourtman/Pulse/actions/runs/39174382045/artifacts/12153956818)
+has GitHub digest
+`7ae238c3982119d122f1eb1a5b3c67c7059894bd61b2e1ecd602b43b2ad82b1a`.
+Neither the faster cold benchmark nor this invalid-capacity SLO run establishes
+the idle-worker 90 ms budget.
+
+A fixed-executable native rollup profile attributed 34.09% cumulative CPU to
+`BtreeIndexMoveto` and 17.18% to text comparisons in the aggregate sorter.
+The old rollup scanned the tier/time secondary index, then sought the wider
+composite primary tree once for every value. Clustered rollups now discover
+only the resource identities in the source window through that covering
+index, then read each resource's contiguous primary range. A query-local
+integer resource number reduces repeated identity-string group comparisons.
+It never changes persisted resource identity. The legacy aggregate remains
+in use while the physical table is rowid-based. No extra index, cached data,
+changed bucket limit or wider write budget is introduced.
+
+Two retained native SQL probes compared the original aggregate, primary-order
+grouping, resource ranges and integer resource grouping. Every variant
+returned the same 4,000 rows from 40,000 source points on both schemas, digest
+`eb61ee4816ff653ad69d7415d28fd234f04496a1c257bc4aad3ad31be8c1e288`.
+In eight alternating rounds of the second probe, the clustered original and
+integer-grouped medians were 161.180 and 103.572 ms. The legacy original median
+was 123.433 ms. Shared-worker timing is diagnostic only; the retained paired
+CI comparison must establish whether the rollup regression is removed.
+Both temporary probes were hash-checked and moved outside the package before
+compiling the revised production test executable.
+
+The revised fixed executable passed targeted rollup tests, the full metrics
+suite and the rollup race suite (3.866 s). Its SHA-256 is
+`b439f1c36464ad14f6ea1392b31dbb83ced8f66948af3c06a504bd24856e2867`;
+five-file source hashes are retained in `clustered-rollup/source.sha256`.
+Tests compare legacy and clustered aggregates across three resource types,
+reused resource names, three metrics, all rollup tiers, nullable/zero bounds,
+window edges and authoritative pre-existing outputs. The query-plan guard
+requires covering time-window discovery and a primary-key resource/time range.
+
+Ten native 100 ms samples per benchmark gave revised/previous clustered
+medians of 4.735637/5.7315405 ms for the small rollup and
+118.3057575/165.052439 ms for the 500-node rollup. These sequential diagnostics
+are retained without substituting for the revised paired CI result or the
+still-pending strict idle-worker chart proof. The previous executable and
+initial adverse CI results are preserved.
diff --git a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md
index 8699e9e6a8..87e05ed6e2 100644
--- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md
+++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md
@@ -2774,7 +2774,17 @@ nullable bounds and all tiers, metadata, concurrent old snapshots and queued
writes, cross-Store cached reads, cancellation after swap, crash before commit,
conflicting replacement tables and primary-table range plans. Historical write
profiles reconstruct their original rowid tables; `clustered` profiles exercise
-the current production schema. The
+the current production schema. Clustered rollups first discover resource
+identities within the bounded source window through the covering tier/time
+index, then read each resource through its primary-key time range. They must
+not seek the primary table once per source observation through the secondary
+index. A statement-local resource number gives the aggregate an integer group
+key without changing stored resource identity, bucket boundaries, nullable
+extrema fallback or `INSERT OR IGNORE` idempotence. It is neither persisted nor
+cached. Legacy rowid databases retain their existing aggregate query until
+physical migration. The clustered rollup plan must preserve covering discovery
+and bounded primary ranges; full-table scans and extra write indexes are not
+accepted substitutes. The
deterministic issue-1124 profile must continue to attribute writes by logical
payload, table/index `dbstat` pages, page-cache writes/spills, WAL frames,
checkpoint frames, process write bytes where the host exposes them, and traced
diff --git a/pkg/metrics/clustered_layout.go b/pkg/metrics/clustered_layout.go
index 68f55d9112..ee2623e8e6 100644
--- a/pkg/metrics/clustered_layout.go
+++ b/pkg/metrics/clustered_layout.go
@@ -16,6 +16,32 @@ import (
// Keep the time-major key shared by ingestion, reconciliation and range reads.
const clusteredMetricsIndex = "sqlite_autoindex_metrics_1"
+// The tier/time tree covers resource identity, but fetching every value through
+// it would seek the composite primary tree once per source point. Discover
+// only the resources in this bounded window, then read their contiguous primary
+// ranges. Group by a statement-local resource number to avoid comparing both
+// identity strings for every input row; the number is unique within this query
+// and never becomes a persisted or externally visible resource identity.
+const clusteredMetricsRollupSQL = `
+ WITH selected_resources AS MATERIALIZED (
+ SELECT DISTINCT resource_type, resource_id FROM metrics
+ WHERE tier = ? AND timestamp >= ? AND timestamp < ?
+ ), resources AS MATERIALIZED (
+ SELECT ROW_NUMBER() OVER (ORDER BY resource_type, resource_id) AS resource_number,
+ resource_type, resource_id FROM selected_resources
+ )
+ INSERT OR IGNORE INTO metrics
+ (resource_type, resource_id, metric_type, value, min_value, max_value, timestamp, tier)
+ SELECT r.resource_type, r.resource_id, m.metric_type, AVG(m.value),
+ MIN(COALESCE(m.min_value, m.value)), MAX(COALESCE(m.max_value, m.value)),
+ (m.timestamp / ?) * ? AS bucket_ts, ?
+ FROM resources AS r CROSS JOIN metrics AS m
+ WHERE m.resource_type = r.resource_type AND m.resource_id = r.resource_id
+ AND m.tier = ? AND m.timestamp >= ? AND m.timestamp < ?
+ GROUP BY r.resource_number, bucket_ts, m.metric_type
+ ORDER BY r.resource_number, bucket_ts, m.metric_type
+`
+
const metricsTableColumns = `
resource_type TEXT NOT NULL,
resource_id TEXT NOT NULL,
diff --git a/pkg/metrics/clustered_layout_test.go b/pkg/metrics/clustered_layout_test.go
index a1a6c1c14e..d61082f284 100644
--- a/pkg/metrics/clustered_layout_test.go
+++ b/pkg/metrics/clustered_layout_test.go
@@ -60,6 +60,135 @@ func assertClusteredStore(t *testing.T, store *Store) {
}
}
+func TestClusteredMetricsRollupPlanUsesBoundedPrimaryRanges(t *testing.T) {
+ store, err := NewStore(DefaultConfig(t.TempDir()))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = store.Close() })
+ rows, err := store.db.Query("EXPLAIN QUERY PLAN "+clusteredMetricsRollupSQL,
+ "raw", int64(1_700_000_000), int64(1_700_000_120),
+ int64(60), int64(60), "minute", "raw", int64(1_700_000_000), int64(1_700_000_120))
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer rows.Close()
+ var plan []string
+ for rows.Next() {
+ var id, parent, unused int
+ var detail string
+ if err := rows.Scan(&id, &parent, &unused, &detail); err != nil {
+ t.Fatal(err)
+ }
+ plan = append(plan, detail)
+ }
+ if err := rows.Err(); err != nil {
+ t.Fatal(err)
+ }
+ joined := strings.Join(plan, "\n")
+ for _, want := range []string{
+ "SEARCH metrics USING COVERING INDEX idx_metrics_tier_time (tier=? AND timestamp>? AND timestamp<?)",
+ "SEARCH m USING PRIMARY KEY (resource_type=? AND resource_id=? AND tier=? AND timestamp>? AND timestamp<?)",
+ } {
+ if !strings.Contains(joined, want) {
+ t.Fatalf("rollup lost its bounded covering discovery/primary range: missing %q in\n%s", want, joined)
+ }
+ }
+ if strings.Contains(joined, "SCAN metrics") || strings.Contains(joined, "SCAN m ") {
+ t.Fatalf("rollup scans observations outside its indexed window:\n%s", joined)
+ }
+}
+
+func TestClusteredMetricsRollupsMatchLegacyAggregation(t *testing.T) {
+ suppressLogs(t)
+ stores := make([]*Store, 2)
+ for i := range stores {
+ cfg := DefaultConfig(t.TempDir())
+ cfg.FlushInterval, cfg.RollupInterval = time.Hour, time.Hour
+ store, err := NewStore(cfg)
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = store.Close() })
+ if err := store.WaitForMaintenance(30 * time.Second); err != nil {
+ t.Fatal(err)
+ }
+ if i == 0 {
+ issue1124InstallRowidSchema(t, store)
+ }
+ stores[i] = store
+ }
+ base := int64(1_700_006_400) // Aligned to a day, hour and minute.
+ for _, store := range stores {
+ for _, resourceType := range []string{"vm", "node", "dockercontainer"} {
+ for _, resourceID := range []string{"same-id", "same-id:child", "z"} {
+ for _, metric := range []string{"cpu", "memory", "bytes_total"} {
+ for i, offset := range []int64{-1, 0, 5, 59, 60, 65, 119, 120} {
+ var low, high any
+ if i%2 == 0 {
+ low = 0.0 // Explicit zero must remain a bound.
+ }
+ if i%3 == 0 {
+ high = 60.0
+ }
+ if _, err := store.db.Exec(`INSERT INTO metrics
+ (resource_type,resource_id,metric_type,value,min_value,max_value,timestamp,tier)
+ VALUES (?,?,?,?,?,?,?,'raw')`, resourceType, resourceID, metric,
+ float64(i*6), low, high, base+offset); err != nil {
+ t.Fatal(err)
+ }
+ }
+ }
+ }
+ }
+ // A completed output is authoritative. Re-running a window must retain it.
+ if _, err := store.db.Exec(`INSERT INTO metrics
+ (resource_type,resource_id,metric_type,value,min_value,max_value,timestamp,tier)
+ VALUES ('vm','same-id','cpu',900,0,1000,?,'minute')`, base); err != nil {
+ t.Fatal(err)
+ }
+ }
+ read := func(store *Store) []string {
+ t.Helper()
+ rows, err := store.db.Query(`SELECT resource_type,resource_id,metric_type,
+ quote(value),quote(min_value),quote(max_value),timestamp,tier
+ FROM metrics ORDER BY resource_type,resource_id,tier,timestamp,metric_type`)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer rows.Close()
+ var result []string
+ for rows.Next() {
+ var rt, id, metric, value, low, high, tier string
+ var ts int64
+ if err := rows.Scan(&rt, &id, &metric, &value, &low, &high, &ts, &tier); err != nil {
+ t.Fatal(err)
+ }
+ result = append(result, fmt.Sprint(rt, id, metric, value, low, high, ts, tier))
+ }
+ if err := rows.Err(); err != nil {
+ t.Fatal(err)
+ }
+ return result
+ }
+ for _, stage := range []struct {
+ from, to Tier
+ bucket int64
+ end int64
+ }{{TierRaw, TierMinute, 60, base + 120}, {TierMinute, TierHourly, 3600, base + 3600}, {TierHourly, TierDaily, 86400, base + 86400}} {
+ for repeat := 0; repeat < 2; repeat++ {
+ for _, store := range stores {
+ if !store.rollupTierWindow(stage.from, stage.to, stage.bucket, base, stage.end) {
+ t.Fatalf("%s -> %s rollup failed", stage.from, stage.to)
+ }
+ }
+ if old, current := read(stores[0]), read(stores[1]); !reflect.DeepEqual(old, current) {
+ t.Fatalf("%s -> %s changed nullable bounds, identity, bucket edges or idempotence\nlegacy=%v\nclustered=%v", stage.from, stage.to, old, current)
+ }
+ }
+ }
+}
+
func TestClusteredMetricsFreshAndRestart(t *testing.T) {
cfg := DefaultConfig(t.TempDir())
for i := 0; i < 2; i++ {
diff --git a/pkg/metrics/store.go b/pkg/metrics/store.go
index 87c7fb0d78..dacc21032f 100644
--- a/pkg/metrics/store.go
+++ b/pkg/metrics/store.go
@@ -2359,7 +2359,7 @@ func (s *Store) rollupTierWindow(fromTier, toTier Tier, bucketSecs, startTs, end
}
defer func() { _ = tx.Rollback() }()
- _, err = tx.Exec(`
+ query := `
INSERT OR IGNORE INTO metrics (resource_type, resource_id, metric_type, value, min_value, max_value, timestamp, tier)
SELECT
resource_type,
@@ -2373,7 +2373,13 @@ func (s *Store) rollupTierWindow(fromTier, toTier Tier, bucketSecs, startTs, end
FROM metrics
WHERE tier = ? AND timestamp >= ? AND timestamp < ?
GROUP BY resource_type, resource_id, metric_type, bucket_ts
- `, bucketSecs, bucketSecs, string(toTier), string(fromTier), startTs, endTs)
+ `
+ params := []interface{}{bucketSecs, bucketSecs, string(toTier), string(fromTier), startTs, endTs}
+ if s.clusteredMetrics.Load() {
+ query = clusteredMetricsRollupSQL
+ params = append([]interface{}{string(fromTier), startTs, endTs}, params...)
+ }
+ _, err = tx.Exec(query, params...)
if err != nil {
log.Warn().Err(err).
Str("from", string(fromTier)).