Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Cluster retained metrics storage to remove chart row lookups
rcourtman5c37e620b45b
diff --git a/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md b/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.mdnew file mode 100644index 0000000000..9b938810c5--- /dev/null+++ b/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md@@ -0,0 +1,119 @@+# Workload chart storage and idle-worker qualification, 18 October 2026++Scope: the remaining cold-service four-hour workload-chart latency gap under+L10 `performance-post-rc-headroom`. The acceptance budget remains 90 ms p95.+Source landing, a relative benchmark, and an idle-worker SLO proof are separate+observations. This record does not close the broad L10 residual.++## Retained failed proof and worker inventory++The installed disposable `pulse-ci-vm-large` profile provides two vCPUs and+4 GiB of RAM. The persistent worker shares other workloads. A configured+performance profile alone is not an available trusted proof route. The manual+source-bound route is owned by the L10 contract at+`pulse-enterprise:.github/workflows/verify-workload-chart-latency.yml` and its+runbook `pulse-enterprise:scripts/workload-chart-latency-proof.md`.++[Run 39168929815](https://github.com/rcourtman/pulse-enterprise/actions/runs/39168929815)+used public source `8a590102fef2089e938815bea280fee5423334fc`, private control+`4f8f9542ee122471c0bfe4f51334e7ef0d19796b`, Go 1.26.9 and the disposable runner+`pulse-ci-vm-enterprise-large-1018214122-359c52` (two Intel Xeon CPUs). Executable+SHA-256: `0bec46b33b3f3a1e23adac8309093966d528c4d6b8c9b2b2786b3e1786255553`.++Its unchanged strict proof failed: p50 99.322850 ms, p95 112.164352 ms and p99+121.091084 ms. Admission was quiet, but one proof interval recorded 0.49505%+CPU steal against the existing 0.1% limit. The result therefore establishes+neither the latency budget nor an idle-worker proof. All 24 during-proof+capacity samples, admission/after samples and diagnostics are retained. The+logs were extracted with original per-file byte counts verified, without+omitting the failing sample or substituting a second SLO attempt.++The separate diagnostic benchmark measured 99.635869 ms/op. Its CPU profile+attributed 80.55% cumulative CPU to the retained query, 52.74% to SQLite's VM and+21.18% to `BtreeTableMoveto`. Finding each identity entry then seeking its rowid+table payload remained a substantial cost. The pinned SQLite driver opens+FULLMUTEX connections; an unsupported connection-string option would not+remove that cost. Cache and mmap size experiments did not demonstrate a+reliable benefit and were not adopted.++## Canonical storage change++`pkg/metrics/clustered_layout.go` stores observations in+`metrics WITHOUT ROWID`, keyed by+`(resource_type, resource_id, tier, timestamp, metric_type)`. Values and nullable+bounds now live in the same B-tree used by retained range reads. The only+secondary tree remains `(tier, timestamp)` for retention. The identity column+set and all eight observation columns retain their existing semantics.++Installed rowid databases keep their authoritative unique index while deferred+startup maintenance copies and swaps the table in one `BEGIN IMMEDIATE` WAL+transaction. Existing readers retain their old SQLite snapshots. Writes queued+behind the transaction resume after commit. Copy failure, cancellation or a+crash before commit rolls back the entire schema and payload change. A+conflicting replacement table is preserved. The free-space guard requires+three times the used database pages plus 16 MiB; inadequate capacity defers the+migration and leaves the installed format readable. Layout and auto-vacuum+share the existing 30-minute rewrite context, canceled on Store shutdown.++Legacy stores with no authoritative unique index still establish uniqueness+synchronously before upserts. Already-authoritative stores skip the redundant+old index rebuild and migrate directly. Readers fix their index selection for+each snapshot; a stale legacy-index error retries once only after verifying+the new primary key. The physical layout participates in the bounded SQL cache+shape. No results or tier presence are cached. An older executable can reopen+the row-compatible table and recreate its old index; subsequent maintenance+removes those known redundant indexes after checking the clustered primary key.++The retained query engine, tier reconciliation, Docker observation provenance,+nullable-bound fallback, value projection, LTTB sampling, chart point limits,+cache bounds, write retry policy and SLO thresholds remain the same.++## Qualification evidence++The parent-owned native evidence directory is+`/opt/pulse-release-worker/task-proofs/chart-latency-slo-20261018/clustered-qualification`.+The initial full metrics suite passed in 219.902 s. The 12-tick deterministic+write profile preserved 157,452 observations across 2,197 resources in 72 writes:+20,842 WAL frames (85,869,072 bytes), below the unchanged 23,000-frame ceiling.+The rowid comparison used 20,223 frames, so the clustered format trades about+3.06% more WAL traffic for removal of the per-observation table lookup. Its+checkpoint wrote 20,672,512 bytes and the main database used 20,725,760 bytes,+versus 23,797,760 and 23,846,912 respectively in the rowid comparison. These+physical write/size observations are distinct from latency qualification.++With the production 4,000-frame checkpoint threshold, four connections and a+paced concurrent reader, the initial 30-tick profile preserved 393,630 rows+through restart and integrity checking. It completed 2,107 reads (p95 0.497286+ms, maximum 17.202902 ms), and retention removed every row. The shared native+worker is unsuitable for an absolute chart SLO claim; these profiles qualify+storage behavior only. Final source-bound verification is recorded separately+from these initial diagnostics.++The final fixed-executable production comparison used 393,630 observations,+180 writes, 4,000-frame checkpointing and the same paced-reader configuration.+The historical rowid profile wrote 728,674,304 process bytes; the clustered+profile wrote 789,917,696 (+8.40%). Final explicit checkpoint writes were+9,256,960 versus 10,158,080 bytes (+9.73%). Main database size was 61,054,976+versus 52,621,312 bytes (-13.81%). Both profiles preserved every row through+restart and integrity checking, then retained zero rows after purge. Their+respective read counts were 18,917 and 4,080. The shared worker produced large+write/read timing outliers, so these sequential profile timings do not prove+a throughput or tail-latency improvement. Retain the increased write volume+as a measured tradeoff; the production checkpoint threshold, 23,000-frame+no-auto-checkpoint ceiling and steady-state write guard are unchanged.++The final full metrics suite and migration race tests passed. The race run+completed in 64.407 s, including the atomic-swap cancellation/crash paths,+independent Store handles and cached read shapes. Temporary layout/cache probe+code was hash-checked and moved out of the package before compiling the final+fixed test executable.++Final fixed metrics executable SHA-256:+`aa501813ca9cd8d6de5d61ae03ac36005e640dd9aa812c9b6da7ca651b6dd689`.+Per-file Go source hashes are retained in `clustered-final/source.sha256`.+The v5 metrics compatibility tests, downstream monitoring/chart sampling tests+(67.669 s), and 11 chart API tests (0.625 s) passed. The separate cold-service+benchmark reported 111.005964 ms/op, 22,262,176 B/op and 508,181 allocs/op on the+shared AMD Ryzen worker. That timing is retained without treating it as a+90 ms SLO proof. Status and contract audits passed; the new explicitly routed+test must be staged before the registry audit can recognize its tracked path.diff --git a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md b/docs/release-control/v6/internal/subsystems/performance-and-scalability.mdindex c3be9f638f..8699e9e6a8 100644--- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md+++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md@@ -545,7 +545,7 @@ which retention tiers have observations in that snapshot. Every present tierparticipates in the shared indexed overlap query. Presence never stands in forper-series coverage. The store retains at most 32 compiled read-statement shapesand 32 SQL templates keyed only by parameter counts, tier order and aggregation-shape. Alphabetic named SQLite bindings supply each current identity, window and+shape, projection and physical layout. Alphabetic named SQLite bindings supply each current identity, window anddisplay step once across all branches. Bindings use `database/sql.Named` so thedriver need not repeatedly convert ordinals to strings while matching a largeparameter set. Names describe parameter positions only. Resource IDs and all@@ -633,6 +633,10 @@ in-place click and chevron activation over a selection.- `frontend-modern/src/utils/filesystemByteUsage.ts`: shared finite byte/capacity admission for filesystem utilization.1. `pkg/metrics/store.go`+ 1a. `pkg/metrics/clustered_layout.go`+ 1b. `pkg/metrics/clustered_layout_test.go`+ 1c. `pkg/metrics/identity_index_test.go`+ 1d. `pkg/metrics/issue1124_write_amplification_test.go`2. `pkg/metrics/store_query_plan_test.go`3. `pkg/metrics/store_slo_test.go`4. `pkg/metrics/store_additional_test.go`@@ -2698,9 +2702,13 @@ WAL checkpoints more aggressive again, reopens duplicate-write failures,removes the bounded rollup-window checkpointing, or removes the metrics DBpath/cadence controls must re-prove the metrics-store hot path with the ownedstore tests rather than assuming the earlier vacuum fixes are sufficient.-Metric identity and every range read must also share one unique, time-major-`idx_metrics_query_all(resource_type, resource_id, tier, timestamp,-metric_type)` B-tree. Reintroducing a second full-width identity index is a+Metric identity, stored values and every range read share the time-major+`PRIMARY KEY(resource_type, resource_id, tier, timestamp, metric_type)` of+`metrics WITHOUT ROWID`. The primary table itself supplies the samples; a+separate rowid-table lookup per observation is no longer required. Its only+secondary tree is the `(tier, timestamp)` retention index. Legacy rowid stores+continue to use `idx_metrics_query_all` until deferred migration commits.+Reintroducing a second full-width identity index is awrite-amplification regression even when retained `metrics.db` size looks small:every sample would dirty both trees in the WAL and every checkpoint would copyboth sets of pages back into the main file. The column order is part of the@@ -2738,12 +2746,35 @@ value points afterward. Full and value projections share scope normalization,filters, tier reconciliation, time ordering, display buckets and Dockerobservation provenance; their distinct SQL column shapes must have distinctkeys within the existing bounded query cache. Neither mode caches results or-tier presence. Full readers continue to preserve extrema. Legacy databases keep their-existing unique index authoritative while the replacement index is built, but-that O(rows) migration must run on deferred startup maintenance and swap indexes-in one SQLite transaction so constructor latency, concurrent reads, uniqueness,-and crash recovery remain intact; a database with no unique identity index at-all migrates synchronously so upserts never run without a conflict target. The+tier presence. Full readers continue to preserve extrema. Legacy databases keep+their existing unique index authoritative while deferred startup maintenance+copies the eight observation columns into the clustered table. `BEGIN IMMEDIATE`+serializes writers and independent Store migrations before inspection. Copy,+old-table removal, rename and retention-index creation commit in one WAL+transaction; cancellation, crash, insufficient disk space or a conflicting+replacement table leaves the installed table and observations intact. Readers+retain their old SQLite snapshots during the copy; queued writes resume after+commit. The migration checks free filesystem space for three times the used+database pages plus 16 MiB before copying. Layout migration and auto-vacuum+share the existing 30-minute rewrite context, canceled when the Store closes.+There is no redundant legacy index rebuild before the copy. A database with+no unique identity index still deduplicates and establishes identity+synchronously before ingestion starts.++Each retained read fixes its index choice for its snapshot. A stale legacy+index error may retry once with a new snapshot only after inspecting the+clustered primary key. This covers another Store committing the upgrade while+prepared SQL is cached. Layout participates in the bounded SQL-template key;+results and tier presence remain uncached. Restart recognizes the primary key+without recreating the legacy index. If an older executable reopens the+row-compatible table and recreates its legacy identity index, deferred+maintenance removes only the known redundant trees after verifying the primary+key. `clustered_layout_test.go` covers fresh/reopened schemas, exact stored+nullable bounds and all tiers, metadata, concurrent old snapshots and queued+writes, cross-Store cached reads, cancellation after swap, crash before commit,+conflicting replacement tables and primary-table range plans. Historical write+profiles reconstruct their original rowid tables; `clustered` profiles exercise+the current production schema. Thedeterministic issue-1124 profile must continue to attribute writes by logicalpayload, table/index `dbstat` pages, page-cache writes/spills, WAL frames,checkpoint frames, process write bytes where the host exposes them, and traced@@ -4093,7 +4124,7 @@ mode in `TestStoreAutoVacuumPersistsAcrossRestart`.### Metrics-store writes survive the one-time startup write-lock hold-The deferred identity-index rebuild and the auto-vacuum VACUUM can hold the+The deferred clustered-table migration and the auto-vacuum VACUUM can hold theSQLite write lock for minutes on a large legacy database. `BEGIN` is deferred,so the lock is contended at the first `INSERT` or at `COMMIT`, not at `Begin`;a retry loop around `Begin` alone skipped rows and committed an emptydiff --git a/docs/release-control/v6/internal/subsystems/registry.json b/docs/release-control/v6/internal/subsystems/registry.jsonindex 751155a6e8..0f83a1ac30 100644--- a/docs/release-control/v6/internal/subsystems/registry.json+++ b/docs/release-control/v6/internal/subsystems/registry.json@@ -7458,6 +7458,9 @@"allow_same_subsystem_tests": false,"test_prefixes": [],"exact_files": [+ "pkg/metrics/clustered_layout_test.go",+ "pkg/metrics/identity_index_test.go",+ "pkg/metrics/issue1124_write_amplification_test.go","pkg/metrics/store_additional_test.go","pkg/metrics/store_bench_test.go","pkg/metrics/store_docker_observation_contract_test.go",diff --git a/pkg/metrics/clustered_layout.go b/pkg/metrics/clustered_layout.gonew file mode 100644index 0000000000..68f55d9112--- /dev/null+++ b/pkg/metrics/clustered_layout.go@@ -0,0 +1,232 @@+package metrics++import (+ "context"+ "database/sql"+ "fmt"+ "math"+ "path/filepath"+ "strings"++ "github.com/shirou/gopsutil/v4/disk"+)++// Store samples in the range-read B-tree itself. The old rowid table required+// a second B-tree lookup for every value after finding its identity index entry.+// Keep the time-major key shared by ingestion, reconciliation and range reads.+const clusteredMetricsIndex = "sqlite_autoindex_metrics_1"++const metricsTableColumns = `+ resource_type TEXT NOT NULL,+ resource_id TEXT NOT NULL,+ metric_type TEXT NOT NULL,+ value REAL NOT NULL,+ min_value REAL,+ max_value REAL,+ timestamp INTEGER NOT NULL,+ tier TEXT NOT NULL DEFAULT 'raw',+ PRIMARY KEY(resource_type, resource_id, tier, timestamp, metric_type)+`++type metricsSchemaReader interface {+ QueryContext(context.Context, string, ...any) (*sql.Rows, error)+}++// Inspect the primary key rather than a schema version marker: a restored+// backup or another Store handle can change the actual physical schema.+func metricsTableIsClustered(ctx context.Context, db metricsSchemaReader) (bool, error) {+ rows, err := db.QueryContext(ctx, `SELECT wr FROM pragma_table_list WHERE schema = 'main' AND name = 'metrics'`)+ if err != nil {+ return false, err+ }+ var withoutRowid bool+ if rows.Next() {+ err = rows.Scan(&withoutRowid)+ } else {+ err = rows.Err()+ if err == nil {+ err = fmt.Errorf("metrics table is missing")+ }+ }+ if closeErr := rows.Close(); err == nil {+ err = closeErr+ }+ if err != nil || !withoutRowid {+ return false, err+ }+ rows, err = db.QueryContext(ctx, `PRAGMA index_info(`+clusteredMetricsIndex+`)`)+ if err != nil {+ return false, err+ }+ defer rows.Close()+ var columns []string+ for rows.Next() {+ var sequence, cid int+ var name string+ if err := rows.Scan(&sequence, &cid, &name); err != nil {+ return false, err+ }+ columns = append(columns, name)+ }+ if err := rows.Err(); err != nil {+ return false, err+ }+ return strings.Join(columns, ",") == strings.Join(metricsIdentityColumns, ","), nil+}++func (s *Store) metricsReadIndex() string {+ if s.clusteredMetrics.Load() {+ return clusteredMetricsIndex+ }+ return metricsIdentityIndex+}++// Reserve room for the replacement pages and the transaction's WAL before+// copying pages. Keep the readable legacy layout if the filesystem cannot+// safely hold a copy; a later startup can retry without a partial migration.+func (s *Store) checkClusteredMigrationSpace(ctx context.Context, conn *sql.Conn) error {+ var pages, freePages, pageSize int64+ for _, item := range []struct {+ query string+ dest *int64+ }{{"PRAGMA page_count", &pages}, {"PRAGMA freelist_count", &freePages}, {"PRAGMA page_size", &pageSize}} {+ if err := conn.QueryRowContext(ctx, item.query).Scan(item.dest); err != nil {+ return err+ }+ }+ usedPages := pages - freePages+ const margin = int64(16 << 20)+ if usedPages < 0 || pageSize <= 0 || usedPages > (math.MaxInt64-margin)/3/pageSize {+ return fmt.Errorf("invalid metrics page accounting")+ }+ required := uint64(usedPages*pageSize*3 + margin)+ usage, err := disk.UsageWithContext(ctx, filepath.Dir(s.config.DBPath))+ if err != nil {+ return fmt.Errorf("inspect metrics migration free space: %w", err)+ }+ if usage.Free < required {+ return fmt.Errorf("metrics clustered migration needs %d free bytes, available %d", required, usage.Free)+ }+ return nil+}++// Copy and swap in one WAL transaction. Existing readers keep their old+// snapshot throughout the copy; queued writes resume against the new table+// after commit. An interruption rolls back the table, indexes and every row.+// BEGIN IMMEDIATE also serializes independent Store handles before inspection.+func (s *Store) migrateClusteredMetrics(ctx context.Context) error {+ if s.clusteredMetrics.Load() {+ return s.cleanupClusteredMetricsIndexes(ctx)+ }+ conn, err := s.db.DB.Conn(ctx)+ if err != nil {+ return err+ }+ defer conn.Close()+ if _, err := conn.ExecContext(ctx, `BEGIN IMMEDIATE`); err != nil {+ return fmt.Errorf("begin clustered metrics migration: %w", err)+ }+ defer func() { _, _ = conn.ExecContext(context.Background(), `ROLLBACK`) }()+ clustered, err := metricsTableIsClustered(ctx, conn)+ if err != nil {+ return err+ }+ if !clustered {+ if err := s.checkClusteredMigrationSpace(ctx, conn); err != nil {+ return err+ }+ if _, err := conn.ExecContext(ctx, `CREATE TABLE metrics_clustered_upgrade (`+metricsTableColumns+`) WITHOUT ROWID`); err != nil {+ return fmt.Errorf("create clustered metrics replacement: %w", err)+ }+ if s.clusteredMigrationHook != nil {+ s.clusteredMigrationHook("before-copy")+ }+ if _, err := conn.ExecContext(ctx, `+ INSERT INTO metrics_clustered_upgrade+ (resource_type, resource_id, metric_type, value, min_value, max_value, timestamp, tier)+ SELECT resource_type, resource_id, metric_type, value, min_value, max_value, timestamp, tier+ FROM metrics ORDER BY resource_type, resource_id, tier, timestamp, metric_type+ `); err != nil {+ return fmt.Errorf("copy clustered metrics: %w", err)+ }+ if s.clusteredMigrationHook != nil {+ s.clusteredMigrationHook("after-copy")+ }+ if _, err := conn.ExecContext(ctx, `+ DROP TABLE metrics;+ ALTER TABLE metrics_clustered_upgrade RENAME TO metrics;+ CREATE INDEX idx_metrics_tier_time ON metrics(tier, timestamp);+ `); err != nil {+ return fmt.Errorf("swap clustered metrics: %w", err)+ }+ }+ if s.clusteredMigrationHook != nil {+ s.clusteredMigrationHook("before-commit")+ }+ if _, err := conn.ExecContext(ctx, `COMMIT`); err != nil {+ return fmt.Errorf("commit clustered metrics migration: %w", err)+ }+ s.clusteredMetrics.Store(true)+ return nil+}++// A previous executable can reopen this row-compatible table and recreate its+// old read index. Remove only those known redundant trees on a later startup.+// The ordinary clustered restart is a read-only probe, without a writer lock.+func (s *Store) cleanupClusteredMetricsIndexes(ctx context.Context) error {+ rows, err := s.db.DB.QueryContext(ctx, `SELECT name FROM pragma_index_list('metrics')+ WHERE name IN ('idx_metrics_query_all', 'idx_metrics_lookup', 'idx_metrics_unique')`)+ if err != nil {+ return err+ }+ var indexes []string+ for rows.Next() {+ var name string+ if err := rows.Scan(&name); err != nil {+ _ = rows.Close()+ return err+ }+ indexes = append(indexes, name)+ }+ err = rows.Err()+ _ = rows.Close()+ if err != nil || len(indexes) == 0 {+ return err+ }+ conn, err := s.db.DB.Conn(ctx)+ if err != nil {+ return err+ }+ defer conn.Close()+ if _, err := conn.ExecContext(ctx, `BEGIN IMMEDIATE`); err != nil {+ return err+ }+ defer func() { _, _ = conn.ExecContext(context.Background(), `ROLLBACK`) }()+ clustered, err := metricsTableIsClustered(ctx, conn)+ if err != nil || !clustered {+ return fmt.Errorf("cannot prune identity indexes without the clustered primary key: %v", err)+ }+ for _, index := range indexes {+ // Names come from the fixed whitelist above, never user input.+ if _, err := conn.ExecContext(ctx, `DROP INDEX IF EXISTS `+index); err != nil {+ return err+ }+ }+ _, err = conn.ExecContext(ctx, `COMMIT`)+ return err+}++// A different Store can commit the migration between preparing and executing+// a legacy read. Retry that missing-index error once, after verifying the new+// primary key, with a new snapshot and the same canonical read semantics.+func (s *Store) refreshClusteredMetrics(err error) bool {+ if !strings.Contains(err.Error(), "no such index: "+metricsIdentityIndex) {+ return false+ }+ clustered, inspectErr := metricsTableIsClustered(context.Background(), s.db.DB)+ if inspectErr != nil || !clustered {+ return false+ }+ s.clusteredMetrics.Store(true)+ return true+}diff --git a/pkg/metrics/clustered_layout_test.go b/pkg/metrics/clustered_layout_test.gonew file mode 100644index 0000000000..a1a6c1c14e--- /dev/null+++ b/pkg/metrics/clustered_layout_test.go@@ -0,0 +1,405 @@+package metrics++import (+ "context"+ "database/sql"+ "fmt"+ "os"+ "os/exec"+ "path/filepath"+ "reflect"+ "strings"+ "sync"+ "testing"+ "time"++ pdb "github.com/rcourtman/pulse-go-rewrite/pkg/db"+)++// Pause deferred maintenance, not the constructor, so tests can exercise the+// installed legacy schema and the cutover without racing a background upgrade.+func clusteredLegacyStore(t *testing.T) (*Store, func()) {+ t.Helper()+ suppressTestLogs(t)+ dbPath := filepath.Join(t.TempDir(), "metrics.db")+ issue1124CreateLegacyStore(t, dbPath, 100, false)+ entered, release := make(chan struct{}), make(chan struct{})+ previous := startupMaintenanceHook+ startupMaintenanceHook = func() { close(entered); <-release }+ cfg := DefaultConfig(filepath.Dir(dbPath))+ cfg.DBPath = dbPath+ cfg.FlushInterval, cfg.RollupInterval = time.Hour, time.Hour+ cfg.RetentionRaw = 10 * 365 * 24 * time.Hour+ store, err := NewStore(cfg)+ startupMaintenanceHook = previous+ var once sync.Once+ unblock := func() { once.Do(func() { close(release) }) }+ if err != nil {+ unblock()+ t.Fatal(err)+ }+ t.Cleanup(func() { unblock(); _ = store.Close() })+ select {+ case <-entered:+ case <-time.After(5 * time.Second):+ t.Fatal("deferred maintenance did not start")+ }+ return store, unblock+}++func assertClusteredStore(t *testing.T, store *Store) {+ t.Helper()+ clustered, err := metricsTableIsClustered(context.Background(), store.db.DB)+ if err != nil || !clustered {+ t.Fatalf("clustered schema=%t, error=%v", clustered, err)+ }+ issue1124AssertConsolidatedIndexes(t, store)+ var integrity string+ if err := store.db.QueryRow(`PRAGMA integrity_check`).Scan(&integrity); err != nil || integrity != "ok" {+ t.Fatalf("integrity=%q, error=%v", integrity, err)+ }+}++func TestClusteredMetricsFreshAndRestart(t *testing.T) {+ cfg := DefaultConfig(t.TempDir())+ for i := 0; i < 2; i++ {+ store, err := NewStore(cfg)+ if err != nil {+ t.Fatal(err)+ }+ if err := store.WaitForMaintenance(30 * time.Second); err != nil {+ t.Fatal(err)+ }+ assertClusteredStore(t, store)+ if err := store.Close(); err != nil {+ t.Fatal(err)+ }+ }+}++func TestClusteredMetricsPreservesObservationsAndMetadata(t *testing.T) {+ store, _ := clusteredLegacyStore(t)+ base := time.Unix(1_700_000_000, 0)+ for i, tier := range []Tier{TierRaw, TierMinute, TierHourly, TierDaily} {+ for j, resource := range []string{"vm", "dockercontainer"} {+ for k, bounds := range [][2]any{{nil, nil}, {0.0, nil}, {nil, 0.0}, {-4.0, 12.0}} {+ _, err := store.db.Exec(`INSERT INTO metrics+ (resource_type, resource_id, metric_type, value, min_value, max_value, timestamp, tier)+ VALUES (?, ?, 'cpu', ?, ?, ?, ?, ?)`, resource, fmt.Sprintf("case-%d", j),+ float64(k+1), bounds[0], bounds[1], base.Add(time.Duration(i*86400+k*60)*time.Second).Unix(), tier)+ if err != nil {+ t.Fatal(err)+ }+ }+ }+ }+ if _, err := store.db.Exec(`INSERT INTO metrics_meta VALUES ('rollup_sentinel', '1700000000')`); err != nil {+ t.Fatal(err)+ }+ // The exact stored nullable bounds, observation names, tiers and values+ // must survive, independently of any reader's normalization or fallback.+ read := func() []string {+ rows, err := store.db.Query(`SELECT resource_type, resource_id, metric_type,+ quote(value), quote(min_value), quote(max_value), timestamp, tier+ FROM metrics ORDER BY resource_type, resource_id, tier, timestamp, metric_type`)+ if err != nil {+ t.Fatal(err)+ }+ defer rows.Close()+ var result []string+ for rows.Next() {+ var resource, id, metric, value, low, high, tier string+ var ts int64+ if err := rows.Scan(&resource, &id, &metric, &value, &low, &high, &ts, &tier); err != nil {+ t.Fatal(err)+ }+ result = append(result, fmt.Sprint(resource, id, metric, value, low, high, ts, tier))+ }+ if err := rows.Err(); err != nil {+ t.Fatal(err)+ }+ return result+ }+ before := read()+ if err := store.migrateClusteredMetrics(context.Background()); err != nil {+ t.Fatal(err)+ }+ if after := read(); !reflect.DeepEqual(before, after) {+ t.Fatalf("stored observations changed across clustered migration")+ }+ var sentinel string+ if err := store.db.QueryRow(`SELECT value FROM metrics_meta WHERE key='rollup_sentinel'`).Scan(&sentinel); err != nil || sentinel != "1700000000" {+ t.Fatalf("rollup metadata=%q, error=%v", sentinel, err)+ }+ assertClusteredStore(t, store)+}++func TestClusteredMetricsReadersAndQueuedWritesSurviveCutover(t *testing.T) {+ store, _ := clusteredLegacyStore(t)+ entered, release := make(chan struct{}), make(chan struct{})+ var once sync.Once+ unblock := func() { once.Do(func() { close(release) }) }+ defer unblock()+ store.clusteredMigrationHook = func(stage string) {+ if stage == "after-copy" {+ close(entered)+ <-release+ }+ }+ // An already open SQLite snapshot must retain its original schema while+ // the other connection swaps and drops the physical table.+ tx, err := store.db.BeginTx(context.Background(), &sql.TxOptions{ReadOnly: true})+ if err != nil {+ t.Fatal(err)+ }+ defer tx.Rollback()+ var count int+ if err := tx.QueryRow(`SELECT COUNT(*) FROM metrics INDEXED BY idx_metrics_query_all`).Scan(&count); err != nil {+ t.Fatal(err)+ }+ migrated := make(chan error, 1)+ go func() { migrated <- store.migrateClusteredMetrics(context.Background()) }()+ select {+ case <-entered:+ case <-time.After(5 * time.Second):+ t.Fatal("migration did not copy")+ }+ points, err := store.Query("vm", "legacy-0001", "cpu", time.Unix(1_699_999_999, 0), time.Unix(1_700_001_000, 0), 0)+ if err != nil || len(points) != 1 {+ t.Fatalf("reader during copy points=%d, error=%v", len(points), err)+ }+ written := make(chan struct{})+ go func() {+ store.WriteBatchSync([]WriteMetric{{ResourceType: "vm", ResourceID: "queued", MetricType: "cpu", Value: 42, Timestamp: time.Unix(1_700_020_000, 0), Tier: TierRaw}})+ close(written)+ }()+ unblock()+ if err := <-migrated; err != nil {+ t.Fatal(err)+ }+ if err := tx.QueryRow(`SELECT COUNT(*) FROM metrics INDEXED BY idx_metrics_query_all`).Scan(&count); err != nil || count != 100 {+ t.Fatalf("old read snapshot count=%d, error=%v", count, err)+ }+ _ = tx.Rollback()+ select {+ case <-written:+ case <-time.After(35 * time.Second):+ t.Fatal("queued write did not resume")+ }+ var value float64+ if err := store.db.QueryRow(`SELECT value FROM metrics WHERE resource_id='queued'`).Scan(&value); err != nil || value != 42 {+ t.Fatalf("queued write=%v, error=%v", value, err)+ }+ assertClusteredStore(t, store)+}++func TestClusteredMetricsCancellationRollsBackSwap(t *testing.T) {+ store, _ := clusteredLegacyStore(t)+ ctx, cancel := context.WithCancel(context.Background())+ defer cancel()+ store.clusteredMigrationHook = func(stage string) {+ if stage == "before-commit" {+ cancel()+ }+ }+ if err := store.migrateClusteredMetrics(ctx); err == nil {+ t.Fatal("canceled migration succeeded")+ }+ clustered, err := metricsTableIsClustered(context.Background(), store.db.DB)+ if err != nil || clustered {+ t.Fatalf("canceled swap changed layout=%t, error=%v", clustered, err)+ }+ var count int+ if err := store.db.QueryRow(`SELECT COUNT(*) FROM metrics`).Scan(&count); err != nil || count != 100 {+ t.Fatalf("rollback rows=%d, error=%v", count, err)+ }+ if err := store.db.QueryRow(`SELECT COUNT(*) FROM sqlite_schema WHERE name='metrics_clustered_upgrade'`).Scan(&count); err != nil || count != 0 {+ t.Fatalf("rollback replacement tables=%d, error=%v", count, err)+ }+ store.clusteredMigrationHook = nil+ if err := store.migrateClusteredMetrics(context.Background()); err != nil {+ t.Fatal(err)+ }+ assertClusteredStore(t, store)+}++func TestClusteredMetricsPreservesConflictingReplacementTable(t *testing.T) {+ store, _ := clusteredLegacyStore(t)+ if _, err := store.db.Exec(`CREATE TABLE metrics_clustered_upgrade(value TEXT);+ INSERT INTO metrics_clustered_upgrade VALUES ('preserve')`); err != nil {+ t.Fatal(err)+ }+ if err := store.migrateClusteredMetrics(context.Background()); err == nil {+ t.Fatal("conflicting table was overwritten")+ }+ var value string+ if err := store.db.QueryRow(`SELECT value FROM metrics_clustered_upgrade`).Scan(&value); err != nil || value != "preserve" {+ t.Fatalf("conflicting table value=%q, error=%v", value, err)+ }+ var count int+ if err := store.db.QueryRow(`SELECT COUNT(*) FROM metrics`).Scan(&count); err != nil || count != 100 {+ t.Fatalf("original rows=%d, error=%v", count, err)+ }+}++func TestClusteredMetricsCrashRollsBackSwap(t *testing.T) {+ if dbPath := os.Getenv("PULSE_CLUSTERED_CRASH_DB"); dbPath != "" {+ metricsClusteredMigrationHook = func(stage string) {+ if stage == "before-commit" {+ os.Exit(88)+ }+ }+ cfg := DefaultConfig(filepath.Dir(dbPath))+ cfg.DBPath = dbPath+ cfg.RetentionRaw = 10 * 365 * 24 * time.Hour+ store, err := NewStore(cfg)+ if err != nil {+ os.Exit(90)+ }+ _ = store.WaitForMaintenance(30 * time.Second)+ os.Exit(89)+ }+ suppressTestLogs(t)+ dbPath := filepath.Join(t.TempDir(), "metrics.db")+ issue1124CreateLegacyStore(t, dbPath, 10_000, false)+ cmd := exec.Command(os.Args[0], "-test.run=^TestClusteredMetricsCrashRollsBackSwap$")+ cmd.Env = append(os.Environ(), "PULSE_CLUSTERED_CRASH_DB="+dbPath)+ err := cmd.Run()+ if exit, ok := err.(*exec.ExitError); !ok || exit.ExitCode() != 88 {+ t.Fatalf("crash child=%v, want exit 88", err)+ }+ cfg := DefaultConfig(filepath.Dir(dbPath))+ cfg.DBPath = dbPath+ cfg.RetentionRaw = 10 * 365 * 24 * time.Hour+ store, err := NewStore(cfg)+ if err != nil {+ t.Fatal(err)+ }+ defer store.Close()+ if err := store.WaitForMaintenance(30 * time.Second); err != nil {+ t.Fatal(err)+ }+ var count int+ if err := store.db.QueryRow(`SELECT COUNT(*) FROM metrics`).Scan(&count); err != nil || count != 10_000 {+ t.Fatalf("recovered rows=%d, error=%v", count, err)+ }+ assertClusteredStore(t, store)+}++func TestClusteredMetricsRefreshesAnotherStoreAndCachedShapes(t *testing.T) {+ store, _ := clusteredLegacyStore(t)+ // A second handle has its own physical-schema bit and bounded SQL caches.+ release := make(chan struct{})+ previous := startupMaintenanceHook+ startupMaintenanceHook = func() { <-release }+ other, err := NewStore(store.config)+ startupMaintenanceHook = previous+ if err != nil {+ close(release)+ t.Fatal(err)+ }+ t.Cleanup(func() { close(release); _ = other.Close() })+ start, end := time.Unix(1_699_999_999, 0), time.Unix(1_700_001_000, 0)+ for _, step := range []int64{0, 60} {+ if _, err := other.QueryAllBatch("vm", []string{"legacy-0001", "legacy-0002"}, start, end, step); err != nil {+ t.Fatal(err)+ }+ }+ if err := store.migrateClusteredMetrics(context.Background()); err != nil {+ t.Fatal(err)+ }+ for _, step := range []int64{0, 60} {+ // Force the other handle's stale layout, including a cached presence+ // statement, so both snapshot paths exercise the real missing-index error.+ other.clusteredMetrics.Store(false)+ points, err := other.QueryAllBatch("vm", []string{"legacy-0001", "legacy-0002"}, start, end, step)+ if err != nil || len(points) != 2 || !other.clusteredMetrics.Load() {+ t.Fatalf("stale full reader step=%d: resources=%d, refreshed=%t, error=%v", step, len(points), other.clusteredMetrics.Load(), err)+ }+ other.clusteredMetrics.Store(false)+ values, err := other.QueryMetricValuesBatch("vm", []string{"legacy-0001", "legacy-0002"}, nil, start, end, step)+ if err != nil || len(values) != 2 || !other.clusteredMetrics.Load() {+ t.Fatalf("stale value reader step=%d: resources=%d, refreshed=%t, error=%v", step, len(values), other.clusteredMetrics.Load(), err)+ }+ }+}++func TestClusteredMetricsReopeningLegacyExecutableIndexes(t *testing.T) {+ cfg := DefaultConfig(t.TempDir())+ store, err := NewStore(cfg)+ if err != nil {+ t.Fatal(err)+ }+ if err := store.WaitForMaintenance(30 * time.Second); err != nil {+ t.Fatal(err)+ }+ // Reproduce the previous runtime's initSchema/unique-index behavior on+ // the new table. Its unchanged upsert conflict target remains functional.+ if _, err := store.db.Exec(`CREATE UNIQUE INDEX idx_metrics_query_all+ ON metrics(resource_type, resource_id, tier, timestamp, metric_type);+ INSERT INTO metrics(resource_type, resource_id, metric_type, value, timestamp, tier)+ VALUES ('vm', 'downgrade', 'cpu', 1, 2000000000, 'raw')+ ON CONFLICT(resource_type, resource_id, metric_type, tier, timestamp) DO UPDATE SET value=excluded.value;+ INSERT INTO metrics(resource_type, resource_id, metric_type, value, timestamp, tier)+ VALUES ('vm', 'downgrade', 'cpu', 2, 2000000000, 'raw')+ ON CONFLICT(resource_type, resource_id, metric_type, tier, timestamp) DO UPDATE SET value=excluded.value;`); err != nil {+ t.Fatal(err)+ }+ if err := store.Close(); err != nil {+ t.Fatal(err)+ }+ reopened, err := NewStore(cfg)+ if err != nil {+ t.Fatal(err)+ }+ defer reopened.Close()+ if err := reopened.WaitForMaintenance(30 * time.Second); err != nil {+ t.Fatal(err)+ }+ assertClusteredStore(t, reopened)+ var value float64+ if err := reopened.db.QueryRow(`SELECT value FROM metrics WHERE resource_id='downgrade'`).Scan(&value); err != nil || value != 2 {+ t.Fatalf("legacy-compatible upsert value=%v, error=%v", value, err)+ }+}++func TestClusteredRetainedPlansUsePrimaryTable(t *testing.T) {+ db := newPlanTestDB(t)+ store := &Store{db: pdb.Wrap(db, "metrics"), config: DefaultConfig(t.TempDir())}+ if err := store.migrateClusteredMetrics(context.Background()); err != nil {+ t.Fatal(err)+ }+ end := time.Unix(2_000_000_040, 0)+ for _, valuesOnly := range []bool{false, true} {+ for _, window := range []time.Duration{time.Hour, 24 * time.Hour, 7 * 24 * time.Hour, 30 * 24 * time.Hour} {+ for _, ids := range [][]string{{"vm-1"}, {"vm-1", "vm-2"}} {+ for _, metrics := range [][]string{nil, {"cpu"}, {"cpu", "memory"}} {+ for _, step := range []int64{0, 60} {+ t.Run(fmt.Sprintf("values_%t/%s/resources_%d/metrics_%d/step_%d", valuesOnly, window, len(ids), len(metrics), step), func(t *testing.T) {+ query, args := store.retainedQuerySQL("vm", ids, metrics, end.Add(-window), end, step, store.tierFallbacks(window), false, valuesOnly)+ plan := explainQueryPlan(t, db, query, args)+ for _, line := range strings.Split(plan, "\n") {+ if strings.Contains(line, "SCAN m ") || strings.Contains(line, "SCAN h ") {+ t.Fatalf("unbounded clustered read: %s", plan)+ }+ if strings.Contains(line, "SEARCH m ") || strings.Contains(line, "SEARCH h ") {+ for _, constraint := range []string{"USING PRIMARY KEY", "resource_type=?", "resource_id=?", "tier=?", "timestamp>?", "timestamp<?"} {+ if !strings.Contains(line, constraint) {+ t.Fatalf("clustered lookup missing %s: %s", constraint, plan)+ }+ }+ }+ if step == 0 && strings.Contains(line, "TEMP B-TREE") {+ t.Fatalf("clustered unaggregated read sorts: %s", plan)+ }+ }+ if !strings.Contains(plan, "SEARCH m USING PRIMARY KEY") {+ t.Fatalf("missing primary table lookup: %s", plan)+ }+ })+ }+ }+ }+ }+ }+}diff --git a/pkg/metrics/identity_index_test.go b/pkg/metrics/identity_index_test.goindex d02d3d623d..334b7aca1f 100644--- a/pkg/metrics/identity_index_test.go+++ b/pkg/metrics/identity_index_test.go@@ -48,6 +48,7 @@ func newSteadyStateStore(tb testing.TB, layout string) (*Store, string) {tb.Fatalf("startup maintenance: %v", err)}if layout == "metric-major" {+ issue1124InstallRowidSchema(tb, store)if _, err := store.db.Exec(`DROP INDEX idx_metrics_query_all;CREATE INDEX idx_metrics_query_alldiff --git a/pkg/metrics/issue1124_write_amplification_test.go b/pkg/metrics/issue1124_write_amplification_test.goindex 6959f93dbc..1e0aaa255c 100644--- a/pkg/metrics/issue1124_write_amplification_test.go+++ b/pkg/metrics/issue1124_write_amplification_test.go@@ -188,8 +188,10 @@ func TestStoreIdentityMigrationDeduplicatesLegacyRows(t *testing.T) {func TestStoreIdentityMigrationRecoversAfterCrash(t *testing.T) {if os.Getenv("PULSE_METRICS_IDENTITY_CRASH_CHILD") == "1" {- metricsIdentityMigrationHook = func() {- os.Exit(88)+ metricsClusteredMigrationHook = func(stage string) {+ if stage == "before-commit" {+ os.Exit(88)+ }}dbPath := os.Getenv("PULSE_METRICS_IDENTITY_CRASH_DB")cfg := DefaultConfig(filepath.Dir(dbPath))@@ -315,12 +317,14 @@ func TestStoreIdentityMigrationKeepsReadersAvailable(t *testing.T) {migrationEntered := make(chan struct{})releaseMigration := make(chan struct{})var hookOnce sync.Once- metricsIdentityMigrationHook = func() {- hookOnce.Do(func() { close(migrationEntered) })- <-releaseMigration+ metricsClusteredMigrationHook = func(stage string) {+ if stage == "after-copy" {+ hookOnce.Do(func() { close(migrationEntered) })+ <-releaseMigration+ }}t.Cleanup(func() {- metricsIdentityMigrationHook = nil+ metricsClusteredMigrationHook = nil})cfg := DefaultConfig(filepath.Dir(dbPath))@@ -497,8 +501,9 @@ func TestMetricsWriteAmplificationInvariant(t *testing.T) {//// Run with PULSE_METRICS_WRITE_PROFILE=full (v6.1.1 layout), lookup (the// v6.1.2 to v6.4 metric-major unique index) or consolidated (the time-major-// identity index), each optionally -batched, and optionally-// PULSE_METRICS_WRITE_PROFILE_TICKS=N. The default 30 ticks+// identity index), each optionally -batched, or clustered (the production+// time-major primary table). PULSE_METRICS_WRITE_PROFILE_TICKS=N+// overrides the default 30 ticks, which// persist 393,630 samples across 2,197 resources. Set// PULSE_METRICS_WRITE_PROFILE_AUTOCHECKPOINT=1 for production checkpoint/fsync// tracing, PULSE_METRICS_WRITE_PROFILE_CHECKPOINT_PAGES=N for a bounded@@ -511,8 +516,8 @@ func TestIssue1124WriteAmplificationProfile(t *testing.T) {t.Skip("set " + issue1124ProfileEnv + " to run the write-amplification profile")}if mode != "full" && mode != "lookup" && mode != "consolidated" && mode != "full-batched" &&- mode != "lookup-batched" && mode != "consolidated-batched" && mode != "without-rowid" {- t.Fatalf("%s must be full, lookup, consolidated, full-batched, lookup-batched, consolidated-batched, or without-rowid, got %q", issue1124ProfileEnv, mode)+ mode != "lookup-batched" && mode != "consolidated-batched" && mode != "without-rowid" && mode != "clustered" {+ t.Fatalf("%s must be full, lookup, consolidated, full-batched, lookup-batched, consolidated-batched, without-rowid, or clustered, got %q", issue1124ProfileEnv, mode)}ticks := 30@@ -562,6 +567,12 @@ func TestIssue1124WriteAmplificationProfile(t *testing.T) {issue1124ConfigureConnections(t, store, connectionCount, checkpointPages)autoCheckpoint := checkpointPages > 0+ if mode != "clustered" {+ // Keep historical profiles as actual rowid schemas after production+ // moves to clustered storage. No rows have been written yet.+ issue1124InstallRowidSchema(t, store)+ }+if mode == "full" || mode == "full-batched" {// Reconstruct the v6.1.1 schema so released and consolidated layouts// remain comparable after the production migration lands.@@ -1211,12 +1222,13 @@ func issue1124CreateLegacyStore(t *testing.T, dbPath string, rows int, duplicatefunc issue1124AssertConsolidatedIndexes(t *testing.T, store *Store) {t.Helper()- matches, err := store.metricsIndexMatches(metricsIdentityIndex, true, metricsIdentityColumns)+ index := store.metricsReadIndex()+ matches, err := store.metricsIndexMatches(index, true, metricsIdentityColumns)if err != nil {t.Fatalf("inspect consolidated identity index: %v", err)}if !matches {- t.Fatalf("%s is not the expected unique time-major identity index", metricsIdentityIndex)+ t.Fatalf("%s is not the expected unique time-major identity index", index)}rows, err := store.db.Query(`PRAGMA index_list(metrics)`)@@ -1244,7 +1256,7 @@ func issue1124AssertConsolidatedIndexes(t *testing.T, store *Store) {// Retired metric-major trees (idx_metrics_lookup, idx_metrics_unique)// must be gone: each would be dirtied by every insert.want := map[string]bool{- "idx_metrics_query_all": true,+ index: true,"idx_metrics_tier_time": false,}if len(indexes) != len(want) {@@ -1256,3 +1268,25 @@ func issue1124AssertConsolidatedIndexes(t *testing.T, store *Store) {}}}++// issue1124InstallRowidSchema reconstructs the released time-major rowid table+// in an empty measurement fixture, preserving comparisons across schema changes.+func issue1124InstallRowidSchema(tb testing.TB, store *Store) {+ tb.Helper()+ if _, err := store.db.Exec(`+ DROP TABLE metrics;+ CREATE TABLE metrics (+ id INTEGER PRIMARY KEY AUTOINCREMENT,+ resource_type TEXT NOT NULL, resource_id TEXT NOT NULL,+ metric_type TEXT NOT NULL, value REAL NOT NULL,+ min_value REAL, max_value REAL,+ timestamp INTEGER NOT NULL, tier TEXT NOT NULL DEFAULT 'raw'+ );+ CREATE INDEX idx_metrics_tier_time ON metrics(tier, timestamp);+ CREATE UNIQUE INDEX idx_metrics_query_all+ ON metrics(resource_type, resource_id, tier, timestamp, metric_type);+ `); err != nil {+ tb.Fatalf("install historical rowid table: %v", err)+ }+ store.clusteredMetrics.Store(false)+}diff --git a/pkg/metrics/store.go b/pkg/metrics/store.goindex 88328223b2..87c7fb0d78 100644--- a/pkg/metrics/store.go+++ b/pkg/metrics/store.go@@ -258,8 +258,8 @@ type maintenanceRequest struct {}var (- startupMaintenanceHook func()- metricsIdentityMigrationHook func()+ startupMaintenanceHook func()+ metricsClusteredMigrationHook func(string))// Store provides persistent metrics storage@@ -295,14 +295,15 @@ type Store struct {stopOnce sync.Oncestopping atomic.BoolsyncWaitWarnNano atomic.Int64- identityMigrationPending atomic.Bool+ clusteredMetrics atomic.BoolstartupMaintenanceActive atomic.BoolcommercialRetentionSeconds atomic.Int64commercialPurgeEligibleAt atomic.Int64// startupHook is captured once at construction so a store that outlives// the test that built it never invokes a later test's hook.- startupHook func()+ startupHook func()+ clusteredMigrationHook func(string)}// SetCommercialHistoryRetention applies a delayed commercial ceiling to the@@ -387,15 +388,16 @@ func NewStore(config StoreConfig) (*Store, error) {db := pdb.Wrap(rawDB, "metrics")store := &Store{- db: db,- config: config,- buffer: make([]bufferedMetric, 0, config.WriteBufferSize),- writeCh: make(chan writeRequest, 100), // Buffer for write batches- maintenanceCh: make(chan maintenanceRequest, 1),- stopCh: make(chan struct{}),- doneCh: make(chan struct{}),- maintenanceDoneCh: make(chan struct{}),- startupHook: startupMaintenanceHook,+ db: db,+ config: config,+ buffer: make([]bufferedMetric, 0, config.WriteBufferSize),+ writeCh: make(chan writeRequest, 100), // Buffer for write batches+ maintenanceCh: make(chan maintenanceRequest, 1),+ stopCh: make(chan struct{}),+ doneCh: make(chan struct{}),+ maintenanceDoneCh: make(chan struct{}),+ startupHook: startupMaintenanceHook,+ clusteredMigrationHook: metricsClusteredMigrationHook,}// Initialize schema@@ -453,28 +455,12 @@ func NewStore(config StoreConfig) (*Store, error) {func (s *Store) initSchema() error {schema := `-- Main metrics table- CREATE TABLE IF NOT EXISTS metrics (- id INTEGER PRIMARY KEY AUTOINCREMENT,- resource_type TEXT NOT NULL,- resource_id TEXT NOT NULL,- metric_type TEXT NOT NULL,- value REAL NOT NULL,- min_value REAL,- max_value REAL,- timestamp INTEGER NOT NULL,- tier TEXT NOT NULL DEFAULT 'raw'- );+ CREATE TABLE IF NOT EXISTS metrics (` + metricsTableColumns + `) WITHOUT ROWID;-- Index for retention pruningCREATE INDEX IF NOT EXISTS idx_metrics_tier_timeON metrics(tier, timestamp);- -- Identity and range-read index. Created non-unique here so reads- -- can name it on every schema generation; ensureMetricsIdentityIndex- -- makes it the unique identity index.- CREATE INDEX IF NOT EXISTS idx_metrics_query_all- ON metrics(resource_type, resource_id, tier, timestamp, metric_type);--- Metadata table for tracking rollup stateCREATE TABLE IF NOT EXISTS metrics_meta (key TEXT PRIMARY KEY,@@ -487,8 +473,21 @@ func (s *Store) initSchema() error {return fmt.Errorf("failed to create schema: %w", err)}- if err := s.ensureMetricsIdentityIndex(); err != nil {- return err+ clustered, err := metricsTableIsClustered(context.Background(), s.db.DB)+ if err != nil {+ return fmt.Errorf("inspect metrics physical layout: %w", err)+ }+ s.clusteredMetrics.Store(clustered)+ if !clustered {+ // Existing rowid stores continue serving on their authoritative index+ // until deferred maintenance can atomically replace the table.+ if _, err := s.db.Exec(`CREATE INDEX IF NOT EXISTS idx_metrics_query_all+ ON metrics(resource_type, resource_id, tier, timestamp, metric_type)`); err != nil {+ return err+ }+ if err := s.ensureMetricsIdentityIndex(); err != nil {+ return err+ }}if err := s.initAvailabilityHistorySchema(); err != nil {return err@@ -498,7 +497,7 @@ func (s *Store) initSchema() error {return nil}-// metricsIdentityIndex is the one B-tree that enforces sample identity and+// metricsIdentityIndex is the legacy B-tree that enforces sample identity and// serves every range read. Its columns are time-major within a resource, so// the samples one poll writes for a resource share leaf pages. The retired// metric-major order gave every series its own insertion point, and each@@ -531,10 +530,10 @@ func (s *Store) ensureMetricsIdentityIndex() error {}if current || retiredUnique {// Identity is already enforced by a unique index. Keep serving with- // that crash-safe schema and defer the O(rows) rebuild to the+ // that crash-safe schema and defer the O(rows) table replacement to the// startup-maintenance worker so NewStore latency stays bounded.- s.identityMigrationPending.Store(true)- log.Info().Strs("retired_indexes", retired).Msg("Scheduled metrics identity index consolidation")+ // Rebuilding its index first would immediately discard that work.+ log.Info().Strs("retired_indexes", retired).Msg("Scheduled clustered metrics table migration")return nil}@@ -573,6 +572,16 @@ func (s *Store) existingRetiredMetricsIdentityIndexes() (existing []string, anyU}func (s *Store) migrateMetricsIdentityIndex() error {+ // Another handle may have completed the physical upgrade while this+ // store's deferred maintenance was queued. Do not rebuild a redundant tree.+ clustered, err := metricsTableIsClustered(context.Background(), s.db.DB)+ if err != nil {+ return fmt.Errorf("inspect metrics layout before identity migration: %w", err)+ }+ if clustered {+ s.clusteredMetrics.Store(true)+ return nil+ }if err := s.replaceMetricsIdentityIndex(); err == nil {log.Info().Msg("Consolidated metrics identity index")return nil@@ -611,17 +620,12 @@ func (s *Store) replaceMetricsIdentityIndex() error {if _, err := tx.Exec(`DROP INDEX IF EXISTS ` + metricsIdentityIndex); err != nil {return fmt.Errorf("drop non-unique identity index: %w", err)}- if metricsIdentityMigrationHook != nil {- metricsIdentityMigrationHook()- }if _, err := tx.Exec(`CREATE UNIQUE INDEX ` + metricsIdentityIndex + `ON metrics(` + strings.Join(metricsIdentityColumns, ", ") + `)`); err != nil {return fmt.Errorf("create unique identity index: %w", err)}- } else if metricsIdentityMigrationHook != nil {- metricsIdentityMigrationHook()}for _, name := range retiredMetricsIdentityIndexes {if _, err := tx.Exec(`DROP INDEX IF EXISTS ` + name); err != nil {@@ -762,6 +766,10 @@ func (s *Store) migrateAutoVacuum() {ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)defer cancel()+ s.migrateAutoVacuumContext(ctx)+}++func (s *Store) migrateAutoVacuumContext(ctx context.Context) {// auto_vacuum is only a per-connection setting until a VACUUM on that same// connection rewrites the file header. Running the pragma and the VACUUM on// separate pool connections could leave the file at NONE, so pin both to one@@ -1092,7 +1100,7 @@ func (s *Store) runStartupMaintenance() {start := time.Now()// Signal the ingestion worker that the write lock may be held for a long- // time by the deferred identity-index rebuild and the auto-vacuum VACUUM.+ // time by the deferred table migration and the auto-vacuum VACUUM.// Without this a large legacy database drops every metric batch enqueued// during the one-time upgrade (#2079 finding 5).s.startupMaintenanceActive.Store(true)@@ -1102,10 +1110,20 @@ func (s *Store) runStartupMaintenance() {s.startupHook()}- if s.identityMigrationPending.Swap(false) {- if err := s.migrateMetricsIdentityIndex(); err != nil {- log.Error().Err(err).Msg("Deferred metrics identity index consolidation failed")+ // Share the existing half-hour rewrite budget between physical layout+ // and auto-vacuum, and interrupt both when the Store is closed.+ ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)+ defer cancel()+ go func() {+ select {+ case <-s.stopCh:+ cancel()+ case <-ctx.Done():}+ }()++ if err := s.migrateClusteredMetrics(ctx); err != nil {+ log.Warn().Err(err).Msg("Clustered metrics migration deferred until next startup")}// Run retention before the deferred auto-vacuum conversion so restart@@ -1113,7 +1131,7 @@ func (s *Store) runStartupMaintenance() {// potentially rewrites the file.s.runRetention()s.runAvailabilityRetention()- s.migrateAutoVacuum()+ s.migrateAutoVacuumContext(ctx)log.Info().Dur("duration", time.Since(start)).Msg("Deferred metrics startup maintenance completed")}@@ -1614,7 +1632,7 @@ type retainedQueryShape struct {resources, metrics inttiers [4]Tieraggregate, groupSeries bool- valuesOnly bool+ valuesOnly, clustered bool}func retainedQueryParameters(resourceType string, resourceIDs, metricTypes []string, start, end time.Time, stepSecs int64, tiers []Tier) []interface{} {@@ -1642,7 +1660,11 @@ func retainedQueryParameters(resourceType string, resourceIDs, metricTypes []str}func (s *Store) retainedQuerySQL(resourceType string, resourceIDs, metricTypes []string, start, end time.Time, stepSecs int64, tiers []Tier, groupSeries, valuesOnly bool) (string, []interface{}) {- shape := retainedQueryShape{resources: len(resourceIDs), metrics: len(metricTypes), aggregate: stepSecs > 1, groupSeries: groupSeries, valuesOnly: valuesOnly}+ return s.retainedQuerySQLForIndex(s.metricsReadIndex(), resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers, groupSeries, valuesOnly)+}++func (s *Store) retainedQuerySQLForIndex(index string, resourceType string, resourceIDs, metricTypes []string, start, end time.Time, stepSecs int64, tiers []Tier, groupSeries, valuesOnly bool) (string, []interface{}) {+ shape := retainedQueryShape{resources: len(resourceIDs), metrics: len(metricTypes), aggregate: stepSecs > 1, groupSeries: groupSeries, valuesOnly: valuesOnly, clustered: index == clusteredMetricsIndex}cacheable := len(tiers) <= len(shape.tiers)copy(shape.tiers[:], tiers)if cacheable {@@ -1653,7 +1675,7 @@ func (s *Store) retainedQuerySQL(resourceType string, resourceIDs, metricTypes [return query, retainedQueryParameters(resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers)}}- query, params := retainedQuerySQL(resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers, groupSeries, valuesOnly)+ query, params := retainedQuerySQLWithIndex(index, resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers, groupSeries, valuesOnly)if cacheable {s.queryMu.Lock()if len(s.readQueryShapes) < maxRetainedReadStatements {@@ -1673,6 +1695,10 @@ func (s *Store) retainedQuerySQL(resourceType string, resourceIDs, metricTypes [// Presence never establishes continuous collection or per-series coverage.// Every probe is restricted to the requested identities and timestamp window.func retainedQuerySQL(resourceType string, resourceIDs, metricTypes []string, start, end time.Time, stepSecs int64, tiers []Tier, groupSeries, valuesOnly bool) (string, []interface{}) {+ return retainedQuerySQLWithIndex(metricsIdentityIndex, resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers, groupSeries, valuesOnly)+}++func retainedQuerySQLWithIndex(index string, resourceType string, resourceIDs, metricTypes []string, start, end time.Time, stepSecs int64, tiers []Tier, groupSeries, valuesOnly bool) (string, []interface{}) {params := retainedQueryParameters(resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers)identityColumns := ""if len(resourceIDs) != 1 {@@ -1696,7 +1722,6 @@ func retainedQuerySQL(resourceType string, resourceIDs, metricTypes []string, stendParam := startParam + 1tierParam := endParam + 1stepParam := tierParam + len(tiers)- index := metricsIdentityIndexscope := func(alias string, tierIndex int) string {clause := alias + ".resource_type = :p1 AND " + alias + ".resource_id IN (" + idSlots + ")"if len(metricTypes) > 0 {@@ -1806,6 +1831,16 @@ func (s *Store) retainedReadStatement(query string) (*pdb.InstrumentedStmt, erro}func queryRetainedChunk[T any](s *Store, resourceType string, resourceIDs []string, metricTypes []string, start, end time.Time, stepSecs int64, tiers []Tier, projection retainedProjection[T]) (map[string]map[string][]T, error) {+ result, err := queryRetainedChunkAttempt(s, resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers, projection)+ if err != nil && s.refreshClusteredMetrics(err) {+ return queryRetainedChunkAttempt(s, resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers, projection)+ }+ return result, err+}++func queryRetainedChunkAttempt[T any](s *Store, resourceType string, resourceIDs []string, metricTypes []string, start, end time.Time, stepSecs int64, tiers []Tier, projection retainedProjection[T]) (map[string]map[string][]T, error) {+ // Keep the index selection fixed for this read snapshot across cutover.+ index := s.metricsReadIndex()var tx *pdb.InstrumentedTxvar err errorif stepSecs > 1 {@@ -1817,7 +1852,6 @@ func queryRetainedChunk[T any](s *Store, resourceType string, resourceIDs []stri// order the value projection for each presence probe.idSlots := strings.TrimSuffix(strings.Repeat("?,", len(resourceIDs)), ",")presenceScope := "resource_type = ? AND resource_id IN (" + idSlots + ")"- index := metricsIdentityIndexif len(metricTypes) > 0 {presenceScope += " AND metric_type IN (" + strings.TrimSuffix(strings.Repeat("?,", len(metricTypes)), ",") + ")"}@@ -1881,7 +1915,7 @@ func queryRetainedChunk[T any](s *Store, resourceType string, resourceIDs []striif streamBuckets {queryStep = 0}- sqlQuery, params := s.retainedQuerySQL(resourceType, resourceIDs, metricTypes, start, end, queryStep, tiers, streamBuckets, projection.valuesOnly)+ sqlQuery, params := s.retainedQuerySQLForIndex(index, resourceType, resourceIDs, metricTypes, start, end, queryStep, tiers, streamBuckets, projection.valuesOnly)queryRows := func() (*sql.Rows, error) { return tx.Query(sqlQuery, params...) }if tx == nil {