Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Cluster retained metrics storage to remove chart row lookups

rcourtman5c37e620b45b
diff --git a/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md b/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md
new file mode 100644
index 0000000000..9b938810c5
--- /dev/null
+++ b/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md
@@ -0,0 +1,119 @@
+# Workload chart storage and idle-worker qualification, 18 October 2026
+
+Scope: the remaining cold-service four-hour workload-chart latency gap under
+L10 `performance-post-rc-headroom`. The acceptance budget remains 90 ms p95.
+Source landing, a relative benchmark, and an idle-worker SLO proof are separate
+observations. This record does not close the broad L10 residual.
+
+## Retained failed proof and worker inventory
+
+The installed disposable `pulse-ci-vm-large` profile provides two vCPUs and
+4 GiB of RAM. The persistent worker shares other workloads. A configured
+performance profile alone is not an available trusted proof route. The manual
+source-bound route is owned by the L10 contract at
+`pulse-enterprise:.github/workflows/verify-workload-chart-latency.yml` and its
+runbook `pulse-enterprise:scripts/workload-chart-latency-proof.md`.
+
+[Run 39168929815](https://github.com/rcourtman/pulse-enterprise/actions/runs/39168929815)
+used public source `8a590102fef2089e938815bea280fee5423334fc`, private control
+`4f8f9542ee122471c0bfe4f51334e7ef0d19796b`, Go 1.26.9 and the disposable runner
+`pulse-ci-vm-enterprise-large-1018214122-359c52` (two Intel Xeon CPUs). Executable
+SHA-256: `0bec46b33b3f3a1e23adac8309093966d528c4d6b8c9b2b2786b3e1786255553`.
+
+Its unchanged strict proof failed: p50 99.322850 ms, p95 112.164352 ms and p99
+121.091084 ms. Admission was quiet, but one proof interval recorded 0.49505%
+CPU steal against the existing 0.1% limit. The result therefore establishes
+neither the latency budget nor an idle-worker proof. All 24 during-proof
+capacity samples, admission/after samples and diagnostics are retained. The
+logs were extracted with original per-file byte counts verified, without
+omitting the failing sample or substituting a second SLO attempt.
+
+The separate diagnostic benchmark measured 99.635869 ms/op. Its CPU profile
+attributed 80.55% cumulative CPU to the retained query, 52.74% to SQLite's VM and
+21.18% to `BtreeTableMoveto`. Finding each identity entry then seeking its rowid
+table payload remained a substantial cost. The pinned SQLite driver opens
+FULLMUTEX connections; an unsupported connection-string option would not
+remove that cost. Cache and mmap size experiments did not demonstrate a
+reliable benefit and were not adopted.
+
+## Canonical storage change
+
+`pkg/metrics/clustered_layout.go` stores observations in
+`metrics WITHOUT ROWID`, keyed by
+`(resource_type, resource_id, tier, timestamp, metric_type)`. Values and nullable
+bounds now live in the same B-tree used by retained range reads. The only
+secondary tree remains `(tier, timestamp)` for retention. The identity column
+set and all eight observation columns retain their existing semantics.
+
+Installed rowid databases keep their authoritative unique index while deferred
+startup maintenance copies and swaps the table in one `BEGIN IMMEDIATE` WAL
+transaction. Existing readers retain their old SQLite snapshots. Writes queued
+behind the transaction resume after commit. Copy failure, cancellation or a
+crash before commit rolls back the entire schema and payload change. A
+conflicting replacement table is preserved. The free-space guard requires
+three times the used database pages plus 16 MiB; inadequate capacity defers the
+migration and leaves the installed format readable. Layout and auto-vacuum
+share the existing 30-minute rewrite context, canceled on Store shutdown.
+
+Legacy stores with no authoritative unique index still establish uniqueness
+synchronously before upserts. Already-authoritative stores skip the redundant
+old index rebuild and migrate directly. Readers fix their index selection for
+each snapshot; a stale legacy-index error retries once only after verifying
+the new primary key. The physical layout participates in the bounded SQL cache
+shape. No results or tier presence are cached. An older executable can reopen
+the row-compatible table and recreate its old index; subsequent maintenance
+removes those known redundant indexes after checking the clustered primary key.
+
+The retained query engine, tier reconciliation, Docker observation provenance,
+nullable-bound fallback, value projection, LTTB sampling, chart point limits,
+cache bounds, write retry policy and SLO thresholds remain the same.
+
+## Qualification evidence
+
+The parent-owned native evidence directory is
+`/opt/pulse-release-worker/task-proofs/chart-latency-slo-20261018/clustered-qualification`.
+The initial full metrics suite passed in 219.902 s. The 12-tick deterministic
+write profile preserved 157,452 observations across 2,197 resources in 72 writes:
+20,842 WAL frames (85,869,072 bytes), below the unchanged 23,000-frame ceiling.
+The rowid comparison used 20,223 frames, so the clustered format trades about
+3.06% more WAL traffic for removal of the per-observation table lookup. Its
+checkpoint wrote 20,672,512 bytes and the main database used 20,725,760 bytes,
+versus 23,797,760 and 23,846,912 respectively in the rowid comparison. These
+physical write/size observations are distinct from latency qualification.
+
+With the production 4,000-frame checkpoint threshold, four connections and a
+paced concurrent reader, the initial 30-tick profile preserved 393,630 rows
+through restart and integrity checking. It completed 2,107 reads (p95 0.497286
+ms, maximum 17.202902 ms), and retention removed every row. The shared native
+worker is unsuitable for an absolute chart SLO claim; these profiles qualify
+storage behavior only. Final source-bound verification is recorded separately
+from these initial diagnostics.
+
+The final fixed-executable production comparison used 393,630 observations,
+180 writes, 4,000-frame checkpointing and the same paced-reader configuration.
+The historical rowid profile wrote 728,674,304 process bytes; the clustered
+profile wrote 789,917,696 (+8.40%). Final explicit checkpoint writes were
+9,256,960 versus 10,158,080 bytes (+9.73%). Main database size was 61,054,976
+versus 52,621,312 bytes (-13.81%). Both profiles preserved every row through
+restart and integrity checking, then retained zero rows after purge. Their
+respective read counts were 18,917 and 4,080. The shared worker produced large
+write/read timing outliers, so these sequential profile timings do not prove
+a throughput or tail-latency improvement. Retain the increased write volume
+as a measured tradeoff; the production checkpoint threshold, 23,000-frame
+no-auto-checkpoint ceiling and steady-state write guard are unchanged.
+
+The final full metrics suite and migration race tests passed. The race run
+completed in 64.407 s, including the atomic-swap cancellation/crash paths,
+independent Store handles and cached read shapes. Temporary layout/cache probe
+code was hash-checked and moved out of the package before compiling the final
+fixed test executable.
+
+Final fixed metrics executable SHA-256:
+`aa501813ca9cd8d6de5d61ae03ac36005e640dd9aa812c9b6da7ca651b6dd689`.
+Per-file Go source hashes are retained in `clustered-final/source.sha256`.
+The v5 metrics compatibility tests, downstream monitoring/chart sampling tests
+(67.669 s), and 11 chart API tests (0.625 s) passed. The separate cold-service
+benchmark reported 111.005964 ms/op, 22,262,176 B/op and 508,181 allocs/op on the
+shared AMD Ryzen worker. That timing is retained without treating it as a
+90 ms SLO proof. Status and contract audits passed; the new explicitly routed
+test must be staged before the registry audit can recognize its tracked path.
diff --git a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md
index c3be9f638f..8699e9e6a8 100644
--- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md
+++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md
@@ -545,7 +545,7 @@ which retention tiers have observations in that snapshot. Every present tier
participates in the shared indexed overlap query. Presence never stands in for
per-series coverage. The store retains at most 32 compiled read-statement shapes
and 32 SQL templates keyed only by parameter counts, tier order and aggregation
-shape. Alphabetic named SQLite bindings supply each current identity, window and
+shape, projection and physical layout. Alphabetic named SQLite bindings supply each current identity, window and
display step once across all branches. Bindings use `database/sql.Named` so the
driver need not repeatedly convert ordinals to strings while matching a large
parameter set. Names describe parameter positions only. Resource IDs and all
@@ -633,6 +633,10 @@ in-place click and chevron activation over a selection.
- `frontend-modern/src/utils/filesystemByteUsage.ts`: shared finite byte/capacity admission for filesystem utilization.
1. `pkg/metrics/store.go`
+ 1a. `pkg/metrics/clustered_layout.go`
+ 1b. `pkg/metrics/clustered_layout_test.go`
+ 1c. `pkg/metrics/identity_index_test.go`
+ 1d. `pkg/metrics/issue1124_write_amplification_test.go`
2. `pkg/metrics/store_query_plan_test.go`
3. `pkg/metrics/store_slo_test.go`
4. `pkg/metrics/store_additional_test.go`
@@ -2698,9 +2702,13 @@ WAL checkpoints more aggressive again, reopens duplicate-write failures,
removes the bounded rollup-window checkpointing, or removes the metrics DB
path/cadence controls must re-prove the metrics-store hot path with the owned
store tests rather than assuming the earlier vacuum fixes are sufficient.
-Metric identity and every range read must also share one unique, time-major
-`idx_metrics_query_all(resource_type, resource_id, tier, timestamp,
-metric_type)` B-tree. Reintroducing a second full-width identity index is a
+Metric identity, stored values and every range read share the time-major
+`PRIMARY KEY(resource_type, resource_id, tier, timestamp, metric_type)` of
+`metrics WITHOUT ROWID`. The primary table itself supplies the samples; a
+separate rowid-table lookup per observation is no longer required. Its only
+secondary tree is the `(tier, timestamp)` retention index. Legacy rowid stores
+continue to use `idx_metrics_query_all` until deferred migration commits.
+Reintroducing a second full-width identity index is a
write-amplification regression even when retained `metrics.db` size looks small:
every sample would dirty both trees in the WAL and every checkpoint would copy
both sets of pages back into the main file. The column order is part of the
@@ -2738,12 +2746,35 @@ value points afterward. Full and value projections share scope normalization,
filters, tier reconciliation, time ordering, display buckets and Docker
observation provenance; their distinct SQL column shapes must have distinct
keys within the existing bounded query cache. Neither mode caches results or
-tier presence. Full readers continue to preserve extrema. Legacy databases keep their
-existing unique index authoritative while the replacement index is built, but
-that O(rows) migration must run on deferred startup maintenance and swap indexes
-in one SQLite transaction so constructor latency, concurrent reads, uniqueness,
-and crash recovery remain intact; a database with no unique identity index at
-all migrates synchronously so upserts never run without a conflict target. The
+tier presence. Full readers continue to preserve extrema. Legacy databases keep
+their existing unique index authoritative while deferred startup maintenance
+copies the eight observation columns into the clustered table. `BEGIN IMMEDIATE`
+serializes writers and independent Store migrations before inspection. Copy,
+old-table removal, rename and retention-index creation commit in one WAL
+transaction; cancellation, crash, insufficient disk space or a conflicting
+replacement table leaves the installed table and observations intact. Readers
+retain their old SQLite snapshots during the copy; queued writes resume after
+commit. The migration checks free filesystem space for three times the used
+database pages plus 16 MiB before copying. Layout migration and auto-vacuum
+share the existing 30-minute rewrite context, canceled when the Store closes.
+There is no redundant legacy index rebuild before the copy. A database with
+no unique identity index still deduplicates and establishes identity
+synchronously before ingestion starts.
+
+Each retained read fixes its index choice for its snapshot. A stale legacy
+index error may retry once with a new snapshot only after inspecting the
+clustered primary key. This covers another Store committing the upgrade while
+prepared SQL is cached. Layout participates in the bounded SQL-template key;
+results and tier presence remain uncached. Restart recognizes the primary key
+without recreating the legacy index. If an older executable reopens the
+row-compatible table and recreates its legacy identity index, deferred
+maintenance removes only the known redundant trees after verifying the primary
+key. `clustered_layout_test.go` covers fresh/reopened schemas, exact stored
+nullable bounds and all tiers, metadata, concurrent old snapshots and queued
+writes, cross-Store cached reads, cancellation after swap, crash before commit,
+conflicting replacement tables and primary-table range plans. Historical write
+profiles reconstruct their original rowid tables; `clustered` profiles exercise
+the current production schema. The
deterministic issue-1124 profile must continue to attribute writes by logical
payload, table/index `dbstat` pages, page-cache writes/spills, WAL frames,
checkpoint frames, process write bytes where the host exposes them, and traced
@@ -4093,7 +4124,7 @@ mode in `TestStoreAutoVacuumPersistsAcrossRestart`.
### Metrics-store writes survive the one-time startup write-lock hold
-The deferred identity-index rebuild and the auto-vacuum VACUUM can hold the
+The deferred clustered-table migration and the auto-vacuum VACUUM can hold the
SQLite write lock for minutes on a large legacy database. `BEGIN` is deferred,
so the lock is contended at the first `INSERT` or at `COMMIT`, not at `Begin`;
a retry loop around `Begin` alone skipped rows and committed an empty
diff --git a/docs/release-control/v6/internal/subsystems/registry.json b/docs/release-control/v6/internal/subsystems/registry.json
index 751155a6e8..0f83a1ac30 100644
--- a/docs/release-control/v6/internal/subsystems/registry.json
+++ b/docs/release-control/v6/internal/subsystems/registry.json
@@ -7458,6 +7458,9 @@
"allow_same_subsystem_tests": false,
"test_prefixes": [],
"exact_files": [
+ "pkg/metrics/clustered_layout_test.go",
+ "pkg/metrics/identity_index_test.go",
+ "pkg/metrics/issue1124_write_amplification_test.go",
"pkg/metrics/store_additional_test.go",
"pkg/metrics/store_bench_test.go",
"pkg/metrics/store_docker_observation_contract_test.go",
diff --git a/pkg/metrics/clustered_layout.go b/pkg/metrics/clustered_layout.go
new file mode 100644
index 0000000000..68f55d9112
--- /dev/null
+++ b/pkg/metrics/clustered_layout.go
@@ -0,0 +1,232 @@
+package metrics
+
+import (
+ "context"
+ "database/sql"
+ "fmt"
+ "math"
+ "path/filepath"
+ "strings"
+
+ "github.com/shirou/gopsutil/v4/disk"
+)
+
+// Store samples in the range-read B-tree itself. The old rowid table required
+// a second B-tree lookup for every value after finding its identity index entry.
+// Keep the time-major key shared by ingestion, reconciliation and range reads.
+const clusteredMetricsIndex = "sqlite_autoindex_metrics_1"
+
+const metricsTableColumns = `
+ resource_type TEXT NOT NULL,
+ resource_id TEXT NOT NULL,
+ metric_type TEXT NOT NULL,
+ value REAL NOT NULL,
+ min_value REAL,
+ max_value REAL,
+ timestamp INTEGER NOT NULL,
+ tier TEXT NOT NULL DEFAULT 'raw',
+ PRIMARY KEY(resource_type, resource_id, tier, timestamp, metric_type)
+`
+
+type metricsSchemaReader interface {
+ QueryContext(context.Context, string, ...any) (*sql.Rows, error)
+}
+
+// Inspect the primary key rather than a schema version marker: a restored
+// backup or another Store handle can change the actual physical schema.
+func metricsTableIsClustered(ctx context.Context, db metricsSchemaReader) (bool, error) {
+ rows, err := db.QueryContext(ctx, `SELECT wr FROM pragma_table_list WHERE schema = 'main' AND name = 'metrics'`)
+ if err != nil {
+ return false, err
+ }
+ var withoutRowid bool
+ if rows.Next() {
+ err = rows.Scan(&withoutRowid)
+ } else {
+ err = rows.Err()
+ if err == nil {
+ err = fmt.Errorf("metrics table is missing")
+ }
+ }
+ if closeErr := rows.Close(); err == nil {
+ err = closeErr
+ }
+ if err != nil || !withoutRowid {
+ return false, err
+ }
+ rows, err = db.QueryContext(ctx, `PRAGMA index_info(`+clusteredMetricsIndex+`)`)
+ if err != nil {
+ return false, err
+ }
+ defer rows.Close()
+ var columns []string
+ for rows.Next() {
+ var sequence, cid int
+ var name string
+ if err := rows.Scan(&sequence, &cid, &name); err != nil {
+ return false, err
+ }
+ columns = append(columns, name)
+ }
+ if err := rows.Err(); err != nil {
+ return false, err
+ }
+ return strings.Join(columns, ",") == strings.Join(metricsIdentityColumns, ","), nil
+}
+
+func (s *Store) metricsReadIndex() string {
+ if s.clusteredMetrics.Load() {
+ return clusteredMetricsIndex
+ }
+ return metricsIdentityIndex
+}
+
+// Reserve room for the replacement pages and the transaction's WAL before
+// copying pages. Keep the readable legacy layout if the filesystem cannot
+// safely hold a copy; a later startup can retry without a partial migration.
+func (s *Store) checkClusteredMigrationSpace(ctx context.Context, conn *sql.Conn) error {
+ var pages, freePages, pageSize int64
+ for _, item := range []struct {
+ query string
+ dest *int64
+ }{{"PRAGMA page_count", &pages}, {"PRAGMA freelist_count", &freePages}, {"PRAGMA page_size", &pageSize}} {
+ if err := conn.QueryRowContext(ctx, item.query).Scan(item.dest); err != nil {
+ return err
+ }
+ }
+ usedPages := pages - freePages
+ const margin = int64(16 << 20)
+ if usedPages < 0 || pageSize <= 0 || usedPages > (math.MaxInt64-margin)/3/pageSize {
+ return fmt.Errorf("invalid metrics page accounting")
+ }
+ required := uint64(usedPages*pageSize*3 + margin)
+ usage, err := disk.UsageWithContext(ctx, filepath.Dir(s.config.DBPath))
+ if err != nil {
+ return fmt.Errorf("inspect metrics migration free space: %w", err)
+ }
+ if usage.Free < required {
+ return fmt.Errorf("metrics clustered migration needs %d free bytes, available %d", required, usage.Free)
+ }
+ return nil
+}
+
+// Copy and swap in one WAL transaction. Existing readers keep their old
+// snapshot throughout the copy; queued writes resume against the new table
+// after commit. An interruption rolls back the table, indexes and every row.
+// BEGIN IMMEDIATE also serializes independent Store handles before inspection.
+func (s *Store) migrateClusteredMetrics(ctx context.Context) error {
+ if s.clusteredMetrics.Load() {
+ return s.cleanupClusteredMetricsIndexes(ctx)
+ }
+ conn, err := s.db.DB.Conn(ctx)
+ if err != nil {
+ return err
+ }
+ defer conn.Close()
+ if _, err := conn.ExecContext(ctx, `BEGIN IMMEDIATE`); err != nil {
+ return fmt.Errorf("begin clustered metrics migration: %w", err)
+ }
+ defer func() { _, _ = conn.ExecContext(context.Background(), `ROLLBACK`) }()
+ clustered, err := metricsTableIsClustered(ctx, conn)
+ if err != nil {
+ return err
+ }
+ if !clustered {
+ if err := s.checkClusteredMigrationSpace(ctx, conn); err != nil {
+ return err
+ }
+ if _, err := conn.ExecContext(ctx, `CREATE TABLE metrics_clustered_upgrade (`+metricsTableColumns+`) WITHOUT ROWID`); err != nil {
+ return fmt.Errorf("create clustered metrics replacement: %w", err)
+ }
+ if s.clusteredMigrationHook != nil {
+ s.clusteredMigrationHook("before-copy")
+ }
+ if _, err := conn.ExecContext(ctx, `
+ INSERT INTO metrics_clustered_upgrade
+ (resource_type, resource_id, metric_type, value, min_value, max_value, timestamp, tier)
+ SELECT resource_type, resource_id, metric_type, value, min_value, max_value, timestamp, tier
+ FROM metrics ORDER BY resource_type, resource_id, tier, timestamp, metric_type
+ `); err != nil {
+ return fmt.Errorf("copy clustered metrics: %w", err)
+ }
+ if s.clusteredMigrationHook != nil {
+ s.clusteredMigrationHook("after-copy")
+ }
+ if _, err := conn.ExecContext(ctx, `
+ DROP TABLE metrics;
+ ALTER TABLE metrics_clustered_upgrade RENAME TO metrics;
+ CREATE INDEX idx_metrics_tier_time ON metrics(tier, timestamp);
+ `); err != nil {
+ return fmt.Errorf("swap clustered metrics: %w", err)
+ }
+ }
+ if s.clusteredMigrationHook != nil {
+ s.clusteredMigrationHook("before-commit")
+ }
+ if _, err := conn.ExecContext(ctx, `COMMIT`); err != nil {
+ return fmt.Errorf("commit clustered metrics migration: %w", err)
+ }
+ s.clusteredMetrics.Store(true)
+ return nil
+}
+
+// A previous executable can reopen this row-compatible table and recreate its
+// old read index. Remove only those known redundant trees on a later startup.
+// The ordinary clustered restart is a read-only probe, without a writer lock.
+func (s *Store) cleanupClusteredMetricsIndexes(ctx context.Context) error {
+ rows, err := s.db.DB.QueryContext(ctx, `SELECT name FROM pragma_index_list('metrics')
+ WHERE name IN ('idx_metrics_query_all', 'idx_metrics_lookup', 'idx_metrics_unique')`)
+ if err != nil {
+ return err
+ }
+ var indexes []string
+ for rows.Next() {
+ var name string
+ if err := rows.Scan(&name); err != nil {
+ _ = rows.Close()
+ return err
+ }
+ indexes = append(indexes, name)
+ }
+ err = rows.Err()
+ _ = rows.Close()
+ if err != nil || len(indexes) == 0 {
+ return err
+ }
+ conn, err := s.db.DB.Conn(ctx)
+ if err != nil {
+ return err
+ }
+ defer conn.Close()
+ if _, err := conn.ExecContext(ctx, `BEGIN IMMEDIATE`); err != nil {
+ return err
+ }
+ defer func() { _, _ = conn.ExecContext(context.Background(), `ROLLBACK`) }()
+ clustered, err := metricsTableIsClustered(ctx, conn)
+ if err != nil || !clustered {
+ return fmt.Errorf("cannot prune identity indexes without the clustered primary key: %v", err)
+ }
+ for _, index := range indexes {
+ // Names come from the fixed whitelist above, never user input.
+ if _, err := conn.ExecContext(ctx, `DROP INDEX IF EXISTS `+index); err != nil {
+ return err
+ }
+ }
+ _, err = conn.ExecContext(ctx, `COMMIT`)
+ return err
+}
+
+// A different Store can commit the migration between preparing and executing
+// a legacy read. Retry that missing-index error once, after verifying the new
+// primary key, with a new snapshot and the same canonical read semantics.
+func (s *Store) refreshClusteredMetrics(err error) bool {
+ if !strings.Contains(err.Error(), "no such index: "+metricsIdentityIndex) {
+ return false
+ }
+ clustered, inspectErr := metricsTableIsClustered(context.Background(), s.db.DB)
+ if inspectErr != nil || !clustered {
+ return false
+ }
+ s.clusteredMetrics.Store(true)
+ return true
+}
diff --git a/pkg/metrics/clustered_layout_test.go b/pkg/metrics/clustered_layout_test.go
new file mode 100644
index 0000000000..a1a6c1c14e
--- /dev/null
+++ b/pkg/metrics/clustered_layout_test.go
@@ -0,0 +1,405 @@
+package metrics
+
+import (
+ "context"
+ "database/sql"
+ "fmt"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "reflect"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ pdb "github.com/rcourtman/pulse-go-rewrite/pkg/db"
+)
+
+// Pause deferred maintenance, not the constructor, so tests can exercise the
+// installed legacy schema and the cutover without racing a background upgrade.
+func clusteredLegacyStore(t *testing.T) (*Store, func()) {
+ t.Helper()
+ suppressTestLogs(t)
+ dbPath := filepath.Join(t.TempDir(), "metrics.db")
+ issue1124CreateLegacyStore(t, dbPath, 100, false)
+ entered, release := make(chan struct{}), make(chan struct{})
+ previous := startupMaintenanceHook
+ startupMaintenanceHook = func() { close(entered); <-release }
+ cfg := DefaultConfig(filepath.Dir(dbPath))
+ cfg.DBPath = dbPath
+ cfg.FlushInterval, cfg.RollupInterval = time.Hour, time.Hour
+ cfg.RetentionRaw = 10 * 365 * 24 * time.Hour
+ store, err := NewStore(cfg)
+ startupMaintenanceHook = previous
+ var once sync.Once
+ unblock := func() { once.Do(func() { close(release) }) }
+ if err != nil {
+ unblock()
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { unblock(); _ = store.Close() })
+ select {
+ case <-entered:
+ case <-time.After(5 * time.Second):
+ t.Fatal("deferred maintenance did not start")
+ }
+ return store, unblock
+}
+
+func assertClusteredStore(t *testing.T, store *Store) {
+ t.Helper()
+ clustered, err := metricsTableIsClustered(context.Background(), store.db.DB)
+ if err != nil || !clustered {
+ t.Fatalf("clustered schema=%t, error=%v", clustered, err)
+ }
+ issue1124AssertConsolidatedIndexes(t, store)
+ var integrity string
+ if err := store.db.QueryRow(`PRAGMA integrity_check`).Scan(&integrity); err != nil || integrity != "ok" {
+ t.Fatalf("integrity=%q, error=%v", integrity, err)
+ }
+}
+
+func TestClusteredMetricsFreshAndRestart(t *testing.T) {
+ cfg := DefaultConfig(t.TempDir())
+ for i := 0; i < 2; i++ {
+ store, err := NewStore(cfg)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := store.WaitForMaintenance(30 * time.Second); err != nil {
+ t.Fatal(err)
+ }
+ assertClusteredStore(t, store)
+ if err := store.Close(); err != nil {
+ t.Fatal(err)
+ }
+ }
+}
+
+func TestClusteredMetricsPreservesObservationsAndMetadata(t *testing.T) {
+ store, _ := clusteredLegacyStore(t)
+ base := time.Unix(1_700_000_000, 0)
+ for i, tier := range []Tier{TierRaw, TierMinute, TierHourly, TierDaily} {
+ for j, resource := range []string{"vm", "dockercontainer"} {
+ for k, bounds := range [][2]any{{nil, nil}, {0.0, nil}, {nil, 0.0}, {-4.0, 12.0}} {
+ _, err := store.db.Exec(`INSERT INTO metrics
+ (resource_type, resource_id, metric_type, value, min_value, max_value, timestamp, tier)
+ VALUES (?, ?, 'cpu', ?, ?, ?, ?, ?)`, resource, fmt.Sprintf("case-%d", j),
+ float64(k+1), bounds[0], bounds[1], base.Add(time.Duration(i*86400+k*60)*time.Second).Unix(), tier)
+ if err != nil {
+ t.Fatal(err)
+ }
+ }
+ }
+ }
+ if _, err := store.db.Exec(`INSERT INTO metrics_meta VALUES ('rollup_sentinel', '1700000000')`); err != nil {
+ t.Fatal(err)
+ }
+ // The exact stored nullable bounds, observation names, tiers and values
+ // must survive, independently of any reader's normalization or fallback.
+ read := func() []string {
+ rows, err := store.db.Query(`SELECT resource_type, resource_id, metric_type,
+ quote(value), quote(min_value), quote(max_value), timestamp, tier
+ FROM metrics ORDER BY resource_type, resource_id, tier, timestamp, metric_type`)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer rows.Close()
+ var result []string
+ for rows.Next() {
+ var resource, id, metric, value, low, high, tier string
+ var ts int64
+ if err := rows.Scan(&resource, &id, &metric, &value, &low, &high, &ts, &tier); err != nil {
+ t.Fatal(err)
+ }
+ result = append(result, fmt.Sprint(resource, id, metric, value, low, high, ts, tier))
+ }
+ if err := rows.Err(); err != nil {
+ t.Fatal(err)
+ }
+ return result
+ }
+ before := read()
+ if err := store.migrateClusteredMetrics(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if after := read(); !reflect.DeepEqual(before, after) {
+ t.Fatalf("stored observations changed across clustered migration")
+ }
+ var sentinel string
+ if err := store.db.QueryRow(`SELECT value FROM metrics_meta WHERE key='rollup_sentinel'`).Scan(&sentinel); err != nil || sentinel != "1700000000" {
+ t.Fatalf("rollup metadata=%q, error=%v", sentinel, err)
+ }
+ assertClusteredStore(t, store)
+}
+
+func TestClusteredMetricsReadersAndQueuedWritesSurviveCutover(t *testing.T) {
+ store, _ := clusteredLegacyStore(t)
+ entered, release := make(chan struct{}), make(chan struct{})
+ var once sync.Once
+ unblock := func() { once.Do(func() { close(release) }) }
+ defer unblock()
+ store.clusteredMigrationHook = func(stage string) {
+ if stage == "after-copy" {
+ close(entered)
+ <-release
+ }
+ }
+ // An already open SQLite snapshot must retain its original schema while
+ // the other connection swaps and drops the physical table.
+ tx, err := store.db.BeginTx(context.Background(), &sql.TxOptions{ReadOnly: true})
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer tx.Rollback()
+ var count int
+ if err := tx.QueryRow(`SELECT COUNT(*) FROM metrics INDEXED BY idx_metrics_query_all`).Scan(&count); err != nil {
+ t.Fatal(err)
+ }
+ migrated := make(chan error, 1)
+ go func() { migrated <- store.migrateClusteredMetrics(context.Background()) }()
+ select {
+ case <-entered:
+ case <-time.After(5 * time.Second):
+ t.Fatal("migration did not copy")
+ }
+ points, err := store.Query("vm", "legacy-0001", "cpu", time.Unix(1_699_999_999, 0), time.Unix(1_700_001_000, 0), 0)
+ if err != nil || len(points) != 1 {
+ t.Fatalf("reader during copy points=%d, error=%v", len(points), err)
+ }
+ written := make(chan struct{})
+ go func() {
+ store.WriteBatchSync([]WriteMetric{{ResourceType: "vm", ResourceID: "queued", MetricType: "cpu", Value: 42, Timestamp: time.Unix(1_700_020_000, 0), Tier: TierRaw}})
+ close(written)
+ }()
+ unblock()
+ if err := <-migrated; err != nil {
+ t.Fatal(err)
+ }
+ if err := tx.QueryRow(`SELECT COUNT(*) FROM metrics INDEXED BY idx_metrics_query_all`).Scan(&count); err != nil || count != 100 {
+ t.Fatalf("old read snapshot count=%d, error=%v", count, err)
+ }
+ _ = tx.Rollback()
+ select {
+ case <-written:
+ case <-time.After(35 * time.Second):
+ t.Fatal("queued write did not resume")
+ }
+ var value float64
+ if err := store.db.QueryRow(`SELECT value FROM metrics WHERE resource_id='queued'`).Scan(&value); err != nil || value != 42 {
+ t.Fatalf("queued write=%v, error=%v", value, err)
+ }
+ assertClusteredStore(t, store)
+}
+
+func TestClusteredMetricsCancellationRollsBackSwap(t *testing.T) {
+ store, _ := clusteredLegacyStore(t)
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+ store.clusteredMigrationHook = func(stage string) {
+ if stage == "before-commit" {
+ cancel()
+ }
+ }
+ if err := store.migrateClusteredMetrics(ctx); err == nil {
+ t.Fatal("canceled migration succeeded")
+ }
+ clustered, err := metricsTableIsClustered(context.Background(), store.db.DB)
+ if err != nil || clustered {
+ t.Fatalf("canceled swap changed layout=%t, error=%v", clustered, err)
+ }
+ var count int
+ if err := store.db.QueryRow(`SELECT COUNT(*) FROM metrics`).Scan(&count); err != nil || count != 100 {
+ t.Fatalf("rollback rows=%d, error=%v", count, err)
+ }
+ if err := store.db.QueryRow(`SELECT COUNT(*) FROM sqlite_schema WHERE name='metrics_clustered_upgrade'`).Scan(&count); err != nil || count != 0 {
+ t.Fatalf("rollback replacement tables=%d, error=%v", count, err)
+ }
+ store.clusteredMigrationHook = nil
+ if err := store.migrateClusteredMetrics(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ assertClusteredStore(t, store)
+}
+
+func TestClusteredMetricsPreservesConflictingReplacementTable(t *testing.T) {
+ store, _ := clusteredLegacyStore(t)
+ if _, err := store.db.Exec(`CREATE TABLE metrics_clustered_upgrade(value TEXT);
+ INSERT INTO metrics_clustered_upgrade VALUES ('preserve')`); err != nil {
+ t.Fatal(err)
+ }
+ if err := store.migrateClusteredMetrics(context.Background()); err == nil {
+ t.Fatal("conflicting table was overwritten")
+ }
+ var value string
+ if err := store.db.QueryRow(`SELECT value FROM metrics_clustered_upgrade`).Scan(&value); err != nil || value != "preserve" {
+ t.Fatalf("conflicting table value=%q, error=%v", value, err)
+ }
+ var count int
+ if err := store.db.QueryRow(`SELECT COUNT(*) FROM metrics`).Scan(&count); err != nil || count != 100 {
+ t.Fatalf("original rows=%d, error=%v", count, err)
+ }
+}
+
+func TestClusteredMetricsCrashRollsBackSwap(t *testing.T) {
+ if dbPath := os.Getenv("PULSE_CLUSTERED_CRASH_DB"); dbPath != "" {
+ metricsClusteredMigrationHook = func(stage string) {
+ if stage == "before-commit" {
+ os.Exit(88)
+ }
+ }
+ cfg := DefaultConfig(filepath.Dir(dbPath))
+ cfg.DBPath = dbPath
+ cfg.RetentionRaw = 10 * 365 * 24 * time.Hour
+ store, err := NewStore(cfg)
+ if err != nil {
+ os.Exit(90)
+ }
+ _ = store.WaitForMaintenance(30 * time.Second)
+ os.Exit(89)
+ }
+ suppressTestLogs(t)
+ dbPath := filepath.Join(t.TempDir(), "metrics.db")
+ issue1124CreateLegacyStore(t, dbPath, 10_000, false)
+ cmd := exec.Command(os.Args[0], "-test.run=^TestClusteredMetricsCrashRollsBackSwap$")
+ cmd.Env = append(os.Environ(), "PULSE_CLUSTERED_CRASH_DB="+dbPath)
+ err := cmd.Run()
+ if exit, ok := err.(*exec.ExitError); !ok || exit.ExitCode() != 88 {
+ t.Fatalf("crash child=%v, want exit 88", err)
+ }
+ cfg := DefaultConfig(filepath.Dir(dbPath))
+ cfg.DBPath = dbPath
+ cfg.RetentionRaw = 10 * 365 * 24 * time.Hour
+ store, err := NewStore(cfg)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer store.Close()
+ if err := store.WaitForMaintenance(30 * time.Second); err != nil {
+ t.Fatal(err)
+ }
+ var count int
+ if err := store.db.QueryRow(`SELECT COUNT(*) FROM metrics`).Scan(&count); err != nil || count != 10_000 {
+ t.Fatalf("recovered rows=%d, error=%v", count, err)
+ }
+ assertClusteredStore(t, store)
+}
+
+func TestClusteredMetricsRefreshesAnotherStoreAndCachedShapes(t *testing.T) {
+ store, _ := clusteredLegacyStore(t)
+ // A second handle has its own physical-schema bit and bounded SQL caches.
+ release := make(chan struct{})
+ previous := startupMaintenanceHook
+ startupMaintenanceHook = func() { <-release }
+ other, err := NewStore(store.config)
+ startupMaintenanceHook = previous
+ if err != nil {
+ close(release)
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { close(release); _ = other.Close() })
+ start, end := time.Unix(1_699_999_999, 0), time.Unix(1_700_001_000, 0)
+ for _, step := range []int64{0, 60} {
+ if _, err := other.QueryAllBatch("vm", []string{"legacy-0001", "legacy-0002"}, start, end, step); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if err := store.migrateClusteredMetrics(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ for _, step := range []int64{0, 60} {
+ // Force the other handle's stale layout, including a cached presence
+ // statement, so both snapshot paths exercise the real missing-index error.
+ other.clusteredMetrics.Store(false)
+ points, err := other.QueryAllBatch("vm", []string{"legacy-0001", "legacy-0002"}, start, end, step)
+ if err != nil || len(points) != 2 || !other.clusteredMetrics.Load() {
+ t.Fatalf("stale full reader step=%d: resources=%d, refreshed=%t, error=%v", step, len(points), other.clusteredMetrics.Load(), err)
+ }
+ other.clusteredMetrics.Store(false)
+ values, err := other.QueryMetricValuesBatch("vm", []string{"legacy-0001", "legacy-0002"}, nil, start, end, step)
+ if err != nil || len(values) != 2 || !other.clusteredMetrics.Load() {
+ t.Fatalf("stale value reader step=%d: resources=%d, refreshed=%t, error=%v", step, len(values), other.clusteredMetrics.Load(), err)
+ }
+ }
+}
+
+func TestClusteredMetricsReopeningLegacyExecutableIndexes(t *testing.T) {
+ cfg := DefaultConfig(t.TempDir())
+ store, err := NewStore(cfg)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := store.WaitForMaintenance(30 * time.Second); err != nil {
+ t.Fatal(err)
+ }
+ // Reproduce the previous runtime's initSchema/unique-index behavior on
+ // the new table. Its unchanged upsert conflict target remains functional.
+ if _, err := store.db.Exec(`CREATE UNIQUE INDEX idx_metrics_query_all
+ ON metrics(resource_type, resource_id, tier, timestamp, metric_type);
+ INSERT INTO metrics(resource_type, resource_id, metric_type, value, timestamp, tier)
+ VALUES ('vm', 'downgrade', 'cpu', 1, 2000000000, 'raw')
+ ON CONFLICT(resource_type, resource_id, metric_type, tier, timestamp) DO UPDATE SET value=excluded.value;
+ INSERT INTO metrics(resource_type, resource_id, metric_type, value, timestamp, tier)
+ VALUES ('vm', 'downgrade', 'cpu', 2, 2000000000, 'raw')
+ ON CONFLICT(resource_type, resource_id, metric_type, tier, timestamp) DO UPDATE SET value=excluded.value;`); err != nil {
+ t.Fatal(err)
+ }
+ if err := store.Close(); err != nil {
+ t.Fatal(err)
+ }
+ reopened, err := NewStore(cfg)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer reopened.Close()
+ if err := reopened.WaitForMaintenance(30 * time.Second); err != nil {
+ t.Fatal(err)
+ }
+ assertClusteredStore(t, reopened)
+ var value float64
+ if err := reopened.db.QueryRow(`SELECT value FROM metrics WHERE resource_id='downgrade'`).Scan(&value); err != nil || value != 2 {
+ t.Fatalf("legacy-compatible upsert value=%v, error=%v", value, err)
+ }
+}
+
+func TestClusteredRetainedPlansUsePrimaryTable(t *testing.T) {
+ db := newPlanTestDB(t)
+ store := &Store{db: pdb.Wrap(db, "metrics"), config: DefaultConfig(t.TempDir())}
+ if err := store.migrateClusteredMetrics(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ end := time.Unix(2_000_000_040, 0)
+ for _, valuesOnly := range []bool{false, true} {
+ for _, window := range []time.Duration{time.Hour, 24 * time.Hour, 7 * 24 * time.Hour, 30 * 24 * time.Hour} {
+ for _, ids := range [][]string{{"vm-1"}, {"vm-1", "vm-2"}} {
+ for _, metrics := range [][]string{nil, {"cpu"}, {"cpu", "memory"}} {
+ for _, step := range []int64{0, 60} {
+ t.Run(fmt.Sprintf("values_%t/%s/resources_%d/metrics_%d/step_%d", valuesOnly, window, len(ids), len(metrics), step), func(t *testing.T) {
+ query, args := store.retainedQuerySQL("vm", ids, metrics, end.Add(-window), end, step, store.tierFallbacks(window), false, valuesOnly)
+ plan := explainQueryPlan(t, db, query, args)
+ for _, line := range strings.Split(plan, "\n") {
+ if strings.Contains(line, "SCAN m ") || strings.Contains(line, "SCAN h ") {
+ t.Fatalf("unbounded clustered read: %s", plan)
+ }
+ if strings.Contains(line, "SEARCH m ") || strings.Contains(line, "SEARCH h ") {
+ for _, constraint := range []string{"USING PRIMARY KEY", "resource_type=?", "resource_id=?", "tier=?", "timestamp>?", "timestamp<?"} {
+ if !strings.Contains(line, constraint) {
+ t.Fatalf("clustered lookup missing %s: %s", constraint, plan)
+ }
+ }
+ }
+ if step == 0 && strings.Contains(line, "TEMP B-TREE") {
+ t.Fatalf("clustered unaggregated read sorts: %s", plan)
+ }
+ }
+ if !strings.Contains(plan, "SEARCH m USING PRIMARY KEY") {
+ t.Fatalf("missing primary table lookup: %s", plan)
+ }
+ })
+ }
+ }
+ }
+ }
+ }
+}
diff --git a/pkg/metrics/identity_index_test.go b/pkg/metrics/identity_index_test.go
index d02d3d623d..334b7aca1f 100644
--- a/pkg/metrics/identity_index_test.go
+++ b/pkg/metrics/identity_index_test.go
@@ -48,6 +48,7 @@ func newSteadyStateStore(tb testing.TB, layout string) (*Store, string) {
tb.Fatalf("startup maintenance: %v", err)
}
if layout == "metric-major" {
+ issue1124InstallRowidSchema(tb, store)
if _, err := store.db.Exec(`
DROP INDEX idx_metrics_query_all;
CREATE INDEX idx_metrics_query_all
diff --git a/pkg/metrics/issue1124_write_amplification_test.go b/pkg/metrics/issue1124_write_amplification_test.go
index 6959f93dbc..1e0aaa255c 100644
--- a/pkg/metrics/issue1124_write_amplification_test.go
+++ b/pkg/metrics/issue1124_write_amplification_test.go
@@ -188,8 +188,10 @@ func TestStoreIdentityMigrationDeduplicatesLegacyRows(t *testing.T) {
func TestStoreIdentityMigrationRecoversAfterCrash(t *testing.T) {
if os.Getenv("PULSE_METRICS_IDENTITY_CRASH_CHILD") == "1" {
- metricsIdentityMigrationHook = func() {
- os.Exit(88)
+ metricsClusteredMigrationHook = func(stage string) {
+ if stage == "before-commit" {
+ os.Exit(88)
+ }
}
dbPath := os.Getenv("PULSE_METRICS_IDENTITY_CRASH_DB")
cfg := DefaultConfig(filepath.Dir(dbPath))
@@ -315,12 +317,14 @@ func TestStoreIdentityMigrationKeepsReadersAvailable(t *testing.T) {
migrationEntered := make(chan struct{})
releaseMigration := make(chan struct{})
var hookOnce sync.Once
- metricsIdentityMigrationHook = func() {
- hookOnce.Do(func() { close(migrationEntered) })
- <-releaseMigration
+ metricsClusteredMigrationHook = func(stage string) {
+ if stage == "after-copy" {
+ hookOnce.Do(func() { close(migrationEntered) })
+ <-releaseMigration
+ }
}
t.Cleanup(func() {
- metricsIdentityMigrationHook = nil
+ metricsClusteredMigrationHook = nil
})
cfg := DefaultConfig(filepath.Dir(dbPath))
@@ -497,8 +501,9 @@ func TestMetricsWriteAmplificationInvariant(t *testing.T) {
//
// Run with PULSE_METRICS_WRITE_PROFILE=full (v6.1.1 layout), lookup (the
// v6.1.2 to v6.4 metric-major unique index) or consolidated (the time-major
-// identity index), each optionally -batched, and optionally
-// PULSE_METRICS_WRITE_PROFILE_TICKS=N. The default 30 ticks
+// identity index), each optionally -batched, or clustered (the production
+// time-major primary table). PULSE_METRICS_WRITE_PROFILE_TICKS=N
+// overrides the default 30 ticks, which
// persist 393,630 samples across 2,197 resources. Set
// PULSE_METRICS_WRITE_PROFILE_AUTOCHECKPOINT=1 for production checkpoint/fsync
// tracing, PULSE_METRICS_WRITE_PROFILE_CHECKPOINT_PAGES=N for a bounded
@@ -511,8 +516,8 @@ func TestIssue1124WriteAmplificationProfile(t *testing.T) {
t.Skip("set " + issue1124ProfileEnv + " to run the write-amplification profile")
}
if mode != "full" && mode != "lookup" && mode != "consolidated" && mode != "full-batched" &&
- mode != "lookup-batched" && mode != "consolidated-batched" && mode != "without-rowid" {
- t.Fatalf("%s must be full, lookup, consolidated, full-batched, lookup-batched, consolidated-batched, or without-rowid, got %q", issue1124ProfileEnv, mode)
+ mode != "lookup-batched" && mode != "consolidated-batched" && mode != "without-rowid" && mode != "clustered" {
+ t.Fatalf("%s must be full, lookup, consolidated, full-batched, lookup-batched, consolidated-batched, without-rowid, or clustered, got %q", issue1124ProfileEnv, mode)
}
ticks := 30
@@ -562,6 +567,12 @@ func TestIssue1124WriteAmplificationProfile(t *testing.T) {
issue1124ConfigureConnections(t, store, connectionCount, checkpointPages)
autoCheckpoint := checkpointPages > 0
+ if mode != "clustered" {
+ // Keep historical profiles as actual rowid schemas after production
+ // moves to clustered storage. No rows have been written yet.
+ issue1124InstallRowidSchema(t, store)
+ }
+
if mode == "full" || mode == "full-batched" {
// Reconstruct the v6.1.1 schema so released and consolidated layouts
// remain comparable after the production migration lands.
@@ -1211,12 +1222,13 @@ func issue1124CreateLegacyStore(t *testing.T, dbPath string, rows int, duplicate
func issue1124AssertConsolidatedIndexes(t *testing.T, store *Store) {
t.Helper()
- matches, err := store.metricsIndexMatches(metricsIdentityIndex, true, metricsIdentityColumns)
+ index := store.metricsReadIndex()
+ matches, err := store.metricsIndexMatches(index, true, metricsIdentityColumns)
if err != nil {
t.Fatalf("inspect consolidated identity index: %v", err)
}
if !matches {
- t.Fatalf("%s is not the expected unique time-major identity index", metricsIdentityIndex)
+ t.Fatalf("%s is not the expected unique time-major identity index", index)
}
rows, err := store.db.Query(`PRAGMA index_list(metrics)`)
@@ -1244,7 +1256,7 @@ func issue1124AssertConsolidatedIndexes(t *testing.T, store *Store) {
// Retired metric-major trees (idx_metrics_lookup, idx_metrics_unique)
// must be gone: each would be dirtied by every insert.
want := map[string]bool{
- "idx_metrics_query_all": true,
+ index: true,
"idx_metrics_tier_time": false,
}
if len(indexes) != len(want) {
@@ -1256,3 +1268,25 @@ func issue1124AssertConsolidatedIndexes(t *testing.T, store *Store) {
}
}
}
+
+// issue1124InstallRowidSchema reconstructs the released time-major rowid table
+// in an empty measurement fixture, preserving comparisons across schema changes.
+func issue1124InstallRowidSchema(tb testing.TB, store *Store) {
+ tb.Helper()
+ if _, err := store.db.Exec(`
+ DROP TABLE metrics;
+ CREATE TABLE metrics (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ resource_type TEXT NOT NULL, resource_id TEXT NOT NULL,
+ metric_type TEXT NOT NULL, value REAL NOT NULL,
+ min_value REAL, max_value REAL,
+ timestamp INTEGER NOT NULL, tier TEXT NOT NULL DEFAULT 'raw'
+ );
+ CREATE INDEX idx_metrics_tier_time ON metrics(tier, timestamp);
+ CREATE UNIQUE INDEX idx_metrics_query_all
+ ON metrics(resource_type, resource_id, tier, timestamp, metric_type);
+ `); err != nil {
+ tb.Fatalf("install historical rowid table: %v", err)
+ }
+ store.clusteredMetrics.Store(false)
+}
diff --git a/pkg/metrics/store.go b/pkg/metrics/store.go
index 88328223b2..87c7fb0d78 100644
--- a/pkg/metrics/store.go
+++ b/pkg/metrics/store.go
@@ -258,8 +258,8 @@ type maintenanceRequest struct {
}
var (
- startupMaintenanceHook func()
- metricsIdentityMigrationHook func()
+ startupMaintenanceHook func()
+ metricsClusteredMigrationHook func(string)
)
// Store provides persistent metrics storage
@@ -295,14 +295,15 @@ type Store struct {
stopOnce sync.Once
stopping atomic.Bool
syncWaitWarnNano atomic.Int64
- identityMigrationPending atomic.Bool
+ clusteredMetrics atomic.Bool
startupMaintenanceActive atomic.Bool
commercialRetentionSeconds atomic.Int64
commercialPurgeEligibleAt atomic.Int64
// startupHook is captured once at construction so a store that outlives
// the test that built it never invokes a later test's hook.
- startupHook func()
+ startupHook func()
+ clusteredMigrationHook func(string)
}
// SetCommercialHistoryRetention applies a delayed commercial ceiling to the
@@ -387,15 +388,16 @@ func NewStore(config StoreConfig) (*Store, error) {
db := pdb.Wrap(rawDB, "metrics")
store := &Store{
- db: db,
- config: config,
- buffer: make([]bufferedMetric, 0, config.WriteBufferSize),
- writeCh: make(chan writeRequest, 100), // Buffer for write batches
- maintenanceCh: make(chan maintenanceRequest, 1),
- stopCh: make(chan struct{}),
- doneCh: make(chan struct{}),
- maintenanceDoneCh: make(chan struct{}),
- startupHook: startupMaintenanceHook,
+ db: db,
+ config: config,
+ buffer: make([]bufferedMetric, 0, config.WriteBufferSize),
+ writeCh: make(chan writeRequest, 100), // Buffer for write batches
+ maintenanceCh: make(chan maintenanceRequest, 1),
+ stopCh: make(chan struct{}),
+ doneCh: make(chan struct{}),
+ maintenanceDoneCh: make(chan struct{}),
+ startupHook: startupMaintenanceHook,
+ clusteredMigrationHook: metricsClusteredMigrationHook,
}
// Initialize schema
@@ -453,28 +455,12 @@ func NewStore(config StoreConfig) (*Store, error) {
func (s *Store) initSchema() error {
schema := `
-- Main metrics table
- CREATE TABLE IF NOT EXISTS metrics (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- resource_type TEXT NOT NULL,
- resource_id TEXT NOT NULL,
- metric_type TEXT NOT NULL,
- value REAL NOT NULL,
- min_value REAL,
- max_value REAL,
- timestamp INTEGER NOT NULL,
- tier TEXT NOT NULL DEFAULT 'raw'
- );
+ CREATE TABLE IF NOT EXISTS metrics (` + metricsTableColumns + `) WITHOUT ROWID;
-- Index for retention pruning
CREATE INDEX IF NOT EXISTS idx_metrics_tier_time
ON metrics(tier, timestamp);
- -- Identity and range-read index. Created non-unique here so reads
- -- can name it on every schema generation; ensureMetricsIdentityIndex
- -- makes it the unique identity index.
- CREATE INDEX IF NOT EXISTS idx_metrics_query_all
- ON metrics(resource_type, resource_id, tier, timestamp, metric_type);
-
-- Metadata table for tracking rollup state
CREATE TABLE IF NOT EXISTS metrics_meta (
key TEXT PRIMARY KEY,
@@ -487,8 +473,21 @@ func (s *Store) initSchema() error {
return fmt.Errorf("failed to create schema: %w", err)
}
- if err := s.ensureMetricsIdentityIndex(); err != nil {
- return err
+ clustered, err := metricsTableIsClustered(context.Background(), s.db.DB)
+ if err != nil {
+ return fmt.Errorf("inspect metrics physical layout: %w", err)
+ }
+ s.clusteredMetrics.Store(clustered)
+ if !clustered {
+ // Existing rowid stores continue serving on their authoritative index
+ // until deferred maintenance can atomically replace the table.
+ if _, err := s.db.Exec(`CREATE INDEX IF NOT EXISTS idx_metrics_query_all
+ ON metrics(resource_type, resource_id, tier, timestamp, metric_type)`); err != nil {
+ return err
+ }
+ if err := s.ensureMetricsIdentityIndex(); err != nil {
+ return err
+ }
}
if err := s.initAvailabilityHistorySchema(); err != nil {
return err
@@ -498,7 +497,7 @@ func (s *Store) initSchema() error {
return nil
}
-// metricsIdentityIndex is the one B-tree that enforces sample identity and
+// metricsIdentityIndex is the legacy B-tree that enforces sample identity and
// serves every range read. Its columns are time-major within a resource, so
// the samples one poll writes for a resource share leaf pages. The retired
// metric-major order gave every series its own insertion point, and each
@@ -531,10 +530,10 @@ func (s *Store) ensureMetricsIdentityIndex() error {
}
if current || retiredUnique {
// Identity is already enforced by a unique index. Keep serving with
- // that crash-safe schema and defer the O(rows) rebuild to the
+ // that crash-safe schema and defer the O(rows) table replacement to the
// startup-maintenance worker so NewStore latency stays bounded.
- s.identityMigrationPending.Store(true)
- log.Info().Strs("retired_indexes", retired).Msg("Scheduled metrics identity index consolidation")
+ // Rebuilding its index first would immediately discard that work.
+ log.Info().Strs("retired_indexes", retired).Msg("Scheduled clustered metrics table migration")
return nil
}
@@ -573,6 +572,16 @@ func (s *Store) existingRetiredMetricsIdentityIndexes() (existing []string, anyU
}
func (s *Store) migrateMetricsIdentityIndex() error {
+ // Another handle may have completed the physical upgrade while this
+ // store's deferred maintenance was queued. Do not rebuild a redundant tree.
+ clustered, err := metricsTableIsClustered(context.Background(), s.db.DB)
+ if err != nil {
+ return fmt.Errorf("inspect metrics layout before identity migration: %w", err)
+ }
+ if clustered {
+ s.clusteredMetrics.Store(true)
+ return nil
+ }
if err := s.replaceMetricsIdentityIndex(); err == nil {
log.Info().Msg("Consolidated metrics identity index")
return nil
@@ -611,17 +620,12 @@ func (s *Store) replaceMetricsIdentityIndex() error {
if _, err := tx.Exec(`DROP INDEX IF EXISTS ` + metricsIdentityIndex); err != nil {
return fmt.Errorf("drop non-unique identity index: %w", err)
}
- if metricsIdentityMigrationHook != nil {
- metricsIdentityMigrationHook()
- }
if _, err := tx.Exec(`
CREATE UNIQUE INDEX ` + metricsIdentityIndex + `
ON metrics(` + strings.Join(metricsIdentityColumns, ", ") + `)
`); err != nil {
return fmt.Errorf("create unique identity index: %w", err)
}
- } else if metricsIdentityMigrationHook != nil {
- metricsIdentityMigrationHook()
}
for _, name := range retiredMetricsIdentityIndexes {
if _, err := tx.Exec(`DROP INDEX IF EXISTS ` + name); err != nil {
@@ -762,6 +766,10 @@ func (s *Store) migrateAutoVacuum() {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
defer cancel()
+ s.migrateAutoVacuumContext(ctx)
+}
+
+func (s *Store) migrateAutoVacuumContext(ctx context.Context) {
// auto_vacuum is only a per-connection setting until a VACUUM on that same
// connection rewrites the file header. Running the pragma and the VACUUM on
// separate pool connections could leave the file at NONE, so pin both to one
@@ -1092,7 +1100,7 @@ func (s *Store) runStartupMaintenance() {
start := time.Now()
// Signal the ingestion worker that the write lock may be held for a long
- // time by the deferred identity-index rebuild and the auto-vacuum VACUUM.
+ // time by the deferred table migration and the auto-vacuum VACUUM.
// Without this a large legacy database drops every metric batch enqueued
// during the one-time upgrade (#2079 finding 5).
s.startupMaintenanceActive.Store(true)
@@ -1102,10 +1110,20 @@ func (s *Store) runStartupMaintenance() {
s.startupHook()
}
- if s.identityMigrationPending.Swap(false) {
- if err := s.migrateMetricsIdentityIndex(); err != nil {
- log.Error().Err(err).Msg("Deferred metrics identity index consolidation failed")
+ // Share the existing half-hour rewrite budget between physical layout
+ // and auto-vacuum, and interrupt both when the Store is closed.
+ ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
+ defer cancel()
+ go func() {
+ select {
+ case <-s.stopCh:
+ cancel()
+ case <-ctx.Done():
}
+ }()
+
+ if err := s.migrateClusteredMetrics(ctx); err != nil {
+ log.Warn().Err(err).Msg("Clustered metrics migration deferred until next startup")
}
// Run retention before the deferred auto-vacuum conversion so restart
@@ -1113,7 +1131,7 @@ func (s *Store) runStartupMaintenance() {
// potentially rewrites the file.
s.runRetention()
s.runAvailabilityRetention()
- s.migrateAutoVacuum()
+ s.migrateAutoVacuumContext(ctx)
log.Info().Dur("duration", time.Since(start)).Msg("Deferred metrics startup maintenance completed")
}
@@ -1614,7 +1632,7 @@ type retainedQueryShape struct {
resources, metrics int
tiers [4]Tier
aggregate, groupSeries bool
- valuesOnly bool
+ valuesOnly, clustered bool
}
func retainedQueryParameters(resourceType string, resourceIDs, metricTypes []string, start, end time.Time, stepSecs int64, tiers []Tier) []interface{} {
@@ -1642,7 +1660,11 @@ func retainedQueryParameters(resourceType string, resourceIDs, metricTypes []str
}
func (s *Store) retainedQuerySQL(resourceType string, resourceIDs, metricTypes []string, start, end time.Time, stepSecs int64, tiers []Tier, groupSeries, valuesOnly bool) (string, []interface{}) {
- shape := retainedQueryShape{resources: len(resourceIDs), metrics: len(metricTypes), aggregate: stepSecs > 1, groupSeries: groupSeries, valuesOnly: valuesOnly}
+ return s.retainedQuerySQLForIndex(s.metricsReadIndex(), resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers, groupSeries, valuesOnly)
+}
+
+func (s *Store) retainedQuerySQLForIndex(index string, resourceType string, resourceIDs, metricTypes []string, start, end time.Time, stepSecs int64, tiers []Tier, groupSeries, valuesOnly bool) (string, []interface{}) {
+ shape := retainedQueryShape{resources: len(resourceIDs), metrics: len(metricTypes), aggregate: stepSecs > 1, groupSeries: groupSeries, valuesOnly: valuesOnly, clustered: index == clusteredMetricsIndex}
cacheable := len(tiers) <= len(shape.tiers)
copy(shape.tiers[:], tiers)
if cacheable {
@@ -1653,7 +1675,7 @@ func (s *Store) retainedQuerySQL(resourceType string, resourceIDs, metricTypes [
return query, retainedQueryParameters(resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers)
}
}
- query, params := retainedQuerySQL(resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers, groupSeries, valuesOnly)
+ query, params := retainedQuerySQLWithIndex(index, resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers, groupSeries, valuesOnly)
if cacheable {
s.queryMu.Lock()
if len(s.readQueryShapes) < maxRetainedReadStatements {
@@ -1673,6 +1695,10 @@ func (s *Store) retainedQuerySQL(resourceType string, resourceIDs, metricTypes [
// Presence never establishes continuous collection or per-series coverage.
// Every probe is restricted to the requested identities and timestamp window.
func retainedQuerySQL(resourceType string, resourceIDs, metricTypes []string, start, end time.Time, stepSecs int64, tiers []Tier, groupSeries, valuesOnly bool) (string, []interface{}) {
+ return retainedQuerySQLWithIndex(metricsIdentityIndex, resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers, groupSeries, valuesOnly)
+}
+
+func retainedQuerySQLWithIndex(index string, resourceType string, resourceIDs, metricTypes []string, start, end time.Time, stepSecs int64, tiers []Tier, groupSeries, valuesOnly bool) (string, []interface{}) {
params := retainedQueryParameters(resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers)
identityColumns := ""
if len(resourceIDs) != 1 {
@@ -1696,7 +1722,6 @@ func retainedQuerySQL(resourceType string, resourceIDs, metricTypes []string, st
endParam := startParam + 1
tierParam := endParam + 1
stepParam := tierParam + len(tiers)
- index := metricsIdentityIndex
scope := func(alias string, tierIndex int) string {
clause := alias + ".resource_type = :p1 AND " + alias + ".resource_id IN (" + idSlots + ")"
if len(metricTypes) > 0 {
@@ -1806,6 +1831,16 @@ func (s *Store) retainedReadStatement(query string) (*pdb.InstrumentedStmt, erro
}
func queryRetainedChunk[T any](s *Store, resourceType string, resourceIDs []string, metricTypes []string, start, end time.Time, stepSecs int64, tiers []Tier, projection retainedProjection[T]) (map[string]map[string][]T, error) {
+ result, err := queryRetainedChunkAttempt(s, resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers, projection)
+ if err != nil && s.refreshClusteredMetrics(err) {
+ return queryRetainedChunkAttempt(s, resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers, projection)
+ }
+ return result, err
+}
+
+func queryRetainedChunkAttempt[T any](s *Store, resourceType string, resourceIDs []string, metricTypes []string, start, end time.Time, stepSecs int64, tiers []Tier, projection retainedProjection[T]) (map[string]map[string][]T, error) {
+ // Keep the index selection fixed for this read snapshot across cutover.
+ index := s.metricsReadIndex()
var tx *pdb.InstrumentedTx
var err error
if stepSecs > 1 {
@@ -1817,7 +1852,6 @@ func queryRetainedChunk[T any](s *Store, resourceType string, resourceIDs []stri
// order the value projection for each presence probe.
idSlots := strings.TrimSuffix(strings.Repeat("?,", len(resourceIDs)), ",")
presenceScope := "resource_type = ? AND resource_id IN (" + idSlots + ")"
- index := metricsIdentityIndex
if len(metricTypes) > 0 {
presenceScope += " AND metric_type IN (" + strings.TrimSuffix(strings.Repeat("?,", len(metricTypes)), ",") + ")"
}
@@ -1881,7 +1915,7 @@ func queryRetainedChunk[T any](s *Store, resourceType string, resourceIDs []stri
if streamBuckets {
queryStep = 0
}
- sqlQuery, params := s.retainedQuerySQL(resourceType, resourceIDs, metricTypes, start, end, queryStep, tiers, streamBuckets, projection.valuesOnly)
+ sqlQuery, params := s.retainedQuerySQLForIndex(index, resourceType, resourceIDs, metricTypes, start, end, queryStep, tiers, streamBuckets, projection.valuesOnly)
queryRows := func() (*sql.Rows, error) { return tx.Query(sqlQuery, params...) }
if tx == nil {