Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Reduce repeated resource decoding in retained metrics reads

rcourtman7a622c0b76c8
diff --git a/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md b/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md
index bf598bd46c..9af2ea3081 100644
--- a/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md
+++ b/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md
@@ -254,3 +254,153 @@ the final native proof exercises all affected rollup paths and boundaries.
All nine required source checks for `fc58c597caf0051389c7bb3a5a6e10e7dfb3ac9b`
ultimately passed, including REST-0 at 23:57:41Z. Its allocation flag and
capacity-invalidated SLO are preserved separately from those source checks.
+
+
+## Final source comparison and absolute qualification
+
+The final shared-binding head is
+`e9dbca7c2b7159e84b922eeb5e4895ad4bb95611`. Its hosted proof in
+[run 39180037366](https://github.com/rcourtman/Pulse/actions/runs/39180037366)
+used synthetic merge `1cbbe76f2d6e063c3c59508f6da7c8688cbc82b3`, tree
+`7ab6a9f73e466e362e2fd45f877c480f4d2fe160`, with current main parent
+`6209d03d5a3895005f48b365cc2eeb8f1483f072` and that final PR head. The runner
+exposed four AMD EPYC 7763 CPUs, Go 1.26.9 and GOMAXPROCS=2. Fixed executable
+SHA-256: `74965528935c2a0da9c21dc9b33d874727aa12e430ab6916244f0b44088bbe25`.
+
+This hosted attempt failed both the explicit latency assertion and capacity
+validation: p50 85.225129 ms, p95 119.002157 ms and p99 122.928564 ms. Four of
+21 proof intervals recorded 12.16% to 25.56% background CPU, predominantly
+`provjobd2939212`. Admission's six intervals were at least 98.998% idle and the
+three after intervals at least 99.649% idle; no guest, steal or swap activity
+was recorded. Quiet admission does not excuse interference during measurement.
+All intervals and the adverse result remain retained. Artifact
+[12154824982](https://github.com/rcourtman/Pulse/actions/runs/39180037366/artifacts/12154824982)
+has GitHub digest
+`ed22be6f8ebc55bebeb5d5ba6015159b206f6f6271192d8c93665608bc6ddb7f6`.
+The entire job log and a per-file hash manifest are retained in the task's
+`pr-3070/publication-e9dbca` evidence directory. This attempt does not establish
+the idle-worker SLO.
+
+
+The final paired comparison passed. It sampled baseline
+`ed4a2c6b54ee1d139d5f198950c53b403596c17c` (tree
+`d11056db9ba2b097525b90000a550695b3a3355c`) and synthetic merge
+`1cbbe76f2d6e063c3c59508f6da7c8688cbc82b3` on the same Intel Xeon Platinum
+8573C worker, Go 1.26.9, runtime-default GOMAXPROCS, ten alternating 100 ms
+samples each. Four-hour cold charts improved 117.84 to 79.53 ms (-32.51%,
+p < 0.001); warm-handler and HTTP-transfer differences were not significant.
+Cold-chart allocation volume remained 21.47 MiB and approximately 508,200
+allocations. Small rollups improved 3.400 to 3.205 ms (-5.73%, p < 0.001),
+with 3.106 to 3.126 KiB (+0.64%) and 100 to 106 allocations (+6%). The
+500-node rollup improved 101.20 to 81.24 ms (-19.73%, p < 0.001), with
+3.875 to 3.480 KiB (-10.18%) and median 109 allocations in both samples.
+The 100-point write batch measured 1.568 to 1.672 ms (+6.58%, p = 0.007).
+The complete time, bytes and allocation comparison raised no flags against
+the unchanged >10%, p < 0.05 rule; the smaller increases are retained.
+Artifact
+[12156880181](https://github.com/rcourtman/Pulse/actions/runs/39180037366/artifacts/12156880181)
+has GitHub digest
+`d0d5af9567df10b4aa8b12e4d9495138b0ab3dd9026b03188161211edbf9ad6b`.
+Baseline/candidate metrics executable SHA-256 values were
+`52c6ab5b03a351a4a2ecdbff7875bd834a6eeb47647cae1daf463a2e51216173` and
+`c6b9500e6f752acde29a0c3441edfb0de3e39098ee05d5cddb2b08dbc245f9d3`.
+Full metadata retain the other affected-package executable identities and
+all 20 sample timestamps.
+
+All nine required source checks passed at the final PR head. Protected
+[PR 3070](https://github.com/rcourtman/Pulse/pull/3070) merged at
+2026-10-19T00:36:33Z as `5153c4ccda35cc50f12f32e3f6b98a87cb90f6b5`, with
+the same tree `7ab6a9f73e466e362e2fd45f877c480f4d2fe160` as the tested synthetic
+merge. That source landing neither waives the capacity-invalidated hosted
+proof nor constitutes release qualification.
+
+## Observed idle failure and remaining row-boundary cost
+
+The next disposable proof,
+[run 39181569505](https://github.com/rcourtman/pulse-enterprise/actions/runs/39181569505),
+used that exact landed public source, private control
+`4f8f9542ee122471c0bfe4f51334e7ef0d19796b`, Go 1.26.9 and two Intel Xeon
+3.30 GHz vCPUs with GOMAXPROCS=2. The fixed executable SHA-256 was
+`416ac833fc16778efe0fe09e94775d25e93ffe8fde0d4bf0e1b42d060f0beed9`.
+This time capacity validation passed: six admission samples established
+30.003 seconds of quiet capacity (at least 98.596% idle), all 19 measurement
+intervals had at most 2% background CPU, and guest, steal and swap were zero.
+The three after samples were at least 99.4% idle. The unchanged strict test
+still failed: p50 77.749800 ms, p95 90.425504 ms and p99 98.667173 ms against
+90 ms. The 0.425504 ms miss remains a failure; rounding or another unchanged
+attempt does not establish the SLO.
+
+All ten retained textual files were extracted with original byte counts and
+SHA-256 hashes verified. The complete job log SHA-256 is
+`b1f55ca54d3dc9b500eb1cde3e12db7950d4a037020d1b737964ebe6cc4b63d6`.
+The package also resides in the native task's
+`idle-evidence/private-proof-39181569505` directory, with its per-file manifest
+verified after transfer. No Actions diagnostic artifact was published.
+
+The separate diagnostic measured 78.198713 ms/op, 22,322,470 B/op and 508,181
+allocations. Its profile attributed 77.81% cumulative CPU to the retained
+reader, 63.66% to `database/sql.Rows.Next`, 40.78% to SQLite's VM and 11.79%
+to record comparisons. Text-column decoding accounted for 12.62%. These CPU
+samples include the benchmark's setup; they are attribution evidence, not a
+replacement SLO measurement.
+
+A bounded linear metric-filter experiment returned identical observations
+but did not demonstrate a consistent timing improvement, so the production
+membership predicate remains unchanged. A second alternating probe replaced
+repeated resource-ID text columns with a request-local integer and explicitly
+ordered completed series in Go. Both variants returned 36,000 observations
+with digest
+`0720c6e78804f01075db817e20fb5920f078519523af67e5fce7b2d1db04f8b0`.
+Twelve native rounds had median 58.2208005 ms for text identities versus
+51.237231 ms for resource numbers. The new plan retained bounded primary
+ranges and no temporary value sort. Shared-worker timings remain diagnostic.
+
+The shared retained reader now binds a bounded requested-resource table for
+raw fleet reads and returns the statement-local resource number rather than
+decoding/copying its resource ID for every observation. Numbers resolve against
+that statement's normalized resource list and never become persisted or public
+identity. The same metric/tier/time predicates and coverage probes select the
+observations. Completed detached series explicitly restore stable timestamp
+order in Go, removing SQLite's merge of tier branches by repeated identity
+strings. Single-resource reads and ordered streamed display buckets retain
+their existing paths and bounds. Regression coverage compares both point
+projections with single-resource reads on rowid and clustered layouts, including
+mixed tiers, nullable/zero extrema, reused names in another resource type,
+unrequested metrics, reordered scopes, missing/duplicate IDs and the
+500-resource chunk boundary. Final qualification is recorded below.
+
+The candidate passed the native full metrics suite, retained-reader correctness
+and query-plan checks, retained race checks (31.366 s), chart API tests, selected
+downstream chart sampling/SLO tests (24.315 s), and v5 metrics compatibility
+(0.143 s). The probe was hash-checked and moved outside the package before
+compiling either baseline or candidate. All qualification exits were zero.
+
+The baseline metrics/API executable SHA-256 values were
+`ce878df8c9df1193b457742ef913dac8073f29777b53aeb9148fecfd8b9a1686` and
+`ce964286839a7621c149a0ee3b7ffec06a1114f999f70a1053bc3c0b34159469`.
+The corresponding candidate values were
+`3de1f7a264c1b09a2cb89d2c4440fa4b66417576853b4755757c255665ef5b79` and
+`51c7b0cdb6d2fa371b9513e355f4a681e9b4bb134732f6967fd009ed4f0becd4`.
+Four source hashes are retained in `ordinal-query/candidate-source.sha256`;
+`store.go` is `535bce025ab3e544b6104bbd537b9c40022debdad0b96df91e6d5469c4162cba`
+and `store_query_plan_test.go` is
+`5aa9149ea8c4e1e201112fbe83d636240d1435f5bee8b686c87506e3d4b4f9b3`.
+
+Ten alternating native 100 ms samples per variant/case, with fixed binaries
+and GOMAXPROCS=2, measured cold-chart median allocation volume of 22,509,840
+versus 19,696,848 bytes (-12.50%) and counts of 508,193 versus 340,887.5
+(-32.92%). Median chart times were 102.660828 versus 103.846864 ms; their
+respective full ranges were 87.728537–134.246896 and
+85.662542–146.783133 ms. The adverse median and all samples remain retained.
+This shared-worker comparison establishes neither a chart latency improvement
+nor the absolute budget.
+
+Batch-read medians for 10, 50 and 100 resources were respectively
+7.429404/5.795782, 36.9792995/31.2997015 and 69.734954/62.305637 ms
+(baseline/candidate). The unchanged single-resource path measured
+0.6401185/0.5719735 ms. The unchanged streamed 50-resource display-bucket path
+measured 43.810747/44.7852855 ms, with median 120,605 allocations in both
+variants. These native timings are diagnostic. Complete logs, exits, source
+and executable manifests and the 120-row raw sample table are retained in
+`ordinal-query-qualified`. Fresh paired CI and the unchanged strict proof on
+observed idle capacity remain required after source landing.
diff --git a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md
index 15cd7d7fe8..3e467ac6d5 100644
--- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md
+++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md
@@ -2765,7 +2765,17 @@ Each retained read fixes its index choice for its snapshot. A stale legacy
index error may retry once with a new snapshot only after inspecting the
clustered primary key. This covers another Store committing the upgrade while
prepared SQL is cached. Layout participates in the bounded SQL-template key;
-results and tier presence remain uncached. Restart recognizes the primary key
+results and tier presence remain uncached. Raw multi-resource reads bind one
+bounded request table and return statement-local resource numbers across the
+SQLite row boundary. Resolve each number against that statement's normalized
+resource list; never persist or expose it as resource identity. Tier discovery,
+coverage probes and value ranges retain the same scoped snapshot and index
+bounds. These detached raw series explicitly restore stable timestamp order in
+Go before Docker observation projection, without a fleet-wide SQLite sort or
+merge by repeated identity strings. Ordered streamed display buckets retain
+their existing SQL path and allocation bound. The row shape follows the same
+resource-count, step and grouping inputs as the bounded SQL-template cache.
+Restart recognizes the primary key
without recreating the legacy index. If an older executable reopens the
row-compatible table and recreates its legacy identity index, deferred
maintenance removes only the known redundant trees after verifying the primary
diff --git a/pkg/metrics/store.go b/pkg/metrics/store.go
index 3d73865290..05662eb846 100644
--- a/pkg/metrics/store.go
+++ b/pkg/metrics/store.go
@@ -1529,12 +1529,20 @@ func (s *Store) QueryMetricValuesBatch(resourceType string, resourceIDs, metricT
type retainedProjection[T any] struct {
valuesOnly bool
point func(MetricPoint) T
+ compare func(T, T) int
}
-var retainedMetricPoints = retainedProjection[MetricPoint]{point: func(p MetricPoint) MetricPoint { return p }}
-var retainedMetricValues = retainedProjection[models.MetricPoint]{valuesOnly: true, point: func(p MetricPoint) models.MetricPoint {
- return models.MetricPoint{Value: p.Value, Timestamp: p.Timestamp}
-}}
+var retainedMetricPoints = retainedProjection[MetricPoint]{
+ point: func(p MetricPoint) MetricPoint { return p },
+ compare: func(a, b MetricPoint) int { return a.Timestamp.Compare(b.Timestamp) },
+}
+var retainedMetricValues = retainedProjection[models.MetricPoint]{
+ valuesOnly: true,
+ point: func(p MetricPoint) models.MetricPoint {
+ return models.MetricPoint{Value: p.Value, Timestamp: p.Timestamp}
+ },
+ compare: func(a, b models.MetricPoint) int { return a.Timestamp.Compare(b.Timestamp) },
+}
func (s *Store) queryBatch(
resourceType string,
@@ -1698,11 +1706,25 @@ func retainedQuerySQL(resourceType string, resourceIDs, metricTypes []string, st
return retainedQuerySQLWithIndex(metricsIdentityIndex, resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers, groupSeries, valuesOnly)
}
+// Raw fleet reads return a statement-local resource number instead of decoding
+// the same resource ID for every observation. Their detached output already
+// holds every point, so order each completed series in Go rather than merging
+// tier branches by repeated identity strings in SQLite. Streamed display
+// buckets keep their ordered SQL path and bounded output allocation.
+func retainedQueryUsesResourceOrdinals(resources int, stepSecs int64, groupSeries bool) bool {
+ return resources > 1 && stepSecs <= 1 && !groupSeries
+}
+
func retainedQuerySQLWithIndex(index string, resourceType string, resourceIDs, metricTypes []string, start, end time.Time, stepSecs int64, tiers []Tier, groupSeries, valuesOnly bool) (string, []interface{}) {
params := retainedQueryParameters(resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers)
+ resourceOrdinals := retainedQueryUsesResourceOrdinals(len(resourceIDs), stepSecs, groupSeries)
identityColumns := ""
if len(resourceIDs) != 1 {
- identityColumns += "resource_id, "
+ if resourceOrdinals {
+ identityColumns += "r.resource_number, "
+ } else {
+ identityColumns += "resource_id, "
+ }
}
if len(metricTypes) != 1 {
identityColumns += "metric_type, "
@@ -1724,6 +1746,9 @@ func retainedQuerySQLWithIndex(index string, resourceType string, resourceIDs, m
stepParam := tierParam + len(tiers)
scope := func(alias string, tierIndex int) string {
clause := alias + ".resource_type = :p1 AND " + alias + ".resource_id IN (" + idSlots + ")"
+ if resourceOrdinals && alias == "m" {
+ clause = alias + ".resource_type = :p1 AND " + alias + ".resource_id = r.resource_id"
+ }
if len(metricTypes) > 0 {
clause += " AND " + alias + ".metric_type IN (" + metricSlots + ")"
}
@@ -1746,7 +1771,13 @@ func retainedQuerySQLWithIndex(index string, resourceType string, resourceIDs, m
}
branches := make([]string, 0, len(tiers))
for i, tier := range tiers {
- branch := "SELECT " + projection + " FROM metrics AS m INDEXED BY " + index + " WHERE " + scope("m", i)
+ from := "metrics AS m INDEXED BY " + index
+ if resourceOrdinals {
+ // Keep each requested resource outside its indexed value range. The
+ // request table is bounded by the existing 500-resource chunk size.
+ from = "requested_resources AS r CROSS JOIN " + from
+ }
+ branch := "SELECT " + projection + " FROM " + from + " WHERE " + scope("m", i)
for j, preferred := range tiers[:i] {
// SQLite evaluates the uncorrelated existence check once. An empty
// preferred tier must not incur a correlated index probe for every
@@ -1775,7 +1806,7 @@ func retainedQuerySQLWithIndex(index string, resourceType string, resourceIDs, m
aggregate += `, MIN(COALESCE(min_value, value)), MAX(COALESCE(max_value, value))`
}
query = aggregate + " FROM (" + query + ") GROUP BY " + identityColumns + "bucket_ts ORDER BY " + identityColumns + "bucket_ts ASC"
- } else {
+ } else if !resourceOrdinals {
orderColumns := identityColumns
if !groupSeries {
// The identity index is time-major within each resource. Metric
@@ -1789,6 +1820,13 @@ func retainedQuerySQLWithIndex(index string, resourceType string, resourceIDs, m
}
query += " ORDER BY " + orderColumns + "timestamp ASC"
}
+ if resourceOrdinals {
+ resources := make([]string, len(resourceIDs))
+ for i := range resources {
+ resources[i] = fmt.Sprintf("(%d,:p%d)", i, i+2)
+ }
+ query = "WITH requested_resources(resource_number, resource_id) AS MATERIALIZED (VALUES " + strings.Join(resources, ",") + ") " + query
+ }
return query, params
}
@@ -1916,6 +1954,7 @@ func queryRetainedChunkAttempt[T any](s *Store, resourceType string, resourceIDs
queryStep = 0
}
sqlQuery, params := s.retainedQuerySQLForIndex(index, resourceType, resourceIDs, metricTypes, start, end, queryStep, tiers, streamBuckets, projection.valuesOnly)
+ resourceOrdinals := retainedQueryUsesResourceOrdinals(len(resourceIDs), queryStep, streamBuckets)
queryRows := func() (*sql.Rows, error) { return tx.Query(sqlQuery, params...) }
if tx == nil {
@@ -1995,12 +2034,15 @@ func queryRetainedChunkAttempt[T any](s *Store, resourceType string, resourceIDs
bucketCount = 0
}
var resourceID, metricType string
+ var resourceNumber int
var ts int64
var p MetricPoint
var low, high sql.NullFloat64
destinations := make([]interface{}, 0, 6)
if len(resourceIDs) == 1 {
resourceID = resourceIDs[0]
+ } else if resourceOrdinals {
+ destinations = append(destinations, &resourceNumber)
} else {
destinations = append(destinations, &resourceID)
}
@@ -2018,6 +2060,12 @@ func queryRetainedChunkAttempt[T any](s *Store, resourceType string, resourceIDs
log.Warn().Err(err).Msg("Failed to scan batch metric row")
continue
}
+ if resourceOrdinals {
+ if resourceNumber < 0 || resourceNumber >= len(resourceIDs) {
+ return nil, fmt.Errorf("invalid retained resource number %d for %d resources", resourceNumber, len(resourceIDs))
+ }
+ resourceID = resourceIDs[resourceNumber]
+ }
if !projection.valuesOnly {
p.Min, p.Max = p.Value, p.Value
if low.Valid {
@@ -2053,6 +2101,15 @@ func queryRetainedChunkAttempt[T any](s *Store, resourceType string, resourceIDs
flushBucket()
flushSeries()
+ if resourceOrdinals {
+ for _, metrics := range result {
+ for _, points := range metrics {
+ if !slices.IsSortedFunc(points, projection.compare) {
+ slices.SortStableFunc(points, projection.compare)
+ }
+ }
+ }
+ }
if hasDockerObservationContract(resourceType) {
projectDockerObservations(result)
}
diff --git a/pkg/metrics/store_query_plan_test.go b/pkg/metrics/store_query_plan_test.go
index e9e7fafde2..b7b1a74f3b 100644
--- a/pkg/metrics/store_query_plan_test.go
+++ b/pkg/metrics/store_query_plan_test.go
@@ -4,10 +4,13 @@ import (
"database/sql"
"fmt"
"net/url"
+ "reflect"
"strings"
"testing"
"time"
+ pdb "github.com/rcourtman/pulse-go-rewrite/pkg/db"
+
_ "modernc.org/sqlite"
)
@@ -550,6 +553,9 @@ func TestRetainedUnaggregatedQueryAvoidsMetricSort(t *testing.T) {
if strings.Contains(strings.ToUpper(line), "TEMP B-TREE") {
t.Fatalf("unaggregated read sorts despite time-ordered index: %s", plan)
}
+ if len(ids) > 1 && strings.Contains(line, "MERGE (UNION ALL)") {
+ t.Fatalf("raw fleet read merges repeated identity strings across tiers: %s", plan)
+ }
}
})
}
@@ -557,3 +563,109 @@ func TestRetainedUnaggregatedQueryAvoidsMetricSort(t *testing.T) {
}
}
}
+
+func TestRetainedResourceNumbersPreserveScopeAndOrderedSeries(t *testing.T) {
+ for _, clustered := range []bool{false, true} {
+ t.Run(fmt.Sprintf("clustered_%t", clustered), func(t *testing.T) {
+ db := newPlanTestDB(t)
+ if clustered {
+ if _, err := db.Exec(`DROP TABLE metrics; CREATE TABLE metrics (` + metricsTableColumns + `) WITHOUT ROWID;
+ CREATE INDEX idx_metrics_tier_time ON metrics(tier,timestamp)`); err != nil {
+ t.Fatal(err)
+ }
+ }
+ store := &Store{db: pdb.Wrap(db, "retained-resource-number-test")}
+ store.clusteredMetrics.Store(clustered)
+ start := time.Unix(1_800_003_600, 0)
+ end := start.Add(3 * time.Hour)
+ ids := make([]string, queryAllBatchChunkSize+3)
+ tx, err := db.Begin()
+ if err != nil {
+ t.Fatal(err)
+ }
+ insert, err := tx.Prepare(`INSERT INTO metrics(resource_type,resource_id,metric_type,value,min_value,max_value,timestamp,tier) VALUES (?,?,?,?,?,?,?,?)`)
+ if err != nil {
+ t.Fatal(err)
+ }
+ for i := range ids {
+ // Deliberately reverse lexical order, including IDs beyond the
+ // chunk boundary. A resource number belongs only to its statement.
+ ids[i] = fmt.Sprintf("guest-'-%04d", len(ids)-i)
+ for _, metric := range []string{"cpu", "memory", "unrequested"} {
+ for _, row := range []struct {
+ offset int64
+ tier Tier
+ value float64
+ low, high any
+ }{
+ {60, TierMinute, 10, float64(0), float64(20)},
+ {65, TierRaw, 999, nil, nil}, // owned by the minute bucket
+ {180, TierRaw, 30, nil, nil},
+ {300, TierMinute, 50, nil, nil},
+ {15, TierRaw, 5, nil, nil}, // emitted after minute, must sort
+ {0, TierHourly, -100, nil, nil},
+ {-1, TierRaw, -999, nil, nil},
+ {3*3600 + 1, TierRaw, -999, nil, nil},
+ } {
+ if _, err := insert.Exec("vm", ids[i], metric, row.value+float64(i), row.low, row.high, start.Unix()+row.offset, string(row.tier)); err != nil {
+ t.Fatal(err)
+ }
+ }
+ }
+ if _, err := insert.Exec("container", ids[i], "cpu", 9999, nil, nil, start.Unix()+60, string(TierMinute)); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if _, err := insert.Exec("vm", "outside-scope", "cpu", 9999, nil, nil, start.Unix()+60, string(TierMinute)); err != nil {
+ t.Fatal(err)
+ }
+ if err := insert.Close(); err != nil {
+ t.Fatal(err)
+ }
+ if err := tx.Commit(); err != nil {
+ t.Fatal(err)
+ }
+
+ check := func(requested, filters []string, step int64) {
+ t.Helper()
+ full, err := store.QueryMetricTypesBatch("vm", requested, filters, start, end, step)
+ if err != nil {
+ t.Fatal(err)
+ }
+ values, err := store.QueryMetricValuesBatch("vm", requested, filters, start, end, step)
+ if err != nil {
+ t.Fatal(err)
+ }
+ assertMetricValueProjection(t, full, values)
+ seen := make(map[string]bool)
+ for _, id := range requested {
+ id = normalizeMetricIdentifier(id)
+ if id == "" || seen[id] {
+ continue
+ }
+ seen[id] = true
+ one, err := store.QueryMetricTypesBatch("vm", []string{id}, filters, start, end, step)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !reflect.DeepEqual(full[id], one[id]) {
+ t.Fatalf("scope/order changed for %q step=%d: batch=%+v single=%+v", id, step, full[id], one[id])
+ }
+ }
+ if _, exists := full["outside-scope"]; exists {
+ t.Fatal("unrequested resource returned")
+ }
+ if len(full) != len(seen)-1 { // one requested ID is deliberately absent
+ t.Fatalf("resource count=%d requested distinct=%d", len(full), len(seen))
+ }
+ }
+ for _, filters := range [][]string{nil, {"cpu"}, {"memory", "cpu", "absent"}} {
+ for _, step := range []int64{0, 1, 60} {
+ check([]string{ids[2], ids[0], ids[1], ids[2], "", "missing"}, filters, step)
+ check([]string{ids[1], ids[2], ids[0], "missing"}, filters, step)
+ }
+ }
+ check(append(append([]string{}, ids...), "missing", ids[0]), nil, 0)
+ })
+ }
+}