Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Reduce repeated resource decoding in retained metrics reads
rcourtman7a622c0b76c8
diff --git a/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md b/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.mdindex bf598bd46c..9af2ea3081 100644--- a/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md+++ b/docs/release-control/v6/internal/records/workload-chart-idle-proof-2026-10-18.md@@ -254,3 +254,153 @@ the final native proof exercises all affected rollup paths and boundaries.All nine required source checks for `fc58c597caf0051389c7bb3a5a6e10e7dfb3ac9b`ultimately passed, including REST-0 at 23:57:41Z. Its allocation flag andcapacity-invalidated SLO are preserved separately from those source checks.+++## Final source comparison and absolute qualification++The final shared-binding head is+`e9dbca7c2b7159e84b922eeb5e4895ad4bb95611`. Its hosted proof in+[run 39180037366](https://github.com/rcourtman/Pulse/actions/runs/39180037366)+used synthetic merge `1cbbe76f2d6e063c3c59508f6da7c8688cbc82b3`, tree+`7ab6a9f73e466e362e2fd45f877c480f4d2fe160`, with current main parent+`6209d03d5a3895005f48b365cc2eeb8f1483f072` and that final PR head. The runner+exposed four AMD EPYC 7763 CPUs, Go 1.26.9 and GOMAXPROCS=2. Fixed executable+SHA-256: `74965528935c2a0da9c21dc9b33d874727aa12e430ab6916244f0b44088bbe25`.++This hosted attempt failed both the explicit latency assertion and capacity+validation: p50 85.225129 ms, p95 119.002157 ms and p99 122.928564 ms. Four of+21 proof intervals recorded 12.16% to 25.56% background CPU, predominantly+`provjobd2939212`. Admission's six intervals were at least 98.998% idle and the+three after intervals at least 99.649% idle; no guest, steal or swap activity+was recorded. Quiet admission does not excuse interference during measurement.+All intervals and the adverse result remain retained. Artifact+[12154824982](https://github.com/rcourtman/Pulse/actions/runs/39180037366/artifacts/12154824982)+has GitHub digest+`ed22be6f8ebc55bebeb5d5ba6015159b206f6f6271192d8c93665608bc6ddb7f6`.+The entire job log and a per-file hash manifest are retained in the task's+`pr-3070/publication-e9dbca` evidence directory. This attempt does not establish+the idle-worker SLO.+++The final paired comparison passed. It sampled baseline+`ed4a2c6b54ee1d139d5f198950c53b403596c17c` (tree+`d11056db9ba2b097525b90000a550695b3a3355c`) and synthetic merge+`1cbbe76f2d6e063c3c59508f6da7c8688cbc82b3` on the same Intel Xeon Platinum+8573C worker, Go 1.26.9, runtime-default GOMAXPROCS, ten alternating 100 ms+samples each. Four-hour cold charts improved 117.84 to 79.53 ms (-32.51%,+p < 0.001); warm-handler and HTTP-transfer differences were not significant.+Cold-chart allocation volume remained 21.47 MiB and approximately 508,200+allocations. Small rollups improved 3.400 to 3.205 ms (-5.73%, p < 0.001),+with 3.106 to 3.126 KiB (+0.64%) and 100 to 106 allocations (+6%). The+500-node rollup improved 101.20 to 81.24 ms (-19.73%, p < 0.001), with+3.875 to 3.480 KiB (-10.18%) and median 109 allocations in both samples.+The 100-point write batch measured 1.568 to 1.672 ms (+6.58%, p = 0.007).+The complete time, bytes and allocation comparison raised no flags against+the unchanged >10%, p < 0.05 rule; the smaller increases are retained.+Artifact+[12156880181](https://github.com/rcourtman/Pulse/actions/runs/39180037366/artifacts/12156880181)+has GitHub digest+`d0d5af9567df10b4aa8b12e4d9495138b0ab3dd9026b03188161211edbf9ad6b`.+Baseline/candidate metrics executable SHA-256 values were+`52c6ab5b03a351a4a2ecdbff7875bd834a6eeb47647cae1daf463a2e51216173` and+`c6b9500e6f752acde29a0c3441edfb0de3e39098ee05d5cddb2b08dbc245f9d3`.+Full metadata retain the other affected-package executable identities and+all 20 sample timestamps.++All nine required source checks passed at the final PR head. Protected+[PR 3070](https://github.com/rcourtman/Pulse/pull/3070) merged at+2026-10-19T00:36:33Z as `5153c4ccda35cc50f12f32e3f6b98a87cb90f6b5`, with+the same tree `7ab6a9f73e466e362e2fd45f877c480f4d2fe160` as the tested synthetic+merge. That source landing neither waives the capacity-invalidated hosted+proof nor constitutes release qualification.++## Observed idle failure and remaining row-boundary cost++The next disposable proof,+[run 39181569505](https://github.com/rcourtman/pulse-enterprise/actions/runs/39181569505),+used that exact landed public source, private control+`4f8f9542ee122471c0bfe4f51334e7ef0d19796b`, Go 1.26.9 and two Intel Xeon+3.30 GHz vCPUs with GOMAXPROCS=2. The fixed executable SHA-256 was+`416ac833fc16778efe0fe09e94775d25e93ffe8fde0d4bf0e1b42d060f0beed9`.+This time capacity validation passed: six admission samples established+30.003 seconds of quiet capacity (at least 98.596% idle), all 19 measurement+intervals had at most 2% background CPU, and guest, steal and swap were zero.+The three after samples were at least 99.4% idle. The unchanged strict test+still failed: p50 77.749800 ms, p95 90.425504 ms and p99 98.667173 ms against+90 ms. The 0.425504 ms miss remains a failure; rounding or another unchanged+attempt does not establish the SLO.++All ten retained textual files were extracted with original byte counts and+SHA-256 hashes verified. The complete job log SHA-256 is+`b1f55ca54d3dc9b500eb1cde3e12db7950d4a037020d1b737964ebe6cc4b63d6`.+The package also resides in the native task's+`idle-evidence/private-proof-39181569505` directory, with its per-file manifest+verified after transfer. No Actions diagnostic artifact was published.++The separate diagnostic measured 78.198713 ms/op, 22,322,470 B/op and 508,181+allocations. Its profile attributed 77.81% cumulative CPU to the retained+reader, 63.66% to `database/sql.Rows.Next`, 40.78% to SQLite's VM and 11.79%+to record comparisons. Text-column decoding accounted for 12.62%. These CPU+samples include the benchmark's setup; they are attribution evidence, not a+replacement SLO measurement.++A bounded linear metric-filter experiment returned identical observations+but did not demonstrate a consistent timing improvement, so the production+membership predicate remains unchanged. A second alternating probe replaced+repeated resource-ID text columns with a request-local integer and explicitly+ordered completed series in Go. Both variants returned 36,000 observations+with digest+`0720c6e78804f01075db817e20fb5920f078519523af67e5fce7b2d1db04f8b0`.+Twelve native rounds had median 58.2208005 ms for text identities versus+51.237231 ms for resource numbers. The new plan retained bounded primary+ranges and no temporary value sort. Shared-worker timings remain diagnostic.++The shared retained reader now binds a bounded requested-resource table for+raw fleet reads and returns the statement-local resource number rather than+decoding/copying its resource ID for every observation. Numbers resolve against+that statement's normalized resource list and never become persisted or public+identity. The same metric/tier/time predicates and coverage probes select the+observations. Completed detached series explicitly restore stable timestamp+order in Go, removing SQLite's merge of tier branches by repeated identity+strings. Single-resource reads and ordered streamed display buckets retain+their existing paths and bounds. Regression coverage compares both point+projections with single-resource reads on rowid and clustered layouts, including+mixed tiers, nullable/zero extrema, reused names in another resource type,+unrequested metrics, reordered scopes, missing/duplicate IDs and the+500-resource chunk boundary. Final qualification is recorded below.++The candidate passed the native full metrics suite, retained-reader correctness+and query-plan checks, retained race checks (31.366 s), chart API tests, selected+downstream chart sampling/SLO tests (24.315 s), and v5 metrics compatibility+(0.143 s). The probe was hash-checked and moved outside the package before+compiling either baseline or candidate. All qualification exits were zero.++The baseline metrics/API executable SHA-256 values were+`ce878df8c9df1193b457742ef913dac8073f29777b53aeb9148fecfd8b9a1686` and+`ce964286839a7621c149a0ee3b7ffec06a1114f999f70a1053bc3c0b34159469`.+The corresponding candidate values were+`3de1f7a264c1b09a2cb89d2c4440fa4b66417576853b4755757c255665ef5b79` and+`51c7b0cdb6d2fa371b9513e355f4a681e9b4bb134732f6967fd009ed4f0becd4`.+Four source hashes are retained in `ordinal-query/candidate-source.sha256`;+`store.go` is `535bce025ab3e544b6104bbd537b9c40022debdad0b96df91e6d5469c4162cba`+and `store_query_plan_test.go` is+`5aa9149ea8c4e1e201112fbe83d636240d1435f5bee8b686c87506e3d4b4f9b3`.++Ten alternating native 100 ms samples per variant/case, with fixed binaries+and GOMAXPROCS=2, measured cold-chart median allocation volume of 22,509,840+versus 19,696,848 bytes (-12.50%) and counts of 508,193 versus 340,887.5+(-32.92%). Median chart times were 102.660828 versus 103.846864 ms; their+respective full ranges were 87.728537–134.246896 and+85.662542–146.783133 ms. The adverse median and all samples remain retained.+This shared-worker comparison establishes neither a chart latency improvement+nor the absolute budget.++Batch-read medians for 10, 50 and 100 resources were respectively+7.429404/5.795782, 36.9792995/31.2997015 and 69.734954/62.305637 ms+(baseline/candidate). The unchanged single-resource path measured+0.6401185/0.5719735 ms. The unchanged streamed 50-resource display-bucket path+measured 43.810747/44.7852855 ms, with median 120,605 allocations in both+variants. These native timings are diagnostic. Complete logs, exits, source+and executable manifests and the 120-row raw sample table are retained in+`ordinal-query-qualified`. Fresh paired CI and the unchanged strict proof on+observed idle capacity remain required after source landing.diff --git a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md b/docs/release-control/v6/internal/subsystems/performance-and-scalability.mdindex 15cd7d7fe8..3e467ac6d5 100644--- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md+++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md@@ -2765,7 +2765,17 @@ Each retained read fixes its index choice for its snapshot. A stale legacyindex error may retry once with a new snapshot only after inspecting theclustered primary key. This covers another Store committing the upgrade whileprepared SQL is cached. Layout participates in the bounded SQL-template key;-results and tier presence remain uncached. Restart recognizes the primary key+results and tier presence remain uncached. Raw multi-resource reads bind one+bounded request table and return statement-local resource numbers across the+SQLite row boundary. Resolve each number against that statement's normalized+resource list; never persist or expose it as resource identity. Tier discovery,+coverage probes and value ranges retain the same scoped snapshot and index+bounds. These detached raw series explicitly restore stable timestamp order in+Go before Docker observation projection, without a fleet-wide SQLite sort or+merge by repeated identity strings. Ordered streamed display buckets retain+their existing SQL path and allocation bound. The row shape follows the same+resource-count, step and grouping inputs as the bounded SQL-template cache.+Restart recognizes the primary keywithout recreating the legacy index. If an older executable reopens therow-compatible table and recreates its legacy identity index, deferredmaintenance removes only the known redundant trees after verifying the primarydiff --git a/pkg/metrics/store.go b/pkg/metrics/store.goindex 3d73865290..05662eb846 100644--- a/pkg/metrics/store.go+++ b/pkg/metrics/store.go@@ -1529,12 +1529,20 @@ func (s *Store) QueryMetricValuesBatch(resourceType string, resourceIDs, metricTtype retainedProjection[T any] struct {valuesOnly boolpoint func(MetricPoint) T+ compare func(T, T) int}-var retainedMetricPoints = retainedProjection[MetricPoint]{point: func(p MetricPoint) MetricPoint { return p }}-var retainedMetricValues = retainedProjection[models.MetricPoint]{valuesOnly: true, point: func(p MetricPoint) models.MetricPoint {- return models.MetricPoint{Value: p.Value, Timestamp: p.Timestamp}-}}+var retainedMetricPoints = retainedProjection[MetricPoint]{+ point: func(p MetricPoint) MetricPoint { return p },+ compare: func(a, b MetricPoint) int { return a.Timestamp.Compare(b.Timestamp) },+}+var retainedMetricValues = retainedProjection[models.MetricPoint]{+ valuesOnly: true,+ point: func(p MetricPoint) models.MetricPoint {+ return models.MetricPoint{Value: p.Value, Timestamp: p.Timestamp}+ },+ compare: func(a, b models.MetricPoint) int { return a.Timestamp.Compare(b.Timestamp) },+}func (s *Store) queryBatch(resourceType string,@@ -1698,11 +1706,25 @@ func retainedQuerySQL(resourceType string, resourceIDs, metricTypes []string, streturn retainedQuerySQLWithIndex(metricsIdentityIndex, resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers, groupSeries, valuesOnly)}+// Raw fleet reads return a statement-local resource number instead of decoding+// the same resource ID for every observation. Their detached output already+// holds every point, so order each completed series in Go rather than merging+// tier branches by repeated identity strings in SQLite. Streamed display+// buckets keep their ordered SQL path and bounded output allocation.+func retainedQueryUsesResourceOrdinals(resources int, stepSecs int64, groupSeries bool) bool {+ return resources > 1 && stepSecs <= 1 && !groupSeries+}+func retainedQuerySQLWithIndex(index string, resourceType string, resourceIDs, metricTypes []string, start, end time.Time, stepSecs int64, tiers []Tier, groupSeries, valuesOnly bool) (string, []interface{}) {params := retainedQueryParameters(resourceType, resourceIDs, metricTypes, start, end, stepSecs, tiers)+ resourceOrdinals := retainedQueryUsesResourceOrdinals(len(resourceIDs), stepSecs, groupSeries)identityColumns := ""if len(resourceIDs) != 1 {- identityColumns += "resource_id, "+ if resourceOrdinals {+ identityColumns += "r.resource_number, "+ } else {+ identityColumns += "resource_id, "+ }}if len(metricTypes) != 1 {identityColumns += "metric_type, "@@ -1724,6 +1746,9 @@ func retainedQuerySQLWithIndex(index string, resourceType string, resourceIDs, mstepParam := tierParam + len(tiers)scope := func(alias string, tierIndex int) string {clause := alias + ".resource_type = :p1 AND " + alias + ".resource_id IN (" + idSlots + ")"+ if resourceOrdinals && alias == "m" {+ clause = alias + ".resource_type = :p1 AND " + alias + ".resource_id = r.resource_id"+ }if len(metricTypes) > 0 {clause += " AND " + alias + ".metric_type IN (" + metricSlots + ")"}@@ -1746,7 +1771,13 @@ func retainedQuerySQLWithIndex(index string, resourceType string, resourceIDs, m}branches := make([]string, 0, len(tiers))for i, tier := range tiers {- branch := "SELECT " + projection + " FROM metrics AS m INDEXED BY " + index + " WHERE " + scope("m", i)+ from := "metrics AS m INDEXED BY " + index+ if resourceOrdinals {+ // Keep each requested resource outside its indexed value range. The+ // request table is bounded by the existing 500-resource chunk size.+ from = "requested_resources AS r CROSS JOIN " + from+ }+ branch := "SELECT " + projection + " FROM " + from + " WHERE " + scope("m", i)for j, preferred := range tiers[:i] {// SQLite evaluates the uncorrelated existence check once. An empty// preferred tier must not incur a correlated index probe for every@@ -1775,7 +1806,7 @@ func retainedQuerySQLWithIndex(index string, resourceType string, resourceIDs, maggregate += `, MIN(COALESCE(min_value, value)), MAX(COALESCE(max_value, value))`}query = aggregate + " FROM (" + query + ") GROUP BY " + identityColumns + "bucket_ts ORDER BY " + identityColumns + "bucket_ts ASC"- } else {+ } else if !resourceOrdinals {orderColumns := identityColumnsif !groupSeries {// The identity index is time-major within each resource. Metric@@ -1789,6 +1820,13 @@ func retainedQuerySQLWithIndex(index string, resourceType string, resourceIDs, m}query += " ORDER BY " + orderColumns + "timestamp ASC"}+ if resourceOrdinals {+ resources := make([]string, len(resourceIDs))+ for i := range resources {+ resources[i] = fmt.Sprintf("(%d,:p%d)", i, i+2)+ }+ query = "WITH requested_resources(resource_number, resource_id) AS MATERIALIZED (VALUES " + strings.Join(resources, ",") + ") " + query+ }return query, params}@@ -1916,6 +1954,7 @@ func queryRetainedChunkAttempt[T any](s *Store, resourceType string, resourceIDsqueryStep = 0}sqlQuery, params := s.retainedQuerySQLForIndex(index, resourceType, resourceIDs, metricTypes, start, end, queryStep, tiers, streamBuckets, projection.valuesOnly)+ resourceOrdinals := retainedQueryUsesResourceOrdinals(len(resourceIDs), queryStep, streamBuckets)queryRows := func() (*sql.Rows, error) { return tx.Query(sqlQuery, params...) }if tx == nil {@@ -1995,12 +2034,15 @@ func queryRetainedChunkAttempt[T any](s *Store, resourceType string, resourceIDsbucketCount = 0}var resourceID, metricType string+ var resourceNumber intvar ts int64var p MetricPointvar low, high sql.NullFloat64destinations := make([]interface{}, 0, 6)if len(resourceIDs) == 1 {resourceID = resourceIDs[0]+ } else if resourceOrdinals {+ destinations = append(destinations, &resourceNumber)} else {destinations = append(destinations, &resourceID)}@@ -2018,6 +2060,12 @@ func queryRetainedChunkAttempt[T any](s *Store, resourceType string, resourceIDslog.Warn().Err(err).Msg("Failed to scan batch metric row")continue}+ if resourceOrdinals {+ if resourceNumber < 0 || resourceNumber >= len(resourceIDs) {+ return nil, fmt.Errorf("invalid retained resource number %d for %d resources", resourceNumber, len(resourceIDs))+ }+ resourceID = resourceIDs[resourceNumber]+ }if !projection.valuesOnly {p.Min, p.Max = p.Value, p.Valueif low.Valid {@@ -2053,6 +2101,15 @@ func queryRetainedChunkAttempt[T any](s *Store, resourceType string, resourceIDsflushBucket()flushSeries()+ if resourceOrdinals {+ for _, metrics := range result {+ for _, points := range metrics {+ if !slices.IsSortedFunc(points, projection.compare) {+ slices.SortStableFunc(points, projection.compare)+ }+ }+ }+ }if hasDockerObservationContract(resourceType) {projectDockerObservations(result)}diff --git a/pkg/metrics/store_query_plan_test.go b/pkg/metrics/store_query_plan_test.goindex e9e7fafde2..b7b1a74f3b 100644--- a/pkg/metrics/store_query_plan_test.go+++ b/pkg/metrics/store_query_plan_test.go@@ -4,10 +4,13 @@ import ("database/sql""fmt""net/url"+ "reflect""strings""testing""time"+ pdb "github.com/rcourtman/pulse-go-rewrite/pkg/db"+_ "modernc.org/sqlite")@@ -550,6 +553,9 @@ func TestRetainedUnaggregatedQueryAvoidsMetricSort(t *testing.T) {if strings.Contains(strings.ToUpper(line), "TEMP B-TREE") {t.Fatalf("unaggregated read sorts despite time-ordered index: %s", plan)}+ if len(ids) > 1 && strings.Contains(line, "MERGE (UNION ALL)") {+ t.Fatalf("raw fleet read merges repeated identity strings across tiers: %s", plan)+ }}})}@@ -557,3 +563,109 @@ func TestRetainedUnaggregatedQueryAvoidsMetricSort(t *testing.T) {}}}++func TestRetainedResourceNumbersPreserveScopeAndOrderedSeries(t *testing.T) {+ for _, clustered := range []bool{false, true} {+ t.Run(fmt.Sprintf("clustered_%t", clustered), func(t *testing.T) {+ db := newPlanTestDB(t)+ if clustered {+ if _, err := db.Exec(`DROP TABLE metrics; CREATE TABLE metrics (` + metricsTableColumns + `) WITHOUT ROWID;+ CREATE INDEX idx_metrics_tier_time ON metrics(tier,timestamp)`); err != nil {+ t.Fatal(err)+ }+ }+ store := &Store{db: pdb.Wrap(db, "retained-resource-number-test")}+ store.clusteredMetrics.Store(clustered)+ start := time.Unix(1_800_003_600, 0)+ end := start.Add(3 * time.Hour)+ ids := make([]string, queryAllBatchChunkSize+3)+ tx, err := db.Begin()+ if err != nil {+ t.Fatal(err)+ }+ insert, err := tx.Prepare(`INSERT INTO metrics(resource_type,resource_id,metric_type,value,min_value,max_value,timestamp,tier) VALUES (?,?,?,?,?,?,?,?)`)+ if err != nil {+ t.Fatal(err)+ }+ for i := range ids {+ // Deliberately reverse lexical order, including IDs beyond the+ // chunk boundary. A resource number belongs only to its statement.+ ids[i] = fmt.Sprintf("guest-'-%04d", len(ids)-i)+ for _, metric := range []string{"cpu", "memory", "unrequested"} {+ for _, row := range []struct {+ offset int64+ tier Tier+ value float64+ low, high any+ }{+ {60, TierMinute, 10, float64(0), float64(20)},+ {65, TierRaw, 999, nil, nil}, // owned by the minute bucket+ {180, TierRaw, 30, nil, nil},+ {300, TierMinute, 50, nil, nil},+ {15, TierRaw, 5, nil, nil}, // emitted after minute, must sort+ {0, TierHourly, -100, nil, nil},+ {-1, TierRaw, -999, nil, nil},+ {3*3600 + 1, TierRaw, -999, nil, nil},+ } {+ if _, err := insert.Exec("vm", ids[i], metric, row.value+float64(i), row.low, row.high, start.Unix()+row.offset, string(row.tier)); err != nil {+ t.Fatal(err)+ }+ }+ }+ if _, err := insert.Exec("container", ids[i], "cpu", 9999, nil, nil, start.Unix()+60, string(TierMinute)); err != nil {+ t.Fatal(err)+ }+ }+ if _, err := insert.Exec("vm", "outside-scope", "cpu", 9999, nil, nil, start.Unix()+60, string(TierMinute)); err != nil {+ t.Fatal(err)+ }+ if err := insert.Close(); err != nil {+ t.Fatal(err)+ }+ if err := tx.Commit(); err != nil {+ t.Fatal(err)+ }++ check := func(requested, filters []string, step int64) {+ t.Helper()+ full, err := store.QueryMetricTypesBatch("vm", requested, filters, start, end, step)+ if err != nil {+ t.Fatal(err)+ }+ values, err := store.QueryMetricValuesBatch("vm", requested, filters, start, end, step)+ if err != nil {+ t.Fatal(err)+ }+ assertMetricValueProjection(t, full, values)+ seen := make(map[string]bool)+ for _, id := range requested {+ id = normalizeMetricIdentifier(id)+ if id == "" || seen[id] {+ continue+ }+ seen[id] = true+ one, err := store.QueryMetricTypesBatch("vm", []string{id}, filters, start, end, step)+ if err != nil {+ t.Fatal(err)+ }+ if !reflect.DeepEqual(full[id], one[id]) {+ t.Fatalf("scope/order changed for %q step=%d: batch=%+v single=%+v", id, step, full[id], one[id])+ }+ }+ if _, exists := full["outside-scope"]; exists {+ t.Fatal("unrequested resource returned")+ }+ if len(full) != len(seen)-1 { // one requested ID is deliberately absent+ t.Fatalf("resource count=%d requested distinct=%d", len(full), len(seen))+ }+ }+ for _, filters := range [][]string{nil, {"cpu"}, {"memory", "cpu", "absent"}} {+ for _, step := range []int64{0, 1, 60} {+ check([]string{ids[2], ids[0], ids[1], ids[2], "", "missing"}, filters, step)+ check([]string{ids[1], ids[2], ids[0], "missing"}, filters, step)+ }+ }+ check(append(append([]string{}, ids...), "missing", ids[0]), nil, 0)+ })+ }+}