Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep native disk History reachable through populated stores
pulse-triage[bot]716b419f888f
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 39f2b25906..8311a06f92 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -3575,6 +3575,16 @@ Agent` secondary handoff against the live setup wizard instead of relying## Current State+### Disk History supplementation does not change agent lifecycle++The disk History handler can replace a shallow API-backed thermal series with+monitor-selected native history even when stored I/O exists. This read does+not enrol, refresh, remove or grant an agent; it changes no agent identity,+helper profile, report completeness or missing-series handling. Ordinary+agent-backed stored metrics remain preferred when coverage is sufficient.+The authenticated native-route control covers the store/native boundary.++The partial-result path no longer discards every successful row when another guest fails. The helper remains in its private network on every platform; pmxcfs abstract IPC is still unavailable there on the reported PVE estate (#2511). That defect is not fixed by the partial protocol (#2513), and API monitoring remains the safe existing alternative. The earlier host-network proposal is not part of this implementation.### VM guest execution admission (backup precaution)diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.mdindex 12452c53b1..fdea524f6a 100644--- a/docs/release-control/v6/internal/subsystems/api-contracts.md+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md@@ -20,6 +20,26 @@## Purpose+### Existing disk History keeps native thermal fallback reachable++The authenticated disk History endpoint supplements a missing or shallow+`smart_temp` series even when independent I/O metrics already exist in the+persistent store. Single-metric and the drawer's all-metric requests use the+same monitor-owned native/local selection. Only a broader series (or more+actual samples at equal span) replaces the thermal series; unrelated stored+values and min/max bounds stay unchanged. Samples stay within the issued+window and use the existing aggregation and fallback-source wire contract.+Complete local coverage makes no native history call. Store errors remain+HTTP 500 rather than being concealed by native fallback.++`TestTrueNASDiskHistoryAuthenticatedRoute` traverses the real authenticated TLS+client, provider, tenant poller, monitor and router using declared appliance+responses and ordinary store writes. It covers nil store, I/O-only store, a+single thermal row, complete local coverage, store errors, token scope, tenant+binding and revocation. It preserves original Celsius samples/timestamps and+independent I/O, not synthetic temperature padding or CPU substitution. This+is connected source proof, not an installed appliance or containing release.+### Organisation deletion retains data after incomplete monitoring shutdownAuthenticated organisation-owner deletion waits for tenant-loop exit and sealeddiff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex 26197f3180..9f97b1d658 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -2351,6 +2351,12 @@ truthfulness, not native thaw, containing-release or workload acceptance.### TrueNAS physical-disk History — issue #2519+The actual all-metric drawer request also reaches this reader when stored I/O+exists but thermal coverage does not. API single/all-metric supplementation+keeps independent stored metrics, issued-window filtering, existing aggregate+bounds and store-error behaviour; complete local coverage makes no native+call. The connected authenticated route includes ordinary store-backed controls.+The existing disk drawer's `smart_temp` chart consumes the same tenant-scopednative temperature-history provider as physical-disk sparklines when localhistory is shallow. Sufficient local coverage and unrelated SMART/I/O metricsdiff --git a/docs/release-control/v6/internal/subsystems/registry.json b/docs/release-control/v6/internal/subsystems/registry.jsonindex cd00c396ca..9c73aa40bc 100644--- a/docs/release-control/v6/internal/subsystems/registry.json+++ b/docs/release-control/v6/internal/subsystems/registry.json@@ -3312,6 +3312,7 @@"internal/api/metadata_handlers_test.go","internal/api/patrol_autopilot_test.go","internal/api/runtime_inventory_sources_test.go",+ "internal/api/truenas_disk_history_integration_test.go","pulse-enterprise:test/extensions_contract_test.go"]},diff --git a/docs/release-control/v6/internal/subsystems/security-privacy.md b/docs/release-control/v6/internal/subsystems/security-privacy.mdindex 57d46e2885..f20812fe34 100644--- a/docs/release-control/v6/internal/subsystems/security-privacy.md+++ b/docs/release-control/v6/internal/subsystems/security-privacy.md@@ -966,6 +966,17 @@ tokens, and path-normalization variants.## Current State+### Native disk History stays inside authenticated reads++Store-backed single/all-metric disk History supplementation occurs only after+the existing monitoring-read, tenant binding, resource selection and licensed+window checks. Native reads use the current tenant's monitored provider, not+a caller-selected host or credential. Failed store queries stay errors; a+revoked or wrong-scope token cannot initiate native reads. The connected+TrueNAS History route tests assert those refusals and unchanged native read+counts before and after revocation. No write or cross-tenant fallback is added.++This repair preserves the existing isolation boundary and introduces no host-IPC grant. Successful partial filesystem rows retain explicit completeness and degraded health; sanitized status exposes no raw provider errors. Native PVE recovery remains unproved. No old v6/v7 receipt is reinterpreted and no v8 qualification layer is introduced.### Container diagnostics keeps copied material credential-free (1 October 2026)diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex 035d5ed4ab..509fe0e6c4 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -2954,6 +2954,17 @@ vdev layout is reported` in## Current State+### Stored disk I/O cannot suppress native thermal History++The existing disk History response can supplement only its absent/shallow+thermal series after a successful persistent-store read. Independent stored+I/O values, timestamps and aggregate bounds stay intact; a closed/failed+store returns its existing error rather than native success. No retention,+pruning, schema, writer, recovery or database-internal change is involved.+The connected native History route covers ordinary store writes, shallow+readback, sufficient coverage and the error boundary.++The agent config-fetch audit suppression cache is bounded to 4,096organisation/agent pairs, independently of persisted security audit history.Expiry-index pruning forgets only suppression state, never audit rows ordiff --git a/internal/api/router.go b/internal/api/router.goindex 082251c472..9efaa70f5d 100644--- a/internal/api/router.go+++ b/internal/api/router.go@@ -7143,6 +7143,37 @@ func (r *Router) handleMetricsHistory(w http.ResponseWriter, req *http.Request)return}+ // A populated I/O series (or one recent temperature row) must not hide the+ // native disk-temperature history. The monitor owns local/native selection;+ // supplement only a better series, after successful store and access checks.+ // Keep the issued window and ordinary aggregation contract, and never replace+ // independent metrics or turn a failed store read into a successful fallback.+ supplementDiskTemperature := func(current []metricstore.MetricPoint) []map[string]interface{} {+ if !fallbackAllowed || runtimeResourceType != "disk" || mock.IsMockEnabled() {+ return nil+ }+ candidate := monitor.GetDiskMetricsForChart(resourceID, "smart_temp", duration)+ withinWindow := make([]monitoring.MetricPoint, 0, len(candidate))+ for _, point := range candidate {+ if !point.Timestamp.Before(start) && !point.Timestamp.After(end) {+ withinWindow = append(withinWindow, point)+ }+ }+ apiPoints := buildHistoryPoints(withinWindow, stepSecs)+ if len(apiPoints) == 0 {+ return nil+ }+ var currentSpan int64+ if len(current) > 1 {+ currentSpan = current[len(current)-1].Timestamp.UnixMilli() - current[0].Timestamp.UnixMilli()+ }+ candidateSpan := apiPoints[len(apiPoints)-1]["timestamp"].(int64) - apiPoints[0]["timestamp"].(int64)+ if candidateSpan > currentSpan || (candidateSpan == currentSpan && len(apiPoints) > len(current)) {+ return apiPoints+ }+ return nil+ }+var response interface{}if metricType != "" {@@ -7182,6 +7213,21 @@ func (r *Router) handleMetricsHistory(w http.ResponseWriter, req *http.Request)}}+ if response == nil && len(points) > 0 && metricType == "smart_temp" {+ if apiPoints := supplementDiskTemperature(points); len(apiPoints) > 0 {+ response = map[string]interface{}{+ "resourceType": responseResourceType,+ "resourceId": resourceID,+ "metric": metricType,+ "range": timeRange,+ "start": start.UnixMilli(),+ "end": end.UnixMilli(),+ "points": apiPoints,+ "source": historySourceMemory,+ }+ }+ }+if response == nil && mock.IsMockEnabled() && runtimeResourceType == "disk" &&(metricType == "smart_temp" || metricType == "disk" || metricType == "diskread" || metricType == "diskwrite") {targetPoints := chartapi.TargetMockSeriesPoints(duration, historyMaxPoints)@@ -7292,6 +7338,13 @@ func (r *Router) handleMetricsHistory(w http.ResponseWriter, req *http.Request)apiData[metric] = apiPoints}+ if runtimeResourceType == "disk" {+ if points := supplementDiskTemperature(metricsMap["smart_temp"]); len(points) > 0 {+ apiData["smart_temp"] = points+ source = historySourceMemory+ }+ }+// QueryAll can return a non-empty but incomplete mock metric map. Fill// only the absent series from the deterministic chart fallback so one// populated metric cannot suppress entire drawer groups.diff --git a/internal/api/truenas_disk_history_integration_test.go b/internal/api/truenas_disk_history_integration_test.goindex 00d75fff24..8fe96473ab 100644--- a/internal/api/truenas_disk_history_integration_test.go+++ b/internal/api/truenas_disk_history_integration_test.go@@ -17,6 +17,7 @@ import ("github.com/rcourtman/pulse-go-rewrite/internal/monitoring""github.com/rcourtman/pulse-go-rewrite/internal/truenas""github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"+ "github.com/rcourtman/pulse-go-rewrite/pkg/metrics")// Inventory is a declared fixture; History traverses the real authenticated@@ -156,8 +157,8 @@ func TestTrueNASDiskHistoryAuthenticatedRoute(t *testing.T) {mtm := monitoring.NewMultiTenantMonitor(cfg, nil, nil)setUnexportedField(t, mtm, "monitors", map[string]*monitoring.Monitor{"org-a": monitor, "org-b": {}})router.mtMonitor = mtm- request := func(raw, org string) *httptest.ResponseRecorder {- req := httptest.NewRequest(http.MethodGet, "/api/metrics-store/history?resourceType=disk&resourceId=disk-serial&metric=smart_temp&range=1h", nil)+ request := func(raw, org, metric string) *httptest.ResponseRecorder {+ req := httptest.NewRequest(http.MethodGet, "/api/metrics-store/history?resourceType=disk&resourceId=disk-serial&metric="+metric+"&range=1h", nil)req.Header.Set("X-Pulse-Org-ID", org)if raw != "" {req.Header.Set("X-API-Token", raw)@@ -170,12 +171,12 @@ func TestTrueNASDiskHistoryAuthenticatedRoute(t *testing.T) {raw, org stringwant int}{{"", "org-a", http.StatusUnauthorized}, {wrongScope, "org-a", http.StatusForbidden}, {token, "org-b", http.StatusForbidden}} {- rec := request(test.raw, test.org)+ rec := request(test.raw, test.org, "smart_temp")if rec.Code != test.want || reads.Load() != 0 {t.Fatalf("auth/scope/tenant refusal lost: status %d want %d native reads %d body %s", rec.Code, test.want, reads.Load(), rec.Body.String())}}- rec := request(token, "org-a")+ rec := request(token, "org-a", "smart_temp")if rec.Code != http.StatusOK {t.Fatalf("authenticated History status %d: %s", rec.Code, rec.Body.String())}@@ -192,12 +193,88 @@ func TestTrueNASDiskHistoryAuthenticatedRoute(t *testing.T) {t.Fatalf("native timestamp/Celsius sample %d changed: %+v", i, got.Points[i])}}+ type point struct {+ Timestamp int64 `json:"timestamp"`+ Value float64 `json:"value"`+ Min float64 `json:"min"`+ Max float64 `json:"max"`+ }+ readAll := func() map[string][]point {+ t.Helper()+ rec := request(token, "org-a", "")+ var all struct {+ ResourceType string `json:"resourceType"`+ ResourceID string `json:"resourceId"`+ Metrics map[string][]point `json:"metrics"`+ }+ if rec.Code != http.StatusOK || json.Unmarshal(rec.Body.Bytes(), &all) != nil || all.ResourceType != "disk" || all.ResourceID != "disk-serial" {+ t.Fatalf("all-metric drawer contract lost: status %d body %s", rec.Code, rec.Body.String())+ }+ return all.Metrics+ }+ assertNative := func(points []point, wantReads int32) {+ t.Helper()+ if len(points) != 3 || reads.Load() != wantReads {+ t.Fatalf("all-metric drawer suppressed native History: %+v, reads %d want %d", points, reads.Load(), wantReads)+ }+ for i, value := range []float64{31.25, 33.5, 32.75} {+ wantTime := time.Unix(nativeEnd.Load()-3540+int64(i)*1740, 0).UnixMilli()+ if points[i].Value != value || points[i].Min != value || points[i].Max != value || points[i].Timestamp != wantTime {+ t.Fatalf("native all-metric timestamp/Celsius/bounds changed: %+v", points[i])+ }+ }+ }+ assertNative(readAll()["smart_temp"], 4)++ // The real drawer asks for all metrics. An unrelated I/O row, then a+ // single shallow temperature row, must not bypass the native reader.+ store, err := metrics.NewStore(metrics.DefaultConfig(t.TempDir()))+ if err != nil {+ t.Fatal(err)+ }+ t.Cleanup(func() { _ = store.Close() })+ setUnexportedField(t, monitor, "metricsStore", store)+ ioTime := time.Now().UTC().Add(-20 * time.Second)+ store.WriteBatchSync([]metrics.WriteMetric{{ResourceType: "disk", ResourceID: "disk-serial", MetricType: "diskread", Value: 1024, Timestamp: ioTime}})+ all := readAll()+ assertNative(all["smart_temp"], 6)+ if len(all["diskread"]) != 1 || all["diskread"][0].Value != 1024 || all["diskread"][0].Min != 1024 || all["diskread"][0].Max != 1024 || all["diskread"][0].Timestamp != ioTime.UnixMilli() {+ t.Fatalf("native supplement changed independent stored I/O: %+v", all["diskread"])+ }+ store.WriteBatchSync([]metrics.WriteMetric{{ResourceType: "disk", ResourceID: "disk-serial", MetricType: "smart_temp", Value: 44, Timestamp: ioTime}})+ assertNative(readAll()["smart_temp"], 8)+ rec = request(token, "org-a", "smart_temp")+ if rec.Code != http.StatusOK || json.Unmarshal(rec.Body.Bytes(), &got) != nil || len(got.Points) != 3 || reads.Load() != 10 {+ t.Fatalf("single-metric sparse store hid native History: status %d points %+v reads %d", rec.Code, got.Points, reads.Load())+ }+ for i, value := range []float64{31.25, 33.5, 32.75} {+ if got.Points[i].Value != value {+ t.Fatalf("sparse stored/native samples were spliced: %+v", got.Points)+ }+ }+ // Adequate local coverage keeps the native session quiet and its original+ // independent values intact. Ordinary store errors still fail closed.+ now := time.Now().UTC()+ for _, offset := range []time.Duration{-55 * time.Minute, -30 * time.Minute, -time.Minute} {+ store.WriteBatchSync([]metrics.WriteMetric{{ResourceType: "disk", ResourceID: "disk-serial", MetricType: "smart_temp", Value: 35, Timestamp: now.Add(offset)}})+ }+ all = readAll()+ if len(all["smart_temp"]) != 4 || all["smart_temp"][0].Value != 35 || reads.Load() != 10 {+ t.Fatalf("complete store needlessly invoked/replaced native history: %+v, reads %d", all["smart_temp"], reads.Load())+ }+ if err := store.Close(); err != nil {+ t.Fatal(err)+ }+ rec = request(token, "org-a", "")+ if rec.Code != http.StatusInternalServerError || reads.Load() != 10 {+ t.Fatalf("failed store read silently became native success: status %d reads %d", rec.Code, reads.Load())+ }if removed := cfg.RemoveAPIToken(readToken.ID); removed == nil {t.Fatal("fixture token missing")}cfg.SortAPITokens()- rec = request(token, "org-a")- if rec.Code != http.StatusUnauthorized || reads.Load() != 2 {+ rec = request(token, "org-a", "smart_temp")+ if rec.Code != http.StatusUnauthorized || reads.Load() != 10 {t.Fatalf("revoked token still reached History: status %d reads %d", rec.Code, reads.Load())}}