Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Keep native disk History reachable through populated stores

pulse-triage[bot]716b419f888f
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 39f2b25906..8311a06f92 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -3575,6 +3575,16 @@ Agent` secondary handoff against the live setup wizard instead of relying
## Current State
+### Disk History supplementation does not change agent lifecycle
+
+The disk History handler can replace a shallow API-backed thermal series with
+monitor-selected native history even when stored I/O exists. This read does
+not enrol, refresh, remove or grant an agent; it changes no agent identity,
+helper profile, report completeness or missing-series handling. Ordinary
+agent-backed stored metrics remain preferred when coverage is sufficient.
+The authenticated native-route control covers the store/native boundary.
+
+
The partial-result path no longer discards every successful row when another guest fails. The helper remains in its private network on every platform; pmxcfs abstract IPC is still unavailable there on the reported PVE estate (#2511). That defect is not fixed by the partial protocol (#2513), and API monitoring remains the safe existing alternative. The earlier host-network proposal is not part of this implementation.
### VM guest execution admission (backup precaution)
diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md
index 12452c53b1..fdea524f6a 100644
--- a/docs/release-control/v6/internal/subsystems/api-contracts.md
+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md
@@ -20,6 +20,26 @@
## Purpose
+### Existing disk History keeps native thermal fallback reachable
+
+The authenticated disk History endpoint supplements a missing or shallow
+`smart_temp` series even when independent I/O metrics already exist in the
+persistent store. Single-metric and the drawer's all-metric requests use the
+same monitor-owned native/local selection. Only a broader series (or more
+actual samples at equal span) replaces the thermal series; unrelated stored
+values and min/max bounds stay unchanged. Samples stay within the issued
+window and use the existing aggregation and fallback-source wire contract.
+Complete local coverage makes no native history call. Store errors remain
+HTTP 500 rather than being concealed by native fallback.
+
+`TestTrueNASDiskHistoryAuthenticatedRoute` traverses the real authenticated TLS
+client, provider, tenant poller, monitor and router using declared appliance
+responses and ordinary store writes. It covers nil store, I/O-only store, a
+single thermal row, complete local coverage, store errors, token scope, tenant
+binding and revocation. It preserves original Celsius samples/timestamps and
+independent I/O, not synthetic temperature padding or CPU substitution. This
+is connected source proof, not an installed appliance or containing release.
+
### Organisation deletion retains data after incomplete monitoring shutdown
Authenticated organisation-owner deletion waits for tenant-loop exit and sealed
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 26197f3180..9f97b1d658 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -2351,6 +2351,12 @@ truthfulness, not native thaw, containing-release or workload acceptance.
### TrueNAS physical-disk History — issue #2519
+The actual all-metric drawer request also reaches this reader when stored I/O
+exists but thermal coverage does not. API single/all-metric supplementation
+keeps independent stored metrics, issued-window filtering, existing aggregate
+bounds and store-error behaviour; complete local coverage makes no native
+call. The connected authenticated route includes ordinary store-backed controls.
+
The existing disk drawer's `smart_temp` chart consumes the same tenant-scoped
native temperature-history provider as physical-disk sparklines when local
history is shallow. Sufficient local coverage and unrelated SMART/I/O metrics
diff --git a/docs/release-control/v6/internal/subsystems/registry.json b/docs/release-control/v6/internal/subsystems/registry.json
index cd00c396ca..9c73aa40bc 100644
--- a/docs/release-control/v6/internal/subsystems/registry.json
+++ b/docs/release-control/v6/internal/subsystems/registry.json
@@ -3312,6 +3312,7 @@
"internal/api/metadata_handlers_test.go",
"internal/api/patrol_autopilot_test.go",
"internal/api/runtime_inventory_sources_test.go",
+ "internal/api/truenas_disk_history_integration_test.go",
"pulse-enterprise:test/extensions_contract_test.go"
]
},
diff --git a/docs/release-control/v6/internal/subsystems/security-privacy.md b/docs/release-control/v6/internal/subsystems/security-privacy.md
index 57d46e2885..f20812fe34 100644
--- a/docs/release-control/v6/internal/subsystems/security-privacy.md
+++ b/docs/release-control/v6/internal/subsystems/security-privacy.md
@@ -966,6 +966,17 @@ tokens, and path-normalization variants.
## Current State
+### Native disk History stays inside authenticated reads
+
+Store-backed single/all-metric disk History supplementation occurs only after
+the existing monitoring-read, tenant binding, resource selection and licensed
+window checks. Native reads use the current tenant's monitored provider, not
+a caller-selected host or credential. Failed store queries stay errors; a
+revoked or wrong-scope token cannot initiate native reads. The connected
+TrueNAS History route tests assert those refusals and unchanged native read
+counts before and after revocation. No write or cross-tenant fallback is added.
+
+
This repair preserves the existing isolation boundary and introduces no host-IPC grant. Successful partial filesystem rows retain explicit completeness and degraded health; sanitized status exposes no raw provider errors. Native PVE recovery remains unproved. No old v6/v7 receipt is reinterpreted and no v8 qualification layer is introduced.
### Container diagnostics keeps copied material credential-free (1 October 2026)
diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md
index 035d5ed4ab..509fe0e6c4 100644
--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md
+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md
@@ -2954,6 +2954,17 @@ vdev layout is reported` in
## Current State
+### Stored disk I/O cannot suppress native thermal History
+
+The existing disk History response can supplement only its absent/shallow
+thermal series after a successful persistent-store read. Independent stored
+I/O values, timestamps and aggregate bounds stay intact; a closed/failed
+store returns its existing error rather than native success. No retention,
+pruning, schema, writer, recovery or database-internal change is involved.
+The connected native History route covers ordinary store writes, shallow
+readback, sufficient coverage and the error boundary.
+
+
The agent config-fetch audit suppression cache is bounded to 4,096
organisation/agent pairs, independently of persisted security audit history.
Expiry-index pruning forgets only suppression state, never audit rows or
diff --git a/internal/api/router.go b/internal/api/router.go
index 082251c472..9efaa70f5d 100644
--- a/internal/api/router.go
+++ b/internal/api/router.go
@@ -7143,6 +7143,37 @@ func (r *Router) handleMetricsHistory(w http.ResponseWriter, req *http.Request)
return
}
+ // A populated I/O series (or one recent temperature row) must not hide the
+ // native disk-temperature history. The monitor owns local/native selection;
+ // supplement only a better series, after successful store and access checks.
+ // Keep the issued window and ordinary aggregation contract, and never replace
+ // independent metrics or turn a failed store read into a successful fallback.
+ supplementDiskTemperature := func(current []metricstore.MetricPoint) []map[string]interface{} {
+ if !fallbackAllowed || runtimeResourceType != "disk" || mock.IsMockEnabled() {
+ return nil
+ }
+ candidate := monitor.GetDiskMetricsForChart(resourceID, "smart_temp", duration)
+ withinWindow := make([]monitoring.MetricPoint, 0, len(candidate))
+ for _, point := range candidate {
+ if !point.Timestamp.Before(start) && !point.Timestamp.After(end) {
+ withinWindow = append(withinWindow, point)
+ }
+ }
+ apiPoints := buildHistoryPoints(withinWindow, stepSecs)
+ if len(apiPoints) == 0 {
+ return nil
+ }
+ var currentSpan int64
+ if len(current) > 1 {
+ currentSpan = current[len(current)-1].Timestamp.UnixMilli() - current[0].Timestamp.UnixMilli()
+ }
+ candidateSpan := apiPoints[len(apiPoints)-1]["timestamp"].(int64) - apiPoints[0]["timestamp"].(int64)
+ if candidateSpan > currentSpan || (candidateSpan == currentSpan && len(apiPoints) > len(current)) {
+ return apiPoints
+ }
+ return nil
+ }
+
var response interface{}
if metricType != "" {
@@ -7182,6 +7213,21 @@ func (r *Router) handleMetricsHistory(w http.ResponseWriter, req *http.Request)
}
}
+ if response == nil && len(points) > 0 && metricType == "smart_temp" {
+ if apiPoints := supplementDiskTemperature(points); len(apiPoints) > 0 {
+ response = map[string]interface{}{
+ "resourceType": responseResourceType,
+ "resourceId": resourceID,
+ "metric": metricType,
+ "range": timeRange,
+ "start": start.UnixMilli(),
+ "end": end.UnixMilli(),
+ "points": apiPoints,
+ "source": historySourceMemory,
+ }
+ }
+ }
+
if response == nil && mock.IsMockEnabled() && runtimeResourceType == "disk" &&
(metricType == "smart_temp" || metricType == "disk" || metricType == "diskread" || metricType == "diskwrite") {
targetPoints := chartapi.TargetMockSeriesPoints(duration, historyMaxPoints)
@@ -7292,6 +7338,13 @@ func (r *Router) handleMetricsHistory(w http.ResponseWriter, req *http.Request)
apiData[metric] = apiPoints
}
+ if runtimeResourceType == "disk" {
+ if points := supplementDiskTemperature(metricsMap["smart_temp"]); len(points) > 0 {
+ apiData["smart_temp"] = points
+ source = historySourceMemory
+ }
+ }
+
// QueryAll can return a non-empty but incomplete mock metric map. Fill
// only the absent series from the deterministic chart fallback so one
// populated metric cannot suppress entire drawer groups.
diff --git a/internal/api/truenas_disk_history_integration_test.go b/internal/api/truenas_disk_history_integration_test.go
index 00d75fff24..8fe96473ab 100644
--- a/internal/api/truenas_disk_history_integration_test.go
+++ b/internal/api/truenas_disk_history_integration_test.go
@@ -17,6 +17,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
"github.com/rcourtman/pulse-go-rewrite/internal/truenas"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/metrics"
)
// Inventory is a declared fixture; History traverses the real authenticated
@@ -156,8 +157,8 @@ func TestTrueNASDiskHistoryAuthenticatedRoute(t *testing.T) {
mtm := monitoring.NewMultiTenantMonitor(cfg, nil, nil)
setUnexportedField(t, mtm, "monitors", map[string]*monitoring.Monitor{"org-a": monitor, "org-b": {}})
router.mtMonitor = mtm
- request := func(raw, org string) *httptest.ResponseRecorder {
- req := httptest.NewRequest(http.MethodGet, "/api/metrics-store/history?resourceType=disk&resourceId=disk-serial&metric=smart_temp&range=1h", nil)
+ request := func(raw, org, metric string) *httptest.ResponseRecorder {
+ req := httptest.NewRequest(http.MethodGet, "/api/metrics-store/history?resourceType=disk&resourceId=disk-serial&metric="+metric+"&range=1h", nil)
req.Header.Set("X-Pulse-Org-ID", org)
if raw != "" {
req.Header.Set("X-API-Token", raw)
@@ -170,12 +171,12 @@ func TestTrueNASDiskHistoryAuthenticatedRoute(t *testing.T) {
raw, org string
want int
}{{"", "org-a", http.StatusUnauthorized}, {wrongScope, "org-a", http.StatusForbidden}, {token, "org-b", http.StatusForbidden}} {
- rec := request(test.raw, test.org)
+ rec := request(test.raw, test.org, "smart_temp")
if rec.Code != test.want || reads.Load() != 0 {
t.Fatalf("auth/scope/tenant refusal lost: status %d want %d native reads %d body %s", rec.Code, test.want, reads.Load(), rec.Body.String())
}
}
- rec := request(token, "org-a")
+ rec := request(token, "org-a", "smart_temp")
if rec.Code != http.StatusOK {
t.Fatalf("authenticated History status %d: %s", rec.Code, rec.Body.String())
}
@@ -192,12 +193,88 @@ func TestTrueNASDiskHistoryAuthenticatedRoute(t *testing.T) {
t.Fatalf("native timestamp/Celsius sample %d changed: %+v", i, got.Points[i])
}
}
+ type point struct {
+ Timestamp int64 `json:"timestamp"`
+ Value float64 `json:"value"`
+ Min float64 `json:"min"`
+ Max float64 `json:"max"`
+ }
+ readAll := func() map[string][]point {
+ t.Helper()
+ rec := request(token, "org-a", "")
+ var all struct {
+ ResourceType string `json:"resourceType"`
+ ResourceID string `json:"resourceId"`
+ Metrics map[string][]point `json:"metrics"`
+ }
+ if rec.Code != http.StatusOK || json.Unmarshal(rec.Body.Bytes(), &all) != nil || all.ResourceType != "disk" || all.ResourceID != "disk-serial" {
+ t.Fatalf("all-metric drawer contract lost: status %d body %s", rec.Code, rec.Body.String())
+ }
+ return all.Metrics
+ }
+ assertNative := func(points []point, wantReads int32) {
+ t.Helper()
+ if len(points) != 3 || reads.Load() != wantReads {
+ t.Fatalf("all-metric drawer suppressed native History: %+v, reads %d want %d", points, reads.Load(), wantReads)
+ }
+ for i, value := range []float64{31.25, 33.5, 32.75} {
+ wantTime := time.Unix(nativeEnd.Load()-3540+int64(i)*1740, 0).UnixMilli()
+ if points[i].Value != value || points[i].Min != value || points[i].Max != value || points[i].Timestamp != wantTime {
+ t.Fatalf("native all-metric timestamp/Celsius/bounds changed: %+v", points[i])
+ }
+ }
+ }
+ assertNative(readAll()["smart_temp"], 4)
+
+ // The real drawer asks for all metrics. An unrelated I/O row, then a
+ // single shallow temperature row, must not bypass the native reader.
+ store, err := metrics.NewStore(metrics.DefaultConfig(t.TempDir()))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = store.Close() })
+ setUnexportedField(t, monitor, "metricsStore", store)
+ ioTime := time.Now().UTC().Add(-20 * time.Second)
+ store.WriteBatchSync([]metrics.WriteMetric{{ResourceType: "disk", ResourceID: "disk-serial", MetricType: "diskread", Value: 1024, Timestamp: ioTime}})
+ all := readAll()
+ assertNative(all["smart_temp"], 6)
+ if len(all["diskread"]) != 1 || all["diskread"][0].Value != 1024 || all["diskread"][0].Min != 1024 || all["diskread"][0].Max != 1024 || all["diskread"][0].Timestamp != ioTime.UnixMilli() {
+ t.Fatalf("native supplement changed independent stored I/O: %+v", all["diskread"])
+ }
+ store.WriteBatchSync([]metrics.WriteMetric{{ResourceType: "disk", ResourceID: "disk-serial", MetricType: "smart_temp", Value: 44, Timestamp: ioTime}})
+ assertNative(readAll()["smart_temp"], 8)
+ rec = request(token, "org-a", "smart_temp")
+ if rec.Code != http.StatusOK || json.Unmarshal(rec.Body.Bytes(), &got) != nil || len(got.Points) != 3 || reads.Load() != 10 {
+ t.Fatalf("single-metric sparse store hid native History: status %d points %+v reads %d", rec.Code, got.Points, reads.Load())
+ }
+ for i, value := range []float64{31.25, 33.5, 32.75} {
+ if got.Points[i].Value != value {
+ t.Fatalf("sparse stored/native samples were spliced: %+v", got.Points)
+ }
+ }
+ // Adequate local coverage keeps the native session quiet and its original
+ // independent values intact. Ordinary store errors still fail closed.
+ now := time.Now().UTC()
+ for _, offset := range []time.Duration{-55 * time.Minute, -30 * time.Minute, -time.Minute} {
+ store.WriteBatchSync([]metrics.WriteMetric{{ResourceType: "disk", ResourceID: "disk-serial", MetricType: "smart_temp", Value: 35, Timestamp: now.Add(offset)}})
+ }
+ all = readAll()
+ if len(all["smart_temp"]) != 4 || all["smart_temp"][0].Value != 35 || reads.Load() != 10 {
+ t.Fatalf("complete store needlessly invoked/replaced native history: %+v, reads %d", all["smart_temp"], reads.Load())
+ }
+ if err := store.Close(); err != nil {
+ t.Fatal(err)
+ }
+ rec = request(token, "org-a", "")
+ if rec.Code != http.StatusInternalServerError || reads.Load() != 10 {
+ t.Fatalf("failed store read silently became native success: status %d reads %d", rec.Code, reads.Load())
+ }
if removed := cfg.RemoveAPIToken(readToken.ID); removed == nil {
t.Fatal("fixture token missing")
}
cfg.SortAPITokens()
- rec = request(token, "org-a")
- if rec.Code != http.StatusUnauthorized || reads.Load() != 2 {
+ rec = request(token, "org-a", "smart_temp")
+ if rec.Code != http.StatusUnauthorized || reads.Load() != 10 {
t.Fatalf("revoked token still reached History: status %d reads %d", rec.Code, reads.Load())
}
}