Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Retain CORE physical-disk throughput through chart history

pulse-triage[bot]91d6c26323c5
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 1cb1db1d81..4529058761 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -15,6 +15,16 @@
## Purpose
+### API-backed disk rates preserve native-writer authority
+
+The physical-disk sync excludes Agent/Proxmox native writers and mock-owned
+records before writing TrueNAS per-device rates. A rate is not a kernel counter,
+agent report or continuity signal. Existing target resolution stays authoritative.
+The physical-disk roundtrip pipeline covers the API rate route and its adjacent
+native-writer regressions remain required. No enrolment, removal, lease, host
+identity or installed state transition changes.
+
+
A member test announces and reveals its result beside that exact member rather
than leaving feedback in the off-screen global footer. The normal primary
Test connection result stays in the footer; both remain endpoint-qualified
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 56dd5ea027..a69893d171 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -17,6 +17,30 @@
## Purpose
+### CORE per-disk throughput keeps its native meaning
+
+The existing reporting batch retains each selected disk's read/write rate even
+when a missing member prevents a complete host sum. Only an exact inventory
+device name within the connection matches; serials, pools, partitions, namesakes
+and unscoped replies never substitute. Native values remain bytes/s, not counters
+or busy percentages. Live age filtering still applies. Read/write presence and
+sample times remain independent; zero is observed and null is absent.
+The canonical physical-disk writer preserves native instants, rejects invalid,
+future or timeless readings, deduplicates rebuilds and keeps existing mock,
+native-writer and metric-target gates. Rehydrated scalars cannot mint samples.
+
+`TestCOREPerDiskRatesKeepPartialNativeSamples` and
+`TestCOREPerDiskRatesRejectInvalidAndPreserveScope` in `core_reporting_test.go`
+cover partial/duplicate/unscoped/stale/zero/null/identity and clone boundaries.
+`TestCOREPhysicalDiskReportingPipeline` in `physical_disk_roundtrip_test.go`
+crosses pinned TLS REST, snapshot, poller, registry, resolver, writer and local/
+persistent drawer reads for both boot names and complete/partial/idle cycles.
+`TestTrueNASDiskRateWriterAdmission` rejects invalid observations. Desktop and
+narrow browser proof must use actual produced disk History in the existing
+drawer. These controls do not establish the reported boot-only host cause,
+native missing ARC, operation/busy counters or original appliance recovery.
+
+
### Explicit Proxmox endpoint trust survives save and reload
A managed primary fingerprint edit reconciles only that exact HTTP authority
diff --git a/docs/release-control/v6/internal/subsystems/registry.json b/docs/release-control/v6/internal/subsystems/registry.json
index 3a2f593f69..51ae1e9590 100644
--- a/docs/release-control/v6/internal/subsystems/registry.json
+++ b/docs/release-control/v6/internal/subsystems/registry.json
@@ -6511,6 +6511,7 @@
"match_prefixes": [],
"match_files": [
"internal/truenas/client.go",
+ "internal/truenas/disk_rates.go",
"internal/truenas/core_reporting.go",
"internal/truenas/disk_health.go",
"internal/truenas/fixtures.go",
diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md
index 8cf254ff01..3a3e057335 100644
--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md
+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md
@@ -21,6 +21,17 @@
## Purpose
+### CORE disk throughput is not protection or counter evidence
+
+Native per-disk read/write rates reach existing disk chart History under the
+canonical disk target and original sample instant. They never populate cumulative
+`PhysicalDisk.IO`, operation totals, busy percentage or backup/protection evidence.
+A partial catalogue preserves individual samples without implying complete host
+I/O. The physical-disk roundtrip pipeline checks both boot names and persisted
+readback. Device/pool identity and recovery operations remain unchanged; browser
+proof concerns the consuming drawer, not original appliance recovery.
+
+
### Snapshot subjects retain their observed guest kind
A Proxmox snapshot's explicit VM/LXC kind takes precedence over best-effort
diff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.md
index 176d95bdb5..4c83a7b30f 100644
--- a/docs/release-control/v6/internal/subsystems/unified-resources.md
+++ b/docs/release-control/v6/internal/subsystems/unified-resources.md
@@ -23,6 +23,16 @@ and sort the complete canonical change table while startup and ingestion wait.
## Purpose
+### Native rate instants survive canonical cloning
+
+`MetricValue.ObservedAt` is an optional internal-only sample instant copied with
+the selected scalar, never part of public JSON. TrueNAS CORE per-disk rate writers
+use it instead of registry rebuild time. Rehydrated scalars without it are not
+new observations. `TestCloneNativeRateObservation` checks independent cloning,
+zero preservation and unchanged wire shape; the physical-disk pipeline verifies
+resolved targets. Identity, source precedence and cumulative counters do not change.
+
+
### Coverage evidence expansion preserves workload ownership
An uncertain backup date remains inspectable beside the newest dated evidence
diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.go
index b7fe3f44dd..38dc7bac80 100644
--- a/internal/monitoring/monitor.go
+++ b/internal/monitoring/monitor.go
@@ -6666,6 +6666,7 @@ func (m *Monitor) syncUnifiedPhysicalDiskMetrics(store ResourceStoreInterface) {
// re-issuing the same snapshot maps to one sample instead of a fresh
// wall-clock row (#1966).
m.writeSMARTMetrics(disk, unifiedResourceObservedAt(resource, now))
+ m.writeTrueNASDiskRates(resource, targetID, now)
}
}
diff --git a/internal/monitoring/physical_disk_roundtrip_test.go b/internal/monitoring/physical_disk_roundtrip_test.go
index 285dcd3eab..4ef6290794 100644
--- a/internal/monitoring/physical_disk_roundtrip_test.go
+++ b/internal/monitoring/physical_disk_roundtrip_test.go
@@ -2,14 +2,10 @@ package monitoring
import (
"context"
+ "crypto/sha256"
"encoding/json"
"errors"
- "slices"
- "strconv"
- "sync"
- "testing"
- "time"
-
+ "fmt"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
@@ -19,6 +15,16 @@ import (
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
"github.com/rcourtman/pulse-go-rewrite/pkg/metrics"
"github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"
+ "math"
+ "net/http"
+ "net/http/httptest"
+ "slices"
+ "strconv"
+ "strings"
+ "sync"
+ "sync/atomic"
+ "testing"
+ "time"
)
// Monitor construction restores notification destinations before admitting
@@ -1966,3 +1972,225 @@ func TestMergeHostAgentSMARTIntoDisksKeepsSSHNodeSensorReadingOfSplitDisk(t *tes
t.Fatalf("split disk with a reporting agent kept the agent-supplied reading: %+v / %+v", live, live.Collection)
}
}
+
+// Registered end-to-end source regression: production pinned REST -> snapshot
+// -> poller -> canonical registry/target -> writer -> local/persistent drawer
+// readback. Only provider HTTP responses are synthetic; there is no appliance
+// mutation, credentials or simulated cumulative kernel counter.
+func TestCOREPhysicalDiskReportingPipeline(t *testing.T) {
+ previous := truenas.IsFeatureEnabled()
+ truenas.SetFeatureEnabled(true)
+ t.Cleanup(func() { truenas.SetFeatureEnabled(previous) })
+ for _, boot := range []string{"boot-pool", "freenas-boot"} {
+ t.Run(boot, func(t *testing.T) {
+ var cycle atomic.Int64
+ sampleEnd := time.Now().UTC().Add(-2 * time.Minute).Truncate(time.Second).Unix()
+ server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.Header().Set("Content-Type", "application/json")
+ switch r.URL.Path {
+ case "/api/v2.0/system/info":
+ fmt.Fprint(w, `{"hostname":"namesake-core","version":"TrueNAS-13.0-U6.1","cores":8,"physmem":34359738368}`)
+ case "/api/v2.0/pool":
+ fmt.Fprintf(w, `[{"id":1,"name":%q,"status":"ONLINE","size":1000,"allocated":400,"free":600},{"id":2,"name":"data-pool","status":"ONLINE","size":100000,"allocated":60000,"free":40000}]`, boot)
+ case "/api/v2.0/disk":
+ fmt.Fprintf(w, `[{"identifier":"{disk-a}","name":"disk-a","size":1000,"pool":%q,"model":"Synthetic boot disk"},{"identifier":"{disk-b}","name":"disk-b","size":100000,"pool":"data-pool","model":"Synthetic data disk"},{"identifier":"{disk-c}","name":"disk-c","size":100000,"pool":"data-pool","model":"Synthetic data disk"}]`, boot)
+ case "/api/v2.0/disk/temperatures":
+ fmt.Fprint(w, `{"disk-a":30,"disk-b":31,"disk-c":32}`)
+ case "/api/v2.0/pool/dataset", "/api/v2.0/alert/list":
+ fmt.Fprint(w, `[]`)
+ case "/api/v2.0/reporting/graphs":
+ fmt.Fprint(w, `[{"name":"disk","identifiers":["disk-a","disk-b","disk-c"]}]`)
+ case "/api/v2.0/reporting/get_data":
+ var request struct {
+ Graphs []map[string]any `json:"graphs"`
+ Query map[string]any `json:"reporting_query"`
+ }
+ if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
+ t.Error(err)
+ w.WriteHeader(400)
+ return
+ }
+ end := sampleEnd + cycle.Load()*10
+ var replies []map[string]any
+ for _, g := range request.Graphs {
+ name, _ := g["name"].(string)
+ id, _ := g["identifier"].(string)
+ reply := map[string]any{"name": name, "identifier": id, "start": end - 10, "end": end, "step": 10}
+ switch name {
+ case "memory":
+ reply["legend"] = []string{"memory-free_value"}
+ reply["data"] = [][]any{{float64(2 << 30)}, {float64(2 << 30)}}
+ case "arcsize":
+ reply["legend"] = []string{"arcsize_value"}
+ reply["data"] = [][]any{{float64(20 << 30)}, {float64(20 << 30)}}
+ case "disk":
+ if cycle.Load() == 1 && id == "disk-c" {
+ continue
+ } // selected but absent, not idle
+ rates := map[string][]any{"disk-a": {100., 200.}, "disk-b": {1000., 2000.}, "disk-c": {10000., 20000.}}
+ rate, ok := rates[id]
+ if !ok {
+ t.Errorf("unexpected disk query %q", id)
+ w.WriteHeader(400)
+ return
+ }
+ if cycle.Load() == 2 {
+ rate = []any{0., nil}
+ } // true idle reads, missing writes
+ reply["legend"] = []string{"disk_octets_read", "disk_octets_write"}
+ reply["data"] = [][]any{rate, rate}
+ default:
+ continue
+ }
+ replies = append(replies, reply)
+ }
+ _ = json.NewEncoder(w).Encode(replies)
+ default:
+ http.NotFound(w, r)
+ }
+ }))
+ defer server.Close()
+ fingerprint := fmt.Sprintf("%x", sha256.Sum256(server.Certificate().Raw))
+ client, err := truenas.NewClient(truenas.ClientConfig{Host: server.URL, APIKey: "synthetic", Fingerprint: fingerprint, InsecureSkipVerify: true})
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer client.Close()
+ provider := truenas.NewLiveProviderForConnection(&truenas.APIFetcher{Client: client}, "core-a")
+ instance := config.TrueNASInstance{ID: "core-a", Host: server.URL, Enabled: true}
+ poller := NewTrueNASPoller(nil, 0, nil)
+ poller.providersByOrg["default"] = map[string]*truenas.Provider{instance.ID: provider}
+ poller.configsByOrg["default"] = map[string]config.TrueNASInstance{instance.ID: instance}
+ persistent, err := metrics.NewStore(metrics.DefaultConfig(t.TempDir()))
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer func() { _ = persistent.Close() }()
+ resourceStore := unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(unifiedresources.NewMemoryStore()))
+ monitor := &Monitor{resourceStore: resourceStore, metricsHistory: NewMetricsHistory(1024, 24*time.Hour), metricsStore: persistent}
+ for c := int64(0); c < 3; c++ {
+ cycle.Store(c)
+ poller.ensureConnectionRuntimeStatusLocked("default", instance.ID).nextPollAt = time.Now().Add(-time.Second)
+ poller.pollAll(context.Background())
+ records := poller.GetCurrentRecordsForOrg("default")
+ if len(records) == 0 {
+ t.Fatal("production poll lost inventory")
+ }
+ resourceStore.PopulateSnapshotAndSupplemental(models.StateSnapshot{}, map[unifiedresources.DataSource][]unifiedresources.IngestRecord{unifiedresources.SourceTrueNAS: records})
+ monitor.syncUnifiedPhysicalDiskMetrics(resourceStore)
+ monitor.syncUnifiedPhysicalDiskMetrics(resourceStore)
+ diskCount := 0
+ for _, resource := range resourceStore.GetAll() {
+ if resource.Type == unifiedresources.ResourceTypeAgent {
+ if resource.Agent.Memory.Free != 2<<30 || resource.Agent.Memory.Cache != 20<<30 {
+ t.Fatalf("memory regression: %+v", resource.Agent.Memory)
+ }
+ if c == 0 && (resource.Metrics.DiskRead == nil || resource.Metrics.DiskRead.Value != 11100 || resource.Metrics.DiskWrite.Value != 22200) {
+ t.Fatal("complete host total changed")
+ }
+ if c == 1 && resource.Metrics != nil && (resource.Metrics.DiskRead != nil || resource.Metrics.DiskWrite != nil) {
+ t.Fatal("partial catalogue became complete host IO")
+ }
+ continue
+ }
+ if resource.Type != unifiedresources.ResourceTypePhysicalDisk {
+ continue
+ }
+ diskCount++
+ if resource.PhysicalDisk.IO != nil {
+ t.Fatal("RRD rates fabricated cumulative counters")
+ }
+ target := resourceStore.MetricsTargetForResource(resource.ID)
+ if target == nil || target.ResourceType != "disk" {
+ t.Fatal("disk lost its real history identity")
+ }
+ want := map[string]float64{"disk-a": 100, "disk-b": 1000, "disk-c": 10000}[resource.Name]
+ present := !(c == 1 && resource.Name == "disk-c")
+ if present && (resource.Metrics == nil || resource.Metrics.DiskRead == nil) {
+ t.Fatalf("cycle %d %s lost native per-disk rate", c, resource.Name)
+ }
+ if !present && resource.Metrics != nil && (resource.Metrics.DiskRead != nil || resource.Metrics.DiskWrite != nil) {
+ t.Fatal("absent disk retained a current rate")
+ }
+ if c == 2 && resource.Metrics.DiskWrite != nil {
+ t.Fatal("missing write was carried forward")
+ }
+
+ for _, kind := range []string{"diskread", "diskwrite"} {
+ expected := 1
+ if c > 0 && resource.Name != "disk-c" {
+ expected++
+ }
+ if c == 2 && kind == "diskread" {
+ expected++
+ }
+ points, err := persistent.Query("disk", target.ResourceID, kind, time.Unix(sampleEnd-3600, 0), time.Now(), 0)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(points) != expected {
+ t.Fatalf("cycle %d %s %s: %d history points, want %d (missing/zero/rebuild)", c, resource.Name, kind, len(points), expected)
+ }
+ if points[0].Value != want*map[string]float64{"diskread": 1, "diskwrite": 2}[kind] || points[0].Timestamp.Unix() != sampleEnd {
+ t.Fatalf("native rate/time altered: %+v", points)
+ }
+ if c == 2 && kind == "diskread" && points[len(points)-1].Value != 0 {
+ t.Fatal("idle read became unavailable")
+ }
+ local := monitor.GetDiskMetricsForChart(target.ResourceID, kind, time.Hour)
+ // Sparse local windows may choose the persistent store. Either path must
+ // return the same actual rate observations, not temperatures or counters.
+ if len(local) != expected {
+ t.Fatalf("drawer %s %s lost history: %+v", target.ResourceID, kind, local)
+ }
+ }
+ if c == 0 {
+ raw, _ := json.Marshal(map[string]any{"resource": resource, "target": target, "read": monitor.GetDiskMetricsForChart(target.ResourceID, "diskread", time.Hour), "write": monitor.GetDiskMetricsForChart(target.ResourceID, "diskwrite", time.Hour)})
+ t.Logf("CORE_DISK_CONSUMER %s", raw)
+ }
+ }
+ if diskCount != 3 {
+ t.Fatalf("disk inventory lost: %d", diskCount)
+ }
+ }
+ })
+ }
+}
+
+func TestTrueNASDiskRateWriterAdmission(t *testing.T) {
+ now := time.Now().UTC().Truncate(time.Second)
+ persistent, err := metrics.NewStore(metrics.DefaultConfig(t.TempDir()))
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer func() { _ = persistent.Close() }()
+ monitor := &Monitor{metricsStore: persistent, metricsHistory: NewMetricsHistory(100, time.Hour)}
+ for _, tc := range []struct {
+ name string
+ value float64
+ at time.Time
+ source unifiedresources.DataSource
+ unit string
+ }{
+ {"missing_time", 1, time.Time{}, unifiedresources.SourceTrueNAS, "bytes/s"},
+ {"future", 1, now.Add(time.Hour), unifiedresources.SourceTrueNAS, "bytes/s"},
+ {"negative", -1, now, unifiedresources.SourceTrueNAS, "bytes/s"},
+ {"nan", math.NaN(), now, unifiedresources.SourceTrueNAS, "bytes/s"},
+ {"infinite", math.Inf(1), now, unifiedresources.SourceTrueNAS, "bytes/s"},
+ {"other_source", 1, now, unifiedresources.SourceAgent, "bytes/s"},
+ {"counters", 1, now, unifiedresources.SourceTrueNAS, "bytes"},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ r := unifiedresources.Resource{Metrics: &unifiedresources.ResourceMetrics{DiskRead: &unifiedresources.MetricValue{Value: tc.value, ObservedAt: tc.at, Source: tc.source, Unit: tc.unit}}}
+ monitor.writeTrueNASDiskRates(r, tc.name, now)
+ points, err := persistent.Query("disk", tc.name, "diskread", now.Add(-time.Hour), now.Add(time.Hour), 0)
+ if err != nil || len(points) > 0 {
+ t.Fatalf("invalid sample persisted: %+v %v", points, err)
+ }
+ })
+ }
+ encoded, _ := json.Marshal(unifiedresources.MetricValue{Value: 1, ObservedAt: now, Source: unifiedresources.SourceTrueNAS})
+ if strings.Contains(string(encoded), "ObservedAt") || strings.Contains(string(encoded), "observedAt") {
+ t.Fatal("internal timestamp changed the wire shape")
+ }
+}
diff --git a/internal/monitoring/truenas_disk_rates.go b/internal/monitoring/truenas_disk_rates.go
new file mode 100644
index 0000000000..1d9329c7b6
--- /dev/null
+++ b/internal/monitoring/truenas_disk_rates.go
@@ -0,0 +1,41 @@
+package monitoring
+
+import (
+ "math"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/metrics"
+)
+
+// CORE reporting supplies rates, not kernel counters. Record the canonical
+// selection directly; never fabricate counters or divide a rate a second time.
+// Called only after the physical-disk sync's native-writer/mock/identity gates.
+func (m *Monitor) writeTrueNASDiskRates(resource unifiedresources.Resource, targetID string, now time.Time) {
+ if resource.Metrics == nil || m.metricsStore == nil {
+ return
+ }
+ writes := make([]metrics.WriteMetric, 0, 2)
+ for _, sample := range []struct {
+ kind string
+ metric *unifiedresources.MetricValue
+ }{
+ {"diskread", resource.Metrics.DiskRead}, {"diskwrite", resource.Metrics.DiskWrite},
+ } {
+ metric := sample.metric
+ if metric == nil || metric.Source != unifiedresources.SourceTrueNAS || metric.Unit != "bytes/s" ||
+ metric.ObservedAt.IsZero() || metric.ObservedAt.After(now) || metric.Value < 0 || math.IsNaN(metric.Value) || math.IsInf(metric.Value, 0) {
+ continue
+ }
+ writes = append(writes, metrics.WriteMetric{ResourceType: "disk", ResourceID: targetID, MetricType: sample.kind, Value: metric.Value, Timestamp: metric.ObservedAt, Tier: metrics.TierRaw})
+ }
+ writes = m.dedupeUnifiedMetricWrites(writes)
+ for _, write := range writes {
+ if m.metricsHistory != nil {
+ m.metricsHistory.AddDiskMetric(targetID, write.MetricType, write.Value, write.Timestamp)
+ }
+ }
+ if len(writes) > 0 {
+ m.metricsStore.WriteBatchBounded(writes)
+ }
+}
diff --git a/internal/truenas/client.go b/internal/truenas/client.go
index 47b98c3409..a0d17857fa 100644
--- a/internal/truenas/client.go
+++ b/internal/truenas/client.go
@@ -302,7 +302,11 @@ func systemInfoFromMetricHistory(history *SystemMetricHistory) *SystemInfo {
if history == nil {
return nil
}
- system := &SystemInfo{Telemetry: &SystemTelemetryAvailability{}}
+ system := &SystemInfo{
+ Telemetry: &SystemTelemetryAvailability{},
+ DiskReadRates: latestDeviceRates(history.DiskReadRates),
+ DiskWriteRates: latestDeviceRates(history.DiskWriteRates),
+ }
for _, series := range [][]TimeSeriesPoint{history.CPUPercent, history.MemoryAvailableBytes, history.NetInRate, history.NetOutRate, history.DiskReadRate, history.DiskWriteRate} {
for _, point := range series {
if point.Timestamp.After(system.CollectedAt) {
@@ -2174,6 +2178,8 @@ func mergeSystemTelemetry(system *SystemInfo, telemetry *SystemInfo) {
system.NetOutRate = telemetry.NetOutRate
system.DiskReadRate = telemetry.DiskReadRate
system.DiskWriteRate = telemetry.DiskWriteRate
+ system.DiskReadRates = cloneDeviceRates(telemetry.DiskReadRates)
+ system.DiskWriteRates = cloneDeviceRates(telemetry.DiskWriteRates)
if len(telemetry.TemperatureCelsius) > 0 {
system.TemperatureCelsius = cloneTemperatureMap(telemetry.TemperatureCelsius)
}
@@ -3454,7 +3460,9 @@ func parseSystemMetricHistory(responses []trueNASReportingGetDataResponse) *Syst
history.NetOutRate = sumReportingDevices(io["netout"])
history.DiskReadRate = sumReportingDevices(io["diskread"])
history.DiskWriteRate = sumReportingDevices(io["diskwrite"])
- if len(history.CPUPercent)+len(history.MemoryPercent)+len(history.MemoryUsedBytes)+len(history.MemoryAvailableBytes)+len(history.MemoryTotalBytes)+len(history.ARCSizeBytes)+len(history.NetInRate)+len(history.NetOutRate)+len(history.DiskReadRate)+len(history.DiskWriteRate)+len(history.TemperatureCelsius) == 0 {
+ history.DiskReadRates = reportingDiskRates(io["diskread"])
+ history.DiskWriteRates = reportingDiskRates(io["diskwrite"])
+ if len(history.CPUPercent)+len(history.MemoryPercent)+len(history.MemoryUsedBytes)+len(history.MemoryAvailableBytes)+len(history.MemoryTotalBytes)+len(history.ARCSizeBytes)+len(history.NetInRate)+len(history.NetOutRate)+len(history.DiskReadRate)+len(history.DiskWriteRate)+len(history.TemperatureCelsius)+len(history.DiskReadRates)+len(history.DiskWriteRates) == 0 {
return nil
}
return history
diff --git a/internal/truenas/core_reporting_test.go b/internal/truenas/core_reporting_test.go
index 6a6bddfffd..3f20d7298a 100644
--- a/internal/truenas/core_reporting_test.go
+++ b/internal/truenas/core_reporting_test.go
@@ -5,6 +5,7 @@ import (
"encoding/json"
"errors"
"fmt"
+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"math"
"net/http"
"os"
@@ -12,8 +13,6 @@ import (
"strings"
"testing"
"time"
-
- "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
)
// Five native result excerpts supplied in #2077 on 1 October 2026. Identifiers
@@ -450,3 +449,96 @@ func TestCORE13DiskTemperatureHistoryExternalTiming(t *testing.T) {
assertCOREPoint(t, points, 0, response.Start, 40)
assertCOREPoint(t, points, 1, response.Start+20, 42)
}
+
+func TestCOREPerDiskRatesKeepPartialNativeSamples(t *testing.T) {
+ const start int64 = 1789000000
+ graph := func(device string, rows []any) trueNASReportingGetDataResponse {
+ return trueNASReportingGetDataResponse{Name: "disk", Identifier: device, Start: start, End: start + 20, Step: 10, Legend: []string{"disk_octets_read", "disk_octets_write"}, Data: rows}
+ }
+ a := graph("disk-a", []any{[]any{100., 200.}, []any{0., nil}, []any{nil, 0.}})
+ b := graph("disk-b", []any{[]any{1000., 2000.}, []any{nil, 2200.}, []any{1200., 2400.}})
+ history := parseSystemMetricHistory([]trueNASReportingGetDataResponse{a, b, a, graph("disk-c", nil)})
+ if history == nil || len(history.DiskReadRate) > 0 || len(history.DiskWriteRate) > 0 {
+ t.Fatal("incomplete catalogue invented host totals or discarded good per-disk samples")
+ }
+ assertCOREPoint(t, history.DiskReadRates["disk-a"], 1, start+10, 0)
+ assertCOREPoint(t, history.DiskWriteRates["disk-a"], 1, start+20, 0)
+ if len(history.DiskReadRates["disk-a"]) != 2 {
+ t.Fatal("duplicate graph doubled rates or null became zero")
+ }
+ system := systemInfoFromMetricHistory(history)
+ metrics := metricsFromTrueNASDisk(*system, Disk{Name: "disk-a"})
+ if metrics == nil || metrics.DiskRead.Value != 0 || metrics.DiskWrite.Value != 0 || metrics.DiskRead.ObservedAt.Unix() != start+10 || metrics.DiskWrite.ObservedAt.Unix() != start+20 {
+ t.Fatalf("directional native time/zero lost: %+v", metrics)
+ }
+ if system.Telemetry.DiskRead || system.Telemetry.DiskWrite {
+ t.Fatal("partial device readings made host total complete")
+ }
+ live := parseSystemMetricHistory(liveReportingResponses([]trueNASReportingGetDataResponse{a, b}, start+100))
+ if live != nil {
+ t.Fatal("stale native disk buckets became live samples")
+ }
+ if metricsFromTrueNASDisk(*system, Disk{Name: "unknown", ID: "disk-a", Serial: "disk-b"}) != nil {
+ t.Fatal("guessed disk identity")
+ }
+ clone := cloneSystemInfo(*system)
+ clone.DiskReadRates["disk-a"] = TimeSeriesPoint{Value: 999}
+ if system.DiskReadRates["disk-a"].Value != 0 {
+ t.Fatal("snapshot rates alias caller mutation")
+ }
+ unscoped := parseSystemMetricHistory([]trueNASReportingGetDataResponse{graph("", []any{[]any{4., 5.}})})
+ if unscoped == nil || len(unscoped.DiskReadRates) != 0 || len(unscoped.DiskWriteRates) != 0 || unscoped.DiskReadRate[0].Value != 4 {
+ t.Fatal("unscoped compatibility graph was assigned to a physical disk or lost host data")
+ }
+}
+
+func TestCOREPerDiskRatesRejectInvalidAndPreserveScope(t *testing.T) {
+ for _, value := range []float64{-1, math.NaN(), math.Inf(1), math.Inf(-1)} {
+ if metricsFromTrueNASDisk(SystemInfo{DiskReadRates: map[string]TimeSeriesPoint{"da0": {Timestamp: time.Unix(1789000000, 0), Value: value}}}, Disk{Name: "da0"}) != nil {
+ t.Fatalf("invalid rate became a reading: %v", value)
+ }
+ }
+ for _, device := range []string{"", "other", "/dev/da0", "da0p2", "DA0"} {
+ if metricsFromTrueNASDisk(SystemInfo{DiskReadRates: map[string]TimeSeriesPoint{device: {Timestamp: time.Unix(1789000000, 0), Value: 1}}}, Disk{Name: "da0", ID: device, Serial: device}) != nil {
+ t.Fatalf("unmatched reporting identifier assigned: %q", device)
+ }
+ }
+ if metricsFromTrueNASDisk(SystemInfo{DiskReadRates: map[string]TimeSeriesPoint{"da0": {Value: 1}}}, Disk{Name: "da0"}) != nil {
+ t.Fatal("sample without time accepted")
+ }
+}
+
+func TestCOREPerDiskRatesConnectionIsolation(t *testing.T) {
+ previous := IsFeatureEnabled()
+ SetFeatureEnabled(true)
+ t.Cleanup(func() { SetFeatureEnabled(previous) })
+ at := time.Unix(1789000000, 0).UTC()
+ fixtures := FixtureSnapshot{CollectedAt: at, System: SystemInfo{Hostname: "namesake", DiskReadRates: map[string]TimeSeriesPoint{"da0": {Timestamp: at, Value: 13}}}, Disks: []Disk{{Name: "da0", ID: "{da0}", Serial: "UNKNOWN"}}}
+ first := NewLiveProviderForConnection(&FixtureFetcher{Snapshot: fixtures}, "core-first")
+ second := NewLiveProviderForConnection(&FixtureFetcher{Snapshot: fixtures}, "core-second")
+ if err := first.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ fixtures.System.DiskReadRates["da0"] = TimeSeriesPoint{Timestamp: at, Value: 29}
+ if err := second.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ registry := unifiedresources.NewRegistry(nil)
+ registry.IngestRecords(unifiedresources.SourceTrueNAS, append(first.Records(), second.Records()...))
+ seen := map[string]float64{}
+ for _, r := range registry.List() {
+ if r.Type == unifiedresources.ResourceTypePhysicalDisk {
+ seen[r.ID] = r.Metrics.DiskRead.Value
+ }
+ }
+ if len(seen) != 2 {
+ t.Fatalf("namesake connection/disk identity merged: %+v", seen)
+ }
+ total := 0.
+ for _, v := range seen {
+ total += v
+ }
+ if total != 42 {
+ t.Fatalf("sample crossed snapshot/connection: %+v", seen)
+ }
+}
diff --git a/internal/truenas/disk_rates.go b/internal/truenas/disk_rates.go
new file mode 100644
index 0000000000..53fe5a1dd3
--- /dev/null
+++ b/internal/truenas/disk_rates.go
@@ -0,0 +1,70 @@
+package truenas
+
+import (
+ "math"
+ "strings"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
+)
+
+// Keep each selected disk's valid samples even when another selected graph is
+// missing. Host totals still require every selected member at the same instant.
+// An unscoped compatibility graph cannot identify any physical disk.
+func reportingDiskRates(devices map[string][]TimeSeriesPoint) map[string][]TimeSeriesPoint {
+ rates := make(map[string][]TimeSeriesPoint)
+ for key, points := range devices {
+ identifier := strings.TrimPrefix(key, "disk\x00")
+ if identifier != "" && len(points) > 0 {
+ rates[identifier] = cloneTimeSeriesPoints(points)
+ }
+ }
+ return rates
+}
+
+func latestDeviceRates(series map[string][]TimeSeriesPoint) map[string]TimeSeriesPoint {
+ rates := make(map[string]TimeSeriesPoint, len(series))
+ for device, points := range series {
+ var latest TimeSeriesPoint
+ for _, point := range points {
+ if point.Timestamp.After(latest.Timestamp) && point.Value >= 0 && !math.IsNaN(point.Value) && !math.IsInf(point.Value, 0) {
+ latest = point
+ }
+ }
+ if !latest.Timestamp.IsZero() {
+ rates[device] = latest
+ }
+ }
+ return rates
+}
+
+func cloneDeviceRates(rates map[string]TimeSeriesPoint) map[string]TimeSeriesPoint {
+ if rates == nil {
+ return nil
+ }
+ cloned := make(map[string]TimeSeriesPoint, len(rates))
+ for device, point := range rates {
+ cloned[device] = point
+ }
+ return cloned
+}
+
+func metricsFromTrueNASDisk(system SystemInfo, disk Disk) *unifiedresources.ResourceMetrics {
+ // CORE's reporting catalogue keys disks by the native device name. Do not
+ // guess with serials, pool names, disk IDs, suffixes or a different connection.
+ device := strings.TrimSpace(disk.Name)
+ if device == "" {
+ return nil
+ }
+ metric := func(rates map[string]TimeSeriesPoint) *unifiedresources.MetricValue {
+ point, ok := rates[device]
+ if !ok || point.Timestamp.IsZero() || point.Value < 0 || math.IsNaN(point.Value) || math.IsInf(point.Value, 0) {
+ return nil
+ }
+ return &unifiedresources.MetricValue{Value: point.Value, Unit: "bytes/s", Source: unifiedresources.SourceTrueNAS, ObservedAt: point.Timestamp}
+ }
+ read, write := metric(system.DiskReadRates), metric(system.DiskWriteRates)
+ if read == nil && write == nil {
+ return nil
+ }
+ return &unifiedresources.ResourceMetrics{DiskRead: read, DiskWrite: write}
+}
diff --git a/internal/truenas/provider.go b/internal/truenas/provider.go
index bac1127425..991534bfcf 100644
--- a/internal/truenas/provider.go
+++ b/internal/truenas/provider.go
@@ -853,6 +853,7 @@ func truenasRecordsFromSnapshot(snapshot *FixtureSnapshot, connectionID string,
Resource: unifiedresources.Resource{
Type: unifiedresources.ResourceTypePhysicalDisk,
Name: disk.Name,
+ Metrics: metricsFromTrueNASDisk(snapshot.System, disk),
Status: unifiedresources.IncidentsStatus(unifiedresources.PhysicalDiskStatus(disk.Model, healthFromDisk(disk), assessment), incidents),
LastSeen: collectedAt,
UpdatedAt: collectedAt,
@@ -3103,6 +3104,8 @@ func clonePools(pools []Pool) []Pool {
func cloneSystemInfo(system SystemInfo) SystemInfo {
cloned := system
+ cloned.DiskReadRates = cloneDeviceRates(system.DiskReadRates)
+ cloned.DiskWriteRates = cloneDeviceRates(system.DiskWriteRates)
if system.Telemetry != nil {
availability := *system.Telemetry
cloned.Telemetry = &availability
diff --git a/internal/truenas/types.go b/internal/truenas/types.go
index dfffdaa911..ce12660b33 100644
--- a/internal/truenas/types.go
+++ b/internal/truenas/types.go
@@ -32,12 +32,16 @@ type SystemInfo struct {
// ARCSizeBytes is the ZFS ARC cache size. The kernel's MemAvailable does
// not count ARC even though it shrinks under memory pressure, so usage
// derived from total-available alone reads ~95% on every ZFS system.
- ARCSizeBytes int64
- CPUPercent float64
- NetInRate float64
- NetOutRate float64
- DiskReadRate float64
- DiskWriteRate float64
+ ARCSizeBytes int64
+ CPUPercent float64
+ NetInRate float64
+ NetOutRate float64
+ DiskReadRate float64
+ DiskWriteRate float64
+ // Native per-device rates retain their own sample times and presence.
+ // They are not cumulative counters and must never populate PhysicalDisk.IO.
+ DiskReadRates map[string]TimeSeriesPoint
+ DiskWriteRates map[string]TimeSeriesPoint
TemperatureCelsius map[string]float64
IntervalSeconds int
CollectedAt time.Time
@@ -215,6 +219,8 @@ type SystemMetricHistory struct {
NetOutRate []TimeSeriesPoint
DiskReadRate []TimeSeriesPoint
DiskWriteRate []TimeSeriesPoint
+ DiskReadRates map[string][]TimeSeriesPoint
+ DiskWriteRates map[string][]TimeSeriesPoint
}
// Alert mirrors a TrueNAS alert listing entry.
diff --git a/internal/unifiedresources/clone_test.go b/internal/unifiedresources/clone_test.go
index a5508f29c6..dbf977dee3 100644
--- a/internal/unifiedresources/clone_test.go
+++ b/internal/unifiedresources/clone_test.go
@@ -1,6 +1,7 @@
package unifiedresources
import (
+ "encoding/json"
"reflect"
"testing"
"time"
@@ -1071,3 +1072,24 @@ func TestCloneResourceKeepsContinuityMarker(t *testing.T) {
t.Fatalf("seeded row = %+v (listed=%v), want the continuity marker kept", seeded, ok)
}
}
+
+func TestCloneNativeRateObservation(t *testing.T) {
+ at := time.Unix(1789000000, 0).UTC()
+ original := &Resource{ID: "disk:core:da0", Type: ResourceTypePhysicalDisk, Metrics: &ResourceMetrics{DiskRead: &MetricValue{Value: 0, Unit: "bytes/s", Source: SourceTrueNAS, ObservedAt: at}}}
+ cloned := cloneResource(original)
+ if cloned.Metrics.DiskRead == original.Metrics.DiskRead || !cloned.Metrics.DiskRead.ObservedAt.Equal(at) || cloned.Metrics.DiskRead.Value != 0 {
+ t.Fatal("rate sample time/zero not cloned independently")
+ }
+ cloned.Metrics.DiskRead.ObservedAt = at.Add(time.Hour)
+ if !original.Metrics.DiskRead.ObservedAt.Equal(at) {
+ t.Fatal("clone mutated original sample instant")
+ }
+ raw, err := json.Marshal(original.Metrics.DiskRead)
+ if err != nil || string(raw) != `{"value":0,"unit":"bytes/s"}` {
+ t.Fatalf("rate instant changed wire contract: %s %v", raw, err)
+ }
+ var hydrated MetricValue
+ if err := json.Unmarshal(raw, &hydrated); err != nil || !hydrated.ObservedAt.IsZero() {
+ t.Fatal("rehydrated scalar invented a sample time")
+ }
+}
diff --git a/internal/unifiedresources/types.go b/internal/unifiedresources/types.go
index 246555146f..7c16a1f51d 100644
--- a/internal/unifiedresources/types.go
+++ b/internal/unifiedresources/types.go
@@ -324,12 +324,15 @@ type ResourceMetrics struct {
// MetricValue represents a metric value, optionally with totals.
type MetricValue struct {
- Value float64 `json:"value"`
- Used *int64 `json:"used,omitempty"`
- Total *int64 `json:"total,omitempty"`
- Percent float64 `json:"percent,omitempty"`
- Unit string `json:"unit,omitempty"`
- Source DataSource `json:"-"`
+ // ObservedAt retains a provider-native sample instant for rate writers.
+ // Internal only: a rehydrated scalar without that instant is not a new sample.
+ ObservedAt time.Time `json:"-"`
+ Value float64 `json:"value"`
+ Used *int64 `json:"used,omitempty"`
+ Total *int64 `json:"total,omitempty"`
+ Percent float64 `json:"percent,omitempty"`
+ Unit string `json:"unit,omitempty"`
+ Source DataSource `json:"-"`
// Memory provenance travels with the selected numeric metric, not a
// different platform facet retained on a cross-source merged guest.
Observation models.MemoryObservation `json:"observation,omitzero"`