Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Retain CORE physical-disk throughput through chart history
pulse-triage[bot]91d6c26323c5
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 1cb1db1d81..4529058761 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -15,6 +15,16 @@## Purpose+### API-backed disk rates preserve native-writer authority++The physical-disk sync excludes Agent/Proxmox native writers and mock-owned+records before writing TrueNAS per-device rates. A rate is not a kernel counter,+agent report or continuity signal. Existing target resolution stays authoritative.+The physical-disk roundtrip pipeline covers the API rate route and its adjacent+native-writer regressions remain required. No enrolment, removal, lease, host+identity or installed state transition changes.++A member test announces and reveals its result beside that exact member ratherthan leaving feedback in the off-screen global footer. The normal primaryTest connection result stays in the footer; both remain endpoint-qualifieddiff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex 56dd5ea027..a69893d171 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -17,6 +17,30 @@## Purpose+### CORE per-disk throughput keeps its native meaning++The existing reporting batch retains each selected disk's read/write rate even+when a missing member prevents a complete host sum. Only an exact inventory+device name within the connection matches; serials, pools, partitions, namesakes+and unscoped replies never substitute. Native values remain bytes/s, not counters+or busy percentages. Live age filtering still applies. Read/write presence and+sample times remain independent; zero is observed and null is absent.+The canonical physical-disk writer preserves native instants, rejects invalid,+future or timeless readings, deduplicates rebuilds and keeps existing mock,+native-writer and metric-target gates. Rehydrated scalars cannot mint samples.++`TestCOREPerDiskRatesKeepPartialNativeSamples` and+`TestCOREPerDiskRatesRejectInvalidAndPreserveScope` in `core_reporting_test.go`+cover partial/duplicate/unscoped/stale/zero/null/identity and clone boundaries.+`TestCOREPhysicalDiskReportingPipeline` in `physical_disk_roundtrip_test.go`+crosses pinned TLS REST, snapshot, poller, registry, resolver, writer and local/+persistent drawer reads for both boot names and complete/partial/idle cycles.+`TestTrueNASDiskRateWriterAdmission` rejects invalid observations. Desktop and+narrow browser proof must use actual produced disk History in the existing+drawer. These controls do not establish the reported boot-only host cause,+native missing ARC, operation/busy counters or original appliance recovery.++### Explicit Proxmox endpoint trust survives save and reloadA managed primary fingerprint edit reconciles only that exact HTTP authoritydiff --git a/docs/release-control/v6/internal/subsystems/registry.json b/docs/release-control/v6/internal/subsystems/registry.jsonindex 3a2f593f69..51ae1e9590 100644--- a/docs/release-control/v6/internal/subsystems/registry.json+++ b/docs/release-control/v6/internal/subsystems/registry.json@@ -6511,6 +6511,7 @@"match_prefixes": [],"match_files": ["internal/truenas/client.go",+ "internal/truenas/disk_rates.go","internal/truenas/core_reporting.go","internal/truenas/disk_health.go","internal/truenas/fixtures.go",diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex 8cf254ff01..3a3e057335 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -21,6 +21,17 @@## Purpose+### CORE disk throughput is not protection or counter evidence++Native per-disk read/write rates reach existing disk chart History under the+canonical disk target and original sample instant. They never populate cumulative+`PhysicalDisk.IO`, operation totals, busy percentage or backup/protection evidence.+A partial catalogue preserves individual samples without implying complete host+I/O. The physical-disk roundtrip pipeline checks both boot names and persisted+readback. Device/pool identity and recovery operations remain unchanged; browser+proof concerns the consuming drawer, not original appliance recovery.++### Snapshot subjects retain their observed guest kindA Proxmox snapshot's explicit VM/LXC kind takes precedence over best-effortdiff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.mdindex 176d95bdb5..4c83a7b30f 100644--- a/docs/release-control/v6/internal/subsystems/unified-resources.md+++ b/docs/release-control/v6/internal/subsystems/unified-resources.md@@ -23,6 +23,16 @@ and sort the complete canonical change table while startup and ingestion wait.## Purpose+### Native rate instants survive canonical cloning++`MetricValue.ObservedAt` is an optional internal-only sample instant copied with+the selected scalar, never part of public JSON. TrueNAS CORE per-disk rate writers+use it instead of registry rebuild time. Rehydrated scalars without it are not+new observations. `TestCloneNativeRateObservation` checks independent cloning,+zero preservation and unchanged wire shape; the physical-disk pipeline verifies+resolved targets. Identity, source precedence and cumulative counters do not change.++### Coverage evidence expansion preserves workload ownershipAn uncertain backup date remains inspectable beside the newest dated evidencediff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.goindex b7fe3f44dd..38dc7bac80 100644--- a/internal/monitoring/monitor.go+++ b/internal/monitoring/monitor.go@@ -6666,6 +6666,7 @@ func (m *Monitor) syncUnifiedPhysicalDiskMetrics(store ResourceStoreInterface) {// re-issuing the same snapshot maps to one sample instead of a fresh// wall-clock row (#1966).m.writeSMARTMetrics(disk, unifiedResourceObservedAt(resource, now))+ m.writeTrueNASDiskRates(resource, targetID, now)}}diff --git a/internal/monitoring/physical_disk_roundtrip_test.go b/internal/monitoring/physical_disk_roundtrip_test.goindex 285dcd3eab..4ef6290794 100644--- a/internal/monitoring/physical_disk_roundtrip_test.go+++ b/internal/monitoring/physical_disk_roundtrip_test.go@@ -2,14 +2,10 @@ package monitoringimport ("context"+ "crypto/sha256""encoding/json""errors"- "slices"- "strconv"- "sync"- "testing"- "time"-+ "fmt""github.com/rcourtman/pulse-go-rewrite/internal/alerts""github.com/rcourtman/pulse-go-rewrite/internal/config""github.com/rcourtman/pulse-go-rewrite/internal/models"@@ -19,6 +15,16 @@ import ("github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory""github.com/rcourtman/pulse-go-rewrite/pkg/metrics""github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"+ "math"+ "net/http"+ "net/http/httptest"+ "slices"+ "strconv"+ "strings"+ "sync"+ "sync/atomic"+ "testing"+ "time")// Monitor construction restores notification destinations before admitting@@ -1966,3 +1972,225 @@ func TestMergeHostAgentSMARTIntoDisksKeepsSSHNodeSensorReadingOfSplitDisk(t *test.Fatalf("split disk with a reporting agent kept the agent-supplied reading: %+v / %+v", live, live.Collection)}}++// Registered end-to-end source regression: production pinned REST -> snapshot+// -> poller -> canonical registry/target -> writer -> local/persistent drawer+// readback. Only provider HTTP responses are synthetic; there is no appliance+// mutation, credentials or simulated cumulative kernel counter.+func TestCOREPhysicalDiskReportingPipeline(t *testing.T) {+ previous := truenas.IsFeatureEnabled()+ truenas.SetFeatureEnabled(true)+ t.Cleanup(func() { truenas.SetFeatureEnabled(previous) })+ for _, boot := range []string{"boot-pool", "freenas-boot"} {+ t.Run(boot, func(t *testing.T) {+ var cycle atomic.Int64+ sampleEnd := time.Now().UTC().Add(-2 * time.Minute).Truncate(time.Second).Unix()+ server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ w.Header().Set("Content-Type", "application/json")+ switch r.URL.Path {+ case "/api/v2.0/system/info":+ fmt.Fprint(w, `{"hostname":"namesake-core","version":"TrueNAS-13.0-U6.1","cores":8,"physmem":34359738368}`)+ case "/api/v2.0/pool":+ fmt.Fprintf(w, `[{"id":1,"name":%q,"status":"ONLINE","size":1000,"allocated":400,"free":600},{"id":2,"name":"data-pool","status":"ONLINE","size":100000,"allocated":60000,"free":40000}]`, boot)+ case "/api/v2.0/disk":+ fmt.Fprintf(w, `[{"identifier":"{disk-a}","name":"disk-a","size":1000,"pool":%q,"model":"Synthetic boot disk"},{"identifier":"{disk-b}","name":"disk-b","size":100000,"pool":"data-pool","model":"Synthetic data disk"},{"identifier":"{disk-c}","name":"disk-c","size":100000,"pool":"data-pool","model":"Synthetic data disk"}]`, boot)+ case "/api/v2.0/disk/temperatures":+ fmt.Fprint(w, `{"disk-a":30,"disk-b":31,"disk-c":32}`)+ case "/api/v2.0/pool/dataset", "/api/v2.0/alert/list":+ fmt.Fprint(w, `[]`)+ case "/api/v2.0/reporting/graphs":+ fmt.Fprint(w, `[{"name":"disk","identifiers":["disk-a","disk-b","disk-c"]}]`)+ case "/api/v2.0/reporting/get_data":+ var request struct {+ Graphs []map[string]any `json:"graphs"`+ Query map[string]any `json:"reporting_query"`+ }+ if err := json.NewDecoder(r.Body).Decode(&request); err != nil {+ t.Error(err)+ w.WriteHeader(400)+ return+ }+ end := sampleEnd + cycle.Load()*10+ var replies []map[string]any+ for _, g := range request.Graphs {+ name, _ := g["name"].(string)+ id, _ := g["identifier"].(string)+ reply := map[string]any{"name": name, "identifier": id, "start": end - 10, "end": end, "step": 10}+ switch name {+ case "memory":+ reply["legend"] = []string{"memory-free_value"}+ reply["data"] = [][]any{{float64(2 << 30)}, {float64(2 << 30)}}+ case "arcsize":+ reply["legend"] = []string{"arcsize_value"}+ reply["data"] = [][]any{{float64(20 << 30)}, {float64(20 << 30)}}+ case "disk":+ if cycle.Load() == 1 && id == "disk-c" {+ continue+ } // selected but absent, not idle+ rates := map[string][]any{"disk-a": {100., 200.}, "disk-b": {1000., 2000.}, "disk-c": {10000., 20000.}}+ rate, ok := rates[id]+ if !ok {+ t.Errorf("unexpected disk query %q", id)+ w.WriteHeader(400)+ return+ }+ if cycle.Load() == 2 {+ rate = []any{0., nil}+ } // true idle reads, missing writes+ reply["legend"] = []string{"disk_octets_read", "disk_octets_write"}+ reply["data"] = [][]any{rate, rate}+ default:+ continue+ }+ replies = append(replies, reply)+ }+ _ = json.NewEncoder(w).Encode(replies)+ default:+ http.NotFound(w, r)+ }+ }))+ defer server.Close()+ fingerprint := fmt.Sprintf("%x", sha256.Sum256(server.Certificate().Raw))+ client, err := truenas.NewClient(truenas.ClientConfig{Host: server.URL, APIKey: "synthetic", Fingerprint: fingerprint, InsecureSkipVerify: true})+ if err != nil {+ t.Fatal(err)+ }+ defer client.Close()+ provider := truenas.NewLiveProviderForConnection(&truenas.APIFetcher{Client: client}, "core-a")+ instance := config.TrueNASInstance{ID: "core-a", Host: server.URL, Enabled: true}+ poller := NewTrueNASPoller(nil, 0, nil)+ poller.providersByOrg["default"] = map[string]*truenas.Provider{instance.ID: provider}+ poller.configsByOrg["default"] = map[string]config.TrueNASInstance{instance.ID: instance}+ persistent, err := metrics.NewStore(metrics.DefaultConfig(t.TempDir()))+ if err != nil {+ t.Fatal(err)+ }+ defer func() { _ = persistent.Close() }()+ resourceStore := unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(unifiedresources.NewMemoryStore()))+ monitor := &Monitor{resourceStore: resourceStore, metricsHistory: NewMetricsHistory(1024, 24*time.Hour), metricsStore: persistent}+ for c := int64(0); c < 3; c++ {+ cycle.Store(c)+ poller.ensureConnectionRuntimeStatusLocked("default", instance.ID).nextPollAt = time.Now().Add(-time.Second)+ poller.pollAll(context.Background())+ records := poller.GetCurrentRecordsForOrg("default")+ if len(records) == 0 {+ t.Fatal("production poll lost inventory")+ }+ resourceStore.PopulateSnapshotAndSupplemental(models.StateSnapshot{}, map[unifiedresources.DataSource][]unifiedresources.IngestRecord{unifiedresources.SourceTrueNAS: records})+ monitor.syncUnifiedPhysicalDiskMetrics(resourceStore)+ monitor.syncUnifiedPhysicalDiskMetrics(resourceStore)+ diskCount := 0+ for _, resource := range resourceStore.GetAll() {+ if resource.Type == unifiedresources.ResourceTypeAgent {+ if resource.Agent.Memory.Free != 2<<30 || resource.Agent.Memory.Cache != 20<<30 {+ t.Fatalf("memory regression: %+v", resource.Agent.Memory)+ }+ if c == 0 && (resource.Metrics.DiskRead == nil || resource.Metrics.DiskRead.Value != 11100 || resource.Metrics.DiskWrite.Value != 22200) {+ t.Fatal("complete host total changed")+ }+ if c == 1 && resource.Metrics != nil && (resource.Metrics.DiskRead != nil || resource.Metrics.DiskWrite != nil) {+ t.Fatal("partial catalogue became complete host IO")+ }+ continue+ }+ if resource.Type != unifiedresources.ResourceTypePhysicalDisk {+ continue+ }+ diskCount+++ if resource.PhysicalDisk.IO != nil {+ t.Fatal("RRD rates fabricated cumulative counters")+ }+ target := resourceStore.MetricsTargetForResource(resource.ID)+ if target == nil || target.ResourceType != "disk" {+ t.Fatal("disk lost its real history identity")+ }+ want := map[string]float64{"disk-a": 100, "disk-b": 1000, "disk-c": 10000}[resource.Name]+ present := !(c == 1 && resource.Name == "disk-c")+ if present && (resource.Metrics == nil || resource.Metrics.DiskRead == nil) {+ t.Fatalf("cycle %d %s lost native per-disk rate", c, resource.Name)+ }+ if !present && resource.Metrics != nil && (resource.Metrics.DiskRead != nil || resource.Metrics.DiskWrite != nil) {+ t.Fatal("absent disk retained a current rate")+ }+ if c == 2 && resource.Metrics.DiskWrite != nil {+ t.Fatal("missing write was carried forward")+ }++ for _, kind := range []string{"diskread", "diskwrite"} {+ expected := 1+ if c > 0 && resource.Name != "disk-c" {+ expected+++ }+ if c == 2 && kind == "diskread" {+ expected+++ }+ points, err := persistent.Query("disk", target.ResourceID, kind, time.Unix(sampleEnd-3600, 0), time.Now(), 0)+ if err != nil {+ t.Fatal(err)+ }+ if len(points) != expected {+ t.Fatalf("cycle %d %s %s: %d history points, want %d (missing/zero/rebuild)", c, resource.Name, kind, len(points), expected)+ }+ if points[0].Value != want*map[string]float64{"diskread": 1, "diskwrite": 2}[kind] || points[0].Timestamp.Unix() != sampleEnd {+ t.Fatalf("native rate/time altered: %+v", points)+ }+ if c == 2 && kind == "diskread" && points[len(points)-1].Value != 0 {+ t.Fatal("idle read became unavailable")+ }+ local := monitor.GetDiskMetricsForChart(target.ResourceID, kind, time.Hour)+ // Sparse local windows may choose the persistent store. Either path must+ // return the same actual rate observations, not temperatures or counters.+ if len(local) != expected {+ t.Fatalf("drawer %s %s lost history: %+v", target.ResourceID, kind, local)+ }+ }+ if c == 0 {+ raw, _ := json.Marshal(map[string]any{"resource": resource, "target": target, "read": monitor.GetDiskMetricsForChart(target.ResourceID, "diskread", time.Hour), "write": monitor.GetDiskMetricsForChart(target.ResourceID, "diskwrite", time.Hour)})+ t.Logf("CORE_DISK_CONSUMER %s", raw)+ }+ }+ if diskCount != 3 {+ t.Fatalf("disk inventory lost: %d", diskCount)+ }+ }+ })+ }+}++func TestTrueNASDiskRateWriterAdmission(t *testing.T) {+ now := time.Now().UTC().Truncate(time.Second)+ persistent, err := metrics.NewStore(metrics.DefaultConfig(t.TempDir()))+ if err != nil {+ t.Fatal(err)+ }+ defer func() { _ = persistent.Close() }()+ monitor := &Monitor{metricsStore: persistent, metricsHistory: NewMetricsHistory(100, time.Hour)}+ for _, tc := range []struct {+ name string+ value float64+ at time.Time+ source unifiedresources.DataSource+ unit string+ }{+ {"missing_time", 1, time.Time{}, unifiedresources.SourceTrueNAS, "bytes/s"},+ {"future", 1, now.Add(time.Hour), unifiedresources.SourceTrueNAS, "bytes/s"},+ {"negative", -1, now, unifiedresources.SourceTrueNAS, "bytes/s"},+ {"nan", math.NaN(), now, unifiedresources.SourceTrueNAS, "bytes/s"},+ {"infinite", math.Inf(1), now, unifiedresources.SourceTrueNAS, "bytes/s"},+ {"other_source", 1, now, unifiedresources.SourceAgent, "bytes/s"},+ {"counters", 1, now, unifiedresources.SourceTrueNAS, "bytes"},+ } {+ t.Run(tc.name, func(t *testing.T) {+ r := unifiedresources.Resource{Metrics: &unifiedresources.ResourceMetrics{DiskRead: &unifiedresources.MetricValue{Value: tc.value, ObservedAt: tc.at, Source: tc.source, Unit: tc.unit}}}+ monitor.writeTrueNASDiskRates(r, tc.name, now)+ points, err := persistent.Query("disk", tc.name, "diskread", now.Add(-time.Hour), now.Add(time.Hour), 0)+ if err != nil || len(points) > 0 {+ t.Fatalf("invalid sample persisted: %+v %v", points, err)+ }+ })+ }+ encoded, _ := json.Marshal(unifiedresources.MetricValue{Value: 1, ObservedAt: now, Source: unifiedresources.SourceTrueNAS})+ if strings.Contains(string(encoded), "ObservedAt") || strings.Contains(string(encoded), "observedAt") {+ t.Fatal("internal timestamp changed the wire shape")+ }+}diff --git a/internal/monitoring/truenas_disk_rates.go b/internal/monitoring/truenas_disk_rates.gonew file mode 100644index 0000000000..1d9329c7b6--- /dev/null+++ b/internal/monitoring/truenas_disk_rates.go@@ -0,0 +1,41 @@+package monitoring++import (+ "math"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"+ "github.com/rcourtman/pulse-go-rewrite/pkg/metrics"+)++// CORE reporting supplies rates, not kernel counters. Record the canonical+// selection directly; never fabricate counters or divide a rate a second time.+// Called only after the physical-disk sync's native-writer/mock/identity gates.+func (m *Monitor) writeTrueNASDiskRates(resource unifiedresources.Resource, targetID string, now time.Time) {+ if resource.Metrics == nil || m.metricsStore == nil {+ return+ }+ writes := make([]metrics.WriteMetric, 0, 2)+ for _, sample := range []struct {+ kind string+ metric *unifiedresources.MetricValue+ }{+ {"diskread", resource.Metrics.DiskRead}, {"diskwrite", resource.Metrics.DiskWrite},+ } {+ metric := sample.metric+ if metric == nil || metric.Source != unifiedresources.SourceTrueNAS || metric.Unit != "bytes/s" ||+ metric.ObservedAt.IsZero() || metric.ObservedAt.After(now) || metric.Value < 0 || math.IsNaN(metric.Value) || math.IsInf(metric.Value, 0) {+ continue+ }+ writes = append(writes, metrics.WriteMetric{ResourceType: "disk", ResourceID: targetID, MetricType: sample.kind, Value: metric.Value, Timestamp: metric.ObservedAt, Tier: metrics.TierRaw})+ }+ writes = m.dedupeUnifiedMetricWrites(writes)+ for _, write := range writes {+ if m.metricsHistory != nil {+ m.metricsHistory.AddDiskMetric(targetID, write.MetricType, write.Value, write.Timestamp)+ }+ }+ if len(writes) > 0 {+ m.metricsStore.WriteBatchBounded(writes)+ }+}diff --git a/internal/truenas/client.go b/internal/truenas/client.goindex 47b98c3409..a0d17857fa 100644--- a/internal/truenas/client.go+++ b/internal/truenas/client.go@@ -302,7 +302,11 @@ func systemInfoFromMetricHistory(history *SystemMetricHistory) *SystemInfo {if history == nil {return nil}- system := &SystemInfo{Telemetry: &SystemTelemetryAvailability{}}+ system := &SystemInfo{+ Telemetry: &SystemTelemetryAvailability{},+ DiskReadRates: latestDeviceRates(history.DiskReadRates),+ DiskWriteRates: latestDeviceRates(history.DiskWriteRates),+ }for _, series := range [][]TimeSeriesPoint{history.CPUPercent, history.MemoryAvailableBytes, history.NetInRate, history.NetOutRate, history.DiskReadRate, history.DiskWriteRate} {for _, point := range series {if point.Timestamp.After(system.CollectedAt) {@@ -2174,6 +2178,8 @@ func mergeSystemTelemetry(system *SystemInfo, telemetry *SystemInfo) {system.NetOutRate = telemetry.NetOutRatesystem.DiskReadRate = telemetry.DiskReadRatesystem.DiskWriteRate = telemetry.DiskWriteRate+ system.DiskReadRates = cloneDeviceRates(telemetry.DiskReadRates)+ system.DiskWriteRates = cloneDeviceRates(telemetry.DiskWriteRates)if len(telemetry.TemperatureCelsius) > 0 {system.TemperatureCelsius = cloneTemperatureMap(telemetry.TemperatureCelsius)}@@ -3454,7 +3460,9 @@ func parseSystemMetricHistory(responses []trueNASReportingGetDataResponse) *Systhistory.NetOutRate = sumReportingDevices(io["netout"])history.DiskReadRate = sumReportingDevices(io["diskread"])history.DiskWriteRate = sumReportingDevices(io["diskwrite"])- if len(history.CPUPercent)+len(history.MemoryPercent)+len(history.MemoryUsedBytes)+len(history.MemoryAvailableBytes)+len(history.MemoryTotalBytes)+len(history.ARCSizeBytes)+len(history.NetInRate)+len(history.NetOutRate)+len(history.DiskReadRate)+len(history.DiskWriteRate)+len(history.TemperatureCelsius) == 0 {+ history.DiskReadRates = reportingDiskRates(io["diskread"])+ history.DiskWriteRates = reportingDiskRates(io["diskwrite"])+ if len(history.CPUPercent)+len(history.MemoryPercent)+len(history.MemoryUsedBytes)+len(history.MemoryAvailableBytes)+len(history.MemoryTotalBytes)+len(history.ARCSizeBytes)+len(history.NetInRate)+len(history.NetOutRate)+len(history.DiskReadRate)+len(history.DiskWriteRate)+len(history.TemperatureCelsius)+len(history.DiskReadRates)+len(history.DiskWriteRates) == 0 {return nil}return historydiff --git a/internal/truenas/core_reporting_test.go b/internal/truenas/core_reporting_test.goindex 6a6bddfffd..3f20d7298a 100644--- a/internal/truenas/core_reporting_test.go+++ b/internal/truenas/core_reporting_test.go@@ -5,6 +5,7 @@ import ("encoding/json""errors""fmt"+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources""math""net/http""os"@@ -12,8 +13,6 @@ import ("strings""testing""time"-- "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources")// Five native result excerpts supplied in #2077 on 1 October 2026. Identifiers@@ -450,3 +449,96 @@ func TestCORE13DiskTemperatureHistoryExternalTiming(t *testing.T) {assertCOREPoint(t, points, 0, response.Start, 40)assertCOREPoint(t, points, 1, response.Start+20, 42)}++func TestCOREPerDiskRatesKeepPartialNativeSamples(t *testing.T) {+ const start int64 = 1789000000+ graph := func(device string, rows []any) trueNASReportingGetDataResponse {+ return trueNASReportingGetDataResponse{Name: "disk", Identifier: device, Start: start, End: start + 20, Step: 10, Legend: []string{"disk_octets_read", "disk_octets_write"}, Data: rows}+ }+ a := graph("disk-a", []any{[]any{100., 200.}, []any{0., nil}, []any{nil, 0.}})+ b := graph("disk-b", []any{[]any{1000., 2000.}, []any{nil, 2200.}, []any{1200., 2400.}})+ history := parseSystemMetricHistory([]trueNASReportingGetDataResponse{a, b, a, graph("disk-c", nil)})+ if history == nil || len(history.DiskReadRate) > 0 || len(history.DiskWriteRate) > 0 {+ t.Fatal("incomplete catalogue invented host totals or discarded good per-disk samples")+ }+ assertCOREPoint(t, history.DiskReadRates["disk-a"], 1, start+10, 0)+ assertCOREPoint(t, history.DiskWriteRates["disk-a"], 1, start+20, 0)+ if len(history.DiskReadRates["disk-a"]) != 2 {+ t.Fatal("duplicate graph doubled rates or null became zero")+ }+ system := systemInfoFromMetricHistory(history)+ metrics := metricsFromTrueNASDisk(*system, Disk{Name: "disk-a"})+ if metrics == nil || metrics.DiskRead.Value != 0 || metrics.DiskWrite.Value != 0 || metrics.DiskRead.ObservedAt.Unix() != start+10 || metrics.DiskWrite.ObservedAt.Unix() != start+20 {+ t.Fatalf("directional native time/zero lost: %+v", metrics)+ }+ if system.Telemetry.DiskRead || system.Telemetry.DiskWrite {+ t.Fatal("partial device readings made host total complete")+ }+ live := parseSystemMetricHistory(liveReportingResponses([]trueNASReportingGetDataResponse{a, b}, start+100))+ if live != nil {+ t.Fatal("stale native disk buckets became live samples")+ }+ if metricsFromTrueNASDisk(*system, Disk{Name: "unknown", ID: "disk-a", Serial: "disk-b"}) != nil {+ t.Fatal("guessed disk identity")+ }+ clone := cloneSystemInfo(*system)+ clone.DiskReadRates["disk-a"] = TimeSeriesPoint{Value: 999}+ if system.DiskReadRates["disk-a"].Value != 0 {+ t.Fatal("snapshot rates alias caller mutation")+ }+ unscoped := parseSystemMetricHistory([]trueNASReportingGetDataResponse{graph("", []any{[]any{4., 5.}})})+ if unscoped == nil || len(unscoped.DiskReadRates) != 0 || len(unscoped.DiskWriteRates) != 0 || unscoped.DiskReadRate[0].Value != 4 {+ t.Fatal("unscoped compatibility graph was assigned to a physical disk or lost host data")+ }+}++func TestCOREPerDiskRatesRejectInvalidAndPreserveScope(t *testing.T) {+ for _, value := range []float64{-1, math.NaN(), math.Inf(1), math.Inf(-1)} {+ if metricsFromTrueNASDisk(SystemInfo{DiskReadRates: map[string]TimeSeriesPoint{"da0": {Timestamp: time.Unix(1789000000, 0), Value: value}}}, Disk{Name: "da0"}) != nil {+ t.Fatalf("invalid rate became a reading: %v", value)+ }+ }+ for _, device := range []string{"", "other", "/dev/da0", "da0p2", "DA0"} {+ if metricsFromTrueNASDisk(SystemInfo{DiskReadRates: map[string]TimeSeriesPoint{device: {Timestamp: time.Unix(1789000000, 0), Value: 1}}}, Disk{Name: "da0", ID: device, Serial: device}) != nil {+ t.Fatalf("unmatched reporting identifier assigned: %q", device)+ }+ }+ if metricsFromTrueNASDisk(SystemInfo{DiskReadRates: map[string]TimeSeriesPoint{"da0": {Value: 1}}}, Disk{Name: "da0"}) != nil {+ t.Fatal("sample without time accepted")+ }+}++func TestCOREPerDiskRatesConnectionIsolation(t *testing.T) {+ previous := IsFeatureEnabled()+ SetFeatureEnabled(true)+ t.Cleanup(func() { SetFeatureEnabled(previous) })+ at := time.Unix(1789000000, 0).UTC()+ fixtures := FixtureSnapshot{CollectedAt: at, System: SystemInfo{Hostname: "namesake", DiskReadRates: map[string]TimeSeriesPoint{"da0": {Timestamp: at, Value: 13}}}, Disks: []Disk{{Name: "da0", ID: "{da0}", Serial: "UNKNOWN"}}}+ first := NewLiveProviderForConnection(&FixtureFetcher{Snapshot: fixtures}, "core-first")+ second := NewLiveProviderForConnection(&FixtureFetcher{Snapshot: fixtures}, "core-second")+ if err := first.Refresh(context.Background()); err != nil {+ t.Fatal(err)+ }+ fixtures.System.DiskReadRates["da0"] = TimeSeriesPoint{Timestamp: at, Value: 29}+ if err := second.Refresh(context.Background()); err != nil {+ t.Fatal(err)+ }+ registry := unifiedresources.NewRegistry(nil)+ registry.IngestRecords(unifiedresources.SourceTrueNAS, append(first.Records(), second.Records()...))+ seen := map[string]float64{}+ for _, r := range registry.List() {+ if r.Type == unifiedresources.ResourceTypePhysicalDisk {+ seen[r.ID] = r.Metrics.DiskRead.Value+ }+ }+ if len(seen) != 2 {+ t.Fatalf("namesake connection/disk identity merged: %+v", seen)+ }+ total := 0.+ for _, v := range seen {+ total += v+ }+ if total != 42 {+ t.Fatalf("sample crossed snapshot/connection: %+v", seen)+ }+}diff --git a/internal/truenas/disk_rates.go b/internal/truenas/disk_rates.gonew file mode 100644index 0000000000..53fe5a1dd3--- /dev/null+++ b/internal/truenas/disk_rates.go@@ -0,0 +1,70 @@+package truenas++import (+ "math"+ "strings"++ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"+)++// Keep each selected disk's valid samples even when another selected graph is+// missing. Host totals still require every selected member at the same instant.+// An unscoped compatibility graph cannot identify any physical disk.+func reportingDiskRates(devices map[string][]TimeSeriesPoint) map[string][]TimeSeriesPoint {+ rates := make(map[string][]TimeSeriesPoint)+ for key, points := range devices {+ identifier := strings.TrimPrefix(key, "disk\x00")+ if identifier != "" && len(points) > 0 {+ rates[identifier] = cloneTimeSeriesPoints(points)+ }+ }+ return rates+}++func latestDeviceRates(series map[string][]TimeSeriesPoint) map[string]TimeSeriesPoint {+ rates := make(map[string]TimeSeriesPoint, len(series))+ for device, points := range series {+ var latest TimeSeriesPoint+ for _, point := range points {+ if point.Timestamp.After(latest.Timestamp) && point.Value >= 0 && !math.IsNaN(point.Value) && !math.IsInf(point.Value, 0) {+ latest = point+ }+ }+ if !latest.Timestamp.IsZero() {+ rates[device] = latest+ }+ }+ return rates+}++func cloneDeviceRates(rates map[string]TimeSeriesPoint) map[string]TimeSeriesPoint {+ if rates == nil {+ return nil+ }+ cloned := make(map[string]TimeSeriesPoint, len(rates))+ for device, point := range rates {+ cloned[device] = point+ }+ return cloned+}++func metricsFromTrueNASDisk(system SystemInfo, disk Disk) *unifiedresources.ResourceMetrics {+ // CORE's reporting catalogue keys disks by the native device name. Do not+ // guess with serials, pool names, disk IDs, suffixes or a different connection.+ device := strings.TrimSpace(disk.Name)+ if device == "" {+ return nil+ }+ metric := func(rates map[string]TimeSeriesPoint) *unifiedresources.MetricValue {+ point, ok := rates[device]+ if !ok || point.Timestamp.IsZero() || point.Value < 0 || math.IsNaN(point.Value) || math.IsInf(point.Value, 0) {+ return nil+ }+ return &unifiedresources.MetricValue{Value: point.Value, Unit: "bytes/s", Source: unifiedresources.SourceTrueNAS, ObservedAt: point.Timestamp}+ }+ read, write := metric(system.DiskReadRates), metric(system.DiskWriteRates)+ if read == nil && write == nil {+ return nil+ }+ return &unifiedresources.ResourceMetrics{DiskRead: read, DiskWrite: write}+}diff --git a/internal/truenas/provider.go b/internal/truenas/provider.goindex bac1127425..991534bfcf 100644--- a/internal/truenas/provider.go+++ b/internal/truenas/provider.go@@ -853,6 +853,7 @@ func truenasRecordsFromSnapshot(snapshot *FixtureSnapshot, connectionID string,Resource: unifiedresources.Resource{Type: unifiedresources.ResourceTypePhysicalDisk,Name: disk.Name,+ Metrics: metricsFromTrueNASDisk(snapshot.System, disk),Status: unifiedresources.IncidentsStatus(unifiedresources.PhysicalDiskStatus(disk.Model, healthFromDisk(disk), assessment), incidents),LastSeen: collectedAt,UpdatedAt: collectedAt,@@ -3103,6 +3104,8 @@ func clonePools(pools []Pool) []Pool {func cloneSystemInfo(system SystemInfo) SystemInfo {cloned := system+ cloned.DiskReadRates = cloneDeviceRates(system.DiskReadRates)+ cloned.DiskWriteRates = cloneDeviceRates(system.DiskWriteRates)if system.Telemetry != nil {availability := *system.Telemetrycloned.Telemetry = &availabilitydiff --git a/internal/truenas/types.go b/internal/truenas/types.goindex dfffdaa911..ce12660b33 100644--- a/internal/truenas/types.go+++ b/internal/truenas/types.go@@ -32,12 +32,16 @@ type SystemInfo struct {// ARCSizeBytes is the ZFS ARC cache size. The kernel's MemAvailable does// not count ARC even though it shrinks under memory pressure, so usage// derived from total-available alone reads ~95% on every ZFS system.- ARCSizeBytes int64- CPUPercent float64- NetInRate float64- NetOutRate float64- DiskReadRate float64- DiskWriteRate float64+ ARCSizeBytes int64+ CPUPercent float64+ NetInRate float64+ NetOutRate float64+ DiskReadRate float64+ DiskWriteRate float64+ // Native per-device rates retain their own sample times and presence.+ // They are not cumulative counters and must never populate PhysicalDisk.IO.+ DiskReadRates map[string]TimeSeriesPoint+ DiskWriteRates map[string]TimeSeriesPointTemperatureCelsius map[string]float64IntervalSeconds intCollectedAt time.Time@@ -215,6 +219,8 @@ type SystemMetricHistory struct {NetOutRate []TimeSeriesPointDiskReadRate []TimeSeriesPointDiskWriteRate []TimeSeriesPoint+ DiskReadRates map[string][]TimeSeriesPoint+ DiskWriteRates map[string][]TimeSeriesPoint}// Alert mirrors a TrueNAS alert listing entry.diff --git a/internal/unifiedresources/clone_test.go b/internal/unifiedresources/clone_test.goindex a5508f29c6..dbf977dee3 100644--- a/internal/unifiedresources/clone_test.go+++ b/internal/unifiedresources/clone_test.go@@ -1,6 +1,7 @@package unifiedresourcesimport (+ "encoding/json""reflect""testing""time"@@ -1071,3 +1072,24 @@ func TestCloneResourceKeepsContinuityMarker(t *testing.T) {t.Fatalf("seeded row = %+v (listed=%v), want the continuity marker kept", seeded, ok)}}++func TestCloneNativeRateObservation(t *testing.T) {+ at := time.Unix(1789000000, 0).UTC()+ original := &Resource{ID: "disk:core:da0", Type: ResourceTypePhysicalDisk, Metrics: &ResourceMetrics{DiskRead: &MetricValue{Value: 0, Unit: "bytes/s", Source: SourceTrueNAS, ObservedAt: at}}}+ cloned := cloneResource(original)+ if cloned.Metrics.DiskRead == original.Metrics.DiskRead || !cloned.Metrics.DiskRead.ObservedAt.Equal(at) || cloned.Metrics.DiskRead.Value != 0 {+ t.Fatal("rate sample time/zero not cloned independently")+ }+ cloned.Metrics.DiskRead.ObservedAt = at.Add(time.Hour)+ if !original.Metrics.DiskRead.ObservedAt.Equal(at) {+ t.Fatal("clone mutated original sample instant")+ }+ raw, err := json.Marshal(original.Metrics.DiskRead)+ if err != nil || string(raw) != `{"value":0,"unit":"bytes/s"}` {+ t.Fatalf("rate instant changed wire contract: %s %v", raw, err)+ }+ var hydrated MetricValue+ if err := json.Unmarshal(raw, &hydrated); err != nil || !hydrated.ObservedAt.IsZero() {+ t.Fatal("rehydrated scalar invented a sample time")+ }+}diff --git a/internal/unifiedresources/types.go b/internal/unifiedresources/types.goindex 246555146f..7c16a1f51d 100644--- a/internal/unifiedresources/types.go+++ b/internal/unifiedresources/types.go@@ -324,12 +324,15 @@ type ResourceMetrics struct {// MetricValue represents a metric value, optionally with totals.type MetricValue struct {- Value float64 `json:"value"`- Used *int64 `json:"used,omitempty"`- Total *int64 `json:"total,omitempty"`- Percent float64 `json:"percent,omitempty"`- Unit string `json:"unit,omitempty"`- Source DataSource `json:"-"`+ // ObservedAt retains a provider-native sample instant for rate writers.+ // Internal only: a rehydrated scalar without that instant is not a new sample.+ ObservedAt time.Time `json:"-"`+ Value float64 `json:"value"`+ Used *int64 `json:"used,omitempty"`+ Total *int64 `json:"total,omitempty"`+ Percent float64 `json:"percent,omitempty"`+ Unit string `json:"unit,omitempty"`+ Source DataSource `json:"-"`// Memory provenance travels with the selected numeric metric, not a// different platform facet retained on a cross-source merged guest.Observation models.MemoryObservation `json:"observation,omitzero"`