Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Alert on the Unraid temperature an agent disk shows
rcourtmanac631eadb87f
diff --git a/docs/release-control/v6/internal/subsystems/alerts.md b/docs/release-control/v6/internal/subsystems/alerts.mdindex 9d7946179e..a07724fb6b 100644--- a/docs/release-control/v6/internal/subsystems/alerts.md+++ b/docs/release-control/v6/internal/subsystems/alerts.md@@ -1867,6 +1867,61 @@ well as its alerts.`internal/alerts/host_unraid_lifecycle_test.go` pins the clear, the holds, therestarted count and the pending-run and count cleanup.+### Agent disk temperature alerts judge the reading the disk shows++`CheckHost` judges each agent disk on the temperature, collection state and+disk type the disk shows, read by `unifiedresources.HostDiskTemperatureReadings`+from the same adapters the registry builds the disks with. A SMART row shows its+own reading or, when the agent's probe returned none, the one the host's Unraid+inventory reports for that disk (`HostSMARTDiskTemperature`), and keeps its+`agent:<host>/disk_temp:<device>` resource. An Unraid inventory row whose disk+key no SMART row carries, a disk only the inventory lists, shows that row's+reading (`HostUnraidDiskTemperature`) and its own transport as disk type, under+its kernel block device token, the resource a plain Linux SMART row for the+same device uses. A SMART row and an Unraid row with one key are one registry+disk even under different device labels, such as a controller member+(`0 [megaraid,0]`) and its Unraid device (`sda`), and alert once, under the+SMART row's resource. When smartctl and Unraid name+one device with different serials the registry shows two disks on it, and the+device's one alert judges the reading that stands highest against its own disk+type's threshold (`judgedHostDiskTemperature`): over its trigger first, then+inside its recovery band, then the larger margin over the trigger, with a tie+keeping the reading listed first.+Before this `CheckHost` read only the SMART row's own field, while the Physical+Disks Temp cell, the Running Hot verdict ("at or above its alert threshold")+and Patrol judged the Unraid reading against the same trigger, so such a disk+could read as at its alert threshold with no alert. Unraid's own warning and+critical disk temperature notifications do not make this redundant: Pulse+already alerts on the smartctl readings and array state of the same hosts,+which Unraid also notifies on, and a user who routes alerts through Pulse sets+these thresholds here.++Only a reading collected now fires or resolves+(`HostDiskTemperatureReading.Collected`: no standby row, and+`diskinventory.TemperatureCollected`), so a spun-down disk holds the alert+like a listed disk without a temperature. A disk only the+inventory lists whose device matches the agent's `--disk-exclude` patterns+(`fsfilters.MatchesDeviceExclude`) raises nothing, and an alert it raised before+the exclusion resolves on the next report (`clearHostDiskTemperatureResources`),+whatever the SMART list holds, unless another listed reading uses the same+resource: `docs/UNIFIED_AGENT.md` documents exclusion as+removing the disk from monitoring, and linked Proxmox hosts already judge an+excluded disk healthy. A disk only the inventory lists is also judged in a+report whose SMART list is empty, because the disk shows the inventory's+reading in that report. The absence cleanup above still runs only on a+non-empty SMART list, and such a report neither counts toward nor restarts a+disk's three, so a disk that leaves the inventory while every report's SMART+list stays empty keeps its alert until the 24-hour stale-alert sweep.+`TestCheckHostJudgesTheUnraidTemperatureADiskShows` and+`TestCheckHostHoldsUnraidDiskTemperatureAlertUntilAReadingOrDeparture` in+`internal/alerts/host_unraid_lifecycle_test.go` pin the fallback, the+inventory-only disk, the empty SMART list, two disks on one device (also of+different types), a controller member under another device label, standby,+exclusion, the disk type+and host override thresholds, the holds, the handover to a SMART reading,+departure, and the exclusion clear with and without SMART rows and beside a+listed SMART row. `TestJudgedHostDiskTemperatureOutranks` pins the ranking.+### Configured flapping thresholds remain reachableEvery accepted positive `FlappingThreshold`, including values above ten, mustdiff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.mdindex 826a91d7c7..ecd57dc3dc 100644--- a/docs/release-control/v6/internal/subsystems/unified-resources.md+++ b/docs/release-control/v6/internal/subsystems/unified-resources.md@@ -5452,6 +5452,23 @@ own reading, so a temperature shown from the Unraid inventory, for a disk whoseSMART probe returned none or one excluded from SMART collection, was nevercharted. Proof: `TestAgentDiskChartsTheUnraidTemperatureItShows` and`TestHostUnraidDiskMetricIDMatchesItsDiskMetricsTarget`.+`HostDiskTemperatureReadings` lists the reading each disk of a host agent shows+for consumers that judge disk heat, the agent disk temperature alerts in+`CheckHost`: one per SMART row the registry ingests (no virtual block device),+then one per Unraid row whose disk key (`HostUnraidDiskMetricID`) no SMART row's+`HostSMARTDiskMetricID` equals, the history writer's rule, with a row without a+device skipped as the registry skips it. Rows with one key are one registry+disk whatever device labels they carry, such as a controller member+(`0 [megaraid,0]`) and its Unraid device (`sda`). When smartctl and Unraid+report different usable serials for one device the registry shows both disks,+and both are listed. Each reading is taken from the disk resource+`resourceFromHostSMARTDisk` or `resourceFromHostUnraidPhysicalDisk` builds, so+its temperature, collection state and disk type are the ones that row's disk+shows. Unraid rows on different devices that share a key and no SMART row are+one registry disk but keep a reading each, since alerts are keyed by device.+Proof: `TestHostDiskTemperatureReadingsMatchTheDisksTheRegistryShows` compares+each reading with the ingested disk, including two disks on one device and a+controller member Unraid lists under its block device.That same canonical physical-disk view must also expose source-independent hostcontext. When a disk is API-backed rather than node-backed, typed views shouldfall back to canonical host identity such as `identity.hostnames` instead ofdiff --git a/internal/alerts/host.go b/internal/alerts/host.goindex 65ae1cbd40..00d3b3ed9d 100644--- a/internal/alerts/host.go+++ b/internal/alerts/host.go@@ -10,6 +10,7 @@ import ("github.com/rcourtman/pulse-go-rewrite/internal/models""github.com/rcourtman/pulse-go-rewrite/internal/storagehealth""github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"+ "github.com/rcourtman/pulse-go-rewrite/pkg/fsfilters""github.com/rs/zerolog/log")@@ -378,43 +379,79 @@ func (m *Manager) CheckHost(host models.Host) {}if thresholds.DiskTemperature != nil && thresholds.DiskTemperature.Trigger > 0 {+ // Each disk is judged on the temperature it shows: its SMART row's+ // reading, or the one the host's Unraid inventory reports for it when+ // the agent's probe returned none, and the inventory's reading for a+ // disk only the inventory lists.+ readings := unifiedresources.HostDiskTemperatureReadings(host)+ seenDiskTemps := make(map[string]struct{}, len(readings))+ excludedDiskTemps := make(map[string]struct{})+ judgedDiskTemps := make(map[string]judgedHostDiskTemperature, len(readings))+ judgedOrder := make([]string, 0, len(readings))+ for _, disk := range readings {+ tempResourceID := hostDiskTemperatureResourceID(host.ID, disk.Device)+ // A disk the agent's --disk-exclude patterns match is out of+ // monitoring, so the Unraid inventory's reading raises nothing+ // for it either.+ if disk.UnraidOnly && fsfilters.MatchesDeviceExclude(disk.Device, host.DiskExclude) {+ excludedDiskTemps[tempResourceID] = struct{}{}+ continue+ }+ // A listed disk in standby, or without a current temperature after+ // a failed probe, is still present. Its alert holds until a fresh+ // reading resolves it instead of clearing and re-raising.+ seenDiskTemps[tempResourceID] = struct{}{}+ if !disk.Collected() {+ continue+ }+ m.mu.RLock()+ threshold := m.hostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature, diskTempOverridden, disk.DiskType)+ m.mu.RUnlock()+ // Sources that disagree on a disk's identity can show two disks+ // on one device. Its alert judges the reading that stands highest+ // against its own disk type's threshold.+ candidate := judgedHostDiskTemperature{reading: disk, threshold: threshold}+ if judged, ok := judgedDiskTemps[tempResourceID]; !ok {+ judgedOrder = append(judgedOrder, tempResourceID)+ } else if !candidate.outranks(judged) {+ continue+ }+ judgedDiskTemps[tempResourceID] = candidate+ }+ for _, tempResourceID := range judgedOrder {+ disk := judgedDiskTemps[tempResourceID].reading+ effectiveTempThreshold := judgedDiskTemps[tempResourceID].threshold++ tempResourceName := fmt.Sprintf("%s (%s Temp)", hostDisplayName(host), disk.Device)++ diskTempMetadata := hostChildAlertMetadata(baseMetadata, hostDiskAlertResourceType)+ diskTempMetadata["metric"] = "diskTemperature"+ diskTempMetadata["device"] = disk.Device+ diskTempMetadata["temperature"] = disk.Temperature+ diskTempMetadata["model"] = disk.Model+ diskTempMetadata["diskType"] = disk.DiskType+ spec, err := buildCanonicalMetricSpec(tempResourceID, tempResourceName, unifiedresources.ResourceType("agent-disk"), "diskTemperature", effectiveTempThreshold)+ if err != nil {+ log.Warn().+ Err(err).+ Str("resourceID", tempResourceID).+ Str("host", resourceName).+ Str("device", disk.Device).+ Msg("Skipping invalid canonical host disk temperature metric spec")+ continue+ }++ m.checkMetricWithCanonicalSpec(spec, tempResourceName, nodeName, disk.Device, "agent", float64(disk.Temperature), effectiveTempThreshold, &metricOptions{Metadata: diskTempMetadata})+ }+ // An excluded disk's alert from before the exclusion resolves now,+ // whatever the SMART list holds, unless another listed disk shares it.+ for tempResourceID := range seenDiskTemps {+ delete(excludedDiskTemps, tempResourceID)+ }+ m.clearHostDiskTemperatureResources(excludedDiskTemps)// An empty SMART list means collection failed or is unsupported for// this report, not that every disk left, so existing alerts are held.if len(host.Sensors.SMART) > 0 {- seenDiskTemps := make(map[string]struct{}, len(host.Sensors.SMART))- for _, disk := range host.Sensors.SMART {- // A listed disk in standby, or without a temperature after a- // failed probe, is still present. Its alert holds until a fresh- // reading resolves it instead of clearing and re-raising.- tempResourceID := hostDiskTemperatureResourceID(host.ID, disk.Device)- seenDiskTemps[tempResourceID] = struct{}{}- if disk.Temperature > 0 && !disk.Standby {- m.mu.RLock()- effectiveTempThreshold := m.hostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature, diskTempOverridden, disk.Type)- m.mu.RUnlock()-- tempResourceName := fmt.Sprintf("%s (%s Temp)", hostDisplayName(host), disk.Device)-- diskTempMetadata := hostChildAlertMetadata(baseMetadata, hostDiskAlertResourceType)- diskTempMetadata["metric"] = "diskTemperature"- diskTempMetadata["device"] = disk.Device- diskTempMetadata["temperature"] = disk.Temperature- diskTempMetadata["model"] = disk.Model- diskTempMetadata["diskType"] = disk.Type- spec, err := buildCanonicalMetricSpec(tempResourceID, tempResourceName, unifiedresources.ResourceType("agent-disk"), "diskTemperature", effectiveTempThreshold)- if err != nil {- log.Warn().- Err(err).- Str("resourceID", tempResourceID).- Str("host", resourceName).- Str("device", disk.Device).- Msg("Skipping invalid canonical host disk temperature metric spec")- continue- }-- m.checkMetricWithCanonicalSpec(spec, tempResourceName, nodeName, disk.Device, "agent", float64(disk.Temperature), effectiveTempThreshold, &metricOptions{Metadata: diskTempMetadata})- }- }// A disk missing from consecutive non-empty reports was removed,// replaced or renamed, so no later reading will resolve its alert.m.cleanupHostDiskTemperatureAlerts(host.ID, seenDiskTemps)diff --git a/internal/alerts/host_disk_temperature_readings.go b/internal/alerts/host_disk_temperature_readings.gonew file mode 100644index 0000000000..f1e997d8fc--- /dev/null+++ b/internal/alerts/host_disk_temperature_readings.go@@ -0,0 +1,72 @@+package alerts++import "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"++// clearHostDiskTemperatureResources resolves the disk temperature alerts+// raised under the given resource IDs and drops their pending threshold runs.+// CheckHost calls it for disks its host still lists but the agent's+// --disk-exclude patterns took out of monitoring, so no reading will resolve+// them.+func (m *Manager) clearHostDiskTemperatureResources(resourceIDs map[string]struct{}) {+ if len(resourceIDs) == 0 {+ return+ }++ m.mu.Lock()+ defer m.mu.Unlock()++ var storageKeys []string+ for storageKey, alert := range m.activeAlerts {+ if alert == nil {+ continue+ }+ if _, ok := resourceIDs[alert.ResourceID]; ok {+ storageKeys = append(storageKeys, storageKey)+ }+ }+ for resourceID := range resourceIDs {+ m.core.DropPendingForResource(resourceID)+ }+ for _, storageKey := range storageKeys {+ m.clearAlertNoLock(storageKey)+ }+}++// judgedHostDiskTemperature is a disk temperature reading with the threshold+// its disk type resolves to under the host's policy.+type judgedHostDiskTemperature struct {+ reading unifiedresources.HostDiskTemperatureReading+ threshold *HysteresisThreshold+}++// standing ranks the reading against its own threshold: 2 at or above the+// trigger, 1 inside the recovery band, where an open alert holds, 0 below it,+// and -1 when the threshold is off. margin is how far the reading sits above+// the trigger.+func (j judgedHostDiskTemperature) standing() (rank int, margin float64) {+ threshold := j.threshold+ if threshold == nil || threshold.Trigger <= 0 {+ return -1, 0+ }+ temperature := float64(j.reading.Temperature)+ margin = temperature - threshold.Trigger+ switch {+ case temperature >= threshold.Trigger:+ return 2, margin+ case threshold.Clear > 0 && threshold.Clear < threshold.Trigger && temperature > threshold.Clear:+ return 1, margin+ default:+ return 0, margin+ }+}++// outranks reports whether j should be judged instead of other for their+// shared alert. A tie keeps the reading listed first.+func (j judgedHostDiskTemperature) outranks(other judgedHostDiskTemperature) bool {+ rank, margin := j.standing()+ otherRank, otherMargin := other.standing()+ if rank != otherRank {+ return rank > otherRank+ }+ return margin > otherMargin+}diff --git a/internal/alerts/host_unraid_lifecycle_test.go b/internal/alerts/host_unraid_lifecycle_test.goindex 1dd52d59b6..a3c217fb20 100644--- a/internal/alerts/host_unraid_lifecycle_test.go+++ b/internal/alerts/host_unraid_lifecycle_test.go@@ -7,6 +7,7 @@ import ("time""github.com/rcourtman/pulse-go-rewrite/internal/models"+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources")func TestHostCustomSensorAlertLifecycle(t *testing.T) {@@ -546,6 +547,303 @@ func TestConfigSaveKeepsAlertTheEvaluatorKeepsWithoutRecoveryBand(t *testing.T)}}+const unraidTempHostID = "host-unraid-temp"++// unraidTempHost is an Unraid agent report with the given SMART rows and+// Unraid inventory rows.+func unraidTempHost(smart []models.HostDiskSMART, unraid ...models.HostUnraidDisk) models.Host {+ host := hostWithSMARTDiskTemp(unraidTempHostID, "sata", 40)+ host.Sensors.SMART = smart+ host.Unraid = &models.HostUnraidStorage{ArrayStarted: true, ArrayState: "STARTED", Disks: unraid}+ return host+}++func unraidTempAlertID(device string) string {+ return canonicalMetricStateID(hostDiskTemperatureResourceID(unraidTempHostID, device), "diskTemperature")+}++func TestCheckHostJudgesTheUnraidTemperatureADiskShows(t *testing.T) {+ cool := models.HostDiskSMART{Device: "sda", Model: "cool-disk", Serial: "COOL-1", Type: "sata", Temperature: 40}+ coolUnraid := models.HostUnraidDisk{Name: "disk1", Device: "/dev/sda", Serial: "COOL-1", Transport: "sata", Temperature: 40}+ hot := models.HostUnraidDisk{Name: "disk2", Device: "/dev/sdc", Model: "hot-disk", Serial: "HOT-1", Transport: "sata", Temperature: 60}++ cases := []struct {+ name string+ host func() models.Host+ configure func(m *Manager)+ device string+ wantAlert bool+ // quiet lists devices that must raise nothing.+ quiet []string+ }{+ {+ // smartctl returned no temperature, and the disk shows Unraid's.+ name: "SMART row without a reading",+ host: func() models.Host {+ return unraidTempHost([]models.HostDiskSMART{+ {Device: "sdc", Model: "hot-disk", Serial: "HOT-1", Type: "sata"},+ }, hot)+ },+ device: "sdc",+ wantAlert: true,+ },+ {+ name: "disk only the Unraid inventory lists",+ host: func() models.Host {+ return unraidTempHost([]models.HostDiskSMART{cool}, coolUnraid, hot)+ },+ device: "sdc",+ wantAlert: true,+ },+ {+ name: "Unraid disk while the SMART list is empty",+ host: func() models.Host {+ return unraidTempHost(nil, coolUnraid, hot)+ },+ device: "sdc",+ wantAlert: true,+ },+ {+ // smartctl and Unraid name the device with different serials, so+ // two disks show on it; its alert judges the hotter reading.+ name: "two disks shown on one device",+ host: func() models.Host {+ return unraidTempHost([]models.HostDiskSMART{+ cool,+ {Device: "sdc", Model: "bridge", Serial: "BRIDGE-9", Type: "sata", Temperature: 41},+ }, coolUnraid, hot)+ },+ device: "sdc",+ wantAlert: true,+ },+ {+ // 64C is under the sas trigger (65); 60C is over the sata one (55).+ name: "two disks of different types on one device",+ host: func() models.Host {+ return unraidTempHost([]models.HostDiskSMART{+ cool,+ {Device: "sdc", Model: "bridge", Serial: "BRIDGE-9", Type: "sas", Temperature: 64},+ }, coolUnraid, hot)+ },+ device: "sdc",+ wantAlert: true,+ },+ {+ // A controller member and its Unraid device are one registry+ // disk, alerted once under the member's own resource.+ name: "controller member under another device label",+ host: func() models.Host {+ member := models.HostDiskSMART{Device: "0 [megaraid,0]", Model: "hot-disk", Serial: "HOT-1", Type: "sata", Controller: "0", Target: "megaraid,0", Temperature: 60}+ return unraidTempHost([]models.HostDiskSMART{cool, member}, coolUnraid, hot)+ },+ device: "0 [megaraid,0]",+ wantAlert: true,+ quiet: []string{"sdc"},+ },+ {+ name: "spun-down Unraid disk",+ host: func() models.Host {+ spunDown := hot+ spunDown.SpunDown = true+ return unraidTempHost([]models.HostDiskSMART{cool}, coolUnraid, spunDown)+ },+ device: "sdc",+ },+ {+ name: "Unraid disk the agent excludes",+ host: func() models.Host {+ host := unraidTempHost([]models.HostDiskSMART{cool}, coolUnraid, hot)+ host.DiskExclude = []string{"/dev/sdc"}+ return host+ },+ device: "sdc",+ },+ {+ // 65C is under the nvme trigger (70) and over the agent default (55).+ name: "Unraid disk judged by its disk type",+ host: func() models.Host {+ nvme := hot+ nvme.Transport = "nvme"+ nvme.Temperature = 65+ return unraidTempHost([]models.HostDiskSMART{cool}, coolUnraid, nvme)+ },+ device: "sdc",+ },+ {+ name: "Unraid disk under a host override",+ host: func() models.Host {+ return unraidTempHost([]models.HostDiskSMART{cool}, coolUnraid, hot)+ },+ configure: func(m *Manager) {+ m.config.Overrides[unraidTempHostID] = ThresholdConfig{DiskTemperature: &HysteresisThreshold{Trigger: 65, Clear: 60}}+ },+ device: "sdc",+ },+ }++ for _, tc := range cases {+ t.Run(tc.name, func(t *testing.T) {+ m := configureDiskTempTypeHostManager(t)+ if tc.configure != nil {+ m.mu.Lock()+ tc.configure(m)+ m.mu.Unlock()+ }+ m.CheckHost(tc.host())++ alert, exists := testLookupActiveAlert(t, m, unraidTempAlertID(tc.device))+ if exists != tc.wantAlert {+ t.Fatalf("disk temperature alert for %s active = %v, want %v; active: %v", tc.device, exists, tc.wantAlert, alertKeys(m))+ }+ for _, device := range append([]string{"sda"}, tc.quiet...) {+ if _, exists := testLookupActiveAlert(t, m, unraidTempAlertID(device)); exists {+ t.Fatalf("%s raised a temperature alert, active: %v", device, alertKeys(m))+ }+ }+ if !tc.wantAlert {+ return+ }+ if alert.Value != 60 || alert.Metadata["temperature"] != 60 || alert.Metadata["diskType"] != "sata" || alert.Metadata["model"] != "hot-disk" {+ t.Fatalf("alert value/metadata = %v/%v, want the 60C sata reading of hot-disk", alert.Value, alert.Metadata)+ }+ })+ }+}++func TestCheckHostHoldsUnraidDiskTemperatureAlertUntilAReadingOrDeparture(t *testing.T) {+ cool := models.HostDiskSMART{Device: "sda", Model: "cool-disk", Serial: "COOL-1", Type: "sata", Temperature: 40}+ hot := models.HostUnraidDisk{Name: "disk2", Device: "/dev/sdc", Model: "hot-disk", Serial: "HOT-1", Transport: "sata", Temperature: 60}+ alertID := unraidTempAlertID("sdc")+ // CheckHost clears a disk's alert once three non-empty reports in a row+ // omit it.+ const absentReports = 3+ active := func(m *Manager) bool {+ _, exists := testLookupActiveAlert(t, m, alertID)+ return exists+ }+ hotHost := func(t *testing.T) *Manager {+ m := configureDiskTempTypeHostManager(t)+ m.CheckHost(unraidTempHost([]models.HostDiskSMART{cool}, hot))+ if !active(m) {+ t.Fatalf("expected the Unraid reading to raise a disk temperature alert, active: %v", alertKeys(m))+ }+ return m+ }++ t.Run("listed without a reading", func(t *testing.T) {+ m := hotHost(t)+ spunDown := hot+ spunDown.SpunDown = true+ for i := 0; i < 4; i++ {+ m.CheckHost(unraidTempHost([]models.HostDiskSMART{cool}, spunDown))+ }+ if !active(m) {+ t.Fatalf("a spun-down disk still in the inventory lost its alert, active: %v", alertKeys(m))+ }+ })++ t.Run("cooled reading resolves it", func(t *testing.T) {+ m := hotHost(t)+ cooled := hot+ cooled.Temperature = 45+ m.CheckHost(unraidTempHost([]models.HostDiskSMART{cool}, cooled))+ if active(m) {+ t.Fatalf("a 45C Unraid reading (clear 50) left the alert active: %v", alertKeys(m))+ }+ })++ t.Run("smartctl reading takes over the same alert", func(t *testing.T) {+ m := hotHost(t)+ withSMART := []models.HostDiskSMART{cool, {Device: "sdc", Model: "hot-disk", Serial: "HOT-1", Type: "sata", Temperature: 62}}+ m.CheckHost(unraidTempHost(withSMART, hot))+ if !active(m) {+ t.Fatalf("the disk's SMART row did not carry its Unraid-raised alert, active: %v", alertKeys(m))+ }+ withSMART[1].Temperature = 45+ m.CheckHost(unraidTempHost(withSMART, hot))+ if active(m) {+ t.Fatalf("a 45C SMART reading left the alert active: %v", alertKeys(m))+ }+ })++ t.Run("left the inventory", func(t *testing.T) {+ m := hotHost(t)+ for i := 1; i < absentReports; i++ {+ m.CheckHost(unraidTempHost([]models.HostDiskSMART{cool}))+ if !active(m) {+ t.Fatalf("alert cleared after %d report(s) without the disk, want %d", i, absentReports)+ }+ }+ m.CheckHost(unraidTempHost([]models.HostDiskSMART{cool}))+ if active(m) {+ t.Fatalf("alert outlived a disk gone from %d reports, active: %v", absentReports, alertKeys(m))+ }+ })++ t.Run("excluded row on a device a SMART row lists", func(t *testing.T) {+ m := hotHost(t)+ excluded := unraidTempHost([]models.HostDiskSMART{+ cool,+ {Device: "sdc", Model: "bridge", Serial: "BRIDGE-9", Type: "sata", Temperature: 62},+ }, hot)+ excluded.DiskExclude = []string{"sdc"}+ m.CheckHost(excluded)+ if !active(m) {+ t.Fatalf("the exclusion cleared an alert the device's SMART row still raises, active: %v", alertKeys(m))+ }+ })++ // The disk is still listed but out of monitoring, so its alert resolves+ // on the first report, even one with an empty SMART list.+ for name, smart := range map[string][]models.HostDiskSMART{+ "excluded after it fired": {cool},+ "excluded after it fired, SMART list empty": nil,+ } {+ t.Run(name, func(t *testing.T) {+ m := hotHost(t)+ excluded := unraidTempHost(smart, hot)+ excluded.DiskExclude = []string{"sdc"}+ m.CheckHost(excluded)+ if active(m) {+ t.Fatalf("alert outlived the disk's exclusion from monitoring, active: %v", alertKeys(m))+ }+ })+ }+}++func TestJudgedHostDiskTemperatureOutranks(t *testing.T) {+ sata := &HysteresisThreshold{Trigger: 55, Clear: 50}+ sas := &HysteresisThreshold{Trigger: 65, Clear: 60}+ noBand := &HysteresisThreshold{Trigger: 55, Clear: 55}+ off := &HysteresisThreshold{Trigger: 0}+ judged := func(temperature int, threshold *HysteresisThreshold) judgedHostDiskTemperature {+ return judgedHostDiskTemperature{+ reading: unifiedresources.HostDiskTemperatureReading{Temperature: temperature},+ threshold: threshold,+ }+ }++ cases := []struct {+ name string+ first, second judgedHostDiskTemperature+ secondWins bool+ }{+ {name: "over its trigger beats a hotter reading under its own", first: judged(64, sas), second: judged(56, sata), secondWins: true},+ {name: "in its recovery band beats one below its clear", first: judged(49, sata), second: judged(62, sas), secondWins: true},+ {name: "no band means under the trigger is below it", first: judged(54, noBand), second: judged(51, sata), secondWins: true},+ {name: "further over its trigger wins", first: judged(56, sata), second: judged(70, sas), secondWins: true},+ {name: "a threshold that is off never wins", first: judged(40, sata), second: judged(90, off)},+ {name: "a tie keeps the first", first: judged(60, sata), second: judged(60, sata)},+ }+ for _, tc := range cases {+ t.Run(tc.name, func(t *testing.T) {+ if got := tc.second.outranks(tc.first); got != tc.secondWins {+ t.Fatalf("second outranks first = %v, want %v", got, tc.secondWins)+ }+ })+ }+}+func hasAlertType(alerts []Alert, alertType string) bool {for _, alert := range alerts {if alert.Type == alertType {diff --git a/internal/unifiedresources/host_disk_temperatures.go b/internal/unifiedresources/host_disk_temperatures.gonew file mode 100644index 0000000000..12c34d802a--- /dev/null+++ b/internal/unifiedresources/host_disk_temperatures.go@@ -0,0 +1,97 @@+package unifiedresources++import (+ "github.com/rcourtman/pulse-go-rewrite/internal/models"+ "github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"+ "github.com/rcourtman/pulse-go-rewrite/pkg/fsfilters"+)++// HostDiskTemperatureReading is the temperature one physical disk of a host+// agent shows, taken from the disk resource the registry builds for it.+type HostDiskTemperatureReading struct {+ // Device names the disk on the host: a SMART row's device label, or the+ // kernel block device token of an Unraid inventory row.+ Device string+ // MetricID is the disk's history key (HostSMARTDiskMetricID or+ // HostUnraidDiskMetricID).+ MetricID string+ Model string+ DiskType string+ Temperature int+ Collection *diskinventory.CollectionStatus+ // Standby reports a SMART row the agent marked as in standby.+ Standby bool+ // UnraidOnly reports a disk built from the host's Unraid inventory row+ // alone, because no SMART row of the host carries its disk key.+ UnraidOnly bool+}++// Collected reports whether the reading is a temperature collected now: not+// a disk in standby, and not a reading its collection state marks as kept+// from earlier (diskinventory.TemperatureCollected).+func (r HostDiskTemperatureReading) Collected() bool {+ return !r.Standby && diskinventory.TemperatureCollected(r.Temperature, r.Collection)+}++// HostDiskTemperatureReadings returns the temperature each physical disk of a+// host agent shows, in report order: one reading per SMART row the registry+// ingests, then one per Unraid inventory row whose disk key no SMART row of+// the host carries. The registry merges a SMART row and an Unraid row with+// one key into one disk whatever device labels they use, as a controller+// member ("0 [megaraid,0]") and its Unraid device ("sda") do. A SMART row+// without a reading of its own shows the one its Unraid row reports+// (HostSMARTDiskTemperature), and a disk only the Unraid inventory lists, such+// as a --disk-exclude match, shows that row's (HostUnraidDiskTemperature).+// When smartctl and Unraid report different usable serials for one device the+// registry shows both disks, so both are listed. The readings come from the+// adapters the registry builds the disks with, so a consumer judging them+// judges the temperature, collection state and disk type each row's disk+// shows.+func HostDiskTemperatureReadings(host models.Host) []HostDiskTemperatureReading {+ readings := make([]HostDiskTemperatureReading, 0, len(host.Sensors.SMART))+ smartKeys := make(map[string]struct{}, len(host.Sensors.SMART))+ for _, disk := range host.Sensors.SMART {+ // The registry ingests no disk for a virtual block device.+ if fsfilters.IsVirtualBlockDevice(disk.Device) {+ continue+ }+ key := HostSMARTDiskMetricID(host, disk)+ if key != "" {+ smartKeys[key] = struct{}{}+ }+ resource, _ := resourceFromHostSMARTDisk(host, disk)+ reading := hostDiskTemperatureReading(disk.Device, key, resource)+ reading.Standby = disk.Standby+ readings = append(readings, reading)+ }+ if host.Unraid == nil {+ return readings+ }+ for _, disk := range host.Unraid.Disks {+ // HostUnraidDiskMetricID is empty for a row without a device, which+ // the registry ingests no disk for.+ key := HostUnraidDiskMetricID(host, disk)+ if key == "" {+ continue+ }+ if _, ok := smartKeys[key]; ok {+ continue+ }+ resource, _ := resourceFromHostUnraidPhysicalDisk(host, disk)+ reading := hostDiskTemperatureReading(normalizePhysicalDiskDeviceToken(disk.Device), key, resource)+ reading.UnraidOnly = true+ readings = append(readings, reading)+ }+ return readings+}++func hostDiskTemperatureReading(device, metricID string, resource Resource) HostDiskTemperatureReading {+ reading := HostDiskTemperatureReading{Device: device, MetricID: metricID}+ if disk := resource.PhysicalDisk; disk != nil {+ reading.Model = disk.Model+ reading.DiskType = disk.DiskType+ reading.Temperature = disk.Temperature+ reading.Collection = diskinventory.CloneStatus(disk.Collection)+ }+ return reading+}diff --git a/internal/unifiedresources/registry_test.go b/internal/unifiedresources/registry_test.goindex 53214cd891..bc0f2e3b1b 100644--- a/internal/unifiedresources/registry_test.go+++ b/internal/unifiedresources/registry_test.go@@ -2857,6 +2857,101 @@ func TestResourceRegistry_IngestSnapshotCreatesUnraidDisksWithoutSMART(t *testin}}+func TestHostDiskTemperatureReadingsMatchTheDisksTheRegistryShows(t *testing.T) {+ now := time.Date(2026, 10, 7, 12, 0, 0, 0, time.UTC)+ host := models.Host{+ ID: "host-tower",+ Hostname: "tower",+ Status: "online",+ LastSeen: now,+ Sensors: models.HostSensorSummary{+ SMART: []models.HostDiskSMART{+ // smartctl read this disk itself.+ {Device: "sda", Model: "WDC A", Serial: "SER-A", Type: "sata", Temperature: 38},+ // smartctl returned no temperature; Unraid read one.+ {Device: "sdb", Model: "WDC B", Serial: "SER-B", Type: "sata"},+ // smartctl and Unraid name this device with different serials.+ {Device: "sde", Model: "Bridge E", Serial: "SER-E1", Type: "sata", Temperature: 41},+ // The registry ingests no virtual device.+ {Device: "zram0", Temperature: 70},+ // A controller member Unraid lists under its block device.+ {Device: "0 [megaraid,0]", Model: "Seagate F", Serial: "SER-F", Type: "sata", Controller: "0", Target: "megaraid,0", Temperature: 44},+ },+ },+ Unraid: &models.HostUnraidStorage{+ ArrayStarted: true,+ ArrayState: "STARTED",+ Disks: []models.HostUnraidDisk{+ {Name: "disk1", Device: "/dev/sda", Role: "data", Status: "online", Serial: "SER-A", Transport: "sata", Temperature: 36},+ {Name: "disk2", Device: "/dev/sdb", Role: "data", Status: "online", Serial: "SER-B", Transport: "sata", Temperature: 47},+ // No SMART row, as for a --disk-exclude match.+ {Name: "disk3", Device: "/dev/nvme0n1", Role: "data", Status: "online", Model: "Samsung C", Serial: "SER-C", Transport: "nvme", Temperature: 52},+ {Name: "disk4", Device: "/dev/sdd", Role: "data", Status: "online", Model: "WDC D", Serial: "SER-D", Transport: "sata", Temperature: 30, SpunDown: true},+ {Name: "disk5", Device: "/dev/sde", Role: "data", Status: "online", Model: "WDC E", Serial: "SER-E2", Transport: "sata", Temperature: 43},+ {Name: "disk7", Device: "/dev/sdf", Role: "data", Status: "online", Model: "Seagate F", Serial: "SER-F", Transport: "sata", Temperature: 44},+ // An empty slot: the registry ingests no disk for it.+ {Name: "disk6", Role: "data", Status: "missing"},+ },+ },+ }++ type want struct {+ device string+ diskType string+ temperature int+ collected bool+ unraidOnly bool+ }+ wants := map[string]want{+ "SER-A": {device: "sda", diskType: "sata", temperature: 38, collected: true},+ "SER-B": {device: "sdb", diskType: "sata", temperature: 47, collected: true},+ "SER-E1": {device: "sde", diskType: "sata", temperature: 41, collected: true},+ "SER-F": {device: "0 [megaraid,0]", diskType: "sata", temperature: 44, collected: true},+ "SER-C": {device: "nvme0n1", diskType: "nvme", temperature: 52, collected: true, unraidOnly: true},+ "SER-D": {device: "sdd", diskType: "sata", temperature: 30, unraidOnly: true},+ "SER-E2": {device: "sde", diskType: "sata", temperature: 43, collected: true, unraidOnly: true},+ }++ readings := HostDiskTemperatureReadings(host)+ if len(readings) != len(wants) {+ t.Fatalf("expected %d readings, got %d: %+v", len(wants), len(readings), readings)+ }+ byKey := make(map[string]HostDiskTemperatureReading, len(readings))+ for _, reading := range readings {+ w, ok := wants[reading.MetricID]+ if !ok {+ t.Fatalf("unexpected reading %+v", reading)+ }+ if reading.Device != w.device || reading.DiskType != w.diskType || reading.Temperature != w.temperature ||+ reading.Collected() != w.collected || reading.UnraidOnly != w.unraidOnly {+ t.Fatalf("reading for %s = %+v (collected %v), want %+v", reading.MetricID, reading, reading.Collected(), w)+ }+ byKey[reading.MetricID] = reading+ }++ // The readings are the ones the disks the registry builds show, one each.+ rr := NewRegistry(nil)+ rr.IngestSnapshot(models.StateSnapshot{Hosts: []models.Host{host}})+ disks := rr.ListByType(ResourceTypePhysicalDisk)+ if len(disks) != len(wants) {+ t.Fatalf("expected %d physical disks, got %d: %+v", len(wants), len(disks), disks)+ }+ for _, disk := range disks {+ meta := disk.PhysicalDisk+ if meta == nil {+ t.Fatalf("physical disk without meta: %+v", disk)+ }+ reading, ok := byKey[meta.Serial]+ if !ok {+ t.Fatalf("registry disk %q has no reading", meta.Serial)+ }+ if meta.Temperature != reading.Temperature || meta.DiskType != reading.DiskType ||+ diskinventory.TemperatureCollected(meta.Temperature, meta.Collection) != diskinventory.TemperatureCollected(reading.Temperature, reading.Collection) {+ t.Fatalf("registry disk %q shows %dC %q (collection %+v), reading %+v", meta.Serial, meta.Temperature, meta.DiskType, meta.Collection, reading)+ }+ }+}+func TestResourceRegistry_IngestSnapshotCreatesUnraidStorageResource(t *testing.T) {rr := NewRegistry(nil)now := time.Date(2026, 3, 7, 12, 0, 0, 0, time.UTC)