Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Alert on the Unraid temperature an agent disk shows

rcourtmanac631eadb87f
diff --git a/docs/release-control/v6/internal/subsystems/alerts.md b/docs/release-control/v6/internal/subsystems/alerts.md
index 9d7946179e..a07724fb6b 100644
--- a/docs/release-control/v6/internal/subsystems/alerts.md
+++ b/docs/release-control/v6/internal/subsystems/alerts.md
@@ -1867,6 +1867,61 @@ well as its alerts.
`internal/alerts/host_unraid_lifecycle_test.go` pins the clear, the holds, the
restarted count and the pending-run and count cleanup.
+### Agent disk temperature alerts judge the reading the disk shows
+
+`CheckHost` judges each agent disk on the temperature, collection state and
+disk type the disk shows, read by `unifiedresources.HostDiskTemperatureReadings`
+from the same adapters the registry builds the disks with. A SMART row shows its
+own reading or, when the agent's probe returned none, the one the host's Unraid
+inventory reports for that disk (`HostSMARTDiskTemperature`), and keeps its
+`agent:<host>/disk_temp:<device>` resource. An Unraid inventory row whose disk
+key no SMART row carries, a disk only the inventory lists, shows that row's
+reading (`HostUnraidDiskTemperature`) and its own transport as disk type, under
+its kernel block device token, the resource a plain Linux SMART row for the
+same device uses. A SMART row and an Unraid row with one key are one registry
+disk even under different device labels, such as a controller member
+(`0 [megaraid,0]`) and its Unraid device (`sda`), and alert once, under the
+SMART row's resource. When smartctl and Unraid name
+one device with different serials the registry shows two disks on it, and the
+device's one alert judges the reading that stands highest against its own disk
+type's threshold (`judgedHostDiskTemperature`): over its trigger first, then
+inside its recovery band, then the larger margin over the trigger, with a tie
+keeping the reading listed first.
+Before this `CheckHost` read only the SMART row's own field, while the Physical
+Disks Temp cell, the Running Hot verdict ("at or above its alert threshold")
+and Patrol judged the Unraid reading against the same trigger, so such a disk
+could read as at its alert threshold with no alert. Unraid's own warning and
+critical disk temperature notifications do not make this redundant: Pulse
+already alerts on the smartctl readings and array state of the same hosts,
+which Unraid also notifies on, and a user who routes alerts through Pulse sets
+these thresholds here.
+
+Only a reading collected now fires or resolves
+(`HostDiskTemperatureReading.Collected`: no standby row, and
+`diskinventory.TemperatureCollected`), so a spun-down disk holds the alert
+like a listed disk without a temperature. A disk only the
+inventory lists whose device matches the agent's `--disk-exclude` patterns
+(`fsfilters.MatchesDeviceExclude`) raises nothing, and an alert it raised before
+the exclusion resolves on the next report (`clearHostDiskTemperatureResources`),
+whatever the SMART list holds, unless another listed reading uses the same
+resource: `docs/UNIFIED_AGENT.md` documents exclusion as
+removing the disk from monitoring, and linked Proxmox hosts already judge an
+excluded disk healthy. A disk only the inventory lists is also judged in a
+report whose SMART list is empty, because the disk shows the inventory's
+reading in that report. The absence cleanup above still runs only on a
+non-empty SMART list, and such a report neither counts toward nor restarts a
+disk's three, so a disk that leaves the inventory while every report's SMART
+list stays empty keeps its alert until the 24-hour stale-alert sweep.
+`TestCheckHostJudgesTheUnraidTemperatureADiskShows` and
+`TestCheckHostHoldsUnraidDiskTemperatureAlertUntilAReadingOrDeparture` in
+`internal/alerts/host_unraid_lifecycle_test.go` pin the fallback, the
+inventory-only disk, the empty SMART list, two disks on one device (also of
+different types), a controller member under another device label, standby,
+exclusion, the disk type
+and host override thresholds, the holds, the handover to a SMART reading,
+departure, and the exclusion clear with and without SMART rows and beside a
+listed SMART row. `TestJudgedHostDiskTemperatureOutranks` pins the ranking.
+
### Configured flapping thresholds remain reachable
Every accepted positive `FlappingThreshold`, including values above ten, must
diff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.md
index 826a91d7c7..ecd57dc3dc 100644
--- a/docs/release-control/v6/internal/subsystems/unified-resources.md
+++ b/docs/release-control/v6/internal/subsystems/unified-resources.md
@@ -5452,6 +5452,23 @@ own reading, so a temperature shown from the Unraid inventory, for a disk whose
SMART probe returned none or one excluded from SMART collection, was never
charted. Proof: `TestAgentDiskChartsTheUnraidTemperatureItShows` and
`TestHostUnraidDiskMetricIDMatchesItsDiskMetricsTarget`.
+`HostDiskTemperatureReadings` lists the reading each disk of a host agent shows
+for consumers that judge disk heat, the agent disk temperature alerts in
+`CheckHost`: one per SMART row the registry ingests (no virtual block device),
+then one per Unraid row whose disk key (`HostUnraidDiskMetricID`) no SMART row's
+`HostSMARTDiskMetricID` equals, the history writer's rule, with a row without a
+device skipped as the registry skips it. Rows with one key are one registry
+disk whatever device labels they carry, such as a controller member
+(`0 [megaraid,0]`) and its Unraid device (`sda`). When smartctl and Unraid
+report different usable serials for one device the registry shows both disks,
+and both are listed. Each reading is taken from the disk resource
+`resourceFromHostSMARTDisk` or `resourceFromHostUnraidPhysicalDisk` builds, so
+its temperature, collection state and disk type are the ones that row's disk
+shows. Unraid rows on different devices that share a key and no SMART row are
+one registry disk but keep a reading each, since alerts are keyed by device.
+Proof: `TestHostDiskTemperatureReadingsMatchTheDisksTheRegistryShows` compares
+each reading with the ingested disk, including two disks on one device and a
+controller member Unraid lists under its block device.
That same canonical physical-disk view must also expose source-independent host
context. When a disk is API-backed rather than node-backed, typed views should
fall back to canonical host identity such as `identity.hostnames` instead of
diff --git a/internal/alerts/host.go b/internal/alerts/host.go
index 65ae1cbd40..00d3b3ed9d 100644
--- a/internal/alerts/host.go
+++ b/internal/alerts/host.go
@@ -10,6 +10,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/storagehealth"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/fsfilters"
"github.com/rs/zerolog/log"
)
@@ -378,43 +379,79 @@ func (m *Manager) CheckHost(host models.Host) {
}
if thresholds.DiskTemperature != nil && thresholds.DiskTemperature.Trigger > 0 {
+ // Each disk is judged on the temperature it shows: its SMART row's
+ // reading, or the one the host's Unraid inventory reports for it when
+ // the agent's probe returned none, and the inventory's reading for a
+ // disk only the inventory lists.
+ readings := unifiedresources.HostDiskTemperatureReadings(host)
+ seenDiskTemps := make(map[string]struct{}, len(readings))
+ excludedDiskTemps := make(map[string]struct{})
+ judgedDiskTemps := make(map[string]judgedHostDiskTemperature, len(readings))
+ judgedOrder := make([]string, 0, len(readings))
+ for _, disk := range readings {
+ tempResourceID := hostDiskTemperatureResourceID(host.ID, disk.Device)
+ // A disk the agent's --disk-exclude patterns match is out of
+ // monitoring, so the Unraid inventory's reading raises nothing
+ // for it either.
+ if disk.UnraidOnly && fsfilters.MatchesDeviceExclude(disk.Device, host.DiskExclude) {
+ excludedDiskTemps[tempResourceID] = struct{}{}
+ continue
+ }
+ // A listed disk in standby, or without a current temperature after
+ // a failed probe, is still present. Its alert holds until a fresh
+ // reading resolves it instead of clearing and re-raising.
+ seenDiskTemps[tempResourceID] = struct{}{}
+ if !disk.Collected() {
+ continue
+ }
+ m.mu.RLock()
+ threshold := m.hostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature, diskTempOverridden, disk.DiskType)
+ m.mu.RUnlock()
+ // Sources that disagree on a disk's identity can show two disks
+ // on one device. Its alert judges the reading that stands highest
+ // against its own disk type's threshold.
+ candidate := judgedHostDiskTemperature{reading: disk, threshold: threshold}
+ if judged, ok := judgedDiskTemps[tempResourceID]; !ok {
+ judgedOrder = append(judgedOrder, tempResourceID)
+ } else if !candidate.outranks(judged) {
+ continue
+ }
+ judgedDiskTemps[tempResourceID] = candidate
+ }
+ for _, tempResourceID := range judgedOrder {
+ disk := judgedDiskTemps[tempResourceID].reading
+ effectiveTempThreshold := judgedDiskTemps[tempResourceID].threshold
+
+ tempResourceName := fmt.Sprintf("%s (%s Temp)", hostDisplayName(host), disk.Device)
+
+ diskTempMetadata := hostChildAlertMetadata(baseMetadata, hostDiskAlertResourceType)
+ diskTempMetadata["metric"] = "diskTemperature"
+ diskTempMetadata["device"] = disk.Device
+ diskTempMetadata["temperature"] = disk.Temperature
+ diskTempMetadata["model"] = disk.Model
+ diskTempMetadata["diskType"] = disk.DiskType
+ spec, err := buildCanonicalMetricSpec(tempResourceID, tempResourceName, unifiedresources.ResourceType("agent-disk"), "diskTemperature", effectiveTempThreshold)
+ if err != nil {
+ log.Warn().
+ Err(err).
+ Str("resourceID", tempResourceID).
+ Str("host", resourceName).
+ Str("device", disk.Device).
+ Msg("Skipping invalid canonical host disk temperature metric spec")
+ continue
+ }
+
+ m.checkMetricWithCanonicalSpec(spec, tempResourceName, nodeName, disk.Device, "agent", float64(disk.Temperature), effectiveTempThreshold, &metricOptions{Metadata: diskTempMetadata})
+ }
+ // An excluded disk's alert from before the exclusion resolves now,
+ // whatever the SMART list holds, unless another listed disk shares it.
+ for tempResourceID := range seenDiskTemps {
+ delete(excludedDiskTemps, tempResourceID)
+ }
+ m.clearHostDiskTemperatureResources(excludedDiskTemps)
// An empty SMART list means collection failed or is unsupported for
// this report, not that every disk left, so existing alerts are held.
if len(host.Sensors.SMART) > 0 {
- seenDiskTemps := make(map[string]struct{}, len(host.Sensors.SMART))
- for _, disk := range host.Sensors.SMART {
- // A listed disk in standby, or without a temperature after a
- // failed probe, is still present. Its alert holds until a fresh
- // reading resolves it instead of clearing and re-raising.
- tempResourceID := hostDiskTemperatureResourceID(host.ID, disk.Device)
- seenDiskTemps[tempResourceID] = struct{}{}
- if disk.Temperature > 0 && !disk.Standby {
- m.mu.RLock()
- effectiveTempThreshold := m.hostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature, diskTempOverridden, disk.Type)
- m.mu.RUnlock()
-
- tempResourceName := fmt.Sprintf("%s (%s Temp)", hostDisplayName(host), disk.Device)
-
- diskTempMetadata := hostChildAlertMetadata(baseMetadata, hostDiskAlertResourceType)
- diskTempMetadata["metric"] = "diskTemperature"
- diskTempMetadata["device"] = disk.Device
- diskTempMetadata["temperature"] = disk.Temperature
- diskTempMetadata["model"] = disk.Model
- diskTempMetadata["diskType"] = disk.Type
- spec, err := buildCanonicalMetricSpec(tempResourceID, tempResourceName, unifiedresources.ResourceType("agent-disk"), "diskTemperature", effectiveTempThreshold)
- if err != nil {
- log.Warn().
- Err(err).
- Str("resourceID", tempResourceID).
- Str("host", resourceName).
- Str("device", disk.Device).
- Msg("Skipping invalid canonical host disk temperature metric spec")
- continue
- }
-
- m.checkMetricWithCanonicalSpec(spec, tempResourceName, nodeName, disk.Device, "agent", float64(disk.Temperature), effectiveTempThreshold, &metricOptions{Metadata: diskTempMetadata})
- }
- }
// A disk missing from consecutive non-empty reports was removed,
// replaced or renamed, so no later reading will resolve its alert.
m.cleanupHostDiskTemperatureAlerts(host.ID, seenDiskTemps)
diff --git a/internal/alerts/host_disk_temperature_readings.go b/internal/alerts/host_disk_temperature_readings.go
new file mode 100644
index 0000000000..f1e997d8fc
--- /dev/null
+++ b/internal/alerts/host_disk_temperature_readings.go
@@ -0,0 +1,72 @@
+package alerts
+
+import "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
+
+// clearHostDiskTemperatureResources resolves the disk temperature alerts
+// raised under the given resource IDs and drops their pending threshold runs.
+// CheckHost calls it for disks its host still lists but the agent's
+// --disk-exclude patterns took out of monitoring, so no reading will resolve
+// them.
+func (m *Manager) clearHostDiskTemperatureResources(resourceIDs map[string]struct{}) {
+ if len(resourceIDs) == 0 {
+ return
+ }
+
+ m.mu.Lock()
+ defer m.mu.Unlock()
+
+ var storageKeys []string
+ for storageKey, alert := range m.activeAlerts {
+ if alert == nil {
+ continue
+ }
+ if _, ok := resourceIDs[alert.ResourceID]; ok {
+ storageKeys = append(storageKeys, storageKey)
+ }
+ }
+ for resourceID := range resourceIDs {
+ m.core.DropPendingForResource(resourceID)
+ }
+ for _, storageKey := range storageKeys {
+ m.clearAlertNoLock(storageKey)
+ }
+}
+
+// judgedHostDiskTemperature is a disk temperature reading with the threshold
+// its disk type resolves to under the host's policy.
+type judgedHostDiskTemperature struct {
+ reading unifiedresources.HostDiskTemperatureReading
+ threshold *HysteresisThreshold
+}
+
+// standing ranks the reading against its own threshold: 2 at or above the
+// trigger, 1 inside the recovery band, where an open alert holds, 0 below it,
+// and -1 when the threshold is off. margin is how far the reading sits above
+// the trigger.
+func (j judgedHostDiskTemperature) standing() (rank int, margin float64) {
+ threshold := j.threshold
+ if threshold == nil || threshold.Trigger <= 0 {
+ return -1, 0
+ }
+ temperature := float64(j.reading.Temperature)
+ margin = temperature - threshold.Trigger
+ switch {
+ case temperature >= threshold.Trigger:
+ return 2, margin
+ case threshold.Clear > 0 && threshold.Clear < threshold.Trigger && temperature > threshold.Clear:
+ return 1, margin
+ default:
+ return 0, margin
+ }
+}
+
+// outranks reports whether j should be judged instead of other for their
+// shared alert. A tie keeps the reading listed first.
+func (j judgedHostDiskTemperature) outranks(other judgedHostDiskTemperature) bool {
+ rank, margin := j.standing()
+ otherRank, otherMargin := other.standing()
+ if rank != otherRank {
+ return rank > otherRank
+ }
+ return margin > otherMargin
+}
diff --git a/internal/alerts/host_unraid_lifecycle_test.go b/internal/alerts/host_unraid_lifecycle_test.go
index 1dd52d59b6..a3c217fb20 100644
--- a/internal/alerts/host_unraid_lifecycle_test.go
+++ b/internal/alerts/host_unraid_lifecycle_test.go
@@ -7,6 +7,7 @@ import (
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
)
func TestHostCustomSensorAlertLifecycle(t *testing.T) {
@@ -546,6 +547,303 @@ func TestConfigSaveKeepsAlertTheEvaluatorKeepsWithoutRecoveryBand(t *testing.T)
}
}
+const unraidTempHostID = "host-unraid-temp"
+
+// unraidTempHost is an Unraid agent report with the given SMART rows and
+// Unraid inventory rows.
+func unraidTempHost(smart []models.HostDiskSMART, unraid ...models.HostUnraidDisk) models.Host {
+ host := hostWithSMARTDiskTemp(unraidTempHostID, "sata", 40)
+ host.Sensors.SMART = smart
+ host.Unraid = &models.HostUnraidStorage{ArrayStarted: true, ArrayState: "STARTED", Disks: unraid}
+ return host
+}
+
+func unraidTempAlertID(device string) string {
+ return canonicalMetricStateID(hostDiskTemperatureResourceID(unraidTempHostID, device), "diskTemperature")
+}
+
+func TestCheckHostJudgesTheUnraidTemperatureADiskShows(t *testing.T) {
+ cool := models.HostDiskSMART{Device: "sda", Model: "cool-disk", Serial: "COOL-1", Type: "sata", Temperature: 40}
+ coolUnraid := models.HostUnraidDisk{Name: "disk1", Device: "/dev/sda", Serial: "COOL-1", Transport: "sata", Temperature: 40}
+ hot := models.HostUnraidDisk{Name: "disk2", Device: "/dev/sdc", Model: "hot-disk", Serial: "HOT-1", Transport: "sata", Temperature: 60}
+
+ cases := []struct {
+ name string
+ host func() models.Host
+ configure func(m *Manager)
+ device string
+ wantAlert bool
+ // quiet lists devices that must raise nothing.
+ quiet []string
+ }{
+ {
+ // smartctl returned no temperature, and the disk shows Unraid's.
+ name: "SMART row without a reading",
+ host: func() models.Host {
+ return unraidTempHost([]models.HostDiskSMART{
+ {Device: "sdc", Model: "hot-disk", Serial: "HOT-1", Type: "sata"},
+ }, hot)
+ },
+ device: "sdc",
+ wantAlert: true,
+ },
+ {
+ name: "disk only the Unraid inventory lists",
+ host: func() models.Host {
+ return unraidTempHost([]models.HostDiskSMART{cool}, coolUnraid, hot)
+ },
+ device: "sdc",
+ wantAlert: true,
+ },
+ {
+ name: "Unraid disk while the SMART list is empty",
+ host: func() models.Host {
+ return unraidTempHost(nil, coolUnraid, hot)
+ },
+ device: "sdc",
+ wantAlert: true,
+ },
+ {
+ // smartctl and Unraid name the device with different serials, so
+ // two disks show on it; its alert judges the hotter reading.
+ name: "two disks shown on one device",
+ host: func() models.Host {
+ return unraidTempHost([]models.HostDiskSMART{
+ cool,
+ {Device: "sdc", Model: "bridge", Serial: "BRIDGE-9", Type: "sata", Temperature: 41},
+ }, coolUnraid, hot)
+ },
+ device: "sdc",
+ wantAlert: true,
+ },
+ {
+ // 64C is under the sas trigger (65); 60C is over the sata one (55).
+ name: "two disks of different types on one device",
+ host: func() models.Host {
+ return unraidTempHost([]models.HostDiskSMART{
+ cool,
+ {Device: "sdc", Model: "bridge", Serial: "BRIDGE-9", Type: "sas", Temperature: 64},
+ }, coolUnraid, hot)
+ },
+ device: "sdc",
+ wantAlert: true,
+ },
+ {
+ // A controller member and its Unraid device are one registry
+ // disk, alerted once under the member's own resource.
+ name: "controller member under another device label",
+ host: func() models.Host {
+ member := models.HostDiskSMART{Device: "0 [megaraid,0]", Model: "hot-disk", Serial: "HOT-1", Type: "sata", Controller: "0", Target: "megaraid,0", Temperature: 60}
+ return unraidTempHost([]models.HostDiskSMART{cool, member}, coolUnraid, hot)
+ },
+ device: "0 [megaraid,0]",
+ wantAlert: true,
+ quiet: []string{"sdc"},
+ },
+ {
+ name: "spun-down Unraid disk",
+ host: func() models.Host {
+ spunDown := hot
+ spunDown.SpunDown = true
+ return unraidTempHost([]models.HostDiskSMART{cool}, coolUnraid, spunDown)
+ },
+ device: "sdc",
+ },
+ {
+ name: "Unraid disk the agent excludes",
+ host: func() models.Host {
+ host := unraidTempHost([]models.HostDiskSMART{cool}, coolUnraid, hot)
+ host.DiskExclude = []string{"/dev/sdc"}
+ return host
+ },
+ device: "sdc",
+ },
+ {
+ // 65C is under the nvme trigger (70) and over the agent default (55).
+ name: "Unraid disk judged by its disk type",
+ host: func() models.Host {
+ nvme := hot
+ nvme.Transport = "nvme"
+ nvme.Temperature = 65
+ return unraidTempHost([]models.HostDiskSMART{cool}, coolUnraid, nvme)
+ },
+ device: "sdc",
+ },
+ {
+ name: "Unraid disk under a host override",
+ host: func() models.Host {
+ return unraidTempHost([]models.HostDiskSMART{cool}, coolUnraid, hot)
+ },
+ configure: func(m *Manager) {
+ m.config.Overrides[unraidTempHostID] = ThresholdConfig{DiskTemperature: &HysteresisThreshold{Trigger: 65, Clear: 60}}
+ },
+ device: "sdc",
+ },
+ }
+
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ m := configureDiskTempTypeHostManager(t)
+ if tc.configure != nil {
+ m.mu.Lock()
+ tc.configure(m)
+ m.mu.Unlock()
+ }
+ m.CheckHost(tc.host())
+
+ alert, exists := testLookupActiveAlert(t, m, unraidTempAlertID(tc.device))
+ if exists != tc.wantAlert {
+ t.Fatalf("disk temperature alert for %s active = %v, want %v; active: %v", tc.device, exists, tc.wantAlert, alertKeys(m))
+ }
+ for _, device := range append([]string{"sda"}, tc.quiet...) {
+ if _, exists := testLookupActiveAlert(t, m, unraidTempAlertID(device)); exists {
+ t.Fatalf("%s raised a temperature alert, active: %v", device, alertKeys(m))
+ }
+ }
+ if !tc.wantAlert {
+ return
+ }
+ if alert.Value != 60 || alert.Metadata["temperature"] != 60 || alert.Metadata["diskType"] != "sata" || alert.Metadata["model"] != "hot-disk" {
+ t.Fatalf("alert value/metadata = %v/%v, want the 60C sata reading of hot-disk", alert.Value, alert.Metadata)
+ }
+ })
+ }
+}
+
+func TestCheckHostHoldsUnraidDiskTemperatureAlertUntilAReadingOrDeparture(t *testing.T) {
+ cool := models.HostDiskSMART{Device: "sda", Model: "cool-disk", Serial: "COOL-1", Type: "sata", Temperature: 40}
+ hot := models.HostUnraidDisk{Name: "disk2", Device: "/dev/sdc", Model: "hot-disk", Serial: "HOT-1", Transport: "sata", Temperature: 60}
+ alertID := unraidTempAlertID("sdc")
+ // CheckHost clears a disk's alert once three non-empty reports in a row
+ // omit it.
+ const absentReports = 3
+ active := func(m *Manager) bool {
+ _, exists := testLookupActiveAlert(t, m, alertID)
+ return exists
+ }
+ hotHost := func(t *testing.T) *Manager {
+ m := configureDiskTempTypeHostManager(t)
+ m.CheckHost(unraidTempHost([]models.HostDiskSMART{cool}, hot))
+ if !active(m) {
+ t.Fatalf("expected the Unraid reading to raise a disk temperature alert, active: %v", alertKeys(m))
+ }
+ return m
+ }
+
+ t.Run("listed without a reading", func(t *testing.T) {
+ m := hotHost(t)
+ spunDown := hot
+ spunDown.SpunDown = true
+ for i := 0; i < 4; i++ {
+ m.CheckHost(unraidTempHost([]models.HostDiskSMART{cool}, spunDown))
+ }
+ if !active(m) {
+ t.Fatalf("a spun-down disk still in the inventory lost its alert, active: %v", alertKeys(m))
+ }
+ })
+
+ t.Run("cooled reading resolves it", func(t *testing.T) {
+ m := hotHost(t)
+ cooled := hot
+ cooled.Temperature = 45
+ m.CheckHost(unraidTempHost([]models.HostDiskSMART{cool}, cooled))
+ if active(m) {
+ t.Fatalf("a 45C Unraid reading (clear 50) left the alert active: %v", alertKeys(m))
+ }
+ })
+
+ t.Run("smartctl reading takes over the same alert", func(t *testing.T) {
+ m := hotHost(t)
+ withSMART := []models.HostDiskSMART{cool, {Device: "sdc", Model: "hot-disk", Serial: "HOT-1", Type: "sata", Temperature: 62}}
+ m.CheckHost(unraidTempHost(withSMART, hot))
+ if !active(m) {
+ t.Fatalf("the disk's SMART row did not carry its Unraid-raised alert, active: %v", alertKeys(m))
+ }
+ withSMART[1].Temperature = 45
+ m.CheckHost(unraidTempHost(withSMART, hot))
+ if active(m) {
+ t.Fatalf("a 45C SMART reading left the alert active: %v", alertKeys(m))
+ }
+ })
+
+ t.Run("left the inventory", func(t *testing.T) {
+ m := hotHost(t)
+ for i := 1; i < absentReports; i++ {
+ m.CheckHost(unraidTempHost([]models.HostDiskSMART{cool}))
+ if !active(m) {
+ t.Fatalf("alert cleared after %d report(s) without the disk, want %d", i, absentReports)
+ }
+ }
+ m.CheckHost(unraidTempHost([]models.HostDiskSMART{cool}))
+ if active(m) {
+ t.Fatalf("alert outlived a disk gone from %d reports, active: %v", absentReports, alertKeys(m))
+ }
+ })
+
+ t.Run("excluded row on a device a SMART row lists", func(t *testing.T) {
+ m := hotHost(t)
+ excluded := unraidTempHost([]models.HostDiskSMART{
+ cool,
+ {Device: "sdc", Model: "bridge", Serial: "BRIDGE-9", Type: "sata", Temperature: 62},
+ }, hot)
+ excluded.DiskExclude = []string{"sdc"}
+ m.CheckHost(excluded)
+ if !active(m) {
+ t.Fatalf("the exclusion cleared an alert the device's SMART row still raises, active: %v", alertKeys(m))
+ }
+ })
+
+ // The disk is still listed but out of monitoring, so its alert resolves
+ // on the first report, even one with an empty SMART list.
+ for name, smart := range map[string][]models.HostDiskSMART{
+ "excluded after it fired": {cool},
+ "excluded after it fired, SMART list empty": nil,
+ } {
+ t.Run(name, func(t *testing.T) {
+ m := hotHost(t)
+ excluded := unraidTempHost(smart, hot)
+ excluded.DiskExclude = []string{"sdc"}
+ m.CheckHost(excluded)
+ if active(m) {
+ t.Fatalf("alert outlived the disk's exclusion from monitoring, active: %v", alertKeys(m))
+ }
+ })
+ }
+}
+
+func TestJudgedHostDiskTemperatureOutranks(t *testing.T) {
+ sata := &HysteresisThreshold{Trigger: 55, Clear: 50}
+ sas := &HysteresisThreshold{Trigger: 65, Clear: 60}
+ noBand := &HysteresisThreshold{Trigger: 55, Clear: 55}
+ off := &HysteresisThreshold{Trigger: 0}
+ judged := func(temperature int, threshold *HysteresisThreshold) judgedHostDiskTemperature {
+ return judgedHostDiskTemperature{
+ reading: unifiedresources.HostDiskTemperatureReading{Temperature: temperature},
+ threshold: threshold,
+ }
+ }
+
+ cases := []struct {
+ name string
+ first, second judgedHostDiskTemperature
+ secondWins bool
+ }{
+ {name: "over its trigger beats a hotter reading under its own", first: judged(64, sas), second: judged(56, sata), secondWins: true},
+ {name: "in its recovery band beats one below its clear", first: judged(49, sata), second: judged(62, sas), secondWins: true},
+ {name: "no band means under the trigger is below it", first: judged(54, noBand), second: judged(51, sata), secondWins: true},
+ {name: "further over its trigger wins", first: judged(56, sata), second: judged(70, sas), secondWins: true},
+ {name: "a threshold that is off never wins", first: judged(40, sata), second: judged(90, off)},
+ {name: "a tie keeps the first", first: judged(60, sata), second: judged(60, sata)},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ if got := tc.second.outranks(tc.first); got != tc.secondWins {
+ t.Fatalf("second outranks first = %v, want %v", got, tc.secondWins)
+ }
+ })
+ }
+}
+
func hasAlertType(alerts []Alert, alertType string) bool {
for _, alert := range alerts {
if alert.Type == alertType {
diff --git a/internal/unifiedresources/host_disk_temperatures.go b/internal/unifiedresources/host_disk_temperatures.go
new file mode 100644
index 0000000000..12c34d802a
--- /dev/null
+++ b/internal/unifiedresources/host_disk_temperatures.go
@@ -0,0 +1,97 @@
+package unifiedresources
+
+import (
+ "github.com/rcourtman/pulse-go-rewrite/internal/models"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/fsfilters"
+)
+
+// HostDiskTemperatureReading is the temperature one physical disk of a host
+// agent shows, taken from the disk resource the registry builds for it.
+type HostDiskTemperatureReading struct {
+ // Device names the disk on the host: a SMART row's device label, or the
+ // kernel block device token of an Unraid inventory row.
+ Device string
+ // MetricID is the disk's history key (HostSMARTDiskMetricID or
+ // HostUnraidDiskMetricID).
+ MetricID string
+ Model string
+ DiskType string
+ Temperature int
+ Collection *diskinventory.CollectionStatus
+ // Standby reports a SMART row the agent marked as in standby.
+ Standby bool
+ // UnraidOnly reports a disk built from the host's Unraid inventory row
+ // alone, because no SMART row of the host carries its disk key.
+ UnraidOnly bool
+}
+
+// Collected reports whether the reading is a temperature collected now: not
+// a disk in standby, and not a reading its collection state marks as kept
+// from earlier (diskinventory.TemperatureCollected).
+func (r HostDiskTemperatureReading) Collected() bool {
+ return !r.Standby && diskinventory.TemperatureCollected(r.Temperature, r.Collection)
+}
+
+// HostDiskTemperatureReadings returns the temperature each physical disk of a
+// host agent shows, in report order: one reading per SMART row the registry
+// ingests, then one per Unraid inventory row whose disk key no SMART row of
+// the host carries. The registry merges a SMART row and an Unraid row with
+// one key into one disk whatever device labels they use, as a controller
+// member ("0 [megaraid,0]") and its Unraid device ("sda") do. A SMART row
+// without a reading of its own shows the one its Unraid row reports
+// (HostSMARTDiskTemperature), and a disk only the Unraid inventory lists, such
+// as a --disk-exclude match, shows that row's (HostUnraidDiskTemperature).
+// When smartctl and Unraid report different usable serials for one device the
+// registry shows both disks, so both are listed. The readings come from the
+// adapters the registry builds the disks with, so a consumer judging them
+// judges the temperature, collection state and disk type each row's disk
+// shows.
+func HostDiskTemperatureReadings(host models.Host) []HostDiskTemperatureReading {
+ readings := make([]HostDiskTemperatureReading, 0, len(host.Sensors.SMART))
+ smartKeys := make(map[string]struct{}, len(host.Sensors.SMART))
+ for _, disk := range host.Sensors.SMART {
+ // The registry ingests no disk for a virtual block device.
+ if fsfilters.IsVirtualBlockDevice(disk.Device) {
+ continue
+ }
+ key := HostSMARTDiskMetricID(host, disk)
+ if key != "" {
+ smartKeys[key] = struct{}{}
+ }
+ resource, _ := resourceFromHostSMARTDisk(host, disk)
+ reading := hostDiskTemperatureReading(disk.Device, key, resource)
+ reading.Standby = disk.Standby
+ readings = append(readings, reading)
+ }
+ if host.Unraid == nil {
+ return readings
+ }
+ for _, disk := range host.Unraid.Disks {
+ // HostUnraidDiskMetricID is empty for a row without a device, which
+ // the registry ingests no disk for.
+ key := HostUnraidDiskMetricID(host, disk)
+ if key == "" {
+ continue
+ }
+ if _, ok := smartKeys[key]; ok {
+ continue
+ }
+ resource, _ := resourceFromHostUnraidPhysicalDisk(host, disk)
+ reading := hostDiskTemperatureReading(normalizePhysicalDiskDeviceToken(disk.Device), key, resource)
+ reading.UnraidOnly = true
+ readings = append(readings, reading)
+ }
+ return readings
+}
+
+func hostDiskTemperatureReading(device, metricID string, resource Resource) HostDiskTemperatureReading {
+ reading := HostDiskTemperatureReading{Device: device, MetricID: metricID}
+ if disk := resource.PhysicalDisk; disk != nil {
+ reading.Model = disk.Model
+ reading.DiskType = disk.DiskType
+ reading.Temperature = disk.Temperature
+ reading.Collection = diskinventory.CloneStatus(disk.Collection)
+ }
+ return reading
+}
diff --git a/internal/unifiedresources/registry_test.go b/internal/unifiedresources/registry_test.go
index 53214cd891..bc0f2e3b1b 100644
--- a/internal/unifiedresources/registry_test.go
+++ b/internal/unifiedresources/registry_test.go
@@ -2857,6 +2857,101 @@ func TestResourceRegistry_IngestSnapshotCreatesUnraidDisksWithoutSMART(t *testin
}
}
+func TestHostDiskTemperatureReadingsMatchTheDisksTheRegistryShows(t *testing.T) {
+ now := time.Date(2026, 10, 7, 12, 0, 0, 0, time.UTC)
+ host := models.Host{
+ ID: "host-tower",
+ Hostname: "tower",
+ Status: "online",
+ LastSeen: now,
+ Sensors: models.HostSensorSummary{
+ SMART: []models.HostDiskSMART{
+ // smartctl read this disk itself.
+ {Device: "sda", Model: "WDC A", Serial: "SER-A", Type: "sata", Temperature: 38},
+ // smartctl returned no temperature; Unraid read one.
+ {Device: "sdb", Model: "WDC B", Serial: "SER-B", Type: "sata"},
+ // smartctl and Unraid name this device with different serials.
+ {Device: "sde", Model: "Bridge E", Serial: "SER-E1", Type: "sata", Temperature: 41},
+ // The registry ingests no virtual device.
+ {Device: "zram0", Temperature: 70},
+ // A controller member Unraid lists under its block device.
+ {Device: "0 [megaraid,0]", Model: "Seagate F", Serial: "SER-F", Type: "sata", Controller: "0", Target: "megaraid,0", Temperature: 44},
+ },
+ },
+ Unraid: &models.HostUnraidStorage{
+ ArrayStarted: true,
+ ArrayState: "STARTED",
+ Disks: []models.HostUnraidDisk{
+ {Name: "disk1", Device: "/dev/sda", Role: "data", Status: "online", Serial: "SER-A", Transport: "sata", Temperature: 36},
+ {Name: "disk2", Device: "/dev/sdb", Role: "data", Status: "online", Serial: "SER-B", Transport: "sata", Temperature: 47},
+ // No SMART row, as for a --disk-exclude match.
+ {Name: "disk3", Device: "/dev/nvme0n1", Role: "data", Status: "online", Model: "Samsung C", Serial: "SER-C", Transport: "nvme", Temperature: 52},
+ {Name: "disk4", Device: "/dev/sdd", Role: "data", Status: "online", Model: "WDC D", Serial: "SER-D", Transport: "sata", Temperature: 30, SpunDown: true},
+ {Name: "disk5", Device: "/dev/sde", Role: "data", Status: "online", Model: "WDC E", Serial: "SER-E2", Transport: "sata", Temperature: 43},
+ {Name: "disk7", Device: "/dev/sdf", Role: "data", Status: "online", Model: "Seagate F", Serial: "SER-F", Transport: "sata", Temperature: 44},
+ // An empty slot: the registry ingests no disk for it.
+ {Name: "disk6", Role: "data", Status: "missing"},
+ },
+ },
+ }
+
+ type want struct {
+ device string
+ diskType string
+ temperature int
+ collected bool
+ unraidOnly bool
+ }
+ wants := map[string]want{
+ "SER-A": {device: "sda", diskType: "sata", temperature: 38, collected: true},
+ "SER-B": {device: "sdb", diskType: "sata", temperature: 47, collected: true},
+ "SER-E1": {device: "sde", diskType: "sata", temperature: 41, collected: true},
+ "SER-F": {device: "0 [megaraid,0]", diskType: "sata", temperature: 44, collected: true},
+ "SER-C": {device: "nvme0n1", diskType: "nvme", temperature: 52, collected: true, unraidOnly: true},
+ "SER-D": {device: "sdd", diskType: "sata", temperature: 30, unraidOnly: true},
+ "SER-E2": {device: "sde", diskType: "sata", temperature: 43, collected: true, unraidOnly: true},
+ }
+
+ readings := HostDiskTemperatureReadings(host)
+ if len(readings) != len(wants) {
+ t.Fatalf("expected %d readings, got %d: %+v", len(wants), len(readings), readings)
+ }
+ byKey := make(map[string]HostDiskTemperatureReading, len(readings))
+ for _, reading := range readings {
+ w, ok := wants[reading.MetricID]
+ if !ok {
+ t.Fatalf("unexpected reading %+v", reading)
+ }
+ if reading.Device != w.device || reading.DiskType != w.diskType || reading.Temperature != w.temperature ||
+ reading.Collected() != w.collected || reading.UnraidOnly != w.unraidOnly {
+ t.Fatalf("reading for %s = %+v (collected %v), want %+v", reading.MetricID, reading, reading.Collected(), w)
+ }
+ byKey[reading.MetricID] = reading
+ }
+
+ // The readings are the ones the disks the registry builds show, one each.
+ rr := NewRegistry(nil)
+ rr.IngestSnapshot(models.StateSnapshot{Hosts: []models.Host{host}})
+ disks := rr.ListByType(ResourceTypePhysicalDisk)
+ if len(disks) != len(wants) {
+ t.Fatalf("expected %d physical disks, got %d: %+v", len(wants), len(disks), disks)
+ }
+ for _, disk := range disks {
+ meta := disk.PhysicalDisk
+ if meta == nil {
+ t.Fatalf("physical disk without meta: %+v", disk)
+ }
+ reading, ok := byKey[meta.Serial]
+ if !ok {
+ t.Fatalf("registry disk %q has no reading", meta.Serial)
+ }
+ if meta.Temperature != reading.Temperature || meta.DiskType != reading.DiskType ||
+ diskinventory.TemperatureCollected(meta.Temperature, meta.Collection) != diskinventory.TemperatureCollected(reading.Temperature, reading.Collection) {
+ t.Fatalf("registry disk %q shows %dC %q (collection %+v), reading %+v", meta.Serial, meta.Temperature, meta.DiskType, meta.Collection, reading)
+ }
+ }
+}
+
func TestResourceRegistry_IngestSnapshotCreatesUnraidStorageResource(t *testing.T) {
rr := NewRegistry(nil)
now := time.Date(2026, 3, 7, 12, 0, 0, 0, time.UTC)