Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Merge origin/main into fix/unraid-disk-temperature-alerts

rcourtmand02d6441dd5a
diff --cc docs/release-control/v6/internal/subsystems/alerts.md
index a07724fb6b,9ed270b84b..437a8ade5d
--- a/docs/release-control/v6/internal/subsystems/alerts.md
+++ b/docs/release-control/v6/internal/subsystems/alerts.md
@@@ -1867,61 -1867,23 +1867,78 @@@ well as its alerts
`internal/alerts/host_unraid_lifecycle_test.go` pins the clear, the holds, the
restarted count and the pending-run and count cleanup.
+ ### Proxmox node cleanup keeps platform metric alerts
+
+ `CleanupAlertsForNodes` runs on every Proxmox node poll (and every mock alert
+ tick) and removes alerts whose `Node` is not a current Proxmox node or PBS
+ instance. Kubernetes, TrueNAS and vSphere metric alerts carry a cluster or
+ platform host in `Node` and a display label such as `TrueNAS Disk` or
+ `Kubernetes Pod` in `metadata.resourceType`, which the preserve list's
+ platform words never matched. The cleanup used to delete them silently on
+ each poll while their reducer incident stayed firing, so the next evaluation
+ re-created each one and handed it to the alert callback again as a new
+ alert. `shouldPreserveAlertOutsideNodeCleanup` now keeps any alert carrying
+ `metadata.platformType`, or whose label maps through
+ `CanonicalAlertResourceType` to a Kubernetes, TrueNAS or vSphere type.
+ `TestCleanupAlertsForNodesKeepsPlatformMetricAlerts` in
+ `internal/alerts/alerts_test.go` pins both signals, and its
+ sibling pins that a guest alert on a removed Proxmox node is still removed.
+
+### Agent disk temperature alerts judge the reading the disk shows
+
+`CheckHost` judges each agent disk on the temperature, collection state and
+disk type the disk shows, read by `unifiedresources.HostDiskTemperatureReadings`
+from the same adapters the registry builds the disks with. A SMART row shows its
+own reading or, when the agent's probe returned none, the one the host's Unraid
+inventory reports for that disk (`HostSMARTDiskTemperature`), and keeps its
+`agent:<host>/disk_temp:<device>` resource. An Unraid inventory row whose disk
+key no SMART row carries, a disk only the inventory lists, shows that row's
+reading (`HostUnraidDiskTemperature`) and its own transport as disk type, under
+its kernel block device token, the resource a plain Linux SMART row for the
+same device uses. A SMART row and an Unraid row with one key are one registry
+disk even under different device labels, such as a controller member
+(`0 [megaraid,0]`) and its Unraid device (`sda`), and alert once, under the
+SMART row's resource. When smartctl and Unraid name
+one device with different serials the registry shows two disks on it, and the
+device's one alert judges the reading that stands highest against its own disk
+type's threshold (`judgedHostDiskTemperature`): over its trigger first, then
+inside its recovery band, then the larger margin over the trigger, with a tie
+keeping the reading listed first.
+Before this `CheckHost` read only the SMART row's own field, while the Physical
+Disks Temp cell, the Running Hot verdict ("at or above its alert threshold")
+and Patrol judged the Unraid reading against the same trigger, so such a disk
+could read as at its alert threshold with no alert. Unraid's own warning and
+critical disk temperature notifications do not make this redundant: Pulse
+already alerts on the smartctl readings and array state of the same hosts,
+which Unraid also notifies on, and a user who routes alerts through Pulse sets
+these thresholds here.
+
+Only a reading collected now fires or resolves
+(`HostDiskTemperatureReading.Collected`: no standby row, and
+`diskinventory.TemperatureCollected`), so a spun-down disk holds the alert
+like a listed disk without a temperature. A disk only the
+inventory lists whose device matches the agent's `--disk-exclude` patterns
+(`fsfilters.MatchesDeviceExclude`) raises nothing, and an alert it raised before
+the exclusion resolves on the next report (`clearHostDiskTemperatureResources`),
+whatever the SMART list holds, unless another listed reading uses the same
+resource: `docs/UNIFIED_AGENT.md` documents exclusion as
+removing the disk from monitoring, and linked Proxmox hosts already judge an
+excluded disk healthy. A disk only the inventory lists is also judged in a
+report whose SMART list is empty, because the disk shows the inventory's
+reading in that report. The absence cleanup above still runs only on a
+non-empty SMART list, and such a report neither counts toward nor restarts a
+disk's three, so a disk that leaves the inventory while every report's SMART
+list stays empty keeps its alert until the 24-hour stale-alert sweep.
+`TestCheckHostJudgesTheUnraidTemperatureADiskShows` and
+`TestCheckHostHoldsUnraidDiskTemperatureAlertUntilAReadingOrDeparture` in
+`internal/alerts/host_unraid_lifecycle_test.go` pin the fallback, the
+inventory-only disk, the empty SMART list, two disks on one device (also of
+different types), a controller member under another device label, standby,
+exclusion, the disk type
+and host override thresholds, the holds, the handover to a SMART reading,
+departure, and the exclusion clear with and without SMART rows and beside a
+listed SMART row. `TestJudgedHostDiskTemperatureOutranks` pins the ranking.
+
### Configured flapping thresholds remain reachable
Every accepted positive `FlappingThreshold`, including values above ten, must