Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Preserve reviewed maintenance with current upstream report and identity ownership

pulse-triage[bot]1052f214195e
diff --cc docs/release-control/v6/internal/subsystems/ai-runtime.md
index 9559c01b62,bc4f13bd3f..f07697a0d0
--- a/docs/release-control/v6/internal/subsystems/ai-runtime.md
+++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md
@@@ -142,6 -128,25 +142,19 @@@ on component mount. Closing clears the
commands cannot target a detached composer. A handoff must leave Escape and
keyboard input in Assistant, not the underlying alert search.
-### Process-wide memory ID sequences
-
-`internal/ai/memory` mints incident, incident-event, remediation-record and
-context-memory IDs from package-level sequences that every store in the process
-shares. Multi-tenant monitoring gives each tenant monitor its own
-`IncidentStore`, and their alert checks call `RecordAlertFired` from separate
-goroutines, while a store's mutex covers only its own state. Each sequence is
-therefore an `atomic.Int64` minted through `nextSequencedID` in
-`internal/ai/memory/ids.go`, never a plain package counter. The ID shape stays
-`<prefix><local second>-<n>`, but `n` no longer wraps at 1000. The old
-`counter%1000` suffix repeated an ID whenever one generator minted more than a
-thousand in a wall-clock second, and incident notes resolve an incident by ID
-while timeline merges deduplicate events by ID. Within one process IDs are now
-unique by construction. Across restarts only the timestamp separates them, so a
-repeated local second (DST fall-back or a backward clock step) can still meet a
-restarted sequence's suffix. Persisted IDs and the stored format are unchanged.
-`internal/ai/memory/ids_test.go` mints from every generator concurrently and
-records alerts into two tenant stores concurrently; run it under `-race`.
++### Process-wide memory ID ownership
++
++`internal/ai/memory` preserves random UUID incident and incident-event IDs from
++the reviewed occurrence repair. They have no shared mutable counter and do not
++reuse a local timestamp after restart. Existing persisted references remain valid.
++Remediation-record and context-memory IDs use the upstream package-wide atomic
++`Int64` sequences through `nextSequencedID` in `ids.go`, without wrapping at 1000.
++Their existing `<prefix><local second>-<n>` format remains; across restarts those
++four sequences still depend on the local timestamp and retain that limitation.
++`ids_test.go` checks all six actual formats and concurrent uniqueness, plus
++independent concurrent tenant-store recording under `-race`. The reviewed
++`incidents_identity_test.go` continues to pin UUID occurrence/event ownership.
+
### Canonical incident-history queries
Incident context reads `IncidentStore.QueryIncidents`, whose page contains
diff --cc docs/release-control/v6/internal/subsystems/alerts.md
index d3c44eb5f3,513b44c27d..5f93f3738d
--- a/docs/release-control/v6/internal/subsystems/alerts.md
+++ b/docs/release-control/v6/internal/subsystems/alerts.md
@@@ -4219,103 -4184,3 +4274,114 @@@ display name differs from its hostname
had no alert highlighting at all. `getAlertStyles` keeps its
id-plus-node-name matching for the platform tables whose alerts are keyed that
way.
+
+### SMART temperature continuity and pending-only policy intervals
+
+Disk temperature activation/grace and recovery require continuous accepted
+observations. Empty SMART inventory, one disk omission, standby, non-positive
+readings, explicit missing/unavailable/unsupported collection and reporting-lease
+expiry interrupt pending/recovery timing without changing an open alert, its
+acknowledgement or lifecycle events. Positive legacy readings without temperature
+provenance remain accepted; an expired last-known positive value is not live
+health. Canonical and shadow reducers plus explicit intent grace are interrupted
+together. A non-empty inventory still resolves a departed disk only after three
+consecutive omissions; empty/expired inventory breaks that confirmation sequence.
+
+Configuration saves reset pending-only runs when global agent policy, host or
+inherited node/guest overrides, or the applicable disk-type threshold disables
+evaluation, even if no report arrives before re-enablement. Existing known-type
+default normalisation is unchanged; the zero-type save control uses a custom
+type whose zero threshold the normaliser preserves. Only live pending
+identity/link/type context is retained; unrelated saves preserve a known enabled
+run and explicit overrides retain precedence over type defaults. Restored intent
+without that context restarts conservatively on a save rather than inventing
+continuous enablement. Cleared intent grace is checkpointed. Existing active-alert
+per-type/save/offline/removal resolution remains unchanged.
+
+`metric_observation_continuity_test.go` and `host_unraid_lifecycle_test.go` cover
+valid/legacy/provenance controls, missing/rejected/expiry gaps on both timing
+edges, explicit grace, acknowledgement/event preservation, host isolation,
+configuration intervals, override precedence and departure. These deterministic
+source tests do not establish installed temperatures or native guest recovery.
+
+### SMART risk inventory gaps and disabled rules
+
+Agent `disk-health` and `disk-wearout` alerts follow the SMART inventory, not
+filesystem usage cleanup. Empty inventory, standby and unobserved risk fields
+hold the existing occurrence and acknowledgement without recovery/refiring.
+Departure requires three consecutive non-empty reports omitting the disk;
+listing it again, an empty inventory or reporting-lease expiry restarts that
+confirmation. Both risk families on the same disk count once per report.
+Filesystem absence still resolves usage on its first missing report. Temperature
+pending/grace/recovery interruption and departure semantics remain unchanged.
+
+Disabling every recorded cause releases that risk alert without a reading,
+both on a configuration save and on an empty report. Partially enabled causes,
+missing legacy cause metadata and unknown codes remain conservative. Re-enabling
+cannot raise an alert from absent evidence; fresh bad evidence uses the existing
+stateful refire/cooldown history policy. Node linking retains existing node ownership, including when the
+agent's SMART inventory is empty; host removal and policy disablement release
+agent risk state and stale absence counts. Counts are ephemeral and restart
+conservatively after a process restart.
+
+`TestHostSMARTRisk*` in `host_unraid_lifecycle_test.go` exercises actual `CheckHost`, expiry, configuration
+save, firing/recovery callbacks and event history with synthetic telemetry.
+It covers both families sharing a disk, interruption/reconfirmation, partial
+rule disablement, re-enablement, legacy/unknown causes, filesystem and host
+isolation, removal and node-link ownership. These are source lifecycle controls,
+not collector, smartctl, installed notification or native disk-recovery proof.
+
+### Storage observation gaps restart confirmation and timing
+
+An absent or `unknown` storage connectivity status interrupts activation,
+recovery confirmations and pending offline intent grace without resolving an
+existing outage or changing its acknowledgement. Capacity is independently
+observable while connectivity is unknown. Missing/unconfirmed-zero capacity,
+negative usage and offline/unavailable storage interrupt capacity activation,
+and intent grace timing; they do not supply healthy capacity evidence.
+Capacity recovery retains its existing immediate measured-clear behaviour,
+not the recovery stability window used by memory and temperature gauges.
+A confirmed empty store still supplies a genuine zero-usage recovery reading.
+Fresh observations must satisfy the existing full confirmation/delay after a
+gap. Interruption is scoped to the storage's canonical identity and applied to
+both live and shadow reducers; interrupted intent is checkpointed for restart.
+
+`TestStorageConnectivityObservationGaps` and
+`TestStorageConnectivityGapRestartsIntentGrace` exercise the real storage
+entry point, acknowledged occurrence retention, neighbour isolation, fresh
+reconfirmation and empty intent checkpoint. Storage routes in
+`TestMetricObservationGapRestartsActivation` covers capacity gaps and explicit
+intent grace; `TestStorageCapacityGapHoldsOccurrence` verifies retained,
+acknowledged capacity incidents and genuine immediate empty-store recovery. Existing restart, empty-capacity and predictive-capacity controls
+remain required. These source controls are not native storage recovery or
+installed destination-delivery evidence.
+
+`TestStorageConnectivityGapWebhook` also exercises the production Monitor
+callbacks, ordinary notification queue and guest-local HTTP receiver: the gap
+and first fresh healthy poll send no recovery; confirmed recovery sends one
+resolved receipt with the original occurrence identity, and the queue drains
+without failed/DLQ work. This is synthetic callback/receiver acceptance, not a
+native appliance or external webhook-provider result.
+
+### Composed host filesystem threshold ownership
+
+The shared filesystem resolver preserves the explicit host or linked-resource
+Disk override ahead of per-hardware-type defaults, for both host reports and
+configuration-save reevaluation. A filesystem-specific override still takes
+precedence. Disabling that filesystem releases its canonical usage metric and
+pending intent, retaining the reviewed agent/node ownership semantics.
+`TestHostDiskOverrideBeatsDiskFillByType` checks agent and inherited overrides
+through a configuration save and the next report without losing the occurrence;
+`TestConfigSaveJudgesFilesystemAlertsByTheirOwnThreshold` retains per-filesystem
+and per-type controls. This is source composition, not installed delivery proof.
++
++### Shared disk-temperature composition and pending ownership
++
++Host temperature evaluation retains the upstream shared SMART/Unraid reading and
++per-type threshold judgement, including Unraid-only disks and competing readings
++on one device. It also retains reviewed observation-gap interruption and pending
++host/link/type context for configuration saves. A live Unraid-only row is not an
++observation gap merely because SMART is empty; absent or uncollected rows still
++restart confirmation. `TestUnraidOnlyTemperatureRetainsPendingContinuity` checks
++continuous, spun-down, omitted and type-off intervals under both grace paths;
++existing SMART pending, override, departure and upstream Unraid controls remain.
diff --cc internal/ai/memory/ids_test.go
index 0000000000,530381625e..bf2138e2b3
mode 000000,100644..100644
--- a/internal/ai/memory/ids_test.go
+++ b/internal/ai/memory/ids_test.go
@@@ -1,0 -1,122 +1,126 @@@
+ package memory
+
+ import (
+ "fmt"
+ "regexp"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/alerts"
+ )
+
+ // TestSequencedIDsStayUniqueUnderConcurrentMinting mints from every
+ // package-level ID sequence on many goroutines at once. Under -race it proves
+ // the sequences are synchronized. The uniqueness check proves the suffix does
+ // not wrap: more than a thousand IDs from one generator land in the same
+ // wall-clock second, which the old counter%1000 suffix turned into duplicates.
+ func TestSequencedIDsStayUniqueUnderConcurrentMinting(t *testing.T) {
+ generators := []struct {
+ prefix string
+ mint func() string
++ uuid bool
+ }{
- {"inc-", generateIncidentID},
- {"inc-evt-", generateIncidentEventID},
- {"rem-", generateRecordID},
- {"mem-", generateMemoryID},
- {"inc-mem-", generateIncidentMemoryID},
- {"pat-mem-", generatePatternMemoryID},
++ {"inc-", generateIncidentID, true},
++ {"inc-evt-", generateIncidentEventID, true},
++ {"rem-", generateRecordID, false},
++ {"mem-", generateMemoryID, false},
++ {"inc-mem-", generateIncidentMemoryID, false},
++ {"pat-mem-", generatePatternMemoryID, false},
+ }
+ const goroutines, perGoroutine = 16, 150
+
+ for _, gen := range generators {
+ t.Run(strings.TrimSuffix(gen.prefix, "-"), func(t *testing.T) {
+ shape := regexp.MustCompile("^" + regexp.QuoteMeta(gen.prefix) + `\d{14}-[1-9]\d*$`)
++ if gen.uuid {
++ shape = regexp.MustCompile("^" + regexp.QuoteMeta(gen.prefix) + `[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$`)
++ }
+ minted := make([][]string, goroutines)
+ start := make(chan struct{})
+ var wg sync.WaitGroup
+ for g := range goroutines {
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ <-start
+ batch := make([]string, perGoroutine)
+ for i := range batch {
+ batch[i] = gen.mint()
+ }
+ minted[g] = batch
+ }()
+ }
+ close(start)
+ wg.Wait()
+
+ seen := make(map[string]bool, goroutines*perGoroutine)
+ for _, batch := range minted {
+ for _, id := range batch {
+ if !shape.MatchString(id) {
+ t.Fatalf("ID %q does not match %s", id, shape)
+ }
+ if seen[id] {
+ t.Fatalf("duplicate ID %q among %d concurrent mints", id, goroutines*perGoroutine)
+ }
+ seen[id] = true
+ }
+ }
+ })
+ }
+ }
+
+ // TestIncidentStoresMintIDsConcurrentlyAcrossTenants mirrors multi-tenant
+ // monitoring: each tenant monitor owns its own IncidentStore, and their alert
+ // checks call RecordAlertFired from separate goroutines. Each store's mutex
+ // covers only its own state, so the shared ID sequence must be synchronized on
+ // its own. Run under -race.
+ func TestIncidentStoresMintIDsConcurrentlyAcrossTenants(t *testing.T) {
+ const tenants, checkersPerTenant, alertsPerChecker = 2, 4, 50
+ stores := make([]*IncidentStore, tenants)
+ for i := range stores {
+ stores[i] = NewIncidentStore(IncidentStoreConfig{MaxIncidents: checkersPerTenant * alertsPerChecker})
+ }
+
+ startedAt := time.Now().Add(-time.Minute)
+ start := make(chan struct{})
+ var wg sync.WaitGroup
+ for tenant, store := range stores {
+ for checker := range checkersPerTenant {
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ <-start
+ for n := range alertsPerChecker {
+ store.RecordAlertFired(&alerts.Alert{
+ ID: fmt.Sprintf("tenant-%d-checker-%d-alert-%d", tenant, checker, n),
+ Type: "cpu",
+ Level: alerts.AlertLevelWarning,
+ ResourceID: fmt.Sprintf("tenant-%d-vm-%d-%d", tenant, checker, n),
+ ResourceName: "vm",
+ StartTime: startedAt,
+ })
+ }
+ }()
+ }
+ }
+ close(start)
+ wg.Wait()
+
+ seen := make(map[string]bool, tenants*checkersPerTenant*alertsPerChecker)
+ for tenant, store := range stores {
+ store.mu.RLock()
+ incidents := len(store.incidents)
+ for _, shell := range store.incidents {
+ if seen[shell.ID] {
+ t.Errorf("tenant %d incident ID %q was already minted", tenant, shell.ID)
+ }
+ seen[shell.ID] = true
+ }
+ store.mu.RUnlock()
+ if incidents != checkersPerTenant*alertsPerChecker {
+ t.Errorf("tenant %d recorded %d incidents, want %d", tenant, incidents, checkersPerTenant*alertsPerChecker)
+ }
+ }
+ }
diff --cc internal/alerts/host.go
index 781969d9c4,be3d9aa092..2d3546b5c6
--- a/internal/alerts/host.go
+++ b/internal/alerts/host.go
@@@ -429,52 -420,82 +430,93 @@@ func (m *Manager) CheckHost(host models
}
if thresholds.DiskTemperature != nil && thresholds.DiskTemperature.Trigger > 0 {
+ // Each disk is judged on the temperature it shows: its SMART row's
+ // reading, or the one the host's Unraid inventory reports for it when
+ // the agent's probe returned none, and the inventory's reading for a
+ // disk only the inventory lists.
+ readings := unifiedresources.HostDiskTemperatureReadings(host)
+ seenDiskTemps := make(map[string]struct{}, len(readings))
+ excludedDiskTemps := make(map[string]struct{})
+ judgedDiskTemps := make(map[string]judgedHostDiskTemperature, len(readings))
+ judgedOrder := make([]string, 0, len(readings))
+ for _, disk := range readings {
+ tempResourceID := hostDiskTemperatureResourceID(host.ID, disk.Device)
+ // A disk the agent's --disk-exclude patterns match is out of
+ // monitoring, so the Unraid inventory's reading raises nothing
+ // for it either.
+ if disk.UnraidOnly && fsfilters.MatchesDeviceExclude(disk.Device, host.DiskExclude) {
+ excludedDiskTemps[tempResourceID] = struct{}{}
+ continue
+ }
+ // A listed disk in standby, or without a current temperature after
+ // a failed probe, is still present. Its alert holds until a fresh
+ // reading resolves it instead of clearing and re-raising.
+ seenDiskTemps[tempResourceID] = struct{}{}
+ if !disk.Collected() {
+ continue
+ }
+ m.mu.RLock()
+ threshold := m.hostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature, diskTempOverridden, disk.DiskType)
+ m.mu.RUnlock()
+ // Sources that disagree on a disk's identity can show two disks
+ // on one device. Its alert judges the reading that stands highest
+ // against its own disk type's threshold.
+ candidate := judgedHostDiskTemperature{reading: disk, threshold: threshold}
+ if judged, ok := judgedDiskTemps[tempResourceID]; !ok {
+ judgedOrder = append(judgedOrder, tempResourceID)
+ } else if !candidate.outranks(judged) {
+ continue
+ }
+ judgedDiskTemps[tempResourceID] = candidate
+ }
+ for _, tempResourceID := range judgedOrder {
+ disk := judgedDiskTemps[tempResourceID].reading
+ effectiveTempThreshold := judgedDiskTemps[tempResourceID].threshold
+
+ tempResourceName := fmt.Sprintf("%s (%s Temp)", hostDisplayName(host), disk.Device)
+
+ diskTempMetadata := hostChildAlertMetadata(baseMetadata, hostDiskAlertResourceType)
+ diskTempMetadata["metric"] = "diskTemperature"
+ diskTempMetadata["device"] = disk.Device
+ diskTempMetadata["temperature"] = disk.Temperature
+ diskTempMetadata["model"] = disk.Model
+ diskTempMetadata["diskType"] = disk.DiskType
+ spec, err := buildCanonicalMetricSpec(tempResourceID, tempResourceName, unifiedresources.ResourceType("agent-disk"), "diskTemperature", effectiveTempThreshold)
+ if err != nil {
+ log.Warn().
+ Err(err).
+ Str("resourceID", tempResourceID).
+ Str("host", resourceName).
+ Str("device", disk.Device).
+ Msg("Skipping invalid canonical host disk temperature metric spec")
++ m.interruptHostDiskTemperatureRun(tempResourceID)
+ continue
+ }
+
+ m.checkMetricWithCanonicalSpec(spec, tempResourceName, nodeName, disk.Device, "agent", float64(disk.Temperature), effectiveTempThreshold, &metricOptions{Metadata: diskTempMetadata})
++ m.rememberHostDiskTemperaturePending(host, models.HostDiskSMART{Type: disk.DiskType}, tempResourceID)
++ }
++ // A present disk without any collected source interrupts pending timing.
++ // Another usable source for the same device keeps its run continuous.
++ for tempResourceID := range seenDiskTemps {
++ if _, observed := judgedDiskTemps[tempResourceID]; !observed {
++ m.interruptHostDiskTemperatureRun(tempResourceID)
++ }
+ }
+ // An excluded disk's alert from before the exclusion resolves now,
+ // whatever the SMART list holds, unless another listed disk shares it.
+ for tempResourceID := range seenDiskTemps {
+ delete(excludedDiskTemps, tempResourceID)
+ }
+ m.clearHostDiskTemperatureResources(excludedDiskTemps)
// An empty SMART list means collection failed or is unsupported for
// this report, not that every disk left, so existing alerts are held.
if len(host.Sensors.SMART) > 0 {
// A disk missing from consecutive non-empty reports was removed,
// replaced or renamed, so no later reading will resolve its alert.
m.cleanupHostDiskTemperatureAlerts(host.ID, seenDiskTemps)
+ } else {
- m.interruptHostDiskTemperatureRuns(host.ID)
++ m.interruptHostDiskTemperatureRunsExcept(host.ID, seenDiskTemps)
}
} else {
// Disk temperature alerting is off for this host, so no later reading
diff --cc internal/alerts/host_disk_temperature.go
index 4f7dec14ab,0000000000..b575fef4c0
mode 100644,000000..100644
--- a/internal/alerts/host_disk_temperature.go
+++ b/internal/alerts/host_disk_temperature.go
@@@ -1,135 -1,0 +1,143 @@@
+package alerts
+
+import (
+ "strings"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/alerts/reducer"
+ "github.com/rcourtman/pulse-go-rewrite/internal/models"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
+)
+
+type hostDiskTempContext struct {
+ hostID, linkedNodeID, linkedVMID, linkedContainerID, diskType string
+}
+
+// A positive legacy reading is still accepted. Explicit failed/unsupported
+// collection, including expired last-known values, is never live evidence.
+func hostDiskTemperatureObserved(disk models.HostDiskSMART) bool {
+ if disk.Temperature <= 0 || disk.Standby {
+ return false
+ }
+ return disk.Collection == nil || disk.Collection.Temperature.State == "" ||
+ disk.Collection.Temperature.State == diskinventory.FieldAvailable
+}
+
+func (m *Manager) hostDiskTemperaturePendingNoLock(resourceID string) bool {
+ for _, state := range m.mirrorStatesNoLock() {
+ if incident, ok := state.Incident(resourceID, canonicalMetricSpecID(resourceID, "diskTemperature")); ok && incident.State == reducer.StatePending {
+ return true
+ }
+ }
+ _, pending := m.intentPending[canonicalMetricStateID(resourceID, "diskTemperature")]
+ return pending
+}
+
+// Keep only the identity of a live pending run, not another telemetry cache.
+func (m *Manager) rememberHostDiskTemperaturePending(host models.Host, disk models.HostDiskSMART, resourceID string) {
+ m.mu.Lock()
+ defer m.mu.Unlock()
+ if !m.hostDiskTemperaturePendingNoLock(resourceID) {
+ delete(m.hostDiskTempPendingContexts, resourceID)
+ return
+ }
+ if m.hostDiskTempPendingContexts == nil {
+ m.hostDiskTempPendingContexts = make(map[string]hostDiskTempContext)
+ }
+ m.hostDiskTempPendingContexts[resourceID] = hostDiskTempContext{host.ID, host.LinkedNodeID, host.LinkedVMID, host.LinkedContainerID, disk.Type}
+}
+
+func (m *Manager) hostDiskTemperatureTrackedResourcesNoLock() map[string]struct{} {
+ resources := make(map[string]struct{})
+ for resourceID := range m.hostDiskTempPendingContexts {
+ resources[resourceID] = struct{}{}
+ }
+ for _, state := range m.mirrorStatesNoLock() {
+ for _, resourceID := range state.PendingResourceIDs() {
+ resources[resourceID] = struct{}{}
+ }
+ }
+ for _, alert := range m.activeAlerts {
+ if alert != nil {
+ resources[alert.ResourceID] = struct{}{}
+ }
+ }
+ for _, pending := range m.intentPending {
+ if pending.Signal == MetricAlertIntentSignal("diskTemperature") {
+ resources[pending.ResourceID] = struct{}{}
+ }
+ }
+ return resources
+}
+
+func (m *Manager) interruptHostDiskTemperatureRunNoLock(resourceID string) bool {
+ delete(m.hostDiskTempPendingContexts, resourceID)
+ return m.interruptMetricRunNoLock(resourceID, canonicalMetricSpecID(resourceID, "diskTemperature"), canonicalMetricStateID(resourceID, "diskTemperature"))
+}
+
+func (m *Manager) interruptHostDiskTemperatureRun(resourceID string) {
+ m.mu.Lock()
+ changed := m.interruptHostDiskTemperatureRunNoLock(resourceID)
+ m.mu.Unlock()
+ if changed {
+ m.saveActiveAlertsAsync("disk temperature observation gap")
+ }
+}
+
+func (m *Manager) interruptHostDiskTemperatureRuns(hostID string) {
++ m.interruptHostDiskTemperatureRunsExcept(hostID, nil)
++}
++
++// Live shared SMART/Unraid rows are not observation gaps when SMART is empty.
++func (m *Manager) interruptHostDiskTemperatureRunsExcept(hostID string, observed map[string]struct{}) {
+ m.mu.Lock()
+ changed := false
+ for resourceID := range m.hostDiskTemperatureTrackedResourcesNoLock() {
++ if _, present := observed[resourceID]; present {
++ continue
++ }
+ if strings.HasPrefix(resourceID, hostDiskTemperatureResourcePrefix(hostID)) {
+ changed = m.interruptHostDiskTemperatureRunNoLock(resourceID) || changed
+ }
+ }
+ // Empty/expired inventory is not another confirmation that a disk left.
+ for resourceID := range m.hostDiskTempAbsences {
+ if strings.HasPrefix(resourceID, hostDiskTemperatureResourcePrefix(hostID)) {
+ delete(m.hostDiskTempAbsences, resourceID)
+ }
+ }
+ m.mu.Unlock()
+ if changed {
+ m.saveActiveAlertsAsync("host disk temperature observation gap")
+ }
+}
+
+// Caller holds m.mu after config normalisation. Active alerts still use the
+// existing per-type threshold re-evaluation; this only drops pending evidence.
+func (m *Manager) reevaluateHostDiskTemperaturePendingNoLock() bool {
+ intentChanged := false
+ for resourceID := range m.hostDiskTemperatureTrackedResourcesNoLock() {
+ if !strings.HasPrefix(resourceID, "agent:") || !strings.Contains(resourceID, "/disk_temp:") || !m.hostDiskTemperaturePendingNoLock(resourceID) {
+ continue
+ }
+ ctx, known := m.hostDiskTempPendingContexts[resourceID]
+ // Restarted intent grace has no retained link/type context. Do not
+ // assume it was continuously enabled across a configuration save.
+ if !known {
+ intentChanged = m.interruptHostDiskTemperatureRunNoLock(resourceID) || intentChanged
+ continue
+ }
+ thresholds := m.resolveHostThresholdsNoLock(ctx.hostID, ctx.linkedNodeID, ctx.linkedVMID, ctx.linkedContainerID)
+ override, exists := m.hostThresholdOverrideNoLock(ctx.hostID, ctx.linkedNodeID, ctx.linkedVMID, ctx.linkedContainerID)
+ threshold := m.hostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature, exists && override.DiskTemperature != nil, ctx.diskType)
+ allDisabled, _ := m.alertPolicyTypeSwitchesNoLock("agent")
+ if !m.config.Enabled || allDisabled || thresholds.Disabled || threshold == nil || threshold.Trigger <= 0 {
+ intentChanged = m.interruptHostDiskTemperatureRunNoLock(resourceID) || intentChanged
+ }
+ }
+ for resourceID := range m.hostDiskTempPendingContexts {
+ if !m.hostDiskTemperaturePendingNoLock(resourceID) {
+ delete(m.hostDiskTempPendingContexts, resourceID)
+ }
+ }
+ return intentChanged
+}
diff --cc internal/alerts/host_unraid_lifecycle_test.go
index de8c3bf92f,a3c217fb20..17bed230e3
--- a/internal/alerts/host_unraid_lifecycle_test.go
+++ b/internal/alerts/host_unraid_lifecycle_test.go
@@@ -8,10 -6,8 +8,11 @@@ import
"testing"
"time"
+ "github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
+ "github.com/rcourtman/pulse-go-rewrite/internal/alerts/reducer"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
)
func TestHostCustomSensorAlertLifecycle(t *testing.T) {
@@@ -896,573 -1189,3 +1194,632 @@@ func TestHostChildAlertsSurviveProxmoxN
}
}
}
+
+// Disabling between two reports is a real policy interval even when there is
+// no firing alert for UpdateConfig to visit. Linked overrides keep precedence.
+func TestDiskTemperaturePendingConfigDisablement(t *testing.T) {
+ for _, explicit := range []bool{false, true} {
+ for _, scope := range []string{"global", "agent-default", "all-agents", "host", "host-disabled", "node", "vm", "container", "type", "unrelated"} {
+ t.Run(fmt.Sprintf("%s/explicit=%v", scope, explicit), func(t *testing.T) {
+ m, elapsed := continuityManager(t, explicit)
+ host := hostWithSMARTDiskTemp("pending-temp", "custom", 85)
+ host.LinkedNodeID, host.LinkedVMID, host.LinkedContainerID = "site:node", "site:node:101", "site:node:102"
+ id := hostDiskTemperatureResourceID(host.ID, host.Sensors.SMART[0].Device)
+ m.CheckHost(host)
+ if incident, ok := continuityIncident(m, id, "diskTemperature"); !ok || incident.State != reducer.StatePending {
+ t.Fatalf("no initial pending run: %+v %v", incident, ok)
+ }
+ cfg := m.GetConfig()
+ switch scope {
+ case "global":
+ cfg.Enabled = false
+ case "agent-default":
+ cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 0}
+ case "host-disabled":
+ cfg.Overrides[host.ID] = ThresholdConfig{Disabled: true}
+ case "all-agents":
+ cfg.DisableAllAgents = true
+ case "host":
+ cfg.Overrides[host.ID] = ThresholdConfig{DiskTemperature: &HysteresisThreshold{Trigger: 0}}
+ case "node":
+ cfg.Overrides[host.LinkedNodeID] = ThresholdConfig{DiskTemperature: &HysteresisThreshold{Trigger: 0}}
+ case "vm":
+ host.LinkedNodeID = "" // The linked node override must not mask the VM.
+ m.CheckHost(host)
+ cfg.Overrides[host.LinkedVMID] = ThresholdConfig{DiskTemperature: &HysteresisThreshold{Trigger: 0}}
+ case "container":
+ host.LinkedNodeID, host.LinkedVMID = "", ""
+ m.CheckHost(host)
+ cfg.Overrides[host.LinkedContainerID] = ThresholdConfig{DiskTemperature: &HysteresisThreshold{Trigger: 0}}
+ case "type":
+ cfg.DiskTempByType = map[string]HysteresisThreshold{"custom": {Trigger: 0}}
+ case "unrelated":
+ cfg.Schedule.QuietHours.Timezone = "UTC"
+ }
+ m.UpdateConfig(cfg)
+ _, pending := continuityIncident(m, id, "diskTemperature")
+ if pending != (scope == "unrelated") {
+ t.Fatalf("after save pending=%v; unrelated saves alone must preserve evidence", pending)
+ }
+ if scope == "unrelated" {
+ return
+ }
+ cfg.Enabled, cfg.DisableAllAgents = true, false
+ cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 80, Clear: 70}
+ cfg.Overrides = map[string]ThresholdConfig{}
+ cfg.DiskTempByType = map[string]HysteresisThreshold{"custom": {Trigger: 80, Clear: 70}}
+ m.UpdateConfig(cfg)
+ elapsed.Store(int64(2 * time.Minute))
+ m.CheckHost(host)
+ if len(m.GetActiveAlerts()) != 0 {
+ t.Fatal("re-enable inherited the pre-disable start")
+ }
+ elapsed.Store(int64(2*time.Minute + 59*time.Second))
+ m.CheckHost(host)
+ if len(m.GetActiveAlerts()) != 0 {
+ t.Fatal("fresh grace fired early")
+ }
+ elapsed.Store(int64(3 * time.Minute))
+ m.CheckHost(host)
+ if len(m.GetActiveAlerts()) != 1 {
+ t.Fatal("fresh continuous breach never fired")
+ }
+ m.mu.RLock()
+ defer m.mu.RUnlock()
+ if len(m.hostDiskTempPendingContexts) != 0 {
+ t.Fatal("context outlived the pending run")
+ }
+ })
+ }
+ }
+}
+
+func TestDiskTemperaturePendingRestartContextIsNotInvented(t *testing.T) {
+ m, elapsed := continuityManager(t, true)
+ host := hostWithSMARTDiskTemp("restored-temp", "sata", 85)
+ id := hostDiskTemperatureResourceID(host.ID, host.Sensors.SMART[0].Device)
+ m.CheckHost(host)
+ m.mu.Lock()
+ // Persisted intent carries grace but not host link/type context. Reducer
+ // pending incidents are not restored; this is the actual restart shape.
+ m.core.Reset()
+ m.hostDiskTempPendingContexts = nil
+ key := canonicalMetricStateID(id, "diskTemperature")
+ _, pending := m.intentPending[key]
+ m.mu.Unlock()
+ if !pending {
+ t.Fatal("explicit intent control did not retain grace")
+ }
+ m.UpdateConfig(m.GetConfig())
+ m.mu.RLock()
+ _, pending = m.intentPending[key]
+ m.mu.RUnlock()
+ if pending {
+ t.Fatal("save retained grace whose link/type enablement is unknown")
+ }
+ elapsed.Store(int64(2 * time.Minute))
+ m.CheckHost(host)
+ if len(m.GetActiveAlerts()) != 0 {
+ t.Fatal("restored grace fired without fresh evidence")
+ }
+}
+
+func TestDiskTemperatureCurrentCollectionAndOverrideControls(t *testing.T) {
+ for _, provenance := range []string{"legacy", "available", "other-field-only"} {
+ t.Run(provenance, func(t *testing.T) {
+ m, elapsed := continuityManager(t, false)
+ host := hostWithSMARTDiskTemp("current-temp", "sata", 85)
+ switch provenance {
+ case "available":
+ host.Sensors.SMART[0].Collection = &diskinventory.CollectionStatus{Temperature: diskinventory.Available("smartctl")}
+ case "other-field-only":
+ host.Sensors.SMART[0].Collection = &diskinventory.CollectionStatus{Serial: diskinventory.Available("smartctl")}
+ }
+ m.CheckHost(host)
+ elapsed.Store(int64(time.Minute))
+ m.CheckHost(host)
+ if len(m.GetActiveAlerts()) != 1 {
+ t.Fatal("continuous accepted temperature did not fire")
+ }
+ })
+ }
+ t.Run("host override wins over disabled type", func(t *testing.T) {
+ m, elapsed := continuityManager(t, false)
+ host := hostWithSMARTDiskTemp("override-temp", "custom", 85)
+ cfg := m.GetConfig()
+ cfg.DiskTempByType = map[string]HysteresisThreshold{"custom": {Trigger: 0}}
+ cfg.Overrides[host.ID] = ThresholdConfig{DiskTemperature: &HysteresisThreshold{Trigger: 80, Clear: 70}}
+ m.UpdateConfig(cfg)
+ m.CheckHost(host)
+ m.UpdateConfig(m.GetConfig())
+ elapsed.Store(int64(time.Minute))
+ m.CheckHost(host)
+ if len(m.GetActiveAlerts()) != 1 {
+ t.Fatal("save discarded an enabled explicit override")
+ }
+ })
+}
+
+func TestDiskTemperaturePendingContextCleanup(t *testing.T) {
+ for _, removal := range []bool{false, true} {
+ t.Run(fmt.Sprintf("removed=%v", removal), func(t *testing.T) {
+ m, _ := continuityManager(t, false)
+ host := hostWithSMARTDiskTemp("pending-cleanup", "custom", 85)
+ m.CheckHost(host)
+ m.mu.RLock()
+ tracked := len(m.hostDiskTempPendingContexts)
+ m.mu.RUnlock()
+ if tracked != 1 || len(m.GetActiveAlerts()) != 0 {
+ t.Fatal("control must have pending context and no active alert")
+ }
+ if removal {
+ m.HandleHostRemoved(host)
+ } else {
+ m.ClearActiveAlerts()
+ }
+ m.mu.RLock()
+ defer m.mu.RUnlock()
+ if len(m.hostDiskTempPendingContexts) != 0 || m.hostDiskTemperaturePendingNoLock(hostDiskTemperatureResourceID(host.ID, host.Sensors.SMART[0].Device)) {
+ t.Fatal("pending context/state outlived its host or explicit clear")
+ }
+ })
+ }
+}
+
+// Feed only synthetic telemetry through the real CheckHost, config and callback
+// paths. No collector, smartctl process, native host or notification provider.
+type smartLifecycleFixture struct {
+ m *Manager
+ host models.Host
+ healthID, wearID string
+ initial map[string]Alert
+ fires atomic.Int32
+ recoveries chan *ResolvedAlert
+}
+
+func newSMARTLifecycleFixture(t *testing.T) *smartLifecycleFixture {
+ t.Helper()
+ m := newEventLogManager(t)
+ cfg := m.GetConfig()
+ cfg.FlappingEnabled = false
+ cfg.Schedule.Cooldown = 0
+ cfg.Schedule.MaxAlertsHour = 0
+ cfg.AgentDefaults.Disk = &HysteresisThreshold{Trigger: 80, Clear: 70}
+ cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 80, Clear: 70}
+ cfg.DiskTempByType = map[string]HysteresisThreshold{"sata": {Trigger: 80, Clear: 70}}
+ m.UpdateConfig(cfg)
+ m.mu.Lock()
+ m.config.TimeThresholds = map[string]int{}
+ m.config.MetricTimeThresholds = map[string]map[string]int{"all": {"disk": 0, "diskTemperature": 0}}
+ m.mu.Unlock()
+ used, pending := 96, int64(2)
+ host := models.Host{ID: "smart-lifecycle", Hostname: "storage", Status: "online",
+ Disks: []models.Disk{{Mountpoint: "/data", Device: "/dev/sda1", Total: 1000, Used: 950, Usage: 95}},
+ Sensors: models.HostSensorSummary{SMART: []models.HostDiskSMART{
+ {Device: "/dev/sda", Type: "sata", Health: "FAILED", Temperature: 95, Attributes: &models.SMARTAttributes{PercentageUsed: &used, PendingSectors: &pending}},
+ {Device: "/dev/sdb", Type: "sata", Health: "PASSED", Temperature: 20},
+ }},
+ }
+ resourceID, _ := hostSMARTDiskResourceID(host, host.Sensors.SMART[0])
+ f := &smartLifecycleFixture{m: m, host: host,
+ healthID: buildCanonicalStateID(resourceID, resourceID+"-disk-health"),
+ wearID: buildCanonicalStateID(resourceID, resourceID+"-disk-wearout"),
+ initial: make(map[string]Alert), recoveries: make(chan *ResolvedAlert, 32),
+ }
+ m.SetAlertCallback(func(a *Alert) {
+ if a.ID == f.healthID || a.ID == f.wearID {
+ f.fires.Add(1)
+ }
+ })
+ m.SetResolvedAlertCallback(func(r *ResolvedAlert) {
+ if r.Alert.ID == f.healthID || r.Alert.ID == f.wearID {
+ f.recoveries <- r
+ }
+ })
+ m.CheckHost(host)
+ for _, id := range []string{f.healthID, f.wearID} {
+ f.initial[id] = *testRequireActiveAlert(t, m, id)
+ }
+ if f.fires.Load() != 2 {
+ t.Fatalf("initial risk callbacks = %d, want 2", f.fires.Load())
+ }
+ return f
+}
+
+func (f *smartLifecycleFixture) held(t *testing.T) {
+ t.Helper()
+ for _, id := range []string{f.healthID, f.wearID} {
+ a := testRequireActiveAlert(t, f.m, id)
+ if !a.StartTime.Equal(f.initial[id].StartTime) {
+ t.Fatalf("%s was replaced by a new occurrence", id)
+ }
+ }
+ if f.fires.Load() != 2 {
+ t.Fatalf("missing report refired risk notifications: %d", f.fires.Load())
+ }
+ for _, e := range queryAlertEvents(t, f.m, eventlog.Filter{Types: []string{eventlog.TypeResolved}}) {
+ if e.AlertID == f.healthID || e.AlertID == f.wearID {
+ t.Fatalf("missing evidence published recovery: %+v", e)
+ }
+ }
+}
+
+func (f *smartLifecycleFixture) resolved(t *testing.T, ids ...string) {
+ t.Helper()
+ want := make(map[string]bool)
+ for _, id := range ids {
+ if testHasActiveAlert(t, f.m, id) {
+ t.Fatalf("risk alert remains active: %s", id)
+ }
+ want[id] = true
+ }
+ for range ids {
+ select {
+ case r := <-f.recoveries:
+ if !want[r.Alert.ID] || !r.Alert.StartTime.Equal(f.initial[r.Alert.ID].StartTime) {
+ t.Fatalf("wrong recovery occurrence: %+v", r)
+ }
+ delete(want, r.Alert.ID)
+ case <-time.After(3 * time.Second):
+ t.Fatal("ordinary recovery callback did not complete")
+ }
+ }
+ for _, id := range ids {
+ events := queryAlertEvents(t, f.m, eventlog.Filter{AlertID: id, Types: []string{eventlog.TypeResolved}})
+ if len(events) != 1 {
+ t.Fatalf("%s has %d resolution events, want 1", id, len(events))
+ }
+ }
+}
+
+func TestHostSMARTRiskMissingReportsPreserveOccurrenceAndCallbacks(t *testing.T) {
+ f := newSMARTLifecycleFixture(t)
+ if err := f.m.AcknowledgeAlert(f.healthID, "operator"); err != nil {
+ t.Fatal(err)
+ }
+ present := f.host.Sensors.SMART
+ // Both risk families on one disk must count once per report, not twice.
+ for _, list := range [][]models.HostDiskSMART{nil, {present[1]}, {present[1]}, nil, {present[1]}, present, nil, present} {
+ f.host.Sensors.SMART = list
+ f.m.CheckHost(f.host)
+ f.held(t)
+ if !testRequireActiveAlert(t, f.m, f.healthID).Acknowledged {
+ t.Fatal("acknowledgement lost during observation gap")
+ }
+ }
+ // Standby and unavailable health/wear fields are presence, not recovery.
+ f.host.Sensors.SMART = []models.HostDiskSMART{{Device: "/dev/sda", Standby: true}, present[1]}
+ f.m.CheckHost(f.host)
+ f.held(t)
+ f.host.Sensors.SMART[0].Standby = false
+ f.m.CheckHost(f.host)
+ f.held(t)
+ used, pending := 10, int64(0)
+ f.host.Sensors.SMART[0] = models.HostDiskSMART{Device: "/dev/sda", Health: "PASSED", Temperature: 20, Attributes: &models.SMARTAttributes{PercentageUsed: &used, PendingSectors: &pending}}
+ f.m.CheckHost(f.host)
+ f.resolved(t, f.healthID, f.wearID)
+ if f.fires.Load() != 2 {
+ t.Fatalf("healthy recovery refired: %d", f.fires.Load())
+ }
+ t.Log("empty/partial/standby/unknown reports preserved occurrence and acknowledgement; fresh complete evidence recovered once")
+}
+
+func TestHostSMARTRiskDepartureRequiresFreshConsecutiveReports(t *testing.T) {
+ for _, gap := range []string{"empty", "expiry", "present"} {
+ t.Run(gap, func(t *testing.T) {
+ f := newSMARTLifecycleFixture(t)
+ present := f.host.Sensors.SMART
+ f.host.Sensors.SMART = []models.HostDiskSMART{present[1]}
+ for i := 0; i < 2; i++ {
+ f.m.CheckHost(f.host)
+ f.held(t)
+ }
+ switch gap {
+ case "empty":
+ f.host.Sensors.SMART = nil
+ f.m.CheckHost(f.host)
+ case "expiry":
+ f.m.HandleHostTelemetryExpired(f.host)
+ case "present":
+ f.host.Sensors.SMART = present
+ f.m.CheckHost(f.host)
+ }
+ f.held(t)
+ f.host.Sensors.SMART = []models.HostDiskSMART{present[1]}
+ for i := 1; i <= 3; i++ {
+ f.m.CheckHost(f.host)
+ if i < 3 {
+ f.held(t)
+ }
+ }
+ f.resolved(t, f.healthID, f.wearID)
+ if f.fires.Load() != 2 {
+ t.Fatalf("departure refired %d callbacks", f.fires.Load())
+ }
+ })
+ }
+}
+
+func TestHostSMARTRiskRulesDisableBetweenReports(t *testing.T) {
+ f := newSMARTLifecycleFixture(t)
+ cfg := f.m.GetConfig()
+ off := 0
+ cfg.AgentDefaults.SMARTHealthFailure = &off
+ f.m.UpdateConfig(cfg)
+ f.held(t) // Pending sectors remain enabled; disabling only one cause is not recovery.
+ zero := int64(0)
+ cfg.AgentDefaults.SMARTPending = &zero
+ f.m.UpdateConfig(cfg)
+ f.resolved(t, f.healthID)
+ if !testHasActiveAlert(t, f.m, f.wearID) {
+ t.Fatal("health policy disabled unrelated wearout")
+ }
+ // A missing report cannot resurrect it. Re-enable before another report;
+ // only fresh failing evidence may reactivate it, under the unchanged
+ // stateful refire/cooldown history policy.
+ f.host.Sensors.SMART = nil
+ f.m.CheckHost(f.host)
+ if testHasActiveAlert(t, f.m, f.healthID) {
+ t.Fatal("missing report raised disabled health")
+ }
+ on, one := 1, int64(1)
+ cfg.AgentDefaults.SMARTHealthFailure = &on
+ cfg.AgentDefaults.SMARTPending = &one
+ f.m.UpdateConfig(cfg)
+ f.m.CheckHost(f.host)
+ if testHasActiveAlert(t, f.m, f.healthID) {
+ t.Fatal("re-enable raised health without evidence")
+ }
+ used, pending := 96, int64(2)
+ f.host.Sensors.SMART = []models.HostDiskSMART{{Device: "/dev/sda", Health: "FAILED", Attributes: &models.SMARTAttributes{PercentageUsed: &used, PendingSectors: &pending}}}
+ f.m.CheckHost(f.host)
+ a := testRequireActiveAlert(t, f.m, f.healthID)
+ if !a.StartTime.Equal(f.initial[f.healthID].StartTime) || !a.LastSeen.After(f.initial[f.healthID].LastSeen) || f.fires.Load() != 3 {
+ t.Fatalf("fresh evidence must refire once within the existing cooldown: start %v, prior %v, seen %v, prior %v, callbacks %d", a.StartTime, f.initial[f.healthID].StartTime, a.LastSeen, f.initial[f.healthID].LastSeen, f.fires.Load())
+ }
+}
+
+func TestHostSMARTRiskEmptyReportHonoursRuleOffAndLegacyUnknowns(t *testing.T) {
+ for _, metadata := range []string{"current", "legacy", "unknown"} {
+ t.Run(metadata, func(t *testing.T) {
+ f := newSMARTLifecycleFixture(t)
+ off, zero := 0, int64(0)
+ f.m.mu.Lock()
+ f.m.config.AgentDefaults.SMARTHealthFailure = &off
+ f.m.config.AgentDefaults.SMARTPending = &zero
+ if metadata != "current" {
+ a, _ := f.m.getActiveAlertNoLock(f.healthID)
+ if metadata == "legacy" {
+ delete(a.Metadata, "riskCodes")
+ } else {
+ a.Metadata["riskCodes"] = []interface{}{"future-code"}
+ }
+ }
+ f.m.mu.Unlock()
+ f.host.Sensors.SMART = nil
+ f.m.CheckHost(f.host)
+ if metadata == "current" {
+ f.resolved(t, f.healthID)
+ } else {
+ f.held(t)
+ }
+ if !testHasActiveAlert(t, f.m, f.wearID) {
+ t.Fatal("wearout was disabled with health")
+ }
+ })
+ }
+}
+
+func TestHostSMARTRiskFilesystemAndHostLifecycleStayIndependent(t *testing.T) {
+ for _, action := range []string{"filesystem-absent", "node-link", "removed", "host-disabled", "agents-disabled", "host-isolation"} {
+ t.Run(action, func(t *testing.T) {
+ f := newSMARTLifecycleFixture(t)
+ switch action {
+ case "filesystem-absent":
+ fsID, _ := hostDiskResourceID(f.host, f.host.Disks[0])
+ id := canonicalMetricStateID(fsID, "disk")
+ testRequireActiveAlert(t, f.m, id)
+ f.host.Disks = nil
+ f.host.Sensors.SMART = nil
+ f.m.CheckHost(f.host)
+ if testHasActiveAlert(t, f.m, id) {
+ t.Fatal("filesystem absence incorrectly held usage")
+ }
+ f.held(t)
+ case "host-isolation":
+ other := f.host
+ other.ID = "smart-lifecycle-other"
+ f.m.CheckHost(other)
+ other.Sensors.SMART = []models.HostDiskSMART{other.Sensors.SMART[1]}
+ for i := 0; i < 3; i++ {
+ f.m.CheckHost(other)
+ }
+ f.held(t)
+ default:
+ present := f.host.Sensors.SMART
+ f.host.Sensors.SMART = []models.HostDiskSMART{present[1]}
+ f.m.CheckHost(f.host)
+ f.held(t)
+ switch action {
+ case "node-link":
+ f.host.LinkedNodeID = "pve-node"
+ f.host.Sensors.SMART = nil
+ f.m.CheckHost(f.host)
+ case "removed":
+ f.m.HandleHostRemoved(f.host)
+ case "host-disabled":
+ cfg := f.m.GetConfig()
+ cfg.Overrides[f.host.ID] = ThresholdConfig{Disabled: true}
+ f.m.UpdateConfig(cfg)
+ case "agents-disabled":
+ cfg := f.m.GetConfig()
+ cfg.DisableAllAgents = true
+ f.m.UpdateConfig(cfg)
+ }
+ f.resolved(t, f.healthID, f.wearID)
+ // Returning ownership gets a new occurrence and a fresh departure run.
+ f.host.LinkedNodeID = ""
+ cfg := f.m.GetConfig()
+ cfg.DisableAllAgents = false
+ delete(cfg.Overrides, f.host.ID)
+ f.m.UpdateConfig(cfg)
+ f.host.Sensors.SMART = present
+ f.m.CheckHost(f.host)
+ f.host.Sensors.SMART = []models.HostDiskSMART{present[1]}
+ for i := 0; i < 2; i++ {
+ f.m.CheckHost(f.host)
+ testRequireActiveAlert(t, f.m, f.healthID)
+ testRequireActiveAlert(t, f.m, f.wearID)
+ }
+ }
+ })
+ }
+}
+
+func TestHostSMARTRiskFilesystemIdentityCannotSubstituteInventory(t *testing.T) {
+ for _, direction := range []string{"SMART-only", "filesystem-only"} {
+ t.Run(direction, func(t *testing.T) {
+ f := newSMARTLifecycleFixture(t)
+ f.host.Disks[0].Mountpoint = "/sda"
+ f.m.CheckHost(f.host)
+ diskResourceID, _ := hostDiskResourceID(f.host, f.host.Disks[0])
+ riskResourceID, _ := hostSMARTDiskResourceID(f.host, f.host.Sensors.SMART[0])
+ if diskResourceID != riskResourceID {
+ t.Fatal("control must share the disk resource ID")
+ }
+ usageID := canonicalMetricStateID(diskResourceID, "disk")
+ testRequireActiveAlert(t, f.m, usageID)
+ if direction == "SMART-only" {
+ f.host.Disks = nil
+ f.m.CheckHost(f.host)
+ if testHasActiveAlert(t, f.m, usageID) {
+ t.Fatal("SMART identity held missing filesystem usage")
+ }
+ f.held(t)
+ } else {
+ f.host.Sensors.SMART = []models.HostDiskSMART{f.host.Sensors.SMART[1]}
+ for i := 1; i <= 3; i++ {
+ f.m.CheckHost(f.host)
+ testRequireActiveAlert(t, f.m, usageID)
+ if i < 3 {
+ f.held(t)
+ }
+ }
+ f.resolved(t, f.healthID, f.wearID)
+ }
+ })
+ }
+}
+
+func TestHostSMARTRiskRestartAndGlobalPauseRestartDeparture(t *testing.T) {
+ for _, gap := range []string{"restart", "global-pause"} {
+ t.Run(gap, func(t *testing.T) {
+ f := newSMARTLifecycleFixture(t)
+ if err := f.m.AcknowledgeAlert(f.healthID, "operator"); err != nil {
+ t.Fatal(err)
+ }
+ f.host.Sensors.SMART = []models.HostDiskSMART{f.host.Sensors.SMART[1]}
+ for i := 0; i < 2; i++ {
+ f.m.CheckHost(f.host)
+ f.held(t)
+ }
+ cfg := f.m.GetConfig()
+ if gap == "restart" {
+ dir := filepath.Dir(f.m.alertsDir)
+ f.m.Stop() // Actual checkpoint and ordinary persisted-alert restoration.
+ f.m = NewManagerWithDataDir(dir)
+ t.Cleanup(f.m.Stop)
+ store, err := eventlog.OpenInMemory()
+ if err != nil {
+ t.Fatal(err)
+ }
+ f.m.SetEventLog(store) // Observe post-restart lifecycle, not nonexistent old in-memory events.
+ f.m.UpdateConfig(cfg)
+ f.m.SetAlertCallback(func(a *Alert) {
+ if a.ID == f.healthID || a.ID == f.wearID {
+ f.fires.Add(1)
+ }
+ })
+ f.m.SetResolvedAlertCallback(func(r *ResolvedAlert) {
+ if r.Alert.ID == f.healthID || r.Alert.ID == f.wearID {
+ f.recoveries <- r
+ }
+ })
+ } else {
+ cfg.Enabled = false
+ f.m.UpdateConfig(cfg)
+ cfg.Enabled = true
+ f.m.UpdateConfig(cfg)
+ }
+ for i := 1; i <= 3; i++ {
+ f.m.CheckHost(f.host)
+ if i < 3 {
+ f.held(t)
+ if !testRequireActiveAlert(t, f.m, f.healthID).Acknowledged {
+ t.Fatal("gap lost persisted acknowledgement")
+ }
+ }
+ }
+ f.resolved(t, f.healthID, f.wearID)
+ })
+ }
+}
++
++// Upstream's shared Unraid readings must retain reviewed pending-grace
++// ownership even when no SMART row exists; genuine gaps still restart grace.
++func TestUnraidOnlyTemperatureRetainsPendingContinuity(t *testing.T) {
++ for _, explicit := range []bool{false, true} {
++ for _, gap := range []string{"continuous", "spun-down", "omitted", "type-off"} {
++ t.Run(fmt.Sprintf("%s/explicit=%v", gap, explicit), func(t *testing.T) {
++ m, elapsed := continuityManager(t, explicit)
++ disk := models.HostUnraidDisk{Name: "disk1", Device: "sda", Serial: "CONTINUITY-1", Transport: "sata", Temperature: 85}
++ host := unraidTempHost(nil, disk)
++ id := hostDiskTemperatureResourceID(host.ID, "sda")
++ m.CheckHost(host)
++ if incident, ok := continuityIncident(m, id, "diskTemperature"); !ok || incident.State != reducer.StatePending {
++ t.Fatalf("no initial pending Unraid run: %+v %v", incident, ok)
++ }
++ elapsed.Store(int64(30 * time.Second))
++ switch gap {
++ case "continuous":
++ m.CheckHost(host)
++ case "spun-down":
++ host.Unraid.Disks[0].Temperature = 0
++ m.CheckHost(host)
++ host.Unraid.Disks[0] = disk
++ case "omitted":
++ host.Unraid.Disks = nil
++ m.CheckHost(host)
++ host.Unraid.Disks = []models.HostUnraidDisk{disk}
++ case "type-off":
++ cfg := m.GetConfig()
++ cfg.DiskTempByType = map[string]HysteresisThreshold{"sata": {Trigger: 0}}
++ m.UpdateConfig(cfg)
++ cfg.DiskTempByType = map[string]HysteresisThreshold{"sata": {Trigger: 80, Clear: 70}}
++ m.UpdateConfig(cfg)
++ }
++ elapsed.Store(int64(time.Minute))
++ m.CheckHost(host)
++ if gap == "continuous" {
++ if len(m.GetActiveAlerts()) != 1 {
++ t.Fatal("live Unraid-only reports lost continuous pending grace")
++ }
++ return
++ }
++ if len(m.GetActiveAlerts()) != 0 {
++ t.Fatal("Unraid gap inherited earlier pending grace")
++ }
++ elapsed.Store(int64(119 * time.Second))
++ m.CheckHost(host)
++ if len(m.GetActiveAlerts()) != 0 {
++ t.Fatal("fresh Unraid grace fired early")
++ }
++ elapsed.Store(int64(2 * time.Minute))
++ m.CheckHost(host)
++ if len(m.GetActiveAlerts()) != 1 {
++ t.Fatal("fresh continuous Unraid breach did not fire")
++ }
++ })
++ }
++ }
++}