Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Preserve reviewed maintenance with current upstream report and identity ownership
pulse-triage[bot]1052f214195e
diff --cc docs/release-control/v6/internal/subsystems/ai-runtime.mdindex 9559c01b62,bc4f13bd3f..f07697a0d0--- a/docs/release-control/v6/internal/subsystems/ai-runtime.md+++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md@@@ -142,6 -128,25 +142,19 @@@ on component mount. Closing clears thecommands cannot target a detached composer. A handoff must leave Escape andkeyboard input in Assistant, not the underlying alert search.-### Process-wide memory ID sequences--`internal/ai/memory` mints incident, incident-event, remediation-record and-context-memory IDs from package-level sequences that every store in the process-shares. Multi-tenant monitoring gives each tenant monitor its own-`IncidentStore`, and their alert checks call `RecordAlertFired` from separate-goroutines, while a store's mutex covers only its own state. Each sequence is-therefore an `atomic.Int64` minted through `nextSequencedID` in-`internal/ai/memory/ids.go`, never a plain package counter. The ID shape stays-`<prefix><local second>-<n>`, but `n` no longer wraps at 1000. The old-`counter%1000` suffix repeated an ID whenever one generator minted more than a-thousand in a wall-clock second, and incident notes resolve an incident by ID-while timeline merges deduplicate events by ID. Within one process IDs are now-unique by construction. Across restarts only the timestamp separates them, so a-repeated local second (DST fall-back or a backward clock step) can still meet a-restarted sequence's suffix. Persisted IDs and the stored format are unchanged.-`internal/ai/memory/ids_test.go` mints from every generator concurrently and-records alerts into two tenant stores concurrently; run it under `-race`.++### Process-wide memory ID ownership++++`internal/ai/memory` preserves random UUID incident and incident-event IDs from++the reviewed occurrence repair. They have no shared mutable counter and do not++reuse a local timestamp after restart. Existing persisted references remain valid.++Remediation-record and context-memory IDs use the upstream package-wide atomic++`Int64` sequences through `nextSequencedID` in `ids.go`, without wrapping at 1000.++Their existing `<prefix><local second>-<n>` format remains; across restarts those++four sequences still depend on the local timestamp and retain that limitation.++`ids_test.go` checks all six actual formats and concurrent uniqueness, plus++independent concurrent tenant-store recording under `-race`. The reviewed++`incidents_identity_test.go` continues to pin UUID occurrence/event ownership.+### Canonical incident-history queriesIncident context reads `IncidentStore.QueryIncidents`, whose page containsdiff --cc docs/release-control/v6/internal/subsystems/alerts.mdindex d3c44eb5f3,513b44c27d..5f93f3738d--- a/docs/release-control/v6/internal/subsystems/alerts.md+++ b/docs/release-control/v6/internal/subsystems/alerts.md@@@ -4219,103 -4184,3 +4274,114 @@@ display name differs from its hostnamehad no alert highlighting at all. `getAlertStyles` keeps itsid-plus-node-name matching for the platform tables whose alerts are keyed thatway.++### SMART temperature continuity and pending-only policy intervals++Disk temperature activation/grace and recovery require continuous accepted+observations. Empty SMART inventory, one disk omission, standby, non-positive+readings, explicit missing/unavailable/unsupported collection and reporting-lease+expiry interrupt pending/recovery timing without changing an open alert, its+acknowledgement or lifecycle events. Positive legacy readings without temperature+provenance remain accepted; an expired last-known positive value is not live+health. Canonical and shadow reducers plus explicit intent grace are interrupted+together. A non-empty inventory still resolves a departed disk only after three+consecutive omissions; empty/expired inventory breaks that confirmation sequence.++Configuration saves reset pending-only runs when global agent policy, host or+inherited node/guest overrides, or the applicable disk-type threshold disables+evaluation, even if no report arrives before re-enablement. Existing known-type+default normalisation is unchanged; the zero-type save control uses a custom+type whose zero threshold the normaliser preserves. Only live pending+identity/link/type context is retained; unrelated saves preserve a known enabled+run and explicit overrides retain precedence over type defaults. Restored intent+without that context restarts conservatively on a save rather than inventing+continuous enablement. Cleared intent grace is checkpointed. Existing active-alert+per-type/save/offline/removal resolution remains unchanged.++`metric_observation_continuity_test.go` and `host_unraid_lifecycle_test.go` cover+valid/legacy/provenance controls, missing/rejected/expiry gaps on both timing+edges, explicit grace, acknowledgement/event preservation, host isolation,+configuration intervals, override precedence and departure. These deterministic+source tests do not establish installed temperatures or native guest recovery.++### SMART risk inventory gaps and disabled rules++Agent `disk-health` and `disk-wearout` alerts follow the SMART inventory, not+filesystem usage cleanup. Empty inventory, standby and unobserved risk fields+hold the existing occurrence and acknowledgement without recovery/refiring.+Departure requires three consecutive non-empty reports omitting the disk;+listing it again, an empty inventory or reporting-lease expiry restarts that+confirmation. Both risk families on the same disk count once per report.+Filesystem absence still resolves usage on its first missing report. Temperature+pending/grace/recovery interruption and departure semantics remain unchanged.++Disabling every recorded cause releases that risk alert without a reading,+both on a configuration save and on an empty report. Partially enabled causes,+missing legacy cause metadata and unknown codes remain conservative. Re-enabling+cannot raise an alert from absent evidence; fresh bad evidence uses the existing+stateful refire/cooldown history policy. Node linking retains existing node ownership, including when the+agent's SMART inventory is empty; host removal and policy disablement release+agent risk state and stale absence counts. Counts are ephemeral and restart+conservatively after a process restart.++`TestHostSMARTRisk*` in `host_unraid_lifecycle_test.go` exercises actual `CheckHost`, expiry, configuration+save, firing/recovery callbacks and event history with synthetic telemetry.+It covers both families sharing a disk, interruption/reconfirmation, partial+rule disablement, re-enablement, legacy/unknown causes, filesystem and host+isolation, removal and node-link ownership. These are source lifecycle controls,+not collector, smartctl, installed notification or native disk-recovery proof.++### Storage observation gaps restart confirmation and timing++An absent or `unknown` storage connectivity status interrupts activation,+recovery confirmations and pending offline intent grace without resolving an+existing outage or changing its acknowledgement. Capacity is independently+observable while connectivity is unknown. Missing/unconfirmed-zero capacity,+negative usage and offline/unavailable storage interrupt capacity activation,+and intent grace timing; they do not supply healthy capacity evidence.+Capacity recovery retains its existing immediate measured-clear behaviour,+not the recovery stability window used by memory and temperature gauges.+A confirmed empty store still supplies a genuine zero-usage recovery reading.+Fresh observations must satisfy the existing full confirmation/delay after a+gap. Interruption is scoped to the storage's canonical identity and applied to+both live and shadow reducers; interrupted intent is checkpointed for restart.++`TestStorageConnectivityObservationGaps` and+`TestStorageConnectivityGapRestartsIntentGrace` exercise the real storage+entry point, acknowledged occurrence retention, neighbour isolation, fresh+reconfirmation and empty intent checkpoint. Storage routes in+`TestMetricObservationGapRestartsActivation` covers capacity gaps and explicit+intent grace; `TestStorageCapacityGapHoldsOccurrence` verifies retained,+acknowledged capacity incidents and genuine immediate empty-store recovery. Existing restart, empty-capacity and predictive-capacity controls+remain required. These source controls are not native storage recovery or+installed destination-delivery evidence.++`TestStorageConnectivityGapWebhook` also exercises the production Monitor+callbacks, ordinary notification queue and guest-local HTTP receiver: the gap+and first fresh healthy poll send no recovery; confirmed recovery sends one+resolved receipt with the original occurrence identity, and the queue drains+without failed/DLQ work. This is synthetic callback/receiver acceptance, not a+native appliance or external webhook-provider result.++### Composed host filesystem threshold ownership++The shared filesystem resolver preserves the explicit host or linked-resource+Disk override ahead of per-hardware-type defaults, for both host reports and+configuration-save reevaluation. A filesystem-specific override still takes+precedence. Disabling that filesystem releases its canonical usage metric and+pending intent, retaining the reviewed agent/node ownership semantics.+`TestHostDiskOverrideBeatsDiskFillByType` checks agent and inherited overrides+through a configuration save and the next report without losing the occurrence;+`TestConfigSaveJudgesFilesystemAlertsByTheirOwnThreshold` retains per-filesystem+and per-type controls. This is source composition, not installed delivery proof.++++### Shared disk-temperature composition and pending ownership++++Host temperature evaluation retains the upstream shared SMART/Unraid reading and++per-type threshold judgement, including Unraid-only disks and competing readings++on one device. It also retains reviewed observation-gap interruption and pending++host/link/type context for configuration saves. A live Unraid-only row is not an++observation gap merely because SMART is empty; absent or uncollected rows still++restart confirmation. `TestUnraidOnlyTemperatureRetainsPendingContinuity` checks++continuous, spun-down, omitted and type-off intervals under both grace paths;++existing SMART pending, override, departure and upstream Unraid controls remain.diff --cc internal/ai/memory/ids_test.goindex 0000000000,530381625e..bf2138e2b3mode 000000,100644..100644--- a/internal/ai/memory/ids_test.go+++ b/internal/ai/memory/ids_test.go@@@ -1,0 -1,122 +1,126 @@@+ package memory++ import (+ "fmt"+ "regexp"+ "strings"+ "sync"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/alerts"+ )++ // TestSequencedIDsStayUniqueUnderConcurrentMinting mints from every+ // package-level ID sequence on many goroutines at once. Under -race it proves+ // the sequences are synchronized. The uniqueness check proves the suffix does+ // not wrap: more than a thousand IDs from one generator land in the same+ // wall-clock second, which the old counter%1000 suffix turned into duplicates.+ func TestSequencedIDsStayUniqueUnderConcurrentMinting(t *testing.T) {+ generators := []struct {+ prefix string+ mint func() string++ uuid bool+ }{- {"inc-", generateIncidentID},- {"inc-evt-", generateIncidentEventID},- {"rem-", generateRecordID},- {"mem-", generateMemoryID},- {"inc-mem-", generateIncidentMemoryID},- {"pat-mem-", generatePatternMemoryID},++ {"inc-", generateIncidentID, true},++ {"inc-evt-", generateIncidentEventID, true},++ {"rem-", generateRecordID, false},++ {"mem-", generateMemoryID, false},++ {"inc-mem-", generateIncidentMemoryID, false},++ {"pat-mem-", generatePatternMemoryID, false},+ }+ const goroutines, perGoroutine = 16, 150++ for _, gen := range generators {+ t.Run(strings.TrimSuffix(gen.prefix, "-"), func(t *testing.T) {+ shape := regexp.MustCompile("^" + regexp.QuoteMeta(gen.prefix) + `\d{14}-[1-9]\d*$`)++ if gen.uuid {++ shape = regexp.MustCompile("^" + regexp.QuoteMeta(gen.prefix) + `[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$`)++ }+ minted := make([][]string, goroutines)+ start := make(chan struct{})+ var wg sync.WaitGroup+ for g := range goroutines {+ wg.Add(1)+ go func() {+ defer wg.Done()+ <-start+ batch := make([]string, perGoroutine)+ for i := range batch {+ batch[i] = gen.mint()+ }+ minted[g] = batch+ }()+ }+ close(start)+ wg.Wait()++ seen := make(map[string]bool, goroutines*perGoroutine)+ for _, batch := range minted {+ for _, id := range batch {+ if !shape.MatchString(id) {+ t.Fatalf("ID %q does not match %s", id, shape)+ }+ if seen[id] {+ t.Fatalf("duplicate ID %q among %d concurrent mints", id, goroutines*perGoroutine)+ }+ seen[id] = true+ }+ }+ })+ }+ }++ // TestIncidentStoresMintIDsConcurrentlyAcrossTenants mirrors multi-tenant+ // monitoring: each tenant monitor owns its own IncidentStore, and their alert+ // checks call RecordAlertFired from separate goroutines. Each store's mutex+ // covers only its own state, so the shared ID sequence must be synchronized on+ // its own. Run under -race.+ func TestIncidentStoresMintIDsConcurrentlyAcrossTenants(t *testing.T) {+ const tenants, checkersPerTenant, alertsPerChecker = 2, 4, 50+ stores := make([]*IncidentStore, tenants)+ for i := range stores {+ stores[i] = NewIncidentStore(IncidentStoreConfig{MaxIncidents: checkersPerTenant * alertsPerChecker})+ }++ startedAt := time.Now().Add(-time.Minute)+ start := make(chan struct{})+ var wg sync.WaitGroup+ for tenant, store := range stores {+ for checker := range checkersPerTenant {+ wg.Add(1)+ go func() {+ defer wg.Done()+ <-start+ for n := range alertsPerChecker {+ store.RecordAlertFired(&alerts.Alert{+ ID: fmt.Sprintf("tenant-%d-checker-%d-alert-%d", tenant, checker, n),+ Type: "cpu",+ Level: alerts.AlertLevelWarning,+ ResourceID: fmt.Sprintf("tenant-%d-vm-%d-%d", tenant, checker, n),+ ResourceName: "vm",+ StartTime: startedAt,+ })+ }+ }()+ }+ }+ close(start)+ wg.Wait()++ seen := make(map[string]bool, tenants*checkersPerTenant*alertsPerChecker)+ for tenant, store := range stores {+ store.mu.RLock()+ incidents := len(store.incidents)+ for _, shell := range store.incidents {+ if seen[shell.ID] {+ t.Errorf("tenant %d incident ID %q was already minted", tenant, shell.ID)+ }+ seen[shell.ID] = true+ }+ store.mu.RUnlock()+ if incidents != checkersPerTenant*alertsPerChecker {+ t.Errorf("tenant %d recorded %d incidents, want %d", tenant, incidents, checkersPerTenant*alertsPerChecker)+ }+ }+ }diff --cc internal/alerts/host.goindex 781969d9c4,be3d9aa092..2d3546b5c6--- a/internal/alerts/host.go+++ b/internal/alerts/host.go@@@ -429,52 -420,82 +430,93 @@@ func (m *Manager) CheckHost(host models}if thresholds.DiskTemperature != nil && thresholds.DiskTemperature.Trigger > 0 {+ // Each disk is judged on the temperature it shows: its SMART row's+ // reading, or the one the host's Unraid inventory reports for it when+ // the agent's probe returned none, and the inventory's reading for a+ // disk only the inventory lists.+ readings := unifiedresources.HostDiskTemperatureReadings(host)+ seenDiskTemps := make(map[string]struct{}, len(readings))+ excludedDiskTemps := make(map[string]struct{})+ judgedDiskTemps := make(map[string]judgedHostDiskTemperature, len(readings))+ judgedOrder := make([]string, 0, len(readings))+ for _, disk := range readings {+ tempResourceID := hostDiskTemperatureResourceID(host.ID, disk.Device)+ // A disk the agent's --disk-exclude patterns match is out of+ // monitoring, so the Unraid inventory's reading raises nothing+ // for it either.+ if disk.UnraidOnly && fsfilters.MatchesDeviceExclude(disk.Device, host.DiskExclude) {+ excludedDiskTemps[tempResourceID] = struct{}{}+ continue+ }+ // A listed disk in standby, or without a current temperature after+ // a failed probe, is still present. Its alert holds until a fresh+ // reading resolves it instead of clearing and re-raising.+ seenDiskTemps[tempResourceID] = struct{}{}+ if !disk.Collected() {+ continue+ }+ m.mu.RLock()+ threshold := m.hostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature, diskTempOverridden, disk.DiskType)+ m.mu.RUnlock()+ // Sources that disagree on a disk's identity can show two disks+ // on one device. Its alert judges the reading that stands highest+ // against its own disk type's threshold.+ candidate := judgedHostDiskTemperature{reading: disk, threshold: threshold}+ if judged, ok := judgedDiskTemps[tempResourceID]; !ok {+ judgedOrder = append(judgedOrder, tempResourceID)+ } else if !candidate.outranks(judged) {+ continue+ }+ judgedDiskTemps[tempResourceID] = candidate+ }+ for _, tempResourceID := range judgedOrder {+ disk := judgedDiskTemps[tempResourceID].reading+ effectiveTempThreshold := judgedDiskTemps[tempResourceID].threshold++ tempResourceName := fmt.Sprintf("%s (%s Temp)", hostDisplayName(host), disk.Device)++ diskTempMetadata := hostChildAlertMetadata(baseMetadata, hostDiskAlertResourceType)+ diskTempMetadata["metric"] = "diskTemperature"+ diskTempMetadata["device"] = disk.Device+ diskTempMetadata["temperature"] = disk.Temperature+ diskTempMetadata["model"] = disk.Model+ diskTempMetadata["diskType"] = disk.DiskType+ spec, err := buildCanonicalMetricSpec(tempResourceID, tempResourceName, unifiedresources.ResourceType("agent-disk"), "diskTemperature", effectiveTempThreshold)+ if err != nil {+ log.Warn().+ Err(err).+ Str("resourceID", tempResourceID).+ Str("host", resourceName).+ Str("device", disk.Device).+ Msg("Skipping invalid canonical host disk temperature metric spec")++ m.interruptHostDiskTemperatureRun(tempResourceID)+ continue+ }++ m.checkMetricWithCanonicalSpec(spec, tempResourceName, nodeName, disk.Device, "agent", float64(disk.Temperature), effectiveTempThreshold, &metricOptions{Metadata: diskTempMetadata})++ m.rememberHostDiskTemperaturePending(host, models.HostDiskSMART{Type: disk.DiskType}, tempResourceID)++ }++ // A present disk without any collected source interrupts pending timing.++ // Another usable source for the same device keeps its run continuous.++ for tempResourceID := range seenDiskTemps {++ if _, observed := judgedDiskTemps[tempResourceID]; !observed {++ m.interruptHostDiskTemperatureRun(tempResourceID)++ }+ }+ // An excluded disk's alert from before the exclusion resolves now,+ // whatever the SMART list holds, unless another listed disk shares it.+ for tempResourceID := range seenDiskTemps {+ delete(excludedDiskTemps, tempResourceID)+ }+ m.clearHostDiskTemperatureResources(excludedDiskTemps)// An empty SMART list means collection failed or is unsupported for// this report, not that every disk left, so existing alerts are held.if len(host.Sensors.SMART) > 0 {// A disk missing from consecutive non-empty reports was removed,// replaced or renamed, so no later reading will resolve its alert.m.cleanupHostDiskTemperatureAlerts(host.ID, seenDiskTemps)+ } else {- m.interruptHostDiskTemperatureRuns(host.ID)++ m.interruptHostDiskTemperatureRunsExcept(host.ID, seenDiskTemps)}} else {// Disk temperature alerting is off for this host, so no later readingdiff --cc internal/alerts/host_disk_temperature.goindex 4f7dec14ab,0000000000..b575fef4c0mode 100644,000000..100644--- a/internal/alerts/host_disk_temperature.go+++ b/internal/alerts/host_disk_temperature.go@@@ -1,135 -1,0 +1,143 @@@+package alerts++import (+ "strings"++ "github.com/rcourtman/pulse-go-rewrite/internal/alerts/reducer"+ "github.com/rcourtman/pulse-go-rewrite/internal/models"+ "github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"+)++type hostDiskTempContext struct {+ hostID, linkedNodeID, linkedVMID, linkedContainerID, diskType string+}++// A positive legacy reading is still accepted. Explicit failed/unsupported+// collection, including expired last-known values, is never live evidence.+func hostDiskTemperatureObserved(disk models.HostDiskSMART) bool {+ if disk.Temperature <= 0 || disk.Standby {+ return false+ }+ return disk.Collection == nil || disk.Collection.Temperature.State == "" ||+ disk.Collection.Temperature.State == diskinventory.FieldAvailable+}++func (m *Manager) hostDiskTemperaturePendingNoLock(resourceID string) bool {+ for _, state := range m.mirrorStatesNoLock() {+ if incident, ok := state.Incident(resourceID, canonicalMetricSpecID(resourceID, "diskTemperature")); ok && incident.State == reducer.StatePending {+ return true+ }+ }+ _, pending := m.intentPending[canonicalMetricStateID(resourceID, "diskTemperature")]+ return pending+}++// Keep only the identity of a live pending run, not another telemetry cache.+func (m *Manager) rememberHostDiskTemperaturePending(host models.Host, disk models.HostDiskSMART, resourceID string) {+ m.mu.Lock()+ defer m.mu.Unlock()+ if !m.hostDiskTemperaturePendingNoLock(resourceID) {+ delete(m.hostDiskTempPendingContexts, resourceID)+ return+ }+ if m.hostDiskTempPendingContexts == nil {+ m.hostDiskTempPendingContexts = make(map[string]hostDiskTempContext)+ }+ m.hostDiskTempPendingContexts[resourceID] = hostDiskTempContext{host.ID, host.LinkedNodeID, host.LinkedVMID, host.LinkedContainerID, disk.Type}+}++func (m *Manager) hostDiskTemperatureTrackedResourcesNoLock() map[string]struct{} {+ resources := make(map[string]struct{})+ for resourceID := range m.hostDiskTempPendingContexts {+ resources[resourceID] = struct{}{}+ }+ for _, state := range m.mirrorStatesNoLock() {+ for _, resourceID := range state.PendingResourceIDs() {+ resources[resourceID] = struct{}{}+ }+ }+ for _, alert := range m.activeAlerts {+ if alert != nil {+ resources[alert.ResourceID] = struct{}{}+ }+ }+ for _, pending := range m.intentPending {+ if pending.Signal == MetricAlertIntentSignal("diskTemperature") {+ resources[pending.ResourceID] = struct{}{}+ }+ }+ return resources+}++func (m *Manager) interruptHostDiskTemperatureRunNoLock(resourceID string) bool {+ delete(m.hostDiskTempPendingContexts, resourceID)+ return m.interruptMetricRunNoLock(resourceID, canonicalMetricSpecID(resourceID, "diskTemperature"), canonicalMetricStateID(resourceID, "diskTemperature"))+}++func (m *Manager) interruptHostDiskTemperatureRun(resourceID string) {+ m.mu.Lock()+ changed := m.interruptHostDiskTemperatureRunNoLock(resourceID)+ m.mu.Unlock()+ if changed {+ m.saveActiveAlertsAsync("disk temperature observation gap")+ }+}++func (m *Manager) interruptHostDiskTemperatureRuns(hostID string) {++ m.interruptHostDiskTemperatureRunsExcept(hostID, nil)++}++++// Live shared SMART/Unraid rows are not observation gaps when SMART is empty.++func (m *Manager) interruptHostDiskTemperatureRunsExcept(hostID string, observed map[string]struct{}) {+ m.mu.Lock()+ changed := false+ for resourceID := range m.hostDiskTemperatureTrackedResourcesNoLock() {++ if _, present := observed[resourceID]; present {++ continue++ }+ if strings.HasPrefix(resourceID, hostDiskTemperatureResourcePrefix(hostID)) {+ changed = m.interruptHostDiskTemperatureRunNoLock(resourceID) || changed+ }+ }+ // Empty/expired inventory is not another confirmation that a disk left.+ for resourceID := range m.hostDiskTempAbsences {+ if strings.HasPrefix(resourceID, hostDiskTemperatureResourcePrefix(hostID)) {+ delete(m.hostDiskTempAbsences, resourceID)+ }+ }+ m.mu.Unlock()+ if changed {+ m.saveActiveAlertsAsync("host disk temperature observation gap")+ }+}++// Caller holds m.mu after config normalisation. Active alerts still use the+// existing per-type threshold re-evaluation; this only drops pending evidence.+func (m *Manager) reevaluateHostDiskTemperaturePendingNoLock() bool {+ intentChanged := false+ for resourceID := range m.hostDiskTemperatureTrackedResourcesNoLock() {+ if !strings.HasPrefix(resourceID, "agent:") || !strings.Contains(resourceID, "/disk_temp:") || !m.hostDiskTemperaturePendingNoLock(resourceID) {+ continue+ }+ ctx, known := m.hostDiskTempPendingContexts[resourceID]+ // Restarted intent grace has no retained link/type context. Do not+ // assume it was continuously enabled across a configuration save.+ if !known {+ intentChanged = m.interruptHostDiskTemperatureRunNoLock(resourceID) || intentChanged+ continue+ }+ thresholds := m.resolveHostThresholdsNoLock(ctx.hostID, ctx.linkedNodeID, ctx.linkedVMID, ctx.linkedContainerID)+ override, exists := m.hostThresholdOverrideNoLock(ctx.hostID, ctx.linkedNodeID, ctx.linkedVMID, ctx.linkedContainerID)+ threshold := m.hostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature, exists && override.DiskTemperature != nil, ctx.diskType)+ allDisabled, _ := m.alertPolicyTypeSwitchesNoLock("agent")+ if !m.config.Enabled || allDisabled || thresholds.Disabled || threshold == nil || threshold.Trigger <= 0 {+ intentChanged = m.interruptHostDiskTemperatureRunNoLock(resourceID) || intentChanged+ }+ }+ for resourceID := range m.hostDiskTempPendingContexts {+ if !m.hostDiskTemperaturePendingNoLock(resourceID) {+ delete(m.hostDiskTempPendingContexts, resourceID)+ }+ }+ return intentChanged+}diff --cc internal/alerts/host_unraid_lifecycle_test.goindex de8c3bf92f,a3c217fb20..17bed230e3--- a/internal/alerts/host_unraid_lifecycle_test.go+++ b/internal/alerts/host_unraid_lifecycle_test.go@@@ -8,10 -6,8 +8,11 @@@ import"testing""time"+ "github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"+ "github.com/rcourtman/pulse-go-rewrite/internal/alerts/reducer""github.com/rcourtman/pulse-go-rewrite/internal/models"+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"+ "github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory")func TestHostCustomSensorAlertLifecycle(t *testing.T) {@@@ -896,573 -1189,3 +1194,632 @@@ func TestHostChildAlertsSurviveProxmoxN}}}++// Disabling between two reports is a real policy interval even when there is+// no firing alert for UpdateConfig to visit. Linked overrides keep precedence.+func TestDiskTemperaturePendingConfigDisablement(t *testing.T) {+ for _, explicit := range []bool{false, true} {+ for _, scope := range []string{"global", "agent-default", "all-agents", "host", "host-disabled", "node", "vm", "container", "type", "unrelated"} {+ t.Run(fmt.Sprintf("%s/explicit=%v", scope, explicit), func(t *testing.T) {+ m, elapsed := continuityManager(t, explicit)+ host := hostWithSMARTDiskTemp("pending-temp", "custom", 85)+ host.LinkedNodeID, host.LinkedVMID, host.LinkedContainerID = "site:node", "site:node:101", "site:node:102"+ id := hostDiskTemperatureResourceID(host.ID, host.Sensors.SMART[0].Device)+ m.CheckHost(host)+ if incident, ok := continuityIncident(m, id, "diskTemperature"); !ok || incident.State != reducer.StatePending {+ t.Fatalf("no initial pending run: %+v %v", incident, ok)+ }+ cfg := m.GetConfig()+ switch scope {+ case "global":+ cfg.Enabled = false+ case "agent-default":+ cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 0}+ case "host-disabled":+ cfg.Overrides[host.ID] = ThresholdConfig{Disabled: true}+ case "all-agents":+ cfg.DisableAllAgents = true+ case "host":+ cfg.Overrides[host.ID] = ThresholdConfig{DiskTemperature: &HysteresisThreshold{Trigger: 0}}+ case "node":+ cfg.Overrides[host.LinkedNodeID] = ThresholdConfig{DiskTemperature: &HysteresisThreshold{Trigger: 0}}+ case "vm":+ host.LinkedNodeID = "" // The linked node override must not mask the VM.+ m.CheckHost(host)+ cfg.Overrides[host.LinkedVMID] = ThresholdConfig{DiskTemperature: &HysteresisThreshold{Trigger: 0}}+ case "container":+ host.LinkedNodeID, host.LinkedVMID = "", ""+ m.CheckHost(host)+ cfg.Overrides[host.LinkedContainerID] = ThresholdConfig{DiskTemperature: &HysteresisThreshold{Trigger: 0}}+ case "type":+ cfg.DiskTempByType = map[string]HysteresisThreshold{"custom": {Trigger: 0}}+ case "unrelated":+ cfg.Schedule.QuietHours.Timezone = "UTC"+ }+ m.UpdateConfig(cfg)+ _, pending := continuityIncident(m, id, "diskTemperature")+ if pending != (scope == "unrelated") {+ t.Fatalf("after save pending=%v; unrelated saves alone must preserve evidence", pending)+ }+ if scope == "unrelated" {+ return+ }+ cfg.Enabled, cfg.DisableAllAgents = true, false+ cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 80, Clear: 70}+ cfg.Overrides = map[string]ThresholdConfig{}+ cfg.DiskTempByType = map[string]HysteresisThreshold{"custom": {Trigger: 80, Clear: 70}}+ m.UpdateConfig(cfg)+ elapsed.Store(int64(2 * time.Minute))+ m.CheckHost(host)+ if len(m.GetActiveAlerts()) != 0 {+ t.Fatal("re-enable inherited the pre-disable start")+ }+ elapsed.Store(int64(2*time.Minute + 59*time.Second))+ m.CheckHost(host)+ if len(m.GetActiveAlerts()) != 0 {+ t.Fatal("fresh grace fired early")+ }+ elapsed.Store(int64(3 * time.Minute))+ m.CheckHost(host)+ if len(m.GetActiveAlerts()) != 1 {+ t.Fatal("fresh continuous breach never fired")+ }+ m.mu.RLock()+ defer m.mu.RUnlock()+ if len(m.hostDiskTempPendingContexts) != 0 {+ t.Fatal("context outlived the pending run")+ }+ })+ }+ }+}++func TestDiskTemperaturePendingRestartContextIsNotInvented(t *testing.T) {+ m, elapsed := continuityManager(t, true)+ host := hostWithSMARTDiskTemp("restored-temp", "sata", 85)+ id := hostDiskTemperatureResourceID(host.ID, host.Sensors.SMART[0].Device)+ m.CheckHost(host)+ m.mu.Lock()+ // Persisted intent carries grace but not host link/type context. Reducer+ // pending incidents are not restored; this is the actual restart shape.+ m.core.Reset()+ m.hostDiskTempPendingContexts = nil+ key := canonicalMetricStateID(id, "diskTemperature")+ _, pending := m.intentPending[key]+ m.mu.Unlock()+ if !pending {+ t.Fatal("explicit intent control did not retain grace")+ }+ m.UpdateConfig(m.GetConfig())+ m.mu.RLock()+ _, pending = m.intentPending[key]+ m.mu.RUnlock()+ if pending {+ t.Fatal("save retained grace whose link/type enablement is unknown")+ }+ elapsed.Store(int64(2 * time.Minute))+ m.CheckHost(host)+ if len(m.GetActiveAlerts()) != 0 {+ t.Fatal("restored grace fired without fresh evidence")+ }+}++func TestDiskTemperatureCurrentCollectionAndOverrideControls(t *testing.T) {+ for _, provenance := range []string{"legacy", "available", "other-field-only"} {+ t.Run(provenance, func(t *testing.T) {+ m, elapsed := continuityManager(t, false)+ host := hostWithSMARTDiskTemp("current-temp", "sata", 85)+ switch provenance {+ case "available":+ host.Sensors.SMART[0].Collection = &diskinventory.CollectionStatus{Temperature: diskinventory.Available("smartctl")}+ case "other-field-only":+ host.Sensors.SMART[0].Collection = &diskinventory.CollectionStatus{Serial: diskinventory.Available("smartctl")}+ }+ m.CheckHost(host)+ elapsed.Store(int64(time.Minute))+ m.CheckHost(host)+ if len(m.GetActiveAlerts()) != 1 {+ t.Fatal("continuous accepted temperature did not fire")+ }+ })+ }+ t.Run("host override wins over disabled type", func(t *testing.T) {+ m, elapsed := continuityManager(t, false)+ host := hostWithSMARTDiskTemp("override-temp", "custom", 85)+ cfg := m.GetConfig()+ cfg.DiskTempByType = map[string]HysteresisThreshold{"custom": {Trigger: 0}}+ cfg.Overrides[host.ID] = ThresholdConfig{DiskTemperature: &HysteresisThreshold{Trigger: 80, Clear: 70}}+ m.UpdateConfig(cfg)+ m.CheckHost(host)+ m.UpdateConfig(m.GetConfig())+ elapsed.Store(int64(time.Minute))+ m.CheckHost(host)+ if len(m.GetActiveAlerts()) != 1 {+ t.Fatal("save discarded an enabled explicit override")+ }+ })+}++func TestDiskTemperaturePendingContextCleanup(t *testing.T) {+ for _, removal := range []bool{false, true} {+ t.Run(fmt.Sprintf("removed=%v", removal), func(t *testing.T) {+ m, _ := continuityManager(t, false)+ host := hostWithSMARTDiskTemp("pending-cleanup", "custom", 85)+ m.CheckHost(host)+ m.mu.RLock()+ tracked := len(m.hostDiskTempPendingContexts)+ m.mu.RUnlock()+ if tracked != 1 || len(m.GetActiveAlerts()) != 0 {+ t.Fatal("control must have pending context and no active alert")+ }+ if removal {+ m.HandleHostRemoved(host)+ } else {+ m.ClearActiveAlerts()+ }+ m.mu.RLock()+ defer m.mu.RUnlock()+ if len(m.hostDiskTempPendingContexts) != 0 || m.hostDiskTemperaturePendingNoLock(hostDiskTemperatureResourceID(host.ID, host.Sensors.SMART[0].Device)) {+ t.Fatal("pending context/state outlived its host or explicit clear")+ }+ })+ }+}++// Feed only synthetic telemetry through the real CheckHost, config and callback+// paths. No collector, smartctl process, native host or notification provider.+type smartLifecycleFixture struct {+ m *Manager+ host models.Host+ healthID, wearID string+ initial map[string]Alert+ fires atomic.Int32+ recoveries chan *ResolvedAlert+}++func newSMARTLifecycleFixture(t *testing.T) *smartLifecycleFixture {+ t.Helper()+ m := newEventLogManager(t)+ cfg := m.GetConfig()+ cfg.FlappingEnabled = false+ cfg.Schedule.Cooldown = 0+ cfg.Schedule.MaxAlertsHour = 0+ cfg.AgentDefaults.Disk = &HysteresisThreshold{Trigger: 80, Clear: 70}+ cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 80, Clear: 70}+ cfg.DiskTempByType = map[string]HysteresisThreshold{"sata": {Trigger: 80, Clear: 70}}+ m.UpdateConfig(cfg)+ m.mu.Lock()+ m.config.TimeThresholds = map[string]int{}+ m.config.MetricTimeThresholds = map[string]map[string]int{"all": {"disk": 0, "diskTemperature": 0}}+ m.mu.Unlock()+ used, pending := 96, int64(2)+ host := models.Host{ID: "smart-lifecycle", Hostname: "storage", Status: "online",+ Disks: []models.Disk{{Mountpoint: "/data", Device: "/dev/sda1", Total: 1000, Used: 950, Usage: 95}},+ Sensors: models.HostSensorSummary{SMART: []models.HostDiskSMART{+ {Device: "/dev/sda", Type: "sata", Health: "FAILED", Temperature: 95, Attributes: &models.SMARTAttributes{PercentageUsed: &used, PendingSectors: &pending}},+ {Device: "/dev/sdb", Type: "sata", Health: "PASSED", Temperature: 20},+ }},+ }+ resourceID, _ := hostSMARTDiskResourceID(host, host.Sensors.SMART[0])+ f := &smartLifecycleFixture{m: m, host: host,+ healthID: buildCanonicalStateID(resourceID, resourceID+"-disk-health"),+ wearID: buildCanonicalStateID(resourceID, resourceID+"-disk-wearout"),+ initial: make(map[string]Alert), recoveries: make(chan *ResolvedAlert, 32),+ }+ m.SetAlertCallback(func(a *Alert) {+ if a.ID == f.healthID || a.ID == f.wearID {+ f.fires.Add(1)+ }+ })+ m.SetResolvedAlertCallback(func(r *ResolvedAlert) {+ if r.Alert.ID == f.healthID || r.Alert.ID == f.wearID {+ f.recoveries <- r+ }+ })+ m.CheckHost(host)+ for _, id := range []string{f.healthID, f.wearID} {+ f.initial[id] = *testRequireActiveAlert(t, m, id)+ }+ if f.fires.Load() != 2 {+ t.Fatalf("initial risk callbacks = %d, want 2", f.fires.Load())+ }+ return f+}++func (f *smartLifecycleFixture) held(t *testing.T) {+ t.Helper()+ for _, id := range []string{f.healthID, f.wearID} {+ a := testRequireActiveAlert(t, f.m, id)+ if !a.StartTime.Equal(f.initial[id].StartTime) {+ t.Fatalf("%s was replaced by a new occurrence", id)+ }+ }+ if f.fires.Load() != 2 {+ t.Fatalf("missing report refired risk notifications: %d", f.fires.Load())+ }+ for _, e := range queryAlertEvents(t, f.m, eventlog.Filter{Types: []string{eventlog.TypeResolved}}) {+ if e.AlertID == f.healthID || e.AlertID == f.wearID {+ t.Fatalf("missing evidence published recovery: %+v", e)+ }+ }+}++func (f *smartLifecycleFixture) resolved(t *testing.T, ids ...string) {+ t.Helper()+ want := make(map[string]bool)+ for _, id := range ids {+ if testHasActiveAlert(t, f.m, id) {+ t.Fatalf("risk alert remains active: %s", id)+ }+ want[id] = true+ }+ for range ids {+ select {+ case r := <-f.recoveries:+ if !want[r.Alert.ID] || !r.Alert.StartTime.Equal(f.initial[r.Alert.ID].StartTime) {+ t.Fatalf("wrong recovery occurrence: %+v", r)+ }+ delete(want, r.Alert.ID)+ case <-time.After(3 * time.Second):+ t.Fatal("ordinary recovery callback did not complete")+ }+ }+ for _, id := range ids {+ events := queryAlertEvents(t, f.m, eventlog.Filter{AlertID: id, Types: []string{eventlog.TypeResolved}})+ if len(events) != 1 {+ t.Fatalf("%s has %d resolution events, want 1", id, len(events))+ }+ }+}++func TestHostSMARTRiskMissingReportsPreserveOccurrenceAndCallbacks(t *testing.T) {+ f := newSMARTLifecycleFixture(t)+ if err := f.m.AcknowledgeAlert(f.healthID, "operator"); err != nil {+ t.Fatal(err)+ }+ present := f.host.Sensors.SMART+ // Both risk families on one disk must count once per report, not twice.+ for _, list := range [][]models.HostDiskSMART{nil, {present[1]}, {present[1]}, nil, {present[1]}, present, nil, present} {+ f.host.Sensors.SMART = list+ f.m.CheckHost(f.host)+ f.held(t)+ if !testRequireActiveAlert(t, f.m, f.healthID).Acknowledged {+ t.Fatal("acknowledgement lost during observation gap")+ }+ }+ // Standby and unavailable health/wear fields are presence, not recovery.+ f.host.Sensors.SMART = []models.HostDiskSMART{{Device: "/dev/sda", Standby: true}, present[1]}+ f.m.CheckHost(f.host)+ f.held(t)+ f.host.Sensors.SMART[0].Standby = false+ f.m.CheckHost(f.host)+ f.held(t)+ used, pending := 10, int64(0)+ f.host.Sensors.SMART[0] = models.HostDiskSMART{Device: "/dev/sda", Health: "PASSED", Temperature: 20, Attributes: &models.SMARTAttributes{PercentageUsed: &used, PendingSectors: &pending}}+ f.m.CheckHost(f.host)+ f.resolved(t, f.healthID, f.wearID)+ if f.fires.Load() != 2 {+ t.Fatalf("healthy recovery refired: %d", f.fires.Load())+ }+ t.Log("empty/partial/standby/unknown reports preserved occurrence and acknowledgement; fresh complete evidence recovered once")+}++func TestHostSMARTRiskDepartureRequiresFreshConsecutiveReports(t *testing.T) {+ for _, gap := range []string{"empty", "expiry", "present"} {+ t.Run(gap, func(t *testing.T) {+ f := newSMARTLifecycleFixture(t)+ present := f.host.Sensors.SMART+ f.host.Sensors.SMART = []models.HostDiskSMART{present[1]}+ for i := 0; i < 2; i++ {+ f.m.CheckHost(f.host)+ f.held(t)+ }+ switch gap {+ case "empty":+ f.host.Sensors.SMART = nil+ f.m.CheckHost(f.host)+ case "expiry":+ f.m.HandleHostTelemetryExpired(f.host)+ case "present":+ f.host.Sensors.SMART = present+ f.m.CheckHost(f.host)+ }+ f.held(t)+ f.host.Sensors.SMART = []models.HostDiskSMART{present[1]}+ for i := 1; i <= 3; i++ {+ f.m.CheckHost(f.host)+ if i < 3 {+ f.held(t)+ }+ }+ f.resolved(t, f.healthID, f.wearID)+ if f.fires.Load() != 2 {+ t.Fatalf("departure refired %d callbacks", f.fires.Load())+ }+ })+ }+}++func TestHostSMARTRiskRulesDisableBetweenReports(t *testing.T) {+ f := newSMARTLifecycleFixture(t)+ cfg := f.m.GetConfig()+ off := 0+ cfg.AgentDefaults.SMARTHealthFailure = &off+ f.m.UpdateConfig(cfg)+ f.held(t) // Pending sectors remain enabled; disabling only one cause is not recovery.+ zero := int64(0)+ cfg.AgentDefaults.SMARTPending = &zero+ f.m.UpdateConfig(cfg)+ f.resolved(t, f.healthID)+ if !testHasActiveAlert(t, f.m, f.wearID) {+ t.Fatal("health policy disabled unrelated wearout")+ }+ // A missing report cannot resurrect it. Re-enable before another report;+ // only fresh failing evidence may reactivate it, under the unchanged+ // stateful refire/cooldown history policy.+ f.host.Sensors.SMART = nil+ f.m.CheckHost(f.host)+ if testHasActiveAlert(t, f.m, f.healthID) {+ t.Fatal("missing report raised disabled health")+ }+ on, one := 1, int64(1)+ cfg.AgentDefaults.SMARTHealthFailure = &on+ cfg.AgentDefaults.SMARTPending = &one+ f.m.UpdateConfig(cfg)+ f.m.CheckHost(f.host)+ if testHasActiveAlert(t, f.m, f.healthID) {+ t.Fatal("re-enable raised health without evidence")+ }+ used, pending := 96, int64(2)+ f.host.Sensors.SMART = []models.HostDiskSMART{{Device: "/dev/sda", Health: "FAILED", Attributes: &models.SMARTAttributes{PercentageUsed: &used, PendingSectors: &pending}}}+ f.m.CheckHost(f.host)+ a := testRequireActiveAlert(t, f.m, f.healthID)+ if !a.StartTime.Equal(f.initial[f.healthID].StartTime) || !a.LastSeen.After(f.initial[f.healthID].LastSeen) || f.fires.Load() != 3 {+ t.Fatalf("fresh evidence must refire once within the existing cooldown: start %v, prior %v, seen %v, prior %v, callbacks %d", a.StartTime, f.initial[f.healthID].StartTime, a.LastSeen, f.initial[f.healthID].LastSeen, f.fires.Load())+ }+}++func TestHostSMARTRiskEmptyReportHonoursRuleOffAndLegacyUnknowns(t *testing.T) {+ for _, metadata := range []string{"current", "legacy", "unknown"} {+ t.Run(metadata, func(t *testing.T) {+ f := newSMARTLifecycleFixture(t)+ off, zero := 0, int64(0)+ f.m.mu.Lock()+ f.m.config.AgentDefaults.SMARTHealthFailure = &off+ f.m.config.AgentDefaults.SMARTPending = &zero+ if metadata != "current" {+ a, _ := f.m.getActiveAlertNoLock(f.healthID)+ if metadata == "legacy" {+ delete(a.Metadata, "riskCodes")+ } else {+ a.Metadata["riskCodes"] = []interface{}{"future-code"}+ }+ }+ f.m.mu.Unlock()+ f.host.Sensors.SMART = nil+ f.m.CheckHost(f.host)+ if metadata == "current" {+ f.resolved(t, f.healthID)+ } else {+ f.held(t)+ }+ if !testHasActiveAlert(t, f.m, f.wearID) {+ t.Fatal("wearout was disabled with health")+ }+ })+ }+}++func TestHostSMARTRiskFilesystemAndHostLifecycleStayIndependent(t *testing.T) {+ for _, action := range []string{"filesystem-absent", "node-link", "removed", "host-disabled", "agents-disabled", "host-isolation"} {+ t.Run(action, func(t *testing.T) {+ f := newSMARTLifecycleFixture(t)+ switch action {+ case "filesystem-absent":+ fsID, _ := hostDiskResourceID(f.host, f.host.Disks[0])+ id := canonicalMetricStateID(fsID, "disk")+ testRequireActiveAlert(t, f.m, id)+ f.host.Disks = nil+ f.host.Sensors.SMART = nil+ f.m.CheckHost(f.host)+ if testHasActiveAlert(t, f.m, id) {+ t.Fatal("filesystem absence incorrectly held usage")+ }+ f.held(t)+ case "host-isolation":+ other := f.host+ other.ID = "smart-lifecycle-other"+ f.m.CheckHost(other)+ other.Sensors.SMART = []models.HostDiskSMART{other.Sensors.SMART[1]}+ for i := 0; i < 3; i++ {+ f.m.CheckHost(other)+ }+ f.held(t)+ default:+ present := f.host.Sensors.SMART+ f.host.Sensors.SMART = []models.HostDiskSMART{present[1]}+ f.m.CheckHost(f.host)+ f.held(t)+ switch action {+ case "node-link":+ f.host.LinkedNodeID = "pve-node"+ f.host.Sensors.SMART = nil+ f.m.CheckHost(f.host)+ case "removed":+ f.m.HandleHostRemoved(f.host)+ case "host-disabled":+ cfg := f.m.GetConfig()+ cfg.Overrides[f.host.ID] = ThresholdConfig{Disabled: true}+ f.m.UpdateConfig(cfg)+ case "agents-disabled":+ cfg := f.m.GetConfig()+ cfg.DisableAllAgents = true+ f.m.UpdateConfig(cfg)+ }+ f.resolved(t, f.healthID, f.wearID)+ // Returning ownership gets a new occurrence and a fresh departure run.+ f.host.LinkedNodeID = ""+ cfg := f.m.GetConfig()+ cfg.DisableAllAgents = false+ delete(cfg.Overrides, f.host.ID)+ f.m.UpdateConfig(cfg)+ f.host.Sensors.SMART = present+ f.m.CheckHost(f.host)+ f.host.Sensors.SMART = []models.HostDiskSMART{present[1]}+ for i := 0; i < 2; i++ {+ f.m.CheckHost(f.host)+ testRequireActiveAlert(t, f.m, f.healthID)+ testRequireActiveAlert(t, f.m, f.wearID)+ }+ }+ })+ }+}++func TestHostSMARTRiskFilesystemIdentityCannotSubstituteInventory(t *testing.T) {+ for _, direction := range []string{"SMART-only", "filesystem-only"} {+ t.Run(direction, func(t *testing.T) {+ f := newSMARTLifecycleFixture(t)+ f.host.Disks[0].Mountpoint = "/sda"+ f.m.CheckHost(f.host)+ diskResourceID, _ := hostDiskResourceID(f.host, f.host.Disks[0])+ riskResourceID, _ := hostSMARTDiskResourceID(f.host, f.host.Sensors.SMART[0])+ if diskResourceID != riskResourceID {+ t.Fatal("control must share the disk resource ID")+ }+ usageID := canonicalMetricStateID(diskResourceID, "disk")+ testRequireActiveAlert(t, f.m, usageID)+ if direction == "SMART-only" {+ f.host.Disks = nil+ f.m.CheckHost(f.host)+ if testHasActiveAlert(t, f.m, usageID) {+ t.Fatal("SMART identity held missing filesystem usage")+ }+ f.held(t)+ } else {+ f.host.Sensors.SMART = []models.HostDiskSMART{f.host.Sensors.SMART[1]}+ for i := 1; i <= 3; i++ {+ f.m.CheckHost(f.host)+ testRequireActiveAlert(t, f.m, usageID)+ if i < 3 {+ f.held(t)+ }+ }+ f.resolved(t, f.healthID, f.wearID)+ }+ })+ }+}++func TestHostSMARTRiskRestartAndGlobalPauseRestartDeparture(t *testing.T) {+ for _, gap := range []string{"restart", "global-pause"} {+ t.Run(gap, func(t *testing.T) {+ f := newSMARTLifecycleFixture(t)+ if err := f.m.AcknowledgeAlert(f.healthID, "operator"); err != nil {+ t.Fatal(err)+ }+ f.host.Sensors.SMART = []models.HostDiskSMART{f.host.Sensors.SMART[1]}+ for i := 0; i < 2; i++ {+ f.m.CheckHost(f.host)+ f.held(t)+ }+ cfg := f.m.GetConfig()+ if gap == "restart" {+ dir := filepath.Dir(f.m.alertsDir)+ f.m.Stop() // Actual checkpoint and ordinary persisted-alert restoration.+ f.m = NewManagerWithDataDir(dir)+ t.Cleanup(f.m.Stop)+ store, err := eventlog.OpenInMemory()+ if err != nil {+ t.Fatal(err)+ }+ f.m.SetEventLog(store) // Observe post-restart lifecycle, not nonexistent old in-memory events.+ f.m.UpdateConfig(cfg)+ f.m.SetAlertCallback(func(a *Alert) {+ if a.ID == f.healthID || a.ID == f.wearID {+ f.fires.Add(1)+ }+ })+ f.m.SetResolvedAlertCallback(func(r *ResolvedAlert) {+ if r.Alert.ID == f.healthID || r.Alert.ID == f.wearID {+ f.recoveries <- r+ }+ })+ } else {+ cfg.Enabled = false+ f.m.UpdateConfig(cfg)+ cfg.Enabled = true+ f.m.UpdateConfig(cfg)+ }+ for i := 1; i <= 3; i++ {+ f.m.CheckHost(f.host)+ if i < 3 {+ f.held(t)+ if !testRequireActiveAlert(t, f.m, f.healthID).Acknowledged {+ t.Fatal("gap lost persisted acknowledgement")+ }+ }+ }+ f.resolved(t, f.healthID, f.wearID)+ })+ }+}++++// Upstream's shared Unraid readings must retain reviewed pending-grace++// ownership even when no SMART row exists; genuine gaps still restart grace.++func TestUnraidOnlyTemperatureRetainsPendingContinuity(t *testing.T) {++ for _, explicit := range []bool{false, true} {++ for _, gap := range []string{"continuous", "spun-down", "omitted", "type-off"} {++ t.Run(fmt.Sprintf("%s/explicit=%v", gap, explicit), func(t *testing.T) {++ m, elapsed := continuityManager(t, explicit)++ disk := models.HostUnraidDisk{Name: "disk1", Device: "sda", Serial: "CONTINUITY-1", Transport: "sata", Temperature: 85}++ host := unraidTempHost(nil, disk)++ id := hostDiskTemperatureResourceID(host.ID, "sda")++ m.CheckHost(host)++ if incident, ok := continuityIncident(m, id, "diskTemperature"); !ok || incident.State != reducer.StatePending {++ t.Fatalf("no initial pending Unraid run: %+v %v", incident, ok)++ }++ elapsed.Store(int64(30 * time.Second))++ switch gap {++ case "continuous":++ m.CheckHost(host)++ case "spun-down":++ host.Unraid.Disks[0].Temperature = 0++ m.CheckHost(host)++ host.Unraid.Disks[0] = disk++ case "omitted":++ host.Unraid.Disks = nil++ m.CheckHost(host)++ host.Unraid.Disks = []models.HostUnraidDisk{disk}++ case "type-off":++ cfg := m.GetConfig()++ cfg.DiskTempByType = map[string]HysteresisThreshold{"sata": {Trigger: 0}}++ m.UpdateConfig(cfg)++ cfg.DiskTempByType = map[string]HysteresisThreshold{"sata": {Trigger: 80, Clear: 70}}++ m.UpdateConfig(cfg)++ }++ elapsed.Store(int64(time.Minute))++ m.CheckHost(host)++ if gap == "continuous" {++ if len(m.GetActiveAlerts()) != 1 {++ t.Fatal("live Unraid-only reports lost continuous pending grace")++ }++ return++ }++ if len(m.GetActiveAlerts()) != 0 {++ t.Fatal("Unraid gap inherited earlier pending grace")++ }++ elapsed.Store(int64(119 * time.Second))++ m.CheckHost(host)++ if len(m.GetActiveAlerts()) != 0 {++ t.Fatal("fresh Unraid grace fired early")++ }++ elapsed.Store(int64(2 * time.Minute))++ m.CheckHost(host)++ if len(m.GetActiveAlerts()) != 1 {++ t.Fatal("fresh continuous Unraid breach did not fire")++ }++ })++ }++ }++}