Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Preserve reviewed monitoring repairs with current filesystem alert ownership
pulse-triage[bot]9c378ffa942e
diff --cc docs/release-control/v6/internal/subsystems/alerts.mdindex 3e31d17fa7,1b3fd083fd..625b94fdce--- a/docs/release-control/v6/internal/subsystems/alerts.md+++ b/docs/release-control/v6/internal/subsystems/alerts.md@@@ -4020,91 -4000,3 +4090,103 @@@ display name differs from its hostnamehad no alert highlighting at all. `getAlertStyles` keeps itsid-plus-node-name matching for the platform tables whose alerts are keyed thatway.++### SMART temperature continuity and pending-only policy intervals++Disk temperature activation/grace and recovery require continuous accepted+observations. Empty SMART inventory, one disk omission, standby, non-positive+readings, explicit missing/unavailable/unsupported collection and reporting-lease+expiry interrupt pending/recovery timing without changing an open alert, its+acknowledgement or lifecycle events. Positive legacy readings without temperature+provenance remain accepted; an expired last-known positive value is not live+health. Canonical and shadow reducers plus explicit intent grace are interrupted+together. A non-empty inventory still resolves a departed disk only after three+consecutive omissions; empty/expired inventory breaks that confirmation sequence.++Configuration saves reset pending-only runs when global agent policy, host or+inherited node/guest overrides, or the applicable disk-type threshold disables+evaluation, even if no report arrives before re-enablement. Existing known-type+default normalisation is unchanged; the zero-type save control uses a custom+type whose zero threshold the normaliser preserves. Only live pending+identity/link/type context is retained; unrelated saves preserve a known enabled+run and explicit overrides retain precedence over type defaults. Restored intent+without that context restarts conservatively on a save rather than inventing+continuous enablement. Cleared intent grace is checkpointed. Existing active-alert+per-type/save/offline/removal resolution remains unchanged.++`metric_observation_continuity_test.go` and `host_unraid_lifecycle_test.go` cover+valid/legacy/provenance controls, missing/rejected/expiry gaps on both timing+edges, explicit grace, acknowledgement/event preservation, host isolation,+configuration intervals, override precedence and departure. These deterministic+source tests do not establish installed temperatures or native guest recovery.++### SMART risk inventory gaps and disabled rules++Agent `disk-health` and `disk-wearout` alerts follow the SMART inventory, not+filesystem usage cleanup. Empty inventory, standby and unobserved risk fields+hold the existing occurrence and acknowledgement without recovery/refiring.+Departure requires three consecutive non-empty reports omitting the disk;+listing it again, an empty inventory or reporting-lease expiry restarts that+confirmation. Both risk families on the same disk count once per report.+Filesystem absence still resolves usage on its first missing report. Temperature+pending/grace/recovery interruption and departure semantics remain unchanged.++Disabling every recorded cause releases that risk alert without a reading,+both on a configuration save and on an empty report. Partially enabled causes,+missing legacy cause metadata and unknown codes remain conservative. Re-enabling+cannot raise an alert from absent evidence; fresh bad evidence uses the existing+stateful refire/cooldown history policy. Node linking retains existing node ownership, including when the+agent's SMART inventory is empty; host removal and policy disablement release+agent risk state and stale absence counts. Counts are ephemeral and restart+conservatively after a process restart.++`TestHostSMARTRisk*` in `host_unraid_lifecycle_test.go` exercises actual `CheckHost`, expiry, configuration+save, firing/recovery callbacks and event history with synthetic telemetry.+It covers both families sharing a disk, interruption/reconfirmation, partial+rule disablement, re-enablement, legacy/unknown causes, filesystem and host+isolation, removal and node-link ownership. These are source lifecycle controls,+not collector, smartctl, installed notification or native disk-recovery proof.++### Storage observation gaps restart confirmation and timing++An absent or `unknown` storage connectivity status interrupts activation,+recovery confirmations and pending offline intent grace without resolving an+existing outage or changing its acknowledgement. Capacity is independently+observable while connectivity is unknown. Missing/unconfirmed-zero capacity,+negative usage and offline/unavailable storage interrupt capacity activation,+and intent grace timing; they do not supply healthy capacity evidence.+Capacity recovery retains its existing immediate measured-clear behaviour,+not the recovery stability window used by memory and temperature gauges.+A confirmed empty store still supplies a genuine zero-usage recovery reading.+Fresh observations must satisfy the existing full confirmation/delay after a+gap. Interruption is scoped to the storage's canonical identity and applied to+both live and shadow reducers; interrupted intent is checkpointed for restart.++`TestStorageConnectivityObservationGaps` and+`TestStorageConnectivityGapRestartsIntentGrace` exercise the real storage+entry point, acknowledged occurrence retention, neighbour isolation, fresh+reconfirmation and empty intent checkpoint. Storage routes in+`TestMetricObservationGapRestartsActivation` covers capacity gaps and explicit+intent grace; `TestStorageCapacityGapHoldsOccurrence` verifies retained,+acknowledged capacity incidents and genuine immediate empty-store recovery. Existing restart, empty-capacity and predictive-capacity controls+remain required. These source controls are not native storage recovery or+installed destination-delivery evidence.++`TestStorageConnectivityGapWebhook` also exercises the production Monitor+callbacks, ordinary notification queue and guest-local HTTP receiver: the gap+and first fresh healthy poll send no recovery; confirmed recovery sends one+resolved receipt with the original occurrence identity, and the queue drains+without failed/DLQ work. This is synthetic callback/receiver acceptance, not a+native appliance or external webhook-provider result.++++### Composed host filesystem threshold ownership++++The shared filesystem resolver preserves the explicit host or linked-resource++Disk override ahead of per-hardware-type defaults, for both host reports and++configuration-save reevaluation. A filesystem-specific override still takes++precedence. Disabling that filesystem releases its canonical usage metric and++pending intent, retaining the reviewed agent/node ownership semantics.++`TestHostDiskOverrideBeatsDiskFillByType` checks agent and inherited overrides++through a configuration save and the next report without losing the occurrence;++`TestConfigSaveJudgesFilesystemAlertsByTheirOwnThreshold` retains per-filesystem++and per-type controls. This is source composition, not installed delivery proof.diff --cc internal/alerts/host.goindex b46a2c00b8,db47b5bdf6..781969d9c4--- a/internal/alerts/host.go+++ b/internal/alerts/host.go@@@ -172,9 -172,50 +172,51 @@@ func (m *Manager) resolveHostAlertThresthresholds.DiskTemperature = m.lowestHostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature)}}+ // A filesystem alert is judged by the threshold CheckHost evaluates that+ // filesystem against, so a config save does not resolve an alert its+ // override or per-type fill threshold still holds. A disabled filesystem+ // resolves with no threshold, as CheckHost clears it.+ if alert != nil && alert.Type == "disk" && isHostFilesystemResourceID(resourceID) {- thresholds.Disk, _ = m.hostDiskUsageThresholdNoLock(strings.TrimSpace(resourceID), metadataStringValue(alert.Metadata, "device"), thresholds.Disk)++ override, exists := m.hostThresholdOverrideNoLock(hostID, linkedNodeID, linkedVMID, linkedContainerID)++ thresholds.Disk, _ = m.hostDiskUsageThresholdNoLock(strings.TrimSpace(resourceID), metadataStringValue(alert.Metadata, "device"), thresholds.Disk, exists && override.Disk != nil)+ }return thresholds}+ // isHostFilesystemResourceID reports whether resourceID names one agent+ // filesystem ("agent:<host>/disk:<label>") rather than the host itself.+ func isHostFilesystemResourceID(resourceID string) bool {+ _, child, ok := strings.Cut(stripHostResourcePrefix(resourceID), "/")+ return ok && strings.HasPrefix(child, "disk:")+ }++ // hostDiskUsageThresholdNoLock resolves the usage threshold one agent-// filesystem is judged by: the filesystem's own override, then the fill-// threshold for its hardware type while the host's disk alerting is on, then-// the host's disk threshold. disabled reports an override that switches the++// filesystem is judged by: the filesystem's own override, then an explicit++// host or inherited disk override, then the fill threshold for its hardware++// type while the host's disk alerting is on, then the host's disk threshold. disabled reports an override that switches the+ // filesystem's alerts off, and then the threshold is nil. CheckHost and+ // config-save reevaluation both resolve through it. An override or per-type+ // threshold is returned as an owned copy, so normalizing it never writes into+ // the live config. Callers must hold m.mu.-func (m *Manager) hostDiskUsageThresholdNoLock(diskResourceID, device string, hostDisk *HysteresisThreshold) (threshold *HysteresisThreshold, disabled bool) {++func (m *Manager) hostDiskUsageThresholdNoLock(diskResourceID, device string, hostDisk *HysteresisThreshold, hostDiskOverridden bool) (threshold *HysteresisThreshold, disabled bool) {+ if override, ok := m.config.Overrides[diskResourceID]; ok {+ if override.Disabled {+ return nil, true+ }+ if override.Disk != nil {+ return ensureHysteresisThreshold(cloneThreshold(override.Disk)), false+ }+ }- if hostDisk != nil && hostDisk.Trigger > 0 {++ if !hostDiskOverridden && hostDisk != nil && hostDisk.Trigger > 0 {+ if hwType := inferDiskHardwareType(device); hwType != "" {+ if byType, ok := m.config.DiskFillByType[hwType]; ok {+ return &byType, false+ }+ }+ }+ return hostDisk, false+ }+func sanitizeHostComponent(value string) string {value = strings.TrimSpace(strings.ToLower(value))if value == "" {@@@ -469,39 -491,12 +511,12 @@@ func (m *Manager) CheckHost(host modelsdiskResourceID, diskName := hostDiskResourceID(host, disk)seenDisks[diskResourceID] = struct{}{}- // Check for disk-specific overridem.mu.RLock()- diskOverride, hasDiskOverride := m.config.Overrides[diskResourceID]- effectiveDiskThreshold, diskDisabled := m.hostDiskUsageThresholdNoLock(diskResourceID, disk.Device, thresholds.Disk)++ effectiveDiskThreshold, diskDisabled := m.hostDiskUsageThresholdNoLock(diskResourceID, disk.Device, thresholds.Disk, diskOverridden)m.mu.RUnlock()-- // Determine the effective disk threshold- var effectiveDiskThreshold *HysteresisThreshold- if hasDiskOverride {- // If disk is disabled via override, skip alerting- if diskOverride.Disabled {- m.releaseHostUsageMetric(diskResourceID, diskName, nodeName, instanceName, unifiedresources.ResourceType("agent-disk"), "agent-disk", "disk")- continue- }- // Use disk-specific threshold if set- if diskOverride.Disk != nil {- effectiveDiskThreshold = ensureHysteresisThreshold(diskOverride.Disk)- }- }- // Per-type override: consult DiskFillByType if hardware type is inferable- // from the device path and no disk-specific override applied above.- if effectiveDiskThreshold == nil && !diskOverridden && thresholds.Disk != nil && thresholds.Disk.Trigger > 0 {- if hwType := inferDiskHardwareType(disk.Device); hwType != "" {- m.mu.RLock()- if th, ok := m.config.DiskFillByType[hwType]; ok {- t := th- effectiveDiskThreshold = &t- }- m.mu.RUnlock()- }- }- // Fall back to host-level threshold- if effectiveDiskThreshold == nil {- effectiveDiskThreshold = thresholds.Disk+ if diskDisabled {- m.clearAlert(canonicalMetricStateID(diskResourceID, "disk"))++ m.releaseHostUsageMetric(diskResourceID, diskName, nodeName, instanceName, unifiedresources.ResourceType("agent-disk"), "agent-disk", "disk")+ continue}// Skip if no threshold configured (nil)diff --cc internal/alerts/threshold_resolution_shared_test.goindex 314a235fad,fac87ae429..77f6dfd1d2--- a/internal/alerts/threshold_resolution_shared_test.go+++ b/internal/alerts/threshold_resolution_shared_test.go@@@ -1451,414 -1563,6 +1568,429 @@@ func TestDiskTemperatureThresholdMatche}}+// An agent owns a usage metric only with usable evidence: it evaluated the+// metric this report, or it still holds an alert or run for it. Configuration+// alone is not coverage.+func TestCheckNodeKeepsMetricsTheAgentCannotEvaluate(t *testing.T) {+ setup := func(t *testing.T) (*Manager, models.Node, models.Host) {+ m := newTestManager(t)+ m.mu.Lock()+ m.config.Enabled = true+ m.config.TimeThresholds = map[string]int{}+ m.config.NodeDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}+ m.config.NodeDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}+ m.config.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}+ m.config.AgentDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}+ m.mu.Unlock()+ node, host := testNodeWithHostAgent()+ node.CPU = 0.95+ node.Memory = models.Memory{Total: 100, Used: 95, Free: 5, Usage: 95}+ return m, node, host+ }++ t.Run("memory_unknown_from_the_start", func(t *testing.T) {+ m, node, host := setup(t)+ host.Memory = models.Memory{Total: 100, UsageUnavailable: true}+ m.CheckHost(host)+ m.CheckNode(node)+ if !testHasActiveAlert(t, m, canonicalMetricStateID(node.ID, "memory")) {+ t.Fatalf("expected the node to alert on memory while the agent has no memory reading")+ }+ })++ t.Run("non_finite_cpu", func(t *testing.T) {+ m, node, host := setup(t)+ host.CPUUsage = math.NaN()+ m.CheckHost(host)+ m.CheckNode(node)+ if !testHasActiveAlert(t, m, canonicalMetricStateID(node.ID, "cpu")) {+ t.Fatalf("expected the node to alert on CPU while the agent reports no usable CPU value")+ }+ })+}++// Turning an agent threshold off releases its pending run even when the agent+// reports no reading for that metric, so the old run cannot keep ownership and+// leave the node silent.+func TestDisabledAgentThresholdReleasesPendingRunWithoutReading(t *testing.T) {+ m := newTestManager(t)+ m.mu.Lock()+ m.config.Enabled = true+ m.config.TimeThresholds = map[string]int{}+ m.config.MetricTimeThresholds = map[string]map[string]int{"agent": {"memory": 300}}+ m.config.NodeDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}+ m.config.AgentDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}+ m.mu.Unlock()+ node, host := testNodeWithHostAgent()+ // 90% stays below the 95% critical level, which would bypass the delay.+ node.Memory = models.Memory{Total: 100, Used: 90, Free: 10, Usage: 90}+ host.Memory = node.Memory+ agentResourceID := hostResourceID(host.ID)+ agentMemorySpec := canonicalMetricSpecID(agentResourceID, "memory")++ m.CheckHost(host)+ m.mu.RLock()+ pending := testCoreIsPending(m, agentResourceID, agentMemorySpec)+ m.mu.RUnlock()+ if !pending {+ t.Fatalf("expected a pending agent memory run")+ }++ m.mu.Lock()+ m.config.AgentDefaults.Memory = &HysteresisThreshold{Trigger: 0, Clear: 0}+ m.mu.Unlock()+ host.Memory = models.Memory{Total: 100, UsageUnavailable: true}+ m.CheckHost(host)+ m.CheckNode(node)++ m.mu.RLock()+ agentIncident := testCoreHasIncident(m, agentResourceID, agentMemorySpec)+ nodeIncident := testCoreHasIncident(m, node.ID, canonicalMetricSpecID(node.ID, "memory"))+ m.mu.RUnlock()+ if agentIncident {+ t.Fatalf("expected the disabled agent memory threshold to release its pending run")+ }+ if !nodeIncident {+ t.Fatalf("expected the node to evaluate memory once the agent no longer covers it")+ }+}++// Ownership follows the config at once and link updates apply in report order:+// a config change revokes what it no longer supports without waiting for the+// agent's next report, an older report can neither restore revoked ownership+// nor remove a newer report's link, and a report started before the agent went+// offline is stale.+func TestHostAgentNodeLinkFollowsConfigAndReportOrder(t *testing.T) {+ m := newTestManager(t)+ m.mu.Lock()+ m.config.Enabled = true+ m.config.TimeThresholds = map[string]int{}+ m.config.NodeDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}+ m.config.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}+ m.mu.Unlock()+ node, host := testNodeWithHostAgent()+ node.CPU = 0.95+ host.CPUUsage = 50++ m.CheckHost(host)+ m.CheckNode(node)+ nodeCPUSpec := canonicalMetricSpecID(node.ID, "cpu")+ m.mu.RLock()+ evaluatedBefore := testCoreHasIncident(m, node.ID, nodeCPUSpec)+ m.mu.RUnlock()+ if evaluatedBefore {+ t.Fatalf("expected the agent to own CPU before the config change")+ }++ // A report already in flight when agent alerts are disabled must not+ // restore ownership once it lands.+ inFlight := m.beginHostAgentReport(host.ID)++ // No agent report follows the config change; the node must evaluate CPU on+ // its very next check. (UpdateConfig restores the default activation delay,+ // so the run is pending rather than firing.)+ config := m.GetConfig()+ config.DisableAllAgents = true+ m.UpdateConfig(config)+ m.CheckNode(node)+ m.mu.RLock()+ evaluatedAfter := testCoreHasIncident(m, node.ID, nodeCPUSpec)+ m.mu.RUnlock()+ if !evaluatedAfter {+ t.Fatalf("expected the node to evaluate CPU right after agent alerts were disabled")+ }++ covering := hostAgentNodeLink{agentID: host.ID, nodeID: node.ID, cpu: true}+ m.applyHostAgentNodeLink(covering, inFlight)+ if m.hasHostAgentForNode(node.ID) {+ t.Fatalf("expected a report evaluated before agent alerts were disabled not to restore ownership")+ }++ config.DisableAllAgents = false+ m.UpdateConfig(config)+ older := m.beginHostAgentReport(host.ID)+ newer := m.beginHostAgentReport(host.ID)+ m.applyHostAgentNodeLink(covering, newer)+ m.applyHostAgentNodeLink(hostAgentNodeLink{agentID: host.ID, nodeID: node.ID}, older)+ if !m.hasHostAgentForNode(node.ID) {+ t.Fatalf("expected an older report not to remove the newer report's link")+ }++ olderCovering := m.beginHostAgentReport(host.ID)+ newerRemoving := m.beginHostAgentReport(host.ID)+ m.applyHostAgentNodeLink(hostAgentNodeLink{agentID: host.ID, nodeID: node.ID}, newerRemoving)+ m.applyHostAgentNodeLink(covering, olderCovering)+ if m.hasHostAgentForNode(node.ID) {+ t.Fatalf("expected an older report not to restore ownership a newer report removed")+ }++ beforeOffline := m.beginHostAgentReport(host.ID)+ m.HandleHostOffline(host)+ m.applyHostAgentNodeLink(covering, beforeOffline)+ if m.hasHostAgentForNode(node.ID) {+ t.Fatalf("expected a report started before the agent went offline not to re-register")+ }+}++// An agent that reports unlinked hands its former node the usage metrics back+// on the node's next check, even while the agent's own alert is still firing.+func TestCheckNodeTakesBackMetricsWhenAgentUnlinks(t *testing.T) {+ m := newTestManager(t)+ m.mu.Lock()+ m.config.Enabled = true+ m.config.TimeThresholds = map[string]int{}+ m.config.NodeDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}+ m.config.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}+ m.mu.Unlock()+ node, host := testNodeWithHostAgent()+ node.CPU = 0.95+ host.CPUUsage = 95++ m.CheckHost(host)+ m.CheckNode(node)+ nodeCPU := canonicalMetricStateID(node.ID, "cpu")+ if testHasActiveAlert(t, m, nodeCPU) {+ t.Fatalf("expected the linked agent to own CPU")+ }++ host.LinkedNodeID = ""+ m.CheckHost(host)+ m.CheckNode(node)+ if !testHasActiveAlert(t, m, nodeCPU) {+ t.Fatalf("expected the node to alert on CPU once its agent reports unlinked")+ }+}++// An explicit disk override, including one inherited from the linked node,+// beats the per-type DiskFillByType default; otherwise the node released its+// disk alert to an agent applying a looser per-type threshold.+func TestHostDiskOverrideBeatsDiskFillByType(t *testing.T) {+ for _, overrideOn := range []string{"linked node", "agent"} {+ t.Run(overrideOn, func(t *testing.T) {+ m := newTestManager(t)+ node, host := testNodeWithHostAgent()+ overrideID := node.ID+ if overrideOn == "agent" {+ overrideID = host.ID+ }+ m.mu.Lock()+ m.config.Enabled = true+ m.config.TimeThresholds = map[string]int{}+ m.config.AgentDefaults.Disk = &HysteresisThreshold{Trigger: 90, Clear: 85}+ m.config.DiskFillByType = map[string]HysteresisThreshold{"nvme": {Trigger: 92, Clear: 88}}+ m.config.Overrides = map[string]ThresholdConfig{+ overrideID: {Disk: &HysteresisThreshold{Trigger: 80, Clear: 75}},+ }+ m.mu.Unlock()+ root := models.Disk{Mountpoint: "/", Device: "/dev/nvme0n1p2", Total: 100, Used: 85, Free: 15, Usage: 85}+ host.Disks = []models.Disk{root}++ m.CheckHost(host)+ rootResourceID, _ := hostDiskResourceID(host, root)+ if !testHasActiveAlert(t, m, canonicalMetricStateID(rootResourceID, "disk")) {+ t.Fatalf("expected the 80%% %s disk override to raise the agent's root disk alert at 85%%", overrideOn)+ }++++ // Config-save reevaluation must preserve the same explicit override++ // when the upstream per-filesystem resolver is composed with agent++ // ownership. The looser NVMe default must not clear this occurrence.++ alertID := canonicalMetricStateID(rootResourceID, "disk")++ before, _ := testLookupActiveAlert(t, m, alertID)++ m.UpdateConfig(m.GetConfig())++ after, exists := testLookupActiveAlert(t, m, alertID)++ if !exists || after.StartTime != before.StartTime {++ t.Fatalf("config save lost the %s disk override occurrence", overrideOn)++ }++ m.CheckHost(host)++ if !testHasActiveAlert(t, m, alertID) {++ t.Fatalf("next host report lost the %s disk override alert", overrideOn)++ }+ })+ }+}++// Releasing a metric drops its explicit intent grace run too, so a later+// resume does not activate at once on time counted while nothing evaluated.+func TestReleaseCanonicalMetricAlertClearsIntentPending(t *testing.T) {+ m := newTestManager(t)+ spec, err := buildCanonicalMetricSpec("homelab-delly2", "delly2", "node", "cpu", nil)+ if err != nil {+ t.Fatalf("spec: %v", err)+ }+ trackingKey := canonicalTrackingKeyForSpec(spec, spec.ID)+ m.mu.Lock()+ m.intentPending[trackingKey] = IntentPendingState{+ TrackingKey: trackingKey,+ ResourceID: spec.ResourceID,+ Signal: MetricAlertIntentSignal("cpu"),+ FirstMatchedAt: time.Now().Add(-10 * time.Minute),+ LastObservedAt: time.Now().Add(-9 * time.Minute),+ }+ m.mu.Unlock()++ m.releaseCanonicalMetricAlert(spec, "delly2", "delly2", "homelab", "node", 0, nil)++ m.mu.RLock()+ _, stillPending := m.intentPending[trackingKey]+ m.mu.RUnlock()+ if stillPending {+ t.Fatalf("expected releasing the metric to clear its intent pending state")+ }+}++// Missing storage observations break confirmation runs, not the occurrence.+// Exercise the public evaluator and both live/shadow reducers, not seeded state.+func TestStorageConnectivityObservationGaps(t *testing.T) {+ for _, gap := range []string{"", "unknown", " UNKNOWN "} {+ t.Run("activation/"+gap, func(t *testing.T) {+ m := newShadowFeedManager(t)+ s := models.Storage{ID: "storage-a", Name: "backups", Status: "offline"}+ other := s+ other.ID = "storage-ab"+ id := canonicalConnectivityStateID(s.ID)+ m.CheckStorage(s)+ m.CheckStorage(other)+ s.Status = gap+ m.CheckStorage(s)+ s.Status = "unavailable"+ m.CheckStorage(s)+ if testHasActiveAlert(t, m, id) {+ t.Fatal("non-consecutive outage observations fired storage alert")+ }+ m.CheckStorage(other)+ testRequireActiveAlert(t, m, canonicalConnectivityStateID(other.ID))+ m.CheckStorage(s)+ testRequireActiveAlert(t, m, id)+ if m.ShadowDivergences() != 0 {+ t.Fatal("storage interruption diverged from shadow")+ }+ })+ t.Run("recovery/"+gap, func(t *testing.T) {+ m := newShadowFeedManager(t)+ s := models.Storage{ID: "storage-a", Name: "backups", Status: "offline"}+ id := canonicalConnectivityStateID(s.ID)+ for range 2 {+ m.CheckStorage(s)+ }+ if err := m.AcknowledgeAlert(id, "operator"); err != nil {+ t.Fatal(err)+ }+ s.Status = "available"+ m.CheckStorage(s)+ before := testRequireActiveAlert(t, m, id).Clone()+ s.Status = gap+ m.CheckStorage(s)+ if !reflect.DeepEqual(before, testRequireActiveAlert(t, m, id).Clone()) || m.GetResolvedAlert(id) != nil {+ t.Fatal("missing connectivity changed acknowledged occurrence")+ }+ s.Status = "available"+ m.CheckStorage(s)+ if !testHasActiveAlert(t, m, id) {+ t.Fatal("non-consecutive healthy observations resolved storage outage")+ }+ m.CheckStorage(s)+ if testHasActiveAlert(t, m, id) {+ t.Fatal("fresh consecutive healthy observations failed to recover")+ }+ resolved := m.GetResolvedAlert(id)+ if resolved == nil || !resolved.StartTime.Equal(before.StartTime) {+ t.Fatal("confirmed recovery lost original occurrence")+ }+ if m.ShadowDivergences() != 0 {+ t.Fatal("storage recovery diverged from shadow")+ }+ })+ }+}++func TestStorageConnectivityGapRestartsIntentGrace(t *testing.T) {+ m := newShadowFeedManager(t)+ var tick time.Duration+ m.intentClock = func() time.Duration { return tick }+ doc := NewAlertIntentPolicyDocument()+ doc.Defaults[string(AlertIntentSignalOffline)] = AlertIntentRule{GraceSeconds: intPointer(60)}+ if err := m.LoadIntentPolicies(doc); err != nil {+ t.Fatal(err)+ }+ s := models.Storage{ID: "storage-a", Name: "backups", Status: "offline"}+ id := canonicalConnectivityStateID(s.ID)+ m.CheckStorage(s)+ tick = 40 * time.Second+ m.CheckStorage(s)+ s.Status = "unknown"+ m.CheckStorage(s)+ m.mu.RLock()+ _, pending := m.intentPending[id]+ _, ticking := m.intentRuntimeTicks[id]+ m.mu.RUnlock()+ if pending || ticking {+ t.Error("missing status retained intent grace bookkeeping")+ }+ if err := m.SaveActiveAlerts(); err != nil {+ t.Fatal(err)+ }+ data, err := os.ReadFile(filepath.Join(m.getAlertsDir(), intentPendingFileName))+ if err != nil {+ t.Fatal(err)+ }+ var saved []IntentPendingState+ if err := json.Unmarshal(data, &saved); err != nil || len(saved) != 0 {+ t.Fatalf("interrupted intent remained checkpointed: %s, %v", data, err)+ }+ s.Status = "offline"+ for _, next := range []time.Duration{70, 90, 129} {+ tick = next * time.Second+ m.CheckStorage(s)+ if testHasActiveAlert(t, m, id) {+ t.Fatal("intent grace accrued across missing storage status")+ }+ }+ tick = 130 * time.Second+ m.CheckStorage(s)+ testRequireActiveAlert(t, m, id)+ if m.ShadowDivergences() != 0 {+ t.Fatal("intent interruption diverged from shadow")+ }+}++// Usage has delayed activation but immediate measured recovery. Do not invent+// a recovery stability window to fit the memory/temperature timing fixtures.+func TestStorageCapacityGapHoldsOccurrence(t *testing.T) {+ for _, gap := range []string{"missing", "unconfirmed-zero", "negative", "offline", "unavailable"} {+ t.Run(gap, func(t *testing.T) {+ m, elapsed := continuityManager(t, false)+ id, metric, observe := continuityObserver(t, m, "storage", gap)+ alertID := canonicalMetricStateID(id, metric)+ observe(95, false)+ elapsed.Store(int64(time.Minute))+ observe(95, false)+ if err := m.AcknowledgeAlert(alertID, "operator"); err != nil {+ t.Fatal(err)+ }+ before := testRequireActiveAlert(t, m, alertID).Clone()+ elapsed.Store(int64(2 * time.Minute))+ observe(10, true)+ if !reflect.DeepEqual(before, testRequireActiveAlert(t, m, alertID).Clone()) || m.GetResolvedAlert(alertID) != nil {+ t.Fatal("missing capacity changed the acknowledged occurrence")+ }+ // A real, independently observed empty capacity can clear even+ // while connectivity is unknown; neither channel stands in for the other.+ empty := models.Storage{ID: id, Name: "backups", Status: "unknown", Total: 1000, Free: 1000}+ m.CheckStorage(empty)+ if testHasActiveAlert(t, m, alertID) {+ t.Fatal("confirmed empty capacity did not recover immediately")+ }+ resolved := m.GetResolvedAlert(alertID)+ if resolved == nil || !resolved.StartTime.Equal(before.StartTime) || resolved.Value != 0 {+ t.Fatal("measured recovery lost the original occurrence or clearing value")+ }+ if m.ShadowDivergences() != 0 {+ t.Fatal("capacity interruption diverged from shadow")+ }+ })+ }+}+// HostDiskTemperatureThreshold is the disk heat policy for a disk a host agent// reports, so it must agree with CheckHost on every host-level override: an// explicit Disk Temp override on the host (or the canonical resource its ID