Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Preserve reviewed monitoring repairs with current filesystem alert ownership

pulse-triage[bot]9c378ffa942e
diff --cc docs/release-control/v6/internal/subsystems/alerts.md
index 3e31d17fa7,1b3fd083fd..625b94fdce
--- a/docs/release-control/v6/internal/subsystems/alerts.md
+++ b/docs/release-control/v6/internal/subsystems/alerts.md
@@@ -4020,91 -4000,3 +4090,103 @@@ display name differs from its hostname
had no alert highlighting at all. `getAlertStyles` keeps its
id-plus-node-name matching for the platform tables whose alerts are keyed that
way.
+
+### SMART temperature continuity and pending-only policy intervals
+
+Disk temperature activation/grace and recovery require continuous accepted
+observations. Empty SMART inventory, one disk omission, standby, non-positive
+readings, explicit missing/unavailable/unsupported collection and reporting-lease
+expiry interrupt pending/recovery timing without changing an open alert, its
+acknowledgement or lifecycle events. Positive legacy readings without temperature
+provenance remain accepted; an expired last-known positive value is not live
+health. Canonical and shadow reducers plus explicit intent grace are interrupted
+together. A non-empty inventory still resolves a departed disk only after three
+consecutive omissions; empty/expired inventory breaks that confirmation sequence.
+
+Configuration saves reset pending-only runs when global agent policy, host or
+inherited node/guest overrides, or the applicable disk-type threshold disables
+evaluation, even if no report arrives before re-enablement. Existing known-type
+default normalisation is unchanged; the zero-type save control uses a custom
+type whose zero threshold the normaliser preserves. Only live pending
+identity/link/type context is retained; unrelated saves preserve a known enabled
+run and explicit overrides retain precedence over type defaults. Restored intent
+without that context restarts conservatively on a save rather than inventing
+continuous enablement. Cleared intent grace is checkpointed. Existing active-alert
+per-type/save/offline/removal resolution remains unchanged.
+
+`metric_observation_continuity_test.go` and `host_unraid_lifecycle_test.go` cover
+valid/legacy/provenance controls, missing/rejected/expiry gaps on both timing
+edges, explicit grace, acknowledgement/event preservation, host isolation,
+configuration intervals, override precedence and departure. These deterministic
+source tests do not establish installed temperatures or native guest recovery.
+
+### SMART risk inventory gaps and disabled rules
+
+Agent `disk-health` and `disk-wearout` alerts follow the SMART inventory, not
+filesystem usage cleanup. Empty inventory, standby and unobserved risk fields
+hold the existing occurrence and acknowledgement without recovery/refiring.
+Departure requires three consecutive non-empty reports omitting the disk;
+listing it again, an empty inventory or reporting-lease expiry restarts that
+confirmation. Both risk families on the same disk count once per report.
+Filesystem absence still resolves usage on its first missing report. Temperature
+pending/grace/recovery interruption and departure semantics remain unchanged.
+
+Disabling every recorded cause releases that risk alert without a reading,
+both on a configuration save and on an empty report. Partially enabled causes,
+missing legacy cause metadata and unknown codes remain conservative. Re-enabling
+cannot raise an alert from absent evidence; fresh bad evidence uses the existing
+stateful refire/cooldown history policy. Node linking retains existing node ownership, including when the
+agent's SMART inventory is empty; host removal and policy disablement release
+agent risk state and stale absence counts. Counts are ephemeral and restart
+conservatively after a process restart.
+
+`TestHostSMARTRisk*` in `host_unraid_lifecycle_test.go` exercises actual `CheckHost`, expiry, configuration
+save, firing/recovery callbacks and event history with synthetic telemetry.
+It covers both families sharing a disk, interruption/reconfirmation, partial
+rule disablement, re-enablement, legacy/unknown causes, filesystem and host
+isolation, removal and node-link ownership. These are source lifecycle controls,
+not collector, smartctl, installed notification or native disk-recovery proof.
+
+### Storage observation gaps restart confirmation and timing
+
+An absent or `unknown` storage connectivity status interrupts activation,
+recovery confirmations and pending offline intent grace without resolving an
+existing outage or changing its acknowledgement. Capacity is independently
+observable while connectivity is unknown. Missing/unconfirmed-zero capacity,
+negative usage and offline/unavailable storage interrupt capacity activation,
+and intent grace timing; they do not supply healthy capacity evidence.
+Capacity recovery retains its existing immediate measured-clear behaviour,
+not the recovery stability window used by memory and temperature gauges.
+A confirmed empty store still supplies a genuine zero-usage recovery reading.
+Fresh observations must satisfy the existing full confirmation/delay after a
+gap. Interruption is scoped to the storage's canonical identity and applied to
+both live and shadow reducers; interrupted intent is checkpointed for restart.
+
+`TestStorageConnectivityObservationGaps` and
+`TestStorageConnectivityGapRestartsIntentGrace` exercise the real storage
+entry point, acknowledged occurrence retention, neighbour isolation, fresh
+reconfirmation and empty intent checkpoint. Storage routes in
+`TestMetricObservationGapRestartsActivation` covers capacity gaps and explicit
+intent grace; `TestStorageCapacityGapHoldsOccurrence` verifies retained,
+acknowledged capacity incidents and genuine immediate empty-store recovery. Existing restart, empty-capacity and predictive-capacity controls
+remain required. These source controls are not native storage recovery or
+installed destination-delivery evidence.
+
+`TestStorageConnectivityGapWebhook` also exercises the production Monitor
+callbacks, ordinary notification queue and guest-local HTTP receiver: the gap
+and first fresh healthy poll send no recovery; confirmed recovery sends one
+resolved receipt with the original occurrence identity, and the queue drains
+without failed/DLQ work. This is synthetic callback/receiver acceptance, not a
+native appliance or external webhook-provider result.
++
++### Composed host filesystem threshold ownership
++
++The shared filesystem resolver preserves the explicit host or linked-resource
++Disk override ahead of per-hardware-type defaults, for both host reports and
++configuration-save reevaluation. A filesystem-specific override still takes
++precedence. Disabling that filesystem releases its canonical usage metric and
++pending intent, retaining the reviewed agent/node ownership semantics.
++`TestHostDiskOverrideBeatsDiskFillByType` checks agent and inherited overrides
++through a configuration save and the next report without losing the occurrence;
++`TestConfigSaveJudgesFilesystemAlertsByTheirOwnThreshold` retains per-filesystem
++and per-type controls. This is source composition, not installed delivery proof.
diff --cc internal/alerts/host.go
index b46a2c00b8,db47b5bdf6..781969d9c4
--- a/internal/alerts/host.go
+++ b/internal/alerts/host.go
@@@ -172,9 -172,50 +172,51 @@@ func (m *Manager) resolveHostAlertThres
thresholds.DiskTemperature = m.lowestHostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature)
}
}
+ // A filesystem alert is judged by the threshold CheckHost evaluates that
+ // filesystem against, so a config save does not resolve an alert its
+ // override or per-type fill threshold still holds. A disabled filesystem
+ // resolves with no threshold, as CheckHost clears it.
+ if alert != nil && alert.Type == "disk" && isHostFilesystemResourceID(resourceID) {
- thresholds.Disk, _ = m.hostDiskUsageThresholdNoLock(strings.TrimSpace(resourceID), metadataStringValue(alert.Metadata, "device"), thresholds.Disk)
++ override, exists := m.hostThresholdOverrideNoLock(hostID, linkedNodeID, linkedVMID, linkedContainerID)
++ thresholds.Disk, _ = m.hostDiskUsageThresholdNoLock(strings.TrimSpace(resourceID), metadataStringValue(alert.Metadata, "device"), thresholds.Disk, exists && override.Disk != nil)
+ }
return thresholds
}
+ // isHostFilesystemResourceID reports whether resourceID names one agent
+ // filesystem ("agent:<host>/disk:<label>") rather than the host itself.
+ func isHostFilesystemResourceID(resourceID string) bool {
+ _, child, ok := strings.Cut(stripHostResourcePrefix(resourceID), "/")
+ return ok && strings.HasPrefix(child, "disk:")
+ }
+
+ // hostDiskUsageThresholdNoLock resolves the usage threshold one agent
-// filesystem is judged by: the filesystem's own override, then the fill
-// threshold for its hardware type while the host's disk alerting is on, then
-// the host's disk threshold. disabled reports an override that switches the
++// filesystem is judged by: the filesystem's own override, then an explicit
++// host or inherited disk override, then the fill threshold for its hardware
++// type while the host's disk alerting is on, then the host's disk threshold. disabled reports an override that switches the
+ // filesystem's alerts off, and then the threshold is nil. CheckHost and
+ // config-save reevaluation both resolve through it. An override or per-type
+ // threshold is returned as an owned copy, so normalizing it never writes into
+ // the live config. Callers must hold m.mu.
-func (m *Manager) hostDiskUsageThresholdNoLock(diskResourceID, device string, hostDisk *HysteresisThreshold) (threshold *HysteresisThreshold, disabled bool) {
++func (m *Manager) hostDiskUsageThresholdNoLock(diskResourceID, device string, hostDisk *HysteresisThreshold, hostDiskOverridden bool) (threshold *HysteresisThreshold, disabled bool) {
+ if override, ok := m.config.Overrides[diskResourceID]; ok {
+ if override.Disabled {
+ return nil, true
+ }
+ if override.Disk != nil {
+ return ensureHysteresisThreshold(cloneThreshold(override.Disk)), false
+ }
+ }
- if hostDisk != nil && hostDisk.Trigger > 0 {
++ if !hostDiskOverridden && hostDisk != nil && hostDisk.Trigger > 0 {
+ if hwType := inferDiskHardwareType(device); hwType != "" {
+ if byType, ok := m.config.DiskFillByType[hwType]; ok {
+ return &byType, false
+ }
+ }
+ }
+ return hostDisk, false
+ }
+
func sanitizeHostComponent(value string) string {
value = strings.TrimSpace(strings.ToLower(value))
if value == "" {
@@@ -469,39 -491,12 +511,12 @@@ func (m *Manager) CheckHost(host models
diskResourceID, diskName := hostDiskResourceID(host, disk)
seenDisks[diskResourceID] = struct{}{}
- // Check for disk-specific override
m.mu.RLock()
- diskOverride, hasDiskOverride := m.config.Overrides[diskResourceID]
- effectiveDiskThreshold, diskDisabled := m.hostDiskUsageThresholdNoLock(diskResourceID, disk.Device, thresholds.Disk)
++ effectiveDiskThreshold, diskDisabled := m.hostDiskUsageThresholdNoLock(diskResourceID, disk.Device, thresholds.Disk, diskOverridden)
m.mu.RUnlock()
-
- // Determine the effective disk threshold
- var effectiveDiskThreshold *HysteresisThreshold
- if hasDiskOverride {
- // If disk is disabled via override, skip alerting
- if diskOverride.Disabled {
- m.releaseHostUsageMetric(diskResourceID, diskName, nodeName, instanceName, unifiedresources.ResourceType("agent-disk"), "agent-disk", "disk")
- continue
- }
- // Use disk-specific threshold if set
- if diskOverride.Disk != nil {
- effectiveDiskThreshold = ensureHysteresisThreshold(diskOverride.Disk)
- }
- }
- // Per-type override: consult DiskFillByType if hardware type is inferable
- // from the device path and no disk-specific override applied above.
- if effectiveDiskThreshold == nil && !diskOverridden && thresholds.Disk != nil && thresholds.Disk.Trigger > 0 {
- if hwType := inferDiskHardwareType(disk.Device); hwType != "" {
- m.mu.RLock()
- if th, ok := m.config.DiskFillByType[hwType]; ok {
- t := th
- effectiveDiskThreshold = &t
- }
- m.mu.RUnlock()
- }
- }
- // Fall back to host-level threshold
- if effectiveDiskThreshold == nil {
- effectiveDiskThreshold = thresholds.Disk
+ if diskDisabled {
- m.clearAlert(canonicalMetricStateID(diskResourceID, "disk"))
++ m.releaseHostUsageMetric(diskResourceID, diskName, nodeName, instanceName, unifiedresources.ResourceType("agent-disk"), "agent-disk", "disk")
+ continue
}
// Skip if no threshold configured (nil)
diff --cc internal/alerts/threshold_resolution_shared_test.go
index 314a235fad,fac87ae429..77f6dfd1d2
--- a/internal/alerts/threshold_resolution_shared_test.go
+++ b/internal/alerts/threshold_resolution_shared_test.go
@@@ -1451,414 -1563,6 +1568,429 @@@ func TestDiskTemperatureThresholdMatche
}
}
+// An agent owns a usage metric only with usable evidence: it evaluated the
+// metric this report, or it still holds an alert or run for it. Configuration
+// alone is not coverage.
+func TestCheckNodeKeepsMetricsTheAgentCannotEvaluate(t *testing.T) {
+ setup := func(t *testing.T) (*Manager, models.Node, models.Host) {
+ m := newTestManager(t)
+ m.mu.Lock()
+ m.config.Enabled = true
+ m.config.TimeThresholds = map[string]int{}
+ m.config.NodeDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}
+ m.config.NodeDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}
+ m.config.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}
+ m.config.AgentDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}
+ m.mu.Unlock()
+ node, host := testNodeWithHostAgent()
+ node.CPU = 0.95
+ node.Memory = models.Memory{Total: 100, Used: 95, Free: 5, Usage: 95}
+ return m, node, host
+ }
+
+ t.Run("memory_unknown_from_the_start", func(t *testing.T) {
+ m, node, host := setup(t)
+ host.Memory = models.Memory{Total: 100, UsageUnavailable: true}
+ m.CheckHost(host)
+ m.CheckNode(node)
+ if !testHasActiveAlert(t, m, canonicalMetricStateID(node.ID, "memory")) {
+ t.Fatalf("expected the node to alert on memory while the agent has no memory reading")
+ }
+ })
+
+ t.Run("non_finite_cpu", func(t *testing.T) {
+ m, node, host := setup(t)
+ host.CPUUsage = math.NaN()
+ m.CheckHost(host)
+ m.CheckNode(node)
+ if !testHasActiveAlert(t, m, canonicalMetricStateID(node.ID, "cpu")) {
+ t.Fatalf("expected the node to alert on CPU while the agent reports no usable CPU value")
+ }
+ })
+}
+
+// Turning an agent threshold off releases its pending run even when the agent
+// reports no reading for that metric, so the old run cannot keep ownership and
+// leave the node silent.
+func TestDisabledAgentThresholdReleasesPendingRunWithoutReading(t *testing.T) {
+ m := newTestManager(t)
+ m.mu.Lock()
+ m.config.Enabled = true
+ m.config.TimeThresholds = map[string]int{}
+ m.config.MetricTimeThresholds = map[string]map[string]int{"agent": {"memory": 300}}
+ m.config.NodeDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}
+ m.config.AgentDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}
+ m.mu.Unlock()
+ node, host := testNodeWithHostAgent()
+ // 90% stays below the 95% critical level, which would bypass the delay.
+ node.Memory = models.Memory{Total: 100, Used: 90, Free: 10, Usage: 90}
+ host.Memory = node.Memory
+ agentResourceID := hostResourceID(host.ID)
+ agentMemorySpec := canonicalMetricSpecID(agentResourceID, "memory")
+
+ m.CheckHost(host)
+ m.mu.RLock()
+ pending := testCoreIsPending(m, agentResourceID, agentMemorySpec)
+ m.mu.RUnlock()
+ if !pending {
+ t.Fatalf("expected a pending agent memory run")
+ }
+
+ m.mu.Lock()
+ m.config.AgentDefaults.Memory = &HysteresisThreshold{Trigger: 0, Clear: 0}
+ m.mu.Unlock()
+ host.Memory = models.Memory{Total: 100, UsageUnavailable: true}
+ m.CheckHost(host)
+ m.CheckNode(node)
+
+ m.mu.RLock()
+ agentIncident := testCoreHasIncident(m, agentResourceID, agentMemorySpec)
+ nodeIncident := testCoreHasIncident(m, node.ID, canonicalMetricSpecID(node.ID, "memory"))
+ m.mu.RUnlock()
+ if agentIncident {
+ t.Fatalf("expected the disabled agent memory threshold to release its pending run")
+ }
+ if !nodeIncident {
+ t.Fatalf("expected the node to evaluate memory once the agent no longer covers it")
+ }
+}
+
+// Ownership follows the config at once and link updates apply in report order:
+// a config change revokes what it no longer supports without waiting for the
+// agent's next report, an older report can neither restore revoked ownership
+// nor remove a newer report's link, and a report started before the agent went
+// offline is stale.
+func TestHostAgentNodeLinkFollowsConfigAndReportOrder(t *testing.T) {
+ m := newTestManager(t)
+ m.mu.Lock()
+ m.config.Enabled = true
+ m.config.TimeThresholds = map[string]int{}
+ m.config.NodeDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}
+ m.config.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}
+ m.mu.Unlock()
+ node, host := testNodeWithHostAgent()
+ node.CPU = 0.95
+ host.CPUUsage = 50
+
+ m.CheckHost(host)
+ m.CheckNode(node)
+ nodeCPUSpec := canonicalMetricSpecID(node.ID, "cpu")
+ m.mu.RLock()
+ evaluatedBefore := testCoreHasIncident(m, node.ID, nodeCPUSpec)
+ m.mu.RUnlock()
+ if evaluatedBefore {
+ t.Fatalf("expected the agent to own CPU before the config change")
+ }
+
+ // A report already in flight when agent alerts are disabled must not
+ // restore ownership once it lands.
+ inFlight := m.beginHostAgentReport(host.ID)
+
+ // No agent report follows the config change; the node must evaluate CPU on
+ // its very next check. (UpdateConfig restores the default activation delay,
+ // so the run is pending rather than firing.)
+ config := m.GetConfig()
+ config.DisableAllAgents = true
+ m.UpdateConfig(config)
+ m.CheckNode(node)
+ m.mu.RLock()
+ evaluatedAfter := testCoreHasIncident(m, node.ID, nodeCPUSpec)
+ m.mu.RUnlock()
+ if !evaluatedAfter {
+ t.Fatalf("expected the node to evaluate CPU right after agent alerts were disabled")
+ }
+
+ covering := hostAgentNodeLink{agentID: host.ID, nodeID: node.ID, cpu: true}
+ m.applyHostAgentNodeLink(covering, inFlight)
+ if m.hasHostAgentForNode(node.ID) {
+ t.Fatalf("expected a report evaluated before agent alerts were disabled not to restore ownership")
+ }
+
+ config.DisableAllAgents = false
+ m.UpdateConfig(config)
+ older := m.beginHostAgentReport(host.ID)
+ newer := m.beginHostAgentReport(host.ID)
+ m.applyHostAgentNodeLink(covering, newer)
+ m.applyHostAgentNodeLink(hostAgentNodeLink{agentID: host.ID, nodeID: node.ID}, older)
+ if !m.hasHostAgentForNode(node.ID) {
+ t.Fatalf("expected an older report not to remove the newer report's link")
+ }
+
+ olderCovering := m.beginHostAgentReport(host.ID)
+ newerRemoving := m.beginHostAgentReport(host.ID)
+ m.applyHostAgentNodeLink(hostAgentNodeLink{agentID: host.ID, nodeID: node.ID}, newerRemoving)
+ m.applyHostAgentNodeLink(covering, olderCovering)
+ if m.hasHostAgentForNode(node.ID) {
+ t.Fatalf("expected an older report not to restore ownership a newer report removed")
+ }
+
+ beforeOffline := m.beginHostAgentReport(host.ID)
+ m.HandleHostOffline(host)
+ m.applyHostAgentNodeLink(covering, beforeOffline)
+ if m.hasHostAgentForNode(node.ID) {
+ t.Fatalf("expected a report started before the agent went offline not to re-register")
+ }
+}
+
+// An agent that reports unlinked hands its former node the usage metrics back
+// on the node's next check, even while the agent's own alert is still firing.
+func TestCheckNodeTakesBackMetricsWhenAgentUnlinks(t *testing.T) {
+ m := newTestManager(t)
+ m.mu.Lock()
+ m.config.Enabled = true
+ m.config.TimeThresholds = map[string]int{}
+ m.config.NodeDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}
+ m.config.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}
+ m.mu.Unlock()
+ node, host := testNodeWithHostAgent()
+ node.CPU = 0.95
+ host.CPUUsage = 95
+
+ m.CheckHost(host)
+ m.CheckNode(node)
+ nodeCPU := canonicalMetricStateID(node.ID, "cpu")
+ if testHasActiveAlert(t, m, nodeCPU) {
+ t.Fatalf("expected the linked agent to own CPU")
+ }
+
+ host.LinkedNodeID = ""
+ m.CheckHost(host)
+ m.CheckNode(node)
+ if !testHasActiveAlert(t, m, nodeCPU) {
+ t.Fatalf("expected the node to alert on CPU once its agent reports unlinked")
+ }
+}
+
+// An explicit disk override, including one inherited from the linked node,
+// beats the per-type DiskFillByType default; otherwise the node released its
+// disk alert to an agent applying a looser per-type threshold.
+func TestHostDiskOverrideBeatsDiskFillByType(t *testing.T) {
+ for _, overrideOn := range []string{"linked node", "agent"} {
+ t.Run(overrideOn, func(t *testing.T) {
+ m := newTestManager(t)
+ node, host := testNodeWithHostAgent()
+ overrideID := node.ID
+ if overrideOn == "agent" {
+ overrideID = host.ID
+ }
+ m.mu.Lock()
+ m.config.Enabled = true
+ m.config.TimeThresholds = map[string]int{}
+ m.config.AgentDefaults.Disk = &HysteresisThreshold{Trigger: 90, Clear: 85}
+ m.config.DiskFillByType = map[string]HysteresisThreshold{"nvme": {Trigger: 92, Clear: 88}}
+ m.config.Overrides = map[string]ThresholdConfig{
+ overrideID: {Disk: &HysteresisThreshold{Trigger: 80, Clear: 75}},
+ }
+ m.mu.Unlock()
+ root := models.Disk{Mountpoint: "/", Device: "/dev/nvme0n1p2", Total: 100, Used: 85, Free: 15, Usage: 85}
+ host.Disks = []models.Disk{root}
+
+ m.CheckHost(host)
+ rootResourceID, _ := hostDiskResourceID(host, root)
+ if !testHasActiveAlert(t, m, canonicalMetricStateID(rootResourceID, "disk")) {
+ t.Fatalf("expected the 80%% %s disk override to raise the agent's root disk alert at 85%%", overrideOn)
+ }
++
++ // Config-save reevaluation must preserve the same explicit override
++ // when the upstream per-filesystem resolver is composed with agent
++ // ownership. The looser NVMe default must not clear this occurrence.
++ alertID := canonicalMetricStateID(rootResourceID, "disk")
++ before, _ := testLookupActiveAlert(t, m, alertID)
++ m.UpdateConfig(m.GetConfig())
++ after, exists := testLookupActiveAlert(t, m, alertID)
++ if !exists || after.StartTime != before.StartTime {
++ t.Fatalf("config save lost the %s disk override occurrence", overrideOn)
++ }
++ m.CheckHost(host)
++ if !testHasActiveAlert(t, m, alertID) {
++ t.Fatalf("next host report lost the %s disk override alert", overrideOn)
++ }
+ })
+ }
+}
+
+// Releasing a metric drops its explicit intent grace run too, so a later
+// resume does not activate at once on time counted while nothing evaluated.
+func TestReleaseCanonicalMetricAlertClearsIntentPending(t *testing.T) {
+ m := newTestManager(t)
+ spec, err := buildCanonicalMetricSpec("homelab-delly2", "delly2", "node", "cpu", nil)
+ if err != nil {
+ t.Fatalf("spec: %v", err)
+ }
+ trackingKey := canonicalTrackingKeyForSpec(spec, spec.ID)
+ m.mu.Lock()
+ m.intentPending[trackingKey] = IntentPendingState{
+ TrackingKey: trackingKey,
+ ResourceID: spec.ResourceID,
+ Signal: MetricAlertIntentSignal("cpu"),
+ FirstMatchedAt: time.Now().Add(-10 * time.Minute),
+ LastObservedAt: time.Now().Add(-9 * time.Minute),
+ }
+ m.mu.Unlock()
+
+ m.releaseCanonicalMetricAlert(spec, "delly2", "delly2", "homelab", "node", 0, nil)
+
+ m.mu.RLock()
+ _, stillPending := m.intentPending[trackingKey]
+ m.mu.RUnlock()
+ if stillPending {
+ t.Fatalf("expected releasing the metric to clear its intent pending state")
+ }
+}
+
+// Missing storage observations break confirmation runs, not the occurrence.
+// Exercise the public evaluator and both live/shadow reducers, not seeded state.
+func TestStorageConnectivityObservationGaps(t *testing.T) {
+ for _, gap := range []string{"", "unknown", " UNKNOWN "} {
+ t.Run("activation/"+gap, func(t *testing.T) {
+ m := newShadowFeedManager(t)
+ s := models.Storage{ID: "storage-a", Name: "backups", Status: "offline"}
+ other := s
+ other.ID = "storage-ab"
+ id := canonicalConnectivityStateID(s.ID)
+ m.CheckStorage(s)
+ m.CheckStorage(other)
+ s.Status = gap
+ m.CheckStorage(s)
+ s.Status = "unavailable"
+ m.CheckStorage(s)
+ if testHasActiveAlert(t, m, id) {
+ t.Fatal("non-consecutive outage observations fired storage alert")
+ }
+ m.CheckStorage(other)
+ testRequireActiveAlert(t, m, canonicalConnectivityStateID(other.ID))
+ m.CheckStorage(s)
+ testRequireActiveAlert(t, m, id)
+ if m.ShadowDivergences() != 0 {
+ t.Fatal("storage interruption diverged from shadow")
+ }
+ })
+ t.Run("recovery/"+gap, func(t *testing.T) {
+ m := newShadowFeedManager(t)
+ s := models.Storage{ID: "storage-a", Name: "backups", Status: "offline"}
+ id := canonicalConnectivityStateID(s.ID)
+ for range 2 {
+ m.CheckStorage(s)
+ }
+ if err := m.AcknowledgeAlert(id, "operator"); err != nil {
+ t.Fatal(err)
+ }
+ s.Status = "available"
+ m.CheckStorage(s)
+ before := testRequireActiveAlert(t, m, id).Clone()
+ s.Status = gap
+ m.CheckStorage(s)
+ if !reflect.DeepEqual(before, testRequireActiveAlert(t, m, id).Clone()) || m.GetResolvedAlert(id) != nil {
+ t.Fatal("missing connectivity changed acknowledged occurrence")
+ }
+ s.Status = "available"
+ m.CheckStorage(s)
+ if !testHasActiveAlert(t, m, id) {
+ t.Fatal("non-consecutive healthy observations resolved storage outage")
+ }
+ m.CheckStorage(s)
+ if testHasActiveAlert(t, m, id) {
+ t.Fatal("fresh consecutive healthy observations failed to recover")
+ }
+ resolved := m.GetResolvedAlert(id)
+ if resolved == nil || !resolved.StartTime.Equal(before.StartTime) {
+ t.Fatal("confirmed recovery lost original occurrence")
+ }
+ if m.ShadowDivergences() != 0 {
+ t.Fatal("storage recovery diverged from shadow")
+ }
+ })
+ }
+}
+
+func TestStorageConnectivityGapRestartsIntentGrace(t *testing.T) {
+ m := newShadowFeedManager(t)
+ var tick time.Duration
+ m.intentClock = func() time.Duration { return tick }
+ doc := NewAlertIntentPolicyDocument()
+ doc.Defaults[string(AlertIntentSignalOffline)] = AlertIntentRule{GraceSeconds: intPointer(60)}
+ if err := m.LoadIntentPolicies(doc); err != nil {
+ t.Fatal(err)
+ }
+ s := models.Storage{ID: "storage-a", Name: "backups", Status: "offline"}
+ id := canonicalConnectivityStateID(s.ID)
+ m.CheckStorage(s)
+ tick = 40 * time.Second
+ m.CheckStorage(s)
+ s.Status = "unknown"
+ m.CheckStorage(s)
+ m.mu.RLock()
+ _, pending := m.intentPending[id]
+ _, ticking := m.intentRuntimeTicks[id]
+ m.mu.RUnlock()
+ if pending || ticking {
+ t.Error("missing status retained intent grace bookkeeping")
+ }
+ if err := m.SaveActiveAlerts(); err != nil {
+ t.Fatal(err)
+ }
+ data, err := os.ReadFile(filepath.Join(m.getAlertsDir(), intentPendingFileName))
+ if err != nil {
+ t.Fatal(err)
+ }
+ var saved []IntentPendingState
+ if err := json.Unmarshal(data, &saved); err != nil || len(saved) != 0 {
+ t.Fatalf("interrupted intent remained checkpointed: %s, %v", data, err)
+ }
+ s.Status = "offline"
+ for _, next := range []time.Duration{70, 90, 129} {
+ tick = next * time.Second
+ m.CheckStorage(s)
+ if testHasActiveAlert(t, m, id) {
+ t.Fatal("intent grace accrued across missing storage status")
+ }
+ }
+ tick = 130 * time.Second
+ m.CheckStorage(s)
+ testRequireActiveAlert(t, m, id)
+ if m.ShadowDivergences() != 0 {
+ t.Fatal("intent interruption diverged from shadow")
+ }
+}
+
+// Usage has delayed activation but immediate measured recovery. Do not invent
+// a recovery stability window to fit the memory/temperature timing fixtures.
+func TestStorageCapacityGapHoldsOccurrence(t *testing.T) {
+ for _, gap := range []string{"missing", "unconfirmed-zero", "negative", "offline", "unavailable"} {
+ t.Run(gap, func(t *testing.T) {
+ m, elapsed := continuityManager(t, false)
+ id, metric, observe := continuityObserver(t, m, "storage", gap)
+ alertID := canonicalMetricStateID(id, metric)
+ observe(95, false)
+ elapsed.Store(int64(time.Minute))
+ observe(95, false)
+ if err := m.AcknowledgeAlert(alertID, "operator"); err != nil {
+ t.Fatal(err)
+ }
+ before := testRequireActiveAlert(t, m, alertID).Clone()
+ elapsed.Store(int64(2 * time.Minute))
+ observe(10, true)
+ if !reflect.DeepEqual(before, testRequireActiveAlert(t, m, alertID).Clone()) || m.GetResolvedAlert(alertID) != nil {
+ t.Fatal("missing capacity changed the acknowledged occurrence")
+ }
+ // A real, independently observed empty capacity can clear even
+ // while connectivity is unknown; neither channel stands in for the other.
+ empty := models.Storage{ID: id, Name: "backups", Status: "unknown", Total: 1000, Free: 1000}
+ m.CheckStorage(empty)
+ if testHasActiveAlert(t, m, alertID) {
+ t.Fatal("confirmed empty capacity did not recover immediately")
+ }
+ resolved := m.GetResolvedAlert(alertID)
+ if resolved == nil || !resolved.StartTime.Equal(before.StartTime) || resolved.Value != 0 {
+ t.Fatal("measured recovery lost the original occurrence or clearing value")
+ }
+ if m.ShadowDivergences() != 0 {
+ t.Fatal("capacity interruption diverged from shadow")
+ }
+ })
+ }
+}
+
// HostDiskTemperatureThreshold is the disk heat policy for a disk a host agent
// reports, so it must agree with CheckHost on every host-level override: an
// explicit Disk Temp override on the host (or the canonical resource its ID