Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Compose reviewed monitoring repairs with current upstream without losing either fence

pulse-triage[bot]0840845673fc
diff --cc docs/release-control/v6/internal/subsystems/monitoring.md
index 0817bbccc2,3c011447f5..f5dae19921
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@@ -1883,23 -1758,19 +1927,25 @@@ boundaries
report and would mark the host online.
- Agents are evaluated before nodes, so a node with an online linked agent
hands its CPU, memory and disk alerts to the agent on the first tick, as
- host-agent hostname deduplication does in production.
+ node-link deduplication does in production.
- A runtime mock config change rebuilds the estate. An agent that leaves it
goes through `HandleHostRemoved`, as a deleted live agent does, so its alerts
- and hostname deduplication do not outlive it.
+ and node link do not outlive it.
- Leaving mock mode routes the fixture agents through `HandleHostRemoved`.
- `ClearActiveAlerts` drops their alerts but not their hostname registrations,
- and a real node named like a fixture agent (`pve1`) would otherwise keep its
- metric alerts suppressed with no agent to own them. `SetMockMode` ends the
- mock-mode epoch before it forgets them, so a pass whose snapshot predates the
- switch finds its epoch ended under the same lock and evaluates nothing, and
- an agent whose departure the switch refused stays in the forgotten set. Two
- overlapping passes in one epoch can still evaluate an older snapshot after a
- newer one; the next pass reconciles the set again.
+ `ClearActiveAlerts` drops their alerts but not their node links, and a
+ leftover link would keep suppressing CPU, memory and disk alerts on any node
- that later carries the linked fixture node's ID. The agent and Docker
- steps of a pass whose snapshot predates the switch find mock mode off under
- the same lock and evaluate nothing.
++ that later carries the linked fixture node's ID. `SetMockMode` ends the
++ mock-mode epoch before forgetting fixture hosts, so evaluations holding an
++ older epoch cannot recreate alerts after the clear. Departures refused by
++ the epoch fence remain tracked for the switch to remove.
+- Ticks start passes concurrently, so a pass whose snapshot predates an
- estate rebuild can reach the lock after a newer one. Each pass carries the
- fixture's structural revision, and the agent and Docker steps of a pass
- older than the last applied or disabled revision are skipped, so they
- cannot remove the new estate's hosts or re-evaluate retired ones, even
- across a disable and re-enable. The pass's other loops (guests, nodes,
- storage, PBS, PMG) take no part in this tracking and are not guarded.
++ estate rebuild can reach the lock after a newer one. The agent and Docker
++ steps also carry the fixture's structural revision and reject revisions
++ older than the last applied or disabled one. Both checks apply: the epoch
++ stops mode-crossing evaluations, while the revision stops an older estate
++ replacing newer host ownership within one epoch. Guest, node, storage, PBS,
++ PMG and physical-disk evaluations use the epoch fence, not host revision
++ tracking.
Fixture agents must carry readings a real agent could report. Mock Kubernetes
pods share 0.7 single-pod memory footprints per node, which keeps a node's pods
diff --cc internal/alerts/threshold_resolution_shared_test.go
index 93911cba42,cf040367e4..314a235fad
--- a/internal/alerts/threshold_resolution_shared_test.go
+++ b/internal/alerts/threshold_resolution_shared_test.go
@@@ -1451,410 -1446,147 +1451,555 @@@ func TestDiskTemperatureThresholdMatche
}
}
+// An agent owns a usage metric only with usable evidence: it evaluated the
+// metric this report, or it still holds an alert or run for it. Configuration
+// alone is not coverage.
+func TestCheckNodeKeepsMetricsTheAgentCannotEvaluate(t *testing.T) {
+ setup := func(t *testing.T) (*Manager, models.Node, models.Host) {
+ m := newTestManager(t)
+ m.mu.Lock()
+ m.config.Enabled = true
+ m.config.TimeThresholds = map[string]int{}
+ m.config.NodeDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}
+ m.config.NodeDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}
+ m.config.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}
+ m.config.AgentDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}
+ m.mu.Unlock()
+ node, host := testNodeWithHostAgent()
+ node.CPU = 0.95
+ node.Memory = models.Memory{Total: 100, Used: 95, Free: 5, Usage: 95}
+ return m, node, host
+ }
+
+ t.Run("memory_unknown_from_the_start", func(t *testing.T) {
+ m, node, host := setup(t)
+ host.Memory = models.Memory{Total: 100, UsageUnavailable: true}
+ m.CheckHost(host)
+ m.CheckNode(node)
+ if !testHasActiveAlert(t, m, canonicalMetricStateID(node.ID, "memory")) {
+ t.Fatalf("expected the node to alert on memory while the agent has no memory reading")
+ }
+ })
+
+ t.Run("non_finite_cpu", func(t *testing.T) {
+ m, node, host := setup(t)
+ host.CPUUsage = math.NaN()
+ m.CheckHost(host)
+ m.CheckNode(node)
+ if !testHasActiveAlert(t, m, canonicalMetricStateID(node.ID, "cpu")) {
+ t.Fatalf("expected the node to alert on CPU while the agent reports no usable CPU value")
+ }
+ })
+}
+
+// Turning an agent threshold off releases its pending run even when the agent
+// reports no reading for that metric, so the old run cannot keep ownership and
+// leave the node silent.
+func TestDisabledAgentThresholdReleasesPendingRunWithoutReading(t *testing.T) {
+ m := newTestManager(t)
+ m.mu.Lock()
+ m.config.Enabled = true
+ m.config.TimeThresholds = map[string]int{}
+ m.config.MetricTimeThresholds = map[string]map[string]int{"agent": {"memory": 300}}
+ m.config.NodeDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}
+ m.config.AgentDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}
+ m.mu.Unlock()
+ node, host := testNodeWithHostAgent()
+ // 90% stays below the 95% critical level, which would bypass the delay.
+ node.Memory = models.Memory{Total: 100, Used: 90, Free: 10, Usage: 90}
+ host.Memory = node.Memory
+ agentResourceID := hostResourceID(host.ID)
+ agentMemorySpec := canonicalMetricSpecID(agentResourceID, "memory")
+
+ m.CheckHost(host)
+ m.mu.RLock()
+ pending := testCoreIsPending(m, agentResourceID, agentMemorySpec)
+ m.mu.RUnlock()
+ if !pending {
+ t.Fatalf("expected a pending agent memory run")
+ }
+
+ m.mu.Lock()
+ m.config.AgentDefaults.Memory = &HysteresisThreshold{Trigger: 0, Clear: 0}
+ m.mu.Unlock()
+ host.Memory = models.Memory{Total: 100, UsageUnavailable: true}
+ m.CheckHost(host)
+ m.CheckNode(node)
+
+ m.mu.RLock()
+ agentIncident := testCoreHasIncident(m, agentResourceID, agentMemorySpec)
+ nodeIncident := testCoreHasIncident(m, node.ID, canonicalMetricSpecID(node.ID, "memory"))
+ m.mu.RUnlock()
+ if agentIncident {
+ t.Fatalf("expected the disabled agent memory threshold to release its pending run")
+ }
+ if !nodeIncident {
+ t.Fatalf("expected the node to evaluate memory once the agent no longer covers it")
+ }
+}
+
+// Ownership follows the config at once and link updates apply in report order:
+// a config change revokes what it no longer supports without waiting for the
+// agent's next report, an older report can neither restore revoked ownership
+// nor remove a newer report's link, and a report started before the agent went
+// offline is stale.
+func TestHostAgentNodeLinkFollowsConfigAndReportOrder(t *testing.T) {
+ m := newTestManager(t)
+ m.mu.Lock()
+ m.config.Enabled = true
+ m.config.TimeThresholds = map[string]int{}
+ m.config.NodeDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}
+ m.config.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}
+ m.mu.Unlock()
+ node, host := testNodeWithHostAgent()
+ node.CPU = 0.95
+ host.CPUUsage = 50
+
+ m.CheckHost(host)
+ m.CheckNode(node)
+ nodeCPUSpec := canonicalMetricSpecID(node.ID, "cpu")
+ m.mu.RLock()
+ evaluatedBefore := testCoreHasIncident(m, node.ID, nodeCPUSpec)
+ m.mu.RUnlock()
+ if evaluatedBefore {
+ t.Fatalf("expected the agent to own CPU before the config change")
+ }
+
+ // A report already in flight when agent alerts are disabled must not
+ // restore ownership once it lands.
+ inFlight := m.beginHostAgentReport(host.ID)
+
+ // No agent report follows the config change; the node must evaluate CPU on
+ // its very next check. (UpdateConfig restores the default activation delay,
+ // so the run is pending rather than firing.)
+ config := m.GetConfig()
+ config.DisableAllAgents = true
+ m.UpdateConfig(config)
+ m.CheckNode(node)
+ m.mu.RLock()
+ evaluatedAfter := testCoreHasIncident(m, node.ID, nodeCPUSpec)
+ m.mu.RUnlock()
+ if !evaluatedAfter {
+ t.Fatalf("expected the node to evaluate CPU right after agent alerts were disabled")
+ }
+
+ covering := hostAgentNodeLink{agentID: host.ID, nodeID: node.ID, cpu: true}
+ m.applyHostAgentNodeLink(covering, inFlight)
+ if m.hasHostAgentForNode(node.ID) {
+ t.Fatalf("expected a report evaluated before agent alerts were disabled not to restore ownership")
+ }
+
+ config.DisableAllAgents = false
+ m.UpdateConfig(config)
+ older := m.beginHostAgentReport(host.ID)
+ newer := m.beginHostAgentReport(host.ID)
+ m.applyHostAgentNodeLink(covering, newer)
+ m.applyHostAgentNodeLink(hostAgentNodeLink{agentID: host.ID, nodeID: node.ID}, older)
+ if !m.hasHostAgentForNode(node.ID) {
+ t.Fatalf("expected an older report not to remove the newer report's link")
+ }
+
+ olderCovering := m.beginHostAgentReport(host.ID)
+ newerRemoving := m.beginHostAgentReport(host.ID)
+ m.applyHostAgentNodeLink(hostAgentNodeLink{agentID: host.ID, nodeID: node.ID}, newerRemoving)
+ m.applyHostAgentNodeLink(covering, olderCovering)
+ if m.hasHostAgentForNode(node.ID) {
+ t.Fatalf("expected an older report not to restore ownership a newer report removed")
+ }
+
+ beforeOffline := m.beginHostAgentReport(host.ID)
+ m.HandleHostOffline(host)
+ m.applyHostAgentNodeLink(covering, beforeOffline)
+ if m.hasHostAgentForNode(node.ID) {
+ t.Fatalf("expected a report started before the agent went offline not to re-register")
+ }
+}
+
+// An agent that reports unlinked hands its former node the usage metrics back
+// on the node's next check, even while the agent's own alert is still firing.
+func TestCheckNodeTakesBackMetricsWhenAgentUnlinks(t *testing.T) {
+ m := newTestManager(t)
+ m.mu.Lock()
+ m.config.Enabled = true
+ m.config.TimeThresholds = map[string]int{}
+ m.config.NodeDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}
+ m.config.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}
+ m.mu.Unlock()
+ node, host := testNodeWithHostAgent()
+ node.CPU = 0.95
+ host.CPUUsage = 95
+
+ m.CheckHost(host)
+ m.CheckNode(node)
+ nodeCPU := canonicalMetricStateID(node.ID, "cpu")
+ if testHasActiveAlert(t, m, nodeCPU) {
+ t.Fatalf("expected the linked agent to own CPU")
+ }
+
+ host.LinkedNodeID = ""
+ m.CheckHost(host)
+ m.CheckNode(node)
+ if !testHasActiveAlert(t, m, nodeCPU) {
+ t.Fatalf("expected the node to alert on CPU once its agent reports unlinked")
+ }
+}
+
+// An explicit disk override, including one inherited from the linked node,
+// beats the per-type DiskFillByType default; otherwise the node released its
+// disk alert to an agent applying a looser per-type threshold.
+func TestHostDiskOverrideBeatsDiskFillByType(t *testing.T) {
+ for _, overrideOn := range []string{"linked node", "agent"} {
+ t.Run(overrideOn, func(t *testing.T) {
+ m := newTestManager(t)
+ node, host := testNodeWithHostAgent()
+ overrideID := node.ID
+ if overrideOn == "agent" {
+ overrideID = host.ID
+ }
+ m.mu.Lock()
+ m.config.Enabled = true
+ m.config.TimeThresholds = map[string]int{}
+ m.config.AgentDefaults.Disk = &HysteresisThreshold{Trigger: 90, Clear: 85}
+ m.config.DiskFillByType = map[string]HysteresisThreshold{"nvme": {Trigger: 92, Clear: 88}}
+ m.config.Overrides = map[string]ThresholdConfig{
+ overrideID: {Disk: &HysteresisThreshold{Trigger: 80, Clear: 75}},
+ }
+ m.mu.Unlock()
+ root := models.Disk{Mountpoint: "/", Device: "/dev/nvme0n1p2", Total: 100, Used: 85, Free: 15, Usage: 85}
+ host.Disks = []models.Disk{root}
+
+ m.CheckHost(host)
+ rootResourceID, _ := hostDiskResourceID(host, root)
+ if !testHasActiveAlert(t, m, canonicalMetricStateID(rootResourceID, "disk")) {
+ t.Fatalf("expected the 80%% %s disk override to raise the agent's root disk alert at 85%%", overrideOn)
+ }
+ })
+ }
+}
+
+// Releasing a metric drops its explicit intent grace run too, so a later
+// resume does not activate at once on time counted while nothing evaluated.
+func TestReleaseCanonicalMetricAlertClearsIntentPending(t *testing.T) {
+ m := newTestManager(t)
+ spec, err := buildCanonicalMetricSpec("homelab-delly2", "delly2", "node", "cpu", nil)
+ if err != nil {
+ t.Fatalf("spec: %v", err)
+ }
+ trackingKey := canonicalTrackingKeyForSpec(spec, spec.ID)
+ m.mu.Lock()
+ m.intentPending[trackingKey] = IntentPendingState{
+ TrackingKey: trackingKey,
+ ResourceID: spec.ResourceID,
+ Signal: MetricAlertIntentSignal("cpu"),
+ FirstMatchedAt: time.Now().Add(-10 * time.Minute),
+ LastObservedAt: time.Now().Add(-9 * time.Minute),
+ }
+ m.mu.Unlock()
+
+ m.releaseCanonicalMetricAlert(spec, "delly2", "delly2", "homelab", "node", 0, nil)
+
+ m.mu.RLock()
+ _, stillPending := m.intentPending[trackingKey]
+ m.mu.RUnlock()
+ if stillPending {
+ t.Fatalf("expected releasing the metric to clear its intent pending state")
+ }
+}
+
+// Missing storage observations break confirmation runs, not the occurrence.
+// Exercise the public evaluator and both live/shadow reducers, not seeded state.
+func TestStorageConnectivityObservationGaps(t *testing.T) {
+ for _, gap := range []string{"", "unknown", " UNKNOWN "} {
+ t.Run("activation/"+gap, func(t *testing.T) {
+ m := newShadowFeedManager(t)
+ s := models.Storage{ID: "storage-a", Name: "backups", Status: "offline"}
+ other := s
+ other.ID = "storage-ab"
+ id := canonicalConnectivityStateID(s.ID)
+ m.CheckStorage(s)
+ m.CheckStorage(other)
+ s.Status = gap
+ m.CheckStorage(s)
+ s.Status = "unavailable"
+ m.CheckStorage(s)
+ if testHasActiveAlert(t, m, id) {
+ t.Fatal("non-consecutive outage observations fired storage alert")
+ }
+ m.CheckStorage(other)
+ testRequireActiveAlert(t, m, canonicalConnectivityStateID(other.ID))
+ m.CheckStorage(s)
+ testRequireActiveAlert(t, m, id)
+ if m.ShadowDivergences() != 0 {
+ t.Fatal("storage interruption diverged from shadow")
+ }
+ })
+ t.Run("recovery/"+gap, func(t *testing.T) {
+ m := newShadowFeedManager(t)
+ s := models.Storage{ID: "storage-a", Name: "backups", Status: "offline"}
+ id := canonicalConnectivityStateID(s.ID)
+ for range 2 {
+ m.CheckStorage(s)
+ }
+ if err := m.AcknowledgeAlert(id, "operator"); err != nil {
+ t.Fatal(err)
+ }
+ s.Status = "available"
+ m.CheckStorage(s)
+ before := testRequireActiveAlert(t, m, id).Clone()
+ s.Status = gap
+ m.CheckStorage(s)
+ if !reflect.DeepEqual(before, testRequireActiveAlert(t, m, id).Clone()) || m.GetResolvedAlert(id) != nil {
+ t.Fatal("missing connectivity changed acknowledged occurrence")
+ }
+ s.Status = "available"
+ m.CheckStorage(s)
+ if !testHasActiveAlert(t, m, id) {
+ t.Fatal("non-consecutive healthy observations resolved storage outage")
+ }
+ m.CheckStorage(s)
+ if testHasActiveAlert(t, m, id) {
+ t.Fatal("fresh consecutive healthy observations failed to recover")
+ }
+ resolved := m.GetResolvedAlert(id)
+ if resolved == nil || !resolved.StartTime.Equal(before.StartTime) {
+ t.Fatal("confirmed recovery lost original occurrence")
+ }
+ if m.ShadowDivergences() != 0 {
+ t.Fatal("storage recovery diverged from shadow")
+ }
+ })
+ }
+}
+
+func TestStorageConnectivityGapRestartsIntentGrace(t *testing.T) {
+ m := newShadowFeedManager(t)
+ var tick time.Duration
+ m.intentClock = func() time.Duration { return tick }
+ doc := NewAlertIntentPolicyDocument()
+ doc.Defaults[string(AlertIntentSignalOffline)] = AlertIntentRule{GraceSeconds: intPointer(60)}
+ if err := m.LoadIntentPolicies(doc); err != nil {
+ t.Fatal(err)
+ }
+ s := models.Storage{ID: "storage-a", Name: "backups", Status: "offline"}
+ id := canonicalConnectivityStateID(s.ID)
+ m.CheckStorage(s)
+ tick = 40 * time.Second
+ m.CheckStorage(s)
+ s.Status = "unknown"
+ m.CheckStorage(s)
+ m.mu.RLock()
+ _, pending := m.intentPending[id]
+ _, ticking := m.intentRuntimeTicks[id]
+ m.mu.RUnlock()
+ if pending || ticking {
+ t.Error("missing status retained intent grace bookkeeping")
+ }
+ if err := m.SaveActiveAlerts(); err != nil {
+ t.Fatal(err)
+ }
+ data, err := os.ReadFile(filepath.Join(m.getAlertsDir(), intentPendingFileName))
+ if err != nil {
+ t.Fatal(err)
+ }
+ var saved []IntentPendingState
+ if err := json.Unmarshal(data, &saved); err != nil || len(saved) != 0 {
+ t.Fatalf("interrupted intent remained checkpointed: %s, %v", data, err)
+ }
+ s.Status = "offline"
+ for _, next := range []time.Duration{70, 90, 129} {
+ tick = next * time.Second
+ m.CheckStorage(s)
+ if testHasActiveAlert(t, m, id) {
+ t.Fatal("intent grace accrued across missing storage status")
+ }
+ }
+ tick = 130 * time.Second
+ m.CheckStorage(s)
+ testRequireActiveAlert(t, m, id)
+ if m.ShadowDivergences() != 0 {
+ t.Fatal("intent interruption diverged from shadow")
+ }
+}
+
+// Usage has delayed activation but immediate measured recovery. Do not invent
+// a recovery stability window to fit the memory/temperature timing fixtures.
+func TestStorageCapacityGapHoldsOccurrence(t *testing.T) {
+ for _, gap := range []string{"missing", "unconfirmed-zero", "negative", "offline", "unavailable"} {
+ t.Run(gap, func(t *testing.T) {
+ m, elapsed := continuityManager(t, false)
+ id, metric, observe := continuityObserver(t, m, "storage", gap)
+ alertID := canonicalMetricStateID(id, metric)
+ observe(95, false)
+ elapsed.Store(int64(time.Minute))
+ observe(95, false)
+ if err := m.AcknowledgeAlert(alertID, "operator"); err != nil {
+ t.Fatal(err)
+ }
+ before := testRequireActiveAlert(t, m, alertID).Clone()
+ elapsed.Store(int64(2 * time.Minute))
+ observe(10, true)
+ if !reflect.DeepEqual(before, testRequireActiveAlert(t, m, alertID).Clone()) || m.GetResolvedAlert(alertID) != nil {
+ t.Fatal("missing capacity changed the acknowledged occurrence")
+ }
+ // A real, independently observed empty capacity can clear even
+ // while connectivity is unknown; neither channel stands in for the other.
+ empty := models.Storage{ID: id, Name: "backups", Status: "unknown", Total: 1000, Free: 1000}
+ m.CheckStorage(empty)
+ if testHasActiveAlert(t, m, alertID) {
+ t.Fatal("confirmed empty capacity did not recover immediately")
+ }
+ resolved := m.GetResolvedAlert(alertID)
+ if resolved == nil || !resolved.StartTime.Equal(before.StartTime) || resolved.Value != 0 {
+ t.Fatal("measured recovery lost the original occurrence or clearing value")
+ }
+ if m.ShadowDivergences() != 0 {
+ t.Fatal("capacity interruption diverged from shadow")
+ }
+ })
+ }
+}
++
+ // HostDiskTemperatureThreshold is the disk heat policy for a disk a host agent
+ // reports, so it must agree with CheckHost on every host-level override: an
+ // explicit Disk Temp override on the host (or the canonical resource its ID
+ // resolves to), or inherited from its linked node or guest, replaces the
+ // per-type entry; the first override in the chain decides even when it sets no
+ // Disk Temp; and a host whose alerts are off judges no heat.
+ func TestHostDiskTemperatureThresholdMatchesCheckHostOverrides(t *testing.T) {
+ const hostID = "host-temp-policy"
+ const nodeID = "pve1-node1"
+ const vmID = "pve1-node1-105"
+ const canonicalID = "agent-canonical-temp-policy"
+ perType := &HysteresisThreshold{Trigger: 70, Clear: 65}
+
+ for _, tc := range []struct {
+ name string
+ overrides map[string]ThresholdConfig
+ linkNode bool
+ linkVM bool
+ canonical bool
+ want *HysteresisThreshold
+ off bool
+ }{
+ {name: "no override", want: perType},
+ {
+ name: "host disk temp override",
+ overrides: map[string]ThresholdConfig{hostID: {DiskTemperature: &HysteresisThreshold{Trigger: 80, Clear: 75}}},
+ want: &HysteresisThreshold{Trigger: 80, Clear: 75},
+ },
+ {
+ name: "host override without disk temp",
+ overrides: map[string]ThresholdConfig{hostID: {Memory: &HysteresisThreshold{Trigger: 90, Clear: 85}}},
+ want: perType,
+ },
+ {
+ name: "linked node disk temp override",
+ overrides: map[string]ThresholdConfig{nodeID: {DiskTemperature: &HysteresisThreshold{Trigger: 78, Clear: 72}}},
+ linkNode: true,
+ want: &HysteresisThreshold{Trigger: 78, Clear: 72},
+ },
+ {
+ name: "host override shadows linked node",
+ overrides: map[string]ThresholdConfig{
+ hostID: {Memory: &HysteresisThreshold{Trigger: 90, Clear: 85}},
+ nodeID: {DiskTemperature: &HysteresisThreshold{Trigger: 78, Clear: 72}},
+ },
+ linkNode: true,
+ want: perType,
+ },
+ {
+ // The agent ID resolves to the machine's canonical resource ID.
+ name: "canonical machine disk temp override",
+ overrides: map[string]ThresholdConfig{canonicalID: {DiskTemperature: &HysteresisThreshold{Trigger: 82, Clear: 77}}},
+ canonical: true,
+ want: &HysteresisThreshold{Trigger: 82, Clear: 77},
+ },
+ {
+ name: "linked guest disk temp override",
+ overrides: map[string]ThresholdConfig{vmID: {DiskTemperature: &HysteresisThreshold{Trigger: 76, Clear: 71}}},
+ linkVM: true,
+ want: &HysteresisThreshold{Trigger: 76, Clear: 71},
+ },
+ {
+ name: "linked guest alerts disabled",
+ overrides: map[string]ThresholdConfig{vmID: {Disabled: true}},
+ linkVM: true,
+ },
+ {
+ name: "host disk temp switched off",
+ overrides: map[string]ThresholdConfig{hostID: {DiskTemperature: &HysteresisThreshold{}}},
+ off: true,
+ },
+ {
+ name: "host alerts disabled",
+ overrides: map[string]ThresholdConfig{hostID: {Disabled: true}},
+ },
+ {
+ name: "linked node alerts disabled",
+ overrides: map[string]ThresholdConfig{nodeID: {Disabled: true}},
+ linkNode: true,
+ },
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ m := configureDiskTempTypeHostManager(t)
+ m.mu.Lock()
+ for id, override := range tc.overrides {
+ m.config.Overrides[id] = override
+ }
+ m.mu.Unlock()
+ if tc.canonical {
+ m.SetResourceIntentIdentityResolver(func(resourceID string) (string, bool) {
+ return canonicalID, resourceID == hostID
+ })
+ }
+
+ owner := DiskTemperatureHost{ID: hostID}
+ if tc.linkNode {
+ owner.LinkedNodeID = nodeID
+ }
+ if tc.linkVM {
+ owner.LinkedVMID = vmID
+ }
+ got := m.HostDiskTemperatureThreshold(owner, "nvme")
+ switch {
+ case tc.off:
+ if got == nil || got.Trigger > 0 {
+ t.Fatalf("HostDiskTemperatureThreshold = %+v, want a switched-off threshold", got)
+ }
+ case (got == nil) != (tc.want == nil) || (got != nil && *got != *tc.want):
+ t.Fatalf("HostDiskTemperatureThreshold = %+v, want %+v", got, tc.want)
+ }
+
+ check := func(temperature int) bool {
+ host := hostWithSMARTDiskTemp(hostID, "nvme", temperature)
+ host.LinkedNodeID = owner.LinkedNodeID
+ host.LinkedVMID = owner.LinkedVMID
+ m.CheckHost(host)
+ _, exists := testLookupActiveAlert(t, m, hostDiskTempAlertID(host))
+ return exists
+ }
+ if got == nil || got.Trigger <= 0 {
+ if check(99) {
+ t.Fatalf("CheckHost alerted at 99C on a host the policy judges no heat for, active: %v", alertKeys(m))
+ }
+ return
+ }
+ trigger := int(got.Trigger)
+ if check(trigger - 1) {
+ t.Fatalf("CheckHost alerted at %dC, under the policy trigger %v", trigger-1, got.Trigger)
+ }
+ if !check(trigger) {
+ t.Fatalf("CheckHost stayed quiet at the policy trigger %dC, active: %v", trigger, alertKeys(m))
+ }
+ })
+ }
+
+ m := configureDiskTempTypeHostManager(t)
+ if got, want := m.HostDiskTemperatureThreshold(DiskTemperatureHost{}, "sas"), m.DiskTemperatureThreshold("sas"); got == nil || want == nil || *got != *want {
+ t.Fatalf("hostless policy = %+v, want DiskTemperatureThreshold %+v", got, want)
+ }
+ var nilManager *Manager
+ if got := nilManager.HostDiskTemperatureThreshold(DiskTemperatureHost{ID: hostID}, "nvme"); got == nil || got.Trigger != 70 {
+ t.Fatalf("nil manager host policy = %+v, want the factory nvme 70", got)
+ }
+ }
diff --cc internal/monitoring/monitor.go
index 8bc8a5519f,25be288f63..176e1c6952
--- a/internal/monitoring/monitor.go
+++ b/internal/monitoring/monitor.go
@@@ -1282,9 -1330,8 +1331,10 @@@ type Monitor struct
mockMetricsCancel context.CancelFunc
mockMetricsWg sync.WaitGroup
mockHostAgentsMu sync.Mutex
+ mockDockerHosts map[string]models.DockerHost
+ mockFixtureRevision uint64
mockHostAgents map[string]models.Host // Fixture agents evaluated by the last mock alert pass
+ mockModeFence mockModeFence // Keeps mode-dependent alert evaluations inside the epoch they read in
dockerChecker DockerChecker // Optional Docker checker for LXC containers
dockerCheckerConfiguredAt time.Time // Last time the Docker checker was configured
dockerCheckAllowedVMIDs map[int]struct{} // Optional VMID allowlist gating the LXC Docker socket probe; empty means all guests
@@@ -4616,8 -4671,9 +4680,9 @@@ func (m *Monitor) SetMockMode(enable bo
if err := mock.SetEnabled(false); err != nil {
return err
}
+ m.mockModeFence.advance()
m.alertManager.ClearActiveAlerts()
- m.forgetMockHostAgents()
+ m.forgetMockFixtureHosts()
m.mu.Lock()
m.resetStateLocked()
m.metricsHistory.Reset()
diff --cc internal/monitoring/monitor_alerts.go
index 00dc149c28,afb150ad62..d8f3f956d8
--- a/internal/monitoring/monitor_alerts.go
+++ b/internal/monitoring/monitor_alerts.go
@@@ -697,10 -702,10 +703,10 @@@ func (m *Monitor) checkMockAlerts()
// the monitor state the host health sweep reads, so without this pass no
// agent CPU, memory, disk, temperature or offline alert can open against
// mock data. Agents go before nodes so a linked node hands its CPU, memory
- // and disk alerts to its agent on the first tick, as the host-agent
- // hostname deduplication does in production.
+ // and disk alerts to its agent on the first tick, as node-link
+ // deduplication does in production.
log.Debug().Int("hostCount", len(state.Hosts)).Msg("checking host agent alerts")
- m.evaluateMockHostAgents(state.Hosts, state.Nodes, fixtureRevision)
- m.evaluateMockHostAgents(scope, state.Hosts, state.Nodes)
++ m.evaluateMockHostAgents(scope, state.Hosts, state.Nodes, fixtureRevision)
// Check alerts for each node
for _, node := range state.Nodes {
@@@ -742,7 -748,9 +749,7 @@@
// so skipping this loop leaves the docker alert lifecycle unexercisable
// against mock data.
log.Debug().Int("dockerHostCount", len(state.DockerHosts)).Msg("checking docker alerts")
- m.evaluateMockDockerHosts(state.DockerHosts, fixtureRevision)
- for _, dockerHost := range state.DockerHosts {
- scope.run(func() { m.checkMockDockerHostAlerts(dockerHost) })
- }
++ m.evaluateMockDockerHosts(scope, state.DockerHosts, fixtureRevision)
// Cache the latest alert snapshots directly in the mock data so the API can serve
// mock state without needing to grab the alert manager lock again.
@@@ -752,12 -762,16 +761,13 @@@
// evaluateMockHostAgents evaluates every fixture agent and remembers the set.
// A runtime mock config change rebuilds the estate, so an agent can leave it
// between passes; it then goes through HandleHostRemoved, as a deleted live
-// agent does, or its alerts and hostname deduplication would outlive it.
-func (m *Monitor) evaluateMockHostAgents(scope mockModeScope, hosts []models.Host, nodes []models.Node) {
+// agent does, or its alerts and node link would outlive it.
- func (m *Monitor) evaluateMockHostAgents(hosts []models.Host, nodes []models.Node, fixtureRevision uint64) {
++func (m *Monitor) evaluateMockHostAgents(scope mockModeScope, hosts []models.Host, nodes []models.Node, fixtureRevision uint64) {
m.mockHostAgentsMu.Lock()
defer m.mockHostAgentsMu.Unlock()
- if !m.acceptMockFixturePassLocked(fixtureRevision) {
- // A pass that took its snapshot before the monitor left mock mode must
- // not re-register fixture agents after forgetMockHostAgents ran.
- // SetMockMode ends the epoch before it forgets them, so either this check
- // fails or forgetMockHostAgents waits for this pass to record its set.
- if !scope.current() {
++ // Both the mode epoch and structural revision must still match.
++ if !scope.current() || !m.acceptMockFixturePassLocked(fixtureRevision) {
return
}
@@@ -767,9 -781,16 +777,14 @@@
current[host.ID] = host
}
}
- // Remove first so a departed agent cannot unregister a hostname that a
- // remaining agent registers on this same pass.
for id, host := range m.mockHostAgents {
- if _, ok := current[id]; !ok {
- m.alertManager.HandleHostRemoved(host)
+ if _, ok := current[id]; ok {
+ continue
+ }
+ if !scope.run(func() { m.alertManager.HandleHostRemoved(host) }) {
+ // Leaving mock mode refused the removal; keep the agent so
- // forgetMockHostAgents still removes it.
++ // forgetMockFixtureHosts still removes it.
+ current[id] = host
}
}
for _, host := range hosts {
@@@ -778,69 -799,16 +793,73 @@@
m.mockHostAgents = current
}
-// forgetMockHostAgents removes the fixture agents when the monitor leaves mock
-// mode. ClearActiveAlerts drops their alerts but not their hostname
-// deduplication, so a real node named like a fixture agent (pve1) would keep
-// its CPU, memory and disk alerts suppressed with no agent to own them.
-// SetMockMode ends the mock epoch before this runs, so the set it removes is
-// final: no pass of that epoch can register an agent or evaluate one again.
-func (m *Monitor) forgetMockHostAgents() {
+// evaluateMockDockerHosts is the Docker counterpart of evaluateMockHostAgents,
+// under the same lock and mode check. Nested Docker-in-LXC hosts are named
+// after their guest's VMID, so a runtime mock config change can drop a host
+// from the estate; it then goes through HandleDockerHostRemoved, as a deleted
+// live host does. pruneStaleDockerAlerts clears the same alerts, but only when
+// something reads state, so an unwatched stack would keep them active.
- func (m *Monitor) evaluateMockDockerHosts(hosts []models.DockerHost, fixtureRevision uint64) {
++func (m *Monitor) evaluateMockDockerHosts(scope mockModeScope, hosts []models.DockerHost, fixtureRevision uint64) {
m.mockHostAgentsMu.Lock()
defer m.mockHostAgentsMu.Unlock()
- if !m.acceptMockFixturePassLocked(fixtureRevision) {
++ // Both the mode epoch and structural revision must still match.
++ if !scope.current() || !m.acceptMockFixturePassLocked(fixtureRevision) {
+ return
+ }
+
+ current := make(map[string]models.DockerHost, len(hosts))
+ for _, host := range hosts {
+ if host.ID != "" {
+ current[host.ID] = host
+ }
+ }
+ for id, host := range m.mockDockerHosts {
+ if _, ok := current[id]; !ok {
- m.alertManager.HandleDockerHostRemoved(host)
++ if !scope.run(func() { m.alertManager.HandleDockerHostRemoved(host) }) {
++ // Keep refused departures for the mode switch to remove.
++ current[id] = host
++ }
+ }
+ }
+ for _, host := range hosts {
- m.checkMockDockerHostAlerts(host)
++ scope.run(func() { m.checkMockDockerHostAlerts(host) })
+ }
+ m.mockDockerHosts = current
+}
+
+// acceptMockFixturePassLocked reports whether a mock alert pass may update the
+// tracked fixture hosts. A pass that took its snapshot before the monitor left
+// mock mode must not evaluate fixtures after forgetMockFixtureHosts ran. Ticks
+// start passes concurrently, so one that took its snapshot before an estate
+// rebuild can also reach this lock after a newer pass; applying it would
+// remove the new estate's hosts and re-evaluate retired ones. The fixture
+// revision only advances on such rebuilds, so an older revision is rejected.
+// Callers hold mockHostAgentsMu.
+func (m *Monitor) acceptMockFixturePassLocked(fixtureRevision uint64) bool {
+ if !mock.IsMockEnabled() || fixtureRevision < m.mockFixtureRevision {
+ return false
+ }
+ m.mockFixtureRevision = fixtureRevision
+ return true
+}
+
+// forgetMockFixtureHosts removes the fixture agents and Docker hosts when the
+// monitor leaves mock mode. ClearActiveAlerts drops their alerts but not the
+// agents' node links, and a leftover link would keep suppressing CPU, memory
+// and disk alerts on any node that later carries the linked fixture node's ID.
+// The removal also clears anything a pass already in flight
+// recreated; mock mode is off before this runs, so no later pass can evaluate
+// them again while it stays off. Disabling advanced the fixture revision, and
+// recording it here keeps a pass paused across a disable and re-enable from
+// evaluating the old estate once mock mode is back on.
+func (m *Monitor) forgetMockFixtureHosts() {
+ m.mockHostAgentsMu.Lock()
+ defer m.mockHostAgentsMu.Unlock()
+
+ if _, revision := mock.CurrentFixtureGraphWithRevision(); revision > m.mockFixtureRevision {
+ m.mockFixtureRevision = revision
+ }
for _, host := range m.mockHostAgents {
m.alertManager.HandleHostRemoved(host)
}
diff --cc internal/monitoring/monitor_host_agent_removal_lifecycle_test.go
index 1cc2468142,dca2a68b1c..b921bf36d2
--- a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go
+++ b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go
@@@ -821,11 -821,11 +821,11 @@@ func TestMockHostAgentLeavingFixtureUse
memory := models.Memory{Total: 32 << 30, Used: 30 << 30, Free: 2 << 30, Usage: 93.75}
kept := models.Host{ID: "host-linux-1", Hostname: "apollo-114", Status: "online", Memory: memory}
- departed := models.Host{ID: "host-node-pve9", Hostname: "pve9", Status: "online", Memory: memory}
+ departed := models.Host{ID: "host-node-pve9", Hostname: "pve9", LinkedNodeID: "mock-cluster-1-pve9", Status: "online", Memory: memory}
- monitor.evaluateMockHostAgents([]models.Host{kept, departed}, nil, 1)
- monitor.evaluateMockHostAgents(monitor.mockModeFence.begin(), []models.Host{kept, departed}, nil)
++ monitor.evaluateMockHostAgents(monitor.mockModeFence.begin(), []models.Host{kept, departed}, nil, 1)
// A runtime mock config change rebuilt the estate without pve9.
- monitor.evaluateMockHostAgents([]models.Host{kept}, nil, 2)
- monitor.evaluateMockHostAgents(monitor.mockModeFence.begin(), []models.Host{kept}, nil)
++ monitor.evaluateMockHostAgents(monitor.mockModeFence.begin(), []models.Host{kept}, nil, 2)
keptAlert := false
for _, alert := range manager.GetActiveAlerts() {
@@@ -857,30 -857,3 +857,30 @@@
}
t.Fatal("node pve9 raised no memory alert after its agent left the mock estate")
}
+
+func TestMockDockerHostLeavingFixtureUsesRemovalLifecycle(t *testing.T) {
+ mustSetMockEnabled(t, true)
+ t.Cleanup(func() { mustSetMockEnabled(t, false) })
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{alertManager: manager}
+
+ kept := newMockDockerHostWithExitedContainers("nebula-1-mock", "nebula-1")
+ // Nested Docker-in-LXC hosts are named after their guest's VMID, which a
+ // runtime mock config change can shift.
+ departed := newMockDockerHostWithExitedContainers("proxmox-lxc-docker:Production West:pve1:108", "pve1-ct108")
+
- monitor.evaluateMockDockerHosts([]models.DockerHost{kept, departed}, 1)
- monitor.evaluateMockDockerHosts([]models.DockerHost{kept, departed}, 1)
++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{kept, departed}, 1)
++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{kept, departed}, 1)
+ if ids := dockerAlertIDsForHost(manager, departed.ID); len(ids) != len(departed.Containers) {
+ t.Fatalf("departing host opened %v, want one alert per exited container", ids)
+ }
+
- monitor.evaluateMockDockerHosts([]models.DockerHost{kept}, 2)
++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{kept}, 2)
+
+ if ids := dockerAlertIDsForHost(manager, departed.ID); len(ids) != 0 {
+ t.Fatalf("Docker host that left the mock estate kept alerts %v", ids)
+ }
+ if ids := dockerAlertIDsForHost(manager, kept.ID); len(ids) != len(kept.Containers) {
+ t.Fatalf("Docker host still in the mock estate has alerts %v, want one per exited container", ids)
+ }
+}
diff --cc internal/monitoring/monitor_mock_alerts_test.go
index 8110f2ba1d,3c02565ece..e45c690ff8
--- a/internal/monitoring/monitor_mock_alerts_test.go
+++ b/internal/monitoring/monitor_mock_alerts_test.go
@@@ -211,18 -219,18 +221,19 @@@ func TestLeavingMockModeReleasesFixture
func TestMockHostAgentPassAfterLeavingMockModeRegistersNothing(t *testing.T) {
manager := newMockHostAlertTestManager(t)
monitor := &Monitor{alertManager: manager}
- mustSetMockEnabled(t, false)
// A pass whose snapshot predates leaving mock mode reaches the agent step
- // after forgetMockFixtureHosts ran.
- // after SetMockMode ended its epoch and forgetMockHostAgents ran.
++ // after SetMockMode ended its epoch and forgetMockFixtureHosts ran.
+ scope := monitor.mockModeFence.begin()
+ monitor.mockModeFence.advance()
stale := models.Host{
- ID: "host-node-mock-cluster-1-pve1",
- Hostname: "pve1",
- Status: "online",
- Memory: models.Memory{Total: 32 << 30, Used: 30 << 30, Free: 2 << 30, Usage: 93.75},
+ ID: "host-node-mock-cluster-1-pve1",
+ Hostname: "pve1",
+ LinkedNodeID: "real-pve1",
+ Status: "online",
+ Memory: models.Memory{Total: 32 << 30, Used: 30 << 30, Free: 2 << 30, Usage: 93.75},
}
- monitor.evaluateMockHostAgents([]models.Host{stale}, nil, 1)
- monitor.evaluateMockHostAgents(scope, []models.Host{stale}, nil)
++ monitor.evaluateMockHostAgents(scope, []models.Host{stale}, nil, 1)
if active := manager.GetActiveAlerts(); len(active) != 0 {
t.Fatalf("stale mock pass opened %d alerts in live mode", len(active))
@@@ -240,121 -251,564 +254,707 @@@
return
}
}
- t.Fatal("stale mock pass registered fixture hostname pve1 and suppressed the real node's memory alert")
+ t.Fatal("stale mock pass linked a fixture agent to real-pve1 and suppressed its memory alert")
+}
+
+// newMockDockerHostWithExitedContainers returns a reporting Docker host whose
+// containers exited, so two evaluation passes open one alert per container.
+func newMockDockerHostWithExitedContainers(id, hostname string) models.DockerHost {
+ return models.DockerHost{
+ ID: id,
+ Hostname: hostname,
+ DisplayName: hostname,
+ Status: "online",
+ Containers: []models.DockerContainer{
+ {ID: id + "-web", Name: "web", State: "exited", Status: "Exited (137)"},
+ {ID: id + "-db", Name: "db", State: "exited", Status: "Exited (137)"},
+ },
+ }
+}
+
+func dockerAlertIDsForHost(manager *alerts.Manager, hostID string) []string {
+ var ids []string
+ for _, alert := range manager.GetActiveAlerts() {
+ if alert.ResourceID == "docker:"+hostID || strings.HasPrefix(alert.ResourceID, "docker:"+hostID+"/") {
+ ids = append(ids, alert.ID)
+ }
+ }
+ return ids
+}
+
+func TestMockDockerPassAfterLeavingMockModeRaisesNothing(t *testing.T) {
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{alertManager: manager}
+ mustSetMockEnabled(t, false)
+
+ // A pass whose snapshot predates leaving mock mode reaches the Docker
+ // step after forgetMockFixtureHosts ran.
+ host := newMockDockerHostWithExitedContainers("nebula-1-mock", "nebula-1")
- monitor.evaluateMockDockerHosts([]models.DockerHost{host}, 1)
- monitor.evaluateMockDockerHosts([]models.DockerHost{host}, 1)
++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{host}, 1)
++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{host}, 1)
+
+ if ids := dockerAlertIDsForHost(manager, host.ID); len(ids) != 0 {
+ t.Fatalf("stale mock pass opened Docker alerts in live mode: %v", ids)
+ }
+}
+
+func TestLeavingMockModeRemovesFixtureDockerHostAlerts(t *testing.T) {
+ mustSetMockEnabled(t, true)
+ t.Cleanup(func() { mustSetMockEnabled(t, false) })
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{alertManager: manager}
+
+ host := newMockDockerHostWithExitedContainers("nebula-1-mock", "nebula-1")
- monitor.evaluateMockDockerHosts([]models.DockerHost{host}, 1)
- monitor.evaluateMockDockerHosts([]models.DockerHost{host}, 1)
++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{host}, 1)
++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{host}, 1)
+ if ids := dockerAlertIDsForHost(manager, host.ID); len(ids) != len(host.Containers) {
+ t.Fatalf("fixture Docker host opened %v, want one alert per exited container", ids)
+ }
+
+ // The alerts stand in for ones a pass in flight recreated after
+ // SetMockMode(false) cleared the manager.
+ mustSetMockEnabled(t, false)
+ monitor.forgetMockFixtureHosts()
+
+ if ids := dockerAlertIDsForHost(manager, host.ID); len(ids) != 0 {
+ t.Fatalf("fixture Docker host alerts outlived mock mode: %v", ids)
+ }
+}
+
++// Upstream mode epochs and the reviewed structural revisions protect different
++// boundaries. A current revision cannot make an ended epoch eligible again.
++func TestMockFixtureCurrentRevisionCannotReviveAnEndedEpoch(t *testing.T) {
++ mustSetMockEnabled(t, true)
++ t.Cleanup(func() { mustSetMockEnabled(t, false) })
++ manager := newMockHostAlertTestManager(t)
++ monitor := &Monitor{alertManager: manager}
++ stale := monitor.mockModeFence.begin()
++ monitor.mockModeFence.advance()
++ _, revision := mock.CurrentFixtureGraphWithRevision()
++ agent := models.Host{ID: "epoch-agent", Hostname: "epoch-agent", Status: "online"}
++ docker := newMockDockerHostWithExitedContainers("epoch-docker", "epoch-docker")
++
++ monitor.evaluateMockHostAgents(stale, []models.Host{agent}, nil, revision)
++ for range 2 {
++ monitor.evaluateMockDockerHosts(stale, []models.DockerHost{docker}, revision)
++ }
++ if len(monitor.mockHostAgents) != 0 || len(monitor.mockDockerHosts) != 0 || len(manager.GetActiveAlerts()) != 0 {
++ t.Fatal("a current fixture revision revived an ended mode epoch")
++ }
++
++ current := monitor.mockModeFence.begin()
++ monitor.evaluateMockHostAgents(current, []models.Host{agent}, nil, revision)
++ for range 2 {
++ monitor.evaluateMockDockerHosts(current, []models.DockerHost{docker}, revision)
++ }
++ if len(monitor.mockHostAgents) != 1 || len(monitor.mockDockerHosts) != 1 || len(dockerAlertIDsForHost(manager, docker.ID)) != len(docker.Containers) {
++ t.Fatal("the current epoch lost normal fixture ownership or Docker evaluation")
++ }
++}
++
+func TestMockFixturePassFromBeforeAnEstateRebuildIsRejected(t *testing.T) {
+ mustSetMockEnabled(t, true)
+ t.Cleanup(func() { mustSetMockEnabled(t, false) })
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{alertManager: manager}
+
+ retired := newMockDockerHostWithExitedContainers("proxmox-lxc-docker:Production West:pve1:108", "pve1-ct108")
+ current := newMockDockerHostWithExitedContainers("proxmox-lxc-docker:Production West:pve1:105", "pve1-ct105")
- monitor.evaluateMockDockerHosts([]models.DockerHost{current}, 2)
- monitor.evaluateMockDockerHosts([]models.DockerHost{current}, 2)
++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{current}, 2)
++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{current}, 2)
+
+ // A slow pass whose snapshot predates the rebuild reaches the lock after
+ // the newer passes. Applying it would remove the current host and
+ // re-evaluate the retired one.
- monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, 1)
- monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, 1)
++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{retired}, 1)
++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{retired}, 1)
+
+ if ids := dockerAlertIDsForHost(manager, current.ID); len(ids) != len(current.Containers) {
+ t.Fatalf("superseded pass disturbed the current host's alerts: %v", ids)
+ }
+ if ids := dockerAlertIDsForHost(manager, retired.ID); len(ids) != 0 {
+ t.Fatalf("superseded pass re-evaluated a retired host: %v", ids)
+ }
+}
+
+func TestMockFixturePassPausedAcrossDisableAndReenableIsRejected(t *testing.T) {
+ mustSetMockEnabled(t, true)
+ t.Cleanup(func() { mustSetMockEnabled(t, false) })
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{alertManager: manager}
+
+ retired := newMockDockerHostWithExitedContainers("proxmox-lxc-docker:Production West:pve1:108", "pve1-ct108")
+ _, staleRevision := mock.CurrentFixtureGraphWithRevision()
- monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, staleRevision)
++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{retired}, staleRevision)
+
+ // Mock mode goes off and back on while a pass holding the old snapshot
+ // is paused; it resumes before any pass of the rebuilt estate applies.
+ mustSetMockEnabled(t, false)
+ monitor.forgetMockFixtureHosts()
+ mustSetMockEnabled(t, true)
- monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, staleRevision)
- monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, staleRevision)
++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{retired}, staleRevision)
++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{retired}, staleRevision)
+
+ if ids := dockerAlertIDsForHost(manager, retired.ID); len(ids) != 0 {
+ t.Fatalf("pass from before the disable re-evaluated the old estate: %v", ids)
+ }
}
+ // pinDefaultMockEstate enables mock mode on the default estate, whatever
+ // PULSE_MOCK_* the shell exports, and restores the previous mode and config.
+ func pinDefaultMockEstate(t *testing.T) {
+ t.Helper()
+ previousEnabled, previousConfig := mock.IsMockEnabled(), mock.GetConfig()
+ t.Cleanup(func() {
+ mustSetMockEnabled(t, false)
+ mock.SetMockConfig(previousConfig)
+ mustSetMockEnabled(t, previousEnabled)
+ })
+ mustSetMockEnabled(t, false)
+ cfg := mock.DefaultConfig
+ cfg.UpdateInterval = 5 * time.Minute
+ mock.SetMockConfig(cfg)
+ }
+
+ // signalFirstFiring closes the returned channel when manager fires its first
+ // alert. Lifecycle callbacks run inside the evaluation that fired, so the
+ // pass that fired is past its fixture read when the channel closes.
+ func signalFirstFiring(t *testing.T, manager *alerts.Manager) <-chan struct{} {
+ t.Helper()
+ fired := make(chan struct{})
+ var once sync.Once
+ unsubscribe := manager.SubscribeLifecycleCallback(func(event alerts.LifecycleEvent) {
+ if event.Type == eventlog.TypeFired {
+ once.Do(func() { close(fired) })
+ }
+ })
+ t.Cleanup(unsubscribe)
+ return fired
+ }
+
+ func describeAlerts(active []alerts.Alert) string {
+ keys := make([]string, 0, len(active))
+ for _, alert := range active {
+ keys = append(keys, alert.Type+" "+alert.ResourceID)
+ }
+ sort.Strings(keys)
+ return strings.Join(keys, ", ")
+ }
+
+ // switchMockModeAsync runs SetMockMode on its own goroutine, so a test that is
+ // holding an evaluation parked can still bound how long the switch takes.
+ func switchMockModeAsync(t *testing.T, monitor *Monitor, enable bool) <-chan struct{} {
+ t.Helper()
+ done := make(chan struct{})
+ go func() {
+ defer close(done)
+ if err := monitor.SetMockMode(enable); err != nil {
+ t.Errorf("SetMockMode(%v): %v", enable, err)
+ }
+ }()
+ return done
+ }
+
+ // goroutineIn reports whether one goroutine's stack holds every frame.
+ func goroutineIn(frames ...string) bool {
+ buf := make([]byte, 4<<20)
+ n := runtime.Stack(buf, true)
+ for _, goroutine := range bytes.Split(buf[:n], []byte("\n\n")) {
+ all := true
+ for _, frame := range frames {
+ all = all && bytes.Contains(goroutine, []byte(frame))
+ }
+ if all {
+ return true
+ }
+ }
+ return false
+ }
+
+ // waitForGoroutineIn waits until one goroutine's stack holds every frame. A
+ // test holding a lock uses it to see the goroutine it started parked on that
+ // lock at a known point.
+ func waitForGoroutineIn(t *testing.T, frames ...string) {
+ t.Helper()
+ waitForCondition(t, 30*time.Second, func() bool { return goroutineIn(frames...) },
+ fmt.Sprintf("no goroutine reached %s", strings.Join(frames, " via ")))
+ }
+
+ func TestLeavingMockModeRefusesTheRestOfAnInFlightPass(t *testing.T) {
+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+ pinDefaultMockEstate(t)
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{
+ state: models.NewState(),
+ alertManager: manager,
+ metricsHistory: NewMetricsHistory(10, time.Hour),
+ }
+ mustSetMonitorMockMode(t, monitor, true)
+
+ // Park a real pass at its first storage capacity read: it has read the
+ // fixture and evaluated guests, agents and nodes; storage, disks, PBS,
+ // PMG and Docker are still ahead of it.
+ monitor.metricsHistory.capacityForecastMu.Lock()
+ parked := true
+ defer func() {
+ if parked {
+ monitor.metricsHistory.capacityForecastMu.Unlock()
+ }
+ }()
+ passDone := make(chan struct{})
+ go func() {
+ defer close(passDone)
+ monitor.checkMockAlerts()
+ }()
+ waitForGoroutineIn(t, "monitoring.(*Monitor).storageCapacityTrendFor", "monitoring.(*Monitor).checkMockAlerts")
+
+ waitForSignal(t, switchMockModeAsync(t, monitor, false), 30*time.Second, "leaving mock mode waited on a pass parked outside any evaluation")
+ monitor.metricsHistory.capacityForecastMu.Unlock()
+ parked = false
+ waitForSignal(t, passDone, 30*time.Second, "mock pass did not finish after it was released")
+
+ if active := manager.GetActiveAlerts(); len(active) != 0 {
+ t.Fatalf("a mock pass in flight when mock mode ended left %d fixture alerts in live mode: %s",
+ len(active), describeAlerts(active))
+ }
+ }
+
+ func TestLeavingMockModeWaitsForAnInFlightEvaluation(t *testing.T) {
+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+ pinDefaultMockEstate(t)
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{
+ state: models.NewState(),
+ alertManager: manager,
+ metricsHistory: NewMetricsHistory(10, time.Hour),
+ }
+ mustSetMonitorMockMode(t, monitor, true)
+
+ // Park a real pass inside its first guest evaluation, in the CPU window
+ // read the alert manager makes outside its own lock, so nothing but the
+ // fence holds the switch back.
+ entered, release := make(chan struct{}), make(chan struct{})
+ var once sync.Once
+ manager.SetMetricWindowProvider(func(alerts.MetricWindowRequest) ([]alerts.MetricWindowPoint, error) {
+ once.Do(func() {
+ close(entered)
+ <-release
+ })
+ return nil, nil
+ })
+ released := false
+ defer func() {
+ if !released {
+ close(release)
+ }
+ }()
+ passDone := make(chan struct{})
+ go func() {
+ defer close(passDone)
+ monitor.checkMockAlerts()
+ }()
+ waitForSignal(t, entered, 30*time.Second, "mock pass never evaluated a windowed metric")
+
+ // The switch must stop in advance until the evaluation returns. One
+ // that finishes first cleared alerts the evaluation can still raise.
+ leftMockMode := switchMockModeAsync(t, monitor, false)
+ waitForCondition(t, 30*time.Second, func() bool {
+ select {
+ case <-leftMockMode:
+ return true
+ default:
+ }
+ return goroutineIn("monitoring.(*mockModeFence).advance", "monitoring.(*Monitor).SetMockMode")
+ }, "leaving mock mode neither finished nor waited for the running evaluation")
+ select {
+ case <-leftMockMode:
+ t.Fatal("leaving mock mode cleared alerts while a fixture evaluation was still running")
+ default:
+ }
+
+ close(release)
+ released = true
+ waitForSignal(t, leftMockMode, 30*time.Second, "leaving mock mode did not finish once the evaluation returned")
+ waitForSignal(t, passDone, 30*time.Second, "mock pass did not finish after it was released")
+ if active := manager.GetActiveAlerts(); len(active) != 0 {
+ t.Fatalf("the evaluation running when mock mode ended left %d fixture alerts in live mode: %s",
+ len(active), describeAlerts(active))
+ }
+ }
+
+ // blockingFixtureRecords serves one source's fixture records the way the
+ // router's mock adapter does, reading them on each call, and holds its first
+ // call until released. Later calls are not held, not even while the first is.
+ type blockingFixtureRecords struct {
+ source unifiedresources.DataSource
+ entered chan struct{}
+ release chan struct{}
+ held atomic.Bool
+ }
+
+ func newBlockingFixtureRecords(source unifiedresources.DataSource) *blockingFixtureRecords {
+ return &blockingFixtureRecords{source: source, entered: make(chan struct{}), release: make(chan struct{})}
+ }
+
+ func (p *blockingFixtureRecords) SupplementalRecords(_ *Monitor, _ string) []unifiedresources.IngestRecord {
+ records := mock.SupplementalRecords(p.source)
+ if p.held.CompareAndSwap(false, true) {
+ close(p.entered)
+ <-p.release
+ }
+ return records
+ }
+
+ func TestLeavingMockModeDiscardsAnInFlightFrontendRefresh(t *testing.T) {
+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+ pinDefaultMockEstate(t)
+ manager := newMockHostAlertTestManager(t)
+ provider := newBlockingFixtureRecords(unifiedresources.SourceVMware)
+ monitor := &Monitor{
+ state: models.NewState(),
+ alertManager: manager,
+ metricsHistory: NewMetricsHistory(10, time.Hour),
+ resourceStore: unifiedresources.NewMonitorAdapter(nil),
+ supplementalProviders: map[unifiedresources.DataSource]MonitorSupplementalRecordsProvider{
+ unifiedresources.SourceVMware: provider,
+ },
+ }
+ mustSetMonitorMockMode(t, monitor, true)
+
+ // Hold a frontend read in its registry rebuild: it has the fixture
+ // snapshot and the vSphere fixture records, whose datastore, host and
+ // network incidents its unified alert sync would raise.
+ released := false
+ defer func() {
+ if !released {
+ close(provider.release)
+ }
+ }()
+ readDone := make(chan struct{})
+ go func() {
+ defer close(readDone)
+ monitor.BuildFrontendState()
+ }()
+ waitForSignal(t, provider.entered, 30*time.Second, "frontend read never reached the supplemental records")
+
+ waitForSignal(t, switchMockModeAsync(t, monitor, false), 30*time.Second, "leaving mock mode waited on a registry rebuild")
+ close(provider.release)
+ released = true
+ waitForSignal(t, readDone, 30*time.Second, "frontend read did not finish after it was released")
+
+ if active := manager.GetActiveAlerts(); len(active) != 0 {
+ t.Fatalf("a frontend read in flight when mock mode ended left %d fixture alerts in live mode: %s",
+ len(active), describeAlerts(active))
+ }
+ }
+
+ // gatedListStore holds the first registry listing made after open is called,
+ // so a test can publish another generation between a refresh's own publish
+ // and its read back.
+ type gatedListStore struct {
+ *unifiedresources.MonitorAdapter
+ mu sync.Mutex
+ armed bool
+ entered chan struct{}
+ release chan struct{}
+ }
+
+ func (s *gatedListStore) arm() {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ s.armed = true
+ }
+
+ func (s *gatedListStore) GetAll() []unifiedresources.Resource {
+ s.mu.Lock()
+ hold := s.armed
+ s.armed = false
+ s.mu.Unlock()
+ if hold {
+ close(s.entered)
+ <-s.release
+ }
+ return s.MonitorAdapter.GetAll()
+ }
+
+ // fixedRecords stands in for a live provider: it serves the same records in
+ // either mode.
+ type fixedRecords []unifiedresources.IngestRecord
+
+ func (r fixedRecords) SupplementalRecords(_ *Monitor, _ string) []unifiedresources.IngestRecord {
+ return r
+ }
+
+ func alertTypes(active []alerts.Alert) map[string]int {
+ types := make(map[string]int)
+ for _, alert := range active {
+ types[alert.Type]++
+ }
+ return types
+ }
+
+ func TestLiveRefreshIgnoresARegistryRepublishedFromMockMode(t *testing.T) {
+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+ pinDefaultMockEstate(t)
+ manager := newMockHostAlertTestManager(t)
+ mustSetMockEnabled(t, true)
+ // TrueNAS records stand in for a live provider's data; the vSphere
+ // fixture source serves nothing once mock mode is off.
+ liveTrueNAS := fixedRecords(mock.SupplementalRecords(unifiedresources.SourceTrueNAS))
+ mustSetMockEnabled(t, false)
+ if len(liveTrueNAS) == 0 {
+ t.Fatal("mock fixture has no TrueNAS records")
+ }
+ provider := newBlockingFixtureRecords(unifiedresources.SourceVMware)
+ store := &gatedListStore{
+ MonitorAdapter: unifiedresources.NewMonitorAdapter(nil),
+ entered: make(chan struct{}),
+ release: make(chan struct{}),
+ }
+ monitor := &Monitor{
+ state: models.NewState(),
+ alertManager: manager,
+ metricsHistory: NewMetricsHistory(10, time.Hour),
+ resourceStore: store,
+ supplementalProviders: map[unifiedresources.DataSource]MonitorSupplementalRecordsProvider{
+ unifiedresources.SourceVMware: provider,
+ unifiedresources.SourceTrueNAS: liveTrueNAS,
+ },
+ }
+ mustSetMonitorMockMode(t, monitor, true)
+ releasedProvider, releasedList := false, false
+ defer func() {
+ if !releasedProvider {
+ close(provider.release)
+ }
+ if !releasedList {
+ close(store.release)
+ }
+ }()
+
+ // A mock-mode refresh holds the fixture snapshot and vSphere records and
+ // has not published them yet.
+ staleDone := make(chan struct{})
+ go func() {
+ defer close(staleDone)
+ monitor.updateResourceStore(monitor.currentStateWithScope())
+ }()
+ waitForSignal(t, provider.entered, 30*time.Second, "mock-mode refresh never reached the supplemental records")
+ waitForSignal(t, switchMockModeAsync(t, monitor, false), 30*time.Second, "leaving mock mode waited on a registry rebuild")
+
+ // A live refresh publishes the live TrueNAS records alone and is held
+ // before reading the registry back.
+ store.arm()
+ liveDone := make(chan struct{})
+ go func() {
+ defer close(liveDone)
+ monitor.updateResourceStore(monitor.currentStateWithScope())
+ }()
+ waitForSignal(t, store.entered, 30*time.Second, "live refresh never read the registry back")
+ for _, resource := range store.MonitorAdapter.GetAll() {
+ if slices.Contains(resource.Sources, unifiedresources.SourceVMware) {
+ t.Fatalf("the live refresh published vSphere fixture resource %s", resource.ID)
+ }
+ }
+
+ // The mock-mode refresh now publishes the fixture estate over it.
+ close(provider.release)
+ releasedProvider = true
+ waitForSignal(t, staleDone, 30*time.Second, "mock-mode refresh did not finish after it was released")
+ close(store.release)
+ releasedList = true
+ waitForSignal(t, liveDone, 30*time.Second, "live refresh did not finish after it was released")
+
+ if active := manager.GetActiveAlerts(); len(active) != 0 {
+ t.Fatalf("a live refresh evaluated a registry the mock-mode refresh republished, leaving %d fixture alerts: %s",
+ len(active), describeAlerts(active))
+ }
+ if monitor.mockModeFence.registryCurrent() {
+ t.Fatal("the registry the mock-mode refresh republished counted as current")
+ }
+
+ // The next live refresh is undisturbed: it replaces the registry and
+ // evaluates the live records.
+ monitor.updateResourceStore(monitor.currentStateWithScope())
+ if !monitor.mockModeFence.registryCurrent() {
+ t.Fatal("an undisturbed live rebuild did not make the registry current")
+ }
+ types := alertTypes(manager.GetActiveAlerts())
+ if types["zfs-pool-state"] == 0 {
+ t.Fatalf("the undisturbed live refresh raised no alert from the live TrueNAS records: %v", types)
+ }
+ for _, fixtureType := range []string{"resource-incident", "storage-incident"} {
+ if types[fixtureType] != 0 {
+ t.Fatalf("vSphere fixture incidents reached live mode: %v", types)
+ }
+ }
+ }
+
+ func TestDockerPruneIgnoresTheRegistryMockModeLeft(t *testing.T) {
+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+ pinDefaultMockEstate(t)
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{
+ state: models.NewState(),
+ alertManager: manager,
+ metricsHistory: NewMetricsHistory(10, time.Hour),
+ resourceStore: unifiedresources.NewMonitorAdapter(nil),
+ }
+ mustSetMonitorMockMode(t, monitor, true)
+ monitor.updateResourceStore(monitor.currentStateWithScope())
+ mustSetMonitorMockMode(t, monitor, false)
+
+ // Before any live rebuild replaces the fixture registry, a live Docker
+ // host reports and then goes offline.
+ host := models.DockerHost{ID: "live-docker-1", Hostname: "live-docker-1", DisplayName: "live-docker-1", Status: "offline"}
+ monitor.state.UpsertDockerHost(host)
+ for range 3 {
+ manager.HandleDockerHostOffline(host)
+ }
+ if types := alertTypes(manager.GetActiveAlerts()); types["docker-host-offline"] != 1 {
+ t.Fatalf("live Docker host raised no offline alert: %v", types)
+ }
+
+ monitor.SyncAlertState()
+ if types := alertTypes(manager.GetActiveAlerts()); types["docker-host-offline"] != 1 {
+ t.Fatalf("Docker pruning removed a live host's alert against the fixture registry: %v", types)
+ }
+ }
+
+ func TestPreviousStateCarryIgnoresTheRegistryMockModeLeft(t *testing.T) {
+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+ pinDefaultMockEstate(t)
+ monitor := &Monitor{
+ state: models.NewState(),
+ alertManager: newMockHostAlertTestManager(t),
+ metricsHistory: NewMetricsHistory(10, time.Hour),
+ resourceStore: unifiedresources.NewMonitorAdapter(nil),
+ }
+ mustSetMonitorMockMode(t, monitor, true)
+ monitor.updateResourceStore(monitor.currentStateWithScope())
+ fixture := mock.CurrentFixtureGraph().State
+ if len(fixture.Nodes) == 0 || len(fixture.VMs) == 0 {
+ t.Fatal("mock fixture has no nodes or VMs")
+ }
+ instance := fixture.VMs[0].Instance
+ mustSetMonitorMockMode(t, monitor, false)
+
+ // A live instance named like a fixture one polls before any live rebuild
+ // replaces the fixture registry. Its previous-state carry, which writes
+ // back into live state, must not see the fixture nodes and guests.
+ if nodes := monitor.previousNodesForInstance(instance); len(nodes) != 0 {
+ t.Fatalf("previous-node carry for %q returned %d fixture nodes after leaving mock mode", instance, len(nodes))
+ }
+ if guests := monitor.previousGuestContextForInstance(instance); len(guests.vms) != 0 || len(guests.containers) != 0 {
+ t.Fatalf("previous-guest carry for %q returned %d fixture VMs and %d containers after leaving mock mode",
+ instance, len(guests.vms), len(guests.containers))
+ }
+ }
+
+ func TestLeavingMockModeDiscardsAnInFlightBackupEvaluation(t *testing.T) {
+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+ pinDefaultMockEstate(t)
+ manager := newMockHostAlertTestManager(t)
+ config := manager.GetConfig()
+ config.BackupDefaults.Enabled = true
+ config.BackupDefaults.WarningDays = 1
+ config.BackupDefaults.CriticalDays = 2
+ manager.UpdateConfig(config)
+ monitor := &Monitor{
+ state: models.NewState(),
+ alertManager: manager,
+ metricsHistory: NewMetricsHistory(10, time.Hour),
+ }
+ mustSetMonitorMockMode(t, monitor, true)
+ // Cache the fixture read view the guest lookup will be served from.
+ monitor.GetUnifiedReadStateOrSnapshot()
+
+ // Park a real evaluation at its guest lookup, after it read the fixture
+ // recovery rollups whose stale backups raise backup-age alerts.
+ monitor.mockUnifiedViewMu.Lock()
+ parked := true
+ defer func() {
+ if parked {
+ monitor.mockUnifiedViewMu.Unlock()
+ }
+ }()
+ evalDone := make(chan struct{})
+ go func() {
+ defer close(evalDone)
+ monitor.checkBackupAlerts(context.Background())
+ }()
+ waitForGoroutineIn(t, "monitoring.(*Monitor).currentUnifiedStateView", "monitoring.(*Monitor).checkBackupAlerts")
+
+ waitForSignal(t, switchMockModeAsync(t, monitor, false), 30*time.Second, "leaving mock mode waited on a backup evaluation parked outside the alert manager")
+ monitor.mockUnifiedViewMu.Unlock()
+ parked = false
+ waitForSignal(t, evalDone, 30*time.Second, "backup evaluation did not finish after it was released")
+
+ if active := manager.GetActiveAlerts(); len(active) != 0 {
+ t.Fatalf("a backup evaluation in flight when mock mode ended left %d fixture alerts in live mode: %.500s",
+ len(active), describeAlerts(active))
+ }
+ }
+
+ func TestLeavingMockModeDiscardsAnInFlightConnectionCheck(t *testing.T) {
+ pinDefaultMockEstate(t)
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{
+ state: models.NewState(),
+ alertManager: manager,
+ metricsHistory: NewMetricsHistory(10, time.Hour),
+ }
+ mustSetMonitorMockMode(t, monitor, true)
+
+ // A hybrid mock ledger lists real connections beside fixture ones, so the
+ // same connection can be read degraded in mock mode and again live.
+ degraded := alerts.ConnectionSnapshot{
+ ID: "pve-cluster-a",
+ Name: "cluster-a",
+ Type: alerts.ConnectionTypePVE,
+ State: alerts.ConnectionStateUnreachable,
+ Enabled: true,
+ }
+ entered, release := make(chan struct{}), make(chan struct{})
+ var once sync.Once
+ monitor.SetConnectionsSnapshotLister(func() []alerts.ConnectionSnapshot {
+ once.Do(func() {
+ close(entered)
+ <-release
+ })
+ return []alerts.ConnectionSnapshot{degraded}
+ })
+ released := false
+ defer func() {
+ if !released {
+ close(release)
+ }
+ }()
+ checkDone := make(chan struct{})
+ go func() {
+ defer close(checkDone)
+ monitor.checkConnectionAlerts()
+ }()
+ waitForSignal(t, entered, 30*time.Second, "connection check never read the ledger")
+
+ waitForSignal(t, switchMockModeAsync(t, monitor, false), 30*time.Second, "leaving mock mode waited on a connection check parked in its ledger read")
+ close(release)
+ released = true
+ waitForSignal(t, checkDone, 30*time.Second, "connection check did not finish after it was released")
+
+ // The connection-degraded alert confirms on its third consecutive
+ // observation; the one read in mock mode must not be one of them.
+ monitor.checkConnectionAlerts()
+ monitor.checkConnectionAlerts()
+ if active := manager.GetActiveAlerts(); len(active) != 0 {
+ t.Fatalf("a connection observation read in mock mode counted toward live confirmation: %s", describeAlerts(active))
+ }
+ monitor.checkConnectionAlerts()
+ if active := manager.GetActiveAlerts(); len(active) != 1 {
+ t.Fatalf("three live degraded observations raised %d alerts, want 1: %s", len(active), describeAlerts(active))
+ }
+ }
+
// diskAlertKeys lists the active disk-health and disk-wearout alerts as
// "type instance/node device" so assertions name the disk, not the alert ID.
func diskAlertKeys(manager *alerts.Manager) map[string]bool {
diff --cc internal/truenas/provider.go
index cc12fd63b2,a023688f6c..bac1127425
--- a/internal/truenas/provider.go
+++ b/internal/truenas/provider.go
@@@ -620,10 -605,15 +621,15 @@@ func truenasRecordsFromSnapshot(snapsho
LastSeen: collectedAt,
UpdatedAt: collectedAt,
Metrics: &unifiedresources.ResourceMetrics{
- Disk: diskMetric(pool.TotalBytes, pool.UsedBytes),
+ Disk: poolDiskMetric(pool),
},
+ // TrueNAS has no disabled state for an imported pool, so
+ // Enabled is always true; Active follows whether the pool is
+ // usable now.
Storage: &unifiedresources.StorageMeta{
Type: "zfs-pool",
+ Enabled: true,
+ Active: poolStatus != unifiedresources.StatusOffline,
IsZFS: true,
Platform: "truenas",
Topology: "pool",