Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Compose reviewed monitoring repairs with current upstream without losing either fence
pulse-triage[bot]0840845673fc
diff --cc docs/release-control/v6/internal/subsystems/monitoring.mdindex 0817bbccc2,3c011447f5..f5dae19921--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@@ -1883,23 -1758,19 +1927,25 @@@ boundariesreport and would mark the host online.- Agents are evaluated before nodes, so a node with an online linked agenthands its CPU, memory and disk alerts to the agent on the first tick, as- host-agent hostname deduplication does in production.+ node-link deduplication does in production.- A runtime mock config change rebuilds the estate. An agent that leaves itgoes through `HandleHostRemoved`, as a deleted live agent does, so its alerts- and hostname deduplication do not outlive it.+ and node link do not outlive it.- Leaving mock mode routes the fixture agents through `HandleHostRemoved`.- `ClearActiveAlerts` drops their alerts but not their hostname registrations,- and a real node named like a fixture agent (`pve1`) would otherwise keep its- metric alerts suppressed with no agent to own them. `SetMockMode` ends the- mock-mode epoch before it forgets them, so a pass whose snapshot predates the- switch finds its epoch ended under the same lock and evaluates nothing, and- an agent whose departure the switch refused stays in the forgotten set. Two- overlapping passes in one epoch can still evaluate an older snapshot after a- newer one; the next pass reconciles the set again.+ `ClearActiveAlerts` drops their alerts but not their node links, and a+ leftover link would keep suppressing CPU, memory and disk alerts on any node- that later carries the linked fixture node's ID. The agent and Docker- steps of a pass whose snapshot predates the switch find mock mode off under- the same lock and evaluate nothing.++ that later carries the linked fixture node's ID. `SetMockMode` ends the++ mock-mode epoch before forgetting fixture hosts, so evaluations holding an++ older epoch cannot recreate alerts after the clear. Departures refused by++ the epoch fence remain tracked for the switch to remove.+- Ticks start passes concurrently, so a pass whose snapshot predates an- estate rebuild can reach the lock after a newer one. Each pass carries the- fixture's structural revision, and the agent and Docker steps of a pass- older than the last applied or disabled revision are skipped, so they- cannot remove the new estate's hosts or re-evaluate retired ones, even- across a disable and re-enable. The pass's other loops (guests, nodes,- storage, PBS, PMG) take no part in this tracking and are not guarded.++ estate rebuild can reach the lock after a newer one. The agent and Docker++ steps also carry the fixture's structural revision and reject revisions++ older than the last applied or disabled one. Both checks apply: the epoch++ stops mode-crossing evaluations, while the revision stops an older estate++ replacing newer host ownership within one epoch. Guest, node, storage, PBS,++ PMG and physical-disk evaluations use the epoch fence, not host revision++ tracking.Fixture agents must carry readings a real agent could report. Mock Kubernetespods share 0.7 single-pod memory footprints per node, which keeps a node's podsdiff --cc internal/alerts/threshold_resolution_shared_test.goindex 93911cba42,cf040367e4..314a235fad--- a/internal/alerts/threshold_resolution_shared_test.go+++ b/internal/alerts/threshold_resolution_shared_test.go@@@ -1451,410 -1446,147 +1451,555 @@@ func TestDiskTemperatureThresholdMatche}}+// An agent owns a usage metric only with usable evidence: it evaluated the+// metric this report, or it still holds an alert or run for it. Configuration+// alone is not coverage.+func TestCheckNodeKeepsMetricsTheAgentCannotEvaluate(t *testing.T) {+ setup := func(t *testing.T) (*Manager, models.Node, models.Host) {+ m := newTestManager(t)+ m.mu.Lock()+ m.config.Enabled = true+ m.config.TimeThresholds = map[string]int{}+ m.config.NodeDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}+ m.config.NodeDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}+ m.config.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}+ m.config.AgentDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}+ m.mu.Unlock()+ node, host := testNodeWithHostAgent()+ node.CPU = 0.95+ node.Memory = models.Memory{Total: 100, Used: 95, Free: 5, Usage: 95}+ return m, node, host+ }++ t.Run("memory_unknown_from_the_start", func(t *testing.T) {+ m, node, host := setup(t)+ host.Memory = models.Memory{Total: 100, UsageUnavailable: true}+ m.CheckHost(host)+ m.CheckNode(node)+ if !testHasActiveAlert(t, m, canonicalMetricStateID(node.ID, "memory")) {+ t.Fatalf("expected the node to alert on memory while the agent has no memory reading")+ }+ })++ t.Run("non_finite_cpu", func(t *testing.T) {+ m, node, host := setup(t)+ host.CPUUsage = math.NaN()+ m.CheckHost(host)+ m.CheckNode(node)+ if !testHasActiveAlert(t, m, canonicalMetricStateID(node.ID, "cpu")) {+ t.Fatalf("expected the node to alert on CPU while the agent reports no usable CPU value")+ }+ })+}++// Turning an agent threshold off releases its pending run even when the agent+// reports no reading for that metric, so the old run cannot keep ownership and+// leave the node silent.+func TestDisabledAgentThresholdReleasesPendingRunWithoutReading(t *testing.T) {+ m := newTestManager(t)+ m.mu.Lock()+ m.config.Enabled = true+ m.config.TimeThresholds = map[string]int{}+ m.config.MetricTimeThresholds = map[string]map[string]int{"agent": {"memory": 300}}+ m.config.NodeDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}+ m.config.AgentDefaults.Memory = &HysteresisThreshold{Trigger: 85, Clear: 80}+ m.mu.Unlock()+ node, host := testNodeWithHostAgent()+ // 90% stays below the 95% critical level, which would bypass the delay.+ node.Memory = models.Memory{Total: 100, Used: 90, Free: 10, Usage: 90}+ host.Memory = node.Memory+ agentResourceID := hostResourceID(host.ID)+ agentMemorySpec := canonicalMetricSpecID(agentResourceID, "memory")++ m.CheckHost(host)+ m.mu.RLock()+ pending := testCoreIsPending(m, agentResourceID, agentMemorySpec)+ m.mu.RUnlock()+ if !pending {+ t.Fatalf("expected a pending agent memory run")+ }++ m.mu.Lock()+ m.config.AgentDefaults.Memory = &HysteresisThreshold{Trigger: 0, Clear: 0}+ m.mu.Unlock()+ host.Memory = models.Memory{Total: 100, UsageUnavailable: true}+ m.CheckHost(host)+ m.CheckNode(node)++ m.mu.RLock()+ agentIncident := testCoreHasIncident(m, agentResourceID, agentMemorySpec)+ nodeIncident := testCoreHasIncident(m, node.ID, canonicalMetricSpecID(node.ID, "memory"))+ m.mu.RUnlock()+ if agentIncident {+ t.Fatalf("expected the disabled agent memory threshold to release its pending run")+ }+ if !nodeIncident {+ t.Fatalf("expected the node to evaluate memory once the agent no longer covers it")+ }+}++// Ownership follows the config at once and link updates apply in report order:+// a config change revokes what it no longer supports without waiting for the+// agent's next report, an older report can neither restore revoked ownership+// nor remove a newer report's link, and a report started before the agent went+// offline is stale.+func TestHostAgentNodeLinkFollowsConfigAndReportOrder(t *testing.T) {+ m := newTestManager(t)+ m.mu.Lock()+ m.config.Enabled = true+ m.config.TimeThresholds = map[string]int{}+ m.config.NodeDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}+ m.config.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}+ m.mu.Unlock()+ node, host := testNodeWithHostAgent()+ node.CPU = 0.95+ host.CPUUsage = 50++ m.CheckHost(host)+ m.CheckNode(node)+ nodeCPUSpec := canonicalMetricSpecID(node.ID, "cpu")+ m.mu.RLock()+ evaluatedBefore := testCoreHasIncident(m, node.ID, nodeCPUSpec)+ m.mu.RUnlock()+ if evaluatedBefore {+ t.Fatalf("expected the agent to own CPU before the config change")+ }++ // A report already in flight when agent alerts are disabled must not+ // restore ownership once it lands.+ inFlight := m.beginHostAgentReport(host.ID)++ // No agent report follows the config change; the node must evaluate CPU on+ // its very next check. (UpdateConfig restores the default activation delay,+ // so the run is pending rather than firing.)+ config := m.GetConfig()+ config.DisableAllAgents = true+ m.UpdateConfig(config)+ m.CheckNode(node)+ m.mu.RLock()+ evaluatedAfter := testCoreHasIncident(m, node.ID, nodeCPUSpec)+ m.mu.RUnlock()+ if !evaluatedAfter {+ t.Fatalf("expected the node to evaluate CPU right after agent alerts were disabled")+ }++ covering := hostAgentNodeLink{agentID: host.ID, nodeID: node.ID, cpu: true}+ m.applyHostAgentNodeLink(covering, inFlight)+ if m.hasHostAgentForNode(node.ID) {+ t.Fatalf("expected a report evaluated before agent alerts were disabled not to restore ownership")+ }++ config.DisableAllAgents = false+ m.UpdateConfig(config)+ older := m.beginHostAgentReport(host.ID)+ newer := m.beginHostAgentReport(host.ID)+ m.applyHostAgentNodeLink(covering, newer)+ m.applyHostAgentNodeLink(hostAgentNodeLink{agentID: host.ID, nodeID: node.ID}, older)+ if !m.hasHostAgentForNode(node.ID) {+ t.Fatalf("expected an older report not to remove the newer report's link")+ }++ olderCovering := m.beginHostAgentReport(host.ID)+ newerRemoving := m.beginHostAgentReport(host.ID)+ m.applyHostAgentNodeLink(hostAgentNodeLink{agentID: host.ID, nodeID: node.ID}, newerRemoving)+ m.applyHostAgentNodeLink(covering, olderCovering)+ if m.hasHostAgentForNode(node.ID) {+ t.Fatalf("expected an older report not to restore ownership a newer report removed")+ }++ beforeOffline := m.beginHostAgentReport(host.ID)+ m.HandleHostOffline(host)+ m.applyHostAgentNodeLink(covering, beforeOffline)+ if m.hasHostAgentForNode(node.ID) {+ t.Fatalf("expected a report started before the agent went offline not to re-register")+ }+}++// An agent that reports unlinked hands its former node the usage metrics back+// on the node's next check, even while the agent's own alert is still firing.+func TestCheckNodeTakesBackMetricsWhenAgentUnlinks(t *testing.T) {+ m := newTestManager(t)+ m.mu.Lock()+ m.config.Enabled = true+ m.config.TimeThresholds = map[string]int{}+ m.config.NodeDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}+ m.config.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 80, Clear: 75}+ m.mu.Unlock()+ node, host := testNodeWithHostAgent()+ node.CPU = 0.95+ host.CPUUsage = 95++ m.CheckHost(host)+ m.CheckNode(node)+ nodeCPU := canonicalMetricStateID(node.ID, "cpu")+ if testHasActiveAlert(t, m, nodeCPU) {+ t.Fatalf("expected the linked agent to own CPU")+ }++ host.LinkedNodeID = ""+ m.CheckHost(host)+ m.CheckNode(node)+ if !testHasActiveAlert(t, m, nodeCPU) {+ t.Fatalf("expected the node to alert on CPU once its agent reports unlinked")+ }+}++// An explicit disk override, including one inherited from the linked node,+// beats the per-type DiskFillByType default; otherwise the node released its+// disk alert to an agent applying a looser per-type threshold.+func TestHostDiskOverrideBeatsDiskFillByType(t *testing.T) {+ for _, overrideOn := range []string{"linked node", "agent"} {+ t.Run(overrideOn, func(t *testing.T) {+ m := newTestManager(t)+ node, host := testNodeWithHostAgent()+ overrideID := node.ID+ if overrideOn == "agent" {+ overrideID = host.ID+ }+ m.mu.Lock()+ m.config.Enabled = true+ m.config.TimeThresholds = map[string]int{}+ m.config.AgentDefaults.Disk = &HysteresisThreshold{Trigger: 90, Clear: 85}+ m.config.DiskFillByType = map[string]HysteresisThreshold{"nvme": {Trigger: 92, Clear: 88}}+ m.config.Overrides = map[string]ThresholdConfig{+ overrideID: {Disk: &HysteresisThreshold{Trigger: 80, Clear: 75}},+ }+ m.mu.Unlock()+ root := models.Disk{Mountpoint: "/", Device: "/dev/nvme0n1p2", Total: 100, Used: 85, Free: 15, Usage: 85}+ host.Disks = []models.Disk{root}++ m.CheckHost(host)+ rootResourceID, _ := hostDiskResourceID(host, root)+ if !testHasActiveAlert(t, m, canonicalMetricStateID(rootResourceID, "disk")) {+ t.Fatalf("expected the 80%% %s disk override to raise the agent's root disk alert at 85%%", overrideOn)+ }+ })+ }+}++// Releasing a metric drops its explicit intent grace run too, so a later+// resume does not activate at once on time counted while nothing evaluated.+func TestReleaseCanonicalMetricAlertClearsIntentPending(t *testing.T) {+ m := newTestManager(t)+ spec, err := buildCanonicalMetricSpec("homelab-delly2", "delly2", "node", "cpu", nil)+ if err != nil {+ t.Fatalf("spec: %v", err)+ }+ trackingKey := canonicalTrackingKeyForSpec(spec, spec.ID)+ m.mu.Lock()+ m.intentPending[trackingKey] = IntentPendingState{+ TrackingKey: trackingKey,+ ResourceID: spec.ResourceID,+ Signal: MetricAlertIntentSignal("cpu"),+ FirstMatchedAt: time.Now().Add(-10 * time.Minute),+ LastObservedAt: time.Now().Add(-9 * time.Minute),+ }+ m.mu.Unlock()++ m.releaseCanonicalMetricAlert(spec, "delly2", "delly2", "homelab", "node", 0, nil)++ m.mu.RLock()+ _, stillPending := m.intentPending[trackingKey]+ m.mu.RUnlock()+ if stillPending {+ t.Fatalf("expected releasing the metric to clear its intent pending state")+ }+}++// Missing storage observations break confirmation runs, not the occurrence.+// Exercise the public evaluator and both live/shadow reducers, not seeded state.+func TestStorageConnectivityObservationGaps(t *testing.T) {+ for _, gap := range []string{"", "unknown", " UNKNOWN "} {+ t.Run("activation/"+gap, func(t *testing.T) {+ m := newShadowFeedManager(t)+ s := models.Storage{ID: "storage-a", Name: "backups", Status: "offline"}+ other := s+ other.ID = "storage-ab"+ id := canonicalConnectivityStateID(s.ID)+ m.CheckStorage(s)+ m.CheckStorage(other)+ s.Status = gap+ m.CheckStorage(s)+ s.Status = "unavailable"+ m.CheckStorage(s)+ if testHasActiveAlert(t, m, id) {+ t.Fatal("non-consecutive outage observations fired storage alert")+ }+ m.CheckStorage(other)+ testRequireActiveAlert(t, m, canonicalConnectivityStateID(other.ID))+ m.CheckStorage(s)+ testRequireActiveAlert(t, m, id)+ if m.ShadowDivergences() != 0 {+ t.Fatal("storage interruption diverged from shadow")+ }+ })+ t.Run("recovery/"+gap, func(t *testing.T) {+ m := newShadowFeedManager(t)+ s := models.Storage{ID: "storage-a", Name: "backups", Status: "offline"}+ id := canonicalConnectivityStateID(s.ID)+ for range 2 {+ m.CheckStorage(s)+ }+ if err := m.AcknowledgeAlert(id, "operator"); err != nil {+ t.Fatal(err)+ }+ s.Status = "available"+ m.CheckStorage(s)+ before := testRequireActiveAlert(t, m, id).Clone()+ s.Status = gap+ m.CheckStorage(s)+ if !reflect.DeepEqual(before, testRequireActiveAlert(t, m, id).Clone()) || m.GetResolvedAlert(id) != nil {+ t.Fatal("missing connectivity changed acknowledged occurrence")+ }+ s.Status = "available"+ m.CheckStorage(s)+ if !testHasActiveAlert(t, m, id) {+ t.Fatal("non-consecutive healthy observations resolved storage outage")+ }+ m.CheckStorage(s)+ if testHasActiveAlert(t, m, id) {+ t.Fatal("fresh consecutive healthy observations failed to recover")+ }+ resolved := m.GetResolvedAlert(id)+ if resolved == nil || !resolved.StartTime.Equal(before.StartTime) {+ t.Fatal("confirmed recovery lost original occurrence")+ }+ if m.ShadowDivergences() != 0 {+ t.Fatal("storage recovery diverged from shadow")+ }+ })+ }+}++func TestStorageConnectivityGapRestartsIntentGrace(t *testing.T) {+ m := newShadowFeedManager(t)+ var tick time.Duration+ m.intentClock = func() time.Duration { return tick }+ doc := NewAlertIntentPolicyDocument()+ doc.Defaults[string(AlertIntentSignalOffline)] = AlertIntentRule{GraceSeconds: intPointer(60)}+ if err := m.LoadIntentPolicies(doc); err != nil {+ t.Fatal(err)+ }+ s := models.Storage{ID: "storage-a", Name: "backups", Status: "offline"}+ id := canonicalConnectivityStateID(s.ID)+ m.CheckStorage(s)+ tick = 40 * time.Second+ m.CheckStorage(s)+ s.Status = "unknown"+ m.CheckStorage(s)+ m.mu.RLock()+ _, pending := m.intentPending[id]+ _, ticking := m.intentRuntimeTicks[id]+ m.mu.RUnlock()+ if pending || ticking {+ t.Error("missing status retained intent grace bookkeeping")+ }+ if err := m.SaveActiveAlerts(); err != nil {+ t.Fatal(err)+ }+ data, err := os.ReadFile(filepath.Join(m.getAlertsDir(), intentPendingFileName))+ if err != nil {+ t.Fatal(err)+ }+ var saved []IntentPendingState+ if err := json.Unmarshal(data, &saved); err != nil || len(saved) != 0 {+ t.Fatalf("interrupted intent remained checkpointed: %s, %v", data, err)+ }+ s.Status = "offline"+ for _, next := range []time.Duration{70, 90, 129} {+ tick = next * time.Second+ m.CheckStorage(s)+ if testHasActiveAlert(t, m, id) {+ t.Fatal("intent grace accrued across missing storage status")+ }+ }+ tick = 130 * time.Second+ m.CheckStorage(s)+ testRequireActiveAlert(t, m, id)+ if m.ShadowDivergences() != 0 {+ t.Fatal("intent interruption diverged from shadow")+ }+}++// Usage has delayed activation but immediate measured recovery. Do not invent+// a recovery stability window to fit the memory/temperature timing fixtures.+func TestStorageCapacityGapHoldsOccurrence(t *testing.T) {+ for _, gap := range []string{"missing", "unconfirmed-zero", "negative", "offline", "unavailable"} {+ t.Run(gap, func(t *testing.T) {+ m, elapsed := continuityManager(t, false)+ id, metric, observe := continuityObserver(t, m, "storage", gap)+ alertID := canonicalMetricStateID(id, metric)+ observe(95, false)+ elapsed.Store(int64(time.Minute))+ observe(95, false)+ if err := m.AcknowledgeAlert(alertID, "operator"); err != nil {+ t.Fatal(err)+ }+ before := testRequireActiveAlert(t, m, alertID).Clone()+ elapsed.Store(int64(2 * time.Minute))+ observe(10, true)+ if !reflect.DeepEqual(before, testRequireActiveAlert(t, m, alertID).Clone()) || m.GetResolvedAlert(alertID) != nil {+ t.Fatal("missing capacity changed the acknowledged occurrence")+ }+ // A real, independently observed empty capacity can clear even+ // while connectivity is unknown; neither channel stands in for the other.+ empty := models.Storage{ID: id, Name: "backups", Status: "unknown", Total: 1000, Free: 1000}+ m.CheckStorage(empty)+ if testHasActiveAlert(t, m, alertID) {+ t.Fatal("confirmed empty capacity did not recover immediately")+ }+ resolved := m.GetResolvedAlert(alertID)+ if resolved == nil || !resolved.StartTime.Equal(before.StartTime) || resolved.Value != 0 {+ t.Fatal("measured recovery lost the original occurrence or clearing value")+ }+ if m.ShadowDivergences() != 0 {+ t.Fatal("capacity interruption diverged from shadow")+ }+ })+ }+}+++ // HostDiskTemperatureThreshold is the disk heat policy for a disk a host agent+ // reports, so it must agree with CheckHost on every host-level override: an+ // explicit Disk Temp override on the host (or the canonical resource its ID+ // resolves to), or inherited from its linked node or guest, replaces the+ // per-type entry; the first override in the chain decides even when it sets no+ // Disk Temp; and a host whose alerts are off judges no heat.+ func TestHostDiskTemperatureThresholdMatchesCheckHostOverrides(t *testing.T) {+ const hostID = "host-temp-policy"+ const nodeID = "pve1-node1"+ const vmID = "pve1-node1-105"+ const canonicalID = "agent-canonical-temp-policy"+ perType := &HysteresisThreshold{Trigger: 70, Clear: 65}++ for _, tc := range []struct {+ name string+ overrides map[string]ThresholdConfig+ linkNode bool+ linkVM bool+ canonical bool+ want *HysteresisThreshold+ off bool+ }{+ {name: "no override", want: perType},+ {+ name: "host disk temp override",+ overrides: map[string]ThresholdConfig{hostID: {DiskTemperature: &HysteresisThreshold{Trigger: 80, Clear: 75}}},+ want: &HysteresisThreshold{Trigger: 80, Clear: 75},+ },+ {+ name: "host override without disk temp",+ overrides: map[string]ThresholdConfig{hostID: {Memory: &HysteresisThreshold{Trigger: 90, Clear: 85}}},+ want: perType,+ },+ {+ name: "linked node disk temp override",+ overrides: map[string]ThresholdConfig{nodeID: {DiskTemperature: &HysteresisThreshold{Trigger: 78, Clear: 72}}},+ linkNode: true,+ want: &HysteresisThreshold{Trigger: 78, Clear: 72},+ },+ {+ name: "host override shadows linked node",+ overrides: map[string]ThresholdConfig{+ hostID: {Memory: &HysteresisThreshold{Trigger: 90, Clear: 85}},+ nodeID: {DiskTemperature: &HysteresisThreshold{Trigger: 78, Clear: 72}},+ },+ linkNode: true,+ want: perType,+ },+ {+ // The agent ID resolves to the machine's canonical resource ID.+ name: "canonical machine disk temp override",+ overrides: map[string]ThresholdConfig{canonicalID: {DiskTemperature: &HysteresisThreshold{Trigger: 82, Clear: 77}}},+ canonical: true,+ want: &HysteresisThreshold{Trigger: 82, Clear: 77},+ },+ {+ name: "linked guest disk temp override",+ overrides: map[string]ThresholdConfig{vmID: {DiskTemperature: &HysteresisThreshold{Trigger: 76, Clear: 71}}},+ linkVM: true,+ want: &HysteresisThreshold{Trigger: 76, Clear: 71},+ },+ {+ name: "linked guest alerts disabled",+ overrides: map[string]ThresholdConfig{vmID: {Disabled: true}},+ linkVM: true,+ },+ {+ name: "host disk temp switched off",+ overrides: map[string]ThresholdConfig{hostID: {DiskTemperature: &HysteresisThreshold{}}},+ off: true,+ },+ {+ name: "host alerts disabled",+ overrides: map[string]ThresholdConfig{hostID: {Disabled: true}},+ },+ {+ name: "linked node alerts disabled",+ overrides: map[string]ThresholdConfig{nodeID: {Disabled: true}},+ linkNode: true,+ },+ } {+ t.Run(tc.name, func(t *testing.T) {+ m := configureDiskTempTypeHostManager(t)+ m.mu.Lock()+ for id, override := range tc.overrides {+ m.config.Overrides[id] = override+ }+ m.mu.Unlock()+ if tc.canonical {+ m.SetResourceIntentIdentityResolver(func(resourceID string) (string, bool) {+ return canonicalID, resourceID == hostID+ })+ }++ owner := DiskTemperatureHost{ID: hostID}+ if tc.linkNode {+ owner.LinkedNodeID = nodeID+ }+ if tc.linkVM {+ owner.LinkedVMID = vmID+ }+ got := m.HostDiskTemperatureThreshold(owner, "nvme")+ switch {+ case tc.off:+ if got == nil || got.Trigger > 0 {+ t.Fatalf("HostDiskTemperatureThreshold = %+v, want a switched-off threshold", got)+ }+ case (got == nil) != (tc.want == nil) || (got != nil && *got != *tc.want):+ t.Fatalf("HostDiskTemperatureThreshold = %+v, want %+v", got, tc.want)+ }++ check := func(temperature int) bool {+ host := hostWithSMARTDiskTemp(hostID, "nvme", temperature)+ host.LinkedNodeID = owner.LinkedNodeID+ host.LinkedVMID = owner.LinkedVMID+ m.CheckHost(host)+ _, exists := testLookupActiveAlert(t, m, hostDiskTempAlertID(host))+ return exists+ }+ if got == nil || got.Trigger <= 0 {+ if check(99) {+ t.Fatalf("CheckHost alerted at 99C on a host the policy judges no heat for, active: %v", alertKeys(m))+ }+ return+ }+ trigger := int(got.Trigger)+ if check(trigger - 1) {+ t.Fatalf("CheckHost alerted at %dC, under the policy trigger %v", trigger-1, got.Trigger)+ }+ if !check(trigger) {+ t.Fatalf("CheckHost stayed quiet at the policy trigger %dC, active: %v", trigger, alertKeys(m))+ }+ })+ }++ m := configureDiskTempTypeHostManager(t)+ if got, want := m.HostDiskTemperatureThreshold(DiskTemperatureHost{}, "sas"), m.DiskTemperatureThreshold("sas"); got == nil || want == nil || *got != *want {+ t.Fatalf("hostless policy = %+v, want DiskTemperatureThreshold %+v", got, want)+ }+ var nilManager *Manager+ if got := nilManager.HostDiskTemperatureThreshold(DiskTemperatureHost{ID: hostID}, "nvme"); got == nil || got.Trigger != 70 {+ t.Fatalf("nil manager host policy = %+v, want the factory nvme 70", got)+ }+ }diff --cc internal/monitoring/monitor.goindex 8bc8a5519f,25be288f63..176e1c6952--- a/internal/monitoring/monitor.go+++ b/internal/monitoring/monitor.go@@@ -1282,9 -1330,8 +1331,10 @@@ type Monitor structmockMetricsCancel context.CancelFuncmockMetricsWg sync.WaitGroupmockHostAgentsMu sync.Mutex+ mockDockerHosts map[string]models.DockerHost+ mockFixtureRevision uint64mockHostAgents map[string]models.Host // Fixture agents evaluated by the last mock alert pass+ mockModeFence mockModeFence // Keeps mode-dependent alert evaluations inside the epoch they read indockerChecker DockerChecker // Optional Docker checker for LXC containersdockerCheckerConfiguredAt time.Time // Last time the Docker checker was configureddockerCheckAllowedVMIDs map[int]struct{} // Optional VMID allowlist gating the LXC Docker socket probe; empty means all guests@@@ -4616,8 -4671,9 +4680,9 @@@ func (m *Monitor) SetMockMode(enable boif err := mock.SetEnabled(false); err != nil {return err}+ m.mockModeFence.advance()m.alertManager.ClearActiveAlerts()- m.forgetMockHostAgents()+ m.forgetMockFixtureHosts()m.mu.Lock()m.resetStateLocked()m.metricsHistory.Reset()diff --cc internal/monitoring/monitor_alerts.goindex 00dc149c28,afb150ad62..d8f3f956d8--- a/internal/monitoring/monitor_alerts.go+++ b/internal/monitoring/monitor_alerts.go@@@ -697,10 -702,10 +703,10 @@@ func (m *Monitor) checkMockAlerts()// the monitor state the host health sweep reads, so without this pass no// agent CPU, memory, disk, temperature or offline alert can open against// mock data. Agents go before nodes so a linked node hands its CPU, memory- // and disk alerts to its agent on the first tick, as the host-agent- // hostname deduplication does in production.+ // and disk alerts to its agent on the first tick, as node-link+ // deduplication does in production.log.Debug().Int("hostCount", len(state.Hosts)).Msg("checking host agent alerts")- m.evaluateMockHostAgents(state.Hosts, state.Nodes, fixtureRevision)- m.evaluateMockHostAgents(scope, state.Hosts, state.Nodes)++ m.evaluateMockHostAgents(scope, state.Hosts, state.Nodes, fixtureRevision)// Check alerts for each nodefor _, node := range state.Nodes {@@@ -742,7 -748,9 +749,7 @@@// so skipping this loop leaves the docker alert lifecycle unexercisable// against mock data.log.Debug().Int("dockerHostCount", len(state.DockerHosts)).Msg("checking docker alerts")- m.evaluateMockDockerHosts(state.DockerHosts, fixtureRevision)- for _, dockerHost := range state.DockerHosts {- scope.run(func() { m.checkMockDockerHostAlerts(dockerHost) })- }++ m.evaluateMockDockerHosts(scope, state.DockerHosts, fixtureRevision)// Cache the latest alert snapshots directly in the mock data so the API can serve// mock state without needing to grab the alert manager lock again.@@@ -752,12 -762,16 +761,13 @@@// evaluateMockHostAgents evaluates every fixture agent and remembers the set.// A runtime mock config change rebuilds the estate, so an agent can leave it// between passes; it then goes through HandleHostRemoved, as a deleted live-// agent does, or its alerts and hostname deduplication would outlive it.-func (m *Monitor) evaluateMockHostAgents(scope mockModeScope, hosts []models.Host, nodes []models.Node) {+// agent does, or its alerts and node link would outlive it.- func (m *Monitor) evaluateMockHostAgents(hosts []models.Host, nodes []models.Node, fixtureRevision uint64) {++func (m *Monitor) evaluateMockHostAgents(scope mockModeScope, hosts []models.Host, nodes []models.Node, fixtureRevision uint64) {m.mockHostAgentsMu.Lock()defer m.mockHostAgentsMu.Unlock()- if !m.acceptMockFixturePassLocked(fixtureRevision) {- // A pass that took its snapshot before the monitor left mock mode must- // not re-register fixture agents after forgetMockHostAgents ran.- // SetMockMode ends the epoch before it forgets them, so either this check- // fails or forgetMockHostAgents waits for this pass to record its set.- if !scope.current() {++ // Both the mode epoch and structural revision must still match.++ if !scope.current() || !m.acceptMockFixturePassLocked(fixtureRevision) {return}@@@ -767,9 -781,16 +777,14 @@@current[host.ID] = host}}- // Remove first so a departed agent cannot unregister a hostname that a- // remaining agent registers on this same pass.for id, host := range m.mockHostAgents {- if _, ok := current[id]; !ok {- m.alertManager.HandleHostRemoved(host)+ if _, ok := current[id]; ok {+ continue+ }+ if !scope.run(func() { m.alertManager.HandleHostRemoved(host) }) {+ // Leaving mock mode refused the removal; keep the agent so- // forgetMockHostAgents still removes it.++ // forgetMockFixtureHosts still removes it.+ current[id] = host}}for _, host := range hosts {@@@ -778,69 -799,16 +793,73 @@@m.mockHostAgents = current}-// forgetMockHostAgents removes the fixture agents when the monitor leaves mock-// mode. ClearActiveAlerts drops their alerts but not their hostname-// deduplication, so a real node named like a fixture agent (pve1) would keep-// its CPU, memory and disk alerts suppressed with no agent to own them.-// SetMockMode ends the mock epoch before this runs, so the set it removes is-// final: no pass of that epoch can register an agent or evaluate one again.-func (m *Monitor) forgetMockHostAgents() {+// evaluateMockDockerHosts is the Docker counterpart of evaluateMockHostAgents,+// under the same lock and mode check. Nested Docker-in-LXC hosts are named+// after their guest's VMID, so a runtime mock config change can drop a host+// from the estate; it then goes through HandleDockerHostRemoved, as a deleted+// live host does. pruneStaleDockerAlerts clears the same alerts, but only when+// something reads state, so an unwatched stack would keep them active.- func (m *Monitor) evaluateMockDockerHosts(hosts []models.DockerHost, fixtureRevision uint64) {++func (m *Monitor) evaluateMockDockerHosts(scope mockModeScope, hosts []models.DockerHost, fixtureRevision uint64) {m.mockHostAgentsMu.Lock()defer m.mockHostAgentsMu.Unlock()- if !m.acceptMockFixturePassLocked(fixtureRevision) {++ // Both the mode epoch and structural revision must still match.++ if !scope.current() || !m.acceptMockFixturePassLocked(fixtureRevision) {+ return+ }++ current := make(map[string]models.DockerHost, len(hosts))+ for _, host := range hosts {+ if host.ID != "" {+ current[host.ID] = host+ }+ }+ for id, host := range m.mockDockerHosts {+ if _, ok := current[id]; !ok {- m.alertManager.HandleDockerHostRemoved(host)++ if !scope.run(func() { m.alertManager.HandleDockerHostRemoved(host) }) {++ // Keep refused departures for the mode switch to remove.++ current[id] = host++ }+ }+ }+ for _, host := range hosts {- m.checkMockDockerHostAlerts(host)++ scope.run(func() { m.checkMockDockerHostAlerts(host) })+ }+ m.mockDockerHosts = current+}++// acceptMockFixturePassLocked reports whether a mock alert pass may update the+// tracked fixture hosts. A pass that took its snapshot before the monitor left+// mock mode must not evaluate fixtures after forgetMockFixtureHosts ran. Ticks+// start passes concurrently, so one that took its snapshot before an estate+// rebuild can also reach this lock after a newer pass; applying it would+// remove the new estate's hosts and re-evaluate retired ones. The fixture+// revision only advances on such rebuilds, so an older revision is rejected.+// Callers hold mockHostAgentsMu.+func (m *Monitor) acceptMockFixturePassLocked(fixtureRevision uint64) bool {+ if !mock.IsMockEnabled() || fixtureRevision < m.mockFixtureRevision {+ return false+ }+ m.mockFixtureRevision = fixtureRevision+ return true+}++// forgetMockFixtureHosts removes the fixture agents and Docker hosts when the+// monitor leaves mock mode. ClearActiveAlerts drops their alerts but not the+// agents' node links, and a leftover link would keep suppressing CPU, memory+// and disk alerts on any node that later carries the linked fixture node's ID.+// The removal also clears anything a pass already in flight+// recreated; mock mode is off before this runs, so no later pass can evaluate+// them again while it stays off. Disabling advanced the fixture revision, and+// recording it here keeps a pass paused across a disable and re-enable from+// evaluating the old estate once mock mode is back on.+func (m *Monitor) forgetMockFixtureHosts() {+ m.mockHostAgentsMu.Lock()+ defer m.mockHostAgentsMu.Unlock()++ if _, revision := mock.CurrentFixtureGraphWithRevision(); revision > m.mockFixtureRevision {+ m.mockFixtureRevision = revision+ }for _, host := range m.mockHostAgents {m.alertManager.HandleHostRemoved(host)}diff --cc internal/monitoring/monitor_host_agent_removal_lifecycle_test.goindex 1cc2468142,dca2a68b1c..b921bf36d2--- a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go+++ b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go@@@ -821,11 -821,11 +821,11 @@@ func TestMockHostAgentLeavingFixtureUsememory := models.Memory{Total: 32 << 30, Used: 30 << 30, Free: 2 << 30, Usage: 93.75}kept := models.Host{ID: "host-linux-1", Hostname: "apollo-114", Status: "online", Memory: memory}- departed := models.Host{ID: "host-node-pve9", Hostname: "pve9", Status: "online", Memory: memory}+ departed := models.Host{ID: "host-node-pve9", Hostname: "pve9", LinkedNodeID: "mock-cluster-1-pve9", Status: "online", Memory: memory}- monitor.evaluateMockHostAgents([]models.Host{kept, departed}, nil, 1)- monitor.evaluateMockHostAgents(monitor.mockModeFence.begin(), []models.Host{kept, departed}, nil)++ monitor.evaluateMockHostAgents(monitor.mockModeFence.begin(), []models.Host{kept, departed}, nil, 1)// A runtime mock config change rebuilt the estate without pve9.- monitor.evaluateMockHostAgents([]models.Host{kept}, nil, 2)- monitor.evaluateMockHostAgents(monitor.mockModeFence.begin(), []models.Host{kept}, nil)++ monitor.evaluateMockHostAgents(monitor.mockModeFence.begin(), []models.Host{kept}, nil, 2)keptAlert := falsefor _, alert := range manager.GetActiveAlerts() {@@@ -857,30 -857,3 +857,30 @@@}t.Fatal("node pve9 raised no memory alert after its agent left the mock estate")}++func TestMockDockerHostLeavingFixtureUsesRemovalLifecycle(t *testing.T) {+ mustSetMockEnabled(t, true)+ t.Cleanup(func() { mustSetMockEnabled(t, false) })+ manager := newMockHostAlertTestManager(t)+ monitor := &Monitor{alertManager: manager}++ kept := newMockDockerHostWithExitedContainers("nebula-1-mock", "nebula-1")+ // Nested Docker-in-LXC hosts are named after their guest's VMID, which a+ // runtime mock config change can shift.+ departed := newMockDockerHostWithExitedContainers("proxmox-lxc-docker:Production West:pve1:108", "pve1-ct108")+- monitor.evaluateMockDockerHosts([]models.DockerHost{kept, departed}, 1)- monitor.evaluateMockDockerHosts([]models.DockerHost{kept, departed}, 1)++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{kept, departed}, 1)++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{kept, departed}, 1)+ if ids := dockerAlertIDsForHost(manager, departed.ID); len(ids) != len(departed.Containers) {+ t.Fatalf("departing host opened %v, want one alert per exited container", ids)+ }+- monitor.evaluateMockDockerHosts([]models.DockerHost{kept}, 2)++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{kept}, 2)++ if ids := dockerAlertIDsForHost(manager, departed.ID); len(ids) != 0 {+ t.Fatalf("Docker host that left the mock estate kept alerts %v", ids)+ }+ if ids := dockerAlertIDsForHost(manager, kept.ID); len(ids) != len(kept.Containers) {+ t.Fatalf("Docker host still in the mock estate has alerts %v, want one per exited container", ids)+ }+}diff --cc internal/monitoring/monitor_mock_alerts_test.goindex 8110f2ba1d,3c02565ece..e45c690ff8--- a/internal/monitoring/monitor_mock_alerts_test.go+++ b/internal/monitoring/monitor_mock_alerts_test.go@@@ -211,18 -219,18 +221,19 @@@ func TestLeavingMockModeReleasesFixturefunc TestMockHostAgentPassAfterLeavingMockModeRegistersNothing(t *testing.T) {manager := newMockHostAlertTestManager(t)monitor := &Monitor{alertManager: manager}- mustSetMockEnabled(t, false)// A pass whose snapshot predates leaving mock mode reaches the agent step- // after forgetMockFixtureHosts ran.- // after SetMockMode ended its epoch and forgetMockHostAgents ran.++ // after SetMockMode ended its epoch and forgetMockFixtureHosts ran.+ scope := monitor.mockModeFence.begin()+ monitor.mockModeFence.advance()stale := models.Host{- ID: "host-node-mock-cluster-1-pve1",- Hostname: "pve1",- Status: "online",- Memory: models.Memory{Total: 32 << 30, Used: 30 << 30, Free: 2 << 30, Usage: 93.75},+ ID: "host-node-mock-cluster-1-pve1",+ Hostname: "pve1",+ LinkedNodeID: "real-pve1",+ Status: "online",+ Memory: models.Memory{Total: 32 << 30, Used: 30 << 30, Free: 2 << 30, Usage: 93.75},}- monitor.evaluateMockHostAgents([]models.Host{stale}, nil, 1)- monitor.evaluateMockHostAgents(scope, []models.Host{stale}, nil)++ monitor.evaluateMockHostAgents(scope, []models.Host{stale}, nil, 1)if active := manager.GetActiveAlerts(); len(active) != 0 {t.Fatalf("stale mock pass opened %d alerts in live mode", len(active))@@@ -240,121 -251,564 +254,707 @@@return}}- t.Fatal("stale mock pass registered fixture hostname pve1 and suppressed the real node's memory alert")+ t.Fatal("stale mock pass linked a fixture agent to real-pve1 and suppressed its memory alert")+}++// newMockDockerHostWithExitedContainers returns a reporting Docker host whose+// containers exited, so two evaluation passes open one alert per container.+func newMockDockerHostWithExitedContainers(id, hostname string) models.DockerHost {+ return models.DockerHost{+ ID: id,+ Hostname: hostname,+ DisplayName: hostname,+ Status: "online",+ Containers: []models.DockerContainer{+ {ID: id + "-web", Name: "web", State: "exited", Status: "Exited (137)"},+ {ID: id + "-db", Name: "db", State: "exited", Status: "Exited (137)"},+ },+ }+}++func dockerAlertIDsForHost(manager *alerts.Manager, hostID string) []string {+ var ids []string+ for _, alert := range manager.GetActiveAlerts() {+ if alert.ResourceID == "docker:"+hostID || strings.HasPrefix(alert.ResourceID, "docker:"+hostID+"/") {+ ids = append(ids, alert.ID)+ }+ }+ return ids+}++func TestMockDockerPassAfterLeavingMockModeRaisesNothing(t *testing.T) {+ manager := newMockHostAlertTestManager(t)+ monitor := &Monitor{alertManager: manager}+ mustSetMockEnabled(t, false)++ // A pass whose snapshot predates leaving mock mode reaches the Docker+ // step after forgetMockFixtureHosts ran.+ host := newMockDockerHostWithExitedContainers("nebula-1-mock", "nebula-1")- monitor.evaluateMockDockerHosts([]models.DockerHost{host}, 1)- monitor.evaluateMockDockerHosts([]models.DockerHost{host}, 1)++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{host}, 1)++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{host}, 1)++ if ids := dockerAlertIDsForHost(manager, host.ID); len(ids) != 0 {+ t.Fatalf("stale mock pass opened Docker alerts in live mode: %v", ids)+ }+}++func TestLeavingMockModeRemovesFixtureDockerHostAlerts(t *testing.T) {+ mustSetMockEnabled(t, true)+ t.Cleanup(func() { mustSetMockEnabled(t, false) })+ manager := newMockHostAlertTestManager(t)+ monitor := &Monitor{alertManager: manager}++ host := newMockDockerHostWithExitedContainers("nebula-1-mock", "nebula-1")- monitor.evaluateMockDockerHosts([]models.DockerHost{host}, 1)- monitor.evaluateMockDockerHosts([]models.DockerHost{host}, 1)++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{host}, 1)++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{host}, 1)+ if ids := dockerAlertIDsForHost(manager, host.ID); len(ids) != len(host.Containers) {+ t.Fatalf("fixture Docker host opened %v, want one alert per exited container", ids)+ }++ // The alerts stand in for ones a pass in flight recreated after+ // SetMockMode(false) cleared the manager.+ mustSetMockEnabled(t, false)+ monitor.forgetMockFixtureHosts()++ if ids := dockerAlertIDsForHost(manager, host.ID); len(ids) != 0 {+ t.Fatalf("fixture Docker host alerts outlived mock mode: %v", ids)+ }+}+++// Upstream mode epochs and the reviewed structural revisions protect different++// boundaries. A current revision cannot make an ended epoch eligible again.++func TestMockFixtureCurrentRevisionCannotReviveAnEndedEpoch(t *testing.T) {++ mustSetMockEnabled(t, true)++ t.Cleanup(func() { mustSetMockEnabled(t, false) })++ manager := newMockHostAlertTestManager(t)++ monitor := &Monitor{alertManager: manager}++ stale := monitor.mockModeFence.begin()++ monitor.mockModeFence.advance()++ _, revision := mock.CurrentFixtureGraphWithRevision()++ agent := models.Host{ID: "epoch-agent", Hostname: "epoch-agent", Status: "online"}++ docker := newMockDockerHostWithExitedContainers("epoch-docker", "epoch-docker")++++ monitor.evaluateMockHostAgents(stale, []models.Host{agent}, nil, revision)++ for range 2 {++ monitor.evaluateMockDockerHosts(stale, []models.DockerHost{docker}, revision)++ }++ if len(monitor.mockHostAgents) != 0 || len(monitor.mockDockerHosts) != 0 || len(manager.GetActiveAlerts()) != 0 {++ t.Fatal("a current fixture revision revived an ended mode epoch")++ }++++ current := monitor.mockModeFence.begin()++ monitor.evaluateMockHostAgents(current, []models.Host{agent}, nil, revision)++ for range 2 {++ monitor.evaluateMockDockerHosts(current, []models.DockerHost{docker}, revision)++ }++ if len(monitor.mockHostAgents) != 1 || len(monitor.mockDockerHosts) != 1 || len(dockerAlertIDsForHost(manager, docker.ID)) != len(docker.Containers) {++ t.Fatal("the current epoch lost normal fixture ownership or Docker evaluation")++ }++}+++func TestMockFixturePassFromBeforeAnEstateRebuildIsRejected(t *testing.T) {+ mustSetMockEnabled(t, true)+ t.Cleanup(func() { mustSetMockEnabled(t, false) })+ manager := newMockHostAlertTestManager(t)+ monitor := &Monitor{alertManager: manager}++ retired := newMockDockerHostWithExitedContainers("proxmox-lxc-docker:Production West:pve1:108", "pve1-ct108")+ current := newMockDockerHostWithExitedContainers("proxmox-lxc-docker:Production West:pve1:105", "pve1-ct105")- monitor.evaluateMockDockerHosts([]models.DockerHost{current}, 2)- monitor.evaluateMockDockerHosts([]models.DockerHost{current}, 2)++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{current}, 2)++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{current}, 2)++ // A slow pass whose snapshot predates the rebuild reaches the lock after+ // the newer passes. Applying it would remove the current host and+ // re-evaluate the retired one.- monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, 1)- monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, 1)++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{retired}, 1)++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{retired}, 1)++ if ids := dockerAlertIDsForHost(manager, current.ID); len(ids) != len(current.Containers) {+ t.Fatalf("superseded pass disturbed the current host's alerts: %v", ids)+ }+ if ids := dockerAlertIDsForHost(manager, retired.ID); len(ids) != 0 {+ t.Fatalf("superseded pass re-evaluated a retired host: %v", ids)+ }+}++func TestMockFixturePassPausedAcrossDisableAndReenableIsRejected(t *testing.T) {+ mustSetMockEnabled(t, true)+ t.Cleanup(func() { mustSetMockEnabled(t, false) })+ manager := newMockHostAlertTestManager(t)+ monitor := &Monitor{alertManager: manager}++ retired := newMockDockerHostWithExitedContainers("proxmox-lxc-docker:Production West:pve1:108", "pve1-ct108")+ _, staleRevision := mock.CurrentFixtureGraphWithRevision()- monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, staleRevision)++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{retired}, staleRevision)++ // Mock mode goes off and back on while a pass holding the old snapshot+ // is paused; it resumes before any pass of the rebuilt estate applies.+ mustSetMockEnabled(t, false)+ monitor.forgetMockFixtureHosts()+ mustSetMockEnabled(t, true)- monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, staleRevision)- monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, staleRevision)++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{retired}, staleRevision)++ monitor.evaluateMockDockerHosts(monitor.mockModeFence.begin(), []models.DockerHost{retired}, staleRevision)++ if ids := dockerAlertIDsForHost(manager, retired.ID); len(ids) != 0 {+ t.Fatalf("pass from before the disable re-evaluated the old estate: %v", ids)+ }}+ // pinDefaultMockEstate enables mock mode on the default estate, whatever+ // PULSE_MOCK_* the shell exports, and restores the previous mode and config.+ func pinDefaultMockEstate(t *testing.T) {+ t.Helper()+ previousEnabled, previousConfig := mock.IsMockEnabled(), mock.GetConfig()+ t.Cleanup(func() {+ mustSetMockEnabled(t, false)+ mock.SetMockConfig(previousConfig)+ mustSetMockEnabled(t, previousEnabled)+ })+ mustSetMockEnabled(t, false)+ cfg := mock.DefaultConfig+ cfg.UpdateInterval = 5 * time.Minute+ mock.SetMockConfig(cfg)+ }++ // signalFirstFiring closes the returned channel when manager fires its first+ // alert. Lifecycle callbacks run inside the evaluation that fired, so the+ // pass that fired is past its fixture read when the channel closes.+ func signalFirstFiring(t *testing.T, manager *alerts.Manager) <-chan struct{} {+ t.Helper()+ fired := make(chan struct{})+ var once sync.Once+ unsubscribe := manager.SubscribeLifecycleCallback(func(event alerts.LifecycleEvent) {+ if event.Type == eventlog.TypeFired {+ once.Do(func() { close(fired) })+ }+ })+ t.Cleanup(unsubscribe)+ return fired+ }++ func describeAlerts(active []alerts.Alert) string {+ keys := make([]string, 0, len(active))+ for _, alert := range active {+ keys = append(keys, alert.Type+" "+alert.ResourceID)+ }+ sort.Strings(keys)+ return strings.Join(keys, ", ")+ }++ // switchMockModeAsync runs SetMockMode on its own goroutine, so a test that is+ // holding an evaluation parked can still bound how long the switch takes.+ func switchMockModeAsync(t *testing.T, monitor *Monitor, enable bool) <-chan struct{} {+ t.Helper()+ done := make(chan struct{})+ go func() {+ defer close(done)+ if err := monitor.SetMockMode(enable); err != nil {+ t.Errorf("SetMockMode(%v): %v", enable, err)+ }+ }()+ return done+ }++ // goroutineIn reports whether one goroutine's stack holds every frame.+ func goroutineIn(frames ...string) bool {+ buf := make([]byte, 4<<20)+ n := runtime.Stack(buf, true)+ for _, goroutine := range bytes.Split(buf[:n], []byte("\n\n")) {+ all := true+ for _, frame := range frames {+ all = all && bytes.Contains(goroutine, []byte(frame))+ }+ if all {+ return true+ }+ }+ return false+ }++ // waitForGoroutineIn waits until one goroutine's stack holds every frame. A+ // test holding a lock uses it to see the goroutine it started parked on that+ // lock at a known point.+ func waitForGoroutineIn(t *testing.T, frames ...string) {+ t.Helper()+ waitForCondition(t, 30*time.Second, func() bool { return goroutineIn(frames...) },+ fmt.Sprintf("no goroutine reached %s", strings.Join(frames, " via ")))+ }++ func TestLeavingMockModeRefusesTheRestOfAnInFlightPass(t *testing.T) {+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)+ pinDefaultMockEstate(t)+ manager := newMockHostAlertTestManager(t)+ monitor := &Monitor{+ state: models.NewState(),+ alertManager: manager,+ metricsHistory: NewMetricsHistory(10, time.Hour),+ }+ mustSetMonitorMockMode(t, monitor, true)++ // Park a real pass at its first storage capacity read: it has read the+ // fixture and evaluated guests, agents and nodes; storage, disks, PBS,+ // PMG and Docker are still ahead of it.+ monitor.metricsHistory.capacityForecastMu.Lock()+ parked := true+ defer func() {+ if parked {+ monitor.metricsHistory.capacityForecastMu.Unlock()+ }+ }()+ passDone := make(chan struct{})+ go func() {+ defer close(passDone)+ monitor.checkMockAlerts()+ }()+ waitForGoroutineIn(t, "monitoring.(*Monitor).storageCapacityTrendFor", "monitoring.(*Monitor).checkMockAlerts")++ waitForSignal(t, switchMockModeAsync(t, monitor, false), 30*time.Second, "leaving mock mode waited on a pass parked outside any evaluation")+ monitor.metricsHistory.capacityForecastMu.Unlock()+ parked = false+ waitForSignal(t, passDone, 30*time.Second, "mock pass did not finish after it was released")++ if active := manager.GetActiveAlerts(); len(active) != 0 {+ t.Fatalf("a mock pass in flight when mock mode ended left %d fixture alerts in live mode: %s",+ len(active), describeAlerts(active))+ }+ }++ func TestLeavingMockModeWaitsForAnInFlightEvaluation(t *testing.T) {+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)+ pinDefaultMockEstate(t)+ manager := newMockHostAlertTestManager(t)+ monitor := &Monitor{+ state: models.NewState(),+ alertManager: manager,+ metricsHistory: NewMetricsHistory(10, time.Hour),+ }+ mustSetMonitorMockMode(t, monitor, true)++ // Park a real pass inside its first guest evaluation, in the CPU window+ // read the alert manager makes outside its own lock, so nothing but the+ // fence holds the switch back.+ entered, release := make(chan struct{}), make(chan struct{})+ var once sync.Once+ manager.SetMetricWindowProvider(func(alerts.MetricWindowRequest) ([]alerts.MetricWindowPoint, error) {+ once.Do(func() {+ close(entered)+ <-release+ })+ return nil, nil+ })+ released := false+ defer func() {+ if !released {+ close(release)+ }+ }()+ passDone := make(chan struct{})+ go func() {+ defer close(passDone)+ monitor.checkMockAlerts()+ }()+ waitForSignal(t, entered, 30*time.Second, "mock pass never evaluated a windowed metric")++ // The switch must stop in advance until the evaluation returns. One+ // that finishes first cleared alerts the evaluation can still raise.+ leftMockMode := switchMockModeAsync(t, monitor, false)+ waitForCondition(t, 30*time.Second, func() bool {+ select {+ case <-leftMockMode:+ return true+ default:+ }+ return goroutineIn("monitoring.(*mockModeFence).advance", "monitoring.(*Monitor).SetMockMode")+ }, "leaving mock mode neither finished nor waited for the running evaluation")+ select {+ case <-leftMockMode:+ t.Fatal("leaving mock mode cleared alerts while a fixture evaluation was still running")+ default:+ }++ close(release)+ released = true+ waitForSignal(t, leftMockMode, 30*time.Second, "leaving mock mode did not finish once the evaluation returned")+ waitForSignal(t, passDone, 30*time.Second, "mock pass did not finish after it was released")+ if active := manager.GetActiveAlerts(); len(active) != 0 {+ t.Fatalf("the evaluation running when mock mode ended left %d fixture alerts in live mode: %s",+ len(active), describeAlerts(active))+ }+ }++ // blockingFixtureRecords serves one source's fixture records the way the+ // router's mock adapter does, reading them on each call, and holds its first+ // call until released. Later calls are not held, not even while the first is.+ type blockingFixtureRecords struct {+ source unifiedresources.DataSource+ entered chan struct{}+ release chan struct{}+ held atomic.Bool+ }++ func newBlockingFixtureRecords(source unifiedresources.DataSource) *blockingFixtureRecords {+ return &blockingFixtureRecords{source: source, entered: make(chan struct{}), release: make(chan struct{})}+ }++ func (p *blockingFixtureRecords) SupplementalRecords(_ *Monitor, _ string) []unifiedresources.IngestRecord {+ records := mock.SupplementalRecords(p.source)+ if p.held.CompareAndSwap(false, true) {+ close(p.entered)+ <-p.release+ }+ return records+ }++ func TestLeavingMockModeDiscardsAnInFlightFrontendRefresh(t *testing.T) {+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)+ pinDefaultMockEstate(t)+ manager := newMockHostAlertTestManager(t)+ provider := newBlockingFixtureRecords(unifiedresources.SourceVMware)+ monitor := &Monitor{+ state: models.NewState(),+ alertManager: manager,+ metricsHistory: NewMetricsHistory(10, time.Hour),+ resourceStore: unifiedresources.NewMonitorAdapter(nil),+ supplementalProviders: map[unifiedresources.DataSource]MonitorSupplementalRecordsProvider{+ unifiedresources.SourceVMware: provider,+ },+ }+ mustSetMonitorMockMode(t, monitor, true)++ // Hold a frontend read in its registry rebuild: it has the fixture+ // snapshot and the vSphere fixture records, whose datastore, host and+ // network incidents its unified alert sync would raise.+ released := false+ defer func() {+ if !released {+ close(provider.release)+ }+ }()+ readDone := make(chan struct{})+ go func() {+ defer close(readDone)+ monitor.BuildFrontendState()+ }()+ waitForSignal(t, provider.entered, 30*time.Second, "frontend read never reached the supplemental records")++ waitForSignal(t, switchMockModeAsync(t, monitor, false), 30*time.Second, "leaving mock mode waited on a registry rebuild")+ close(provider.release)+ released = true+ waitForSignal(t, readDone, 30*time.Second, "frontend read did not finish after it was released")++ if active := manager.GetActiveAlerts(); len(active) != 0 {+ t.Fatalf("a frontend read in flight when mock mode ended left %d fixture alerts in live mode: %s",+ len(active), describeAlerts(active))+ }+ }++ // gatedListStore holds the first registry listing made after open is called,+ // so a test can publish another generation between a refresh's own publish+ // and its read back.+ type gatedListStore struct {+ *unifiedresources.MonitorAdapter+ mu sync.Mutex+ armed bool+ entered chan struct{}+ release chan struct{}+ }++ func (s *gatedListStore) arm() {+ s.mu.Lock()+ defer s.mu.Unlock()+ s.armed = true+ }++ func (s *gatedListStore) GetAll() []unifiedresources.Resource {+ s.mu.Lock()+ hold := s.armed+ s.armed = false+ s.mu.Unlock()+ if hold {+ close(s.entered)+ <-s.release+ }+ return s.MonitorAdapter.GetAll()+ }++ // fixedRecords stands in for a live provider: it serves the same records in+ // either mode.+ type fixedRecords []unifiedresources.IngestRecord++ func (r fixedRecords) SupplementalRecords(_ *Monitor, _ string) []unifiedresources.IngestRecord {+ return r+ }++ func alertTypes(active []alerts.Alert) map[string]int {+ types := make(map[string]int)+ for _, alert := range active {+ types[alert.Type]+++ }+ return types+ }++ func TestLiveRefreshIgnoresARegistryRepublishedFromMockMode(t *testing.T) {+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)+ pinDefaultMockEstate(t)+ manager := newMockHostAlertTestManager(t)+ mustSetMockEnabled(t, true)+ // TrueNAS records stand in for a live provider's data; the vSphere+ // fixture source serves nothing once mock mode is off.+ liveTrueNAS := fixedRecords(mock.SupplementalRecords(unifiedresources.SourceTrueNAS))+ mustSetMockEnabled(t, false)+ if len(liveTrueNAS) == 0 {+ t.Fatal("mock fixture has no TrueNAS records")+ }+ provider := newBlockingFixtureRecords(unifiedresources.SourceVMware)+ store := &gatedListStore{+ MonitorAdapter: unifiedresources.NewMonitorAdapter(nil),+ entered: make(chan struct{}),+ release: make(chan struct{}),+ }+ monitor := &Monitor{+ state: models.NewState(),+ alertManager: manager,+ metricsHistory: NewMetricsHistory(10, time.Hour),+ resourceStore: store,+ supplementalProviders: map[unifiedresources.DataSource]MonitorSupplementalRecordsProvider{+ unifiedresources.SourceVMware: provider,+ unifiedresources.SourceTrueNAS: liveTrueNAS,+ },+ }+ mustSetMonitorMockMode(t, monitor, true)+ releasedProvider, releasedList := false, false+ defer func() {+ if !releasedProvider {+ close(provider.release)+ }+ if !releasedList {+ close(store.release)+ }+ }()++ // A mock-mode refresh holds the fixture snapshot and vSphere records and+ // has not published them yet.+ staleDone := make(chan struct{})+ go func() {+ defer close(staleDone)+ monitor.updateResourceStore(monitor.currentStateWithScope())+ }()+ waitForSignal(t, provider.entered, 30*time.Second, "mock-mode refresh never reached the supplemental records")+ waitForSignal(t, switchMockModeAsync(t, monitor, false), 30*time.Second, "leaving mock mode waited on a registry rebuild")++ // A live refresh publishes the live TrueNAS records alone and is held+ // before reading the registry back.+ store.arm()+ liveDone := make(chan struct{})+ go func() {+ defer close(liveDone)+ monitor.updateResourceStore(monitor.currentStateWithScope())+ }()+ waitForSignal(t, store.entered, 30*time.Second, "live refresh never read the registry back")+ for _, resource := range store.MonitorAdapter.GetAll() {+ if slices.Contains(resource.Sources, unifiedresources.SourceVMware) {+ t.Fatalf("the live refresh published vSphere fixture resource %s", resource.ID)+ }+ }++ // The mock-mode refresh now publishes the fixture estate over it.+ close(provider.release)+ releasedProvider = true+ waitForSignal(t, staleDone, 30*time.Second, "mock-mode refresh did not finish after it was released")+ close(store.release)+ releasedList = true+ waitForSignal(t, liveDone, 30*time.Second, "live refresh did not finish after it was released")++ if active := manager.GetActiveAlerts(); len(active) != 0 {+ t.Fatalf("a live refresh evaluated a registry the mock-mode refresh republished, leaving %d fixture alerts: %s",+ len(active), describeAlerts(active))+ }+ if monitor.mockModeFence.registryCurrent() {+ t.Fatal("the registry the mock-mode refresh republished counted as current")+ }++ // The next live refresh is undisturbed: it replaces the registry and+ // evaluates the live records.+ monitor.updateResourceStore(monitor.currentStateWithScope())+ if !monitor.mockModeFence.registryCurrent() {+ t.Fatal("an undisturbed live rebuild did not make the registry current")+ }+ types := alertTypes(manager.GetActiveAlerts())+ if types["zfs-pool-state"] == 0 {+ t.Fatalf("the undisturbed live refresh raised no alert from the live TrueNAS records: %v", types)+ }+ for _, fixtureType := range []string{"resource-incident", "storage-incident"} {+ if types[fixtureType] != 0 {+ t.Fatalf("vSphere fixture incidents reached live mode: %v", types)+ }+ }+ }++ func TestDockerPruneIgnoresTheRegistryMockModeLeft(t *testing.T) {+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)+ pinDefaultMockEstate(t)+ manager := newMockHostAlertTestManager(t)+ monitor := &Monitor{+ state: models.NewState(),+ alertManager: manager,+ metricsHistory: NewMetricsHistory(10, time.Hour),+ resourceStore: unifiedresources.NewMonitorAdapter(nil),+ }+ mustSetMonitorMockMode(t, monitor, true)+ monitor.updateResourceStore(monitor.currentStateWithScope())+ mustSetMonitorMockMode(t, monitor, false)++ // Before any live rebuild replaces the fixture registry, a live Docker+ // host reports and then goes offline.+ host := models.DockerHost{ID: "live-docker-1", Hostname: "live-docker-1", DisplayName: "live-docker-1", Status: "offline"}+ monitor.state.UpsertDockerHost(host)+ for range 3 {+ manager.HandleDockerHostOffline(host)+ }+ if types := alertTypes(manager.GetActiveAlerts()); types["docker-host-offline"] != 1 {+ t.Fatalf("live Docker host raised no offline alert: %v", types)+ }++ monitor.SyncAlertState()+ if types := alertTypes(manager.GetActiveAlerts()); types["docker-host-offline"] != 1 {+ t.Fatalf("Docker pruning removed a live host's alert against the fixture registry: %v", types)+ }+ }++ func TestPreviousStateCarryIgnoresTheRegistryMockModeLeft(t *testing.T) {+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)+ pinDefaultMockEstate(t)+ monitor := &Monitor{+ state: models.NewState(),+ alertManager: newMockHostAlertTestManager(t),+ metricsHistory: NewMetricsHistory(10, time.Hour),+ resourceStore: unifiedresources.NewMonitorAdapter(nil),+ }+ mustSetMonitorMockMode(t, monitor, true)+ monitor.updateResourceStore(monitor.currentStateWithScope())+ fixture := mock.CurrentFixtureGraph().State+ if len(fixture.Nodes) == 0 || len(fixture.VMs) == 0 {+ t.Fatal("mock fixture has no nodes or VMs")+ }+ instance := fixture.VMs[0].Instance+ mustSetMonitorMockMode(t, monitor, false)++ // A live instance named like a fixture one polls before any live rebuild+ // replaces the fixture registry. Its previous-state carry, which writes+ // back into live state, must not see the fixture nodes and guests.+ if nodes := monitor.previousNodesForInstance(instance); len(nodes) != 0 {+ t.Fatalf("previous-node carry for %q returned %d fixture nodes after leaving mock mode", instance, len(nodes))+ }+ if guests := monitor.previousGuestContextForInstance(instance); len(guests.vms) != 0 || len(guests.containers) != 0 {+ t.Fatalf("previous-guest carry for %q returned %d fixture VMs and %d containers after leaving mock mode",+ instance, len(guests.vms), len(guests.containers))+ }+ }++ func TestLeavingMockModeDiscardsAnInFlightBackupEvaluation(t *testing.T) {+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)+ pinDefaultMockEstate(t)+ manager := newMockHostAlertTestManager(t)+ config := manager.GetConfig()+ config.BackupDefaults.Enabled = true+ config.BackupDefaults.WarningDays = 1+ config.BackupDefaults.CriticalDays = 2+ manager.UpdateConfig(config)+ monitor := &Monitor{+ state: models.NewState(),+ alertManager: manager,+ metricsHistory: NewMetricsHistory(10, time.Hour),+ }+ mustSetMonitorMockMode(t, monitor, true)+ // Cache the fixture read view the guest lookup will be served from.+ monitor.GetUnifiedReadStateOrSnapshot()++ // Park a real evaluation at its guest lookup, after it read the fixture+ // recovery rollups whose stale backups raise backup-age alerts.+ monitor.mockUnifiedViewMu.Lock()+ parked := true+ defer func() {+ if parked {+ monitor.mockUnifiedViewMu.Unlock()+ }+ }()+ evalDone := make(chan struct{})+ go func() {+ defer close(evalDone)+ monitor.checkBackupAlerts(context.Background())+ }()+ waitForGoroutineIn(t, "monitoring.(*Monitor).currentUnifiedStateView", "monitoring.(*Monitor).checkBackupAlerts")++ waitForSignal(t, switchMockModeAsync(t, monitor, false), 30*time.Second, "leaving mock mode waited on a backup evaluation parked outside the alert manager")+ monitor.mockUnifiedViewMu.Unlock()+ parked = false+ waitForSignal(t, evalDone, 30*time.Second, "backup evaluation did not finish after it was released")++ if active := manager.GetActiveAlerts(); len(active) != 0 {+ t.Fatalf("a backup evaluation in flight when mock mode ended left %d fixture alerts in live mode: %.500s",+ len(active), describeAlerts(active))+ }+ }++ func TestLeavingMockModeDiscardsAnInFlightConnectionCheck(t *testing.T) {+ pinDefaultMockEstate(t)+ manager := newMockHostAlertTestManager(t)+ monitor := &Monitor{+ state: models.NewState(),+ alertManager: manager,+ metricsHistory: NewMetricsHistory(10, time.Hour),+ }+ mustSetMonitorMockMode(t, monitor, true)++ // A hybrid mock ledger lists real connections beside fixture ones, so the+ // same connection can be read degraded in mock mode and again live.+ degraded := alerts.ConnectionSnapshot{+ ID: "pve-cluster-a",+ Name: "cluster-a",+ Type: alerts.ConnectionTypePVE,+ State: alerts.ConnectionStateUnreachable,+ Enabled: true,+ }+ entered, release := make(chan struct{}), make(chan struct{})+ var once sync.Once+ monitor.SetConnectionsSnapshotLister(func() []alerts.ConnectionSnapshot {+ once.Do(func() {+ close(entered)+ <-release+ })+ return []alerts.ConnectionSnapshot{degraded}+ })+ released := false+ defer func() {+ if !released {+ close(release)+ }+ }()+ checkDone := make(chan struct{})+ go func() {+ defer close(checkDone)+ monitor.checkConnectionAlerts()+ }()+ waitForSignal(t, entered, 30*time.Second, "connection check never read the ledger")++ waitForSignal(t, switchMockModeAsync(t, monitor, false), 30*time.Second, "leaving mock mode waited on a connection check parked in its ledger read")+ close(release)+ released = true+ waitForSignal(t, checkDone, 30*time.Second, "connection check did not finish after it was released")++ // The connection-degraded alert confirms on its third consecutive+ // observation; the one read in mock mode must not be one of them.+ monitor.checkConnectionAlerts()+ monitor.checkConnectionAlerts()+ if active := manager.GetActiveAlerts(); len(active) != 0 {+ t.Fatalf("a connection observation read in mock mode counted toward live confirmation: %s", describeAlerts(active))+ }+ monitor.checkConnectionAlerts()+ if active := manager.GetActiveAlerts(); len(active) != 1 {+ t.Fatalf("three live degraded observations raised %d alerts, want 1: %s", len(active), describeAlerts(active))+ }+ }+// diskAlertKeys lists the active disk-health and disk-wearout alerts as// "type instance/node device" so assertions name the disk, not the alert ID.func diskAlertKeys(manager *alerts.Manager) map[string]bool {diff --cc internal/truenas/provider.goindex cc12fd63b2,a023688f6c..bac1127425--- a/internal/truenas/provider.go+++ b/internal/truenas/provider.go@@@ -620,10 -605,15 +621,15 @@@ func truenasRecordsFromSnapshot(snapshoLastSeen: collectedAt,UpdatedAt: collectedAt,Metrics: &unifiedresources.ResourceMetrics{- Disk: diskMetric(pool.TotalBytes, pool.UsedBytes),+ Disk: poolDiskMetric(pool),},+ // TrueNAS has no disabled state for an imported pool, so+ // Enabled is always true; Active follows whether the pool is+ // usable now.Storage: &unifiedresources.StorageMeta{Type: "zfs-pool",+ Enabled: true,+ Active: poolStatus != unifiedresources.StatusOffline,IsZFS: true,Platform: "truenas",Topology: "pool",