Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Evaluate host agents in mock mode like live agents

rcourtmane3a42d86c3c9
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index b51b2d57dc..ea945c3572 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -8206,6 +8206,21 @@ or create agent continuity evidence. The removal lifecycle proof keeps a
removed host blocked across a PBS storage sync so metric timestamp maintenance
cannot become an accidental re-enrollment transition.
+### Mock fixture agents reuse only the removal alert boundary
+
+`internal/monitoring/monitor.go` now holds the set of fixture agents the last
+mock alert pass evaluated. When a runtime mock config change drops an agent
+from the estate, `evaluateMockHostAgents` routes it through the alert
+manager's `HandleHostRemoved`, the same alert boundary a deleted live agent
+crosses. Leaving mock mode routes every fixture agent through the same call,
+which also releases their hostname deduplication. Neither path writes a
+removal tombstone, revokes a token, touches continuity evidence or admits a
+report: fixture agents have no credentials or durable identity, and real
+reports stay discarded while mock mode is on.
+`TestMockHostAgentLeavingFixtureUsesRemovalLifecycle` in
+`internal/monitoring/monitor_host_agent_removal_lifecycle_test.go` pins the
+alert cleanup.
+
### Deploy enrollment swaps credentials as one durable transition
A deploy bootstrap token remains the live credential until Pulse can durably
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 9ae9a5d83a..a1c215ab3b 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -1596,6 +1596,58 @@ supporting inventory rather than a new batch of independent exits, so the
confirmed host incident clears child alerts instead of producing one alert per
container.
+Mock alert evaluation covers host agents the same way. Live agents are
+evaluated by `CheckHost` as each report lands. Mock mode discards reports, and
+the `evaluateHostAgents` health sweep reads monitor state, which never holds
+fixture agents, so `checkMockAlerts` evaluates every fixture agent itself
+through `evaluateMockHostAgents`. Without that pass no agent CPU, memory, disk,
+temperature or offline alert could open against mock data, while the same
+reading on a real agent would alert. The pass keeps the live lifecycle
+boundaries:
+
+- An offline fixture goes through `HandleHostOfflineWithCorrelation`, as a
+ lapsed live report does, because `CheckHost` treats its input as a fresh
+ report and would mark the host online.
+- Agents are evaluated before nodes, so a node with an online linked agent
+ hands its CPU, memory and disk alerts to the agent on the first tick, as
+ host-agent hostname deduplication does in production.
+- A runtime mock config change rebuilds the estate. An agent that leaves it
+ goes through `HandleHostRemoved`, as a deleted live agent does, so its alerts
+ and hostname deduplication do not outlive it.
+- Leaving mock mode routes the fixture agents through `HandleHostRemoved`.
+ `ClearActiveAlerts` drops their alerts but not their hostname registrations,
+ and a real node named like a fixture agent (`pve1`) would otherwise keep its
+ metric alerts suppressed with no agent to own them. A pass whose snapshot
+ predates the switch finds mock mode off under the same lock and evaluates
+ nothing. Two overlapping passes can still evaluate an older snapshot after
+ a newer one; the next pass reconciles the set again.
+
+Fixture agents must carry readings a real agent could report. Mock Kubernetes
+pods share 0.7 single-pod memory footprints per node, which keeps a node's pods
+at or below 58% of allocatable memory in steady state, and a 60% per-node
+ceiling clips the transient while rescheduled pods' readings settle. A dense
+node and its linked agent therefore no longer read a constant 100% and raise a
+critical alert no real cluster would. Whether a demo agent alerts at default thresholds depends on
+the readings each boot draws. On 2026-10-06, private stacks with the public
+demo's estate (8 nodes, 2 standalone agents and 3 Kubernetes node agents)
+opened none on one boot and one (a standalone agent at 91% memory) on another.
+With the agent memory trigger lowered to 60%, exactly the agents above it
+alerted and their linked Proxmox nodes raised none, and shrinking the estate
+through the mock settings API cleared the departed agents' alerts on the next
+pass. Those are measurements, not test guarantees. The tests pin the
+boundaries:
+`TestCheckMockAlertsEvaluatesHostAgentsBeforeLinkedNodes`,
+`TestCheckMockHostAlertsUsesHostLifecycleForOfflineFixtures`,
+`TestLeavingMockModeReleasesFixtureAgentHostnames` and
+`TestMockHostAgentPassAfterLeavingMockModeRegistersNothing` in
+`internal/monitoring/monitor_mock_alerts_test.go`,
+`TestMockHostAgentLeavingFixtureUsesRemovalLifecycle` in
+`internal/monitoring/monitor_host_agent_removal_lifecycle_test.go`, and
+`TestMockKubernetesDenseNodesStayBelowAllocatableMemory`,
+`TestMockKubernetesRescheduledPodsDoNotSaturateReceivingNode` and
+`TestMockKubernetesLoneHeavyPodStaysUnderNodeCeiling` in
+`internal/mock/generator_test.go`.
+
Host and container-runtime disk collection supports an explicit include list
for filesystems hidden by Pulse's automatic virtual/container filtering. The
include list is bounded to that automatic filter; explicit disk exclusions
diff --git a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md
index 50344121e9..9bc2a19077 100644
--- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md
+++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md
@@ -610,6 +610,14 @@ count times the number of shared storages.
Demo Docker container names stay unique per host (numbered suffixes on reused
profiles), so image "used by" lists and container rows never collapse two
distinct containers into what reads as a duplicated row.
+Mock Kubernetes pods share their node's allocatable memory: each active pod's
+memory target is scaled by `mockKubernetesPodMemoryShare` over the active pods
+on its node, which keeps a node's pods at or below 58% of allocatable memory in
+steady state, and `capMockKubernetesNodePodMemory` clips them to 60% only while
+rescheduled pods' smoothed readings settle, so a dense node does not pin itself
+and its linked agent at 100%. The count and the clip are linear passes over the
+cluster's pods in each usage update, keyed by node name, with no per-pod scan
+of other pods or nodes.
The browser applies resource deltas to its connection-scoped raw baseline, but
canonicalizes and reconciles only changed resources plus the host-merge groups
diff --git a/internal/mock/generator.go b/internal/mock/generator.go
index 658d7c48a6..0e641a5cc1 100644
--- a/internal/mock/generator.go
+++ b/internal/mock/generator.go
@@ -1738,16 +1738,97 @@ func initializeMockKubernetesClusterUsage(cluster *models.KubernetesCluster, now
}
}
+ activePodsByNode := make(map[string]int, len(cluster.Nodes))
+ for i := range cluster.Pods {
+ pod := &cluster.Pods[i]
+ nodeName := strings.TrimSpace(pod.NodeName)
+ if mockKubernetesPodIsActive(pod, nodeByName[nodeName]) {
+ activePodsByNode[nodeName]++
+ }
+ }
+
clusterOffline := strings.EqualFold(strings.TrimSpace(cluster.Status), "offline")
for i := range cluster.Pods {
pod := &cluster.Pods[i]
- node := nodeByName[strings.TrimSpace(pod.NodeName)]
- updateMockKubernetesPodUsage(pod, node, now, randomize, clusterOffline)
+ nodeName := strings.TrimSpace(pod.NodeName)
+ updateMockKubernetesPodUsage(pod, nodeByName[nodeName], activePodsByNode[nodeName], now, randomize, clusterOffline)
}
+ capMockKubernetesNodePodMemory(cluster, nodeByName)
recomputeMockKubernetesNodeUsage(cluster, now)
}
+// mockKubernetesNodePodMemoryBudget is how many single-pod memory footprints
+// the active pods on one node share. A pod's memory percent is measured
+// against its node's allocatable memory, as the live agent reports it, so
+// giving every pod a whole footprint (roughly 10-83%) committed any node
+// running four or more pods past its allocatable memory. The node, and the
+// host agent linked to it, then read a constant 100%. The largest footprint
+// is about 83% (a Guaranteed StatefulSet pod at full burst), so a budget of
+// 0.7 footprints keeps a node's pods at or below 58% in steady state, under
+// mockKubernetesNodePodMemoryCeiling.
+const mockKubernetesNodePodMemoryBudget = 0.7
+
+// mockKubernetesPodMemoryShare scales one pod's memory target by the number
+// of active pods on its node, so the node total stays within the budget plus
+// system overhead however many pods are scheduled there.
+func mockKubernetesPodMemoryShare(activePodsOnNode int) float64 {
+ if activePodsOnNode < 1 {
+ activePodsOnNode = 1
+ }
+ return mockKubernetesNodePodMemoryBudget / float64(activePodsOnNode)
+}
+
+// mockKubernetesNodePodMemoryCeiling bounds the fraction of a node's
+// allocatable memory its active pods may hold together. Pod readings smooth
+// toward their share of the budget, so they lag a change in the node's pod
+// count: a pod rescheduled off a failed node arrives with a reading sized for
+// its old node. Share-scaled targets stay under the ceiling, so it binds only
+// on that transient, keeping the node, with its system overhead of up to 26%,
+// below allocatable memory while readings settle.
+const mockKubernetesNodePodMemoryCeiling = 0.6
+
+// capMockKubernetesNodePodMemory applies the ceiling and reports how many
+// nodes it clipped.
+func capMockKubernetesNodePodMemory(cluster *models.KubernetesCluster, nodeByName map[string]*models.KubernetesNode) int {
+ usedByNode := make(map[string]int64, len(cluster.Nodes))
+ for i := range cluster.Pods {
+ pod := &cluster.Pods[i]
+ nodeName := strings.TrimSpace(pod.NodeName)
+ if nodeByName[nodeName] != nil && mockKubernetesPodIsActive(pod, nodeByName[nodeName]) {
+ usedByNode[nodeName] += pod.UsageMemoryBytes
+ }
+ }
+
+ scaleByNode := make(map[string]float64, len(usedByNode))
+ for nodeName, used := range usedByNode {
+ node := nodeByName[nodeName]
+ allocMemory := node.AllocMemoryBytes
+ if allocMemory <= 0 {
+ allocMemory = node.CapacityMemoryBytes
+ }
+ ceiling := float64(allocMemory) * mockKubernetesNodePodMemoryCeiling
+ if allocMemory > 0 && float64(used) > ceiling {
+ scaleByNode[nodeName] = ceiling / float64(used)
+ }
+ }
+ if len(scaleByNode) == 0 {
+ return 0
+ }
+
+ for i := range cluster.Pods {
+ pod := &cluster.Pods[i]
+ nodeName := strings.TrimSpace(pod.NodeName)
+ scale, ok := scaleByNode[nodeName]
+ if !ok || !mockKubernetesPodIsActive(pod, nodeByName[nodeName]) {
+ continue
+ }
+ pod.UsageMemoryPercent *= scale
+ pod.UsageMemoryBytes = int64(float64(pod.UsageMemoryBytes) * scale)
+ }
+ return len(scaleByNode)
+}
+
// reconcileMockKubernetesPodScheduling keeps pod placement coherent with
// node readiness. It runs on every mock update tick, both mid-tick (with
// randomize) and again after the demo scenario re-pins pod profiles, so it
@@ -1916,6 +1997,7 @@ func normalizeMockKubernetesNodeCapacity(node *models.KubernetesNode) {
func updateMockKubernetesPodUsage(
pod *models.KubernetesPod,
node *models.KubernetesNode,
+ activePodsOnNode int,
now time.Time,
randomize bool,
clusterOffline bool,
@@ -2002,8 +2084,9 @@ func updateMockKubernetesPodUsage(
baseMemory := 9.0 + float64(mockStableHash64(seedID, "mem-base")%20)
burstMemory := 14.0 + float64(mockStableHash64(seedID, "mem-burst")%36)
- targetMemory := (baseMemory + activity*burstMemory) * (0.92 + ownerScale*0.12)
- pod.UsageMemoryPercent = clampFloat(smoothMetricToward(pod.UsageMemoryPercent, targetMemory, 0.08), 3, 97)
+ memoryShare := mockKubernetesPodMemoryShare(activePodsOnNode)
+ targetMemory := (baseMemory + activity*burstMemory) * (0.92 + ownerScale*0.12) * memoryShare
+ pod.UsageMemoryPercent = clampFloat(smoothMetricToward(pod.UsageMemoryPercent, targetMemory, 0.08), 3*memoryShare, 97)
pod.UsageMemoryBytes = int64(float64(allocMemory) * (pod.UsageMemoryPercent / 100.0))
baseNetIn := float64((24 + int(mockStableHash64(seedID, "netin-base")%180)) * 1024)
diff --git a/internal/mock/generator_test.go b/internal/mock/generator_test.go
index 98772cd2ff..0c76b8346d 100644
--- a/internal/mock/generator_test.go
+++ b/internal/mock/generator_test.go
@@ -478,6 +478,144 @@ func TestBuildFixtureStateCreatesHostEntriesForKubernetesNodes(t *testing.T) {
}
}
+// newMockKubernetesTestCluster builds ready, schedulable nodes with a fixed
+// number of running pods each, so placement does not depend on the random
+// readiness the fixture generator rolls.
+func newMockKubernetesTestCluster(nodeCount, podsPerNode int) models.KubernetesCluster {
+ cluster := models.KubernetesCluster{ID: "usage-test", Name: "usage-test", Status: "online"}
+ for n := 0; n < nodeCount; n++ {
+ nodeName := fmt.Sprintf("usage-test-%d", n)
+ cluster.Nodes = append(cluster.Nodes, models.KubernetesNode{
+ UID: nodeName + "-uid",
+ Name: nodeName,
+ Ready: true,
+ CapacityCPU: 8,
+ AllocCPU: 8,
+ CapacityMemoryBytes: 32 << 30,
+ AllocMemoryBytes: 32 << 30,
+ })
+ for p := 0; p < podsPerNode; p++ {
+ cluster.Pods = append(cluster.Pods, models.KubernetesPod{
+ UID: fmt.Sprintf("%s-pod-%d-uid", nodeName, p),
+ Name: fmt.Sprintf("%s-pod-%d", nodeName, p),
+ Namespace: "apps",
+ NodeName: nodeName,
+ Phase: "Running",
+ OwnerKind: "Deployment",
+ })
+ }
+ }
+ return cluster
+}
+
+func TestMockKubernetesDenseNodesStayBelowAllocatableMemory(t *testing.T) {
+ cluster := newMockKubernetesTestCluster(3, 13)
+ data := models.StateSnapshot{KubernetesClusters: []models.KubernetesCluster{cluster}}
+ for _, node := range cluster.Nodes {
+ data.Hosts = append(data.Hosts, models.Host{ID: "host-" + node.Name, Hostname: node.Name, Status: "online"})
+ }
+
+ start := time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC)
+ for tick := 0; tick < 40; tick++ {
+ initializeMockKubernetesClusterUsage(&data.KubernetesClusters[0], start.Add(time.Duration(tick)*15*time.Second), false)
+ // The mock tick re-derives each linked host's memory from its node.
+ syncMockKubernetesNodeHosts(&data)
+
+ podMemory := make(map[string]float64, len(cluster.Nodes))
+ for _, pod := range data.KubernetesClusters[0].Pods {
+ podMemory[pod.NodeName] += pod.UsageMemoryPercent
+ }
+ for _, node := range data.KubernetesClusters[0].Nodes {
+ if node.UsageMemoryBytes >= node.AllocMemoryBytes || node.UsageMemoryPercent >= 100 {
+ t.Fatalf("tick %d: node %s with 13 running pods is pinned at allocatable memory (%.1f%%)", tick, node.Name, node.UsageMemoryPercent)
+ }
+ // Steady-state shares stay under the ceiling, which clips to it
+ // exactly; a total at the ceiling means it bound without a transient.
+ if podMemory[node.Name] >= mockKubernetesNodePodMemoryCeiling*100-0.01 {
+ t.Fatalf("tick %d: node %s pods hold %.2f%%, the ceiling bound in steady state", tick, node.Name, podMemory[node.Name])
+ }
+ }
+ for _, host := range data.Hosts {
+ if host.Memory.Usage >= 100 {
+ t.Fatalf("tick %d: linked host %s memory = %.1f%%, want the node's unsaturated reading", tick, host.Hostname, host.Memory.Usage)
+ }
+ }
+ }
+}
+
+func TestMockKubernetesRescheduledPodsDoNotSaturateReceivingNode(t *testing.T) {
+ cluster := newMockKubernetesTestCluster(3, 8)
+ start := time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC)
+ for tick := 0; tick < 40; tick++ {
+ initializeMockKubernetesClusterUsage(&cluster, start.Add(time.Duration(tick)*15*time.Second), false)
+ }
+
+ // Two nodes fail: their sixteen pods move to the survivor carrying
+ // readings sized for nodes that each ran a third as many pods.
+ cluster.Nodes[0].Ready = false
+ cluster.Nodes[1].Ready = false
+ initializeMockKubernetesClusterUsage(&cluster, start.Add(40*15*time.Second), false)
+
+ survivor := cluster.Nodes[2]
+ running := 0
+ for _, pod := range cluster.Pods {
+ if pod.NodeName == survivor.Name {
+ running++
+ }
+ }
+ if running != 24 {
+ t.Fatalf("survivor runs %d pods after the failures, want all 24", running)
+ }
+ // Pods hold at most the ceiling and system overhead tops out at 26%.
+ if limit := (mockKubernetesNodePodMemoryCeiling + 0.26) * 100; survivor.UsageMemoryPercent > limit+1e-9 {
+ t.Fatalf("survivor memory = %.1f%% right after rescheduling, want at most %.1f%%", survivor.UsageMemoryPercent, limit)
+ }
+}
+
+func TestMockKubernetesPodMemoryShareKeepsNodeWithinBudget(t *testing.T) {
+ for pods := 0; pods <= 64; pods++ {
+ share := mockKubernetesPodMemoryShare(pods)
+ if share <= 0 || share > 1 {
+ t.Fatalf("share(%d) = %f, want (0, 1]", pods, share)
+ }
+ if committed := float64(pods) * share; committed > mockKubernetesNodePodMemoryBudget+1e-9 {
+ t.Fatalf("%d pods commit %.2f single-pod footprints, want at most %.2f", pods, committed, mockKubernetesNodePodMemoryBudget)
+ }
+ }
+}
+
+func TestMockKubernetesLoneHeavyPodStaysUnderNodeCeiling(t *testing.T) {
+ // A five-minute cadence lets smoothing track each target closely, so the
+ // pod reaches its real peaks.
+ setMockUpdateInterval(5 * time.Minute)
+ t.Cleanup(func() { setMockUpdateInterval(DefaultConfig.UpdateInterval) })
+
+ // The heaviest footprint: the highest base and burst seeds on a
+ // Guaranteed StatefulSet pod.
+ uid, best := "", uint64(0)
+ for i := 0; i < 2000; i++ {
+ candidate := fmt.Sprintf("lone-pod-%d", i)
+ weight := mockStableHash64(candidate, "mem-base")%20 + mockStableHash64(candidate, "mem-burst")%36
+ if uid == "" || weight > best {
+ uid, best = candidate, weight
+ }
+ }
+ cluster := newMockKubernetesTestCluster(1, 1)
+ cluster.Pods[0].UID = uid
+ cluster.Pods[0].OwnerKind = "StatefulSet"
+ cluster.Pods[0].QoSClass = "Guaranteed"
+
+ start := time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)
+ peak := 0.0
+ for tick := 0; tick < 288; tick++ {
+ initializeMockKubernetesClusterUsage(&cluster, start.Add(time.Duration(tick)*5*time.Minute), false)
+ peak = math.Max(peak, cluster.Pods[0].UsageMemoryPercent)
+ }
+ if peak >= mockKubernetesNodePodMemoryCeiling*100-0.01 {
+ t.Fatalf("lone pod %s peaked at %.2f%%: the ceiling clipped a steady-state reading", uid, peak)
+ }
+}
+
func TestGenerateNodesBoundsDirectConfiguration(t *testing.T) {
cfg := DefaultConfig
cfg.NodeCount = 1 << 30
diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.go
index 53182cf222..baab181eb9 100644
--- a/internal/monitoring/monitor.go
+++ b/internal/monitoring/monitor.go
@@ -1263,6 +1263,8 @@ type Monitor struct {
recoveryIngestPending []recoveryIngestBatch
mockMetricsCancel context.CancelFunc
mockMetricsWg sync.WaitGroup
+ mockHostAgentsMu sync.Mutex
+ mockHostAgents map[string]models.Host // Fixture agents evaluated by the last mock alert pass
dockerChecker DockerChecker // Optional Docker checker for LXC containers
dockerCheckerConfiguredAt time.Time // Last time the Docker checker was configured
dockerCheckAllowedVMIDs map[int]struct{} // Optional VMID allowlist gating the LXC Docker socket probe; empty means all guests
@@ -4567,6 +4569,7 @@ func (m *Monitor) SetMockMode(enable bool) error {
return err
}
m.alertManager.ClearActiveAlerts()
+ m.forgetMockHostAgents()
m.mu.Lock()
m.resetStateLocked()
m.metricsHistory.Reset()
diff --git a/internal/monitoring/monitor_alerts.go b/internal/monitoring/monitor_alerts.go
index 46445c3ec7..96fecbde9e 100644
--- a/internal/monitoring/monitor_alerts.go
+++ b/internal/monitoring/monitor_alerts.go
@@ -686,6 +686,16 @@ func (m *Monitor) checkMockAlerts() {
guestsChecked++
}
+ // Check alerts for host agents. Live agents are evaluated as each report
+ // lands, and mock mode discards reports and keeps fixture agents out of
+ // the monitor state the host health sweep reads, so without this pass no
+ // agent CPU, memory, disk, temperature or offline alert can open against
+ // mock data. Agents go before nodes so a linked node hands its CPU, memory
+ // and disk alerts to its agent on the first tick, as the host-agent
+ // hostname deduplication does in production.
+ log.Debug().Int("hostCount", len(state.Hosts)).Msg("checking host agent alerts")
+ m.evaluateMockHostAgents(state.Hosts, state.Nodes)
+
// Check alerts for each node
for _, node := range state.Nodes {
m.alertManager.CheckNode(node)
@@ -727,6 +737,67 @@ func (m *Monitor) checkMockAlerts() {
mock.UpdateAlertSnapshots(m.alertManager.GetActiveAlerts(), m.alertManager.GetRecentlyResolved())
}
+// evaluateMockHostAgents evaluates every fixture agent and remembers the set.
+// A runtime mock config change rebuilds the estate, so an agent can leave it
+// between passes; it then goes through HandleHostRemoved, as a deleted live
+// agent does, or its alerts and hostname deduplication would outlive it.
+func (m *Monitor) evaluateMockHostAgents(hosts []models.Host, nodes []models.Node) {
+ m.mockHostAgentsMu.Lock()
+ defer m.mockHostAgentsMu.Unlock()
+
+ // A pass that took its snapshot before the monitor left mock mode must
+ // not re-register fixture agents after forgetMockHostAgents ran.
+ if !mock.IsMockEnabled() {
+ return
+ }
+
+ current := make(map[string]models.Host, len(hosts))
+ for _, host := range hosts {
+ if host.ID != "" {
+ current[host.ID] = host
+ }
+ }
+ // Remove first so a departed agent cannot unregister a hostname that a
+ // remaining agent registers on this same pass.
+ for id, host := range m.mockHostAgents {
+ if _, ok := current[id]; !ok {
+ m.alertManager.HandleHostRemoved(host)
+ }
+ }
+ for _, host := range hosts {
+ m.checkMockHostAlerts(host, nodes)
+ }
+ m.mockHostAgents = current
+}
+
+// forgetMockHostAgents removes the fixture agents when the monitor leaves mock
+// mode. ClearActiveAlerts drops their alerts but not their hostname
+// deduplication, so a real node named like a fixture agent (pve1) would keep
+// its CPU, memory and disk alerts suppressed with no agent to own them. The
+// removal also clears anything a pass already in flight recreated; mock mode
+// is off before this runs, so no later pass can register them again.
+func (m *Monitor) forgetMockHostAgents() {
+ m.mockHostAgentsMu.Lock()
+ defer m.mockHostAgentsMu.Unlock()
+
+ for _, host := range m.mockHostAgents {
+ m.alertManager.HandleHostRemoved(host)
+ }
+ m.mockHostAgents = nil
+}
+
+// checkMockHostAlerts applies the live host-agent boundary to a fixture. An
+// offline fixture has no fresh telemetry, so it goes through the host
+// connectivity lifecycle that evaluateHostAgents uses for a lapsed report.
+// CheckHost treats its input as a fresh report and would mark the host online.
+func (m *Monitor) checkMockHostAlerts(host models.Host, nodes []models.Node) {
+ if strings.EqualFold(strings.TrimSpace(host.Status), "offline") {
+ m.alertManager.HandleHostOfflineWithCorrelation(host, sharedSystemAlertCorrelationForHost(host, nodes))
+ return
+ }
+ m.alertManager.CheckHost(host)
+}
+
// checkMockDockerHostAlerts preserves the same evidence boundary as live
// agent monitoring. An explicitly offline fixture is missing fresh container
// telemetry; its last container states must not be reinterpreted as a fresh
diff --git a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go
index 3b9eac2272..7459a3c610 100644
--- a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go
+++ b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go
@@ -812,3 +812,48 @@ func TestUnifiedStorageMetricSyncPreservesHostRemovalBlock(t *testing.T) {
t.Fatal("storage metric sync allowed a removed host to report with its denied token")
}
}
+
+func TestMockHostAgentLeavingFixtureUsesRemovalLifecycle(t *testing.T) {
+ mustSetMockEnabled(t, true)
+ t.Cleanup(func() { mustSetMockEnabled(t, false) })
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{alertManager: manager}
+
+ memory := models.Memory{Total: 32 << 30, Used: 30 << 30, Free: 2 << 30, Usage: 93.75}
+ kept := models.Host{ID: "host-linux-1", Hostname: "apollo-114", Status: "online", Memory: memory}
+ departed := models.Host{ID: "host-node-pve9", Hostname: "pve9", Status: "online", Memory: memory}
+
+ monitor.evaluateMockHostAgents([]models.Host{kept, departed}, nil)
+ // A runtime mock config change rebuilt the estate without pve9.
+ monitor.evaluateMockHostAgents([]models.Host{kept}, nil)
+
+ keptAlert := false
+ for _, alert := range manager.GetActiveAlerts() {
+ switch alert.ResourceID {
+ case "agent:" + departed.ID:
+ t.Fatalf("agent that left the mock estate kept alert %s", alert.ID)
+ case "agent:" + kept.ID:
+ keptAlert = keptAlert || alert.Type == "memory"
+ }
+ }
+ if !keptAlert {
+ t.Fatal("agent still in the mock estate lost its memory alert")
+ }
+
+ // The departed agent's hostname deduplication is released too, so the
+ // node it was linked to owns its metric alerts again.
+ node := models.Node{
+ ID: "mock-cluster-1-pve9",
+ Name: "pve9",
+ Instance: "mock-cluster-1",
+ Status: "online",
+ Memory: memory,
+ }
+ manager.CheckNode(node)
+ for _, alert := range manager.GetActiveAlerts() {
+ if alert.ResourceID == node.ID && alert.Type == "memory" {
+ return
+ }
+ }
+ t.Fatal("node pve9 raised no memory alert after its agent left the mock estate")
+}
diff --git a/internal/monitoring/monitor_mock_alerts_test.go b/internal/monitoring/monitor_mock_alerts_test.go
index bd0bb9b10a..66e69d616a 100644
--- a/internal/monitoring/monitor_mock_alerts_test.go
+++ b/internal/monitoring/monitor_mock_alerts_test.go
@@ -1,9 +1,12 @@
package monitoring
import (
+ "strings"
"testing"
+ "time"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
+ "github.com/rcourtman/pulse-go-rewrite/internal/mock"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
)
@@ -50,3 +53,189 @@ func TestCheckMockDockerHostAlertsUsesHostLifecycleForOfflineFixtures(t *testing
t.Fatalf("offline mock host alert = %+v, want canonical host connectivity incident", active[0])
}
}
+
+func newMockHostAlertTestManager(t *testing.T) *alerts.Manager {
+ t.Helper()
+ manager := alerts.NewManagerWithDataDir(t.TempDir())
+ t.Cleanup(manager.Stop)
+ config := manager.GetConfig()
+ config.Enabled = true
+ config.ActivationState = alerts.ActivationPending
+ config.TimeThresholds = map[string]int{}
+ config.MetricTimeThresholds = nil
+ config.BackupDefaults.Enabled = false
+ config.AgentDefaults.Memory = &alerts.HysteresisThreshold{Trigger: 1, Clear: 0.5}
+ config.NodeDefaults.Memory = &alerts.HysteresisThreshold{Trigger: 1, Clear: 0.5}
+ manager.UpdateConfig(config)
+ return manager
+}
+
+func TestCheckMockAlertsEvaluatesHostAgentsBeforeLinkedNodes(t *testing.T) {
+ mustSetMockEnabled(t, true)
+ t.Cleanup(func() { mustSetMockEnabled(t, false) })
+ manager := newMockHostAlertTestManager(t)
+
+ monitor := &Monitor{alertManager: manager}
+ monitor.checkMockAlerts()
+
+ // Agents reading well above the 1% trigger must alert. A NotReady
+ // Kubernetes node's agent validly reads 0%, and a later fixture read may
+ // drift slightly from the pass's own snapshot, so the bound leaves margin.
+ state := mock.CurrentFixtureGraph().State
+ onlineHosts := make(map[string]models.Host, len(state.Hosts))
+ for _, host := range state.Hosts {
+ if !strings.EqualFold(strings.TrimSpace(host.Status), "offline") && host.Memory.HasKnownUsage() && host.Memory.Usage >= 5 {
+ onlineHosts[host.ID] = host
+ }
+ }
+ if len(onlineHosts) == 0 {
+ t.Fatal("mock fixture has no online host agents reading above 5% memory")
+ }
+
+ active := manager.GetActiveAlerts()
+ agentMemoryAlerts := 0
+ alertByResourceMetric := make(map[string]bool, len(active))
+ for _, alert := range active {
+ alertByResourceMetric[alert.ResourceID+"/"+alert.Type] = true
+ if alert.Type != "memory" || !strings.HasPrefix(alert.ResourceID, "agent:") {
+ continue
+ }
+ if _, ok := onlineHosts[strings.TrimPrefix(alert.ResourceID, "agent:")]; ok {
+ agentMemoryAlerts++
+ }
+ }
+ if agentMemoryAlerts != len(onlineHosts) {
+ t.Fatalf("mock pass opened %d agent memory alerts at a 1%% trigger, want one per online agent above 5%% (%d)", agentMemoryAlerts, len(onlineHosts))
+ }
+
+ // A node whose linked agent reported on the same pass must hand its
+ // memory alert to the agent on the first tick, as hostname
+ // deduplication does for live agents.
+ linkedNodes := 0
+ for _, node := range state.Nodes {
+ host, ok := onlineHosts[strings.TrimSpace(node.LinkedAgentID)]
+ if !ok || !strings.EqualFold(host.Hostname, node.Name) {
+ continue
+ }
+ linkedNodes++
+ if alertByResourceMetric[node.ID+"/memory"] {
+ t.Fatalf("node %s raised its own memory alert although linked agent %s reported on the same pass", node.Name, host.ID)
+ }
+ }
+ if linkedNodes == 0 {
+ t.Fatal("mock fixture has no online node with a linked agent")
+ }
+}
+
+func TestCheckMockHostAlertsUsesHostLifecycleForOfflineFixtures(t *testing.T) {
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{alertManager: manager}
+
+ host := models.Host{
+ ID: "host-linux-9",
+ Hostname: "branch-gateway",
+ DisplayName: "Branch Gateway",
+ Platform: "linux",
+ Status: "offline",
+ Memory: models.Memory{Total: 16 << 30, Used: 15 << 30, Usage: 93.75},
+ }
+
+ // Three offline observations confirm the connectivity incident. CheckHost
+ // would treat the stale fixture as a fresh report: it marks the host
+ // online and raises the stale 94% memory reading instead.
+ monitor.checkMockHostAlerts(host, nil)
+ monitor.checkMockHostAlerts(host, nil)
+ monitor.checkMockHostAlerts(host, nil)
+
+ active := manager.GetActiveAlerts()
+ if len(active) != 1 || active[0].Type != alerts.HostOfflineAlertType || active[0].ResourceID != "agent:host-linux-9" {
+ got := make([]string, 0, len(active))
+ for _, alert := range active {
+ got = append(got, alert.ResourceID+" "+alert.Type)
+ }
+ t.Fatalf("offline mock agent alerts = %v, want only the host-offline incident for agent:host-linux-9", got)
+ }
+
+ host.Status = "online"
+ monitor.checkMockHostAlerts(host, nil)
+ for _, alert := range manager.GetActiveAlerts() {
+ if alert.Type == alerts.HostOfflineAlertType {
+ t.Fatalf("recovered mock agent kept its offline incident %s", alert.ID)
+ }
+ }
+}
+
+func TestLeavingMockModeReleasesFixtureAgentHostnames(t *testing.T) {
+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{
+ state: models.NewState(),
+ alertManager: manager,
+ metricsHistory: NewMetricsHistory(10, time.Hour),
+ }
+ mustSetMonitorMockMode(t, monitor, true)
+ t.Cleanup(func() { mustSetMockEnabled(t, false) })
+ monitor.checkMockAlerts()
+
+ registeredPVE1 := false
+ for _, host := range mock.CurrentFixtureGraph().State.Hosts {
+ if host.Hostname == "pve1" && !strings.EqualFold(host.Status, "offline") {
+ registeredPVE1 = true
+ }
+ }
+ if !registeredPVE1 {
+ t.Fatal("mock fixture has no online agent named pve1")
+ }
+ mustSetMonitorMockMode(t, monitor, false)
+
+ // A real node named like a fixture agent, with no agent of its own,
+ // must own its metric alerts once the monitor shows live data again.
+ node := models.Node{
+ ID: "real-pve1",
+ Name: "pve1",
+ Instance: "real",
+ Status: "online",
+ Memory: models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75},
+ }
+ manager.CheckNode(node)
+ for _, alert := range manager.GetActiveAlerts() {
+ if alert.ResourceID == node.ID && alert.Type == "memory" {
+ return
+ }
+ }
+ t.Fatal("real node pve1 raised no memory alert: the fixture agent's hostname deduplication outlived mock mode")
+}
+
+func TestMockHostAgentPassAfterLeavingMockModeRegistersNothing(t *testing.T) {
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{alertManager: manager}
+ mustSetMockEnabled(t, false)
+
+ // A pass whose snapshot predates leaving mock mode reaches the agent step
+ // after forgetMockHostAgents ran.
+ stale := models.Host{
+ ID: "host-node-mock-cluster-1-pve1",
+ Hostname: "pve1",
+ Status: "online",
+ Memory: models.Memory{Total: 32 << 30, Used: 30 << 30, Free: 2 << 30, Usage: 93.75},
+ }
+ monitor.evaluateMockHostAgents([]models.Host{stale}, nil)
+
+ if active := manager.GetActiveAlerts(); len(active) != 0 {
+ t.Fatalf("stale mock pass opened %d alerts in live mode", len(active))
+ }
+ node := models.Node{
+ ID: "real-pve1",
+ Name: "pve1",
+ Instance: "real",
+ Status: "online",
+ Memory: models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75},
+ }
+ manager.CheckNode(node)
+ for _, alert := range manager.GetActiveAlerts() {
+ if alert.ResourceID == node.ID && alert.Type == "memory" {
+ return
+ }
+ }
+ t.Fatal("stale mock pass registered fixture hostname pve1 and suppressed the real node's memory alert")
+}