Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Remove departed mock Docker hosts from alert evaluation
pulse-triage[bot]b05b8f352764
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex de0ce3e17d..1de6d23f3e 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -8308,8 +8308,12 @@ cannot become an accidental re-enrollment transition.mock alert pass evaluated. When a runtime mock config change drops an agentfrom the estate, `evaluateMockHostAgents` routes it through the alertmanager's `HandleHostRemoved`, the same alert boundary a deleted live agent-crosses. Leaving mock mode routes every fixture agent through the same call,-which also releases their hostname deduplication. Neither path writes a+crosses. Fixture Docker hosts that leave the estate go through+`HandleDockerHostRemoved` the same way. Leaving mock mode routes every fixture+agent and Docker host through those calls, which also drops the agents' node+links. The monitor also records the fixture's structural revision so a mock+pass that predates an estate rebuild, or a disable and re-enable, cannot+re-run that removal against the current fixture agents. Neither path writes aremoval tombstone, revokes a token, touches continuity evidence or admits areport: fixture agents have no credentials or durable identity, and realreports stay discarded while mock mode is on.diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex 1cc77d03ff..3d2a0c2586 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -1732,6 +1732,21 @@ supporting inventory rather than a new batch of independent exits, so theconfirmed host incident clears child alerts instead of producing one alert percontainer.+Mock Docker hosts also keep the live removal boundary. Nested Docker-in-LXC+hosts are named after their guest's VMID, so a runtime mock config change can+drop one from the estate; `evaluateMockDockerHosts` then routes it through+`HandleDockerHostRemoved`, as a deleted live host is. `pruneStaleDockerAlerts`+clears the same alerts, but only on a state read or subscribed broadcast, so an+unwatched stack would otherwise keep them active for `/api/alerts/active`,+Patrol and escalations. The Docker pass shares the host-agent pass's lock and+mock-mode check, and leaving mock mode removes the fixture Docker hosts too, so+a pass in flight cannot leave Docker alerts behind in live mode.+`TestMockDockerHostLeavingFixtureUsesRemovalLifecycle`,+`TestMockDockerPassAfterLeavingMockModeRaisesNothing`,+`TestLeavingMockModeRemovesFixtureDockerHostAlerts`,+`TestMockFixturePassFromBeforeAnEstateRebuildIsRejected` and+`TestMockFixturePassPausedAcrossDisableAndReenableIsRejected` pin this.+Mock alert evaluation covers host agents the same way. Live agents areevaluated by `CheckHost` as each report lands. Mock mode discards reports, andthe `evaluateHostAgents` health sweep reads monitor state, which never holds@@ -1746,17 +1761,23 @@ boundaries:report and would mark the host online.- Agents are evaluated before nodes, so a node with an online linked agenthands its CPU, memory and disk alerts to the agent on the first tick, as- host-agent hostname deduplication does in production.+ node-link deduplication does in production.- A runtime mock config change rebuilds the estate. An agent that leaves itgoes through `HandleHostRemoved`, as a deleted live agent does, so its alerts- and hostname deduplication do not outlive it.+ and node link do not outlive it.- Leaving mock mode routes the fixture agents through `HandleHostRemoved`.- `ClearActiveAlerts` drops their alerts but not their hostname registrations,- and a real node named like a fixture agent (`pve1`) would otherwise keep its- metric alerts suppressed with no agent to own them. A pass whose snapshot- predates the switch finds mock mode off under the same lock and evaluates- nothing. Two overlapping passes can still evaluate an older snapshot after- a newer one; the next pass reconciles the set again.+ `ClearActiveAlerts` drops their alerts but not their node links, and a+ leftover link would keep suppressing CPU, memory and disk alerts on any node+ that later carries the linked fixture node's ID. The agent and Docker+ steps of a pass whose snapshot predates the switch find mock mode off under+ the same lock and evaluate nothing.+- Ticks start passes concurrently, so a pass whose snapshot predates an+ estate rebuild can reach the lock after a newer one. Each pass carries the+ fixture's structural revision, and the agent and Docker steps of a pass+ older than the last applied or disabled revision are skipped, so they+ cannot remove the new estate's hosts or re-evaluate retired ones, even+ across a disable and re-enable. The pass's other loops (guests, nodes,+ storage, PBS, PMG) take no part in this tracking and are not guarded.Fixture agents must carry readings a real agent could report. Mock Kubernetespods share 0.7 single-pod memory footprints per node, which keeps a node's pods@@ -1774,7 +1795,7 @@ pass. Those are measurements, not test guarantees. The tests pin theboundaries:`TestCheckMockAlertsEvaluatesHostAgentsBeforeLinkedNodes`,`TestCheckMockHostAlertsUsesHostLifecycleForOfflineFixtures`,-`TestLeavingMockModeReleasesFixtureAgentHostnames` and+`TestLeavingMockModeReleasesFixtureAgentNodeLinks` and`TestMockHostAgentPassAfterLeavingMockModeRegistersNothing` in`internal/monitoring/monitor_mock_alerts_test.go`,`TestMockHostAgentLeavingFixtureUsesRemovalLifecycle` indiff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.goindex 65df2b3a2e..388a3e7c90 100644--- a/internal/monitoring/monitor.go+++ b/internal/monitoring/monitor.go@@ -1303,6 +1303,8 @@ type Monitor struct {mockMetricsCancel context.CancelFuncmockMetricsWg sync.WaitGroupmockHostAgentsMu sync.Mutex+ mockDockerHosts map[string]models.DockerHost+ mockFixtureRevision uint64mockHostAgents map[string]models.Host // Fixture agents evaluated by the last mock alert passdockerChecker DockerChecker // Optional Docker checker for LXC containersdockerCheckerConfiguredAt time.Time // Last time the Docker checker was configured@@ -4628,7 +4630,7 @@ func (m *Monitor) SetMockMode(enable bool) error {return err}m.alertManager.ClearActiveAlerts()- m.forgetMockHostAgents()+ m.forgetMockFixtureHosts()m.mu.Lock()m.resetStateLocked()m.metricsHistory.Reset()diff --git a/internal/monitoring/monitor_alerts.go b/internal/monitoring/monitor_alerts.goindex 89e870836f..bb581d4da0 100644--- a/internal/monitoring/monitor_alerts.go+++ b/internal/monitoring/monitor_alerts.go@@ -635,7 +635,8 @@ func (m *Monitor) checkMockAlerts() {}// Get mock state- state := mock.CurrentFixtureGraph().State+ graph, fixtureRevision := mock.CurrentFixtureGraphWithRevision()+ state := graph.Statelog.Debug().Int("vms", len(state.VMs)).@@ -696,10 +697,10 @@ func (m *Monitor) checkMockAlerts() {// the monitor state the host health sweep reads, so without this pass no// agent CPU, memory, disk, temperature or offline alert can open against// mock data. Agents go before nodes so a linked node hands its CPU, memory- // and disk alerts to its agent on the first tick, as the host-agent- // hostname deduplication does in production.+ // and disk alerts to its agent on the first tick, as node-link+ // deduplication does in production.log.Debug().Int("hostCount", len(state.Hosts)).Msg("checking host agent alerts")- m.evaluateMockHostAgents(state.Hosts, state.Nodes)+ m.evaluateMockHostAgents(state.Hosts, state.Nodes, fixtureRevision)// Check alerts for each nodefor _, node := range state.Nodes {@@ -733,9 +734,7 @@ func (m *Monitor) checkMockAlerts() {// so skipping this loop leaves the docker alert lifecycle unexercisable// against mock data.log.Debug().Int("dockerHostCount", len(state.DockerHosts)).Msg("checking docker alerts")- for _, dockerHost := range state.DockerHosts {- m.checkMockDockerHostAlerts(dockerHost)- }+ m.evaluateMockDockerHosts(state.DockerHosts, fixtureRevision)// Cache the latest alert snapshots directly in the mock data so the API can serve// mock state without needing to grab the alert manager lock again.@@ -745,14 +744,12 @@ func (m *Monitor) checkMockAlerts() {// evaluateMockHostAgents evaluates every fixture agent and remembers the set.// A runtime mock config change rebuilds the estate, so an agent can leave it// between passes; it then goes through HandleHostRemoved, as a deleted live-// agent does, or its alerts and hostname deduplication would outlive it.-func (m *Monitor) evaluateMockHostAgents(hosts []models.Host, nodes []models.Node) {+// agent does, or its alerts and node link would outlive it.+func (m *Monitor) evaluateMockHostAgents(hosts []models.Host, nodes []models.Node, fixtureRevision uint64) {m.mockHostAgentsMu.Lock()defer m.mockHostAgentsMu.Unlock()- // A pass that took its snapshot before the monitor left mock mode must- // not re-register fixture agents after forgetMockHostAgents ran.- if !mock.IsMockEnabled() {+ if !m.acceptMockFixturePassLocked(fixtureRevision) {return}@@ -762,8 +759,6 @@ func (m *Monitor) evaluateMockHostAgents(hosts []models.Host, nodes []models.Nodcurrent[host.ID] = host}}- // Remove first so a departed agent cannot unregister a hostname that a- // remaining agent registers on this same pass.for id, host := range m.mockHostAgents {if _, ok := current[id]; !ok {m.alertManager.HandleHostRemoved(host)@@ -775,20 +770,77 @@ func (m *Monitor) evaluateMockHostAgents(hosts []models.Host, nodes []models.Nodm.mockHostAgents = current}-// forgetMockHostAgents removes the fixture agents when the monitor leaves mock-// mode. ClearActiveAlerts drops their alerts but not their hostname-// deduplication, so a real node named like a fixture agent (pve1) would keep-// its CPU, memory and disk alerts suppressed with no agent to own them. The-// removal also clears anything a pass already in flight recreated; mock mode-// is off before this runs, so no later pass can register them again.-func (m *Monitor) forgetMockHostAgents() {+// evaluateMockDockerHosts is the Docker counterpart of evaluateMockHostAgents,+// under the same lock and mode check. Nested Docker-in-LXC hosts are named+// after their guest's VMID, so a runtime mock config change can drop a host+// from the estate; it then goes through HandleDockerHostRemoved, as a deleted+// live host does. pruneStaleDockerAlerts clears the same alerts, but only when+// something reads state, so an unwatched stack would keep them active.+func (m *Monitor) evaluateMockDockerHosts(hosts []models.DockerHost, fixtureRevision uint64) {+ m.mockHostAgentsMu.Lock()+ defer m.mockHostAgentsMu.Unlock()++ if !m.acceptMockFixturePassLocked(fixtureRevision) {+ return+ }++ current := make(map[string]models.DockerHost, len(hosts))+ for _, host := range hosts {+ if host.ID != "" {+ current[host.ID] = host+ }+ }+ for id, host := range m.mockDockerHosts {+ if _, ok := current[id]; !ok {+ m.alertManager.HandleDockerHostRemoved(host)+ }+ }+ for _, host := range hosts {+ m.checkMockDockerHostAlerts(host)+ }+ m.mockDockerHosts = current+}++// acceptMockFixturePassLocked reports whether a mock alert pass may update the+// tracked fixture hosts. A pass that took its snapshot before the monitor left+// mock mode must not evaluate fixtures after forgetMockFixtureHosts ran. Ticks+// start passes concurrently, so one that took its snapshot before an estate+// rebuild can also reach this lock after a newer pass; applying it would+// remove the new estate's hosts and re-evaluate retired ones. The fixture+// revision only advances on such rebuilds, so an older revision is rejected.+// Callers hold mockHostAgentsMu.+func (m *Monitor) acceptMockFixturePassLocked(fixtureRevision uint64) bool {+ if !mock.IsMockEnabled() || fixtureRevision < m.mockFixtureRevision {+ return false+ }+ m.mockFixtureRevision = fixtureRevision+ return true+}++// forgetMockFixtureHosts removes the fixture agents and Docker hosts when the+// monitor leaves mock mode. ClearActiveAlerts drops their alerts but not the+// agents' node links, and a leftover link would keep suppressing CPU, memory+// and disk alerts on any node that later carries the linked fixture node's ID.+// The removal also clears anything a pass already in flight+// recreated; mock mode is off before this runs, so no later pass can evaluate+// them again while it stays off. Disabling advanced the fixture revision, and+// recording it here keeps a pass paused across a disable and re-enable from+// evaluating the old estate once mock mode is back on.+func (m *Monitor) forgetMockFixtureHosts() {m.mockHostAgentsMu.Lock()defer m.mockHostAgentsMu.Unlock()+ if _, revision := mock.CurrentFixtureGraphWithRevision(); revision > m.mockFixtureRevision {+ m.mockFixtureRevision = revision+ }for _, host := range m.mockHostAgents {m.alertManager.HandleHostRemoved(host)}m.mockHostAgents = nil+ for _, host := range m.mockDockerHosts {+ m.alertManager.HandleDockerHostRemoved(host)+ }+ m.mockDockerHosts = nil}// checkMockHostAlerts applies the live host-agent boundary to a fixture. Andiff --git a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.goindex 7459a3c610..1cc2468142 100644--- a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go+++ b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go@@ -821,11 +821,11 @@ func TestMockHostAgentLeavingFixtureUsesRemovalLifecycle(t *testing.T) {memory := models.Memory{Total: 32 << 30, Used: 30 << 30, Free: 2 << 30, Usage: 93.75}kept := models.Host{ID: "host-linux-1", Hostname: "apollo-114", Status: "online", Memory: memory}- departed := models.Host{ID: "host-node-pve9", Hostname: "pve9", Status: "online", Memory: memory}+ departed := models.Host{ID: "host-node-pve9", Hostname: "pve9", LinkedNodeID: "mock-cluster-1-pve9", Status: "online", Memory: memory}- monitor.evaluateMockHostAgents([]models.Host{kept, departed}, nil)+ monitor.evaluateMockHostAgents([]models.Host{kept, departed}, nil, 1)// A runtime mock config change rebuilt the estate without pve9.- monitor.evaluateMockHostAgents([]models.Host{kept}, nil)+ monitor.evaluateMockHostAgents([]models.Host{kept}, nil, 2)keptAlert := falsefor _, alert := range manager.GetActiveAlerts() {@@ -840,8 +840,8 @@ func TestMockHostAgentLeavingFixtureUsesRemovalLifecycle(t *testing.T) {t.Fatal("agent still in the mock estate lost its memory alert")}- // The departed agent's hostname deduplication is released too, so the- // node it was linked to owns its metric alerts again.+ // The departed agent's node link is dropped too, so the node it was+ // linked to owns its metric alerts again.node := models.Node{ID: "mock-cluster-1-pve9",Name: "pve9",@@ -857,3 +857,30 @@ func TestMockHostAgentLeavingFixtureUsesRemovalLifecycle(t *testing.T) {}t.Fatal("node pve9 raised no memory alert after its agent left the mock estate")}++func TestMockDockerHostLeavingFixtureUsesRemovalLifecycle(t *testing.T) {+ mustSetMockEnabled(t, true)+ t.Cleanup(func() { mustSetMockEnabled(t, false) })+ manager := newMockHostAlertTestManager(t)+ monitor := &Monitor{alertManager: manager}++ kept := newMockDockerHostWithExitedContainers("nebula-1-mock", "nebula-1")+ // Nested Docker-in-LXC hosts are named after their guest's VMID, which a+ // runtime mock config change can shift.+ departed := newMockDockerHostWithExitedContainers("proxmox-lxc-docker:Production West:pve1:108", "pve1-ct108")++ monitor.evaluateMockDockerHosts([]models.DockerHost{kept, departed}, 1)+ monitor.evaluateMockDockerHosts([]models.DockerHost{kept, departed}, 1)+ if ids := dockerAlertIDsForHost(manager, departed.ID); len(ids) != len(departed.Containers) {+ t.Fatalf("departing host opened %v, want one alert per exited container", ids)+ }++ monitor.evaluateMockDockerHosts([]models.DockerHost{kept}, 2)++ if ids := dockerAlertIDsForHost(manager, departed.ID); len(ids) != 0 {+ t.Fatalf("Docker host that left the mock estate kept alerts %v", ids)+ }+ if ids := dockerAlertIDsForHost(manager, kept.ID); len(ids) != len(kept.Containers) {+ t.Fatalf("Docker host still in the mock estate has alerts %v, want one per exited container", ids)+ }+}diff --git a/internal/monitoring/monitor_mock_alerts_test.go b/internal/monitoring/monitor_mock_alerts_test.goindex 66e69d616a..a1c37dc3e8 100644--- a/internal/monitoring/monitor_mock_alerts_test.go+++ b/internal/monitoring/monitor_mock_alerts_test.go@@ -64,6 +64,7 @@ func newMockHostAlertTestManager(t *testing.T) *alerts.Manager {config.TimeThresholds = map[string]int{}config.MetricTimeThresholds = nilconfig.BackupDefaults.Enabled = false+ config.DockerDefaults.StateDisableConnectivity = falseconfig.AgentDefaults.Memory = &alerts.HysteresisThreshold{Trigger: 1, Clear: 0.5}config.NodeDefaults.Memory = &alerts.HysteresisThreshold{Trigger: 1, Clear: 0.5}manager.UpdateConfig(config)@@ -109,12 +110,12 @@ func TestCheckMockAlertsEvaluatesHostAgentsBeforeLinkedNodes(t *testing.T) {}// A node whose linked agent reported on the same pass must hand its- // memory alert to the agent on the first tick, as hostname+ // memory alert to the agent on the first tick, as node-link// deduplication does for live agents.linkedNodes := 0for _, node := range state.Nodes {host, ok := onlineHosts[strings.TrimSpace(node.LinkedAgentID)]- if !ok || !strings.EqualFold(host.Hostname, node.Name) {+ if !ok || strings.TrimSpace(host.LinkedNodeID) != node.ID {continue}linkedNodes++@@ -165,7 +166,7 @@ func TestCheckMockHostAlertsUsesHostLifecycleForOfflineFixtures(t *testing.T) {}}-func TestLeavingMockModeReleasesFixtureAgentHostnames(t *testing.T) {+func TestLeavingMockModeReleasesFixtureAgentNodeLinks(t *testing.T) {setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)manager := newMockHostAlertTestManager(t)monitor := &Monitor{@@ -177,21 +178,22 @@ func TestLeavingMockModeReleasesFixtureAgentHostnames(t *testing.T) {t.Cleanup(func() { mustSetMockEnabled(t, false) })monitor.checkMockAlerts()- registeredPVE1 := false+ linkedNodeID := ""for _, host := range mock.CurrentFixtureGraph().State.Hosts {- if host.Hostname == "pve1" && !strings.EqualFold(host.Status, "offline") {- registeredPVE1 = true+ if host.LinkedNodeID != "" && !strings.EqualFold(host.Status, "offline") {+ linkedNodeID = host.LinkedNodeID+ break}}- if !registeredPVE1 {- t.Fatal("mock fixture has no online agent named pve1")+ if linkedNodeID == "" {+ t.Fatal("mock fixture has no online agent linked to a node")}mustSetMonitorMockMode(t, monitor, false)- // A real node named like a fixture agent, with no agent of its own,+ // A node carrying the linked fixture node's ID, with no agent of its own,// must own its metric alerts once the monitor shows live data again.node := models.Node{- ID: "real-pve1",+ ID: linkedNodeID,Name: "pve1",Instance: "real",Status: "online",@@ -203,7 +205,7 @@ func TestLeavingMockModeReleasesFixtureAgentHostnames(t *testing.T) {return}}- t.Fatal("real node pve1 raised no memory alert: the fixture agent's hostname deduplication outlived mock mode")+ t.Fatalf("node %s raised no memory alert: a fixture agent's node link outlived mock mode", node.ID)}func TestMockHostAgentPassAfterLeavingMockModeRegistersNothing(t *testing.T) {@@ -212,14 +214,15 @@ func TestMockHostAgentPassAfterLeavingMockModeRegistersNothing(t *testing.T) {mustSetMockEnabled(t, false)// A pass whose snapshot predates leaving mock mode reaches the agent step- // after forgetMockHostAgents ran.+ // after forgetMockFixtureHosts ran.stale := models.Host{- ID: "host-node-mock-cluster-1-pve1",- Hostname: "pve1",- Status: "online",- Memory: models.Memory{Total: 32 << 30, Used: 30 << 30, Free: 2 << 30, Usage: 93.75},+ ID: "host-node-mock-cluster-1-pve1",+ Hostname: "pve1",+ LinkedNodeID: "real-pve1",+ Status: "online",+ Memory: models.Memory{Total: 32 << 30, Used: 30 << 30, Free: 2 << 30, Usage: 93.75},}- monitor.evaluateMockHostAgents([]models.Host{stale}, nil)+ monitor.evaluateMockHostAgents([]models.Host{stale}, nil, 1)if active := manager.GetActiveAlerts(); len(active) != 0 {t.Fatalf("stale mock pass opened %d alerts in live mode", len(active))@@ -237,5 +240,117 @@ func TestMockHostAgentPassAfterLeavingMockModeRegistersNothing(t *testing.T) {return}}- t.Fatal("stale mock pass registered fixture hostname pve1 and suppressed the real node's memory alert")+ t.Fatal("stale mock pass linked a fixture agent to real-pve1 and suppressed its memory alert")+}++// newMockDockerHostWithExitedContainers returns a reporting Docker host whose+// containers exited, so two evaluation passes open one alert per container.+func newMockDockerHostWithExitedContainers(id, hostname string) models.DockerHost {+ return models.DockerHost{+ ID: id,+ Hostname: hostname,+ DisplayName: hostname,+ Status: "online",+ Containers: []models.DockerContainer{+ {ID: id + "-web", Name: "web", State: "exited", Status: "Exited (137)"},+ {ID: id + "-db", Name: "db", State: "exited", Status: "Exited (137)"},+ },+ }+}++func dockerAlertIDsForHost(manager *alerts.Manager, hostID string) []string {+ var ids []string+ for _, alert := range manager.GetActiveAlerts() {+ if alert.ResourceID == "docker:"+hostID || strings.HasPrefix(alert.ResourceID, "docker:"+hostID+"/") {+ ids = append(ids, alert.ID)+ }+ }+ return ids+}++func TestMockDockerPassAfterLeavingMockModeRaisesNothing(t *testing.T) {+ manager := newMockHostAlertTestManager(t)+ monitor := &Monitor{alertManager: manager}+ mustSetMockEnabled(t, false)++ // A pass whose snapshot predates leaving mock mode reaches the Docker+ // step after forgetMockFixtureHosts ran.+ host := newMockDockerHostWithExitedContainers("nebula-1-mock", "nebula-1")+ monitor.evaluateMockDockerHosts([]models.DockerHost{host}, 1)+ monitor.evaluateMockDockerHosts([]models.DockerHost{host}, 1)++ if ids := dockerAlertIDsForHost(manager, host.ID); len(ids) != 0 {+ t.Fatalf("stale mock pass opened Docker alerts in live mode: %v", ids)+ }+}++func TestLeavingMockModeRemovesFixtureDockerHostAlerts(t *testing.T) {+ mustSetMockEnabled(t, true)+ t.Cleanup(func() { mustSetMockEnabled(t, false) })+ manager := newMockHostAlertTestManager(t)+ monitor := &Monitor{alertManager: manager}++ host := newMockDockerHostWithExitedContainers("nebula-1-mock", "nebula-1")+ monitor.evaluateMockDockerHosts([]models.DockerHost{host}, 1)+ monitor.evaluateMockDockerHosts([]models.DockerHost{host}, 1)+ if ids := dockerAlertIDsForHost(manager, host.ID); len(ids) != len(host.Containers) {+ t.Fatalf("fixture Docker host opened %v, want one alert per exited container", ids)+ }++ // The alerts stand in for ones a pass in flight recreated after+ // SetMockMode(false) cleared the manager.+ mustSetMockEnabled(t, false)+ monitor.forgetMockFixtureHosts()++ if ids := dockerAlertIDsForHost(manager, host.ID); len(ids) != 0 {+ t.Fatalf("fixture Docker host alerts outlived mock mode: %v", ids)+ }+}++func TestMockFixturePassFromBeforeAnEstateRebuildIsRejected(t *testing.T) {+ mustSetMockEnabled(t, true)+ t.Cleanup(func() { mustSetMockEnabled(t, false) })+ manager := newMockHostAlertTestManager(t)+ monitor := &Monitor{alertManager: manager}++ retired := newMockDockerHostWithExitedContainers("proxmox-lxc-docker:Production West:pve1:108", "pve1-ct108")+ current := newMockDockerHostWithExitedContainers("proxmox-lxc-docker:Production West:pve1:105", "pve1-ct105")+ monitor.evaluateMockDockerHosts([]models.DockerHost{current}, 2)+ monitor.evaluateMockDockerHosts([]models.DockerHost{current}, 2)++ // A slow pass whose snapshot predates the rebuild reaches the lock after+ // the newer passes. Applying it would remove the current host and+ // re-evaluate the retired one.+ monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, 1)+ monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, 1)++ if ids := dockerAlertIDsForHost(manager, current.ID); len(ids) != len(current.Containers) {+ t.Fatalf("superseded pass disturbed the current host's alerts: %v", ids)+ }+ if ids := dockerAlertIDsForHost(manager, retired.ID); len(ids) != 0 {+ t.Fatalf("superseded pass re-evaluated a retired host: %v", ids)+ }+}++func TestMockFixturePassPausedAcrossDisableAndReenableIsRejected(t *testing.T) {+ mustSetMockEnabled(t, true)+ t.Cleanup(func() { mustSetMockEnabled(t, false) })+ manager := newMockHostAlertTestManager(t)+ monitor := &Monitor{alertManager: manager}++ retired := newMockDockerHostWithExitedContainers("proxmox-lxc-docker:Production West:pve1:108", "pve1-ct108")+ _, staleRevision := mock.CurrentFixtureGraphWithRevision()+ monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, staleRevision)++ // Mock mode goes off and back on while a pass holding the old snapshot+ // is paused; it resumes before any pass of the rebuilt estate applies.+ mustSetMockEnabled(t, false)+ monitor.forgetMockFixtureHosts()+ mustSetMockEnabled(t, true)+ monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, staleRevision)+ monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, staleRevision)++ if ids := dockerAlertIDsForHost(manager, retired.ID); len(ids) != 0 {+ t.Fatalf("pass from before the disable re-evaluated the old estate: %v", ids)+ }}