Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Remove departed mock Docker hosts from alert evaluation

pulse-triage[bot]b05b8f352764
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index de0ce3e17d..1de6d23f3e 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -8308,8 +8308,12 @@ cannot become an accidental re-enrollment transition.
mock alert pass evaluated. When a runtime mock config change drops an agent
from the estate, `evaluateMockHostAgents` routes it through the alert
manager's `HandleHostRemoved`, the same alert boundary a deleted live agent
-crosses. Leaving mock mode routes every fixture agent through the same call,
-which also releases their hostname deduplication. Neither path writes a
+crosses. Fixture Docker hosts that leave the estate go through
+`HandleDockerHostRemoved` the same way. Leaving mock mode routes every fixture
+agent and Docker host through those calls, which also drops the agents' node
+links. The monitor also records the fixture's structural revision so a mock
+pass that predates an estate rebuild, or a disable and re-enable, cannot
+re-run that removal against the current fixture agents. Neither path writes a
removal tombstone, revokes a token, touches continuity evidence or admits a
report: fixture agents have no credentials or durable identity, and real
reports stay discarded while mock mode is on.
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 1cc77d03ff..3d2a0c2586 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -1732,6 +1732,21 @@ supporting inventory rather than a new batch of independent exits, so the
confirmed host incident clears child alerts instead of producing one alert per
container.
+Mock Docker hosts also keep the live removal boundary. Nested Docker-in-LXC
+hosts are named after their guest's VMID, so a runtime mock config change can
+drop one from the estate; `evaluateMockDockerHosts` then routes it through
+`HandleDockerHostRemoved`, as a deleted live host is. `pruneStaleDockerAlerts`
+clears the same alerts, but only on a state read or subscribed broadcast, so an
+unwatched stack would otherwise keep them active for `/api/alerts/active`,
+Patrol and escalations. The Docker pass shares the host-agent pass's lock and
+mock-mode check, and leaving mock mode removes the fixture Docker hosts too, so
+a pass in flight cannot leave Docker alerts behind in live mode.
+`TestMockDockerHostLeavingFixtureUsesRemovalLifecycle`,
+`TestMockDockerPassAfterLeavingMockModeRaisesNothing`,
+`TestLeavingMockModeRemovesFixtureDockerHostAlerts`,
+`TestMockFixturePassFromBeforeAnEstateRebuildIsRejected` and
+`TestMockFixturePassPausedAcrossDisableAndReenableIsRejected` pin this.
+
Mock alert evaluation covers host agents the same way. Live agents are
evaluated by `CheckHost` as each report lands. Mock mode discards reports, and
the `evaluateHostAgents` health sweep reads monitor state, which never holds
@@ -1746,17 +1761,23 @@ boundaries:
report and would mark the host online.
- Agents are evaluated before nodes, so a node with an online linked agent
hands its CPU, memory and disk alerts to the agent on the first tick, as
- host-agent hostname deduplication does in production.
+ node-link deduplication does in production.
- A runtime mock config change rebuilds the estate. An agent that leaves it
goes through `HandleHostRemoved`, as a deleted live agent does, so its alerts
- and hostname deduplication do not outlive it.
+ and node link do not outlive it.
- Leaving mock mode routes the fixture agents through `HandleHostRemoved`.
- `ClearActiveAlerts` drops their alerts but not their hostname registrations,
- and a real node named like a fixture agent (`pve1`) would otherwise keep its
- metric alerts suppressed with no agent to own them. A pass whose snapshot
- predates the switch finds mock mode off under the same lock and evaluates
- nothing. Two overlapping passes can still evaluate an older snapshot after
- a newer one; the next pass reconciles the set again.
+ `ClearActiveAlerts` drops their alerts but not their node links, and a
+ leftover link would keep suppressing CPU, memory and disk alerts on any node
+ that later carries the linked fixture node's ID. The agent and Docker
+ steps of a pass whose snapshot predates the switch find mock mode off under
+ the same lock and evaluate nothing.
+- Ticks start passes concurrently, so a pass whose snapshot predates an
+ estate rebuild can reach the lock after a newer one. Each pass carries the
+ fixture's structural revision, and the agent and Docker steps of a pass
+ older than the last applied or disabled revision are skipped, so they
+ cannot remove the new estate's hosts or re-evaluate retired ones, even
+ across a disable and re-enable. The pass's other loops (guests, nodes,
+ storage, PBS, PMG) take no part in this tracking and are not guarded.
Fixture agents must carry readings a real agent could report. Mock Kubernetes
pods share 0.7 single-pod memory footprints per node, which keeps a node's pods
@@ -1774,7 +1795,7 @@ pass. Those are measurements, not test guarantees. The tests pin the
boundaries:
`TestCheckMockAlertsEvaluatesHostAgentsBeforeLinkedNodes`,
`TestCheckMockHostAlertsUsesHostLifecycleForOfflineFixtures`,
-`TestLeavingMockModeReleasesFixtureAgentHostnames` and
+`TestLeavingMockModeReleasesFixtureAgentNodeLinks` and
`TestMockHostAgentPassAfterLeavingMockModeRegistersNothing` in
`internal/monitoring/monitor_mock_alerts_test.go`,
`TestMockHostAgentLeavingFixtureUsesRemovalLifecycle` in
diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.go
index 65df2b3a2e..388a3e7c90 100644
--- a/internal/monitoring/monitor.go
+++ b/internal/monitoring/monitor.go
@@ -1303,6 +1303,8 @@ type Monitor struct {
mockMetricsCancel context.CancelFunc
mockMetricsWg sync.WaitGroup
mockHostAgentsMu sync.Mutex
+ mockDockerHosts map[string]models.DockerHost
+ mockFixtureRevision uint64
mockHostAgents map[string]models.Host // Fixture agents evaluated by the last mock alert pass
dockerChecker DockerChecker // Optional Docker checker for LXC containers
dockerCheckerConfiguredAt time.Time // Last time the Docker checker was configured
@@ -4628,7 +4630,7 @@ func (m *Monitor) SetMockMode(enable bool) error {
return err
}
m.alertManager.ClearActiveAlerts()
- m.forgetMockHostAgents()
+ m.forgetMockFixtureHosts()
m.mu.Lock()
m.resetStateLocked()
m.metricsHistory.Reset()
diff --git a/internal/monitoring/monitor_alerts.go b/internal/monitoring/monitor_alerts.go
index 89e870836f..bb581d4da0 100644
--- a/internal/monitoring/monitor_alerts.go
+++ b/internal/monitoring/monitor_alerts.go
@@ -635,7 +635,8 @@ func (m *Monitor) checkMockAlerts() {
}
// Get mock state
- state := mock.CurrentFixtureGraph().State
+ graph, fixtureRevision := mock.CurrentFixtureGraphWithRevision()
+ state := graph.State
log.Debug().
Int("vms", len(state.VMs)).
@@ -696,10 +697,10 @@ func (m *Monitor) checkMockAlerts() {
// the monitor state the host health sweep reads, so without this pass no
// agent CPU, memory, disk, temperature or offline alert can open against
// mock data. Agents go before nodes so a linked node hands its CPU, memory
- // and disk alerts to its agent on the first tick, as the host-agent
- // hostname deduplication does in production.
+ // and disk alerts to its agent on the first tick, as node-link
+ // deduplication does in production.
log.Debug().Int("hostCount", len(state.Hosts)).Msg("checking host agent alerts")
- m.evaluateMockHostAgents(state.Hosts, state.Nodes)
+ m.evaluateMockHostAgents(state.Hosts, state.Nodes, fixtureRevision)
// Check alerts for each node
for _, node := range state.Nodes {
@@ -733,9 +734,7 @@ func (m *Monitor) checkMockAlerts() {
// so skipping this loop leaves the docker alert lifecycle unexercisable
// against mock data.
log.Debug().Int("dockerHostCount", len(state.DockerHosts)).Msg("checking docker alerts")
- for _, dockerHost := range state.DockerHosts {
- m.checkMockDockerHostAlerts(dockerHost)
- }
+ m.evaluateMockDockerHosts(state.DockerHosts, fixtureRevision)
// Cache the latest alert snapshots directly in the mock data so the API can serve
// mock state without needing to grab the alert manager lock again.
@@ -745,14 +744,12 @@ func (m *Monitor) checkMockAlerts() {
// evaluateMockHostAgents evaluates every fixture agent and remembers the set.
// A runtime mock config change rebuilds the estate, so an agent can leave it
// between passes; it then goes through HandleHostRemoved, as a deleted live
-// agent does, or its alerts and hostname deduplication would outlive it.
-func (m *Monitor) evaluateMockHostAgents(hosts []models.Host, nodes []models.Node) {
+// agent does, or its alerts and node link would outlive it.
+func (m *Monitor) evaluateMockHostAgents(hosts []models.Host, nodes []models.Node, fixtureRevision uint64) {
m.mockHostAgentsMu.Lock()
defer m.mockHostAgentsMu.Unlock()
- // A pass that took its snapshot before the monitor left mock mode must
- // not re-register fixture agents after forgetMockHostAgents ran.
- if !mock.IsMockEnabled() {
+ if !m.acceptMockFixturePassLocked(fixtureRevision) {
return
}
@@ -762,8 +759,6 @@ func (m *Monitor) evaluateMockHostAgents(hosts []models.Host, nodes []models.Nod
current[host.ID] = host
}
}
- // Remove first so a departed agent cannot unregister a hostname that a
- // remaining agent registers on this same pass.
for id, host := range m.mockHostAgents {
if _, ok := current[id]; !ok {
m.alertManager.HandleHostRemoved(host)
@@ -775,20 +770,77 @@ func (m *Monitor) evaluateMockHostAgents(hosts []models.Host, nodes []models.Nod
m.mockHostAgents = current
}
-// forgetMockHostAgents removes the fixture agents when the monitor leaves mock
-// mode. ClearActiveAlerts drops their alerts but not their hostname
-// deduplication, so a real node named like a fixture agent (pve1) would keep
-// its CPU, memory and disk alerts suppressed with no agent to own them. The
-// removal also clears anything a pass already in flight recreated; mock mode
-// is off before this runs, so no later pass can register them again.
-func (m *Monitor) forgetMockHostAgents() {
+// evaluateMockDockerHosts is the Docker counterpart of evaluateMockHostAgents,
+// under the same lock and mode check. Nested Docker-in-LXC hosts are named
+// after their guest's VMID, so a runtime mock config change can drop a host
+// from the estate; it then goes through HandleDockerHostRemoved, as a deleted
+// live host does. pruneStaleDockerAlerts clears the same alerts, but only when
+// something reads state, so an unwatched stack would keep them active.
+func (m *Monitor) evaluateMockDockerHosts(hosts []models.DockerHost, fixtureRevision uint64) {
+ m.mockHostAgentsMu.Lock()
+ defer m.mockHostAgentsMu.Unlock()
+
+ if !m.acceptMockFixturePassLocked(fixtureRevision) {
+ return
+ }
+
+ current := make(map[string]models.DockerHost, len(hosts))
+ for _, host := range hosts {
+ if host.ID != "" {
+ current[host.ID] = host
+ }
+ }
+ for id, host := range m.mockDockerHosts {
+ if _, ok := current[id]; !ok {
+ m.alertManager.HandleDockerHostRemoved(host)
+ }
+ }
+ for _, host := range hosts {
+ m.checkMockDockerHostAlerts(host)
+ }
+ m.mockDockerHosts = current
+}
+
+// acceptMockFixturePassLocked reports whether a mock alert pass may update the
+// tracked fixture hosts. A pass that took its snapshot before the monitor left
+// mock mode must not evaluate fixtures after forgetMockFixtureHosts ran. Ticks
+// start passes concurrently, so one that took its snapshot before an estate
+// rebuild can also reach this lock after a newer pass; applying it would
+// remove the new estate's hosts and re-evaluate retired ones. The fixture
+// revision only advances on such rebuilds, so an older revision is rejected.
+// Callers hold mockHostAgentsMu.
+func (m *Monitor) acceptMockFixturePassLocked(fixtureRevision uint64) bool {
+ if !mock.IsMockEnabled() || fixtureRevision < m.mockFixtureRevision {
+ return false
+ }
+ m.mockFixtureRevision = fixtureRevision
+ return true
+}
+
+// forgetMockFixtureHosts removes the fixture agents and Docker hosts when the
+// monitor leaves mock mode. ClearActiveAlerts drops their alerts but not the
+// agents' node links, and a leftover link would keep suppressing CPU, memory
+// and disk alerts on any node that later carries the linked fixture node's ID.
+// The removal also clears anything a pass already in flight
+// recreated; mock mode is off before this runs, so no later pass can evaluate
+// them again while it stays off. Disabling advanced the fixture revision, and
+// recording it here keeps a pass paused across a disable and re-enable from
+// evaluating the old estate once mock mode is back on.
+func (m *Monitor) forgetMockFixtureHosts() {
m.mockHostAgentsMu.Lock()
defer m.mockHostAgentsMu.Unlock()
+ if _, revision := mock.CurrentFixtureGraphWithRevision(); revision > m.mockFixtureRevision {
+ m.mockFixtureRevision = revision
+ }
for _, host := range m.mockHostAgents {
m.alertManager.HandleHostRemoved(host)
}
m.mockHostAgents = nil
+ for _, host := range m.mockDockerHosts {
+ m.alertManager.HandleDockerHostRemoved(host)
+ }
+ m.mockDockerHosts = nil
}
// checkMockHostAlerts applies the live host-agent boundary to a fixture. An
diff --git a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go
index 7459a3c610..1cc2468142 100644
--- a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go
+++ b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go
@@ -821,11 +821,11 @@ func TestMockHostAgentLeavingFixtureUsesRemovalLifecycle(t *testing.T) {
memory := models.Memory{Total: 32 << 30, Used: 30 << 30, Free: 2 << 30, Usage: 93.75}
kept := models.Host{ID: "host-linux-1", Hostname: "apollo-114", Status: "online", Memory: memory}
- departed := models.Host{ID: "host-node-pve9", Hostname: "pve9", Status: "online", Memory: memory}
+ departed := models.Host{ID: "host-node-pve9", Hostname: "pve9", LinkedNodeID: "mock-cluster-1-pve9", Status: "online", Memory: memory}
- monitor.evaluateMockHostAgents([]models.Host{kept, departed}, nil)
+ monitor.evaluateMockHostAgents([]models.Host{kept, departed}, nil, 1)
// A runtime mock config change rebuilt the estate without pve9.
- monitor.evaluateMockHostAgents([]models.Host{kept}, nil)
+ monitor.evaluateMockHostAgents([]models.Host{kept}, nil, 2)
keptAlert := false
for _, alert := range manager.GetActiveAlerts() {
@@ -840,8 +840,8 @@ func TestMockHostAgentLeavingFixtureUsesRemovalLifecycle(t *testing.T) {
t.Fatal("agent still in the mock estate lost its memory alert")
}
- // The departed agent's hostname deduplication is released too, so the
- // node it was linked to owns its metric alerts again.
+ // The departed agent's node link is dropped too, so the node it was
+ // linked to owns its metric alerts again.
node := models.Node{
ID: "mock-cluster-1-pve9",
Name: "pve9",
@@ -857,3 +857,30 @@ func TestMockHostAgentLeavingFixtureUsesRemovalLifecycle(t *testing.T) {
}
t.Fatal("node pve9 raised no memory alert after its agent left the mock estate")
}
+
+func TestMockDockerHostLeavingFixtureUsesRemovalLifecycle(t *testing.T) {
+ mustSetMockEnabled(t, true)
+ t.Cleanup(func() { mustSetMockEnabled(t, false) })
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{alertManager: manager}
+
+ kept := newMockDockerHostWithExitedContainers("nebula-1-mock", "nebula-1")
+ // Nested Docker-in-LXC hosts are named after their guest's VMID, which a
+ // runtime mock config change can shift.
+ departed := newMockDockerHostWithExitedContainers("proxmox-lxc-docker:Production West:pve1:108", "pve1-ct108")
+
+ monitor.evaluateMockDockerHosts([]models.DockerHost{kept, departed}, 1)
+ monitor.evaluateMockDockerHosts([]models.DockerHost{kept, departed}, 1)
+ if ids := dockerAlertIDsForHost(manager, departed.ID); len(ids) != len(departed.Containers) {
+ t.Fatalf("departing host opened %v, want one alert per exited container", ids)
+ }
+
+ monitor.evaluateMockDockerHosts([]models.DockerHost{kept}, 2)
+
+ if ids := dockerAlertIDsForHost(manager, departed.ID); len(ids) != 0 {
+ t.Fatalf("Docker host that left the mock estate kept alerts %v", ids)
+ }
+ if ids := dockerAlertIDsForHost(manager, kept.ID); len(ids) != len(kept.Containers) {
+ t.Fatalf("Docker host still in the mock estate has alerts %v, want one per exited container", ids)
+ }
+}
diff --git a/internal/monitoring/monitor_mock_alerts_test.go b/internal/monitoring/monitor_mock_alerts_test.go
index 66e69d616a..a1c37dc3e8 100644
--- a/internal/monitoring/monitor_mock_alerts_test.go
+++ b/internal/monitoring/monitor_mock_alerts_test.go
@@ -64,6 +64,7 @@ func newMockHostAlertTestManager(t *testing.T) *alerts.Manager {
config.TimeThresholds = map[string]int{}
config.MetricTimeThresholds = nil
config.BackupDefaults.Enabled = false
+ config.DockerDefaults.StateDisableConnectivity = false
config.AgentDefaults.Memory = &alerts.HysteresisThreshold{Trigger: 1, Clear: 0.5}
config.NodeDefaults.Memory = &alerts.HysteresisThreshold{Trigger: 1, Clear: 0.5}
manager.UpdateConfig(config)
@@ -109,12 +110,12 @@ func TestCheckMockAlertsEvaluatesHostAgentsBeforeLinkedNodes(t *testing.T) {
}
// A node whose linked agent reported on the same pass must hand its
- // memory alert to the agent on the first tick, as hostname
+ // memory alert to the agent on the first tick, as node-link
// deduplication does for live agents.
linkedNodes := 0
for _, node := range state.Nodes {
host, ok := onlineHosts[strings.TrimSpace(node.LinkedAgentID)]
- if !ok || !strings.EqualFold(host.Hostname, node.Name) {
+ if !ok || strings.TrimSpace(host.LinkedNodeID) != node.ID {
continue
}
linkedNodes++
@@ -165,7 +166,7 @@ func TestCheckMockHostAlertsUsesHostLifecycleForOfflineFixtures(t *testing.T) {
}
}
-func TestLeavingMockModeReleasesFixtureAgentHostnames(t *testing.T) {
+func TestLeavingMockModeReleasesFixtureAgentNodeLinks(t *testing.T) {
setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
manager := newMockHostAlertTestManager(t)
monitor := &Monitor{
@@ -177,21 +178,22 @@ func TestLeavingMockModeReleasesFixtureAgentHostnames(t *testing.T) {
t.Cleanup(func() { mustSetMockEnabled(t, false) })
monitor.checkMockAlerts()
- registeredPVE1 := false
+ linkedNodeID := ""
for _, host := range mock.CurrentFixtureGraph().State.Hosts {
- if host.Hostname == "pve1" && !strings.EqualFold(host.Status, "offline") {
- registeredPVE1 = true
+ if host.LinkedNodeID != "" && !strings.EqualFold(host.Status, "offline") {
+ linkedNodeID = host.LinkedNodeID
+ break
}
}
- if !registeredPVE1 {
- t.Fatal("mock fixture has no online agent named pve1")
+ if linkedNodeID == "" {
+ t.Fatal("mock fixture has no online agent linked to a node")
}
mustSetMonitorMockMode(t, monitor, false)
- // A real node named like a fixture agent, with no agent of its own,
+ // A node carrying the linked fixture node's ID, with no agent of its own,
// must own its metric alerts once the monitor shows live data again.
node := models.Node{
- ID: "real-pve1",
+ ID: linkedNodeID,
Name: "pve1",
Instance: "real",
Status: "online",
@@ -203,7 +205,7 @@ func TestLeavingMockModeReleasesFixtureAgentHostnames(t *testing.T) {
return
}
}
- t.Fatal("real node pve1 raised no memory alert: the fixture agent's hostname deduplication outlived mock mode")
+ t.Fatalf("node %s raised no memory alert: a fixture agent's node link outlived mock mode", node.ID)
}
func TestMockHostAgentPassAfterLeavingMockModeRegistersNothing(t *testing.T) {
@@ -212,14 +214,15 @@ func TestMockHostAgentPassAfterLeavingMockModeRegistersNothing(t *testing.T) {
mustSetMockEnabled(t, false)
// A pass whose snapshot predates leaving mock mode reaches the agent step
- // after forgetMockHostAgents ran.
+ // after forgetMockFixtureHosts ran.
stale := models.Host{
- ID: "host-node-mock-cluster-1-pve1",
- Hostname: "pve1",
- Status: "online",
- Memory: models.Memory{Total: 32 << 30, Used: 30 << 30, Free: 2 << 30, Usage: 93.75},
+ ID: "host-node-mock-cluster-1-pve1",
+ Hostname: "pve1",
+ LinkedNodeID: "real-pve1",
+ Status: "online",
+ Memory: models.Memory{Total: 32 << 30, Used: 30 << 30, Free: 2 << 30, Usage: 93.75},
}
- monitor.evaluateMockHostAgents([]models.Host{stale}, nil)
+ monitor.evaluateMockHostAgents([]models.Host{stale}, nil, 1)
if active := manager.GetActiveAlerts(); len(active) != 0 {
t.Fatalf("stale mock pass opened %d alerts in live mode", len(active))
@@ -237,5 +240,117 @@ func TestMockHostAgentPassAfterLeavingMockModeRegistersNothing(t *testing.T) {
return
}
}
- t.Fatal("stale mock pass registered fixture hostname pve1 and suppressed the real node's memory alert")
+ t.Fatal("stale mock pass linked a fixture agent to real-pve1 and suppressed its memory alert")
+}
+
+// newMockDockerHostWithExitedContainers returns a reporting Docker host whose
+// containers exited, so two evaluation passes open one alert per container.
+func newMockDockerHostWithExitedContainers(id, hostname string) models.DockerHost {
+ return models.DockerHost{
+ ID: id,
+ Hostname: hostname,
+ DisplayName: hostname,
+ Status: "online",
+ Containers: []models.DockerContainer{
+ {ID: id + "-web", Name: "web", State: "exited", Status: "Exited (137)"},
+ {ID: id + "-db", Name: "db", State: "exited", Status: "Exited (137)"},
+ },
+ }
+}
+
+func dockerAlertIDsForHost(manager *alerts.Manager, hostID string) []string {
+ var ids []string
+ for _, alert := range manager.GetActiveAlerts() {
+ if alert.ResourceID == "docker:"+hostID || strings.HasPrefix(alert.ResourceID, "docker:"+hostID+"/") {
+ ids = append(ids, alert.ID)
+ }
+ }
+ return ids
+}
+
+func TestMockDockerPassAfterLeavingMockModeRaisesNothing(t *testing.T) {
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{alertManager: manager}
+ mustSetMockEnabled(t, false)
+
+ // A pass whose snapshot predates leaving mock mode reaches the Docker
+ // step after forgetMockFixtureHosts ran.
+ host := newMockDockerHostWithExitedContainers("nebula-1-mock", "nebula-1")
+ monitor.evaluateMockDockerHosts([]models.DockerHost{host}, 1)
+ monitor.evaluateMockDockerHosts([]models.DockerHost{host}, 1)
+
+ if ids := dockerAlertIDsForHost(manager, host.ID); len(ids) != 0 {
+ t.Fatalf("stale mock pass opened Docker alerts in live mode: %v", ids)
+ }
+}
+
+func TestLeavingMockModeRemovesFixtureDockerHostAlerts(t *testing.T) {
+ mustSetMockEnabled(t, true)
+ t.Cleanup(func() { mustSetMockEnabled(t, false) })
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{alertManager: manager}
+
+ host := newMockDockerHostWithExitedContainers("nebula-1-mock", "nebula-1")
+ monitor.evaluateMockDockerHosts([]models.DockerHost{host}, 1)
+ monitor.evaluateMockDockerHosts([]models.DockerHost{host}, 1)
+ if ids := dockerAlertIDsForHost(manager, host.ID); len(ids) != len(host.Containers) {
+ t.Fatalf("fixture Docker host opened %v, want one alert per exited container", ids)
+ }
+
+ // The alerts stand in for ones a pass in flight recreated after
+ // SetMockMode(false) cleared the manager.
+ mustSetMockEnabled(t, false)
+ monitor.forgetMockFixtureHosts()
+
+ if ids := dockerAlertIDsForHost(manager, host.ID); len(ids) != 0 {
+ t.Fatalf("fixture Docker host alerts outlived mock mode: %v", ids)
+ }
+}
+
+func TestMockFixturePassFromBeforeAnEstateRebuildIsRejected(t *testing.T) {
+ mustSetMockEnabled(t, true)
+ t.Cleanup(func() { mustSetMockEnabled(t, false) })
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{alertManager: manager}
+
+ retired := newMockDockerHostWithExitedContainers("proxmox-lxc-docker:Production West:pve1:108", "pve1-ct108")
+ current := newMockDockerHostWithExitedContainers("proxmox-lxc-docker:Production West:pve1:105", "pve1-ct105")
+ monitor.evaluateMockDockerHosts([]models.DockerHost{current}, 2)
+ monitor.evaluateMockDockerHosts([]models.DockerHost{current}, 2)
+
+ // A slow pass whose snapshot predates the rebuild reaches the lock after
+ // the newer passes. Applying it would remove the current host and
+ // re-evaluate the retired one.
+ monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, 1)
+ monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, 1)
+
+ if ids := dockerAlertIDsForHost(manager, current.ID); len(ids) != len(current.Containers) {
+ t.Fatalf("superseded pass disturbed the current host's alerts: %v", ids)
+ }
+ if ids := dockerAlertIDsForHost(manager, retired.ID); len(ids) != 0 {
+ t.Fatalf("superseded pass re-evaluated a retired host: %v", ids)
+ }
+}
+
+func TestMockFixturePassPausedAcrossDisableAndReenableIsRejected(t *testing.T) {
+ mustSetMockEnabled(t, true)
+ t.Cleanup(func() { mustSetMockEnabled(t, false) })
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{alertManager: manager}
+
+ retired := newMockDockerHostWithExitedContainers("proxmox-lxc-docker:Production West:pve1:108", "pve1-ct108")
+ _, staleRevision := mock.CurrentFixtureGraphWithRevision()
+ monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, staleRevision)
+
+ // Mock mode goes off and back on while a pass holding the old snapshot
+ // is paused; it resumes before any pass of the rebuilt estate applies.
+ mustSetMockEnabled(t, false)
+ monitor.forgetMockFixtureHosts()
+ mustSetMockEnabled(t, true)
+ monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, staleRevision)
+ monitor.evaluateMockDockerHosts([]models.DockerHost{retired}, staleRevision)
+
+ if ids := dockerAlertIDsForHost(manager, retired.ID); len(ids) != 0 {
+ t.Fatalf("pass from before the disable re-evaluated the old estate: %v", ids)
+ }
}