Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Switch every running tenant monitor when mock mode flips

rcourtman27e440339b7a
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index e7b4f59ab3..7582e7db33 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -754,6 +754,15 @@ not fenced: a live report already past its mock-mode check when mock mode is
switched on can still be evaluated after the clear, and its alerts can persist
until mock mode is left.
+Every tenant monitor registers its own fixture agents, and their node links,
+in its own alert manager, so the toggle forgets the fixture agents of every
+running monitor, not only those of the monitor it was called through. A link
+left behind would keep owning its fixture node's usage alerts after mock mode
+ends. `TestLeavingMockModeReleasesFixtureAgentNodeLinksOnEveryRunningMonitor`
+in `internal/monitoring/monitor_host_agent_removal_lifecycle_test.go` switches
+through the default monitor and fails if the other tenant's node stays
+suppressed.
+
Physical-disk evidence collected by a host agent must survive projection back
into monitoring's models. Absent evidence has to carry its declared sentinel
rather than a zero value that reads as a real measurement: an absent
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 60ff94e27d..7e54def128 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -1855,11 +1855,33 @@ a live report or poll already past its mock-mode check when mock mode is
switched on can still evaluate after that clear, and its alerts can persist
until mock mode is left.
-The fence and the clear are per monitor while the mode is process-wide. A
-tenant monitor other than the one `SetMockMode` was called on neither drains
-nor clears, so fixture alerts it raised outlive the switch as they did before
-the fence; switching mock mode with several tenant monitors running is not
-covered.
+The mode is process-wide while each monitor keeps its own fence, alert manager,
+fixture agents and state, and the server runs one monitor per organization, so
+`SetMockMode` switches every running monitor, whichever tenant's monitor it is
+called on (the mock-mode API calls the requesting tenant's, the demo-fixture
+licence sync the default organization's). `Start` joins a process-wide set of
+running monitors under `mockModeSwitchMu`, and leaves it when its loop returns.
+A switch flips the mode once, then ends each monitor's epoch, waits for its
+admitted calls, clears its alerts, forgets its fixture agents when leaving mock
+mode and resets its state, and only then restarts the mode-dependent runtime of
+each monitor in the set. `Start` sets its runtime context before it joins and
+chooses the mock metrics sampler or discovery while it joins, under the same
+lock, so no switch lands between reading the mode and starting the runtime for
+it, and a switch that reaches a monitor not yet joined leaves the runtime to
+its `Start`. A monitor that missed switches between `New` and `Start` first
+leaves what it holds: `New` records the mode and the number of switches made so
+far, and `Start` compares both, so even a round trip that brought the flag back
+drops the fixture alerts a read path raised meanwhile, as well as the live
+alerts `New` restored outside mock mode. A monitor that is not running is
+switched only when the switch is made through it; a call for the mode the flag
+already holds still moves any reached monitor whose alerts belong to the other
+mode, and leaves the rest alone.
+
+Release builds reach several tenant monitors in mock mode only on an instance
+running `DemoMode` with the `demo_fixtures` entitlement (release
+`ValidateEnablement` refuses mock fixtures otherwise) and with more than one
+organization's monitor running; the licence sync switches only for the
+default organization.
The proofs park a real evaluation at a fixed point, switch modes and release
it. `TestLeavingMockModeRefusesTheRestOfAnInFlightPass` holds a mock pass at its
@@ -1889,7 +1911,25 @@ previous-state carry returns fixture nodes or guests. All eight are in
`TestStoreRefreshTakesItsMockModeScopeBeforeReadingState` in
`internal/monitoring/canonical_guardrails_test.go` rejects a call that reads
state in an argument ahead of `m.mockModeFence.begin()` and pins the
-end-epoch-then-clear order in `SetMockMode`.
+end-epoch-then-clear order in `endMockModeEpoch`, which `SetMockMode` runs for
+every monitor it reaches.
+
+`TestLeavingMockModeClearsEveryRunningTenantMonitor` and
+`TestEnteringMockModeClearsEveryRunningTenantMonitor` start the default and one
+other tenant monitor through `MultiTenantMonitor`, wait until both have joined,
+raise alerts in both, switch through one of them and check both. Against a
+switch that reached only its own monitor, the default monitor kept its fixture
+alerts (69 to 104 across runs) and all 69 fixture agents after leaving mock
+mode through the other tenant, and that tenant kept its live alert after mock
+mode was entered through the default monitor. `TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn`
+fails if `Start` joins without leaving the mode its monitor was built in, and
+`TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` fails if a call for
+the flag's current mode returns without moving such a monitor, and
+`TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` fails if a switch
+starts the runtime of a monitor whose `Start` has not joined, and
+`TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile` fails if
+`Start` compares only the mode. All six are in
+`internal/monitoring/monitor_mock_alerts_test.go`.
The disk evaluation runs on the mock alert tick, not the poller's disk
interval (five minutes by default, `PhysicalDiskPollingMinutes` per instance).
diff --git a/internal/monitoring/canonical_guardrails_test.go b/internal/monitoring/canonical_guardrails_test.go
index fd27c2215b..8a13f8b631 100644
--- a/internal/monitoring/canonical_guardrails_test.go
+++ b/internal/monitoring/canonical_guardrails_test.go
@@ -543,8 +543,12 @@ func TestStoreRefreshTakesItsMockModeScopeBeforeReadingState(t *testing.T) {
if err != nil {
t.Fatalf("failed to read monitor.go: %v", err)
}
- if got := strings.Count(string(data), "m.mockModeFence.advance()\n\t\tm.alertManager.ClearActiveAlerts()"); got != 2 {
- t.Fatalf("SetMockMode must end the mock-mode epoch immediately before clearing alerts in both directions, found %d", got)
+ source := string(data)
+ if got := strings.Count(source, "m.mockModeFence.advance()\n\tm.alertManager.ClearActiveAlerts()"); got != 1 {
+ t.Fatalf("endMockModeEpoch must end the mock-mode epoch immediately before clearing alerts, found %d", got)
+ }
+ if !strings.Contains(source, "for _, monitor := range monitors {\n\t\tmonitor.endMockModeEpoch(enable)\n\t}") {
+ t.Fatal("SetMockMode must end the epoch of every monitor the switch reaches")
}
}
@@ -1664,16 +1668,11 @@ func TestMonitorSetMockModeAuthorizesBeforeResettingRuntimeState(t *testing.T) {
if err != nil {
t.Fatalf("failed to read monitor.go: %v", err)
}
- source := string(data)
- requiredSnippets := []string{
- "if err := mock.SetEnabled(true); err != nil {",
- "if err := mock.SetEnabled(false); err != nil {",
- "return err",
- }
- for _, snippet := range requiredSnippets {
- if !strings.Contains(source, snippet) {
- t.Fatalf("monitor.go must contain %q", snippet)
- }
+ source := strings.Join(strings.Fields(string(data)), " ")
+ authorize := strings.Index(source, "if err := mock.SetEnabled(enable); err != nil { return err }")
+ reset := strings.Index(source, "monitor.endMockModeEpoch(enable)")
+ if authorize < 0 || reset < 0 || authorize > reset {
+ t.Fatal("SetMockMode must return the mock.SetEnabled error before it ends any monitor's epoch or resets its state")
}
}
diff --git a/internal/monitoring/mock_mode_switch.go b/internal/monitoring/mock_mode_switch.go
new file mode 100644
index 0000000000..4419f77b88
--- /dev/null
+++ b/internal/monitoring/mock_mode_switch.go
@@ -0,0 +1,75 @@
+package monitoring
+
+import (
+ "sync"
+ "sync/atomic"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/mock"
+)
+
+// Mock mode is process-wide (mock.SetEnabled flips one package flag), while
+// every Monitor keeps its own alert manager, fixture agents, state and
+// mock-mode fence, and the server runs one Monitor per organization. A switch
+// made through one monitor therefore has to reach every running monitor, or
+// the others keep what they raised in the mode the process left: fixture
+// alerts and fixture-agent node links in live mode, live alerts in mock mode,
+// until the 24-hour stale sweep.
+//
+// mockModeSwitchMu serializes switches and guards the set of running monitors
+// and each monitor's mockModeAligned and mockModeSwitchesSeen. A switch is
+// several steps per monitor (end the epoch, clear, reset), so two interleaved
+// switches could clear what the later one just admitted. Start joins the set
+// under the same lock, so a monitor joins either before a switch, which then
+// switches it with the others, or after it, and starts in the new mode.
+//
+// mockModeSwitchCount counts the flips SetMockMode made. A monitor that missed
+// flips while it was not running can hold the alerts and state of a mode the
+// process left even when a later flip brought the flag back to the mode it
+// records (a read path evaluated fixture data before Start), so Start
+// compares the count as well as the mode.
+var (
+ mockModeSwitchMu sync.Mutex
+ mockModeRunningMonitors = make(map[*Monitor]struct{})
+ mockModeSwitchCount atomic.Uint64
+)
+
+// mockModeSwitchTargetsLocked returns the monitors a switch made through m
+// reaches: m itself, which need not be running, and every running monitor.
+func mockModeSwitchTargetsLocked(m *Monitor) []*Monitor {
+ targets := make([]*Monitor, 0, len(mockModeRunningMonitors)+1)
+ targets = append(targets, m)
+ for monitor := range mockModeRunningMonitors {
+ if monitor != m {
+ targets = append(targets, monitor)
+ }
+ }
+ return targets
+}
+
+// joinMockModeSwitches adds m to the monitors every switch reaches, then runs
+// start with the current mode while no switch can run, so the mode-dependent
+// runtime start chooses (the mock metrics sampler or discovery) cannot be
+// overtaken by a switch. It returns the function that removes m again.
+//
+// A switch can land between New and Start, while m is not yet running. New
+// restores persisted alerts only outside mock mode, so m would then carry live
+// alerts into mock mode, or fixture alerts that a read path raised before
+// Start into live mode; m first leaves what it holds, as the switches it
+// missed would have done.
+func (m *Monitor) joinMockModeSwitches(start func(mockEnabled bool)) (leave func()) {
+ mockModeSwitchMu.Lock()
+ defer mockModeSwitchMu.Unlock()
+
+ enabled := mock.IsMockEnabled()
+ if m.mockModeAligned != enabled || m.mockModeSwitchesSeen != mockModeSwitchCount.Load() {
+ m.endMockModeEpoch(enabled)
+ }
+ mockModeRunningMonitors[m] = struct{}{}
+ start(enabled)
+
+ return func() {
+ mockModeSwitchMu.Lock()
+ defer mockModeSwitchMu.Unlock()
+ delete(mockModeRunningMonitors, m)
+ }
+}
diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.go
index 69c5c33f37..80fc4aa163 100644
--- a/internal/monitoring/monitor.go
+++ b/internal/monitoring/monitor.go
@@ -13,6 +13,7 @@ import (
"os"
"path/filepath"
"reflect"
+ "slices"
"sort"
"strings"
"sync"
@@ -67,9 +68,9 @@ func keepRealPollingInMockMode() bool {
// process that boots with mock mode already enabled never runs that path, so
// alerts raised against real infrastructure are restored from disk and served
// beside fixture data until something clears them.
-func alertManagerRestoreOptions() []alerts.ManagerOption {
+func alertManagerRestoreOptions(mockEnabled bool) []alerts.ManagerOption {
options := []alerts.ManagerOption{alerts.WithDurableAlertStore()}
- if mock.IsMockEnabled() {
+ if mockEnabled {
options = append(options, alerts.WithoutPersistedAlertRestore())
}
return options
@@ -1283,6 +1284,8 @@ type Monitor struct {
mockHostAgentsMu sync.Mutex
mockHostAgents map[string]models.Host // Fixture agents evaluated by the last mock alert pass
mockModeFence mockModeFence // Keeps mode-dependent alert evaluations inside the epoch they read in
+ mockModeAligned bool // Mock mode the alerts and state belong to; guarded by mockModeSwitchMu
+ mockModeSwitchesSeen uint64 // mockModeSwitchCount when they were last aligned; guarded by mockModeSwitchMu
dockerChecker DockerChecker // Optional Docker checker for LXC containers
dockerCheckerConfiguredAt time.Time // Last time the Docker checker was configured
dockerCheckAllowedVMIDs map[int]struct{} // Optional VMID allowlist gating the LXC Docker socket probe; empty means all guests
@@ -1771,6 +1774,12 @@ func New(cfg *config.Config) (*Monitor, error) {
DataDir: cfg.DataPath,
})
+ // One read decides the alert restore and records the mode it belongs to,
+ // so Start can tell whether a mock-mode switch landed in between. The
+ // count is read first: a switch flips the flag before it counts, so one
+ // that overlaps these reads leaves a stale count, never a stale mode.
+ switchesSeen := mockModeSwitchCount.Load()
+ builtInMockMode := mock.IsMockEnabled()
m := &Monitor{
config: cfg,
state: models.NewState(),
@@ -1804,7 +1813,9 @@ func New(cfg *config.Config) (*Monitor, error) {
rateTracker: NewRateTracker(),
metricsHistory: NewMetricsHistory(1000, 24*time.Hour), // Keep up to 1000 points (~8h @ 30s)
metricsStore: metricsStore, // Persistent SQLite storage
- alertManager: alerts.NewManagerWithDataDir(cfg.DataPath, alertManagerRestoreOptions()...),
+ alertManager: alerts.NewManagerWithDataDir(cfg.DataPath, alertManagerRestoreOptions(builtInMockMode)...),
+ mockModeAligned: builtInMockMode,
+ mockModeSwitchesSeen: switchesSeen,
incidentStore: incidentStore,
notificationMgr: notifications.NewNotificationManagerWithDeferredQueue(cfg.PublicURL, cfg.DataPath),
deadMan: newDeadManRuntime(config.ResolveRuntimeDataDir(cfg.DataPath)),
@@ -2086,33 +2097,13 @@ func (m *Monitor) Start(ctx context.Context, wsHub *websocket.Hub) {
// PVE-proxied PBS backup points to prevent duplicate recovery entries.
m.purgeStalePVEPBSBackupsBestEffort(ctx)
- if mock.IsMockEnabled() {
- m.startMockMetricsSampler(ctx)
- }
-
- // Initialize and start discovery service if enabled
- if mock.IsMockEnabled() {
- log.Info().Msg("mock mode enabled - skipping discovery service")
- m.discoveryService = nil
- } else if m.config.DiscoveryEnabled {
- discoverySubnet := m.config.DiscoverySubnet
- if discoverySubnet == "" {
- discoverySubnet = "auto"
- }
- cfgProvider := func() config.DiscoveryConfig {
- return m.discoveryConfigSnapshot()
- }
- m.discoveryService = discovery.NewService(wsHub, 5*time.Minute, discoverySubnet, cfgProvider)
- if m.discoveryService != nil {
- m.discoveryService.Start(ctx)
- log.Info().Msg("discovery service initialized and started")
- } else {
- log.Error().Msg("failed to initialize discovery service")
- }
- } else {
- log.Info().Msg("discovery service disabled by configuration")
- m.discoveryService = nil
- }
+ // From here every mock-mode switch reaches this monitor. The sampler or
+ // discovery is chosen while no switch can run, so a switch cannot land
+ // between reading the mode and starting the runtime for it.
+ leaveMockModeSwitches := m.joinMockModeSwitches(func(mockEnabled bool) {
+ m.startModeRuntime(ctx, wsHub, mockEnabled)
+ })
+ defer leaveMockModeSwitches()
// Set up alert callbacks. Projection replay is deliberately absent here:
// the canonical resource store is not attached yet, so a replay now could
@@ -2251,6 +2242,38 @@ func (m *Monitor) Start(ctx context.Context, wsHub *websocket.Hub) {
}
}
+// startModeRuntime starts the mock metrics sampler in mock mode, or the
+// discovery service in live mode when it is enabled.
+func (m *Monitor) startModeRuntime(ctx context.Context, wsHub *websocket.Hub, mockEnabled bool) {
+ if mockEnabled {
+ m.startMockMetricsSampler(ctx)
+ }
+
+ // Initialize and start discovery service if enabled
+ if mockEnabled {
+ log.Info().Msg("mock mode enabled - skipping discovery service")
+ m.discoveryService = nil
+ } else if m.config.DiscoveryEnabled {
+ discoverySubnet := m.config.DiscoverySubnet
+ if discoverySubnet == "" {
+ discoverySubnet = "auto"
+ }
+ cfgProvider := func() config.DiscoveryConfig {
+ return m.discoveryConfigSnapshot()
+ }
+ m.discoveryService = discovery.NewService(wsHub, 5*time.Minute, discoverySubnet, cfgProvider)
+ if m.discoveryService != nil {
+ m.discoveryService.Start(ctx)
+ log.Info().Msg("discovery service initialized and started")
+ } else {
+ log.Error().Msg("failed to initialize discovery service")
+ }
+ } else {
+ log.Info().Msg("discovery service disabled by configuration")
+ m.discoveryService = nil
+ }
+}
+
// wireExternalAlertCallbacks installs callbacks whose effects leave the alert
// manager. It is safe to call again: these are single callback slots, so Start
// replaces the constructor-time escalation closure with one carrying its hub.
@@ -4577,65 +4600,93 @@ func (m *Monitor) broadcastEscalatedAlert(hub *websocket.Hub, alert *alerts.Aler
hub.BroadcastAlertToTenant(m.GetOrgID(), alert)
}
-// mockModeSwitchMu serializes mock-mode switches. The mode is process-wide,
-// and a switch is several steps (flip, end the epoch, clear, reset state), so
-// two interleaved switches could clear what the later one just admitted.
-var mockModeSwitchMu sync.Mutex
-
-// SetMockMode switches between mock data and real infrastructure data at runtime.
-// It must not be called from inside an alert evaluation, whose completion it
-// may wait for.
+// SetMockMode switches between mock data and real infrastructure data at
+// runtime. The mode is process-wide, so the switch reaches m and every running
+// monitor (see mockModeSwitchMu), whichever tenant's monitor it was made
+// through. It must not be called from inside an alert evaluation of any of
+// them, whose completion it may wait for.
func (m *Monitor) SetMockMode(enable bool) error {
mockModeSwitchMu.Lock()
defer mockModeSwitchMu.Unlock()
- current := mock.IsMockEnabled()
- if current == enable {
- log.Info().Bool("mockMode", enable).Msg("mock mode already in desired state")
- return nil
+ monitors := mockModeSwitchTargetsLocked(m)
+ flip := mock.IsMockEnabled() != enable
+ if !flip {
+ // The flag already holds the mode, but a monitor that was not running
+ // during an earlier switch can still hold the other mode's alerts
+ // and state; only those leave it.
+ monitors = slices.DeleteFunc(monitors, func(monitor *Monitor) bool {
+ return monitor.mockModeAligned == enable
+ })
+ if len(monitors) == 0 {
+ log.Info().Bool("mockMode", enable).Msg("mock mode already in desired state")
+ return nil
+ }
}
- // Every evaluation of mode-dependent data that started before the flip
- // must finish or be refused before the clear, or it reopens alerts for
- // the side the monitor just left (see mockModeFence).
- if enable {
- m.stopMockMetricsSampler()
- if err := mock.SetEnabled(true); err != nil {
+ for _, monitor := range monitors {
+ monitor.stopMockMetricsSampler()
+ }
+ if flip {
+ if err := mock.SetEnabled(enable); err != nil {
return err
}
- m.mockModeFence.advance()
- m.alertManager.ClearActiveAlerts()
- m.mu.Lock()
- m.resetStateLocked()
- m.metricsHistory.Reset()
- m.mu.Unlock()
- m.StopDiscoveryService()
- m.mu.RLock()
- ctx := m.runtimeCtx
- m.mu.RUnlock()
- if ctx != nil {
- m.startMockMetricsSampler(ctx)
- }
- log.Info().Msg("switched monitor to mock mode")
+ mockModeSwitchCount.Add(1)
+ }
+ for _, monitor := range monitors {
+ monitor.endMockModeEpoch(enable)
+ }
+ for _, monitor := range monitors {
+ _, running := mockModeRunningMonitors[monitor]
+ monitor.resumeAfterMockModeSwitch(enable, running)
+ }
+ if enable {
+ log.Info().Int("monitors", len(monitors)).Msg("switched monitors to mock mode")
} else {
- m.stopMockMetricsSampler()
- if err := mock.SetEnabled(false); err != nil {
- return err
- }
- m.mockModeFence.advance()
- m.alertManager.ClearActiveAlerts()
+ log.Info().Int("monitors", len(monitors)).Msg("switched monitors to real data mode")
+ }
+ return nil
+}
+
+// endMockModeEpoch leaves the mode the monitor's alerts and state belong to
+// once the process flag already holds the new one. Every evaluation of
+// mode-dependent data that started before the flip must finish or be refused
+// before the clear, or it reopens alerts for the side the monitor just left
+// (see mockModeFence). The caller holds mockModeSwitchMu.
+func (m *Monitor) endMockModeEpoch(enable bool) {
+ m.mockModeFence.advance()
+ m.alertManager.ClearActiveAlerts()
+ if !enable {
m.forgetMockHostAgents()
- m.mu.Lock()
- m.resetStateLocked()
- m.metricsHistory.Reset()
- m.mu.Unlock()
- log.Info().Msg("switched monitor to real data mode")
}
+ m.mu.Lock()
+ m.resetStateLocked()
+ m.metricsHistory.Reset()
+ m.mu.Unlock()
+ if enable {
+ m.StopDiscoveryService()
+ }
+ m.mockModeAligned = enable
+ m.mockModeSwitchesSeen = mockModeSwitchCount.Load()
+}
+// resumeAfterMockModeSwitch restarts the mode-dependent runtime of a monitor
+// whose Start loop has joined the switches, once every monitor has ended its
+// epoch. A monitor that has not joined yet starts that runtime itself when it
+// does (Start sets its runtime context first), and one whose loop has stopped
+// only broadcasts.
+func (m *Monitor) resumeAfterMockModeSwitch(enable, running bool) {
m.mu.RLock()
ctx := m.runtimeCtx
hub := m.wsHub
m.mu.RUnlock()
+ if !running || (ctx != nil && ctx.Err() != nil) {
+ ctx = nil
+ }
+
+ if enable && ctx != nil {
+ m.startMockMetricsSampler(ctx)
+ }
if hub != nil {
m.broadcastCurrentState(hub)
@@ -4653,8 +4704,6 @@ func (m *Monitor) SetMockMode(enable bool) error {
go m.StartDiscoveryService(ctx, hub, m.config.DiscoverySubnet)
}
}
-
- return nil
}
func (m *Monitor) resetStateLocked() {
diff --git a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go
index dca2a68b1c..e9d81cf562 100644
--- a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go
+++ b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go
@@ -11,6 +11,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
+ "github.com/rcourtman/pulse-go-rewrite/internal/mock"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"
@@ -857,3 +858,45 @@ func TestMockHostAgentLeavingFixtureUsesRemovalLifecycle(t *testing.T) {
}
t.Fatal("node pve9 raised no memory alert after its agent left the mock estate")
}
+
+func TestLeavingMockModeReleasesFixtureAgentNodeLinksOnEveryRunningMonitor(t *testing.T) {
+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+ pinDefaultMockEstate(t)
+ mustSetMockEnabled(t, true)
+ defaultMonitor, tenantMonitor := startTenantMonitors(t)
+
+ // org-b's start pass registers the fixture agents in org-b's own alert
+ // manager, each owning the usage alerts of the fixture node it runs on.
+ var linked models.Node
+ waitForCondition(t, 30*time.Second, func() bool {
+ tenantMonitor.mockHostAgentsMu.Lock()
+ defer tenantMonitor.mockHostAgentsMu.Unlock()
+ for _, host := range tenantMonitor.mockHostAgents {
+ if host.Status != "online" || host.LinkedNodeID == "" {
+ continue
+ }
+ for _, node := range mock.CurrentFixtureGraph().State.Nodes {
+ if node.ID == host.LinkedNodeID {
+ linked = node
+ return true
+ }
+ }
+ }
+ return false
+ }, "org-b registered no online fixture agent linked to a fixture node")
+
+ // The demo-fixture licence sync switches through the default monitor.
+ mustSetMonitorMockMode(t, defaultMonitor, false)
+
+ // org-b's fixture agents go through the removal lifecycle too, so the
+ // node an agent covered owns its metric alerts again.
+ linked.Status = "online"
+ linked.Memory = models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75}
+ tenantMonitor.alertManager.CheckNode(linked)
+ for _, alert := range tenantMonitor.alertManager.GetActiveAlerts() {
+ if alert.ResourceID == linked.ID && alert.Type == "memory" {
+ return
+ }
+ }
+ t.Fatalf("org-b's node %s raised no memory alert: its fixture agent's node link outlived a switch made through the default monitor", linked.ID)
+}
diff --git a/internal/monitoring/monitor_mock_alerts_test.go b/internal/monitoring/monitor_mock_alerts_test.go
index 3c02565ece..ce0b725e54 100644
--- a/internal/monitoring/monitor_mock_alerts_test.go
+++ b/internal/monitoring/monitor_mock_alerts_test.go
@@ -15,6 +15,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
+ "github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
@@ -809,6 +810,254 @@ func TestLeavingMockModeDiscardsAnInFlightConnectionCheck(t *testing.T) {
}
}
+// startTenantMonitors starts the default organization's monitor and one
+// other tenant's through MultiTenantMonitor, as the server does, each with its
+// own alert manager, and waits until both monitoring loops run. An hour
+// between ticks leaves each loop the pass it runs on start, so the test drives
+// every later pass itself.
+func startTenantMonitors(t *testing.T) (defaultMonitor, tenantMonitor *Monitor) {
+ t.Helper()
+ mtp, _ := newTestTenantPersistence(t)
+ mtm := NewMultiTenantMonitor(&config.Config{DataPath: t.TempDir(), PVEPollingInterval: time.Hour}, mtp, nil)
+ t.Cleanup(mtm.Stop)
+ mtm.SetMonitorInitializer(func(monitor *Monitor) {
+ cfg := monitor.alertManager.GetConfig()
+ cfg.Enabled = true
+ cfg.TimeThresholds = map[string]int{}
+ cfg.MetricTimeThresholds = nil
+ cfg.BackupDefaults.Enabled = false
+ cfg.AgentDefaults.Memory = &alerts.HysteresisThreshold{Trigger: 1, Clear: 0.5}
+ cfg.NodeDefaults.Memory = &alerts.HysteresisThreshold{Trigger: 1, Clear: 0.5}
+ monitor.alertManager.UpdateConfig(cfg)
+ })
+ if err := mtp.SaveOrganization(&models.Organization{ID: "org-b", DisplayName: "Org B"}); err != nil {
+ t.Fatalf("SaveOrganization(org-b): %v", err)
+ }
+ monitors := make([]*Monitor, 0, 2)
+ for _, orgID := range []string{"default", "org-b"} {
+ monitor, err := mtm.GetMonitor(orgID)
+ if err != nil {
+ t.Fatalf("GetMonitor(%s): %v", orgID, err)
+ }
+ // Start sets its runtime context before it joins the monitors a
+ // switch reaches, so wait for the join itself.
+ waitForCondition(t, 30*time.Second, func() bool {
+ mockModeSwitchMu.Lock()
+ defer mockModeSwitchMu.Unlock()
+ _, running := mockModeRunningMonitors[monitor]
+ return running
+ }, orgID+" monitoring loop did not join mock-mode switches")
+ monitors = append(monitors, monitor)
+ }
+ return monitors[0], monitors[1]
+}
+
+func mockHostAgentCount(monitor *Monitor) int {
+ monitor.mockHostAgentsMu.Lock()
+ defer monitor.mockHostAgentsMu.Unlock()
+ return len(monitor.mockHostAgents)
+}
+
+func TestLeavingMockModeClearsEveryRunningTenantMonitor(t *testing.T) {
+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+ pinDefaultMockEstate(t)
+ mustSetMockEnabled(t, true)
+ defaultMonitor, tenantMonitor := startTenantMonitors(t)
+
+ // Each loop runs a mock pass on start and raises the fixture's alerts in
+ // its own alert manager.
+ monitors := map[string]*Monitor{"default": defaultMonitor, "org-b": tenantMonitor}
+ for name, monitor := range monitors {
+ waitForCondition(t, 30*time.Second, func() bool {
+ return len(monitor.alertManager.GetActiveAlerts()) > 0 && mockHostAgentCount(monitor) > 0
+ }, name+" monitor raised no fixture alerts in mock mode")
+ }
+
+ // The mock-mode API switches through the requesting tenant's monitor.
+ mustSetMonitorMockMode(t, tenantMonitor, false)
+
+ for name, monitor := range monitors {
+ if active := monitor.alertManager.GetActiveAlerts(); len(active) != 0 {
+ t.Errorf("%s monitor kept %d fixture alerts after mock mode was left through org-b: %s",
+ name, len(active), describeAlerts(active))
+ }
+ if count := mockHostAgentCount(monitor); count != 0 {
+ t.Errorf("%s monitor still tracks %d fixture agents in live mode", name, count)
+ }
+ }
+}
+
+func TestEnteringMockModeClearsEveryRunningTenantMonitor(t *testing.T) {
+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+ pinDefaultMockEstate(t)
+ defaultMonitor, tenantMonitor := startTenantMonitors(t)
+
+ monitors := map[string]*Monitor{"default": defaultMonitor, "org-b": tenantMonitor}
+ for name, monitor := range monitors {
+ monitor.alertManager.CheckNode(models.Node{
+ ID: name + "-live-node",
+ Name: "live-node",
+ Instance: "live",
+ Status: "online",
+ Memory: models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75},
+ })
+ if len(monitor.alertManager.GetActiveAlerts()) == 0 {
+ t.Fatalf("live node raised no alert in %s", name)
+ }
+ }
+
+ // The demo-fixture licence sync switches through the default monitor.
+ mustSetMonitorMockMode(t, defaultMonitor, true)
+
+ for name, monitor := range monitors {
+ for _, alert := range monitor.alertManager.GetActiveAlerts() {
+ if alert.ResourceID == name+"-live-node" {
+ t.Errorf("%s kept its live %s alert after mock mode was entered through the default monitor", name, alert.Type)
+ }
+ }
+ }
+}
+
+func TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn(t *testing.T) {
+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+ pinDefaultMockEstate(t)
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{
+ state: models.NewState(),
+ alertManager: manager,
+ metricsHistory: NewMetricsHistory(10, time.Hour),
+ }
+ // Built in live mode: New restored this live alert from disk.
+ manager.CheckNode(models.Node{
+ ID: "live-node",
+ Name: "live-node",
+ Instance: "live",
+ Status: "online",
+ Memory: models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75},
+ })
+ if len(manager.GetActiveAlerts()) == 0 {
+ t.Fatal("live node raised no alert")
+ }
+
+ // Another monitor switched to mock mode before this one's Start joined.
+ mustSetMockEnabled(t, true)
+ started := false
+ leave := monitor.joinMockModeSwitches(func(mockEnabled bool) { started = mockEnabled })
+ defer leave()
+
+ if !started {
+ t.Fatal("Start was not handed mock mode")
+ }
+ if active := manager.GetActiveAlerts(); len(active) != 0 {
+ t.Fatalf("a monitor built in live mode started in mock mode with its live alerts: %s", describeAlerts(active))
+ }
+ mockModeSwitchMu.Lock()
+ _, running := mockModeRunningMonitors[monitor]
+ mockModeSwitchMu.Unlock()
+ if !running {
+ t.Fatal("joined monitor is not reached by later switches")
+ }
+}
+
+func TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile(t *testing.T) {
+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+ pinDefaultMockEstate(t)
+ newMonitor := func() *Monitor {
+ return &Monitor{
+ state: models.NewState(),
+ alertManager: newMockHostAlertTestManager(t),
+ metricsHistory: NewMetricsHistory(10, time.Hour),
+ }
+ }
+ switcher, starting := newMonitor(), newMonitor()
+ // Built in live mode, as New records it, and not running yet.
+ mockModeSwitchMu.Lock()
+ starting.mockModeSwitchesSeen = mockModeSwitchCount.Load()
+ mockModeSwitchMu.Unlock()
+
+ mustSetMonitorMockMode(t, switcher, true)
+ // A read path evaluates fixture data on the monitor before its Start runs.
+ fixture := mock.CurrentFixtureGraph().State.Nodes[0]
+ fixture.Status = "online"
+ fixture.Memory = models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75}
+ starting.alertManager.CheckNode(fixture)
+ if len(starting.alertManager.GetActiveAlerts()) == 0 {
+ t.Fatal("fixture node raised no alert")
+ }
+ mustSetMonitorMockMode(t, switcher, false)
+
+ leave := starting.joinMockModeSwitches(func(bool) {})
+ defer leave()
+ if active := starting.alertManager.GetActiveAlerts(); len(active) != 0 {
+ t.Fatalf("a monitor that missed a mock-mode round trip started in live mode with fixture alerts: %s", describeAlerts(active))
+ }
+}
+
+func TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds(t *testing.T) {
+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+ pinDefaultMockEstate(t)
+ manager := newMockHostAlertTestManager(t)
+ monitor := &Monitor{
+ state: models.NewState(),
+ alertManager: manager,
+ metricsHistory: NewMetricsHistory(10, time.Hour),
+ }
+ // Built in live mode, with a live alert, and not running yet.
+ manager.CheckNode(models.Node{
+ ID: "live-node",
+ Name: "live-node",
+ Instance: "live",
+ Status: "online",
+ Memory: models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75},
+ })
+ if len(manager.GetActiveAlerts()) == 0 {
+ t.Fatal("live node raised no alert")
+ }
+
+ // Another monitor's switch already set the flag.
+ mustSetMockEnabled(t, true)
+ mustSetMonitorMockMode(t, monitor, true)
+
+ if active := manager.GetActiveAlerts(); len(active) != 0 {
+ t.Fatalf("SetMockMode(true) left a monitor built in live mode with its live alerts: %s", describeAlerts(active))
+ }
+}
+
+func TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart(t *testing.T) {
+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+ pinDefaultMockEstate(t)
+ ctx, cancel := context.WithCancel(context.Background())
+ t.Cleanup(cancel)
+ // Start has set its runtime context but not joined the switches yet.
+ monitor := &Monitor{
+ config: &config.Config{},
+ state: models.NewState(),
+ alertManager: newMockHostAlertTestManager(t),
+ metricsHistory: NewMetricsHistory(10, time.Hour),
+ runtimeCtx: ctx,
+ }
+ t.Cleanup(monitor.stopMockMetricsSampler)
+
+ mustSetMonitorMockMode(t, monitor, true)
+ monitor.mu.RLock()
+ started := monitor.mockMetricsCancel != nil
+ monitor.mu.RUnlock()
+ if started {
+ t.Fatal("a switch started the runtime of a monitor whose Start has not joined; that runtime belongs to its Start")
+ }
+
+ leave := monitor.joinMockModeSwitches(func(mockEnabled bool) {
+ monitor.startModeRuntime(ctx, nil, mockEnabled)
+ })
+ defer leave()
+ monitor.mu.RLock()
+ started = monitor.mockMetricsCancel != nil
+ monitor.mu.RUnlock()
+ if !started {
+ t.Fatal("joining in mock mode did not start the mock sampler")
+ }
+}
+
// diskAlertKeys lists the active disk-health and disk-wearout alerts as
// "type instance/node device" so assertions name the disk, not the alert ID.
func diskAlertKeys(manager *alerts.Manager) map[string]bool {