Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Switch every running tenant monitor when mock mode flips
rcourtman27e440339b7a
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex e7b4f59ab3..7582e7db33 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -754,6 +754,15 @@ not fenced: a live report already past its mock-mode check when mock mode isswitched on can still be evaluated after the clear, and its alerts can persistuntil mock mode is left.+Every tenant monitor registers its own fixture agents, and their node links,+in its own alert manager, so the toggle forgets the fixture agents of every+running monitor, not only those of the monitor it was called through. A link+left behind would keep owning its fixture node's usage alerts after mock mode+ends. `TestLeavingMockModeReleasesFixtureAgentNodeLinksOnEveryRunningMonitor`+in `internal/monitoring/monitor_host_agent_removal_lifecycle_test.go` switches+through the default monitor and fails if the other tenant's node stays+suppressed.+Physical-disk evidence collected by a host agent must survive projection backinto monitoring's models. Absent evidence has to carry its declared sentinelrather than a zero value that reads as a real measurement: an absentdiff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex 60ff94e27d..7e54def128 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -1855,11 +1855,33 @@ a live report or poll already past its mock-mode check when mock mode isswitched on can still evaluate after that clear, and its alerts can persistuntil mock mode is left.-The fence and the clear are per monitor while the mode is process-wide. A-tenant monitor other than the one `SetMockMode` was called on neither drains-nor clears, so fixture alerts it raised outlive the switch as they did before-the fence; switching mock mode with several tenant monitors running is not-covered.+The mode is process-wide while each monitor keeps its own fence, alert manager,+fixture agents and state, and the server runs one monitor per organization, so+`SetMockMode` switches every running monitor, whichever tenant's monitor it is+called on (the mock-mode API calls the requesting tenant's, the demo-fixture+licence sync the default organization's). `Start` joins a process-wide set of+running monitors under `mockModeSwitchMu`, and leaves it when its loop returns.+A switch flips the mode once, then ends each monitor's epoch, waits for its+admitted calls, clears its alerts, forgets its fixture agents when leaving mock+mode and resets its state, and only then restarts the mode-dependent runtime of+each monitor in the set. `Start` sets its runtime context before it joins and+chooses the mock metrics sampler or discovery while it joins, under the same+lock, so no switch lands between reading the mode and starting the runtime for+it, and a switch that reaches a monitor not yet joined leaves the runtime to+its `Start`. A monitor that missed switches between `New` and `Start` first+leaves what it holds: `New` records the mode and the number of switches made so+far, and `Start` compares both, so even a round trip that brought the flag back+drops the fixture alerts a read path raised meanwhile, as well as the live+alerts `New` restored outside mock mode. A monitor that is not running is+switched only when the switch is made through it; a call for the mode the flag+already holds still moves any reached monitor whose alerts belong to the other+mode, and leaves the rest alone.++Release builds reach several tenant monitors in mock mode only on an instance+running `DemoMode` with the `demo_fixtures` entitlement (release+`ValidateEnablement` refuses mock fixtures otherwise) and with more than one+organization's monitor running; the licence sync switches only for the+default organization.The proofs park a real evaluation at a fixed point, switch modes and releaseit. `TestLeavingMockModeRefusesTheRestOfAnInFlightPass` holds a mock pass at its@@ -1889,7 +1911,25 @@ previous-state carry returns fixture nodes or guests. All eight are in`TestStoreRefreshTakesItsMockModeScopeBeforeReadingState` in`internal/monitoring/canonical_guardrails_test.go` rejects a call that readsstate in an argument ahead of `m.mockModeFence.begin()` and pins the-end-epoch-then-clear order in `SetMockMode`.+end-epoch-then-clear order in `endMockModeEpoch`, which `SetMockMode` runs for+every monitor it reaches.++`TestLeavingMockModeClearsEveryRunningTenantMonitor` and+`TestEnteringMockModeClearsEveryRunningTenantMonitor` start the default and one+other tenant monitor through `MultiTenantMonitor`, wait until both have joined,+raise alerts in both, switch through one of them and check both. Against a+switch that reached only its own monitor, the default monitor kept its fixture+alerts (69 to 104 across runs) and all 69 fixture agents after leaving mock+mode through the other tenant, and that tenant kept its live alert after mock+mode was entered through the default monitor. `TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn`+fails if `Start` joins without leaving the mode its monitor was built in, and+`TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` fails if a call for+the flag's current mode returns without moving such a monitor, and+`TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` fails if a switch+starts the runtime of a monitor whose `Start` has not joined, and+`TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile` fails if+`Start` compares only the mode. All six are in+`internal/monitoring/monitor_mock_alerts_test.go`.The disk evaluation runs on the mock alert tick, not the poller's diskinterval (five minutes by default, `PhysicalDiskPollingMinutes` per instance).diff --git a/internal/monitoring/canonical_guardrails_test.go b/internal/monitoring/canonical_guardrails_test.goindex fd27c2215b..8a13f8b631 100644--- a/internal/monitoring/canonical_guardrails_test.go+++ b/internal/monitoring/canonical_guardrails_test.go@@ -543,8 +543,12 @@ func TestStoreRefreshTakesItsMockModeScopeBeforeReadingState(t *testing.T) {if err != nil {t.Fatalf("failed to read monitor.go: %v", err)}- if got := strings.Count(string(data), "m.mockModeFence.advance()\n\t\tm.alertManager.ClearActiveAlerts()"); got != 2 {- t.Fatalf("SetMockMode must end the mock-mode epoch immediately before clearing alerts in both directions, found %d", got)+ source := string(data)+ if got := strings.Count(source, "m.mockModeFence.advance()\n\tm.alertManager.ClearActiveAlerts()"); got != 1 {+ t.Fatalf("endMockModeEpoch must end the mock-mode epoch immediately before clearing alerts, found %d", got)+ }+ if !strings.Contains(source, "for _, monitor := range monitors {\n\t\tmonitor.endMockModeEpoch(enable)\n\t}") {+ t.Fatal("SetMockMode must end the epoch of every monitor the switch reaches")}}@@ -1664,16 +1668,11 @@ func TestMonitorSetMockModeAuthorizesBeforeResettingRuntimeState(t *testing.T) {if err != nil {t.Fatalf("failed to read monitor.go: %v", err)}- source := string(data)- requiredSnippets := []string{- "if err := mock.SetEnabled(true); err != nil {",- "if err := mock.SetEnabled(false); err != nil {",- "return err",- }- for _, snippet := range requiredSnippets {- if !strings.Contains(source, snippet) {- t.Fatalf("monitor.go must contain %q", snippet)- }+ source := strings.Join(strings.Fields(string(data)), " ")+ authorize := strings.Index(source, "if err := mock.SetEnabled(enable); err != nil { return err }")+ reset := strings.Index(source, "monitor.endMockModeEpoch(enable)")+ if authorize < 0 || reset < 0 || authorize > reset {+ t.Fatal("SetMockMode must return the mock.SetEnabled error before it ends any monitor's epoch or resets its state")}}diff --git a/internal/monitoring/mock_mode_switch.go b/internal/monitoring/mock_mode_switch.gonew file mode 100644index 0000000000..4419f77b88--- /dev/null+++ b/internal/monitoring/mock_mode_switch.go@@ -0,0 +1,75 @@+package monitoring++import (+ "sync"+ "sync/atomic"++ "github.com/rcourtman/pulse-go-rewrite/internal/mock"+)++// Mock mode is process-wide (mock.SetEnabled flips one package flag), while+// every Monitor keeps its own alert manager, fixture agents, state and+// mock-mode fence, and the server runs one Monitor per organization. A switch+// made through one monitor therefore has to reach every running monitor, or+// the others keep what they raised in the mode the process left: fixture+// alerts and fixture-agent node links in live mode, live alerts in mock mode,+// until the 24-hour stale sweep.+//+// mockModeSwitchMu serializes switches and guards the set of running monitors+// and each monitor's mockModeAligned and mockModeSwitchesSeen. A switch is+// several steps per monitor (end the epoch, clear, reset), so two interleaved+// switches could clear what the later one just admitted. Start joins the set+// under the same lock, so a monitor joins either before a switch, which then+// switches it with the others, or after it, and starts in the new mode.+//+// mockModeSwitchCount counts the flips SetMockMode made. A monitor that missed+// flips while it was not running can hold the alerts and state of a mode the+// process left even when a later flip brought the flag back to the mode it+// records (a read path evaluated fixture data before Start), so Start+// compares the count as well as the mode.+var (+ mockModeSwitchMu sync.Mutex+ mockModeRunningMonitors = make(map[*Monitor]struct{})+ mockModeSwitchCount atomic.Uint64+)++// mockModeSwitchTargetsLocked returns the monitors a switch made through m+// reaches: m itself, which need not be running, and every running monitor.+func mockModeSwitchTargetsLocked(m *Monitor) []*Monitor {+ targets := make([]*Monitor, 0, len(mockModeRunningMonitors)+1)+ targets = append(targets, m)+ for monitor := range mockModeRunningMonitors {+ if monitor != m {+ targets = append(targets, monitor)+ }+ }+ return targets+}++// joinMockModeSwitches adds m to the monitors every switch reaches, then runs+// start with the current mode while no switch can run, so the mode-dependent+// runtime start chooses (the mock metrics sampler or discovery) cannot be+// overtaken by a switch. It returns the function that removes m again.+//+// A switch can land between New and Start, while m is not yet running. New+// restores persisted alerts only outside mock mode, so m would then carry live+// alerts into mock mode, or fixture alerts that a read path raised before+// Start into live mode; m first leaves what it holds, as the switches it+// missed would have done.+func (m *Monitor) joinMockModeSwitches(start func(mockEnabled bool)) (leave func()) {+ mockModeSwitchMu.Lock()+ defer mockModeSwitchMu.Unlock()++ enabled := mock.IsMockEnabled()+ if m.mockModeAligned != enabled || m.mockModeSwitchesSeen != mockModeSwitchCount.Load() {+ m.endMockModeEpoch(enabled)+ }+ mockModeRunningMonitors[m] = struct{}{}+ start(enabled)++ return func() {+ mockModeSwitchMu.Lock()+ defer mockModeSwitchMu.Unlock()+ delete(mockModeRunningMonitors, m)+ }+}diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.goindex 69c5c33f37..80fc4aa163 100644--- a/internal/monitoring/monitor.go+++ b/internal/monitoring/monitor.go@@ -13,6 +13,7 @@ import ("os""path/filepath""reflect"+ "slices""sort""strings""sync"@@ -67,9 +68,9 @@ func keepRealPollingInMockMode() bool {// process that boots with mock mode already enabled never runs that path, so// alerts raised against real infrastructure are restored from disk and served// beside fixture data until something clears them.-func alertManagerRestoreOptions() []alerts.ManagerOption {+func alertManagerRestoreOptions(mockEnabled bool) []alerts.ManagerOption {options := []alerts.ManagerOption{alerts.WithDurableAlertStore()}- if mock.IsMockEnabled() {+ if mockEnabled {options = append(options, alerts.WithoutPersistedAlertRestore())}return options@@ -1283,6 +1284,8 @@ type Monitor struct {mockHostAgentsMu sync.MutexmockHostAgents map[string]models.Host // Fixture agents evaluated by the last mock alert passmockModeFence mockModeFence // Keeps mode-dependent alert evaluations inside the epoch they read in+ mockModeAligned bool // Mock mode the alerts and state belong to; guarded by mockModeSwitchMu+ mockModeSwitchesSeen uint64 // mockModeSwitchCount when they were last aligned; guarded by mockModeSwitchMudockerChecker DockerChecker // Optional Docker checker for LXC containersdockerCheckerConfiguredAt time.Time // Last time the Docker checker was configureddockerCheckAllowedVMIDs map[int]struct{} // Optional VMID allowlist gating the LXC Docker socket probe; empty means all guests@@ -1771,6 +1774,12 @@ func New(cfg *config.Config) (*Monitor, error) {DataDir: cfg.DataPath,})+ // One read decides the alert restore and records the mode it belongs to,+ // so Start can tell whether a mock-mode switch landed in between. The+ // count is read first: a switch flips the flag before it counts, so one+ // that overlaps these reads leaves a stale count, never a stale mode.+ switchesSeen := mockModeSwitchCount.Load()+ builtInMockMode := mock.IsMockEnabled()m := &Monitor{config: cfg,state: models.NewState(),@@ -1804,7 +1813,9 @@ func New(cfg *config.Config) (*Monitor, error) {rateTracker: NewRateTracker(),metricsHistory: NewMetricsHistory(1000, 24*time.Hour), // Keep up to 1000 points (~8h @ 30s)metricsStore: metricsStore, // Persistent SQLite storage- alertManager: alerts.NewManagerWithDataDir(cfg.DataPath, alertManagerRestoreOptions()...),+ alertManager: alerts.NewManagerWithDataDir(cfg.DataPath, alertManagerRestoreOptions(builtInMockMode)...),+ mockModeAligned: builtInMockMode,+ mockModeSwitchesSeen: switchesSeen,incidentStore: incidentStore,notificationMgr: notifications.NewNotificationManagerWithDeferredQueue(cfg.PublicURL, cfg.DataPath),deadMan: newDeadManRuntime(config.ResolveRuntimeDataDir(cfg.DataPath)),@@ -2086,33 +2097,13 @@ func (m *Monitor) Start(ctx context.Context, wsHub *websocket.Hub) {// PVE-proxied PBS backup points to prevent duplicate recovery entries.m.purgeStalePVEPBSBackupsBestEffort(ctx)- if mock.IsMockEnabled() {- m.startMockMetricsSampler(ctx)- }-- // Initialize and start discovery service if enabled- if mock.IsMockEnabled() {- log.Info().Msg("mock mode enabled - skipping discovery service")- m.discoveryService = nil- } else if m.config.DiscoveryEnabled {- discoverySubnet := m.config.DiscoverySubnet- if discoverySubnet == "" {- discoverySubnet = "auto"- }- cfgProvider := func() config.DiscoveryConfig {- return m.discoveryConfigSnapshot()- }- m.discoveryService = discovery.NewService(wsHub, 5*time.Minute, discoverySubnet, cfgProvider)- if m.discoveryService != nil {- m.discoveryService.Start(ctx)- log.Info().Msg("discovery service initialized and started")- } else {- log.Error().Msg("failed to initialize discovery service")- }- } else {- log.Info().Msg("discovery service disabled by configuration")- m.discoveryService = nil- }+ // From here every mock-mode switch reaches this monitor. The sampler or+ // discovery is chosen while no switch can run, so a switch cannot land+ // between reading the mode and starting the runtime for it.+ leaveMockModeSwitches := m.joinMockModeSwitches(func(mockEnabled bool) {+ m.startModeRuntime(ctx, wsHub, mockEnabled)+ })+ defer leaveMockModeSwitches()// Set up alert callbacks. Projection replay is deliberately absent here:// the canonical resource store is not attached yet, so a replay now could@@ -2251,6 +2242,38 @@ func (m *Monitor) Start(ctx context.Context, wsHub *websocket.Hub) {}}+// startModeRuntime starts the mock metrics sampler in mock mode, or the+// discovery service in live mode when it is enabled.+func (m *Monitor) startModeRuntime(ctx context.Context, wsHub *websocket.Hub, mockEnabled bool) {+ if mockEnabled {+ m.startMockMetricsSampler(ctx)+ }++ // Initialize and start discovery service if enabled+ if mockEnabled {+ log.Info().Msg("mock mode enabled - skipping discovery service")+ m.discoveryService = nil+ } else if m.config.DiscoveryEnabled {+ discoverySubnet := m.config.DiscoverySubnet+ if discoverySubnet == "" {+ discoverySubnet = "auto"+ }+ cfgProvider := func() config.DiscoveryConfig {+ return m.discoveryConfigSnapshot()+ }+ m.discoveryService = discovery.NewService(wsHub, 5*time.Minute, discoverySubnet, cfgProvider)+ if m.discoveryService != nil {+ m.discoveryService.Start(ctx)+ log.Info().Msg("discovery service initialized and started")+ } else {+ log.Error().Msg("failed to initialize discovery service")+ }+ } else {+ log.Info().Msg("discovery service disabled by configuration")+ m.discoveryService = nil+ }+}+// wireExternalAlertCallbacks installs callbacks whose effects leave the alert// manager. It is safe to call again: these are single callback slots, so Start// replaces the constructor-time escalation closure with one carrying its hub.@@ -4577,65 +4600,93 @@ func (m *Monitor) broadcastEscalatedAlert(hub *websocket.Hub, alert *alerts.Alerhub.BroadcastAlertToTenant(m.GetOrgID(), alert)}-// mockModeSwitchMu serializes mock-mode switches. The mode is process-wide,-// and a switch is several steps (flip, end the epoch, clear, reset state), so-// two interleaved switches could clear what the later one just admitted.-var mockModeSwitchMu sync.Mutex--// SetMockMode switches between mock data and real infrastructure data at runtime.-// It must not be called from inside an alert evaluation, whose completion it-// may wait for.+// SetMockMode switches between mock data and real infrastructure data at+// runtime. The mode is process-wide, so the switch reaches m and every running+// monitor (see mockModeSwitchMu), whichever tenant's monitor it was made+// through. It must not be called from inside an alert evaluation of any of+// them, whose completion it may wait for.func (m *Monitor) SetMockMode(enable bool) error {mockModeSwitchMu.Lock()defer mockModeSwitchMu.Unlock()- current := mock.IsMockEnabled()- if current == enable {- log.Info().Bool("mockMode", enable).Msg("mock mode already in desired state")- return nil+ monitors := mockModeSwitchTargetsLocked(m)+ flip := mock.IsMockEnabled() != enable+ if !flip {+ // The flag already holds the mode, but a monitor that was not running+ // during an earlier switch can still hold the other mode's alerts+ // and state; only those leave it.+ monitors = slices.DeleteFunc(monitors, func(monitor *Monitor) bool {+ return monitor.mockModeAligned == enable+ })+ if len(monitors) == 0 {+ log.Info().Bool("mockMode", enable).Msg("mock mode already in desired state")+ return nil+ }}- // Every evaluation of mode-dependent data that started before the flip- // must finish or be refused before the clear, or it reopens alerts for- // the side the monitor just left (see mockModeFence).- if enable {- m.stopMockMetricsSampler()- if err := mock.SetEnabled(true); err != nil {+ for _, monitor := range monitors {+ monitor.stopMockMetricsSampler()+ }+ if flip {+ if err := mock.SetEnabled(enable); err != nil {return err}- m.mockModeFence.advance()- m.alertManager.ClearActiveAlerts()- m.mu.Lock()- m.resetStateLocked()- m.metricsHistory.Reset()- m.mu.Unlock()- m.StopDiscoveryService()- m.mu.RLock()- ctx := m.runtimeCtx- m.mu.RUnlock()- if ctx != nil {- m.startMockMetricsSampler(ctx)- }- log.Info().Msg("switched monitor to mock mode")+ mockModeSwitchCount.Add(1)+ }+ for _, monitor := range monitors {+ monitor.endMockModeEpoch(enable)+ }+ for _, monitor := range monitors {+ _, running := mockModeRunningMonitors[monitor]+ monitor.resumeAfterMockModeSwitch(enable, running)+ }+ if enable {+ log.Info().Int("monitors", len(monitors)).Msg("switched monitors to mock mode")} else {- m.stopMockMetricsSampler()- if err := mock.SetEnabled(false); err != nil {- return err- }- m.mockModeFence.advance()- m.alertManager.ClearActiveAlerts()+ log.Info().Int("monitors", len(monitors)).Msg("switched monitors to real data mode")+ }+ return nil+}++// endMockModeEpoch leaves the mode the monitor's alerts and state belong to+// once the process flag already holds the new one. Every evaluation of+// mode-dependent data that started before the flip must finish or be refused+// before the clear, or it reopens alerts for the side the monitor just left+// (see mockModeFence). The caller holds mockModeSwitchMu.+func (m *Monitor) endMockModeEpoch(enable bool) {+ m.mockModeFence.advance()+ m.alertManager.ClearActiveAlerts()+ if !enable {m.forgetMockHostAgents()- m.mu.Lock()- m.resetStateLocked()- m.metricsHistory.Reset()- m.mu.Unlock()- log.Info().Msg("switched monitor to real data mode")}+ m.mu.Lock()+ m.resetStateLocked()+ m.metricsHistory.Reset()+ m.mu.Unlock()+ if enable {+ m.StopDiscoveryService()+ }+ m.mockModeAligned = enable+ m.mockModeSwitchesSeen = mockModeSwitchCount.Load()+}+// resumeAfterMockModeSwitch restarts the mode-dependent runtime of a monitor+// whose Start loop has joined the switches, once every monitor has ended its+// epoch. A monitor that has not joined yet starts that runtime itself when it+// does (Start sets its runtime context first), and one whose loop has stopped+// only broadcasts.+func (m *Monitor) resumeAfterMockModeSwitch(enable, running bool) {m.mu.RLock()ctx := m.runtimeCtxhub := m.wsHubm.mu.RUnlock()+ if !running || (ctx != nil && ctx.Err() != nil) {+ ctx = nil+ }++ if enable && ctx != nil {+ m.startMockMetricsSampler(ctx)+ }if hub != nil {m.broadcastCurrentState(hub)@@ -4653,8 +4704,6 @@ func (m *Monitor) SetMockMode(enable bool) error {go m.StartDiscoveryService(ctx, hub, m.config.DiscoverySubnet)}}-- return nil}func (m *Monitor) resetStateLocked() {diff --git a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.goindex dca2a68b1c..e9d81cf562 100644--- a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go+++ b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go@@ -11,6 +11,7 @@ import ("github.com/rcourtman/pulse-go-rewrite/internal/alerts""github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog""github.com/rcourtman/pulse-go-rewrite/internal/config"+ "github.com/rcourtman/pulse-go-rewrite/internal/mock""github.com/rcourtman/pulse-go-rewrite/internal/models""github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"@@ -857,3 +858,45 @@ func TestMockHostAgentLeavingFixtureUsesRemovalLifecycle(t *testing.T) {}t.Fatal("node pve9 raised no memory alert after its agent left the mock estate")}++func TestLeavingMockModeReleasesFixtureAgentNodeLinksOnEveryRunningMonitor(t *testing.T) {+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)+ pinDefaultMockEstate(t)+ mustSetMockEnabled(t, true)+ defaultMonitor, tenantMonitor := startTenantMonitors(t)++ // org-b's start pass registers the fixture agents in org-b's own alert+ // manager, each owning the usage alerts of the fixture node it runs on.+ var linked models.Node+ waitForCondition(t, 30*time.Second, func() bool {+ tenantMonitor.mockHostAgentsMu.Lock()+ defer tenantMonitor.mockHostAgentsMu.Unlock()+ for _, host := range tenantMonitor.mockHostAgents {+ if host.Status != "online" || host.LinkedNodeID == "" {+ continue+ }+ for _, node := range mock.CurrentFixtureGraph().State.Nodes {+ if node.ID == host.LinkedNodeID {+ linked = node+ return true+ }+ }+ }+ return false+ }, "org-b registered no online fixture agent linked to a fixture node")++ // The demo-fixture licence sync switches through the default monitor.+ mustSetMonitorMockMode(t, defaultMonitor, false)++ // org-b's fixture agents go through the removal lifecycle too, so the+ // node an agent covered owns its metric alerts again.+ linked.Status = "online"+ linked.Memory = models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75}+ tenantMonitor.alertManager.CheckNode(linked)+ for _, alert := range tenantMonitor.alertManager.GetActiveAlerts() {+ if alert.ResourceID == linked.ID && alert.Type == "memory" {+ return+ }+ }+ t.Fatalf("org-b's node %s raised no memory alert: its fixture agent's node link outlived a switch made through the default monitor", linked.ID)+}diff --git a/internal/monitoring/monitor_mock_alerts_test.go b/internal/monitoring/monitor_mock_alerts_test.goindex 3c02565ece..ce0b725e54 100644--- a/internal/monitoring/monitor_mock_alerts_test.go+++ b/internal/monitoring/monitor_mock_alerts_test.go@@ -15,6 +15,7 @@ import ("github.com/rcourtman/pulse-go-rewrite/internal/alerts""github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"+ "github.com/rcourtman/pulse-go-rewrite/internal/config""github.com/rcourtman/pulse-go-rewrite/internal/mock""github.com/rcourtman/pulse-go-rewrite/internal/models""github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"@@ -809,6 +810,254 @@ func TestLeavingMockModeDiscardsAnInFlightConnectionCheck(t *testing.T) {}}+// startTenantMonitors starts the default organization's monitor and one+// other tenant's through MultiTenantMonitor, as the server does, each with its+// own alert manager, and waits until both monitoring loops run. An hour+// between ticks leaves each loop the pass it runs on start, so the test drives+// every later pass itself.+func startTenantMonitors(t *testing.T) (defaultMonitor, tenantMonitor *Monitor) {+ t.Helper()+ mtp, _ := newTestTenantPersistence(t)+ mtm := NewMultiTenantMonitor(&config.Config{DataPath: t.TempDir(), PVEPollingInterval: time.Hour}, mtp, nil)+ t.Cleanup(mtm.Stop)+ mtm.SetMonitorInitializer(func(monitor *Monitor) {+ cfg := monitor.alertManager.GetConfig()+ cfg.Enabled = true+ cfg.TimeThresholds = map[string]int{}+ cfg.MetricTimeThresholds = nil+ cfg.BackupDefaults.Enabled = false+ cfg.AgentDefaults.Memory = &alerts.HysteresisThreshold{Trigger: 1, Clear: 0.5}+ cfg.NodeDefaults.Memory = &alerts.HysteresisThreshold{Trigger: 1, Clear: 0.5}+ monitor.alertManager.UpdateConfig(cfg)+ })+ if err := mtp.SaveOrganization(&models.Organization{ID: "org-b", DisplayName: "Org B"}); err != nil {+ t.Fatalf("SaveOrganization(org-b): %v", err)+ }+ monitors := make([]*Monitor, 0, 2)+ for _, orgID := range []string{"default", "org-b"} {+ monitor, err := mtm.GetMonitor(orgID)+ if err != nil {+ t.Fatalf("GetMonitor(%s): %v", orgID, err)+ }+ // Start sets its runtime context before it joins the monitors a+ // switch reaches, so wait for the join itself.+ waitForCondition(t, 30*time.Second, func() bool {+ mockModeSwitchMu.Lock()+ defer mockModeSwitchMu.Unlock()+ _, running := mockModeRunningMonitors[monitor]+ return running+ }, orgID+" monitoring loop did not join mock-mode switches")+ monitors = append(monitors, monitor)+ }+ return monitors[0], monitors[1]+}++func mockHostAgentCount(monitor *Monitor) int {+ monitor.mockHostAgentsMu.Lock()+ defer monitor.mockHostAgentsMu.Unlock()+ return len(monitor.mockHostAgents)+}++func TestLeavingMockModeClearsEveryRunningTenantMonitor(t *testing.T) {+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)+ pinDefaultMockEstate(t)+ mustSetMockEnabled(t, true)+ defaultMonitor, tenantMonitor := startTenantMonitors(t)++ // Each loop runs a mock pass on start and raises the fixture's alerts in+ // its own alert manager.+ monitors := map[string]*Monitor{"default": defaultMonitor, "org-b": tenantMonitor}+ for name, monitor := range monitors {+ waitForCondition(t, 30*time.Second, func() bool {+ return len(monitor.alertManager.GetActiveAlerts()) > 0 && mockHostAgentCount(monitor) > 0+ }, name+" monitor raised no fixture alerts in mock mode")+ }++ // The mock-mode API switches through the requesting tenant's monitor.+ mustSetMonitorMockMode(t, tenantMonitor, false)++ for name, monitor := range monitors {+ if active := monitor.alertManager.GetActiveAlerts(); len(active) != 0 {+ t.Errorf("%s monitor kept %d fixture alerts after mock mode was left through org-b: %s",+ name, len(active), describeAlerts(active))+ }+ if count := mockHostAgentCount(monitor); count != 0 {+ t.Errorf("%s monitor still tracks %d fixture agents in live mode", name, count)+ }+ }+}++func TestEnteringMockModeClearsEveryRunningTenantMonitor(t *testing.T) {+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)+ pinDefaultMockEstate(t)+ defaultMonitor, tenantMonitor := startTenantMonitors(t)++ monitors := map[string]*Monitor{"default": defaultMonitor, "org-b": tenantMonitor}+ for name, monitor := range monitors {+ monitor.alertManager.CheckNode(models.Node{+ ID: name + "-live-node",+ Name: "live-node",+ Instance: "live",+ Status: "online",+ Memory: models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75},+ })+ if len(monitor.alertManager.GetActiveAlerts()) == 0 {+ t.Fatalf("live node raised no alert in %s", name)+ }+ }++ // The demo-fixture licence sync switches through the default monitor.+ mustSetMonitorMockMode(t, defaultMonitor, true)++ for name, monitor := range monitors {+ for _, alert := range monitor.alertManager.GetActiveAlerts() {+ if alert.ResourceID == name+"-live-node" {+ t.Errorf("%s kept its live %s alert after mock mode was entered through the default monitor", name, alert.Type)+ }+ }+ }+}++func TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn(t *testing.T) {+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)+ pinDefaultMockEstate(t)+ manager := newMockHostAlertTestManager(t)+ monitor := &Monitor{+ state: models.NewState(),+ alertManager: manager,+ metricsHistory: NewMetricsHistory(10, time.Hour),+ }+ // Built in live mode: New restored this live alert from disk.+ manager.CheckNode(models.Node{+ ID: "live-node",+ Name: "live-node",+ Instance: "live",+ Status: "online",+ Memory: models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75},+ })+ if len(manager.GetActiveAlerts()) == 0 {+ t.Fatal("live node raised no alert")+ }++ // Another monitor switched to mock mode before this one's Start joined.+ mustSetMockEnabled(t, true)+ started := false+ leave := monitor.joinMockModeSwitches(func(mockEnabled bool) { started = mockEnabled })+ defer leave()++ if !started {+ t.Fatal("Start was not handed mock mode")+ }+ if active := manager.GetActiveAlerts(); len(active) != 0 {+ t.Fatalf("a monitor built in live mode started in mock mode with its live alerts: %s", describeAlerts(active))+ }+ mockModeSwitchMu.Lock()+ _, running := mockModeRunningMonitors[monitor]+ mockModeSwitchMu.Unlock()+ if !running {+ t.Fatal("joined monitor is not reached by later switches")+ }+}++func TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile(t *testing.T) {+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)+ pinDefaultMockEstate(t)+ newMonitor := func() *Monitor {+ return &Monitor{+ state: models.NewState(),+ alertManager: newMockHostAlertTestManager(t),+ metricsHistory: NewMetricsHistory(10, time.Hour),+ }+ }+ switcher, starting := newMonitor(), newMonitor()+ // Built in live mode, as New records it, and not running yet.+ mockModeSwitchMu.Lock()+ starting.mockModeSwitchesSeen = mockModeSwitchCount.Load()+ mockModeSwitchMu.Unlock()++ mustSetMonitorMockMode(t, switcher, true)+ // A read path evaluates fixture data on the monitor before its Start runs.+ fixture := mock.CurrentFixtureGraph().State.Nodes[0]+ fixture.Status = "online"+ fixture.Memory = models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75}+ starting.alertManager.CheckNode(fixture)+ if len(starting.alertManager.GetActiveAlerts()) == 0 {+ t.Fatal("fixture node raised no alert")+ }+ mustSetMonitorMockMode(t, switcher, false)++ leave := starting.joinMockModeSwitches(func(bool) {})+ defer leave()+ if active := starting.alertManager.GetActiveAlerts(); len(active) != 0 {+ t.Fatalf("a monitor that missed a mock-mode round trip started in live mode with fixture alerts: %s", describeAlerts(active))+ }+}++func TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds(t *testing.T) {+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)+ pinDefaultMockEstate(t)+ manager := newMockHostAlertTestManager(t)+ monitor := &Monitor{+ state: models.NewState(),+ alertManager: manager,+ metricsHistory: NewMetricsHistory(10, time.Hour),+ }+ // Built in live mode, with a live alert, and not running yet.+ manager.CheckNode(models.Node{+ ID: "live-node",+ Name: "live-node",+ Instance: "live",+ Status: "online",+ Memory: models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75},+ })+ if len(manager.GetActiveAlerts()) == 0 {+ t.Fatal("live node raised no alert")+ }++ // Another monitor's switch already set the flag.+ mustSetMockEnabled(t, true)+ mustSetMonitorMockMode(t, monitor, true)++ if active := manager.GetActiveAlerts(); len(active) != 0 {+ t.Fatalf("SetMockMode(true) left a monitor built in live mode with its live alerts: %s", describeAlerts(active))+ }+}++func TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart(t *testing.T) {+ setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)+ pinDefaultMockEstate(t)+ ctx, cancel := context.WithCancel(context.Background())+ t.Cleanup(cancel)+ // Start has set its runtime context but not joined the switches yet.+ monitor := &Monitor{+ config: &config.Config{},+ state: models.NewState(),+ alertManager: newMockHostAlertTestManager(t),+ metricsHistory: NewMetricsHistory(10, time.Hour),+ runtimeCtx: ctx,+ }+ t.Cleanup(monitor.stopMockMetricsSampler)++ mustSetMonitorMockMode(t, monitor, true)+ monitor.mu.RLock()+ started := monitor.mockMetricsCancel != nil+ monitor.mu.RUnlock()+ if started {+ t.Fatal("a switch started the runtime of a monitor whose Start has not joined; that runtime belongs to its Start")+ }++ leave := monitor.joinMockModeSwitches(func(mockEnabled bool) {+ monitor.startModeRuntime(ctx, nil, mockEnabled)+ })+ defer leave()+ monitor.mu.RLock()+ started = monitor.mockMetricsCancel != nil+ monitor.mu.RUnlock()+ if !started {+ t.Fatal("joining in mock mode did not start the mock sampler")+ }+}+// diskAlertKeys lists the active disk-health and disk-wearout alerts as// "type instance/node device" so assertions name the disk, not the alert ID.func diskAlertKeys(manager *alerts.Manager) map[string]bool {