Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Stop mock-mode callers that read identity rebuilding the estate view
rcourtman8823440f33b5
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 2625c04957..8847722e10 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -9459,3 +9459,20 @@ pair the operator has decided about since. The agents API's unlink isunchanged. Registration, enrolment, install, update,removal, token binding and report identity are unchanged. Theunified-resources contract records the split's rules.++### Host-agent evaluation lists nodes only when an offline host needs them++`Monitor.evaluateHostAgents` (`internal/monitoring/monitor_agents.go`) runs on+every monitor tick. It used to list every Proxmox node through+`NodesSnapshot` before looking at a single host, only to give an offline host+its shared-system alert correlation (`sharedSystemAlertCorrelationForHost`,+which reads node ID, linked agent and instance). It now lists the nodes when+the first offline host that names a linked node asks, through+`structureNodes`, which in mock mode serves the view built for the current+fixture structure instead of rebuilding the estate after each metric tick. The+liveness window, the online and offline transitions, the host lifecycle, the+telemetry expiry and the alert correlation itself are unchanged, and no agent+report, enrollment, lease or command path is touched+(`TestSharedSystemAlertCorrelationForHostLazyListsNodesOnDemand` and+`TestMockStructureConsumersDoNotRebuildTheViewAfterFixtureTicks` in+`internal/monitoring/monitor_host_agents_test.go`).diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.mdindex 3ea9c617e3..79ab1face9 100644--- a/docs/release-control/v6/internal/subsystems/api-contracts.md+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md@@ -12080,3 +12080,22 @@ it. The request, status codes, response shape and the meaning of an emptydecision is added: `exclusions` still counts one link pair each.`TestResourceReportMergeSourceFilterDetachesChainMembers` reports a chainthrough the handler and the REST listing.++### The connection-degraded alert feed reads platform rows only++`buildAlertConnectionSnapshotsWithRuntimeSources` (`internal/api/connections_alerts.go`)+feeds the connection-degraded alert, and `snapshotConnectionsForAlerts` keeps+only the PVE, PBS, PMG, VMware and TrueNAS rows. It now builds the aggregator's+inputs with `aggregatorRuntimeSources.platformRowsOnly`, which leaves out the+monitor's hosts, their desired-config fingerprints and the PBS reported node+names: they only feed agent rows, which that snapshot drops, and a PBS host+alias, which it does not carry. The connections endpoint, diagnostics and the+runtime inventory sources still read them. No route, request, response, status+code, header or payload changes, and a platform row is the same with and+without those reads (`TestAlertConnectionSnapshotsMatchTheFullAggregatorOnPlatformRows`+and `TestAlertConnectionSnapshotsKeepConfiguredPlatformRowsWithoutTheHosts` in+`internal/api/runtime_inventory_sources_test.go`). The demo Patrol resolver+(`patrolResourceOperatorStateProvider` in `internal/api/router.go`) resolves a+finding's resource reference through `Monitor.GetUnifiedStructureReadState`; in+mock mode that is the view built for the current fixture structure, outside it+the read state it used before.diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex d7ee52daa5..53043e47c1 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -3634,9 +3634,34 @@ being reconfigured or a link change rebuilds the view on the next lookup; asbefore, two lookups that miss together can each build one, and a cold orinvalidated cache builds even for an ID the estate does not list. Once theview is current an unknown ID resolves to nothing and falls to the raw-resource store. The view stays keyed on the data version for every other-consumer (broadcast, `/api/state`, `GetUnifiedReadStateOrSnapshot`), which-still rebuild it on a data-version miss.+resource store. The view stays keyed on the data version for the consumers+that read what a tick moves (broadcast, `/api/state`,+`GetUnifiedReadStateOrSnapshot`), which still rebuild it on a data-version+miss.++Callers that read the estate's identity and topology only share the+structure-keyed view instead (`currentStructureUnifiedStateView`, with+`GetUnifiedStructureReadState` and `currentModeStructureReadState` over it in+`internal/monitoring/unified_structure_view.go`): the Docker-alert prune+(`pruneStaleDockerAlerts`, host IDs), the host-agent evaluation+(`evaluateHostAgents`, which runs on every monitor tick and lists nodes through+`structureNodes`, for their IDs and links, only when an offline host names a+linked node), the chart cache prewarm (metric IDs) and the demo patrol's+operator-state resolver (`patrolResourceOperatorStateProvider`, canonical ID+resolution). Such a caller can be handed a view several ticks old, so it must+not read status, metric values, timestamps, sensors or anything a link merges by+freshness, and a new consumer that does belongs on the data-version view. The+mock storage chart history (`resolveMockStorageTotal`) is one of those: a linked+pool takes its capacity from whichever source was seen last, which moves+without a structure change, so it totals each pool whose chart is not cached+from the data-version view, and `GetStorageMetricsForChartBatch` reads that view+once for all the pools of a request instead of once per pool+(`TestMockStorageChartReadsTheEstateOnlyForAnUncachedChart`). The connection-degraded alert feed+(`buildAlertConnectionSnapshotsWithRuntimeSources`) reads neither the hosts nor+the PBS instances: it keeps the platform rows and drops every agent row, and a+PBS row's reported node name only becomes a host alias that the alert snapshot+does not carry, so it asks the aggregator for platform rows only+(`aggregatorRuntimeSources.platformRowsOnly`).The revision is read before the data version when a view is cached, and thefixture advances its data version before its revision@@ -3658,16 +3683,36 @@ busy loops missed the view cache on 675 of 929 requests at a 16-second averagebuild and had not served by 606 s; with this change it served at 148 s (13 son the core alone). This was not a regression: builds from 1a094646d7 andd2ac0c5112 stall identically under equal contention and both start in 14 to-16 seconds at normal load. The remaining time on a starved host is the other-consumers of the shared view, which rebuild it whenever they find it stale (on-the starved run, 42 view requests in a start of 194 s were all misses, from the-Docker-alert prune, the connection snapshots and the host-agent evaluation),-and the monitor's repeated full resource-store passes at startup. Loops that-still reach the data-version view once per item are unchanged: the mock storage-chart history (`resolveMockStorageTotal` per uncached pool), the demo patrol's-finding adds (`patrolResourceOperatorStateProvider`) and, with-`PULSE_MOCK_KEEP_REAL_POLLING`, the per-node polling lookups-(`linkedHostForNode`, `getHostAgentTemperatureForNode`).+16 seconds at normal load. The remaining time on a starved host was the other+consumers of the shared view, which rebuilt it whenever they found it stale (on+the starved run, 42 view requests in a start of 194 s were all misses: the+Docker-alert prune 18, the connection snapshots 13, the host-agent evaluation 8,+the chart prewarm 2 and the metrics-target lookup 1), and the monitor's repeated full resource-store+passes at startup. The consumers above now ride the structure view or skip the+read, and the passes are one (see "Store and provider wiring publishes the+estate once, and only when it changed"). The loops that still reach the+data-version view once per item are the per-node polling lookups of a monitor+that keeps real polling in mock mode (`PULSE_MOCK_KEEP_REAL_POLLING`):+`linkedHostForNode`, `getHostAgentTemperatureForNode` and+`carriedTemperatureOutlivesAgentLease`. They read a host's status, disks,+sensors and report lease, which a tick moves, so the structure view cannot+answer them; hoisting the read out of the per-node loop would need the read+state threaded through the node poller, and they run in the polling goroutines+after startup, not before the listener opens.++Measured on pulse-dev (single runs on a shared host at load 50 to 90, the+process pinned to one core with GOMAXPROCS=1, probe builds counting estate+builds): before the listener answered, a start of the previous build made 19+store passes and 25 estate view builds; with the wiring change alone it made one+pass and two builds (the first metrics-target lookup and the Docker-alert+prune); with the structure view as well it makes one pass and one build. With no+client connected, a mock-mode process kept building the estate about once every+nine seconds (33 builds in the 299-second life of a run with the wiring change+alone) and used 12.3 seconds of CPU over a 180-second window; with the+structure view it built the estate twice in all, both at startup, and used 3.3+seconds over the same window. The previous build used 9.0 seconds over that+window in a separate run, so read the 12.3 against 3.3 as the comparison, not as+exact percentages.`TestMockMetricsTargetLookupsDoNotRebuildTheViewAfterFixtureTicks` and`TestMockMetricsTargetForUnknownIDDoesNotRebuildTheViewAfterFixtureTicks` fail ondiff --git a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md b/docs/release-control/v6/internal/subsystems/performance-and-scalability.mdindex a7372f932d..eeb88823d2 100644--- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md+++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md@@ -3748,9 +3748,18 @@ alert loop, which is what the one-off measurement above showed. Run bothbenchmarks with `-benchtime=3x`, since the registry case rebuilds its ingestoutside the timer. `TestMockMetricsTargetLookupsDoNotRebuildTheViewAfterFixtureTicks`is the guard that fails on a rebuild per lookup. The remaining startup cost-under starvation is the monitor's repeated full resource-store passes (one per-supplemental provider registration) and the other consumers of the shared view,-which rebuild it whenever they find it stale; neither is changed here.+under starvation was the monitor's repeated full resource-store passes (one per+supplemental provider registration, 19 before the listener answered) and the+other consumers of the shared view, which rebuilt it whenever they found it+stale. A start now makes one pass and one estate build, and a mock-mode process+with no client connected no longer builds the estate on every monitor tick (the+data-version consumers, broadcast and `/api/state`, still rebuild it after a+tick for a connected client). The callers that read+identity and topology only share the view built for the current fixture+structure, and the connection-degraded alert feed no longer asks the monitor for+hosts or PBS instances it throws away (see "Mock-mode metrics-target lookups+ride the fixture structure revision" in the monitoring contract for the callers+and the measurements).### Update evidence reuses the bounded node observationdiff --git a/docs/release-control/v6/internal/subsystems/security-privacy.md b/docs/release-control/v6/internal/subsystems/security-privacy.mdindex 39579156bc..268ff47963 100644--- a/docs/release-control/v6/internal/subsystems/security-privacy.md+++ b/docs/release-control/v6/internal/subsystems/security-privacy.md@@ -3168,3 +3168,18 @@ that node's row in the same monitor. This only withholds or substitutes valuesthe caller could already read; the new `NodeView` and `DockerHostView``SourceStatus` accessors expose no field beyond the delivery state the registryalready records on the resource.++### Patrol operator-state resolution keeps its org scoping++`patrolResourceOperatorStateProvider` (`internal/api/router.go`) resolves a+finding's resource reference to a canonical ID before it reads the operator+state, and now reads the monitor's structure read state+(`Monitor.GetUnifiedStructureReadState`) for that. The org is resolved by+`resolveMonitorForOrg` as before, the operator-state store is the org's own, and+the resolver is identity-only. In mock mode the read state can be several metric+ticks old; the fixture moves no pod between nodes and re-IDs nothing on a tick+(pod node names are set when the estate is built, and they feed the platform+aliases a reference can resolve through), so a reference resolves to the same+canonical ID for as long as the fixture structure holds. Outside mock mode it+is the read state the resolver used before. No authentication, authorization, token or+tenant-isolation path is touched.diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex a1b9dc254c..ab402ed7fe 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -6784,3 +6784,13 @@ excludes the links that join a member carrying the named source to the restof the merged resource. That changes which pairs the report excludes, throughthe same exclusion write as before; no schema, backup, retention, migration or recovery path is added ormoved.++### Mock-mode structure views open no storage or recovery path++`internal/api/connections_alerts.go` and `internal/api/router.go` now ask the+monitor for less in mock mode: the connection-degraded alert feed no longer+reads the hosts and PBS instances it discards, and the demo Patrol resolver+reads the structure view (see "Mock-mode metrics-target lookups ride the+fixture structure revision" in the monitoring contract). Real mode, its stores,+backups, retention, snapshots, migrations and recovery paths are unchanged, and+none is added or moved.diff --git a/internal/api/connections_alerts.go b/internal/api/connections_alerts.goindex 0a8dbdbf41..acd8c948d3 100644--- a/internal/api/connections_alerts.go+++ b/internal/api/connections_alerts.go@@ -15,6 +15,12 @@ type aggregatorRuntimeSources struct {orgID stringvmwarePoller *monitoring.VMwarePollertruenasPoller *monitoring.TrueNASPoller+ // platformRowsOnly leaves out what only the agent rows and the PBS host+ // aliases read: the monitor's hosts and PBS instances. A caller that keeps+ // the platform rows alone (the connection-degraded alert feed) would+ // otherwise ask the monitor for both, and in mock mode each ask rebuilds+ // the whole estate view whenever a metric tick has made it stale.+ platformRowsOnly bool}func buildAggregatorInputsWithRuntimeSources(@@ -61,11 +67,15 @@ func buildAggregatorInputsWithRuntimeSources(}if monitor != nil {- inputs.hosts = monitor.HostsSnapshot()- inputs.agentDesiredConfigs = connectionAgentDesiredConfigFingerprints(monitor, inputs.hosts, inputs.apiTokens)+ if runtime.platformRowsOnly {+ inputs.hosts = []models.Host{}+ } else {+ inputs.hosts = monitor.HostsSnapshot()+ inputs.agentDesiredConfigs = connectionAgentDesiredConfigFingerprints(monitor, inputs.hosts, inputs.apiTokens)+ inputs.pbsReportedNodeNames = pbsReportedNodeNamesByInstance(monitor.PBSInstancesSnapshot())+ }inputs.instanceHealth = instanceHealthByKey(monitor.SchedulerHealth())inputs.availabilityStatuses = monitor.AvailabilityStatusSnapshot()- inputs.pbsReportedNodeNames = pbsReportedNodeNamesByInstance(monitor.PBSInstancesSnapshot())// A non-default org's cfg is its monitor's detached config copy// (#1619), which a settings save does not update: saved PBS and PMG// intervals reach the monitor only as runtime overrides. Scale the@@ -155,6 +165,10 @@ func buildAlertConnectionSnapshotsWithRuntimeSources(monitor *monitoring.Monitor,runtime aggregatorRuntimeSources,) []alerts.ConnectionSnapshot {+ // snapshotConnectionsForAlerts keeps the platform rows (PVE, PBS, PMG,+ // VMware, TrueNAS) and drops every agent row, and a PBS row's reported node+ // name only becomes a host alias that the snapshot does not carry.+ runtime.platformRowsOnly = trueinputs := buildAggregatorInputsWithRuntimeSources(ctx, cfg, persistence, monitor, runtime)return snapshotConnectionsForAlerts(buildConnections(inputs))}diff --git a/internal/api/router.go b/internal/api/router.goindex 30c3971cb5..fe2766d9e5 100644--- a/internal/api/router.go+++ b/internal/api/router.go@@ -2242,7 +2242,7 @@ func (r *Router) patrolResourceOperatorStateProvider(orgID string) ai.ResourceOpreturn ai.ResourceOperatorStateProjection{}, false}canonicalID := resourceRef- if resolver, ok := r.resolveMonitorForOrg(orgID).GetUnifiedReadStateOrSnapshot().(canonicalResourceIDResolver); ok {+ if resolver, ok := r.resolveMonitorForOrg(orgID).GetUnifiedStructureReadState().(canonicalResourceIDResolver); ok {if resolvedID, resolved := resolver.ResolveCanonicalResourceID(resourceRef); resolved {canonicalID = resolvedID}diff --git a/internal/api/runtime_inventory_sources_test.go b/internal/api/runtime_inventory_sources_test.goindex cdbf40953f..5b56c83c24 100644--- a/internal/api/runtime_inventory_sources_test.go+++ b/internal/api/runtime_inventory_sources_test.go@@ -11,6 +11,7 @@ import ("time""github.com/rcourtman/pulse-go-rewrite/internal/config"+ "github.com/rcourtman/pulse-go-rewrite/internal/mock""github.com/rcourtman/pulse-go-rewrite/internal/monitoring")@@ -314,3 +315,149 @@ func TestRuntimeInventorySourcesHandlerRejectsNonGET(t *testing.T) {t.Fatalf("status = %d, want 405 (%s)", rec.Code, rec.Body.String())}}++// mockMonitorWithStaleView returns a mock-mode monitor whose estate view was+// built before the fixture last ticked, and the structure read state it holds.+func mockMonitorWithStaleView(t *testing.T) (*monitoring.Monitor, *config.Config) {+ t.Helper()+ previous := mock.IsMockEnabled()+ previousConfig := mock.GetConfig()+ estate := mock.DefaultConfig+ estate.NodeCount = 4+ estate.UpdateInterval = time.Second+ mock.SetMockConfig(estate)+ if err := mock.SetEnabled(true); err != nil {+ t.Fatalf("enable mock mode: %v", err)+ }+ t.Cleanup(func() {+ _ = mock.SetEnabled(false)+ mock.SetMockConfig(previousConfig)+ _ = mock.SetEnabled(previous)+ })++ cfg := &config.Config{DataPath: t.TempDir()}+ monitor, err := monitoring.New(cfg)+ if err != nil {+ t.Fatalf("new monitor: %v", err)+ }+ t.Cleanup(monitor.Stop)++ monitor.GetUnifiedReadStateOrSnapshot()+ tick := mock.FixtureDataVersion()+ deadline := time.Now().Add(5 * time.Minute)+ for mock.FixtureDataVersion() <= tick {+ if time.Now().After(deadline) {+ t.Fatal("the mock fixture did not tick")+ }+ time.Sleep(20 * time.Millisecond)+ }+ return monitor, cfg+}++// The connection-degraded alert feed keeps the platform rows (PVE, PBS, PMG,+// VMware, TrueNAS) and drops every agent row, so it has no use for the+// monitor's hosts or PBS instances. In mock mode each of those reads rebuilds+// the whole estate view once a metric tick has made it stale, and the feed+// runs on startup's critical path.+func TestAlertConnectionSnapshotsDoNotRebuildTheMockViewAfterFixtureTicks(t *testing.T) {+ monitor, cfg := mockMonitorWithStaleView(t)+ warm := monitor.GetUnifiedStructureReadState()++ buildAlertConnectionSnapshotsWithRuntimeSources(context.Background(), cfg, nil, monitor, aggregatorRuntimeSources{orgID: "default"})+ if monitor.GetUnifiedStructureReadState() != warm {+ t.Fatal("the alert connection feed rebuilt the unified view after a metric tick; it reads no host or PBS instance")+ }++ // Control: the connections endpoint does read them, and the test would+ // pass vacuously if a rebuild left no trace.+ buildAggregatorInputsWithRuntimeSources(context.Background(), cfg, nil, monitor, aggregatorRuntimeSources{orgID: "default"})+ if monitor.GetUnifiedStructureReadState() == warm {+ t.Fatal("reading the hosts did not rebuild the stale view, so this test cannot see a rebuild")+ }+}++// Leaving the hosts and PBS instances out must not change a platform row. The+// lean inputs are the full ones with exactly what platformRowsOnly omits taken+// away, so a supplemental fixture tick between two reads cannot make the rows+// differ for a reason other than the omission.+func TestAlertConnectionSnapshotsMatchTheFullAggregatorOnPlatformRows(t *testing.T) {+ monitor, cfg := mockMonitorWithStaleView(t)++ full := buildAggregatorInputsWithRuntimeSources(context.Background(), cfg, nil, monitor, aggregatorRuntimeSources{orgID: "default"})+ lean := buildAggregatorInputsWithRuntimeSources(context.Background(), cfg, nil, monitor, aggregatorRuntimeSources{orgID: "default", platformRowsOnly: true})++ if len(full.hosts) == 0 {+ t.Fatal("the mock estate lists no host, so the comparison proves nothing")+ }+ if len(lean.hosts) != 0 || lean.agentDesiredConfigs != nil || lean.pbsReportedNodeNames != nil {+ t.Fatalf("platformRowsOnly kept hosts=%d desiredConfigs=%v pbsNodeNames=%v", len(lean.hosts), lean.agentDesiredConfigs, lean.pbsReportedNodeNames)+ }++ stripped := full+ stripped.hosts = lean.hosts+ stripped.agentDesiredConfigs = nil+ stripped.pbsReportedNodeNames = nil+ want := snapshotConnectionsForAlerts(buildConnections(full))+ got := snapshotConnectionsForAlerts(buildConnections(stripped))+ if len(want) == 0 {+ t.Fatal("the mock estate yields no platform connection")+ }+ if !reflect.DeepEqual(got, want) {+ t.Fatalf("platform connections changed without the hosts:\n got %+v\nwant %+v", got, want)+ }+}++// The patrol resolver turns a finding's resource reference into a canonical ID.+// It reads identity only, so in mock mode it must not rebuild the estate view+// after a metric tick (it runs once per finding).+func TestPatrolOperatorStateResolverDoesNotRebuildTheMockViewAfterFixtureTicks(t *testing.T) {+ monitor, cfg := mockMonitorWithStaleView(t)+ warm := monitor.GetUnifiedStructureReadState()++ router := &Router{+ resourceHandlers: NewResourceHandlers(&config.Config{DataPath: cfg.DataPath}),+ monitor: monitor,+ }+ t.Cleanup(router.ShutdownResourceStores)+ provider := router.patrolResourceOperatorStateProvider("default")+ provider.OperatorStateProjection("some-finding-resource", time.Now())++ if monitor.GetUnifiedStructureReadState() != warm {+ t.Fatal("the patrol operator-state resolver rebuilt the unified view after a metric tick; it only resolves identity")+ }+}++// Mock mode drops the configured PVE, PBS and PMG instances from the ledger, so+// their rows are checked here with mock mode off: the platform rows an alert+// is raised from are the same with and without the hosts and PBS reads.+func TestAlertConnectionSnapshotsKeepConfiguredPlatformRowsWithoutTheHosts(t *testing.T) {+ previousMock := mock.IsMockEnabled()+ if err := mock.SetEnabled(false); err != nil {+ t.Fatalf("disable mock mode: %v", err)+ }+ t.Cleanup(func() { _ = mock.SetEnabled(previousMock) })+ monitor, err := monitoring.New(&config.Config{DataPath: t.TempDir()})+ if err != nil {+ t.Fatalf("new monitor: %v", err)+ }+ t.Cleanup(monitor.Stop)+ cfg := &config.Config{+ DataPath: t.TempDir(),+ PVEInstances: []config.PVEInstance{{Name: "pve-lab", Host: "https://pve-lab.example:8006"}},+ PBSInstances: []config.PBSInstance{{Name: "pbs-lab", Host: "https://pbs-lab.example:8007"}},+ PMGInstances: []config.PMGInstance{{Name: "pmg-lab", Host: "https://pmg-lab.example:8006"}},+ }++ full := buildAggregatorInputsWithRuntimeSources(context.Background(), cfg, nil, monitor, aggregatorRuntimeSources{orgID: "default"})+ lean := buildAggregatorInputsWithRuntimeSources(context.Background(), cfg, nil, monitor, aggregatorRuntimeSources{orgID: "default", platformRowsOnly: true})+ lean.now = full.now++ want := snapshotConnectionsForAlerts(buildConnections(full))+ got := snapshotConnectionsForAlerts(buildConnections(lean))+ if len(want) != 3 {+ t.Fatalf("expected the PVE, PBS and PMG rows, got %+v", want)+ }+ if !reflect.DeepEqual(got, want) {+ t.Fatalf("platform connections changed without the hosts:\n got %+v\nwant %+v", got, want)+ }+}diff --git a/internal/monitoring/metrics_target_resolution.go b/internal/monitoring/metrics_target_resolution.goindex 6d81f731a3..8d6f57f4b3 100644--- a/internal/monitoring/metrics_target_resolution.go+++ b/internal/monitoring/metrics_target_resolution.go@@ -43,19 +43,7 @@ func (m *Monitor) mockMetricsTarget(resourceID string) (*unifiedresources.Metricreturn nil, false}- structure := mock.FixtureStructureRevision()- links := m.resourceStoreManualLinks()-- m.mockUnifiedViewMu.Lock()- view := m.mockUnifiedView- reusable := m.mockUnifiedViewValid &&- m.mockUnifiedViewStructure == structure &&- sameManualLinks(m.mockUnifiedViewLinks, links)- m.mockUnifiedViewMu.Unlock()-- if !reusable {- view = m.currentUnifiedStateView()- }+ view := m.currentStructureUnifiedStateView()if view.metricsTargets == nil {return nil, true}diff --git a/internal/monitoring/mock_chart_history.go b/internal/monitoring/mock_chart_history.goindex 507cd858ba..500e1ed165 100644--- a/internal/monitoring/mock_chart_history.go+++ b/internal/monitoring/mock_chart_history.go@@ -6,6 +6,7 @@ import ("time""github.com/rcourtman/pulse-go-rewrite/internal/mock"+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources")var (@@ -171,9 +172,14 @@ func latestMetricPoint(points []MetricPoint) (MetricPoint, bool) {return latest, true}-func (m *Monitor) resolveMockStorageTotal(storageID string, inMemory map[string][]MetricPoint) float64 {- readState := m.GetUnifiedReadStateOrSnapshot()- if readState != nil {+// resolveMockStorageTotal totals one pool for a chart. readState lists the pools+// and is read here only if the pool's chart is not cached, so a request over+// many pools reads the estate view once (see GetStorageMetricsForChartBatch)+// and not once per pool, which on a starved process rebuilt the view for each.+// It is not the structure view: a linked pool takes its capacity from whichever+// source was seen last, and that moves without a structure change.+func (m *Monitor) resolveMockStorageTotal(storageID string, inMemory map[string][]MetricPoint, readState func() unifiedresources.ReadState) float64 {+ if readState := readState(); readState != nil {for _, pool := range readState.StoragePools() {if pool == nil || strings.TrimSpace(pool.SourceID()) != strings.TrimSpace(storageID) {continue@@ -191,8 +197,8 @@ func (m *Monitor) resolveMockStorageTotal(storageID string, inMemory map[string]return 0}-func (m *Monitor) mockStorageMetricsForChart(storageID string, duration time.Duration, inMemory map[string][]MetricPoint) map[string][]MetricPoint {- total := m.resolveMockStorageTotal(storageID, inMemory)+func (m *Monitor) mockStorageMetricsForChart(storageID string, duration time.Duration, inMemory map[string][]MetricPoint, readState func() unifiedresources.ReadState) map[string][]MetricPoint {+ total := m.resolveMockStorageTotal(storageID, inMemory, readState)if total <= 0 {return cloneMetricPointMap(inMemory)}diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.goindex 72aff4182a..a3daa80544 100644--- a/internal/monitoring/monitor.go+++ b/internal/monitoring/monitor.go@@ -3610,7 +3610,10 @@ func (m *Monitor) NodesSnapshot() []models.Node {if m == nil {return nil}- readState := m.GetUnifiedReadStateOrSnapshot()+ return nodesSnapshotFromReadState(m.GetUnifiedReadStateOrSnapshot())+}++func nodesSnapshotFromReadState(readState unifiedresources.ReadState) []models.Node {if readState == nil {return nil}diff --git a/internal/monitoring/monitor_agents.go b/internal/monitoring/monitor_agents.goindex 0d93930105..7a391fc4a7 100644--- a/internal/monitoring/monitor_agents.go+++ b/internal/monitoring/monitor_agents.go@@ -5227,7 +5227,15 @@ func hostAgentHealthWindow(interval int) time.Duration {// evaluateHostAgents updates health for host agents based on last report time.func (m *Monitor) evaluateHostAgents(now time.Time) {hosts := m.state.GetHosts()- nodes := m.NodesSnapshot()+ // Only an offline host that names a linked node asks which node it is+ // linked to, and only for the node's identity, so the nodes are listed when+ // the first one does, from the structure view: this runs on every monitor+ // tick, and in mock mode a listing of the data-version view rebuilds the+ // whole estate after each metric tick.+ nodes := sync.OnceValue(m.structureNodes)+ offlineCorrelation := func(host models.Host) *alerts.AlertCorrelation {+ return sharedSystemAlertCorrelationForHostLazy(host, nodes)+ }liveHostIDs := make(map[string]struct{}, len(hosts))resourceRefreshNeeded := falsefor _, host := range hosts {@@ -5279,7 +5287,7 @@ func (m *Monitor) evaluateHostAgents(now time.Time) {if m.alertManager != nil {m.alertManager.HandleHostOfflineWithCorrelation(hostCopy,- sharedSystemAlertCorrelationForHost(hostCopy, nodes),+ offlineCorrelation(hostCopy),)}}@@ -5304,7 +5312,7 @@ func (m *Monitor) evaluateHostAgents(now time.Time) {if m.alertManager != nil {m.alertManager.HandleHostOfflineWithCorrelation(host,- sharedSystemAlertCorrelationForHost(host, nodes),+ offlineCorrelation(host),)}}@@ -5314,6 +5322,15 @@ func (m *Monitor) evaluateHostAgents(now time.Time) {}}+// sharedSystemAlertCorrelationForHostLazy is sharedSystemAlertCorrelationForHost+// for a caller that lists the nodes only when the host names a linked node.+func sharedSystemAlertCorrelationForHostLazy(host models.Host, nodes func() []models.Node) *alerts.AlertCorrelation {+ if strings.TrimSpace(host.LinkedNodeID) == "" || strings.TrimSpace(host.ID) == "" {+ return nil+ }+ return sharedSystemAlertCorrelationForHost(host, nodes())+}+func sharedSystemAlertCorrelationForHost(host models.Host, nodes []models.Node) *alerts.AlertCorrelation {linkedNodeID := strings.TrimSpace(host.LinkedNodeID)hostID := strings.TrimSpace(host.ID)diff --git a/internal/monitoring/monitor_alert_sync.go b/internal/monitoring/monitor_alert_sync.goindex a53f30a6e6..eb55d9aa5d 100644--- a/internal/monitoring/monitor_alert_sync.go+++ b/internal/monitoring/monitor_alert_sync.go@@ -175,8 +175,9 @@ func (m *Monitor) pruneStaleDockerAlerts() bool {// The inventory is the fixture read view in mock mode. A prune that read// it must not remove Docker alerts the other mode raised after a switch.+ // Only the hosts' IDs matter, so the view may predate the last metric ticks.scope := m.mockModeFence.begin()- readState := m.currentModeReadState()+ readState := m.currentModeStructureReadState()if readState == nil {return false}diff --git a/internal/monitoring/monitor_host_agents_test.go b/internal/monitoring/monitor_host_agents_test.goindex 46cf04cea2..a859ed65f4 100644--- a/internal/monitoring/monitor_host_agents_test.go+++ b/internal/monitoring/monitor_host_agents_test.go@@ -6173,6 +6173,151 @@ func TestMockMetricsTargetRebuildsWhenTheLinkListMoves(t *testing.T) {}}+// The estate view is keyed on the fixture data version, which advances on+// every metric tick, and a build costs about half a second of CPU on an idle+// core (several seconds on a starved process). Callers that read the estate's+// identity and topology only share the view built for the current fixture+// structure instead (currentStructureUnifiedStateView); startup runs every one+// of them before the listener opens, and on a starved process each used to+// build its own estate.+func TestMockStructureConsumersDoNotRebuildTheViewAfterFixtureTicks(t *testing.T) {+ useMockEstate(t, 4, time.Second)+ m := &Monitor{+ state: models.NewState(),+ resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(unifiedresources.NewMemoryStore())),+ alertManager: alerts.NewManager(),+ }+ t.Cleanup(m.alertManager.Stop)++ warm := m.currentUnifiedStateView()+ waitForFixtureTicks(t, 2)++ consumers := []struct {+ name string+ run func()+ }{+ {"structure read state", func() { m.GetUnifiedStructureReadState() }},+ {"mode structure read state", func() { m.currentModeStructureReadState() }},+ {"structure node listing", func() { m.structureNodes() }},+ {"docker alert prune", func() { m.pruneStaleDockerAlerts() }},+ {"host agent evaluation", func() { m.evaluateHostAgents(time.Now()) }},+ {"chart cache prewarm", func() { m.prewarmMockDashboardChartCaches() }},+ {"host agent evaluation with an offline linked host", func() {+ m.state.UpsertHost(models.Host{ID: "offline-agent", Hostname: "offline-agent", LinkedNodeID: "pve1-node", Status: "online", LastSeen: time.Now().Add(-24 * time.Hour)})+ m.evaluateHostAgents(time.Now())+ }},+ }+ for _, consumer := range consumers {+ consumer.run()+ after, ok := publishedMockView(m)+ if !ok || after.readState != warm.readState {+ t.Fatalf("%s rebuilt the unified view after a metric tick; it only reads identity and topology, which the view built for the current fixture structure holds", consumer.name)+ }+ }+}++// The host-agent evaluation lists nodes only for a host that names a linked+// node, and only when it has to.+func TestSharedSystemAlertCorrelationForHostLazyListsNodesOnDemand(t *testing.T) {+ listed := 0+ nodes := func() []models.Node {+ listed+++ return []models.Node{{ID: "node-1", Instance: "lab", LinkedAgentID: "host-1"}}+ }++ if got := sharedSystemAlertCorrelationForHostLazy(models.Host{ID: "host-1"}, nodes); got != nil || listed != 0 {+ t.Fatalf("an unlinked host listed nodes %d times and correlated %v", listed, got)+ }+ if got := sharedSystemAlertCorrelationForHostLazy(models.Host{LinkedNodeID: "node-1"}, nodes); got != nil || listed != 0 {+ t.Fatalf("a host without an ID listed nodes %d times and correlated %v", listed, got)+ }+ got := sharedSystemAlertCorrelationForHostLazy(models.Host{ID: "host-1", LinkedNodeID: "node-1"}, nodes)+ if listed != 1 {+ t.Fatalf("a linked host listed nodes %d times, want once", listed)+ }+ want := sharedSystemAlertCorrelationForHost(models.Host{ID: "host-1", LinkedNodeID: "node-1"}, nodes())+ if got == nil || want == nil || got.Key != want.Key {+ t.Fatalf("a linked host correlated %+v, want %+v", got, want)+ }+}++// The data-version view must keep serving the callers that read values: a+// consumer of statuses, sensors and metrics must see the tick, so the+// structure view cannot have replaced it.+func TestMockDataVersionViewStillRebuildsAfterFixtureTicks(t *testing.T) {+ useMockEstate(t, 3, time.Second)+ m := newMockEstateMonitor()++ warm := m.currentUnifiedStateView()+ waitForFixtureTicks(t, 1)++ if m.GetUnifiedReadStateOrSnapshot() == warm.readState {+ t.Fatal("GetUnifiedReadStateOrSnapshot served the view from before a metric tick")+ }+}++// The structure view answers for the estate it was built from. A structural+// change must reach its callers, not keep serving the old resource set.+func TestMockStructureReadStateFollowsAStructuralFixtureChange(t *testing.T) {+ useMockEstate(t, 3, 5*time.Minute)+ m := newMockEstateMonitor()++ before := m.GetUnifiedStructureReadState()+ if before == nil || len(before.Nodes()) == 0 {+ t.Fatal("the mock estate lists no nodes")+ }++ grown := mock.GetConfig()+ grown.NodeCount+++ mock.SetMockConfig(grown)++ after := m.GetUnifiedStructureReadState()+ if after == before {+ t.Fatal("the structure view outlived a structural fixture change")+ }+ if len(after.Nodes()) <= len(before.Nodes()) {+ t.Fatalf("the structure view lists %d nodes after the estate grew from %d", len(after.Nodes()), len(before.Nodes()))+ }+}++// An operator link folds one resource into another; the structure view must+// stop listing the folded resource once the resource store applies the link.+func TestMockStructureReadStateFollowsAnOperatorLink(t *testing.T) {+ useMockEstate(t, 4, 5*time.Minute)+ store := unifiedresources.NewMemoryStore()+ m := &Monitor{+ state: models.NewState(),+ resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(store)),+ alertManager: alerts.NewManager(),+ }+ t.Cleanup(m.alertManager.Stop)++ var vmID, agentID string+ for _, resource := range m.currentStructureUnifiedStateView().resources {+ switch {+ case vmID == "" && resource.Type == unifiedresources.ResourceTypeVM && resource.Agent == nil:+ vmID = resource.ID+ case agentID == "" && resource.Type == unifiedresources.ResourceTypeAgent && resource.Agent != nil &&+ len(resource.Sources) == 1 && resource.Sources[0] == unifiedresources.SourceAgent:+ agentID = resource.ID+ }+ }+ if vmID == "" || agentID == "" {+ t.Fatalf("fixture graph needs an agentless VM and a standalone agent, got vm=%q agent=%q", vmID, agentID)+ }++ if err := store.AddLink(unifiedresources.ResourceLink{ResourceA: vmID, ResourceB: agentID, PrimaryID: vmID}); err != nil {+ t.Fatalf("AddLink: %v", err)+ }+ m.updateResourceStore(models.StateSnapshot{}, m.mockModeFence.begin())++ for _, resource := range m.currentStructureUnifiedStateView().resources {+ if resource.ID == agentID {+ t.Fatalf("the structure view still lists the agent %q the operator folded into %q", agentID, vmID)+ }+ }+}+// TestMockModeDiscardsRealHostReports pins the push-side of the mock clean// room. Mock mode already suspends pull-based PVE/PBS/PMG collection, but a// real agent keeps POSTing regardless; ingesting those reports lands a realdiff --git a/internal/monitoring/monitor_metrics.go b/internal/monitoring/monitor_metrics.goindex e2eb1e9a16..4b0c811f95 100644--- a/internal/monitoring/monitor_metrics.go+++ b/internal/monitoring/monitor_metrics.go@@ -3,6 +3,7 @@ package monitoringimport ("sort""strings"+ "sync""time""github.com/rcourtman/pulse-go-rewrite/internal/mock"@@ -226,7 +227,7 @@ func (m *Monitor) GetStorageMetricsForChart(storageID string, duration time.DurainMemoryResult = m.metricsHistory.GetAllStorageMetrics(storageID, duration)}if mock.IsMockEnabled() {- return m.mockStorageMetricsForChartCached(storageID, duration, inMemoryResult)+ return m.mockStorageMetricsForChartCached(storageID, duration, inMemoryResult, m.GetUnifiedReadStateOrSnapshot)}if m.metricsStore == nil {return inMemoryResult@@ -729,12 +730,14 @@ func (m *Monitor) GetStorageMetricsForChartBatch(result := make(map[string]map[string][]MetricPoint, len(storageIDs))if mock.IsMockEnabled() {+ // One read of the estate view for every pool that needs a total.+ readState := sync.OnceValue(m.GetUnifiedReadStateOrSnapshot)for _, sid := range storageIDs {inMemory := map[string][]MetricPoint{}if m.metricsHistory != nil {inMemory = m.metricsHistory.GetAllStorageMetrics(sid, duration)}- result[sid] = m.mockStorageMetricsForChartCached(sid, duration, inMemory)+ result[sid] = m.mockStorageMetricsForChartCached(sid, duration, inMemory, readState)}return result}@@ -1179,7 +1182,7 @@ func (m *Monitor) prewarmMockWorkloadChartCaches(duration time.Duration) {}requests := boundMockWorkloadChartPrewarmRequests(- mockWorkloadChartRequestsForReadState(m.GetUnifiedReadStateOrSnapshot()),+ mockWorkloadChartRequestsForReadState(m.GetUnifiedStructureReadState()),)if len(requests.vms) > 0 {_ = m.GetGuestMetricsForChartBatch("vm", requests.vms, duration)@@ -1309,6 +1312,7 @@ func (m *Monitor) mockStorageMetricsForChartCached(storageID string,duration time.Duration,inMemoryResult map[string][]MetricPoint,+ readState func() unifiedresources.ReadState,) map[string][]MetricPoint {cacheKey := mockChartMetricMapCacheKey{kind: "storage",@@ -1325,7 +1329,7 @@ func (m *Monitor) mockStorageMetricsForChartCached(computed = downsampleMetricMapForMockChart(inMemoryResult, duration)} else {computed = downsampleMetricMapForMockChart(- m.mockStorageMetricsForChart(storageID, duration, inMemoryResult),+ m.mockStorageMetricsForChart(storageID, duration, inMemoryResult, readState),duration,)}diff --git a/internal/monitoring/monitor_metrics_slo_test.go b/internal/monitoring/monitor_metrics_slo_test.goindex f59de6ea6e..756177dd3a 100644--- a/internal/monitoring/monitor_metrics_slo_test.go+++ b/internal/monitoring/monitor_metrics_slo_test.go@@ -823,3 +823,33 @@ func TestMockChartCacheInvalidatesAfterMockHistoryRefresh(t *testing.T) {}, lastPoint)}}++// A storage chart request totals each pool whose chart is not cached, and a+// linked pool takes its capacity from whichever source was seen last, which+// moves without a structure change, so the totals read the data-version view.+// A batch hands every pool one shared read (GetStorageMetricsForChartBatch):+// the view is rebuilt after a tick, and on a starved process a read per pool+// rebuilt it for each. A pool whose chart is cached reads nothing.+func TestMockStorageChartReadsTheEstateOnlyForAnUncachedChart(t *testing.T) {+ useMockEstate(t, 4, 5*time.Minute)+ m := newMockEstateMonitor()+ pools := m.currentUnifiedStateView().readState.StoragePools()+ if len(pools) == 0 {+ t.Fatal("the mock estate lists no storage pool")+ }+ id := pools[0].SourceID()++ reads := 0+ readState := func() unifiedresources.ReadState {+ reads+++ return m.GetUnifiedReadStateOrSnapshot()+ }+ first := m.mockStorageMetricsForChartCached(id, time.Hour, nil, readState)+ if reads != 1 || len(first["total"]) == 0 {+ t.Fatalf("an uncached chart read the estate %d times and returned %d total points, want one read and a total series", reads, len(first["total"]))+ }+ m.mockStorageMetricsForChartCached(id, time.Hour, nil, readState)+ if reads != 1 {+ t.Fatalf("a cached chart read the estate again: %d reads", reads)+ }+}diff --git a/internal/monitoring/unified_structure_view.go b/internal/monitoring/unified_structure_view.gonew file mode 100644index 0000000000..00c7fe8d24--- /dev/null+++ b/internal/monitoring/unified_structure_view.go@@ -0,0 +1,78 @@+package monitoring++import (+ "github.com/rcourtman/pulse-go-rewrite/internal/mock"+ "github.com/rcourtman/pulse-go-rewrite/internal/models"+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"+)++// cachedMockStructureView returns the mock view the monitor has cached when it+// was built for the fixture's current structure revision and operator link+// list, however many metric ticks ago. The second result is false when there is+// no such view, and the first is then not for use.+func (m *Monitor) cachedMockStructureView() (monitorUnifiedStateView, bool) {+ structure := mock.FixtureStructureRevision()+ links := m.resourceStoreManualLinks()++ m.mockUnifiedViewMu.Lock()+ defer m.mockUnifiedViewMu.Unlock()+ reusable := m.mockUnifiedViewValid &&+ m.mockUnifiedViewStructure == structure &&+ sameManualLinks(m.mockUnifiedViewLinks, links)+ return m.mockUnifiedView, reusable+}++// currentStructureUnifiedStateView is currentUnifiedStateView for a caller that+// reads what the estate lists and how it connects: IDs, names, types, source+// mappings, parents, operator links and the metrics targets those yield. In+// mock mode that answers from the cached view for as long as the fixture+// structure and the link list are unchanged. The view is keyed on the data+// version otherwise, which advances on every metric tick, so a caller that+// needs only structure but asked for the data-version view built the whole+// estate once per tick: about half a second of CPU on an idle core, and on a+// starved process a build longer than the tick, so every call missed. A+// caller must not read what a tick moves (status, metric values, timestamps,+// sensors); the view it gets can be any number of ticks old. Outside mock mode+// this is currentUnifiedStateView.+func (m *Monitor) currentStructureUnifiedStateView() monitorUnifiedStateView {+ if m == nil {+ return monitorUnifiedStateView{}+ }+ if mock.IsMockEnabled() {+ if view, ok := m.cachedMockStructureView(); ok {+ return view+ }+ }+ return m.currentUnifiedStateView()+}++// GetUnifiedStructureReadState is GetUnifiedReadStateOrSnapshot for a caller+// that reads identity and topology only (see currentStructureUnifiedStateView):+// in mock mode it can be several metric ticks old.+func (m *Monitor) GetUnifiedStructureReadState() unifiedresources.ReadState {+ if m == nil {+ return nil+ }+ if !mock.IsMockEnabled() {+ return m.GetUnifiedReadStateOrSnapshot()+ }+ return m.currentStructureUnifiedStateView().readState+}++// currentModeStructureReadState is currentModeReadState for a caller that reads+// identity and topology only.+func (m *Monitor) currentModeStructureReadState() unifiedresources.ReadState {+ if m == nil {+ return nil+ }+ if mock.IsMockEnabled() {+ return m.currentStructureUnifiedStateView().readState+ }+ return m.currentModeReadState()+}++// structureNodes is NodesSnapshot for a caller that reads the nodes' identity+// and links only (ID, instance, linked agent).+func (m *Monitor) structureNodes() []models.Node {+ return nodesSnapshotFromReadState(m.GetUnifiedStructureReadState())+}