Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Stop mock target lookups rebuilding the estate view per tick
rcourtmanad122ed150a2
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 6a8d9af826..8456fef29e 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -9385,3 +9385,13 @@ for report-merge. No synthetic telemetry, identity key, enrollment or servicelink is added. `TestOperatorSplitOverridesProxmoxNodeAgentLink` checks thatthe monitor and its re-ingested API seed retain the fold across all splitshapes, then survive pin persistence and a repeated split.++### Mock-mode metrics-target lookups ride the fixture structure revision++`internal/monitoring/monitor.go` changed only so the mock-mode unified view+records the fixture structure revision it was built at and+`MetricsTargetForResource` resolves mock-mode targets from that view while the+revision and link list hold (monitoring contract, "Mock-mode metrics-target+lookups ride the fixture structure revision"). Agent registration, enrolment,+install, update, removal, report identity and continuity are unchanged, and+real-mode resolution still goes through the live registry.diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex 5bb69533b2..78ed8035a5 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -3530,6 +3530,73 @@ and `TestUnifiedResourceSnapshotWithLinksLeavesTheSharedSnapshotUnlinked` in`TestMockUnifiedStateViewUsesCanonicalMockFixtureGraph` in`internal/monitoring/canonical_guardrails_test.go` pins the call.+### Mock-mode metrics-target lookups ride the fixture structure revision++`Monitor.MetricsTargetForResource` is the lookup behind every windowed alert+metric: an alert pass resolves a target once per resource and metric+(`metricWindowPoints`). In mock mode it answers from the `metricsTargets` the+cached mock view captured when it was built (`mockMetricsTarget` in+`internal/monitoring/metrics_target_resolution.go`), and that view serves+lookups for as long as `mock.FixtureStructureRevision` and the operator link+list it was built with are unchanged. A metric tick advances+`mock.FixtureDataVersion` but moves neither the revision nor any input of a+target (`BuildMetricsTarget` reads a resource's type, its source mappings and+the technology, Docker container ID and disk identity of its facets, with the+resource's own `MetricsTarget` as fallback), so a warm cache no longer sends+the next lookup off to build an estate. Mock mode being toggled, the estate+being reconfigured or a link change rebuilds the view on the next lookup; as+before, two lookups that miss together can each build one, and a cold or+invalidated cache builds even for an ID the estate does not list. Once the+view is current an unknown ID resolves to nothing and falls to the raw+resource store. The view stays keyed on the data version for every other+consumer (broadcast, `/api/state`, `GetUnifiedReadStateOrSnapshot`), which+still rebuild it on a data-version miss.++The revision is read before the data version when a view is cached, and the+fixture advances its data version before its revision+(`advanceFixtureStructure` in `internal/mock/integration.go`). A reader that+sees a new revision therefore also sees the new data version and cannot cache+the previous estate's memoized snapshot under it. The interleaving cannot be+forced from outside, so `TestFixtureStructureRevisionOrderingContract` pins the+order of the writes and of the reads in source.++Keyed on the data version, a lookup after a tick built the view (about 0.5 s+of CPU for the default 1,776-resource estate on an idle core), and a build+slower than the 2-second tick published an already-stale view, so the next+lookup built again. Startup runs the first alert pass before the listener+opens, so on a starved host it never finished: on 2026-10-08 the startup+watchdog's dump on pulse-dev parked the main goroutine in+`evaluateMetricWindow` -> `MetricsTargetForResource` -> `currentUnifiedStateView`+and `/api/health` timed out. The same binary pinned to one core behind four+busy loops missed the view cache on 675 of 929 requests at a 16-second average+build and had not served by 606 s; with this change it served at 148 s (13 s+on the core alone). This was not a regression: builds from 1a094646d7 and+d2ac0c5112 stall identically under equal contention and both start in 14 to+16 seconds at normal load. The remaining time on a starved host is the other+consumers of the shared view, which rebuild it whenever they find it stale (on+the starved run, 42 view requests in a start of 194 s were all misses, from the+Docker-alert prune, the connection snapshots and the host-agent evaluation),+and the monitor's repeated full resource-store passes at startup. Loops that+still reach the data-version view once per item are unchanged: the mock storage+chart history (`resolveMockStorageTotal` per uncached pool), the demo patrol's+finding adds (`patrolResourceOperatorStateProvider`) and, with+`PULSE_MOCK_KEEP_REAL_POLLING`, the per-node polling lookups+(`linkedHostForNode`, `getHostAgentTemperatureForNode`).++`TestMockMetricsTargetLookupsDoNotRebuildTheViewAfterFixtureTicks` and+`TestMockMetricsTargetForUnknownIDDoesNotRebuildTheViewAfterFixtureTicks` fail on+the data-version key; `TestMockMetricsTargetFollowsAStructuralFixtureChange`,+`TestMockMetricsTargetFollowsAnOperatorLink`,+`TestMockMetricsTargetRebuildsWhenTheLinkListMoves` and+`TestMockMetricsTargetMatchesTheRegistryResolution` pin that the reused view is+replaced when the structure or the links change and answers as the registry+does. All six are in `internal/monitoring/monitor_host_agents_test.go`, with+their helpers in `internal/monitoring/metrics_target_resolution_test.go`.+`TestMetricTicksNeverChangeTheFixtureStructure` in+`internal/mock/platform_fixtures_test.go` pins the assumption itself: after each+tick of two cohort rotations the revision, the resource set and every+resolved target are unchanged.+### Saved quiet-hours policy before queue activation`Monitor.New` binds `alerts.Manager.QuietHoursNotificationPolicy` to thediff --git a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md b/docs/release-control/v6/internal/subsystems/performance-and-scalability.mdindex 351ec00504..229c6fdf6a 100644--- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md+++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md@@ -3724,6 +3724,35 @@ nonempty machine and agent IDs and canonical identity across fixture rebuilds.This changes fixture generation only, without relaxing production identityresolution or adding work to the recurring update path.+### Alert passes resolve metrics targets without materializing the estate++An alert pass makes one `Monitor.MetricsTargetForResource` call per resource+and windowed metric, so that call has to be a lookup, not a view build. In mock+mode it reads the targets the cached view captured and keeps reading them across+metric ticks (see "Mock-mode metrics-target lookups ride the fixture structure+revision" in the monitoring contract). On pulse-dev,+`BenchmarkMockMetricsTargetResolutionAfterTick` measured a pass of 4,731+lookups against the default estate, after a tick, at 1.8 ms and 151 KB. The+data-version key made the first lookup after every tick build the estate, and+made every lookup of a pass build it once a build outlasted the 2-second tick,+which stalled startup under CPU starvation. Real mode did not show this cost in+steady state: the store is itself the read state, and a one-off measurement+with a temporary counter (not asserted by a test) found a full+`updateResourceStore` pass over 1,694 resources (populate, metric syncs and+alert sync) taking 108 ms with 121 registry resolutions that fell back to the+full-mapping scan, because the registry's typed views are built before the+alert loop runs. `BenchmarkLiveRegistryMetricsTargetResolution` records only+the lookup cost of a registry whose views are unbuilt against one whose views+are built (136 ms against 5 ms per 5,079 lookups); it controls the warming+itself, so it cannot detect production failing to warm the views before the+alert loop, which is what the one-off measurement above showed. Run both+benchmarks with `-benchtime=3x`, since the registry case rebuilds its ingest+outside the timer. `TestMockMetricsTargetLookupsDoNotRebuildTheViewAfterFixtureTicks`+is the guard that fails on a rebuild per lookup. The remaining startup cost+under starvation is the monitor's repeated full resource-store passes (one per+supplemental provider registration) and the other consumers of the shared view,+which rebuild it whenever they find it stale; neither is changed here.+### Update evidence reuses the bounded node observationPackage evidence adds only one bounded status, one bounded reason, and thediff --git a/internal/mock/integration.go b/internal/mock/integration.goindex 219946857c..8ae4c87c58 100644--- a/internal/mock/integration.go+++ b/internal/mock/integration.go@@ -218,8 +218,7 @@ func enableMockMode(config MockConfig, fromInit bool) {dataMu.Lock()mockConfig = configmockGraph = buildFixtureGraph(config, now)- fixtureRevision.Add(1)- fixtureDataVersion.Add(1)+ advanceFixtureStructure()metricCohort.Store(0)enabled.Store(true)dataMu.Unlock()@@ -255,8 +254,7 @@ func disableMockMode() {dataMu.Lock()mockGraph = emptyFixtureGraph()- fixtureRevision.Add(1)- fixtureDataVersion.Add(1)+ advanceFixtureStructure()dataMu.Unlock()log.Info().Msg("mock mode disabled")@@ -334,6 +332,33 @@ func FixtureDataVersion() uint64 {return fixtureDataVersion.Load()}+// FixtureStructureRevision returns a token that advances only when the mock+// estate itself changes (mock mode enabled or disabled, or a configuration+// that rebuilds the graph), never on a metric tick. Which resources exist,+// their identities and their source mappings are fixed between two reads of+// the same revision, so a lookup that depends only on those, such as a+// resource's metrics target, may be served from a view built at any data+// version of that revision.+//+// A cache that tags a snapshot with the revision must read the revision+// BEFORE the data version it memoizes the snapshot under. The two counters+// advance under dataMu, data version first (advanceFixtureStructure), so a+// reader that sees a new revision is guaranteed the new data version after+// it and cannot be handed the previous estate's memoized snapshot under the+// new revision; the other interleaving only tags a newer snapshot with an+// older revision, which costs one rebuild.+func FixtureStructureRevision() uint64 {+ return fixtureRevision.Load()+}++// advanceFixtureStructure records a structural change of the fixture graph.+// Callers hold dataMu for writing. The data version moves before the+// structure revision; see FixtureStructureRevision for the ordering contract.+func advanceFixtureStructure() {+ fixtureDataVersion.Add(1)+ fixtureRevision.Add(1)+}+// GetConfig returns the current mock configuration.func GetConfig() MockConfig {dataMu.RLock()@@ -539,8 +564,7 @@ func SetMockConfig(cfg MockConfig) {setMockUpdateInterval(normalized.UpdateInterval)if configChanged && enabled.Load() {mockGraph = buildFixtureGraph(normalized, time.Now())- fixtureRevision.Add(1)- fixtureDataVersion.Add(1)+ advanceFixtureStructure()}dataMu.Unlock()diff --git a/internal/mock/platform_fixtures_test.go b/internal/mock/platform_fixtures_test.goindex 7f25908432..bf5db89b7f 100644--- a/internal/mock/platform_fixtures_test.go+++ b/internal/mock/platform_fixtures_test.go@@ -4,8 +4,15 @@ import ("bytes""encoding/json""fmt"+ "go/ast"+ "go/parser"+ "go/token""math"+ "os"+ "path/filepath"+ "reflect""slices"+ "strings""testing""time"@@ -1017,3 +1024,280 @@ func TestFixtureGraphMetricCohortProducesSparseUnifiedResourceChanges(t *testingt.Fatalf("expected sparse unified resource churn, changed=%d total=%d", changed, len(after))}}++func fixtureMetricsTargets(t *testing.T) map[string]unifiedresources.MetricsTarget {+ t.Helper()+ resources, _ := UnifiedResourceSnapshot()+ if len(resources) == 0 {+ t.Fatal("expected mock resources")+ }+ registry := unifiedresources.NewRegistry(nil)+ registry.IngestResources(resources)+ _, targets := registry.ListWithMetricsTargets()+ if len(targets) == 0 {+ t.Fatal("expected the mock estate to resolve metrics targets")+ }+ return targets+}++func fixtureResourceIDs(t *testing.T) []string {+ t.Helper()+ resources, _ := UnifiedResourceSnapshot()+ ids := make([]string, 0, len(resources))+ for _, resource := range resources {+ ids = append(ids, resource.ID)+ }+ slices.Sort(ids)+ return ids+}++// smallMockEstateForTest replaces the ambient estate with a fixed small one,+// so a test neither depends on the process's PULSE_MOCK_* settings (a node+// count at the supported maximum clamps back when raised) nor pays for a+// 929-guest snapshot per assertion. The ambient configuration returns when+// the test ends.+func smallMockEstateForTest(t *testing.T, nodes int) MockConfig {+ t.Helper()+ previous := GetConfig()+ t.Cleanup(func() { SetMockConfig(previous) })+ small := previous+ small.NodeCount = nodes+ SetMockConfig(small)+ return small+}++// Callers cache a resource's metrics target for as long as the structure+// revision holds (see Monitor.MetricsTargetForResource), so a metric tick must+// leave the revision, the resource set and every target exactly as they were.+// Checked after every tick of two full cohort rotations, not only at the end,+// so a change that a later tick reverts is still caught.+func TestMetricTicksNeverChangeTheFixtureStructure(t *testing.T) {+ withMockEnabledForTest(t)+ stopUpdateLoop()+ t.Cleanup(func() {+ if IsMockEnabled() {+ startUpdateLoop()+ }+ })+ smallMockEstateForTest(t, 6)+ stopUpdateLoop()++ revision := FixtureStructureRevision()+ ids := fixtureResourceIDs(t)+ targets := fixtureMetricsTargets(t)+ version := FixtureDataVersion()++ for tick := 1; tick <= mockMetricCohortCount*2+1; tick++ {+ updateMetrics(GetConfig())++ if got := FixtureDataVersion(); got <= version {+ t.Fatalf("tick %d did not advance the data version (%d)", tick, got)+ } else {+ version = got+ }+ if got := FixtureStructureRevision(); got != revision {+ t.Fatalf("tick %d moved the fixture structure revision from %d to %d", tick, revision, got)+ }+ if got := fixtureResourceIDs(t); !slices.Equal(got, ids) {+ t.Fatalf("tick %d changed the resource set: %d resources before, %d after", tick, len(ids), len(got))+ }+ if got := fixtureMetricsTargets(t); !reflect.DeepEqual(got, targets) {+ t.Fatalf("tick %d changed the resolved metrics targets", tick)+ }+ }+}++func TestFixtureStructureRevisionAdvancesOnStructuralChange(t *testing.T) {+ withMockEnabledForTest(t)+ stopUpdateLoop()+ t.Cleanup(func() {+ if IsMockEnabled() {+ startUpdateLoop()+ }+ })+ small := smallMockEstateForTest(t, 3)+ stopUpdateLoop()++ revision := FixtureStructureRevision()+ grown := small+ grown.NodeCount = 4+ SetMockConfig(grown)+ if got := FixtureStructureRevision(); got <= revision {+ t.Fatalf("reconfiguring the estate left the structure revision at %d (was %d)", got, revision)+ }++ revision = FixtureStructureRevision()+ if err := SetEnabled(false); err != nil {+ t.Fatalf("disable mock mode: %v", err)+ }+ if got := FixtureStructureRevision(); got <= revision {+ t.Fatalf("disabling mock mode left the structure revision at %d (was %d)", got, revision)+ }+ revision = FixtureStructureRevision()+ if err := SetEnabled(true); err != nil {+ t.Fatalf("enable mock mode: %v", err)+ }+ if got := FixtureStructureRevision(); got <= revision {+ t.Fatalf("enabling mock mode left the structure revision at %d (was %d)", got, revision)+ }+}++// A cache fill reads the structure revision before the data version, and the+// fixture advances its data version before its revision, so a reader that+// sees a new revision is guaranteed the new data version after it and is+// never handed the previous estate's memoized snapshot under the new+// revision. The interleaving cannot be forced from outside, so the order of+// the two writes and of the two reads is pinned on the syntax tree: inside+// the named functions only, so comments, local names and the position of the+// functions in their package do not matter.+func TestFixtureStructureRevisionOrderingContract(t *testing.T) {+ mockDecls := parsePackageFuncs(t, ".")+ advance := packageFunc(mockDecls["advanceFixtureStructure"])+ if advance == nil {+ t.Fatal("the mock package lost advanceFixtureStructure")+ }+ version := synchronousCallOffsets(advance, "fixtureDataVersion", "Add")+ revision := synchronousCallOffsets(advance, "fixtureRevision", "Add")+ if len(version) != 1 || len(revision) != 1 || version[0] > revision[0] {+ t.Fatal("advanceFixtureStructure must advance the data version once, before the structure revision once, as plain statements")+ }+ if got := countCalls(advance, "fixtureRevision", "Add") + countCalls(advance, "fixtureDataVersion", "Add"); got != 2 {+ t.Fatalf("advanceFixtureStructure must not defer or spawn its counter writes; found %d writes in all", got)+ }++ // Every declaration counts for the bypass scan, including methods and the+ // build-tagged duplicates of a function.+ for name, decls := range mockDecls {+ if name == "advanceFixtureStructure" {+ continue+ }+ for _, decl := range decls {+ if got := countCalls(decl, "fixtureRevision", "Add"); got != 0 {+ t.Fatalf("%s advances the structure revision directly; it must go through advanceFixtureStructure", name)+ }+ }+ }+ for _, name := range []string{"enableMockMode", "disableMockMode", "SetMockConfig"} {+ decl := packageFunc(mockDecls[name])+ if decl == nil {+ t.Fatalf("the mock package lost %s", name)+ }+ if got := len(synchronousIdentCallOffsets(decl, "advanceFixtureStructure")); got != 1 {+ t.Fatalf("%s must advance the fixture structure through advanceFixtureStructure exactly once, as a plain statement; found %d", name, got)+ }+ }++ view := packageFunc(parsePackageFuncs(t, "../monitoring")["currentUnifiedStateView"])+ if view == nil {+ t.Fatal("the monitoring package lost currentUnifiedStateView")+ }+ structureRead := synchronousCallOffsets(view, "mock", "FixtureStructureRevision")+ versionRead := synchronousCallOffsets(view, "mock", "FixtureDataVersion")+ if len(structureRead) != 1 || len(versionRead) != 1 || structureRead[0] > versionRead[0] {+ t.Fatal("currentUnifiedStateView must read the structure revision once, before it reads the data version")+ }+}++// parsePackageFuncs returns every non-test function and method declaration of+// the package in dir, keyed by name. Names repeat across build-tagged files+// and between functions and methods, so each key holds all of them.+func parsePackageFuncs(t *testing.T, dir string) map[string][]*ast.FuncDecl {+ t.Helper()+ entries, err := os.ReadDir(dir)+ if err != nil {+ t.Fatalf("read %s: %v", dir, err)+ }+ funcs := make(map[string][]*ast.FuncDecl)+ fset := token.NewFileSet()+ for _, entry := range entries {+ name := entry.Name()+ if entry.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") {+ continue+ }+ file, err := parser.ParseFile(fset, filepath.Join(dir, name), nil, 0)+ if err != nil {+ t.Fatalf("parse %s: %v", name, err)+ }+ for _, decl := range file.Decls {+ if fn, ok := decl.(*ast.FuncDecl); ok {+ funcs[fn.Name.Name] = append(funcs[fn.Name.Name], fn)+ }+ }+ }+ return funcs+}++// packageFunc returns the declaration when it is the only one of its name in+// the package (function or method), and nil otherwise.+func packageFunc(decls []*ast.FuncDecl) *ast.FuncDecl {+ if len(decls) != 1 {+ return nil+ }+ return decls[0]+}++// walkSynchronous visits the nodes of fn that run in the function's own+// control flow: it does not descend into function literals, deferred calls or+// goroutine starts, whose source position says nothing about when they run.+func walkSynchronous(fn *ast.FuncDecl, visit func(ast.Node)) {+ ast.Inspect(fn, func(node ast.Node) bool {+ switch node.(type) {+ case *ast.FuncLit, *ast.DeferStmt, *ast.GoStmt:+ return false+ }+ visit(node)+ return true+ })+}++func isSelectorCall(node ast.Node, receiver, method string) bool {+ call, ok := node.(*ast.CallExpr)+ if !ok {+ return false+ }+ selector, ok := call.Fun.(*ast.SelectorExpr)+ if !ok || selector.Sel.Name != method {+ return false+ }+ ident, ok := selector.X.(*ast.Ident)+ return ok && ident.Name == receiver+}++// synchronousCallOffsets lists the positions of receiver.method calls that run+// in fn's own control flow, in source order.+func synchronousCallOffsets(fn *ast.FuncDecl, receiver, method string) []token.Pos {+ var offsets []token.Pos+ walkSynchronous(fn, func(node ast.Node) {+ if isSelectorCall(node, receiver, method) {+ offsets = append(offsets, node.Pos())+ }+ })+ return offsets+}++// synchronousIdentCallOffsets lists the positions of calls to the package-level+// function name that run in fn's own control flow.+func synchronousIdentCallOffsets(fn *ast.FuncDecl, name string) []token.Pos {+ var offsets []token.Pos+ walkSynchronous(fn, func(node ast.Node) {+ if call, ok := node.(*ast.CallExpr); ok {+ if ident, ok := call.Fun.(*ast.Ident); ok && ident.Name == name {+ offsets = append(offsets, call.Pos())+ }+ }+ })+ return offsets+}++// countCalls counts every receiver.method call anywhere in fn, including+// deferred calls, goroutines and function literals.+func countCalls(fn *ast.FuncDecl, receiver, method string) int {+ count := 0+ ast.Inspect(fn, func(node ast.Node) bool {+ if isSelectorCall(node, receiver, method) {+ count+++ }+ return true+ })+ return count+}diff --git a/internal/monitoring/metrics_target_resolution.go b/internal/monitoring/metrics_target_resolution.gonew file mode 100644index 0000000000..6d81f731a3--- /dev/null+++ b/internal/monitoring/metrics_target_resolution.go@@ -0,0 +1,63 @@+package monitoring++import (+ "github.com/rcourtman/pulse-go-rewrite/internal/mock"+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"+)++// mockMetricsTarget resolves a canonical resource ID to its metrics-store+// target in mock mode. The second result is false when mock mode is off, so+// the caller takes its ordinary path; when it is true the answer is final for+// this structure, a nil target meaning the view captured no metrics target for+// the ID (the estate does not list it, or lists a resource that has none, such+// as a VMware network); the caller still tries the raw resource store.+//+// Alert evaluation resolves a target for every resource and every windowed+// metric (CPU by default) on every pass, through MetricsTargetForResource.+// The mock view that answers it is cached per fixture data version, and that+// version advances on every metric tick (2 s by default). Rebuilding the view+// ingests the whole estate twice (the memoized fixture snapshot, then the+// view's own registry), about half a second of CPU for the default estate+// (1,776 resources) on an idle core. A build that takes longer than one tick publishes+// its view under an already-stale version, so the next lookup rebuilds again:+// on a host with CPU to spare that costs one build per tick, but when the+// process is starved (a loaded dev VM, a small demo host) the cache never+// hits, every lookup in the pass builds an estate, and the monitor never+// finishes its first evaluation. Startup stalled in exactly that state,+// before the listener opened, with the watchdog dump parked in the view build+// beneath evaluateMetricWindow.+//+// A metrics target depends only on identity inputs: a resource's type, its+// source mappings, and the technology, Docker container ID and disk identity+// of its facets (unifiedresources.BuildMetricsTarget), falling back to the+// resource's own MetricsTarget. A metric tick moves values and timestamps,+// never those inputs (TestMetricTicksNeverChangeTheFixtureStructure), so the+// captured targets of a view built at any data version of the current+// structure answer the lookup. A data-version change alone therefore never+// triggers a build here: only a structural change (mock mode toggled, the+// fixture reconfigured) or a link change makes the next lookup rebuild the+// view. Two lookups that miss together can each still build one, and a cold+// cache builds even for an ID the estate does not list.+func (m *Monitor) mockMetricsTarget(resourceID string) (*unifiedresources.MetricsTarget, bool) {+ if m == nil || !mock.IsMockEnabled() {+ return nil, false+ }++ structure := mock.FixtureStructureRevision()+ links := m.resourceStoreManualLinks()++ m.mockUnifiedViewMu.Lock()+ view := m.mockUnifiedView+ reusable := m.mockUnifiedViewValid &&+ m.mockUnifiedViewStructure == structure &&+ sameManualLinks(m.mockUnifiedViewLinks, links)+ m.mockUnifiedViewMu.Unlock()++ if !reusable {+ view = m.currentUnifiedStateView()+ }+ if view.metricsTargets == nil {+ return nil, true+ }+ return view.metricsTargets.MetricsTargetForResource(resourceID), true+}diff --git a/internal/monitoring/metrics_target_resolution_test.go b/internal/monitoring/metrics_target_resolution_test.gonew file mode 100644index 0000000000..79122e93ee--- /dev/null+++ b/internal/monitoring/metrics_target_resolution_test.go@@ -0,0 +1,158 @@+package monitoring++import (+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/mock"+ "github.com/rcourtman/pulse-go-rewrite/internal/models"+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"+)++// useMockEstate enables the default mock estate with the given node count,+// whose fixture ticks every interval, and restores the previous mock mode and+// configuration when the test ends.+func useMockEstate(tb testing.TB, nodes int, interval time.Duration) {+ tb.Helper()+ previous := mock.IsMockEnabled()+ previousConfig := mock.GetConfig()+ if previous {+ mustSetMockEnabled(tb, false)+ }+ // Start from the default estate, not the ambient one: PULSE_MOCK_* settings+ // can remove the standalone agents and clusters these tests look for.+ config := mock.DefaultConfig+ config.NodeCount = nodes+ config.UpdateInterval = interval+ mock.SetMockConfig(config)+ mustSetMockEnabled(tb, true)+ tb.Cleanup(func() {+ mustSetMockEnabled(tb, false)+ mock.SetMockConfig(previousConfig)+ mustSetMockEnabled(tb, previous)+ })+}++func newMockEstateMonitor() *Monitor {+ return &Monitor{+ state: models.NewState(),+ resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(unifiedresources.NewMemoryStore())),+ }+}++// publishedMockView returns the view the monitor currently caches for mock+// mode without building one.+func publishedMockView(m *Monitor) (monitorUnifiedStateView, bool) {+ m.mockUnifiedViewMu.Lock()+ defer m.mockUnifiedViewMu.Unlock()+ return m.mockUnifiedView, m.mockUnifiedViewValid+}++// sameMetricsTarget reports whether two lookups resolved the same target (or+// both resolved none).+func sameMetricsTarget(a, b *unifiedresources.MetricsTarget) bool {+ if a == nil || b == nil {+ return a == b+ }+ return *a == *b+}++func waitForFixtureTicks(tb testing.TB, ticks uint64) {+ tb.Helper()+ target := mock.FixtureDataVersion() + ticks+ // A starved host ticks slowly; the deadline only catches a ticker that+ // is not running at all.+ deadline := time.Now().Add(5 * time.Minute)+ for mock.FixtureDataVersion() < target {+ if time.Now().After(deadline) {+ tb.Fatalf("fixture data version stayed below %d (at %d) for five minutes; the mock ticker is not running", target, mock.FixtureDataVersion())+ }+ time.Sleep(20 * time.Millisecond)+ }+}++func resourceIDsWithTargets(tb testing.TB, view monitorUnifiedStateView) []string {+ tb.Helper()+ ids := make([]string, 0, len(view.resources))+ for _, resource := range view.resources {+ if view.metricsTargets.MetricsTargetForResource(resource.ID) != nil {+ ids = append(ids, resource.ID)+ }+ }+ if len(ids) == 0 {+ tb.Fatal("expected the mock estate to list resources with metrics targets")+ }+ return ids+}++// BenchmarkMockMetricsTargetResolutionAfterTick measures one alert pass's+// lookups (every resource, three windowed metrics) against a view that was+// built before the fixture last ticked. It must stay a map read per lookup, so+// a pass costs microseconds however slowly the process runs, not an estate+// build per lookup.+func BenchmarkMockMetricsTargetResolutionAfterTick(b *testing.B) {+ useMockEstate(b, mock.DefaultConfig.NodeCount, time.Second)+ m := newMockEstateMonitor()+ ids := resourceIDsWithTargets(b, m.currentUnifiedStateView())+ waitForFixtureTicks(b, 1)++ b.ReportAllocs()+ b.ResetTimer()+ for i := 0; i < b.N; i++ {+ for _, id := range ids {+ for metric := 0; metric < 3; metric++ {+ if m.MetricsTargetForResource(id) == nil {+ b.Fatalf("no target for %q", id)+ }+ }+ }+ }+ b.ReportMetric(float64(len(ids)*3), "lookups/op")+}++// BenchmarkLiveRegistryMetricsTargetResolution measures the same pass against a+// store-backed registry, as real mode resolves it, right after an ingest (the+// registry's typed views not yet built) and once they are.+func BenchmarkLiveRegistryMetricsTargetResolution(b *testing.B) {+ useMockEstate(b, mock.DefaultConfig.NodeCount, 5*time.Minute)+ graph := mock.CurrentFixtureGraph()+ snapshot := unifiedresources.SnapshotWithoutSources(graph.State, mock.SupplementalOwnedSources())+ mustSetMockEnabled(b, false)++ for _, views := range []string{"after-ingest", "views-built"} {+ b.Run(views, func(b *testing.B) {+ registry := unifiedresources.NewRegistry(nil)+ registry.IngestSnapshot(snapshot)+ adapter := unifiedresources.NewMonitorAdapter(registry)+ m := &Monitor{state: models.NewState(), resourceStore: adapter}++ listed := registry.List()+ ids := make([]string, 0, len(listed))+ for _, resource := range listed {+ ids = append(ids, resource.ID)+ }+ if views == "views-built" {+ adapter.VMs()+ }++ b.ReportAllocs()+ b.ResetTimer()+ for i := 0; i < b.N; i++ {+ if views == "after-ingest" {+ b.StopTimer()+ registry = unifiedresources.NewRegistry(nil)+ registry.IngestSnapshot(snapshot)+ adapter = unifiedresources.NewMonitorAdapter(registry)+ m.resourceStore = adapter+ b.StartTimer()+ }+ for _, id := range ids {+ for metric := 0; metric < 3; metric++ {+ m.MetricsTargetForResource(id)+ }+ }+ }+ b.ReportMetric(float64(len(ids)*3), "lookups/op")+ })+ }+}diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.goindex 317d4130dc..5fe35958a8 100644--- a/internal/monitoring/monitor.go+++ b/internal/monitoring/monitor.go@@ -1216,6 +1216,7 @@ type Monitor struct {mockUnifiedViewMu sync.MutexmockUnifiedView monitorUnifiedStateViewmockUnifiedViewVersion uint64+ mockUnifiedViewStructure uint64 // fixture structure revision the published view was built atmockUnifiedViewLinks []unifiedresources.ResourceLinkmockUnifiedViewFixtureAt time.Time // fixture freshness the published view was built frommockUnifiedViewValid bool@@ -5049,10 +5050,20 @@ func (m *Monitor) MetricsTargetForResource(resourceID string) *unifiedresources.return nil}- if view := m.GetUnifiedReadStateOrSnapshot(); view != nil {- if resolver, ok := view.(MetricsTargetResourceStore); ok {- if target := resolver.MetricsTargetForResource(resourceID); target != nil {- return target+ // Mock mode resolves from the cached view's captured targets and never+ // asks the view again: this runs once per resource and windowed metric in+ // every alert pass, and the fixture's data version moves on every tick+ // (see mockMetricsTarget).+ target, fromMockView := m.mockMetricsTarget(resourceID)+ if target != nil {+ return target+ }+ if !fromMockView {+ if view := m.GetUnifiedReadStateOrSnapshot(); view != nil {+ if resolver, ok := view.(MetricsTargetResourceStore); ok {+ if target := resolver.MetricsTargetForResource(resourceID); target != nil {+ return target+ }}}}@@ -5210,7 +5221,12 @@ func (m *Monitor) currentUnifiedStateView() monitorUnifiedStateView {// Read the version and links before the snapshot so a tick or// rebuild landing in between caches newer data under an older// token (harmless rebuild next call) rather than ever serving stale- // data under a newer one.+ // data under a newer one. The structure revision goes first: the+ // fixture advances its data version before its revision, so a+ // reader that sees a new revision also reads the new version and+ // never memoizes the previous estate under it+ // (mock.FixtureStructureRevision).+ structure := mock.FixtureStructureRevision()version := mock.FixtureDataVersion()links := m.resourceStoreManualLinks()m.mockUnifiedViewMu.Lock()@@ -5255,6 +5271,7 @@ func (m *Monitor) currentUnifiedStateView() monitorUnifiedStateView {m.mockUnifiedView = viewm.mockUnifiedViewFixtureAt = fixtureFreshnessm.mockUnifiedViewVersion = version+ m.mockUnifiedViewStructure = structurem.mockUnifiedViewLinks = linksm.mockUnifiedViewValid = truem.mockUnifiedViewMu.Unlock()diff --git a/internal/monitoring/monitor_host_agents_test.go b/internal/monitoring/monitor_host_agents_test.goindex b4127d27bf..46cf04cea2 100644--- a/internal/monitoring/monitor_host_agents_test.go+++ b/internal/monitoring/monitor_host_agents_test.go@@ -5992,6 +5992,187 @@ func TestMockUnifiedViewAppliesOperatorManualLinks(t *testing.T) {}}+// Alert evaluation resolves a target per resource and windowed metric on every+// pass. A metric tick moves the fixture's data version, not what resources+// exist, so a lookup after a tick must be served from the cached view instead+// of building an estate. Before this was pinned, every lookup after a tick+// rebuilt the view, and a process too starved to finish one build within a tick+// rebuilt it for every lookup of the pass: startup never reached its listener.+func TestMockMetricsTargetLookupsDoNotRebuildTheViewAfterFixtureTicks(t *testing.T) {+ useMockEstate(t, 4, time.Second)+ m := newMockEstateMonitor()++ warm := m.currentUnifiedStateView()+ ids := resourceIDsWithTargets(t, warm)++ waitForFixtureTicks(t, 2)++ for _, id := range ids {+ want := warm.metricsTargets.MetricsTargetForResource(id)+ got := m.MetricsTargetForResource(id)+ if !sameMetricsTarget(got, want) {+ t.Fatalf("MetricsTargetForResource(%q) = %+v after fixture ticks, want %+v", id, got, want)+ }+ }++ after, ok := publishedMockView(m)+ if !ok || after.readState != warm.readState {+ t.Fatal("target lookups rebuilt the unified view after a metric tick; they must reuse the view built for the current fixture structure")+ }+}++// The reused view answers for the estate it was built from. A structural+// change (here a larger estate) must reach the lookup, not keep serving the+// old resource set.+func TestMockMetricsTargetFollowsAStructuralFixtureChange(t *testing.T) {+ useMockEstate(t, 3, 5*time.Minute)+ m := newMockEstateMonitor()++ before := m.currentUnifiedStateView()+ known := make(map[string]struct{}, len(before.resources))+ for _, resource := range before.resources {+ known[resource.ID] = struct{}{}+ }++ grown := mock.GetConfig()+ grown.NodeCount+++ mock.SetMockConfig(grown)++ // Another monitor's view of the grown estate names the resources the+ // first one lacks. The first monitor has not been asked for a view since+ // the change, so the lookup itself has to notice the structure moved.+ var added string+ for _, resource := range newMockEstateMonitor().currentUnifiedStateView().resources {+ if _, ok := known[resource.ID]; !ok && resource.Type == unifiedresources.ResourceTypeAgent {+ added = resource.ID+ break+ }+ }+ if added == "" {+ t.Fatal("the larger fixture estate lists no agent the first one did not")+ }+ if target := m.MetricsTargetForResource(added); target == nil {+ t.Fatalf("MetricsTargetForResource(%q) = nil after the fixture grew; the cached view outlived its structure", added)+ }+}++// Reusing a view across ticks must not change an answer: every resource+// resolves to what the view's own registry resolves, and an ID the estate does+// not list resolves to nothing without building a view for it.+func TestMockMetricsTargetMatchesTheRegistryResolution(t *testing.T) {+ useMockEstate(t, 4, 5*time.Minute)+ m := newMockEstateMonitor()++ view := m.currentUnifiedStateView()+ registryResolver, ok := view.readState.(MetricsTargetResourceStore)+ if !ok {+ t.Fatalf("mock view read state %T does not resolve metrics targets", view.readState)+ }+ for _, resource := range view.resources {+ want := registryResolver.MetricsTargetForResource(resource.ID)+ got := m.MetricsTargetForResource(resource.ID)+ if !sameMetricsTarget(got, want) {+ t.Fatalf("MetricsTargetForResource(%q) = %+v, want the registry's %+v", resource.ID, got, want)+ }+ if padded := m.MetricsTargetForResource(" " + resource.ID + " "); !sameMetricsTarget(padded, want) {+ t.Fatalf("MetricsTargetForResource(%q padded) = %+v, want the canonical ID's %+v", resource.ID, padded, want)+ }+ }+ if got := m.MetricsTargetForResource("not-a-mock-resource"); got != nil {+ t.Fatalf("MetricsTargetForResource(unknown) = %+v, want nil", got)+ }+ if after, _ := publishedMockView(m); after.readState != view.readState {+ t.Fatal("resolving an unknown ID rebuilt the cached view")+ }+}++// A lookup for an ID the estate does not list must stay cheap after a tick+// too: the view has no more to say about it than the one it replaces.+func TestMockMetricsTargetForUnknownIDDoesNotRebuildTheViewAfterFixtureTicks(t *testing.T) {+ useMockEstate(t, 3, time.Second)+ m := newMockEstateMonitor()++ warm := m.currentUnifiedStateView()+ waitForFixtureTicks(t, 2)+ for i := 0; i < 50; i++ {+ if got := m.MetricsTargetForResource(fmt.Sprintf("unknown-%d", i)); got != nil {+ t.Fatalf("MetricsTargetForResource(unknown) = %+v, want nil", got)+ }+ }+ if after, _ := publishedMockView(m); after.readState != warm.readState {+ t.Fatal("unknown-ID lookups rebuilt the unified view after a metric tick")+ }+}++// An operator link folds one resource into another, which changes what the+// folded ID resolves to. Whatever rebuilds the view first, a lookup after the+// resource store applied the link must not resolve the folded resource.+func TestMockMetricsTargetFollowsAnOperatorLink(t *testing.T) {+ useMockEstate(t, 4, 5*time.Minute)+ store := unifiedresources.NewMemoryStore()+ m := &Monitor{+ state: models.NewState(),+ resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(store)),+ alertManager: alerts.NewManager(),+ }+ t.Cleanup(m.alertManager.Stop)++ var vmID, agentID string+ for _, resource := range m.currentUnifiedStateView().resources {+ switch {+ case vmID == "" && resource.Type == unifiedresources.ResourceTypeVM && resource.Agent == nil:+ vmID = resource.ID+ case agentID == "" && resource.Type == unifiedresources.ResourceTypeAgent && resource.Agent != nil &&+ len(resource.Sources) == 1 && resource.Sources[0] == unifiedresources.SourceAgent:+ agentID = resource.ID+ }+ }+ if vmID == "" || agentID == "" {+ t.Fatalf("fixture graph needs an agentless VM and a standalone agent, got vm=%q agent=%q", vmID, agentID)+ }+ if m.MetricsTargetForResource(agentID) == nil || m.MetricsTargetForResource(vmID) == nil {+ t.Fatal("expected the unlinked VM and agent to resolve metrics targets")+ }++ if err := store.AddLink(unifiedresources.ResourceLink{ResourceA: vmID, ResourceB: agentID, PrimaryID: vmID}); err != nil {+ t.Fatalf("AddLink: %v", err)+ }+ // Rebuild the resource store so it loads the link, without asking for a+ // mock view: the lookup itself has to see the link list change.+ m.updateResourceStore(models.StateSnapshot{}, m.mockModeFence.begin())++ if got := m.MetricsTargetForResource(agentID); got != nil {+ t.Fatalf("MetricsTargetForResource(folded agent) = %+v after the operator link, want nothing", got)+ }+ if m.MetricsTargetForResource(vmID) == nil {+ t.Fatal("the linked VM lost its metrics target")+ }+}++// The reused view was built under one link list; once the resource store+// applies a different one, a lookup must rebuild rather than answer from it.+// A rebuild by any other consumer would hide a missing check, so the list the+// view was built under is changed directly.+func TestMockMetricsTargetRebuildsWhenTheLinkListMoves(t *testing.T) {+ useMockEstate(t, 3, 5*time.Minute)+ m := newMockEstateMonitor()++ warm := m.currentUnifiedStateView()+ id := resourceIDsWithTargets(t, warm)[0]++ m.mockUnifiedViewMu.Lock()+ m.mockUnifiedViewLinks = []unifiedresources.ResourceLink{{ResourceA: "stale-a", ResourceB: "stale-b", PrimaryID: "stale-a"}}+ m.mockUnifiedViewMu.Unlock()++ if m.MetricsTargetForResource(id) == nil {+ t.Fatalf("MetricsTargetForResource(%q) = nil", id)+ }+ after, _ := publishedMockView(m)+ if after.readState == warm.readState {+ t.Fatal("a lookup answered from a view built under a different operator link list")+ }+}+// TestMockModeDiscardsRealHostReports pins the push-side of the mock clean// room. Mock mode already suspends pull-based PVE/PBS/PMG collection, but a// real agent keeps POSTing regardless; ingesting those reports lands a real