Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Stop mock target lookups rebuilding the estate view per tick

rcourtmanad122ed150a2
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 6a8d9af826..8456fef29e 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -9385,3 +9385,13 @@ for report-merge. No synthetic telemetry, identity key, enrollment or service
link is added. `TestOperatorSplitOverridesProxmoxNodeAgentLink` checks that
the monitor and its re-ingested API seed retain the fold across all split
shapes, then survive pin persistence and a repeated split.
+
+### Mock-mode metrics-target lookups ride the fixture structure revision
+
+`internal/monitoring/monitor.go` changed only so the mock-mode unified view
+records the fixture structure revision it was built at and
+`MetricsTargetForResource` resolves mock-mode targets from that view while the
+revision and link list hold (monitoring contract, "Mock-mode metrics-target
+lookups ride the fixture structure revision"). Agent registration, enrolment,
+install, update, removal, report identity and continuity are unchanged, and
+real-mode resolution still goes through the live registry.
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 5bb69533b2..78ed8035a5 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -3530,6 +3530,73 @@ and `TestUnifiedResourceSnapshotWithLinksLeavesTheSharedSnapshotUnlinked` in
`TestMockUnifiedStateViewUsesCanonicalMockFixtureGraph` in
`internal/monitoring/canonical_guardrails_test.go` pins the call.
+### Mock-mode metrics-target lookups ride the fixture structure revision
+
+`Monitor.MetricsTargetForResource` is the lookup behind every windowed alert
+metric: an alert pass resolves a target once per resource and metric
+(`metricWindowPoints`). In mock mode it answers from the `metricsTargets` the
+cached mock view captured when it was built (`mockMetricsTarget` in
+`internal/monitoring/metrics_target_resolution.go`), and that view serves
+lookups for as long as `mock.FixtureStructureRevision` and the operator link
+list it was built with are unchanged. A metric tick advances
+`mock.FixtureDataVersion` but moves neither the revision nor any input of a
+target (`BuildMetricsTarget` reads a resource's type, its source mappings and
+the technology, Docker container ID and disk identity of its facets, with the
+resource's own `MetricsTarget` as fallback), so a warm cache no longer sends
+the next lookup off to build an estate. Mock mode being toggled, the estate
+being reconfigured or a link change rebuilds the view on the next lookup; as
+before, two lookups that miss together can each build one, and a cold or
+invalidated cache builds even for an ID the estate does not list. Once the
+view is current an unknown ID resolves to nothing and falls to the raw
+resource store. The view stays keyed on the data version for every other
+consumer (broadcast, `/api/state`, `GetUnifiedReadStateOrSnapshot`), which
+still rebuild it on a data-version miss.
+
+The revision is read before the data version when a view is cached, and the
+fixture advances its data version before its revision
+(`advanceFixtureStructure` in `internal/mock/integration.go`). A reader that
+sees a new revision therefore also sees the new data version and cannot cache
+the previous estate's memoized snapshot under it. The interleaving cannot be
+forced from outside, so `TestFixtureStructureRevisionOrderingContract` pins the
+order of the writes and of the reads in source.
+
+Keyed on the data version, a lookup after a tick built the view (about 0.5 s
+of CPU for the default 1,776-resource estate on an idle core), and a build
+slower than the 2-second tick published an already-stale view, so the next
+lookup built again. Startup runs the first alert pass before the listener
+opens, so on a starved host it never finished: on 2026-10-08 the startup
+watchdog's dump on pulse-dev parked the main goroutine in
+`evaluateMetricWindow` -> `MetricsTargetForResource` -> `currentUnifiedStateView`
+and `/api/health` timed out. The same binary pinned to one core behind four
+busy loops missed the view cache on 675 of 929 requests at a 16-second average
+build and had not served by 606 s; with this change it served at 148 s (13 s
+on the core alone). This was not a regression: builds from 1a094646d7 and
+d2ac0c5112 stall identically under equal contention and both start in 14 to
+16 seconds at normal load. The remaining time on a starved host is the other
+consumers of the shared view, which rebuild it whenever they find it stale (on
+the starved run, 42 view requests in a start of 194 s were all misses, from the
+Docker-alert prune, the connection snapshots and the host-agent evaluation),
+and the monitor's repeated full resource-store passes at startup. Loops that
+still reach the data-version view once per item are unchanged: the mock storage
+chart history (`resolveMockStorageTotal` per uncached pool), the demo patrol's
+finding adds (`patrolResourceOperatorStateProvider`) and, with
+`PULSE_MOCK_KEEP_REAL_POLLING`, the per-node polling lookups
+(`linkedHostForNode`, `getHostAgentTemperatureForNode`).
+
+`TestMockMetricsTargetLookupsDoNotRebuildTheViewAfterFixtureTicks` and
+`TestMockMetricsTargetForUnknownIDDoesNotRebuildTheViewAfterFixtureTicks` fail on
+the data-version key; `TestMockMetricsTargetFollowsAStructuralFixtureChange`,
+`TestMockMetricsTargetFollowsAnOperatorLink`,
+`TestMockMetricsTargetRebuildsWhenTheLinkListMoves` and
+`TestMockMetricsTargetMatchesTheRegistryResolution` pin that the reused view is
+replaced when the structure or the links change and answers as the registry
+does. All six are in `internal/monitoring/monitor_host_agents_test.go`, with
+their helpers in `internal/monitoring/metrics_target_resolution_test.go`.
+`TestMetricTicksNeverChangeTheFixtureStructure` in
+`internal/mock/platform_fixtures_test.go` pins the assumption itself: after each
+tick of two cohort rotations the revision, the resource set and every
+resolved target are unchanged.
+
### Saved quiet-hours policy before queue activation
`Monitor.New` binds `alerts.Manager.QuietHoursNotificationPolicy` to the
diff --git a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md
index 351ec00504..229c6fdf6a 100644
--- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md
+++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md
@@ -3724,6 +3724,35 @@ nonempty machine and agent IDs and canonical identity across fixture rebuilds.
This changes fixture generation only, without relaxing production identity
resolution or adding work to the recurring update path.
+### Alert passes resolve metrics targets without materializing the estate
+
+An alert pass makes one `Monitor.MetricsTargetForResource` call per resource
+and windowed metric, so that call has to be a lookup, not a view build. In mock
+mode it reads the targets the cached view captured and keeps reading them across
+metric ticks (see "Mock-mode metrics-target lookups ride the fixture structure
+revision" in the monitoring contract). On pulse-dev,
+`BenchmarkMockMetricsTargetResolutionAfterTick` measured a pass of 4,731
+lookups against the default estate, after a tick, at 1.8 ms and 151 KB. The
+data-version key made the first lookup after every tick build the estate, and
+made every lookup of a pass build it once a build outlasted the 2-second tick,
+which stalled startup under CPU starvation. Real mode did not show this cost in
+steady state: the store is itself the read state, and a one-off measurement
+with a temporary counter (not asserted by a test) found a full
+`updateResourceStore` pass over 1,694 resources (populate, metric syncs and
+alert sync) taking 108 ms with 121 registry resolutions that fell back to the
+full-mapping scan, because the registry's typed views are built before the
+alert loop runs. `BenchmarkLiveRegistryMetricsTargetResolution` records only
+the lookup cost of a registry whose views are unbuilt against one whose views
+are built (136 ms against 5 ms per 5,079 lookups); it controls the warming
+itself, so it cannot detect production failing to warm the views before the
+alert loop, which is what the one-off measurement above showed. Run both
+benchmarks with `-benchtime=3x`, since the registry case rebuilds its ingest
+outside the timer. `TestMockMetricsTargetLookupsDoNotRebuildTheViewAfterFixtureTicks`
+is the guard that fails on a rebuild per lookup. The remaining startup cost
+under starvation is the monitor's repeated full resource-store passes (one per
+supplemental provider registration) and the other consumers of the shared view,
+which rebuild it whenever they find it stale; neither is changed here.
+
### Update evidence reuses the bounded node observation
Package evidence adds only one bounded status, one bounded reason, and the
diff --git a/internal/mock/integration.go b/internal/mock/integration.go
index 219946857c..8ae4c87c58 100644
--- a/internal/mock/integration.go
+++ b/internal/mock/integration.go
@@ -218,8 +218,7 @@ func enableMockMode(config MockConfig, fromInit bool) {
dataMu.Lock()
mockConfig = config
mockGraph = buildFixtureGraph(config, now)
- fixtureRevision.Add(1)
- fixtureDataVersion.Add(1)
+ advanceFixtureStructure()
metricCohort.Store(0)
enabled.Store(true)
dataMu.Unlock()
@@ -255,8 +254,7 @@ func disableMockMode() {
dataMu.Lock()
mockGraph = emptyFixtureGraph()
- fixtureRevision.Add(1)
- fixtureDataVersion.Add(1)
+ advanceFixtureStructure()
dataMu.Unlock()
log.Info().Msg("mock mode disabled")
@@ -334,6 +332,33 @@ func FixtureDataVersion() uint64 {
return fixtureDataVersion.Load()
}
+// FixtureStructureRevision returns a token that advances only when the mock
+// estate itself changes (mock mode enabled or disabled, or a configuration
+// that rebuilds the graph), never on a metric tick. Which resources exist,
+// their identities and their source mappings are fixed between two reads of
+// the same revision, so a lookup that depends only on those, such as a
+// resource's metrics target, may be served from a view built at any data
+// version of that revision.
+//
+// A cache that tags a snapshot with the revision must read the revision
+// BEFORE the data version it memoizes the snapshot under. The two counters
+// advance under dataMu, data version first (advanceFixtureStructure), so a
+// reader that sees a new revision is guaranteed the new data version after
+// it and cannot be handed the previous estate's memoized snapshot under the
+// new revision; the other interleaving only tags a newer snapshot with an
+// older revision, which costs one rebuild.
+func FixtureStructureRevision() uint64 {
+ return fixtureRevision.Load()
+}
+
+// advanceFixtureStructure records a structural change of the fixture graph.
+// Callers hold dataMu for writing. The data version moves before the
+// structure revision; see FixtureStructureRevision for the ordering contract.
+func advanceFixtureStructure() {
+ fixtureDataVersion.Add(1)
+ fixtureRevision.Add(1)
+}
+
// GetConfig returns the current mock configuration.
func GetConfig() MockConfig {
dataMu.RLock()
@@ -539,8 +564,7 @@ func SetMockConfig(cfg MockConfig) {
setMockUpdateInterval(normalized.UpdateInterval)
if configChanged && enabled.Load() {
mockGraph = buildFixtureGraph(normalized, time.Now())
- fixtureRevision.Add(1)
- fixtureDataVersion.Add(1)
+ advanceFixtureStructure()
}
dataMu.Unlock()
diff --git a/internal/mock/platform_fixtures_test.go b/internal/mock/platform_fixtures_test.go
index 7f25908432..bf5db89b7f 100644
--- a/internal/mock/platform_fixtures_test.go
+++ b/internal/mock/platform_fixtures_test.go
@@ -4,8 +4,15 @@ import (
"bytes"
"encoding/json"
"fmt"
+ "go/ast"
+ "go/parser"
+ "go/token"
"math"
+ "os"
+ "path/filepath"
+ "reflect"
"slices"
+ "strings"
"testing"
"time"
@@ -1017,3 +1024,280 @@ func TestFixtureGraphMetricCohortProducesSparseUnifiedResourceChanges(t *testing
t.Fatalf("expected sparse unified resource churn, changed=%d total=%d", changed, len(after))
}
}
+
+func fixtureMetricsTargets(t *testing.T) map[string]unifiedresources.MetricsTarget {
+ t.Helper()
+ resources, _ := UnifiedResourceSnapshot()
+ if len(resources) == 0 {
+ t.Fatal("expected mock resources")
+ }
+ registry := unifiedresources.NewRegistry(nil)
+ registry.IngestResources(resources)
+ _, targets := registry.ListWithMetricsTargets()
+ if len(targets) == 0 {
+ t.Fatal("expected the mock estate to resolve metrics targets")
+ }
+ return targets
+}
+
+func fixtureResourceIDs(t *testing.T) []string {
+ t.Helper()
+ resources, _ := UnifiedResourceSnapshot()
+ ids := make([]string, 0, len(resources))
+ for _, resource := range resources {
+ ids = append(ids, resource.ID)
+ }
+ slices.Sort(ids)
+ return ids
+}
+
+// smallMockEstateForTest replaces the ambient estate with a fixed small one,
+// so a test neither depends on the process's PULSE_MOCK_* settings (a node
+// count at the supported maximum clamps back when raised) nor pays for a
+// 929-guest snapshot per assertion. The ambient configuration returns when
+// the test ends.
+func smallMockEstateForTest(t *testing.T, nodes int) MockConfig {
+ t.Helper()
+ previous := GetConfig()
+ t.Cleanup(func() { SetMockConfig(previous) })
+ small := previous
+ small.NodeCount = nodes
+ SetMockConfig(small)
+ return small
+}
+
+// Callers cache a resource's metrics target for as long as the structure
+// revision holds (see Monitor.MetricsTargetForResource), so a metric tick must
+// leave the revision, the resource set and every target exactly as they were.
+// Checked after every tick of two full cohort rotations, not only at the end,
+// so a change that a later tick reverts is still caught.
+func TestMetricTicksNeverChangeTheFixtureStructure(t *testing.T) {
+ withMockEnabledForTest(t)
+ stopUpdateLoop()
+ t.Cleanup(func() {
+ if IsMockEnabled() {
+ startUpdateLoop()
+ }
+ })
+ smallMockEstateForTest(t, 6)
+ stopUpdateLoop()
+
+ revision := FixtureStructureRevision()
+ ids := fixtureResourceIDs(t)
+ targets := fixtureMetricsTargets(t)
+ version := FixtureDataVersion()
+
+ for tick := 1; tick <= mockMetricCohortCount*2+1; tick++ {
+ updateMetrics(GetConfig())
+
+ if got := FixtureDataVersion(); got <= version {
+ t.Fatalf("tick %d did not advance the data version (%d)", tick, got)
+ } else {
+ version = got
+ }
+ if got := FixtureStructureRevision(); got != revision {
+ t.Fatalf("tick %d moved the fixture structure revision from %d to %d", tick, revision, got)
+ }
+ if got := fixtureResourceIDs(t); !slices.Equal(got, ids) {
+ t.Fatalf("tick %d changed the resource set: %d resources before, %d after", tick, len(ids), len(got))
+ }
+ if got := fixtureMetricsTargets(t); !reflect.DeepEqual(got, targets) {
+ t.Fatalf("tick %d changed the resolved metrics targets", tick)
+ }
+ }
+}
+
+func TestFixtureStructureRevisionAdvancesOnStructuralChange(t *testing.T) {
+ withMockEnabledForTest(t)
+ stopUpdateLoop()
+ t.Cleanup(func() {
+ if IsMockEnabled() {
+ startUpdateLoop()
+ }
+ })
+ small := smallMockEstateForTest(t, 3)
+ stopUpdateLoop()
+
+ revision := FixtureStructureRevision()
+ grown := small
+ grown.NodeCount = 4
+ SetMockConfig(grown)
+ if got := FixtureStructureRevision(); got <= revision {
+ t.Fatalf("reconfiguring the estate left the structure revision at %d (was %d)", got, revision)
+ }
+
+ revision = FixtureStructureRevision()
+ if err := SetEnabled(false); err != nil {
+ t.Fatalf("disable mock mode: %v", err)
+ }
+ if got := FixtureStructureRevision(); got <= revision {
+ t.Fatalf("disabling mock mode left the structure revision at %d (was %d)", got, revision)
+ }
+ revision = FixtureStructureRevision()
+ if err := SetEnabled(true); err != nil {
+ t.Fatalf("enable mock mode: %v", err)
+ }
+ if got := FixtureStructureRevision(); got <= revision {
+ t.Fatalf("enabling mock mode left the structure revision at %d (was %d)", got, revision)
+ }
+}
+
+// A cache fill reads the structure revision before the data version, and the
+// fixture advances its data version before its revision, so a reader that
+// sees a new revision is guaranteed the new data version after it and is
+// never handed the previous estate's memoized snapshot under the new
+// revision. The interleaving cannot be forced from outside, so the order of
+// the two writes and of the two reads is pinned on the syntax tree: inside
+// the named functions only, so comments, local names and the position of the
+// functions in their package do not matter.
+func TestFixtureStructureRevisionOrderingContract(t *testing.T) {
+ mockDecls := parsePackageFuncs(t, ".")
+ advance := packageFunc(mockDecls["advanceFixtureStructure"])
+ if advance == nil {
+ t.Fatal("the mock package lost advanceFixtureStructure")
+ }
+ version := synchronousCallOffsets(advance, "fixtureDataVersion", "Add")
+ revision := synchronousCallOffsets(advance, "fixtureRevision", "Add")
+ if len(version) != 1 || len(revision) != 1 || version[0] > revision[0] {
+ t.Fatal("advanceFixtureStructure must advance the data version once, before the structure revision once, as plain statements")
+ }
+ if got := countCalls(advance, "fixtureRevision", "Add") + countCalls(advance, "fixtureDataVersion", "Add"); got != 2 {
+ t.Fatalf("advanceFixtureStructure must not defer or spawn its counter writes; found %d writes in all", got)
+ }
+
+ // Every declaration counts for the bypass scan, including methods and the
+ // build-tagged duplicates of a function.
+ for name, decls := range mockDecls {
+ if name == "advanceFixtureStructure" {
+ continue
+ }
+ for _, decl := range decls {
+ if got := countCalls(decl, "fixtureRevision", "Add"); got != 0 {
+ t.Fatalf("%s advances the structure revision directly; it must go through advanceFixtureStructure", name)
+ }
+ }
+ }
+ for _, name := range []string{"enableMockMode", "disableMockMode", "SetMockConfig"} {
+ decl := packageFunc(mockDecls[name])
+ if decl == nil {
+ t.Fatalf("the mock package lost %s", name)
+ }
+ if got := len(synchronousIdentCallOffsets(decl, "advanceFixtureStructure")); got != 1 {
+ t.Fatalf("%s must advance the fixture structure through advanceFixtureStructure exactly once, as a plain statement; found %d", name, got)
+ }
+ }
+
+ view := packageFunc(parsePackageFuncs(t, "../monitoring")["currentUnifiedStateView"])
+ if view == nil {
+ t.Fatal("the monitoring package lost currentUnifiedStateView")
+ }
+ structureRead := synchronousCallOffsets(view, "mock", "FixtureStructureRevision")
+ versionRead := synchronousCallOffsets(view, "mock", "FixtureDataVersion")
+ if len(structureRead) != 1 || len(versionRead) != 1 || structureRead[0] > versionRead[0] {
+ t.Fatal("currentUnifiedStateView must read the structure revision once, before it reads the data version")
+ }
+}
+
+// parsePackageFuncs returns every non-test function and method declaration of
+// the package in dir, keyed by name. Names repeat across build-tagged files
+// and between functions and methods, so each key holds all of them.
+func parsePackageFuncs(t *testing.T, dir string) map[string][]*ast.FuncDecl {
+ t.Helper()
+ entries, err := os.ReadDir(dir)
+ if err != nil {
+ t.Fatalf("read %s: %v", dir, err)
+ }
+ funcs := make(map[string][]*ast.FuncDecl)
+ fset := token.NewFileSet()
+ for _, entry := range entries {
+ name := entry.Name()
+ if entry.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") {
+ continue
+ }
+ file, err := parser.ParseFile(fset, filepath.Join(dir, name), nil, 0)
+ if err != nil {
+ t.Fatalf("parse %s: %v", name, err)
+ }
+ for _, decl := range file.Decls {
+ if fn, ok := decl.(*ast.FuncDecl); ok {
+ funcs[fn.Name.Name] = append(funcs[fn.Name.Name], fn)
+ }
+ }
+ }
+ return funcs
+}
+
+// packageFunc returns the declaration when it is the only one of its name in
+// the package (function or method), and nil otherwise.
+func packageFunc(decls []*ast.FuncDecl) *ast.FuncDecl {
+ if len(decls) != 1 {
+ return nil
+ }
+ return decls[0]
+}
+
+// walkSynchronous visits the nodes of fn that run in the function's own
+// control flow: it does not descend into function literals, deferred calls or
+// goroutine starts, whose source position says nothing about when they run.
+func walkSynchronous(fn *ast.FuncDecl, visit func(ast.Node)) {
+ ast.Inspect(fn, func(node ast.Node) bool {
+ switch node.(type) {
+ case *ast.FuncLit, *ast.DeferStmt, *ast.GoStmt:
+ return false
+ }
+ visit(node)
+ return true
+ })
+}
+
+func isSelectorCall(node ast.Node, receiver, method string) bool {
+ call, ok := node.(*ast.CallExpr)
+ if !ok {
+ return false
+ }
+ selector, ok := call.Fun.(*ast.SelectorExpr)
+ if !ok || selector.Sel.Name != method {
+ return false
+ }
+ ident, ok := selector.X.(*ast.Ident)
+ return ok && ident.Name == receiver
+}
+
+// synchronousCallOffsets lists the positions of receiver.method calls that run
+// in fn's own control flow, in source order.
+func synchronousCallOffsets(fn *ast.FuncDecl, receiver, method string) []token.Pos {
+ var offsets []token.Pos
+ walkSynchronous(fn, func(node ast.Node) {
+ if isSelectorCall(node, receiver, method) {
+ offsets = append(offsets, node.Pos())
+ }
+ })
+ return offsets
+}
+
+// synchronousIdentCallOffsets lists the positions of calls to the package-level
+// function name that run in fn's own control flow.
+func synchronousIdentCallOffsets(fn *ast.FuncDecl, name string) []token.Pos {
+ var offsets []token.Pos
+ walkSynchronous(fn, func(node ast.Node) {
+ if call, ok := node.(*ast.CallExpr); ok {
+ if ident, ok := call.Fun.(*ast.Ident); ok && ident.Name == name {
+ offsets = append(offsets, call.Pos())
+ }
+ }
+ })
+ return offsets
+}
+
+// countCalls counts every receiver.method call anywhere in fn, including
+// deferred calls, goroutines and function literals.
+func countCalls(fn *ast.FuncDecl, receiver, method string) int {
+ count := 0
+ ast.Inspect(fn, func(node ast.Node) bool {
+ if isSelectorCall(node, receiver, method) {
+ count++
+ }
+ return true
+ })
+ return count
+}
diff --git a/internal/monitoring/metrics_target_resolution.go b/internal/monitoring/metrics_target_resolution.go
new file mode 100644
index 0000000000..6d81f731a3
--- /dev/null
+++ b/internal/monitoring/metrics_target_resolution.go
@@ -0,0 +1,63 @@
+package monitoring
+
+import (
+ "github.com/rcourtman/pulse-go-rewrite/internal/mock"
+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
+)
+
+// mockMetricsTarget resolves a canonical resource ID to its metrics-store
+// target in mock mode. The second result is false when mock mode is off, so
+// the caller takes its ordinary path; when it is true the answer is final for
+// this structure, a nil target meaning the view captured no metrics target for
+// the ID (the estate does not list it, or lists a resource that has none, such
+// as a VMware network); the caller still tries the raw resource store.
+//
+// Alert evaluation resolves a target for every resource and every windowed
+// metric (CPU by default) on every pass, through MetricsTargetForResource.
+// The mock view that answers it is cached per fixture data version, and that
+// version advances on every metric tick (2 s by default). Rebuilding the view
+// ingests the whole estate twice (the memoized fixture snapshot, then the
+// view's own registry), about half a second of CPU for the default estate
+// (1,776 resources) on an idle core. A build that takes longer than one tick publishes
+// its view under an already-stale version, so the next lookup rebuilds again:
+// on a host with CPU to spare that costs one build per tick, but when the
+// process is starved (a loaded dev VM, a small demo host) the cache never
+// hits, every lookup in the pass builds an estate, and the monitor never
+// finishes its first evaluation. Startup stalled in exactly that state,
+// before the listener opened, with the watchdog dump parked in the view build
+// beneath evaluateMetricWindow.
+//
+// A metrics target depends only on identity inputs: a resource's type, its
+// source mappings, and the technology, Docker container ID and disk identity
+// of its facets (unifiedresources.BuildMetricsTarget), falling back to the
+// resource's own MetricsTarget. A metric tick moves values and timestamps,
+// never those inputs (TestMetricTicksNeverChangeTheFixtureStructure), so the
+// captured targets of a view built at any data version of the current
+// structure answer the lookup. A data-version change alone therefore never
+// triggers a build here: only a structural change (mock mode toggled, the
+// fixture reconfigured) or a link change makes the next lookup rebuild the
+// view. Two lookups that miss together can each still build one, and a cold
+// cache builds even for an ID the estate does not list.
+func (m *Monitor) mockMetricsTarget(resourceID string) (*unifiedresources.MetricsTarget, bool) {
+ if m == nil || !mock.IsMockEnabled() {
+ return nil, false
+ }
+
+ structure := mock.FixtureStructureRevision()
+ links := m.resourceStoreManualLinks()
+
+ m.mockUnifiedViewMu.Lock()
+ view := m.mockUnifiedView
+ reusable := m.mockUnifiedViewValid &&
+ m.mockUnifiedViewStructure == structure &&
+ sameManualLinks(m.mockUnifiedViewLinks, links)
+ m.mockUnifiedViewMu.Unlock()
+
+ if !reusable {
+ view = m.currentUnifiedStateView()
+ }
+ if view.metricsTargets == nil {
+ return nil, true
+ }
+ return view.metricsTargets.MetricsTargetForResource(resourceID), true
+}
diff --git a/internal/monitoring/metrics_target_resolution_test.go b/internal/monitoring/metrics_target_resolution_test.go
new file mode 100644
index 0000000000..79122e93ee
--- /dev/null
+++ b/internal/monitoring/metrics_target_resolution_test.go
@@ -0,0 +1,158 @@
+package monitoring
+
+import (
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/mock"
+ "github.com/rcourtman/pulse-go-rewrite/internal/models"
+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
+)
+
+// useMockEstate enables the default mock estate with the given node count,
+// whose fixture ticks every interval, and restores the previous mock mode and
+// configuration when the test ends.
+func useMockEstate(tb testing.TB, nodes int, interval time.Duration) {
+ tb.Helper()
+ previous := mock.IsMockEnabled()
+ previousConfig := mock.GetConfig()
+ if previous {
+ mustSetMockEnabled(tb, false)
+ }
+ // Start from the default estate, not the ambient one: PULSE_MOCK_* settings
+ // can remove the standalone agents and clusters these tests look for.
+ config := mock.DefaultConfig
+ config.NodeCount = nodes
+ config.UpdateInterval = interval
+ mock.SetMockConfig(config)
+ mustSetMockEnabled(tb, true)
+ tb.Cleanup(func() {
+ mustSetMockEnabled(tb, false)
+ mock.SetMockConfig(previousConfig)
+ mustSetMockEnabled(tb, previous)
+ })
+}
+
+func newMockEstateMonitor() *Monitor {
+ return &Monitor{
+ state: models.NewState(),
+ resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(unifiedresources.NewMemoryStore())),
+ }
+}
+
+// publishedMockView returns the view the monitor currently caches for mock
+// mode without building one.
+func publishedMockView(m *Monitor) (monitorUnifiedStateView, bool) {
+ m.mockUnifiedViewMu.Lock()
+ defer m.mockUnifiedViewMu.Unlock()
+ return m.mockUnifiedView, m.mockUnifiedViewValid
+}
+
+// sameMetricsTarget reports whether two lookups resolved the same target (or
+// both resolved none).
+func sameMetricsTarget(a, b *unifiedresources.MetricsTarget) bool {
+ if a == nil || b == nil {
+ return a == b
+ }
+ return *a == *b
+}
+
+func waitForFixtureTicks(tb testing.TB, ticks uint64) {
+ tb.Helper()
+ target := mock.FixtureDataVersion() + ticks
+ // A starved host ticks slowly; the deadline only catches a ticker that
+ // is not running at all.
+ deadline := time.Now().Add(5 * time.Minute)
+ for mock.FixtureDataVersion() < target {
+ if time.Now().After(deadline) {
+ tb.Fatalf("fixture data version stayed below %d (at %d) for five minutes; the mock ticker is not running", target, mock.FixtureDataVersion())
+ }
+ time.Sleep(20 * time.Millisecond)
+ }
+}
+
+func resourceIDsWithTargets(tb testing.TB, view monitorUnifiedStateView) []string {
+ tb.Helper()
+ ids := make([]string, 0, len(view.resources))
+ for _, resource := range view.resources {
+ if view.metricsTargets.MetricsTargetForResource(resource.ID) != nil {
+ ids = append(ids, resource.ID)
+ }
+ }
+ if len(ids) == 0 {
+ tb.Fatal("expected the mock estate to list resources with metrics targets")
+ }
+ return ids
+}
+
+// BenchmarkMockMetricsTargetResolutionAfterTick measures one alert pass's
+// lookups (every resource, three windowed metrics) against a view that was
+// built before the fixture last ticked. It must stay a map read per lookup, so
+// a pass costs microseconds however slowly the process runs, not an estate
+// build per lookup.
+func BenchmarkMockMetricsTargetResolutionAfterTick(b *testing.B) {
+ useMockEstate(b, mock.DefaultConfig.NodeCount, time.Second)
+ m := newMockEstateMonitor()
+ ids := resourceIDsWithTargets(b, m.currentUnifiedStateView())
+ waitForFixtureTicks(b, 1)
+
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ for _, id := range ids {
+ for metric := 0; metric < 3; metric++ {
+ if m.MetricsTargetForResource(id) == nil {
+ b.Fatalf("no target for %q", id)
+ }
+ }
+ }
+ }
+ b.ReportMetric(float64(len(ids)*3), "lookups/op")
+}
+
+// BenchmarkLiveRegistryMetricsTargetResolution measures the same pass against a
+// store-backed registry, as real mode resolves it, right after an ingest (the
+// registry's typed views not yet built) and once they are.
+func BenchmarkLiveRegistryMetricsTargetResolution(b *testing.B) {
+ useMockEstate(b, mock.DefaultConfig.NodeCount, 5*time.Minute)
+ graph := mock.CurrentFixtureGraph()
+ snapshot := unifiedresources.SnapshotWithoutSources(graph.State, mock.SupplementalOwnedSources())
+ mustSetMockEnabled(b, false)
+
+ for _, views := range []string{"after-ingest", "views-built"} {
+ b.Run(views, func(b *testing.B) {
+ registry := unifiedresources.NewRegistry(nil)
+ registry.IngestSnapshot(snapshot)
+ adapter := unifiedresources.NewMonitorAdapter(registry)
+ m := &Monitor{state: models.NewState(), resourceStore: adapter}
+
+ listed := registry.List()
+ ids := make([]string, 0, len(listed))
+ for _, resource := range listed {
+ ids = append(ids, resource.ID)
+ }
+ if views == "views-built" {
+ adapter.VMs()
+ }
+
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ if views == "after-ingest" {
+ b.StopTimer()
+ registry = unifiedresources.NewRegistry(nil)
+ registry.IngestSnapshot(snapshot)
+ adapter = unifiedresources.NewMonitorAdapter(registry)
+ m.resourceStore = adapter
+ b.StartTimer()
+ }
+ for _, id := range ids {
+ for metric := 0; metric < 3; metric++ {
+ m.MetricsTargetForResource(id)
+ }
+ }
+ }
+ b.ReportMetric(float64(len(ids)*3), "lookups/op")
+ })
+ }
+}
diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.go
index 317d4130dc..5fe35958a8 100644
--- a/internal/monitoring/monitor.go
+++ b/internal/monitoring/monitor.go
@@ -1216,6 +1216,7 @@ type Monitor struct {
mockUnifiedViewMu sync.Mutex
mockUnifiedView monitorUnifiedStateView
mockUnifiedViewVersion uint64
+ mockUnifiedViewStructure uint64 // fixture structure revision the published view was built at
mockUnifiedViewLinks []unifiedresources.ResourceLink
mockUnifiedViewFixtureAt time.Time // fixture freshness the published view was built from
mockUnifiedViewValid bool
@@ -5049,10 +5050,20 @@ func (m *Monitor) MetricsTargetForResource(resourceID string) *unifiedresources.
return nil
}
- if view := m.GetUnifiedReadStateOrSnapshot(); view != nil {
- if resolver, ok := view.(MetricsTargetResourceStore); ok {
- if target := resolver.MetricsTargetForResource(resourceID); target != nil {
- return target
+ // Mock mode resolves from the cached view's captured targets and never
+ // asks the view again: this runs once per resource and windowed metric in
+ // every alert pass, and the fixture's data version moves on every tick
+ // (see mockMetricsTarget).
+ target, fromMockView := m.mockMetricsTarget(resourceID)
+ if target != nil {
+ return target
+ }
+ if !fromMockView {
+ if view := m.GetUnifiedReadStateOrSnapshot(); view != nil {
+ if resolver, ok := view.(MetricsTargetResourceStore); ok {
+ if target := resolver.MetricsTargetForResource(resourceID); target != nil {
+ return target
+ }
}
}
}
@@ -5210,7 +5221,12 @@ func (m *Monitor) currentUnifiedStateView() monitorUnifiedStateView {
// Read the version and links before the snapshot so a tick or
// rebuild landing in between caches newer data under an older
// token (harmless rebuild next call) rather than ever serving stale
- // data under a newer one.
+ // data under a newer one. The structure revision goes first: the
+ // fixture advances its data version before its revision, so a
+ // reader that sees a new revision also reads the new version and
+ // never memoizes the previous estate under it
+ // (mock.FixtureStructureRevision).
+ structure := mock.FixtureStructureRevision()
version := mock.FixtureDataVersion()
links := m.resourceStoreManualLinks()
m.mockUnifiedViewMu.Lock()
@@ -5255,6 +5271,7 @@ func (m *Monitor) currentUnifiedStateView() monitorUnifiedStateView {
m.mockUnifiedView = view
m.mockUnifiedViewFixtureAt = fixtureFreshness
m.mockUnifiedViewVersion = version
+ m.mockUnifiedViewStructure = structure
m.mockUnifiedViewLinks = links
m.mockUnifiedViewValid = true
m.mockUnifiedViewMu.Unlock()
diff --git a/internal/monitoring/monitor_host_agents_test.go b/internal/monitoring/monitor_host_agents_test.go
index b4127d27bf..46cf04cea2 100644
--- a/internal/monitoring/monitor_host_agents_test.go
+++ b/internal/monitoring/monitor_host_agents_test.go
@@ -5992,6 +5992,187 @@ func TestMockUnifiedViewAppliesOperatorManualLinks(t *testing.T) {
}
}
+// Alert evaluation resolves a target per resource and windowed metric on every
+// pass. A metric tick moves the fixture's data version, not what resources
+// exist, so a lookup after a tick must be served from the cached view instead
+// of building an estate. Before this was pinned, every lookup after a tick
+// rebuilt the view, and a process too starved to finish one build within a tick
+// rebuilt it for every lookup of the pass: startup never reached its listener.
+func TestMockMetricsTargetLookupsDoNotRebuildTheViewAfterFixtureTicks(t *testing.T) {
+ useMockEstate(t, 4, time.Second)
+ m := newMockEstateMonitor()
+
+ warm := m.currentUnifiedStateView()
+ ids := resourceIDsWithTargets(t, warm)
+
+ waitForFixtureTicks(t, 2)
+
+ for _, id := range ids {
+ want := warm.metricsTargets.MetricsTargetForResource(id)
+ got := m.MetricsTargetForResource(id)
+ if !sameMetricsTarget(got, want) {
+ t.Fatalf("MetricsTargetForResource(%q) = %+v after fixture ticks, want %+v", id, got, want)
+ }
+ }
+
+ after, ok := publishedMockView(m)
+ if !ok || after.readState != warm.readState {
+ t.Fatal("target lookups rebuilt the unified view after a metric tick; they must reuse the view built for the current fixture structure")
+ }
+}
+
+// The reused view answers for the estate it was built from. A structural
+// change (here a larger estate) must reach the lookup, not keep serving the
+// old resource set.
+func TestMockMetricsTargetFollowsAStructuralFixtureChange(t *testing.T) {
+ useMockEstate(t, 3, 5*time.Minute)
+ m := newMockEstateMonitor()
+
+ before := m.currentUnifiedStateView()
+ known := make(map[string]struct{}, len(before.resources))
+ for _, resource := range before.resources {
+ known[resource.ID] = struct{}{}
+ }
+
+ grown := mock.GetConfig()
+ grown.NodeCount++
+ mock.SetMockConfig(grown)
+
+ // Another monitor's view of the grown estate names the resources the
+ // first one lacks. The first monitor has not been asked for a view since
+ // the change, so the lookup itself has to notice the structure moved.
+ var added string
+ for _, resource := range newMockEstateMonitor().currentUnifiedStateView().resources {
+ if _, ok := known[resource.ID]; !ok && resource.Type == unifiedresources.ResourceTypeAgent {
+ added = resource.ID
+ break
+ }
+ }
+ if added == "" {
+ t.Fatal("the larger fixture estate lists no agent the first one did not")
+ }
+ if target := m.MetricsTargetForResource(added); target == nil {
+ t.Fatalf("MetricsTargetForResource(%q) = nil after the fixture grew; the cached view outlived its structure", added)
+ }
+}
+
+// Reusing a view across ticks must not change an answer: every resource
+// resolves to what the view's own registry resolves, and an ID the estate does
+// not list resolves to nothing without building a view for it.
+func TestMockMetricsTargetMatchesTheRegistryResolution(t *testing.T) {
+ useMockEstate(t, 4, 5*time.Minute)
+ m := newMockEstateMonitor()
+
+ view := m.currentUnifiedStateView()
+ registryResolver, ok := view.readState.(MetricsTargetResourceStore)
+ if !ok {
+ t.Fatalf("mock view read state %T does not resolve metrics targets", view.readState)
+ }
+ for _, resource := range view.resources {
+ want := registryResolver.MetricsTargetForResource(resource.ID)
+ got := m.MetricsTargetForResource(resource.ID)
+ if !sameMetricsTarget(got, want) {
+ t.Fatalf("MetricsTargetForResource(%q) = %+v, want the registry's %+v", resource.ID, got, want)
+ }
+ if padded := m.MetricsTargetForResource(" " + resource.ID + " "); !sameMetricsTarget(padded, want) {
+ t.Fatalf("MetricsTargetForResource(%q padded) = %+v, want the canonical ID's %+v", resource.ID, padded, want)
+ }
+ }
+ if got := m.MetricsTargetForResource("not-a-mock-resource"); got != nil {
+ t.Fatalf("MetricsTargetForResource(unknown) = %+v, want nil", got)
+ }
+ if after, _ := publishedMockView(m); after.readState != view.readState {
+ t.Fatal("resolving an unknown ID rebuilt the cached view")
+ }
+}
+
+// A lookup for an ID the estate does not list must stay cheap after a tick
+// too: the view has no more to say about it than the one it replaces.
+func TestMockMetricsTargetForUnknownIDDoesNotRebuildTheViewAfterFixtureTicks(t *testing.T) {
+ useMockEstate(t, 3, time.Second)
+ m := newMockEstateMonitor()
+
+ warm := m.currentUnifiedStateView()
+ waitForFixtureTicks(t, 2)
+ for i := 0; i < 50; i++ {
+ if got := m.MetricsTargetForResource(fmt.Sprintf("unknown-%d", i)); got != nil {
+ t.Fatalf("MetricsTargetForResource(unknown) = %+v, want nil", got)
+ }
+ }
+ if after, _ := publishedMockView(m); after.readState != warm.readState {
+ t.Fatal("unknown-ID lookups rebuilt the unified view after a metric tick")
+ }
+}
+
+// An operator link folds one resource into another, which changes what the
+// folded ID resolves to. Whatever rebuilds the view first, a lookup after the
+// resource store applied the link must not resolve the folded resource.
+func TestMockMetricsTargetFollowsAnOperatorLink(t *testing.T) {
+ useMockEstate(t, 4, 5*time.Minute)
+ store := unifiedresources.NewMemoryStore()
+ m := &Monitor{
+ state: models.NewState(),
+ resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(store)),
+ alertManager: alerts.NewManager(),
+ }
+ t.Cleanup(m.alertManager.Stop)
+
+ var vmID, agentID string
+ for _, resource := range m.currentUnifiedStateView().resources {
+ switch {
+ case vmID == "" && resource.Type == unifiedresources.ResourceTypeVM && resource.Agent == nil:
+ vmID = resource.ID
+ case agentID == "" && resource.Type == unifiedresources.ResourceTypeAgent && resource.Agent != nil &&
+ len(resource.Sources) == 1 && resource.Sources[0] == unifiedresources.SourceAgent:
+ agentID = resource.ID
+ }
+ }
+ if vmID == "" || agentID == "" {
+ t.Fatalf("fixture graph needs an agentless VM and a standalone agent, got vm=%q agent=%q", vmID, agentID)
+ }
+ if m.MetricsTargetForResource(agentID) == nil || m.MetricsTargetForResource(vmID) == nil {
+ t.Fatal("expected the unlinked VM and agent to resolve metrics targets")
+ }
+
+ if err := store.AddLink(unifiedresources.ResourceLink{ResourceA: vmID, ResourceB: agentID, PrimaryID: vmID}); err != nil {
+ t.Fatalf("AddLink: %v", err)
+ }
+ // Rebuild the resource store so it loads the link, without asking for a
+ // mock view: the lookup itself has to see the link list change.
+ m.updateResourceStore(models.StateSnapshot{}, m.mockModeFence.begin())
+
+ if got := m.MetricsTargetForResource(agentID); got != nil {
+ t.Fatalf("MetricsTargetForResource(folded agent) = %+v after the operator link, want nothing", got)
+ }
+ if m.MetricsTargetForResource(vmID) == nil {
+ t.Fatal("the linked VM lost its metrics target")
+ }
+}
+
+// The reused view was built under one link list; once the resource store
+// applies a different one, a lookup must rebuild rather than answer from it.
+// A rebuild by any other consumer would hide a missing check, so the list the
+// view was built under is changed directly.
+func TestMockMetricsTargetRebuildsWhenTheLinkListMoves(t *testing.T) {
+ useMockEstate(t, 3, 5*time.Minute)
+ m := newMockEstateMonitor()
+
+ warm := m.currentUnifiedStateView()
+ id := resourceIDsWithTargets(t, warm)[0]
+
+ m.mockUnifiedViewMu.Lock()
+ m.mockUnifiedViewLinks = []unifiedresources.ResourceLink{{ResourceA: "stale-a", ResourceB: "stale-b", PrimaryID: "stale-a"}}
+ m.mockUnifiedViewMu.Unlock()
+
+ if m.MetricsTargetForResource(id) == nil {
+ t.Fatalf("MetricsTargetForResource(%q) = nil", id)
+ }
+ after, _ := publishedMockView(m)
+ if after.readState == warm.readState {
+ t.Fatal("a lookup answered from a view built under a different operator link list")
+ }
+}
+
// TestMockModeDiscardsRealHostReports pins the push-side of the mock clean
// room. Mock mode already suspends pull-based PVE/PBS/PMG collection, but a
// real agent keeps POSTing regardless; ingesting those reports lands a real