Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Rebuild the mock view when only a link's creation time changes

rcourtman0582d2922aad
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 4d44926cc5..bf0471ad33 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -9480,3 +9480,14 @@ report, enrollment, lease or command path is touched
(`TestSharedSystemAlertCorrelationForHostLazyListsNodesOnDemand` and
`TestMockStructureConsumersDoNotRebuildTheViewAfterFixtureTicks` in
`internal/monitoring/monitor_host_agents_test.go`).
+
+### Mock-mode unified view cache follows a link's creation time
+
+`internal/monitoring/monitor.go` changed only so `sameManualLinks`, the
+comparison behind the mock-mode unified view's cache, also compares each
+link's creation time (monitoring contract, "Mock-mode unified view applies
+operator links"): among otherwise equal candidates a chain of links folds into
+the primary of its earliest-created link, so lists that differ only in a time
+can leave a different row standing. Agent registration, enrolment, install, update,
+removal, report identity and continuity are unchanged, and real-mode
+resolution still goes through the live registry.
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 80d939cbf4..b198549729 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -3596,8 +3596,16 @@ current generation (`resourceStoreManualLinks`,
broadcast when the resource store's next rebuild loads it, as in live mode; in
mock mode that is normally the broadcast's own read-path refresh once the
generation is older than `readPathRegistryFreshness`. The cached view is keyed
-on the fixture data version and the link list, so a link change rebuilds it
-without a fixture tick; an unchanged list keeps the shared build. Freshness,
+on the fixture data version and the link list, each link by its resource pair,
+its primary and its creation time (`sameManualLinks`, which the structure view
+reuses), so a link change rebuilds it without a fixture tick; an unchanged
+list keeps the shared build. The creation time is in the key because, among
+otherwise equal candidates, a chain of links folds into the primary of its
+earliest-created link (unified-resources contract, "Operator links reach
+record-ingested resources", "Links apply as chains"): lists that differ only
+in a time can leave a different row standing, and a key that ignored it kept
+serving the row the old times chose while the live rebuild, which reloads the
+links from the store, served the new one. Freshness,
which the resources API keys its registry cache on, is assigned in
publication order under `mockUnifiedViewMu`: a view built from the same
fixture freshness and links as the one it replaces keeps its freshness, and
@@ -3610,7 +3618,13 @@ the mock-mode fence above): it stays a plain projection of current state for
the window before a rebuild in the new epoch, after which the registry applies
the links again. `TestMockUnifiedViewAppliesOperatorManualLinks` in
`internal/monitoring/monitor_host_agents_test.go` pins the broadcast, the
-freshness advance and the cache, `TestFixtureGraphAppliesManualLinksAtTheRebuildsBoundary`
+freshness advance and the cache,
+`TestMockUnifiedViewFollowsALinkCreationTimeChange` in the same file pins the
+creation time in that key (a Proxmox storage, a TrueNAS VM and a vSphere VM
+linked in a cycle swap the surviving VM when only one link's time moves, with
+the fixture data version held, through the structure view and the
+data-version view alike, and match what a linked mock build keeps),
+`TestFixtureGraphAppliesManualLinksAtTheRebuildsBoundary`
and `TestUnifiedResourceSnapshotWithLinksLeavesTheSharedSnapshotUnlinked` in
`internal/mock/platform_fixtures_test.go` pin the linked fixture build, and
`TestMockUnifiedStateViewUsesCanonicalMockFixtureGraph` in
diff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.md
index 540d0ec4e4..8346ea112c 100644
--- a/docs/release-control/v6/internal/subsystems/unified-resources.md
+++ b/docs/release-control/v6/internal/subsystems/unified-resources.md
@@ -7308,8 +7308,23 @@ of durable state through it.
finished list instead dropped the check with the folded row.
- A link reaches the view when the adapter's next rebuild loads it, the same
point at which the live broadcast picks it up. The view's cache is keyed on
- the fixture data version and on the links' resource pairs and primaries, so
- a link change rebuilds it without waiting for a fixture tick. Freshness is
+ the fixture data version and on each link's resource pair, primary and
+ creation time, so a link change rebuilds it without waiting for a fixture
+ tick. The creation time is part of the key because, among otherwise equal
+ candidates, a chain folds into the primary of its earliest-created link
+ (the "Links apply as chains" bullet of "Operator links reach
+ record-ingested resources"): link lists that differ only in a creation
+ time can leave a different row standing. In the storage, TrueNAS VM and
+ vSphere VM cycle above both VMs outrank the storage, so the TrueNAS VM
+ keeps its row while its link is older than the vSphere VM's and the
+ vSphere VM keeps it once the TrueNAS VM's link is newer. A key that
+ compared pairs and primaries alone kept serving the row the old times
+ chose with the fixture data version unchanged, while the live rebuild,
+ which reloads the links from the store, served the new one. The structure
+ view that serves
+ callers reading identity and topology only keys on the fixture structure
+ revision and the same link comparison, so a creation time change rebuilds
+ it too. Freshness is
assigned in publication order: a view built from the same fixture
freshness and links as the one it replaces keeps its freshness, and any
other view gets a later one than every view published before it, even when
@@ -7336,6 +7351,11 @@ the rebuild and freshness advance on a link change with the fixture data
version held, the cached build afterwards, the same rows as a store-backed
registry seeded through resource ingest the way the resources API seeds, and
no change rows reaching the store through the view.
+`TestMockUnifiedViewFollowsALinkCreationTimeChange` in the same file pins the
+creation time in the cache key: the cycle above, with only the TrueNAS VM's
+link time moved and the fixture data version held, swaps the surviving VM in
+the structure view and in the data-version view, each matching the row a
+linked mock build keeps.
`TestFixtureGraphAppliesManualLinksAtTheRebuildsBoundary` and
`TestUnifiedResourceSnapshotWithLinksLeavesTheSharedSnapshotUnlinked` in
`internal/mock/platform_fixtures_test.go` pin the rebuild's boundary (the
diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.go
index c81fe89b13..33505f2c4d 100644
--- a/internal/monitoring/monitor.go
+++ b/internal/monitoring/monitor.go
@@ -5445,10 +5445,15 @@ func (m *Monitor) resourceStoreManualLinks() []unifiedresources.ResourceLink {
}
// sameManualLinks reports whether two link lists fold the same resources in
-// the same order. Link metadata (reason, author, time) does not change a fold.
+// the same order. The reason and author do not change a fold, but the creation
+// time can: among otherwise equal candidates a chain of links folds into the
+// primary of its earliest-created link (unifiedresources linkChainRootLocked),
+// so lists that differ only in a creation time can leave a different row
+// standing.
func sameManualLinks(a, b []unifiedresources.ResourceLink) bool {
return slices.EqualFunc(a, b, func(x, y unifiedresources.ResourceLink) bool {
- return x.ResourceA == y.ResourceA && x.ResourceB == y.ResourceB && x.PrimaryID == y.PrimaryID
+ return x.ResourceA == y.ResourceA && x.ResourceB == y.ResourceB && x.PrimaryID == y.PrimaryID &&
+ x.CreatedAt.Equal(y.CreatedAt)
})
}
diff --git a/internal/monitoring/monitor_host_agents_test.go b/internal/monitoring/monitor_host_agents_test.go
index cbe431bec8..b43522ba4a 100644
--- a/internal/monitoring/monitor_host_agents_test.go
+++ b/internal/monitoring/monitor_host_agents_test.go
@@ -5992,6 +5992,171 @@ func TestMockUnifiedViewAppliesOperatorManualLinks(t *testing.T) {
}
}
+// TestMockUnifiedViewFollowsALinkCreationTimeChange pins the creation time as
+// part of the mock view's cache key. Among otherwise equal candidates a chain
+// of links folds into the primary of its earliest-created link, so a link list
+// that differs from the cached one only in a creation time can leave a
+// different row standing: in a cycle of a Proxmox storage, a TrueNAS VM and a
+// vSphere VM the storage never competes, and the TrueNAS VM keeps its row
+// while its link is older than the vSphere VM's and gives it to the vSphere VM
+// once it is newer. The cache compared resource pairs and primaries only, so
+// with the fixture data version held it kept serving the row the old times
+// chose while the live rebuild, which reloads the links from the store, served
+// the new one. Both readers of the key are held to it: the structure view
+// (cachedMockStructureView) and the data-version view
+// (currentUnifiedStateView).
+func TestMockUnifiedViewFollowsALinkCreationTimeChange(t *testing.T) {
+ useMockEstate(t, 4, 5*time.Minute)
+
+ store := unifiedresources.NewMemoryStore()
+ m := &Monitor{
+ state: models.NewState(),
+ config: &config.Config{PVEPollingInterval: 2 * time.Minute},
+ resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(store)),
+ alertManager: alerts.NewManager(),
+ }
+ t.Cleanup(m.alertManager.Stop)
+
+ soleSource := func(resource unifiedresources.Resource, source unifiedresources.DataSource) bool {
+ return len(resource.Sources) == 1 && resource.Sources[0] == source
+ }
+ var storageID, nasVMID, guestID string
+ for _, resource := range m.currentUnifiedStateView().resources {
+ switch {
+ case storageID == "" && resource.Type == unifiedresources.ResourceTypeStorage &&
+ soleSource(resource, unifiedresources.SourceProxmox):
+ storageID = resource.ID
+ case nasVMID == "" && resource.Type == unifiedresources.ResourceTypeVM &&
+ soleSource(resource, unifiedresources.SourceTrueNAS):
+ nasVMID = resource.ID
+ case guestID == "" && resource.Type == unifiedresources.ResourceTypeVM &&
+ soleSource(resource, unifiedresources.SourceVMware):
+ guestID = resource.ID
+ }
+ }
+ if storageID == "" || nasVMID == "" || guestID == "" {
+ t.Fatalf("fixture graph needs a Proxmox storage, a TrueNAS VM and a vSphere VM, got storage=%q nas=%q guest=%q",
+ storageID, nasVMID, guestID)
+ }
+
+ created := time.Date(2026, time.October, 1, 9, 0, 0, 0, time.UTC)
+ // The link whose primary is the TrueNAS VM is the last of the three, so
+ // replacing it keeps the list's order and changes only its creation time.
+ cycle := func(nasLinkCreated time.Time) []unifiedresources.ResourceLink {
+ return []unifiedresources.ResourceLink{
+ {ResourceA: nasVMID, ResourceB: storageID, PrimaryID: storageID, CreatedAt: created.Add(time.Minute)},
+ {ResourceA: storageID, ResourceB: guestID, PrimaryID: guestID, CreatedAt: created.Add(2 * time.Minute)},
+ {ResourceA: guestID, ResourceB: nasVMID, PrimaryID: nasVMID, CreatedAt: nasLinkCreated},
+ }
+ }
+ oldest, newest := created, created.Add(3*time.Minute)
+ setNasLinkCreated := func(at time.Time) {
+ t.Helper()
+ for _, link := range cycle(at) {
+ if err := store.AddLink(link); err != nil {
+ t.Fatalf("AddLink: %v", err)
+ }
+ }
+ // Reads deliberately reuse a generation younger than two seconds, so
+ // apply the rebuild boundary that loads the links instead of waiting.
+ m.updateResourceStore(m.currentStateWithScope())
+ }
+
+ // cycleRows is what is left of the three members: ID and sources.
+ cycleRows := func(resources []unifiedresources.Resource) []string {
+ var rows []string
+ for _, resource := range resources {
+ if resource.ID != storageID && resource.ID != nasVMID && resource.ID != guestID {
+ continue
+ }
+ sources := make([]string, 0, len(resource.Sources))
+ for _, source := range resource.Sources {
+ sources = append(sources, string(source))
+ }
+ slices.Sort(sources)
+ rows = append(rows, fmt.Sprintf("%s %v", resource.ID, sources))
+ }
+ return rows
+ }
+ // mockRows is the reference: the linked build the view is made of.
+ mockRows := func(links []unifiedresources.ResourceLink) []string {
+ resources, _ := mock.UnifiedResourceSnapshotWithLinks(links)
+ return cycleRows(resources)
+ }
+ wantNas, wantGuest := mockRows(cycle(oldest)), mockRows(cycle(newest))
+ if slices.Equal(wantNas, wantGuest) || len(wantNas) != 1 || len(wantGuest) != 1 ||
+ !strings.HasPrefix(wantNas[0], nasVMID+" ") || !strings.HasPrefix(wantGuest[0], guestID+" ") {
+ t.Fatalf("a creation time alone must move the cycle between the TrueNAS VM %s and the vSphere VM %s; got %v then %v",
+ nasVMID, guestID, wantNas, wantGuest)
+ }
+ version := mock.FixtureDataVersion()
+ assertLinksDifferOnlyInTime := func(before, after []unifiedresources.ResourceLink) {
+ t.Helper()
+ if len(before) != len(after) || len(before) != 3 {
+ t.Fatalf("link lists %+v and %+v need three links each", before, after)
+ }
+ differing := 0
+ for i := range before {
+ if before[i].ResourceA != after[i].ResourceA || before[i].ResourceB != after[i].ResourceB ||
+ before[i].PrimaryID != after[i].PrimaryID {
+ t.Fatalf("link %d changed its pair or primary: %+v -> %+v", i, before[i], after[i])
+ }
+ if !before[i].CreatedAt.Equal(after[i].CreatedAt) {
+ differing++
+ }
+ }
+ if differing != 1 {
+ t.Fatalf("%d links changed their creation time, want exactly one: %+v -> %+v", differing, before, after)
+ }
+ }
+ assertRows := func(label string, view monitorUnifiedStateView, want []string) {
+ t.Helper()
+ if got := cycleRows(view.resources); !slices.Equal(got, want) {
+ t.Fatalf("%s kept %v, want the row a linked mock build keeps: %v", label, got, want)
+ }
+ if got := mock.FixtureDataVersion(); got != version {
+ t.Fatalf("fixture data version moved from %d to %d; a creation time alone must refresh the view", version, got)
+ }
+ }
+
+ setNasLinkCreated(oldest)
+ first := m.currentUnifiedStateView()
+ assertRows("view with the TrueNAS VM's link oldest", first, wantNas)
+ firstLinks := m.resourceStoreManualLinks()
+
+ // The link list changes in its creation time alone. The structure reader
+ // answers from the cached view while the list is unchanged, so it must
+ // notice before anything else rebuilds the view.
+ setNasLinkCreated(newest)
+ assertLinksDifferOnlyInTime(firstLinks, m.resourceStoreManualLinks())
+ structure := m.currentStructureUnifiedStateView()
+ if structure.readState == first.readState {
+ t.Fatal("structure view served the view built before the creation time changed")
+ }
+ assertRows("structure view after the creation time changed", structure, wantGuest)
+ second := m.currentUnifiedStateView()
+ assertRows("view after the creation time changed", second, wantGuest)
+ if !second.freshness.After(first.freshness) {
+ t.Fatalf("view freshness %v did not advance past %v on a creation time change", second.freshness, first.freshness)
+ }
+
+ // And back: the data-version reader is held to the key on its own, with
+ // the structure reader no longer the one that rebuilt the view.
+ setNasLinkCreated(oldest)
+ assertLinksDifferOnlyInTime(m.resourceStoreManualLinks(), cycle(newest))
+ third := m.currentUnifiedStateView()
+ if third.readState == second.readState {
+ t.Fatal("view served the build made before the creation time changed back")
+ }
+ assertRows("view after the creation time changed back", third, wantNas)
+ if !third.freshness.After(second.freshness) {
+ t.Fatalf("view freshness %v did not advance past %v on a creation time change", third.freshness, second.freshness)
+ }
+ if again := m.currentUnifiedStateView(); again.readState != third.readState {
+ t.Fatal("expected the view to stay cached while neither the fixtures nor the links change")
+ }
+}
+
// Alert evaluation resolves a target per resource and windowed metric on every
// pass. A metric tick moves the fixture's data version, not what resources
// exist, so a lookup after a tick must be served from the cached view instead