Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Rebuild the mock view when only a link's creation time changes
rcourtman0582d2922aad
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 4d44926cc5..bf0471ad33 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -9480,3 +9480,14 @@ report, enrollment, lease or command path is touched(`TestSharedSystemAlertCorrelationForHostLazyListsNodesOnDemand` and`TestMockStructureConsumersDoNotRebuildTheViewAfterFixtureTicks` in`internal/monitoring/monitor_host_agents_test.go`).++### Mock-mode unified view cache follows a link's creation time++`internal/monitoring/monitor.go` changed only so `sameManualLinks`, the+comparison behind the mock-mode unified view's cache, also compares each+link's creation time (monitoring contract, "Mock-mode unified view applies+operator links"): among otherwise equal candidates a chain of links folds into+the primary of its earliest-created link, so lists that differ only in a time+can leave a different row standing. Agent registration, enrolment, install, update,+removal, report identity and continuity are unchanged, and real-mode+resolution still goes through the live registry.diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex 80d939cbf4..b198549729 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -3596,8 +3596,16 @@ current generation (`resourceStoreManualLinks`,broadcast when the resource store's next rebuild loads it, as in live mode; inmock mode that is normally the broadcast's own read-path refresh once thegeneration is older than `readPathRegistryFreshness`. The cached view is keyed-on the fixture data version and the link list, so a link change rebuilds it-without a fixture tick; an unchanged list keeps the shared build. Freshness,+on the fixture data version and the link list, each link by its resource pair,+its primary and its creation time (`sameManualLinks`, which the structure view+reuses), so a link change rebuilds it without a fixture tick; an unchanged+list keeps the shared build. The creation time is in the key because, among+otherwise equal candidates, a chain of links folds into the primary of its+earliest-created link (unified-resources contract, "Operator links reach+record-ingested resources", "Links apply as chains"): lists that differ only+in a time can leave a different row standing, and a key that ignored it kept+serving the row the old times chose while the live rebuild, which reloads the+links from the store, served the new one. Freshness,which the resources API keys its registry cache on, is assigned inpublication order under `mockUnifiedViewMu`: a view built from the samefixture freshness and links as the one it replaces keeps its freshness, and@@ -3610,7 +3618,13 @@ the mock-mode fence above): it stays a plain projection of current state forthe window before a rebuild in the new epoch, after which the registry appliesthe links again. `TestMockUnifiedViewAppliesOperatorManualLinks` in`internal/monitoring/monitor_host_agents_test.go` pins the broadcast, the-freshness advance and the cache, `TestFixtureGraphAppliesManualLinksAtTheRebuildsBoundary`+freshness advance and the cache,+`TestMockUnifiedViewFollowsALinkCreationTimeChange` in the same file pins the+creation time in that key (a Proxmox storage, a TrueNAS VM and a vSphere VM+linked in a cycle swap the surviving VM when only one link's time moves, with+the fixture data version held, through the structure view and the+data-version view alike, and match what a linked mock build keeps),+`TestFixtureGraphAppliesManualLinksAtTheRebuildsBoundary`and `TestUnifiedResourceSnapshotWithLinksLeavesTheSharedSnapshotUnlinked` in`internal/mock/platform_fixtures_test.go` pin the linked fixture build, and`TestMockUnifiedStateViewUsesCanonicalMockFixtureGraph` indiff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.mdindex 540d0ec4e4..8346ea112c 100644--- a/docs/release-control/v6/internal/subsystems/unified-resources.md+++ b/docs/release-control/v6/internal/subsystems/unified-resources.md@@ -7308,8 +7308,23 @@ of durable state through it.finished list instead dropped the check with the folded row.- A link reaches the view when the adapter's next rebuild loads it, the samepoint at which the live broadcast picks it up. The view's cache is keyed on- the fixture data version and on the links' resource pairs and primaries, so- a link change rebuilds it without waiting for a fixture tick. Freshness is+ the fixture data version and on each link's resource pair, primary and+ creation time, so a link change rebuilds it without waiting for a fixture+ tick. The creation time is part of the key because, among otherwise equal+ candidates, a chain folds into the primary of its earliest-created link+ (the "Links apply as chains" bullet of "Operator links reach+ record-ingested resources"): link lists that differ only in a creation+ time can leave a different row standing. In the storage, TrueNAS VM and+ vSphere VM cycle above both VMs outrank the storage, so the TrueNAS VM+ keeps its row while its link is older than the vSphere VM's and the+ vSphere VM keeps it once the TrueNAS VM's link is newer. A key that+ compared pairs and primaries alone kept serving the row the old times+ chose with the fixture data version unchanged, while the live rebuild,+ which reloads the links from the store, served the new one. The structure+ view that serves+ callers reading identity and topology only keys on the fixture structure+ revision and the same link comparison, so a creation time change rebuilds+ it too. Freshness isassigned in publication order: a view built from the same fixturefreshness and links as the one it replaces keeps its freshness, and anyother view gets a later one than every view published before it, even when@@ -7336,6 +7351,11 @@ the rebuild and freshness advance on a link change with the fixture dataversion held, the cached build afterwards, the same rows as a store-backedregistry seeded through resource ingest the way the resources API seeds, andno change rows reaching the store through the view.+`TestMockUnifiedViewFollowsALinkCreationTimeChange` in the same file pins the+creation time in the cache key: the cycle above, with only the TrueNAS VM's+link time moved and the fixture data version held, swaps the surviving VM in+the structure view and in the data-version view, each matching the row a+linked mock build keeps.`TestFixtureGraphAppliesManualLinksAtTheRebuildsBoundary` and`TestUnifiedResourceSnapshotWithLinksLeavesTheSharedSnapshotUnlinked` in`internal/mock/platform_fixtures_test.go` pin the rebuild's boundary (thediff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.goindex c81fe89b13..33505f2c4d 100644--- a/internal/monitoring/monitor.go+++ b/internal/monitoring/monitor.go@@ -5445,10 +5445,15 @@ func (m *Monitor) resourceStoreManualLinks() []unifiedresources.ResourceLink {}// sameManualLinks reports whether two link lists fold the same resources in-// the same order. Link metadata (reason, author, time) does not change a fold.+// the same order. The reason and author do not change a fold, but the creation+// time can: among otherwise equal candidates a chain of links folds into the+// primary of its earliest-created link (unifiedresources linkChainRootLocked),+// so lists that differ only in a creation time can leave a different row+// standing.func sameManualLinks(a, b []unifiedresources.ResourceLink) bool {return slices.EqualFunc(a, b, func(x, y unifiedresources.ResourceLink) bool {- return x.ResourceA == y.ResourceA && x.ResourceB == y.ResourceB && x.PrimaryID == y.PrimaryID+ return x.ResourceA == y.ResourceA && x.ResourceB == y.ResourceB && x.PrimaryID == y.PrimaryID &&+ x.CreatedAt.Equal(y.CreatedAt)})}diff --git a/internal/monitoring/monitor_host_agents_test.go b/internal/monitoring/monitor_host_agents_test.goindex cbe431bec8..b43522ba4a 100644--- a/internal/monitoring/monitor_host_agents_test.go+++ b/internal/monitoring/monitor_host_agents_test.go@@ -5992,6 +5992,171 @@ func TestMockUnifiedViewAppliesOperatorManualLinks(t *testing.T) {}}+// TestMockUnifiedViewFollowsALinkCreationTimeChange pins the creation time as+// part of the mock view's cache key. Among otherwise equal candidates a chain+// of links folds into the primary of its earliest-created link, so a link list+// that differs from the cached one only in a creation time can leave a+// different row standing: in a cycle of a Proxmox storage, a TrueNAS VM and a+// vSphere VM the storage never competes, and the TrueNAS VM keeps its row+// while its link is older than the vSphere VM's and gives it to the vSphere VM+// once it is newer. The cache compared resource pairs and primaries only, so+// with the fixture data version held it kept serving the row the old times+// chose while the live rebuild, which reloads the links from the store, served+// the new one. Both readers of the key are held to it: the structure view+// (cachedMockStructureView) and the data-version view+// (currentUnifiedStateView).+func TestMockUnifiedViewFollowsALinkCreationTimeChange(t *testing.T) {+ useMockEstate(t, 4, 5*time.Minute)++ store := unifiedresources.NewMemoryStore()+ m := &Monitor{+ state: models.NewState(),+ config: &config.Config{PVEPollingInterval: 2 * time.Minute},+ resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(store)),+ alertManager: alerts.NewManager(),+ }+ t.Cleanup(m.alertManager.Stop)++ soleSource := func(resource unifiedresources.Resource, source unifiedresources.DataSource) bool {+ return len(resource.Sources) == 1 && resource.Sources[0] == source+ }+ var storageID, nasVMID, guestID string+ for _, resource := range m.currentUnifiedStateView().resources {+ switch {+ case storageID == "" && resource.Type == unifiedresources.ResourceTypeStorage &&+ soleSource(resource, unifiedresources.SourceProxmox):+ storageID = resource.ID+ case nasVMID == "" && resource.Type == unifiedresources.ResourceTypeVM &&+ soleSource(resource, unifiedresources.SourceTrueNAS):+ nasVMID = resource.ID+ case guestID == "" && resource.Type == unifiedresources.ResourceTypeVM &&+ soleSource(resource, unifiedresources.SourceVMware):+ guestID = resource.ID+ }+ }+ if storageID == "" || nasVMID == "" || guestID == "" {+ t.Fatalf("fixture graph needs a Proxmox storage, a TrueNAS VM and a vSphere VM, got storage=%q nas=%q guest=%q",+ storageID, nasVMID, guestID)+ }++ created := time.Date(2026, time.October, 1, 9, 0, 0, 0, time.UTC)+ // The link whose primary is the TrueNAS VM is the last of the three, so+ // replacing it keeps the list's order and changes only its creation time.+ cycle := func(nasLinkCreated time.Time) []unifiedresources.ResourceLink {+ return []unifiedresources.ResourceLink{+ {ResourceA: nasVMID, ResourceB: storageID, PrimaryID: storageID, CreatedAt: created.Add(time.Minute)},+ {ResourceA: storageID, ResourceB: guestID, PrimaryID: guestID, CreatedAt: created.Add(2 * time.Minute)},+ {ResourceA: guestID, ResourceB: nasVMID, PrimaryID: nasVMID, CreatedAt: nasLinkCreated},+ }+ }+ oldest, newest := created, created.Add(3*time.Minute)+ setNasLinkCreated := func(at time.Time) {+ t.Helper()+ for _, link := range cycle(at) {+ if err := store.AddLink(link); err != nil {+ t.Fatalf("AddLink: %v", err)+ }+ }+ // Reads deliberately reuse a generation younger than two seconds, so+ // apply the rebuild boundary that loads the links instead of waiting.+ m.updateResourceStore(m.currentStateWithScope())+ }++ // cycleRows is what is left of the three members: ID and sources.+ cycleRows := func(resources []unifiedresources.Resource) []string {+ var rows []string+ for _, resource := range resources {+ if resource.ID != storageID && resource.ID != nasVMID && resource.ID != guestID {+ continue+ }+ sources := make([]string, 0, len(resource.Sources))+ for _, source := range resource.Sources {+ sources = append(sources, string(source))+ }+ slices.Sort(sources)+ rows = append(rows, fmt.Sprintf("%s %v", resource.ID, sources))+ }+ return rows+ }+ // mockRows is the reference: the linked build the view is made of.+ mockRows := func(links []unifiedresources.ResourceLink) []string {+ resources, _ := mock.UnifiedResourceSnapshotWithLinks(links)+ return cycleRows(resources)+ }+ wantNas, wantGuest := mockRows(cycle(oldest)), mockRows(cycle(newest))+ if slices.Equal(wantNas, wantGuest) || len(wantNas) != 1 || len(wantGuest) != 1 ||+ !strings.HasPrefix(wantNas[0], nasVMID+" ") || !strings.HasPrefix(wantGuest[0], guestID+" ") {+ t.Fatalf("a creation time alone must move the cycle between the TrueNAS VM %s and the vSphere VM %s; got %v then %v",+ nasVMID, guestID, wantNas, wantGuest)+ }+ version := mock.FixtureDataVersion()+ assertLinksDifferOnlyInTime := func(before, after []unifiedresources.ResourceLink) {+ t.Helper()+ if len(before) != len(after) || len(before) != 3 {+ t.Fatalf("link lists %+v and %+v need three links each", before, after)+ }+ differing := 0+ for i := range before {+ if before[i].ResourceA != after[i].ResourceA || before[i].ResourceB != after[i].ResourceB ||+ before[i].PrimaryID != after[i].PrimaryID {+ t.Fatalf("link %d changed its pair or primary: %+v -> %+v", i, before[i], after[i])+ }+ if !before[i].CreatedAt.Equal(after[i].CreatedAt) {+ differing+++ }+ }+ if differing != 1 {+ t.Fatalf("%d links changed their creation time, want exactly one: %+v -> %+v", differing, before, after)+ }+ }+ assertRows := func(label string, view monitorUnifiedStateView, want []string) {+ t.Helper()+ if got := cycleRows(view.resources); !slices.Equal(got, want) {+ t.Fatalf("%s kept %v, want the row a linked mock build keeps: %v", label, got, want)+ }+ if got := mock.FixtureDataVersion(); got != version {+ t.Fatalf("fixture data version moved from %d to %d; a creation time alone must refresh the view", version, got)+ }+ }++ setNasLinkCreated(oldest)+ first := m.currentUnifiedStateView()+ assertRows("view with the TrueNAS VM's link oldest", first, wantNas)+ firstLinks := m.resourceStoreManualLinks()++ // The link list changes in its creation time alone. The structure reader+ // answers from the cached view while the list is unchanged, so it must+ // notice before anything else rebuilds the view.+ setNasLinkCreated(newest)+ assertLinksDifferOnlyInTime(firstLinks, m.resourceStoreManualLinks())+ structure := m.currentStructureUnifiedStateView()+ if structure.readState == first.readState {+ t.Fatal("structure view served the view built before the creation time changed")+ }+ assertRows("structure view after the creation time changed", structure, wantGuest)+ second := m.currentUnifiedStateView()+ assertRows("view after the creation time changed", second, wantGuest)+ if !second.freshness.After(first.freshness) {+ t.Fatalf("view freshness %v did not advance past %v on a creation time change", second.freshness, first.freshness)+ }++ // And back: the data-version reader is held to the key on its own, with+ // the structure reader no longer the one that rebuilt the view.+ setNasLinkCreated(oldest)+ assertLinksDifferOnlyInTime(m.resourceStoreManualLinks(), cycle(newest))+ third := m.currentUnifiedStateView()+ if third.readState == second.readState {+ t.Fatal("view served the build made before the creation time changed back")+ }+ assertRows("view after the creation time changed back", third, wantNas)+ if !third.freshness.After(second.freshness) {+ t.Fatalf("view freshness %v did not advance past %v on a creation time change", third.freshness, second.freshness)+ }+ if again := m.currentUnifiedStateView(); again.readState != third.readState {+ t.Fatal("expected the view to stay cached while neither the fixtures nor the links change")+ }+}+// Alert evaluation resolves a target per resource and windowed metric on every// pass. A metric tick moves the fixture's data version, not what resources// exist, so a lookup after a tick must be served from the cached view instead