Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Fold chained manual links the same way in any link order

rcourtman95ff7f2804a8
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index ab25b4cbda..ef0ff1d3c9 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -3386,7 +3386,19 @@ truthfulness, not native thaw, containing-release or workload acceptance.
`TestMonitorAdapterSourceMergesUseConfiguredStaleThresholds` in
`internal/unifiedresources/monitor_adapter_read_state_test.go` and
`TestManualLinkToSupplementalGuestHoldsWithAndWithoutContinuity` in
- `internal/monitoring/issue1913_host_continuity_test.go`.
+ `internal/monitoring/issue1913_host_continuity_test.go`. The rebuild
+ holds the operator-link pass back until the snapshot and every record
+ source except availability are ingested, then folds each chain of links
+ once over the assembled estate and ingests availability after
+ (`deferManualLinks`, `applyDeferredManualLinks`; unified-resources
+ contract, "Links apply as chains"): a pass after each source folded
+ members as their sources arrived, and a later source's member could not
+ outrank one already folded, so the surviving identity depended on which
+ source reported it. The live supplemental refresh and the read-state
+ overlay still run a pass per call over the rows they find. Regression
+ coverage:
+ `TestMonitorRebuildJudgesLinkChainsOverTheAssembledEstate` in
+ `internal/unifiedresources/monitor_adapter_read_state_test.go`.
11. The TrueNAS provider projects pools with `Storage.Topology` fixed to
`pool` and the ZFS data vdev layout in `Storage.VDevLayout`. The
diff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.md
index eb6b8fd3a7..a1c3748f93 100644
--- a/docs/release-control/v6/internal/subsystems/unified-resources.md
+++ b/docs/release-control/v6/internal/subsystems/unified-resources.md
@@ -6921,19 +6921,142 @@ links" below).
answer for themselves until the agent reports and the link folds them, so
intent set on the pair under the agent's ID does not reach the live
member's references in that window. Holds are recomputed on every link
- pass; a hold whose primary a later link folds follows the fold, a saved
- row held under a saved primary follows that primary's hold, and a cycle of
- saved rows resolves each to itself. Exact reads are untouched: `Get` still
+ pass. A saved row answers to the row its link partner folds into, found
+ through the fold record when an earlier pass of the rebuild, or the
+ monitor a registry was seeded from, already folded the partner away; a
+ saved row held under a saved primary follows that primary's hold, and a
+ cycle of saved rows resolves each to itself. A saved row that several
+ links would fold answers to the best of the rows their partners fold into,
+ found after the folds (a partner a seeded listing carries inside a row the
+ links now fold away is ranked by the row it survives in), by the chain
+ precedence below with each row judged by its own identity and an observed
+ row ranked before a saved one; the first link the store lists no longer
+ decides. Exact reads are untouched: `Get` still
returns the saved row, so the saved agent's own drawer reads its own facet
bundle and history, while its operator-state section reads and writes the
guest's state.
-- Holds cover a link whose two members are both listed. A saved member
- whose partner an earlier link already folded into a third resource, such
- as an agent linked into a VM that is itself linked into another resource,
- is not held: the monitor's published registry folds the VM before
- continuity adds the agent, so the agent answers for itself until it
- reports. Live chains already depend on the order their links are stored
- in, which this does not change.
+- Links apply as chains (`manual_link_chains.go`). Links that share a
+ member name one identity, so a link pass folds every connected set of
+ observed members into one row, whatever order the store lists the links
+ in. Applied one at a time, a link naming a member an earlier link had
+ folded away found nothing: an agent linked into a VM that a second link
+ folds into storage joined the chain in one order and stayed its own row in
+ the other, and a saved agent whose partner the published registry had
+ already folded answered for itself. Standing the row that holds a folded
+ member in for it fixed neither, because that row then outranked the
+ link's own primary (a VM folded into a pool that had taken the agent
+ first) and hid the agent-into-guest direction once the guest was folded.
+ A link member is the resource the link names, judged by its own shape
+ (agent, hypervisor-managed guest) after a link folds it too: the holding
+ row takes a guest's Proxmox payload and would read as a guest itself.
+ Fold records keep both sides' shapes, so later passes and a registry
+ seeded from a listing judge a member alike. Each link keeps its pair rule:
+ the stated primary, except that an agent named primary over a guest folds
+ into the guest. A chain's root is picked from the members no link folds,
+ or from every member when the directions form a cycle. Rows compete, each
+ by its own identity: a member an earlier pass folded stands as the row
+ holding it, since a fold is not undone. When every candidate row is an
+ agent and the chain holds a guest, the rows holding guests compete
+ instead, including a row an earlier fold put a guest into, so an agent
+ never ends up holding a guest. Among candidates a hypervisor-managed guest
+ wins, then any non-agent resource over an agent, then the primary of the
+ earliest-created link, then the lowest canonical ID. A single link keeps
+ its effective primary, and so does a chain whose directions agree, unless
+ that primary is an agent and the chain holds a guest. The node and agent
+ rules judge each link before it joins a chain: a link between a Proxmox
+ node and an agent a newer split separates is dropped, and a link whose one
+ side a joined node-and-agent row already holds records that side's fold on
+ the row before any link resolves its members, so a second link naming the
+ folded side joins the chain whichever link the store lists first; the
+ fold record keeps both sides' agent shape, so a guest linked to the
+ folded node still outranks storage.
+- The monitor's rebuild holds the link pass back until the snapshot and
+ every record source except availability are in
+ (`deferManualLinks`, `applyDeferredManualLinks`), so its chains are judged
+ over the assembled estate; the availability batch that follows runs its
+ own pass, which finds the chains already folded. A link pass after each source folded members as
+ their sources arrived, and a later source's member could then outrank a
+ member already folded: a cycle through a Proxmox storage, a TrueNAS VM
+ and a vSphere VM kept the storage or the vSphere VM, where judging the
+ whole cycle keeps the TrueNAS VM. Availability checks still follow the
+ links, so a check linked to a folded resource projects onto its primary.
+ Record ingest into an existing registry still runs a link pass per call:
+ a live supplemental refresh that brings a member the last rebuild had not
+ seen judges it against the rows the rebuild left, until the next rebuild
+ judges the whole chain. A refresh can also recreate a folded member whose
+ source maps to a holder of another type; the recreated row competes as
+ itself, as it did in the rebuild, and folds back along the fold records
+ its chain carries, so identical records leave the pair set and every
+ folded ID as they were. The resources API's snapshot-seeded fallback,
+ used only while the monitor has published no unified listing, still
+ ingests the snapshot and each supplemental source with a pass per call,
+ and so does the mock-mode fixture build (`internal/mock`): a chain whose
+ answer depends on the order its members' sources arrive in can settle
+ differently there than in the monitor's rebuild.
+- Rows fold along the chain's links, deepest first, so each fold record
+ names a stored link's pair. The links form a spanning tree from the root
+ that follows the fold records the rows already carry before any other
+ link: a row a record pass recreated folds back through a link a record
+ names in the direction it folded, and a chain with several recreated
+ members folds back along its recorded ancestry, instead of reaching one
+ through a member another row holds, which would turn that member's record
+ round and leave its ID resolving nowhere. A cycle's closing link is
+ recorded in its own direction, so it counts as a recorded link only where
+ that direction matches, and a recreated row may fold back through it; the
+ pair set and every folded ID stay as they were. Links no record names
+ join once those are exhausted, breadth first from the root. The pass then records every other link of the
+ chain on the root (the link closing a cycle, or one whose sides an earlier
+ pass already joined, recorded with the link's effective direction though
+ it folded nothing in), so a report-merge that excludes every recorded pair
+ undoes every link of the chain. The pass records each fold before it
+ merges the folded row into the holder, as the own-sources record needs. A
+ pair is recorded once and keeps the
+ direction it was first recorded in, since a seeded listing can carry
+ records from an older generation of the links, and turning one would leave
+ the ID it folded in without a row. When a row must fold into another
+ and its own records name members it took in, its records are oriented
+ away from an anchor: the surviving row's own ID when they name it (a
+ recreated row that now roots the chain, or a link changed after the
+ listing was published), else the member the link names. Each record points
+ from the side nearer the anchor to the side farther (records between two
+ members equally far keep their direction), so the folded row's own ID
+ becomes a folded ID and keeps resolving, and the anchor is never recorded
+ as folded: a record naming the surviving row as folded, such as a cycle's
+ closing link, would leave report-merge no holder that no link folded to
+ infer the root from once that row is re-keyed. Hanging the records from
+ the member when the surviving row lies between that member and the folded
+ row would leave a member with no incoming fold. Each turned record lists
+ all of the row's sources, since the row no longer knows which member
+ brought which. A row whose records reach neither, such as a physical disk re-keyed
+ after it took the member in, stays standing rather than leave its own ID
+ resolving nowhere. A turned record keeps each side's own sources with the
+ side, and a record the pass adds for a link closing a cycle carries them
+ too, so report-merge (above), which picks links by member from these
+ records, can cut a reported member from every path of the chain.
+ Availability-owned members stay
+ unmerged, and saved members never join a chain: two live rows that only a
+ saved row links stay apart until it reports.
+ `TestManualLinkChainsFoldTheSameWayInAnyLinkOrder` (`registry_test.go`)
+ rebuilds twelve chains in every link order and checks both rebuilds, a live
+ refresh of each record source, the read-state overlay and a registry
+ seeded from it: one row of the expected type, every member resolving to
+ it, every link pair recorded and the same fold records in every order. On
+ main ten of the twelve leave a member standing, pick another root or miss
+ a link pair in some order; the other two (a record-sourced storage chain
+ and one recreated member) guard what already worked.
+ `TestManualLinkChainRefreshNeverLeavesAnAgentHoldingAGuest` refreshes in
+ a vSphere host the rebuild had not seen and checks the storage holding the
+ guest keeps the chain until the next rebuild keeps the guest.
+ `TestManualLinkToAJoinedNodeFoldsTheSameWayInAnyLinkOrder` links a
+ Proxmox node's own ID, which the inferred join with its agent consumed,
+ and a pool, in both orders: the joined row's fold of the node is recorded
+ before any link resolves its members, so the pool link joins the chain
+ whichever link the store lists first.
+ `TestSavedLinkMemberHoldsFollowTheChainInAnyLinkOrder` (`resolve_test.go`)
+ holds a saved agent under the pool its VM partner folds into, and under
+ its guest over a competing pool, in both orders, in the overlay and the
+ resources API's seeded registry; on main each case answers to the wrong
+ row in one order.
- A link merge keeps the TrueNAS and vSphere payloads the primary lacks, so
an agent chosen as primary over a TrueNAS system still carries the
system's facet, and the system's pools re-parent to the merged row.
diff --git a/internal/unifiedresources/manual_link_chains.go b/internal/unifiedresources/manual_link_chains.go
new file mode 100644
index 0000000000..2f9352b0f8
--- /dev/null
+++ b/internal/unifiedresources/manual_link_chains.go
@@ -0,0 +1,744 @@
+package unifiedresources
+
+import (
+ "maps"
+ "slices"
+ "strings"
+ "time"
+)
+
+// Operator links apply as chains. Links that share a member name one
+// identity, so a link pass folds every connected set of observed members into
+// one row, chosen by an explicit precedence (linkChainRoot), whatever order
+// the store lists the links in. Applying the links one at a time made the
+// result depend on that order: once an earlier link folded a member away, a
+// later link naming it found nothing and left its other side standing, and
+// standing in the row holding the folded member for it let that row outrank
+// the link's own primary and lose the agent-into-guest direction.
+//
+// Links are judged member by member. A member is the resource a link names:
+// its own row, or the row an earlier pass of the same rebuild, or the monitor
+// a registry was seeded from, already folded it into (linkMemberLocked).
+
+// linkMemberShape is what link precedence reads from a link member: whether
+// it is an agent, and whether a hypervisor platform accounts it as a guest.
+type linkMemberShape struct {
+ agent bool
+ guest bool
+}
+
+func linkShapeOfResource(resource *Resource) linkMemberShape {
+ return linkMemberShape{agent: resource.Type == ResourceTypeAgent, guest: hypervisorManagedGuest(resource)}
+}
+
+// linkFoldShapes keeps the shapes of a fold record's two members as the link
+// pass judged them, before either took in the other. A row that takes in a
+// guest also takes its Proxmox payload and would read as a guest itself, and
+// a folded member has no row of its own, so a later pass of the same rebuild
+// and a registry seeded from a listing read the shapes from the record.
+type linkFoldShapes struct {
+ recorded bool
+ holder linkMemberShape
+ folded linkMemberShape
+}
+
+func (s linkFoldShapes) turned() linkFoldShapes {
+ return linkFoldShapes{recorded: s.recorded, holder: s.folded, folded: s.holder}
+}
+
+// linkMember is one resource a stored link names, as the pass finds it.
+type linkMember struct {
+ id string
+ row *Resource
+ shape linkMemberShape
+ saved bool
+}
+
+// linkEdge is a stored link in its effective direction: other folds into,
+// or is held under, the primary's side.
+type linkEdge struct {
+ primary linkMember
+ other linkMember
+ createdAt time.Time
+}
+
+// live reports whether both members are observed. A link with a member that
+// exists only through saved-host continuity never merges anything.
+func (e linkEdge) live() bool {
+ return !e.primary.saved && !e.other.saved
+}
+
+// linkMemberLocked finds the resource a link names: its own listed row, which
+// answers for itself, or else the one row a link folded it into. A member
+// that is neither, or that two rows hold, takes no part in the pass.
+func (rr *ResourceRegistry) linkMemberLocked(id string) (linkMember, bool) {
+ row := rr.resources[id]
+ if row == nil {
+ holderID, ambiguous := rr.linkFoldHolderLocked(id)
+ if holderID == "" || ambiguous {
+ return linkMember{}, false
+ }
+ row = rr.resources[holderID]
+ if row == nil {
+ return linkMember{}, false
+ }
+ }
+ shape, recorded := recordedLinkMemberShape(row, CanonicalResourceID(id))
+ if !recorded && row.ID == id {
+ shape = linkShapeOfResource(row)
+ }
+ return linkMember{id: id, row: row, shape: shape, saved: row.continuityOnly}, true
+}
+
+func recordedLinkMemberShape(row *Resource, id string) (linkMemberShape, bool) {
+ for _, fold := range row.linkFolds {
+ if !fold.shapes.recorded {
+ continue
+ }
+ if fold.FoldedID == id {
+ return fold.shapes.folded, true
+ }
+ if fold.HolderID == id {
+ return fold.shapes.holder, true
+ }
+ }
+ return linkMemberShape{}, false
+}
+
+// manualLinkEdgesLocked resolves every stored link the pass can apply, in a
+// canonical order so nothing downstream depends on the store's.
+func (rr *ResourceRegistry) manualLinkEdgesLocked() []linkEdge {
+ rr.recordJoinedPairFoldsLocked()
+ edges := make([]linkEdge, 0, len(rr.links))
+ decisions := rr.operatorPairDecisionsLocked()
+ for _, link := range rr.links {
+ // A link joins only the pair its row names. Canonical-ID succession
+ // can leave a row whose endpoint re-key collided with the successor's
+ // own row while its primary moved on to the successor; honouring that
+ // primary merged the successor through a row naming a retired ID, which
+ // an unlink or report-merge of the merged pair leaves behind.
+ primaryID := link.PrimaryID
+ if primaryID != link.ResourceA && primaryID != link.ResourceB {
+ primaryID = link.ResourceA
+ }
+ otherID := link.ResourceB
+ if otherID == primaryID {
+ otherID = link.ResourceA
+ }
+ if otherID == primaryID {
+ continue
+ }
+ primary, primaryFound := rr.linkMemberLocked(primaryID)
+ other, otherFound := rr.linkMemberLocked(otherID)
+ if !primaryFound || !otherFound {
+ continue
+ }
+ // Availability checks retain their source-owned identity even when an
+ // operator links them to another resource. Correlation is represented
+ // by RelChecks plus the additive facet projection.
+ if isAvailabilityOwnedResource(*primary.row) || isAvailabilityOwnedResource(*other.row) {
+ continue
+ }
+ // A link between a Proxmox node and an agent loses to a newer split
+ // of the two read across their IDs: report-merge cannot delete a
+ // link naming the rows an earlier split left (nodeAgentSplit).
+ if rr.nodeAgentRowsSplit(decisions, primary.row, other.row) {
+ continue
+ }
+ // A manual link records operator intent to unify identities, but its
+ // direction must not change the semantic resource shape. In particular,
+ // an agent running inside a VM/LXC supplements that guest; it does not
+ // turn the guest into an agent resource with an agent metrics target.
+ // The members' own shapes decide, never the rows holding them.
+ if primary.shape.agent && other.shape.guest {
+ primary, other = other, primary
+ }
+ edges = append(edges, linkEdge{primary: primary, other: other, createdAt: link.CreatedAt})
+ }
+ slices.SortFunc(edges, func(a, b linkEdge) int {
+ if c := strings.Compare(a.primary.id, b.primary.id); c != 0 {
+ return c
+ }
+ return strings.Compare(a.other.id, b.other.id)
+ })
+ return edges
+}
+
+// recordJoinedPairFoldsLocked records, for every link with exactly one member
+// no row holds, the fold of that member on a joined Proxmox node+agent row
+// that holds the other member: a relink of a split node and its agent names
+// the two rows, and the declared link folds one of them into the other's ID
+// once the relink rejoins them. Recording the fold keeps pin succession from
+// re-keying the link onto the joined ID, which would let the split decide
+// again, and lets report-merge of the joined row name this pair. It runs
+// before any link is resolved to its members, because a recorded fold lets
+// the absent ID resolve and other links naming it join the chain, whatever
+// order the store lists them in; a recorded fold can in turn make another
+// link's one absent member resolve, so it repeats until none is left (a fold
+// the row already holds is not recorded again, so an ambiguous holder cannot
+// keep it going).
+func (rr *ResourceRegistry) recordJoinedPairFoldsLocked() {
+ for recorded := true; recorded; {
+ recorded = false
+ for _, link := range rr.links {
+ a, aFound := rr.linkMemberLocked(link.ResourceA)
+ b, bFound := rr.linkMemberLocked(link.ResourceB)
+ if aFound == bFound {
+ continue
+ }
+ present, absentID := a, link.ResourceB
+ if bFound {
+ present, absentID = b, link.ResourceA
+ }
+ if present.id != present.row.ID || holdsLinkFold(present.row, absentID) {
+ continue
+ }
+ source, ok := rr.joinedPairSideLocked(present.row, absentID)
+ if !ok {
+ continue
+ }
+ recordManualLinkFold(present.row, present.id, &Resource{Sources: []DataSource{source}}, absentID)
+ // Both sides of a joined node and agent are agent-type rows, so
+ // the absent side keeps the agent shape a guest linked to it is
+ // judged against.
+ stampManualLinkFoldShapes(present.row, present.id, absentID, present.shape, linkMemberShape{agent: true})
+ rr.indexLinkFoldsLocked(present.row)
+ recorded = true
+ }
+ }
+}
+
+// linkChainRootLocked picks the row a connected set of linked members folds
+// into and returns that row's own member. The candidates are the members no
+// link folds (each link's effective primary, never its other side), or every
+// member when the links' directions form a cycle. Rows compete, each by its
+// own identity: a member an earlier pass of the rebuild already folded stands
+// as the row holding it, because a fold cannot be undone, so the row's own
+// identity is what would survive. When every candidate row is an agent and the
+// chain holds a guest, the rows holding guests compete instead, including a
+// row an earlier fold put a guest into: an agent never ends up holding a
+// guest, as the pair rule already guarantees for one link. Among candidate
+// rows:
+// 1. an observed row before a saved one (holds only);
+// 2. a hypervisor-managed guest, the most specific identity a chain can
+// reach, as the pair rule already decides between an agent and its guest;
+// 3. any other resource before an agent, which supplements the resource it
+// runs on;
+// 4. the primary of the earliest-created link, so the operator's first
+// decision stands;
+// 5. the lowest canonical ID.
+//
+// A single link keeps its effective primary, and so does a chain whose
+// directions agree unless that primary is an agent and the chain holds a
+// guest, whichever order the links are stored in.
+func (rr *ResourceRegistry) linkChainRootLocked(members map[string]linkMember, edges []linkEdge) linkMember {
+ folded := make(map[string]bool, len(edges))
+ firstLinked := make(map[string]time.Time, len(edges))
+ for _, edge := range edges {
+ folded[edge.other.id] = true
+ if at, ok := firstLinked[edge.primary.id]; !ok || edge.createdAt.Before(at) {
+ firstLinked[edge.primary.id] = edge.createdAt
+ }
+ }
+ owners := make(map[string]linkMember, len(members))
+ ownerOf := func(member linkMember) linkMember {
+ owner, ok := owners[member.row.ID]
+ if !ok {
+ owner = rr.linkRowOwnerLocked(member.row, members)
+ owners[member.row.ID] = owner
+ }
+ return owner
+ }
+ collect := func(include func(linkMember) bool) []linkMember {
+ var rows []linkMember
+ seen := make(map[string]bool, len(members))
+ for _, member := range members {
+ owner := ownerOf(member)
+ if seen[owner.id] || !include(member) {
+ continue
+ }
+ seen[owner.id] = true
+ rows = append(rows, owner)
+ }
+ return rows
+ }
+ candidates := collect(func(member linkMember) bool { return !folded[member.id] })
+ if len(candidates) == 0 {
+ candidates = collect(func(linkMember) bool { return true })
+ }
+ if !slices.ContainsFunc(candidates, func(row linkMember) bool { return !row.shape.agent }) {
+ guests := collect(func(member linkMember) bool { return member.shape.guest || ownerOf(member).shape.guest })
+ if len(guests) > 0 {
+ candidates = guests
+ }
+ }
+ return bestLinkRoot(candidates, firstLinked)
+}
+
+// linkRowOwnerLocked returns the member a row is: its own canonical ID, with
+// the shape the row had before any link folded something into it.
+func (rr *ResourceRegistry) linkRowOwnerLocked(row *Resource, members map[string]linkMember) linkMember {
+ if member, ok := members[row.ID]; ok && member.row == row {
+ return member
+ }
+ shape, recorded := recordedLinkMemberShape(row, row.ID)
+ if !recorded {
+ shape = linkShapeOfResource(row)
+ }
+ return linkMember{id: row.ID, row: row, shape: shape, saved: row.continuityOnly}
+}
+
+func bestLinkRoot(candidates []linkMember, firstLinked map[string]time.Time) linkMember {
+ best := candidates[0]
+ for _, candidate := range candidates[1:] {
+ if linkRootBefore(candidate, best, firstLinked) {
+ best = candidate
+ }
+ }
+ return best
+}
+
+func linkRootBefore(a, b linkMember, firstLinked map[string]time.Time) bool {
+ if a.saved != b.saved {
+ return !a.saved
+ }
+ if a.shape.guest != b.shape.guest {
+ return a.shape.guest
+ }
+ if a.shape.agent != b.shape.agent {
+ return !a.shape.agent
+ }
+ aAt, aLinked := firstLinked[a.id]
+ bAt, bLinked := firstLinked[b.id]
+ if aLinked != bLinked {
+ return aLinked
+ }
+ if aLinked && !aAt.Equal(bAt) {
+ return aAt.Before(bAt)
+ }
+ return a.id < b.id
+}
+
+// foldLinkChainsLocked folds each connected set of observed link members into
+// its root row.
+func (rr *ResourceRegistry) foldLinkChainsLocked(edges []linkEdge, thresholds map[DataSource]time.Duration) {
+ // Rows join through links between their members; members a row already
+ // holds belong to it even when no stored link names them both any more.
+ sets := make(map[string]string)
+ var find func(string) string
+ find = func(id string) string {
+ parent, ok := sets[id]
+ if !ok || parent == id {
+ sets[id] = id
+ return id
+ }
+ root := find(parent)
+ sets[id] = root
+ return root
+ }
+ for _, edge := range edges {
+ if !edge.live() {
+ continue
+ }
+ a, b := find(edge.primary.row.ID), find(edge.other.row.ID)
+ if a == b {
+ continue
+ }
+ if b < a {
+ a, b = b, a
+ }
+ sets[b] = a
+ }
+ chains := make(map[string][]linkEdge)
+ for _, edge := range edges {
+ if edge.live() {
+ key := find(edge.primary.row.ID)
+ chains[key] = append(chains[key], edge)
+ }
+ }
+
+ for _, key := range slices.Sorted(maps.Keys(chains)) {
+ chain := chains[key]
+ members := make(map[string]linkMember, len(chain)+1)
+ for _, edge := range chain {
+ members[edge.primary.id] = edge.primary
+ members[edge.other.id] = edge.other
+ }
+ root := rr.linkChainRootLocked(members, chain)
+ rr.foldLinkChainLocked(members, chain, root.row, thresholds)
+ }
+}
+
+// foldLinkChainLocked merges every row of one chain into the root's row. Rows
+// fold along the chain's links, deepest first, so each fold record names a
+// stored link's own pair and lists what its folded side brought, including
+// what that side had already taken in. A row joins only through a link naming
+// its own ID, or a member whose fold records can be turned to hang from it
+// (rehangManualLinkFolds); a row neither reaches, such as one re-keyed after
+// it took the member in, stays standing, since folding it would leave its own
+// ID resolving nowhere.
+func (rr *ResourceRegistry) foldLinkChainLocked(members map[string]linkMember, edges []linkEdge, root *Resource, thresholds map[DataSource]time.Duration) {
+ type rowLink struct {
+ to *Resource
+ near string // the member on this row's side
+ far string // the member on the neighbouring row's side
+ recorded bool // a fold record on either row says this link folded far in
+ }
+ // Every fold record any row of the chain carries, in the direction it
+ // folded. A link counts as having taken its far member in only when a
+ // record says so in that direction: a cycle's closing link is recorded
+ // the other way round.
+ foldedThrough := make(map[[2]string]bool)
+ for _, edge := range edges {
+ for _, row := range []*Resource{edge.primary.row, edge.other.row} {
+ for _, fold := range row.linkFolds {
+ foldedThrough[[2]string{fold.HolderID, fold.FoldedID}] = true
+ }
+ }
+ }
+ adjacent := make(map[string][]rowLink)
+ join := func(from, to linkMember) {
+ if to.id != to.row.ID && !canHangManualLinkFolds(to.row, to.id) && !canHangManualLinkFolds(to.row, from.row.ID) {
+ return
+ }
+ adjacent[from.row.ID] = append(adjacent[from.row.ID], rowLink{
+ to: to.row, near: from.id, far: to.id,
+ recorded: foldedThrough[[2]string{from.id, to.id}],
+ })
+ }
+ for _, edge := range edges {
+ if edge.primary.row == edge.other.row {
+ continue
+ }
+ join(edge.primary, edge.other)
+ join(edge.other, edge.primary)
+ }
+ for rowID, links := range adjacent {
+ // Prefer the link a fold record says took the neighbour in, so a row
+ // a record pass recreated folds back through that same link; then
+ // links naming each row by its own ID, so records re-hang only when
+ // no such link joins the row.
+ slices.SortFunc(links, func(a, b rowLink) int {
+ if c := strings.Compare(a.to.ID, b.to.ID); c != 0 {
+ return c
+ }
+ if c := compareFalseFirst(!a.recorded, !b.recorded); c != 0 {
+ return c
+ }
+ if c := compareFalseFirst(a.far != a.to.ID, b.far != b.to.ID); c != 0 {
+ return c
+ }
+ if c := compareFalseFirst(a.near != rowID, b.near != rowID); c != 0 {
+ return c
+ }
+ if c := strings.Compare(a.far, b.far); c != 0 {
+ return c
+ }
+ return strings.Compare(a.near, b.near)
+ })
+ }
+
+ type step struct {
+ holder *Resource
+ link rowLink
+ }
+ // The spanning tree follows the recorded folds first, wherever they lead:
+ // a row reached through a member another row already holds would turn
+ // that member's record round and lose its ID (recreated interior rows of
+ // a chain fold back along the chain, not through the root's shortcuts).
+ // Links no record names join only once the recorded ones are exhausted,
+ // in breadth-first order, so a fresh rebuild keeps its breadth-first tree.
+ type deferredLink struct {
+ holder *Resource
+ link rowLink
+ }
+ order := []*Resource{root}
+ steps := map[string]step{root.ID: {}}
+ var deferred []deferredLink
+ visit := func(holder *Resource, link rowLink) {
+ steps[link.to.ID] = step{holder: holder, link: link}
+ order = append(order, link.to)
+ }
+ for scanned := 0; ; {
+ for ; scanned < len(order); scanned++ {
+ row := order[scanned]
+ for _, link := range adjacent[row.ID] {
+ if _, seen := steps[link.to.ID]; seen {
+ continue
+ }
+ if link.recorded {
+ visit(row, link)
+ } else {
+ deferred = append(deferred, deferredLink{holder: row, link: link})
+ }
+ }
+ }
+ joinedLater := false
+ for len(deferred) > 0 && !joinedLater {
+ next := deferred[0]
+ deferred = deferred[1:]
+ if _, seen := steps[next.link.to.ID]; !seen {
+ visit(next.holder, next.link)
+ joinedLater = true
+ }
+ }
+ if !joinedLater {
+ break
+ }
+ }
+ joined := make(map[string]bool, len(order))
+ for _, row := range order {
+ joined[row.ID] = true
+ if len(order) > 1 {
+ rr.recordLinkOwnPin(row.ID, row)
+ }
+ }
+ for i := len(order) - 1; i > 0; i-- {
+ folded := order[i]
+ s := steps[folded.ID]
+ if anchor := manualLinkFoldAnchor(folded, s.holder.ID, s.link.far); anchor != folded.ID {
+ rehangManualLinkFolds(folded, anchor)
+ }
+ // The fold records what each side is on its own, so it reads the
+ // holder before the folded row's sources join its list.
+ recordManualLinkFold(s.holder, s.link.near, folded, s.link.far)
+ rr.mergeResourceData(s.holder, folded, thresholds)
+ stampManualLinkFoldShapes(s.holder, s.link.near, s.link.far, members[s.link.near].shape, members[s.link.far].shape)
+ delete(rr.resources, folded.ID)
+ // The fold record names every ID along a chain of links, so the chain
+ // resolves to its root; the folded row's own index entries lapse on
+ // read now that it has left the registry.
+ rr.indexLinkFoldsLocked(s.holder)
+ rr.updateSourceMappings(folded.ID, s.holder.ID)
+ }
+ recordChainLinkFolds(root, slices.DeleteFunc(slices.Clone(edges), func(edge linkEdge) bool {
+ return !joined[edge.primary.row.ID] || !joined[edge.other.row.ID]
+ }))
+ rr.indexLinkFoldsLocked(root)
+}
+
+// recordChainLinkFolds records on a chain's root row every link of the chain
+// that no fold names yet: the link that closes a cycle, or one joining two
+// members an earlier pass already put in one row. The row holds both sides,
+// and report-merge undoes a merge by excluding each recorded pair, so a link
+// left unrecorded would keep the pair joined after a report. The link's folded
+// side is its effective other unless that is the row itself, and the record
+// lists what that side brought, as the fold that took it in recorded.
+func recordChainLinkFolds(root *Resource, edges []linkEdge) {
+ recorded := make(map[string]bool, len(root.linkFolds))
+ brought := make(map[string][]DataSource, len(root.linkFolds))
+ for _, fold := range root.linkFolds {
+ recorded[exclusionKey(fold.HolderID, fold.FoldedID)] = true
+ if _, ok := brought[fold.FoldedID]; !ok {
+ brought[fold.FoldedID] = fold.Sources
+ }
+ }
+ for _, edge := range edges {
+ key := exclusionKey(edge.primary.id, edge.other.id)
+ if recorded[key] {
+ continue
+ }
+ holder, folded := edge.primary, edge.other
+ if folded.id == root.ID {
+ holder, folded = folded, holder
+ }
+ recorded[key] = true
+ root.linkFolds = append(root.linkFolds, ManualLinkFold{
+ HolderID: holder.id,
+ FoldedID: folded.id,
+ Sources: cloneDataSourceSlice(brought[folded.id]),
+ HolderOwn: linkMemberOwnSources(root, holder.id),
+ FoldedOwn: linkMemberOwnSources(root, folded.id),
+ shapes: linkFoldShapes{recorded: true, holder: holder.shape, folded: folded.shape},
+ })
+ }
+}
+
+func compareFalseFirst(a, b bool) int {
+ switch {
+ case a == b:
+ return 0
+ case !a:
+ return -1
+ default:
+ return 1
+ }
+}
+
+// canHangManualLinkFolds reports whether a row's fold records join memberID to
+// the row's own ID, treating each record as a link between its two IDs. A row
+// can hang its records from a member only when they do; the row's own ID
+// counts trivially.
+func canHangManualLinkFolds(row *Resource, memberID string) bool {
+ start := CanonicalResourceID(memberID)
+ if start == row.ID {
+ return true
+ }
+ adjacent := make(map[string][]string, len(row.linkFolds))
+ for _, fold := range row.linkFolds {
+ adjacent[fold.HolderID] = append(adjacent[fold.HolderID], fold.FoldedID)
+ adjacent[fold.FoldedID] = append(adjacent[fold.FoldedID], fold.HolderID)
+ }
+ seen := map[string]bool{start: true}
+ for queue := []string{start}; len(queue) > 0; {
+ current := queue[0]
+ queue = queue[1:]
+ for _, next := range adjacent[current] {
+ if next == row.ID {
+ return true
+ }
+ if !seen[next] {
+ seen[next] = true
+ queue = append(queue, next)
+ }
+ }
+ }
+ return false
+}
+
+// manualLinkFoldAnchor picks the ID a row's fold records hang from when the
+// row folds into holderID through the link's member memberID: the holder's
+// own ID when the row's records name it (the holder is one of the row's folds,
+// as when a recreated row becomes the chain's root, or a link changed after
+// the listing was published), else the member. Hanging from the member would
+// turn the records between the two round the holder's ID and leave a member
+// the row held with no incoming fold.
+func manualLinkFoldAnchor(row *Resource, holderID, memberID string) string {
+ if canHangManualLinkFolds(row, holderID) {
+ return holderID
+ }
+ if CanonicalResourceID(memberID) == row.ID {
+ return row.ID
+ }
+ return memberID
+}
+
+// rehangManualLinkFolds turns a row's fold records round so they hang from
+// anchorID, an ID the row took in, before a chain folds the row into another
+// (manualLinkFoldAnchor). Every record turns to point away from the anchor,
+// from the side nearer it to the side farther, so the row's own ID becomes a
+// folded ID like any other and keeps resolving, and the anchor is never a
+// folded ID: a record that names it as folded (a cycle's closing link, or one
+// an older generation of the links made) would leave the surviving row
+// looking like a member some link folded, and the one holder no link folded,
+// which report-merge infers the root from, would be gone. Records between two
+// members equally far from the anchor keep their direction. This happens when
+// a member the row already holds is its only link to the chain's root: in a
+// live refresh, or in a registry seeded from a listing. The row no longer
+// knows which of its sources each member brought, so a turned record lists
+// all of them, which errs towards undoing the link on a report-merge.
+func rehangManualLinkFolds(row *Resource, anchorID string) {
+ anchor := CanonicalResourceID(anchorID)
+ if !canHangManualLinkFolds(row, anchor) {
+ return
+ }
+ adjacent := make(map[string][]string, len(row.linkFolds))
+ for _, fold := range row.linkFolds {
+ adjacent[fold.HolderID] = append(adjacent[fold.HolderID], fold.FoldedID)
+ adjacent[fold.FoldedID] = append(adjacent[fold.FoldedID], fold.HolderID)
+ }
+ depth := map[string]int{anchor: 0}
+ for queue := []string{anchor}; len(queue) > 0; {
+ current := queue[0]
+ queue = queue[1:]
+ for _, next := range adjacent[current] {
+ if _, seen := depth[next]; seen {
+ continue
+ }
+ depth[next] = depth[current] + 1
+ queue = append(queue, next)
+ }
+ }
+ folds := cloneManualLinkFolds(row.linkFolds)
+ for i, fold := range folds {
+ holderDepth, holderIn := depth[fold.HolderID]
+ foldedDepth, foldedIn := depth[fold.FoldedID]
+ if !holderIn || !foldedIn || foldedDepth >= holderDepth {
+ continue
+ }
+ folds[i] = ManualLinkFold{
+ HolderID: fold.FoldedID,
+ FoldedID: fold.HolderID,
+ Sources: cloneDataSourceSlice(row.Sources),
+ HolderOwn: fold.FoldedOwn,
+ FoldedOwn: fold.HolderOwn,
+ shapes: fold.shapes.turned(),
+ }
+ }
+ row.linkFolds = folds
+}
+
+// stampManualLinkFoldShapes records on a holder's fold record for the pair
+// (holderID, foldedID) the shapes the pass judged its members by.
+func stampManualLinkFoldShapes(holder *Resource, holderID, foldedID string, holderShape, foldedShape linkMemberShape) {
+ key := exclusionKey(holderID, foldedID)
+ for i, fold := range holder.linkFolds {
+ if exclusionKey(fold.HolderID, fold.FoldedID) != key {
+ continue
+ }
+ shapes := linkFoldShapes{recorded: true, holder: holderShape, folded: foldedShape}
+ if fold.HolderID != holderID {
+ shapes = shapes.turned()
+ }
+ holder.linkFolds[i].shapes = shapes
+ return
+ }
+}
+
+// holdSavedLinkMembersLocked holds each saved member a link would fold under
+// the row its partner folds into. A saved member several links would fold
+// answers to the best of those rows by the chain precedence, judged on each
+// row's root member; the earliest link to each counts.
+func (rr *ResourceRegistry) holdSavedLinkMembersLocked(edges []linkEdge) {
+ members := make(map[string]linkMember)
+ for _, edge := range edges {
+ members[edge.primary.id] = edge.primary
+ members[edge.other.id] = edge.other
+ }
+ targets := make(map[string]map[string]linkMember)
+ linked := make(map[string]map[string]time.Time)
+ for _, edge := range edges {
+ // A live member is never held, and a saved primary holds nothing: a
+ // live row never resolves to a saved one.
+ if !edge.other.saved {
+ continue
+ }
+ // The partner's row after the folds: the row the edge named may have
+ // folded into another since, and ranking it would answer for a row
+ // that has left the registry.
+ partner, ok := rr.linkMemberLocked(edge.primary.id)
+ if !ok {
+ continue
+ }
+ target := rr.linkRowOwnerLocked(partner.row, members)
+ if target.id == edge.other.id {
+ continue
+ }
+ savedID := edge.other.id
+ if targets[savedID] == nil {
+ targets[savedID] = make(map[string]linkMember)
+ linked[savedID] = make(map[string]time.Time)
+ }
+ targets[savedID][target.id] = target
+ if at, ok := linked[savedID][target.id]; !ok || edge.createdAt.Before(at) {
+ linked[savedID][target.id] = edge.createdAt
+ }
+ }
+ for savedID, candidates := range targets {
+ rr.holdLinkedResourceLocked(savedID, bestLinkRoot(slices.Collect(maps.Values(candidates)), linked[savedID]).id)
+ }
+}
+
+// holdLinkedResourceLocked keeps a saved link member out of the fold. A saved
+// enrollment is not an observation: folded into a live resource it would lend
+// that resource its offline verdict and old payload, and a live resource
+// folded into it would hide live telemetry behind it. The link still names
+// one identity, so the saved member answers to the row the fold would have
+// taken it into, as a folded member does.
+func (rr *ResourceRegistry) holdLinkedResourceLocked(savedID, targetID string) {
+ if rr.linkHolds == nil {
+ rr.linkHolds = make(map[string]string)
+ }
+ rr.linkHolds[savedID] = targetID
+ rr.canonicalIdentityIndex = nil
+}
diff --git a/internal/unifiedresources/manual_link_folds.go b/internal/unifiedresources/manual_link_folds.go
index 916c8883b2..d99be553f5 100644
--- a/internal/unifiedresources/manual_link_folds.go
+++ b/internal/unifiedresources/manual_link_folds.go
@@ -17,6 +17,9 @@ type ManualLinkFold struct {
Sources []DataSource
HolderOwn []DataSource
FoldedOwn []DataSource
+ // shapes keeps the two members' link shapes for later link passes
+ // (manual_link_chains.go); ManualLinkFolds leaves them out.
+ shapes linkFoldShapes
}
// recordManualLinkFold notes on the link's merged resource that it took in
@@ -80,7 +83,11 @@ func (rr *ResourceRegistry) ManualLinkFolds(resourceID string) []ManualLinkFold
if resource == nil {
return nil
}
- return cloneManualLinkFolds(resource.linkFolds)
+ folds := cloneManualLinkFolds(resource.linkFolds)
+ for i := range folds {
+ folds[i].shapes = linkFoldShapes{}
+ }
+ return folds
}
func cloneManualLinkFolds(in []ManualLinkFold) []ManualLinkFold {
diff --git a/internal/unifiedresources/monitor_adapter.go b/internal/unifiedresources/monitor_adapter.go
index fa996645e2..c584ed2e13 100644
--- a/internal/unifiedresources/monitor_adapter.go
+++ b/internal/unifiedresources/monitor_adapter.go
@@ -489,6 +489,13 @@ func (a *MonitorAdapter) replaceRegistryLocked(snapshot models.StateSnapshot, re
rebuilt := NewRegistry(registry.store)
staleThresholds := a.currentStaleThresholds()
+ // Operator links apply once the snapshot and every record source are in,
+ // so a chain is judged over the whole estate rather than folded as its
+ // members' sources arrive: a fold cannot be undone when a later source
+ // brings a member that outranks the earlier choice. Availability checks
+ // still come after the links, so a check linked to a folded resource
+ // projects onto its primary.
+ rebuilt.deferManualLinks = true
rebuilt.IngestSnapshotWithStaleThresholds(snapshot, staleThresholds)
sources := make([]DataSource, 0, len(recordsBySource))
for source := range recordsBySource {
@@ -504,12 +511,16 @@ func (a *MonitorAdapter) replaceRegistryLocked(snapshot models.StateSnapshot, re
return string(sources[i]) < string(sources[j])
})
for _, source := range sources {
+ if source == SourceAvailability {
+ rebuilt.applyDeferredManualLinks(staleThresholds)
+ }
records := recordsBySource[source]
if len(records) == 0 || strings.TrimSpace(string(source)) == "" {
continue
}
rebuilt.IngestRecordsWithStaleThresholds(source, records, staleThresholds)
}
+ rebuilt.applyDeferredManualLinks(staleThresholds)
// IngestSnapshot runs its stale pass before the record sources above are
// ingested, so record-sourced resources would otherwise keep their
// ingest-time "online" stamp regardless of how old their data is.
diff --git a/internal/unifiedresources/monitor_adapter_read_state_test.go b/internal/unifiedresources/monitor_adapter_read_state_test.go
index 0ab82b5b43..477db0bb98 100644
--- a/internal/unifiedresources/monitor_adapter_read_state_test.go
+++ b/internal/unifiedresources/monitor_adapter_read_state_test.go
@@ -1291,3 +1291,40 @@ func TestMonitorAdapterProjectionSnapshotKeepsReplacedGeneration(t *testing.T) {
t.Fatal("separate adapters shared a tenant projection")
}
}
+
+// The monitor's rebuild folds a chain of operator links once, over the
+// snapshot and every record source, instead of after each source. A Proxmox
+// storage, a TrueNAS VM and a vSphere VM linked in a cycle keep the TrueNAS
+// VM, whose link is the oldest; folding as the sources arrived (TrueNAS
+// before vSphere) had already put the TrueNAS VM inside the storage when the
+// vSphere VM's link closed the cycle, and kept the storage or the vSphere VM.
+func TestMonitorRebuildJudgesLinkChainsOverTheAssembledEstate(t *testing.T) {
+ estate := newChainLinkEstate(t)
+ created := time.Date(2026, 10, 1, 9, 0, 0, 0, time.UTC)
+ nasVM, storage, guest := estate.ids["nas-vm"], estate.ids["pve-storage"], estate.ids["vsphere-vm"]
+ store := NewMemoryStore()
+ for _, link := range []ResourceLink{
+ {ResourceA: nasVM, ResourceB: storage, PrimaryID: storage, CreatedAt: created.Add(time.Minute)},
+ {ResourceA: storage, ResourceB: guest, PrimaryID: guest, CreatedAt: created.Add(2 * time.Minute)},
+ {ResourceA: guest, ResourceB: nasVM, PrimaryID: nasVM, CreatedAt: created},
+ } {
+ if err := store.AddLink(link); err != nil {
+ t.Fatalf("add link: %v", err)
+ }
+ }
+
+ adapter := NewMonitorAdapter(NewRegistry(store))
+ adapter.PopulateSnapshotAndSupplemental(estate.snapshot, estate.records)
+ assertChainRoot(t, "rebuild", adapter.currentRegistry(), nasVM, ResourceTypeVM, storage, guest)
+
+ // A registry that ingests the same sources one call at a time, as a live
+ // refresh does, judges the chain at each call and can settle elsewhere:
+ // the rebuild's whole-estate pass is what the monitor publishes.
+ staged := NewRegistry(store)
+ staged.IngestSnapshot(estate.snapshot)
+ staged.IngestRecords(SourceTrueNAS, estate.records[SourceTrueNAS])
+ staged.IngestRecords(SourceVMware, estate.records[SourceVMware])
+ if _, listed := staged.Get(nasVM); listed {
+ t.Fatalf("staged ingest kept the TrueNAS VM, so this fixture no longer separates the rebuild from per-call passes")
+ }
+}
diff --git a/internal/unifiedresources/registry.go b/internal/unifiedresources/registry.go
index 1e1aa35333..32ec78f261 100644
--- a/internal/unifiedresources/registry.go
+++ b/internal/unifiedresources/registry.go
@@ -115,10 +115,16 @@ type ResourceRegistry struct {
linkOwnPins map[string]*ResourceIdentityPin
// linkHolds maps a resource that exists only through saved-host
// continuity, and that an operator link would otherwise fold away, to
- // the link's primary (holdLinkedResourceLocked). Both rows stay listed
- // with their own telemetry; references to the held row resolve to the
- // primary. Every link pass recomputes it.
+ // the member whose row the fold would have taken it into
+ // (holdLinkedResourceLocked, holdSavedLinkMembersLocked). Both rows stay
+ // listed with their own telemetry; references to the held row resolve to
+ // that row. Every link pass recomputes it.
linkHolds map[string]string
+ // deferManualLinks holds the link pass back while a monitor rebuild
+ // ingests its snapshot and record sources (MonitorAdapter.replaceRegistryLocked),
+ // so applyDeferredManualLinks judges every chain once, over the
+ // assembled estate, instead of folding members as their sources arrive.
+ deferManualLinks bool
// supersededIndex maps record-declared retired canonical IDs to the live
// resource that superseded them, so references persisted under a retired
// ID (availability links, API reads) keep resolving. An empty value marks
@@ -652,7 +658,9 @@ func (rr *ResourceRegistry) ingestSnapshot(snapshot models.StateSnapshot, thresh
rr.mu.Lock()
rr.pbsBackups = clonePBSBackups(snapshot.PBSBackups)
- rr.applyManualLinks(thresholds)
+ if !rr.deferManualLinks {
+ rr.applyManualLinks(thresholds)
+ }
rr.refreshStorageConsumersLocked()
rr.refreshPBSRollupsLocked()
rr.refreshStoragePostureLocked()
@@ -727,7 +735,9 @@ func (rr *ResourceRegistry) ingestRecords(source DataSource, records []IngestRec
// and old agent payload), but the link still names their identity: the
// pass holds them beside the primary and resolves references to them
// there.
- rr.applyManualLinks(thresholds)
+ if !rr.deferManualLinks {
+ rr.applyManualLinks(thresholds)
+ }
rr.refreshStorageConsumersLocked()
rr.refreshPBSRollupsLocked()
rr.refreshStoragePostureLocked()
@@ -4989,6 +4999,27 @@ func mergeVMwareData(existing *VMwareData, incoming *VMwareData) *VMwareData {
return &merged
}
+// applyDeferredManualLinks runs the link pass a rebuild held back while it
+// ingested the snapshot and record sources (deferManualLinks), then refreshes
+// what folding changes, as record ingest does after its own pass.
+func (rr *ResourceRegistry) applyDeferredManualLinks(thresholds map[DataSource]time.Duration) {
+ rr.mu.Lock()
+ defer rr.mu.Unlock()
+ if !rr.deferManualLinks {
+ return
+ }
+ rr.deferManualLinks = false
+ rr.applyManualLinks(thresholds)
+ rr.refreshStorageConsumersLocked()
+ rr.refreshPBSRollupsLocked()
+ rr.refreshStoragePostureLocked()
+ rr.refreshIncidentRollupsLocked()
+ rr.buildChildCounts()
+ rr.refreshProxmoxSensorSetupLocked()
+ rr.refreshCanonicalIdentitiesLocked()
+ rr.invalidateViewsLocked()
+}
+
func (rr *ResourceRegistry) applyManualLinks(thresholds map[DataSource]time.Duration) {
if rr.linkHolds != nil {
rr.linkHolds = nil
@@ -4997,88 +5028,12 @@ func (rr *ResourceRegistry) applyManualLinks(thresholds map[DataSource]time.Dura
if len(rr.links) == 0 {
return
}
- for _, link := range rr.links {
- // A link joins only the pair its row names. Canonical-ID succession
- // can leave a row whose endpoint re-key collided with the successor's
- // own row while its primary moved on to the successor; honouring that
- // primary merged the successor through a row naming a retired ID, which
- // an unlink or report-merge of the merged pair leaves behind.
- primaryID := link.PrimaryID
- if primaryID != link.ResourceA && primaryID != link.ResourceB {
- primaryID = link.ResourceA
- }
- primary := rr.resources[primaryID]
- if primary == nil {
- primary = rr.resources[link.ResourceA]
- primaryID = link.ResourceA
- }
- if primary == nil {
- primary = rr.resources[link.ResourceB]
- primaryID = link.ResourceB
- }
- if primary == nil {
- continue
- }
-
- otherID := link.ResourceB
- if otherID == primaryID {
- otherID = link.ResourceA
- }
- other := rr.resources[otherID]
- if other == nil && otherID != primaryID {
- if source, ok := rr.joinedPairSideLocked(primary, otherID); ok {
- // A relink of a split node and its agent names the two rows,
- // and the declared link folds one of them into the other's
- // ID once the relink rejoins them. Record the fold so pin
- // succession keeps the link rather than re-keying it onto
- // the joined ID, which would let the split decide again,
- // and so report-merge of the joined row names this pair.
- recordManualLinkFold(primary, primaryID, &Resource{Sources: []DataSource{source}}, otherID)
- rr.indexLinkFoldsLocked(primary)
- }
- }
- if other == nil || otherID == primaryID {
- continue
- }
- // Availability checks retain their source-owned identity even when an
- // operator links them to another resource. Correlation is represented
- // by RelChecks plus the additive facet projection.
- if isAvailabilityOwnedResource(*primary) || isAvailabilityOwnedResource(*other) {
- continue
- }
- // A link between a Proxmox node and an agent loses to a newer split
- // of the two read across their IDs: report-merge cannot delete a
- // link naming the rows an earlier split left (nodeAgentSplit).
- if rr.nodeAgentRowsSplit(rr.operatorPairDecisionsLocked(), primary, other) {
- continue
- }
-
- // A manual link records operator intent to unify identities, but its
- // direction must not change the semantic resource shape. In particular,
- // an agent running inside a VM/LXC supplements that guest; it does not
- // turn the guest into an agent resource with an agent metrics target.
- if primary.Type == ResourceTypeAgent && hypervisorManagedGuest(other) {
- primary, other = other, primary
- primaryID, otherID = otherID, primaryID
- }
- if primary.continuityOnly || other.continuityOnly {
- rr.holdLinkedResourceLocked(primaryID, otherID, other)
- continue
- }
-
- rr.recordLinkOwnPin(primaryID, primary)
- rr.recordLinkOwnPin(otherID, other)
- // The fold records what each side is on its own, so it reads the
- // primary before the other side's sources join its list.
- recordManualLinkFold(primary, primaryID, other, otherID)
- rr.mergeResourceData(primary, other, thresholds)
- delete(rr.resources, otherID)
- // The fold record names every ID along a chain of links, so the chain
- // resolves to its last primary; the folded row's own index entries
- // lapse on read now that it has left the registry.
- rr.indexLinkFoldsLocked(primary)
- rr.updateSourceMappings(otherID, primaryID)
- }
+ // Links apply as chains (manual_link_chains.go): each connected set of
+ // observed members folds into one root, and saved members are held under
+ // the row their partners fold into, whatever order the links are stored in.
+ edges := rr.manualLinkEdgesLocked()
+ rr.foldLinkChainsLocked(edges, thresholds)
+ rr.holdSavedLinkMembersLocked(edges)
}
// recordLinkOwnPin captures the identity pin a manual-link side derives from
@@ -5099,29 +5054,6 @@ func (rr *ResourceRegistry) recordLinkOwnPin(id string, resource *Resource) {
rr.linkOwnPins[id] = nil
}
-// holdLinkedResourceLocked keeps a link whose member exists only through
-// saved-host continuity out of the fold. A saved enrollment is not an
-// observation: folded into a live resource it would lend that resource its
-// offline verdict and old payload, and a live resource folded into it would
-// hide live telemetry behind it. The link still names one identity, so a
-// saved member the fold would have taken in answers to the primary,
-// as a folded one does. A live member keeps its own references even when the
-// saved member is the primary: a live row never resolves to a saved one. The
-// first link to claim a member holds it, as the first fold takes it in.
-func (rr *ResourceRegistry) holdLinkedResourceLocked(primaryID, otherID string, other *Resource) {
- if !other.continuityOnly {
- return
- }
- if _, held := rr.linkHolds[otherID]; held {
- return
- }
- if rr.linkHolds == nil {
- rr.linkHolds = make(map[string]string)
- }
- rr.linkHolds[otherID] = primaryID
- rr.canonicalIdentityIndex = nil
-}
-
// linkHoldPrimaryLocked returns the row a held link member's references
// resolve to: its primary, followed through a later fold of the primary into
// another resource and through further holds (a saved primary held under
diff --git a/internal/unifiedresources/registry_test.go b/internal/unifiedresources/registry_test.go
index 599346178b..d3200a099c 100644
--- a/internal/unifiedresources/registry_test.go
+++ b/internal/unifiedresources/registry_test.go
@@ -4,6 +4,7 @@ import (
"encoding/json"
"fmt"
"reflect"
+ "slices"
"strings"
"testing"
"time"
@@ -8533,3 +8534,662 @@ func TestSuccessionBatchOrderKeepsLinkedRemediationLock(t *testing.T) {
})
}
}
+
+// chainLinkEstate arrives the way the monitor ingests it: two agents, a
+// Proxmox VM and its storage in the snapshot, then two TrueNAS pools and a
+// TrueNAS VM, then a vSphere VM and a vSphere datastore, each record source
+// in its own pass.
+type chainLinkEstate struct {
+ snapshot models.StateSnapshot
+ records map[DataSource][]IngestRecord
+ // saved is an unlinked host that only saved-host continuity lists.
+ saved models.Host
+ ids map[string]string
+}
+
+func newChainLinkEstate(t *testing.T) chainLinkEstate {
+ t.Helper()
+ now := time.Now().UTC()
+ pool := func(name string) IngestRecord {
+ return IngestRecord{
+ SourceID: "system:tn-1:pool:" + name,
+ Resource: Resource{
+ Type: ResourceTypeStorage, Name: name, Status: StatusOnline, LastSeen: now,
+ Storage: &StorageMeta{Type: "zfs-pool", Platform: "truenas", Topology: "pool"},
+ },
+ }
+ }
+ estate := chainLinkEstate{
+ snapshot: models.StateSnapshot{
+ LastUpdate: now,
+ VMs: []models.VM{
+ {ID: "lab:pve1:101", VMID: 101, Name: "web", Node: "pve1", Instance: "lab", Status: "running", Type: "qemu", LastSeen: now},
+ {ID: "lab:pve1:111", VMID: 111, Name: "app-r", Node: "pve1", Instance: "lab", Status: "running", Type: "qemu", LastSeen: now},
+ {ID: "lab:pve1:112", VMID: 112, Name: "app-u", Node: "pve1", Instance: "lab", Status: "running", Type: "qemu", LastSeen: now},
+ {ID: "lab:pve1:113", VMID: 113, Name: "app-x", Node: "pve1", Instance: "lab", Status: "running", Type: "qemu", LastSeen: now},
+ {ID: "lab:pve1:114", VMID: 114, Name: "app-y", Node: "pve1", Instance: "lab", Status: "running", Type: "qemu", LastSeen: now},
+ },
+ Storage: []models.Storage{{
+ ID: "lab-pve1-local-zfs", Name: "local-zfs", Node: "pve1", Instance: "lab", Type: "zfspool", Status: "available",
+ Total: 1000, Used: 400, Free: 600, Usage: 40, Enabled: true, Active: true, LastSeen: now,
+ }},
+ Hosts: []models.Host{
+ {ID: "host-app", Hostname: "app-agent", MachineID: "0123456789abcdef", Status: "online", LastSeen: now},
+ {ID: "host-ops", Hostname: "ops-agent", MachineID: "fedcba9876543210", Status: "online", LastSeen: now},
+ },
+ },
+ records: map[DataSource][]IngestRecord{
+ SourceTrueNAS: {
+ pool("tank"), pool("vault"),
+ {
+ SourceID: "system:tn-1:vm:nas-vm",
+ Resource: Resource{Type: ResourceTypeVM, Name: "nas-vm", Status: StatusOnline, LastSeen: now},
+ },
+ },
+ SourceVMware: {
+ {
+ SourceID: "vc-1:vm:vm-42",
+ Resource: Resource{
+ Type: ResourceTypeVM, Technology: "vmware", Name: "app-guest", Status: StatusOnline, LastSeen: now,
+ VMware: &VMwareData{ConnectionID: "vc-1", ManagedObjectID: "vm-42", EntityType: "vm"},
+ },
+ Identity: ResourceIdentity{Hostnames: []string{"app-guest"}},
+ },
+ {
+ SourceID: "vc-1:datastore:datastore-7",
+ Resource: Resource{
+ Type: ResourceTypeStorage, Technology: "vmware", Name: "ds-7", Status: StatusOnline, LastSeen: now,
+ VMware: &VMwareData{ConnectionID: "vc-1", ManagedObjectID: "datastore-7", EntityType: "datastore"},
+ },
+ },
+ },
+ },
+ saved: models.Host{ID: "host-saved", Hostname: "saved-agent", MachineID: "00112233445566aa", Status: "offline", LastSeen: now.Add(-time.Hour)},
+ ids: map[string]string{},
+ }
+ unlinked := NewMonitorAdapter(NewRegistry(nil))
+ unlinked.PopulateSnapshotAndSupplemental(estate.snapshot, estate.records)
+ roles := map[string]string{
+ "app-agent": "agent", "ops-agent": "ops-agent", "web": "pve-vm", "local-zfs": "pve-storage",
+ "tank": "tank", "vault": "vault", "nas-vm": "nas-vm", "app-guest": "vsphere-vm", "ds-7": "datastore",
+ "app-r": "vm-r", "app-u": "vm-u", "app-x": "vm-x", "app-y": "vm-y",
+ }
+ for _, resource := range unlinked.GetAll() {
+ if role, ok := roles[resource.Name]; ok {
+ estate.ids[role] = resource.ID
+ }
+ }
+ if len(estate.ids) != len(roles) || len(unlinked.GetAll()) != len(roles) {
+ t.Fatalf("unlinked estate = %+v, want one row per role %v", unlinked.GetAll(), roles)
+ }
+ return estate
+}
+
+// linkOrders lists every order of n links.
+func linkOrders(n int) [][]int {
+ if n == 0 {
+ return [][]int{{}}
+ }
+ var orders [][]int
+ for _, rest := range linkOrders(n - 1) {
+ for i := 0; i <= len(rest); i++ {
+ order := append(append(append([]int{}, rest[:i]...), n-1), rest[i:]...)
+ orders = append(orders, order)
+ }
+ }
+ return orders
+}
+
+// Links that share a member name one identity, so a chain folds into one row
+// chosen by precedence, never by the order the store lists the links in or
+// the order its members' sources arrive. Each estate is rebuilt in every link
+// order and checked after each of two rebuilds (identity pins persist on the
+// first), after a live refresh of every record source, in the read state that overlays a saved host, and in
+// a registry seeded from that read state, as the resources API seeds when the
+// listing already carries every source: each view lists the same row,
+// resolves every member to it and records every link's pair, with the same
+// fold records in every order, refreshes included. Applied one link at a
+// time, a link naming a member an earlier link had folded away found nothing,
+// so one order folded the chain and another left a member standing.
+func TestManualLinkChainsFoldTheSameWayInAnyLinkOrder(t *testing.T) {
+ estate := newChainLinkEstate(t)
+ created := time.Date(2026, 10, 1, 9, 0, 0, 0, time.UTC)
+ link := func(a, b, primary string, minute int) ResourceLink {
+ return ResourceLink{
+ ResourceA: estate.ids[a], ResourceB: estate.ids[b], PrimaryID: estate.ids[primary],
+ CreatedAt: created.Add(time.Duration(minute) * time.Minute),
+ }
+ }
+ // The cycle case below needs its two tie-broken VMs in canonical-ID order.
+ vmB, vmC := "vm-x", "vm-y"
+ if estate.ids[vmC] > estate.ids[vmB] {
+ vmB, vmC = vmC, vmB
+ }
+ for _, tc := range []struct {
+ name string
+ links []ResourceLink
+ root string
+ rootType ResourceType
+ }{
+ {
+ // The agent linked from its own page still folds into its guest, and
+ // the guest's own link takes both into the storage it names, all in
+ // the snapshot pass.
+ name: "agent primary over its guest, guest into storage",
+ links: []ResourceLink{link("agent", "pve-vm", "agent", 0), link("pve-vm", "pve-storage", "pve-storage", 1)},
+ root: "pve-storage",
+ rootType: ResourceTypeStorage,
+ },
+ {
+ // The same chain with record-sourced storage: the rebuild judges
+ // links once every source is in.
+ name: "agent primary over its guest, guest into record-sourced storage",
+ links: []ResourceLink{link("agent", "pve-vm", "agent", 0), link("pve-vm", "tank", "tank", 1)},
+ root: "tank",
+ rootType: ResourceTypeStorage,
+ },
+ {
+ // Both links name a primary over the agent, and the guest outranks
+ // the storage, though the pools' source sorts before the VM's.
+ name: "guest outranks storage as competing primary",
+ links: []ResourceLink{link("agent", "tank", "tank", 0), link("agent", "vsphere-vm", "vsphere-vm", 1)},
+ root: "vsphere-vm",
+ rootType: ResourceTypeVM,
+ },
+ {
+ name: "competing storage primaries keep the earliest link's",
+ links: []ResourceLink{link("agent", "tank", "tank", 0), link("pve-vm", "vault", "vault", 1), link("agent", "pve-vm", "agent", 2)},
+ root: "tank",
+ rootType: ResourceTypeStorage,
+ },
+ {
+ name: "competing storage primaries follow creation time, not names",
+ links: []ResourceLink{link("agent", "tank", "tank", 2), link("pve-vm", "vault", "vault", 1), link("agent", "pve-vm", "agent", 0)},
+ root: "vault",
+ rootType: ResourceTypeStorage,
+ },
+ {
+ // Every candidate is an agent, so the guest takes its place.
+ name: "an agent never holds a guest",
+ links: []ResourceLink{link("pve-vm", "tank", "tank", 0), link("tank", "ops-agent", "ops-agent", 1)},
+ root: "pve-vm",
+ rootType: ResourceTypeVM,
+ },
+ {
+ name: "a cycle folds into its best member",
+ links: []ResourceLink{link("agent", "tank", "tank", 0), link("tank", "vault", "vault", 1), link("vault", "agent", "agent", 2)},
+ root: "tank",
+ rootType: ResourceTypeStorage,
+ },
+ {
+ // A cycle across the snapshot and two record sources. Folding as the
+ // sources arrived put the TrueNAS VM inside the Proxmox storage
+ // before the vSphere VM's link closed the cycle, so the storage or
+ // the vSphere VM survived; the rebuild judges the whole cycle and
+ // keeps the guest whose link is oldest.
+ name: "a cycle across record sources keeps its best guest",
+ links: []ResourceLink{
+ link("nas-vm", "pve-storage", "pve-storage", 1),
+ link("pve-storage", "vsphere-vm", "vsphere-vm", 2),
+ link("vsphere-vm", "nas-vm", "nas-vm", 0),
+ },
+ root: "nas-vm",
+ rootType: ResourceTypeVM,
+ },
+ {
+ // The pool folds in through one VM's link while a second VM's link
+ // to it closes a cycle. A TrueNAS refresh recreates the pool's row
+ // (its source maps to a VM); it folds back through the link that
+ // took it in, not the cycle's closing link, so no record turns.
+ name: "a recreated cycle member folds back through its own link",
+ links: []ResourceLink{
+ link(vmB, "vm-r", "vm-r", 0),
+ link("vm-u", "vm-r", "vm-r", 1),
+ link(vmC, "vm-u", "vm-u", 2),
+ link(vmC, "tank", "tank", 3),
+ link("tank", vmB, vmB, 4),
+ },
+ root: "vm-r",
+ rootType: ResourceTypeVM,
+ },
+ {
+ // Two interior members of the chain are recreated by one refresh.
+ // Each folds back along its record; reaching the second through
+ // the VM the first holds would turn that VM's record round and
+ // leave its ID without a row.
+ name: "two recreated interior members fold back along their records",
+ links: []ResourceLink{
+ link("vm-u", "vault", "vault", 0),
+ link("vault", "tank", "tank", 1),
+ link("tank", "vm-r", "vm-r", 2),
+ },
+ root: "vm-r",
+ rootType: ResourceTypeVM,
+ },
+ {
+ name: "two recreated interior members with a closing link",
+ links: []ResourceLink{
+ link("vm-u", "vault", "vault", 0),
+ link("vault", "tank", "tank", 1),
+ link("tank", "vm-r", "vm-r", 2),
+ link("vm-u", "vm-r", "vm-r", 3),
+ },
+ root: "vm-r",
+ rootType: ResourceTypeVM,
+ },
+ {
+ // A TrueNAS refresh recreates the VM's row, since its source maps to
+ // a row of another type; the row folds back through the same pair.
+ name: "a member a refresh recreates folds back into its root",
+ links: []ResourceLink{link("nas-vm", "pve-storage", "pve-storage", 0)},
+ root: "pve-storage",
+ rootType: ResourceTypeStorage,
+ },
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ rootID := estate.ids[tc.root]
+ var reference []ManualLinkFold
+ check := func(order []int, view string, rr *ResourceRegistry) {
+ t.Helper()
+ root, ok := rr.Get(rootID)
+ if !ok || root.Type != tc.rootType {
+ t.Fatalf("order %v, %s: root %s = %+v (listed=%v), want a listed %s", order, view, rootID, root, ok, tc.rootType)
+ }
+ pairs := make(map[string]bool, len(tc.links))
+ for _, l := range tc.links {
+ pairs[exclusionKey(l.ResourceA, l.ResourceB)] = true
+ for _, member := range []string{l.ResourceA, l.ResourceB} {
+ if _, listed := rr.Get(member); listed && member != rootID {
+ t.Fatalf("order %v, %s: member %s still listed beside %s", order, view, member, rootID)
+ }
+ if got, ok := rr.ResolveReferenceID(member); !ok || got != rootID {
+ t.Fatalf("order %v, %s: %s resolved to %q (%v), want %s", order, view, member, got, ok, rootID)
+ }
+ }
+ }
+ folds := rr.ManualLinkFolds(rootID)
+ slices.SortFunc(folds, func(a, b ManualLinkFold) int {
+ return strings.Compare(a.HolderID+"|"+a.FoldedID, b.HolderID+"|"+b.FoldedID)
+ })
+ for _, fold := range folds {
+ delete(pairs, exclusionKey(fold.HolderID, fold.FoldedID))
+ }
+ if len(folds) != len(tc.links) || len(pairs) != 0 {
+ t.Fatalf("order %v, %s: folds %+v, want one per link pair %+v", order, view, folds, tc.links)
+ }
+ if reference == nil {
+ reference = folds
+ } else if !reflect.DeepEqual(folds, reference) {
+ t.Fatalf("order %v, %s: folds %+v, want %+v as in the first order", order, view, folds, reference)
+ }
+ }
+ for _, order := range linkOrders(len(tc.links)) {
+ store := NewMemoryStore()
+ for _, i := range order {
+ if err := store.AddLink(tc.links[i]); err != nil {
+ t.Fatalf("add link: %v", err)
+ }
+ }
+ adapter := NewMonitorAdapter(NewRegistry(store))
+ adapter.PopulateSnapshotAndSupplemental(estate.snapshot, estate.records)
+ check(order, "first rebuild", adapter.currentRegistry())
+ adapter.PopulateSnapshotAndSupplemental(estate.snapshot, estate.records)
+ check(order, "second rebuild", adapter.currentRegistry())
+ readState := ReadStateWithHostContinuity(adapter, []IngestRecord{HostIngestRecord(estate.saved)}).(*MonitorAdapter)
+ check(order, "read state", readState.registry)
+ seeded := NewRegistry(store)
+ seeded.IngestResources(readState.GetAll())
+ check(order, "resources API", seeded)
+ for _, source := range []DataSource{SourceTrueNAS, SourceVMware} {
+ adapter.PopulateSupplementalRecords(source, estate.records[source])
+ check(order, "refresh of "+string(source), adapter.currentRegistry())
+ }
+ }
+ })
+ }
+}
+
+// A live refresh can bring a member the last rebuild had not seen, and it
+// cannot undo the folds the rebuild made. The TrueNAS VM is linked into the
+// Proxmox storage, and the storage into a vSphere host that only the refresh
+// reports. The host is the chain's only candidate, but an agent never holds a
+// guest, so the storage that already holds the VM keeps the chain; the next
+// rebuild judges the whole chain and keeps the VM itself.
+func TestManualLinkChainRefreshNeverLeavesAnAgentHoldingAGuest(t *testing.T) {
+ estate := newChainLinkEstate(t)
+ now := time.Now().UTC()
+ host := IngestRecord{
+ SourceID: "vc-1:host:host-9",
+ Resource: Resource{
+ Type: ResourceTypeAgent, Technology: "vmware", Name: "esx-9", Status: StatusOnline, LastSeen: now,
+ VMware: &VMwareData{ConnectionID: "vc-1", ManagedObjectID: "host-9", EntityType: "host"},
+ },
+ Identity: ResourceIdentity{Hostnames: []string{"esx-9"}},
+ }
+ withHost := make(map[DataSource][]IngestRecord, len(estate.records))
+ for source, records := range estate.records {
+ withHost[source] = records
+ }
+ withHost[SourceVMware] = append(slices.Clone(estate.records[SourceVMware]), host)
+ probe := NewMonitorAdapter(NewRegistry(nil))
+ probe.PopulateSnapshotAndSupplemental(estate.snapshot, withHost)
+ var hostID string
+ for _, resource := range probe.GetAll() {
+ if resource.Name == "esx-9" {
+ hostID = resource.ID
+ }
+ }
+ if hostID == "" {
+ t.Fatalf("probe estate %+v has no vSphere host", probe.GetAll())
+ }
+ vmID, storageID := estate.ids["nas-vm"], estate.ids["pve-storage"]
+ created := time.Date(2026, 10, 1, 9, 0, 0, 0, time.UTC)
+ store := NewMemoryStore()
+ for _, link := range []ResourceLink{
+ {ResourceA: vmID, ResourceB: storageID, PrimaryID: storageID, CreatedAt: created},
+ {ResourceA: storageID, ResourceB: hostID, PrimaryID: hostID, CreatedAt: created.Add(time.Minute)},
+ } {
+ if err := store.AddLink(link); err != nil {
+ t.Fatalf("add link: %v", err)
+ }
+ }
+ adapter := NewMonitorAdapter(NewRegistry(store))
+ adapter.PopulateSnapshotAndSupplemental(estate.snapshot, estate.records)
+ assertChainRoot(t, "rebuild without the host", adapter.currentRegistry(), storageID, ResourceTypeStorage, vmID, storageID)
+ adapter.PopulateSupplementalRecords(SourceVMware, withHost[SourceVMware])
+ assertChainRoot(t, "refresh bringing the host", adapter.currentRegistry(), storageID, ResourceTypeStorage, vmID, storageID, hostID)
+ adapter.PopulateSnapshotAndSupplemental(estate.snapshot, withHost)
+ assertChainRoot(t, "rebuild with the host", adapter.currentRegistry(), vmID, ResourceTypeVM, vmID, storageID, hostID)
+}
+
+func assertChainRoot(t *testing.T, step string, rr *ResourceRegistry, rootID string, rootType ResourceType, members ...string) {
+ t.Helper()
+ root, ok := rr.Get(rootID)
+ if !ok || root.Type != rootType {
+ t.Fatalf("%s: root %s = %+v (listed=%v), want a listed %s", step, rootID, root, ok, rootType)
+ }
+ for _, member := range members {
+ if _, listed := rr.Get(member); listed && member != rootID {
+ t.Fatalf("%s: member %s still listed beside %s", step, member, rootID)
+ }
+ if got, ok := rr.ResolveReferenceID(member); !ok || got != rootID {
+ t.Fatalf("%s: %s resolved to %q (%v), want %s", step, member, got, ok, rootID)
+ }
+ }
+}
+
+// A relink of a Proxmox node and its agent names the node's own ID, which the
+// inferred join consumed before any link applies. The join records that fold
+// before the links resolve their members, so a second link naming the node
+// joins the chain whichever link the store lists first: applied in store
+// order, a link naming the node ahead of the relink found no node and was
+// skipped for good. The folded node keeps the agent shape it had, so a guest
+// linked to it still outranks storage as the chain's root.
+func TestManualLinkToAJoinedNodeFoldsTheSameWayInAnyLinkOrder(t *testing.T) {
+ now := time.Now().UTC()
+ node := models.Node{ID: "lab-pve1", Name: "pve1", Instance: "lab", Host: "https://10.0.0.5:8006", Status: "online", LastSeen: now, LinkedAgentID: "host-pve1"}
+ host := models.Host{
+ ID: "host-pve1", Hostname: "pve1", MachineID: "0123456789abcdef", Status: "online", LastSeen: now, LinkedNodeID: node.ID,
+ NetworkInterfaces: []models.HostNetworkInterface{{Name: "vmbr0", MAC: "aa:bb:cc:dd:ee:01", Addresses: []string{"10.0.0.5/24"}}},
+ }
+ pool := IngestRecord{
+ SourceID: "system:tn-1:pool:tank",
+ Resource: Resource{
+ Type: ResourceTypeStorage, Name: "tank", Status: StatusOnline, LastSeen: now,
+ Storage: &StorageMeta{Type: "zfs-pool", Platform: "truenas", Topology: "pool"},
+ },
+ }
+ snapshot := models.StateSnapshot{
+ LastUpdate: now, Nodes: []models.Node{node}, Hosts: []models.Host{host},
+ VMs: []models.VM{{ID: "lab:pve1:101", VMID: 101, Name: "web", Node: "pve1", Instance: "lab", Status: "running", Type: "qemu", LastSeen: now}},
+ }
+ records := map[DataSource][]IngestRecord{SourceTrueNAS: {pool}}
+
+ alone := NewRegistry(nil)
+ alone.IngestSnapshot(models.StateSnapshot{LastUpdate: now, Nodes: []models.Node{node}})
+ nodeID := hostRowsByFacet(t, alone.List()).node
+ probe := NewMonitorAdapter(NewRegistry(nil))
+ probe.PopulateSnapshotAndSupplemental(snapshot, records)
+ joinedID := hostRowsByFacet(t, probe.GetAll()).joined
+ var poolID, vmID string
+ for _, resource := range probe.GetAll() {
+ switch resource.Type {
+ case ResourceTypeStorage:
+ poolID = resource.ID
+ case ResourceTypeVM:
+ vmID = resource.ID
+ }
+ }
+ if nodeID == "" || joinedID == "" || poolID == "" || vmID == "" || nodeID == joinedID {
+ t.Fatalf("fixture joined node %q and agent into %q beside pool %q and VM %q", nodeID, joinedID, poolID, vmID)
+ }
+
+ created := time.Date(2026, 10, 1, 9, 0, 0, 0, time.UTC)
+ relink := ResourceLink{ResourceA: joinedID, ResourceB: nodeID, PrimaryID: joinedID, CreatedAt: created}
+ for _, tc := range []struct {
+ name string
+ links []ResourceLink
+ root string
+ rootType ResourceType
+ }{
+ {
+ name: "a pool linked to the node",
+ links: []ResourceLink{relink, {ResourceA: nodeID, ResourceB: poolID, PrimaryID: poolID, CreatedAt: created.Add(time.Minute)}},
+ root: poolID,
+ rootType: ResourceTypeStorage,
+ },
+ {
+ // The node is an agent-type row whichever way it was folded, so a
+ // guest linked to it folds the chain, not the pool.
+ name: "a guest and a pool linked to the node",
+ links: []ResourceLink{
+ relink,
+ {ResourceA: nodeID, ResourceB: vmID, PrimaryID: nodeID, CreatedAt: created.Add(time.Minute)},
+ {ResourceA: nodeID, ResourceB: poolID, PrimaryID: poolID, CreatedAt: created.Add(2 * time.Minute)},
+ },
+ root: vmID,
+ rootType: ResourceTypeVM,
+ },
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ var reference []ManualLinkFold
+ for _, order := range linkOrders(len(tc.links)) {
+ store := NewMemoryStore()
+ for _, i := range order {
+ if err := store.AddLink(tc.links[i]); err != nil {
+ t.Fatalf("add link: %v", err)
+ }
+ }
+ adapter := NewMonitorAdapter(NewRegistry(store))
+ adapter.PopulateSnapshotAndSupplemental(snapshot, records)
+ seeded := NewRegistry(store)
+ seeded.IngestResources(adapter.GetAll())
+ for _, view := range []struct {
+ name string
+ rr *ResourceRegistry
+ }{{"monitor", adapter.currentRegistry()}, {"resources API", seeded}} {
+ assertChainRoot(t, fmt.Sprintf("order %v, %s", order, view.name), view.rr, tc.root, tc.rootType, joinedID, nodeID, poolID)
+ folds := view.rr.ManualLinkFolds(tc.root)
+ slices.SortFunc(folds, func(a, b ManualLinkFold) int {
+ return strings.Compare(a.HolderID+"|"+a.FoldedID, b.HolderID+"|"+b.FoldedID)
+ })
+ if reference == nil {
+ reference = folds
+ } else if !reflect.DeepEqual(folds, reference) {
+ t.Fatalf("order %v, %s: folds %+v, want %+v as in the first order", order, view.name, folds, reference)
+ }
+ }
+ }
+ })
+ }
+}
+
+// A registry seeded from a listing keeps the folds that listing's links made,
+// whatever the store holds now: the links changed after the monitor published
+// it. A record that recreates a folded member then folds it under the new
+// links, and every ID the listing's fold records fold in keeps answering to
+// the row that survives. Turning a record the surviving row carries to match
+// the new link would leave the ID it folded in without a row, and so would
+// hanging the folded row's records from the linking member when the surviving
+// row's own ID lies between that member and the folded row.
+func TestManualLinkChangedAfterSeedingKeepsFoldedIDsResolving(t *testing.T) {
+ estate := newChainLinkEstate(t)
+ created := time.Date(2026, 10, 1, 9, 0, 0, 0, time.UTC)
+ vmR, vmT, pool, otherPool := estate.ids["vm-r"], estate.ids["vm-u"], estate.ids["tank"], estate.ids["vault"]
+ chain := []ResourceLink{
+ {ResourceA: vmT, ResourceB: pool, PrimaryID: pool},
+ {ResourceA: pool, ResourceB: vmR, PrimaryID: vmR},
+ }
+ // The VM, the pool and the second pool linked in a cycle, rooted at the VM.
+ cycle := []ResourceLink{
+ {ResourceA: pool, ResourceB: vmR, PrimaryID: vmR},
+ {ResourceA: otherPool, ResourceB: vmR, PrimaryID: vmR},
+ {ResourceA: pool, ResourceB: otherPool, PrimaryID: otherPool},
+ }
+
+ for _, tc := range []struct {
+ name string
+ published []ResourceLink
+ changed []ResourceLink
+ // root and rootType name the row left after the replay.
+ root string
+ rootType ResourceType
+ members []string
+ }{
+ {
+ // The pool now folds into the VM inside the published root.
+ name: "the pool folds into a member of the root",
+ published: chain,
+ changed: []ResourceLink{{ResourceA: pool, ResourceB: vmT, PrimaryID: vmT}},
+ root: vmR,
+ rootType: ResourceTypeVM,
+ members: []string{vmR, vmT, pool},
+ },
+ {
+ // Only the link to the pool's folded VM stays, so the recreated pool
+ // is the chain's root and the published root folds into it through
+ // that VM, a member its records reach only through the pool.
+ name: "the recreated pool becomes the root through a member",
+ published: chain,
+ changed: []ResourceLink{{ResourceA: vmT, ResourceB: pool, PrimaryID: pool}},
+ root: pool,
+ rootType: ResourceTypeStorage,
+ members: []string{vmR, vmT, pool},
+ },
+ {
+ // Only the link between the pool and the published root stays, with
+ // the pool primary: the root folds into the recreated pool through
+ // the very ID the pool's records fold it under.
+ name: "the recreated pool becomes the root over the published root",
+ published: chain,
+ changed: []ResourceLink{{ResourceA: pool, ResourceB: vmR, PrimaryID: pool}},
+ root: pool,
+ rootType: ResourceTypeStorage,
+ members: []string{vmR, vmT, pool},
+ },
+ {
+ // The published records form a cycle, and the links now root it at a
+ // recreated pool. Every record that named the pool as folded turns
+ // away from it, so the new root is never a folded ID.
+ name: "a recreated pool roots a cycle of records",
+ published: cycle,
+ changed: []ResourceLink{
+ {ResourceA: vmR, ResourceB: pool, PrimaryID: pool},
+ {ResourceA: otherPool, ResourceB: pool, PrimaryID: pool},
+ {ResourceA: vmR, ResourceB: otherPool, PrimaryID: otherPool},
+ },
+ root: pool,
+ rootType: ResourceTypeStorage,
+ members: []string{vmR, pool, otherPool},
+ },
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ published := NewMemoryStore()
+ for i, link := range tc.published {
+ link.CreatedAt = created.Add(time.Duration(i) * time.Minute)
+ if err := published.AddLink(link); err != nil {
+ t.Fatalf("add link: %v", err)
+ }
+ }
+ adapter := NewMonitorAdapter(NewRegistry(published))
+ adapter.PopulateSnapshotAndSupplemental(estate.snapshot, estate.records)
+ assertChainRoot(t, "published", adapter.currentRegistry(), vmR, ResourceTypeVM, tc.members...)
+
+ changed := NewMemoryStore()
+ for i, link := range tc.changed {
+ link.CreatedAt = created.Add(time.Duration(10+i) * time.Minute)
+ if err := changed.AddLink(link); err != nil {
+ t.Fatalf("add link: %v", err)
+ }
+ }
+ seeded := NewRegistry(changed)
+ seeded.IngestResources(adapter.GetAll())
+ assertChainRoot(t, "seeded", seeded, vmR, ResourceTypeVM, tc.members...)
+ seeded.IngestRecords(SourceTrueNAS, estate.records[SourceTrueNAS])
+ assertChainRoot(t, "seeded then replayed", seeded, tc.root, tc.rootType, tc.members...)
+ // Each side keeps what it is on its own through the turned and the
+ // carried records, which report-merge selects links by, and the
+ // surviving row is never recorded as a folded ID, which report-merge
+ // infers its root from when the row is re-keyed.
+ own := map[string][]DataSource{vmR: {SourceProxmox}, vmT: {SourceProxmox}, pool: {SourceTrueNAS}, otherPool: {SourceTrueNAS}}
+ for _, fold := range seeded.ManualLinkFolds(tc.root) {
+ if !reflect.DeepEqual(fold.HolderOwn, own[fold.HolderID]) || !reflect.DeepEqual(fold.FoldedOwn, own[fold.FoldedID]) {
+ t.Fatalf("fold %+v: own sources not %v for holder and %v for folded", fold, own[fold.HolderID], own[fold.FoldedID])
+ }
+ if fold.FoldedID == tc.root {
+ t.Fatalf("fold %+v records the surviving row %s as folded", fold, tc.root)
+ }
+ }
+ })
+ }
+}
+
+// A cycle of links is recorded link by link, so a report-merge naming one
+// member's source cuts every link that member holds: the pair set the chain
+// pass records is what ReportedManualLinkFolds selects from, and the link that
+// closes the cycle is in it with its sides' own sources. A fold recorded for
+// only the links that folded rows in left the reported member joined through
+// the link that closed the cycle.
+func TestManualLinkChainCycleLinksAreAllCutByAReportedMember(t *testing.T) {
+ estate := newChainLinkEstate(t)
+ created := time.Date(2026, 10, 1, 9, 0, 0, 0, time.UTC)
+ nasVM, storage, guest := estate.ids["nas-vm"], estate.ids["pve-storage"], estate.ids["vsphere-vm"]
+ links := []ResourceLink{
+ {ResourceA: nasVM, ResourceB: storage, PrimaryID: storage, CreatedAt: created.Add(time.Minute)},
+ {ResourceA: storage, ResourceB: guest, PrimaryID: guest, CreatedAt: created.Add(2 * time.Minute)},
+ {ResourceA: guest, ResourceB: nasVM, PrimaryID: nasVM, CreatedAt: created},
+ }
+ for _, order := range linkOrders(len(links)) {
+ store := NewMemoryStore()
+ for _, i := range order {
+ if err := store.AddLink(links[i]); err != nil {
+ t.Fatalf("add link: %v", err)
+ }
+ }
+ adapter := NewMonitorAdapter(NewRegistry(store))
+ adapter.PopulateSnapshotAndSupplemental(estate.snapshot, estate.records)
+ seeded := NewRegistry(store)
+ seeded.IngestResources(adapter.GetAll())
+ for name, rr := range map[string]*ResourceRegistry{"monitor": adapter.currentRegistry(), "resources API": seeded} {
+ folds := rr.ManualLinkFolds(nasVM)
+ cut := ReportedManualLinkFolds(nasVM, folds, func(sources ...DataSource) bool {
+ return slices.Contains(sources, SourceVMware)
+ })
+ pairs := make(map[string]bool, len(cut))
+ for _, fold := range cut {
+ pairs[exclusionKey(fold.HolderID, fold.FoldedID)] = true
+ }
+ for _, link := range links {
+ if link.ResourceA != guest && link.ResourceB != guest {
+ continue
+ }
+ if !pairs[exclusionKey(link.ResourceA, link.ResourceB)] {
+ t.Fatalf("order %v, %s: reporting vmware leaves the link %s - %s joined; cut %+v of %+v", order, name, link.ResourceA, link.ResourceB, cut, folds)
+ }
+ }
+ if len(cut) != 2 {
+ t.Fatalf("order %v, %s: reporting vmware cuts %d links, want the two the vSphere VM holds: %+v", order, name, len(cut), cut)
+ }
+ }
+ }
+}
diff --git a/internal/unifiedresources/resolve_test.go b/internal/unifiedresources/resolve_test.go
index ee236ddcc2..7c305f7dd9 100644
--- a/internal/unifiedresources/resolve_test.go
+++ b/internal/unifiedresources/resolve_test.go
@@ -722,3 +722,157 @@ func TestLinkHoldsFollowTheirPrimaryThroughThePass(t *testing.T) {
})
}
}
+
+// A saved agent answers to the row its link partner folds into, in every
+// link order: the read-state overlay and the resources API, which seeds a
+// registry from that listing and re-applies the links, hold it alike. The partner here is the vSphere VM the agent was linked into from
+// its own page, and the VM is itself linked into a TrueNAS pool, so the
+// published registry folds the VM away before continuity adds the agent;
+// the hold used to need both members listed and left the agent answering for
+// itself. When two links would fold the agent, the chain precedence picks:
+// the guest outranks the pool, whichever link the store lists first.
+func TestSavedLinkMemberHoldsFollowTheChainInAnyLinkOrder(t *testing.T) {
+ estate := newLinkedGuestEstate(t)
+ now := time.Now().UTC()
+ pool := func(name string) IngestRecord {
+ return IngestRecord{
+ SourceID: "system:tn-1:pool:" + name,
+ Resource: Resource{
+ Type: ResourceTypeStorage, Name: name, Status: StatusOnline, LastSeen: now,
+ Storage: &StorageMeta{Type: "zfs-pool", Platform: "truenas", Topology: "pool"},
+ },
+ }
+ }
+ records := map[DataSource][]IngestRecord{SourceVMware: estate.vmRecords, SourceTrueNAS: {pool("tank"), pool("vault")}}
+ probe := NewMonitorAdapter(NewRegistry(nil))
+ probe.PopulateSnapshotAndSupplemental(models.StateSnapshot{LastUpdate: now}, records)
+ poolIDs := map[string]string{}
+ for _, resource := range probe.GetAll() {
+ if resource.Type == ResourceTypeStorage {
+ poolIDs[resource.Name] = resource.ID
+ }
+ }
+ if len(poolIDs) != 2 {
+ t.Fatalf("probe estate = %+v, want two pools", probe.GetAll())
+ }
+ created := time.Date(2026, 10, 1, 9, 0, 0, 0, time.UTC)
+
+ for _, tc := range []struct {
+ name string
+ links []ResourceLink
+ want string
+ }{
+ {
+ name: "partner folded into a third resource",
+ links: []ResourceLink{
+ {ResourceA: estate.agentID, ResourceB: estate.vmID, PrimaryID: estate.agentID, CreatedAt: created},
+ {ResourceA: estate.vmID, ResourceB: poolIDs["tank"], PrimaryID: poolIDs["tank"], CreatedAt: created.Add(time.Minute)},
+ },
+ want: poolIDs["tank"],
+ },
+ {
+ name: "competing links pick the guest",
+ links: []ResourceLink{
+ {ResourceA: estate.agentID, ResourceB: poolIDs["vault"], PrimaryID: poolIDs["vault"], CreatedAt: created},
+ {ResourceA: estate.agentID, ResourceB: estate.vmID, PrimaryID: estate.agentID, CreatedAt: created.Add(time.Minute)},
+ },
+ want: estate.vmID,
+ },
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ for _, order := range linkOrders(len(tc.links)) {
+ store := NewMemoryStore()
+ for _, i := range order {
+ if err := store.AddLink(tc.links[i]); err != nil {
+ t.Fatalf("add link: %v", err)
+ }
+ }
+ published := NewMonitorAdapter(NewRegistry(store))
+ published.PopulateSnapshotAndSupplemental(models.StateSnapshot{LastUpdate: time.Now().UTC()}, records)
+ readState := ReadStateWithHostContinuity(published, []IngestRecord{HostIngestRecord(estate.saved)}).(*MonitorAdapter)
+ resources := NewRegistry(store)
+ resources.IngestResources(readState.GetAll())
+ for _, view := range []struct {
+ name string
+ rr *ResourceRegistry
+ }{{"read state", readState.registry}, {"resources API", resources}} {
+ if agent, ok := view.rr.Get(estate.agentID); !ok || agent.Status != StatusOffline {
+ t.Fatalf("order %v, %s: saved agent row = %+v (listed=%v), want its own offline row", order, view.name, agent, ok)
+ }
+ if target, ok := view.rr.Get(tc.want); !ok || target.Agent != nil {
+ t.Fatalf("order %v, %s: hold target %s = %+v (listed=%v), want it listed without the saved agent's payload", order, view.name, tc.want, target, ok)
+ }
+ for _, ref := range []string{estate.agentID, "agent:" + estate.saved.ID} {
+ if got, ok := view.rr.ResolveReferenceID(ref); !ok || got != tc.want {
+ t.Fatalf("order %v, %s: ResolveReferenceID(%q) = %q (%v), want %s", order, view.name, ref, got, ok, tc.want)
+ }
+ if _, got, ok := view.rr.GetByReference(ref); !ok || got != tc.want {
+ t.Fatalf("order %v, %s: GetByReference(%q) = %q (%v), want %s", order, view.name, ref, got, ok, tc.want)
+ }
+ }
+ }
+ }
+ })
+ }
+}
+
+// A saved member held under a partner answers to the row that partner
+// survives in. A seeded listing can carry the partner inside a storage row
+// that the changed links now fold into a guest, so the hold ranks the guest,
+// not the row that left the registry, and repeating the pass changes nothing.
+func TestSavedLinkMemberHoldFollowsThePartnersSurvivingRow(t *testing.T) {
+ estate := newChainLinkEstate(t)
+ created := time.Date(2026, 10, 1, 9, 0, 0, 0, time.UTC)
+ vm, pool, otherPool, guest := estate.ids["pve-vm"], estate.ids["tank"], estate.ids["vault"], estate.ids["vsphere-vm"]
+
+ probe := NewMonitorAdapter(NewRegistry(nil))
+ probe.PopulateSnapshotAndSupplemental(estate.snapshot, estate.records)
+ var savedID string
+ for _, resource := range ReadStateWithHostContinuity(probe, []IngestRecord{HostIngestRecord(estate.saved)}).(*MonitorAdapter).GetAll() {
+ if resource.Agent != nil && resource.Agent.AgentID == estate.saved.ID {
+ savedID = resource.ID
+ }
+ }
+ if savedID == "" {
+ t.Fatal("overlay did not list the saved host")
+ }
+
+ savedLinks := []ResourceLink{
+ {ResourceA: savedID, ResourceB: otherPool, PrimaryID: otherPool, CreatedAt: created},
+ {ResourceA: savedID, ResourceB: vm, PrimaryID: vm, CreatedAt: created.Add(time.Minute)},
+ }
+ published := NewMemoryStore()
+ for _, link := range append([]ResourceLink{{ResourceA: vm, ResourceB: pool, PrimaryID: pool, CreatedAt: created}}, savedLinks...) {
+ if err := published.AddLink(link); err != nil {
+ t.Fatalf("add link: %v", err)
+ }
+ }
+ adapter := NewMonitorAdapter(NewRegistry(published))
+ adapter.PopulateSnapshotAndSupplemental(estate.snapshot, estate.records)
+ readState := ReadStateWithHostContinuity(adapter, []IngestRecord{HostIngestRecord(estate.saved)}).(*MonitorAdapter)
+
+ // The links change after the listing was published: the pool now folds
+ // into the vSphere VM, which takes the Proxmox VM the pool held with it.
+ changed := NewMemoryStore()
+ for _, link := range append([]ResourceLink{{ResourceA: pool, ResourceB: guest, PrimaryID: guest, CreatedAt: created.Add(2 * time.Minute)}}, savedLinks...) {
+ if err := changed.AddLink(link); err != nil {
+ t.Fatalf("add link: %v", err)
+ }
+ }
+ seeded := NewRegistry(changed)
+ seeded.IngestResources(readState.GetAll())
+ check := func(step string) {
+ t.Helper()
+ if got, ok := seeded.ResolveReferenceID(savedID); !ok || got != guest {
+ t.Fatalf("%s: saved agent resolved to %q (%v), want the guest %s its partner folded into", step, got, ok, guest)
+ }
+ if _, listed := seeded.Get(pool); listed {
+ t.Fatalf("%s: pool still listed beside the guest", step)
+ }
+ }
+ check("seeded")
+ seeded.mu.Lock()
+ seeded.applyManualLinks(nil)
+ seeded.mu.Unlock()
+ check("second pass")
+}