Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep a remediation lock across links and ID successions
rcourtmanb5578136775b
diff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.mdindex 710e2f90b5..6c1f8cc5a4 100644--- a/docs/release-control/v6/internal/subsystems/unified-resources.md+++ b/docs/release-control/v6/internal/subsystems/unified-resources.md@@ -3565,6 +3565,86 @@ stable token. Finding-suppression integrations consume the same`ResourceOperatorState` shape via the`ResourceOperatorStateProvider` interface in `internal/ai`.+A remediation lock follows the resource through the identity changes the+stores can see. The planner, the action broker and the dispatch gate+(`validateExecutionPolicy`) read the row keyed by the one canonical ID the+registry lists, so a lock left on an ID the registry no longer lists would let+Pulse act on a resource the operator told it never to touch. The members of a+live manual-link component therefore share the block+(`shareRemediationLockAcrossLinksSQL`, `CarryRemediationLock`). `AddLink` writes+a block held by any member of the linked component to every member, in the+link's own transaction, because the store cannot see resource types and so+cannot know which member the registry keeps (a guest outlives an agent running+inside it whichever way the link points); a chain of links is one component.+`AddExclusion` does the same just before an unlink splits the pair, so each+member the unlink leaves keeps the lock the merged resource carried.+`ApplyCanonicalIDSuccessions` carries the predecessor's block onto a successor+row that already exists, which would otherwise shadow it. The IDs the+predecessor's and successor's links joined before the re-key stay one group:+once the re-key has run (so a successor with no row still receives the+predecessor's whole row, retirement, maintenance, notes and attribution+included), the group shares the block even where re-keying a link onto the+successor lets a newer exclusion supersede it. Successions apply hop by hop in+the order they are declared; the stores do not reorder a batch, so a chain+declared against its direction in one batch, or into an intermediate ID whose+own succession was recorded earlier, carries one hop per succession. A+succession applies once per predecessor in both stores, so a lock the operator+lifts on the successor, by editing or clearing its row, is not brought back by+a re-declared succession. Opening a store that has not run the one-shot+upgrade carry (`migrateRemediationLockCarry`, recorded in+`resource_store_migrations`) restores the locks that earlier links and+already-recorded successions dropped, following a recorded chain to every ID+that still has a row and repeating with the links until nothing changes; it+records itself done only once it has converged, does not run again after that,+so a lock lifted afterwards stays lifted, and a failure is logged and retried+at the next start rather than keeping the store closed. Sharing after a+succession batch waits until every predecessor's row in the batch has moved+and follows each member through the batch's re-keys, so a block-only row made+for one predecessor cannot take the place of another's whole row and an ID a+later succession in the batch re-keyed away is not given a row. The link reads+that settle a lock take only the columns that decide a pair and treat a time+the driver cannot read as the zero time, so a malformed link row cannot stop a+succession, a link or an unlink.++What crosses is the block only, as `NeverAutoRemediate`, and `Retired` counts+as a block: retirement also silences alerts and Patrol attention, which is+monitoring posture rather than a safety property, so a retired donor does not+retire the survivor. Every other setting an existing survivor row holds is left+untouched, a member with no row gets a row carrying the block alone, and the+carried row is attributed `system:identity-change` rather than to a person.+The operator lifts the lock on the resource the registry lists, which is where+the operator-state API resolves a folded ID to, and a restart, a rebuild or a+re-declared succession does not put it back. The copies the sharing wrote on+folded members cannot be reached through that API while the pair is linked, so+they share again at the component's next link, unlink or succession. Unlinking+the pair removes no carried lock from either side; once unlinked, each member+can be edited or cleared on its own.++The sharing is bounded by when it runs, and the contract does not claim more.+A lock written to a linked ID after the link was recorded reaches the other+members at the component's next link, unlink or succession, and reaches the+survivor at once only when the ID resolves through the registry, as the operator-state API+resolves a folded ID to its holder; a lock written while the registry cannot+resolve the ID (a member absent from the current generation) stays on that ID+until then. The store cannot see which links the registry declines to fold+(availability-owned endpoints, a node and agent pair a newer split separated),+so those share too, which can lock a host because a linked probe was retired.+Re-recording an existing link shares again. Canonical ID changes that reach+neither a link nor `ApplyCanonicalIDSuccessions` are not covered here: a+physical disk re-keyed in place to a machine-scoped ID+(`rekeyPhysicalDiskLocked` moves no operator state), and a host whose earlier+ID never held an identity pin, which no pin-driven succession can name. Proof:+`TestResourceRegistry_ManualLinkFoldKeepsRemediationLock`,+`TestResourceRegistry_ManualLinkFoldLockLeavesSurvivorSettings`,+`TestLinkedComponentSharesRemediationLockThroughLinksAndUnlinks`,+`TestSQLiteStoreRestoresLocksEarlierIdentityChangesDropped` and+`TestSQLiteStoreRestoresLocksAcrossRecordedSuccessionChains` in+`internal/unifiedresources/registry_test.go`, and+`TestCanonicalIDSuccessionCarriesRemediationLockOntoExistingSuccessorRow`,+`TestCanonicalIDSuccessionLockReachesLinkedSurvivor` and+`TestCanonicalIDSuccessionKeepsLinkedMembersTogether` in+`internal/unifiedresources/canonical_id_succession_test.go`.+`actions.go` now owns the canonical action preflight and audit-normalizationcontract. Action plans must carry dry-run availability, safety checks, andverification steps through `preflight`, and `RecordActionAudit` plus@@ -3897,7 +3977,10 @@ reported hostname or machine-identity changes, and separate connections withthe same hostname remain distinct. Legacy hostname-derived IDs may be exposedas explicit `SupersededCanonicalIDs`; registry persistence may re-key ownedoperator state and delete the retired pin, while ordinary aliases remain-read-only lookup compatibility. Identity pins from an unrelated Pulse Agent+read-only lookup compatibility. A successor row that already exists keeps+its own settings, but on the succession's first application the predecessor's+remediation lock is carried onto it (`CarryRemediationLock`) and across the+successor's links rather than shadowed. Identity pins from an unrelated Pulse Agentmust not complete or merge a TrueNAS connection identity.`canonicalIdentity.supersededIds` is preserved across backend projection and@@ -6919,7 +7002,12 @@ links" below).merged row's maintenance and monitoring mode, not an operator-staterow left under the folded ID, whenever that read state folds theagent, or holds the saved agent beside the guest after a restart- (above). History does+ (above). The folded row's remediation lock is the one part of it that+ does not stay behind: the store shares it across the link's component when+ the link is recorded or unlinked, and once on upgrade for earlier links, so+ the gates, which read the merged row's ID, still refuse (the remediation+ lock note under the operator-state contract above, which also states its+ limits). History doesnot follow the fold: a history binding persists, and`expandHistoryAliases` walks bindings in both directions, so binding afolded ID would join the two journals for good. Folded IDs are neverdiff --git a/internal/unifiedresources/canonical_id_succession.go b/internal/unifiedresources/canonical_id_succession.goindex a6e4ce4044..60ac83f43b 100644--- a/internal/unifiedresources/canonical_id_succession.go+++ b/internal/unifiedresources/canonical_id_succession.go@@ -1,6 +1,9 @@package unifiedresources-import "fmt"+import (+ "fmt"+ "time"+)// CanonicalIDSuccession records that the resource durably pinned under// OldCanonicalID now mints NewCanonicalID. Canonical IDs hash the strongest@@ -37,7 +40,12 @@ type canonicalIDSuccessor interface {// and lets change-journal reads merge the old era without rewriting history.// Re-keys never clobber rows already present under the new ID (those are// fresher); a shadowed old row is left behind, which matches the-// pre-succession orphaning behavior.+// pre-succession orphaning behavior. The one thing a shadowed row still+// contributes is its remediation lock: the operator's "never touch this" is+// carried onto the surviving row (CarryRemediationLock) rather than lost to+// whichever row happened to be keyed first. The carry happens with the+// succession's first application, so an operator who later lifts the lock is+// not overruled by the shadowed row on a re-declared succession.func (s *SQLiteResourceStore) ApplyCanonicalIDSuccessions(successions []CanonicalIDSuccession) error {if len(successions) == 0 {return nil@@ -75,6 +83,9 @@ func (s *SQLiteResourceStore) ApplyCanonicalIDSuccessions(successions []Canonica}()applied := make([]CanonicalIDSuccession, 0, len(pending))+ var carried []carriedLock+ var groups [][]string+ rekeyed := make(map[string]string, len(pending))for _, succession := range pending {oldID := succession.OldCanonicalIDnewID := succession.NewCanonicalID@@ -90,6 +101,26 @@ func (s *SQLiteResourceStore) ApplyCanonicalIDSuccessions(successions []Canonicaapplied = append(applied, succession)continue}+ carriedAt := time.Now().UTC()+ // The re-key below moves the predecessor's links onto the successor,+ // where a newer exclusion naming either ID can make one read as+ // unlinked. The IDs those links joined stay one group for the sharing+ // after the re-key.+ members, err := linkedMembersSQL(tx, []string{oldID, newID})+ if err != nil {+ return fmt.Errorf("read links of %q and %q: %w", oldID, newID, err)+ }+ // A successor row that already exists wins the re-key below, so the+ // predecessor's remediation lock would be shadowed by it. Carry the+ // block onto it first; with no successor row the re-key moves the+ // predecessor's whole row and there is nothing to carry.+ shadowed, err := carryRemediationLockSQL(tx, newID, oldID, carriedAt, false)+ if err != nil {+ return fmt.Errorf("carry remediation lock %q -> %q: %w", oldID, newID, err)+ }+ if shadowed {+ carried = append(carried, carriedLock{to: newID, from: oldID})+ }for _, stmt := range []string{`UPDATE OR IGNORE resource_operator_state SET canonical_id = ? WHERE canonical_id = ?`,`UPDATE action_audits SET canonical_id = ? WHERE canonical_id = ?`,@@ -108,13 +139,31 @@ func (s *SQLiteResourceStore) ApplyCanonicalIDSuccessions(successions []Canonicaif _, err := tx.Exec(`DELETE FROM resource_identities WHERE canonical_id = ?`, oldID); err != nil {return fmt.Errorf("delete superseded identity pin %q: %w", oldID, err)}+ groups = append(groups, groupAfterRekey(members, oldID, newID))+ rekeyed[oldID] = newIDapplied = append(applied, succession)}+ // Whether a successor took its predecessor's whole row or kept its own, it+ // may be linked into another resource the registry keeps instead; the lock+ // has to reach whichever member survives. Sharing waits for the whole batch+ // so a member that had no row is given the block alone only once every+ // predecessor's whole row has had its move: a row made earlier would win+ // the re-key of a later predecessor into the same successor.+ sharedAt := time.Now().UTC()+ for _, group := range groups {+ shared, err := shareRemediationLockAmongSQL(tx, settledGroup(group, rekeyed), sharedAt)+ if err != nil {+ return fmt.Errorf("share remediation lock across linked resources %v: %w", group, err)+ }+ carried = append(carried, shared...)+ }+if err := tx.Commit(); err != nil {return fmt.Errorf("commit canonical ID succession: %w", err)}committed = true+ logCarriedLocks("a canonical ID succession", carried)s.identityPinMu.Lock()s.identityPinFresh = false@@ -181,17 +230,30 @@ func (m *MemoryStore) ApplyCanonicalIDSuccessions(successions []CanonicalIDSuccem.mu.Lock()defer m.mu.Unlock()+ var groups [][]string+ rekeyed := make(map[string]string, len(successions))for _, succession := range successions {oldID := CanonicalResourceID(succession.OldCanonicalID)newID := CanonicalResourceID(succession.NewCanonicalID)if oldID == "" || newID == "" || oldID == newID {continue}+ // A succession applies once per predecessor, as SQLite's recorded+ // successions do: a repeat neither re-keys nor re-carries.+ if _, recorded := m.canonicalSuccessions[oldID]; recorded {+ continue+ }+ carriedAt := time.Now().UTC()+ // See the SQLite store: the IDs the links joined before the re-key stay+ // one group for the sharing after it.+ members := flattenComponents(linkComponents(m.livePairsLocked(), []string{oldID, newID}))if state, ok := m.resourceOperatorState[oldID]; ok {if _, taken := m.resourceOperatorState[newID]; !taken {state.CanonicalID = newIDm.resourceOperatorState[newID] = statedelete(m.resourceOperatorState, oldID)+ } else {+ m.carryRemediationLockLocked(newID, oldID, carriedAt, false)}}for i := range m.links {@@ -217,9 +279,14 @@ func (m *MemoryStore) ApplyCanonicalIDSuccessions(successions []CanonicalIDSucceif m.canonicalSuccessions == nil {m.canonicalSuccessions = make(map[string]string)}- if _, exists := m.canonicalSuccessions[oldID]; !exists {- m.canonicalSuccessions[oldID] = newID- }+ m.canonicalSuccessions[oldID] = newID+ groups = append(groups, groupAfterRekey(members, oldID, newID))+ rekeyed[oldID] = newID+ }+ // See the SQLite store: sharing waits for the whole batch.+ sharedAt := time.Now().UTC()+ for _, group := range groups {+ m.shareRemediationLockAmongLocked(settledGroup(group, rekeyed), sharedAt)}return nil}diff --git a/internal/unifiedresources/canonical_id_succession_test.go b/internal/unifiedresources/canonical_id_succession_test.goindex ae4e31db76..1b09509c5f 100644--- a/internal/unifiedresources/canonical_id_succession_test.go+++ b/internal/unifiedresources/canonical_id_succession_test.go@@ -1,6 +1,7 @@package unifiedresourcesimport (+ "fmt""testing""time")@@ -132,3 +133,373 @@ func TestSQLiteCanonicalIDSuccessionKeepsExistingTargetRow(t *testing.T) {t.Fatalf("succession overwrote the fresher successor row: %+v", state)}}++// The successor's row wins over the superseded one (it is fresher), but a+// remediation lock is the operator's "never touch this" and must not be shed+// by whichever row happened to be written second. The successor keeps every+// setting of its own and gains only the block.+func TestCanonicalIDSuccessionCarriesRemediationLockOntoExistingSuccessorRow(t *testing.T) {+ stores := map[string]func(t *testing.T) ResourceStore{+ "memory": func(t *testing.T) ResourceStore { return NewMemoryStore() },+ "sqlite": func(t *testing.T) ResourceStore {+ store, err := NewSQLiteResourceStore(t.TempDir(), "default")+ if err != nil {+ t.Fatalf("NewSQLiteResourceStore: %v", err)+ }+ t.Cleanup(func() { _ = store.Close() })+ return store+ },+ }+ locks := map[string]ResourceOperatorState{+ "never auto-remediate": {NeverAutoRemediate: true},+ "retired": {LifecycleState: LifecycleStateRetired},+ }+ for storeName, newStore := range stores {+ for lockName, lock := range locks {+ t.Run(storeName+"/"+lockName, func(t *testing.T) {+ store := newStore(t)+ succeeder, ok := store.(canonicalIDSuccessor)+ if !ok {+ t.Fatalf("%T cannot apply canonical ID successions", store)+ }+ oldID := buildHashID(ResourceTypeAgent, "cluster:prod-swarm:vault")+ newID := buildHashID(ResourceTypeAgent, "cluster:prod-swarm:vault.a")+ now := time.Now().UTC()++ predecessor := lock+ predecessor.CanonicalID = oldID+ predecessor.SetAt = now.Add(-72 * time.Hour)+ predecessor.SetBy = "operator"+ if err := store.SetResourceOperatorState(predecessor); err != nil {+ t.Fatalf("seed predecessor: %v", err)+ }+ if err := store.SetResourceOperatorState(ResourceOperatorState{+ CanonicalID: newID,+ MonitoringMode: MonitoringModeExpectedOffline,+ Criticality: CriticalityHigh,+ Note: "set under the new ID before the era change was declared",+ SetAt: now.Add(-time.Hour),+ SetBy: "operator",+ }); err != nil {+ t.Fatalf("seed successor: %v", err)+ }++ if err := succeeder.ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{{OldCanonicalID: oldID, NewCanonicalID: newID}}); err != nil {+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)+ }++ got, found, err := store.GetResourceOperatorState(newID)+ if err != nil || !found {+ t.Fatalf("successor state missing: found=%v err=%v", found, err)+ }+ if !got.BlocksRemediation() {+ t.Fatalf("the successor's unlocked row shadowed the predecessor's lock: %+v", got)+ }+ if got.MonitoringMode != MonitoringModeExpectedOffline || got.Criticality != CriticalityHigh ||+ got.Note != "set under the new ID before the era change was declared" || got.LifecycleState != LifecycleStateActive {+ t.Fatalf("carrying the lock rewrote the successor's own settings: %+v", got)+ }++ // The block is the operator's to lift on the successor. Declaring+ // the same succession again (every steady-state rebuild does) must+ // not bring it back.+ got.NeverAutoRemediate = false+ if err := store.SetResourceOperatorState(got); err != nil {+ t.Fatalf("operator lifts the lock: %v", err)+ }+ if err := succeeder.ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{{OldCanonicalID: oldID, NewCanonicalID: newID}}); err != nil {+ t.Fatalf("ApplyCanonicalIDSuccessions (repeat): %v", err)+ }+ if again, _, err := store.GetResourceOperatorState(newID); err != nil || again.BlocksRemediation() {+ t.Fatalf("a repeated succession re-locked the successor the operator cleared: err=%v state=%+v", err, again)+ }++ // Clearing the row outright is the other way to lift it, and the+ // repeat must not move the shadowed predecessor's row back in.+ if err := store.ClearResourceOperatorState(newID); err != nil {+ t.Fatalf("operator clears the successor: %v", err)+ }+ if err := succeeder.ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{{OldCanonicalID: oldID, NewCanonicalID: newID}}); err != nil {+ t.Fatalf("ApplyCanonicalIDSuccessions (repeat after clear): %v", err)+ }+ if again, found, err := store.GetResourceOperatorState(newID); err != nil || found {+ t.Fatalf("a repeated succession restored a row the operator cleared: found=%v err=%v state=%+v", found, err, again)+ }+ })+ }+ }+}++// The registry keeps one member of a manual link, and a succession re-keys the+// link onto the successor. The successor's lock (whether it kept its own row+// or took the predecessor's whole row) has to reach the member that survives.+func TestCanonicalIDSuccessionLockReachesLinkedSurvivor(t *testing.T) {+ stores := map[string]func(t *testing.T) ResourceStore{+ "memory": func(t *testing.T) ResourceStore { return NewMemoryStore() },+ "sqlite": func(t *testing.T) ResourceStore {+ store, err := NewSQLiteResourceStore(t.TempDir(), "default")+ if err != nil {+ t.Fatalf("NewSQLiteResourceStore: %v", err)+ }+ t.Cleanup(func() { _ = store.Close() })+ return store+ },+ }+ for name, newStore := range stores {+ for _, successorHasRow := range []bool{false, true} {+ t.Run(fmt.Sprintf("%s/successor row %v", name, successorHasRow), func(t *testing.T) {+ store := newStore(t)+ succeeder, ok := store.(canonicalIDSuccessor)+ if !ok {+ t.Fatalf("%T cannot apply canonical ID successions", store)+ }+ const guestID = "vm-linked-survivor"+ oldID := buildHashID(ResourceTypeAgent, "linked:old-era")+ newID := buildHashID(ResourceTypeAgent, "linked:new-era")+ now := time.Now().UTC()++ // The link names the superseded ID, so the succession re-keys it.+ if err := store.AddLink(ResourceLink{ResourceA: guestID, ResourceB: oldID, PrimaryID: guestID}); err != nil {+ t.Fatalf("link: %v", err)+ }+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: oldID, NeverAutoRemediate: true, SetAt: now}); err != nil {+ t.Fatalf("lock predecessor: %v", err)+ }+ if successorHasRow {+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: newID, Note: "unlocked", SetAt: now}); err != nil {+ t.Fatalf("seed successor: %v", err)+ }+ }+ if err := succeeder.ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{{OldCanonicalID: oldID, NewCanonicalID: newID}}); err != nil {+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)+ }++ for _, id := range []string{newID, guestID} {+ state, found, err := store.GetResourceOperatorState(id)+ if err != nil || !found || !state.BlocksRemediation() {+ t.Fatalf("%s lost the lock across the succession: found=%v err=%v state=%+v", id, found, err, state)+ }+ }+ })+ }+ }+}++// A succession keeps the IDs its links joined together. The lock follows a+// chain hop by hop as each succession applies, a link that a newer exclusion+// supersedes once it is re-keyed still hands its lock to the member it leaves+// behind (on either side of the succession), and a predecessor linked to its+// own successor still has its whole row move across.+func TestCanonicalIDSuccessionKeepsLinkedMembersTogether(t *testing.T) {+ stores := map[string]func(t *testing.T) ResourceStore{+ "memory": func(t *testing.T) ResourceStore { return NewMemoryStore() },+ "sqlite": func(t *testing.T) ResourceStore {+ store, err := NewSQLiteResourceStore(t.TempDir(), "default")+ if err != nil {+ t.Fatalf("NewSQLiteResourceStore: %v", err)+ }+ t.Cleanup(func() { _ = store.Close() })+ return store+ },+ }+ now := time.Now().UTC()+ locked := func(t *testing.T, store ResourceStore, id string) bool {+ t.Helper()+ state, found, err := store.GetResourceOperatorState(id)+ if err != nil {+ t.Fatalf("read operator state of %s: %v", id, err)+ }+ return found && state.BlocksRemediation()+ }+ apply := func(t *testing.T, store ResourceStore, oldID, newID string) {+ t.Helper()+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{{OldCanonicalID: oldID, NewCanonicalID: newID}}); err != nil {+ t.Fatalf("ApplyCanonicalIDSuccessions %s -> %s: %v", oldID, newID, err)+ }+ }+ for name, newStore := range stores {+ t.Run(name+"/chain across two batches", func(t *testing.T) {+ store := newStore(t)+ for id, state := range map[string]ResourceOperatorState{+ "hop-a": {NeverAutoRemediate: true},+ "hop-b": {Note: "unlocked"},+ "hop-c": {Note: "unlocked"},+ } {+ state.CanonicalID, state.SetAt = id, now+ if err := store.SetResourceOperatorState(state); err != nil {+ t.Fatalf("seed %s: %v", id, err)+ }+ }+ apply(t, store, "hop-a", "hop-b")+ apply(t, store, "hop-b", "hop-c")+ for _, id := range []string{"hop-b", "hop-c"} {+ if !locked(t, store, id) {+ t.Fatalf("%s did not get the lock of the chain's first predecessor", id)+ }+ }+ })+ t.Run(name+"/predecessor's link superseded once re-keyed", func(t *testing.T) {+ store := newStore(t)+ if err := store.AddLink(ResourceLink{ResourceA: "split-a", ResourceB: "split-b", PrimaryID: "split-a", CreatedAt: now.Add(-2 * time.Hour)}); err != nil {+ t.Fatalf("link: %v", err)+ }+ // An exclusion naming the successor, newer than the link: once the+ // link is re-keyed onto the successor this exclusion supersedes it.+ if err := store.AddExclusion(ResourceExclusion{ResourceA: "split-a2", ResourceB: "split-b", CreatedAt: now.Add(-time.Hour)}); err != nil {+ t.Fatalf("exclude: %v", err)+ }+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "split-a", NeverAutoRemediate: true, SetAt: now}); err != nil {+ t.Fatalf("operator locks the merged resource: %v", err)+ }+ apply(t, store, "split-a", "split-a2")+ for _, id := range []string{"split-a2", "split-b"} {+ if !locked(t, store, id) {+ t.Fatalf("%s lost the lock when the succession split the link", id)+ }+ }+ })+ t.Run(name+"/successor's link superseded once re-keyed", func(t *testing.T) {+ store := newStore(t)+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "side-b", Note: "unlocked", SetAt: now}); err != nil {+ t.Fatalf("seed successor: %v", err)+ }+ if err := store.AddLink(ResourceLink{ResourceA: "side-b", ResourceB: "side-x", PrimaryID: "side-x", CreatedAt: now.Add(-2 * time.Hour)}); err != nil {+ t.Fatalf("link: %v", err)+ }+ if err := store.AddExclusion(ResourceExclusion{ResourceA: "side-a", ResourceB: "side-x", CreatedAt: now.Add(-time.Hour)}); err != nil {+ t.Fatalf("exclude: %v", err)+ }+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "side-x", NeverAutoRemediate: true, SetAt: now}); err != nil {+ t.Fatalf("operator locks the merged resource: %v", err)+ }+ apply(t, store, "side-a", "side-b")+ for _, id := range []string{"side-b", "side-x"} {+ if !locked(t, store, id) {+ t.Fatalf("%s lost the lock when the succession split the successor's link", id)+ }+ }+ })+ t.Run(name+"/star batch: sharing waits until every predecessor's row has moved", func(t *testing.T) {+ store := newStore(t)+ if err := store.AddLink(ResourceLink{ResourceA: "star-b", ResourceB: "star-x", PrimaryID: "star-x"}); err != nil {+ t.Fatalf("link: %v", err)+ }+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "star-x", NeverAutoRemediate: true, SetAt: now}); err != nil {+ t.Fatalf("operator locks the linked resource: %v", err)+ }+ if err := store.SetResourceOperatorState(ResourceOperatorState{+ CanonicalID: "star-old2", Note: "keep", Criticality: CriticalityHigh, SetAt: now.Add(-time.Hour), SetBy: "alice",+ }); err != nil {+ t.Fatalf("seed second predecessor: %v", err)+ }+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{+ {OldCanonicalID: "star-old1", NewCanonicalID: "star-b"},+ {OldCanonicalID: "star-old2", NewCanonicalID: "star-b"},+ }); err != nil {+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)+ }+ got, found, err := store.GetResourceOperatorState("star-b")+ if err != nil || !found {+ t.Fatalf("successor row missing: found=%v err=%v", found, err)+ }+ if got.Note != "keep" || got.Criticality != CriticalityHigh {+ t.Fatalf("a block-only row made for the first predecessor shadowed the second one's whole row: %+v", got)+ }+ if !got.BlocksRemediation() {+ t.Fatalf("the successor did not get the lock of the resource its links joined: %+v", got)+ }+ })+ t.Run(name+"/forward chain in one batch, link superseded once re-keyed", func(t *testing.T) {+ store := newStore(t)+ if err := store.AddLink(ResourceLink{ResourceA: "fwd-a", ResourceB: "fwd-x", PrimaryID: "fwd-x", CreatedAt: now.Add(-2 * time.Hour)}); err != nil {+ t.Fatalf("link: %v", err)+ }+ if err := store.AddExclusion(ResourceExclusion{ResourceA: "fwd-b", ResourceB: "fwd-x", CreatedAt: now.Add(-time.Hour)}); err != nil {+ t.Fatalf("exclude: %v", err)+ }+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "fwd-x", NeverAutoRemediate: true, SetAt: now}); err != nil {+ t.Fatalf("operator locks the linked resource: %v", err)+ }+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{+ {OldCanonicalID: "fwd-a", NewCanonicalID: "fwd-b"},+ {OldCanonicalID: "fwd-b", NewCanonicalID: "fwd-c"},+ }); err != nil {+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)+ }+ for _, id := range []string{"fwd-c", "fwd-x"} {+ if !locked(t, store, id) {+ t.Fatalf("%s lost the lock across a chain declared in one batch", id)+ }+ }+ if state, found, err := store.GetResourceOperatorState("fwd-b"); err != nil || found {+ t.Fatalf("the re-keyed intermediate ID was given a row: found=%v err=%v state=%+v", found, err, state)+ }+ })+ t.Run(name+"/predecessor linked to its successor keeps its whole row", func(t *testing.T) {+ store := newStore(t)+ if err := store.AddLink(ResourceLink{ResourceA: "own-a", ResourceB: "own-b", PrimaryID: "own-a"}); err != nil {+ t.Fatalf("link: %v", err)+ }+ if err := store.SetResourceOperatorState(ResourceOperatorState{+ CanonicalID: "own-a", LifecycleState: LifecycleStateRetired, Note: "keep", SetAt: now.Add(-time.Hour), SetBy: "alice",+ }); err != nil {+ t.Fatalf("seed predecessor: %v", err)+ }+ apply(t, store, "own-a", "own-b")+ got, found, err := store.GetResourceOperatorState("own-b")+ if err != nil || !found {+ t.Fatalf("successor row missing: found=%v err=%v", found, err)+ }+ if got.LifecycleState != LifecycleStateRetired || got.Note != "keep" || got.SetBy != "alice" {+ t.Fatalf("the predecessor's whole row did not move across: %+v", got)+ }+ })+ }+}++// Link rows may leave the optional reason and author empty (NULL) or hold a+// time the driver cannot read. Reading links to settle a lock must not stop an+// unrelated succession, or the upgrade carry, from completing.+func TestSQLiteSuccessionAndUpgradeCarryIgnoreMalformedLinkRows(t *testing.T) {+ dir := t.TempDir()+ store, err := NewSQLiteResourceStore(dir, "default")+ if err != nil {+ t.Fatalf("NewSQLiteResourceStore: %v", err)+ }+ // A row without reason or author, one with an integer where the driver+ // expects a time, and one with no time at all.+ for _, row := range []string{+ `INSERT INTO resource_links (resource_a, resource_b, primary_id) VALUES ('unrelated', 'unrelated', 'unrelated')`,+ `INSERT INTO resource_links (resource_a, resource_b, primary_id, reason, created_by, created_at) VALUES ('z1', 'z2', 'z1', '', '', 123)`,+ `INSERT INTO resource_links (resource_a, resource_b, primary_id, created_at) VALUES ('z3', 'z4', 'z3', NULL)`,+ } {+ if _, err := store.db.Exec(row); err != nil {+ t.Fatalf("seed malformed link row: %v", err)+ }+ }+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "null-old", NeverAutoRemediate: true, Note: "moves", SetAt: time.Now().UTC()}); err != nil {+ t.Fatalf("seed predecessor: %v", err)+ }+ if err := store.ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{{OldCanonicalID: "null-old", NewCanonicalID: "null-new"}}); err != nil {+ t.Fatalf("succession stopped by an unrelated link row: %v", err)+ }+ if got, found, err := store.GetResourceOperatorState("null-new"); err != nil || !found || got.Note != "moves" {+ t.Fatalf("predecessor's row did not move: found=%v err=%v state=%+v", found, err, got)+ }+ if _, err := store.db.Exec(`DELETE FROM resource_store_migrations`); err != nil {+ t.Fatalf("forget the carry marker: %v", err)+ }+ if err := store.Close(); err != nil {+ t.Fatalf("close: %v", err)+ }++ store, err = NewSQLiteResourceStore(dir, "default")+ if err != nil {+ t.Fatalf("reopen: %v", err)+ }+ defer store.Close()+ var recorded int+ if err := store.db.QueryRow(`SELECT COUNT(*) FROM resource_store_migrations`).Scan(&recorded); err != nil || recorded != 1 {+ t.Fatalf("the upgrade carry did not complete past the unrelated link row: markers=%d err=%v", recorded, err)+ }+}diff --git a/internal/unifiedresources/registry_test.go b/internal/unifiedresources/registry_test.goindex 8fcdf65ac0..a7118ef715 100644--- a/internal/unifiedresources/registry_test.go+++ b/internal/unifiedresources/registry_test.go@@ -8032,3 +8032,432 @@ func proxmoxNodeSensorSetupOutdated(t *testing.T, resources []Resource, nodeName}return *found.Proxmox.SensorSetupOutdated}++// An operator's remediation lock is a safety property of the thing they+// locked, not of one canonical ID. Linking a locked resource into another+// folds it out of the registry, and the surviving resource is the only ID the+// planner and the dispatch gate ever read, so the lock has to be readable+// there. Lock state is judged the way the gates judge it: the stored row of+// the ID the registry lists.+func TestResourceRegistry_ManualLinkFoldKeepsRemediationLock(t *testing.T) {+ now := time.Now().UTC()+ const (+ guestID = "vm-cluster-a-node-1-501"+ agentID = "agent-inside-vm-501"+ )+ ingest := func(rr *ResourceRegistry) {+ rr.IngestResources([]Resource{+ {+ ID: guestID, Type: ResourceTypeVM, Name: "vm-501", Status: StatusOnline, LastSeen: now,+ Sources: []DataSource{SourceProxmox},+ SourceStatus: map[DataSource]SourceStatus{SourceProxmox: {Status: "online", LastSeen: now}},+ Proxmox: &ProxmoxData{SourceID: "cluster-a-node-1-501", VMID: 501},+ },+ {+ ID: agentID, Type: ResourceTypeAgent, Name: "vm-501", Status: StatusOnline, LastSeen: now,+ Sources: []DataSource{SourceAgent},+ SourceStatus: map[DataSource]SourceStatus{SourceAgent: {Status: "online", LastSeen: now}},+ Agent: &AgentData{AgentID: agentID},+ },+ })+ }+ stores := map[string]func(t *testing.T) ResourceStore{+ "memory": func(t *testing.T) ResourceStore { return NewMemoryStore() },+ "sqlite": func(t *testing.T) ResourceStore {+ store, err := NewSQLiteResourceStore(t.TempDir(), "default")+ if err != nil {+ t.Fatalf("NewSQLiteResourceStore: %v", err)+ }+ t.Cleanup(func() { _ = store.Close() })+ return store+ },+ }+ locks := map[string]ResourceOperatorState{+ "never auto-remediate": {NeverAutoRemediate: true},+ "retired": {LifecycleState: LifecycleStateRetired},+ }+ for storeName, newStore := range stores {+ for lockName, lock := range locks {+ for _, tc := range []struct {+ name string+ locked string+ // primary is the side the link names as its primary. The registry+ // keeps a guest over an agent running inside it whichever way the+ // link points, so a lock on the guest-folded agent has to survive+ // a link that names the agent the primary and a link that names the+ // guest.+ primary string+ }{+ {name: "locked agent folded into the guest, agent named primary", locked: agentID, primary: agentID},+ {name: "locked agent folded into the guest, guest named primary", locked: agentID, primary: guestID},+ {name: "control: locked guest absorbing the agent", locked: guestID, primary: agentID},+ } {+ t.Run(storeName+"/"+lockName+"/"+tc.name, func(t *testing.T) {+ store := newStore(t)+ state := lock+ state.CanonicalID = tc.locked+ state.SetAt = now.Add(-time.Hour)+ state.SetBy = "operator"+ if err := store.SetResourceOperatorState(state); err != nil {+ t.Fatalf("lock %s: %v", tc.locked, err)+ }+ if err := store.AddLink(ResourceLink{ResourceA: guestID, ResourceB: agentID, PrimaryID: tc.primary}); err != nil {+ t.Fatalf("add link: %v", err)+ }++ rr := NewRegistry(store)+ ingest(rr)+ survivors := rr.List()+ if len(survivors) != 1 {+ t.Fatalf("link did not fold the pair into one resource: %d resources", len(survivors))+ }+ survivor := survivors[0].ID++ got, found, err := store.GetResourceOperatorState(survivor)+ if err != nil {+ t.Fatalf("read operator state of survivor %s: %v", survivor, err)+ }+ if !found || !got.BlocksRemediation() {+ t.Fatalf("surviving resource %s lost the lock on %s: found=%v state=%+v", survivor, tc.locked, found, got)+ }+ if got.SuppressesAllAttention() != (lockName == "retired" && tc.locked == survivor) {+ t.Fatalf("lock on %s changed the survivor's alert attention: %+v", tc.locked, got)+ }+ })+ }+ }+ }+}++// Carrying the lock must not rewrite anything else the survivor's operator+// said: its note, priority and monitoring posture stay as they were.+func TestResourceRegistry_ManualLinkFoldLockLeavesSurvivorSettings(t *testing.T) {+ now := time.Now().UTC()+ store := NewMemoryStore()+ const (+ keeperID = "vm-cluster-a-node-1-502"+ foldedID = "agent-inside-vm-502"+ )+ if err := store.SetResourceOperatorState(ResourceOperatorState{+ CanonicalID: keeperID,+ MonitoringMode: MonitoringModeExpectedOffline,+ Criticality: CriticalityHigh,+ Note: "database host",+ SetAt: now.Add(-time.Hour),+ SetBy: "alice",+ }); err != nil {+ t.Fatalf("seed survivor state: %v", err)+ }+ if err := store.SetResourceOperatorState(ResourceOperatorState{+ CanonicalID: foldedID, NeverAutoRemediate: true, SetAt: now.Add(-2 * time.Hour), SetBy: "bob",+ }); err != nil {+ t.Fatalf("seed folded state: %v", err)+ }+ if err := store.AddLink(ResourceLink{ResourceA: keeperID, ResourceB: foldedID, PrimaryID: keeperID}); err != nil {+ t.Fatalf("add link: %v", err)+ }++ got, found, err := store.GetResourceOperatorState(keeperID)+ if err != nil || !found {+ t.Fatalf("survivor state missing: found=%v err=%v", found, err)+ }+ if !got.NeverAutoRemediate {+ t.Fatalf("survivor did not take the folded resource's lock: %+v", got)+ }+ if got.MonitoringMode != MonitoringModeExpectedOffline || got.Criticality != CriticalityHigh || got.Note != "database host" || got.LifecycleState != LifecycleStateActive {+ t.Fatalf("carrying the lock rewrote the survivor's own settings: %+v", got)+ }+ if got.SetBy != RemediationLockCarriedBy {+ t.Fatalf("carried lock is attributed to %q, not to the identity change", got.SetBy)+ }++ // Unlinking splits the pair again; the lock was the operator's on both, so+ // neither side loses it.+ if err := store.AddExclusion(ResourceExclusion{ResourceA: keeperID, ResourceB: foldedID}); err != nil {+ t.Fatalf("unlink: %v", err)+ }+ for _, id := range []string{keeperID, foldedID} {+ if state, found, err := store.GetResourceOperatorState(id); err != nil || !found || !state.NeverAutoRemediate {+ t.Fatalf("unlinking dropped the lock on %s: found=%v err=%v state=%+v", id, found, err, state)+ }+ }+}++// Links recorded before locks travelled with them already folded a locked+// resource out from under its lock, and a succession that landed on an existing+// successor row left its predecessor's lock behind. Opening a store that has+// not yet run the carry restores those once; an operator who then lifts the+// survivor's lock is not overruled by the dormant row on every later start.+func TestSQLiteStoreRestoresLocksEarlierIdentityChangesDropped(t *testing.T) {+ dir := t.TempDir()+ const (+ lockedID = "agent-legacy-locked"+ survivorID = "vm-legacy-survivor"+ chainEndID = "vm-legacy-chain-end"+ unlinkedA = "agent-legacy-unlinked-locked"+ unlinkedB = "vm-legacy-unlinked-open"+ oldEraID = "agent-legacy-old-era"+ newEraID = "agent-legacy-new-era"+ linkedAt = "2026-09-01T10:00:00Z"+ unlinkedAt = "2026-09-02T10:00:00Z"+ operator = "operator"+ legacyReason = "recorded before locks travelled"+ )+ open := func() *SQLiteResourceStore {+ t.Helper()+ store, err := NewSQLiteResourceStore(dir, "default")+ if err != nil {+ t.Fatalf("NewSQLiteResourceStore: %v", err)+ }+ return store+ }+ now := time.Now().UTC()++ store := open()+ for _, id := range []string{lockedID, unlinkedA, oldEraID} {+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: id, NeverAutoRemediate: true, SetAt: now.Add(-48 * time.Hour), SetBy: operator}); err != nil {+ t.Fatalf("lock %s: %v", id, err)+ }+ }+ // A successor row written before the succession was declared, as the+ // succession that shadowed the predecessor's lock found it.+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: newEraID, Note: "set under the new ID", SetAt: now.Add(-24 * time.Hour), SetBy: operator}); err != nil {+ t.Fatalf("seed successor: %v", err)+ }+ // Rows exactly as a store that did not carry locks wrote them: a live link,+ // a chain extension of it, one a later unlink superseded (its exclusion is+ // newer), and a succession already recorded with the predecessor's row left+ // behind.+ for _, row := range []struct{ a, b, primary, at string }{+ {lockedID, survivorID, survivorID, linkedAt},+ {survivorID, chainEndID, chainEndID, linkedAt},+ {unlinkedA, unlinkedB, unlinkedB, linkedAt},+ } {+ if _, err := store.db.Exec(`INSERT INTO resource_links (resource_a, resource_b, primary_id, reason, created_by, created_at) VALUES (?, ?, ?, ?, ?, ?)`,+ row.a, row.b, row.primary, legacyReason, operator, row.at); err != nil {+ t.Fatalf("seed legacy link: %v", err)+ }+ }+ if _, err := store.db.Exec(`INSERT INTO resource_exclusions (resource_a, resource_b, reason, created_by, created_at) VALUES (?, ?, ?, ?, ?)`,+ unlinkedA, unlinkedB, "unlinked", operator, unlinkedAt); err != nil {+ t.Fatalf("seed legacy exclusion: %v", err)+ }+ if _, err := store.db.Exec(`INSERT INTO canonical_id_successions (old_canonical_id, new_canonical_id) VALUES (?, ?)`, oldEraID, newEraID); err != nil {+ t.Fatalf("seed recorded succession: %v", err)+ }+ // This store has never run the carry.+ if _, err := store.db.Exec(`DELETE FROM resource_store_migrations`); err != nil {+ t.Fatalf("forget the carry marker: %v", err)+ }+ if err := store.Close(); err != nil {+ t.Fatalf("close: %v", err)+ }++ store = open()+ for _, id := range []string{survivorID, chainEndID, newEraID} {+ got, found, err := store.GetResourceOperatorState(id)+ if err != nil || !found || !got.BlocksRemediation() {+ t.Fatalf("%s did not get the lock an earlier identity change dropped: found=%v err=%v state=%+v", id, found, err, got)+ }+ }+ if got, _, _ := store.GetResourceOperatorState(newEraID); got.Note != "set under the new ID" {+ t.Fatalf("restoring the successor's lock rewrote its own settings: %+v", got)+ }+ if other, found, err := store.GetResourceOperatorState(unlinkedB); err != nil || found {+ t.Fatalf("a pair the operator unlinked was carried: found=%v err=%v state=%+v", found, err, other)+ }++ for _, id := range []string{survivorID, newEraID} {+ if err := store.ClearResourceOperatorState(id); err != nil {+ t.Fatalf("operator clears %s: %v", id, err)+ }+ }+ if err := store.Close(); err != nil {+ t.Fatalf("close: %v", err)+ }+ store = open()+ defer store.Close()+ for _, id := range []string{survivorID, newEraID} {+ if got, found, err := store.GetResourceOperatorState(id); err != nil || found {+ t.Fatalf("a restart re-locked %s, which the operator cleared: found=%v err=%v state=%+v", id, found, err, got)+ }+ }+}++// The registry keeps one member of a link and acts only on that ID, so a lock+// has to be on every member of the linked component, not just the two IDs of+// the newest link, and it has to stay on each member the unlink splits off.+func TestLinkedComponentSharesRemediationLockThroughLinksAndUnlinks(t *testing.T) {+ stores := map[string]func(t *testing.T) ResourceStore{+ "memory": func(t *testing.T) ResourceStore { return NewMemoryStore() },+ "sqlite": func(t *testing.T) ResourceStore {+ store, err := NewSQLiteResourceStore(t.TempDir(), "default")+ if err != nil {+ t.Fatalf("NewSQLiteResourceStore: %v", err)+ }+ t.Cleanup(func() { _ = store.Close() })+ return store+ },+ }+ locked := func(t *testing.T, store ResourceStore, id string) bool {+ t.Helper()+ state, found, err := store.GetResourceOperatorState(id)+ if err != nil {+ t.Fatalf("read operator state of %s: %v", id, err)+ }+ return found && state.BlocksRemediation()+ }+ now := time.Now().UTC()+ for name, newStore := range stores {+ t.Run(name+"/chain", func(t *testing.T) {+ store := newStore(t)+ if err := store.AddLink(ResourceLink{ResourceA: "host-a", ResourceB: "host-b", PrimaryID: "host-a"}); err != nil {+ t.Fatalf("link a-b: %v", err)+ }+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "host-c", NeverAutoRemediate: true, SetAt: now}); err != nil {+ t.Fatalf("lock c: %v", err)+ }+ if err := store.AddLink(ResourceLink{ResourceA: "host-b", ResourceB: "host-c", PrimaryID: "host-c"}); err != nil {+ t.Fatalf("link b-c: %v", err)+ }+ for _, id := range []string{"host-a", "host-b", "host-c"} {+ if !locked(t, store, id) {+ t.Fatalf("%s is outside the lock of its linked component", id)+ }+ }+ })+ t.Run(name+"/lock set on the survivor after the link, then unlinked", func(t *testing.T) {+ store := newStore(t)+ if err := store.AddLink(ResourceLink{ResourceA: "vm-1", ResourceB: "agent-1", PrimaryID: "vm-1"}); err != nil {+ t.Fatalf("link: %v", err)+ }+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "vm-1", NeverAutoRemediate: true, SetAt: now}); err != nil {+ t.Fatalf("operator locks the merged resource: %v", err)+ }+ if locked(t, store, "agent-1") {+ t.Fatalf("the lock reached the folded member before any identity change")+ }+ if err := store.AddExclusion(ResourceExclusion{ResourceA: "vm-1", ResourceB: "agent-1"}); err != nil {+ t.Fatalf("unlink: %v", err)+ }+ for _, id := range []string{"vm-1", "agent-1"} {+ if !locked(t, store, id) {+ t.Fatalf("unlinking left %s without the merged resource's lock", id)+ }+ }+ })+ t.Run(name+"/lock on one end of a chain, unlinking the other end", func(t *testing.T) {+ store := newStore(t)+ for _, link := range []ResourceLink{+ {ResourceA: "chain-a", ResourceB: "chain-b", PrimaryID: "chain-a"},+ {ResourceA: "chain-b", ResourceB: "chain-c", PrimaryID: "chain-b"},+ } {+ if err := store.AddLink(link); err != nil {+ t.Fatalf("link %s-%s: %v", link.ResourceA, link.ResourceB, err)+ }+ }+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "chain-a", NeverAutoRemediate: true, SetAt: now}); err != nil {+ t.Fatalf("operator locks the merged resource: %v", err)+ }+ if err := store.AddExclusion(ResourceExclusion{ResourceA: "chain-b", ResourceB: "chain-c"}); err != nil {+ t.Fatalf("unlink b-c: %v", err)+ }+ for _, id := range []string{"chain-a", "chain-b", "chain-c"} {+ if !locked(t, store, id) {+ t.Fatalf("%s left the chain without the lock the whole chain carried", id)+ }+ }+ })+ t.Run(name+"/unlinking a pair that was never linked carries nothing", func(t *testing.T) {+ store := newStore(t)+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "vm-2", NeverAutoRemediate: true, SetAt: now}); err != nil {+ t.Fatalf("lock: %v", err)+ }+ if err := store.AddExclusion(ResourceExclusion{ResourceA: "vm-2", ResourceB: "agent-2"}); err != nil {+ t.Fatalf("exclude: %v", err)+ }+ if locked(t, store, "agent-2") {+ t.Fatalf("an exclusion between unlinked resources locked one of them")+ }+ })+ }+}++// Recorded successions can chain, and an intermediate row may have moved on to+// its own successor, so the upgrade carry follows the chain instead of+// stepping one succession at a time; a lock it restores then spreads over the+// links of the ID it lands on.+func TestSQLiteStoreRestoresLocksAcrossRecordedSuccessionChains(t *testing.T) {+ dir := t.TempDir()+ open := func() *SQLiteResourceStore {+ t.Helper()+ store, err := NewSQLiteResourceStore(dir, "default")+ if err != nil {+ t.Fatalf("NewSQLiteResourceStore: %v", err)+ }+ return store+ }+ now := time.Now().UTC()+ store := open()+ defer func() { _ = store.Close() }()++ // a1 -> b1 -> c1: b1's settings row moved to c1, so b1 has no row and the+ // shadowed lock on a1 has to reach c1 across the gap.+ // a2 -> b2 -> c2, recorded in the opposite order from the chain: both+ // intermediate rows still exist.+ // a3 -> b3 with a live link b3 - c3: the restored lock must reach c3.+ seed := func(id string, state ResourceOperatorState) {+ t.Helper()+ state.CanonicalID, state.SetAt = id, now.Add(-72*time.Hour)+ if err := store.SetResourceOperatorState(state); err != nil {+ t.Fatalf("seed %s: %v", id, err)+ }+ }+ seed("chain-a1", ResourceOperatorState{NeverAutoRemediate: true})+ seed("chain-c1", ResourceOperatorState{Note: "moved here from b1"})+ seed("chain-a2", ResourceOperatorState{NeverAutoRemediate: true})+ seed("chain-b2", ResourceOperatorState{Note: "unlocked"})+ seed("chain-c2", ResourceOperatorState{Note: "unlocked"})+ seed("chain-a3", ResourceOperatorState{LifecycleState: LifecycleStateRetired})+ seed("chain-b3", ResourceOperatorState{Note: "unlocked"})+ // a4 -> b4 and c4 -> d4 with a live link b4 - c4: the lock restored onto b4+ // spreads over the link to c4, and from c4 on to d4, which the first pass+ // over the successions had already passed.+ seed("chain-a4", ResourceOperatorState{NeverAutoRemediate: true})+ seed("chain-b4", ResourceOperatorState{Note: "unlocked"})+ seed("chain-c4", ResourceOperatorState{Note: "unlocked"})+ seed("chain-d4", ResourceOperatorState{Note: "unlocked"})+ for _, succession := range [][2]string{+ {"chain-b2", "chain-c2"}, {"chain-a2", "chain-b2"},+ {"chain-a1", "chain-b1"}, {"chain-b1", "chain-c1"},+ {"chain-a3", "chain-b3"},+ {"chain-a4", "chain-b4"}, {"chain-c4", "chain-d4"},+ } {+ if _, err := store.db.Exec(`INSERT INTO canonical_id_successions (old_canonical_id, new_canonical_id) VALUES (?, ?)`, succession[0], succession[1]); err != nil {+ t.Fatalf("record succession %v: %v", succession, err)+ }+ }+ for _, link := range [][2]string{{"chain-b3", "chain-c3"}, {"chain-b4", "chain-c4"}} {+ if _, err := store.db.Exec(`INSERT INTO resource_links (resource_a, resource_b, primary_id, reason, created_by, created_at) VALUES (?, ?, ?, ?, ?, ?)`,+ link[0], link[1], link[0], "linked before locks travelled", "operator", "2026-09-01T10:00:00Z"); err != nil {+ t.Fatalf("seed link: %v", err)+ }+ }+ if _, err := store.db.Exec(`DELETE FROM resource_store_migrations`); err != nil {+ t.Fatalf("forget the carry marker: %v", err)+ }+ if err := store.Close(); err != nil {+ t.Fatalf("close: %v", err)+ }++ store = open()+ for _, id := range []string{"chain-c1", "chain-b2", "chain-c2", "chain-b3", "chain-c3", "chain-b4", "chain-c4", "chain-d4"} {+ got, found, err := store.GetResourceOperatorState(id)+ if err != nil || !found || !got.BlocksRemediation() {+ t.Fatalf("%s did not get the lock its succession chain dropped: found=%v err=%v state=%+v", id, found, err, got)+ }+ }+ if got, _, _ := store.GetResourceOperatorState("chain-c1"); got.Note != "moved here from b1" {+ t.Fatalf("restoring the lock rewrote the successor's own settings: %+v", got)+ }+}diff --git a/internal/unifiedresources/remediation_lock_carry.go b/internal/unifiedresources/remediation_lock_carry.gonew file mode 100644index 0000000000..d38e968ebe--- /dev/null+++ b/internal/unifiedresources/remediation_lock_carry.go@@ -0,0 +1,552 @@+package unifiedresources++import (+ "database/sql"+ "fmt"+ "log"+ "sort"+ "strings"+ "time"+)++// An operator's remediation lock (ResourceOperatorState.BlocksRemediation) is+// stored on the row of one canonical ID, and every gate reads the row of the ID+// a resource currently has. Identity changes move resources between IDs, so the+// stores carry the block with them: a live manual link makes its members one+// resource, and the members of a link component share the block whenever the+// component changes (a link, an unlink) and whenever a succession lands on one+// of them. CarryRemediationLock owns what is written onto a row.++// carriedLock records one remediation block written onto `to` from `from`.+type carriedLock struct {+ to string+ from string+}++func logCarriedLocks(cause string, carried []carriedLock) {+ for _, lock := range carried {+ log.Printf("unifiedresources: carried the remediation lock of %q onto %q (%s)", lock.from, lock.to, cause)+ }+}++// rowsQueryer is the read side shared by *sql.DB and *sql.Tx.+type rowsQueryer interface {+ Query(query string, args ...any) (*sql.Rows, error)+}++// liveLinkPairsSQL returns the ID pairs of the links that are their pair's+// latest operator decision, read through q so a caller inside a transaction+// sees its own writes. It reads only the columns that decide a pair (IDs and+// time), which the schema never lets be NULL except the time, so a row with an+// empty reason or author cannot stop a succession or the upgrade carry.+func liveLinkPairsSQL(q rowsQueryer) ([][2]string, error) {+ links, err := queryPairDecisions(q, `SELECT resource_a, resource_b, created_at FROM resource_links`)+ if err != nil || len(links) == 0 {+ return nil, err+ }+ decided, err := queryPairDecisions(q, `SELECT resource_a, resource_b, created_at FROM resource_exclusions`)+ if err != nil {+ return nil, err+ }+ exclusions := make([]ResourceExclusion, 0, len(decided))+ for _, decision := range decided {+ exclusions = append(exclusions, ResourceExclusion{ResourceA: decision.ResourceA, ResourceB: decision.ResourceB, CreatedAt: decision.CreatedAt})+ }+ live, _ := effectiveManualPairDecisions(links, exclusions)+ return linkPairs(live), nil+}++// decisionTime reads a stored created_at leniently: a time the driver cannot+// parse (NULL, an integer, free text) counts as the zero time, which only+// affects whether a newer decision supersedes it. One malformed row must not+// stop an unrelated succession, link or unlink from settling a lock.+func decisionTime(raw any) time.Time {+ switch value := raw.(type) {+ case time.Time:+ return value+ case string:+ for _, layout := range []string{time.RFC3339Nano, "2006-01-02 15:04:05.999999999-07:00", "2006-01-02 15:04:05"} {+ if parsed, err := time.Parse(layout, value); err == nil {+ return parsed+ }+ }+ }+ return time.Time{}+}++// queryPairDecisions reads (resource_a, resource_b, created_at) rows as links;+// only the pair and its time are meaningful.+func queryPairDecisions(q rowsQueryer, query string) (decisions []ResourceLink, err error) {+ rows, err := q.Query(query)+ if err != nil {+ return nil, fmt.Errorf("query manual pair decisions: %w", err)+ }+ defer func() {+ if closeErr := rows.Close(); closeErr != nil && err == nil {+ err = fmt.Errorf("close manual pair decisions: %w", closeErr)+ }+ }()+ for rows.Next() {+ var (+ a, b string+ at any+ )+ if scanErr := rows.Scan(&a, &b, &at); scanErr != nil {+ return nil, fmt.Errorf("scan manual pair decision: %w", scanErr)+ }+ decisions = append(decisions, ResourceLink{ResourceA: CanonicalResourceID(a), ResourceB: CanonicalResourceID(b), CreatedAt: decisionTime(at)})+ }+ if rowsErr := rows.Err(); rowsErr != nil {+ return nil, fmt.Errorf("iterate manual pair decisions: %w", rowsErr)+ }+ return decisions, nil+}++func linkPairs(links []ResourceLink) [][2]string {+ pairs := make([][2]string, 0, len(links))+ for _, link := range links {+ pairs = append(pairs, [2]string{link.ResourceA, link.ResourceB})+ }+ return pairs+}++// linkComponents groups the IDs reachable from seeds through pairs, one sorted+// slice per connected component that holds at least one seed. A pair names a+// link the registry treats as one resource, in either direction, so a chain of+// links is one component.+func linkComponents(pairs [][2]string, seeds []string) [][]string {+ adjacent := make(map[string][]string, len(pairs)*2)+ for _, pair := range pairs {+ a, b := strings.TrimSpace(pair[0]), strings.TrimSpace(pair[1])+ if a == "" || b == "" || a == b {+ continue+ }+ adjacent[a] = append(adjacent[a], b)+ adjacent[b] = append(adjacent[b], a)+ }+ seen := make(map[string]struct{}, len(seeds))+ var components [][]string+ for _, seed := range seeds {+ seed = strings.TrimSpace(seed)+ if seed == "" {+ continue+ }+ if _, done := seen[seed]; done {+ continue+ }+ seen[seed] = struct{}{}+ component := []string{seed}+ for queue := []string{seed}; len(queue) > 0; queue = queue[1:] {+ for _, next := range adjacent[queue[0]] {+ if _, done := seen[next]; done {+ continue+ }+ seen[next] = struct{}{}+ component = append(component, next)+ queue = append(queue, next)+ }+ }+ sort.Strings(component)+ components = append(components, component)+ }+ return components+}++// shareRemediationLock returns the rows a component must hold so that every+// member carries the block any one of them holds. states holds the members'+// stored rows; a member with no row is absent from it. The donor is the first+// blocking member in ID order, so the result does not depend on map order.+func shareRemediationLock(component []string, states map[string]ResourceOperatorState, at time.Time) (writes []ResourceOperatorState, carried []carriedLock) {+ donorID := ""+ for _, id := range component {+ if state, found := states[id]; found && state.BlocksRemediation() {+ donorID = id+ break+ }+ }+ if donorID == "" {+ return nil, nil+ }+ for _, id := range component {+ if id == donorID {+ continue+ }+ state, found := states[id]+ if next, changed := CarryRemediationLock(id, state, found, states[donorID], true, at); changed {+ writes = append(writes, next)+ carried = append(carried, carriedLock{to: id, from: donorID})+ }+ }+ return writes, carried+}++// carryRemediationLockSQL writes donor's remediation block onto survivor's+// operator-state row inside the caller's transaction (CarryRemediationLock owns+// what is written). createRow false leaves a survivor with no row alone, for+// a caller whose next statement moves the donor's whole row onto it. Reports+// whether a row was written.+func carryRemediationLockSQL(tx *sql.Tx, survivorID, donorID string, at time.Time, createRow bool) (bool, error) {+ survivorID = strings.TrimSpace(survivorID)+ donorID = strings.TrimSpace(donorID)+ if survivorID == "" || donorID == "" || survivorID == donorID {+ return false, nil+ }+ donor, donorFound, err := getResourceOperatorStateSQL(tx, donorID)+ if err != nil || !donorFound || !donor.BlocksRemediation() {+ return false, err+ }+ survivor, survivorFound, err := getResourceOperatorStateSQL(tx, survivorID)+ if err != nil {+ return false, err+ }+ if !survivorFound && !createRow {+ return false, nil+ }+ next, changed := CarryRemediationLock(survivorID, survivor, survivorFound, donor, donorFound, at)+ if !changed {+ return false, nil+ }+ if err := setResourceOperatorStateSQL(tx, next); err != nil {+ return false, err+ }+ return true, nil+}++// shareRemediationLockAmongSQL makes every ID in group carry the block any of+// them holds, inside the caller's transaction, and reports what it wrote.+func shareRemediationLockAmongSQL(tx *sql.Tx, group []string, at time.Time) ([]carriedLock, error) {+ if len(group) < 2 {+ return nil, nil+ }+ states := make(map[string]ResourceOperatorState, len(group))+ for _, id := range group {+ state, found, err := getResourceOperatorStateSQL(tx, id)+ if err != nil {+ return nil, err+ }+ if found {+ states[id] = state+ }+ }+ writes, carried := shareRemediationLock(group, states, at)+ for _, next := range writes {+ if err := setResourceOperatorStateSQL(tx, next); err != nil {+ return nil, err+ }+ }+ return carried, nil+}++// shareRemediationLockAcrossLinksSQL makes every member of each live link+// component holding a seed carry the block any member holds, inside the+// caller's transaction. A link written in the same transaction counts as live.+func shareRemediationLockAcrossLinksSQL(tx *sql.Tx, seeds []string, at time.Time) ([]carriedLock, error) {+ pairs, err := liveLinkPairsSQL(tx)+ if err != nil {+ return nil, err+ }+ var carried []carriedLock+ for _, component := range linkComponents(pairs, seeds) {+ shared, err := shareRemediationLockAmongSQL(tx, component, at)+ if err != nil {+ return nil, err+ }+ carried = append(carried, shared...)+ }+ return carried, nil+}++// linkedMembersSQL returns the IDs in the live link components of seeds, seeds+// included, as one sorted list.+func linkedMembersSQL(tx *sql.Tx, seeds []string) ([]string, error) {+ pairs, err := liveLinkPairsSQL(tx)+ if err != nil {+ return nil, err+ }+ return flattenComponents(linkComponents(pairs, seeds)), nil+}++func flattenComponents(components [][]string) []string {+ seen := make(map[string]struct{})+ var out []string+ for _, component := range components {+ for _, id := range component {+ if _, dup := seen[id]; dup {+ continue+ }+ seen[id] = struct{}{}+ out = append(out, id)+ }+ }+ sort.Strings(out)+ return out+}++// resolveThroughRekeys follows id through a batch's re-keys (a -> b, then+// b -> c) to the ID that holds it once the whole batch has applied.+func resolveThroughRekeys(id string, rekeyed map[string]string) string {+ for hops := 0; hops <= len(rekeyed); hops++ {+ next, moved := rekeyed[id]+ if !moved {+ return id+ }+ id = next+ }+ return id+}++// settledGroup maps a succession's saved sharing group through the batch's+// re-keys, so a member a later succession in the batch re-keyed away is+// replaced by its successor instead of being given a row of its own.+func settledGroup(group []string, rekeyed map[string]string) []string {+ resolved := make([]string, 0, len(group))+ for _, id := range group {+ resolved = append(resolved, resolveThroughRekeys(id, rekeyed))+ }+ return flattenComponents([][]string{resolved})+}++// groupAfterRekey is the set a succession's pre-re-key members form once the+// predecessor's ID has become the successor's: they were one linked resource+// before the re-key and stay one group for sharing whatever the re-keyed+// links now read as.+func groupAfterRekey(members []string, oldID, newID string) []string {+ group := make([]string, 0, len(members)+1)+ group = append(group, newID)+ for _, id := range members {+ if id == oldID {+ continue+ }+ group = append(group, id)+ }+ return flattenComponents([][]string{group})+}++// carryRemediationLockLocked is carryRemediationLockSQL for the in-memory+// store. The caller holds m.mu.+func (m *MemoryStore) carryRemediationLockLocked(survivorID, donorID string, at time.Time, createRow bool) {+ survivorID = strings.TrimSpace(survivorID)+ donorID = strings.TrimSpace(donorID)+ if survivorID == "" || donorID == "" || survivorID == donorID {+ return+ }+ donor, donorFound := m.resourceOperatorState[donorID]+ survivor, survivorFound := m.resourceOperatorState[survivorID]+ if !survivorFound && !createRow {+ return+ }+ if next, changed := CarryRemediationLock(survivorID, survivor, survivorFound, donor, donorFound, at); changed {+ if m.resourceOperatorState == nil {+ m.resourceOperatorState = make(map[string]ResourceOperatorState)+ }+ m.resourceOperatorState[survivorID] = next+ }+}++// shareRemediationLockAmongLocked is shareRemediationLockAmongSQL for the+// in-memory store. The caller holds m.mu.+func (m *MemoryStore) shareRemediationLockAmongLocked(group []string, at time.Time) {+ if len(group) < 2 {+ return+ }+ states := make(map[string]ResourceOperatorState, len(group))+ for _, id := range group {+ if state, found := m.resourceOperatorState[id]; found {+ states[id] = state+ }+ }+ writes, _ := shareRemediationLock(group, states, at)+ for _, next := range writes {+ if m.resourceOperatorState == nil {+ m.resourceOperatorState = make(map[string]ResourceOperatorState)+ }+ m.resourceOperatorState[next.CanonicalID] = next+ }+}++// shareRemediationLockAcrossLinksLocked is shareRemediationLockAcrossLinksSQL+// for the in-memory store. The caller holds m.mu.+func (m *MemoryStore) shareRemediationLockAcrossLinksLocked(seeds []string, at time.Time) {+ for _, component := range linkComponents(m.livePairsLocked(), seeds) {+ m.shareRemediationLockAmongLocked(component, at)+ }+}++// livePairsLocked returns the in-memory store's live link pairs. The caller+// holds m.mu.+func (m *MemoryStore) livePairsLocked() [][2]string {+ live, _ := effectiveManualPairDecisions(m.links, m.exclusions)+ return linkPairs(live)+}++const remediationLockCarryMigration = "remediation-lock-carry-v1"++// migrateRemediationLockCarry carries the locks that identity changes dropped+// before the stores carried them. It runs once per store, so an operator who+// later lifts a lock is not overruled by a dormant row on every start:+//+// - a succession applied while its successor already had a row left the+// predecessor's row behind, shadowed, and with it the predecessor's lock;+// - a link recorded before a lock followed it left the lock on a member the+// registry folded away.+//+// Successions are carried first so a lock they restore then spreads across the+// successor's links. A failure is logged and retried on the next start rather+// than keeping the store from opening.+func (s *SQLiteResourceStore) migrateRemediationLockCarry() error {+ var applied int+ if err := s.db.QueryRow(`SELECT COUNT(*) FROM resource_store_migrations WHERE name = ?`, remediationLockCarryMigration).Scan(&applied); err != nil {+ return fmt.Errorf("read remediation lock carry marker: %w", err)+ }+ if applied > 0 {+ return nil+ }++ rows, err := s.db.Query(`SELECT old_canonical_id, new_canonical_id FROM canonical_id_successions`)+ if err != nil {+ return fmt.Errorf("query recorded canonical ID successions: %w", err)+ }+ var successions []CanonicalIDSuccession+ for rows.Next() {+ var succession CanonicalIDSuccession+ if err := rows.Scan(&succession.OldCanonicalID, &succession.NewCanonicalID); err != nil {+ _ = rows.Close()+ return fmt.Errorf("scan recorded canonical ID succession: %w", err)+ }+ successions = append(successions, succession)+ }+ rowsErr := rows.Err()+ if err := rows.Close(); err != nil {+ return fmt.Errorf("close recorded canonical ID successions: %w", err)+ }+ if rowsErr != nil {+ return fmt.Errorf("iterate recorded canonical ID successions: %w", rowsErr)+ }++ s.mu.Lock()+ defer s.mu.Unlock()+ tx, err := s.db.Begin()+ if err != nil {+ return fmt.Errorf("begin remediation lock carry: %w", err)+ }+ committed := false+ defer func() {+ if !committed {+ _ = tx.Rollback()+ }+ }()++ at := time.Now().UTC()+ var carried []carriedLock+ // Successions chain (a -> b, then b -> c), an intermediate row may have+ // moved on to its own successor, and a lock a succession restores can reach+ // a link whose far member heads another succession. A block therefore goes+ // to every ID its chain reaches that still has a row, and the two steps+ // repeat until a pass changes nothing. Every productive pass sets a block+ // that was unset on some ID, so the passes needed are bounded by the number+ // of IDs the successions and links name.+ livePairs, err := liveLinkPairsSQL(tx)+ if err != nil {+ return err+ }+ maxPasses := 2*(len(successions)+len(livePairs)) + 8+ converged := false+ for pass := 0; pass <= maxPasses; pass++ {+ changedAny := false+ for _, oldID := range recordedPredecessors(successions) {+ for _, descendant := range successionDescendants(successions, oldID) {+ changed, err := carryRemediationLockSQL(tx, descendant, oldID, at, false)+ if err != nil {+ return fmt.Errorf("carry remediation lock %q -> %q: %w", oldID, descendant, err)+ }+ if changed {+ changedAny = true+ carried = append(carried, carriedLock{to: descendant, from: oldID})+ }+ }+ }+ pairs, err := liveLinkPairsSQL(tx)+ if err != nil {+ return err+ }+ var seeds []string+ for _, pair := range pairs {+ seeds = append(seeds, pair[0], pair[1])+ }+ shared, err := shareRemediationLockAcrossLinksSQL(tx, seeds, at)+ if err != nil {+ return err+ }+ if len(shared) > 0 {+ changedAny = true+ carried = append(carried, shared...)+ }+ if !changedAny {+ converged = true+ break+ }+ }+ if !converged {+ // Cannot happen while every change sets a block that was unset, but a+ // store that did not finish must not record that it did.+ return fmt.Errorf("remediation lock carry did not converge")+ }++ if _, err := tx.Exec(`INSERT INTO resource_store_migrations (name) VALUES (?)`, remediationLockCarryMigration); err != nil {+ return fmt.Errorf("record remediation lock carry marker: %w", err)+ }+ if err := tx.Commit(); err != nil {+ return fmt.Errorf("commit remediation lock carry: %w", err)+ }+ committed = true+ logCarriedLocks("restoring a lock an earlier link or succession dropped", carried)+ return nil+}++// recordedPredecessors lists the predecessor IDs of successions, sorted and+// deduplicated.+func recordedPredecessors(successions []CanonicalIDSuccession) []string {+ seen := make(map[string]struct{}, len(successions))+ var out []string+ for _, succession := range successions {+ oldID := CanonicalResourceID(succession.OldCanonicalID)+ if oldID == "" {+ continue+ }+ if _, dup := seen[oldID]; dup {+ continue+ }+ seen[oldID] = struct{}{}+ out = append(out, oldID)+ }+ sort.Strings(out)+ return out+}++// successionDescendants returns every ID the successions lead from oldID to,+// directly or through further successions, in sorted order. oldID itself is+// never returned, and a cycle ends the walk instead of repeating it.+func successionDescendants(successions []CanonicalIDSuccession, oldID string) []string {+ successors := make(map[string][]string, len(successions))+ for _, succession := range successions {+ from, to := CanonicalResourceID(succession.OldCanonicalID), CanonicalResourceID(succession.NewCanonicalID)+ if from == "" || to == "" || from == to {+ continue+ }+ successors[from] = append(successors[from], to)+ }+ seen := map[string]struct{}{oldID: {}}+ var out []string+ for queue := []string{oldID}; len(queue) > 0; queue = queue[1:] {+ for _, next := range successors[queue[0]] {+ if _, dup := seen[next]; dup {+ continue+ }+ seen[next] = struct{}{}+ out = append(out, next)+ queue = append(queue, next)+ }+ }+ sort.Strings(out)+ return out+}diff --git a/internal/unifiedresources/resource_operator_state.go b/internal/unifiedresources/resource_operator_state.goindex ad929480e7..754896cc3b 100644--- a/internal/unifiedresources/resource_operator_state.go+++ b/internal/unifiedresources/resource_operator_state.go@@ -256,6 +256,43 @@ func (s ResourceOperatorState) BlocksRemediation() bool {return s.NeverAutoRemediate || s.LifecycleState == LifecycleStateRetired}+// RemediationLockCarriedBy attributes a row whose remediation lock was written+// by an identity change (a manual link, a canonical-ID succession) rather than+// by an operator edit, so the operator-state header does not credit a person+// with a lock they never set on this ID.+const RemediationLockCarriedBy = "system:identity-change"++// CarryRemediationLock returns the state a surviving resource must hold once+// an identity change has absorbed donor, and whether that differs from the+// survivor's current state. A remediation lock is the operator's "never touch+// this", so it follows the resource through a manual link fold or a+// canonical-ID succession instead of belonging to whichever ID happened to be+// keyed first: the lock is ORed onto the survivor, never lost to an unlocked+// row that happens to win the key.+//+// Only the remediation block crosses, as NeverAutoRemediate. Retirement also+// silences alerts and Patrol attention, which is a monitoring posture rather+// than a safety property; copying it would mute a survivor the operator still+// watches. Every other setting the survivor holds is returned untouched, and+// a survivor with no row gets a row carrying the block alone. The result is+// attributed to the identity change at `at`. An already-blocking survivor is+// returned as it was.+func CarryRemediationLock(survivorID string, survivor ResourceOperatorState, survivorFound bool, donor ResourceOperatorState, donorFound bool, at time.Time) (ResourceOperatorState, bool) {+ if !donorFound || !donor.BlocksRemediation() {+ return survivor, false+ }+ if survivorFound && survivor.BlocksRemediation() {+ return survivor, false+ }+ if !survivorFound {+ survivor = ResourceOperatorState{CanonicalID: survivorID}+ }+ survivor.NeverAutoRemediate = true+ survivor.SetAt = at.UTC()+ survivor.SetBy = RemediationLockCarriedBy+ return NormalizeResourceOperatorState(survivor), true+}+// IsInMaintenanceAt reports whether `now` falls within the configured// maintenance window. Returns false when no window is configured, when only// one of start/end is set (treated as no window), or when end <= start.diff --git a/internal/unifiedresources/store.go b/internal/unifiedresources/store.goindex 8f5d7fa85c..de9c296892 100644--- a/internal/unifiedresources/store.go+++ b/internal/unifiedresources/store.go@@ -477,6 +477,12 @@ func (s *SQLiteResourceStore) initSchema() error {metadata_json TEXT);+ -- One row per one-shot data migration that must not repeat on every start.+ CREATE TABLE IF NOT EXISTS resource_store_migrations (+ name TEXT PRIMARY KEY,+ applied_at DATETIME DEFAULT CURRENT_TIMESTAMP+ );+CREATE TABLE IF NOT EXISTS canonical_id_successions (old_canonical_id TEXT PRIMARY KEY,new_canonical_id TEXT NOT NULL,@@ -635,6 +641,11 @@ func (s *SQLiteResourceStore) initSchema() error {if err := s.migrateResourceOperatorStateSchema(); err != nil {return err}+ if err := s.migrateRemediationLockCarry(); err != nil {+ // A repair of earlier lock losses, not part of opening the store: keep+ // the store usable and retry on the next start.+ log.Printf("[WARN] unified_resources: remediation lock carry migration failed; it will retry on the next start: %v", err)+ }if err := s.migrateActionAuditRedaction(); err != nil {return err}@@ -1232,10 +1243,21 @@ func (s *SQLiteResourceStore) AddLink(link ResourceLink) error {`, a, b, link.PrimaryID, link.Reason, link.CreatedBy, link.CreatedAt); err != nil {return fmt.Errorf("upsert resource link %q<->%q: %w", a, b, err)}+ // The link declares both IDs one resource, and only the ID the registry+ // keeps is ever planned or dispatched against. Which side that is depends+ // on resource types the store cannot see (a guest outlives an agent+ // running inside it whichever way the link points), so a lock on any+ // member of the linked component is written to every member, in the+ // link's own transaction.+ carried, err := shareRemediationLockAcrossLinksSQL(tx, []string{a, b}, time.Now().UTC())+ if err != nil {+ return fmt.Errorf("share remediation lock across resource link %q<->%q: %w", a, b, err)+ }if err := tx.Commit(); err != nil {return fmt.Errorf("commit resource link %q<->%q: %w", a, b, err)}committed = true+ logCarriedLocks("a manual link", carried)return nil}@@ -1261,6 +1283,13 @@ func (s *SQLiteResourceStore) AddExclusion(exclusion ResourceExclusion) error {_ = tx.Rollback()}}()+ // Unlinking splits one resource into the members it joined. The lock the+ // merged resource carried stays with each of them: it was the operator's on+ // all of them, and only the surviving ID was ever able to hold it since.+ carried, err := shareRemediationLockAcrossLinksSQL(tx, []string{a, b}, time.Now().UTC())+ if err != nil {+ return fmt.Errorf("share remediation lock across resource link %q<->%q before unlinking: %w", a, b, err)+ }// An exclusion replaces the pair's earlier decision, so unlinking a linked// pair splits it (manual_link_decisions.go): its links in either order,// and an exclusion canonical-ID succession stored in the other order,@@ -1288,6 +1317,7 @@ func (s *SQLiteResourceStore) AddExclusion(exclusion ResourceExclusion) error {return fmt.Errorf("commit resource exclusion %q<->%q: %w", a, b, err)}committed = true+ logCarriedLocks("an unlink", carried)return nil}@@ -3610,6 +3640,9 @@ func (m *MemoryStore) AddLink(link ResourceLink) error {return sameManualPair(existing.ResourceA, existing.ResourceB, link.ResourceA, link.ResourceB)})m.links = append(m.links, link)+ // Every member of the linked component carries the block; see+ // SQLiteResourceStore.AddLink.+ m.shareRemediationLockAcrossLinksLocked([]string{link.ResourceA, link.ResourceB}, time.Now().UTC())return nil}@@ -3621,6 +3654,9 @@ func (m *MemoryStore) AddExclusion(exclusion ResourceExclusion) error {if exclusion.CreatedAt.IsZero() {exclusion.CreatedAt = time.Now().UTC()}+ // The merged resource's lock stays with each member the unlink splits off;+ // see SQLiteResourceStore.AddExclusion.+ m.shareRemediationLockAcrossLinksLocked([]string{exclusion.ResourceA, exclusion.ResourceB}, time.Now().UTC())// An exclusion replaces the pair's earlier decision, so unlinking a// linked pair splits it (manual_link_decisions.go).m.links = slices.DeleteFunc(m.links, func(link ResourceLink) bool {