Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Keep a remediation lock across links and ID successions

rcourtmanb5578136775b
diff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.md
index 710e2f90b5..6c1f8cc5a4 100644
--- a/docs/release-control/v6/internal/subsystems/unified-resources.md
+++ b/docs/release-control/v6/internal/subsystems/unified-resources.md
@@ -3565,6 +3565,86 @@ stable token. Finding-suppression integrations consume the same
`ResourceOperatorState` shape via the
`ResourceOperatorStateProvider` interface in `internal/ai`.
+A remediation lock follows the resource through the identity changes the
+stores can see. The planner, the action broker and the dispatch gate
+(`validateExecutionPolicy`) read the row keyed by the one canonical ID the
+registry lists, so a lock left on an ID the registry no longer lists would let
+Pulse act on a resource the operator told it never to touch. The members of a
+live manual-link component therefore share the block
+(`shareRemediationLockAcrossLinksSQL`, `CarryRemediationLock`). `AddLink` writes
+a block held by any member of the linked component to every member, in the
+link's own transaction, because the store cannot see resource types and so
+cannot know which member the registry keeps (a guest outlives an agent running
+inside it whichever way the link points); a chain of links is one component.
+`AddExclusion` does the same just before an unlink splits the pair, so each
+member the unlink leaves keeps the lock the merged resource carried.
+`ApplyCanonicalIDSuccessions` carries the predecessor's block onto a successor
+row that already exists, which would otherwise shadow it. The IDs the
+predecessor's and successor's links joined before the re-key stay one group:
+once the re-key has run (so a successor with no row still receives the
+predecessor's whole row, retirement, maintenance, notes and attribution
+included), the group shares the block even where re-keying a link onto the
+successor lets a newer exclusion supersede it. Successions apply hop by hop in
+the order they are declared; the stores do not reorder a batch, so a chain
+declared against its direction in one batch, or into an intermediate ID whose
+own succession was recorded earlier, carries one hop per succession. A
+succession applies once per predecessor in both stores, so a lock the operator
+lifts on the successor, by editing or clearing its row, is not brought back by
+a re-declared succession. Opening a store that has not run the one-shot
+upgrade carry (`migrateRemediationLockCarry`, recorded in
+`resource_store_migrations`) restores the locks that earlier links and
+already-recorded successions dropped, following a recorded chain to every ID
+that still has a row and repeating with the links until nothing changes; it
+records itself done only once it has converged, does not run again after that,
+so a lock lifted afterwards stays lifted, and a failure is logged and retried
+at the next start rather than keeping the store closed. Sharing after a
+succession batch waits until every predecessor's row in the batch has moved
+and follows each member through the batch's re-keys, so a block-only row made
+for one predecessor cannot take the place of another's whole row and an ID a
+later succession in the batch re-keyed away is not given a row. The link reads
+that settle a lock take only the columns that decide a pair and treat a time
+the driver cannot read as the zero time, so a malformed link row cannot stop a
+succession, a link or an unlink.
+
+What crosses is the block only, as `NeverAutoRemediate`, and `Retired` counts
+as a block: retirement also silences alerts and Patrol attention, which is
+monitoring posture rather than a safety property, so a retired donor does not
+retire the survivor. Every other setting an existing survivor row holds is left
+untouched, a member with no row gets a row carrying the block alone, and the
+carried row is attributed `system:identity-change` rather than to a person.
+The operator lifts the lock on the resource the registry lists, which is where
+the operator-state API resolves a folded ID to, and a restart, a rebuild or a
+re-declared succession does not put it back. The copies the sharing wrote on
+folded members cannot be reached through that API while the pair is linked, so
+they share again at the component's next link, unlink or succession. Unlinking
+the pair removes no carried lock from either side; once unlinked, each member
+can be edited or cleared on its own.
+
+The sharing is bounded by when it runs, and the contract does not claim more.
+A lock written to a linked ID after the link was recorded reaches the other
+members at the component's next link, unlink or succession, and reaches the
+survivor at once only when the ID resolves through the registry, as the operator-state API
+resolves a folded ID to its holder; a lock written while the registry cannot
+resolve the ID (a member absent from the current generation) stays on that ID
+until then. The store cannot see which links the registry declines to fold
+(availability-owned endpoints, a node and agent pair a newer split separated),
+so those share too, which can lock a host because a linked probe was retired.
+Re-recording an existing link shares again. Canonical ID changes that reach
+neither a link nor `ApplyCanonicalIDSuccessions` are not covered here: a
+physical disk re-keyed in place to a machine-scoped ID
+(`rekeyPhysicalDiskLocked` moves no operator state), and a host whose earlier
+ID never held an identity pin, which no pin-driven succession can name. Proof:
+`TestResourceRegistry_ManualLinkFoldKeepsRemediationLock`,
+`TestResourceRegistry_ManualLinkFoldLockLeavesSurvivorSettings`,
+`TestLinkedComponentSharesRemediationLockThroughLinksAndUnlinks`,
+`TestSQLiteStoreRestoresLocksEarlierIdentityChangesDropped` and
+`TestSQLiteStoreRestoresLocksAcrossRecordedSuccessionChains` in
+`internal/unifiedresources/registry_test.go`, and
+`TestCanonicalIDSuccessionCarriesRemediationLockOntoExistingSuccessorRow`,
+`TestCanonicalIDSuccessionLockReachesLinkedSurvivor` and
+`TestCanonicalIDSuccessionKeepsLinkedMembersTogether` in
+`internal/unifiedresources/canonical_id_succession_test.go`.
+
`actions.go` now owns the canonical action preflight and audit-normalization
contract. Action plans must carry dry-run availability, safety checks, and
verification steps through `preflight`, and `RecordActionAudit` plus
@@ -3897,7 +3977,10 @@ reported hostname or machine-identity changes, and separate connections with
the same hostname remain distinct. Legacy hostname-derived IDs may be exposed
as explicit `SupersededCanonicalIDs`; registry persistence may re-key owned
operator state and delete the retired pin, while ordinary aliases remain
-read-only lookup compatibility. Identity pins from an unrelated Pulse Agent
+read-only lookup compatibility. A successor row that already exists keeps
+its own settings, but on the succession's first application the predecessor's
+remediation lock is carried onto it (`CarryRemediationLock`) and across the
+successor's links rather than shadowed. Identity pins from an unrelated Pulse Agent
must not complete or merge a TrueNAS connection identity.
`canonicalIdentity.supersededIds` is preserved across backend projection and
@@ -6919,7 +7002,12 @@ links" below).
merged row's maintenance and monitoring mode, not an operator-state
row left under the folded ID, whenever that read state folds the
agent, or holds the saved agent beside the guest after a restart
- (above). History does
+ (above). The folded row's remediation lock is the one part of it that
+ does not stay behind: the store shares it across the link's component when
+ the link is recorded or unlinked, and once on upgrade for earlier links, so
+ the gates, which read the merged row's ID, still refuse (the remediation
+ lock note under the operator-state contract above, which also states its
+ limits). History does
not follow the fold: a history binding persists, and
`expandHistoryAliases` walks bindings in both directions, so binding a
folded ID would join the two journals for good. Folded IDs are never
diff --git a/internal/unifiedresources/canonical_id_succession.go b/internal/unifiedresources/canonical_id_succession.go
index a6e4ce4044..60ac83f43b 100644
--- a/internal/unifiedresources/canonical_id_succession.go
+++ b/internal/unifiedresources/canonical_id_succession.go
@@ -1,6 +1,9 @@
package unifiedresources
-import "fmt"
+import (
+ "fmt"
+ "time"
+)
// CanonicalIDSuccession records that the resource durably pinned under
// OldCanonicalID now mints NewCanonicalID. Canonical IDs hash the strongest
@@ -37,7 +40,12 @@ type canonicalIDSuccessor interface {
// and lets change-journal reads merge the old era without rewriting history.
// Re-keys never clobber rows already present under the new ID (those are
// fresher); a shadowed old row is left behind, which matches the
-// pre-succession orphaning behavior.
+// pre-succession orphaning behavior. The one thing a shadowed row still
+// contributes is its remediation lock: the operator's "never touch this" is
+// carried onto the surviving row (CarryRemediationLock) rather than lost to
+// whichever row happened to be keyed first. The carry happens with the
+// succession's first application, so an operator who later lifts the lock is
+// not overruled by the shadowed row on a re-declared succession.
func (s *SQLiteResourceStore) ApplyCanonicalIDSuccessions(successions []CanonicalIDSuccession) error {
if len(successions) == 0 {
return nil
@@ -75,6 +83,9 @@ func (s *SQLiteResourceStore) ApplyCanonicalIDSuccessions(successions []Canonica
}()
applied := make([]CanonicalIDSuccession, 0, len(pending))
+ var carried []carriedLock
+ var groups [][]string
+ rekeyed := make(map[string]string, len(pending))
for _, succession := range pending {
oldID := succession.OldCanonicalID
newID := succession.NewCanonicalID
@@ -90,6 +101,26 @@ func (s *SQLiteResourceStore) ApplyCanonicalIDSuccessions(successions []Canonica
applied = append(applied, succession)
continue
}
+ carriedAt := time.Now().UTC()
+ // The re-key below moves the predecessor's links onto the successor,
+ // where a newer exclusion naming either ID can make one read as
+ // unlinked. The IDs those links joined stay one group for the sharing
+ // after the re-key.
+ members, err := linkedMembersSQL(tx, []string{oldID, newID})
+ if err != nil {
+ return fmt.Errorf("read links of %q and %q: %w", oldID, newID, err)
+ }
+ // A successor row that already exists wins the re-key below, so the
+ // predecessor's remediation lock would be shadowed by it. Carry the
+ // block onto it first; with no successor row the re-key moves the
+ // predecessor's whole row and there is nothing to carry.
+ shadowed, err := carryRemediationLockSQL(tx, newID, oldID, carriedAt, false)
+ if err != nil {
+ return fmt.Errorf("carry remediation lock %q -> %q: %w", oldID, newID, err)
+ }
+ if shadowed {
+ carried = append(carried, carriedLock{to: newID, from: oldID})
+ }
for _, stmt := range []string{
`UPDATE OR IGNORE resource_operator_state SET canonical_id = ? WHERE canonical_id = ?`,
`UPDATE action_audits SET canonical_id = ? WHERE canonical_id = ?`,
@@ -108,13 +139,31 @@ func (s *SQLiteResourceStore) ApplyCanonicalIDSuccessions(successions []Canonica
if _, err := tx.Exec(`DELETE FROM resource_identities WHERE canonical_id = ?`, oldID); err != nil {
return fmt.Errorf("delete superseded identity pin %q: %w", oldID, err)
}
+ groups = append(groups, groupAfterRekey(members, oldID, newID))
+ rekeyed[oldID] = newID
applied = append(applied, succession)
}
+ // Whether a successor took its predecessor's whole row or kept its own, it
+ // may be linked into another resource the registry keeps instead; the lock
+ // has to reach whichever member survives. Sharing waits for the whole batch
+ // so a member that had no row is given the block alone only once every
+ // predecessor's whole row has had its move: a row made earlier would win
+ // the re-key of a later predecessor into the same successor.
+ sharedAt := time.Now().UTC()
+ for _, group := range groups {
+ shared, err := shareRemediationLockAmongSQL(tx, settledGroup(group, rekeyed), sharedAt)
+ if err != nil {
+ return fmt.Errorf("share remediation lock across linked resources %v: %w", group, err)
+ }
+ carried = append(carried, shared...)
+ }
+
if err := tx.Commit(); err != nil {
return fmt.Errorf("commit canonical ID succession: %w", err)
}
committed = true
+ logCarriedLocks("a canonical ID succession", carried)
s.identityPinMu.Lock()
s.identityPinFresh = false
@@ -181,17 +230,30 @@ func (m *MemoryStore) ApplyCanonicalIDSuccessions(successions []CanonicalIDSucce
m.mu.Lock()
defer m.mu.Unlock()
+ var groups [][]string
+ rekeyed := make(map[string]string, len(successions))
for _, succession := range successions {
oldID := CanonicalResourceID(succession.OldCanonicalID)
newID := CanonicalResourceID(succession.NewCanonicalID)
if oldID == "" || newID == "" || oldID == newID {
continue
}
+ // A succession applies once per predecessor, as SQLite's recorded
+ // successions do: a repeat neither re-keys nor re-carries.
+ if _, recorded := m.canonicalSuccessions[oldID]; recorded {
+ continue
+ }
+ carriedAt := time.Now().UTC()
+ // See the SQLite store: the IDs the links joined before the re-key stay
+ // one group for the sharing after it.
+ members := flattenComponents(linkComponents(m.livePairsLocked(), []string{oldID, newID}))
if state, ok := m.resourceOperatorState[oldID]; ok {
if _, taken := m.resourceOperatorState[newID]; !taken {
state.CanonicalID = newID
m.resourceOperatorState[newID] = state
delete(m.resourceOperatorState, oldID)
+ } else {
+ m.carryRemediationLockLocked(newID, oldID, carriedAt, false)
}
}
for i := range m.links {
@@ -217,9 +279,14 @@ func (m *MemoryStore) ApplyCanonicalIDSuccessions(successions []CanonicalIDSucce
if m.canonicalSuccessions == nil {
m.canonicalSuccessions = make(map[string]string)
}
- if _, exists := m.canonicalSuccessions[oldID]; !exists {
- m.canonicalSuccessions[oldID] = newID
- }
+ m.canonicalSuccessions[oldID] = newID
+ groups = append(groups, groupAfterRekey(members, oldID, newID))
+ rekeyed[oldID] = newID
+ }
+ // See the SQLite store: sharing waits for the whole batch.
+ sharedAt := time.Now().UTC()
+ for _, group := range groups {
+ m.shareRemediationLockAmongLocked(settledGroup(group, rekeyed), sharedAt)
}
return nil
}
diff --git a/internal/unifiedresources/canonical_id_succession_test.go b/internal/unifiedresources/canonical_id_succession_test.go
index ae4e31db76..1b09509c5f 100644
--- a/internal/unifiedresources/canonical_id_succession_test.go
+++ b/internal/unifiedresources/canonical_id_succession_test.go
@@ -1,6 +1,7 @@
package unifiedresources
import (
+ "fmt"
"testing"
"time"
)
@@ -132,3 +133,373 @@ func TestSQLiteCanonicalIDSuccessionKeepsExistingTargetRow(t *testing.T) {
t.Fatalf("succession overwrote the fresher successor row: %+v", state)
}
}
+
+// The successor's row wins over the superseded one (it is fresher), but a
+// remediation lock is the operator's "never touch this" and must not be shed
+// by whichever row happened to be written second. The successor keeps every
+// setting of its own and gains only the block.
+func TestCanonicalIDSuccessionCarriesRemediationLockOntoExistingSuccessorRow(t *testing.T) {
+ stores := map[string]func(t *testing.T) ResourceStore{
+ "memory": func(t *testing.T) ResourceStore { return NewMemoryStore() },
+ "sqlite": func(t *testing.T) ResourceStore {
+ store, err := NewSQLiteResourceStore(t.TempDir(), "default")
+ if err != nil {
+ t.Fatalf("NewSQLiteResourceStore: %v", err)
+ }
+ t.Cleanup(func() { _ = store.Close() })
+ return store
+ },
+ }
+ locks := map[string]ResourceOperatorState{
+ "never auto-remediate": {NeverAutoRemediate: true},
+ "retired": {LifecycleState: LifecycleStateRetired},
+ }
+ for storeName, newStore := range stores {
+ for lockName, lock := range locks {
+ t.Run(storeName+"/"+lockName, func(t *testing.T) {
+ store := newStore(t)
+ succeeder, ok := store.(canonicalIDSuccessor)
+ if !ok {
+ t.Fatalf("%T cannot apply canonical ID successions", store)
+ }
+ oldID := buildHashID(ResourceTypeAgent, "cluster:prod-swarm:vault")
+ newID := buildHashID(ResourceTypeAgent, "cluster:prod-swarm:vault.a")
+ now := time.Now().UTC()
+
+ predecessor := lock
+ predecessor.CanonicalID = oldID
+ predecessor.SetAt = now.Add(-72 * time.Hour)
+ predecessor.SetBy = "operator"
+ if err := store.SetResourceOperatorState(predecessor); err != nil {
+ t.Fatalf("seed predecessor: %v", err)
+ }
+ if err := store.SetResourceOperatorState(ResourceOperatorState{
+ CanonicalID: newID,
+ MonitoringMode: MonitoringModeExpectedOffline,
+ Criticality: CriticalityHigh,
+ Note: "set under the new ID before the era change was declared",
+ SetAt: now.Add(-time.Hour),
+ SetBy: "operator",
+ }); err != nil {
+ t.Fatalf("seed successor: %v", err)
+ }
+
+ if err := succeeder.ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{{OldCanonicalID: oldID, NewCanonicalID: newID}}); err != nil {
+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)
+ }
+
+ got, found, err := store.GetResourceOperatorState(newID)
+ if err != nil || !found {
+ t.Fatalf("successor state missing: found=%v err=%v", found, err)
+ }
+ if !got.BlocksRemediation() {
+ t.Fatalf("the successor's unlocked row shadowed the predecessor's lock: %+v", got)
+ }
+ if got.MonitoringMode != MonitoringModeExpectedOffline || got.Criticality != CriticalityHigh ||
+ got.Note != "set under the new ID before the era change was declared" || got.LifecycleState != LifecycleStateActive {
+ t.Fatalf("carrying the lock rewrote the successor's own settings: %+v", got)
+ }
+
+ // The block is the operator's to lift on the successor. Declaring
+ // the same succession again (every steady-state rebuild does) must
+ // not bring it back.
+ got.NeverAutoRemediate = false
+ if err := store.SetResourceOperatorState(got); err != nil {
+ t.Fatalf("operator lifts the lock: %v", err)
+ }
+ if err := succeeder.ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{{OldCanonicalID: oldID, NewCanonicalID: newID}}); err != nil {
+ t.Fatalf("ApplyCanonicalIDSuccessions (repeat): %v", err)
+ }
+ if again, _, err := store.GetResourceOperatorState(newID); err != nil || again.BlocksRemediation() {
+ t.Fatalf("a repeated succession re-locked the successor the operator cleared: err=%v state=%+v", err, again)
+ }
+
+ // Clearing the row outright is the other way to lift it, and the
+ // repeat must not move the shadowed predecessor's row back in.
+ if err := store.ClearResourceOperatorState(newID); err != nil {
+ t.Fatalf("operator clears the successor: %v", err)
+ }
+ if err := succeeder.ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{{OldCanonicalID: oldID, NewCanonicalID: newID}}); err != nil {
+ t.Fatalf("ApplyCanonicalIDSuccessions (repeat after clear): %v", err)
+ }
+ if again, found, err := store.GetResourceOperatorState(newID); err != nil || found {
+ t.Fatalf("a repeated succession restored a row the operator cleared: found=%v err=%v state=%+v", found, err, again)
+ }
+ })
+ }
+ }
+}
+
+// The registry keeps one member of a manual link, and a succession re-keys the
+// link onto the successor. The successor's lock (whether it kept its own row
+// or took the predecessor's whole row) has to reach the member that survives.
+func TestCanonicalIDSuccessionLockReachesLinkedSurvivor(t *testing.T) {
+ stores := map[string]func(t *testing.T) ResourceStore{
+ "memory": func(t *testing.T) ResourceStore { return NewMemoryStore() },
+ "sqlite": func(t *testing.T) ResourceStore {
+ store, err := NewSQLiteResourceStore(t.TempDir(), "default")
+ if err != nil {
+ t.Fatalf("NewSQLiteResourceStore: %v", err)
+ }
+ t.Cleanup(func() { _ = store.Close() })
+ return store
+ },
+ }
+ for name, newStore := range stores {
+ for _, successorHasRow := range []bool{false, true} {
+ t.Run(fmt.Sprintf("%s/successor row %v", name, successorHasRow), func(t *testing.T) {
+ store := newStore(t)
+ succeeder, ok := store.(canonicalIDSuccessor)
+ if !ok {
+ t.Fatalf("%T cannot apply canonical ID successions", store)
+ }
+ const guestID = "vm-linked-survivor"
+ oldID := buildHashID(ResourceTypeAgent, "linked:old-era")
+ newID := buildHashID(ResourceTypeAgent, "linked:new-era")
+ now := time.Now().UTC()
+
+ // The link names the superseded ID, so the succession re-keys it.
+ if err := store.AddLink(ResourceLink{ResourceA: guestID, ResourceB: oldID, PrimaryID: guestID}); err != nil {
+ t.Fatalf("link: %v", err)
+ }
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: oldID, NeverAutoRemediate: true, SetAt: now}); err != nil {
+ t.Fatalf("lock predecessor: %v", err)
+ }
+ if successorHasRow {
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: newID, Note: "unlocked", SetAt: now}); err != nil {
+ t.Fatalf("seed successor: %v", err)
+ }
+ }
+ if err := succeeder.ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{{OldCanonicalID: oldID, NewCanonicalID: newID}}); err != nil {
+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)
+ }
+
+ for _, id := range []string{newID, guestID} {
+ state, found, err := store.GetResourceOperatorState(id)
+ if err != nil || !found || !state.BlocksRemediation() {
+ t.Fatalf("%s lost the lock across the succession: found=%v err=%v state=%+v", id, found, err, state)
+ }
+ }
+ })
+ }
+ }
+}
+
+// A succession keeps the IDs its links joined together. The lock follows a
+// chain hop by hop as each succession applies, a link that a newer exclusion
+// supersedes once it is re-keyed still hands its lock to the member it leaves
+// behind (on either side of the succession), and a predecessor linked to its
+// own successor still has its whole row move across.
+func TestCanonicalIDSuccessionKeepsLinkedMembersTogether(t *testing.T) {
+ stores := map[string]func(t *testing.T) ResourceStore{
+ "memory": func(t *testing.T) ResourceStore { return NewMemoryStore() },
+ "sqlite": func(t *testing.T) ResourceStore {
+ store, err := NewSQLiteResourceStore(t.TempDir(), "default")
+ if err != nil {
+ t.Fatalf("NewSQLiteResourceStore: %v", err)
+ }
+ t.Cleanup(func() { _ = store.Close() })
+ return store
+ },
+ }
+ now := time.Now().UTC()
+ locked := func(t *testing.T, store ResourceStore, id string) bool {
+ t.Helper()
+ state, found, err := store.GetResourceOperatorState(id)
+ if err != nil {
+ t.Fatalf("read operator state of %s: %v", id, err)
+ }
+ return found && state.BlocksRemediation()
+ }
+ apply := func(t *testing.T, store ResourceStore, oldID, newID string) {
+ t.Helper()
+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{{OldCanonicalID: oldID, NewCanonicalID: newID}}); err != nil {
+ t.Fatalf("ApplyCanonicalIDSuccessions %s -> %s: %v", oldID, newID, err)
+ }
+ }
+ for name, newStore := range stores {
+ t.Run(name+"/chain across two batches", func(t *testing.T) {
+ store := newStore(t)
+ for id, state := range map[string]ResourceOperatorState{
+ "hop-a": {NeverAutoRemediate: true},
+ "hop-b": {Note: "unlocked"},
+ "hop-c": {Note: "unlocked"},
+ } {
+ state.CanonicalID, state.SetAt = id, now
+ if err := store.SetResourceOperatorState(state); err != nil {
+ t.Fatalf("seed %s: %v", id, err)
+ }
+ }
+ apply(t, store, "hop-a", "hop-b")
+ apply(t, store, "hop-b", "hop-c")
+ for _, id := range []string{"hop-b", "hop-c"} {
+ if !locked(t, store, id) {
+ t.Fatalf("%s did not get the lock of the chain's first predecessor", id)
+ }
+ }
+ })
+ t.Run(name+"/predecessor's link superseded once re-keyed", func(t *testing.T) {
+ store := newStore(t)
+ if err := store.AddLink(ResourceLink{ResourceA: "split-a", ResourceB: "split-b", PrimaryID: "split-a", CreatedAt: now.Add(-2 * time.Hour)}); err != nil {
+ t.Fatalf("link: %v", err)
+ }
+ // An exclusion naming the successor, newer than the link: once the
+ // link is re-keyed onto the successor this exclusion supersedes it.
+ if err := store.AddExclusion(ResourceExclusion{ResourceA: "split-a2", ResourceB: "split-b", CreatedAt: now.Add(-time.Hour)}); err != nil {
+ t.Fatalf("exclude: %v", err)
+ }
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "split-a", NeverAutoRemediate: true, SetAt: now}); err != nil {
+ t.Fatalf("operator locks the merged resource: %v", err)
+ }
+ apply(t, store, "split-a", "split-a2")
+ for _, id := range []string{"split-a2", "split-b"} {
+ if !locked(t, store, id) {
+ t.Fatalf("%s lost the lock when the succession split the link", id)
+ }
+ }
+ })
+ t.Run(name+"/successor's link superseded once re-keyed", func(t *testing.T) {
+ store := newStore(t)
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "side-b", Note: "unlocked", SetAt: now}); err != nil {
+ t.Fatalf("seed successor: %v", err)
+ }
+ if err := store.AddLink(ResourceLink{ResourceA: "side-b", ResourceB: "side-x", PrimaryID: "side-x", CreatedAt: now.Add(-2 * time.Hour)}); err != nil {
+ t.Fatalf("link: %v", err)
+ }
+ if err := store.AddExclusion(ResourceExclusion{ResourceA: "side-a", ResourceB: "side-x", CreatedAt: now.Add(-time.Hour)}); err != nil {
+ t.Fatalf("exclude: %v", err)
+ }
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "side-x", NeverAutoRemediate: true, SetAt: now}); err != nil {
+ t.Fatalf("operator locks the merged resource: %v", err)
+ }
+ apply(t, store, "side-a", "side-b")
+ for _, id := range []string{"side-b", "side-x"} {
+ if !locked(t, store, id) {
+ t.Fatalf("%s lost the lock when the succession split the successor's link", id)
+ }
+ }
+ })
+ t.Run(name+"/star batch: sharing waits until every predecessor's row has moved", func(t *testing.T) {
+ store := newStore(t)
+ if err := store.AddLink(ResourceLink{ResourceA: "star-b", ResourceB: "star-x", PrimaryID: "star-x"}); err != nil {
+ t.Fatalf("link: %v", err)
+ }
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "star-x", NeverAutoRemediate: true, SetAt: now}); err != nil {
+ t.Fatalf("operator locks the linked resource: %v", err)
+ }
+ if err := store.SetResourceOperatorState(ResourceOperatorState{
+ CanonicalID: "star-old2", Note: "keep", Criticality: CriticalityHigh, SetAt: now.Add(-time.Hour), SetBy: "alice",
+ }); err != nil {
+ t.Fatalf("seed second predecessor: %v", err)
+ }
+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{
+ {OldCanonicalID: "star-old1", NewCanonicalID: "star-b"},
+ {OldCanonicalID: "star-old2", NewCanonicalID: "star-b"},
+ }); err != nil {
+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)
+ }
+ got, found, err := store.GetResourceOperatorState("star-b")
+ if err != nil || !found {
+ t.Fatalf("successor row missing: found=%v err=%v", found, err)
+ }
+ if got.Note != "keep" || got.Criticality != CriticalityHigh {
+ t.Fatalf("a block-only row made for the first predecessor shadowed the second one's whole row: %+v", got)
+ }
+ if !got.BlocksRemediation() {
+ t.Fatalf("the successor did not get the lock of the resource its links joined: %+v", got)
+ }
+ })
+ t.Run(name+"/forward chain in one batch, link superseded once re-keyed", func(t *testing.T) {
+ store := newStore(t)
+ if err := store.AddLink(ResourceLink{ResourceA: "fwd-a", ResourceB: "fwd-x", PrimaryID: "fwd-x", CreatedAt: now.Add(-2 * time.Hour)}); err != nil {
+ t.Fatalf("link: %v", err)
+ }
+ if err := store.AddExclusion(ResourceExclusion{ResourceA: "fwd-b", ResourceB: "fwd-x", CreatedAt: now.Add(-time.Hour)}); err != nil {
+ t.Fatalf("exclude: %v", err)
+ }
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "fwd-x", NeverAutoRemediate: true, SetAt: now}); err != nil {
+ t.Fatalf("operator locks the linked resource: %v", err)
+ }
+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{
+ {OldCanonicalID: "fwd-a", NewCanonicalID: "fwd-b"},
+ {OldCanonicalID: "fwd-b", NewCanonicalID: "fwd-c"},
+ }); err != nil {
+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)
+ }
+ for _, id := range []string{"fwd-c", "fwd-x"} {
+ if !locked(t, store, id) {
+ t.Fatalf("%s lost the lock across a chain declared in one batch", id)
+ }
+ }
+ if state, found, err := store.GetResourceOperatorState("fwd-b"); err != nil || found {
+ t.Fatalf("the re-keyed intermediate ID was given a row: found=%v err=%v state=%+v", found, err, state)
+ }
+ })
+ t.Run(name+"/predecessor linked to its successor keeps its whole row", func(t *testing.T) {
+ store := newStore(t)
+ if err := store.AddLink(ResourceLink{ResourceA: "own-a", ResourceB: "own-b", PrimaryID: "own-a"}); err != nil {
+ t.Fatalf("link: %v", err)
+ }
+ if err := store.SetResourceOperatorState(ResourceOperatorState{
+ CanonicalID: "own-a", LifecycleState: LifecycleStateRetired, Note: "keep", SetAt: now.Add(-time.Hour), SetBy: "alice",
+ }); err != nil {
+ t.Fatalf("seed predecessor: %v", err)
+ }
+ apply(t, store, "own-a", "own-b")
+ got, found, err := store.GetResourceOperatorState("own-b")
+ if err != nil || !found {
+ t.Fatalf("successor row missing: found=%v err=%v", found, err)
+ }
+ if got.LifecycleState != LifecycleStateRetired || got.Note != "keep" || got.SetBy != "alice" {
+ t.Fatalf("the predecessor's whole row did not move across: %+v", got)
+ }
+ })
+ }
+}
+
+// Link rows may leave the optional reason and author empty (NULL) or hold a
+// time the driver cannot read. Reading links to settle a lock must not stop an
+// unrelated succession, or the upgrade carry, from completing.
+func TestSQLiteSuccessionAndUpgradeCarryIgnoreMalformedLinkRows(t *testing.T) {
+ dir := t.TempDir()
+ store, err := NewSQLiteResourceStore(dir, "default")
+ if err != nil {
+ t.Fatalf("NewSQLiteResourceStore: %v", err)
+ }
+ // A row without reason or author, one with an integer where the driver
+ // expects a time, and one with no time at all.
+ for _, row := range []string{
+ `INSERT INTO resource_links (resource_a, resource_b, primary_id) VALUES ('unrelated', 'unrelated', 'unrelated')`,
+ `INSERT INTO resource_links (resource_a, resource_b, primary_id, reason, created_by, created_at) VALUES ('z1', 'z2', 'z1', '', '', 123)`,
+ `INSERT INTO resource_links (resource_a, resource_b, primary_id, created_at) VALUES ('z3', 'z4', 'z3', NULL)`,
+ } {
+ if _, err := store.db.Exec(row); err != nil {
+ t.Fatalf("seed malformed link row: %v", err)
+ }
+ }
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "null-old", NeverAutoRemediate: true, Note: "moves", SetAt: time.Now().UTC()}); err != nil {
+ t.Fatalf("seed predecessor: %v", err)
+ }
+ if err := store.ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{{OldCanonicalID: "null-old", NewCanonicalID: "null-new"}}); err != nil {
+ t.Fatalf("succession stopped by an unrelated link row: %v", err)
+ }
+ if got, found, err := store.GetResourceOperatorState("null-new"); err != nil || !found || got.Note != "moves" {
+ t.Fatalf("predecessor's row did not move: found=%v err=%v state=%+v", found, err, got)
+ }
+ if _, err := store.db.Exec(`DELETE FROM resource_store_migrations`); err != nil {
+ t.Fatalf("forget the carry marker: %v", err)
+ }
+ if err := store.Close(); err != nil {
+ t.Fatalf("close: %v", err)
+ }
+
+ store, err = NewSQLiteResourceStore(dir, "default")
+ if err != nil {
+ t.Fatalf("reopen: %v", err)
+ }
+ defer store.Close()
+ var recorded int
+ if err := store.db.QueryRow(`SELECT COUNT(*) FROM resource_store_migrations`).Scan(&recorded); err != nil || recorded != 1 {
+ t.Fatalf("the upgrade carry did not complete past the unrelated link row: markers=%d err=%v", recorded, err)
+ }
+}
diff --git a/internal/unifiedresources/registry_test.go b/internal/unifiedresources/registry_test.go
index 8fcdf65ac0..a7118ef715 100644
--- a/internal/unifiedresources/registry_test.go
+++ b/internal/unifiedresources/registry_test.go
@@ -8032,3 +8032,432 @@ func proxmoxNodeSensorSetupOutdated(t *testing.T, resources []Resource, nodeName
}
return *found.Proxmox.SensorSetupOutdated
}
+
+// An operator's remediation lock is a safety property of the thing they
+// locked, not of one canonical ID. Linking a locked resource into another
+// folds it out of the registry, and the surviving resource is the only ID the
+// planner and the dispatch gate ever read, so the lock has to be readable
+// there. Lock state is judged the way the gates judge it: the stored row of
+// the ID the registry lists.
+func TestResourceRegistry_ManualLinkFoldKeepsRemediationLock(t *testing.T) {
+ now := time.Now().UTC()
+ const (
+ guestID = "vm-cluster-a-node-1-501"
+ agentID = "agent-inside-vm-501"
+ )
+ ingest := func(rr *ResourceRegistry) {
+ rr.IngestResources([]Resource{
+ {
+ ID: guestID, Type: ResourceTypeVM, Name: "vm-501", Status: StatusOnline, LastSeen: now,
+ Sources: []DataSource{SourceProxmox},
+ SourceStatus: map[DataSource]SourceStatus{SourceProxmox: {Status: "online", LastSeen: now}},
+ Proxmox: &ProxmoxData{SourceID: "cluster-a-node-1-501", VMID: 501},
+ },
+ {
+ ID: agentID, Type: ResourceTypeAgent, Name: "vm-501", Status: StatusOnline, LastSeen: now,
+ Sources: []DataSource{SourceAgent},
+ SourceStatus: map[DataSource]SourceStatus{SourceAgent: {Status: "online", LastSeen: now}},
+ Agent: &AgentData{AgentID: agentID},
+ },
+ })
+ }
+ stores := map[string]func(t *testing.T) ResourceStore{
+ "memory": func(t *testing.T) ResourceStore { return NewMemoryStore() },
+ "sqlite": func(t *testing.T) ResourceStore {
+ store, err := NewSQLiteResourceStore(t.TempDir(), "default")
+ if err != nil {
+ t.Fatalf("NewSQLiteResourceStore: %v", err)
+ }
+ t.Cleanup(func() { _ = store.Close() })
+ return store
+ },
+ }
+ locks := map[string]ResourceOperatorState{
+ "never auto-remediate": {NeverAutoRemediate: true},
+ "retired": {LifecycleState: LifecycleStateRetired},
+ }
+ for storeName, newStore := range stores {
+ for lockName, lock := range locks {
+ for _, tc := range []struct {
+ name string
+ locked string
+ // primary is the side the link names as its primary. The registry
+ // keeps a guest over an agent running inside it whichever way the
+ // link points, so a lock on the guest-folded agent has to survive
+ // a link that names the agent the primary and a link that names the
+ // guest.
+ primary string
+ }{
+ {name: "locked agent folded into the guest, agent named primary", locked: agentID, primary: agentID},
+ {name: "locked agent folded into the guest, guest named primary", locked: agentID, primary: guestID},
+ {name: "control: locked guest absorbing the agent", locked: guestID, primary: agentID},
+ } {
+ t.Run(storeName+"/"+lockName+"/"+tc.name, func(t *testing.T) {
+ store := newStore(t)
+ state := lock
+ state.CanonicalID = tc.locked
+ state.SetAt = now.Add(-time.Hour)
+ state.SetBy = "operator"
+ if err := store.SetResourceOperatorState(state); err != nil {
+ t.Fatalf("lock %s: %v", tc.locked, err)
+ }
+ if err := store.AddLink(ResourceLink{ResourceA: guestID, ResourceB: agentID, PrimaryID: tc.primary}); err != nil {
+ t.Fatalf("add link: %v", err)
+ }
+
+ rr := NewRegistry(store)
+ ingest(rr)
+ survivors := rr.List()
+ if len(survivors) != 1 {
+ t.Fatalf("link did not fold the pair into one resource: %d resources", len(survivors))
+ }
+ survivor := survivors[0].ID
+
+ got, found, err := store.GetResourceOperatorState(survivor)
+ if err != nil {
+ t.Fatalf("read operator state of survivor %s: %v", survivor, err)
+ }
+ if !found || !got.BlocksRemediation() {
+ t.Fatalf("surviving resource %s lost the lock on %s: found=%v state=%+v", survivor, tc.locked, found, got)
+ }
+ if got.SuppressesAllAttention() != (lockName == "retired" && tc.locked == survivor) {
+ t.Fatalf("lock on %s changed the survivor's alert attention: %+v", tc.locked, got)
+ }
+ })
+ }
+ }
+ }
+}
+
+// Carrying the lock must not rewrite anything else the survivor's operator
+// said: its note, priority and monitoring posture stay as they were.
+func TestResourceRegistry_ManualLinkFoldLockLeavesSurvivorSettings(t *testing.T) {
+ now := time.Now().UTC()
+ store := NewMemoryStore()
+ const (
+ keeperID = "vm-cluster-a-node-1-502"
+ foldedID = "agent-inside-vm-502"
+ )
+ if err := store.SetResourceOperatorState(ResourceOperatorState{
+ CanonicalID: keeperID,
+ MonitoringMode: MonitoringModeExpectedOffline,
+ Criticality: CriticalityHigh,
+ Note: "database host",
+ SetAt: now.Add(-time.Hour),
+ SetBy: "alice",
+ }); err != nil {
+ t.Fatalf("seed survivor state: %v", err)
+ }
+ if err := store.SetResourceOperatorState(ResourceOperatorState{
+ CanonicalID: foldedID, NeverAutoRemediate: true, SetAt: now.Add(-2 * time.Hour), SetBy: "bob",
+ }); err != nil {
+ t.Fatalf("seed folded state: %v", err)
+ }
+ if err := store.AddLink(ResourceLink{ResourceA: keeperID, ResourceB: foldedID, PrimaryID: keeperID}); err != nil {
+ t.Fatalf("add link: %v", err)
+ }
+
+ got, found, err := store.GetResourceOperatorState(keeperID)
+ if err != nil || !found {
+ t.Fatalf("survivor state missing: found=%v err=%v", found, err)
+ }
+ if !got.NeverAutoRemediate {
+ t.Fatalf("survivor did not take the folded resource's lock: %+v", got)
+ }
+ if got.MonitoringMode != MonitoringModeExpectedOffline || got.Criticality != CriticalityHigh || got.Note != "database host" || got.LifecycleState != LifecycleStateActive {
+ t.Fatalf("carrying the lock rewrote the survivor's own settings: %+v", got)
+ }
+ if got.SetBy != RemediationLockCarriedBy {
+ t.Fatalf("carried lock is attributed to %q, not to the identity change", got.SetBy)
+ }
+
+ // Unlinking splits the pair again; the lock was the operator's on both, so
+ // neither side loses it.
+ if err := store.AddExclusion(ResourceExclusion{ResourceA: keeperID, ResourceB: foldedID}); err != nil {
+ t.Fatalf("unlink: %v", err)
+ }
+ for _, id := range []string{keeperID, foldedID} {
+ if state, found, err := store.GetResourceOperatorState(id); err != nil || !found || !state.NeverAutoRemediate {
+ t.Fatalf("unlinking dropped the lock on %s: found=%v err=%v state=%+v", id, found, err, state)
+ }
+ }
+}
+
+// Links recorded before locks travelled with them already folded a locked
+// resource out from under its lock, and a succession that landed on an existing
+// successor row left its predecessor's lock behind. Opening a store that has
+// not yet run the carry restores those once; an operator who then lifts the
+// survivor's lock is not overruled by the dormant row on every later start.
+func TestSQLiteStoreRestoresLocksEarlierIdentityChangesDropped(t *testing.T) {
+ dir := t.TempDir()
+ const (
+ lockedID = "agent-legacy-locked"
+ survivorID = "vm-legacy-survivor"
+ chainEndID = "vm-legacy-chain-end"
+ unlinkedA = "agent-legacy-unlinked-locked"
+ unlinkedB = "vm-legacy-unlinked-open"
+ oldEraID = "agent-legacy-old-era"
+ newEraID = "agent-legacy-new-era"
+ linkedAt = "2026-09-01T10:00:00Z"
+ unlinkedAt = "2026-09-02T10:00:00Z"
+ operator = "operator"
+ legacyReason = "recorded before locks travelled"
+ )
+ open := func() *SQLiteResourceStore {
+ t.Helper()
+ store, err := NewSQLiteResourceStore(dir, "default")
+ if err != nil {
+ t.Fatalf("NewSQLiteResourceStore: %v", err)
+ }
+ return store
+ }
+ now := time.Now().UTC()
+
+ store := open()
+ for _, id := range []string{lockedID, unlinkedA, oldEraID} {
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: id, NeverAutoRemediate: true, SetAt: now.Add(-48 * time.Hour), SetBy: operator}); err != nil {
+ t.Fatalf("lock %s: %v", id, err)
+ }
+ }
+ // A successor row written before the succession was declared, as the
+ // succession that shadowed the predecessor's lock found it.
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: newEraID, Note: "set under the new ID", SetAt: now.Add(-24 * time.Hour), SetBy: operator}); err != nil {
+ t.Fatalf("seed successor: %v", err)
+ }
+ // Rows exactly as a store that did not carry locks wrote them: a live link,
+ // a chain extension of it, one a later unlink superseded (its exclusion is
+ // newer), and a succession already recorded with the predecessor's row left
+ // behind.
+ for _, row := range []struct{ a, b, primary, at string }{
+ {lockedID, survivorID, survivorID, linkedAt},
+ {survivorID, chainEndID, chainEndID, linkedAt},
+ {unlinkedA, unlinkedB, unlinkedB, linkedAt},
+ } {
+ if _, err := store.db.Exec(`INSERT INTO resource_links (resource_a, resource_b, primary_id, reason, created_by, created_at) VALUES (?, ?, ?, ?, ?, ?)`,
+ row.a, row.b, row.primary, legacyReason, operator, row.at); err != nil {
+ t.Fatalf("seed legacy link: %v", err)
+ }
+ }
+ if _, err := store.db.Exec(`INSERT INTO resource_exclusions (resource_a, resource_b, reason, created_by, created_at) VALUES (?, ?, ?, ?, ?)`,
+ unlinkedA, unlinkedB, "unlinked", operator, unlinkedAt); err != nil {
+ t.Fatalf("seed legacy exclusion: %v", err)
+ }
+ if _, err := store.db.Exec(`INSERT INTO canonical_id_successions (old_canonical_id, new_canonical_id) VALUES (?, ?)`, oldEraID, newEraID); err != nil {
+ t.Fatalf("seed recorded succession: %v", err)
+ }
+ // This store has never run the carry.
+ if _, err := store.db.Exec(`DELETE FROM resource_store_migrations`); err != nil {
+ t.Fatalf("forget the carry marker: %v", err)
+ }
+ if err := store.Close(); err != nil {
+ t.Fatalf("close: %v", err)
+ }
+
+ store = open()
+ for _, id := range []string{survivorID, chainEndID, newEraID} {
+ got, found, err := store.GetResourceOperatorState(id)
+ if err != nil || !found || !got.BlocksRemediation() {
+ t.Fatalf("%s did not get the lock an earlier identity change dropped: found=%v err=%v state=%+v", id, found, err, got)
+ }
+ }
+ if got, _, _ := store.GetResourceOperatorState(newEraID); got.Note != "set under the new ID" {
+ t.Fatalf("restoring the successor's lock rewrote its own settings: %+v", got)
+ }
+ if other, found, err := store.GetResourceOperatorState(unlinkedB); err != nil || found {
+ t.Fatalf("a pair the operator unlinked was carried: found=%v err=%v state=%+v", found, err, other)
+ }
+
+ for _, id := range []string{survivorID, newEraID} {
+ if err := store.ClearResourceOperatorState(id); err != nil {
+ t.Fatalf("operator clears %s: %v", id, err)
+ }
+ }
+ if err := store.Close(); err != nil {
+ t.Fatalf("close: %v", err)
+ }
+ store = open()
+ defer store.Close()
+ for _, id := range []string{survivorID, newEraID} {
+ if got, found, err := store.GetResourceOperatorState(id); err != nil || found {
+ t.Fatalf("a restart re-locked %s, which the operator cleared: found=%v err=%v state=%+v", id, found, err, got)
+ }
+ }
+}
+
+// The registry keeps one member of a link and acts only on that ID, so a lock
+// has to be on every member of the linked component, not just the two IDs of
+// the newest link, and it has to stay on each member the unlink splits off.
+func TestLinkedComponentSharesRemediationLockThroughLinksAndUnlinks(t *testing.T) {
+ stores := map[string]func(t *testing.T) ResourceStore{
+ "memory": func(t *testing.T) ResourceStore { return NewMemoryStore() },
+ "sqlite": func(t *testing.T) ResourceStore {
+ store, err := NewSQLiteResourceStore(t.TempDir(), "default")
+ if err != nil {
+ t.Fatalf("NewSQLiteResourceStore: %v", err)
+ }
+ t.Cleanup(func() { _ = store.Close() })
+ return store
+ },
+ }
+ locked := func(t *testing.T, store ResourceStore, id string) bool {
+ t.Helper()
+ state, found, err := store.GetResourceOperatorState(id)
+ if err != nil {
+ t.Fatalf("read operator state of %s: %v", id, err)
+ }
+ return found && state.BlocksRemediation()
+ }
+ now := time.Now().UTC()
+ for name, newStore := range stores {
+ t.Run(name+"/chain", func(t *testing.T) {
+ store := newStore(t)
+ if err := store.AddLink(ResourceLink{ResourceA: "host-a", ResourceB: "host-b", PrimaryID: "host-a"}); err != nil {
+ t.Fatalf("link a-b: %v", err)
+ }
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "host-c", NeverAutoRemediate: true, SetAt: now}); err != nil {
+ t.Fatalf("lock c: %v", err)
+ }
+ if err := store.AddLink(ResourceLink{ResourceA: "host-b", ResourceB: "host-c", PrimaryID: "host-c"}); err != nil {
+ t.Fatalf("link b-c: %v", err)
+ }
+ for _, id := range []string{"host-a", "host-b", "host-c"} {
+ if !locked(t, store, id) {
+ t.Fatalf("%s is outside the lock of its linked component", id)
+ }
+ }
+ })
+ t.Run(name+"/lock set on the survivor after the link, then unlinked", func(t *testing.T) {
+ store := newStore(t)
+ if err := store.AddLink(ResourceLink{ResourceA: "vm-1", ResourceB: "agent-1", PrimaryID: "vm-1"}); err != nil {
+ t.Fatalf("link: %v", err)
+ }
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "vm-1", NeverAutoRemediate: true, SetAt: now}); err != nil {
+ t.Fatalf("operator locks the merged resource: %v", err)
+ }
+ if locked(t, store, "agent-1") {
+ t.Fatalf("the lock reached the folded member before any identity change")
+ }
+ if err := store.AddExclusion(ResourceExclusion{ResourceA: "vm-1", ResourceB: "agent-1"}); err != nil {
+ t.Fatalf("unlink: %v", err)
+ }
+ for _, id := range []string{"vm-1", "agent-1"} {
+ if !locked(t, store, id) {
+ t.Fatalf("unlinking left %s without the merged resource's lock", id)
+ }
+ }
+ })
+ t.Run(name+"/lock on one end of a chain, unlinking the other end", func(t *testing.T) {
+ store := newStore(t)
+ for _, link := range []ResourceLink{
+ {ResourceA: "chain-a", ResourceB: "chain-b", PrimaryID: "chain-a"},
+ {ResourceA: "chain-b", ResourceB: "chain-c", PrimaryID: "chain-b"},
+ } {
+ if err := store.AddLink(link); err != nil {
+ t.Fatalf("link %s-%s: %v", link.ResourceA, link.ResourceB, err)
+ }
+ }
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "chain-a", NeverAutoRemediate: true, SetAt: now}); err != nil {
+ t.Fatalf("operator locks the merged resource: %v", err)
+ }
+ if err := store.AddExclusion(ResourceExclusion{ResourceA: "chain-b", ResourceB: "chain-c"}); err != nil {
+ t.Fatalf("unlink b-c: %v", err)
+ }
+ for _, id := range []string{"chain-a", "chain-b", "chain-c"} {
+ if !locked(t, store, id) {
+ t.Fatalf("%s left the chain without the lock the whole chain carried", id)
+ }
+ }
+ })
+ t.Run(name+"/unlinking a pair that was never linked carries nothing", func(t *testing.T) {
+ store := newStore(t)
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "vm-2", NeverAutoRemediate: true, SetAt: now}); err != nil {
+ t.Fatalf("lock: %v", err)
+ }
+ if err := store.AddExclusion(ResourceExclusion{ResourceA: "vm-2", ResourceB: "agent-2"}); err != nil {
+ t.Fatalf("exclude: %v", err)
+ }
+ if locked(t, store, "agent-2") {
+ t.Fatalf("an exclusion between unlinked resources locked one of them")
+ }
+ })
+ }
+}
+
+// Recorded successions can chain, and an intermediate row may have moved on to
+// its own successor, so the upgrade carry follows the chain instead of
+// stepping one succession at a time; a lock it restores then spreads over the
+// links of the ID it lands on.
+func TestSQLiteStoreRestoresLocksAcrossRecordedSuccessionChains(t *testing.T) {
+ dir := t.TempDir()
+ open := func() *SQLiteResourceStore {
+ t.Helper()
+ store, err := NewSQLiteResourceStore(dir, "default")
+ if err != nil {
+ t.Fatalf("NewSQLiteResourceStore: %v", err)
+ }
+ return store
+ }
+ now := time.Now().UTC()
+ store := open()
+ defer func() { _ = store.Close() }()
+
+ // a1 -> b1 -> c1: b1's settings row moved to c1, so b1 has no row and the
+ // shadowed lock on a1 has to reach c1 across the gap.
+ // a2 -> b2 -> c2, recorded in the opposite order from the chain: both
+ // intermediate rows still exist.
+ // a3 -> b3 with a live link b3 - c3: the restored lock must reach c3.
+ seed := func(id string, state ResourceOperatorState) {
+ t.Helper()
+ state.CanonicalID, state.SetAt = id, now.Add(-72*time.Hour)
+ if err := store.SetResourceOperatorState(state); err != nil {
+ t.Fatalf("seed %s: %v", id, err)
+ }
+ }
+ seed("chain-a1", ResourceOperatorState{NeverAutoRemediate: true})
+ seed("chain-c1", ResourceOperatorState{Note: "moved here from b1"})
+ seed("chain-a2", ResourceOperatorState{NeverAutoRemediate: true})
+ seed("chain-b2", ResourceOperatorState{Note: "unlocked"})
+ seed("chain-c2", ResourceOperatorState{Note: "unlocked"})
+ seed("chain-a3", ResourceOperatorState{LifecycleState: LifecycleStateRetired})
+ seed("chain-b3", ResourceOperatorState{Note: "unlocked"})
+ // a4 -> b4 and c4 -> d4 with a live link b4 - c4: the lock restored onto b4
+ // spreads over the link to c4, and from c4 on to d4, which the first pass
+ // over the successions had already passed.
+ seed("chain-a4", ResourceOperatorState{NeverAutoRemediate: true})
+ seed("chain-b4", ResourceOperatorState{Note: "unlocked"})
+ seed("chain-c4", ResourceOperatorState{Note: "unlocked"})
+ seed("chain-d4", ResourceOperatorState{Note: "unlocked"})
+ for _, succession := range [][2]string{
+ {"chain-b2", "chain-c2"}, {"chain-a2", "chain-b2"},
+ {"chain-a1", "chain-b1"}, {"chain-b1", "chain-c1"},
+ {"chain-a3", "chain-b3"},
+ {"chain-a4", "chain-b4"}, {"chain-c4", "chain-d4"},
+ } {
+ if _, err := store.db.Exec(`INSERT INTO canonical_id_successions (old_canonical_id, new_canonical_id) VALUES (?, ?)`, succession[0], succession[1]); err != nil {
+ t.Fatalf("record succession %v: %v", succession, err)
+ }
+ }
+ for _, link := range [][2]string{{"chain-b3", "chain-c3"}, {"chain-b4", "chain-c4"}} {
+ if _, err := store.db.Exec(`INSERT INTO resource_links (resource_a, resource_b, primary_id, reason, created_by, created_at) VALUES (?, ?, ?, ?, ?, ?)`,
+ link[0], link[1], link[0], "linked before locks travelled", "operator", "2026-09-01T10:00:00Z"); err != nil {
+ t.Fatalf("seed link: %v", err)
+ }
+ }
+ if _, err := store.db.Exec(`DELETE FROM resource_store_migrations`); err != nil {
+ t.Fatalf("forget the carry marker: %v", err)
+ }
+ if err := store.Close(); err != nil {
+ t.Fatalf("close: %v", err)
+ }
+
+ store = open()
+ for _, id := range []string{"chain-c1", "chain-b2", "chain-c2", "chain-b3", "chain-c3", "chain-b4", "chain-c4", "chain-d4"} {
+ got, found, err := store.GetResourceOperatorState(id)
+ if err != nil || !found || !got.BlocksRemediation() {
+ t.Fatalf("%s did not get the lock its succession chain dropped: found=%v err=%v state=%+v", id, found, err, got)
+ }
+ }
+ if got, _, _ := store.GetResourceOperatorState("chain-c1"); got.Note != "moved here from b1" {
+ t.Fatalf("restoring the lock rewrote the successor's own settings: %+v", got)
+ }
+}
diff --git a/internal/unifiedresources/remediation_lock_carry.go b/internal/unifiedresources/remediation_lock_carry.go
new file mode 100644
index 0000000000..d38e968ebe
--- /dev/null
+++ b/internal/unifiedresources/remediation_lock_carry.go
@@ -0,0 +1,552 @@
+package unifiedresources
+
+import (
+ "database/sql"
+ "fmt"
+ "log"
+ "sort"
+ "strings"
+ "time"
+)
+
+// An operator's remediation lock (ResourceOperatorState.BlocksRemediation) is
+// stored on the row of one canonical ID, and every gate reads the row of the ID
+// a resource currently has. Identity changes move resources between IDs, so the
+// stores carry the block with them: a live manual link makes its members one
+// resource, and the members of a link component share the block whenever the
+// component changes (a link, an unlink) and whenever a succession lands on one
+// of them. CarryRemediationLock owns what is written onto a row.
+
+// carriedLock records one remediation block written onto `to` from `from`.
+type carriedLock struct {
+ to string
+ from string
+}
+
+func logCarriedLocks(cause string, carried []carriedLock) {
+ for _, lock := range carried {
+ log.Printf("unifiedresources: carried the remediation lock of %q onto %q (%s)", lock.from, lock.to, cause)
+ }
+}
+
+// rowsQueryer is the read side shared by *sql.DB and *sql.Tx.
+type rowsQueryer interface {
+ Query(query string, args ...any) (*sql.Rows, error)
+}
+
+// liveLinkPairsSQL returns the ID pairs of the links that are their pair's
+// latest operator decision, read through q so a caller inside a transaction
+// sees its own writes. It reads only the columns that decide a pair (IDs and
+// time), which the schema never lets be NULL except the time, so a row with an
+// empty reason or author cannot stop a succession or the upgrade carry.
+func liveLinkPairsSQL(q rowsQueryer) ([][2]string, error) {
+ links, err := queryPairDecisions(q, `SELECT resource_a, resource_b, created_at FROM resource_links`)
+ if err != nil || len(links) == 0 {
+ return nil, err
+ }
+ decided, err := queryPairDecisions(q, `SELECT resource_a, resource_b, created_at FROM resource_exclusions`)
+ if err != nil {
+ return nil, err
+ }
+ exclusions := make([]ResourceExclusion, 0, len(decided))
+ for _, decision := range decided {
+ exclusions = append(exclusions, ResourceExclusion{ResourceA: decision.ResourceA, ResourceB: decision.ResourceB, CreatedAt: decision.CreatedAt})
+ }
+ live, _ := effectiveManualPairDecisions(links, exclusions)
+ return linkPairs(live), nil
+}
+
+// decisionTime reads a stored created_at leniently: a time the driver cannot
+// parse (NULL, an integer, free text) counts as the zero time, which only
+// affects whether a newer decision supersedes it. One malformed row must not
+// stop an unrelated succession, link or unlink from settling a lock.
+func decisionTime(raw any) time.Time {
+ switch value := raw.(type) {
+ case time.Time:
+ return value
+ case string:
+ for _, layout := range []string{time.RFC3339Nano, "2006-01-02 15:04:05.999999999-07:00", "2006-01-02 15:04:05"} {
+ if parsed, err := time.Parse(layout, value); err == nil {
+ return parsed
+ }
+ }
+ }
+ return time.Time{}
+}
+
+// queryPairDecisions reads (resource_a, resource_b, created_at) rows as links;
+// only the pair and its time are meaningful.
+func queryPairDecisions(q rowsQueryer, query string) (decisions []ResourceLink, err error) {
+ rows, err := q.Query(query)
+ if err != nil {
+ return nil, fmt.Errorf("query manual pair decisions: %w", err)
+ }
+ defer func() {
+ if closeErr := rows.Close(); closeErr != nil && err == nil {
+ err = fmt.Errorf("close manual pair decisions: %w", closeErr)
+ }
+ }()
+ for rows.Next() {
+ var (
+ a, b string
+ at any
+ )
+ if scanErr := rows.Scan(&a, &b, &at); scanErr != nil {
+ return nil, fmt.Errorf("scan manual pair decision: %w", scanErr)
+ }
+ decisions = append(decisions, ResourceLink{ResourceA: CanonicalResourceID(a), ResourceB: CanonicalResourceID(b), CreatedAt: decisionTime(at)})
+ }
+ if rowsErr := rows.Err(); rowsErr != nil {
+ return nil, fmt.Errorf("iterate manual pair decisions: %w", rowsErr)
+ }
+ return decisions, nil
+}
+
+func linkPairs(links []ResourceLink) [][2]string {
+ pairs := make([][2]string, 0, len(links))
+ for _, link := range links {
+ pairs = append(pairs, [2]string{link.ResourceA, link.ResourceB})
+ }
+ return pairs
+}
+
+// linkComponents groups the IDs reachable from seeds through pairs, one sorted
+// slice per connected component that holds at least one seed. A pair names a
+// link the registry treats as one resource, in either direction, so a chain of
+// links is one component.
+func linkComponents(pairs [][2]string, seeds []string) [][]string {
+ adjacent := make(map[string][]string, len(pairs)*2)
+ for _, pair := range pairs {
+ a, b := strings.TrimSpace(pair[0]), strings.TrimSpace(pair[1])
+ if a == "" || b == "" || a == b {
+ continue
+ }
+ adjacent[a] = append(adjacent[a], b)
+ adjacent[b] = append(adjacent[b], a)
+ }
+ seen := make(map[string]struct{}, len(seeds))
+ var components [][]string
+ for _, seed := range seeds {
+ seed = strings.TrimSpace(seed)
+ if seed == "" {
+ continue
+ }
+ if _, done := seen[seed]; done {
+ continue
+ }
+ seen[seed] = struct{}{}
+ component := []string{seed}
+ for queue := []string{seed}; len(queue) > 0; queue = queue[1:] {
+ for _, next := range adjacent[queue[0]] {
+ if _, done := seen[next]; done {
+ continue
+ }
+ seen[next] = struct{}{}
+ component = append(component, next)
+ queue = append(queue, next)
+ }
+ }
+ sort.Strings(component)
+ components = append(components, component)
+ }
+ return components
+}
+
+// shareRemediationLock returns the rows a component must hold so that every
+// member carries the block any one of them holds. states holds the members'
+// stored rows; a member with no row is absent from it. The donor is the first
+// blocking member in ID order, so the result does not depend on map order.
+func shareRemediationLock(component []string, states map[string]ResourceOperatorState, at time.Time) (writes []ResourceOperatorState, carried []carriedLock) {
+ donorID := ""
+ for _, id := range component {
+ if state, found := states[id]; found && state.BlocksRemediation() {
+ donorID = id
+ break
+ }
+ }
+ if donorID == "" {
+ return nil, nil
+ }
+ for _, id := range component {
+ if id == donorID {
+ continue
+ }
+ state, found := states[id]
+ if next, changed := CarryRemediationLock(id, state, found, states[donorID], true, at); changed {
+ writes = append(writes, next)
+ carried = append(carried, carriedLock{to: id, from: donorID})
+ }
+ }
+ return writes, carried
+}
+
+// carryRemediationLockSQL writes donor's remediation block onto survivor's
+// operator-state row inside the caller's transaction (CarryRemediationLock owns
+// what is written). createRow false leaves a survivor with no row alone, for
+// a caller whose next statement moves the donor's whole row onto it. Reports
+// whether a row was written.
+func carryRemediationLockSQL(tx *sql.Tx, survivorID, donorID string, at time.Time, createRow bool) (bool, error) {
+ survivorID = strings.TrimSpace(survivorID)
+ donorID = strings.TrimSpace(donorID)
+ if survivorID == "" || donorID == "" || survivorID == donorID {
+ return false, nil
+ }
+ donor, donorFound, err := getResourceOperatorStateSQL(tx, donorID)
+ if err != nil || !donorFound || !donor.BlocksRemediation() {
+ return false, err
+ }
+ survivor, survivorFound, err := getResourceOperatorStateSQL(tx, survivorID)
+ if err != nil {
+ return false, err
+ }
+ if !survivorFound && !createRow {
+ return false, nil
+ }
+ next, changed := CarryRemediationLock(survivorID, survivor, survivorFound, donor, donorFound, at)
+ if !changed {
+ return false, nil
+ }
+ if err := setResourceOperatorStateSQL(tx, next); err != nil {
+ return false, err
+ }
+ return true, nil
+}
+
+// shareRemediationLockAmongSQL makes every ID in group carry the block any of
+// them holds, inside the caller's transaction, and reports what it wrote.
+func shareRemediationLockAmongSQL(tx *sql.Tx, group []string, at time.Time) ([]carriedLock, error) {
+ if len(group) < 2 {
+ return nil, nil
+ }
+ states := make(map[string]ResourceOperatorState, len(group))
+ for _, id := range group {
+ state, found, err := getResourceOperatorStateSQL(tx, id)
+ if err != nil {
+ return nil, err
+ }
+ if found {
+ states[id] = state
+ }
+ }
+ writes, carried := shareRemediationLock(group, states, at)
+ for _, next := range writes {
+ if err := setResourceOperatorStateSQL(tx, next); err != nil {
+ return nil, err
+ }
+ }
+ return carried, nil
+}
+
+// shareRemediationLockAcrossLinksSQL makes every member of each live link
+// component holding a seed carry the block any member holds, inside the
+// caller's transaction. A link written in the same transaction counts as live.
+func shareRemediationLockAcrossLinksSQL(tx *sql.Tx, seeds []string, at time.Time) ([]carriedLock, error) {
+ pairs, err := liveLinkPairsSQL(tx)
+ if err != nil {
+ return nil, err
+ }
+ var carried []carriedLock
+ for _, component := range linkComponents(pairs, seeds) {
+ shared, err := shareRemediationLockAmongSQL(tx, component, at)
+ if err != nil {
+ return nil, err
+ }
+ carried = append(carried, shared...)
+ }
+ return carried, nil
+}
+
+// linkedMembersSQL returns the IDs in the live link components of seeds, seeds
+// included, as one sorted list.
+func linkedMembersSQL(tx *sql.Tx, seeds []string) ([]string, error) {
+ pairs, err := liveLinkPairsSQL(tx)
+ if err != nil {
+ return nil, err
+ }
+ return flattenComponents(linkComponents(pairs, seeds)), nil
+}
+
+func flattenComponents(components [][]string) []string {
+ seen := make(map[string]struct{})
+ var out []string
+ for _, component := range components {
+ for _, id := range component {
+ if _, dup := seen[id]; dup {
+ continue
+ }
+ seen[id] = struct{}{}
+ out = append(out, id)
+ }
+ }
+ sort.Strings(out)
+ return out
+}
+
+// resolveThroughRekeys follows id through a batch's re-keys (a -> b, then
+// b -> c) to the ID that holds it once the whole batch has applied.
+func resolveThroughRekeys(id string, rekeyed map[string]string) string {
+ for hops := 0; hops <= len(rekeyed); hops++ {
+ next, moved := rekeyed[id]
+ if !moved {
+ return id
+ }
+ id = next
+ }
+ return id
+}
+
+// settledGroup maps a succession's saved sharing group through the batch's
+// re-keys, so a member a later succession in the batch re-keyed away is
+// replaced by its successor instead of being given a row of its own.
+func settledGroup(group []string, rekeyed map[string]string) []string {
+ resolved := make([]string, 0, len(group))
+ for _, id := range group {
+ resolved = append(resolved, resolveThroughRekeys(id, rekeyed))
+ }
+ return flattenComponents([][]string{resolved})
+}
+
+// groupAfterRekey is the set a succession's pre-re-key members form once the
+// predecessor's ID has become the successor's: they were one linked resource
+// before the re-key and stay one group for sharing whatever the re-keyed
+// links now read as.
+func groupAfterRekey(members []string, oldID, newID string) []string {
+ group := make([]string, 0, len(members)+1)
+ group = append(group, newID)
+ for _, id := range members {
+ if id == oldID {
+ continue
+ }
+ group = append(group, id)
+ }
+ return flattenComponents([][]string{group})
+}
+
+// carryRemediationLockLocked is carryRemediationLockSQL for the in-memory
+// store. The caller holds m.mu.
+func (m *MemoryStore) carryRemediationLockLocked(survivorID, donorID string, at time.Time, createRow bool) {
+ survivorID = strings.TrimSpace(survivorID)
+ donorID = strings.TrimSpace(donorID)
+ if survivorID == "" || donorID == "" || survivorID == donorID {
+ return
+ }
+ donor, donorFound := m.resourceOperatorState[donorID]
+ survivor, survivorFound := m.resourceOperatorState[survivorID]
+ if !survivorFound && !createRow {
+ return
+ }
+ if next, changed := CarryRemediationLock(survivorID, survivor, survivorFound, donor, donorFound, at); changed {
+ if m.resourceOperatorState == nil {
+ m.resourceOperatorState = make(map[string]ResourceOperatorState)
+ }
+ m.resourceOperatorState[survivorID] = next
+ }
+}
+
+// shareRemediationLockAmongLocked is shareRemediationLockAmongSQL for the
+// in-memory store. The caller holds m.mu.
+func (m *MemoryStore) shareRemediationLockAmongLocked(group []string, at time.Time) {
+ if len(group) < 2 {
+ return
+ }
+ states := make(map[string]ResourceOperatorState, len(group))
+ for _, id := range group {
+ if state, found := m.resourceOperatorState[id]; found {
+ states[id] = state
+ }
+ }
+ writes, _ := shareRemediationLock(group, states, at)
+ for _, next := range writes {
+ if m.resourceOperatorState == nil {
+ m.resourceOperatorState = make(map[string]ResourceOperatorState)
+ }
+ m.resourceOperatorState[next.CanonicalID] = next
+ }
+}
+
+// shareRemediationLockAcrossLinksLocked is shareRemediationLockAcrossLinksSQL
+// for the in-memory store. The caller holds m.mu.
+func (m *MemoryStore) shareRemediationLockAcrossLinksLocked(seeds []string, at time.Time) {
+ for _, component := range linkComponents(m.livePairsLocked(), seeds) {
+ m.shareRemediationLockAmongLocked(component, at)
+ }
+}
+
+// livePairsLocked returns the in-memory store's live link pairs. The caller
+// holds m.mu.
+func (m *MemoryStore) livePairsLocked() [][2]string {
+ live, _ := effectiveManualPairDecisions(m.links, m.exclusions)
+ return linkPairs(live)
+}
+
+const remediationLockCarryMigration = "remediation-lock-carry-v1"
+
+// migrateRemediationLockCarry carries the locks that identity changes dropped
+// before the stores carried them. It runs once per store, so an operator who
+// later lifts a lock is not overruled by a dormant row on every start:
+//
+// - a succession applied while its successor already had a row left the
+// predecessor's row behind, shadowed, and with it the predecessor's lock;
+// - a link recorded before a lock followed it left the lock on a member the
+// registry folded away.
+//
+// Successions are carried first so a lock they restore then spreads across the
+// successor's links. A failure is logged and retried on the next start rather
+// than keeping the store from opening.
+func (s *SQLiteResourceStore) migrateRemediationLockCarry() error {
+ var applied int
+ if err := s.db.QueryRow(`SELECT COUNT(*) FROM resource_store_migrations WHERE name = ?`, remediationLockCarryMigration).Scan(&applied); err != nil {
+ return fmt.Errorf("read remediation lock carry marker: %w", err)
+ }
+ if applied > 0 {
+ return nil
+ }
+
+ rows, err := s.db.Query(`SELECT old_canonical_id, new_canonical_id FROM canonical_id_successions`)
+ if err != nil {
+ return fmt.Errorf("query recorded canonical ID successions: %w", err)
+ }
+ var successions []CanonicalIDSuccession
+ for rows.Next() {
+ var succession CanonicalIDSuccession
+ if err := rows.Scan(&succession.OldCanonicalID, &succession.NewCanonicalID); err != nil {
+ _ = rows.Close()
+ return fmt.Errorf("scan recorded canonical ID succession: %w", err)
+ }
+ successions = append(successions, succession)
+ }
+ rowsErr := rows.Err()
+ if err := rows.Close(); err != nil {
+ return fmt.Errorf("close recorded canonical ID successions: %w", err)
+ }
+ if rowsErr != nil {
+ return fmt.Errorf("iterate recorded canonical ID successions: %w", rowsErr)
+ }
+
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ tx, err := s.db.Begin()
+ if err != nil {
+ return fmt.Errorf("begin remediation lock carry: %w", err)
+ }
+ committed := false
+ defer func() {
+ if !committed {
+ _ = tx.Rollback()
+ }
+ }()
+
+ at := time.Now().UTC()
+ var carried []carriedLock
+ // Successions chain (a -> b, then b -> c), an intermediate row may have
+ // moved on to its own successor, and a lock a succession restores can reach
+ // a link whose far member heads another succession. A block therefore goes
+ // to every ID its chain reaches that still has a row, and the two steps
+ // repeat until a pass changes nothing. Every productive pass sets a block
+ // that was unset on some ID, so the passes needed are bounded by the number
+ // of IDs the successions and links name.
+ livePairs, err := liveLinkPairsSQL(tx)
+ if err != nil {
+ return err
+ }
+ maxPasses := 2*(len(successions)+len(livePairs)) + 8
+ converged := false
+ for pass := 0; pass <= maxPasses; pass++ {
+ changedAny := false
+ for _, oldID := range recordedPredecessors(successions) {
+ for _, descendant := range successionDescendants(successions, oldID) {
+ changed, err := carryRemediationLockSQL(tx, descendant, oldID, at, false)
+ if err != nil {
+ return fmt.Errorf("carry remediation lock %q -> %q: %w", oldID, descendant, err)
+ }
+ if changed {
+ changedAny = true
+ carried = append(carried, carriedLock{to: descendant, from: oldID})
+ }
+ }
+ }
+ pairs, err := liveLinkPairsSQL(tx)
+ if err != nil {
+ return err
+ }
+ var seeds []string
+ for _, pair := range pairs {
+ seeds = append(seeds, pair[0], pair[1])
+ }
+ shared, err := shareRemediationLockAcrossLinksSQL(tx, seeds, at)
+ if err != nil {
+ return err
+ }
+ if len(shared) > 0 {
+ changedAny = true
+ carried = append(carried, shared...)
+ }
+ if !changedAny {
+ converged = true
+ break
+ }
+ }
+ if !converged {
+ // Cannot happen while every change sets a block that was unset, but a
+ // store that did not finish must not record that it did.
+ return fmt.Errorf("remediation lock carry did not converge")
+ }
+
+ if _, err := tx.Exec(`INSERT INTO resource_store_migrations (name) VALUES (?)`, remediationLockCarryMigration); err != nil {
+ return fmt.Errorf("record remediation lock carry marker: %w", err)
+ }
+ if err := tx.Commit(); err != nil {
+ return fmt.Errorf("commit remediation lock carry: %w", err)
+ }
+ committed = true
+ logCarriedLocks("restoring a lock an earlier link or succession dropped", carried)
+ return nil
+}
+
+// recordedPredecessors lists the predecessor IDs of successions, sorted and
+// deduplicated.
+func recordedPredecessors(successions []CanonicalIDSuccession) []string {
+ seen := make(map[string]struct{}, len(successions))
+ var out []string
+ for _, succession := range successions {
+ oldID := CanonicalResourceID(succession.OldCanonicalID)
+ if oldID == "" {
+ continue
+ }
+ if _, dup := seen[oldID]; dup {
+ continue
+ }
+ seen[oldID] = struct{}{}
+ out = append(out, oldID)
+ }
+ sort.Strings(out)
+ return out
+}
+
+// successionDescendants returns every ID the successions lead from oldID to,
+// directly or through further successions, in sorted order. oldID itself is
+// never returned, and a cycle ends the walk instead of repeating it.
+func successionDescendants(successions []CanonicalIDSuccession, oldID string) []string {
+ successors := make(map[string][]string, len(successions))
+ for _, succession := range successions {
+ from, to := CanonicalResourceID(succession.OldCanonicalID), CanonicalResourceID(succession.NewCanonicalID)
+ if from == "" || to == "" || from == to {
+ continue
+ }
+ successors[from] = append(successors[from], to)
+ }
+ seen := map[string]struct{}{oldID: {}}
+ var out []string
+ for queue := []string{oldID}; len(queue) > 0; queue = queue[1:] {
+ for _, next := range successors[queue[0]] {
+ if _, dup := seen[next]; dup {
+ continue
+ }
+ seen[next] = struct{}{}
+ out = append(out, next)
+ queue = append(queue, next)
+ }
+ }
+ sort.Strings(out)
+ return out
+}
diff --git a/internal/unifiedresources/resource_operator_state.go b/internal/unifiedresources/resource_operator_state.go
index ad929480e7..754896cc3b 100644
--- a/internal/unifiedresources/resource_operator_state.go
+++ b/internal/unifiedresources/resource_operator_state.go
@@ -256,6 +256,43 @@ func (s ResourceOperatorState) BlocksRemediation() bool {
return s.NeverAutoRemediate || s.LifecycleState == LifecycleStateRetired
}
+// RemediationLockCarriedBy attributes a row whose remediation lock was written
+// by an identity change (a manual link, a canonical-ID succession) rather than
+// by an operator edit, so the operator-state header does not credit a person
+// with a lock they never set on this ID.
+const RemediationLockCarriedBy = "system:identity-change"
+
+// CarryRemediationLock returns the state a surviving resource must hold once
+// an identity change has absorbed donor, and whether that differs from the
+// survivor's current state. A remediation lock is the operator's "never touch
+// this", so it follows the resource through a manual link fold or a
+// canonical-ID succession instead of belonging to whichever ID happened to be
+// keyed first: the lock is ORed onto the survivor, never lost to an unlocked
+// row that happens to win the key.
+//
+// Only the remediation block crosses, as NeverAutoRemediate. Retirement also
+// silences alerts and Patrol attention, which is a monitoring posture rather
+// than a safety property; copying it would mute a survivor the operator still
+// watches. Every other setting the survivor holds is returned untouched, and
+// a survivor with no row gets a row carrying the block alone. The result is
+// attributed to the identity change at `at`. An already-blocking survivor is
+// returned as it was.
+func CarryRemediationLock(survivorID string, survivor ResourceOperatorState, survivorFound bool, donor ResourceOperatorState, donorFound bool, at time.Time) (ResourceOperatorState, bool) {
+ if !donorFound || !donor.BlocksRemediation() {
+ return survivor, false
+ }
+ if survivorFound && survivor.BlocksRemediation() {
+ return survivor, false
+ }
+ if !survivorFound {
+ survivor = ResourceOperatorState{CanonicalID: survivorID}
+ }
+ survivor.NeverAutoRemediate = true
+ survivor.SetAt = at.UTC()
+ survivor.SetBy = RemediationLockCarriedBy
+ return NormalizeResourceOperatorState(survivor), true
+}
+
// IsInMaintenanceAt reports whether `now` falls within the configured
// maintenance window. Returns false when no window is configured, when only
// one of start/end is set (treated as no window), or when end <= start.
diff --git a/internal/unifiedresources/store.go b/internal/unifiedresources/store.go
index 8f5d7fa85c..de9c296892 100644
--- a/internal/unifiedresources/store.go
+++ b/internal/unifiedresources/store.go
@@ -477,6 +477,12 @@ func (s *SQLiteResourceStore) initSchema() error {
metadata_json TEXT
);
+ -- One row per one-shot data migration that must not repeat on every start.
+ CREATE TABLE IF NOT EXISTS resource_store_migrations (
+ name TEXT PRIMARY KEY,
+ applied_at DATETIME DEFAULT CURRENT_TIMESTAMP
+ );
+
CREATE TABLE IF NOT EXISTS canonical_id_successions (
old_canonical_id TEXT PRIMARY KEY,
new_canonical_id TEXT NOT NULL,
@@ -635,6 +641,11 @@ func (s *SQLiteResourceStore) initSchema() error {
if err := s.migrateResourceOperatorStateSchema(); err != nil {
return err
}
+ if err := s.migrateRemediationLockCarry(); err != nil {
+ // A repair of earlier lock losses, not part of opening the store: keep
+ // the store usable and retry on the next start.
+ log.Printf("[WARN] unified_resources: remediation lock carry migration failed; it will retry on the next start: %v", err)
+ }
if err := s.migrateActionAuditRedaction(); err != nil {
return err
}
@@ -1232,10 +1243,21 @@ func (s *SQLiteResourceStore) AddLink(link ResourceLink) error {
`, a, b, link.PrimaryID, link.Reason, link.CreatedBy, link.CreatedAt); err != nil {
return fmt.Errorf("upsert resource link %q<->%q: %w", a, b, err)
}
+ // The link declares both IDs one resource, and only the ID the registry
+ // keeps is ever planned or dispatched against. Which side that is depends
+ // on resource types the store cannot see (a guest outlives an agent
+ // running inside it whichever way the link points), so a lock on any
+ // member of the linked component is written to every member, in the
+ // link's own transaction.
+ carried, err := shareRemediationLockAcrossLinksSQL(tx, []string{a, b}, time.Now().UTC())
+ if err != nil {
+ return fmt.Errorf("share remediation lock across resource link %q<->%q: %w", a, b, err)
+ }
if err := tx.Commit(); err != nil {
return fmt.Errorf("commit resource link %q<->%q: %w", a, b, err)
}
committed = true
+ logCarriedLocks("a manual link", carried)
return nil
}
@@ -1261,6 +1283,13 @@ func (s *SQLiteResourceStore) AddExclusion(exclusion ResourceExclusion) error {
_ = tx.Rollback()
}
}()
+ // Unlinking splits one resource into the members it joined. The lock the
+ // merged resource carried stays with each of them: it was the operator's on
+ // all of them, and only the surviving ID was ever able to hold it since.
+ carried, err := shareRemediationLockAcrossLinksSQL(tx, []string{a, b}, time.Now().UTC())
+ if err != nil {
+ return fmt.Errorf("share remediation lock across resource link %q<->%q before unlinking: %w", a, b, err)
+ }
// An exclusion replaces the pair's earlier decision, so unlinking a linked
// pair splits it (manual_link_decisions.go): its links in either order,
// and an exclusion canonical-ID succession stored in the other order,
@@ -1288,6 +1317,7 @@ func (s *SQLiteResourceStore) AddExclusion(exclusion ResourceExclusion) error {
return fmt.Errorf("commit resource exclusion %q<->%q: %w", a, b, err)
}
committed = true
+ logCarriedLocks("an unlink", carried)
return nil
}
@@ -3610,6 +3640,9 @@ func (m *MemoryStore) AddLink(link ResourceLink) error {
return sameManualPair(existing.ResourceA, existing.ResourceB, link.ResourceA, link.ResourceB)
})
m.links = append(m.links, link)
+ // Every member of the linked component carries the block; see
+ // SQLiteResourceStore.AddLink.
+ m.shareRemediationLockAcrossLinksLocked([]string{link.ResourceA, link.ResourceB}, time.Now().UTC())
return nil
}
@@ -3621,6 +3654,9 @@ func (m *MemoryStore) AddExclusion(exclusion ResourceExclusion) error {
if exclusion.CreatedAt.IsZero() {
exclusion.CreatedAt = time.Now().UTC()
}
+ // The merged resource's lock stays with each member the unlink splits off;
+ // see SQLiteResourceStore.AddExclusion.
+ m.shareRemediationLockAcrossLinksLocked([]string{exclusion.ResourceA, exclusion.ResourceB}, time.Now().UTC())
// An exclusion replaces the pair's earlier decision, so unlinking a
// linked pair splits it (manual_link_decisions.go).
m.links = slices.DeleteFunc(m.links, func(link ResourceLink) bool {