Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Keep a remediation lock readable on an ID a succession retired

rcourtmanb329730e6bd6
diff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.md
index 540d0ec4e4..7b9e971aa6 100644
--- a/docs/release-control/v6/internal/subsystems/unified-resources.md
+++ b/docs/release-control/v6/internal/subsystems/unified-resources.md
@@ -3635,6 +3635,43 @@ until then. The store cannot see which links the registry declines to fold
so those share too, which can lock a host because a linked probe was retired.
Re-recording an existing link shares again.
+A rebuild applies the successions it finds to the store before it publishes
+the generation that lists the successors, and the generation it replaces keeps
+serving in between. The planner, the Patrol broker and the dispatch gates
+(`validateExecutionPolicy`, `isResourceRemediationLocked`) read the row of the ID
+that generation lists, so for the length of the rebuild they ask for an ID whose
+row has just moved. `GetResourceOperatorState` therefore follows the recorded
+successions (`canonical_id_successions`) in both stores when the ID it is asked
+about holds no row (`followRetiredID`, one walker for both): it answers as the
+ID its chain ends on would, with that row's remediation block alone
+(`retiredIDRemediationLock`, which is `CarryRemediationLock` into an empty
+survivor), as `NeverAutoRemediate`, attributed `system:identity-change` and
+dated by the row it came from so two reads agree (a plan's policy revision
+hashes the state). Nothing is written, so `ListResourceOperatorStates` still
+lists only the rows the stores hold. The end of the chain decides, because it is
+the ID the next generation lists: a lock the operator lifted there, by editing
+or clearing its row, is not read back from the stale row a succession leaves on
+an ID before it when its successor already had a row, and an unlocked stale row
+does not hide a lock held at the end. An ID that holds a row of its own answers
+with that row. A chain is as long as the identity changes were and has no cap;
+a chain that returns to an ID it has seen has no end, so a lock held anywhere on
+it counts. Only the block crosses, so while the window is open the retired ID
+reads none of the row's other settings (retirement as a lifecycle state,
+maintenance windows, criticality, notes), as it read none before. The walk is
+one snapshot: the SQLite store reads the ID's own row and every hop in one
+transaction, because a succession that commits between two of the statements
+can move the lock onto an ID the walk has already passed. The window ends when
+the new generation publishes: that generation lists the successor, and a
+lifecycle plan made against the retired ID drifts (`resource ... is no longer
+present`). An operator edit to the retired ID inside the window goes to the
+retired ID's own row, not the successor's: the operator-state API writes the ID
+the registry it consults lists, and a succession applies once per predecessor.
+A lock set there does not reach the successor, which the next generation lists
+and reads, and a lock lifted there leaves the successor's lock in place, which
+is the safe direction; the retired ID's own row still answers reads of that ID.
+A retired ID that a different resource later mints again, with no row of its
+own, reads the successor's lock until the operator writes it a row.
+
A host or Docker reporter that had no machine ID, DMI UUID or cluster slot is
keyed by its source-specific ID (unless a link or match folds it into another
resource) and held no identity pin (a pin needs one of those keys), so when a
@@ -3667,10 +3704,10 @@ era of every keyed reporter that passes those checks, in the batches of its
ingest passes, and also restores a lock or row an earlier keyless-to-keyed
change left behind, even where the operator has edited the keyed row since. An ID held under a report ID the reporter no longer sends is
not recomputable. Like every succession, the move happens while the previous
-generation is still being served, so when the keyed ID had no row an action
-planned against the old ID in that window finds no row there, and a link the
-succession rewrites takes effect in the next generation, because a registry
-loads its links when it is built (the lock is shared across the pair at once).
+generation is still being served, which is why a read of the old ID follows the
+succession (above); a link the succession rewrites takes effect in the next
+generation, because a registry loads its links when it is built (the lock is
+shared across the pair at once).
Canonical ID changes that reach neither a link nor `ApplyCanonicalIDSuccessions`
carry nothing: a physical disk re-keyed in place to a machine-scoped ID
@@ -3686,6 +3723,7 @@ array disk, a record-fed disk), and `TestProviderDiskRecordsAdvertiseNoActionCap
in `internal/truenas/provider_test.go` when the TrueNAS provider's records do.
The disk's other operator settings (retirement, maintenance, notes) are
orphaned by the same re-key and are not carried. Proof:
+`TestRemediationLockHoldsOnTheRetiredIDWhileThePreviousGenerationServes`,
`TestReporterGainingAStrongKeyKeepsItsRemediationLock`,
`TestReporterEraIsDeclaredOnlyForAKeyTheReporterGains`,
`TestPhysicalDiskResourcesAdvertiseNoActionCapability`,
@@ -3696,6 +3734,10 @@ orphaned by the same re-key and are not carried. Proof:
`TestSQLiteStoreRestoresLocksAcrossRecordedSuccessionChains` in
`internal/unifiedresources/registry_test.go`, and
`TestCanonicalIDSuccessionCarriesRemediationLockOntoExistingSuccessorRow`,
+`TestRetiredCanonicalIDReadsTheLockItsSuccessorHolds`,
+`TestRetiredCanonicalIDReadFollowsASuccessionChain`,
+`TestRetiredCanonicalIDReadsWhatTheEndOfItsChainHolds`,
+`TestRetiredCanonicalIDReadEndsOnARecordedLoop`,
`TestCanonicalIDSuccessionRekeysAPredecessorOnce`,
`TestCanonicalIDSuccessionLockReachesLinkedSurvivor` and
`TestCanonicalIDSuccessionKeepsLinkedMembersTogether` in
diff --git a/internal/actionlifecycle/service_test.go b/internal/actionlifecycle/service_test.go
index d1d7cc7f5f..168ef3fab1 100644
--- a/internal/actionlifecycle/service_test.go
+++ b/internal/actionlifecycle/service_test.go
@@ -1473,6 +1473,60 @@ func TestExecuteRefusesRemediationLockedResource(t *testing.T) {
}
}
+// A rebuild moves the operator's rows to the successor ID before it publishes
+// the registry generation that lists it, so an action planned against the old
+// ID can be dispatched, and the planner can be asked about it, while the
+// registry the lifecycle consults still lists the old ID. The lock the
+// operator set on it has moved with its row; the gates must still find it.
+func TestLockMovedBySuccessionStillRefusesTheIDThePreviousGenerationLists(t *testing.T) {
+ locks := map[string]unified.ResourceOperatorState{
+ "never auto-remediate": {NeverAutoRemediate: true},
+ "retired": {LifecycleState: unified.LifecycleStateRetired},
+ }
+ for lockName, lock := range locks {
+ t.Run(lockName, func(t *testing.T) {
+ now := time.Now().UTC()
+ env := newServiceEnv(t, testResource(now, unified.ApprovalNone))
+
+ plan, err := env.service.Plan(context.Background(), "default", restartRequest(), testActionActor("requester", "default"))
+ if err != nil {
+ t.Fatalf("Plan: %v", err)
+ }
+ state := lock
+ state.CanonicalID = "vm:42"
+ if err := env.store.SetResourceOperatorState(state); err != nil {
+ t.Fatalf("SetResourceOperatorState: %v", err)
+ }
+ if err := env.store.(*unified.MemoryStore).ApplyCanonicalIDSuccessions([]unified.CanonicalIDSuccession{{OldCanonicalID: "vm:42", NewCanonicalID: "vm:42-successor"}}); err != nil {
+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)
+ }
+ if _, listed := env.registry.Get("vm:42"); !listed {
+ t.Fatal("the registry no longer lists vm:42; the fixture does not model the window before the new generation publishes")
+ }
+ if row, found, err := env.store.GetResourceOperatorState("vm:42-successor"); err != nil || !found || !row.BlocksRemediation() {
+ t.Fatalf("the successor does not hold the moved lock: found=%v err=%v state=%+v", found, err, row)
+ }
+
+ // The planner and the Patrol broker read the policy through the lifecycle.
+ planned, found, err := env.service.ResourceOperatorState("default", "vm:42")
+ if err != nil || !found || !planned.BlocksRemediation() {
+ t.Fatalf("the policy read for vm:42 lost the lock: found=%v err=%v state=%+v", found, err, planned)
+ }
+
+ failed, err := env.service.Execute(context.Background(), "default", plan.ActionID, testActionActor("agent:test", "default"), "")
+ if !errors.Is(err, unified.ErrResourceRemediationLocked) {
+ t.Fatalf("error = %v, want ErrResourceRemediationLocked", err)
+ }
+ if env.executor.calls != 0 {
+ t.Fatalf("executor must not run on a locked resource whose row moved, calls = %d", env.executor.calls)
+ }
+ if failed.State != unified.ActionStateFailed {
+ t.Fatalf("refusal must persist a terminal failed audit, state = %q", failed.State)
+ }
+ })
+ }
+}
+
func TestExecuteRefusesRetiredResourceWithoutDispatch(t *testing.T) {
now := time.Now().UTC()
env := newServiceEnv(t, testResource(now, unified.ApprovalNone))
diff --git a/internal/ai/tools/action_audit_remediation_lock_test.go b/internal/ai/tools/action_audit_remediation_lock_test.go
index 1e3ed2ee8f..65aaefba80 100644
--- a/internal/ai/tools/action_audit_remediation_lock_test.go
+++ b/internal/ai/tools/action_audit_remediation_lock_test.go
@@ -378,3 +378,28 @@ func TestExecuteNativeActionWithAuditAllowsHumanApprovedDispatchOnUnknownLockSta
t.Fatalf("expected successful ExecutionResult, got %#v", result)
}
}
+
+// A succession moves the operator's row to the successor ID when a rebuild
+// applies it, before the registry generation listing the successor is
+// published; the dispatch a model asks for in between still names the old ID.
+// The lock the operator set on it must still refuse.
+func TestRemediationLockGateFollowsASuccessionOffTheIDItNames(t *testing.T) {
+ store := unifiedresources.NewMemoryStore()
+ if err := store.SetResourceOperatorState(unifiedresources.ResourceOperatorState{CanonicalID: "agent-before-key", NeverAutoRemediate: true}); err != nil {
+ t.Fatalf("lock the agent: %v", err)
+ }
+ if err := store.ApplyCanonicalIDSuccessions([]unifiedresources.CanonicalIDSuccession{{OldCanonicalID: "agent-before-key", NewCanonicalID: "agent-keyed"}}); err != nil {
+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)
+ }
+ executor := NewPulseToolExecutor(ExecutorConfig{ActionAuditStore: store})
+
+ for _, humanApproved := range []bool{false, true} {
+ err := executor.checkRemediationLockForDispatch("action-1", "agent-before-key", "restart", humanApproved)
+ if !errors.Is(err, unifiedresources.ErrResourceRemediationLocked) {
+ t.Fatalf("humanApproved=%v: error = %v, want ErrResourceRemediationLocked", humanApproved, err)
+ }
+ }
+ if err := executor.checkRemediationLockForDispatch("action-2", "agent-unrelated", "restart", false); err != nil {
+ t.Fatalf("an ID no succession names was refused: %v", err)
+ }
+}
diff --git a/internal/unifiedresources/canonical_id_succession_test.go b/internal/unifiedresources/canonical_id_succession_test.go
index 132f62f009..92ef371d3d 100644
--- a/internal/unifiedresources/canonical_id_succession_test.go
+++ b/internal/unifiedresources/canonical_id_succession_test.go
@@ -2,6 +2,8 @@ package unifiedresources
import (
"fmt"
+ "reflect"
+ "sort"
"testing"
"time"
)
@@ -54,8 +56,8 @@ func TestSQLiteApplyCanonicalIDSuccessions(t *testing.T) {
t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)
}
- if _, found, err := store.GetResourceOperatorState(oldID); err != nil || found {
- t.Fatalf("operator state still keyed by superseded ID (found=%v, err=%v)", found, err)
+ if _, found := operatorStateRow(t, store, oldID); found {
+ t.Fatalf("operator state still keyed by superseded ID")
}
state, found, err := store.GetResourceOperatorState(newID)
if err != nil || !found {
@@ -262,8 +264,8 @@ func TestCanonicalIDSuccessionRekeysAPredecessorOnce(t *testing.T) {
t.Fatalf("the first successor lost the lock: found=%v err=%v state=%+v", found, err, state)
}
for _, id := range []string{"once-old", "once-second"} {
- if state, found, err := store.GetResourceOperatorState(id); err != nil || found {
- t.Fatalf("%s holds a row after the predecessor re-keyed once: found=%v err=%v state=%+v", id, found, err, state)
+ if state, found := operatorStateRow(t, store, id); found {
+ t.Fatalf("%s holds a row after the predecessor re-keyed once: state=%+v", id, state)
}
}
})
@@ -471,8 +473,8 @@ func TestCanonicalIDSuccessionKeepsLinkedMembersTogether(t *testing.T) {
t.Fatalf("%s lost the lock across a chain declared in one batch", id)
}
}
- if state, found, err := store.GetResourceOperatorState("fwd-b"); err != nil || found {
- t.Fatalf("the re-keyed intermediate ID was given a row: found=%v err=%v state=%+v", found, err, state)
+ if state, found := operatorStateRow(t, store, "fwd-b"); found {
+ t.Fatalf("the re-keyed intermediate ID was given a row: state=%+v", state)
}
})
t.Run(name+"/predecessor linked to its successor keeps its whole row", func(t *testing.T) {
@@ -543,3 +545,408 @@ func TestSQLiteSuccessionAndUpgradeCarryIgnoreMalformedLinkRows(t *testing.T) {
t.Fatalf("the upgrade carry did not complete past the unrelated link row: markers=%d err=%v", recorded, err)
}
}
+
+// retiredIDTestStores builds each store the succession record lives in.
+func retiredIDTestStores() map[string]func(t *testing.T) ResourceStore {
+ return map[string]func(t *testing.T) ResourceStore{
+ "memory": func(t *testing.T) ResourceStore { return NewMemoryStore() },
+ "sqlite": func(t *testing.T) ResourceStore {
+ store, err := NewSQLiteResourceStore(t.TempDir(), "default")
+ if err != nil {
+ t.Fatalf("NewSQLiteResourceStore: %v", err)
+ }
+ t.Cleanup(func() { _ = store.Close() })
+ return store
+ },
+ }
+}
+
+// operatorStateRowIDs lists the IDs that hold a stored operator-state row, which
+// GetResourceOperatorState does not: it answers for an ID a succession retired
+// with the lock its successor holds, though the retired ID holds nothing.
+func operatorStateRowIDs(t *testing.T, store ResourceStore) []string {
+ t.Helper()
+ states, err := store.ListResourceOperatorStates()
+ if err != nil {
+ t.Fatalf("ListResourceOperatorStates: %v", err)
+ }
+ ids := make([]string, 0, len(states))
+ for _, state := range states {
+ ids = append(ids, state.CanonicalID)
+ }
+ sort.Strings(ids)
+ return ids
+}
+
+// operatorStateRow returns the row an ID itself holds, for the tests that assert
+// where a succession left a row rather than what a read of an ID answers.
+func operatorStateRow(t *testing.T, store ResourceStore, id string) (ResourceOperatorState, bool) {
+ t.Helper()
+ states, err := store.ListResourceOperatorStates()
+ if err != nil {
+ t.Fatalf("ListResourceOperatorStates: %v", err)
+ }
+ for _, state := range states {
+ if state.CanonicalID == id {
+ return state, true
+ }
+ }
+ return ResourceOperatorState{}, false
+}
+
+// A rebuild applies a succession to the store while the previous registry
+// generation, which still lists the retired ID, keeps serving. The row moved
+// with the succession, so a read of the retired ID has to find the lock where
+// it went: the dispatch gates and the planner read the row of the ID the
+// generation they consult lists, and a retired ID that answers "no row" lets
+// Pulse act on a resource the operator locked.
+func TestRetiredCanonicalIDReadsTheLockItsSuccessorHolds(t *testing.T) {
+ locks := map[string]ResourceOperatorState{
+ "never auto-remediate": {NeverAutoRemediate: true},
+ "retired": {LifecycleState: LifecycleStateRetired},
+ }
+ for storeName, newStore := range retiredIDTestStores() {
+ for lockName, lock := range locks {
+ t.Run(storeName+"/"+lockName, func(t *testing.T) {
+ store := newStore(t)
+ const oldID, newID = "retired-old", "retired-new"
+ setAt := time.Now().UTC().Add(-72 * time.Hour).Truncate(time.Second)
+
+ predecessor := lock
+ predecessor.CanonicalID = oldID
+ predecessor.Criticality = CriticalityHigh
+ predecessor.Note = "the operator's own words"
+ predecessor.SetAt, predecessor.SetBy = setAt, "operator"
+ if err := store.SetResourceOperatorState(predecessor); err != nil {
+ t.Fatalf("seed predecessor: %v", err)
+ }
+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{{OldCanonicalID: oldID, NewCanonicalID: newID}}); err != nil {
+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)
+ }
+
+ moved, found, err := store.GetResourceOperatorState(newID)
+ if err != nil || !found || moved.Note != predecessor.Note {
+ t.Fatalf("the predecessor's row did not move to %s: found=%v err=%v state=%+v", newID, found, err, moved)
+ }
+ got, found, err := store.GetResourceOperatorState(oldID)
+ if err != nil {
+ t.Fatalf("read operator state of %s: %v", oldID, err)
+ }
+ if !found || !got.BlocksRemediation() {
+ t.Fatalf("the retired ID %s reads no lock although %s holds it: found=%v state=%+v", oldID, newID, found, got)
+ }
+ // Only the block crosses, as it does for a link fold: retirement
+ // and the rest of the row are monitoring posture the retired ID
+ // never carried over.
+ if got.CanonicalID != oldID || !got.NeverAutoRemediate || got.LifecycleState == LifecycleStateRetired ||
+ got.Criticality != "" || got.Note != "" || got.SetBy != RemediationLockCarriedBy || !got.SetAt.Equal(setAt) {
+ t.Fatalf("the retired ID reads more or less than the carried block: %+v", got)
+ }
+ // The policy revision of a plan hashes this state, so it must not
+ // change between two reads.
+ if again, _, _ := store.GetResourceOperatorState(oldID); !reflect.DeepEqual(again, got) {
+ t.Fatalf("two reads of the retired ID differ: %+v vs %+v", got, again)
+ }
+ if ids := operatorStateRowIDs(t, store); !reflect.DeepEqual(ids, []string{newID}) {
+ t.Fatalf("reading the retired ID wrote a row; rows are %v", ids)
+ }
+
+ // A row the operator writes on the retired ID is theirs to decide
+ // with: it answers, and clearing it hands the read back.
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: oldID, Note: "explicit", SetAt: setAt, SetBy: "operator"}); err != nil {
+ t.Fatalf("operator writes the retired ID: %v", err)
+ }
+ if explicit, found, err := store.GetResourceOperatorState(oldID); err != nil || !found || explicit.BlocksRemediation() || explicit.Note != "explicit" {
+ t.Fatalf("an explicit row on the retired ID did not answer: found=%v err=%v state=%+v", found, err, explicit)
+ }
+ if err := store.ClearResourceOperatorState(oldID); err != nil {
+ t.Fatalf("operator clears the retired ID: %v", err)
+ }
+ if again, found, err := store.GetResourceOperatorState(oldID); err != nil || !found || !again.BlocksRemediation() {
+ t.Fatalf("clearing the explicit row did not hand the read back to the successor: found=%v err=%v state=%+v", found, err, again)
+ }
+
+ // The lock is the operator's to lift on the successor, and the
+ // retired ID must not keep reporting what was lifted, whether the
+ // row was edited or cleared.
+ lifted := moved
+ lifted.NeverAutoRemediate = false
+ lifted.LifecycleState = LifecycleStateActive
+ if err := store.SetResourceOperatorState(lifted); err != nil {
+ t.Fatalf("operator lifts the lock on %s: %v", newID, err)
+ }
+ if after, found, err := store.GetResourceOperatorState(oldID); err != nil || found {
+ t.Fatalf("the retired ID still reports a lock the operator lifted: found=%v err=%v state=%+v", found, err, after)
+ }
+ if err := store.SetResourceOperatorState(moved); err != nil {
+ t.Fatalf("operator locks %s again: %v", newID, err)
+ }
+ if again, found, err := store.GetResourceOperatorState(oldID); err != nil || !found || !again.BlocksRemediation() {
+ t.Fatalf("the retired ID did not follow a lock set on the successor: found=%v err=%v state=%+v", found, err, again)
+ }
+ if err := store.ClearResourceOperatorState(newID); err != nil {
+ t.Fatalf("operator clears %s: %v", newID, err)
+ }
+ if after, found, err := store.GetResourceOperatorState(oldID); err != nil || found {
+ t.Fatalf("the retired ID still reports a lock the operator cleared: found=%v err=%v state=%+v", found, err, after)
+ }
+ })
+ }
+ }
+}
+
+// The era an ID leaves can itself be left again, one succession at a time or in
+// one batch, and the read follows the record to wherever the row ended up. A
+// chain is as long as the identity changes were, so nothing caps it.
+func TestRetiredCanonicalIDReadFollowsASuccessionChain(t *testing.T) {
+ const length = 150
+ chain := make([]string, length)
+ for i := range chain {
+ chain[i] = fmt.Sprintf("era-%03d", i)
+ }
+ hops := make([]CanonicalIDSuccession, 0, length-1)
+ for i := 0; i+1 < length; i++ {
+ hops = append(hops, CanonicalIDSuccession{OldCanonicalID: chain[i], NewCanonicalID: chain[i+1]})
+ }
+ apply := map[string]func(t *testing.T, store ResourceStore){
+ "one hop per succession": func(t *testing.T, store ResourceStore) {
+ for _, hop := range hops {
+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{hop}); err != nil {
+ t.Fatalf("ApplyCanonicalIDSuccessions %v: %v", hop, err)
+ }
+ }
+ },
+ "one batch": func(t *testing.T, store ResourceStore) {
+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions(hops); err != nil {
+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)
+ }
+ },
+ }
+ for storeName, newStore := range retiredIDTestStores() {
+ for applyName, applyChain := range apply {
+ t.Run(storeName+"/"+applyName, func(t *testing.T) {
+ store := newStore(t)
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: chain[0], NeverAutoRemediate: true, SetAt: time.Now().UTC(), SetBy: "operator"}); err != nil {
+ t.Fatalf("lock %s: %v", chain[0], err)
+ }
+ applyChain(t, store)
+
+ end := chain[length-1]
+ if ids := operatorStateRowIDs(t, store); !reflect.DeepEqual(ids, []string{end}) {
+ t.Fatalf("the row did not end up on %s: rows are %v", end, ids)
+ }
+ for _, id := range []string{chain[0], chain[1], chain[length/2], chain[length-2], end} {
+ if state, found, err := store.GetResourceOperatorState(id); err != nil || !found || !state.BlocksRemediation() {
+ t.Fatalf("%s does not read the lock %s holds: found=%v err=%v state=%+v", id, end, found, err, state)
+ }
+ }
+ if state, found, err := store.GetResourceOperatorState("era-unrelated"); err != nil || found {
+ t.Fatalf("an ID no succession names read a row: found=%v err=%v state=%+v", found, err, state)
+ }
+
+ if err := store.ClearResourceOperatorState(end); err != nil {
+ t.Fatalf("operator clears %s: %v", end, err)
+ }
+ for _, id := range []string{chain[0], chain[length/2], end} {
+ if state, found, err := store.GetResourceOperatorState(id); err != nil || found {
+ t.Fatalf("%s still reads a lock the operator cleared on %s: found=%v err=%v state=%+v", id, end, found, err, state)
+ }
+ }
+ })
+ }
+ }
+}
+
+// Successions are recorded by predecessor, so two eras can name each other
+// (a reporter that gains a key, loses it and gains it again). A read of an ID
+// on such a loop ends, and finds the row wherever the loop left it.
+func TestRetiredCanonicalIDReadEndsOnARecordedLoop(t *testing.T) {
+ for storeName, newStore := range retiredIDTestStores() {
+ t.Run(storeName, func(t *testing.T) {
+ store := newStore(t)
+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{
+ {OldCanonicalID: "loop-a", NewCanonicalID: "loop-b"},
+ {OldCanonicalID: "loop-b", NewCanonicalID: "loop-c"},
+ {OldCanonicalID: "loop-c", NewCanonicalID: "loop-a"},
+ }); err != nil {
+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)
+ }
+ readsEnd := func(id string, wantLock bool) {
+ t.Helper()
+ done := make(chan struct{})
+ go func() {
+ defer close(done)
+ state, found, err := store.GetResourceOperatorState(id)
+ if err != nil {
+ t.Errorf("read operator state of %s: %v", id, err)
+ return
+ }
+ if got := found && state.BlocksRemediation(); got != wantLock {
+ t.Errorf("%s: lock read = %v, want %v (found=%v state=%+v)", id, got, wantLock, found, state)
+ }
+ }()
+ select {
+ case <-done:
+ case <-time.After(10 * time.Second):
+ t.Fatalf("reading %s did not end on a loop of recorded successions", id)
+ }
+ }
+ for _, id := range []string{"loop-a", "loop-b", "loop-c"} {
+ readsEnd(id, false)
+ }
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "loop-b", NeverAutoRemediate: true, SetAt: time.Now().UTC()}); err != nil {
+ t.Fatalf("lock loop-b: %v", err)
+ }
+ for _, id := range []string{"loop-a", "loop-b", "loop-c"} {
+ readsEnd(id, true)
+ }
+ })
+ }
+}
+
+// A succession that lands on an ID with a row of its own leaves the
+// predecessor's row where it was, so an ID early on a chain can hold a row that
+// no longer says what the resource is. A retired ID reads what the end of its
+// chain holds, the ID the next generation will list.
+func TestRetiredCanonicalIDReadsWhatTheEndOfItsChainHolds(t *testing.T) {
+ for storeName, newStore := range retiredIDTestStores() {
+ t.Run(storeName+"/an unlocked row left behind does not hide the lock at the end", func(t *testing.T) {
+ store := newStore(t)
+ now := time.Now().UTC()
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "stale-b", Note: "set under the middle ID", SetAt: now, SetBy: "operator"}); err != nil {
+ t.Fatalf("seed the middle ID: %v", err)
+ }
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "stale-c", NeverAutoRemediate: true, SetAt: now, SetBy: "operator"}); err != nil {
+ t.Fatalf("lock the last ID: %v", err)
+ }
+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{
+ {OldCanonicalID: "stale-a", NewCanonicalID: "stale-b"},
+ {OldCanonicalID: "stale-b", NewCanonicalID: "stale-c"},
+ }); err != nil {
+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)
+ }
+ if _, left := operatorStateRow(t, store, "stale-b"); !left {
+ t.Fatal("the middle ID's row moved; the fixture no longer leaves one behind")
+ }
+ if state, found, err := store.GetResourceOperatorState("stale-a"); err != nil || !found || !state.BlocksRemediation() {
+ t.Fatalf("a row left on the way hid the lock at the end of the chain: found=%v err=%v state=%+v", found, err, state)
+ }
+ })
+ t.Run(storeName+"/a lock left behind is not read once the end lifted it", func(t *testing.T) {
+ store := newStore(t)
+ now := time.Now().UTC()
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "lift-a", NeverAutoRemediate: true, SetAt: now, SetBy: "operator"}); err != nil {
+ t.Fatalf("lock the first ID: %v", err)
+ }
+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "lift-c", Note: "set under the last ID", SetAt: now, SetBy: "operator"}); err != nil {
+ t.Fatalf("seed the last ID: %v", err)
+ }
+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{
+ {OldCanonicalID: "lift-a", NewCanonicalID: "lift-b"},
+ {OldCanonicalID: "lift-b", NewCanonicalID: "lift-c"},
+ }); err != nil {
+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)
+ }
+ end, found, err := store.GetResourceOperatorState("lift-c")
+ if err != nil || !found || !end.BlocksRemediation() {
+ t.Fatalf("the last ID did not take the lock: found=%v err=%v state=%+v", found, err, end)
+ }
+ if left, found := operatorStateRow(t, store, "lift-b"); !found || !left.BlocksRemediation() {
+ t.Fatalf("the middle ID's locked row moved; the fixture no longer leaves one behind: found=%v state=%+v", found, left)
+ }
+ if state, found, err := store.GetResourceOperatorState("lift-a"); err != nil || !found || !state.BlocksRemediation() {
+ t.Fatalf("the first ID does not read the lock the last holds: found=%v err=%v state=%+v", found, err, state)
+ }
+
+ lifted := end
+ lifted.NeverAutoRemediate = false
+ if err := store.SetResourceOperatorState(lifted); err != nil {
+ t.Fatalf("operator lifts the lock on lift-c: %v", err)
+ }
+ if state, found, err := store.GetResourceOperatorState("lift-a"); err != nil || found {
+ t.Fatalf("the first ID still reads a lock the operator lifted on the last: found=%v err=%v state=%+v", found, err, state)
+ }
+ if err := store.ClearResourceOperatorState("lift-c"); err != nil {
+ t.Fatalf("operator clears lift-c: %v", err)
+ }
+ if state, found, err := store.GetResourceOperatorState("lift-a"); err != nil || found {
+ t.Fatalf("the first ID still reads a lock the operator cleared on the last: found=%v err=%v state=%+v", found, err, state)
+ }
+ })
+ }
+}
+
+// The lock can move between two IDs on a chain that loops while a read walks
+// it, since every move commits on its own. At every instant one of the two holds
+// the lock (the mover writes the new holder before it clears the old one), so a
+// read of either ID that answers "not locked" has walked the chain across a
+// move instead of reading one state.
+func TestRetiredCanonicalIDReadSeesOneStateWhileTheLockMoves(t *testing.T) {
+ for storeName, newStore := range retiredIDTestStores() {
+ t.Run(storeName, func(t *testing.T) {
+ store := newStore(t)
+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{
+ {OldCanonicalID: "move-a", NewCanonicalID: "move-b"},
+ {OldCanonicalID: "move-b", NewCanonicalID: "move-a"},
+ }); err != nil {
+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)
+ }
+ lock := func(id string) ResourceOperatorState {
+ return ResourceOperatorState{CanonicalID: id, NeverAutoRemediate: true, SetAt: time.Now().UTC(), SetBy: "operator"}
+ }
+ if err := store.SetResourceOperatorState(lock("move-b")); err != nil {
+ t.Fatalf("lock move-b: %v", err)
+ }
+
+ stop := make(chan struct{})
+ moverDone := make(chan error, 1)
+ go func() {
+ holder, other := "move-b", "move-a"
+ for {
+ select {
+ case <-stop:
+ moverDone <- nil
+ return
+ default:
+ }
+ if err := store.SetResourceOperatorState(lock(other)); err != nil {
+ moverDone <- err
+ return
+ }
+ if err := store.ClearResourceOperatorState(holder); err != nil {
+ moverDone <- err
+ return
+ }
+ holder, other = other, holder
+ }
+ }()
+
+ reads, misses := 0, 0
+ var firstMiss string
+ deadline := time.Now().Add(2 * time.Second)
+ for time.Now().Before(deadline) {
+ for _, id := range []string{"move-a", "move-b"} {
+ state, found, err := store.GetResourceOperatorState(id)
+ if err != nil {
+ close(stop)
+ <-moverDone
+ t.Fatalf("read operator state of %s: %v", id, err)
+ }
+ reads++
+ if !found || !state.BlocksRemediation() {
+ if misses++; firstMiss == "" {
+ firstMiss = fmt.Sprintf("%s: found=%v state=%+v", id, found, state)
+ }
+ }
+ }
+ }
+ close(stop)
+ if err := <-moverDone; err != nil {
+ t.Fatalf("moving the lock: %v", err)
+ }
+ if misses > 0 {
+ t.Fatalf("%d of %d reads answered unlocked while the lock was held throughout; first: %s", misses, reads, firstMiss)
+ }
+ })
+ }
+}
diff --git a/internal/unifiedresources/registry_test.go b/internal/unifiedresources/registry_test.go
index 55a16a6e69..3a15133ada 100644
--- a/internal/unifiedresources/registry_test.go
+++ b/internal/unifiedresources/registry_test.go
@@ -6583,8 +6583,8 @@ func TestCanonicalIDSuccessionMovesOperatorState(t *testing.T) {
}
registry.PersistIdentityPins()
- if _, found, err := store.GetResourceOperatorState(shortEraID); err != nil || found {
- t.Fatalf("operator state still keyed by superseded ID (found=%v, err=%v)", found, err)
+ if _, found := operatorStateRow(t, store, shortEraID); found {
+ t.Fatalf("operator state still keyed by superseded ID")
}
state, found, err := store.GetResourceOperatorState(newID)
if err != nil || !found {
@@ -6683,8 +6683,8 @@ func TestIngestRecordsRecordDeclaredSuccessions(t *testing.T) {
}})
newID := SourceSpecificID(ResourceTypeStorage, SourceTrueNAS, "system:conn-a/pool:tank")
- if _, found, err := store.GetResourceOperatorState(oldID); err != nil || found {
- t.Fatalf("expected operator state to leave superseded ID (found=%v err=%v)", found, err)
+ if _, found := operatorStateRow(t, store, oldID); found {
+ t.Fatalf("expected operator state to leave superseded ID")
}
state, found, err := store.GetResourceOperatorState(newID)
if err != nil || !found || !state.NeverAutoRemediate {
@@ -6958,7 +6958,7 @@ func TestProxmoxGuestSuccessionRekeysOperatorState(t *testing.T) {
if !state.NeverAutoRemediate {
t.Fatalf("re-keyed operator state lost its fields: %+v", state)
}
- if _, stale, _ := store.GetResourceOperatorState(legacyID); stale {
+ if _, stale := operatorStateRow(t, store, legacyID); stale {
t.Fatalf("operator state still present under retired ID %q", legacyID)
}
}
@@ -8940,8 +8940,8 @@ func TestSuccessionBatchOrderKeepsLinkedRemediationLock(t *testing.T) {
t.Fatalf("%s lost the lock across a batch that returned to its start", id)
}
}
- if state, found, err := store.GetResourceOperatorState("loop-b"); err != nil || found {
- t.Fatalf("the re-keyed intermediate ID was given a row: found=%v err=%v state=%+v", found, err, state)
+ if state, found := operatorStateRow(t, store, "loop-b"); found {
+ t.Fatalf("the re-keyed intermediate ID was given a row: state=%+v", state)
}
})
}
@@ -9605,3 +9605,137 @@ func TestManualLinkChainCycleLinksAreAllCutByAReportedMember(t *testing.T) {
}
}
}
+
+// successionHookStore runs a callback from inside a rebuild, right after the
+// store applied a canonical-ID succession and before the rebuilt registry is
+// published: the moment the previous generation is still served while the
+// store already answers for the successor.
+type successionHookStore struct {
+ ResourceStore
+ afterSuccession func()
+}
+
+func (s *successionHookStore) ApplyCanonicalIDSuccessions(successions []CanonicalIDSuccession) error {
+ err := s.ResourceStore.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions(successions)
+ if err == nil && s.afterSuccession != nil {
+ s.afterSuccession()
+ }
+ return err
+}
+
+// A rebuild applies a succession to the store (the operator's rows move to the
+// successor) before the generation that lists the successor is published, and
+// the previous generation keeps serving until then. The dispatch gates and the
+// planner read the operator state of the ID the generation they consult lists,
+// so in that window they ask for an ID whose row has gone. A lock the operator
+// set must still be found there: the old ID is the resource the previous
+// generation would act on.
+func TestRemediationLockHoldsOnTheRetiredIDWhileThePreviousGenerationServes(t *testing.T) {
+ now := time.Now().UTC()
+ type scenario struct {
+ before, after models.StateSnapshot
+ // kind is the resource type that changes ID.
+ kind ResourceType
+ }
+ scenarios := map[string]scenario{
+ // A record-declared era (the reporter's source-specific ID), applied
+ // while the snapshot is ingested.
+ "host agent gaining a machine ID": {
+ kind: ResourceTypeAgent,
+ before: models.StateSnapshot{Hosts: []models.Host{{ID: "agent-window-01", Hostname: "window-01", Status: "online", LastSeen: now}}},
+ after: models.StateSnapshot{Hosts: []models.Host{{ID: "agent-window-01", Hostname: "window-01", MachineID: "machine-window-01", Status: "online", LastSeen: now}}},
+ },
+ // A pin-driven era, applied once the rebuilt registry persists its pins.
+ "swarm host gaining a full hostname": {
+ kind: ResourceTypeAgent,
+ before: models.StateSnapshot{DockerHosts: []models.DockerHost{{
+ ID: "docker-window-02", Hostname: "cloud", Status: "online", LastSeen: now,
+ Swarm: &models.DockerSwarmInfo{ClusterID: "swarm-window", ClusterName: "prod-swarm"},
+ }}},
+ after: models.StateSnapshot{DockerHosts: []models.DockerHost{{
+ ID: "docker-window-02", Hostname: "cloud.a", Status: "online", LastSeen: now,
+ Swarm: &models.DockerSwarmInfo{ClusterID: "swarm-window", ClusterName: "prod-swarm"},
+ }}},
+ },
+ }
+ stores := map[string]func(t *testing.T) ResourceStore{
+ "memory": func(t *testing.T) ResourceStore { return NewMemoryStore() },
+ "sqlite": func(t *testing.T) ResourceStore {
+ store, err := NewSQLiteResourceStore(t.TempDir(), "default")
+ if err != nil {
+ t.Fatalf("NewSQLiteResourceStore: %v", err)
+ }
+ t.Cleanup(func() { _ = store.Close() })
+ return store
+ },
+ }
+ locks := map[string]ResourceOperatorState{
+ "never auto-remediate": {NeverAutoRemediate: true},
+ "retired": {LifecycleState: LifecycleStateRetired},
+ }
+ idsOf := func(adapter *MonitorAdapter, kind ResourceType) []string {
+ var ids []string
+ for _, resource := range adapter.GetByType(kind) {
+ ids = append(ids, resource.ID)
+ }
+ return ids
+ }
+ for scenarioName, sc := range scenarios {
+ for storeName, newStore := range stores {
+ for lockName, lock := range locks {
+ t.Run(scenarioName+"/"+storeName+"/"+lockName, func(t *testing.T) {
+ hooked := &successionHookStore{ResourceStore: newStore(t)}
+ adapter := NewMonitorAdapter(NewRegistry(hooked))
+ adapter.PopulateFromSnapshot(sc.before)
+ before := idsOf(adapter, sc.kind)
+ if len(before) != 1 {
+ t.Fatalf("first generation lists %v, want the one resource", before)
+ }
+ oldID := before[0]
+ state := lock
+ state.CanonicalID, state.SetAt, state.SetBy = oldID, now.Add(-time.Hour), "operator"
+ if err := hooked.SetResourceOperatorState(state); err != nil {
+ t.Fatalf("lock %s: %v", oldID, err)
+ }
+
+ var (
+ applied int
+ listedDuring []string
+ foundDuring bool
+ lockedDuring bool
+ readErrDuring error
+ )
+ hooked.afterSuccession = func() {
+ applied++
+ listedDuring = idsOf(adapter, sc.kind)
+ var got ResourceOperatorState
+ got, foundDuring, readErrDuring = adapter.GetResourceOperatorState(oldID)
+ lockedDuring = foundDuring && got.BlocksRemediation()
+ }
+ adapter.PopulateFromSnapshot(sc.after)
+
+ if applied == 0 {
+ t.Fatalf("the rebuild applied no succession; the fixture no longer changes %s's era", oldID)
+ }
+ if !slices.Equal(listedDuring, []string{oldID}) {
+ t.Fatalf("during the rebuild the served generation lists %v, want the previous generation's %s", listedDuring, oldID)
+ }
+ if readErrDuring != nil {
+ t.Fatalf("read operator state of %s during the rebuild: %v", oldID, readErrDuring)
+ }
+ if !lockedDuring {
+ t.Fatalf("while the previous generation still lists %s the lock the operator set on it reads as missing (found=%v)", oldID, foundDuring)
+ }
+
+ after := idsOf(adapter, sc.kind)
+ if len(after) != 1 || after[0] == oldID {
+ t.Fatalf("the published generation lists %v, want the resource under a new ID", after)
+ }
+ if got, found, err := adapter.GetResourceOperatorState(after[0]); err != nil || !found || !got.BlocksRemediation() {
+ t.Fatalf("the published ID %s does not hold the lock: found=%v err=%v state=%+v", after[0], found, err, got)
+ }
+ })
+ }
+ }
+ }
+}
diff --git a/internal/unifiedresources/remediation_lock_carry.go b/internal/unifiedresources/remediation_lock_carry.go
index 109d8f0aeb..964a38107a 100644
--- a/internal/unifiedresources/remediation_lock_carry.go
+++ b/internal/unifiedresources/remediation_lock_carry.go
@@ -2,6 +2,7 @@ package unifiedresources
import (
"database/sql"
+ "errors"
"fmt"
"log"
"sort"
@@ -17,6 +18,93 @@ import (
// component changes (a link, an unlink) and whenever a succession lands on one
// of them. CarryRemediationLock owns what is written onto a row.
+// A rebuild applies a succession to the store before it publishes the registry
+// generation that lists the successor, and the previous generation, which still
+// lists the retired ID, keeps serving in between. The gates read the operator
+// state of the ID they are asked about, and the succession moved that ID's row,
+// so a retired ID that answers "no row" lets Pulse act on a resource the
+// operator locked. A read of an ID that holds no row therefore follows the
+// recorded successions (canonical_id_successions) to the end of its chain and
+// answers as the ID the chain ends on would: with the remediation block that
+// ID's row carries, the block alone as a link fold carries it, attributed to the
+// identity change and dated by the row it came from so two reads agree (a
+// plan's policy revision hashes the state). The end of the chain decides, so a
+// lock the operator lifted on the successor, by editing or clearing its row, is
+// not read back from a stale row left behind on an ID before it; an ID that
+// holds a row of its own answers with it; and a chain that returns to an ID it
+// has seen has no end, so a lock held anywhere on it counts.
+func followRetiredID(
+ retiredID string,
+ successorOf func(id string) (successor string, retired bool, err error),
+ rowOf func(id string) (ResourceOperatorState, bool, error),
+) (ResourceOperatorState, bool, error) {
+ seen := map[string]struct{}{retiredID: {}}
+ var end, nearestLock ResourceOperatorState
+ var endFound, lockFound bool
+ for current := retiredID; ; {
+ successor, retired, err := successorOf(current)
+ if err != nil {
+ return ResourceOperatorState{}, false, err
+ }
+ successor = CanonicalResourceID(successor)
+ if !retired || successor == "" {
+ if endFound {
+ lock, carried := retiredIDRemediationLock(retiredID, end)
+ return lock, carried, nil
+ }
+ return ResourceOperatorState{}, false, nil
+ }
+ if _, again := seen[successor]; again {
+ if lockFound {
+ lock, carried := retiredIDRemediationLock(retiredID, nearestLock)
+ return lock, carried, nil
+ }
+ return ResourceOperatorState{}, false, nil
+ }
+ seen[successor] = struct{}{}
+ row, found, err := rowOf(successor)
+ if err != nil {
+ return ResourceOperatorState{}, false, err
+ }
+ end, endFound = row, found
+ if found && !lockFound && row.BlocksRemediation() {
+ nearestLock, lockFound = row, true
+ }
+ current = successor
+ }
+}
+
+// retiredIDRemediationLock is what a retired ID reads of the row an ID on its
+// chain holds: that row's remediation block, if it has one, and nothing else.
+func retiredIDRemediationLock(retiredID string, holder ResourceOperatorState) (ResourceOperatorState, bool) {
+ return CarryRemediationLock(retiredID, ResourceOperatorState{}, false, holder, true, holder.SetAt)
+}
+
+// retiredIDRemediationLockSQL is the read above for the SQLite store: one
+// statement per hop, on the connection the succession's transaction commits on,
+// so a read that finds the row gone from an ID also finds the succession that
+// moved it.
+func retiredIDRemediationLockSQL(queryer resourceOperatorStateQueryRower, canonicalID string) (ResourceOperatorState, bool, error) {
+ retiredID := CanonicalResourceID(canonicalID)
+ if retiredID == "" {
+ return ResourceOperatorState{}, false, nil
+ }
+ return followRetiredID(retiredID,
+ func(id string) (string, bool, error) {
+ var successor string
+ err := queryer.QueryRow(`SELECT new_canonical_id FROM canonical_id_successions WHERE old_canonical_id = ?`, id).Scan(&successor)
+ if errors.Is(err, sql.ErrNoRows) {
+ return "", false, nil
+ }
+ if err != nil {
+ return "", false, fmt.Errorf("query canonical ID succession of %q: %w", id, err)
+ }
+ return successor, true, nil
+ },
+ func(id string) (ResourceOperatorState, bool, error) { return getResourceOperatorStateSQL(queryer, id) },
+ )
+}
+
// carriedLock records one remediation block written onto `to` from `from`.
type carriedLock struct {
to string
diff --git a/internal/unifiedresources/store.go b/internal/unifiedresources/store.go
index a67f5b03a4..9d1021bf66 100644
--- a/internal/unifiedresources/store.go
+++ b/internal/unifiedresources/store.go
@@ -3307,9 +3307,33 @@ func (s *SQLiteResourceStore) GetExportAudits(since time.Time, limit int) ([]Exp
// canonical ID. Returns (zero, false, nil) on no entry; (state, true,
// nil) on a hit; (zero, false, error) on a query failure. The found
// signal is meaningful for the API GET path which returns 404 vs
-// returning the default no-state record.
+// returning the default no-state record. An ID that holds no row but that a
+// canonical-ID succession retired answers with the remediation block its
+// successor carries (followRetiredID), which is what a registry
+// generation still listing the retired ID needs while a rebuild is in flight.
func (s *SQLiteResourceStore) GetResourceOperatorState(canonicalID string) (ResourceOperatorState, bool, error) {
- return getResourceOperatorStateSQL(s.db, canonicalID)
+ state, found, err := getResourceOperatorStateSQL(s.db, canonicalID)
+ if err != nil || found {
+ return state, found, err
+ }
+ return s.retiredIDOperatorState(canonicalID)
+}
+
+// retiredIDOperatorState answers for an ID that held no row when it was first
+// read. The walk down its chain is several statements, and a succession that
+// commits between two of them can move the lock off an ID the walk already
+// passed onto one it has seen (a chain that loops back), so the ID's own row and
+// every hop are read in one transaction, which is one snapshot.
+func (s *SQLiteResourceStore) retiredIDOperatorState(canonicalID string) (ResourceOperatorState, bool, error) {
+ tx, err := s.db.Begin()
+ if err != nil {
+ return ResourceOperatorState{}, false, fmt.Errorf("begin resource operator state read: %w", err)
+ }
+ defer func() { _ = tx.Rollback() }()
+ if state, found, err := getResourceOperatorStateSQL(tx, canonicalID); err != nil || found {
+ return state, found, err
+ }
+ return retiredIDRemediationLockSQL(tx, canonicalID)
}
type resourceOperatorStateQueryRower interface {
@@ -4683,7 +4707,9 @@ func (m *MemoryStore) GetExportAudits(since time.Time, limit int) ([]ExportAudit
// canonical resource ID. The found bool distinguishes "no entry" (the
// default no-state posture) from "explicit empty entry the operator
// cleared" — both have the same effective behavior but the latter may
-// be surfaced differently on the audit timeline.
+// be surfaced differently on the audit timeline. An ID that holds no row but
+// that a canonical-ID succession retired answers as the SQLite store does, with
+// the remediation block its successor carries.
func (m *MemoryStore) GetResourceOperatorState(canonicalID string) (ResourceOperatorState, bool, error) {
canonicalID = strings.TrimSpace(canonicalID)
if canonicalID == "" {
@@ -4693,11 +4719,28 @@ func (m *MemoryStore) GetResourceOperatorState(canonicalID string) (ResourceOper
defer m.mu.RUnlock()
state, ok := m.resourceOperatorState[canonicalID]
if !ok {
- return ResourceOperatorState{}, false, nil
+ lock, carried := m.retiredIDRemediationLockLocked(canonicalID)
+ return lock, carried, nil
}
return state, true, nil
}
+// retiredIDRemediationLockLocked is the read described at followRetiredID over
+// the in-memory succession record; callers hold m.mu.
+func (m *MemoryStore) retiredIDRemediationLockLocked(retiredID string) (ResourceOperatorState, bool) {
+ lock, carried, _ := followRetiredID(retiredID,
+ func(id string) (string, bool, error) {
+ successor, retired := m.canonicalSuccessions[id]
+ return successor, retired, nil
+ },
+ func(id string) (ResourceOperatorState, bool, error) {
+ state, found := m.resourceOperatorState[id]
+ return state, found, nil
+ },
+ )
+ return lock, carried
+}
+
// SetResourceOperatorState upserts the supplied state, validating it
// before persistence. Callers should pass a fully-populated record;
// per-field merge is not supported, so reading the existing state and