Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep a remediation lock readable on an ID a succession retired
rcourtmanb329730e6bd6
diff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.mdindex 540d0ec4e4..7b9e971aa6 100644--- a/docs/release-control/v6/internal/subsystems/unified-resources.md+++ b/docs/release-control/v6/internal/subsystems/unified-resources.md@@ -3635,6 +3635,43 @@ until then. The store cannot see which links the registry declines to foldso those share too, which can lock a host because a linked probe was retired.Re-recording an existing link shares again.+A rebuild applies the successions it finds to the store before it publishes+the generation that lists the successors, and the generation it replaces keeps+serving in between. The planner, the Patrol broker and the dispatch gates+(`validateExecutionPolicy`, `isResourceRemediationLocked`) read the row of the ID+that generation lists, so for the length of the rebuild they ask for an ID whose+row has just moved. `GetResourceOperatorState` therefore follows the recorded+successions (`canonical_id_successions`) in both stores when the ID it is asked+about holds no row (`followRetiredID`, one walker for both): it answers as the+ID its chain ends on would, with that row's remediation block alone+(`retiredIDRemediationLock`, which is `CarryRemediationLock` into an empty+survivor), as `NeverAutoRemediate`, attributed `system:identity-change` and+dated by the row it came from so two reads agree (a plan's policy revision+hashes the state). Nothing is written, so `ListResourceOperatorStates` still+lists only the rows the stores hold. The end of the chain decides, because it is+the ID the next generation lists: a lock the operator lifted there, by editing+or clearing its row, is not read back from the stale row a succession leaves on+an ID before it when its successor already had a row, and an unlocked stale row+does not hide a lock held at the end. An ID that holds a row of its own answers+with that row. A chain is as long as the identity changes were and has no cap;+a chain that returns to an ID it has seen has no end, so a lock held anywhere on+it counts. Only the block crosses, so while the window is open the retired ID+reads none of the row's other settings (retirement as a lifecycle state,+maintenance windows, criticality, notes), as it read none before. The walk is+one snapshot: the SQLite store reads the ID's own row and every hop in one+transaction, because a succession that commits between two of the statements+can move the lock onto an ID the walk has already passed. The window ends when+the new generation publishes: that generation lists the successor, and a+lifecycle plan made against the retired ID drifts (`resource ... is no longer+present`). An operator edit to the retired ID inside the window goes to the+retired ID's own row, not the successor's: the operator-state API writes the ID+the registry it consults lists, and a succession applies once per predecessor.+A lock set there does not reach the successor, which the next generation lists+and reads, and a lock lifted there leaves the successor's lock in place, which+is the safe direction; the retired ID's own row still answers reads of that ID.+A retired ID that a different resource later mints again, with no row of its+own, reads the successor's lock until the operator writes it a row.+A host or Docker reporter that had no machine ID, DMI UUID or cluster slot iskeyed by its source-specific ID (unless a link or match folds it into anotherresource) and held no identity pin (a pin needs one of those keys), so when a@@ -3667,10 +3704,10 @@ era of every keyed reporter that passes those checks, in the batches of itsingest passes, and also restores a lock or row an earlier keyless-to-keyedchange left behind, even where the operator has edited the keyed row since. An ID held under a report ID the reporter no longer sends isnot recomputable. Like every succession, the move happens while the previous-generation is still being served, so when the keyed ID had no row an action-planned against the old ID in that window finds no row there, and a link the-succession rewrites takes effect in the next generation, because a registry-loads its links when it is built (the lock is shared across the pair at once).+generation is still being served, which is why a read of the old ID follows the+succession (above); a link the succession rewrites takes effect in the next+generation, because a registry loads its links when it is built (the lock is+shared across the pair at once).Canonical ID changes that reach neither a link nor `ApplyCanonicalIDSuccessions`carry nothing: a physical disk re-keyed in place to a machine-scoped ID@@ -3686,6 +3723,7 @@ array disk, a record-fed disk), and `TestProviderDiskRecordsAdvertiseNoActionCapin `internal/truenas/provider_test.go` when the TrueNAS provider's records do.The disk's other operator settings (retirement, maintenance, notes) areorphaned by the same re-key and are not carried. Proof:+`TestRemediationLockHoldsOnTheRetiredIDWhileThePreviousGenerationServes`,`TestReporterGainingAStrongKeyKeepsItsRemediationLock`,`TestReporterEraIsDeclaredOnlyForAKeyTheReporterGains`,`TestPhysicalDiskResourcesAdvertiseNoActionCapability`,@@ -3696,6 +3734,10 @@ orphaned by the same re-key and are not carried. Proof:`TestSQLiteStoreRestoresLocksAcrossRecordedSuccessionChains` in`internal/unifiedresources/registry_test.go`, and`TestCanonicalIDSuccessionCarriesRemediationLockOntoExistingSuccessorRow`,+`TestRetiredCanonicalIDReadsTheLockItsSuccessorHolds`,+`TestRetiredCanonicalIDReadFollowsASuccessionChain`,+`TestRetiredCanonicalIDReadsWhatTheEndOfItsChainHolds`,+`TestRetiredCanonicalIDReadEndsOnARecordedLoop`,`TestCanonicalIDSuccessionRekeysAPredecessorOnce`,`TestCanonicalIDSuccessionLockReachesLinkedSurvivor` and`TestCanonicalIDSuccessionKeepsLinkedMembersTogether` indiff --git a/internal/actionlifecycle/service_test.go b/internal/actionlifecycle/service_test.goindex d1d7cc7f5f..168ef3fab1 100644--- a/internal/actionlifecycle/service_test.go+++ b/internal/actionlifecycle/service_test.go@@ -1473,6 +1473,60 @@ func TestExecuteRefusesRemediationLockedResource(t *testing.T) {}}+// A rebuild moves the operator's rows to the successor ID before it publishes+// the registry generation that lists it, so an action planned against the old+// ID can be dispatched, and the planner can be asked about it, while the+// registry the lifecycle consults still lists the old ID. The lock the+// operator set on it has moved with its row; the gates must still find it.+func TestLockMovedBySuccessionStillRefusesTheIDThePreviousGenerationLists(t *testing.T) {+ locks := map[string]unified.ResourceOperatorState{+ "never auto-remediate": {NeverAutoRemediate: true},+ "retired": {LifecycleState: unified.LifecycleStateRetired},+ }+ for lockName, lock := range locks {+ t.Run(lockName, func(t *testing.T) {+ now := time.Now().UTC()+ env := newServiceEnv(t, testResource(now, unified.ApprovalNone))++ plan, err := env.service.Plan(context.Background(), "default", restartRequest(), testActionActor("requester", "default"))+ if err != nil {+ t.Fatalf("Plan: %v", err)+ }+ state := lock+ state.CanonicalID = "vm:42"+ if err := env.store.SetResourceOperatorState(state); err != nil {+ t.Fatalf("SetResourceOperatorState: %v", err)+ }+ if err := env.store.(*unified.MemoryStore).ApplyCanonicalIDSuccessions([]unified.CanonicalIDSuccession{{OldCanonicalID: "vm:42", NewCanonicalID: "vm:42-successor"}}); err != nil {+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)+ }+ if _, listed := env.registry.Get("vm:42"); !listed {+ t.Fatal("the registry no longer lists vm:42; the fixture does not model the window before the new generation publishes")+ }+ if row, found, err := env.store.GetResourceOperatorState("vm:42-successor"); err != nil || !found || !row.BlocksRemediation() {+ t.Fatalf("the successor does not hold the moved lock: found=%v err=%v state=%+v", found, err, row)+ }++ // The planner and the Patrol broker read the policy through the lifecycle.+ planned, found, err := env.service.ResourceOperatorState("default", "vm:42")+ if err != nil || !found || !planned.BlocksRemediation() {+ t.Fatalf("the policy read for vm:42 lost the lock: found=%v err=%v state=%+v", found, err, planned)+ }++ failed, err := env.service.Execute(context.Background(), "default", plan.ActionID, testActionActor("agent:test", "default"), "")+ if !errors.Is(err, unified.ErrResourceRemediationLocked) {+ t.Fatalf("error = %v, want ErrResourceRemediationLocked", err)+ }+ if env.executor.calls != 0 {+ t.Fatalf("executor must not run on a locked resource whose row moved, calls = %d", env.executor.calls)+ }+ if failed.State != unified.ActionStateFailed {+ t.Fatalf("refusal must persist a terminal failed audit, state = %q", failed.State)+ }+ })+ }+}+func TestExecuteRefusesRetiredResourceWithoutDispatch(t *testing.T) {now := time.Now().UTC()env := newServiceEnv(t, testResource(now, unified.ApprovalNone))diff --git a/internal/ai/tools/action_audit_remediation_lock_test.go b/internal/ai/tools/action_audit_remediation_lock_test.goindex 1e3ed2ee8f..65aaefba80 100644--- a/internal/ai/tools/action_audit_remediation_lock_test.go+++ b/internal/ai/tools/action_audit_remediation_lock_test.go@@ -378,3 +378,28 @@ func TestExecuteNativeActionWithAuditAllowsHumanApprovedDispatchOnUnknownLockStat.Fatalf("expected successful ExecutionResult, got %#v", result)}}++// A succession moves the operator's row to the successor ID when a rebuild+// applies it, before the registry generation listing the successor is+// published; the dispatch a model asks for in between still names the old ID.+// The lock the operator set on it must still refuse.+func TestRemediationLockGateFollowsASuccessionOffTheIDItNames(t *testing.T) {+ store := unifiedresources.NewMemoryStore()+ if err := store.SetResourceOperatorState(unifiedresources.ResourceOperatorState{CanonicalID: "agent-before-key", NeverAutoRemediate: true}); err != nil {+ t.Fatalf("lock the agent: %v", err)+ }+ if err := store.ApplyCanonicalIDSuccessions([]unifiedresources.CanonicalIDSuccession{{OldCanonicalID: "agent-before-key", NewCanonicalID: "agent-keyed"}}); err != nil {+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)+ }+ executor := NewPulseToolExecutor(ExecutorConfig{ActionAuditStore: store})++ for _, humanApproved := range []bool{false, true} {+ err := executor.checkRemediationLockForDispatch("action-1", "agent-before-key", "restart", humanApproved)+ if !errors.Is(err, unifiedresources.ErrResourceRemediationLocked) {+ t.Fatalf("humanApproved=%v: error = %v, want ErrResourceRemediationLocked", humanApproved, err)+ }+ }+ if err := executor.checkRemediationLockForDispatch("action-2", "agent-unrelated", "restart", false); err != nil {+ t.Fatalf("an ID no succession names was refused: %v", err)+ }+}diff --git a/internal/unifiedresources/canonical_id_succession_test.go b/internal/unifiedresources/canonical_id_succession_test.goindex 132f62f009..92ef371d3d 100644--- a/internal/unifiedresources/canonical_id_succession_test.go+++ b/internal/unifiedresources/canonical_id_succession_test.go@@ -2,6 +2,8 @@ package unifiedresourcesimport ("fmt"+ "reflect"+ "sort""testing""time")@@ -54,8 +56,8 @@ func TestSQLiteApplyCanonicalIDSuccessions(t *testing.T) {t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)}- if _, found, err := store.GetResourceOperatorState(oldID); err != nil || found {- t.Fatalf("operator state still keyed by superseded ID (found=%v, err=%v)", found, err)+ if _, found := operatorStateRow(t, store, oldID); found {+ t.Fatalf("operator state still keyed by superseded ID")}state, found, err := store.GetResourceOperatorState(newID)if err != nil || !found {@@ -262,8 +264,8 @@ func TestCanonicalIDSuccessionRekeysAPredecessorOnce(t *testing.T) {t.Fatalf("the first successor lost the lock: found=%v err=%v state=%+v", found, err, state)}for _, id := range []string{"once-old", "once-second"} {- if state, found, err := store.GetResourceOperatorState(id); err != nil || found {- t.Fatalf("%s holds a row after the predecessor re-keyed once: found=%v err=%v state=%+v", id, found, err, state)+ if state, found := operatorStateRow(t, store, id); found {+ t.Fatalf("%s holds a row after the predecessor re-keyed once: state=%+v", id, state)}}})@@ -471,8 +473,8 @@ func TestCanonicalIDSuccessionKeepsLinkedMembersTogether(t *testing.T) {t.Fatalf("%s lost the lock across a chain declared in one batch", id)}}- if state, found, err := store.GetResourceOperatorState("fwd-b"); err != nil || found {- t.Fatalf("the re-keyed intermediate ID was given a row: found=%v err=%v state=%+v", found, err, state)+ if state, found := operatorStateRow(t, store, "fwd-b"); found {+ t.Fatalf("the re-keyed intermediate ID was given a row: state=%+v", state)}})t.Run(name+"/predecessor linked to its successor keeps its whole row", func(t *testing.T) {@@ -543,3 +545,408 @@ func TestSQLiteSuccessionAndUpgradeCarryIgnoreMalformedLinkRows(t *testing.T) {t.Fatalf("the upgrade carry did not complete past the unrelated link row: markers=%d err=%v", recorded, err)}}++// retiredIDTestStores builds each store the succession record lives in.+func retiredIDTestStores() map[string]func(t *testing.T) ResourceStore {+ return map[string]func(t *testing.T) ResourceStore{+ "memory": func(t *testing.T) ResourceStore { return NewMemoryStore() },+ "sqlite": func(t *testing.T) ResourceStore {+ store, err := NewSQLiteResourceStore(t.TempDir(), "default")+ if err != nil {+ t.Fatalf("NewSQLiteResourceStore: %v", err)+ }+ t.Cleanup(func() { _ = store.Close() })+ return store+ },+ }+}++// operatorStateRowIDs lists the IDs that hold a stored operator-state row, which+// GetResourceOperatorState does not: it answers for an ID a succession retired+// with the lock its successor holds, though the retired ID holds nothing.+func operatorStateRowIDs(t *testing.T, store ResourceStore) []string {+ t.Helper()+ states, err := store.ListResourceOperatorStates()+ if err != nil {+ t.Fatalf("ListResourceOperatorStates: %v", err)+ }+ ids := make([]string, 0, len(states))+ for _, state := range states {+ ids = append(ids, state.CanonicalID)+ }+ sort.Strings(ids)+ return ids+}++// operatorStateRow returns the row an ID itself holds, for the tests that assert+// where a succession left a row rather than what a read of an ID answers.+func operatorStateRow(t *testing.T, store ResourceStore, id string) (ResourceOperatorState, bool) {+ t.Helper()+ states, err := store.ListResourceOperatorStates()+ if err != nil {+ t.Fatalf("ListResourceOperatorStates: %v", err)+ }+ for _, state := range states {+ if state.CanonicalID == id {+ return state, true+ }+ }+ return ResourceOperatorState{}, false+}++// A rebuild applies a succession to the store while the previous registry+// generation, which still lists the retired ID, keeps serving. The row moved+// with the succession, so a read of the retired ID has to find the lock where+// it went: the dispatch gates and the planner read the row of the ID the+// generation they consult lists, and a retired ID that answers "no row" lets+// Pulse act on a resource the operator locked.+func TestRetiredCanonicalIDReadsTheLockItsSuccessorHolds(t *testing.T) {+ locks := map[string]ResourceOperatorState{+ "never auto-remediate": {NeverAutoRemediate: true},+ "retired": {LifecycleState: LifecycleStateRetired},+ }+ for storeName, newStore := range retiredIDTestStores() {+ for lockName, lock := range locks {+ t.Run(storeName+"/"+lockName, func(t *testing.T) {+ store := newStore(t)+ const oldID, newID = "retired-old", "retired-new"+ setAt := time.Now().UTC().Add(-72 * time.Hour).Truncate(time.Second)++ predecessor := lock+ predecessor.CanonicalID = oldID+ predecessor.Criticality = CriticalityHigh+ predecessor.Note = "the operator's own words"+ predecessor.SetAt, predecessor.SetBy = setAt, "operator"+ if err := store.SetResourceOperatorState(predecessor); err != nil {+ t.Fatalf("seed predecessor: %v", err)+ }+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{{OldCanonicalID: oldID, NewCanonicalID: newID}}); err != nil {+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)+ }++ moved, found, err := store.GetResourceOperatorState(newID)+ if err != nil || !found || moved.Note != predecessor.Note {+ t.Fatalf("the predecessor's row did not move to %s: found=%v err=%v state=%+v", newID, found, err, moved)+ }+ got, found, err := store.GetResourceOperatorState(oldID)+ if err != nil {+ t.Fatalf("read operator state of %s: %v", oldID, err)+ }+ if !found || !got.BlocksRemediation() {+ t.Fatalf("the retired ID %s reads no lock although %s holds it: found=%v state=%+v", oldID, newID, found, got)+ }+ // Only the block crosses, as it does for a link fold: retirement+ // and the rest of the row are monitoring posture the retired ID+ // never carried over.+ if got.CanonicalID != oldID || !got.NeverAutoRemediate || got.LifecycleState == LifecycleStateRetired ||+ got.Criticality != "" || got.Note != "" || got.SetBy != RemediationLockCarriedBy || !got.SetAt.Equal(setAt) {+ t.Fatalf("the retired ID reads more or less than the carried block: %+v", got)+ }+ // The policy revision of a plan hashes this state, so it must not+ // change between two reads.+ if again, _, _ := store.GetResourceOperatorState(oldID); !reflect.DeepEqual(again, got) {+ t.Fatalf("two reads of the retired ID differ: %+v vs %+v", got, again)+ }+ if ids := operatorStateRowIDs(t, store); !reflect.DeepEqual(ids, []string{newID}) {+ t.Fatalf("reading the retired ID wrote a row; rows are %v", ids)+ }++ // A row the operator writes on the retired ID is theirs to decide+ // with: it answers, and clearing it hands the read back.+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: oldID, Note: "explicit", SetAt: setAt, SetBy: "operator"}); err != nil {+ t.Fatalf("operator writes the retired ID: %v", err)+ }+ if explicit, found, err := store.GetResourceOperatorState(oldID); err != nil || !found || explicit.BlocksRemediation() || explicit.Note != "explicit" {+ t.Fatalf("an explicit row on the retired ID did not answer: found=%v err=%v state=%+v", found, err, explicit)+ }+ if err := store.ClearResourceOperatorState(oldID); err != nil {+ t.Fatalf("operator clears the retired ID: %v", err)+ }+ if again, found, err := store.GetResourceOperatorState(oldID); err != nil || !found || !again.BlocksRemediation() {+ t.Fatalf("clearing the explicit row did not hand the read back to the successor: found=%v err=%v state=%+v", found, err, again)+ }++ // The lock is the operator's to lift on the successor, and the+ // retired ID must not keep reporting what was lifted, whether the+ // row was edited or cleared.+ lifted := moved+ lifted.NeverAutoRemediate = false+ lifted.LifecycleState = LifecycleStateActive+ if err := store.SetResourceOperatorState(lifted); err != nil {+ t.Fatalf("operator lifts the lock on %s: %v", newID, err)+ }+ if after, found, err := store.GetResourceOperatorState(oldID); err != nil || found {+ t.Fatalf("the retired ID still reports a lock the operator lifted: found=%v err=%v state=%+v", found, err, after)+ }+ if err := store.SetResourceOperatorState(moved); err != nil {+ t.Fatalf("operator locks %s again: %v", newID, err)+ }+ if again, found, err := store.GetResourceOperatorState(oldID); err != nil || !found || !again.BlocksRemediation() {+ t.Fatalf("the retired ID did not follow a lock set on the successor: found=%v err=%v state=%+v", found, err, again)+ }+ if err := store.ClearResourceOperatorState(newID); err != nil {+ t.Fatalf("operator clears %s: %v", newID, err)+ }+ if after, found, err := store.GetResourceOperatorState(oldID); err != nil || found {+ t.Fatalf("the retired ID still reports a lock the operator cleared: found=%v err=%v state=%+v", found, err, after)+ }+ })+ }+ }+}++// The era an ID leaves can itself be left again, one succession at a time or in+// one batch, and the read follows the record to wherever the row ended up. A+// chain is as long as the identity changes were, so nothing caps it.+func TestRetiredCanonicalIDReadFollowsASuccessionChain(t *testing.T) {+ const length = 150+ chain := make([]string, length)+ for i := range chain {+ chain[i] = fmt.Sprintf("era-%03d", i)+ }+ hops := make([]CanonicalIDSuccession, 0, length-1)+ for i := 0; i+1 < length; i++ {+ hops = append(hops, CanonicalIDSuccession{OldCanonicalID: chain[i], NewCanonicalID: chain[i+1]})+ }+ apply := map[string]func(t *testing.T, store ResourceStore){+ "one hop per succession": func(t *testing.T, store ResourceStore) {+ for _, hop := range hops {+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{hop}); err != nil {+ t.Fatalf("ApplyCanonicalIDSuccessions %v: %v", hop, err)+ }+ }+ },+ "one batch": func(t *testing.T, store ResourceStore) {+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions(hops); err != nil {+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)+ }+ },+ }+ for storeName, newStore := range retiredIDTestStores() {+ for applyName, applyChain := range apply {+ t.Run(storeName+"/"+applyName, func(t *testing.T) {+ store := newStore(t)+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: chain[0], NeverAutoRemediate: true, SetAt: time.Now().UTC(), SetBy: "operator"}); err != nil {+ t.Fatalf("lock %s: %v", chain[0], err)+ }+ applyChain(t, store)++ end := chain[length-1]+ if ids := operatorStateRowIDs(t, store); !reflect.DeepEqual(ids, []string{end}) {+ t.Fatalf("the row did not end up on %s: rows are %v", end, ids)+ }+ for _, id := range []string{chain[0], chain[1], chain[length/2], chain[length-2], end} {+ if state, found, err := store.GetResourceOperatorState(id); err != nil || !found || !state.BlocksRemediation() {+ t.Fatalf("%s does not read the lock %s holds: found=%v err=%v state=%+v", id, end, found, err, state)+ }+ }+ if state, found, err := store.GetResourceOperatorState("era-unrelated"); err != nil || found {+ t.Fatalf("an ID no succession names read a row: found=%v err=%v state=%+v", found, err, state)+ }++ if err := store.ClearResourceOperatorState(end); err != nil {+ t.Fatalf("operator clears %s: %v", end, err)+ }+ for _, id := range []string{chain[0], chain[length/2], end} {+ if state, found, err := store.GetResourceOperatorState(id); err != nil || found {+ t.Fatalf("%s still reads a lock the operator cleared on %s: found=%v err=%v state=%+v", id, end, found, err, state)+ }+ }+ })+ }+ }+}++// Successions are recorded by predecessor, so two eras can name each other+// (a reporter that gains a key, loses it and gains it again). A read of an ID+// on such a loop ends, and finds the row wherever the loop left it.+func TestRetiredCanonicalIDReadEndsOnARecordedLoop(t *testing.T) {+ for storeName, newStore := range retiredIDTestStores() {+ t.Run(storeName, func(t *testing.T) {+ store := newStore(t)+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{+ {OldCanonicalID: "loop-a", NewCanonicalID: "loop-b"},+ {OldCanonicalID: "loop-b", NewCanonicalID: "loop-c"},+ {OldCanonicalID: "loop-c", NewCanonicalID: "loop-a"},+ }); err != nil {+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)+ }+ readsEnd := func(id string, wantLock bool) {+ t.Helper()+ done := make(chan struct{})+ go func() {+ defer close(done)+ state, found, err := store.GetResourceOperatorState(id)+ if err != nil {+ t.Errorf("read operator state of %s: %v", id, err)+ return+ }+ if got := found && state.BlocksRemediation(); got != wantLock {+ t.Errorf("%s: lock read = %v, want %v (found=%v state=%+v)", id, got, wantLock, found, state)+ }+ }()+ select {+ case <-done:+ case <-time.After(10 * time.Second):+ t.Fatalf("reading %s did not end on a loop of recorded successions", id)+ }+ }+ for _, id := range []string{"loop-a", "loop-b", "loop-c"} {+ readsEnd(id, false)+ }+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "loop-b", NeverAutoRemediate: true, SetAt: time.Now().UTC()}); err != nil {+ t.Fatalf("lock loop-b: %v", err)+ }+ for _, id := range []string{"loop-a", "loop-b", "loop-c"} {+ readsEnd(id, true)+ }+ })+ }+}++// A succession that lands on an ID with a row of its own leaves the+// predecessor's row where it was, so an ID early on a chain can hold a row that+// no longer says what the resource is. A retired ID reads what the end of its+// chain holds, the ID the next generation will list.+func TestRetiredCanonicalIDReadsWhatTheEndOfItsChainHolds(t *testing.T) {+ for storeName, newStore := range retiredIDTestStores() {+ t.Run(storeName+"/an unlocked row left behind does not hide the lock at the end", func(t *testing.T) {+ store := newStore(t)+ now := time.Now().UTC()+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "stale-b", Note: "set under the middle ID", SetAt: now, SetBy: "operator"}); err != nil {+ t.Fatalf("seed the middle ID: %v", err)+ }+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "stale-c", NeverAutoRemediate: true, SetAt: now, SetBy: "operator"}); err != nil {+ t.Fatalf("lock the last ID: %v", err)+ }+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{+ {OldCanonicalID: "stale-a", NewCanonicalID: "stale-b"},+ {OldCanonicalID: "stale-b", NewCanonicalID: "stale-c"},+ }); err != nil {+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)+ }+ if _, left := operatorStateRow(t, store, "stale-b"); !left {+ t.Fatal("the middle ID's row moved; the fixture no longer leaves one behind")+ }+ if state, found, err := store.GetResourceOperatorState("stale-a"); err != nil || !found || !state.BlocksRemediation() {+ t.Fatalf("a row left on the way hid the lock at the end of the chain: found=%v err=%v state=%+v", found, err, state)+ }+ })+ t.Run(storeName+"/a lock left behind is not read once the end lifted it", func(t *testing.T) {+ store := newStore(t)+ now := time.Now().UTC()+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "lift-a", NeverAutoRemediate: true, SetAt: now, SetBy: "operator"}); err != nil {+ t.Fatalf("lock the first ID: %v", err)+ }+ if err := store.SetResourceOperatorState(ResourceOperatorState{CanonicalID: "lift-c", Note: "set under the last ID", SetAt: now, SetBy: "operator"}); err != nil {+ t.Fatalf("seed the last ID: %v", err)+ }+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{+ {OldCanonicalID: "lift-a", NewCanonicalID: "lift-b"},+ {OldCanonicalID: "lift-b", NewCanonicalID: "lift-c"},+ }); err != nil {+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)+ }+ end, found, err := store.GetResourceOperatorState("lift-c")+ if err != nil || !found || !end.BlocksRemediation() {+ t.Fatalf("the last ID did not take the lock: found=%v err=%v state=%+v", found, err, end)+ }+ if left, found := operatorStateRow(t, store, "lift-b"); !found || !left.BlocksRemediation() {+ t.Fatalf("the middle ID's locked row moved; the fixture no longer leaves one behind: found=%v state=%+v", found, left)+ }+ if state, found, err := store.GetResourceOperatorState("lift-a"); err != nil || !found || !state.BlocksRemediation() {+ t.Fatalf("the first ID does not read the lock the last holds: found=%v err=%v state=%+v", found, err, state)+ }++ lifted := end+ lifted.NeverAutoRemediate = false+ if err := store.SetResourceOperatorState(lifted); err != nil {+ t.Fatalf("operator lifts the lock on lift-c: %v", err)+ }+ if state, found, err := store.GetResourceOperatorState("lift-a"); err != nil || found {+ t.Fatalf("the first ID still reads a lock the operator lifted on the last: found=%v err=%v state=%+v", found, err, state)+ }+ if err := store.ClearResourceOperatorState("lift-c"); err != nil {+ t.Fatalf("operator clears lift-c: %v", err)+ }+ if state, found, err := store.GetResourceOperatorState("lift-a"); err != nil || found {+ t.Fatalf("the first ID still reads a lock the operator cleared on the last: found=%v err=%v state=%+v", found, err, state)+ }+ })+ }+}++// The lock can move between two IDs on a chain that loops while a read walks+// it, since every move commits on its own. At every instant one of the two holds+// the lock (the mover writes the new holder before it clears the old one), so a+// read of either ID that answers "not locked" has walked the chain across a+// move instead of reading one state.+func TestRetiredCanonicalIDReadSeesOneStateWhileTheLockMoves(t *testing.T) {+ for storeName, newStore := range retiredIDTestStores() {+ t.Run(storeName, func(t *testing.T) {+ store := newStore(t)+ if err := store.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions([]CanonicalIDSuccession{+ {OldCanonicalID: "move-a", NewCanonicalID: "move-b"},+ {OldCanonicalID: "move-b", NewCanonicalID: "move-a"},+ }); err != nil {+ t.Fatalf("ApplyCanonicalIDSuccessions: %v", err)+ }+ lock := func(id string) ResourceOperatorState {+ return ResourceOperatorState{CanonicalID: id, NeverAutoRemediate: true, SetAt: time.Now().UTC(), SetBy: "operator"}+ }+ if err := store.SetResourceOperatorState(lock("move-b")); err != nil {+ t.Fatalf("lock move-b: %v", err)+ }++ stop := make(chan struct{})+ moverDone := make(chan error, 1)+ go func() {+ holder, other := "move-b", "move-a"+ for {+ select {+ case <-stop:+ moverDone <- nil+ return+ default:+ }+ if err := store.SetResourceOperatorState(lock(other)); err != nil {+ moverDone <- err+ return+ }+ if err := store.ClearResourceOperatorState(holder); err != nil {+ moverDone <- err+ return+ }+ holder, other = other, holder+ }+ }()++ reads, misses := 0, 0+ var firstMiss string+ deadline := time.Now().Add(2 * time.Second)+ for time.Now().Before(deadline) {+ for _, id := range []string{"move-a", "move-b"} {+ state, found, err := store.GetResourceOperatorState(id)+ if err != nil {+ close(stop)+ <-moverDone+ t.Fatalf("read operator state of %s: %v", id, err)+ }+ reads+++ if !found || !state.BlocksRemediation() {+ if misses++; firstMiss == "" {+ firstMiss = fmt.Sprintf("%s: found=%v state=%+v", id, found, state)+ }+ }+ }+ }+ close(stop)+ if err := <-moverDone; err != nil {+ t.Fatalf("moving the lock: %v", err)+ }+ if misses > 0 {+ t.Fatalf("%d of %d reads answered unlocked while the lock was held throughout; first: %s", misses, reads, firstMiss)+ }+ })+ }+}diff --git a/internal/unifiedresources/registry_test.go b/internal/unifiedresources/registry_test.goindex 55a16a6e69..3a15133ada 100644--- a/internal/unifiedresources/registry_test.go+++ b/internal/unifiedresources/registry_test.go@@ -6583,8 +6583,8 @@ func TestCanonicalIDSuccessionMovesOperatorState(t *testing.T) {}registry.PersistIdentityPins()- if _, found, err := store.GetResourceOperatorState(shortEraID); err != nil || found {- t.Fatalf("operator state still keyed by superseded ID (found=%v, err=%v)", found, err)+ if _, found := operatorStateRow(t, store, shortEraID); found {+ t.Fatalf("operator state still keyed by superseded ID")}state, found, err := store.GetResourceOperatorState(newID)if err != nil || !found {@@ -6683,8 +6683,8 @@ func TestIngestRecordsRecordDeclaredSuccessions(t *testing.T) {}})newID := SourceSpecificID(ResourceTypeStorage, SourceTrueNAS, "system:conn-a/pool:tank")- if _, found, err := store.GetResourceOperatorState(oldID); err != nil || found {- t.Fatalf("expected operator state to leave superseded ID (found=%v err=%v)", found, err)+ if _, found := operatorStateRow(t, store, oldID); found {+ t.Fatalf("expected operator state to leave superseded ID")}state, found, err := store.GetResourceOperatorState(newID)if err != nil || !found || !state.NeverAutoRemediate {@@ -6958,7 +6958,7 @@ func TestProxmoxGuestSuccessionRekeysOperatorState(t *testing.T) {if !state.NeverAutoRemediate {t.Fatalf("re-keyed operator state lost its fields: %+v", state)}- if _, stale, _ := store.GetResourceOperatorState(legacyID); stale {+ if _, stale := operatorStateRow(t, store, legacyID); stale {t.Fatalf("operator state still present under retired ID %q", legacyID)}}@@ -8940,8 +8940,8 @@ func TestSuccessionBatchOrderKeepsLinkedRemediationLock(t *testing.T) {t.Fatalf("%s lost the lock across a batch that returned to its start", id)}}- if state, found, err := store.GetResourceOperatorState("loop-b"); err != nil || found {- t.Fatalf("the re-keyed intermediate ID was given a row: found=%v err=%v state=%+v", found, err, state)+ if state, found := operatorStateRow(t, store, "loop-b"); found {+ t.Fatalf("the re-keyed intermediate ID was given a row: state=%+v", state)}})}@@ -9605,3 +9605,137 @@ func TestManualLinkChainCycleLinksAreAllCutByAReportedMember(t *testing.T) {}}}++// successionHookStore runs a callback from inside a rebuild, right after the+// store applied a canonical-ID succession and before the rebuilt registry is+// published: the moment the previous generation is still served while the+// store already answers for the successor.+type successionHookStore struct {+ ResourceStore+ afterSuccession func()+}++func (s *successionHookStore) ApplyCanonicalIDSuccessions(successions []CanonicalIDSuccession) error {+ err := s.ResourceStore.(canonicalIDSuccessor).ApplyCanonicalIDSuccessions(successions)+ if err == nil && s.afterSuccession != nil {+ s.afterSuccession()+ }+ return err+}++// A rebuild applies a succession to the store (the operator's rows move to the+// successor) before the generation that lists the successor is published, and+// the previous generation keeps serving until then. The dispatch gates and the+// planner read the operator state of the ID the generation they consult lists,+// so in that window they ask for an ID whose row has gone. A lock the operator+// set must still be found there: the old ID is the resource the previous+// generation would act on.+func TestRemediationLockHoldsOnTheRetiredIDWhileThePreviousGenerationServes(t *testing.T) {+ now := time.Now().UTC()+ type scenario struct {+ before, after models.StateSnapshot+ // kind is the resource type that changes ID.+ kind ResourceType+ }+ scenarios := map[string]scenario{+ // A record-declared era (the reporter's source-specific ID), applied+ // while the snapshot is ingested.+ "host agent gaining a machine ID": {+ kind: ResourceTypeAgent,+ before: models.StateSnapshot{Hosts: []models.Host{{ID: "agent-window-01", Hostname: "window-01", Status: "online", LastSeen: now}}},+ after: models.StateSnapshot{Hosts: []models.Host{{ID: "agent-window-01", Hostname: "window-01", MachineID: "machine-window-01", Status: "online", LastSeen: now}}},+ },+ // A pin-driven era, applied once the rebuilt registry persists its pins.+ "swarm host gaining a full hostname": {+ kind: ResourceTypeAgent,+ before: models.StateSnapshot{DockerHosts: []models.DockerHost{{+ ID: "docker-window-02", Hostname: "cloud", Status: "online", LastSeen: now,+ Swarm: &models.DockerSwarmInfo{ClusterID: "swarm-window", ClusterName: "prod-swarm"},+ }}},+ after: models.StateSnapshot{DockerHosts: []models.DockerHost{{+ ID: "docker-window-02", Hostname: "cloud.a", Status: "online", LastSeen: now,+ Swarm: &models.DockerSwarmInfo{ClusterID: "swarm-window", ClusterName: "prod-swarm"},+ }}},+ },+ }+ stores := map[string]func(t *testing.T) ResourceStore{+ "memory": func(t *testing.T) ResourceStore { return NewMemoryStore() },+ "sqlite": func(t *testing.T) ResourceStore {+ store, err := NewSQLiteResourceStore(t.TempDir(), "default")+ if err != nil {+ t.Fatalf("NewSQLiteResourceStore: %v", err)+ }+ t.Cleanup(func() { _ = store.Close() })+ return store+ },+ }+ locks := map[string]ResourceOperatorState{+ "never auto-remediate": {NeverAutoRemediate: true},+ "retired": {LifecycleState: LifecycleStateRetired},+ }+ idsOf := func(adapter *MonitorAdapter, kind ResourceType) []string {+ var ids []string+ for _, resource := range adapter.GetByType(kind) {+ ids = append(ids, resource.ID)+ }+ return ids+ }+ for scenarioName, sc := range scenarios {+ for storeName, newStore := range stores {+ for lockName, lock := range locks {+ t.Run(scenarioName+"/"+storeName+"/"+lockName, func(t *testing.T) {+ hooked := &successionHookStore{ResourceStore: newStore(t)}+ adapter := NewMonitorAdapter(NewRegistry(hooked))+ adapter.PopulateFromSnapshot(sc.before)+ before := idsOf(adapter, sc.kind)+ if len(before) != 1 {+ t.Fatalf("first generation lists %v, want the one resource", before)+ }+ oldID := before[0]+ state := lock+ state.CanonicalID, state.SetAt, state.SetBy = oldID, now.Add(-time.Hour), "operator"+ if err := hooked.SetResourceOperatorState(state); err != nil {+ t.Fatalf("lock %s: %v", oldID, err)+ }++ var (+ applied int+ listedDuring []string+ foundDuring bool+ lockedDuring bool+ readErrDuring error+ )+ hooked.afterSuccession = func() {+ applied+++ listedDuring = idsOf(adapter, sc.kind)+ var got ResourceOperatorState+ got, foundDuring, readErrDuring = adapter.GetResourceOperatorState(oldID)+ lockedDuring = foundDuring && got.BlocksRemediation()+ }+ adapter.PopulateFromSnapshot(sc.after)++ if applied == 0 {+ t.Fatalf("the rebuild applied no succession; the fixture no longer changes %s's era", oldID)+ }+ if !slices.Equal(listedDuring, []string{oldID}) {+ t.Fatalf("during the rebuild the served generation lists %v, want the previous generation's %s", listedDuring, oldID)+ }+ if readErrDuring != nil {+ t.Fatalf("read operator state of %s during the rebuild: %v", oldID, readErrDuring)+ }+ if !lockedDuring {+ t.Fatalf("while the previous generation still lists %s the lock the operator set on it reads as missing (found=%v)", oldID, foundDuring)+ }++ after := idsOf(adapter, sc.kind)+ if len(after) != 1 || after[0] == oldID {+ t.Fatalf("the published generation lists %v, want the resource under a new ID", after)+ }+ if got, found, err := adapter.GetResourceOperatorState(after[0]); err != nil || !found || !got.BlocksRemediation() {+ t.Fatalf("the published ID %s does not hold the lock: found=%v err=%v state=%+v", after[0], found, err, got)+ }+ })+ }+ }+ }+}diff --git a/internal/unifiedresources/remediation_lock_carry.go b/internal/unifiedresources/remediation_lock_carry.goindex 109d8f0aeb..964a38107a 100644--- a/internal/unifiedresources/remediation_lock_carry.go+++ b/internal/unifiedresources/remediation_lock_carry.go@@ -2,6 +2,7 @@ package unifiedresourcesimport ("database/sql"+ "errors""fmt""log""sort"@@ -17,6 +18,93 @@ import (// component changes (a link, an unlink) and whenever a succession lands on one// of them. CarryRemediationLock owns what is written onto a row.+// A rebuild applies a succession to the store before it publishes the registry+// generation that lists the successor, and the previous generation, which still+// lists the retired ID, keeps serving in between. The gates read the operator+// state of the ID they are asked about, and the succession moved that ID's row,+// so a retired ID that answers "no row" lets Pulse act on a resource the+// operator locked. A read of an ID that holds no row therefore follows the+// recorded successions (canonical_id_successions) to the end of its chain and+// answers as the ID the chain ends on would: with the remediation block that+// ID's row carries, the block alone as a link fold carries it, attributed to the+// identity change and dated by the row it came from so two reads agree (a+// plan's policy revision hashes the state). The end of the chain decides, so a+// lock the operator lifted on the successor, by editing or clearing its row, is+// not read back from a stale row left behind on an ID before it; an ID that+// holds a row of its own answers with it; and a chain that returns to an ID it+// has seen has no end, so a lock held anywhere on it counts.+func followRetiredID(+ retiredID string,+ successorOf func(id string) (successor string, retired bool, err error),+ rowOf func(id string) (ResourceOperatorState, bool, error),+) (ResourceOperatorState, bool, error) {+ seen := map[string]struct{}{retiredID: {}}+ var end, nearestLock ResourceOperatorState+ var endFound, lockFound bool+ for current := retiredID; ; {+ successor, retired, err := successorOf(current)+ if err != nil {+ return ResourceOperatorState{}, false, err+ }+ successor = CanonicalResourceID(successor)+ if !retired || successor == "" {+ if endFound {+ lock, carried := retiredIDRemediationLock(retiredID, end)+ return lock, carried, nil+ }+ return ResourceOperatorState{}, false, nil+ }+ if _, again := seen[successor]; again {+ if lockFound {+ lock, carried := retiredIDRemediationLock(retiredID, nearestLock)+ return lock, carried, nil+ }+ return ResourceOperatorState{}, false, nil+ }+ seen[successor] = struct{}{}+ row, found, err := rowOf(successor)+ if err != nil {+ return ResourceOperatorState{}, false, err+ }+ end, endFound = row, found+ if found && !lockFound && row.BlocksRemediation() {+ nearestLock, lockFound = row, true+ }+ current = successor+ }+}++// retiredIDRemediationLock is what a retired ID reads of the row an ID on its+// chain holds: that row's remediation block, if it has one, and nothing else.+func retiredIDRemediationLock(retiredID string, holder ResourceOperatorState) (ResourceOperatorState, bool) {+ return CarryRemediationLock(retiredID, ResourceOperatorState{}, false, holder, true, holder.SetAt)+}++// retiredIDRemediationLockSQL is the read above for the SQLite store: one+// statement per hop, on the connection the succession's transaction commits on,+// so a read that finds the row gone from an ID also finds the succession that+// moved it.+func retiredIDRemediationLockSQL(queryer resourceOperatorStateQueryRower, canonicalID string) (ResourceOperatorState, bool, error) {+ retiredID := CanonicalResourceID(canonicalID)+ if retiredID == "" {+ return ResourceOperatorState{}, false, nil+ }+ return followRetiredID(retiredID,+ func(id string) (string, bool, error) {+ var successor string+ err := queryer.QueryRow(`SELECT new_canonical_id FROM canonical_id_successions WHERE old_canonical_id = ?`, id).Scan(&successor)+ if errors.Is(err, sql.ErrNoRows) {+ return "", false, nil+ }+ if err != nil {+ return "", false, fmt.Errorf("query canonical ID succession of %q: %w", id, err)+ }+ return successor, true, nil+ },+ func(id string) (ResourceOperatorState, bool, error) { return getResourceOperatorStateSQL(queryer, id) },+ )+}+// carriedLock records one remediation block written onto `to` from `from`.type carriedLock struct {to stringdiff --git a/internal/unifiedresources/store.go b/internal/unifiedresources/store.goindex a67f5b03a4..9d1021bf66 100644--- a/internal/unifiedresources/store.go+++ b/internal/unifiedresources/store.go@@ -3307,9 +3307,33 @@ func (s *SQLiteResourceStore) GetExportAudits(since time.Time, limit int) ([]Exp// canonical ID. Returns (zero, false, nil) on no entry; (state, true,// nil) on a hit; (zero, false, error) on a query failure. The found// signal is meaningful for the API GET path which returns 404 vs-// returning the default no-state record.+// returning the default no-state record. An ID that holds no row but that a+// canonical-ID succession retired answers with the remediation block its+// successor carries (followRetiredID), which is what a registry+// generation still listing the retired ID needs while a rebuild is in flight.func (s *SQLiteResourceStore) GetResourceOperatorState(canonicalID string) (ResourceOperatorState, bool, error) {- return getResourceOperatorStateSQL(s.db, canonicalID)+ state, found, err := getResourceOperatorStateSQL(s.db, canonicalID)+ if err != nil || found {+ return state, found, err+ }+ return s.retiredIDOperatorState(canonicalID)+}++// retiredIDOperatorState answers for an ID that held no row when it was first+// read. The walk down its chain is several statements, and a succession that+// commits between two of them can move the lock off an ID the walk already+// passed onto one it has seen (a chain that loops back), so the ID's own row and+// every hop are read in one transaction, which is one snapshot.+func (s *SQLiteResourceStore) retiredIDOperatorState(canonicalID string) (ResourceOperatorState, bool, error) {+ tx, err := s.db.Begin()+ if err != nil {+ return ResourceOperatorState{}, false, fmt.Errorf("begin resource operator state read: %w", err)+ }+ defer func() { _ = tx.Rollback() }()+ if state, found, err := getResourceOperatorStateSQL(tx, canonicalID); err != nil || found {+ return state, found, err+ }+ return retiredIDRemediationLockSQL(tx, canonicalID)}type resourceOperatorStateQueryRower interface {@@ -4683,7 +4707,9 @@ func (m *MemoryStore) GetExportAudits(since time.Time, limit int) ([]ExportAudit// canonical resource ID. The found bool distinguishes "no entry" (the// default no-state posture) from "explicit empty entry the operator// cleared" — both have the same effective behavior but the latter may-// be surfaced differently on the audit timeline.+// be surfaced differently on the audit timeline. An ID that holds no row but+// that a canonical-ID succession retired answers as the SQLite store does, with+// the remediation block its successor carries.func (m *MemoryStore) GetResourceOperatorState(canonicalID string) (ResourceOperatorState, bool, error) {canonicalID = strings.TrimSpace(canonicalID)if canonicalID == "" {@@ -4693,11 +4719,28 @@ func (m *MemoryStore) GetResourceOperatorState(canonicalID string) (ResourceOperdefer m.mu.RUnlock()state, ok := m.resourceOperatorState[canonicalID]if !ok {- return ResourceOperatorState{}, false, nil+ lock, carried := m.retiredIDRemediationLockLocked(canonicalID)+ return lock, carried, nil}return state, true, nil}+// retiredIDRemediationLockLocked is the read described at followRetiredID over+// the in-memory succession record; callers hold m.mu.+func (m *MemoryStore) retiredIDRemediationLockLocked(retiredID string) (ResourceOperatorState, bool) {+ lock, carried, _ := followRetiredID(retiredID,+ func(id string) (string, bool, error) {+ successor, retired := m.canonicalSuccessions[id]+ return successor, retired, nil+ },+ func(id string) (ResourceOperatorState, bool, error) {+ state, found := m.resourceOperatorState[id]+ return state, found, nil+ },+ )+ return lock, carried+}+// SetResourceOperatorState upserts the supplied state, validating it// before persistence. Callers should pass a fully-populated record;// per-field merge is not supported, so reading the existing state and