Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep historical fix results from closing recurring findings
rcourtmanc1b5f4b13fa2
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 02d1f7610b..ac69b751e8 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -15,6 +15,18 @@## Purpose+### Verified agent actions remain attached to their originating occurrence++An agent action's execution and verification evidence belongs to its original+action and investigation. Replaying that result after a finding reopens must+preserve the new occurrence's investigation eligibility and current record.+The shared findings store validates session identity and the persisted+regression boundary atomically; current finding reads select current work+while explicit action history retains earlier receipts. The three-occurrence+Patrol action regression in+`internal/api/ai_handlers_patrol_actions_additional_test.go` also reconstructs+persisted findings and action audit state before the third occurrence.+### Collected backup evidence does not require command-agent accessGuest investigation manifests include the existing read-only storage tool fordiff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.mdindex afafd36fd5..adac4daab1 100644--- a/docs/release-control/v6/internal/subsystems/ai-runtime.md+++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md@@ -421,6 +421,20 @@ evidence change and remains recorded in the customer-journey qualification.Action reconciliation refreshes the durable product investigation record fromthe authoritative session/action even when the finding outcome already matches.+Action truth is authoritative for its originating investigation, not for every+later occurrence sharing the finding ID. Finding outcome and record writes+check the current session and persisted regression boundary under the findings+store lock. Historical callbacks and review hydration retain their original+action result without resolving, queuing, or replacing a newer occurrence,+including after restart and before its next investigation starts. A verified+action from the new occurrence can still resolve that occurrence normally.+Finding-scoped investigation and transcript reads return no current+investigation while only an older occurrence's session exists. Explicit+session and action history retains that older evidence.+The canonical Patrol handler suite in+`internal/api/ai_handlers_patrol_actions_additional_test.go` exercises three+occurrences with a persisted restart and historical action replay; findings+store controls also exercise concurrent re-detection and reconciliation.The same builder owns initial completion and later refresh. Completion replacesan early action projection with the final investigation evidence. Later actionrefresh preserves original prose, impact, confidence, evidence and rollback. Unchanged hydration is a no-op, and adiff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.mdindex 7472788178..c70a330981 100644--- a/docs/release-control/v6/internal/subsystems/api-contracts.md+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md@@ -20,6 +20,18 @@## Purpose+### Finding-scoped investigation reads belong to the current occurrence++The finding investigation and transcript endpoints return only the session+owned by the finding's current occurrence. After a resolved finding is+re-detected, a retained earlier session returns `404` through these current+views until a new session exists. Explicit session and action history retain+their original evidence. Outcome and record reconciliation check session+identity and the persisted regression boundary under the findings lock before+changing current state. `TestPatrolHistoricalActionCannotResolveRecurringFinding`+in `internal/api/ai_handlers_patrol_actions_additional_test.go` covers three+occurrences, restart, historical reads and duplicate action callbacks.+### Recovery point reads stay coherent between HTTP and investigationsThe points endpoint and investigation adapter use the same org-scoped managerdiff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex 0009138940..8184232917 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -21,6 +21,17 @@## Purpose+### Recurring findings preserve the ownership of recovery evidence++Storage and recovery findings use the same occurrence-bound investigation+projection as other Patrol findings. A previous verified action remains+historical evidence when the finding is re-detected; it cannot resolve the new+occurrence or replace its investigation record. Finding-scoped investigation+and transcript reads select the current session, including an explicit absent+state before a new session starts. Explicit session and action history retain+the earlier result. The shared three-occurrence and persisted-restart controls+live in `internal/api/ai_handlers_patrol_actions_additional_test.go`.+### Recovery APIs and investigations share the org-scoped point reader`Manager.ListPointsForOrg` selects the composed recovery evidence source fordiff --git a/internal/ai/findings.go b/internal/ai/findings.goindex 0f9b076cc3..5b994aee8e 100644--- a/internal/ai/findings.go+++ b/internal/ai/findings.go@@ -2460,6 +2460,10 @@ func (s *FindingsStore) UpdateInvestigationRecord(id string, record *aicontractss.mu.Unlock()return false}+ if record != nil && !f.acceptsInvestigationResult(record.SessionID, record.StartedAt) {+ s.mu.Unlock()+ return false+ }f.InvestigationRecord = recordmeta := map[string]string{}@@ -2561,6 +2565,56 @@ func (s *FindingsStore) UpdateInvestigationOutcome(id, outcome string) bool {return false}+ s.updateInvestigationOutcomeLocked(f, outcome)+ s.mu.Unlock()+ s.scheduleSave()+ return true+}++// ReconcileInvestigationOutcome projects an authoritative session result only+// onto the occurrence that owns it. A finding ID survives regressions, while+// the investigation session and regression boundary identify the current work.+// Check and mutation share the store lock so a concurrent re-detection cannot+// be closed by a result accepted against its previous occurrence.+func (s *FindingsStore) ReconcileInvestigationOutcome(id string, session *aicontracts.InvestigationSession, outcome string) (applied, changed bool) {+ if session == nil || session.FindingID != id {+ return false, false+ }+ s.mu.Lock()+ f := s.findings[id]+ if f == nil || !f.IsCurrentInvestigation(session) {+ s.mu.Unlock()+ return false, false+ }+ changed = f.InvestigationOutcome != outcome+ if changed {+ s.updateInvestigationOutcomeLocked(f, outcome)+ }+ s.mu.Unlock()+ if changed {+ s.scheduleSave()+ }+ return true, changed+}++// IsCurrentInvestigation distinguishes current finding work from historical+// sessions retained under the same finding ID.+func (f *Finding) IsCurrentInvestigation(session *aicontracts.InvestigationSession) bool {+ return f != nil && session != nil && session.FindingID == f.ID &&+ f.acceptsInvestigationResult(session.SessionID, session.StartedAt)+}++func (f *Finding) acceptsInvestigationResult(sessionID string, startedAt time.Time) bool {+ if f.InvestigationSessionID != "" && f.InvestigationSessionID != sessionID {+ return false+ }+ // Re-detection clears the session reference. Its persisted regression time+ // still excludes old sessions during the gap before a new one starts, and+ // after a process restart. Missing historical time cannot prove ownership.+ return f.LastRegressionAt == nil || (!startedAt.IsZero() && !startedAt.Before(*f.LastRegressionAt))+}++func (s *FindingsStore) updateInvestigationOutcomeLocked(f *Finding, outcome string) {prevOutcome := f.InvestigationOutcomef.InvestigationOutcome = outcome// Replaying a persisted result must not invent a later recovery event.@@ -2577,9 +2631,6 @@ func (s *FindingsStore) UpdateInvestigationOutcome(id, outcome string) bool {})}s.syncLoopStateLocked(f)- s.mu.Unlock()- s.scheduleSave()- return true}// UpdateInvestigation updates all investigation fields on a findingdiff --git a/internal/ai/findings_recurrence_test.go b/internal/ai/findings_recurrence_test.gonew file mode 100644index 0000000000..195821fa85--- /dev/null+++ b/internal/ai/findings_recurrence_test.go@@ -0,0 +1,74 @@+package ai++import (+ "reflect"+ "sync"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/pkg/aicontracts"+)++func TestFindingInvestigationReconciliationUsesCurrentOccurrence(t *testing.T) {+ boundary := time.Now().Add(-time.Minute)+ for _, tc := range []struct {+ name, currentSession, resultSession string+ startedAt time.Time+ accepted bool+ }{+ {"old result before new investigation", "", "old", boundary.Add(-time.Minute), false},+ {"missing occurrence time", "", "old", time.Time{}, false},+ {"old result after new investigation", "current", "old", boundary.Add(time.Second), false},+ {"stale persisted session reference", "old", "old", boundary.Add(-time.Minute), false},+ {"current result", "current", "current", boundary.Add(time.Second), true},+ } {+ t.Run(tc.name, func(t *testing.T) {+ for _, outcome := range []InvestigationOutcome{InvestigationOutcomeFixQueued, InvestigationOutcomeFixVerified,+ InvestigationOutcomeFixRejected, InvestigationOutcomeNeedsAttention, InvestigationOutcomeFixVerificationUnknown} {+ store := NewFindingsStore()+ store.Add(&Finding{ID: "failure", ResourceID: "resource", Severity: FindingSeverityWarning,+ LastRegressionAt: &boundary, RegressionCount: 1, InvestigationSessionID: tc.currentSession})+ before := store.Get("failure")+ session := &aicontracts.InvestigationSession{ID: "investigation", FindingID: "failure", SessionID: tc.resultSession, StartedAt: tc.startedAt}+ applied, changed := store.ReconcileInvestigationOutcome("failure", session, string(outcome))+ if applied != tc.accepted || changed != tc.accepted {+ t.Fatalf("outcome=%s applied=%v changed=%v, want %v", outcome, applied, changed, tc.accepted)+ }+ if !tc.accepted && !reflect.DeepEqual(before, store.Get("failure")) {+ t.Fatal("historical result changed current finding")+ }+ record := &aicontracts.InvestigationRecord{ID: session.ID, FindingID: session.FindingID,+ SessionID: session.SessionID, StartedAt: session.StartedAt, Outcome: aicontracts.InvestigationOutcome(outcome)}+ if got := store.UpdateInvestigationRecord("failure", record); got != tc.accepted {+ t.Fatalf("record accepted=%v, want %v", got, tc.accepted)+ }+ }+ })+ }+}++func TestFindingRecurrenceAndHistoricalReconciliationAreAtomic(t *testing.T) {+ for attempt := 0; attempt < 100; attempt++ {+ store := NewFindingsStore()+ report := func() {+ store.Add(&Finding{ID: "failure", ResourceID: "resource", Severity: FindingSeverityWarning, Title: "Failure"})+ }+ report()+ session := &aicontracts.InvestigationSession{ID: "old", FindingID: "failure", SessionID: "old", StartedAt: time.Now().Add(-time.Minute)}+ store.UpdateInvestigation("failure", "old", "completed", string(InvestigationOutcomeFixVerified), nil, 1)+ var wg sync.WaitGroup+ start := make(chan struct{})+ wg.Add(2)+ go func() { defer wg.Done(); <-start; report() }()+ go func() {+ defer wg.Done()+ <-start+ store.ReconcileInvestigationOutcome("failure", session, string(InvestigationOutcomeFixVerified))+ }()+ close(start)+ wg.Wait()+ if got := store.Get("failure"); got.ResolvedAt != nil || !got.ShouldInvestigate("full") || got.RegressionCount != 1 {+ t.Fatalf("historical result closed concurrent recurrence: %#v", got)+ }+ }+}diff --git a/internal/api/ai_handlers.go b/internal/api/ai_handlers.goindex 0ce8adf078..a345e6f71e 100644--- a/internal/api/ai_handlers.go+++ b/internal/api/ai_handlers.go@@ -7630,7 +7630,15 @@ func (h *AISettingsHandler) updateFindingInvestigationOutcome(ctx context.Contexreturn}outcomeChanged := existing.InvestigationOutcome != outcome- if outcomeChanged && !findingsStore.UpdateInvestigationOutcome(findingID, outcome) {+ if investigation != nil {+ var applied bool+ applied, outcomeChanged = findingsStore.ReconcileInvestigationOutcome(findingID, investigation, outcome)+ if !applied {+ log.Debug().Str("findingID", findingID).Str("investigationID", investigation.ID).+ Msg("Historical investigation result does not belong to current finding occurrence")+ return+ }+ } else if outcomeChanged && !findingsStore.UpdateInvestigationOutcome(findingID, outcome) {log.Warn().Str("findingID", findingID).Msg("Finding not found for outcome update")return}@@ -8258,7 +8266,8 @@ func (h *AISettingsHandler) HandleGetInvestigation(w http.ResponseWriter, r *htt}investigation := orchestrator.GetInvestigationByFinding(findingID)- if investigation == nil {+ finding := patrol.GetFindings().Get(findingID)+ if investigation == nil || (finding != nil && !finding.IsCurrentInvestigation(investigation)) {writeErrorResponse(w, http.StatusNotFound, "not_found", "No investigation found for this finding", nil)return}@@ -8311,7 +8320,8 @@ func (h *AISettingsHandler) HandleGetInvestigationMessages(w http.ResponseWriter}investigation := orchestrator.GetInvestigationByFinding(findingID)- if investigation == nil {+ finding := patrol.GetFindings().Get(findingID)+ if investigation == nil || (finding != nil && !finding.IsCurrentInvestigation(investigation)) {writeErrorResponse(w, http.StatusNotFound, "not_found", "No investigation found for this finding", nil)return}diff --git a/internal/api/ai_handlers_patrol_actions_additional_test.go b/internal/api/ai_handlers_patrol_actions_additional_test.goindex 8a0eaa4422..caf31115b3 100644--- a/internal/api/ai_handlers_patrol_actions_additional_test.go+++ b/internal/api/ai_handlers_patrol_actions_additional_test.go@@ -4,6 +4,7 @@ import ("bytes""context""encoding/json"+ "fmt""net/http""net/http/httptest""os"@@ -18,6 +19,8 @@ import ("github.com/rcourtman/pulse-go-rewrite/internal/ai/unified""github.com/rcourtman/pulse-go-rewrite/internal/config""github.com/rcourtman/pulse-go-rewrite/internal/models"+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"+ "github.com/rcourtman/pulse-go-rewrite/pkg/aicontracts")type scopedPatrolStateProvider struct {@@ -1381,3 +1384,142 @@ func TestPatrolObjectiveAPIContractKeepsCoverageServerOwned(t *testing.T) {t.Fatalf("forged coverage status = %d, body=%s", forgedRec.Code, forgedRec.Body.String())}}++// A completed action remains authoritative for its original investigation,+// even after the same finding represents a new occurrence of the failure.+func TestPatrolHistoricalActionCannotResolveRecurringFinding(t *testing.T) {+ persistence := config.NewConfigPersistence(t.TempDir())+ newService := func() *ai.Service {+ svc := ai.NewService(nil, nil)+ svc.SetStateProvider(&MockStateProvider{})+ if err := svc.GetPatrolService().GetFindings().SetPersistence(ai.NewFindingsPersistenceAdapter(persistence)); err != nil {+ t.Fatal(err)+ }+ return svc+ }+ svc := newService()+ findings := svc.GetPatrolService().GetFindings()+ report := func() {+ findings.Add(&ai.Finding{ID: "recurring", Key: "health-check-failed", ResourceID: "vm:42", ResourceType: "vm",+ Title: "Service health check failed", Severity: ai.FindingSeverityWarning, Category: ai.FindingCategoryReliability})+ }+ report()+ investigations := newTestInvestigationStore()+ auditDirectory := t.TempDir()+ audits, err := unifiedresources.NewSQLiteResourceStore(auditDirectory, "default")+ if err != nil {+ t.Fatal(err)+ }+ t.Cleanup(func() { audits.Close() })+ handler := &AISettingsHandler{defaultAIService: svc,+ investigationStores: map[string]aicontracts.InvestigationStore{"default": investigations},+ resourceStoreProvider: func(string) (unifiedresources.ResourceStore, error) { return audits, nil },+ }+ var history []*aicontracts.InvestigationSession+ for occurrence := 1; occurrence <= 3; occurrence++ {+ t.Run(fmt.Sprintf("occurrence-%d", occurrence), func(t *testing.T) {+ for _, previous := range history {+ before := findings.Get("recurring")+ svc.GetPatrolService().SetInvestigationOrchestrator(&stubInvestigationOrchestrator{session: previous})+ for _, suffix := range []string{"/investigation", "/investigation/messages"} {+ req := httptest.NewRequest(http.MethodGet, "/api/ai/findings/recurring"+suffix, nil)+ rec := httptest.NewRecorder()+ if suffix == "/investigation" {+ handler.HandleGetInvestigation(rec, req)+ } else {+ handler.HandleGetInvestigationMessages(rec, req)+ }+ if rec.Code != http.StatusNotFound {+ t.Fatalf("current occurrence read returned historical session: status=%d body=%s", rec.Code, rec.Body.String())+ }+ }+ hydrated := handler.hydratePatrolInvestigationAction("default", previous)+ if hydrated.Outcome != aicontracts.OutcomeFixVerified || hydrated.Action == nil {+ t.Fatal("historical investigation lost its verified action")+ }+ audit, found, err := audits.GetActionAudit(hydrated.Action.ActionID)+ if err != nil || !found {+ t.Fatalf("historical audit: found=%v err=%v", found, err)+ }+ handler.ReconcilePatrolActionTransition("default", audit)+ after := findings.Get("recurring")+ if after.ResolvedAt != nil || !after.ShouldInvestigate("full") || after.InvestigationRecord != nil {+ t.Fatalf("historical action closed occurrence %d: resolved=%v outcome=%q session=%q", occurrence, after.ResolvedAt, after.InvestigationOutcome, after.InvestigationSessionID)+ }+ if len(after.Lifecycle) != len(before.Lifecycle) {+ t.Fatal("historical action changed the current occurrence's lifecycle")+ }+ }+ if !findings.Get("recurring").ShouldInvestigate("full") {+ t.Fatal("new occurrence is not eligible for investigation")+ }+ current := investigations.Create("recurring", fmt.Sprintf("session-%d", occurrence))+ current.Status = aicontracts.InvestigationStatusCompleted+ current.Summary = "Current process stalled; restart proposed."+ investigations.Update(current)+ findings.UpdateInvestigation("recurring", current.SessionID, "running", "", nil, 1)+ for _, previous := range history {+ handler.hydratePatrolInvestigationAction("default", previous)+ if got := findings.Get("recurring"); got.ResolvedAt != nil || got.InvestigationSessionID != current.SessionID || got.InvestigationOutcome != "" {+ t.Fatal("historical result overwrote the new running investigation")+ }+ }+ audit := recurrenceVerifiedAudit(t, current)+ if _, _, err := audits.CreateActionAudit(audit, nil); err != nil {+ t.Fatal(err)+ }+ handler.ReconcilePatrolActionTransition("default", audit)+ if got := findings.Get("recurring"); got.ResolvedAt == nil || got.InvestigationOutcome != string(aicontracts.OutcomeFixVerified) || got.InvestigationRecord == nil || got.InvestigationRecord.ID != current.ID {+ t.Fatalf("current verified action did not resolve its occurrence: %#v", got)+ }+ history = append(history, current)+ })+ if t.Failed() {+ return+ }+ if occurrence < 3 {+ report()+ if occurrence == 2 {+ // Reconstruct persisted findings and action audit before the third+ // occurrence, matching the unattended controller restart.+ if err := findings.ForceSave(); err != nil {+ t.Fatal(err)+ }+ if err := audits.Close(); err != nil {+ t.Fatal(err)+ }+ audits, err = unifiedresources.NewSQLiteResourceStore(auditDirectory, "default")+ if err != nil {+ t.Fatal(err)+ }+ svc = newService()+ findings = svc.GetPatrolService().GetFindings()+ handler.defaultAIService = svc+ }+ }+ }+}++func recurrenceVerifiedAudit(t *testing.T, investigation *aicontracts.InvestigationSession) unifiedresources.ActionAuditRecord {+ t.Helper()+ now := time.Now().UTC()+ evidence := unifiedresources.ActionEvidence{Version: 1, ID: "health-" + investigation.ID,+ ObserverID: "daemon-observer", ObserverKind: "docker_daemon_observer", ObserverTrustDomain: "daemon", ExecutorTrustDomain: "agent",+ Method: "typed_docker_daemon_observation", SubjectID: "vm:42", ObservedAt: now, ReceivedAt: now, Summary: "Service health check passed"}+ canonical, err := unifiedresources.NormalizeActionResultV2(unifiedresources.ActionResultV2{Version: 2,+ Execution: unifiedresources.ActionExecutionTruth{Status: unifiedresources.ActionExecutionSucceeded},+ Verification: unifiedresources.ActionVerificationTruth{Status: unifiedresources.ActionVerificationConfirmed,+ EvidenceClass: unifiedresources.ActionEvidenceIndependent, Evidence: []unifiedresources.ActionEvidence{evidence}},+ Compensation: unifiedresources.ActionCompensationTruth{Support: unifiedresources.ActionCompensationUnavailable, Status: unifiedresources.ActionCompensationNotAvailable},+ })+ if err != nil {+ t.Fatal(err)+ }+ id := "action-" + investigation.ID+ return unifiedresources.ActionAuditRecord{ID: id, CreatedAt: now, UpdatedAt: now, State: unifiedresources.ActionStateCompleted,+ Request: unifiedresources.ActionRequest{RequestID: id, ResourceID: "vm:42", CapabilityName: "restart", RequestedBy: "pulse_patrol"},+ Plan: unifiedresources.ActionPlan{ActionID: id, RequestID: id, Allowed: true},+ Origin: &unifiedresources.ActionOrigin{Surface: patrolActionOriginSurface, FindingID: investigation.FindingID, InvestigationID: investigation.ID, ProposalID: id},+ Result: &unifiedresources.ExecutionResult{Success: true, ActionResultV2: &canonical},+ }+}