Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep stale investigation runs from overwriting newer finding state
rcourtmana408736bc4ac
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 045699ff64..9ae0f9e7ec 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -10185,3 +10185,10 @@ and `interrupted` outcomes. None of this enrols, commands, updates or removes anagent, or touches agent identity or credentials. Investigations still reachhosts only through the existing read-only evidence tools and the governedaction lifecycle.++The investigation orchestrator's findings adapter in `internal/api/ai_handlers.go`+now refuses a closing write from a run the finding no longer names, and a+starting write from a run whose session began before a regression this process+observed, so a run that finishes after its finding was detected again or+re-investigated leaves the newer state alone. It enrols, commands, updates or+removes no agent and touches no agent identity or credential.diff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.mdindex 1c5d9f5bf3..f03e048fec 100644--- a/docs/release-control/v6/internal/subsystems/ai-runtime.md+++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md@@ -7527,3 +7527,75 @@ Proofs:This changes outcome recording and run limits only. It widens no mutation,approval or execution boundary, and it is not measured improvement oninstalled systems.++### Orchestrator writes land only on the run's own occurrence (23 October 2026)++The private orchestrator writes a finding's investigation fields through+`orchestratorFindingsAdapter.Update` in `internal/api/ai_handlers.go`, which now+takes each run's start from the orchestrator's own session store. A write that+closes a run goes through `FindingsStore.CompleteInvestigation`. That records+the run's status, outcome, close time and attempt count only while the finding+still names the run: its session must be the finding's current+`InvestigationSessionID`, and its start must not predate the finding's latest+regression. A write that starts a run (`pending` or `running`) goes through+`FindingsStore.StartInvestigation`. That installs the run as before, unless its+session started before the finding's latest regression, which means the run was+started for an earlier occurrence.++Before this, a run still in flight when its finding was resolved and detected+again, or when the stuck-run sweep gave it up and a retry started another run,+could overwrite the newer state with its own session, status, outcome and+attempts. A stale `resolved` then closed the new occurrence's investigation and+stopped Patrol investigating it again. A regression that landed between a run+recording its session and marking the finding as running let that run install+the earlier occurrence's attempt count, or replace a newer run. A late result+from the run the finding still names still lands after the sweep gave it up, so+the run's real result replaces the sweep's `timed_out`.++Starts and closes compare a run's start only with an occurrence start this+process observed: a regression, or the finding's removal by `ClearAll` or+`Cleanup`, after which the same ID detected again is a new finding. The store+keeps these, unpersisted and with the clock's monotonic reading, for a day,+and refuses a start from a session older than that, which could predate one it+has forgotten. A regression an earlier process recorded predates every run of+this one, so it orders none of them, even when the clock was set back across a+restart. Comparing wall times there would refuse every new run until the clock+caught up, starting a run every Patrol cycle or exhausting the attempt limit.+Investigation records and action outcomes keep comparing with+`LastRegressionAt`, because they also arrive for sessions of earlier processes,+stamped by the same clock. After a clock set back across a restart they can+still refuse a newer run's results, as before.++A run whose result does not count, but which the finding still shows as+running, is closed at once through `RecoverStuckInvestigation`, the way the+sweep would close it later: as `interrupted`, or completed with an outcome the+action lifecycle recorded during the run, which is kept. Its attempt count+stays as its admission wrote it. Through the adapter this is rare, because the+start check keeps a run from an earlier occurrence off the finding: it takes a+run with no recorded start, or findings reloaded from disk that still name an+earlier run. A run that records its session after a regression, from a+snapshot taken before it, is still installed with that snapshot's attempt+count. When two runs are dispatched for one finding at once, the run whose+start lands last is the finding's run and only its result is recorded, where+before the run that closed last won. The `aicontracts` boundary is unchanged,+and the orchestrator needs no change.++Proofs:+- `findings_test.go`: `StartInvestigation` skips a run from an earlier+ occurrence or from before the finding was cleared or cleaned up, and installs+ one with no recorded start or against a regression an earlier process+ recorded; occurrence starts past the retention window are forgotten, and a+ session older than that window is refused.+ `CompleteInvestigation` refuses another session, an unnamed run and an+ unknown finding; records the finding's own run, including a late result, a+ run with no recorded start before any regression, and a regression an+ earlier process recorded; and closes a run from before a regression the way+ the sweep does, without projecting its verified fix and keeping a queued fix.+ An earlier process's action outcome stays refused against `LastRegressionAt`.+- `ai_handlers_investigation_additional_test.go`: through the adapter the+ orchestrator receives, a run's result skips a later occurrence and a newer+ run, a run started for an earlier occurrence is not installed and does not+ replace the current run, and the current run's result lands.++This changes which writes land, not what they record. It widens no mutation,+approval or execution boundary.diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.mdindex 1567cc8f6f..8effc9c85b 100644--- a/docs/release-control/v6/internal/subsystems/api-contracts.md+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md@@ -12526,3 +12526,21 @@ payloads gain one optional field. An older orchestrator ignores the conclusionand keeps reading prose markers, and records no deadline, so core measures itsruns from their start. Proofs: `patrol_scope_tools_test.go`, `names_test.go`,`tools_conclude_test.go` and `contracts_test.go`.++### Orchestrator finding writes check run ownership (23 October 2026)++`orchestratorFindingsAdapter.Update`, the `aicontracts.OrchestratorFindingsStore`+the private orchestrator receives, now returns false for a write that the+finding's current occurrence does not own, and the newer state stays. A write+that closes a run is refused when another session is current, or when the run+cannot be shown to have started after the finding's current occurrence began:+a regression this process observed, or the finding's removal by `ClearAll` or+`Cleanup`, after which the same ID detected again is a new finding. If the+finding still shows such a run as running, the run is closed the way the+stuck-run sweep closes it, not with its own result. A write that starts a run+is refused when the run's session started before that occurrence began, or+more than a day ago, and applies as before otherwise. A regression an earlier+process recorded refuses neither. The adapter takes each run's start from the+orchestrator's session store, so the Pulse–enterprise boundary is unchanged: no+`aicontracts` type, field or method is added, and no route or payload changes.+Proof: `ai_handlers_investigation_additional_test.go`.diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex 1b49b14167..7c6853b53c 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -7174,3 +7174,8 @@ private orchestrator, make investigation concurrency and timeout follow thePatrol provider, and label the `unclassified` and `interrupted` outcomes in thedigest email. Recovery evidence, restore rules and storage capabilities areunchanged, and Patrol gains no storage remediation capability.++The `internal/api` change that makes the investigation orchestrator's finding+writes check run ownership opens no storage, backup or recovery path. It decides+only whether a run's start or result is recorded on a Patrol finding. Recovery+evidence, restore rules and storage capabilities are unchanged.diff --git a/internal/ai/findings.go b/internal/ai/findings.goindex 4126defa10..c01b5c7ded 100644--- a/internal/ai/findings.go+++ b/internal/ai/findings.go@@ -20,6 +20,9 @@ const (investigationTimeoutCooldown = 10 * time.Minute // Shorter cooldown for timeout failuresinvestigationFixRetryCooldown = 20 * time.Minute // Retry faster after failed fix attemptsmaxInvestigationAttempts = 3 // Maximum number of investigation retries+ // How long occurrence starts are kept. StartInvestigation refuses a run+ // whose session is older, so a forgotten start cannot let one in.+ occurrenceStartRetention = 24 * time.Hour)// FindingSeverity represents how urgent a finding is@@ -841,6 +844,9 @@ type FindingsStore struct {activeCounts map[FindingSeverity]int// User-defined suppression rules (separate from dismissed findings)suppressionRules map[string]*SuppressionRule+ // When this process last saw each finding begin a new occurrence, with the+ // clock's monotonic reading. Never persisted: see liveRunOccurrenceLocked.+ occurrenceStarts map[string]time.Time// Persistence layer (optional)persistence FindingsPersistence// Debounce save operations@@ -965,6 +971,7 @@ func NewFindingsStore() *FindingsStore {byResource: make(map[string][]string),activeCounts: make(map[FindingSeverity]int),suppressionRules: make(map[string]*SuppressionRule),+ occurrenceStarts: make(map[string]time.Time),saveDebounce: 5 * time.Second, // Debounce saves by 5 seconds}}@@ -2097,6 +2104,7 @@ func (s *FindingsStore) Add(f *Finding) bool {existing.RegressionCount++now := time.Now()existing.LastRegressionAt = &now+ s.occurrenceStarts[existing.ID] = nowmeta := map[string]string{}if prevResolvedAt != nil {meta["previous_resolved_at"] = prevResolvedAt.Format(time.RFC3339)@@ -2636,6 +2644,19 @@ func (f *Finding) acceptsInvestigationResult(sessionID string, startedAt time.Tireturn f.LastRegressionAt == nil || (!startedAt.IsZero() && !startedAt.Before(*f.LastRegressionAt))}+// liveRunOccurrenceLocked returns when the finding's current occurrence began,+// as far as a run of this process can tell: the latest regression this process+// saw, or the finding's removal, after which the same ID detected again is a new+// finding. The monotonic clock orders it exactly against the run's start. A+// regression an earlier process recorded predates every run of this one, so it+// orders none of them, whatever the clock now reads. A result for a session of+// an earlier process, such as an action outcome, is still judged against+// LastRegressionAt.+func (s *FindingsStore) liveRunOccurrenceLocked(id string) (time.Time, bool) {+ startedAt, ok := s.occurrenceStarts[id]+ return startedAt, ok+}+func (s *FindingsStore) updateInvestigationOutcomeLocked(f *Finding, outcome string) {prevOutcome := f.InvestigationOutcomef.InvestigationOutcome = outcome@@ -2665,6 +2686,73 @@ func (s *FindingsStore) UpdateInvestigation(id, sessionID, status, outcome strinreturn false}+ s.updateInvestigationLocked(f, sessionID, status, outcome, lastInvestigatedAt, attempts)+ s.mu.Unlock()+ s.scheduleSave()+ return true+}++// StartInvestigation installs a run that is starting on a finding. A run whose+// session started before the finding's current occurrence began, as+// liveRunOccurrenceLocked tells it, was started for an earlier occurrence, so+// it is not installed: it would replace the current occurrence's investigation+// state, or a newer run's. Neither is a run whose session started longer ago+// than the store keeps occurrence starts, since it could predate one already+// forgotten. runStartedAt is the start of the run's session, or zero when the+// orchestrator holds none; such a run is installed as before.+func (s *FindingsStore) StartInvestigation(id, sessionID string, runStartedAt time.Time, status, outcome string, lastInvestigatedAt *time.Time, attempts int) bool {+ s.mu.Lock()+ f, exists := s.findings[id]+ if !exists {+ s.mu.Unlock()+ return false+ }+ occurrenceAt, ok := s.liveRunOccurrenceLocked(id)+ if !runStartedAt.IsZero() && ((ok && runStartedAt.Before(occurrenceAt)) || time.Since(runStartedAt) > occurrenceStartRetention) {+ s.mu.Unlock()+ return false+ }+ s.updateInvestigationLocked(f, sessionID, status, outcome, lastInvestigatedAt, attempts)+ s.mu.Unlock()+ s.scheduleSave()+ return true+}++// CompleteInvestigation records how a run of this process ended: its status,+// outcome, close time and attempt count. It records them only while the finding+// still names that run, so a run that ends after its finding moved on cannot+// overwrite the newer state: the run's session must be the finding's current+// one, and the run must not have started before the finding's current+// occurrence began, as liveRunOccurrenceLocked tells it. runStartedAt is the+// start of the run's session, or zero when the orchestrator holds none, which+// cannot prove ownership once this process has seen an occurrence begin. A run+// whose result does not count, but which the finding still shows as running,+// is closed now through RecoverStuckInvestigation, as the stuck-run sweep would+// later close it, rather than left running. It reports whether the run's+// result was recorded.+func (s *FindingsStore) CompleteInvestigation(id, sessionID string, runStartedAt time.Time, status, outcome string, lastInvestigatedAt *time.Time, attempts int) bool {+ s.mu.Lock()+ f, exists := s.findings[id]+ if !exists || sessionID == "" || f.InvestigationSessionID != sessionID {+ s.mu.Unlock()+ return false+ }+ if occurrenceAt, ok := s.liveRunOccurrenceLocked(id); ok && (runStartedAt.IsZero() || runStartedAt.Before(occurrenceAt)) {+ s.mu.Unlock()+ closedAt := time.Now()+ if lastInvestigatedAt != nil {+ closedAt = *lastInvestigatedAt+ }+ s.RecoverStuckInvestigation(id, sessionID, runStartedAt, InvestigationStatusFailed, InvestigationOutcomeInterrupted, closedAt)+ return false+ }+ s.updateInvestigationLocked(f, sessionID, status, outcome, lastInvestigatedAt, attempts)+ s.mu.Unlock()+ s.scheduleSave()+ return true+}++func (s *FindingsStore) updateInvestigationLocked(f *Finding, sessionID, status, outcome string, lastInvestigatedAt *time.Time, attempts int) {prevStatus := f.InvestigationStatusprevOutcome := f.InvestigationOutcomef.InvestigationSessionID = sessionID@@ -2683,9 +2771,6 @@ func (s *FindingsStore) UpdateInvestigation(id, sessionID, status, outcome strin})}s.syncLoopStateLocked(f)- s.mu.Unlock()- s.scheduleSave()- return true}// RecoverStuckInvestigation closes a run that is no longer live with the given@@ -2911,6 +2996,10 @@ func (s *FindingsStore) GetAll(startTime *time.Time) []*Finding {func (s *FindingsStore) ClearAll() int {s.mu.Lock()count := len(s.findings)+ now := time.Now()+ for id := range s.findings {+ s.occurrenceStarts[id] = now+ }s.findings = make(map[string]*Finding)s.byResource = make(map[string][]string)s.activeCounts = make(map[FindingSeverity]int)@@ -2944,6 +3033,7 @@ func (s *FindingsStore) Cleanup(maxAge time.Duration) int {if shouldRemove {delete(s.findings, id)+ s.occurrenceStarts[id] = now// Clean up resource indexids := s.byResource[f.ResourceID]for i, fid := range ids {@@ -2955,6 +3045,11 @@ func (s *FindingsStore) Cleanup(maxAge time.Duration) int {removed++}}+ for id, startedAt := range s.occurrenceStarts {+ if startedAt.Before(now.Add(-occurrenceStartRetention)) {+ delete(s.occurrenceStarts, id)+ }+ }return removed}diff --git a/internal/ai/findings_test.go b/internal/ai/findings_test.goindex 0210625557..ad8b397150 100644--- a/internal/ai/findings_test.go+++ b/internal/ai/findings_test.go@@ -2483,3 +2483,259 @@ func TestInterruptedAndUnclassifiedInvestigationsStayEligible(t *testing.T) {t.Fatal("a real needs_attention conclusion stays terminal")}}++// regressHere resolves a finding and detects it again, a regression this+// process observes, and returns when it happened.+func regressHere(t *testing.T, findings *FindingsStore, id string) time.Time {+ t.Helper()+ f := findings.Get(id)+ if f == nil || !findings.Resolve(id, true) {+ t.Fatalf("could not resolve %s", id)+ }+ findings.Add(&Finding{ID: id, ResourceID: f.ResourceID, ResourceType: f.ResourceType, Title: f.Title, Severity: f.Severity, Category: f.Category})+ got := findings.Get(id)+ if got.LastRegressionAt == nil {+ t.Fatalf("%s did not regress", id)+ }+ return *got.LastRegressionAt+}++func TestStartInvestigationSkipsARunFromAnEarlierOccurrence(t *testing.T) {+ findings := NewFindingsStore()+ for _, id := range []string{"regressed", "unrecorded", "first", "recorded-earlier"} {+ f := runningInvestigationFinding(id, "", nil)+ f.InvestigationStatus = ""+ f.InvestigationAttempts = 0+ if id == "recorded-earlier" {+ // Loaded after a restart, with the clock since set back.+ recordedAt := time.Now().Add(time.Hour)+ f.LastRegressionAt = &recordedAt+ }+ findings.Add(f)+ }+ regressedAt := regressHere(t, findings, "regressed")+ regressHere(t, findings, "unrecorded")+ start := func(id, sessionID string, runStartedAt time.Time) bool {+ return findings.StartInvestigation(id, sessionID, runStartedAt, string(InvestigationStatusRunning), "", nil, 3)+ }++ if start("regressed", "chat-earlier", regressedAt.Add(-time.Minute)) {+ t.Fatal("a run started for the earlier occurrence was installed")+ }+ if got := findings.Get("regressed"); got.InvestigationSessionID != "" || got.InvestigationStatus != "" || got.InvestigationAttempts != 0 {+ t.Fatalf("regressed = session %q %s, %d attempts; want the current occurrence untouched", got.InvestigationSessionID, got.InvestigationStatus, got.InvestigationAttempts)+ }+ if !start("regressed", "chat-current", time.Now()) {+ t.Fatal("a run of the current occurrence was refused")+ }+ assertInvestigationState(t, findings, "regressed", InvestigationStatusRunning, "")+ // A run with no recorded start, or on a finding that never regressed, is+ // installed as before.+ if !start("unrecorded", "chat-unrecorded", time.Time{}) {+ t.Fatal("a run with no recorded start was refused")+ }+ if !start("first", "chat-first", time.Now().Add(-time.Hour)) {+ t.Fatal("a run on a finding that never regressed was refused")+ }+ // A regression an earlier process recorded predates every run of this one,+ // however the clock now reads; refusing would start a run every cycle.+ if !start("recorded-earlier", "chat-recorded-earlier", time.Now()) {+ t.Fatal("a run was refused against a regression an earlier process recorded")+ }+ if start("missing", "chat-missing", time.Now()) {+ t.Fatal("a run was installed on an unknown finding")+ }+}++func TestCompleteInvestigationRecordsOnlyTheRunTheFindingNames(t *testing.T) {+ started := time.Now().Add(-time.Minute)+ findings := NewFindingsStore()+ findings.Add(runningInvestigationFinding("current", "chat-current", nil))+ findings.Add(runningInvestigationFinding("unrecorded", "chat-unrecorded", nil))+ findings.Add(runningInvestigationFinding("replaced", "chat-newer", nil))+ idle := runningInvestigationFinding("idle", "", nil)+ idle.InvestigationStatus = ""+ findings.Add(idle)+ // The stuck-run sweep gave this run up as timed out; it reports back after all.+ late := runningInvestigationFinding("late", "chat-late", nil)+ late.InvestigationStatus = string(InvestigationStatusFailed)+ late.InvestigationOutcome = string(InvestigationOutcomeTimedOut)+ findings.Add(late)+ // Loaded after a restart, with the clock since set back.+ recordedEarlier := runningInvestigationFinding("recorded-earlier", "chat-recorded-earlier", nil)+ recordedAt := time.Now().Add(time.Hour)+ recordedEarlier.LastRegressionAt = &recordedAt+ findings.Add(recordedEarlier)++ closedAt := time.Now()+ complete := func(id, sessionID string, runStartedAt time.Time, status InvestigationStatus, outcome InvestigationOutcome, attempts int) bool {+ return findings.CompleteInvestigation(id, sessionID, runStartedAt, string(status), string(outcome), &closedAt, attempts)+ }+ if complete("replaced", "chat-older", started, InvestigationStatusCompleted, InvestigationOutcomeResolved, 1) {+ t.Fatal("an older run's result overwrote the run that replaced it")+ }+ assertInvestigationState(t, findings, "replaced", InvestigationStatusRunning, "")+ if complete("idle", "chat-other", started, InvestigationStatusCompleted, InvestigationOutcomeResolved, 1) {+ t.Fatal("a run the finding does not name recorded its result")+ }+ assertInvestigationState(t, findings, "idle", "", "")+ if complete("missing", "chat-missing", started, InvestigationStatusCompleted, InvestigationOutcomeResolved, 1) {+ t.Fatal("a result was recorded for an unknown finding")+ }++ if !complete("current", "chat-current", started, InvestigationStatusCompleted, InvestigationOutcomeCannotFix, 2) {+ t.Fatal("the finding's own run was refused")+ }+ assertInvestigationState(t, findings, "current", InvestigationStatusCompleted, InvestigationOutcomeCannotFix)+ if got := findings.Get("current"); got.InvestigationAttempts != 2 || got.LastInvestigatedAt == nil || !got.LastInvestigatedAt.Equal(closedAt) {+ t.Fatalf("current = %d attempts, last investigated %v; want 2 at %v", got.InvestigationAttempts, got.LastInvestigatedAt, closedAt)+ }+ // With no regression on record, a run with no recorded start still owns+ // the only occurrence there has been.+ if !complete("unrecorded", "chat-unrecorded", time.Time{}, InvestigationStatusFailed, InvestigationOutcomeTimedOut, 1) {+ t.Fatal("a run with no recorded start was refused before any regression")+ }+ assertInvestigationState(t, findings, "unrecorded", InvestigationStatusFailed, InvestigationOutcomeTimedOut)+ if !complete("late", "chat-late", started, InvestigationStatusCompleted, InvestigationOutcomeFixVerified, 1) {+ t.Fatal("a late result from the finding's own run was refused")+ }+ assertInvestigationState(t, findings, "late", InvestigationStatusCompleted, InvestigationOutcomeFixVerified)+ if findings.Get("late").ResolvedAt == nil {+ t.Fatal("a verified fix from the finding's own run did not resolve it")+ }+ // A regression an earlier process recorded predates every run of this one.+ if !complete("recorded-earlier", "chat-recorded-earlier", started, InvestigationStatusCompleted, InvestigationOutcomeNeedsAttention, 1) {+ t.Fatal("a run's result was refused against a regression an earlier process recorded")+ }+ assertInvestigationState(t, findings, "recorded-earlier", InvestigationStatusCompleted, InvestigationOutcomeNeedsAttention)+ // An action outcome for a session of an earlier process is still judged+ // against LastRegressionAt, which the same clock stamped.+ earlier := &aicontracts.InvestigationSession{ID: "inv-earlier", FindingID: "recorded-earlier", SessionID: "chat-earlier", StartedAt: recordedAt.Add(-time.Minute)}+ findings.UpdateInvestigation("recorded-earlier", "", "", "", nil, 0) // as the regression left it+ if applied, _ := findings.ReconcileInvestigationOutcome("recorded-earlier", earlier, string(InvestigationOutcomeFixVerified)); applied {+ t.Fatal("an earlier process's action outcome landed on the current occurrence")+ }+}++func TestCompleteInvestigationClosesARunFromAnEarlierOccurrenceAsTheSweepWould(t *testing.T) {+ ids := []string{"stale", "unrecorded", "settled", "queued", "current"}+ findings := NewFindingsStore()+ for _, id := range ids {+ findings.Add(runningInvestigationFinding(id, "", nil))+ }+ var regressedAt time.Time+ for _, id := range ids {+ regressedAt = regressHere(t, findings, id)+ // The run is installed on the current occurrence, as through a path+ // that does not check when it started.+ status, outcome := InvestigationStatusRunning, InvestigationOutcome("")+ switch id {+ case "settled":+ // The stuck-run sweep already closed it.+ status, outcome = InvestigationStatusFailed, InvestigationOutcomeInterrupted+ case "queued":+ // The action lifecycle recorded a queued fix during the run.+ outcome = InvestigationOutcomeFixQueued+ }+ findings.UpdateInvestigation(id, "chat-"+id, string(status), string(outcome), nil, 1)+ }+ beforeRegression := regressedAt.Add(-time.Minute)++ closedAt := time.Now()+ complete := func(id string, runStartedAt time.Time) bool {+ return findings.CompleteInvestigation(id, "chat-"+id, runStartedAt, string(InvestigationStatusCompleted), string(InvestigationOutcomeFixVerified), &closedAt, 2)+ }+ // The run started before the finding's current occurrence: its verified+ // fix was for the earlier one.+ if complete("stale", beforeRegression) {+ t.Fatal("a run from before the regression recorded its result")+ }+ assertInvestigationState(t, findings, "stale", InvestigationStatusFailed, InvestigationOutcomeInterrupted)+ stale := findings.Get("stale")+ if stale.ResolvedAt != nil {+ t.Fatal("an earlier occurrence's verified fix resolved the current one")+ }+ if stale.LastInvestigatedAt == nil || !stale.LastInvestigatedAt.Equal(closedAt) || stale.InvestigationAttempts != 1 {+ t.Fatalf("stale = %d attempts, last investigated %v; want 1 at the close %v", stale.InvestigationAttempts, stale.LastInvestigatedAt, closedAt)+ }+ if !stale.ShouldInvestigate("approval") {+ t.Fatal("the current occurrence must be free for another run")+ }+ // Without a recorded start the run cannot be shown to belong to the+ // current occurrence.+ if complete("unrecorded", time.Time{}) {+ t.Fatal("a run with no recorded start recorded its result after a regression")+ }+ assertInvestigationState(t, findings, "unrecorded", InvestigationStatusFailed, InvestigationOutcomeInterrupted)+ if complete("settled", beforeRegression) {+ t.Fatal("a run from before the regression recorded its result")+ }+ assertInvestigationState(t, findings, "settled", InvestigationStatusFailed, InvestigationOutcomeInterrupted)+ // The close keeps an outcome the action lifecycle recorded, as the sweep+ // does, so the queued fix still blocks another investigation.+ if complete("queued", beforeRegression) {+ t.Fatal("a run from before the regression recorded its result")+ }+ assertInvestigationState(t, findings, "queued", InvestigationStatusCompleted, InvestigationOutcomeFixQueued)+ if findings.Get("queued").ShouldInvestigate("approval") {+ t.Fatal("a queued fix must keep blocking another investigation")+ }++ if !complete("current", time.Now()) {+ t.Fatal("a run of the current occurrence was refused")+ }+ assertInvestigationState(t, findings, "current", InvestigationStatusCompleted, InvestigationOutcomeFixVerified)+ if current := findings.Get("current"); current.ResolvedAt == nil || current.InvestigationAttempts != 2 {+ t.Fatalf("current = resolved at %v, %d attempts; want resolved with the run's 2 attempts", current.ResolvedAt, current.InvestigationAttempts)+ }+}++func TestStartInvestigationSkipsARunFromBeforeTheFindingWasRemoved(t *testing.T) {+ start := func(findings *FindingsStore, id, sessionID string, runStartedAt time.Time) bool {+ return findings.StartInvestigation(id, sessionID, runStartedAt, string(InvestigationStatusRunning), "", nil, 1)+ }++ // Cleared by the operator, then detected again: the same ID, a new finding.+ cleared := NewFindingsStore()+ cleared.Add(runningInvestigationFinding("cleared", "", nil))+ runStartedAt := time.Now()+ cleared.ClearAll()+ cleared.Add(runningInvestigationFinding("cleared", "", nil))+ if start(cleared, "cleared", "chat-before-clear", runStartedAt) {+ t.Fatal("a run from before the finding was cleared was installed on the new one")+ }+ if !start(cleared, "cleared", "chat-after-clear", time.Now()) {+ t.Fatal("a run of the new finding was refused")+ }++ // Removed as an old resolved finding, then detected again.+ cleaned := NewFindingsStore()+ cleaned.Add(runningInvestigationFinding("cleaned", "", nil))+ runStartedAt = time.Now()+ cleaned.Resolve("cleaned", true)+ if removed := cleaned.Cleanup(0); removed != 1 {+ t.Fatalf("Cleanup removed %d findings, want 1", removed)+ }+ cleaned.Add(runningInvestigationFinding("cleaned", "", nil))+ if start(cleaned, "cleaned", "chat-before-cleanup", runStartedAt) {+ t.Fatal("a run from before the finding was removed was installed on the new one")+ }++ // An occurrence start no live run can predate is forgotten.+ cleaned.occurrenceStarts["long-gone"] = time.Now().Add(-occurrenceStartRetention - time.Minute)+ cleaned.Cleanup(0)+ if _, ok := cleaned.occurrenceStarts["long-gone"]; ok {+ t.Fatal("an occurrence start past the retention window was kept")+ }++ // So a session older than that could predate an occurrence start the store+ // no longer holds, even where none is on record.+ fresh := NewFindingsStore()+ fresh.Add(runningInvestigationFinding("fresh", "", nil))+ if start(fresh, "fresh", "chat-ancient", time.Now().Add(-occurrenceStartRetention-time.Minute)) {+ t.Fatal("a run from a session older than the retention window was installed")+ }+ if !start(fresh, "fresh", "chat-recent", time.Now().Add(-time.Hour)) {+ t.Fatal("a run from a recent session was refused")+ }+}diff --git a/internal/api/ai_handlers.go b/internal/api/ai_handlers.goindex 02713e706b..07abae14b8 100644--- a/internal/api/ai_handlers.go+++ b/internal/api/ai_handlers.go@@ -1932,7 +1932,7 @@ func (h *AISettingsHandler) setupInvestigationOrchestrator(orgID string, svc *ailog.Warn().Str("orgID", orgID).Msg("Findings store not available for orchestrator")return}- findingsAdapter := &orchestratorFindingsAdapter{store: &findingsStoreWrapper{store: findingsStore}}+ findingsAdapter := &orchestratorFindingsAdapter{store: &findingsStoreWrapper{store: findingsStore}, sessions: store}// Get config for investigation settingscfg := svc.GetConfig()@@ -2111,6 +2111,9 @@ func (a *orchestratorChatAdapter) ListInvestigationTools(ctx context.Context) []// aicontracts.OrchestratorFindingsStore.type orchestratorFindingsAdapter struct {store *findingsStoreWrapper+ // sessions is the orchestrator's session store, which records when each+ // run started.+ sessions aicontracts.InvestigationStore}func (a *orchestratorFindingsAdapter) Get(id string) *aicontracts.Finding {@@ -2140,13 +2143,32 @@ func (a *orchestratorFindingsAdapter) Get(id string) *aicontracts.Finding {}}+// Update applies the orchestrator's investigation fields to a finding. Starting+// a run installs it, unless the run was started for an earlier occurrence of+// the finding. Every other write closes a run and lands only while the finding+// still names that run, so a run that ends after its finding was re-detected,+// or after a newer run took over, cannot overwrite the newer state.func (a *orchestratorFindingsAdapter) Update(f *aicontracts.Finding) bool {if a.store == nil || f == nil {return false}- return a.store.UpdateInvestigation(+ runStartedAt := a.runStartedAt(f.ID, f.InvestigationSessionID)+ switch aicontracts.InvestigationStatus(f.InvestigationStatus) {+ case aicontracts.InvestigationStatusPending, aicontracts.InvestigationStatusRunning:+ return a.store.StartInvestigation(+ f.ID,+ f.InvestigationSessionID,+ runStartedAt,+ f.InvestigationStatus,+ f.InvestigationOutcome,+ f.LastInvestigatedAt,+ f.InvestigationAttempts,+ )+ }+ return a.store.CompleteInvestigation(f.ID,f.InvestigationSessionID,+ runStartedAt,f.InvestigationStatus,f.InvestigationOutcome,f.LastInvestigatedAt,@@ -2154,6 +2176,20 @@ func (a *orchestratorFindingsAdapter) Update(f *aicontracts.Finding) bool {)}+// runStartedAt returns when the orchestrator's session for a run started, or+// zero when its session store holds no record of the run.+func (a *orchestratorFindingsAdapter) runStartedAt(findingID, sessionID string) time.Time {+ if a.sessions == nil {+ return time.Time{}+ }+ for _, session := range a.sessions.GetByFinding(findingID) {+ if session != nil && session.SessionID == sessionID {+ return session.StartedAt+ }+ }+ return time.Time{}+}+// patrolMetricsCallbackAdapter implements aicontracts.OrchestratorMetricsCallback// by delegating to the global PatrolMetrics singleton.type patrolMetricsCallbackAdapter struct{}@@ -2189,6 +2225,20 @@ func (w *findingsStoreWrapper) UpdateInvestigation(id, sessionID, status, outcomreturn w.store.UpdateInvestigation(id, sessionID, status, outcome, lastInvestigatedAt, attempts)}+func (w *findingsStoreWrapper) StartInvestigation(id, sessionID string, runStartedAt time.Time, status, outcome string, lastInvestigatedAt *time.Time, attempts int) bool {+ if w.store == nil {+ return false+ }+ return w.store.StartInvestigation(id, sessionID, runStartedAt, status, outcome, lastInvestigatedAt, attempts)+}++func (w *findingsStoreWrapper) CompleteInvestigation(id, sessionID string, runStartedAt time.Time, status, outcome string, lastInvestigatedAt *time.Time, attempts int) bool {+ if w.store == nil {+ return false+ }+ return w.store.CompleteInvestigation(id, sessionID, runStartedAt, status, outcome, lastInvestigatedAt, attempts)+}+// AISettingsResponse is returned by GET /api/settings/ai// API keys are masked for securitytype AISettingsResponse struct {diff --git a/internal/api/ai_handlers_investigation_additional_test.go b/internal/api/ai_handlers_investigation_additional_test.goindex c51e310375..b29f0054eb 100644--- a/internal/api/ai_handlers_investigation_additional_test.go+++ b/internal/api/ai_handlers_investigation_additional_test.go@@ -530,6 +530,12 @@ func TestFindingsStoreWrapper_GetAndUpdate(t *testing.T) {if nilWrapper.UpdateInvestigation("finding-1", "session-1", "running", "outcome", nil, 1) {t.Fatalf("expected nil store update to return false")}+ if nilWrapper.StartInvestigation("finding-1", "session-1", time.Time{}, "running", "", nil, 1) {+ t.Fatalf("expected nil store start to return false")+ }+ if nilWrapper.CompleteInvestigation("finding-1", "session-1", time.Time{}, "completed", "outcome", nil, 1) {+ t.Fatalf("expected nil store completion to return false")+ }}func TestHandleClearAllFindings(t *testing.T) {@@ -933,3 +939,195 @@ func TestSetupInvestigationOrchestratorUsesSubscriptionAgentLimits(t *testing.T)t.Fatalf("Timeout = %s, want %s", created[0].Timeout, want)}}++// orchestratorFindingsFor wires a tenant's investigation orchestrator and+// returns the findings and session stores it was built with, plus the+// tenant's findings.+func orchestratorFindingsFor(t *testing.T) (aicontracts.OrchestratorFindingsStore, aicontracts.InvestigationStore, *ai.FindingsStore) {+ t.Helper()+ prevStoreFactory := getCreateInvestigationStore()+ SetCreateInvestigationStore(func(dataDir string) aicontracts.InvestigationStore {+ return newTestInvestigationStore()+ })+ t.Cleanup(func() { SetCreateInvestigationStore(prevStoreFactory) })++ var deps aicontracts.OrchestratorDeps+ prevOrchestratorFactory := getCreateInvestigationOrchestrator()+ SetCreateInvestigationOrchestrator(func(d aicontracts.OrchestratorDeps) aicontracts.InvestigationOrchestrator {+ deps = d+ return &stubInvestigationOrchestrator{}+ })+ t.Cleanup(func() { SetCreateInvestigationOrchestrator(prevOrchestratorFactory) })++ tenantSvc := ai.NewService(nil, nil)+ tenantSvc.SetStateProvider(&MockStateProvider{})+ handler := &AISettingsHandler{+ investigationStores: make(map[string]aicontracts.InvestigationStore),+ }+ handler.chatHandler = &AIHandler{+ services: map[string]AIService{+ "tenant-1": chat.NewService(chat.Config{AIConfig: config.NewDefaultAIConfig()}),+ },+ }+ handler.setupInvestigationOrchestrator("tenant-1", tenantSvc)+ if deps.FindingsStore == nil || deps.Store == nil {+ t.Fatal("investigation orchestrator was not wired")+ }+ return deps.FindingsStore, deps.Store, tenantSvc.GetPatrolService().GetFindings()+}++func investigatedFinding(id string) *ai.Finding {+ return &ai.Finding{+ ID: id,+ Severity: ai.FindingSeverityWarning,+ Category: ai.FindingCategoryPerformance,+ ResourceID: "vm-" + id,+ ResourceName: "vm-" + id,+ ResourceType: "vm",+ Title: "Memory pressure on vm-" + id,+ Description: "Memory usage is above 90%",+ }+}++// startOrchestratorRun does what the orchestrator does when it starts a run:+// it records the run's session, then marks the finding as running under it.+func startOrchestratorRun(t *testing.T, findings aicontracts.OrchestratorFindingsStore, sessions aicontracts.InvestigationStore, findingID, chatSessionID string) aicontracts.Finding {+ t.Helper()+ run := findings.Get(findingID)+ if run == nil {+ t.Fatalf("finding %s not found", findingID)+ }+ sessions.Create(findingID, chatSessionID)+ run.InvestigationSessionID = chatSessionID+ run.InvestigationStatus = string(aicontracts.InvestigationStatusRunning)+ run.InvestigationOutcome = ""+ run.InvestigationAttempts+++ if !findings.Update(run) {+ t.Fatalf("starting run %s was not recorded", chatSessionID)+ }+ return *run+}++// finishOrchestratorRun is the orchestrator's finding write when a run ends.+func finishOrchestratorRun(findings aicontracts.OrchestratorFindingsStore, run aicontracts.Finding, status aicontracts.InvestigationStatus, outcome aicontracts.InvestigationOutcome) bool {+ now := time.Now()+ run.InvestigationStatus = string(status)+ run.InvestigationOutcome = string(outcome)+ run.LastInvestigatedAt = &now+ return findings.Update(&run)+}++func assertOrchestratorRunState(t *testing.T, findings *ai.FindingsStore, id, sessionID string, status aicontracts.InvestigationStatus, outcome aicontracts.InvestigationOutcome, attempts int) {+ t.Helper()+ got := findings.Get(id)+ if got == nil {+ t.Fatalf("finding %s not found", id)+ }+ if got.InvestigationSessionID != sessionID || got.InvestigationStatus != string(status) ||+ got.InvestigationOutcome != string(outcome) || got.InvestigationAttempts != attempts {+ t.Fatalf("%s investigation = session %q %s/%s, %d attempts; want session %q %s/%s, %d attempts",+ id, got.InvestigationSessionID, got.InvestigationStatus, got.InvestigationOutcome, got.InvestigationAttempts,+ sessionID, status, outcome, attempts)+ }+}++func TestOrchestratorRunResultSkipsALaterOccurrence(t *testing.T) {+ orchestratorFindings, sessions, findings := orchestratorFindingsFor(t)+ findings.Add(investigatedFinding("finding-1"))+ runA := startOrchestratorRun(t, orchestratorFindings, sessions, "finding-1", "chat-a")++ // While A is in flight the finding resolves and is detected again.+ findings.Resolve("finding-1", true)+ findings.Add(investigatedFinding("finding-1"))++ // A concludes for the occurrence it investigated, not the current one.+ applied := finishOrchestratorRun(orchestratorFindings, runA, aicontracts.InvestigationStatusCompleted, aicontracts.OutcomeResolved)+ assertOrchestratorRunState(t, findings, "finding-1", "", "", "", 0)+ if applied {+ t.Fatal("a run of the earlier occurrence reported closing the current one")+ }+ if got := findings.Get("finding-1"); got.LastInvestigatedAt != nil || !got.ShouldInvestigate("approval") {+ t.Fatalf("the earlier occurrence's result kept the current one from being investigated: %+v", got)+ }++ // Nor does it land once a run of the current occurrence has started.+ runB := startOrchestratorRun(t, orchestratorFindings, sessions, "finding-1", "chat-b")+ if finishOrchestratorRun(orchestratorFindings, runA, aicontracts.InvestigationStatusFailed, aicontracts.OutcomeTimedOut) {+ t.Fatal("a run of the earlier occurrence overwrote the current run")+ }+ assertOrchestratorRunState(t, findings, "finding-1", "chat-b", aicontracts.InvestigationStatusRunning, "", 1)++ if !finishOrchestratorRun(orchestratorFindings, runB, aicontracts.InvestigationStatusCompleted, aicontracts.OutcomeNeedsAttention) {+ t.Fatal("the current run's result was refused")+ }+ assertOrchestratorRunState(t, findings, "finding-1", "chat-b", aicontracts.InvestigationStatusCompleted, aicontracts.OutcomeNeedsAttention, 1)+}++func TestOrchestratorRunFromAnEarlierOccurrenceIsNotInstalled(t *testing.T) {+ orchestratorFindings, sessions, findings := orchestratorFindingsFor(t)+ findings.Add(investigatedFinding("finding-1"))++ // A records its session for the first occurrence, with that occurrence's+ // attempt count; the finding resolves and is detected again before A marks+ // it as running.+ runA := orchestratorFindings.Get("finding-1")+ sessions.Create("finding-1", "chat-a")+ findings.Resolve("finding-1", true)+ findings.Add(investigatedFinding("finding-1"))+ runA.InvestigationSessionID = "chat-a"+ runA.InvestigationStatus = string(aicontracts.InvestigationStatusRunning)+ runA.InvestigationAttempts = 3+ if orchestratorFindings.Update(runA) {+ t.Fatal("a run started for the earlier occurrence was installed on the current one")+ }+ assertOrchestratorRunState(t, findings, "finding-1", "", "", "", 0)++ // Nor does A's admission replace a run of the current occurrence.+ runB := startOrchestratorRun(t, orchestratorFindings, sessions, "finding-1", "chat-b")+ if orchestratorFindings.Update(runA) {+ t.Fatal("a run started for the earlier occurrence replaced the current run")+ }+ assertOrchestratorRunState(t, findings, "finding-1", "chat-b", aicontracts.InvestigationStatusRunning, "", 1)+ if finishOrchestratorRun(orchestratorFindings, *runA, aicontracts.InvestigationStatusCompleted, aicontracts.OutcomeFixVerified) {+ t.Fatal("a run of the earlier occurrence recorded its result on the current one")+ }++ if !finishOrchestratorRun(orchestratorFindings, runB, aicontracts.InvestigationStatusCompleted, aicontracts.OutcomeCannotFix) {+ t.Fatal("the current run's result was refused")+ }+ assertOrchestratorRunState(t, findings, "finding-1", "chat-b", aicontracts.InvestigationStatusCompleted, aicontracts.OutcomeCannotFix, 1)+ if findings.Get("finding-1").ResolvedAt != nil {+ t.Fatal("the earlier occurrence's verified fix resolved the current one")+ }+}++func TestOrchestratorRunResultSkipsANewerRun(t *testing.T) {+ orchestratorFindings, sessions, findings := orchestratorFindingsFor(t)+ findings.Add(investigatedFinding("retried"))+ findings.Add(investigatedFinding("late"))+ retriedA := startOrchestratorRun(t, orchestratorFindings, sessions, "retried", "chat-retried-a")+ lateA := startOrchestratorRun(t, orchestratorFindings, sessions, "late", "chat-late-a")++ // The stuck-run sweep gave both runs up as timed out, and the retry+ // started another run for one of them.+ now := time.Now()+ findings.RecoverStuckInvestigation("retried", "chat-retried-a", time.Time{}, ai.InvestigationStatusFailed, ai.InvestigationOutcomeTimedOut, now)+ findings.RecoverStuckInvestigation("late", "chat-late-a", time.Time{}, ai.InvestigationStatusFailed, ai.InvestigationOutcomeTimedOut, now)+ retriedB := startOrchestratorRun(t, orchestratorFindings, sessions, "retried", "chat-retried-b")++ // The first runs report back after all.+ if finishOrchestratorRun(orchestratorFindings, retriedA, aicontracts.InvestigationStatusCompleted, aicontracts.OutcomeResolved) {+ t.Fatal("an earlier run overwrote the run that replaced it")+ }+ assertOrchestratorRunState(t, findings, "retried", "chat-retried-b", aicontracts.InvestigationStatusRunning, "", 2)+ // With no newer run, the run the finding still names has the real result.+ if !finishOrchestratorRun(orchestratorFindings, lateA, aicontracts.InvestigationStatusCompleted, aicontracts.OutcomeCannotFix) {+ t.Fatal("a late result from the finding's own run was refused")+ }+ assertOrchestratorRunState(t, findings, "late", "chat-late-a", aicontracts.InvestigationStatusCompleted, aicontracts.OutcomeCannotFix, 1)++ if !finishOrchestratorRun(orchestratorFindings, retriedB, aicontracts.InvestigationStatusFailed, aicontracts.OutcomeTimedOut) {+ t.Fatal("the newer run's result was refused")+ }+ assertOrchestratorRunState(t, findings, "retried", "chat-retried-b", aicontracts.InvestigationStatusFailed, aicontracts.OutcomeTimedOut, 2)+}