Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Keep stale investigation runs from overwriting newer finding state

rcourtmana408736bc4ac
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 045699ff64..9ae0f9e7ec 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -10185,3 +10185,10 @@ and `interrupted` outcomes. None of this enrols, commands, updates or removes an
agent, or touches agent identity or credentials. Investigations still reach
hosts only through the existing read-only evidence tools and the governed
action lifecycle.
+
+The investigation orchestrator's findings adapter in `internal/api/ai_handlers.go`
+now refuses a closing write from a run the finding no longer names, and a
+starting write from a run whose session began before a regression this process
+observed, so a run that finishes after its finding was detected again or
+re-investigated leaves the newer state alone. It enrols, commands, updates or
+removes no agent and touches no agent identity or credential.
diff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.md
index 1c5d9f5bf3..f03e048fec 100644
--- a/docs/release-control/v6/internal/subsystems/ai-runtime.md
+++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md
@@ -7527,3 +7527,75 @@ Proofs:
This changes outcome recording and run limits only. It widens no mutation,
approval or execution boundary, and it is not measured improvement on
installed systems.
+
+### Orchestrator writes land only on the run's own occurrence (23 October 2026)
+
+The private orchestrator writes a finding's investigation fields through
+`orchestratorFindingsAdapter.Update` in `internal/api/ai_handlers.go`, which now
+takes each run's start from the orchestrator's own session store. A write that
+closes a run goes through `FindingsStore.CompleteInvestigation`. That records
+the run's status, outcome, close time and attempt count only while the finding
+still names the run: its session must be the finding's current
+`InvestigationSessionID`, and its start must not predate the finding's latest
+regression. A write that starts a run (`pending` or `running`) goes through
+`FindingsStore.StartInvestigation`. That installs the run as before, unless its
+session started before the finding's latest regression, which means the run was
+started for an earlier occurrence.
+
+Before this, a run still in flight when its finding was resolved and detected
+again, or when the stuck-run sweep gave it up and a retry started another run,
+could overwrite the newer state with its own session, status, outcome and
+attempts. A stale `resolved` then closed the new occurrence's investigation and
+stopped Patrol investigating it again. A regression that landed between a run
+recording its session and marking the finding as running let that run install
+the earlier occurrence's attempt count, or replace a newer run. A late result
+from the run the finding still names still lands after the sweep gave it up, so
+the run's real result replaces the sweep's `timed_out`.
+
+Starts and closes compare a run's start only with an occurrence start this
+process observed: a regression, or the finding's removal by `ClearAll` or
+`Cleanup`, after which the same ID detected again is a new finding. The store
+keeps these, unpersisted and with the clock's monotonic reading, for a day,
+and refuses a start from a session older than that, which could predate one it
+has forgotten. A regression an earlier process recorded predates every run of
+this one, so it orders none of them, even when the clock was set back across a
+restart. Comparing wall times there would refuse every new run until the clock
+caught up, starting a run every Patrol cycle or exhausting the attempt limit.
+Investigation records and action outcomes keep comparing with
+`LastRegressionAt`, because they also arrive for sessions of earlier processes,
+stamped by the same clock. After a clock set back across a restart they can
+still refuse a newer run's results, as before.
+
+A run whose result does not count, but which the finding still shows as
+running, is closed at once through `RecoverStuckInvestigation`, the way the
+sweep would close it later: as `interrupted`, or completed with an outcome the
+action lifecycle recorded during the run, which is kept. Its attempt count
+stays as its admission wrote it. Through the adapter this is rare, because the
+start check keeps a run from an earlier occurrence off the finding: it takes a
+run with no recorded start, or findings reloaded from disk that still name an
+earlier run. A run that records its session after a regression, from a
+snapshot taken before it, is still installed with that snapshot's attempt
+count. When two runs are dispatched for one finding at once, the run whose
+start lands last is the finding's run and only its result is recorded, where
+before the run that closed last won. The `aicontracts` boundary is unchanged,
+and the orchestrator needs no change.
+
+Proofs:
+- `findings_test.go`: `StartInvestigation` skips a run from an earlier
+ occurrence or from before the finding was cleared or cleaned up, and installs
+ one with no recorded start or against a regression an earlier process
+ recorded; occurrence starts past the retention window are forgotten, and a
+ session older than that window is refused.
+ `CompleteInvestigation` refuses another session, an unnamed run and an
+ unknown finding; records the finding's own run, including a late result, a
+ run with no recorded start before any regression, and a regression an
+ earlier process recorded; and closes a run from before a regression the way
+ the sweep does, without projecting its verified fix and keeping a queued fix.
+ An earlier process's action outcome stays refused against `LastRegressionAt`.
+- `ai_handlers_investigation_additional_test.go`: through the adapter the
+ orchestrator receives, a run's result skips a later occurrence and a newer
+ run, a run started for an earlier occurrence is not installed and does not
+ replace the current run, and the current run's result lands.
+
+This changes which writes land, not what they record. It widens no mutation,
+approval or execution boundary.
diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md
index 1567cc8f6f..8effc9c85b 100644
--- a/docs/release-control/v6/internal/subsystems/api-contracts.md
+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md
@@ -12526,3 +12526,21 @@ payloads gain one optional field. An older orchestrator ignores the conclusion
and keeps reading prose markers, and records no deadline, so core measures its
runs from their start. Proofs: `patrol_scope_tools_test.go`, `names_test.go`,
`tools_conclude_test.go` and `contracts_test.go`.
+
+### Orchestrator finding writes check run ownership (23 October 2026)
+
+`orchestratorFindingsAdapter.Update`, the `aicontracts.OrchestratorFindingsStore`
+the private orchestrator receives, now returns false for a write that the
+finding's current occurrence does not own, and the newer state stays. A write
+that closes a run is refused when another session is current, or when the run
+cannot be shown to have started after the finding's current occurrence began:
+a regression this process observed, or the finding's removal by `ClearAll` or
+`Cleanup`, after which the same ID detected again is a new finding. If the
+finding still shows such a run as running, the run is closed the way the
+stuck-run sweep closes it, not with its own result. A write that starts a run
+is refused when the run's session started before that occurrence began, or
+more than a day ago, and applies as before otherwise. A regression an earlier
+process recorded refuses neither. The adapter takes each run's start from the
+orchestrator's session store, so the Pulse–enterprise boundary is unchanged: no
+`aicontracts` type, field or method is added, and no route or payload changes.
+Proof: `ai_handlers_investigation_additional_test.go`.
diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md
index 1b49b14167..7c6853b53c 100644
--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md
+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md
@@ -7174,3 +7174,8 @@ private orchestrator, make investigation concurrency and timeout follow the
Patrol provider, and label the `unclassified` and `interrupted` outcomes in the
digest email. Recovery evidence, restore rules and storage capabilities are
unchanged, and Patrol gains no storage remediation capability.
+
+The `internal/api` change that makes the investigation orchestrator's finding
+writes check run ownership opens no storage, backup or recovery path. It decides
+only whether a run's start or result is recorded on a Patrol finding. Recovery
+evidence, restore rules and storage capabilities are unchanged.
diff --git a/internal/ai/findings.go b/internal/ai/findings.go
index 4126defa10..c01b5c7ded 100644
--- a/internal/ai/findings.go
+++ b/internal/ai/findings.go
@@ -20,6 +20,9 @@ const (
investigationTimeoutCooldown = 10 * time.Minute // Shorter cooldown for timeout failures
investigationFixRetryCooldown = 20 * time.Minute // Retry faster after failed fix attempts
maxInvestigationAttempts = 3 // Maximum number of investigation retries
+ // How long occurrence starts are kept. StartInvestigation refuses a run
+ // whose session is older, so a forgotten start cannot let one in.
+ occurrenceStartRetention = 24 * time.Hour
)
// FindingSeverity represents how urgent a finding is
@@ -841,6 +844,9 @@ type FindingsStore struct {
activeCounts map[FindingSeverity]int
// User-defined suppression rules (separate from dismissed findings)
suppressionRules map[string]*SuppressionRule
+ // When this process last saw each finding begin a new occurrence, with the
+ // clock's monotonic reading. Never persisted: see liveRunOccurrenceLocked.
+ occurrenceStarts map[string]time.Time
// Persistence layer (optional)
persistence FindingsPersistence
// Debounce save operations
@@ -965,6 +971,7 @@ func NewFindingsStore() *FindingsStore {
byResource: make(map[string][]string),
activeCounts: make(map[FindingSeverity]int),
suppressionRules: make(map[string]*SuppressionRule),
+ occurrenceStarts: make(map[string]time.Time),
saveDebounce: 5 * time.Second, // Debounce saves by 5 seconds
}
}
@@ -2097,6 +2104,7 @@ func (s *FindingsStore) Add(f *Finding) bool {
existing.RegressionCount++
now := time.Now()
existing.LastRegressionAt = &now
+ s.occurrenceStarts[existing.ID] = now
meta := map[string]string{}
if prevResolvedAt != nil {
meta["previous_resolved_at"] = prevResolvedAt.Format(time.RFC3339)
@@ -2636,6 +2644,19 @@ func (f *Finding) acceptsInvestigationResult(sessionID string, startedAt time.Ti
return f.LastRegressionAt == nil || (!startedAt.IsZero() && !startedAt.Before(*f.LastRegressionAt))
}
+// liveRunOccurrenceLocked returns when the finding's current occurrence began,
+// as far as a run of this process can tell: the latest regression this process
+// saw, or the finding's removal, after which the same ID detected again is a new
+// finding. The monotonic clock orders it exactly against the run's start. A
+// regression an earlier process recorded predates every run of this one, so it
+// orders none of them, whatever the clock now reads. A result for a session of
+// an earlier process, such as an action outcome, is still judged against
+// LastRegressionAt.
+func (s *FindingsStore) liveRunOccurrenceLocked(id string) (time.Time, bool) {
+ startedAt, ok := s.occurrenceStarts[id]
+ return startedAt, ok
+}
+
func (s *FindingsStore) updateInvestigationOutcomeLocked(f *Finding, outcome string) {
prevOutcome := f.InvestigationOutcome
f.InvestigationOutcome = outcome
@@ -2665,6 +2686,73 @@ func (s *FindingsStore) UpdateInvestigation(id, sessionID, status, outcome strin
return false
}
+ s.updateInvestigationLocked(f, sessionID, status, outcome, lastInvestigatedAt, attempts)
+ s.mu.Unlock()
+ s.scheduleSave()
+ return true
+}
+
+// StartInvestigation installs a run that is starting on a finding. A run whose
+// session started before the finding's current occurrence began, as
+// liveRunOccurrenceLocked tells it, was started for an earlier occurrence, so
+// it is not installed: it would replace the current occurrence's investigation
+// state, or a newer run's. Neither is a run whose session started longer ago
+// than the store keeps occurrence starts, since it could predate one already
+// forgotten. runStartedAt is the start of the run's session, or zero when the
+// orchestrator holds none; such a run is installed as before.
+func (s *FindingsStore) StartInvestigation(id, sessionID string, runStartedAt time.Time, status, outcome string, lastInvestigatedAt *time.Time, attempts int) bool {
+ s.mu.Lock()
+ f, exists := s.findings[id]
+ if !exists {
+ s.mu.Unlock()
+ return false
+ }
+ occurrenceAt, ok := s.liveRunOccurrenceLocked(id)
+ if !runStartedAt.IsZero() && ((ok && runStartedAt.Before(occurrenceAt)) || time.Since(runStartedAt) > occurrenceStartRetention) {
+ s.mu.Unlock()
+ return false
+ }
+ s.updateInvestigationLocked(f, sessionID, status, outcome, lastInvestigatedAt, attempts)
+ s.mu.Unlock()
+ s.scheduleSave()
+ return true
+}
+
+// CompleteInvestigation records how a run of this process ended: its status,
+// outcome, close time and attempt count. It records them only while the finding
+// still names that run, so a run that ends after its finding moved on cannot
+// overwrite the newer state: the run's session must be the finding's current
+// one, and the run must not have started before the finding's current
+// occurrence began, as liveRunOccurrenceLocked tells it. runStartedAt is the
+// start of the run's session, or zero when the orchestrator holds none, which
+// cannot prove ownership once this process has seen an occurrence begin. A run
+// whose result does not count, but which the finding still shows as running,
+// is closed now through RecoverStuckInvestigation, as the stuck-run sweep would
+// later close it, rather than left running. It reports whether the run's
+// result was recorded.
+func (s *FindingsStore) CompleteInvestigation(id, sessionID string, runStartedAt time.Time, status, outcome string, lastInvestigatedAt *time.Time, attempts int) bool {
+ s.mu.Lock()
+ f, exists := s.findings[id]
+ if !exists || sessionID == "" || f.InvestigationSessionID != sessionID {
+ s.mu.Unlock()
+ return false
+ }
+ if occurrenceAt, ok := s.liveRunOccurrenceLocked(id); ok && (runStartedAt.IsZero() || runStartedAt.Before(occurrenceAt)) {
+ s.mu.Unlock()
+ closedAt := time.Now()
+ if lastInvestigatedAt != nil {
+ closedAt = *lastInvestigatedAt
+ }
+ s.RecoverStuckInvestigation(id, sessionID, runStartedAt, InvestigationStatusFailed, InvestigationOutcomeInterrupted, closedAt)
+ return false
+ }
+ s.updateInvestigationLocked(f, sessionID, status, outcome, lastInvestigatedAt, attempts)
+ s.mu.Unlock()
+ s.scheduleSave()
+ return true
+}
+
+func (s *FindingsStore) updateInvestigationLocked(f *Finding, sessionID, status, outcome string, lastInvestigatedAt *time.Time, attempts int) {
prevStatus := f.InvestigationStatus
prevOutcome := f.InvestigationOutcome
f.InvestigationSessionID = sessionID
@@ -2683,9 +2771,6 @@ func (s *FindingsStore) UpdateInvestigation(id, sessionID, status, outcome strin
})
}
s.syncLoopStateLocked(f)
- s.mu.Unlock()
- s.scheduleSave()
- return true
}
// RecoverStuckInvestigation closes a run that is no longer live with the given
@@ -2911,6 +2996,10 @@ func (s *FindingsStore) GetAll(startTime *time.Time) []*Finding {
func (s *FindingsStore) ClearAll() int {
s.mu.Lock()
count := len(s.findings)
+ now := time.Now()
+ for id := range s.findings {
+ s.occurrenceStarts[id] = now
+ }
s.findings = make(map[string]*Finding)
s.byResource = make(map[string][]string)
s.activeCounts = make(map[FindingSeverity]int)
@@ -2944,6 +3033,7 @@ func (s *FindingsStore) Cleanup(maxAge time.Duration) int {
if shouldRemove {
delete(s.findings, id)
+ s.occurrenceStarts[id] = now
// Clean up resource index
ids := s.byResource[f.ResourceID]
for i, fid := range ids {
@@ -2955,6 +3045,11 @@ func (s *FindingsStore) Cleanup(maxAge time.Duration) int {
removed++
}
}
+ for id, startedAt := range s.occurrenceStarts {
+ if startedAt.Before(now.Add(-occurrenceStartRetention)) {
+ delete(s.occurrenceStarts, id)
+ }
+ }
return removed
}
diff --git a/internal/ai/findings_test.go b/internal/ai/findings_test.go
index 0210625557..ad8b397150 100644
--- a/internal/ai/findings_test.go
+++ b/internal/ai/findings_test.go
@@ -2483,3 +2483,259 @@ func TestInterruptedAndUnclassifiedInvestigationsStayEligible(t *testing.T) {
t.Fatal("a real needs_attention conclusion stays terminal")
}
}
+
+// regressHere resolves a finding and detects it again, a regression this
+// process observes, and returns when it happened.
+func regressHere(t *testing.T, findings *FindingsStore, id string) time.Time {
+ t.Helper()
+ f := findings.Get(id)
+ if f == nil || !findings.Resolve(id, true) {
+ t.Fatalf("could not resolve %s", id)
+ }
+ findings.Add(&Finding{ID: id, ResourceID: f.ResourceID, ResourceType: f.ResourceType, Title: f.Title, Severity: f.Severity, Category: f.Category})
+ got := findings.Get(id)
+ if got.LastRegressionAt == nil {
+ t.Fatalf("%s did not regress", id)
+ }
+ return *got.LastRegressionAt
+}
+
+func TestStartInvestigationSkipsARunFromAnEarlierOccurrence(t *testing.T) {
+ findings := NewFindingsStore()
+ for _, id := range []string{"regressed", "unrecorded", "first", "recorded-earlier"} {
+ f := runningInvestigationFinding(id, "", nil)
+ f.InvestigationStatus = ""
+ f.InvestigationAttempts = 0
+ if id == "recorded-earlier" {
+ // Loaded after a restart, with the clock since set back.
+ recordedAt := time.Now().Add(time.Hour)
+ f.LastRegressionAt = &recordedAt
+ }
+ findings.Add(f)
+ }
+ regressedAt := regressHere(t, findings, "regressed")
+ regressHere(t, findings, "unrecorded")
+ start := func(id, sessionID string, runStartedAt time.Time) bool {
+ return findings.StartInvestigation(id, sessionID, runStartedAt, string(InvestigationStatusRunning), "", nil, 3)
+ }
+
+ if start("regressed", "chat-earlier", regressedAt.Add(-time.Minute)) {
+ t.Fatal("a run started for the earlier occurrence was installed")
+ }
+ if got := findings.Get("regressed"); got.InvestigationSessionID != "" || got.InvestigationStatus != "" || got.InvestigationAttempts != 0 {
+ t.Fatalf("regressed = session %q %s, %d attempts; want the current occurrence untouched", got.InvestigationSessionID, got.InvestigationStatus, got.InvestigationAttempts)
+ }
+ if !start("regressed", "chat-current", time.Now()) {
+ t.Fatal("a run of the current occurrence was refused")
+ }
+ assertInvestigationState(t, findings, "regressed", InvestigationStatusRunning, "")
+ // A run with no recorded start, or on a finding that never regressed, is
+ // installed as before.
+ if !start("unrecorded", "chat-unrecorded", time.Time{}) {
+ t.Fatal("a run with no recorded start was refused")
+ }
+ if !start("first", "chat-first", time.Now().Add(-time.Hour)) {
+ t.Fatal("a run on a finding that never regressed was refused")
+ }
+ // A regression an earlier process recorded predates every run of this one,
+ // however the clock now reads; refusing would start a run every cycle.
+ if !start("recorded-earlier", "chat-recorded-earlier", time.Now()) {
+ t.Fatal("a run was refused against a regression an earlier process recorded")
+ }
+ if start("missing", "chat-missing", time.Now()) {
+ t.Fatal("a run was installed on an unknown finding")
+ }
+}
+
+func TestCompleteInvestigationRecordsOnlyTheRunTheFindingNames(t *testing.T) {
+ started := time.Now().Add(-time.Minute)
+ findings := NewFindingsStore()
+ findings.Add(runningInvestigationFinding("current", "chat-current", nil))
+ findings.Add(runningInvestigationFinding("unrecorded", "chat-unrecorded", nil))
+ findings.Add(runningInvestigationFinding("replaced", "chat-newer", nil))
+ idle := runningInvestigationFinding("idle", "", nil)
+ idle.InvestigationStatus = ""
+ findings.Add(idle)
+ // The stuck-run sweep gave this run up as timed out; it reports back after all.
+ late := runningInvestigationFinding("late", "chat-late", nil)
+ late.InvestigationStatus = string(InvestigationStatusFailed)
+ late.InvestigationOutcome = string(InvestigationOutcomeTimedOut)
+ findings.Add(late)
+ // Loaded after a restart, with the clock since set back.
+ recordedEarlier := runningInvestigationFinding("recorded-earlier", "chat-recorded-earlier", nil)
+ recordedAt := time.Now().Add(time.Hour)
+ recordedEarlier.LastRegressionAt = &recordedAt
+ findings.Add(recordedEarlier)
+
+ closedAt := time.Now()
+ complete := func(id, sessionID string, runStartedAt time.Time, status InvestigationStatus, outcome InvestigationOutcome, attempts int) bool {
+ return findings.CompleteInvestigation(id, sessionID, runStartedAt, string(status), string(outcome), &closedAt, attempts)
+ }
+ if complete("replaced", "chat-older", started, InvestigationStatusCompleted, InvestigationOutcomeResolved, 1) {
+ t.Fatal("an older run's result overwrote the run that replaced it")
+ }
+ assertInvestigationState(t, findings, "replaced", InvestigationStatusRunning, "")
+ if complete("idle", "chat-other", started, InvestigationStatusCompleted, InvestigationOutcomeResolved, 1) {
+ t.Fatal("a run the finding does not name recorded its result")
+ }
+ assertInvestigationState(t, findings, "idle", "", "")
+ if complete("missing", "chat-missing", started, InvestigationStatusCompleted, InvestigationOutcomeResolved, 1) {
+ t.Fatal("a result was recorded for an unknown finding")
+ }
+
+ if !complete("current", "chat-current", started, InvestigationStatusCompleted, InvestigationOutcomeCannotFix, 2) {
+ t.Fatal("the finding's own run was refused")
+ }
+ assertInvestigationState(t, findings, "current", InvestigationStatusCompleted, InvestigationOutcomeCannotFix)
+ if got := findings.Get("current"); got.InvestigationAttempts != 2 || got.LastInvestigatedAt == nil || !got.LastInvestigatedAt.Equal(closedAt) {
+ t.Fatalf("current = %d attempts, last investigated %v; want 2 at %v", got.InvestigationAttempts, got.LastInvestigatedAt, closedAt)
+ }
+ // With no regression on record, a run with no recorded start still owns
+ // the only occurrence there has been.
+ if !complete("unrecorded", "chat-unrecorded", time.Time{}, InvestigationStatusFailed, InvestigationOutcomeTimedOut, 1) {
+ t.Fatal("a run with no recorded start was refused before any regression")
+ }
+ assertInvestigationState(t, findings, "unrecorded", InvestigationStatusFailed, InvestigationOutcomeTimedOut)
+ if !complete("late", "chat-late", started, InvestigationStatusCompleted, InvestigationOutcomeFixVerified, 1) {
+ t.Fatal("a late result from the finding's own run was refused")
+ }
+ assertInvestigationState(t, findings, "late", InvestigationStatusCompleted, InvestigationOutcomeFixVerified)
+ if findings.Get("late").ResolvedAt == nil {
+ t.Fatal("a verified fix from the finding's own run did not resolve it")
+ }
+ // A regression an earlier process recorded predates every run of this one.
+ if !complete("recorded-earlier", "chat-recorded-earlier", started, InvestigationStatusCompleted, InvestigationOutcomeNeedsAttention, 1) {
+ t.Fatal("a run's result was refused against a regression an earlier process recorded")
+ }
+ assertInvestigationState(t, findings, "recorded-earlier", InvestigationStatusCompleted, InvestigationOutcomeNeedsAttention)
+ // An action outcome for a session of an earlier process is still judged
+ // against LastRegressionAt, which the same clock stamped.
+ earlier := &aicontracts.InvestigationSession{ID: "inv-earlier", FindingID: "recorded-earlier", SessionID: "chat-earlier", StartedAt: recordedAt.Add(-time.Minute)}
+ findings.UpdateInvestigation("recorded-earlier", "", "", "", nil, 0) // as the regression left it
+ if applied, _ := findings.ReconcileInvestigationOutcome("recorded-earlier", earlier, string(InvestigationOutcomeFixVerified)); applied {
+ t.Fatal("an earlier process's action outcome landed on the current occurrence")
+ }
+}
+
+func TestCompleteInvestigationClosesARunFromAnEarlierOccurrenceAsTheSweepWould(t *testing.T) {
+ ids := []string{"stale", "unrecorded", "settled", "queued", "current"}
+ findings := NewFindingsStore()
+ for _, id := range ids {
+ findings.Add(runningInvestigationFinding(id, "", nil))
+ }
+ var regressedAt time.Time
+ for _, id := range ids {
+ regressedAt = regressHere(t, findings, id)
+ // The run is installed on the current occurrence, as through a path
+ // that does not check when it started.
+ status, outcome := InvestigationStatusRunning, InvestigationOutcome("")
+ switch id {
+ case "settled":
+ // The stuck-run sweep already closed it.
+ status, outcome = InvestigationStatusFailed, InvestigationOutcomeInterrupted
+ case "queued":
+ // The action lifecycle recorded a queued fix during the run.
+ outcome = InvestigationOutcomeFixQueued
+ }
+ findings.UpdateInvestigation(id, "chat-"+id, string(status), string(outcome), nil, 1)
+ }
+ beforeRegression := regressedAt.Add(-time.Minute)
+
+ closedAt := time.Now()
+ complete := func(id string, runStartedAt time.Time) bool {
+ return findings.CompleteInvestigation(id, "chat-"+id, runStartedAt, string(InvestigationStatusCompleted), string(InvestigationOutcomeFixVerified), &closedAt, 2)
+ }
+ // The run started before the finding's current occurrence: its verified
+ // fix was for the earlier one.
+ if complete("stale", beforeRegression) {
+ t.Fatal("a run from before the regression recorded its result")
+ }
+ assertInvestigationState(t, findings, "stale", InvestigationStatusFailed, InvestigationOutcomeInterrupted)
+ stale := findings.Get("stale")
+ if stale.ResolvedAt != nil {
+ t.Fatal("an earlier occurrence's verified fix resolved the current one")
+ }
+ if stale.LastInvestigatedAt == nil || !stale.LastInvestigatedAt.Equal(closedAt) || stale.InvestigationAttempts != 1 {
+ t.Fatalf("stale = %d attempts, last investigated %v; want 1 at the close %v", stale.InvestigationAttempts, stale.LastInvestigatedAt, closedAt)
+ }
+ if !stale.ShouldInvestigate("approval") {
+ t.Fatal("the current occurrence must be free for another run")
+ }
+ // Without a recorded start the run cannot be shown to belong to the
+ // current occurrence.
+ if complete("unrecorded", time.Time{}) {
+ t.Fatal("a run with no recorded start recorded its result after a regression")
+ }
+ assertInvestigationState(t, findings, "unrecorded", InvestigationStatusFailed, InvestigationOutcomeInterrupted)
+ if complete("settled", beforeRegression) {
+ t.Fatal("a run from before the regression recorded its result")
+ }
+ assertInvestigationState(t, findings, "settled", InvestigationStatusFailed, InvestigationOutcomeInterrupted)
+ // The close keeps an outcome the action lifecycle recorded, as the sweep
+ // does, so the queued fix still blocks another investigation.
+ if complete("queued", beforeRegression) {
+ t.Fatal("a run from before the regression recorded its result")
+ }
+ assertInvestigationState(t, findings, "queued", InvestigationStatusCompleted, InvestigationOutcomeFixQueued)
+ if findings.Get("queued").ShouldInvestigate("approval") {
+ t.Fatal("a queued fix must keep blocking another investigation")
+ }
+
+ if !complete("current", time.Now()) {
+ t.Fatal("a run of the current occurrence was refused")
+ }
+ assertInvestigationState(t, findings, "current", InvestigationStatusCompleted, InvestigationOutcomeFixVerified)
+ if current := findings.Get("current"); current.ResolvedAt == nil || current.InvestigationAttempts != 2 {
+ t.Fatalf("current = resolved at %v, %d attempts; want resolved with the run's 2 attempts", current.ResolvedAt, current.InvestigationAttempts)
+ }
+}
+
+func TestStartInvestigationSkipsARunFromBeforeTheFindingWasRemoved(t *testing.T) {
+ start := func(findings *FindingsStore, id, sessionID string, runStartedAt time.Time) bool {
+ return findings.StartInvestigation(id, sessionID, runStartedAt, string(InvestigationStatusRunning), "", nil, 1)
+ }
+
+ // Cleared by the operator, then detected again: the same ID, a new finding.
+ cleared := NewFindingsStore()
+ cleared.Add(runningInvestigationFinding("cleared", "", nil))
+ runStartedAt := time.Now()
+ cleared.ClearAll()
+ cleared.Add(runningInvestigationFinding("cleared", "", nil))
+ if start(cleared, "cleared", "chat-before-clear", runStartedAt) {
+ t.Fatal("a run from before the finding was cleared was installed on the new one")
+ }
+ if !start(cleared, "cleared", "chat-after-clear", time.Now()) {
+ t.Fatal("a run of the new finding was refused")
+ }
+
+ // Removed as an old resolved finding, then detected again.
+ cleaned := NewFindingsStore()
+ cleaned.Add(runningInvestigationFinding("cleaned", "", nil))
+ runStartedAt = time.Now()
+ cleaned.Resolve("cleaned", true)
+ if removed := cleaned.Cleanup(0); removed != 1 {
+ t.Fatalf("Cleanup removed %d findings, want 1", removed)
+ }
+ cleaned.Add(runningInvestigationFinding("cleaned", "", nil))
+ if start(cleaned, "cleaned", "chat-before-cleanup", runStartedAt) {
+ t.Fatal("a run from before the finding was removed was installed on the new one")
+ }
+
+ // An occurrence start no live run can predate is forgotten.
+ cleaned.occurrenceStarts["long-gone"] = time.Now().Add(-occurrenceStartRetention - time.Minute)
+ cleaned.Cleanup(0)
+ if _, ok := cleaned.occurrenceStarts["long-gone"]; ok {
+ t.Fatal("an occurrence start past the retention window was kept")
+ }
+
+ // So a session older than that could predate an occurrence start the store
+ // no longer holds, even where none is on record.
+ fresh := NewFindingsStore()
+ fresh.Add(runningInvestigationFinding("fresh", "", nil))
+ if start(fresh, "fresh", "chat-ancient", time.Now().Add(-occurrenceStartRetention-time.Minute)) {
+ t.Fatal("a run from a session older than the retention window was installed")
+ }
+ if !start(fresh, "fresh", "chat-recent", time.Now().Add(-time.Hour)) {
+ t.Fatal("a run from a recent session was refused")
+ }
+}
diff --git a/internal/api/ai_handlers.go b/internal/api/ai_handlers.go
index 02713e706b..07abae14b8 100644
--- a/internal/api/ai_handlers.go
+++ b/internal/api/ai_handlers.go
@@ -1932,7 +1932,7 @@ func (h *AISettingsHandler) setupInvestigationOrchestrator(orgID string, svc *ai
log.Warn().Str("orgID", orgID).Msg("Findings store not available for orchestrator")
return
}
- findingsAdapter := &orchestratorFindingsAdapter{store: &findingsStoreWrapper{store: findingsStore}}
+ findingsAdapter := &orchestratorFindingsAdapter{store: &findingsStoreWrapper{store: findingsStore}, sessions: store}
// Get config for investigation settings
cfg := svc.GetConfig()
@@ -2111,6 +2111,9 @@ func (a *orchestratorChatAdapter) ListInvestigationTools(ctx context.Context) []
// aicontracts.OrchestratorFindingsStore.
type orchestratorFindingsAdapter struct {
store *findingsStoreWrapper
+ // sessions is the orchestrator's session store, which records when each
+ // run started.
+ sessions aicontracts.InvestigationStore
}
func (a *orchestratorFindingsAdapter) Get(id string) *aicontracts.Finding {
@@ -2140,13 +2143,32 @@ func (a *orchestratorFindingsAdapter) Get(id string) *aicontracts.Finding {
}
}
+// Update applies the orchestrator's investigation fields to a finding. Starting
+// a run installs it, unless the run was started for an earlier occurrence of
+// the finding. Every other write closes a run and lands only while the finding
+// still names that run, so a run that ends after its finding was re-detected,
+// or after a newer run took over, cannot overwrite the newer state.
func (a *orchestratorFindingsAdapter) Update(f *aicontracts.Finding) bool {
if a.store == nil || f == nil {
return false
}
- return a.store.UpdateInvestigation(
+ runStartedAt := a.runStartedAt(f.ID, f.InvestigationSessionID)
+ switch aicontracts.InvestigationStatus(f.InvestigationStatus) {
+ case aicontracts.InvestigationStatusPending, aicontracts.InvestigationStatusRunning:
+ return a.store.StartInvestigation(
+ f.ID,
+ f.InvestigationSessionID,
+ runStartedAt,
+ f.InvestigationStatus,
+ f.InvestigationOutcome,
+ f.LastInvestigatedAt,
+ f.InvestigationAttempts,
+ )
+ }
+ return a.store.CompleteInvestigation(
f.ID,
f.InvestigationSessionID,
+ runStartedAt,
f.InvestigationStatus,
f.InvestigationOutcome,
f.LastInvestigatedAt,
@@ -2154,6 +2176,20 @@ func (a *orchestratorFindingsAdapter) Update(f *aicontracts.Finding) bool {
)
}
+// runStartedAt returns when the orchestrator's session for a run started, or
+// zero when its session store holds no record of the run.
+func (a *orchestratorFindingsAdapter) runStartedAt(findingID, sessionID string) time.Time {
+ if a.sessions == nil {
+ return time.Time{}
+ }
+ for _, session := range a.sessions.GetByFinding(findingID) {
+ if session != nil && session.SessionID == sessionID {
+ return session.StartedAt
+ }
+ }
+ return time.Time{}
+}
+
// patrolMetricsCallbackAdapter implements aicontracts.OrchestratorMetricsCallback
// by delegating to the global PatrolMetrics singleton.
type patrolMetricsCallbackAdapter struct{}
@@ -2189,6 +2225,20 @@ func (w *findingsStoreWrapper) UpdateInvestigation(id, sessionID, status, outcom
return w.store.UpdateInvestigation(id, sessionID, status, outcome, lastInvestigatedAt, attempts)
}
+func (w *findingsStoreWrapper) StartInvestigation(id, sessionID string, runStartedAt time.Time, status, outcome string, lastInvestigatedAt *time.Time, attempts int) bool {
+ if w.store == nil {
+ return false
+ }
+ return w.store.StartInvestigation(id, sessionID, runStartedAt, status, outcome, lastInvestigatedAt, attempts)
+}
+
+func (w *findingsStoreWrapper) CompleteInvestigation(id, sessionID string, runStartedAt time.Time, status, outcome string, lastInvestigatedAt *time.Time, attempts int) bool {
+ if w.store == nil {
+ return false
+ }
+ return w.store.CompleteInvestigation(id, sessionID, runStartedAt, status, outcome, lastInvestigatedAt, attempts)
+}
+
// AISettingsResponse is returned by GET /api/settings/ai
// API keys are masked for security
type AISettingsResponse struct {
diff --git a/internal/api/ai_handlers_investigation_additional_test.go b/internal/api/ai_handlers_investigation_additional_test.go
index c51e310375..b29f0054eb 100644
--- a/internal/api/ai_handlers_investigation_additional_test.go
+++ b/internal/api/ai_handlers_investigation_additional_test.go
@@ -530,6 +530,12 @@ func TestFindingsStoreWrapper_GetAndUpdate(t *testing.T) {
if nilWrapper.UpdateInvestigation("finding-1", "session-1", "running", "outcome", nil, 1) {
t.Fatalf("expected nil store update to return false")
}
+ if nilWrapper.StartInvestigation("finding-1", "session-1", time.Time{}, "running", "", nil, 1) {
+ t.Fatalf("expected nil store start to return false")
+ }
+ if nilWrapper.CompleteInvestigation("finding-1", "session-1", time.Time{}, "completed", "outcome", nil, 1) {
+ t.Fatalf("expected nil store completion to return false")
+ }
}
func TestHandleClearAllFindings(t *testing.T) {
@@ -933,3 +939,195 @@ func TestSetupInvestigationOrchestratorUsesSubscriptionAgentLimits(t *testing.T)
t.Fatalf("Timeout = %s, want %s", created[0].Timeout, want)
}
}
+
+// orchestratorFindingsFor wires a tenant's investigation orchestrator and
+// returns the findings and session stores it was built with, plus the
+// tenant's findings.
+func orchestratorFindingsFor(t *testing.T) (aicontracts.OrchestratorFindingsStore, aicontracts.InvestigationStore, *ai.FindingsStore) {
+ t.Helper()
+ prevStoreFactory := getCreateInvestigationStore()
+ SetCreateInvestigationStore(func(dataDir string) aicontracts.InvestigationStore {
+ return newTestInvestigationStore()
+ })
+ t.Cleanup(func() { SetCreateInvestigationStore(prevStoreFactory) })
+
+ var deps aicontracts.OrchestratorDeps
+ prevOrchestratorFactory := getCreateInvestigationOrchestrator()
+ SetCreateInvestigationOrchestrator(func(d aicontracts.OrchestratorDeps) aicontracts.InvestigationOrchestrator {
+ deps = d
+ return &stubInvestigationOrchestrator{}
+ })
+ t.Cleanup(func() { SetCreateInvestigationOrchestrator(prevOrchestratorFactory) })
+
+ tenantSvc := ai.NewService(nil, nil)
+ tenantSvc.SetStateProvider(&MockStateProvider{})
+ handler := &AISettingsHandler{
+ investigationStores: make(map[string]aicontracts.InvestigationStore),
+ }
+ handler.chatHandler = &AIHandler{
+ services: map[string]AIService{
+ "tenant-1": chat.NewService(chat.Config{AIConfig: config.NewDefaultAIConfig()}),
+ },
+ }
+ handler.setupInvestigationOrchestrator("tenant-1", tenantSvc)
+ if deps.FindingsStore == nil || deps.Store == nil {
+ t.Fatal("investigation orchestrator was not wired")
+ }
+ return deps.FindingsStore, deps.Store, tenantSvc.GetPatrolService().GetFindings()
+}
+
+func investigatedFinding(id string) *ai.Finding {
+ return &ai.Finding{
+ ID: id,
+ Severity: ai.FindingSeverityWarning,
+ Category: ai.FindingCategoryPerformance,
+ ResourceID: "vm-" + id,
+ ResourceName: "vm-" + id,
+ ResourceType: "vm",
+ Title: "Memory pressure on vm-" + id,
+ Description: "Memory usage is above 90%",
+ }
+}
+
+// startOrchestratorRun does what the orchestrator does when it starts a run:
+// it records the run's session, then marks the finding as running under it.
+func startOrchestratorRun(t *testing.T, findings aicontracts.OrchestratorFindingsStore, sessions aicontracts.InvestigationStore, findingID, chatSessionID string) aicontracts.Finding {
+ t.Helper()
+ run := findings.Get(findingID)
+ if run == nil {
+ t.Fatalf("finding %s not found", findingID)
+ }
+ sessions.Create(findingID, chatSessionID)
+ run.InvestigationSessionID = chatSessionID
+ run.InvestigationStatus = string(aicontracts.InvestigationStatusRunning)
+ run.InvestigationOutcome = ""
+ run.InvestigationAttempts++
+ if !findings.Update(run) {
+ t.Fatalf("starting run %s was not recorded", chatSessionID)
+ }
+ return *run
+}
+
+// finishOrchestratorRun is the orchestrator's finding write when a run ends.
+func finishOrchestratorRun(findings aicontracts.OrchestratorFindingsStore, run aicontracts.Finding, status aicontracts.InvestigationStatus, outcome aicontracts.InvestigationOutcome) bool {
+ now := time.Now()
+ run.InvestigationStatus = string(status)
+ run.InvestigationOutcome = string(outcome)
+ run.LastInvestigatedAt = &now
+ return findings.Update(&run)
+}
+
+func assertOrchestratorRunState(t *testing.T, findings *ai.FindingsStore, id, sessionID string, status aicontracts.InvestigationStatus, outcome aicontracts.InvestigationOutcome, attempts int) {
+ t.Helper()
+ got := findings.Get(id)
+ if got == nil {
+ t.Fatalf("finding %s not found", id)
+ }
+ if got.InvestigationSessionID != sessionID || got.InvestigationStatus != string(status) ||
+ got.InvestigationOutcome != string(outcome) || got.InvestigationAttempts != attempts {
+ t.Fatalf("%s investigation = session %q %s/%s, %d attempts; want session %q %s/%s, %d attempts",
+ id, got.InvestigationSessionID, got.InvestigationStatus, got.InvestigationOutcome, got.InvestigationAttempts,
+ sessionID, status, outcome, attempts)
+ }
+}
+
+func TestOrchestratorRunResultSkipsALaterOccurrence(t *testing.T) {
+ orchestratorFindings, sessions, findings := orchestratorFindingsFor(t)
+ findings.Add(investigatedFinding("finding-1"))
+ runA := startOrchestratorRun(t, orchestratorFindings, sessions, "finding-1", "chat-a")
+
+ // While A is in flight the finding resolves and is detected again.
+ findings.Resolve("finding-1", true)
+ findings.Add(investigatedFinding("finding-1"))
+
+ // A concludes for the occurrence it investigated, not the current one.
+ applied := finishOrchestratorRun(orchestratorFindings, runA, aicontracts.InvestigationStatusCompleted, aicontracts.OutcomeResolved)
+ assertOrchestratorRunState(t, findings, "finding-1", "", "", "", 0)
+ if applied {
+ t.Fatal("a run of the earlier occurrence reported closing the current one")
+ }
+ if got := findings.Get("finding-1"); got.LastInvestigatedAt != nil || !got.ShouldInvestigate("approval") {
+ t.Fatalf("the earlier occurrence's result kept the current one from being investigated: %+v", got)
+ }
+
+ // Nor does it land once a run of the current occurrence has started.
+ runB := startOrchestratorRun(t, orchestratorFindings, sessions, "finding-1", "chat-b")
+ if finishOrchestratorRun(orchestratorFindings, runA, aicontracts.InvestigationStatusFailed, aicontracts.OutcomeTimedOut) {
+ t.Fatal("a run of the earlier occurrence overwrote the current run")
+ }
+ assertOrchestratorRunState(t, findings, "finding-1", "chat-b", aicontracts.InvestigationStatusRunning, "", 1)
+
+ if !finishOrchestratorRun(orchestratorFindings, runB, aicontracts.InvestigationStatusCompleted, aicontracts.OutcomeNeedsAttention) {
+ t.Fatal("the current run's result was refused")
+ }
+ assertOrchestratorRunState(t, findings, "finding-1", "chat-b", aicontracts.InvestigationStatusCompleted, aicontracts.OutcomeNeedsAttention, 1)
+}
+
+func TestOrchestratorRunFromAnEarlierOccurrenceIsNotInstalled(t *testing.T) {
+ orchestratorFindings, sessions, findings := orchestratorFindingsFor(t)
+ findings.Add(investigatedFinding("finding-1"))
+
+ // A records its session for the first occurrence, with that occurrence's
+ // attempt count; the finding resolves and is detected again before A marks
+ // it as running.
+ runA := orchestratorFindings.Get("finding-1")
+ sessions.Create("finding-1", "chat-a")
+ findings.Resolve("finding-1", true)
+ findings.Add(investigatedFinding("finding-1"))
+ runA.InvestigationSessionID = "chat-a"
+ runA.InvestigationStatus = string(aicontracts.InvestigationStatusRunning)
+ runA.InvestigationAttempts = 3
+ if orchestratorFindings.Update(runA) {
+ t.Fatal("a run started for the earlier occurrence was installed on the current one")
+ }
+ assertOrchestratorRunState(t, findings, "finding-1", "", "", "", 0)
+
+ // Nor does A's admission replace a run of the current occurrence.
+ runB := startOrchestratorRun(t, orchestratorFindings, sessions, "finding-1", "chat-b")
+ if orchestratorFindings.Update(runA) {
+ t.Fatal("a run started for the earlier occurrence replaced the current run")
+ }
+ assertOrchestratorRunState(t, findings, "finding-1", "chat-b", aicontracts.InvestigationStatusRunning, "", 1)
+ if finishOrchestratorRun(orchestratorFindings, *runA, aicontracts.InvestigationStatusCompleted, aicontracts.OutcomeFixVerified) {
+ t.Fatal("a run of the earlier occurrence recorded its result on the current one")
+ }
+
+ if !finishOrchestratorRun(orchestratorFindings, runB, aicontracts.InvestigationStatusCompleted, aicontracts.OutcomeCannotFix) {
+ t.Fatal("the current run's result was refused")
+ }
+ assertOrchestratorRunState(t, findings, "finding-1", "chat-b", aicontracts.InvestigationStatusCompleted, aicontracts.OutcomeCannotFix, 1)
+ if findings.Get("finding-1").ResolvedAt != nil {
+ t.Fatal("the earlier occurrence's verified fix resolved the current one")
+ }
+}
+
+func TestOrchestratorRunResultSkipsANewerRun(t *testing.T) {
+ orchestratorFindings, sessions, findings := orchestratorFindingsFor(t)
+ findings.Add(investigatedFinding("retried"))
+ findings.Add(investigatedFinding("late"))
+ retriedA := startOrchestratorRun(t, orchestratorFindings, sessions, "retried", "chat-retried-a")
+ lateA := startOrchestratorRun(t, orchestratorFindings, sessions, "late", "chat-late-a")
+
+ // The stuck-run sweep gave both runs up as timed out, and the retry
+ // started another run for one of them.
+ now := time.Now()
+ findings.RecoverStuckInvestigation("retried", "chat-retried-a", time.Time{}, ai.InvestigationStatusFailed, ai.InvestigationOutcomeTimedOut, now)
+ findings.RecoverStuckInvestigation("late", "chat-late-a", time.Time{}, ai.InvestigationStatusFailed, ai.InvestigationOutcomeTimedOut, now)
+ retriedB := startOrchestratorRun(t, orchestratorFindings, sessions, "retried", "chat-retried-b")
+
+ // The first runs report back after all.
+ if finishOrchestratorRun(orchestratorFindings, retriedA, aicontracts.InvestigationStatusCompleted, aicontracts.OutcomeResolved) {
+ t.Fatal("an earlier run overwrote the run that replaced it")
+ }
+ assertOrchestratorRunState(t, findings, "retried", "chat-retried-b", aicontracts.InvestigationStatusRunning, "", 2)
+ // With no newer run, the run the finding still names has the real result.
+ if !finishOrchestratorRun(orchestratorFindings, lateA, aicontracts.InvestigationStatusCompleted, aicontracts.OutcomeCannotFix) {
+ t.Fatal("a late result from the finding's own run was refused")
+ }
+ assertOrchestratorRunState(t, findings, "late", "chat-late-a", aicontracts.InvestigationStatusCompleted, aicontracts.OutcomeCannotFix, 1)
+
+ if !finishOrchestratorRun(orchestratorFindings, retriedB, aicontracts.InvestigationStatusFailed, aicontracts.OutcomeTimedOut) {
+ t.Fatal("the newer run's result was refused")
+ }
+ assertOrchestratorRunState(t, findings, "retried", "chat-retried-b", aicontracts.InvestigationStatusFailed, aicontracts.OutcomeTimedOut, 2)
+}