Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Finalize Patrol runs when they close, failed runs included
rcourtman45b5a8cde821
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 7a591c9248..fabcbb890c 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -10204,3 +10204,9 @@ The `internal/api` change of 2026-10-23 removes browser-only chat routes`/redo`, `/steer`, `/diff`, `/revert`, and `/unrevert` session sub-resources). The adjacent-boundary notes above that name those routes are nowhistory; none of them was agent-lifecycle state. No agent enrollment,command, update, identity, or credential path changes.++The `internal/api` change of 2026-10-23 that finalizes Patrol investigation+runs when they close, failed runs included, and publishes the action plans the+inbox expiry sweep expires, enrols, commands, updates or removes no agent and+touches no agent identity or credential. Investigations still reach hosts only+through the read-only evidence tools and the governed action lifecycle.diff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.mdindex 67a49aab94..8dacbc3be6 100644--- a/docs/release-control/v6/internal/subsystems/ai-runtime.md+++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md@@ -7590,3 +7590,48 @@ affordances for an undeclared Assistant surface in`frontend-modern/src/api/agentCapabilities.ts` no longer include interactivequestions, matching `DefaultSurfaceAffordancesForID`. Surface IDs, labels,tools, and approval boundaries are unchanged.++### Investigation runs are finalized when they close (23 October 2026)++A Patrol investigation run is finalized when it closes, however it ended and+whoever started it: when the private orchestrator's write that closes the run+lands on the finding through `orchestratorFindingsAdapter.Update`, or when the+stuck-run sweep recovers a run whose session has already closed, such as one a+restart interrupted. Core first reconciles the run's action from the action+audit (`hydratePatrolInvestigationAction`), then calls+`PatrolService.FinalizeInvestigationRun`. That rebuilds the durable+investigation record from the run's final session, publishes the finding to the+unified store, resolving it there when a verified fix resolved it, and asks the+operator to approve a planned fix that still awaits approval, once per action.+It finalizes only a run the finding still names, checked again after the record+is written. An action's outcome, such as a verified fix, is pushed once, when+its transition is reconciled (`PublishFindingLifecycleUpdate`); a close no+longer sends a second fix-completed push.++Before this, only an automatic investigation that returned without an error+was finalized, by the goroutine that started it, which now only logs a run's+error. A run that planned a fix and then failed, normally on its final model+call, kept the record core projected when the plan was recorded, which showed+the run as still running, and the operator was never asked to approve the+queued fix. A manual re-investigation was never finalized, even when it+succeeded. The action is reconciled first because the orchestrator writes it+back as it was planned, which an approval, rejection or expiry during the final+model call may have overtaken. For the same reason `CompleteInvestigation` no+longer moves a finding back to `fix_queued` from an outcome the plan's action+reached during the run: a run starts with no outcome, so one present at its+close came from the action lifecycle. Proofs:+`internal/ai/patrol_investigation_finalize_test.go` and+`internal/api/ai_handlers_investigation_finalize_test.go`.++The action inbox and decision queue expire every plan past its expiry, and now+publish each one through the lifecycle's transition callback, as a single+action's expiry already was. `ReconcilePatrolActionTransition` therefore moves a+Patrol investigation and its finding from `fix_queued` to `needs_attention` when+the plan expires in a sweep, rather than when someone next opens the+investigation. Reconciliation ignores a transition for a plan that the+investigation's current action replaced through one or more refreshes, which it+follows through `actionlifecycle.RefreshedActionID` rather than the clock: a+refresh plans a replacement under the same investigation and leaves the+replaced plan to expire on its own, and that expiry must not take the+investigation back from its replacement. Proof:+`internal/api/patrol_action_expiry_sweep_test.go`.diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.mdindex 2c2a21facc..fd7762b031 100644--- a/docs/release-control/v6/internal/subsystems/api-contracts.md+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md@@ -12519,3 +12519,29 @@ affordance fields keep their shape. The browser normalizer's fallback for anundeclared Assistant surface (`frontend-modern/src/api/agentCapabilities.ts`)now matches the backend default of tools only, so the External agents settingspanel never shows an "Interactive questions" badge.++### Closed runs are finalized and swept expiries published (23 October 2026)++`orchestratorFindingsAdapter.Update` finalizes a run when the write that closes+it lands, after reconciling the run's action from the action audit. The record,+the unified finding and the approval request that only a successful automatic+run got before now follow every closed run, failed or successful, automatic or+manual, and a run the stuck-run sweep recovers once its session has closed (see+the AI runtime contract). An action's outcome keeps one push, from transition+reconciliation, and an approval request is sent once per action. A close that+reports a planned fix as queued no longer moves the finding back from an+outcome the action reached during the run. The adapter takes the run's session+from the orchestrator's session store, so the Pulse–enterprise boundary is+unchanged: no `aicontracts` type, field or method is added. The+transport-independent action lifecycle's `List` and `DecisionQueue` now publish+each plan their expiry sweep expires through `OnActionTransition`, including+plans expired before a sweep fails, as `materializeExpiry` already did for one+action. `actionlifecycle.RefreshedActionID` names the action a refresh's+replacement replaced, from the `refresh:` request ID `Refresh` already gives+it, and Patrol reconciliation ignores a transition for a plan that the+investigation's current action replaced. No route, payload or action state+changes. Proofs: `internal/actionlifecycle/service_test.go`+(`TestExpirySweepsPublishEachExpiredPlan`,+`TestRefreshedActionIDNamesTheReplacedAction`),+`internal/api/patrol_action_expiry_sweep_test.go` and+`internal/api/ai_handlers_investigation_finalize_test.go`.diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex 2b5ad3347c..4c18b15efa 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -7193,3 +7193,8 @@ name those routes are now history. Persisted chat session files no longercarry `turn_redo_stack` or the `steered` message flag; older files that stillhave them load unchanged and drop them on the next write. No backup, restore,or recovery path changes.++The `internal/api` change of 2026-10-23 that finalizes Patrol investigation+runs when they close and publishes the action plans the inbox expiry sweep+expires opens no storage, backup or recovery path. Recovery evidence, restore+rules and storage capabilities are unchanged.diff --git a/internal/actionlifecycle/service.go b/internal/actionlifecycle/service.goindex a6ca0de5f6..b7b470fe3a 100644--- a/internal/actionlifecycle/service.go+++ b/internal/actionlifecycle/service.go@@ -557,6 +557,17 @@ func (s *Service) PlanWithOptions(ctx context.Context, orgID string, req unifiedreturn plan, nil}+// refreshRequestPrefix starts the request ID a refresh plans its replacement+// under; the replaced action's ID follows it.+const refreshRequestPrefix = "refresh:"++// RefreshedActionID returns the action that a refresh replaced with record,+// when record is such a replacement.+func RefreshedActionID(record unified.ActionAuditRecord) (string, bool) {+ replaced, ok := strings.CutPrefix(strings.TrimSpace(record.Request.RequestID), refreshRequestPrefix)+ return replaced, ok && replaced != ""+}+// Refresh creates a fresh immutable plan for an expired or drifted action.// The previous record remains historical (and an unexpired drifted record// naturally expires at its original short TTL); callers always receive the@@ -583,7 +594,7 @@ func (s *Service) Refresh(ctx context.Context, orgID, actionID string, actor uniif !refreshable {return unified.ActionAuditRecord{}, ErrActionRefreshNotAllowed}- requestID := "refresh:" + previous.ID+ requestID := refreshRequestPrefix + previous.IDreq := previous.Requestreq.RequestID = requestIDreq.RequestedBy = ""@@ -667,8 +678,8 @@ func (s *Service) List(orgID string, view ActionListView, limit int) ([]unified.if err != nil {return nil, err}- if _, err := store.ExpireActionAudits(s.now(), 500); err != nil {- return nil, &PersistError{Op: "action expiry sweep", Err: err}+ if err := s.sweepExpiry(orgID, store); err != nil {+ return nil, err}var states []unified.ActionStateswitch view {@@ -693,8 +704,8 @@ func (s *Service) DecisionQueue(orgID string, limit int) ([]unified.ActionAuditRif err != nil {return nil, err}- if _, err := store.ExpireActionAudits(s.now(), 500); err != nil {- return nil, &PersistError{Op: "action expiry sweep", Err: err}+ if err := s.sweepExpiry(orgID, store); err != nil {+ return nil, err}records, err := store.GetPendingActionAudits(limit)if err != nil {@@ -703,6 +714,21 @@ func (s *Service) DecisionQueue(orgID string, limit int) ([]unified.ActionAuditRreturn records, nil}+// sweepExpiry materializes every plan past its expiry and publishes each one,+// as a single action's expiry is published, so surfaces that project actions,+// such as the queued fix on a Patrol finding, learn of it. Plans the sweep+// expired before failing are published too.+func (s *Service) sweepExpiry(orgID string, store Store) error {+ expired, err := store.ExpireActionAudits(s.now(), 500)+ for _, record := range expired {+ s.publishTransition(orgID, record)+ }+ if err != nil {+ return &PersistError{Op: "action expiry sweep", Err: err}+ }+ return nil+}+// ResourceOperatorState returns the authoritative per-resource policy used by// execution and Patrol policy authorization.func (s *Service) ResourceOperatorState(orgID, resourceID string) (unified.ResourceOperatorState, bool, error) {diff --git a/internal/actionlifecycle/service_test.go b/internal/actionlifecycle/service_test.goindex 168ef3fab1..e0aa5caf3a 100644--- a/internal/actionlifecycle/service_test.go+++ b/internal/actionlifecycle/service_test.go@@ -2123,3 +2123,81 @@ func TestIssue1649ForceFailRefusesTerminalAndUndispatchedActions(t *testing.T) {t.Fatalf("missing action ForceFail error=%v", err)}}++// A refresh's replacement names the action it replaced, which is how a+// surface that projects actions tells the replaced plan's later transitions+// from its replacement's.+func TestRefreshedActionIDNamesTheReplacedAction(t *testing.T) {+ now := time.Date(2026, 10, 23, 18, 0, 0, 0, time.UTC)+ store := unified.NewMemoryStore()+ service := serviceForStore(t, store, testResource(now, unified.ApprovalAdmin), &stubExecutor{})+ service.Now = func() time.Time { return now }+ plan, err := service.Plan(context.Background(), "default", restartRequest(), testActionActor("requester", "default"))+ if err != nil {+ t.Fatal(err)+ }+ service.Now = func() time.Time { return plan.ExpiresAt.Add(time.Second) }+ replacement, err := service.Refresh(context.Background(), "default", plan.ActionID, testActionActor("operator", "default"))+ if err != nil {+ t.Fatal(err)+ }+ if replaced, ok := RefreshedActionID(replacement); !ok || replaced != plan.ActionID {+ t.Fatalf("RefreshedActionID(replacement) = %q, %v, want %q", replaced, ok, plan.ActionID)+ }+ original, found, err := store.GetActionAudit(plan.ActionID)+ if err != nil || !found {+ t.Fatalf("original: found=%v err=%v", found, err)+ }+ if replaced, ok := RefreshedActionID(original); ok {+ t.Fatalf("a first plan names a replaced action: %q", replaced)+ }+}++// The inbox and decision-queue sweeps publish each plan they expire, as a+// single action's expiry is published, so a surface that projects the action,+// such as a Patrol finding's queued fix, does not outlive the plan. Each expiry+// is published once.+func TestExpirySweepsPublishEachExpiredPlan(t *testing.T) {+ sweeps := map[string]func(*Service) error{+ "inbox": func(s *Service) error {+ _, err := s.List("default", ActionListPending, 100)+ return err+ },+ "decision queue": func(s *Service) error {+ _, err := s.DecisionQueue("default", 100)+ return err+ },+ }+ for name, sweep := range sweeps {+ t.Run(name, func(t *testing.T) {+ now := time.Date(2026, 10, 23, 18, 0, 0, 0, time.UTC)+ store := unified.NewMemoryStore()+ service := serviceForStore(t, store, testResource(now, unified.ApprovalAdmin), &stubExecutor{})+ service.Now = func() time.Time { return now }+ plan, err := service.Plan(context.Background(), "default", restartRequest(), testActionActor("requester", "default"))+ if err != nil {+ t.Fatal(err)+ }+ var published []unified.ActionAuditRecord+ var orgs []string+ service.OnActionTransition = func(orgID string, record unified.ActionAuditRecord) {+ published = append(published, record)+ orgs = append(orgs, orgID)+ }++ if err := sweep(service); err != nil || len(published) != 0 {+ t.Fatalf("a sweep before expiry: err=%v published=%+v", err, published)+ }+ service.Now = func() time.Time { return plan.ExpiresAt.Add(time.Second) }+ if err := sweep(service); err != nil {+ t.Fatal(err)+ }+ if len(published) != 1 || published[0].ID != plan.ActionID || published[0].State != unified.ActionStateExpired || orgs[0] != "default" {+ t.Fatalf("published = %+v for orgs %v, want one expiry of %s", published, orgs, plan.ActionID)+ }+ if err := sweep(service); err != nil || len(published) != 1 {+ t.Fatalf("a later sweep published the expiry again: err=%v published=%+v", err, published)+ }+ })+ }+}diff --git a/internal/ai/findings.go b/internal/ai/findings.goindex c01b5c7ded..b644eb0f03 100644--- a/internal/ai/findings.go+++ b/internal/ai/findings.go@@ -2728,7 +2728,10 @@ func (s *FindingsStore) StartInvestigation(id, sessionID string, runStartedAt ti// cannot prove ownership once this process has seen an occurrence begin. A run// whose result does not count, but which the finding still shows as running,// is closed now through RecoverStuckInvestigation, as the stuck-run sweep would-// later close it, rather than left running. It reports whether the run's+// later close it, rather than left running. A run that reports its planned fix+// as queued does not move the finding back from an outcome the fix's action+// has reached since: a run starts with no outcome, so one already present came+// from the action lifecycle during this run. It reports whether the run's// result was recorded.func (s *FindingsStore) CompleteInvestigation(id, sessionID string, runStartedAt time.Time, status, outcome string, lastInvestigatedAt *time.Time, attempts int) bool {s.mu.Lock()@@ -2746,6 +2749,9 @@ func (s *FindingsStore) CompleteInvestigation(id, sessionID string, runStartedAts.RecoverStuckInvestigation(id, sessionID, runStartedAt, InvestigationStatusFailed, InvestigationOutcomeInterrupted, closedAt)return false}+ if outcome == string(InvestigationOutcomeFixQueued) && f.InvestigationOutcome != "" {+ outcome = f.InvestigationOutcome+ }s.updateInvestigationLocked(f, sessionID, status, outcome, lastInvestigatedAt, attempts)s.mu.Unlock()s.scheduleSave()diff --git a/internal/ai/patrol.go b/internal/ai/patrol.goindex d053975a8f..09bdc6baa3 100644--- a/internal/ai/patrol.go+++ b/internal/ai/patrol.go@@ -551,6 +551,11 @@ type PatrolService struct {// Investigation orchestrator for autonomous investigation of findingsinvestigationOrchestrator InvestigationOrchestratorinvestigationWg sync.WaitGroup // Tracks in-flight investigation goroutines+ // investigationRunCloser finalizes a closed run; see SetInvestigationRunCloser.+ investigationRunCloser func(findingID string, session *InvestigationSession)+ // approvalRequests holds when each action's approval was last requested,+ // so a run finalized twice does not ask twice; see claimApprovalRequest.+ approvalRequests map[string]time.Time// Unified findings callback - pushes findings to unified storeunifiedFindingCallback UnifiedFindingCallbackdiff --git a/internal/ai/patrol_findings.go b/internal/ai/patrol_findings.goindex adad51a674..ea79755cf7 100644--- a/internal/ai/patrol_findings.go+++ b/internal/ai/patrol_findings.go@@ -1594,6 +1594,12 @@ func (p *PatrolService) recoverStuckInvestigations() {Str("resource", f.ResourceName).Str("outcome", string(result.outcome)).Msg("AI Patrol: Recovered stuck investigation")+ // A run whose session closed without its close reaching the finding,+ // such as one a restart interrupted after it planned a fix, is+ // finalized here. A run still going finalizes when its close lands.+ if result.session != nil {+ p.closeInvestigationRun(f.ID, result.session)+ }}if recovered > 0 {log.Info().Int("recovered", recovered).@@ -1604,11 +1610,13 @@ func (p *PatrolService) recoverStuckInvestigations() {// stuckInvestigation is how to close a run that is no longer live.// runStartedAt is the start of the run's session, zero when the orchestrator// holds none, so the store can check the run belongs to the finding's current-// occurrence before it applies the session's result.+// occurrence before it applies the session's result. session is the run's+// session once it has closed, which finalizes the run after recovery.type stuckInvestigation struct {status InvestigationStatusoutcome InvestigationOutcomerunStartedAt time.Time+ session *InvestigationSession}// stuckInvestigationResult reports whether a finding marked as running has a@@ -1647,6 +1655,7 @@ func stuckInvestigationResult(f *Finding, orchestrator InvestigationOrchestrator}return closed, true}+ closed.session = sessionif session.Outcome == aicontracts.OutcomeInterrupted {// Closed when the orchestrator restarted, or cancelled by shutdown.closed.outcome = InvestigationOutcomeInterrupted@@ -1880,57 +1889,15 @@ func (p *PatrolService) maybeInvestigateFinding(f *Finding) bool {// cutting longer configured or subscription-agent runs short.ctx, cancel := context.WithTimeout(context.Background(), currentCfg.GetPatrolInvestigationTimeout()+investigationRunGrace)defer cancel()+ // The run is finalized when it closes, successfully or not, through+ // the orchestrator's findings store (FinalizeInvestigationRun), not+ // here: a failed run that planned a fix still needs its record, its+ // unified finding and the approval push.if err := orchestrator.InvestigateFinding(ctx, invFinding, currentAutonomy); err != nil {log.Error().Err(err).Str("finding_id", f.ID).- Msg("Failed to start investigation")- return- }-- // The orchestrator updates the patrol findings store; sync the latest state to the unified store.- // This makes fix verification and resolution visible as an actual closed loop in the UI.- var pushUnified UnifiedFindingCallback- var resolveUnified func(string)- var pushCb PushNotifyCallback- p.mu.RLock()- pushUnified = p.unifiedFindingCallback- resolveUnified = p.unifiedFindingResolver- pushCb = p.pushNotifyCallback- p.mu.RUnlock()- if latest := p.findings.Get(f.ID); latest != nil {- var latestInvestigation *InvestigationSession- if orchestrator != nil {- latestInvestigation = orchestrator.GetInvestigationByFinding(latest.ID)- }- if p.storeFindingInvestigationRecord(latest.ID, latestInvestigation, false) {- if refreshed := p.findings.Get(latest.ID); refreshed != nil {- latest = refreshed- }- }- if pushUnified != nil {- pushUnified(latest)- }- if latest.ResolvedAt != nil && resolveUnified != nil {- resolveUnified(latest.ID)- }-- // Send push notifications for investigation outcomes- if pushCb != nil {- switch latest.InvestigationOutcome {- case string(InvestigationOutcomeFixQueued):- if latestInvestigation != nil && latestInvestigation.Action != nil && latestInvestigation.Action.State == "pending_approval" {- pushCb(relay.NewActionDecisionNotification(- latestInvestigation.Action.ActionID,- latest.Title,- ))- }- case string(InvestigationOutcomeFixExecuted), string(InvestigationOutcomeFixVerified):- pushCb(relay.NewFixCompletedNotification(latest.ID, latest.Title, true))- case string(InvestigationOutcomeFixFailed), string(InvestigationOutcomeFixVerificationFailed):- pushCb(relay.NewFixCompletedNotification(latest.ID, latest.Title, false))- }- }+ Msg("Investigation ended with an error")}// Typed remediation now flows exclusively through the action@@ -1948,6 +1915,96 @@ func (p *PatrolService) maybeInvestigateFinding(f *Finding) bool {return true}+// SetInvestigationRunCloser sets how a closed run is finalized. Pulse's API+// layer reconciles the run's action from the action audit, then calls+// FinalizeInvestigationRun. Without a closer a run is finalized as it is.+func (p *PatrolService) SetInvestigationRunCloser(closer func(findingID string, session *InvestigationSession)) {+ p.mu.Lock()+ defer p.mu.Unlock()+ p.investigationRunCloser = closer+}++func (p *PatrolService) closeInvestigationRun(findingID string, session *InvestigationSession) {+ p.mu.RLock()+ closer := p.investigationRunCloser+ p.mu.RUnlock()+ if closer != nil {+ closer(findingID, session)+ return+ }+ p.FinalizeInvestigationRun(findingID, session)+}++// FinalizeInvestigationRun projects a run that has just closed, successfully or+// not, onto the finding: it rebuilds the durable record from the run's final+// session, publishes the unified finding and, when the run left a planned fix+// awaiting approval, asks the operator to approve it, once per action. It runs+// when the run's close lands on the finding or the stuck-run sweep recovers a+// run whose session closed, so a manual re-investigation and a run that failed+// after planning are finalized like any other run. A run the finding no longer+// names, because a newer run or occurrence took it over, is not finalized.+func (p *PatrolService) FinalizeInvestigationRun(findingID string, session *InvestigationSession) {+ if p == nil || p.findings == nil || session == nil {+ return+ }+ latest := p.findings.Get(findingID)+ if latest == nil || latest.InvestigationSessionID != session.SessionID || !latest.IsCurrentInvestigation(session) {+ return+ }+ p.storeFindingInvestigationRecord(latest.ID, session, false)+ // A newer run or occurrence can take the finding while the record is+ // written, and the store then refuses this run's record.+ latest = p.findings.Get(findingID)+ if latest == nil || latest.InvestigationSessionID != session.SessionID || !latest.IsCurrentInvestigation(session) {+ return+ }+ p.mu.RLock()+ pushUnified := p.unifiedFindingCallback+ resolveUnified := p.unifiedFindingResolver+ pushCb := p.pushNotifyCallback+ p.mu.RUnlock()+ // Sync the closed run to the unified store, so fix verification and+ // resolution show as a closed loop in the UI.+ if pushUnified != nil {+ pushUnified(latest)+ }+ if latest.ResolvedAt != nil && resolveUnified != nil {+ resolveUnified(latest.ID)+ }+ // An action's outcome, such as a verified fix, is pushed once when its+ // transition is reconciled (PublishFindingLifecycleUpdate), not here.+ if pushCb != nil && latest.InvestigationOutcome == string(InvestigationOutcomeFixQueued) &&+ session.Action != nil && session.Action.State == "pending_approval" &&+ p.claimApprovalRequest(session.Action.ActionID, time.Now()) {+ pushCb(relay.NewActionDecisionNotification(session.Action.ActionID, latest.Title))+ }+}++// approvalRequestMemory is how long an action's approval request is+// remembered, comfortably past any plan's expiry.+const approvalRequestMemory = 24 * time.Hour++// claimApprovalRequest reports whether an approval request for actionID may be+// sent: once per action, so a run finalized again, as when a late close lands+// after the stuck-run sweep recovered the run, does not ask twice.+func (p *PatrolService) claimApprovalRequest(actionID string, now time.Time) bool {+ p.mu.Lock()+ defer p.mu.Unlock()+ for id, at := range p.approvalRequests {+ if now.Sub(at) > approvalRequestMemory {+ delete(p.approvalRequests, id)+ }+ }+ if _, requested := p.approvalRequests[actionID]; requested {+ return false+ }+ if p.approvalRequests == nil {+ p.approvalRequests = make(map[string]time.Time)+ }+ p.approvalRequests[actionID] = now+ return true+}+// RefreshFindingInvestigationRecord preserves the latest investigation and// reconciled action in the durable record shared by product surfaces.func (p *PatrolService) RefreshFindingInvestigationRecord(findingID string, session *InvestigationSession) bool {diff --git a/internal/ai/patrol_investigation_finalize_test.go b/internal/ai/patrol_investigation_finalize_test.gonew file mode 100644index 0000000000..096f05a255--- /dev/null+++ b/internal/ai/patrol_investigation_finalize_test.go@@ -0,0 +1,179 @@+package ai++import (+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/relay"+ "github.com/rcourtman/pulse-go-rewrite/pkg/aicontracts"+)++// finalizedRun is a Patrol service whose unified publications and pushes are+// recorded, with one finding whose run, chat-1, closed with outcome.+type finalizedRun struct {+ patrol *PatrolService+ store *FindingsStore+ published []*Finding+ pushes []relay.PushNotificationPayload+}++func newFinalizedRun(t *testing.T, outcome InvestigationOutcome) *finalizedRun {+ t.Helper()+ run := &finalizedRun{store: NewFindingsStore()}+ run.patrol = &PatrolService{findings: run.store}+ run.patrol.SetUnifiedFindingCallback(func(f *Finding) bool {+ run.published = append(run.published, f)+ return true+ })+ run.patrol.SetPushNotifyCallback(func(n relay.PushNotificationPayload) { run.pushes = append(run.pushes, n) })+ run.store.Add(&Finding{ID: "finding-1", ResourceID: "vm-100", Title: "Backup service is crash-looping", DetectedAt: time.Now(),+ InvestigationSessionID: "chat-1", InvestigationStatus: string(InvestigationStatusFailed),+ InvestigationOutcome: string(outcome)})+ return run+}++// A run that planned a fix and then failed is finalized like a completed run:+// its record replaces the projection made when the plan was recorded, the+// unified finding is published and the queued fix asks for approval.+func TestFinalizeInvestigationRunFinalizesAFailedRunThatPlannedAFix(t *testing.T) {+ run := newFinalizedRun(t, InvestigationOutcomeFixQueued)++ // Core projects the plan onto the record while the run is still going.+ session := &InvestigationSession{ID: "investigation-1", FindingID: "finding-1", SessionID: "chat-1",+ Status: aicontracts.InvestigationStatusRunning, StartedAt: time.Now(),+ Action: &aicontracts.ActionReference{ActionID: "action-1", State: "pending_approval"}}+ if !run.patrol.RefreshFindingInvestigationRecord("finding-1", session) {+ t.Fatal("the plan's projection was not recorded")+ }++ // The final model call then fails, and the run closes with the plan's outcome.+ closed := time.Now()+ session.Status = aicontracts.InvestigationStatusFailed+ session.Outcome = aicontracts.OutcomeFixQueued+ session.CompletedAt = &closed+ session.Error = "investigation final response failed: context deadline exceeded"+ run.patrol.FinalizeInvestigationRun("finding-1", session)++ record := run.store.Get("finding-1").InvestigationRecord+ if record == nil || record.Status != aicontracts.InvestigationStatusFailed || record.Error != session.Error || record.CompletedAt == nil {+ t.Fatalf("the record kept the projection made while the run was going: %#v", record)+ }+ if len(run.published) != 1 || run.published[0].ID != "finding-1" {+ t.Fatalf("published %d findings, want the closed run once", len(run.published))+ }+ if len(run.pushes) != 1 || run.pushes[0].Type != relay.PushTypeApprovalRequest || run.pushes[0].ActionID != "action-1" {+ t.Fatalf("pushes = %+v, want one approval request for action-1", run.pushes)+ }+}++// A run the finding no longer names, because a newer run took it over after+// this one's close landed, is not finalized: its action would otherwise be+// offered for approval under the newer run's outcome.+func TestFinalizeInvestigationRunSkipsARunTheFindingNoLongerNames(t *testing.T) {+ run := newFinalizedRun(t, InvestigationOutcomeFixQueued)+ closed := time.Now()+ older := &InvestigationSession{ID: "investigation-0", FindingID: "finding-1", SessionID: "chat-0",+ Status: aicontracts.InvestigationStatusFailed, Outcome: aicontracts.OutcomeFixQueued, StartedAt: closed.Add(-time.Hour), CompletedAt: &closed,+ Action: &aicontracts.ActionReference{ActionID: "action-0", State: "pending_approval"}}++ run.patrol.FinalizeInvestigationRun("finding-1", older)++ if record := run.store.Get("finding-1").InvestigationRecord; record != nil || len(run.published) != 0 || len(run.pushes) != 0 {+ t.Fatalf("an older run was finalized: record=%#v published=%d pushes=%+v", record, len(run.published), run.pushes)+ }+}++// A run finalized twice, as when a late close lands after the stuck-run sweep+// recovered the run, asks for approval once.+func TestFinalizeInvestigationRunAsksForApprovalOnce(t *testing.T) {+ run := newFinalizedRun(t, InvestigationOutcomeFixQueued)+ closed := time.Now()+ session := &InvestigationSession{ID: "investigation-1", FindingID: "finding-1", SessionID: "chat-1",+ Status: aicontracts.InvestigationStatusFailed, Outcome: aicontracts.OutcomeFixQueued, StartedAt: closed.Add(-time.Minute), CompletedAt: &closed,+ Action: &aicontracts.ActionReference{ActionID: "action-1", State: "pending_approval"}}++ run.patrol.FinalizeInvestigationRun("finding-1", session)+ run.patrol.FinalizeInvestigationRun("finding-1", session)++ if len(run.pushes) != 1 || run.pushes[0].ActionID != "action-1" {+ t.Fatalf("pushes = %+v, want one approval request", run.pushes)+ }+}++// A run that reports its planned fix as queued does not move the finding back+// from an outcome the fix's action reached while the run was closing, here a+// rejection. A run starts with no outcome, so one present at its close came+// from the action lifecycle.+func TestCompleteInvestigationKeepsAnOutcomeTheActionReached(t *testing.T) {+ findings := NewFindingsStore()+ findings.Add(runningInvestigationFinding("rejected", "chat-rejected", nil))+ findings.Add(runningInvestigationFinding("unplanned", "chat-unplanned", nil))+ started := time.Now()+ rejected := &InvestigationSession{ID: "inv-rejected", FindingID: "rejected", SessionID: "chat-rejected", StartedAt: started}+ if applied, _ := findings.ReconcileInvestigationOutcome("rejected", rejected, string(InvestigationOutcomeFixRejected)); !applied {+ t.Fatal("the action's rejection was not reconciled")+ }+ closed := time.Now()++ if !findings.CompleteInvestigation("rejected", "chat-rejected", started, string(InvestigationStatusFailed), string(InvestigationOutcomeFixQueued), &closed, 1) {+ t.Fatal("the run's close was refused")+ }+ if !findings.CompleteInvestigation("unplanned", "chat-unplanned", started, string(InvestigationStatusFailed), string(InvestigationOutcomeTimedOut), &closed, 1) {+ t.Fatal("the run's close was refused")+ }++ assertInvestigationState(t, findings, "rejected", InvestigationStatusFailed, InvestigationOutcomeFixRejected)+ assertInvestigationState(t, findings, "unplanned", InvestigationStatusFailed, InvestigationOutcomeTimedOut)+}++// An action's outcome is pushed when its transition is reconciled+// (PublishFindingLifecycleUpdate), so a run that closes on a verified fix is+// published without a second push.+func TestFinalizeInvestigationRunLeavesActionOutcomePushesToReconciliation(t *testing.T) {+ run := newFinalizedRun(t, InvestigationOutcomeFixVerified)+ closed := time.Now()+ session := &InvestigationSession{ID: "investigation-1", FindingID: "finding-1", SessionID: "chat-1",+ Status: aicontracts.InvestigationStatusCompleted, Outcome: aicontracts.OutcomeFixVerified, StartedAt: closed.Add(-time.Minute), CompletedAt: &closed,+ Action: &aicontracts.ActionReference{ActionID: "action-1", State: "completed", VerificationStatus: "verified"}}++ run.patrol.FinalizeInvestigationRun("finding-1", session)++ if len(run.published) != 1 || len(run.pushes) != 0 {+ t.Fatalf("published=%d pushes=%+v, want one publication and no push", len(run.published), run.pushes)+ }+}++// The stuck-run sweep finalizes a run it recovers once the run's session has+// closed without its close reaching the finding, such as a run a restart+// interrupted after it planned a fix. A run still going past its limit is not+// finalized: it finalizes when its own close lands.+func TestRecoverStuckInvestigationsFinalizesAClosedRun(t *testing.T) {+ now := time.Now()+ closedAt := now.Add(-time.Second)+ findings := NewFindingsStore()+ interrupted := runningInvestigationFinding("restarted", "chat-restarted", nil)+ // Core recorded the plan's queued fix while the run was going.+ interrupted.InvestigationOutcome = string(InvestigationOutcomeFixQueued)+ findings.Add(interrupted)+ findings.Add(runningInvestigationFinding("hung", "chat-hung", nil))++ orchestrator := &sessionReportingOrchestrator{sessions: map[string]*InvestigationSession{+ "restarted": {ID: "inv-restarted", FindingID: "restarted", SessionID: "chat-restarted", Status: aicontracts.InvestigationStatusFailed,+ Outcome: aicontracts.OutcomeInterrupted, StartedAt: now.Add(-3 * time.Minute), CompletedAt: &closedAt,+ Action: &aicontracts.ActionReference{ActionID: "action-1", State: "pending_approval"}},+ "hung": {ID: "inv-hung", FindingID: "hung", SessionID: "chat-hung", Status: aicontracts.InvestigationStatusRunning, StartedAt: now.Add(-time.Hour)},+ }}+ patrol := &PatrolService{findings: findings, investigationOrchestrator: orchestrator}+ var closed []string+ patrol.SetInvestigationRunCloser(func(findingID string, session *InvestigationSession) {+ closed = append(closed, findingID+"/"+session.SessionID)+ })++ patrol.recoverStuckInvestigations()++ assertInvestigationState(t, findings, "restarted", InvestigationStatusCompleted, InvestigationOutcomeFixQueued)+ assertInvestigationState(t, findings, "hung", InvestigationStatusFailed, InvestigationOutcomeTimedOut)+ if len(closed) != 1 || closed[0] != "restarted/chat-restarted" {+ t.Fatalf("closed runs = %v, want only the restarted run's closed session", closed)+ }+}diff --git a/internal/api/ai_handlers.go b/internal/api/ai_handlers.goindex 07abae14b8..bc1a41f1de 100644--- a/internal/api/ai_handlers.go+++ b/internal/api/ai_handlers.go@@ -1932,7 +1932,18 @@ func (h *AISettingsHandler) setupInvestigationOrchestrator(orgID string, svc *ailog.Warn().Str("orgID", orgID).Msg("Findings store not available for orchestrator")return}- findingsAdapter := &orchestratorFindingsAdapter{store: &findingsStoreWrapper{store: findingsStore}, sessions: store}+ // A closed run's action is reconciled from the action audit before the run+ // is finalized: the orchestrator writes the action back as it was planned,+ // which an approval or rejection during the run may have overtaken.+ closeRun := func(findingID string, session *aicontracts.InvestigationSession) {+ patrol.FinalizeInvestigationRun(findingID, h.hydratePatrolInvestigationAction(orgID, session))+ }+ patrol.SetInvestigationRunCloser(closeRun)+ findingsAdapter := &orchestratorFindingsAdapter{+ store: &findingsStoreWrapper{store: findingsStore},+ sessions: store,+ closed: closeRun,+ }// Get config for investigation settingscfg := svc.GetConfig()@@ -2114,6 +2125,8 @@ type orchestratorFindingsAdapter struct {// sessions is the orchestrator's session store, which records when each// run started.sessions aicontracts.InvestigationStore+ // closed finalizes a run once its close has landed on the finding.+ closed func(findingID string, session *aicontracts.InvestigationSession)}func (a *orchestratorFindingsAdapter) Get(id string) *aicontracts.Finding {@@ -2147,7 +2160,9 @@ func (a *orchestratorFindingsAdapter) Get(id string) *aicontracts.Finding {// a run installs it, unless the run was started for an earlier occurrence of// the finding. Every other write closes a run and lands only while the finding// still names that run, so a run that ends after its finding was re-detected,-// or after a newer run took over, cannot overwrite the newer state.+// or after a newer run took over, cannot overwrite the newer state. A close+// that lands finalizes the run, whether it succeeded or failed and whether+// Patrol or an operator started it.func (a *orchestratorFindingsAdapter) Update(f *aicontracts.Finding) bool {if a.store == nil || f == nil {return false@@ -2165,7 +2180,7 @@ func (a *orchestratorFindingsAdapter) Update(f *aicontracts.Finding) bool {f.InvestigationAttempts,)}- return a.store.CompleteInvestigation(+ if !a.store.CompleteInvestigation(f.ID,f.InvestigationSessionID,runStartedAt,@@ -2173,20 +2188,37 @@ func (a *orchestratorFindingsAdapter) Update(f *aicontracts.Finding) bool {f.InvestigationOutcome,f.LastInvestigatedAt,f.InvestigationAttempts,- )+ ) {+ return false+ }+ if a.closed != nil {+ if session := a.session(f.ID, f.InvestigationSessionID); session != nil {+ a.closed(f.ID, session)+ }+ }+ return true}-// runStartedAt returns when the orchestrator's session for a run started, or-// zero when its session store holds no record of the run.-func (a *orchestratorFindingsAdapter) runStartedAt(findingID, sessionID string) time.Time {+// session returns the orchestrator's session for a run, or nil when its+// session store holds no record of the run.+func (a *orchestratorFindingsAdapter) session(findingID, sessionID string) *aicontracts.InvestigationSession {if a.sessions == nil {- return time.Time{}+ return nil}for _, session := range a.sessions.GetByFinding(findingID) {if session != nil && session.SessionID == sessionID {- return session.StartedAt+ return session}}+ return nil+}++// runStartedAt returns when the orchestrator's session for a run started, or+// zero when its session store holds no record of the run.+func (a *orchestratorFindingsAdapter) runStartedAt(findingID, sessionID string) time.Time {+ if session := a.session(findingID, sessionID); session != nil {+ return session.StartedAt+ }return time.Time{}}diff --git a/internal/api/ai_handlers_investigation_additional_test.go b/internal/api/ai_handlers_investigation_additional_test.goindex b29f0054eb..5aa368ce5a 100644--- a/internal/api/ai_handlers_investigation_additional_test.go+++ b/internal/api/ai_handlers_investigation_additional_test.go@@ -944,6 +944,15 @@ func TestSetupInvestigationOrchestratorUsesSubscriptionAgentLimits(t *testing.T)// returns the findings and session stores it was built with, plus the// tenant's findings.func orchestratorFindingsFor(t *testing.T) (aicontracts.OrchestratorFindingsStore, aicontracts.InvestigationStore, *ai.FindingsStore) {+ t.Helper()+ findings, sessions, patrol := orchestratorPatrolFor(t, nil)+ return findings, sessions, patrol.GetFindings()+}++// orchestratorPatrolFor is orchestratorFindingsFor with the tenant's Patrol+// service instead of only its findings, and audits as the action audit store+// when it is not nil.+func orchestratorPatrolFor(t *testing.T, audits unifiedresources.ResourceStore) (aicontracts.OrchestratorFindingsStore, aicontracts.InvestigationStore, *ai.PatrolService) {t.Helper()prevStoreFactory := getCreateInvestigationStore()SetCreateInvestigationStore(func(dataDir string) aicontracts.InvestigationStore {@@ -964,6 +973,9 @@ func orchestratorFindingsFor(t *testing.T) (aicontracts.OrchestratorFindingsStorhandler := &AISettingsHandler{investigationStores: make(map[string]aicontracts.InvestigationStore),}+ if audits != nil {+ handler.resourceStoreProvider = func(string) (unifiedresources.ResourceStore, error) { return audits, nil }+ }handler.chatHandler = &AIHandler{services: map[string]AIService{"tenant-1": chat.NewService(chat.Config{AIConfig: config.NewDefaultAIConfig()}),@@ -973,7 +985,7 @@ func orchestratorFindingsFor(t *testing.T) (aicontracts.OrchestratorFindingsStorif deps.FindingsStore == nil || deps.Store == nil {t.Fatal("investigation orchestrator was not wired")}- return deps.FindingsStore, deps.Store, tenantSvc.GetPatrolService().GetFindings()+ return deps.FindingsStore, deps.Store, tenantSvc.GetPatrolService()}func investigatedFinding(id string) *ai.Finding {diff --git a/internal/api/ai_handlers_investigation_finalize_test.go b/internal/api/ai_handlers_investigation_finalize_test.gonew file mode 100644index 0000000000..959f38a091--- /dev/null+++ b/internal/api/ai_handlers_investigation_finalize_test.go@@ -0,0 +1,184 @@+package api++import (+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/ai"+ "github.com/rcourtman/pulse-go-rewrite/internal/relay"+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"+ "github.com/rcourtman/pulse-go-rewrite/pkg/aicontracts"+)++// runFinalization records what finalizing a closed run sent out.+type runFinalization struct {+ published []*ai.Finding+ pushes []relay.PushNotificationPayload+}++// watchRunFinalization captures the Patrol service's unified publications and+// pushes. Set it before adding findings: setting the unified callback syncs+// the active findings at once.+func watchRunFinalization(patrol *ai.PatrolService) *runFinalization {+ sent := &runFinalization{}+ patrol.SetUnifiedFindingCallback(func(f *ai.Finding) bool {+ sent.published = append(sent.published, f)+ return true+ })+ patrol.SetPushNotifyCallback(func(n relay.PushNotificationPayload) {+ sent.pushes = append(sent.pushes, n)+ })+ return sent+}++// A run is finalized when its close lands, whether it completed or failed: its+// record shows the run's final state, the unified finding is published once and+// a planned fix awaiting approval asks the operator to approve it. A run that+// planned a fix and then failed, normally on its final model call, used to get+// none of this. An action's outcome is pushed when its transition is+// reconciled, so a close does not push it again.+func TestOrchestratorRunCloseFinalizesTheRun(t *testing.T) {+ cases := []struct {+ name string+ status aicontracts.InvestigationStatus+ outcome aicontracts.InvestigationOutcome+ fail string+ action *aicontracts.ActionReference+ approval bool+ }{+ {+ name: "failed after planning a fix",+ status: aicontracts.InvestigationStatusFailed,+ outcome: aicontracts.OutcomeFixQueued,+ fail: "investigation final response failed: context deadline exceeded",+ action: &aicontracts.ActionReference{ActionID: "action-1", State: "pending_approval"},+ approval: true,+ },+ {+ name: "completed with a planned fix",+ status: aicontracts.InvestigationStatusCompleted,+ outcome: aicontracts.OutcomeFixQueued,+ action: &aicontracts.ActionReference{ActionID: "action-1", State: "pending_approval"},+ approval: true,+ },+ {+ name: "completed with a verified fix",+ status: aicontracts.InvestigationStatusCompleted,+ outcome: aicontracts.OutcomeFixVerified,+ action: &aicontracts.ActionReference{ActionID: "action-1", State: "completed", VerificationStatus: "verified"},+ },+ {+ name: "timed out without a plan",+ status: aicontracts.InvestigationStatusFailed,+ outcome: aicontracts.OutcomeTimedOut,+ fail: "context deadline exceeded",+ },+ }+ for _, tc := range cases {+ t.Run(tc.name, func(t *testing.T) {+ orchestratorFindings, sessions, patrol := orchestratorPatrolFor(t, nil)+ sent := watchRunFinalization(patrol)+ findings := patrol.GetFindings()+ findings.Add(investigatedFinding("finding-1"))+ run := startOrchestratorRun(t, orchestratorFindings, sessions, "finding-1", "chat-1")+ if len(sent.published) != 0 || len(sent.pushes) != 0 {+ t.Fatalf("starting a run finalized it: %+v", sent)+ }++ // The orchestrator records the run's end on its session before+ // it closes the run on the finding.+ session := sessions.GetLatestByFinding("finding-1")+ session.Action = tc.action+ sessions.Update(session)+ if tc.fail != "" {+ sessions.Fail(session.ID, tc.fail)+ sessions.SetOutcome(session.ID, tc.outcome)+ } else {+ sessions.Complete(session.ID, tc.outcome, "The backup service exits on a stale mount.", nil)+ }+ if !finishOrchestratorRun(orchestratorFindings, run, tc.status, tc.outcome) {+ t.Fatal("the run's close was refused")+ }++ record := findings.Get("finding-1").InvestigationRecord+ if record == nil || record.Status != tc.status || record.Outcome != tc.outcome || record.CompletedAt == nil || record.Error != tc.fail {+ t.Fatalf("record does not show the closed run: %+v", record)+ }+ if tc.action != nil && (record.Action == nil || record.Action.ActionID != tc.action.ActionID) {+ t.Fatalf("record lost the planned action: %+v", record.Action)+ }+ if len(sent.published) != 1 || sent.published[0].InvestigationOutcome != string(tc.outcome) {+ t.Fatalf("published %d findings, want the closed run once: %+v", len(sent.published), sent.published)+ }+ switch {+ case tc.approval && (len(sent.pushes) != 1 || sent.pushes[0].Type != relay.PushTypeApprovalRequest || sent.pushes[0].ActionID != tc.action.ActionID):+ t.Fatalf("pushes = %+v, want one approval request for %s", sent.pushes, tc.action.ActionID)+ case !tc.approval && len(sent.pushes) != 0:+ t.Fatalf("pushes = %+v, want none", sent.pushes)+ }+ })+ }+}++// The orchestrator writes a run's action back as it was planned. When the plan+// moved on while the final model call was pending, here by expiring, the close+// takes the action from the action audit: the record shows the expired plan and+// nobody is asked to approve it.+func TestOrchestratorRunCloseTakesTheActionFromTheAudit(t *testing.T) {+ audits := unifiedresources.NewMemoryStore()+ orchestratorFindings, sessions, patrol := orchestratorPatrolFor(t, audits)+ sent := watchRunFinalization(patrol)+ findings := patrol.GetFindings()+ findings.Add(investigatedFinding("finding-1"))+ run := startOrchestratorRun(t, orchestratorFindings, sessions, "finding-1", "chat-1")++ session := sessions.GetLatestByFinding("finding-1")+ now := time.Now().UTC()+ if _, _, err := audits.CreateActionAudit(unifiedresources.ActionAuditRecord{+ ID: "act-1", CreatedAt: now, UpdatedAt: now, State: unifiedresources.ActionStateExpired,+ Request: unifiedresources.ActionRequest{RequestID: "proposal-1", ResourceID: "vm-finding-1", CapabilityName: "restart", RequestedBy: "pulse_patrol"},+ Plan: unifiedresources.ActionPlan{ActionID: "act-1", RequestID: "proposal-1", Allowed: true},+ Origin: &unifiedresources.ActionOrigin{Surface: patrolActionOriginSurface, FindingID: "finding-1", InvestigationID: session.ID, ProposalID: "proposal-1"},+ }, nil); err != nil {+ t.Fatal(err)+ }+ session.Action = &aicontracts.ActionReference{ActionID: "act-1", State: "pending_approval"}+ sessions.Update(session)+ sessions.Fail(session.ID, "investigation final response failed: context deadline exceeded")+ sessions.SetOutcome(session.ID, aicontracts.OutcomeFixQueued)+ if !finishOrchestratorRun(orchestratorFindings, run, aicontracts.InvestigationStatusFailed, aicontracts.OutcomeFixQueued) {+ t.Fatal("the run's close was refused")+ }++ record := findings.Get("finding-1").InvestigationRecord+ if record == nil || record.Action == nil || record.Action.State != string(unifiedresources.ActionStateExpired) || record.Outcome != aicontracts.OutcomeNeedsAttention {+ t.Fatalf("record kept the plan as written back: %+v", record)+ }+ if got := sessions.GetLatestByFinding("finding-1"); got.Action == nil || got.Action.State != string(unifiedresources.ActionStateExpired) {+ t.Fatalf("session kept the plan as written back: %+v", got.Action)+ }+ for _, push := range sent.pushes {+ if push.Type == relay.PushTypeApprovalRequest {+ t.Fatalf("an expired plan asked for approval: %+v", sent.pushes)+ }+ }+}++// A close that does not land, because a newer run replaced the one reporting+// back, finalizes nothing.+func TestOrchestratorRunCloseThatDoesNotLandIsNotFinalized(t *testing.T) {+ orchestratorFindings, sessions, patrol := orchestratorPatrolFor(t, nil)+ sent := watchRunFinalization(patrol)+ findings := patrol.GetFindings()+ findings.Add(investigatedFinding("finding-1"))+ runA := startOrchestratorRun(t, orchestratorFindings, sessions, "finding-1", "chat-a")+ findings.RecoverStuckInvestigation("finding-1", "chat-a", time.Time{}, ai.InvestigationStatusFailed, ai.InvestigationOutcomeTimedOut, time.Now())+ startOrchestratorRun(t, orchestratorFindings, sessions, "finding-1", "chat-b")++ if finishOrchestratorRun(orchestratorFindings, runA, aicontracts.InvestigationStatusFailed, aicontracts.OutcomeFixQueued) {+ t.Fatal("an earlier run overwrote the run that replaced it")+ }+ if len(sent.published) != 0 || len(sent.pushes) != 0 || findings.Get("finding-1").InvestigationRecord != nil {+ t.Fatalf("a close that did not land was finalized: %+v", sent)+ }+}diff --git a/internal/api/patrol_action_expiry_sweep_test.go b/internal/api/patrol_action_expiry_sweep_test.gonew file mode 100644index 0000000000..33d50d183e--- /dev/null+++ b/internal/api/patrol_action_expiry_sweep_test.go@@ -0,0 +1,140 @@+package api++import (+ "net/http"+ "net/http/httptest"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/ai"+ "github.com/rcourtman/pulse-go-rewrite/internal/config"+ unified "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"+ "github.com/rcourtman/pulse-go-rewrite/pkg/aicontracts"+)++// patrolPlanAudit is a pending Patrol plan for an investigation, planned at+// planned and expiring at expires.+func patrolPlanAudit(id, investigationID string, planned, expires time.Time) unified.ActionAuditRecord {+ return unified.ActionAuditRecord{+ ID: id,+ CreatedAt: planned,+ UpdatedAt: planned,+ State: unified.ActionStatePending,+ Request: boundActionTestRequest("proposal-"+id, "vm:42", "restart", "Clear the stuck mount", "pulse_patrol"),+ Plan: unified.ActionPlan{+ ActionID: id,+ RequestID: "proposal-" + id,+ Allowed: true,+ RequiresApproval: true,+ ApprovalPolicy: unified.ApprovalAdmin,+ ApprovalRequirement: unified.ApprovalRequirementForFloor(unified.ApprovalAdmin),+ PlannedAt: planned,+ ExpiresAt: expires,+ ResourceVersion: "resource:sha256:test",+ PolicyVersion: "policy:sha256:test",+ PlanHash: "sha256:" + id,+ },+ Origin: &unified.ActionOrigin{Surface: patrolActionOriginSurface, FindingID: "finding-1", InvestigationID: investigationID, ProposalID: "proposal-" + id},+ }+}++// patrolSweepFixture is a finding whose run failed after planning a fix, with+// Patrol reconciling the action handlers' transitions.+type patrolSweepFixture struct {+ handlers *ResourceHandlers+ store unified.ResourceStore+ findings *ai.FindingsStore+ investigations *testInvestigationStore+ investigation *aicontracts.InvestigationSession+}++func newPatrolSweepFixture(t *testing.T, currentActionID string) patrolSweepFixture {+ t.Helper()+ h := newActionTestResourceHandlers(t, &config.Config{DataPath: t.TempDir()})+ store, err := h.getStore("default")+ if err != nil {+ t.Fatalf("get store: %v", err)+ }+ svc := ai.NewService(nil, nil)+ svc.SetStateProvider(&MockStateProvider{})+ findings := svc.GetPatrolService().GetFindings()+ findings.Add(&ai.Finding{ID: "finding-1", ResourceID: "vm:42", Title: "Unhealthy service", Severity: ai.FindingSeverityWarning,+ InvestigationSessionID: "session-1", InvestigationStatus: string(aicontracts.InvestigationStatusFailed),+ InvestigationOutcome: string(aicontracts.OutcomeFixQueued)})+ investigations := newTestInvestigationStore()+ investigation := investigations.Create("finding-1", "session-1")+ investigation.Status = aicontracts.InvestigationStatusFailed+ investigation.Outcome = aicontracts.OutcomeFixQueued+ investigation.Action = &aicontracts.ActionReference{ActionID: currentActionID, State: string(unified.ActionStatePending)}+ investigations.Update(investigation)++ patrolHandler := &AISettingsHandler{defaultAIService: svc,+ investigationStores: map[string]aicontracts.InvestigationStore{"default": investigations},+ resourceStoreProvider: func(string) (unified.ResourceStore, error) { return store, nil },+ }+ h.SetActionTransitionPublisher(patrolHandler.ReconcilePatrolActionTransition)+ return patrolSweepFixture{handlers: h, store: store, findings: findings, investigations: investigations, investigation: investigation}+}++func (f patrolSweepFixture) readInbox(t *testing.T) {+ t.Helper()+ rec := httptest.NewRecorder()+ req := httptest.NewRequest(http.MethodGet, "/api/actions?view=pending", nil)+ f.handlers.HandleListActions(rec, actionHandlerTestRequest(req, ""))+ if rec.Code != http.StatusOK {+ t.Fatalf("inbox status = %d, body=%s", rec.Code, rec.Body.String())+ }+}++// Reading the action inbox sweeps plans past their expiry. A Patrol fix whose+// plan expires in that sweep moves its investigation and finding off the+// queued fix, as the action's own expiry does, rather than leaving them queued+// until someone opens the investigation.+func TestActionInboxSweepReconcilesAnExpiredPatrolPlan(t *testing.T) {+ now := time.Now().UTC().Truncate(time.Second)+ f := newPatrolSweepFixture(t, "act-1")+ if err := f.store.RecordActionAudit(patrolPlanAudit("act-1", f.investigation.ID, now.Add(-10*time.Minute), now.Add(-5*time.Minute))); err != nil {+ t.Fatalf("RecordActionAudit: %v", err)+ }++ f.readInbox(t)++ if got := f.investigations.Get(f.investigation.ID); got.Outcome != aicontracts.OutcomeNeedsAttention || got.Action == nil || got.Action.State != string(unified.ActionStateExpired) {+ t.Fatalf("investigation kept the expired plan queued: outcome=%q action=%+v", got.Outcome, got.Action)+ }+ if got := f.findings.Get("finding-1"); got.InvestigationOutcome != string(aicontracts.OutcomeNeedsAttention) {+ t.Fatalf("finding kept the expired plan queued: outcome=%q", got.InvestigationOutcome)+ }+}++// A refresh plans a replacement under the same investigation and leaves the+// replaced plan to expire on its own. That plan's swept expiry must not take+// the investigation back from its replacement, which still waits for approval.+// The refresh lineage decides, not the clock: here the replacement's planning+// time reads earlier, as after the clock was set back.+func TestActionInboxSweepLeavesAReplacementPlanInPlace(t *testing.T) {+ now := time.Now().UTC().Truncate(time.Second)+ f := newPatrolSweepFixture(t, "act-2")+ if err := f.store.RecordActionAudit(patrolPlanAudit("act-1", f.investigation.ID, now.Add(-10*time.Minute), now.Add(-5*time.Minute))); err != nil {+ t.Fatalf("RecordActionAudit(replaced): %v", err)+ }+ replacement := patrolPlanAudit("act-2", f.investigation.ID, now.Add(-20*time.Minute), now.Add(time.Hour))+ replacement.Request.RequestID = "refresh:act-1"+ replacement.Plan.RequestID = "refresh:act-1"+ replacement.Origin.ProposalID = "proposal-act-1"+ if err := f.store.RecordActionAudit(replacement); err != nil {+ t.Fatalf("RecordActionAudit(replacement): %v", err)+ }++ f.readInbox(t)++ if replaced, found, err := f.store.GetActionAudit("act-1"); err != nil || !found || replaced.State != unified.ActionStateExpired {+ t.Fatalf("the replaced plan did not expire in the sweep: %+v found=%v err=%v", replaced, found, err)+ }+ if got := f.investigations.Get(f.investigation.ID); got.Outcome != aicontracts.OutcomeFixQueued || got.Action == nil || got.Action.ActionID != "act-2" {+ t.Fatalf("the replaced plan's expiry took the investigation: outcome=%q action=%+v", got.Outcome, got.Action)+ }+ if got := f.findings.Get("finding-1"); got.InvestigationOutcome != string(aicontracts.OutcomeFixQueued) {+ t.Fatalf("the replaced plan's expiry moved the finding: outcome=%q", got.InvestigationOutcome)+ }+}diff --git a/internal/api/patrol_action_reconciliation.go b/internal/api/patrol_action_reconciliation.goindex e38ea247b2..76a055d612 100644--- a/internal/api/patrol_action_reconciliation.go+++ b/internal/api/patrol_action_reconciliation.go@@ -6,6 +6,7 @@ import ("reflect""strings"+ "github.com/rcourtman/pulse-go-rewrite/internal/actionlifecycle""github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources""github.com/rcourtman/pulse-go-rewrite/pkg/aicontracts""github.com/rs/zerolog/log"@@ -90,6 +91,9 @@ func (h *AISettingsHandler) applyPatrolActionAudit(orgID string, audit unifiedreif investigation == nil || strings.TrimSpace(investigation.FindingID) != origin.FindingID {return}+ if h.patrolActionSuperseded(orgID, investigation.Action, audit) {+ return+ }reference := actionReferenceFromAudit(audit)changed := falseif !reflect.DeepEqual(investigation.Action, reference) {@@ -113,6 +117,40 @@ func (h *AISettingsHandler) applyPatrolActionAudit(orgID string, audit unifiedre}}+// maxPatrolActionRefreshLineage bounds the walk back through refreshes.+const maxPatrolActionRefreshLineage = 32++// patrolActionSuperseded reports whether audit is for a plan that the+// investigation's current action replaced through one or more refreshes. A+// refresh plans the replacement under the same origin and leaves the replaced+// plan to expire on its own; that plan's later transitions, such as its+// expiry, must not take the investigation back from its replacement.+func (h *AISettingsHandler) patrolActionSuperseded(orgID string, current *aicontracts.ActionReference, audit unifiedresources.ActionAuditRecord) bool {+ if current == nil || strings.TrimSpace(current.ActionID) == "" || current.ActionID == audit.ID {+ return false+ }+ store, err := h.actionAuditStore(orgID)+ if err != nil {+ return false+ }+ actionID := current.ActionID+ for range maxPatrolActionRefreshLineage {+ record, found, err := store.GetActionAudit(actionID)+ if err != nil || !found {+ return false+ }+ replaced, ok := actionlifecycle.RefreshedActionID(record)+ if !ok {+ return false+ }+ if replaced == audit.ID {+ return true+ }+ actionID = replaced+ }+ return false+}+func (h *AISettingsHandler) actionAuditStore(orgID string) (unifiedresources.ResourceStore, error) {h.stateMu.RLock()provider := h.resourceStoreProvider