Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Finalize Patrol runs when they close, failed runs included

rcourtman45b5a8cde821
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 7a591c9248..fabcbb890c 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -10204,3 +10204,9 @@ The `internal/api` change of 2026-10-23 removes browser-only chat routes
`/redo`, `/steer`, `/diff`, `/revert`, and `/unrevert` session sub-resources). The adjacent-boundary notes above that name those routes are now
history; none of them was agent-lifecycle state. No agent enrollment,
command, update, identity, or credential path changes.
+
+The `internal/api` change of 2026-10-23 that finalizes Patrol investigation
+runs when they close, failed runs included, and publishes the action plans the
+inbox expiry sweep expires, enrols, commands, updates or removes no agent and
+touches no agent identity or credential. Investigations still reach hosts only
+through the read-only evidence tools and the governed action lifecycle.
diff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.md
index 67a49aab94..8dacbc3be6 100644
--- a/docs/release-control/v6/internal/subsystems/ai-runtime.md
+++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md
@@ -7590,3 +7590,48 @@ affordances for an undeclared Assistant surface in
`frontend-modern/src/api/agentCapabilities.ts` no longer include interactive
questions, matching `DefaultSurfaceAffordancesForID`. Surface IDs, labels,
tools, and approval boundaries are unchanged.
+
+### Investigation runs are finalized when they close (23 October 2026)
+
+A Patrol investigation run is finalized when it closes, however it ended and
+whoever started it: when the private orchestrator's write that closes the run
+lands on the finding through `orchestratorFindingsAdapter.Update`, or when the
+stuck-run sweep recovers a run whose session has already closed, such as one a
+restart interrupted. Core first reconciles the run's action from the action
+audit (`hydratePatrolInvestigationAction`), then calls
+`PatrolService.FinalizeInvestigationRun`. That rebuilds the durable
+investigation record from the run's final session, publishes the finding to the
+unified store, resolving it there when a verified fix resolved it, and asks the
+operator to approve a planned fix that still awaits approval, once per action.
+It finalizes only a run the finding still names, checked again after the record
+is written. An action's outcome, such as a verified fix, is pushed once, when
+its transition is reconciled (`PublishFindingLifecycleUpdate`); a close no
+longer sends a second fix-completed push.
+
+Before this, only an automatic investigation that returned without an error
+was finalized, by the goroutine that started it, which now only logs a run's
+error. A run that planned a fix and then failed, normally on its final model
+call, kept the record core projected when the plan was recorded, which showed
+the run as still running, and the operator was never asked to approve the
+queued fix. A manual re-investigation was never finalized, even when it
+succeeded. The action is reconciled first because the orchestrator writes it
+back as it was planned, which an approval, rejection or expiry during the final
+model call may have overtaken. For the same reason `CompleteInvestigation` no
+longer moves a finding back to `fix_queued` from an outcome the plan's action
+reached during the run: a run starts with no outcome, so one present at its
+close came from the action lifecycle. Proofs:
+`internal/ai/patrol_investigation_finalize_test.go` and
+`internal/api/ai_handlers_investigation_finalize_test.go`.
+
+The action inbox and decision queue expire every plan past its expiry, and now
+publish each one through the lifecycle's transition callback, as a single
+action's expiry already was. `ReconcilePatrolActionTransition` therefore moves a
+Patrol investigation and its finding from `fix_queued` to `needs_attention` when
+the plan expires in a sweep, rather than when someone next opens the
+investigation. Reconciliation ignores a transition for a plan that the
+investigation's current action replaced through one or more refreshes, which it
+follows through `actionlifecycle.RefreshedActionID` rather than the clock: a
+refresh plans a replacement under the same investigation and leaves the
+replaced plan to expire on its own, and that expiry must not take the
+investigation back from its replacement. Proof:
+`internal/api/patrol_action_expiry_sweep_test.go`.
diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md
index 2c2a21facc..fd7762b031 100644
--- a/docs/release-control/v6/internal/subsystems/api-contracts.md
+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md
@@ -12519,3 +12519,29 @@ affordance fields keep their shape. The browser normalizer's fallback for an
undeclared Assistant surface (`frontend-modern/src/api/agentCapabilities.ts`)
now matches the backend default of tools only, so the External agents settings
panel never shows an "Interactive questions" badge.
+
+### Closed runs are finalized and swept expiries published (23 October 2026)
+
+`orchestratorFindingsAdapter.Update` finalizes a run when the write that closes
+it lands, after reconciling the run's action from the action audit. The record,
+the unified finding and the approval request that only a successful automatic
+run got before now follow every closed run, failed or successful, automatic or
+manual, and a run the stuck-run sweep recovers once its session has closed (see
+the AI runtime contract). An action's outcome keeps one push, from transition
+reconciliation, and an approval request is sent once per action. A close that
+reports a planned fix as queued no longer moves the finding back from an
+outcome the action reached during the run. The adapter takes the run's session
+from the orchestrator's session store, so the Pulse–enterprise boundary is
+unchanged: no `aicontracts` type, field or method is added. The
+transport-independent action lifecycle's `List` and `DecisionQueue` now publish
+each plan their expiry sweep expires through `OnActionTransition`, including
+plans expired before a sweep fails, as `materializeExpiry` already did for one
+action. `actionlifecycle.RefreshedActionID` names the action a refresh's
+replacement replaced, from the `refresh:` request ID `Refresh` already gives
+it, and Patrol reconciliation ignores a transition for a plan that the
+investigation's current action replaced. No route, payload or action state
+changes. Proofs: `internal/actionlifecycle/service_test.go`
+(`TestExpirySweepsPublishEachExpiredPlan`,
+`TestRefreshedActionIDNamesTheReplacedAction`),
+`internal/api/patrol_action_expiry_sweep_test.go` and
+`internal/api/ai_handlers_investigation_finalize_test.go`.
diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md
index 2b5ad3347c..4c18b15efa 100644
--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md
+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md
@@ -7193,3 +7193,8 @@ name those routes are now history. Persisted chat session files no longer
carry `turn_redo_stack` or the `steered` message flag; older files that still
have them load unchanged and drop them on the next write. No backup, restore,
or recovery path changes.
+
+The `internal/api` change of 2026-10-23 that finalizes Patrol investigation
+runs when they close and publishes the action plans the inbox expiry sweep
+expires opens no storage, backup or recovery path. Recovery evidence, restore
+rules and storage capabilities are unchanged.
diff --git a/internal/actionlifecycle/service.go b/internal/actionlifecycle/service.go
index a6ca0de5f6..b7b470fe3a 100644
--- a/internal/actionlifecycle/service.go
+++ b/internal/actionlifecycle/service.go
@@ -557,6 +557,17 @@ func (s *Service) PlanWithOptions(ctx context.Context, orgID string, req unified
return plan, nil
}
+// refreshRequestPrefix starts the request ID a refresh plans its replacement
+// under; the replaced action's ID follows it.
+const refreshRequestPrefix = "refresh:"
+
+// RefreshedActionID returns the action that a refresh replaced with record,
+// when record is such a replacement.
+func RefreshedActionID(record unified.ActionAuditRecord) (string, bool) {
+ replaced, ok := strings.CutPrefix(strings.TrimSpace(record.Request.RequestID), refreshRequestPrefix)
+ return replaced, ok && replaced != ""
+}
+
// Refresh creates a fresh immutable plan for an expired or drifted action.
// The previous record remains historical (and an unexpired drifted record
// naturally expires at its original short TTL); callers always receive the
@@ -583,7 +594,7 @@ func (s *Service) Refresh(ctx context.Context, orgID, actionID string, actor uni
if !refreshable {
return unified.ActionAuditRecord{}, ErrActionRefreshNotAllowed
}
- requestID := "refresh:" + previous.ID
+ requestID := refreshRequestPrefix + previous.ID
req := previous.Request
req.RequestID = requestID
req.RequestedBy = ""
@@ -667,8 +678,8 @@ func (s *Service) List(orgID string, view ActionListView, limit int) ([]unified.
if err != nil {
return nil, err
}
- if _, err := store.ExpireActionAudits(s.now(), 500); err != nil {
- return nil, &PersistError{Op: "action expiry sweep", Err: err}
+ if err := s.sweepExpiry(orgID, store); err != nil {
+ return nil, err
}
var states []unified.ActionState
switch view {
@@ -693,8 +704,8 @@ func (s *Service) DecisionQueue(orgID string, limit int) ([]unified.ActionAuditR
if err != nil {
return nil, err
}
- if _, err := store.ExpireActionAudits(s.now(), 500); err != nil {
- return nil, &PersistError{Op: "action expiry sweep", Err: err}
+ if err := s.sweepExpiry(orgID, store); err != nil {
+ return nil, err
}
records, err := store.GetPendingActionAudits(limit)
if err != nil {
@@ -703,6 +714,21 @@ func (s *Service) DecisionQueue(orgID string, limit int) ([]unified.ActionAuditR
return records, nil
}
+// sweepExpiry materializes every plan past its expiry and publishes each one,
+// as a single action's expiry is published, so surfaces that project actions,
+// such as the queued fix on a Patrol finding, learn of it. Plans the sweep
+// expired before failing are published too.
+func (s *Service) sweepExpiry(orgID string, store Store) error {
+ expired, err := store.ExpireActionAudits(s.now(), 500)
+ for _, record := range expired {
+ s.publishTransition(orgID, record)
+ }
+ if err != nil {
+ return &PersistError{Op: "action expiry sweep", Err: err}
+ }
+ return nil
+}
+
// ResourceOperatorState returns the authoritative per-resource policy used by
// execution and Patrol policy authorization.
func (s *Service) ResourceOperatorState(orgID, resourceID string) (unified.ResourceOperatorState, bool, error) {
diff --git a/internal/actionlifecycle/service_test.go b/internal/actionlifecycle/service_test.go
index 168ef3fab1..e0aa5caf3a 100644
--- a/internal/actionlifecycle/service_test.go
+++ b/internal/actionlifecycle/service_test.go
@@ -2123,3 +2123,81 @@ func TestIssue1649ForceFailRefusesTerminalAndUndispatchedActions(t *testing.T) {
t.Fatalf("missing action ForceFail error=%v", err)
}
}
+
+// A refresh's replacement names the action it replaced, which is how a
+// surface that projects actions tells the replaced plan's later transitions
+// from its replacement's.
+func TestRefreshedActionIDNamesTheReplacedAction(t *testing.T) {
+ now := time.Date(2026, 10, 23, 18, 0, 0, 0, time.UTC)
+ store := unified.NewMemoryStore()
+ service := serviceForStore(t, store, testResource(now, unified.ApprovalAdmin), &stubExecutor{})
+ service.Now = func() time.Time { return now }
+ plan, err := service.Plan(context.Background(), "default", restartRequest(), testActionActor("requester", "default"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ service.Now = func() time.Time { return plan.ExpiresAt.Add(time.Second) }
+ replacement, err := service.Refresh(context.Background(), "default", plan.ActionID, testActionActor("operator", "default"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if replaced, ok := RefreshedActionID(replacement); !ok || replaced != plan.ActionID {
+ t.Fatalf("RefreshedActionID(replacement) = %q, %v, want %q", replaced, ok, plan.ActionID)
+ }
+ original, found, err := store.GetActionAudit(plan.ActionID)
+ if err != nil || !found {
+ t.Fatalf("original: found=%v err=%v", found, err)
+ }
+ if replaced, ok := RefreshedActionID(original); ok {
+ t.Fatalf("a first plan names a replaced action: %q", replaced)
+ }
+}
+
+// The inbox and decision-queue sweeps publish each plan they expire, as a
+// single action's expiry is published, so a surface that projects the action,
+// such as a Patrol finding's queued fix, does not outlive the plan. Each expiry
+// is published once.
+func TestExpirySweepsPublishEachExpiredPlan(t *testing.T) {
+ sweeps := map[string]func(*Service) error{
+ "inbox": func(s *Service) error {
+ _, err := s.List("default", ActionListPending, 100)
+ return err
+ },
+ "decision queue": func(s *Service) error {
+ _, err := s.DecisionQueue("default", 100)
+ return err
+ },
+ }
+ for name, sweep := range sweeps {
+ t.Run(name, func(t *testing.T) {
+ now := time.Date(2026, 10, 23, 18, 0, 0, 0, time.UTC)
+ store := unified.NewMemoryStore()
+ service := serviceForStore(t, store, testResource(now, unified.ApprovalAdmin), &stubExecutor{})
+ service.Now = func() time.Time { return now }
+ plan, err := service.Plan(context.Background(), "default", restartRequest(), testActionActor("requester", "default"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ var published []unified.ActionAuditRecord
+ var orgs []string
+ service.OnActionTransition = func(orgID string, record unified.ActionAuditRecord) {
+ published = append(published, record)
+ orgs = append(orgs, orgID)
+ }
+
+ if err := sweep(service); err != nil || len(published) != 0 {
+ t.Fatalf("a sweep before expiry: err=%v published=%+v", err, published)
+ }
+ service.Now = func() time.Time { return plan.ExpiresAt.Add(time.Second) }
+ if err := sweep(service); err != nil {
+ t.Fatal(err)
+ }
+ if len(published) != 1 || published[0].ID != plan.ActionID || published[0].State != unified.ActionStateExpired || orgs[0] != "default" {
+ t.Fatalf("published = %+v for orgs %v, want one expiry of %s", published, orgs, plan.ActionID)
+ }
+ if err := sweep(service); err != nil || len(published) != 1 {
+ t.Fatalf("a later sweep published the expiry again: err=%v published=%+v", err, published)
+ }
+ })
+ }
+}
diff --git a/internal/ai/findings.go b/internal/ai/findings.go
index c01b5c7ded..b644eb0f03 100644
--- a/internal/ai/findings.go
+++ b/internal/ai/findings.go
@@ -2728,7 +2728,10 @@ func (s *FindingsStore) StartInvestigation(id, sessionID string, runStartedAt ti
// cannot prove ownership once this process has seen an occurrence begin. A run
// whose result does not count, but which the finding still shows as running,
// is closed now through RecoverStuckInvestigation, as the stuck-run sweep would
-// later close it, rather than left running. It reports whether the run's
+// later close it, rather than left running. A run that reports its planned fix
+// as queued does not move the finding back from an outcome the fix's action
+// has reached since: a run starts with no outcome, so one already present came
+// from the action lifecycle during this run. It reports whether the run's
// result was recorded.
func (s *FindingsStore) CompleteInvestigation(id, sessionID string, runStartedAt time.Time, status, outcome string, lastInvestigatedAt *time.Time, attempts int) bool {
s.mu.Lock()
@@ -2746,6 +2749,9 @@ func (s *FindingsStore) CompleteInvestigation(id, sessionID string, runStartedAt
s.RecoverStuckInvestigation(id, sessionID, runStartedAt, InvestigationStatusFailed, InvestigationOutcomeInterrupted, closedAt)
return false
}
+ if outcome == string(InvestigationOutcomeFixQueued) && f.InvestigationOutcome != "" {
+ outcome = f.InvestigationOutcome
+ }
s.updateInvestigationLocked(f, sessionID, status, outcome, lastInvestigatedAt, attempts)
s.mu.Unlock()
s.scheduleSave()
diff --git a/internal/ai/patrol.go b/internal/ai/patrol.go
index d053975a8f..09bdc6baa3 100644
--- a/internal/ai/patrol.go
+++ b/internal/ai/patrol.go
@@ -551,6 +551,11 @@ type PatrolService struct {
// Investigation orchestrator for autonomous investigation of findings
investigationOrchestrator InvestigationOrchestrator
investigationWg sync.WaitGroup // Tracks in-flight investigation goroutines
+ // investigationRunCloser finalizes a closed run; see SetInvestigationRunCloser.
+ investigationRunCloser func(findingID string, session *InvestigationSession)
+ // approvalRequests holds when each action's approval was last requested,
+ // so a run finalized twice does not ask twice; see claimApprovalRequest.
+ approvalRequests map[string]time.Time
// Unified findings callback - pushes findings to unified store
unifiedFindingCallback UnifiedFindingCallback
diff --git a/internal/ai/patrol_findings.go b/internal/ai/patrol_findings.go
index adad51a674..ea79755cf7 100644
--- a/internal/ai/patrol_findings.go
+++ b/internal/ai/patrol_findings.go
@@ -1594,6 +1594,12 @@ func (p *PatrolService) recoverStuckInvestigations() {
Str("resource", f.ResourceName).
Str("outcome", string(result.outcome)).
Msg("AI Patrol: Recovered stuck investigation")
+ // A run whose session closed without its close reaching the finding,
+ // such as one a restart interrupted after it planned a fix, is
+ // finalized here. A run still going finalizes when its close lands.
+ if result.session != nil {
+ p.closeInvestigationRun(f.ID, result.session)
+ }
}
if recovered > 0 {
log.Info().Int("recovered", recovered).
@@ -1604,11 +1610,13 @@ func (p *PatrolService) recoverStuckInvestigations() {
// stuckInvestigation is how to close a run that is no longer live.
// runStartedAt is the start of the run's session, zero when the orchestrator
// holds none, so the store can check the run belongs to the finding's current
-// occurrence before it applies the session's result.
+// occurrence before it applies the session's result. session is the run's
+// session once it has closed, which finalizes the run after recovery.
type stuckInvestigation struct {
status InvestigationStatus
outcome InvestigationOutcome
runStartedAt time.Time
+ session *InvestigationSession
}
// stuckInvestigationResult reports whether a finding marked as running has a
@@ -1647,6 +1655,7 @@ func stuckInvestigationResult(f *Finding, orchestrator InvestigationOrchestrator
}
return closed, true
}
+ closed.session = session
if session.Outcome == aicontracts.OutcomeInterrupted {
// Closed when the orchestrator restarted, or cancelled by shutdown.
closed.outcome = InvestigationOutcomeInterrupted
@@ -1880,57 +1889,15 @@ func (p *PatrolService) maybeInvestigateFinding(f *Finding) bool {
// cutting longer configured or subscription-agent runs short.
ctx, cancel := context.WithTimeout(context.Background(), currentCfg.GetPatrolInvestigationTimeout()+investigationRunGrace)
defer cancel()
+ // The run is finalized when it closes, successfully or not, through
+ // the orchestrator's findings store (FinalizeInvestigationRun), not
+ // here: a failed run that planned a fix still needs its record, its
+ // unified finding and the approval push.
if err := orchestrator.InvestigateFinding(ctx, invFinding, currentAutonomy); err != nil {
log.Error().
Err(err).
Str("finding_id", f.ID).
- Msg("Failed to start investigation")
- return
- }
-
- // The orchestrator updates the patrol findings store; sync the latest state to the unified store.
- // This makes fix verification and resolution visible as an actual closed loop in the UI.
- var pushUnified UnifiedFindingCallback
- var resolveUnified func(string)
- var pushCb PushNotifyCallback
- p.mu.RLock()
- pushUnified = p.unifiedFindingCallback
- resolveUnified = p.unifiedFindingResolver
- pushCb = p.pushNotifyCallback
- p.mu.RUnlock()
- if latest := p.findings.Get(f.ID); latest != nil {
- var latestInvestigation *InvestigationSession
- if orchestrator != nil {
- latestInvestigation = orchestrator.GetInvestigationByFinding(latest.ID)
- }
- if p.storeFindingInvestigationRecord(latest.ID, latestInvestigation, false) {
- if refreshed := p.findings.Get(latest.ID); refreshed != nil {
- latest = refreshed
- }
- }
- if pushUnified != nil {
- pushUnified(latest)
- }
- if latest.ResolvedAt != nil && resolveUnified != nil {
- resolveUnified(latest.ID)
- }
-
- // Send push notifications for investigation outcomes
- if pushCb != nil {
- switch latest.InvestigationOutcome {
- case string(InvestigationOutcomeFixQueued):
- if latestInvestigation != nil && latestInvestigation.Action != nil && latestInvestigation.Action.State == "pending_approval" {
- pushCb(relay.NewActionDecisionNotification(
- latestInvestigation.Action.ActionID,
- latest.Title,
- ))
- }
- case string(InvestigationOutcomeFixExecuted), string(InvestigationOutcomeFixVerified):
- pushCb(relay.NewFixCompletedNotification(latest.ID, latest.Title, true))
- case string(InvestigationOutcomeFixFailed), string(InvestigationOutcomeFixVerificationFailed):
- pushCb(relay.NewFixCompletedNotification(latest.ID, latest.Title, false))
- }
- }
+ Msg("Investigation ended with an error")
}
// Typed remediation now flows exclusively through the action
@@ -1948,6 +1915,96 @@ func (p *PatrolService) maybeInvestigateFinding(f *Finding) bool {
return true
}
+// SetInvestigationRunCloser sets how a closed run is finalized. Pulse's API
+// layer reconciles the run's action from the action audit, then calls
+// FinalizeInvestigationRun. Without a closer a run is finalized as it is.
+func (p *PatrolService) SetInvestigationRunCloser(closer func(findingID string, session *InvestigationSession)) {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ p.investigationRunCloser = closer
+}
+
+func (p *PatrolService) closeInvestigationRun(findingID string, session *InvestigationSession) {
+ p.mu.RLock()
+ closer := p.investigationRunCloser
+ p.mu.RUnlock()
+ if closer != nil {
+ closer(findingID, session)
+ return
+ }
+ p.FinalizeInvestigationRun(findingID, session)
+}
+
+// FinalizeInvestigationRun projects a run that has just closed, successfully or
+// not, onto the finding: it rebuilds the durable record from the run's final
+// session, publishes the unified finding and, when the run left a planned fix
+// awaiting approval, asks the operator to approve it, once per action. It runs
+// when the run's close lands on the finding or the stuck-run sweep recovers a
+// run whose session closed, so a manual re-investigation and a run that failed
+// after planning are finalized like any other run. A run the finding no longer
+// names, because a newer run or occurrence took it over, is not finalized.
+func (p *PatrolService) FinalizeInvestigationRun(findingID string, session *InvestigationSession) {
+ if p == nil || p.findings == nil || session == nil {
+ return
+ }
+ latest := p.findings.Get(findingID)
+ if latest == nil || latest.InvestigationSessionID != session.SessionID || !latest.IsCurrentInvestigation(session) {
+ return
+ }
+ p.storeFindingInvestigationRecord(latest.ID, session, false)
+ // A newer run or occurrence can take the finding while the record is
+ // written, and the store then refuses this run's record.
+ latest = p.findings.Get(findingID)
+ if latest == nil || latest.InvestigationSessionID != session.SessionID || !latest.IsCurrentInvestigation(session) {
+ return
+ }
+ p.mu.RLock()
+ pushUnified := p.unifiedFindingCallback
+ resolveUnified := p.unifiedFindingResolver
+ pushCb := p.pushNotifyCallback
+ p.mu.RUnlock()
+ // Sync the closed run to the unified store, so fix verification and
+ // resolution show as a closed loop in the UI.
+ if pushUnified != nil {
+ pushUnified(latest)
+ }
+ if latest.ResolvedAt != nil && resolveUnified != nil {
+ resolveUnified(latest.ID)
+ }
+ // An action's outcome, such as a verified fix, is pushed once when its
+ // transition is reconciled (PublishFindingLifecycleUpdate), not here.
+ if pushCb != nil && latest.InvestigationOutcome == string(InvestigationOutcomeFixQueued) &&
+ session.Action != nil && session.Action.State == "pending_approval" &&
+ p.claimApprovalRequest(session.Action.ActionID, time.Now()) {
+ pushCb(relay.NewActionDecisionNotification(session.Action.ActionID, latest.Title))
+ }
+}
+
+// approvalRequestMemory is how long an action's approval request is
+// remembered, comfortably past any plan's expiry.
+const approvalRequestMemory = 24 * time.Hour
+
+// claimApprovalRequest reports whether an approval request for actionID may be
+// sent: once per action, so a run finalized again, as when a late close lands
+// after the stuck-run sweep recovered the run, does not ask twice.
+func (p *PatrolService) claimApprovalRequest(actionID string, now time.Time) bool {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ for id, at := range p.approvalRequests {
+ if now.Sub(at) > approvalRequestMemory {
+ delete(p.approvalRequests, id)
+ }
+ }
+ if _, requested := p.approvalRequests[actionID]; requested {
+ return false
+ }
+ if p.approvalRequests == nil {
+ p.approvalRequests = make(map[string]time.Time)
+ }
+ p.approvalRequests[actionID] = now
+ return true
+}
+
// RefreshFindingInvestigationRecord preserves the latest investigation and
// reconciled action in the durable record shared by product surfaces.
func (p *PatrolService) RefreshFindingInvestigationRecord(findingID string, session *InvestigationSession) bool {
diff --git a/internal/ai/patrol_investigation_finalize_test.go b/internal/ai/patrol_investigation_finalize_test.go
new file mode 100644
index 0000000000..096f05a255
--- /dev/null
+++ b/internal/ai/patrol_investigation_finalize_test.go
@@ -0,0 +1,179 @@
+package ai
+
+import (
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/relay"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/aicontracts"
+)
+
+// finalizedRun is a Patrol service whose unified publications and pushes are
+// recorded, with one finding whose run, chat-1, closed with outcome.
+type finalizedRun struct {
+ patrol *PatrolService
+ store *FindingsStore
+ published []*Finding
+ pushes []relay.PushNotificationPayload
+}
+
+func newFinalizedRun(t *testing.T, outcome InvestigationOutcome) *finalizedRun {
+ t.Helper()
+ run := &finalizedRun{store: NewFindingsStore()}
+ run.patrol = &PatrolService{findings: run.store}
+ run.patrol.SetUnifiedFindingCallback(func(f *Finding) bool {
+ run.published = append(run.published, f)
+ return true
+ })
+ run.patrol.SetPushNotifyCallback(func(n relay.PushNotificationPayload) { run.pushes = append(run.pushes, n) })
+ run.store.Add(&Finding{ID: "finding-1", ResourceID: "vm-100", Title: "Backup service is crash-looping", DetectedAt: time.Now(),
+ InvestigationSessionID: "chat-1", InvestigationStatus: string(InvestigationStatusFailed),
+ InvestigationOutcome: string(outcome)})
+ return run
+}
+
+// A run that planned a fix and then failed is finalized like a completed run:
+// its record replaces the projection made when the plan was recorded, the
+// unified finding is published and the queued fix asks for approval.
+func TestFinalizeInvestigationRunFinalizesAFailedRunThatPlannedAFix(t *testing.T) {
+ run := newFinalizedRun(t, InvestigationOutcomeFixQueued)
+
+ // Core projects the plan onto the record while the run is still going.
+ session := &InvestigationSession{ID: "investigation-1", FindingID: "finding-1", SessionID: "chat-1",
+ Status: aicontracts.InvestigationStatusRunning, StartedAt: time.Now(),
+ Action: &aicontracts.ActionReference{ActionID: "action-1", State: "pending_approval"}}
+ if !run.patrol.RefreshFindingInvestigationRecord("finding-1", session) {
+ t.Fatal("the plan's projection was not recorded")
+ }
+
+ // The final model call then fails, and the run closes with the plan's outcome.
+ closed := time.Now()
+ session.Status = aicontracts.InvestigationStatusFailed
+ session.Outcome = aicontracts.OutcomeFixQueued
+ session.CompletedAt = &closed
+ session.Error = "investigation final response failed: context deadline exceeded"
+ run.patrol.FinalizeInvestigationRun("finding-1", session)
+
+ record := run.store.Get("finding-1").InvestigationRecord
+ if record == nil || record.Status != aicontracts.InvestigationStatusFailed || record.Error != session.Error || record.CompletedAt == nil {
+ t.Fatalf("the record kept the projection made while the run was going: %#v", record)
+ }
+ if len(run.published) != 1 || run.published[0].ID != "finding-1" {
+ t.Fatalf("published %d findings, want the closed run once", len(run.published))
+ }
+ if len(run.pushes) != 1 || run.pushes[0].Type != relay.PushTypeApprovalRequest || run.pushes[0].ActionID != "action-1" {
+ t.Fatalf("pushes = %+v, want one approval request for action-1", run.pushes)
+ }
+}
+
+// A run the finding no longer names, because a newer run took it over after
+// this one's close landed, is not finalized: its action would otherwise be
+// offered for approval under the newer run's outcome.
+func TestFinalizeInvestigationRunSkipsARunTheFindingNoLongerNames(t *testing.T) {
+ run := newFinalizedRun(t, InvestigationOutcomeFixQueued)
+ closed := time.Now()
+ older := &InvestigationSession{ID: "investigation-0", FindingID: "finding-1", SessionID: "chat-0",
+ Status: aicontracts.InvestigationStatusFailed, Outcome: aicontracts.OutcomeFixQueued, StartedAt: closed.Add(-time.Hour), CompletedAt: &closed,
+ Action: &aicontracts.ActionReference{ActionID: "action-0", State: "pending_approval"}}
+
+ run.patrol.FinalizeInvestigationRun("finding-1", older)
+
+ if record := run.store.Get("finding-1").InvestigationRecord; record != nil || len(run.published) != 0 || len(run.pushes) != 0 {
+ t.Fatalf("an older run was finalized: record=%#v published=%d pushes=%+v", record, len(run.published), run.pushes)
+ }
+}
+
+// A run finalized twice, as when a late close lands after the stuck-run sweep
+// recovered the run, asks for approval once.
+func TestFinalizeInvestigationRunAsksForApprovalOnce(t *testing.T) {
+ run := newFinalizedRun(t, InvestigationOutcomeFixQueued)
+ closed := time.Now()
+ session := &InvestigationSession{ID: "investigation-1", FindingID: "finding-1", SessionID: "chat-1",
+ Status: aicontracts.InvestigationStatusFailed, Outcome: aicontracts.OutcomeFixQueued, StartedAt: closed.Add(-time.Minute), CompletedAt: &closed,
+ Action: &aicontracts.ActionReference{ActionID: "action-1", State: "pending_approval"}}
+
+ run.patrol.FinalizeInvestigationRun("finding-1", session)
+ run.patrol.FinalizeInvestigationRun("finding-1", session)
+
+ if len(run.pushes) != 1 || run.pushes[0].ActionID != "action-1" {
+ t.Fatalf("pushes = %+v, want one approval request", run.pushes)
+ }
+}
+
+// A run that reports its planned fix as queued does not move the finding back
+// from an outcome the fix's action reached while the run was closing, here a
+// rejection. A run starts with no outcome, so one present at its close came
+// from the action lifecycle.
+func TestCompleteInvestigationKeepsAnOutcomeTheActionReached(t *testing.T) {
+ findings := NewFindingsStore()
+ findings.Add(runningInvestigationFinding("rejected", "chat-rejected", nil))
+ findings.Add(runningInvestigationFinding("unplanned", "chat-unplanned", nil))
+ started := time.Now()
+ rejected := &InvestigationSession{ID: "inv-rejected", FindingID: "rejected", SessionID: "chat-rejected", StartedAt: started}
+ if applied, _ := findings.ReconcileInvestigationOutcome("rejected", rejected, string(InvestigationOutcomeFixRejected)); !applied {
+ t.Fatal("the action's rejection was not reconciled")
+ }
+ closed := time.Now()
+
+ if !findings.CompleteInvestigation("rejected", "chat-rejected", started, string(InvestigationStatusFailed), string(InvestigationOutcomeFixQueued), &closed, 1) {
+ t.Fatal("the run's close was refused")
+ }
+ if !findings.CompleteInvestigation("unplanned", "chat-unplanned", started, string(InvestigationStatusFailed), string(InvestigationOutcomeTimedOut), &closed, 1) {
+ t.Fatal("the run's close was refused")
+ }
+
+ assertInvestigationState(t, findings, "rejected", InvestigationStatusFailed, InvestigationOutcomeFixRejected)
+ assertInvestigationState(t, findings, "unplanned", InvestigationStatusFailed, InvestigationOutcomeTimedOut)
+}
+
+// An action's outcome is pushed when its transition is reconciled
+// (PublishFindingLifecycleUpdate), so a run that closes on a verified fix is
+// published without a second push.
+func TestFinalizeInvestigationRunLeavesActionOutcomePushesToReconciliation(t *testing.T) {
+ run := newFinalizedRun(t, InvestigationOutcomeFixVerified)
+ closed := time.Now()
+ session := &InvestigationSession{ID: "investigation-1", FindingID: "finding-1", SessionID: "chat-1",
+ Status: aicontracts.InvestigationStatusCompleted, Outcome: aicontracts.OutcomeFixVerified, StartedAt: closed.Add(-time.Minute), CompletedAt: &closed,
+ Action: &aicontracts.ActionReference{ActionID: "action-1", State: "completed", VerificationStatus: "verified"}}
+
+ run.patrol.FinalizeInvestigationRun("finding-1", session)
+
+ if len(run.published) != 1 || len(run.pushes) != 0 {
+ t.Fatalf("published=%d pushes=%+v, want one publication and no push", len(run.published), run.pushes)
+ }
+}
+
+// The stuck-run sweep finalizes a run it recovers once the run's session has
+// closed without its close reaching the finding, such as a run a restart
+// interrupted after it planned a fix. A run still going past its limit is not
+// finalized: it finalizes when its own close lands.
+func TestRecoverStuckInvestigationsFinalizesAClosedRun(t *testing.T) {
+ now := time.Now()
+ closedAt := now.Add(-time.Second)
+ findings := NewFindingsStore()
+ interrupted := runningInvestigationFinding("restarted", "chat-restarted", nil)
+ // Core recorded the plan's queued fix while the run was going.
+ interrupted.InvestigationOutcome = string(InvestigationOutcomeFixQueued)
+ findings.Add(interrupted)
+ findings.Add(runningInvestigationFinding("hung", "chat-hung", nil))
+
+ orchestrator := &sessionReportingOrchestrator{sessions: map[string]*InvestigationSession{
+ "restarted": {ID: "inv-restarted", FindingID: "restarted", SessionID: "chat-restarted", Status: aicontracts.InvestigationStatusFailed,
+ Outcome: aicontracts.OutcomeInterrupted, StartedAt: now.Add(-3 * time.Minute), CompletedAt: &closedAt,
+ Action: &aicontracts.ActionReference{ActionID: "action-1", State: "pending_approval"}},
+ "hung": {ID: "inv-hung", FindingID: "hung", SessionID: "chat-hung", Status: aicontracts.InvestigationStatusRunning, StartedAt: now.Add(-time.Hour)},
+ }}
+ patrol := &PatrolService{findings: findings, investigationOrchestrator: orchestrator}
+ var closed []string
+ patrol.SetInvestigationRunCloser(func(findingID string, session *InvestigationSession) {
+ closed = append(closed, findingID+"/"+session.SessionID)
+ })
+
+ patrol.recoverStuckInvestigations()
+
+ assertInvestigationState(t, findings, "restarted", InvestigationStatusCompleted, InvestigationOutcomeFixQueued)
+ assertInvestigationState(t, findings, "hung", InvestigationStatusFailed, InvestigationOutcomeTimedOut)
+ if len(closed) != 1 || closed[0] != "restarted/chat-restarted" {
+ t.Fatalf("closed runs = %v, want only the restarted run's closed session", closed)
+ }
+}
diff --git a/internal/api/ai_handlers.go b/internal/api/ai_handlers.go
index 07abae14b8..bc1a41f1de 100644
--- a/internal/api/ai_handlers.go
+++ b/internal/api/ai_handlers.go
@@ -1932,7 +1932,18 @@ func (h *AISettingsHandler) setupInvestigationOrchestrator(orgID string, svc *ai
log.Warn().Str("orgID", orgID).Msg("Findings store not available for orchestrator")
return
}
- findingsAdapter := &orchestratorFindingsAdapter{store: &findingsStoreWrapper{store: findingsStore}, sessions: store}
+ // A closed run's action is reconciled from the action audit before the run
+ // is finalized: the orchestrator writes the action back as it was planned,
+ // which an approval or rejection during the run may have overtaken.
+ closeRun := func(findingID string, session *aicontracts.InvestigationSession) {
+ patrol.FinalizeInvestigationRun(findingID, h.hydratePatrolInvestigationAction(orgID, session))
+ }
+ patrol.SetInvestigationRunCloser(closeRun)
+ findingsAdapter := &orchestratorFindingsAdapter{
+ store: &findingsStoreWrapper{store: findingsStore},
+ sessions: store,
+ closed: closeRun,
+ }
// Get config for investigation settings
cfg := svc.GetConfig()
@@ -2114,6 +2125,8 @@ type orchestratorFindingsAdapter struct {
// sessions is the orchestrator's session store, which records when each
// run started.
sessions aicontracts.InvestigationStore
+ // closed finalizes a run once its close has landed on the finding.
+ closed func(findingID string, session *aicontracts.InvestigationSession)
}
func (a *orchestratorFindingsAdapter) Get(id string) *aicontracts.Finding {
@@ -2147,7 +2160,9 @@ func (a *orchestratorFindingsAdapter) Get(id string) *aicontracts.Finding {
// a run installs it, unless the run was started for an earlier occurrence of
// the finding. Every other write closes a run and lands only while the finding
// still names that run, so a run that ends after its finding was re-detected,
-// or after a newer run took over, cannot overwrite the newer state.
+// or after a newer run took over, cannot overwrite the newer state. A close
+// that lands finalizes the run, whether it succeeded or failed and whether
+// Patrol or an operator started it.
func (a *orchestratorFindingsAdapter) Update(f *aicontracts.Finding) bool {
if a.store == nil || f == nil {
return false
@@ -2165,7 +2180,7 @@ func (a *orchestratorFindingsAdapter) Update(f *aicontracts.Finding) bool {
f.InvestigationAttempts,
)
}
- return a.store.CompleteInvestigation(
+ if !a.store.CompleteInvestigation(
f.ID,
f.InvestigationSessionID,
runStartedAt,
@@ -2173,20 +2188,37 @@ func (a *orchestratorFindingsAdapter) Update(f *aicontracts.Finding) bool {
f.InvestigationOutcome,
f.LastInvestigatedAt,
f.InvestigationAttempts,
- )
+ ) {
+ return false
+ }
+ if a.closed != nil {
+ if session := a.session(f.ID, f.InvestigationSessionID); session != nil {
+ a.closed(f.ID, session)
+ }
+ }
+ return true
}
-// runStartedAt returns when the orchestrator's session for a run started, or
-// zero when its session store holds no record of the run.
-func (a *orchestratorFindingsAdapter) runStartedAt(findingID, sessionID string) time.Time {
+// session returns the orchestrator's session for a run, or nil when its
+// session store holds no record of the run.
+func (a *orchestratorFindingsAdapter) session(findingID, sessionID string) *aicontracts.InvestigationSession {
if a.sessions == nil {
- return time.Time{}
+ return nil
}
for _, session := range a.sessions.GetByFinding(findingID) {
if session != nil && session.SessionID == sessionID {
- return session.StartedAt
+ return session
}
}
+ return nil
+}
+
+// runStartedAt returns when the orchestrator's session for a run started, or
+// zero when its session store holds no record of the run.
+func (a *orchestratorFindingsAdapter) runStartedAt(findingID, sessionID string) time.Time {
+ if session := a.session(findingID, sessionID); session != nil {
+ return session.StartedAt
+ }
return time.Time{}
}
diff --git a/internal/api/ai_handlers_investigation_additional_test.go b/internal/api/ai_handlers_investigation_additional_test.go
index b29f0054eb..5aa368ce5a 100644
--- a/internal/api/ai_handlers_investigation_additional_test.go
+++ b/internal/api/ai_handlers_investigation_additional_test.go
@@ -944,6 +944,15 @@ func TestSetupInvestigationOrchestratorUsesSubscriptionAgentLimits(t *testing.T)
// returns the findings and session stores it was built with, plus the
// tenant's findings.
func orchestratorFindingsFor(t *testing.T) (aicontracts.OrchestratorFindingsStore, aicontracts.InvestigationStore, *ai.FindingsStore) {
+ t.Helper()
+ findings, sessions, patrol := orchestratorPatrolFor(t, nil)
+ return findings, sessions, patrol.GetFindings()
+}
+
+// orchestratorPatrolFor is orchestratorFindingsFor with the tenant's Patrol
+// service instead of only its findings, and audits as the action audit store
+// when it is not nil.
+func orchestratorPatrolFor(t *testing.T, audits unifiedresources.ResourceStore) (aicontracts.OrchestratorFindingsStore, aicontracts.InvestigationStore, *ai.PatrolService) {
t.Helper()
prevStoreFactory := getCreateInvestigationStore()
SetCreateInvestigationStore(func(dataDir string) aicontracts.InvestigationStore {
@@ -964,6 +973,9 @@ func orchestratorFindingsFor(t *testing.T) (aicontracts.OrchestratorFindingsStor
handler := &AISettingsHandler{
investigationStores: make(map[string]aicontracts.InvestigationStore),
}
+ if audits != nil {
+ handler.resourceStoreProvider = func(string) (unifiedresources.ResourceStore, error) { return audits, nil }
+ }
handler.chatHandler = &AIHandler{
services: map[string]AIService{
"tenant-1": chat.NewService(chat.Config{AIConfig: config.NewDefaultAIConfig()}),
@@ -973,7 +985,7 @@ func orchestratorFindingsFor(t *testing.T) (aicontracts.OrchestratorFindingsStor
if deps.FindingsStore == nil || deps.Store == nil {
t.Fatal("investigation orchestrator was not wired")
}
- return deps.FindingsStore, deps.Store, tenantSvc.GetPatrolService().GetFindings()
+ return deps.FindingsStore, deps.Store, tenantSvc.GetPatrolService()
}
func investigatedFinding(id string) *ai.Finding {
diff --git a/internal/api/ai_handlers_investigation_finalize_test.go b/internal/api/ai_handlers_investigation_finalize_test.go
new file mode 100644
index 0000000000..959f38a091
--- /dev/null
+++ b/internal/api/ai_handlers_investigation_finalize_test.go
@@ -0,0 +1,184 @@
+package api
+
+import (
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/ai"
+ "github.com/rcourtman/pulse-go-rewrite/internal/relay"
+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/aicontracts"
+)
+
+// runFinalization records what finalizing a closed run sent out.
+type runFinalization struct {
+ published []*ai.Finding
+ pushes []relay.PushNotificationPayload
+}
+
+// watchRunFinalization captures the Patrol service's unified publications and
+// pushes. Set it before adding findings: setting the unified callback syncs
+// the active findings at once.
+func watchRunFinalization(patrol *ai.PatrolService) *runFinalization {
+ sent := &runFinalization{}
+ patrol.SetUnifiedFindingCallback(func(f *ai.Finding) bool {
+ sent.published = append(sent.published, f)
+ return true
+ })
+ patrol.SetPushNotifyCallback(func(n relay.PushNotificationPayload) {
+ sent.pushes = append(sent.pushes, n)
+ })
+ return sent
+}
+
+// A run is finalized when its close lands, whether it completed or failed: its
+// record shows the run's final state, the unified finding is published once and
+// a planned fix awaiting approval asks the operator to approve it. A run that
+// planned a fix and then failed, normally on its final model call, used to get
+// none of this. An action's outcome is pushed when its transition is
+// reconciled, so a close does not push it again.
+func TestOrchestratorRunCloseFinalizesTheRun(t *testing.T) {
+ cases := []struct {
+ name string
+ status aicontracts.InvestigationStatus
+ outcome aicontracts.InvestigationOutcome
+ fail string
+ action *aicontracts.ActionReference
+ approval bool
+ }{
+ {
+ name: "failed after planning a fix",
+ status: aicontracts.InvestigationStatusFailed,
+ outcome: aicontracts.OutcomeFixQueued,
+ fail: "investigation final response failed: context deadline exceeded",
+ action: &aicontracts.ActionReference{ActionID: "action-1", State: "pending_approval"},
+ approval: true,
+ },
+ {
+ name: "completed with a planned fix",
+ status: aicontracts.InvestigationStatusCompleted,
+ outcome: aicontracts.OutcomeFixQueued,
+ action: &aicontracts.ActionReference{ActionID: "action-1", State: "pending_approval"},
+ approval: true,
+ },
+ {
+ name: "completed with a verified fix",
+ status: aicontracts.InvestigationStatusCompleted,
+ outcome: aicontracts.OutcomeFixVerified,
+ action: &aicontracts.ActionReference{ActionID: "action-1", State: "completed", VerificationStatus: "verified"},
+ },
+ {
+ name: "timed out without a plan",
+ status: aicontracts.InvestigationStatusFailed,
+ outcome: aicontracts.OutcomeTimedOut,
+ fail: "context deadline exceeded",
+ },
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ orchestratorFindings, sessions, patrol := orchestratorPatrolFor(t, nil)
+ sent := watchRunFinalization(patrol)
+ findings := patrol.GetFindings()
+ findings.Add(investigatedFinding("finding-1"))
+ run := startOrchestratorRun(t, orchestratorFindings, sessions, "finding-1", "chat-1")
+ if len(sent.published) != 0 || len(sent.pushes) != 0 {
+ t.Fatalf("starting a run finalized it: %+v", sent)
+ }
+
+ // The orchestrator records the run's end on its session before
+ // it closes the run on the finding.
+ session := sessions.GetLatestByFinding("finding-1")
+ session.Action = tc.action
+ sessions.Update(session)
+ if tc.fail != "" {
+ sessions.Fail(session.ID, tc.fail)
+ sessions.SetOutcome(session.ID, tc.outcome)
+ } else {
+ sessions.Complete(session.ID, tc.outcome, "The backup service exits on a stale mount.", nil)
+ }
+ if !finishOrchestratorRun(orchestratorFindings, run, tc.status, tc.outcome) {
+ t.Fatal("the run's close was refused")
+ }
+
+ record := findings.Get("finding-1").InvestigationRecord
+ if record == nil || record.Status != tc.status || record.Outcome != tc.outcome || record.CompletedAt == nil || record.Error != tc.fail {
+ t.Fatalf("record does not show the closed run: %+v", record)
+ }
+ if tc.action != nil && (record.Action == nil || record.Action.ActionID != tc.action.ActionID) {
+ t.Fatalf("record lost the planned action: %+v", record.Action)
+ }
+ if len(sent.published) != 1 || sent.published[0].InvestigationOutcome != string(tc.outcome) {
+ t.Fatalf("published %d findings, want the closed run once: %+v", len(sent.published), sent.published)
+ }
+ switch {
+ case tc.approval && (len(sent.pushes) != 1 || sent.pushes[0].Type != relay.PushTypeApprovalRequest || sent.pushes[0].ActionID != tc.action.ActionID):
+ t.Fatalf("pushes = %+v, want one approval request for %s", sent.pushes, tc.action.ActionID)
+ case !tc.approval && len(sent.pushes) != 0:
+ t.Fatalf("pushes = %+v, want none", sent.pushes)
+ }
+ })
+ }
+}
+
+// The orchestrator writes a run's action back as it was planned. When the plan
+// moved on while the final model call was pending, here by expiring, the close
+// takes the action from the action audit: the record shows the expired plan and
+// nobody is asked to approve it.
+func TestOrchestratorRunCloseTakesTheActionFromTheAudit(t *testing.T) {
+ audits := unifiedresources.NewMemoryStore()
+ orchestratorFindings, sessions, patrol := orchestratorPatrolFor(t, audits)
+ sent := watchRunFinalization(patrol)
+ findings := patrol.GetFindings()
+ findings.Add(investigatedFinding("finding-1"))
+ run := startOrchestratorRun(t, orchestratorFindings, sessions, "finding-1", "chat-1")
+
+ session := sessions.GetLatestByFinding("finding-1")
+ now := time.Now().UTC()
+ if _, _, err := audits.CreateActionAudit(unifiedresources.ActionAuditRecord{
+ ID: "act-1", CreatedAt: now, UpdatedAt: now, State: unifiedresources.ActionStateExpired,
+ Request: unifiedresources.ActionRequest{RequestID: "proposal-1", ResourceID: "vm-finding-1", CapabilityName: "restart", RequestedBy: "pulse_patrol"},
+ Plan: unifiedresources.ActionPlan{ActionID: "act-1", RequestID: "proposal-1", Allowed: true},
+ Origin: &unifiedresources.ActionOrigin{Surface: patrolActionOriginSurface, FindingID: "finding-1", InvestigationID: session.ID, ProposalID: "proposal-1"},
+ }, nil); err != nil {
+ t.Fatal(err)
+ }
+ session.Action = &aicontracts.ActionReference{ActionID: "act-1", State: "pending_approval"}
+ sessions.Update(session)
+ sessions.Fail(session.ID, "investigation final response failed: context deadline exceeded")
+ sessions.SetOutcome(session.ID, aicontracts.OutcomeFixQueued)
+ if !finishOrchestratorRun(orchestratorFindings, run, aicontracts.InvestigationStatusFailed, aicontracts.OutcomeFixQueued) {
+ t.Fatal("the run's close was refused")
+ }
+
+ record := findings.Get("finding-1").InvestigationRecord
+ if record == nil || record.Action == nil || record.Action.State != string(unifiedresources.ActionStateExpired) || record.Outcome != aicontracts.OutcomeNeedsAttention {
+ t.Fatalf("record kept the plan as written back: %+v", record)
+ }
+ if got := sessions.GetLatestByFinding("finding-1"); got.Action == nil || got.Action.State != string(unifiedresources.ActionStateExpired) {
+ t.Fatalf("session kept the plan as written back: %+v", got.Action)
+ }
+ for _, push := range sent.pushes {
+ if push.Type == relay.PushTypeApprovalRequest {
+ t.Fatalf("an expired plan asked for approval: %+v", sent.pushes)
+ }
+ }
+}
+
+// A close that does not land, because a newer run replaced the one reporting
+// back, finalizes nothing.
+func TestOrchestratorRunCloseThatDoesNotLandIsNotFinalized(t *testing.T) {
+ orchestratorFindings, sessions, patrol := orchestratorPatrolFor(t, nil)
+ sent := watchRunFinalization(patrol)
+ findings := patrol.GetFindings()
+ findings.Add(investigatedFinding("finding-1"))
+ runA := startOrchestratorRun(t, orchestratorFindings, sessions, "finding-1", "chat-a")
+ findings.RecoverStuckInvestigation("finding-1", "chat-a", time.Time{}, ai.InvestigationStatusFailed, ai.InvestigationOutcomeTimedOut, time.Now())
+ startOrchestratorRun(t, orchestratorFindings, sessions, "finding-1", "chat-b")
+
+ if finishOrchestratorRun(orchestratorFindings, runA, aicontracts.InvestigationStatusFailed, aicontracts.OutcomeFixQueued) {
+ t.Fatal("an earlier run overwrote the run that replaced it")
+ }
+ if len(sent.published) != 0 || len(sent.pushes) != 0 || findings.Get("finding-1").InvestigationRecord != nil {
+ t.Fatalf("a close that did not land was finalized: %+v", sent)
+ }
+}
diff --git a/internal/api/patrol_action_expiry_sweep_test.go b/internal/api/patrol_action_expiry_sweep_test.go
new file mode 100644
index 0000000000..33d50d183e
--- /dev/null
+++ b/internal/api/patrol_action_expiry_sweep_test.go
@@ -0,0 +1,140 @@
+package api
+
+import (
+ "net/http"
+ "net/http/httptest"
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/ai"
+ "github.com/rcourtman/pulse-go-rewrite/internal/config"
+ unified "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/aicontracts"
+)
+
+// patrolPlanAudit is a pending Patrol plan for an investigation, planned at
+// planned and expiring at expires.
+func patrolPlanAudit(id, investigationID string, planned, expires time.Time) unified.ActionAuditRecord {
+ return unified.ActionAuditRecord{
+ ID: id,
+ CreatedAt: planned,
+ UpdatedAt: planned,
+ State: unified.ActionStatePending,
+ Request: boundActionTestRequest("proposal-"+id, "vm:42", "restart", "Clear the stuck mount", "pulse_patrol"),
+ Plan: unified.ActionPlan{
+ ActionID: id,
+ RequestID: "proposal-" + id,
+ Allowed: true,
+ RequiresApproval: true,
+ ApprovalPolicy: unified.ApprovalAdmin,
+ ApprovalRequirement: unified.ApprovalRequirementForFloor(unified.ApprovalAdmin),
+ PlannedAt: planned,
+ ExpiresAt: expires,
+ ResourceVersion: "resource:sha256:test",
+ PolicyVersion: "policy:sha256:test",
+ PlanHash: "sha256:" + id,
+ },
+ Origin: &unified.ActionOrigin{Surface: patrolActionOriginSurface, FindingID: "finding-1", InvestigationID: investigationID, ProposalID: "proposal-" + id},
+ }
+}
+
+// patrolSweepFixture is a finding whose run failed after planning a fix, with
+// Patrol reconciling the action handlers' transitions.
+type patrolSweepFixture struct {
+ handlers *ResourceHandlers
+ store unified.ResourceStore
+ findings *ai.FindingsStore
+ investigations *testInvestigationStore
+ investigation *aicontracts.InvestigationSession
+}
+
+func newPatrolSweepFixture(t *testing.T, currentActionID string) patrolSweepFixture {
+ t.Helper()
+ h := newActionTestResourceHandlers(t, &config.Config{DataPath: t.TempDir()})
+ store, err := h.getStore("default")
+ if err != nil {
+ t.Fatalf("get store: %v", err)
+ }
+ svc := ai.NewService(nil, nil)
+ svc.SetStateProvider(&MockStateProvider{})
+ findings := svc.GetPatrolService().GetFindings()
+ findings.Add(&ai.Finding{ID: "finding-1", ResourceID: "vm:42", Title: "Unhealthy service", Severity: ai.FindingSeverityWarning,
+ InvestigationSessionID: "session-1", InvestigationStatus: string(aicontracts.InvestigationStatusFailed),
+ InvestigationOutcome: string(aicontracts.OutcomeFixQueued)})
+ investigations := newTestInvestigationStore()
+ investigation := investigations.Create("finding-1", "session-1")
+ investigation.Status = aicontracts.InvestigationStatusFailed
+ investigation.Outcome = aicontracts.OutcomeFixQueued
+ investigation.Action = &aicontracts.ActionReference{ActionID: currentActionID, State: string(unified.ActionStatePending)}
+ investigations.Update(investigation)
+
+ patrolHandler := &AISettingsHandler{defaultAIService: svc,
+ investigationStores: map[string]aicontracts.InvestigationStore{"default": investigations},
+ resourceStoreProvider: func(string) (unified.ResourceStore, error) { return store, nil },
+ }
+ h.SetActionTransitionPublisher(patrolHandler.ReconcilePatrolActionTransition)
+ return patrolSweepFixture{handlers: h, store: store, findings: findings, investigations: investigations, investigation: investigation}
+}
+
+func (f patrolSweepFixture) readInbox(t *testing.T) {
+ t.Helper()
+ rec := httptest.NewRecorder()
+ req := httptest.NewRequest(http.MethodGet, "/api/actions?view=pending", nil)
+ f.handlers.HandleListActions(rec, actionHandlerTestRequest(req, ""))
+ if rec.Code != http.StatusOK {
+ t.Fatalf("inbox status = %d, body=%s", rec.Code, rec.Body.String())
+ }
+}
+
+// Reading the action inbox sweeps plans past their expiry. A Patrol fix whose
+// plan expires in that sweep moves its investigation and finding off the
+// queued fix, as the action's own expiry does, rather than leaving them queued
+// until someone opens the investigation.
+func TestActionInboxSweepReconcilesAnExpiredPatrolPlan(t *testing.T) {
+ now := time.Now().UTC().Truncate(time.Second)
+ f := newPatrolSweepFixture(t, "act-1")
+ if err := f.store.RecordActionAudit(patrolPlanAudit("act-1", f.investigation.ID, now.Add(-10*time.Minute), now.Add(-5*time.Minute))); err != nil {
+ t.Fatalf("RecordActionAudit: %v", err)
+ }
+
+ f.readInbox(t)
+
+ if got := f.investigations.Get(f.investigation.ID); got.Outcome != aicontracts.OutcomeNeedsAttention || got.Action == nil || got.Action.State != string(unified.ActionStateExpired) {
+ t.Fatalf("investigation kept the expired plan queued: outcome=%q action=%+v", got.Outcome, got.Action)
+ }
+ if got := f.findings.Get("finding-1"); got.InvestigationOutcome != string(aicontracts.OutcomeNeedsAttention) {
+ t.Fatalf("finding kept the expired plan queued: outcome=%q", got.InvestigationOutcome)
+ }
+}
+
+// A refresh plans a replacement under the same investigation and leaves the
+// replaced plan to expire on its own. That plan's swept expiry must not take
+// the investigation back from its replacement, which still waits for approval.
+// The refresh lineage decides, not the clock: here the replacement's planning
+// time reads earlier, as after the clock was set back.
+func TestActionInboxSweepLeavesAReplacementPlanInPlace(t *testing.T) {
+ now := time.Now().UTC().Truncate(time.Second)
+ f := newPatrolSweepFixture(t, "act-2")
+ if err := f.store.RecordActionAudit(patrolPlanAudit("act-1", f.investigation.ID, now.Add(-10*time.Minute), now.Add(-5*time.Minute))); err != nil {
+ t.Fatalf("RecordActionAudit(replaced): %v", err)
+ }
+ replacement := patrolPlanAudit("act-2", f.investigation.ID, now.Add(-20*time.Minute), now.Add(time.Hour))
+ replacement.Request.RequestID = "refresh:act-1"
+ replacement.Plan.RequestID = "refresh:act-1"
+ replacement.Origin.ProposalID = "proposal-act-1"
+ if err := f.store.RecordActionAudit(replacement); err != nil {
+ t.Fatalf("RecordActionAudit(replacement): %v", err)
+ }
+
+ f.readInbox(t)
+
+ if replaced, found, err := f.store.GetActionAudit("act-1"); err != nil || !found || replaced.State != unified.ActionStateExpired {
+ t.Fatalf("the replaced plan did not expire in the sweep: %+v found=%v err=%v", replaced, found, err)
+ }
+ if got := f.investigations.Get(f.investigation.ID); got.Outcome != aicontracts.OutcomeFixQueued || got.Action == nil || got.Action.ActionID != "act-2" {
+ t.Fatalf("the replaced plan's expiry took the investigation: outcome=%q action=%+v", got.Outcome, got.Action)
+ }
+ if got := f.findings.Get("finding-1"); got.InvestigationOutcome != string(aicontracts.OutcomeFixQueued) {
+ t.Fatalf("the replaced plan's expiry moved the finding: outcome=%q", got.InvestigationOutcome)
+ }
+}
diff --git a/internal/api/patrol_action_reconciliation.go b/internal/api/patrol_action_reconciliation.go
index e38ea247b2..76a055d612 100644
--- a/internal/api/patrol_action_reconciliation.go
+++ b/internal/api/patrol_action_reconciliation.go
@@ -6,6 +6,7 @@ import (
"reflect"
"strings"
+ "github.com/rcourtman/pulse-go-rewrite/internal/actionlifecycle"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rcourtman/pulse-go-rewrite/pkg/aicontracts"
"github.com/rs/zerolog/log"
@@ -90,6 +91,9 @@ func (h *AISettingsHandler) applyPatrolActionAudit(orgID string, audit unifiedre
if investigation == nil || strings.TrimSpace(investigation.FindingID) != origin.FindingID {
return
}
+ if h.patrolActionSuperseded(orgID, investigation.Action, audit) {
+ return
+ }
reference := actionReferenceFromAudit(audit)
changed := false
if !reflect.DeepEqual(investigation.Action, reference) {
@@ -113,6 +117,40 @@ func (h *AISettingsHandler) applyPatrolActionAudit(orgID string, audit unifiedre
}
}
+// maxPatrolActionRefreshLineage bounds the walk back through refreshes.
+const maxPatrolActionRefreshLineage = 32
+
+// patrolActionSuperseded reports whether audit is for a plan that the
+// investigation's current action replaced through one or more refreshes. A
+// refresh plans the replacement under the same origin and leaves the replaced
+// plan to expire on its own; that plan's later transitions, such as its
+// expiry, must not take the investigation back from its replacement.
+func (h *AISettingsHandler) patrolActionSuperseded(orgID string, current *aicontracts.ActionReference, audit unifiedresources.ActionAuditRecord) bool {
+ if current == nil || strings.TrimSpace(current.ActionID) == "" || current.ActionID == audit.ID {
+ return false
+ }
+ store, err := h.actionAuditStore(orgID)
+ if err != nil {
+ return false
+ }
+ actionID := current.ActionID
+ for range maxPatrolActionRefreshLineage {
+ record, found, err := store.GetActionAudit(actionID)
+ if err != nil || !found {
+ return false
+ }
+ replaced, ok := actionlifecycle.RefreshedActionID(record)
+ if !ok {
+ return false
+ }
+ if replaced == audit.ID {
+ return true
+ }
+ actionID = replaced
+ }
+ return false
+}
+
func (h *AISettingsHandler) actionAuditStore(orgID string) (unifiedresources.ResourceStore, error) {
h.stateMu.RLock()
provider := h.resourceStoreProvider