Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Record Patrol investigation conclusions as a typed tool call
rcourtmanb3b5ad6e1fc4
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 2140384fe0..267021b175 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -10165,3 +10165,13 @@ the credential reveal render as shared `InlineNotice`s; agent and securityfacts as a `DetailSectionTable`; the connection id and identity evidence sitin a closed Details disclosure. No diagnosis, repair or credential flowchanges: this is presentation only.++The Patrol investigation wiring in `internal/api/ai_handlers.go` now passes the+typed conclusion an investigation records with `patrol_conclude_investigation`+to the private orchestrator. It also seeds the orchestrator's concurrency and+timeout from the Patrol provider: one investigation at a time and a 25-minute+default on a subscription agent. The digest email names the new `unclassified`+and `interrupted` outcomes. None of this enrols, commands, updates or removes an+agent, or touches agent identity or credentials. Investigations still reach+hosts only through the existing read-only evidence tools and the governed+action lifecycle.diff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.mdindex 34d4912b1e..05931e26bd 100644--- a/docs/release-control/v6/internal/subsystems/ai-runtime.md+++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md@@ -9058,3 +9058,109 @@ diagnostics (`journalctl -u ... --since`, `dmesg -T`, `ss -tlnp`, `ip addr`,read-only. Proof: `TestExecutionIntent_ArgumentAwareReadCommands` and`TestExecutionIntent_RouteOperationBoundary` in`internal/ai/tools/tools_query_readonly_test.go`.++### Typed investigation conclusions and subscription-aware investigation limits (22 October 2026)++Patrol investigations record their zero-action conclusion through+`patrol_conclude_investigation`, an investigation-only, side-effect-free,+read-kind, mutation-none tool on the request-local `ProposalCapture`. The model+supplies one of `resolved`, `cannot_fix` or `needs_attention` and a reason.+The capture keeps the first conclusion: an identical replay is accepted and a+different outcome is refused. `InvestigationRunResult.Conclusion` carries it to+the orchestrator beside any proposal. A typed proposal still owns remediation+and outranks a conclusion, and the conclusion consumes no evidence budget.+Once a conclusion is recorded, the next turn is text-only: the summary.+The final turn, whether reached by the turn limit or by the deadline reserve,+offers only the tools that end a run: the conclusion, plus the typed proposal+once the evidence budget is spent. Evidence exhaustion offers the proposal and+the conclusion.++A run reserves time to conclude before its context deadline. The reserve is+one more evidence turn plus the conclusion and summary turns, each estimated as+the slowest turn so far (20 s minimum), plus 15 s, capped at half the run's+budget. When the time left reaches the reserve, the current turn becomes the+final turn. The final summary call may use the time left before that deadline+instead of the 20 s interactive bound, which interactive chat keeps.++On subscription agents, a positive `StreamIdleTimeout` remains an earlier+complete-turn deadline, because the transport buffers the whole turn.+Investigation turns there get a four-minute caller-owned stall budget, while+Watch keeps 60 s. Without an explicit setting, an investigation on a+subscription agent defaults to 25 minutes and to one investigation at a time,+because that transport serves one call at a time.++`LoadConfig` and autonomy saves republish the timeout and, through the optional+`aicontracts.InvestigationConcurrencyLimitUpdater`, the concurrency limit, so a+provider change applies without rebuilding the orchestrator. Both read the+orchestrator before taking the service lock, because `PatrolService` takes its+own lock before the service lock. Patrol's outer deadline for an automatic+investigation follows the effective timeout plus a 5-minute grace instead of a+fixed 10 minutes.++The stuck-run sweep looks up the session of the run each running finding names+among every session the orchestrator's store holds for the finding, so a newer+session cannot hide a live run. Without the store it uses the finding's latest+session, and leaves the finding alone when a different run is the latest. A+live run is reset as+`timed_out` only after its own `InvestigationSession.Deadline` plus that+grace, or its start plus the current timeout and grace when no deadline is+recorded. So a timeout lowered mid-run does not cut it short, and the previous+run's `LastInvestigatedAt` no longer resets a live re-investigation. A run the+orchestrator holds no session for, or closed as `interrupted` when it reloaded+after a restart, is recorded as `interrupted`. Such a run keeps its attempt, so+a run that crashes Pulse cannot retry forever. A run the orchestrator closed+without updating the finding takes the result its session recorded, once the+orchestrator's own update has had the grace to land, or `timed_out` when the+session recorded none. Without an orchestrator the sweep keeps the+`LastInvestigatedAt` rule. A first investigation cut short by a hard restart no+longer stays running for good. `FindingsStore.RecoverStuckInvestigation`+applies the recovery only while the finding still shows that run as running,+so a result or a newer run that landed after the sweep looked is kept. A run+that started before the finding's current occurrence is recorded only as+`interrupted`, never with its session's result. An outcome the action+lifecycle recorded on the finding during the run, such as+`fix_queued`, is kept and the run completes, so a queued fix still blocks+another investigation. The close stamps `LastInvestigatedAt` like any other+closed run.++Two outcomes join the vocabulary. `unclassified` records a completed run that+left neither a typed conclusion nor a recognisable marker. The private+orchestrator no longer defaults that case to `needs_attention`, which ends+investigation of the finding. `unclassified` keeps the normal cooldown and+attempt limit, presents as needs-attention in the loop state, and lands in the+telemetry `other` bucket with no schema change. `interrupted` records a run+cut short by a restart, shutdown or caller cancellation. It is not+`timed_out`, has no cooldown, and reports as `in_progress`. A run cancelled by+shutdown spends no attempt.++A typed conclusion recorded before a later provider failure or summary timeout+stands when the run holds no proposal or action. A failure without one remains+failed, and fallback prose is still never classified as a conclusion. The+prose-marker reader remains a fallback. It accepts only line-start markers+inside the conclusion section. It ignores fenced and indented-code lines,+reading any fence-like line as a fence and ending it only where every+CommonMark reading would, and it treats a quote as running to the next blank+line. Where that differs from CommonMark it hides more, which can only leave+a run unclassified. It now tolerates markdown emphasis, list markers,+numbering, case and heading punctuation.++Proofs:+- `agentic_investigation_conclusion_test.go`: final-turn tools, the summary+ after a conclusion, the deadline reserve, and the reserve, summary-timeout+ and subscription turn-budget helpers.+- `tools_conclude_test.go`: the closed vocabulary, first conclusion wins,+ profile and capture gating, and refusals.+- `findings_test.go`: `interrupted` and `unclassified` eligibility.+- `patrol_investigation_limits_test.go`: the sweep (live runs held to their+ own deadline, a live or pending run behind a newer session, restart+ reconciliation, the settle window, session results, kept action outcomes,+ a result from before a regression, the guarded recovery write and the+ no-orchestrator rule) and the run ceiling.+- `service_patrol_limits_test.go`: provider-aware limits on reload.+- `ai_config_test.go`, `contracts_test.go` (including the session `deadline`+ wire shape), `patrol_cost_test.go` and the orchestrator-setup case in+ `ai_handlers_investigation_additional_test.go`.++This changes outcome recording and run limits only. It widens no mutation,+approval or execution boundary, and it is not measured improvement on+installed systems.diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.mdindex f2ea283cae..6fd8b32499 100644--- a/docs/release-control/v6/internal/subsystems/api-contracts.md+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md@@ -12586,3 +12586,25 @@ Routes, caller scope/token filtering, cache policy and probe count are unchanged`TestContract_DiagnosticsAPIRejectionGuidanceSurvivesSerialization` cover realHTTP statuses, private-body controls, same-credential fixture recovery and wireattribution. They are not installed permission or guest-recovery acceptance.++### Typed investigation conclusion tool (22 October 2026)++`patrol_conclude_investigation` joins `patrol_propose_action` and+`patrol_action_capabilities` as an investigation-only tool whose canonical name+and descriptor live in `internal/agentcapabilities/`. It is statically+read-kind and mutation-none, projected into the investigation scope surface,+rejected by registry policy outside the investigation profile, and unavailable+without the request-local capture. The model supplies only `outcome`+(`resolved`, `cannot_fix` or `needs_attention`) and `reason`. The tool records+the run's conclusion for the private orchestrator, which persists the outcome;+it creates no proposal and grants no plan, approval or execution authority.+The Pulse–enterprise boundary stays additive:+`aicontracts.OrchestratorInvestigationResult.Conclusion`,+`aicontracts.InvestigationConclusion`, the `unclassified` and `interrupted`+outcome constants and the optional `InvestigationConcurrencyLimitUpdater` are+new. `aicontracts.InvestigationSession.Deadline` (`deadline`, omitted when+unrecorded) carries when a run's own timeout expires, so investigation session+payloads gain one optional field. An older orchestrator ignores the conclusion+and keeps reading prose markers, and records no deadline, so core measures its+runs from their start. Proofs: `patrol_scope_tools_test.go`, `names_test.go`,+`tools_conclude_test.go` and `contracts_test.go`.diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex 61df861998..1c75958192 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -7158,3 +7158,10 @@ The Proxmox page surface's Overview guest heading ("Guests on <node>", Show allnodes) now renders through the shared `PlatformHostScopedTableTitle` that theDocker and vSphere overviews also use; its Storage, Replication, Backups, Cephand Mail Gateway tabs, and every storage and recovery rule, are unchanged.++The `internal/api` changes for typed Patrol investigation conclusions open no+storage, backup or recovery path. They hand the run's typed conclusion to the+private orchestrator, make investigation concurrency and timeout follow the+Patrol provider, and label the `unclassified` and `interrupted` outcomes in the+digest email. Recovery evidence, restore rules and storage capabilities are+unchanged, and Patrol gains no storage remediation capability.diff --git a/internal/agentcapabilities/invocation.go b/internal/agentcapabilities/invocation.goindex d9f121ef6c..9c89aac6dc 100644--- a/internal/agentcapabilities/invocation.go+++ b/internal/agentcapabilities/invocation.go@@ -223,6 +223,9 @@ var registryInvocationDescriptors = map[string]InvocationDescriptor{PatrolReportFindingToolName: staticClass(ToolCallKindWrite, MutationPulseState),PatrolResolveFindingToolName: staticClass(ToolCallKindWrite, MutationPulseState),PatrolProposeObserverToolName: staticClass(ToolCallKindWrite, MutationPulseState),+ // patrol_conclude_investigation only records the run's typed conclusion+ // in the request-local capture; the orchestrator persists the outcome.+ PatrolConcludeInvestigationToolName: staticClass(ToolCallKindRead, MutationNone),}// Clone returns a deep copy of the descriptor so callers can neverdiff --git a/internal/agentcapabilities/patrol_scope_tools.go b/internal/agentcapabilities/patrol_scope_tools.goindex 2479a3ffc4..f707677faf 100644--- a/internal/agentcapabilities/patrol_scope_tools.go+++ b/internal/agentcapabilities/patrol_scope_tools.go@@ -111,8 +111,9 @@ func PatrolDetectionToolNamesForResourceTypes(resourceTypes []string) ([]string,}// PatrolInvestigationToolNamesForResourceTypes composes scoped evidence with-// the side-effect-free capability lookup and typed proposal sink. Planning,-// approval, execution, and verification remain outside this surface.+// the side-effect-free capability lookup, typed proposal sink and typed+// conclusion capture. Planning, approval, execution, and verification remain+// outside this surface.func PatrolInvestigationToolNamesForResourceTypes(resourceTypes []string) ([]string, bool) {names, ok := PatrolEvidenceToolNamesForResourceTypes(resourceTypes)if !ok {@@ -121,5 +122,6 @@ func PatrolInvestigationToolNamesForResourceTypes(resourceTypes []string) ([]strreturn append(names,PatrolActionCapabilitiesToolName,PatrolProposeActionToolName,+ PatrolConcludeInvestigationToolName,), true}diff --git a/internal/agentcapabilities/patrol_scope_tools_test.go b/internal/agentcapabilities/patrol_scope_tools_test.goindex 35b782172c..cbd3839984 100644--- a/internal/agentcapabilities/patrol_scope_tools_test.go+++ b/internal/agentcapabilities/patrol_scope_tools_test.go@@ -23,7 +23,7 @@ func TestPatrolScopedToolProjectionUsesTypedLeastManifest(t *testing.T) {if !ok {t.Fatal("physical-disk scope was not recognized")}- for _, required := range []string{PulseQueryToolName, PulseReadToolName, PulseMetricsToolName, PulseStorageToolName, PatrolActionCapabilitiesToolName, PatrolProposeActionToolName} {+ for _, required := range []string{PulseQueryToolName, PulseReadToolName, PulseMetricsToolName, PulseStorageToolName, PatrolActionCapabilitiesToolName, PatrolProposeActionToolName, PatrolConcludeInvestigationToolName} {if !slices.Contains(storageInvestigation, required) {t.Fatalf("storage investigation tools missing %q: %v", required, storageInvestigation)}@@ -61,6 +61,9 @@ func TestPatrolScopedToolProjectionComposesProfileOwnedTools(t *testing.T) {if slices.Contains(detection, PatrolProposeActionToolName) {t.Fatalf("detection surface gained investigation proposal authority: %v", detection)}+ if slices.Contains(detection, PatrolConcludeInvestigationToolName) {+ t.Fatalf("detection surface gained the investigation conclusion capture: %v", detection)+ }}func TestGuestInvestigationsCanReadCollectedBackupEvidence(t *testing.T) {@@ -71,7 +74,7 @@ func TestGuestInvestigationsCanReadCollectedBackupEvidence(t *testing.T) {t.Fatalf("guest investigation lacks backup evidence: %v", names)}for _, name := range names {- if !IsPatrolInfrastructureEvidenceToolName(name) && name != PatrolActionCapabilitiesToolName && name != PatrolProposeActionToolName {+ if !IsPatrolInfrastructureEvidenceToolName(name) && name != PatrolActionCapabilitiesToolName && name != PatrolProposeActionToolName && name != PatrolConcludeInvestigationToolName {t.Fatalf("read projection gained action authority: %s", name)}}diff --git a/internal/agentcapabilities/tool_names.go b/internal/agentcapabilities/tool_names.goindex 864925603d..5ca83af966 100644--- a/internal/agentcapabilities/tool_names.go+++ b/internal/agentcapabilities/tool_names.go@@ -49,4 +49,10 @@ const (// model choose an action target from evidence instead of pinning// remediation to the finding's initial resource.PatrolActionCapabilitiesToolName = "patrol_action_capabilities"+ // PatrolConcludeInvestigationToolName is the investigation-only,+ // side-effect-free conclusion capture. It records one typed zero-action+ // conclusion (resolved, cannot_fix or needs_attention) in the+ // request-local capture sink so the orchestrator never has to infer the+ // outcome from prose. It grants no planning, approval or execution.+ PatrolConcludeInvestigationToolName = "patrol_conclude_investigation")diff --git a/internal/ai/chat/agentic.go b/internal/ai/chat/agentic.goindex 792ed4976d..d759e2c418 100644--- a/internal/ai/chat/agentic.go+++ b/internal/ai/chat/agentic.go@@ -812,6 +812,18 @@ func (a *AgenticLoop) executeWithTools(ctx context.Context, sessionID string, me// limit check below owns compaction when the request actually needs it.currentTurnStartIndex := len(providerMessages)+ // A Patrol investigation reserves time to conclude before its deadline.+ // Once another evidence turn would leave too little for the conclusion+ // and its summary, the current turn becomes the final turn, so the run+ // ends with a recorded conclusion instead of a timeout.+ investigationDeadline, investigationHasDeadline := ctx.Deadline()+ investigationHasDeadline = investigationHasDeadline && isPatrolInvestigationExecution(a.currentExecutionProfile())+ var investigationBudget, slowestInvestigationTurn time.Duration+ var investigationTurnStartedAt time.Time+ if investigationHasDeadline {+ investigationBudget = time.Until(investigationDeadline)+ }+for turn < maxTurns ||(patrolOutputLimitRecoveryPending && !patrolOutputLimitRecoveryAttempted) ||(investigationOutputLimitRecoveryPending && !investigationOutputLimitRecoveryAttempted) {@@ -863,6 +875,24 @@ func (a *AgenticLoop) executeWithTools(ctx context.Context, sessionID string, medefault:}+ if investigationHasDeadline {+ now := time.Now()+ if !investigationTurnStartedAt.IsZero() {+ slowestInvestigationTurn = max(slowestInvestigationTurn, now.Sub(investigationTurnStartedAt))+ }+ investigationTurnStartedAt = now+ remaining := investigationDeadline.Sub(now)+ if turn < maxTurns-1 && remaining <= investigationConclusionReserve(slowestInvestigationTurn, investigationBudget) {+ log.Warn().+ Int("turn", turn).+ Dur("remaining", remaining).+ Dur("slowest_turn", slowestInvestigationTurn).+ Str("session_id", sessionID).+ Msg("[AgenticLoop] Investigation deadline approaching — making this the final turn")+ maxTurns = turn + 1+ }+ }+log.Info().Int("turn", turn).Int("messages", len(providerMessages)).@@ -884,7 +914,7 @@ func (a *AgenticLoop) executeWithTools(ctx context.Context, sessionID string, meSystemCacheablePrefix: stablePrompt,Tools: tools,ExecutionID: a.executionID,- StreamIdleTimeout: streamIdleTimeout,+ StreamIdleTimeout: investigationStreamIdleTimeout(a.currentExecutionProfile(), providerName, streamIdleTimeout),}if isPatrolDetectionExecution(a.currentExecutionProfile()) && patrolFindingsReadCompleted {req.Tools = withoutProviderTool(req.Tools, agentcapabilities.PatrolGetFindingsToolName)@@ -941,6 +971,9 @@ func (a *AgenticLoop) executeWithTools(ctx context.Context, sessionID string, meInt("max_turns", maxTurns).Str("session_id", sessionID).Msg("[AgenticLoop] Approaching max turns — restricting Watch to finding decisions")+ } else if isPatrolInvestigationExecution(a.currentExecutionProfile()) && !investigationOutputLimitRecoveryTurn {+ // The investigation block below restricts the final turn to the+ // tools that conclude the run.} else {req.Tools = niltextOnlySafetyBrake = true@@ -964,8 +997,30 @@ func (a *AgenticLoop) executeWithTools(ctx context.Context, sessionID string, meif a.maxEvidenceCalls > 0 {req.System += fmt.Sprintf("\nEvidence-tool calls remaining within this run's configured limit: %d.", max(0, a.maxEvidenceCalls-a.totalEvidenceCalls))}- if a.maxEvidenceCalls > 0 && a.totalEvidenceCalls >= a.maxEvidenceCalls {- req.Tools = investigationTerminalTools(tools)+ evidenceBudgetSpent := a.maxEvidenceCalls > 0 && a.totalEvidenceCalls >= a.maxEvidenceCalls+ switch {+ case a.executor.InvestigationConcluded():+ // The conclusion is recorded; the next response is its summary.+ req.Tools = nil+ textOnlySafetyBrake = true+ case turn >= maxTurns-1:+ // Final turn (turn limit or deadline reserve): record a typed+ // conclusion rather than leave the outcome to be inferred from+ // prose. A proposal stays available only once evidence is spent,+ // as before.+ req.Tools = investigationTerminalTools(tools, evidenceBudgetSpent)+ textOnlySafetyBrake = len(req.Tools) == 0+ if !textOnlySafetyBrake {+ req.System += investigationFinalConclusionSystemNote+ }+ log.Warn().+ Int("turn", turn).+ Int("max_turns", maxTurns).+ Int("tools", len(req.Tools)).+ Str("session_id", sessionID).+ Msg("[AgenticLoop] Final investigation turn — restricting tools to the conclusion")+ case evidenceBudgetSpent:+ req.Tools = investigationTerminalTools(tools, true)textOnlySafetyBrake = len(req.Tools) == 0}}diff --git a/internal/ai/chat/agentic_additional_test.go b/internal/ai/chat/agentic_additional_test.goindex ce8f48ddcd..a8641c8d05 100644--- a/internal/ai/chat/agentic_additional_test.go+++ b/internal/ai/chat/agentic_additional_test.go@@ -137,10 +137,22 @@ func TestInvestigationEvidenceBudgetHelpers(t *testing.T) {{Name: agentcapabilities.PatrolProposeActionToolName},{Name: agentcapabilities.PatrolActionCapabilitiesToolName},}- terminal := investigationTerminalTools(available)+ terminal := investigationTerminalTools(available, true)if len(terminal) != 1 || terminal[0].Name != agentcapabilities.PatrolProposeActionToolName {t.Fatalf("terminal tools = %+v, want only patrol_propose_action", terminal)}+ withConclusion := append(available, providers.Tool{Name: agentcapabilities.PatrolConcludeInvestigationToolName})+ terminal = investigationTerminalTools(withConclusion, true)+ if len(terminal) != 2 || terminal[0].Name != agentcapabilities.PatrolProposeActionToolName || terminal[1].Name != agentcapabilities.PatrolConcludeInvestigationToolName {+ t.Fatalf("terminal tools = %+v, want patrol_propose_action then patrol_conclude_investigation", terminal)+ }+ terminal = investigationTerminalTools(withConclusion, false)+ if len(terminal) != 1 || terminal[0].Name != agentcapabilities.PatrolConcludeInvestigationToolName {+ t.Fatalf("final-turn terminal tools = %+v, want only patrol_conclude_investigation", terminal)+ }+ if isInvestigationEvidenceTool(agentcapabilities.PatrolConcludeInvestigationToolName) {+ t.Fatal("recording a conclusion must not consume evidence budget")+ }if !isInvestigationEvidenceTool(agentcapabilities.PulseQueryToolName) {t.Fatal("read-only query must consume evidence budget")}diff --git a/internal/ai/chat/agentic_final.go b/internal/ai/chat/agentic_final.goindex c162f673b5..aadc6d7ce0 100644--- a/internal/ai/chat/agentic_final.go+++ b/internal/ai/chat/agentic_final.go@@ -94,8 +94,9 @@ func (a *AgenticLoop) ensureFinalTextResponseWithSystemPrompt(var suppressLeakedToolContent boolvar pendingVisibleContent string- // Keep this bounded so a stuck provider stream doesn't drag the whole request.- summaryCtx, cancel := context.WithTimeout(ctx, 20*time.Second)+ // Keep this bounded so a stuck provider stream doesn't drag the whole+ // request. An investigation may use the time left before its deadline.+ summaryCtx, cancel := context.WithTimeout(ctx, investigationSummaryTimeout(ctx, a.currentExecutionProfile(), 20*time.Second))defer cancel()summaryErr := a.provider.ChatStream(summaryCtx, summaryReq, func(event providers.StreamEvent) {diff --git a/internal/ai/chat/agentic_investigation_budget.go b/internal/ai/chat/agentic_investigation_budget.goindex e2dc990c14..29f3f06bb4 100644--- a/internal/ai/chat/agentic_investigation_budget.go+++ b/internal/ai/chat/agentic_investigation_budget.go@@ -1,6 +1,9 @@package chatimport (+ "context"+ "time"+"github.com/rcourtman/pulse-go-rewrite/internal/agentcapabilities""github.com/rcourtman/pulse-go-rewrite/internal/ai/providers"aitools "github.com/rcourtman/pulse-go-rewrite/internal/ai/tools"@@ -30,11 +33,88 @@ func isInvestigationEvidenceTool(name string) bool {return agentcapabilities.IsPatrolInfrastructureEvidenceToolName(name)}-func investigationTerminalTools(available []providers.Tool) []providers.Tool {+// investigationTerminalTools are the tools that end an investigation: the+// typed action proposal and the typed conclusion. includePropose is false on+// a final turn that historically offered no proposal.+func investigationTerminalTools(available []providers.Tool, includePropose bool) []providers.Tool {+ var terminal []providers.Tool+ seen := make(map[string]bool, 2)for _, tool := range available {- if tool.Name == agentcapabilities.PatrolProposeActionToolName {- return []providers.Tool{tool}+ if seen[tool.Name] {+ continue+ }+ switch tool.Name {+ case agentcapabilities.PatrolProposeActionToolName:+ if !includePropose {+ continue+ }+ case agentcapabilities.PatrolConcludeInvestigationToolName:+ default:+ continue}+ seen[tool.Name] = true+ terminal = append(terminal, tool)+ }+ return terminal+}++const investigationFinalConclusionSystemNote = "\nThis is the investigation's final turn. Do not gather more evidence. Record your conclusion now with patrol_conclude_investigation, based only on the evidence already collected; you will then be asked for the final summary."++// Turn-time estimates for reserving a conclusion before the investigation+// deadline. Concluding takes up to two provider calls: the conclusion turn+// and the final summary.+const (+ investigationMinTurnEstimate = 20 * time.Second+ investigationConclusionTimeMargin = 15 * time.Second+)++// investigationConclusionReserve is how much of the deadline must remain to+// start another evidence turn: that turn plus the conclusion and summary+// turns, each estimated as the slowest turn seen so far, plus a margin. It is+// capped at half the run's budget so short timeouts still gather evidence.+func investigationConclusionReserve(slowestTurn, budget time.Duration) time.Duration {+ reserve := 3*max(slowestTurn, investigationMinTurnEstimate) + investigationConclusionTimeMargin+ if budget > 0 && reserve > budget/2 {+ reserve = budget / 2+ }+ return reserve+}++// patrolSubscriptionInvestigationTurnTimeout is the complete-turn budget for an+// investigation on a subscription agent. That transport buffers the whole turn,+// so Patrol's stream idle timeout is the deadline for the entire turn. Watch+// keeps its short stall budget for latency; an investigation turn starts a CLI+// process, resends the conversation and reasons at high effort.+const patrolSubscriptionInvestigationTurnTimeout = 4 * time.Minute++// investigationStreamIdleTimeout returns the caller-owned provider stall+// budget for one turn. Only an investigation on a buffered subscription+// transport is widened; every other profile and provider keeps base.+func investigationStreamIdleTimeout(profile aitools.ExecutionProfile, providerName string, base time.Duration) time.Duration {+ if !isPatrolInvestigationExecution(profile) || base <= 0 || base >= patrolSubscriptionInvestigationTurnTimeout {+ return base+ }+ switch providers.SubscriptionAgent(providerName) {+ case providers.SubscriptionAgentCodex, providers.SubscriptionAgentClaude:+ return patrolSubscriptionInvestigationTurnTimeout+ default:+ return base+ }+}++// investigationSummaryTimeout bounds the final summary call. Interactive chat+// keeps the historical 20s bound; an investigation may use the time left+// before its deadline, because a slow provider's summary is its result.+func investigationSummaryTimeout(ctx context.Context, profile aitools.ExecutionProfile, base time.Duration) time.Duration {+ if !isPatrolInvestigationExecution(profile) {+ return base+ }+ deadline, ok := ctx.Deadline()+ if !ok {+ return base+ }+ if remaining := time.Until(deadline); remaining > base {+ return remaining}- return nil+ return base}diff --git a/internal/ai/chat/agentic_investigation_budget_w0716_coverage_test.go b/internal/ai/chat/agentic_investigation_budget_w0716_coverage_test.goindex c396f64ffa..4694f449b0 100644--- a/internal/ai/chat/agentic_investigation_budget_w0716_coverage_test.go+++ b/internal/ai/chat/agentic_investigation_budget_w0716_coverage_test.go@@ -71,7 +71,7 @@ func Test_w0716_budget_InvestigationTerminalTools(t *testing.T) {}for _, tt := range tests {t.Run(tt.name, func(t *testing.T) {- got := investigationTerminalTools(tt.available)+ got := investigationTerminalTools(tt.available, true)if !tt.wantFound {if got != nil {t.Fatalf("investigationTerminalTools() = %+v, want nil", got)diff --git a/internal/ai/chat/agentic_investigation_conclusion_test.go b/internal/ai/chat/agentic_investigation_conclusion_test.gonew file mode 100644index 0000000000..3200c60621--- /dev/null+++ b/internal/ai/chat/agentic_investigation_conclusion_test.go@@ -0,0 +1,211 @@+package chat++import (+ "context"+ "strings"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/agentcapabilities"+ "github.com/rcourtman/pulse-go-rewrite/internal/ai/providers"+ "github.com/rcourtman/pulse-go-rewrite/internal/ai/tools"+ "github.com/rcourtman/pulse-go-rewrite/pkg/aicontracts"+)++func investigationConclusionTestTools() []providers.Tool {+ return []providers.Tool{+ {Name: agentcapabilities.PulseQueryToolName},+ {Name: agentcapabilities.PatrolProposeActionToolName},+ {Name: agentcapabilities.PatrolConcludeInvestigationToolName},+ }+}++func newConclusionTestExecutor(capture *tools.ProposalCapture) *tools.PulseToolExecutor {+ executor := tools.NewPulseToolExecutor(tools.ExecutorConfig{StateProvider: &mockStateProvider{}})+ executor.ApplyExecutionProfile(tools.ProfilePatrolInvestigation)+ if capture != nil {+ executor.SetProposalCapture(capture)+ }+ return executor+}++func toolNames(list []providers.Tool) []string {+ names := make([]string, 0, len(list))+ for _, tool := range list {+ names = append(names, tool.Name)+ }+ return names+}++func TestAgenticLoopPatrolInvestigationFinalTurnOffersOnlyTheConclusion(t *testing.T) {+ provider := &stubStreamingProvider{}+ var requests []providers.ChatRequest+ provider.chatStream = func(_ context.Context, req providers.ChatRequest, callback providers.StreamCallback) error {+ requests = append(requests, req)+ if len(requests) == 1 {+ callback(providers.StreamEvent{Type: "done", Data: providers.DoneEvent{ToolCalls: []providers.ToolCall{{ID: "evidence-1", Name: agentcapabilities.PulseQueryToolName, Input: map[string]interface{}{"action": "search", "query": "disks"}}}}})+ return nil+ }+ callback(providers.StreamEvent{Type: "content", Data: providers.ContentEvent{Text: "final investigation summary"}})+ callback(providers.StreamEvent{Type: "done", Data: providers.DoneEvent{}})+ return nil+ }++ loop := NewAgenticLoop(provider, newConclusionTestExecutor(nil), "system")+ loop.SetExecutionProfile(tools.ProfilePatrolInvestigation)+ loop.SetMaxTurns(2)++ if _, err := loop.ExecuteWithTools(context.Background(), "final-turn", []Message{{Role: "user", Content: "investigate"}}, investigationConclusionTestTools(), func(StreamEvent) {}); err != nil {+ t.Fatalf("ExecuteWithTools: %v", err)+ }+ if len(requests) != 2 {+ t.Fatalf("provider requests = %d, want 2", len(requests))+ }+ if names := toolNames(requests[1].Tools); len(names) != 1 || names[0] != agentcapabilities.PatrolConcludeInvestigationToolName {+ t.Fatalf("final-turn tools = %v, want only patrol_conclude_investigation (no proposal before the evidence budget is spent)", names)+ }+ if !strings.Contains(requests[1].System, "final turn") {+ t.Fatalf("final-turn system prompt missing the conclusion instruction: %q", requests[1].System)+ }+}++func TestAgenticLoopPatrolInvestigationAsksForSummaryAfterConclusion(t *testing.T) {+ provider := &stubStreamingProvider{}+ var requests []providers.ChatRequest+ provider.chatStream = func(_ context.Context, req providers.ChatRequest, callback providers.StreamCallback) error {+ requests = append(requests, req)+ if len(requests) == 1 {+ callback(providers.StreamEvent{Type: "done", Data: providers.DoneEvent{ToolCalls: []providers.ToolCall{{+ ID: "conclude-1",+ Name: agentcapabilities.PatrolConcludeInvestigationToolName,+ Input: map[string]interface{}{"outcome": "needs_attention", "reason": "the pool needs a replacement disk"},+ }}}})+ return nil+ }+ callback(providers.StreamEvent{Type: "content", Data: providers.ContentEvent{Text: "### Investigation Summary\nThe pool is degraded."}})+ callback(providers.StreamEvent{Type: "done", Data: providers.DoneEvent{}})+ return nil+ }++ capture := tools.NewProposalCapture(tools.ProposalIdentity{ProposalID: "p1", FindingID: "f1", InvestigationID: "i1"}, nil)+ loop := NewAgenticLoop(provider, newConclusionTestExecutor(capture), "system")+ loop.SetExecutionProfile(tools.ProfilePatrolInvestigation)+ loop.SetMaxTurns(6)++ result, err := loop.ExecuteWithTools(context.Background(), "conclude-then-summarize", []Message{{Role: "user", Content: "investigate"}}, investigationConclusionTestTools(), func(StreamEvent) {})+ if err != nil {+ t.Fatalf("ExecuteWithTools: %v", err)+ }+ conclusion := capture.Conclusion()+ if conclusion == nil || conclusion.Outcome != aicontracts.OutcomeNeedsAttention {+ t.Fatalf("recorded conclusion = %+v, want needs_attention", conclusion)+ }+ if len(requests) != 2 {+ t.Fatalf("provider requests = %d, want 2 (conclusion, then summary)", len(requests))+ }+ if len(requests[1].Tools) != 0 {+ t.Fatalf("post-conclusion tools = %v, want none: the next response is the summary", toolNames(requests[1].Tools))+ }+ if len(result) == 0 || !strings.Contains(result[len(result)-1].Content, "The pool is degraded.") {+ t.Fatalf("final result = %+v, want the summary", result)+ }+}++func TestAgenticLoopPatrolInvestigationConcludesBeforeItsDeadline(t *testing.T) {+ provider := &stubStreamingProvider{}+ var requests []providers.ChatRequest+ provider.chatStream = func(_ context.Context, req providers.ChatRequest, callback providers.StreamCallback) error {+ requests = append(requests, req)+ if len(requests) == 1 {+ // A slow evidence turn leaves less than the conclusion reserve.+ time.Sleep(1100 * time.Millisecond)+ callback(providers.StreamEvent{Type: "done", Data: providers.DoneEvent{ToolCalls: []providers.ToolCall{{ID: "evidence-1", Name: agentcapabilities.PulseQueryToolName, Input: map[string]interface{}{"action": "search", "query": "disks"}}}}})+ return nil+ }+ callback(providers.StreamEvent{Type: "content", Data: providers.ContentEvent{Text: "final investigation summary"}})+ callback(providers.StreamEvent{Type: "done", Data: providers.DoneEvent{}})+ return nil+ }++ loop := NewAgenticLoop(provider, newConclusionTestExecutor(nil), "system")+ loop.SetExecutionProfile(tools.ProfilePatrolInvestigation)+ loop.SetMaxTurns(12)++ // A 2s budget reserves 1s (half the budget) to conclude.+ ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)+ defer cancel()+ if _, err := loop.ExecuteWithTools(ctx, "deadline-reserve", []Message{{Role: "user", Content: "investigate"}}, investigationConclusionTestTools(), func(StreamEvent) {}); err != nil {+ t.Fatalf("ExecuteWithTools: %v", err)+ }+ if len(requests) != 2 {+ t.Fatalf("provider requests = %d, want 2", len(requests))+ }+ if names := toolNames(requests[1].Tools); len(names) != 1 || names[0] != agentcapabilities.PatrolConcludeInvestigationToolName {+ t.Fatalf("tools after the reserve was reached = %v, want only patrol_conclude_investigation", names)+ }+}++func TestInvestigationConclusionReserve(t *testing.T) {+ cases := []struct {+ name string+ slowest time.Duration+ budget time.Duration+ want time.Duration+ }{+ {name: "no turn seen yet uses the minimum estimate", slowest: 0, budget: 10 * time.Minute, want: 75 * time.Second},+ // Another evidence turn plus the conclusion and summary turns.+ {name: "slow subscription turns reserve three of them", slowest: 90 * time.Second, budget: 25 * time.Minute, want: 285 * time.Second},+ {name: "near-limit subscription turns still fit", slowest: 225 * time.Second, budget: 25 * time.Minute, want: 690 * time.Second},+ {name: "short budgets keep half for evidence", slowest: 0, budget: time.Minute, want: 30 * time.Second},+ }+ for _, tc := range cases {+ if got := investigationConclusionReserve(tc.slowest, tc.budget); got != tc.want {+ t.Fatalf("%s: reserve = %s, want %s", tc.name, got, tc.want)+ }+ }+}++func TestInvestigationStreamIdleTimeoutWidensOnlySubscriptionInvestigationTurns(t *testing.T) {+ base := PatrolProviderStreamIdleTimeout+ cases := []struct {+ name string+ profile tools.ExecutionProfile+ provider string+ want time.Duration+ }{+ {name: "codex subscription investigation", profile: tools.ProfilePatrolInvestigation, provider: string(providers.SubscriptionAgentCodex), want: patrolSubscriptionInvestigationTurnTimeout},+ {name: "claude subscription investigation", profile: tools.ProfilePatrolInvestigation, provider: string(providers.SubscriptionAgentClaude), want: patrolSubscriptionInvestigationTurnTimeout},+ {name: "subscription Watch keeps its stall budget", profile: tools.ProfilePatrolDetection, provider: string(providers.SubscriptionAgentCodex), want: base},+ {name: "streaming provider investigation keeps its idle budget", profile: tools.ProfilePatrolInvestigation, provider: "anthropic", want: base},+ }+ for _, tc := range cases {+ if got := investigationStreamIdleTimeout(tc.profile, tc.provider, base); got != tc.want {+ t.Fatalf("%s: stream idle timeout = %s, want %s", tc.name, got, tc.want)+ }+ }+ if got := investigationStreamIdleTimeout(tools.ProfileInteractiveAssistant, string(providers.SubscriptionAgentCodex), 0); got != 0 {+ t.Fatalf("interactive runs without a stall budget must stay unbounded, got %s", got)+ }+}++func TestInvestigationSummaryTimeoutUsesTheRemainingDeadline(t *testing.T) {+ base := 20 * time.Second+ if got := investigationSummaryTimeout(context.Background(), tools.ProfilePatrolInvestigation, base); got != base {+ t.Fatalf("no deadline: timeout = %s, want %s", got, base)+ }++ long, cancelLong := context.WithTimeout(context.Background(), 5*time.Minute)+ defer cancelLong()+ if got := investigationSummaryTimeout(long, tools.ProfilePatrolInvestigation, base); got <= 4*time.Minute {+ t.Fatalf("investigation summary timeout = %s, want the remaining ~5m", got)+ }+ if got := investigationSummaryTimeout(long, tools.ProfileInteractiveAssistant, base); got != base {+ t.Fatalf("interactive summary timeout = %s, want %s", got, base)+ }++ short, cancelShort := context.WithTimeout(context.Background(), 5*time.Second)+ defer cancelShort()+ if got := investigationSummaryTimeout(short, tools.ProfilePatrolInvestigation, base); got != base {+ t.Fatalf("near-deadline summary timeout = %s, want %s (the deadline still bounds it)", got, base)+ }+}diff --git a/internal/ai/chat/service_investigation.go b/internal/ai/chat/service_investigation.goindex aa85733fd5..70997806ae 100644--- a/internal/ai/chat/service_investigation.go+++ b/internal/ai/chat/service_investigation.go@@ -53,6 +53,10 @@ type InvestigationRunResult struct {ModelTurns intEvidenceCalls intToolCalls int+ // Conclusion is the typed conclusion the model recorded with+ // patrol_conclude_investigation, nil when it recorded none. It survives+ // a later provider failure, like a persisted proposal.+ Conclusion *tools.CapturedConclusion}// ExecuteInvestigationStream runs one Patrol investigation under the@@ -193,6 +197,7 @@ func (s *Service) ExecuteInvestigationStream(ctx context.Context, req Investigatresult := &InvestigationRunResult{Content: content,Proposal: proposal,+ Conclusion: capture.Conclusion(),InputTokens: loop.GetTotalInputTokens(),OutputTokens: loop.GetTotalOutputTokens(),ModelTurns: loop.GetTotalModelTurns(),diff --git a/internal/ai/findings.go b/internal/ai/findings.goindex 1c0fefb1cb..4126defa10 100644--- a/internal/ai/findings.go+++ b/internal/ai/findings.go@@ -137,6 +137,8 @@ const (InvestigationOutcomeNeedsAttention InvestigationOutcome = "needs_attention" // Requires user interventionInvestigationOutcomeCannotFix InvestigationOutcome = "cannot_fix" // AI determined it cannot fix thisInvestigationOutcomeTimedOut InvestigationOutcome = "timed_out" // Transient timeout, will retry sooner+ InvestigationOutcomeUnclassified InvestigationOutcome = "unclassified" // Finished, but its conclusion could not be read+ InvestigationOutcomeInterrupted InvestigationOutcome = "interrupted" // Cancelled by a restart or shutdown, will retry)// FindingLifecycleEvent represents an append-only log entry of finding lifecycle transitions.@@ -457,6 +459,9 @@ func investigationCooldownForOutcome(outcome string) time.Duration {return 0case InvestigationOutcomeTimedOut:return investigationTimeoutCooldown+ case InvestigationOutcomeInterrupted:+ // A restart cut the run short; nothing about the finding changed.+ return 0case InvestigationOutcomeFixFailed, InvestigationOutcomeFixVerificationFailed:return investigationFixRetryCooldowndefault:@@ -466,7 +471,9 @@ func investigationCooldownForOutcome(outcome string) time.Duration {func (f *Finding) investigationCooldown() time.Duration {if InvestigationStatus(f.InvestigationStatus) == InvestigationStatusFailed &&- (f.InvestigationOutcome == "" || InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeTimedOut) {+ (f.InvestigationOutcome == "" ||+ InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeTimedOut ||+ InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeInterrupted) {// The run failed before producing a conclusion. The scheduled Watch// cadence owns the next assessment; there is no additional cooldown.// Admission, attempt limits and action authority still apply.@@ -505,7 +512,9 @@ func deriveLoopState(f *Finding) FindingLoopState {return FindingLoopStateRemediatingcase InvestigationOutcomeFixFailed, InvestigationOutcomeFixVerificationFailed:return FindingLoopStateRemediationFailed- case InvestigationOutcomeFixRejected, InvestigationOutcomeFixVerificationUnknown, InvestigationOutcomeNeedsAttention, InvestigationOutcomeCannotFix:+ case InvestigationOutcomeFixRejected, InvestigationOutcomeFixVerificationUnknown, InvestigationOutcomeNeedsAttention, InvestigationOutcomeCannotFix, InvestigationOutcomeUnclassified:+ // An unclassified run still left a summary for the operator to read;+ // unlike needs_attention it stays eligible for another run.return FindingLoopStateNeedsAttentioncase InvestigationOutcomeTimedOut:return FindingLoopStateTimedOut@@ -2679,6 +2688,50 @@ func (s *FindingsStore) UpdateInvestigation(id, sessionID, status, outcome strinreturn true}+// RecoverStuckInvestigation closes a run that is no longer live with the given+// status and outcome, but only while the finding still shows that run as+// running, so a result or a newer run that landed since the caller looked is+// never overwritten. runStartedAt is the start of the run's session, or zero+// when the orchestrator holds none. A run that started before the finding's+// current occurrence is only recorded as interrupted: its result is not this+// occurrence's. A run starts with no outcome, so an outcome already present+// came from the action lifecycle during this run; it is kept, and the run+// completes, so a queued fix still blocks another investigation. The close+// becomes the finding's latest investigation time.+func (s *FindingsStore) RecoverStuckInvestigation(id, sessionID string, runStartedAt time.Time, status InvestigationStatus, outcome InvestigationOutcome, at time.Time) bool {+ s.mu.Lock()+ f, exists := s.findings[id]+ if !exists || f.InvestigationStatus != string(InvestigationStatusRunning) || f.InvestigationSessionID != sessionID {+ s.mu.Unlock()+ return false+ }+ if !runStartedAt.IsZero() && !f.acceptsInvestigationResult(sessionID, runStartedAt) {+ status, outcome = InvestigationStatusFailed, InvestigationOutcomeInterrupted+ }+ prevStatus := f.InvestigationStatus+ prevOutcome := f.InvestigationOutcome+ if prevOutcome != "" {+ status = InvestigationStatusCompleted+ } else {+ f.InvestigationOutcome = string(outcome)+ if outcome == InvestigationOutcomeFixVerified {+ s.resolveVerifiedFindingLocked(f, prevOutcome)+ }+ }+ f.InvestigationStatus = string(status)+ f.LastInvestigatedAt = &at+ s.appendLifecycleLocked(f, "investigation_updated", "Investigation state updated", f.LoopState, f.LoopState, map[string]string{+ "prev_status": prevStatus,+ "next_status": string(status),+ "prev_outcome": prevOutcome,+ "next_outcome": f.InvestigationOutcome,+ })+ s.syncLoopStateLocked(f)+ s.mu.Unlock()+ s.scheduleSave()+ return true+}+// resolveVerifiedFindingLocked projects the first verified resolution without// moving its timestamp or appending another verification when history is replayed.// A genuine regression clears ResolvedAt and can therefore resolve again.diff --git a/internal/ai/findings_test.go b/internal/ai/findings_test.goindex 5edd24d25e..0210625557 100644--- a/internal/ai/findings_test.go+++ b/internal/ai/findings_test.go@@ -2443,3 +2443,43 @@ func TestFindingsStore_OperatorStateProjectionFor_RejectsEmptyResourceID(t *testt.Error("expected false on empty resource id")}}++func newEligibleInvestigationFinding(status InvestigationStatus, outcome InvestigationOutcome, lastInvestigated time.Time) *Finding {+ return &Finding{+ ID: "eligible-" + string(outcome),+ Severity: FindingSeverityWarning,+ InvestigationStatus: string(status),+ InvestigationOutcome: string(outcome),+ InvestigationAttempts: 1,+ LastInvestigatedAt: &lastInvestigated,+ }+}++func TestInterruptedAndUnclassifiedInvestigationsStayEligible(t *testing.T) {+ now := time.Now()++ interrupted := newEligibleInvestigationFinding(InvestigationStatusFailed, InvestigationOutcomeInterrupted, now)+ if !interrupted.ShouldInvestigate("approval") || !interrupted.CanRetryInvestigation() {+ t.Fatal("a run cut short by a restart must be retried at the next scheduled check")+ }+ if got := deriveLoopState(interrupted); got != FindingLoopStateDetected {+ t.Fatalf("interrupted loop state = %q, want %q", got, FindingLoopStateDetected)+ }++ unclassified := newEligibleInvestigationFinding(InvestigationStatusCompleted, InvestigationOutcomeUnclassified, now)+ if got := deriveLoopState(unclassified); got != FindingLoopStateNeedsAttention {+ t.Fatalf("unclassified loop state = %q, want %q", got, FindingLoopStateNeedsAttention)+ }+ if unclassified.ShouldInvestigate("approval") {+ t.Fatal("an unclassified run must still respect the normal cooldown")+ }+ unclassified = newEligibleInvestigationFinding(InvestigationStatusCompleted, InvestigationOutcomeUnclassified, now.Add(-2*time.Hour))+ if !unclassified.ShouldInvestigate("approval") || !unclassified.CanRetryInvestigation() {+ t.Fatal("an unreadable conclusion must not become terminal the way needs_attention is")+ }++ needsAttention := newEligibleInvestigationFinding(InvestigationStatusCompleted, InvestigationOutcomeNeedsAttention, now.Add(-2*time.Hour))+ if needsAttention.ShouldInvestigate("approval") {+ t.Fatal("a real needs_attention conclusion stays terminal")+ }+}diff --git a/internal/ai/investigation_records.go b/internal/ai/investigation_records.goindex f4ee5ab0a2..bb20509e4c 100644--- a/internal/ai/investigation_records.go+++ b/internal/ai/investigation_records.go@@ -203,7 +203,8 @@ func deriveInvestigationRecordConfidence(record *aicontracts.InvestigationRecordreturn aicontracts.InvestigationRecordConfidenceHighcase aicontracts.OutcomeFixQueued, aicontracts.OutcomeFixExecuted, aicontracts.OutcomeCannotFix, aicontracts.OutcomeNeedsAttention:return aicontracts.InvestigationRecordConfidenceMedium- case aicontracts.OutcomeFixRejected, aicontracts.OutcomeFixFailed, aicontracts.OutcomeFixVerificationFailed, aicontracts.OutcomeFixVerificationUnknown, aicontracts.OutcomeTimedOut:+ case aicontracts.OutcomeFixRejected, aicontracts.OutcomeFixFailed, aicontracts.OutcomeFixVerificationFailed, aicontracts.OutcomeFixVerificationUnknown, aicontracts.OutcomeTimedOut,+ aicontracts.OutcomeUnclassified, aicontracts.OutcomeInterrupted:return aicontracts.InvestigationRecordConfidenceLow}if strings.TrimSpace(record.Conclusion) != "" || len(record.Evidence) > 0 {diff --git a/internal/ai/patrol_digest.go b/internal/ai/patrol_digest.goindex 1a97e19772..ff9cdaf554 100644--- a/internal/ai/patrol_digest.go+++ b/internal/ai/patrol_digest.go@@ -430,12 +430,14 @@ func PatrolDigestOutcomeOrder(byOutcome map[string]int) []string {"fix_verification_failed": 2,"cannot_fix": 3,"timed_out": 4,- "fix_queued": 5,- "fix_executed": 6,- "fix_verification_unknown": 7,- "fix_verified": 8,- "resolved": 9,- "fix_rejected": 10,+ "unclassified": 5,+ "fix_queued": 6,+ "fix_executed": 7,+ "fix_verification_unknown": 8,+ "fix_verified": 9,+ "resolved": 10,+ "fix_rejected": 11,+ "interrupted": 12,}keys := make([]string, 0, len(byOutcome))for key := range byOutcome {diff --git a/internal/ai/patrol_findings.go b/internal/ai/patrol_findings.goindex 86f47d42d7..adad51a674 100644--- a/internal/ai/patrol_findings.go+++ b/internal/ai/patrol_findings.go@@ -1551,42 +1551,49 @@ func normalizeFindingKey(key string) string {return normalized}-// recoverStuckInvestigations detects findings stuck in "running" state for longer than-// the investigation timeout and resets them to "failed/timed_out" so they can be retried.-// This handles the case where an investigation goroutine panics or is killed without-// properly updating the finding status.+// investigationRunGrace covers work after the orchestrator's own deadline,+// such as recording the failure, before Patrol treats a run as abandoned.+const investigationRunGrace = 5 * time.Minute++// investigationRunCeiling is the longest one investigation may legitimately+// run: the effective configured timeout plus grace.+func (p *PatrolService) investigationRunCeiling() time.Duration {+ timeout := aicontracts.DefaultInvestigationConfig().Timeout+ if p.aiService != nil {+ if cfg := p.aiService.GetConfig(); cfg != nil {+ timeout = cfg.GetPatrolInvestigationTimeout()+ }+ }+ return timeout + investigationRunGrace+}++// recoverStuckInvestigations releases findings left in "running" by a run that+// is no longer live, so they can be investigated again. This handles an+// investigation goroutine that panicked or hung, and a run cut short by a+// restart before it could update the finding.func (p *PatrolService) recoverStuckInvestigations() {if p.findings == nil {return}- const stuckThreshold = 15 * time.Minute // investigation timeout is 10min; allow 5min grace+ ceiling := p.investigationRunCeiling()+ orchestrator := p.GetInvestigationOrchestrator()+ now := time.Now()active := p.findings.GetActive(FindingSeverityWarning)recovered := 0for _, f := range active {if f.InvestigationStatus != string(InvestigationStatusRunning) {continue}- if f.LastInvestigatedAt == nil {+ result, stuck := stuckInvestigationResult(f, orchestrator, ceiling, now)+ if !stuck || !p.findings.RecoverStuckInvestigation(f.ID, f.InvestigationSessionID, result.runStartedAt, result.status, result.outcome, now) {continue}- if time.Since(*f.LastInvestigatedAt) < stuckThreshold {- continue- }- // This finding has been "running" for too long — reset it- p.findings.UpdateInvestigation(- f.ID,- f.InvestigationSessionID,- string(InvestigationStatusFailed),- string(InvestigationOutcomeTimedOut),- f.LastInvestigatedAt,- f.InvestigationAttempts,- )recovered++log.Warn().Str("finding_id", f.ID).Str("resource", f.ResourceName).- Time("last_investigated", *f.LastInvestigatedAt).- Msg("AI Patrol: Recovered stuck investigation (exceeded timeout)")+ Str("outcome", string(result.outcome)).+ Msg("AI Patrol: Recovered stuck investigation")}if recovered > 0 {log.Info().Int("recovered", recovered).@@ -1594,6 +1601,107 @@ func (p *PatrolService) recoverStuckInvestigations() {}}+// stuckInvestigation is how to close a run that is no longer live.+// runStartedAt is the start of the run's session, zero when the orchestrator+// holds none, so the store can check the run belongs to the finding's current+// occurrence before it applies the session's result.+type stuckInvestigation struct {+ status InvestigationStatus+ outcome InvestigationOutcome+ runStartedAt time.Time+}++// stuckInvestigationResult reports whether a finding marked as running has a+// run that is no longer live, and how to close it. The orchestrator's session+// for that run decides liveness and, once the run is closed, its result.+// Restarts are recorded as interrupted rather than timed out, but the run+// still spends its attempt: a run that crashes Pulse must not retry forever.+func stuckInvestigationResult(f *Finding, orchestrator InvestigationOrchestrator, ceiling time.Duration, now time.Time) (stuckInvestigation, bool) {+ if orchestrator == nil {+ // No session to consult: measure from the last recorded investigation.+ if f.LastInvestigatedAt == nil || now.Sub(*f.LastInvestigatedAt) < ceiling {+ return stuckInvestigation{}, false+ }+ return stuckInvestigation{status: InvestigationStatusFailed, outcome: InvestigationOutcomeTimedOut}, true+ }+ session, known := recordedInvestigationSession(orchestrator, f)+ if !known {+ return stuckInvestigation{}, false+ }+ if session == nil {+ // The orchestrator records its session before it marks the finding as+ // running, so a run it holds no record of ended with an earlier process.+ return stuckInvestigation{status: InvestigationStatusFailed, outcome: InvestigationOutcomeInterrupted}, true+ }+ closed := stuckInvestigation{status: InvestigationStatusFailed, outcome: InvestigationOutcomeTimedOut, runStartedAt: session.StartedAt}+ switch session.Status {+ case aicontracts.InvestigationStatusPending, aicontracts.InvestigationStatusRunning:+ // A live run keeps the timeout it started with, so a timeout lowered+ // since then does not cut it short.+ limit := session.StartedAt.Add(ceiling)+ if session.Deadline != nil {+ limit = session.Deadline.Add(investigationRunGrace)+ }+ if now.Before(limit) {+ return stuckInvestigation{}, false+ }+ return closed, true+ }+ if session.Outcome == aicontracts.OutcomeInterrupted {+ // Closed when the orchestrator restarted, or cancelled by shutdown.+ closed.outcome = InvestigationOutcomeInterrupted+ return closed, true+ }+ // The orchestrator closed the run without updating the finding. Its own+ // update follows the close at once, so give it time to land first.+ closedAt := session.StartedAt+ if session.CompletedAt != nil {+ closedAt = *session.CompletedAt+ }+ if now.Sub(closedAt) < investigationRunGrace {+ return stuckInvestigation{}, false+ }+ if session.Outcome != "" {+ // The session holds the run's result; the finding never received it.+ // Without one the run closed with no result, such as past its limit.+ closed.status = InvestigationStatus(session.Status)+ closed.outcome = InvestigationOutcome(session.Outcome)+ }+ return closed, true+}++// investigationStoreProvider is implemented by orchestrators that expose their+// session store, which holds every recorded run of a finding.+type investigationStoreProvider interface {+ GetStore() aicontracts.InvestigationStore+}++// recordedInvestigationSession returns the orchestrator's session for the run+// a finding names, or nil when the orchestrator holds no record of it. known+// is false when the orchestrator cannot tell: it does not expose its store,+// and a different run is the finding's latest.+func recordedInvestigationSession(orchestrator InvestigationOrchestrator, f *Finding) (session *InvestigationSession, known bool) {+ if provider, ok := orchestrator.(investigationStoreProvider); ok {+ if store := provider.GetStore(); store != nil {+ for _, recorded := range store.GetByFinding(f.ID) {+ if recorded != nil && recorded.SessionID == f.InvestigationSessionID {+ return recorded, true+ }+ }+ return nil, true+ }+ }+ latest := orchestrator.GetInvestigationByFinding(f.ID)+ switch {+ case latest == nil:+ return nil, true+ case latest.SessionID == f.InvestigationSessionID:+ return latest, true+ default:+ return nil, false+ }+}+// retryTimedOutInvestigations re-triggers investigation for findings that failed due to timeout.// Called at the end of each patrol run to give timed-out investigations another chance// without waiting for the full 1-hour cooldown.@@ -1767,7 +1875,10 @@ func (p *PatrolService) maybeInvestigateFinding(f *Finding) bool {}currentAutonomy := aiService.GetEffectivePatrolAutonomyLevel()- ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)+ // The orchestrator enforces the configured investigation timeout; this+ // outer deadline is a safety net, so it follows that timeout rather than+ // cutting longer configured or subscription-agent runs short.+ ctx, cancel := context.WithTimeout(context.Background(), currentCfg.GetPatrolInvestigationTimeout()+investigationRunGrace)defer cancel()if err := orchestrator.InvestigateFinding(ctx, invFinding, currentAutonomy); err != nil {log.Error().diff --git a/internal/ai/patrol_investigation_limits_test.go b/internal/ai/patrol_investigation_limits_test.gonew file mode 100644index 0000000000..205e20b95e--- /dev/null+++ b/internal/ai/patrol_investigation_limits_test.go@@ -0,0 +1,295 @@+package ai++import (+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/config"+ "github.com/rcourtman/pulse-go-rewrite/pkg/aicontracts"+)++type sessionReportingOrchestrator struct {+ stubInvestigationOrchestrator+ sessions map[string]*InvestigationSession+ store aicontracts.InvestigationStore+}++func (s *sessionReportingOrchestrator) GetInvestigationByFinding(findingID string) *InvestigationSession {+ return s.sessions[findingID]+}++func (s *sessionReportingOrchestrator) GetStore() aicontracts.InvestigationStore {+ return s.store+}++// findingSessionsStore is a session store that only answers which sessions a+// finding has recorded, the one query the stuck-run sweep makes.+type findingSessionsStore struct {+ aicontracts.InvestigationStore+ byFinding map[string][]*InvestigationSession+}++func (s findingSessionsStore) GetByFinding(findingID string) []*InvestigationSession {+ return s.byFinding[findingID]+}++func runningInvestigationFinding(id, sessionID string, lastInvestigated *time.Time) *Finding {+ return &Finding{+ ID: id,+ ResourceID: "vm:" + id,+ ResourceType: "vm",+ Title: "Investigation " + id,+ Severity: FindingSeverityWarning,+ Category: FindingCategoryReliability,+ InvestigationSessionID: sessionID,+ InvestigationStatus: string(InvestigationStatusRunning),+ LastInvestigatedAt: lastInvestigated,+ InvestigationAttempts: 1,+ }+}++func assertInvestigationState(t *testing.T, findings *FindingsStore, id string, status InvestigationStatus, outcome InvestigationOutcome) {+ t.Helper()+ got := findings.Get(id)+ if got.InvestigationStatus != string(status) || got.InvestigationOutcome != string(outcome) {+ t.Fatalf("%s = %s/%s, want %s/%s", id, got.InvestigationStatus, got.InvestigationOutcome, status, outcome)+ }+}++func TestRecoverStuckInvestigationsMeasuresLiveRunsFromTheirOwnStart(t *testing.T) {+ now := time.Now()+ previousRun := now.Add(-2 * time.Hour)+ findings := NewFindingsStore()+ // A re-investigation still running: LastInvestigatedAt is the previous run.+ findings.Add(runningInvestigationFinding("live", "chat-live", &previousRun))+ // A first investigation whose run outlived its ceiling.+ findings.Add(runningInvestigationFinding("stuck", "chat-stuck", nil))++ orchestrator := &sessionReportingOrchestrator{sessions: map[string]*InvestigationSession{+ "live": {ID: "inv-live", FindingID: "live", SessionID: "chat-live", Status: aicontracts.InvestigationStatusRunning, StartedAt: now.Add(-time.Minute)},+ "stuck": {ID: "inv-stuck", FindingID: "stuck", SessionID: "chat-stuck", Status: aicontracts.InvestigationStatusRunning, StartedAt: now.Add(-time.Hour)},+ }}+ patrol := &PatrolService{findings: findings, investigationOrchestrator: orchestrator}+ patrol.recoverStuckInvestigations()++ assertInvestigationState(t, findings, "live", InvestigationStatusRunning, "")+ assertInvestigationState(t, findings, "stuck", InvestigationStatusFailed, InvestigationOutcomeTimedOut)+}++func TestRecoverStuckInvestigationsKeepsALiveRunToItsOwnDeadline(t *testing.T) {+ now := time.Now()+ findings := NewFindingsStore()+ findings.Add(runningInvestigationFinding("long", "chat-long", nil))+ findings.Add(runningInvestigationFinding("overdue", "chat-overdue", nil))++ // Both runs started 16 minutes ago under a 25-minute timeout that has+ // since been lowered to the 10-minute default, a 15-minute ceiling.+ started := now.Add(-16 * time.Minute)+ longDeadline := started.Add(25 * time.Minute)+ overdueDeadline := now.Add(-investigationRunGrace - time.Minute)+ orchestrator := &sessionReportingOrchestrator{sessions: map[string]*InvestigationSession{+ "long": {ID: "inv-long", FindingID: "long", SessionID: "chat-long", Status: aicontracts.InvestigationStatusRunning, StartedAt: started, Deadline: &longDeadline},+ "overdue": {ID: "inv-overdue", FindingID: "overdue", SessionID: "chat-overdue", Status: aicontracts.InvestigationStatusRunning, StartedAt: started, Deadline: &overdueDeadline},+ }}+ patrol := &PatrolService{findings: findings, investigationOrchestrator: orchestrator}+ patrol.recoverStuckInvestigations()++ assertInvestigationState(t, findings, "long", InvestigationStatusRunning, "")+ assertInvestigationState(t, findings, "overdue", InvestigationStatusFailed, InvestigationOutcomeTimedOut)+}++func TestRecoverStuckInvestigationsRecordsRestartsAsInterrupted(t *testing.T) {+ now := time.Now()+ previousRun := now.Add(-2 * time.Hour)+ justClosed := now.Add(-time.Second)+ closedAWhileAgo := now.Add(-investigationRunGrace - time.Minute)+ findings := NewFindingsStore()+ // Closed as interrupted when the orchestrator reloaded after a restart.+ findings.Add(runningInvestigationFinding("restarted", "chat-restarted", &previousRun))+ // A first run the orchestrator holds no record of. Before, nothing ever+ // released it, which also blocked a manual re-run.+ findings.Add(runningInvestigationFinding("orphan", "chat-gone", nil))+ // Closed with a result moments ago; the orchestrator's own update is due.+ findings.Add(runningInvestigationFinding("settling", "chat-settling", nil))+ // Closed without a result long enough ago that no update is coming.+ findings.Add(runningInvestigationFinding("abandoned", "chat-abandoned", nil))++ orchestrator := &sessionReportingOrchestrator{sessions: map[string]*InvestigationSession{+ "restarted": {ID: "inv-restarted", FindingID: "restarted", SessionID: "chat-restarted", Status: aicontracts.InvestigationStatusFailed, Outcome: aicontracts.OutcomeInterrupted, StartedAt: now.Add(-3 * time.Minute), CompletedAt: &justClosed},+ "settling": {ID: "inv-settling", FindingID: "settling", SessionID: "chat-settling", Status: aicontracts.InvestigationStatusCompleted, Outcome: aicontracts.OutcomeResolved, StartedAt: now.Add(-3 * time.Minute), CompletedAt: &justClosed},+ "abandoned": {ID: "inv-abandoned", FindingID: "abandoned", SessionID: "chat-abandoned", Status: aicontracts.InvestigationStatusFailed, StartedAt: now.Add(-time.Hour), CompletedAt: &closedAWhileAgo},+ }}+ patrol := &PatrolService{findings: findings, investigationOrchestrator: orchestrator}+ patrol.recoverStuckInvestigations()++ assertInvestigationState(t, findings, "restarted", InvestigationStatusFailed, InvestigationOutcomeInterrupted)+ assertInvestigationState(t, findings, "orphan", InvestigationStatusFailed, InvestigationOutcomeInterrupted)+ assertInvestigationState(t, findings, "settling", InvestigationStatusRunning, "")+ assertInvestigationState(t, findings, "abandoned", InvestigationStatusFailed, InvestigationOutcomeTimedOut)+ // The interrupted run still spends its attempt, so a run that crashes+ // Pulse cannot retry forever.+ if got := findings.Get("orphan").InvestigationAttempts; got != 1 {+ t.Fatalf("orphan attempts = %d, want 1", got)+ }+ if !findings.Get("orphan").ShouldInvestigate("approval") {+ t.Fatal("an interrupted first run must be eligible for another investigation")+ }+ // The close counts as the latest investigation, so the digest sees it.+ if got := findings.Get("orphan").LastInvestigatedAt; got == nil || got.Before(now) {+ t.Fatalf("orphan LastInvestigatedAt = %v, want the recovery time", got)+ }+}++func TestRecoverStuckInvestigationsKeepsAnActionOutcome(t *testing.T) {+ now := time.Now()+ closedAWhileAgo := now.Add(-investigationRunGrace - time.Minute)+ findings := NewFindingsStore()+ for _, id := range []string{"queued", "queued-interrupted"} {+ f := runningInvestigationFinding(id, "chat-"+id, nil)+ // The action lifecycle recorded the queued fix while the run was+ // still running, and a restart came before the run finished.+ f.InvestigationOutcome = string(InvestigationOutcomeFixQueued)+ findings.Add(f)+ }+ orchestrator := &sessionReportingOrchestrator{sessions: map[string]*InvestigationSession{+ "queued": {ID: "inv-queued", FindingID: "queued", SessionID: "chat-queued", Status: aicontracts.InvestigationStatusFailed, Outcome: aicontracts.OutcomeFixQueued, StartedAt: now.Add(-time.Hour), CompletedAt: &closedAWhileAgo},+ "queued-interrupted": {ID: "inv-queued-interrupted", FindingID: "queued-interrupted", SessionID: "chat-queued-interrupted", Status: aicontracts.InvestigationStatusFailed, Outcome: aicontracts.OutcomeInterrupted, StartedAt: now.Add(-time.Hour), CompletedAt: &closedAWhileAgo},+ }}+ patrol := &PatrolService{findings: findings, investigationOrchestrator: orchestrator}+ patrol.recoverStuckInvestigations()++ for _, id := range []string{"queued", "queued-interrupted"} {+ assertInvestigationState(t, findings, id, InvestigationStatusCompleted, InvestigationOutcomeFixQueued)+ if findings.Get(id).ShouldInvestigate("approval") {+ t.Fatalf("%s: a queued fix must keep blocking another investigation", id)+ }+ }+}++func TestRecoverStuckInvestigationOnlyClosesTheRunItWasGiven(t *testing.T) {+ now := time.Now()+ findings := NewFindingsStore()+ // A newer run started after the sweep read the finding.+ findings.Add(runningInvestigationFinding("moved-on", "chat-newer", nil))+ // The run finished on its own after the sweep read the finding.+ finished := runningInvestigationFinding("finished", "chat-finished", nil)+ finished.InvestigationStatus = string(InvestigationStatusCompleted)+ finished.InvestigationOutcome = string(InvestigationOutcomeResolved)+ findings.Add(finished)++ if findings.RecoverStuckInvestigation("moved-on", "chat-older", time.Time{}, InvestigationStatusFailed, InvestigationOutcomeInterrupted, now) {+ t.Fatal("recovering an older run overwrote the newer one")+ }+ assertInvestigationState(t, findings, "moved-on", InvestigationStatusRunning, "")+ if findings.RecoverStuckInvestigation("finished", "chat-finished", time.Time{}, InvestigationStatusFailed, InvestigationOutcomeTimedOut, now) {+ t.Fatal("recovery overwrote a finished run")+ }+ assertInvestigationState(t, findings, "finished", InvestigationStatusCompleted, InvestigationOutcomeResolved)+ if findings.RecoverStuckInvestigation("missing", "chat-missing", time.Time{}, InvestigationStatusFailed, InvestigationOutcomeTimedOut, now) {+ t.Fatal("recovery reported success for an unknown finding")+ }+}++func TestRecoverStuckInvestigationsKeepsALiveRunBehindANewerSession(t *testing.T) {+ now := time.Now()+ findings := NewFindingsStore()+ findings.Add(runningInvestigationFinding("running", "chat-running", nil))+ findings.Add(runningInvestigationFinding("pending", "chat-pending", nil))+ // A second admission for each finding created its session but has not+ // marked the finding yet, so it is the finding's latest session.+ newer := func(id string) *InvestigationSession {+ return &InvestigationSession{ID: "inv-newer-" + id, FindingID: id, SessionID: "chat-newer-" + id, Status: aicontracts.InvestigationStatusPending, StartedAt: now}+ }+ store := findingSessionsStore{byFinding: map[string][]*InvestigationSession{+ "running": {{ID: "inv-running", FindingID: "running", SessionID: "chat-running", Status: aicontracts.InvestigationStatusRunning, StartedAt: now.Add(-time.Minute)}, newer("running")},+ // Marked on the finding, not yet running in the store.+ "pending": {{ID: "inv-pending", FindingID: "pending", SessionID: "chat-pending", Status: aicontracts.InvestigationStatusPending, StartedAt: now.Add(-time.Minute)}, newer("pending")},+ }}+ orchestrator := &sessionReportingOrchestrator{+ sessions: map[string]*InvestigationSession{"running": newer("running"), "pending": newer("pending")},+ store: store,+ }+ patrol := &PatrolService{findings: findings, investigationOrchestrator: orchestrator}+ patrol.recoverStuckInvestigations()++ assertInvestigationState(t, findings, "running", InvestigationStatusRunning, "")+ assertInvestigationState(t, findings, "pending", InvestigationStatusRunning, "")++ // Without the store, a newer latest session leaves the named run unknown,+ // and the sweep leaves it alone rather than guess.+ patrol.investigationOrchestrator = &sessionReportingOrchestrator{sessions: orchestrator.sessions}+ patrol.recoverStuckInvestigations()+ assertInvestigationState(t, findings, "running", InvestigationStatusRunning, "")+}++func TestRecoverStuckInvestigationsKeepsAResultFromBeforeARegression(t *testing.T) {+ now := time.Now()+ closedAWhileAgo := now.Add(-investigationRunGrace - time.Minute)+ regressedAt := now.Add(-10 * time.Minute)+ findings := NewFindingsStore()+ f := runningInvestigationFinding("regressed", "chat-old", nil)+ f.LastRegressionAt = ®ressedAt+ findings.Add(f)+ // The run started before the finding regressed, and its session recorded+ // a verified fix for the earlier occurrence.+ store := findingSessionsStore{byFinding: map[string][]*InvestigationSession{+ "regressed": {{ID: "inv-old", FindingID: "regressed", SessionID: "chat-old", Status: aicontracts.InvestigationStatusCompleted, Outcome: aicontracts.OutcomeFixVerified, StartedAt: regressedAt.Add(-10 * time.Minute), CompletedAt: &closedAWhileAgo}},+ }}+ patrol := &PatrolService{findings: findings, investigationOrchestrator: &sessionReportingOrchestrator{store: store}}+ patrol.recoverStuckInvestigations()++ assertInvestigationState(t, findings, "regressed", InvestigationStatusFailed, InvestigationOutcomeInterrupted)+ if got := findings.Get("regressed"); got.ResolvedAt != nil {+ t.Fatal("an earlier occurrence's verified fix resolved the current one")+ }+}++func TestRecoverStuckInvestigationsAppliesTheSessionResult(t *testing.T) {+ now := time.Now()+ closedAWhileAgo := now.Add(-investigationRunGrace - time.Minute)+ findings := NewFindingsStore()+ // The session store saved the run's result; the finding store did not+ // before a crash.+ findings.Add(runningInvestigationFinding("resolved", "chat-resolved", nil))+ findings.Add(runningInvestigationFinding("queued", "chat-queued", nil))+ orchestrator := &sessionReportingOrchestrator{sessions: map[string]*InvestigationSession{+ "resolved": {ID: "inv-resolved", FindingID: "resolved", SessionID: "chat-resolved", Status: aicontracts.InvestigationStatusCompleted, Outcome: aicontracts.OutcomeResolved, StartedAt: now.Add(-time.Hour), CompletedAt: &closedAWhileAgo},+ // Closed at load after a restart while its proposal was being submitted.+ "queued": {ID: "inv-queued", FindingID: "queued", SessionID: "chat-queued", Status: aicontracts.InvestigationStatusFailed, Outcome: aicontracts.OutcomeFixQueued, StartedAt: now.Add(-time.Hour), CompletedAt: &closedAWhileAgo},+ }}+ patrol := &PatrolService{findings: findings, investigationOrchestrator: orchestrator}+ patrol.recoverStuckInvestigations()++ assertInvestigationState(t, findings, "resolved", InvestigationStatusCompleted, InvestigationOutcomeResolved)+ assertInvestigationState(t, findings, "queued", InvestigationStatusFailed, InvestigationOutcomeFixQueued)+ for _, id := range []string{"resolved", "queued"} {+ if findings.Get(id).ShouldInvestigate("approval") {+ t.Fatalf("%s: the run's recorded result must not be retried as a timeout", id)+ }+ }+}++func TestRecoverStuckInvestigationsWithoutOrchestratorUsesLastInvestigation(t *testing.T) {+ now := time.Now()+ previousRun := now.Add(-2 * time.Hour)+ findings := NewFindingsStore()+ findings.Add(runningInvestigationFinding("old", "chat-old", &previousRun))+ findings.Add(runningInvestigationFinding("unknown", "chat-unknown", nil))+ patrol := &PatrolService{findings: findings}+ patrol.recoverStuckInvestigations()++ assertInvestigationState(t, findings, "old", InvestigationStatusFailed, InvestigationOutcomeTimedOut)+ assertInvestigationState(t, findings, "unknown", InvestigationStatusRunning, "")+}++func TestInvestigationRunCeilingFollowsTheConfiguredTimeout(t *testing.T) {+ if got, want := (&PatrolService{}).investigationRunCeiling(), 10*time.Minute+investigationRunGrace; got != want {+ t.Fatalf("default ceiling = %s, want %s", got, want)+ }+ cfg := config.NewDefaultAIConfig()+ cfg.PatrolModel = config.AIProviderCodexSubscription + ":gpt-5.6-luna"+ patrol := &PatrolService{aiService: &Service{cfg: cfg}}+ if got, want := patrol.investigationRunCeiling(), time.Duration(config.DefaultPatrolSubscriptionInvestigationTimeoutSec)*time.Second+investigationRunGrace; got != want {+ t.Fatalf("subscription ceiling = %s, want %s", got, want)+ }+}diff --git a/internal/ai/patrol_lifecycle_integration_test.go b/internal/ai/patrol_lifecycle_integration_test.goindex c2feed1871..0f11975f7f 100644--- a/internal/ai/patrol_lifecycle_integration_test.go+++ b/internal/ai/patrol_lifecycle_integration_test.go@@ -9,6 +9,7 @@ import ("github.com/rcourtman/pulse-go-rewrite/internal/ai/tools""github.com/rcourtman/pulse-go-rewrite/internal/config""github.com/rcourtman/pulse-go-rewrite/internal/models"+ "github.com/rcourtman/pulse-go-rewrite/pkg/aicontracts")// TestLifecycle_FindingToInvestigationToApproval verifies the end-to-end lifecycle:@@ -151,12 +152,20 @@ func TestLifecycle_StuckInvestigationRecovery(t *testing.T) {AnalyzeGuests: true,})- // Set up orchestrator (required for MaybeInvestigateFinding path)- orch := newIntegrationOrchestrator(true)- ps.SetInvestigationOrchestrator(orch)-// Create a finding and manually set it to "running" with LastInvestigatedAt 20 minutes agotwentyMinAgo := time.Now().Add(-20 * time.Minute)++ // The orchestrator still holds the run's session, started 20 minutes ago,+ // so recovery treats it as a hung run rather than one lost to a restart.+ ps.SetInvestigationOrchestrator(&sessionReportingOrchestrator{sessions: map[string]*InvestigationSession{+ "stuck-finding-1": {+ ID: "inv-stuck-1",+ FindingID: "stuck-finding-1",+ SessionID: "session-stuck-1",+ Status: aicontracts.InvestigationStatusRunning,+ StartedAt: twentyMinAgo,+ },+ }})stuckFinding := &Finding{ID: "stuck-finding-1",Key: "high-mem",diff --git a/internal/ai/service.go b/internal/ai/service.goindex 3cb2d32b6d..55cef9860a 100644--- a/internal/ai/service.go+++ b/internal/ai/service.go@@ -903,6 +903,7 @@ func (s *Service) ApplyPatrolAutonomyConfig(cfg *config.AIConfig) error {if s == nil || cfg == nil {return fmt.Errorf("Pulse Patrol config is unavailable")}+ orchestrator := s.liveInvestigationOrchestrator()s.mu.Lock()defer s.mu.Unlock()if s.cfg == nil {@@ -920,18 +921,42 @@ func (s *Service) ApplyPatrolAutonomyConfig(cfg *config.AIConfig) error {activation := *cfg.PatrolAutopilotActivations.cfg.PatrolAutopilotActivation = &activation}- if s.patrolService != nil {- orchestrator := s.patrolService.GetInvestigationOrchestrator()- if updater, ok := orchestrator.(aicontracts.InvestigationExecutionLimitUpdater); ok {- updater.SetInvestigationExecutionLimits(- s.cfg.GetPatrolInvestigationBudget(),- s.cfg.GetPatrolInvestigationTimeout(),- )- }- }+ publishInvestigationExecutionLimits(orchestrator, s.cfg)return nil}+// liveInvestigationOrchestrator returns the Patrol investigation orchestrator+// without holding s.mu while it takes the Patrol lock: PatrolService takes its+// own lock before s.mu (GetIntelligence), so the reverse order could deadlock.+func (s *Service) liveInvestigationOrchestrator() InvestigationOrchestrator {+ s.mu.RLock()+ patrol := s.patrolService+ s.mu.RUnlock()+ if patrol == nil {+ return nil+ }+ return patrol.GetInvestigationOrchestrator()+}++// publishInvestigationExecutionLimits publishes the configured evidence+// budget, timeout and concurrency to a live investigation orchestrator. The+// timeout default and the concurrency follow the Patrol provider, so a+// provider change applies without rebuilding the orchestrator.+func publishInvestigationExecutionLimits(orchestrator InvestigationOrchestrator, cfg *config.AIConfig) {+ if orchestrator == nil || cfg == nil {+ return+ }+ if updater, ok := orchestrator.(aicontracts.InvestigationExecutionLimitUpdater); ok {+ updater.SetInvestigationExecutionLimits(+ cfg.GetPatrolInvestigationBudget(),+ cfg.GetPatrolInvestigationTimeout(),+ )+ }+ if updater, ok := orchestrator.(aicontracts.InvestigationConcurrencyLimitUpdater); ok {+ updater.SetInvestigationConcurrencyLimit(cfg.GetPatrolInvestigationConcurrency())+ }+}+func clonePatrolAutopilotAcknowledgements(records []unifiedresources.PatrolAutopilotAcknowledgement) []unifiedresources.PatrolAutopilotAcknowledgement {cloned := append([]unifiedresources.PatrolAutopilotAcknowledgement(nil), records...)for index := range cloned {@@ -1564,10 +1589,12 @@ func (s *Service) LoadConfig() error {providerClient, providerInitErr = buildConfiguredProvider(context.Background(), cfg, orgID, true)}+ orchestrator := s.liveInvestigationOrchestrator()s.mu.Lock()s.cfg = cfgs.provider = providerClients.providerInitErr = providerInitErr+ publishInvestigationExecutionLimits(orchestrator, cfg)s.mu.Unlock()return nildiff --git a/internal/ai/service_patrol_limits_test.go b/internal/ai/service_patrol_limits_test.gonew file mode 100644index 0000000000..6ffed4b9d5--- /dev/null+++ b/internal/ai/service_patrol_limits_test.go@@ -0,0 +1,57 @@+package ai++import (+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/config"+)++type concurrencyRecordingOrchestrator struct {+ stubInvestigationOrchestrator+ maxConcurrent int+}++func (s *concurrencyRecordingOrchestrator) SetInvestigationConcurrencyLimit(maxConcurrent int) {+ s.maxConcurrent = maxConcurrent+}++func TestLoadConfigPublishesProviderAwareInvestigationLimits(t *testing.T) {+ persistence := config.NewConfigPersistence(t.TempDir())+ saved := config.NewDefaultAIConfig()+ saved.PatrolModel = config.AIProviderCodexSubscription + ":gpt-5.6-luna"+ if err := persistence.SaveAIConfig(*saved); err != nil {+ t.Fatalf("SaveAIConfig: %v", err)+ }++ orchestrator := &concurrencyRecordingOrchestrator{}+ service := &Service{+ persistence: persistence,+ patrolService: &PatrolService{investigationOrchestrator: orchestrator},+ }+ if err := service.LoadConfig(); err != nil {+ t.Fatalf("LoadConfig: %v", err)+ }+ if orchestrator.maxConcurrent != 1 {+ t.Fatalf("concurrency = %d, want 1 for a subscription agent that serves one call at a time", orchestrator.maxConcurrent)+ }+ if want := time.Duration(config.DefaultPatrolSubscriptionInvestigationTimeoutSec) * time.Second; orchestrator.timeout != want {+ t.Fatalf("timeout = %s, want %s", orchestrator.timeout, want)+ }++ // Switching Patrol to an API-key provider restores the normal limits on+ // the next config load, without rebuilding the orchestrator.+ saved.PatrolModel = config.AIProviderAnthropic + ":claude-sonnet-4-5"+ if err := persistence.SaveAIConfig(*saved); err != nil {+ t.Fatalf("SaveAIConfig: %v", err)+ }+ if err := service.LoadConfig(); err != nil {+ t.Fatalf("LoadConfig: %v", err)+ }+ if orchestrator.maxConcurrent != config.MaxConcurrentInvestigations {+ t.Fatalf("concurrency = %d, want %d", orchestrator.maxConcurrent, config.MaxConcurrentInvestigations)+ }+ if want := time.Duration(config.DefaultPatrolInvestigationTimeoutSec) * time.Second; orchestrator.timeout != want {+ t.Fatalf("timeout = %s, want %s", orchestrator.timeout, want)+ }+}diff --git a/internal/ai/tools/executor.go b/internal/ai/tools/executor.goindex 874846cf6e..7b4fb6d7eb 100644--- a/internal/ai/tools/executor.go+++ b/internal/ai/tools/executor.go@@ -1161,10 +1161,11 @@ func (e *PulseToolExecutor) isToolAvailable(name string) bool {return e.GetPatrolFindingCreator() != nilcase agentcapabilities.PatrolProposeObserverToolName:return e.GetPatrolObserverProposer() != nil- case agentcapabilities.PatrolProposeActionToolName, agentcapabilities.PatrolActionCapabilitiesToolName:+ case agentcapabilities.PatrolProposeActionToolName, agentcapabilities.PatrolActionCapabilitiesToolName, agentcapabilities.PatrolConcludeInvestigationToolName:// These investigation-only tools share the request-local proposal- // capture: it supplies both trusted correlation and the tenant-bound- // capability catalog. They must never appear without that boundary.+ // capture: it supplies trusted correlation, the tenant-bound+ // capability catalog and the run's conclusion sink. They must never+ // appear without that boundary.return e.proposalCapture != nildefault:return e.hasReadState()@@ -1230,6 +1231,9 @@ func (e *PulseToolExecutor) registerTools() {e.registerProposeTools()+ // patrol_conclude_investigation - typed zero-action investigation conclusion+ e.registerConcludeTool()+// pulse_summarize - retrospective synthesis for one resource or a fleete.registerSummarizeTools()}diff --git a/internal/ai/tools/names_test.go b/internal/ai/tools/names_test.goindex 57e35fec4d..56ae238b61 100644--- a/internal/ai/tools/names_test.go+++ b/internal/ai/tools/names_test.go@@ -30,6 +30,7 @@ func TestKnownToolNamesIncludesRegisteredTools(t *testing.T) {agentcapabilities.PatrolAssessFindingToolName,agentcapabilities.PatrolActionCapabilitiesToolName,agentcapabilities.PatrolProposeActionToolName,+ agentcapabilities.PatrolConcludeInvestigationToolName,}for _, name := range expected {if !IsKnownToolName(name) {diff --git a/internal/ai/tools/proposal_capture.go b/internal/ai/tools/proposal_capture.goindex 58897e6a89..1c5732e6d7 100644--- a/internal/ai/tools/proposal_capture.go+++ b/internal/ai/tools/proposal_capture.go@@ -85,6 +85,9 @@ type ProposalCapture struct {catalog ProposalCatalogplanner ProposalPlannerproposal *CapturedProposal+ // conclusion is the run's one typed zero-action conclusion, recorded+ // by patrol_conclude_investigation (see tools_conclude.go).+ conclusion *CapturedConclusion}func (i ProposalIdentity) clone() ProposalIdentity {diff --git a/internal/ai/tools/registry.go b/internal/ai/tools/registry.goindex 674c1e9324..b190e1d57e 100644--- a/internal/ai/tools/registry.go+++ b/internal/ai/tools/registry.go@@ -161,7 +161,7 @@ func (p InvocationPolicy) Allows(toolName string, class agentcapabilities.Invocafunc isPatrolInvestigationOnlyTool(toolName string) bool {switch toolName {- case agentcapabilities.PatrolProposeActionToolName, agentcapabilities.PatrolActionCapabilitiesToolName:+ case agentcapabilities.PatrolProposeActionToolName, agentcapabilities.PatrolActionCapabilitiesToolName, agentcapabilities.PatrolConcludeInvestigationToolName:return truedefault:return falsediff --git a/internal/ai/tools/tools_conclude.go b/internal/ai/tools/tools_conclude.gonew file mode 100644index 0000000000..f290ee85d6--- /dev/null+++ b/internal/ai/tools/tools_conclude.go@@ -0,0 +1,149 @@+package tools++import (+ "context"+ "errors"+ "fmt"+ "strings"++ "github.com/rcourtman/pulse-go-rewrite/internal/agentcapabilities"+ "github.com/rcourtman/pulse-go-rewrite/pkg/aicontracts"+)++// maxConclusionReasonRunes bounds the recorded reason. The run's summary+// carries the full narrative; the conclusion only needs its justification.+const maxConclusionReasonRunes = 2000++// CapturedConclusion is the one typed zero-action conclusion an investigation+// recorded. Outcome is aicontracts.OutcomeResolved, OutcomeCannotFix or+// OutcomeNeedsAttention.+type CapturedConclusion struct {+ Outcome aicontracts.InvestigationOutcome+ Reason string+ InvocationID string+}++// ParseConclusionOutcome maps a model-authored outcome onto the closed+// conclusion vocabulary. Anything else is refused, never defaulted.+func ParseConclusionOutcome(value string) (aicontracts.InvestigationOutcome, bool) {+ switch strings.ToLower(strings.TrimSpace(value)) {+ case string(aicontracts.OutcomeResolved):+ return aicontracts.OutcomeResolved, true+ case string(aicontracts.OutcomeCannotFix):+ return aicontracts.OutcomeCannotFix, true+ case string(aicontracts.OutcomeNeedsAttention):+ return aicontracts.OutcomeNeedsAttention, true+ default:+ return "", false+ }+}++// RecordConclusion keeps the first conclusion for the run. Replaying the same+// outcome is accepted; a different outcome is refused so the recorded+// conclusion cannot flip after the model has acted on it.+func (c *ProposalCapture) RecordConclusion(invocationID string, outcome aicontracts.InvestigationOutcome, reason string) (*CapturedConclusion, error) {+ if c == nil {+ return nil, errors.New("investigation conclusions are not available in this run")+ }+ reason = strings.TrimSpace(reason)+ if runes := []rune(reason); len(runes) > maxConclusionReasonRunes {+ reason = string(runes[:maxConclusionReasonRunes])+ }+ c.mu.Lock()+ defer c.mu.Unlock()+ if c.conclusion != nil {+ existing := *c.conclusion+ if existing.Outcome == outcome {+ return &existing, nil+ }+ return &existing, fmt.Errorf("this investigation already concluded %s; a different conclusion needs a new investigation", existing.Outcome)+ }+ c.conclusion = &CapturedConclusion{Outcome: outcome, Reason: reason, InvocationID: invocationID}+ recorded := *c.conclusion+ return &recorded, nil+}++// Conclusion returns a copy of the recorded conclusion, nil when none exists.+func (c *ProposalCapture) Conclusion() *CapturedConclusion {+ if c == nil {+ return nil+ }+ c.mu.Lock()+ defer c.mu.Unlock()+ if c.conclusion == nil {+ return nil+ }+ recorded := *c.conclusion+ return &recorded+}++// InvestigationConcluded reports whether this run has recorded its+// conclusion. The agentic loop then asks only for the final summary.+func (e *PulseToolExecutor) InvestigationConcluded() bool {+ return e != nil && e.proposalCapture.Conclusion() != nil+}++// registerConcludeTool registers the investigation-only conclusion capture.+// The registry policy rejects it outside the Patrol investigation profile,+// and it is offered only when the request-local capture exists.+func (e *PulseToolExecutor) registerConcludeTool() {+ e.registry.registerBuiltin(RegisteredTool{+ Definition: Tool{+ Name: agentcapabilities.PatrolConcludeInvestigationToolName,+ Description: `Record this investigation's conclusion. Call it once, when the evidence supports a conclusion, then write the final summary. Choose exactly one outcome:+- resolved: the evidence shows the finding's condition no longer holds.+- cannot_fix: the problem is real, but no advertised capability can remediate it.+- needs_attention: a person must act or decide; the reason says what they should do.++If you proposed an action, its planning result is the investigation outcome and this call is optional. Side-effect-free: it records the conclusion for this run only and plans, approves or executes nothing.`,+ InputSchema: InputSchema{+ Type: "object",+ Properties: map[string]PropertySchema{+ "outcome": {+ Type: "string",+ Description: "The conclusion this investigation reached",+ Enum: []string{string(aicontracts.OutcomeResolved), string(aicontracts.OutcomeCannotFix), string(aicontracts.OutcomeNeedsAttention)},+ },+ "reason": {+ Type: "string",+ Description: "The evidence for this conclusion and, for needs_attention, what the person should do",+ },+ },+ Required: []string{"outcome", "reason"},+ },+ },+ Handler: func(ctx context.Context, exec *PulseToolExecutor, args map[string]interface{}) (CallToolResult, error) {+ return exec.executeConcludeInvestigation(ctx, args)+ },+ Governance: ToolGovernance{+ ActionMode: ToolActionRead,+ ApprovalPolicy: ToolApprovalScopeOnly,+ ApprovalSummary: "records the investigation's typed conclusion without changing infrastructure or Pulse state",+ Summary: "Records one typed zero-action conclusion during a Patrol investigation; the orchestrator persists the outcome.",+ },+ })+}++// executeConcludeInvestigation validates and records the run's conclusion.+func (e *PulseToolExecutor) executeConcludeInvestigation(ctx context.Context, args map[string]interface{}) (CallToolResult, error) {+ capture := e.proposalCapture+ if capture == nil {+ return NewErrorResult(fmt.Errorf("investigation conclusions are not available in this run")), nil+ }+ outcome, ok := ParseConclusionOutcome(stringArg(args, "outcome"))+ if !ok {+ return NewErrorResult(fmt.Errorf("outcome must be one of resolved, cannot_fix or needs_attention")), nil+ }+ reason := strings.TrimSpace(stringArg(args, "reason"))+ if reason == "" {+ return NewErrorResult(fmt.Errorf("reason is required")), nil+ }+ recorded, err := capture.RecordConclusion(InvocationIDFromContext(ctx), outcome, reason)+ if err != nil {+ return NewErrorResult(err), nil+ }+ return NewJSONResult(map[string]interface{}{+ "recorded_outcome": string(recorded.Outcome),+ "next_step": "Write the final summary sections now: Investigation Summary, Root Cause, Affected Resources and Recommendation. Do not call further tools.",+ }), nil+}diff --git a/internal/ai/tools/tools_conclude_test.go b/internal/ai/tools/tools_conclude_test.gonew file mode 100644index 0000000000..3960ae9074--- /dev/null+++ b/internal/ai/tools/tools_conclude_test.go@@ -0,0 +1,126 @@+package tools++import (+ "context"+ "strings"+ "testing"++ "github.com/rcourtman/pulse-go-rewrite/internal/agentcapabilities"+ "github.com/rcourtman/pulse-go-rewrite/pkg/aicontracts"+ "github.com/stretchr/testify/assert"+ "github.com/stretchr/testify/require"+)++func executeConclude(t *testing.T, exec *PulseToolExecutor, id string, args map[string]interface{}) CallToolResult {+ t.Helper()+ result, err := exec.ExecuteInvocation(context.Background(), ToolInvocation{+ ID: id,+ Name: agentcapabilities.PatrolConcludeInvestigationToolName,+ Arguments: args,+ })+ require.NoError(t, err)+ require.NotEmpty(t, result.Content)+ return result+}++func TestParseConclusionOutcomeAcceptsOnlyTheClosedVocabulary(t *testing.T) {+ for input, want := range map[string]aicontracts.InvestigationOutcome{+ "resolved": aicontracts.OutcomeResolved,+ " CANNOT_FIX ": aicontracts.OutcomeCannotFix,+ "Needs_Attention": aicontracts.OutcomeNeedsAttention,+ } {+ got, ok := ParseConclusionOutcome(input)+ assert.True(t, ok, input)+ assert.Equal(t, want, got, input)+ }+ for _, input := range []string{"", "fix_verified", "timed_out", "unclassified", "resolved: yes"} {+ _, ok := ParseConclusionOutcome(input)+ assert.False(t, ok, "%q must not parse as a conclusion", input)+ }+}++func TestRecordConclusionKeepsTheFirstConclusion(t *testing.T) {+ capture := NewProposalCapture(ProposalIdentity{ProposalID: "p1", FindingID: "f1", InvestigationID: "i1"}, nil)+ assert.Nil(t, capture.Conclusion())++ recorded, err := capture.RecordConclusion("call-1", aicontracts.OutcomeResolved, " usage is back under threshold ")+ require.NoError(t, err)+ assert.Equal(t, aicontracts.OutcomeResolved, recorded.Outcome)+ assert.Equal(t, "usage is back under threshold", recorded.Reason)++ _, err = capture.RecordConclusion("call-2", aicontracts.OutcomeResolved, "replayed")+ require.NoError(t, err, "replaying the same outcome is accepted")++ _, err = capture.RecordConclusion("call-3", aicontracts.OutcomeNeedsAttention, "changed my mind")+ require.Error(t, err, "a different outcome must not replace the recorded conclusion")++ got := capture.Conclusion()+ require.NotNil(t, got)+ assert.Equal(t, aicontracts.OutcomeResolved, got.Outcome)+ assert.Equal(t, "usage is back under threshold", got.Reason)+ assert.Equal(t, "call-1", got.InvocationID)++ long := NewProposalCapture(ProposalIdentity{}, nil)+ recorded, err = long.RecordConclusion("call-1", aicontracts.OutcomeCannotFix, strings.Repeat("x", maxConclusionReasonRunes+50))+ require.NoError(t, err)+ assert.Len(t, []rune(recorded.Reason), maxConclusionReasonRunes)+}++func TestConcludeInvestigationIsInvestigationProfileOnly(t *testing.T) {+ args := map[string]interface{}{"outcome": "needs_attention", "reason": "replace the failing disk"}+ for _, profile := range []ExecutionProfile{ProfileInteractiveAssistant, ProfilePatrolDetection} {+ exec := NewPulseToolExecutor(ExecutorConfig{})+ exec.SetControlLevel(ControlLevelAutonomous)+ exec.ApplyExecutionProfile(profile)+ exec.SetProposalCapture(NewProposalCapture(ProposalIdentity{}, nil))++ result := executeConclude(t, exec, "call-x", args)+ assert.Contains(t, result.Content[0].Text, "Invocation blocked",+ "profile %d must reject the conclusion capture at the registry boundary", profile)+ for _, tool := range exec.registry.ListTools(exec.invocationPolicy()) {+ assert.NotEqual(t, agentcapabilities.PatrolConcludeInvestigationToolName, tool.Name,+ "profile %d must not offer the conclusion capture", profile)+ }+ }++ // Without the request-local capture the tool is not offered at all.+ bare := NewPulseToolExecutor(ExecutorConfig{})+ bare.ApplyExecutionProfile(ProfilePatrolInvestigation)+ for _, tool := range bare.ListTools() {+ assert.NotEqual(t, agentcapabilities.PatrolConcludeInvestigationToolName, tool.Name,+ "an investigation without a capture must not offer the conclusion tool")+ }++ capture := NewProposalCapture(ProposalIdentity{ProposalID: "p1", FindingID: "f1", InvestigationID: "i1"}, nil)+ exec := newInvestigationExecutor(t, capture)+ offered := false+ for _, tool := range exec.ListTools() {+ if tool.Name == agentcapabilities.PatrolConcludeInvestigationToolName {+ offered = true+ }+ }+ assert.True(t, offered, "investigation profile must offer patrol_conclude_investigation")+ assert.False(t, exec.InvestigationConcluded())++ result := executeConclude(t, exec, "call-a", args)+ assert.False(t, result.IsError, result.Content[0].Text)+ assert.Contains(t, result.Content[0].Text, "needs_attention")+ assert.True(t, exec.InvestigationConcluded())+ require.NotNil(t, capture.Conclusion())+ assert.Equal(t, aicontracts.OutcomeNeedsAttention, capture.Conclusion().Outcome)+ assert.Equal(t, "call-a", capture.Conclusion().InvocationID)+}++func TestConcludeInvestigationRefusesInvalidArguments(t *testing.T) {+ exec := newInvestigationExecutor(t, NewProposalCapture(ProposalIdentity{ProposalID: "p1", FindingID: "f1", InvestigationID: "i1"}, nil))++ for name, args := range map[string]map[string]interface{}{+ "unknown outcome": {"outcome": "fix_verified", "reason": "done"},+ "missing reason": {"outcome": "resolved"},+ "blank reason": {"outcome": "resolved", "reason": " "},+ } {+ result := executeConclude(t, exec, "call-"+strings.ReplaceAll(name, " ", "-"), args)+ assert.True(t, result.IsError, "%s must be refused: %s", name, result.Content[0].Text)+ }+ assert.False(t, exec.InvestigationConcluded(), "refused calls must not record a conclusion")+}diff --git a/internal/api/ai_handlers.go b/internal/api/ai_handlers.goindex a345e6f71e..02713e706b 100644--- a/internal/api/ai_handlers.go+++ b/internal/api/ai_handlers.go@@ -1941,6 +1941,7 @@ func (h *AISettingsHandler) setupInvestigationOrchestrator(orgID string, svc *aiinvConfig.MaxEvidenceCalls = cfg.GetPatrolInvestigationBudget()invConfig.MaxTurns = aicontracts.InvestigationModelTurnLimit(invConfig.MaxEvidenceCalls)invConfig.Timeout = cfg.GetPatrolInvestigationTimeout()+ invConfig.MaxConcurrent = cfg.GetPatrolInvestigationConcurrency()}// The knowledge store supplies the operator's infrastructure notes.@@ -2058,6 +2059,12 @@ func (a *orchestratorChatAdapter) ExecuteInvestigationStream(ctx context.ContextEvidenceIDs: captured.Identity.EvidenceIDs,}}+ if runResult.Conclusion != nil {+ result.Conclusion = &aicontracts.InvestigationConclusion{+ Outcome: runResult.Conclusion.Outcome,+ Reason: runResult.Conclusion.Reason,+ }+ }return result, err}diff --git a/internal/api/ai_handlers_investigation_additional_test.go b/internal/api/ai_handlers_investigation_additional_test.goindex b00f92d771..c51e310375 100644--- a/internal/api/ai_handlers_investigation_additional_test.go+++ b/internal/api/ai_handlers_investigation_additional_test.go@@ -885,3 +885,51 @@ func TestPatrolExpiredActionHydrationRemovesQueuedOutcomeWithoutRewritingEvidenct.Fatal("duplicate hydration republished unchanged record")}}++func TestSetupInvestigationOrchestratorUsesSubscriptionAgentLimits(t *testing.T) {+ prevStoreFactory := getCreateInvestigationStore()+ SetCreateInvestigationStore(func(dataDir string) aicontracts.InvestigationStore {+ return newTestInvestigationStore()+ })+ t.Cleanup(func() { SetCreateInvestigationStore(prevStoreFactory) })++ var created []aicontracts.InvestigationConfig+ prevOrchestratorFactory := getCreateInvestigationOrchestrator()+ SetCreateInvestigationOrchestrator(func(deps aicontracts.OrchestratorDeps) aicontracts.InvestigationOrchestrator {+ created = append(created, deps.Config)+ return &stubInvestigationOrchestrator{}+ })+ t.Cleanup(func() { SetCreateInvestigationOrchestrator(prevOrchestratorFactory) })++ persistence := config.NewConfigPersistence(t.TempDir())+ saved := config.NewDefaultAIConfig()+ saved.PatrolModel = config.AIProviderCodexSubscription + ":gpt-5.6-luna"+ if err := persistence.SaveAIConfig(*saved); err != nil {+ t.Fatalf("SaveAIConfig: %v", err)+ }+ tenantSvc := ai.NewService(persistence, nil)+ if err := tenantSvc.LoadConfig(); err != nil {+ t.Fatalf("LoadConfig: %v", err)+ }+ tenantSvc.SetStateProvider(&MockStateProvider{})++ handler := &AISettingsHandler{+ investigationStores: make(map[string]aicontracts.InvestigationStore),+ }+ handler.chatHandler = &AIHandler{+ services: map[string]AIService{+ "tenant-1": chat.NewService(chat.Config{AIConfig: config.NewDefaultAIConfig()}),+ },+ }+ handler.setupInvestigationOrchestrator("tenant-1", tenantSvc)++ if len(created) != 1 {+ t.Fatalf("orchestrator factory calls = %d, want 1", len(created))+ }+ if created[0].MaxConcurrent != 1 {+ t.Fatalf("MaxConcurrent = %d, want 1 for a subscription agent that serves one call at a time", created[0].MaxConcurrent)+ }+ if want := time.Duration(config.DefaultPatrolSubscriptionInvestigationTimeoutSec) * time.Second; created[0].Timeout != want {+ t.Fatalf("Timeout = %s, want %s", created[0].Timeout, want)+ }+}diff --git a/internal/api/patrol_digest_email.go b/internal/api/patrol_digest_email.goindex 3479acc372..ba9202b8b6 100644--- a/internal/api/patrol_digest_email.go+++ b/internal/api/patrol_digest_email.go@@ -144,6 +144,7 @@ func patrolDigestEmailLines(digest ai.PatrolDigest) []patrolDigestEmailLine {"cannot_fix": "could not fix", "timed_out": "timed out", "fix_queued": "fix waiting for approval","fix_executed": "fix run", "fix_verification_unknown": "fix run, result unknown", "fix_verified": "fixed and verified","resolved": "resolved", "fix_rejected": "fix declined",+ "unclassified": "finished without a clear conclusion", "interrupted": "interrupted by a restart",}outcomeDetails := []string{}for _, key := range ai.PatrolDigestOutcomeOrder(digest.Investigations.ByOutcome) {diff --git a/internal/config/ai.go b/internal/config/ai.goindex aafdd672a8..e57ab85dbc 100644--- a/internal/config/ai.go+++ b/internal/config/ai.go@@ -179,6 +179,11 @@ const (InvestigationCooldownHours = 1 // Hours before re-investigating same finding)+// DefaultPatrolSubscriptionInvestigationTimeoutSec is the default when Patrol+// runs on a subscription agent CLI, which starts a new process and resends+// the conversation on every turn (25 minutes).+const DefaultPatrolSubscriptionInvestigationTimeoutSec = 1500+const (// DefaultAIModelQuickstart is retained only to recognize and retire legacy// Pulse-hosted model aliases from pre-GA config.@@ -1093,9 +1098,32 @@ func (c *AIConfig) GetPatrolInvestigationBudget() int {return c.PatrolInvestigationBudget}-// GetPatrolInvestigationTimeout returns the investigation timeout as a duration+// PatrolUsesSubscriptionAgent reports whether Patrol runs on a local+// subscription agent CLI (Codex or Claude). Such an agent serves one call at+// a time for all of Patrol and starts a new process for every turn.+func (c *AIConfig) PatrolUsesSubscriptionAgent() bool {+ provider, _ := ParseModelString(c.GetPatrolModel())+ return provider == AIProviderCodexSubscription || provider == AIProviderClaudeSubscription+}++// GetPatrolInvestigationConcurrency returns how many investigations may run+// at once. A subscription agent serves one call at a time, so parallel+// investigations would only queue against each other's deadlines.+func (c *AIConfig) GetPatrolInvestigationConcurrency() int {+ if c.PatrolUsesSubscriptionAgent() {+ return 1+ }+ return MaxConcurrentInvestigations+}++// GetPatrolInvestigationTimeout returns the investigation timeout as a+// duration. Without an explicit setting, a subscription agent gets the longer+// default because each of its turns takes minutes rather than seconds.func (c *AIConfig) GetPatrolInvestigationTimeout() time.Duration {if c.PatrolInvestigationTimeoutSec <= 0 {+ if c.PatrolUsesSubscriptionAgent() {+ return time.Duration(DefaultPatrolSubscriptionInvestigationTimeoutSec) * time.Second+ }return time.Duration(DefaultPatrolInvestigationTimeoutSec) * time.Second}// Clamp to reasonable range (60-1800 seconds / 30 minutes)diff --git a/internal/config/ai_config_test.go b/internal/config/ai_config_test.goindex 685503d24a..6832eb1c04 100644--- a/internal/config/ai_config_test.go+++ b/internal/config/ai_config_test.go@@ -1410,3 +1410,54 @@ func TestResolvedPatrolAutonomyLevelDefaultsOnlyNeverChosenLicensedInstalls(t *t})}}++func TestPatrolSubscriptionAgentGetsLongerTimeoutAndOneInvestigationAtATime(t *testing.T) {+ cases := []struct {+ name string+ cfg AIConfig+ subscription bool+ wantTimeout time.Duration+ wantConcurrency int+ }{+ {+ name: "codex subscription patrol model",+ cfg: AIConfig{PatrolModel: AIProviderCodexSubscription + ":gpt-5.6-luna"},+ subscription: true,+ wantTimeout: DefaultPatrolSubscriptionInvestigationTimeoutSec * time.Second,+ wantConcurrency: 1,+ },+ {+ name: "claude subscription inherited from the main model",+ cfg: AIConfig{Model: AIProviderClaudeSubscription + ":sonnet"},+ subscription: true,+ wantTimeout: DefaultPatrolSubscriptionInvestigationTimeoutSec * time.Second,+ wantConcurrency: 1,+ },+ {+ name: "api key provider",+ cfg: AIConfig{PatrolModel: AIProviderAnthropic + ":claude-sonnet-4-5"},+ wantTimeout: DefaultPatrolInvestigationTimeoutSec * time.Second,+ wantConcurrency: MaxConcurrentInvestigations,+ },+ {+ name: "explicit timeout wins over the subscription default",+ cfg: AIConfig{PatrolModel: AIProviderCodexSubscription + ":gpt-5.6-luna", PatrolInvestigationTimeoutSec: 900},+ subscription: true,+ wantTimeout: 900 * time.Second,+ wantConcurrency: 1,+ },+ }+ for _, tc := range cases {+ t.Run(tc.name, func(t *testing.T) {+ if got := tc.cfg.PatrolUsesSubscriptionAgent(); got != tc.subscription {+ t.Fatalf("PatrolUsesSubscriptionAgent() = %v, want %v", got, tc.subscription)+ }+ if got := tc.cfg.GetPatrolInvestigationTimeout(); got != tc.wantTimeout {+ t.Fatalf("GetPatrolInvestigationTimeout() = %s, want %s", got, tc.wantTimeout)+ }+ if got := tc.cfg.GetPatrolInvestigationConcurrency(); got != tc.wantConcurrency {+ t.Fatalf("GetPatrolInvestigationConcurrency() = %d, want %d", got, tc.wantConcurrency)+ }+ })+ }+}diff --git a/internal/telemetry/patrol_cost.go b/internal/telemetry/patrol_cost.goindex 8a3009e222..e1e8e5cc65 100644--- a/internal/telemetry/patrol_cost.go+++ b/internal/telemetry/patrol_cost.go@@ -253,6 +253,15 @@ func PatrolInvestigationOutcomeBucket(outcome, status string) string {return PatrolInvestigationOutcomeCannotFixcase "timed_out":return PatrolInvestigationOutcomeTimedOut+ case "unclassified":+ // A finished run whose conclusion could not be read. Every other+ // outcome the orchestrator writes has its own bucket, so "other"+ // effectively isolates these runs.+ return PatrolInvestigationOutcomeOther+ case "interrupted":+ // A restart cut the run short and the finding awaits another run,+ // so it is not yet concluded.+ return PatrolInvestigationOutcomeInProgresscase "":switch strings.ToLower(strings.TrimSpace(status)) {case "pending", "running":diff --git a/internal/telemetry/patrol_cost_test.go b/internal/telemetry/patrol_cost_test.goindex 3975fc6aa2..575d150919 100644--- a/internal/telemetry/patrol_cost_test.go+++ b/internal/telemetry/patrol_cost_test.go@@ -234,6 +234,8 @@ func TestPatrolInvestigationOutcomeBucketPartitionsEveryState(t *testing.T) {{"needs_attention", "needs_attention", PatrolInvestigationOutcomeNeedsAttention},{"cannot_fix", "completed", PatrolInvestigationOutcomeCannotFix},{"timed_out", "failed", PatrolInvestigationOutcomeTimedOut},+ {"unclassified", "completed", PatrolInvestigationOutcomeOther},+ {"interrupted", "failed", PatrolInvestigationOutcomeInProgress},{"", "pending", PatrolInvestigationOutcomeInProgress},{"", "running", PatrolInvestigationOutcomeInProgress},{"", "failed", PatrolInvestigationOutcomeFailed},@@ -252,7 +254,7 @@ func TestPatrolInvestigationOutcomeBucketPartitionsEveryState(t *testing.T) {if counts.Total() != len(cases) {t.Fatalf("outcome counts total = %d, want %d (every investigated finding must land in exactly one bucket)", counts.Total(), len(cases))}- if counts.FixFailed != 2 || counts.InProgress != 2 || counts.Other != 3 || counts.NeedsAttention != 2 {+ if counts.FixFailed != 2 || counts.InProgress != 3 || counts.Other != 4 || counts.NeedsAttention != 2 || counts.TimedOut != 1 {t.Fatalf("unexpected partition: %+v", counts)}}diff --git a/pkg/aicontracts/contracts_test.go b/pkg/aicontracts/contracts_test.goindex be93a154a6..061cc77832 100644--- a/pkg/aicontracts/contracts_test.go+++ b/pkg/aicontracts/contracts_test.go@@ -420,3 +420,63 @@ func TestInvestigationBudgetSeparatesEvidenceCallsFromModelTurns(t *testing.T) {}}}++func TestInvestigationSessionCarriesItsRunDeadline(t *testing.T) {+ payload, err := json.Marshal(EmptyInvestigationSession())+ if err != nil {+ t.Fatalf("marshal session: %v", err)+ }+ if strings.Contains(string(payload), `"deadline"`) {+ t.Fatalf("a session with no recorded deadline must omit it: %s", payload)+ }+ deadline := time.Date(2026, 10, 23, 1, 30, 0, 0, time.UTC)+ session := EmptyInvestigationSession()+ session.Deadline = &deadline+ payload, err = json.Marshal(session)+ if err != nil {+ t.Fatalf("marshal session: %v", err)+ }+ if !strings.Contains(string(payload), `"deadline":"2026-10-23T01:30:00Z"`) {+ t.Fatalf("session must carry its run deadline: %s", payload)+ }+}++func TestInvestigationResultCarriesTypedConclusion(t *testing.T) {+ resultType := reflect.TypeOf(OrchestratorInvestigationResult{})+ field, ok := resultType.FieldByName("Conclusion")+ if !ok {+ t.Fatal("investigation result must carry a typed Conclusion")+ }+ if field.Type != reflect.TypeOf((*InvestigationConclusion)(nil)) {+ t.Fatalf("Conclusion field type = %v, want *InvestigationConclusion", field.Type)+ }++ empty, err := json.Marshal(OrchestratorInvestigationResult{})+ if err != nil {+ t.Fatalf("marshal empty result: %v", err)+ }+ if strings.Contains(string(empty), "conclusion") {+ t.Fatalf("a run without a conclusion must omit the field: %s", empty)+ }+ payload, err := json.Marshal(OrchestratorInvestigationResult{Conclusion: &InvestigationConclusion{Outcome: OutcomeCannotFix, Reason: "no capability covers pool expansion"}})+ if err != nil {+ t.Fatalf("marshal concluded result: %v", err)+ }+ if !strings.Contains(string(payload), `"conclusion":{"outcome":"cannot_fix","reason":"no capability covers pool expansion"}`) {+ t.Fatalf("conclusion wire shape changed: %s", payload)+ }+}++func TestUnclassifiedAndInterruptedAreCanonicalWireValues(t *testing.T) {+ for outcome, want := range map[InvestigationOutcome]string{+ OutcomeUnclassified: "unclassified",+ OutcomeInterrupted: "interrupted",+ } {+ if string(outcome) != want {+ t.Fatalf("outcome %q must keep wire value %q", outcome, want)+ }+ }+ if _, ok := reflect.TypeOf((*InvestigationConcurrencyLimitUpdater)(nil)).Elem().MethodByName("SetInvestigationConcurrencyLimit"); !ok {+ t.Fatal("the optional concurrency updater must keep SetInvestigationConcurrencyLimit")+ }+}diff --git a/pkg/aicontracts/interfaces.go b/pkg/aicontracts/interfaces.goindex 8f6ecc8161..4ed094003c 100644--- a/pkg/aicontracts/interfaces.go+++ b/pkg/aicontracts/interfaces.go@@ -35,6 +35,14 @@ type InvestigationExecutionLimitUpdater interface {SetInvestigationExecutionLimits(maxTurns int, timeout time.Duration)}+// InvestigationConcurrencyLimitUpdater is the optional live-configuration+// contract for orchestrators whose concurrent-investigation limit can change+// without rebuilding the orchestrator. A Patrol provider that serves one call+// at a time needs one investigation at a time.+type InvestigationConcurrencyLimitUpdater interface {+ SetInvestigationConcurrencyLimit(maxConcurrent int)+}+// InvestigationStoreMaintainer is an optional interface for orchestrators that// expose their investigation store for periodic maintenance.type InvestigationStoreMaintainer interface {diff --git a/pkg/aicontracts/investigation.go b/pkg/aicontracts/investigation.goindex 5693f0e497..a1bd4158d1 100644--- a/pkg/aicontracts/investigation.go+++ b/pkg/aicontracts/investigation.go@@ -46,8 +46,26 @@ const (OutcomeNeedsAttention InvestigationOutcome = "needs_attention"OutcomeCannotFix InvestigationOutcome = "cannot_fix"OutcomeTimedOut InvestigationOutcome = "timed_out"+ // OutcomeUnclassified records a completed run whose conclusion could+ // not be read: it recorded no structured conclusion and its summary has+ // no recognisable conclusion marker. It is not a request for human+ // intervention, so the finding stays eligible for another run.+ OutcomeUnclassified InvestigationOutcome = "unclassified"+ // OutcomeInterrupted records a run cancelled before its deadline, for+ // example by a Pulse restart or shutdown. It is neither a timeout nor a+ // conclusion, so the finding stays eligible for another run.+ OutcomeInterrupted InvestigationOutcome = "interrupted")+// InvestigationConclusion is the structured zero-action conclusion a Patrol+// investigation records with patrol_conclude_investigation. Outcome is+// OutcomeResolved, OutcomeCannotFix or OutcomeNeedsAttention. Remediation+// still travels only as a typed action proposal.+type InvestigationConclusion struct {+ Outcome InvestigationOutcome `json:"outcome"`+ Reason string `json:"reason"`+}+// ErrVerificationUnknown indicates the verifier could not conclusively determine// whether a fix resolved the underlying issue. Callers may treat this as a// distinct outcome from "verification failed" (issue persists).@@ -187,6 +205,10 @@ type InvestigationSession struct {EvidenceIDs []string `json:"evidence_ids"`Summary string `json:"summary,omitempty"`Error string `json:"error,omitempty"`+ // Deadline is when this run's own execution timeout expires. A run keeps+ // the timeout it started with, so recovery measures a live run against+ // this deadline rather than the timeout configured now.+ Deadline *time.Time `json:"deadline,omitempty"`}func EmptyInvestigationSession() InvestigationSession {diff --git a/pkg/aicontracts/orchestrator_deps.go b/pkg/aicontracts/orchestrator_deps.goindex 45fc75aa66..fcf3e1d875 100644--- a/pkg/aicontracts/orchestrator_deps.go+++ b/pkg/aicontracts/orchestrator_deps.go@@ -69,6 +69,9 @@ type OrchestratorInvestigationResult struct {ModelTurns int `json:"model_turns"`EvidenceCalls int `json:"evidence_calls"`ToolCalls int `json:"tool_calls"`+ // Conclusion is the model's structured zero-action conclusion, nil when+ // the run recorded none. It carries no remediation authority.+ Conclusion *InvestigationConclusion `json:"conclusion,omitempty"`}// OrchestratorFindingsStore provides access to patrol findings for the orchestrator.