Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
fix: reconcile uncertain restarts and preserve Patrol follow-up
rcourtman2214d270f37d
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex bf5cc47f59..997b5df75f 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -15,6 +15,17 @@## Purpose+### Lost container mutation acknowledgements reconcile through reads++Typed container lifecycle execution attempts its mutation once. A transport+error preserves uncertain execution and, while the original operation context+remains live, reads the same container and waits for its existing health check.+Readback never manufactures a mutation acknowledgement or extends a canceled+operation. Fixed, non-sensitive timeout/cancellation/unknown messages survive+receipt sanitization. `docker_lifecycle_test.go` covers lost acknowledgement,+health recovery, cancellation, exactly-once mutation and receipt redaction.++### Verified agent actions remain attached to their originating occurrenceAn agent action's execution and verification evidence belongs to its originaldiff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.mdindex 0aa0e0a286..16cc08de1b 100644--- a/docs/release-control/v6/internal/subsystems/ai-runtime.md+++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md@@ -25,6 +25,28 @@ that same result. Successful reads retain their content and execution provenance## Purpose+### Uncertain remediation retains a model-led follow-up++An inconclusive verification remains unresolved and may be investigated again+when the model reports it present, without another cooldown on top of Watch's+schedule. Existing attempt admission and autonomy checks still apply.+A model's fresh `present` assessment records evidence during Watch and hands off+only after the Watch adapter has unwound. It does not dispatch a second+investigation inside the detection conversation. Deliberate rejection,+needs-attention and cannot-fix outcomes keep their existing terminal meaning.++`pulse-enterprise:internal/investigation/orchestrator.go` captures the prior+investigation context before replacing the finding's session identity. It reads+only the exact retained session for the current occurrence and gives the model+its bounded summary and canonical action receipt as historical evidence.+Reopened occurrences with no current session cannot borrow an old receipt.+The model must reconcile an uncertain effect using current reads before deciding+whether a new action is justified. This adds no mutation or approval authority.+`findings_verification_followup_test.go` covers eligibility, persistence and the+Watch handoff. The enterprise orchestrator test covers receipt continuity and+session isolation. Native unattended runs remain necessary outcome evidence.++### Patrol resumes its actual next-check deadline after restartThe main Patrol timer checkpoints its next deadline and effective interval in@@ -1300,6 +1322,8 @@ a measured cost reduction or installed-deployment cache-read rate.## Canonical Files+- `pulse-enterprise:internal/investigation/orchestrator.go`: private investigation execution and current-occurrence follow-up context.+- `pkg/reporting/evidence.go`: measured retained-history evidence contract for model-facing summaries, with no report narrator or heuristic health judgement.1. `internal/ai/`diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.mdindex a76fb3ca64..ebc17011e4 100644--- a/docs/release-control/v6/internal/subsystems/api-contracts.md+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md@@ -20,6 +20,16 @@## Purpose+### Container verification distinguishes transitional health++Both agent and independent lifecycle readbacks classify a running container's+`starting` health as inconclusive, not a final postcondition contradiction.+Execution and verification remain separate axes: a confirmed readback cannot+turn a lost mutation acknowledgement into acknowledged execution. The action+summary retains the agent's sanitized error. The Docker action-result tests+cover this projection without changing receipt protocol versions or authority.++### Finding-scoped investigation reads belong to the current occurrenceThe finding investigation and transcript endpoints return only the sessiondiff --git a/docs/release-control/v6/internal/subsystems/registry.json b/docs/release-control/v6/internal/subsystems/registry.jsonindex dc0d410e3e..ed76139af5 100644--- a/docs/release-control/v6/internal/subsystems/registry.json+++ b/docs/release-control/v6/internal/subsystems/registry.json@@ -2041,6 +2041,7 @@"pkg/aicontracts/orchestrator_deps.go","pkg/extensions/ai_autofix.go","pkg/reporting/evidence.go",+ "pulse-enterprise:internal/investigation/orchestrator.go","scripts/generate-pulse-intelligence-docs.go"],"verification": {@@ -2049,6 +2050,19 @@"exact_files": [],"require_explicit_path_policy_coverage": true,"path_policies": [+ {+ "id": "investigation-followup-context",+ "label": "Current-occurrence investigation continuity proof",+ "match_prefixes": [],+ "match_files": [+ "pulse-enterprise:internal/investigation/orchestrator.go"+ ],+ "allow_same_subsystem_tests": false,+ "test_prefixes": [],+ "exact_files": [+ "pulse-enterprise:internal/investigation/orchestrator_test.go"+ ]+ },{"id": "legacy-incident-archive","label": "Explicit read-only legacy recording archive proof",diff --git a/internal/ai/findings.go b/internal/ai/findings.goindex 5b994aee8e..15d32e60ab 100644--- a/internal/ai/findings.go+++ b/internal/ai/findings.go@@ -451,6 +451,10 @@ func (f *Finding) IsResolved() bool {func investigationCooldownForOutcome(outcome string) time.Duration {switch InvestigationOutcome(outcome) {+ case InvestigationOutcomeFixVerificationUnknown:+ // The next model assessment owns follow-up. Investigating an uncertain+ // effect is a read, not permission to retry the infrastructure write.+ return 0case InvestigationOutcomeTimedOut:return investigationTimeoutCooldowncase InvestigationOutcomeFixFailed, InvestigationOutcomeFixVerificationFailed:@@ -549,8 +553,7 @@ func (f *Finding) ShouldInvestigate(autonomyLevel string) bool {InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeResolved ||InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeFixRejected ||InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeCannotFix ||- InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeNeedsAttention ||- InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeFixVerificationUnknown {+ InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeNeedsAttention {return false}@@ -684,8 +687,7 @@ func (f *Finding) CanRetryInvestigation() bool {InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeResolved ||InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeFixRejected ||InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeCannotFix ||- InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeNeedsAttention ||- InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeFixVerificationUnknown {+ InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeNeedsAttention {return false}// Can't retry if currently runningdiff --git a/internal/ai/findings_verification_followup_test.go b/internal/ai/findings_verification_followup_test.gonew file mode 100644index 0000000000..5ebb0f7197--- /dev/null+++ b/internal/ai/findings_verification_followup_test.go@@ -0,0 +1,78 @@+package ai++import (+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/ai/tools"+ "github.com/rcourtman/pulse-go-rewrite/internal/config"+)++func TestUnknownVerificationAllowsFollowupWithoutAnotherCooldown(t *testing.T) {+ completed := time.Now().Add(-15 * time.Minute)+ f := &Finding{ID: "uncertain", Severity: FindingSeverityWarning,+ InvestigationStatus: string(InvestigationStatusCompleted),+ InvestigationOutcome: string(InvestigationOutcomeFixVerificationUnknown),+ LastInvestigatedAt: &completed, InvestigationAttempts: 1}+ if !f.ShouldInvestigate("full") || !f.CanRetryInvestigation() {+ t.Fatal("an inconclusive action permanently suppressed follow-up")+ }+ recent := time.Now().Add(-5 * time.Minute)+ f.LastInvestigatedAt = &recent+ if !f.ShouldInvestigate("full") || !f.CanRetryInvestigation() {+ t.Fatal("uncertain verification delayed a fresh model assessment")+ }+ f.LastInvestigatedAt = &completed+ f.InvestigationStatus = string(InvestigationStatusRunning)+ if f.ShouldInvestigate("full") || f.CanRetryInvestigation() {+ t.Fatal("follow-up duplicated a running investigation")+ }+}++func TestUnknownVerificationFollowupSurvivesRestartAndWaitsForWatchHandoff(t *testing.T) {+ persistence := config.NewConfigPersistence(t.TempDir())+ store := NewFindingsStore()+ if err := store.SetPersistence(NewFindingsPersistenceAdapter(persistence)); err != nil {+ t.Fatal(err)+ }+ completed := time.Now().Add(-time.Minute)+ store.Add(&Finding{ID: "uncertain", ResourceID: "app-container:service", ResourceType: "app-container",+ Title: "Service unavailable", Severity: FindingSeverityWarning, Category: FindingCategoryReliability,+ InvestigationSessionID: "original", InvestigationStatus: string(InvestigationStatusCompleted),+ InvestigationOutcome: string(InvestigationOutcomeFixVerificationUnknown), LastInvestigatedAt: &completed, InvestigationAttempts: 1})+ if err := store.ForceSave(); err != nil {+ t.Fatal(err)+ }+ restored := NewFindingsStore()+ if err := restored.SetPersistence(NewFindingsPersistenceAdapter(persistence)); err != nil {+ t.Fatal(err)+ }+ service := NewService(persistence, nil)+ service.cfg = &config.AIConfig{Enabled: true, PatrolAutonomyLevel: config.PatrolAutonomyApproval}+ patrol := NewPatrolService(service, nil)+ patrol.findings = restored+ orchestrator := newIntegrationOrchestrator(true)+ patrol.SetInvestigationOrchestrator(orchestrator)+ adapter := &patrolFindingCreatorAdapter{patrol: patrol}+ if err := adapter.AssessFinding(tools.PatrolFindingAssessmentInput{FindingID: "uncertain", Verdict: "present", Evidence: "Current HTTP health remains unavailable after the uncertain restart"}); err != nil {+ t.Fatal(err)+ }+ patrol.investigationWg.Wait()+ if len(orchestrator.investigateCh) != 0 {+ t.Fatal("investigation started while the Watch adapter still owned the executor")+ }+ current := restored.Get("uncertain")+ if current.InvestigationSessionID != "original" || current.InvestigationAttempts != 1 || current.ResolvedAt != nil {+ t.Fatalf("assessment discarded prior uncertainty: %#v", current)+ }+ patrol.dispatchPatrolInvestigations(&AIAnalysisResult{Findings: adapter.getCollectedFindings()})+ patrol.investigationWg.Wait()+ select {+ case id := <-orchestrator.investigateCh:+ if id != "uncertain" {+ t.Fatalf("follow-up finding=%s", id)+ }+ default:+ t.Fatal("fresh model assessment did not hand the unresolved finding to investigation")+ }+}diff --git a/internal/ai/patrol_findings.go b/internal/ai/patrol_findings.goindex b286ce17cc..86f47d42d7 100644--- a/internal/ai/patrol_findings.go+++ b/internal/ai/patrol_findings.go@@ -945,7 +945,7 @@ func (a *patrolFindingCreatorAdapter) AssessFinding(input tools.PatrolFindingAssif a.objectiveContext != nil {refreshed.ObjectiveContext = clonePatrolObjectiveContext(a.objectiveContext)}- if a.patrol.recordFinding(&refreshed) {+ if a.patrol.recordFindingDuringPatrol(&refreshed) {return fmt.Errorf("finding %s unexpectedly became a new finding during assessment", assessment.FindingID)}if stored := a.patrol.findings.Get(assessment.FindingID); stored != nil {diff --git a/internal/api/docker_container_action_result.go b/internal/api/docker_container_action_result.goindex b5837501fc..4d18670a8a 100644--- a/internal/api/docker_container_action_result.go+++ b/internal/api/docker_container_action_result.go@@ -33,15 +33,7 @@ func dockerContainerExecutionResult(resourceID, agentID string, req agentexec.Doif err != nil {return nil, fmt.Errorf("normalize docker lifecycle evidence: %w", err)}- status := unified.ActionVerificationContradicted- reason := "postcondition_contradicted"- if dockerLifecycleRunningHealthUnknown(req.Operation, facts.After) {- status = unified.ActionVerificationInconclusive- reason = "container_health_unknown"- } else if dockerLifecycleFactsMatch(req.Operation, facts.Before, facts.After) {- status = unified.ActionVerificationConfirmed- reason = ""- }+ status, reason := dockerLifecycleVerification(req.Operation, facts.Before, facts.After)verification = unified.ActionVerificationTruth{Status: status, EvidenceClass: unified.ActionEvidenceAgentAttested, ReasonCode: reason, Summary: summary, Evidence: []unified.ActionEvidence{evidence}}}}@@ -65,12 +57,10 @@ func dockerContainerExecutionResult(resourceID, agentID string, req agentexec.Doif err != nil {return nil, fmt.Errorf("normalize independent docker observation: %w", err)}- status := unified.ActionVerificationContradicted- reason := "postcondition_contradicted"- if dockerLifecycleObservationFactsMatch(req.Operation, facts.Before, independent.Snapshot) {- status = unified.ActionVerificationConfirmed- reason = ""- }+ status, reason := dockerLifecycleVerification(req.Operation, facts.Before, agentexec.DockerContainerLifecycleSnapshot{+ ContainerID: independent.Snapshot.ContainerID, State: independent.Snapshot.State, Running: independent.Snapshot.Running,+ Health: independent.Snapshot.Health, StartedAt: independent.Snapshot.StartedAt,+ })verification = unified.ActionVerificationTruth{Status: status, EvidenceClass: unified.ActionEvidenceIndependent, ReasonCode: reason, Summary: independentSummary, Evidence: []unified.ActionEvidence{evidence}}}@@ -265,7 +255,11 @@ func dockerUpdateFactsMatch(facts agentexec.DockerContainerUpdateResultPayload,}func dockerContainerFactSummary(facts agentexec.DockerContainerLifecycleResultPayload) string {- return fmt.Sprintf("Container lifecycle: phase=%s; mutation started=%t; mutation completed=%t; readback ran=%t; before=%s/%s; after=%s/%s", strings.TrimSpace(facts.ExecutionPhase), facts.MutationStarted, facts.MutationCompleted, facts.ReadbackRan, strings.TrimSpace(facts.Before.State), dockerLifecycleHealthSummary(facts.Before.Health), strings.TrimSpace(facts.After.State), dockerLifecycleHealthSummary(facts.After.Health))+ summary := fmt.Sprintf("Container lifecycle: phase=%s; mutation started=%t; mutation completed=%t; readback ran=%t; before=%s/%s; after=%s/%s", strings.TrimSpace(facts.ExecutionPhase), facts.MutationStarted, facts.MutationCompleted, facts.ReadbackRan, strings.TrimSpace(facts.Before.State), dockerLifecycleHealthSummary(facts.Before.Health), strings.TrimSpace(facts.After.State), dockerLifecycleHealthSummary(facts.After.Health))+ if errText := strings.TrimSpace(facts.Error); errText != "" {+ summary += "; error=" + errText+ }+ return summary}func dockerLifecycleHealthSummary(health string) string {@@ -298,13 +292,20 @@ func dockerLifecycleFactsMatch(operation string, before, after agentexec.DockerC}}-func dockerLifecycleObservationFactsMatch(operation string, before agentexec.DockerContainerLifecycleSnapshot, after agentexec.DockerContainerObservationSnapshot) bool {- return dockerLifecycleFactsMatch(operation, before, agentexec.DockerContainerLifecycleSnapshot{- ContainerID: after.ContainerID, State: after.State, Running: after.Running, Health: after.Health,- StartedAt: after.StartedAt, RestartCount: after.RestartCount, ObservedAt: after.ObservedAt,- })-}-func dockerLifecycleRunningHealthUnknown(operation string, after agentexec.DockerContainerLifecycleSnapshot) bool {return (operation == agentexec.DockerContainerOperationStart || operation == agentexec.DockerContainerOperationRestart) && !agentexec.IsDockerContainerHealth(after.Health)}++func dockerLifecycleVerification(operation string, before, after agentexec.DockerContainerLifecycleSnapshot) (unified.ActionVerificationStatus, string) {+ if dockerLifecycleFactsMatch(operation, before, after) {+ return unified.ActionVerificationConfirmed, ""+ }+ if dockerLifecycleRunningHealthUnknown(operation, after) {+ return unified.ActionVerificationInconclusive, "container_health_unknown"+ }+ if (operation == agentexec.DockerContainerOperationStart || operation == agentexec.DockerContainerOperationRestart) &&+ after.Running && after.State == "running" && after.Health == agentexec.DockerContainerHealthStarting {+ return unified.ActionVerificationInconclusive, "container_health_starting"+ }+ return unified.ActionVerificationContradicted, "postcondition_contradicted"+}diff --git a/internal/api/docker_container_action_result_test.go b/internal/api/docker_container_action_result_test.goindex 5292fe4d8f..01ee11a8a1 100644--- a/internal/api/docker_container_action_result_test.go+++ b/internal/api/docker_container_action_result_test.go@@ -161,6 +161,24 @@ func TestDockerContainerExecutionResultDoesNotVerifyUnrecoveredHealth(t *testing}}+func TestDockerContainerExecutionResultStartingHealthRemainsInconclusive(t *testing.T) {+ now := time.Now().UTC()+ facts := dockerResultFacts(now, true, false, false, true)+ facts.ContainerID = dockerLifecycleTestID+ observation := &dockerContainerPostconditionObservation{+ ObserverID: "daemon", TrustDomain: "docker-daemon:agent-1", Method: "docker_api_inspect",+ Snapshot: dockerObservationSnapshotFromLifecycle(facts.After), ReceivedAt: now,+ }+ observation.Snapshot.Health = agentexec.DockerContainerHealthStarting+ result, err := dockerContainerExecutionResult("app-container:fixture", "agent-1", agentexec.DockerContainerLifecyclePayload{ActionID: "action", Operation: agentexec.DockerContainerOperationRestart}, facts, observation, now)+ if err != nil {+ t.Fatal(err)+ }+ if result.ActionResultV2.Execution.Status != unified.ActionExecutionInconclusive || result.ActionResultV2.Verification.Status != unified.ActionVerificationInconclusive || result.ActionResultV2.Verification.ReasonCode != "container_health_starting" {+ t.Fatalf("transitional health was treated as a final contradiction: %#v", result.ActionResultV2)+ }+}+func dockerObservationSnapshotFromLifecycle(snapshot agentexec.DockerContainerLifecycleSnapshot) agentexec.DockerContainerObservationSnapshot {return agentexec.DockerContainerObservationSnapshot{ContainerID: snapshot.ContainerID, State: snapshot.State, Running: snapshot.Running, Health: snapshot.Health,diff --git a/internal/hostagent/commands.go b/internal/hostagent/commands.goindex c9f527762d..a85a4beb6f 100644--- a/internal/hostagent/commands.go+++ b/internal/hostagent/commands.go@@ -1196,8 +1196,14 @@ func (c *CommandClient) replayDockerContainerLifecycle(conn *websocket.Conn, rec}func sanitizeDockerLifecycleError(value string) string {- if strings.TrimSpace(value) == "" {+ switch strings.TrimSpace(value) {+ case "":return ""+ case "container lifecycle mutation acknowledgement timed out",+ "container lifecycle mutation acknowledgement canceled",+ "container lifecycle mutation acknowledgement unavailable":+ // Fixed messages retain the cause without daemon output or endpoints.+ return strings.TrimSpace(value)}return "typed container lifecycle did not complete as requested"}diff --git a/internal/hostagent/docker_lifecycle.go b/internal/hostagent/docker_lifecycle.goindex 41c76e7799..7b99a3fdb3 100644--- a/internal/hostagent/docker_lifecycle.go+++ b/internal/hostagent/docker_lifecycle.go@@ -3,7 +3,9 @@ package hostagentimport ("context""encoding/json"+ "errors""fmt"+ "net""os""os/exec""regexp"@@ -94,21 +96,32 @@ func (m *localDockerLifecycleManager) Apply(ctx context.Context, req agentexec.Dresult.ExecutionPhase = agentexec.DockerContainerPhaseMutateresult.MutationStarted = trueverb := strings.TrimSuffix(req.Operation, "_container")- if err := m.mutate(ctx, req.Runtime, verb, req.ContainerID); err != nil {- result.Error = "container lifecycle mutation did not complete"- return result+ mutationErr := m.mutate(ctx, req.Runtime, verb, req.ContainerID)+ if mutationErr != nil {+ // A lost acknowledgement does not prove that the daemon did nothing.+ // Reconcile using reads under the original deadline, without retrying+ // the mutation or claiming its execution was acknowledged.+ result.Error = dockerLifecycleMutationError(mutationErr)+ if ctx.Err() != nil {+ return result+ }+ } else {+ result.MutationCompleted = true}- result.MutationCompleted = trueresult.ExecutionPhase = agentexec.DockerContainerPhaseVerifyafter, err := m.inspect(ctx, req.Runtime, req.ContainerID)result.After = afterif err != nil {- result.Error = "container postcondition inspect unavailable"+ if mutationErr == nil {+ result.Error = "container postcondition inspect unavailable"+ }return result}result.ReadbackRan = trueif dockerLifecyclePostcondition(req, before, after) {- result.ExecutionPhase = agentexec.DockerContainerPhaseComplete+ if result.MutationCompleted {+ result.ExecutionPhase = agentexec.DockerContainerPhaseComplete+ }return result}if dockerLifecycleHealthPending(req, before, after) {@@ -120,20 +133,38 @@ func (m *localDockerLifecycleManager) Apply(ctx context.Context, req agentexec.D}after, err = m.inspect(verifyCtx, req.Runtime, req.ContainerID)if err != nil {- result.Error = "container postcondition inspect unavailable"+ if mutationErr == nil {+ result.Error = "container postcondition inspect unavailable"+ }return result}result.After = afterif dockerLifecyclePostcondition(req, before, after) {- result.ExecutionPhase = agentexec.DockerContainerPhaseComplete+ if result.MutationCompleted {+ result.ExecutionPhase = agentexec.DockerContainerPhaseComplete+ }return result}}}- result.Error = "container postcondition contradicted the requested state"+ if mutationErr == nil {+ result.Error = "container postcondition did not reach the requested state"+ }return result}+func dockerLifecycleMutationError(err error) string {+ var timeout net.Error+ switch {+ case errors.Is(err, context.DeadlineExceeded), errors.As(err, &timeout) && timeout.Timeout():+ return "container lifecycle mutation acknowledgement timed out"+ case errors.Is(err, context.Canceled):+ return "container lifecycle mutation acknowledgement canceled"+ default:+ return "container lifecycle mutation acknowledgement unavailable"+ }+}+func (m *localDockerLifecycleManager) Preflight(ctx context.Context, req agentexec.DockerContainerLifecyclePayload) (bool, string) {if err := agentexec.ValidateDockerContainerLifecyclePayload(&req); err != nil {return false, agentexec.ActionRefusalContractInvaliddiff --git a/internal/hostagent/docker_lifecycle_test.go b/internal/hostagent/docker_lifecycle_test.goindex 66729b9b10..31a831e2dc 100644--- a/internal/hostagent/docker_lifecycle_test.go+++ b/internal/hostagent/docker_lifecycle_test.go@@ -20,8 +20,9 @@ import (const dockerLifecycleTestContainerID = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"type stubDockerLifecycleOperator struct {- snapshots []agentexec.DockerContainerLifecycleSnapshot- mutations []string+ snapshots []agentexec.DockerContainerLifecycleSnapshot+ mutations []string+ mutationError error}func (s *stubDockerLifecycleOperator) InspectDockerContainerLifecycle(context.Context, string, string) (agentexec.DockerContainerLifecycleSnapshot, error) {@@ -35,7 +36,62 @@ func (s *stubDockerLifecycleOperator) InspectDockerContainerLifecycle(context.Cofunc (s *stubDockerLifecycleOperator) MutateDockerContainerLifecycle(_ context.Context, _, operation, _ string) error {s.mutations = append(s.mutations, operation)- return nil+ return s.mutationError+}++func TestDockerLifecycleManagerReconcilesLostMutationAcknowledgement(t *testing.T) {+ before := time.Now().UTC().Add(-time.Minute)+ after := before.Add(time.Minute)+ operator := &stubDockerLifecycleOperator{mutationError: context.DeadlineExceeded, snapshots: []agentexec.DockerContainerLifecycleSnapshot{+ {ContainerID: dockerLifecycleTestContainerID, State: "running", Running: true, Health: agentexec.DockerContainerHealthUnhealthy, StartedAt: before, ObservedAt: before},+ {ContainerID: dockerLifecycleTestContainerID, State: "running", Running: true, Health: agentexec.DockerContainerHealthStarting, StartedAt: after, ObservedAt: after},+ {ContainerID: dockerLifecycleTestContainerID, State: "running", Running: true, Health: agentexec.DockerContainerHealthHealthy, StartedAt: after, ObservedAt: after.Add(time.Millisecond)},+ }}+ manager := newLocalDockerLifecycleManager(operator)+ manager.healthPollInterval = time.Millisecond+ result := manager.Apply(context.Background(), dockerLifecycleTestRequest(t, before))+ if !result.MutationStarted || result.MutationCompleted || !result.ReadbackRan || result.After.Health != agentexec.DockerContainerHealthHealthy {+ t.Fatalf("lost acknowledgement was not reconciled without inventing completion: %#v", result)+ }+ if result.Error == "" || len(operator.mutations) != 1 {+ t.Fatalf("uncertainty or exactly-once mutation lost: result=%#v writes=%v", result, operator.mutations)+ }+}++func TestDockerLifecycleManagerDoesNotDetachCanceledReconciliation(t *testing.T) {+ t.Setenv("DOCKER_CONTEXT", "")+ before := time.Now().UTC().Add(-time.Minute)+ ctx, cancel := context.WithCancel(context.Background())+ manager := newLocalDockerLifecycleManager()+ writes, reads := 0, 0+ manager.run = func(_ context.Context, _ string, args ...string) ([]byte, error) {+ if args[0] == "restart" {+ writes+++ cancel()+ return nil, context.Canceled+ }+ reads+++ if args[2] == "{{.RestartCount}}" {+ return []byte("0"), nil+ }+ return []byte(fmt.Sprintf(`{"Status":"running","Running":true,"StartedAt":%q}`, before.Format(time.RFC3339Nano))), nil+ }+ result := manager.Apply(ctx, dockerLifecycleTestRequest(t, before))+ if writes != 1 || reads != 2 || result.ReadbackRan || result.MutationCompleted {+ t.Fatalf("canceled operation detached or retried: result=%#v reads=%d writes=%d", result, reads, writes)+ }+}++func TestDockerLifecycleErrorReceiptRetainsOnlySafeCauses(t *testing.T) {+ for _, cause := range []error{context.DeadlineExceeded, context.Canceled, fmt.Errorf("daemon at https://user:[email protected] refused operation")} {+ message := dockerLifecycleMutationError(cause)+ if got := sanitizeDockerLifecycleError(message); got != message || strings.Contains(got, "secret") {+ t.Fatalf("safe cause lost or leaked: %q", got)+ }+ }+ if got := sanitizeDockerLifecycleError("daemon response: secret"); strings.Contains(got, "secret") {+ t.Fatalf("raw daemon output leaked: %q", got)+ }}func TestDockerLifecycleManagerUsesConnectedModuleWithoutExternalCLI(t *testing.T) {