Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

fix: reconcile uncertain restarts and preserve Patrol follow-up

rcourtman2214d270f37d
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index bf5cc47f59..997b5df75f 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -15,6 +15,17 @@
## Purpose
+### Lost container mutation acknowledgements reconcile through reads
+
+Typed container lifecycle execution attempts its mutation once. A transport
+error preserves uncertain execution and, while the original operation context
+remains live, reads the same container and waits for its existing health check.
+Readback never manufactures a mutation acknowledgement or extends a canceled
+operation. Fixed, non-sensitive timeout/cancellation/unknown messages survive
+receipt sanitization. `docker_lifecycle_test.go` covers lost acknowledgement,
+health recovery, cancellation, exactly-once mutation and receipt redaction.
+
+
### Verified agent actions remain attached to their originating occurrence
An agent action's execution and verification evidence belongs to its original
diff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.md
index 0aa0e0a286..16cc08de1b 100644
--- a/docs/release-control/v6/internal/subsystems/ai-runtime.md
+++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md
@@ -25,6 +25,28 @@ that same result. Successful reads retain their content and execution provenance
## Purpose
+### Uncertain remediation retains a model-led follow-up
+
+An inconclusive verification remains unresolved and may be investigated again
+when the model reports it present, without another cooldown on top of Watch's
+schedule. Existing attempt admission and autonomy checks still apply.
+A model's fresh `present` assessment records evidence during Watch and hands off
+only after the Watch adapter has unwound. It does not dispatch a second
+investigation inside the detection conversation. Deliberate rejection,
+needs-attention and cannot-fix outcomes keep their existing terminal meaning.
+
+`pulse-enterprise:internal/investigation/orchestrator.go` captures the prior
+investigation context before replacing the finding's session identity. It reads
+only the exact retained session for the current occurrence and gives the model
+its bounded summary and canonical action receipt as historical evidence.
+Reopened occurrences with no current session cannot borrow an old receipt.
+The model must reconcile an uncertain effect using current reads before deciding
+whether a new action is justified. This adds no mutation or approval authority.
+`findings_verification_followup_test.go` covers eligibility, persistence and the
+Watch handoff. The enterprise orchestrator test covers receipt continuity and
+session isolation. Native unattended runs remain necessary outcome evidence.
+
+
### Patrol resumes its actual next-check deadline after restart
The main Patrol timer checkpoints its next deadline and effective interval in
@@ -1300,6 +1322,8 @@ a measured cost reduction or installed-deployment cache-read rate.
## Canonical Files
+- `pulse-enterprise:internal/investigation/orchestrator.go`: private investigation execution and current-occurrence follow-up context.
+
- `pkg/reporting/evidence.go`: measured retained-history evidence contract for model-facing summaries, with no report narrator or heuristic health judgement.
1. `internal/ai/`
diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md
index a76fb3ca64..ebc17011e4 100644
--- a/docs/release-control/v6/internal/subsystems/api-contracts.md
+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md
@@ -20,6 +20,16 @@
## Purpose
+### Container verification distinguishes transitional health
+
+Both agent and independent lifecycle readbacks classify a running container's
+`starting` health as inconclusive, not a final postcondition contradiction.
+Execution and verification remain separate axes: a confirmed readback cannot
+turn a lost mutation acknowledgement into acknowledged execution. The action
+summary retains the agent's sanitized error. The Docker action-result tests
+cover this projection without changing receipt protocol versions or authority.
+
+
### Finding-scoped investigation reads belong to the current occurrence
The finding investigation and transcript endpoints return only the session
diff --git a/docs/release-control/v6/internal/subsystems/registry.json b/docs/release-control/v6/internal/subsystems/registry.json
index dc0d410e3e..ed76139af5 100644
--- a/docs/release-control/v6/internal/subsystems/registry.json
+++ b/docs/release-control/v6/internal/subsystems/registry.json
@@ -2041,6 +2041,7 @@
"pkg/aicontracts/orchestrator_deps.go",
"pkg/extensions/ai_autofix.go",
"pkg/reporting/evidence.go",
+ "pulse-enterprise:internal/investigation/orchestrator.go",
"scripts/generate-pulse-intelligence-docs.go"
],
"verification": {
@@ -2049,6 +2050,19 @@
"exact_files": [],
"require_explicit_path_policy_coverage": true,
"path_policies": [
+ {
+ "id": "investigation-followup-context",
+ "label": "Current-occurrence investigation continuity proof",
+ "match_prefixes": [],
+ "match_files": [
+ "pulse-enterprise:internal/investigation/orchestrator.go"
+ ],
+ "allow_same_subsystem_tests": false,
+ "test_prefixes": [],
+ "exact_files": [
+ "pulse-enterprise:internal/investigation/orchestrator_test.go"
+ ]
+ },
{
"id": "legacy-incident-archive",
"label": "Explicit read-only legacy recording archive proof",
diff --git a/internal/ai/findings.go b/internal/ai/findings.go
index 5b994aee8e..15d32e60ab 100644
--- a/internal/ai/findings.go
+++ b/internal/ai/findings.go
@@ -451,6 +451,10 @@ func (f *Finding) IsResolved() bool {
func investigationCooldownForOutcome(outcome string) time.Duration {
switch InvestigationOutcome(outcome) {
+ case InvestigationOutcomeFixVerificationUnknown:
+ // The next model assessment owns follow-up. Investigating an uncertain
+ // effect is a read, not permission to retry the infrastructure write.
+ return 0
case InvestigationOutcomeTimedOut:
return investigationTimeoutCooldown
case InvestigationOutcomeFixFailed, InvestigationOutcomeFixVerificationFailed:
@@ -549,8 +553,7 @@ func (f *Finding) ShouldInvestigate(autonomyLevel string) bool {
InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeResolved ||
InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeFixRejected ||
InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeCannotFix ||
- InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeNeedsAttention ||
- InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeFixVerificationUnknown {
+ InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeNeedsAttention {
return false
}
@@ -684,8 +687,7 @@ func (f *Finding) CanRetryInvestigation() bool {
InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeResolved ||
InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeFixRejected ||
InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeCannotFix ||
- InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeNeedsAttention ||
- InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeFixVerificationUnknown {
+ InvestigationOutcome(f.InvestigationOutcome) == InvestigationOutcomeNeedsAttention {
return false
}
// Can't retry if currently running
diff --git a/internal/ai/findings_verification_followup_test.go b/internal/ai/findings_verification_followup_test.go
new file mode 100644
index 0000000000..5ebb0f7197
--- /dev/null
+++ b/internal/ai/findings_verification_followup_test.go
@@ -0,0 +1,78 @@
+package ai
+
+import (
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/ai/tools"
+ "github.com/rcourtman/pulse-go-rewrite/internal/config"
+)
+
+func TestUnknownVerificationAllowsFollowupWithoutAnotherCooldown(t *testing.T) {
+ completed := time.Now().Add(-15 * time.Minute)
+ f := &Finding{ID: "uncertain", Severity: FindingSeverityWarning,
+ InvestigationStatus: string(InvestigationStatusCompleted),
+ InvestigationOutcome: string(InvestigationOutcomeFixVerificationUnknown),
+ LastInvestigatedAt: &completed, InvestigationAttempts: 1}
+ if !f.ShouldInvestigate("full") || !f.CanRetryInvestigation() {
+ t.Fatal("an inconclusive action permanently suppressed follow-up")
+ }
+ recent := time.Now().Add(-5 * time.Minute)
+ f.LastInvestigatedAt = &recent
+ if !f.ShouldInvestigate("full") || !f.CanRetryInvestigation() {
+ t.Fatal("uncertain verification delayed a fresh model assessment")
+ }
+ f.LastInvestigatedAt = &completed
+ f.InvestigationStatus = string(InvestigationStatusRunning)
+ if f.ShouldInvestigate("full") || f.CanRetryInvestigation() {
+ t.Fatal("follow-up duplicated a running investigation")
+ }
+}
+
+func TestUnknownVerificationFollowupSurvivesRestartAndWaitsForWatchHandoff(t *testing.T) {
+ persistence := config.NewConfigPersistence(t.TempDir())
+ store := NewFindingsStore()
+ if err := store.SetPersistence(NewFindingsPersistenceAdapter(persistence)); err != nil {
+ t.Fatal(err)
+ }
+ completed := time.Now().Add(-time.Minute)
+ store.Add(&Finding{ID: "uncertain", ResourceID: "app-container:service", ResourceType: "app-container",
+ Title: "Service unavailable", Severity: FindingSeverityWarning, Category: FindingCategoryReliability,
+ InvestigationSessionID: "original", InvestigationStatus: string(InvestigationStatusCompleted),
+ InvestigationOutcome: string(InvestigationOutcomeFixVerificationUnknown), LastInvestigatedAt: &completed, InvestigationAttempts: 1})
+ if err := store.ForceSave(); err != nil {
+ t.Fatal(err)
+ }
+ restored := NewFindingsStore()
+ if err := restored.SetPersistence(NewFindingsPersistenceAdapter(persistence)); err != nil {
+ t.Fatal(err)
+ }
+ service := NewService(persistence, nil)
+ service.cfg = &config.AIConfig{Enabled: true, PatrolAutonomyLevel: config.PatrolAutonomyApproval}
+ patrol := NewPatrolService(service, nil)
+ patrol.findings = restored
+ orchestrator := newIntegrationOrchestrator(true)
+ patrol.SetInvestigationOrchestrator(orchestrator)
+ adapter := &patrolFindingCreatorAdapter{patrol: patrol}
+ if err := adapter.AssessFinding(tools.PatrolFindingAssessmentInput{FindingID: "uncertain", Verdict: "present", Evidence: "Current HTTP health remains unavailable after the uncertain restart"}); err != nil {
+ t.Fatal(err)
+ }
+ patrol.investigationWg.Wait()
+ if len(orchestrator.investigateCh) != 0 {
+ t.Fatal("investigation started while the Watch adapter still owned the executor")
+ }
+ current := restored.Get("uncertain")
+ if current.InvestigationSessionID != "original" || current.InvestigationAttempts != 1 || current.ResolvedAt != nil {
+ t.Fatalf("assessment discarded prior uncertainty: %#v", current)
+ }
+ patrol.dispatchPatrolInvestigations(&AIAnalysisResult{Findings: adapter.getCollectedFindings()})
+ patrol.investigationWg.Wait()
+ select {
+ case id := <-orchestrator.investigateCh:
+ if id != "uncertain" {
+ t.Fatalf("follow-up finding=%s", id)
+ }
+ default:
+ t.Fatal("fresh model assessment did not hand the unresolved finding to investigation")
+ }
+}
diff --git a/internal/ai/patrol_findings.go b/internal/ai/patrol_findings.go
index b286ce17cc..86f47d42d7 100644
--- a/internal/ai/patrol_findings.go
+++ b/internal/ai/patrol_findings.go
@@ -945,7 +945,7 @@ func (a *patrolFindingCreatorAdapter) AssessFinding(input tools.PatrolFindingAss
if a.objectiveContext != nil {
refreshed.ObjectiveContext = clonePatrolObjectiveContext(a.objectiveContext)
}
- if a.patrol.recordFinding(&refreshed) {
+ if a.patrol.recordFindingDuringPatrol(&refreshed) {
return fmt.Errorf("finding %s unexpectedly became a new finding during assessment", assessment.FindingID)
}
if stored := a.patrol.findings.Get(assessment.FindingID); stored != nil {
diff --git a/internal/api/docker_container_action_result.go b/internal/api/docker_container_action_result.go
index b5837501fc..4d18670a8a 100644
--- a/internal/api/docker_container_action_result.go
+++ b/internal/api/docker_container_action_result.go
@@ -33,15 +33,7 @@ func dockerContainerExecutionResult(resourceID, agentID string, req agentexec.Do
if err != nil {
return nil, fmt.Errorf("normalize docker lifecycle evidence: %w", err)
}
- status := unified.ActionVerificationContradicted
- reason := "postcondition_contradicted"
- if dockerLifecycleRunningHealthUnknown(req.Operation, facts.After) {
- status = unified.ActionVerificationInconclusive
- reason = "container_health_unknown"
- } else if dockerLifecycleFactsMatch(req.Operation, facts.Before, facts.After) {
- status = unified.ActionVerificationConfirmed
- reason = ""
- }
+ status, reason := dockerLifecycleVerification(req.Operation, facts.Before, facts.After)
verification = unified.ActionVerificationTruth{Status: status, EvidenceClass: unified.ActionEvidenceAgentAttested, ReasonCode: reason, Summary: summary, Evidence: []unified.ActionEvidence{evidence}}
}
}
@@ -65,12 +57,10 @@ func dockerContainerExecutionResult(resourceID, agentID string, req agentexec.Do
if err != nil {
return nil, fmt.Errorf("normalize independent docker observation: %w", err)
}
- status := unified.ActionVerificationContradicted
- reason := "postcondition_contradicted"
- if dockerLifecycleObservationFactsMatch(req.Operation, facts.Before, independent.Snapshot) {
- status = unified.ActionVerificationConfirmed
- reason = ""
- }
+ status, reason := dockerLifecycleVerification(req.Operation, facts.Before, agentexec.DockerContainerLifecycleSnapshot{
+ ContainerID: independent.Snapshot.ContainerID, State: independent.Snapshot.State, Running: independent.Snapshot.Running,
+ Health: independent.Snapshot.Health, StartedAt: independent.Snapshot.StartedAt,
+ })
verification = unified.ActionVerificationTruth{Status: status, EvidenceClass: unified.ActionEvidenceIndependent, ReasonCode: reason, Summary: independentSummary, Evidence: []unified.ActionEvidence{evidence}}
}
@@ -265,7 +255,11 @@ func dockerUpdateFactsMatch(facts agentexec.DockerContainerUpdateResultPayload,
}
func dockerContainerFactSummary(facts agentexec.DockerContainerLifecycleResultPayload) string {
- return fmt.Sprintf("Container lifecycle: phase=%s; mutation started=%t; mutation completed=%t; readback ran=%t; before=%s/%s; after=%s/%s", strings.TrimSpace(facts.ExecutionPhase), facts.MutationStarted, facts.MutationCompleted, facts.ReadbackRan, strings.TrimSpace(facts.Before.State), dockerLifecycleHealthSummary(facts.Before.Health), strings.TrimSpace(facts.After.State), dockerLifecycleHealthSummary(facts.After.Health))
+ summary := fmt.Sprintf("Container lifecycle: phase=%s; mutation started=%t; mutation completed=%t; readback ran=%t; before=%s/%s; after=%s/%s", strings.TrimSpace(facts.ExecutionPhase), facts.MutationStarted, facts.MutationCompleted, facts.ReadbackRan, strings.TrimSpace(facts.Before.State), dockerLifecycleHealthSummary(facts.Before.Health), strings.TrimSpace(facts.After.State), dockerLifecycleHealthSummary(facts.After.Health))
+ if errText := strings.TrimSpace(facts.Error); errText != "" {
+ summary += "; error=" + errText
+ }
+ return summary
}
func dockerLifecycleHealthSummary(health string) string {
@@ -298,13 +292,20 @@ func dockerLifecycleFactsMatch(operation string, before, after agentexec.DockerC
}
}
-func dockerLifecycleObservationFactsMatch(operation string, before agentexec.DockerContainerLifecycleSnapshot, after agentexec.DockerContainerObservationSnapshot) bool {
- return dockerLifecycleFactsMatch(operation, before, agentexec.DockerContainerLifecycleSnapshot{
- ContainerID: after.ContainerID, State: after.State, Running: after.Running, Health: after.Health,
- StartedAt: after.StartedAt, RestartCount: after.RestartCount, ObservedAt: after.ObservedAt,
- })
-}
-
func dockerLifecycleRunningHealthUnknown(operation string, after agentexec.DockerContainerLifecycleSnapshot) bool {
return (operation == agentexec.DockerContainerOperationStart || operation == agentexec.DockerContainerOperationRestart) && !agentexec.IsDockerContainerHealth(after.Health)
}
+
+func dockerLifecycleVerification(operation string, before, after agentexec.DockerContainerLifecycleSnapshot) (unified.ActionVerificationStatus, string) {
+ if dockerLifecycleFactsMatch(operation, before, after) {
+ return unified.ActionVerificationConfirmed, ""
+ }
+ if dockerLifecycleRunningHealthUnknown(operation, after) {
+ return unified.ActionVerificationInconclusive, "container_health_unknown"
+ }
+ if (operation == agentexec.DockerContainerOperationStart || operation == agentexec.DockerContainerOperationRestart) &&
+ after.Running && after.State == "running" && after.Health == agentexec.DockerContainerHealthStarting {
+ return unified.ActionVerificationInconclusive, "container_health_starting"
+ }
+ return unified.ActionVerificationContradicted, "postcondition_contradicted"
+}
diff --git a/internal/api/docker_container_action_result_test.go b/internal/api/docker_container_action_result_test.go
index 5292fe4d8f..01ee11a8a1 100644
--- a/internal/api/docker_container_action_result_test.go
+++ b/internal/api/docker_container_action_result_test.go
@@ -161,6 +161,24 @@ func TestDockerContainerExecutionResultDoesNotVerifyUnrecoveredHealth(t *testing
}
}
+func TestDockerContainerExecutionResultStartingHealthRemainsInconclusive(t *testing.T) {
+ now := time.Now().UTC()
+ facts := dockerResultFacts(now, true, false, false, true)
+ facts.ContainerID = dockerLifecycleTestID
+ observation := &dockerContainerPostconditionObservation{
+ ObserverID: "daemon", TrustDomain: "docker-daemon:agent-1", Method: "docker_api_inspect",
+ Snapshot: dockerObservationSnapshotFromLifecycle(facts.After), ReceivedAt: now,
+ }
+ observation.Snapshot.Health = agentexec.DockerContainerHealthStarting
+ result, err := dockerContainerExecutionResult("app-container:fixture", "agent-1", agentexec.DockerContainerLifecyclePayload{ActionID: "action", Operation: agentexec.DockerContainerOperationRestart}, facts, observation, now)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if result.ActionResultV2.Execution.Status != unified.ActionExecutionInconclusive || result.ActionResultV2.Verification.Status != unified.ActionVerificationInconclusive || result.ActionResultV2.Verification.ReasonCode != "container_health_starting" {
+ t.Fatalf("transitional health was treated as a final contradiction: %#v", result.ActionResultV2)
+ }
+}
+
func dockerObservationSnapshotFromLifecycle(snapshot agentexec.DockerContainerLifecycleSnapshot) agentexec.DockerContainerObservationSnapshot {
return agentexec.DockerContainerObservationSnapshot{
ContainerID: snapshot.ContainerID, State: snapshot.State, Running: snapshot.Running, Health: snapshot.Health,
diff --git a/internal/hostagent/commands.go b/internal/hostagent/commands.go
index c9f527762d..a85a4beb6f 100644
--- a/internal/hostagent/commands.go
+++ b/internal/hostagent/commands.go
@@ -1196,8 +1196,14 @@ func (c *CommandClient) replayDockerContainerLifecycle(conn *websocket.Conn, rec
}
func sanitizeDockerLifecycleError(value string) string {
- if strings.TrimSpace(value) == "" {
+ switch strings.TrimSpace(value) {
+ case "":
return ""
+ case "container lifecycle mutation acknowledgement timed out",
+ "container lifecycle mutation acknowledgement canceled",
+ "container lifecycle mutation acknowledgement unavailable":
+ // Fixed messages retain the cause without daemon output or endpoints.
+ return strings.TrimSpace(value)
}
return "typed container lifecycle did not complete as requested"
}
diff --git a/internal/hostagent/docker_lifecycle.go b/internal/hostagent/docker_lifecycle.go
index 41c76e7799..7b99a3fdb3 100644
--- a/internal/hostagent/docker_lifecycle.go
+++ b/internal/hostagent/docker_lifecycle.go
@@ -3,7 +3,9 @@ package hostagent
import (
"context"
"encoding/json"
+ "errors"
"fmt"
+ "net"
"os"
"os/exec"
"regexp"
@@ -94,21 +96,32 @@ func (m *localDockerLifecycleManager) Apply(ctx context.Context, req agentexec.D
result.ExecutionPhase = agentexec.DockerContainerPhaseMutate
result.MutationStarted = true
verb := strings.TrimSuffix(req.Operation, "_container")
- if err := m.mutate(ctx, req.Runtime, verb, req.ContainerID); err != nil {
- result.Error = "container lifecycle mutation did not complete"
- return result
+ mutationErr := m.mutate(ctx, req.Runtime, verb, req.ContainerID)
+ if mutationErr != nil {
+ // A lost acknowledgement does not prove that the daemon did nothing.
+ // Reconcile using reads under the original deadline, without retrying
+ // the mutation or claiming its execution was acknowledged.
+ result.Error = dockerLifecycleMutationError(mutationErr)
+ if ctx.Err() != nil {
+ return result
+ }
+ } else {
+ result.MutationCompleted = true
}
- result.MutationCompleted = true
result.ExecutionPhase = agentexec.DockerContainerPhaseVerify
after, err := m.inspect(ctx, req.Runtime, req.ContainerID)
result.After = after
if err != nil {
- result.Error = "container postcondition inspect unavailable"
+ if mutationErr == nil {
+ result.Error = "container postcondition inspect unavailable"
+ }
return result
}
result.ReadbackRan = true
if dockerLifecyclePostcondition(req, before, after) {
- result.ExecutionPhase = agentexec.DockerContainerPhaseComplete
+ if result.MutationCompleted {
+ result.ExecutionPhase = agentexec.DockerContainerPhaseComplete
+ }
return result
}
if dockerLifecycleHealthPending(req, before, after) {
@@ -120,20 +133,38 @@ func (m *localDockerLifecycleManager) Apply(ctx context.Context, req agentexec.D
}
after, err = m.inspect(verifyCtx, req.Runtime, req.ContainerID)
if err != nil {
- result.Error = "container postcondition inspect unavailable"
+ if mutationErr == nil {
+ result.Error = "container postcondition inspect unavailable"
+ }
return result
}
result.After = after
if dockerLifecyclePostcondition(req, before, after) {
- result.ExecutionPhase = agentexec.DockerContainerPhaseComplete
+ if result.MutationCompleted {
+ result.ExecutionPhase = agentexec.DockerContainerPhaseComplete
+ }
return result
}
}
}
- result.Error = "container postcondition contradicted the requested state"
+ if mutationErr == nil {
+ result.Error = "container postcondition did not reach the requested state"
+ }
return result
}
+func dockerLifecycleMutationError(err error) string {
+ var timeout net.Error
+ switch {
+ case errors.Is(err, context.DeadlineExceeded), errors.As(err, &timeout) && timeout.Timeout():
+ return "container lifecycle mutation acknowledgement timed out"
+ case errors.Is(err, context.Canceled):
+ return "container lifecycle mutation acknowledgement canceled"
+ default:
+ return "container lifecycle mutation acknowledgement unavailable"
+ }
+}
+
func (m *localDockerLifecycleManager) Preflight(ctx context.Context, req agentexec.DockerContainerLifecyclePayload) (bool, string) {
if err := agentexec.ValidateDockerContainerLifecyclePayload(&req); err != nil {
return false, agentexec.ActionRefusalContractInvalid
diff --git a/internal/hostagent/docker_lifecycle_test.go b/internal/hostagent/docker_lifecycle_test.go
index 66729b9b10..31a831e2dc 100644
--- a/internal/hostagent/docker_lifecycle_test.go
+++ b/internal/hostagent/docker_lifecycle_test.go
@@ -20,8 +20,9 @@ import (
const dockerLifecycleTestContainerID = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
type stubDockerLifecycleOperator struct {
- snapshots []agentexec.DockerContainerLifecycleSnapshot
- mutations []string
+ snapshots []agentexec.DockerContainerLifecycleSnapshot
+ mutations []string
+ mutationError error
}
func (s *stubDockerLifecycleOperator) InspectDockerContainerLifecycle(context.Context, string, string) (agentexec.DockerContainerLifecycleSnapshot, error) {
@@ -35,7 +36,62 @@ func (s *stubDockerLifecycleOperator) InspectDockerContainerLifecycle(context.Co
func (s *stubDockerLifecycleOperator) MutateDockerContainerLifecycle(_ context.Context, _, operation, _ string) error {
s.mutations = append(s.mutations, operation)
- return nil
+ return s.mutationError
+}
+
+func TestDockerLifecycleManagerReconcilesLostMutationAcknowledgement(t *testing.T) {
+ before := time.Now().UTC().Add(-time.Minute)
+ after := before.Add(time.Minute)
+ operator := &stubDockerLifecycleOperator{mutationError: context.DeadlineExceeded, snapshots: []agentexec.DockerContainerLifecycleSnapshot{
+ {ContainerID: dockerLifecycleTestContainerID, State: "running", Running: true, Health: agentexec.DockerContainerHealthUnhealthy, StartedAt: before, ObservedAt: before},
+ {ContainerID: dockerLifecycleTestContainerID, State: "running", Running: true, Health: agentexec.DockerContainerHealthStarting, StartedAt: after, ObservedAt: after},
+ {ContainerID: dockerLifecycleTestContainerID, State: "running", Running: true, Health: agentexec.DockerContainerHealthHealthy, StartedAt: after, ObservedAt: after.Add(time.Millisecond)},
+ }}
+ manager := newLocalDockerLifecycleManager(operator)
+ manager.healthPollInterval = time.Millisecond
+ result := manager.Apply(context.Background(), dockerLifecycleTestRequest(t, before))
+ if !result.MutationStarted || result.MutationCompleted || !result.ReadbackRan || result.After.Health != agentexec.DockerContainerHealthHealthy {
+ t.Fatalf("lost acknowledgement was not reconciled without inventing completion: %#v", result)
+ }
+ if result.Error == "" || len(operator.mutations) != 1 {
+ t.Fatalf("uncertainty or exactly-once mutation lost: result=%#v writes=%v", result, operator.mutations)
+ }
+}
+
+func TestDockerLifecycleManagerDoesNotDetachCanceledReconciliation(t *testing.T) {
+ t.Setenv("DOCKER_CONTEXT", "")
+ before := time.Now().UTC().Add(-time.Minute)
+ ctx, cancel := context.WithCancel(context.Background())
+ manager := newLocalDockerLifecycleManager()
+ writes, reads := 0, 0
+ manager.run = func(_ context.Context, _ string, args ...string) ([]byte, error) {
+ if args[0] == "restart" {
+ writes++
+ cancel()
+ return nil, context.Canceled
+ }
+ reads++
+ if args[2] == "{{.RestartCount}}" {
+ return []byte("0"), nil
+ }
+ return []byte(fmt.Sprintf(`{"Status":"running","Running":true,"StartedAt":%q}`, before.Format(time.RFC3339Nano))), nil
+ }
+ result := manager.Apply(ctx, dockerLifecycleTestRequest(t, before))
+ if writes != 1 || reads != 2 || result.ReadbackRan || result.MutationCompleted {
+ t.Fatalf("canceled operation detached or retried: result=%#v reads=%d writes=%d", result, reads, writes)
+ }
+}
+
+func TestDockerLifecycleErrorReceiptRetainsOnlySafeCauses(t *testing.T) {
+ for _, cause := range []error{context.DeadlineExceeded, context.Canceled, fmt.Errorf("daemon at https://user:[email protected] refused operation")} {
+ message := dockerLifecycleMutationError(cause)
+ if got := sanitizeDockerLifecycleError(message); got != message || strings.Contains(got, "secret") {
+ t.Fatalf("safe cause lost or leaked: %q", got)
+ }
+ }
+ if got := sanitizeDockerLifecycleError("daemon response: secret"); strings.Contains(got, "secret") {
+ t.Fatalf("raw daemon output leaked: %q", got)
+ }
}
func TestDockerLifecycleManagerUsesConnectedModuleWithoutExternalCLI(t *testing.T) {