Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

fix(docker): wait for update health checks before verifying

rcourtman7045d6f6e1bc
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 200bca9f7b..a79aea2f2a 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -4112,13 +4112,29 @@ has not settled or has deteriorated after module success. The API must retain
the completed mutation as execution history without treating replacement
identity alone as confirmed running health. Agent-attested verification requires
running state and healthy or explicit no-healthcheck evidence for running
-replacements; unknown health remains inconclusive. This does not trigger an
+replacements; unknown health remains inconclusive, and so does health that is
+still `starting` (`container_health_starting`). This does not trigger an
automatic resend or rollback and preserves deliberately stopped updates.
+Before the module declares success, it waits for a started replacement's
+health check to settle. Docker reports `starting` until the first probe passes,
+which with the default 30s interval is long after the 5s crash check, so that
+check alone rarely saw `unhealthy`. The module re-inspects every 2s for at
+most 3 minutes while health is `starting`. A replacement that turns
+`unhealthy`, stops, is `restarting` (Docker keeps `Running` set while a crashed
+container waits to restart), or cannot be inspected is rolled back to the
+original. One still `starting` at the cap is kept. Reaching the cap during an
+inspect ends the wait and does not fail the update. The cap is the per-Agent
+`healthSettleCap` seam (zero means 3 minutes), not a package global.
+
Verification: `TestDockerContainerUpdateAgentReadbackMustSupportRunningClaim`
asserts both verification outcomes and unchanged successful execution.
`TestDockerContainerActionExecutorDispatchesTypedUpdate` retains typed-only
dispatch with explicit no-healthcheck readback evidence.
+`TestUpdateContainer_WaitsForReplacementHealthToSettle` and
+`TestUpdateContainer_SettleCapEndsHungHealthInspect` in
+`internal/dockeragent/container_update_test.go` cover the settle wait, each
+rollback trigger, the cap and a hung inspect.
Command-capable agent completion must prove more than fresh telemetry. The
diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md
index 21cd9989fe..eb5f1c89ad 100644
--- a/docs/release-control/v6/internal/subsystems/api-contracts.md
+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md
@@ -4259,13 +4259,32 @@ container ID is insufficient to confirm a running agent readback. A running
replacement must report state `running` and health `healthy` or explicit
`none`. Known unsuitable health or lifecycle states contradict verification;
missing or unrecognised running health yields `inconclusive` with
-`container_health_unknown`. This does not change execution or compensation
-history, introduce a wire field, or require stopped replacements to run.
+`container_health_unknown`. A running replacement whose health check is still
+`starting` yields `inconclusive` with `container_health_starting`, matching
+start and restart verification. Docker has not yet settled the health check in
+that state (no probe has run, or failures have not reached the retry
+threshold), so it is not evidence against the update. This does not change
+execution or compensation history, introduce a wire field, or require stopped
+replacements to run.
+
+The Docker agent (`internal/dockeragent/container_update.go`) polls a started
+replacement whose health is `starting` every 2s, for at most 3 minutes, before
+judging it. Docker's default probe first runs 30s after start, so the 5s crash
+check alone usually saw `starting` on health-checked images and rarely reached
+its unhealthy rollback. A replacement that turns `unhealthy`, stops, or is
+`restarting` (Docker keeps `Running` set while a crashed container waits to
+restart) is rolled back. One still `starting` at the cap is kept, and the host
+agent's readback then reports `starting`.
Verification: `TestDockerContainerUpdateAgentReadbackMustSupportRunningClaim`
in `internal/api/docker_container_action_result_test.go` covers healthy,
no-healthcheck, stopped, unhealthy, starting, restarting and unknown-health
readbacks. The typed-dispatch fixture supplies explicit no-healthcheck evidence.
+`TestUpdateContainer_WaitsForReplacementHealthToSettle` in
+`internal/dockeragent/container_update_test.go` covers settling healthy,
+settling unhealthy, exiting, restarting or failing inspection during the wait,
+the cap and no health check. `TestUpdateContainer_SettleCapEndsHungHealthInspect`
+covers an inspect that hangs past the cap, which ends the wait without rollback.
The public connection ledger and action APIs must project telemetry liveness
diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md
index 40db273b47..1aa7438c8d 100644
--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md
+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md
@@ -918,6 +918,14 @@ retention or recovery authority is added. The focused
verification changes never rewrite successful execution history, including
stopped replacements and missing readback.
+A running replacement whose health check is still `starting` is now
+inconclusive rather than contradicted, and the Docker module waits for that
+health to settle before deciding whether to roll back to the original
+container. Both are action-verification and container-lifecycle behaviour.
+Restoring the renamed original container is still action compensation, not a
+storage backup, recovery point or restore, and no storage or recovery path
+is added.
+
Recovery consumers of security status must distinguish currentUsername (the validated caller identity) from privileged authUsername (administrator configuration). Presence of currentUsername in a scoped response does not authorise export, import, recovery or unprotected backup transfer; existing settings capabilities and backend enforcement still gate those operations.
diff --git a/internal/api/docker_container_action_result.go b/internal/api/docker_container_action_result.go
index 4d18670a8a..680d3f34eb 100644
--- a/internal/api/docker_container_action_result.go
+++ b/internal/api/docker_container_action_result.go
@@ -108,6 +108,11 @@ func dockerContainerUpdateExecutionResult(resourceID, agentID string, facts agen
if facts.After.Running && !agentexec.IsDockerContainerHealth(facts.After.Health) {
status = unified.ActionVerificationInconclusive
reason = "container_health_unknown"
+ } else if dockerUpdateFactsMatch(facts, facts.After.ContainerID) && dockerUpdateHealthStarting(facts.After.State, facts.After.Running, facts.After.Health) {
+ // A first probe that has not run yet is not evidence against
+ // the update, matching start and restart verification.
+ status = unified.ActionVerificationInconclusive
+ reason = "container_health_starting"
} else if dockerUpdateFactsMatch(facts, facts.After.ContainerID) &&
(!facts.After.Running || (facts.After.State == "running" &&
agentexec.DockerContainerHealthAllowsVerifiedRunningState(facts.After.Health))) {
@@ -146,6 +151,12 @@ func dockerContainerUpdateExecutionResult(resourceID, agentID string, facts agen
if !facts.ReadbackRan || facts.After.State == "" {
status = unified.ActionVerificationInconclusive
reason = "agent_readback_unavailable"
+ } else if dockerUpdateFactsMatch(facts, independent.Snapshot.ContainerID) &&
+ independent.Snapshot.State == facts.After.State &&
+ independent.Snapshot.Running == facts.After.Running &&
+ dockerUpdateHealthStarting(independent.Snapshot.State, independent.Snapshot.Running, independent.Snapshot.Health) {
+ status = unified.ActionVerificationInconclusive
+ reason = "container_health_starting"
} else if dockerUpdateFactsMatch(facts, independent.Snapshot.ContainerID) &&
independent.Snapshot.State == facts.After.State &&
independent.Snapshot.Running == facts.After.Running &&
@@ -254,6 +265,10 @@ func dockerUpdateFactsMatch(facts agentexec.DockerContainerUpdateResultPayload,
return strings.EqualFold(facts.NewContainerID, observedContainerID)
}
+func dockerUpdateHealthStarting(state string, running bool, health string) bool {
+ return running && state == "running" && strings.EqualFold(strings.TrimSpace(health), agentexec.DockerContainerHealthStarting)
+}
+
func dockerContainerFactSummary(facts agentexec.DockerContainerLifecycleResultPayload) string {
summary := fmt.Sprintf("Container lifecycle: phase=%s; mutation started=%t; mutation completed=%t; readback ran=%t; before=%s/%s; after=%s/%s", strings.TrimSpace(facts.ExecutionPhase), facts.MutationStarted, facts.MutationCompleted, facts.ReadbackRan, strings.TrimSpace(facts.Before.State), dockerLifecycleHealthSummary(facts.Before.Health), strings.TrimSpace(facts.After.State), dockerLifecycleHealthSummary(facts.After.Health))
if errText := strings.TrimSpace(facts.Error); errText != "" {
diff --git a/internal/api/docker_container_action_result_test.go b/internal/api/docker_container_action_result_test.go
index 01ee11a8a1..600d4ba403 100644
--- a/internal/api/docker_container_action_result_test.go
+++ b/internal/api/docker_container_action_result_test.go
@@ -232,7 +232,7 @@ func TestDockerContainerUpdateIndependentObservationMustMatchState(t *testing.T)
{"stopped after update", "running", "exited", "", false, unified.ActionVerificationContradicted},
{"restarting", "running", "restarting", "", true, unified.ActionVerificationContradicted},
{"unhealthy", "running", "running", "unhealthy", true, unified.ActionVerificationContradicted},
- {"starting healthcheck", "running", "running", "starting", true, unified.ActionVerificationContradicted},
+ {"starting healthcheck", "running", "running", "starting", true, unified.ActionVerificationInconclusive},
{"unknown health", "running", "running", "", true, unified.ActionVerificationContradicted},
{"missing agent readback", "", "running", "healthy", true, unified.ActionVerificationInconclusive},
} {
@@ -256,6 +256,9 @@ func TestDockerContainerUpdateIndependentObservationMustMatchState(t *testing.T)
if got := result.ActionResultV2.Verification; got.Status != tc.want || got.EvidenceClass != unified.ActionEvidenceIndependent {
t.Fatalf("verification = %+v, want %s / independent", got, tc.want)
}
+ if tc.health == "starting" && result.ActionResultV2.Verification.ReasonCode != "container_health_starting" {
+ t.Fatalf("reason = %q, want container_health_starting", result.ActionResultV2.Verification.ReasonCode)
+ }
if result.ActionResultV2.Execution.Status != unified.ActionExecutionSucceeded {
t.Fatal("readback changed execution history")
}
@@ -274,7 +277,7 @@ func TestDockerContainerUpdateAgentReadbackMustSupportRunningClaim(t *testing.T)
{"no healthcheck", "running", "none", true, unified.ActionVerificationConfirmed},
{"stopped preserved", "created", "none", false, unified.ActionVerificationConfirmed},
{"unhealthy", "running", "unhealthy", true, unified.ActionVerificationContradicted},
- {"starting", "running", "starting", true, unified.ActionVerificationContradicted},
+ {"starting", "running", "starting", true, unified.ActionVerificationInconclusive},
{"restarting", "restarting", "healthy", true, unified.ActionVerificationContradicted},
{"unknown health", "running", "", true, unified.ActionVerificationInconclusive},
} {
@@ -291,6 +294,9 @@ func TestDockerContainerUpdateAgentReadbackMustSupportRunningClaim(t *testing.T)
if got := result.ActionResultV2.Verification; got.Status != tc.want || got.EvidenceClass != unified.ActionEvidenceAgentAttested {
t.Fatalf("verification = %+v, want %s / agent_attested", got, tc.want)
}
+ if tc.health == "starting" && result.ActionResultV2.Verification.ReasonCode != "container_health_starting" {
+ t.Fatalf("reason = %q, want container_health_starting", result.ActionResultV2.Verification.ReasonCode)
+ }
if result.ActionResultV2.Execution.Status != unified.ActionExecutionSucceeded {
t.Fatal("readback changed execution history")
}
diff --git a/internal/dockeragent/agent.go b/internal/dockeragent/agent.go
index 971f202ab2..d7fc4bffd8 100644
--- a/internal/dockeragent/agent.go
+++ b/internal/dockeragent/agent.go
@@ -176,6 +176,7 @@ type Agent struct {
manualCheckCollect func(context.Context) (agentsdocker.Report, error) // test seam for bounded manual checks
newTimerFn func(time.Duration) *time.Timer // test seam; per-Agent so async goroutines never read a shared global (nil = time.NewTimer)
jsonMarshalFn func(any) ([]byte, error) // test seam; per-Agent so async goroutines never read a shared global (nil = json.Marshal)
+ healthSettleCap time.Duration // test seam; per-Agent update health wait (zero = dockerUpdateHealthSettleCap)
backgroundMu sync.Mutex // protects updateCheckRunning, cleanupTaskRunning
updateCheckRunning bool
cleanupTaskRunning bool
diff --git a/internal/dockeragent/agent_internal_test.go b/internal/dockeragent/agent_internal_test.go
index 6d5275553c..b8dd28b366 100644
--- a/internal/dockeragent/agent_internal_test.go
+++ b/internal/dockeragent/agent_internal_test.go
@@ -1415,6 +1415,7 @@ func TestAgentHookSeamsArePerAgentFields(t *testing.T) {
}{
{field: "newTimerFn", want: reflect.TypeOf((func(time.Duration) *time.Timer)(nil))},
{field: "jsonMarshalFn", want: reflect.TypeOf((func(any) ([]byte, error))(nil))},
+ {field: "healthSettleCap", want: reflect.TypeOf(time.Duration(0))},
} {
field, ok := agentType.FieldByName(tc.field)
if !ok {
@@ -1435,8 +1436,9 @@ func TestAgentHookSeamsArePerAgentFields(t *testing.T) {
// is exactly the race the per-Agent fields removed.
func TestAgentHookSeamsHaveNoPackageGlobals(t *testing.T) {
forbidden := map[string]struct{}{
- "newTimerFn": {},
- "jsonMarshalFn": {},
+ "newTimerFn": {},
+ "jsonMarshalFn": {},
+ "healthSettleCap": {},
}
sources, err := filepath.Glob("*.go")
diff --git a/internal/dockeragent/container_update.go b/internal/dockeragent/container_update.go
index 0dd94897d5..81b78f49b8 100644
--- a/internal/dockeragent/container_update.go
+++ b/internal/dockeragent/container_update.go
@@ -465,6 +465,13 @@ func (a *Agent) updateContainerWithProgress(ctx context.Context, containerID str
verifyInspect, err := dockerCallWithRetry(ctx, dockerUpdateCallTimeout, func(callCtx context.Context) (containertypes.InspectResponse, error) {
return a.docker.ContainerInspect(callCtx, newContainerID)
})
+ if err == nil && wasRunning && replacementHealthStarting(verifyInspect) {
+ // Docker reports "starting" until the first probe passes, which with
+ // the default 30s interval is long after the crash window above, so
+ // an unhealthy replacement would otherwise never be rolled back.
+ reportProgress("Waiting for container health check...")
+ verifyInspect, err = a.waitForReplacementHealth(ctx, newContainerID, verifyInspect)
+ }
if err != nil {
result.Error = fmt.Sprintf("Failed to inspect new container: %v", annotateDockerConnectionError(err))
a.logger.Error().Err(err).Str("container", result.ContainerName).Msg("Failed to inspect new container")
@@ -474,8 +481,9 @@ func (a *Agent) updateContainerWithProgress(ctx context.Context, containerID str
}
if wasRunning {
- // Check if running
- if verifyInspect.State != nil && !verifyInspect.State.Running {
+ // Check if running. Docker keeps Running set while a crashed
+ // container waits to be restarted, so restarting counts as a crash.
+ if verifyInspect.State != nil && (!verifyInspect.State.Running || verifyInspect.State.Restarting) {
exitCode := 0
stateErr := ""
if verifyInspect.State != nil {
@@ -526,6 +534,55 @@ func (a *Agent) updateContainerWithProgress(ctx context.Context, containerID str
return result
}
+const (
+ dockerUpdateHealthPollInterval = 2 * time.Second
+ // Docker's default probe settles unhealthy after three 30s intervals, so
+ // this covers default health checks without holding the update open
+ // indefinitely. A check still starting at the cap is reported as such.
+ dockerUpdateHealthSettleCap = 3 * time.Minute
+)
+
+func (a *Agent) updateHealthSettleCap() time.Duration {
+ if a.healthSettleCap > 0 {
+ return a.healthSettleCap
+ }
+ return dockerUpdateHealthSettleCap
+}
+
+func replacementHealthStarting(inspect containertypes.InspectResponse) bool {
+ return inspect.State != nil && inspect.State.Running && !inspect.State.Restarting && inspect.State.Health != nil &&
+ strings.EqualFold(strings.TrimSpace(string(inspect.State.Health.Status)), "starting")
+}
+
+// waitForReplacementHealth polls a started replacement until its health check
+// leaves "starting", the container stops running or restarts, or the settle
+// cap elapses, returning the latest inspect for the caller's crash and health
+// checks. Inspects run under the settle cap, so reaching it mid-inspect ends
+// the wait rather than failing the update; the poll count also bounds the
+// loop when sleepFn does not sleep.
+func (a *Agent) waitForReplacementHealth(ctx context.Context, containerID string, inspect containertypes.InspectResponse) (containertypes.InspectResponse, error) {
+ settleCap := a.updateHealthSettleCap()
+ settleCtx, cancel := context.WithTimeout(ctx, settleCap)
+ defer cancel()
+ for polls := int(settleCap / dockerUpdateHealthPollInterval); polls > 0 && replacementHealthStarting(inspect); polls-- {
+ if settleCtx.Err() != nil {
+ break
+ }
+ sleepFn(dockerUpdateHealthPollInterval)
+ next, err := dockerCallWithRetry(settleCtx, dockerUpdateCallTimeout, func(callCtx context.Context) (containertypes.InspectResponse, error) {
+ return a.docker.ContainerInspect(callCtx, containerID)
+ })
+ if err != nil {
+ if ctx.Err() == nil && settleCtx.Err() != nil {
+ break
+ }
+ return inspect, err
+ }
+ inspect = next
+ }
+ return inspect, nil
+}
+
func (a *Agent) rollbackRenameAndRestart(ctx context.Context, backupName, originalName, originalID string, restart bool) bool {
restored := true
if err := a.docker.ContainerRename(ctx, backupName, originalName); err != nil {
diff --git a/internal/dockeragent/container_update_test.go b/internal/dockeragent/container_update_test.go
index 5835efa576..52e069cdd8 100644
--- a/internal/dockeragent/container_update_test.go
+++ b/internal/dockeragent/container_update_test.go
@@ -834,3 +834,169 @@ func TestDecodeUpdateContainerPayloadUsesReceiverMarshaller(t *testing.T) {
}
})
}
+
+func TestUpdateContainer_WaitsForReplacementHealthToSettle(t *testing.T) {
+ swap(t, &sleepFn, func(time.Duration) {})
+ maxPolls := int(dockerUpdateHealthSettleCap / dockerUpdateHealthPollInterval)
+
+ for _, tc := range []struct {
+ name string
+ sequence []string // replacement health per inspect; the last value repeats
+ after string // once the sequence is exhausted: "exited", "restarting" or "error"
+ wantOK bool
+ wantPolls int
+ }{
+ {name: "starting then healthy", sequence: []string{"starting", "starting", "healthy"}, wantOK: true, wantPolls: 3},
+ {name: "starting then unhealthy rolls back", sequence: []string{"starting", "unhealthy"}, wantPolls: 2},
+ {name: "exits while starting rolls back", sequence: []string{"starting", "starting"}, after: "exited", wantPolls: 3},
+ {name: "restarting while starting rolls back", sequence: []string{"starting"}, after: "restarting", wantPolls: 2},
+ {name: "inspect failure while starting rolls back", sequence: []string{"starting"}, after: "error", wantPolls: 2},
+ {name: "still starting at the cap is kept", sequence: []string{"starting"}, wantOK: true, wantPolls: maxPolls + 1},
+ {name: "no health check does not wait", sequence: []string{""}, wantOK: true, wantPolls: 1},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ var (
+ mu sync.Mutex
+ polls int
+ removed []string
+ cleanupCh = make(chan struct{}, 1)
+ )
+ agent := &Agent{
+ docker: &fakeDockerClient{
+ containerInspectFn: func(_ context.Context, id string) (containertypes.InspectResponse, error) {
+ if id != "new123" {
+ return baseInspect(), nil
+ }
+ mu.Lock()
+ defer mu.Unlock()
+ inspect := baseInspect()
+ inspect.Image = "sha256:new0000000000"
+ health := tc.sequence[min(polls, len(tc.sequence)-1)]
+ exhausted := polls >= len(tc.sequence)
+ polls++
+ switch {
+ case exhausted && tc.after == "error":
+ return containertypes.InspectResponse{}, errors.New("daemon unavailable")
+ case exhausted && tc.after == "exited":
+ inspect.State = &containertypes.State{Running: false, ExitCode: 1}
+ case exhausted && tc.after == "restarting":
+ inspect.State = &containertypes.State{Running: true, Restarting: true, ExitCode: 1, Health: &containertypes.Health{Status: containertypes.Starting}}
+ case health != "":
+ inspect.State = &containertypes.State{Running: true, Health: &containertypes.Health{Status: containertypes.HealthStatus(health)}}
+ }
+ return inspect, nil
+ },
+ imagePullFn: func(context.Context, string, dockerImagePullOptions) (io.ReadCloser, error) {
+ return io.NopCloser(strings.NewReader("{}")), nil
+ },
+ containerStopFn: func(context.Context, string, dockerContainerStopOptions) error { return nil },
+ containerRenameFn: func(context.Context, string, string) error { return nil },
+ containerCreateFn: func(context.Context, *containertypes.Config, *containertypes.HostConfig, *network.NetworkingConfig, *v1.Platform, string) (containertypes.CreateResponse, error) {
+ return containertypes.CreateResponse{ID: "new123"}, nil
+ },
+ containerStartFn: func(context.Context, string, dockerContainerStartOptions) error { return nil },
+ containerRemoveFn: func(_ context.Context, id string, _ dockerContainerRemoveOptions) error {
+ mu.Lock()
+ removed = append(removed, id)
+ mu.Unlock()
+ select {
+ case cleanupCh <- struct{}{}:
+ default:
+ }
+ return nil
+ },
+ },
+ logger: zerolog.Nop(),
+ newTimerFn: immediateTimer,
+ }
+
+ result := agent.updateContainerWithProgress(context.Background(), "container1", nil)
+ if result.Success != tc.wantOK {
+ t.Fatalf("success = %t (error %q), want %t", result.Success, result.Error, tc.wantOK)
+ }
+ if tc.wantOK {
+ select { // the backup is removed asynchronously after success
+ case <-cleanupCh:
+ case <-time.After(5 * time.Second):
+ t.Fatal("backup cleanup did not run")
+ }
+ }
+ mu.Lock()
+ defer mu.Unlock()
+ if polls != tc.wantPolls {
+ t.Fatalf("replacement inspected %d times, want %d", polls, tc.wantPolls)
+ }
+ if tc.wantOK {
+ if result.RollbackAttempted {
+ t.Fatal("a healthy or still-starting replacement must not be rolled back")
+ }
+ return
+ }
+ if !result.RollbackAttempted || len(removed) == 0 || removed[0] != "new123" {
+ t.Fatalf("expected rollback to remove the replacement, removed=%v rollback=%t", removed, result.RollbackAttempted)
+ }
+ })
+ }
+}
+
+func TestUpdateContainer_SettleCapEndsHungHealthInspect(t *testing.T) {
+ swap(t, &sleepFn, func(time.Duration) {})
+
+ var (
+ mu sync.Mutex
+ polls int
+ rolled bool
+ )
+ agent := &Agent{
+ docker: &fakeDockerClient{
+ containerInspectFn: func(ctx context.Context, id string) (containertypes.InspectResponse, error) {
+ if id != "new123" {
+ return baseInspect(), nil
+ }
+ mu.Lock()
+ polls++
+ first := polls == 1
+ mu.Unlock()
+ if !first {
+ <-ctx.Done() // a daemon that stops answering mid-wait
+ return containertypes.InspectResponse{}, ctx.Err()
+ }
+ inspect := baseInspect()
+ inspect.State = &containertypes.State{Running: true, Health: &containertypes.Health{Status: containertypes.Starting}}
+ return inspect, nil
+ },
+ imagePullFn: func(context.Context, string, dockerImagePullOptions) (io.ReadCloser, error) {
+ return io.NopCloser(strings.NewReader("{}")), nil
+ },
+ containerStopFn: func(context.Context, string, dockerContainerStopOptions) error { return nil },
+ containerRenameFn: func(context.Context, string, string) error { return nil },
+ containerCreateFn: func(context.Context, *containertypes.Config, *containertypes.HostConfig, *network.NetworkingConfig, *v1.Platform, string) (containertypes.CreateResponse, error) {
+ return containertypes.CreateResponse{ID: "new123"}, nil
+ },
+ containerStartFn: func(context.Context, string, dockerContainerStartOptions) error { return nil },
+ containerRemoveFn: func(_ context.Context, id string, _ dockerContainerRemoveOptions) error {
+ mu.Lock()
+ rolled = rolled || id == "new123"
+ mu.Unlock()
+ return nil
+ },
+ },
+ logger: zerolog.Nop(),
+ newTimerFn: immediateTimer,
+ healthSettleCap: 4 * time.Second,
+ }
+
+ started := time.Now()
+ result := agent.updateContainerWithProgress(context.Background(), "container1", nil)
+ if elapsed := time.Since(started); elapsed > 30*time.Second {
+ t.Fatalf("hung inspect held the update for %s, want it bounded by the settle cap", elapsed)
+ }
+ if !result.Success || result.RollbackAttempted {
+ t.Fatalf("reaching the settle cap mid-inspect must keep the replacement: success=%t rollback=%t error=%q", result.Success, result.RollbackAttempted, result.Error)
+ }
+ mu.Lock()
+ defer mu.Unlock()
+ if rolled {
+ t.Fatal("the replacement was removed")
+ }
+}