Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
fix(docker): wait for update health checks before verifying
rcourtman7045d6f6e1bc
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 200bca9f7b..a79aea2f2a 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -4112,13 +4112,29 @@ has not settled or has deteriorated after module success. The API must retainthe completed mutation as execution history without treating replacementidentity alone as confirmed running health. Agent-attested verification requiresrunning state and healthy or explicit no-healthcheck evidence for running-replacements; unknown health remains inconclusive. This does not trigger an+replacements; unknown health remains inconclusive, and so does health that is+still `starting` (`container_health_starting`). This does not trigger anautomatic resend or rollback and preserves deliberately stopped updates.+Before the module declares success, it waits for a started replacement's+health check to settle. Docker reports `starting` until the first probe passes,+which with the default 30s interval is long after the 5s crash check, so that+check alone rarely saw `unhealthy`. The module re-inspects every 2s for at+most 3 minutes while health is `starting`. A replacement that turns+`unhealthy`, stops, is `restarting` (Docker keeps `Running` set while a crashed+container waits to restart), or cannot be inspected is rolled back to the+original. One still `starting` at the cap is kept. Reaching the cap during an+inspect ends the wait and does not fail the update. The cap is the per-Agent+`healthSettleCap` seam (zero means 3 minutes), not a package global.+Verification: `TestDockerContainerUpdateAgentReadbackMustSupportRunningClaim`asserts both verification outcomes and unchanged successful execution.`TestDockerContainerActionExecutorDispatchesTypedUpdate` retains typed-onlydispatch with explicit no-healthcheck readback evidence.+`TestUpdateContainer_WaitsForReplacementHealthToSettle` and+`TestUpdateContainer_SettleCapEndsHungHealthInspect` in+`internal/dockeragent/container_update_test.go` cover the settle wait, each+rollback trigger, the cap and a hung inspect.Command-capable agent completion must prove more than fresh telemetry. Thediff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.mdindex 21cd9989fe..eb5f1c89ad 100644--- a/docs/release-control/v6/internal/subsystems/api-contracts.md+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md@@ -4259,13 +4259,32 @@ container ID is insufficient to confirm a running agent readback. A runningreplacement must report state `running` and health `healthy` or explicit`none`. Known unsuitable health or lifecycle states contradict verification;missing or unrecognised running health yields `inconclusive` with-`container_health_unknown`. This does not change execution or compensation-history, introduce a wire field, or require stopped replacements to run.+`container_health_unknown`. A running replacement whose health check is still+`starting` yields `inconclusive` with `container_health_starting`, matching+start and restart verification. Docker has not yet settled the health check in+that state (no probe has run, or failures have not reached the retry+threshold), so it is not evidence against the update. This does not change+execution or compensation history, introduce a wire field, or require stopped+replacements to run.++The Docker agent (`internal/dockeragent/container_update.go`) polls a started+replacement whose health is `starting` every 2s, for at most 3 minutes, before+judging it. Docker's default probe first runs 30s after start, so the 5s crash+check alone usually saw `starting` on health-checked images and rarely reached+its unhealthy rollback. A replacement that turns `unhealthy`, stops, or is+`restarting` (Docker keeps `Running` set while a crashed container waits to+restart) is rolled back. One still `starting` at the cap is kept, and the host+agent's readback then reports `starting`.Verification: `TestDockerContainerUpdateAgentReadbackMustSupportRunningClaim`in `internal/api/docker_container_action_result_test.go` covers healthy,no-healthcheck, stopped, unhealthy, starting, restarting and unknown-healthreadbacks. The typed-dispatch fixture supplies explicit no-healthcheck evidence.+`TestUpdateContainer_WaitsForReplacementHealthToSettle` in+`internal/dockeragent/container_update_test.go` covers settling healthy,+settling unhealthy, exiting, restarting or failing inspection during the wait,+the cap and no health check. `TestUpdateContainer_SettleCapEndsHungHealthInspect`+covers an inspect that hangs past the cap, which ends the wait without rollback.The public connection ledger and action APIs must project telemetry livenessdiff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex 40db273b47..1aa7438c8d 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -918,6 +918,14 @@ retention or recovery authority is added. The focusedverification changes never rewrite successful execution history, includingstopped replacements and missing readback.+A running replacement whose health check is still `starting` is now+inconclusive rather than contradicted, and the Docker module waits for that+health to settle before deciding whether to roll back to the original+container. Both are action-verification and container-lifecycle behaviour.+Restoring the renamed original container is still action compensation, not a+storage backup, recovery point or restore, and no storage or recovery path+is added.+Recovery consumers of security status must distinguish currentUsername (the validated caller identity) from privileged authUsername (administrator configuration). Presence of currentUsername in a scoped response does not authorise export, import, recovery or unprotected backup transfer; existing settings capabilities and backend enforcement still gate those operations.diff --git a/internal/api/docker_container_action_result.go b/internal/api/docker_container_action_result.goindex 4d18670a8a..680d3f34eb 100644--- a/internal/api/docker_container_action_result.go+++ b/internal/api/docker_container_action_result.go@@ -108,6 +108,11 @@ func dockerContainerUpdateExecutionResult(resourceID, agentID string, facts agenif facts.After.Running && !agentexec.IsDockerContainerHealth(facts.After.Health) {status = unified.ActionVerificationInconclusivereason = "container_health_unknown"+ } else if dockerUpdateFactsMatch(facts, facts.After.ContainerID) && dockerUpdateHealthStarting(facts.After.State, facts.After.Running, facts.After.Health) {+ // A first probe that has not run yet is not evidence against+ // the update, matching start and restart verification.+ status = unified.ActionVerificationInconclusive+ reason = "container_health_starting"} else if dockerUpdateFactsMatch(facts, facts.After.ContainerID) &&(!facts.After.Running || (facts.After.State == "running" &&agentexec.DockerContainerHealthAllowsVerifiedRunningState(facts.After.Health))) {@@ -146,6 +151,12 @@ func dockerContainerUpdateExecutionResult(resourceID, agentID string, facts agenif !facts.ReadbackRan || facts.After.State == "" {status = unified.ActionVerificationInconclusivereason = "agent_readback_unavailable"+ } else if dockerUpdateFactsMatch(facts, independent.Snapshot.ContainerID) &&+ independent.Snapshot.State == facts.After.State &&+ independent.Snapshot.Running == facts.After.Running &&+ dockerUpdateHealthStarting(independent.Snapshot.State, independent.Snapshot.Running, independent.Snapshot.Health) {+ status = unified.ActionVerificationInconclusive+ reason = "container_health_starting"} else if dockerUpdateFactsMatch(facts, independent.Snapshot.ContainerID) &&independent.Snapshot.State == facts.After.State &&independent.Snapshot.Running == facts.After.Running &&@@ -254,6 +265,10 @@ func dockerUpdateFactsMatch(facts agentexec.DockerContainerUpdateResultPayload,return strings.EqualFold(facts.NewContainerID, observedContainerID)}+func dockerUpdateHealthStarting(state string, running bool, health string) bool {+ return running && state == "running" && strings.EqualFold(strings.TrimSpace(health), agentexec.DockerContainerHealthStarting)+}+func dockerContainerFactSummary(facts agentexec.DockerContainerLifecycleResultPayload) string {summary := fmt.Sprintf("Container lifecycle: phase=%s; mutation started=%t; mutation completed=%t; readback ran=%t; before=%s/%s; after=%s/%s", strings.TrimSpace(facts.ExecutionPhase), facts.MutationStarted, facts.MutationCompleted, facts.ReadbackRan, strings.TrimSpace(facts.Before.State), dockerLifecycleHealthSummary(facts.Before.Health), strings.TrimSpace(facts.After.State), dockerLifecycleHealthSummary(facts.After.Health))if errText := strings.TrimSpace(facts.Error); errText != "" {diff --git a/internal/api/docker_container_action_result_test.go b/internal/api/docker_container_action_result_test.goindex 01ee11a8a1..600d4ba403 100644--- a/internal/api/docker_container_action_result_test.go+++ b/internal/api/docker_container_action_result_test.go@@ -232,7 +232,7 @@ func TestDockerContainerUpdateIndependentObservationMustMatchState(t *testing.T){"stopped after update", "running", "exited", "", false, unified.ActionVerificationContradicted},{"restarting", "running", "restarting", "", true, unified.ActionVerificationContradicted},{"unhealthy", "running", "running", "unhealthy", true, unified.ActionVerificationContradicted},- {"starting healthcheck", "running", "running", "starting", true, unified.ActionVerificationContradicted},+ {"starting healthcheck", "running", "running", "starting", true, unified.ActionVerificationInconclusive},{"unknown health", "running", "running", "", true, unified.ActionVerificationContradicted},{"missing agent readback", "", "running", "healthy", true, unified.ActionVerificationInconclusive},} {@@ -256,6 +256,9 @@ func TestDockerContainerUpdateIndependentObservationMustMatchState(t *testing.T)if got := result.ActionResultV2.Verification; got.Status != tc.want || got.EvidenceClass != unified.ActionEvidenceIndependent {t.Fatalf("verification = %+v, want %s / independent", got, tc.want)}+ if tc.health == "starting" && result.ActionResultV2.Verification.ReasonCode != "container_health_starting" {+ t.Fatalf("reason = %q, want container_health_starting", result.ActionResultV2.Verification.ReasonCode)+ }if result.ActionResultV2.Execution.Status != unified.ActionExecutionSucceeded {t.Fatal("readback changed execution history")}@@ -274,7 +277,7 @@ func TestDockerContainerUpdateAgentReadbackMustSupportRunningClaim(t *testing.T){"no healthcheck", "running", "none", true, unified.ActionVerificationConfirmed},{"stopped preserved", "created", "none", false, unified.ActionVerificationConfirmed},{"unhealthy", "running", "unhealthy", true, unified.ActionVerificationContradicted},- {"starting", "running", "starting", true, unified.ActionVerificationContradicted},+ {"starting", "running", "starting", true, unified.ActionVerificationInconclusive},{"restarting", "restarting", "healthy", true, unified.ActionVerificationContradicted},{"unknown health", "running", "", true, unified.ActionVerificationInconclusive},} {@@ -291,6 +294,9 @@ func TestDockerContainerUpdateAgentReadbackMustSupportRunningClaim(t *testing.T)if got := result.ActionResultV2.Verification; got.Status != tc.want || got.EvidenceClass != unified.ActionEvidenceAgentAttested {t.Fatalf("verification = %+v, want %s / agent_attested", got, tc.want)}+ if tc.health == "starting" && result.ActionResultV2.Verification.ReasonCode != "container_health_starting" {+ t.Fatalf("reason = %q, want container_health_starting", result.ActionResultV2.Verification.ReasonCode)+ }if result.ActionResultV2.Execution.Status != unified.ActionExecutionSucceeded {t.Fatal("readback changed execution history")}diff --git a/internal/dockeragent/agent.go b/internal/dockeragent/agent.goindex 971f202ab2..d7fc4bffd8 100644--- a/internal/dockeragent/agent.go+++ b/internal/dockeragent/agent.go@@ -176,6 +176,7 @@ type Agent struct {manualCheckCollect func(context.Context) (agentsdocker.Report, error) // test seam for bounded manual checksnewTimerFn func(time.Duration) *time.Timer // test seam; per-Agent so async goroutines never read a shared global (nil = time.NewTimer)jsonMarshalFn func(any) ([]byte, error) // test seam; per-Agent so async goroutines never read a shared global (nil = json.Marshal)+ healthSettleCap time.Duration // test seam; per-Agent update health wait (zero = dockerUpdateHealthSettleCap)backgroundMu sync.Mutex // protects updateCheckRunning, cleanupTaskRunningupdateCheckRunning boolcleanupTaskRunning booldiff --git a/internal/dockeragent/agent_internal_test.go b/internal/dockeragent/agent_internal_test.goindex 6d5275553c..b8dd28b366 100644--- a/internal/dockeragent/agent_internal_test.go+++ b/internal/dockeragent/agent_internal_test.go@@ -1415,6 +1415,7 @@ func TestAgentHookSeamsArePerAgentFields(t *testing.T) {}{{field: "newTimerFn", want: reflect.TypeOf((func(time.Duration) *time.Timer)(nil))},{field: "jsonMarshalFn", want: reflect.TypeOf((func(any) ([]byte, error))(nil))},+ {field: "healthSettleCap", want: reflect.TypeOf(time.Duration(0))},} {field, ok := agentType.FieldByName(tc.field)if !ok {@@ -1435,8 +1436,9 @@ func TestAgentHookSeamsArePerAgentFields(t *testing.T) {// is exactly the race the per-Agent fields removed.func TestAgentHookSeamsHaveNoPackageGlobals(t *testing.T) {forbidden := map[string]struct{}{- "newTimerFn": {},- "jsonMarshalFn": {},+ "newTimerFn": {},+ "jsonMarshalFn": {},+ "healthSettleCap": {},}sources, err := filepath.Glob("*.go")diff --git a/internal/dockeragent/container_update.go b/internal/dockeragent/container_update.goindex 0dd94897d5..81b78f49b8 100644--- a/internal/dockeragent/container_update.go+++ b/internal/dockeragent/container_update.go@@ -465,6 +465,13 @@ func (a *Agent) updateContainerWithProgress(ctx context.Context, containerID strverifyInspect, err := dockerCallWithRetry(ctx, dockerUpdateCallTimeout, func(callCtx context.Context) (containertypes.InspectResponse, error) {return a.docker.ContainerInspect(callCtx, newContainerID)})+ if err == nil && wasRunning && replacementHealthStarting(verifyInspect) {+ // Docker reports "starting" until the first probe passes, which with+ // the default 30s interval is long after the crash window above, so+ // an unhealthy replacement would otherwise never be rolled back.+ reportProgress("Waiting for container health check...")+ verifyInspect, err = a.waitForReplacementHealth(ctx, newContainerID, verifyInspect)+ }if err != nil {result.Error = fmt.Sprintf("Failed to inspect new container: %v", annotateDockerConnectionError(err))a.logger.Error().Err(err).Str("container", result.ContainerName).Msg("Failed to inspect new container")@@ -474,8 +481,9 @@ func (a *Agent) updateContainerWithProgress(ctx context.Context, containerID str}if wasRunning {- // Check if running- if verifyInspect.State != nil && !verifyInspect.State.Running {+ // Check if running. Docker keeps Running set while a crashed+ // container waits to be restarted, so restarting counts as a crash.+ if verifyInspect.State != nil && (!verifyInspect.State.Running || verifyInspect.State.Restarting) {exitCode := 0stateErr := ""if verifyInspect.State != nil {@@ -526,6 +534,55 @@ func (a *Agent) updateContainerWithProgress(ctx context.Context, containerID strreturn result}+const (+ dockerUpdateHealthPollInterval = 2 * time.Second+ // Docker's default probe settles unhealthy after three 30s intervals, so+ // this covers default health checks without holding the update open+ // indefinitely. A check still starting at the cap is reported as such.+ dockerUpdateHealthSettleCap = 3 * time.Minute+)++func (a *Agent) updateHealthSettleCap() time.Duration {+ if a.healthSettleCap > 0 {+ return a.healthSettleCap+ }+ return dockerUpdateHealthSettleCap+}++func replacementHealthStarting(inspect containertypes.InspectResponse) bool {+ return inspect.State != nil && inspect.State.Running && !inspect.State.Restarting && inspect.State.Health != nil &&+ strings.EqualFold(strings.TrimSpace(string(inspect.State.Health.Status)), "starting")+}++// waitForReplacementHealth polls a started replacement until its health check+// leaves "starting", the container stops running or restarts, or the settle+// cap elapses, returning the latest inspect for the caller's crash and health+// checks. Inspects run under the settle cap, so reaching it mid-inspect ends+// the wait rather than failing the update; the poll count also bounds the+// loop when sleepFn does not sleep.+func (a *Agent) waitForReplacementHealth(ctx context.Context, containerID string, inspect containertypes.InspectResponse) (containertypes.InspectResponse, error) {+ settleCap := a.updateHealthSettleCap()+ settleCtx, cancel := context.WithTimeout(ctx, settleCap)+ defer cancel()+ for polls := int(settleCap / dockerUpdateHealthPollInterval); polls > 0 && replacementHealthStarting(inspect); polls-- {+ if settleCtx.Err() != nil {+ break+ }+ sleepFn(dockerUpdateHealthPollInterval)+ next, err := dockerCallWithRetry(settleCtx, dockerUpdateCallTimeout, func(callCtx context.Context) (containertypes.InspectResponse, error) {+ return a.docker.ContainerInspect(callCtx, containerID)+ })+ if err != nil {+ if ctx.Err() == nil && settleCtx.Err() != nil {+ break+ }+ return inspect, err+ }+ inspect = next+ }+ return inspect, nil+}+func (a *Agent) rollbackRenameAndRestart(ctx context.Context, backupName, originalName, originalID string, restart bool) bool {restored := trueif err := a.docker.ContainerRename(ctx, backupName, originalName); err != nil {diff --git a/internal/dockeragent/container_update_test.go b/internal/dockeragent/container_update_test.goindex 5835efa576..52e069cdd8 100644--- a/internal/dockeragent/container_update_test.go+++ b/internal/dockeragent/container_update_test.go@@ -834,3 +834,169 @@ func TestDecodeUpdateContainerPayloadUsesReceiverMarshaller(t *testing.T) {}})}++func TestUpdateContainer_WaitsForReplacementHealthToSettle(t *testing.T) {+ swap(t, &sleepFn, func(time.Duration) {})+ maxPolls := int(dockerUpdateHealthSettleCap / dockerUpdateHealthPollInterval)++ for _, tc := range []struct {+ name string+ sequence []string // replacement health per inspect; the last value repeats+ after string // once the sequence is exhausted: "exited", "restarting" or "error"+ wantOK bool+ wantPolls int+ }{+ {name: "starting then healthy", sequence: []string{"starting", "starting", "healthy"}, wantOK: true, wantPolls: 3},+ {name: "starting then unhealthy rolls back", sequence: []string{"starting", "unhealthy"}, wantPolls: 2},+ {name: "exits while starting rolls back", sequence: []string{"starting", "starting"}, after: "exited", wantPolls: 3},+ {name: "restarting while starting rolls back", sequence: []string{"starting"}, after: "restarting", wantPolls: 2},+ {name: "inspect failure while starting rolls back", sequence: []string{"starting"}, after: "error", wantPolls: 2},+ {name: "still starting at the cap is kept", sequence: []string{"starting"}, wantOK: true, wantPolls: maxPolls + 1},+ {name: "no health check does not wait", sequence: []string{""}, wantOK: true, wantPolls: 1},+ } {+ t.Run(tc.name, func(t *testing.T) {+ var (+ mu sync.Mutex+ polls int+ removed []string+ cleanupCh = make(chan struct{}, 1)+ )+ agent := &Agent{+ docker: &fakeDockerClient{+ containerInspectFn: func(_ context.Context, id string) (containertypes.InspectResponse, error) {+ if id != "new123" {+ return baseInspect(), nil+ }+ mu.Lock()+ defer mu.Unlock()+ inspect := baseInspect()+ inspect.Image = "sha256:new0000000000"+ health := tc.sequence[min(polls, len(tc.sequence)-1)]+ exhausted := polls >= len(tc.sequence)+ polls+++ switch {+ case exhausted && tc.after == "error":+ return containertypes.InspectResponse{}, errors.New("daemon unavailable")+ case exhausted && tc.after == "exited":+ inspect.State = &containertypes.State{Running: false, ExitCode: 1}+ case exhausted && tc.after == "restarting":+ inspect.State = &containertypes.State{Running: true, Restarting: true, ExitCode: 1, Health: &containertypes.Health{Status: containertypes.Starting}}+ case health != "":+ inspect.State = &containertypes.State{Running: true, Health: &containertypes.Health{Status: containertypes.HealthStatus(health)}}+ }+ return inspect, nil+ },+ imagePullFn: func(context.Context, string, dockerImagePullOptions) (io.ReadCloser, error) {+ return io.NopCloser(strings.NewReader("{}")), nil+ },+ containerStopFn: func(context.Context, string, dockerContainerStopOptions) error { return nil },+ containerRenameFn: func(context.Context, string, string) error { return nil },+ containerCreateFn: func(context.Context, *containertypes.Config, *containertypes.HostConfig, *network.NetworkingConfig, *v1.Platform, string) (containertypes.CreateResponse, error) {+ return containertypes.CreateResponse{ID: "new123"}, nil+ },+ containerStartFn: func(context.Context, string, dockerContainerStartOptions) error { return nil },+ containerRemoveFn: func(_ context.Context, id string, _ dockerContainerRemoveOptions) error {+ mu.Lock()+ removed = append(removed, id)+ mu.Unlock()+ select {+ case cleanupCh <- struct{}{}:+ default:+ }+ return nil+ },+ },+ logger: zerolog.Nop(),+ newTimerFn: immediateTimer,+ }++ result := agent.updateContainerWithProgress(context.Background(), "container1", nil)+ if result.Success != tc.wantOK {+ t.Fatalf("success = %t (error %q), want %t", result.Success, result.Error, tc.wantOK)+ }+ if tc.wantOK {+ select { // the backup is removed asynchronously after success+ case <-cleanupCh:+ case <-time.After(5 * time.Second):+ t.Fatal("backup cleanup did not run")+ }+ }+ mu.Lock()+ defer mu.Unlock()+ if polls != tc.wantPolls {+ t.Fatalf("replacement inspected %d times, want %d", polls, tc.wantPolls)+ }+ if tc.wantOK {+ if result.RollbackAttempted {+ t.Fatal("a healthy or still-starting replacement must not be rolled back")+ }+ return+ }+ if !result.RollbackAttempted || len(removed) == 0 || removed[0] != "new123" {+ t.Fatalf("expected rollback to remove the replacement, removed=%v rollback=%t", removed, result.RollbackAttempted)+ }+ })+ }+}++func TestUpdateContainer_SettleCapEndsHungHealthInspect(t *testing.T) {+ swap(t, &sleepFn, func(time.Duration) {})++ var (+ mu sync.Mutex+ polls int+ rolled bool+ )+ agent := &Agent{+ docker: &fakeDockerClient{+ containerInspectFn: func(ctx context.Context, id string) (containertypes.InspectResponse, error) {+ if id != "new123" {+ return baseInspect(), nil+ }+ mu.Lock()+ polls+++ first := polls == 1+ mu.Unlock()+ if !first {+ <-ctx.Done() // a daemon that stops answering mid-wait+ return containertypes.InspectResponse{}, ctx.Err()+ }+ inspect := baseInspect()+ inspect.State = &containertypes.State{Running: true, Health: &containertypes.Health{Status: containertypes.Starting}}+ return inspect, nil+ },+ imagePullFn: func(context.Context, string, dockerImagePullOptions) (io.ReadCloser, error) {+ return io.NopCloser(strings.NewReader("{}")), nil+ },+ containerStopFn: func(context.Context, string, dockerContainerStopOptions) error { return nil },+ containerRenameFn: func(context.Context, string, string) error { return nil },+ containerCreateFn: func(context.Context, *containertypes.Config, *containertypes.HostConfig, *network.NetworkingConfig, *v1.Platform, string) (containertypes.CreateResponse, error) {+ return containertypes.CreateResponse{ID: "new123"}, nil+ },+ containerStartFn: func(context.Context, string, dockerContainerStartOptions) error { return nil },+ containerRemoveFn: func(_ context.Context, id string, _ dockerContainerRemoveOptions) error {+ mu.Lock()+ rolled = rolled || id == "new123"+ mu.Unlock()+ return nil+ },+ },+ logger: zerolog.Nop(),+ newTimerFn: immediateTimer,+ healthSettleCap: 4 * time.Second,+ }++ started := time.Now()+ result := agent.updateContainerWithProgress(context.Background(), "container1", nil)+ if elapsed := time.Since(started); elapsed > 30*time.Second {+ t.Fatalf("hung inspect held the update for %s, want it bounded by the settle cap", elapsed)+ }+ if !result.Success || result.RollbackAttempted {+ t.Fatalf("reaching the settle cap mid-inspect must keep the replacement: success=%t rollback=%t error=%q", result.Success, result.RollbackAttempted, result.Error)+ }+ mu.Lock()+ defer mu.Unlock()+ if rolled {+ t.Fatal("the replacement was removed")+ }+}