Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

docs(#391): pipeline-owned HCC Recreate recovery + e2e undo path

Alfredo Lopez58dc455ec1e3
diff --git a/.github/workflows/ci-public.yml b/.github/workflows/ci-public.yml
index 8da700bbe..35b2d3d3a 100644
--- a/.github/workflows/ci-public.yml
+++ b/.github/workflows/ci-public.yml
@@ -397,6 +397,9 @@ jobs:
- name: Test HCC watch-churn readiness gate contract
run: bash scripts/tests/test-hcc-watch-churn-readiness-gate.sh
+ - name: Test HCC rollout readiness gate contract
+ run: bash scripts/tests/test-hcc-rollout-readiness-gate.sh
+
- name: Test HCC API gateway readiness separation
run: bash scripts/tests/test-hcc-api-gateway-readiness.sh
diff --git a/deploy/base/control-plane/host-context-controller.yaml b/deploy/base/control-plane/host-context-controller.yaml
index 24a4c21dd..8b48cb482 100644
--- a/deploy/base/control-plane/host-context-controller.yaml
+++ b/deploy/base/control-plane/host-context-controller.yaml
@@ -24,11 +24,12 @@ spec:
strategy:
type: Recreate
# Recreate + replicas:1 means a botched rollout (new pod never Ready) leaves
- # the control plane down until a manual `kubectl rollout undo` — the single-
- # writer invariant forbids any overlap-based auto-recovery. This deadline is
- # DETECTION ONLY: it flips Progressing=False (ProgressDeadlineExceeded) and
- # fails `kubectl rollout status` for callers without their own --timeout; it
- # does not roll back.
+ # /ready 503. Kubernetes will not self-heal that; the evenfire-infra deploy
+ # workflows own automatic `rollout undo --to-revision` for this Deployment
+ # only (evenfire#391). The deploy job still fails after a successful undo.
+ # This deadline is DETECTION ONLY: it flips Progressing=False
+ # (ProgressDeadlineExceeded) and fails `kubectl rollout status` for callers
+ # without their own --timeout; it does not roll back.
#
# Value (origin/dev, 2026-08-05): HCC's /ready stays 503 until its initial fleet
# reconciliation finishes, so time-to-Ready scales with the McpServer/Context
diff --git a/docs/architecture/hcc-readiness-flows.md b/docs/architecture/hcc-readiness-flows.md
index b20373519..5d158b3e4 100644
--- a/docs/architecture/hcc-readiness-flows.md
+++ b/docs/architecture/hcc-readiness-flows.md
@@ -84,7 +84,13 @@ The coordinator is the **single owner** of external-egress reconcile cadence
The HCC Deployment ships `strategy: Recreate`, `replicas: 1`, and
`progressDeadlineSeconds: 1200`: the stateless single-writer safety model has no
leader election, so two concurrently-running controllers must be impossible during a
-rollout. The trade-off (full HCC control-plane downtime per rollout; a rollout whose
-new pod never becomes Ready needs a manual `kubectl rollout undo`) is declared in the
-manifest and asserted by the bootstrap gate. `validate-postgres-single-writer-strategy.sh`
-enforces the invariant on the rendered manifest in CI.
+rollout. The trade-off (full HCC control-plane downtime per rollout; a botched Recreate
+leaves `/ready` 503) is declared in the manifest and asserted by the bootstrap
+gate. `progressDeadlineSeconds: 1200` is detection-only. Recovery is owned by
+the evenfire-infra deploy workflows: they capture the last-known-good revision,
+wait 90s for HCC, and run `rollout undo --to-revision` for
+`control-plane/host-context-controller` only. The cluster may return to
+last-known-good; the deploy job still fails (evenfire#391). Do not treat e2e
+`cleanup()` image restore as that pipeline recovery.
+`validate-postgres-single-writer-strategy.sh` enforces the Recreate +
+replicas:1 invariant on the rendered manifest in CI.
diff --git a/host-context-controller/package-lock.json b/host-context-controller/package-lock.json
index bbe7bf588..84cf93bac 100644
--- a/host-context-controller/package-lock.json
+++ b/host-context-controller/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "clerum-host-context-controller",
- "version": "0.1.228",
+ "version": "0.1.229",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "clerum-host-context-controller",
- "version": "0.1.228",
+ "version": "0.1.229",
"license": "SEE LICENSE IN LICENSE",
"dependencies": {
"@clerum/image-policy": "file:../packages/image-policy",
diff --git a/host-context-controller/package.json b/host-context-controller/package.json
index 96e9a797d..efe9ae9da 100644
--- a/host-context-controller/package.json
+++ b/host-context-controller/package.json
@@ -1,6 +1,6 @@
{
"name": "clerum-host-context-controller",
- "version": "0.1.228",
+ "version": "0.1.229",
"description": "Host Context Controller - provides McpServer CRDs to mcp-host via REST API",
"main": "dist/main.js",
"engines": {
diff --git a/host-context-controller/src/deploymentStrategy.test.ts b/host-context-controller/src/deploymentStrategy.test.ts
index 7a9258139..0779328a6 100644
--- a/host-context-controller/src/deploymentStrategy.test.ts
+++ b/host-context-controller/src/deploymentStrategy.test.ts
@@ -52,7 +52,9 @@ describe('host-context-controller Deployment strategy', () => {
it('bounds rollout-stall detection above the worst legitimate startup', () => {
// Detection only, not recovery: Recreate + replicas:1 means a botched
- // rollout is healed operationally (kubectl rollout undo), never by K8s.
+ // rollout is healed by the evenfire-infra deploy pipeline
+ // (rollout undo --to-revision; evenfire#391), never by Kubernetes itself.
+ // The deadline stays 1200; this test must not lower it.
// This deadline just flips Progressing=False / fails `kubectl rollout
// status` deterministically for observers without their own --timeout. It
// must exceed the worst legitimate startup, which — until #205 fully
diff --git a/scripts/e2e/e2e-hcc-rollout-readiness.sh b/scripts/e2e/e2e-hcc-rollout-readiness.sh
index acdd55eec..b6f5e699a 100755
--- a/scripts/e2e/e2e-hcc-rollout-readiness.sh
+++ b/scripts/e2e/e2e-hcc-rollout-readiness.sh
@@ -18,6 +18,12 @@
# keeps zero old replicas to fall back on.
#
# Modes:
+# EXPECT_RECOVERY=1 — exclusive with EXPECT_STUCK. Botch the image, prove the
+# D1b 503 outage, then run last-good revision restore as the TEST ACTION
+# (not cleanup). Recovery must recertify /ready 200, restore the original
+# image and last-good revision, and replace the botched pod. An undo no-op
+# is FAIL. After a proven undo, IMAGE_BROKEN=0 so cleanup does not set the
+# image again. This is the evenfire#391 evidence path.
# EXPECT_STUCK=0 (default) — healthy rollout via `kubectl rollout restart`.
# rollout restart mutates the pod template (restartedAt annotation), which
# drives the SAME Deployment machinery — terminate old, then create new —
@@ -67,7 +73,7 @@ command -v jq >/dev/null 2>&1 || {
}
[ "${E2E_HCC_ROLLOUT_FAULT_INJECTION:-0}" = 1 ] || {
echo "Set E2E_HCC_ROLLOUT_FAULT_INJECTION=1 to acknowledge that this gate rolls" >&2
- echo "(and, with EXPECT_STUCK=1, deliberately breaks) the HCC deployment." >&2
+ echo "(and, with EXPECT_STUCK=1 or EXPECT_RECOVERY=1, deliberately breaks) the HCC deployment." >&2
exit 1
}
case "${EXPECT_STUCK:-0}" in
@@ -78,6 +84,18 @@ case "${EXPECT_STUCK:-0}" in
;;
esac
EXPECT_STUCK="${EXPECT_STUCK:-0}"
+EXPECT_RECOVERY="${EXPECT_RECOVERY:-0}"
+case "${EXPECT_RECOVERY}" in
+ 0 | 1) ;;
+ *)
+ echo "EXPECT_RECOVERY must be 0 or 1." >&2
+ exit 1
+ ;;
+esac
+if [ "$EXPECT_STUCK" = 1 ] && [ "$EXPECT_RECOVERY" = 1 ]; then
+ echo "EXPECT_STUCK=1 and EXPECT_RECOVERY=1 are exclusive." >&2
+ exit 1
+fi
kctl get nodes -o json | jq -e --arg c "$E2E_KUBECONTEXT" \
'any(.items[]; .metadata.labels["minikube.k8s.io/name"] == $c)' >/dev/null ||
{
@@ -229,7 +247,7 @@ write_evidence_artifact() {
# sibling churn gate: an empty grep under `set -euo pipefail` must never
# abort the gate before its verdict).
{
- echo "=== mode: EXPECT_STUCK=${EXPECT_STUCK} fleet=${WITH_SYNTHETIC_FLEET} (${FLEET_CONTEXTS}/${FLEET_MCPSERVERS}/${FLEET_HOSTS}) ==="
+ echo "=== mode: EXPECT_STUCK=${EXPECT_STUCK} EXPECT_RECOVERY=${EXPECT_RECOVERY} fleet=${WITH_SYNTHETIC_FLEET} (${FLEET_CONTEXTS}/${FLEET_MCPSERVERS}/${FLEET_HOSTS}) ==="
echo "=== HCC deploy ==="
kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o wide || true
echo "=== deployment conditions (progressDeadlineSeconds is DETECTION only) ==="
@@ -322,7 +340,11 @@ OLD_POD_NAME="$(running_hcc_pod)"
OLD_POD_UID="$(kctl get pod "$OLD_POD_NAME" -n "$HCC_NS" -o jsonpath='{.metadata.uid}')"
[ -n "$OLD_POD_UID" ] || die "could not pin the pre-rollout pod uid"
-if [ "$EXPECT_STUCK" = 0 ]; then
+
+LAST_GOOD_REVISION="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')"
+[ -n "$LAST_GOOD_REVISION" ] || die "could not capture the last-known-good HCC revision"
+
+if [ "$EXPECT_STUCK" = 0 ] && [ "$EXPECT_RECOVERY" = 0 ]; then
# ── FASE D (healthy): Recreate rollout, sampled at 1Hz end to end ──
log "Triggering a Recreate rollout (kubectl rollout restart) and sampling /ready at 1Hz"
kctl rollout restart deployment/"$HCC_DEPLOY" -n "$HCC_NS" >/dev/null
@@ -364,7 +386,7 @@ if [ "$EXPECT_STUCK" = 0 ]; then
[ "$hold_total" -gt 0 ] && [ "$hold_503" -eq 0 ] &&
ok "post-rollout hold: ${hold_total} samples over ${STABILITY_WINDOW_SEC}s, zero 503 — the replacement is stable" ||
fail "post-rollout hold saw ${hold_503} 503(s) across ${hold_total} sample(s) — the replacement did not stabilize"
-else
+elif [ "$EXPECT_STUCK" = 1 ]; then
# ── FASE D (botched, D1b): roll to an unpullable image and watch the outage ──
BOTCHED_IMAGE="$(botched_image_ref)"
log "Botched rollout (D1b): rolling to unpullable image ${BOTCHED_IMAGE}; observing /ready for ${STUCK_OBSERVE_SEC}s"
@@ -406,8 +428,80 @@ else
[ "$stuck_maxstreak" -gt "$ROLLOUT_DOWNTIME_BUDGET_SEC" ] &&
ok "SUSTAINED outage measured: ${stuck_maxstreak}s continuous 503 (> ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget) and still open at observation end — recovery requires the manual image rollback cleanup performs (D1b reproduced)" ||
fail "outage lasted only ${stuck_maxstreak}s and did not exceed the ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget — the botched rollout did not produce the D1b total outage"
+else
+ # ── FASE D (recovery, evenfire#391): botch, prove the outage, then undo ──
+ BOTCHED_IMAGE="$(botched_image_ref)"
+ PRE_BOTCH_UID="$OLD_POD_UID"
+ log "Botched rollout (EXPECT_RECOVERY): observing /ready for ${STUCK_OBSERVE_SEC}s"
+ kctl set image deployment/"$HCC_DEPLOY" -n "$HCC_NS" host-context-controller="$BOTCHED_IMAGE" >/dev/null
+ ROLLOUT_TRIGGERED=1
+ IMAGE_BROKEN=1
+ sample_ready_series "$STUCK_OBSERVE_SEC" stuck 0
+ read -r stuck_total stuck_200 stuck_503 stuck_maxstreak stuck_transitions <<<"$(series_metrics stuck)"
+ deployed_image_now="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.spec.template.spec.containers[?(@.name=="host-context-controller")].image}' 2>/dev/null || true)"
+ old_uid_live="$(kctl get pods -n "$HCC_NS" -l "app=${HCC_DEPLOY}" -o jsonpath='{range .items[*]}{.metadata.uid}{"\n"}{end}' 2>/dev/null | grep -cxF "$OLD_POD_UID" || true)"
+ stuck_pod_rows="$(kctl get pods -n "$HCC_NS" -l "app=${HCC_DEPLOY}" -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.uid}{"\t"}{.status.phase}{"\t"}{.status.containerStatuses[*].state.waiting.reason}{"\n"}{end}' 2>/dev/null || true)"
+ replacement_row="$(awk -F '\t' -v old="$OLD_POD_UID" '$2 != old && $2 != "" { print; exit }' <<<"$stuck_pod_rows")"
+ BOTCHED_POD_UID="$(awk -F '\t' '{ print $2; exit }' <<<"$replacement_row")"
+
+ [ "$stuck_total" -gt 0 ] && ok "sampled /ready ${stuck_total}x across the botched rollout" ||
+ fail "ZERO /ready samples across the botched rollout — the sampler never ran, and a zero-sample run must never pass"
+ [ "$deployed_image_now" = "$BOTCHED_IMAGE" ] &&
+ ok "the botched image is what the Deployment is actually rolling (${BOTCHED_IMAGE})" ||
+ fail "deployment image is '${deployed_image_now:-unset}', not the injected ${BOTCHED_IMAGE}"
+ [ "$old_uid_live" = 0 ] &&
+ ok "Recreate terminated the old HEALTHY pod before any viable replacement existed" ||
+ fail "old pod ${OLD_POD_NAME} is still alive under a Recreate rollout"
+ if [ -n "$replacement_row" ] && grep -Eq 'ErrImagePull|ImagePullBackOff' <<<"$replacement_row"; then
+ ok "replacement pod is pinned by the injected image failure: ${replacement_row}"
+ else
+ fail "no replacement pod pinned in ErrImagePull|ImagePullBackOff (rows: ${stuck_pod_rows:-none})"
+ fi
+ [ "$stuck_503" -ge 1 ] &&
+ ok "the botched rollout bit the readiness path: ${stuck_503} sample(s) at 503" ||
+ fail "zero 503 samples under a botched Recreate rollout — the outage claim is VACUOUS"
+ [ "$stuck_transitions" -eq 0 ] &&
+ ok "no 503->200 transition closed the outage before undo — Kubernetes did NOT self-heal" ||
+ fail "the outage closed on its own (${stuck_transitions} 503->200 transition(s)) before undo"
+
+ live_revision_before_undo="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')"
+ [ "$live_revision_before_undo" != "$LAST_GOOD_REVISION" ] ||
+ fail "live revision ${live_revision_before_undo} still equals last-good ${LAST_GOOD_REVISION} — undo would be a no-op"
+
+ log "TEST ACTION (evenfire#391): rollout undo --to-revision=${LAST_GOOD_REVISION}"
+ kctl rollout undo deployment/"$HCC_DEPLOY" -n "$HCC_NS" --to-revision="$LAST_GOOD_REVISION" >/dev/null ||
+ fail "rollout undo --to-revision=${LAST_GOOD_REVISION} failed"
+ kctl rollout status deployment/"$HCC_DEPLOY" -n "$HCC_NS" --timeout=240s >/dev/null ||
+ fail "last-known-good revision ${LAST_GOOD_REVISION} did not become Ready"
+ wait_until 90 "HCC /ready after undo" hcc_ready_now ||
+ fail "HCC /ready did not return 200 after last-good restore"
+ IMAGE_BROKEN=0
+
+ recovered_image="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.spec.template.spec.containers[?(@.name=="host-context-controller")].image}')"
+ recovered_revision="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')"
+ NEW_POD_NAME="$(running_hcc_pod)"
+ NEW_POD_UID="$(kctl get pod "$NEW_POD_NAME" -n "$HCC_NS" -o jsonpath='{.metadata.uid}')"
+ sample_ready_series "$STABILITY_WINDOW_SEC" post-hold 0
+ read -r hold_total _ hold_503 _ _ <<<"$(series_metrics post-hold)"
+ write_evidence_artifact
+
+ [ "$recovered_image" = "$HCC_IMAGE" ] &&
+ ok "undo restored the original image ${HCC_IMAGE}" ||
+ fail "image after undo is '${recovered_image:-unset}', expected ${HCC_IMAGE}"
+ [ "$recovered_revision" != "$live_revision_before_undo" ] &&
+ ok "undo created a new revision ${recovered_revision} from last-good ${LAST_GOOD_REVISION} (was ${live_revision_before_undo})" ||
+ fail "revision after undo is still ${recovered_revision:-unset} — undo was a no-op"
+ if [ -n "$NEW_POD_UID" ] && [ "$NEW_POD_UID" != "$PRE_BOTCH_UID" ] && [ "$NEW_POD_UID" != "$BOTCHED_POD_UID" ]; then
+ ok "recovered pod uid ${NEW_POD_UID} is neither the pre-botch nor the botched pod"
+ else
+ fail "recovered pod uid '${NEW_POD_UID:-none}' matches pre-botch ${PRE_BOTCH_UID} or botched ${BOTCHED_POD_UID:-none} — undo was a no-op"
+ fi
+ [ "$hold_total" -gt 0 ] && [ "$hold_503" -eq 0 ] &&
+ ok "post-undo hold: ${hold_total} samples over ${STABILITY_WINDOW_SEC}s, zero 503" ||
+ fail "post-undo hold saw ${hold_503} 503(s) across ${hold_total} sample(s)"
fi
+
log "Evidence artifact: ${LOG_ARTIFACT}"
# cleanup() (EXIT trap) restores the image if broken, waits for re-certification,
# deletes the fleet, finalizes the single-writer lock, and runs print_results.
diff --git a/scripts/tests/test-hcc-rollout-readiness-gate.sh b/scripts/tests/test-hcc-rollout-readiness-gate.sh
index 7ed24988d..b7f7ded3e 100755
--- a/scripts/tests/test-hcc-rollout-readiness-gate.sh
+++ b/scripts/tests/test-hcc-rollout-readiness-gate.sh
@@ -121,4 +121,19 @@ grep -Fq 'create_synthetic_fleet' "$GATE" &&
pass "measured window includes the synthetic clerum-dev-scale fleet" ||
fail "gate measures a bare cluster and calls it the production window"
+
+# 12. evenfire#391 recovery mode: exclusive with EXPECT_STUCK, undo is the test
+# action (not cleanup), last-good revision is captured, IMAGE_BROKEN is
+# cleared after a proven undo, and a no-op undo fails.
+grep -Fq 'EXPECT_RECOVERY' "$GATE" &&
+ grep -Fq 'EXPECT_STUCK=1 and EXPECT_RECOVERY=1 are exclusive' "$GATE" &&
+ grep -Fq 'TEST ACTION' "$GATE" &&
+ grep -Fq -- '--to-revision' "$GATE" &&
+ grep -Fq 'LAST_GOOD_REVISION' "$GATE" &&
+ grep -Fq 'IMAGE_BROKEN=0' "$GATE" &&
+ grep -Fq 'undo was a no-op' "$GATE" &&
+ ! grep -q 'EXPECT_RECOVERY=1' <<<"$(awk '/^cleanup\(\)/,/^}/' "$GATE")" &&
+ pass "EXPECT_RECOVERY undoes to last-good outside cleanup and fails a no-op" ||
+ fail "EXPECT_RECOVERY is missing, shares cleanup, or can pass on a no-op undo"
+
exit "$FAIL"