Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
docs(#391): pipeline-owned HCC Recreate recovery + e2e undo path
Alfredo Lopez58dc455ec1e3
diff --git a/.github/workflows/ci-public.yml b/.github/workflows/ci-public.ymlindex 8da700bbe..35b2d3d3a 100644--- a/.github/workflows/ci-public.yml+++ b/.github/workflows/ci-public.yml@@ -397,6 +397,9 @@ jobs:- name: Test HCC watch-churn readiness gate contractrun: bash scripts/tests/test-hcc-watch-churn-readiness-gate.sh+ - name: Test HCC rollout readiness gate contract+ run: bash scripts/tests/test-hcc-rollout-readiness-gate.sh+- name: Test HCC API gateway readiness separationrun: bash scripts/tests/test-hcc-api-gateway-readiness.shdiff --git a/deploy/base/control-plane/host-context-controller.yaml b/deploy/base/control-plane/host-context-controller.yamlindex 24a4c21dd..8b48cb482 100644--- a/deploy/base/control-plane/host-context-controller.yaml+++ b/deploy/base/control-plane/host-context-controller.yaml@@ -24,11 +24,12 @@ spec:strategy:type: Recreate# Recreate + replicas:1 means a botched rollout (new pod never Ready) leaves- # the control plane down until a manual `kubectl rollout undo` — the single-- # writer invariant forbids any overlap-based auto-recovery. This deadline is- # DETECTION ONLY: it flips Progressing=False (ProgressDeadlineExceeded) and- # fails `kubectl rollout status` for callers without their own --timeout; it- # does not roll back.+ # /ready 503. Kubernetes will not self-heal that; the evenfire-infra deploy+ # workflows own automatic `rollout undo --to-revision` for this Deployment+ # only (evenfire#391). The deploy job still fails after a successful undo.+ # This deadline is DETECTION ONLY: it flips Progressing=False+ # (ProgressDeadlineExceeded) and fails `kubectl rollout status` for callers+ # without their own --timeout; it does not roll back.## Value (origin/dev, 2026-08-05): HCC's /ready stays 503 until its initial fleet# reconciliation finishes, so time-to-Ready scales with the McpServer/Contextdiff --git a/docs/architecture/hcc-readiness-flows.md b/docs/architecture/hcc-readiness-flows.mdindex b20373519..5d158b3e4 100644--- a/docs/architecture/hcc-readiness-flows.md+++ b/docs/architecture/hcc-readiness-flows.md@@ -84,7 +84,13 @@ The coordinator is the **single owner** of external-egress reconcile cadenceThe HCC Deployment ships `strategy: Recreate`, `replicas: 1`, and`progressDeadlineSeconds: 1200`: the stateless single-writer safety model has noleader election, so two concurrently-running controllers must be impossible during a-rollout. The trade-off (full HCC control-plane downtime per rollout; a rollout whose-new pod never becomes Ready needs a manual `kubectl rollout undo`) is declared in the-manifest and asserted by the bootstrap gate. `validate-postgres-single-writer-strategy.sh`-enforces the invariant on the rendered manifest in CI.+rollout. The trade-off (full HCC control-plane downtime per rollout; a botched Recreate+leaves `/ready` 503) is declared in the manifest and asserted by the bootstrap+gate. `progressDeadlineSeconds: 1200` is detection-only. Recovery is owned by+the evenfire-infra deploy workflows: they capture the last-known-good revision,+wait 90s for HCC, and run `rollout undo --to-revision` for+`control-plane/host-context-controller` only. The cluster may return to+last-known-good; the deploy job still fails (evenfire#391). Do not treat e2e+`cleanup()` image restore as that pipeline recovery.+`validate-postgres-single-writer-strategy.sh` enforces the Recreate ++replicas:1 invariant on the rendered manifest in CI.diff --git a/host-context-controller/package-lock.json b/host-context-controller/package-lock.jsonindex bbe7bf588..84cf93bac 100644--- a/host-context-controller/package-lock.json+++ b/host-context-controller/package-lock.json@@ -1,12 +1,12 @@{"name": "clerum-host-context-controller",- "version": "0.1.228",+ "version": "0.1.229","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "clerum-host-context-controller",- "version": "0.1.228",+ "version": "0.1.229","license": "SEE LICENSE IN LICENSE","dependencies": {"@clerum/image-policy": "file:../packages/image-policy",diff --git a/host-context-controller/package.json b/host-context-controller/package.jsonindex 96e9a797d..efe9ae9da 100644--- a/host-context-controller/package.json+++ b/host-context-controller/package.json@@ -1,6 +1,6 @@{"name": "clerum-host-context-controller",- "version": "0.1.228",+ "version": "0.1.229","description": "Host Context Controller - provides McpServer CRDs to mcp-host via REST API","main": "dist/main.js","engines": {diff --git a/host-context-controller/src/deploymentStrategy.test.ts b/host-context-controller/src/deploymentStrategy.test.tsindex 7a9258139..0779328a6 100644--- a/host-context-controller/src/deploymentStrategy.test.ts+++ b/host-context-controller/src/deploymentStrategy.test.ts@@ -52,7 +52,9 @@ describe('host-context-controller Deployment strategy', () => {it('bounds rollout-stall detection above the worst legitimate startup', () => {// Detection only, not recovery: Recreate + replicas:1 means a botched- // rollout is healed operationally (kubectl rollout undo), never by K8s.+ // rollout is healed by the evenfire-infra deploy pipeline+ // (rollout undo --to-revision; evenfire#391), never by Kubernetes itself.+ // The deadline stays 1200; this test must not lower it.// This deadline just flips Progressing=False / fails `kubectl rollout// status` deterministically for observers without their own --timeout. It// must exceed the worst legitimate startup, which — until #205 fullydiff --git a/scripts/e2e/e2e-hcc-rollout-readiness.sh b/scripts/e2e/e2e-hcc-rollout-readiness.shindex acdd55eec..b6f5e699a 100755--- a/scripts/e2e/e2e-hcc-rollout-readiness.sh+++ b/scripts/e2e/e2e-hcc-rollout-readiness.sh@@ -18,6 +18,12 @@# keeps zero old replicas to fall back on.## Modes:+# EXPECT_RECOVERY=1 — exclusive with EXPECT_STUCK. Botch the image, prove the+# D1b 503 outage, then run last-good revision restore as the TEST ACTION+# (not cleanup). Recovery must recertify /ready 200, restore the original+# image and last-good revision, and replace the botched pod. An undo no-op+# is FAIL. After a proven undo, IMAGE_BROKEN=0 so cleanup does not set the+# image again. This is the evenfire#391 evidence path.# EXPECT_STUCK=0 (default) — healthy rollout via `kubectl rollout restart`.# rollout restart mutates the pod template (restartedAt annotation), which# drives the SAME Deployment machinery — terminate old, then create new —@@ -67,7 +73,7 @@ command -v jq >/dev/null 2>&1 || {}[ "${E2E_HCC_ROLLOUT_FAULT_INJECTION:-0}" = 1 ] || {echo "Set E2E_HCC_ROLLOUT_FAULT_INJECTION=1 to acknowledge that this gate rolls" >&2- echo "(and, with EXPECT_STUCK=1, deliberately breaks) the HCC deployment." >&2+ echo "(and, with EXPECT_STUCK=1 or EXPECT_RECOVERY=1, deliberately breaks) the HCC deployment." >&2exit 1}case "${EXPECT_STUCK:-0}" in@@ -78,6 +84,18 @@ case "${EXPECT_STUCK:-0}" in;;esacEXPECT_STUCK="${EXPECT_STUCK:-0}"+EXPECT_RECOVERY="${EXPECT_RECOVERY:-0}"+case "${EXPECT_RECOVERY}" in+ 0 | 1) ;;+ *)+ echo "EXPECT_RECOVERY must be 0 or 1." >&2+ exit 1+ ;;+esac+if [ "$EXPECT_STUCK" = 1 ] && [ "$EXPECT_RECOVERY" = 1 ]; then+ echo "EXPECT_STUCK=1 and EXPECT_RECOVERY=1 are exclusive." >&2+ exit 1+fikctl get nodes -o json | jq -e --arg c "$E2E_KUBECONTEXT" \'any(.items[]; .metadata.labels["minikube.k8s.io/name"] == $c)' >/dev/null ||{@@ -229,7 +247,7 @@ write_evidence_artifact() {# sibling churn gate: an empty grep under `set -euo pipefail` must never# abort the gate before its verdict).{- echo "=== mode: EXPECT_STUCK=${EXPECT_STUCK} fleet=${WITH_SYNTHETIC_FLEET} (${FLEET_CONTEXTS}/${FLEET_MCPSERVERS}/${FLEET_HOSTS}) ==="+ echo "=== mode: EXPECT_STUCK=${EXPECT_STUCK} EXPECT_RECOVERY=${EXPECT_RECOVERY} fleet=${WITH_SYNTHETIC_FLEET} (${FLEET_CONTEXTS}/${FLEET_MCPSERVERS}/${FLEET_HOSTS}) ==="echo "=== HCC deploy ==="kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o wide || trueecho "=== deployment conditions (progressDeadlineSeconds is DETECTION only) ==="@@ -322,7 +340,11 @@ OLD_POD_NAME="$(running_hcc_pod)"OLD_POD_UID="$(kctl get pod "$OLD_POD_NAME" -n "$HCC_NS" -o jsonpath='{.metadata.uid}')"[ -n "$OLD_POD_UID" ] || die "could not pin the pre-rollout pod uid"-if [ "$EXPECT_STUCK" = 0 ]; then++LAST_GOOD_REVISION="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')"+[ -n "$LAST_GOOD_REVISION" ] || die "could not capture the last-known-good HCC revision"++if [ "$EXPECT_STUCK" = 0 ] && [ "$EXPECT_RECOVERY" = 0 ]; then# ── FASE D (healthy): Recreate rollout, sampled at 1Hz end to end ──log "Triggering a Recreate rollout (kubectl rollout restart) and sampling /ready at 1Hz"kctl rollout restart deployment/"$HCC_DEPLOY" -n "$HCC_NS" >/dev/null@@ -364,7 +386,7 @@ if [ "$EXPECT_STUCK" = 0 ]; then[ "$hold_total" -gt 0 ] && [ "$hold_503" -eq 0 ] &&ok "post-rollout hold: ${hold_total} samples over ${STABILITY_WINDOW_SEC}s, zero 503 — the replacement is stable" ||fail "post-rollout hold saw ${hold_503} 503(s) across ${hold_total} sample(s) — the replacement did not stabilize"-else+elif [ "$EXPECT_STUCK" = 1 ]; then# ── FASE D (botched, D1b): roll to an unpullable image and watch the outage ──BOTCHED_IMAGE="$(botched_image_ref)"log "Botched rollout (D1b): rolling to unpullable image ${BOTCHED_IMAGE}; observing /ready for ${STUCK_OBSERVE_SEC}s"@@ -406,8 +428,80 @@ else[ "$stuck_maxstreak" -gt "$ROLLOUT_DOWNTIME_BUDGET_SEC" ] &&ok "SUSTAINED outage measured: ${stuck_maxstreak}s continuous 503 (> ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget) and still open at observation end — recovery requires the manual image rollback cleanup performs (D1b reproduced)" ||fail "outage lasted only ${stuck_maxstreak}s and did not exceed the ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget — the botched rollout did not produce the D1b total outage"+else+ # ── FASE D (recovery, evenfire#391): botch, prove the outage, then undo ──+ BOTCHED_IMAGE="$(botched_image_ref)"+ PRE_BOTCH_UID="$OLD_POD_UID"+ log "Botched rollout (EXPECT_RECOVERY): observing /ready for ${STUCK_OBSERVE_SEC}s"+ kctl set image deployment/"$HCC_DEPLOY" -n "$HCC_NS" host-context-controller="$BOTCHED_IMAGE" >/dev/null+ ROLLOUT_TRIGGERED=1+ IMAGE_BROKEN=1+ sample_ready_series "$STUCK_OBSERVE_SEC" stuck 0+ read -r stuck_total stuck_200 stuck_503 stuck_maxstreak stuck_transitions <<<"$(series_metrics stuck)"+ deployed_image_now="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.spec.template.spec.containers[?(@.name=="host-context-controller")].image}' 2>/dev/null || true)"+ old_uid_live="$(kctl get pods -n "$HCC_NS" -l "app=${HCC_DEPLOY}" -o jsonpath='{range .items[*]}{.metadata.uid}{"\n"}{end}' 2>/dev/null | grep -cxF "$OLD_POD_UID" || true)"+ stuck_pod_rows="$(kctl get pods -n "$HCC_NS" -l "app=${HCC_DEPLOY}" -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.uid}{"\t"}{.status.phase}{"\t"}{.status.containerStatuses[*].state.waiting.reason}{"\n"}{end}' 2>/dev/null || true)"+ replacement_row="$(awk -F '\t' -v old="$OLD_POD_UID" '$2 != old && $2 != "" { print; exit }' <<<"$stuck_pod_rows")"+ BOTCHED_POD_UID="$(awk -F '\t' '{ print $2; exit }' <<<"$replacement_row")"++ [ "$stuck_total" -gt 0 ] && ok "sampled /ready ${stuck_total}x across the botched rollout" ||+ fail "ZERO /ready samples across the botched rollout — the sampler never ran, and a zero-sample run must never pass"+ [ "$deployed_image_now" = "$BOTCHED_IMAGE" ] &&+ ok "the botched image is what the Deployment is actually rolling (${BOTCHED_IMAGE})" ||+ fail "deployment image is '${deployed_image_now:-unset}', not the injected ${BOTCHED_IMAGE}"+ [ "$old_uid_live" = 0 ] &&+ ok "Recreate terminated the old HEALTHY pod before any viable replacement existed" ||+ fail "old pod ${OLD_POD_NAME} is still alive under a Recreate rollout"+ if [ -n "$replacement_row" ] && grep -Eq 'ErrImagePull|ImagePullBackOff' <<<"$replacement_row"; then+ ok "replacement pod is pinned by the injected image failure: ${replacement_row}"+ else+ fail "no replacement pod pinned in ErrImagePull|ImagePullBackOff (rows: ${stuck_pod_rows:-none})"+ fi+ [ "$stuck_503" -ge 1 ] &&+ ok "the botched rollout bit the readiness path: ${stuck_503} sample(s) at 503" ||+ fail "zero 503 samples under a botched Recreate rollout — the outage claim is VACUOUS"+ [ "$stuck_transitions" -eq 0 ] &&+ ok "no 503->200 transition closed the outage before undo — Kubernetes did NOT self-heal" ||+ fail "the outage closed on its own (${stuck_transitions} 503->200 transition(s)) before undo"++ live_revision_before_undo="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')"+ [ "$live_revision_before_undo" != "$LAST_GOOD_REVISION" ] ||+ fail "live revision ${live_revision_before_undo} still equals last-good ${LAST_GOOD_REVISION} — undo would be a no-op"++ log "TEST ACTION (evenfire#391): rollout undo --to-revision=${LAST_GOOD_REVISION}"+ kctl rollout undo deployment/"$HCC_DEPLOY" -n "$HCC_NS" --to-revision="$LAST_GOOD_REVISION" >/dev/null ||+ fail "rollout undo --to-revision=${LAST_GOOD_REVISION} failed"+ kctl rollout status deployment/"$HCC_DEPLOY" -n "$HCC_NS" --timeout=240s >/dev/null ||+ fail "last-known-good revision ${LAST_GOOD_REVISION} did not become Ready"+ wait_until 90 "HCC /ready after undo" hcc_ready_now ||+ fail "HCC /ready did not return 200 after last-good restore"+ IMAGE_BROKEN=0++ recovered_image="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.spec.template.spec.containers[?(@.name=="host-context-controller")].image}')"+ recovered_revision="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')"+ NEW_POD_NAME="$(running_hcc_pod)"+ NEW_POD_UID="$(kctl get pod "$NEW_POD_NAME" -n "$HCC_NS" -o jsonpath='{.metadata.uid}')"+ sample_ready_series "$STABILITY_WINDOW_SEC" post-hold 0+ read -r hold_total _ hold_503 _ _ <<<"$(series_metrics post-hold)"+ write_evidence_artifact++ [ "$recovered_image" = "$HCC_IMAGE" ] &&+ ok "undo restored the original image ${HCC_IMAGE}" ||+ fail "image after undo is '${recovered_image:-unset}', expected ${HCC_IMAGE}"+ [ "$recovered_revision" != "$live_revision_before_undo" ] &&+ ok "undo created a new revision ${recovered_revision} from last-good ${LAST_GOOD_REVISION} (was ${live_revision_before_undo})" ||+ fail "revision after undo is still ${recovered_revision:-unset} — undo was a no-op"+ if [ -n "$NEW_POD_UID" ] && [ "$NEW_POD_UID" != "$PRE_BOTCH_UID" ] && [ "$NEW_POD_UID" != "$BOTCHED_POD_UID" ]; then+ ok "recovered pod uid ${NEW_POD_UID} is neither the pre-botch nor the botched pod"+ else+ fail "recovered pod uid '${NEW_POD_UID:-none}' matches pre-botch ${PRE_BOTCH_UID} or botched ${BOTCHED_POD_UID:-none} — undo was a no-op"+ fi+ [ "$hold_total" -gt 0 ] && [ "$hold_503" -eq 0 ] &&+ ok "post-undo hold: ${hold_total} samples over ${STABILITY_WINDOW_SEC}s, zero 503" ||+ fail "post-undo hold saw ${hold_503} 503(s) across ${hold_total} sample(s)"fi+log "Evidence artifact: ${LOG_ARTIFACT}"# cleanup() (EXIT trap) restores the image if broken, waits for re-certification,# deletes the fleet, finalizes the single-writer lock, and runs print_results.diff --git a/scripts/tests/test-hcc-rollout-readiness-gate.sh b/scripts/tests/test-hcc-rollout-readiness-gate.shindex 7ed24988d..b7f7ded3e 100755--- a/scripts/tests/test-hcc-rollout-readiness-gate.sh+++ b/scripts/tests/test-hcc-rollout-readiness-gate.sh@@ -121,4 +121,19 @@ grep -Fq 'create_synthetic_fleet' "$GATE" &&pass "measured window includes the synthetic clerum-dev-scale fleet" ||fail "gate measures a bare cluster and calls it the production window"++# 12. evenfire#391 recovery mode: exclusive with EXPECT_STUCK, undo is the test+# action (not cleanup), last-good revision is captured, IMAGE_BROKEN is+# cleared after a proven undo, and a no-op undo fails.+grep -Fq 'EXPECT_RECOVERY' "$GATE" &&+ grep -Fq 'EXPECT_STUCK=1 and EXPECT_RECOVERY=1 are exclusive' "$GATE" &&+ grep -Fq 'TEST ACTION' "$GATE" &&+ grep -Fq -- '--to-revision' "$GATE" &&+ grep -Fq 'LAST_GOOD_REVISION' "$GATE" &&+ grep -Fq 'IMAGE_BROKEN=0' "$GATE" &&+ grep -Fq 'undo was a no-op' "$GATE" &&+ ! grep -q 'EXPECT_RECOVERY=1' <<<"$(awk '/^cleanup\(\)/,/^}/' "$GATE")" &&+ pass "EXPECT_RECOVERY undoes to last-good outside cleanup and fails a no-op" ||+ fail "EXPECT_RECOVERY is missing, shares cleanup, or can pass on a no-op undo"+exit "$FAIL"