Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
fix(e2e): scope the watch-churn gate to the POSITIVE case; the RED is hermetic
Alfredo Lopezd010109eda08
diff --git a/scripts/e2e/e2e-hcc-watch-churn-readiness.sh b/scripts/e2e/e2e-hcc-watch-churn-readiness.shindex 50995d8fe..578d71b18 100755--- a/scripts/e2e/e2e-hcc-watch-churn-readiness.sh+++ b/scripts/e2e/e2e-hcc-watch-churn-readiness.sh@@ -1,28 +1,31 @@#!/usr/bin/env bash# HCC readiness under SUSTAINED apiserver watch churn (Premature close regime).#-# Reproduces the PR #205 GKE livelock in minikube — a regime minikube does NOT-# exhibit by default (single stable apiserver, no balancer idle-timeout). A-# clerum-dev-scale synthetic fleet widens each NetworkPolicy revocation pass; a-# self-flapping API proxy resets the HCC watches mid-pass, bumping the watch-# generation so a generation-pinned isReadinessInventoryAuthoritative() could-# never certify. The content-identity fix must certify through the churn.+# SCOPE — this e2e proves ONLY the POSITIVE case: the content-identity fix+# (PR #382) certifies /ready THROUGH sustained watch churn in a REAL cluster,+# plus anti-vacuity (the churn actually cut every watch, repeatedly). It does+# NOT reproduce the pre-fix livelock as a NEGATIVE — that is STRUCTURALLY+# impossible in minikube. The pre-fix's authoritative NetworkPolicy revocation+# pass (where the watch generation matters) runs at connection age < the+# flapper's min-age (~0.3s on a local apiserver), so it is never cut; and any+# min-age small enough to cut it also kills the re-LIST/re-WATCH that BOTH the+# fixed and pre-fix builds need to recover — so both would livelock (a false+# positive), never a clean discrimination. Empirically confirmed: fix and+# pre-fix converge identically under 7s AND 3s churn.#-# The fail-closed contract means a cut watch drops /ready to 503 TRANSIENTLY-# (~recovery-timer + re-LIST) and re-certifies with NO content change; under-# sustained churn the correct regime is a bounded 503/200 duty cycle, NOT an-# unbroken 200. The gate therefore measures streaks, not the absence of 503s:+# The livelock RED lives, hermetically and MUTATION-VERIFIED, in+# host-context-controller/src/k8sClient.test.ts+# "McpServerWatcher readiness under sustained watch churn (GKE Premature-close+# regime)"+# which bumps the generation DURING each authoritative pass — the interleaving+# minikube cannot produce. RED there + GREEN here is the complete proof.#-# EXPECT_LIVELOCK=1 -> proves the reproducer: an UNBROKEN 503 streak for the-# whole budget (>> RECOVERY_BUDGET_SEC), kubelet-# cross-check, plus the divergent-generation pattern-# (guards against a vacuous test).-# EXPECT_LIVELOCK=0 -> proves the fix: /ready reaches 200 under churn, every-# in-churn 503 streak closes within RECOVERY_BUDGET_SEC-# with repeated 503->200 recoveries, and once the churn-# is paused (flag file — no proxy or HCC restart) the-# SAME pod re-certifies in <= POST_STOP_READY_SEC and-# holds 200 for STABILITY_WINDOW_SEC with zero 503s.+# The fail-closed contract means a cut watch drops /ready to 503 TRANSIENTLY+# (~recovery + re-LIST) and re-certifies with NO content change; under sustained+# churn the correct regime is a bounded 503/200 duty cycle, NOT an unbroken 200.+# The gate measures CONVERGENCE — bounded 503 streaks, repeated 503->200+# recoveries, and post-churn re-certification of the SAME non-restarted pod —+# never the absence of 503s. EXPECT_LIVELOCK=1 is refused (see the guard below).set -euo pipefailSCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"# shellcheck source=scripts/e2e/e2e-lib.sh@@ -54,6 +57,15 @@ command -v jq >/dev/null 2>&1 || {echo "Set E2E_HCC_WATCH_FAULT_INJECTION=1 to acknowledge fault injection." >&2exit 1}+# The NEGATIVE (pre-fix livelock) is not reproducible in minikube — see the SCOPE+# note above. Refuse EXPECT_LIVELOCK=1 rather than pretend to reproduce it.+[ "${EXPECT_LIVELOCK:-0}" != 1 ] || {+ echo "EXPECT_LIVELOCK=1 is not supported: the pre-fix livelock cannot be reproduced" >&2+ echo "in minikube (the authoritative pass is structurally uncuttable). The livelock" >&2+ echo "RED lives in host-context-controller/src/k8sClient.test.ts ('McpServerWatcher" >&2+ echo "readiness under sustained watch churn (GKE Premature-close regime)')." >&2+ exit 1+}kctl get nodes -o json | jq -e --arg c "$E2E_KUBECONTEXT" \'any(.items[]; .metadata.labels["minikube.k8s.io/name"] == $c)' >/dev/null ||{@@ -85,12 +97,9 @@ WATCH_CUT_MIN_LINES="${WATCH_CUT_MIN_LINES:-6}"# Minimum distinct 503->200 transitions under churn: one recovery could be a# fluke; repeated recovery refutes the livelock.MIN_CHURN_RECOVERIES="${MIN_CHURN_RECOVERIES:-2}"-# Livelock signature: an unbroken 503 streak at least this long (3x budget).-LIVELOCK_MIN_STREAK_SEC="${LIVELOCK_MIN_STREAK_SEC:-45}"FLEET_CONTEXTS="${FLEET_CONTEXTS:-24}"FLEET_MCPSERVERS="${FLEET_MCPSERVERS:-115}"FLEET_HOSTS="${FLEET_HOSTS:-8}"-EXPECT_LIVELOCK="${EXPECT_LIVELOCK:-0}"RUN_ID="$(date +%s)-$$"PROXY_NAME="$(truncate_rfc1123 "e2e-hcc-churn-proxy-${RUN_ID}")"@@ -387,7 +396,7 @@ write_evidence_artifact() {# asserted a property the fail-closed contract deliberately violates: a cut# watch drops /ready to 503 transiently while it re-syncs, so "no 503 after# first 200" failed the CORRECT regime and measured the wrong thing.-if [ "$EXPECT_LIVELOCK" = 0 ] && [ -n "$first_ready" ]; then+if [ -n "$first_ready" ]; thenlog "Observing /ready at 1Hz for ${CHURN_OBSERVE_SEC}s under continuous churn"sample_ready_series "$CHURN_OBSERVE_SEC" churn 0fi@@ -431,86 +440,59 @@ read -r churn_total churn_200 churn_503 churn_maxstreak churn_transitions <<<"$(read -r hold_total _ hold_503 _ _ <<<"$(series_metrics post-hold)"write_evidence_artifact-# ── Verdict ──-if [ "$EXPECT_LIVELOCK" = 1 ]; then- [ "$cuts" -ge "$CHURN_MIN_CUTS" ] &&- ok "churn cut the Context watch ${cuts}x (>= ${CHURN_MIN_CUTS}; ${total_watch_cut_lines} watch-ended lines total)" ||- fail "churn cut the Context watch only ${cuts}x — reproducer is VACUOUS"- [ "$gen_pattern" -ge 1 ] && ok "observed divergent-generation admission pattern ${gen_pattern}x" ||- fail "no divergent-generation pattern — bug mechanism NOT exercised"- [ "$churn_total" -gt 0 ] && ok "sampled /ready ${churn_total}x under churn" ||- fail "ZERO /ready samples under churn — the sampler never ran, and a zero-sample run must never pass"- if [ -z "$first_ready" ]; then- # Cross-check the exec probe against the kubelet's own view: a broken- # ready_status (wrong container/port) reports 503 forever and would fake- # the livelock. readyReplicas is omitted from status when it is 0.- ready_replicas="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"- if [ -z "$ready_replicas" ] || [ "$ready_replicas" = 0 ]; then- ok "livelock reproduced: /ready stayed 503 for ${READINESS_BUDGET_SEC}s under churn (kubelet agrees: 0 ready replicas)"- else- fail "exec probe reported 503 but the Deployment shows ${ready_replicas} ready replica(s) — probe is broken, reproducer VACUOUS"- fi- [ "$churn_maxstreak" -ge "$LIVELOCK_MIN_STREAK_SEC" ] &&- ok "the 503 streak never closed: ${churn_maxstreak}s continuous (>= ${LIVELOCK_MIN_STREAK_SEC}s; recovery budget is ${RECOVERY_BUDGET_SEC}s)" ||- fail "longest 503 streak was only ${churn_maxstreak}s — not a livelock signature"- else- fail "expected livelock but /ready reached 200 at ${first_ready}s — reproducer does not exercise the bug"- fi- # Post-churn behaviour is DIAGNOSTIC in this mode: the generation pin only- # livelocks WHILE the generation keeps bumping, so once the churn is paused- # the pre-fix code may legitimately certify. Asserting either outcome would- # turn an implementation detail of the pre-fix build into a gate verdict.- log "post-churn (diagnostic only): converged=${post_converged} ready-after=${post_ready_secs:-never}s hold-503s=${hold_503}"+# ── Verdict — POSITIVE case only ──+# EXPECT_LIVELOCK=1 is refused at the guard: the pre-fix livelock is not+# reproducible in minikube (see the SCOPE note). Its RED lives, mutation-verified,+# in host-context-controller/src/k8sClient.test.ts. This verdict proves the fix+# CONVERGES under real churn, plus anti-vacuity that the churn was real.+[ "$cuts" -ge "$CHURN_MIN_CUTS" ] &&+ ok "churn was real: ${cuts} Context-watch cuts (${total_watch_cut_lines} watch-ended lines) across readiness+observation" ||+ fail "fix validated WITHOUT churn (${cuts} Context-watch cuts) — result is worthless"+[ "$total_watch_cut_lines" -ge "$WATCH_CUT_MIN_LINES" ] &&+ ok "every informer was cut repeatedly: ${total_watch_cut_lines} watch-ended lines (>= ${WATCH_CUT_MIN_LINES} = 2 cycles x 3 watches)" ||+ fail "only ${total_watch_cut_lines} watch-ended lines (< ${WATCH_CUT_MIN_LINES}) — churn did not cut all three informers at least twice"+[ "$churn_total" -gt 0 ] && ok "sampled /ready ${churn_total}x under churn" ||+ fail "ZERO /ready samples under churn — the sampler never ran, and a zero-sample run must never pass"+if [ -n "$first_ready" ]; then+ ok "fix converged to Ready under sustained Premature close (time-to-Ready=${first_ready}s)"else- [ "$cuts" -ge "$CHURN_MIN_CUTS" ] &&- ok "churn was real: ${cuts} Context-watch cuts (${total_watch_cut_lines} watch-ended lines) across readiness+observation" ||- fail "fix validated WITHOUT churn (${cuts} Context-watch cuts) — result is worthless"- [ "$total_watch_cut_lines" -ge "$WATCH_CUT_MIN_LINES" ] &&- ok "every informer was cut repeatedly: ${total_watch_cut_lines} watch-ended lines (>= ${WATCH_CUT_MIN_LINES} = 2 cycles x 3 watches)" ||- fail "only ${total_watch_cut_lines} watch-ended lines (< ${WATCH_CUT_MIN_LINES}) — churn did not cut all three informers at least twice"- [ "$churn_total" -gt 0 ] && ok "sampled /ready ${churn_total}x under churn" ||- fail "ZERO /ready samples under churn — the sampler never ran, and a zero-sample run must never pass"- if [ -n "$first_ready" ]; then- ok "fix converged to Ready under sustained Premature close (time-to-Ready=${first_ready}s)"- else- fail "fix did NOT reach Ready under churn within ${READINESS_BUDGET_SEC}s — livelock regression (see ${LOG_ARTIFACT})"- fi- # Anti-vacuity: the proxy must actually have bitten the readiness path. A- # transient 503 under churn is the fail-closed CONTRACT working, not a bug;- # an observation window with zero 503s means the cuts never reached the- # watches and the bounded-recovery claim below would be vacuously true.- [ "$churn_503" -ge 1 ] &&- ok "churn bit the readiness path: ${churn_503} sample(s) at 503 (transient fail-closed is the contract)" ||- fail "zero 503 samples under churn — the proxy never bit the watches; bounded-recovery evidence is VACUOUS"- [ "$churn_transitions" -ge "$MIN_CHURN_RECOVERIES" ] &&- ok "repeated in-churn recovery: ${churn_transitions} distinct 503->200 transitions (>= ${MIN_CHURN_RECOVERIES})" ||- fail "only ${churn_transitions} 503->200 transition(s) under churn — a livelock is not refuted by fewer than ${MIN_CHURN_RECOVERIES} recoveries"- [ "$churn_maxstreak" -le "$RECOVERY_BUDGET_SEC" ] &&- ok "every in-churn 503 outage closed within budget: max streak ${churn_maxstreak}s <= ${RECOVERY_BUDGET_SEC}s" ||- fail "a 503 streak lasted ${churn_maxstreak}s (> ${RECOVERY_BUDGET_SEC}s recovery budget) — recovery is not bounded under churn"- # Post-churn: absorbing-state detector (the decisive livelock/transient split).- if [ "$post_converged" = 1 ]; then- ok "post-churn: /ready re-certified in ${post_ready_secs}s with NO content change injected"- [ "$hold_total" -gt 0 ] && [ "$hold_503" -eq 0 ] &&- ok "post-churn hold: ${hold_total} samples over ${STABILITY_WINDOW_SEC}s, zero 503 — no absorbing state" ||- fail "post-churn hold saw ${hold_503} 503(s) across ${hold_total} sample(s) — readiness did not stabilize once the churn stopped"- else- fail "post-churn: /ready did NOT converge within ${POST_STOP_READY_SEC}s after the churn was paused — absorbing state (livelock signature)"- fi- # In-situ guarantee: the convergence above must belong to the SAME process- # that lived through the churn. A restarted or replaced pod proves fresh-boot- # readiness, which the bootstrap gate already covers — not in-situ recovery.- hcc_pod_now="$(running_hcc_pod || true)"- hcc_restarts_now=""- if [ -n "$hcc_pod_now" ]; then- hcc_restarts_now="$(hcc_restart_count "$hcc_pod_now" || true)"- fi- if [ -n "$hcc_pod_at_stop" ] && [ "$hcc_pod_now" = "$hcc_pod_at_stop" ] &&- [ -n "$hcc_restarts_at_stop" ] && [ "$hcc_restarts_now" = "$hcc_restarts_at_stop" ]; then- ok "post-churn recovery was in-situ: pod ${hcc_pod_at_stop} unchanged, restartCount ${hcc_restarts_now}"- else- fail "HCC pod changed or restarted across the post-churn phase (${hcc_pod_at_stop:-none}/restarts=${hcc_restarts_at_stop:-?} -> ${hcc_pod_now:-none}/restarts=${hcc_restarts_now:-?}) — convergence could be a fresh boot, not in-situ recovery"- fi+ fail "fix did NOT reach Ready under churn within ${READINESS_BUDGET_SEC}s — livelock regression (see ${LOG_ARTIFACT})"+fi+# Anti-vacuity: the proxy must actually have bitten the readiness path. A+# transient 503 under churn is the fail-closed CONTRACT working, not a bug;+# an observation window with zero 503s means the cuts never reached the+# watches and the bounded-recovery claim below would be vacuously true.+[ "$churn_503" -ge 1 ] &&+ ok "churn bit the readiness path: ${churn_503} sample(s) at 503 (transient fail-closed is the contract)" ||+ fail "zero 503 samples under churn — the proxy never bit the watches; bounded-recovery evidence is VACUOUS"+[ "$churn_transitions" -ge "$MIN_CHURN_RECOVERIES" ] &&+ ok "repeated in-churn recovery: ${churn_transitions} distinct 503->200 transitions (>= ${MIN_CHURN_RECOVERIES})" ||+ fail "only ${churn_transitions} 503->200 transition(s) under churn — a livelock is not refuted by fewer than ${MIN_CHURN_RECOVERIES} recoveries"+[ "$churn_maxstreak" -le "$RECOVERY_BUDGET_SEC" ] &&+ ok "every in-churn 503 outage closed within budget: max streak ${churn_maxstreak}s <= ${RECOVERY_BUDGET_SEC}s" ||+ fail "a 503 streak lasted ${churn_maxstreak}s (> ${RECOVERY_BUDGET_SEC}s recovery budget) — recovery is not bounded under churn"+# Post-churn: absorbing-state detector (the decisive livelock/transient split).+if [ "$post_converged" = 1 ]; then+ ok "post-churn: /ready re-certified in ${post_ready_secs}s with NO content change injected"+ [ "$hold_total" -gt 0 ] && [ "$hold_503" -eq 0 ] &&+ ok "post-churn hold: ${hold_total} samples over ${STABILITY_WINDOW_SEC}s, zero 503 — no absorbing state" ||+ fail "post-churn hold saw ${hold_503} 503(s) across ${hold_total} sample(s) — readiness did not stabilize once the churn stopped"+else+ fail "post-churn: /ready did NOT converge within ${POST_STOP_READY_SEC}s after the churn was paused — absorbing state (livelock signature)"+fi+# In-situ guarantee: the convergence above must belong to the SAME process+# that lived through the churn. A restarted or replaced pod proves fresh-boot+# readiness, which the bootstrap gate already covers — not in-situ recovery.+hcc_pod_now="$(running_hcc_pod || true)"+hcc_restarts_now=""+if [ -n "$hcc_pod_now" ]; then+ hcc_restarts_now="$(hcc_restart_count "$hcc_pod_now" || true)"+fi+if [ -n "$hcc_pod_at_stop" ] && [ "$hcc_pod_now" = "$hcc_pod_at_stop" ] &&+ [ -n "$hcc_restarts_at_stop" ] && [ "$hcc_restarts_now" = "$hcc_restarts_at_stop" ]; then+ ok "post-churn recovery was in-situ: pod ${hcc_pod_at_stop} unchanged, restartCount ${hcc_restarts_now}"+else+ fail "HCC pod changed or restarted across the post-churn phase (${hcc_pod_at_stop:-none}/restarts=${hcc_restarts_at_stop:-?} -> ${hcc_pod_now:-none}/restarts=${hcc_restarts_now:-?}) — convergence could be a fresh boot, not in-situ recovery"filog "Evidence artifact: ${LOG_ARTIFACT}"# cleanup() (EXIT trap) restores HCC, deletes the fleet, and runs print_results.diff --git a/scripts/tests/test-hcc-watch-churn-readiness-gate.sh b/scripts/tests/test-hcc-watch-churn-readiness-gate.shindex 2ef41f306..c5b0162d4 100755--- a/scripts/tests/test-hcc-watch-churn-readiness-gate.sh+++ b/scripts/tests/test-hcc-watch-churn-readiness-gate.sh@@ -2,7 +2,9 @@# Offline contract gate for the HCC watch-churn readiness e2e. Runs in CI: no# cluster. Proves the gate keeps its fail-loud, branch-scoped, restorable shape# and its anti-vacuity assertions, so a refactor cannot quietly gut the one test-# that reproduces the clerum-dev livelock under real watch churn.+# that proves the fix converges under real watch churn. The livelock NEGATIVE is+# not reproducible in minikube (see the gate's SCOPE note); its RED lives in+# host-context-controller/src/k8sClient.test.ts, and check 2 pins that pointer.set -uFAIL=0ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"@@ -18,15 +20,18 @@ for s in "$GATE" "$FIXTURE"; dobash -n "$s" && pass "$(basename "$s") valid bash" || fail "$(basename "$s") invalid bash"done-# 1. Anti-vacuity: both regimes assert a minimum number of real watch cuts.-[ "$(grep -Ec 'cuts.*-ge.*CHURN_MIN_CUTS' "$GATE")" -ge 2 ] &&- pass "both livelock and fix regimes assert >= CHURN_MIN_CUTS real cuts" ||- fail "a regime can pass without proving churn actually cut the watches"+# 1. Anti-vacuity: the fix regime asserts a minimum number of real watch cuts.+[ "$(grep -Ec 'cuts.*-ge.*CHURN_MIN_CUTS' "$GATE")" -ge 1 ] &&+ pass "fix regime asserts >= CHURN_MIN_CUTS real cuts" ||+ fail "the gate can pass without proving churn actually cut the watches"-# 2. Livelock regime demands the divergent-generation evidence.-grep -Fq 'authority changed before .* admission' "$GATE" &&- pass "livelock regime keys on the real generation-divergence log" ||- fail "livelock regime does not check the generation-divergence signal"+# 2. The gate must refuse EXPECT_LIVELOCK=1 and point at the hermetic test that+# owns the livelock RED, so nobody can wire an irreproducible NEGATIVE here or+# delete the pointer to where the NEGATIVE actually lives.+grep -Eq 'EXPECT_LIVELOCK.*!=.*1' "$GATE" &&+ grep -Fq 'k8sClient.test.ts' "$GATE" &&+ pass "gate refuses EXPECT_LIVELOCK=1 and points at the hermetic livelock RED" ||+ fail "gate does not refuse EXPECT_LIVELOCK=1 or lost the pointer to the hermetic RED"# 3. Fix regime bounds in-churn recovery instead of banning transient 503s.# The fail-closed contract drops /ready to 503 TRANSIENTLY while a cut watch