Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
#!/usr/bin/env bash
# ======================================================================
# E2E — Host STORM gate (Issue #791 follow-up, plan Addendum 5)
# ======================================================================
#
# Reproduces the incident's *concurrency class* at realistic fleet scale
# against the branch HCC image: with the seeded fleet present, this gate
# OVERLAPS four storm ingredients in time (start timestamps are recorded
# and asserted, proving genuine concurrency rather than sequential phases):
#
# (a) create STORM_CREATE_COUNT (>=10) fixture Hosts, mixed
# stateful/stateless, storm-labeled;
# (b) delete STORM_DELETE_COUNT (>=3) pre-created fixture Hosts
# mid-storm (supported CR deletion path — never kubectl scale);
# (c) trigger 1 wake of a suspended stateless Host through the SUPPORTED
# rpc-proxy user route (POST /api/v1/rpc/hosts/:hostRef/wake with a
# per-user RPC token minted the Desktop App way — the same helper
# pattern as e2e-stateless-suspend-wake.sh);
# (d) kill the HCC pod once mid-storm (kubectl delete pod — a supported
# operational event forcing LIST->WATCH recovery under load).
#
# Assertions (each fail-loud with diagnostics), per plan §14.1/§14.2:
# A1 concurrency: delete/wake/kill start timestamps all fall strictly
# inside the Host-creation span (first create .. last create).
# Framing (for PR evidence): the interleaving is enforced by
# construction, so A1 is primarily a RECORDING of that evidence — it
# can only fail if the harness itself degenerates into sequential
# phases; the load-bearing falsifiable claims are A2-A8.
# A2 urgent queue-wait coverage has two independent same-pod windows:
# (a) the supported wake window immediately before the HCC kill requires
# every completed urgent sample <= 5s, after proving the wake reconcile
# finished and the urgent lane returned idle; and
# (b) ten dedicated Host ADDED events issued only after the replacement
# HCC completed LIST→WATCH recovery use a D - C1 p95 delta. A slow wake
# can therefore never be diluted by later fast post-recovery probes, and
# neither window crosses the intentional process restart.
# A3 no starvation: EVERY created Host's Deployment materializes within
# STORM_MATERIALIZATION_BUDGET_S (20s, §14.2 "New Host
# materialization") of that Host's OWN metadata.creationTimestamp,
# independent of the others. All deltas are listed; worst case printed.
# A4 watch recovery <= 15s across the HCC kill, from the new pod's
# clerum_hcc_host_watch_recovery_seconds{phase="total"} histogram
# (scrape C1): zero failure samples, >=1 success sample, and every
# success sample inside the le="15" bucket.
# A5 every deleted Host's bundle fully cleaned: ownership-label
# (clerum.io/managed-by + clerum.io/host) leftover check across the
# three bundle namespaces (mcp-host, channels, rpc-proxy) — the same
# predicate measure-host-bundle-reconcile.sh teardown uses. FAIL-
# CLOSED: a kubectl read that cannot be verified (API error without
# positive NotFound evidence) counts as failure, never as absence.
# A6 delete-cleanup counters consistent (zero lost deletes):
# clerum_hcc_host_delete_cleanup_total must be REGISTERED on the
# branch image (TYPE line present) and its outcomes must be
# consistent: retried == 0, superseded == 0, and
# queued == confirmed == completed. Read from scrape C2 (taken after
# A5 cleanup convergence, so the outcome counters are settled).
# All-zero is legitimate (deletes fully handled by the direct
# watch-DELETE path, which by design does not touch this counter) —
# A5 remains the falsifiable cleanup proof.
# A7 recovered fleet convergence is bounded, from
# clerum_hcc_host_fleet_requests_total (scrape C2) plus the replacement
# pod's explicit initial-fleet COMPLETED log: failed == 0 in both windows,
# the exact replacement pod completes its startup fleet pass, and
# trailing <= coalesced. The pod identity, completion marker, and A4's
# successful recovery histogram form the causal fence; started is not
# capped because an independent request may legitimately begin after the
# recovery pass settles. Requiring a coalesced request here would invent
# a second overlapping signal; the concurrent-request coalescing contract
# is covered deterministically by hostFleetScheduler.test.ts.
# A8 wake ingredient integrity: the woken Host leaves state=suspended and
# reaches a Ready pod within STORM_WAKE_READY_BUDGET_S of the wake
# POST. This is an ingredient-integrity bound (a silently no-op wake
# would weaken the gate), NOT a §14.2 wake SLO measurement — the §14.2
# 45s wake clock assumes a healthy controller, and this gate kills the
# controller seconds after the wake on purpose.
#
# HONEST SCOPE (plan Addendum 5.3 — recorded in the evidence): local
# minikube cannot reproduce API-server instability, which was the
# incident's amplifier. This gate validates scheduling fairness and
# convergence under realistic concurrency, NOT API degradation — that
# dimension remains covered by the bounded-deadline unit contracts and
# post-deploy telemetry.
#
# Fixtures are storm-labeled (e2e-storm=791 + e2e-storm-run=<run-id>) and
# torn down fail-loud; the profile and seeded data stay. Non-clerum.io
# label keys are used deliberately: clerum.io/* is platform-owned
# (HCC/WRC ownership semantics) and fixtures must never squat on it.
#
# Binding plan:
# .ralph/plans/2026-07-22-issue-791-hcc-priority-reconciliation-wake-plan.md
# Addendum 5 (storm shape + assertions), §14.1 (metric names),
# §14.2 (measurement clocks — start/end points are BINDING).
#
# House style: self-contained like measure-host-bundle-reconcile.sh (kctl
# --context wrapper, strict local-profile CONTEXT guard, log/ok/fail
# helpers, python3 for RFC3339->epoch and histogram math) with the
# rpc-token/wake helpers of e2e-stateless-suspend-wake.sh.
#
# ----------------------------------------------------------------------
# INPUTS (environment; fail-loud if missing/invalid)
# KUBECONTEXT / E2E_K8S_CONTEXT
# Exact generated branch-owned minikube context.
# MINIKUBE_PROFILE
# Must equal that context.
# E2E_EXPECTED_PRE_GATE_GATE
# Exact gate recorded by the branch-owned pre-gate sync.
# E2E_HCC_HOST_STORM_FAULT_INJECTION=1
# Explicit acknowledgement of the disruptive HCC pod restart.
#
# OPTIONAL INPUTS
# E2E_STORM_WAKE_HOST_REF suspended-wake target (default chatllm-stateless,
# the seeded stateless Host — it must exist with
# spec.lifecycle.stateless=true and be associated to the E2E
# user; seed via scripts/e2e/seed-stateless-host.sh).
# E2E_DEV_LOGIN_EMAIL default [email protected]
# E2E_USER_PASSWORD default ${ADMIN_PASSWORD:-changeme123!}
# STORM_CONTEXT_REF Context CR fixtures reference (default context1)
# STORM_MODEL_PROVIDER / STORM_MODEL_NAME fixture model (default zai/glm-5.1)
# CLERUM_PROFILE_PORTS_ENV optional explicit path to the selected branch
# profile's generated ports.env.
#
# Prereqs (each a HARD FAIL with the concrete reason):
# - branch HCC image deployed (clerum_hcc_host_delete_cleanup_total
# registered — A6 proves it)
# - seeded stateless Host present + E2E user associated (wake mintable)
# - external-rest-api + rpc-proxy reachable (make minikube-pf-all)
#
# Usage:
# MINIKUBE_PROFILE=<profile> KUBECONTEXT=<profile> \
# E2E_EXPECTED_PRE_GATE_GATE=<gate> \
# E2E_HCC_HOST_STORM_FAULT_INJECTION=1 \
# bash scripts/e2e/e2e-host-storm-gate.sh
#
# EXIT CODES
# 0 every storm assertion passed and teardown left zero fixtures
# 1 an assertion, precondition, or teardown failed loud — never masked
# 2 usage / input error
# ======================================================================
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=scripts/e2e/e2e-lib.sh
# shellcheck disable=SC1091
source "${SCRIPT_DIR}/e2e-lib.sh"
# shellcheck source=scripts/e2e/_lib/hcc-watch-recovery-fixture.sh
# shellcheck disable=SC1091
source "${SCRIPT_DIR}/_lib/hcc-watch-recovery-fixture.sh"
# shellcheck source=scripts/e2e/_lib/hcc-watch-recovery-lock.sh
# shellcheck disable=SC1091
source "${SCRIPT_DIR}/_lib/hcc-watch-recovery-lock.sh"
# ─── Constants mirrored from host-context-controller/src/{constants,config}.ts ──
readonly MANAGED_BY_LABEL='clerum.io/managed-by'
readonly MANAGED_BY_VALUE='host-context-controller'
readonly HOST_LABEL='clerum.io/host'
readonly OWNED_SELECTOR="${MANAGED_BY_LABEL}=${MANAGED_BY_VALUE}"
readonly MCP_HOST_NS='mcp-host' # config.hostNamespace
readonly CHANNELS_NS='channels' # config.channelsNamespace
readonly RPC_PROXY_NS='rpc-proxy' # config.rpcProxyNamespace
readonly CONTROL_NS='control-plane'
readonly HCC_DEPLOY='host-context-controller'
readonly HCC_PORT='8081' # containerPort http (deploy/base/control-plane)
readonly HCC_METRICS_PATH='/metrics' # server.ts GET /metrics
readonly HCC_PASS_COMPLETED_MARKER='Completed Host reconciliation after initial Host reconciliation'
readonly HCC_PASS_FAILED_MARKER='Host reconciliation after initial Host reconciliation failed'
# Shared HCC ownership/lock helpers use HCC_NS.
HCC_NS="$CONTROL_NS"
# Owned-child kinds + namespaces of a per-Host bundle (mirrors
# measure-host-bundle-reconcile.sh — includes per-Host NetworkPolicies).
readonly BUNDLE_KINDS='deployment service serviceaccount secret persistentvolumeclaim networkpolicy role rolebinding configmap'
readonly BUNDLE_NAMESPACES="${MCP_HOST_NS} ${CHANNELS_NS} ${RPC_PROXY_NS}"
# §14.1 metric names (host-context-controller/src/metrics.ts).
# EXPORTED so the adjudication python reads them from the environment. They are
# `readonly`, so passing them as same-name command-prefix assignments
# (`QUEUE_WAIT_METRIC="$QUEUE_WAIT_METRIC" python3 …`) would fail as a readonly
# reassignment under `set -e` — export once here and drop the inline prefixes.
readonly QUEUE_WAIT_METRIC='clerum_hcc_host_reconcile_queue_wait_seconds'
readonly IN_FLIGHT_METRIC='clerum_hcc_host_reconcile_in_flight'
readonly WATCH_RECOVERY_METRIC='clerum_hcc_host_watch_recovery_seconds'
readonly FLEET_METRIC='clerum_hcc_host_fleet_requests_total'
readonly DELETE_CLEANUP_METRIC='clerum_hcc_host_delete_cleanup_total'
export QUEUE_WAIT_METRIC WATCH_RECOVERY_METRIC FLEET_METRIC DELETE_CLEANUP_METRIC
# ─── Storm shape + budgets (each constant states its contract) ────────
# Addendum 5.1: ">=10" created fixture Hosts, mixed stateful/stateless.
readonly STORM_CREATE_COUNT=10
# A2's direct-watch probe is deliberately separate from the restart window:
# a cold-start LIST legitimately converges objects on the fleet lane. These
# ten ADDED events are emitted only once the replacement's LIST→WATCH recovery
# has succeeded, so their queue histogram is an unambiguous urgent-lane SLO.
readonly STORM_URGENT_PROBE_COUNT=10
# Addendum 5.1: ">=3" pre-created fixture Hosts deleted mid-storm.
readonly STORM_DELETE_COUNT=3
# §14.2 "urgent new/wake queue wait": local hard gate <= 5s (histogram
# bucket boundary le="5" exists in RECONCILE_LATENCY_BUCKETS).
readonly STORM_URGENT_P95_BUDGET_S=5
# The queue value itself is adjudicated against the 5s hard gate. This larger
# deadline bounds collection of the completed reconcile evidence and produces
# a loud operational failure if the wake never settles or telemetry is absent.
readonly STORM_WAKE_QUEUE_EVIDENCE_DEADLINE_S=60
# §14.2 "New Host materialization": Host CR -> Deployment <= 20s.
readonly STORM_MATERIALIZATION_BUDGET_S=20
# §14.2 "Watch recovery": recovery requested -> WATCH installed <= 15s
# (histogram bucket boundary le="15" exists).
readonly STORM_WATCH_RECOVERY_BUDGET_S=15
# A8 ingredient-integrity bound for the mid-storm wake: §14.2 wake budget
# (45s) + watch-recovery budget (15s, the controller is killed after the
# wake) + scheduling margin. NOT a §14.2 SLO measurement (see header).
readonly STORM_WAKE_READY_BUDGET_S=90
# Bounded discovery wait for ALL created Hosts' Deployments to exist before
# per-host deltas are adjudicated. Expiry is a HARD FAIL with diagnostics —
# the per-host 20s budget is judged from creationTimestamps regardless.
readonly STORM_CONVERGENCE_DEADLINE_S=120
# Bounded wait for a deleted Host's CR + full owned bundle to disappear.
readonly STORM_CLEANUP_DEADLINE_S=180
# Bounded wait for the delete-target fixtures' bundles to exist pre-storm
# (a delete of a bundle-less Host would prove nothing).
readonly STORM_PREP_MATERIALIZE_DEADLINE_S=90
# Bounded wait for the replacement HCC pod after the mid-storm kill.
readonly STORM_HCC_READY_DEADLINE_S=120
# Idle->suspend window for the wake target when it is not already
# suspended: emitter tick (~30s) + test idle floor (60s) + drain grace
# (20s) + HCC poll (5s) + slack (mirrors e2e-stateless-suspend-wake.sh
# SUSPEND_WINDOW rationale under the same shortened cadences).
readonly STORM_SUSPEND_WINDOW_S=210
# Bounded teardown wait before the fail-loud leftover check.
readonly STORM_TEARDOWN_DEADLINE_S=180
# Bounded EXTENSION for owned children that are already Terminating
# (deletionTimestamp set) at the leftover check — e.g. a workspace PVC whose
# kubernetes.io/pvc-protection finalizer drains for the deleting pod's
# termination grace. A leftover with NO deletionTimestamp is never given this
# extension (the delete was never issued → immediate fail).
readonly STORM_TEARDOWN_CONVERGE_S=120
# HCC test cadences applied ONLY when the wake target must first be driven
# to suspension (identical values to e2e-stateless-suspend-wake.sh; saved
# from the live deployment and restored in the cleanup trap).
readonly TEST_IDLE_MINUTES='1'
readonly TEST_IDLE_FLOOR_MINUTES='1'
readonly TEST_DRAIN_GRACE_MS='20000'
readonly TEST_POLL_MS='5000'
# Fixture provenance labels. Every object this gate creates carries BOTH so
# teardown deletes ONLY this run's fixtures. Deliberately NOT clerum.io/*
# (platform-owned label namespace — see header).
readonly FIXTURE_LABEL_KEY='e2e-storm'
readonly FIXTURE_LABEL_VAL='791'
readonly FIXTURE_RUN_KEY='e2e-storm-run'
# ─── Colors (mirrors e2e-lib.sh) ──────────────────────────────────────
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; CYAN='\033[0;36m'; NC='\033[0m'; BOLD='\033[1m'
# ─── Counters ─────────────────────────────────────────────────────────
g_pass=0; g_fail=0; g_total=0
# ─── Logging helpers (mirrors e2e-lib.sh / measure script) ────────────
log() { echo -e "${CYAN}[STORM]${NC} $*"; }
ok() { g_pass=$((g_pass+1)); g_total=$((g_total+1)); echo -e "${GREEN} PASS${NC} — $*"; }
fail() { g_fail=$((g_fail+1)); g_total=$((g_total+1)); echo -e "${RED} FAIL${NC} — $*" >&2; }
warn() { echo -e "${YELLOW} WARN${NC} — $*" >&2; }
header() { echo -e "\n${BOLD}=== $* ===${NC}"; }
die() { echo -e "${RED}[STORM] FATAL${NC} — ${1}" >&2; exit "${2:-1}"; }
print_results() {
echo -e "\n${BOLD}=== Storm gate summary (run-id ${RUN_ID:-?}) ===${NC}"
echo -e " ${GREEN}pass=${g_pass}${NC} ${RED}fail=${g_fail}${NC} total=${g_total}"
log "Honest scope (Addendum 5.3): local minikube cannot reproduce API-server"
log "instability (the incident's amplifier); this gate validates scheduling"
log "fairness and convergence under realistic concurrency, not API degradation."
}
# ─── Portable time helper (python3; mirrors measure script) ───────────
now_ms() { python3 -c 'import time;print(int(time.time()*1000))'; }
# ─── Strict local-profile context guard (fail-loud) ───────────────────
require_storm_context() {
[[ -n "${E2E_KUBECONTEXT:-}" ]] ||
die "KUBECONTEXT or E2E_K8S_CONTEXT must select an explicit branch-scoped minikube context." 2
is_branch_scoped_e2e_context "$E2E_KUBECONTEXT" ||
die "Refusing Host storm fault injection on non-branch context '${E2E_KUBECONTEXT}'." 2
require_safe_kube_context ||
die "context '${E2E_KUBECONTEXT}' is not an allowed local E2E context." 2
if ! kctl get nodes -o name >/dev/null 2>&1; then
die "context '${E2E_KUBECONTEXT}' is not reachable (kubectl get nodes failed)." 1
fi
kctl get nodes -o json | jq -e --arg context "$E2E_KUBECONTEXT" \
'any(.items[]; .metadata.labels["minikube.k8s.io/name"] == $context)' >/dev/null ||
die "target context is not the exact selected minikube profile '${E2E_KUBECONTEXT}'." 2
log "branch-owned minikube context OK: ${E2E_KUBECONTEXT}"
}
load_branch_profile_endpoints() {
local ports_env external_url rpc_url
ports_env="${CLERUM_PROFILE_PORTS_ENV:-${HOME}/.cache/clerum/minikube-profiles/${E2E_KUBECONTEXT}/ports.env}"
[[ -r "$ports_env" ]] ||
die "branch profile ports evidence is missing at ${ports_env}; refresh the branch-profile helper." 2
external_url="$(profile_env_value EXTERNAL_REST_API_BASE_URL "$ports_env")"
[[ -n "$external_url" ]] ||
external_url="$(profile_env_value EXTERNAL_REST_API_URL "$ports_env")"
rpc_url="$(profile_env_value RPC_PROXY_BASE_URL "$ports_env")"
[[ -n "$rpc_url" ]] ||
rpc_url="$(profile_env_value RPC_PROXY_URL "$ports_env")"
[[ "$external_url" =~ ^http://127\.0\.0\.1:[0-9]{2,5}$ ]] ||
die "branch ports.env has no valid loopback External REST URL." 2
[[ "$rpc_url" =~ ^http://127\.0\.0\.1:[0-9]{2,5}$ ]] ||
die "branch ports.env has no valid loopback RPC proxy URL." 2
EXT_BASE="$external_url"
RPC_BASE="$rpc_url"
log "using branch-profile endpoints from ${ports_env}"
}
require_deps() {
local missing=()
for bin in kubectl jq python3 curl mktemp; do
command -v "$bin" >/dev/null 2>&1 || missing+=("$bin")
done
[[ ${#missing[@]} -eq 0 ]] || die "missing required tools: ${missing[*]}" 2
}
# ─── Bounded wait helper: poll <fn> until success or <deadline_s>; on
# expiry dump <diag_fn> and return 1 (never silently continue) ────────
wait_until() {
local deadline_s="$1" interval_s="$2" desc="$3" probe_fn="$4" diag_fn="${5:-:}"
local start now
start="$(now_ms)"
while :; do
if "$probe_fn"; then return 0; fi
now="$(now_ms)"
if (( (now - start) / 1000 >= deadline_s )); then
warn "deadline ${deadline_s}s expired waiting for: ${desc}"
"$diag_fn" || true
return 1
fi
sleep "$interval_s"
done
}
# ======================================================================
# Run state
# ======================================================================
RUN_ID=''; WORK_DIR=''
FIXTURE_SECRET=''
STORM_CREATED_HOSTS=()
STORM_DELETE_HOSTS=()
STORM_URGENT_PROBE_HOSTS=()
WAKE_HOST_REF="${E2E_STORM_WAKE_HOST_REF:-chatllm-stateless}"
EXT_BASE=''
RPC_BASE=''
DEV_EMAIL="${E2E_DEV_LOGIN_EMAIL:-test@clerum.io}"
DEV_PASSWORD="${E2E_USER_PASSWORD:-${ADMIN_PASSWORD:-changeme123!}}"
STORM_CTX_REF="${STORM_CONTEXT_REF:-context1}"
STORM_PROVIDER="${STORM_MODEL_PROVIDER:-zai}"
STORM_MODEL="${STORM_MODEL_NAME:-glm-5.1}"
# Storm timeline (epoch-ms; A1 concurrency evidence).
T_CREATE_FIRST_MS=''; T_DELETE_MS=''; T_WAKE_MS=''; T_KILL_MS=''; T_CREATE_LAST_MS=''
# ─── Cleanup trap: best-effort fixture removal + HCC cadence restore ──
PF_PID=''
HCC_ENV_SAVED=''
TEARDOWN_DONE=0
TEARDOWN_CLEANUP_FAILED=0
HCC_RESTORE_OK=1
STORM_FIXTURES_CREATED=0
# Read and mutated by the sourced compare-and-swap lock helper.
HCC_GATE_LOCK_ACQUIRED=0
HCC_GATE_LOCK_NAME=''
HCC_GATE_LOCK_UID=''
HCC_GATE_FINALIZATION_FAILURE=''
cleanup_portforward() {
if [[ -n "$PF_PID" ]] && kill -0 "$PF_PID" 2>/dev/null; then
kill "$PF_PID" 2>/dev/null || true
wait "$PF_PID" 2>/dev/null || true
fi
PF_PID=''
}
# Build the kubectl set env restore args from HCC_ENV_SAVED into the
# caller's `args` array (empty saved value => remove the key).
_hcc_restore_args() {
args=()
local key val
while IFS='=' read -r key val; do
[[ -n "$key" ]] || continue
if [[ -n "$val" ]]; then args+=("${key}=${val}"); else args+=("${key}-"); fi
done <<< "$HCC_ENV_SAVED"
}
# Lenient restore — EXIT-trap path only: best-effort with warnings, so an
# abnormal exit still attempts to undo the cadence mutation.
# shellcheck disable=SC2329 # invoked from cleanup_on_exit (EXIT trap)
restore_hcc_env() {
[[ -n "$HCC_ENV_SAVED" ]] || return 0
local args=()
_hcc_restore_args
[[ ${#args[@]} -gt 0 ]] || { HCC_ENV_SAVED=''; return 0; }
log "Restoring HCC cadence env on deployment/${HCC_DEPLOY}"
if ! kctl set env "deployment/${HCC_DEPLOY}" -n "$CONTROL_NS" "${args[@]}" >/dev/null 2>&1; then
warn "failed to restore HCC env (manual check required)"
HCC_RESTORE_OK=0
return 1
fi
if ! kctl rollout status "deployment/${HCC_DEPLOY}" -n "$CONTROL_NS" --timeout=180s >/dev/null 2>&1; then
warn "HCC rollout did not settle after env restore"
HCC_RESTORE_OK=0
return 1
fi
HCC_ENV_SAVED=''
}
# Strict restore — SUCCESS path (M3): a failed cadence restore is real
# profile damage and must flip the gate to FAIL (counted via fail());
# HCC_ENV_SAVED is kept on failure so the EXIT trap retries best-effort.
# Always returns 0 so the final print_results/exit-code path still runs.
restore_hcc_env_strict() {
[[ -n "$HCC_ENV_SAVED" ]] || return 0
local args=()
_hcc_restore_args
[[ ${#args[@]} -gt 0 ]] || { HCC_ENV_SAVED=''; return 0; }
log "Restoring HCC cadence env on deployment/${HCC_DEPLOY} (strict)"
if ! kctl set env "deployment/${HCC_DEPLOY}" -n "$CONTROL_NS" "${args[@]}" >/dev/null 2>&1; then
HCC_RESTORE_OK=0
fail "cadence restore FAILED — the profile is left with test cadences on deployment/${HCC_DEPLOY} (the EXIT trap retries best-effort; verify manually)"
return 0
fi
if ! kctl rollout status "deployment/${HCC_DEPLOY}" -n "$CONTROL_NS" --timeout=180s >/dev/null 2>&1; then
HCC_RESTORE_OK=0
fail "HCC rollout did not settle after the cadence restore — verify deployment/${HCC_DEPLOY} manually"
return 0
fi
HCC_ENV_SAVED=''
ok "HCC test cadences restored and rolled out"
}
# shellcheck disable=SC2329 # invoked via the EXIT trap below
cleanup_on_exit() {
local status=$? cleanup_failed=0
set +e
cleanup_portforward
if [[ "$TEARDOWN_DONE" != '1' && "$STORM_FIXTURES_CREATED" == '1' && -n "$RUN_ID" ]]; then
cleanup_failed=1
local sel="${FIXTURE_LABEL_KEY}=${FIXTURE_LABEL_VAL},${FIXTURE_RUN_KEY}=${RUN_ID}"
warn "abnormal exit — best-effort deletion of storm fixtures (${sel})"
kctl -n "$MCP_HOST_NS" delete host -l "$sel" --wait=false >/dev/null 2>&1 || \
warn "best-effort fixture Host deletion failed (manual check advised)"
kctl -n "$MCP_HOST_NS" delete secret -l "$sel" --wait=false >/dev/null 2>&1 || \
warn "best-effort fixture Secret deletion failed (manual check advised)"
fi
restore_hcc_env || HCC_RESTORE_OK=0
if ! finalize_hcc_watch_gate_lock "$cleanup_failed" "$HCC_RESTORE_OK"; then
status=1
fi
exit "$status"
}
trap cleanup_on_exit EXIT
# ======================================================================
# Fixture + observable helpers
# ======================================================================
init_run() {
# Run discriminator: FULL epoch-ms plus a random suffix — no truncation,
# so there is no periodic tail-collision window between runs. Longest
# fixture name stays s791-<13ms>-<5rand>-c%02d = 28 chars, inside the
# 34-char budget that keeps host-<name>-mcp-host-runtime-tokens <= 63.
RUN_ID="$(now_ms)-${RANDOM}"
WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/e2e-host-storm-${RUN_ID}.XXXXXX")"
FIXTURE_SECRET="s791-${RUN_ID}-keys"
local i
for i in $(seq 1 "$STORM_CREATE_COUNT"); do
STORM_CREATED_HOSTS+=("$(printf 's791-%s-c%02d' "$RUN_ID" "$i")")
done
for i in $(seq 1 "$STORM_DELETE_COUNT"); do
STORM_DELETE_HOSTS+=("$(printf 's791-%s-d%d' "$RUN_ID" "$i")")
done
for i in $(seq 1 "$STORM_URGENT_PROBE_COUNT"); do
STORM_URGENT_PROBE_HOSTS+=("$(printf 's791-%s-u%02d' "$RUN_ID" "$i")")
done
log "run-id ${RUN_ID}; work dir ${WORK_DIR}"
log "create fixtures: ${STORM_CREATED_HOSTS[*]}"
log "delete fixtures: ${STORM_DELETE_HOSTS[*]}"
}
ensure_fixture_secret() {
# Throwaway placeholder Secret (mirrors measure script): the mcp-host
# readiness probe is an HTTP health endpoint that never validates the LLM
# key, so a placeholder is sufficient for materialization clocks.
cat <<YAML | kctl apply -f - >/dev/null
apiVersion: v1
kind: Secret
metadata:
name: ${FIXTURE_SECRET}
namespace: ${MCP_HOST_NS}
labels:
${FIXTURE_LABEL_KEY}: "${FIXTURE_LABEL_VAL}"
${FIXTURE_RUN_KEY}: "${RUN_ID}"
type: Opaque
stringData:
ZAI_API_KEY: "placeholder-e2e-storm-not-a-real-key"
OPENAI_API_KEY: "placeholder-e2e-storm-not-a-real-key"
YAML
log "created placeholder Secret ${FIXTURE_SECRET}"
}
# apply_storm_host <name> <stateless:true|false>
apply_storm_host() {
local name="$1" stateless="$2" lifecycle_block=''
if [[ "$stateless" == 'true' ]]; then
lifecycle_block=$' lifecycle:\n stateless: true'
fi
cat <<YAML | kctl apply -f - >/dev/null
apiVersion: clerum.io/v1alpha1
kind: Host
metadata:
name: ${name}
namespace: ${MCP_HOST_NS}
labels:
${FIXTURE_LABEL_KEY}: "${FIXTURE_LABEL_VAL}"
${FIXTURE_RUN_KEY}: "${RUN_ID}"
spec:
host: ${name}
contextRef: ${STORM_CTX_REF}
secretRef: ${FIXTURE_SECRET}
model:
provider: ${STORM_PROVIDER}
name: ${STORM_MODEL}
${lifecycle_block}
YAML
}
# Collect every owned child of <host> across the bundle namespaces —
# the authoritative HCC ownership selector (managed-by + host), same
# predicate as measure-host-bundle-reconcile.sh / HostReconciler.
#
# FAIL-CLOSED (B2): prints "ns/kind/name" lines to stdout and returns 0
# even when empty; returns 2 when ANY kubectl read FAILED (the error goes
# to stderr). Every label-selector list of these standard kinds returns
# exit 0 with empty output when nothing matches, so a nonzero kubectl exit
# is a real read failure (API outage, RBAC, unknown kind) — callers MUST
# treat rc=2 as "verification impossible" (a failure), never as "zero
# leftovers".
collect_children() {
local host="$1" ns kind out rc=0
for ns in $BUNDLE_NAMESPACES; do
for kind in $BUNDLE_KINDS; do
if ! out="$(kctl -n "$ns" get "$kind" \
-l "${OWNED_SELECTOR},${HOST_LABEL}=${host}" -o name 2>"${WORK_DIR}/collect-children.err")"; then
echo "collect_children: kubectl get ${kind} in ${ns} FAILED for host ${host}: $(cat "${WORK_DIR}/collect-children.err" 2>/dev/null)" >&2
rc=2
continue
fi
if [[ -n "$out" ]]; then
printf '%s\n' "$out" | sed "s|^|${ns}/|"
fi
done
done
return "$rc"
}
# FAIL-CLOSED absence check (B2): returns 0 ONLY when the API positively
# answered NotFound for the Host. "Present" returns 1; any other kubectl
# failure (outage, RBAC, timeout) returns 2 with the error on stderr —
# absence must never be inferred from an unreadable API.
# shellcheck disable=SC2329 # invoked from _deleted_hosts_clean_probe (runs via wait_until)
host_absent_confirmed() {
local h="$1" err
if kctl -n "$MCP_HOST_NS" get host "$h" -o name >/dev/null 2>"${WORK_DIR}/host-absent.err"; then
return 1
fi
err="$(cat "${WORK_DIR}/host-absent.err" 2>/dev/null || true)"
# Anchored match: the API-server reason '(NotFound)' or the kubectl
# trailing form naming EXACTLY the requested Host. A kubeconfig-level
# error like `context "..." not found` must never confirm absence.
if printf '%s' "$err" | grep -Eq "\(NotFound\)|\"${h}\" not found\$"; then
return 0
fi
echo "host_absent_confirmed: kubectl get host ${h} FAILED without NotFound evidence: ${err}" >&2
return 2
}
hcc_pod_name() {
kctl -n "$CONTROL_NS" get pods -l "app=${HCC_DEPLOY}" \
--field-selector=status.phase=Running \
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.deletionTimestamp}{"\n"}{end}' 2>/dev/null \
| awk -F'\t' '$1 != "" && $2 == "" {print $1; exit}'
}
storm_diagnostics() {
{
echo "----- storm diagnostics (run-id ${RUN_ID}) -----"
kctl -n "$MCP_HOST_NS" get host -l "${FIXTURE_LABEL_KEY}=${FIXTURE_LABEL_VAL},${FIXTURE_RUN_KEY}=${RUN_ID}" -o wide 2>&1 || true
kctl -n "$MCP_HOST_NS" get deploy -l "$OWNED_SELECTOR" -o wide 2>&1 | head -40 || true
kctl -n "$CONTROL_NS" get pods -l "app=${HCC_DEPLOY}" -o wide 2>&1 || true
echo "----- recent HCC logs -----"
kctl -n "$CONTROL_NS" logs "deploy/${HCC_DEPLOY}" --tail=60 2>&1 || true
} >&2
}
# ─── Metrics scrape (ephemeral port-forward; mirrors measure script) ──
# scrape_hcc_metrics <outfile> <pod-name> — M1: the port-forward binds the
# POD BY NAME (never deploy/) so a scrape can never silently read a
# different pod (e.g. a terminating one during a rollout) than the one the
# identity guard checked; cross-pod counter deltas would false-PASS A4.
scrape_hcc_metrics() {
local out="$1" pod="$2" lport='' url pf_log
pf_log="${WORK_DIR}/pf-${pod}.log"
# Let kubectl allocate a process-owned loopback port. A fixed port is
# machine-global and can silently collide with another branch/profile gate.
kctl -n "$CONTROL_NS" port-forward --address=127.0.0.1 \
"pod/${pod}" ":${HCC_PORT}" >"$pf_log" 2>&1 &
PF_PID=$!
# shellcheck disable=SC2329 # invoked by name via wait_until
_pf_ready_probe() {
kill -0 "$PF_PID" 2>/dev/null || return 1
lport="$(sed -nE 's/^Forwarding from 127\.0\.0\.1:([0-9]+) -> .*/\1/p' "$pf_log" | head -n1)"
[[ "$lport" =~ ^[0-9]+$ ]]
}
# shellcheck disable=SC2329 # invoked by name via wait_until
_pf_diag() { { echo '----- port-forward log -----'; cat "$pf_log" 2>/dev/null; } >&2; }
if ! wait_until 30 1 "random HCC metrics port-forward" _pf_ready_probe _pf_diag; then
cleanup_portforward
return 1
fi
url="http://127.0.0.1:${lport}${HCC_METRICS_PATH}"
# shellcheck disable=SC2329 # invoked by name via wait_until
_scrape_probe() { curl -fsS --max-time 3 "$url" -o "$out" 2>/dev/null; }
# shellcheck disable=SC2329 # invoked by name via wait_until
_scrape_diag() { _pf_diag; }
if ! wait_until 30 2 "HCC metrics scrape -> ${out}" _scrape_probe _scrape_diag; then
cleanup_portforward
return 1
fi
cleanup_portforward
}
# ─── Wake-target lifecycle observables ────────────────────────────────
wake_lifecycle_state() {
kctl -n "$MCP_HOST_NS" get host "$WAKE_HOST_REF" -o jsonpath='{.status.lifecycle.state}' 2>/dev/null || true
}
# shellcheck disable=SC2329 # invoked from the wait_until callback below
wake_handled_generation() {
kctl -n "$MCP_HOST_NS" get host "$WAKE_HOST_REF" \
-o jsonpath='{.status.lifecycle.wakeHandledGeneration}' 2>/dev/null || true
}
# Correlate the public wake request with HCC's durable lifecycle status. This
# predicate is deliberately pure so the static gate test can exercise the
# same ordering contract used by the cluster E2E.
# shellcheck disable=SC2329 # invoked from the wait_until callback below
wake_generation_is_handled() {
local requested="$1" handled="$2"
[[ "$requested" =~ ^[1-9][0-9]*$ ]] || return 1
[[ "$handled" =~ ^[0-9]+$ ]] || return 1
(( handled >= requested ))
}
# shellcheck disable=SC2329 # invoked from _wake_converged_probe (runs via wait_until)
wake_ready_pod_exists() {
kctl -n "$MCP_HOST_NS" get pods -l "app=${WAKE_HOST_REF}" \
--field-selector=status.phase=Running \
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' 2>/dev/null \
| grep -qx 'True'
}
# ─── RPC helpers (Desktop App auth path; mirrors e2e-stateless-suspend-wake.sh) ──
SESSION_TOKEN=''; RPC_TOKEN=''
RPC_SCOPES_JSON='["host:message:invoke","host:approval:write","host:session:read","host:status:read","host:health:read","host:wake:write"]'
mint_session_token() {
local resp code body
resp=$(curl -sS -m 30 -w '\n%{http_code}' -X POST "${EXT_BASE}/api/v1/auth/password-login" \
-H 'Content-Type: application/json' \
-d "$(jq -cn --arg e "$DEV_EMAIL" --arg p "$DEV_PASSWORD" '{email:$e,password:$p}')") || {
echo "password-login request failed at ${EXT_BASE}" >&2; return 1; }
code="$(echo "$resp" | tail -n1)"; body="$(echo "$resp" | sed '$d')"
[[ "$code" == '200' ]] || { echo "password-login -> HTTP ${code}: ${body}" >&2; return 1; }
SESSION_TOKEN="$(echo "$body" | jq -r '.token // empty')"
[[ -n "$SESSION_TOKEN" ]] || { echo "password-login returned no .token" >&2; return 1; }
}
mint_rpc_token() {
local resp code body
resp=$(curl -sS -m 30 -w '\n%{http_code}' -X POST "${EXT_BASE}/api/v1/rpc/token" \
-H "Authorization: Bearer ${SESSION_TOKEN}" -H 'Content-Type: application/json' \
-d "$(jq -cn --arg h "$WAKE_HOST_REF" --argjson s "$RPC_SCOPES_JSON" '{hostRefs:[$h],scopes:$s}')") || {
echo "rpc/token request failed at ${EXT_BASE}" >&2; return 1; }
code="$(echo "$resp" | tail -n1)"; body="$(echo "$resp" | sed '$d')"
[[ "$code" == '200' ]] || { echo "rpc/token -> HTTP ${code}: ${body}" >&2; return 1; }
RPC_TOKEN="$(echo "$body" | jq -r '.token // empty')"
[[ -n "$RPC_TOKEN" ]] || { echo "rpc/token returned no .token" >&2; return 1; }
}
# The wake route is the user-facing rpc-proxy endpoint Desktop uses;
# rpc-proxy owns the control-api hop. Never kubectl scale.
WAKE_STATUS=''; WAKE_BODY=''; WAKE_GENERATION=''
post_wake() {
local resp response_status response_generation
WAKE_STATUS=''; WAKE_BODY=''; WAKE_GENERATION=''
mint_rpc_token || return 1
resp=$(curl -sS -m 30 -w '\n%{http_code}' -X POST \
"${RPC_BASE}/api/v1/rpc/hosts/${WAKE_HOST_REF}/wake" \
-H "Authorization: Bearer ${RPC_TOKEN}" -H 'Content-Type: application/json') || {
echo "wake POST failed at ${RPC_BASE}" >&2; return 1; }
WAKE_STATUS="$(echo "$resp" | tail -n1)"; WAKE_BODY="$(echo "$resp" | sed '$d')"
[[ "$WAKE_STATUS" == '202' ]] || {
echo "wake POST returned HTTP ${WAKE_STATUS}, expected 202: ${WAKE_BODY}" >&2
return 1
}
response_status="$(jq -r '.status // empty' <<<"$WAKE_BODY" 2>/dev/null || true)"
response_generation="$(jq -r '.wakeGeneration // empty' <<<"$WAKE_BODY" 2>/dev/null || true)"
[[ "$response_status" == 'wake-requested' ]] || {
echo "wake POST returned unexpected status '${response_status:-<missing>}': ${WAKE_BODY}" >&2
return 1
}
[[ "$response_generation" =~ ^[1-9][0-9]*$ ]] || {
echo "wake POST returned invalid wakeGeneration '${response_generation:-<missing>}': ${WAKE_BODY}" >&2
return 1
}
WAKE_GENERATION="$response_generation"
}
# ─── HCC cadence shortening (only when the wake target must suspend) ──
save_and_set_hcc_cadences() {
header "Setting HCC test cadences (idle=${TEST_IDLE_MINUTES}m, drain=${TEST_DRAIN_GRACE_MS}ms, poll=${TEST_POLL_MS}ms)"
local keys=(CONTEXT_MAPPER_STATELESS_IDLE_MINUTES CONTEXT_MAPPER_STATELESS_IDLE_FLOOR_MINUTES \
CONTEXT_MAPPER_STATELESS_DRAIN_GRACE_MS CONTEXT_MAPPER_HEARTBEAT_POLL_MS)
local k cur saved=''
for k in "${keys[@]}"; do
cur="$(kctl get "deployment/${HCC_DEPLOY}" -n "$CONTROL_NS" \
-o jsonpath="{.spec.template.spec.containers[0].env[?(@.name=='${k}')].value}" 2>/dev/null || true)"
saved+="${k}=${cur}"$'\n'
done
HCC_ENV_SAVED="$saved"
kctl set env "deployment/${HCC_DEPLOY}" -n "$CONTROL_NS" \
"CONTEXT_MAPPER_STATELESS_IDLE_MINUTES=${TEST_IDLE_MINUTES}" \
"CONTEXT_MAPPER_STATELESS_IDLE_FLOOR_MINUTES=${TEST_IDLE_FLOOR_MINUTES}" \
"CONTEXT_MAPPER_STATELESS_DRAIN_GRACE_MS=${TEST_DRAIN_GRACE_MS}" \
"CONTEXT_MAPPER_HEARTBEAT_POLL_MS=${TEST_POLL_MS}" >/dev/null 2>&1 || {
fail "failed to set HCC test cadences via kubectl set env"; return 1; }
if kctl rollout status "deployment/${HCC_DEPLOY}" -n "$CONTROL_NS" --timeout=180s >/dev/null 2>&1; then
ok "HCC test cadences applied and rolled out"
else
fail "HCC rollout did not settle after cadence set"; storm_diagnostics; return 1
fi
}
# shellcheck disable=SC2329 # invoked by name via wait_until
_wake_target_suspended_probe() {
local st reps
st="$(wake_lifecycle_state)"
reps="$(kctl -n "$MCP_HOST_NS" get deployment "$WAKE_HOST_REF" -o jsonpath='{.spec.replicas}' 2>/dev/null || echo '')"
[[ "$st" == 'suspended' && "${reps:-1}" == '0' ]]
}
# shellcheck disable=SC2329 # invoked by name via wait_until
_wake_target_diag() {
{
echo "----- wake target diagnostics (${WAKE_HOST_REF}) -----"
kctl -n "$MCP_HOST_NS" get host "$WAKE_HOST_REF" \
-o jsonpath='{"lifecycle="}{.status.lifecycle}{"\n"}' 2>&1 || true
kctl -n "$MCP_HOST_NS" get pods -l "app=${WAKE_HOST_REF}" -o wide 2>&1 || true
kctl -n "$CONTROL_NS" logs "deploy/${HCC_DEPLOY}" --tail=40 2>&1 || true
} >&2
}
# ======================================================================
# Prerequisites
# ======================================================================
require_deps
require_storm_context
init_run
[[ "${KEEP_FIXTURES:-0}" != '1' ]] ||
die "KEEP_FIXTURES=1 is diagnostic-only and cannot produce a probative T2 Host-storm result; refusing before mutation." 2
[[ "${E2E_HCC_HOST_STORM_FAULT_INJECTION:-0}" == '1' ]] ||
die "Set E2E_HCC_HOST_STORM_FAULT_INJECTION=1 to acknowledge the temporary HCC pod restart." 2
require_branch_owned_hcc_gate "$CONTROL_NS"
load_branch_profile_endpoints
header "Prerequisites"
ok "branch helper, profile, exact HEAD/fingerprint/gate, and minikube node label verified"
if kctl get deployment "$HCC_DEPLOY" -n "$CONTROL_NS" >/dev/null 2>&1; then
ok "HCC deployment present in ${CONTROL_NS}"
else
fail "deployment/${HCC_DEPLOY} not found in ${CONTROL_NS}"; print_results; exit 1
fi
stateless_flag="$(kctl -n "$MCP_HOST_NS" get host "$WAKE_HOST_REF" -o jsonpath='{.spec.lifecycle.stateless}' 2>/dev/null || echo '')"
if [[ "$stateless_flag" == 'true' ]]; then
ok "wake target '${WAKE_HOST_REF}' exists with spec.lifecycle.stateless=true"
else
fail "wake target '${WAKE_HOST_REF}' missing or not stateless (got '${stateless_flag:-<absent>}'). Seed: bash scripts/e2e/seed-stateless-host.sh"
print_results; exit 1
fi
if kctl -n "$MCP_HOST_NS" get context.clerum.io "$STORM_CTX_REF" >/dev/null 2>&1 \
|| kctl get context.clerum.io "$STORM_CTX_REF" -n mcp-server >/dev/null 2>&1; then
ok "fixture contextRef '${STORM_CTX_REF}' exists"
else
fail "fixture contextRef '${STORM_CTX_REF}' not found (set STORM_CONTEXT_REF to an existing Context CR)"
print_results; exit 1
fi
for svc in "external-rest-api ${EXT_BASE}" "rpc-proxy ${RPC_BASE}"; do
name="${svc%% *}"; base="${svc##* }"
if curl -fsS -m 10 "${base}/health" >/dev/null 2>&1; then
ok "${name} reachable at ${base}"
else
fail "${name} NOT reachable at ${base} -- start port-forwards (make minikube-pf-all)"; print_results; exit 1
fi
done
if mint_session_token; then ok "session token minted for ${DEV_EMAIL}"; else
fail "cannot login as ${DEV_EMAIL} on ${EXT_BASE}"; print_results; exit 1; fi
if mint_rpc_token; then
ok "user RPC token minted for hostRef=${WAKE_HOST_REF} (host:wake:write among scopes)"
else
fail "cannot mint RPC token for hostRef=${WAKE_HOST_REF} -- is ${DEV_EMAIL} associated to it? Seed: bash scripts/e2e/seed-stateless-host.sh"
print_results; exit 1
fi
if acquire_hcc_watch_gate_lock; then
ok "exclusive shared HCC fault-injection lock acquired"
else
die "another disruptive HCC gate owns context ${E2E_KUBECONTEXT}"
fi
# ======================================================================
# Prep 1 — storm fixtures that will be DELETED mid-storm (bundles must
# exist first: deleting a bundle-less Host proves nothing)
# ======================================================================
header "Prep — pre-create ${STORM_DELETE_COUNT} delete-target fixture Hosts"
STORM_FIXTURES_CREATED=1
ensure_fixture_secret
for h in "${STORM_DELETE_HOSTS[@]}"; do
apply_storm_host "$h" false
done
# shellcheck disable=SC2329 # invoked by name via wait_until
_delete_targets_materialized_probe() {
local h
for h in "${STORM_DELETE_HOSTS[@]}"; do
kctl -n "$MCP_HOST_NS" get deployment "$h" >/dev/null 2>&1 || return 1
done
}
if wait_until "$STORM_PREP_MATERIALIZE_DEADLINE_S" 3 \
"delete-target bundles (Deployments) to materialize" \
_delete_targets_materialized_probe storm_diagnostics; then
ok "all ${STORM_DELETE_COUNT} delete-target Hosts have materialized bundles"
else
fail "delete-target bundles did not materialize within ${STORM_PREP_MATERIALIZE_DEADLINE_S}s — cannot run a meaningful mid-storm delete"
print_results; exit 1
fi
# ======================================================================
# Prep 2 — ensure the wake target is SUSPENDED (idle-driven, supported
# path; shortened cadences only if needed, restored in the trap)
# ======================================================================
header "Prep — wake target suspension (${WAKE_HOST_REF})"
if _wake_target_suspended_probe; then
ok "wake target already suspended (state=suspended, replicas=0)"
else
save_and_set_hcc_cadences || { print_results; exit 1; }
if wait_until "$STORM_SUSPEND_WINDOW_S" 5 \
"wake target ${WAKE_HOST_REF} to idle-suspend" \
_wake_target_suspended_probe _wake_target_diag; then
ok "wake target suspended via idle expiry (supported D8 path)"
else
fail "wake target did not suspend within ${STORM_SUSPEND_WINDOW_S}s (state=$(wake_lifecycle_state)) — cannot exercise the mid-storm wake ingredient"
print_results; exit 1
fi
fi
# ======================================================================
# Prep 3 — metrics baseline (scrape A) + pre-storm HCC pod identity
# ======================================================================
header "Prep — metrics baseline (scrape A)"
SCRAPE_A="${WORK_DIR}/metrics-A.prom"
SCRAPE_B="${WORK_DIR}/metrics-B.prom"
SCRAPE_C1="${WORK_DIR}/metrics-C1.prom"
SCRAPE_C2="${WORK_DIR}/metrics-C2.prom"
SCRAPE_D="${WORK_DIR}/metrics-D.prom"
SCRAPE_WAKE_PRE="${WORK_DIR}/metrics-wake-pre.prom"
SCRAPE_WAKE_POST="${WORK_DIR}/metrics-wake-post.prom"
HCC_POD_A="$(hcc_pod_name)"
if [[ -z "$HCC_POD_A" ]]; then
fail "no Running HCC pod found for baseline identity"; storm_diagnostics; print_results; exit 1
fi
if scrape_hcc_metrics "$SCRAPE_A" "$HCC_POD_A"; then
ok "baseline scrape A captured (pod ${HCC_POD_A})"
else
fail "could not scrape HCC ${HCC_METRICS_PATH} for baseline"; print_results; exit 1
fi
# ======================================================================
# STORM — overlapped ingredients (timestamps recorded for A1)
# ======================================================================
header "STORM — overlapped create/delete/wake/kill (run-id ${RUN_ID})"
STORM_HALF=$(( STORM_CREATE_COUNT / 2 ))
T_CREATE_FIRST_MS="$(now_ms)"
i=0
for h in "${STORM_CREATED_HOSTS[@]}"; do
i=$((i + 1))
[[ $i -gt $STORM_HALF ]] && break
# Mixed lifecycle: even index stateless, odd stateful (Addendum 5.1).
if (( i % 2 == 0 )); then apply_storm_host "$h" true; else apply_storm_host "$h" false; fi
done
log "storm: first ${STORM_HALF} Host creations issued"
T_DELETE_MS="$(now_ms)"
for h in "${STORM_DELETE_HOSTS[@]}"; do
kctl -n "$MCP_HOST_NS" delete host "$h" --wait=false >/dev/null || {
fail "storm: mid-storm delete of Host ${h} failed"; print_results; exit 1; }
done
log "storm: ${STORM_DELETE_COUNT} mid-storm Host deletions issued"
# Isolate the supported wake on the pre-kill pod. Samples from the first half
# of Host creation may still settle inside this window; they are valid urgent
# traffic and the hard-gate assertion below requires every one of them, not
# only a percentile, to remain within budget.
HCC_POD_WAKE="$(hcc_pod_name)"
if [[ -z "$HCC_POD_WAKE" || "$HCC_POD_WAKE" != "$HCC_POD_A" ]]; then
fail "storm: HCC pod changed before the wake metric baseline (${HCC_POD_A} -> ${HCC_POD_WAKE:-<none>})"
storm_diagnostics; print_results; exit 1
fi
if ! scrape_hcc_metrics "$SCRAPE_WAKE_PRE" "$HCC_POD_WAKE"; then
fail "storm: could not capture the same-pod wake metric baseline"
print_results; exit 1
fi
T_WAKE_MS="$(now_ms)"
if post_wake; then
log "storm: wake POST accepted for ${WAKE_HOST_REF} (HTTP ${WAKE_STATUS}, generation ${WAKE_GENERATION})"
else
fail "storm: mid-storm wake POST failed for ${WAKE_HOST_REF}"; print_results; exit 1
fi
# Prove that this specific supported wake reached HCC: its response generation
# must be durably reflected in lifecycle status. Then require a new urgent
# queue sample and an idle urgent lane on the same pod. Prometheus collectors
# are read independently, so this deliberately uses a two-scrape fence instead
# of assuming cross-metric atomicity: once the first scrape observes the urgent
# lane idle, HCC's observe-before-in-flight-decrement ordering guarantees the
# wake observation is durable and therefore visible to the second scrape.
# shellcheck disable=SC2329 # invoked by name via wait_until
_wake_urgent_evidence_recorded() {
local current_pod lifecycle_state handled_generation
local fence_in_flight pre_count post_count in_flight
current_pod="$(hcc_pod_name)"
[[ -n "$current_pod" && "$current_pod" == "$HCC_POD_WAKE" ]] || return 1
lifecycle_state="$(wake_lifecycle_state)"
[[ -n "$lifecycle_state" && "$lifecycle_state" != 'suspended' ]] || return 1
handled_generation="$(wake_handled_generation)"
wake_generation_is_handled "$WAKE_GENERATION" "$handled_generation" || return 1
scrape_hcc_metrics "$SCRAPE_WAKE_POST" "$HCC_POD_WAKE" || return 1
fence_in_flight="$(awk -v metric="${IN_FLIGHT_METRIC}" '
index($0, metric "{") == 1 && $0 ~ /lane="urgent"/ { total += $NF }
END { print total + 0 }
' "$SCRAPE_WAKE_POST")"
awk -v active="$fence_in_flight" 'BEGIN { exit !(active == 0) }' || return 1
# The wake histogram observation precedes the decrement seen by the fence.
# Re-scrape so the adjudicated histogram cannot predate that observation.
scrape_hcc_metrics "$SCRAPE_WAKE_POST" "$HCC_POD_WAKE" || return 1
pre_count="$(awk -v metric="${QUEUE_WAIT_METRIC}_bucket" '
index($0, metric "{") == 1 && $0 ~ /lane="urgent"/ && $0 ~ /le="\+Inf"/ { total += $NF }
END { print total + 0 }
' "$SCRAPE_WAKE_PRE")"
post_count="$(awk -v metric="${QUEUE_WAIT_METRIC}_bucket" '
index($0, metric "{") == 1 && $0 ~ /lane="urgent"/ && $0 ~ /le="\+Inf"/ { total += $NF }
END { print total + 0 }
' "$SCRAPE_WAKE_POST")"
in_flight="$(awk -v metric="${IN_FLIGHT_METRIC}" '
index($0, metric "{") == 1 && $0 ~ /lane="urgent"/ { total += $NF }
END { print total + 0 }
' "$SCRAPE_WAKE_POST")"
awk -v before="$pre_count" -v after="$post_count" -v active="$in_flight" \
'BEGIN { exit !((after - before) >= 1 && active == 0) }'
}
if wait_until "$STORM_WAKE_QUEUE_EVIDENCE_DEADLINE_S" 1 \
"supported wake urgent reconcile + same-pod queue evidence" \
_wake_urgent_evidence_recorded _wake_target_diag; then
ok "A2 wake input: supported wake reconcile completed and published same-pod urgent queue evidence"
else
fail "A2 wake input: wake did not complete with durable same-pod urgent queue evidence within ${STORM_WAKE_QUEUE_EVIDENCE_DEADLINE_S}s"
print_results; exit 1
fi
# Capture A7's fleet-failure baseline immediately before the kill. The wake
# scrape remains the A2 wake-window oracle; a separate fresh scrape prevents
# fleet failures after wake adjudication from falling outside A7's delta.
HCC_POD_B="$(hcc_pod_name)"
if [[ "$HCC_POD_B" != "$HCC_POD_A" ]]; then
fail "storm: HCC pod changed unexpectedly before the kill (${HCC_POD_A} -> ${HCC_POD_B:-<none>}) — counter deltas would be invalid"
storm_diagnostics; print_results; exit 1
fi
if ! scrape_hcc_metrics "$SCRAPE_B" "$HCC_POD_B"; then
fail "storm: could not capture the fresh pre-kill metric baseline"
print_results; exit 1
fi
log "storm: pre-kill scrape B captured (pod ${HCC_POD_B})"
T_KILL_MS="$(now_ms)"
kctl -n "$CONTROL_NS" delete pod "$HCC_POD_B" --wait=false >/dev/null || {
fail "storm: could not delete HCC pod ${HCC_POD_B}"; print_results; exit 1; }
log "storm: HCC pod ${HCC_POD_B} kill issued (forces LIST->WATCH recovery under load)"
i=0
for h in "${STORM_CREATED_HOSTS[@]}"; do
i=$((i + 1))
[[ $i -le $STORM_HALF ]] && continue
if (( i % 2 == 0 )); then apply_storm_host "$h" true; else apply_storm_host "$h" false; fi
done
T_CREATE_LAST_MS="$(now_ms)"
log "storm: remaining $(( STORM_CREATE_COUNT - STORM_HALF )) Host creations issued"
# ─── A1: concurrency evidence ─────────────────────────────────────────
header "A1 — storm concurrency (recorded start timestamps)"
printf ' %-28s %s\n' 'ingredient' 'start (epoch-ms)'
printf ' %-28s %s\n' 'create[first]' "$T_CREATE_FIRST_MS"
printf ' %-28s %s\n' 'delete[batch]' "$T_DELETE_MS"
printf ' %-28s %s\n' 'wake[POST]' "$T_WAKE_MS"
printf ' %-28s %s\n' 'hcc-kill' "$T_KILL_MS"
printf ' %-28s %s\n' 'create[last]' "$T_CREATE_LAST_MS"
if [[ "$T_CREATE_FIRST_MS" -lt "$T_DELETE_MS" && "$T_DELETE_MS" -lt "$T_WAKE_MS" \
&& "$T_WAKE_MS" -lt "$T_KILL_MS" && "$T_KILL_MS" -lt "$T_CREATE_LAST_MS" ]]; then
ok "A1: deletes, wake, and HCC kill all started strictly inside the creation span (genuinely overlapped, not sequential phases)"
else
fail "A1: storm ingredient timestamps are not interleaved (first=${T_CREATE_FIRST_MS} delete=${T_DELETE_MS} wake=${T_WAKE_MS} kill=${T_KILL_MS} last=${T_CREATE_LAST_MS})"
print_results; exit 1
fi
# ======================================================================
# Convergence — replacement HCC pod, then all created Hosts' Deployments
# ======================================================================
header "Convergence — replacement HCC pod + created-Host materialization"
# shellcheck disable=SC2329 # invoked by name via wait_until
_new_hcc_pod_probe() {
local name; name="$(hcc_pod_name)"
[[ -n "$name" && "$name" != "$HCC_POD_B" ]]
}
if wait_until "$STORM_HCC_READY_DEADLINE_S" 2 "replacement HCC pod after the kill" \
_new_hcc_pod_probe storm_diagnostics; then
HCC_POD_C="$(hcc_pod_name)"
ok "replacement HCC pod Running: ${HCC_POD_C}"
else
fail "no replacement HCC pod within ${STORM_HCC_READY_DEADLINE_S}s of the kill"
print_results; exit 1
fi
# A7's causal business signal is the replacement pod's own completion log for
# its startup fleet reconciliation. A scheduled/started counter is insufficient:
# it can increment before the pass fails, stalls, or is superseded.
FLEET_COMPLETION_PROVEN=0
replacement_fleet_state=''
# shellcheck disable=SC2329 # invoked by name via wait_until
_replacement_fleet_completed_probe() {
local logs current_pod
current_pod="$(hcc_pod_name)"
[[ -n "$current_pod" && "$current_pod" == "$HCC_POD_C" ]] || return 1
logs="$(kctl -n "$CONTROL_NS" logs "pod/${HCC_POD_C}" \
-c host-context-controller 2>/dev/null)" || return 1
if grep -Fq -- "$HCC_PASS_FAILED_MARKER" <<<"$logs"; then
replacement_fleet_state='failed'
return 0
fi
if grep -Fq -- "$HCC_PASS_COMPLETED_MARKER" <<<"$logs"; then
replacement_fleet_state='completed'
return 0
fi
return 1
}
if ! wait_until "$STORM_CONVERGENCE_DEADLINE_S" 2 \
"replacement HCC initial Host fleet reconciliation completion" \
_replacement_fleet_completed_probe storm_diagnostics; then
fail "A7: replacement pod did not complete its initial Host fleet reconciliation within ${STORM_CONVERGENCE_DEADLINE_S}s"
print_results; exit 1
fi
if [[ "$replacement_fleet_state" != 'completed' ]]; then
fail "A7: replacement pod's initial Host fleet reconciliation failed"
storm_diagnostics; print_results; exit 1
fi
FLEET_COMPLETION_PROVEN=1
ok "A7 input: replacement pod logged completed initial Host fleet reconciliation"
# shellcheck disable=SC2329 # invoked by name via wait_until
_all_created_deploys_probe() {
local h
for h in "${STORM_CREATED_HOSTS[@]}"; do
kctl -n "$MCP_HOST_NS" get deployment "$h" >/dev/null 2>&1 || return 1
done
}
if wait_until "$STORM_CONVERGENCE_DEADLINE_S" 3 \
"all ${STORM_CREATE_COUNT} created Hosts' Deployments to exist" \
_all_created_deploys_probe storm_diagnostics; then
ok "all ${STORM_CREATE_COUNT} created Hosts have Deployments"
else
fail "created-Host Deployments incomplete after ${STORM_CONVERGENCE_DEADLINE_S}s — starvation or lost events"
print_results; exit 1
fi
# ─── Scrape C1 (M2): taken IMMEDIATELY after created-Host convergence so
# post-storm quiet-period samples cannot dilute the urgent p95 (A2) —
# recovery (A4) necessarily completed before any post-kill Deployment
# materialized. A6/A7 counters are read later from scrape C2, after the
# A5 cleanup convergence, so their outcome counters are settled. ─────────
header "Metrics — early post-convergence scrape C1 (replacement pod)"
HCC_POD_C_NOW="$(hcc_pod_name)"
if [[ -z "$HCC_POD_C_NOW" || "$HCC_POD_C_NOW" != "$HCC_POD_C" ]]; then
fail "replacement HCC pod changed again before scrape C1 (${HCC_POD_C} -> ${HCC_POD_C_NOW:-<none>}) — an unexpected second restart invalidates the counter windows"
storm_diagnostics; print_results; exit 1
fi
if ! scrape_hcc_metrics "$SCRAPE_C1" "$HCC_POD_C"; then
fail "could not scrape HCC metrics (C1) on replacement pod ${HCC_POD_C}"; print_results; exit 1
fi
log "scrape C1 captured (replacement pod ${HCC_POD_C})"
# ─── A2 input: direct-watch urgent probe after replacement LIST→WATCH ──
# The storm's second half is intentionally issued immediately after pod
# deletion, when a cold-start LIST may legitimately own those Hosts on the
# fleet lane. C1 proves the replacement has completed LIST→WATCH recovery;
# issue a fresh, independent set now so every observed sample is a direct
# watch ADDED admission on the urgent lane.
header "A2 input — urgent direct-watch probe (${STORM_URGENT_PROBE_COUNT} Hosts)"
for i in $(seq 1 "${#STORM_URGENT_PROBE_HOSTS[@]}"); do
h="${STORM_URGENT_PROBE_HOSTS[$((i - 1))]}"
if (( i % 2 == 0 )); then apply_storm_host "$h" true; else apply_storm_host "$h" false; fi
done
log "urgent probe: ${STORM_URGENT_PROBE_COUNT} Host ADDED events issued after replacement LIST→WATCH"
# shellcheck disable=SC2329 # invoked by name via wait_until
_urgent_probe_deploys_probe() {
local h
for h in "${STORM_URGENT_PROBE_HOSTS[@]}"; do
kctl -n "$MCP_HOST_NS" get deployment "$h" >/dev/null 2>&1 || return 1
done
}
if wait_until "$STORM_CONVERGENCE_DEADLINE_S" 3 \
"all ${STORM_URGENT_PROBE_COUNT} urgent-probe Hosts' Deployments to exist" \
_urgent_probe_deploys_probe storm_diagnostics; then
ok "A2 input: every direct-watch urgent-probe Host materialized"
else
fail "A2 input: urgent-probe Host materialization incomplete after ${STORM_CONVERGENCE_DEADLINE_S}s"
print_results; exit 1
fi
HCC_POD_C_NOW="$(hcc_pod_name)"
if [[ -z "$HCC_POD_C_NOW" || "$HCC_POD_C_NOW" != "$HCC_POD_C" ]]; then
fail "replacement HCC pod changed before urgent-probe scrape D (${HCC_POD_C} -> ${HCC_POD_C_NOW:-<none>}) — urgent counter window is invalid"
storm_diagnostics; print_results; exit 1
fi
# Deployment existence can precede the end of reconcileCore, where the queue
# histogram is observed. Poll the metric itself instead of sleeping, otherwise
# a fast API server can make this gate race its own telemetry.
# shellcheck disable=SC2329 # invoked by name via wait_until
_urgent_probe_metrics_recorded() {
local current_pod pre_count post_count
current_pod="$(hcc_pod_name)"
[[ -n "$current_pod" && "$current_pod" == "$HCC_POD_C" ]] || return 1
scrape_hcc_metrics "$SCRAPE_D" "$HCC_POD_C" || return 1
pre_count="$(awk -v metric="${QUEUE_WAIT_METRIC}_bucket" '
index($0, metric "{") == 1 && $0 ~ /lane="urgent"/ && $0 ~ /le="\+Inf"/ { total += $NF }
END { print total + 0 }
' "$SCRAPE_C1")"
post_count="$(awk -v metric="${QUEUE_WAIT_METRIC}_bucket" '
index($0, metric "{") == 1 && $0 ~ /lane="urgent"/ && $0 ~ /le="\+Inf"/ { total += $NF }
END { print total + 0 }
' "$SCRAPE_D")"
awk -v before="$pre_count" -v after="$post_count" -v required="$STORM_URGENT_PROBE_COUNT" \
'BEGIN { exit !((after - before) >= required) }'
}
if ! wait_until 60 2 \
"at least ${STORM_URGENT_PROBE_COUNT} direct-watch urgent metric samples" \
_urgent_probe_metrics_recorded storm_diagnostics; then
fail "urgent-probe reconciles did not publish ${STORM_URGENT_PROBE_COUNT} queue samples within 60s"
print_results; exit 1
fi
log "scrape D captured (replacement pod ${HCC_POD_C}; direct-watch urgent window C1→D)"
# ─── A3: per-host materialization <= 20s from ITS OWN creation ───────
header "A3 — per-host materialization (budget ${STORM_MATERIALIZATION_BUDGET_S}s, §14.2)"
MAT_ROWS="${WORK_DIR}/materialization.rows"
: >"$MAT_ROWS"
for h in "${STORM_CREATED_HOSTS[@]}"; do
hc="$(kctl -n "$MCP_HOST_NS" get host "$h" -o jsonpath='{.metadata.creationTimestamp}' 2>/dev/null || true)"
dc="$(kctl -n "$MCP_HOST_NS" get deployment "$h" -o jsonpath='{.metadata.creationTimestamp}' 2>/dev/null || true)"
if [[ -z "$hc" || -z "$dc" ]]; then
fail "A3: missing creationTimestamp for ${h} (host='${hc}' deploy='${dc}')"
print_results; exit 1
fi
printf '%s %s %s\n' "$h" "$hc" "$dc" >>"$MAT_ROWS"
done
if MAT_ROWS="$MAT_ROWS" BUDGET_S="$STORM_MATERIALIZATION_BUDGET_S" python3 <<'PY'
import datetime, os, sys
def ms(v):
return int(datetime.datetime.fromisoformat(v.replace("Z", "+00:00")).timestamp() * 1000)
budget = int(os.environ["BUDGET_S"])
rows = []
for line in open(os.environ["MAT_ROWS"], encoding="utf-8"):
host, host_ts, dep_ts = line.split()
delta = round((ms(dep_ts) - ms(host_ts)) / 1000.0, 3)
rows.append((host, delta))
print(f" {host}: Host->Deployment {delta:.1f}s ({'within' if delta <= budget else 'OVER_BUDGET'})")
worst = max(rows, key=lambda r: r[1])
print(f" worst case: {worst[0]} at {worst[1]:.1f}s (budget {budget}s)")
offenders = [r for r in rows if r[1] > budget]
if offenders:
print(" OVER BUDGET: " + ", ".join(f"{h}={d:.1f}s" for h, d in offenders))
sys.exit(1)
PY
then
ok "A3: every created Host materialized <= ${STORM_MATERIALIZATION_BUDGET_S}s from its own creation (no starvation)"
else
fail "A3: at least one created Host exceeded the ${STORM_MATERIALIZATION_BUDGET_S}s materialization budget (offenders listed above)"
storm_diagnostics; print_results; exit 1
fi
# ─── A8: wake ingredient integrity ────────────────────────────────────
header "A8 — mid-storm wake converges (bound ${STORM_WAKE_READY_BUDGET_S}s)"
# shellcheck disable=SC2329 # invoked by name via wait_until
_wake_converged_probe() {
local st; st="$(wake_lifecycle_state)"
[[ -n "$st" && "$st" != 'suspended' ]] || return 1
wake_ready_pod_exists
}
wake_elapsed_s=$(( ( $(now_ms) - T_WAKE_MS ) / 1000 ))
wake_remaining_s=$(( STORM_WAKE_READY_BUDGET_S - wake_elapsed_s ))
[[ "$wake_remaining_s" -lt 1 ]] && wake_remaining_s=1
if wait_until "$wake_remaining_s" 3 "woken host ${WAKE_HOST_REF} to leave suspended + reach a Ready pod" \
_wake_converged_probe _wake_target_diag; then
ok "A8: wake converged $(( ( $(now_ms) - T_WAKE_MS ) / 1000 ))s after the wake POST (state=$(wake_lifecycle_state))"
else
fail "A8: woken host did not reach Ready within ${STORM_WAKE_READY_BUDGET_S}s of the wake POST (state=$(wake_lifecycle_state)) — the wake ingredient did not complete"
print_results; exit 1
fi
# ─── A5: deleted bundles fully cleaned (ownership-label leftover check) ──
header "A5 — deleted-Host bundle cleanup across ${BUNDLE_NAMESPACES}"
# FAIL-CLOSED (B2): the probe succeeds ONLY on positive evidence — the CR
# absence must be a confirmed NotFound and every leftover list must have
# been readable. A kubectl error keeps the probe failing, so an API outage
# ends in the bounded deadline FAILING (with the read errors on stderr),
# never in a false "cleanup complete".
# shellcheck disable=SC2329 # invoked by name via wait_until
_deleted_hosts_clean_probe() {
local h leftovers
for h in "${STORM_DELETE_HOSTS[@]}"; do
host_absent_confirmed "$h" || return 1
leftovers="$(collect_children "$h")" || return 1
[[ -z "$leftovers" ]] || return 1
done
}
# shellcheck disable=SC2329 # invoked by name via wait_until
_deleted_hosts_diag() {
local h
{
echo "----- deleted-host leftover diagnostics -----"
for h in "${STORM_DELETE_HOSTS[@]}"; do
echo "host ${h}:"
kctl -n "$MCP_HOST_NS" get host "$h" -o wide 2>&1 || true
collect_children "$h" 2>&1 || true
done
kctl -n "$CONTROL_NS" logs "deploy/${HCC_DEPLOY}" --tail=60 2>&1 || true
} >&2
}
if wait_until "$STORM_CLEANUP_DEADLINE_S" 3 \
"deleted Hosts' CRs + owned bundles to disappear" \
_deleted_hosts_clean_probe _deleted_hosts_diag; then
ok "A5: every deleted Host's CR and owned bundle fully cleaned (confirmed NotFound + zero ownership-labeled leftovers in ${BUNDLE_NAMESPACES})"
else
fail "A5: deleted-Host cleanup incomplete OR unverifiable after ${STORM_CLEANUP_DEADLINE_S}s (leftovers/read failures listed above) — lost delete, cleanup regression, or an API read that could not be verified (fail-closed)"
print_results; exit 1
fi
# ======================================================================
# Metrics adjudication (scrape C2 after cleanup convergence, then A2/A4/A6/A7)
# ======================================================================
header "Metrics — settled scrape C2 + §14.1 adjudication"
HCC_POD_C_NOW="$(hcc_pod_name)"
if [[ -z "$HCC_POD_C_NOW" || "$HCC_POD_C_NOW" != "$HCC_POD_C" ]]; then
fail "replacement HCC pod changed between scrapes C1 and C2 (${HCC_POD_C} -> ${HCC_POD_C_NOW:-<none>}) — an unexpected second restart invalidates the counter windows"
storm_diagnostics; print_results; exit 1
fi
if ! scrape_hcc_metrics "$SCRAPE_C2" "$HCC_POD_C"; then
fail "could not scrape HCC metrics (C2) on replacement pod ${HCC_POD_C}"; print_results; exit 1
fi
log "scrape C2 captured (replacement pod ${HCC_POD_C}; A6/A7 counters settled post-A5)"
# One python pass performs the five metric assertions and emits one
# VERDICT|<name>|PASS/FAIL|<detail> line each. python exits nonzero only
# when it cannot parse its inputs (a hard failure of the gate itself).
METRIC_VERDICTS="${WORK_DIR}/metric-verdicts.txt"
if ! SCRAPE_A="$SCRAPE_A" SCRAPE_B="$SCRAPE_B" SCRAPE_C1="$SCRAPE_C1" SCRAPE_C2="$SCRAPE_C2" SCRAPE_D="$SCRAPE_D" \
SCRAPE_WAKE_PRE="$SCRAPE_WAKE_PRE" SCRAPE_WAKE_POST="$SCRAPE_WAKE_POST" \
P95_BUDGET_S="$STORM_URGENT_P95_BUDGET_S" RECOVERY_BUDGET_S="$STORM_WATCH_RECOVERY_BUDGET_S" \
MIN_URGENT_SAMPLES="$STORM_URGENT_PROBE_COUNT" FLEET_COMPLETION_PROVEN="$FLEET_COMPLETION_PROVEN" \
python3 >"$METRIC_VERDICTS" <<'PY'
import os, re, sys
from collections import defaultdict
LINE = re.compile(r'^([a-zA-Z_:][a-zA-Z0-9_:]*)(\{[^}]*\})?\s+(\S+)')
LABEL = re.compile(r'(\w+)="([^"]*)"')
def load(path):
samples, types = [], set()
for line in open(path, encoding="utf-8", errors="replace"):
if line.startswith("# TYPE "):
types.add(line.split()[2])
continue
if line.startswith("#"):
continue
m = LINE.match(line)
if not m:
continue
labels = dict(LABEL.findall(m.group(2) or ""))
try:
value = float(m.group(3))
except ValueError:
continue
samples.append((m.group(1), labels, value))
return samples, types
# Two replacement-pod windows (M2): C1 is scraped immediately after
# created-Host convergence (A2 p95 undiluted by quiet-period samples; A4
# recovery necessarily recorded by then); C2 is scraped after the A5
# cleanup convergence (A6/A7 outcome counters settled). Pod identity for
# A/B and C1/C2 is guarded in bash before this adjudication runs.
a, _ = load(os.environ["SCRAPE_A"])
b, _ = load(os.environ["SCRAPE_B"])
c1, _ = load(os.environ["SCRAPE_C1"])
c2, c2_types = load(os.environ["SCRAPE_C2"])
d, _ = load(os.environ["SCRAPE_D"])
wake_pre, _ = load(os.environ["SCRAPE_WAKE_PRE"])
wake_post, _ = load(os.environ["SCRAPE_WAKE_POST"])
def buckets(samples, metric, match):
"""Cumulative histogram buckets {le: count} summed over matching series."""
out = defaultdict(float)
for name, labels, value in samples:
if name != metric + "_bucket":
continue
if any(labels.get(k) != v for k, v in match.items()):
continue
out[labels.get("le", "")] += value
return out
def counter(samples, metric, label, value):
total = 0.0
for name, labels, val in samples:
if name == metric and labels.get(label) == value:
total += val
return total
def le_key(le):
return float("inf") if le == "+Inf" else float(le)
verdicts = []
# ── A2a: every completed urgent reconcile in the supported wake window ─
qm = os.environ["QUEUE_WAIT_METRIC"]
budget = float(os.environ["P95_BUDGET_S"])
wake_before = buckets(wake_pre, qm, {"lane": "urgent"})
wake_after = buckets(wake_post, qm, {"lane": "urgent"})
wake_delta = defaultdict(float)
wake_reset = False
for le in set(wake_before) | set(wake_after):
delta = wake_after.get(le, 0.0) - wake_before.get(le, 0.0)
if delta < 0:
wake_reset = True
wake_delta[le] = delta
if wake_reset:
verdicts.append(("A2-wake-urgent-hard-gate", False,
"negative bucket delta in the same-pod wake window"))
else:
wake_total = wake_delta.get("+Inf", 0.0)
wake_within = wake_delta.get(str(int(budget)), wake_delta.get(str(budget), 0.0))
wake_dist = " ".join(
f"le{le}={wake_delta[le]:.0f}" for le in sorted(wake_delta, key=le_key)
)
if wake_total < 1:
verdicts.append(("A2-wake-urgent-hard-gate", False,
"no completed urgent queue sample in the supported wake window"))
elif wake_within != wake_total:
verdicts.append(("A2-wake-urgent-hard-gate", False,
f"{wake_total - wake_within:.0f} of {wake_total:.0f} urgent sample(s) exceeded "
f"{budget:.0f}s in the supported wake window ({wake_dist})"))
else:
verdicts.append(("A2-wake-urgent-hard-gate", True,
f"all {wake_total:.0f} urgent sample(s) <= {budget:.0f}s in the supported "
f"same-pod wake window ({wake_dist})"))
# ── A2b: urgent queue-wait p95 across direct-watch window D-C1 ────────
window_note = "window: replacement-pod direct-watch probe delta (D-C1) after LIST→WATCH recovery"
pre = buckets(c1, qm, {"lane": "urgent"})
post = buckets(d, qm, {"lane": "urgent"})
combined = defaultdict(float)
reset = False
for le in set(pre) | set(post):
delta = post.get(le, 0.0) - pre.get(le, 0.0)
if delta < 0:
reset = True
combined[le] = delta
if reset:
verdicts.append(("A2-direct-watch-urgent-p95", False,
"negative bucket delta between scrapes C1 and D (unexpected counter reset)"))
else:
total = combined.get("+Inf", 0.0)
min_samples = float(os.environ["MIN_URGENT_SAMPLES"])
if total < min_samples:
verdicts.append(("A2-direct-watch-urgent-p95", False,
f"only {total:.0f} direct-watch urgent samples (need >= {min_samples:.0f} — "
"the urgent probe did not exercise its declared lane)"))
else:
p95_ub = None
for le in sorted(combined, key=le_key):
if total > 0 and combined[le] / total >= 0.95:
p95_ub = le
break
dist = " ".join(f"le{le}={combined[le]:.0f}" for le in sorted(combined, key=le_key))
if p95_ub is not None and le_key(p95_ub) <= budget:
verdicts.append(("A2-direct-watch-urgent-p95", True,
f"p95 <= {p95_ub}s over {total:.0f} samples (budget {budget:.0f}s; {dist}; {window_note})"))
else:
verdicts.append(("A2-direct-watch-urgent-p95", False,
f"p95 bucket upper bound {p95_ub} exceeds {budget:.0f}s over {total:.0f} samples ({dist}; {window_note})"))
# ── A4: watch recovery <= budget on the replacement pod (C1) ──────────
wm = os.environ["WATCH_RECOVERY_METRIC"]
rbudget = os.environ["RECOVERY_BUDGET_S"]
succ = buckets(c1, wm, {"phase": "total", "outcome": "success"})
failed_total = buckets(c1, wm, {"phase": "total", "outcome": "failure"}).get("+Inf", 0.0)
succ_total = succ.get("+Inf", 0.0)
within = succ.get(rbudget, 0.0)
if failed_total > 0:
verdicts.append(("A4-watch-recovery", False,
f"{failed_total:.0f} failed recovery attempt(s) on the replacement pod"))
elif succ_total < 1:
verdicts.append(("A4-watch-recovery", False,
"no successful LIST->WATCH recovery sample on the replacement pod"))
elif within != succ_total:
verdicts.append(("A4-watch-recovery", False,
f"{succ_total - within:.0f} of {succ_total:.0f} recovery sample(s) exceeded {rbudget}s"))
else:
bound = next((le for le in sorted(succ, key=le_key) if succ[le] == succ_total), "+Inf")
verdicts.append(("A4-watch-recovery", True,
f"{succ_total:.0f} recovery sample(s), all <= {bound}s (budget {rbudget}s)"))
# ── A7: recovered fleet convergence remains bounded (settled C2) ─────
fm = os.environ["FLEET_METRIC"]
# Presence check (A6 pattern): counter() returns 0 for an absent family, so
# without this a renamed/unregistered FLEET_METRIC makes every count 0 and A7
# passes vacuously. The causal leg (FLEET_COMPLETION_PROVEN) alone cannot catch
# a metric rename — the metric itself must be registered on the running image.
if fm not in c2_types:
verdicts.append(("A7-fleet-recovery-bounded", False,
f"{fm} not registered on the running image — is the branch HCC image deployed?"))
else:
pre_failed = counter(b, fm, "result", "failed") - counter(a, fm, "result", "failed")
post_started = counter(c2, fm, "result", "started")
post_coalesced = counter(c2, fm, "result", "coalesced")
post_trailing = counter(c2, fm, "result", "trailing")
post_failed = counter(c2, fm, "result", "failed")
problems = []
if pre_failed != 0 or post_failed > 0:
problems.append(f"fleet pass failures (pre={pre_failed:.0f} post={post_failed:.0f})")
if os.environ.get("FLEET_COMPLETION_PROVEN") != "1":
problems.append("replacement-pod initial fleet completion was not causally proven")
if post_trailing > post_coalesced:
problems.append(f"trailing={post_trailing:.0f} > coalesced={post_coalesced:.0f} (violates the single-pending-slot structure)")
detail = (f"completion-proven={os.environ.get('FLEET_COMPLETION_PROVEN', 'missing')} "
f"started={post_started:.0f} coalesced={post_coalesced:.0f} "
f"trailing={post_trailing:.0f} failed={post_failed:.0f} "
f"(replacement pod; A4 recovery-success fenced; pre-kill failed delta={pre_failed:.0f})")
if problems:
verdicts.append(("A7-fleet-recovery-bounded", False, "; ".join(problems) + f" [{detail}]"))
else:
verdicts.append(("A7-fleet-recovery-bounded", True, detail))
# ── A6: delete-cleanup counters consistent (settled window C2) ────────
dm = os.environ["DELETE_CLEANUP_METRIC"]
if dm not in c2_types:
verdicts.append(("A6-delete-cleanup-counters", False,
f"{dm} not registered on the running image — is the branch HCC image deployed?"))
else:
queued = counter(c2, dm, "outcome", "queued")
confirmed = counter(c2, dm, "outcome", "confirmed")
completed = counter(c2, dm, "outcome", "completed")
retried = counter(c2, dm, "outcome", "retried")
superseded = counter(c2, dm, "outcome", "superseded")
detail = (f"queued={queued:.0f} confirmed={confirmed:.0f} completed={completed:.0f} "
f"retried={retried:.0f} superseded={superseded:.0f}")
if retried == 0 and superseded == 0 and queued == confirmed == completed:
note = "" if queued > 0 else " (all-zero: deletes handled by the direct watch-DELETE path; A5 proved cleanup)"
verdicts.append(("A6-delete-cleanup-counters", True, detail + note))
else:
verdicts.append(("A6-delete-cleanup-counters", False,
f"inconsistent outcomes — lost or failed delete cleanup [{detail}]"))
for name, passed, detail in verdicts:
print(f"VERDICT|{name}|{'PASS' if passed else 'FAIL'}|{detail}")
PY
then
fail "metric adjudication crashed (could not parse the §14.1 scrapes) — see ${WORK_DIR}"
print_results; exit 1
fi
VERDICT_COUNT=0
while IFS='|' read -r tag name verdict detail; do
[[ "$tag" == 'VERDICT' ]] || continue
VERDICT_COUNT=$((VERDICT_COUNT + 1))
if [[ "$verdict" == 'PASS' ]]; then
ok "${name}: ${detail}"
else
fail "${name}: ${detail}"
fi
done <"$METRIC_VERDICTS"
if [[ "$VERDICT_COUNT" -ne 5 ]]; then
fail "metric adjudication emitted ${VERDICT_COUNT}/5 verdicts — truncated output is a failure, not a pass"
fi
# ======================================================================
# Teardown — delete ONLY this run's fixtures; fail-loud leftover check.
# The profile and seeded data (including the wake target) stay.
# ======================================================================
header "Teardown — storm fixtures (${FIXTURE_RUN_KEY}=${RUN_ID})"
STORM_SEL="${FIXTURE_LABEL_KEY}=${FIXTURE_LABEL_VAL},${FIXTURE_RUN_KEY}=${RUN_ID}"
kctl -n "$MCP_HOST_NS" delete host -l "$STORM_SEL" --wait=false >/dev/null 2>&1 || \
warn "teardown: label-selector Host deletion returned nonzero (leftover check below is authoritative)"
kctl -n "$MCP_HOST_NS" delete secret -l "$STORM_SEL" --wait=false >/dev/null 2>&1 || \
warn "teardown: label-selector Secret deletion returned nonzero (leftover check below is authoritative)"
# FAIL-CLOSED (B2): a kubectl list failure must never read as "all
# gone" — the probe fails (with the error on stderr) until the API
# positively returns an empty list, and the bounded deadline then FAILS.
# shellcheck disable=SC2329 # invoked by name via wait_until
_storm_hosts_gone_probe() {
local out
if ! out="$(kctl -n "$MCP_HOST_NS" get host -l "$STORM_SEL" -o name 2>"${WORK_DIR}/hosts-gone.err")"; then
echo "_storm_hosts_gone_probe: kubectl list FAILED: $(cat "${WORK_DIR}/hosts-gone.err" 2>/dev/null)" >&2
return 1
fi
[[ -z "$out" ]]
}
if wait_until "$STORM_TEARDOWN_DEADLINE_S" 3 "storm fixture Hosts (${STORM_SEL}) deletion" \
_storm_hosts_gone_probe storm_diagnostics; then
ok "teardown: all storm fixture Hosts deleted via the supported CR path"
else
fail "teardown: storm fixture Hosts still present after ${STORM_TEARDOWN_DEADLINE_S}s (run-id ${RUN_ID})"
TEARDOWN_CLEANUP_FAILED=1
fi
# FAIL-CLOSED (B2): an unverifiable read counts as a leftover — the teardown
# check may only pass on positive evidence of absence. Convergence-aware: a
# leftover with a deletionTimestamp is CONVERGING (its delete WAS issued; a
# finalizer like a workspace PVC's kubernetes.io/pvc-protection is draining for
# the deleting pod's grace) — allow it a bounded extension. A leftover with NO
# deletionTimestamp means the delete was NEVER issued (a real cleanup gap) —
# fail immediately, no wait. An unverifiable read is fail-closed.
TEARDOWN_LEFTOVER=0
teardown_converge_deadline=$((SECONDS + STORM_TEARDOWN_CONVERGE_S))
while :; do
TEARDOWN_LEFTOVER=0; teardown_orphan=0; teardown_unverifiable=0; teardown_report=''
for h in "${STORM_CREATED_HOSTS[@]}" "${STORM_URGENT_PROBE_HOSTS[@]}" "${STORM_DELETE_HOSTS[@]}"; do
if ! leftovers="$(collect_children "$h")"; then
teardown_report+=" ${h}: kubectl read FAILED (unverifiable, fail-closed)"$'\n'
TEARDOWN_LEFTOVER=$((TEARDOWN_LEFTOVER + 1)); teardown_unverifiable=1
continue
fi
if [[ -z "$leftovers" ]]; then
continue
fi
while IFS= read -r res; do
if [[ -z "$res" ]]; then
continue
fi
r_ns="${res%%/*}"; r_kn="${res#*/}"
if dts="$(kctl -n "$r_ns" get "$r_kn" -o jsonpath='{.metadata.deletionTimestamp}' 2>/dev/null)"; then
if [[ -n "$dts" ]]; then
teardown_report+=" ${res} — Terminating since ${dts} (converging)"$'\n'
TEARDOWN_LEFTOVER=$((TEARDOWN_LEFTOVER + 1))
else
teardown_report+=" ${res} — NO deletionTimestamp (delete never issued — REAL cleanup gap)"$'\n'
TEARDOWN_LEFTOVER=$((TEARDOWN_LEFTOVER + 1)); teardown_orphan=1
fi
fi
# get failed (NotFound between collect and read) => converged; not a leftover
done <<< "$leftovers"
done
if [[ "$TEARDOWN_LEFTOVER" -eq 0 ]]; then
break
fi
# A never-issued delete or an unverifiable read is a hard finding — do not wait.
if [[ "$teardown_orphan" -eq 1 || "$teardown_unverifiable" -eq 1 ]]; then
break
fi
if [[ "$SECONDS" -ge "$teardown_converge_deadline" ]]; then
break
fi
sleep 3
done
if [[ "$TEARDOWN_LEFTOVER" -gt 0 ]]; then
warn "teardown: ownership-labeled leftovers after up to ${STORM_TEARDOWN_CONVERGE_S}s convergence extension:"
printf '%s' "$teardown_report" >&2
fi
if ! sec_out="$(kctl -n "$MCP_HOST_NS" get secret -l "$STORM_SEL" -o name 2>"${WORK_DIR}/secret-gone.err")"; then
warn "teardown: could not verify fixture Secret deletion: $(cat "${WORK_DIR}/secret-gone.err" 2>/dev/null) — counting as leftover"
TEARDOWN_LEFTOVER=$((TEARDOWN_LEFTOVER + 1))
elif [[ -n "$sec_out" ]]; then
warn "teardown: fixture Secret still present (${STORM_SEL})"
TEARDOWN_LEFTOVER=$((TEARDOWN_LEFTOVER + 1))
fi
if [[ "$TEARDOWN_LEFTOVER" -eq 0 ]]; then
ok "teardown: zero storm-fixture leftovers across ${BUNDLE_NAMESPACES} (profile and seeded data untouched)"
else
fail "teardown: ${TEARDOWN_LEFTOVER} storm fixture subject(s) left resources behind (run-id ${RUN_ID}) — investigate before trusting results"
TEARDOWN_CLEANUP_FAILED=1
fi
TEARDOWN_DONE=1
# M3: on the success path a failed cadence restore must FAIL the gate
# (restore_hcc_env_strict registers fail() and leaves HCC_ENV_SAVED for
# the trap's best-effort retry); the EXIT trap keeps the lenient restore.
restore_hcc_env_strict
if ! finalize_hcc_watch_gate_lock "$TEARDOWN_CLEANUP_FAILED" "$HCC_RESTORE_OK"; then
fail "shared HCC fault-injection lock was retained fail-closed; follow the printed repair instructions"
else
ok "exclusive shared HCC fault-injection lock finalized"
fi
print_results
if [[ "$g_fail" -gt 0 ]]; then
exit 1
fi
exit 0