Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
fix: judge gfsc-reader rollout by readiness, not the generation HCC rewrites
Alfredo Lopez6b421789aa82
diff --git a/.cursor/rules/minikube-t0-t1-t2.mdc b/.cursor/rules/minikube-t0-t1-t2.mdcindex e71767825..8c0997ed9 100644--- a/.cursor/rules/minikube-t0-t1-t2.mdc+++ b/.cursor/rules/minikube-t0-t1-t2.mdc@@ -52,7 +52,11 @@ repository — never mix its PRs or clusters into this lane.fail-loud. T1 restores branch-profile GFS credentials on exit, and`pre-gate-sync` provisions GFS serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true`and `GFS_RECOVER_ABANDONED_STATE=true` in every plan, so no manual repair- script sits between plan and verdict.+ script sits between plan and verdict. `settle-gfs-reader-rollout.sh` clears+ Ready-reader leftovers (claim, stale ReplicaSets, CrashLoopBackOff pods)+ first, and the `gfs-rollout-shim` PATH prefix replaces the reader+ `rollout status` wait with a readiness poll because HCC's gfsReconciler+ strips `restartedAt` and makes generation waits time out.- Evidence lives under ignored `.local-notes/infra/runs/`; never commit it.Full workflow: skill `.cursor/skills/minikube-t0-t1-t2/SKILL.md`.diff --git a/.cursor/rules/minikube-t2-harness.mdc b/.cursor/rules/minikube-t2-harness.mdcindex 114df0b9c..9eaa90c13 100644--- a/.cursor/rules/minikube-t2-harness.mdc+++ b/.cursor/rules/minikube-t2-harness.mdc@@ -43,8 +43,15 @@ Invariants that must never regress:NOLOGIN reader or a leftover `rollout-running` claim from a dead priorsetup. The T2 profile lock makes that abandoned-state recover safe.When `gfsc-reader` is already Ready, `settle-gfs-reader-rollout.sh`- marks that leftover claim ready before reconcile so a second- `rollout restart` cannot race HCC's gfsReconciler.+ marks that leftover claim ready before reconcile, scales leftover+ non-current unready reader ReplicaSets to 0, and deletes CrashLoopBackOff+ reader pods so a second `rollout restart` is not triggered and cannot race+ HCC's gfsReconciler. HCC strips the `restartedAt` annotation, so every+ harness GFS reconcile runs with the `scripts/minikube/gfs-rollout-shim`+ PATH prefix: the reader `rollout status` wait becomes the readiness poll+ in `scripts/minikube/wait-gfs-reader-ready.sh` (desired replicas Ready and+ no live non-terminating unready reader pod), never a generation wait.+ `pre-gate-sync` reader convergence uses the same readiness contract.- T1 stays fail-loud (no green on unavailable DSN or zero tests); the JSONreporter is the suite verdict, not a leftover Vitest process exit after acomplete green reporter. Role-reset suites keep the throwaway Postgres 16.diff --git a/.cursor/skills/minikube-t0-t1-t2/SKILL.md b/.cursor/skills/minikube-t0-t1-t2/SKILL.mdindex ae7cdb9b5..22962c6da 100644--- a/.cursor/skills/minikube-t0-t1-t2/SKILL.md+++ b/.cursor/skills/minikube-t0-t1-t2/SKILL.md@@ -95,6 +95,12 @@ T1 runs inside `make minikube-t2`, or explicitly viaand poison the T2 preflight. Do not run that restore by hand. T2full-reconcile reaches `full-setup.sh` before pre-gate-sync; that REUSE_DBpath must pass the same opt-in on both GFS reconcile calls.+- Every harness GFS reconcile first settles Ready-reader leftovers+ (`settle-gfs-reader-rollout.sh`: leftover claim, stale non-current+ ReplicaSets, CrashLoopBackOff pods) and runs with the `gfs-rollout-shim`+ PATH prefix, which replaces the reader `rollout status` wait with the+ readiness poll in `wait-gfs-reader-ready.sh` — HCC's gfsReconciler strips+ the `restartedAt` annotation, so a generation-based wait times out.## Step 4 — Verdict and evidence reportingdiff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.mdindex e0b8e40a2..fb15ced0b 100644--- a/.cursor/skills/minikube-t0-t1-t2/reference.md+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md@@ -66,7 +66,14 @@ serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true` and`GFS_RECOVER_ABANDONED_STATE=true` in every plan and restarts an unready`gfsc-reader` after restore. A Ready leftover `rollout-running` claim issettled by `scripts/minikube/settle-gfs-reader-rollout.sh` before-reconcile so HCC's gfsReconciler cannot race a second restart. Do not run+reconcile: it marks the claim ready, scales leftover non-current unready+reader ReplicaSets to 0, and deletes CrashLoopBackOff reader pods so they+re-read the restored Secret without waiting out kubelet backoff. HCC's+gfsReconciler strips the `restartedAt` annotation, so every harness GFS+reconcile runs with the `scripts/minikube/gfs-rollout-shim` PATH prefix:+the reader `rollout status` wait is replaced by the readiness poll in+`scripts/minikube/wait-gfs-reader-ready.sh` instead of chasing the template+generation HCC keeps rewriting. Do not run`reconcile-gfs-deploy-credentials.sh` or `kubectl rollout restartdeploy/gfsc-reader` by hand as a T2-repair step — re-run the entry point.diff --git a/AGENTS.md b/AGENTS.mdindex e36507789..ba679a8cb 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -99,7 +99,11 @@ has an unready required deployment, the orchestrator planner selectsbranch-profile GFS credentials on exit and `pre-gate-sync` provisions GFSserving with `GFS_RESTORE_ACTIVE_NOLOGIN=true` and`GFS_RECOVER_ABANDONED_STATE=true` in every plan, so no manual repair-script belongs between plan and verdict. The standalone preflight and+script belongs between plan and verdict. Harness GFS reconciles settle+Ready-reader leftovers first (`settle-gfs-reader-rollout.sh`) and wait on+reader readiness through the `gfs-rollout-shim` PATH prefix instead of a+generation-based `rollout status`, because HCC's gfsReconciler strips the+`restartedAt` annotation and makes that wait time out. The standalone preflight andthe final exact-head T2 check stay fail-loud on an unready deployment.Real PostgreSQL suites are opt-in in the ordinary testmatrix, but a T1 run that requires them must fail when the database/DSN isdiff --git a/CLAUDE.md b/CLAUDE.mdindex 5ce1871b5..2e1346eb0 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -24,7 +24,11 @@ bootstrapped profile with an unready deployment (never `PROFILE_UNHEALTHY`before a transition), T1 restores branch-profile GFS credentials on exit, and`pre-gate-sync` provisions GFS serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true`and `GFS_RECOVER_ABANDONED_STATE=true` in every plan; do not insert manual-repair scripts between runs. Follow the rule+repair scripts between runs. Harness GFS reconciles settle Ready-reader+leftovers first (`settle-gfs-reader-rollout.sh`) and judge reader rollout+completion by readiness via the `gfs-rollout-shim` PATH prefix, because+HCC's gfsReconciler strips the `restartedAt` annotation and a+generation-based `kubectl rollout status` would time out. Follow the rule`.cursor/rules/minikube-t0-t1-t2.mdc` and the skill`.cursor/skills/minikube-t0-t1-t2/SKILL.md` for the certification workflow.diff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.mdindex 959885f95..d2e28d08f 100644--- a/docs/testing/minikube-t2-runbook.md+++ b/docs/testing/minikube-t2-runbook.md@@ -142,14 +142,23 @@ the same contract as the standalone GFS T1 gate, plus resume of a leftover`rollout-running` claim from a timed-out prior setup. The T2 profile lockmakes that recover safe: the prior process is dead. When `gfsc-reader`is already Ready, `scripts/minikube/settle-gfs-reader-rollout.sh` marks-the leftover reader claim ready before reconcile so a second-`rollout restart` cannot race HCC's gfsReconciler (that patch loop-makes `kubectl rollout status` wait for a spec update until timeout).-`pre-gate-sync`+the leftover reader claim ready before reconcile, scales to 0 any leftover+non-current ReplicaSet that contributes no Ready pod (its live unready pod+would otherwise keep the stale-pod recovery pending forever), and deletes+CrashLoopBackOff reader pods so they re-read the restored Secret without+waiting out kubelet backoff. HCC's gfsReconciler owns the reader Deployment+template and strips the `restartedAt` annotation `kubectl rollout restart`+adds, so a generation-based `kubectl rollout status` chases flapping+revisions until timeout; every harness GFS reconcile therefore runs with the+`scripts/minikube/gfs-rollout-shim` PATH prefix, which intercepts exactly+the reader `rollout status` wait and judges readiness instead+(`scripts/minikube/wait-gfs-reader-ready.sh`: desired replicas Ready and no+live non-terminating unready reader pod). `pre-gate-sync`provisions GFS serving with the same opt-ins in every sync plan (including-"no cluster sync required") and restarts an unready `gfsc-reader` after a-successful restore. `full-setup.sh` on the REUSE_DB / T2 full-reconcile path-passes the same opt-ins on both GFS reconcile calls, because setup runs+"no cluster sync required") and, after a successful restore, restarts an+unready `gfsc-reader`, deletes its live unready pods once, and waits on the+same readiness contract. `full-setup.sh` on the REUSE_DB / T2 full-reconcile+path passes the same opt-ins on both GFS reconcile calls, because setup runsbefore pre-gate-sync and must not abort on a NOLOGIN reader or an abandonedreader rollout. The recovery helper restores aNOLOGIN role only from the committed Secret DSN and still fails loud whendiff --git a/scripts/e2e/minikube-real-postgres.sh b/scripts/e2e/minikube-real-postgres.shindex 4c4072538..b8993e846 100755--- a/scripts/e2e/minikube-real-postgres.sh+++ b/scripts/e2e/minikube-real-postgres.sh@@ -69,7 +69,16 @@ restore_gfs_runtime_credentials() {if ! t2_kc -n gfs get secret gfs-controller-db >/dev/null 2>&1; thenreturn 0fi- if ! GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \+ # Settle a Ready reader's leftover rollout claim first so restore does not+ # rollout restart it, and let any wait reconcile still needs judge+ # readiness via the gfs-rollout-shim instead of the template generation+ # HCC's gfsReconciler keeps rewriting.+ if ! CONTEXT="$T2_CONTEXT" \+ bash "$PROJECT_DIR/scripts/minikube/settle-gfs-reader-rollout.sh"; then+ printf '[minikube-t1] WARN: could not settle a leftover gfsc-reader rollout claim before restore\n' >&2+ fi+ if ! PATH="$PROJECT_DIR/scripts/minikube/gfs-rollout-shim:$PATH" \+ GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \CONTEXT="$T2_CONTEXT" \bash "$PROJECT_DIR/deploy/scripts/reconcile-gfs-deploy-credentials.sh"; thenprintf '[minikube-t1] ERROR: failed to restore branch-profile GFS credentials\n' >&2diff --git a/scripts/minikube/full-setup.sh b/scripts/minikube/full-setup.shindex 580e0ed5a..fb605ff41 100755--- a/scripts/minikube/full-setup.sh+++ b/scripts/minikube/full-setup.sh@@ -987,9 +987,14 @@ else# a manual retry flag between runs.# If gfsc-reader is already Ready, settle the leftover claim first so# reconcile does not rollout restart and race HCC's gfsReconciler.+ # The gfs-rollout-shim PATH prefix makes any reader rollout wait inside+ # reconcile judge readiness instead of the template generation HCC keeps+ # rewriting (it strips the restartedAt annotation, so a generation wait+ # loops until timeout).CONTEXT="${PROFILE}" \bash "${PROJECT_DIR}/scripts/minikube/settle-gfs-reader-rollout.sh"- GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \+ PATH="${PROJECT_DIR}/scripts/minikube/gfs-rollout-shim:${PATH}" \+ GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \CONTEXT="${PROFILE}" \bash "${PROJECT_DIR}/deploy/scripts/reconcile-gfs-deploy-credentials.sh"else@@ -1055,7 +1060,8 @@ elseCONTEXT="${PROFILE}" bash "${PROJECT_DIR}/deploy/scripts/wait-gfsc-secret-references.sh"CONTEXT="${PROFILE}" \bash "${PROJECT_DIR}/scripts/minikube/settle-gfs-reader-rollout.sh"- GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \+ PATH="${PROJECT_DIR}/scripts/minikube/gfs-rollout-shim:${PATH}" \+ GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \CONTEXT="${PROFILE}" \bash "${PROJECT_DIR}/deploy/scripts/reconcile-gfs-deploy-credentials.sh"ok "GFS credentials reconciled and writer bootstrap verified"diff --git a/scripts/minikube/gfs-rollout-shim/kubectl b/scripts/minikube/gfs-rollout-shim/kubectlnew file mode 100755index 000000000..133e6dd75--- /dev/null+++ b/scripts/minikube/gfs-rollout-shim/kubectl@@ -0,0 +1,92 @@+#!/usr/bin/env bash+# Development harness PATH shim, prepended only for the GFS credential+# reconcile subprocess. The shared rollout library finishes a reader rollout+# with `kubectl rollout status deployment/gfsc-reader`, but HCC's+# gfsReconciler owns that Deployment template and strips the restartedAt+# annotation `kubectl rollout restart` adds, so the generation-based wait+# chases flapping revisions until it times out. Intercept exactly that wait+# and judge readiness instead (wait-gfs-reader-ready.sh); every other kubectl+# invocation is passed through to the real binary unchanged.+set -euo pipefail++SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"++REAL_KUBECTL=""+old_ifs="${IFS}"+IFS=:+for dir in $PATH; do+ IFS="$old_ifs"+ [ -n "$dir" ] || continue+ resolved="$(cd "$dir" 2>/dev/null && pwd -P)" || continue+ [ "$resolved" = "$SELF_DIR" ] && continue+ if [ -f "${dir}/kubectl" ] && [ -x "${dir}/kubectl" ]; then+ REAL_KUBECTL="${dir}/kubectl"+ break+ fi+done+IFS="$old_ifs"+if [ -z "$REAL_KUBECTL" ]; then+ printf '[gfs-rollout-shim] no real kubectl found on PATH\n' >&2+ exit 127+fi++DEPLOY="${GFS_READER_DEPLOYMENT:-gfsc-reader}"+joined=" $* "+intercept=false+case "$joined" in+ *' rollout status '*)+ case "$joined" in+ *" deployment/${DEPLOY} "*|*" deploy/${DEPLOY} "*|\+ *" deployment.apps/${DEPLOY} "*|*" deployment ${DEPLOY} "*)+ intercept=true ;;+ esac+ ;;+esac+if [ "$intercept" != true ]; then+ exec "$REAL_KUBECTL" "$@"+fi++context="${CONTEXT:-}"+namespace="gfs"+timeout_raw=""+args=("$@")+i=0+while [ "$i" -lt "${#args[@]}" ]; do+ a="${args[$i]}"+ case "$a" in+ --context=*) context="${a#--context=}" ;;+ --context) i=$((i + 1)); context="${args[$i]}" ;;+ --namespace=*) namespace="${a#--namespace=}" ;;+ -n|--namespace) i=$((i + 1)); namespace="${args[$i]}" ;;+ --timeout=*) timeout_raw="${a#--timeout=}" ;;+ --timeout) i=$((i + 1)); timeout_raw="${args[$i]}" ;;+ esac+ i=$((i + 1))+done++timeout_seconds=600+case "$timeout_raw" in+ '') ;;+ *m) timeout_seconds=$((${timeout_raw%m} * 60)) ;;+ *s) timeout_seconds="${timeout_raw%s}" ;;+ *[!0-9]*) ;;+ *) timeout_seconds="$timeout_raw" ;;+esac+# A CrashLoopBackOff retry delay reaches five minutes; a shorter caller+# timeout cannot observe the credential-restore recovery this wait exists+# for, so enforce a floor. Overridable only for the stubbed contract test.+min_timeout="${GFS_ROLLOUT_SHIM_MIN_TIMEOUT_SECONDS:-600}"+if [ "$timeout_seconds" -lt "$min_timeout" ]; then+ timeout_seconds="$min_timeout"+fi++if [ -z "$context" ]; then+ printf '[gfs-rollout-shim] cannot determine the kube-context for the %s readiness wait\n' "$DEPLOY" >&2+ exit 1+fi++printf '[gfs-rollout-shim] waiting on %s/%s readiness instead of a generation-based rollout status (HCC owns the template)\n' \+ "$namespace" "$DEPLOY" >&2+CONTEXT="$context" GFS_NS="$namespace" GFS_READER_DEPLOYMENT="$DEPLOY" \+ GFS_READER_WAIT_TIMEOUT_SECONDS="$timeout_seconds" \+ exec bash "${SELF_DIR}/../wait-gfs-reader-ready.sh"diff --git a/scripts/minikube/pre-gate-sync.sh b/scripts/minikube/pre-gate-sync.shindex ce96c2584..ec1185a46 100755--- a/scripts/minikube/pre-gate-sync.sh+++ b/scripts/minikube/pre-gate-sync.sh@@ -316,7 +316,23 @@ converge_gfs_reader_after_restore() {filog "Restarting gfs/${deployment} after credential restore (${ready:-0}/${desired} Ready)"rollout_restart_with_retry gfs "${deployment}"- ${KC} rollout status "deployment/${deployment}" -n gfs --timeout=180s >/dev/null+ # HCC's gfsReconciler strips the restartedAt annotation, so the restart may+ # not replace pods and a generation-based rollout status loops until+ # timeout. Delete live unready reader pods once so they re-read the+ # restored Secret without waiting out CrashLoopBackOff, then judge+ # readiness directly.+ local pod_rows pod_name pod_ready pod_deleting+ pod_rows="$(${KC} get pods -n gfs -l 'app=gfs-controller,clerum.io/gfsc-role=reader' -o \+ 'jsonpath={range .items[*]}{.metadata.name}{"|"}{.status.conditions[?(@.type=="Ready")].status}{"|"}{.metadata.deletionTimestamp}{"\n"}{end}' \+ 2>/dev/null || true)"+ while IFS='|' read -r pod_name pod_ready pod_deleting; do+ [ -n "$pod_name" ] || continue+ [ -z "$pod_deleting" ] || continue+ [ "$pod_ready" != True ] || continue+ ${KC} delete pod "$pod_name" -n gfs --wait=false >/dev/null 2>&1 || true+ done <<<"$pod_rows"+ CONTEXT="${PROFILE}" \+ bash "${PROJECT_DIR}/scripts/minikube/wait-gfs-reader-ready.sh"}provision_gfs_serving() {@@ -334,10 +350,14 @@ provision_gfs_serving() {# GFS_RECOVER_ABANDONED_STATE: a dead prior setup can leave# rollout-running; this sync holds the T2 lock, so resume that claim.# Settle a Ready reader first so reconcile does not restart it and- # race HCC's gfsReconciler during kubectl rollout status.+ # race HCC's gfsReconciler during kubectl rollout status. If reconcile+ # still needs a reader rollout, the gfs-rollout-shim PATH prefix makes+ # that wait judge readiness instead of the template generation HCC keeps+ # rewriting.CONTEXT="${PROFILE}" \bash "${PROJECT_DIR}/scripts/minikube/settle-gfs-reader-rollout.sh"- if ! GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \+ if ! PATH="${PROJECT_DIR}/scripts/minikube/gfs-rollout-shim:${PATH}" \+ GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \CONTEXT="${PROFILE}" \bash "${PROJECT_DIR}/deploy/scripts/reconcile-gfs-deploy-credentials.sh"; thenlog "ERROR: gfs DB provisioning FAILED — gfsc cannot authorize any operation. Aborting ${GATE_NAME} pre-gate sync."diff --git a/scripts/minikube/settle-gfs-reader-rollout.sh b/scripts/minikube/settle-gfs-reader-rollout.shindex 97b0140fc..bcdd71141 100755--- a/scripts/minikube/settle-gfs-reader-rollout.sh+++ b/scripts/minikube/settle-gfs-reader-rollout.sh@@ -1,15 +1,23 @@#!/usr/bin/env bash-# If gfsc-reader is already Ready, clear a leftover rollout-running claim-# on the reader Secret so the next reconcile does not rollout restart.-# HCC's gfsReconciler patches gfsc-reader; a second restart races-# `kubectl rollout status` ("Waiting for deployment spec update to be-# observed") until the wait times out.+# If gfsc-reader is already Ready, settle every leftover that would make the+# next reconcile rollout restart it and race HCC's gfsReconciler:+# - scale to 0 any leftover non-current ReplicaSet that contributes no Ready+# pod (a stale-template RS whose live unready pod keeps+# credential_rollout_pending true forever);+# - delete CrashLoopBackOff reader pods so kubelet's up-to-5m backoff resets+# and the pod re-reads the restored Secret immediately;+# - clear a leftover rollout-running claim on the reader Secret.+# HCC's gfsReconciler owns the reader template and strips the restartedAt+# annotation, so `kubectl rollout status` after a restart loops on+# "Waiting for deployment spec update to be observed" until the wait+# times out; not restarting a Ready reader is the only safe path.set -euo pipefailCONTEXT="${CONTEXT:?set CONTEXT to the target kube-context}"GFS_NS="${GFS_NS:-gfs}"DEPLOY="${GFS_READER_DEPLOYMENT:-gfsc-reader}"SECRET="${GFS_READER_DB_SECRET:-gfs-controller-reader-db}"+SELECTOR="${GFS_READER_SELECTOR:-app=gfs-controller,clerum.io/gfsc-role=reader}"kc() { kubectl --context="$CONTEXT" "$@"; }log() { printf '[settle-gfs-reader-rollout] %s\n' "$*" >&2; }@@ -24,6 +32,35 @@ if [ -z "$desired" ] || [ "$desired" = "0" ] || [ "${ready:-0}" != "$desired" ];exit 0fi+# Desired Ready is met from here on: leftovers below serve nothing.+current_rev="$(kc -n "$GFS_NS" get deployment "$DEPLOY" \+ -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')"+rs_rows="$(kc -n "$GFS_NS" get rs -l "$SELECTOR" -o \+ 'jsonpath={range .items[*]}{.metadata.name}{"|"}{.metadata.ownerReferences[0].name}{"|"}{.spec.replicas}{"|"}{.status.readyReplicas}{"|"}{.metadata.annotations.deployment\.kubernetes\.io/revision}{"\n"}{end}' \+ 2>/dev/null || true)"+while IFS='|' read -r rs_name rs_owner rs_replicas rs_ready rs_rev; do+ [ -n "$rs_name" ] || continue+ [ "$rs_owner" = "$DEPLOY" ] || continue+ [ "${rs_replicas:-0}" -gt 0 ] || continue+ [ "${rs_ready:-0}" -eq 0 ] || continue+ # Never fight the deployment controller over its current-revision RS.+ [ -n "$current_rev" ] && [ "$rs_rev" = "$current_rev" ] && continue+ log "scaling leftover unready ReplicaSet ${rs_name} to 0 (revision ${rs_rev:-unknown}, current ${current_rev:-unknown})"+ kc -n "$GFS_NS" scale rs "$rs_name" --replicas=0 >/dev/null+done <<<"$rs_rows"++pod_rows="$(kc -n "$GFS_NS" get pods -l "$SELECTOR" -o \+ 'jsonpath={range .items[*]}{.metadata.name}{"|"}{.status.conditions[?(@.type=="Ready")].status}{"|"}{.metadata.deletionTimestamp}{"|"}{.status.containerStatuses[0].state.waiting.reason}{"\n"}{end}' \+ 2>/dev/null || true)"+while IFS='|' read -r pod_name pod_ready pod_deleting pod_reason; do+ [ -n "$pod_name" ] || continue+ [ -z "$pod_deleting" ] || continue+ [ "$pod_ready" != True ] || continue+ [ "$pod_reason" = CrashLoopBackOff ] || continue+ log "deleting CrashLoopBackOff pod ${pod_name} so it re-reads the Secret without waiting out kubelet backoff"+ kc -n "$GFS_NS" delete pod "$pod_name" --wait=false >/dev/null+done <<<"$pod_rows"+if ! kc -n "$GFS_NS" get secret "$SECRET" >/dev/null 2>&1; thenexit 0fidiff --git a/scripts/minikube/wait-gfs-reader-ready.sh b/scripts/minikube/wait-gfs-reader-ready.shnew file mode 100755index 000000000..0a5d652fb--- /dev/null+++ b/scripts/minikube/wait-gfs-reader-ready.sh@@ -0,0 +1,43 @@+#!/usr/bin/env bash+# Development harness only. HCC's gfsReconciler owns the gfsc-reader+# Deployment template and strips the restartedAt annotation that+# `kubectl rollout restart` adds, so a generation-based+# `kubectl rollout status deployment/gfsc-reader` chases flapping revisions+# ("Waiting for deployment spec update to be observed") until it times out.+# Judge readiness directly instead: the desired replica count is Ready and no+# live, non-terminating reader pod is unready. A CrashLoopBackOff pod whose+# credential was just restored converges through kubelet retries; this wait+# observes that recovery instead of the template generation HCC keeps moving.+set -euo pipefail++CONTEXT="${CONTEXT:?set CONTEXT to the target kube-context}"+GFS_NS="${GFS_NS:-gfs}"+DEPLOY="${GFS_READER_DEPLOYMENT:-gfsc-reader}"+SELECTOR="${GFS_READER_SELECTOR:-app=gfs-controller,clerum.io/gfsc-role=reader}"+TIMEOUT_SECONDS="${GFS_READER_WAIT_TIMEOUT_SECONDS:-600}"+POLL_SECONDS="${GFS_READER_WAIT_POLL_SECONDS:-5}"++kc() { kubectl --context="$CONTEXT" "$@"; }+log() { printf '[wait-gfs-reader-ready] %s\n' "$*" >&2; }++deadline=$((SECONDS + TIMEOUT_SECONDS))+while :; do+ desired="$(kc -n "$GFS_NS" get deployment "$DEPLOY" \+ -o jsonpath='{.spec.replicas}' 2>/dev/null || true)"+ ready="$(kc -n "$GFS_NS" get deployment "$DEPLOY" \+ -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"+ if [ -n "$desired" ] && [ "$desired" != 0 ] && [ "${ready:-0}" -ge "$desired" ]; then+ unready="$(kc -n "$GFS_NS" get pods -l "$SELECTOR" -o \+ 'jsonpath={range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"|"}{.metadata.deletionTimestamp}{"\n"}{end}' \+ 2>/dev/null | awk -F'|' 'NF && $2 == "" && $1 != "True" { n++ } END { print n+0 }')"+ if [ "${unready:-0}" -eq 0 ]; then+ log "${GFS_NS}/${DEPLOY} is Ready (${ready}/${desired}) with no live unready reader pod"+ exit 0+ fi+ fi+ if [ "$SECONDS" -ge "$deadline" ]; then+ log "timed out after ${TIMEOUT_SECONDS}s waiting for ${GFS_NS}/${DEPLOY} readiness (ready=${ready:-0}/${desired:-unknown})"+ exit 1+ fi+ sleep "$POLL_SECONDS"+donediff --git a/scripts/tests/test-minikube-full-setup.sh b/scripts/tests/test-minikube-full-setup.shindex 870f2c78d..c9336f17d 100755--- a/scripts/tests/test-minikube-full-setup.sh+++ b/scripts/tests/test-minikube-full-setup.sh@@ -284,6 +284,7 @@ assert_gfs_provisioning_follows_migrations_and_core_readiness() {! grep -q 'delete pvc control-postgres-data' scripts/minikube/full-setup.sh && \[[ "$(grep -c 'GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true' scripts/minikube/full-setup.sh)" -ge 2 ]] && \[[ "$(grep -c 'scripts/minikube/settle-gfs-reader-rollout.sh' scripts/minikube/full-setup.sh)" -ge 2 ]] && \+ [[ "$(grep -c 'gfs-rollout-shim' scripts/minikube/full-setup.sh)" -ge 2 ]] && \[[ "$core_block" != *"gfs-controller"* ]]; thenpass "full-setup centralizes explicit UID-bound reset and never auto-deletes on WAL logs"elsediff --git a/scripts/tests/test-minikube-gfs-rollout-shim.sh b/scripts/tests/test-minikube-gfs-rollout-shim.shnew file mode 100644index 000000000..c56580d51--- /dev/null+++ b/scripts/tests/test-minikube-gfs-rollout-shim.sh@@ -0,0 +1,81 @@+#!/usr/bin/env bash+# Stubbed contract for scripts/minikube/gfs-rollout-shim/kubectl and+# scripts/minikube/wait-gfs-reader-ready.sh: the shim passes every ordinary+# kubectl invocation through unchanged and replaces only the gfsc-reader+# `rollout status` wait with a readiness poll that ignores the template+# generation HCC's gfsReconciler keeps rewriting.+set -euo pipefail++ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"+SHIM="$ROOT/scripts/minikube/gfs-rollout-shim/kubectl"+WAIT="$ROOT/scripts/minikube/wait-gfs-reader-ready.sh"+fail() { printf 'FAIL: %s\n' "$*" >&2; exit 1; }++bash -n "$SHIM"+bash -n "$WAIT"++tmp="$(mktemp -d)"+cleanup() { rm -rf "$tmp"; }+trap cleanup EXIT++# Fake "real" kubectl the shim must fall through to.+mkdir -p "$tmp/realbin"+cat >"$tmp/realbin/kubectl" <<'STUB'+#!/usr/bin/env bash+set -euo pipefail+printf '%s\n' "$*" >>"${FAKE_KUBECTL_LOG:?}"+case " $* " in+ *' get deployment gfsc-reader '*'-o jsonpath={.spec.replicas}'*)+ printf '%s' "${FAKE_DESIRED:-1}"; exit 0 ;;+ *' get deployment gfsc-reader '*'-o jsonpath={.status.readyReplicas}'*)+ printf '%s' "${FAKE_READY:-1}"; exit 0 ;;+ *' get pods -l '*)+ printf '%b' "${FAKE_POD_ROWS:-True|\n}"; exit 0 ;;+esac+exit 0+STUB+chmod +x "$tmp/realbin/kubectl"++export FAKE_KUBECTL_LOG="$tmp/kubectl.log"+shim_path="$(dirname "$SHIM"):$tmp/realbin:/usr/bin:/bin"++# 1. Ordinary invocations pass through to the real kubectl unchanged.+: >"$FAKE_KUBECTL_LOG"+PATH="$shim_path" kubectl --context=fake -n gfs get secret whatever >/dev/null+grep -q -- '--context=fake -n gfs get secret whatever' "$FAKE_KUBECTL_LOG" \+ || fail 'shim did not pass an ordinary invocation through to the real kubectl'++# 2. A rollout status on another deployment also passes through.+: >"$FAKE_KUBECTL_LOG"+PATH="$shim_path" kubectl --context=fake -n gfs rollout status deployment/gfsc-writer --timeout=1s >/dev/null+grep -q 'rollout status deployment/gfsc-writer' "$FAKE_KUBECTL_LOG" \+ || fail 'shim intercepted a non-reader rollout status'++# 3. The reader rollout status is intercepted and succeeds on readiness+# without calling the real `rollout status` at all.+: >"$FAKE_KUBECTL_LOG"+out="$(PATH="$shim_path" FAKE_DESIRED=1 FAKE_READY=1 FAKE_POD_ROWS='True|\n' \+ GFS_READER_WAIT_POLL_SECONDS=0 \+ kubectl --context=fake rollout status deployment/gfsc-reader -n gfs --timeout=240s 2>&1)" \+ || fail 'shim readiness wait failed on a Ready reader'+grep -q 'gfs-rollout-shim' <<<"$out" || fail 'shim did not announce the readiness wait'+grep -q 'rollout status deployment/gfsc-reader' "$FAKE_KUBECTL_LOG" \+ && fail 'shim leaked the generation-based rollout status to the real kubectl'++# 4. A live unready reader pod blocks the wait until timeout (fail-loud).+: >"$FAKE_KUBECTL_LOG"+if PATH="$shim_path" FAKE_DESIRED=1 FAKE_READY=1 FAKE_POD_ROWS='True|\nFalse|\n' \+ GFS_ROLLOUT_SHIM_MIN_TIMEOUT_SECONDS=1 GFS_READER_WAIT_POLL_SECONDS=0 \+ kubectl --context=fake rollout status deployment/gfsc-reader -n gfs --timeout=1s 2>/dev/null; then+ fail 'shim readiness wait went green with a live unready reader pod'+fi++# 5. A terminating unready pod does not block the wait.+: >"$FAKE_KUBECTL_LOG"+PATH="$shim_path" FAKE_DESIRED=1 FAKE_READY=1 \+ FAKE_POD_ROWS='True|\nFalse|2026-01-01T00:00:00Z\n' \+ GFS_READER_WAIT_POLL_SECONDS=0 \+ kubectl --context=fake rollout status deployment/gfsc-reader -n gfs --timeout=240s >/dev/null 2>&1 \+ || fail 'shim readiness wait blocked on a terminating pod'++printf 'PASS: gfs-rollout-shim readiness interception contract\n'diff --git a/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh b/scripts/tests/test-minikube-settle-gfs-reader-rollout.shindex 6ca7cfc9b..98ea7a298 100755--- a/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh+++ b/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh@@ -20,9 +20,19 @@ case " $* " inprintf '%s' "${FAKE_DESIRED:-1}"; exit 0 ;;*' get deployment gfsc-reader '*'-o jsonpath={.status.readyReplicas}'*)printf '%s' "${FAKE_READY:-1}"; exit 0 ;;+ *' get deployment gfsc-reader '*revision*)+ printf '%s' "${FAKE_REVISION:-19}"; exit 0 ;;*' get deployment gfsc-reader'*)[ "${FAKE_DEPLOY_EXISTS:-1}" = 1 ] || exit 1exit 0 ;;+ *' get rs -l '*)+ printf '%b' "${FAKE_RS_ROWS:-}"; exit 0 ;;+ *' scale rs '*)+ exit 0 ;;+ *' get pods -l '*)+ printf '%b' "${FAKE_POD_ROWS:-}"; exit 0 ;;+ *' delete pod '*)+ exit 0 ;;*' get secret gfs-controller-reader-db '*gfs-dsn-state*)printf '%s' "${FAKE_STATE:-rollout-running}"; exit 0 ;;*' get secret gfs-controller-reader-db '*gfs-dsn-rotated-at*)@@ -47,6 +57,9 @@ STUBFAKE_STATE="${5:-rollout-running}" \FAKE_ROTATED="${6:-2026-01-01T00:00:00Z}" \FAKE_PATCH_OK="${7:-1}" \+ FAKE_RS_ROWS="${8:-}" \+ FAKE_POD_ROWS="${9:-}" \+ FAKE_REVISION="${10:-19}" \bash "$SCRIPT"printf '%s' "$fake_dir"}@@ -59,6 +72,8 @@ rm -rf "$dir"dir="$(run_settle 1 1 0 1 rollout-running '2026-01-01T00:00:00Z' 1)"grep -q 'patch secret' "$dir/kubectl.log" \&& fail 'unready reader must not settle the leftover claim'+grep -Eq 'scale rs|delete pod' "$dir/kubectl.log" \+ && fail 'unready reader must not scale ReplicaSets or delete pods'rm -rf "$dir"dir="$(run_settle 1 1 1 1 ready '2026-01-01T00:00:00Z' 1)"@@ -66,4 +81,29 @@ grep -q 'patch secret' "$dir/kubectl.log" \&& fail 'ready Secret state must not be patched again'rm -rf "$dir"-printf 'PASS: settle-gfs-reader-rollout leftover Ready claim\n'+# A leftover non-current ReplicaSet with live unready pods keeps+# credential_rollout_pending true forever; it must be scaled to 0 while the+# current-revision ReplicaSet is never touched.+rs_rows='gfsc-reader-stale|gfsc-reader|1|0|12\ngfsc-reader-current|gfsc-reader|1|0|19\ngfsc-reader-old-empty|gfsc-reader|0|0|11\n'+dir="$(run_settle 1 1 1 1 ready '2026-01-01T00:00:00Z' 1 "$rs_rows" '' 19)"+grep -q 'scale rs gfsc-reader-stale --replicas=0' "$dir/kubectl.log" \+ || fail 'leftover non-current unready ReplicaSet was not scaled to 0'+grep -q 'scale rs gfsc-reader-current' "$dir/kubectl.log" \+ && fail 'the current-revision ReplicaSet must never be scaled by settle'+grep -q 'scale rs gfsc-reader-old-empty' "$dir/kubectl.log" \+ && fail 'an already-empty ReplicaSet must not be scaled'+rm -rf "$dir"++# CrashLoopBackOff pods must be deleted (to reset kubelet backoff and re-read+# the Secret); Ready and terminating pods must be left alone.+pod_rows='gfsc-reader-a-crash|False||CrashLoopBackOff\ngfsc-reader-b-ready|True||\ngfsc-reader-c-term|False|2026-01-01T00:00:00Z|CrashLoopBackOff\n'+dir="$(run_settle 1 1 1 1 ready '2026-01-01T00:00:00Z' 1 '' "$pod_rows" 19)"+grep -q 'delete pod gfsc-reader-a-crash' "$dir/kubectl.log" \+ || fail 'CrashLoopBackOff reader pod was not deleted'+grep -q 'delete pod gfsc-reader-b-ready' "$dir/kubectl.log" \+ && fail 'a Ready reader pod must never be deleted'+grep -q 'delete pod gfsc-reader-c-term' "$dir/kubectl.log" \+ && fail 'a terminating reader pod must not be deleted again'+rm -rf "$dir"++printf 'PASS: settle-gfs-reader-rollout leftover Ready claim and stale leftovers\n'diff --git a/scripts/tests/test-minikube-t2-contract.sh b/scripts/tests/test-minikube-t2-contract.shindex 7a3b270b3..d09707a0c 100755--- a/scripts/tests/test-minikube-t2-contract.sh+++ b/scripts/tests/test-minikube-t2-contract.sh@@ -17,9 +17,12 @@ set -ufor file in "$COMMON" "$PREFLIGHT" "$T2" "$T1" \"$ROOT/scripts/e2e/e2e-np08-hcc-authorization.sh" \"$ROOT/scripts/minikube/settle-gfs-reader-rollout.sh" \+ "$ROOT/scripts/minikube/wait-gfs-reader-ready.sh" \+ "$ROOT/scripts/minikube/gfs-rollout-shim/kubectl" \"$ROOT/scripts/tests/test-minikube-t2-public-boundary.sh" \"$ROOT/scripts/tests/test-minikube-t2-scenarios.sh" \- "$ROOT/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh"; do+ "$ROOT/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh" \+ "$ROOT/scripts/tests/test-minikube-gfs-rollout-shim.sh"; dobash -n "$file"done@@ -91,6 +94,8 @@ grep -Fq 'GFS_RESTORE_ACTIVE_NOLOGIN=true' "$T1"grep -Fq 'GFS_RECOVER_ABANDONED_STATE=true' "$T1"grep -Fq 'reconcile-gfs-deploy-credentials.sh' "$T1"grep -Fq 'T1_GFS_RESTORE_REQUIRED=true' "$T1"+grep -Fq 'settle-gfs-reader-rollout.sh' "$T1"+grep -Fq 'gfs-rollout-shim' "$T1"grep -Fq 'T2_UNREADY_DEPLOYMENTS' "$COMMON"# pre-gate-sync GFS serving provisioning must opt into restoring a NOLOGIN@@ -108,6 +113,14 @@ if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'settle-gfecho 'FAIL: provision_gfs_serving does not settle a Ready gfsc-reader leftover rollout claim before reconcile' >&2exit 1fi+if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'gfs-rollout-shim'; then+ echo 'FAIL: provision_gfs_serving reconcile does not use the HCC-safe reader rollout wait' >&2+ exit 1+fi+if ! sed -n '/^converge_gfs_reader_after_restore()/,/^}/p' "$PRE_GATE" | grep -Fq 'wait-gfs-reader-ready.sh'; then+ echo 'FAIL: converge_gfs_reader_after_restore still waits on a generation-based rollout status HCC keeps rewriting' >&2+ exit 1+fiif ! sed -n '/No cluster sync required before/,$p' "$PRE_GATE" | grep -Fq 'provision_gfs_serving'; thenecho 'FAIL: pre-gate-sync skips GFS serving convergence when no cluster sync is required' >&2exit 1@@ -121,6 +134,10 @@ if [[ "$(grep -c 'scripts/minikube/settle-gfs-reader-rollout.sh' "$ROOT/scripts/echo 'FAIL: full-setup REUSE_DB path must settle a Ready gfsc-reader leftover rollout claim before both reconciles' >&2exit 1fi+if [[ "$(grep -c 'gfs-rollout-shim' "$ROOT/scripts/minikube/full-setup.sh")" -lt 2 ]]; then+ echo 'FAIL: full-setup REUSE_DB reconciles must use the HCC-safe reader rollout wait on both calls' >&2+ exit 1+fiif grep -Eq 'make minikube-pre-gate-sync|pre-gate-sync\.sh' "$PREFLIGHT"; thenecho 'FAIL: preflight invokes pre-gate-sync' >&2@@ -267,5 +284,6 @@ if T2_PUBLIC_ROOT="$public_repo" T2_PUBLIC_BASE_REF="$public_base" \fibash "$ROOT/scripts/tests/test-minikube-t2-scenarios.sh"bash "$ROOT/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh"+bash "$ROOT/scripts/tests/test-minikube-gfs-rollout-shim.sh"printf 'PASS: local Minikube T0/T1/T2 contract checks\n'diff --git a/scripts/tests/test-minikube-t2-scenarios.sh b/scripts/tests/test-minikube-t2-scenarios.shindex 679fac3c7..b45fc32b5 100755--- a/scripts/tests/test-minikube-t2-scenarios.sh+++ b/scripts/tests/test-minikube-t2-scenarios.sh@@ -276,11 +276,16 @@ grep -Fq 'Real PostgreSQL suite failed in $package ($lane)' "$ROOT/scripts/e2e/mgrep -Fq 'restore_gfs_runtime_credentials' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 must restore branch-profile GFS credentials on exit'grep -Fq 'GFS_RESTORE_ACTIVE_NOLOGIN=true' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 GFS restore must opt into the NOLOGIN recovery contract'grep -Fq 'GFS_RECOVER_ABANDONED_STATE=true' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 GFS restore must resume an interrupted gfsc-reader rollout claim'+grep -Fq 'settle-gfs-reader-rollout.sh' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 GFS restore must settle a Ready gfsc-reader leftover rollout claim first'+grep -Fq 'gfs-rollout-shim' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 GFS restore must use the HCC-safe reader rollout wait'grep -Fq 'GFS_RESTORE_ACTIVE_NOLOGIN=true' "$ROOT/scripts/minikube/pre-gate-sync.sh" || fail 'pre-gate-sync must restore a NOLOGIN GFS role from the committed Secret DSN'grep -Fq 'GFS_RECOVER_ABANDONED_STATE=true' "$ROOT/scripts/minikube/pre-gate-sync.sh" || fail 'pre-gate-sync must resume an interrupted gfsc-reader rollout claim'grep -Fq 'settle-gfs-reader-rollout.sh' "$ROOT/scripts/minikube/pre-gate-sync.sh" || fail 'pre-gate-sync must settle a Ready gfsc-reader leftover rollout claim before reconcile'+grep -Fq 'gfs-rollout-shim' "$ROOT/scripts/minikube/pre-gate-sync.sh" || fail 'pre-gate-sync reconcile must use the HCC-safe reader rollout wait'+grep -Fq 'wait-gfs-reader-ready.sh' "$ROOT/scripts/minikube/pre-gate-sync.sh" || fail 'pre-gate-sync reader convergence must judge readiness, not the template generation HCC rewrites'[[ "$(grep -c 'GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true' "$ROOT/scripts/minikube/full-setup.sh")" -ge 2 ]] || fail 'full-setup REUSE_DB must restore a NOLOGIN GFS role and resume an abandoned reader rollout on both reconcile calls'[[ "$(grep -c 'scripts/minikube/settle-gfs-reader-rollout.sh' "$ROOT/scripts/minikube/full-setup.sh")" -ge 2 ]] || fail 'full-setup REUSE_DB must settle a Ready gfsc-reader leftover rollout claim before both reconciles'+[[ "$(grep -c 'gfs-rollout-shim' "$ROOT/scripts/minikube/full-setup.sh")" -ge 2 ]] || fail 'full-setup REUSE_DB reconciles must use the HCC-safe reader rollout wait on both calls'grep -Fq 'T2_LOCK_ROOT' "$COMMON"grep -Fq 'trap t2_lock_release EXIT INT TERM' "$COMMON"grep -Fq 'PORT_FORWARD_CONFLICT' "$COMMON"