Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix: judge gfsc-reader rollout by readiness, not the generation HCC rewrites

Alfredo Lopez33e377c71ec2
diff --git a/.cursor/rules/minikube-t0-t1-t2.mdc b/.cursor/rules/minikube-t0-t1-t2.mdc
index e71767825..8c0997ed9 100644
--- a/.cursor/rules/minikube-t0-t1-t2.mdc
+++ b/.cursor/rules/minikube-t0-t1-t2.mdc
@@ -52,7 +52,11 @@ repository — never mix its PRs or clusters into this lane.
fail-loud. T1 restores branch-profile GFS credentials on exit, and
`pre-gate-sync` provisions GFS serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true`
and `GFS_RECOVER_ABANDONED_STATE=true` in every plan, so no manual repair
- script sits between plan and verdict.
+ script sits between plan and verdict. `settle-gfs-reader-rollout.sh` clears
+ Ready-reader leftovers (claim, stale ReplicaSets, CrashLoopBackOff pods)
+ first, and the `gfs-rollout-shim` PATH prefix replaces the reader
+ `rollout status` wait with a readiness poll because HCC's gfsReconciler
+ strips `restartedAt` and makes generation waits time out.
- Evidence lives under ignored `.local-notes/infra/runs/`; never commit it.
Full workflow: skill `.cursor/skills/minikube-t0-t1-t2/SKILL.md`.
diff --git a/.cursor/rules/minikube-t2-harness.mdc b/.cursor/rules/minikube-t2-harness.mdc
index 114df0b9c..9eaa90c13 100644
--- a/.cursor/rules/minikube-t2-harness.mdc
+++ b/.cursor/rules/minikube-t2-harness.mdc
@@ -43,8 +43,15 @@ Invariants that must never regress:
NOLOGIN reader or a leftover `rollout-running` claim from a dead prior
setup. The T2 profile lock makes that abandoned-state recover safe.
When `gfsc-reader` is already Ready, `settle-gfs-reader-rollout.sh`
- marks that leftover claim ready before reconcile so a second
- `rollout restart` cannot race HCC's gfsReconciler.
+ marks that leftover claim ready before reconcile, scales leftover
+ non-current unready reader ReplicaSets to 0, and deletes CrashLoopBackOff
+ reader pods so a second `rollout restart` is not triggered and cannot race
+ HCC's gfsReconciler. HCC strips the `restartedAt` annotation, so every
+ harness GFS reconcile runs with the `scripts/minikube/gfs-rollout-shim`
+ PATH prefix: the reader `rollout status` wait becomes the readiness poll
+ in `scripts/minikube/wait-gfs-reader-ready.sh` (desired replicas Ready and
+ no live non-terminating unready reader pod), never a generation wait.
+ `pre-gate-sync` reader convergence uses the same readiness contract.
- T1 stays fail-loud (no green on unavailable DSN or zero tests); the JSON
reporter is the suite verdict, not a leftover Vitest process exit after a
complete green reporter. Role-reset suites keep the throwaway Postgres 16.
diff --git a/.cursor/skills/minikube-t0-t1-t2/SKILL.md b/.cursor/skills/minikube-t0-t1-t2/SKILL.md
index ae7cdb9b5..22962c6da 100644
--- a/.cursor/skills/minikube-t0-t1-t2/SKILL.md
+++ b/.cursor/skills/minikube-t0-t1-t2/SKILL.md
@@ -95,6 +95,12 @@ T1 runs inside `make minikube-t2`, or explicitly via
and poison the T2 preflight. Do not run that restore by hand. T2
full-reconcile reaches `full-setup.sh` before pre-gate-sync; that REUSE_DB
path must pass the same opt-in on both GFS reconcile calls.
+- Every harness GFS reconcile first settles Ready-reader leftovers
+ (`settle-gfs-reader-rollout.sh`: leftover claim, stale non-current
+ ReplicaSets, CrashLoopBackOff pods) and runs with the `gfs-rollout-shim`
+ PATH prefix, which replaces the reader `rollout status` wait with the
+ readiness poll in `wait-gfs-reader-ready.sh` — HCC's gfsReconciler strips
+ the `restartedAt` annotation, so a generation-based wait times out.
## Step 4 — Verdict and evidence reporting
diff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.md
index e0b8e40a2..fb15ced0b 100644
--- a/.cursor/skills/minikube-t0-t1-t2/reference.md
+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md
@@ -66,7 +66,14 @@ serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true` and
`GFS_RECOVER_ABANDONED_STATE=true` in every plan and restarts an unready
`gfsc-reader` after restore. A Ready leftover `rollout-running` claim is
settled by `scripts/minikube/settle-gfs-reader-rollout.sh` before
-reconcile so HCC's gfsReconciler cannot race a second restart. Do not run
+reconcile: it marks the claim ready, scales leftover non-current unready
+reader ReplicaSets to 0, and deletes CrashLoopBackOff reader pods so they
+re-read the restored Secret without waiting out kubelet backoff. HCC's
+gfsReconciler strips the `restartedAt` annotation, so every harness GFS
+reconcile runs with the `scripts/minikube/gfs-rollout-shim` PATH prefix:
+the reader `rollout status` wait is replaced by the readiness poll in
+`scripts/minikube/wait-gfs-reader-ready.sh` instead of chasing the template
+generation HCC keeps rewriting. Do not run
`reconcile-gfs-deploy-credentials.sh` or `kubectl rollout restart
deploy/gfsc-reader` by hand as a T2-repair step — re-run the entry point.
diff --git a/AGENTS.md b/AGENTS.md
index e36507789..ba679a8cb 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -99,7 +99,11 @@ has an unready required deployment, the orchestrator planner selects
branch-profile GFS credentials on exit and `pre-gate-sync` provisions GFS
serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true` and
`GFS_RECOVER_ABANDONED_STATE=true` in every plan, so no manual repair
-script belongs between plan and verdict. The standalone preflight and
+script belongs between plan and verdict. Harness GFS reconciles settle
+Ready-reader leftovers first (`settle-gfs-reader-rollout.sh`) and wait on
+reader readiness through the `gfs-rollout-shim` PATH prefix instead of a
+generation-based `rollout status`, because HCC's gfsReconciler strips the
+`restartedAt` annotation and makes that wait time out. The standalone preflight and
the final exact-head T2 check stay fail-loud on an unready deployment.
Real PostgreSQL suites are opt-in in the ordinary test
matrix, but a T1 run that requires them must fail when the database/DSN is
diff --git a/CLAUDE.md b/CLAUDE.md
index 5ce1871b5..2e1346eb0 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -24,7 +24,11 @@ bootstrapped profile with an unready deployment (never `PROFILE_UNHEALTHY`
before a transition), T1 restores branch-profile GFS credentials on exit, and
`pre-gate-sync` provisions GFS serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true`
and `GFS_RECOVER_ABANDONED_STATE=true` in every plan; do not insert manual
-repair scripts between runs. Follow the rule
+repair scripts between runs. Harness GFS reconciles settle Ready-reader
+leftovers first (`settle-gfs-reader-rollout.sh`) and judge reader rollout
+completion by readiness via the `gfs-rollout-shim` PATH prefix, because
+HCC's gfsReconciler strips the `restartedAt` annotation and a
+generation-based `kubectl rollout status` would time out. Follow the rule
`.cursor/rules/minikube-t0-t1-t2.mdc` and the skill
`.cursor/skills/minikube-t0-t1-t2/SKILL.md` for the certification workflow.
diff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.md
index ab771c495..e2d263f60 100644
--- a/docs/testing/minikube-t2-runbook.md
+++ b/docs/testing/minikube-t2-runbook.md
@@ -142,14 +142,23 @@ the same contract as the standalone GFS T1 gate, plus resume of a leftover
`rollout-running` claim from a timed-out prior setup. The T2 profile lock
makes that recover safe: the prior process is dead. When `gfsc-reader`
is already Ready, `scripts/minikube/settle-gfs-reader-rollout.sh` marks
-the leftover reader claim ready before reconcile so a second
-`rollout restart` cannot race HCC's gfsReconciler (that patch loop
-makes `kubectl rollout status` wait for a spec update until timeout).
-`pre-gate-sync`
+the leftover reader claim ready before reconcile, scales to 0 any leftover
+non-current ReplicaSet that contributes no Ready pod (its live unready pod
+would otherwise keep the stale-pod recovery pending forever), and deletes
+CrashLoopBackOff reader pods so they re-read the restored Secret without
+waiting out kubelet backoff. HCC's gfsReconciler owns the reader Deployment
+template and strips the `restartedAt` annotation `kubectl rollout restart`
+adds, so a generation-based `kubectl rollout status` chases flapping
+revisions until timeout; every harness GFS reconcile therefore runs with the
+`scripts/minikube/gfs-rollout-shim` PATH prefix, which intercepts exactly
+the reader `rollout status` wait and judges readiness instead
+(`scripts/minikube/wait-gfs-reader-ready.sh`: desired replicas Ready and no
+live non-terminating unready reader pod). `pre-gate-sync`
provisions GFS serving with the same opt-ins in every sync plan (including
-"no cluster sync required") and restarts an unready `gfsc-reader` after a
-successful restore. `full-setup.sh` on the REUSE_DB / T2 full-reconcile path
-passes the same opt-ins on both GFS reconcile calls, because setup runs
+"no cluster sync required") and, after a successful restore, restarts an
+unready `gfsc-reader`, deletes its live unready pods once, and waits on the
+same readiness contract. `full-setup.sh` on the REUSE_DB / T2 full-reconcile
+path passes the same opt-ins on both GFS reconcile calls, because setup runs
before pre-gate-sync and must not abort on a NOLOGIN reader or an abandoned
reader rollout. The recovery helper restores a
NOLOGIN role only from the committed Secret DSN and still fails loud when
diff --git a/scripts/e2e/minikube-real-postgres.sh b/scripts/e2e/minikube-real-postgres.sh
index 9c6e21db1..293624f68 100755
--- a/scripts/e2e/minikube-real-postgres.sh
+++ b/scripts/e2e/minikube-real-postgres.sh
@@ -69,7 +69,16 @@ restore_gfs_runtime_credentials() {
if ! t2_kc -n gfs get secret gfs-controller-db >/dev/null 2>&1; then
return 0
fi
- if ! GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \
+ # Settle a Ready reader's leftover rollout claim first so restore does not
+ # rollout restart it, and let any wait reconcile still needs judge
+ # readiness via the gfs-rollout-shim instead of the template generation
+ # HCC's gfsReconciler keeps rewriting.
+ if ! CONTEXT="$T2_CONTEXT" \
+ bash "$PROJECT_DIR/scripts/minikube/settle-gfs-reader-rollout.sh"; then
+ printf '[minikube-t1] WARN: could not settle a leftover gfsc-reader rollout claim before restore\n' >&2
+ fi
+ if ! PATH="$PROJECT_DIR/scripts/minikube/gfs-rollout-shim:$PATH" \
+ GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \
CONTEXT="$T2_CONTEXT" \
bash "$PROJECT_DIR/deploy/scripts/reconcile-gfs-deploy-credentials.sh"; then
printf '[minikube-t1] ERROR: failed to restore branch-profile GFS credentials\n' >&2
diff --git a/scripts/minikube/full-setup.sh b/scripts/minikube/full-setup.sh
index 580e0ed5a..fb605ff41 100755
--- a/scripts/minikube/full-setup.sh
+++ b/scripts/minikube/full-setup.sh
@@ -987,9 +987,14 @@ else
# a manual retry flag between runs.
# If gfsc-reader is already Ready, settle the leftover claim first so
# reconcile does not rollout restart and race HCC's gfsReconciler.
+ # The gfs-rollout-shim PATH prefix makes any reader rollout wait inside
+ # reconcile judge readiness instead of the template generation HCC keeps
+ # rewriting (it strips the restartedAt annotation, so a generation wait
+ # loops until timeout).
CONTEXT="${PROFILE}" \
bash "${PROJECT_DIR}/scripts/minikube/settle-gfs-reader-rollout.sh"
- GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \
+ PATH="${PROJECT_DIR}/scripts/minikube/gfs-rollout-shim:${PATH}" \
+ GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \
CONTEXT="${PROFILE}" \
bash "${PROJECT_DIR}/deploy/scripts/reconcile-gfs-deploy-credentials.sh"
else
@@ -1055,7 +1060,8 @@ else
CONTEXT="${PROFILE}" bash "${PROJECT_DIR}/deploy/scripts/wait-gfsc-secret-references.sh"
CONTEXT="${PROFILE}" \
bash "${PROJECT_DIR}/scripts/minikube/settle-gfs-reader-rollout.sh"
- GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \
+ PATH="${PROJECT_DIR}/scripts/minikube/gfs-rollout-shim:${PATH}" \
+ GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \
CONTEXT="${PROFILE}" \
bash "${PROJECT_DIR}/deploy/scripts/reconcile-gfs-deploy-credentials.sh"
ok "GFS credentials reconciled and writer bootstrap verified"
diff --git a/scripts/minikube/gfs-rollout-shim/kubectl b/scripts/minikube/gfs-rollout-shim/kubectl
new file mode 100755
index 000000000..133e6dd75
--- /dev/null
+++ b/scripts/minikube/gfs-rollout-shim/kubectl
@@ -0,0 +1,92 @@
+#!/usr/bin/env bash
+# Development harness PATH shim, prepended only for the GFS credential
+# reconcile subprocess. The shared rollout library finishes a reader rollout
+# with `kubectl rollout status deployment/gfsc-reader`, but HCC's
+# gfsReconciler owns that Deployment template and strips the restartedAt
+# annotation `kubectl rollout restart` adds, so the generation-based wait
+# chases flapping revisions until it times out. Intercept exactly that wait
+# and judge readiness instead (wait-gfs-reader-ready.sh); every other kubectl
+# invocation is passed through to the real binary unchanged.
+set -euo pipefail
+
+SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
+
+REAL_KUBECTL=""
+old_ifs="${IFS}"
+IFS=:
+for dir in $PATH; do
+ IFS="$old_ifs"
+ [ -n "$dir" ] || continue
+ resolved="$(cd "$dir" 2>/dev/null && pwd -P)" || continue
+ [ "$resolved" = "$SELF_DIR" ] && continue
+ if [ -f "${dir}/kubectl" ] && [ -x "${dir}/kubectl" ]; then
+ REAL_KUBECTL="${dir}/kubectl"
+ break
+ fi
+done
+IFS="$old_ifs"
+if [ -z "$REAL_KUBECTL" ]; then
+ printf '[gfs-rollout-shim] no real kubectl found on PATH\n' >&2
+ exit 127
+fi
+
+DEPLOY="${GFS_READER_DEPLOYMENT:-gfsc-reader}"
+joined=" $* "
+intercept=false
+case "$joined" in
+ *' rollout status '*)
+ case "$joined" in
+ *" deployment/${DEPLOY} "*|*" deploy/${DEPLOY} "*|\
+ *" deployment.apps/${DEPLOY} "*|*" deployment ${DEPLOY} "*)
+ intercept=true ;;
+ esac
+ ;;
+esac
+if [ "$intercept" != true ]; then
+ exec "$REAL_KUBECTL" "$@"
+fi
+
+context="${CONTEXT:-}"
+namespace="gfs"
+timeout_raw=""
+args=("$@")
+i=0
+while [ "$i" -lt "${#args[@]}" ]; do
+ a="${args[$i]}"
+ case "$a" in
+ --context=*) context="${a#--context=}" ;;
+ --context) i=$((i + 1)); context="${args[$i]}" ;;
+ --namespace=*) namespace="${a#--namespace=}" ;;
+ -n|--namespace) i=$((i + 1)); namespace="${args[$i]}" ;;
+ --timeout=*) timeout_raw="${a#--timeout=}" ;;
+ --timeout) i=$((i + 1)); timeout_raw="${args[$i]}" ;;
+ esac
+ i=$((i + 1))
+done
+
+timeout_seconds=600
+case "$timeout_raw" in
+ '') ;;
+ *m) timeout_seconds=$((${timeout_raw%m} * 60)) ;;
+ *s) timeout_seconds="${timeout_raw%s}" ;;
+ *[!0-9]*) ;;
+ *) timeout_seconds="$timeout_raw" ;;
+esac
+# A CrashLoopBackOff retry delay reaches five minutes; a shorter caller
+# timeout cannot observe the credential-restore recovery this wait exists
+# for, so enforce a floor. Overridable only for the stubbed contract test.
+min_timeout="${GFS_ROLLOUT_SHIM_MIN_TIMEOUT_SECONDS:-600}"
+if [ "$timeout_seconds" -lt "$min_timeout" ]; then
+ timeout_seconds="$min_timeout"
+fi
+
+if [ -z "$context" ]; then
+ printf '[gfs-rollout-shim] cannot determine the kube-context for the %s readiness wait\n' "$DEPLOY" >&2
+ exit 1
+fi
+
+printf '[gfs-rollout-shim] waiting on %s/%s readiness instead of a generation-based rollout status (HCC owns the template)\n' \
+ "$namespace" "$DEPLOY" >&2
+CONTEXT="$context" GFS_NS="$namespace" GFS_READER_DEPLOYMENT="$DEPLOY" \
+ GFS_READER_WAIT_TIMEOUT_SECONDS="$timeout_seconds" \
+ exec bash "${SELF_DIR}/../wait-gfs-reader-ready.sh"
diff --git a/scripts/minikube/pre-gate-sync.sh b/scripts/minikube/pre-gate-sync.sh
index 1f7bf5876..99cfae62b 100755
--- a/scripts/minikube/pre-gate-sync.sh
+++ b/scripts/minikube/pre-gate-sync.sh
@@ -332,7 +332,23 @@ converge_gfs_reader_after_restore() {
fi
log "Restarting gfs/${deployment} after credential restore (${ready:-0}/${desired} Ready)"
rollout_restart_with_retry gfs "${deployment}"
- ${KC} rollout status "deployment/${deployment}" -n gfs --timeout=180s >/dev/null
+ # HCC's gfsReconciler strips the restartedAt annotation, so the restart may
+ # not replace pods and a generation-based rollout status loops until
+ # timeout. Delete live unready reader pods once so they re-read the
+ # restored Secret without waiting out CrashLoopBackOff, then judge
+ # readiness directly.
+ local pod_rows pod_name pod_ready pod_deleting
+ pod_rows="$(${KC} get pods -n gfs -l 'app=gfs-controller,clerum.io/gfsc-role=reader' -o \
+ 'jsonpath={range .items[*]}{.metadata.name}{"|"}{.status.conditions[?(@.type=="Ready")].status}{"|"}{.metadata.deletionTimestamp}{"\n"}{end}' \
+ 2>/dev/null || true)"
+ while IFS='|' read -r pod_name pod_ready pod_deleting; do
+ [ -n "$pod_name" ] || continue
+ [ -z "$pod_deleting" ] || continue
+ [ "$pod_ready" != True ] || continue
+ ${KC} delete pod "$pod_name" -n gfs --wait=false >/dev/null 2>&1 || true
+ done <<<"$pod_rows"
+ CONTEXT="${PROFILE}" \
+ bash "${PROJECT_DIR}/scripts/minikube/wait-gfs-reader-ready.sh"
}
provision_gfs_serving() {
@@ -350,10 +366,14 @@ provision_gfs_serving() {
# GFS_RECOVER_ABANDONED_STATE: a dead prior setup can leave
# rollout-running; this sync holds the T2 lock, so resume that claim.
# Settle a Ready reader first so reconcile does not restart it and
- # race HCC's gfsReconciler during kubectl rollout status.
+ # race HCC's gfsReconciler during kubectl rollout status. If reconcile
+ # still needs a reader rollout, the gfs-rollout-shim PATH prefix makes
+ # that wait judge readiness instead of the template generation HCC keeps
+ # rewriting.
CONTEXT="${PROFILE}" \
bash "${PROJECT_DIR}/scripts/minikube/settle-gfs-reader-rollout.sh"
- if ! GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \
+ if ! PATH="${PROJECT_DIR}/scripts/minikube/gfs-rollout-shim:${PATH}" \
+ GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \
CONTEXT="${PROFILE}" \
bash "${PROJECT_DIR}/deploy/scripts/reconcile-gfs-deploy-credentials.sh"; then
log "ERROR: gfs DB provisioning FAILED — gfsc cannot authorize any operation. Aborting ${GATE_NAME} pre-gate sync."
diff --git a/scripts/minikube/settle-gfs-reader-rollout.sh b/scripts/minikube/settle-gfs-reader-rollout.sh
index 97b0140fc..bcdd71141 100755
--- a/scripts/minikube/settle-gfs-reader-rollout.sh
+++ b/scripts/minikube/settle-gfs-reader-rollout.sh
@@ -1,15 +1,23 @@
#!/usr/bin/env bash
-# If gfsc-reader is already Ready, clear a leftover rollout-running claim
-# on the reader Secret so the next reconcile does not rollout restart.
-# HCC's gfsReconciler patches gfsc-reader; a second restart races
-# `kubectl rollout status` ("Waiting for deployment spec update to be
-# observed") until the wait times out.
+# If gfsc-reader is already Ready, settle every leftover that would make the
+# next reconcile rollout restart it and race HCC's gfsReconciler:
+# - scale to 0 any leftover non-current ReplicaSet that contributes no Ready
+# pod (a stale-template RS whose live unready pod keeps
+# credential_rollout_pending true forever);
+# - delete CrashLoopBackOff reader pods so kubelet's up-to-5m backoff resets
+# and the pod re-reads the restored Secret immediately;
+# - clear a leftover rollout-running claim on the reader Secret.
+# HCC's gfsReconciler owns the reader template and strips the restartedAt
+# annotation, so `kubectl rollout status` after a restart loops on
+# "Waiting for deployment spec update to be observed" until the wait
+# times out; not restarting a Ready reader is the only safe path.
set -euo pipefail
CONTEXT="${CONTEXT:?set CONTEXT to the target kube-context}"
GFS_NS="${GFS_NS:-gfs}"
DEPLOY="${GFS_READER_DEPLOYMENT:-gfsc-reader}"
SECRET="${GFS_READER_DB_SECRET:-gfs-controller-reader-db}"
+SELECTOR="${GFS_READER_SELECTOR:-app=gfs-controller,clerum.io/gfsc-role=reader}"
kc() { kubectl --context="$CONTEXT" "$@"; }
log() { printf '[settle-gfs-reader-rollout] %s\n' "$*" >&2; }
@@ -24,6 +32,35 @@ if [ -z "$desired" ] || [ "$desired" = "0" ] || [ "${ready:-0}" != "$desired" ];
exit 0
fi
+# Desired Ready is met from here on: leftovers below serve nothing.
+current_rev="$(kc -n "$GFS_NS" get deployment "$DEPLOY" \
+ -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')"
+rs_rows="$(kc -n "$GFS_NS" get rs -l "$SELECTOR" -o \
+ 'jsonpath={range .items[*]}{.metadata.name}{"|"}{.metadata.ownerReferences[0].name}{"|"}{.spec.replicas}{"|"}{.status.readyReplicas}{"|"}{.metadata.annotations.deployment\.kubernetes\.io/revision}{"\n"}{end}' \
+ 2>/dev/null || true)"
+while IFS='|' read -r rs_name rs_owner rs_replicas rs_ready rs_rev; do
+ [ -n "$rs_name" ] || continue
+ [ "$rs_owner" = "$DEPLOY" ] || continue
+ [ "${rs_replicas:-0}" -gt 0 ] || continue
+ [ "${rs_ready:-0}" -eq 0 ] || continue
+ # Never fight the deployment controller over its current-revision RS.
+ [ -n "$current_rev" ] && [ "$rs_rev" = "$current_rev" ] && continue
+ log "scaling leftover unready ReplicaSet ${rs_name} to 0 (revision ${rs_rev:-unknown}, current ${current_rev:-unknown})"
+ kc -n "$GFS_NS" scale rs "$rs_name" --replicas=0 >/dev/null
+done <<<"$rs_rows"
+
+pod_rows="$(kc -n "$GFS_NS" get pods -l "$SELECTOR" -o \
+ 'jsonpath={range .items[*]}{.metadata.name}{"|"}{.status.conditions[?(@.type=="Ready")].status}{"|"}{.metadata.deletionTimestamp}{"|"}{.status.containerStatuses[0].state.waiting.reason}{"\n"}{end}' \
+ 2>/dev/null || true)"
+while IFS='|' read -r pod_name pod_ready pod_deleting pod_reason; do
+ [ -n "$pod_name" ] || continue
+ [ -z "$pod_deleting" ] || continue
+ [ "$pod_ready" != True ] || continue
+ [ "$pod_reason" = CrashLoopBackOff ] || continue
+ log "deleting CrashLoopBackOff pod ${pod_name} so it re-reads the Secret without waiting out kubelet backoff"
+ kc -n "$GFS_NS" delete pod "$pod_name" --wait=false >/dev/null
+done <<<"$pod_rows"
+
if ! kc -n "$GFS_NS" get secret "$SECRET" >/dev/null 2>&1; then
exit 0
fi
diff --git a/scripts/minikube/wait-gfs-reader-ready.sh b/scripts/minikube/wait-gfs-reader-ready.sh
new file mode 100755
index 000000000..0a5d652fb
--- /dev/null
+++ b/scripts/minikube/wait-gfs-reader-ready.sh
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+# Development harness only. HCC's gfsReconciler owns the gfsc-reader
+# Deployment template and strips the restartedAt annotation that
+# `kubectl rollout restart` adds, so a generation-based
+# `kubectl rollout status deployment/gfsc-reader` chases flapping revisions
+# ("Waiting for deployment spec update to be observed") until it times out.
+# Judge readiness directly instead: the desired replica count is Ready and no
+# live, non-terminating reader pod is unready. A CrashLoopBackOff pod whose
+# credential was just restored converges through kubelet retries; this wait
+# observes that recovery instead of the template generation HCC keeps moving.
+set -euo pipefail
+
+CONTEXT="${CONTEXT:?set CONTEXT to the target kube-context}"
+GFS_NS="${GFS_NS:-gfs}"
+DEPLOY="${GFS_READER_DEPLOYMENT:-gfsc-reader}"
+SELECTOR="${GFS_READER_SELECTOR:-app=gfs-controller,clerum.io/gfsc-role=reader}"
+TIMEOUT_SECONDS="${GFS_READER_WAIT_TIMEOUT_SECONDS:-600}"
+POLL_SECONDS="${GFS_READER_WAIT_POLL_SECONDS:-5}"
+
+kc() { kubectl --context="$CONTEXT" "$@"; }
+log() { printf '[wait-gfs-reader-ready] %s\n' "$*" >&2; }
+
+deadline=$((SECONDS + TIMEOUT_SECONDS))
+while :; do
+ desired="$(kc -n "$GFS_NS" get deployment "$DEPLOY" \
+ -o jsonpath='{.spec.replicas}' 2>/dev/null || true)"
+ ready="$(kc -n "$GFS_NS" get deployment "$DEPLOY" \
+ -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
+ if [ -n "$desired" ] && [ "$desired" != 0 ] && [ "${ready:-0}" -ge "$desired" ]; then
+ unready="$(kc -n "$GFS_NS" get pods -l "$SELECTOR" -o \
+ 'jsonpath={range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"|"}{.metadata.deletionTimestamp}{"\n"}{end}' \
+ 2>/dev/null | awk -F'|' 'NF && $2 == "" && $1 != "True" { n++ } END { print n+0 }')"
+ if [ "${unready:-0}" -eq 0 ]; then
+ log "${GFS_NS}/${DEPLOY} is Ready (${ready}/${desired}) with no live unready reader pod"
+ exit 0
+ fi
+ fi
+ if [ "$SECONDS" -ge "$deadline" ]; then
+ log "timed out after ${TIMEOUT_SECONDS}s waiting for ${GFS_NS}/${DEPLOY} readiness (ready=${ready:-0}/${desired:-unknown})"
+ exit 1
+ fi
+ sleep "$POLL_SECONDS"
+done
diff --git a/scripts/tests/test-minikube-full-setup.sh b/scripts/tests/test-minikube-full-setup.sh
index 870f2c78d..c9336f17d 100755
--- a/scripts/tests/test-minikube-full-setup.sh
+++ b/scripts/tests/test-minikube-full-setup.sh
@@ -284,6 +284,7 @@ assert_gfs_provisioning_follows_migrations_and_core_readiness() {
! grep -q 'delete pvc control-postgres-data' scripts/minikube/full-setup.sh && \
[[ "$(grep -c 'GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true' scripts/minikube/full-setup.sh)" -ge 2 ]] && \
[[ "$(grep -c 'scripts/minikube/settle-gfs-reader-rollout.sh' scripts/minikube/full-setup.sh)" -ge 2 ]] && \
+ [[ "$(grep -c 'gfs-rollout-shim' scripts/minikube/full-setup.sh)" -ge 2 ]] && \
[[ "$core_block" != *"gfs-controller"* ]]; then
pass "full-setup centralizes explicit UID-bound reset and never auto-deletes on WAL logs"
else
diff --git a/scripts/tests/test-minikube-gfs-rollout-shim.sh b/scripts/tests/test-minikube-gfs-rollout-shim.sh
new file mode 100644
index 000000000..c56580d51
--- /dev/null
+++ b/scripts/tests/test-minikube-gfs-rollout-shim.sh
@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+# Stubbed contract for scripts/minikube/gfs-rollout-shim/kubectl and
+# scripts/minikube/wait-gfs-reader-ready.sh: the shim passes every ordinary
+# kubectl invocation through unchanged and replaces only the gfsc-reader
+# `rollout status` wait with a readiness poll that ignores the template
+# generation HCC's gfsReconciler keeps rewriting.
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+SHIM="$ROOT/scripts/minikube/gfs-rollout-shim/kubectl"
+WAIT="$ROOT/scripts/minikube/wait-gfs-reader-ready.sh"
+fail() { printf 'FAIL: %s\n' "$*" >&2; exit 1; }
+
+bash -n "$SHIM"
+bash -n "$WAIT"
+
+tmp="$(mktemp -d)"
+cleanup() { rm -rf "$tmp"; }
+trap cleanup EXIT
+
+# Fake "real" kubectl the shim must fall through to.
+mkdir -p "$tmp/realbin"
+cat >"$tmp/realbin/kubectl" <<'STUB'
+#!/usr/bin/env bash
+set -euo pipefail
+printf '%s\n' "$*" >>"${FAKE_KUBECTL_LOG:?}"
+case " $* " in
+ *' get deployment gfsc-reader '*'-o jsonpath={.spec.replicas}'*)
+ printf '%s' "${FAKE_DESIRED:-1}"; exit 0 ;;
+ *' get deployment gfsc-reader '*'-o jsonpath={.status.readyReplicas}'*)
+ printf '%s' "${FAKE_READY:-1}"; exit 0 ;;
+ *' get pods -l '*)
+ printf '%b' "${FAKE_POD_ROWS:-True|\n}"; exit 0 ;;
+esac
+exit 0
+STUB
+chmod +x "$tmp/realbin/kubectl"
+
+export FAKE_KUBECTL_LOG="$tmp/kubectl.log"
+shim_path="$(dirname "$SHIM"):$tmp/realbin:/usr/bin:/bin"
+
+# 1. Ordinary invocations pass through to the real kubectl unchanged.
+: >"$FAKE_KUBECTL_LOG"
+PATH="$shim_path" kubectl --context=fake -n gfs get secret whatever >/dev/null
+grep -q -- '--context=fake -n gfs get secret whatever' "$FAKE_KUBECTL_LOG" \
+ || fail 'shim did not pass an ordinary invocation through to the real kubectl'
+
+# 2. A rollout status on another deployment also passes through.
+: >"$FAKE_KUBECTL_LOG"
+PATH="$shim_path" kubectl --context=fake -n gfs rollout status deployment/gfsc-writer --timeout=1s >/dev/null
+grep -q 'rollout status deployment/gfsc-writer' "$FAKE_KUBECTL_LOG" \
+ || fail 'shim intercepted a non-reader rollout status'
+
+# 3. The reader rollout status is intercepted and succeeds on readiness
+# without calling the real `rollout status` at all.
+: >"$FAKE_KUBECTL_LOG"
+out="$(PATH="$shim_path" FAKE_DESIRED=1 FAKE_READY=1 FAKE_POD_ROWS='True|\n' \
+ GFS_READER_WAIT_POLL_SECONDS=0 \
+ kubectl --context=fake rollout status deployment/gfsc-reader -n gfs --timeout=240s 2>&1)" \
+ || fail 'shim readiness wait failed on a Ready reader'
+grep -q 'gfs-rollout-shim' <<<"$out" || fail 'shim did not announce the readiness wait'
+grep -q 'rollout status deployment/gfsc-reader' "$FAKE_KUBECTL_LOG" \
+ && fail 'shim leaked the generation-based rollout status to the real kubectl'
+
+# 4. A live unready reader pod blocks the wait until timeout (fail-loud).
+: >"$FAKE_KUBECTL_LOG"
+if PATH="$shim_path" FAKE_DESIRED=1 FAKE_READY=1 FAKE_POD_ROWS='True|\nFalse|\n' \
+ GFS_ROLLOUT_SHIM_MIN_TIMEOUT_SECONDS=1 GFS_READER_WAIT_POLL_SECONDS=0 \
+ kubectl --context=fake rollout status deployment/gfsc-reader -n gfs --timeout=1s 2>/dev/null; then
+ fail 'shim readiness wait went green with a live unready reader pod'
+fi
+
+# 5. A terminating unready pod does not block the wait.
+: >"$FAKE_KUBECTL_LOG"
+PATH="$shim_path" FAKE_DESIRED=1 FAKE_READY=1 \
+ FAKE_POD_ROWS='True|\nFalse|2026-01-01T00:00:00Z\n' \
+ GFS_READER_WAIT_POLL_SECONDS=0 \
+ kubectl --context=fake rollout status deployment/gfsc-reader -n gfs --timeout=240s >/dev/null 2>&1 \
+ || fail 'shim readiness wait blocked on a terminating pod'
+
+printf 'PASS: gfs-rollout-shim readiness interception contract\n'
diff --git a/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh b/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh
index 6ca7cfc9b..98ea7a298 100755
--- a/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh
+++ b/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh
@@ -20,9 +20,19 @@ case " $* " in
printf '%s' "${FAKE_DESIRED:-1}"; exit 0 ;;
*' get deployment gfsc-reader '*'-o jsonpath={.status.readyReplicas}'*)
printf '%s' "${FAKE_READY:-1}"; exit 0 ;;
+ *' get deployment gfsc-reader '*revision*)
+ printf '%s' "${FAKE_REVISION:-19}"; exit 0 ;;
*' get deployment gfsc-reader'*)
[ "${FAKE_DEPLOY_EXISTS:-1}" = 1 ] || exit 1
exit 0 ;;
+ *' get rs -l '*)
+ printf '%b' "${FAKE_RS_ROWS:-}"; exit 0 ;;
+ *' scale rs '*)
+ exit 0 ;;
+ *' get pods -l '*)
+ printf '%b' "${FAKE_POD_ROWS:-}"; exit 0 ;;
+ *' delete pod '*)
+ exit 0 ;;
*' get secret gfs-controller-reader-db '*gfs-dsn-state*)
printf '%s' "${FAKE_STATE:-rollout-running}"; exit 0 ;;
*' get secret gfs-controller-reader-db '*gfs-dsn-rotated-at*)
@@ -47,6 +57,9 @@ STUB
FAKE_STATE="${5:-rollout-running}" \
FAKE_ROTATED="${6:-2026-01-01T00:00:00Z}" \
FAKE_PATCH_OK="${7:-1}" \
+ FAKE_RS_ROWS="${8:-}" \
+ FAKE_POD_ROWS="${9:-}" \
+ FAKE_REVISION="${10:-19}" \
bash "$SCRIPT"
printf '%s' "$fake_dir"
}
@@ -59,6 +72,8 @@ rm -rf "$dir"
dir="$(run_settle 1 1 0 1 rollout-running '2026-01-01T00:00:00Z' 1)"
grep -q 'patch secret' "$dir/kubectl.log" \
&& fail 'unready reader must not settle the leftover claim'
+grep -Eq 'scale rs|delete pod' "$dir/kubectl.log" \
+ && fail 'unready reader must not scale ReplicaSets or delete pods'
rm -rf "$dir"
dir="$(run_settle 1 1 1 1 ready '2026-01-01T00:00:00Z' 1)"
@@ -66,4 +81,29 @@ grep -q 'patch secret' "$dir/kubectl.log" \
&& fail 'ready Secret state must not be patched again'
rm -rf "$dir"
-printf 'PASS: settle-gfs-reader-rollout leftover Ready claim\n'
+# A leftover non-current ReplicaSet with live unready pods keeps
+# credential_rollout_pending true forever; it must be scaled to 0 while the
+# current-revision ReplicaSet is never touched.
+rs_rows='gfsc-reader-stale|gfsc-reader|1|0|12\ngfsc-reader-current|gfsc-reader|1|0|19\ngfsc-reader-old-empty|gfsc-reader|0|0|11\n'
+dir="$(run_settle 1 1 1 1 ready '2026-01-01T00:00:00Z' 1 "$rs_rows" '' 19)"
+grep -q 'scale rs gfsc-reader-stale --replicas=0' "$dir/kubectl.log" \
+ || fail 'leftover non-current unready ReplicaSet was not scaled to 0'
+grep -q 'scale rs gfsc-reader-current' "$dir/kubectl.log" \
+ && fail 'the current-revision ReplicaSet must never be scaled by settle'
+grep -q 'scale rs gfsc-reader-old-empty' "$dir/kubectl.log" \
+ && fail 'an already-empty ReplicaSet must not be scaled'
+rm -rf "$dir"
+
+# CrashLoopBackOff pods must be deleted (to reset kubelet backoff and re-read
+# the Secret); Ready and terminating pods must be left alone.
+pod_rows='gfsc-reader-a-crash|False||CrashLoopBackOff\ngfsc-reader-b-ready|True||\ngfsc-reader-c-term|False|2026-01-01T00:00:00Z|CrashLoopBackOff\n'
+dir="$(run_settle 1 1 1 1 ready '2026-01-01T00:00:00Z' 1 '' "$pod_rows" 19)"
+grep -q 'delete pod gfsc-reader-a-crash' "$dir/kubectl.log" \
+ || fail 'CrashLoopBackOff reader pod was not deleted'
+grep -q 'delete pod gfsc-reader-b-ready' "$dir/kubectl.log" \
+ && fail 'a Ready reader pod must never be deleted'
+grep -q 'delete pod gfsc-reader-c-term' "$dir/kubectl.log" \
+ && fail 'a terminating reader pod must not be deleted again'
+rm -rf "$dir"
+
+printf 'PASS: settle-gfs-reader-rollout leftover Ready claim and stale leftovers\n'
diff --git a/scripts/tests/test-minikube-t2-contract.sh b/scripts/tests/test-minikube-t2-contract.sh
index adec785e7..c481aa39d 100755
--- a/scripts/tests/test-minikube-t2-contract.sh
+++ b/scripts/tests/test-minikube-t2-contract.sh
@@ -16,9 +16,12 @@ set -u
for file in "$COMMON" "$PREFLIGHT" "$T2" "$T1" \
"$ROOT/scripts/minikube/settle-gfs-reader-rollout.sh" \
+ "$ROOT/scripts/minikube/wait-gfs-reader-ready.sh" \
+ "$ROOT/scripts/minikube/gfs-rollout-shim/kubectl" \
"$ROOT/scripts/tests/test-minikube-t2-public-boundary.sh" \
"$ROOT/scripts/tests/test-minikube-t2-scenarios.sh" \
- "$ROOT/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh"; do
+ "$ROOT/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh" \
+ "$ROOT/scripts/tests/test-minikube-gfs-rollout-shim.sh"; do
bash -n "$file"
done
@@ -82,6 +85,8 @@ grep -Fq 'GFS_RESTORE_ACTIVE_NOLOGIN=true' "$T1"
grep -Fq 'GFS_RECOVER_ABANDONED_STATE=true' "$T1"
grep -Fq 'reconcile-gfs-deploy-credentials.sh' "$T1"
grep -Fq 'T1_GFS_RESTORE_REQUIRED=true' "$T1"
+grep -Fq 'settle-gfs-reader-rollout.sh' "$T1"
+grep -Fq 'gfs-rollout-shim' "$T1"
grep -Fq 'T2_UNREADY_DEPLOYMENTS' "$COMMON"
# pre-gate-sync GFS serving provisioning must opt into restoring a NOLOGIN
@@ -99,6 +104,14 @@ if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'settle-gf
echo 'FAIL: provision_gfs_serving does not settle a Ready gfsc-reader leftover rollout claim before reconcile' >&2
exit 1
fi
+if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'gfs-rollout-shim'; then
+ echo 'FAIL: provision_gfs_serving reconcile does not use the HCC-safe reader rollout wait' >&2
+ exit 1
+fi
+if ! sed -n '/^converge_gfs_reader_after_restore()/,/^}/p' "$PRE_GATE" | grep -Fq 'wait-gfs-reader-ready.sh'; then
+ echo 'FAIL: converge_gfs_reader_after_restore still waits on a generation-based rollout status HCC keeps rewriting' >&2
+ exit 1
+fi
if ! sed -n '/No cluster sync required before/,$p' "$PRE_GATE" | grep -Fq 'provision_gfs_serving'; then
echo 'FAIL: pre-gate-sync skips GFS serving convergence when no cluster sync is required' >&2
exit 1
@@ -112,6 +125,10 @@ if [[ "$(grep -c 'scripts/minikube/settle-gfs-reader-rollout.sh' "$ROOT/scripts/
echo 'FAIL: full-setup REUSE_DB path must settle a Ready gfsc-reader leftover rollout claim before both reconciles' >&2
exit 1
fi
+if [[ "$(grep -c 'gfs-rollout-shim' "$ROOT/scripts/minikube/full-setup.sh")" -lt 2 ]]; then
+ echo 'FAIL: full-setup REUSE_DB reconciles must use the HCC-safe reader rollout wait on both calls' >&2
+ exit 1
+fi
if grep -Eq 'make minikube-pre-gate-sync|pre-gate-sync\.sh' "$PREFLIGHT"; then
echo 'FAIL: preflight invokes pre-gate-sync' >&2
@@ -246,5 +263,6 @@ if T2_PUBLIC_ROOT="$public_repo" T2_PUBLIC_BASE_REF="$public_base" \
fi
bash "$ROOT/scripts/tests/test-minikube-t2-scenarios.sh"
bash "$ROOT/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh"
+bash "$ROOT/scripts/tests/test-minikube-gfs-rollout-shim.sh"
printf 'PASS: local Minikube T0/T1/T2 contract checks\n'
diff --git a/scripts/tests/test-minikube-t2-scenarios.sh b/scripts/tests/test-minikube-t2-scenarios.sh
index bcd49a441..aab20c166 100755
--- a/scripts/tests/test-minikube-t2-scenarios.sh
+++ b/scripts/tests/test-minikube-t2-scenarios.sh
@@ -270,11 +270,16 @@ grep -Fq 'Real PostgreSQL suite failed in $package ($lane)' "$ROOT/scripts/e2e/m
grep -Fq 'restore_gfs_runtime_credentials' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 must restore branch-profile GFS credentials on exit'
grep -Fq 'GFS_RESTORE_ACTIVE_NOLOGIN=true' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 GFS restore must opt into the NOLOGIN recovery contract'
grep -Fq 'GFS_RECOVER_ABANDONED_STATE=true' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 GFS restore must resume an interrupted gfsc-reader rollout claim'
+grep -Fq 'settle-gfs-reader-rollout.sh' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 GFS restore must settle a Ready gfsc-reader leftover rollout claim first'
+grep -Fq 'gfs-rollout-shim' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 GFS restore must use the HCC-safe reader rollout wait'
grep -Fq 'GFS_RESTORE_ACTIVE_NOLOGIN=true' "$ROOT/scripts/minikube/pre-gate-sync.sh" || fail 'pre-gate-sync must restore a NOLOGIN GFS role from the committed Secret DSN'
grep -Fq 'GFS_RECOVER_ABANDONED_STATE=true' "$ROOT/scripts/minikube/pre-gate-sync.sh" || fail 'pre-gate-sync must resume an interrupted gfsc-reader rollout claim'
grep -Fq 'settle-gfs-reader-rollout.sh' "$ROOT/scripts/minikube/pre-gate-sync.sh" || fail 'pre-gate-sync must settle a Ready gfsc-reader leftover rollout claim before reconcile'
+grep -Fq 'gfs-rollout-shim' "$ROOT/scripts/minikube/pre-gate-sync.sh" || fail 'pre-gate-sync reconcile must use the HCC-safe reader rollout wait'
+grep -Fq 'wait-gfs-reader-ready.sh' "$ROOT/scripts/minikube/pre-gate-sync.sh" || fail 'pre-gate-sync reader convergence must judge readiness, not the template generation HCC rewrites'
[[ "$(grep -c 'GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true' "$ROOT/scripts/minikube/full-setup.sh")" -ge 2 ]] || fail 'full-setup REUSE_DB must restore a NOLOGIN GFS role and resume an abandoned reader rollout on both reconcile calls'
[[ "$(grep -c 'scripts/minikube/settle-gfs-reader-rollout.sh' "$ROOT/scripts/minikube/full-setup.sh")" -ge 2 ]] || fail 'full-setup REUSE_DB must settle a Ready gfsc-reader leftover rollout claim before both reconciles'
+[[ "$(grep -c 'gfs-rollout-shim' "$ROOT/scripts/minikube/full-setup.sh")" -ge 2 ]] || fail 'full-setup REUSE_DB reconciles must use the HCC-safe reader rollout wait on both calls'
grep -Fq 'T2_LOCK_ROOT' "$COMMON"
grep -Fq 'trap t2_lock_release EXIT INT TERM' "$COMMON"
grep -Fq 'PORT_FORWARD_CONFLICT' "$COMMON"