Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

docs(minikube): explain why GFS reader waits stay readiness-based

Alfredo Lopez6c93427c7998
diff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.md
index 0d3a0bdd0..3eacad1a3 100644
--- a/.cursor/skills/minikube-t0-t1-t2/reference.md
+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md
@@ -147,11 +147,11 @@ settled by `scripts/minikube/settle-gfs-reader-rollout.sh` before
reconcile: it marks the claim ready, scales leftover non-current unready
reader ReplicaSets to 0, and deletes CrashLoopBackOff reader pods so they
re-read the restored Secret without waiting out kubelet backoff. HCC's
-gfsReconciler strips the `restartedAt` annotation, so every harness GFS
-reconcile runs with the `scripts/minikube/gfs-rollout-shim` PATH prefix:
+gfsReconciler now preserves the `restartedAt` annotation. Every harness GFS
+reconcile still runs with the `scripts/minikube/gfs-rollout-shim` PATH prefix:
the reader `rollout status` wait is replaced by the readiness poll in
-`scripts/minikube/wait-gfs-reader-ready.sh` instead of chasing the template
-generation HCC keeps rewriting. `full-setup.sh` and `pre-gate-sync` also
+`scripts/minikube/wait-gfs-reader-ready.sh` so leftover ReplicaSets cannot
+turn a generation wait into a timeout. `full-setup.sh` and `pre-gate-sync` also
re-run `sync-auth-key.sh` before each GFS reconcile because the overlay
re-applies `gfs-config.jwt-public-key` empty and a reader pod fails closed
without it. Do not run
diff --git a/CLAUDE.md b/CLAUDE.md
index 06f86fa3f..c2439a64f 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -33,8 +33,10 @@ other security gates refresh MCP auth without mutating GFS. Do not insert
manual repair scripts between runs. Harness GFS reconciles settle Ready-reader
leftovers first (`settle-gfs-reader-rollout.sh`) and judge reader rollout
completion by readiness via the `gfs-rollout-shim` PATH prefix, because
-HCC's gfsReconciler strips the `restartedAt` annotation and a
-generation-based `kubectl rollout status` would time out. Follow the rule
+HCC's gfsReconciler now preserves the `restartedAt` annotation, and the
+harness still judges reader readiness via the shim so leftover ReplicaSets
+cannot turn a generation-based `kubectl rollout status` into a timeout.
+Follow the rule
`.cursor/rules/minikube-t0-t1-t2.mdc` and the skill
`.cursor/skills/minikube-t0-t1-t2/SKILL.md` for the certification workflow.
diff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.md
index 2b117d869..5a076fd83 100644
--- a/docs/testing/minikube-t2-runbook.md
+++ b/docs/testing/minikube-t2-runbook.md
@@ -279,11 +279,11 @@ non-current ReplicaSet that contributes no Ready pod (its live unready pod
would otherwise keep the stale-pod recovery pending forever), and deletes
CrashLoopBackOff reader pods so they re-read the restored Secret without
waiting out kubelet backoff. HCC's gfsReconciler owns the reader Deployment
-template and strips the `restartedAt` annotation `kubectl rollout restart`
-adds, so a generation-based `kubectl rollout status` chases flapping
-revisions until timeout; every harness GFS reconcile therefore runs with the
-`scripts/minikube/gfs-rollout-shim` PATH prefix, which intercepts exactly
-the reader `rollout status` wait and judges readiness instead
+template and now preserves the `restartedAt` annotation `kubectl rollout restart`
+adds. Leftover reader ReplicaSets can still make a generation-based
+`kubectl rollout status` wait the wrong revision; every harness GFS reconcile
+therefore runs with the `scripts/minikube/gfs-rollout-shim` PATH prefix, which
+intercepts exactly the reader `rollout status` wait and judges readiness instead
(`scripts/minikube/wait-gfs-reader-ready.sh`: desired replicas Ready and no
live non-terminating unready reader pod). A reader pod also fails closed
when `gfs-config.jwt-public-key` is empty — the overlay re-applies the base
diff --git a/scripts/e2e/minikube-real-postgres.sh b/scripts/e2e/minikube-real-postgres.sh
index d1accf74e..4f8f1a1c2 100755
--- a/scripts/e2e/minikube-real-postgres.sh
+++ b/scripts/e2e/minikube-real-postgres.sh
@@ -140,8 +140,8 @@ restore_gfs_runtime_credentials() {
fi
# Settle a Ready reader's leftover rollout claim first so restore does not
# rollout restart it, and let any wait reconcile still needs judge
- # readiness via the gfs-rollout-shim instead of the template generation
- # HCC's gfsReconciler keeps rewriting.
+ # readiness via the gfs-rollout-shim instead of a leftover template
+ # generation.
if ! T2_SKIP_LOCK=true T2_LOCK_TOKEN="$T2_LOCK_TOKEN" \
T2_PROJECT_DIR="$T2_PROJECT_DIR" MINIKUBE_PROFILE="$T2_PROFILE" \
CONTROL_API_REAL_PG_CONTEXT="$T2_CONTEXT" T2_PROFILE_ROOT="$T2_PROFILE_ROOT" \
diff --git a/scripts/minikube/full-setup.sh b/scripts/minikube/full-setup.sh
index 514df9120..211a8237c 100755
--- a/scripts/minikube/full-setup.sh
+++ b/scripts/minikube/full-setup.sh
@@ -1664,8 +1664,8 @@ reconcile_existing_gfs_credentials() {
bash "${SCRIPT_DIR}/sync-auth-key.sh" --context="${PROFILE}" --require-gfs
# If gfsc-reader is already Ready, settle the leftover claim first so
# reconcile does not rollout restart and race HCC's gfsReconciler. The
- # gfs-rollout-shim makes the wait judge readiness rather than the template
- # generation HCC rewrites.
+ # gfs-rollout-shim makes the wait judge readiness rather than a leftover
+ # template generation.
GFS_READER_ROLLOUT_AUTHORIZED=true ALLOWED_CONTEXTS="${PROFILE}" CONTEXT="${PROFILE}" \
T2_SKIP_LOCK=true \
bash "${PROJECT_DIR}/scripts/minikube/settle-gfs-reader-rollout.sh"
diff --git a/scripts/minikube/gfs-rollout-shim/kubectl b/scripts/minikube/gfs-rollout-shim/kubectl
index 0d618fec8..9ae5daf2c 100755
--- a/scripts/minikube/gfs-rollout-shim/kubectl
+++ b/scripts/minikube/gfs-rollout-shim/kubectl
@@ -1,11 +1,11 @@
#!/usr/bin/env bash
# Development harness PATH shim, prepended only for the GFS credential
# reconcile subprocess. The shared rollout library finishes a reader rollout
-# with `kubectl rollout status deployment/gfsc-reader`, but HCC's
-# gfsReconciler owns that Deployment template and strips the restartedAt
-# annotation `kubectl rollout restart` adds, so the generation-based wait
-# chases flapping revisions until it times out. Intercept exactly that wait
-# and judge readiness instead (wait-gfs-reader-ready.sh); every other kubectl
+# with `kubectl rollout status deployment/gfsc-reader`. HCC's gfsReconciler
+# now preserves the restartedAt annotation `kubectl rollout restart` adds;
+# leftover reader ReplicaSets can still make a generation-based wait chase
+# the wrong revision until timeout. Intercept exactly that wait and judge
+# readiness instead (wait-gfs-reader-ready.sh); every other kubectl
# invocation is passed through to the real binary unchanged.
set -euo pipefail
diff --git a/scripts/minikube/pre-gate-sync.sh b/scripts/minikube/pre-gate-sync.sh
index 15e860867..d1008ffdb 100755
--- a/scripts/minikube/pre-gate-sync.sh
+++ b/scripts/minikube/pre-gate-sync.sh
@@ -485,11 +485,10 @@ converge_gfs_reader_after_restore() {
fi
log "Restarting gfs/${deployment} after credential restore (${ready:-0}/${desired} Ready)"
rollout_restart_with_retry gfs "${deployment}"
- # HCC's gfsReconciler strips the restartedAt annotation, so the restart may
- # not replace pods and a generation-based rollout status loops until
- # timeout. Delete live unready reader pods once so they re-read the
- # restored Secret without waiting out CrashLoopBackOff, then judge
- # readiness directly.
+ # HCC's gfsReconciler now preserves restartedAt. Delete live unready reader
+ # pods once so they re-read the restored Secret without waiting out
+ # CrashLoopBackOff, then judge readiness directly (leftover ReplicaSets can
+ # still make a generation-based rollout status wait the wrong revision).
local pod_rows pod_name pod_ready pod_deleting
if ! pod_rows="$(${KC} get pods -n gfs -l 'app=gfs-controller,clerum.io/gfsc-role=reader' -o \
'jsonpath={range .items[*]}{.metadata.name}{"|"}{.status.conditions[?(@.type=="Ready")].status}{"|"}{.metadata.deletionTimestamp}{"\n"}{end}' \
diff --git a/scripts/minikube/settle-gfs-reader-rollout.sh b/scripts/minikube/settle-gfs-reader-rollout.sh
index e98266c65..119ed9b0b 100755
--- a/scripts/minikube/settle-gfs-reader-rollout.sh
+++ b/scripts/minikube/settle-gfs-reader-rollout.sh
@@ -7,10 +7,10 @@
# - delete CrashLoopBackOff reader pods so kubelet's up-to-5m backoff resets
# and the pod re-reads the restored Secret immediately;
# - clear a leftover rollout-running claim on the reader Secret.
-# HCC's gfsReconciler owns the reader template and strips the restartedAt
-# annotation, so `kubectl rollout status` after a restart loops on
-# "Waiting for deployment spec update to be observed" until the wait
-# times out; not restarting a Ready reader is the only safe path.
+# HCC's gfsReconciler now preserves restartedAt on the reader template.
+# A leftover claim or stale ReplicaSet can still make `kubectl rollout status`
+# wait on the wrong generation; not restarting a Ready reader is the only
+# safe path.
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/../.." && pwd -P)"
diff --git a/scripts/minikube/wait-gfs-reader-ready.sh b/scripts/minikube/wait-gfs-reader-ready.sh
index 5683b7e6d..07e56a067 100755
--- a/scripts/minikube/wait-gfs-reader-ready.sh
+++ b/scripts/minikube/wait-gfs-reader-ready.sh
@@ -1,13 +1,12 @@
#!/usr/bin/env bash
-# Development harness only. HCC's gfsReconciler owns the gfsc-reader
-# Deployment template and strips the restartedAt annotation that
-# `kubectl rollout restart` adds, so a generation-based
-# `kubectl rollout status deployment/gfsc-reader` chases flapping revisions
-# ("Waiting for deployment spec update to be observed") until it times out.
+# Development harness only. HCC's gfsReconciler now preserves the restartedAt
+# annotation that `kubectl rollout restart` adds. Leftover reader ReplicaSets
+# can still make a generation-based `kubectl rollout status
+# deployment/gfsc-reader` wait on the wrong revision until timeout.
# Judge readiness directly instead: the desired replica count is Ready and no
# live, non-terminating reader pod is unready. A CrashLoopBackOff pod whose
# credential was just restored converges through kubelet retries; this wait
-# observes that recovery instead of the template generation HCC keeps moving.
+# observes that recovery instead of a leftover template generation.
set -euo pipefail
CONTEXT="${CONTEXT:?set CONTEXT to the target kube-context}"