Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
docs(minikube): explain why GFS reader waits stay readiness-based
Alfredo Lopez6c93427c7998
diff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.mdindex 0d3a0bdd0..3eacad1a3 100644--- a/.cursor/skills/minikube-t0-t1-t2/reference.md+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md@@ -147,11 +147,11 @@ settled by `scripts/minikube/settle-gfs-reader-rollout.sh` beforereconcile: it marks the claim ready, scales leftover non-current unreadyreader ReplicaSets to 0, and deletes CrashLoopBackOff reader pods so theyre-read the restored Secret without waiting out kubelet backoff. HCC's-gfsReconciler strips the `restartedAt` annotation, so every harness GFS-reconcile runs with the `scripts/minikube/gfs-rollout-shim` PATH prefix:+gfsReconciler now preserves the `restartedAt` annotation. Every harness GFS+reconcile still runs with the `scripts/minikube/gfs-rollout-shim` PATH prefix:the reader `rollout status` wait is replaced by the readiness poll in-`scripts/minikube/wait-gfs-reader-ready.sh` instead of chasing the template-generation HCC keeps rewriting. `full-setup.sh` and `pre-gate-sync` also+`scripts/minikube/wait-gfs-reader-ready.sh` so leftover ReplicaSets cannot+turn a generation wait into a timeout. `full-setup.sh` and `pre-gate-sync` alsore-run `sync-auth-key.sh` before each GFS reconcile because the overlayre-applies `gfs-config.jwt-public-key` empty and a reader pod fails closedwithout it. Do not rundiff --git a/CLAUDE.md b/CLAUDE.mdindex 06f86fa3f..c2439a64f 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -33,8 +33,10 @@ other security gates refresh MCP auth without mutating GFS. Do not insertmanual repair scripts between runs. Harness GFS reconciles settle Ready-readerleftovers first (`settle-gfs-reader-rollout.sh`) and judge reader rolloutcompletion by readiness via the `gfs-rollout-shim` PATH prefix, because-HCC's gfsReconciler strips the `restartedAt` annotation and a-generation-based `kubectl rollout status` would time out. Follow the rule+HCC's gfsReconciler now preserves the `restartedAt` annotation, and the+harness still judges reader readiness via the shim so leftover ReplicaSets+cannot turn a generation-based `kubectl rollout status` into a timeout.+Follow the rule`.cursor/rules/minikube-t0-t1-t2.mdc` and the skill`.cursor/skills/minikube-t0-t1-t2/SKILL.md` for the certification workflow.diff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.mdindex 2b117d869..5a076fd83 100644--- a/docs/testing/minikube-t2-runbook.md+++ b/docs/testing/minikube-t2-runbook.md@@ -279,11 +279,11 @@ non-current ReplicaSet that contributes no Ready pod (its live unready podwould otherwise keep the stale-pod recovery pending forever), and deletesCrashLoopBackOff reader pods so they re-read the restored Secret withoutwaiting out kubelet backoff. HCC's gfsReconciler owns the reader Deployment-template and strips the `restartedAt` annotation `kubectl rollout restart`-adds, so a generation-based `kubectl rollout status` chases flapping-revisions until timeout; every harness GFS reconcile therefore runs with the-`scripts/minikube/gfs-rollout-shim` PATH prefix, which intercepts exactly-the reader `rollout status` wait and judges readiness instead+template and now preserves the `restartedAt` annotation `kubectl rollout restart`+adds. Leftover reader ReplicaSets can still make a generation-based+`kubectl rollout status` wait the wrong revision; every harness GFS reconcile+therefore runs with the `scripts/minikube/gfs-rollout-shim` PATH prefix, which+intercepts exactly the reader `rollout status` wait and judges readiness instead(`scripts/minikube/wait-gfs-reader-ready.sh`: desired replicas Ready and nolive non-terminating unready reader pod). A reader pod also fails closedwhen `gfs-config.jwt-public-key` is empty — the overlay re-applies the basediff --git a/scripts/e2e/minikube-real-postgres.sh b/scripts/e2e/minikube-real-postgres.shindex d1accf74e..4f8f1a1c2 100755--- a/scripts/e2e/minikube-real-postgres.sh+++ b/scripts/e2e/minikube-real-postgres.sh@@ -140,8 +140,8 @@ restore_gfs_runtime_credentials() {fi# Settle a Ready reader's leftover rollout claim first so restore does not# rollout restart it, and let any wait reconcile still needs judge- # readiness via the gfs-rollout-shim instead of the template generation- # HCC's gfsReconciler keeps rewriting.+ # readiness via the gfs-rollout-shim instead of a leftover template+ # generation.if ! T2_SKIP_LOCK=true T2_LOCK_TOKEN="$T2_LOCK_TOKEN" \T2_PROJECT_DIR="$T2_PROJECT_DIR" MINIKUBE_PROFILE="$T2_PROFILE" \CONTROL_API_REAL_PG_CONTEXT="$T2_CONTEXT" T2_PROFILE_ROOT="$T2_PROFILE_ROOT" \diff --git a/scripts/minikube/full-setup.sh b/scripts/minikube/full-setup.shindex 514df9120..211a8237c 100755--- a/scripts/minikube/full-setup.sh+++ b/scripts/minikube/full-setup.sh@@ -1664,8 +1664,8 @@ reconcile_existing_gfs_credentials() {bash "${SCRIPT_DIR}/sync-auth-key.sh" --context="${PROFILE}" --require-gfs# If gfsc-reader is already Ready, settle the leftover claim first so# reconcile does not rollout restart and race HCC's gfsReconciler. The- # gfs-rollout-shim makes the wait judge readiness rather than the template- # generation HCC rewrites.+ # gfs-rollout-shim makes the wait judge readiness rather than a leftover+ # template generation.GFS_READER_ROLLOUT_AUTHORIZED=true ALLOWED_CONTEXTS="${PROFILE}" CONTEXT="${PROFILE}" \T2_SKIP_LOCK=true \bash "${PROJECT_DIR}/scripts/minikube/settle-gfs-reader-rollout.sh"diff --git a/scripts/minikube/gfs-rollout-shim/kubectl b/scripts/minikube/gfs-rollout-shim/kubectlindex 0d618fec8..9ae5daf2c 100755--- a/scripts/minikube/gfs-rollout-shim/kubectl+++ b/scripts/minikube/gfs-rollout-shim/kubectl@@ -1,11 +1,11 @@#!/usr/bin/env bash# Development harness PATH shim, prepended only for the GFS credential# reconcile subprocess. The shared rollout library finishes a reader rollout-# with `kubectl rollout status deployment/gfsc-reader`, but HCC's-# gfsReconciler owns that Deployment template and strips the restartedAt-# annotation `kubectl rollout restart` adds, so the generation-based wait-# chases flapping revisions until it times out. Intercept exactly that wait-# and judge readiness instead (wait-gfs-reader-ready.sh); every other kubectl+# with `kubectl rollout status deployment/gfsc-reader`. HCC's gfsReconciler+# now preserves the restartedAt annotation `kubectl rollout restart` adds;+# leftover reader ReplicaSets can still make a generation-based wait chase+# the wrong revision until timeout. Intercept exactly that wait and judge+# readiness instead (wait-gfs-reader-ready.sh); every other kubectl# invocation is passed through to the real binary unchanged.set -euo pipefaildiff --git a/scripts/minikube/pre-gate-sync.sh b/scripts/minikube/pre-gate-sync.shindex 15e860867..d1008ffdb 100755--- a/scripts/minikube/pre-gate-sync.sh+++ b/scripts/minikube/pre-gate-sync.sh@@ -485,11 +485,10 @@ converge_gfs_reader_after_restore() {filog "Restarting gfs/${deployment} after credential restore (${ready:-0}/${desired} Ready)"rollout_restart_with_retry gfs "${deployment}"- # HCC's gfsReconciler strips the restartedAt annotation, so the restart may- # not replace pods and a generation-based rollout status loops until- # timeout. Delete live unready reader pods once so they re-read the- # restored Secret without waiting out CrashLoopBackOff, then judge- # readiness directly.+ # HCC's gfsReconciler now preserves restartedAt. Delete live unready reader+ # pods once so they re-read the restored Secret without waiting out+ # CrashLoopBackOff, then judge readiness directly (leftover ReplicaSets can+ # still make a generation-based rollout status wait the wrong revision).local pod_rows pod_name pod_ready pod_deletingif ! pod_rows="$(${KC} get pods -n gfs -l 'app=gfs-controller,clerum.io/gfsc-role=reader' -o \'jsonpath={range .items[*]}{.metadata.name}{"|"}{.status.conditions[?(@.type=="Ready")].status}{"|"}{.metadata.deletionTimestamp}{"\n"}{end}' \diff --git a/scripts/minikube/settle-gfs-reader-rollout.sh b/scripts/minikube/settle-gfs-reader-rollout.shindex e98266c65..119ed9b0b 100755--- a/scripts/minikube/settle-gfs-reader-rollout.sh+++ b/scripts/minikube/settle-gfs-reader-rollout.sh@@ -7,10 +7,10 @@# - delete CrashLoopBackOff reader pods so kubelet's up-to-5m backoff resets# and the pod re-reads the restored Secret immediately;# - clear a leftover rollout-running claim on the reader Secret.-# HCC's gfsReconciler owns the reader template and strips the restartedAt-# annotation, so `kubectl rollout status` after a restart loops on-# "Waiting for deployment spec update to be observed" until the wait-# times out; not restarting a Ready reader is the only safe path.+# HCC's gfsReconciler now preserves restartedAt on the reader template.+# A leftover claim or stale ReplicaSet can still make `kubectl rollout status`+# wait on the wrong generation; not restarting a Ready reader is the only+# safe path.set -euo pipefailROOT="$(cd "$(dirname "$0")/../.." && pwd -P)"diff --git a/scripts/minikube/wait-gfs-reader-ready.sh b/scripts/minikube/wait-gfs-reader-ready.shindex 5683b7e6d..07e56a067 100755--- a/scripts/minikube/wait-gfs-reader-ready.sh+++ b/scripts/minikube/wait-gfs-reader-ready.sh@@ -1,13 +1,12 @@#!/usr/bin/env bash-# Development harness only. HCC's gfsReconciler owns the gfsc-reader-# Deployment template and strips the restartedAt annotation that-# `kubectl rollout restart` adds, so a generation-based-# `kubectl rollout status deployment/gfsc-reader` chases flapping revisions-# ("Waiting for deployment spec update to be observed") until it times out.+# Development harness only. HCC's gfsReconciler now preserves the restartedAt+# annotation that `kubectl rollout restart` adds. Leftover reader ReplicaSets+# can still make a generation-based `kubectl rollout status+# deployment/gfsc-reader` wait on the wrong revision until timeout.# Judge readiness directly instead: the desired replica count is Ready and no# live, non-terminating reader pod is unready. A CrashLoopBackOff pod whose# credential was just restored converges through kubelet retries; this wait-# observes that recovery instead of the template generation HCC keeps moving.+# observes that recovery instead of a leftover template generation.set -euo pipefailCONTEXT="${CONTEXT:?set CONTEXT to the target kube-context}"