Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix: re-sync gfs-config.jwt-public-key before each harness GFS reconcile

Alfredo Lopeza7bbf70fbf0c
diff --git a/.cursor/rules/minikube-t2-harness.mdc b/.cursor/rules/minikube-t2-harness.mdc
index 9eaa90c13..1ecd6f67c 100644
--- a/.cursor/rules/minikube-t2-harness.mdc
+++ b/.cursor/rules/minikube-t2-harness.mdc
@@ -52,6 +52,9 @@ Invariants that must never regress:
in `scripts/minikube/wait-gfs-reader-ready.sh` (desired replicas Ready and
no live non-terminating unready reader pod), never a generation wait.
`pre-gate-sync` reader convergence uses the same readiness contract.
+ Both re-run `sync-auth-key.sh` before each GFS reconcile: the overlay
+ re-applies `gfs-config.jwt-public-key` empty and a reader pod fails
+ closed without it, so the readiness wait could otherwise only time out.
- T1 stays fail-loud (no green on unavailable DSN or zero tests); the JSON
reporter is the suite verdict, not a leftover Vitest process exit after a
complete green reporter. Role-reset suites keep the throwaway Postgres 16.
diff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.md
index fb15ced0b..f425bd9dc 100644
--- a/.cursor/skills/minikube-t0-t1-t2/reference.md
+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md
@@ -73,7 +73,10 @@ gfsReconciler strips the `restartedAt` annotation, so every harness GFS
reconcile runs with the `scripts/minikube/gfs-rollout-shim` PATH prefix:
the reader `rollout status` wait is replaced by the readiness poll in
`scripts/minikube/wait-gfs-reader-ready.sh` instead of chasing the template
-generation HCC keeps rewriting. Do not run
+generation HCC keeps rewriting. `full-setup.sh` and `pre-gate-sync` also
+re-run `sync-auth-key.sh` before each GFS reconcile because the overlay
+re-applies `gfs-config.jwt-public-key` empty and a reader pod fails closed
+without it. Do not run
`reconcile-gfs-deploy-credentials.sh` or `kubectl rollout restart
deploy/gfsc-reader` by hand as a T2-repair step — re-run the entry point.
diff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.md
index d2e28d08f..5bc2f6698 100644
--- a/docs/testing/minikube-t2-runbook.md
+++ b/docs/testing/minikube-t2-runbook.md
@@ -153,7 +153,12 @@ revisions until timeout; every harness GFS reconcile therefore runs with the
`scripts/minikube/gfs-rollout-shim` PATH prefix, which intercepts exactly
the reader `rollout status` wait and judges readiness instead
(`scripts/minikube/wait-gfs-reader-ready.sh`: desired replicas Ready and no
-live non-terminating unready reader pod). `pre-gate-sync`
+live non-terminating unready reader pod). A reader pod also fails closed
+when `gfs-config.jwt-public-key` is empty — the overlay re-applies the base
+ConfigMap with an empty value — so `full-setup.sh` and `pre-gate-sync` re-run
+`scripts/minikube/sync-auth-key.sh` before each GFS reconcile; otherwise no
+new reader pod can start and the readiness wait can only time out.
+`pre-gate-sync`
provisions GFS serving with the same opt-ins in every sync plan (including
"no cluster sync required") and, after a successful restore, restarts an
unready `gfsc-reader`, deletes its live unready pods once, and waits on the
diff --git a/scripts/minikube/full-setup.sh b/scripts/minikube/full-setup.sh
index fb605ff41..b15335370 100755
--- a/scripts/minikube/full-setup.sh
+++ b/scripts/minikube/full-setup.sh
@@ -985,6 +985,12 @@ else
# reader Secret in rollout-running. This path holds the T2 profile lock,
# so the prior process is dead; resume that claim instead of asking for
# a manual retry flag between runs.
+ # A reader pod resolves gfs-config.jwt-public-key at container start and
+ # fails closed when it is empty (a prior interrupted run can leave it
+ # wiped by the overlay re-apply). Re-sync it from the live platform
+ # Secret before reconcile so a reader pod can actually start; this is a
+ # no-op when the key already matches.
+ bash "${SCRIPT_DIR}/sync-auth-key.sh" --context="${PROFILE}"
# If gfsc-reader is already Ready, settle the leftover claim first so
# reconcile does not rollout restart and race HCC's gfsReconciler.
# The gfs-rollout-shim PATH prefix makes any reader rollout wait inside
@@ -1058,6 +1064,10 @@ else
# the split writer/reader templates; reconciling before that lands leaves
# the staged reader credential rollout-pending and fails the final verify.
CONTEXT="${PROFILE}" bash "${PROJECT_DIR}/deploy/scripts/wait-gfsc-secret-references.sh"
+ # The overlay apply above re-declared the base gfs-config with an empty
+ # jwt-public-key; re-sync it before any reader pod may need to start,
+ # otherwise the reconcile readiness wait can only time out.
+ bash "${SCRIPT_DIR}/sync-auth-key.sh" --context="${PROFILE}"
CONTEXT="${PROFILE}" \
bash "${PROJECT_DIR}/scripts/minikube/settle-gfs-reader-rollout.sh"
PATH="${PROJECT_DIR}/scripts/minikube/gfs-rollout-shim:${PATH}" \
diff --git a/scripts/minikube/pre-gate-sync.sh b/scripts/minikube/pre-gate-sync.sh
index ec1185a46..025e130df 100755
--- a/scripts/minikube/pre-gate-sync.sh
+++ b/scripts/minikube/pre-gate-sync.sh
@@ -349,6 +349,11 @@ provision_gfs_serving() {
# credential cannot authenticate; no password is invented here.
# GFS_RECOVER_ABANDONED_STATE: a dead prior setup can leave
# rollout-running; this sync holds the T2 lock, so resume that claim.
+ # An interrupted prior setup can leave gfs-config.jwt-public-key empty
+ # (the overlay re-applies the base ConfigMap with an empty value); a
+ # reader pod cannot start without it and any readiness wait would only
+ # time out. Re-sync it first; this is a no-op when the key matches.
+ bash "${PROJECT_DIR}/scripts/minikube/sync-auth-key.sh" --context "${PROFILE}"
# Settle a Ready reader first so reconcile does not restart it and
# race HCC's gfsReconciler during kubectl rollout status. If reconcile
# still needs a reader rollout, the gfs-rollout-shim PATH prefix makes
diff --git a/scripts/tests/test-minikube-t2-contract.sh b/scripts/tests/test-minikube-t2-contract.sh
index d09707a0c..c9f32a369 100755
--- a/scripts/tests/test-minikube-t2-contract.sh
+++ b/scripts/tests/test-minikube-t2-contract.sh
@@ -117,6 +117,10 @@ if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'gfs-rollo
echo 'FAIL: provision_gfs_serving reconcile does not use the HCC-safe reader rollout wait' >&2
exit 1
fi
+if ! sed -n '/^provision_gfs_serving()/,/settle-gfs-reader-rollout.sh/p' "$PRE_GATE" | grep -Fq 'sync-auth-key.sh'; then
+ echo 'FAIL: provision_gfs_serving does not re-sync gfs-config.jwt-public-key before reconcile (an empty key blocks every new reader pod)' >&2
+ exit 1
+fi
if ! sed -n '/^converge_gfs_reader_after_restore()/,/^}/p' "$PRE_GATE" | grep -Fq 'wait-gfs-reader-ready.sh'; then
echo 'FAIL: converge_gfs_reader_after_restore still waits on a generation-based rollout status HCC keeps rewriting' >&2
exit 1
@@ -138,6 +142,10 @@ if [[ "$(grep -c 'gfs-rollout-shim' "$ROOT/scripts/minikube/full-setup.sh")" -lt
echo 'FAIL: full-setup REUSE_DB reconciles must use the HCC-safe reader rollout wait on both calls' >&2
exit 1
fi
+if [[ "$(grep -c 'sync-auth-key.sh' "$ROOT/scripts/minikube/full-setup.sh")" -lt 3 ]]; then
+ echo 'FAIL: full-setup must re-sync gfs-config.jwt-public-key before both GFS reconciles (the overlay re-applies it empty)' >&2
+ exit 1
+fi
if grep -Eq 'make minikube-pre-gate-sync|pre-gate-sync\.sh' "$PREFLIGHT"; then
echo 'FAIL: preflight invokes pre-gate-sync' >&2
diff --git a/scripts/tests/test-minikube-t2-scenarios.sh b/scripts/tests/test-minikube-t2-scenarios.sh
index b45fc32b5..6aff9ad3d 100755
--- a/scripts/tests/test-minikube-t2-scenarios.sh
+++ b/scripts/tests/test-minikube-t2-scenarios.sh
@@ -286,6 +286,7 @@ grep -Fq 'wait-gfs-reader-ready.sh' "$ROOT/scripts/minikube/pre-gate-sync.sh" ||
[[ "$(grep -c 'GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true' "$ROOT/scripts/minikube/full-setup.sh")" -ge 2 ]] || fail 'full-setup REUSE_DB must restore a NOLOGIN GFS role and resume an abandoned reader rollout on both reconcile calls'
[[ "$(grep -c 'scripts/minikube/settle-gfs-reader-rollout.sh' "$ROOT/scripts/minikube/full-setup.sh")" -ge 2 ]] || fail 'full-setup REUSE_DB must settle a Ready gfsc-reader leftover rollout claim before both reconciles'
[[ "$(grep -c 'gfs-rollout-shim' "$ROOT/scripts/minikube/full-setup.sh")" -ge 2 ]] || fail 'full-setup REUSE_DB reconciles must use the HCC-safe reader rollout wait on both calls'
+[[ "$(grep -c 'sync-auth-key.sh' "$ROOT/scripts/minikube/full-setup.sh")" -ge 3 ]] || fail 'full-setup must re-sync gfs-config.jwt-public-key before both GFS reconciles'
grep -Fq 'T2_LOCK_ROOT' "$COMMON"
grep -Fq 'trap t2_lock_release EXIT INT TERM' "$COMMON"
grep -Fq 'PORT_FORWARD_CONFLICT' "$COMMON"