Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
fix: re-sync gfs-config.jwt-public-key before each harness GFS reconcile
Alfredo Lopezc8790109b8fc
diff --git a/.cursor/rules/minikube-t2-harness.mdc b/.cursor/rules/minikube-t2-harness.mdcindex 9eaa90c13..1ecd6f67c 100644--- a/.cursor/rules/minikube-t2-harness.mdc+++ b/.cursor/rules/minikube-t2-harness.mdc@@ -52,6 +52,9 @@ Invariants that must never regress:in `scripts/minikube/wait-gfs-reader-ready.sh` (desired replicas Ready andno live non-terminating unready reader pod), never a generation wait.`pre-gate-sync` reader convergence uses the same readiness contract.+ Both re-run `sync-auth-key.sh` before each GFS reconcile: the overlay+ re-applies `gfs-config.jwt-public-key` empty and a reader pod fails+ closed without it, so the readiness wait could otherwise only time out.- T1 stays fail-loud (no green on unavailable DSN or zero tests); the JSONreporter is the suite verdict, not a leftover Vitest process exit after acomplete green reporter. Role-reset suites keep the throwaway Postgres 16.diff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.mdindex fb15ced0b..f425bd9dc 100644--- a/.cursor/skills/minikube-t0-t1-t2/reference.md+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md@@ -73,7 +73,10 @@ gfsReconciler strips the `restartedAt` annotation, so every harness GFSreconcile runs with the `scripts/minikube/gfs-rollout-shim` PATH prefix:the reader `rollout status` wait is replaced by the readiness poll in`scripts/minikube/wait-gfs-reader-ready.sh` instead of chasing the template-generation HCC keeps rewriting. Do not run+generation HCC keeps rewriting. `full-setup.sh` and `pre-gate-sync` also+re-run `sync-auth-key.sh` before each GFS reconcile because the overlay+re-applies `gfs-config.jwt-public-key` empty and a reader pod fails closed+without it. Do not run`reconcile-gfs-deploy-credentials.sh` or `kubectl rollout restartdeploy/gfsc-reader` by hand as a T2-repair step — re-run the entry point.diff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.mdindex e2d263f60..8d35bfccd 100644--- a/docs/testing/minikube-t2-runbook.md+++ b/docs/testing/minikube-t2-runbook.md@@ -153,7 +153,12 @@ revisions until timeout; every harness GFS reconcile therefore runs with the`scripts/minikube/gfs-rollout-shim` PATH prefix, which intercepts exactlythe reader `rollout status` wait and judges readiness instead(`scripts/minikube/wait-gfs-reader-ready.sh`: desired replicas Ready and no-live non-terminating unready reader pod). `pre-gate-sync`+live non-terminating unready reader pod). A reader pod also fails closed+when `gfs-config.jwt-public-key` is empty — the overlay re-applies the base+ConfigMap with an empty value — so `full-setup.sh` and `pre-gate-sync` re-run+`scripts/minikube/sync-auth-key.sh` before each GFS reconcile; otherwise no+new reader pod can start and the readiness wait can only time out.+`pre-gate-sync`provisions GFS serving with the same opt-ins in every sync plan (including"no cluster sync required") and, after a successful restore, restarts anunready `gfsc-reader`, deletes its live unready pods once, and waits on thediff --git a/scripts/minikube/full-setup.sh b/scripts/minikube/full-setup.shindex fb605ff41..b15335370 100755--- a/scripts/minikube/full-setup.sh+++ b/scripts/minikube/full-setup.sh@@ -985,6 +985,12 @@ else# reader Secret in rollout-running. This path holds the T2 profile lock,# so the prior process is dead; resume that claim instead of asking for# a manual retry flag between runs.+ # A reader pod resolves gfs-config.jwt-public-key at container start and+ # fails closed when it is empty (a prior interrupted run can leave it+ # wiped by the overlay re-apply). Re-sync it from the live platform+ # Secret before reconcile so a reader pod can actually start; this is a+ # no-op when the key already matches.+ bash "${SCRIPT_DIR}/sync-auth-key.sh" --context="${PROFILE}"# If gfsc-reader is already Ready, settle the leftover claim first so# reconcile does not rollout restart and race HCC's gfsReconciler.# The gfs-rollout-shim PATH prefix makes any reader rollout wait inside@@ -1058,6 +1064,10 @@ else# the split writer/reader templates; reconciling before that lands leaves# the staged reader credential rollout-pending and fails the final verify.CONTEXT="${PROFILE}" bash "${PROJECT_DIR}/deploy/scripts/wait-gfsc-secret-references.sh"+ # The overlay apply above re-declared the base gfs-config with an empty+ # jwt-public-key; re-sync it before any reader pod may need to start,+ # otherwise the reconcile readiness wait can only time out.+ bash "${SCRIPT_DIR}/sync-auth-key.sh" --context="${PROFILE}"CONTEXT="${PROFILE}" \bash "${PROJECT_DIR}/scripts/minikube/settle-gfs-reader-rollout.sh"PATH="${PROJECT_DIR}/scripts/minikube/gfs-rollout-shim:${PATH}" \diff --git a/scripts/minikube/pre-gate-sync.sh b/scripts/minikube/pre-gate-sync.shindex 99cfae62b..213956ebf 100755--- a/scripts/minikube/pre-gate-sync.sh+++ b/scripts/minikube/pre-gate-sync.sh@@ -365,6 +365,11 @@ provision_gfs_serving() {# credential cannot authenticate; no password is invented here.# GFS_RECOVER_ABANDONED_STATE: a dead prior setup can leave# rollout-running; this sync holds the T2 lock, so resume that claim.+ # An interrupted prior setup can leave gfs-config.jwt-public-key empty+ # (the overlay re-applies the base ConfigMap with an empty value); a+ # reader pod cannot start without it and any readiness wait would only+ # time out. Re-sync it first; this is a no-op when the key matches.+ bash "${PROJECT_DIR}/scripts/minikube/sync-auth-key.sh" --context "${PROFILE}"# Settle a Ready reader first so reconcile does not restart it and# race HCC's gfsReconciler during kubectl rollout status. If reconcile# still needs a reader rollout, the gfs-rollout-shim PATH prefix makesdiff --git a/scripts/tests/test-minikube-t2-contract.sh b/scripts/tests/test-minikube-t2-contract.shindex c481aa39d..8360f97bd 100755--- a/scripts/tests/test-minikube-t2-contract.sh+++ b/scripts/tests/test-minikube-t2-contract.sh@@ -108,6 +108,10 @@ if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'gfs-rolloecho 'FAIL: provision_gfs_serving reconcile does not use the HCC-safe reader rollout wait' >&2exit 1fi+if ! sed -n '/^provision_gfs_serving()/,/settle-gfs-reader-rollout.sh/p' "$PRE_GATE" | grep -Fq 'sync-auth-key.sh'; then+ echo 'FAIL: provision_gfs_serving does not re-sync gfs-config.jwt-public-key before reconcile (an empty key blocks every new reader pod)' >&2+ exit 1+fiif ! sed -n '/^converge_gfs_reader_after_restore()/,/^}/p' "$PRE_GATE" | grep -Fq 'wait-gfs-reader-ready.sh'; thenecho 'FAIL: converge_gfs_reader_after_restore still waits on a generation-based rollout status HCC keeps rewriting' >&2exit 1@@ -129,6 +133,10 @@ if [[ "$(grep -c 'gfs-rollout-shim' "$ROOT/scripts/minikube/full-setup.sh")" -ltecho 'FAIL: full-setup REUSE_DB reconciles must use the HCC-safe reader rollout wait on both calls' >&2exit 1fi+if [[ "$(grep -c 'sync-auth-key.sh' "$ROOT/scripts/minikube/full-setup.sh")" -lt 3 ]]; then+ echo 'FAIL: full-setup must re-sync gfs-config.jwt-public-key before both GFS reconciles (the overlay re-applies it empty)' >&2+ exit 1+fiif grep -Eq 'make minikube-pre-gate-sync|pre-gate-sync\.sh' "$PREFLIGHT"; thenecho 'FAIL: preflight invokes pre-gate-sync' >&2diff --git a/scripts/tests/test-minikube-t2-scenarios.sh b/scripts/tests/test-minikube-t2-scenarios.shindex aab20c166..9444e0eed 100755--- a/scripts/tests/test-minikube-t2-scenarios.sh+++ b/scripts/tests/test-minikube-t2-scenarios.sh@@ -280,6 +280,7 @@ grep -Fq 'wait-gfs-reader-ready.sh' "$ROOT/scripts/minikube/pre-gate-sync.sh" ||[[ "$(grep -c 'GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true' "$ROOT/scripts/minikube/full-setup.sh")" -ge 2 ]] || fail 'full-setup REUSE_DB must restore a NOLOGIN GFS role and resume an abandoned reader rollout on both reconcile calls'[[ "$(grep -c 'scripts/minikube/settle-gfs-reader-rollout.sh' "$ROOT/scripts/minikube/full-setup.sh")" -ge 2 ]] || fail 'full-setup REUSE_DB must settle a Ready gfsc-reader leftover rollout claim before both reconciles'[[ "$(grep -c 'gfs-rollout-shim' "$ROOT/scripts/minikube/full-setup.sh")" -ge 2 ]] || fail 'full-setup REUSE_DB reconciles must use the HCC-safe reader rollout wait on both calls'+[[ "$(grep -c 'sync-auth-key.sh' "$ROOT/scripts/minikube/full-setup.sh")" -ge 3 ]] || fail 'full-setup must re-sync gfs-config.jwt-public-key before both GFS reconciles'grep -Fq 'T2_LOCK_ROOT' "$COMMON"grep -Fq 'trap t2_lock_release EXIT INT TERM' "$COMMON"grep -Fq 'PORT_FORWARD_CONFLICT' "$COMMON"