Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix: resume abandoned gfsc-reader rollout during a single T2 run

Alfredo Lopez98b8fe01c7ca
diff --git a/.cursor/rules/minikube-t0-t1-t2.mdc b/.cursor/rules/minikube-t0-t1-t2.mdc
index b8020078e..e71767825 100644
--- a/.cursor/rules/minikube-t0-t1-t2.mdc
+++ b/.cursor/rules/minikube-t0-t1-t2.mdc
@@ -51,7 +51,8 @@ repository — never mix its PRs or clusters into this lane.
transition. Standalone preflight and the final exact-head check stay
fail-loud. T1 restores branch-profile GFS credentials on exit, and
`pre-gate-sync` provisions GFS serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true`
- in every plan, so no manual repair script sits between plan and verdict.
+ and `GFS_RECOVER_ABANDONED_STATE=true` in every plan, so no manual repair
+ script sits between plan and verdict.
- Evidence lives under ignored `.local-notes/infra/runs/`; never commit it.
Full workflow: skill `.cursor/skills/minikube-t0-t1-t2/SKILL.md`.
diff --git a/.cursor/rules/minikube-t2-harness.mdc b/.cursor/rules/minikube-t2-harness.mdc
index b2ae6968c..fa122a8fb 100644
--- a/.cursor/rules/minikube-t2-harness.mdc
+++ b/.cursor/rules/minikube-t2-harness.mdc
@@ -34,12 +34,14 @@ Invariants that must never regress:
naming it. `T2_PLAN_MODE=false` keeps failing `PROFILE_UNHEALTHY`.
- T1 (`minikube-real-postgres.sh`) restores branch-profile GFS credentials on
exit via `reconcile-gfs-deploy-credentials.sh` with
- `GFS_RESTORE_ACTIVE_NOLOGIN=true` (armed only once suites started), and
- `pre-gate-sync.sh` `provision_gfs_serving` passes the same opt-in and runs
- in every sync plan, restarting an unready `gfsc-reader` after restore.
- `full-setup.sh` REUSE_DB / T2 full-reconcile also passes the opt-in on both
- GFS reconcile calls — setup runs before pre-gate-sync and must not abort
- on a NOLOGIN reader left by T1.
+ `GFS_RESTORE_ACTIVE_NOLOGIN=true` and `GFS_RECOVER_ABANDONED_STATE=true`
+ (armed only once suites started), and `pre-gate-sync.sh`
+ `provision_gfs_serving` passes the same opt-ins and runs in every sync
+ plan, restarting an unready `gfsc-reader` after restore. `full-setup.sh`
+ REUSE_DB / T2 full-reconcile also passes both opt-ins on both GFS
+ reconcile calls — setup runs before pre-gate-sync and must not abort on a
+ NOLOGIN reader or a leftover `rollout-running` claim from a dead prior
+ setup. The T2 profile lock makes that abandoned-state recover safe.
- T1 stays fail-loud (no green on unavailable DSN or zero tests); the JSON
reporter is the suite verdict, not a leftover Vitest process exit after a
complete green reporter. Role-reset suites keep the throwaway Postgres 16.
diff --git a/.cursor/skills/minikube-t0-t1-t2/SKILL.md b/.cursor/skills/minikube-t0-t1-t2/SKILL.md
index 55aa65530..ae7cdb9b5 100644
--- a/.cursor/skills/minikube-t0-t1-t2/SKILL.md
+++ b/.cursor/skills/minikube-t0-t1-t2/SKILL.md
@@ -58,7 +58,8 @@ transition = full-reconcile (deploy/* or charts/* changed, OR a required
deployment is unready on a bootstrapped profile)?
└── make minikube-t2. Never downgrade an infra change to a
service-only restart. An unready deployment is repaired inside the
- run (planner full-reconcile + pre-gate GFS restore), not by a manual
+ run (planner full-reconcile + pre-gate GFS restore, including resume
+ of an abandoned `gfsc-reader` rollout claim), not by a manual
script between runs.
transition = targeted-sync (service-only diff)?
diff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.md
index 53e43975f..6f3e34ea4 100644
--- a/.cursor/skills/minikube-t0-t1-t2/reference.md
+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md
@@ -60,9 +60,10 @@ profile. Restore/DSN provisioning for `gfs` is ordered by full-setup /
`minikube-deploy-all` note in the Makefile). Do not hand-provision GFS DSNs,
and never copy DSNs or credentials between namespaces or into evidence.
-The harness owns NOLOGIN recovery: T1 restores branch-profile GFS credentials
-on exit, and `pre-gate-sync` provisions serving with
-`GFS_RESTORE_ACTIVE_NOLOGIN=true` in every plan and restarts an unready
+The harness owns NOLOGIN recovery and abandoned-rollout resume: T1 restores
+branch-profile GFS credentials on exit, and `pre-gate-sync` provisions
+serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true` and
+`GFS_RECOVER_ABANDONED_STATE=true` in every plan and restarts an unready
`gfsc-reader` after restore. Do not run
`reconcile-gfs-deploy-credentials.sh` or `kubectl rollout restart
deploy/gfsc-reader` by hand as a T2-repair step — re-run the entry point.
diff --git a/AGENTS.md b/AGENTS.md
index 356921130..e36507789 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -97,8 +97,9 @@ has an unready required deployment, the orchestrator planner selects
`full-reconcile` (naming the deployment) instead of failing
`PROFILE_UNHEALTHY` before a transition exists; the T1 lane restores
branch-profile GFS credentials on exit and `pre-gate-sync` provisions GFS
-serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true` in every plan, so no manual
-repair script belongs between plan and verdict. The standalone preflight and
+serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true` and
+`GFS_RECOVER_ABANDONED_STATE=true` in every plan, so no manual repair
+script belongs between plan and verdict. The standalone preflight and
the final exact-head T2 check stay fail-loud on an unready deployment.
Real PostgreSQL suites are opt-in in the ordinary test
matrix, but a T1 run that requires them must fail when the database/DSN is
diff --git a/CLAUDE.md b/CLAUDE.md
index 72635b17b..5ce1871b5 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -23,7 +23,8 @@ self-healing: the orchestrator planner selects `full-reconcile` for a
bootstrapped profile with an unready deployment (never `PROFILE_UNHEALTHY`
before a transition), T1 restores branch-profile GFS credentials on exit, and
`pre-gate-sync` provisions GFS serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true`
-in every plan; do not insert manual repair scripts between runs. Follow the rule
+and `GFS_RECOVER_ABANDONED_STATE=true` in every plan; do not insert manual
+repair scripts between runs. Follow the rule
`.cursor/rules/minikube-t0-t1-t2.mdc` and the skill
`.cursor/skills/minikube-t0-t1-t2/SKILL.md` for the certification workflow.
diff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.md
index 224e7857b..b740afccd 100644
--- a/docs/testing/minikube-t2-runbook.md
+++ b/docs/testing/minikube-t2-runbook.md
@@ -137,13 +137,16 @@ gfs-controller shared suites exercise cluster-global role names, so the T1
lane restores the branch-profile GFS credentials on exit (success, failure,
or interrupt) using the canonical
`deploy/scripts/reconcile-gfs-deploy-credentials.sh` with
-`GFS_RESTORE_ACTIVE_NOLOGIN=true` — the same contract as the standalone GFS
-T1 gate. `pre-gate-sync` provisions GFS serving with the same opt-in in every
-sync plan (including "no cluster sync required") and restarts an unready
-`gfsc-reader` after a successful restore. `full-setup.sh` on the REUSE_DB /
-T2 full-reconcile path passes the same opt-in on both GFS reconcile calls,
-because setup runs before pre-gate-sync and must not abort on a NOLOGIN
-reader. The recovery helper restores a
+`GFS_RESTORE_ACTIVE_NOLOGIN=true` and `GFS_RECOVER_ABANDONED_STATE=true` —
+the same contract as the standalone GFS T1 gate, plus resume of a leftover
+`rollout-running` claim from a timed-out prior setup. The T2 profile lock
+makes that recover safe: the prior process is dead. `pre-gate-sync`
+provisions GFS serving with the same opt-ins in every sync plan (including
+"no cluster sync required") and restarts an unready `gfsc-reader` after a
+successful restore. `full-setup.sh` on the REUSE_DB / T2 full-reconcile path
+passes the same opt-ins on both GFS reconcile calls, because setup runs
+before pre-gate-sync and must not abort on a NOLOGIN reader or an abandoned
+reader rollout. The recovery helper restores a
NOLOGIN role only from the committed Secret DSN and still fails loud when
that credential cannot authenticate; no password is ever invented and no DSN
is printed.
diff --git a/scripts/e2e/minikube-real-postgres.sh b/scripts/e2e/minikube-real-postgres.sh
index 5d89f3332..9c6e21db1 100755
--- a/scripts/e2e/minikube-real-postgres.sh
+++ b/scripts/e2e/minikube-real-postgres.sh
@@ -69,7 +69,8 @@ restore_gfs_runtime_credentials() {
if ! t2_kc -n gfs get secret gfs-controller-db >/dev/null 2>&1; then
return 0
fi
- if ! GFS_RESTORE_ACTIVE_NOLOGIN=true CONTEXT="$T2_CONTEXT" \
+ if ! GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \
+ CONTEXT="$T2_CONTEXT" \
bash "$PROJECT_DIR/deploy/scripts/reconcile-gfs-deploy-credentials.sh"; then
printf '[minikube-t1] ERROR: failed to restore branch-profile GFS credentials\n' >&2
return 1
diff --git a/scripts/minikube/full-setup.sh b/scripts/minikube/full-setup.sh
index ace16883d..7a1d128b9 100755
--- a/scripts/minikube/full-setup.sh
+++ b/scripts/minikube/full-setup.sh
@@ -981,7 +981,12 @@ else
# REUSE_DB / T2 full-reconcile reaches this before pre-gate-sync. A prior T1
# can leave gfs_controller_reader NOLOGIN; restore from the committed
# Secret DSN (same opt-in as the GFS T1 gate). Still fail-loud on auth.
- GFS_RESTORE_ACTIVE_NOLOGIN=true CONTEXT="${PROFILE}" \
+ # GFS_RECOVER_ABANDONED_STATE: a timed-out prior setup can leave the
+ # reader Secret in rollout-running. This path holds the T2 profile lock,
+ # so the prior process is dead; resume that claim instead of asking for
+ # a manual retry flag between runs.
+ GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \
+ CONTEXT="${PROFILE}" \
bash "${PROJECT_DIR}/deploy/scripts/reconcile-gfs-deploy-credentials.sh"
else
log "Fresh bootstrap detected — reader staging deferred until migrations; GFSC remains fail-closed"
@@ -1044,7 +1049,8 @@ else
# the split writer/reader templates; reconciling before that lands leaves
# the staged reader credential rollout-pending and fails the final verify.
CONTEXT="${PROFILE}" bash "${PROJECT_DIR}/deploy/scripts/wait-gfsc-secret-references.sh"
- GFS_RESTORE_ACTIVE_NOLOGIN=true CONTEXT="${PROFILE}" \
+ GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \
+ CONTEXT="${PROFILE}" \
bash "${PROJECT_DIR}/deploy/scripts/reconcile-gfs-deploy-credentials.sh"
ok "GFS credentials reconciled and writer bootstrap verified"
fi
diff --git a/scripts/minikube/pre-gate-sync.sh b/scripts/minikube/pre-gate-sync.sh
index 312b6e8d6..c4c1676af 100755
--- a/scripts/minikube/pre-gate-sync.sh
+++ b/scripts/minikube/pre-gate-sync.sh
@@ -347,7 +347,10 @@ provision_gfs_serving() {
# may restore the role from it — the same contract the GFS T1 gate already
# applies on exit. The helper still fails loud when the restored
# credential cannot authenticate; no password is invented here.
- if ! GFS_RESTORE_ACTIVE_NOLOGIN=true CONTEXT="${PROFILE}" \
+ # GFS_RECOVER_ABANDONED_STATE: a dead prior setup can leave
+ # rollout-running; this sync holds the T2 lock, so resume that claim.
+ if ! GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \
+ CONTEXT="${PROFILE}" \
bash "${PROJECT_DIR}/deploy/scripts/reconcile-gfs-deploy-credentials.sh"; then
log "ERROR: gfs DB provisioning FAILED — gfsc cannot authorize any operation. Aborting ${GATE_NAME} pre-gate sync."
exit 1
diff --git a/scripts/tests/test-minikube-full-setup.sh b/scripts/tests/test-minikube-full-setup.sh
index 2e8c0e30e..f6aab554c 100755
--- a/scripts/tests/test-minikube-full-setup.sh
+++ b/scripts/tests/test-minikube-full-setup.sh
@@ -282,7 +282,7 @@ assert_gfs_provisioning_follows_migrations_and_core_readiness() {
grep -q 'scale deployment/gfsc-reader --replicas="\$RESET_READER_REPLICAS"' <<<"$reset_block" && \
grep -q 'scale deployment/host-context-controller --replicas="\$RESET_HCC_REPLICAS"' <<<"$reset_block" && \
! grep -q 'delete pvc control-postgres-data' scripts/minikube/full-setup.sh && \
- [[ "$(grep -c 'GFS_RESTORE_ACTIVE_NOLOGIN=true CONTEXT="${PROFILE}"' scripts/minikube/full-setup.sh)" -ge 2 ]] && \
+ [[ "$(grep -c 'GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true' scripts/minikube/full-setup.sh)" -ge 2 ]] && \
[[ "$core_block" != *"gfs-controller"* ]]; then
pass "full-setup centralizes explicit UID-bound reset and never auto-deletes on WAL logs"
else
diff --git a/scripts/tests/test-minikube-t2-contract.sh b/scripts/tests/test-minikube-t2-contract.sh
index ea0065fbf..347cb5cae 100755
--- a/scripts/tests/test-minikube-t2-contract.sh
+++ b/scripts/tests/test-minikube-t2-contract.sh
@@ -77,6 +77,7 @@ grep -Fq 'set +x' "$T1" "$PREFLIGHT" "$T2"
# cannot leave gfsc-reader unready and fail the exact-head T2 preflight.
grep -Fq 'restore_gfs_runtime_credentials' "$T1"
grep -Fq 'GFS_RESTORE_ACTIVE_NOLOGIN=true' "$T1"
+grep -Fq 'GFS_RECOVER_ABANDONED_STATE=true' "$T1"
grep -Fq 'reconcile-gfs-deploy-credentials.sh' "$T1"
grep -Fq 'T1_GFS_RESTORE_REQUIRED=true' "$T1"
grep -Fq 'T2_UNREADY_DEPLOYMENTS' "$COMMON"
@@ -88,13 +89,17 @@ if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'GFS_RESTO
echo 'FAIL: provision_gfs_serving does not restore a NOLOGIN GFS role from the committed Secret DSN' >&2
exit 1
fi
+if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'GFS_RECOVER_ABANDONED_STATE=true'; then
+ echo 'FAIL: provision_gfs_serving does not resume an interrupted gfsc-reader rollout claim' >&2
+ exit 1
+fi
if ! sed -n '/No cluster sync required before/,$p' "$PRE_GATE" | grep -Fq 'provision_gfs_serving'; then
echo 'FAIL: pre-gate-sync skips GFS serving convergence when no cluster sync is required' >&2
exit 1
fi
grep -Fq 'converge_gfs_reader_after_restore' "$PRE_GATE"
-if [[ "$(grep -c 'GFS_RESTORE_ACTIVE_NOLOGIN=true CONTEXT="${PROFILE}"' "$ROOT/scripts/minikube/full-setup.sh")" -lt 2 ]]; then
- echo 'FAIL: full-setup REUSE_DB path must restore a NOLOGIN GFS role before and after overlay' >&2
+if [[ "$(grep -c 'GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true' "$ROOT/scripts/minikube/full-setup.sh")" -lt 2 ]]; then
+ echo 'FAIL: full-setup REUSE_DB path must restore a NOLOGIN GFS role and resume an abandoned reader rollout before and after overlay' >&2
exit 1
fi
diff --git a/scripts/tests/test-minikube-t2-scenarios.sh b/scripts/tests/test-minikube-t2-scenarios.sh
index 10cfdca11..fe869bff9 100755
--- a/scripts/tests/test-minikube-t2-scenarios.sh
+++ b/scripts/tests/test-minikube-t2-scenarios.sh
@@ -269,8 +269,10 @@ grep -Fq 'complete green reporter' "$ROOT/scripts/e2e/minikube-real-postgres.sh"
grep -Fq 'Real PostgreSQL suite failed in $package ($lane)' "$ROOT/scripts/e2e/minikube-real-postgres.sh" && fail 'T1 still treats npm exit as a suite failure before the JSON reporter'
grep -Fq 'restore_gfs_runtime_credentials' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 must restore branch-profile GFS credentials on exit'
grep -Fq 'GFS_RESTORE_ACTIVE_NOLOGIN=true' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 GFS restore must opt into the NOLOGIN recovery contract'
+grep -Fq 'GFS_RECOVER_ABANDONED_STATE=true' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 GFS restore must resume an interrupted gfsc-reader rollout claim'
grep -Fq 'GFS_RESTORE_ACTIVE_NOLOGIN=true' "$ROOT/scripts/minikube/pre-gate-sync.sh" || fail 'pre-gate-sync must restore a NOLOGIN GFS role from the committed Secret DSN'
-[[ "$(grep -c 'GFS_RESTORE_ACTIVE_NOLOGIN=true CONTEXT="${PROFILE}"' "$ROOT/scripts/minikube/full-setup.sh")" -ge 2 ]] || fail 'full-setup REUSE_DB must restore a NOLOGIN GFS role on both reconcile calls'
+grep -Fq 'GFS_RECOVER_ABANDONED_STATE=true' "$ROOT/scripts/minikube/pre-gate-sync.sh" || fail 'pre-gate-sync must resume an interrupted gfsc-reader rollout claim'
+[[ "$(grep -c 'GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true' "$ROOT/scripts/minikube/full-setup.sh")" -ge 2 ]] || fail 'full-setup REUSE_DB must restore a NOLOGIN GFS role and resume an abandoned reader rollout on both reconcile calls'
grep -Fq 'T2_LOCK_ROOT' "$COMMON"
grep -Fq 'trap t2_lock_release EXIT INT TERM' "$COMMON"
grep -Fq 'PORT_FORWARD_CONFLICT' "$COMMON"