Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
fix: resume abandoned gfsc-reader rollout during a single T2 run
Alfredo Lopez98b8fe01c7ca
diff --git a/.cursor/rules/minikube-t0-t1-t2.mdc b/.cursor/rules/minikube-t0-t1-t2.mdcindex b8020078e..e71767825 100644--- a/.cursor/rules/minikube-t0-t1-t2.mdc+++ b/.cursor/rules/minikube-t0-t1-t2.mdc@@ -51,7 +51,8 @@ repository — never mix its PRs or clusters into this lane.transition. Standalone preflight and the final exact-head check stayfail-loud. T1 restores branch-profile GFS credentials on exit, and`pre-gate-sync` provisions GFS serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true`- in every plan, so no manual repair script sits between plan and verdict.+ and `GFS_RECOVER_ABANDONED_STATE=true` in every plan, so no manual repair+ script sits between plan and verdict.- Evidence lives under ignored `.local-notes/infra/runs/`; never commit it.Full workflow: skill `.cursor/skills/minikube-t0-t1-t2/SKILL.md`.diff --git a/.cursor/rules/minikube-t2-harness.mdc b/.cursor/rules/minikube-t2-harness.mdcindex b2ae6968c..fa122a8fb 100644--- a/.cursor/rules/minikube-t2-harness.mdc+++ b/.cursor/rules/minikube-t2-harness.mdc@@ -34,12 +34,14 @@ Invariants that must never regress:naming it. `T2_PLAN_MODE=false` keeps failing `PROFILE_UNHEALTHY`.- T1 (`minikube-real-postgres.sh`) restores branch-profile GFS credentials onexit via `reconcile-gfs-deploy-credentials.sh` with- `GFS_RESTORE_ACTIVE_NOLOGIN=true` (armed only once suites started), and- `pre-gate-sync.sh` `provision_gfs_serving` passes the same opt-in and runs- in every sync plan, restarting an unready `gfsc-reader` after restore.- `full-setup.sh` REUSE_DB / T2 full-reconcile also passes the opt-in on both- GFS reconcile calls — setup runs before pre-gate-sync and must not abort- on a NOLOGIN reader left by T1.+ `GFS_RESTORE_ACTIVE_NOLOGIN=true` and `GFS_RECOVER_ABANDONED_STATE=true`+ (armed only once suites started), and `pre-gate-sync.sh`+ `provision_gfs_serving` passes the same opt-ins and runs in every sync+ plan, restarting an unready `gfsc-reader` after restore. `full-setup.sh`+ REUSE_DB / T2 full-reconcile also passes both opt-ins on both GFS+ reconcile calls — setup runs before pre-gate-sync and must not abort on a+ NOLOGIN reader or a leftover `rollout-running` claim from a dead prior+ setup. The T2 profile lock makes that abandoned-state recover safe.- T1 stays fail-loud (no green on unavailable DSN or zero tests); the JSONreporter is the suite verdict, not a leftover Vitest process exit after acomplete green reporter. Role-reset suites keep the throwaway Postgres 16.diff --git a/.cursor/skills/minikube-t0-t1-t2/SKILL.md b/.cursor/skills/minikube-t0-t1-t2/SKILL.mdindex 55aa65530..ae7cdb9b5 100644--- a/.cursor/skills/minikube-t0-t1-t2/SKILL.md+++ b/.cursor/skills/minikube-t0-t1-t2/SKILL.md@@ -58,7 +58,8 @@ transition = full-reconcile (deploy/* or charts/* changed, OR a requireddeployment is unready on a bootstrapped profile)?└── make minikube-t2. Never downgrade an infra change to aservice-only restart. An unready deployment is repaired inside the- run (planner full-reconcile + pre-gate GFS restore), not by a manual+ run (planner full-reconcile + pre-gate GFS restore, including resume+ of an abandoned `gfsc-reader` rollout claim), not by a manualscript between runs.transition = targeted-sync (service-only diff)?diff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.mdindex 53e43975f..6f3e34ea4 100644--- a/.cursor/skills/minikube-t0-t1-t2/reference.md+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md@@ -60,9 +60,10 @@ profile. Restore/DSN provisioning for `gfs` is ordered by full-setup /`minikube-deploy-all` note in the Makefile). Do not hand-provision GFS DSNs,and never copy DSNs or credentials between namespaces or into evidence.-The harness owns NOLOGIN recovery: T1 restores branch-profile GFS credentials-on exit, and `pre-gate-sync` provisions serving with-`GFS_RESTORE_ACTIVE_NOLOGIN=true` in every plan and restarts an unready+The harness owns NOLOGIN recovery and abandoned-rollout resume: T1 restores+branch-profile GFS credentials on exit, and `pre-gate-sync` provisions+serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true` and+`GFS_RECOVER_ABANDONED_STATE=true` in every plan and restarts an unready`gfsc-reader` after restore. Do not run`reconcile-gfs-deploy-credentials.sh` or `kubectl rollout restartdeploy/gfsc-reader` by hand as a T2-repair step — re-run the entry point.diff --git a/AGENTS.md b/AGENTS.mdindex 356921130..e36507789 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -97,8 +97,9 @@ has an unready required deployment, the orchestrator planner selects`full-reconcile` (naming the deployment) instead of failing`PROFILE_UNHEALTHY` before a transition exists; the T1 lane restoresbranch-profile GFS credentials on exit and `pre-gate-sync` provisions GFS-serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true` in every plan, so no manual-repair script belongs between plan and verdict. The standalone preflight and+serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true` and+`GFS_RECOVER_ABANDONED_STATE=true` in every plan, so no manual repair+script belongs between plan and verdict. The standalone preflight andthe final exact-head T2 check stay fail-loud on an unready deployment.Real PostgreSQL suites are opt-in in the ordinary testmatrix, but a T1 run that requires them must fail when the database/DSN isdiff --git a/CLAUDE.md b/CLAUDE.mdindex 72635b17b..5ce1871b5 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -23,7 +23,8 @@ self-healing: the orchestrator planner selects `full-reconcile` for abootstrapped profile with an unready deployment (never `PROFILE_UNHEALTHY`before a transition), T1 restores branch-profile GFS credentials on exit, and`pre-gate-sync` provisions GFS serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true`-in every plan; do not insert manual repair scripts between runs. Follow the rule+and `GFS_RECOVER_ABANDONED_STATE=true` in every plan; do not insert manual+repair scripts between runs. Follow the rule`.cursor/rules/minikube-t0-t1-t2.mdc` and the skill`.cursor/skills/minikube-t0-t1-t2/SKILL.md` for the certification workflow.diff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.mdindex 224e7857b..b740afccd 100644--- a/docs/testing/minikube-t2-runbook.md+++ b/docs/testing/minikube-t2-runbook.md@@ -137,13 +137,16 @@ gfs-controller shared suites exercise cluster-global role names, so the T1lane restores the branch-profile GFS credentials on exit (success, failure,or interrupt) using the canonical`deploy/scripts/reconcile-gfs-deploy-credentials.sh` with-`GFS_RESTORE_ACTIVE_NOLOGIN=true` — the same contract as the standalone GFS-T1 gate. `pre-gate-sync` provisions GFS serving with the same opt-in in every-sync plan (including "no cluster sync required") and restarts an unready-`gfsc-reader` after a successful restore. `full-setup.sh` on the REUSE_DB /-T2 full-reconcile path passes the same opt-in on both GFS reconcile calls,-because setup runs before pre-gate-sync and must not abort on a NOLOGIN-reader. The recovery helper restores a+`GFS_RESTORE_ACTIVE_NOLOGIN=true` and `GFS_RECOVER_ABANDONED_STATE=true` —+the same contract as the standalone GFS T1 gate, plus resume of a leftover+`rollout-running` claim from a timed-out prior setup. The T2 profile lock+makes that recover safe: the prior process is dead. `pre-gate-sync`+provisions GFS serving with the same opt-ins in every sync plan (including+"no cluster sync required") and restarts an unready `gfsc-reader` after a+successful restore. `full-setup.sh` on the REUSE_DB / T2 full-reconcile path+passes the same opt-ins on both GFS reconcile calls, because setup runs+before pre-gate-sync and must not abort on a NOLOGIN reader or an abandoned+reader rollout. The recovery helper restores aNOLOGIN role only from the committed Secret DSN and still fails loud whenthat credential cannot authenticate; no password is ever invented and no DSNis printed.diff --git a/scripts/e2e/minikube-real-postgres.sh b/scripts/e2e/minikube-real-postgres.shindex 5d89f3332..9c6e21db1 100755--- a/scripts/e2e/minikube-real-postgres.sh+++ b/scripts/e2e/minikube-real-postgres.sh@@ -69,7 +69,8 @@ restore_gfs_runtime_credentials() {if ! t2_kc -n gfs get secret gfs-controller-db >/dev/null 2>&1; thenreturn 0fi- if ! GFS_RESTORE_ACTIVE_NOLOGIN=true CONTEXT="$T2_CONTEXT" \+ if ! GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \+ CONTEXT="$T2_CONTEXT" \bash "$PROJECT_DIR/deploy/scripts/reconcile-gfs-deploy-credentials.sh"; thenprintf '[minikube-t1] ERROR: failed to restore branch-profile GFS credentials\n' >&2return 1diff --git a/scripts/minikube/full-setup.sh b/scripts/minikube/full-setup.shindex ace16883d..7a1d128b9 100755--- a/scripts/minikube/full-setup.sh+++ b/scripts/minikube/full-setup.sh@@ -981,7 +981,12 @@ else# REUSE_DB / T2 full-reconcile reaches this before pre-gate-sync. A prior T1# can leave gfs_controller_reader NOLOGIN; restore from the committed# Secret DSN (same opt-in as the GFS T1 gate). Still fail-loud on auth.- GFS_RESTORE_ACTIVE_NOLOGIN=true CONTEXT="${PROFILE}" \+ # GFS_RECOVER_ABANDONED_STATE: a timed-out prior setup can leave the+ # reader Secret in rollout-running. This path holds the T2 profile lock,+ # so the prior process is dead; resume that claim instead of asking for+ # a manual retry flag between runs.+ GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \+ CONTEXT="${PROFILE}" \bash "${PROJECT_DIR}/deploy/scripts/reconcile-gfs-deploy-credentials.sh"elselog "Fresh bootstrap detected — reader staging deferred until migrations; GFSC remains fail-closed"@@ -1044,7 +1049,8 @@ else# the split writer/reader templates; reconciling before that lands leaves# the staged reader credential rollout-pending and fails the final verify.CONTEXT="${PROFILE}" bash "${PROJECT_DIR}/deploy/scripts/wait-gfsc-secret-references.sh"- GFS_RESTORE_ACTIVE_NOLOGIN=true CONTEXT="${PROFILE}" \+ GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \+ CONTEXT="${PROFILE}" \bash "${PROJECT_DIR}/deploy/scripts/reconcile-gfs-deploy-credentials.sh"ok "GFS credentials reconciled and writer bootstrap verified"fidiff --git a/scripts/minikube/pre-gate-sync.sh b/scripts/minikube/pre-gate-sync.shindex 312b6e8d6..c4c1676af 100755--- a/scripts/minikube/pre-gate-sync.sh+++ b/scripts/minikube/pre-gate-sync.sh@@ -347,7 +347,10 @@ provision_gfs_serving() {# may restore the role from it — the same contract the GFS T1 gate already# applies on exit. The helper still fails loud when the restored# credential cannot authenticate; no password is invented here.- if ! GFS_RESTORE_ACTIVE_NOLOGIN=true CONTEXT="${PROFILE}" \+ # GFS_RECOVER_ABANDONED_STATE: a dead prior setup can leave+ # rollout-running; this sync holds the T2 lock, so resume that claim.+ if ! GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \+ CONTEXT="${PROFILE}" \bash "${PROJECT_DIR}/deploy/scripts/reconcile-gfs-deploy-credentials.sh"; thenlog "ERROR: gfs DB provisioning FAILED — gfsc cannot authorize any operation. Aborting ${GATE_NAME} pre-gate sync."exit 1diff --git a/scripts/tests/test-minikube-full-setup.sh b/scripts/tests/test-minikube-full-setup.shindex 2e8c0e30e..f6aab554c 100755--- a/scripts/tests/test-minikube-full-setup.sh+++ b/scripts/tests/test-minikube-full-setup.sh@@ -282,7 +282,7 @@ assert_gfs_provisioning_follows_migrations_and_core_readiness() {grep -q 'scale deployment/gfsc-reader --replicas="\$RESET_READER_REPLICAS"' <<<"$reset_block" && \grep -q 'scale deployment/host-context-controller --replicas="\$RESET_HCC_REPLICAS"' <<<"$reset_block" && \! grep -q 'delete pvc control-postgres-data' scripts/minikube/full-setup.sh && \- [[ "$(grep -c 'GFS_RESTORE_ACTIVE_NOLOGIN=true CONTEXT="${PROFILE}"' scripts/minikube/full-setup.sh)" -ge 2 ]] && \+ [[ "$(grep -c 'GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true' scripts/minikube/full-setup.sh)" -ge 2 ]] && \[[ "$core_block" != *"gfs-controller"* ]]; thenpass "full-setup centralizes explicit UID-bound reset and never auto-deletes on WAL logs"elsediff --git a/scripts/tests/test-minikube-t2-contract.sh b/scripts/tests/test-minikube-t2-contract.shindex ea0065fbf..347cb5cae 100755--- a/scripts/tests/test-minikube-t2-contract.sh+++ b/scripts/tests/test-minikube-t2-contract.sh@@ -77,6 +77,7 @@ grep -Fq 'set +x' "$T1" "$PREFLIGHT" "$T2"# cannot leave gfsc-reader unready and fail the exact-head T2 preflight.grep -Fq 'restore_gfs_runtime_credentials' "$T1"grep -Fq 'GFS_RESTORE_ACTIVE_NOLOGIN=true' "$T1"+grep -Fq 'GFS_RECOVER_ABANDONED_STATE=true' "$T1"grep -Fq 'reconcile-gfs-deploy-credentials.sh' "$T1"grep -Fq 'T1_GFS_RESTORE_REQUIRED=true' "$T1"grep -Fq 'T2_UNREADY_DEPLOYMENTS' "$COMMON"@@ -88,13 +89,17 @@ if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'GFS_RESTOecho 'FAIL: provision_gfs_serving does not restore a NOLOGIN GFS role from the committed Secret DSN' >&2exit 1fi+if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'GFS_RECOVER_ABANDONED_STATE=true'; then+ echo 'FAIL: provision_gfs_serving does not resume an interrupted gfsc-reader rollout claim' >&2+ exit 1+fiif ! sed -n '/No cluster sync required before/,$p' "$PRE_GATE" | grep -Fq 'provision_gfs_serving'; thenecho 'FAIL: pre-gate-sync skips GFS serving convergence when no cluster sync is required' >&2exit 1figrep -Fq 'converge_gfs_reader_after_restore' "$PRE_GATE"-if [[ "$(grep -c 'GFS_RESTORE_ACTIVE_NOLOGIN=true CONTEXT="${PROFILE}"' "$ROOT/scripts/minikube/full-setup.sh")" -lt 2 ]]; then- echo 'FAIL: full-setup REUSE_DB path must restore a NOLOGIN GFS role before and after overlay' >&2+if [[ "$(grep -c 'GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true' "$ROOT/scripts/minikube/full-setup.sh")" -lt 2 ]]; then+ echo 'FAIL: full-setup REUSE_DB path must restore a NOLOGIN GFS role and resume an abandoned reader rollout before and after overlay' >&2exit 1fidiff --git a/scripts/tests/test-minikube-t2-scenarios.sh b/scripts/tests/test-minikube-t2-scenarios.shindex 10cfdca11..fe869bff9 100755--- a/scripts/tests/test-minikube-t2-scenarios.sh+++ b/scripts/tests/test-minikube-t2-scenarios.sh@@ -269,8 +269,10 @@ grep -Fq 'complete green reporter' "$ROOT/scripts/e2e/minikube-real-postgres.sh"grep -Fq 'Real PostgreSQL suite failed in $package ($lane)' "$ROOT/scripts/e2e/minikube-real-postgres.sh" && fail 'T1 still treats npm exit as a suite failure before the JSON reporter'grep -Fq 'restore_gfs_runtime_credentials' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 must restore branch-profile GFS credentials on exit'grep -Fq 'GFS_RESTORE_ACTIVE_NOLOGIN=true' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 GFS restore must opt into the NOLOGIN recovery contract'+grep -Fq 'GFS_RECOVER_ABANDONED_STATE=true' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 GFS restore must resume an interrupted gfsc-reader rollout claim'grep -Fq 'GFS_RESTORE_ACTIVE_NOLOGIN=true' "$ROOT/scripts/minikube/pre-gate-sync.sh" || fail 'pre-gate-sync must restore a NOLOGIN GFS role from the committed Secret DSN'-[[ "$(grep -c 'GFS_RESTORE_ACTIVE_NOLOGIN=true CONTEXT="${PROFILE}"' "$ROOT/scripts/minikube/full-setup.sh")" -ge 2 ]] || fail 'full-setup REUSE_DB must restore a NOLOGIN GFS role on both reconcile calls'+grep -Fq 'GFS_RECOVER_ABANDONED_STATE=true' "$ROOT/scripts/minikube/pre-gate-sync.sh" || fail 'pre-gate-sync must resume an interrupted gfsc-reader rollout claim'+[[ "$(grep -c 'GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true' "$ROOT/scripts/minikube/full-setup.sh")" -ge 2 ]] || fail 'full-setup REUSE_DB must restore a NOLOGIN GFS role and resume an abandoned reader rollout on both reconcile calls'grep -Fq 'T2_LOCK_ROOT' "$COMMON"grep -Fq 'trap t2_lock_release EXIT INT TERM' "$COMMON"grep -Fq 'PORT_FORWARD_CONFLICT' "$COMMON"