Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix: close PR 416 review findings

Alfredo Lopez5c68c7cc61cb
diff --git a/.cursor/rules/minikube-t0-t1-t2.mdc b/.cursor/rules/minikube-t0-t1-t2.mdc
index 8c0997ed9..88086635a 100644
--- a/.cursor/rules/minikube-t0-t1-t2.mdc
+++ b/.cursor/rules/minikube-t0-t1-t2.mdc
@@ -42,17 +42,19 @@ repository — never mix its PRs or clusters into this lane.
current-context.
- Bootstrap/reconcile must build the worktree: `IMAGE_SOURCE=local`.
- T1 is fail-loud: unavailable DSN or zero executed tests is FAIL, never green.
- The JSON reporter is the suite verdict; a leftover Vitest exit after a
- complete green reporter is not a failed suite. Role-reset suites use the
- throwaway `postgres:16-alpine`, never shared `control-postgres`.
+ The JSON reporter must be complete and green and the Vitest process must also
+ exit zero. Role-reset suites use the throwaway `postgres:16-alpine`, never
+ shared `control-postgres`.
- Self-healing single run: the orchestrator planner (`T2_PLAN_MODE=true`)
selects `full-reconcile` (naming the deployment) for a bootstrapped profile
with an unready required deployment — never `PROFILE_UNHEALTHY` before a
transition. Standalone preflight and the final exact-head check stay
fail-loud. T1 restores branch-profile GFS credentials on exit, and
`pre-gate-sync` provisions GFS serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true`
- and `GFS_RECOVER_ABANDONED_STATE=true` in every plan, so no manual repair
- script sits between plan and verdict. `settle-gfs-reader-rollout.sh` clears
+ and `GFS_RECOVER_ABANDONED_STATE=true` only in the `minikube-t2` transition;
+ other security gates refresh MCP auth with `--skip-gfs` and do not mutate GFS,
+ so no manual repair script sits between plan and verdict.
+ `settle-gfs-reader-rollout.sh` clears
Ready-reader leftovers (claim, stale ReplicaSets, CrashLoopBackOff pods)
first, and the `gfs-rollout-shim` PATH prefix replaces the reader
`rollout status` wait with a readiness poll because HCC's gfsReconciler
diff --git a/.cursor/rules/minikube-t2-harness.mdc b/.cursor/rules/minikube-t2-harness.mdc
index 3119de28f..861f583d8 100644
--- a/.cursor/rules/minikube-t2-harness.mdc
+++ b/.cursor/rules/minikube-t2-harness.mdc
@@ -36,8 +36,10 @@ Invariants that must never regress:
exit via `reconcile-gfs-deploy-credentials.sh` with
`GFS_RESTORE_ACTIVE_NOLOGIN=true` and `GFS_RECOVER_ABANDONED_STATE=true`
(armed only once suites started), and `pre-gate-sync.sh`
- `provision_gfs_serving` passes the same opt-ins and runs in every sync
- plan, restarting an unready `gfsc-reader` after restore. `full-setup.sh`
+ `provision_gfs_serving` passes the same opt-ins only for the `minikube-t2`
+ transition; other security gates refresh MCP auth with `--skip-gfs` and do
+ not mutate GFS. It restarts an unready `gfsc-reader` after T2 restore.
+ `full-setup.sh`
REUSE_DB / T2 full-reconcile also passes both opt-ins on both GFS
reconcile calls — setup runs before pre-gate-sync and must not abort on a
NOLOGIN reader or a leftover `rollout-running` claim from a dead prior
@@ -52,12 +54,14 @@ Invariants that must never regress:
in `scripts/minikube/wait-gfs-reader-ready.sh` (desired replicas Ready and
no live non-terminating unready reader pod), never a generation wait.
`pre-gate-sync` reader convergence uses the same readiness contract.
+ `scripts/tests/test-minikube-gfs-provision-order.sh` exercises the composed
+ T2 order and proves the non-T2 scope guard.
Both re-run `sync-auth-key.sh` before each GFS reconcile: the overlay
re-applies `gfs-config.jwt-public-key` empty and a reader pod fails
closed without it, so the readiness wait could otherwise only time out.
- T1 stays fail-loud (no green on unavailable DSN or zero tests); the JSON
- reporter is the suite verdict, not a leftover Vitest process exit after a
- complete green reporter. Role-reset suites keep the throwaway Postgres 16.
+ reporter must be complete and green and the Vitest process must also exit
+ zero. Role-reset suites keep the throwaway Postgres 16.
- `t2.sh` keeps `IMAGE_SOURCE=local` for bootstrap/reconcile and calls
`pre-gate-sync` with `--skip-port-forwards`.
- `t2_process_check` accepts only real `kubectl` (`comm=kubectl`) port-forward
diff --git a/.cursor/skills/minikube-t0-t1-t2/SKILL.md b/.cursor/skills/minikube-t0-t1-t2/SKILL.md
index 22962c6da..a089af573 100644
--- a/.cursor/skills/minikube-t0-t1-t2/SKILL.md
+++ b/.cursor/skills/minikube-t0-t1-t2/SKILL.md
@@ -83,8 +83,9 @@ T1 runs inside `make minikube-t2`, or explicitly via
- T1 is fail-loud: an unavailable DSN, an isolated server that did not start,
or zero executed tests is FAIL. Never report green from skipped suites. The
- JSON reporter is the suite verdict; a leftover Vitest process exit after a
- complete green reporter is not a failed suite.
+ JSON reporter must be complete and green, and the Vitest process must also
+ exit zero; a green reporter cannot hide teardown, worker, OOM, or signal
+ failure.
- Role-reset suites (cluster-global role drop/rewrite) run against a
throwaway `postgres:16-alpine`, never the shared `control-postgres`.
- The admin DSN is resolved in-process from the cluster Secret; never print,
diff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.md
index f425bd9dc..a1f8a9b1d 100644
--- a/.cursor/skills/minikube-t0-t1-t2/reference.md
+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md
@@ -33,8 +33,8 @@ Companion to `SKILL.md`. Source of truth: `scripts/minikube/t2.sh`,
`npm ci`, generated ports/profile metadata, or anything secret-like. Run
`make minikube-t2-public-boundary` first.
- **Do not weaken fail-loud T1**: no green on unavailable DSN, zero executed
- tests, or silently skipped suites; the JSON reporter is the suite verdict,
- not a leftover Vitest process exit after a complete green reporter.
+ tests, or silently skipped suites; the JSON reporter must be complete and
+ green, and the Vitest process must also exit zero.
Role-reset suites stay on the throwaway `postgres:16-alpine` (#412), never
shared `control-postgres`.
@@ -63,8 +63,9 @@ and never copy DSNs or credentials between namespaces or into evidence.
The harness owns NOLOGIN recovery and abandoned-rollout resume: T1 restores
branch-profile GFS credentials on exit, and `pre-gate-sync` provisions
serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true` and
-`GFS_RECOVER_ABANDONED_STATE=true` in every plan and restarts an unready
-`gfsc-reader` after restore. A Ready leftover `rollout-running` claim is
+`GFS_RECOVER_ABANDONED_STATE=true` only in the `minikube-t2` transition.
+Other security gates refresh MCP auth with `--skip-gfs` and do not mutate GFS.
+The T2 path restarts an unready `gfsc-reader` after restore. A Ready leftover `rollout-running` claim is
settled by `scripts/minikube/settle-gfs-reader-rollout.sh` before
reconcile: it marks the claim ready, scales leftover non-current unready
reader ReplicaSets to 0, and deletes CrashLoopBackOff reader pods so they
diff --git a/AGENTS.md b/AGENTS.md
index ba679a8cb..a21b3f620 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -98,8 +98,9 @@ has an unready required deployment, the orchestrator planner selects
`PROFILE_UNHEALTHY` before a transition exists; the T1 lane restores
branch-profile GFS credentials on exit and `pre-gate-sync` provisions GFS
serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true` and
-`GFS_RECOVER_ABANDONED_STATE=true` in every plan, so no manual repair
-script belongs between plan and verdict. Harness GFS reconciles settle
+`GFS_RECOVER_ABANDONED_STATE=true` in the `minikube-t2` transition, so other
+security gates do not mutate GFS and no manual repair script belongs between
+plan and verdict. Harness GFS reconciles settle
Ready-reader leftovers first (`settle-gfs-reader-rollout.sh`) and wait on
reader readiness through the `gfs-rollout-shim` PATH prefix instead of a
generation-based `rollout status`, because HCC's gfsReconciler strips the
@@ -108,8 +109,9 @@ the final exact-head T2 check stay fail-loud on an unready deployment.
Real PostgreSQL suites are opt-in in the ordinary test
matrix, but a T1 run that requires them must fail when the database/DSN is
unavailable or when zero tests execute; a green run must never be produced by
-silently skipping the suites. The JSON reporter is the suite verdict; a leftover
-Vitest process exit after a complete green reporter is not a failed suite.
+silently skipping the suites. The JSON reporter must be complete and green,
+and the Vitest process must also exit zero; a green reporter cannot hide a
+teardown, worker, OOM, or signal failure.
Suites that drop or rewrite cluster-global roles must use the harness throwaway
Postgres 16, never the shared `control-postgres`.
diff --git a/CLAUDE.md b/CLAUDE.md
index 2e1346eb0..717a2c1ab 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -16,15 +16,17 @@ evidence rules in `AGENTS.md` and `docs/testing/minikube-t2-runbook.md`.
produced only by the final exact-head preflight inside `make minikube-t2` (or
`make minikube-t2-runtime` after T0 and T1 are already green on the same HEAD).
`make minikube-pre-gate-sync` alone is not T2. Playwright and product E2E
-scripts are separate lanes. T1 judges the JSON reporter (expected files,
-executed/passed, zero failures, zero pending); a leftover Vitest process exit
-after a complete green reporter is not a failed suite. The single run is
+scripts are separate lanes. T1 requires both a complete green JSON reporter
+(expected files, executed/passed, zero failures, zero pending) and a zero
+Vitest process exit; a green reporter cannot hide teardown, worker, OOM, or
+signal failure. The single run is
self-healing: the orchestrator planner selects `full-reconcile` for a
bootstrapped profile with an unready deployment (never `PROFILE_UNHEALTHY`
before a transition), T1 restores branch-profile GFS credentials on exit, and
`pre-gate-sync` provisions GFS serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true`
-and `GFS_RECOVER_ABANDONED_STATE=true` in every plan; do not insert manual
-repair scripts between runs. Harness GFS reconciles settle Ready-reader
+and `GFS_RECOVER_ABANDONED_STATE=true` only in the `minikube-t2` transition;
+other security gates refresh MCP auth without mutating GFS. Do not insert
+manual repair scripts between runs. Harness GFS reconciles settle Ready-reader
leftovers first (`settle-gfs-reader-rollout.sh`) and judge reader rollout
completion by readiness via the `gfs-rollout-shim` PATH prefix, because
HCC's gfsReconciler strips the `restartedAt` annotation and a
diff --git a/deploy/scripts/provision-gfs-runtime.sh b/deploy/scripts/provision-gfs-runtime.sh
index 1853b8873..c4895b1db 100755
--- a/deploy/scripts/provision-gfs-runtime.sh
+++ b/deploy/scripts/provision-gfs-runtime.sh
@@ -189,7 +189,12 @@ finalize_credentials_after_overlay() {
wait_for_gfsc_secret_references
log "Finalizing staged GFS credentials after the overlay"
- CONTEXT="$CONTEXT" bash "$ROOT/deploy/scripts/reconcile-gfs-deploy-credentials.sh"
+ # This entrypoint has already validated the target context (including the
+ # explicit --allow-prod path). Carry that decision into the credential
+ # mutator as an exact, single-context remote authorization instead of
+ # relying on a gke_* name prefix.
+ GFS_REMOTE_RECONCILE_AUTHORIZED=true ALLOWED_CONTEXTS="$CONTEXT" \
+ CONTEXT="$CONTEXT" bash "$ROOT/deploy/scripts/reconcile-gfs-deploy-credentials.sh"
log "Waiting for the exact GFSC writer and reader rollouts"
kctl -n gfs rollout status deployment/gfsc-writer --timeout=240s
diff --git a/deploy/scripts/reconcile-gfs-deploy-credentials.sh b/deploy/scripts/reconcile-gfs-deploy-credentials.sh
index a9a718a21..daede2656 100755
--- a/deploy/scripts/reconcile-gfs-deploy-credentials.sh
+++ b/deploy/scripts/reconcile-gfs-deploy-credentials.sh
@@ -18,6 +18,21 @@ if [ -z "$T2_PROFILE" ]; then T2_PROFILE="$MINIKUBE_PROFILE"; fi
if [ -z "$T2_PROFILE" ]; then T2_PROFILE="$CONTEXT"; fi
if [ -z "$T2_CONTEXT" ]; then T2_CONTEXT="$CONTROL_API_REAL_PG_CONTEXT"; fi
if [ -z "$T2_CONTEXT" ]; then T2_CONTEXT="$CONTEXT"; fi
+
+remote_context_allowed() {
+ local allowed_context
+ local -a allowed_contexts=()
+ [ "${GFS_REMOTE_RECONCILE_AUTHORIZED:-false}" = true ] || return 1
+ [ -n "${ALLOWED_CONTEXTS:-}" ] || return 1
+ IFS=',' read -r -a allowed_contexts <<<"${ALLOWED_CONTEXTS}"
+ for allowed_context in "${allowed_contexts[@]}"; do
+ if [ "$allowed_context" = "$CONTEXT" ]; then
+ return 0
+ fi
+ done
+ return 1
+}
+
if [ -n "$T2_LOCK_TOKEN" ] || [ -n "$MINIKUBE_PROFILE" ] || [[ "$CONTEXT" == clerum-* ]]; then
# shellcheck source=scripts/minikube/t2-common.sh
source "$ROOT/scripts/minikube/t2-common.sh"
@@ -38,9 +53,13 @@ else
case "$CONTEXT" in
gke_*)
# GKE deploys are owned by the explicitly selected deploy workflow rather
- # than the local Minikube lease. Keep this exception narrow: an arbitrary
- # CONTEXT must never turn this credential mutator into a remote-cluster
- # command without either the local fence or the deploy-context shape.
+ # than the local Minikube lease. The workflow must opt in explicitly and
+ # pass the exact context allowlist; the name prefix alone is not
+ # authorization to mutate a remote cluster.
+ if ! remote_context_allowed; then
+ printf '[reconcile-gfs-deploy] ERROR: refusing remote context without explicit authorization and exact ALLOWED_CONTEXTS membership: %s\n' "$CONTEXT" >&2
+ exit 1
+ fi
;;
*)
printf '[reconcile-gfs-deploy] ERROR: refusing unverified Kubernetes context: %s\n' "$CONTEXT" >&2
diff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.md
index 5bc2f6698..a691dc48d 100644
--- a/docs/testing/minikube-t2-runbook.md
+++ b/docs/testing/minikube-t2-runbook.md
@@ -158,17 +158,37 @@ when `gfs-config.jwt-public-key` is empty — the overlay re-applies the base
ConfigMap with an empty value — so `full-setup.sh` and `pre-gate-sync` re-run
`scripts/minikube/sync-auth-key.sh` before each GFS reconcile; otherwise no
new reader pod can start and the readiness wait can only time out.
-`pre-gate-sync`
-provisions GFS serving with the same opt-ins in every sync plan (including
-"no cluster sync required") and, after a successful restore, restarts an
-unready `gfsc-reader`, deletes its live unready pods once, and waits on the
-same readiness contract. `full-setup.sh` on the REUSE_DB / T2 full-reconcile
+`pre-gate-sync` provisions GFS serving with the same opt-ins only in the
+`minikube-t2` transition (including its "no cluster sync required" fast path).
+Other platform security gates refresh MCP auth with `--skip-gfs` and do not
+mutate the GFS plane. After a successful T2 restore, it restarts an unready
+`gfsc-reader`, deletes its live unready pods once, and waits on the same
+readiness contract. `full-setup.sh` on the REUSE_DB / T2 full-reconcile
path passes the same opt-ins on both GFS reconcile calls, because setup runs
before pre-gate-sync and must not abort on a NOLOGIN reader or an abandoned
reader rollout. The recovery helper restores a
NOLOGIN role only from the committed Secret DSN and still fails loud when
-that credential cannot authenticate; no password is ever invented and no DSN
-is printed.
+that credential cannot authenticate; a missing or unreadable required Secret
+also fails the T1 cleanup instead of producing a green run. No password is
+ever invented and no DSN is printed.
+
+The composed GFS recovery decision table is:
+
+| Observed state | Authoritative action | Safety boundary |
+| --- | --- | --- |
+| `gfs-config` or the GFS reader Deployment is absent | Skip GFS recovery | The profile has no adopted GFS serving plane. |
+| Reader is Ready and the reader Secret says `rollout-running` | `settle-gfs-reader-rollout.sh` marks the claim ready | Preserve the Ready reader; do not trigger a second restart. |
+| A non-current reader ReplicaSet has no Ready pod, including an empty `readyReplicas` field | Scale that ReplicaSet to zero | Never scale the current revision; leave terminating pods alone. |
+| A live reader pod is `CrashLoopBackOff`, unready, and not terminating | Delete that pod once | Kubelet backoff is reset so it can re-read restored credentials. |
+| Desired reader replicas are not Ready, or a live non-terminating reader is unready | Reconcile with the GFS rollout shim, then converge and wait | Do not mark the credential rollout ready from an unready observation. |
+| Source auth Secret/key or GFS ConfigMap is missing/empty during a strict GFS sync | Fail before any consumer patch or restart | `--require-gfs` makes an empty source fail closed. |
+| Caller supplies an explicit reader rollout timeout | Honor that timeout; an omitted timeout defaults to 600 seconds | The shim never silently floors a caller's fail-fast timeout. |
+| Gate is not `minikube-t2` | Refresh MCP auth with `--skip-gfs` only | Security gates cannot patch GFS Secrets/ConfigMaps or delete/restart GFS pods. |
+
+The executable composition contract in
+`scripts/tests/test-minikube-gfs-provision-order.sh` verifies the T2 order
+`sync-auth-key → settle → reconcile-with-shim-on-PATH → converge` and the
+non-T2 scope guard.
## T0, T1, and T2 boundaries
@@ -178,9 +198,10 @@ is printed.
`control-postgres`, except the role-reset suites which use an isolated
Postgres 16; the lane reports `PASS`, `FAIL`, `SKIPPED`, and `NOT_RUN`
separately and fails on an unavailable DSN, an isolated server that did not
- start, or zero executed tests. The JSON reporter is the suite verdict
- (expected files, executed/passed, zero failures, zero pending); a leftover
- Vitest process exit after a complete green reporter is not a failed suite.
+ start, or zero executed tests. The JSON reporter must be complete and green
+ (expected files, executed/passed, zero failures, zero pending), and the
+ Vitest process must also exit zero; a green reporter cannot hide teardown,
+ worker, OOM, or signal failure.
* **T2** — the final exact-head preflight inside `make minikube-t2` (or
`make minikube-t2-runtime`): the marker matches this worktree/`HEAD`, the
image manifest is current, PostgreSQL and required namespaces/Services are
diff --git a/scripts/e2e/e2e-hcc-rollout-readiness.sh b/scripts/e2e/e2e-hcc-rollout-readiness.sh
index a3f3a66a8..aa1b550f7 100755
--- a/scripts/e2e/e2e-hcc-rollout-readiness.sh
+++ b/scripts/e2e/e2e-hcc-rollout-readiness.sh
@@ -228,6 +228,8 @@ botched_image_ref() {
# A failed pod-UID listing must not count as "old pod terminated".
assert_botched_outage() {
local flavor="$1" old_uid_query
+ local image_fail old_pod_ok old_pod_fail replacement_fail
+ local transitions_ok transitions_fail streak_ok streak_fail
case "$flavor" in
stuck | recovery) ;;
*) die "assert_botched_outage: flavor must be stuck or recovery, got '${flavor}'" ;;
@@ -246,51 +248,45 @@ assert_botched_outage() {
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.uid}{"\t"}{.status.phase}{"\t"}{.status.containerStatuses[*].state.waiting.reason}{"\n"}{end}' 2>/dev/null || true)"
replacement_row="$(awk -F '\t' -v old="$OLD_POD_UID" '$2 != old && $2 != "" { print; exit }' <<<"$stuck_pod_rows")"
+ case "$flavor" in
+ recovery)
+ image_fail="deployment image is '${deployed_image_now:-unset}', not the injected ${BOTCHED_IMAGE}"
+ old_pod_ok="Recreate terminated the old HEALTHY pod before any viable replacement existed"
+ old_pod_fail="old pod ${OLD_POD_NAME} is still alive under a Recreate rollout"
+ replacement_fail="no replacement pod pinned in ErrImagePull|ImagePullBackOff (rows: ${stuck_pod_rows:-none})"
+ transitions_ok="no 503->200 transition closed the outage before undo — Kubernetes did NOT self-heal"
+ transitions_fail="the outage closed on its own (${stuck_transitions} 503->200 transition(s)) before undo"
+ streak_ok="SUSTAINED outage before undo: ${stuck_maxstreak}s continuous 503 (> ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget) — undo recovers D1b, not a blip"
+ streak_fail="outage lasted only ${stuck_maxstreak}s and did not exceed the ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget — undo would recover a blip, not D1b"
+ ;;
+ stuck)
+ image_fail="deployment image is '${deployed_image_now:-unset}', not the injected ${BOTCHED_IMAGE} — the outage (if any) has a different cause"
+ old_pod_ok="Recreate terminated the old HEALTHY pod before any viable replacement existed — zero fallback replicas (the D1b mechanism)"
+ old_pod_fail="old pod ${OLD_POD_NAME} is still alive under a Recreate rollout — the terminate-before-create premise did not hold"
+ replacement_fail="no replacement pod pinned in ErrImagePull|ImagePullBackOff (rows: ${stuck_pod_rows:-none}) — the outage is not attributable to the injected bad rollout"
+ transitions_ok="no 503->200 transition ever closed the outage — Kubernetes did NOT self-heal the botched Recreate rollout"
+ transitions_fail="the outage closed on its own (${stuck_transitions} 503->200 transition(s)) — the D1b premise that recovery requires an explicit rollback is refuted; investigate before trusting this gate"
+ streak_ok="SUSTAINED outage measured: ${stuck_maxstreak}s continuous 503 (> ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget) and still open at observation end — recovery requires an explicit rollback (infra-owned rollout undo --to-revision in production; cleanup's set image restore here) (D1b reproduced)"
+ streak_fail="outage lasted only ${stuck_maxstreak}s and did not exceed the ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget — the botched rollout did not produce the D1b total outage"
+ ;;
+ esac
+
[ "$stuck_total" -gt 0 ] && ok "sampled /ready ${stuck_total}x across the botched rollout" ||
fail "ZERO /ready samples across the botched rollout — the sampler never ran, and a zero-sample run must never pass"
- if [ "$flavor" = recovery ]; then
- [ "$deployed_image_now" = "$BOTCHED_IMAGE" ] &&
- ok "the botched image is what the Deployment is actually rolling (${BOTCHED_IMAGE})" ||
- fail "deployment image is '${deployed_image_now:-unset}', not the injected ${BOTCHED_IMAGE}"
- [ "$old_uid_live" = 0 ] &&
- ok "Recreate terminated the old HEALTHY pod before any viable replacement existed" ||
- fail "old pod ${OLD_POD_NAME} is still alive under a Recreate rollout"
- if [ -n "$replacement_row" ] && grep -Eq 'ErrImagePull|ImagePullBackOff' <<<"$replacement_row"; then
- ok "replacement pod is pinned by the injected image failure: ${replacement_row}"
- else
- fail "no replacement pod pinned in ErrImagePull|ImagePullBackOff (rows: ${stuck_pod_rows:-none})"
- fi
- [ "$stuck_503" -ge 1 ] &&
- ok "the botched rollout bit the readiness path: ${stuck_503} sample(s) at 503" ||
- fail "zero 503 samples under a botched Recreate rollout — the outage claim is VACUOUS"
- [ "$stuck_transitions" -eq 0 ] &&
- ok "no 503->200 transition closed the outage before undo — Kubernetes did NOT self-heal" ||
- fail "the outage closed on its own (${stuck_transitions} 503->200 transition(s)) before undo"
- [ "$stuck_maxstreak" -gt "$ROLLOUT_DOWNTIME_BUDGET_SEC" ] &&
- ok "SUSTAINED outage before undo: ${stuck_maxstreak}s continuous 503 (> ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget) — undo recovers D1b, not a blip" ||
- fail "outage lasted only ${stuck_maxstreak}s and did not exceed the ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget — undo would recover a blip, not D1b"
+ [ "$deployed_image_now" = "$BOTCHED_IMAGE" ] &&
+ ok "the botched image is what the Deployment is actually rolling (${BOTCHED_IMAGE})" ||
+ fail "$image_fail"
+ [ "$old_uid_live" = 0 ] && ok "$old_pod_ok" || fail "$old_pod_fail"
+ if [ -n "$replacement_row" ] && grep -Eq 'ErrImagePull|ImagePullBackOff' <<<"$replacement_row"; then
+ ok "replacement pod is pinned by the injected image failure: ${replacement_row}"
else
- [ "$deployed_image_now" = "$BOTCHED_IMAGE" ] &&
- ok "the botched image is what the Deployment is actually rolling (${BOTCHED_IMAGE})" ||
- fail "deployment image is '${deployed_image_now:-unset}', not the injected ${BOTCHED_IMAGE} — the outage (if any) has a different cause"
- [ "$old_uid_live" = 0 ] &&
- ok "Recreate terminated the old HEALTHY pod before any viable replacement existed — zero fallback replicas (the D1b mechanism)" ||
- fail "old pod ${OLD_POD_NAME} is still alive under a Recreate rollout — the terminate-before-create premise did not hold"
- if [ -n "$replacement_row" ] && grep -Eq 'ErrImagePull|ImagePullBackOff' <<<"$replacement_row"; then
- ok "replacement pod is pinned by the injected image failure: ${replacement_row}"
- else
- fail "no replacement pod pinned in ErrImagePull|ImagePullBackOff (rows: ${stuck_pod_rows:-none}) — the outage is not attributable to the injected bad rollout"
- fi
- [ "$stuck_503" -ge 1 ] &&
- ok "the botched rollout bit the readiness path: ${stuck_503} sample(s) at 503" ||
- fail "zero 503 samples under a botched Recreate rollout — the outage claim is VACUOUS"
- [ "$stuck_transitions" -eq 0 ] &&
- ok "no 503->200 transition ever closed the outage — Kubernetes did NOT self-heal the botched Recreate rollout" ||
- fail "the outage closed on its own (${stuck_transitions} 503->200 transition(s)) — the D1b premise that recovery requires an explicit rollback is refuted; investigate before trusting this gate"
- [ "$stuck_maxstreak" -gt "$ROLLOUT_DOWNTIME_BUDGET_SEC" ] &&
- ok "SUSTAINED outage measured: ${stuck_maxstreak}s continuous 503 (> ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget) and still open at observation end — recovery requires an explicit rollback (infra-owned rollout undo --to-revision in production; cleanup's set image restore here) (D1b reproduced)" ||
- fail "outage lasted only ${stuck_maxstreak}s and did not exceed the ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget — the botched rollout did not produce the D1b total outage"
+ fail "$replacement_fail"
fi
+ [ "$stuck_503" -ge 1 ] &&
+ ok "the botched rollout bit the readiness path: ${stuck_503} sample(s) at 503" ||
+ fail "zero 503 samples under a botched Recreate rollout — the outage claim is VACUOUS"
+ [ "$stuck_transitions" -eq 0 ] && ok "$transitions_ok" || fail "$transitions_fail"
+ [ "$stuck_maxstreak" -gt "$ROLLOUT_DOWNTIME_BUDGET_SEC" ] && ok "$streak_ok" || fail "$streak_fail"
}
print_repair_instructions() {
@@ -317,6 +313,11 @@ EOF
}
write_evidence_artifact() {
+ local rollout_transition="not-observed"
+ if [ -n "$rollout_recovered_at" ] && [ -n "$rollout_t0" ]; then
+ rollout_transition="$((rollout_recovered_at - rollout_t0))s"
+ fi
+
# Diagnostics ONLY — every verdict input is computed before this runs, so
# every command here is deliberately best-effort (same rationale as the
# sibling churn gate: an empty grep under `set -euo pipefail` must never
@@ -331,11 +332,20 @@ write_evidence_artifact() {
echo ""
echo "=== pods ==="
kctl get pods -n "$HCC_NS" -l "app=${HCC_DEPLOY}" -o wide || true
- echo "=== pod identity: old ${OLD_POD_NAME:-none} (${OLD_POD_UID:-?}) -> new ${NEW_POD_NAME:-none} (${NEW_POD_UID:-?}) restarts=${NEW_POD_RESTARTS:-?} ==="
echo "=== baseline: samples=${baseline_total} 200=${baseline_200} 503=${baseline_503} ==="
- echo "=== rollout: samples=${roll_total} 200=${roll_200} 503=${roll_503} MEASURED-window=${roll_maxstreak}s 503->200=${roll_transitions} budget=${ROLLOUT_DOWNTIME_BUDGET_SEC}s total-transition=${rollout_recovered_at:+$((rollout_recovered_at - rollout_t0))}s ==="
- echo "=== stuck: samples=${stuck_total} 200=${stuck_200} 503=${stuck_503} max-503-streak=${stuck_maxstreak}s 503->200=${stuck_transitions} observe=${STUCK_OBSERVE_SEC}s ==="
- echo "=== post-hold: samples=${hold_total} 503=${hold_503} ==="
+ if [ "$EXPECT_RECOVERY" = 1 ]; then
+ echo "=== recovered pod identity: old ${OLD_POD_NAME:-none} (${OLD_POD_UID:-not-observed}) -> recovered ${NEW_POD_NAME:-none} (${NEW_POD_UID:-not-observed}) restarts=${NEW_POD_RESTARTS:-not-observed} ==="
+ echo "=== recovery revisions: last-good=${LAST_GOOD_REVISION:-not-captured} botched=${live_revision_before_undo:-not-observed} recovered=${recovered_revision:-not-observed} ==="
+ echo "=== pre-undo outage: samples=${stuck_total} 200=${stuck_200} 503=${stuck_503} max-503-streak=${stuck_maxstreak}s 503->200=${stuck_transitions} observe=${STUCK_OBSERVE_SEC}s ==="
+ echo "=== post-undo hold: samples=${hold_total} 503=${hold_503} ==="
+ elif [ "$EXPECT_STUCK" = 1 ]; then
+ echo "=== botched replacement: ${replacement_row:-not-observed} ==="
+ echo "=== stuck outage: samples=${stuck_total} 200=${stuck_200} 503=${stuck_503} max-503-streak=${stuck_maxstreak}s 503->200=${stuck_transitions} observe=${STUCK_OBSERVE_SEC}s ==="
+ else
+ echo "=== pod identity: old ${OLD_POD_NAME:-none} (${OLD_POD_UID:-not-observed}) -> new ${NEW_POD_NAME:-none} (${NEW_POD_UID:-not-observed}) restarts=${NEW_POD_RESTARTS:-not-observed} ==="
+ echo "=== rollout: samples=${roll_total} 200=${roll_200} 503=${roll_503} MEASURED-window=${roll_maxstreak}s 503->200=${roll_transitions} budget=${ROLLOUT_DOWNTIME_BUDGET_SEC}s total-transition=${rollout_transition} ==="
+ echo "=== post-rollout hold: samples=${hold_total} 503=${hold_503} ==="
+ fi
echo "=== /ready series (epoch status phase) ==="
cat "$READY_SERIES" || true
} >"$LOG_ARTIFACT" 2>&1 || true
@@ -416,11 +426,6 @@ OLD_POD_NAME="$(running_hcc_pod)"
[ -n "$OLD_POD_NAME" ] || die "no running HCC pod after a clean baseline"
OLD_POD_UID="$(kctl get pod "$OLD_POD_NAME" -n "$HCC_NS" -o jsonpath='{.metadata.uid}')"
[ -n "$OLD_POD_UID" ] || die "could not pin the pre-rollout pod uid"
-
-
-LAST_GOOD_REVISION="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')"
-[ -n "$LAST_GOOD_REVISION" ] || die "could not capture the last-known-good HCC revision"
-
if [ "$EXPECT_STUCK" = 0 ] && [ "$EXPECT_RECOVERY" = 0 ]; then
# ── FASE D (healthy): Recreate rollout, sampled at 1Hz end to end ──
log "Triggering a Recreate rollout (kubectl rollout restart) and sampling /ready at 1Hz"
@@ -477,6 +482,8 @@ elif [ "$EXPECT_STUCK" = 1 ]; then
write_evidence_artifact
else
# ── FASE D (recovery, evenfire#391): botch, prove the outage, then undo ──
+ LAST_GOOD_REVISION="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')"
+ [ -n "$LAST_GOOD_REVISION" ] || die "could not capture the last-known-good HCC revision"
BOTCHED_IMAGE="$(botched_image_ref)"
PRE_BOTCH_UID="$OLD_POD_UID"
log "Botched rollout (EXPECT_RECOVERY): observing /ready for ${STUCK_OBSERVE_SEC}s"
@@ -505,6 +512,7 @@ else
recovered_revision="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')"
NEW_POD_NAME="$(running_hcc_pod)"
NEW_POD_UID="$(kctl get pod "$NEW_POD_NAME" -n "$HCC_NS" -o jsonpath='{.metadata.uid}')"
+ NEW_POD_RESTARTS="$(hcc_restart_count "$NEW_POD_NAME" || true)"
sample_ready_series "$STABILITY_WINDOW_SEC" post-hold 0
read -r hold_total _ hold_503 _ _ <<<"$(series_metrics post-hold)"
write_evidence_artifact
diff --git a/scripts/e2e/gfs-real-pg-minikube-gate.sh b/scripts/e2e/gfs-real-pg-minikube-gate.sh
index 0654f525c..e743a3afa 100755
--- a/scripts/e2e/gfs-real-pg-minikube-gate.sh
+++ b/scripts/e2e/gfs-real-pg-minikube-gate.sh
@@ -240,7 +240,8 @@ restore_gfs_runtime_credentials() {
# Restore the branch profile before this gate exits so a failed or interrupted
# T1 run cannot leave GFSC in NOLOGIN and poison the following T2 journey.
if ! kc -n gfs get secret gfs-controller-db >/dev/null 2>&1; then
- return 0
+ printf '[gfs-real-pg-minikube] ERROR: required gfs-controller-db Secret is missing or unreadable; refusing to finish with GFS credentials unreconciled\n' >&2
+ return 1
fi
if ! GFS_RESTORE_ACTIVE_NOLOGIN=true CONTEXT="${CONTEXT}" \
bash "${PROJECT_DIR}/deploy/scripts/reconcile-gfs-deploy-credentials.sh"; then
diff --git a/scripts/e2e/minikube-real-postgres.sh b/scripts/e2e/minikube-real-postgres.sh
index c696b1662..b7eae25aa 100755
--- a/scripts/e2e/minikube-real-postgres.sh
+++ b/scripts/e2e/minikube-real-postgres.sh
@@ -68,7 +68,8 @@ restore_gfs_runtime_credentials() {
return 0
fi
if ! t2_kc -n gfs get secret gfs-controller-db >/dev/null 2>&1; then
- return 0
+ printf '[minikube-t1] ERROR: required gfs-controller-db Secret is missing or unreadable; refusing to finish T1 with GFS credentials unreconciled\n' >&2
+ return 1
fi
# Settle a Ready reader's leftover rollout claim first so restore does not
# rollout restart it, and let any wait reconcile still needs judge
diff --git a/scripts/minikube/full-setup.sh b/scripts/minikube/full-setup.sh
index b15335370..a7f661fb1 100755
--- a/scripts/minikube/full-setup.sh
+++ b/scripts/minikube/full-setup.sh
@@ -990,7 +990,7 @@ else
# wiped by the overlay re-apply). Re-sync it from the live platform
# Secret before reconcile so a reader pod can actually start; this is a
# no-op when the key already matches.
- bash "${SCRIPT_DIR}/sync-auth-key.sh" --context="${PROFILE}"
+ bash "${SCRIPT_DIR}/sync-auth-key.sh" --context="${PROFILE}" --require-gfs
# If gfsc-reader is already Ready, settle the leftover claim first so
# reconcile does not rollout restart and race HCC's gfsReconciler.
# The gfs-rollout-shim PATH prefix makes any reader rollout wait inside
@@ -1067,7 +1067,7 @@ else
# The overlay apply above re-declared the base gfs-config with an empty
# jwt-public-key; re-sync it before any reader pod may need to start,
# otherwise the reconcile readiness wait can only time out.
- bash "${SCRIPT_DIR}/sync-auth-key.sh" --context="${PROFILE}"
+ bash "${SCRIPT_DIR}/sync-auth-key.sh" --context="${PROFILE}" --require-gfs
CONTEXT="${PROFILE}" \
bash "${PROJECT_DIR}/scripts/minikube/settle-gfs-reader-rollout.sh"
PATH="${PROJECT_DIR}/scripts/minikube/gfs-rollout-shim:${PATH}" \
@@ -1384,7 +1384,7 @@ done
# ----------------------------------------------------------------------
if $KC get configmap gfs-config -n gfs &>/dev/null; then
log "Provisioning gfs serving (public key → gfs-config + gfs_controller DB login)..."
- if ! bash "${SCRIPT_DIR}/sync-auth-key.sh" --context="${PROFILE}"; then
+ if ! bash "${SCRIPT_DIR}/sync-auth-key.sh" --context="${PROFILE}" --require-gfs; then
err "gfs public-key sync FAILED — gfsc cannot verify tokens. Fix and re-run."
exit 1
fi
diff --git a/scripts/minikube/gfs-rollout-shim/kubectl b/scripts/minikube/gfs-rollout-shim/kubectl
index 1b6749264..8d34ed686 100755
--- a/scripts/minikube/gfs-rollout-shim/kubectl
+++ b/scripts/minikube/gfs-rollout-shim/kubectl
@@ -64,11 +64,15 @@ while [ "$i" -lt "${#args[@]}" ]; do
i=$((i + 1))
done
-if ! timeout_seconds="$(python3 - "$timeout_raw" "${GFS_ROLLOUT_SHIM_MIN_TIMEOUT_SECONDS:-600}" <<'PY'
+# Honor the caller's rollout timeout. The harness caller that needs the
+# readiness default omits --timeout and receives 600s; silently flooring an
+# explicit short timeout would hide a dead reader and invalidate the caller's
+# fail-fast contract.
+if ! timeout_seconds="$(python3 - "$timeout_raw" <<'PY'
import re
import sys
-timeout_raw, minimum_raw = sys.argv[1:]
+timeout_raw = sys.argv[1]
def parse_duration(raw, default, label):
if raw == "":
@@ -83,9 +87,7 @@ def parse_duration(raw, default, label):
raise SystemExit(f"invalid {label}: value must be between 1 and 86400 seconds")
return value
-timeout = parse_duration(timeout_raw, 600, "rollout timeout")
-minimum = parse_duration(minimum_raw, 600, "minimum rollout timeout")
-print(max(timeout, minimum))
+print(parse_duration(timeout_raw, 600, "rollout timeout"))
PY
)"; then
printf '[gfs-rollout-shim] refusing an invalid rollout timeout\n' >&2
diff --git a/scripts/minikube/pre-gate-sync.sh b/scripts/minikube/pre-gate-sync.sh
index 3da798c11..6a482c709 100755
--- a/scripts/minikube/pre-gate-sync.sh
+++ b/scripts/minikube/pre-gate-sync.sh
@@ -133,12 +133,15 @@ restore_workflow_reconciler() {
fi
set +e
log "Restoring workflow reconciler to ${WRC_REPLICAS} replica(s)"
- ${KC} scale deployment/workflow-recipes -n control-plane --replicas="${WRC_REPLICAS}" >/dev/null
- ${KC} rollout status deployment/workflow-recipes -n control-plane --timeout=120s >/dev/null
- local restore_rc=$?
+ local scale_rc=0 rollout_rc=0
+ ${KC} scale deployment/workflow-recipes -n control-plane --replicas="${WRC_REPLICAS}" >/dev/null || scale_rc=$?
+ if [[ "${scale_rc}" -eq 0 ]]; then
+ ${KC} rollout status deployment/workflow-recipes -n control-plane --timeout=120s >/dev/null || rollout_rc=$?
+ fi
set -e
- if [[ "${restore_rc}" -ne 0 ]]; then
- log "WARNING: workflow reconciler did not become Ready during pre-gate cleanup"
+ if [[ "${scale_rc}" -ne 0 || "${rollout_rc}" -ne 0 ]]; then
+ log "ERROR: workflow reconciler restore failed (scale=${scale_rc}, rollout=${rollout_rc}); leaving the writer fence armed"
+ return 1
fi
WRC_FENCED=false
}
@@ -166,12 +169,15 @@ restore_control_api() {
fi
set +e
log "Restoring Control API to ${CONTROL_API_REPLICAS} replica(s)"
- ${KC} scale deployment/control-api -n control-plane --replicas="${CONTROL_API_REPLICAS}" >/dev/null
- ${KC} rollout status deployment/control-api -n control-plane --timeout=120s >/dev/null
- local restore_rc=$?
+ local scale_rc=0 rollout_rc=0
+ ${KC} scale deployment/control-api -n control-plane --replicas="${CONTROL_API_REPLICAS}" >/dev/null || scale_rc=$?
+ if [[ "${scale_rc}" -eq 0 ]]; then
+ ${KC} rollout status deployment/control-api -n control-plane --timeout=120s >/dev/null || rollout_rc=$?
+ fi
set -e
- if [[ "${restore_rc}" -ne 0 ]]; then
- log "WARNING: Control API did not become Ready during pre-gate cleanup"
+ if [[ "${scale_rc}" -ne 0 || "${rollout_rc}" -ne 0 ]]; then
+ log "ERROR: Control API restore failed (scale=${scale_rc}, rollout=${rollout_rc}); leaving the writer fence armed"
+ return 1
fi
CONTROL_API_FENCED=false
}
@@ -324,7 +330,31 @@ sync_mcp_host_auth_key() {
return 0
fi
- bash "${PROJECT_DIR}/scripts/minikube/sync-auth-key.sh" --context "${PROFILE}"
+ bash "${PROJECT_DIR}/scripts/minikube/sync-auth-key.sh" --context "${PROFILE}" --skip-gfs
+}
+
+sync_gfs_auth_key() {
+ bash "${PROJECT_DIR}/scripts/minikube/sync-auth-key.sh" --context "${PROFILE}" --require-gfs
+}
+
+settle_gfs_reader_rollout() {
+ T2_SKIP_LOCK=true T2_LOCK_TOKEN="$T2_LOCK_TOKEN" \
+ T2_PROJECT_DIR="$T2_PROJECT_DIR" MINIKUBE_PROFILE="$T2_PROFILE" \
+ CONTROL_API_REAL_PG_CONTEXT="$T2_CONTEXT" T2_PROFILE_ROOT="$T2_PROFILE_ROOT" \
+ T2_PROFILE_ENV="$T2_PROFILE_ENV" T2_PORTS_ENV="$T2_PORTS_ENV" \
+ CONTEXT="$T2_CONTEXT" \
+ bash "${PROJECT_DIR}/scripts/minikube/settle-gfs-reader-rollout.sh"
+}
+
+reconcile_gfs_credentials() {
+ PATH="${PROJECT_DIR}/scripts/minikube/gfs-rollout-shim:${PATH}" \
+ GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \
+ T2_SKIP_LOCK=true T2_LOCK_TOKEN="$T2_LOCK_TOKEN" \
+ T2_PROJECT_DIR="$T2_PROJECT_DIR" MINIKUBE_PROFILE="$T2_PROFILE" \
+ CONTROL_API_REAL_PG_CONTEXT="$T2_CONTEXT" T2_PROFILE_ROOT="$T2_PROFILE_ROOT" \
+ T2_PROFILE_ENV="$T2_PROFILE_ENV" T2_PORTS_ENV="$T2_PORTS_ENV" \
+ CONTEXT="$T2_CONTEXT" \
+ bash "${PROJECT_DIR}/deploy/scripts/reconcile-gfs-deploy-credentials.sh"
}
converge_gfs_reader_after_restore() {
@@ -332,13 +362,34 @@ converge_gfs_reader_after_restore() {
# already-started pod even after the role is restored. Converging it here
# keeps the repair inside the single orchestrated run instead of a manual
# `kubectl rollout restart deploy/gfsc-reader` side quest.
- local deployment=gfsc-reader desired ready
- if ! ${KC} get deployment "${deployment}" -n gfs >/dev/null 2>&1; then
+ local deployment=gfsc-reader deployment_probe desired ready
+ if ! deployment_probe="$(${KC} get deployment "${deployment}" -n gfs 2>&1)"; then
+ if [[ "${deployment_probe}" == *NotFound* || "${deployment_probe}" == *"not found"* ]]; then
+ return 0
+ fi
+ log "ERROR: unable to inspect gfs/${deployment} before credential convergence"
+ return 1
+ fi
+ if ! desired="$(${KC} get deployment "${deployment}" -n gfs -o jsonpath='{.spec.replicas}' 2>/dev/null)"; then
+ log "ERROR: unable to read desired replicas for gfs/${deployment}"
+ return 1
+ fi
+ if [[ ! "${desired}" =~ ^[0-9]+$ ]]; then
+ log "ERROR: desired replicas for gfs/${deployment} is not numeric"
+ return 1
+ fi
+ if [[ "${desired}" == "0" ]]; then
return 0
fi
- desired="$(${KC} get deployment "${deployment}" -n gfs -o jsonpath='{.spec.replicas}' 2>/dev/null || true)"
- ready="$(${KC} get deployment "${deployment}" -n gfs -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
- if [[ -z "${desired}" || "${desired}" == "0" || "${ready:-0}" == "${desired}" ]]; then
+ if ! ready="$(${KC} get deployment "${deployment}" -n gfs -o jsonpath='{.status.readyReplicas}' 2>/dev/null)"; then
+ log "ERROR: unable to read Ready replicas for gfs/${deployment}"
+ return 1
+ fi
+ if [[ ! "${ready}" =~ ^[0-9]+$ ]]; then
+ log "ERROR: Ready replicas for gfs/${deployment} is not numeric"
+ return 1
+ fi
+ if [[ "${ready}" == "${desired}" ]]; then
return 0
fi
log "Restarting gfs/${deployment} after credential restore (${ready:-0}/${desired} Ready)"
@@ -363,7 +414,25 @@ converge_gfs_reader_after_restore() {
}
provision_gfs_serving() {
+ if [[ "${GATE_NAME}" != "minikube-t2" ]]; then
+ # Other platform security gates must retain their read/verify contract.
+ # GFS credential recovery is a T2 transition because it can patch Secrets,
+ # restart GFSC, delete pods, and resume an abandoned rollout claim.
+ log "Skipping GFS serving mutation before ${GATE_NAME}; only minikube-t2 owns GFS recovery"
+ sync_mcp_host_auth_key
+ return 0
+ fi
if ${KC} get configmap gfs-config -n gfs >/dev/null 2>&1; then
+ local reader_probe
+ if ! reader_probe="$(${KC} get deployment gfsc-reader -n gfs 2>&1)"; then
+ if [[ "${reader_probe}" == *NotFound* || "${reader_probe}" == *"not found"* ]]; then
+ log "Skipping gfs serving provisioning (gfsc-reader not found — GFS reader is not deployed)"
+ sync_mcp_host_auth_key
+ return 0
+ fi
+ log "ERROR: unable to inspect gfs/gfsc-reader before credential recovery"
+ return 1
+ fi
log "Provisioning gfs serving before ${GATE_NAME}"
# FAIL LOUD: with the GFS stack deployed, a broken gfs_controller credential
# means every GFS operation 503s (issue #775). Continuing would burn the
@@ -380,35 +449,28 @@ provision_gfs_serving() {
# (the overlay re-applies the base ConfigMap with an empty value); a
# reader pod cannot start without it and any readiness wait would only
# time out. Re-sync it first; this is a no-op when the key matches.
- bash "${PROJECT_DIR}/scripts/minikube/sync-auth-key.sh" --context "${PROFILE}"
+ sync_gfs_auth_key
# Settle a Ready reader first so reconcile does not restart it and
# race HCC's gfsReconciler during kubectl rollout status. If reconcile
# still needs a reader rollout, the gfs-rollout-shim PATH prefix makes
# that wait judge readiness instead of the template generation HCC keeps
# rewriting.
- T2_SKIP_LOCK=true T2_LOCK_TOKEN="$T2_LOCK_TOKEN" \
- T2_PROJECT_DIR="$T2_PROJECT_DIR" MINIKUBE_PROFILE="$T2_PROFILE" \
- CONTROL_API_REAL_PG_CONTEXT="$T2_CONTEXT" T2_PROFILE_ROOT="$T2_PROFILE_ROOT" \
- T2_PROFILE_ENV="$T2_PROFILE_ENV" T2_PORTS_ENV="$T2_PORTS_ENV" \
- CONTEXT="$T2_CONTEXT" \
- bash "${PROJECT_DIR}/scripts/minikube/settle-gfs-reader-rollout.sh"
- if ! PATH="${PROJECT_DIR}/scripts/minikube/gfs-rollout-shim:${PATH}" \
- GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \
- T2_SKIP_LOCK=true T2_LOCK_TOKEN="$T2_LOCK_TOKEN" \
- T2_PROJECT_DIR="$T2_PROJECT_DIR" MINIKUBE_PROFILE="$T2_PROFILE" \
- CONTROL_API_REAL_PG_CONTEXT="$T2_CONTEXT" T2_PROFILE_ROOT="$T2_PROFILE_ROOT" \
- T2_PROFILE_ENV="$T2_PROFILE_ENV" T2_PORTS_ENV="$T2_PORTS_ENV" \
- CONTEXT="$T2_CONTEXT" \
- bash "${PROJECT_DIR}/deploy/scripts/reconcile-gfs-deploy-credentials.sh"; then
+ if ! settle_gfs_reader_rollout; then
+ log "ERROR: unable to settle the branch-owned GFS reader rollout before credential reconciliation"
+ exit 1
+ fi
+ if ! reconcile_gfs_credentials; then
log "ERROR: gfs DB provisioning FAILED — gfsc cannot authorize any operation. Aborting ${GATE_NAME} pre-gate sync."
exit 1
fi
- converge_gfs_reader_after_restore
+ if ! converge_gfs_reader_after_restore; then
+ log "ERROR: gfs reader did not converge after credential restoration"
+ return 1
+ fi
else
log "Skipping gfs serving provisioning (gfs-config not found — GFS stack not deployed)"
+ sync_mcp_host_auth_key
fi
-
- sync_mcp_host_auth_key
}
assert_no_legacy_prompt_bridge_grants() {
@@ -568,9 +630,8 @@ if [[ "${cluster_changed}" == "true" ]]; then
if ! incremental_requires_database_reconcile; then
# No schema work is planned, but a role-reset T1 run may still have left
# the cluster-global GFS runtime roles NOLOGIN while every image and
- # manifest matched. Serving provisioning is idempotent, so converge it in
- # every sync plan instead of leaving an unready gfsc-reader to fail the
- # exact-head T2 preflight after this sync reports success.
+ # manifest matched. The T2 transition owns this idempotent convergence;
+ # security-gate invocations return through the non-mutating guard above.
provision_gfs_serving
fi
@@ -657,8 +718,8 @@ else
log "No cluster sync required before ${GATE_NAME}"
# Images and manifests match, yet the GFS runtime credentials can still be
# broken (a role-reset T1 leaves the reader NOLOGIN without changing any
- # fingerprint). Converge serving credentials on every pre-gate pass so a
- # single orchestrated run repairs them without a manual script.
+ # fingerprint). The T2 fast path repairs them in this single orchestrated
+ # run; non-T2 gates are fenced out by provision_gfs_serving.
provision_gfs_serving
assert_no_legacy_prompt_bridge_grants
ensure_evenfire_registry
diff --git a/scripts/minikube/sync-auth-key.sh b/scripts/minikube/sync-auth-key.sh
index ad519aeb7..661e9b978 100755
--- a/scripts/minikube/sync-auth-key.sh
+++ b/scripts/minikube/sync-auth-key.sh
@@ -3,6 +3,7 @@ set -euo pipefail
PROFILE="clerum-test"
REQUIRE_GFS=false
+SYNC_GFS=true
while [[ $# -gt 0 ]]; do
case "$1" in
@@ -21,8 +22,14 @@ while [[ $# -gt 0 ]]; do
REQUIRE_GFS=true
shift
;;
+ --skip-gfs)
+ # Pre-gate security lanes need the MCP host key refresh without mutating
+ # the optional GFS plane. The T2 GFS path uses --require-gfs instead.
+ SYNC_GFS=false
+ shift
+ ;;
-h|--help)
- echo "Usage: scripts/minikube/sync-auth-key.sh [--context <kubectl-context>] [--require-gfs]" >&2
+ echo "Usage: scripts/minikube/sync-auth-key.sh [--context <kubectl-context>] [--require-gfs|--skip-gfs]" >&2
exit 0
;;
*)
@@ -32,6 +39,11 @@ while [[ $# -gt 0 ]]; do
esac
done
+if [[ "${REQUIRE_GFS}" == "true" && "${SYNC_GFS}" != "true" ]]; then
+ echo "--require-gfs cannot be combined with --skip-gfs" >&2
+ exit 2
+fi
+
KCTL=(kubectl --context="${PROFILE}")
SOURCE_SECRET="rpc-proxy-secrets"
SOURCE_NAMESPACE="rpc-proxy"
@@ -97,8 +109,11 @@ if ! "${KCTL[@]}" get secret "${SOURCE_SECRET}" -n "${SOURCE_NAMESPACE}" >/dev/n
exit 0
fi
source_key="$("${KCTL[@]}" get secret "${SOURCE_SECRET}" -n "${SOURCE_NAMESPACE}" -o "jsonpath={.data.${SOURCE_KEY}}" | base64 -d)"
-if [[ "${REQUIRE_GFS}" == "true" && -z "${source_key}" ]]; then
- die "required GFS auth source key ${SOURCE_KEY} is empty"
+if [[ -z "${source_key}" ]]; then
+ if [[ "${REQUIRE_GFS}" == "true" ]]; then
+ die "required GFS auth source key ${SOURCE_KEY} is empty"
+ fi
+ die "auth source key ${SOURCE_KEY} is empty; refusing to mutate consumers"
fi
if [[ "${REQUIRE_GFS}" == "true" ]] && \
! "${KCTL[@]}" get configmap "${GFS_CONFIGMAP}" -n "${GFS_NAMESPACE}" >/dev/null 2>&1; then
@@ -168,6 +183,8 @@ sync_gfs() {
}
sync_mcp_host
-sync_gfs
+if [[ "${SYNC_GFS}" == "true" ]]; then
+ sync_gfs
+fi
log "Auth key synced"
diff --git a/scripts/minikube/t2-common.sh b/scripts/minikube/t2-common.sh
index 7b1ed9018..b25186628 100755
--- a/scripts/minikube/t2-common.sh
+++ b/scripts/minikube/t2-common.sh
@@ -489,7 +489,7 @@ t2_deployment_check() {
T2_NEXT_COMMAND="MINIKUBE_PROFILE=$T2_PROFILE make minikube-setup-local"
t2_fail PROFILE_UNHEALTHY 'deployment readiness inventory is unavailable'
fi
- unready="$(python3 - "$deployment_json" <<'PY'
+ if ! unready="$(python3 - "$deployment_json" <<'PY'
import json
import sys
payload = json.loads(sys.argv[1])
@@ -507,7 +507,15 @@ for item in payload.get("items", []):
bad.append("%s/%s %s/%s" % (metadata.get("namespace", "?"), metadata.get("name", "?"), ready, desired))
print("; ".join(bad))
PY
- )"
+ )"; then
+ if [ "$T2_BOOTSTRAP_REQUIRED" = true ]; then
+ T2_PLAN_STATE=full-bootstrap
+ T2_PLAN_REASON='deployment readiness inventory is invalid'
+ return 0
+ fi
+ T2_NEXT_COMMAND="MINIKUBE_PROFILE=$T2_PROFILE make minikube-t2"
+ t2_fail PROFILE_UNHEALTHY 'deployment readiness inventory is invalid'
+ fi
if [ -n "$unready" ]; then
if [ "$T2_BOOTSTRAP_REQUIRED" = true ]; then
T2_PLAN_STATE=full-bootstrap
diff --git a/scripts/tests/test-gfs-deploy-contract.sh b/scripts/tests/test-gfs-deploy-contract.sh
index 7cd1e40b3..8a12acfb1 100755
--- a/scripts/tests/test-gfs-deploy-contract.sh
+++ b/scripts/tests/test-gfs-deploy-contract.sh
@@ -51,6 +51,19 @@ fi
grep -Fq 'refusing unverified Kubernetes context' "$reconcile_context_err" \
|| fail 'credential reconciliation did not explain its unverified-context rejection'
rm -f "$reconcile_context_err"
+reconcile_remote_context_err="$(mktemp)"
+if CONTEXT=gke_unapproved GFS_REMOTE_RECONCILE_AUTHORIZED=true ALLOWED_CONTEXTS=gke_approved \
+ bash deploy/scripts/reconcile-gfs-deploy-credentials.sh 2>"$reconcile_remote_context_err"; then
+ rm -f "$reconcile_remote_context_err"
+ fail 'credential reconciliation accepted a GKE context outside the explicit allowlist'
+fi
+grep -Fq 'explicit authorization and exact ALLOWED_CONTEXTS membership' "$reconcile_remote_context_err" \
+ || fail 'credential reconciliation did not fail closed for an unauthorized GKE context'
+rm -f "$reconcile_remote_context_err"
+grep -Fq 'GFS_REMOTE_RECONCILE_AUTHORIZED' deploy/scripts/reconcile-gfs-deploy-credentials.sh \
+ || fail 'remote GFS reconciliation lacks an explicit authorization gate'
+grep -Fq 'ALLOWED_CONTEXTS' deploy/scripts/reconcile-gfs-deploy-credentials.sh \
+ || fail 'remote GFS reconciliation lacks an exact context allowlist'
make_block() {
awk -v target="$1" '$0 ~ "^" target ":" {active=1} active && /^\.PHONY:/ {exit} active {print}' Makefile
}
diff --git a/scripts/tests/test-gfs-real-pg-minikube-gate.sh b/scripts/tests/test-gfs-real-pg-minikube-gate.sh
index 3a1e47640..99c89b2a0 100755
--- a/scripts/tests/test-gfs-real-pg-minikube-gate.sh
+++ b/scripts/tests/test-gfs-real-pg-minikube-gate.sh
@@ -16,6 +16,7 @@ grep -Fq "gfs*.realPostgres.integration.test.ts" "${SCRIPT}"
grep -Fq "services.rateLimiter.realPostgres.integration.test.ts" "${SCRIPT}"
grep -Fq 'isolated processes' "${SCRIPT}"
grep -Fq 'restore_gfs_runtime_credentials' "${SCRIPT}"
+grep -Fq 'required gfs-controller-db Secret is missing or unreadable' "${SCRIPT}"
grep -Fq 'GFS_RESTORE_ACTIVE_NOLOGIN=true' "${SCRIPT}"
grep -Fq 'failed to restore branch-profile GFS credentials' "${SCRIPT}"
grep -Fq 'gke_*)' "${SCRIPT}"
diff --git a/scripts/tests/test-hcc-rollout-readiness-gate.sh b/scripts/tests/test-hcc-rollout-readiness-gate.sh
index 4478693ed..201df98c7 100755
--- a/scripts/tests/test-hcc-rollout-readiness-gate.sh
+++ b/scripts/tests/test-hcc-rollout-readiness-gate.sh
@@ -127,6 +127,7 @@ grep -Fq 'create_synthetic_fleet' "$GATE" &&
# is the same sustained D1b pin as EXPECT_STUCK, IMAGE_BROKEN is cleared
# only AFTER every post-undo proof, and a no-op undo fails.
recovery_block="$(awk '/FASE D \(recovery, evenfire#391\)/,0' "$GATE")"
+stuck_block="$(awk '/FASE D \(botched, D1b\)/,/^else$/' "$GATE")"
# The sustained-outage comparison lives in the shared assert_botched_outage
# helper; pin the LITERAL test expression in the helper body (a comment
# mentioning the variables must never satisfy this check) plus the recovery
@@ -149,7 +150,88 @@ grep -Fq 'EXPECT_RECOVERY' "$GATE" &&
pass "EXPECT_RECOVERY undoes to last-good outside cleanup and fails a no-op" ||
fail "EXPECT_RECOVERY is missing, shares cleanup, skips sustained outage, or clears IMAGE_BROKEN before proofs"
-# 13. Makefile entry point: the gate is reachable through the same shaped
+# 13. The stuck and recovery modes share one implementation of every D1b
+# predicate. Only their explanatory messages differ, so one mode cannot
+# silently lose a correctness pin while the other keeps it.
+[ "$(grep -Fc '[ "$deployed_image_now" = "$BOTCHED_IMAGE" ]' <<<"$outage_helper")" = 1 ] &&
+ [ "$(grep -Fc '[ "$old_uid_live" = 0 ]' <<<"$outage_helper")" = 1 ] &&
+ [ "$(grep -Fc "grep -Eq 'ErrImagePull|ImagePullBackOff'" <<<"$outage_helper")" = 1 ] &&
+ [ "$(grep -Fc '[ "$stuck_503" -ge 1 ]' <<<"$outage_helper")" = 1 ] &&
+ [ "$(grep -Fc '[ "$stuck_transitions" -eq 0 ]' <<<"$outage_helper")" = 1 ] &&
+ [ "$(grep -Fc '[ "$stuck_maxstreak" -gt "$ROLLOUT_DOWNTIME_BUDGET_SEC" ]' <<<"$outage_helper")" = 1 ] &&
+ grep -Fq 'assert_botched_outage stuck' <<<"$stuck_block" &&
+ grep -Fq 'assert_botched_outage recovery' <<<"$recovery_block" &&
+ pass "stuck and recovery modes share one complete D1b assertion implementation" ||
+ fail "assert_botched_outage duplicates a predicate or one mode bypasses the shared D1b pins"
+
+# 14. Evidence is mode-aware: each mode emits only the samples it collected,
+# and an unavailable restart count is explicit rather than the ambiguous
+# `restarts=?` placeholder.
+evidence_helper="$(awk '/^write_evidence_artifact\(\) \{/,/^\}/' "$GATE")"
+render_evidence_mode() (
+ local mode="$1"
+ eval "$evidence_helper"
+ EXPECT_STUCK=0 EXPECT_RECOVERY=0
+ case "$mode" in
+ stuck) EXPECT_STUCK=1 ;;
+ recovery) EXPECT_RECOVERY=1 ;;
+ esac
+ WITH_SYNTHETIC_FLEET=0 FLEET_CONTEXTS=24 FLEET_MCPSERVERS=115 FLEET_HOSTS=8
+ HCC_DEPLOY=host-context-controller HCC_NS=control-plane
+ OLD_POD_NAME=old-pod OLD_POD_UID=uid-old
+ NEW_POD_NAME=new-pod NEW_POD_UID=uid-new NEW_POD_RESTARTS=0
+ LAST_GOOD_REVISION=7 live_revision_before_undo=8 recovered_revision=9
+ replacement_row=$'botched-pod\tuid-botched\tPending\tImagePullBackOff'
+ baseline_total=10 baseline_200=10 baseline_503=0
+ roll_total=3 roll_200=2 roll_503=1 roll_maxstreak=1 roll_transitions=1
+ stuck_total=4 stuck_200=0 stuck_503=4 stuck_maxstreak=4 stuck_transitions=0
+ hold_total=2 hold_503=0
+ ROLLOUT_DOWNTIME_BUDGET_SEC=120 STUCK_OBSERVE_SEC=210
+ rollout_t0=100 rollout_recovered_at=103
+ READY_SERIES=/dev/null
+ LOG_ARTIFACT="$(mktemp "${TMPDIR:-/tmp}/hcc-rollout-evidence-test.XXXXXX")"
+ trap 'rm -f "$LOG_ARTIFACT"' EXIT
+ kctl() { return 0; }
+ write_evidence_artifact
+ cat "$LOG_ARTIFACT"
+)
+healthy_evidence="$(render_evidence_mode healthy)"
+stuck_evidence="$(render_evidence_mode stuck)"
+recovery_evidence="$(render_evidence_mode recovery)"
+grep -Fq 'pre-undo outage: samples=' <<<"$recovery_evidence" &&
+ grep -Fq 'post-undo hold: samples=' <<<"$recovery_evidence" &&
+ grep -Fq 'restarts=0' <<<"$recovery_evidence" &&
+ ! grep -Fq '=== rollout: samples=' <<<"$recovery_evidence" &&
+ grep -Fq 'stuck outage: samples=' <<<"$stuck_evidence" &&
+ ! grep -Fq 'restarts=' <<<"$stuck_evidence" &&
+ ! grep -Fq '=== rollout: samples=' <<<"$stuck_evidence" &&
+ grep -Fq '=== rollout: samples=' <<<"$healthy_evidence" &&
+ grep -Fq 'post-rollout hold: samples=' <<<"$healthy_evidence" &&
+ grep -Fq 'restarts=0' <<<"$healthy_evidence" &&
+ ! grep -Fq 'stuck outage: samples=' <<<"$healthy_evidence" &&
+ ! grep -Fq 'samples=0' <<<"${healthy_evidence}${stuck_evidence}${recovery_evidence}" &&
+ ! grep -Fq 'restarts=?' <<<"${healthy_evidence}${stuck_evidence}${recovery_evidence}" &&
+ ! grep -Fq 'restarts=${NEW_POD_RESTARTS:-?}' <<<"$evidence_helper" &&
+ pass "evidence reports only mode-relevant samples and never emits restarts=?" ||
+ fail "evidence mixes uncollected mode counters or retains an ambiguous restart placeholder"
+
+# 15. Recovery-only state stays in the recovery branch: capture last-good
+# before injecting the bad image, then record the recovered pod's restart
+# count after undo and before writing evidence.
+last_good_capture='LAST_GOOD_REVISION="$(kctl get deployment "$HCC_DEPLOY"'
+capture_line="$(grep -nF "$last_good_capture" <<<"$recovery_block" | head -1 | cut -d: -f1)"
+botch_line="$(grep -nF 'kctl set image deployment/"$HCC_DEPLOY"' <<<"$recovery_block" | head -1 | cut -d: -f1)"
+undo_line="$(grep -nF 'kctl rollout undo deployment/"$HCC_DEPLOY"' <<<"$recovery_block" | head -1 | cut -d: -f1)"
+restart_line="$(grep -nF 'NEW_POD_RESTARTS="$(hcc_restart_count "$NEW_POD_NAME" || true)"' <<<"$recovery_block" | head -1 | cut -d: -f1)"
+evidence_line="$(grep -nF 'write_evidence_artifact' <<<"$recovery_block" | head -1 | cut -d: -f1)"
+[ "$(grep -Fc "$last_good_capture" "$GATE")" = 1 ] &&
+ [ -n "$capture_line" ] && [ -n "$botch_line" ] && [ "$capture_line" -lt "$botch_line" ] &&
+ [ -n "$undo_line" ] && [ -n "$restart_line" ] && [ -n "$evidence_line" ] &&
+ [ "$restart_line" -gt "$undo_line" ] && [ "$restart_line" -lt "$evidence_line" ] &&
+ pass "recovery alone captures last-good and records recovered pod restarts after undo" ||
+ fail "last-good/restart evidence escaped recovery scope or occurs on the wrong side of undo"
+
+# 16. Makefile entry point: the gate is reachable through the same shaped
# target as the sibling HCC gates — fault-injection ack + pre-gate-sync
# expectation + explicit profile=context pin, with the exclusive
# EXPECT_STUCK/EXPECT_RECOVERY modes surfaced (and documented as
@@ -168,7 +250,7 @@ grep -Fq '.PHONY: test-e2e-hcc-rollout-readiness' "$MAKEFILE" &&
pass "Makefile target test-e2e-hcc-rollout-readiness keeps ack + profile=context + modes + gate invocation" ||
fail "Makefile target test-e2e-hcc-rollout-readiness is missing or lost its ack/profile/mode/gate contract"
-# 14. Executable guard suite: CI must RUN the cluster-free guard tests (not
+# 17. Executable guard suite: CI must RUN the cluster-free guard tests (not
# only this grep contract), and that suite must actually EXECUTE the gate
# — a guards script degraded back to grep would be this file twice.
GUARDS="${ROOT}/scripts/tests/test-hcc-rollout-readiness-guards.sh"
diff --git a/scripts/tests/test-minikube-gfs-provision-order.sh b/scripts/tests/test-minikube-gfs-provision-order.sh
new file mode 100755
index 000000000..422d0adcd
--- /dev/null
+++ b/scripts/tests/test-minikube-gfs-provision-order.sh
@@ -0,0 +1,118 @@
+#!/usr/bin/env bash
+# Executable contract for the composed GFS pre-gate recovery sequence.
+set -euo pipefail
+
+ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
+PRE_GATE="$ROOT/scripts/minikube/pre-gate-sync.sh"
+fail() { printf 'FAIL: %s\n' "$*" >&2; exit 1; }
+
+tmp="$(mktemp -d)"
+cleanup() { rm -rf "$tmp"; }
+trap cleanup EXIT
+
+fake_project="$tmp/project"
+mkdir -p "$fake_project/scripts/minikube" "$fake_project/deploy/scripts" \
+ "$fake_project/scripts/minikube/gfs-rollout-shim"
+order_file="$tmp/order.log"
+
+cat >"$fake_project/scripts/minikube/sync-auth-key.sh" <<'STUB'
+#!/usr/bin/env bash
+set -eu
+case " $* " in
+ *' --require-gfs '*) printf 'sync\n' >>"${ORDER_FILE:?}" ;;
+ *) printf 'unexpected-sync-args:%s\n' "$*" >>"${ORDER_FILE:?}"; exit 1 ;;
+esac
+STUB
+cat >"$fake_project/scripts/minikube/settle-gfs-reader-rollout.sh" <<'STUB'
+#!/usr/bin/env bash
+set -eu
+printf 'settle\n' >>"${ORDER_FILE:?}"
+STUB
+cat >"$fake_project/deploy/scripts/reconcile-gfs-deploy-credentials.sh" <<'STUB'
+#!/usr/bin/env bash
+set -eu
+case ":${PATH}:" in
+ *":${T2_PROJECT_DIR}/scripts/minikube/gfs-rollout-shim:"*) ;;
+ *) printf 'reconcile-missing-shim-path\n' >>"${ORDER_FILE:?}"; exit 1 ;;
+esac
+printf 'reconcile\n' >>"${ORDER_FILE:?}"
+STUB
+chmod +x "$fake_project/scripts/minikube/sync-auth-key.sh" \
+ "$fake_project/scripts/minikube/settle-gfs-reader-rollout.sh" \
+ "$fake_project/deploy/scripts/reconcile-gfs-deploy-credentials.sh"
+
+cat >"$tmp/kubectl" <<'STUB'
+#!/usr/bin/env bash
+set -eu
+if [[ "${FAKE_KC_MODE:-present}" == reader-absent && "$*" == *'get deployment gfsc-reader -n gfs'* ]]; then
+ printf 'Error from server (NotFound): deployments.apps "gfsc-reader" not found\n' >&2
+ exit 1
+fi
+if [[ "${FAKE_KC_MODE:-present}" == reader-ready-read-error && "$*" == *'get deployment gfsc-reader -n gfs -o jsonpath='* ]]; then
+ printf 'simulated jsonpath read failure\n' >&2
+ exit 1
+fi
+exit 0
+STUB
+chmod +x "$tmp/kubectl"
+
+export PRE_GATE_SYNC_CONFIG_ONLY=true
+export MINIKUBE_PROFILE=gfs-order-test
+export IMAGE_SOURCE=local
+# shellcheck source=/dev/null
+source "$PRE_GATE" >/dev/null
+
+PROJECT_DIR="$fake_project"
+T2_PROJECT_DIR="$fake_project"
+T2_PROFILE=gfs-order-test
+T2_CONTEXT=gfs-order-test
+T2_LOCK_TOKEN=test-lock
+T2_PROFILE_ROOT="$tmp/profile"
+T2_PROFILE_ENV="$tmp/profile.env"
+T2_PORTS_ENV="$tmp/ports.env"
+PROFILE=gfs-order-test
+KC="$tmp/kubectl"
+GATE_NAME=minikube-t2
+ORDER_FILE="$order_file"
+export PROJECT_DIR T2_PROJECT_DIR T2_PROFILE T2_CONTEXT T2_LOCK_TOKEN \
+ T2_PROFILE_ROOT T2_PROFILE_ENV T2_PORTS_ENV PROFILE KC GATE_NAME ORDER_FILE
+
+# Keep the final convergence observable without touching a real cluster; the
+# three wrappers above remain the production implementations under test.
+production_converge="$(declare -f converge_gfs_reader_after_restore)"
+converge_gfs_reader_after_restore() {
+ printf 'converge\n' >>"${ORDER_FILE:?}"
+}
+sync_mcp_host_auth_key() {
+ printf 'mcp\n' >>"${ORDER_FILE:?}"
+}
+
+provision_gfs_serving
+expected=$'sync\nsettle\nreconcile\nconverge'
+actual="$(cat "$order_file")"
+[ "$actual" = "$expected" ] || {
+ printf 'expected:\n%s\nactual:\n%s\n' "$expected" "$actual" >&2
+ fail 'T2 GFS recovery order or shim PATH contract changed'
+}
+
+: >"$order_file"
+GATE_NAME=security-gate-2
+provision_gfs_serving
+actual="$(cat "$order_file")"
+[ "$actual" = 'mcp' ] || fail 'non-T2 pre-gate mutated the GFS recovery sequence'
+
+: >"$order_file"
+FAKE_KC_MODE=reader-absent
+export FAKE_KC_MODE
+GATE_NAME=minikube-t2
+provision_gfs_serving
+actual="$(cat "$order_file")"
+[ "$actual" = 'mcp' ] || fail 'GFS recovery did not skip all GFS mutations when gfsc-reader is absent'
+
+FAKE_KC_MODE=reader-ready-read-error
+eval "$production_converge"
+if converge_gfs_reader_after_restore; then
+ fail 'GFS reader convergence accepted an unreadable replica status'
+fi
+
+printf 'PASS: composed GFS pre-gate order and gate scope\n'
diff --git a/scripts/tests/test-minikube-gfs-rollout-shim.sh b/scripts/tests/test-minikube-gfs-rollout-shim.sh
index 084f78570..6664581df 100755
--- a/scripts/tests/test-minikube-gfs-rollout-shim.sh
+++ b/scripts/tests/test-minikube-gfs-rollout-shim.sh
@@ -63,9 +63,11 @@ grep -q 'rollout status deployment/gfsc-reader' "$FAKE_KUBECTL_LOG" \
&& fail 'shim leaked the generation-based rollout status to the real kubectl'
# 4. A live unready reader pod blocks the wait until timeout (fail-loud).
+# The explicit 1s caller timeout is honored; the shim must not silently floor
+# it to an unrelated minimum.
: >"$FAKE_KUBECTL_LOG"
if PATH="$shim_path" FAKE_DESIRED=1 FAKE_READY=1 FAKE_POD_ROWS='True|\nFalse|\n' \
- GFS_ROLLOUT_SHIM_MIN_TIMEOUT_SECONDS=1 GFS_READER_WAIT_POLL_SECONDS=0 \
+ GFS_READER_WAIT_POLL_SECONDS=0 \
kubectl --context=fake rollout status deployment/gfsc-reader -n gfs --timeout=1s 2>/dev/null; then
fail 'shim readiness wait went green with a live unready reader pod'
fi
diff --git a/scripts/tests/test-minikube-pre-gate-restore.sh b/scripts/tests/test-minikube-pre-gate-restore.sh
new file mode 100755
index 000000000..865fb65ce
--- /dev/null
+++ b/scripts/tests/test-minikube-pre-gate-restore.sh
@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
+fail() { printf 'FAIL: %s\n' "$*" >&2; exit 1; }
+
+tmp="$(mktemp -d)"
+cleanup() { rm -rf "$tmp"; }
+trap cleanup EXIT
+
+cat >"$tmp/kubectl" <<'STUB'
+#!/usr/bin/env bash
+set -eu
+case "${FAKE_RESTORE_FAILURE:-none}:$*" in
+ workflow-scale:*scale\ deployment/workflow-recipes*) exit 17 ;;
+ workflow-rollout:*rollout\ status\ deployment/workflow-recipes*) exit 18 ;;
+ control-scale:*scale\ deployment/control-api*) exit 27 ;;
+ control-rollout:*rollout\ status\ deployment/control-api*) exit 28 ;;
+esac
+exit 0
+STUB
+chmod +x "$tmp/kubectl"
+
+export PRE_GATE_SYNC_CONFIG_ONLY=true
+export MINIKUBE_PROFILE=restore-contract
+export IMAGE_SOURCE=local
+# shellcheck source=/dev/null
+source "$ROOT/scripts/minikube/pre-gate-sync.sh" >/dev/null
+
+KC="$tmp/kubectl"
+export KC
+
+for failure in workflow-scale workflow-rollout; do
+ WRC_FENCED=true
+ WRC_REPLICAS=1
+ export WRC_FENCED WRC_REPLICAS FAKE_RESTORE_FAILURE
+ FAKE_RESTORE_FAILURE="$failure"
+ if restore_workflow_reconciler; then
+ fail "workflow reconciler restore accepted ${failure}"
+ fi
+ [ "$WRC_FENCED" = true ] || fail "workflow fence was cleared after ${failure}"
+done
+
+for failure in control-scale control-rollout; do
+ CONTROL_API_FENCED=true
+ CONTROL_API_REPLICAS=1
+ export CONTROL_API_FENCED CONTROL_API_REPLICAS FAKE_RESTORE_FAILURE
+ FAKE_RESTORE_FAILURE="$failure"
+ if restore_control_api; then
+ fail "Control API restore accepted ${failure}"
+ fi
+ [ "$CONTROL_API_FENCED" = true ] || fail "Control API fence was cleared after ${failure}"
+done
+
+FAKE_RESTORE_FAILURE=none
+WRC_FENCED=true
+restore_workflow_reconciler || fail 'workflow reconciler restore rejected a successful scale and rollout'
+[ "$WRC_FENCED" = false ] || fail 'workflow fence remained armed after a successful restore'
+CONTROL_API_FENCED=true
+restore_control_api || fail 'Control API restore rejected a successful scale and rollout'
+[ "$CONTROL_API_FENCED" = false ] || fail 'Control API fence remained armed after a successful restore'
+
+printf 'PASS: pre-gate restores fail closed and preserve writer fences\n'
diff --git a/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh b/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh
index cbbb29f42..ab4243ce7 100755
--- a/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh
+++ b/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh
@@ -158,7 +158,7 @@ rm -rf "$dir"
# A leftover non-current ReplicaSet with live unready pods keeps
# credential_rollout_pending true forever; it must be scaled to 0 while the
# current-revision ReplicaSet is never touched.
-rs_rows='gfsc-reader-stale|gfsc-reader|1|0|12\ngfsc-reader-current|gfsc-reader|1|0|19\ngfsc-reader-old-empty|gfsc-reader|0|0|11\n'
+rs_rows='gfsc-reader-stale|gfsc-reader|1|0|12\ngfsc-reader-current|gfsc-reader|1|0|19\ngfsc-reader-old-empty|gfsc-reader|0||11\n'
dir="$(run_settle 1 1 1 1 ready '2026-01-01T00:00:00Z' 1 "$rs_rows" '' 19)"
grep -q 'scale rs gfsc-reader-stale --replicas=0' "$dir/kubectl.log" \
|| fail 'leftover non-current unready ReplicaSet was not scaled to 0'
diff --git a/scripts/tests/test-minikube-sync-auth-key.sh b/scripts/tests/test-minikube-sync-auth-key.sh
index dd533086e..48237d75c 100755
--- a/scripts/tests/test-minikube-sync-auth-key.sh
+++ b/scripts/tests/test-minikube-sync-auth-key.sh
@@ -200,6 +200,24 @@ grep -q 'required GFS auth source key RPC_PROXY_JWT_PUBLIC_KEY is empty' "${OUT_
exit 1
}
+: >"${LOG_FILE}"
+if TEST_KUBECTL_LOG="${LOG_FILE}" TEST_STATE_DIR="${STATE_DIR}" TEST_SOURCE_KEY_EMPTY=1 \
+ PATH="${TMP_DIR}:$PATH" bash "${ROOT}/scripts/minikube/sync-auth-key.sh" \
+ --context fake --skip-gfs >"${OUT_FILE}" 2>&1; then
+ echo "FAIL: optional auth sync accepted an empty source key" >&2
+ exit 1
+fi
+grep -q 'auth source key RPC_PROXY_JWT_PUBLIC_KEY is empty; refusing to mutate consumers' "${OUT_FILE}" || {
+ cat "${OUT_FILE}" >&2
+ echo "FAIL: optional auth sync did not fail closed on an empty source key" >&2
+ exit 1
+}
+if grep -Eq '^(patch|rollout restart)' "${LOG_FILE}"; then
+ echo "FAIL: empty source key mutated an auth consumer in optional mode" >&2
+ cat "${LOG_FILE}" >&2
+ exit 1
+fi
+
: >"${LOG_FILE}"
if TEST_KUBECTL_LOG="${LOG_FILE}" TEST_STATE_DIR="${STATE_DIR}" \
PATH="${TMP_DIR}:$PATH" bash "${ROOT}/scripts/minikube/sync-auth-key.sh" \
diff --git a/scripts/tests/test-minikube-t1-gfs-restore.sh b/scripts/tests/test-minikube-t1-gfs-restore.sh
new file mode 100755
index 000000000..e77a26a6f
--- /dev/null
+++ b/scripts/tests/test-minikube-t1-gfs-restore.sh
@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
+fail() { printf 'FAIL: %s\n' "$*" >&2; exit 1; }
+
+# Source only the T1 definitions, then replace the Kubernetes probe with a
+# deterministic unreadable-Secret response. The production cleanup trap is
+# removed before the subshell exits so this test cannot touch a cluster.
+if ! (
+ set +e
+ export MINIKUBE_PROFILE=t1-restore-contract
+ export T2_CONTEXT=t1-restore-contract
+ source "$ROOT/scripts/e2e/minikube-real-postgres.sh" >/dev/null 2>&1
+ trap - EXIT INT TERM
+ set +e
+ T1_GFS_RESTORE_REQUIRED=true
+ t2_kc() { return 1; }
+ if restore_gfs_runtime_credentials; then
+ exit 1
+ fi
+ exit 0
+); then
+ fail 'T1 accepted an unreadable required gfs-controller-db Secret'
+fi
+
+printf 'PASS: T1 GFS restore fails closed on a missing or unreadable Secret\n'
diff --git a/scripts/tests/test-minikube-t2-contract.sh b/scripts/tests/test-minikube-t2-contract.sh
index 60059d83e..5da0cfc9b 100755
--- a/scripts/tests/test-minikube-t2-contract.sh
+++ b/scripts/tests/test-minikube-t2-contract.sh
@@ -22,7 +22,10 @@ for file in "$COMMON" "$PREFLIGHT" "$T2" "$T1" \
"$ROOT/scripts/tests/test-minikube-t2-public-boundary.sh" \
"$ROOT/scripts/tests/test-minikube-t2-scenarios.sh" \
"$ROOT/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh" \
- "$ROOT/scripts/tests/test-minikube-gfs-rollout-shim.sh"; do
+ "$ROOT/scripts/tests/test-minikube-gfs-rollout-shim.sh" \
+ "$ROOT/scripts/tests/test-minikube-gfs-provision-order.sh" \
+ "$ROOT/scripts/tests/test-minikube-pre-gate-restore.sh" \
+ "$ROOT/scripts/tests/test-minikube-t1-gfs-restore.sh"; do
bash -n "$file"
done
@@ -104,36 +107,48 @@ grep -Fq 'T1_GFS_RESTORE_REQUIRED=true' "$T1"
grep -Fq 'settle-gfs-reader-rollout.sh' "$T1"
grep -Fq 'gfs-rollout-shim' "$T1"
grep -Fq 'T2_UNREADY_DEPLOYMENTS' "$COMMON"
+grep -Fq 'required gfs-controller-db Secret is missing or unreadable' "$T1"
# pre-gate-sync GFS serving provisioning must opt into restoring a NOLOGIN
-# role from the committed Secret DSN, and must run in every sync plan.
+# role from the committed Secret DSN. Only the T2 transition owns this
+# mutation; security gates refresh MCP auth without touching the GFS plane.
PRE_GATE="$MINIKUBE_DIR/pre-gate-sync.sh"
-if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'GFS_RESTORE_ACTIVE_NOLOGIN=true'; then
- echo 'FAIL: provision_gfs_serving does not restore a NOLOGIN GFS role from the committed Secret DSN' >&2
+grep -Fq 'scale_rc=0 rollout_rc=0' "$PRE_GATE"
+grep -Fq 'leaving the writer fence armed' "$PRE_GATE"
+if ! sed -n '/^reconcile_gfs_credentials()/,/^}/p' "$PRE_GATE" | grep -Fq 'GFS_RESTORE_ACTIVE_NOLOGIN=true'; then
+ echo 'FAIL: reconcile_gfs_credentials does not restore a NOLOGIN GFS role from the committed Secret DSN' >&2
exit 1
fi
-if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'GFS_RECOVER_ABANDONED_STATE=true'; then
- echo 'FAIL: provision_gfs_serving does not resume an interrupted gfsc-reader rollout claim' >&2
+if ! sed -n '/^reconcile_gfs_credentials()/,/^}/p' "$PRE_GATE" | grep -Fq 'GFS_RECOVER_ABANDONED_STATE=true'; then
+ echo 'FAIL: reconcile_gfs_credentials does not resume an interrupted gfsc-reader rollout claim' >&2
exit 1
fi
-if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'settle-gfs-reader-rollout.sh'; then
- echo 'FAIL: provision_gfs_serving does not settle a Ready gfsc-reader leftover rollout claim before reconcile' >&2
+if ! sed -n '/^settle_gfs_reader_rollout()/,/^}/p' "$PRE_GATE" | grep -Fq 'settle-gfs-reader-rollout.sh'; then
+ echo 'FAIL: settle_gfs_reader_rollout does not settle a Ready gfsc-reader leftover rollout claim before reconcile' >&2
exit 1
fi
-if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'gfs-rollout-shim'; then
- echo 'FAIL: provision_gfs_serving reconcile does not use the HCC-safe reader rollout wait' >&2
+if ! sed -n '/^reconcile_gfs_credentials()/,/^}/p' "$PRE_GATE" | grep -Fq 'gfs-rollout-shim'; then
+ echo 'FAIL: reconcile_gfs_credentials does not use the HCC-safe reader rollout wait' >&2
exit 1
fi
-if ! sed -n '/^provision_gfs_serving()/,/settle-gfs-reader-rollout.sh/p' "$PRE_GATE" | grep -Fq 'sync-auth-key.sh'; then
- echo 'FAIL: provision_gfs_serving does not re-sync gfs-config.jwt-public-key before reconcile (an empty key blocks every new reader pod)' >&2
+if ! sed -n '/^sync_gfs_auth_key()/,/^}/p' "$PRE_GATE" | grep -Fq 'sync-auth-key.sh'; then
+ echo 'FAIL: sync_gfs_auth_key does not re-sync gfs-config.jwt-public-key before reconcile (an empty key blocks every new reader pod)' >&2
exit 1
fi
if ! sed -n '/^converge_gfs_reader_after_restore()/,/^}/p' "$PRE_GATE" | grep -Fq 'wait-gfs-reader-ready.sh'; then
echo 'FAIL: converge_gfs_reader_after_restore still waits on a generation-based rollout status HCC keeps rewriting' >&2
exit 1
fi
-if ! sed -n '/No cluster sync required before/,$p' "$PRE_GATE" | grep -Fq 'provision_gfs_serving'; then
- echo 'FAIL: pre-gate-sync skips GFS serving convergence when no cluster sync is required' >&2
+if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'only minikube-t2 owns GFS recovery'; then
+ echo 'FAIL: pre-gate-sync does not scope GFS recovery mutations to minikube-t2' >&2
+ exit 1
+fi
+if ! sed -n '/^sync_mcp_host_auth_key()/,/^}/p' "$PRE_GATE" | grep -Fq -- '--skip-gfs'; then
+ echo 'FAIL: non-T2 MCP auth refresh can still mutate GFS' >&2
+ exit 1
+fi
+if ! sed -n '/^sync_gfs_auth_key()/,/^}/p' "$PRE_GATE" | grep -Fq -- '--require-gfs'; then
+ echo 'FAIL: T2 GFS auth refresh is not strict about its source key' >&2
exit 1
fi
grep -Fq 'converge_gfs_reader_after_restore' "$PRE_GATE"
@@ -153,6 +168,10 @@ if [[ "$(grep -c 'sync-auth-key.sh' "$ROOT/scripts/minikube/full-setup.sh")" -lt
echo 'FAIL: full-setup must re-sync gfs-config.jwt-public-key before both GFS reconciles (the overlay re-applies it empty)' >&2
exit 1
fi
+if [[ "$(grep -c -- '--require-gfs' "$ROOT/scripts/minikube/full-setup.sh")" -lt 3 ]]; then
+ echo 'FAIL: every GFS-specific full-setup auth sync must require a non-empty source key' >&2
+ exit 1
+fi
if grep -Eq 'make minikube-pre-gate-sync|pre-gate-sync\.sh' "$PREFLIGHT"; then
echo 'FAIL: preflight invokes pre-gate-sync' >&2
@@ -469,5 +488,8 @@ fi
bash "$ROOT/scripts/tests/test-minikube-t2-scenarios.sh"
bash "$ROOT/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh"
bash "$ROOT/scripts/tests/test-minikube-gfs-rollout-shim.sh"
+bash "$ROOT/scripts/tests/test-minikube-gfs-provision-order.sh"
+bash "$ROOT/scripts/tests/test-minikube-pre-gate-restore.sh"
+bash "$ROOT/scripts/tests/test-minikube-t1-gfs-restore.sh"
printf 'PASS: local Minikube T0/T1/T2 contract checks\n'
diff --git a/scripts/tests/test-minikube-t2-scenarios.sh b/scripts/tests/test-minikube-t2-scenarios.sh
index 9d8da0bee..d7db81126 100755
--- a/scripts/tests/test-minikube-t2-scenarios.sh
+++ b/scripts/tests/test-minikube-t2-scenarios.sh
@@ -261,6 +261,17 @@ expect_code PROFILE_UNHEALTHY unready-final-preflight unready-final-preflight \
env "${repo_env[@]}" UNREADY_JSON="$unready_json" \
bash -c 'source "$1"; T2_PLAN_MODE=false; T2_BOOTSTRAP_REQUIRED=false; t2_kc(){ printf "%s" "$UNREADY_JSON"; }; t2_deployment_check' bash "$COMMON"
+# A kubectl command can return a non-JSON diagnostic (for example during a
+# transient API/proxy failure) or JSON with an invalid replica field. The
+# parser failure must be adjudicated as the stable profile failure, not leak a
+# Python traceback and exit before t2_fail can emit next:.
+expect_code PROFILE_UNHEALTHY invalid-deployment-inventory invalid-deployment-inventory \
+ env "${repo_env[@]}" INVALID_INVENTORY='{not-json' \
+ bash -c 'source "$1"; T2_PLAN_MODE=false; T2_BOOTSTRAP_REQUIRED=false; t2_kc(){ printf "%s" "$INVALID_INVENTORY"; }; t2_deployment_check' bash "$COMMON"
+expect_code PROFILE_UNHEALTHY malformed-deployment-inventory malformed-deployment-inventory \
+ env "${repo_env[@]}" INVALID_INVENTORY='{"items":[{"spec":{"replicas":"not-an-integer"}}]}' \
+ bash -c 'source "$1"; T2_PLAN_MODE=false; T2_BOOTSTRAP_REQUIRED=false; t2_kc(){ printf "%s" "$INVALID_INVENTORY"; }; t2_deployment_check' bash "$COMMON"
+
plan_mode_head="$(env "${marker_env[@]}" T2_PLAN_MODE=true FAKE_MARKER='{"data":{"clusterFingerprint":"fp","gitHead":"old","worktreeId":"worktree-a","imageSource":"local","imageTag":"test"}}' \
bash -c 'source "$1"; T2_WORKTREE_ID=worktree-a; T2_HEAD=feature; T2_PLAN_MODE=true; t2_kc(){ printf "%s" "$FAKE_MARKER"; }; t2_marker_check; printf "%s" "$T2_MARKER_MATCHES_HEAD"' bash "$COMMON")"
[ "$plan_mode_head" = false ] || fail "planner mode still treated a stale marker as matching HEAD"
diff --git a/scripts/tests/test-provision-gfs-runtime.sh b/scripts/tests/test-provision-gfs-runtime.sh
index 2740a6431..29d80fbba 100755
--- a/scripts/tests/test-provision-gfs-runtime.sh
+++ b/scripts/tests/test-provision-gfs-runtime.sh
@@ -57,6 +57,11 @@ reconcile="$ROOT/deploy/scripts/reconcile-gfs-deploy-credentials.sh"
echo 'FAIL: post-overlay credential reconciliation did not run once per deploy' >&2
exit 1
}
+grep -Fq 'GFS_REMOTE_RECONCILE_AUTHORIZED=true ALLOWED_CONTEXTS="$CONTEXT"' \
+ "$ROOT/deploy/scripts/provision-gfs-runtime.sh" || {
+ echo 'FAIL: post-overlay credential reconciliation did not carry explicit context authorization' >&2
+ exit 1
+}
[ "$(grep -Fc "rollout status deployment/host-context-controller --timeout=${HCC_ROLLOUT_TIMEOUT_S}s" "$CALL_LOG")" -eq 3 ] || {
echo "FAIL: HCC post-overlay rollout did not use its dedicated ${HCC_ROLLOUT_TIMEOUT_S}s timeout" >&2
exit 1