Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
fix: close PR 416 review findings
Alfredo Lopez5c68c7cc61cb
diff --git a/.cursor/rules/minikube-t0-t1-t2.mdc b/.cursor/rules/minikube-t0-t1-t2.mdcindex 8c0997ed9..88086635a 100644--- a/.cursor/rules/minikube-t0-t1-t2.mdc+++ b/.cursor/rules/minikube-t0-t1-t2.mdc@@ -42,17 +42,19 @@ repository — never mix its PRs or clusters into this lane.current-context.- Bootstrap/reconcile must build the worktree: `IMAGE_SOURCE=local`.- T1 is fail-loud: unavailable DSN or zero executed tests is FAIL, never green.- The JSON reporter is the suite verdict; a leftover Vitest exit after a- complete green reporter is not a failed suite. Role-reset suites use the- throwaway `postgres:16-alpine`, never shared `control-postgres`.+ The JSON reporter must be complete and green and the Vitest process must also+ exit zero. Role-reset suites use the throwaway `postgres:16-alpine`, never+ shared `control-postgres`.- Self-healing single run: the orchestrator planner (`T2_PLAN_MODE=true`)selects `full-reconcile` (naming the deployment) for a bootstrapped profilewith an unready required deployment — never `PROFILE_UNHEALTHY` before atransition. Standalone preflight and the final exact-head check stayfail-loud. T1 restores branch-profile GFS credentials on exit, and`pre-gate-sync` provisions GFS serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true`- and `GFS_RECOVER_ABANDONED_STATE=true` in every plan, so no manual repair- script sits between plan and verdict. `settle-gfs-reader-rollout.sh` clears+ and `GFS_RECOVER_ABANDONED_STATE=true` only in the `minikube-t2` transition;+ other security gates refresh MCP auth with `--skip-gfs` and do not mutate GFS,+ so no manual repair script sits between plan and verdict.+ `settle-gfs-reader-rollout.sh` clearsReady-reader leftovers (claim, stale ReplicaSets, CrashLoopBackOff pods)first, and the `gfs-rollout-shim` PATH prefix replaces the reader`rollout status` wait with a readiness poll because HCC's gfsReconcilerdiff --git a/.cursor/rules/minikube-t2-harness.mdc b/.cursor/rules/minikube-t2-harness.mdcindex 3119de28f..861f583d8 100644--- a/.cursor/rules/minikube-t2-harness.mdc+++ b/.cursor/rules/minikube-t2-harness.mdc@@ -36,8 +36,10 @@ Invariants that must never regress:exit via `reconcile-gfs-deploy-credentials.sh` with`GFS_RESTORE_ACTIVE_NOLOGIN=true` and `GFS_RECOVER_ABANDONED_STATE=true`(armed only once suites started), and `pre-gate-sync.sh`- `provision_gfs_serving` passes the same opt-ins and runs in every sync- plan, restarting an unready `gfsc-reader` after restore. `full-setup.sh`+ `provision_gfs_serving` passes the same opt-ins only for the `minikube-t2`+ transition; other security gates refresh MCP auth with `--skip-gfs` and do+ not mutate GFS. It restarts an unready `gfsc-reader` after T2 restore.+ `full-setup.sh`REUSE_DB / T2 full-reconcile also passes both opt-ins on both GFSreconcile calls — setup runs before pre-gate-sync and must not abort on aNOLOGIN reader or a leftover `rollout-running` claim from a dead prior@@ -52,12 +54,14 @@ Invariants that must never regress:in `scripts/minikube/wait-gfs-reader-ready.sh` (desired replicas Ready andno live non-terminating unready reader pod), never a generation wait.`pre-gate-sync` reader convergence uses the same readiness contract.+ `scripts/tests/test-minikube-gfs-provision-order.sh` exercises the composed+ T2 order and proves the non-T2 scope guard.Both re-run `sync-auth-key.sh` before each GFS reconcile: the overlayre-applies `gfs-config.jwt-public-key` empty and a reader pod failsclosed without it, so the readiness wait could otherwise only time out.- T1 stays fail-loud (no green on unavailable DSN or zero tests); the JSON- reporter is the suite verdict, not a leftover Vitest process exit after a- complete green reporter. Role-reset suites keep the throwaway Postgres 16.+ reporter must be complete and green and the Vitest process must also exit+ zero. Role-reset suites keep the throwaway Postgres 16.- `t2.sh` keeps `IMAGE_SOURCE=local` for bootstrap/reconcile and calls`pre-gate-sync` with `--skip-port-forwards`.- `t2_process_check` accepts only real `kubectl` (`comm=kubectl`) port-forwarddiff --git a/.cursor/skills/minikube-t0-t1-t2/SKILL.md b/.cursor/skills/minikube-t0-t1-t2/SKILL.mdindex 22962c6da..a089af573 100644--- a/.cursor/skills/minikube-t0-t1-t2/SKILL.md+++ b/.cursor/skills/minikube-t0-t1-t2/SKILL.md@@ -83,8 +83,9 @@ T1 runs inside `make minikube-t2`, or explicitly via- T1 is fail-loud: an unavailable DSN, an isolated server that did not start,or zero executed tests is FAIL. Never report green from skipped suites. The- JSON reporter is the suite verdict; a leftover Vitest process exit after a- complete green reporter is not a failed suite.+ JSON reporter must be complete and green, and the Vitest process must also+ exit zero; a green reporter cannot hide teardown, worker, OOM, or signal+ failure.- Role-reset suites (cluster-global role drop/rewrite) run against athrowaway `postgres:16-alpine`, never the shared `control-postgres`.- The admin DSN is resolved in-process from the cluster Secret; never print,diff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.mdindex f425bd9dc..a1f8a9b1d 100644--- a/.cursor/skills/minikube-t0-t1-t2/reference.md+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md@@ -33,8 +33,8 @@ Companion to `SKILL.md`. Source of truth: `scripts/minikube/t2.sh`,`npm ci`, generated ports/profile metadata, or anything secret-like. Run`make minikube-t2-public-boundary` first.- **Do not weaken fail-loud T1**: no green on unavailable DSN, zero executed- tests, or silently skipped suites; the JSON reporter is the suite verdict,- not a leftover Vitest process exit after a complete green reporter.+ tests, or silently skipped suites; the JSON reporter must be complete and+ green, and the Vitest process must also exit zero.Role-reset suites stay on the throwaway `postgres:16-alpine` (#412), nevershared `control-postgres`.@@ -63,8 +63,9 @@ and never copy DSNs or credentials between namespaces or into evidence.The harness owns NOLOGIN recovery and abandoned-rollout resume: T1 restoresbranch-profile GFS credentials on exit, and `pre-gate-sync` provisionsserving with `GFS_RESTORE_ACTIVE_NOLOGIN=true` and-`GFS_RECOVER_ABANDONED_STATE=true` in every plan and restarts an unready-`gfsc-reader` after restore. A Ready leftover `rollout-running` claim is+`GFS_RECOVER_ABANDONED_STATE=true` only in the `minikube-t2` transition.+Other security gates refresh MCP auth with `--skip-gfs` and do not mutate GFS.+The T2 path restarts an unready `gfsc-reader` after restore. A Ready leftover `rollout-running` claim issettled by `scripts/minikube/settle-gfs-reader-rollout.sh` beforereconcile: it marks the claim ready, scales leftover non-current unreadyreader ReplicaSets to 0, and deletes CrashLoopBackOff reader pods so theydiff --git a/AGENTS.md b/AGENTS.mdindex ba679a8cb..a21b3f620 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -98,8 +98,9 @@ has an unready required deployment, the orchestrator planner selects`PROFILE_UNHEALTHY` before a transition exists; the T1 lane restoresbranch-profile GFS credentials on exit and `pre-gate-sync` provisions GFSserving with `GFS_RESTORE_ACTIVE_NOLOGIN=true` and-`GFS_RECOVER_ABANDONED_STATE=true` in every plan, so no manual repair-script belongs between plan and verdict. Harness GFS reconciles settle+`GFS_RECOVER_ABANDONED_STATE=true` in the `minikube-t2` transition, so other+security gates do not mutate GFS and no manual repair script belongs between+plan and verdict. Harness GFS reconciles settleReady-reader leftovers first (`settle-gfs-reader-rollout.sh`) and wait onreader readiness through the `gfs-rollout-shim` PATH prefix instead of ageneration-based `rollout status`, because HCC's gfsReconciler strips the@@ -108,8 +109,9 @@ the final exact-head T2 check stay fail-loud on an unready deployment.Real PostgreSQL suites are opt-in in the ordinary testmatrix, but a T1 run that requires them must fail when the database/DSN isunavailable or when zero tests execute; a green run must never be produced by-silently skipping the suites. The JSON reporter is the suite verdict; a leftover-Vitest process exit after a complete green reporter is not a failed suite.+silently skipping the suites. The JSON reporter must be complete and green,+and the Vitest process must also exit zero; a green reporter cannot hide a+teardown, worker, OOM, or signal failure.Suites that drop or rewrite cluster-global roles must use the harness throwawayPostgres 16, never the shared `control-postgres`.diff --git a/CLAUDE.md b/CLAUDE.mdindex 2e1346eb0..717a2c1ab 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -16,15 +16,17 @@ evidence rules in `AGENTS.md` and `docs/testing/minikube-t2-runbook.md`.produced only by the final exact-head preflight inside `make minikube-t2` (or`make minikube-t2-runtime` after T0 and T1 are already green on the same HEAD).`make minikube-pre-gate-sync` alone is not T2. Playwright and product E2E-scripts are separate lanes. T1 judges the JSON reporter (expected files,-executed/passed, zero failures, zero pending); a leftover Vitest process exit-after a complete green reporter is not a failed suite. The single run is+scripts are separate lanes. T1 requires both a complete green JSON reporter+(expected files, executed/passed, zero failures, zero pending) and a zero+Vitest process exit; a green reporter cannot hide teardown, worker, OOM, or+signal failure. The single run isself-healing: the orchestrator planner selects `full-reconcile` for abootstrapped profile with an unready deployment (never `PROFILE_UNHEALTHY`before a transition), T1 restores branch-profile GFS credentials on exit, and`pre-gate-sync` provisions GFS serving with `GFS_RESTORE_ACTIVE_NOLOGIN=true`-and `GFS_RECOVER_ABANDONED_STATE=true` in every plan; do not insert manual-repair scripts between runs. Harness GFS reconciles settle Ready-reader+and `GFS_RECOVER_ABANDONED_STATE=true` only in the `minikube-t2` transition;+other security gates refresh MCP auth without mutating GFS. Do not insert+manual repair scripts between runs. Harness GFS reconciles settle Ready-readerleftovers first (`settle-gfs-reader-rollout.sh`) and judge reader rolloutcompletion by readiness via the `gfs-rollout-shim` PATH prefix, becauseHCC's gfsReconciler strips the `restartedAt` annotation and adiff --git a/deploy/scripts/provision-gfs-runtime.sh b/deploy/scripts/provision-gfs-runtime.shindex 1853b8873..c4895b1db 100755--- a/deploy/scripts/provision-gfs-runtime.sh+++ b/deploy/scripts/provision-gfs-runtime.sh@@ -189,7 +189,12 @@ finalize_credentials_after_overlay() {wait_for_gfsc_secret_referenceslog "Finalizing staged GFS credentials after the overlay"- CONTEXT="$CONTEXT" bash "$ROOT/deploy/scripts/reconcile-gfs-deploy-credentials.sh"+ # This entrypoint has already validated the target context (including the+ # explicit --allow-prod path). Carry that decision into the credential+ # mutator as an exact, single-context remote authorization instead of+ # relying on a gke_* name prefix.+ GFS_REMOTE_RECONCILE_AUTHORIZED=true ALLOWED_CONTEXTS="$CONTEXT" \+ CONTEXT="$CONTEXT" bash "$ROOT/deploy/scripts/reconcile-gfs-deploy-credentials.sh"log "Waiting for the exact GFSC writer and reader rollouts"kctl -n gfs rollout status deployment/gfsc-writer --timeout=240sdiff --git a/deploy/scripts/reconcile-gfs-deploy-credentials.sh b/deploy/scripts/reconcile-gfs-deploy-credentials.shindex a9a718a21..daede2656 100755--- a/deploy/scripts/reconcile-gfs-deploy-credentials.sh+++ b/deploy/scripts/reconcile-gfs-deploy-credentials.sh@@ -18,6 +18,21 @@ if [ -z "$T2_PROFILE" ]; then T2_PROFILE="$MINIKUBE_PROFILE"; fiif [ -z "$T2_PROFILE" ]; then T2_PROFILE="$CONTEXT"; fiif [ -z "$T2_CONTEXT" ]; then T2_CONTEXT="$CONTROL_API_REAL_PG_CONTEXT"; fiif [ -z "$T2_CONTEXT" ]; then T2_CONTEXT="$CONTEXT"; fi++remote_context_allowed() {+ local allowed_context+ local -a allowed_contexts=()+ [ "${GFS_REMOTE_RECONCILE_AUTHORIZED:-false}" = true ] || return 1+ [ -n "${ALLOWED_CONTEXTS:-}" ] || return 1+ IFS=',' read -r -a allowed_contexts <<<"${ALLOWED_CONTEXTS}"+ for allowed_context in "${allowed_contexts[@]}"; do+ if [ "$allowed_context" = "$CONTEXT" ]; then+ return 0+ fi+ done+ return 1+}+if [ -n "$T2_LOCK_TOKEN" ] || [ -n "$MINIKUBE_PROFILE" ] || [[ "$CONTEXT" == clerum-* ]]; then# shellcheck source=scripts/minikube/t2-common.shsource "$ROOT/scripts/minikube/t2-common.sh"@@ -38,9 +53,13 @@ elsecase "$CONTEXT" ingke_*)# GKE deploys are owned by the explicitly selected deploy workflow rather- # than the local Minikube lease. Keep this exception narrow: an arbitrary- # CONTEXT must never turn this credential mutator into a remote-cluster- # command without either the local fence or the deploy-context shape.+ # than the local Minikube lease. The workflow must opt in explicitly and+ # pass the exact context allowlist; the name prefix alone is not+ # authorization to mutate a remote cluster.+ if ! remote_context_allowed; then+ printf '[reconcile-gfs-deploy] ERROR: refusing remote context without explicit authorization and exact ALLOWED_CONTEXTS membership: %s\n' "$CONTEXT" >&2+ exit 1+ fi;;*)printf '[reconcile-gfs-deploy] ERROR: refusing unverified Kubernetes context: %s\n' "$CONTEXT" >&2diff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.mdindex 5bc2f6698..a691dc48d 100644--- a/docs/testing/minikube-t2-runbook.md+++ b/docs/testing/minikube-t2-runbook.md@@ -158,17 +158,37 @@ when `gfs-config.jwt-public-key` is empty — the overlay re-applies the baseConfigMap with an empty value — so `full-setup.sh` and `pre-gate-sync` re-run`scripts/minikube/sync-auth-key.sh` before each GFS reconcile; otherwise nonew reader pod can start and the readiness wait can only time out.-`pre-gate-sync`-provisions GFS serving with the same opt-ins in every sync plan (including-"no cluster sync required") and, after a successful restore, restarts an-unready `gfsc-reader`, deletes its live unready pods once, and waits on the-same readiness contract. `full-setup.sh` on the REUSE_DB / T2 full-reconcile+`pre-gate-sync` provisions GFS serving with the same opt-ins only in the+`minikube-t2` transition (including its "no cluster sync required" fast path).+Other platform security gates refresh MCP auth with `--skip-gfs` and do not+mutate the GFS plane. After a successful T2 restore, it restarts an unready+`gfsc-reader`, deletes its live unready pods once, and waits on the same+readiness contract. `full-setup.sh` on the REUSE_DB / T2 full-reconcilepath passes the same opt-ins on both GFS reconcile calls, because setup runsbefore pre-gate-sync and must not abort on a NOLOGIN reader or an abandonedreader rollout. The recovery helper restores aNOLOGIN role only from the committed Secret DSN and still fails loud when-that credential cannot authenticate; no password is ever invented and no DSN-is printed.+that credential cannot authenticate; a missing or unreadable required Secret+also fails the T1 cleanup instead of producing a green run. No password is+ever invented and no DSN is printed.++The composed GFS recovery decision table is:++| Observed state | Authoritative action | Safety boundary |+| --- | --- | --- |+| `gfs-config` or the GFS reader Deployment is absent | Skip GFS recovery | The profile has no adopted GFS serving plane. |+| Reader is Ready and the reader Secret says `rollout-running` | `settle-gfs-reader-rollout.sh` marks the claim ready | Preserve the Ready reader; do not trigger a second restart. |+| A non-current reader ReplicaSet has no Ready pod, including an empty `readyReplicas` field | Scale that ReplicaSet to zero | Never scale the current revision; leave terminating pods alone. |+| A live reader pod is `CrashLoopBackOff`, unready, and not terminating | Delete that pod once | Kubelet backoff is reset so it can re-read restored credentials. |+| Desired reader replicas are not Ready, or a live non-terminating reader is unready | Reconcile with the GFS rollout shim, then converge and wait | Do not mark the credential rollout ready from an unready observation. |+| Source auth Secret/key or GFS ConfigMap is missing/empty during a strict GFS sync | Fail before any consumer patch or restart | `--require-gfs` makes an empty source fail closed. |+| Caller supplies an explicit reader rollout timeout | Honor that timeout; an omitted timeout defaults to 600 seconds | The shim never silently floors a caller's fail-fast timeout. |+| Gate is not `minikube-t2` | Refresh MCP auth with `--skip-gfs` only | Security gates cannot patch GFS Secrets/ConfigMaps or delete/restart GFS pods. |++The executable composition contract in+`scripts/tests/test-minikube-gfs-provision-order.sh` verifies the T2 order+`sync-auth-key → settle → reconcile-with-shim-on-PATH → converge` and the+non-T2 scope guard.## T0, T1, and T2 boundaries@@ -178,9 +198,10 @@ is printed.`control-postgres`, except the role-reset suites which use an isolatedPostgres 16; the lane reports `PASS`, `FAIL`, `SKIPPED`, and `NOT_RUN`separately and fails on an unavailable DSN, an isolated server that did not- start, or zero executed tests. The JSON reporter is the suite verdict- (expected files, executed/passed, zero failures, zero pending); a leftover- Vitest process exit after a complete green reporter is not a failed suite.+ start, or zero executed tests. The JSON reporter must be complete and green+ (expected files, executed/passed, zero failures, zero pending), and the+ Vitest process must also exit zero; a green reporter cannot hide teardown,+ worker, OOM, or signal failure.* **T2** — the final exact-head preflight inside `make minikube-t2` (or`make minikube-t2-runtime`): the marker matches this worktree/`HEAD`, theimage manifest is current, PostgreSQL and required namespaces/Services arediff --git a/scripts/e2e/e2e-hcc-rollout-readiness.sh b/scripts/e2e/e2e-hcc-rollout-readiness.shindex a3f3a66a8..aa1b550f7 100755--- a/scripts/e2e/e2e-hcc-rollout-readiness.sh+++ b/scripts/e2e/e2e-hcc-rollout-readiness.sh@@ -228,6 +228,8 @@ botched_image_ref() {# A failed pod-UID listing must not count as "old pod terminated".assert_botched_outage() {local flavor="$1" old_uid_query+ local image_fail old_pod_ok old_pod_fail replacement_fail+ local transitions_ok transitions_fail streak_ok streak_failcase "$flavor" instuck | recovery) ;;*) die "assert_botched_outage: flavor must be stuck or recovery, got '${flavor}'" ;;@@ -246,51 +248,45 @@ assert_botched_outage() {-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.uid}{"\t"}{.status.phase}{"\t"}{.status.containerStatuses[*].state.waiting.reason}{"\n"}{end}' 2>/dev/null || true)"replacement_row="$(awk -F '\t' -v old="$OLD_POD_UID" '$2 != old && $2 != "" { print; exit }' <<<"$stuck_pod_rows")"+ case "$flavor" in+ recovery)+ image_fail="deployment image is '${deployed_image_now:-unset}', not the injected ${BOTCHED_IMAGE}"+ old_pod_ok="Recreate terminated the old HEALTHY pod before any viable replacement existed"+ old_pod_fail="old pod ${OLD_POD_NAME} is still alive under a Recreate rollout"+ replacement_fail="no replacement pod pinned in ErrImagePull|ImagePullBackOff (rows: ${stuck_pod_rows:-none})"+ transitions_ok="no 503->200 transition closed the outage before undo — Kubernetes did NOT self-heal"+ transitions_fail="the outage closed on its own (${stuck_transitions} 503->200 transition(s)) before undo"+ streak_ok="SUSTAINED outage before undo: ${stuck_maxstreak}s continuous 503 (> ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget) — undo recovers D1b, not a blip"+ streak_fail="outage lasted only ${stuck_maxstreak}s and did not exceed the ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget — undo would recover a blip, not D1b"+ ;;+ stuck)+ image_fail="deployment image is '${deployed_image_now:-unset}', not the injected ${BOTCHED_IMAGE} — the outage (if any) has a different cause"+ old_pod_ok="Recreate terminated the old HEALTHY pod before any viable replacement existed — zero fallback replicas (the D1b mechanism)"+ old_pod_fail="old pod ${OLD_POD_NAME} is still alive under a Recreate rollout — the terminate-before-create premise did not hold"+ replacement_fail="no replacement pod pinned in ErrImagePull|ImagePullBackOff (rows: ${stuck_pod_rows:-none}) — the outage is not attributable to the injected bad rollout"+ transitions_ok="no 503->200 transition ever closed the outage — Kubernetes did NOT self-heal the botched Recreate rollout"+ transitions_fail="the outage closed on its own (${stuck_transitions} 503->200 transition(s)) — the D1b premise that recovery requires an explicit rollback is refuted; investigate before trusting this gate"+ streak_ok="SUSTAINED outage measured: ${stuck_maxstreak}s continuous 503 (> ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget) and still open at observation end — recovery requires an explicit rollback (infra-owned rollout undo --to-revision in production; cleanup's set image restore here) (D1b reproduced)"+ streak_fail="outage lasted only ${stuck_maxstreak}s and did not exceed the ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget — the botched rollout did not produce the D1b total outage"+ ;;+ esac+[ "$stuck_total" -gt 0 ] && ok "sampled /ready ${stuck_total}x across the botched rollout" ||fail "ZERO /ready samples across the botched rollout — the sampler never ran, and a zero-sample run must never pass"- if [ "$flavor" = recovery ]; then- [ "$deployed_image_now" = "$BOTCHED_IMAGE" ] &&- ok "the botched image is what the Deployment is actually rolling (${BOTCHED_IMAGE})" ||- fail "deployment image is '${deployed_image_now:-unset}', not the injected ${BOTCHED_IMAGE}"- [ "$old_uid_live" = 0 ] &&- ok "Recreate terminated the old HEALTHY pod before any viable replacement existed" ||- fail "old pod ${OLD_POD_NAME} is still alive under a Recreate rollout"- if [ -n "$replacement_row" ] && grep -Eq 'ErrImagePull|ImagePullBackOff' <<<"$replacement_row"; then- ok "replacement pod is pinned by the injected image failure: ${replacement_row}"- else- fail "no replacement pod pinned in ErrImagePull|ImagePullBackOff (rows: ${stuck_pod_rows:-none})"- fi- [ "$stuck_503" -ge 1 ] &&- ok "the botched rollout bit the readiness path: ${stuck_503} sample(s) at 503" ||- fail "zero 503 samples under a botched Recreate rollout — the outage claim is VACUOUS"- [ "$stuck_transitions" -eq 0 ] &&- ok "no 503->200 transition closed the outage before undo — Kubernetes did NOT self-heal" ||- fail "the outage closed on its own (${stuck_transitions} 503->200 transition(s)) before undo"- [ "$stuck_maxstreak" -gt "$ROLLOUT_DOWNTIME_BUDGET_SEC" ] &&- ok "SUSTAINED outage before undo: ${stuck_maxstreak}s continuous 503 (> ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget) — undo recovers D1b, not a blip" ||- fail "outage lasted only ${stuck_maxstreak}s and did not exceed the ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget — undo would recover a blip, not D1b"+ [ "$deployed_image_now" = "$BOTCHED_IMAGE" ] &&+ ok "the botched image is what the Deployment is actually rolling (${BOTCHED_IMAGE})" ||+ fail "$image_fail"+ [ "$old_uid_live" = 0 ] && ok "$old_pod_ok" || fail "$old_pod_fail"+ if [ -n "$replacement_row" ] && grep -Eq 'ErrImagePull|ImagePullBackOff' <<<"$replacement_row"; then+ ok "replacement pod is pinned by the injected image failure: ${replacement_row}"else- [ "$deployed_image_now" = "$BOTCHED_IMAGE" ] &&- ok "the botched image is what the Deployment is actually rolling (${BOTCHED_IMAGE})" ||- fail "deployment image is '${deployed_image_now:-unset}', not the injected ${BOTCHED_IMAGE} — the outage (if any) has a different cause"- [ "$old_uid_live" = 0 ] &&- ok "Recreate terminated the old HEALTHY pod before any viable replacement existed — zero fallback replicas (the D1b mechanism)" ||- fail "old pod ${OLD_POD_NAME} is still alive under a Recreate rollout — the terminate-before-create premise did not hold"- if [ -n "$replacement_row" ] && grep -Eq 'ErrImagePull|ImagePullBackOff' <<<"$replacement_row"; then- ok "replacement pod is pinned by the injected image failure: ${replacement_row}"- else- fail "no replacement pod pinned in ErrImagePull|ImagePullBackOff (rows: ${stuck_pod_rows:-none}) — the outage is not attributable to the injected bad rollout"- fi- [ "$stuck_503" -ge 1 ] &&- ok "the botched rollout bit the readiness path: ${stuck_503} sample(s) at 503" ||- fail "zero 503 samples under a botched Recreate rollout — the outage claim is VACUOUS"- [ "$stuck_transitions" -eq 0 ] &&- ok "no 503->200 transition ever closed the outage — Kubernetes did NOT self-heal the botched Recreate rollout" ||- fail "the outage closed on its own (${stuck_transitions} 503->200 transition(s)) — the D1b premise that recovery requires an explicit rollback is refuted; investigate before trusting this gate"- [ "$stuck_maxstreak" -gt "$ROLLOUT_DOWNTIME_BUDGET_SEC" ] &&- ok "SUSTAINED outage measured: ${stuck_maxstreak}s continuous 503 (> ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget) and still open at observation end — recovery requires an explicit rollback (infra-owned rollout undo --to-revision in production; cleanup's set image restore here) (D1b reproduced)" ||- fail "outage lasted only ${stuck_maxstreak}s and did not exceed the ${ROLLOUT_DOWNTIME_BUDGET_SEC}s healthy budget — the botched rollout did not produce the D1b total outage"+ fail "$replacement_fail"fi+ [ "$stuck_503" -ge 1 ] &&+ ok "the botched rollout bit the readiness path: ${stuck_503} sample(s) at 503" ||+ fail "zero 503 samples under a botched Recreate rollout — the outage claim is VACUOUS"+ [ "$stuck_transitions" -eq 0 ] && ok "$transitions_ok" || fail "$transitions_fail"+ [ "$stuck_maxstreak" -gt "$ROLLOUT_DOWNTIME_BUDGET_SEC" ] && ok "$streak_ok" || fail "$streak_fail"}print_repair_instructions() {@@ -317,6 +313,11 @@ EOF}write_evidence_artifact() {+ local rollout_transition="not-observed"+ if [ -n "$rollout_recovered_at" ] && [ -n "$rollout_t0" ]; then+ rollout_transition="$((rollout_recovered_at - rollout_t0))s"+ fi+# Diagnostics ONLY — every verdict input is computed before this runs, so# every command here is deliberately best-effort (same rationale as the# sibling churn gate: an empty grep under `set -euo pipefail` must never@@ -331,11 +332,20 @@ write_evidence_artifact() {echo ""echo "=== pods ==="kctl get pods -n "$HCC_NS" -l "app=${HCC_DEPLOY}" -o wide || true- echo "=== pod identity: old ${OLD_POD_NAME:-none} (${OLD_POD_UID:-?}) -> new ${NEW_POD_NAME:-none} (${NEW_POD_UID:-?}) restarts=${NEW_POD_RESTARTS:-?} ==="echo "=== baseline: samples=${baseline_total} 200=${baseline_200} 503=${baseline_503} ==="- echo "=== rollout: samples=${roll_total} 200=${roll_200} 503=${roll_503} MEASURED-window=${roll_maxstreak}s 503->200=${roll_transitions} budget=${ROLLOUT_DOWNTIME_BUDGET_SEC}s total-transition=${rollout_recovered_at:+$((rollout_recovered_at - rollout_t0))}s ==="- echo "=== stuck: samples=${stuck_total} 200=${stuck_200} 503=${stuck_503} max-503-streak=${stuck_maxstreak}s 503->200=${stuck_transitions} observe=${STUCK_OBSERVE_SEC}s ==="- echo "=== post-hold: samples=${hold_total} 503=${hold_503} ==="+ if [ "$EXPECT_RECOVERY" = 1 ]; then+ echo "=== recovered pod identity: old ${OLD_POD_NAME:-none} (${OLD_POD_UID:-not-observed}) -> recovered ${NEW_POD_NAME:-none} (${NEW_POD_UID:-not-observed}) restarts=${NEW_POD_RESTARTS:-not-observed} ==="+ echo "=== recovery revisions: last-good=${LAST_GOOD_REVISION:-not-captured} botched=${live_revision_before_undo:-not-observed} recovered=${recovered_revision:-not-observed} ==="+ echo "=== pre-undo outage: samples=${stuck_total} 200=${stuck_200} 503=${stuck_503} max-503-streak=${stuck_maxstreak}s 503->200=${stuck_transitions} observe=${STUCK_OBSERVE_SEC}s ==="+ echo "=== post-undo hold: samples=${hold_total} 503=${hold_503} ==="+ elif [ "$EXPECT_STUCK" = 1 ]; then+ echo "=== botched replacement: ${replacement_row:-not-observed} ==="+ echo "=== stuck outage: samples=${stuck_total} 200=${stuck_200} 503=${stuck_503} max-503-streak=${stuck_maxstreak}s 503->200=${stuck_transitions} observe=${STUCK_OBSERVE_SEC}s ==="+ else+ echo "=== pod identity: old ${OLD_POD_NAME:-none} (${OLD_POD_UID:-not-observed}) -> new ${NEW_POD_NAME:-none} (${NEW_POD_UID:-not-observed}) restarts=${NEW_POD_RESTARTS:-not-observed} ==="+ echo "=== rollout: samples=${roll_total} 200=${roll_200} 503=${roll_503} MEASURED-window=${roll_maxstreak}s 503->200=${roll_transitions} budget=${ROLLOUT_DOWNTIME_BUDGET_SEC}s total-transition=${rollout_transition} ==="+ echo "=== post-rollout hold: samples=${hold_total} 503=${hold_503} ==="+ fiecho "=== /ready series (epoch status phase) ==="cat "$READY_SERIES" || true} >"$LOG_ARTIFACT" 2>&1 || true@@ -416,11 +426,6 @@ OLD_POD_NAME="$(running_hcc_pod)"[ -n "$OLD_POD_NAME" ] || die "no running HCC pod after a clean baseline"OLD_POD_UID="$(kctl get pod "$OLD_POD_NAME" -n "$HCC_NS" -o jsonpath='{.metadata.uid}')"[ -n "$OLD_POD_UID" ] || die "could not pin the pre-rollout pod uid"---LAST_GOOD_REVISION="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')"-[ -n "$LAST_GOOD_REVISION" ] || die "could not capture the last-known-good HCC revision"-if [ "$EXPECT_STUCK" = 0 ] && [ "$EXPECT_RECOVERY" = 0 ]; then# ── FASE D (healthy): Recreate rollout, sampled at 1Hz end to end ──log "Triggering a Recreate rollout (kubectl rollout restart) and sampling /ready at 1Hz"@@ -477,6 +482,8 @@ elif [ "$EXPECT_STUCK" = 1 ]; thenwrite_evidence_artifactelse# ── FASE D (recovery, evenfire#391): botch, prove the outage, then undo ──+ LAST_GOOD_REVISION="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')"+ [ -n "$LAST_GOOD_REVISION" ] || die "could not capture the last-known-good HCC revision"BOTCHED_IMAGE="$(botched_image_ref)"PRE_BOTCH_UID="$OLD_POD_UID"log "Botched rollout (EXPECT_RECOVERY): observing /ready for ${STUCK_OBSERVE_SEC}s"@@ -505,6 +512,7 @@ elserecovered_revision="$(kctl get deployment "$HCC_DEPLOY" -n "$HCC_NS" -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')"NEW_POD_NAME="$(running_hcc_pod)"NEW_POD_UID="$(kctl get pod "$NEW_POD_NAME" -n "$HCC_NS" -o jsonpath='{.metadata.uid}')"+ NEW_POD_RESTARTS="$(hcc_restart_count "$NEW_POD_NAME" || true)"sample_ready_series "$STABILITY_WINDOW_SEC" post-hold 0read -r hold_total _ hold_503 _ _ <<<"$(series_metrics post-hold)"write_evidence_artifactdiff --git a/scripts/e2e/gfs-real-pg-minikube-gate.sh b/scripts/e2e/gfs-real-pg-minikube-gate.shindex 0654f525c..e743a3afa 100755--- a/scripts/e2e/gfs-real-pg-minikube-gate.sh+++ b/scripts/e2e/gfs-real-pg-minikube-gate.sh@@ -240,7 +240,8 @@ restore_gfs_runtime_credentials() {# Restore the branch profile before this gate exits so a failed or interrupted# T1 run cannot leave GFSC in NOLOGIN and poison the following T2 journey.if ! kc -n gfs get secret gfs-controller-db >/dev/null 2>&1; then- return 0+ printf '[gfs-real-pg-minikube] ERROR: required gfs-controller-db Secret is missing or unreadable; refusing to finish with GFS credentials unreconciled\n' >&2+ return 1fiif ! GFS_RESTORE_ACTIVE_NOLOGIN=true CONTEXT="${CONTEXT}" \bash "${PROJECT_DIR}/deploy/scripts/reconcile-gfs-deploy-credentials.sh"; thendiff --git a/scripts/e2e/minikube-real-postgres.sh b/scripts/e2e/minikube-real-postgres.shindex c696b1662..b7eae25aa 100755--- a/scripts/e2e/minikube-real-postgres.sh+++ b/scripts/e2e/minikube-real-postgres.sh@@ -68,7 +68,8 @@ restore_gfs_runtime_credentials() {return 0fiif ! t2_kc -n gfs get secret gfs-controller-db >/dev/null 2>&1; then- return 0+ printf '[minikube-t1] ERROR: required gfs-controller-db Secret is missing or unreadable; refusing to finish T1 with GFS credentials unreconciled\n' >&2+ return 1fi# Settle a Ready reader's leftover rollout claim first so restore does not# rollout restart it, and let any wait reconcile still needs judgediff --git a/scripts/minikube/full-setup.sh b/scripts/minikube/full-setup.shindex b15335370..a7f661fb1 100755--- a/scripts/minikube/full-setup.sh+++ b/scripts/minikube/full-setup.sh@@ -990,7 +990,7 @@ else# wiped by the overlay re-apply). Re-sync it from the live platform# Secret before reconcile so a reader pod can actually start; this is a# no-op when the key already matches.- bash "${SCRIPT_DIR}/sync-auth-key.sh" --context="${PROFILE}"+ bash "${SCRIPT_DIR}/sync-auth-key.sh" --context="${PROFILE}" --require-gfs# If gfsc-reader is already Ready, settle the leftover claim first so# reconcile does not rollout restart and race HCC's gfsReconciler.# The gfs-rollout-shim PATH prefix makes any reader rollout wait inside@@ -1067,7 +1067,7 @@ else# The overlay apply above re-declared the base gfs-config with an empty# jwt-public-key; re-sync it before any reader pod may need to start,# otherwise the reconcile readiness wait can only time out.- bash "${SCRIPT_DIR}/sync-auth-key.sh" --context="${PROFILE}"+ bash "${SCRIPT_DIR}/sync-auth-key.sh" --context="${PROFILE}" --require-gfsCONTEXT="${PROFILE}" \bash "${PROJECT_DIR}/scripts/minikube/settle-gfs-reader-rollout.sh"PATH="${PROJECT_DIR}/scripts/minikube/gfs-rollout-shim:${PATH}" \@@ -1384,7 +1384,7 @@ done# ----------------------------------------------------------------------if $KC get configmap gfs-config -n gfs &>/dev/null; thenlog "Provisioning gfs serving (public key → gfs-config + gfs_controller DB login)..."- if ! bash "${SCRIPT_DIR}/sync-auth-key.sh" --context="${PROFILE}"; then+ if ! bash "${SCRIPT_DIR}/sync-auth-key.sh" --context="${PROFILE}" --require-gfs; thenerr "gfs public-key sync FAILED — gfsc cannot verify tokens. Fix and re-run."exit 1fidiff --git a/scripts/minikube/gfs-rollout-shim/kubectl b/scripts/minikube/gfs-rollout-shim/kubectlindex 1b6749264..8d34ed686 100755--- a/scripts/minikube/gfs-rollout-shim/kubectl+++ b/scripts/minikube/gfs-rollout-shim/kubectl@@ -64,11 +64,15 @@ while [ "$i" -lt "${#args[@]}" ]; doi=$((i + 1))done-if ! timeout_seconds="$(python3 - "$timeout_raw" "${GFS_ROLLOUT_SHIM_MIN_TIMEOUT_SECONDS:-600}" <<'PY'+# Honor the caller's rollout timeout. The harness caller that needs the+# readiness default omits --timeout and receives 600s; silently flooring an+# explicit short timeout would hide a dead reader and invalidate the caller's+# fail-fast contract.+if ! timeout_seconds="$(python3 - "$timeout_raw" <<'PY'import reimport sys-timeout_raw, minimum_raw = sys.argv[1:]+timeout_raw = sys.argv[1]def parse_duration(raw, default, label):if raw == "":@@ -83,9 +87,7 @@ def parse_duration(raw, default, label):raise SystemExit(f"invalid {label}: value must be between 1 and 86400 seconds")return value-timeout = parse_duration(timeout_raw, 600, "rollout timeout")-minimum = parse_duration(minimum_raw, 600, "minimum rollout timeout")-print(max(timeout, minimum))+print(parse_duration(timeout_raw, 600, "rollout timeout"))PY)"; thenprintf '[gfs-rollout-shim] refusing an invalid rollout timeout\n' >&2diff --git a/scripts/minikube/pre-gate-sync.sh b/scripts/minikube/pre-gate-sync.shindex 3da798c11..6a482c709 100755--- a/scripts/minikube/pre-gate-sync.sh+++ b/scripts/minikube/pre-gate-sync.sh@@ -133,12 +133,15 @@ restore_workflow_reconciler() {fiset +elog "Restoring workflow reconciler to ${WRC_REPLICAS} replica(s)"- ${KC} scale deployment/workflow-recipes -n control-plane --replicas="${WRC_REPLICAS}" >/dev/null- ${KC} rollout status deployment/workflow-recipes -n control-plane --timeout=120s >/dev/null- local restore_rc=$?+ local scale_rc=0 rollout_rc=0+ ${KC} scale deployment/workflow-recipes -n control-plane --replicas="${WRC_REPLICAS}" >/dev/null || scale_rc=$?+ if [[ "${scale_rc}" -eq 0 ]]; then+ ${KC} rollout status deployment/workflow-recipes -n control-plane --timeout=120s >/dev/null || rollout_rc=$?+ fiset -e- if [[ "${restore_rc}" -ne 0 ]]; then- log "WARNING: workflow reconciler did not become Ready during pre-gate cleanup"+ if [[ "${scale_rc}" -ne 0 || "${rollout_rc}" -ne 0 ]]; then+ log "ERROR: workflow reconciler restore failed (scale=${scale_rc}, rollout=${rollout_rc}); leaving the writer fence armed"+ return 1fiWRC_FENCED=false}@@ -166,12 +169,15 @@ restore_control_api() {fiset +elog "Restoring Control API to ${CONTROL_API_REPLICAS} replica(s)"- ${KC} scale deployment/control-api -n control-plane --replicas="${CONTROL_API_REPLICAS}" >/dev/null- ${KC} rollout status deployment/control-api -n control-plane --timeout=120s >/dev/null- local restore_rc=$?+ local scale_rc=0 rollout_rc=0+ ${KC} scale deployment/control-api -n control-plane --replicas="${CONTROL_API_REPLICAS}" >/dev/null || scale_rc=$?+ if [[ "${scale_rc}" -eq 0 ]]; then+ ${KC} rollout status deployment/control-api -n control-plane --timeout=120s >/dev/null || rollout_rc=$?+ fiset -e- if [[ "${restore_rc}" -ne 0 ]]; then- log "WARNING: Control API did not become Ready during pre-gate cleanup"+ if [[ "${scale_rc}" -ne 0 || "${rollout_rc}" -ne 0 ]]; then+ log "ERROR: Control API restore failed (scale=${scale_rc}, rollout=${rollout_rc}); leaving the writer fence armed"+ return 1fiCONTROL_API_FENCED=false}@@ -324,7 +330,31 @@ sync_mcp_host_auth_key() {return 0fi- bash "${PROJECT_DIR}/scripts/minikube/sync-auth-key.sh" --context "${PROFILE}"+ bash "${PROJECT_DIR}/scripts/minikube/sync-auth-key.sh" --context "${PROFILE}" --skip-gfs+}++sync_gfs_auth_key() {+ bash "${PROJECT_DIR}/scripts/minikube/sync-auth-key.sh" --context "${PROFILE}" --require-gfs+}++settle_gfs_reader_rollout() {+ T2_SKIP_LOCK=true T2_LOCK_TOKEN="$T2_LOCK_TOKEN" \+ T2_PROJECT_DIR="$T2_PROJECT_DIR" MINIKUBE_PROFILE="$T2_PROFILE" \+ CONTROL_API_REAL_PG_CONTEXT="$T2_CONTEXT" T2_PROFILE_ROOT="$T2_PROFILE_ROOT" \+ T2_PROFILE_ENV="$T2_PROFILE_ENV" T2_PORTS_ENV="$T2_PORTS_ENV" \+ CONTEXT="$T2_CONTEXT" \+ bash "${PROJECT_DIR}/scripts/minikube/settle-gfs-reader-rollout.sh"+}++reconcile_gfs_credentials() {+ PATH="${PROJECT_DIR}/scripts/minikube/gfs-rollout-shim:${PATH}" \+ GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \+ T2_SKIP_LOCK=true T2_LOCK_TOKEN="$T2_LOCK_TOKEN" \+ T2_PROJECT_DIR="$T2_PROJECT_DIR" MINIKUBE_PROFILE="$T2_PROFILE" \+ CONTROL_API_REAL_PG_CONTEXT="$T2_CONTEXT" T2_PROFILE_ROOT="$T2_PROFILE_ROOT" \+ T2_PROFILE_ENV="$T2_PROFILE_ENV" T2_PORTS_ENV="$T2_PORTS_ENV" \+ CONTEXT="$T2_CONTEXT" \+ bash "${PROJECT_DIR}/deploy/scripts/reconcile-gfs-deploy-credentials.sh"}converge_gfs_reader_after_restore() {@@ -332,13 +362,34 @@ converge_gfs_reader_after_restore() {# already-started pod even after the role is restored. Converging it here# keeps the repair inside the single orchestrated run instead of a manual# `kubectl rollout restart deploy/gfsc-reader` side quest.- local deployment=gfsc-reader desired ready- if ! ${KC} get deployment "${deployment}" -n gfs >/dev/null 2>&1; then+ local deployment=gfsc-reader deployment_probe desired ready+ if ! deployment_probe="$(${KC} get deployment "${deployment}" -n gfs 2>&1)"; then+ if [[ "${deployment_probe}" == *NotFound* || "${deployment_probe}" == *"not found"* ]]; then+ return 0+ fi+ log "ERROR: unable to inspect gfs/${deployment} before credential convergence"+ return 1+ fi+ if ! desired="$(${KC} get deployment "${deployment}" -n gfs -o jsonpath='{.spec.replicas}' 2>/dev/null)"; then+ log "ERROR: unable to read desired replicas for gfs/${deployment}"+ return 1+ fi+ if [[ ! "${desired}" =~ ^[0-9]+$ ]]; then+ log "ERROR: desired replicas for gfs/${deployment} is not numeric"+ return 1+ fi+ if [[ "${desired}" == "0" ]]; thenreturn 0fi- desired="$(${KC} get deployment "${deployment}" -n gfs -o jsonpath='{.spec.replicas}' 2>/dev/null || true)"- ready="$(${KC} get deployment "${deployment}" -n gfs -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"- if [[ -z "${desired}" || "${desired}" == "0" || "${ready:-0}" == "${desired}" ]]; then+ if ! ready="$(${KC} get deployment "${deployment}" -n gfs -o jsonpath='{.status.readyReplicas}' 2>/dev/null)"; then+ log "ERROR: unable to read Ready replicas for gfs/${deployment}"+ return 1+ fi+ if [[ ! "${ready}" =~ ^[0-9]+$ ]]; then+ log "ERROR: Ready replicas for gfs/${deployment} is not numeric"+ return 1+ fi+ if [[ "${ready}" == "${desired}" ]]; thenreturn 0filog "Restarting gfs/${deployment} after credential restore (${ready:-0}/${desired} Ready)"@@ -363,7 +414,25 @@ converge_gfs_reader_after_restore() {}provision_gfs_serving() {+ if [[ "${GATE_NAME}" != "minikube-t2" ]]; then+ # Other platform security gates must retain their read/verify contract.+ # GFS credential recovery is a T2 transition because it can patch Secrets,+ # restart GFSC, delete pods, and resume an abandoned rollout claim.+ log "Skipping GFS serving mutation before ${GATE_NAME}; only minikube-t2 owns GFS recovery"+ sync_mcp_host_auth_key+ return 0+ fiif ${KC} get configmap gfs-config -n gfs >/dev/null 2>&1; then+ local reader_probe+ if ! reader_probe="$(${KC} get deployment gfsc-reader -n gfs 2>&1)"; then+ if [[ "${reader_probe}" == *NotFound* || "${reader_probe}" == *"not found"* ]]; then+ log "Skipping gfs serving provisioning (gfsc-reader not found — GFS reader is not deployed)"+ sync_mcp_host_auth_key+ return 0+ fi+ log "ERROR: unable to inspect gfs/gfsc-reader before credential recovery"+ return 1+ filog "Provisioning gfs serving before ${GATE_NAME}"# FAIL LOUD: with the GFS stack deployed, a broken gfs_controller credential# means every GFS operation 503s (issue #775). Continuing would burn the@@ -380,35 +449,28 @@ provision_gfs_serving() {# (the overlay re-applies the base ConfigMap with an empty value); a# reader pod cannot start without it and any readiness wait would only# time out. Re-sync it first; this is a no-op when the key matches.- bash "${PROJECT_DIR}/scripts/minikube/sync-auth-key.sh" --context "${PROFILE}"+ sync_gfs_auth_key# Settle a Ready reader first so reconcile does not restart it and# race HCC's gfsReconciler during kubectl rollout status. If reconcile# still needs a reader rollout, the gfs-rollout-shim PATH prefix makes# that wait judge readiness instead of the template generation HCC keeps# rewriting.- T2_SKIP_LOCK=true T2_LOCK_TOKEN="$T2_LOCK_TOKEN" \- T2_PROJECT_DIR="$T2_PROJECT_DIR" MINIKUBE_PROFILE="$T2_PROFILE" \- CONTROL_API_REAL_PG_CONTEXT="$T2_CONTEXT" T2_PROFILE_ROOT="$T2_PROFILE_ROOT" \- T2_PROFILE_ENV="$T2_PROFILE_ENV" T2_PORTS_ENV="$T2_PORTS_ENV" \- CONTEXT="$T2_CONTEXT" \- bash "${PROJECT_DIR}/scripts/minikube/settle-gfs-reader-rollout.sh"- if ! PATH="${PROJECT_DIR}/scripts/minikube/gfs-rollout-shim:${PATH}" \- GFS_RESTORE_ACTIVE_NOLOGIN=true GFS_RECOVER_ABANDONED_STATE=true \- T2_SKIP_LOCK=true T2_LOCK_TOKEN="$T2_LOCK_TOKEN" \- T2_PROJECT_DIR="$T2_PROJECT_DIR" MINIKUBE_PROFILE="$T2_PROFILE" \- CONTROL_API_REAL_PG_CONTEXT="$T2_CONTEXT" T2_PROFILE_ROOT="$T2_PROFILE_ROOT" \- T2_PROFILE_ENV="$T2_PROFILE_ENV" T2_PORTS_ENV="$T2_PORTS_ENV" \- CONTEXT="$T2_CONTEXT" \- bash "${PROJECT_DIR}/deploy/scripts/reconcile-gfs-deploy-credentials.sh"; then+ if ! settle_gfs_reader_rollout; then+ log "ERROR: unable to settle the branch-owned GFS reader rollout before credential reconciliation"+ exit 1+ fi+ if ! reconcile_gfs_credentials; thenlog "ERROR: gfs DB provisioning FAILED — gfsc cannot authorize any operation. Aborting ${GATE_NAME} pre-gate sync."exit 1fi- converge_gfs_reader_after_restore+ if ! converge_gfs_reader_after_restore; then+ log "ERROR: gfs reader did not converge after credential restoration"+ return 1+ fielselog "Skipping gfs serving provisioning (gfs-config not found — GFS stack not deployed)"+ sync_mcp_host_auth_keyfi-- sync_mcp_host_auth_key}assert_no_legacy_prompt_bridge_grants() {@@ -568,9 +630,8 @@ if [[ "${cluster_changed}" == "true" ]]; thenif ! incremental_requires_database_reconcile; then# No schema work is planned, but a role-reset T1 run may still have left# the cluster-global GFS runtime roles NOLOGIN while every image and- # manifest matched. Serving provisioning is idempotent, so converge it in- # every sync plan instead of leaving an unready gfsc-reader to fail the- # exact-head T2 preflight after this sync reports success.+ # manifest matched. The T2 transition owns this idempotent convergence;+ # security-gate invocations return through the non-mutating guard above.provision_gfs_servingfi@@ -657,8 +718,8 @@ elselog "No cluster sync required before ${GATE_NAME}"# Images and manifests match, yet the GFS runtime credentials can still be# broken (a role-reset T1 leaves the reader NOLOGIN without changing any- # fingerprint). Converge serving credentials on every pre-gate pass so a- # single orchestrated run repairs them without a manual script.+ # fingerprint). The T2 fast path repairs them in this single orchestrated+ # run; non-T2 gates are fenced out by provision_gfs_serving.provision_gfs_servingassert_no_legacy_prompt_bridge_grantsensure_evenfire_registrydiff --git a/scripts/minikube/sync-auth-key.sh b/scripts/minikube/sync-auth-key.shindex ad519aeb7..661e9b978 100755--- a/scripts/minikube/sync-auth-key.sh+++ b/scripts/minikube/sync-auth-key.sh@@ -3,6 +3,7 @@ set -euo pipefailPROFILE="clerum-test"REQUIRE_GFS=false+SYNC_GFS=truewhile [[ $# -gt 0 ]]; docase "$1" in@@ -21,8 +22,14 @@ while [[ $# -gt 0 ]]; doREQUIRE_GFS=trueshift;;+ --skip-gfs)+ # Pre-gate security lanes need the MCP host key refresh without mutating+ # the optional GFS plane. The T2 GFS path uses --require-gfs instead.+ SYNC_GFS=false+ shift+ ;;-h|--help)- echo "Usage: scripts/minikube/sync-auth-key.sh [--context <kubectl-context>] [--require-gfs]" >&2+ echo "Usage: scripts/minikube/sync-auth-key.sh [--context <kubectl-context>] [--require-gfs|--skip-gfs]" >&2exit 0;;*)@@ -32,6 +39,11 @@ while [[ $# -gt 0 ]]; doesacdone+if [[ "${REQUIRE_GFS}" == "true" && "${SYNC_GFS}" != "true" ]]; then+ echo "--require-gfs cannot be combined with --skip-gfs" >&2+ exit 2+fi+KCTL=(kubectl --context="${PROFILE}")SOURCE_SECRET="rpc-proxy-secrets"SOURCE_NAMESPACE="rpc-proxy"@@ -97,8 +109,11 @@ if ! "${KCTL[@]}" get secret "${SOURCE_SECRET}" -n "${SOURCE_NAMESPACE}" >/dev/nexit 0fisource_key="$("${KCTL[@]}" get secret "${SOURCE_SECRET}" -n "${SOURCE_NAMESPACE}" -o "jsonpath={.data.${SOURCE_KEY}}" | base64 -d)"-if [[ "${REQUIRE_GFS}" == "true" && -z "${source_key}" ]]; then- die "required GFS auth source key ${SOURCE_KEY} is empty"+if [[ -z "${source_key}" ]]; then+ if [[ "${REQUIRE_GFS}" == "true" ]]; then+ die "required GFS auth source key ${SOURCE_KEY} is empty"+ fi+ die "auth source key ${SOURCE_KEY} is empty; refusing to mutate consumers"fiif [[ "${REQUIRE_GFS}" == "true" ]] && \! "${KCTL[@]}" get configmap "${GFS_CONFIGMAP}" -n "${GFS_NAMESPACE}" >/dev/null 2>&1; then@@ -168,6 +183,8 @@ sync_gfs() {}sync_mcp_host-sync_gfs+if [[ "${SYNC_GFS}" == "true" ]]; then+ sync_gfs+filog "Auth key synced"diff --git a/scripts/minikube/t2-common.sh b/scripts/minikube/t2-common.shindex 7b1ed9018..b25186628 100755--- a/scripts/minikube/t2-common.sh+++ b/scripts/minikube/t2-common.sh@@ -489,7 +489,7 @@ t2_deployment_check() {T2_NEXT_COMMAND="MINIKUBE_PROFILE=$T2_PROFILE make minikube-setup-local"t2_fail PROFILE_UNHEALTHY 'deployment readiness inventory is unavailable'fi- unready="$(python3 - "$deployment_json" <<'PY'+ if ! unready="$(python3 - "$deployment_json" <<'PY'import jsonimport syspayload = json.loads(sys.argv[1])@@ -507,7 +507,15 @@ for item in payload.get("items", []):bad.append("%s/%s %s/%s" % (metadata.get("namespace", "?"), metadata.get("name", "?"), ready, desired))print("; ".join(bad))PY- )"+ )"; then+ if [ "$T2_BOOTSTRAP_REQUIRED" = true ]; then+ T2_PLAN_STATE=full-bootstrap+ T2_PLAN_REASON='deployment readiness inventory is invalid'+ return 0+ fi+ T2_NEXT_COMMAND="MINIKUBE_PROFILE=$T2_PROFILE make minikube-t2"+ t2_fail PROFILE_UNHEALTHY 'deployment readiness inventory is invalid'+ fiif [ -n "$unready" ]; thenif [ "$T2_BOOTSTRAP_REQUIRED" = true ]; thenT2_PLAN_STATE=full-bootstrapdiff --git a/scripts/tests/test-gfs-deploy-contract.sh b/scripts/tests/test-gfs-deploy-contract.shindex 7cd1e40b3..8a12acfb1 100755--- a/scripts/tests/test-gfs-deploy-contract.sh+++ b/scripts/tests/test-gfs-deploy-contract.sh@@ -51,6 +51,19 @@ figrep -Fq 'refusing unverified Kubernetes context' "$reconcile_context_err" \|| fail 'credential reconciliation did not explain its unverified-context rejection'rm -f "$reconcile_context_err"+reconcile_remote_context_err="$(mktemp)"+if CONTEXT=gke_unapproved GFS_REMOTE_RECONCILE_AUTHORIZED=true ALLOWED_CONTEXTS=gke_approved \+ bash deploy/scripts/reconcile-gfs-deploy-credentials.sh 2>"$reconcile_remote_context_err"; then+ rm -f "$reconcile_remote_context_err"+ fail 'credential reconciliation accepted a GKE context outside the explicit allowlist'+fi+grep -Fq 'explicit authorization and exact ALLOWED_CONTEXTS membership' "$reconcile_remote_context_err" \+ || fail 'credential reconciliation did not fail closed for an unauthorized GKE context'+rm -f "$reconcile_remote_context_err"+grep -Fq 'GFS_REMOTE_RECONCILE_AUTHORIZED' deploy/scripts/reconcile-gfs-deploy-credentials.sh \+ || fail 'remote GFS reconciliation lacks an explicit authorization gate'+grep -Fq 'ALLOWED_CONTEXTS' deploy/scripts/reconcile-gfs-deploy-credentials.sh \+ || fail 'remote GFS reconciliation lacks an exact context allowlist'make_block() {awk -v target="$1" '$0 ~ "^" target ":" {active=1} active && /^\.PHONY:/ {exit} active {print}' Makefile}diff --git a/scripts/tests/test-gfs-real-pg-minikube-gate.sh b/scripts/tests/test-gfs-real-pg-minikube-gate.shindex 3a1e47640..99c89b2a0 100755--- a/scripts/tests/test-gfs-real-pg-minikube-gate.sh+++ b/scripts/tests/test-gfs-real-pg-minikube-gate.sh@@ -16,6 +16,7 @@ grep -Fq "gfs*.realPostgres.integration.test.ts" "${SCRIPT}"grep -Fq "services.rateLimiter.realPostgres.integration.test.ts" "${SCRIPT}"grep -Fq 'isolated processes' "${SCRIPT}"grep -Fq 'restore_gfs_runtime_credentials' "${SCRIPT}"+grep -Fq 'required gfs-controller-db Secret is missing or unreadable' "${SCRIPT}"grep -Fq 'GFS_RESTORE_ACTIVE_NOLOGIN=true' "${SCRIPT}"grep -Fq 'failed to restore branch-profile GFS credentials' "${SCRIPT}"grep -Fq 'gke_*)' "${SCRIPT}"diff --git a/scripts/tests/test-hcc-rollout-readiness-gate.sh b/scripts/tests/test-hcc-rollout-readiness-gate.shindex 4478693ed..201df98c7 100755--- a/scripts/tests/test-hcc-rollout-readiness-gate.sh+++ b/scripts/tests/test-hcc-rollout-readiness-gate.sh@@ -127,6 +127,7 @@ grep -Fq 'create_synthetic_fleet' "$GATE" &&# is the same sustained D1b pin as EXPECT_STUCK, IMAGE_BROKEN is cleared# only AFTER every post-undo proof, and a no-op undo fails.recovery_block="$(awk '/FASE D \(recovery, evenfire#391\)/,0' "$GATE")"+stuck_block="$(awk '/FASE D \(botched, D1b\)/,/^else$/' "$GATE")"# The sustained-outage comparison lives in the shared assert_botched_outage# helper; pin the LITERAL test expression in the helper body (a comment# mentioning the variables must never satisfy this check) plus the recovery@@ -149,7 +150,88 @@ grep -Fq 'EXPECT_RECOVERY' "$GATE" &&pass "EXPECT_RECOVERY undoes to last-good outside cleanup and fails a no-op" ||fail "EXPECT_RECOVERY is missing, shares cleanup, skips sustained outage, or clears IMAGE_BROKEN before proofs"-# 13. Makefile entry point: the gate is reachable through the same shaped+# 13. The stuck and recovery modes share one implementation of every D1b+# predicate. Only their explanatory messages differ, so one mode cannot+# silently lose a correctness pin while the other keeps it.+[ "$(grep -Fc '[ "$deployed_image_now" = "$BOTCHED_IMAGE" ]' <<<"$outage_helper")" = 1 ] &&+ [ "$(grep -Fc '[ "$old_uid_live" = 0 ]' <<<"$outage_helper")" = 1 ] &&+ [ "$(grep -Fc "grep -Eq 'ErrImagePull|ImagePullBackOff'" <<<"$outage_helper")" = 1 ] &&+ [ "$(grep -Fc '[ "$stuck_503" -ge 1 ]' <<<"$outage_helper")" = 1 ] &&+ [ "$(grep -Fc '[ "$stuck_transitions" -eq 0 ]' <<<"$outage_helper")" = 1 ] &&+ [ "$(grep -Fc '[ "$stuck_maxstreak" -gt "$ROLLOUT_DOWNTIME_BUDGET_SEC" ]' <<<"$outage_helper")" = 1 ] &&+ grep -Fq 'assert_botched_outage stuck' <<<"$stuck_block" &&+ grep -Fq 'assert_botched_outage recovery' <<<"$recovery_block" &&+ pass "stuck and recovery modes share one complete D1b assertion implementation" ||+ fail "assert_botched_outage duplicates a predicate or one mode bypasses the shared D1b pins"++# 14. Evidence is mode-aware: each mode emits only the samples it collected,+# and an unavailable restart count is explicit rather than the ambiguous+# `restarts=?` placeholder.+evidence_helper="$(awk '/^write_evidence_artifact\(\) \{/,/^\}/' "$GATE")"+render_evidence_mode() (+ local mode="$1"+ eval "$evidence_helper"+ EXPECT_STUCK=0 EXPECT_RECOVERY=0+ case "$mode" in+ stuck) EXPECT_STUCK=1 ;;+ recovery) EXPECT_RECOVERY=1 ;;+ esac+ WITH_SYNTHETIC_FLEET=0 FLEET_CONTEXTS=24 FLEET_MCPSERVERS=115 FLEET_HOSTS=8+ HCC_DEPLOY=host-context-controller HCC_NS=control-plane+ OLD_POD_NAME=old-pod OLD_POD_UID=uid-old+ NEW_POD_NAME=new-pod NEW_POD_UID=uid-new NEW_POD_RESTARTS=0+ LAST_GOOD_REVISION=7 live_revision_before_undo=8 recovered_revision=9+ replacement_row=$'botched-pod\tuid-botched\tPending\tImagePullBackOff'+ baseline_total=10 baseline_200=10 baseline_503=0+ roll_total=3 roll_200=2 roll_503=1 roll_maxstreak=1 roll_transitions=1+ stuck_total=4 stuck_200=0 stuck_503=4 stuck_maxstreak=4 stuck_transitions=0+ hold_total=2 hold_503=0+ ROLLOUT_DOWNTIME_BUDGET_SEC=120 STUCK_OBSERVE_SEC=210+ rollout_t0=100 rollout_recovered_at=103+ READY_SERIES=/dev/null+ LOG_ARTIFACT="$(mktemp "${TMPDIR:-/tmp}/hcc-rollout-evidence-test.XXXXXX")"+ trap 'rm -f "$LOG_ARTIFACT"' EXIT+ kctl() { return 0; }+ write_evidence_artifact+ cat "$LOG_ARTIFACT"+)+healthy_evidence="$(render_evidence_mode healthy)"+stuck_evidence="$(render_evidence_mode stuck)"+recovery_evidence="$(render_evidence_mode recovery)"+grep -Fq 'pre-undo outage: samples=' <<<"$recovery_evidence" &&+ grep -Fq 'post-undo hold: samples=' <<<"$recovery_evidence" &&+ grep -Fq 'restarts=0' <<<"$recovery_evidence" &&+ ! grep -Fq '=== rollout: samples=' <<<"$recovery_evidence" &&+ grep -Fq 'stuck outage: samples=' <<<"$stuck_evidence" &&+ ! grep -Fq 'restarts=' <<<"$stuck_evidence" &&+ ! grep -Fq '=== rollout: samples=' <<<"$stuck_evidence" &&+ grep -Fq '=== rollout: samples=' <<<"$healthy_evidence" &&+ grep -Fq 'post-rollout hold: samples=' <<<"$healthy_evidence" &&+ grep -Fq 'restarts=0' <<<"$healthy_evidence" &&+ ! grep -Fq 'stuck outage: samples=' <<<"$healthy_evidence" &&+ ! grep -Fq 'samples=0' <<<"${healthy_evidence}${stuck_evidence}${recovery_evidence}" &&+ ! grep -Fq 'restarts=?' <<<"${healthy_evidence}${stuck_evidence}${recovery_evidence}" &&+ ! grep -Fq 'restarts=${NEW_POD_RESTARTS:-?}' <<<"$evidence_helper" &&+ pass "evidence reports only mode-relevant samples and never emits restarts=?" ||+ fail "evidence mixes uncollected mode counters or retains an ambiguous restart placeholder"++# 15. Recovery-only state stays in the recovery branch: capture last-good+# before injecting the bad image, then record the recovered pod's restart+# count after undo and before writing evidence.+last_good_capture='LAST_GOOD_REVISION="$(kctl get deployment "$HCC_DEPLOY"'+capture_line="$(grep -nF "$last_good_capture" <<<"$recovery_block" | head -1 | cut -d: -f1)"+botch_line="$(grep -nF 'kctl set image deployment/"$HCC_DEPLOY"' <<<"$recovery_block" | head -1 | cut -d: -f1)"+undo_line="$(grep -nF 'kctl rollout undo deployment/"$HCC_DEPLOY"' <<<"$recovery_block" | head -1 | cut -d: -f1)"+restart_line="$(grep -nF 'NEW_POD_RESTARTS="$(hcc_restart_count "$NEW_POD_NAME" || true)"' <<<"$recovery_block" | head -1 | cut -d: -f1)"+evidence_line="$(grep -nF 'write_evidence_artifact' <<<"$recovery_block" | head -1 | cut -d: -f1)"+[ "$(grep -Fc "$last_good_capture" "$GATE")" = 1 ] &&+ [ -n "$capture_line" ] && [ -n "$botch_line" ] && [ "$capture_line" -lt "$botch_line" ] &&+ [ -n "$undo_line" ] && [ -n "$restart_line" ] && [ -n "$evidence_line" ] &&+ [ "$restart_line" -gt "$undo_line" ] && [ "$restart_line" -lt "$evidence_line" ] &&+ pass "recovery alone captures last-good and records recovered pod restarts after undo" ||+ fail "last-good/restart evidence escaped recovery scope or occurs on the wrong side of undo"++# 16. Makefile entry point: the gate is reachable through the same shaped# target as the sibling HCC gates — fault-injection ack + pre-gate-sync# expectation + explicit profile=context pin, with the exclusive# EXPECT_STUCK/EXPECT_RECOVERY modes surfaced (and documented as@@ -168,7 +250,7 @@ grep -Fq '.PHONY: test-e2e-hcc-rollout-readiness' "$MAKEFILE" &&pass "Makefile target test-e2e-hcc-rollout-readiness keeps ack + profile=context + modes + gate invocation" ||fail "Makefile target test-e2e-hcc-rollout-readiness is missing or lost its ack/profile/mode/gate contract"-# 14. Executable guard suite: CI must RUN the cluster-free guard tests (not+# 17. Executable guard suite: CI must RUN the cluster-free guard tests (not# only this grep contract), and that suite must actually EXECUTE the gate# — a guards script degraded back to grep would be this file twice.GUARDS="${ROOT}/scripts/tests/test-hcc-rollout-readiness-guards.sh"diff --git a/scripts/tests/test-minikube-gfs-provision-order.sh b/scripts/tests/test-minikube-gfs-provision-order.shnew file mode 100755index 000000000..422d0adcd--- /dev/null+++ b/scripts/tests/test-minikube-gfs-provision-order.sh@@ -0,0 +1,118 @@+#!/usr/bin/env bash+# Executable contract for the composed GFS pre-gate recovery sequence.+set -euo pipefail++ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"+PRE_GATE="$ROOT/scripts/minikube/pre-gate-sync.sh"+fail() { printf 'FAIL: %s\n' "$*" >&2; exit 1; }++tmp="$(mktemp -d)"+cleanup() { rm -rf "$tmp"; }+trap cleanup EXIT++fake_project="$tmp/project"+mkdir -p "$fake_project/scripts/minikube" "$fake_project/deploy/scripts" \+ "$fake_project/scripts/minikube/gfs-rollout-shim"+order_file="$tmp/order.log"++cat >"$fake_project/scripts/minikube/sync-auth-key.sh" <<'STUB'+#!/usr/bin/env bash+set -eu+case " $* " in+ *' --require-gfs '*) printf 'sync\n' >>"${ORDER_FILE:?}" ;;+ *) printf 'unexpected-sync-args:%s\n' "$*" >>"${ORDER_FILE:?}"; exit 1 ;;+esac+STUB+cat >"$fake_project/scripts/minikube/settle-gfs-reader-rollout.sh" <<'STUB'+#!/usr/bin/env bash+set -eu+printf 'settle\n' >>"${ORDER_FILE:?}"+STUB+cat >"$fake_project/deploy/scripts/reconcile-gfs-deploy-credentials.sh" <<'STUB'+#!/usr/bin/env bash+set -eu+case ":${PATH}:" in+ *":${T2_PROJECT_DIR}/scripts/minikube/gfs-rollout-shim:"*) ;;+ *) printf 'reconcile-missing-shim-path\n' >>"${ORDER_FILE:?}"; exit 1 ;;+esac+printf 'reconcile\n' >>"${ORDER_FILE:?}"+STUB+chmod +x "$fake_project/scripts/minikube/sync-auth-key.sh" \+ "$fake_project/scripts/minikube/settle-gfs-reader-rollout.sh" \+ "$fake_project/deploy/scripts/reconcile-gfs-deploy-credentials.sh"++cat >"$tmp/kubectl" <<'STUB'+#!/usr/bin/env bash+set -eu+if [[ "${FAKE_KC_MODE:-present}" == reader-absent && "$*" == *'get deployment gfsc-reader -n gfs'* ]]; then+ printf 'Error from server (NotFound): deployments.apps "gfsc-reader" not found\n' >&2+ exit 1+fi+if [[ "${FAKE_KC_MODE:-present}" == reader-ready-read-error && "$*" == *'get deployment gfsc-reader -n gfs -o jsonpath='* ]]; then+ printf 'simulated jsonpath read failure\n' >&2+ exit 1+fi+exit 0+STUB+chmod +x "$tmp/kubectl"++export PRE_GATE_SYNC_CONFIG_ONLY=true+export MINIKUBE_PROFILE=gfs-order-test+export IMAGE_SOURCE=local+# shellcheck source=/dev/null+source "$PRE_GATE" >/dev/null++PROJECT_DIR="$fake_project"+T2_PROJECT_DIR="$fake_project"+T2_PROFILE=gfs-order-test+T2_CONTEXT=gfs-order-test+T2_LOCK_TOKEN=test-lock+T2_PROFILE_ROOT="$tmp/profile"+T2_PROFILE_ENV="$tmp/profile.env"+T2_PORTS_ENV="$tmp/ports.env"+PROFILE=gfs-order-test+KC="$tmp/kubectl"+GATE_NAME=minikube-t2+ORDER_FILE="$order_file"+export PROJECT_DIR T2_PROJECT_DIR T2_PROFILE T2_CONTEXT T2_LOCK_TOKEN \+ T2_PROFILE_ROOT T2_PROFILE_ENV T2_PORTS_ENV PROFILE KC GATE_NAME ORDER_FILE++# Keep the final convergence observable without touching a real cluster; the+# three wrappers above remain the production implementations under test.+production_converge="$(declare -f converge_gfs_reader_after_restore)"+converge_gfs_reader_after_restore() {+ printf 'converge\n' >>"${ORDER_FILE:?}"+}+sync_mcp_host_auth_key() {+ printf 'mcp\n' >>"${ORDER_FILE:?}"+}++provision_gfs_serving+expected=$'sync\nsettle\nreconcile\nconverge'+actual="$(cat "$order_file")"+[ "$actual" = "$expected" ] || {+ printf 'expected:\n%s\nactual:\n%s\n' "$expected" "$actual" >&2+ fail 'T2 GFS recovery order or shim PATH contract changed'+}++: >"$order_file"+GATE_NAME=security-gate-2+provision_gfs_serving+actual="$(cat "$order_file")"+[ "$actual" = 'mcp' ] || fail 'non-T2 pre-gate mutated the GFS recovery sequence'++: >"$order_file"+FAKE_KC_MODE=reader-absent+export FAKE_KC_MODE+GATE_NAME=minikube-t2+provision_gfs_serving+actual="$(cat "$order_file")"+[ "$actual" = 'mcp' ] || fail 'GFS recovery did not skip all GFS mutations when gfsc-reader is absent'++FAKE_KC_MODE=reader-ready-read-error+eval "$production_converge"+if converge_gfs_reader_after_restore; then+ fail 'GFS reader convergence accepted an unreadable replica status'+fi++printf 'PASS: composed GFS pre-gate order and gate scope\n'diff --git a/scripts/tests/test-minikube-gfs-rollout-shim.sh b/scripts/tests/test-minikube-gfs-rollout-shim.shindex 084f78570..6664581df 100755--- a/scripts/tests/test-minikube-gfs-rollout-shim.sh+++ b/scripts/tests/test-minikube-gfs-rollout-shim.sh@@ -63,9 +63,11 @@ grep -q 'rollout status deployment/gfsc-reader' "$FAKE_KUBECTL_LOG" \&& fail 'shim leaked the generation-based rollout status to the real kubectl'# 4. A live unready reader pod blocks the wait until timeout (fail-loud).+# The explicit 1s caller timeout is honored; the shim must not silently floor+# it to an unrelated minimum.: >"$FAKE_KUBECTL_LOG"if PATH="$shim_path" FAKE_DESIRED=1 FAKE_READY=1 FAKE_POD_ROWS='True|\nFalse|\n' \- GFS_ROLLOUT_SHIM_MIN_TIMEOUT_SECONDS=1 GFS_READER_WAIT_POLL_SECONDS=0 \+ GFS_READER_WAIT_POLL_SECONDS=0 \kubectl --context=fake rollout status deployment/gfsc-reader -n gfs --timeout=1s 2>/dev/null; thenfail 'shim readiness wait went green with a live unready reader pod'fidiff --git a/scripts/tests/test-minikube-pre-gate-restore.sh b/scripts/tests/test-minikube-pre-gate-restore.shnew file mode 100755index 000000000..865fb65ce--- /dev/null+++ b/scripts/tests/test-minikube-pre-gate-restore.sh@@ -0,0 +1,63 @@+#!/usr/bin/env bash+set -euo pipefail++ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"+fail() { printf 'FAIL: %s\n' "$*" >&2; exit 1; }++tmp="$(mktemp -d)"+cleanup() { rm -rf "$tmp"; }+trap cleanup EXIT++cat >"$tmp/kubectl" <<'STUB'+#!/usr/bin/env bash+set -eu+case "${FAKE_RESTORE_FAILURE:-none}:$*" in+ workflow-scale:*scale\ deployment/workflow-recipes*) exit 17 ;;+ workflow-rollout:*rollout\ status\ deployment/workflow-recipes*) exit 18 ;;+ control-scale:*scale\ deployment/control-api*) exit 27 ;;+ control-rollout:*rollout\ status\ deployment/control-api*) exit 28 ;;+esac+exit 0+STUB+chmod +x "$tmp/kubectl"++export PRE_GATE_SYNC_CONFIG_ONLY=true+export MINIKUBE_PROFILE=restore-contract+export IMAGE_SOURCE=local+# shellcheck source=/dev/null+source "$ROOT/scripts/minikube/pre-gate-sync.sh" >/dev/null++KC="$tmp/kubectl"+export KC++for failure in workflow-scale workflow-rollout; do+ WRC_FENCED=true+ WRC_REPLICAS=1+ export WRC_FENCED WRC_REPLICAS FAKE_RESTORE_FAILURE+ FAKE_RESTORE_FAILURE="$failure"+ if restore_workflow_reconciler; then+ fail "workflow reconciler restore accepted ${failure}"+ fi+ [ "$WRC_FENCED" = true ] || fail "workflow fence was cleared after ${failure}"+done++for failure in control-scale control-rollout; do+ CONTROL_API_FENCED=true+ CONTROL_API_REPLICAS=1+ export CONTROL_API_FENCED CONTROL_API_REPLICAS FAKE_RESTORE_FAILURE+ FAKE_RESTORE_FAILURE="$failure"+ if restore_control_api; then+ fail "Control API restore accepted ${failure}"+ fi+ [ "$CONTROL_API_FENCED" = true ] || fail "Control API fence was cleared after ${failure}"+done++FAKE_RESTORE_FAILURE=none+WRC_FENCED=true+restore_workflow_reconciler || fail 'workflow reconciler restore rejected a successful scale and rollout'+[ "$WRC_FENCED" = false ] || fail 'workflow fence remained armed after a successful restore'+CONTROL_API_FENCED=true+restore_control_api || fail 'Control API restore rejected a successful scale and rollout'+[ "$CONTROL_API_FENCED" = false ] || fail 'Control API fence remained armed after a successful restore'++printf 'PASS: pre-gate restores fail closed and preserve writer fences\n'diff --git a/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh b/scripts/tests/test-minikube-settle-gfs-reader-rollout.shindex cbbb29f42..ab4243ce7 100755--- a/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh+++ b/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh@@ -158,7 +158,7 @@ rm -rf "$dir"# A leftover non-current ReplicaSet with live unready pods keeps# credential_rollout_pending true forever; it must be scaled to 0 while the# current-revision ReplicaSet is never touched.-rs_rows='gfsc-reader-stale|gfsc-reader|1|0|12\ngfsc-reader-current|gfsc-reader|1|0|19\ngfsc-reader-old-empty|gfsc-reader|0|0|11\n'+rs_rows='gfsc-reader-stale|gfsc-reader|1|0|12\ngfsc-reader-current|gfsc-reader|1|0|19\ngfsc-reader-old-empty|gfsc-reader|0||11\n'dir="$(run_settle 1 1 1 1 ready '2026-01-01T00:00:00Z' 1 "$rs_rows" '' 19)"grep -q 'scale rs gfsc-reader-stale --replicas=0' "$dir/kubectl.log" \|| fail 'leftover non-current unready ReplicaSet was not scaled to 0'diff --git a/scripts/tests/test-minikube-sync-auth-key.sh b/scripts/tests/test-minikube-sync-auth-key.shindex dd533086e..48237d75c 100755--- a/scripts/tests/test-minikube-sync-auth-key.sh+++ b/scripts/tests/test-minikube-sync-auth-key.sh@@ -200,6 +200,24 @@ grep -q 'required GFS auth source key RPC_PROXY_JWT_PUBLIC_KEY is empty' "${OUT_exit 1}+: >"${LOG_FILE}"+if TEST_KUBECTL_LOG="${LOG_FILE}" TEST_STATE_DIR="${STATE_DIR}" TEST_SOURCE_KEY_EMPTY=1 \+ PATH="${TMP_DIR}:$PATH" bash "${ROOT}/scripts/minikube/sync-auth-key.sh" \+ --context fake --skip-gfs >"${OUT_FILE}" 2>&1; then+ echo "FAIL: optional auth sync accepted an empty source key" >&2+ exit 1+fi+grep -q 'auth source key RPC_PROXY_JWT_PUBLIC_KEY is empty; refusing to mutate consumers' "${OUT_FILE}" || {+ cat "${OUT_FILE}" >&2+ echo "FAIL: optional auth sync did not fail closed on an empty source key" >&2+ exit 1+}+if grep -Eq '^(patch|rollout restart)' "${LOG_FILE}"; then+ echo "FAIL: empty source key mutated an auth consumer in optional mode" >&2+ cat "${LOG_FILE}" >&2+ exit 1+fi+: >"${LOG_FILE}"if TEST_KUBECTL_LOG="${LOG_FILE}" TEST_STATE_DIR="${STATE_DIR}" \PATH="${TMP_DIR}:$PATH" bash "${ROOT}/scripts/minikube/sync-auth-key.sh" \diff --git a/scripts/tests/test-minikube-t1-gfs-restore.sh b/scripts/tests/test-minikube-t1-gfs-restore.shnew file mode 100755index 000000000..e77a26a6f--- /dev/null+++ b/scripts/tests/test-minikube-t1-gfs-restore.sh@@ -0,0 +1,27 @@+#!/usr/bin/env bash+set -euo pipefail++ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"+fail() { printf 'FAIL: %s\n' "$*" >&2; exit 1; }++# Source only the T1 definitions, then replace the Kubernetes probe with a+# deterministic unreadable-Secret response. The production cleanup trap is+# removed before the subshell exits so this test cannot touch a cluster.+if ! (+ set +e+ export MINIKUBE_PROFILE=t1-restore-contract+ export T2_CONTEXT=t1-restore-contract+ source "$ROOT/scripts/e2e/minikube-real-postgres.sh" >/dev/null 2>&1+ trap - EXIT INT TERM+ set +e+ T1_GFS_RESTORE_REQUIRED=true+ t2_kc() { return 1; }+ if restore_gfs_runtime_credentials; then+ exit 1+ fi+ exit 0+); then+ fail 'T1 accepted an unreadable required gfs-controller-db Secret'+fi++printf 'PASS: T1 GFS restore fails closed on a missing or unreadable Secret\n'diff --git a/scripts/tests/test-minikube-t2-contract.sh b/scripts/tests/test-minikube-t2-contract.shindex 60059d83e..5da0cfc9b 100755--- a/scripts/tests/test-minikube-t2-contract.sh+++ b/scripts/tests/test-minikube-t2-contract.sh@@ -22,7 +22,10 @@ for file in "$COMMON" "$PREFLIGHT" "$T2" "$T1" \"$ROOT/scripts/tests/test-minikube-t2-public-boundary.sh" \"$ROOT/scripts/tests/test-minikube-t2-scenarios.sh" \"$ROOT/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh" \- "$ROOT/scripts/tests/test-minikube-gfs-rollout-shim.sh"; do+ "$ROOT/scripts/tests/test-minikube-gfs-rollout-shim.sh" \+ "$ROOT/scripts/tests/test-minikube-gfs-provision-order.sh" \+ "$ROOT/scripts/tests/test-minikube-pre-gate-restore.sh" \+ "$ROOT/scripts/tests/test-minikube-t1-gfs-restore.sh"; dobash -n "$file"done@@ -104,36 +107,48 @@ grep -Fq 'T1_GFS_RESTORE_REQUIRED=true' "$T1"grep -Fq 'settle-gfs-reader-rollout.sh' "$T1"grep -Fq 'gfs-rollout-shim' "$T1"grep -Fq 'T2_UNREADY_DEPLOYMENTS' "$COMMON"+grep -Fq 'required gfs-controller-db Secret is missing or unreadable' "$T1"# pre-gate-sync GFS serving provisioning must opt into restoring a NOLOGIN-# role from the committed Secret DSN, and must run in every sync plan.+# role from the committed Secret DSN. Only the T2 transition owns this+# mutation; security gates refresh MCP auth without touching the GFS plane.PRE_GATE="$MINIKUBE_DIR/pre-gate-sync.sh"-if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'GFS_RESTORE_ACTIVE_NOLOGIN=true'; then- echo 'FAIL: provision_gfs_serving does not restore a NOLOGIN GFS role from the committed Secret DSN' >&2+grep -Fq 'scale_rc=0 rollout_rc=0' "$PRE_GATE"+grep -Fq 'leaving the writer fence armed' "$PRE_GATE"+if ! sed -n '/^reconcile_gfs_credentials()/,/^}/p' "$PRE_GATE" | grep -Fq 'GFS_RESTORE_ACTIVE_NOLOGIN=true'; then+ echo 'FAIL: reconcile_gfs_credentials does not restore a NOLOGIN GFS role from the committed Secret DSN' >&2exit 1fi-if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'GFS_RECOVER_ABANDONED_STATE=true'; then- echo 'FAIL: provision_gfs_serving does not resume an interrupted gfsc-reader rollout claim' >&2+if ! sed -n '/^reconcile_gfs_credentials()/,/^}/p' "$PRE_GATE" | grep -Fq 'GFS_RECOVER_ABANDONED_STATE=true'; then+ echo 'FAIL: reconcile_gfs_credentials does not resume an interrupted gfsc-reader rollout claim' >&2exit 1fi-if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'settle-gfs-reader-rollout.sh'; then- echo 'FAIL: provision_gfs_serving does not settle a Ready gfsc-reader leftover rollout claim before reconcile' >&2+if ! sed -n '/^settle_gfs_reader_rollout()/,/^}/p' "$PRE_GATE" | grep -Fq 'settle-gfs-reader-rollout.sh'; then+ echo 'FAIL: settle_gfs_reader_rollout does not settle a Ready gfsc-reader leftover rollout claim before reconcile' >&2exit 1fi-if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'gfs-rollout-shim'; then- echo 'FAIL: provision_gfs_serving reconcile does not use the HCC-safe reader rollout wait' >&2+if ! sed -n '/^reconcile_gfs_credentials()/,/^}/p' "$PRE_GATE" | grep -Fq 'gfs-rollout-shim'; then+ echo 'FAIL: reconcile_gfs_credentials does not use the HCC-safe reader rollout wait' >&2exit 1fi-if ! sed -n '/^provision_gfs_serving()/,/settle-gfs-reader-rollout.sh/p' "$PRE_GATE" | grep -Fq 'sync-auth-key.sh'; then- echo 'FAIL: provision_gfs_serving does not re-sync gfs-config.jwt-public-key before reconcile (an empty key blocks every new reader pod)' >&2+if ! sed -n '/^sync_gfs_auth_key()/,/^}/p' "$PRE_GATE" | grep -Fq 'sync-auth-key.sh'; then+ echo 'FAIL: sync_gfs_auth_key does not re-sync gfs-config.jwt-public-key before reconcile (an empty key blocks every new reader pod)' >&2exit 1fiif ! sed -n '/^converge_gfs_reader_after_restore()/,/^}/p' "$PRE_GATE" | grep -Fq 'wait-gfs-reader-ready.sh'; thenecho 'FAIL: converge_gfs_reader_after_restore still waits on a generation-based rollout status HCC keeps rewriting' >&2exit 1fi-if ! sed -n '/No cluster sync required before/,$p' "$PRE_GATE" | grep -Fq 'provision_gfs_serving'; then- echo 'FAIL: pre-gate-sync skips GFS serving convergence when no cluster sync is required' >&2+if ! sed -n '/^provision_gfs_serving()/,/^}/p' "$PRE_GATE" | grep -Fq 'only minikube-t2 owns GFS recovery'; then+ echo 'FAIL: pre-gate-sync does not scope GFS recovery mutations to minikube-t2' >&2+ exit 1+fi+if ! sed -n '/^sync_mcp_host_auth_key()/,/^}/p' "$PRE_GATE" | grep -Fq -- '--skip-gfs'; then+ echo 'FAIL: non-T2 MCP auth refresh can still mutate GFS' >&2+ exit 1+fi+if ! sed -n '/^sync_gfs_auth_key()/,/^}/p' "$PRE_GATE" | grep -Fq -- '--require-gfs'; then+ echo 'FAIL: T2 GFS auth refresh is not strict about its source key' >&2exit 1figrep -Fq 'converge_gfs_reader_after_restore' "$PRE_GATE"@@ -153,6 +168,10 @@ if [[ "$(grep -c 'sync-auth-key.sh' "$ROOT/scripts/minikube/full-setup.sh")" -ltecho 'FAIL: full-setup must re-sync gfs-config.jwt-public-key before both GFS reconciles (the overlay re-applies it empty)' >&2exit 1fi+if [[ "$(grep -c -- '--require-gfs' "$ROOT/scripts/minikube/full-setup.sh")" -lt 3 ]]; then+ echo 'FAIL: every GFS-specific full-setup auth sync must require a non-empty source key' >&2+ exit 1+fiif grep -Eq 'make minikube-pre-gate-sync|pre-gate-sync\.sh' "$PREFLIGHT"; thenecho 'FAIL: preflight invokes pre-gate-sync' >&2@@ -469,5 +488,8 @@ fibash "$ROOT/scripts/tests/test-minikube-t2-scenarios.sh"bash "$ROOT/scripts/tests/test-minikube-settle-gfs-reader-rollout.sh"bash "$ROOT/scripts/tests/test-minikube-gfs-rollout-shim.sh"+bash "$ROOT/scripts/tests/test-minikube-gfs-provision-order.sh"+bash "$ROOT/scripts/tests/test-minikube-pre-gate-restore.sh"+bash "$ROOT/scripts/tests/test-minikube-t1-gfs-restore.sh"printf 'PASS: local Minikube T0/T1/T2 contract checks\n'diff --git a/scripts/tests/test-minikube-t2-scenarios.sh b/scripts/tests/test-minikube-t2-scenarios.shindex 9d8da0bee..d7db81126 100755--- a/scripts/tests/test-minikube-t2-scenarios.sh+++ b/scripts/tests/test-minikube-t2-scenarios.sh@@ -261,6 +261,17 @@ expect_code PROFILE_UNHEALTHY unready-final-preflight unready-final-preflight \env "${repo_env[@]}" UNREADY_JSON="$unready_json" \bash -c 'source "$1"; T2_PLAN_MODE=false; T2_BOOTSTRAP_REQUIRED=false; t2_kc(){ printf "%s" "$UNREADY_JSON"; }; t2_deployment_check' bash "$COMMON"+# A kubectl command can return a non-JSON diagnostic (for example during a+# transient API/proxy failure) or JSON with an invalid replica field. The+# parser failure must be adjudicated as the stable profile failure, not leak a+# Python traceback and exit before t2_fail can emit next:.+expect_code PROFILE_UNHEALTHY invalid-deployment-inventory invalid-deployment-inventory \+ env "${repo_env[@]}" INVALID_INVENTORY='{not-json' \+ bash -c 'source "$1"; T2_PLAN_MODE=false; T2_BOOTSTRAP_REQUIRED=false; t2_kc(){ printf "%s" "$INVALID_INVENTORY"; }; t2_deployment_check' bash "$COMMON"+expect_code PROFILE_UNHEALTHY malformed-deployment-inventory malformed-deployment-inventory \+ env "${repo_env[@]}" INVALID_INVENTORY='{"items":[{"spec":{"replicas":"not-an-integer"}}]}' \+ bash -c 'source "$1"; T2_PLAN_MODE=false; T2_BOOTSTRAP_REQUIRED=false; t2_kc(){ printf "%s" "$INVALID_INVENTORY"; }; t2_deployment_check' bash "$COMMON"+plan_mode_head="$(env "${marker_env[@]}" T2_PLAN_MODE=true FAKE_MARKER='{"data":{"clusterFingerprint":"fp","gitHead":"old","worktreeId":"worktree-a","imageSource":"local","imageTag":"test"}}' \bash -c 'source "$1"; T2_WORKTREE_ID=worktree-a; T2_HEAD=feature; T2_PLAN_MODE=true; t2_kc(){ printf "%s" "$FAKE_MARKER"; }; t2_marker_check; printf "%s" "$T2_MARKER_MATCHES_HEAD"' bash "$COMMON")"[ "$plan_mode_head" = false ] || fail "planner mode still treated a stale marker as matching HEAD"diff --git a/scripts/tests/test-provision-gfs-runtime.sh b/scripts/tests/test-provision-gfs-runtime.shindex 2740a6431..29d80fbba 100755--- a/scripts/tests/test-provision-gfs-runtime.sh+++ b/scripts/tests/test-provision-gfs-runtime.sh@@ -57,6 +57,11 @@ reconcile="$ROOT/deploy/scripts/reconcile-gfs-deploy-credentials.sh"echo 'FAIL: post-overlay credential reconciliation did not run once per deploy' >&2exit 1}+grep -Fq 'GFS_REMOTE_RECONCILE_AUTHORIZED=true ALLOWED_CONTEXTS="$CONTEXT"' \+ "$ROOT/deploy/scripts/provision-gfs-runtime.sh" || {+ echo 'FAIL: post-overlay credential reconciliation did not carry explicit context authorization' >&2+ exit 1+}[ "$(grep -Fc "rollout status deployment/host-context-controller --timeout=${HCC_ROLLOUT_TIMEOUT_S}s" "$CALL_LOG")" -eq 3 ] || {echo "FAIL: HCC post-overlay rollout did not use its dedicated ${HCC_ROLLOUT_TIMEOUT_S}s timeout" >&2exit 1