Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
fix(minikube): close remaining recovery and puller gaps
Alfredo Lopez37a9ad3ec737
diff --git a/.cursor/skills/minikube-t0-t1-t2/SKILL.md b/.cursor/skills/minikube-t0-t1-t2/SKILL.mdindex 6d2047815..032d470c5 100644--- a/.cursor/skills/minikube-t0-t1-t2/SKILL.md+++ b/.cursor/skills/minikube-t0-t1-t2/SKILL.md@@ -37,6 +37,10 @@ Before running anything, verify ALL of these:- [ ] Mutating image acquisition/builds use the public Make target/orchestratorand inherit its exact profile lease. Do not call `build-images.sh` or`pull-images.sh` directly; `--verify-only` is the read-only exception.+ The legacy `setup.sh` path is also lease-protected for every invocation,+ requires matching explicit profile/context variables, and never uses the+ shared `clerum-test` profile. Pull configuration values are finite and an+ empty successful `minikube docker-env` result is a hard failure.- [ ] Never read `~/.cache/clerum/minikube-profiles/` directly (HARD DENY —it holds private profile state). The harness reads it for you.@@ -155,8 +159,12 @@ Before committing anything else, run `make minikube-t2-public-boundary`.The final exact-head check also rejects a marker whose `imagesGeneratedAt`does not equal the current image manifest. During REUSE_DB recovery, thedurable fence covers HCC, workflow-recipes, trace-maintenance-worker, and-control-api; a changed HEAD on the same owned lane is historical state, not a-reason to discard recovery evidence.+control-api. The incremental baseline applies that image-stamp check in both+GHCR and local image modes, because a matching gitHead does not prove that a+newly acquired image ID is running. The trace worker must be at zero desired+replicas with no remaining pods before `trace-fenced` is persisted. A changed+HEAD on the same owned lane is historical state, not a reason to discard+recovery evidence.## Step 5 — On failurediff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.mdindex 6facbeecf..c25b515ee 100644--- a/.cursor/skills/minikube-t0-t1-t2/reference.md+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md@@ -66,6 +66,8 @@ Companion to `SKILL.md`. Source of truth: `scripts/minikube/t2.sh`,| `PROFILE_LOCK_REQUIRED` | A mutating child did not inherit the exact parent lease. | Invoke the public Make target/orchestrator; do not fabricate or bypass its lock token. || `DOCKER_ENDPOINT_REQUIRED` / `DOCKER_ENDPOINT_UNSAFE` / `DOCKER_ENDPOINT_MISMATCH` | Docker did not resolve exactly one approved local endpoint, or isolation selected a different endpoint. | Select a local Docker Desktop/rootless context or set an explicit local `DOCKER_HOST`; do not weaken the endpoint check. || `DOCKER_CONFIG_REQUIRED` / `REGISTRY_AUTH_REQUIRED` | A Docker operation lacks its empty task-local config, or a private pull lacks an explicit readable auth config. | Use the canonical Make/orchestrator path; for a genuinely private source, pass only the intended config directory through `MINIKUBE_DOCKER_AUTH_CONFIG`. |+| `MINIKUBE_PULL_CONFIG_INVALID` | Published-image pull parallelism, retry count, or retry delay is outside its finite range. | Correct the override (`parallelism 1-64`, retries 1-10, delay 0-300 seconds); never disable the bound. |+| `DOCKER_ENV_UNRESOLVED` | Minikube returned no usable Docker environment, including a successful empty response. | Repair the profile/runtime endpoint and retry the same owned Make target; no image pull may proceed. || `DOCKER_DEADLINE_INVALID` | A runtime deadline/retry value is malformed or exceeds its finite maximum. | Remove the override or set it within the documented range; do not disable the deadline. || `BOOTSTRAP_REQUIRED` | Profile missing/uninitialized, or the planner produced no transition. Standalone preflight refuses to bootstrap. | Run `MINIKUBE_PROFILE=<owned> CONTROL_API_REAL_PG_CONTEXT=<owned> make minikube-t2`; its internal planner makes `full-bootstrap` reachable. || `HEAD_MARKER_MISMATCH` | Pre-gate marker does not match this worktree/HEAD; the final T2 preflight selected something other than `already-synced`. | Re-run the full target on this HEAD so `pre-gate-sync` updates the marker. Never hand-edit the marker. |diff --git a/.github/workflows/ci-public.yml b/.github/workflows/ci-public.ymlindex 91c1be1ad..73e46030c 100644--- a/.github/workflows/ci-public.yml+++ b/.github/workflows/ci-public.yml@@ -484,6 +484,12 @@ jobs:- name: Test minikube Makefile targetsrun: bash scripts/tests/test-minikube-makefile.sh+ - name: Test local Minikube T0/T1/T2 contract+ run: bash scripts/tests/test-minikube-t2-contract.sh++ - name: Test local Minikube T2 setup handoff+ run: bash scripts/tests/test-minikube-t2-setup-handoff.sh+- name: Test minikube GFS Upload v2 profile contractrun: bash scripts/tests/test-minikube-gfs-upload-v2-profile.shdiff --git a/AGENTS.md b/AGENTS.mdindex 8036b2be2..043e92307 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -50,6 +50,13 @@ use finite deadlines. Docker endpoint discovery must resolveto an explicit local Unix socket or loopback TCP endpoint before switching toan empty task-local Docker config. Ambient registry credentials are nevercopied; a private pull requires an explicit `MINIKUBE_DOCKER_AUTH_CONFIG`.+The legacy `scripts/minikube/setup.sh` compatibility path follows the same+boundary: it requires matching `MINIKUBE_PROFILE` and+`CONTROL_API_REAL_PG_CONTEXT`, acquires the branch-owned lease before its first+cluster operation for every invocation (with or without `--build`), and never+falls back to the shared `clerum-test` profile. The published-image puller+rejects invalid parallelism/retry/delay values and an empty successful+`minikube docker-env`; neither condition may produce a green acquisition.## Incremental local image updates@@ -80,6 +87,12 @@ profile is already healthy; record it as a *targeted sync* and prove theaffected deployment is Ready plus its user-facing health endpoint. Do notreport that as a full reconcile or as T2.+The image acquisition stamp is part of the baseline in both GHCR and local+image modes. If the manifest's `generated` value differs from the pre-gate+marker, the incremental planner must invalidate the old `gitHead` baseline and+reconcile before certifying; a matching commit alone does not prove that pods+run the newly acquired image IDs.+Expect a full reconcile when deployment manifests, CRDs, charts, networkpolicy, or other infrastructure inputs changed; do not replace that safe pathwith partial image updates. Every `kubectl` invocation must use the verified@@ -139,6 +152,10 @@ When REUSE_DB recovery is needed, the fence covers all four database writers:HCC, workflow-recipes, trace-maintenance-worker, and control-api. The durablerecovery identity is the owned profile/context/worktree/branch; the interruptedHEAD is retained as audit data while the exact-head marker controls freshness.+The trace-maintenance-worker fence must scale its Deployment to zero and prove+both `.spec.replicas == 0` and no remaining writer pods before persisting the+`trace-fenced` phase; restoring its saved replica count happens only after the+recovery window.Real PostgreSQL suites are opt-in in the ordinary testmatrix, but a T1 run that requires them must fail when the database/DSN isunavailable or when zero tests execute; a green run must never be produced bydiff --git a/control-api/package-lock.json b/control-api/package-lock.jsonindex bff1abf2b..63275ff3e 100644--- a/control-api/package-lock.json+++ b/control-api/package-lock.json@@ -1,12 +1,12 @@{"name": "clerum-control-api",- "version": "0.1.491",+ "version": "0.1.492","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "clerum-control-api",- "version": "0.1.491",+ "version": "0.1.492","dependencies": {"@clerum/display-field": "file:../packages/display-field","@clerum/image-policy": "file:../packages/image-policy",diff --git a/control-api/package.json b/control-api/package.jsonindex f4fad6068..c57856d50 100644--- a/control-api/package.json+++ b/control-api/package.json@@ -1,6 +1,6 @@{"name": "clerum-control-api",- "version": "0.1.491",+ "version": "0.1.492","description": "Internal Clerum control plane REST API + UI for managing CRDs and secrets","main": "dist/main.js","engines": {diff --git a/control-api/src/k8s.ts b/control-api/src/k8s.tsindex 73aaef994..a8bcae7a8 100644--- a/control-api/src/k8s.ts+++ b/control-api/src/k8s.ts@@ -573,6 +573,10 @@ export class K8sGateway {podName,namespace,containerName,+ err: {+ name: findError instanceof Error ? findError.name : typeof findError,+ message: findError instanceof Error ? findError.message : String(findError),+ },},'find -printf failed; falling back to find -type f')diff --git a/control-api/src/routes/admin/secrets.ts b/control-api/src/routes/admin/secrets.tsindex 8951c8e3d..623302d0f 100644--- a/control-api/src/routes/admin/secrets.ts+++ b/control-api/src/routes/admin/secrets.ts@@ -514,13 +514,17 @@ export function createAdminSecretsRouter(gateway: K8sGateway): Router {.map(item => (item as { metadata?: { name?: string } }).metadata?.name).filter((connector): connector is string => typeof connector === 'string').sort((a, b) => a.localeCompare(b))- } catch {+ } catch (err) {logger.warn({module: 'admin-secrets',event: 'mcp-secret-rotated-affected-connectors-unavailable',name,namespace: targetNs,+ err: {+ name: err instanceof Error ? err.name : typeof err,+ message: err instanceof Error ? err.message : String(err),+ },},'Affected connectors unavailable after MCP Secret rotation')diff --git a/control-api/test/k8s.execSecurity.test.ts b/control-api/test/k8s.execSecurity.test.tsindex 5efa652c6..745fa3426 100644--- a/control-api/test/k8s.execSecurity.test.ts+++ b/control-api/test/k8s.execSecurity.test.ts@@ -357,7 +357,11 @@ describe('artifact listing commands', () => {'find -printf failed; falling back to find -type f')const [fields] = log.warn.mock.calls[0]- expect(fields).not.toHaveProperty('err')+ expect(fields).toEqual(+ expect.objectContaining({+ err: { name: 'Error', message: 'find: unrecognized: -printf' },+ })+ )expect(fields).not.toHaveProperty('directory')expect(JSON.stringify(log.warn.mock.calls)).not.toContain(sentinel)})diff --git a/control-api/test/routes.mcpSecrets.test.ts b/control-api/test/routes.mcpSecrets.test.tsindex 6cbcc4e8a..9b19190ea 100644--- a/control-api/test/routes.mcpSecrets.test.ts+++ b/control-api/test/routes.mcpSecrets.test.ts@@ -690,7 +690,11 @@ describe('PUT /admin/mcp-secrets/:name (credential rotation, issue #223)', () =>'Affected connectors unavailable after MCP Secret rotation')const [fields] = warnSpy.mock.calls[0]- expect(fields).not.toHaveProperty('err')+ expect(fields).toEqual(+ expect.objectContaining({+ err: { name: 'Error', message: 'mcpservers list forbidden' },+ })+ )expect(JSON.stringify(warnSpy.mock.calls)).not.toContain(sentinel)} finally {warnSpy.mockRestore()diff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.mdindex bad0b5504..9c192d18b 100644--- a/docs/testing/minikube-t2-runbook.md+++ b/docs/testing/minikube-t2-runbook.md@@ -115,7 +115,12 @@ same exact profile lease. Public Make targets acquire it; private body targets,before the first Docker,Minikube, or Kubernetes operation. `build-images.sh --verify-only` is theread-only exception. Empty/unknown selectors fail before the lease or runtime-is touched.+is touched. The legacy `scripts/minikube/setup.sh` path is protected the same+way for both `--build` and no-build invocations and requires matching explicit+branch-profile/context variables; it never defaults to the shared+`clerum-test` profile. The published-image puller bounds parallelism to 1-64,+retries to 1-10, and retry delay to 0-300 seconds; empty successful+`minikube docker-env` output is a hard failure.### Orphaned lock recovery@@ -310,7 +315,10 @@ REUSE_DB recovery fences all four database writers—HCC, workflow-recipes,trace-maintenance-worker, and control-api—before migrations, role provisioning,or the recovery overlay. The interrupted `HEAD` remains historical state forthe owned profile/context/worktree/branch; exact-head freshness is decided by-the pre-gate marker and image stamp.+the pre-gate marker and image stamp. The trace worker is not considered fenced+until its Deployment reports zero desired replicas and no matching pods remain.+The incremental planner treats a changed image stamp as stale in both local and+GHCR modes, even when the recorded gitHead is unchanged.Retry by phase. During a T1 failure, iterate with`minikube-t2-real-postgres`, then run one full `minikube-t2` certification oncediff --git a/scripts/minikube/full-setup.sh b/scripts/minikube/full-setup.shindex 0e140367c..99f0b25b2 100755--- a/scripts/minikube/full-setup.sh+++ b/scripts/minikube/full-setup.sh@@ -1372,20 +1372,32 @@ fence_partial_workflow_reconciler() {}fence_partial_trace_worker() {- local pods+ local pods desiredif [[ ! "${PARTIAL_TRACE_REPLICAS}" =~ ^[0-9]+$ ]]; thenerr "trace-maintenance-worker replica count is invalid; refusing partial-bootstrap recovery"return 1fi- log "Fencing trace-maintenance-worker at ${PARTIAL_TRACE_REPLICAS} replica(s)"+ log "Fencing trace-maintenance-worker at zero replicas (restore ${PARTIAL_TRACE_REPLICAS} after recovery)"writer_recovery_state_phase trace-fencing- $KC -n control-plane scale deployment/trace-maintenance-worker --replicas="${PARTIAL_TRACE_REPLICAS}" >/dev/null+ $KC -n control-plane scale deployment/trace-maintenance-worker --replicas=0 >/dev/null+ desired="$($KC -n control-plane get deployment/trace-maintenance-worker \+ -o 'jsonpath={.spec.replicas}')" || return 1+ if [[ "${desired}" != 0 ]]; then+ err "trace-maintenance-worker did not converge to zero desired replicas; refusing recovery"+ return 1+ fipods="$($KC -n control-plane get pods -l app=trace-maintenance-worker -o name)" \|| return 1if [ -n "${pods}" ]; then$KC -n control-plane wait --for=delete pod \-l app=trace-maintenance-worker --timeout=180s >/dev/nullfi+ pods="$($KC -n control-plane get pods -l app=trace-maintenance-worker -o name)" \+ || return 1+ if [ -n "${pods}" ]; then+ err "trace-maintenance-worker pods remain after the zero-replica fence; refusing recovery"+ return 1+ fiPARTIAL_TRACE_FENCED=truewriter_recovery_state_phase trace-fenced}diff --git a/scripts/minikube/pre-gate-incremental.sh b/scripts/minikube/pre-gate-incremental.shindex 7c2f7c11a..b2c8cd3b9 100755--- a/scripts/minikube/pre-gate-incremental.sh+++ b/scripts/minikube/pre-gate-incremental.sh@@ -69,13 +69,13 @@ incremental_add_target() {# The commit the cluster's images were last synced to, or nothing.#-# In ghcr mode the marker is only a usable baseline while the image set it was-# stamped against is still in place. `make minikube-setup` re-pulls every-# release image, discarding any shadow build, and does not touch this marker ---# so a marker whose acquisition stamp predates the current one describes a-# cluster that no longer exists, and trusting its gitHead would compute an empty-# delta and gate on release code. Local mode has no such hazard: a full local-# build puts every image at the tree state the marker names.+# The marker is only a usable baseline while the image set it was stamped+# against is still the set the profile can prove. Both a GHCR acquisition and a+# local build can replace image IDs without changing gitHead; trusting that+# head after a newer acquisition would compute an empty delta and gate without+# proving that the running pods use the newly acquired images. A stamp mismatch+# therefore invalidates the baseline in every image mode; the caller chooses+# the established full local build or GHCR recovery path below.incremental_marker_git_head() {local git_head marker_generatedgit_head="$(${KC} get configmap "${CLUSTER_SYNC_STATE_CONFIGMAP}" -n control-plane \@@ -83,12 +83,11 @@ incremental_marker_git_head() {if [[ -z "${git_head}" ]]; thenreturn 0fi- if [[ "${IMAGE_SOURCE}" == "ghcr" ]]; then- marker_generated="$(${KC} get configmap "${CLUSTER_SYNC_STATE_CONFIGMAP}" -n control-plane \- -o jsonpath='{.data.imagesGeneratedAt}' 2>/dev/null || true)"- if [[ "${marker_generated}" != "${IMAGES_GENERATED_AT:-}" ]]; then- return 0- fi+ marker_generated="$(${KC} get configmap "${CLUSTER_SYNC_STATE_CONFIGMAP}" -n control-plane \+ -o jsonpath='{.data.imagesGeneratedAt}' 2>/dev/null || true)"+ if [[ -z "${marker_generated}" || -z "${IMAGES_GENERATED_AT:-}" ||+ "${marker_generated}" != "${IMAGES_GENERATED_AT}" ]]; then+ return 0fiprintf '%s' "${git_head}"}diff --git a/scripts/minikube/pull-images.sh b/scripts/minikube/pull-images.shindex afa99b748..9c582a8a8 100755--- a/scripts/minikube/pull-images.sh+++ b/scripts/minikube/pull-images.sh@@ -11,7 +11,7 @@# counterpart is scripts/minikube/build-images.sh.## Usage:-# MINIKUBE_PROFILE=clerum-test ./scripts/minikube/pull-images.sh [--only=<svc>]+# MINIKUBE_PROFILE=<branch-profile> ./scripts/minikube/pull-images.sh [--only=<svc>]## Env:# MINIKUBE_IMAGE_TAG Override the recorded/committed tag (render-time@@ -19,18 +19,19 @@# the pull path before a release tag exists.# MINIKUBE_SKIP_UIS true -> do not pull control-ui/profile-ui, which# the -no-uis overlays delete anyway (see below).-# MINIKUBE_PROFILE Target profile (default: clerum-test)+# MINIKUBE_PROFILE Target branch-owned profile (provided by Make/T2)# MINIKUBE_MULTI_NODE true -> pull on the host + `minikube image load`-# MINIKUBE_PULL_PARALLELISM Concurrent pulls (default: 6)+# MINIKUBE_PULL_PARALLELISM Concurrent pulls (default: 6, range: 1-64)# MINIKUBE_IMAGE_PULL_RETRIES Attempts per image before it is reported-# failed (default: 3). A transient registry blip+# failed (default: 3, range: 1-10). A transient registry blip# (one bad pull 24 images into a run) should not# abort the whole setup. Mirrors the naming and# defaults of build-images.sh's# MINIKUBE_BASE_IMAGE_PULL_RETRIES, applied here to# the ghcr application-image pull instead of# Dockerfile base images.-# MINIKUBE_IMAGE_PULL_DELAY_SECS Delay between failed attempts (default: 5).+# MINIKUBE_IMAGE_PULL_DELAY_SECS Delay between failed attempts (default: 5,+# range: 0-300 seconds).## Two behaviours here are load-bearing; do not "optimize" them away:#@@ -94,6 +95,15 @@ for arg in "$@"; doesacdone+validate_pull_integer() {+ local name="$1" value="$2" minimum="$3" maximum="$4"+ if ! [[ "${value}" =~ ^[0-9]+$ ]] ||+ (( 10#${value} < minimum || 10#${value} > maximum )); then+ err "MINIKUBE_PULL_CONFIG_INVALID: ${name} must be an integer from ${minimum} to ${maximum}"+ exit 2+ fi+}+RED='\033[0;31m'GREEN='\033[0;32m'YELLOW='\033[1;33m'@@ -106,6 +116,10 @@ ok() { echo -e "${GREEN} OK${NC} -- $*"; }warn() { echo -e "${YELLOW} WARN${NC} -- $*"; }err() { echo -e "${RED} ERROR${NC} -- $*" >&2; }+validate_pull_integer MINIKUBE_PULL_PARALLELISM "${PULL_PARALLELISM}" 1 64+validate_pull_integer MINIKUBE_IMAGE_PULL_RETRIES "${MINIKUBE_IMAGE_PULL_RETRIES}" 1 10+validate_pull_integer MINIKUBE_IMAGE_PULL_DELAY_SECS "${MINIKUBE_IMAGE_PULL_DELAY_SECS}" 0 300+# ---- Resolve the effective tag -------------------------------------------# THE TAG FOLLOWS THE CLUSTER, NOT JUST THE COMMIT. Reading the committed pin# here (with MINIKUBE_IMAGE_TAG as the only override) ignored the tag the last@@ -199,8 +213,11 @@ if [ "$MINIKUBE_MULTI_NODE" = false ]; then"$MINIKUBE_DOCKER_ENV_TIMEOUT_SECONDS" \minikube -p "$PROFILE" docker-env --shell bash)" || docker_env_status=$?if [ "$docker_env_status" -ne 0 ] || [ -z "$docker_env_output" ]; then- err "Could not resolve Docker environment for minikube '${PROFILE}'"- exit "${docker_env_status:-1}"+ err "DOCKER_ENV_UNRESOLVED: could not resolve Docker environment for minikube '${PROFILE}'"+ if [ "$docker_env_status" -ne 0 ]; then+ exit "$docker_env_status"+ fi+ exit 1fieval "$docker_env_output"unset DOCKER_API_VERSION 2>/dev/null || truediff --git a/scripts/minikube/setup.sh b/scripts/minikube/setup.shindex fae24489b..208f33982 100755--- a/scripts/minikube/setup.sh+++ b/scripts/minikube/setup.sh@@ -8,8 +8,13 @@# model provider configuration.## Usage:-# ./scripts/minikube/setup.sh-# ./scripts/minikube/setup.sh --build # Build images under the branch lease+# MINIKUBE_PROFILE=<branch-profile> \+# CONTROL_API_REAL_PG_CONTEXT=<branch-profile> \+# ./scripts/minikube/setup.sh [--build]+#+# This is a legacy compatibility path. The supported T0/T1/T2 entrypoint is+# `make minikube-t2`; every legacy invocation still needs the same branch-owned+# profile lease and never falls back to the shared `clerum-test` profile.# ======================================================================set -euo pipefail@@ -25,7 +30,8 @@ if [ -f "${PROJECT_DIR}/.env" ]; thenset +afi-PROFILE="clerum-test"+PROFILE="${MINIKUBE_PROFILE:-${T2_PROFILE:-}}"+CONTEXT="${CONTROL_API_REAL_PG_CONTEXT:-${T2_CONTEXT:-${K8S_CONTEXT:-${KUBECONTEXT:-}}}}"BUILD_IMAGES=falsefor arg in "$@"; do@@ -34,20 +40,27 @@ for arg in "$@"; doesacdone-# This legacy setup path mutates the profile before it reaches its optional-# image build. Re-enter it through the canonical lease wrapper first, so a-# failed lease check cannot leave namespaces, keys, or Secrets half-applied.-# The wrapper marks the inherited lease; the second invocation then proceeds-# with the exact opaque token held by the parent. Fresh/unowned profiles must-# use the supported T2 bootstrap instead of bypassing this boundary.-if [ "$BUILD_IMAGES" = true ] && [ "${T2_MUTATION_LOCK_WRAPPED:-false}" != true ]; then+if [[ -z "${PROFILE}" || -z "${CONTEXT}" || "${PROFILE}" != "${CONTEXT}" ]]; then+ printf 'PROFILE_LOCK_REQUIRED: setup.sh requires matching MINIKUBE_PROFILE and CONTROL_API_REAL_PG_CONTEXT for a branch-owned profile\n' >&2+ exit 1+fi++# This legacy setup path mutates namespaces, CRDs, Secrets, workloads, and+# optionally images. Re-enter it through the canonical lease wrapper before+# the first mutation, regardless of --build. The wrapper marks the inherited+# lease; the second invocation validates the exact opaque token. Fresh or+# unowned profiles must use the supported T2 bootstrap instead of bypassing it.+if [ "${T2_MUTATION_LOCK_WRAPPED:-false}" != true ]; thenexec env \- T2_PROJECT_DIR="$PROJECT_DIR" T2_PROFILE="$PROFILE" T2_CONTEXT="$PROFILE" \- MINIKUBE_PROFILE="$PROFILE" CONTROL_API_REAL_PG_CONTEXT="$PROFILE" \+ T2_PROJECT_DIR="$PROJECT_DIR" T2_PROFILE="$PROFILE" T2_CONTEXT="$CONTEXT" \+ MINIKUBE_PROFILE="$PROFILE" CONTROL_API_REAL_PG_CONTEXT="$CONTEXT" \T2_SKIP_LOCK=false T2_GATE_ID=legacy-minikube-setup \"$SCRIPT_DIR/with-t2-mutation-lock.sh" -- "$SCRIPT_DIR/setup.sh" "$@"fi+T2_PROJECT_DIR="$PROJECT_DIR" T2_PROFILE="$PROFILE" T2_CONTEXT="$CONTEXT" \+ T2_SKIP_LOCK=true bash "$SCRIPT_DIR/require-t2-mutation-lock.sh"+RED='\033[0;31m'GREEN='\033[0;32m'YELLOW='\033[1;33m'@@ -59,7 +72,7 @@ log() { echo -e "${CYAN}[SETUP]${NC} $*"; }ok() { echo -e "${GREEN} OK${NC} -- $*"; }err() { echo -e "${RED} ERROR${NC} -- $*"; }-KC="kubectl --context=${PROFILE}"+KC="kubectl --context=${CONTEXT}"# ---- Step 1: Verify cluster -----------------------------------------echo -e "\n${BOLD}=== Step 1/12: Verify Cluster ===${NC}"diff --git a/scripts/tests/test-minikube-full-setup.sh b/scripts/tests/test-minikube-full-setup.shindex 0274f6aa6..d178585f6 100755--- a/scripts/tests/test-minikube-full-setup.sh+++ b/scripts/tests/test-minikube-full-setup.sh@@ -1245,6 +1245,94 @@ assert_the_unoverridden_path_generates_the_api_ip_patch_once() {rm -rf "$d"}+assert_trace_writer_fence_closes_and_proves_zero_replicas() {+ local fence_block+ fence_block="$(sed -n '/^fence_partial_trace_worker()/,/^}/p' scripts/minikube/full-setup.sh)"+ if grep -Fq -- '--replicas=0' <<<"$fence_block" && \+ grep -Fq "jsonpath={.spec.replicas}" <<<"$fence_block" && \+ grep -Fq 'trace-maintenance-worker did not converge to zero desired replicas' <<<"$fence_block" && \+ grep -Fq 'trace-maintenance-worker pods remain after the zero-replica fence' <<<"$fence_block"; then+ pass "trace-maintenance-worker recovery fence scales to zero and proves no writer pod remains"+ else+ fail "trace-maintenance-worker recovery can be marked fenced without proving a zero-replica state"+ fi+}++assert_trace_writer_fence_is_behaviorally_fail_closed() {+ local d fence_block run_script out rc problems=""+ d="$(mktemp -d)"+ fence_block="$(sed -n '/^fence_partial_trace_worker()/,/^}/p' scripts/minikube/full-setup.sh)"++ mkdir -p "$d/bin"+ cat > "$d/bin/kubectl" <<'STUB'+#!/usr/bin/env bash+set -u+state_dir="${TEST_TRACE_STATE_DIR:?}"+case "$*" in+ *"scale deployment/trace-maintenance-worker --replicas=0"*)+ : >"$state_dir/scaled"+ exit 0+ ;;+ *"get deployment/trace-maintenance-worker"*)+ printf '%s' "${TEST_TRACE_DESIRED:-0}"+ exit 0+ ;;+ *"get pods -l app=trace-maintenance-worker"*)+ if [ -f "$state_dir/waited" ]; then+ exit 0+ fi+ printf '%s' "${TEST_TRACE_PODS:-}"+ exit 0+ ;;+ *"wait --for=delete pod"*)+ : >"$state_dir/waited"+ exit 0+ ;;+esac+exit 0+STUB+ chmod +x "$d/bin/kubectl"++ run_script="$d/run.sh"+ {+ printf 'set -u\n'+ printf 'KC=%q\n' 'kubectl --context=test'+ printf 'PARTIAL_TRACE_REPLICAS=2\n'+ printf 'PARTIAL_TRACE_FENCED=false\n'+ printf 'STATE_DIR=%q\n' "$d"+ printf 'log() { :; }\n'+ printf 'err() { printf "ERR:%%s\\n" "$*" >&2; }\n'+ printf 'writer_recovery_state_phase() { printf "%%s\\n" "$1" >>"$STATE_DIR/phases"; }\n'+ printf '%s\n' "$fence_block"+ printf 'fence_partial_trace_worker\n'+ } >"$run_script"++ out="$(TEST_TRACE_STATE_DIR="$d" TEST_TRACE_DESIRED=0 TEST_TRACE_PODS='pod/trace-0' \+ PATH="$d/bin:$PATH" bash "$run_script" 2>&1)"+ rc=$?+ [ "$rc" -eq 0 ] || problems+="zero-replica convergence exited $rc; "+ [ -f "$d/scaled" ] || problems+="zero-replica convergence never scaled; "+ grep -Fxq 'trace-fenced' "$d/phases" || problems+="successful fence never persisted trace-fenced; "++ rm -f "$d/scaled" "$d/waited" "$d/phases"+ out="$(TEST_TRACE_STATE_DIR="$d" TEST_TRACE_DESIRED=1 TEST_TRACE_PODS= \+ PATH="$d/bin:$PATH" bash "$run_script" 2>&1)"+ rc=$?+ [ "$rc" -ne 0 ] || problems+="nonzero desired replicas were accepted; "+ if [ -f "$d/phases" ] && grep -Fxq 'trace-fenced' "$d/phases"; then+ problems+="trace-fenced was persisted after nonzero convergence; "+ fi+ grep -Fq 'did not converge to zero desired replicas' <<<"$out" \+ || problems+="nonzero convergence did not report the stable failure; "++ if [ -z "$problems" ]; then+ pass "trace-maintenance-worker fence proves convergence and rejects nonzero desired replicas"+ else+ fail "$problems out=$out"+ fi+ rm -rf "$d"+}+# The guard that keeps this file honest: a case defined but never added to the# call block below reports nothing at all, which reads as a green run.assert_every_defined_case_is_invoked() {@@ -1293,6 +1381,8 @@ assert_the_banner_prints_the_image_source_tag_and_originassert_the_tag_origin_distinguishes_pin_from_overrideassert_the_tag_override_also_generates_the_api_ip_patch_in_the_working_treeassert_the_unoverridden_path_generates_the_api_ip_patch_once+assert_trace_writer_fence_closes_and_proves_zero_replicas+assert_trace_writer_fence_is_behaviorally_fail_closedassert_skip_build_follows_the_recorded_image_sourceassert_skip_build_says_it_is_following_the_clusterassert_an_acquiring_run_still_honours_image_sourcediff --git a/scripts/tests/test-minikube-pre-gate-shadow.sh b/scripts/tests/test-minikube-pre-gate-shadow.shindex 95eb19084..93eae3297 100755--- a/scripts/tests/test-minikube-pre-gate-shadow.sh+++ b/scripts/tests/test-minikube-pre-gate-shadow.sh@@ -541,6 +541,32 @@ assert_a_re_acquisition_invalidates_the_marker_baseline_in_ghcr_mode() {rm -rf "$d"}+# A local rebuild can also replace image IDs without changing gitHead. The+# incremental planner must not treat that newer acquisition as proof that the+# old deployment is still the image set being tested.+assert_a_re_acquisition_invalidates_the_marker_baseline_in_local_mode() {+ local d out rc head+ d="$(mktemp -d)"+ prepare_repo "$d"+ head="$(repo_head "$d")"+ mkdir -p "$d/marker"+ printf '%s' "$head" > "$d/marker/gitHead"+ printf '%s' "2026-08-06T00:00:00Z" > "$d/marker/imagesGeneratedAt"+ write_manifest "$d" '{"generated":"2026-08-06T09:99:99Z","imageSource":"local","imageTag":"","images":{}}'+ out="$(+ PATH="$d/bin:$PATH" TEST_LOG_FILE="$d/ops.log" TEST_MARKER_DIR="$d/marker" \+ IMAGE_SOURCE=local IMAGE_TAG="" IMAGES_GENERATED_AT="2026-08-06T09:99:99Z" \+ run_incremental "$d" 'echo "baseline=[$(incremental_marker_git_head)]"'+ )"+ rc=$?+ if [ "$rc" -eq 0 ] && grep -Fq 'baseline=[]' <<< "$out"; then+ pass "a newer local image acquisition invalidates its gitHead baseline"+ else+ fail "expected an empty local baseline after a re-acquisition; rc=$rc out=$out"+ fi+ rm -rf "$d"+}+# The complement, so the fix cannot be "never trust the marker": an untouched# image set keeps its baseline, which is what makes the incremental path fast.assert_an_untouched_image_set_keeps_the_marker_baseline() {@@ -1024,6 +1050,7 @@ assert_the_release_image_revision_label_supplies_the_missing_baselineassert_no_marker_and_no_revision_label_fails_closedassert_an_unreachable_docker_daemon_cannot_become_a_baselineassert_a_re_acquisition_invalidates_the_marker_baseline_in_ghcr_mode+assert_a_re_acquisition_invalidates_the_marker_baseline_in_local_modeassert_an_untouched_image_set_keeps_the_marker_baselineassert_the_pre_gate_adopts_the_recorded_cluster_mode_over_the_environmentassert_a_recorded_local_build_renders_the_local_overlaydiff --git a/scripts/tests/test-minikube-pull-images.sh b/scripts/tests/test-minikube-pull-images.shindex 7f23f2281..632d69172 100755--- a/scripts/tests/test-minikube-pull-images.sh+++ b/scripts/tests/test-minikube-pull-images.sh@@ -79,6 +79,9 @@ STUB#!/usr/bin/env bashprintf 'minikube %s\n' "$*" >>"${TEST_LOG_FILE:?}"if [[ "$*" == *"docker-env"* ]]; then+ if [ "${TEST_EMPTY_DOCKER_ENV:-false}" = true ]; then+ exit 0+ fiecho 'export DOCKER_HOST="tcp://127.0.0.1:2376"'exit 0fi@@ -144,9 +147,11 @@ run_puller_prepared() {TEST_FLAKY_TAGS="${TEST_FLAKY_TAGS:-}" \TEST_FLAKY_FAIL_COUNT="${TEST_FLAKY_FAIL_COUNT:-}" \TEST_FLAKY_COUNTER_DIR="${TEST_FLAKY_COUNTER_DIR:-}" \+ TEST_EMPTY_DOCKER_ENV="${TEST_EMPTY_DOCKER_ENV:-}" \MINIKUBE_IMAGE_TAG="${MINIKUBE_IMAGE_TAG:-}" \MINIKUBE_IMAGE_PULL_RETRIES="${MINIKUBE_IMAGE_PULL_RETRIES:-}" \MINIKUBE_IMAGE_PULL_DELAY_SECS="${MINIKUBE_IMAGE_PULL_DELAY_SECS:-}" \+ MINIKUBE_PULL_PARALLELISM="${MINIKUBE_PULL_PARALLELISM:-}" \TEST_HANG_PULL="${TEST_HANG_PULL:-}" \MINIKUBE_DOCKER_PULL_TIMEOUT_SECONDS="${MINIKUBE_DOCKER_PULL_TIMEOUT_SECONDS:-}" \MINIKUBE_DOCKER_BUILD_TIMEOUT_SECONDS="${MINIKUBE_DOCKER_BUILD_TIMEOUT_SECONDS:-}" \@@ -200,6 +205,46 @@ assert_hung_pull_hits_the_finite_deadline() {rm -rf "$d"}+assert_empty_docker_env_fails_closed() {+ local d out rc+ d="$(mktemp -d)"+ export TEST_EMPTY_DOCKER_ENV=true+ out="$(run_puller "$d" --only=control-api)"; rc=$?+ unset TEST_EMPTY_DOCKER_ENV+ if [ "$rc" -ne 0 ] \+ && grep -Fq 'DOCKER_ENV_UNRESOLVED' <<<"$out" \+ && ! grep -q '^pull ' "$d/ops.log"; then+ pass "empty successful minikube docker-env output fails closed"+ else+ fail "empty docker-env output returned success or reached Docker: rc=$rc out=$out"+ fi+ rm -rf "$d"+}++assert_invalid_pull_knobs_fail_closed() {+ local d out rc+ d="$(mktemp -d)"++ export MINIKUBE_PULL_PARALLELISM=0+ out="$(run_puller "$d" --only=control-api)"; rc=$?+ unset MINIKUBE_PULL_PARALLELISM+ if [ "$rc" -ne 2 ] || ! grep -Fq 'MINIKUBE_PULL_PARALLELISM must be an integer' <<<"$out"; then+ fail "parallelism=0 did not fail with the parameter error: rc=$rc out=$out"+ rm -rf "$d"+ return+ fi++ export MINIKUBE_IMAGE_PULL_DELAY_SECS=301+ out="$(run_puller "$d" --only=control-api)"; rc=$?+ unset MINIKUBE_IMAGE_PULL_DELAY_SECS+ if [ "$rc" -eq 2 ] && grep -Fq 'MINIKUBE_IMAGE_PULL_DELAY_SECS must be an integer' <<<"$out"; then+ pass "parallelism, retry, and delay knobs fail closed outside their finite ranges"+ else+ fail "delay=301 did not fail with the parameter error: rc=$rc out=$out"+ fi+ rm -rf "$d"+}+assert_it_pulls_every_pull_in_ghcr_mode_image() {local d out rc want gotd="$(mktemp -d)"@@ -755,6 +800,8 @@ assert_every_defined_case_is_invoked() {assert_it_pulls_every_pull_in_ghcr_mode_imageassert_puller_requires_the_inherited_profile_leaseassert_hung_pull_hits_the_finite_deadline+assert_empty_docker_env_fails_closed+assert_invalid_pull_knobs_fail_closedassert_it_never_pulls_an_unpublished_imageassert_it_never_pulls_a_registry_distributed_mcp_serverassert_it_repulls_a_tag_already_present_in_the_daemondiff --git a/scripts/tests/test-minikube-t2-contract.sh b/scripts/tests/test-minikube-t2-contract.shindex 6f62da4ba..7c0656a49 100755--- a/scripts/tests/test-minikube-t2-contract.sh+++ b/scripts/tests/test-minikube-t2-contract.sh@@ -149,6 +149,19 @@ grep -Fq 't2-setup-handoff.sh" consume --' "$ROOT/scripts/minikube/pre-gate-syncgrep -Fq 'minikube-verify-images' "$ROOT/scripts/minikube/pre-gate-sync.sh"grep -Fq 'with-t2-mutation-lock.sh' "$ROOT/scripts/minikube/setup.sh"grep -Fq 'T2_MUTATION_LOCK_WRAPPED' "$ROOT/scripts/minikube/setup.sh"+grep -Fq 'require-t2-mutation-lock.sh' "$ROOT/scripts/minikube/setup.sh"+grep -Fq 'PROFILE_LOCK_REQUIRED: setup.sh requires matching' "$ROOT/scripts/minikube/setup.sh"+grep -Fq 'PROFILE="${MINIKUBE_PROFILE:-${T2_PROFILE:-}}"' "$ROOT/scripts/minikube/setup.sh"+if grep -Fq 'PROFILE="clerum-test"' "$ROOT/scripts/minikube/setup.sh"; then+ echo 'FAIL: legacy setup still defaults to the shared clerum-test profile' >&2+ exit 1+fi+setup_guard_line="$(grep -nF 'if [ "${T2_MUTATION_LOCK_WRAPPED:-false}" != true ]; then' "$ROOT/scripts/minikube/setup.sh" | cut -d: -f1)"+setup_cluster_line="$(grep -nF 'cluster-info' "$ROOT/scripts/minikube/setup.sh" | head -1 | cut -d: -f1)"+if [[ -z "$setup_guard_line" || -z "$setup_cluster_line" || "$setup_guard_line" -ge "$setup_cluster_line" ]]; then+ echo 'FAIL: legacy setup can mutate or probe the cluster before the lease wrapper' >&2+ exit 1+fifor timed_phase in 't2_evidence_write planner PASS' 't2_evidence_write transition PASS' \'t2_evidence_write pre-gate-sync PASS' 't2_evidence_write T1 PASS' \'t2_evidence_write T2 PASS' 't2_evidence_write NP08_HCC_AUTHORIZATION PASS'; do