Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix(minikube): close remaining recovery and puller gaps

Alfredo Lopez37a9ad3ec737
diff --git a/.cursor/skills/minikube-t0-t1-t2/SKILL.md b/.cursor/skills/minikube-t0-t1-t2/SKILL.md
index 6d2047815..032d470c5 100644
--- a/.cursor/skills/minikube-t0-t1-t2/SKILL.md
+++ b/.cursor/skills/minikube-t0-t1-t2/SKILL.md
@@ -37,6 +37,10 @@ Before running anything, verify ALL of these:
- [ ] Mutating image acquisition/builds use the public Make target/orchestrator
and inherit its exact profile lease. Do not call `build-images.sh` or
`pull-images.sh` directly; `--verify-only` is the read-only exception.
+ The legacy `setup.sh` path is also lease-protected for every invocation,
+ requires matching explicit profile/context variables, and never uses the
+ shared `clerum-test` profile. Pull configuration values are finite and an
+ empty successful `minikube docker-env` result is a hard failure.
- [ ] Never read `~/.cache/clerum/minikube-profiles/` directly (HARD DENY —
it holds private profile state). The harness reads it for you.
@@ -155,8 +159,12 @@ Before committing anything else, run `make minikube-t2-public-boundary`.
The final exact-head check also rejects a marker whose `imagesGeneratedAt`
does not equal the current image manifest. During REUSE_DB recovery, the
durable fence covers HCC, workflow-recipes, trace-maintenance-worker, and
-control-api; a changed HEAD on the same owned lane is historical state, not a
-reason to discard recovery evidence.
+control-api. The incremental baseline applies that image-stamp check in both
+GHCR and local image modes, because a matching gitHead does not prove that a
+newly acquired image ID is running. The trace worker must be at zero desired
+replicas with no remaining pods before `trace-fenced` is persisted. A changed
+HEAD on the same owned lane is historical state, not a reason to discard
+recovery evidence.
## Step 5 — On failure
diff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.md
index 6facbeecf..c25b515ee 100644
--- a/.cursor/skills/minikube-t0-t1-t2/reference.md
+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md
@@ -66,6 +66,8 @@ Companion to `SKILL.md`. Source of truth: `scripts/minikube/t2.sh`,
| `PROFILE_LOCK_REQUIRED` | A mutating child did not inherit the exact parent lease. | Invoke the public Make target/orchestrator; do not fabricate or bypass its lock token. |
| `DOCKER_ENDPOINT_REQUIRED` / `DOCKER_ENDPOINT_UNSAFE` / `DOCKER_ENDPOINT_MISMATCH` | Docker did not resolve exactly one approved local endpoint, or isolation selected a different endpoint. | Select a local Docker Desktop/rootless context or set an explicit local `DOCKER_HOST`; do not weaken the endpoint check. |
| `DOCKER_CONFIG_REQUIRED` / `REGISTRY_AUTH_REQUIRED` | A Docker operation lacks its empty task-local config, or a private pull lacks an explicit readable auth config. | Use the canonical Make/orchestrator path; for a genuinely private source, pass only the intended config directory through `MINIKUBE_DOCKER_AUTH_CONFIG`. |
+| `MINIKUBE_PULL_CONFIG_INVALID` | Published-image pull parallelism, retry count, or retry delay is outside its finite range. | Correct the override (`parallelism 1-64`, retries 1-10, delay 0-300 seconds); never disable the bound. |
+| `DOCKER_ENV_UNRESOLVED` | Minikube returned no usable Docker environment, including a successful empty response. | Repair the profile/runtime endpoint and retry the same owned Make target; no image pull may proceed. |
| `DOCKER_DEADLINE_INVALID` | A runtime deadline/retry value is malformed or exceeds its finite maximum. | Remove the override or set it within the documented range; do not disable the deadline. |
| `BOOTSTRAP_REQUIRED` | Profile missing/uninitialized, or the planner produced no transition. Standalone preflight refuses to bootstrap. | Run `MINIKUBE_PROFILE=<owned> CONTROL_API_REAL_PG_CONTEXT=<owned> make minikube-t2`; its internal planner makes `full-bootstrap` reachable. |
| `HEAD_MARKER_MISMATCH` | Pre-gate marker does not match this worktree/HEAD; the final T2 preflight selected something other than `already-synced`. | Re-run the full target on this HEAD so `pre-gate-sync` updates the marker. Never hand-edit the marker. |
diff --git a/.github/workflows/ci-public.yml b/.github/workflows/ci-public.yml
index 91c1be1ad..73e46030c 100644
--- a/.github/workflows/ci-public.yml
+++ b/.github/workflows/ci-public.yml
@@ -484,6 +484,12 @@ jobs:
- name: Test minikube Makefile targets
run: bash scripts/tests/test-minikube-makefile.sh
+ - name: Test local Minikube T0/T1/T2 contract
+ run: bash scripts/tests/test-minikube-t2-contract.sh
+
+ - name: Test local Minikube T2 setup handoff
+ run: bash scripts/tests/test-minikube-t2-setup-handoff.sh
+
- name: Test minikube GFS Upload v2 profile contract
run: bash scripts/tests/test-minikube-gfs-upload-v2-profile.sh
diff --git a/AGENTS.md b/AGENTS.md
index 8036b2be2..043e92307 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -50,6 +50,13 @@ use finite deadlines. Docker endpoint discovery must resolve
to an explicit local Unix socket or loopback TCP endpoint before switching to
an empty task-local Docker config. Ambient registry credentials are never
copied; a private pull requires an explicit `MINIKUBE_DOCKER_AUTH_CONFIG`.
+The legacy `scripts/minikube/setup.sh` compatibility path follows the same
+boundary: it requires matching `MINIKUBE_PROFILE` and
+`CONTROL_API_REAL_PG_CONTEXT`, acquires the branch-owned lease before its first
+cluster operation for every invocation (with or without `--build`), and never
+falls back to the shared `clerum-test` profile. The published-image puller
+rejects invalid parallelism/retry/delay values and an empty successful
+`minikube docker-env`; neither condition may produce a green acquisition.
## Incremental local image updates
@@ -80,6 +87,12 @@ profile is already healthy; record it as a *targeted sync* and prove the
affected deployment is Ready plus its user-facing health endpoint. Do not
report that as a full reconcile or as T2.
+The image acquisition stamp is part of the baseline in both GHCR and local
+image modes. If the manifest's `generated` value differs from the pre-gate
+marker, the incremental planner must invalidate the old `gitHead` baseline and
+reconcile before certifying; a matching commit alone does not prove that pods
+run the newly acquired image IDs.
+
Expect a full reconcile when deployment manifests, CRDs, charts, network
policy, or other infrastructure inputs changed; do not replace that safe path
with partial image updates. Every `kubectl` invocation must use the verified
@@ -139,6 +152,10 @@ When REUSE_DB recovery is needed, the fence covers all four database writers:
HCC, workflow-recipes, trace-maintenance-worker, and control-api. The durable
recovery identity is the owned profile/context/worktree/branch; the interrupted
HEAD is retained as audit data while the exact-head marker controls freshness.
+The trace-maintenance-worker fence must scale its Deployment to zero and prove
+both `.spec.replicas == 0` and no remaining writer pods before persisting the
+`trace-fenced` phase; restoring its saved replica count happens only after the
+recovery window.
Real PostgreSQL suites are opt-in in the ordinary test
matrix, but a T1 run that requires them must fail when the database/DSN is
unavailable or when zero tests execute; a green run must never be produced by
diff --git a/control-api/package-lock.json b/control-api/package-lock.json
index bff1abf2b..63275ff3e 100644
--- a/control-api/package-lock.json
+++ b/control-api/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "clerum-control-api",
- "version": "0.1.491",
+ "version": "0.1.492",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "clerum-control-api",
- "version": "0.1.491",
+ "version": "0.1.492",
"dependencies": {
"@clerum/display-field": "file:../packages/display-field",
"@clerum/image-policy": "file:../packages/image-policy",
diff --git a/control-api/package.json b/control-api/package.json
index f4fad6068..c57856d50 100644
--- a/control-api/package.json
+++ b/control-api/package.json
@@ -1,6 +1,6 @@
{
"name": "clerum-control-api",
- "version": "0.1.491",
+ "version": "0.1.492",
"description": "Internal Clerum control plane REST API + UI for managing CRDs and secrets",
"main": "dist/main.js",
"engines": {
diff --git a/control-api/src/k8s.ts b/control-api/src/k8s.ts
index 73aaef994..a8bcae7a8 100644
--- a/control-api/src/k8s.ts
+++ b/control-api/src/k8s.ts
@@ -573,6 +573,10 @@ export class K8sGateway {
podName,
namespace,
containerName,
+ err: {
+ name: findError instanceof Error ? findError.name : typeof findError,
+ message: findError instanceof Error ? findError.message : String(findError),
+ },
},
'find -printf failed; falling back to find -type f'
)
diff --git a/control-api/src/routes/admin/secrets.ts b/control-api/src/routes/admin/secrets.ts
index 8951c8e3d..623302d0f 100644
--- a/control-api/src/routes/admin/secrets.ts
+++ b/control-api/src/routes/admin/secrets.ts
@@ -514,13 +514,17 @@ export function createAdminSecretsRouter(gateway: K8sGateway): Router {
.map(item => (item as { metadata?: { name?: string } }).metadata?.name)
.filter((connector): connector is string => typeof connector === 'string')
.sort((a, b) => a.localeCompare(b))
- } catch {
+ } catch (err) {
logger.warn(
{
module: 'admin-secrets',
event: 'mcp-secret-rotated-affected-connectors-unavailable',
name,
namespace: targetNs,
+ err: {
+ name: err instanceof Error ? err.name : typeof err,
+ message: err instanceof Error ? err.message : String(err),
+ },
},
'Affected connectors unavailable after MCP Secret rotation'
)
diff --git a/control-api/test/k8s.execSecurity.test.ts b/control-api/test/k8s.execSecurity.test.ts
index 5efa652c6..745fa3426 100644
--- a/control-api/test/k8s.execSecurity.test.ts
+++ b/control-api/test/k8s.execSecurity.test.ts
@@ -357,7 +357,11 @@ describe('artifact listing commands', () => {
'find -printf failed; falling back to find -type f'
)
const [fields] = log.warn.mock.calls[0]
- expect(fields).not.toHaveProperty('err')
+ expect(fields).toEqual(
+ expect.objectContaining({
+ err: { name: 'Error', message: 'find: unrecognized: -printf' },
+ })
+ )
expect(fields).not.toHaveProperty('directory')
expect(JSON.stringify(log.warn.mock.calls)).not.toContain(sentinel)
})
diff --git a/control-api/test/routes.mcpSecrets.test.ts b/control-api/test/routes.mcpSecrets.test.ts
index 6cbcc4e8a..9b19190ea 100644
--- a/control-api/test/routes.mcpSecrets.test.ts
+++ b/control-api/test/routes.mcpSecrets.test.ts
@@ -690,7 +690,11 @@ describe('PUT /admin/mcp-secrets/:name (credential rotation, issue #223)', () =>
'Affected connectors unavailable after MCP Secret rotation'
)
const [fields] = warnSpy.mock.calls[0]
- expect(fields).not.toHaveProperty('err')
+ expect(fields).toEqual(
+ expect.objectContaining({
+ err: { name: 'Error', message: 'mcpservers list forbidden' },
+ })
+ )
expect(JSON.stringify(warnSpy.mock.calls)).not.toContain(sentinel)
} finally {
warnSpy.mockRestore()
diff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.md
index bad0b5504..9c192d18b 100644
--- a/docs/testing/minikube-t2-runbook.md
+++ b/docs/testing/minikube-t2-runbook.md
@@ -115,7 +115,12 @@ same exact profile lease. Public Make targets acquire it; private body targets,
before the first Docker,
Minikube, or Kubernetes operation. `build-images.sh --verify-only` is the
read-only exception. Empty/unknown selectors fail before the lease or runtime
-is touched.
+is touched. The legacy `scripts/minikube/setup.sh` path is protected the same
+way for both `--build` and no-build invocations and requires matching explicit
+branch-profile/context variables; it never defaults to the shared
+`clerum-test` profile. The published-image puller bounds parallelism to 1-64,
+retries to 1-10, and retry delay to 0-300 seconds; empty successful
+`minikube docker-env` output is a hard failure.
### Orphaned lock recovery
@@ -310,7 +315,10 @@ REUSE_DB recovery fences all four database writers—HCC, workflow-recipes,
trace-maintenance-worker, and control-api—before migrations, role provisioning,
or the recovery overlay. The interrupted `HEAD` remains historical state for
the owned profile/context/worktree/branch; exact-head freshness is decided by
-the pre-gate marker and image stamp.
+the pre-gate marker and image stamp. The trace worker is not considered fenced
+until its Deployment reports zero desired replicas and no matching pods remain.
+The incremental planner treats a changed image stamp as stale in both local and
+GHCR modes, even when the recorded gitHead is unchanged.
Retry by phase. During a T1 failure, iterate with
`minikube-t2-real-postgres`, then run one full `minikube-t2` certification once
diff --git a/scripts/minikube/full-setup.sh b/scripts/minikube/full-setup.sh
index 0e140367c..99f0b25b2 100755
--- a/scripts/minikube/full-setup.sh
+++ b/scripts/minikube/full-setup.sh
@@ -1372,20 +1372,32 @@ fence_partial_workflow_reconciler() {
}
fence_partial_trace_worker() {
- local pods
+ local pods desired
if [[ ! "${PARTIAL_TRACE_REPLICAS}" =~ ^[0-9]+$ ]]; then
err "trace-maintenance-worker replica count is invalid; refusing partial-bootstrap recovery"
return 1
fi
- log "Fencing trace-maintenance-worker at ${PARTIAL_TRACE_REPLICAS} replica(s)"
+ log "Fencing trace-maintenance-worker at zero replicas (restore ${PARTIAL_TRACE_REPLICAS} after recovery)"
writer_recovery_state_phase trace-fencing
- $KC -n control-plane scale deployment/trace-maintenance-worker --replicas="${PARTIAL_TRACE_REPLICAS}" >/dev/null
+ $KC -n control-plane scale deployment/trace-maintenance-worker --replicas=0 >/dev/null
+ desired="$($KC -n control-plane get deployment/trace-maintenance-worker \
+ -o 'jsonpath={.spec.replicas}')" || return 1
+ if [[ "${desired}" != 0 ]]; then
+ err "trace-maintenance-worker did not converge to zero desired replicas; refusing recovery"
+ return 1
+ fi
pods="$($KC -n control-plane get pods -l app=trace-maintenance-worker -o name)" \
|| return 1
if [ -n "${pods}" ]; then
$KC -n control-plane wait --for=delete pod \
-l app=trace-maintenance-worker --timeout=180s >/dev/null
fi
+ pods="$($KC -n control-plane get pods -l app=trace-maintenance-worker -o name)" \
+ || return 1
+ if [ -n "${pods}" ]; then
+ err "trace-maintenance-worker pods remain after the zero-replica fence; refusing recovery"
+ return 1
+ fi
PARTIAL_TRACE_FENCED=true
writer_recovery_state_phase trace-fenced
}
diff --git a/scripts/minikube/pre-gate-incremental.sh b/scripts/minikube/pre-gate-incremental.sh
index 7c2f7c11a..b2c8cd3b9 100755
--- a/scripts/minikube/pre-gate-incremental.sh
+++ b/scripts/minikube/pre-gate-incremental.sh
@@ -69,13 +69,13 @@ incremental_add_target() {
# The commit the cluster's images were last synced to, or nothing.
#
-# In ghcr mode the marker is only a usable baseline while the image set it was
-# stamped against is still in place. `make minikube-setup` re-pulls every
-# release image, discarding any shadow build, and does not touch this marker --
-# so a marker whose acquisition stamp predates the current one describes a
-# cluster that no longer exists, and trusting its gitHead would compute an empty
-# delta and gate on release code. Local mode has no such hazard: a full local
-# build puts every image at the tree state the marker names.
+# The marker is only a usable baseline while the image set it was stamped
+# against is still the set the profile can prove. Both a GHCR acquisition and a
+# local build can replace image IDs without changing gitHead; trusting that
+# head after a newer acquisition would compute an empty delta and gate without
+# proving that the running pods use the newly acquired images. A stamp mismatch
+# therefore invalidates the baseline in every image mode; the caller chooses
+# the established full local build or GHCR recovery path below.
incremental_marker_git_head() {
local git_head marker_generated
git_head="$(${KC} get configmap "${CLUSTER_SYNC_STATE_CONFIGMAP}" -n control-plane \
@@ -83,12 +83,11 @@ incremental_marker_git_head() {
if [[ -z "${git_head}" ]]; then
return 0
fi
- if [[ "${IMAGE_SOURCE}" == "ghcr" ]]; then
- marker_generated="$(${KC} get configmap "${CLUSTER_SYNC_STATE_CONFIGMAP}" -n control-plane \
- -o jsonpath='{.data.imagesGeneratedAt}' 2>/dev/null || true)"
- if [[ "${marker_generated}" != "${IMAGES_GENERATED_AT:-}" ]]; then
- return 0
- fi
+ marker_generated="$(${KC} get configmap "${CLUSTER_SYNC_STATE_CONFIGMAP}" -n control-plane \
+ -o jsonpath='{.data.imagesGeneratedAt}' 2>/dev/null || true)"
+ if [[ -z "${marker_generated}" || -z "${IMAGES_GENERATED_AT:-}" ||
+ "${marker_generated}" != "${IMAGES_GENERATED_AT}" ]]; then
+ return 0
fi
printf '%s' "${git_head}"
}
diff --git a/scripts/minikube/pull-images.sh b/scripts/minikube/pull-images.sh
index afa99b748..9c582a8a8 100755
--- a/scripts/minikube/pull-images.sh
+++ b/scripts/minikube/pull-images.sh
@@ -11,7 +11,7 @@
# counterpart is scripts/minikube/build-images.sh.
#
# Usage:
-# MINIKUBE_PROFILE=clerum-test ./scripts/minikube/pull-images.sh [--only=<svc>]
+# MINIKUBE_PROFILE=<branch-profile> ./scripts/minikube/pull-images.sh [--only=<svc>]
#
# Env:
# MINIKUBE_IMAGE_TAG Override the recorded/committed tag (render-time
@@ -19,18 +19,19 @@
# the pull path before a release tag exists.
# MINIKUBE_SKIP_UIS true -> do not pull control-ui/profile-ui, which
# the -no-uis overlays delete anyway (see below).
-# MINIKUBE_PROFILE Target profile (default: clerum-test)
+# MINIKUBE_PROFILE Target branch-owned profile (provided by Make/T2)
# MINIKUBE_MULTI_NODE true -> pull on the host + `minikube image load`
-# MINIKUBE_PULL_PARALLELISM Concurrent pulls (default: 6)
+# MINIKUBE_PULL_PARALLELISM Concurrent pulls (default: 6, range: 1-64)
# MINIKUBE_IMAGE_PULL_RETRIES Attempts per image before it is reported
-# failed (default: 3). A transient registry blip
+# failed (default: 3, range: 1-10). A transient registry blip
# (one bad pull 24 images into a run) should not
# abort the whole setup. Mirrors the naming and
# defaults of build-images.sh's
# MINIKUBE_BASE_IMAGE_PULL_RETRIES, applied here to
# the ghcr application-image pull instead of
# Dockerfile base images.
-# MINIKUBE_IMAGE_PULL_DELAY_SECS Delay between failed attempts (default: 5).
+# MINIKUBE_IMAGE_PULL_DELAY_SECS Delay between failed attempts (default: 5,
+# range: 0-300 seconds).
#
# Two behaviours here are load-bearing; do not "optimize" them away:
#
@@ -94,6 +95,15 @@ for arg in "$@"; do
esac
done
+validate_pull_integer() {
+ local name="$1" value="$2" minimum="$3" maximum="$4"
+ if ! [[ "${value}" =~ ^[0-9]+$ ]] ||
+ (( 10#${value} < minimum || 10#${value} > maximum )); then
+ err "MINIKUBE_PULL_CONFIG_INVALID: ${name} must be an integer from ${minimum} to ${maximum}"
+ exit 2
+ fi
+}
+
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
@@ -106,6 +116,10 @@ ok() { echo -e "${GREEN} OK${NC} -- $*"; }
warn() { echo -e "${YELLOW} WARN${NC} -- $*"; }
err() { echo -e "${RED} ERROR${NC} -- $*" >&2; }
+validate_pull_integer MINIKUBE_PULL_PARALLELISM "${PULL_PARALLELISM}" 1 64
+validate_pull_integer MINIKUBE_IMAGE_PULL_RETRIES "${MINIKUBE_IMAGE_PULL_RETRIES}" 1 10
+validate_pull_integer MINIKUBE_IMAGE_PULL_DELAY_SECS "${MINIKUBE_IMAGE_PULL_DELAY_SECS}" 0 300
+
# ---- Resolve the effective tag -------------------------------------------
# THE TAG FOLLOWS THE CLUSTER, NOT JUST THE COMMIT. Reading the committed pin
# here (with MINIKUBE_IMAGE_TAG as the only override) ignored the tag the last
@@ -199,8 +213,11 @@ if [ "$MINIKUBE_MULTI_NODE" = false ]; then
"$MINIKUBE_DOCKER_ENV_TIMEOUT_SECONDS" \
minikube -p "$PROFILE" docker-env --shell bash)" || docker_env_status=$?
if [ "$docker_env_status" -ne 0 ] || [ -z "$docker_env_output" ]; then
- err "Could not resolve Docker environment for minikube '${PROFILE}'"
- exit "${docker_env_status:-1}"
+ err "DOCKER_ENV_UNRESOLVED: could not resolve Docker environment for minikube '${PROFILE}'"
+ if [ "$docker_env_status" -ne 0 ]; then
+ exit "$docker_env_status"
+ fi
+ exit 1
fi
eval "$docker_env_output"
unset DOCKER_API_VERSION 2>/dev/null || true
diff --git a/scripts/minikube/setup.sh b/scripts/minikube/setup.sh
index fae24489b..208f33982 100755
--- a/scripts/minikube/setup.sh
+++ b/scripts/minikube/setup.sh
@@ -8,8 +8,13 @@
# model provider configuration.
#
# Usage:
-# ./scripts/minikube/setup.sh
-# ./scripts/minikube/setup.sh --build # Build images under the branch lease
+# MINIKUBE_PROFILE=<branch-profile> \
+# CONTROL_API_REAL_PG_CONTEXT=<branch-profile> \
+# ./scripts/minikube/setup.sh [--build]
+#
+# This is a legacy compatibility path. The supported T0/T1/T2 entrypoint is
+# `make minikube-t2`; every legacy invocation still needs the same branch-owned
+# profile lease and never falls back to the shared `clerum-test` profile.
# ======================================================================
set -euo pipefail
@@ -25,7 +30,8 @@ if [ -f "${PROJECT_DIR}/.env" ]; then
set +a
fi
-PROFILE="clerum-test"
+PROFILE="${MINIKUBE_PROFILE:-${T2_PROFILE:-}}"
+CONTEXT="${CONTROL_API_REAL_PG_CONTEXT:-${T2_CONTEXT:-${K8S_CONTEXT:-${KUBECONTEXT:-}}}}"
BUILD_IMAGES=false
for arg in "$@"; do
@@ -34,20 +40,27 @@ for arg in "$@"; do
esac
done
-# This legacy setup path mutates the profile before it reaches its optional
-# image build. Re-enter it through the canonical lease wrapper first, so a
-# failed lease check cannot leave namespaces, keys, or Secrets half-applied.
-# The wrapper marks the inherited lease; the second invocation then proceeds
-# with the exact opaque token held by the parent. Fresh/unowned profiles must
-# use the supported T2 bootstrap instead of bypassing this boundary.
-if [ "$BUILD_IMAGES" = true ] && [ "${T2_MUTATION_LOCK_WRAPPED:-false}" != true ]; then
+if [[ -z "${PROFILE}" || -z "${CONTEXT}" || "${PROFILE}" != "${CONTEXT}" ]]; then
+ printf 'PROFILE_LOCK_REQUIRED: setup.sh requires matching MINIKUBE_PROFILE and CONTROL_API_REAL_PG_CONTEXT for a branch-owned profile\n' >&2
+ exit 1
+fi
+
+# This legacy setup path mutates namespaces, CRDs, Secrets, workloads, and
+# optionally images. Re-enter it through the canonical lease wrapper before
+# the first mutation, regardless of --build. The wrapper marks the inherited
+# lease; the second invocation validates the exact opaque token. Fresh or
+# unowned profiles must use the supported T2 bootstrap instead of bypassing it.
+if [ "${T2_MUTATION_LOCK_WRAPPED:-false}" != true ]; then
exec env \
- T2_PROJECT_DIR="$PROJECT_DIR" T2_PROFILE="$PROFILE" T2_CONTEXT="$PROFILE" \
- MINIKUBE_PROFILE="$PROFILE" CONTROL_API_REAL_PG_CONTEXT="$PROFILE" \
+ T2_PROJECT_DIR="$PROJECT_DIR" T2_PROFILE="$PROFILE" T2_CONTEXT="$CONTEXT" \
+ MINIKUBE_PROFILE="$PROFILE" CONTROL_API_REAL_PG_CONTEXT="$CONTEXT" \
T2_SKIP_LOCK=false T2_GATE_ID=legacy-minikube-setup \
"$SCRIPT_DIR/with-t2-mutation-lock.sh" -- "$SCRIPT_DIR/setup.sh" "$@"
fi
+T2_PROJECT_DIR="$PROJECT_DIR" T2_PROFILE="$PROFILE" T2_CONTEXT="$CONTEXT" \
+ T2_SKIP_LOCK=true bash "$SCRIPT_DIR/require-t2-mutation-lock.sh"
+
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
@@ -59,7 +72,7 @@ log() { echo -e "${CYAN}[SETUP]${NC} $*"; }
ok() { echo -e "${GREEN} OK${NC} -- $*"; }
err() { echo -e "${RED} ERROR${NC} -- $*"; }
-KC="kubectl --context=${PROFILE}"
+KC="kubectl --context=${CONTEXT}"
# ---- Step 1: Verify cluster -----------------------------------------
echo -e "\n${BOLD}=== Step 1/12: Verify Cluster ===${NC}"
diff --git a/scripts/tests/test-minikube-full-setup.sh b/scripts/tests/test-minikube-full-setup.sh
index 0274f6aa6..d178585f6 100755
--- a/scripts/tests/test-minikube-full-setup.sh
+++ b/scripts/tests/test-minikube-full-setup.sh
@@ -1245,6 +1245,94 @@ assert_the_unoverridden_path_generates_the_api_ip_patch_once() {
rm -rf "$d"
}
+assert_trace_writer_fence_closes_and_proves_zero_replicas() {
+ local fence_block
+ fence_block="$(sed -n '/^fence_partial_trace_worker()/,/^}/p' scripts/minikube/full-setup.sh)"
+ if grep -Fq -- '--replicas=0' <<<"$fence_block" && \
+ grep -Fq "jsonpath={.spec.replicas}" <<<"$fence_block" && \
+ grep -Fq 'trace-maintenance-worker did not converge to zero desired replicas' <<<"$fence_block" && \
+ grep -Fq 'trace-maintenance-worker pods remain after the zero-replica fence' <<<"$fence_block"; then
+ pass "trace-maintenance-worker recovery fence scales to zero and proves no writer pod remains"
+ else
+ fail "trace-maintenance-worker recovery can be marked fenced without proving a zero-replica state"
+ fi
+}
+
+assert_trace_writer_fence_is_behaviorally_fail_closed() {
+ local d fence_block run_script out rc problems=""
+ d="$(mktemp -d)"
+ fence_block="$(sed -n '/^fence_partial_trace_worker()/,/^}/p' scripts/minikube/full-setup.sh)"
+
+ mkdir -p "$d/bin"
+ cat > "$d/bin/kubectl" <<'STUB'
+#!/usr/bin/env bash
+set -u
+state_dir="${TEST_TRACE_STATE_DIR:?}"
+case "$*" in
+ *"scale deployment/trace-maintenance-worker --replicas=0"*)
+ : >"$state_dir/scaled"
+ exit 0
+ ;;
+ *"get deployment/trace-maintenance-worker"*)
+ printf '%s' "${TEST_TRACE_DESIRED:-0}"
+ exit 0
+ ;;
+ *"get pods -l app=trace-maintenance-worker"*)
+ if [ -f "$state_dir/waited" ]; then
+ exit 0
+ fi
+ printf '%s' "${TEST_TRACE_PODS:-}"
+ exit 0
+ ;;
+ *"wait --for=delete pod"*)
+ : >"$state_dir/waited"
+ exit 0
+ ;;
+esac
+exit 0
+STUB
+ chmod +x "$d/bin/kubectl"
+
+ run_script="$d/run.sh"
+ {
+ printf 'set -u\n'
+ printf 'KC=%q\n' 'kubectl --context=test'
+ printf 'PARTIAL_TRACE_REPLICAS=2\n'
+ printf 'PARTIAL_TRACE_FENCED=false\n'
+ printf 'STATE_DIR=%q\n' "$d"
+ printf 'log() { :; }\n'
+ printf 'err() { printf "ERR:%%s\\n" "$*" >&2; }\n'
+ printf 'writer_recovery_state_phase() { printf "%%s\\n" "$1" >>"$STATE_DIR/phases"; }\n'
+ printf '%s\n' "$fence_block"
+ printf 'fence_partial_trace_worker\n'
+ } >"$run_script"
+
+ out="$(TEST_TRACE_STATE_DIR="$d" TEST_TRACE_DESIRED=0 TEST_TRACE_PODS='pod/trace-0' \
+ PATH="$d/bin:$PATH" bash "$run_script" 2>&1)"
+ rc=$?
+ [ "$rc" -eq 0 ] || problems+="zero-replica convergence exited $rc; "
+ [ -f "$d/scaled" ] || problems+="zero-replica convergence never scaled; "
+ grep -Fxq 'trace-fenced' "$d/phases" || problems+="successful fence never persisted trace-fenced; "
+
+ rm -f "$d/scaled" "$d/waited" "$d/phases"
+ out="$(TEST_TRACE_STATE_DIR="$d" TEST_TRACE_DESIRED=1 TEST_TRACE_PODS= \
+ PATH="$d/bin:$PATH" bash "$run_script" 2>&1)"
+ rc=$?
+ [ "$rc" -ne 0 ] || problems+="nonzero desired replicas were accepted; "
+ if [ -f "$d/phases" ] && grep -Fxq 'trace-fenced' "$d/phases"; then
+ problems+="trace-fenced was persisted after nonzero convergence; "
+ fi
+ grep -Fq 'did not converge to zero desired replicas' <<<"$out" \
+ || problems+="nonzero convergence did not report the stable failure; "
+
+ if [ -z "$problems" ]; then
+ pass "trace-maintenance-worker fence proves convergence and rejects nonzero desired replicas"
+ else
+ fail "$problems out=$out"
+ fi
+ rm -rf "$d"
+}
+
# The guard that keeps this file honest: a case defined but never added to the
# call block below reports nothing at all, which reads as a green run.
assert_every_defined_case_is_invoked() {
@@ -1293,6 +1381,8 @@ assert_the_banner_prints_the_image_source_tag_and_origin
assert_the_tag_origin_distinguishes_pin_from_override
assert_the_tag_override_also_generates_the_api_ip_patch_in_the_working_tree
assert_the_unoverridden_path_generates_the_api_ip_patch_once
+assert_trace_writer_fence_closes_and_proves_zero_replicas
+assert_trace_writer_fence_is_behaviorally_fail_closed
assert_skip_build_follows_the_recorded_image_source
assert_skip_build_says_it_is_following_the_cluster
assert_an_acquiring_run_still_honours_image_source
diff --git a/scripts/tests/test-minikube-pre-gate-shadow.sh b/scripts/tests/test-minikube-pre-gate-shadow.sh
index 95eb19084..93eae3297 100755
--- a/scripts/tests/test-minikube-pre-gate-shadow.sh
+++ b/scripts/tests/test-minikube-pre-gate-shadow.sh
@@ -541,6 +541,32 @@ assert_a_re_acquisition_invalidates_the_marker_baseline_in_ghcr_mode() {
rm -rf "$d"
}
+# A local rebuild can also replace image IDs without changing gitHead. The
+# incremental planner must not treat that newer acquisition as proof that the
+# old deployment is still the image set being tested.
+assert_a_re_acquisition_invalidates_the_marker_baseline_in_local_mode() {
+ local d out rc head
+ d="$(mktemp -d)"
+ prepare_repo "$d"
+ head="$(repo_head "$d")"
+ mkdir -p "$d/marker"
+ printf '%s' "$head" > "$d/marker/gitHead"
+ printf '%s' "2026-08-06T00:00:00Z" > "$d/marker/imagesGeneratedAt"
+ write_manifest "$d" '{"generated":"2026-08-06T09:99:99Z","imageSource":"local","imageTag":"","images":{}}'
+ out="$(
+ PATH="$d/bin:$PATH" TEST_LOG_FILE="$d/ops.log" TEST_MARKER_DIR="$d/marker" \
+ IMAGE_SOURCE=local IMAGE_TAG="" IMAGES_GENERATED_AT="2026-08-06T09:99:99Z" \
+ run_incremental "$d" 'echo "baseline=[$(incremental_marker_git_head)]"'
+ )"
+ rc=$?
+ if [ "$rc" -eq 0 ] && grep -Fq 'baseline=[]' <<< "$out"; then
+ pass "a newer local image acquisition invalidates its gitHead baseline"
+ else
+ fail "expected an empty local baseline after a re-acquisition; rc=$rc out=$out"
+ fi
+ rm -rf "$d"
+}
+
# The complement, so the fix cannot be "never trust the marker": an untouched
# image set keeps its baseline, which is what makes the incremental path fast.
assert_an_untouched_image_set_keeps_the_marker_baseline() {
@@ -1024,6 +1050,7 @@ assert_the_release_image_revision_label_supplies_the_missing_baseline
assert_no_marker_and_no_revision_label_fails_closed
assert_an_unreachable_docker_daemon_cannot_become_a_baseline
assert_a_re_acquisition_invalidates_the_marker_baseline_in_ghcr_mode
+assert_a_re_acquisition_invalidates_the_marker_baseline_in_local_mode
assert_an_untouched_image_set_keeps_the_marker_baseline
assert_the_pre_gate_adopts_the_recorded_cluster_mode_over_the_environment
assert_a_recorded_local_build_renders_the_local_overlay
diff --git a/scripts/tests/test-minikube-pull-images.sh b/scripts/tests/test-minikube-pull-images.sh
index 7f23f2281..632d69172 100755
--- a/scripts/tests/test-minikube-pull-images.sh
+++ b/scripts/tests/test-minikube-pull-images.sh
@@ -79,6 +79,9 @@ STUB
#!/usr/bin/env bash
printf 'minikube %s\n' "$*" >>"${TEST_LOG_FILE:?}"
if [[ "$*" == *"docker-env"* ]]; then
+ if [ "${TEST_EMPTY_DOCKER_ENV:-false}" = true ]; then
+ exit 0
+ fi
echo 'export DOCKER_HOST="tcp://127.0.0.1:2376"'
exit 0
fi
@@ -144,9 +147,11 @@ run_puller_prepared() {
TEST_FLAKY_TAGS="${TEST_FLAKY_TAGS:-}" \
TEST_FLAKY_FAIL_COUNT="${TEST_FLAKY_FAIL_COUNT:-}" \
TEST_FLAKY_COUNTER_DIR="${TEST_FLAKY_COUNTER_DIR:-}" \
+ TEST_EMPTY_DOCKER_ENV="${TEST_EMPTY_DOCKER_ENV:-}" \
MINIKUBE_IMAGE_TAG="${MINIKUBE_IMAGE_TAG:-}" \
MINIKUBE_IMAGE_PULL_RETRIES="${MINIKUBE_IMAGE_PULL_RETRIES:-}" \
MINIKUBE_IMAGE_PULL_DELAY_SECS="${MINIKUBE_IMAGE_PULL_DELAY_SECS:-}" \
+ MINIKUBE_PULL_PARALLELISM="${MINIKUBE_PULL_PARALLELISM:-}" \
TEST_HANG_PULL="${TEST_HANG_PULL:-}" \
MINIKUBE_DOCKER_PULL_TIMEOUT_SECONDS="${MINIKUBE_DOCKER_PULL_TIMEOUT_SECONDS:-}" \
MINIKUBE_DOCKER_BUILD_TIMEOUT_SECONDS="${MINIKUBE_DOCKER_BUILD_TIMEOUT_SECONDS:-}" \
@@ -200,6 +205,46 @@ assert_hung_pull_hits_the_finite_deadline() {
rm -rf "$d"
}
+assert_empty_docker_env_fails_closed() {
+ local d out rc
+ d="$(mktemp -d)"
+ export TEST_EMPTY_DOCKER_ENV=true
+ out="$(run_puller "$d" --only=control-api)"; rc=$?
+ unset TEST_EMPTY_DOCKER_ENV
+ if [ "$rc" -ne 0 ] \
+ && grep -Fq 'DOCKER_ENV_UNRESOLVED' <<<"$out" \
+ && ! grep -q '^pull ' "$d/ops.log"; then
+ pass "empty successful minikube docker-env output fails closed"
+ else
+ fail "empty docker-env output returned success or reached Docker: rc=$rc out=$out"
+ fi
+ rm -rf "$d"
+}
+
+assert_invalid_pull_knobs_fail_closed() {
+ local d out rc
+ d="$(mktemp -d)"
+
+ export MINIKUBE_PULL_PARALLELISM=0
+ out="$(run_puller "$d" --only=control-api)"; rc=$?
+ unset MINIKUBE_PULL_PARALLELISM
+ if [ "$rc" -ne 2 ] || ! grep -Fq 'MINIKUBE_PULL_PARALLELISM must be an integer' <<<"$out"; then
+ fail "parallelism=0 did not fail with the parameter error: rc=$rc out=$out"
+ rm -rf "$d"
+ return
+ fi
+
+ export MINIKUBE_IMAGE_PULL_DELAY_SECS=301
+ out="$(run_puller "$d" --only=control-api)"; rc=$?
+ unset MINIKUBE_IMAGE_PULL_DELAY_SECS
+ if [ "$rc" -eq 2 ] && grep -Fq 'MINIKUBE_IMAGE_PULL_DELAY_SECS must be an integer' <<<"$out"; then
+ pass "parallelism, retry, and delay knobs fail closed outside their finite ranges"
+ else
+ fail "delay=301 did not fail with the parameter error: rc=$rc out=$out"
+ fi
+ rm -rf "$d"
+}
+
assert_it_pulls_every_pull_in_ghcr_mode_image() {
local d out rc want got
d="$(mktemp -d)"
@@ -755,6 +800,8 @@ assert_every_defined_case_is_invoked() {
assert_it_pulls_every_pull_in_ghcr_mode_image
assert_puller_requires_the_inherited_profile_lease
assert_hung_pull_hits_the_finite_deadline
+assert_empty_docker_env_fails_closed
+assert_invalid_pull_knobs_fail_closed
assert_it_never_pulls_an_unpublished_image
assert_it_never_pulls_a_registry_distributed_mcp_server
assert_it_repulls_a_tag_already_present_in_the_daemon
diff --git a/scripts/tests/test-minikube-t2-contract.sh b/scripts/tests/test-minikube-t2-contract.sh
index 6f62da4ba..7c0656a49 100755
--- a/scripts/tests/test-minikube-t2-contract.sh
+++ b/scripts/tests/test-minikube-t2-contract.sh
@@ -149,6 +149,19 @@ grep -Fq 't2-setup-handoff.sh" consume --' "$ROOT/scripts/minikube/pre-gate-sync
grep -Fq 'minikube-verify-images' "$ROOT/scripts/minikube/pre-gate-sync.sh"
grep -Fq 'with-t2-mutation-lock.sh' "$ROOT/scripts/minikube/setup.sh"
grep -Fq 'T2_MUTATION_LOCK_WRAPPED' "$ROOT/scripts/minikube/setup.sh"
+grep -Fq 'require-t2-mutation-lock.sh' "$ROOT/scripts/minikube/setup.sh"
+grep -Fq 'PROFILE_LOCK_REQUIRED: setup.sh requires matching' "$ROOT/scripts/minikube/setup.sh"
+grep -Fq 'PROFILE="${MINIKUBE_PROFILE:-${T2_PROFILE:-}}"' "$ROOT/scripts/minikube/setup.sh"
+if grep -Fq 'PROFILE="clerum-test"' "$ROOT/scripts/minikube/setup.sh"; then
+ echo 'FAIL: legacy setup still defaults to the shared clerum-test profile' >&2
+ exit 1
+fi
+setup_guard_line="$(grep -nF 'if [ "${T2_MUTATION_LOCK_WRAPPED:-false}" != true ]; then' "$ROOT/scripts/minikube/setup.sh" | cut -d: -f1)"
+setup_cluster_line="$(grep -nF 'cluster-info' "$ROOT/scripts/minikube/setup.sh" | head -1 | cut -d: -f1)"
+if [[ -z "$setup_guard_line" || -z "$setup_cluster_line" || "$setup_guard_line" -ge "$setup_cluster_line" ]]; then
+ echo 'FAIL: legacy setup can mutate or probe the cluster before the lease wrapper' >&2
+ exit 1
+fi
for timed_phase in 't2_evidence_write planner PASS' 't2_evidence_write transition PASS' \
't2_evidence_write pre-gate-sync PASS' 't2_evidence_write T1 PASS' \
't2_evidence_write T2 PASS' 't2_evidence_write NP08_HCC_AUTHORIZATION PASS'; do