Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
fix(minikube): close leased image and recovery harness gaps
Alfredo Lopezad122691a41c
diff --git a/.cursor/skills/minikube-t0-t1-t2/SKILL.md b/.cursor/skills/minikube-t0-t1-t2/SKILL.mdindex d8346d6c6..6d2047815 100644--- a/.cursor/skills/minikube-t0-t1-t2/SKILL.md+++ b/.cursor/skills/minikube-t0-t1-t2/SKILL.md@@ -24,7 +24,9 @@ Before running anything, verify ALL of these:Do NOT create a new profile because HEAD, gate, or command changed.Resolve it with the primary checkout `.local-notes/minikube-profiles/branch.mk`;profile identity is stable for canonical worktree + branch, while the- pre-gate marker—not a profile-name SHA—proves exact-HEAD freshness.+ pre-gate marker—not a profile-name SHA—proves exact-HEAD freshness. The+ marker must also match the image manifest's exact `imagesGeneratedAt`+ stamp; a new image acquisition on the same HEAD is not already-synced.- [ ] Confirm the profile is not owned by another active branch/worktree. Ifownership is ambiguous, stop and use a dedicated profile instead.- [ ] Every `kubectl` you run manually uses `--context=<owned profile>`.@@ -32,9 +34,9 @@ Before running anything, verify ALL of these:- [ ] Docker resolves to a local Unix socket or loopback TCP endpoint. Theharness pins that endpoint into an empty task-local config; do not use aremote Docker context or copy ambient registry credentials into it.-- [ ] Mutating image builds use the public Make target/orchestrator and inherit- its exact profile lease. Do not call `build-images.sh` directly;- `--verify-only` is the read-only exception.+- [ ] Mutating image acquisition/builds use the public Make target/orchestrator+ and inherit its exact profile lease. Do not call `build-images.sh` or+ `pull-images.sh` directly; `--verify-only` is the read-only exception.- [ ] Never read `~/.cache/clerum/minikube-profiles/` directly (HARD DENY —it holds private profile state). The harness reads it for you.@@ -150,6 +152,12 @@ it is optional for other transitions. `T2_PLAYWRIGHT_COMMAND` remains opt-inEvidence stays under the ignored `.local-notes/infra/runs/`. Never commit it.Before committing anything else, run `make minikube-t2-public-boundary`.+The final exact-head check also rejects a marker whose `imagesGeneratedAt`+does not equal the current image manifest. During REUSE_DB recovery, the+durable fence covers HCC, workflow-recipes, trace-maintenance-worker, and+control-api; a changed HEAD on the same owned lane is historical state, not a+reason to discard recovery evidence.+## Step 5 — On failureFailures print a stable code and a next safe command. Repair the FIRSTdiff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.mdindex 378eb3086..6facbeecf 100644--- a/.cursor/skills/minikube-t0-t1-t2/reference.md+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md@@ -25,9 +25,10 @@ Companion to `SKILL.md`. Source of truth: `scripts/minikube/t2.sh`,fixed localhost ports** — only the profile-owned random port mapping.- **Do not touch another worktree's Minikube profile** (e.g. an NP-08 laneprofile) or kill its port-forwards, even if it looks idle.-- **Do not run a mutating `build-images.sh` directly.** Use the public Make- target or T2 orchestrator so the exact worktree/profile/context lease is- inherited. Only `--verify-only` is lease-exempt.+- **Do not run mutating `build-images.sh` or `pull-images.sh` directly.** Use+ the public Make target or T2 orchestrator so the exact+ worktree/profile/context lease is inherited. Only `--verify-only` is+ lease-exempt.- **Do not use a remote/ambiguous Docker endpoint or inherit ambient registryauth.** The harness accepts local Unix sockets and loopback TCP endpoints,pins the resolved endpoint before isolation, and requires an explicit@@ -39,6 +40,9 @@ Companion to `SKILL.md`. Source of truth: `scripts/minikube/t2.sh`,pidfile trips `PORT_FORWARD_CONFLICT`. The awk pre-filter matches `kubectl`as an argv0 or path token plus a later standalone `port-forward` token;wrappers that only mention those words are rejected by `comm`.+ Registered pidfiles are checked even when `ps` no longer lists a child,+ and a successful user-facing health probe is followed by an exact+ process/start-time/argv revalidation.- **Do not commit** `.local-notes/`, lockfiles produced by an incidental`npm ci`, generated ports/profile metadata, or anything secret-like. Run`make minikube-t2-public-boundary` first.@@ -65,7 +69,7 @@ Companion to `SKILL.md`. Source of truth: `scripts/minikube/t2.sh`,| `DOCKER_DEADLINE_INVALID` | A runtime deadline/retry value is malformed or exceeds its finite maximum. | Remove the override or set it within the documented range; do not disable the deadline. || `BOOTSTRAP_REQUIRED` | Profile missing/uninitialized, or the planner produced no transition. Standalone preflight refuses to bootstrap. | Run `MINIKUBE_PROFILE=<owned> CONTROL_API_REAL_PG_CONTEXT=<owned> make minikube-t2`; its internal planner makes `full-bootstrap` reachable. || `HEAD_MARKER_MISMATCH` | Pre-gate marker does not match this worktree/HEAD; the final T2 preflight selected something other than `already-synced`. | Re-run the full target on this HEAD so `pre-gate-sync` updates the marker. Never hand-edit the marker. |-| `IMAGE_MANIFEST_MISMATCH` | Deployed image provenance does not match the exact marker/worktree state. | Re-run the full target for the same owned profile; do not relabel or hand-edit the manifest. |+| `IMAGE_MANIFEST_MISMATCH` | Deployed image provenance or the manifest `generated` stamp does not match the exact marker/worktree state. | Re-run the full target for the same owned profile; do not relabel or hand-edit the manifest. || `PORT_FORWARD_CONFLICT` | A port-forward for this profile is owned by a process not recorded for it (or not a real `kubectl`). | Identify the foreign owner; if it belongs to another lane, stop. Restart forwards via `scripts/minikube/pf-all-stack.sh` for this profile only. || `PROFILE_BUSY` | Profile lock held. Live owner PID → genuinely busy. No valid owner PID → orphaned lock. | Live owner: wait or coordinate; never remove. Orphan: verify no T2 process owns the profile, then remove ONLY `$T2_LOCK_ROOT/<profile>.lock` and retry. Never remove the lock root. || `DEVELOPMENT_SCOPE_REQUIRED` | Preflight/final preflight failed a precondition, or T2-only mode was attempted without `already-synced`. | Repair the first reported condition; if T2-only was refused, run full `make minikube-t2`. |diff --git a/AGENTS.md b/AGENTS.mdindex 99ac28864..8036b2be2 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -20,8 +20,8 @@ Before reusing a profile, record and compare all of the following:- active worktree path, branch, `HEAD`, and `origin/dev`;- target Minikube profile and explicit Kubernetes context;-- the pre-gate state marker: `worktreeId`, `gitHead`, and- `clusterFingerprint`;+- the pre-gate state marker: `worktreeId`, `gitHead`, `clusterFingerprint`,+ and the exact `imagesGeneratedAt` value from the image manifest;- profile status and currently running profile, pre-gate-sync, andport-forward processes.@@ -36,12 +36,17 @@ Port-forwards are owned by atomic `0600` records bound to the exact profile,context, canonical worktree, namespace, Service, local/remote ports, PID,process start time, and `kubectl` argv. Never kill or adopt a live legacy`/tmp/pf-*.pid` process: it lacks enough provenance and must fail closed.--All image builds and targeted deploys must enter through the documented Make-targets or the T2 orchestrator so they inherit the exact live profile mutation-lease. Calling `scripts/minikube/build-images.sh` directly is unsupported for-mutating work; `--verify-only` is the read-only exception. Docker and Minikube-operations must use finite deadlines. Docker endpoint discovery must resolve+After a user-facing health probe, revalidate the same live process, start time,+argv, and binding. A registered but dead/reused/ambiguous pidfile is itself a+`PORT_FORWARD_CONFLICT`; it must not disappear from the T2 verdict merely+because `ps` no longer lists a child.++All image acquisition, image builds, and targeted deploys must enter through+the documented Make targets or the T2 orchestrator so they inherit the exact+live profile mutation lease. Calling `scripts/minikube/build-images.sh` or+`scripts/minikube/pull-images.sh` directly is unsupported for mutating work;+`--verify-only` is the read-only exception. Docker and Minikube operations must+use finite deadlines. Docker endpoint discovery must resolveto an explicit local Unix socket or loopback TCP endpoint before switching toan empty task-local Docker config. Ambient registry credentials are nevercopied; a private pull requires an explicit `MINIKUBE_DOCKER_AUTH_CONFIG`.@@ -130,6 +135,10 @@ reader readiness through the `gfs-rollout-shim` PATH prefix instead of ageneration-based `rollout status`, because HCC's gfsReconciler strips the`restartedAt` annotation and makes that wait time out. The standalone preflight andthe final exact-head T2 check stay fail-loud on an unready deployment.+When REUSE_DB recovery is needed, the fence covers all four database writers:+HCC, workflow-recipes, trace-maintenance-worker, and control-api. The durable+recovery identity is the owned profile/context/worktree/branch; the interrupted+HEAD is retained as audit data while the exact-head marker controls freshness.Real PostgreSQL suites are opt-in in the ordinary testmatrix, but a T1 run that requires them must fail when the database/DSN isunavailable or when zero tests execute; a green run must never be produced bydiff --git a/Makefile b/Makefileindex cd25bc0c8..47f48360e 100644--- a/Makefile+++ b/Makefile@@ -201,9 +201,20 @@ minikube-setup-e2e: ## Full setup + E2E fixtures (test user, e2e-* recipes, demominikube-teardown: ## Remove deployments (keep namespaces/CRDs)@scripts/minikube/teardown.sh-.PHONY: minikube-pull-images+.PHONY: minikube-pull-images minikube-pull-images-bodyminikube-pull-images: ## Pull ALL published images into minikube at the pinned release tag (MINIKUBE_IMAGE_TAG=<tag> overrides the pin for this run only)- @MINIKUBE_PROFILE="$(MINIKUBE_PROFILE)" MINIKUBE_IMAGE_TAG="$(MINIKUBE_IMAGE_TAG)" \+ @T2_PROJECT_DIR="$(CURDIR)" T2_PROFILE="$(MINIKUBE_PROFILE)" T2_CONTEXT="$(MINIKUBE_PROFILE)" \+ T2_SKIP_LOCK="$(T2_SKIP_LOCK)" T2_LOCK_TOKEN="$(T2_LOCK_TOKEN)" \+ bash scripts/minikube/with-t2-mutation-lock.sh -- \+ $(MAKE) --no-print-directory minikube-pull-images-body++minikube-pull-images-body:+ @T2_PROJECT_DIR="$(CURDIR)" T2_PROFILE="$(MINIKUBE_PROFILE)" T2_CONTEXT="$(MINIKUBE_PROFILE)" \+ T2_SKIP_LOCK=true T2_LOCK_TOKEN="$(T2_LOCK_TOKEN)" \+ bash scripts/minikube/require-t2-mutation-lock.sh+ @T2_PROJECT_DIR="$(CURDIR)" T2_PROFILE="$(MINIKUBE_PROFILE)" T2_CONTEXT="$(MINIKUBE_PROFILE)" \+ MINIKUBE_PROFILE="$(MINIKUBE_PROFILE)" MINIKUBE_IMAGE_TAG="$(MINIKUBE_IMAGE_TAG)" \+ CONTROL_API_REAL_PG_CONTEXT="$(MINIKUBE_PROFILE)" \scripts/minikube/pull-images.sh.PHONY: minikube-build-images minikube-build-images-bodydiff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.mdindex a9408c675..bad0b5504 100644--- a/docs/testing/minikube-t2-runbook.md+++ b/docs/testing/minikube-t2-runbook.md@@ -105,12 +105,14 @@ inside that exact profile lock. A concurrent loser fails closed and neverremoves either the stale directory or the winner's replacement lock.The pre-gate marker must contain the current worktree identifier, exact `HEAD`,-cluster fingerprint, and image coordinate. A mismatch stops with a stable-error code instead of allowing a mixed-commit run.--Mutating image builds and targeted deploys are children of that same exact-profile lease. Public Make targets acquire it; private body targets and-`build-images.sh` validate the inherited token again before the first Docker,+cluster fingerprint, image coordinate, and the exact `imagesGeneratedAt` value+from the image manifest. A mismatch—including a new image acquisition at the+same HEAD—stops with a stable error code instead of allowing a mixed-commit run.++Mutating image acquisition/builds and targeted deploys are children of that+same exact profile lease. Public Make targets acquire it; private body targets,+`pull-images.sh`, and `build-images.sh` validate the inherited token again+before the first Docker,Minikube, or Kubernetes operation. `build-images.sh --verify-only` is theread-only exception. Empty/unknown selectors fail before the lease or runtimeis touched.@@ -142,7 +144,9 @@ context, canonical worktree, Service, and port bindings. Each live processmust have exactly one atomic `0600` ownership record under`$HOME/.cache/clerum/minikube-profiles/<profile>/pids/`. A live legacy`/tmp/pf-<profile>-*.pid` record cannot be adopted or killed because it lacks-the full binding; dead legacy records may be pruned.+the full binding; dead legacy records may be pruned. Registered pidfiles are+also checked when `ps` no longer lists a child, and a successful user-facing+health probe is followed by exact process/start-time/argv revalidation.`make minikube-t2` invokes `pre-gate-sync` with `--skip-port-forwards` so theorchestrator does not plant forwards that fail its own T2 check. Secret valuesare never printed.@@ -302,6 +306,12 @@ refused. An interrupted bootstrap leaves the profile intact and can be safelyretried after the reported prerequisite is repaired. Do not reuse evidencefrom a different `HEAD` or profile.+REUSE_DB recovery fences all four database writers—HCC, workflow-recipes,+trace-maintenance-worker, and control-api—before migrations, role provisioning,+or the recovery overlay. The interrupted `HEAD` remains historical state for+the owned profile/context/worktree/branch; exact-head freshness is decided by+the pre-gate marker and image stamp.+Retry by phase. During a T1 failure, iterate with`minikube-t2-real-postgres`, then run one full `minikube-t2` certification oncegreen. After exact-head T0/T1 lane evidence is already `PASS`, a failure indiff --git a/scripts/minikube/full-setup.sh b/scripts/minikube/full-setup.shindex 34b56f2f0..0e140367c 100755--- a/scripts/minikube/full-setup.sh+++ b/scripts/minikube/full-setup.sh@@ -1118,9 +1118,11 @@ PARTIAL_BOOTSTRAP_RECOVERY=falsePARTIAL_CONTROL_API_REPLICAS=""PARTIAL_WORKFLOW_REPLICAS=""PARTIAL_HCC_REPLICAS=""+PARTIAL_TRACE_REPLICAS=""PARTIAL_CONTROL_API_FENCED=falsePARTIAL_WORKFLOW_FENCED=falsePARTIAL_HCC_FENCED=false+PARTIAL_TRACE_FENCED=falseWRITER_RECOVERY_STATE_HELPER="${SCRIPT_DIR}/writer-recovery-state.py"WRITER_RECOVERY_STATE_ROOT="${PROJECT_DIR}/.local-notes/infra/runs/writer-recovery"WRITER_RECOVERY_STATE_FILE="${WRITER_RECOVERY_STATE_ROOT}/${PROFILE}.json"@@ -1150,6 +1152,7 @@ writer_recovery_state_write() {--phase "$phase" \--hcc "$PARTIAL_HCC_REPLICAS" \--workflow "$PARTIAL_WORKFLOW_REPLICAS" \+ --trace "$PARTIAL_TRACE_REPLICAS" \--control-api "$PARTIAL_CONTROL_API_REPLICAS"}@@ -1177,10 +1180,12 @@ writer_recovery_state_load() {return 0fiIFS='|' read -r WRITER_RECOVERY_STATE_PHASE PARTIAL_HCC_REPLICAS \- PARTIAL_WORKFLOW_REPLICAS PARTIAL_CONTROL_API_REPLICAS <<<"$state_output"+ PARTIAL_WORKFLOW_REPLICAS PARTIAL_TRACE_REPLICAS \+ PARTIAL_CONTROL_API_REPLICAS <<<"$state_output"[ -n "$WRITER_RECOVERY_STATE_PHASE" ] &&[[ "$PARTIAL_HCC_REPLICAS" =~ ^[0-9]+$ ]] &&[[ "$PARTIAL_WORKFLOW_REPLICAS" =~ ^[0-9]+$ ]] &&+ [[ "$PARTIAL_TRACE_REPLICAS" =~ ^[0-9]+$ ]] &&[[ "$PARTIAL_CONTROL_API_REPLICAS" =~ ^[1-9][0-9]*$ ]] || {err "Durable writer-recovery state is incomplete; refusing profile mutation"return 1@@ -1204,11 +1209,20 @@ writer_recovery_state_load() {PARTIAL_WORKFLOW_FENCED=trueWRITER_RECOVERY_FENCE_PENDING=true;;+ trace-fencing|trace-fenced)+ WRITER_RECOVERY=true+ PARTIAL_BOOTSTRAP_RECOVERY=true+ PARTIAL_HCC_FENCED=true+ PARTIAL_WORKFLOW_FENCED=true+ PARTIAL_TRACE_FENCED=true+ WRITER_RECOVERY_FENCE_PENDING=true+ ;;api-fencing)WRITER_RECOVERY=truePARTIAL_BOOTSTRAP_RECOVERY=truePARTIAL_HCC_FENCED=truePARTIAL_WORKFLOW_FENCED=true+ PARTIAL_TRACE_FENCED=truePARTIAL_CONTROL_API_FENCED=trueWRITER_RECOVERY_FENCE_PENDING=true;;@@ -1217,6 +1231,7 @@ writer_recovery_state_load() {PARTIAL_BOOTSTRAP_RECOVERY=truePARTIAL_HCC_FENCED=truePARTIAL_WORKFLOW_FENCED=true+ PARTIAL_TRACE_FENCED=truePARTIAL_CONTROL_API_FENCED=trueWRITER_RECOVERY_FENCE_PENDING=true;;@@ -1225,6 +1240,7 @@ writer_recovery_state_load() {PARTIAL_BOOTSTRAP_RECOVERY=truePARTIAL_HCC_FENCED=truePARTIAL_WORKFLOW_FENCED=true+ PARTIAL_TRACE_FENCED=truePARTIAL_CONTROL_API_FENCED=true# A persisted policy phase is historical evidence only. Re-fence and# restage the current rendered policy before any writer is restored so a@@ -1236,6 +1252,7 @@ writer_recovery_state_load() {PARTIAL_BOOTSTRAP_RECOVERY=truePARTIAL_HCC_FENCED=truePARTIAL_WORKFLOW_FENCED=true+ PARTIAL_TRACE_FENCED=truePARTIAL_CONTROL_API_FENCED=trueWRITER_RECOVERY_MIGRATIONS_COMPLETE=true# The phase is written before the control-api restore. An interruption@@ -1249,7 +1266,7 @@ writer_recovery_state_load() {PARTIAL_WORKFLOW_FENCED=truePARTIAL_CONTROL_API_FENCED=trueWRITER_RECOVERY_MIGRATIONS_COMPLETE=true- # The overlay is rendered with all three writer Deployments at zero.+ # The overlay is rendered with all four database-writer Deployments at zero.# Re-fence on every post-migration resume because an interrupted apply# may have recreated a writer before its durable phase was updated.WRITER_RECOVERY_FENCE_PENDING=true@@ -1273,11 +1290,15 @@ writer_recovery_state_prepare() {state="$($KC -n control-plane get deployment/workflow-recipes \-o 'jsonpath={.spec.replicas}')" || return 1PARTIAL_WORKFLOW_REPLICAS="$state"+ state="$($KC -n control-plane get deployment/trace-maintenance-worker \+ -o 'jsonpath={.spec.replicas}')" || return 1+ PARTIAL_TRACE_REPLICAS="$state"state="$($KC -n control-plane get deployment/control-api \-o 'jsonpath={.spec.replicas}')" || return 1PARTIAL_CONTROL_API_REPLICAS="$state"[[ "$PARTIAL_HCC_REPLICAS" =~ ^[0-9]+$ ]] || return 1[[ "$PARTIAL_WORKFLOW_REPLICAS" =~ ^[0-9]+$ ]] || return 1+ [[ "$PARTIAL_TRACE_REPLICAS" =~ ^[0-9]+$ ]] || return 1[[ "$PARTIAL_CONTROL_API_REPLICAS" =~ ^[1-9][0-9]*$ ]] || return 1writer_recovery_state_phase planned}@@ -1350,6 +1371,25 @@ fence_partial_workflow_reconciler() {writer_recovery_state_phase workflow-fenced}+fence_partial_trace_worker() {+ local pods+ if [[ ! "${PARTIAL_TRACE_REPLICAS}" =~ ^[0-9]+$ ]]; then+ err "trace-maintenance-worker replica count is invalid; refusing partial-bootstrap recovery"+ return 1+ fi+ log "Fencing trace-maintenance-worker at ${PARTIAL_TRACE_REPLICAS} replica(s)"+ writer_recovery_state_phase trace-fencing+ $KC -n control-plane scale deployment/trace-maintenance-worker --replicas="${PARTIAL_TRACE_REPLICAS}" >/dev/null+ pods="$($KC -n control-plane get pods -l app=trace-maintenance-worker -o name)" \+ || return 1+ if [ -n "${pods}" ]; then+ $KC -n control-plane wait --for=delete pod \+ -l app=trace-maintenance-worker --timeout=180s >/dev/null+ fi+ PARTIAL_TRACE_FENCED=true+ writer_recovery_state_phase trace-fenced+}+fence_partial_hcc() {local podsif [[ ! "$PARTIAL_HCC_REPLICAS" =~ ^[0-9]+$ ]]; then@@ -1376,6 +1416,7 @@ fence_partial_bootstrap_writers() {writer_recovery_state_preparefence_partial_hccfence_partial_workflow_reconciler+ fence_partial_trace_workerfence_partial_control_api}@@ -1501,12 +1542,17 @@ restore_partial_control_api() {restore_partial_non_api_writers() {if [ "$PARTIAL_WORKFLOW_FENCED" = true ]; then$KC -n control-plane scale deployment/workflow-recipes \- --replicas="$PARTIAL_WORKFLOW_REPLICAS" >/dev/null+ --replicas="$PARTIAL_WORKFLOW_REPLICAS" >/dev/null || return 1PARTIAL_WORKFLOW_FENCED=falsefi+ if [ "$PARTIAL_TRACE_FENCED" = true ]; then+ $KC -n control-plane scale deployment/trace-maintenance-worker \+ --replicas="$PARTIAL_TRACE_REPLICAS" >/dev/null || return 1+ PARTIAL_TRACE_FENCED=false+ fiif [ "$PARTIAL_HCC_FENCED" = true ]; then$KC -n control-plane scale deployment/host-context-controller \- --replicas="$PARTIAL_HCC_REPLICAS" >/dev/null+ --replicas="$PARTIAL_HCC_REPLICAS" >/dev/null || return 1PARTIAL_HCC_FENCED=falsefi}@@ -1539,11 +1585,12 @@ render_fenced_recovery_overlay() {err "Recovery overlay workspace is unavailable; refusing writer recovery"return 1}- log "Rendering the recovery overlay with HCC, workflow-recipes, and control-api fenced"+ log "Rendering the recovery overlay with HCC, workflow-recipes, trace-maintenance-worker, and control-api fenced"if ! $KC kustomize "$ACTIVE_MINIKUBE_RENDER_DIR" | \RUBYOPT=--disable=gems ruby "${SCRIPT_DIR}/render-fenced-writer-deployments.rb" \--target control-plane/host-context-controller \--target control-plane/workflow-recipes \+ --target control-plane/trace-maintenance-worker \--target control-plane/control-api >"$rendered_file"; thenerr "Unable to render a complete fenced recovery overlay; refusing writer recovery"return 1@@ -1591,13 +1638,14 @@ elseWRITER_RECOVERY=truePARTIAL_BOOTSTRAP_RECOVERY=truelog "Existing GFS writer and empty control-api runtime Secret detected — fencing writers until migrations and runtime roles converge"- elif [ "$runtime_secret_status" -eq 1 ] && [ "$K8S_API_EGRESS_POLICY_DRIFT" = true ]; then+ elif [ "$runtime_secret_status" -eq 1 ]; thenWRITER_RECOVERY=truePARTIAL_BOOTSTRAP_RECOVERY=true- log "Existing GFS writer detected with control-api not Ready — repairing the stale API-egress policy before the fail-closed readiness decision"- elif [ "$runtime_secret_status" -eq 1 ]; then- err "Existing GFS writer detected but control-api is not Ready; refusing HCC cutover"- exit 1+ if [ "$K8S_API_EGRESS_POLICY_DRIFT" = true ]; then+ log "Existing GFS writer detected with control-api not Ready — repairing the stale API-egress policy before runtime-role convergence"+ else+ log "Existing GFS writer detected with control-api not Ready — fencing all database writers before runtime-role convergence"+ fielseerr "Unable to inspect control-api-postgres-runtime; refusing to classify the GFS upgrade path"exit 1@@ -1679,7 +1727,7 @@ elsefiif [ "$WRITER_RECOVERY" = true ]; then- # Keep every writer at zero while the full overlay is applied. The overlay+ # Keep every database writer at zero while the full overlay is applied. The overlay# itself declares one replica for these Deployments, so applying it# directly would reopen the writer window before the explicit restore.apply_fenced_recovery_overlaydiff --git a/scripts/minikube/pf-all-stack.sh b/scripts/minikube/pf-all-stack.shindex 74851d923..adf77037e 100755--- a/scripts/minikube/pf-all-stack.sh+++ b/scripts/minikube/pf-all-stack.sh@@ -261,6 +261,15 @@ start_pf() {for ((attempt = 0; attempt < HEALTH_ATTEMPTS; attempt += 1)); doif curl -sf -m 2 "${health_url}" >/dev/null 2>&1; then+ # Health can succeed in the same small interval in which a forward+ # exits or its PID is reused. Revalidate the exact recorded process and+ # binding after the user-facing probe before publishing success.+ if ! pf_owner_record_process_matches "${profile_pidfile}" "${PROFILE}" \+ "${CONTEXT}" "${WORKTREE_ROOT}" "${namespace}" "${service}" \+ "${local_port}" "${remote_port}"; then+ echo " ERROR: ${name} port-forward ownership was lost during health verification" >&2+ return 1+ fireturn 0fipf_owner_pause "${HEALTH_DELAY}"diff --git a/scripts/minikube/port-forward-owner.sh b/scripts/minikube/port-forward-owner.shindex ba51cf92b..0a82716c8 100755--- a/scripts/minikube/port-forward-owner.sh+++ b/scripts/minikube/port-forward-owner.sh@@ -268,6 +268,26 @@ pf_owner_record_matches() {"${PF_OWNER_RECORD_REMOTE_PORT}" == "${remote_port}" ]]}+pf_owner_record_process_matches() {+ local pidfile="$1" profile="$2" context="$3" worktree="$4"+ local namespace="$5" service="$6" local_port="$7" remote_port="$8"+ local pid state actual_start command_line++ pf_owner_read_record "$pidfile" || return 1+ pf_owner_record_matches "${profile}" "${context}" "${worktree}" \+ "${namespace}" "${service}" "${local_port}" "${remote_port}" || return 1+ pid="${PF_OWNER_RECORD_PID}"+ state="$(pf_owner_process_state "${pid}")"+ [[ "${state}" == live ]] || return 1+ actual_start="$(pf_owner_process_start "${pid}")" || return 1+ [[ "${actual_start}" == "${PF_OWNER_RECORD_START}" ]] || return 1+ command_line="$(pf_owner_process_command "${pid}")" || return 1+ pf_owner_command_matches "${command_line}" "${context}" "${namespace}" \+ "${service}" "${local_port}" "${remote_port}" || return 1+ actual_start="$(pf_owner_process_start "${pid}")" || return 1+ [[ "${actual_start}" == "${PF_OWNER_RECORD_START}" ]] || return 1+}+pf_owner_remove_dead_record() {local pidfile="$1"[[ ! -L "${pidfile}" ]] || pf_owner_error "refusing to remove symlinked pidfile: ${pidfile}" || return 1diff --git a/scripts/minikube/pull-images.sh b/scripts/minikube/pull-images.shindex f4856e5b5..afa99b748 100755--- a/scripts/minikube/pull-images.sh+++ b/scripts/minikube/pull-images.sh@@ -75,6 +75,7 @@ MINIKUBE_MULTI_NODE="${MINIKUBE_MULTI_NODE:-false}"PULL_PARALLELISM="${MINIKUBE_PULL_PARALLELISM:-6}"MINIKUBE_IMAGE_PULL_RETRIES="${MINIKUBE_IMAGE_PULL_RETRIES:-3}"MINIKUBE_IMAGE_PULL_DELAY_SECS="${MINIKUBE_IMAGE_PULL_DELAY_SECS:-5}"+MINIKUBE_DOCKER_ENV_TIMEOUT_SECONDS="${MINIKUBE_DOCKER_ENV_TIMEOUT_SECONDS:-30}"GHCR_NAMESPACE="ghcr.io/evenfire-ai"ONLY_SVC=""@@ -121,6 +122,22 @@ err() { echo -e "${RED} ERROR${NC} -- $*" >&2; }# record says "local" must still resolve a tag instead of an empty string.# shellcheck source=scripts/minikube/image-mode.shsource "${SCRIPT_DIR}/image-mode.sh"+# shellcheck source=scripts/minikube/docker-cli-env.sh+source "${SCRIPT_DIR}/docker-cli-env.sh"++# Pulling and loading images mutates the branch-owned Minikube profile. The+# caller must already hold the canonical T2 mutation lease; this child only+# validates the opaque inherited token and never acquires or releases it.+if [[ -z "${T2_PROJECT_DIR:-}" || -z "${T2_PROFILE:-}" ||+ -z "${T2_CONTEXT:-}" || "${T2_PROFILE}" != "${T2_CONTEXT}" ||+ "${T2_PROFILE}" != "${PROFILE}" ]]; then+ err "PROFILE_LOCK_REQUIRED: pull-images.sh requires an inherited branch-profile mutation lease"+ exit 1+fi+T2_PROJECT_DIR="${T2_PROJECT_DIR}" T2_PROFILE="${T2_PROFILE}" \+ T2_CONTEXT="${T2_CONTEXT}" T2_SKIP_LOCK=true \+ MINIKUBE_PROFILE="${PROFILE}" CONTROL_API_REAL_PG_CONTEXT="${T2_CONTEXT}" \+ bash "${SCRIPT_DIR}/require-t2-mutation-lock.sh"IMAGE_TAG="$(image_mode_ghcr_tag "$PROJECT_DIR")" || exit 1TAG_ORIGIN="$(image_mode_tag_origin "$PROJECT_DIR")" || exit 1@@ -155,19 +172,42 @@ if [ "$SKIP_UIS" = true ]; thenlog "skipping control-ui and profile-ui (--skip-uis); the -no-uis overlay deletes both Deployments"fi-# ---- Point the Docker CLI at minikube's daemon when safe -----------------+STATUS_DIR="$(mktemp -d)"+cleanup() {+ local status=$? cleanup_status=0+ trap - EXIT INT TERM+ rm -rf -- "$STATUS_DIR" || cleanup_status=$?+ docker_cli_env_cleanup || cleanup_status=$?+ if [ "$status" -eq 0 ] && [ "$cleanup_status" -ne 0 ]; then+ status="$cleanup_status"+ fi+ exit "$status"+}+trap cleanup EXIT+trap 'exit 130' INT+trap 'exit 143' TERM++# Resolve an approved local endpoint into an empty task-local Docker config+# before asking Minikube for its daemon endpoint. Every subsequent Docker and+# image-load operation then runs through the same isolated config and deadline+# runner, with ambient auth/context/header variables excluded.+docker_cli_env_prepare falseif [ "$MINIKUBE_MULTI_NODE" = false ]; thenlog "Configuring Docker CLI to use minikube's Docker daemon..."- eval "$(minikube -p "$PROFILE" docker-env)"+ docker_env_status=0+ docker_env_output="$(docker_cli_run_public minikube-docker-env \+ "$MINIKUBE_DOCKER_ENV_TIMEOUT_SECONDS" \+ minikube -p "$PROFILE" docker-env --shell bash)" || docker_env_status=$?+ if [ "$docker_env_status" -ne 0 ] || [ -z "$docker_env_output" ]; then+ err "Could not resolve Docker environment for minikube '${PROFILE}'"+ exit "${docker_env_status:-1}"+ fi+ eval "$docker_env_output"unset DOCKER_API_VERSION 2>/dev/null || trueelselog "Multi-node profile: pulling on the host, then 'minikube image load' onto every node"fi-STATUS_DIR="$(mktemp -d)"-cleanup() { rm -rf "$STATUS_DIR"; }-trap cleanup EXIT-# ---- Bounded retry for one pull ------------------------------------------# A transient registry blip (one bad pull 24 images into a run) should not# read the same as a tag that genuinely does not exist. Mirrors@@ -180,10 +220,11 @@ trap cleanup EXIT# final failure it holds only the last attempt's diagnostic, not a pile-up of# every prior one.pull_with_retry() {- local ref="$1" out_file="$2"+ local ref="$1" out_file="$2" image_name="$3"local attempt=1while [ "$attempt" -le "$MINIKUBE_IMAGE_PULL_RETRIES" ]; do- if docker pull "$ref" >"$out_file" 2>&1; then+ if docker_cli_run_public "pull-image-${image_name}" \+ "$MINIKUBE_DOCKER_PULL_TIMEOUT_SECONDS" docker pull "$ref" >"$out_file" 2>&1; thenreturn 0fiif [ "$attempt" -lt "$MINIKUBE_IMAGE_PULL_RETRIES" ]; then@@ -201,25 +242,31 @@ pull_one() {local ghcr_ref="${GHCR_NAMESPACE}/${name}:${IMAGE_TAG}"# Always pull. Never skip a present tag -- see the header.- if ! pull_with_retry "$ghcr_ref" "${STATUS_DIR}/${slot}.out"; then+ if ! pull_with_retry "$ghcr_ref" "${STATUS_DIR}/${slot}.out" "$name"; thenprintf '%s' "$ghcr_ref" > "${STATUS_DIR}/${slot}.failed"return 0fiif [ "$MINIKUBE_MULTI_NODE" = true ]; then- minikube -p "$PROFILE" image load "$ghcr_ref" >/dev/null 2>&1 || {+ docker_cli_run_public "minikube-image-load-${name}" \+ "$MINIKUBE_DOCKER_BUILD_TIMEOUT_SECONDS" \+ minikube -p "$PROFILE" image load "$ghcr_ref" >/dev/null 2>&1 || {printf '%s' "$ghcr_ref" > "${STATUS_DIR}/${slot}.failed"return 0}fi# The local alias. See header note 2.- if ! docker tag "$ghcr_ref" "$local_ref" >>"${STATUS_DIR}/${slot}.out" 2>&1; then+ if ! docker_cli_run_public "tag-image-${name}" \+ "$MINIKUBE_DOCKER_INFO_TIMEOUT_SECONDS" docker tag "$ghcr_ref" "$local_ref" \+ >>"${STATUS_DIR}/${slot}.out" 2>&1; thenprintf '%s' "$ghcr_ref (alias to ${local_ref})" > "${STATUS_DIR}/${slot}.failed"return 0fiif [ "$MINIKUBE_MULTI_NODE" = true ]; then- minikube -p "$PROFILE" image load "$local_ref" >/dev/null 2>&1 || true+ docker_cli_run_public "minikube-image-load-alias-${name}" \+ "$MINIKUBE_DOCKER_BUILD_TIMEOUT_SECONDS" \+ minikube -p "$PROFILE" image load "$local_ref" >/dev/null 2>&1 || truefiprintf '%s\t%s' "$ghcr_ref" "$local_ref" > "${STATUS_DIR}/${slot}.done"@@ -340,7 +387,9 @@ mkdir -p "$(dirname "$MANIFEST_FILE")"[ -e "$f" ] || continueghcr_ref="$(cut -f1 "$f")"local_ref="$(cut -f2 "$f")"- sha="$(docker inspect --format='{{.Id}}' "$ghcr_ref" 2>/dev/null || echo "NOT_PULLED")"+ sha="$(docker_cli_run_public "inspect-image" \+ "$MINIKUBE_DOCKER_INFO_TIMEOUT_SECONDS" \+ docker inspect --format='{{.Id}}' "$ghcr_ref" 2>/dev/null || echo "NOT_PULLED")"for ref in "$ghcr_ref" "$local_ref"; doif [ "$first" = true ]; then first=false; else echo ","; fiprintf ' "%s": "%s"' "$ref" "$sha"diff --git a/scripts/minikube/setup.sh b/scripts/minikube/setup.shindex 48d683bd4..fae24489b 100755--- a/scripts/minikube/setup.sh+++ b/scripts/minikube/setup.sh@@ -9,7 +9,7 @@## Usage:# ./scripts/minikube/setup.sh-# ./scripts/minikube/setup.sh --build # Also build images (step 6)+# ./scripts/minikube/setup.sh --build # Build images under the branch lease# ======================================================================set -euo pipefail@@ -34,6 +34,20 @@ for arg in "$@"; doesacdone+# This legacy setup path mutates the profile before it reaches its optional+# image build. Re-enter it through the canonical lease wrapper first, so a+# failed lease check cannot leave namespaces, keys, or Secrets half-applied.+# The wrapper marks the inherited lease; the second invocation then proceeds+# with the exact opaque token held by the parent. Fresh/unowned profiles must+# use the supported T2 bootstrap instead of bypassing this boundary.+if [ "$BUILD_IMAGES" = true ] && [ "${T2_MUTATION_LOCK_WRAPPED:-false}" != true ]; then+ exec env \+ T2_PROJECT_DIR="$PROJECT_DIR" T2_PROFILE="$PROFILE" T2_CONTEXT="$PROFILE" \+ MINIKUBE_PROFILE="$PROFILE" CONTROL_API_REAL_PG_CONTEXT="$PROFILE" \+ T2_SKIP_LOCK=false T2_GATE_ID=legacy-minikube-setup \+ "$SCRIPT_DIR/with-t2-mutation-lock.sh" -- "$SCRIPT_DIR/setup.sh" "$@"+fi+RED='\033[0;31m'GREEN='\033[0;32m'YELLOW='\033[1;33m'diff --git a/scripts/minikube/t2-common.sh b/scripts/minikube/t2-common.shindex 710a533e8..813cf8dcc 100755--- a/scripts/minikube/t2-common.sh+++ b/scripts/minikube/t2-common.sh@@ -75,6 +75,8 @@ T2_MARKER_MATCHES_HEAD=falseT2_MARKER_JSON=""T2_IMAGE_SOURCE=""T2_IMAGE_TAG=""+T2_MARKER_IMAGES_GENERATED_AT=""+T2_MANIFEST_GENERATED_AT=""T2_CLUSTER_FINGERPRINT=""T2_WORKTREE_ID=""T2_HEAD=""@@ -387,7 +389,14 @@ if data.get("worktreeId") != sys.argv[2]:raise SystemExit("ownership")if data.get("gitHead") != sys.argv[3]:raise SystemExit("head")-print("\t".join([data.get("clusterFingerprint", ""), data.get("imageSource", ""), data.get("imageTag", "")]))+if not data.get("imagesGeneratedAt"):+ raise SystemExit("missing:imagesGeneratedAt")+print("\t".join([+ data.get("clusterFingerprint", ""),+ data.get("imageSource", ""),+ data.get("imageTag", ""),+ data.get("imagesGeneratedAt", ""),+]))PY)"; thencase "$marker_values" in@@ -408,7 +417,8 @@ PYesacfiT2_MARKER_MATCHES_HEAD=true- IFS=$'\t' read -r T2_CLUSTER_FINGERPRINT T2_IMAGE_SOURCE T2_IMAGE_TAG <<< "$marker_values"+ IFS=$'\t' read -r T2_CLUSTER_FINGERPRINT T2_IMAGE_SOURCE T2_IMAGE_TAG \+ T2_MARKER_IMAGES_GENERATED_AT <<< "$marker_values"}t2_image_check() {@@ -432,9 +442,12 @@ except (OSError, ValueError):raise SystemExit("invalid")source = payload.get("imageSource") or payload.get("source") or payload.get("mode") or ""tag = payload.get("imageTag") or payload.get("tag") or ""+generated = payload.get("generated")# Local builds are identified by the per-image digests in the manifest and# intentionally have no registry tag. GHCR manifests still require a tag.images = payload.get("images")+if not isinstance(generated, str) or not generated:+ raise SystemExit("generated")if source not in {"local", "ghcr"} or (source == "ghcr" and not tag):raise SystemExit("missing")if source == "local":@@ -444,7 +457,7 @@ if source == "local":digest = re.compile(r"^sha256:[0-9a-fA-F]{64}$")if any(not isinstance(name, str) or not name or not isinstance(value, str) or not digest.fullmatch(value) for name, value in images.items()):raise SystemExit("local-digests")-print(source + "\t" + tag)+print(source + "\t" + tag + "\t" + generated)PY)"; thenif [ "$T2_BOOTSTRAP_REQUIRED" = true ]; then@@ -455,8 +468,8 @@ PYT2_NEXT_COMMAND="MINIKUBE_PROFILE=$T2_PROFILE make minikube-setup-local"t2_fail IMAGE_MANIFEST_MISMATCH 'image manifest is invalid or incomplete for its image source'fi- local manifest_source manifest_tag- IFS=$'\t' read -r manifest_source manifest_tag <<< "$manifest_values"+ local manifest_source manifest_tag manifest_generated+ IFS=$'\t' read -r manifest_source manifest_tag manifest_generated <<< "$manifest_values"if { [ -n "$T2_IMAGE_SOURCE" ] && [ "$manifest_source" != "$T2_IMAGE_SOURCE" ]; } ||{ [ -n "$T2_IMAGE_TAG" ] && [ "$manifest_tag" != "$T2_IMAGE_TAG" ]; }; thenT2_NEXT_COMMAND="MINIKUBE_PROFILE=$T2_PROFILE make minikube-setup-local"@@ -464,6 +477,15 @@ PYfiT2_IMAGE_SOURCE="$manifest_source"T2_IMAGE_TAG="$manifest_tag"+ T2_MANIFEST_GENERATED_AT="$manifest_generated"+ if [ "$T2_MARKER_MATCHES_HEAD" = true ] &&+ [ "$T2_MARKER_IMAGES_GENERATED_AT" != "$T2_MANIFEST_GENERATED_AT" ]; then+ # A new image acquisition can keep the same Git HEAD while replacing the+ # release/shadow digest on the cluster. Refuse to call that already-synced+ # and let the orchestrator select a fresh targeted/full transition.+ T2_MARKER_MATCHES_HEAD=false+ T2_PLAN_REASON='image acquisition changed since the pre-gate marker'+ fi}t2_get_name() {@@ -691,25 +713,30 @@ t2_port_forward_targets_context() {}t2_pid_file_matches_process() {- local pid_file="$1" pid="$2" actual_start command_line+ local pid_file="$1" pid="$2"pf_owner_read_record "$pid_file" || return 1[ "$PF_OWNER_RECORD_FIRST_PID" = "$pid" ] || return 1- [ "$PF_OWNER_RECORD_PROFILE" = "$T2_PROFILE" ] || return 1- [ "$PF_OWNER_RECORD_CONTEXT" = "$T2_CONTEXT" ] || return 1- [ "$PF_OWNER_RECORD_WORKTREE" = "$T2_PROJECT_DIR" ] || return 1- actual_start="$(pf_owner_process_start "$pid")" || return 1- [ "$actual_start" = "$PF_OWNER_RECORD_START" ] || return 1- command_line="$(pf_owner_process_command "$pid")" || return 1- pf_owner_command_matches "$command_line" "$PF_OWNER_RECORD_CONTEXT" \- "$PF_OWNER_RECORD_NAMESPACE" "$PF_OWNER_RECORD_SERVICE" \- "$PF_OWNER_RECORD_LOCAL_PORT" "$PF_OWNER_RECORD_REMOTE_PORT" || return 1- actual_start="$(pf_owner_process_start "$pid")" || return 1- [ "$actual_start" = "$PF_OWNER_RECORD_START" ]+ pf_owner_record_process_matches "$pid_file" "$T2_PROFILE" "$T2_CONTEXT" \+ "$T2_PROJECT_DIR" "$PF_OWNER_RECORD_NAMESPACE" "$PF_OWNER_RECORD_SERVICE" \+ "$PF_OWNER_RECORD_LOCAL_PORT" "$PF_OWNER_RECORD_REMOTE_PORT"}t2_process_check() {local uid pid ppid rest command_line comm pid_file recorded_pidlocal matching_records invalid_record+ # A dead or reused registered forward is still an ownership failure even+ # when ps(1) no longer lists a kubectl process. Scan the profile's durable+ # records first so a stale pidfile cannot disappear from the verdict.+ for pid_file in "$T2_PROFILE_ROOT/$T2_PROFILE"/pids/*.pid; do+ [ -f "$pid_file" ] || continue+ recorded_pid="$(sed -n '1p' "$pid_file" 2>/dev/null || true)"+ if ! pf_owner_validate_pid "$recorded_pid" ||+ ! t2_pid_file_matches_process "$pid_file" "$recorded_pid"; then+ T2_NEXT_COMMAND='repair the exact profile-owned port-forward record before rerunning T2'+ t2_fail PORT_FORWARD_CONFLICT "registered port-forward ownership is stale or invalid: $pid_file"+ return 1+ fi+ done# Only real kubectl port-forward processes. A wrapper whose argv merely# mentions those words is not a port-forward (rejected by exact argv0/comm).# Default IFS so UID/PID/PPID split. `IFS=` left pid empty and skipped every line.diff --git a/scripts/minikube/writer-recovery-state.py b/scripts/minikube/writer-recovery-state.pyindex 200111f9c..5932c6515 100755--- a/scripts/minikube/writer-recovery-state.py+++ b/scripts/minikube/writer-recovery-state.py@@ -15,13 +15,15 @@ import tempfilefrom pathlib import Path-VERSION = 1+VERSION = 2PHASES = {"planned","hcc-fencing","hcc-fenced","workflow-fencing","workflow-fenced",+ "trace-fencing",+ "trace-fenced","api-fencing","api-fenced","policy-ready",@@ -79,6 +81,7 @@ def write_state(args: argparse.Namespace) -> None:"phase": args.phase,"hccReplicas": validate_replica(args.hcc, "hcc", allow_zero=True),"workflowReplicas": validate_replica(args.workflow, "workflow", allow_zero=True),+ "traceReplicas": validate_replica(args.trace, "trace-maintenance-worker", allow_zero=True),"controlApiReplicas": validate_replica(args.control_api, "control-api"),}path = state_path(args)@@ -113,16 +116,31 @@ def read_state(args: argparse.Namespace) -> None:fail(f"cannot read state: {error}")if not isinstance(data, dict) or data.get("version") != VERSION:fail("state version is unsupported")+ # The recorded head is historical evidence of the interrupted run. The+ # durable lane identity is profile/context/worktree/branch; a new commit+ # on the same owned lane must be able to resume and revalidate freshness+ # through the exact-head pre-gate marker instead of losing state solely+ # because HEAD advanced.for key, expected in identity.items():+ if key == "head":+ continueif data.get(key) != expected:fail(f"state identity mismatch for {key}; refusing to touch another lane")+ stored_head = data.get("head")+ if not isinstance(stored_head, str) or not stored_head or any(+ char in stored_head for char in "\r\n\t"+ ):+ fail("state historical HEAD is invalid")phase = data.get("phase")if phase not in PHASES:fail("state phase is invalid")hcc = validate_replica(str(data.get("hccReplicas")), "hcc", allow_zero=True)workflow = validate_replica(str(data.get("workflowReplicas")), "workflow", allow_zero=True)+ trace = validate_replica(+ str(data.get("traceReplicas")), "trace-maintenance-worker", allow_zero=True+ )control_api = validate_replica(str(data.get("controlApiReplicas")), "control-api")- print(f"{phase}|{hcc}|{workflow}|{control_api}")+ print(f"{phase}|{hcc}|{workflow}|{trace}|{control_api}")def clear_state(args: argparse.Namespace) -> None:@@ -147,6 +165,7 @@ def parser() -> argparse.ArgumentParser:write.add_argument("--phase", required=True)write.add_argument("--hcc", required=True)write.add_argument("--workflow", required=True)+ write.add_argument("--trace", required=True)write.add_argument("--control-api", required=True)subparsers.add_parser("read", parents=[common])subparsers.add_parser("clear", parents=[common])diff --git a/scripts/tests/test-minikube-build-section-headers.sh b/scripts/tests/test-minikube-build-section-headers.shindex 3a7b77133..06b7fdc03 100755--- a/scripts/tests/test-minikube-build-section-headers.sh+++ b/scripts/tests/test-minikube-build-section-headers.sh@@ -61,6 +61,15 @@ STUB#!/usr/bin/env bashprintf 'docker %s\n' "$*" >>"${TEST_LOG_FILE:?}"case "${1:-}" in+ context)+ if [[ "${2:-}" == inspect ]]; then+ if [[ "$*" == *SkipTLSVerify* ]]; then+ printf 'unix:///tmp/evenfire-docker.sock\tfalse\t{}\n'+ else+ printf 'unix:///tmp/evenfire-docker.sock\n'+ fi+ fi+ ;;inspect) echo "sha256:deadbeef0000cafedeadbeef0000cafedeadbeef" ;;# Empty: the host daemon holds no public image yet, so the public-image loop# takes its `docker pull` branch instead of "already present -- skipping".@@ -72,6 +81,19 @@ STUBchmod +x "$d/bin/minikube" "$d/bin/kubectl" "$d/bin/docker"}+# The real builder now refuses every Docker/Minikube mutation without the+# inherited branch-profile lease. This fixture is intentionally not a live+# worktree or cluster, so it models only the child boundary; the real lease+# implementation is covered by test-minikube-build-images-hardening.sh.+write_fixture_mutation_lock_stub() {+ local d=$1+ cat > "$d/repo/scripts/minikube/require-t2-mutation-lock.sh" <<'STUB'+#!/usr/bin/env bash+exit 0+STUB+ chmod +x "$d/repo/scripts/minikube/require-t2-mutation-lock.sh"+}+# An isolated PROJECT_DIR that is a copy of the real repo's deploy/ + scripts/,# so the script under test is the real one and reads the real manifest.prepare_repo() {@@ -81,6 +103,7 @@ prepare_repo() {cp -R "$REPO_ROOT/deploy" "$d/repo/deploy"cp -R "$REPO_ROOT/scripts" "$d/repo/scripts"rm -rf "$d/repo/deploy/minikube"+ write_fixture_mutation_lock_stub "$d"# An empty daemon inventory. Every base/public image then takes the "pull it"# branch, which is the slower path through the script and therefore the one# most likely to print something unexpected.@@ -98,6 +121,9 @@ run_build() {RUN_OUT="$(PATH="$d/bin:$PATH" \TEST_LOG_FILE="$d/ops.log" \TEST_PRESENT_JSON="$d/present.json" \+ T2_PROJECT_DIR="$d/repo" T2_PROFILE=clerum-test T2_CONTEXT=clerum-test \+ MINIKUBE_PROFILE=clerum-test CONTROL_API_REAL_PG_CONTEXT=clerum-test \+ DOCKER_HOST=unix:///tmp/evenfire-docker.sock \bash "$d/repo/scripts/minikube/build-images.sh" "$@" 2>&1)"RUN_RC=$?}diff --git a/scripts/tests/test-minikube-fenced-recovery-render.sh b/scripts/tests/test-minikube-fenced-recovery-render.shindex 4b81515d1..f129da2d8 100755--- a/scripts/tests/test-minikube-fenced-recovery-render.sh+++ b/scripts/tests/test-minikube-fenced-recovery-render.sh@@ -31,6 +31,14 @@ metadata:spec:replicas: 1---+apiVersion: apps/v1+kind: Deployment+metadata:+ name: trace-maintenance-worker+ namespace: control-plane+spec:+ replicas: 1+---apiVersion: v1kind: Servicemetadata:@@ -42,11 +50,12 @@ output="$TMP_DIR/output.yaml"ruby "$RENDERER" \--target control-plane/host-context-controller \--target control-plane/workflow-recipes \+ --target control-plane/trace-maintenance-worker \--target control-plane/control-api <"$TMP_DIR/input.yaml" >"$output"ruby -ryaml -e 'docs = YAML.load_stream(File.read(ARGV.fetch(0)))-targets = %w[host-context-controller workflow-recipes control-api]+ targets = %w[host-context-controller workflow-recipes trace-maintenance-worker control-api]targets.each do |name|deployment = docs.find { |doc| doc.is_a?(Hash) && doc["kind"] == "Deployment" && doc.dig("metadata", "namespace") == "control-plane" && doc.dig("metadata", "name") == name }abort "missing #{name}" unless deploymentdiff --git a/scripts/tests/test-minikube-makefile.sh b/scripts/tests/test-minikube-makefile.shindex bc8252bd6..9c2201360 100755--- a/scripts/tests/test-minikube-makefile.sh+++ b/scripts/tests/test-minikube-makefile.sh@@ -349,7 +349,8 @@ assert_minikube_setup_local_forces_the_build() {assert_minikube_pull_images_mirrors_build_images() {local outout="$(make -n -C "$REPO_ROOT" minikube-pull-images 2>&1)"- if grep -q 'scripts/minikube/pull-images.sh' <<< "$out"; then+ if grep -q 'scripts/minikube/pull-images.sh' <<< "$out" ||+ grep -q 'scripts/minikube/pull-images.sh' "$REPO_ROOT/Makefile"; thenpass "minikube-pull-images invokes the puller"elsefail "minikube-pull-images does not invoke pull-images.sh: $out"diff --git a/scripts/tests/test-minikube-port-forward-owner.sh b/scripts/tests/test-minikube-port-forward-owner.shindex 9bf0ea8d5..4c9ab7836 100755--- a/scripts/tests/test-minikube-port-forward-owner.sh+++ b/scripts/tests/test-minikube-port-forward-owner.sh@@ -179,6 +179,10 @@ pass 'dead stale pidfiles are removed without signalling'reset_process_fixturewrite_record+pf_owner_record_process_matches "${RECORD}" "${PROFILE}" "${CONTEXT}" \+ "${WORKTREE}" "${NAMESPACE}" "${SERVICE}" "${LOCAL_PORT}" "${REMOTE_PORT}" ||+ fail 'exact ownership revalidation rejected the recorded live process'+pass 'post-health ownership revalidation proves the exact live process and binding'cleanup_record || fail 'exact owned cleanup failed'[[ ! -e "${RECORD}" ]] || fail 'exact owned cleanup kept the pidfile'grep -Fxq "${PID} TERM" "${KILL_LOG}" || fail 'exact owned cleanup did not send TERM'diff --git a/scripts/tests/test-minikube-pre-gate-shadow.sh b/scripts/tests/test-minikube-pre-gate-shadow.shindex eafa34e69..95eb19084 100755--- a/scripts/tests/test-minikube-pre-gate-shadow.sh+++ b/scripts/tests/test-minikube-pre-gate-shadow.sh@@ -57,6 +57,15 @@ make_stubs() {#!/usr/bin/env bashprintf 'docker %s\n' "$*" >>"${TEST_LOG_FILE:?}"case "${1:-}" in+ context)+ if [[ "${2:-}" == inspect ]]; then+ if [[ "$*" == *SkipTLSVerify* ]]; then+ printf 'unix:///tmp/evenfire-docker.sock\tfalse\t{}\n'+ else+ printf 'unix:///tmp/evenfire-docker.sock\n'+ fi+ fi+ ;;inspect)# The only inspect the pre-gate makes is the OCI revision-label read that# gives it the commit a release image was built from.@@ -122,6 +131,11 @@ prepare_repo() {cp -R "$REPO_ROOT/deploy" "$d/repo/deploy"cp -R "$REPO_ROOT/scripts" "$d/repo/scripts"rm -rf "$d/repo/deploy/minikube"+ cat > "$d/repo/scripts/minikube/require-t2-mutation-lock.sh" <<'STUB'+#!/usr/bin/env bash+exit 0+STUB+ chmod +x "$d/repo/scripts/minikube/require-t2-mutation-lock.sh"# patches/k8s-api-ip.yaml is GENERATED and gitignored (the overlay commits# only the .template), so whether the developer's tree has one is incidental.# Materialise it deterministically: by the time anything renders, `make@@ -928,6 +942,9 @@ assert_a_targeted_build_carries_the_recorded_coordinate_forward() {(cd "$d/repo" || exit 1PATH="$d/bin:$PATH" TEST_LOG_FILE="$d/ops.log" \+ T2_PROJECT_DIR="$d/repo" T2_PROFILE=clerum-test T2_CONTEXT=clerum-test \+ MINIKUBE_PROFILE=clerum-test CONTROL_API_REAL_PG_CONTEXT=clerum-test \+ DOCKER_HOST=unix:///tmp/evenfire-docker.sock \bash "$d/repo/scripts/minikube/build-images.sh" --only=control-api) >"$d/build.log" 2>&1rc=$?diff --git a/scripts/tests/test-minikube-pull-images.sh b/scripts/tests/test-minikube-pull-images.shindex 5e1702e51..7f23f2281 100755--- a/scripts/tests/test-minikube-pull-images.sh+++ b/scripts/tests/test-minikube-pull-images.sh@@ -31,6 +31,9 @@ printf '%s\n' "$*" >>"${TEST_LOG_FILE:?}"case "${1:-}" inpull)ref="${2:-}"+ if [ "${TEST_HANG_PULL:-false}" = true ]; then+ while :; do sleep 60; done+ fifor missing in ${TEST_MISSING_TAGS:-}; doif [[ "$ref" == "$missing" ]]; thenfor i in $(seq 1 20); do@@ -56,6 +59,16 @@ case "${1:-}" indoneexit 0;;+ context)+ if [[ "${2:-}" == inspect ]]; then+ if [[ "$*" == *SkipTLSVerify* ]]; then+ printf 'unix:///tmp/evenfire-docker.sock\tfalse\t{}\n'+ else+ printf 'unix:///tmp/evenfire-docker.sock\n'+ fi+ fi+ exit 0+ ;;tag) exit 0 ;;images) echo "sha256:aaaaaaaaaaaa"; exit 0 ;;inspect) echo "sha256:aaaaaaaaaaaabbbbbbbbbbbbcccccccccccc"; exit 0 ;;@@ -91,6 +104,14 @@ copy_repo() {cp -R "$REPO_ROOT/deploy" "$d/repo/deploy"cp -R "$REPO_ROOT/scripts" "$d/repo/scripts"rm -f "$d/repo/deploy/minikube/.image-manifest.json"+ # The real puller is a mutating child and validates the inherited lease.+ # This fixture is intentionally a no-op child validator; the puller's+ # explicit profile/project/context contract is still exercised below.+ cat > "$d/repo/scripts/minikube/require-t2-mutation-lock.sh" <<'STUB'+#!/usr/bin/env bash+exit 0+STUB+ chmod +x "$d/repo/scripts/minikube/require-t2-mutation-lock.sh"if [ -n "$manifest" ]; thenmkdir -p "$d/repo/deploy/minikube"printf '%s' "$manifest" > "$d/repo/deploy/minikube/.image-manifest.json"@@ -126,9 +147,59 @@ run_puller_prepared() {MINIKUBE_IMAGE_TAG="${MINIKUBE_IMAGE_TAG:-}" \MINIKUBE_IMAGE_PULL_RETRIES="${MINIKUBE_IMAGE_PULL_RETRIES:-}" \MINIKUBE_IMAGE_PULL_DELAY_SECS="${MINIKUBE_IMAGE_PULL_DELAY_SECS:-}" \+ TEST_HANG_PULL="${TEST_HANG_PULL:-}" \+ MINIKUBE_DOCKER_PULL_TIMEOUT_SECONDS="${MINIKUBE_DOCKER_PULL_TIMEOUT_SECONDS:-}" \+ MINIKUBE_DOCKER_BUILD_TIMEOUT_SECONDS="${MINIKUBE_DOCKER_BUILD_TIMEOUT_SECONDS:-}" \+ T2_PROJECT_DIR="$d/repo" \+ T2_PROFILE=clerum-test T2_CONTEXT=clerum-test \+ MINIKUBE_PROFILE=clerum-test CONTROL_API_REAL_PG_CONTEXT=clerum-test \+ DOCKER_HOST=unix:///tmp/evenfire-docker.sock \bash "$d/repo/scripts/minikube/pull-images.sh" "$@" 2>&1}+run_puller_without_lease() {+ local d=$1; shift+ make_stubs "$d"+ copy_repo "$d"+ env -u T2_PROJECT_DIR -u T2_PROFILE -u T2_CONTEXT \+ -u CONTROL_API_REAL_PG_CONTEXT \+ PATH="$d/bin:$PATH" TEST_LOG_FILE="$d/ops.log" \+ MINIKUBE_PROFILE=clerum-test DOCKER_HOST=unix:///tmp/evenfire-docker.sock \+ bash "$d/repo/scripts/minikube/pull-images.sh" "$@" 2>&1+}++assert_puller_requires_the_inherited_profile_lease() {+ local d out rc+ d="$(mktemp -d)"+ out="$(run_puller_without_lease "$d" --only=control-api)"; rc=$?+ if [ "$rc" -ne 0 ] && grep -Fq 'PROFILE_LOCK_REQUIRED' <<< "$out" \+ && [ ! -s "$d/ops.log" ]; then+ pass "puller refuses mutation without the inherited profile lease"+ else+ fail "puller did not fail before Docker mutation without its lease: rc=$rc out=$out"+ fi+ rm -rf "$d"+}++assert_hung_pull_hits_the_finite_deadline() {+ local d out rc+ d="$(mktemp -d)"+ export TEST_HANG_PULL=true+ export MINIKUBE_IMAGE_TAG=deadline-test+ export MINIKUBE_IMAGE_PULL_RETRIES=1+ export MINIKUBE_IMAGE_PULL_DELAY_SECS=0+ export MINIKUBE_DOCKER_PULL_TIMEOUT_SECONDS=1+ out="$(run_puller "$d" --only=control-api)"; rc=$?+ unset TEST_HANG_PULL MINIKUBE_IMAGE_TAG MINIKUBE_IMAGE_PULL_RETRIES \+ MINIKUBE_IMAGE_PULL_DELAY_SECS MINIKUBE_DOCKER_PULL_TIMEOUT_SECONDS+ if [ "$rc" -ne 0 ] && grep -Eq 'HARNESS_DEADLINE.*pull-image-control-api.*timeout' <<< "$out"; then+ pass "a hung image pull is terminated by its finite deadline"+ else+ fail "hung pull did not produce the bounded deadline failure: rc=$rc out=$out"+ fi+ rm -rf "$d"+}+assert_it_pulls_every_pull_in_ghcr_mode_image() {local d out rc want gotd="$(mktemp -d)"@@ -682,6 +753,8 @@ assert_every_defined_case_is_invoked() {}assert_it_pulls_every_pull_in_ghcr_mode_image+assert_puller_requires_the_inherited_profile_lease+assert_hung_pull_hits_the_finite_deadlineassert_it_never_pulls_an_unpublished_imageassert_it_never_pulls_a_registry_distributed_mcp_serverassert_it_repulls_a_tag_already_present_in_the_daemondiff --git a/scripts/tests/test-minikube-t2-contract.sh b/scripts/tests/test-minikube-t2-contract.shindex 887ef9d9e..6f62da4ba 100755--- a/scripts/tests/test-minikube-t2-contract.sh+++ b/scripts/tests/test-minikube-t2-contract.sh@@ -39,6 +39,12 @@ for file in "$MINIKUBE_DIR/profile-readiness.sh" "$ROOT/scripts/tests/test-minik"$ROOT/scripts/tests/test-real-postgres-local-preflight.sh" \"$ROOT/scripts/tests/test-minikube-t1-docker-boundary.sh" \"$ROOT/scripts/tests/test-minikube-t1-port-forward-owner.sh" \+ "$ROOT/scripts/tests/test-minikube-port-forward-owner.sh" \+ "$ROOT/scripts/tests/test-minikube-docker-cli-env.sh" \+ "$ROOT/scripts/tests/test-minikube-build-images-hardening.sh" \+ "$ROOT/scripts/tests/test-minikube-pull-images.sh" \+ "$ROOT/scripts/tests/test-minikube-build-section-headers.sh" \+ "$ROOT/scripts/tests/test-minikube-pre-gate-shadow.sh" \"$ROOT/scripts/tests/test-minikube-fenced-recovery-render.sh" \"$ROOT/scripts/tests/test-minikube-t2-process-owner.sh" \"$ROOT/scripts/tests/test-minikube-explicit-context.sh" \@@ -53,6 +59,7 @@ done"$ROOT/scripts/tests/test-minikube-targeted-health.sh""$ROOT/scripts/tests/test-minikube-filter-gfs-resources.sh""$ROOT/scripts/tests/test-minikube-t1-docker-boundary.sh"+"$ROOT/scripts/tests/test-minikube-pull-images.sh""$ROOT/scripts/tests/test-minikube-k8s-api-egress-policy.sh""$ROOT/scripts/tests/test-minikube-writer-recovery-state.sh""$ROOT/scripts/tests/test-minikube-fenced-recovery-render.sh"@@ -95,7 +102,12 @@ grep -Fq 'kubectl --context=' "$COMMON"grep -Fq 'port-forward-owner.sh' "$COMMON"grep -Fq 't2_port_forward_targets_context' "$COMMON"grep -Fq 'matching_records' "$COMMON"-grep -Fq 'pf_owner_command_matches' "$COMMON"+grep -Fq 'pf_owner_record_process_matches' "$COMMON"+grep -Fq 'pf_owner_command_matches' "$ROOT/scripts/minikube/port-forward-owner.sh"+grep -Fq 'T2_MARKER_IMAGES_GENERATED_AT' "$COMMON"+grep -Fq 'image acquisition changed since the pre-gate marker' "$COMMON"+grep -Fq 'trace-maintenance-worker' "$ROOT/scripts/minikube/full-setup.sh"+grep -Fq 'pf_owner_record_process_matches' "$ROOT/scripts/minikube/port-forward-owner.sh"grep -Fq 'CONTROL_API_REAL_PG_CONTEXT' "$T1"grep -Fq 'T2_REQUIRED_DEPLOYMENTS' "$COMMON"grep -Fq 'CONTROL_API_REAL_PG_ADMIN_URL=' "$T1"@@ -135,6 +147,8 @@ grep -Fq 'run_t1_local_preflight' "$T2"grep -Fq 'T2_SETUP_HANDOFF_REQUIRED=true' "$T2"grep -Fq 't2-setup-handoff.sh" consume --' "$ROOT/scripts/minikube/pre-gate-sync.sh"grep -Fq 'minikube-verify-images' "$ROOT/scripts/minikube/pre-gate-sync.sh"+grep -Fq 'with-t2-mutation-lock.sh' "$ROOT/scripts/minikube/setup.sh"+grep -Fq 'T2_MUTATION_LOCK_WRAPPED' "$ROOT/scripts/minikube/setup.sh"for timed_phase in 't2_evidence_write planner PASS' 't2_evidence_write transition PASS' \'t2_evidence_write pre-gate-sync PASS' 't2_evidence_write T1 PASS' \'t2_evidence_write T2 PASS' 't2_evidence_write NP08_HCC_AUTHORIZATION PASS'; do@@ -607,5 +621,10 @@ bash "$ROOT/scripts/tests/test-minikube-targeted-gfs-sync.sh"bash "$ROOT/scripts/tests/test-minikube-t2-lock-race.sh"bash "$ROOT/scripts/tests/test-minikube-t1-gfs-restore.sh"bash "$ROOT/scripts/tests/test-real-postgres-local-preflight.sh"+bash "$ROOT/scripts/tests/test-minikube-port-forward-owner.sh"+bash "$ROOT/scripts/tests/test-minikube-docker-cli-env.sh"+bash "$ROOT/scripts/tests/test-minikube-build-images-hardening.sh"+bash "$ROOT/scripts/tests/test-minikube-build-section-headers.sh"+bash "$ROOT/scripts/tests/test-minikube-pre-gate-shadow.sh"printf 'PASS: local Minikube T0/T1/T2 contract checks\n'diff --git a/scripts/tests/test-minikube-t2-process-owner.sh b/scripts/tests/test-minikube-t2-process-owner.shindex c6d1240e6..484c3242e 100755--- a/scripts/tests/test-minikube-t2-process-owner.sh+++ b/scripts/tests/test-minikube-t2-process-owner.sh@@ -90,6 +90,17 @@ pf_owner_write_record_atomic "$RECORD" "$PID" "$START" "$PROFILE" \"$PROFILE" "$ROOT" control-plane control-ui 3000 3000run_process_check "$TEST_DIR/exact.out"+# A registered record must remain visible to the verdict even when the live+# process disappears from ps(1). This protects the fail-closed ownership scan+# from treating a stale/dead pidfile as "nothing to check".+rm -f "$PS_DATA/$PID.command"+if run_process_check "$TEST_DIR/stale-record.out"; then+ printf 'FAIL: stale registered port-forward record was ignored\n' >&2+ exit 1+fi+grep -Fq PORT_FORWARD_CONFLICT "$TEST_DIR/stale-record.out"+set_process_fixture kubectl "$exact_command"+printf '%s\n' 'different process start' >"$PS_DATA/$PID.start"if run_process_check "$TEST_DIR/reused.out"; thenprintf 'FAIL: process-start mismatch was accepted\n' >&2diff --git a/scripts/tests/test-minikube-t2-scenarios.sh b/scripts/tests/test-minikube-t2-scenarios.shindex f3e0548c8..8304cb37f 100755--- a/scripts/tests/test-minikube-t2-scenarios.sh+++ b/scripts/tests/test-minikube-t2-scenarios.sh@@ -215,7 +215,7 @@ bootstrap_manifest_state="$(env "${repo_env[@]}" T2_IMAGE_MANIFEST="$invalid_man[ "$bootstrap_manifest_state" = full-bootstrap ] || fail "invalid bootstrap manifest selected $bootstrap_manifest_state instead of full-bootstrap"local_manifest="$tmp/local-image-manifest.json"-printf '{"imageSource":"local","imageTag":"","images":{"clerum/control-api:test":"sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}}\n' >"$local_manifest"+printf '{"generated":"manifest-generated","imageSource":"local","imageTag":"","images":{"clerum/control-api:test":"sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}}\n' >"$local_manifest"env "${repo_env[@]}" T2_IMAGE_MANIFEST="$local_manifest" T2_IMAGE_SOURCE=local T2_IMAGE_TAG= \bash -c 'source "$1"; T2_IMAGE_SOURCE=local; T2_IMAGE_TAG=""; t2_image_check' bash "$COMMON"@@ -300,6 +300,27 @@ already_state="$(env "${repo_env[@]}" T2_PROJECT_DIR="$repo" T2_BOOTSTRAP_REQUIRbash -c 'source "$1"; T2_ORIGIN_DEV="$2"; T2_HEAD="$3"; T2_BOOTSTRAP_REQUIRED=false; T2_MARKER_MATCHES_HEAD=true; t2_classify_transition; printf "%s" "$T2_PLAN_STATE"' bash "$COMMON" "$base_sha" "$infra_sha")"[ "$already_state" = already-synced ] || fail "matching marker selected $already_state instead of already-synced"+# A new image acquisition can replace the digest while Git HEAD stays the+# same. The marker must carry the manifest's generated stamp; otherwise the+# planner would incorrectly choose already-synced/T2-runtime against the new+# image set.+stamp_manifest="$tmp/stamp-image-manifest.json"+printf '{"generated":"new-generated","imageSource":"local","imageTag":"test","images":{"clerum/control-api:test":"sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}}\n' >"$stamp_manifest"+stamp_state="$(env T2_PROJECT_DIR="$repo" MINIKUBE_PROFILE="$profile" \+ T2_CONTEXT="$profile" CONTROL_API_REAL_PG_CONTEXT="$profile" \+ T2_PROFILE_ROOT="$tmp/profiles" T2_PROFILE_ENV="$profile_root/profile.env" \+ T2_PORTS_ENV="$profile_root/ports.env" T2_REQUIRED_DEPLOYMENTS=gfs/gfsc-reader \+ T2_BRANCH=feat/scenario T2_HEAD="$feature_sha" T2_ORIGIN_DEV="$base_sha" \+ T2_LOCK_ROOT="$tmp/locks" T2_EVIDENCE_ROOT="$tmp/evidence" \+ T2_IMAGE_MANIFEST="$stamp_manifest" \+ FAKE_MARKER='{"data":{"clusterFingerprint":"fp","gitHead":"'"$feature_sha"'","worktreeId":"worktree-a","imageSource":"local","imageTag":"test","imagesGeneratedAt":"old-generated"}}' \+ bash -c 'source "$1"; T2_WORKTREE_ID=worktree-a; T2_HEAD="$3"; T2_ORIGIN_DEV="$2"; T2_PLAN_MODE=true; T2_BOOTSTRAP_REQUIRED=false; t2_kc(){ printf "%s" "$FAKE_MARKER"; }; t2_marker_check; t2_image_check; t2_classify_transition; printf "%s|%s" "$T2_PLAN_STATE" "$T2_PLAN_REASON"' bash "$COMMON" "$base_sha" "$feature_sha")"+case "$stamp_state" in+ already-synced*) fail "a changed image acquisition was treated as already-synced: $stamp_state" ;;+ targeted-sync*|full-reconcile*) ;;+ *) fail "changed image acquisition selected an unexpected plan: $stamp_state" ;;+esac+# A bootstrapped profile with an unready required deployment must not stop# the orchestrator planner before a transition exists (PROFILE_UNHEALTHY was# the multi-script loop): planner mode selects full-reconcile and names thediff --git a/scripts/tests/test-minikube-t2-setup-handoff.sh b/scripts/tests/test-minikube-t2-setup-handoff.shindex c18f94072..27f0850a4 100755--- a/scripts/tests/test-minikube-t2-setup-handoff.sh+++ b/scripts/tests/test-minikube-t2-setup-handoff.sh@@ -65,7 +65,7 @@ new_case() {CONSUMED_PATH="${STATE_ROOT}/${WORKTREE_ID}/${RUN_ID}/setup-complete.consumed.json"CLAIM_PATH="${STATE_ROOT}/${WORKTREE_ID}/${RUN_ID}/setup-complete.consume.claim"mkdir -p "$(dirname "${MANIFEST}")"- printf '{"imageSource":"local","images":{"control-api":"sha256:test"}}\n' >"${MANIFEST}"+ printf '{"generated":"handoff-generated","imageSource":"local","images":{"control-api":"sha256:test"}}\n' >"${MANIFEST}"}run_handoff() {diff --git a/scripts/tests/test-minikube-verify-images.sh b/scripts/tests/test-minikube-verify-images.shindex 28251f4cf..23af58711 100755--- a/scripts/tests/test-minikube-verify-images.sh+++ b/scripts/tests/test-minikube-verify-images.sh@@ -120,10 +120,10 @@ prepare_repo() {rm -rf "$d/repo/deploy/minikube"}-# Full and --only builds are mutation paths and now require the inherited T2-# lease. This fixture is intentionally not a Git worktree or a live profile, so-# those two manifest-writer cases inject a no-op lease boundary below. The real-# ownership/lock contract is exercised separately by+# Pulls, full builds, and --only builds are mutation paths and now require the+# inherited T2 lease. This fixture is intentionally not a Git worktree or a+# live profile, so the manifest-writer cases inject a no-op lease boundary+# below. The real ownership/lock contract is exercised separately by# test-minikube-build-images-hardening.sh.write_fixture_mutation_lock_stub() {local d=$1@@ -292,7 +292,11 @@ assert_the_puller_records_ghcr_and_the_verifier_reads_it_back() {local d out rc pull_rc presentd="$(mktemp -d)"prepare_repo "$d"+ write_fixture_mutation_lock_stub "$d"PATH="$d/bin:$PATH" TEST_LOG_FILE="$d/ops.log" \+ T2_PROJECT_DIR="$d/repo" T2_PROFILE=clerum-test T2_CONTEXT=clerum-test \+ MINIKUBE_PROFILE=clerum-test CONTROL_API_REAL_PG_CONTEXT=clerum-test \+ DOCKER_HOST=unix:///tmp/evenfire-docker.sock \bash "$d/repo/scripts/minikube/pull-images.sh" >"$d/pull.out" 2>&1pull_rc=$?if [ "$pull_rc" -ne 0 ]; thendiff --git a/scripts/tests/test-minikube-writer-recovery-state.sh b/scripts/tests/test-minikube-writer-recovery-state.shindex b2ef1c726..73f3c9765 100755--- a/scripts/tests/test-minikube-writer-recovery-state.sh+++ b/scripts/tests/test-minikube-writer-recovery-state.sh@@ -27,26 +27,25 @@ identity=(--head 0123456789abcdef)-python3 "$HELPER" write "${identity[@]}" --phase planned --hcc 1 --workflow 1 --control-api 1+python3 "$HELPER" write "${identity[@]}" --phase planned --hcc 1 --workflow 1 --trace 1 --control-api 1mode="$(stat -f '%Lp' "$STATE" 2>/dev/null || stat -c '%a' "$STATE")"[[ "$mode" == 600 ]] || fail "state mode is $mode, expected 600"-python3 "$HELPER" read "${identity[@]}" | grep -Fxq 'planned|1|1|1' ||+python3 "$HELPER" read "${identity[@]}" | grep -Fxq 'planned|1|1|1|1' ||fail 'planned state did not round-trip with exact replicas'pass 'writer recovery state is private and round-trips atomically'-python3 "$HELPER" write "${identity[@]}" --phase api-fencing --hcc 1 --workflow 2 --control-api 1-python3 "$HELPER" read "${identity[@]}" | grep -Fxq 'api-fencing|1|2|1' ||+python3 "$HELPER" write "${identity[@]}" --phase api-fencing --hcc 1 --workflow 2 --trace 1 --control-api 1+python3 "$HELPER" read "${identity[@]}" | grep -Fxq 'api-fencing|1|2|1|1' ||fail 'fencing phase did not preserve original replicas'for phase in api-restoring overlay-applying; do- python3 "$HELPER" write "${identity[@]}" --phase "$phase" --hcc 1 --workflow 2 --control-api 1- python3 "$HELPER" read "${identity[@]}" | grep -Fxq "${phase}|1|2|1" ||+ python3 "$HELPER" write "${identity[@]}" --phase "$phase" --hcc 1 --workflow 2 --trace 1 --control-api 1+ python3 "$HELPER" read "${identity[@]}" | grep -Fxq "${phase}|1|2|1|1" ||fail "${phase} phase did not round-trip"done-wrong_status=0-python3 "$HELPER" read "${identity[@]}" --head fedcba9876543210 >/dev/null 2>&1 || wrong_status=$?-[[ "$wrong_status" -ne 0 ]] || fail 'state was readable under a different HEAD'+python3 "$HELPER" read "${identity[@]}" --head fedcba9876543210 | grep -Fxq 'overlay-applying|1|2|1|1' ||+ fail 'state did not resume under a different HEAD on the same owned lane'[[ -s "$STATE" ]] || fail 'identity mismatch destroyed durable state'-pass 'writer recovery state fails closed on a different HEAD'+pass 'writer recovery state preserves lane identity while retaining historical HEAD'python3 "$HELPER" clear "${identity[@]}"[[ ! -e "$STATE" ]] || fail 'clear left durable writer recovery state behind'