Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix(minikube): close leased image and recovery harness gaps

Alfredo Lopezad122691a41c
diff --git a/.cursor/skills/minikube-t0-t1-t2/SKILL.md b/.cursor/skills/minikube-t0-t1-t2/SKILL.md
index d8346d6c6..6d2047815 100644
--- a/.cursor/skills/minikube-t0-t1-t2/SKILL.md
+++ b/.cursor/skills/minikube-t0-t1-t2/SKILL.md
@@ -24,7 +24,9 @@ Before running anything, verify ALL of these:
Do NOT create a new profile because HEAD, gate, or command changed.
Resolve it with the primary checkout `.local-notes/minikube-profiles/branch.mk`;
profile identity is stable for canonical worktree + branch, while the
- pre-gate marker—not a profile-name SHA—proves exact-HEAD freshness.
+ pre-gate marker—not a profile-name SHA—proves exact-HEAD freshness. The
+ marker must also match the image manifest's exact `imagesGeneratedAt`
+ stamp; a new image acquisition on the same HEAD is not already-synced.
- [ ] Confirm the profile is not owned by another active branch/worktree. If
ownership is ambiguous, stop and use a dedicated profile instead.
- [ ] Every `kubectl` you run manually uses `--context=<owned profile>`.
@@ -32,9 +34,9 @@ Before running anything, verify ALL of these:
- [ ] Docker resolves to a local Unix socket or loopback TCP endpoint. The
harness pins that endpoint into an empty task-local config; do not use a
remote Docker context or copy ambient registry credentials into it.
-- [ ] Mutating image builds use the public Make target/orchestrator and inherit
- its exact profile lease. Do not call `build-images.sh` directly;
- `--verify-only` is the read-only exception.
+- [ ] Mutating image acquisition/builds use the public Make target/orchestrator
+ and inherit its exact profile lease. Do not call `build-images.sh` or
+ `pull-images.sh` directly; `--verify-only` is the read-only exception.
- [ ] Never read `~/.cache/clerum/minikube-profiles/` directly (HARD DENY —
it holds private profile state). The harness reads it for you.
@@ -150,6 +152,12 @@ it is optional for other transitions. `T2_PLAYWRIGHT_COMMAND` remains opt-in
Evidence stays under the ignored `.local-notes/infra/runs/`. Never commit it.
Before committing anything else, run `make minikube-t2-public-boundary`.
+The final exact-head check also rejects a marker whose `imagesGeneratedAt`
+does not equal the current image manifest. During REUSE_DB recovery, the
+durable fence covers HCC, workflow-recipes, trace-maintenance-worker, and
+control-api; a changed HEAD on the same owned lane is historical state, not a
+reason to discard recovery evidence.
+
## Step 5 — On failure
Failures print a stable code and a next safe command. Repair the FIRST
diff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.md
index 378eb3086..6facbeecf 100644
--- a/.cursor/skills/minikube-t0-t1-t2/reference.md
+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md
@@ -25,9 +25,10 @@ Companion to `SKILL.md`. Source of truth: `scripts/minikube/t2.sh`,
fixed localhost ports** — only the profile-owned random port mapping.
- **Do not touch another worktree's Minikube profile** (e.g. an NP-08 lane
profile) or kill its port-forwards, even if it looks idle.
-- **Do not run a mutating `build-images.sh` directly.** Use the public Make
- target or T2 orchestrator so the exact worktree/profile/context lease is
- inherited. Only `--verify-only` is lease-exempt.
+- **Do not run mutating `build-images.sh` or `pull-images.sh` directly.** Use
+ the public Make target or T2 orchestrator so the exact
+ worktree/profile/context lease is inherited. Only `--verify-only` is
+ lease-exempt.
- **Do not use a remote/ambiguous Docker endpoint or inherit ambient registry
auth.** The harness accepts local Unix sockets and loopback TCP endpoints,
pins the resolved endpoint before isolation, and requires an explicit
@@ -39,6 +40,9 @@ Companion to `SKILL.md`. Source of truth: `scripts/minikube/t2.sh`,
pidfile trips `PORT_FORWARD_CONFLICT`. The awk pre-filter matches `kubectl`
as an argv0 or path token plus a later standalone `port-forward` token;
wrappers that only mention those words are rejected by `comm`.
+ Registered pidfiles are checked even when `ps` no longer lists a child,
+ and a successful user-facing health probe is followed by an exact
+ process/start-time/argv revalidation.
- **Do not commit** `.local-notes/`, lockfiles produced by an incidental
`npm ci`, generated ports/profile metadata, or anything secret-like. Run
`make minikube-t2-public-boundary` first.
@@ -65,7 +69,7 @@ Companion to `SKILL.md`. Source of truth: `scripts/minikube/t2.sh`,
| `DOCKER_DEADLINE_INVALID` | A runtime deadline/retry value is malformed or exceeds its finite maximum. | Remove the override or set it within the documented range; do not disable the deadline. |
| `BOOTSTRAP_REQUIRED` | Profile missing/uninitialized, or the planner produced no transition. Standalone preflight refuses to bootstrap. | Run `MINIKUBE_PROFILE=<owned> CONTROL_API_REAL_PG_CONTEXT=<owned> make minikube-t2`; its internal planner makes `full-bootstrap` reachable. |
| `HEAD_MARKER_MISMATCH` | Pre-gate marker does not match this worktree/HEAD; the final T2 preflight selected something other than `already-synced`. | Re-run the full target on this HEAD so `pre-gate-sync` updates the marker. Never hand-edit the marker. |
-| `IMAGE_MANIFEST_MISMATCH` | Deployed image provenance does not match the exact marker/worktree state. | Re-run the full target for the same owned profile; do not relabel or hand-edit the manifest. |
+| `IMAGE_MANIFEST_MISMATCH` | Deployed image provenance or the manifest `generated` stamp does not match the exact marker/worktree state. | Re-run the full target for the same owned profile; do not relabel or hand-edit the manifest. |
| `PORT_FORWARD_CONFLICT` | A port-forward for this profile is owned by a process not recorded for it (or not a real `kubectl`). | Identify the foreign owner; if it belongs to another lane, stop. Restart forwards via `scripts/minikube/pf-all-stack.sh` for this profile only. |
| `PROFILE_BUSY` | Profile lock held. Live owner PID → genuinely busy. No valid owner PID → orphaned lock. | Live owner: wait or coordinate; never remove. Orphan: verify no T2 process owns the profile, then remove ONLY `$T2_LOCK_ROOT/<profile>.lock` and retry. Never remove the lock root. |
| `DEVELOPMENT_SCOPE_REQUIRED` | Preflight/final preflight failed a precondition, or T2-only mode was attempted without `already-synced`. | Repair the first reported condition; if T2-only was refused, run full `make minikube-t2`. |
diff --git a/AGENTS.md b/AGENTS.md
index 99ac28864..8036b2be2 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -20,8 +20,8 @@ Before reusing a profile, record and compare all of the following:
- active worktree path, branch, `HEAD`, and `origin/dev`;
- target Minikube profile and explicit Kubernetes context;
-- the pre-gate state marker: `worktreeId`, `gitHead`, and
- `clusterFingerprint`;
+- the pre-gate state marker: `worktreeId`, `gitHead`, `clusterFingerprint`,
+ and the exact `imagesGeneratedAt` value from the image manifest;
- profile status and currently running profile, pre-gate-sync, and
port-forward processes.
@@ -36,12 +36,17 @@ Port-forwards are owned by atomic `0600` records bound to the exact profile,
context, canonical worktree, namespace, Service, local/remote ports, PID,
process start time, and `kubectl` argv. Never kill or adopt a live legacy
`/tmp/pf-*.pid` process: it lacks enough provenance and must fail closed.
-
-All image builds and targeted deploys must enter through the documented Make
-targets or the T2 orchestrator so they inherit the exact live profile mutation
-lease. Calling `scripts/minikube/build-images.sh` directly is unsupported for
-mutating work; `--verify-only` is the read-only exception. Docker and Minikube
-operations must use finite deadlines. Docker endpoint discovery must resolve
+After a user-facing health probe, revalidate the same live process, start time,
+argv, and binding. A registered but dead/reused/ambiguous pidfile is itself a
+`PORT_FORWARD_CONFLICT`; it must not disappear from the T2 verdict merely
+because `ps` no longer lists a child.
+
+All image acquisition, image builds, and targeted deploys must enter through
+the documented Make targets or the T2 orchestrator so they inherit the exact
+live profile mutation lease. Calling `scripts/minikube/build-images.sh` or
+`scripts/minikube/pull-images.sh` directly is unsupported for mutating work;
+`--verify-only` is the read-only exception. Docker and Minikube operations must
+use finite deadlines. Docker endpoint discovery must resolve
to an explicit local Unix socket or loopback TCP endpoint before switching to
an empty task-local Docker config. Ambient registry credentials are never
copied; a private pull requires an explicit `MINIKUBE_DOCKER_AUTH_CONFIG`.
@@ -130,6 +135,10 @@ reader readiness through the `gfs-rollout-shim` PATH prefix instead of a
generation-based `rollout status`, because HCC's gfsReconciler strips the
`restartedAt` annotation and makes that wait time out. The standalone preflight and
the final exact-head T2 check stay fail-loud on an unready deployment.
+When REUSE_DB recovery is needed, the fence covers all four database writers:
+HCC, workflow-recipes, trace-maintenance-worker, and control-api. The durable
+recovery identity is the owned profile/context/worktree/branch; the interrupted
+HEAD is retained as audit data while the exact-head marker controls freshness.
Real PostgreSQL suites are opt-in in the ordinary test
matrix, but a T1 run that requires them must fail when the database/DSN is
unavailable or when zero tests execute; a green run must never be produced by
diff --git a/Makefile b/Makefile
index cd25bc0c8..47f48360e 100644
--- a/Makefile
+++ b/Makefile
@@ -201,9 +201,20 @@ minikube-setup-e2e: ## Full setup + E2E fixtures (test user, e2e-* recipes, demo
minikube-teardown: ## Remove deployments (keep namespaces/CRDs)
@scripts/minikube/teardown.sh
-.PHONY: minikube-pull-images
+.PHONY: minikube-pull-images minikube-pull-images-body
minikube-pull-images: ## Pull ALL published images into minikube at the pinned release tag (MINIKUBE_IMAGE_TAG=<tag> overrides the pin for this run only)
- @MINIKUBE_PROFILE="$(MINIKUBE_PROFILE)" MINIKUBE_IMAGE_TAG="$(MINIKUBE_IMAGE_TAG)" \
+ @T2_PROJECT_DIR="$(CURDIR)" T2_PROFILE="$(MINIKUBE_PROFILE)" T2_CONTEXT="$(MINIKUBE_PROFILE)" \
+ T2_SKIP_LOCK="$(T2_SKIP_LOCK)" T2_LOCK_TOKEN="$(T2_LOCK_TOKEN)" \
+ bash scripts/minikube/with-t2-mutation-lock.sh -- \
+ $(MAKE) --no-print-directory minikube-pull-images-body
+
+minikube-pull-images-body:
+ @T2_PROJECT_DIR="$(CURDIR)" T2_PROFILE="$(MINIKUBE_PROFILE)" T2_CONTEXT="$(MINIKUBE_PROFILE)" \
+ T2_SKIP_LOCK=true T2_LOCK_TOKEN="$(T2_LOCK_TOKEN)" \
+ bash scripts/minikube/require-t2-mutation-lock.sh
+ @T2_PROJECT_DIR="$(CURDIR)" T2_PROFILE="$(MINIKUBE_PROFILE)" T2_CONTEXT="$(MINIKUBE_PROFILE)" \
+ MINIKUBE_PROFILE="$(MINIKUBE_PROFILE)" MINIKUBE_IMAGE_TAG="$(MINIKUBE_IMAGE_TAG)" \
+ CONTROL_API_REAL_PG_CONTEXT="$(MINIKUBE_PROFILE)" \
scripts/minikube/pull-images.sh
.PHONY: minikube-build-images minikube-build-images-body
diff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.md
index a9408c675..bad0b5504 100644
--- a/docs/testing/minikube-t2-runbook.md
+++ b/docs/testing/minikube-t2-runbook.md
@@ -105,12 +105,14 @@ inside that exact profile lock. A concurrent loser fails closed and never
removes either the stale directory or the winner's replacement lock.
The pre-gate marker must contain the current worktree identifier, exact `HEAD`,
-cluster fingerprint, and image coordinate. A mismatch stops with a stable
-error code instead of allowing a mixed-commit run.
-
-Mutating image builds and targeted deploys are children of that same exact
-profile lease. Public Make targets acquire it; private body targets and
-`build-images.sh` validate the inherited token again before the first Docker,
+cluster fingerprint, image coordinate, and the exact `imagesGeneratedAt` value
+from the image manifest. A mismatch—including a new image acquisition at the
+same HEAD—stops with a stable error code instead of allowing a mixed-commit run.
+
+Mutating image acquisition/builds and targeted deploys are children of that
+same exact profile lease. Public Make targets acquire it; private body targets,
+`pull-images.sh`, and `build-images.sh` validate the inherited token again
+before the first Docker,
Minikube, or Kubernetes operation. `build-images.sh --verify-only` is the
read-only exception. Empty/unknown selectors fail before the lease or runtime
is touched.
@@ -142,7 +144,9 @@ context, canonical worktree, Service, and port bindings. Each live process
must have exactly one atomic `0600` ownership record under
`$HOME/.cache/clerum/minikube-profiles/<profile>/pids/`. A live legacy
`/tmp/pf-<profile>-*.pid` record cannot be adopted or killed because it lacks
-the full binding; dead legacy records may be pruned.
+the full binding; dead legacy records may be pruned. Registered pidfiles are
+also checked when `ps` no longer lists a child, and a successful user-facing
+health probe is followed by exact process/start-time/argv revalidation.
`make minikube-t2` invokes `pre-gate-sync` with `--skip-port-forwards` so the
orchestrator does not plant forwards that fail its own T2 check. Secret values
are never printed.
@@ -302,6 +306,12 @@ refused. An interrupted bootstrap leaves the profile intact and can be safely
retried after the reported prerequisite is repaired. Do not reuse evidence
from a different `HEAD` or profile.
+REUSE_DB recovery fences all four database writers—HCC, workflow-recipes,
+trace-maintenance-worker, and control-api—before migrations, role provisioning,
+or the recovery overlay. The interrupted `HEAD` remains historical state for
+the owned profile/context/worktree/branch; exact-head freshness is decided by
+the pre-gate marker and image stamp.
+
Retry by phase. During a T1 failure, iterate with
`minikube-t2-real-postgres`, then run one full `minikube-t2` certification once
green. After exact-head T0/T1 lane evidence is already `PASS`, a failure in
diff --git a/scripts/minikube/full-setup.sh b/scripts/minikube/full-setup.sh
index 34b56f2f0..0e140367c 100755
--- a/scripts/minikube/full-setup.sh
+++ b/scripts/minikube/full-setup.sh
@@ -1118,9 +1118,11 @@ PARTIAL_BOOTSTRAP_RECOVERY=false
PARTIAL_CONTROL_API_REPLICAS=""
PARTIAL_WORKFLOW_REPLICAS=""
PARTIAL_HCC_REPLICAS=""
+PARTIAL_TRACE_REPLICAS=""
PARTIAL_CONTROL_API_FENCED=false
PARTIAL_WORKFLOW_FENCED=false
PARTIAL_HCC_FENCED=false
+PARTIAL_TRACE_FENCED=false
WRITER_RECOVERY_STATE_HELPER="${SCRIPT_DIR}/writer-recovery-state.py"
WRITER_RECOVERY_STATE_ROOT="${PROJECT_DIR}/.local-notes/infra/runs/writer-recovery"
WRITER_RECOVERY_STATE_FILE="${WRITER_RECOVERY_STATE_ROOT}/${PROFILE}.json"
@@ -1150,6 +1152,7 @@ writer_recovery_state_write() {
--phase "$phase" \
--hcc "$PARTIAL_HCC_REPLICAS" \
--workflow "$PARTIAL_WORKFLOW_REPLICAS" \
+ --trace "$PARTIAL_TRACE_REPLICAS" \
--control-api "$PARTIAL_CONTROL_API_REPLICAS"
}
@@ -1177,10 +1180,12 @@ writer_recovery_state_load() {
return 0
fi
IFS='|' read -r WRITER_RECOVERY_STATE_PHASE PARTIAL_HCC_REPLICAS \
- PARTIAL_WORKFLOW_REPLICAS PARTIAL_CONTROL_API_REPLICAS <<<"$state_output"
+ PARTIAL_WORKFLOW_REPLICAS PARTIAL_TRACE_REPLICAS \
+ PARTIAL_CONTROL_API_REPLICAS <<<"$state_output"
[ -n "$WRITER_RECOVERY_STATE_PHASE" ] &&
[[ "$PARTIAL_HCC_REPLICAS" =~ ^[0-9]+$ ]] &&
[[ "$PARTIAL_WORKFLOW_REPLICAS" =~ ^[0-9]+$ ]] &&
+ [[ "$PARTIAL_TRACE_REPLICAS" =~ ^[0-9]+$ ]] &&
[[ "$PARTIAL_CONTROL_API_REPLICAS" =~ ^[1-9][0-9]*$ ]] || {
err "Durable writer-recovery state is incomplete; refusing profile mutation"
return 1
@@ -1204,11 +1209,20 @@ writer_recovery_state_load() {
PARTIAL_WORKFLOW_FENCED=true
WRITER_RECOVERY_FENCE_PENDING=true
;;
+ trace-fencing|trace-fenced)
+ WRITER_RECOVERY=true
+ PARTIAL_BOOTSTRAP_RECOVERY=true
+ PARTIAL_HCC_FENCED=true
+ PARTIAL_WORKFLOW_FENCED=true
+ PARTIAL_TRACE_FENCED=true
+ WRITER_RECOVERY_FENCE_PENDING=true
+ ;;
api-fencing)
WRITER_RECOVERY=true
PARTIAL_BOOTSTRAP_RECOVERY=true
PARTIAL_HCC_FENCED=true
PARTIAL_WORKFLOW_FENCED=true
+ PARTIAL_TRACE_FENCED=true
PARTIAL_CONTROL_API_FENCED=true
WRITER_RECOVERY_FENCE_PENDING=true
;;
@@ -1217,6 +1231,7 @@ writer_recovery_state_load() {
PARTIAL_BOOTSTRAP_RECOVERY=true
PARTIAL_HCC_FENCED=true
PARTIAL_WORKFLOW_FENCED=true
+ PARTIAL_TRACE_FENCED=true
PARTIAL_CONTROL_API_FENCED=true
WRITER_RECOVERY_FENCE_PENDING=true
;;
@@ -1225,6 +1240,7 @@ writer_recovery_state_load() {
PARTIAL_BOOTSTRAP_RECOVERY=true
PARTIAL_HCC_FENCED=true
PARTIAL_WORKFLOW_FENCED=true
+ PARTIAL_TRACE_FENCED=true
PARTIAL_CONTROL_API_FENCED=true
# A persisted policy phase is historical evidence only. Re-fence and
# restage the current rendered policy before any writer is restored so a
@@ -1236,6 +1252,7 @@ writer_recovery_state_load() {
PARTIAL_BOOTSTRAP_RECOVERY=true
PARTIAL_HCC_FENCED=true
PARTIAL_WORKFLOW_FENCED=true
+ PARTIAL_TRACE_FENCED=true
PARTIAL_CONTROL_API_FENCED=true
WRITER_RECOVERY_MIGRATIONS_COMPLETE=true
# The phase is written before the control-api restore. An interruption
@@ -1249,7 +1266,7 @@ writer_recovery_state_load() {
PARTIAL_WORKFLOW_FENCED=true
PARTIAL_CONTROL_API_FENCED=true
WRITER_RECOVERY_MIGRATIONS_COMPLETE=true
- # The overlay is rendered with all three writer Deployments at zero.
+ # The overlay is rendered with all four database-writer Deployments at zero.
# Re-fence on every post-migration resume because an interrupted apply
# may have recreated a writer before its durable phase was updated.
WRITER_RECOVERY_FENCE_PENDING=true
@@ -1273,11 +1290,15 @@ writer_recovery_state_prepare() {
state="$($KC -n control-plane get deployment/workflow-recipes \
-o 'jsonpath={.spec.replicas}')" || return 1
PARTIAL_WORKFLOW_REPLICAS="$state"
+ state="$($KC -n control-plane get deployment/trace-maintenance-worker \
+ -o 'jsonpath={.spec.replicas}')" || return 1
+ PARTIAL_TRACE_REPLICAS="$state"
state="$($KC -n control-plane get deployment/control-api \
-o 'jsonpath={.spec.replicas}')" || return 1
PARTIAL_CONTROL_API_REPLICAS="$state"
[[ "$PARTIAL_HCC_REPLICAS" =~ ^[0-9]+$ ]] || return 1
[[ "$PARTIAL_WORKFLOW_REPLICAS" =~ ^[0-9]+$ ]] || return 1
+ [[ "$PARTIAL_TRACE_REPLICAS" =~ ^[0-9]+$ ]] || return 1
[[ "$PARTIAL_CONTROL_API_REPLICAS" =~ ^[1-9][0-9]*$ ]] || return 1
writer_recovery_state_phase planned
}
@@ -1350,6 +1371,25 @@ fence_partial_workflow_reconciler() {
writer_recovery_state_phase workflow-fenced
}
+fence_partial_trace_worker() {
+ local pods
+ if [[ ! "${PARTIAL_TRACE_REPLICAS}" =~ ^[0-9]+$ ]]; then
+ err "trace-maintenance-worker replica count is invalid; refusing partial-bootstrap recovery"
+ return 1
+ fi
+ log "Fencing trace-maintenance-worker at ${PARTIAL_TRACE_REPLICAS} replica(s)"
+ writer_recovery_state_phase trace-fencing
+ $KC -n control-plane scale deployment/trace-maintenance-worker --replicas="${PARTIAL_TRACE_REPLICAS}" >/dev/null
+ pods="$($KC -n control-plane get pods -l app=trace-maintenance-worker -o name)" \
+ || return 1
+ if [ -n "${pods}" ]; then
+ $KC -n control-plane wait --for=delete pod \
+ -l app=trace-maintenance-worker --timeout=180s >/dev/null
+ fi
+ PARTIAL_TRACE_FENCED=true
+ writer_recovery_state_phase trace-fenced
+}
+
fence_partial_hcc() {
local pods
if [[ ! "$PARTIAL_HCC_REPLICAS" =~ ^[0-9]+$ ]]; then
@@ -1376,6 +1416,7 @@ fence_partial_bootstrap_writers() {
writer_recovery_state_prepare
fence_partial_hcc
fence_partial_workflow_reconciler
+ fence_partial_trace_worker
fence_partial_control_api
}
@@ -1501,12 +1542,17 @@ restore_partial_control_api() {
restore_partial_non_api_writers() {
if [ "$PARTIAL_WORKFLOW_FENCED" = true ]; then
$KC -n control-plane scale deployment/workflow-recipes \
- --replicas="$PARTIAL_WORKFLOW_REPLICAS" >/dev/null
+ --replicas="$PARTIAL_WORKFLOW_REPLICAS" >/dev/null || return 1
PARTIAL_WORKFLOW_FENCED=false
fi
+ if [ "$PARTIAL_TRACE_FENCED" = true ]; then
+ $KC -n control-plane scale deployment/trace-maintenance-worker \
+ --replicas="$PARTIAL_TRACE_REPLICAS" >/dev/null || return 1
+ PARTIAL_TRACE_FENCED=false
+ fi
if [ "$PARTIAL_HCC_FENCED" = true ]; then
$KC -n control-plane scale deployment/host-context-controller \
- --replicas="$PARTIAL_HCC_REPLICAS" >/dev/null
+ --replicas="$PARTIAL_HCC_REPLICAS" >/dev/null || return 1
PARTIAL_HCC_FENCED=false
fi
}
@@ -1539,11 +1585,12 @@ render_fenced_recovery_overlay() {
err "Recovery overlay workspace is unavailable; refusing writer recovery"
return 1
}
- log "Rendering the recovery overlay with HCC, workflow-recipes, and control-api fenced"
+ log "Rendering the recovery overlay with HCC, workflow-recipes, trace-maintenance-worker, and control-api fenced"
if ! $KC kustomize "$ACTIVE_MINIKUBE_RENDER_DIR" | \
RUBYOPT=--disable=gems ruby "${SCRIPT_DIR}/render-fenced-writer-deployments.rb" \
--target control-plane/host-context-controller \
--target control-plane/workflow-recipes \
+ --target control-plane/trace-maintenance-worker \
--target control-plane/control-api >"$rendered_file"; then
err "Unable to render a complete fenced recovery overlay; refusing writer recovery"
return 1
@@ -1591,13 +1638,14 @@ else
WRITER_RECOVERY=true
PARTIAL_BOOTSTRAP_RECOVERY=true
log "Existing GFS writer and empty control-api runtime Secret detected — fencing writers until migrations and runtime roles converge"
- elif [ "$runtime_secret_status" -eq 1 ] && [ "$K8S_API_EGRESS_POLICY_DRIFT" = true ]; then
+ elif [ "$runtime_secret_status" -eq 1 ]; then
WRITER_RECOVERY=true
PARTIAL_BOOTSTRAP_RECOVERY=true
- log "Existing GFS writer detected with control-api not Ready — repairing the stale API-egress policy before the fail-closed readiness decision"
- elif [ "$runtime_secret_status" -eq 1 ]; then
- err "Existing GFS writer detected but control-api is not Ready; refusing HCC cutover"
- exit 1
+ if [ "$K8S_API_EGRESS_POLICY_DRIFT" = true ]; then
+ log "Existing GFS writer detected with control-api not Ready — repairing the stale API-egress policy before runtime-role convergence"
+ else
+ log "Existing GFS writer detected with control-api not Ready — fencing all database writers before runtime-role convergence"
+ fi
else
err "Unable to inspect control-api-postgres-runtime; refusing to classify the GFS upgrade path"
exit 1
@@ -1679,7 +1727,7 @@ else
fi
if [ "$WRITER_RECOVERY" = true ]; then
- # Keep every writer at zero while the full overlay is applied. The overlay
+ # Keep every database writer at zero while the full overlay is applied. The overlay
# itself declares one replica for these Deployments, so applying it
# directly would reopen the writer window before the explicit restore.
apply_fenced_recovery_overlay
diff --git a/scripts/minikube/pf-all-stack.sh b/scripts/minikube/pf-all-stack.sh
index 74851d923..adf77037e 100755
--- a/scripts/minikube/pf-all-stack.sh
+++ b/scripts/minikube/pf-all-stack.sh
@@ -261,6 +261,15 @@ start_pf() {
for ((attempt = 0; attempt < HEALTH_ATTEMPTS; attempt += 1)); do
if curl -sf -m 2 "${health_url}" >/dev/null 2>&1; then
+ # Health can succeed in the same small interval in which a forward
+ # exits or its PID is reused. Revalidate the exact recorded process and
+ # binding after the user-facing probe before publishing success.
+ if ! pf_owner_record_process_matches "${profile_pidfile}" "${PROFILE}" \
+ "${CONTEXT}" "${WORKTREE_ROOT}" "${namespace}" "${service}" \
+ "${local_port}" "${remote_port}"; then
+ echo " ERROR: ${name} port-forward ownership was lost during health verification" >&2
+ return 1
+ fi
return 0
fi
pf_owner_pause "${HEALTH_DELAY}"
diff --git a/scripts/minikube/port-forward-owner.sh b/scripts/minikube/port-forward-owner.sh
index ba51cf92b..0a82716c8 100755
--- a/scripts/minikube/port-forward-owner.sh
+++ b/scripts/minikube/port-forward-owner.sh
@@ -268,6 +268,26 @@ pf_owner_record_matches() {
"${PF_OWNER_RECORD_REMOTE_PORT}" == "${remote_port}" ]]
}
+pf_owner_record_process_matches() {
+ local pidfile="$1" profile="$2" context="$3" worktree="$4"
+ local namespace="$5" service="$6" local_port="$7" remote_port="$8"
+ local pid state actual_start command_line
+
+ pf_owner_read_record "$pidfile" || return 1
+ pf_owner_record_matches "${profile}" "${context}" "${worktree}" \
+ "${namespace}" "${service}" "${local_port}" "${remote_port}" || return 1
+ pid="${PF_OWNER_RECORD_PID}"
+ state="$(pf_owner_process_state "${pid}")"
+ [[ "${state}" == live ]] || return 1
+ actual_start="$(pf_owner_process_start "${pid}")" || return 1
+ [[ "${actual_start}" == "${PF_OWNER_RECORD_START}" ]] || return 1
+ command_line="$(pf_owner_process_command "${pid}")" || return 1
+ pf_owner_command_matches "${command_line}" "${context}" "${namespace}" \
+ "${service}" "${local_port}" "${remote_port}" || return 1
+ actual_start="$(pf_owner_process_start "${pid}")" || return 1
+ [[ "${actual_start}" == "${PF_OWNER_RECORD_START}" ]] || return 1
+}
+
pf_owner_remove_dead_record() {
local pidfile="$1"
[[ ! -L "${pidfile}" ]] || pf_owner_error "refusing to remove symlinked pidfile: ${pidfile}" || return 1
diff --git a/scripts/minikube/pull-images.sh b/scripts/minikube/pull-images.sh
index f4856e5b5..afa99b748 100755
--- a/scripts/minikube/pull-images.sh
+++ b/scripts/minikube/pull-images.sh
@@ -75,6 +75,7 @@ MINIKUBE_MULTI_NODE="${MINIKUBE_MULTI_NODE:-false}"
PULL_PARALLELISM="${MINIKUBE_PULL_PARALLELISM:-6}"
MINIKUBE_IMAGE_PULL_RETRIES="${MINIKUBE_IMAGE_PULL_RETRIES:-3}"
MINIKUBE_IMAGE_PULL_DELAY_SECS="${MINIKUBE_IMAGE_PULL_DELAY_SECS:-5}"
+MINIKUBE_DOCKER_ENV_TIMEOUT_SECONDS="${MINIKUBE_DOCKER_ENV_TIMEOUT_SECONDS:-30}"
GHCR_NAMESPACE="ghcr.io/evenfire-ai"
ONLY_SVC=""
@@ -121,6 +122,22 @@ err() { echo -e "${RED} ERROR${NC} -- $*" >&2; }
# record says "local" must still resolve a tag instead of an empty string.
# shellcheck source=scripts/minikube/image-mode.sh
source "${SCRIPT_DIR}/image-mode.sh"
+# shellcheck source=scripts/minikube/docker-cli-env.sh
+source "${SCRIPT_DIR}/docker-cli-env.sh"
+
+# Pulling and loading images mutates the branch-owned Minikube profile. The
+# caller must already hold the canonical T2 mutation lease; this child only
+# validates the opaque inherited token and never acquires or releases it.
+if [[ -z "${T2_PROJECT_DIR:-}" || -z "${T2_PROFILE:-}" ||
+ -z "${T2_CONTEXT:-}" || "${T2_PROFILE}" != "${T2_CONTEXT}" ||
+ "${T2_PROFILE}" != "${PROFILE}" ]]; then
+ err "PROFILE_LOCK_REQUIRED: pull-images.sh requires an inherited branch-profile mutation lease"
+ exit 1
+fi
+T2_PROJECT_DIR="${T2_PROJECT_DIR}" T2_PROFILE="${T2_PROFILE}" \
+ T2_CONTEXT="${T2_CONTEXT}" T2_SKIP_LOCK=true \
+ MINIKUBE_PROFILE="${PROFILE}" CONTROL_API_REAL_PG_CONTEXT="${T2_CONTEXT}" \
+ bash "${SCRIPT_DIR}/require-t2-mutation-lock.sh"
IMAGE_TAG="$(image_mode_ghcr_tag "$PROJECT_DIR")" || exit 1
TAG_ORIGIN="$(image_mode_tag_origin "$PROJECT_DIR")" || exit 1
@@ -155,19 +172,42 @@ if [ "$SKIP_UIS" = true ]; then
log "skipping control-ui and profile-ui (--skip-uis); the -no-uis overlay deletes both Deployments"
fi
-# ---- Point the Docker CLI at minikube's daemon when safe -----------------
+STATUS_DIR="$(mktemp -d)"
+cleanup() {
+ local status=$? cleanup_status=0
+ trap - EXIT INT TERM
+ rm -rf -- "$STATUS_DIR" || cleanup_status=$?
+ docker_cli_env_cleanup || cleanup_status=$?
+ if [ "$status" -eq 0 ] && [ "$cleanup_status" -ne 0 ]; then
+ status="$cleanup_status"
+ fi
+ exit "$status"
+}
+trap cleanup EXIT
+trap 'exit 130' INT
+trap 'exit 143' TERM
+
+# Resolve an approved local endpoint into an empty task-local Docker config
+# before asking Minikube for its daemon endpoint. Every subsequent Docker and
+# image-load operation then runs through the same isolated config and deadline
+# runner, with ambient auth/context/header variables excluded.
+docker_cli_env_prepare false
if [ "$MINIKUBE_MULTI_NODE" = false ]; then
log "Configuring Docker CLI to use minikube's Docker daemon..."
- eval "$(minikube -p "$PROFILE" docker-env)"
+ docker_env_status=0
+ docker_env_output="$(docker_cli_run_public minikube-docker-env \
+ "$MINIKUBE_DOCKER_ENV_TIMEOUT_SECONDS" \
+ minikube -p "$PROFILE" docker-env --shell bash)" || docker_env_status=$?
+ if [ "$docker_env_status" -ne 0 ] || [ -z "$docker_env_output" ]; then
+ err "Could not resolve Docker environment for minikube '${PROFILE}'"
+ exit "${docker_env_status:-1}"
+ fi
+ eval "$docker_env_output"
unset DOCKER_API_VERSION 2>/dev/null || true
else
log "Multi-node profile: pulling on the host, then 'minikube image load' onto every node"
fi
-STATUS_DIR="$(mktemp -d)"
-cleanup() { rm -rf "$STATUS_DIR"; }
-trap cleanup EXIT
-
# ---- Bounded retry for one pull ------------------------------------------
# A transient registry blip (one bad pull 24 images into a run) should not
# read the same as a tag that genuinely does not exist. Mirrors
@@ -180,10 +220,11 @@ trap cleanup EXIT
# final failure it holds only the last attempt's diagnostic, not a pile-up of
# every prior one.
pull_with_retry() {
- local ref="$1" out_file="$2"
+ local ref="$1" out_file="$2" image_name="$3"
local attempt=1
while [ "$attempt" -le "$MINIKUBE_IMAGE_PULL_RETRIES" ]; do
- if docker pull "$ref" >"$out_file" 2>&1; then
+ if docker_cli_run_public "pull-image-${image_name}" \
+ "$MINIKUBE_DOCKER_PULL_TIMEOUT_SECONDS" docker pull "$ref" >"$out_file" 2>&1; then
return 0
fi
if [ "$attempt" -lt "$MINIKUBE_IMAGE_PULL_RETRIES" ]; then
@@ -201,25 +242,31 @@ pull_one() {
local ghcr_ref="${GHCR_NAMESPACE}/${name}:${IMAGE_TAG}"
# Always pull. Never skip a present tag -- see the header.
- if ! pull_with_retry "$ghcr_ref" "${STATUS_DIR}/${slot}.out"; then
+ if ! pull_with_retry "$ghcr_ref" "${STATUS_DIR}/${slot}.out" "$name"; then
printf '%s' "$ghcr_ref" > "${STATUS_DIR}/${slot}.failed"
return 0
fi
if [ "$MINIKUBE_MULTI_NODE" = true ]; then
- minikube -p "$PROFILE" image load "$ghcr_ref" >/dev/null 2>&1 || {
+ docker_cli_run_public "minikube-image-load-${name}" \
+ "$MINIKUBE_DOCKER_BUILD_TIMEOUT_SECONDS" \
+ minikube -p "$PROFILE" image load "$ghcr_ref" >/dev/null 2>&1 || {
printf '%s' "$ghcr_ref" > "${STATUS_DIR}/${slot}.failed"
return 0
}
fi
# The local alias. See header note 2.
- if ! docker tag "$ghcr_ref" "$local_ref" >>"${STATUS_DIR}/${slot}.out" 2>&1; then
+ if ! docker_cli_run_public "tag-image-${name}" \
+ "$MINIKUBE_DOCKER_INFO_TIMEOUT_SECONDS" docker tag "$ghcr_ref" "$local_ref" \
+ >>"${STATUS_DIR}/${slot}.out" 2>&1; then
printf '%s' "$ghcr_ref (alias to ${local_ref})" > "${STATUS_DIR}/${slot}.failed"
return 0
fi
if [ "$MINIKUBE_MULTI_NODE" = true ]; then
- minikube -p "$PROFILE" image load "$local_ref" >/dev/null 2>&1 || true
+ docker_cli_run_public "minikube-image-load-alias-${name}" \
+ "$MINIKUBE_DOCKER_BUILD_TIMEOUT_SECONDS" \
+ minikube -p "$PROFILE" image load "$local_ref" >/dev/null 2>&1 || true
fi
printf '%s\t%s' "$ghcr_ref" "$local_ref" > "${STATUS_DIR}/${slot}.done"
@@ -340,7 +387,9 @@ mkdir -p "$(dirname "$MANIFEST_FILE")"
[ -e "$f" ] || continue
ghcr_ref="$(cut -f1 "$f")"
local_ref="$(cut -f2 "$f")"
- sha="$(docker inspect --format='{{.Id}}' "$ghcr_ref" 2>/dev/null || echo "NOT_PULLED")"
+ sha="$(docker_cli_run_public "inspect-image" \
+ "$MINIKUBE_DOCKER_INFO_TIMEOUT_SECONDS" \
+ docker inspect --format='{{.Id}}' "$ghcr_ref" 2>/dev/null || echo "NOT_PULLED")"
for ref in "$ghcr_ref" "$local_ref"; do
if [ "$first" = true ]; then first=false; else echo ","; fi
printf ' "%s": "%s"' "$ref" "$sha"
diff --git a/scripts/minikube/setup.sh b/scripts/minikube/setup.sh
index 48d683bd4..fae24489b 100755
--- a/scripts/minikube/setup.sh
+++ b/scripts/minikube/setup.sh
@@ -9,7 +9,7 @@
#
# Usage:
# ./scripts/minikube/setup.sh
-# ./scripts/minikube/setup.sh --build # Also build images (step 6)
+# ./scripts/minikube/setup.sh --build # Build images under the branch lease
# ======================================================================
set -euo pipefail
@@ -34,6 +34,20 @@ for arg in "$@"; do
esac
done
+# This legacy setup path mutates the profile before it reaches its optional
+# image build. Re-enter it through the canonical lease wrapper first, so a
+# failed lease check cannot leave namespaces, keys, or Secrets half-applied.
+# The wrapper marks the inherited lease; the second invocation then proceeds
+# with the exact opaque token held by the parent. Fresh/unowned profiles must
+# use the supported T2 bootstrap instead of bypassing this boundary.
+if [ "$BUILD_IMAGES" = true ] && [ "${T2_MUTATION_LOCK_WRAPPED:-false}" != true ]; then
+ exec env \
+ T2_PROJECT_DIR="$PROJECT_DIR" T2_PROFILE="$PROFILE" T2_CONTEXT="$PROFILE" \
+ MINIKUBE_PROFILE="$PROFILE" CONTROL_API_REAL_PG_CONTEXT="$PROFILE" \
+ T2_SKIP_LOCK=false T2_GATE_ID=legacy-minikube-setup \
+ "$SCRIPT_DIR/with-t2-mutation-lock.sh" -- "$SCRIPT_DIR/setup.sh" "$@"
+fi
+
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
diff --git a/scripts/minikube/t2-common.sh b/scripts/minikube/t2-common.sh
index 710a533e8..813cf8dcc 100755
--- a/scripts/minikube/t2-common.sh
+++ b/scripts/minikube/t2-common.sh
@@ -75,6 +75,8 @@ T2_MARKER_MATCHES_HEAD=false
T2_MARKER_JSON=""
T2_IMAGE_SOURCE=""
T2_IMAGE_TAG=""
+T2_MARKER_IMAGES_GENERATED_AT=""
+T2_MANIFEST_GENERATED_AT=""
T2_CLUSTER_FINGERPRINT=""
T2_WORKTREE_ID=""
T2_HEAD=""
@@ -387,7 +389,14 @@ if data.get("worktreeId") != sys.argv[2]:
raise SystemExit("ownership")
if data.get("gitHead") != sys.argv[3]:
raise SystemExit("head")
-print("\t".join([data.get("clusterFingerprint", ""), data.get("imageSource", ""), data.get("imageTag", "")]))
+if not data.get("imagesGeneratedAt"):
+ raise SystemExit("missing:imagesGeneratedAt")
+print("\t".join([
+ data.get("clusterFingerprint", ""),
+ data.get("imageSource", ""),
+ data.get("imageTag", ""),
+ data.get("imagesGeneratedAt", ""),
+]))
PY
)"; then
case "$marker_values" in
@@ -408,7 +417,8 @@ PY
esac
fi
T2_MARKER_MATCHES_HEAD=true
- IFS=$'\t' read -r T2_CLUSTER_FINGERPRINT T2_IMAGE_SOURCE T2_IMAGE_TAG <<< "$marker_values"
+ IFS=$'\t' read -r T2_CLUSTER_FINGERPRINT T2_IMAGE_SOURCE T2_IMAGE_TAG \
+ T2_MARKER_IMAGES_GENERATED_AT <<< "$marker_values"
}
t2_image_check() {
@@ -432,9 +442,12 @@ except (OSError, ValueError):
raise SystemExit("invalid")
source = payload.get("imageSource") or payload.get("source") or payload.get("mode") or ""
tag = payload.get("imageTag") or payload.get("tag") or ""
+generated = payload.get("generated")
# Local builds are identified by the per-image digests in the manifest and
# intentionally have no registry tag. GHCR manifests still require a tag.
images = payload.get("images")
+if not isinstance(generated, str) or not generated:
+ raise SystemExit("generated")
if source not in {"local", "ghcr"} or (source == "ghcr" and not tag):
raise SystemExit("missing")
if source == "local":
@@ -444,7 +457,7 @@ if source == "local":
digest = re.compile(r"^sha256:[0-9a-fA-F]{64}$")
if any(not isinstance(name, str) or not name or not isinstance(value, str) or not digest.fullmatch(value) for name, value in images.items()):
raise SystemExit("local-digests")
-print(source + "\t" + tag)
+print(source + "\t" + tag + "\t" + generated)
PY
)"; then
if [ "$T2_BOOTSTRAP_REQUIRED" = true ]; then
@@ -455,8 +468,8 @@ PY
T2_NEXT_COMMAND="MINIKUBE_PROFILE=$T2_PROFILE make minikube-setup-local"
t2_fail IMAGE_MANIFEST_MISMATCH 'image manifest is invalid or incomplete for its image source'
fi
- local manifest_source manifest_tag
- IFS=$'\t' read -r manifest_source manifest_tag <<< "$manifest_values"
+ local manifest_source manifest_tag manifest_generated
+ IFS=$'\t' read -r manifest_source manifest_tag manifest_generated <<< "$manifest_values"
if { [ -n "$T2_IMAGE_SOURCE" ] && [ "$manifest_source" != "$T2_IMAGE_SOURCE" ]; } ||
{ [ -n "$T2_IMAGE_TAG" ] && [ "$manifest_tag" != "$T2_IMAGE_TAG" ]; }; then
T2_NEXT_COMMAND="MINIKUBE_PROFILE=$T2_PROFILE make minikube-setup-local"
@@ -464,6 +477,15 @@ PY
fi
T2_IMAGE_SOURCE="$manifest_source"
T2_IMAGE_TAG="$manifest_tag"
+ T2_MANIFEST_GENERATED_AT="$manifest_generated"
+ if [ "$T2_MARKER_MATCHES_HEAD" = true ] &&
+ [ "$T2_MARKER_IMAGES_GENERATED_AT" != "$T2_MANIFEST_GENERATED_AT" ]; then
+ # A new image acquisition can keep the same Git HEAD while replacing the
+ # release/shadow digest on the cluster. Refuse to call that already-synced
+ # and let the orchestrator select a fresh targeted/full transition.
+ T2_MARKER_MATCHES_HEAD=false
+ T2_PLAN_REASON='image acquisition changed since the pre-gate marker'
+ fi
}
t2_get_name() {
@@ -691,25 +713,30 @@ t2_port_forward_targets_context() {
}
t2_pid_file_matches_process() {
- local pid_file="$1" pid="$2" actual_start command_line
+ local pid_file="$1" pid="$2"
pf_owner_read_record "$pid_file" || return 1
[ "$PF_OWNER_RECORD_FIRST_PID" = "$pid" ] || return 1
- [ "$PF_OWNER_RECORD_PROFILE" = "$T2_PROFILE" ] || return 1
- [ "$PF_OWNER_RECORD_CONTEXT" = "$T2_CONTEXT" ] || return 1
- [ "$PF_OWNER_RECORD_WORKTREE" = "$T2_PROJECT_DIR" ] || return 1
- actual_start="$(pf_owner_process_start "$pid")" || return 1
- [ "$actual_start" = "$PF_OWNER_RECORD_START" ] || return 1
- command_line="$(pf_owner_process_command "$pid")" || return 1
- pf_owner_command_matches "$command_line" "$PF_OWNER_RECORD_CONTEXT" \
- "$PF_OWNER_RECORD_NAMESPACE" "$PF_OWNER_RECORD_SERVICE" \
- "$PF_OWNER_RECORD_LOCAL_PORT" "$PF_OWNER_RECORD_REMOTE_PORT" || return 1
- actual_start="$(pf_owner_process_start "$pid")" || return 1
- [ "$actual_start" = "$PF_OWNER_RECORD_START" ]
+ pf_owner_record_process_matches "$pid_file" "$T2_PROFILE" "$T2_CONTEXT" \
+ "$T2_PROJECT_DIR" "$PF_OWNER_RECORD_NAMESPACE" "$PF_OWNER_RECORD_SERVICE" \
+ "$PF_OWNER_RECORD_LOCAL_PORT" "$PF_OWNER_RECORD_REMOTE_PORT"
}
t2_process_check() {
local uid pid ppid rest command_line comm pid_file recorded_pid
local matching_records invalid_record
+ # A dead or reused registered forward is still an ownership failure even
+ # when ps(1) no longer lists a kubectl process. Scan the profile's durable
+ # records first so a stale pidfile cannot disappear from the verdict.
+ for pid_file in "$T2_PROFILE_ROOT/$T2_PROFILE"/pids/*.pid; do
+ [ -f "$pid_file" ] || continue
+ recorded_pid="$(sed -n '1p' "$pid_file" 2>/dev/null || true)"
+ if ! pf_owner_validate_pid "$recorded_pid" ||
+ ! t2_pid_file_matches_process "$pid_file" "$recorded_pid"; then
+ T2_NEXT_COMMAND='repair the exact profile-owned port-forward record before rerunning T2'
+ t2_fail PORT_FORWARD_CONFLICT "registered port-forward ownership is stale or invalid: $pid_file"
+ return 1
+ fi
+ done
# Only real kubectl port-forward processes. A wrapper whose argv merely
# mentions those words is not a port-forward (rejected by exact argv0/comm).
# Default IFS so UID/PID/PPID split. `IFS=` left pid empty and skipped every line.
diff --git a/scripts/minikube/writer-recovery-state.py b/scripts/minikube/writer-recovery-state.py
index 200111f9c..5932c6515 100755
--- a/scripts/minikube/writer-recovery-state.py
+++ b/scripts/minikube/writer-recovery-state.py
@@ -15,13 +15,15 @@ import tempfile
from pathlib import Path
-VERSION = 1
+VERSION = 2
PHASES = {
"planned",
"hcc-fencing",
"hcc-fenced",
"workflow-fencing",
"workflow-fenced",
+ "trace-fencing",
+ "trace-fenced",
"api-fencing",
"api-fenced",
"policy-ready",
@@ -79,6 +81,7 @@ def write_state(args: argparse.Namespace) -> None:
"phase": args.phase,
"hccReplicas": validate_replica(args.hcc, "hcc", allow_zero=True),
"workflowReplicas": validate_replica(args.workflow, "workflow", allow_zero=True),
+ "traceReplicas": validate_replica(args.trace, "trace-maintenance-worker", allow_zero=True),
"controlApiReplicas": validate_replica(args.control_api, "control-api"),
}
path = state_path(args)
@@ -113,16 +116,31 @@ def read_state(args: argparse.Namespace) -> None:
fail(f"cannot read state: {error}")
if not isinstance(data, dict) or data.get("version") != VERSION:
fail("state version is unsupported")
+ # The recorded head is historical evidence of the interrupted run. The
+ # durable lane identity is profile/context/worktree/branch; a new commit
+ # on the same owned lane must be able to resume and revalidate freshness
+ # through the exact-head pre-gate marker instead of losing state solely
+ # because HEAD advanced.
for key, expected in identity.items():
+ if key == "head":
+ continue
if data.get(key) != expected:
fail(f"state identity mismatch for {key}; refusing to touch another lane")
+ stored_head = data.get("head")
+ if not isinstance(stored_head, str) or not stored_head or any(
+ char in stored_head for char in "\r\n\t"
+ ):
+ fail("state historical HEAD is invalid")
phase = data.get("phase")
if phase not in PHASES:
fail("state phase is invalid")
hcc = validate_replica(str(data.get("hccReplicas")), "hcc", allow_zero=True)
workflow = validate_replica(str(data.get("workflowReplicas")), "workflow", allow_zero=True)
+ trace = validate_replica(
+ str(data.get("traceReplicas")), "trace-maintenance-worker", allow_zero=True
+ )
control_api = validate_replica(str(data.get("controlApiReplicas")), "control-api")
- print(f"{phase}|{hcc}|{workflow}|{control_api}")
+ print(f"{phase}|{hcc}|{workflow}|{trace}|{control_api}")
def clear_state(args: argparse.Namespace) -> None:
@@ -147,6 +165,7 @@ def parser() -> argparse.ArgumentParser:
write.add_argument("--phase", required=True)
write.add_argument("--hcc", required=True)
write.add_argument("--workflow", required=True)
+ write.add_argument("--trace", required=True)
write.add_argument("--control-api", required=True)
subparsers.add_parser("read", parents=[common])
subparsers.add_parser("clear", parents=[common])
diff --git a/scripts/tests/test-minikube-build-section-headers.sh b/scripts/tests/test-minikube-build-section-headers.sh
index 3a7b77133..06b7fdc03 100755
--- a/scripts/tests/test-minikube-build-section-headers.sh
+++ b/scripts/tests/test-minikube-build-section-headers.sh
@@ -61,6 +61,15 @@ STUB
#!/usr/bin/env bash
printf 'docker %s\n' "$*" >>"${TEST_LOG_FILE:?}"
case "${1:-}" in
+ context)
+ if [[ "${2:-}" == inspect ]]; then
+ if [[ "$*" == *SkipTLSVerify* ]]; then
+ printf 'unix:///tmp/evenfire-docker.sock\tfalse\t{}\n'
+ else
+ printf 'unix:///tmp/evenfire-docker.sock\n'
+ fi
+ fi
+ ;;
inspect) echo "sha256:deadbeef0000cafedeadbeef0000cafedeadbeef" ;;
# Empty: the host daemon holds no public image yet, so the public-image loop
# takes its `docker pull` branch instead of "already present -- skipping".
@@ -72,6 +81,19 @@ STUB
chmod +x "$d/bin/minikube" "$d/bin/kubectl" "$d/bin/docker"
}
+# The real builder now refuses every Docker/Minikube mutation without the
+# inherited branch-profile lease. This fixture is intentionally not a live
+# worktree or cluster, so it models only the child boundary; the real lease
+# implementation is covered by test-minikube-build-images-hardening.sh.
+write_fixture_mutation_lock_stub() {
+ local d=$1
+ cat > "$d/repo/scripts/minikube/require-t2-mutation-lock.sh" <<'STUB'
+#!/usr/bin/env bash
+exit 0
+STUB
+ chmod +x "$d/repo/scripts/minikube/require-t2-mutation-lock.sh"
+}
+
# An isolated PROJECT_DIR that is a copy of the real repo's deploy/ + scripts/,
# so the script under test is the real one and reads the real manifest.
prepare_repo() {
@@ -81,6 +103,7 @@ prepare_repo() {
cp -R "$REPO_ROOT/deploy" "$d/repo/deploy"
cp -R "$REPO_ROOT/scripts" "$d/repo/scripts"
rm -rf "$d/repo/deploy/minikube"
+ write_fixture_mutation_lock_stub "$d"
# An empty daemon inventory. Every base/public image then takes the "pull it"
# branch, which is the slower path through the script and therefore the one
# most likely to print something unexpected.
@@ -98,6 +121,9 @@ run_build() {
RUN_OUT="$(PATH="$d/bin:$PATH" \
TEST_LOG_FILE="$d/ops.log" \
TEST_PRESENT_JSON="$d/present.json" \
+ T2_PROJECT_DIR="$d/repo" T2_PROFILE=clerum-test T2_CONTEXT=clerum-test \
+ MINIKUBE_PROFILE=clerum-test CONTROL_API_REAL_PG_CONTEXT=clerum-test \
+ DOCKER_HOST=unix:///tmp/evenfire-docker.sock \
bash "$d/repo/scripts/minikube/build-images.sh" "$@" 2>&1)"
RUN_RC=$?
}
diff --git a/scripts/tests/test-minikube-fenced-recovery-render.sh b/scripts/tests/test-minikube-fenced-recovery-render.sh
index 4b81515d1..f129da2d8 100755
--- a/scripts/tests/test-minikube-fenced-recovery-render.sh
+++ b/scripts/tests/test-minikube-fenced-recovery-render.sh
@@ -31,6 +31,14 @@ metadata:
spec:
replicas: 1
---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: trace-maintenance-worker
+ namespace: control-plane
+spec:
+ replicas: 1
+---
apiVersion: v1
kind: Service
metadata:
@@ -42,11 +50,12 @@ output="$TMP_DIR/output.yaml"
ruby "$RENDERER" \
--target control-plane/host-context-controller \
--target control-plane/workflow-recipes \
+ --target control-plane/trace-maintenance-worker \
--target control-plane/control-api <"$TMP_DIR/input.yaml" >"$output"
ruby -ryaml -e '
docs = YAML.load_stream(File.read(ARGV.fetch(0)))
-targets = %w[host-context-controller workflow-recipes control-api]
+ targets = %w[host-context-controller workflow-recipes trace-maintenance-worker control-api]
targets.each do |name|
deployment = docs.find { |doc| doc.is_a?(Hash) && doc["kind"] == "Deployment" && doc.dig("metadata", "namespace") == "control-plane" && doc.dig("metadata", "name") == name }
abort "missing #{name}" unless deployment
diff --git a/scripts/tests/test-minikube-makefile.sh b/scripts/tests/test-minikube-makefile.sh
index bc8252bd6..9c2201360 100755
--- a/scripts/tests/test-minikube-makefile.sh
+++ b/scripts/tests/test-minikube-makefile.sh
@@ -349,7 +349,8 @@ assert_minikube_setup_local_forces_the_build() {
assert_minikube_pull_images_mirrors_build_images() {
local out
out="$(make -n -C "$REPO_ROOT" minikube-pull-images 2>&1)"
- if grep -q 'scripts/minikube/pull-images.sh' <<< "$out"; then
+ if grep -q 'scripts/minikube/pull-images.sh' <<< "$out" ||
+ grep -q 'scripts/minikube/pull-images.sh' "$REPO_ROOT/Makefile"; then
pass "minikube-pull-images invokes the puller"
else
fail "minikube-pull-images does not invoke pull-images.sh: $out"
diff --git a/scripts/tests/test-minikube-port-forward-owner.sh b/scripts/tests/test-minikube-port-forward-owner.sh
index 9bf0ea8d5..4c9ab7836 100755
--- a/scripts/tests/test-minikube-port-forward-owner.sh
+++ b/scripts/tests/test-minikube-port-forward-owner.sh
@@ -179,6 +179,10 @@ pass 'dead stale pidfiles are removed without signalling'
reset_process_fixture
write_record
+pf_owner_record_process_matches "${RECORD}" "${PROFILE}" "${CONTEXT}" \
+ "${WORKTREE}" "${NAMESPACE}" "${SERVICE}" "${LOCAL_PORT}" "${REMOTE_PORT}" ||
+ fail 'exact ownership revalidation rejected the recorded live process'
+pass 'post-health ownership revalidation proves the exact live process and binding'
cleanup_record || fail 'exact owned cleanup failed'
[[ ! -e "${RECORD}" ]] || fail 'exact owned cleanup kept the pidfile'
grep -Fxq "${PID} TERM" "${KILL_LOG}" || fail 'exact owned cleanup did not send TERM'
diff --git a/scripts/tests/test-minikube-pre-gate-shadow.sh b/scripts/tests/test-minikube-pre-gate-shadow.sh
index eafa34e69..95eb19084 100755
--- a/scripts/tests/test-minikube-pre-gate-shadow.sh
+++ b/scripts/tests/test-minikube-pre-gate-shadow.sh
@@ -57,6 +57,15 @@ make_stubs() {
#!/usr/bin/env bash
printf 'docker %s\n' "$*" >>"${TEST_LOG_FILE:?}"
case "${1:-}" in
+ context)
+ if [[ "${2:-}" == inspect ]]; then
+ if [[ "$*" == *SkipTLSVerify* ]]; then
+ printf 'unix:///tmp/evenfire-docker.sock\tfalse\t{}\n'
+ else
+ printf 'unix:///tmp/evenfire-docker.sock\n'
+ fi
+ fi
+ ;;
inspect)
# The only inspect the pre-gate makes is the OCI revision-label read that
# gives it the commit a release image was built from.
@@ -122,6 +131,11 @@ prepare_repo() {
cp -R "$REPO_ROOT/deploy" "$d/repo/deploy"
cp -R "$REPO_ROOT/scripts" "$d/repo/scripts"
rm -rf "$d/repo/deploy/minikube"
+ cat > "$d/repo/scripts/minikube/require-t2-mutation-lock.sh" <<'STUB'
+#!/usr/bin/env bash
+exit 0
+STUB
+ chmod +x "$d/repo/scripts/minikube/require-t2-mutation-lock.sh"
# patches/k8s-api-ip.yaml is GENERATED and gitignored (the overlay commits
# only the .template), so whether the developer's tree has one is incidental.
# Materialise it deterministically: by the time anything renders, `make
@@ -928,6 +942,9 @@ assert_a_targeted_build_carries_the_recorded_coordinate_forward() {
(
cd "$d/repo" || exit 1
PATH="$d/bin:$PATH" TEST_LOG_FILE="$d/ops.log" \
+ T2_PROJECT_DIR="$d/repo" T2_PROFILE=clerum-test T2_CONTEXT=clerum-test \
+ MINIKUBE_PROFILE=clerum-test CONTROL_API_REAL_PG_CONTEXT=clerum-test \
+ DOCKER_HOST=unix:///tmp/evenfire-docker.sock \
bash "$d/repo/scripts/minikube/build-images.sh" --only=control-api
) >"$d/build.log" 2>&1
rc=$?
diff --git a/scripts/tests/test-minikube-pull-images.sh b/scripts/tests/test-minikube-pull-images.sh
index 5e1702e51..7f23f2281 100755
--- a/scripts/tests/test-minikube-pull-images.sh
+++ b/scripts/tests/test-minikube-pull-images.sh
@@ -31,6 +31,9 @@ printf '%s\n' "$*" >>"${TEST_LOG_FILE:?}"
case "${1:-}" in
pull)
ref="${2:-}"
+ if [ "${TEST_HANG_PULL:-false}" = true ]; then
+ while :; do sleep 60; done
+ fi
for missing in ${TEST_MISSING_TAGS:-}; do
if [[ "$ref" == "$missing" ]]; then
for i in $(seq 1 20); do
@@ -56,6 +59,16 @@ case "${1:-}" in
done
exit 0
;;
+ context)
+ if [[ "${2:-}" == inspect ]]; then
+ if [[ "$*" == *SkipTLSVerify* ]]; then
+ printf 'unix:///tmp/evenfire-docker.sock\tfalse\t{}\n'
+ else
+ printf 'unix:///tmp/evenfire-docker.sock\n'
+ fi
+ fi
+ exit 0
+ ;;
tag) exit 0 ;;
images) echo "sha256:aaaaaaaaaaaa"; exit 0 ;;
inspect) echo "sha256:aaaaaaaaaaaabbbbbbbbbbbbcccccccccccc"; exit 0 ;;
@@ -91,6 +104,14 @@ copy_repo() {
cp -R "$REPO_ROOT/deploy" "$d/repo/deploy"
cp -R "$REPO_ROOT/scripts" "$d/repo/scripts"
rm -f "$d/repo/deploy/minikube/.image-manifest.json"
+ # The real puller is a mutating child and validates the inherited lease.
+ # This fixture is intentionally a no-op child validator; the puller's
+ # explicit profile/project/context contract is still exercised below.
+ cat > "$d/repo/scripts/minikube/require-t2-mutation-lock.sh" <<'STUB'
+#!/usr/bin/env bash
+exit 0
+STUB
+ chmod +x "$d/repo/scripts/minikube/require-t2-mutation-lock.sh"
if [ -n "$manifest" ]; then
mkdir -p "$d/repo/deploy/minikube"
printf '%s' "$manifest" > "$d/repo/deploy/minikube/.image-manifest.json"
@@ -126,9 +147,59 @@ run_puller_prepared() {
MINIKUBE_IMAGE_TAG="${MINIKUBE_IMAGE_TAG:-}" \
MINIKUBE_IMAGE_PULL_RETRIES="${MINIKUBE_IMAGE_PULL_RETRIES:-}" \
MINIKUBE_IMAGE_PULL_DELAY_SECS="${MINIKUBE_IMAGE_PULL_DELAY_SECS:-}" \
+ TEST_HANG_PULL="${TEST_HANG_PULL:-}" \
+ MINIKUBE_DOCKER_PULL_TIMEOUT_SECONDS="${MINIKUBE_DOCKER_PULL_TIMEOUT_SECONDS:-}" \
+ MINIKUBE_DOCKER_BUILD_TIMEOUT_SECONDS="${MINIKUBE_DOCKER_BUILD_TIMEOUT_SECONDS:-}" \
+ T2_PROJECT_DIR="$d/repo" \
+ T2_PROFILE=clerum-test T2_CONTEXT=clerum-test \
+ MINIKUBE_PROFILE=clerum-test CONTROL_API_REAL_PG_CONTEXT=clerum-test \
+ DOCKER_HOST=unix:///tmp/evenfire-docker.sock \
bash "$d/repo/scripts/minikube/pull-images.sh" "$@" 2>&1
}
+run_puller_without_lease() {
+ local d=$1; shift
+ make_stubs "$d"
+ copy_repo "$d"
+ env -u T2_PROJECT_DIR -u T2_PROFILE -u T2_CONTEXT \
+ -u CONTROL_API_REAL_PG_CONTEXT \
+ PATH="$d/bin:$PATH" TEST_LOG_FILE="$d/ops.log" \
+ MINIKUBE_PROFILE=clerum-test DOCKER_HOST=unix:///tmp/evenfire-docker.sock \
+ bash "$d/repo/scripts/minikube/pull-images.sh" "$@" 2>&1
+}
+
+assert_puller_requires_the_inherited_profile_lease() {
+ local d out rc
+ d="$(mktemp -d)"
+ out="$(run_puller_without_lease "$d" --only=control-api)"; rc=$?
+ if [ "$rc" -ne 0 ] && grep -Fq 'PROFILE_LOCK_REQUIRED' <<< "$out" \
+ && [ ! -s "$d/ops.log" ]; then
+ pass "puller refuses mutation without the inherited profile lease"
+ else
+ fail "puller did not fail before Docker mutation without its lease: rc=$rc out=$out"
+ fi
+ rm -rf "$d"
+}
+
+assert_hung_pull_hits_the_finite_deadline() {
+ local d out rc
+ d="$(mktemp -d)"
+ export TEST_HANG_PULL=true
+ export MINIKUBE_IMAGE_TAG=deadline-test
+ export MINIKUBE_IMAGE_PULL_RETRIES=1
+ export MINIKUBE_IMAGE_PULL_DELAY_SECS=0
+ export MINIKUBE_DOCKER_PULL_TIMEOUT_SECONDS=1
+ out="$(run_puller "$d" --only=control-api)"; rc=$?
+ unset TEST_HANG_PULL MINIKUBE_IMAGE_TAG MINIKUBE_IMAGE_PULL_RETRIES \
+ MINIKUBE_IMAGE_PULL_DELAY_SECS MINIKUBE_DOCKER_PULL_TIMEOUT_SECONDS
+ if [ "$rc" -ne 0 ] && grep -Eq 'HARNESS_DEADLINE.*pull-image-control-api.*timeout' <<< "$out"; then
+ pass "a hung image pull is terminated by its finite deadline"
+ else
+ fail "hung pull did not produce the bounded deadline failure: rc=$rc out=$out"
+ fi
+ rm -rf "$d"
+}
+
assert_it_pulls_every_pull_in_ghcr_mode_image() {
local d out rc want got
d="$(mktemp -d)"
@@ -682,6 +753,8 @@ assert_every_defined_case_is_invoked() {
}
assert_it_pulls_every_pull_in_ghcr_mode_image
+assert_puller_requires_the_inherited_profile_lease
+assert_hung_pull_hits_the_finite_deadline
assert_it_never_pulls_an_unpublished_image
assert_it_never_pulls_a_registry_distributed_mcp_server
assert_it_repulls_a_tag_already_present_in_the_daemon
diff --git a/scripts/tests/test-minikube-t2-contract.sh b/scripts/tests/test-minikube-t2-contract.sh
index 887ef9d9e..6f62da4ba 100755
--- a/scripts/tests/test-minikube-t2-contract.sh
+++ b/scripts/tests/test-minikube-t2-contract.sh
@@ -39,6 +39,12 @@ for file in "$MINIKUBE_DIR/profile-readiness.sh" "$ROOT/scripts/tests/test-minik
"$ROOT/scripts/tests/test-real-postgres-local-preflight.sh" \
"$ROOT/scripts/tests/test-minikube-t1-docker-boundary.sh" \
"$ROOT/scripts/tests/test-minikube-t1-port-forward-owner.sh" \
+ "$ROOT/scripts/tests/test-minikube-port-forward-owner.sh" \
+ "$ROOT/scripts/tests/test-minikube-docker-cli-env.sh" \
+ "$ROOT/scripts/tests/test-minikube-build-images-hardening.sh" \
+ "$ROOT/scripts/tests/test-minikube-pull-images.sh" \
+ "$ROOT/scripts/tests/test-minikube-build-section-headers.sh" \
+ "$ROOT/scripts/tests/test-minikube-pre-gate-shadow.sh" \
"$ROOT/scripts/tests/test-minikube-fenced-recovery-render.sh" \
"$ROOT/scripts/tests/test-minikube-t2-process-owner.sh" \
"$ROOT/scripts/tests/test-minikube-explicit-context.sh" \
@@ -53,6 +59,7 @@ done
"$ROOT/scripts/tests/test-minikube-targeted-health.sh"
"$ROOT/scripts/tests/test-minikube-filter-gfs-resources.sh"
"$ROOT/scripts/tests/test-minikube-t1-docker-boundary.sh"
+"$ROOT/scripts/tests/test-minikube-pull-images.sh"
"$ROOT/scripts/tests/test-minikube-k8s-api-egress-policy.sh"
"$ROOT/scripts/tests/test-minikube-writer-recovery-state.sh"
"$ROOT/scripts/tests/test-minikube-fenced-recovery-render.sh"
@@ -95,7 +102,12 @@ grep -Fq 'kubectl --context=' "$COMMON"
grep -Fq 'port-forward-owner.sh' "$COMMON"
grep -Fq 't2_port_forward_targets_context' "$COMMON"
grep -Fq 'matching_records' "$COMMON"
-grep -Fq 'pf_owner_command_matches' "$COMMON"
+grep -Fq 'pf_owner_record_process_matches' "$COMMON"
+grep -Fq 'pf_owner_command_matches' "$ROOT/scripts/minikube/port-forward-owner.sh"
+grep -Fq 'T2_MARKER_IMAGES_GENERATED_AT' "$COMMON"
+grep -Fq 'image acquisition changed since the pre-gate marker' "$COMMON"
+grep -Fq 'trace-maintenance-worker' "$ROOT/scripts/minikube/full-setup.sh"
+grep -Fq 'pf_owner_record_process_matches' "$ROOT/scripts/minikube/port-forward-owner.sh"
grep -Fq 'CONTROL_API_REAL_PG_CONTEXT' "$T1"
grep -Fq 'T2_REQUIRED_DEPLOYMENTS' "$COMMON"
grep -Fq 'CONTROL_API_REAL_PG_ADMIN_URL=' "$T1"
@@ -135,6 +147,8 @@ grep -Fq 'run_t1_local_preflight' "$T2"
grep -Fq 'T2_SETUP_HANDOFF_REQUIRED=true' "$T2"
grep -Fq 't2-setup-handoff.sh" consume --' "$ROOT/scripts/minikube/pre-gate-sync.sh"
grep -Fq 'minikube-verify-images' "$ROOT/scripts/minikube/pre-gate-sync.sh"
+grep -Fq 'with-t2-mutation-lock.sh' "$ROOT/scripts/minikube/setup.sh"
+grep -Fq 'T2_MUTATION_LOCK_WRAPPED' "$ROOT/scripts/minikube/setup.sh"
for timed_phase in 't2_evidence_write planner PASS' 't2_evidence_write transition PASS' \
't2_evidence_write pre-gate-sync PASS' 't2_evidence_write T1 PASS' \
't2_evidence_write T2 PASS' 't2_evidence_write NP08_HCC_AUTHORIZATION PASS'; do
@@ -607,5 +621,10 @@ bash "$ROOT/scripts/tests/test-minikube-targeted-gfs-sync.sh"
bash "$ROOT/scripts/tests/test-minikube-t2-lock-race.sh"
bash "$ROOT/scripts/tests/test-minikube-t1-gfs-restore.sh"
bash "$ROOT/scripts/tests/test-real-postgres-local-preflight.sh"
+bash "$ROOT/scripts/tests/test-minikube-port-forward-owner.sh"
+bash "$ROOT/scripts/tests/test-minikube-docker-cli-env.sh"
+bash "$ROOT/scripts/tests/test-minikube-build-images-hardening.sh"
+bash "$ROOT/scripts/tests/test-minikube-build-section-headers.sh"
+bash "$ROOT/scripts/tests/test-minikube-pre-gate-shadow.sh"
printf 'PASS: local Minikube T0/T1/T2 contract checks\n'
diff --git a/scripts/tests/test-minikube-t2-process-owner.sh b/scripts/tests/test-minikube-t2-process-owner.sh
index c6d1240e6..484c3242e 100755
--- a/scripts/tests/test-minikube-t2-process-owner.sh
+++ b/scripts/tests/test-minikube-t2-process-owner.sh
@@ -90,6 +90,17 @@ pf_owner_write_record_atomic "$RECORD" "$PID" "$START" "$PROFILE" \
"$PROFILE" "$ROOT" control-plane control-ui 3000 3000
run_process_check "$TEST_DIR/exact.out"
+# A registered record must remain visible to the verdict even when the live
+# process disappears from ps(1). This protects the fail-closed ownership scan
+# from treating a stale/dead pidfile as "nothing to check".
+rm -f "$PS_DATA/$PID.command"
+if run_process_check "$TEST_DIR/stale-record.out"; then
+ printf 'FAIL: stale registered port-forward record was ignored\n' >&2
+ exit 1
+fi
+grep -Fq PORT_FORWARD_CONFLICT "$TEST_DIR/stale-record.out"
+set_process_fixture kubectl "$exact_command"
+
printf '%s\n' 'different process start' >"$PS_DATA/$PID.start"
if run_process_check "$TEST_DIR/reused.out"; then
printf 'FAIL: process-start mismatch was accepted\n' >&2
diff --git a/scripts/tests/test-minikube-t2-scenarios.sh b/scripts/tests/test-minikube-t2-scenarios.sh
index f3e0548c8..8304cb37f 100755
--- a/scripts/tests/test-minikube-t2-scenarios.sh
+++ b/scripts/tests/test-minikube-t2-scenarios.sh
@@ -215,7 +215,7 @@ bootstrap_manifest_state="$(env "${repo_env[@]}" T2_IMAGE_MANIFEST="$invalid_man
[ "$bootstrap_manifest_state" = full-bootstrap ] || fail "invalid bootstrap manifest selected $bootstrap_manifest_state instead of full-bootstrap"
local_manifest="$tmp/local-image-manifest.json"
-printf '{"imageSource":"local","imageTag":"","images":{"clerum/control-api:test":"sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}}\n' >"$local_manifest"
+printf '{"generated":"manifest-generated","imageSource":"local","imageTag":"","images":{"clerum/control-api:test":"sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}}\n' >"$local_manifest"
env "${repo_env[@]}" T2_IMAGE_MANIFEST="$local_manifest" T2_IMAGE_SOURCE=local T2_IMAGE_TAG= \
bash -c 'source "$1"; T2_IMAGE_SOURCE=local; T2_IMAGE_TAG=""; t2_image_check' bash "$COMMON"
@@ -300,6 +300,27 @@ already_state="$(env "${repo_env[@]}" T2_PROJECT_DIR="$repo" T2_BOOTSTRAP_REQUIR
bash -c 'source "$1"; T2_ORIGIN_DEV="$2"; T2_HEAD="$3"; T2_BOOTSTRAP_REQUIRED=false; T2_MARKER_MATCHES_HEAD=true; t2_classify_transition; printf "%s" "$T2_PLAN_STATE"' bash "$COMMON" "$base_sha" "$infra_sha")"
[ "$already_state" = already-synced ] || fail "matching marker selected $already_state instead of already-synced"
+# A new image acquisition can replace the digest while Git HEAD stays the
+# same. The marker must carry the manifest's generated stamp; otherwise the
+# planner would incorrectly choose already-synced/T2-runtime against the new
+# image set.
+stamp_manifest="$tmp/stamp-image-manifest.json"
+printf '{"generated":"new-generated","imageSource":"local","imageTag":"test","images":{"clerum/control-api:test":"sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}}\n' >"$stamp_manifest"
+stamp_state="$(env T2_PROJECT_DIR="$repo" MINIKUBE_PROFILE="$profile" \
+ T2_CONTEXT="$profile" CONTROL_API_REAL_PG_CONTEXT="$profile" \
+ T2_PROFILE_ROOT="$tmp/profiles" T2_PROFILE_ENV="$profile_root/profile.env" \
+ T2_PORTS_ENV="$profile_root/ports.env" T2_REQUIRED_DEPLOYMENTS=gfs/gfsc-reader \
+ T2_BRANCH=feat/scenario T2_HEAD="$feature_sha" T2_ORIGIN_DEV="$base_sha" \
+ T2_LOCK_ROOT="$tmp/locks" T2_EVIDENCE_ROOT="$tmp/evidence" \
+ T2_IMAGE_MANIFEST="$stamp_manifest" \
+ FAKE_MARKER='{"data":{"clusterFingerprint":"fp","gitHead":"'"$feature_sha"'","worktreeId":"worktree-a","imageSource":"local","imageTag":"test","imagesGeneratedAt":"old-generated"}}' \
+ bash -c 'source "$1"; T2_WORKTREE_ID=worktree-a; T2_HEAD="$3"; T2_ORIGIN_DEV="$2"; T2_PLAN_MODE=true; T2_BOOTSTRAP_REQUIRED=false; t2_kc(){ printf "%s" "$FAKE_MARKER"; }; t2_marker_check; t2_image_check; t2_classify_transition; printf "%s|%s" "$T2_PLAN_STATE" "$T2_PLAN_REASON"' bash "$COMMON" "$base_sha" "$feature_sha")"
+case "$stamp_state" in
+ already-synced*) fail "a changed image acquisition was treated as already-synced: $stamp_state" ;;
+ targeted-sync*|full-reconcile*) ;;
+ *) fail "changed image acquisition selected an unexpected plan: $stamp_state" ;;
+esac
+
# A bootstrapped profile with an unready required deployment must not stop
# the orchestrator planner before a transition exists (PROFILE_UNHEALTHY was
# the multi-script loop): planner mode selects full-reconcile and names the
diff --git a/scripts/tests/test-minikube-t2-setup-handoff.sh b/scripts/tests/test-minikube-t2-setup-handoff.sh
index c18f94072..27f0850a4 100755
--- a/scripts/tests/test-minikube-t2-setup-handoff.sh
+++ b/scripts/tests/test-minikube-t2-setup-handoff.sh
@@ -65,7 +65,7 @@ new_case() {
CONSUMED_PATH="${STATE_ROOT}/${WORKTREE_ID}/${RUN_ID}/setup-complete.consumed.json"
CLAIM_PATH="${STATE_ROOT}/${WORKTREE_ID}/${RUN_ID}/setup-complete.consume.claim"
mkdir -p "$(dirname "${MANIFEST}")"
- printf '{"imageSource":"local","images":{"control-api":"sha256:test"}}\n' >"${MANIFEST}"
+ printf '{"generated":"handoff-generated","imageSource":"local","images":{"control-api":"sha256:test"}}\n' >"${MANIFEST}"
}
run_handoff() {
diff --git a/scripts/tests/test-minikube-verify-images.sh b/scripts/tests/test-minikube-verify-images.sh
index 28251f4cf..23af58711 100755
--- a/scripts/tests/test-minikube-verify-images.sh
+++ b/scripts/tests/test-minikube-verify-images.sh
@@ -120,10 +120,10 @@ prepare_repo() {
rm -rf "$d/repo/deploy/minikube"
}
-# Full and --only builds are mutation paths and now require the inherited T2
-# lease. This fixture is intentionally not a Git worktree or a live profile, so
-# those two manifest-writer cases inject a no-op lease boundary below. The real
-# ownership/lock contract is exercised separately by
+# Pulls, full builds, and --only builds are mutation paths and now require the
+# inherited T2 lease. This fixture is intentionally not a Git worktree or a
+# live profile, so the manifest-writer cases inject a no-op lease boundary
+# below. The real ownership/lock contract is exercised separately by
# test-minikube-build-images-hardening.sh.
write_fixture_mutation_lock_stub() {
local d=$1
@@ -292,7 +292,11 @@ assert_the_puller_records_ghcr_and_the_verifier_reads_it_back() {
local d out rc pull_rc present
d="$(mktemp -d)"
prepare_repo "$d"
+ write_fixture_mutation_lock_stub "$d"
PATH="$d/bin:$PATH" TEST_LOG_FILE="$d/ops.log" \
+ T2_PROJECT_DIR="$d/repo" T2_PROFILE=clerum-test T2_CONTEXT=clerum-test \
+ MINIKUBE_PROFILE=clerum-test CONTROL_API_REAL_PG_CONTEXT=clerum-test \
+ DOCKER_HOST=unix:///tmp/evenfire-docker.sock \
bash "$d/repo/scripts/minikube/pull-images.sh" >"$d/pull.out" 2>&1
pull_rc=$?
if [ "$pull_rc" -ne 0 ]; then
diff --git a/scripts/tests/test-minikube-writer-recovery-state.sh b/scripts/tests/test-minikube-writer-recovery-state.sh
index b2ef1c726..73f3c9765 100755
--- a/scripts/tests/test-minikube-writer-recovery-state.sh
+++ b/scripts/tests/test-minikube-writer-recovery-state.sh
@@ -27,26 +27,25 @@ identity=(
--head 0123456789abcdef
)
-python3 "$HELPER" write "${identity[@]}" --phase planned --hcc 1 --workflow 1 --control-api 1
+python3 "$HELPER" write "${identity[@]}" --phase planned --hcc 1 --workflow 1 --trace 1 --control-api 1
mode="$(stat -f '%Lp' "$STATE" 2>/dev/null || stat -c '%a' "$STATE")"
[[ "$mode" == 600 ]] || fail "state mode is $mode, expected 600"
-python3 "$HELPER" read "${identity[@]}" | grep -Fxq 'planned|1|1|1' ||
+python3 "$HELPER" read "${identity[@]}" | grep -Fxq 'planned|1|1|1|1' ||
fail 'planned state did not round-trip with exact replicas'
pass 'writer recovery state is private and round-trips atomically'
-python3 "$HELPER" write "${identity[@]}" --phase api-fencing --hcc 1 --workflow 2 --control-api 1
-python3 "$HELPER" read "${identity[@]}" | grep -Fxq 'api-fencing|1|2|1' ||
+python3 "$HELPER" write "${identity[@]}" --phase api-fencing --hcc 1 --workflow 2 --trace 1 --control-api 1
+python3 "$HELPER" read "${identity[@]}" | grep -Fxq 'api-fencing|1|2|1|1' ||
fail 'fencing phase did not preserve original replicas'
for phase in api-restoring overlay-applying; do
- python3 "$HELPER" write "${identity[@]}" --phase "$phase" --hcc 1 --workflow 2 --control-api 1
- python3 "$HELPER" read "${identity[@]}" | grep -Fxq "${phase}|1|2|1" ||
+ python3 "$HELPER" write "${identity[@]}" --phase "$phase" --hcc 1 --workflow 2 --trace 1 --control-api 1
+ python3 "$HELPER" read "${identity[@]}" | grep -Fxq "${phase}|1|2|1|1" ||
fail "${phase} phase did not round-trip"
done
-wrong_status=0
-python3 "$HELPER" read "${identity[@]}" --head fedcba9876543210 >/dev/null 2>&1 || wrong_status=$?
-[[ "$wrong_status" -ne 0 ]] || fail 'state was readable under a different HEAD'
+python3 "$HELPER" read "${identity[@]}" --head fedcba9876543210 | grep -Fxq 'overlay-applying|1|2|1|1' ||
+ fail 'state did not resume under a different HEAD on the same owned lane'
[[ -s "$STATE" ]] || fail 'identity mismatch destroyed durable state'
-pass 'writer recovery state fails closed on a different HEAD'
+pass 'writer recovery state preserves lane identity while retaining historical HEAD'
python3 "$HELPER" clear "${identity[@]}"
[[ ! -e "$STATE" ]] || fail 'clear left durable writer recovery state behind'