Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

docs: record T2 targeted-sync health and stale port-forward codes

Alfredo Lopez7f4ef51a7bac
diff --git a/.cursor/rules/minikube-t0-t1-t2.mdc b/.cursor/rules/minikube-t0-t1-t2.mdc
index 6fc824a6c..73adfdb0c 100644
--- a/.cursor/rules/minikube-t0-t1-t2.mdc
+++ b/.cursor/rules/minikube-t0-t1-t2.mdc
@@ -31,8 +31,12 @@ repository — never mix its PRs or clusters into this lane.
- `make minikube-pre-gate-sync` is NOT T2.
- Product E2E scripts (e.g. `scripts/e2e/e2e-hcc-rollout-readiness.sh`), CI,
unit suites, and Playwright are separate lanes; none stands in for T2.
-- Health/Playwright are opt-in via `T2_HEALTHCHECK_COMMAND` /
- `T2_PLAYWRIGHT_COMMAND` (`T2_REQUIRE_PLAYWRIGHT=true` refuses `NOT_RUN`).
+- Health is mandatory and bounded for `targeted-sync` via
+ `T2_HEALTHCHECK_COMMAND`. After `T2_PREFLIGHT_PASS` the orchestrator still
+ fail-louds `PROFILE_UNHEALTHY` if that command is unset — planner PASS is
+ not T2. Health stays opt-in for bootstrap, full reconcile, and
+ already-synced. Playwright is opt-in via `T2_PLAYWRIGHT_COMMAND`
+ (`T2_REQUIRE_PLAYWRIGHT=true` refuses `NOT_RUN`).
## Branch-profile UI port-forwards
@@ -48,7 +52,14 @@ MINIKUBE_PROFILE=<owned-profile> \
```
Host-side hold for Control UI / Desktop. Profile-owned random ports only
-(never `:3000` / `:8090`). Run on the host, not a sandboxed agent shell.
+(never `:3000` / `:8090`). Run on the host, not a sandboxed agent shell:
+`make ... branch-profile-pf` that prints `PF` lines and exits 0 can still
+leave registered-but-dead pidfiles when the runner reaps `nohup kubectl`
+children. The planner then fail-louds `PORT_FORWARD_CONFLICT` +
+`DEVELOPMENT_SCOPE_REQUIRED` **before** a transition.
+`T2_PORT_FORWARD_COMMAND` does not skip that pre-transition check. Restore
+with a lasting host `branch-profile-pf` + `branch-profile-health`, then
+re-enter `make minikube-t2`. Not a new profile. Do not `ls`/`cat` the cache.
`make minikube-pf-all-bg` is a gate refresh only — it does not replace
`branch-profile-pf`. Do not kill this lane's forwards.
`branch-profile-pf-health` stops PFs on EXIT. Inner `pre-gate-sync` may use
diff --git a/.cursor/skills/minikube-t0-t1-t2/SKILL.md b/.cursor/skills/minikube-t0-t1-t2/SKILL.md
index 42f0976ea..370c5d096 100644
--- a/.cursor/skills/minikube-t0-t1-t2/SKILL.md
+++ b/.cursor/skills/minikube-t0-t1-t2/SKILL.md
@@ -59,7 +59,13 @@ Before running anything, verify ALL of these:
then `branch-profile-health`. Implementation:
`.local-notes/minikube-profiles/branch-profile.sh`. Do not replace
that hold with `make minikube-pf-all-bg`. Do not start UI PFs from a
- sandboxed agent shell. Do not kill this lane's `branch-profile-pf`.
+ sandboxed agent shell. A `make ... branch-profile-pf` that prints `PF`
+ lines and exits 0 can still leave registered-but-dead pidfiles when the
+ runner reaps `nohup kubectl` children. The planner then fail-louds
+ `PORT_FORWARD_CONFLICT` + `DEVELOPMENT_SCOPE_REQUIRED` before a
+ transition; `T2_PORT_FORWARD_COMMAND` has not run yet. Restore with a
+ lasting host hold + `branch-profile-health`, then re-enter T2. Do not
+ kill this lane's `branch-profile-pf`.
`branch-profile-pf-health` stops PFs on EXIT — not a lasting hold.
- [ ] Run `make minikube-t2` from a host terminal with
`T2_PORT_FORWARD_COMMAND` set to that `branch-profile-pf` command.
@@ -111,8 +117,10 @@ transition = full-reconcile (deploy/* or charts/* changed, OR a required
script between runs.
transition = targeted-sync (service-only diff)?
- └── make minikube-t2 performs the targeted deploy; record it as a
- targeted sync, never as a full reconcile.
+ └── make minikube-t2 with T2_HEALTHCHECK_COMMAND set to the affected
+ service's profile-owned user-facing journey. Record as targeted
+ sync, never as a full reconcile. T2_PREFLIGHT_PASS then
+ PROFILE_UNHEALTHY means the command was missing — not a T2 verdict.
```
Rules that override any shortcut idea:
@@ -181,7 +189,11 @@ Report to the user exactly these lane statuses plus HEAD, profile, and the
evidence path. `SKIPPED` is legitimate only for T0/T1 previously green on the
same HEAD (say so explicitly). `T2_HEALTHCHECK_COMMAND` is mandatory for
`targeted-sync` and bounded by `T2_HEALTHCHECK_TIMEOUT_SECONDS` (default 120s);
-it is optional for other transitions. `T2_PLAYWRIGHT_COMMAND` remains opt-in
+it is optional for other transitions. For `workflow-recipes`, combine
+`branch-profile-health` with an in-pod
+`kubectl --context=<owned> -n control-plane exec deploy/workflow-recipes --
+wget -qO- http://127.0.0.1:8082/health` — do not invent a host port.
+`T2_PLAYWRIGHT_COMMAND` remains opt-in
(`T2_REQUIRE_PLAYWRIGHT=true` refuses a missing journey).
Evidence stays under the ignored `.local-notes/infra/runs/`. Never commit it.
diff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.md
index 2c8f021dc..81d9a5708 100644
--- a/.cursor/skills/minikube-t0-t1-t2/reference.md
+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md
@@ -149,9 +149,9 @@ checkout `branch.mk` (`branch-profile-start`). Never invent ports or
| `BOOTSTRAP_REQUIRED` | Profile missing/uninitialized, or the planner produced no transition. Standalone preflight refuses to bootstrap. | Run `MINIKUBE_PROFILE=<owned> CONTROL_API_REAL_PG_CONTEXT=<owned> make minikube-t2`; its internal planner makes `full-bootstrap` reachable. |
| `HEAD_MARKER_MISMATCH` | Pre-gate marker does not match this worktree/HEAD; the final T2 preflight selected something other than `already-synced`. | Re-run the full target on this HEAD so `pre-gate-sync` updates the marker. Never hand-edit the marker. |
| `IMAGE_MANIFEST_MISMATCH` | Deployed image provenance or the manifest `generated` stamp does not match the exact marker/worktree state. | Re-run the full target for the same owned profile; do not relabel or hand-edit the manifest. |
-| `PORT_FORWARD_CONFLICT` | A port-forward for this profile is owned by a process not recorded for it (or not a real `kubectl`). | Identify the foreign owner; if it belongs to another lane, stop. Do not kill this lane's `branch-profile-pf`. Restore UI PFs with `make -f .local-notes/minikube-profiles/branch.mk branch-profile-pf`. |
+| `PORT_FORWARD_CONFLICT` | A port-forward for this profile is owned by a process not recorded for it, is not a real `kubectl`, or the registered pidfile is stale/dead/ambiguous. A sandboxed `make ... branch-profile-pf` that exits 0 can leave registered-but-dead records; the planner then wraps this as `DEVELOPMENT_SCOPE_REQUIRED` **before** a transition. `T2_PORT_FORWARD_COMMAND` does not skip that check. | Identify a foreign owner and stop if it is another lane. Do not kill this lane's `branch-profile-pf`. Restore with a lasting host `branch-profile-pf` + `branch-profile-health`, then re-enter `make minikube-t2`. Not a new profile. |
| `PROFILE_BUSY` | Profile lock held. Live owner PID → genuinely busy. No valid owner PID → orphaned lock. | Live owner: wait or coordinate; never remove. Orphan: verify no T2 process owns the profile, then remove ONLY `$T2_LOCK_ROOT/<profile>.lock` and retry. Never remove the lock root. |
-| `DEVELOPMENT_SCOPE_REQUIRED` | Preflight/final preflight failed a precondition, T2-only mode was attempted without `already-synced`, or the worktree is dirty (`worktree is dirty`). | Repair the first reported condition. If dirty: commit or stash, do not edit during the run, then re-enter T2. If T2-only was refused, run full `make minikube-t2`. |
+| `DEVELOPMENT_SCOPE_REQUIRED` | Preflight/final preflight failed a precondition (including `PORT_FORWARD_CONFLICT` on a stale pidfile before a transition was selected), T2-only mode was attempted without `already-synced`, or the worktree is dirty (`worktree is dirty`). | Repair the **first** reported condition. Stale PF: lasting host hold, then re-enter T2. If dirty: commit or stash, do not edit during the run, then re-enter T2. If T2-only was refused, run full `make minikube-t2`. |
| `CERTIFICATION_REQUIRED` | Runtime-only was requested without valid exact-head T0/T1 evidence for the full ownership tuple. | Run the full target once; do not hand-create or copy evidence. |
| `SECRET_MISSING` / `CONFIGMAP_MISSING` | A required named runtime input is absent or unreadable. | Reconcile the same owned profile through the full target; never invent values. |
| `ZERO_TESTS_EXECUTED` | A lane was configured off or executed nothing (including a required-but-missing Playwright journey). | Re-run with the lane enabled, or supply the required `T2_PLAYWRIGHT_COMMAND`. |
@@ -159,7 +159,7 @@ checkout `branch.mk` (`branch-profile-start`). Never invent ports or
| `REAL_PG_REPORT_INCOMPLETE` | Vitest JSON did not prove the exact expected physical files. | Repair test selection/reporter output; do not accept suite counters as file identity. |
| `REAL_PG_SUITE_FAILED` | Reporter, test assertions, process exit, or parent result contract failed. | Iterate with standalone T1; once green, run full T2 for final exact-head lane evidence. |
| `POSTGRES_NOT_READY` | PostgreSQL precondition failed, or a PVC reset was requested without the exact expected UID. | Fix DB readiness; never guess a PVC UID. |
-| `PROFILE_UNHEALTHY` | A required deployment is unready in a fail-loud check (`T2_PLAN_MODE=false`: standalone preflight or the final exact-head T2), the mandatory targeted-sync health command is missing, or a supplied user-facing health command failed or timed out. The orchestrator planner never emits this merely for an unready deployment — it selects `full-reconcile` instead. | For a targeted sync, supply the affected service's profile-owned `T2_HEALTHCHECK_COMMAND`; otherwise repair the reported health failure. Re-run on the same HEAD and profile. Do not run manual repair scripts between runs. |
+| `PROFILE_UNHEALTHY` | A required deployment is unready in a fail-loud check (`T2_PLAN_MODE=false`: standalone preflight or the final exact-head T2), the mandatory targeted-sync health command is missing, or a supplied user-facing health command failed or timed out. The orchestrator planner never emits this merely for an unready deployment — it selects `full-reconcile` instead. | For a targeted sync, supply the affected service's profile-owned `T2_HEALTHCHECK_COMMAND` **on the same `make minikube-t2` invocation** (planner `T2_PREFLIGHT_PASS` is not enough). For `workflow-recipes`: `branch-profile-health` plus in-pod `wget` of `deploy/workflow-recipes` `/health`. Otherwise repair the reported health failure. Re-run on the same HEAD and profile. Do not run manual repair scripts between runs. |
| `NP08_HCC_AUTHORIZATION_FAILED` | The deployed Host-to-HCC authorization journey or cleanup failed after T0/T1 passed. | Repair the first NP08 failure, then use `minikube-t2-runtime` on the exact same profile/HEAD; never refresh/reissue in the test. |
| `PLAYWRIGHT_FAILED` | The opt-in user-visible journey failed after runtime and lane evidence passed. | Repair the journey and retry with `minikube-t2-runtime` on the same exact tuple. |
diff --git a/AGENTS.md b/AGENTS.md
index b36ef53fb..cd63e9c95 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -47,16 +47,21 @@ MINIKUBE_PROFILE=<owned-profile> \
```
This is the host-side hold for Control UI / Desktop. Run it on the host, not
-from a sandboxed agent shell. `make minikube-pf-all-bg` is a gate refresh
-only and must not replace `branch-profile-pf`. Do not kill this lane's
-forwards. `branch-profile-pf-health` starts then stops PFs on EXIT — do not
-use it as the lasting hold. Inner `pre-gate-sync` may use
-`--skip-port-forwards`; never pass that globally into `make minikube-t2`.
-`pre-gate-sync` can restart every deployment, which leaves the host hold
-bound to terminated pods. Run `make minikube-t2` from a host terminal with
-`T2_PORT_FORWARD_COMMAND` set to the `branch-profile-pf` command above; T2
-runs it once after an in-run sync, before Health and Playwright, and records
-`PortForwards=` in the evidence.
+from a sandboxed agent shell. A `make ... branch-profile-pf` that prints `PF`
+lines and exits 0 can still leave registered-but-dead pidfiles when the
+runner reaps `nohup kubectl` children. The planner then fail-louds
+`PORT_FORWARD_CONFLICT` + `DEVELOPMENT_SCOPE_REQUIRED` before a transition;
+`T2_PORT_FORWARD_COMMAND` has not run yet. Restore with a lasting host hold
+and `branch-profile-health`, then re-enter `make minikube-t2`. Not a new
+profile. `make minikube-pf-all-bg` is a gate refresh only and must not
+replace `branch-profile-pf`. Do not kill this lane's forwards.
+`branch-profile-pf-health` starts then stops PFs on EXIT — do not use it as
+the lasting hold. Inner `pre-gate-sync` may use `--skip-port-forwards`; never
+pass that globally into `make minikube-t2`. `pre-gate-sync` can restart every
+deployment, which leaves the host hold bound to terminated pods. Run
+`make minikube-t2` from a host terminal with `T2_PORT_FORWARD_COMMAND` set to
+the `branch-profile-pf` command above; T2 runs it once after an in-run sync,
+before Health and Playwright, and records `PortForwards=` in the evidence.
Port-forwards are owned by atomic `0600` records bound to the exact profile,
context, canonical worktree, namespace, Service, local/remote ports, PID,
@@ -245,7 +250,14 @@ CI, Control UI/Desktop Playwright, and product E2E scripts such as
`scripts/e2e/e2e-hcc-rollout-readiness.sh` are separate evidence lanes. One
lane does not stand in for another. User-facing health is mandatory and bounded
for a `targeted-sync` transition via `T2_HEALTHCHECK_COMMAND`; it remains
-opt-in for bootstrap, full reconcile, and already-synced runs. Playwright is
+opt-in for bootstrap, full reconcile, and already-synced runs. A planner
+`T2_PREFLIGHT_PASS` / `transition=targeted-sync` followed by
+`PROFILE_UNHEALTHY` (`targeted sync requires a profile-owned user-facing
+health command`) means the command was missing — re-enter `make minikube-t2`
+with it set. For `workflow-recipes`, use `branch-profile-health` plus
+`kubectl --context=<owned> -n control-plane exec deploy/workflow-recipes --
+wget -qO- http://127.0.0.1:8082/health` (in-pod `/health`; do not invent a
+host port). Playwright is
opt-in via `T2_PLAYWRIGHT_COMMAND` (`T2_REQUIRE_PLAYWRIGHT=true` refuses
`NOT_RUN`). Private operational state,
generated ports, profile metadata, logs, and evidence belong under the ignored
diff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.md
index d8bac646c..8c7f7fcb6 100644
--- a/docs/testing/minikube-t2-runbook.md
+++ b/docs/testing/minikube-t2-runbook.md
@@ -68,8 +68,14 @@ HARD DENY: do not `ls`/`cat` `~/.cache/clerum/minikube-profiles/`.
This is the host-side hold for Control UI / Desktop. Profile-owned random
ports only (never shared `:3000`/`:8090`). `make minikube-pf-all-bg` is a
gate refresh only; it must not replace `branch-profile-pf`. Do not start UI
-PFs from a sandboxed agent shell (hooks/PATH; runners clean children). Run
-the make target on the host. Do not kill this lane's `branch-profile-pf`.
+PFs from a sandboxed agent shell (hooks/PATH; runners clean children). A
+`make ... branch-profile-pf` that prints `PF` lines and exits 0 can still
+leave registered-but-dead pidfiles; the planner then fail-louds
+`PORT_FORWARD_CONFLICT` + `DEVELOPMENT_SCOPE_REQUIRED` before a transition.
+`T2_PORT_FORWARD_COMMAND` does not skip that pre-transition check. Restore
+with a lasting host hold + `branch-profile-health`, then re-enter
+`make minikube-t2`. Run the make target on the host. Do not kill this
+lane's `branch-profile-pf`.
Inner `pre-gate-sync` may use `--skip-port-forwards`; never pass that
globally into `make minikube-t2`. `branch-profile-pf-health` starts PFs then
STOPS them on EXIT — do not use it as the lasting hold.
@@ -382,7 +388,9 @@ non-T2 scope guard.
image manifest is current, PostgreSQL and required namespaces/Services are
present, deployments are Ready, and no foreign `kubectl port-forward` owns
this profile. A targeted sync requires a bounded user-facing journey via
- `T2_HEALTHCHECK_COMMAND`; other transitions may leave it opt-in. Control
+ `T2_HEALTHCHECK_COMMAND` on the same `make minikube-t2` invocation
+ (planner `T2_PREFLIGHT_PASS` then `PROFILE_UNHEALTHY` means the command
+ was missing); other transitions may leave it opt-in. Control
UI/Desktop Playwright remains opt-in via `T2_PLAYWRIGHT_COMMAND`. Both are
recorded as separate evidence statuses (`NOT_RUN` when optional;
`T2_REQUIRE_PLAYWRIGHT=true` refuses a missing journey). Product E2E scripts