Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
docs: record T2 targeted-sync health and stale port-forward codes
Alfredo Lopez7f4ef51a7bac
diff --git a/.cursor/rules/minikube-t0-t1-t2.mdc b/.cursor/rules/minikube-t0-t1-t2.mdcindex 6fc824a6c..73adfdb0c 100644--- a/.cursor/rules/minikube-t0-t1-t2.mdc+++ b/.cursor/rules/minikube-t0-t1-t2.mdc@@ -31,8 +31,12 @@ repository — never mix its PRs or clusters into this lane.- `make minikube-pre-gate-sync` is NOT T2.- Product E2E scripts (e.g. `scripts/e2e/e2e-hcc-rollout-readiness.sh`), CI,unit suites, and Playwright are separate lanes; none stands in for T2.-- Health/Playwright are opt-in via `T2_HEALTHCHECK_COMMAND` /- `T2_PLAYWRIGHT_COMMAND` (`T2_REQUIRE_PLAYWRIGHT=true` refuses `NOT_RUN`).+- Health is mandatory and bounded for `targeted-sync` via+ `T2_HEALTHCHECK_COMMAND`. After `T2_PREFLIGHT_PASS` the orchestrator still+ fail-louds `PROFILE_UNHEALTHY` if that command is unset — planner PASS is+ not T2. Health stays opt-in for bootstrap, full reconcile, and+ already-synced. Playwright is opt-in via `T2_PLAYWRIGHT_COMMAND`+ (`T2_REQUIRE_PLAYWRIGHT=true` refuses `NOT_RUN`).## Branch-profile UI port-forwards@@ -48,7 +52,14 @@ MINIKUBE_PROFILE=<owned-profile> \```Host-side hold for Control UI / Desktop. Profile-owned random ports only-(never `:3000` / `:8090`). Run on the host, not a sandboxed agent shell.+(never `:3000` / `:8090`). Run on the host, not a sandboxed agent shell:+`make ... branch-profile-pf` that prints `PF` lines and exits 0 can still+leave registered-but-dead pidfiles when the runner reaps `nohup kubectl`+children. The planner then fail-louds `PORT_FORWARD_CONFLICT` ++`DEVELOPMENT_SCOPE_REQUIRED` **before** a transition.+`T2_PORT_FORWARD_COMMAND` does not skip that pre-transition check. Restore+with a lasting host `branch-profile-pf` + `branch-profile-health`, then+re-enter `make minikube-t2`. Not a new profile. Do not `ls`/`cat` the cache.`make minikube-pf-all-bg` is a gate refresh only — it does not replace`branch-profile-pf`. Do not kill this lane's forwards.`branch-profile-pf-health` stops PFs on EXIT. Inner `pre-gate-sync` may usediff --git a/.cursor/skills/minikube-t0-t1-t2/SKILL.md b/.cursor/skills/minikube-t0-t1-t2/SKILL.mdindex 42f0976ea..370c5d096 100644--- a/.cursor/skills/minikube-t0-t1-t2/SKILL.md+++ b/.cursor/skills/minikube-t0-t1-t2/SKILL.md@@ -59,7 +59,13 @@ Before running anything, verify ALL of these:then `branch-profile-health`. Implementation:`.local-notes/minikube-profiles/branch-profile.sh`. Do not replacethat hold with `make minikube-pf-all-bg`. Do not start UI PFs from a- sandboxed agent shell. Do not kill this lane's `branch-profile-pf`.+ sandboxed agent shell. A `make ... branch-profile-pf` that prints `PF`+ lines and exits 0 can still leave registered-but-dead pidfiles when the+ runner reaps `nohup kubectl` children. The planner then fail-louds+ `PORT_FORWARD_CONFLICT` + `DEVELOPMENT_SCOPE_REQUIRED` before a+ transition; `T2_PORT_FORWARD_COMMAND` has not run yet. Restore with a+ lasting host hold + `branch-profile-health`, then re-enter T2. Do not+ kill this lane's `branch-profile-pf`.`branch-profile-pf-health` stops PFs on EXIT — not a lasting hold.- [ ] Run `make minikube-t2` from a host terminal with`T2_PORT_FORWARD_COMMAND` set to that `branch-profile-pf` command.@@ -111,8 +117,10 @@ transition = full-reconcile (deploy/* or charts/* changed, OR a requiredscript between runs.transition = targeted-sync (service-only diff)?- └── make minikube-t2 performs the targeted deploy; record it as a- targeted sync, never as a full reconcile.+ └── make minikube-t2 with T2_HEALTHCHECK_COMMAND set to the affected+ service's profile-owned user-facing journey. Record as targeted+ sync, never as a full reconcile. T2_PREFLIGHT_PASS then+ PROFILE_UNHEALTHY means the command was missing — not a T2 verdict.```Rules that override any shortcut idea:@@ -181,7 +189,11 @@ Report to the user exactly these lane statuses plus HEAD, profile, and theevidence path. `SKIPPED` is legitimate only for T0/T1 previously green on thesame HEAD (say so explicitly). `T2_HEALTHCHECK_COMMAND` is mandatory for`targeted-sync` and bounded by `T2_HEALTHCHECK_TIMEOUT_SECONDS` (default 120s);-it is optional for other transitions. `T2_PLAYWRIGHT_COMMAND` remains opt-in+it is optional for other transitions. For `workflow-recipes`, combine+`branch-profile-health` with an in-pod+`kubectl --context=<owned> -n control-plane exec deploy/workflow-recipes --+wget -qO- http://127.0.0.1:8082/health` — do not invent a host port.+`T2_PLAYWRIGHT_COMMAND` remains opt-in(`T2_REQUIRE_PLAYWRIGHT=true` refuses a missing journey).Evidence stays under the ignored `.local-notes/infra/runs/`. Never commit it.diff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.mdindex 2c8f021dc..81d9a5708 100644--- a/.cursor/skills/minikube-t0-t1-t2/reference.md+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md@@ -149,9 +149,9 @@ checkout `branch.mk` (`branch-profile-start`). Never invent ports or| `BOOTSTRAP_REQUIRED` | Profile missing/uninitialized, or the planner produced no transition. Standalone preflight refuses to bootstrap. | Run `MINIKUBE_PROFILE=<owned> CONTROL_API_REAL_PG_CONTEXT=<owned> make minikube-t2`; its internal planner makes `full-bootstrap` reachable. || `HEAD_MARKER_MISMATCH` | Pre-gate marker does not match this worktree/HEAD; the final T2 preflight selected something other than `already-synced`. | Re-run the full target on this HEAD so `pre-gate-sync` updates the marker. Never hand-edit the marker. || `IMAGE_MANIFEST_MISMATCH` | Deployed image provenance or the manifest `generated` stamp does not match the exact marker/worktree state. | Re-run the full target for the same owned profile; do not relabel or hand-edit the manifest. |-| `PORT_FORWARD_CONFLICT` | A port-forward for this profile is owned by a process not recorded for it (or not a real `kubectl`). | Identify the foreign owner; if it belongs to another lane, stop. Do not kill this lane's `branch-profile-pf`. Restore UI PFs with `make -f .local-notes/minikube-profiles/branch.mk branch-profile-pf`. |+| `PORT_FORWARD_CONFLICT` | A port-forward for this profile is owned by a process not recorded for it, is not a real `kubectl`, or the registered pidfile is stale/dead/ambiguous. A sandboxed `make ... branch-profile-pf` that exits 0 can leave registered-but-dead records; the planner then wraps this as `DEVELOPMENT_SCOPE_REQUIRED` **before** a transition. `T2_PORT_FORWARD_COMMAND` does not skip that check. | Identify a foreign owner and stop if it is another lane. Do not kill this lane's `branch-profile-pf`. Restore with a lasting host `branch-profile-pf` + `branch-profile-health`, then re-enter `make minikube-t2`. Not a new profile. || `PROFILE_BUSY` | Profile lock held. Live owner PID → genuinely busy. No valid owner PID → orphaned lock. | Live owner: wait or coordinate; never remove. Orphan: verify no T2 process owns the profile, then remove ONLY `$T2_LOCK_ROOT/<profile>.lock` and retry. Never remove the lock root. |-| `DEVELOPMENT_SCOPE_REQUIRED` | Preflight/final preflight failed a precondition, T2-only mode was attempted without `already-synced`, or the worktree is dirty (`worktree is dirty`). | Repair the first reported condition. If dirty: commit or stash, do not edit during the run, then re-enter T2. If T2-only was refused, run full `make minikube-t2`. |+| `DEVELOPMENT_SCOPE_REQUIRED` | Preflight/final preflight failed a precondition (including `PORT_FORWARD_CONFLICT` on a stale pidfile before a transition was selected), T2-only mode was attempted without `already-synced`, or the worktree is dirty (`worktree is dirty`). | Repair the **first** reported condition. Stale PF: lasting host hold, then re-enter T2. If dirty: commit or stash, do not edit during the run, then re-enter T2. If T2-only was refused, run full `make minikube-t2`. || `CERTIFICATION_REQUIRED` | Runtime-only was requested without valid exact-head T0/T1 evidence for the full ownership tuple. | Run the full target once; do not hand-create or copy evidence. || `SECRET_MISSING` / `CONFIGMAP_MISSING` | A required named runtime input is absent or unreadable. | Reconcile the same owned profile through the full target; never invent values. || `ZERO_TESTS_EXECUTED` | A lane was configured off or executed nothing (including a required-but-missing Playwright journey). | Re-run with the lane enabled, or supply the required `T2_PLAYWRIGHT_COMMAND`. |@@ -159,7 +159,7 @@ checkout `branch.mk` (`branch-profile-start`). Never invent ports or| `REAL_PG_REPORT_INCOMPLETE` | Vitest JSON did not prove the exact expected physical files. | Repair test selection/reporter output; do not accept suite counters as file identity. || `REAL_PG_SUITE_FAILED` | Reporter, test assertions, process exit, or parent result contract failed. | Iterate with standalone T1; once green, run full T2 for final exact-head lane evidence. || `POSTGRES_NOT_READY` | PostgreSQL precondition failed, or a PVC reset was requested without the exact expected UID. | Fix DB readiness; never guess a PVC UID. |-| `PROFILE_UNHEALTHY` | A required deployment is unready in a fail-loud check (`T2_PLAN_MODE=false`: standalone preflight or the final exact-head T2), the mandatory targeted-sync health command is missing, or a supplied user-facing health command failed or timed out. The orchestrator planner never emits this merely for an unready deployment — it selects `full-reconcile` instead. | For a targeted sync, supply the affected service's profile-owned `T2_HEALTHCHECK_COMMAND`; otherwise repair the reported health failure. Re-run on the same HEAD and profile. Do not run manual repair scripts between runs. |+| `PROFILE_UNHEALTHY` | A required deployment is unready in a fail-loud check (`T2_PLAN_MODE=false`: standalone preflight or the final exact-head T2), the mandatory targeted-sync health command is missing, or a supplied user-facing health command failed or timed out. The orchestrator planner never emits this merely for an unready deployment — it selects `full-reconcile` instead. | For a targeted sync, supply the affected service's profile-owned `T2_HEALTHCHECK_COMMAND` **on the same `make minikube-t2` invocation** (planner `T2_PREFLIGHT_PASS` is not enough). For `workflow-recipes`: `branch-profile-health` plus in-pod `wget` of `deploy/workflow-recipes` `/health`. Otherwise repair the reported health failure. Re-run on the same HEAD and profile. Do not run manual repair scripts between runs. || `NP08_HCC_AUTHORIZATION_FAILED` | The deployed Host-to-HCC authorization journey or cleanup failed after T0/T1 passed. | Repair the first NP08 failure, then use `minikube-t2-runtime` on the exact same profile/HEAD; never refresh/reissue in the test. || `PLAYWRIGHT_FAILED` | The opt-in user-visible journey failed after runtime and lane evidence passed. | Repair the journey and retry with `minikube-t2-runtime` on the same exact tuple. |diff --git a/AGENTS.md b/AGENTS.mdindex b36ef53fb..cd63e9c95 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -47,16 +47,21 @@ MINIKUBE_PROFILE=<owned-profile> \```This is the host-side hold for Control UI / Desktop. Run it on the host, not-from a sandboxed agent shell. `make minikube-pf-all-bg` is a gate refresh-only and must not replace `branch-profile-pf`. Do not kill this lane's-forwards. `branch-profile-pf-health` starts then stops PFs on EXIT — do not-use it as the lasting hold. Inner `pre-gate-sync` may use-`--skip-port-forwards`; never pass that globally into `make minikube-t2`.-`pre-gate-sync` can restart every deployment, which leaves the host hold-bound to terminated pods. Run `make minikube-t2` from a host terminal with-`T2_PORT_FORWARD_COMMAND` set to the `branch-profile-pf` command above; T2-runs it once after an in-run sync, before Health and Playwright, and records-`PortForwards=` in the evidence.+from a sandboxed agent shell. A `make ... branch-profile-pf` that prints `PF`+lines and exits 0 can still leave registered-but-dead pidfiles when the+runner reaps `nohup kubectl` children. The planner then fail-louds+`PORT_FORWARD_CONFLICT` + `DEVELOPMENT_SCOPE_REQUIRED` before a transition;+`T2_PORT_FORWARD_COMMAND` has not run yet. Restore with a lasting host hold+and `branch-profile-health`, then re-enter `make minikube-t2`. Not a new+profile. `make minikube-pf-all-bg` is a gate refresh only and must not+replace `branch-profile-pf`. Do not kill this lane's forwards.+`branch-profile-pf-health` starts then stops PFs on EXIT — do not use it as+the lasting hold. Inner `pre-gate-sync` may use `--skip-port-forwards`; never+pass that globally into `make minikube-t2`. `pre-gate-sync` can restart every+deployment, which leaves the host hold bound to terminated pods. Run+`make minikube-t2` from a host terminal with `T2_PORT_FORWARD_COMMAND` set to+the `branch-profile-pf` command above; T2 runs it once after an in-run sync,+before Health and Playwright, and records `PortForwards=` in the evidence.Port-forwards are owned by atomic `0600` records bound to the exact profile,context, canonical worktree, namespace, Service, local/remote ports, PID,@@ -245,7 +250,14 @@ CI, Control UI/Desktop Playwright, and product E2E scripts such as`scripts/e2e/e2e-hcc-rollout-readiness.sh` are separate evidence lanes. Onelane does not stand in for another. User-facing health is mandatory and boundedfor a `targeted-sync` transition via `T2_HEALTHCHECK_COMMAND`; it remains-opt-in for bootstrap, full reconcile, and already-synced runs. Playwright is+opt-in for bootstrap, full reconcile, and already-synced runs. A planner+`T2_PREFLIGHT_PASS` / `transition=targeted-sync` followed by+`PROFILE_UNHEALTHY` (`targeted sync requires a profile-owned user-facing+health command`) means the command was missing — re-enter `make minikube-t2`+with it set. For `workflow-recipes`, use `branch-profile-health` plus+`kubectl --context=<owned> -n control-plane exec deploy/workflow-recipes --+wget -qO- http://127.0.0.1:8082/health` (in-pod `/health`; do not invent a+host port). Playwright isopt-in via `T2_PLAYWRIGHT_COMMAND` (`T2_REQUIRE_PLAYWRIGHT=true` refuses`NOT_RUN`). Private operational state,generated ports, profile metadata, logs, and evidence belong under the ignoreddiff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.mdindex d8bac646c..8c7f7fcb6 100644--- a/docs/testing/minikube-t2-runbook.md+++ b/docs/testing/minikube-t2-runbook.md@@ -68,8 +68,14 @@ HARD DENY: do not `ls`/`cat` `~/.cache/clerum/minikube-profiles/`.This is the host-side hold for Control UI / Desktop. Profile-owned randomports only (never shared `:3000`/`:8090`). `make minikube-pf-all-bg` is agate refresh only; it must not replace `branch-profile-pf`. Do not start UI-PFs from a sandboxed agent shell (hooks/PATH; runners clean children). Run-the make target on the host. Do not kill this lane's `branch-profile-pf`.+PFs from a sandboxed agent shell (hooks/PATH; runners clean children). A+`make ... branch-profile-pf` that prints `PF` lines and exits 0 can still+leave registered-but-dead pidfiles; the planner then fail-louds+`PORT_FORWARD_CONFLICT` + `DEVELOPMENT_SCOPE_REQUIRED` before a transition.+`T2_PORT_FORWARD_COMMAND` does not skip that pre-transition check. Restore+with a lasting host hold + `branch-profile-health`, then re-enter+`make minikube-t2`. Run the make target on the host. Do not kill this+lane's `branch-profile-pf`.Inner `pre-gate-sync` may use `--skip-port-forwards`; never pass thatglobally into `make minikube-t2`. `branch-profile-pf-health` starts PFs thenSTOPS them on EXIT — do not use it as the lasting hold.@@ -382,7 +388,9 @@ non-T2 scope guard.image manifest is current, PostgreSQL and required namespaces/Services arepresent, deployments are Ready, and no foreign `kubectl port-forward` ownsthis profile. A targeted sync requires a bounded user-facing journey via- `T2_HEALTHCHECK_COMMAND`; other transitions may leave it opt-in. Control+ `T2_HEALTHCHECK_COMMAND` on the same `make minikube-t2` invocation+ (planner `T2_PREFLIGHT_PASS` then `PROFILE_UNHEALTHY` means the command+ was missing); other transitions may leave it opt-in. ControlUI/Desktop Playwright remains opt-in via `T2_PLAYWRIGHT_COMMAND`. Both arerecorded as separate evidence statuses (`NOT_RUN` when optional;`T2_REQUIRE_PLAYWRIGHT=true` refuses a missing journey). Product E2E scripts