Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
fix: judge T1 from the JSON reporter, not leftover vitest exit
Alfredo Lopezd234836b7519
diff --git a/.cursor/rules/minikube-t0-t1-t2.mdc b/.cursor/rules/minikube-t0-t1-t2.mdcindex 438ec5379..bfd48354b 100644--- a/.cursor/rules/minikube-t0-t1-t2.mdc+++ b/.cursor/rules/minikube-t0-t1-t2.mdc@@ -42,8 +42,9 @@ repository — never mix its PRs or clusters into this lane.current-context.- Bootstrap/reconcile must build the worktree: `IMAGE_SOURCE=local`.- T1 is fail-loud: unavailable DSN or zero executed tests is FAIL, never green.- Role-reset suites use the throwaway `postgres:16-alpine`, never shared- `control-postgres`.+ The JSON reporter is the suite verdict; a leftover Vitest exit after a+ complete green reporter is not a failed suite. Role-reset suites use the+ throwaway `postgres:16-alpine`, never shared `control-postgres`.- Evidence lives under ignored `.local-notes/infra/runs/`; never commit it.Full workflow: skill `.cursor/skills/minikube-t0-t1-t2/SKILL.md`.diff --git a/.cursor/rules/minikube-t2-harness.mdc b/.cursor/rules/minikube-t2-harness.mdcindex 5634fbfea..e0988178c 100644--- a/.cursor/rules/minikube-t2-harness.mdc+++ b/.cursor/rules/minikube-t2-harness.mdc@@ -28,8 +28,9 @@ Invariants that must never regress:and plan state `already-synced`; anything else is `HEAD_MARKER_MISMATCH`.- Planner mode `T2_PLAN_MODE=true` exists only inside the orchestrator so`full-bootstrap` is reachable; standalone preflight stays fail-loud.-- T1 stays fail-loud (no green on unavailable DSN or zero tests); role-reset- suites keep the throwaway Postgres 16.+- T1 stays fail-loud (no green on unavailable DSN or zero tests); the JSON+ reporter is the suite verdict, not a leftover Vitest process exit after a+ complete green reporter. Role-reset suites keep the throwaway Postgres 16.- `t2.sh` keeps `IMAGE_SOURCE=local` for bootstrap/reconcile and calls`pre-gate-sync` with `--skip-port-forwards`.- `t2_process_check` accepts only real `kubectl` (`comm=kubectl`) port-forwarddiff --git a/.cursor/skills/minikube-t0-t1-t2/SKILL.md b/.cursor/skills/minikube-t0-t1-t2/SKILL.mdindex dc667c6a4..882d7877d 100644--- a/.cursor/skills/minikube-t0-t1-t2/SKILL.md+++ b/.cursor/skills/minikube-t0-t1-t2/SKILL.md@@ -78,7 +78,9 @@ T1 runs inside `make minikube-t2`, or explicitly via`make minikube-t2-real-postgres MINIKUBE_PROFILE=<owned-profile>`.- T1 is fail-loud: an unavailable DSN, an isolated server that did not start,- or zero executed tests is FAIL. Never report green from skipped suites.+ or zero executed tests is FAIL. Never report green from skipped suites. The+ JSON reporter is the suite verdict; a leftover Vitest process exit after a+ complete green reporter is not a failed suite.- Role-reset suites (cluster-global role drop/rewrite) run against athrowaway `postgres:16-alpine`, never the shared `control-postgres`.- The admin DSN is resolved in-process from the cluster Secret; never print,diff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.mdindex b6154f3ec..141d81b94 100644--- a/.cursor/skills/minikube-t0-t1-t2/reference.md+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md@@ -33,8 +33,10 @@ Companion to `SKILL.md`. Source of truth: `scripts/minikube/t2.sh`,`npm ci`, generated ports/profile metadata, or anything secret-like. Run`make minikube-t2-public-boundary` first.- **Do not weaken fail-loud T1**: no green on unavailable DSN, zero executed- tests, or silently skipped suites; role-reset suites stay on the throwaway- `postgres:16-alpine` (#412), never shared `control-postgres`.+ tests, or silently skipped suites; the JSON reporter is the suite verdict,+ not a leftover Vitest process exit after a complete green reporter.+ Role-reset suites stay on the throwaway `postgres:16-alpine` (#412), never+ shared `control-postgres`.## Stable failure codesdiff --git a/AGENTS.md b/AGENTS.mdindex 7e5a527f3..80924984c 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -95,8 +95,10 @@ standalone preflight refuses to call `pre-gate-sync` on an incomplete profile;`full-bootstrap`. Real PostgreSQL suites are opt-in in the ordinary testmatrix, but a T1 run that requires them must fail when the database/DSN isunavailable or when zero tests execute; a green run must never be produced by-silently skipping the suites. Suites that drop or rewrite cluster-global roles-must use the harness throwaway Postgres 16, never the shared `control-postgres`.+silently skipping the suites. The JSON reporter is the suite verdict; a leftover+Vitest process exit after a complete green reporter is not a failed suite.+Suites that drop or rewrite cluster-global roles must use the harness throwaway+Postgres 16, never the shared `control-postgres`.T0 (static/unit/contract checks), T1 (real PostgreSQL), T2 (validated runtime),CI, Control UI/Desktop Playwright, and product E2E scripts such asdiff --git a/CLAUDE.md b/CLAUDE.mdindex e7aa5811e..26cb6cfeb 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -16,9 +16,11 @@ evidence rules in `AGENTS.md` and `docs/testing/minikube-t2-runbook.md`.produced only by the final exact-head preflight inside `make minikube-t2` (or`make minikube-t2-runtime` after T0 and T1 are already green on the same HEAD).`make minikube-pre-gate-sync` alone is not T2. Playwright and product E2E-scripts are separate lanes. Follow the rule `.cursor/rules/minikube-t0-t1-t2.mdc`-and the skill `.cursor/skills/minikube-t0-t1-t2/SKILL.md` for the certification-workflow.+scripts are separate lanes. T1 judges the JSON reporter (expected files,+executed/passed, zero failures, zero pending); a leftover Vitest process exit+after a complete green reporter is not a failed suite. Follow the rule+`.cursor/rules/minikube-t0-t1-t2.mdc` and the skill+`.cursor/skills/minikube-t0-t1-t2/SKILL.md` for the certification workflow.## Logging standarddiff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.mdindex 05ba7a3a8..f2e8ba17c 100644--- a/docs/testing/minikube-t2-runbook.md+++ b/docs/testing/minikube-t2-runbook.md@@ -131,7 +131,9 @@ never share live `control-postgres` (#412). CI continues to use`control-postgres`, except the role-reset suites which use an isolatedPostgres 16; the lane reports `PASS`, `FAIL`, `SKIPPED`, and `NOT_RUN`separately and fails on an unavailable DSN, an isolated server that did not- start, or zero executed tests.+ start, or zero executed tests. The JSON reporter is the suite verdict+ (expected files, executed/passed, zero failures, zero pending); a leftover+ Vitest process exit after a complete green reporter is not a failed suite.* **T2** — the final exact-head preflight inside `make minikube-t2` (or`make minikube-t2-runtime`): the marker matches this worktree/`HEAD`, theimage manifest is current, PostgreSQL and required namespaces/Services arediff --git a/scripts/e2e/minikube-real-postgres.sh b/scripts/e2e/minikube-real-postgres.shindex 92d296d36..cd1eeed4f 100755--- a/scripts/e2e/minikube-real-postgres.sh+++ b/scripts/e2e/minikube-real-postgres.sh@@ -250,22 +250,20 @@ run_suite() {json_file="$T1_TMP_DIR/$(printf '%s' "$package-$lane" | tr / _).json"printf '[minikube-t1] running %s %s Real PostgreSQL suites (%s files)\n' \"$package" "$lane" "$expected_files"- if ! (+ # T1 verdict is the JSON reporter (expected files, executed/passed, zero+ # failures, zero pending). Capture npm's exit but do not treat a leftover+ # process exit after a complete green reporter as a failed suite.+ suite_status=0+ (cd "$PROJECT_DIR/$package"CONTROL_API_REAL_PG_ADMIN_URL="$admin_url" \CONTROL_API_REAL_PG_REQUIRED=1 FORCE_COLOR=0 NO_COLOR=1 \npm test -- --run "${vitest_filters[@]}" --reporter=json --outputFile="$json_file"- ) >"$log_file" 2>&1; then- sanitize_file "$log_file"- sanitize_file "$json_file"- cat "$log_file" >&2 || true- cat "$json_file" >&2 || true- T1_NEXT_COMMAND='repair the first failing Real PostgreSQL suite, then re-run T1 on the same HEAD'- die_t1 REAL_PG_SUITE_FAILED "Real PostgreSQL suite failed in $package ($lane)"- fi+ ) >"$log_file" 2>&1 || suite_status=$?sanitize_file "$log_file"sanitize_file "$json_file"[ -s "$json_file" ] || {+ cat "$log_file" >&2 || trueT1_NEXT_COMMAND='inspect the Vitest reporter output, then re-run T1'die_t1 ZERO_TESTS_EXECUTED "Real PostgreSQL reporter produced no result for $package ($lane)"}@@ -273,7 +271,7 @@ run_suite() {# comparable to the number of physical *realPostgres*.test.ts files above.# Validate the reporter's own total/passed suite counts instead; the file# discovery count remains a separate zero-selection guard.- stats="$(python3 - "$json_file" "$expected_files" <<'PY'+ if ! stats="$(python3 - "$json_file" "$expected_files" <<'PY'import jsonimport sysresult = json.loads(open(sys.argv[1]).read())@@ -293,7 +291,12 @@ print(total_suites, passed_suites, failed_suites, passed_files, passed_tests,failed_tests, pending_files, pending_tests, total_tests, int(success),expected, len(files))PY- )"+ )"; then+ cat "$log_file" >&2 || true+ cat "$json_file" >&2 || true+ T1_NEXT_COMMAND='inspect the Vitest reporter output, then re-run T1'+ die_t1 ZERO_TESTS_EXECUTED "Real PostgreSQL reporter was unreadable for $package ($lane)"+ firead -r total_suites passed_suites failed_suites passed_files passed_tests \failed_tests pending_files pending_tests total_tests success expected \reported_files <<< "$stats"@@ -308,9 +311,15 @@ PYdie_t1 REAL_PG_SUITE_FAILED "Real PostgreSQL reporter did not pass every suite in $package ($lane)"fiif [ "$total_tests" -le 0 ] || [ "$passed_tests" -le 0 ] || [ "$pending_files" -ne 0 ] || [ "$pending_tests" -ne 0 ]; then+ cat "$log_file" >&2 || true+ cat "$json_file" >&2 || trueT1_NEXT_COMMAND='repair the test selection or database route; a T1 run cannot silently skip suites'die_t1 ZERO_TESTS_EXECUTED "Real PostgreSQL lane reported zero tests or skips in $package ($lane)"fi+ if [ "$suite_status" -ne 0 ]; then+ printf '[minikube-t1] vitest process exited %s after a complete green reporter for %s %s; T1 follows the reporter\n' \+ "$suite_status" "$package" "$lane"+ fiprintf '[minikube-t1] PASS %s %s files=%s tests=%s skipped=0\n' \"$package" "$lane" "$passed_files" "$passed_tests"}diff --git a/scripts/tests/test-minikube-t2-contract.sh b/scripts/tests/test-minikube-t2-contract.shindex 12fa9d7e7..995c04cdd 100755--- a/scripts/tests/test-minikube-t2-contract.sh+++ b/scripts/tests/test-minikube-t2-contract.sh@@ -62,6 +62,12 @@ if grep -Eq 'describe\.skip|pending_tests.*allow|exclude.*realPostgresMigration'figrep -Fq 'T1_REDACT_PASSWORD="${PG_PASSWORD}"' "$T1" || grep -Fq 'T1_REDACT_PASSWORD="$PG_PASSWORD"' "$T1"grep -Fq -- '--reporter=json' "$T1"+grep -Fq '|| suite_status=$?' "$T1"+grep -Fq 'complete green reporter' "$T1"+if grep -Fq 'Real PostgreSQL suite failed in $package ($lane)' "$T1"; then+ echo 'FAIL: T1 still treats npm exit as a suite failure before the JSON reporter' >&2+ exit 1+figrep -Fq 'pending_tests' "$T1"grep -Fq 'numTotalTests' "$T1"grep -Fq 'numTotalTestSuites' "$T1"diff --git a/scripts/tests/test-minikube-t2-scenarios.sh b/scripts/tests/test-minikube-t2-scenarios.shindex 5ab4c42dc..945df66b4 100755--- a/scripts/tests/test-minikube-t2-scenarios.sh+++ b/scripts/tests/test-minikube-t2-scenarios.sh@@ -251,6 +251,9 @@ grep -Fq 'CONTROL_API_REAL_PG_REQUIRED=1' "$ROOT/scripts/e2e/minikube-real-postggrep -Fq 'ZERO_TESTS_EXECUTED' "$ROOT/scripts/e2e/minikube-real-postgres.sh"grep -Fq 'start_isolated_postgres' "$ROOT/scripts/e2e/minikube-real-postgres.sh"grep -Fq 'ISOLATED_DSN' "$ROOT/scripts/e2e/minikube-real-postgres.sh"+grep -Fq '|| suite_status=$?' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 must capture vitest exit without aborting before the reporter'+grep -Fq 'complete green reporter' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 must document reporter-authoritative verdict'+grep -Fq 'Real PostgreSQL suite failed in $package ($lane)' "$ROOT/scripts/e2e/minikube-real-postgres.sh" && fail 'T1 still treats npm exit as a suite failure before the JSON reporter'grep -Fq 'T2_LOCK_ROOT' "$COMMON"grep -Fq 'trap t2_lock_release EXIT INT TERM' "$COMMON"grep -Fq 'PORT_FORWARD_CONFLICT' "$COMMON"