Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix: judge T1 from the JSON reporter, not leftover vitest exit

Alfredo Lopezd234836b7519
diff --git a/.cursor/rules/minikube-t0-t1-t2.mdc b/.cursor/rules/minikube-t0-t1-t2.mdc
index 438ec5379..bfd48354b 100644
--- a/.cursor/rules/minikube-t0-t1-t2.mdc
+++ b/.cursor/rules/minikube-t0-t1-t2.mdc
@@ -42,8 +42,9 @@ repository — never mix its PRs or clusters into this lane.
current-context.
- Bootstrap/reconcile must build the worktree: `IMAGE_SOURCE=local`.
- T1 is fail-loud: unavailable DSN or zero executed tests is FAIL, never green.
- Role-reset suites use the throwaway `postgres:16-alpine`, never shared
- `control-postgres`.
+ The JSON reporter is the suite verdict; a leftover Vitest exit after a
+ complete green reporter is not a failed suite. Role-reset suites use the
+ throwaway `postgres:16-alpine`, never shared `control-postgres`.
- Evidence lives under ignored `.local-notes/infra/runs/`; never commit it.
Full workflow: skill `.cursor/skills/minikube-t0-t1-t2/SKILL.md`.
diff --git a/.cursor/rules/minikube-t2-harness.mdc b/.cursor/rules/minikube-t2-harness.mdc
index 5634fbfea..e0988178c 100644
--- a/.cursor/rules/minikube-t2-harness.mdc
+++ b/.cursor/rules/minikube-t2-harness.mdc
@@ -28,8 +28,9 @@ Invariants that must never regress:
and plan state `already-synced`; anything else is `HEAD_MARKER_MISMATCH`.
- Planner mode `T2_PLAN_MODE=true` exists only inside the orchestrator so
`full-bootstrap` is reachable; standalone preflight stays fail-loud.
-- T1 stays fail-loud (no green on unavailable DSN or zero tests); role-reset
- suites keep the throwaway Postgres 16.
+- T1 stays fail-loud (no green on unavailable DSN or zero tests); the JSON
+ reporter is the suite verdict, not a leftover Vitest process exit after a
+ complete green reporter. Role-reset suites keep the throwaway Postgres 16.
- `t2.sh` keeps `IMAGE_SOURCE=local` for bootstrap/reconcile and calls
`pre-gate-sync` with `--skip-port-forwards`.
- `t2_process_check` accepts only real `kubectl` (`comm=kubectl`) port-forward
diff --git a/.cursor/skills/minikube-t0-t1-t2/SKILL.md b/.cursor/skills/minikube-t0-t1-t2/SKILL.md
index dc667c6a4..882d7877d 100644
--- a/.cursor/skills/minikube-t0-t1-t2/SKILL.md
+++ b/.cursor/skills/minikube-t0-t1-t2/SKILL.md
@@ -78,7 +78,9 @@ T1 runs inside `make minikube-t2`, or explicitly via
`make minikube-t2-real-postgres MINIKUBE_PROFILE=<owned-profile>`.
- T1 is fail-loud: an unavailable DSN, an isolated server that did not start,
- or zero executed tests is FAIL. Never report green from skipped suites.
+ or zero executed tests is FAIL. Never report green from skipped suites. The
+ JSON reporter is the suite verdict; a leftover Vitest process exit after a
+ complete green reporter is not a failed suite.
- Role-reset suites (cluster-global role drop/rewrite) run against a
throwaway `postgres:16-alpine`, never the shared `control-postgres`.
- The admin DSN is resolved in-process from the cluster Secret; never print,
diff --git a/.cursor/skills/minikube-t0-t1-t2/reference.md b/.cursor/skills/minikube-t0-t1-t2/reference.md
index b6154f3ec..141d81b94 100644
--- a/.cursor/skills/minikube-t0-t1-t2/reference.md
+++ b/.cursor/skills/minikube-t0-t1-t2/reference.md
@@ -33,8 +33,10 @@ Companion to `SKILL.md`. Source of truth: `scripts/minikube/t2.sh`,
`npm ci`, generated ports/profile metadata, or anything secret-like. Run
`make minikube-t2-public-boundary` first.
- **Do not weaken fail-loud T1**: no green on unavailable DSN, zero executed
- tests, or silently skipped suites; role-reset suites stay on the throwaway
- `postgres:16-alpine` (#412), never shared `control-postgres`.
+ tests, or silently skipped suites; the JSON reporter is the suite verdict,
+ not a leftover Vitest process exit after a complete green reporter.
+ Role-reset suites stay on the throwaway `postgres:16-alpine` (#412), never
+ shared `control-postgres`.
## Stable failure codes
diff --git a/AGENTS.md b/AGENTS.md
index 7e5a527f3..80924984c 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -95,8 +95,10 @@ standalone preflight refuses to call `pre-gate-sync` on an incomplete profile;
`full-bootstrap`. Real PostgreSQL suites are opt-in in the ordinary test
matrix, but a T1 run that requires them must fail when the database/DSN is
unavailable or when zero tests execute; a green run must never be produced by
-silently skipping the suites. Suites that drop or rewrite cluster-global roles
-must use the harness throwaway Postgres 16, never the shared `control-postgres`.
+silently skipping the suites. The JSON reporter is the suite verdict; a leftover
+Vitest process exit after a complete green reporter is not a failed suite.
+Suites that drop or rewrite cluster-global roles must use the harness throwaway
+Postgres 16, never the shared `control-postgres`.
T0 (static/unit/contract checks), T1 (real PostgreSQL), T2 (validated runtime),
CI, Control UI/Desktop Playwright, and product E2E scripts such as
diff --git a/CLAUDE.md b/CLAUDE.md
index e7aa5811e..26cb6cfeb 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -16,9 +16,11 @@ evidence rules in `AGENTS.md` and `docs/testing/minikube-t2-runbook.md`.
produced only by the final exact-head preflight inside `make minikube-t2` (or
`make minikube-t2-runtime` after T0 and T1 are already green on the same HEAD).
`make minikube-pre-gate-sync` alone is not T2. Playwright and product E2E
-scripts are separate lanes. Follow the rule `.cursor/rules/minikube-t0-t1-t2.mdc`
-and the skill `.cursor/skills/minikube-t0-t1-t2/SKILL.md` for the certification
-workflow.
+scripts are separate lanes. T1 judges the JSON reporter (expected files,
+executed/passed, zero failures, zero pending); a leftover Vitest process exit
+after a complete green reporter is not a failed suite. Follow the rule
+`.cursor/rules/minikube-t0-t1-t2.mdc` and the skill
+`.cursor/skills/minikube-t0-t1-t2/SKILL.md` for the certification workflow.
## Logging standard
diff --git a/docs/testing/minikube-t2-runbook.md b/docs/testing/minikube-t2-runbook.md
index 05ba7a3a8..f2e8ba17c 100644
--- a/docs/testing/minikube-t2-runbook.md
+++ b/docs/testing/minikube-t2-runbook.md
@@ -131,7 +131,9 @@ never share live `control-postgres` (#412). CI continues to use
`control-postgres`, except the role-reset suites which use an isolated
Postgres 16; the lane reports `PASS`, `FAIL`, `SKIPPED`, and `NOT_RUN`
separately and fails on an unavailable DSN, an isolated server that did not
- start, or zero executed tests.
+ start, or zero executed tests. The JSON reporter is the suite verdict
+ (expected files, executed/passed, zero failures, zero pending); a leftover
+ Vitest process exit after a complete green reporter is not a failed suite.
* **T2** — the final exact-head preflight inside `make minikube-t2` (or
`make minikube-t2-runtime`): the marker matches this worktree/`HEAD`, the
image manifest is current, PostgreSQL and required namespaces/Services are
diff --git a/scripts/e2e/minikube-real-postgres.sh b/scripts/e2e/minikube-real-postgres.sh
index 92d296d36..cd1eeed4f 100755
--- a/scripts/e2e/minikube-real-postgres.sh
+++ b/scripts/e2e/minikube-real-postgres.sh
@@ -250,22 +250,20 @@ run_suite() {
json_file="$T1_TMP_DIR/$(printf '%s' "$package-$lane" | tr / _).json"
printf '[minikube-t1] running %s %s Real PostgreSQL suites (%s files)\n' \
"$package" "$lane" "$expected_files"
- if ! (
+ # T1 verdict is the JSON reporter (expected files, executed/passed, zero
+ # failures, zero pending). Capture npm's exit but do not treat a leftover
+ # process exit after a complete green reporter as a failed suite.
+ suite_status=0
+ (
cd "$PROJECT_DIR/$package"
CONTROL_API_REAL_PG_ADMIN_URL="$admin_url" \
CONTROL_API_REAL_PG_REQUIRED=1 FORCE_COLOR=0 NO_COLOR=1 \
npm test -- --run "${vitest_filters[@]}" --reporter=json --outputFile="$json_file"
- ) >"$log_file" 2>&1; then
- sanitize_file "$log_file"
- sanitize_file "$json_file"
- cat "$log_file" >&2 || true
- cat "$json_file" >&2 || true
- T1_NEXT_COMMAND='repair the first failing Real PostgreSQL suite, then re-run T1 on the same HEAD'
- die_t1 REAL_PG_SUITE_FAILED "Real PostgreSQL suite failed in $package ($lane)"
- fi
+ ) >"$log_file" 2>&1 || suite_status=$?
sanitize_file "$log_file"
sanitize_file "$json_file"
[ -s "$json_file" ] || {
+ cat "$log_file" >&2 || true
T1_NEXT_COMMAND='inspect the Vitest reporter output, then re-run T1'
die_t1 ZERO_TESTS_EXECUTED "Real PostgreSQL reporter produced no result for $package ($lane)"
}
@@ -273,7 +271,7 @@ run_suite() {
# comparable to the number of physical *realPostgres*.test.ts files above.
# Validate the reporter's own total/passed suite counts instead; the file
# discovery count remains a separate zero-selection guard.
- stats="$(python3 - "$json_file" "$expected_files" <<'PY'
+ if ! stats="$(python3 - "$json_file" "$expected_files" <<'PY'
import json
import sys
result = json.loads(open(sys.argv[1]).read())
@@ -293,7 +291,12 @@ print(total_suites, passed_suites, failed_suites, passed_files, passed_tests,
failed_tests, pending_files, pending_tests, total_tests, int(success),
expected, len(files))
PY
- )"
+ )"; then
+ cat "$log_file" >&2 || true
+ cat "$json_file" >&2 || true
+ T1_NEXT_COMMAND='inspect the Vitest reporter output, then re-run T1'
+ die_t1 ZERO_TESTS_EXECUTED "Real PostgreSQL reporter was unreadable for $package ($lane)"
+ fi
read -r total_suites passed_suites failed_suites passed_files passed_tests \
failed_tests pending_files pending_tests total_tests success expected \
reported_files <<< "$stats"
@@ -308,9 +311,15 @@ PY
die_t1 REAL_PG_SUITE_FAILED "Real PostgreSQL reporter did not pass every suite in $package ($lane)"
fi
if [ "$total_tests" -le 0 ] || [ "$passed_tests" -le 0 ] || [ "$pending_files" -ne 0 ] || [ "$pending_tests" -ne 0 ]; then
+ cat "$log_file" >&2 || true
+ cat "$json_file" >&2 || true
T1_NEXT_COMMAND='repair the test selection or database route; a T1 run cannot silently skip suites'
die_t1 ZERO_TESTS_EXECUTED "Real PostgreSQL lane reported zero tests or skips in $package ($lane)"
fi
+ if [ "$suite_status" -ne 0 ]; then
+ printf '[minikube-t1] vitest process exited %s after a complete green reporter for %s %s; T1 follows the reporter\n' \
+ "$suite_status" "$package" "$lane"
+ fi
printf '[minikube-t1] PASS %s %s files=%s tests=%s skipped=0\n' \
"$package" "$lane" "$passed_files" "$passed_tests"
}
diff --git a/scripts/tests/test-minikube-t2-contract.sh b/scripts/tests/test-minikube-t2-contract.sh
index 12fa9d7e7..995c04cdd 100755
--- a/scripts/tests/test-minikube-t2-contract.sh
+++ b/scripts/tests/test-minikube-t2-contract.sh
@@ -62,6 +62,12 @@ if grep -Eq 'describe\.skip|pending_tests.*allow|exclude.*realPostgresMigration'
fi
grep -Fq 'T1_REDACT_PASSWORD="${PG_PASSWORD}"' "$T1" || grep -Fq 'T1_REDACT_PASSWORD="$PG_PASSWORD"' "$T1"
grep -Fq -- '--reporter=json' "$T1"
+grep -Fq '|| suite_status=$?' "$T1"
+grep -Fq 'complete green reporter' "$T1"
+if grep -Fq 'Real PostgreSQL suite failed in $package ($lane)' "$T1"; then
+ echo 'FAIL: T1 still treats npm exit as a suite failure before the JSON reporter' >&2
+ exit 1
+fi
grep -Fq 'pending_tests' "$T1"
grep -Fq 'numTotalTests' "$T1"
grep -Fq 'numTotalTestSuites' "$T1"
diff --git a/scripts/tests/test-minikube-t2-scenarios.sh b/scripts/tests/test-minikube-t2-scenarios.sh
index 5ab4c42dc..945df66b4 100755
--- a/scripts/tests/test-minikube-t2-scenarios.sh
+++ b/scripts/tests/test-minikube-t2-scenarios.sh
@@ -251,6 +251,9 @@ grep -Fq 'CONTROL_API_REAL_PG_REQUIRED=1' "$ROOT/scripts/e2e/minikube-real-postg
grep -Fq 'ZERO_TESTS_EXECUTED' "$ROOT/scripts/e2e/minikube-real-postgres.sh"
grep -Fq 'start_isolated_postgres' "$ROOT/scripts/e2e/minikube-real-postgres.sh"
grep -Fq 'ISOLATED_DSN' "$ROOT/scripts/e2e/minikube-real-postgres.sh"
+grep -Fq '|| suite_status=$?' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 must capture vitest exit without aborting before the reporter'
+grep -Fq 'complete green reporter' "$ROOT/scripts/e2e/minikube-real-postgres.sh" || fail 'T1 must document reporter-authoritative verdict'
+grep -Fq 'Real PostgreSQL suite failed in $package ($lane)' "$ROOT/scripts/e2e/minikube-real-postgres.sh" && fail 'T1 still treats npm exit as a suite failure before the JSON reporter'
grep -Fq 'T2_LOCK_ROOT' "$COMMON"
grep -Fq 'trap t2_lock_release EXIT INT TERM' "$COMMON"
grep -Fq 'PORT_FORWARD_CONFLICT' "$COMMON"