Skip to content

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

evals: classify Vitest collection failures before scoring

Louis Beaumontcb479138eb3f
diff --git a/evals/coding-agent/README.md b/evals/coding-agent/README.md
index 1a3129ec7..797f92d06 100644
--- a/evals/coding-agent/README.md
+++ b/evals/coding-agent/README.md
@@ -51,7 +51,8 @@ the success denominator. They are never silently converted into model failures.
Grader timeouts and terminating signals cannot establish a failing baseline:
`--verify` requires a behavioral `fail` followed by an oracle `pass`.
Known command-unavailable exits (126/127), Bun unhandled test-load errors, and
-Node missing-module/syntax diagnostics are also reported as `error`, with a
+Node missing-module/syntax diagnostics and Vitest URL-load collection failures
+with zero tests are also reported as `error`, with a
`grader_error_kind` in the result. A missing import followed by an oracle pass
therefore cannot certify a regression. Plain failed assertions and successful
commands containing diagnostic words retain their previous outcomes.
@@ -64,8 +65,9 @@ change, caused it. Report error counts and inspect candidate-caused errors befor
comparing model success rates. No isolation or model-quality claim follows.
Run the synthetic runner controls with `bun test ./evals/coding-agent/run.test.ts`.
-Seventeen end-to-end controls cover baseline/reference timeouts and signals,
-missing ESM/CommonJS modules, Node/Bun syntax and Bun import errors, unavailable
+Nineteen end-to-end controls cover baseline/reference timeouts and signals,
+missing ESM/CommonJS modules, Node/Bun syntax and Bun import errors, Vitest
+collection failures and quoted diagnostics followed by real assertions, unavailable
or non-executable commands, genuine failures, assertions quoting diagnostic words or complete diagnostic blocks,
already-passing baselines and exclusion of known setup/process errors from scores.
The controls use temporary local Git fixtures and invoke no model or provider.
diff --git a/evals/coding-agent/grader-outcome.mjs b/evals/coding-agent/grader-outcome.mjs
index ac613b2f0..f77d6f820 100644
--- a/evals/coding-agent/grader-outcome.mjs
+++ b/evals/coding-agent/grader-outcome.mjs
@@ -10,6 +10,14 @@ export function classifyGraderError(grader) {
const stderr = (grader.stderr ?? "").replace(/\x1b\[[0-9;]*m/g, "");
if (/^# Unhandled error between tests\s*$/m.test(stderr) &&
/^\s*\d+ errors?\s*$/m.test(stderr)) return "bun_unhandled_error";
+ // Vitest prints collection errors on stderr but its final test count on
+ // stdout. Use the last summary so diagnostic text printed by a test cannot
+ // hide a later genuine assertion failure. Unknown formats still need review.
+ const stdout = (grader.stdout ?? "").replace(/\x1b\[[0-9;]*m/g, "");
+ const summary = [...stdout.matchAll(/^\s*Tests\s+(.+)$/gm)].at(-1)?.[1]?.trim();
+ if (summary === "no tests" && /^\s*Test Files\s+\d+ failed/m.test(stdout) &&
+ /Failed Suites [1-9]/.test(stderr) &&
+ /^Error: Failed to load url /m.test(stderr)) return "vitest_collection_error";
// Only the first thrown-error header classifies a Node failure. Assertion
// messages may quote complete setup diagnostics on later lines.
const header = stderr.match(/^([A-Za-z]*Error)(?: \[([A-Z_0-9]+)\])?:[^\n]*/m);
diff --git a/evals/coding-agent/run.test.ts b/evals/coding-agent/run.test.ts
index a9a22025e..a5f361fc0 100644
--- a/evals/coding-agent/run.test.ts
+++ b/evals/coding-agent/run.test.ts
@@ -34,6 +34,8 @@ test("process and setup failures cannot prove a regression or count as scored fa
{ id: "reference-timeout", baseline: "pass", oracle: "error", valid: false },
{ id: "baseline-signal", baseline: "error", oracle: "pass", valid: false },
{ id: "reference-signal", baseline: "pass", oracle: "error", valid: false },
+ { id: "baseline-vitest-load", baseline: "error", oracle: "pass", valid: false },
+ { id: "vitest-assertion-with-quoted-load-error", baseline: "fail", oracle: "pass", valid: true },
{ id: "baseline-missing-module", baseline: "error", oracle: "pass", valid: false },
{ id: "reference-missing-module", baseline: "pass", oracle: "error", valid: false },
{ id: "baseline-syntax", baseline: "error", oracle: "pass", valid: false },
@@ -55,7 +57,15 @@ const id = process.env.SCREENPIPE_EVAL_CASE_ID;
const affected = id.startsWith("baseline-") ? broken : !broken;
if (id.endsWith("-timeout") && affected) setInterval(() => {}, 1000);
else if (id.endsWith("-signal") && affected) process.kill(process.pid, "SIGTERM");
-else if (id.endsWith("-missing-module") && affected) await import("./absent-synthetic-module.mjs");
+else if (id === "baseline-vitest-load" && affected) {
+ process.stdout.write("\\n RUN v2.1.9 /synthetic\\n Test Files 1 failed (1)\\n Tests no tests\\n");
+ process.stderr.write("Failed Suites 1\\nError: Failed to load url ./missing-fixture (resolved id: ./missing-fixture). Does the file exist?\\n");
+ process.exit(1);
+} else if (id === "vitest-assertion-with-quoted-load-error" && broken) {
+ process.stdout.write("Test Files 1 failed (1)\\nTests no tests\\nTest Files 1 failed (1)\\nTests 1 failed (1)\\n");
+ process.stderr.write("Failed Suites 1\\nError: Failed to load url ./quoted-fixture\\nAssertionError: actual did not equal expected\\n");
+ process.exit(1);
+} else if (id.endsWith("-missing-module") && affected) await import("./absent-synthetic-module.mjs");
else if (id === "baseline-syntax" && affected) {
writeFileSync("invalid.mjs", "export const value = ;");
await import("./invalid.mjs");