YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust agentsagiaiai-memoryaudio-recordingcomputer-visionhermeshermes-agentllmlocal-ailocal-firstmachine-learningmcpmultimodalopenclawprivacyrewindscreen-recordingspeech-to-textycombinator
evals: classify Vitest collection failures before scoring
Louis Beaumontcb479138eb3f
diff --git a/evals/coding-agent/README.md b/evals/coding-agent/README.mdindex 1a3129ec7..797f92d06 100644--- a/evals/coding-agent/README.md+++ b/evals/coding-agent/README.md@@ -51,7 +51,8 @@ the success denominator. They are never silently converted into model failures.Grader timeouts and terminating signals cannot establish a failing baseline:`--verify` requires a behavioral `fail` followed by an oracle `pass`.Known command-unavailable exits (126/127), Bun unhandled test-load errors, and-Node missing-module/syntax diagnostics are also reported as `error`, with a+Node missing-module/syntax diagnostics and Vitest URL-load collection failures+with zero tests are also reported as `error`, with a`grader_error_kind` in the result. A missing import followed by an oracle passtherefore cannot certify a regression. Plain failed assertions and successfulcommands containing diagnostic words retain their previous outcomes.@@ -64,8 +65,9 @@ change, caused it. Report error counts and inspect candidate-caused errors beforcomparing model success rates. No isolation or model-quality claim follows.Run the synthetic runner controls with `bun test ./evals/coding-agent/run.test.ts`.-Seventeen end-to-end controls cover baseline/reference timeouts and signals,-missing ESM/CommonJS modules, Node/Bun syntax and Bun import errors, unavailable+Nineteen end-to-end controls cover baseline/reference timeouts and signals,+missing ESM/CommonJS modules, Node/Bun syntax and Bun import errors, Vitest+collection failures and quoted diagnostics followed by real assertions, unavailableor non-executable commands, genuine failures, assertions quoting diagnostic words or complete diagnostic blocks,already-passing baselines and exclusion of known setup/process errors from scores.The controls use temporary local Git fixtures and invoke no model or provider.diff --git a/evals/coding-agent/grader-outcome.mjs b/evals/coding-agent/grader-outcome.mjsindex ac613b2f0..f77d6f820 100644--- a/evals/coding-agent/grader-outcome.mjs+++ b/evals/coding-agent/grader-outcome.mjs@@ -10,6 +10,14 @@ export function classifyGraderError(grader) {const stderr = (grader.stderr ?? "").replace(/\x1b\[[0-9;]*m/g, "");if (/^# Unhandled error between tests\s*$/m.test(stderr) &&/^\s*\d+ errors?\s*$/m.test(stderr)) return "bun_unhandled_error";+ // Vitest prints collection errors on stderr but its final test count on+ // stdout. Use the last summary so diagnostic text printed by a test cannot+ // hide a later genuine assertion failure. Unknown formats still need review.+ const stdout = (grader.stdout ?? "").replace(/\x1b\[[0-9;]*m/g, "");+ const summary = [...stdout.matchAll(/^\s*Tests\s+(.+)$/gm)].at(-1)?.[1]?.trim();+ if (summary === "no tests" && /^\s*Test Files\s+\d+ failed/m.test(stdout) &&+ /Failed Suites [1-9]/.test(stderr) &&+ /^Error: Failed to load url /m.test(stderr)) return "vitest_collection_error";// Only the first thrown-error header classifies a Node failure. Assertion// messages may quote complete setup diagnostics on later lines.const header = stderr.match(/^([A-Za-z]*Error)(?: \[([A-Z_0-9]+)\])?:[^\n]*/m);diff --git a/evals/coding-agent/run.test.ts b/evals/coding-agent/run.test.tsindex a9a22025e..a5f361fc0 100644--- a/evals/coding-agent/run.test.ts+++ b/evals/coding-agent/run.test.ts@@ -34,6 +34,8 @@ test("process and setup failures cannot prove a regression or count as scored fa{ id: "reference-timeout", baseline: "pass", oracle: "error", valid: false },{ id: "baseline-signal", baseline: "error", oracle: "pass", valid: false },{ id: "reference-signal", baseline: "pass", oracle: "error", valid: false },+ { id: "baseline-vitest-load", baseline: "error", oracle: "pass", valid: false },+ { id: "vitest-assertion-with-quoted-load-error", baseline: "fail", oracle: "pass", valid: true },{ id: "baseline-missing-module", baseline: "error", oracle: "pass", valid: false },{ id: "reference-missing-module", baseline: "pass", oracle: "error", valid: false },{ id: "baseline-syntax", baseline: "error", oracle: "pass", valid: false },@@ -55,7 +57,15 @@ const id = process.env.SCREENPIPE_EVAL_CASE_ID;const affected = id.startsWith("baseline-") ? broken : !broken;if (id.endsWith("-timeout") && affected) setInterval(() => {}, 1000);else if (id.endsWith("-signal") && affected) process.kill(process.pid, "SIGTERM");-else if (id.endsWith("-missing-module") && affected) await import("./absent-synthetic-module.mjs");+else if (id === "baseline-vitest-load" && affected) {+ process.stdout.write("\\n RUN v2.1.9 /synthetic\\n Test Files 1 failed (1)\\n Tests no tests\\n");+ process.stderr.write("Failed Suites 1\\nError: Failed to load url ./missing-fixture (resolved id: ./missing-fixture). Does the file exist?\\n");+ process.exit(1);+} else if (id === "vitest-assertion-with-quoted-load-error" && broken) {+ process.stdout.write("Test Files 1 failed (1)\\nTests no tests\\nTest Files 1 failed (1)\\nTests 1 failed (1)\\n");+ process.stderr.write("Failed Suites 1\\nError: Failed to load url ./quoted-fixture\\nAssertionError: actual did not equal expected\\n");+ process.exit(1);+} else if (id.endsWith("-missing-module") && affected) await import("./absent-synthetic-module.mjs");else if (id === "baseline-syntax" && affected) {writeFileSync("invalid.mjs", "export const value = ;");await import("./invalid.mjs");