Skip to content

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

evals: classify missing PostCSS plugin startup failures

Louis Beaumont05d525953435
diff --git a/evals/coding-agent/README.md b/evals/coding-agent/README.md
index ea5a42228..b0a9d6c07 100644
--- a/evals/coding-agent/README.md
+++ b/evals/coding-agent/README.md
@@ -65,9 +65,10 @@ change, caused it. Report error counts and inspect candidate-caused errors befor
comparing model success rates. No isolation or model-quality claim follows.
Run the synthetic runner controls with `bun test ./evals/coding-agent/run.test.ts`.
-Twenty-two end-to-end controls cover baseline/reference timeouts and signals,
+Twenty-seven end-to-end controls cover baseline/reference timeouts and signals,
missing ESM/CommonJS modules, Node/Bun syntax and Bun import errors, Vitest
-collection failures (including alias imports) and quoted diagnostics followed by real assertions, unavailable
+collection failures (including alias imports), missing PostCSS-plugin startup
+failures, quoted diagnostics followed by real assertions, unavailable
or non-executable commands, genuine failures, assertions quoting diagnostic words or complete diagnostic blocks,
already-passing baselines and exclusion of known setup/process errors from scores.
The controls use temporary local Git fixtures and invoke no model or provider.
@@ -148,3 +149,12 @@ real settings-store serialization, validation, native capture or current-head
integration. The adapter retains the existing category-switch UI contract;
module relocation or a different UI/IO abstraction can require recalibration.
No dependency link or model call is needed for historical verification.
+
+Vite can reject a missing PostCSS plugin before reporting any test-count summary.
+The runner recognizes the bounded combination of a Vitest startup banner,
+unhandled rejection, PostCSS load failure, missing-plugin diagnostic and config
+path as `vitest_postcss_setup_error`. Executed-test summaries, assertion headers
+and successful exits retain their previous results. This does not classify all
+PostCSS/compiler errors or prove whether the candidate caused a setup failure.
+The 27 synthetic CLI controls include both baseline/reference startup failure,
+quoted diagnostics with and without a Vitest test summary, and successful output.
diff --git a/evals/coding-agent/grader-outcome.mjs b/evals/coding-agent/grader-outcome.mjs
index ceab61657..50b67de91 100644
--- a/evals/coding-agent/grader-outcome.mjs
+++ b/evals/coding-agent/grader-outcome.mjs
@@ -19,6 +19,15 @@ export function classifyGraderError(grader) {
/Failed Suites [1-9]/.test(stderr) &&
(/^Error: Failed to load url /m.test(stderr) ||
/^Error: Cannot find module ['"][^\n]+['"] imported from ['"][^\n]+['"]\.\s*$/m.test(stderr))) return "vitest_collection_error";
+ // Vite can reject PostCSS startup before printing a test-count summary.
+ // Require the specific missing-plugin diagnostic and startup banner; an
+ // executed-test summary or assertion header must keep its behavioral result.
+ if ((summary === undefined || summary === "no tests") &&
+ /^\s*RUN\s+v\d+\./m.test(stdout) && /Unhandled Rejection/.test(stderr) &&
+ /^Failed to load PostCSS config:/m.test(stderr) &&
+ /Loading PostCSS Plugin failed: Cannot find module ['"][^\n]+['"]/.test(stderr) &&
+ /[/\\]postcss\.config\.[cm]?[jt]s/.test(stderr) &&
+ !/^AssertionError(?: \[[^\]]+\])?:/m.test(stderr)) return "vitest_postcss_setup_error";
// Only the first thrown-error header classifies a Node failure. Assertion
// messages may quote complete setup diagnostics on later lines.
const header = stderr.match(/^([A-Za-z]*Error)(?: \[([A-Z_0-9]+)\])?:[^\n]*/m);
diff --git a/evals/coding-agent/run.test.ts b/evals/coding-agent/run.test.ts
index 40b5b6f57..68fc4f9e6 100644
--- a/evals/coding-agent/run.test.ts
+++ b/evals/coding-agent/run.test.ts
@@ -34,6 +34,11 @@ test("process and setup failures cannot prove a regression or count as scored fa
{ id: "reference-timeout", baseline: "pass", oracle: "error", valid: false },
{ id: "baseline-signal", baseline: "error", oracle: "pass", valid: false },
{ id: "reference-signal", baseline: "pass", oracle: "error", valid: false },
+ { id: "baseline-vitest-postcss", baseline: "error", oracle: "pass", valid: false },
+ { id: "reference-vitest-postcss", baseline: "fail", oracle: "error", valid: false },
+ { id: "vitest-assertion-with-quoted-postcss", baseline: "fail", oracle: "pass", valid: true },
+ { id: "node-assertion-with-quoted-postcss", baseline: "fail", oracle: "pass", valid: true },
+ { id: "both-pass-postcss-diagnostic", baseline: "pass", oracle: "pass", valid: false },
{ id: "baseline-vitest-alias", baseline: "error", oracle: "pass", valid: false },
{ id: "reference-vitest-alias", baseline: "fail", oracle: "error", valid: false },
{ id: "vitest-assertion-with-quoted-alias-error", baseline: "fail", oracle: "pass", valid: true },
@@ -60,6 +65,13 @@ const id = process.env.SCREENPIPE_EVAL_CASE_ID;
const affected = id.startsWith("baseline-") ? broken : !broken;
if (id.endsWith("-timeout") && affected) setInterval(() => {}, 1000);
else if (id.endsWith("-signal") && affected) process.kill(process.pid, "SIGTERM");
+else if ((id.endsWith("vitest-postcss") && affected) || ((id === "vitest-assertion-with-quoted-postcss" || id === "node-assertion-with-quoted-postcss") && broken) || id === "both-pass-postcss-diagnostic") {
+ process.stdout.write(" RUN v2.1.9 /synthetic\\n");
+ process.stderr.write("Unhandled Rejection\\nFailed to load PostCSS config: Failed to load PostCSS config (searchPath: /synthetic): [Error] Loading PostCSS Plugin failed: Cannot find module 'synthetic-css-plugin'\\nRequire stack:\\n- /synthetic/postcss.config.js\\n");
+ if (id === "vitest-assertion-with-quoted-postcss") process.stdout.write("Test Files 1 failed (1)\\nTests 1 failed (1)\\n");
+ if (id === "node-assertion-with-quoted-postcss") assert.fail("synthetic behavior assertion after quoted startup text");
+ process.exit(id === "both-pass-postcss-diagnostic" ? 0 : 1);
+}
else if (id.endsWith("vitest-alias") && affected) {
process.stdout.write("Test Files 1 failed (1)\\nTests no tests\\n");
process.stderr.write("Failed Suites 1\\nError: Cannot find module '@/lib/synthetic-missing' imported from '/synthetic/fixture.test.ts'.\\n");
@@ -100,7 +112,7 @@ else if (id === "baseline-syntax" && affected) {
assert.fail("\\nError [ERR_MODULE_NOT_FOUND]: quoted diagnostic\\nSyntaxError: quoted diagnostic\\n at quotedFixture");
} else if (id === "both-pass-diagnostic-words") {
console.error("Cannot find module; SyntaxError; command not found are fixture words");
-} else if (id === "reference-vitest-alias" && broken) assert.fail("synthetic broken behavior");
+} else if ((id === "reference-vitest-alias" || id === "reference-vitest-postcss") && broken) assert.fail("synthetic broken behavior");
else process.exit(id === "intended-failure" && broken ? 1 : 0);
`);
const manifest = join(repo, "cases.json");