Skip to content

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

feat(evals): add coding-agent regression harness

Louis Beaumont6cc40c4f326e
diff --git a/evals/coding-agent/.gitignore b/evals/coding-agent/.gitignore
new file mode 100644
index 000000000..fbca22537
--- /dev/null
+++ b/evals/coding-agent/.gitignore
@@ -0,0 +1 @@
+results/
diff --git a/evals/coding-agent/README.md b/evals/coding-agent/README.md
new file mode 100644
index 000000000..36890c6ca
--- /dev/null
+++ b/evals/coding-agent/README.md
@@ -0,0 +1,51 @@
+# Coding-agent regression evals
+
+This is an agent eval suite, not a unit-test suite. Every case contains:
+
+- a sanitized task derived from an escaped product failure;
+- the historical broken repository revision;
+- an isolated trial workspace with no future git history;
+- a hidden deterministic outcome grader materialized only after the agent stops;
+- saved prompt, transcript, candidate patch, grader output, runtime fingerprint, and result;
+- repeated-trial reporting with success rate, `pass@k`, and `pass^k`.
+
+The product tests referenced by the manifest are graders. Passing them directly is not the eval; the evaluated object is an agent trajectory and resulting patch from the historical broken state.
+
+## Validate the corpus
+
+```bash
+node evals/coding-agent/run.mjs --validate
+node evals/coding-agent/run.mjs --verify
+```
+
+`--verify` proves each historical base fails its hidden grader and its known fix passes. A case is invalid if either side of that contrast is missing.
+
+## Run agent trials
+
+```bash
+node evals/coding-agent/run.mjs \
+ --case app-chat-concurrent-save \
+ --trials 3 \
+ --agent-command 'codex exec --ephemeral --approve-for-me --json -C {workspace} - < {prompt_file}'
+```
+
+The agent receives only the task and archived broken tree. It does not receive the oracle commit, grader definition, or future repository history.
+
+Results are written under `evals/coding-agent/results/` unless `--results-dir` is supplied. Use `--keep` only for debugging a failed trial.
+
+Agent-process and harness failures are reported as `error` and excluded from
+the success denominator. They are never silently converted into model failures.
+
+The same runner can score another checkout and manifest with `--repo` and
+`--manifest`; the website corpus uses this so both repositories share exactly
+one harness implementation.
+
+## Modes
+
+- `agent`: run the configured coding agent and grade its patch.
+- `baseline`: make no change and grade the broken revision.
+- `oracle`: apply the historical fix and grade it.
+- `regrade`: apply a saved `candidate.patch` and run the current grader without another model call.
+- `--verify`: require baseline failure plus oracle success.
+
+Do not turn capability scores into a release gate after one run. Establish matched-environment repeated baselines first. Regression cases intended to block should target reliable `pass^k`, not a lucky `pass@k`.
diff --git a/evals/coding-agent/cases.json b/evals/coding-agent/cases.json
new file mode 100644
index 000000000..fd6a3ca65
--- /dev/null
+++ b/evals/coding-agent/cases.json
@@ -0,0 +1,112 @@
+{
+ "schema_version": 1,
+ "suite": "screenpipe-app-coding-agent-regressions",
+ "dataset_version": "2026-08-19.1",
+ "cases": [
+ {
+ "id": "app-chat-concurrent-save",
+ "name": "Preserve both writers during concurrent chat saves",
+ "base_ref": "1a74f9c68^",
+ "oracle_ref": "1a74f9c68",
+ "tags": ["regression", "escaped-failure", "chat", "concurrency", "data-loss"],
+ "prompt": "A user has the same chat open in two app windows. Both writers start from the same conversation. Writer A saves a new message, then writer B saves a different new message from its stale snapshot. The later save silently deletes writer A's message. Fix conversation persistence so concurrent saves preserve both suffixes, do not duplicate messages, retain the richer copy of a streaming message, and do not let stale scalar fields revert newer disk state. Keep the solution scoped to chat persistence and add or update tests as needed.",
+ "agent_timeout_seconds": 900,
+ "dependency_links": [
+ {
+ "source_path": "apps/screenpipe-app-tauri/node_modules",
+ "destination_path": "apps/screenpipe-app-tauri/node_modules"
+ }
+ ],
+ "grader": {
+ "timeout_seconds": 300,
+ "fixtures": [
+ {
+ "source_path": "apps/screenpipe-app-tauri/lib/__tests__/chat-merge.test.ts"
+ },
+ {
+ "source_path": "apps/screenpipe-app-tauri/lib/__tests__/chat-storage-concurrency.test.ts"
+ }
+ ],
+ "command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts lib/__tests__/chat-merge.test.ts lib/__tests__/chat-storage-concurrency.test.ts"
+ }
+ },
+ {
+ "id": "app-terminal-quota-retry",
+ "name": "Stop retrying terminal Pi quota failures",
+ "base_ref": "58c6e837f^",
+ "oracle_ref": "58c6e837f",
+ "tags": ["regression", "escaped-failure", "chat", "quota", "retry"],
+ "prompt": "The Pi chat transport receives provider 429 responses that mean the account has exhausted a daily allowance or the selected model is not permitted. The app currently treats these as transient and keeps retrying, leaving the user in a loop. Make terminal quota/plan errors stop the retry path and surface the assistant error, while ordinary transient failures remain retryable. Preserve existing silent prompt-timeout and connection-card behavior.",
+ "agent_timeout_seconds": 720,
+ "dependency_links": [
+ {
+ "source_path": "apps/screenpipe-app-tauri/node_modules",
+ "destination_path": "apps/screenpipe-app-tauri/node_modules"
+ }
+ ],
+ "grader": {
+ "timeout_seconds": 300,
+ "fixtures": [
+ {
+ "local_path": "graders/app-terminal-quota.test.ts",
+ "destination_path": "apps/screenpipe-app-tauri/lib/chat/__tests__/eval-terminal-quota.test.ts"
+ },
+ {
+ "source_ref": "58c6e837f^",
+ "source_path": "apps/screenpipe-app-tauri/lib/chat/__tests__/quota-errors.test.ts"
+ },
+ {
+ "source_ref": "58c6e837f^",
+ "source_path": "apps/screenpipe-app-tauri/lib/chat/__tests__/provider-errors.test.ts"
+ }
+ ],
+ "command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts lib/chat/__tests__/eval-terminal-quota.test.ts lib/chat/__tests__/quota-errors.test.ts lib/chat/__tests__/provider-errors.test.ts"
+ }
+ },
+ {
+ "id": "app-timeline-stream-reliability",
+ "name": "Keep native Timeline streams bounded and complete",
+ "base_ref": "a4c1161d2^",
+ "oracle_ref": "a4c1161d2",
+ "tags": ["regression", "escaped-failure", "timeline", "websocket", "native"],
+ "prompt": "Native Timeline search becomes unreliable on large days: serialized WebSocket batches can exceed the native receiver ceiling, and navigation/search races can surface stale or incomplete frames. Fix the stream path so payloads are bounded by encoded bytes without dropping or duplicating rows, cancellation is respected, and the native search contract remains compatible. Do not solve this by imposing a small arbitrary row limit.",
+ "agent_timeout_seconds": 1200,
+ "grader": {
+ "timeout_seconds": 1200,
+ "fixtures": [
+ {
+ "source_path": "crates/screenpipe-engine/tests/stream_frames_test.rs"
+ }
+ ],
+ "command": "cargo test -p screenpipe-engine --test stream_frames_test"
+ }
+ },
+ {
+ "id": "app-first-run-empty-summary",
+ "name": "Never render an empty first-run summary as success",
+ "base_ref": "91fa9bd1b^",
+ "oracle_ref": "91fa9bd1b",
+ "tags": ["regression", "escaped-failure", "onboarding", "truthfulness"],
+ "prompt": "The first-run learning flow can finish with an empty or unusable generated summary and then render that as if capture produced a meaningful result. Fix the state and UI boundary so empty summaries stay out of the content surface, capture diagnostics remain actionable, and resume/dismiss behavior remains stable. A valid non-empty summary must still render normally.",
+ "agent_timeout_seconds": 900,
+ "dependency_links": [
+ {
+ "source_path": "apps/screenpipe-app-tauri/node_modules",
+ "destination_path": "apps/screenpipe-app-tauri/node_modules"
+ }
+ ],
+ "grader": {
+ "timeout_seconds": 420,
+ "fixtures": [
+ {
+ "source_path": "apps/screenpipe-app-tauri/lib/first-run/learning-window.test.ts"
+ },
+ {
+ "source_path": "apps/screenpipe-app-tauri/components/first-run/learning-banner.test.tsx"
+ }
+ ],
+ "command": "cd apps/screenpipe-app-tauri && bun x vitest run --config vitest.config.ts lib/first-run/learning-window.test.ts components/first-run/learning-banner.test.tsx"
+ }
+ }
+ ]
+}
diff --git a/evals/coding-agent/graders/app-terminal-quota.test.ts b/evals/coding-agent/graders/app-terminal-quota.test.ts
new file mode 100644
index 000000000..d3811d0d7
--- /dev/null
+++ b/evals/coding-agent/graders/app-terminal-quota.test.ts
@@ -0,0 +1,48 @@
+// screenpipe — AI that knows everything you've seen, said, or heard
+// https://screenpi.pe
+
+import { readFileSync } from "node:fs";
+import { join } from "node:path";
+import { describe, expect, it } from "vitest";
+import * as handlers from "../../../components/chat/standalone/hooks/pi-event-handlers";
+import * as quota from "../quota-errors";
+
+type Detector = (error: string) => boolean;
+
+const terminalDetector =
+ (handlers as Record<string, unknown>).isTerminalQuotaError ??
+ (quota as Record<string, unknown>).isTerminalQuotaOrPlanError;
+
+describe("terminal Pi retry outcome", () => {
+ it("provides one pure terminal detector with balanced positive and negative behavior", () => {
+ expect(typeof terminalDetector).toBe("function");
+ const detect = terminalDetector as Detector;
+ for (const error of [
+ 'HTTP 429 {"error":"daily_cost_limit_exceeded","required_plan":"business"}',
+ "HTTP 429 credits_exhausted",
+ '{"error":"model_not_allowed","required_plan":"business"}',
+ ]) {
+ expect(detect(error), error).toBe(true);
+ }
+ for (const error of [
+ "HTTP 429 rate limit exceeded; wait 9 seconds",
+ "500 upstream unavailable",
+ "socket hang up",
+ ]) {
+ expect(detect(error), error).toBe(false);
+ }
+ });
+
+ it("uses the terminal detector in the foreground event path that owns retries", () => {
+ const source = readFileSync(
+ join(
+ process.cwd(),
+ "components/chat/standalone/hooks/use-pi-foreground-events.ts",
+ ),
+ "utf8",
+ );
+ expect(source).toMatch(/\bisTerminalQuota(?:OrPlan)?Error\s*\(/);
+ expect(source).toContain("classifyQuotaError");
+ expect(source).toContain("PI_MAX_RATE_LIMIT_RETRIES");
+ });
+});
diff --git a/evals/coding-agent/run.mjs b/evals/coding-agent/run.mjs
new file mode 100644
index 000000000..261f6e823
--- /dev/null
+++ b/evals/coding-agent/run.mjs
@@ -0,0 +1,445 @@
+// screenpipe — AI that knows everything you've seen, said, or heard
+// https://screenpi.pe
+
+import {
+ chmodSync,
+ copyFileSync,
+ existsSync,
+ mkdirSync,
+ mkdtempSync,
+ readFileSync,
+ rmSync,
+ symlinkSync,
+ writeFileSync,
+} from "node:fs";
+import { arch, platform, release, tmpdir } from "node:os";
+import { dirname, join, resolve } from "node:path";
+import { spawnSync } from "node:child_process";
+import process from "node:process";
+
+const HERE = dirname(new URL(import.meta.url).pathname);
+let REPO = resolve(HERE, "../..");
+let MANIFEST = join(HERE, "cases.json");
+const MAX_BUFFER = 64 * 1024 * 1024;
+
+function parseArgs(argv) {
+ const args = { mode: "agent", trials: 1, keep: false };
+ for (let index = 0; index < argv.length; index += 1) {
+ const value = argv[index];
+ if (value === "--keep") args.keep = true;
+ else if (value === "--list") args.list = true;
+ else if (value === "--validate") args.validate = true;
+ else if (value === "--verify") args.verify = true;
+ else if (value === "--mode") args.mode = argv[++index];
+ else if (value === "--trials") args.trials = Number(argv[++index]);
+ else if (value === "--case") args.caseIds = argv[++index].split(",");
+ else if (value === "--agent-command") args.agentCommand = argv[++index];
+ else if (value === "--results-dir") args.resultsDir = resolve(argv[++index]);
+ else if (value === "--repo") args.repo = resolve(argv[++index]);
+ else if (value === "--manifest") args.manifest = resolve(argv[++index]);
+ else if (value === "--candidate-patch") args.candidatePatch = resolve(argv[++index]);
+ else throw new Error(`unknown argument: ${value}`);
+ }
+ if (!Number.isInteger(args.trials) || args.trials < 1) {
+ throw new Error("--trials must be a positive integer");
+ }
+ if (args.candidatePatch) args.mode = "regrade";
+ if (!["agent", "baseline", "oracle", "regrade"].includes(args.mode)) {
+ throw new Error("--mode must be agent, baseline, oracle, or regrade");
+ }
+ return args;
+}
+
+function command(program, args, options = {}) {
+ const result = spawnSync(program, args, {
+ cwd: options.cwd ?? REPO,
+ encoding: options.encoding ?? "utf8",
+ input: options.input,
+ env: options.env ?? process.env,
+ maxBuffer: MAX_BUFFER,
+ timeout: options.timeout,
+ });
+ return {
+ status: result.status,
+ signal: result.signal,
+ stdout: result.stdout?.toString() ?? "",
+ stderr: result.stderr?.toString() ?? "",
+ error: result.error?.message ?? null,
+ };
+}
+
+function mustRun(program, args, options = {}) {
+ const result = command(program, args, options);
+ if (result.status !== 0) {
+ throw new Error(
+ `${program} ${args.join(" ")} failed (${result.status}): ${result.stderr || result.error}`,
+ );
+ }
+ return result;
+}
+
+function shellQuote(value) {
+ return `'${String(value).replaceAll("'", `'"'"'`)}'`;
+}
+
+function loadManifest() {
+ const manifest = JSON.parse(readFileSync(MANIFEST, "utf8"));
+ if (manifest.schema_version !== 1 || !Array.isArray(manifest.cases)) {
+ throw new Error("unsupported or malformed eval manifest");
+ }
+ return manifest;
+}
+
+function selectCases(manifest, ids) {
+ if (!ids) return manifest.cases;
+ const selected = manifest.cases.filter((candidate) => ids.includes(candidate.id));
+ const found = new Set(selected.map((candidate) => candidate.id));
+ const missing = ids.filter((id) => !found.has(id));
+ if (missing.length) throw new Error(`unknown case(s): ${missing.join(", ")}`);
+ return selected;
+}
+
+function validateCase(evalCase) {
+ for (const key of ["id", "name", "base_ref", "oracle_ref", "prompt", "grader"]) {
+ if (!evalCase[key]) throw new Error(`${evalCase.id ?? "case"} is missing ${key}`);
+ }
+ mustRun("git", ["cat-file", "-e", `${evalCase.base_ref}^{commit}`]);
+ mustRun("git", ["cat-file", "-e", `${evalCase.oracle_ref}^{commit}`]);
+ for (const fixture of evalCase.grader.fixtures ?? []) {
+ if (fixture.local_path) {
+ const source = join(dirname(MANIFEST), fixture.local_path);
+ if (!existsSync(source)) throw new Error(`${evalCase.id} is missing ${fixture.local_path}`);
+ } else {
+ const sourceRef = fixture.source_ref ?? evalCase.oracle_ref;
+ mustRun("git", ["cat-file", "-e", `${sourceRef}:${fixture.source_path}`]);
+ }
+ }
+}
+
+function extractBase(evalCase, workspace) {
+ const baseSha = mustRun("git", ["rev-parse", `${evalCase.base_ref}^{commit}`]).stdout.trim();
+ mustRun("git", ["init", "-q"], { cwd: workspace });
+ const isolatedEnv = { ...process.env, GIT_LFS_SKIP_SMUDGE: "1" };
+ mustRun("git", ["fetch", "-q", "--depth=1", "--no-tags", REPO, baseSha], {
+ cwd: workspace,
+ env: isolatedEnv,
+ timeout: 180_000,
+ });
+ mustRun("git", ["checkout", "-q", "--detach", "FETCH_HEAD"], {
+ cwd: workspace,
+ env: isolatedEnv,
+ timeout: 180_000,
+ });
+ mustRun("git", ["config", "user.email", "[email protected]"], { cwd: workspace });
+ mustRun("git", ["config", "user.name", "screenpipe eval harness"], { cwd: workspace });
+}
+
+function linkDependencies(evalCase, workspace) {
+ for (const link of evalCase.dependency_links ?? []) {
+ const source = join(REPO, link.source_path);
+ const destination = join(workspace, link.destination_path);
+ if (!existsSync(source) || existsSync(destination)) continue;
+ mkdirSync(dirname(destination), { recursive: true });
+ symlinkSync(source, destination, "dir");
+ }
+}
+
+function applyOracle(evalCase, workspace) {
+ const pathspec = evalCase.oracle_paths ?? [];
+ const patch = mustRun(
+ "git",
+ ["diff", "--binary", evalCase.base_ref, evalCase.oracle_ref, "--", ...pathspec],
+ { encoding: null },
+ ).stdout;
+ const applied = command("git", ["apply", "--whitespace=nowarn", "-"], {
+ cwd: workspace,
+ input: Buffer.from(patch),
+ });
+ if (applied.status !== 0) throw new Error(`oracle patch failed: ${applied.stderr}`);
+}
+
+function applySavedCandidate(path, workspace) {
+ const applied = command("git", ["apply", "--whitespace=nowarn", "-"], {
+ cwd: workspace,
+ input: readFileSync(path),
+ });
+ if (applied.status !== 0) throw new Error(`saved candidate patch failed: ${applied.stderr}`);
+}
+
+function materializeGraders(evalCase, workspace) {
+ for (const fixture of evalCase.grader.fixtures ?? []) {
+ let body;
+ if (fixture.local_path) {
+ body = readFileSync(join(dirname(MANIFEST), fixture.local_path));
+ } else {
+ const sourceRef = fixture.source_ref ?? evalCase.oracle_ref;
+ body = mustRun("git", ["show", `${sourceRef}:${fixture.source_path}`], {
+ encoding: null,
+ }).stdout;
+ }
+ const destination = join(
+ workspace,
+ fixture.destination_path ?? fixture.source_path ?? fixture.local_path,
+ );
+ mkdirSync(dirname(destination), { recursive: true });
+ writeFileSync(destination, Buffer.from(body));
+ if (fixture.executable) chmodSync(destination, 0o755);
+ }
+}
+
+function renderAgentCommand(template, paths, evalCase) {
+ return template
+ .replaceAll("{workspace}", shellQuote(paths.workspace))
+ .replaceAll("{prompt_file}", shellQuote(paths.promptFile))
+ .replaceAll("{case_id}", shellQuote(evalCase.id));
+}
+
+function extractAgentUsage(stdout) {
+ let usage = null;
+ for (const line of stdout.split("\n")) {
+ if (!line.trim().startsWith("{")) continue;
+ try {
+ const event = JSON.parse(line);
+ if (event.type === "turn.completed" && event.usage) usage = event.usage;
+ } catch {
+ // Non-JSON agent output is still a valid transcript.
+ }
+ }
+ return usage;
+}
+
+function captureCandidate(evalCase, workspace) {
+ const trackedPatch = command("git", ["diff", "--binary", "HEAD"], { cwd: workspace });
+ const trackedFiles = command("git", ["diff", "--name-only", "HEAD"], {
+ cwd: workspace,
+ }).stdout
+ .split("\n")
+ .filter(Boolean);
+ const dependencyPaths = (evalCase.dependency_links ?? []).map(
+ (link) => link.destination_path.replaceAll(/\/$/g, ""),
+ );
+ const untracked = command(
+ "git",
+ ["ls-files", "--others", "--exclude-standard"],
+ { cwd: workspace },
+ ).stdout
+ .split("\n")
+ .filter(Boolean)
+ .filter(
+ (path) =>
+ !dependencyPaths.some(
+ (dependencyPath) =>
+ path === dependencyPath || path.startsWith(`${dependencyPath}/`),
+ ),
+ );
+ const patches = [trackedPatch.stdout];
+ for (const path of untracked) {
+ const addition = command("git", ["diff", "--no-index", "--binary", "--", "/dev/null", path], {
+ cwd: workspace,
+ });
+ if (![0, 1].includes(addition.status)) {
+ throw new Error(`could not capture untracked file ${path}: ${addition.stderr}`);
+ }
+ patches.push(addition.stdout);
+ }
+ return {
+ patch: patches.join(""),
+ changedFiles: [...new Set([...trackedFiles, ...untracked])].sort(),
+ };
+}
+
+function runTrial(evalCase, mode, trial, options, runDir) {
+ const sandboxRoot = mkdtempSync(join(tmpdir(), `screenpipe-eval-${evalCase.id}-`));
+ const workspace = join(sandboxRoot, "workspace");
+ mkdirSync(workspace);
+ const trialDir = join(runDir, evalCase.id, `trial-${trial}`);
+ mkdirSync(trialDir, { recursive: true });
+ const promptFile = join(sandboxRoot, "prompt.md");
+ writeFileSync(promptFile, `${evalCase.prompt.trim()}\n`);
+
+ let agent = { status: 0, stdout: "", stderr: "", signal: null, error: null };
+ const started = Date.now();
+ try {
+ extractBase(evalCase, workspace);
+ linkDependencies(evalCase, workspace);
+ if (mode === "oracle") applyOracle(evalCase, workspace);
+ if (mode === "regrade") {
+ applySavedCandidate(options.candidatePatch, workspace);
+ const sourceDir = dirname(options.candidatePatch);
+ const stdoutPath = join(sourceDir, "agent.stdout.log");
+ const stderrPath = join(sourceDir, "agent.stderr.log");
+ const metadataPath = join(sourceDir, "agent.json");
+ if (existsSync(stdoutPath)) agent.stdout = readFileSync(stdoutPath, "utf8");
+ if (existsSync(stderrPath)) agent.stderr = readFileSync(stderrPath, "utf8");
+ if (existsSync(metadataPath)) {
+ const metadata = JSON.parse(readFileSync(metadataPath, "utf8"));
+ agent.status = metadata.status ?? 0;
+ agent.signal = metadata.signal ?? null;
+ agent.error = metadata.error ?? null;
+ }
+ }
+ if (mode === "agent") {
+ if (!options.agentCommand) throw new Error("--agent-command is required in agent mode");
+ const rendered = renderAgentCommand(
+ options.agentCommand,
+ { workspace, promptFile },
+ evalCase,
+ );
+ agent = command("/bin/bash", ["-lc", rendered], {
+ cwd: workspace,
+ timeout: (evalCase.agent_timeout_seconds ?? 900) * 1_000,
+ env: {
+ ...process.env,
+ SCREENPIPE_EVAL_CASE_ID: evalCase.id,
+ SCREENPIPE_EVAL_WORKSPACE: workspace,
+ },
+ });
+ }
+
+ const candidate = captureCandidate(evalCase, workspace);
+ writeFileSync(join(trialDir, "candidate.patch"), candidate.patch);
+ writeFileSync(
+ join(trialDir, "agent.json"),
+ JSON.stringify({ ...agent, stdout: undefined, stderr: undefined }, null, 2),
+ );
+ writeFileSync(join(trialDir, "agent.stdout.log"), agent.stdout);
+ writeFileSync(join(trialDir, "agent.stderr.log"), agent.stderr);
+ copyFileSync(promptFile, join(trialDir, "prompt.md"));
+
+ materializeGraders(evalCase, workspace);
+ const grader = command("/bin/bash", ["-lc", evalCase.grader.command], {
+ cwd: workspace,
+ timeout: (evalCase.grader.timeout_seconds ?? 600) * 1_000,
+ env: { ...process.env, SCREENPIPE_EVAL_CASE_ID: evalCase.id },
+ });
+ writeFileSync(join(trialDir, "grader.stdout.log"), grader.stdout);
+ writeFileSync(join(trialDir, "grader.stderr.log"), grader.stderr);
+ const agentOk = !["agent", "regrade"].includes(mode) || (agent.status === 0 && !agent.error);
+ const passed = agentOk && grader.status === 0;
+ const outcome = !agentOk ? "error" : passed ? "pass" : "fail";
+ const result = {
+ case_id: evalCase.id,
+ trial,
+ mode,
+ passed,
+ outcome,
+ duration_ms: Date.now() - started,
+ agent_exit: agent.status,
+ agent_usage: extractAgentUsage(agent.stdout),
+ grader_exit: grader.status,
+ changed_files: candidate.changedFiles,
+ workspace: options.keep ? workspace : undefined,
+ };
+ writeFileSync(join(trialDir, "result.json"), JSON.stringify(result, null, 2));
+ return result;
+ } catch (error) {
+ const result = {
+ case_id: evalCase.id,
+ trial,
+ mode,
+ passed: false,
+ outcome: "error",
+ duration_ms: Date.now() - started,
+ harness_error: error instanceof Error ? error.message : String(error),
+ workspace: options.keep ? workspace : undefined,
+ };
+ writeFileSync(join(trialDir, "result.json"), JSON.stringify(result, null, 2));
+ return result;
+ } finally {
+ if (!options.keep) rmSync(sandboxRoot, { recursive: true, force: true });
+ }
+}
+
+function summarize(manifest, mode, results, options, runDir) {
+ const cases = [...new Set(results.map((result) => result.case_id))].map((caseId) => {
+ const trials = results.filter((result) => result.case_id === caseId);
+ const passes = trials.filter((result) => result.passed).length;
+ const errors = trials.filter((result) => result.outcome === "error").length;
+ const scoredTrials = trials.length - errors;
+ return {
+ case_id: caseId,
+ trials: trials.length,
+ scored_trials: scoredTrials,
+ passes,
+ errors,
+ success_rate: scoredTrials ? passes / scoredTrials : null,
+ pass_at_k: scoredTrials ? passes > 0 : null,
+ pass_pow_k: scoredTrials ? passes === scoredTrials : null,
+ };
+ });
+ const report = {
+ suite: manifest.suite,
+ dataset_version: manifest.dataset_version,
+ mode,
+ created_at: new Date().toISOString(),
+ runtime: {
+ platform: platform(),
+ release: release(),
+ arch: arch(),
+ node: process.version,
+ repo_head: mustRun("git", ["rev-parse", "HEAD"]).stdout.trim(),
+ agent_command: mode === "agent" ? options.agentCommand : undefined,
+ candidate_patch: mode === "regrade" ? options.candidatePatch : undefined,
+ },
+ cases,
+ trials: results,
+ };
+ writeFileSync(join(runDir, "summary.json"), JSON.stringify(report, null, 2));
+ for (const item of cases) {
+ console.log(
+ `${item.case_id}: ${item.passes}/${item.scored_trials} pass, ${item.errors} error ` +
+ `(pass@${item.scored_trials}=${item.pass_at_k}, pass^${item.scored_trials}=${item.pass_pow_k})`,
+ );
+ }
+ console.log(`results: ${join(runDir, "summary.json")}`);
+ return report;
+}
+
+function main() {
+ const options = parseArgs(process.argv.slice(2));
+ if (options.repo) REPO = options.repo;
+ if (options.manifest) MANIFEST = options.manifest;
+ const manifest = loadManifest();
+ const cases = selectCases(manifest, options.caseIds);
+ if (options.candidatePatch && cases.length !== 1) {
+ throw new Error("--candidate-patch requires exactly one --case");
+ }
+ if (options.list) {
+ for (const evalCase of cases) console.log(`${evalCase.id}\t${evalCase.name}`);
+ return;
+ }
+ for (const evalCase of cases) validateCase(evalCase);
+ if (options.validate) {
+ console.log(`validated ${cases.length} eval cases`);
+ return;
+ }
+
+ const runId = new Date().toISOString().replaceAll(/[:.]/g, "-");
+ const runDir = options.resultsDir ?? join(dirname(MANIFEST), "results", runId);
+ mkdirSync(runDir, { recursive: true });
+ if (options.verify) {
+ const verification = [];
+ for (const evalCase of cases) {
+ const baseline = runTrial(evalCase, "baseline", 1, options, join(runDir, "baseline"));
+ const oracle = runTrial(evalCase, "oracle", 1, options, join(runDir, "oracle"));
+ const valid = !baseline.passed && oracle.passed && !baseline.harness_error && !oracle.harness_error;
+ verification.push({ case_id: evalCase.id, valid, baseline, oracle });
+ console.log(`${evalCase.id}: broken=${baseline.passed ? "PASS" : "FAIL"} oracle=${oracle.passed ? "PASS" : "FAIL"}`);
+ }
+ writeFileSync(join(runDir, "verification.json"), JSON.stringify(verification, null, 2));
+ if (verification.some((item) => !item.valid)) process.exitCode = 1;
+ return;
+ }
+
+ const results = [];
+ for (const evalCase of cases) {
+ for (let trial = 1; trial <= options.trials; trial += 1) {
+ const result = runTrial(evalCase, options.mode, trial, options, runDir);
+ results.push(result);
+ console.log(`${evalCase.id} trial ${trial}: ${result.outcome.toUpperCase()}`);
+ }
+ }
+ summarize(manifest, options.mode, results, options, runDir);
+}
+
+main();