Skip to content

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

perf(evals): hide and reuse grader build cache

Louis Beaumont5e3e58e942e9
diff --git a/evals/coding-agent/DESIGN.md b/evals/coding-agent/DESIGN.md
index 13881b1e5..2897fb7e1 100644
--- a/evals/coding-agent/DESIGN.md
+++ b/evals/coding-agent/DESIGN.md
@@ -24,6 +24,8 @@ current guidance instead of copying that older repository's execution model.
- `--verify` must observe the parent fail and the known fix pass.
- The agent gets a depth-one detached checkout of the broken commit, with no
future history, oracle patch, or hidden grader available during its trial.
+- Build caches that could contain future artifacts are linked only after the
+ agent stops; build tools still validate their content hashes before reuse.
- Prompts specify observable behavior and safety boundaries, not an expected
implementation trajectory.
- Existing behavior is retained through positive and negative grader examples.
diff --git a/evals/coding-agent/cases.json b/evals/coding-agent/cases.json
index 53782f254..8ec027c87 100644
--- a/evals/coding-agent/cases.json
+++ b/evals/coding-agent/cases.json
@@ -75,6 +75,12 @@
"tags": ["regression", "escaped-failure", "timeline", "websocket", "native"],
"prompt": "Native Timeline search becomes unreliable on large days: serialized WebSocket batches can exceed the native receiver ceiling, and navigation/search races can surface stale or incomplete frames. Fix the stream path so payloads are bounded by encoded bytes without dropping or duplicating rows, cancellation is respected, and the native search contract remains compatible. Do not solve this by imposing a small arbitrary row limit.",
"agent_timeout_seconds": 1200,
+ "grader_dependency_links": [
+ {
+ "source_path": "target",
+ "destination_path": "target"
+ }
+ ],
"grader": {
"timeout_seconds": 1200,
"fixtures": [
diff --git a/evals/coding-agent/run.mjs b/evals/coding-agent/run.mjs
index 050ee144e..bfd3b0ac4 100644
--- a/evals/coding-agent/run.mjs
+++ b/evals/coding-agent/run.mjs
@@ -145,8 +145,8 @@ function extractBase(evalCase, workspace) {
mustRun("git", ["config", "user.name", "screenpipe eval harness"], { cwd: workspace });
}
-function linkDependencies(evalCase, workspace) {
- for (const link of evalCase.dependency_links ?? []) {
+function linkDependencies(evalCase, workspace, field = "dependency_links") {
+ for (const link of evalCase[field] ?? []) {
const source = join(REPO, link.source_path);
const destination = join(workspace, link.destination_path);
if (!existsSync(source) || existsSync(destination)) continue;
@@ -317,6 +317,9 @@ function runTrial(evalCase, mode, trial, options, runDir) {
writeFileSync(join(trialDir, "agent.stderr.log"), agent.stderr);
copyFileSync(promptFile, join(trialDir, "prompt.md"));
+ // Grader-only caches are withheld until the agent trajectory is complete,
+ // so future build artifacts cannot leak information into the trial.
+ linkDependencies(evalCase, workspace, "grader_dependency_links");
materializeGraders(evalCase, workspace);
const grader = command("/bin/bash", ["-lc", evalCase.grader.command], {
cwd: workspace,