Skip to content

Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.

docs: index preserved SWE-bench research and retire local eval clutter

David Zhang08090f8edffb
diff --git a/.gitignore b/.gitignore
index 19eb286..3361285 100644
--- a/.gitignore
+++ b/.gitignore
@@ -5,7 +5,7 @@ dist/
!.env.example
.DS_Store
# Only the official benchmark workflow belongs in version control.
-# Historical experiments remain local and are fully deprecated.
+# Raw runs stay local; superseded official experiments live on the research archive branch.
/evals/*
!/evals/README.md
!/evals/accounting.md
diff --git a/evals/README.md b/evals/README.md
index 6e78ad1..01729eb 100644
--- a/evals/README.md
+++ b/evals/README.md
@@ -5,11 +5,10 @@ model and harness with and without Jevgrep. [Accepted architecture](../docs/arch
records the spike decision and its evidence limits. The production implementation
must earn its own results; historical wins do not transfer automatically.
-The maintained [installed-package checkpoint](implementation/swebench/installed.md)
-drives the actual `jg` executable with Sol and reuses the retained Requests
-baseline. It preserves raw Jev requests/responses, coding-agent actions, patches,
-official grades and complete generation bills. This single-task checkpoint is
-not the final ten-task confirmation.
+The maintained [installed-package harness](implementation/swebench/installed.md)
+drives the actual `jg` executable with Sol and reuses the fixed baselines.
+It supports a single-task checkpoint or a complete ten-task cohort, preserving raw
+Jev requests/responses, coding-agent actions, patches, official grades and bills.
A baseline is immutable per task/model/harness and is never rerun to improve a
comparison. The benchmark requires initial Jevgrep use to isolate retrieval's
@@ -26,3 +25,6 @@ Raw traces, repository snapshots and generated study artifacts stay in ignored
run storage. Commit reproducible official tooling and concise evidence reports.
Personal-repository fixtures and custom-rubric evaluations are deprecated and
stay outside Git; they are not product acceptance evidence.
+
+The [research archive](implementation/swebench/research-archive.md) preserves the
+superseded official experiments in Git without adding obsolete runners to `main`.
diff --git a/evals/implementation/swebench/research-archive.md b/evals/implementation/swebench/research-archive.md
new file mode 100644
index 0000000..a4c08e6
--- /dev/null
+++ b/evals/implementation/swebench/research-archive.md
@@ -0,0 +1,40 @@
+# Research preservation
+
+The official SWE-bench spike history is committed in this repository on
+`codex/archive-swebench-spikes-2026-09-26`. It is deliberately separate from the
+maintained CLI and installed-package harness on `main`; do not merge its obsolete
+runners or historical instructions into the product.
+
+The immutable [snapshot](https://github.com/dzhng/jevgrep/tree/06a9d79fac2cee77b91f155fc7b3130dc95708f7)
+preserves all 358 previously untracked official research files at their original
+relative paths: experiment implementations, tests, native-agent harness variants,
+frozen configurations, prompts, result reports and analysis. The
+[manifest](https://github.com/dzhng/jevgrep/blob/06a9d79fac2cee77b91f155fc7b3130dc95708f7/research-snapshot-manifest.json)
+records each file's SHA-256; every byte was verified against the local archive.
+No model calls, baseline reruns, or result recomputation accompanied preservation.
+
+## Reading the evidence
+
+- [Paired agent trace findings](https://github.com/dzhng/jevgrep/blob/06a9d79fac2cee77b91f155fc7b3130dc95708f7/evals/implementation/swebench/paired-sol-research-findings.md)
+ explain how baseline and retrieval-assisted agents explored the same tasks.
+- [Context findings](https://github.com/dzhng/jevgrep/blob/06a9d79fac2cee77b91f155fc7b3130dc95708f7/evals/implementation/swebench/context-findings.md)
+ record early handoff and output-consumption problems.
+- [Historical reports and source](https://github.com/dzhng/jevgrep/tree/06a9d79fac2cee77b91f155fc7b3130dc95708f7/evals/implementation/swebench)
+ preserve alternatives, including failed and superseded experiments. Old model,
+ timing, output and status statements describe their own studies, not the release.
+- [Retrieval lessons](architecture-lessons.md) and the
+ [implementation record](../../../specs/done/jevgrep/README.md) own the synthesis
+ and final product evidence, including the failed original quality gates.
+
+The snapshot is source history, not a promise that every old experiment runs from
+a fresh clone. Raw responses, agent traces, baseline artifacts, source snapshots,
+and evaluator datasets remain in ignored local `evals/runs/swebench/` and
+`evals/runs/tooling/`. Their recorded paths and hashes remain unchanged. These
+large artifacts are not included in Git; losing that local storage would lose
+evidence that cannot be recovered from the source snapshot alone.
+
+Deprecated personal-repository evaluations are excluded from the public archive.
+Their tooling and fixtures were moved intact to
+`~/.local/share/jevgrep/retired-personal-evals/`, outside the checkout. Its local
+`moved-paths.json` records original paths. Redundant `.claude` skill links were
+removed; canonical development skills remain in `.agents/skills/`.