Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.
TypeScript ai-sdkclaude-codeclicode-searchcodexcoding-agentscontext-retrievaldeveloper-toolsjevsemantic-searchtypescriptvercel-ai-gateway
docs: index preserved SWE-bench research and retire local eval clutter
David Zhang08090f8edffb
diff --git a/.gitignore b/.gitignoreindex 19eb286..3361285 100644--- a/.gitignore+++ b/.gitignore@@ -5,7 +5,7 @@ dist/!.env.example.DS_Store# Only the official benchmark workflow belongs in version control.-# Historical experiments remain local and are fully deprecated.+# Raw runs stay local; superseded official experiments live on the research archive branch./evals/*!/evals/README.md!/evals/accounting.mddiff --git a/evals/README.md b/evals/README.mdindex 6e78ad1..01729eb 100644--- a/evals/README.md+++ b/evals/README.md@@ -5,11 +5,10 @@ model and harness with and without Jevgrep. [Accepted architecture](../docs/archrecords the spike decision and its evidence limits. The production implementationmust earn its own results; historical wins do not transfer automatically.-The maintained [installed-package checkpoint](implementation/swebench/installed.md)-drives the actual `jg` executable with Sol and reuses the retained Requests-baseline. It preserves raw Jev requests/responses, coding-agent actions, patches,-official grades and complete generation bills. This single-task checkpoint is-not the final ten-task confirmation.+The maintained [installed-package harness](implementation/swebench/installed.md)+drives the actual `jg` executable with Sol and reuses the fixed baselines.+It supports a single-task checkpoint or a complete ten-task cohort, preserving raw+Jev requests/responses, coding-agent actions, patches, official grades and bills.A baseline is immutable per task/model/harness and is never rerun to improve acomparison. The benchmark requires initial Jevgrep use to isolate retrieval's@@ -26,3 +25,6 @@ Raw traces, repository snapshots and generated study artifacts stay in ignoredrun storage. Commit reproducible official tooling and concise evidence reports.Personal-repository fixtures and custom-rubric evaluations are deprecated andstay outside Git; they are not product acceptance evidence.++The [research archive](implementation/swebench/research-archive.md) preserves the+superseded official experiments in Git without adding obsolete runners to `main`.diff --git a/evals/implementation/swebench/research-archive.md b/evals/implementation/swebench/research-archive.mdnew file mode 100644index 0000000..a4c08e6--- /dev/null+++ b/evals/implementation/swebench/research-archive.md@@ -0,0 +1,40 @@+# Research preservation++The official SWE-bench spike history is committed in this repository on+`codex/archive-swebench-spikes-2026-09-26`. It is deliberately separate from the+maintained CLI and installed-package harness on `main`; do not merge its obsolete+runners or historical instructions into the product.++The immutable [snapshot](https://github.com/dzhng/jevgrep/tree/06a9d79fac2cee77b91f155fc7b3130dc95708f7)+preserves all 358 previously untracked official research files at their original+relative paths: experiment implementations, tests, native-agent harness variants,+frozen configurations, prompts, result reports and analysis. The+[manifest](https://github.com/dzhng/jevgrep/blob/06a9d79fac2cee77b91f155fc7b3130dc95708f7/research-snapshot-manifest.json)+records each file's SHA-256; every byte was verified against the local archive.+No model calls, baseline reruns, or result recomputation accompanied preservation.++## Reading the evidence++- [Paired agent trace findings](https://github.com/dzhng/jevgrep/blob/06a9d79fac2cee77b91f155fc7b3130dc95708f7/evals/implementation/swebench/paired-sol-research-findings.md)+ explain how baseline and retrieval-assisted agents explored the same tasks.+- [Context findings](https://github.com/dzhng/jevgrep/blob/06a9d79fac2cee77b91f155fc7b3130dc95708f7/evals/implementation/swebench/context-findings.md)+ record early handoff and output-consumption problems.+- [Historical reports and source](https://github.com/dzhng/jevgrep/tree/06a9d79fac2cee77b91f155fc7b3130dc95708f7/evals/implementation/swebench)+ preserve alternatives, including failed and superseded experiments. Old model,+ timing, output and status statements describe their own studies, not the release.+- [Retrieval lessons](architecture-lessons.md) and the+ [implementation record](../../../specs/done/jevgrep/README.md) own the synthesis+ and final product evidence, including the failed original quality gates.++The snapshot is source history, not a promise that every old experiment runs from+a fresh clone. Raw responses, agent traces, baseline artifacts, source snapshots,+and evaluator datasets remain in ignored local `evals/runs/swebench/` and+`evals/runs/tooling/`. Their recorded paths and hashes remain unchanged. These+large artifacts are not included in Git; losing that local storage would lose+evidence that cannot be recovered from the source snapshot alone.++Deprecated personal-repository evaluations are excluded from the public archive.+Their tooling and fixtures were moved intact to+`~/.local/share/jevgrep/retired-personal-evals/`, outside the checkout. Its local+`moved-paths.json` records original paths. Redundant `.claude` skill links were+removed; canonical development skills remain in `.agents/skills/`.