Skip to content

Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.

docs: consolidate replacement cohort evidence and trace limits

David Zhangbeb3038d27e7
diff --git a/specs/jevgrep/README.md b/specs/jevgrep/README.md
index 51e4693..e062eef 100644
--- a/specs/jevgrep/README.md
+++ b/specs/jevgrep/README.md
@@ -7,15 +7,13 @@ Jev classification through Vercel AI Gateway. This is the build plan; production
## Next Agent Prompt
You are implementing this spec. Status: **implementing**, updated **2026-09-26**.
-The work-clock v1 confirmation stopped after Docker's disk-space failure during
-Astropy. That attempt is failed and recovered separately; two completed wins and
-seven unstarted cells remain retained. Do not resume or replace its failed cell.
-Docker is recovered and disposable build cache was cleared. The new full `installed-jg-cpython-work-clock-v2` study is prepared and all ten
-no-call validations pass with identical package and protocol. Scikit-learn and Django have
-official cost wins; Astropy is running. Continue the same sequential runner,
-execute each cell once and grade/account all outcomes.
-Never pool studies or rerun a baseline. [Timing protocol and evidence](assets/timing-protocol.md)
-own the outage, recovered bills and prospective replacement rationale.
+The active full study is `installed-jg-cpython-work-clock-v2`. Continue collecting
+terminal grades and complete bills from its existing sequential runner, then
+aggregate all ten outcomes. [Confirmation evidence](assets/work-clock-confirmation.md)
+owns results and trace findings. Its identical package and protocol passed every
+no-call validation. No candidate changes, replacement attempts or baseline reruns.
+The earlier v1 study remains stopped after a documented Docker outage; its
+recovered costs and two wins cannot be pooled into v2.
Keep production retrieval and the canonical skill unchanged. The installed
package matches the restored bundled-CPython candidate; only the prospective
diff --git a/specs/jevgrep/assets/timing-protocol.md b/specs/jevgrep/assets/timing-protocol.md
index f355dfc..bb9aeb1 100644
--- a/specs/jevgrep/assets/timing-protocol.md
+++ b/specs/jevgrep/assets/timing-protocol.md
@@ -174,17 +174,5 @@ and timing policy match v1 byte-for-byte where applicable. Proof:
all 1,198 manifest hashes and the unchanged failed receipt before paid continuation.
-V2's first cell, scikit-learn, officially resolved with zero infrastructure/errors.
-Its complete 10-generation Sol cost is **$0.2414888** versus **$0.2944360**
-baseline. Jev observed cost is complete at **$0.070983906** (277 client calls),
-separately excluded from scoring. The same unconditional random-state validation
-caveat applies to its patch. Django is running next; full acceptance remains open.
-
-
-V2 Django officially resolved with zero infrastructure/errors. Its complete
-35-generation Sol cost is **$1.2852270**, below **$1.5055472** baseline. Jev's
-known subtotal is **$0.152654124**; full observed cost remains unknown and excluded
-from scoring. The patch includes SQLite collation-only rebuilding, old/new
-collation comparison for related columns, and MySQL nullability handling. Its
-added regression still combines a type change and collation change; the official
-evaluator supplies the same-type regression. Astropy is running next.
+[Replacement confirmation](work-clock-confirmation.md) owns v2 results and paired
+trace findings. This document retains the protocol and outage rationale.
diff --git a/specs/jevgrep/assets/work-clock-confirmation.md b/specs/jevgrep/assets/work-clock-confirmation.md
new file mode 100644
index 0000000..50f2cd9
--- /dev/null
+++ b/specs/jevgrep/assets/work-clock-confirmation.md
@@ -0,0 +1,63 @@
+# Faithful package: replacement work-clock confirmation
+
+Status: running; full-cohort acceptance remains unproven. The authoritative plan
+and receipts are in `evals/runs/swebench/installed-jg-cpython-work-clock-v2/`.
+[Timing protocol](timing-protocol.md) owns the prospective clock rule, the v1
+Docker outage, retained interruption costs and replacement rationale. No v1
+outcome contributes to this cohort. Saved baselines are never rerun.
+
+The production archive, skill, archived harness, task inputs and all 179 installed
+files match v1. Artifact proof: `/tmp/jg-work-clock-v2-freeze-proof.json`.
+All ten no-call validations passed before execution. The package stays frozen;
+this run changes no retrieval policy.
+
+## Retained results
+
+Sol is full task cost; Jev is separately reported and excluded. Pending results
+are not wins, and observed Jev subtotals are not complete totals. This checkpoint
+will be replaced by the final aggregate after all cells finish.
+
+| Task | Official result | Sol | Baseline Sol | Jev observed API cost |
+| --- | --- | ---: | ---: | --- |
+| scikit-learn__scikit-learn-13124 | solved | $0.2414888 | $0.2944360 | $0.070983906 |
+| django__django-15629 | solved | $1.2852270 | $1.5055472 | known $0.152654124; total unknown |
+| astropy__astropy-13579 | solved | $0.3548522 | $0.4286310 | $0.105473382 |
+| pydata__xarray-3305 | solved | $0.3180666 | $0.4937066 | known $0.174302688; total unknown |
+| psf__requests-1142 | solved | $0.2466230 | $0.2685004 | known $0.030577680; total unknown |
+| sympy__sympy-16792 | pending | pending | $0.5479750 | pending |
+| pytest-dev__pytest-6197 | pending | pending | $2.3444580 | pending |
+| sphinx-doc__sphinx-8638 | pending | pending | $1.0594724 | pending |
+| matplotlib__matplotlib-26466 | pending | pending | $0.3957160 | pending |
+| pylint-dev__pylint-4604 | pending | pending | $0.2836264 | pending |
+
+## Paired trace findings
+
+Django received 43 files without source excerpts and explicitly read SQLite before
+patching. Its patch includes the same-type rebuild trigger, old/new collation
+comparison and MySQL nullability handling. Own regression covers BigAutoField to
+CharField, FK/O2O nullability and constraints; official tests supply same-type,
+reversal and many-to-many coverage. Final broad tests: 283 passes, 29 skips.
+MySQL execution was not tested. More shell reads than v1 accompany higher Sol
+cost, but there is no isolated causal cost attribution. Evidence:
+`/tmp/jg-work-clock-v2-django-comparison/`.
+
+Astropy received 45 files and no excerpts. Its shorter source-read, reproduction,
+patch and test sequence resembles the accepted spike; no concrete functional
+defect was established. Own regression is one scalar low-level case. Focused tests
+passed 41; broader tests passed 107, skipped seven and failed one because of the
+same expired leap-second data seen in baseline/spike. The official evaluator
+separately resolved the task. Evidence: `/tmp/jg-work-clock-v2-astropy-comparison/`.
+
+Xarray received 17 files and five excerpts, retaining the decisive Variable source.
+One patch and four focused passing tests avoided the original faithful run's shell
+status error. New regression checks default attr removal and explicit retention;
+coverage remains narrower than the spike's full-module run. Evidence:
+`/tmp/jg-work-clock-v2-xarray-comparison/`.
+
+Scikit-learn repeats the established compatibility caveat despite its official
+solve: unconditional random-state validation rejects invalid seeds previously
+ignored when shuffling is disabled. Official benchmark success does not establish
+all behavior equivalent. No benchmark patch is silently repaired after grading.
+
+These tasks fit the old wall-clock allowance too. Their improvements do not
+establish a clock effect or determinism across independent live model trajectories.