Skip to content

Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.

docs: retain accepted cohort and reopen final freshness validation

David Zhang35dd4a550c2a
diff --git a/specs/jevgrep/README.md b/specs/jevgrep/README.md
index e062eef..6a4e8c0 100644
--- a/specs/jevgrep/README.md
+++ b/specs/jevgrep/README.md
@@ -7,19 +7,26 @@ Jev classification through Vercel AI Gateway. This is the build plan; production
## Next Agent Prompt
You are implementing this spec. Status: **implementing**, updated **2026-09-26**.
-The active full study is `installed-jg-cpython-work-clock-v2`. Continue collecting
-terminal grades and complete bills from its existing sequential runner, then
-aggregate all ten outcomes. [Confirmation evidence](assets/work-clock-confirmation.md)
-owns results and trace findings. Its identical package and protocol passed every
-no-call validation. No candidate changes, replacement attempts or baseline reruns.
-The earlier v1 study remains stopped after a documented Docker outage; its
-recovered costs and two wins cannot be pooled into v2.
-
-Keep production retrieval and the canonical skill unchanged. The installed
-package matches the restored bundled-CPython candidate; only the prospective
-benchmark timing protocol differs. The failed query and presentation experiments
+The full `installed-jg-cpython-work-clock-v2` study is complete and accepted for
+its frozen artifact: eight baseline solves preserved, eight cost wins, 33.21%
+lower total Sol cost. [Confirmation evidence](assets/work-clock-confirmation.md)
+owns all ten results and their limits. No paid benchmark or grader remains active.
+
+Next: integrate the filesystem-freshness fixes being implemented in the isolated
+`codex/final-freshness` worktree. Final independent review found buffered navigation
+source uploaded after ignore changes and stale non-Python excerpts returned after
+role evaluation. Required final freshness checks and per-network-attempt validation
+must fix those cases without changing healthy spike requests or output. Run their
+red/green installed tests and healthy parity, then determine what remains necessary
+to validate the corrected artifact. Do not transfer the frozen cohort's measured
+results to changed code without evidence. The pre-fix full default gate passed;
+its green result did not cover these newly reproduced cases.
+
+Preserve healthy retrieval semantics and the canonical skill while fixing
+filesystem freshness. Keep the measured archive immutable; it matches the
+restored bundled-CPython candidate, with only the benchmark timing protocol changed. The failed query and presentation experiments
have not earned promotion. Their remaining cells stay unrun. Prior outcomes
-cannot be pooled into the active cohort.
+cannot be pooled into the completed confirmation.
Evidence and boundaries:
@@ -35,9 +42,9 @@ Evidence and boundaries:
[presentation trial](assets/presentation-study.md): rejected experiments,
source-parity probes and limits on causal explanations.
-Priority: finish this frozen quality confirmation; address any failing contract
-without weakening acceptance; run whole-spec closeout review; consolidate choices
-against the final code; then close-spec. Slice 08 remains open. No publication or
+Priority: validate and integrate the freshness correction without weakening
+acceptance; resolve the whole-product review; consolidate choices against the
+final code; then close-spec. Slice 08 remains open. No publication or
release tag is authorized for the intentional `0.0.0` development checkpoint.
Report observed Jev costs separately; they never enter scored Sol task cost.
diff --git a/specs/jevgrep/assets/work-clock-aggregate.json b/specs/jevgrep/assets/work-clock-aggregate.json
new file mode 100644
index 0000000..a28edce
--- /dev/null
+++ b/specs/jevgrep/assets/work-clock-aggregate.json
@@ -0,0 +1,328 @@
+{
+ "prospective_full_cohort": true,
+ "complete": true,
+ "baseline_solves_preserved": 8,
+ "fully_billed_solved_cost_wins": 8,
+ "accepted": true,
+ "known_gateway_subtotal_usd": 5.0907814,
+ "fully_billed_total_usd": 5.0907814,
+ "jev": {
+ "included_in_scored_task_cost": false,
+ "basis": "Observed response API metadata, not invoice reconciliation",
+ "complete": false,
+ "known_cost_usd": 1.0259382,
+ "observed_cost_usd": null
+ },
+ "cells": [
+ {
+ "task": "scikit-learn__scikit-learn-13124",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 0.29443600000000003,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.2414888,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 277,
+ "responses_with_cost": 277,
+ "complete": true,
+ "observed_cost_usd": 0.070983906,
+ "known_cost_usd": 0.070983906,
+ "input_tokens": 1690093,
+ "output_tokens": 26835,
+ "provider_attempts": 294,
+ "known_input_tokens": 1690093,
+ "known_output_tokens": 26835,
+ "known_provider_attempts": 294
+ },
+ "known_gateway_cost_usd": 0.2414888,
+ "successful_cost_win": true
+ },
+ {
+ "task": "django__django-15629",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 1.5055472,
+ "fully_billed": true,
+ "gateway_cost_usd": 1.2852270000000001,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 798,
+ "responses_with_cost": 798,
+ "complete": false,
+ "observed_cost_usd": null,
+ "known_cost_usd": 0.152654124,
+ "input_tokens": null,
+ "output_tokens": null,
+ "provider_attempts": null,
+ "known_input_tokens": 3634622,
+ "known_output_tokens": 88132,
+ "known_provider_attempts": 877
+ },
+ "known_gateway_cost_usd": 1.2852270000000001,
+ "successful_cost_win": true
+ },
+ {
+ "task": "astropy__astropy-13579",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 0.428631,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.3548522,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 441,
+ "responses_with_cost": 441,
+ "complete": true,
+ "observed_cost_usd": 0.105473382,
+ "known_cost_usd": 0.105473382,
+ "input_tokens": 2511271,
+ "output_tokens": 37766,
+ "provider_attempts": 482,
+ "known_input_tokens": 2511271,
+ "known_output_tokens": 37766,
+ "known_provider_attempts": 482
+ },
+ "known_gateway_cost_usd": 0.3548522,
+ "successful_cost_win": true
+ },
+ {
+ "task": "pydata__xarray-3305",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 0.4937066,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.3180666,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 855,
+ "responses_with_cost": 847,
+ "complete": false,
+ "observed_cost_usd": null,
+ "known_cost_usd": 0.174302688,
+ "input_tokens": null,
+ "output_tokens": null,
+ "provider_attempts": null,
+ "known_input_tokens": 4150064,
+ "known_output_tokens": 97755,
+ "known_provider_attempts": 905
+ },
+ "known_gateway_cost_usd": 0.3180666,
+ "successful_cost_win": true
+ },
+ {
+ "task": "psf__requests-1142",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 0.2685004,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.246623,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 116,
+ "responses_with_cost": 114,
+ "complete": false,
+ "observed_cost_usd": null,
+ "known_cost_usd": 0.03057768,
+ "input_tokens": null,
+ "output_tokens": null,
+ "provider_attempts": null,
+ "known_input_tokens": 728040,
+ "known_output_tokens": 12549,
+ "known_provider_attempts": 134
+ },
+ "known_gateway_cost_usd": 0.246623,
+ "successful_cost_win": true
+ },
+ {
+ "task": "sympy__sympy-16792",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 0.547975,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.3628038,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 467,
+ "responses_with_cost": 467,
+ "complete": false,
+ "observed_cost_usd": null,
+ "known_cost_usd": 0.113127588,
+ "input_tokens": null,
+ "output_tokens": null,
+ "provider_attempts": null,
+ "known_input_tokens": 2693514,
+ "known_output_tokens": 45460,
+ "known_provider_attempts": 500
+ },
+ "known_gateway_cost_usd": 0.3628038,
+ "successful_cost_win": true
+ },
+ {
+ "task": "pytest-dev__pytest-6197",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 2.344458,
+ "fully_billed": true,
+ "gateway_cost_usd": 1.0226676,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 342,
+ "responses_with_cost": 342,
+ "complete": false,
+ "observed_cost_usd": null,
+ "known_cost_usd": 0.05827332,
+ "input_tokens": null,
+ "output_tokens": null,
+ "provider_attempts": null,
+ "known_input_tokens": 1387460,
+ "known_output_tokens": 35370,
+ "known_provider_attempts": 384
+ },
+ "known_gateway_cost_usd": 1.0226676,
+ "successful_cost_win": true
+ },
+ {
+ "task": "sphinx-doc__sphinx-8638",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 1.0594724,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.6570284000000001,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 441,
+ "responses_with_cost": 441,
+ "complete": false,
+ "observed_cost_usd": null,
+ "known_cost_usd": 0.079899246,
+ "input_tokens": null,
+ "output_tokens": null,
+ "provider_attempts": null,
+ "known_input_tokens": 1902363,
+ "known_output_tokens": 48235,
+ "known_provider_attempts": 477
+ },
+ "known_gateway_cost_usd": 0.6570284000000001,
+ "successful_cost_win": true
+ },
+ {
+ "task": "matplotlib__matplotlib-26466",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": false,
+ "resolved": false,
+ "baseline_resolved": false,
+ "baseline_cost_usd": 0.395716,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.3444674,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 726,
+ "responses_with_cost": 709,
+ "complete": false,
+ "observed_cost_usd": null,
+ "known_cost_usd": 0.171503976,
+ "input_tokens": null,
+ "output_tokens": null,
+ "provider_attempts": null,
+ "known_input_tokens": 4083428,
+ "known_output_tokens": 71816,
+ "known_provider_attempts": 758
+ },
+ "known_gateway_cost_usd": 0.3444674,
+ "successful_cost_win": false
+ },
+ {
+ "task": "pylint-dev__pylint-4604",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": false,
+ "resolved": false,
+ "baseline_resolved": false,
+ "baseline_cost_usd": 0.2836264,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.2575566,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 350,
+ "responses_with_cost": 350,
+ "complete": true,
+ "observed_cost_usd": 0.06914229,
+ "known_cost_usd": 0.06914229,
+ "input_tokens": 1646245,
+ "output_tokens": 46575,
+ "provider_attempts": 372,
+ "known_input_tokens": 1646245,
+ "known_output_tokens": 46575,
+ "known_provider_attempts": 372
+ },
+ "known_gateway_cost_usd": 0.2575566,
+ "successful_cost_win": false
+ }
+ ]
+}
diff --git a/specs/jevgrep/assets/work-clock-confirmation.md b/specs/jevgrep/assets/work-clock-confirmation.md
index 0b73436..293cc81 100644
--- a/specs/jevgrep/assets/work-clock-confirmation.md
+++ b/specs/jevgrep/assets/work-clock-confirmation.md
@@ -1,6 +1,9 @@
# Faithful package: replacement work-clock confirmation
-Status: running; full-cohort acceptance remains unproven. The authoritative plan
+Status: complete; the registered cohort gate passed for its frozen artifact.
+Final product review subsequently found filesystem-freshness defects; their fixes
+require separate validation before product closeout. This result is not silently
+relabelled as a measurement of later code. The authoritative plan
and receipts are in `evals/runs/swebench/installed-jg-cpython-work-clock-v2/`.
[Timing protocol](timing-protocol.md) owns the prospective clock rule, the v1
Docker outage, retained interruption costs and replacement rationale. No v1
@@ -13,9 +16,12 @@ this run changes no retrieval policy.
## Retained results
-Sol is full task cost; Jev is separately reported and excluded. Pending results
-are not wins, and observed Jev subtotals are not complete totals. This checkpoint
-will be replaced by the final aggregate after all cells finish.
+Sol is full task cost; Jev is separately reported and excluded. Observed Jev subtotals are not complete totals. The retained
+[full aggregate](work-clock-aggregate.json) records all ten graded and fully billed
+Sol outcomes, including the two nonpasses. All eight baseline solves are preserved,
+and all eight are strict cost wins. Full Sol cost is **$5.0907814** versus
+**$7.6220690**, a **33.21% reduction**. Jev known subtotal is **$1.0259382**;
+its full total is unknown and excluded.
| Task | Official result | Sol | Baseline Sol | Jev observed API cost |
| --- | --- | ---: | ---: | --- |
@@ -26,9 +32,9 @@ will be replaced by the final aggregate after all cells finish.
| psf__requests-1142 | solved | $0.2466230 | $0.2685004 | known $0.030577680; total unknown |
| sympy__sympy-16792 | solved | $0.3628038 | $0.5479750 | known $0.113127588; total unknown |
| pytest-dev__pytest-6197 | solved | $1.0226676 | $2.3444580 | known $0.058273320; total unknown |
-| sphinx-doc__sphinx-8638 | pending | pending | $1.0594724 | pending |
-| matplotlib__matplotlib-26466 | pending | pending | $0.3957160 | pending |
-| pylint-dev__pylint-4604 | pending | pending | $0.2836264 | pending |
+| sphinx-doc__sphinx-8638 | solved | $0.6570284 | $1.0594724 | known $0.079899246; total unknown |
+| matplotlib__matplotlib-26466 | unresolved | $0.3444674 | $0.3957160 | known $0.171503976; total unknown |
+| pylint-dev__pylint-4604 | unresolved | $0.2575566 | $0.2836264 | $0.069142290 |
## Paired trace findings
@@ -88,5 +94,17 @@ whose imports were verified, supporting the environmental explanation; this is
a test control, not another baseline agent run. Evidence:
`/tmp/jg-work-clock-v2-sphinx-comparison/`.
+Matplotlib remains unresolved on the official Annotation/OffsetFrom input-copy
+regression. Its production fix matches the failed baseline: Annotation copies
+its input but OffsetFrom retains a mutable caller array. Passing local tests do
+not cover the latter case. Evidence:
+`/tmp/jg-work-clock-v2-matplotlib-comparison/`.
+
+Pylint's production fix matches baseline and passes local checker/fixture tests.
+Official evaluation fails during collection on missing `IS_PYPY`, before any
+assertion runs, as in the retained baseline limitation. It remains a formal
+nonpass, not proof of a patch defect or an excluded task. Evidence:
+`/tmp/jg-work-clock-v2-pylint-comparison/`.
+
These tasks fit the old wall-clock allowance too. Their improvements do not
establish a clock effect or determinism across independent live model trajectories.
diff --git a/specs/jevgrep/slices/08-release.md b/specs/jevgrep/slices/08-release.md
index 8d835ec..ca2f6bb 100644
--- a/specs/jevgrep/slices/08-release.md
+++ b/specs/jevgrep/slices/08-release.md
@@ -1,6 +1,6 @@
# Confirm quality and the release artifact
-Status: open. The first faithful confirmation was rejected; the new frozen work-clock cohort is running after its Django-first continuation gate passed. Depends on: 07. Read [contracts](../contracts.md) first.
+Status: open. Frozen work-clock v2 confirmation passed; final review found filesystem-freshness defects requiring correction and validation. Depends on: 07. Read [contracts](../contracts.md) first.
## Contract and owner