Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.
TypeScript ai-sdkclaude-codeclicode-searchcodexcoding-agentscontext-retrievaldeveloper-toolsjevsemantic-searchtypescriptvercel-ai-gateway
docs: retain accepted cohort and reopen final freshness validation
David Zhang35dd4a550c2a
diff --git a/specs/jevgrep/README.md b/specs/jevgrep/README.mdindex e062eef..6a4e8c0 100644--- a/specs/jevgrep/README.md+++ b/specs/jevgrep/README.md@@ -7,19 +7,26 @@ Jev classification through Vercel AI Gateway. This is the build plan; production## Next Agent PromptYou are implementing this spec. Status: **implementing**, updated **2026-09-26**.-The active full study is `installed-jg-cpython-work-clock-v2`. Continue collecting-terminal grades and complete bills from its existing sequential runner, then-aggregate all ten outcomes. [Confirmation evidence](assets/work-clock-confirmation.md)-owns results and trace findings. Its identical package and protocol passed every-no-call validation. No candidate changes, replacement attempts or baseline reruns.-The earlier v1 study remains stopped after a documented Docker outage; its-recovered costs and two wins cannot be pooled into v2.--Keep production retrieval and the canonical skill unchanged. The installed-package matches the restored bundled-CPython candidate; only the prospective-benchmark timing protocol differs. The failed query and presentation experiments+The full `installed-jg-cpython-work-clock-v2` study is complete and accepted for+its frozen artifact: eight baseline solves preserved, eight cost wins, 33.21%+lower total Sol cost. [Confirmation evidence](assets/work-clock-confirmation.md)+owns all ten results and their limits. No paid benchmark or grader remains active.++Next: integrate the filesystem-freshness fixes being implemented in the isolated+`codex/final-freshness` worktree. Final independent review found buffered navigation+source uploaded after ignore changes and stale non-Python excerpts returned after+role evaluation. Required final freshness checks and per-network-attempt validation+must fix those cases without changing healthy spike requests or output. Run their+red/green installed tests and healthy parity, then determine what remains necessary+to validate the corrected artifact. Do not transfer the frozen cohort's measured+results to changed code without evidence. The pre-fix full default gate passed;+its green result did not cover these newly reproduced cases.++Preserve healthy retrieval semantics and the canonical skill while fixing+filesystem freshness. Keep the measured archive immutable; it matches the+restored bundled-CPython candidate, with only the benchmark timing protocol changed. The failed query and presentation experimentshave not earned promotion. Their remaining cells stay unrun. Prior outcomes-cannot be pooled into the active cohort.+cannot be pooled into the completed confirmation.Evidence and boundaries:@@ -35,9 +42,9 @@ Evidence and boundaries:[presentation trial](assets/presentation-study.md): rejected experiments,source-parity probes and limits on causal explanations.-Priority: finish this frozen quality confirmation; address any failing contract-without weakening acceptance; run whole-spec closeout review; consolidate choices-against the final code; then close-spec. Slice 08 remains open. No publication or+Priority: validate and integrate the freshness correction without weakening+acceptance; resolve the whole-product review; consolidate choices against the+final code; then close-spec. Slice 08 remains open. No publication orrelease tag is authorized for the intentional `0.0.0` development checkpoint.Report observed Jev costs separately; they never enter scored Sol task cost.diff --git a/specs/jevgrep/assets/work-clock-aggregate.json b/specs/jevgrep/assets/work-clock-aggregate.jsonnew file mode 100644index 0000000..a28edce--- /dev/null+++ b/specs/jevgrep/assets/work-clock-aggregate.json@@ -0,0 +1,328 @@+{+ "prospective_full_cohort": true,+ "complete": true,+ "baseline_solves_preserved": 8,+ "fully_billed_solved_cost_wins": 8,+ "accepted": true,+ "known_gateway_subtotal_usd": 5.0907814,+ "fully_billed_total_usd": 5.0907814,+ "jev": {+ "included_in_scored_task_cost": false,+ "basis": "Observed response API metadata, not invoice reconciliation",+ "complete": false,+ "known_cost_usd": 1.0259382,+ "observed_cost_usd": null+ },+ "cells": [+ {+ "task": "scikit-learn__scikit-learn-13124",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": true,+ "resolved": true,+ "baseline_resolved": true,+ "baseline_cost_usd": 0.29443600000000003,+ "fully_billed": true,+ "gateway_cost_usd": 0.2414888,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 277,+ "responses_with_cost": 277,+ "complete": true,+ "observed_cost_usd": 0.070983906,+ "known_cost_usd": 0.070983906,+ "input_tokens": 1690093,+ "output_tokens": 26835,+ "provider_attempts": 294,+ "known_input_tokens": 1690093,+ "known_output_tokens": 26835,+ "known_provider_attempts": 294+ },+ "known_gateway_cost_usd": 0.2414888,+ "successful_cost_win": true+ },+ {+ "task": "django__django-15629",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": true,+ "resolved": true,+ "baseline_resolved": true,+ "baseline_cost_usd": 1.5055472,+ "fully_billed": true,+ "gateway_cost_usd": 1.2852270000000001,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 798,+ "responses_with_cost": 798,+ "complete": false,+ "observed_cost_usd": null,+ "known_cost_usd": 0.152654124,+ "input_tokens": null,+ "output_tokens": null,+ "provider_attempts": null,+ "known_input_tokens": 3634622,+ "known_output_tokens": 88132,+ "known_provider_attempts": 877+ },+ "known_gateway_cost_usd": 1.2852270000000001,+ "successful_cost_win": true+ },+ {+ "task": "astropy__astropy-13579",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": true,+ "resolved": true,+ "baseline_resolved": true,+ "baseline_cost_usd": 0.428631,+ "fully_billed": true,+ "gateway_cost_usd": 0.3548522,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 441,+ "responses_with_cost": 441,+ "complete": true,+ "observed_cost_usd": 0.105473382,+ "known_cost_usd": 0.105473382,+ "input_tokens": 2511271,+ "output_tokens": 37766,+ "provider_attempts": 482,+ "known_input_tokens": 2511271,+ "known_output_tokens": 37766,+ "known_provider_attempts": 482+ },+ "known_gateway_cost_usd": 0.3548522,+ "successful_cost_win": true+ },+ {+ "task": "pydata__xarray-3305",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": true,+ "resolved": true,+ "baseline_resolved": true,+ "baseline_cost_usd": 0.4937066,+ "fully_billed": true,+ "gateway_cost_usd": 0.3180666,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 855,+ "responses_with_cost": 847,+ "complete": false,+ "observed_cost_usd": null,+ "known_cost_usd": 0.174302688,+ "input_tokens": null,+ "output_tokens": null,+ "provider_attempts": null,+ "known_input_tokens": 4150064,+ "known_output_tokens": 97755,+ "known_provider_attempts": 905+ },+ "known_gateway_cost_usd": 0.3180666,+ "successful_cost_win": true+ },+ {+ "task": "psf__requests-1142",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": true,+ "resolved": true,+ "baseline_resolved": true,+ "baseline_cost_usd": 0.2685004,+ "fully_billed": true,+ "gateway_cost_usd": 0.246623,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 116,+ "responses_with_cost": 114,+ "complete": false,+ "observed_cost_usd": null,+ "known_cost_usd": 0.03057768,+ "input_tokens": null,+ "output_tokens": null,+ "provider_attempts": null,+ "known_input_tokens": 728040,+ "known_output_tokens": 12549,+ "known_provider_attempts": 134+ },+ "known_gateway_cost_usd": 0.246623,+ "successful_cost_win": true+ },+ {+ "task": "sympy__sympy-16792",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": true,+ "resolved": true,+ "baseline_resolved": true,+ "baseline_cost_usd": 0.547975,+ "fully_billed": true,+ "gateway_cost_usd": 0.3628038,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 467,+ "responses_with_cost": 467,+ "complete": false,+ "observed_cost_usd": null,+ "known_cost_usd": 0.113127588,+ "input_tokens": null,+ "output_tokens": null,+ "provider_attempts": null,+ "known_input_tokens": 2693514,+ "known_output_tokens": 45460,+ "known_provider_attempts": 500+ },+ "known_gateway_cost_usd": 0.3628038,+ "successful_cost_win": true+ },+ {+ "task": "pytest-dev__pytest-6197",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": true,+ "resolved": true,+ "baseline_resolved": true,+ "baseline_cost_usd": 2.344458,+ "fully_billed": true,+ "gateway_cost_usd": 1.0226676,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 342,+ "responses_with_cost": 342,+ "complete": false,+ "observed_cost_usd": null,+ "known_cost_usd": 0.05827332,+ "input_tokens": null,+ "output_tokens": null,+ "provider_attempts": null,+ "known_input_tokens": 1387460,+ "known_output_tokens": 35370,+ "known_provider_attempts": 384+ },+ "known_gateway_cost_usd": 1.0226676,+ "successful_cost_win": true+ },+ {+ "task": "sphinx-doc__sphinx-8638",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": true,+ "resolved": true,+ "baseline_resolved": true,+ "baseline_cost_usd": 1.0594724,+ "fully_billed": true,+ "gateway_cost_usd": 0.6570284000000001,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 441,+ "responses_with_cost": 441,+ "complete": false,+ "observed_cost_usd": null,+ "known_cost_usd": 0.079899246,+ "input_tokens": null,+ "output_tokens": null,+ "provider_attempts": null,+ "known_input_tokens": 1902363,+ "known_output_tokens": 48235,+ "known_provider_attempts": 477+ },+ "known_gateway_cost_usd": 0.6570284000000001,+ "successful_cost_win": true+ },+ {+ "task": "matplotlib__matplotlib-26466",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": false,+ "resolved": false,+ "baseline_resolved": false,+ "baseline_cost_usd": 0.395716,+ "fully_billed": true,+ "gateway_cost_usd": 0.3444674,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 726,+ "responses_with_cost": 709,+ "complete": false,+ "observed_cost_usd": null,+ "known_cost_usd": 0.171503976,+ "input_tokens": null,+ "output_tokens": null,+ "provider_attempts": null,+ "known_input_tokens": 4083428,+ "known_output_tokens": 71816,+ "known_provider_attempts": 758+ },+ "known_gateway_cost_usd": 0.3444674,+ "successful_cost_win": false+ },+ {+ "task": "pylint-dev__pylint-4604",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": false,+ "resolved": false,+ "baseline_resolved": false,+ "baseline_cost_usd": 0.2836264,+ "fully_billed": true,+ "gateway_cost_usd": 0.2575566,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 350,+ "responses_with_cost": 350,+ "complete": true,+ "observed_cost_usd": 0.06914229,+ "known_cost_usd": 0.06914229,+ "input_tokens": 1646245,+ "output_tokens": 46575,+ "provider_attempts": 372,+ "known_input_tokens": 1646245,+ "known_output_tokens": 46575,+ "known_provider_attempts": 372+ },+ "known_gateway_cost_usd": 0.2575566,+ "successful_cost_win": false+ }+ ]+}diff --git a/specs/jevgrep/assets/work-clock-confirmation.md b/specs/jevgrep/assets/work-clock-confirmation.mdindex 0b73436..293cc81 100644--- a/specs/jevgrep/assets/work-clock-confirmation.md+++ b/specs/jevgrep/assets/work-clock-confirmation.md@@ -1,6 +1,9 @@# Faithful package: replacement work-clock confirmation-Status: running; full-cohort acceptance remains unproven. The authoritative plan+Status: complete; the registered cohort gate passed for its frozen artifact.+Final product review subsequently found filesystem-freshness defects; their fixes+require separate validation before product closeout. This result is not silently+relabelled as a measurement of later code. The authoritative planand receipts are in `evals/runs/swebench/installed-jg-cpython-work-clock-v2/`.[Timing protocol](timing-protocol.md) owns the prospective clock rule, the v1Docker outage, retained interruption costs and replacement rationale. No v1@@ -13,9 +16,12 @@ this run changes no retrieval policy.## Retained results-Sol is full task cost; Jev is separately reported and excluded. Pending results-are not wins, and observed Jev subtotals are not complete totals. This checkpoint-will be replaced by the final aggregate after all cells finish.+Sol is full task cost; Jev is separately reported and excluded. Observed Jev subtotals are not complete totals. The retained+[full aggregate](work-clock-aggregate.json) records all ten graded and fully billed+Sol outcomes, including the two nonpasses. All eight baseline solves are preserved,+and all eight are strict cost wins. Full Sol cost is **$5.0907814** versus+**$7.6220690**, a **33.21% reduction**. Jev known subtotal is **$1.0259382**;+its full total is unknown and excluded.| Task | Official result | Sol | Baseline Sol | Jev observed API cost || --- | --- | ---: | ---: | --- |@@ -26,9 +32,9 @@ will be replaced by the final aggregate after all cells finish.| psf__requests-1142 | solved | $0.2466230 | $0.2685004 | known $0.030577680; total unknown || sympy__sympy-16792 | solved | $0.3628038 | $0.5479750 | known $0.113127588; total unknown || pytest-dev__pytest-6197 | solved | $1.0226676 | $2.3444580 | known $0.058273320; total unknown |-| sphinx-doc__sphinx-8638 | pending | pending | $1.0594724 | pending |-| matplotlib__matplotlib-26466 | pending | pending | $0.3957160 | pending |-| pylint-dev__pylint-4604 | pending | pending | $0.2836264 | pending |+| sphinx-doc__sphinx-8638 | solved | $0.6570284 | $1.0594724 | known $0.079899246; total unknown |+| matplotlib__matplotlib-26466 | unresolved | $0.3444674 | $0.3957160 | known $0.171503976; total unknown |+| pylint-dev__pylint-4604 | unresolved | $0.2575566 | $0.2836264 | $0.069142290 |## Paired trace findings@@ -88,5 +94,17 @@ whose imports were verified, supporting the environmental explanation; this isa test control, not another baseline agent run. Evidence:`/tmp/jg-work-clock-v2-sphinx-comparison/`.+Matplotlib remains unresolved on the official Annotation/OffsetFrom input-copy+regression. Its production fix matches the failed baseline: Annotation copies+its input but OffsetFrom retains a mutable caller array. Passing local tests do+not cover the latter case. Evidence:+`/tmp/jg-work-clock-v2-matplotlib-comparison/`.++Pylint's production fix matches baseline and passes local checker/fixture tests.+Official evaluation fails during collection on missing `IS_PYPY`, before any+assertion runs, as in the retained baseline limitation. It remains a formal+nonpass, not proof of a patch defect or an excluded task. Evidence:+`/tmp/jg-work-clock-v2-pylint-comparison/`.+These tasks fit the old wall-clock allowance too. Their improvements do notestablish a clock effect or determinism across independent live model trajectories.diff --git a/specs/jevgrep/slices/08-release.md b/specs/jevgrep/slices/08-release.mdindex 8d835ec..ca2f6bb 100644--- a/specs/jevgrep/slices/08-release.md+++ b/specs/jevgrep/slices/08-release.md@@ -1,6 +1,6 @@# Confirm quality and the release artifact-Status: open. The first faithful confirmation was rejected; the new frozen work-clock cohort is running after its Django-first continuation gate passed. Depends on: 07. Read [contracts](../contracts.md) first.+Status: open. Frozen work-clock v2 confirmation passed; final review found filesystem-freshness defects requiring correction and validation. Depends on: 07. Read [contracts](../contracts.md) first.## Contract and owner