Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.
TypeScript ai-sdkclaude-codeclicode-searchcodexcoding-agentscontext-retrievaldeveloper-toolsjevsemantic-searchtypescriptvercel-ai-gateway
docs: retain audited final cohort rejection and quality decision
David Zhang3b296d81dab4
diff --git a/specs/jevgrep/README.md b/specs/jevgrep/README.mdindex b451170..cc9b669 100644--- a/specs/jevgrep/README.md+++ b/specs/jevgrep/README.md@@ -7,31 +7,29 @@ Jev classification through Vercel AI Gateway. This is the build plan; production## Next Agent PromptYou are implementing this spec. Status: **implementing**, updated **2026-09-26**.-The full `installed-jg-cpython-work-clock-v2` study is complete and accepted for-its frozen artifact: eight baseline solves preserved, eight cost wins, 33.21%-lower total Sol cost. [Confirmation evidence](assets/work-clock-confirmation.md)-owns all ten results and their limits. That prior study has no active jobs.--Next: finish Pylint grading/accounting and retain the complete-`installed-jg-freshness-v1` aggregate. All ten coding attempts are complete; root-execution session `16656` exited successfully and must not be restarted. Observer-session `12752` owns remaining grading/accounting, with-`/tmp/jg-freshness-observer.log`. The correction is integrated in `92ca7f9`; its isolated worktree is removed.-Full merged verification and exact archive platform checks passed, followed by-all ten frozen no-call validations. [Freshness evidence](assets/source-freshness.md)-owns the archive identity, failure reproductions and verification results. Do not-transfer the previous cohort's measured results to this corrected archive.-[Current confirmation](assets/freshness-confirmation.md) records completed cells;-scikit-learn solved officially but exceeded baseline cost, and Django and Requests-failed official grading, contradicting acceptance for this cohort. Finish the registered-attempts and diagnose that failure; do not replace or pool cells. The choices ledger has-been consolidated against current code; final reconciliation and archive remain.--Preserve healthy retrieval semantics and the canonical skill while fixing-filesystem freshness. Keep the measured archive immutable; it matches the-restored bundled-CPython candidate, with only the benchmark timing protocol changed. The failed query and presentation experiments-have not earned promotion. Their remaining cells stay unrun. Prior outcomes-cannot be pooled into the completed confirmation.+The corrected package's `installed-jg-freshness-v1` cohort is complete and+**rejected**: six of eight baseline solves preserved, four successful cost wins,+and 27.30% lower full Sol cost. [Current confirmation](assets/freshness-confirmation.md)+owns the complete aggregate and all ten trace comparisons. No coding, grading or+accounting process remains active; do not restart terminal sessions or attempts.++The freshness correction is integrated in `92ca7f9`. Full merged verification,+healthy reference parity and exact archive checks on all supported platforms+passed; [freshness evidence](assets/source-freshness.md) owns that proof and its+limits. Django and Requests still failed official behavior cases despite the+agent receiving the relevant code. This does not prove the correction or model+variance alone caused the losses.++Next: audit the final retained evidence and resolve the product-quality decision+with the user before changing the exact spike strategy or acceptance contract.+Do not rerun an unchanged cohort to seek a passing draw, replace attempts, pool+prior outcomes, or archive the spec as accepted. The choices ledger has been+consolidated against current code; final reconciliation and close-spec remain.++The preceding [accepted confirmation](assets/work-clock-confirmation.md) remains+valid for its own frozen archive. It cannot replace this corrected candidate's+failed result. Preserve both archives, the canonical skill and healthy retrieval+semantics. Rejected query/presentation experiments have not earned promotion.Evidence and boundaries:@@ -47,13 +45,14 @@ Evidence and boundaries:[presentation trial](assets/presentation-study.md): rejected experiments,source-parity probes and limits on causal explanations.-Priority: validate the integrated freshness correction without weakening-acceptance; resolve the whole-product review; consolidate choices against the-final code; then close-spec. Slice 08 remains open. No publication or+Priority: resolve the failed quality confirmation without silently weakening+acceptance or changing the measured strategy; then perform final reconciliation+and close-spec. Slice 08 remains open. No publication orrelease tag is authorized for the intentional `0.0.0` development checkpoint.Report observed Jev costs separately; they never enter scored Sol task cost.-No user decision blocks continuing. Use the authorized credentials and retained+Further strategy experiments or acceptance changes require an explicit direction+after reviewing the completed evidence. Preserve authorized credentials and retainedDocker snapshots; do not repair credentials or regenerate baselines. Preserveuser-edited skill files and local historical evidence. Stage only explicit files,keeping deprecated personal-repository evals out of Git.diff --git a/specs/jevgrep/assets/freshness-aggregate.json b/specs/jevgrep/assets/freshness-aggregate.jsonnew file mode 100644index 0000000..f86c13a--- /dev/null+++ b/specs/jevgrep/assets/freshness-aggregate.json@@ -0,0 +1,328 @@+{+ "prospective_full_cohort": true,+ "complete": true,+ "baseline_solves_preserved": 6,+ "fully_billed_solved_cost_wins": 4,+ "accepted": false,+ "known_gateway_subtotal_usd": 5.5410776,+ "fully_billed_total_usd": 5.5410776,+ "jev": {+ "included_in_scored_task_cost": false,+ "basis": "Observed response API metadata, not invoice reconciliation",+ "complete": false,+ "known_cost_usd": 1.035782622,+ "observed_cost_usd": null+ },+ "cells": [+ {+ "task": "scikit-learn__scikit-learn-13124",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": true,+ "resolved": true,+ "baseline_resolved": true,+ "baseline_cost_usd": 0.29443600000000003,+ "fully_billed": true,+ "gateway_cost_usd": 0.343483,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 206,+ "responses_with_cost": 203,+ "complete": false,+ "observed_cost_usd": null,+ "known_cost_usd": 0.046611348,+ "input_tokens": null,+ "output_tokens": null,+ "provider_attempts": null,+ "known_input_tokens": 1109794,+ "known_output_tokens": 20564,+ "known_provider_attempts": 227+ },+ "known_gateway_cost_usd": 0.343483,+ "successful_cost_win": false+ },+ {+ "task": "django__django-15629",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": false,+ "resolved": false,+ "baseline_resolved": true,+ "baseline_cost_usd": 1.5055472,+ "fully_billed": true,+ "gateway_cost_usd": 1.069589,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 870,+ "responses_with_cost": 848,+ "complete": false,+ "observed_cost_usd": null,+ "known_cost_usd": 0.158282796,+ "input_tokens": null,+ "output_tokens": null,+ "provider_attempts": null,+ "known_input_tokens": 3768638,+ "known_output_tokens": 93007,+ "known_provider_attempts": 899+ },+ "known_gateway_cost_usd": 1.069589,+ "successful_cost_win": false+ },+ {+ "task": "astropy__astropy-13579",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": true,+ "resolved": true,+ "baseline_resolved": true,+ "baseline_cost_usd": 0.428631,+ "fully_billed": true,+ "gateway_cost_usd": 0.3914126,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 341,+ "responses_with_cost": 341,+ "complete": false,+ "observed_cost_usd": null,+ "known_cost_usd": 0.07780647,+ "input_tokens": null,+ "output_tokens": null,+ "provider_attempts": null,+ "known_input_tokens": 1852535,+ "known_output_tokens": 29474,+ "known_provider_attempts": 384+ },+ "known_gateway_cost_usd": 0.3914126,+ "successful_cost_win": true+ },+ {+ "task": "pydata__xarray-3305",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": true,+ "resolved": true,+ "baseline_resolved": true,+ "baseline_cost_usd": 0.4937066,+ "fully_billed": true,+ "gateway_cost_usd": 0.3182098,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 517,+ "responses_with_cost": 517,+ "complete": true,+ "observed_cost_usd": 0.123411498,+ "known_cost_usd": 0.123411498,+ "input_tokens": 2938369,+ "output_tokens": 53901,+ "provider_attempts": 644,+ "known_input_tokens": 2938369,+ "known_output_tokens": 53901,+ "known_provider_attempts": 644+ },+ "known_gateway_cost_usd": 0.3182098,+ "successful_cost_win": true+ },+ {+ "task": "psf__requests-1142",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": false,+ "resolved": false,+ "baseline_resolved": true,+ "baseline_cost_usd": 0.2685004,+ "fully_billed": true,+ "gateway_cost_usd": 0.23880800000000002,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 72,+ "responses_with_cost": 72,+ "complete": true,+ "observed_cost_usd": 0.011300058,+ "known_cost_usd": 0.011300058,+ "input_tokens": 269049,+ "output_tokens": 8911,+ "provider_attempts": 83,+ "known_input_tokens": 269049,+ "known_output_tokens": 8911,+ "known_provider_attempts": 83+ },+ "known_gateway_cost_usd": 0.23880800000000002,+ "successful_cost_win": false+ },+ {+ "task": "sympy__sympy-16792",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": true,+ "resolved": true,+ "baseline_resolved": true,+ "baseline_cost_usd": 0.547975,+ "fully_billed": true,+ "gateway_cost_usd": 0.3320948,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 413,+ "responses_with_cost": 413,+ "complete": false,+ "observed_cost_usd": null,+ "known_cost_usd": 0.098661402,+ "input_tokens": null,+ "output_tokens": null,+ "provider_attempts": null,+ "known_input_tokens": 2349081,+ "known_output_tokens": 40385,+ "known_provider_attempts": 490+ },+ "known_gateway_cost_usd": 0.3320948,+ "successful_cost_win": true+ },+ {+ "task": "pytest-dev__pytest-6197",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": true,+ "resolved": true,+ "baseline_resolved": true,+ "baseline_cost_usd": 2.344458,+ "fully_billed": true,+ "gateway_cost_usd": 0.8116844,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 578,+ "responses_with_cost": 578,+ "complete": true,+ "observed_cost_usd": 0.098483154,+ "known_cost_usd": 0.098483154,+ "input_tokens": 2344837,+ "output_tokens": 65717,+ "provider_attempts": 632,+ "known_input_tokens": 2344837,+ "known_output_tokens": 65717,+ "known_provider_attempts": 632+ },+ "known_gateway_cost_usd": 0.8116844,+ "successful_cost_win": true+ },+ {+ "task": "sphinx-doc__sphinx-8638",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": true,+ "resolved": true,+ "baseline_resolved": true,+ "baseline_cost_usd": 1.0594724,+ "fully_billed": true,+ "gateway_cost_usd": 1.2444744,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 1334,+ "responses_with_cost": 1334,+ "complete": false,+ "observed_cost_usd": null,+ "known_cost_usd": 0.196647108,+ "input_tokens": null,+ "output_tokens": null,+ "provider_attempts": null,+ "known_input_tokens": 4682074,+ "known_output_tokens": 156382,+ "known_provider_attempts": 1510+ },+ "known_gateway_cost_usd": 1.2444744,+ "successful_cost_win": false+ },+ {+ "task": "matplotlib__matplotlib-26466",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": false,+ "resolved": false,+ "baseline_resolved": false,+ "baseline_cost_usd": 0.395716,+ "fully_billed": true,+ "gateway_cost_usd": 0.4256784,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 700,+ "responses_with_cost": 700,+ "complete": false,+ "observed_cost_usd": null,+ "known_cost_usd": 0.157293066,+ "input_tokens": null,+ "output_tokens": null,+ "provider_attempts": null,+ "known_input_tokens": 3745073,+ "known_output_tokens": 69302,+ "known_provider_attempts": 764+ },+ "known_gateway_cost_usd": 0.4256784,+ "successful_cost_win": false+ },+ {+ "task": "pylint-dev__pylint-4604",+ "status": "completed",+ "terminal": true,+ "grading_attempted": true,+ "graded": true,+ "protocol_valid": true,+ "official_resolved": false,+ "resolved": false,+ "baseline_resolved": false,+ "baseline_cost_usd": 0.2836264,+ "fully_billed": true,+ "gateway_cost_usd": 0.3656432,+ "jev": {+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",+ "included_in_scored_task_cost": false,+ "client_calls": 349,+ "responses_with_cost": 349,+ "complete": false,+ "observed_cost_usd": null,+ "known_cost_usd": 0.067285722,+ "input_tokens": null,+ "output_tokens": null,+ "provider_attempts": null,+ "known_input_tokens": 1602041,+ "known_output_tokens": 43900,+ "known_provider_attempts": 376+ },+ "known_gateway_cost_usd": 0.3656432,+ "successful_cost_win": false+ }+ ]+}diff --git a/specs/jevgrep/assets/freshness-confirmation.md b/specs/jevgrep/assets/freshness-confirmation.mdindex 57784d3..037264d 100644--- a/specs/jevgrep/assets/freshness-confirmation.md+++ b/specs/jevgrep/assets/freshness-confirmation.md@@ -1,7 +1,13 @@# Corrected-package confirmation-Status: running; acceptance contradicted by lost Django and Requests baseline solves.-Finish and retain the registered cohort rather than replacing that attempt.+Status: complete; rejected by the registered acceptance gate. Six of ten tasks+solved, six of eight baseline solves were preserved, and four solved tasks were+strict cost wins. All ten Sol bills are complete: **$5.5410776** versus baseline+**$7.6220690**, a **27.30% reduction including failed tasks**. Lower total cost+does not compensate for the lost solves. Jev's observed subtotal is+**$1.035782622**; its full total is unknown and excluded from scored cost.+The [complete aggregate](freshness-aggregate.json) is retained byte-for-byte from+the study, SHA-256 `37f2dc0883d0e366b75e74b0f178498a393da9f8ef9cd573d5881630efc0d65a`.This is a separate prospective ten-task cohort for the[source-freshness correction](source-freshness.md), not a replacement of individualcells from the preceding accepted study. The frozen plan, archived runner and@@ -23,6 +29,7 @@ separately and excluded; incomplete observations are subtotals, not totals.| pytest-dev__pytest-6197 | solved; cost win | $0.8116844 | $2.3444580 | $0.098483154 || sphinx-doc__sphinx-8638 | solved; not a cost win | $1.2444744 | $1.0594724 | known $0.196647108; total unknown || matplotlib__matplotlib-26466 | unresolved | $0.4256784 | $0.3957160 | known $0.157293066; total unknown |+| pylint-dev__pylint-4604 | unresolved at collection | $0.3656432 | $0.2836264 | known $0.067285722; total unknown |Scikit-learn returned five relevant files in 5,038 bytes, implementation readingleads and two test source blocks. The agent then read the splitter and tests, searched documentation,@@ -37,8 +44,9 @@ and test work does not isolate a causal cost effect from the freshness correctioRoot execution log: `/tmp/jg-freshness-v1-run.log`. Per-task grade and accountingreceipts, complete event streams, patches and request/response evidence remain-with the attempts. Acceptance stays unproven until all ten outcomes are retained-and the full aggregate is evaluated.+with the attempts. All coding, grading and accounting processes are terminal;+no attempt was replaced and no baseline was rerun. The product quality gate+remains open because the completed aggregate rejects the candidate.Django's packet contained 51 files in 8,071 bytes without excerpts. SQLite waslisted and its critical type-only rebuild guard was read, untruncated, before the@@ -122,3 +130,23 @@ After correcting an initial NumPy slice attempt, local text tests passed 104 wit12 skips and offsetbox tests passed 285 with two skips. Official evaluation stillfailed the combined Annotation/OffsetFrom input-copy regression, as in baseline.Evidence: `/tmp/jg-freshness-v1-matplotlib-comparison/findings.md`.++Pylint returned 32 files in 6,990 bytes with two fixture blocks and a reading lead+to the relevant checker. The agent read that checker, added qualified/unqualified+ABC coverage and passed one focused plus 20 checker tests. Its implementation+differs structurally from earlier variants but handles the inspected paths+equivalently. Official evaluation failed during collection on missing `IS_PYPY`,+before patch assertions ran. That retained environmental limitation remains a+formal nonpass, not a proven patch defect. Evidence:+`/tmp/jg-freshness-v1-pylint-comparison/findings.md`.++## Interpretation boundary++The two lost baseline solves are concrete patch/test omissions despite delivered+code: Django's same-type rebuild condition and Requests' HEAD behavior. This+does not isolate model variance or the freshness correction as the sole cause.+Healthy controlled parity passed, but it is not a substitute for this failed+live confirmation. The preceding accepted cohort remains evidence about its+own frozen archive, not a result that can replace this one. No further unchanged+cohort is justified merely to obtain a passing draw; changes to retrieval, skill+or acceptance require a separate explicit decision and identified experiment.diff --git a/specs/jevgrep/assets/source-freshness.md b/specs/jevgrep/assets/source-freshness.mdindex 21ba5a5..d314ba6 100644--- a/specs/jevgrep/assets/source-freshness.md+++ b/specs/jevgrep/assets/source-freshness.md@@ -47,8 +47,9 @@ tests, 24 benchmark-harness tests and 24 installed tests, plus typecheck and linEvidence: `/tmp/jg-freshness-merged-verify-serial.log`; no test allowance was changed.The accepted work-clock cohort measured the preceding archive. A separate frozen-ten-task treatment cohort is running at+ten-task treatment cohort completed at`evals/runs/swebench/installed-jg-freshness-v1/`. Preparation and all ten no-callvalidations passed before execution. Its archived runner owns the frozen inputs;saved baselines remain immutable and earlier treatment cells cannot be pooled-into its result. Run log: `/tmp/jg-freshness-v1-run.log`.+into its result. The [completed confirmation](freshness-confirmation.md) failed+the quality gate despite passing deterministic verification. Run log: `/tmp/jg-freshness-v1-run.log`.diff --git a/specs/jevgrep/slices/08-release.md b/specs/jevgrep/slices/08-release.mdindex ca2f6bb..1f9bff2 100644--- a/specs/jevgrep/slices/08-release.md+++ b/specs/jevgrep/slices/08-release.md@@ -1,6 +1,6 @@# Confirm quality and the release artifact-Status: open. Frozen work-clock v2 confirmation passed; final review found filesystem-freshness defects requiring correction and validation. Depends on: 07. Read [contracts](../contracts.md) first.+Status: open. Freshness fixes and deterministic verification passed, but the corrected package's complete cohort lost Django and Requests baseline solves. See [final confirmation](../assets/freshness-confirmation.md). Depends on: 07. Read [contracts](../contracts.md) first.## Contract and owner