Skip to content

Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.

docs: retain audited final cohort rejection and quality decision

David Zhang3b296d81dab4
diff --git a/specs/jevgrep/README.md b/specs/jevgrep/README.md
index b451170..cc9b669 100644
--- a/specs/jevgrep/README.md
+++ b/specs/jevgrep/README.md
@@ -7,31 +7,29 @@ Jev classification through Vercel AI Gateway. This is the build plan; production
## Next Agent Prompt
You are implementing this spec. Status: **implementing**, updated **2026-09-26**.
-The full `installed-jg-cpython-work-clock-v2` study is complete and accepted for
-its frozen artifact: eight baseline solves preserved, eight cost wins, 33.21%
-lower total Sol cost. [Confirmation evidence](assets/work-clock-confirmation.md)
-owns all ten results and their limits. That prior study has no active jobs.
-
-Next: finish Pylint grading/accounting and retain the complete
-`installed-jg-freshness-v1` aggregate. All ten coding attempts are complete; root
-execution session `16656` exited successfully and must not be restarted. Observer
-session `12752` owns remaining grading/accounting, with
-`/tmp/jg-freshness-observer.log`. The correction is integrated in `92ca7f9`; its isolated worktree is removed.
-Full merged verification and exact archive platform checks passed, followed by
-all ten frozen no-call validations. [Freshness evidence](assets/source-freshness.md)
-owns the archive identity, failure reproductions and verification results. Do not
-transfer the previous cohort's measured results to this corrected archive.
-[Current confirmation](assets/freshness-confirmation.md) records completed cells;
-scikit-learn solved officially but exceeded baseline cost, and Django and Requests
-failed official grading, contradicting acceptance for this cohort. Finish the registered
-attempts and diagnose that failure; do not replace or pool cells. The choices ledger has
-been consolidated against current code; final reconciliation and archive remain.
-
-Preserve healthy retrieval semantics and the canonical skill while fixing
-filesystem freshness. Keep the measured archive immutable; it matches the
-restored bundled-CPython candidate, with only the benchmark timing protocol changed. The failed query and presentation experiments
-have not earned promotion. Their remaining cells stay unrun. Prior outcomes
-cannot be pooled into the completed confirmation.
+The corrected package's `installed-jg-freshness-v1` cohort is complete and
+**rejected**: six of eight baseline solves preserved, four successful cost wins,
+and 27.30% lower full Sol cost. [Current confirmation](assets/freshness-confirmation.md)
+owns the complete aggregate and all ten trace comparisons. No coding, grading or
+accounting process remains active; do not restart terminal sessions or attempts.
+
+The freshness correction is integrated in `92ca7f9`. Full merged verification,
+healthy reference parity and exact archive checks on all supported platforms
+passed; [freshness evidence](assets/source-freshness.md) owns that proof and its
+limits. Django and Requests still failed official behavior cases despite the
+agent receiving the relevant code. This does not prove the correction or model
+variance alone caused the losses.
+
+Next: audit the final retained evidence and resolve the product-quality decision
+with the user before changing the exact spike strategy or acceptance contract.
+Do not rerun an unchanged cohort to seek a passing draw, replace attempts, pool
+prior outcomes, or archive the spec as accepted. The choices ledger has been
+consolidated against current code; final reconciliation and close-spec remain.
+
+The preceding [accepted confirmation](assets/work-clock-confirmation.md) remains
+valid for its own frozen archive. It cannot replace this corrected candidate's
+failed result. Preserve both archives, the canonical skill and healthy retrieval
+semantics. Rejected query/presentation experiments have not earned promotion.
Evidence and boundaries:
@@ -47,13 +45,14 @@ Evidence and boundaries:
[presentation trial](assets/presentation-study.md): rejected experiments,
source-parity probes and limits on causal explanations.
-Priority: validate the integrated freshness correction without weakening
-acceptance; resolve the whole-product review; consolidate choices against the
-final code; then close-spec. Slice 08 remains open. No publication or
+Priority: resolve the failed quality confirmation without silently weakening
+acceptance or changing the measured strategy; then perform final reconciliation
+and close-spec. Slice 08 remains open. No publication or
release tag is authorized for the intentional `0.0.0` development checkpoint.
Report observed Jev costs separately; they never enter scored Sol task cost.
-No user decision blocks continuing. Use the authorized credentials and retained
+Further strategy experiments or acceptance changes require an explicit direction
+after reviewing the completed evidence. Preserve authorized credentials and retained
Docker snapshots; do not repair credentials or regenerate baselines. Preserve
user-edited skill files and local historical evidence. Stage only explicit files,
keeping deprecated personal-repository evals out of Git.
diff --git a/specs/jevgrep/assets/freshness-aggregate.json b/specs/jevgrep/assets/freshness-aggregate.json
new file mode 100644
index 0000000..f86c13a
--- /dev/null
+++ b/specs/jevgrep/assets/freshness-aggregate.json
@@ -0,0 +1,328 @@
+{
+ "prospective_full_cohort": true,
+ "complete": true,
+ "baseline_solves_preserved": 6,
+ "fully_billed_solved_cost_wins": 4,
+ "accepted": false,
+ "known_gateway_subtotal_usd": 5.5410776,
+ "fully_billed_total_usd": 5.5410776,
+ "jev": {
+ "included_in_scored_task_cost": false,
+ "basis": "Observed response API metadata, not invoice reconciliation",
+ "complete": false,
+ "known_cost_usd": 1.035782622,
+ "observed_cost_usd": null
+ },
+ "cells": [
+ {
+ "task": "scikit-learn__scikit-learn-13124",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 0.29443600000000003,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.343483,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 206,
+ "responses_with_cost": 203,
+ "complete": false,
+ "observed_cost_usd": null,
+ "known_cost_usd": 0.046611348,
+ "input_tokens": null,
+ "output_tokens": null,
+ "provider_attempts": null,
+ "known_input_tokens": 1109794,
+ "known_output_tokens": 20564,
+ "known_provider_attempts": 227
+ },
+ "known_gateway_cost_usd": 0.343483,
+ "successful_cost_win": false
+ },
+ {
+ "task": "django__django-15629",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": false,
+ "resolved": false,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 1.5055472,
+ "fully_billed": true,
+ "gateway_cost_usd": 1.069589,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 870,
+ "responses_with_cost": 848,
+ "complete": false,
+ "observed_cost_usd": null,
+ "known_cost_usd": 0.158282796,
+ "input_tokens": null,
+ "output_tokens": null,
+ "provider_attempts": null,
+ "known_input_tokens": 3768638,
+ "known_output_tokens": 93007,
+ "known_provider_attempts": 899
+ },
+ "known_gateway_cost_usd": 1.069589,
+ "successful_cost_win": false
+ },
+ {
+ "task": "astropy__astropy-13579",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 0.428631,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.3914126,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 341,
+ "responses_with_cost": 341,
+ "complete": false,
+ "observed_cost_usd": null,
+ "known_cost_usd": 0.07780647,
+ "input_tokens": null,
+ "output_tokens": null,
+ "provider_attempts": null,
+ "known_input_tokens": 1852535,
+ "known_output_tokens": 29474,
+ "known_provider_attempts": 384
+ },
+ "known_gateway_cost_usd": 0.3914126,
+ "successful_cost_win": true
+ },
+ {
+ "task": "pydata__xarray-3305",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 0.4937066,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.3182098,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 517,
+ "responses_with_cost": 517,
+ "complete": true,
+ "observed_cost_usd": 0.123411498,
+ "known_cost_usd": 0.123411498,
+ "input_tokens": 2938369,
+ "output_tokens": 53901,
+ "provider_attempts": 644,
+ "known_input_tokens": 2938369,
+ "known_output_tokens": 53901,
+ "known_provider_attempts": 644
+ },
+ "known_gateway_cost_usd": 0.3182098,
+ "successful_cost_win": true
+ },
+ {
+ "task": "psf__requests-1142",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": false,
+ "resolved": false,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 0.2685004,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.23880800000000002,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 72,
+ "responses_with_cost": 72,
+ "complete": true,
+ "observed_cost_usd": 0.011300058,
+ "known_cost_usd": 0.011300058,
+ "input_tokens": 269049,
+ "output_tokens": 8911,
+ "provider_attempts": 83,
+ "known_input_tokens": 269049,
+ "known_output_tokens": 8911,
+ "known_provider_attempts": 83
+ },
+ "known_gateway_cost_usd": 0.23880800000000002,
+ "successful_cost_win": false
+ },
+ {
+ "task": "sympy__sympy-16792",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 0.547975,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.3320948,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 413,
+ "responses_with_cost": 413,
+ "complete": false,
+ "observed_cost_usd": null,
+ "known_cost_usd": 0.098661402,
+ "input_tokens": null,
+ "output_tokens": null,
+ "provider_attempts": null,
+ "known_input_tokens": 2349081,
+ "known_output_tokens": 40385,
+ "known_provider_attempts": 490
+ },
+ "known_gateway_cost_usd": 0.3320948,
+ "successful_cost_win": true
+ },
+ {
+ "task": "pytest-dev__pytest-6197",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 2.344458,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.8116844,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 578,
+ "responses_with_cost": 578,
+ "complete": true,
+ "observed_cost_usd": 0.098483154,
+ "known_cost_usd": 0.098483154,
+ "input_tokens": 2344837,
+ "output_tokens": 65717,
+ "provider_attempts": 632,
+ "known_input_tokens": 2344837,
+ "known_output_tokens": 65717,
+ "known_provider_attempts": 632
+ },
+ "known_gateway_cost_usd": 0.8116844,
+ "successful_cost_win": true
+ },
+ {
+ "task": "sphinx-doc__sphinx-8638",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 1.0594724,
+ "fully_billed": true,
+ "gateway_cost_usd": 1.2444744,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 1334,
+ "responses_with_cost": 1334,
+ "complete": false,
+ "observed_cost_usd": null,
+ "known_cost_usd": 0.196647108,
+ "input_tokens": null,
+ "output_tokens": null,
+ "provider_attempts": null,
+ "known_input_tokens": 4682074,
+ "known_output_tokens": 156382,
+ "known_provider_attempts": 1510
+ },
+ "known_gateway_cost_usd": 1.2444744,
+ "successful_cost_win": false
+ },
+ {
+ "task": "matplotlib__matplotlib-26466",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": false,
+ "resolved": false,
+ "baseline_resolved": false,
+ "baseline_cost_usd": 0.395716,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.4256784,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 700,
+ "responses_with_cost": 700,
+ "complete": false,
+ "observed_cost_usd": null,
+ "known_cost_usd": 0.157293066,
+ "input_tokens": null,
+ "output_tokens": null,
+ "provider_attempts": null,
+ "known_input_tokens": 3745073,
+ "known_output_tokens": 69302,
+ "known_provider_attempts": 764
+ },
+ "known_gateway_cost_usd": 0.4256784,
+ "successful_cost_win": false
+ },
+ {
+ "task": "pylint-dev__pylint-4604",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": false,
+ "resolved": false,
+ "baseline_resolved": false,
+ "baseline_cost_usd": 0.2836264,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.3656432,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 349,
+ "responses_with_cost": 349,
+ "complete": false,
+ "observed_cost_usd": null,
+ "known_cost_usd": 0.067285722,
+ "input_tokens": null,
+ "output_tokens": null,
+ "provider_attempts": null,
+ "known_input_tokens": 1602041,
+ "known_output_tokens": 43900,
+ "known_provider_attempts": 376
+ },
+ "known_gateway_cost_usd": 0.3656432,
+ "successful_cost_win": false
+ }
+ ]
+}
diff --git a/specs/jevgrep/assets/freshness-confirmation.md b/specs/jevgrep/assets/freshness-confirmation.md
index 57784d3..037264d 100644
--- a/specs/jevgrep/assets/freshness-confirmation.md
+++ b/specs/jevgrep/assets/freshness-confirmation.md
@@ -1,7 +1,13 @@
# Corrected-package confirmation
-Status: running; acceptance contradicted by lost Django and Requests baseline solves.
-Finish and retain the registered cohort rather than replacing that attempt.
+Status: complete; rejected by the registered acceptance gate. Six of ten tasks
+solved, six of eight baseline solves were preserved, and four solved tasks were
+strict cost wins. All ten Sol bills are complete: **$5.5410776** versus baseline
+**$7.6220690**, a **27.30% reduction including failed tasks**. Lower total cost
+does not compensate for the lost solves. Jev's observed subtotal is
+**$1.035782622**; its full total is unknown and excluded from scored cost.
+The [complete aggregate](freshness-aggregate.json) is retained byte-for-byte from
+the study, SHA-256 `37f2dc0883d0e366b75e74b0f178498a393da9f8ef9cd573d5881630efc0d65a`.
This is a separate prospective ten-task cohort for the
[source-freshness correction](source-freshness.md), not a replacement of individual
cells from the preceding accepted study. The frozen plan, archived runner and
@@ -23,6 +29,7 @@ separately and excluded; incomplete observations are subtotals, not totals.
| pytest-dev__pytest-6197 | solved; cost win | $0.8116844 | $2.3444580 | $0.098483154 |
| sphinx-doc__sphinx-8638 | solved; not a cost win | $1.2444744 | $1.0594724 | known $0.196647108; total unknown |
| matplotlib__matplotlib-26466 | unresolved | $0.4256784 | $0.3957160 | known $0.157293066; total unknown |
+| pylint-dev__pylint-4604 | unresolved at collection | $0.3656432 | $0.2836264 | known $0.067285722; total unknown |
Scikit-learn returned five relevant files in 5,038 bytes, implementation reading
leads and two test source blocks. The agent then read the splitter and tests, searched documentation,
@@ -37,8 +44,9 @@ and test work does not isolate a causal cost effect from the freshness correctio
Root execution log: `/tmp/jg-freshness-v1-run.log`. Per-task grade and accounting
receipts, complete event streams, patches and request/response evidence remain
-with the attempts. Acceptance stays unproven until all ten outcomes are retained
-and the full aggregate is evaluated.
+with the attempts. All coding, grading and accounting processes are terminal;
+no attempt was replaced and no baseline was rerun. The product quality gate
+remains open because the completed aggregate rejects the candidate.
Django's packet contained 51 files in 8,071 bytes without excerpts. SQLite was
listed and its critical type-only rebuild guard was read, untruncated, before the
@@ -122,3 +130,23 @@ After correcting an initial NumPy slice attempt, local text tests passed 104 wit
12 skips and offsetbox tests passed 285 with two skips. Official evaluation still
failed the combined Annotation/OffsetFrom input-copy regression, as in baseline.
Evidence: `/tmp/jg-freshness-v1-matplotlib-comparison/findings.md`.
+
+Pylint returned 32 files in 6,990 bytes with two fixture blocks and a reading lead
+to the relevant checker. The agent read that checker, added qualified/unqualified
+ABC coverage and passed one focused plus 20 checker tests. Its implementation
+differs structurally from earlier variants but handles the inspected paths
+equivalently. Official evaluation failed during collection on missing `IS_PYPY`,
+before patch assertions ran. That retained environmental limitation remains a
+formal nonpass, not a proven patch defect. Evidence:
+`/tmp/jg-freshness-v1-pylint-comparison/findings.md`.
+
+## Interpretation boundary
+
+The two lost baseline solves are concrete patch/test omissions despite delivered
+code: Django's same-type rebuild condition and Requests' HEAD behavior. This
+does not isolate model variance or the freshness correction as the sole cause.
+Healthy controlled parity passed, but it is not a substitute for this failed
+live confirmation. The preceding accepted cohort remains evidence about its
+own frozen archive, not a result that can replace this one. No further unchanged
+cohort is justified merely to obtain a passing draw; changes to retrieval, skill
+or acceptance require a separate explicit decision and identified experiment.
diff --git a/specs/jevgrep/assets/source-freshness.md b/specs/jevgrep/assets/source-freshness.md
index 21ba5a5..d314ba6 100644
--- a/specs/jevgrep/assets/source-freshness.md
+++ b/specs/jevgrep/assets/source-freshness.md
@@ -47,8 +47,9 @@ tests, 24 benchmark-harness tests and 24 installed tests, plus typecheck and lin
Evidence: `/tmp/jg-freshness-merged-verify-serial.log`; no test allowance was changed.
The accepted work-clock cohort measured the preceding archive. A separate frozen
-ten-task treatment cohort is running at
+ten-task treatment cohort completed at
`evals/runs/swebench/installed-jg-freshness-v1/`. Preparation and all ten no-call
validations passed before execution. Its archived runner owns the frozen inputs;
saved baselines remain immutable and earlier treatment cells cannot be pooled
-into its result. Run log: `/tmp/jg-freshness-v1-run.log`.
+into its result. The [completed confirmation](freshness-confirmation.md) failed
+the quality gate despite passing deterministic verification. Run log: `/tmp/jg-freshness-v1-run.log`.
diff --git a/specs/jevgrep/slices/08-release.md b/specs/jevgrep/slices/08-release.md
index ca2f6bb..1f9bff2 100644
--- a/specs/jevgrep/slices/08-release.md
+++ b/specs/jevgrep/slices/08-release.md
@@ -1,6 +1,6 @@
# Confirm quality and the release artifact
-Status: open. Frozen work-clock v2 confirmation passed; final review found filesystem-freshness defects requiring correction and validation. Depends on: 07. Read [contracts](../contracts.md) first.
+Status: open. Freshness fixes and deterministic verification passed, but the corrected package's complete cohort lost Django and Requests baseline solves. See [final confirmation](../assets/freshness-confirmation.md). Depends on: 07. Read [contracts](../contracts.md) first.
## Contract and owner