Skip to content

Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.

docs: close the independent variance repeat with audited results

David Zhang88e61cd523a2
diff --git a/README.md b/README.md
index e5b6230..34aae9a 100644
--- a/README.md
+++ b/README.md
@@ -6,7 +6,9 @@ The production Node-only CLI is implemented and its installed package is verifie
on macOS and Linux. The [implementation record](specs/done/jevgrep/README.md)
owns the rationale, invariants and evidence limits. The user accepted the documented
quality tradeoff after the corrected candidate failed its original benchmark gate;
-one separately authorized identical-cohort variance check remains pending.
+the separately authorized identical-runtime repeat also failed that gate. It solved
+7/10 tasks versus the saved baseline’s 8/10 while reducing total Sol cost by 40.70%.
+Both complete cohorts remain separately reported in the implementation record.
## Development
@@ -44,5 +46,5 @@ The npm package is `@dzhng/jevgrep`; its executable is `jg`. The project uses th
[MIT license](LICENSE). [Release guidance](scripts/RELEASING.md) describes the
verified tarball and GitHub tag workflow. Publication requires the repository's
`NPM_TOKEN` secret and a deliberate release tag. The user authorized `0.1.0`
-publication after the pending repeat and closeout; the frozen benchmark archive
+publication after the completed repeat and final release checks; the frozen benchmark archive
remains `0.0.0` and is not a publication candidate.
diff --git a/specs/done/jevgrep/README.md b/specs/done/jevgrep/README.md
index e942508..85ee94f 100644
--- a/specs/done/jevgrep/README.md
+++ b/specs/done/jevgrep/README.md
@@ -18,14 +18,15 @@ trace comparisons and immutable aggregate; its `accepted: false` verdict remains
The earlier [passing confirmation](assets/work-clock-confirmation.md) belongs to
its own frozen package and cannot replace the corrected candidate's result.
-The user separately authorized one [identical cohort repeat](assets/variance-repeat.md)
-to double-check variance. **That measurement is pending at this documentation checkpoint.** It
-is not permission to replace failures, pool outcomes, change the strategy or keep
-rerunning until the gate passes. Its terminal evidence and the final independent
-claim audit remain closeout work for the coordinating agent. The user authorized publication of `0.1.0` once the repeat and closeout finish.
-The [0.1.0 release candidate](assets/release-0.1.0.md) passed local validation and
-installed checks; CI publication and registry verification remain pending. The repeated benchmark retains the frozen `0.0.0` archive;
-release preparation does not replace its measured artifact.
+The separately authorized [identical-runtime repeat](assets/variance-repeat.md)
+finished with seven solves, seven of eight baseline solves preserved, four
+successful cost wins and 40.70% lower full Sol cost. Its original gate also remains
+failed. Both cohorts retain every attempt; neither best-cell pooling nor baseline
+reruns contribute to either result. No further repeat was authorized or run.
+The user authorized publication of `0.1.0` after closeout. The
+[release evidence](assets/release-0.1.0.md) distinguishes local validation from
+CI publication and registry verification. Both measured cohorts retain the frozen
+`0.0.0` archive; release preparation changes version identity, not their evidence.
## Why this shape
diff --git a/specs/done/jevgrep/assets/variance-repeat-aggregate.json b/specs/done/jevgrep/assets/variance-repeat-aggregate.json
new file mode 100644
index 0000000..8c6e4ce
--- /dev/null
+++ b/specs/done/jevgrep/assets/variance-repeat-aggregate.json
@@ -0,0 +1,328 @@
+{
+ "prospective_full_cohort": true,
+ "complete": true,
+ "baseline_solves_preserved": 7,
+ "fully_billed_solved_cost_wins": 4,
+ "accepted": false,
+ "known_gateway_subtotal_usd": 4.5195532,
+ "fully_billed_total_usd": 4.5195532,
+ "jev": {
+ "included_in_scored_task_cost": false,
+ "basis": "Observed response API metadata, not invoice reconciliation",
+ "complete": false,
+ "known_cost_usd": 1.021334412,
+ "observed_cost_usd": null
+ },
+ "cells": [
+ {
+ "task": "scikit-learn__scikit-learn-13124",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 0.29443600000000003,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.3745434,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 209,
+ "responses_with_cost": 209,
+ "complete": true,
+ "observed_cost_usd": 0.054481224,
+ "known_cost_usd": 0.054481224,
+ "input_tokens": 1297172,
+ "output_tokens": 19281,
+ "provider_attempts": 240,
+ "known_input_tokens": 1297172,
+ "known_output_tokens": 19281,
+ "known_provider_attempts": 240
+ },
+ "known_gateway_cost_usd": 0.3745434,
+ "successful_cost_win": false
+ },
+ {
+ "task": "django__django-15629",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": false,
+ "resolved": false,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 1.5055472,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.9526992,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 791,
+ "responses_with_cost": 791,
+ "complete": false,
+ "observed_cost_usd": null,
+ "known_cost_usd": 0.151341876,
+ "input_tokens": null,
+ "output_tokens": null,
+ "provider_attempts": null,
+ "known_input_tokens": 3603378,
+ "known_output_tokens": 89126,
+ "known_provider_attempts": 833
+ },
+ "known_gateway_cost_usd": 0.9526992,
+ "successful_cost_win": false
+ },
+ {
+ "task": "astropy__astropy-13579",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 0.428631,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.3507346,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 337,
+ "responses_with_cost": 337,
+ "complete": true,
+ "observed_cost_usd": 0.076535382,
+ "known_cost_usd": 0.076535382,
+ "input_tokens": 1822271,
+ "output_tokens": 29779,
+ "provider_attempts": 359,
+ "known_input_tokens": 1822271,
+ "known_output_tokens": 29779,
+ "known_provider_attempts": 359
+ },
+ "known_gateway_cost_usd": 0.3507346,
+ "successful_cost_win": true
+ },
+ {
+ "task": "pydata__xarray-3305",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 0.4937066,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.511454,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 889,
+ "responses_with_cost": 889,
+ "complete": false,
+ "observed_cost_usd": null,
+ "known_cost_usd": 0.181858194,
+ "input_tokens": null,
+ "output_tokens": null,
+ "provider_attempts": null,
+ "known_input_tokens": 4329957,
+ "known_output_tokens": 104339,
+ "known_provider_attempts": 944
+ },
+ "known_gateway_cost_usd": 0.511454,
+ "successful_cost_win": false
+ },
+ {
+ "task": "psf__requests-1142",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 0.2685004,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.2721328,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 63,
+ "responses_with_cost": 63,
+ "complete": true,
+ "observed_cost_usd": 0.010286388,
+ "known_cost_usd": 0.010286388,
+ "input_tokens": 244914,
+ "output_tokens": 7911,
+ "provider_attempts": 65,
+ "known_input_tokens": 244914,
+ "known_output_tokens": 7911,
+ "known_provider_attempts": 65
+ },
+ "known_gateway_cost_usd": 0.2721328,
+ "successful_cost_win": false
+ },
+ {
+ "task": "sympy__sympy-16792",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 0.547975,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.5426052,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 466,
+ "responses_with_cost": 466,
+ "complete": true,
+ "observed_cost_usd": 0.112039242,
+ "known_cost_usd": 0.112039242,
+ "input_tokens": 2667601,
+ "output_tokens": 45410,
+ "provider_attempts": 494,
+ "known_input_tokens": 2667601,
+ "known_output_tokens": 45410,
+ "known_provider_attempts": 494
+ },
+ "known_gateway_cost_usd": 0.5426052,
+ "successful_cost_win": true
+ },
+ {
+ "task": "pytest-dev__pytest-6197",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 2.344458,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.6108848,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 365,
+ "responses_with_cost": 365,
+ "complete": true,
+ "observed_cost_usd": 0.059945634,
+ "known_cost_usd": 0.059945634,
+ "input_tokens": 1427277,
+ "output_tokens": 38980,
+ "provider_attempts": 427,
+ "known_input_tokens": 1427277,
+ "known_output_tokens": 38980,
+ "known_provider_attempts": 427
+ },
+ "known_gateway_cost_usd": 0.6108848,
+ "successful_cost_win": true
+ },
+ {
+ "task": "sphinx-doc__sphinx-8638",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": true,
+ "resolved": true,
+ "baseline_resolved": true,
+ "baseline_cost_usd": 1.0594724,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.40014679999999997,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 742,
+ "responses_with_cost": 742,
+ "complete": true,
+ "observed_cost_usd": 0.11034114,
+ "known_cost_usd": 0.11034114,
+ "input_tokens": 2627170,
+ "output_tokens": 87714,
+ "provider_attempts": 778,
+ "known_input_tokens": 2627170,
+ "known_output_tokens": 87714,
+ "known_provider_attempts": 778
+ },
+ "known_gateway_cost_usd": 0.40014679999999997,
+ "successful_cost_win": true
+ },
+ {
+ "task": "matplotlib__matplotlib-26466",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": false,
+ "resolved": false,
+ "baseline_resolved": false,
+ "baseline_cost_usd": 0.395716,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.2444352,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 1041,
+ "responses_with_cost": 1040,
+ "complete": false,
+ "observed_cost_usd": null,
+ "known_cost_usd": 0.208445538,
+ "input_tokens": null,
+ "output_tokens": null,
+ "provider_attempts": null,
+ "known_input_tokens": 4962989,
+ "known_output_tokens": 111817,
+ "known_provider_attempts": 1103
+ },
+ "known_gateway_cost_usd": 0.2444352,
+ "successful_cost_win": false
+ },
+ {
+ "task": "pylint-dev__pylint-4604",
+ "status": "completed",
+ "terminal": true,
+ "grading_attempted": true,
+ "graded": true,
+ "protocol_valid": true,
+ "official_resolved": false,
+ "resolved": false,
+ "baseline_resolved": false,
+ "baseline_cost_usd": 0.2836264,
+ "fully_billed": true,
+ "gateway_cost_usd": 0.2599172,
+ "jev": {
+ "basis": "Retained response providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation",
+ "included_in_scored_task_cost": false,
+ "client_calls": 270,
+ "responses_with_cost": 270,
+ "complete": true,
+ "observed_cost_usd": 0.056059794,
+ "known_cost_usd": 0.056059794,
+ "input_tokens": 1334757,
+ "output_tokens": 36266,
+ "provider_attempts": 296,
+ "known_input_tokens": 1334757,
+ "known_output_tokens": 36266,
+ "known_provider_attempts": 296
+ },
+ "known_gateway_cost_usd": 0.2599172,
+ "successful_cost_win": false
+ }
+ ]
+}
diff --git a/specs/done/jevgrep/assets/variance-repeat.md b/specs/done/jevgrep/assets/variance-repeat.md
index 839e21b..9db9596 100644
--- a/specs/done/jevgrep/assets/variance-repeat.md
+++ b/specs/done/jevgrep/assets/variance-repeat.md
@@ -1,6 +1,6 @@
# Identical-runtime variance repeat
-Status: running. After accepting the first corrected cohort's documented
+Status: complete; all ten tasks are officially graded and fully billed for Sol. After accepting the first corrected cohort's documented
tradeoff, the user authorized exactly one additional full cohort to examine
variation. The original outcomes remain intact and no baseline is rerun.
@@ -21,10 +21,34 @@ uses the unchanged archived runner. Its `preparation-method.json` and
`/tmp/jg-freshness-repeat-v1b-freeze-proof.json` retain the method and file hashes.
No product or evaluation policy changed to repair the setup.
-Live execution and sequential grading/accounting belong to sessions `67618` and
-`27936`; logs use `/tmp/jg-freshness-repeat-v1b-{run,observer}.log`. Revalidate
-their handles before any resume. Final results must be shown alongside the first
-cohort, without pooling selected cells or replacing its failed verdict.
+Both execution and sequential grading/accounting exited successfully. Logs use
+`/tmp/jg-freshness-repeat-v1b-{run,observer}.log`. The immutable
+[repeat aggregate](variance-repeat-aggregate.json) retains all ten results.
+
+| Measurement | Saved baseline | First corrected cohort | Identical-runtime repeat |
+| --- | ---: | ---: | ---: |
+| Official solves | 8/10 | 6/10 | 7/10 |
+| Baseline solves preserved | — | 6/8 | 7/8 |
+| Solved tasks cheaper than baseline | — | 4 | 4 |
+| Full Sol cost, including failures | $7.6220690 | $5.5410776 | $4.5195532 |
+| Sol savings | — | 27.30% | 40.70% |
+| Original gate accepted | — | false | false |
+
+The repeat's known Jev subtotal is $1.021334412; its full total is unknown and
+excluded from scored Sol cost. These are provider metadata observations, not
+invoice reconciliation. No baseline was rerun and no outcomes were pooled.
+Requests recovered its solve; Django still lost a baseline solve. This single
+repeat demonstrates variation, not a reliability estimate or causal explanation.
+The user's separate acceptance of the documented tradeoff remains explicit.
+
+An independent read-only audit reconciled all ten receipts, grading reports,
+158 distinct Sol generation charges, frozen runtime artifacts and saved baseline
+hashes: 273 checks with no discrepancies. The two cohorts' generation IDs are
+disjoint. Aggregate SHA-256:
+`2c0a7c78b000e14197fa12364b70d2d258f31eb1a85d416599e0ce2945a6ac90`.
+Audit detail: `/tmp/jg-final-repeat-audit.md`. A separate claim audit checked this
+record, the closure docs and all ten paired trace comparisons without findings.
+
## Completed observations
@@ -112,3 +136,26 @@ passed immediately in one 35-pass domain-suite run, versus three test-authoring
repairs and broader verification in the first run. Omitted broader coverage
does not show that the known environment warning was fixed. Evidence:
`/tmp/jg-repeat-sphinx-comparison.md`.
+
+Matplotlib remains unresolved at $0.2444352 versus first-run $0.4256784 and
+baseline $0.3957160. Its packet grew from 3,846 bytes without excerpts to 9,865
+bytes with eight blocks. It delivered the Annotation constructor and an OffsetFrom
+header, but not the missing OffsetFrom constructor logic. The agent fixed only
+Annotation, using the same tuple conversion as the saved baseline. Local tests
+passed 389 with 14 skips; they did not cover the missing OffsetFrom behavior.
+The official combined annotation/OffsetFrom test still failed. Jev's known subtotal
+is $0.208445538, full total unknown. Evidence:
+`/tmp/jg-repeat-matplotlib-comparison.md`.
+
+Pylint remains officially unresolved at $0.2599172 versus first-run $0.3656432
+and baseline $0.2836264. Official test collection failed with a missing `IS_PYPY`
+import before any tests executed. This is not proof of a patch assertion failure.
+Its complete observed Jev total is $0.056059794, excluded from scored Sol cost.
+
+The repeat's production patch matches the saved baseline and accepted spike;
+the first corrected patch used an equivalent shared traversal for the inspected
+path. Its packet contained 34 files without excerpts versus 32 files with two
+fixture excerpts. Local checks passed one, twenty and six tests; the repeat
+used fewer pre-patch reads and no standalone AST probe. Neither those differences
+nor the lower cost establish a causal effect of retrieval. Evidence:
+`/tmp/jg-repeat-pylint-comparison.md`.
diff --git a/specs/done/jevgrep/contracts.md b/specs/done/jevgrep/contracts.md
index 4ab8322..684f4f4 100644
--- a/specs/done/jevgrep/contracts.md
+++ b/specs/done/jevgrep/contracts.md
@@ -229,4 +229,4 @@ not prerequisites hidden inside this release. Keep all traces and official grade
The corrected candidate did not pass that numerical gate. The user explicitly
accepted implementation closure with the documented tradeoff; this does not
change the retained aggregate verdict. [Closure and evidence bounds](README.md)
-own that decision and the separately authorized pending variance repeat.
+own that decision and the separately authorized completed variance repeat.