Skip to content

Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.

Document paired checkpoint evidence and reconcile current choices

David Zhanga79c172b709e
diff --git a/specs/jevgrep/README.md b/specs/jevgrep/README.md
index 366010f..bde53be 100644
--- a/specs/jevgrep/README.md
+++ b/specs/jevgrep/README.md
@@ -7,10 +7,11 @@ Jev classification through Vercel AI Gateway. This is the build plan; production
## Next Agent Prompt
You are implementing this spec. Status: **implementing**, updated **2026-09-26**.
-Next: finish the frozen `installed-jg-cpython-parity-v1` cohort. Requests is the
-first completed checkpoint: officially solved, fully billed, and 14.0% below
-baseline Sol cost. Its paired trace review is underway; the same plan's remaining
-cells are running. Never rerun an existing baseline or replace a retained
+Next: finish the frozen `installed-jg-cpython-parity-v1` cohort. Requests and
+scikit-learn are officially solved and fully billed cost wins.
+Requests paired trace review is complete; the same plan's remaining cells are
+running. [Cohort confirmation](assets/cpython-confirmation.md) owns the results.
+Never rerun an existing baseline or replace a retained
attempt. The merged `bun run verify` gate passed and whole-product review's sole
finding is fixed with a clean follow-up review. Exact archive checks pass on
macOS arm64 and Linux arm64/amd64; all ten no-call validations passed.
@@ -19,7 +20,8 @@ and interpreter-version limits. Quality acceptance remains unproven.
`installed-jg-reference-parity-v1` is stopped and superseded. Requests and
scikit-learn officially solved but exceeded baseline cost; Django was interrupted
-with its patch and traces retained; seven cells never started. These superseded studies remain stopped; only the new bundled-CPython study is
+with its patch and traces retained; seven cells never started. These superseded
+studies remain stopped; only the new bundled-CPython study is
active.
The changed-policy cohort `installed-jg-final-cohort-v1` was stopped and marked
@@ -52,7 +54,7 @@ not claims about the current scaffold.
- [x] [01 — Reference and real HTTP fixture](slices/01-reference.md)
- [x] [02 — Bundled parser parity](slices/02-parser.md)
- [x] [03 — Filesystem eligibility and snapshots](slices/03-filesystem.md)
-- [ ] [04 — Installed CLI + skill + first real task](slices/04-checkpoint.md)
+- [x] [04 — Installed CLI + skill + first real task](slices/04-checkpoint.md)
- [x] [05 — Faults, cancellation and partial results](slices/05-failures.md)
- [x] [06 — Default-on fresh cache](slices/06-cache.md)
- [x] [07 — Preserved output policy](slices/07-output-policy.md)
diff --git a/specs/jevgrep/assets/cpython-confirmation.md b/specs/jevgrep/assets/cpython-confirmation.md
new file mode 100644
index 0000000..7285b6f
--- /dev/null
+++ b/specs/jevgrep/assets/cpython-confirmation.md
@@ -0,0 +1,53 @@
+# Frozen bundled-CPython confirmation
+
+Study: `evals/runs/swebench/installed-jg-cpython-parity-v1/plan.json`.
+The [runtime evidence](python-runtime.md) owns the exact archive and preflight
+checks. This is one prospective ten-task cohort; superseded studies do not
+contribute results, and saved baselines are never rerun. Acceptance remains open.
+
+## Verified completed pairs
+
+| Task | Official result | Full Sol cost | Saved baseline | Cost change | Observed Jev cost, excluded |
+| --- | --- | ---: | ---: | ---: | ---: |
+| Requests 1142 | Solved, no evaluator errors | $0.2307800 | $0.2685004 | −14.05% | $0.011333322 |
+| scikit-learn 13124 | Solved, no evaluator errors | $0.2161638 | $0.2944360 | −26.58% | $0.061259100 |
+
+Both Sol bills are complete, with nine generations each. Jev response-cost
+coverage is complete: Requests has 72 client calls and 81 provider attempts;
+scikit-learn has 215 calls and 233 provider attempts. These are observed API
+metadata, not invoice reconciliation. Unknown or failed remaining outcomes cannot
+be counted as wins. Original grading and accounting receipts live beside each
+attempt in the study; remaining cells continue under the same frozen policy.
+
+## Requests paired trace finding
+
+Exact CLI query:
+
+```sh
+jg "requests.get always adds Content-Length header; expected GET requests with no body to omit automatically generated Content-Length, while preserving behavior for requests with bodies. Find request preparation/header calculation code and regression tests or test helpers."
+```
+
+The complete packet is 6,880 bytes and six files. Its six `models.py` source blocks
+are byte-identical to the accepted spike, totaling 4,761 unnumbered source bytes.
+It adds `api.py` as a file location and omits the spike's 213-byte `httpbin` helper
+excerpt. Neither packet includes an actual regression-test method.
+
+Sol reads the preparation call sites and test conventions, makes one successful
+patch, runs its new offline unittest, then expands verification to four tests.
+The implementation diff exactly matches the accepted spike's. Compared with the
+baseline, it avoids broad implementation reads and two separate behavior probes.
+Its Git history/blame lookup returns only synthetic initialization and adds little
+information. Unlike the spike, its tests do not explicitly exercise authenticated
+GET; the official task still resolves, and auth recomputation appears in the
+returned source. This boundary must not be rewritten as universal patch quality.
+
+Savings are predominantly lower billed input, not lower generated/reasoning usage
+against every comparator. Different queries and model trajectories prevent
+attributing the result specifically to CPython or any individual excerpt.
+Full raw-output comparison and exact source-byte offsets are retained locally in
+`/tmp/jg-cpython-requests-comparison/`. Authoritative treatment evidence is the
+study's Requests attempt; comparators are the retained baseline at
+`lookahead-native-v88/psf__requests-1142/codex-baseline` and accepted spike at
+`auto-research-80/unit-locators-confirmation/psf__requests-1142`, relative to
+`evals/runs/swebench/`. Use raw rollout files for complete tool output; event
+summaries can retain only the final output segment.
diff --git a/specs/jevgrep/assets/python-runtime.md b/specs/jevgrep/assets/python-runtime.md
index cb4f634..f5b3085 100644
--- a/specs/jevgrep/assets/python-runtime.md
+++ b/specs/jevgrep/assets/python-runtime.md
@@ -74,12 +74,9 @@ whole runtime as Apache-only. Packaging owns those notices and their verificatio
on the archive above and the existing Sol model/harness/baselines. Its exact
archive passed four offline installed checks on each of Linux amd64 and arm64,
and the native macOS smoke identified the same SHA-256. Release content validation
-and all ten dry-run validations passed before paid execution. The Requests cell
-started first and officially resolved with zero evaluator errors. Its nine Sol
-generations are fully accounted at $0.23078 versus baseline $0.2685004 (14.0%
-lower). Jev observed cost is separately $0.011333322 across 72 client calls
-and 81 provider attempts; it is excluded from scored task cost. This is one
-verified cost win, not cohort acceptance. Remaining cells continue unchanged. Superseded studies remain
+and all ten dry-run validations passed before paid execution. Current official
+results and paired trace findings belong in the
+[cohort confirmation](cpython-confirmation.md). Superseded studies remain
historical and contribute no outcomes to this cohort. Platform transcripts are
`/tmp/jg-cpython-frozen-amd64.log` and `/tmp/jg-cpython-frozen-arm64.log`;
preparation and dry-run transcripts use the `/tmp/jg-cpython-cohort-` prefix.
diff --git a/specs/jevgrep/choices.md b/specs/jevgrep/choices.md
index 071a316..0729fa0 100644
--- a/specs/jevgrep/choices.md
+++ b/specs/jevgrep/choices.md
@@ -1,18 +1,10 @@
# Implementation choices
-## Sound — high confidence
-
-### Keep deprecated tooling outside workspace discovery
-
-When: initial implementation checkpoint. The root workspace glob included the old
-personal-repository eval package. An ordinary workspace test or install could
-therefore discover that package even though its evals are deprecated. The active
-workspace now names core and TypeScript config explicitly, and old root eval
-commands are removed; historical local files remain available.
+Current decisions, ordered by confidence. Superseded experiments belong in the
+[restoration evidence](assets/parity-restoration.md). Final closeout must re-audit
+this ledger against the shipped state; benchmark acceptance is still open.
-Gap: the plan required a cutover but did not specify when workspace discovery
-changed. Reach: future packages must be added deliberately. Verdict: sound,
-because ordinary development should not invoke deprecated workflows.
+## Sound — medium confidence
### Pin request contents without pretending network arrival order is meaningful
@@ -25,8 +17,11 @@ request—would hide an input change that could change Jev's answer.
Gap: the spec did not define how to control historical concurrency in the fixture.
Reach: this fixture proves exact request construction, not determinism on every
-repository. Verdict: sound. Confidence: high. The ordering race remains documented
-for the production port to resolve deliberately.
+repository. Production deliberately preserves selection-completion order inside
+cross-file evidence: whichever file finishes first contributes its evidence first.
+A warm cache can change that order and therefore produce a genuinely new request.
+Verdict: sound. Confidence: medium; preserving the measured behavior takes priority
+over making requests deterministic.
### Stream directory entries in native order
@@ -35,10 +30,66 @@ fit in one page. Its open cursor retains the unread entries instead of loading t
whole directory just to alphabetize it. The caller must keep reading even when a
page contains only excluded names, and must close a cursor when pruning a branch.
-Gap: the plan required bounded pages but did not prescribe order. Reach: traversal
-must sort candidate output separately and may not treat a page boundary as the end
-of a directory. Verdict: sound. Confidence: medium; the quality effect of request
-ordering remains part of the installed benchmark confirmation.
+Gap: the plan required bounded pages but did not prescribe reader ordering.
+Reach: this streaming behavior belongs to the filesystem reader. Retrieval gathers
+the pages and alphabetizes entries before constructing requests, preserving the
+frozen strategy. It may not treat a page boundary as the end of a directory.
+Verdict: sound. Confidence: medium; this does not claim bounded total retrieval
+memory for an arbitrarily wide directory.
+
+### Preserve the winner while deferring throughput changes
+
+When: reference restoration. Earlier unverified source-budget changes are
+[superseded experiments](assets/parity-restoration.md), not active product choices. A minified file can contain several declarations on
+one line; the winning strategy may ask about that line repeatedly under different
+declaration names. Combining those questions could change Jev's answers, so this
+port keeps them. Separately, saving a cache answer scans existing entries to
+enforce the disk limit. A large cache therefore has repeated scanning overhead.
+
+Gap: review found performance costs that are outside the user's current solve-rate
+and task-cost acceptance priorities. Reach: the confirmation does not establish
+whole-computer throughput or optimal cache maintenance. Verdict: sound for this
+confirmation, with medium confidence: retain measured retrieval behavior and
+report the cache limitation; test any later optimization as a separately identified
+artifact rather than silently changing the candidate being confirmed.
+
+### Isolate bundled CPython in one Node child process
+
+When: Python parity restoration. If query A is parsing source when its caller
+cancels, terminate the interpreter process. If query B also has a pending helper
+request, replay B's pure parsing request in a replacement process rather than
+failing B along with A. The helpers have no repository side effects, so repeating
+that parsing does not repeat a user action. An idle interpreter does not keep the
+CLI alive and exits when its parent disconnects.
+
+Gap: the spec requires no system Python and cancelable inspection but leaves the
+interpreter isolation mechanism open. One Node child process owns the bundled
+interpreter in production and Bun-hosted development, avoiding a dependency patch
+or separate parser for the development runner.
+
+A helper failure caused by source preserves the reference fallback; a missing
+runtime or failed child process remains fatal. Child diagnostics cannot escape the
+CLI's output handling. Python counts carriage-return-only line endings as lines, while the reference
+caller splits excerpts only at newline characters. Keep those two coordinate
+rules distinct on such files; normalizing them would change reference behavior.
+
+Reach: the package gains runtime assets and a child process, but still requires
+only Node. Verdict: sound. Confidence: medium. Interpreter version boundaries and footprint
+measurements live in [runtime rationale](assets/python-runtime.md).
+
+## Sound — high confidence
+
+### Keep deprecated tooling outside workspace discovery
+
+When: initial implementation checkpoint. The root workspace glob included the old
+personal-repository eval package. An ordinary workspace test or install could
+therefore discover that package even though its evals are deprecated. The active
+workspace now names core and TypeScript config explicitly, and old root eval
+commands are removed; historical local files remain available.
+
+Gap: the plan required a cutover but did not specify when workspace discovery
+changed. Reach: future packages must be added deliberately. Verdict: sound,
+because ordinary development should not invoke deprecated workflows.
### Fix eligibility policy for one reader
@@ -72,16 +123,6 @@ Gap: the result schema did not distinguish cache warnings from evidence failures
Reach: callers can trust a complete search even if it ran without persistence.
Verdict: sound. Confidence: high.
-### Bound pending source batches without limiting admitted file counts
-
-When: traversal integration. Flush accumulated source fragments by bytes before
-many large files can pile up in memory. Keep candidate metadata for every file
-passing the threshold, and revisit snapshots when selecting source.
-
-Gap: the plan delegated batching details. Reach: request grouping differs from the
-reference under large inputs and therefore needs the production quality gate.
-Verdict: provisional until the frozen task comparison. Confidence: medium.
-
### Rebuild the CLI whenever build is requested
When: packaging integration. The bundled CLI includes core source and the canonical
@@ -109,7 +150,10 @@ Confidence: high. Actual publication still requires a later user-triggered tag.
When: release packaging. Use Bun's emitted-input metadata to collect installed
license notices. Retain a version-specific upstream license for the SDK package
whose npm archive omits it; an unknown missing license fails the build. The user
-selected MIT for Jevgrep itself.
+selected MIT for Jevgrep itself. The externally installed Pyodide distribution
+uses separately pinned component notices because emitted JavaScript metadata
+cannot discover licenses for its compiled runtime. An upgrade requires reviewing
+those notices; it does not inherit the old notice set automatically.
Gap: the spec required licenses but left collection mechanics open. Reach: a
new dependency can require a verified notice update, while ordinary builds need
@@ -131,7 +175,7 @@ the same evaluator cancellation. Verdict: sound. Confidence: high.
When: installed failure acceptance. A user can interrupt after some declaration
groups have returned useful source. That interruption ends further work and marks
-the result incomplete; it does not itself mean the source changed. Actual changed
+the result `interrupted`; it does not itself mean the source changed. Actual changed
or newly excluded snapshots still discard their evidence.
Gap: the filesystem interruption and invalidation paths shared a preparation
@@ -139,20 +183,6 @@ boundary, so the implementation had to distinguish their effect on retained data
Reach: partial output remains useful without claiming complete discovery or a
filesystem lock. Verdict: sound. Confidence: high.
-### Superseded source-budget tuning before port parity
-
-When: source-budget trial. I ranked limited source by contained declaration scores
-and selected a 1,500-byte default after one task improved against the already
-changed production port. The winning spike itself was uncapped and expanded
-context differently; the experiment did not establish parity with that winner.
-
-Gap: the spec delegated a measured budget but did not authorize treating an
-unverified port as equivalent to the accepted reference. Reach: the capped study
-and stopped cohort remain historical evidence only. Verdict: unsound sequencing.
-Corrected decision: restore exact reference retrieval/output/skill behavior and
-prove computed-request and output parity before introducing architecture changes.
-Confidence: high. The default is restored to uncapped.
-
### Freeze one installed package across the official cohort
When: maintained benchmark runner. Prepare all ten cells before execution, binding
@@ -175,33 +205,3 @@ Gap: the native smoke mechanism was delegated. Reach: portable assertions stay
shared while Linux tool-absence assertions retain their own Docker scope; a Mac
with Python installed cannot accidentally satisfy a runtime dependency through
PATH. Verdict: sound. Confidence: high.
-
-### Preserve the winner while deferring throughput changes
-
-When: reference restoration. A minified file can contain several declarations on
-one line; the winning strategy may ask about that line repeatedly under different
-declaration names. Combining those questions could change Jev's answers, so this
-port keeps them. Separately, saving a cache answer scans existing entries to
-enforce the disk limit. A large cache therefore has repeated scanning overhead.
-
-Gap: review found performance costs that are outside the user's current solve-rate
-and task-cost acceptance priorities. Reach: the confirmation does not establish
-whole-computer throughput or optimal cache maintenance. Verdict: sound for this
-confirmation, with medium confidence: retain measured retrieval behavior and
-report the cache limitation; test any later optimization as a separately identified
-artifact rather than silently changing the candidate being confirmed.
-
-### Isolate bundled CPython in one Node child process
-
-Gap: the spec requires no system Python and cancelable inspection but leaves the
-interpreter isolation mechanism open. Use the spike's unchanged helpers inside
-bundled CPython, with one Node child process owning interpreter state. A worker
-thread passed Node checks but its Pyodide loader failed under the existing Bun
-development runner; attempting to normalize its environment did not solve it.
-A single Node process path avoids patching the dependency or retaining two parsers.
-
-Reach: the package gains runtime assets and a child process; pure pending helper
-jobs may need replay when cancellation terminates that process. The product still
-requires only Node. Verdict: sound; merged lifecycle, parity and installed checks passed, with a
-clean review follow-up. Confidence: medium. Version coverage and footprint rationale live
-in [runtime evidence](assets/python-runtime.md).
diff --git a/specs/jevgrep/slices/04-checkpoint.md b/specs/jevgrep/slices/04-checkpoint.md
index 2aff648..eacf3ca 100644
--- a/specs/jevgrep/slices/04-checkpoint.md
+++ b/specs/jevgrep/slices/04-checkpoint.md
@@ -1,7 +1,6 @@
# Deliver the installed CLI and agent workflow
-Status: deterministic installed workflow verified; current bundled-CPython
-artifact still requires its frozen cohort checkpoint (2026-09-26). Depends on: 01, 02, 03.
+Status: verified for the frozen bundled-CPython artifact (2026-09-26). Depends on: 01, 02, 03.
Read [contracts](../contracts.md) first.
## Contract and owner
@@ -26,11 +25,11 @@ needed. CLI transcripts replace visual/screenshot gates for this product.
Docker fixture exercises installed auth, help, doctor, query, skill and partial output with empty stderr and isolated HOME/XDG. Pack excludes evals, secrets and workspace dependencies. Run one explicit live Jev query, then Sol through installed skill/CLI on the saved Requests task; inspect query, stdout, subsequent reads, patch, official grade and full bill. Reuse baseline; do not rerun it. One pass proves integration only, not cohort quality.
-The superseded reference-parity cohort's Requests cell officially resolved with complete
-Sol and Jev billing. Its source blocks match the accepted spike packet; the
-live file/lead decisions differ. This is historical integration evidence, not verification of the current
-bundled-CPython artifact or a cost advantage. [Restoration evidence](../assets/parity-restoration.md) owns the
-artifact identity and measured costs.
+The current Requests checkpoint officially solved with complete billing and a
+lower Sol cost than its saved baseline. The paired review checks its exact query,
+packet, subsequent reads, patch and tests against both baseline and accepted
+spike. [Cohort confirmation](../assets/cpython-confirmation.md) owns that evidence
+and its limits; one checkpoint is not cohort acceptance.
## Delegated decisions
@@ -46,5 +45,3 @@ Show the artifact and summarize deviations. This is a non-blocking review checkp
continue on the evidence if the user does not respond. Feedback that changes the
public contract or acceptance measure requires updating this slice before broadening
implementation. Record new choices and update the README handoff before ending.
-
-The earlier installed fixture, live query and official Requests task completed on a superseded package. The task solved but cost more than its saved baseline. Exact query, output, follow-up behavior, grade and billing are in the [checkpoint report](../assets/installed-requests-checkpoint.md). Reproduction uses the maintained [runner workflow](../../../evals/implementation/swebench/installed.md); the frozen checkpoint retains its original runner.