Skip to content

Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.

Preregister isolated mechanism-first skill trial and correct release evidence

David Zhang91d24c19f5ad
diff --git a/specs/jevgrep/README.md b/specs/jevgrep/README.md
index 246e968..2c9c065 100644
--- a/specs/jevgrep/README.md
+++ b/specs/jevgrep/README.md
@@ -7,7 +7,8 @@ Jev classification through Vercel AI Gateway. This is the build plan; production
## Next Agent Prompt
You are implementing this spec. Status: **implementing**, updated **2026-09-26**.
-Next: diagnose the Django regression with one isolated query-framing experiment.
+Next: prepare and run Django first in the isolated
+[mechanism-first skill experiment](assets/query-framing-study.md).
The frozen `installed-jg-cpython-parity-v1` cohort is complete and not accepted:
7/10 solves versus 8/10 baseline, six cost wins, and 35.27% lower full Sol cost.
[Confirmation evidence](assets/cpython-confirmation.md) owns all outcomes, trace
@@ -25,7 +26,8 @@ exercised corpus; task-quality acceptance failed and remains required.
scikit-learn officially solved but exceeded baseline cost; Django was interrupted
with its patch and traces retained; seven cells never started. These superseded
studies remain stopped. The bundled-CPython cohort is terminal; no Sol run is
-active. The isolated diagnostic makes only bounded Jev classification calls.
+active. The completed query diagnostics made only bounded Jev calls. A separately
+identified skill candidate is prepared for an actual Django task test.
The changed-policy cohort `installed-jg-final-cohort-v1` was stopped and marked
superseded, retaining completed outcomes and interrupted traces. It is not a
diff --git a/specs/jevgrep/assets/python-runtime.md b/specs/jevgrep/assets/python-runtime.md
index f5b3085..674f596 100644
--- a/specs/jevgrep/assets/python-runtime.md
+++ b/specs/jevgrep/assets/python-runtime.md
@@ -73,8 +73,11 @@ whole runtime as Apache-only. Packaging owns those notices and their verificatio
`evals/runs/swebench/installed-jg-cpython-parity-v1/plan.json` freezes all ten tasks
on the archive above and the existing Sol model/harness/baselines. Its exact
archive passed four offline installed checks on each of Linux amd64 and arm64,
-and the native macOS smoke identified the same SHA-256. Release content validation
-and all ten dry-run validations passed before paid execution. Current official
+and the native macOS smoke identified the same SHA-256. All ten dry-run
+validations passed before paid execution. Release-validator unit tests passed,
+but the direct archive validation intentionally rejected development version
+`0.0.0` before content checks. An earlier statement that this archive passed
+release validation was incorrect; the recorded output confirms that rejection. Current official
results and paired trace findings belong in the
[cohort confirmation](cpython-confirmation.md). Superseded studies remain
historical and contribute no outcomes to this cohort. Platform transcripts are
diff --git a/specs/jevgrep/assets/query-framing-study.md b/specs/jevgrep/assets/query-framing-study.md
new file mode 100644
index 0000000..dad6ac5
--- /dev/null
+++ b/specs/jevgrep/assets/query-framing-study.md
@@ -0,0 +1,66 @@
+# Mechanism-first query experiment
+
+The faithful bundled-CPython port remains unchanged. Its complete cohort missed
+acceptance; [confirmation evidence](cpython-confirmation.md) owns those results.
+This separately identified experiment tests the caller's initial query scope,
+not a replacement parser, traversal strategy, threshold or source representation.
+
+## Contract registered before the coding-agent run
+
+Candidate branch commit: `99b6afefd0dfe33311a2c1139c67551a2d7e4478`.
+Candidate package SHA-256:
+`2b2b4712fcff4780d747063390bb31a460b7f368c260aefcce5dc5af9e254dcb`.
+Only the packaged skill differs from the parent archive. Every other member's
+payload and metadata is identical; proof lives in `/tmp/jg-mechanism-query-v1/`.
+The production worktree's canonical skill is unchanged. Both archives remain
+unpublishable development version `0.0.0`; release validation intentionally rejects
+that version. No validation-only version shadow is substituted for the candidate.
+
+Replace only the skill's instruction to describe symptoms, expected behavior and
+reproduction clues with:
+
+> Start by asking how the existing mechanism works and where it is tested. Leave the symptom, desired fix, and platform-specific restriction out of this first query; apply those constraints when interpreting the result and implementing.
+
+This is a coupled change to query scope, not just tone. It contains no benchmark
+names, implementation paths, missing conditions, or evaluator answer hints.
+The original issue remains unchanged in the agent prompt, and every later skill
+instruction remains byte-identical.
+
+Prepare one new ten-task frozen plan, `installed-jg-mechanism-query-v1`, using the
+same package installation process, registry, task snapshots, baseline prompts,
+Sol model/effort, 900-second agent limit and official grading/accounting. Baselines
+run zero times. Execute Django first, once. All internal repairs, tool calls and
+tests performed by that attempt remain in its full bill. No interactive hints,
+replacement attempts or mid-run candidate edits are allowed.
+
+The first check is whether the actual initial query follows the new instruction
+and provides useful implementation/test evidence. This diagnoses compliance; it
+does not replace outcome measurement. Local outcome: official Django solve with a
+fully accounted Sol bill, compared with its fixed $1.5055472 baseline and the
+parent's failed $1.0015696 attempt. Jev and experimental research spend stay
+separate from scored Sol cost, including incomplete provider observations.
+
+If Django solves below baseline cost, continue the same frozen plan's remaining
+cells without rerunning Django. If it still fails, retain the result and diagnose
+before spending on the remainder. A solved but more expensive outcome is a
+partial win to investigate, not grounds to discard the strategy automatically.
+Promotion still requires the original full-cohort gate: preserve all eight baseline
+solves and reach at least seven successful lower-cost solves. Never pool outcomes
+from different candidates. A focused result or classifier score cannot complete
+the product goal.
+
+## Evidence and parameter-effect map
+
+| Hypothesis | Controlled observation | Verdict / next discriminator |
+| --- | --- | --- |
+| Query content affects declaration selection | Same sklearn source/questions: current query 0.26, spike query 0.50, neutral mechanism query 0.69 | Supported as query sensitivity in one observation each; no task-quality claim |
+| Neutral wording alone fixes Django selection | Same SQLite method/source/questions: current 0.09, neutral same concepts 0.06 | Not supported |
+| Broader mechanism scope admits relevant missing context | Removing collation and MySQL focus together raises that Django method to 0.52 | Coupled scope effect in one observation; individual contributions and downstream value unknown |
+| Initial mechanism-first skill improves the failed task | Candidate defined above; coding-agent result pending | Inspect actual query, packet, reads, patch, tests, official grade and complete bill |
+
+Each diagnostic used three successful Jev calls, no SDK retries and a 60-second
+diagnostic timeout, without changing production's timeout. Observed research Jev
+cost: $0.000414750 for sklearn and $0.000318360 for Django. Exact preregistration,
+requests, responses and unchanged-source proofs are in `/tmp/jg-query-sensitivity/`
+and `/tmp/jg-django-query-sensitivity/`. These are development diagnostics on
+known cases; they establish no untouched generalization.