Skip to content

Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.

3.1 KB
Raw Blame

Official benchmark evaluation

Use SWE-bench's official behavioral grader to compare product versions on matched tasks, source, model and harness. Keep architecture experiments and packaged-product runs separately identified. The architecture records the retrieval principles; saved no-Jev results provide additional context.

The maintained installed-package harness drives the actual jg executable with Sol and reuses the fixed baselines. It supports a single-task checkpoint or a complete ten-task cohort, preserving raw Jev requests/responses, coding-agent actions, patches, official grades and bills.

A baseline is immutable per task/model/harness and is never rerun to improve a comparison. The benchmark requires initial Jevgrep use to isolate retrieval's effect, while the production skill remains selective. Agents may fill remaining context gaps with ordinary tools.

Cost and quality policy owns acceptance. Count the full coding-agent task, including implementation, verification and failed attempts. Total cost includes Jev; Jev tokens are reported separately from agent tokens. Unknown bills cannot count as cost wins. Timing is diagnostic; report solve outcomes alongside cost changes. The tuned Python cohort does not prove untouched-task or language-wide generalization.

Raw traces, repository snapshots and generated study artifacts stay in ignored run storage. Commit reproducible official tooling and concise evidence reports. Personal-repository fixtures and custom-rubric evaluations are deprecated and stay outside Git; they are not product acceptance evidence.

The research archive preserves the superseded official experiments in Git without adding obsolete runners to main.

The final product research report records the neutral-skill cohort and parameter-effect map. The earlier source-first cohort retains its separate package and skill identity.

The speed study compares the frozen local optimizations and native TypeSafe route against that previous cohort and the same saved no-Jev baselines.

The total-cost rerun uses published 0.4.3 and includes Jev costs, the default for all future runs. Historical reports retain their original accounting basis.

The combined-cost research examines why lower retrieval charges can be offset by greater coding-agent expense.

The skill-impact pilot compares old and new skills on identical CLI builds, separately from automatic-trigger checks.

The Python parser checkpoint records one official task against the merged Tree-sitter runtime.

The bounded-search research records weak-score experiments, cost accounting repairs, and the protocol audit that distinguishes an inventory command from the required search.