Skip to content

Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.

8.0 KB
Raw Blame

Installed package cohort

The runner evaluates one frozen installed jg package against the ten fixed Sol baselines. Baseline receipts, prompts, grades and billing are checked by hash; there is no baseline execution command. The two recorded baseline nonpasses remain nonpasses, including the retained Pylint test-collection limitation.

Preparation installs the packed npm artifact and dependencies once, then checks that same installation offline in every selected runtime image. Each check verifies the original source, agent versions and canonical packaged skill. The coding agent receives the archived prefix and runs the real jg executable on an internal network. The benchmark prompt explicitly invokes $jevgrep and requires a natural-language search before implementation discovery. An inventory command does not satisfy that requirement. Prompt changes require a new frozen baseline protocol; historical plans retain their archived runner. The public skill itself teaches CLI usage without prescribing a research workflow. Retrieval limits come from the frozen package's default policy.

New preparations use the current canonical skills/jevgrep/SKILL.md by default and reject a package carrying different skill bytes. Use --skill only for an explicitly identified comparison; existing frozen plans retain their original skill for reproducibility.

The plan binds one package, installed prefix, skill, safe task export, evaluator dataset and archived runner/broker/registry sources before any treatment runs. Existing studies remain unchanged; their archived source records the procedure used. A fresh full-cohort plan prevents combining the cheapest outcomes from different packages or experiments. A single-task plan is useful for diagnostics but cannot support a claim about the full cohort.

python3 evals/implementation/swebench/installed.py prepare \
  --package /absolute/path/to/package.tgz --output /absolute/path/to/new-study
python3 evals/implementation/swebench/installed.py run \
  --plan /absolute/path/to/new-study/plan.json --all --dry-run

Preparation and dry-run make no model calls. prepare defaults to all ten tasks; --jev-provider typesafe selects native Jev for a separately identified provider experiment. Sol continues through Gateway. The broker holds both real credentials; the agent receives only its scoped broker token. Native Jev cost is estimated from recorded input usage at the documented list price frozen in the runner. Total task cost includes both Sol and Jev. Missing usage or incomplete transport coverage leaves that total unknown. --task selects one registered task. --evidence-root locates the retained baseline and evaluator files when preparing from another checkout. The root eval:swebench alias points to this entry point; --help lists its arguments.

If a retained Docker image was removed, restore its pinned official base and archived toolchain, then prepare that one task with --runtime-image and the new immutable image ID. Preparation verifies the base image's filesystem layers, source identity, tool versions and installed skill. The frozen plan records both image IDs; baseline receipts remain untouched. Preserve the rebuild recipe and toolchain checksum verification beside the study and disclose the rebuild when reporting timing comparisons.

After reviewing the frozen plan, run --all without --dry-run executes the paid treatments sequentially. Load the Gateway credential through the authorized environment workflow. Real keys stay out of plans and command arguments. Every attempt has fresh home/cache state and a durable lifecycle receipt. Failed or completed attempts are retained; they are never replaced. Active processes stop continuation. An abandoned attempt whose runner and containers are gone becomes an interrupted result, preserving its evidence.

Follow execution with grade --plan ... --all and account --plan ... --all. Use --task instead of --all for an individual cell. Grading uses the pinned official harness and unique run IDs. Accounting reuses saved generation metadata, fetches missing receipts, and leaves full cost unknown if any request is missing. Neither operation reruns the coding agent.

Jev is reported separately under jev: the observed cost sums each retained response's providerMetadata.gateway.cost once. Input/output tokens and internal provider attempt counts are reported separately from client calls; provider fallbacks do not multiply the response cost. This is observed API metadata, not invoice reconciliation. Complete totals require matching client starts, completed transport receipts, request/response files and valid cost fields. Missing or invalid evidence leaves the total unknown while preserving a known subtotal. Retained response costs still contribute to that subtotal when the transport log or a request-start entry is missing. Jev charges contribute to total task cost; they remain separate from the Sol component and its token counts.

aggregate --plan ... reports official solves, protocol validity and billing for the selected tasks. It does not issue a promotion verdict. Unknown costs prevent a full cost total; the known subtotal remains explicit. Saved no-Jev results are historical context, not the release acceptance target.

Compare a candidate with the preceding released package using the same tasks, model, harness and public skill. Official task completion is primary, with full Sol cost reported alongside it. Keep both package artifacts and every attempt; do not combine favorable results from different candidates.

Run storage holds the frozen artifacts, agent output, patches, official reports and billing. The broker retains exact Jev request/response bodies in the non-agent container without authentication headers. These contain benchmark source and remain local with the ignored study evidence.

Focused tests run in an isolated Docker container. They cover prompt identity, drift rejection, no-call validation, retained attempts, incomplete billing, aggregate reporting and exact body capture through a synthetic provider.

Treatment work clock

New schema-3 plans freeze a 900-second treatment work budget. The monitor uses host monotonic observation of native command_execution started/completed events. Only native command-execution events establish concurrent work for this clock. It excludes the union of standalone jg search waits only while no other command is active. Overlapping retrieval calls earn no double credit; unrelated concurrent commands count as work. Completion, cancellation, terminal turns and process EOF end credit. Nonterminal error/reconnect events do not end an active command. A separate 24-hour wall limit protects against a runaway retrieval. The task container stays alive for setup and artifact capture until lifecycle cleanup; it has no competing shorter sleep deadline. An expired clock always records a failed attempt, even if the native process has already exited zero while output drains. Receipts retain work, credited retrieval and wall durations plus the disjoint credit intervals and native command IDs, so timing remains auditable.

The recognizer accepts a direct installed jg search, including the native shell wrapper. It recognizes search exclusions and gives no credit to files, auth, doctor, cache, skill, help/version, compound commands, redirections or shell expansions. Ambiguous invocations count as work. Event-observation timestamps measure what the harness sees, not provider execution time; malformed timing events invalidate the timing evidence rather than inventing credit. Process startup and time outside qualifying waits count toward work.

The treatment prompt states this clock explicitly; its timing sentence therefore differs from the retained baseline prompt. Baseline prompt bytes, receipts, model, grading are unchanged. This correction does not rescore old attempts or explain away earlier task failures. Schema-2 studies remain usable with their archived runner and wall-clock policy; the current runner requires a fresh schema-3 plan and rejects timing-policy drift.