Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.
TypeScript ai-sdkclaude-codeclicode-searchcodexcoding-agentscontext-retrievaldeveloper-toolsjevsemantic-searchtypescriptvercel-ai-gateway
Record skill impact pilot and account for inventory in benchmark timing
David Zhangcc3d860d9884
diff --git a/evals/README.md b/evals/README.mdindex 75a52f0..4bb8086 100644--- a/evals/README.md+++ b/evals/README.md@@ -37,3 +37,6 @@ The [total-cost rerun](results/total-cost-2026-09-28.md) uses published 0.4.3 anThe [combined-cost research](results/combined-cost-research-2026-09-28.md) examineswhy lower retrieval charges can be offset by greater coding-agent expense.++The [skill-impact pilot](results/skill-impact-2026-09-29.md) compares old and new+skills on identical CLI builds, separately from automatic-trigger checks.diff --git a/evals/implementation/swebench/installed.md b/evals/implementation/swebench/installed.mdindex 9cf090f..c76d04f 100644--- a/evals/implementation/swebench/installed.md+++ b/evals/implementation/swebench/installed.md@@ -14,6 +14,11 @@ on an internal network. The benchmark prompt explicitly invokes `$jevgrep`, as rthe public skill itself teaches CLI usage without prescribing a research workflow. Retrieval limits comefrom the frozen package's default policy.+New preparations use the current canonical `skills/jevgrep/SKILL.md` by default+and reject a package carrying different skill bytes. Use `--skill` only for an+explicitly identified comparison; existing frozen plans retain their original+skill for reproducibility.+The plan binds one package, installed prefix, skill, safe task export, evaluatordataset and archived runner/broker/registry sources before any treatment runs.Existing studies remain unchanged; their archived source records the procedure used.@@ -107,8 +112,8 @@ Receipts retain work, credited retrieval and wall durations plus the disjointcredit intervals and native command IDs, so timing remains auditable.The recognizer accepts a direct installed `jg` search, including the native shell-wrapper. It gives no credit to auth, doctor, cache, skill, help/version, compound-commands, redirections or shell expansions. Ambiguous invocations count as work.+wrapper. It recognizes search exclusions and gives no credit to files, auth,+doctor, cache, skill, help/version, compound commands, redirections or shell expansions. Ambiguous invocations count as work.Event-observation timestamps measure what the harness sees, not provider executiontime; malformed timing events invalidate the timing evidence rather than inventingcredit. Process startup and time outside qualifying waits count toward work.diff --git a/evals/implementation/swebench/installed.py b/evals/implementation/swebench/installed.pyindex dfbab98..9131a38 100644--- a/evals/implementation/swebench/installed.py+++ b/evals/implementation/swebench/installed.py@@ -320,13 +320,20 @@ def direct_jg_search(command):elif options and arg.startswith('--max-source-bytes='):if not arg.split('=', 1)[1].isdigit():return False+ elif options and arg == '--exclude':+ index += 1+ if index >= len(argv) or not argv[index]:+ return False+ elif options and arg.startswith('--exclude='):+ if not arg.split('=', 1)[1]:+ return Falseelif options and arg.startswith('-'):return Falseelse:positionals.append(arg)index += 1return (1 <= len(positionals) <= 2 and bool(positionals[0].strip()) and- positionals[0] not in ('auth', 'doctor', 'cache', 'skill'))+ positionals[0] not in ('auth', 'doctor', 'cache', 'skill', 'files'))except ValueError:return Falsediff --git a/evals/implementation/swebench/test_installed.py b/evals/implementation/swebench/test_installed.pyindex 185905d..b0629c3 100644--- a/evals/implementation/swebench/test_installed.py+++ b/evals/implementation/swebench/test_installed.py@@ -150,9 +150,9 @@ class InstalledTests(unittest.TestCase):def test_only_unambiguous_standalone_search_gets_credit(self):for command in ['jg "query with ; punctuation"', "/bin/sh -lc 'jg query'",- 'jg --no-cache "query" /testbed', '/opt/jg-install/bin/jg "query" --max-source-bytes=0']:+ 'jg --no-cache "query" /testbed', 'jg "query" --exclude tests/ --exclude=src/generated/', '/opt/jg-install/bin/jg "query" --max-source-bytes=0']:with self.subTest(command=command): self.assertTrue(runner.direct_jg_search(command))- for command in ['jg', 'jg doctor', 'jg auth --stdin', 'jg cache clear', 'jg skill', 'jg --help',+ for command in ['jg', 'jg files', 'jg files /testbed', 'jg files --exclude tests/', 'jg doctor', 'jg auth --stdin', 'jg cache clear', 'jg skill', 'jg --help','jg -h', 'jg --version', 'jg "query" --version', 'jg "query"; sleep 5','jg "query" && sleep 5', 'sleep 5 | jg "query"', 'jg "query" > out','jg "$(sleep 5)"', 'env KEY=value jg "query"', 'echo jg "query"', 'jg "query" &', 'jg doc*', 'jg auth?', 'jg {auth,doctor}',diff --git a/evals/results/skill-impact-2026-09-29.json b/evals/results/skill-impact-2026-09-29.jsonnew file mode 100644index 0000000..5836ac4--- /dev/null+++ b/evals/results/skill-impact-2026-09-29.json@@ -0,0 +1,188 @@+{+ "protocol": "Two exploratory paired tasks; same CLI, source, model, prompts, provider and corrected work clock; only packaged skill differs. One attempt per arm/task.",+ "cli_source_commit": "dd43534c891c5779eedcba50b3d40ae3edbeec7b",+ "skill_sha256": {+ "old": "82d4537183cb9219d60400df1d478ad0b213e14bf05bf6ae9b7f7e751dc3d935",+ "new": "916bf51b9a57f1cb94d04dbd90d361f1190b6dabc0f9d1ca152d2469862a974c"+ },+ "cells": [+ {+ "task": "psf__requests-1142",+ "arm": "old",+ "status": "completed",+ "official_solved": true,+ "grade_report": "/Users/david/dev/jevgrep/evals/runs/tooling/swebench/logs/evaluation/jg-skill-requests-old-v2-pinned-platform/results.json",+ "grading_recovered": true,+ "cost_complete": true,+ "total_usd": 0.29272813799999997,+ "agent_usd": 0.2863362,+ "jev_usd": 0.006391938,+ "known_agent_usd": 0.2863362,+ "timing": {+ "exit_code": 0,+ "timed_out": false,+ "timeout_kind": null,+ "wall_seconds": 74.39526695897803,+ "work_seconds": 58.74530012597097,+ "retrieval_wait_seconds": 15.649966833007056,+ "credit_intervals": [+ {+ "start_seconds": 8.6868124169996,+ "end_seconds": 24.336779250006657,+ "command_ids": [+ "item_2"+ ]+ }+ ],+ "timing_event_errors": 0+ },+ "agent_token_usage": [+ {+ "input_tokens": 216746,+ "cached_input_tokens": 187053,+ "cache_write_input_tokens": 29663,+ "output_tokens": 3154,+ "reasoning_output_tokens": 1519+ }+ ],+ "evidence": "/Users/david/dev/jevgrep/evals/runs/skill-impact-2026-09-29/requests-old-v2/attempts/psf__requests-1142"+ },+ {+ "task": "psf__requests-1142",+ "arm": "new",+ "status": "completed",+ "official_solved": true,+ "grade_report": "/Users/david/dev/jevgrep/evals/runs/tooling/swebench/logs/evaluation/jg-skill-requests-new-pinned-platform/results.json",+ "grading_recovered": true,+ "cost_complete": true,+ "total_usd": 0.20895612800000002,+ "agent_usd": 0.2034632,+ "jev_usd": 0.005492928,+ "known_agent_usd": 0.2034632,+ "timing": {+ "exit_code": 0,+ "timed_out": false,+ "timeout_kind": null,+ "wall_seconds": 65.58349579200149,+ "work_seconds": 49.57501416699961,+ "retrieval_wait_seconds": 16.008481625001878,+ "credit_intervals": [+ {+ "start_seconds": 8.87021837499924,+ "end_seconds": 24.878700000001118,+ "command_ids": [+ "item_2"+ ]+ }+ ],+ "timing_event_errors": 0+ },+ "agent_token_usage": [+ {+ "input_tokens": 116447,+ "cached_input_tokens": 94498,+ "cache_write_input_tokens": 21928,+ "output_tokens": 2797,+ "reasoning_output_tokens": 1445+ }+ ],+ "evidence": "/Users/david/dev/jevgrep/evals/runs/skill-impact-2026-09-29/requests-new/attempts/psf__requests-1142"+ },+ {+ "task": "pytest-dev__pytest-6197",+ "arm": "old",+ "status": "completed",+ "official_solved": true,+ "grade_report": "/Users/david/dev/jevgrep/evals/runs/tooling/swebench/logs/evaluation/jg-installed-pytest-dev__pytest-6197-563b4f6313bb/results.json",+ "grading_recovered": false,+ "cost_complete": true,+ "total_usd": 1.190029434,+ "agent_usd": 1.1434104,+ "jev_usd": 0.046619034,+ "known_agent_usd": 1.1434104,+ "timing": {+ "exit_code": 0,+ "timed_out": false,+ "timeout_kind": null,+ "wall_seconds": 306.988658999966,+ "work_seconds": 269.7163303749985,+ "retrieval_wait_seconds": 37.272328624967486,+ "credit_intervals": [+ {+ "start_seconds": 9.96058895799797,+ "end_seconds": 47.232917582965456,+ "command_ids": [+ "item_2"+ ]+ }+ ],+ "timing_event_errors": 0+ },+ "agent_token_usage": [+ {+ "input_tokens": 1394275,+ "cached_input_tokens": 1330041,+ "cache_write_input_tokens": 64138,+ "output_tokens": 14516,+ "reasoning_output_tokens": 9970+ }+ ],+ "evidence": "/Users/david/dev/jevgrep/evals/runs/skill-impact-2026-09-29/pytest-old/attempts/pytest-dev__pytest-6197"+ },+ {+ "task": "pytest-dev__pytest-6197",+ "arm": "new",+ "status": "completed",+ "official_solved": true,+ "grade_report": "/Users/david/dev/jevgrep/evals/runs/tooling/swebench/logs/evaluation/jg-installed-pytest-dev__pytest-6197-3ebfd911d4b8/results.json",+ "grading_recovered": false,+ "cost_complete": true,+ "total_usd": 0.896717274,+ "agent_usd": 0.856347,+ "jev_usd": 0.040370274,+ "known_agent_usd": 0.856347,+ "timing": {+ "exit_code": 0,+ "timed_out": false,+ "timeout_kind": null,+ "wall_seconds": 227.74059166701045,+ "work_seconds": 207.00520112604136,+ "retrieval_wait_seconds": 20.735390540969092,+ "credit_intervals": [+ {+ "start_seconds": 7.682393792027142,+ "end_seconds": 28.417784332996234,+ "command_ids": [+ "item_2"+ ]+ }+ ],+ "timing_event_errors": 0+ },+ "agent_token_usage": [+ {+ "input_tokens": 947284,+ "cached_input_tokens": 891040,+ "cache_write_input_tokens": 56175,+ "output_tokens": 10939,+ "reasoning_output_tokens": 7811+ }+ ],+ "evidence": "/Users/david/dev/jevgrep/evals/runs/skill-impact-2026-09-29/pytest-new/attempts/pytest-dev__pytest-6197"+ }+ ],+ "trigger_report": "/Users/david/dev/jevgrep/evals/runs/skill-trigger-2026-09-29/REPORT.md",+ "rollout_manifest": "/Users/david/dev/jevgrep/evals/runs/skill-rollout-2026-09-29/manifest.json",+ "totals": {+ "old": {+ "official_solves": 2,+ "graded_tasks": 2,+ "total_usd": 1.482757572+ },+ "new": {+ "official_solves": 2,+ "graded_tasks": 2,+ "total_usd": 1.105673402+ }+ }+}diff --git a/evals/results/skill-impact-2026-09-29.md b/evals/results/skill-impact-2026-09-29.mdnew file mode 100644index 0000000..3304dc2--- /dev/null+++ b/evals/results/skill-impact-2026-09-29.md@@ -0,0 +1,92 @@+# Skill rollout and benchmark pilot++The updated skill preserved both official solves and reduced observed total cost+from **$1.483 to $1.106 (25.4%)** in a two-task paired pilot. Automatic invocation+showed **no movement** in an isolated Codex trigger test. These are exploratory+observations, not a statistically established cost reduction or a full-cohort gain.++## Rollout++Updated existing skill installations in `aiexplainer`, `jevgrep`, `battlegame`, and+one existing Battlegame worktree under `~/dev`. Five physical files and their+Claude aliases (nine paths total) match the merged canonical skill. Historical+benchmark snapshots were preserved. Previous bytes and SHA256 records are in the+[rollout manifest](../runs/skill-rollout-2026-09-29/manifest.json).++This updates skill text, not the globally installed CLI. The global executable+remains 0.5.0; the skill advises checking help for newer commands. The benchmark+used locally packed current-main CLI builds that include `jg files`.++## Automatic triggering++Eighteen fresh native Codex sessions compared the full old and new skills on+three issue-29 prompts, three repetitions per version. Both versions read the+skill and invoked Jevgrep before source-content search on **6/6 behavioral+questions**, and correctly used grep without Jevgrep on **3/3 exact-symbol+questions**. Both scored **0/6** under the stricter before-any-filename-search+measure; that measure includes AGENTS.md discovery, not just source exploration.++The runs used configured `gpt-6-astra`/medium, one enabled task skill, a frozen+aiohttp-session checkout, and an inert command shim. They measure attempted+invocation, not successful retrieval. The isolated catalog creates a ceiling+effect and does not reproduce Claude's competing-plugin setup in issue #29.+A separate reviewer audited all 18 raw traces and confirmed the counts.++The [trigger report](../runs/skill-trigger-2026-09-29/REPORT.md) contains the+protocol, per-run evidence, token usage, invalid initial probe, and limitations.+Native CLI monetary usage was unavailable; the shim made no retrieval calls.++## Official SWE-bench pilot++| Task | Old skill | New skill | Old total cost | New total cost |+|---|---|---|---:|---:|+| Requests 1142 | Solved | Solved | $0.292728 | $0.208956 |+| Pytest 6197 | Solved | Solved | $1.190029 | $0.896717 |+| Total | 2/2 | 2/2 | **$1.482758** | **$1.105673** |++The two npm archives differ only in `dist/skills/jevgrep/SKILL.md`. Both use the+merged CLI source from `dd43534`, identical pinned task sources and runtime images,+`openai/gpt-5.6-sol`/medium through Gateway, native TypeSafe retrieval, and the+same corrected work-clock harness. The old skill is from `a9993c1`; the new one+is from `dd43534`. Full identities and per-task receipts are in the+[result data](skill-impact-2026-09-29.json).++Each arm/task ran once, in fixed old-then-new order. These are new skill-control+attempts on identical CLI code; historical no-Jev baselines were neither rerun nor+selected as the comparison. The benchmark explicitly invokes `$jevgrep`, so it+cannot measure automatic triggering. Both Requests arms and both Pytest arms ran+one semantic search; the new Pytest arm additionally ran `jg files`.++Observed cost fell on both tasks. Most of the difference was coding-agent usage:+combined agent charges fell from $1.429747 to $1.059810; Jev estimates fell from+$0.053011 to $0.045863. All four task totals have complete accounting. Native Jev+uses the harness's retained list-price estimate, not invoice reconciliation.+Four paid coding attempts totalled $2.588431; this excludes the native Codex+trigger/audit sessions whose monetary usage was not returned.++Timing also decreased (Requests 74.4 → 65.6 seconds, Pytest 307.0 → 227.7 seconds),+but remains diagnostic. One pair per task cannot separate skill effects from+agent/provider variance, order, cache effects, or changed query wording. No+repeats were selected or discarded to improve scores. The small, previously+studied Python sample does not establish generalization.++## Validation and recoveries++Before paid trials, corrected the timing classifier to count `jg files` as work+and recognize searches with `--exclude`. Added cases failed against the old+classifier; all 30 harness tests passed in Docker after the fix. Independent+review found no actionable issues. Both arms freeze the same corrected runner.+An earlier no-call plan was retained as superseded.++Requests initially hit an official-grader platform error: a missing local x86+image tag caused an ARM64 pull. Restored the tag from the pinned AMD64 digest and+regraded the unchanged patches under new run IDs. Both passed; initial error+reports and recovery receipts remain available. No coding attempt was rerun.+One candidate Gateway charge became available on a single accounting retry.++## Next experiment++Keep the merged skill. A useful next step is a predeclared expansion to more+matched SWE-bench tasks, plus a separate native Claude trigger test with its+usual competing skills. Do not infer a broad 25% saving or a solve-rate gain from+this pilot. No additional runs are pending.