Skip to content

Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.

Record skill impact pilot and account for inventory in benchmark timing

David Zhangcc3d860d9884
diff --git a/evals/README.md b/evals/README.md
index 75a52f0..4bb8086 100644
--- a/evals/README.md
+++ b/evals/README.md
@@ -37,3 +37,6 @@ The [total-cost rerun](results/total-cost-2026-09-28.md) uses published 0.4.3 an
The [combined-cost research](results/combined-cost-research-2026-09-28.md) examines
why lower retrieval charges can be offset by greater coding-agent expense.
+
+The [skill-impact pilot](results/skill-impact-2026-09-29.md) compares old and new
+skills on identical CLI builds, separately from automatic-trigger checks.
diff --git a/evals/implementation/swebench/installed.md b/evals/implementation/swebench/installed.md
index 9cf090f..c76d04f 100644
--- a/evals/implementation/swebench/installed.md
+++ b/evals/implementation/swebench/installed.md
@@ -14,6 +14,11 @@ on an internal network. The benchmark prompt explicitly invokes `$jevgrep`, as r
the public skill itself teaches CLI usage without prescribing a research workflow. Retrieval limits come
from the frozen package's default policy.
+New preparations use the current canonical `skills/jevgrep/SKILL.md` by default
+and reject a package carrying different skill bytes. Use `--skill` only for an
+explicitly identified comparison; existing frozen plans retain their original
+skill for reproducibility.
+
The plan binds one package, installed prefix, skill, safe task export, evaluator
dataset and archived runner/broker/registry sources before any treatment runs.
Existing studies remain unchanged; their archived source records the procedure used.
@@ -107,8 +112,8 @@ Receipts retain work, credited retrieval and wall durations plus the disjoint
credit intervals and native command IDs, so timing remains auditable.
The recognizer accepts a direct installed `jg` search, including the native shell
-wrapper. It gives no credit to auth, doctor, cache, skill, help/version, compound
-commands, redirections or shell expansions. Ambiguous invocations count as work.
+wrapper. It recognizes search exclusions and gives no credit to files, auth,
+doctor, cache, skill, help/version, compound commands, redirections or shell expansions. Ambiguous invocations count as work.
Event-observation timestamps measure what the harness sees, not provider execution
time; malformed timing events invalidate the timing evidence rather than inventing
credit. Process startup and time outside qualifying waits count toward work.
diff --git a/evals/implementation/swebench/installed.py b/evals/implementation/swebench/installed.py
index dfbab98..9131a38 100644
--- a/evals/implementation/swebench/installed.py
+++ b/evals/implementation/swebench/installed.py
@@ -320,13 +320,20 @@ def direct_jg_search(command):
elif options and arg.startswith('--max-source-bytes='):
if not arg.split('=', 1)[1].isdigit():
return False
+ elif options and arg == '--exclude':
+ index += 1
+ if index >= len(argv) or not argv[index]:
+ return False
+ elif options and arg.startswith('--exclude='):
+ if not arg.split('=', 1)[1]:
+ return False
elif options and arg.startswith('-'):
return False
else:
positionals.append(arg)
index += 1
return (1 <= len(positionals) <= 2 and bool(positionals[0].strip()) and
- positionals[0] not in ('auth', 'doctor', 'cache', 'skill'))
+ positionals[0] not in ('auth', 'doctor', 'cache', 'skill', 'files'))
except ValueError:
return False
diff --git a/evals/implementation/swebench/test_installed.py b/evals/implementation/swebench/test_installed.py
index 185905d..b0629c3 100644
--- a/evals/implementation/swebench/test_installed.py
+++ b/evals/implementation/swebench/test_installed.py
@@ -150,9 +150,9 @@ class InstalledTests(unittest.TestCase):
def test_only_unambiguous_standalone_search_gets_credit(self):
for command in ['jg "query with ; punctuation"', "/bin/sh -lc 'jg query'",
- 'jg --no-cache "query" /testbed', '/opt/jg-install/bin/jg "query" --max-source-bytes=0']:
+ 'jg --no-cache "query" /testbed', 'jg "query" --exclude tests/ --exclude=src/generated/', '/opt/jg-install/bin/jg "query" --max-source-bytes=0']:
with self.subTest(command=command): self.assertTrue(runner.direct_jg_search(command))
- for command in ['jg', 'jg doctor', 'jg auth --stdin', 'jg cache clear', 'jg skill', 'jg --help',
+ for command in ['jg', 'jg files', 'jg files /testbed', 'jg files --exclude tests/', 'jg doctor', 'jg auth --stdin', 'jg cache clear', 'jg skill', 'jg --help',
'jg -h', 'jg --version', 'jg "query" --version', 'jg "query"; sleep 5',
'jg "query" && sleep 5', 'sleep 5 | jg "query"', 'jg "query" > out',
'jg "$(sleep 5)"', 'env KEY=value jg "query"', 'echo jg "query"', 'jg "query" &', 'jg doc*', 'jg auth?', 'jg {auth,doctor}',
diff --git a/evals/results/skill-impact-2026-09-29.json b/evals/results/skill-impact-2026-09-29.json
new file mode 100644
index 0000000..5836ac4
--- /dev/null
+++ b/evals/results/skill-impact-2026-09-29.json
@@ -0,0 +1,188 @@
+{
+ "protocol": "Two exploratory paired tasks; same CLI, source, model, prompts, provider and corrected work clock; only packaged skill differs. One attempt per arm/task.",
+ "cli_source_commit": "dd43534c891c5779eedcba50b3d40ae3edbeec7b",
+ "skill_sha256": {
+ "old": "82d4537183cb9219d60400df1d478ad0b213e14bf05bf6ae9b7f7e751dc3d935",
+ "new": "916bf51b9a57f1cb94d04dbd90d361f1190b6dabc0f9d1ca152d2469862a974c"
+ },
+ "cells": [
+ {
+ "task": "psf__requests-1142",
+ "arm": "old",
+ "status": "completed",
+ "official_solved": true,
+ "grade_report": "/Users/david/dev/jevgrep/evals/runs/tooling/swebench/logs/evaluation/jg-skill-requests-old-v2-pinned-platform/results.json",
+ "grading_recovered": true,
+ "cost_complete": true,
+ "total_usd": 0.29272813799999997,
+ "agent_usd": 0.2863362,
+ "jev_usd": 0.006391938,
+ "known_agent_usd": 0.2863362,
+ "timing": {
+ "exit_code": 0,
+ "timed_out": false,
+ "timeout_kind": null,
+ "wall_seconds": 74.39526695897803,
+ "work_seconds": 58.74530012597097,
+ "retrieval_wait_seconds": 15.649966833007056,
+ "credit_intervals": [
+ {
+ "start_seconds": 8.6868124169996,
+ "end_seconds": 24.336779250006657,
+ "command_ids": [
+ "item_2"
+ ]
+ }
+ ],
+ "timing_event_errors": 0
+ },
+ "agent_token_usage": [
+ {
+ "input_tokens": 216746,
+ "cached_input_tokens": 187053,
+ "cache_write_input_tokens": 29663,
+ "output_tokens": 3154,
+ "reasoning_output_tokens": 1519
+ }
+ ],
+ "evidence": "/Users/david/dev/jevgrep/evals/runs/skill-impact-2026-09-29/requests-old-v2/attempts/psf__requests-1142"
+ },
+ {
+ "task": "psf__requests-1142",
+ "arm": "new",
+ "status": "completed",
+ "official_solved": true,
+ "grade_report": "/Users/david/dev/jevgrep/evals/runs/tooling/swebench/logs/evaluation/jg-skill-requests-new-pinned-platform/results.json",
+ "grading_recovered": true,
+ "cost_complete": true,
+ "total_usd": 0.20895612800000002,
+ "agent_usd": 0.2034632,
+ "jev_usd": 0.005492928,
+ "known_agent_usd": 0.2034632,
+ "timing": {
+ "exit_code": 0,
+ "timed_out": false,
+ "timeout_kind": null,
+ "wall_seconds": 65.58349579200149,
+ "work_seconds": 49.57501416699961,
+ "retrieval_wait_seconds": 16.008481625001878,
+ "credit_intervals": [
+ {
+ "start_seconds": 8.87021837499924,
+ "end_seconds": 24.878700000001118,
+ "command_ids": [
+ "item_2"
+ ]
+ }
+ ],
+ "timing_event_errors": 0
+ },
+ "agent_token_usage": [
+ {
+ "input_tokens": 116447,
+ "cached_input_tokens": 94498,
+ "cache_write_input_tokens": 21928,
+ "output_tokens": 2797,
+ "reasoning_output_tokens": 1445
+ }
+ ],
+ "evidence": "/Users/david/dev/jevgrep/evals/runs/skill-impact-2026-09-29/requests-new/attempts/psf__requests-1142"
+ },
+ {
+ "task": "pytest-dev__pytest-6197",
+ "arm": "old",
+ "status": "completed",
+ "official_solved": true,
+ "grade_report": "/Users/david/dev/jevgrep/evals/runs/tooling/swebench/logs/evaluation/jg-installed-pytest-dev__pytest-6197-563b4f6313bb/results.json",
+ "grading_recovered": false,
+ "cost_complete": true,
+ "total_usd": 1.190029434,
+ "agent_usd": 1.1434104,
+ "jev_usd": 0.046619034,
+ "known_agent_usd": 1.1434104,
+ "timing": {
+ "exit_code": 0,
+ "timed_out": false,
+ "timeout_kind": null,
+ "wall_seconds": 306.988658999966,
+ "work_seconds": 269.7163303749985,
+ "retrieval_wait_seconds": 37.272328624967486,
+ "credit_intervals": [
+ {
+ "start_seconds": 9.96058895799797,
+ "end_seconds": 47.232917582965456,
+ "command_ids": [
+ "item_2"
+ ]
+ }
+ ],
+ "timing_event_errors": 0
+ },
+ "agent_token_usage": [
+ {
+ "input_tokens": 1394275,
+ "cached_input_tokens": 1330041,
+ "cache_write_input_tokens": 64138,
+ "output_tokens": 14516,
+ "reasoning_output_tokens": 9970
+ }
+ ],
+ "evidence": "/Users/david/dev/jevgrep/evals/runs/skill-impact-2026-09-29/pytest-old/attempts/pytest-dev__pytest-6197"
+ },
+ {
+ "task": "pytest-dev__pytest-6197",
+ "arm": "new",
+ "status": "completed",
+ "official_solved": true,
+ "grade_report": "/Users/david/dev/jevgrep/evals/runs/tooling/swebench/logs/evaluation/jg-installed-pytest-dev__pytest-6197-3ebfd911d4b8/results.json",
+ "grading_recovered": false,
+ "cost_complete": true,
+ "total_usd": 0.896717274,
+ "agent_usd": 0.856347,
+ "jev_usd": 0.040370274,
+ "known_agent_usd": 0.856347,
+ "timing": {
+ "exit_code": 0,
+ "timed_out": false,
+ "timeout_kind": null,
+ "wall_seconds": 227.74059166701045,
+ "work_seconds": 207.00520112604136,
+ "retrieval_wait_seconds": 20.735390540969092,
+ "credit_intervals": [
+ {
+ "start_seconds": 7.682393792027142,
+ "end_seconds": 28.417784332996234,
+ "command_ids": [
+ "item_2"
+ ]
+ }
+ ],
+ "timing_event_errors": 0
+ },
+ "agent_token_usage": [
+ {
+ "input_tokens": 947284,
+ "cached_input_tokens": 891040,
+ "cache_write_input_tokens": 56175,
+ "output_tokens": 10939,
+ "reasoning_output_tokens": 7811
+ }
+ ],
+ "evidence": "/Users/david/dev/jevgrep/evals/runs/skill-impact-2026-09-29/pytest-new/attempts/pytest-dev__pytest-6197"
+ }
+ ],
+ "trigger_report": "/Users/david/dev/jevgrep/evals/runs/skill-trigger-2026-09-29/REPORT.md",
+ "rollout_manifest": "/Users/david/dev/jevgrep/evals/runs/skill-rollout-2026-09-29/manifest.json",
+ "totals": {
+ "old": {
+ "official_solves": 2,
+ "graded_tasks": 2,
+ "total_usd": 1.482757572
+ },
+ "new": {
+ "official_solves": 2,
+ "graded_tasks": 2,
+ "total_usd": 1.105673402
+ }
+ }
+}
diff --git a/evals/results/skill-impact-2026-09-29.md b/evals/results/skill-impact-2026-09-29.md
new file mode 100644
index 0000000..3304dc2
--- /dev/null
+++ b/evals/results/skill-impact-2026-09-29.md
@@ -0,0 +1,92 @@
+# Skill rollout and benchmark pilot
+
+The updated skill preserved both official solves and reduced observed total cost
+from **$1.483 to $1.106 (25.4%)** in a two-task paired pilot. Automatic invocation
+showed **no movement** in an isolated Codex trigger test. These are exploratory
+observations, not a statistically established cost reduction or a full-cohort gain.
+
+## Rollout
+
+Updated existing skill installations in `aiexplainer`, `jevgrep`, `battlegame`, and
+one existing Battlegame worktree under `~/dev`. Five physical files and their
+Claude aliases (nine paths total) match the merged canonical skill. Historical
+benchmark snapshots were preserved. Previous bytes and SHA256 records are in the
+[rollout manifest](../runs/skill-rollout-2026-09-29/manifest.json).
+
+This updates skill text, not the globally installed CLI. The global executable
+remains 0.5.0; the skill advises checking help for newer commands. The benchmark
+used locally packed current-main CLI builds that include `jg files`.
+
+## Automatic triggering
+
+Eighteen fresh native Codex sessions compared the full old and new skills on
+three issue-29 prompts, three repetitions per version. Both versions read the
+skill and invoked Jevgrep before source-content search on **6/6 behavioral
+questions**, and correctly used grep without Jevgrep on **3/3 exact-symbol
+questions**. Both scored **0/6** under the stricter before-any-filename-search
+measure; that measure includes AGENTS.md discovery, not just source exploration.
+
+The runs used configured `gpt-6-astra`/medium, one enabled task skill, a frozen
+aiohttp-session checkout, and an inert command shim. They measure attempted
+invocation, not successful retrieval. The isolated catalog creates a ceiling
+effect and does not reproduce Claude's competing-plugin setup in issue #29.
+A separate reviewer audited all 18 raw traces and confirmed the counts.
+
+The [trigger report](../runs/skill-trigger-2026-09-29/REPORT.md) contains the
+protocol, per-run evidence, token usage, invalid initial probe, and limitations.
+Native CLI monetary usage was unavailable; the shim made no retrieval calls.
+
+## Official SWE-bench pilot
+
+| Task | Old skill | New skill | Old total cost | New total cost |
+|---|---|---|---:|---:|
+| Requests 1142 | Solved | Solved | $0.292728 | $0.208956 |
+| Pytest 6197 | Solved | Solved | $1.190029 | $0.896717 |
+| Total | 2/2 | 2/2 | **$1.482758** | **$1.105673** |
+
+The two npm archives differ only in `dist/skills/jevgrep/SKILL.md`. Both use the
+merged CLI source from `dd43534`, identical pinned task sources and runtime images,
+`openai/gpt-5.6-sol`/medium through Gateway, native TypeSafe retrieval, and the
+same corrected work-clock harness. The old skill is from `a9993c1`; the new one
+is from `dd43534`. Full identities and per-task receipts are in the
+[result data](skill-impact-2026-09-29.json).
+
+Each arm/task ran once, in fixed old-then-new order. These are new skill-control
+attempts on identical CLI code; historical no-Jev baselines were neither rerun nor
+selected as the comparison. The benchmark explicitly invokes `$jevgrep`, so it
+cannot measure automatic triggering. Both Requests arms and both Pytest arms ran
+one semantic search; the new Pytest arm additionally ran `jg files`.
+
+Observed cost fell on both tasks. Most of the difference was coding-agent usage:
+combined agent charges fell from $1.429747 to $1.059810; Jev estimates fell from
+$0.053011 to $0.045863. All four task totals have complete accounting. Native Jev
+uses the harness's retained list-price estimate, not invoice reconciliation.
+Four paid coding attempts totalled $2.588431; this excludes the native Codex
+trigger/audit sessions whose monetary usage was not returned.
+
+Timing also decreased (Requests 74.4 → 65.6 seconds, Pytest 307.0 → 227.7 seconds),
+but remains diagnostic. One pair per task cannot separate skill effects from
+agent/provider variance, order, cache effects, or changed query wording. No
+repeats were selected or discarded to improve scores. The small, previously
+studied Python sample does not establish generalization.
+
+## Validation and recoveries
+
+Before paid trials, corrected the timing classifier to count `jg files` as work
+and recognize searches with `--exclude`. Added cases failed against the old
+classifier; all 30 harness tests passed in Docker after the fix. Independent
+review found no actionable issues. Both arms freeze the same corrected runner.
+An earlier no-call plan was retained as superseded.
+
+Requests initially hit an official-grader platform error: a missing local x86
+image tag caused an ARM64 pull. Restored the tag from the pinned AMD64 digest and
+regraded the unchanged patches under new run IDs. Both passed; initial error
+reports and recovery receipts remain available. No coding attempt was rerun.
+One candidate Gateway charge became available on a single accounting retry.
+
+## Next experiment
+
+Keep the merged skill. A useful next step is a predeclared expansion to more
+matched SWE-bench tasks, plus a separate native Claude trigger test with its
+usual competing skills. Do not infer a broad 25% saving or a solve-rate gain from
+this pilot. No additional runs are pending.