Skip to content

Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.

Include Jev in benchmark task costs and record 0.4.3 rerun

David Zhangadbea4cc8560
diff --git a/README.md b/README.md
index 9e09166..cd9763a 100644
--- a/README.md
+++ b/README.md
@@ -113,6 +113,10 @@ for per-task costs, artifact identities and limitations. A separate
[speed study](evals/results/speed-2026-09-28.md) measures the follow-up local
optimizations with Jev’s native TypeSafe endpoint.
+The [latest total-cost rerun](evals/results/total-cost-2026-09-28.md), including
+Jev, measured **25.8% lower total cost with the same 8/10 tasks solved**.
+The older ~30% graphic above reports Sol-only cost. Future benchmark totals include Jev.
+
## Source, credentials, and local state
Searches send eligible source content to Jev through the provider selected during auth. Default
diff --git a/evals/README.md b/evals/README.md
index 31ed49c..5917ad9 100644
--- a/evals/README.md
+++ b/evals/README.md
@@ -17,7 +17,7 @@ context gaps with ordinary tools.
[Cost and quality policy](cost-quality-policy.md) owns acceptance. Count the full
coding-agent task, including implementation, verification and failed attempts.
-Jev cost and tokens are excluded. Unknown bills cannot count as cost wins. Timing
+Total cost includes Jev; Jev tokens are reported separately from agent tokens. Unknown bills cannot count as cost wins. Timing
is diagnostic; report solve outcomes alongside cost changes. The tuned
Python cohort does not prove untouched-task or language-wide generalization.
@@ -32,3 +32,5 @@ superseded official experiments in Git without adding obsolete runners to `main`
The [final product research report](results/relevance-threshold-2026-09-27.md) records the neutral-skill cohort and parameter-effect map. The earlier [source-first cohort](results/source-first-2026-09-27.md) retains its separate package and skill identity.
The [speed study](results/speed-2026-09-28.md) compares the frozen local optimizations and native TypeSafe route against that previous cohort and the same saved no-Jev baselines.
+
+The [total-cost rerun](results/total-cost-2026-09-28.md) uses published 0.4.3 and includes Jev costs, the default for all future runs. Historical reports retain their original accounting basis.
diff --git a/evals/accounting.md b/evals/accounting.md
index 7899221..01c091b 100644
--- a/evals/accounting.md
+++ b/evals/accounting.md
@@ -4,9 +4,11 @@ The required outcome is equal or better official task solve rate with lower cost
and elapsed time. Compare matched tasks within each engine before aggregating.
A cheaper attempt is not a win if it lowers solve rate.
-Jev costs zero by explicit user instruction. Record its requests, bytes, tokens
-and latency separately as diagnostics; never add Jev tokens to coding-agent
-token totals. The context consumed by the coding agent still contributes to
+Task cost includes coding-agent and Jev costs by default. Report each component
+and their sum. Native TypeSafe Jev is estimated from recorded input usage at the
+public rate retained with the frozen runner; output is free under the current
+Jev 1.13 rate. Missing usage leaves the total unknown. Record Jev requests, bytes,
+tokens and latency separately; never add Jev tokens to coding-agent token totals. The context consumed by the coding agent still contributes to
that agent's usage, and retrieval latency remains inside task wall time.
Report native task-cost telemetry when available, labelled as telemetry rather
diff --git a/evals/cost-quality-policy.md b/evals/cost-quality-policy.md
index 3daadf5..0c06aa7 100644
--- a/evals/cost-quality-policy.md
+++ b/evals/cost-quality-policy.md
@@ -1,9 +1,9 @@
-# Product evaluation: task completion and agent cost
+# Product evaluation: task completion and total cost
Official SWE-bench task completion is the primary outcome. Compare a candidate
with the preceding product version's saved benchmark evidence on matched tasks,
and identify exactly which artifacts and protocols were measured. Report the
-full coding-agent cost per attempted task alongside quality; increased cost may
+full task cost (coding agent plus Jev) per attempted task alongside quality; increased cost may
be an acceptable tradeoff for more completed tasks. Old spike cost-win counts,
parity assertions and supplemental compatibility checks are not acceptance gates.
@@ -25,8 +25,11 @@ from confirmation runs; do not select favorable repetitions or combine the best
result from different candidates for each task. Partial cohorts remain explicitly
partial. An additional failure does not erase already verified solves.
-Jev tokens and cost are excluded from the scored coding-agent bill. Retrieved
-source does count toward the coding agent's context cost. Include unsuccessful
+Total scored task cost includes both coding-agent and Jev charges by default.
+Keep their cost components separate. Native TypeSafe usage is priced at the
+retained public model rate and clearly labelled as an estimate; Gateway metadata
+is reported cost, not an invoice. Jev tokens remain separate from coding-agent
+token totals. Retrieved source counts toward the coding agent's context cost. Include unsuccessful
attempts; unknown charges are unknown, never zero. Preserve infrastructure failures
and investigate them separately from product failures. Supplemental observations
can explain behavior but do not rewrite official solve results.
diff --git a/evals/implementation/swebench/installed.md b/evals/implementation/swebench/installed.md
index 2201bda..9cf090f 100644
--- a/evals/implementation/swebench/installed.md
+++ b/evals/implementation/swebench/installed.md
@@ -31,8 +31,9 @@ python3 evals/implementation/swebench/installed.py run \
Preparation and dry-run make no model calls. `prepare` defaults to all ten tasks;
`--jev-provider typesafe` selects native Jev for a separately identified provider
experiment. Sol continues through Gateway. The broker holds both real credentials;
-the agent receives only its scoped broker token. Native Jev billing is unknown
-without Gateway metadata and remains excluded from scored Sol cost.
+the agent receives only its scoped broker token. Native Jev cost is estimated from recorded input usage at the documented
+list price frozen in the runner. Total task cost includes both Sol and Jev.
+Missing usage or incomplete transport coverage leaves that total unknown.
`--task` selects one registered task. `--evidence-root` locates the retained
baseline and evaluator files when preparing from another checkout. The root
`eval:swebench` alias points to this entry point; `--help` lists its arguments.
diff --git a/evals/implementation/swebench/installed.py b/evals/implementation/swebench/installed.py
index 639c907..0bd3032 100644
--- a/evals/implementation/swebench/installed.py
+++ b/evals/implementation/swebench/installed.py
@@ -597,8 +597,8 @@ def valid_cost(value):
return type(value) in (int, float) and math.isfinite(value) and value >= 0
-def observed_jev(out, events, log_valid, traces_copied):
- """Sum response-level gateway.cost once; provider retries are counts, not extra prices."""
+def observed_jev(out, events, log_valid, traces_copied, provider="vercel"):
+ """Sum Gateway costs or estimate native input usage at the retained public rate."""
starts = [event for event in events if event.get('kind') == 'jev-request-start']
ids = [event.get('requestId') for event in starts]
safe_ids = {identifier for identifier in ids if isinstance(identifier, str) and re.fullmatch(r'[a-f0-9]{32}', identifier)}
@@ -611,6 +611,10 @@ def observed_jev(out, events, log_valid, traces_copied):
coverage = (log_valid and traces_copied is True and len(safe_ids) == len(ids) and
safe_ids == set(responses) == set(requests) == safe_end_ids and len(safe_end_ids) == len(ends) == len(ids))
costs, inputs, outputs, attempts = [], [], [], []
+ native_rate = Decimal('0.042') / 1_000_000
+ pricing = {'source': 'https://docs.typesafe.ai/models', 'verified_date': '2026-09-28',
+ 'model': 'jev-1.13.0', 'input_usd_per_million': 0.042, 'output_usd_per_million': 0,
+ 'basis': 'Public list-price estimate; no free-credit or negotiated discounts applied'} if provider == 'typesafe' else None
def count(value):
return type(value) is int and value >= 0
# Retained response metadata remains known even if transport logs were lost.
@@ -625,6 +629,10 @@ def observed_jev(out, events, log_valid, traces_copied):
provider_attempts = gateway.get('routing', {}).get('totalProviderAttemptCount')
if count(provider_attempts): attempts.append(provider_attempts)
value = gateway.get('cost')
+ if provider == 'typesafe':
+ value = None
+ if body.get('model') == 'jev-1.13.0' and count(usage.get('input_tokens')):
+ value = str(Decimal(usage['input_tokens']) * native_rate)
# Billing can be absent while usage and transport coverage remain known.
try:
if type(value) not in (str, int, float): raise ValueError('Missing cost')
@@ -648,8 +656,9 @@ def observed_jev(out, events, log_valid, traces_copied):
if not math.isfinite(known):
known = None
complete = False
- return {'basis': 'Retained response provider_metadata.gateway.cost or historical providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation',
- 'included_in_scored_task_cost': False, 'client_calls': len(ids), 'responses_with_cost': len(costs),
+ return {'basis': 'Native input usage at retained list price' if pricing else 'Retained response Gateway cost metadata; not invoice reconciliation',
+ 'pricing': pricing, 'cost_kind': 'estimate' if pricing else 'reported',
+ 'included_in_scored_task_cost': True, 'client_calls': len(ids), 'responses_with_cost': len(costs),
'complete': complete, 'observed_cost_usd': known if complete else None, 'known_cost_usd': known,
'input_tokens': sum(inputs) if coverage and len(inputs) == len(ids) else None,
'output_tokens': sum(outputs) if coverage and len(outputs) == len(ids) else None,
@@ -695,13 +704,17 @@ def account(args):
write_json(path, lookups)
complete = log_valid and bool(identifiers) and len(request_ids) == len(starts) == len(ends) == len(identifiers) == len(lookups) and request_ids == {event.get('requestId') for event in ends} == {event.get('requestId') for event in observations} and all(event.get('streamTerminal') == 'response.completed' and not event.get('incompleteStream') and not event.get('transportError') and event.get('status') == 200 for event in ends) and all(valid_cost(row.get('metadata', {}).get('total_cost')) for row in lookups)
known = sum(row.get('metadata', {}).get('total_cost', 0) for row in lookups if valid_cost(row.get('metadata', {}).get('total_cost')))
- result = {'cell': plan['cell']['id'], 'request_starts': len(starts), 'generation_ids': len(identifiers), 'all_requests_accounted': complete, 'known_gateway_cost_usd': known, 'gateway_cost_usd': known if complete else None, 'baseline_cost_usd': plan['baseline_cost_usd'], 'jev': observed_jev(out, events, log_valid, receipt.get('jev_traces_copied'))}
+ result = {'cell': plan['cell']['id'], 'request_starts': len(starts), 'generation_ids': len(identifiers), 'all_requests_accounted': complete, 'known_gateway_cost_usd': known, 'gateway_cost_usd': known if complete else None, 'baseline_cost_usd': plan['baseline_cost_usd'], 'jev': observed_jev(out, events, log_valid, receipt.get('jev_traces_copied'), plan.get('jev_provider', 'vercel'))}
+ jev = result['jev']
+ result['task_cost_complete'] = complete and jev['complete']
+ result['task_cost_usd'] = known + jev['observed_cost_usd'] if result['task_cost_complete'] else None
+ result['known_task_cost_usd'] = known + (jev['known_cost_usd'] or 0)
grading = out / 'grading-receipt.json'
if grading.exists():
grade_receipt = json.loads(grading.read_text())
result['official_resolved'] = grade_receipt.get('run_id') == 'jg-' + plan['cell']['id'] and grade_receipt.get('exit_code') == 0 and grade_receipt.get('resolved_instances') == 1
result['protocol_valid'] = protocol_valid(receipt)
- result['successful_cost_win'] = complete and result['official_resolved'] and result['protocol_valid'] and known < plan['baseline_cost_usd']
+ result['successful_cost_win'] = result['task_cost_complete'] and result['official_resolved'] and result['protocol_valid'] and result['task_cost_usd'] < plan['baseline_cost_usd']
write_json(out / 'generation-accounting.json', result)
print(json.dumps(result))
@@ -728,12 +741,13 @@ def aggregate(args):
protocol = terminal and protocol_valid(receipt)
solved = graded and grading.get('resolved_instances') == 1
billed = (terminal and billing.get('cell') == item['cell']['id'] and
- billing.get('all_requests_accounted') is True and valid_cost(billing.get('gateway_cost_usd')))
- cost = billing.get('gateway_cost_usd') if billed else None
+ billing.get('task_cost_complete') is True and valid_cost(billing.get('task_cost_usd')))
+ cost = billing.get('task_cost_usd') if billed else None
rows.append({'task': item['cell']['instance_id'], 'status': receipt.get('status', 'not-started'),
'terminal': terminal, 'grading_attempted': grade_attempted, 'graded': graded, 'protocol_valid': protocol, 'official_resolved': solved, 'resolved': solved and protocol,
'baseline_resolved': item['baseline_resolved'], 'baseline_cost_usd': item['baseline_cost_usd'],
- 'fully_billed': billed, 'gateway_cost_usd': cost,
+ 'fully_billed': billed, 'task_cost_usd': cost,
+ 'gateway_cost_usd': billing.get('gateway_cost_usd') if belongs and billing.get('cell') == item['cell']['id'] else None,
'jev': billing.get('jev', {}) if belongs and billing.get('cell') == item['cell']['id'] else {},
'known_gateway_cost_usd': billing.get('known_gateway_cost_usd') if belongs and billing.get('cell') == item['cell']['id'] and valid_cost(billing.get('known_gateway_cost_usd')) else None,
'successful_cost_win': solved and protocol and billed and cost < item['baseline_cost_usd']})
@@ -746,9 +760,9 @@ def aggregate(args):
'official_solves': sum(row['official_resolved'] for row in rows),
'comparison_basis': 'Saved no-Jev baselines; descriptive only, not version acceptance',
'known_gateway_subtotal_usd': sum(row['known_gateway_cost_usd'] for row in rows if row['known_gateway_cost_usd'] is not None),
- 'fully_billed_total_usd': sum(row['gateway_cost_usd'] for row in rows) if all(row['fully_billed'] for row in rows) else None,
- 'jev': {'included_in_scored_task_cost': False,
- 'basis': 'Observed response API metadata, not invoice reconciliation',
+ 'fully_billed_total_usd': sum(row['task_cost_usd'] for row in rows) if all(row['fully_billed'] for row in rows) else None,
+ 'jev': {'included_in_scored_task_cost': True,
+ 'basis': 'Per-task reported Gateway costs or native list-price estimates; not invoice reconciliation',
'complete': all(row['jev'].get('complete') is True and valid_cost(row['jev'].get('observed_cost_usd')) for row in rows),
'known_cost_usd': sum(row['jev']['known_cost_usd'] for row in rows if valid_cost(row['jev'].get('known_cost_usd'))),
'observed_cost_usd': sum(row['jev']['observed_cost_usd'] for row in rows) if all(row['jev'].get('complete') is True and valid_cost(row['jev'].get('observed_cost_usd')) for row in rows) else None},
diff --git a/evals/implementation/swebench/test_installed.py b/evals/implementation/swebench/test_installed.py
index 016c419..ebec2e7 100644
--- a/evals/implementation/swebench/test_installed.py
+++ b/evals/implementation/swebench/test_installed.py
@@ -319,7 +319,7 @@ class InstalledTests(unittest.TestCase):
for index,item in enumerate(plan['cells']):
out=self.receipt(path,item)
runner.write_json(out/'grading-receipt.json',{'run_id':'jg-'+item['cell']['id'],'exit_code':0,'resolved_instances':int(item['baseline_resolved'])})
- runner.write_json(out/'generation-accounting.json',{'cell':item['cell']['id'],'all_requests_accounted':index<7,'gateway_cost_usd':0.01 if index<7 else None,'known_gateway_cost_usd':0.01})
+ runner.write_json(out/'generation-accounting.json',{'cell':item['cell']['id'],'all_requests_accounted':index<7,'gateway_cost_usd':0.01 if index<7 else None,'known_gateway_cost_usd':0.01,'task_cost_complete':index<7,'task_cost_usd':0.02 if index<7 else None})
with contextlib.redirect_stdout(io.StringIO()):result=runner.aggregate(argparse.Namespace(plan=path))
self.assertEqual(result['official_solves'],8);self.assertEqual(result['baseline_solves_preserved'],8)
self.assertEqual(result['fully_billed_solved_cost_wins'],7);self.assertIsNone(result['fully_billed_total_usd'])
@@ -376,7 +376,7 @@ class InstalledTests(unittest.TestCase):
self.assertTrue(result['complete']);self.assertEqual(result['observed_cost_usd'],0.03)
self.assertEqual(result['client_calls'],2);self.assertEqual(result['provider_attempts'],3)
self.assertEqual(result['input_tokens'],20);self.assertEqual(result['output_tokens'],4)
- self.assertFalse(result['included_in_scored_task_cost'])
+ self.assertTrue(result['included_in_scored_task_cost'])
def test_jev_missing_invalid_or_unfinished_responses_leave_total_unknown(self):
with tempfile.TemporaryDirectory() as temporary:
@@ -412,7 +412,7 @@ class InstalledTests(unittest.TestCase):
self.assertEqual(result['known_provider_attempts'],3)
self.assertFalse(result['complete']);self.assertIsNone(result['observed_cost_usd'])
- def test_observed_jev_cost_does_not_change_scored_sol_cost_win(self):
+ def test_jev_cost_changes_total_task_cost_win(self):
with tempfile.TemporaryDirectory() as temporary:
path,plan=self.fixture(Path(temporary));out=self.receipt(path,plan)
events=self.jev_fixture(out)
@@ -429,7 +429,8 @@ class InstalledTests(unittest.TestCase):
with patch.object(runner.urllib.request,'urlopen',side_effect=AssertionError('No network')),contextlib.redirect_stdout(io.StringIO()):
runner.account(argparse.Namespace(plan=path,task=None))
result=json.loads((out/'generation-accounting.json').read_text())
- self.assertEqual(result['gateway_cost_usd'],0.1);self.assertTrue(result['successful_cost_win'])
+ self.assertEqual(result['gateway_cost_usd'],0.1);self.assertFalse(result['successful_cost_win'])
+ self.assertAlmostEqual(result['task_cost_usd'],100.12)
self.assertEqual(result['jev']['observed_cost_usd'],100.02)
def test_native_usage_survives_missing_cost(self):
@@ -445,6 +446,21 @@ class InstalledTests(unittest.TestCase):
self.assertEqual(result['observed_cost_usd'],float(cost)*2 if cost in ('0.04','0') else None)
self.assertEqual(result['complete'],cost in ('0.04','0'))
+ def test_native_list_price_estimate_requires_full_usage_and_known_model(self):
+ with tempfile.TemporaryDirectory() as temporary:
+ root=Path(temporary);events=self.jev_fixture(root)
+ for index in range(2):
+ response=root/'jev-traces'/(str(index)*32+'.response.json')
+ runner.write_json(response,{'model':'jev-1.13.0','usage':{'input_tokens':1_000_000,'output_tokens':500}})
+ events[index*2+1]['responseBytes']=response.stat().st_size
+ result=runner.observed_jev(root,events,True,True,'typesafe')
+ self.assertTrue(result['complete']);self.assertAlmostEqual(result['observed_cost_usd'],0.084)
+ self.assertEqual(result['cost_kind'],'estimate')
+ response.write_text('{}');events[-1]['responseBytes']=response.stat().st_size
+ result=runner.observed_jev(root,events,True,True,'typesafe')
+ self.assertFalse(result['complete']);self.assertIsNone(result['observed_cost_usd'])
+ self.assertAlmostEqual(result['known_cost_usd'],0.042)
+
def test_preload_is_required_and_identity_checked(self):
with tempfile.TemporaryDirectory() as temporary:
root=Path(temporary);path,plan=self.fixture(root)
diff --git a/evals/results/total-cost-2026-09-28.json b/evals/results/total-cost-2026-09-28.json
new file mode 100644
index 0000000..c0feb25
--- /dev/null
+++ b/evals/results/total-cost-2026-09-28.json
@@ -0,0 +1,306 @@
+{
+ "completed": 10,
+ "solves": 8,
+ "total_tasks": 10,
+ "cost_complete": true,
+ "task_cost_usd": 5.6563990539999995,
+ "known_task_cost_usd": 5.6563990539999995,
+ "baseline_cost_usd": 7.622069,
+ "sol_cost_usd": 4.1779912,
+ "jev_cost_usd": 1.478407854,
+ "wall_seconds": 1488.0545216670725,
+ "rows": [
+ {
+ "name": "xarray",
+ "task": "pydata__xarray-3305",
+ "resolved": true,
+ "baseline_resolved": true,
+ "wall_seconds": 154.6805433750269,
+ "baseline_wall_seconds": 118.45186395899509,
+ "sol_tokens": 470621,
+ "sol_cost_usd": 0.4308676,
+ "jev_cost_usd": 0.270070164,
+ "jev_cost_kind": "estimate",
+ "task_cost_usd": 0.7009377640000001,
+ "known_task_cost_usd": 0.7009377640000001,
+ "cost_complete": true,
+ "baseline_cost_usd": 0.4937066,
+ "jev_input_tokens": 6430242,
+ "jev_requests": 807,
+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/xarray/attempts/pydata__xarray-3305",
+ "artifact_sha256": {
+ "receipt.json": "330ac1f0c3ffae24e1c503d625bbb8ebef2bef7f9d97ba2c7d20afb3752274fb",
+ "generation-accounting.json": "e04b2be0d437416182fb89b0cdb59d15135683844fe97bc537f1ef066503ca4e",
+ "grading-receipt.json": "ed3870236ae4ba74eaa163396c75bb12663990a7cc35d158ec36ee73c7d3f7dd",
+ "agent.patch": "90b52f301c85cc07885b661ed1e11671e065f50561ad80c56d9f404402f60a00"
+ },
+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"
+ },
+ {
+ "name": "sphinx",
+ "task": "sphinx-doc__sphinx-8638",
+ "resolved": true,
+ "baseline_resolved": true,
+ "wall_seconds": 125.24537920800503,
+ "baseline_wall_seconds": 192.01333883299958,
+ "sol_tokens": 306961,
+ "sol_cost_usd": 0.3714718,
+ "jev_cost_usd": 0.167765934,
+ "jev_cost_kind": "estimate",
+ "task_cost_usd": 0.5392377340000001,
+ "known_task_cost_usd": 0.5392377340000001,
+ "cost_complete": true,
+ "baseline_cost_usd": 1.0594724,
+ "jev_input_tokens": 3994427,
+ "jev_requests": 496,
+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/sphinx/attempts/sphinx-doc__sphinx-8638",
+ "artifact_sha256": {
+ "receipt.json": "ec9e1d9b87f97c4e313d97d00ea5965b2efa99133371c8486546f967af7b60aa",
+ "generation-accounting.json": "29eb291e0ef7568c647258481a1ad1fc61a59941aba6048145f0f753553c7c2e",
+ "grading-receipt.json": "6d242d22c60fb0b7a0f4ed1064a90e40da104c189f212753071c2c7eb6a2be8a",
+ "agent.patch": "96cca025030336d27c0ffe411f7b6c39a5f006211e059fe510c9e9ee151b4b11"
+ },
+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"
+ },
+ {
+ "name": "scikit",
+ "task": "scikit-learn__scikit-learn-13124",
+ "resolved": true,
+ "baseline_resolved": true,
+ "wall_seconds": 84.86864645901369,
+ "baseline_wall_seconds": 86.36477591699804,
+ "sol_tokens": 132856,
+ "sol_cost_usd": 0.2128774,
+ "jev_cost_usd": 0.079381092,
+ "jev_cost_kind": "estimate",
+ "task_cost_usd": 0.292258492,
+ "known_task_cost_usd": 0.292258492,
+ "cost_complete": true,
+ "baseline_cost_usd": 0.29443600000000003,
+ "jev_input_tokens": 1890026,
+ "jev_requests": 249,
+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/scikit/attempts/scikit-learn__scikit-learn-13124",
+ "artifact_sha256": {
+ "receipt.json": "6c046bfa1adf979f5e2cb87f74e82f7af0702238c5d556e4cc0112a30f757512",
+ "generation-accounting.json": "e0bd197e9e6ba1ac0b46c8b97f0b4bfe8e0a56682803d20a4ce11f3bda03017f",
+ "grading-receipt.json": "9b68f143383cbdbc572317f07d7219878b1ae25eafc98c25170e7d285f911352",
+ "agent.patch": "a4e758cc434ceba256ac6ee3e1a71cddb9f2f6064d1b5728a4b8712057ed1694"
+ },
+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"
+ },
+ {
+ "name": "django",
+ "task": "django__django-15629",
+ "resolved": true,
+ "baseline_resolved": true,
+ "wall_seconds": 367.173281000054,
+ "baseline_wall_seconds": 402.324371292023,
+ "sol_tokens": 1392546,
+ "sol_cost_usd": 1.1522126,
+ "jev_cost_usd": 0.351474984,
+ "jev_cost_kind": "estimate",
+ "task_cost_usd": 1.503687584,
+ "known_task_cost_usd": 1.503687584,
+ "cost_complete": true,
+ "baseline_cost_usd": 1.5055472,
+ "jev_input_tokens": 8368452,
+ "jev_requests": 921,
+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/django/attempts/django__django-15629",
+ "artifact_sha256": {
+ "receipt.json": "f84de8109c9170ace0117e00b70e19733f46f49d191b9af58cfaa294bfcb8bd1",
+ "generation-accounting.json": "9adfcd3a792d8d5f0c2c029675c6c469d9349043e78c7666bd5a5637c7d1f683",
+ "grading-receipt.json": "8075b7829a96865e95177459f546b51ce571ed95de392f2ec3cdbf534542e768",
+ "agent.patch": "da595f5d2309f45227bd5056a62d01a813edecc07204b13944ceec61381601f8"
+ },
+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"
+ },
+ {
+ "name": "requests",
+ "task": "psf__requests-1142",
+ "resolved": true,
+ "baseline_resolved": true,
+ "wall_seconds": 91.80947487498634,
+ "baseline_wall_seconds": 139.47062074998394,
+ "sol_tokens": 304258,
+ "sol_cost_usd": 0.336777,
+ "jev_cost_usd": 0.012791814,
+ "jev_cost_kind": "estimate",
+ "task_cost_usd": 0.349568814,
+ "known_task_cost_usd": 0.349568814,
+ "cost_complete": true,
+ "baseline_cost_usd": 0.2685004,
+ "jev_input_tokens": 304567,
+ "jev_requests": 61,
+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/requests/attempts/psf__requests-1142",
+ "artifact_sha256": {
+ "receipt.json": "1ee88e8221347b539dd3f74119cc64b0829c6d9ca60bf2a35440a441b2e4a07e",
+ "generation-accounting.json": "ac0d34d8d63864688e07d656854843870c28c18594320acfa2a9e7d2741fb09d",
+ "grading-receipt.json": "adf074584011fce78270d974a5408890b8a217910cc71c8ec43aa16641a7ba58",
+ "agent.patch": "727a41c473fe0158868794ce7ef7730e4f86dfa8d02f5f9cc28a174bf45c9dc4"
+ },
+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"
+ },
+ {
+ "name": "astropy",
+ "task": "astropy__astropy-13579",
+ "resolved": true,
+ "baseline_resolved": true,
+ "wall_seconds": 96.07057837495813,
+ "baseline_wall_seconds": 127.74519329200848,
+ "sol_tokens": 239674,
+ "sol_cost_usd": 0.3436076,
+ "jev_cost_usd": 0.150145212,
+ "jev_cost_kind": "estimate",
+ "task_cost_usd": 0.493752812,
+ "known_task_cost_usd": 0.493752812,
+ "cost_complete": true,
+ "baseline_cost_usd": 0.428631,
+ "jev_input_tokens": 3574886,
+ "jev_requests": 362,
+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/astropy/attempts/astropy__astropy-13579",
+ "artifact_sha256": {
+ "receipt.json": "785effac04b1e2a7f8ad4069a4e0b90262e7e0f4f673f3517e8e0fabc51f6dd5",
+ "generation-accounting.json": "5281b319cfc6627d687cdbd499df449beb89164bc54c8b0ec4e344f4268d463b",
+ "grading-receipt.json": "4e312c871b6e0bc1247c26f6fd08fc1372e67c9b70a63efc9a5439cafcbb1153",
+ "agent.patch": "9c41a7f485a6471914587d668f467e84ce4a7a8896c7a3c2e310bdc9b5ce54d1"
+ },
+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"
+ },
+ {
+ "name": "pytest",
+ "task": "pytest-dev__pytest-6197",
+ "resolved": true,
+ "baseline_resolved": true,
+ "wall_seconds": 130.86104795901338,
+ "baseline_wall_seconds": 596.4330365410133,
+ "sol_tokens": 357497,
+ "sol_cost_usd": 0.41693979999999997,
+ "jev_cost_usd": 0.111920088,
+ "jev_cost_kind": "estimate",
+ "task_cost_usd": 0.528859888,
+ "known_task_cost_usd": 0.528859888,
+ "cost_complete": true,
+ "baseline_cost_usd": 2.344458,
+ "jev_input_tokens": 2664764,
+ "jev_requests": 324,
+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/pytest/attempts/pytest-dev__pytest-6197",
+ "artifact_sha256": {
+ "receipt.json": "d6423cb9f05fab077b003c0270f90ee77ee46ba44bfc95adb2d615a9cb4305e7",
+ "generation-accounting.json": "a52c8c41a8dde5a3eda28e299a3bcd12eefa58d9a93bda79132c6efbcaf381c7",
+ "grading-receipt.json": "042f65c112c1a264677b1a4e87e80101f7800e02fa31de1aba1f654283d52bf2",
+ "agent.patch": "62c861ad12e469f55c4bccf4d1fde4a91c61766508ebdf6afbbf9bb8172ec382"
+ },
+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"
+ },
+ {
+ "name": "sympy",
+ "task": "sympy__sympy-16792",
+ "resolved": true,
+ "baseline_resolved": true,
+ "wall_seconds": 136.16174887504894,
+ "baseline_wall_seconds": 386.5954145830183,
+ "sol_tokens": 222856,
+ "sol_cost_usd": 0.2661758,
+ "jev_cost_usd": 0.093028194,
+ "jev_cost_kind": "estimate",
+ "task_cost_usd": 0.359203994,
+ "known_task_cost_usd": 0.359203994,
+ "cost_complete": true,
+ "baseline_cost_usd": 0.547975,
+ "jev_input_tokens": 2214957,
+ "jev_requests": 281,
+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/sympy/attempts/sympy__sympy-16792",
+ "artifact_sha256": {
+ "receipt.json": "25de9d5fe1ba8b99f23b871323f3b348861f1f4b2af99ac6eaa72f2d13b44c37",
+ "generation-accounting.json": "3b04d82a06e3eb8db8e7f28a8d062badc45b52e5f20e6b6d8467224abdd852c0",
+ "grading-receipt.json": "7510788711691ecc06a7c925a2fbf2f89f01a8c06503b414beddf86f86a9f8d2",
+ "agent.patch": "ceb80f479dda000b0353e28ec59badf3c1fab453e198b639a5de5411698ddef8"
+ },
+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"
+ },
+ {
+ "name": "matplotlib",
+ "task": "matplotlib__matplotlib-26466",
+ "resolved": false,
+ "baseline_resolved": false,
+ "wall_seconds": 211.51842320797732,
+ "baseline_wall_seconds": 106.53679849999025,
+ "sol_tokens": 439949,
+ "sol_cost_usd": 0.4214364,
+ "jev_cost_usd": 0.175165788,
+ "jev_cost_kind": "estimate",
+ "task_cost_usd": 0.596602188,
+ "known_task_cost_usd": 0.596602188,
+ "cost_complete": true,
+ "baseline_cost_usd": 0.395716,
+ "jev_input_tokens": 4170614,
+ "jev_requests": 650,
+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/matplotlib/attempts/matplotlib__matplotlib-26466",
+ "artifact_sha256": {
+ "receipt.json": "134a00e112549c3707ed84d32f70d2d9766375747b92d3a1b37e5834eb3db6bc",
+ "generation-accounting.json": "2a8e9e275bcd2a07ab75321792f06389bcbb663808794a3d5b475d675f52d230",
+ "grading-receipt.json": "93e2af75254d0ef1e4bc5ce4c8c7675fd907f7407683c51aa234249ae97b1a89",
+ "agent.patch": "6781334e557559eebdf035ea3473ad999c560e6b608daf0e89b364cf7402ca69"
+ },
+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"
+ },
+ {
+ "name": "pylint",
+ "task": "pylint-dev__pylint-4604",
+ "resolved": false,
+ "baseline_resolved": false,
+ "wall_seconds": 89.66539833298884,
+ "baseline_wall_seconds": 74.52734654099913,
+ "sol_tokens": 167241,
+ "sol_cost_usd": 0.2256252,
+ "jev_cost_usd": 0.066664584,
+ "jev_cost_kind": "estimate",
+ "task_cost_usd": 0.292289784,
+ "known_task_cost_usd": 0.292289784,
+ "cost_complete": true,
+ "baseline_cost_usd": 0.2836264,
+ "jev_input_tokens": 1587252,
+ "jev_requests": 203,
+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/pylint/attempts/pylint-dev__pylint-4604",
+ "artifact_sha256": {
+ "receipt.json": "b051ab9bdcd6b19c09b74209fefe767758d7bb08e6a000900da37a38436f9ec2",
+ "generation-accounting.json": "74052b7998cf38bee8048a2579bc59b570b7d00ba269a86b6543c4f17d438b14",
+ "grading-receipt.json": "f76fa47afc5af114d2e46845a481242cdc3f067320bd59330274e186c6800769",
+ "agent.patch": "b0976f797af371069b62e0bef425caf38821e20ce70f6271c500aede163aba34"
+ },
+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"
+ }
+ ],
+ "contract": {
+ "package": "@dzhng/[email protected]",
+ "package_sha256": "4b36178c728ee89859b8f34d083351b0f57d0527e2b22140bb8b81728fdd23b3",
+ "tasks": [
+ "xarray",
+ "sphinx",
+ "scikit",
+ "django",
+ "requests",
+ "astropy",
+ "pytest",
+ "sympy",
+ "matplotlib",
+ "pylint"
+ ],
+ "provider": "typesafe",
+ "model": "openai/gpt-5.6-sol",
+ "effort": "medium",
+ "cost": "Sol reported cost plus native Jev usage times $0.042/M input; output free; missing coverage means unknown total",
+ "price_source": "https://docs.typesafe.ai/models",
+ "price_verified": "2026-09-28",
+ "baselines": "Reuse fixed saved baselines; no baseline runs",
+ "attempts": "One fresh first attempt per task, no repair or best-of selection",
+ "skill": "Exact public skill shipped in registry package",
+ "prior_comparison": "speed-2026-09-28 retained separately under original Sol-only policy"
+ },
+ "pricing_document_sha256": "9d20bb3c90a0147532d0b20ddc4c64579391be7842e965543b27bad684eeb4d6",
+ "limitations": "Single fresh repeat of ten tuned Python tasks; not a holdout or variance estimate. Jev is estimated at public list price; Sol is provider-reported cost, not invoice reconciliation. All attempts included; baseline never rerun.",
+ "savings_percent": 25.78919117630659,
+ "cost_per_attempt_usd": 0.5656399054,
+ "cost_per_solve_usd": 0.7070498817499999,
+ "baseline_cost_per_solve_usd": 0.952758625
+}
diff --git a/evals/results/total-cost-2026-09-28.md b/evals/results/total-cost-2026-09-28.md
new file mode 100644
index 0000000..a329de1
--- /dev/null
+++ b/evals/results/total-cost-2026-09-28.md
@@ -0,0 +1,61 @@
+# Total task cost including Jev
+
+A fresh run of published `@dzhng/[email protected]` preserved the same **8/10 official
+SWE-bench solves** as the saved no-Jev baseline. Full task cost was **$5.6563991**:
+**$4.1779912 Sol + $1.4784079 Jev**, versus **$7.6220690** without Jevgrep.
+That is **25.8% lower total cost**, including both unresolved attempts.
+
+| Task | Official outcome | Sol reported cost | Jev estimate | Combined | No-Jev baseline |
+| --- | --- | ---: | ---: | ---: | ---: |
+| `pydata__xarray-3305` | pass | $0.4309 | $0.2701 | $0.7009 | $0.4937 |
+| `sphinx-doc__sphinx-8638` | pass | $0.3715 | $0.1678 | $0.5392 | $1.0595 |
+| `scikit-learn__scikit-learn-13124` | pass | $0.2129 | $0.0794 | $0.2923 | $0.2944 |
+| `django__django-15629` | pass | $1.1522 | $0.3515 | $1.5037 | $1.5055 |
+| `psf__requests-1142` | pass | $0.3368 | $0.0128 | $0.3496 | $0.2685 |
+| `astropy__astropy-13579` | pass | $0.3436 | $0.1501 | $0.4938 | $0.4286 |
+| `pytest-dev__pytest-6197` | pass | $0.4169 | $0.1119 | $0.5289 | $2.3445 |
+| `sympy__sympy-16792` | pass | $0.2662 | $0.0930 | $0.3592 | $0.5480 |
+| `matplotlib__matplotlib-26466` | unresolved | $0.4214 | $0.1752 | $0.5966 | $0.3957 |
+| `pylint-dev__pylint-4604` | unresolved | $0.2256 | $0.0667 | $0.2923 | $0.2836 |
+| **Total** | **8/10** | **$4.1780** | **$1.4784** | **$5.6564** | **$7.6221** |
+
+Cost per attempted task is **$0.5656 versus $0.7622**. Total spend divided by
+solved tasks is **$0.7070 versus $0.9528**. This includes spend on failed tasks;
+it is not the mean cost of only the successful attempts.
+
+## Accounting basis
+
+Total task cost includes both Sol and Jev by default. The
+[accounting policy](../accounting.md) governs future evaluations. Jev tokens are
+still shown separately from coding-agent tokens, and returned context contributes
+to Sol's bill.
+
+Sol charges come from retained Gateway generation metadata. Native TypeSafe Jev
+cost is a **list-price estimate**, calculated from response `usage.input_tokens`
+at **$0.042 per million input tokens** for `jev-1.13.0`; output tokens are free.
+The rate comes from [TypeSafe's model documentation](https://docs.typesafe.ai/models),
+retrieved on September 28, 2026 and retained with the run. It does not subtract
+free credits or assume negotiated discounts. Neither component is invoice
+reconciliation. All ten attempts have complete transport and usage coverage;
+missing coverage would make the combined total unknown rather than zero.
+
+The [machine-readable report](total-cost-2026-09-28.json) retains package and
+receipt hashes, price provenance, per-task components, runtime, token usage and
+links to raw evidence. Raw traces and the frozen pricing document remain in
+ignored `evals/runs/swebench/total-cost-2026-09-28/` storage.
+
+## Protocol and limits
+
+This is one fresh first attempt per task with the published package and its exact
+public skill, native TypeSafe retrieval, and `openai/gpt-5.6-sol` at medium effort.
+Initial Jevgrep use remains required; agents can then use ordinary tools. Tasks
+ran sequentially. Saved no-Jev baselines were reused, never rerun. Official
+SWE-bench grading alone determines the outcomes; Matplotlib and Pylint remained
+unresolved. Every attempt and its costs count.
+
+This is a repeat of ten tuned Python tasks, not a holdout or a variance estimate.
+Sol used different queries and trajectories than the earlier run. Consequently,
+this report is a new measurement, not merely the old bill with Jev added. The
+[previous speed cohort](speed-2026-09-28.md) retains its original Sol-only totals
+for historical comparison. Its advertised savings should not be read as including
+Jev. No product code or skill was tuned during this repeat.