Find code by asking what it does. A CLI for coding agents that uses Jev to discover relevant files and source context.
TypeScript ai-sdkclaude-codeclicode-searchcodexcoding-agentscontext-retrievaldeveloper-toolsjevsemantic-searchtypescriptvercel-ai-gateway
Include Jev in benchmark task costs and record 0.4.3 rerun
David Zhangadbea4cc8560
diff --git a/README.md b/README.mdindex 9e09166..cd9763a 100644--- a/README.md+++ b/README.md@@ -113,6 +113,10 @@ for per-task costs, artifact identities and limitations. A separate[speed study](evals/results/speed-2026-09-28.md) measures the follow-up localoptimizations with Jev’s native TypeSafe endpoint.+The [latest total-cost rerun](evals/results/total-cost-2026-09-28.md), including+Jev, measured **25.8% lower total cost with the same 8/10 tasks solved**.+The older ~30% graphic above reports Sol-only cost. Future benchmark totals include Jev.+## Source, credentials, and local stateSearches send eligible source content to Jev through the provider selected during auth. Defaultdiff --git a/evals/README.md b/evals/README.mdindex 31ed49c..5917ad9 100644--- a/evals/README.md+++ b/evals/README.md@@ -17,7 +17,7 @@ context gaps with ordinary tools.[Cost and quality policy](cost-quality-policy.md) owns acceptance. Count the fullcoding-agent task, including implementation, verification and failed attempts.-Jev cost and tokens are excluded. Unknown bills cannot count as cost wins. Timing+Total cost includes Jev; Jev tokens are reported separately from agent tokens. Unknown bills cannot count as cost wins. Timingis diagnostic; report solve outcomes alongside cost changes. The tunedPython cohort does not prove untouched-task or language-wide generalization.@@ -32,3 +32,5 @@ superseded official experiments in Git without adding obsolete runners to `main`The [final product research report](results/relevance-threshold-2026-09-27.md) records the neutral-skill cohort and parameter-effect map. The earlier [source-first cohort](results/source-first-2026-09-27.md) retains its separate package and skill identity.The [speed study](results/speed-2026-09-28.md) compares the frozen local optimizations and native TypeSafe route against that previous cohort and the same saved no-Jev baselines.++The [total-cost rerun](results/total-cost-2026-09-28.md) uses published 0.4.3 and includes Jev costs, the default for all future runs. Historical reports retain their original accounting basis.diff --git a/evals/accounting.md b/evals/accounting.mdindex 7899221..01c091b 100644--- a/evals/accounting.md+++ b/evals/accounting.md@@ -4,9 +4,11 @@ The required outcome is equal or better official task solve rate with lower costand elapsed time. Compare matched tasks within each engine before aggregating.A cheaper attempt is not a win if it lowers solve rate.-Jev costs zero by explicit user instruction. Record its requests, bytes, tokens-and latency separately as diagnostics; never add Jev tokens to coding-agent-token totals. The context consumed by the coding agent still contributes to+Task cost includes coding-agent and Jev costs by default. Report each component+and their sum. Native TypeSafe Jev is estimated from recorded input usage at the+public rate retained with the frozen runner; output is free under the current+Jev 1.13 rate. Missing usage leaves the total unknown. Record Jev requests, bytes,+tokens and latency separately; never add Jev tokens to coding-agent token totals. The context consumed by the coding agent still contributes tothat agent's usage, and retrieval latency remains inside task wall time.Report native task-cost telemetry when available, labelled as telemetry ratherdiff --git a/evals/cost-quality-policy.md b/evals/cost-quality-policy.mdindex 3daadf5..0c06aa7 100644--- a/evals/cost-quality-policy.md+++ b/evals/cost-quality-policy.md@@ -1,9 +1,9 @@-# Product evaluation: task completion and agent cost+# Product evaluation: task completion and total costOfficial SWE-bench task completion is the primary outcome. Compare a candidatewith the preceding product version's saved benchmark evidence on matched tasks,and identify exactly which artifacts and protocols were measured. Report the-full coding-agent cost per attempted task alongside quality; increased cost may+full task cost (coding agent plus Jev) per attempted task alongside quality; increased cost maybe an acceptable tradeoff for more completed tasks. Old spike cost-win counts,parity assertions and supplemental compatibility checks are not acceptance gates.@@ -25,8 +25,11 @@ from confirmation runs; do not select favorable repetitions or combine the bestresult from different candidates for each task. Partial cohorts remain explicitlypartial. An additional failure does not erase already verified solves.-Jev tokens and cost are excluded from the scored coding-agent bill. Retrieved-source does count toward the coding agent's context cost. Include unsuccessful+Total scored task cost includes both coding-agent and Jev charges by default.+Keep their cost components separate. Native TypeSafe usage is priced at the+retained public model rate and clearly labelled as an estimate; Gateway metadata+is reported cost, not an invoice. Jev tokens remain separate from coding-agent+token totals. Retrieved source counts toward the coding agent's context cost. Include unsuccessfulattempts; unknown charges are unknown, never zero. Preserve infrastructure failuresand investigate them separately from product failures. Supplemental observationscan explain behavior but do not rewrite official solve results.diff --git a/evals/implementation/swebench/installed.md b/evals/implementation/swebench/installed.mdindex 2201bda..9cf090f 100644--- a/evals/implementation/swebench/installed.md+++ b/evals/implementation/swebench/installed.md@@ -31,8 +31,9 @@ python3 evals/implementation/swebench/installed.py run \Preparation and dry-run make no model calls. `prepare` defaults to all ten tasks;`--jev-provider typesafe` selects native Jev for a separately identified providerexperiment. Sol continues through Gateway. The broker holds both real credentials;-the agent receives only its scoped broker token. Native Jev billing is unknown-without Gateway metadata and remains excluded from scored Sol cost.+the agent receives only its scoped broker token. Native Jev cost is estimated from recorded input usage at the documented+list price frozen in the runner. Total task cost includes both Sol and Jev.+Missing usage or incomplete transport coverage leaves that total unknown.`--task` selects one registered task. `--evidence-root` locates the retainedbaseline and evaluator files when preparing from another checkout. The root`eval:swebench` alias points to this entry point; `--help` lists its arguments.diff --git a/evals/implementation/swebench/installed.py b/evals/implementation/swebench/installed.pyindex 639c907..0bd3032 100644--- a/evals/implementation/swebench/installed.py+++ b/evals/implementation/swebench/installed.py@@ -597,8 +597,8 @@ def valid_cost(value):return type(value) in (int, float) and math.isfinite(value) and value >= 0-def observed_jev(out, events, log_valid, traces_copied):- """Sum response-level gateway.cost once; provider retries are counts, not extra prices."""+def observed_jev(out, events, log_valid, traces_copied, provider="vercel"):+ """Sum Gateway costs or estimate native input usage at the retained public rate."""starts = [event for event in events if event.get('kind') == 'jev-request-start']ids = [event.get('requestId') for event in starts]safe_ids = {identifier for identifier in ids if isinstance(identifier, str) and re.fullmatch(r'[a-f0-9]{32}', identifier)}@@ -611,6 +611,10 @@ def observed_jev(out, events, log_valid, traces_copied):coverage = (log_valid and traces_copied is True and len(safe_ids) == len(ids) andsafe_ids == set(responses) == set(requests) == safe_end_ids and len(safe_end_ids) == len(ends) == len(ids))costs, inputs, outputs, attempts = [], [], [], []+ native_rate = Decimal('0.042') / 1_000_000+ pricing = {'source': 'https://docs.typesafe.ai/models', 'verified_date': '2026-09-28',+ 'model': 'jev-1.13.0', 'input_usd_per_million': 0.042, 'output_usd_per_million': 0,+ 'basis': 'Public list-price estimate; no free-credit or negotiated discounts applied'} if provider == 'typesafe' else Nonedef count(value):return type(value) is int and value >= 0# Retained response metadata remains known even if transport logs were lost.@@ -625,6 +629,10 @@ def observed_jev(out, events, log_valid, traces_copied):provider_attempts = gateway.get('routing', {}).get('totalProviderAttemptCount')if count(provider_attempts): attempts.append(provider_attempts)value = gateway.get('cost')+ if provider == 'typesafe':+ value = None+ if body.get('model') == 'jev-1.13.0' and count(usage.get('input_tokens')):+ value = str(Decimal(usage['input_tokens']) * native_rate)# Billing can be absent while usage and transport coverage remain known.try:if type(value) not in (str, int, float): raise ValueError('Missing cost')@@ -648,8 +656,9 @@ def observed_jev(out, events, log_valid, traces_copied):if not math.isfinite(known):known = Nonecomplete = False- return {'basis': 'Retained response provider_metadata.gateway.cost or historical providerMetadata.gateway.cost; observed API metadata, not invoice reconciliation',- 'included_in_scored_task_cost': False, 'client_calls': len(ids), 'responses_with_cost': len(costs),+ return {'basis': 'Native input usage at retained list price' if pricing else 'Retained response Gateway cost metadata; not invoice reconciliation',+ 'pricing': pricing, 'cost_kind': 'estimate' if pricing else 'reported',+ 'included_in_scored_task_cost': True, 'client_calls': len(ids), 'responses_with_cost': len(costs),'complete': complete, 'observed_cost_usd': known if complete else None, 'known_cost_usd': known,'input_tokens': sum(inputs) if coverage and len(inputs) == len(ids) else None,'output_tokens': sum(outputs) if coverage and len(outputs) == len(ids) else None,@@ -695,13 +704,17 @@ def account(args):write_json(path, lookups)complete = log_valid and bool(identifiers) and len(request_ids) == len(starts) == len(ends) == len(identifiers) == len(lookups) and request_ids == {event.get('requestId') for event in ends} == {event.get('requestId') for event in observations} and all(event.get('streamTerminal') == 'response.completed' and not event.get('incompleteStream') and not event.get('transportError') and event.get('status') == 200 for event in ends) and all(valid_cost(row.get('metadata', {}).get('total_cost')) for row in lookups)known = sum(row.get('metadata', {}).get('total_cost', 0) for row in lookups if valid_cost(row.get('metadata', {}).get('total_cost')))- result = {'cell': plan['cell']['id'], 'request_starts': len(starts), 'generation_ids': len(identifiers), 'all_requests_accounted': complete, 'known_gateway_cost_usd': known, 'gateway_cost_usd': known if complete else None, 'baseline_cost_usd': plan['baseline_cost_usd'], 'jev': observed_jev(out, events, log_valid, receipt.get('jev_traces_copied'))}+ result = {'cell': plan['cell']['id'], 'request_starts': len(starts), 'generation_ids': len(identifiers), 'all_requests_accounted': complete, 'known_gateway_cost_usd': known, 'gateway_cost_usd': known if complete else None, 'baseline_cost_usd': plan['baseline_cost_usd'], 'jev': observed_jev(out, events, log_valid, receipt.get('jev_traces_copied'), plan.get('jev_provider', 'vercel'))}+ jev = result['jev']+ result['task_cost_complete'] = complete and jev['complete']+ result['task_cost_usd'] = known + jev['observed_cost_usd'] if result['task_cost_complete'] else None+ result['known_task_cost_usd'] = known + (jev['known_cost_usd'] or 0)grading = out / 'grading-receipt.json'if grading.exists():grade_receipt = json.loads(grading.read_text())result['official_resolved'] = grade_receipt.get('run_id') == 'jg-' + plan['cell']['id'] and grade_receipt.get('exit_code') == 0 and grade_receipt.get('resolved_instances') == 1result['protocol_valid'] = protocol_valid(receipt)- result['successful_cost_win'] = complete and result['official_resolved'] and result['protocol_valid'] and known < plan['baseline_cost_usd']+ result['successful_cost_win'] = result['task_cost_complete'] and result['official_resolved'] and result['protocol_valid'] and result['task_cost_usd'] < plan['baseline_cost_usd']write_json(out / 'generation-accounting.json', result)print(json.dumps(result))@@ -728,12 +741,13 @@ def aggregate(args):protocol = terminal and protocol_valid(receipt)solved = graded and grading.get('resolved_instances') == 1billed = (terminal and billing.get('cell') == item['cell']['id'] and- billing.get('all_requests_accounted') is True and valid_cost(billing.get('gateway_cost_usd')))- cost = billing.get('gateway_cost_usd') if billed else None+ billing.get('task_cost_complete') is True and valid_cost(billing.get('task_cost_usd')))+ cost = billing.get('task_cost_usd') if billed else Nonerows.append({'task': item['cell']['instance_id'], 'status': receipt.get('status', 'not-started'),'terminal': terminal, 'grading_attempted': grade_attempted, 'graded': graded, 'protocol_valid': protocol, 'official_resolved': solved, 'resolved': solved and protocol,'baseline_resolved': item['baseline_resolved'], 'baseline_cost_usd': item['baseline_cost_usd'],- 'fully_billed': billed, 'gateway_cost_usd': cost,+ 'fully_billed': billed, 'task_cost_usd': cost,+ 'gateway_cost_usd': billing.get('gateway_cost_usd') if belongs and billing.get('cell') == item['cell']['id'] else None,'jev': billing.get('jev', {}) if belongs and billing.get('cell') == item['cell']['id'] else {},'known_gateway_cost_usd': billing.get('known_gateway_cost_usd') if belongs and billing.get('cell') == item['cell']['id'] and valid_cost(billing.get('known_gateway_cost_usd')) else None,'successful_cost_win': solved and protocol and billed and cost < item['baseline_cost_usd']})@@ -746,9 +760,9 @@ def aggregate(args):'official_solves': sum(row['official_resolved'] for row in rows),'comparison_basis': 'Saved no-Jev baselines; descriptive only, not version acceptance','known_gateway_subtotal_usd': sum(row['known_gateway_cost_usd'] for row in rows if row['known_gateway_cost_usd'] is not None),- 'fully_billed_total_usd': sum(row['gateway_cost_usd'] for row in rows) if all(row['fully_billed'] for row in rows) else None,- 'jev': {'included_in_scored_task_cost': False,- 'basis': 'Observed response API metadata, not invoice reconciliation',+ 'fully_billed_total_usd': sum(row['task_cost_usd'] for row in rows) if all(row['fully_billed'] for row in rows) else None,+ 'jev': {'included_in_scored_task_cost': True,+ 'basis': 'Per-task reported Gateway costs or native list-price estimates; not invoice reconciliation','complete': all(row['jev'].get('complete') is True and valid_cost(row['jev'].get('observed_cost_usd')) for row in rows),'known_cost_usd': sum(row['jev']['known_cost_usd'] for row in rows if valid_cost(row['jev'].get('known_cost_usd'))),'observed_cost_usd': sum(row['jev']['observed_cost_usd'] for row in rows) if all(row['jev'].get('complete') is True and valid_cost(row['jev'].get('observed_cost_usd')) for row in rows) else None},diff --git a/evals/implementation/swebench/test_installed.py b/evals/implementation/swebench/test_installed.pyindex 016c419..ebec2e7 100644--- a/evals/implementation/swebench/test_installed.py+++ b/evals/implementation/swebench/test_installed.py@@ -319,7 +319,7 @@ class InstalledTests(unittest.TestCase):for index,item in enumerate(plan['cells']):out=self.receipt(path,item)runner.write_json(out/'grading-receipt.json',{'run_id':'jg-'+item['cell']['id'],'exit_code':0,'resolved_instances':int(item['baseline_resolved'])})- runner.write_json(out/'generation-accounting.json',{'cell':item['cell']['id'],'all_requests_accounted':index<7,'gateway_cost_usd':0.01 if index<7 else None,'known_gateway_cost_usd':0.01})+ runner.write_json(out/'generation-accounting.json',{'cell':item['cell']['id'],'all_requests_accounted':index<7,'gateway_cost_usd':0.01 if index<7 else None,'known_gateway_cost_usd':0.01,'task_cost_complete':index<7,'task_cost_usd':0.02 if index<7 else None})with contextlib.redirect_stdout(io.StringIO()):result=runner.aggregate(argparse.Namespace(plan=path))self.assertEqual(result['official_solves'],8);self.assertEqual(result['baseline_solves_preserved'],8)self.assertEqual(result['fully_billed_solved_cost_wins'],7);self.assertIsNone(result['fully_billed_total_usd'])@@ -376,7 +376,7 @@ class InstalledTests(unittest.TestCase):self.assertTrue(result['complete']);self.assertEqual(result['observed_cost_usd'],0.03)self.assertEqual(result['client_calls'],2);self.assertEqual(result['provider_attempts'],3)self.assertEqual(result['input_tokens'],20);self.assertEqual(result['output_tokens'],4)- self.assertFalse(result['included_in_scored_task_cost'])+ self.assertTrue(result['included_in_scored_task_cost'])def test_jev_missing_invalid_or_unfinished_responses_leave_total_unknown(self):with tempfile.TemporaryDirectory() as temporary:@@ -412,7 +412,7 @@ class InstalledTests(unittest.TestCase):self.assertEqual(result['known_provider_attempts'],3)self.assertFalse(result['complete']);self.assertIsNone(result['observed_cost_usd'])- def test_observed_jev_cost_does_not_change_scored_sol_cost_win(self):+ def test_jev_cost_changes_total_task_cost_win(self):with tempfile.TemporaryDirectory() as temporary:path,plan=self.fixture(Path(temporary));out=self.receipt(path,plan)events=self.jev_fixture(out)@@ -429,7 +429,8 @@ class InstalledTests(unittest.TestCase):with patch.object(runner.urllib.request,'urlopen',side_effect=AssertionError('No network')),contextlib.redirect_stdout(io.StringIO()):runner.account(argparse.Namespace(plan=path,task=None))result=json.loads((out/'generation-accounting.json').read_text())- self.assertEqual(result['gateway_cost_usd'],0.1);self.assertTrue(result['successful_cost_win'])+ self.assertEqual(result['gateway_cost_usd'],0.1);self.assertFalse(result['successful_cost_win'])+ self.assertAlmostEqual(result['task_cost_usd'],100.12)self.assertEqual(result['jev']['observed_cost_usd'],100.02)def test_native_usage_survives_missing_cost(self):@@ -445,6 +446,21 @@ class InstalledTests(unittest.TestCase):self.assertEqual(result['observed_cost_usd'],float(cost)*2 if cost in ('0.04','0') else None)self.assertEqual(result['complete'],cost in ('0.04','0'))+ def test_native_list_price_estimate_requires_full_usage_and_known_model(self):+ with tempfile.TemporaryDirectory() as temporary:+ root=Path(temporary);events=self.jev_fixture(root)+ for index in range(2):+ response=root/'jev-traces'/(str(index)*32+'.response.json')+ runner.write_json(response,{'model':'jev-1.13.0','usage':{'input_tokens':1_000_000,'output_tokens':500}})+ events[index*2+1]['responseBytes']=response.stat().st_size+ result=runner.observed_jev(root,events,True,True,'typesafe')+ self.assertTrue(result['complete']);self.assertAlmostEqual(result['observed_cost_usd'],0.084)+ self.assertEqual(result['cost_kind'],'estimate')+ response.write_text('{}');events[-1]['responseBytes']=response.stat().st_size+ result=runner.observed_jev(root,events,True,True,'typesafe')+ self.assertFalse(result['complete']);self.assertIsNone(result['observed_cost_usd'])+ self.assertAlmostEqual(result['known_cost_usd'],0.042)+def test_preload_is_required_and_identity_checked(self):with tempfile.TemporaryDirectory() as temporary:root=Path(temporary);path,plan=self.fixture(root)diff --git a/evals/results/total-cost-2026-09-28.json b/evals/results/total-cost-2026-09-28.jsonnew file mode 100644index 0000000..c0feb25--- /dev/null+++ b/evals/results/total-cost-2026-09-28.json@@ -0,0 +1,306 @@+{+ "completed": 10,+ "solves": 8,+ "total_tasks": 10,+ "cost_complete": true,+ "task_cost_usd": 5.6563990539999995,+ "known_task_cost_usd": 5.6563990539999995,+ "baseline_cost_usd": 7.622069,+ "sol_cost_usd": 4.1779912,+ "jev_cost_usd": 1.478407854,+ "wall_seconds": 1488.0545216670725,+ "rows": [+ {+ "name": "xarray",+ "task": "pydata__xarray-3305",+ "resolved": true,+ "baseline_resolved": true,+ "wall_seconds": 154.6805433750269,+ "baseline_wall_seconds": 118.45186395899509,+ "sol_tokens": 470621,+ "sol_cost_usd": 0.4308676,+ "jev_cost_usd": 0.270070164,+ "jev_cost_kind": "estimate",+ "task_cost_usd": 0.7009377640000001,+ "known_task_cost_usd": 0.7009377640000001,+ "cost_complete": true,+ "baseline_cost_usd": 0.4937066,+ "jev_input_tokens": 6430242,+ "jev_requests": 807,+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/xarray/attempts/pydata__xarray-3305",+ "artifact_sha256": {+ "receipt.json": "330ac1f0c3ffae24e1c503d625bbb8ebef2bef7f9d97ba2c7d20afb3752274fb",+ "generation-accounting.json": "e04b2be0d437416182fb89b0cdb59d15135683844fe97bc537f1ef066503ca4e",+ "grading-receipt.json": "ed3870236ae4ba74eaa163396c75bb12663990a7cc35d158ec36ee73c7d3f7dd",+ "agent.patch": "90b52f301c85cc07885b661ed1e11671e065f50561ad80c56d9f404402f60a00"+ },+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"+ },+ {+ "name": "sphinx",+ "task": "sphinx-doc__sphinx-8638",+ "resolved": true,+ "baseline_resolved": true,+ "wall_seconds": 125.24537920800503,+ "baseline_wall_seconds": 192.01333883299958,+ "sol_tokens": 306961,+ "sol_cost_usd": 0.3714718,+ "jev_cost_usd": 0.167765934,+ "jev_cost_kind": "estimate",+ "task_cost_usd": 0.5392377340000001,+ "known_task_cost_usd": 0.5392377340000001,+ "cost_complete": true,+ "baseline_cost_usd": 1.0594724,+ "jev_input_tokens": 3994427,+ "jev_requests": 496,+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/sphinx/attempts/sphinx-doc__sphinx-8638",+ "artifact_sha256": {+ "receipt.json": "ec9e1d9b87f97c4e313d97d00ea5965b2efa99133371c8486546f967af7b60aa",+ "generation-accounting.json": "29eb291e0ef7568c647258481a1ad1fc61a59941aba6048145f0f753553c7c2e",+ "grading-receipt.json": "6d242d22c60fb0b7a0f4ed1064a90e40da104c189f212753071c2c7eb6a2be8a",+ "agent.patch": "96cca025030336d27c0ffe411f7b6c39a5f006211e059fe510c9e9ee151b4b11"+ },+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"+ },+ {+ "name": "scikit",+ "task": "scikit-learn__scikit-learn-13124",+ "resolved": true,+ "baseline_resolved": true,+ "wall_seconds": 84.86864645901369,+ "baseline_wall_seconds": 86.36477591699804,+ "sol_tokens": 132856,+ "sol_cost_usd": 0.2128774,+ "jev_cost_usd": 0.079381092,+ "jev_cost_kind": "estimate",+ "task_cost_usd": 0.292258492,+ "known_task_cost_usd": 0.292258492,+ "cost_complete": true,+ "baseline_cost_usd": 0.29443600000000003,+ "jev_input_tokens": 1890026,+ "jev_requests": 249,+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/scikit/attempts/scikit-learn__scikit-learn-13124",+ "artifact_sha256": {+ "receipt.json": "6c046bfa1adf979f5e2cb87f74e82f7af0702238c5d556e4cc0112a30f757512",+ "generation-accounting.json": "e0bd197e9e6ba1ac0b46c8b97f0b4bfe8e0a56682803d20a4ce11f3bda03017f",+ "grading-receipt.json": "9b68f143383cbdbc572317f07d7219878b1ae25eafc98c25170e7d285f911352",+ "agent.patch": "a4e758cc434ceba256ac6ee3e1a71cddb9f2f6064d1b5728a4b8712057ed1694"+ },+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"+ },+ {+ "name": "django",+ "task": "django__django-15629",+ "resolved": true,+ "baseline_resolved": true,+ "wall_seconds": 367.173281000054,+ "baseline_wall_seconds": 402.324371292023,+ "sol_tokens": 1392546,+ "sol_cost_usd": 1.1522126,+ "jev_cost_usd": 0.351474984,+ "jev_cost_kind": "estimate",+ "task_cost_usd": 1.503687584,+ "known_task_cost_usd": 1.503687584,+ "cost_complete": true,+ "baseline_cost_usd": 1.5055472,+ "jev_input_tokens": 8368452,+ "jev_requests": 921,+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/django/attempts/django__django-15629",+ "artifact_sha256": {+ "receipt.json": "f84de8109c9170ace0117e00b70e19733f46f49d191b9af58cfaa294bfcb8bd1",+ "generation-accounting.json": "9adfcd3a792d8d5f0c2c029675c6c469d9349043e78c7666bd5a5637c7d1f683",+ "grading-receipt.json": "8075b7829a96865e95177459f546b51ce571ed95de392f2ec3cdbf534542e768",+ "agent.patch": "da595f5d2309f45227bd5056a62d01a813edecc07204b13944ceec61381601f8"+ },+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"+ },+ {+ "name": "requests",+ "task": "psf__requests-1142",+ "resolved": true,+ "baseline_resolved": true,+ "wall_seconds": 91.80947487498634,+ "baseline_wall_seconds": 139.47062074998394,+ "sol_tokens": 304258,+ "sol_cost_usd": 0.336777,+ "jev_cost_usd": 0.012791814,+ "jev_cost_kind": "estimate",+ "task_cost_usd": 0.349568814,+ "known_task_cost_usd": 0.349568814,+ "cost_complete": true,+ "baseline_cost_usd": 0.2685004,+ "jev_input_tokens": 304567,+ "jev_requests": 61,+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/requests/attempts/psf__requests-1142",+ "artifact_sha256": {+ "receipt.json": "1ee88e8221347b539dd3f74119cc64b0829c6d9ca60bf2a35440a441b2e4a07e",+ "generation-accounting.json": "ac0d34d8d63864688e07d656854843870c28c18594320acfa2a9e7d2741fb09d",+ "grading-receipt.json": "adf074584011fce78270d974a5408890b8a217910cc71c8ec43aa16641a7ba58",+ "agent.patch": "727a41c473fe0158868794ce7ef7730e4f86dfa8d02f5f9cc28a174bf45c9dc4"+ },+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"+ },+ {+ "name": "astropy",+ "task": "astropy__astropy-13579",+ "resolved": true,+ "baseline_resolved": true,+ "wall_seconds": 96.07057837495813,+ "baseline_wall_seconds": 127.74519329200848,+ "sol_tokens": 239674,+ "sol_cost_usd": 0.3436076,+ "jev_cost_usd": 0.150145212,+ "jev_cost_kind": "estimate",+ "task_cost_usd": 0.493752812,+ "known_task_cost_usd": 0.493752812,+ "cost_complete": true,+ "baseline_cost_usd": 0.428631,+ "jev_input_tokens": 3574886,+ "jev_requests": 362,+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/astropy/attempts/astropy__astropy-13579",+ "artifact_sha256": {+ "receipt.json": "785effac04b1e2a7f8ad4069a4e0b90262e7e0f4f673f3517e8e0fabc51f6dd5",+ "generation-accounting.json": "5281b319cfc6627d687cdbd499df449beb89164bc54c8b0ec4e344f4268d463b",+ "grading-receipt.json": "4e312c871b6e0bc1247c26f6fd08fc1372e67c9b70a63efc9a5439cafcbb1153",+ "agent.patch": "9c41a7f485a6471914587d668f467e84ce4a7a8896c7a3c2e310bdc9b5ce54d1"+ },+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"+ },+ {+ "name": "pytest",+ "task": "pytest-dev__pytest-6197",+ "resolved": true,+ "baseline_resolved": true,+ "wall_seconds": 130.86104795901338,+ "baseline_wall_seconds": 596.4330365410133,+ "sol_tokens": 357497,+ "sol_cost_usd": 0.41693979999999997,+ "jev_cost_usd": 0.111920088,+ "jev_cost_kind": "estimate",+ "task_cost_usd": 0.528859888,+ "known_task_cost_usd": 0.528859888,+ "cost_complete": true,+ "baseline_cost_usd": 2.344458,+ "jev_input_tokens": 2664764,+ "jev_requests": 324,+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/pytest/attempts/pytest-dev__pytest-6197",+ "artifact_sha256": {+ "receipt.json": "d6423cb9f05fab077b003c0270f90ee77ee46ba44bfc95adb2d615a9cb4305e7",+ "generation-accounting.json": "a52c8c41a8dde5a3eda28e299a3bcd12eefa58d9a93bda79132c6efbcaf381c7",+ "grading-receipt.json": "042f65c112c1a264677b1a4e87e80101f7800e02fa31de1aba1f654283d52bf2",+ "agent.patch": "62c861ad12e469f55c4bccf4d1fde4a91c61766508ebdf6afbbf9bb8172ec382"+ },+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"+ },+ {+ "name": "sympy",+ "task": "sympy__sympy-16792",+ "resolved": true,+ "baseline_resolved": true,+ "wall_seconds": 136.16174887504894,+ "baseline_wall_seconds": 386.5954145830183,+ "sol_tokens": 222856,+ "sol_cost_usd": 0.2661758,+ "jev_cost_usd": 0.093028194,+ "jev_cost_kind": "estimate",+ "task_cost_usd": 0.359203994,+ "known_task_cost_usd": 0.359203994,+ "cost_complete": true,+ "baseline_cost_usd": 0.547975,+ "jev_input_tokens": 2214957,+ "jev_requests": 281,+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/sympy/attempts/sympy__sympy-16792",+ "artifact_sha256": {+ "receipt.json": "25de9d5fe1ba8b99f23b871323f3b348861f1f4b2af99ac6eaa72f2d13b44c37",+ "generation-accounting.json": "3b04d82a06e3eb8db8e7f28a8d062badc45b52e5f20e6b6d8467224abdd852c0",+ "grading-receipt.json": "7510788711691ecc06a7c925a2fbf2f89f01a8c06503b414beddf86f86a9f8d2",+ "agent.patch": "ceb80f479dda000b0353e28ec59badf3c1fab453e198b639a5de5411698ddef8"+ },+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"+ },+ {+ "name": "matplotlib",+ "task": "matplotlib__matplotlib-26466",+ "resolved": false,+ "baseline_resolved": false,+ "wall_seconds": 211.51842320797732,+ "baseline_wall_seconds": 106.53679849999025,+ "sol_tokens": 439949,+ "sol_cost_usd": 0.4214364,+ "jev_cost_usd": 0.175165788,+ "jev_cost_kind": "estimate",+ "task_cost_usd": 0.596602188,+ "known_task_cost_usd": 0.596602188,+ "cost_complete": true,+ "baseline_cost_usd": 0.395716,+ "jev_input_tokens": 4170614,+ "jev_requests": 650,+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/matplotlib/attempts/matplotlib__matplotlib-26466",+ "artifact_sha256": {+ "receipt.json": "134a00e112549c3707ed84d32f70d2d9766375747b92d3a1b37e5834eb3db6bc",+ "generation-accounting.json": "2a8e9e275bcd2a07ab75321792f06389bcbb663808794a3d5b475d675f52d230",+ "grading-receipt.json": "93e2af75254d0ef1e4bc5ce4c8c7675fd907f7407683c51aa234249ae97b1a89",+ "agent.patch": "6781334e557559eebdf035ea3473ad999c560e6b608daf0e89b364cf7402ca69"+ },+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"+ },+ {+ "name": "pylint",+ "task": "pylint-dev__pylint-4604",+ "resolved": false,+ "baseline_resolved": false,+ "wall_seconds": 89.66539833298884,+ "baseline_wall_seconds": 74.52734654099913,+ "sol_tokens": 167241,+ "sol_cost_usd": 0.2256252,+ "jev_cost_usd": 0.066664584,+ "jev_cost_kind": "estimate",+ "task_cost_usd": 0.292289784,+ "known_task_cost_usd": 0.292289784,+ "cost_complete": true,+ "baseline_cost_usd": 0.2836264,+ "jev_input_tokens": 1587252,+ "jev_requests": 203,+ "evidence": "/Users/david/dev/jevgrep/evals/runs/swebench/total-cost-2026-09-28/pylint/attempts/pylint-dev__pylint-4604",+ "artifact_sha256": {+ "receipt.json": "b051ab9bdcd6b19c09b74209fefe767758d7bb08e6a000900da37a38436f9ec2",+ "generation-accounting.json": "74052b7998cf38bee8048a2579bc59b570b7d00ba269a86b6543c4f17d438b14",+ "grading-receipt.json": "f76fa47afc5af114d2e46845a481242cdc3f067320bd59330274e186c6800769",+ "agent.patch": "b0976f797af371069b62e0bef425caf38821e20ce70f6271c500aede163aba34"+ },+ "skill_sha256": "04d4d7b59d122799a863bc399234de171bdb087436075be52ebf3bece1bf3e44"+ }+ ],+ "contract": {+ "package": "@dzhng/[email protected]",+ "package_sha256": "4b36178c728ee89859b8f34d083351b0f57d0527e2b22140bb8b81728fdd23b3",+ "tasks": [+ "xarray",+ "sphinx",+ "scikit",+ "django",+ "requests",+ "astropy",+ "pytest",+ "sympy",+ "matplotlib",+ "pylint"+ ],+ "provider": "typesafe",+ "model": "openai/gpt-5.6-sol",+ "effort": "medium",+ "cost": "Sol reported cost plus native Jev usage times $0.042/M input; output free; missing coverage means unknown total",+ "price_source": "https://docs.typesafe.ai/models",+ "price_verified": "2026-09-28",+ "baselines": "Reuse fixed saved baselines; no baseline runs",+ "attempts": "One fresh first attempt per task, no repair or best-of selection",+ "skill": "Exact public skill shipped in registry package",+ "prior_comparison": "speed-2026-09-28 retained separately under original Sol-only policy"+ },+ "pricing_document_sha256": "9d20bb3c90a0147532d0b20ddc4c64579391be7842e965543b27bad684eeb4d6",+ "limitations": "Single fresh repeat of ten tuned Python tasks; not a holdout or variance estimate. Jev is estimated at public list price; Sol is provider-reported cost, not invoice reconciliation. All attempts included; baseline never rerun.",+ "savings_percent": 25.78919117630659,+ "cost_per_attempt_usd": 0.5656399054,+ "cost_per_solve_usd": 0.7070498817499999,+ "baseline_cost_per_solve_usd": 0.952758625+}diff --git a/evals/results/total-cost-2026-09-28.md b/evals/results/total-cost-2026-09-28.mdnew file mode 100644index 0000000..a329de1--- /dev/null+++ b/evals/results/total-cost-2026-09-28.md@@ -0,0 +1,61 @@+# Total task cost including Jev++A fresh run of published `@dzhng/[email protected]` preserved the same **8/10 official+SWE-bench solves** as the saved no-Jev baseline. Full task cost was **$5.6563991**:+**$4.1779912 Sol + $1.4784079 Jev**, versus **$7.6220690** without Jevgrep.+That is **25.8% lower total cost**, including both unresolved attempts.++| Task | Official outcome | Sol reported cost | Jev estimate | Combined | No-Jev baseline |+| --- | --- | ---: | ---: | ---: | ---: |+| `pydata__xarray-3305` | pass | $0.4309 | $0.2701 | $0.7009 | $0.4937 |+| `sphinx-doc__sphinx-8638` | pass | $0.3715 | $0.1678 | $0.5392 | $1.0595 |+| `scikit-learn__scikit-learn-13124` | pass | $0.2129 | $0.0794 | $0.2923 | $0.2944 |+| `django__django-15629` | pass | $1.1522 | $0.3515 | $1.5037 | $1.5055 |+| `psf__requests-1142` | pass | $0.3368 | $0.0128 | $0.3496 | $0.2685 |+| `astropy__astropy-13579` | pass | $0.3436 | $0.1501 | $0.4938 | $0.4286 |+| `pytest-dev__pytest-6197` | pass | $0.4169 | $0.1119 | $0.5289 | $2.3445 |+| `sympy__sympy-16792` | pass | $0.2662 | $0.0930 | $0.3592 | $0.5480 |+| `matplotlib__matplotlib-26466` | unresolved | $0.4214 | $0.1752 | $0.5966 | $0.3957 |+| `pylint-dev__pylint-4604` | unresolved | $0.2256 | $0.0667 | $0.2923 | $0.2836 |+| **Total** | **8/10** | **$4.1780** | **$1.4784** | **$5.6564** | **$7.6221** |++Cost per attempted task is **$0.5656 versus $0.7622**. Total spend divided by+solved tasks is **$0.7070 versus $0.9528**. This includes spend on failed tasks;+it is not the mean cost of only the successful attempts.++## Accounting basis++Total task cost includes both Sol and Jev by default. The+[accounting policy](../accounting.md) governs future evaluations. Jev tokens are+still shown separately from coding-agent tokens, and returned context contributes+to Sol's bill.++Sol charges come from retained Gateway generation metadata. Native TypeSafe Jev+cost is a **list-price estimate**, calculated from response `usage.input_tokens`+at **$0.042 per million input tokens** for `jev-1.13.0`; output tokens are free.+The rate comes from [TypeSafe's model documentation](https://docs.typesafe.ai/models),+retrieved on September 28, 2026 and retained with the run. It does not subtract+free credits or assume negotiated discounts. Neither component is invoice+reconciliation. All ten attempts have complete transport and usage coverage;+missing coverage would make the combined total unknown rather than zero.++The [machine-readable report](total-cost-2026-09-28.json) retains package and+receipt hashes, price provenance, per-task components, runtime, token usage and+links to raw evidence. Raw traces and the frozen pricing document remain in+ignored `evals/runs/swebench/total-cost-2026-09-28/` storage.++## Protocol and limits++This is one fresh first attempt per task with the published package and its exact+public skill, native TypeSafe retrieval, and `openai/gpt-5.6-sol` at medium effort.+Initial Jevgrep use remains required; agents can then use ordinary tools. Tasks+ran sequentially. Saved no-Jev baselines were reused, never rerun. Official+SWE-bench grading alone determines the outcomes; Matplotlib and Pylint remained+unresolved. Every attempt and its costs count.++This is a repeat of ten tuned Python tasks, not a holdout or a variance estimate.+Sol used different queries and trajectories than the earlier run. Consequently,+this report is a new measurement, not merely the old bill with Jev added. The+[previous speed cohort](speed-2026-09-28.md) retains its original Sol-only totals+for historical comparison. Its advertised savings should not be read as including+Jev. No product code or skill was tuned during this repeat.