Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.
Python aicode-complexitycode-healthcode-intelligencecode-qualitydead-codedeveloper-toolsdocumentationgit-analyticsmcpopen-sourcepythonrefactoringstatic-analysistechnical-debt
docs: put the numbers where people actually look (#1365)
Raghav Chamadiya81d46314634f
diff --git a/.github/assets/bench/agent-output-tokens-dark.svg b/.github/assets/bench/agent-output-tokens-dark.svgnew file mode 100644index 000000000..b6895a6a5--- /dev/null+++ b/.github/assets/bench/agent-output-tokens-dark.svg@@ -0,0 +1,36 @@+<svg xmlns="http://www.w3.org/2000/svg" width="720" height="324" viewBox="0 0 720 324" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,Arial,sans-serif" role="img" aria-label="Work saved in a real agent loop">+<rect width="720" height="324" fill="none"/>+<text x="0" y="20" font-size="15" font-weight="650" fill="#E2E8F0">Work saved in a real agent loop</text>+<text x="0" y="40" font-size="12" fill="#94A3B8">Output tokens the agent itself writes to reach an answer. Lower is better.</text>+<text x="186" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#E2E8F0">repowise</text>+<rect x="196" y="62" width="396" height="24" rx="3" fill="#18202E"/>+<rect x="196" y="62" width="270.8" height="24" rx="3" fill="#F59520"/>+<text x="654" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#E2E8F0">1,250</text>+<text x="716" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#E2E8F0">-31.6%</text>+<text x="186" y="116" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">CodeGraph</text>+<rect x="196" y="100" width="396" height="24" rx="3" fill="#18202E"/>+<rect x="196" y="100" width="299.6" height="24" rx="3" fill="#334155"/>+<text x="654" y="116" text-anchor="end" font-size="12.5" font-weight="400" fill="#E2E8F0">1,383</text>+<text x="716" y="116" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">-24.4%</text>+<text x="186" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">Serena</text>+<rect x="196" y="138" width="396" height="24" rx="3" fill="#18202E"/>+<rect x="196" y="138" width="335.8" height="24" rx="3" fill="#334155"/>+<text x="654" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#E2E8F0">1,550</text>+<text x="716" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">-14.8%</text>+<text x="186" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">Graphify</text>+<rect x="196" y="176" width="396" height="24" rx="3" fill="#18202E"/>+<rect x="196" y="176" width="359.2" height="24" rx="3" fill="#334155"/>+<text x="654" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#E2E8F0">1,658</text>+<text x="716" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">-8.9%</text>+<text x="186" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">code-review-graph</text>+<rect x="196" y="214" width="396" height="24" rx="3" fill="#18202E"/>+<rect x="196" y="214" width="370.4" height="24" rx="3" fill="#334155"/>+<text x="654" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#E2E8F0">1,710</text>+<text x="716" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">-6.0%</text>+<text x="186" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">bare agent (control)</text>+<rect x="196" y="252" width="396" height="24" rx="3" fill="#18202E"/>+<rect x="196" y="252" width="396.0" height="24" rx="3" fill="none" stroke="#64748B" stroke-width="1.5" stroke-dasharray="4 3"/>+<text x="654" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#E2E8F0">1,828</text>+<text x="716" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">baseline</text>+<text x="0" y="316" font-size="11" fill="#94A3B8">48 questions on django/django under Codex (gpt-5.6-sol), n=43 completed by all six arms. repowise p < 0.0001.</text>+</svg>diff --git a/.github/assets/bench/agent-output-tokens.svg b/.github/assets/bench/agent-output-tokens.svgnew file mode 100644index 000000000..4d06ab5e3--- /dev/null+++ b/.github/assets/bench/agent-output-tokens.svg@@ -0,0 +1,36 @@+<svg xmlns="http://www.w3.org/2000/svg" width="720" height="324" viewBox="0 0 720 324" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,Arial,sans-serif" role="img" aria-label="Work saved in a real agent loop">+<rect width="720" height="324" fill="none"/>+<text x="0" y="20" font-size="15" font-weight="650" fill="#0A0A0A">Work saved in a real agent loop</text>+<text x="0" y="40" font-size="12" fill="#64748B">Output tokens the agent itself writes to reach an answer. Lower is better.</text>+<text x="186" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#0A0A0A">repowise</text>+<rect x="196" y="62" width="396" height="24" rx="3" fill="#F1F5F9"/>+<rect x="196" y="62" width="270.8" height="24" rx="3" fill="#F59520"/>+<text x="654" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#0A0A0A">1,250</text>+<text x="716" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#0A0A0A">-31.6%</text>+<text x="186" y="116" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">CodeGraph</text>+<rect x="196" y="100" width="396" height="24" rx="3" fill="#F1F5F9"/>+<rect x="196" y="100" width="299.6" height="24" rx="3" fill="#CBD5E1"/>+<text x="654" y="116" text-anchor="end" font-size="12.5" font-weight="400" fill="#0A0A0A">1,383</text>+<text x="716" y="116" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">-24.4%</text>+<text x="186" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">Serena</text>+<rect x="196" y="138" width="396" height="24" rx="3" fill="#F1F5F9"/>+<rect x="196" y="138" width="335.8" height="24" rx="3" fill="#CBD5E1"/>+<text x="654" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#0A0A0A">1,550</text>+<text x="716" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">-14.8%</text>+<text x="186" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">Graphify</text>+<rect x="196" y="176" width="396" height="24" rx="3" fill="#F1F5F9"/>+<rect x="196" y="176" width="359.2" height="24" rx="3" fill="#CBD5E1"/>+<text x="654" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#0A0A0A">1,658</text>+<text x="716" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">-8.9%</text>+<text x="186" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">code-review-graph</text>+<rect x="196" y="214" width="396" height="24" rx="3" fill="#F1F5F9"/>+<rect x="196" y="214" width="370.4" height="24" rx="3" fill="#CBD5E1"/>+<text x="654" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#0A0A0A">1,710</text>+<text x="716" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">-6.0%</text>+<text x="186" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">bare agent (control)</text>+<rect x="196" y="252" width="396" height="24" rx="3" fill="#F1F5F9"/>+<rect x="196" y="252" width="396.0" height="24" rx="3" fill="none" stroke="#94A3B8" stroke-width="1.5" stroke-dasharray="4 3"/>+<text x="654" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#0A0A0A">1,828</text>+<text x="716" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">baseline</text>+<text x="0" y="316" font-size="11" fill="#64748B">48 questions on django/django under Codex (gpt-5.6-sol), n=43 completed by all six arms. repowise p < 0.0001.</text>+</svg>diff --git a/.github/assets/bench/file-coverage-dark.svg b/.github/assets/bench/file-coverage-dark.svgnew file mode 100644index 000000000..caa1c645d--- /dev/null+++ b/.github/assets/bench/file-coverage-dark.svg@@ -0,0 +1,30 @@+<svg xmlns="http://www.w3.org/2000/svg" width="720" height="324" viewBox="0 0 720 324" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,Arial,sans-serif" role="img" aria-label="Finding the right files, on 42 sealed instances">+<rect width="720" height="324" fill="none"/>+<text x="0" y="20" font-size="15" font-weight="650" fill="#E2E8F0">Finding the right files, on 42 sealed instances</text>+<text x="0" y="40" font-size="12" fill="#94A3B8">File coverage against gold spans. Higher is better. Deterministic grading, no LLM judge.</text>+<text x="186" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#E2E8F0">repowise get_answer</text>+<rect x="196" y="62" width="430" height="24" rx="3" fill="#18202E"/>+<rect x="196" y="62" width="376.7" height="24" rx="3" fill="#F59520"/>+<text x="716" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#E2E8F0">0.876</text>+<text x="186" y="116" text-anchor="end" font-size="12.5" font-weight="650" fill="#E2E8F0">repowise search_codebase</text>+<rect x="196" y="100" width="430" height="24" rx="3" fill="#18202E"/>+<rect x="196" y="100" width="319.1" height="24" rx="3" fill="#B4741F"/>+<text x="716" y="116" text-anchor="end" font-size="12.5" font-weight="650" fill="#E2E8F0">0.742</text>+<text x="186" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">CodeGraph</text>+<rect x="196" y="138" width="430" height="24" rx="3" fill="#18202E"/>+<rect x="196" y="138" width="262.3" height="24" rx="3" fill="#334155"/>+<text x="716" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#E2E8F0">0.610</text>+<text x="186" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">Graphify</text>+<rect x="196" y="176" width="430" height="24" rx="3" fill="#18202E"/>+<rect x="196" y="176" width="234.8" height="24" rx="3" fill="#334155"/>+<text x="716" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#E2E8F0">0.546</text>+<text x="186" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">code-review-graph</text>+<rect x="196" y="214" width="430" height="24" rx="3" fill="#18202E"/>+<rect x="196" y="214" width="191.3" height="24" rx="3" fill="#334155"/>+<text x="716" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#E2E8F0">0.445</text>+<text x="186" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">cocoindex</text>+<rect x="196" y="252" width="430" height="24" rx="3" fill="#18202E"/>+<rect x="196" y="252" width="155.2" height="24" rx="3" fill="#334155"/>+<text x="716" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#E2E8F0">0.361</text>+<text x="0" y="316" font-size="11" fill="#94A3B8">n=42 (cocoindex n=41). Sign test vs CodeGraph p = 0.00004. Sealed half, evaluated once.</text>+</svg>diff --git a/.github/assets/bench/file-coverage.svg b/.github/assets/bench/file-coverage.svgnew file mode 100644index 000000000..b8fac1861--- /dev/null+++ b/.github/assets/bench/file-coverage.svg@@ -0,0 +1,30 @@+<svg xmlns="http://www.w3.org/2000/svg" width="720" height="324" viewBox="0 0 720 324" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,Arial,sans-serif" role="img" aria-label="Finding the right files, on 42 sealed instances">+<rect width="720" height="324" fill="none"/>+<text x="0" y="20" font-size="15" font-weight="650" fill="#0A0A0A">Finding the right files, on 42 sealed instances</text>+<text x="0" y="40" font-size="12" fill="#64748B">File coverage against gold spans. Higher is better. Deterministic grading, no LLM judge.</text>+<text x="186" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#0A0A0A">repowise get_answer</text>+<rect x="196" y="62" width="430" height="24" rx="3" fill="#F1F5F9"/>+<rect x="196" y="62" width="376.7" height="24" rx="3" fill="#F59520"/>+<text x="716" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#0A0A0A">0.876</text>+<text x="186" y="116" text-anchor="end" font-size="12.5" font-weight="650" fill="#0A0A0A">repowise search_codebase</text>+<rect x="196" y="100" width="430" height="24" rx="3" fill="#F1F5F9"/>+<rect x="196" y="100" width="319.1" height="24" rx="3" fill="#F6B15C"/>+<text x="716" y="116" text-anchor="end" font-size="12.5" font-weight="650" fill="#0A0A0A">0.742</text>+<text x="186" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">CodeGraph</text>+<rect x="196" y="138" width="430" height="24" rx="3" fill="#F1F5F9"/>+<rect x="196" y="138" width="262.3" height="24" rx="3" fill="#CBD5E1"/>+<text x="716" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#0A0A0A">0.610</text>+<text x="186" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">Graphify</text>+<rect x="196" y="176" width="430" height="24" rx="3" fill="#F1F5F9"/>+<rect x="196" y="176" width="234.8" height="24" rx="3" fill="#CBD5E1"/>+<text x="716" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#0A0A0A">0.546</text>+<text x="186" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">code-review-graph</text>+<rect x="196" y="214" width="430" height="24" rx="3" fill="#F1F5F9"/>+<rect x="196" y="214" width="191.3" height="24" rx="3" fill="#CBD5E1"/>+<text x="716" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#0A0A0A">0.445</text>+<text x="186" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">cocoindex</text>+<rect x="196" y="252" width="430" height="24" rx="3" fill="#F1F5F9"/>+<rect x="196" y="252" width="155.2" height="24" rx="3" fill="#CBD5E1"/>+<text x="716" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#0A0A0A">0.361</text>+<text x="0" y="316" font-size="11" fill="#64748B">n=42 (cocoindex n=41). Sign test vs CodeGraph p = 0.00004. Sealed half, evaluated once.</text>+</svg>diff --git a/README.md b/README.mdindex f15db1dfc..1cc491070 100644--- a/README.md+++ b/README.md@@ -43,14 +43,24 @@### Your AI agent burns most of its budget rediscovering your codebase. Index it once, and it never has to again.-### One index. A dependency graph, a generated wiki, mined git history, architectural decisions, and a defect-validated health score.--<sub><strong>−31.6%</strong> of an agent's own output tokens across 43 questions (p<0.0001), reached in-3.8 tool calls where a bare agent needed 7.2 · loading one commit's context costs 393 tokens-instead of 13,984 raw · defect-risk <strong>ROC AUC 0.737</strong> across 21 repos and 9 languages, scored-leakage-free, and 0.76–0.78 on a public dataset that played no part in calibration · every-layer computed with <strong>zero LLM calls</strong>. We publish the rows we lose.-<a href="docs/BENCHMARKS.md"><strong>See how the others did →</strong></a><br />+<table align="center">+<tr>+<td align="center" width="250"><h2>#1 of 6</h2></td>+<td align="center" width="250"><h2>−31.6%</h2></td>+<td align="center" width="250"><h2>97%</h2></td>+</tr>+<tr>+<td align="center" valign="top"><sub><strong>at finding the right files.</strong><br />0.876 file coverage against the<br />next tool's 0.610, on a <strong>sealed</strong><br />42-instance split. <em>p=0.00004</em></sub></td>+<td align="center" valign="top"><sub><strong>of your agent's own output tokens,</strong><br />reached in 3.8 tool calls where a<br />bare agent needed 7.2. <em>n=43,<br />p<0.0001, leaner on 37 of 44</em></sub></td>+<td align="center" valign="top"><sub><strong>fewer tokens to load a commit.</strong><br />393 instead of 13,984 raw, counted<br />with deterministic tiktoken across<br />30 commits. <em>35.6x, pooled</em></sub></td>+</tr>+</table>++<sub>Measured head to head against the open-source agent-context field, on instances held out+from every improvement round. Defect risk validated separately at <strong>ROC AUC 0.737</strong>+across 21 repos and 9 languages, leakage-free. Every layer computed with <strong>zero LLM+calls</strong>. <strong>We publish the rows we lose</strong>, and we are the slowest indexer here.+<a href="docs/BENCHMARKS.md"><strong>All of it, including the losses →</strong></a><br />Free and self-hosted, runs on your machine, and the first index needs no API key.</sub><picture>@@ -527,9 +537,20 @@ opt-in tools, and the full reference: **[docs/agent/MCP_TOOLS.md →](docs/agent## Measured against the field-Three numbers, each with its sample size and its test. The tools here are the-open-source agent-context tools we ran head to head, and the full page carries-the rows we lose beside the rows we win.+Six open-source agent-context tools, the same repositories, the same pinned+commits, the same questions, each one given its own full advertised tool surface.+The full page carries the rows we lose beside the rows we win.++<div align="center">+<picture>+ <source media="(prefers-color-scheme: dark)" srcset=".github/assets/bench/file-coverage-dark.svg" />+ <img src=".github/assets/bench/file-coverage.svg" alt="File coverage on 42 sealed ContextBench instances: repowise get_answer 0.876, repowise search_codebase 0.742, CodeGraph 0.610, Graphify 0.546, code-review-graph 0.445, cocoindex 0.361" width="100%" />+</picture>+<picture>+ <source media="(prefers-color-scheme: dark)" srcset=".github/assets/bench/agent-output-tokens-dark.svg" />+ <img src=".github/assets/bench/agent-output-tokens.svg" alt="Output tokens an agent writes to reach an answer across 43 django questions on Codex: repowise 1,250, CodeGraph 1,383, Serena 1,550, Graphify 1,658, code-review-graph 1,710, bare agent 1,828" width="100%" />+</picture>+</div>- **Finds the right files.** 0.876 file coverage against CodeGraph's 0.610 on a**sealed** 42-instance split, held out from every improvement round. 19 wins,diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.mdindex 7cd6f0ffe..d306e886d 100644--- a/docs/BENCHMARKS.md+++ b/docs/BENCHMARKS.md@@ -1,92 +1,60 @@# Benchmarks-Every number repowise publishes, with its sample size, its test, and a link to-the raw data. The harnesses and full reports live in-**[repowise-bench](https://github.com/repowise-dev/repowise-bench)**, and nothing-here is measured on a private corpus.+Every number repowise publishes, with its sample size, its test, and a link to the+raw data. Nothing here is measured on a private corpus. The harnesses, the+pre-registrations and every graded cell live in+**[repowise-bench](https://github.com/repowise-dev/repowise-bench)**, which is also+where the depth is: this page is the summary, that repository is the evidence.-## Why this page is built the way it is--In July 2026, JetBrains took two popular token-saving tools and reran their-headline claims on real agent work with Sonnet 5.--| Tool | Advertised saving | What JetBrains measured |-|---|---|---|-| [Caveman](https://blog.jetbrains.com/ai/2026/07/speak-to-ai-agents-like-cavemen-tosave-tokens/) | 65% | **8.5%** |-| [RTK](https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings/) | 60 to 90% | **7.6% more expensive** at low reasoning effort (p = 0.004), and no change at high effort |--Greptile's 82% became 45% under Augment's rerun. The pattern is consistent-enough to be a rule: in this category, the advertised number and the reran-number are different numbers.--They are different for a reason that is worth understanding before you read-anything below, because it is the single most common way a token claim goes-wrong.--**Measuring one context load is easy. Measuring an agent loop is hard.** You can-show that your representation of a file is smaller than the file. That is a real-measurement and it is the one almost everybody publishes. It is also not the-question anyone actually has, which is whether an agent given your tool finishes-the job having done less work. Agents re-read, backtrack, re-plan and re-explore.-A compression that looks like 90% on one payload routinely nets out near zero-across a real session, and can go negative when the agent has to work harder to-recover what you compressed away.--**This page publishes both numbers and labels which is which.** Section 3 is our-one-context-load figure, the easy one. Section 2 is the agent-loop figure,-measured the way JetBrains measured: real agent, real repository, real tool use,-against a control. Section 2 is the one that matters, and it is deliberately the-smaller number.--That is also why this page prints n beside every mean, states which tool-produced each number, and publishes the rows we lose. It is built to survive a-rerun, because in this field that is the only property that turns out to matter.--So we reran our own headline ourselves, on a second agent harness, and published-what came back. It moved a number we had been quoting, and section 2 now-carries both results and the correction.--## What we measured, and against whom--repowise builds [five intelligence layers](layers/INTELLIGENCE_LAYERS.md) from-one index, so there is no single competitor to measure against. Different tools-overlap on different layers, and this table says exactly which.+**Read this page in one minute:** the two charts below are the two results that+matter. Everything after them is the sample size, the caveat and the row we lose.| Layer | Measured against | Result ||---|---|---|-| Finding the right files | CodeGraph, Graphify, code-review-graph | [§1](#1-finding-the-right-files) **we win**, n=42 held out, p=0.00004 |-| Work saved in a real agent loop | CodeGraph, Serena, Graphify, code-review-graph, bare agent | [§2](#2-what-changes-in-a-real-agent-loop) **we win**, n=43 on Codex at p<0.0001, and the only tool to clear the bar on both agent harnesses we tried |+| Finding the right files | CodeGraph, Graphify, code-review-graph, cocoindex | [§1](#1-finding-the-right-files) **we win**, n=42 held out, p=0.00004 |+| Work saved in a real agent loop | the same field plus Serena and a bare agent | [§2](#2-what-changes-in-a-real-agent-loop) **we win** on all three agent harnesses we tried, n=43 at p<0.0001 || Loading one commit's context | naive file reads, `git diff` | [§3](#3-loading-one-commits-context-the-easy-number) **35.6x** fewer tokens than naive || Command-output compression | RTK | [§4](#4-command-output-compression) **not measured head to head** |-| Code health and defect prediction | CodeScene | [§5](#5-code-health-predicts-defects) **we win**, p=0.003 |+| Code health and defect prediction | CodeScene | [§5](#5-code-health-predicts-defects) **we win** on recall and effort-aware ranking, p=0.003 || Indexing time | CodeGraph, Graphify, code-review-graph | [§6](#6-indexing-time-the-row-we-lose) **we lose**, 22x, because we build four more layers in the same pass || Documentation generation | DeepWiki, Google Code Wiki, Swimm | **not measured** || PR review | CodeRabbit, Greptile | **not measured** |The last two rows are capability comparisons, not measurements, and they live inthe [README's feature table](../README.md#how-it-compares-on-capability) where a-reader can tell the difference. We would rather say "not measured" than let a-checkmark do a number's job. **§4 carries the same label for the same reason**:-RTK does exactly what `repowise distill` does, and we have not run the two-against each other.+reader can tell the difference. **§4 carries the same label for the same reason:**+RTK does exactly what `repowise distill` does, and we have not run the two against+each other. We would rather write "not measured" than let a checkmark do a+number's job.++---++<div align="center">+<picture>+ <source media="(prefers-color-scheme: dark)" srcset="../.github/assets/bench/file-coverage-dark.svg" />+ <img src="../.github/assets/bench/file-coverage.svg" alt="File coverage on 42 sealed ContextBench instances: repowise get_answer 0.876, repowise search_codebase 0.742, CodeGraph 0.610, Graphify 0.546, code-review-graph 0.445, cocoindex 0.361" width="100%" />+</picture>+<picture>+ <source media="(prefers-color-scheme: dark)" srcset="../.github/assets/bench/agent-output-tokens-dark.svg" />+ <img src="../.github/assets/bench/agent-output-tokens.svg" alt="Output tokens an agent writes to reach an answer across 43 django questions on Codex: repowise 1,250, CodeGraph 1,383, Serena 1,550, Graphify 1,658, code-review-graph 1,710, bare agent 1,828 baseline" width="100%" />+</picture>+</div>---## 1. Finding the right files-Before a tool can save an agent any work, it has to point at the right code.-This section measures only that.+Before a tool can save an agent any work, it has to point at the right code. This+section measures only that.-**Grading here is deterministic. No LLM judge is involved anywhere in this-number**, which makes it the most reproducible result on the page. ContextBench-ships gold file spans; a tool either returns them or it does not.+**Grading is deterministic. No LLM judge is involved anywhere in this number**,+which makes it the most reproducible result on the page. ContextBench ships gold+file spans; a tool either returns them or it does not.**Every number below comes from instances this work has never seen.** The 112instances were split 70 / 42 by instance id, **pinned before any of it started**,and the 42 were kept sealed until the final measurement.-### The numbers, on the 42 sealed instances-| Tool | File coverage | n | Precision | Files served ||---|---:|---:|---:|---:|| **repowise** (`get_answer`) | **0.876** | 42 | 0.087 | 19.2 |@@ -94,23 +62,37 @@ and the 42 were kept sealed until the final measurement.| CodeGraph | 0.610 | 42 | 0.093 | 14.0 || Graphify | 0.546 | 42 | 0.033 | 34.5 || code-review-graph | 0.445 | 42 | 0.240 | 5.4 |+| cocoindex | 0.361 | 41 | 0.092 | 7.1 |Head to head per instance against CodeGraph: `get_answer` **19 wins, 1 loss, 22ties, sign test p = 0.00004**. `search_codebase` **13 wins, 3 losses, 26 ties,p = 0.021**.**These are two different tools with two different profiles, and we would rather-say so than average them into one claim.** `get_answer` finds the most, from a-list of ~19 files. `search_codebase` finds fewer but is the most efficient per-file served: **0.742 from 8.2 files**, better coverage-per-file than anything-else in the table. If you are paying by the token, that is the row to read.--### How this number moved, and why it is not benchmark tuning--We first ran this and came **last, at 0.228**. We published that. The cause-turned out to be a bug: a query-time gate was discarding most candidates before-ranking ever happened. Fixing that path is what moved the number, and it is a-fix any user of the tool gets, not a change shaped around these questions.+say so than average them into one claim.** `get_answer` finds the most, from a list+of ~19 files. `search_codebase` finds fewer but is the most efficient per file+served: **0.742 from 8.2 files**, better coverage-per-file than anything else in+the table. If you are paying by the token, that is the row to read.++**Precision is not our column.** code-review-graph's 0.240 is nearly three times+ours. Part of that is mechanical, since precision rises for whoever returns fewest+files, which is exactly why files-served is a column here and not a footnote.++**The cocoindex row is not from the same sitting**, and it is printed rather than+smoothed away: measured 2026-08-09 against the other five from 2026-08-02 to+2026-08-06, same instances, same gold spans, same deterministic grading. Its n is+41 because one named instance never answered even when queried alone, so it is+excluded rather than counted as a zero, which makes its row *better* than counting+it would (0.361 against 0.353). Its placing was+[pre-registered before its index existed](https://github.com/repowise-dev/repowise-bench/tree/master/configs),+including the prediction that it would come last.++### Why this is not benchmark tuning++We first ran this and came **last, at 0.228**. We published that. The cause was a+query-time gate discarding most candidates before ranking ever happened. Fixing+that path is what moved the number, and it is a fix any user gets, not a change+shaped around these questions.The check on that claim is the split, and it points the right way:@@ -121,57 +103,43 @@ The check on that claim is the split, and it points the right way:| CodeGraph | **0.6093** | **0.6095** |**Overfitting makes the unseen half score worse. Ours scores better**, on both-tools. And CodeGraph, which nobody tuned against either half, scores the same on-both to three decimal places, so the two halves are equally hard and the gap is-about the tool rather than the questions.--**We do not quote a pooled 112-instance figure**, though it is easy to compute-and would be 0.835. Averaging the halves loses the only number that matters,-which is how the tool does on instances it has never seen.+tools. CodeGraph, which nobody tuned against either half, scores the same on both+to three decimal places, so the two halves are equally hard and the gap is about+the tool rather than the questions.-### What it cost to produce--Every arm builds its **own** index of **every** instance's repository at that-instance's own `base_commit`. Nothing is shared between arms, nothing is cached-across instances, and a stale checkout is a wrong answer rather than a fast one.-Across the rung-8 matrix that is **748 index builds and roughly 78 machine-hours-of indexing for 1,129 graded (instance, arm) cells.**+**We do not quote a pooled 112-instance figure**, though it is easy to compute and+would be 0.835. Averaging the halves loses the only number that matters.This is retrieval, not task success. It says we find the right files, not that an-agent using us writes better code. That is what section 2 is for.--Measured on repowise at commit `081a59fa` (between v0.37.0 and v0.38.0).+agent using us writes better code. That is what §2 is for. Producing it cost+**748 index builds and roughly 78 machine-hours**, because every arm builds its own+index of every instance at that instance's own `base_commit`, with nothing shared+and nothing cached.-Raw data and harness:-**[bakeoff\_2026\_08/rung8](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08/rung8)**+Measured on repowise `081a59fa` (between v0.37.0 and v0.38.0). Raw cells:+**[rung8](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08/rung8)**.---## 2. What changes in a real agent loop-This is the section modelled on the JetBrains reruns, and the one we would ask a-skeptic to read first.+The section a skeptic should read first, and the one modelled on the JetBrains+reruns described [at the bottom of this page](#how-to-read-a-number-on-this-page).-Every question in `django/django`'s question set, 48 of them, spanning five-question shapes. Six arms: repowise, four competing tools, and a bare agent with-no tools at all. Every arm got a byte-identical prompt, its full advertised tool-surface, and a freshly built index on the same pinned commit. The bare-agent-control was verified free of any local hooks, so it is a real control and not a-contaminated one.+Every question in `django/django`'s question set, 48 of them, across five question+shapes. Six arms: repowise, four competing tools, and a bare agent with no tools.+Byte-identical prompt, full advertised tool surface per arm, a freshly built index+on the same pinned commit, and a bare control verified free of local hooks.-Two things have to be true for a tool to be worth mounting. The agent has to-actually call it, and the loop has to get leaner when it does. One table, both-questions.+Two things have to be true for a tool to be worth mounting: the agent has to+actually call it, and the loop has to get leaner when it does.-We ran this on two agent harnesses, because the answer turned out to depend on-the harness as much as on the tools. The main result is on Codex, where every-tool in the field actually gets used, so the comparison is between the tools-rather than between agents that ignored them. Claude Code follows as a second-proof point.+**We ran this on three agent harnesses, because the answer depends on the harness+as much as on the tools.**### The main run: 48 questions on Codex (`gpt-5.6-sol`)-Every tool called on every question, so this is a like-for-like comparison.+Every tool called on every question, so this is like-for-like.| Tool | Agent used it | Output tokens | vs bare agent | Tool calls | Leaner on | p ||---|---:|---:|---:|---:|---:|---:|@@ -182,26 +150,30 @@ Every tool called on every question, so this is a like-for-like comparison.| code-review-graph | 43 / 43 | 1,710 | -6.0% | 7.2 | 26 of 43 | 0.046 || *bare agent (control)* | 0 / 44 | 1,828 | baseline | 7.2 | n/a | n/a |-**repowise leaves the agent with the least work to do, and gets there in the-fewest steps.** A third less output than working with no tool at all, reached in-**3.8 tool calls against the bare agent's 7.2**. One answered question replacing-roughly six greps, visible directly in the call counts rather than inferred.+A third less output than working with no tool at all, reached in **3.8 tool calls+against the bare agent's 7.2**, opening 3.0 files instead of 7.2. One answered+question replacing roughly six greps, visible in the call counts rather than+inferred.-Correcting for testing five tools at once, three reductions are solid and two-are marginal. **CodeGraph is a genuine second at -24.4%**, and the honest reading-is that we lead a field in which more than one tool works, not that we are the-only one that does.+Correcting for testing five tools at once, three reductions are solid and two are+marginal. **CodeGraph is a genuine second at -24.4%**, and the honest reading is+that we lead a field in which more than one tool works. Serena is the counter-case:+it writes less than the bare agent while calling tools **42% more often**. Busier,+not leaner.-Serena is the interesting counter-case: it writes less than the bare agent while-calling tools **42% more often**. Busier, not leaner.+5 of the 48 questions are missing from every arm equally because the run hit an API+usage cap. Paired comparisons are unaffected; the figures are over the 43 all six+arms completed.-5 of the 48 questions are missing from every arm equally, because the run hit an-API usage cap near the end. Paired comparisons are unaffected and the figures-above are over the 43 questions all six arms completed.+**The saving grows with the work.** Splitting at the median by how much the bare+agent needed: easier half 27.2%, harder half **34.3%**, correlation +0.379.+Pre-computed structure replaces exploration, and harder questions contain more+exploration to replace. That split is post-hoc and the pre-registered comparisons+are the stronger evidence.-### The second proof point: 15 questions on Claude Code (`claude-sonnet-5`)+### Second harness: 15 questions on Claude Code (`claude-sonnet-5`)-| Tool | Tools advertised | Schema cost (chars) | Agent used it | Output tokens | vs bare agent | Leaner on | p |+| Tool | Tools advertised | Schema cost (chars) | Agent used it | Output tokens | vs bare | Leaner on | p ||---|---:|---:|---:|---:|---:|---:|---:|| **repowise** | 10 | 17,561 | **15 / 15** | **2,420** | **-15.9%** | **12 of 15** | **0.035** || CodeGraph | 1 | 1,567 | 13 / 15 | 2,540 | -11.7% | 10 of 15 | 0.302 |@@ -210,202 +182,102 @@ above are over the 43 questions all six arms completed.| Graphify | 10 | 5,482 | 3 / 15 | 2,878 | 0.0% | 7 of 15 | 1.000 || *bare agent (control)* | 0 | 0 | n/a | 2,877 | baseline | n/a | n/a |-**repowise is the only tool that clears the bar on both harnesses**, and it is-the same direction and the same mechanism in each.--The "agent used it" column is doing something different here, and it is the most-interesting number on this page. Under Claude Code most of these tools were-barely called at all: code-review-graph never once, Graphify three times in-fifteen. Nothing was different about the servers, the questions or the indexes-between the two runs. Claude Code loads MCP tool schemas on demand, so the agent-has to go looking before it can call anything, and frequently never does.--**Treat that column as unstable, including our own 15 of 15.** Rerunning the-same setup on later days returned 4 of 15 and then 3 of 15 for us, and 2 of 14-for CodeGraph. It is a property of the pairing of tool and harness on a given-day, not of the tool.--### Is that adoption collapse the model or the harness? Inconclusive.--The same 15 questions on Opus, everything else held fixed, against bands set-before the run: **12 or more means the model**, 6 or fewer means **the harness's-schema deferral**, 7 to 11 is inconclusive.--**It came back at 7.** Published as inconclusive.--The mechanism is more useful than the verdict. Opus goes looking on **11 of 15**-against Sonnet's 13 of 30, then declines about a third of the times it looks. So-schema deferral is part of the story and not all of it. Ordering across the three-instruments is Sonnet 3 to 4, Opus 7, Codex 15.--**That run's token column also fails its own control**, at -9.3% when the tool-was called against -10.7% when it was not, so no token figure is quoted from it-for any tool including ours. Same class of artifact as the 43%-cheaper dollar-control above, from position and variance at n=15 rather than caching. The Codex-run is unaffected: different harness, three times the `n`, every tool called on-every question.--### How to read those tables--**Agent used it** counts the questions where the agent made at least one call the-server actually answered. A tool the agent never reaches for is not a tool it-has, whatever the feature list says. code-review-graph advertises 30 tools over-a built, embedded graph of 40,904 nodes and 380,168 edges, and under Claude Code-the agent never called it once across 15 questions. Under Codex it called it on-all 15. **That number is a fact about the pairing of tool and harness, not about-the tool**, which is why it appears twice above and never as a single figure.--**Output tokens** is how much the agent itself writes to reach an answer: its-reasoning, its tool calls, its final reply. Lower means it went in a straighter-line. This is the honest measure of work saved, for reasons in the box below.--**Leaner on** is the plain-language version of the statistic: on how many-questions did this tool beat the bare agent, head to head. 37 of 44 is not-something a coin does. Roughly half is.--**Tool calls** is how many separate actions the agent took. It is the clearest-view of the mechanism: on Codex the repowise agent finished in 3.8 steps where-the bare agent needed 7.2, and it opened 3.0 files instead of 7.2. Those move-together because they are the same effect, which is work done once, offline,-that the agent would otherwise redo on every query.--### Why this section reports tokens and not dollars--Dollar cost per question is the number every tool in this category wants to-quote, and it is close to meaningless as a measure of a tool. Here is the control-that convinced us, run in our own harness.--**Under Claude Code, code-review-graph never called its server across all 15-questions.** It is behaviourally identical to the bare agent, carrying an extra-28,118 characters of tool schema that should make it cost *more*. Measured on-dollars, it came out-**43% cheaper than the bare agent**. A tool that did nothing produced a-best-in-class saving.--The cause is prompt caching. Cached tokens bill at a fraction of fresh ones, so-whichever arm happens to run first pays to warm the cache and every arm after it-reads it cheaply. In our run the correlation between an arm's position in the-cycle and its dollar cost was **-0.487**. That is not a property of any tool. It-is a property of the schedule.--**Output tokens are immune to this.** They are never cached, and their-correlation with run position is **+0.010**, which is nothing. So that is what we-report. It is a smaller and less impressive number than the dollar figure would-have been, and it is the one that survives.--If you see a token-savings claim in this category that does not say whether it-controls for cache state and arm ordering, this is the first thing to ask about.--### Where the saving is largest--The effect is not uniform. **repowise saves more on questions that require-touching more of the codebase.** Splitting the 48-question Codex run at the-median, by how much work the bare agent needed:--| | n | Bare agent output | Tokens saved | % saved |-|---|---:|---:|---:|---:|-| easier half | 22 | 1,377 | 374 | 27.2% |-| harder half | 22 | 2,279 | **781** | **34.3%** |--The harder half saves **more than twice as many tokens per question**, and the-correlation between how much work a question demands and how much we save is-**+0.379**. The same pattern appeared on the smaller Claude Code run, so it has-now shown up twice.--The mechanism is that pre-computed structure replaces exploration, and harder-questions contain more exploration to replace.--**One honest limit.** Every question here is answered in a single session of-roughly four to seven turns. **We have not measured a long multi-hour task such-as designing a feature across many files, and we will not imply a number for-one.** The reasonable expectation is that a saving which scales with exploration-keeps scaling when there is more exploration to do, but that is an argument from-mechanism rather than a result, and it stays labelled as one until we run it.--### What we will not claim from this run--- **This is a work-saved result, not a quality result.** A blind judge scored- every tool in the field, ours included, a fraction **below** the bare agent on- the 48-question run, in a range of 0.04 to 0.25 points on a 10-point scale.- None of those differences is statistically distinguishable from zero, and all- of them are smaller than the 0.69 points by which this benchmark moves when we- rerun it unchanged. So the correct reading is that no tool here measurably- changed answer quality in either direction. Ours is at the low end of that- band, we are watching it, and we will say so if it turns into a real effect.- "No significant difference" is not the same as parity, and an equivalence- claim needs a test we have not run.-- **The two quality columns cannot be compared with each other.** They were- graded by different judges, because grading a model with a judge from its own- family is a known bias and avoiding it means the judge changes when the agent- does. So no quality number in one table may be subtracted from one in the other.-- **Not a universal saving.** One repository, one commit, one prompt. Two- harnesses, which is two more than most published numbers in this category, and- still not a constant. Treat each figure as that configuration's result.-- **Adoption is not a stable property of a tool.** We used to read the "agent- used it" column as a design result about how well we had named and shaped our- tools. It does not support that. Repeating the Claude Code run with nothing- changed on anyone's side moved us from 15 of 15 to 4 of 15 to 3 of 15, and- CodeGraph from 13 of 15 to 2 of 14. Switching harness moved every tool in the- field to called-on-every-question. **Whether an agent calls a codebase server- at all depends more on the harness than on the server.** Any adoption figure, ours included,- is only meaningful with its harness and its date attached.--### What producing these tables cost--**471 agent runs, about 13 hours of machine time, roughly $44 of API spend.**--| | runs | wall clock | API spend |-|---|---:|---:|---:|-| Codex, 48 questions x 6 tools | 261 | 4.9h | $17.62 |-| Codex, earlier 15-question run | 90 | 1.7h | $6.08 |-| Codex, proof-of-life checks and a second language | 30 | 0.5h | $1.58 |-| Claude Code with Sonnet, 15 questions x 6 tools | 90 | 1.8h | $18.77 |-| Claude Code with Opus, the promised rerun | 90 | 2.1h | $28.57 |--The two harnesses' dollar figures are not comparable and are given only as what-each cost to produce: Claude Code reports its own spend, while Codex reports-token counts only, so its figure is computed from published list rates.--**Before any of that, every tool got a fresh index built from scratch** on the-same pinned commit, **11.3 minutes in total**: repowise 363.6s, Graphify 102.7s,-code-review-graph 54.3s, CodeGraph 15.6s. Serena indexes on demand. We are the-slowest of the four and [say so in §6](#6-indexing-time-the-row-we-lose). Those-indexes were then reused by every later run, so the second harness cost agent-time only.--Two things are not in that table and are most of the real effort. **The-competitor setups**:-[code-review-graph](https://github.com/repowise-dev/repowise-bench/blob/master/head-to-head/arms/code-review-graph.md)-alone needs three steps that are not in its README, each of which produces a-clean, plausible zero when missed, and getting-[Serena](https://github.com/repowise-dev/repowise-bench/blob/master/head-to-head/arms/serena.md)-to answer anything at all needs an explicit project activation. A tool scoring-zero because we set it up wrong is not a result, and finding that out is most of-what this work is. The setup each tool needs is written up per tool.--**And the checks that come before any number is allowed to count.** Every arm is-probed before each run to confirm its server actually answers, and a control-that checks it can also correctly report a tool as unused, so a broken detector-cannot quietly pass everything. Roughly a third of the total runs above are-those checks and repeats rather than headline numbers.--Raw data, every cell including the failures:+The "agent used it" column is the interesting one. Under Claude Code most of these+tools were barely called at all: code-review-graph never once, Graphify three times+in fifteen. Nothing differed about the servers, questions or indexes between the+two harnesses. Claude Code loads MCP tool schemas on demand, so the agent has to go+looking before it can call anything, and frequently never does.++**Treat that column as unstable, including our own 15 of 15.** Reruns on later days+returned 4 of 15 and then 3 of 15 for us, and 2 of 14 for CodeGraph.++**Is the collapse the model or the harness? Inconclusive.** The same 15 questions on+Opus, against bands fixed before the run (12+ means the model, 6 or fewer means+schema deferral, 7 to 11 inconclusive) **came back at 7**. Opus goes looking on 11+of 15 and declines about a third of the times it looks, so deferral is part of the+story and not all of it. **That run's token column fails its own control** (-9.3%+when the tool was called against -10.7% when it was not), so no token figure is+quoted from it for any tool including ours.++### Third harness: a small model on your own hardware++`qwen3:8b` under Ollama, driven by opencode, same 15 django questions, same seed.+Inference cost is zero, so the question becomes "is it faster, and is it better".++**Every cell called its tool: 15 of 15 on both rows**, against 7 of 15 for Opus on+this identical draw. Adoption ordering across four instruments is Sonnet 3 to 4,+Opus 7, Codex 15, local `qwen3:8b` 15.++| Row | Agent used it | Output tokens | vs bare | Leaner on | p | Wall clock | vs bare |+|---|---:|---:|---:|---:|---:|---:|---:|+| **repowise, full surface** | **15 / 15** | **1,319** | **-40.8%** | **15 of 15** | **0.00006** | **117s** | **-27.5%** |+| **repowise, local-only tools** | **15 / 15** | **1,172** | **-47.9%** | **15 of 15** | **0.00006** | **96s** | **-41.5%** |+| *bare agent (control)* | 0 / 15 | 2,336 | baseline | n/a | n/a | 171s | baseline |++**Two repowise rows, never combined.** `get_answer` writes its answer using a hosted+model, so a row using it is not a local-only result. The second row switches it off+and leaves only tools that run against the local index. We verified the restriction+holds rather than assuming it: instructed directly and repeatedly to call+`get_answer`, that agent could not reach it in any of its 15 cells.++**On a local model the win shows up as time, and the mechanism differs from the+hosted case.** repowise roughly doubles the tokens fed in on a single step while+cutting steps from 3.3 to 2.1 and halving what the model writes. Reading a large+payload once is cheap on a GPU; generating text token-by-token across several+rounds is not.++**We are not claiming a quality improvement from this run.** The full surface scored++1.32 against the bare agent on a 0 to 10 judge scale, above the judge's measured+0.69 noise, but the win-loss count is 10 to 5 at p = 0.30 and removing the single+best question drops it to +0.99. The local-only row is **+0.20, a null**. The+defensible statement is narrower: **the local-only configuration answers about as+well as a bare agent in roughly half the wall clock, on hardware you already own.**++### What this section will not claim++- **This is work saved, not quality.** A blind judge scored every tool in the field,+ ours included, a fraction **below** the bare agent on the 48-question run, in a+ range of 0.04 to 0.25 points on a 10-point scale. All are smaller than the 0.69+ points by which this benchmark moves when rerun unchanged. No tool here measurably+ changed answer quality in either direction. Ours is at the low end of that band+ and we will say so if it becomes a real effect. "No significant difference" is not+ parity, and an equivalence claim needs a test we have not run.+- **The quality columns cannot be compared across tables.** Different judges, because+ grading a model with a judge from its own family is a known bias.+- **Not a universal saving.** One repository, one commit, one prompt. Three+ harnesses, which is more than most published numbers in this category, and still+ not a constant.+- **Adoption is not a stable property of a tool.** Whether an agent calls a codebase+ server at all depends more on the harness than on the server. Any adoption figure,+ ours included, is only meaningful with its harness and its date attached.+- **No dollar figure**, and the control that retired it is the most useful thing we+ ran. code-review-graph, having never called its server once in 15 questions and+ carrying 28,118 extra characters of schema that should cost *more*, measured+ **43% cheaper than the bare agent**. The cause is prompt caching: whichever arm+ runs first warms it. An arm's position in the cycle correlated **-0.487** with its+ dollar cost. Output tokens are never cached and correlate **+0.010**, so those are+ what we publish. If you see a token claim here that does not say whether it+ controls for cache state and arm ordering, ask about that first.++**What producing these tables cost: 471 agent runs, about 13 hours of machine time,+roughly $44 of API spend**, plus 11.3 minutes of fresh index builds for every tool.+Roughly a third of those runs are gates and repeats rather than headline numbers.+Full breakdown, and the per-tool setup traps that were most of the real effort, in+[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head).++Raw data, every cell including failures:**[rung9](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08/rung9)**-(the 48-question Codex run) and+(48-question Codex) and**[rung6](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08/rung6)**-(the 15-question runs and the proof-of-life checks).+(the 15-question runs).---## 3. Loading one commit's context, the easy numberThis is the measurement almost every tool in this category publishes, and we are-labelling it as such. It is a real measurement. It is not section 2, and it-should not be read as though it were.--How many tokens does it take to load one commit's context? Measured over the 30-most recent non-merge commits of `pallets/flask`, counted with deterministic-`tiktoken` (`cl100k_base`):+labelling it as such. Over the 30 most recent non-merge commits of `pallets/flask`,+counted with deterministic `tiktoken` (`cl100k_base`):| Strategy | Tokens per commit ||---|---:|@@ -413,22 +285,21 @@ most recent non-merge commits of `pallets/flask`, counted with deterministic| `git diff` only | 1,408 || **`get_context`** | **393** |-**35.6x fewer than naive, pooled**, 29.3x as a mean of per-commit ratios, and-3.6x pooled against `git diff`.+**35.6x fewer than naive, pooled**, 29.3x as a mean of per-commit ratios, and 3.6x+pooled against `git diff`.-**Lead with the pooled figure.** Pooled is sum-of-tokens over sum-of-tokens, so-it weights each commit by the tokens actually at stake. A mean of per-commit-ratios does not: a one-line commit where `get_context` returns 40 tokens-contributes a huge ratio that counts equally against a commit saving a hundred-thousand. That is exactly how a 35x becomes a 209x in a press release, and it is-why the pooled number is the one we print first.+**Lead with the pooled figure.** Pooled is sum over sum, so it weights each commit by+the tokens actually at stake. A mean of per-commit ratios does not: a one-line commit+where `get_context` returns 40 tokens contributes a huge ratio that counts equally+against one saving a hundred thousand. That is how a 35x becomes a 209x in a press+release.-**What this number does not tell you.** It is one payload, not a session. It says-our representation of a commit is smaller than the commit. It does not say an-agent finishes faster, and the honest version of that question is section 2,-where the answer is a much more modest 15.9%.+**What this does not tell you.** It is one payload, not a session. It says our+representation of a commit is smaller than the commit, not that an agent finishes+faster. The honest version of that question is §2, where the answer on the same+harness is a much more modest 15.9%.-Raw CSV committed alongside the harness:+Raw CSV:**[token\_efficiency\_flask30](https://github.com/repowise-dev/repowise-bench/blob/master/results/bakeoff_2026_08/rung1/token_efficiency_flask30_2026-08-01.csv)**---@@ -443,30 +314,26 @@ errors first, exit code preserved, every omission recoverable through an inline|---|---:|---:|---|| `pytest -q` (11 failures) | 3,374 | 1,317 | **61%**, all 11 `FAILED` lines kept || `git log -50` | 3,064 | 331 | **89%** |-| `git diff` (30 commits) | 62,833 | 8,635 | **86%** |+| `git diff` (30 commits) | 62,833 | 8,635 | 86%, **unsupported, see below** || `git log --oneline -30` | 321 | 321 | 0%, already compact || `git status` (clean) | 83 | 83 | 0%, too small to distill |The two 0% rows are the net-positive guard working: distill never inflates small-output. One run per command on one repository, so these are point measurements,-not a distribution. Reduction is also not comprehension: the bytes removed are-measured, and the evidence they were safe to remove is narrower, being preserved-failure lines plus CI-asserted zero-error-line-loss fixtures.+output. One run per command on one repository, so these are point measurements.+Reduction is also not comprehension: the bytes removed are measured, the evidence+they were safe to remove is narrower.++**One row above is inflated, by the mechanism that broke RTK's number.** A saving may+only count output the agent would actually have received, and the host truncates a+command result at 30,000 characters. `git diff`'s 62,833 raw tokens is eight times+that cap and predates it. **Treat that 86% as unsupported until re-measured.**+`pytest` and `git log` sit under the cap and are unaffected.**There is a comparable tool and we have not run it.**-[RTK](https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings/) does-this and essentially only this. It is also the tool in the table at the top of-this page whose advertised 60 to 90% came back **7.6% more expensive** under-JetBrains' rerun. So the table above is a before-and-after of our own output, not-a head to head, and a before-and-after is the weaker design. Only a paired run-against RTK would settle it.--**One row above is inflated, by the mechanism that broke RTK's number.** A saving-may only count output the agent would actually have received, and a host-truncates a command result at 30,000 characters. The engine applies that cap;-`git diff`'s 62,833 raw tokens is eight times it and predates it. **Treat that-86% as unsupported until re-measured.** `pytest` and `git log` sit under the cap-and are unaffected.+[RTK](https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings/) does this+and essentially only this. It is also the tool whose advertised 60 to 90% came back+**7.6% more expensive** under JetBrains' rerun. So this table is a before-and-after+of our own output, which is the weaker design. Only a paired run would settle it.Full guide: **[docs/agent/DISTILL.md](agent/DISTILL.md)**@@ -479,23 +346,22 @@ break. Scores are taken at a historical commit, bug fixes are counted over thefollowing six months, and nothing after the scoring commit feeds the score.Across **21 repositories, 9 languages, 2,826 files**: **ROC AUC 0.737**-(95% CI 0.683 to 0.787), ranging from 0.55 to 0.86 across individual repos. It-beats recent churn by +0.100 AUC and prior-defect history by +0.117 (DeLong-p < 1e-9). On PROMISE/jEdit, a dataset it never saw and which carries no git-history at all, it holds at 0.76 to 0.78.--It is **not** better than raw file size at discrimination: LOC-only scores 0.742-against our 0.737 (p = 0.92, a tie). Where it wins is effort-aware ranking,-Popt +0.134 (95% CI +0.080 to +0.198) — same discrimination as counting lines,-much better at ordering a fixed review budget, and unlike a line count it says-why. Holding size fixed, within-band AUC runs 0.525 / 0.572 / 0.593 / 0.718-across NLOC quartiles: the signal survives cleanly only in the largest quartile,-and a purpose-built positive control confirms that collapse is a real absence-rather than too few positives to detect one.--**Against CodeScene**, the closest commercial product and the only other vendor-in this category with a published empirical defect study. Both tools scored the-same 2,770 files at the same leakage-free commit against the same labels:+(95% CI 0.683 to 0.787), ranging 0.55 to 0.86 across individual repos. It beats+recent churn by +0.100 AUC and prior-defect history by +0.117 (DeLong p < 1e-9). On+PROMISE/jEdit, a dataset it never saw which carries no git history at all, it holds+at **0.76 to 0.78**.++**It is not better than raw file size at discrimination.** LOC-only scores 0.742+against our 0.737 (p = 0.92, a tie). Where it wins is effort-aware ranking, Popt++0.134 (95% CI +0.080 to +0.198): same discrimination as counting lines, much better+at ordering a fixed review budget, and unlike a line count it says why. Holding size+fixed, within-band AUC runs 0.525 / 0.572 / 0.593 / 0.718 across NLOC quartiles, so+the signal survives cleanly only in the largest quartile, and a positive control+confirms that collapse is a real absence rather than too few positives to detect one.++**Against CodeScene**, the closest commercial product and the only other vendor in+this category with a published empirical defect study. Both tools scored the same+2,770 files at the same leakage-free commit against the same labels:| Paired test | repowise | CodeScene | ||---|---:|---:|---|@@ -505,34 +371,23 @@ same 2,770 files at the same leakage-free commit against the same labels:| Discrimination (ROC AUC) | 0.731 | 0.705 | p = 0.054, marginal || Precision at a 20%-of-lines review budget | 0.580 | **0.636** | p = 0.64, a tie |-Ranking by repowise health surfaces **2.3x the defects under a fixed review-budget**, at Popt +0.144 and recall +0.098, both p = 0.003, paired.--**Where CodeScene is ahead, and why it is a real choice.** Its nominal precision-lead is not statistically significant, but the behaviour behind it is worth-having: CodeScene flags about **27 files** where we flag **132**. That is a more-conservative threshold, trading recall for a short list a team will actually work-through. If what you want is a handful of files to fix this quarter rather than-the ranking that catches the most defects, that operating point is the better-one, and it is a deliberate design choice rather than a weaker model. Our AUC-edge is also **marginal**, not significant at 0.05, and we would rather say so-than round it up. So is our raw defect-density lead: 16.9x against 14.2x, but-p = 0.65 on a heavy tail. The size-normalized version of that row, 2.18x against-0.56x, is the one that reaches significance and the one to cite.--**The axis we lost outright.** CodeScene's "Code Red" study reports a Pearson-correlation of **−0.58** between Code Health and mean issue-resolution time, on-proprietary Jira cycle-time data. We tried to replicate that on open data and-could not. Across 17 repositories and 271 files, GitHub PR merge time correlated-with health at **−0.09** (95% CI −0.19 to +0.14), and six queue-independent-effort signals — commit span, commit count, review rounds, changes-requested,-commits after first review, review-comment density — all came back flat with-every interval spanning zero. The likely reason is structural: GitHub merge time-largely measures maintainer review-queue availability, not how hard a change was.-An early three-repo slice did show a review-rounds correlation of about −0.29,-and it did not survive expanding the corpus, so we treat it as small-sample-noise. **The business-impact axis remains CodeScene's, unreplicated on open-data.**+Ranking by repowise health surfaces **2.3x the defects under a fixed review budget**.++**Where CodeScene is ahead, and why it is a real choice.** Its precision lead is not+significant, but the behaviour behind it is worth having: it flags about **27 files**+where we flag **132**, trading recall for a short list a team will actually work+through. That is a deliberate operating point, not a weaker model. Our AUC edge is+**marginal**, and so is our raw defect-density lead (16.9x against 14.2x, p = 0.65 on+a heavy tail); the size-normalized version is the one that reaches significance and+the one to cite.++**The axis we lost outright.** CodeScene's "Code Red" study reports a **-0.58**+correlation between Code Health and mean issue-resolution time, on proprietary Jira+data. We could not replicate it on open data: across 17 repositories and 271 files,+GitHub PR merge time correlated at **-0.09** (95% CI -0.19 to +0.14), and six+queue-independent effort signals all came back flat with every interval spanning+zero. Merge time on GitHub largely measures maintainer review-queue availability.+**The business-impact axis remains CodeScene's, unreplicated on open data.**Reports:**[BENCHMARK\_REPORT.md](https://github.com/repowise-dev/repowise-bench/blob/master/health-defect/BENCHMARK_REPORT.md)** ·@@ -543,10 +398,7 @@ Reports:## 6. Indexing time, the row we loseWe are the slowest indexer in the field, on every repo we measured, and it is not-close. **The reason is not an optimisation we forgot: the tools we are measured-against build a call graph, and in the same pass we also mine git history,-generate and embed documentation, extract decision records and score code-health.** On `django/django`:+close. On `django/django`:| Tool | Index time | What it builds ||---|---:|---|@@ -556,29 +408,32 @@ health.** On `django/django`:| **repowise** (`--no-prose`) | **366.8s** | five layers, below || **repowise** (default, prose on) | **1,058s** | five layers plus generated documentation |-That is **22x** CodeGraph like for like, and **135x** with prose on, which is what-a default `repowise init` actually costs you. Both numbers ship, and the 22x is-not the user-facing one.--**Here is what the extra time buys.** The tools above build a call graph. In the-same run, repowise builds five layers over the same codebase:+That is **22x** CodeGraph like for like, and **135x** with prose on, which is what a+default `repowise init` actually costs you. Both numbers ship, and the 22x is not the+user-facing one.-| Layer | On django, in that run |-|---|---|-| **Graph** | 36,485 nodes, 90,477 edges, 31,384 symbols, plus PageRank, betweenness, Leiden communities and execution-flow tracing |-| **Git** | full history mined across 2,630 files: hotspots, ownership, co-change pairs, bus factor |-| **Documentation** | 3,392 wiki pages rendered and embedded for natural-language search |-| **Decisions** | architectural decision records mined from history and sessions |-| **Code health** | 5,317 findings, plus 155 unreachable files and 98 unused exports |+**Here is what the extra time buys.** The tools above build a call graph. In the same+run, repowise builds: a **graph** of 36,485 nodes, 90,477 edges and 31,384 symbols+plus PageRank, betweenness, Leiden communities and execution-flow tracing; **git**+history mined across 2,630 files for hotspots, ownership, co-change pairs and bus+factor; **3,392 wiki pages** rendered and embedded for natural-language search;+**architectural decisions** mined from history and sessions; and **code health**,+5,317 findings plus 155 unreachable files and 98 unused exports.So "22x slower" and "the index contains categorically more" are both true, and-neither one cancels the other. If all you want is a call graph, CodeGraph builds-one in 16 seconds and you should use it. The comparison that would be dishonest-is quoting the ratio without the column beside it, which is why the column is-here.+neither cancels the other. If all you want is a call graph, CodeGraph builds one in+16 seconds and you should use it. The comparison that would be dishonest is quoting+the ratio without the column beside it, which is why the column is here.It is also a one-time cost. Updates after the first index are incremental.+A fitted build-cost curve for five tools across a 12x repository-size range, which+nobody in this field had published, is in+[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head#the-result-nobody-in-the-field-had-published-a-build-cost-curve).+Our exponent is sublinear at 0.906, and we publish that as sublinear **work** rather+than efficiency, because symbol density halves across the range. That is a product+finding against us.+---## Limits@@ -588,41 +443,96 @@ Beyond the ones stated in each section:- **Every number here is Python or Go.** A JavaScript/TypeScript corpus(`mui/material-ui`, six tools, a 12x size range) is built and half graded, but**its sealed half is unrun and nothing from it is quoted here.** Publishing the- development half is what that split exists to prevent, so the row arrives when the sealed- half is evaluated, once.-- **We index code files only, which depresses our own coverage on repositories- with documentation.** Deliberate: on doc-heavy repositories the docs outweigh- the code, and indexing them polluted the index and degraded retrieval for the- code questions the tool exists to answer. The cost is invisible in a coverage- figure. On the JavaScript corpus above, **21% of gold files are `.md` or- `.json`**, unreachable to us by construction rather than by ranking. A tool- making the opposite trade retrieves some of them and pays for it on code.-- **§2 is one repository**, `django/django` at one commit, which is in every- model's training data.-- **§2 measures single-session questions**, roughly nine turns each. Nothing on- this page measures a long multi-hour engineering task.-- **§1 quotes only the sealed half.** Pooling both halves gives a stronger p and- we do not quote it. The other half appears in §1 as a check on the sealed- number, never as the result.-- **§2's difficulty split is post-hoc.** The median split was chosen after seeing- the data. The pre-registered comparisons on this page are stronger evidence.-- **§4 has no head-to-head and one row that needs re-measuring.** See that- section.--## Method and provenance--The method, drawn end to end with every gate and the failure that put it there,-is **[THE\_LOOP.md](https://github.com/repowise-dev/repowise-bench/blob/master/head-to-head/THE_LOOP.md)**.-One page per tool in the field, with its setup traps, is-**[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head)**.-The pre-registration files with their commit timestamps, the arm-parity rules and-the statistical tests are in-**[repowise-bench](https://github.com/repowise-dev/repowise-bench)**, which also-holds every raw run permanently, including the invalidated ones with their-invalidation notes attached.+ development half is what that split exists to prevent, so the row arrives when the+ sealed half is evaluated, once.+- **We index code files only, which depresses our own coverage on repositories with+ documentation.** Deliberate: on doc-heavy repositories the docs outweigh the code,+ and indexing them polluted the index and degraded retrieval for the code questions+ the tool exists to answer. On the JavaScript corpus above, **21% of gold files are+ `.md` or `.json`**, unreachable to us by construction rather than by ranking.+ cocoindex makes the opposite trade, is the only arm of six to retrieve any+ documentation gold, and pays for it on code gold.+- **§2 is one repository**, `django/django` at one commit, which is in every model's+ training data.+- **§2 measures single-session questions**, roughly four to nine turns each. Nothing+ on this page measures a long multi-hour engineering task, and we will not imply a+ number for one.+- **§1 quotes only the sealed half.** Pooling both halves gives a stronger p and we+ do not quote it.+- **§2's difficulty split is post-hoc.** The pre-registered comparisons are stronger+ evidence.+- **§4 has no head-to-head and one row that needs re-measuring.**+- **§5's signal is weak among files of similar size**, and a prior-defects baseline+ still beats us on Popt by 0.085 even while losing on AUC.++---++## How to read a number on this page++In July 2026 JetBrains took two popular token-saving tools and reran their headline+claims on real agent work.++| Tool | Advertised saving | What JetBrains measured |+|---|---|---|+| [Caveman](https://blog.jetbrains.com/ai/2026/07/speak-to-ai-agents-like-cavemen-tosave-tokens/) | 65% | **8.5%** |+| [RTK](https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings/) | 60 to 90% | **7.6% more expensive** at low reasoning effort (p = 0.004) |++Greptile's 82% became 45% under Augment's rerun. The pattern is consistent enough to+be a rule: in this category, the advertised number and the reran number are different+numbers. They differ for one reason.++**Measuring one context load is easy. Measuring an agent loop is hard.** Showing that+your representation of a file is smaller than the file is a real measurement, and the+one almost everybody publishes. It is not the question anyone has, which is whether+an agent given your tool finishes the job having done less work. Agents re-read,+backtrack, re-plan and re-explore, so a compression that looks like 90% on one+payload routinely nets out near zero across a session, and can go negative when the+agent works harder to recover what you compressed away.++**This page publishes both and labels which is which.** §3 is the easy+one-context-load figure. §2 is the agent-loop figure, measured the way JetBrains+measured, and deliberately the smaller number. We reran our own headline on a second+and then a third harness and published what came back: it moved a number we had been+quoting, and §2 carries the correction.++Four rules apply to every number here, and each exists because breaking it produced a+wrong published figure at least once:++- **Pre-register before spending**, as its own commit, so a favourable result cannot+ become a different question afterwards.+- **Seal a half**, split by instance id before any work begins, evaluated once.+- **Precision and files-served beside coverage**, never averaged into one figure.+- **Prove an arm was alive and its extractor works before recording a zero.** A dead+ server, a wrong tool name and a broken output parser all score exactly like a bad+ tool.++The full method, drawn end to end with every gate and the failure that put it there,+is+**[THE\_LOOP.md](https://github.com/repowise-dev/repowise-bench/blob/master/head-to-head/THE_LOOP.md)**.++---++## Where the depth is++This page is the summary. Stop wherever you like; each level is the evidence for the+one above it.++| Level | What is there |+|---|---|+| **[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head)** | Who wins what, the build-cost curve, and what each index can rank at all |+| **[arms/](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head/arms)** | One page per competitor: what it is, what it serves, and every setup trap. Four of six have a step that produces a clean zero when missed |+| **[THE\_LOOP.md](https://github.com/repowise-dev/repowise-bench/blob/master/head-to-head/THE_LOOP.md)** | The method and all nine gates, each named with the failure that created it |+| **[configs/arms.yaml](https://github.com/repowise-dev/repowise-bench/blob/master/configs/arms.yaml)** | Every launch command, allowlisted tool and exclusion with its reason. Read this if you think an arm was set up unfairly |+| **[configs/\*.PREREGISTRATION.md](https://github.com/repowise-dev/repowise-bench/tree/master/configs)** | One per scored run, each committed before its run spent anything |+| **[results/bakeoff\_2026\_08](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08)** | Every graded cell and every verbatim response, including the runs we invalidated, with their invalidation notes attached |+| **[repro/README.md](https://github.com/repowise-dev/repowise-bench/blob/master/repro/README.md)** | Per claim: what it costs to reproduce, how long it takes, and which ones need credentials we cannot hand you |++**Found a problem with one of these numbers, or want your tool in the field?** Adding+a competitor is a YAML block, no Python and no runner change. See+[CONTRIBUTING.md](https://github.com/repowise-dev/repowise-bench/blob/master/CONTRIBUTING.md).Tool versions as measured: CodeGraph 1.5.0, Graphify 0.9.31, Serena 1.6.2.dev0,-code-review-graph 2.3.7.+code-review-graph 2.3.7, cocoindex as of 2026-08-09.## See also