Skip to content

Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.

docs: put the numbers where people actually look (#1365)

Raghav Chamadiya81d46314634f
diff --git a/.github/assets/bench/agent-output-tokens-dark.svg b/.github/assets/bench/agent-output-tokens-dark.svg
new file mode 100644
index 000000000..b6895a6a5
--- /dev/null
+++ b/.github/assets/bench/agent-output-tokens-dark.svg
@@ -0,0 +1,36 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="720" height="324" viewBox="0 0 720 324" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,Arial,sans-serif" role="img" aria-label="Work saved in a real agent loop">
+<rect width="720" height="324" fill="none"/>
+<text x="0" y="20" font-size="15" font-weight="650" fill="#E2E8F0">Work saved in a real agent loop</text>
+<text x="0" y="40" font-size="12" fill="#94A3B8">Output tokens the agent itself writes to reach an answer. Lower is better.</text>
+<text x="186" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#E2E8F0">repowise</text>
+<rect x="196" y="62" width="396" height="24" rx="3" fill="#18202E"/>
+<rect x="196" y="62" width="270.8" height="24" rx="3" fill="#F59520"/>
+<text x="654" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#E2E8F0">1,250</text>
+<text x="716" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#E2E8F0">-31.6%</text>
+<text x="186" y="116" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">CodeGraph</text>
+<rect x="196" y="100" width="396" height="24" rx="3" fill="#18202E"/>
+<rect x="196" y="100" width="299.6" height="24" rx="3" fill="#334155"/>
+<text x="654" y="116" text-anchor="end" font-size="12.5" font-weight="400" fill="#E2E8F0">1,383</text>
+<text x="716" y="116" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">-24.4%</text>
+<text x="186" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">Serena</text>
+<rect x="196" y="138" width="396" height="24" rx="3" fill="#18202E"/>
+<rect x="196" y="138" width="335.8" height="24" rx="3" fill="#334155"/>
+<text x="654" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#E2E8F0">1,550</text>
+<text x="716" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">-14.8%</text>
+<text x="186" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">Graphify</text>
+<rect x="196" y="176" width="396" height="24" rx="3" fill="#18202E"/>
+<rect x="196" y="176" width="359.2" height="24" rx="3" fill="#334155"/>
+<text x="654" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#E2E8F0">1,658</text>
+<text x="716" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">-8.9%</text>
+<text x="186" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">code-review-graph</text>
+<rect x="196" y="214" width="396" height="24" rx="3" fill="#18202E"/>
+<rect x="196" y="214" width="370.4" height="24" rx="3" fill="#334155"/>
+<text x="654" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#E2E8F0">1,710</text>
+<text x="716" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">-6.0%</text>
+<text x="186" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">bare agent (control)</text>
+<rect x="196" y="252" width="396" height="24" rx="3" fill="#18202E"/>
+<rect x="196" y="252" width="396.0" height="24" rx="3" fill="none" stroke="#64748B" stroke-width="1.5" stroke-dasharray="4 3"/>
+<text x="654" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#E2E8F0">1,828</text>
+<text x="716" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">baseline</text>
+<text x="0" y="316" font-size="11" fill="#94A3B8">48 questions on django/django under Codex (gpt-5.6-sol), n=43 completed by all six arms. repowise p &lt; 0.0001.</text>
+</svg>
diff --git a/.github/assets/bench/agent-output-tokens.svg b/.github/assets/bench/agent-output-tokens.svg
new file mode 100644
index 000000000..4d06ab5e3
--- /dev/null
+++ b/.github/assets/bench/agent-output-tokens.svg
@@ -0,0 +1,36 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="720" height="324" viewBox="0 0 720 324" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,Arial,sans-serif" role="img" aria-label="Work saved in a real agent loop">
+<rect width="720" height="324" fill="none"/>
+<text x="0" y="20" font-size="15" font-weight="650" fill="#0A0A0A">Work saved in a real agent loop</text>
+<text x="0" y="40" font-size="12" fill="#64748B">Output tokens the agent itself writes to reach an answer. Lower is better.</text>
+<text x="186" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#0A0A0A">repowise</text>
+<rect x="196" y="62" width="396" height="24" rx="3" fill="#F1F5F9"/>
+<rect x="196" y="62" width="270.8" height="24" rx="3" fill="#F59520"/>
+<text x="654" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#0A0A0A">1,250</text>
+<text x="716" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#0A0A0A">-31.6%</text>
+<text x="186" y="116" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">CodeGraph</text>
+<rect x="196" y="100" width="396" height="24" rx="3" fill="#F1F5F9"/>
+<rect x="196" y="100" width="299.6" height="24" rx="3" fill="#CBD5E1"/>
+<text x="654" y="116" text-anchor="end" font-size="12.5" font-weight="400" fill="#0A0A0A">1,383</text>
+<text x="716" y="116" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">-24.4%</text>
+<text x="186" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">Serena</text>
+<rect x="196" y="138" width="396" height="24" rx="3" fill="#F1F5F9"/>
+<rect x="196" y="138" width="335.8" height="24" rx="3" fill="#CBD5E1"/>
+<text x="654" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#0A0A0A">1,550</text>
+<text x="716" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">-14.8%</text>
+<text x="186" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">Graphify</text>
+<rect x="196" y="176" width="396" height="24" rx="3" fill="#F1F5F9"/>
+<rect x="196" y="176" width="359.2" height="24" rx="3" fill="#CBD5E1"/>
+<text x="654" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#0A0A0A">1,658</text>
+<text x="716" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">-8.9%</text>
+<text x="186" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">code-review-graph</text>
+<rect x="196" y="214" width="396" height="24" rx="3" fill="#F1F5F9"/>
+<rect x="196" y="214" width="370.4" height="24" rx="3" fill="#CBD5E1"/>
+<text x="654" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#0A0A0A">1,710</text>
+<text x="716" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">-6.0%</text>
+<text x="186" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">bare agent (control)</text>
+<rect x="196" y="252" width="396" height="24" rx="3" fill="#F1F5F9"/>
+<rect x="196" y="252" width="396.0" height="24" rx="3" fill="none" stroke="#94A3B8" stroke-width="1.5" stroke-dasharray="4 3"/>
+<text x="654" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#0A0A0A">1,828</text>
+<text x="716" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">baseline</text>
+<text x="0" y="316" font-size="11" fill="#64748B">48 questions on django/django under Codex (gpt-5.6-sol), n=43 completed by all six arms. repowise p &lt; 0.0001.</text>
+</svg>
diff --git a/.github/assets/bench/file-coverage-dark.svg b/.github/assets/bench/file-coverage-dark.svg
new file mode 100644
index 000000000..caa1c645d
--- /dev/null
+++ b/.github/assets/bench/file-coverage-dark.svg
@@ -0,0 +1,30 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="720" height="324" viewBox="0 0 720 324" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,Arial,sans-serif" role="img" aria-label="Finding the right files, on 42 sealed instances">
+<rect width="720" height="324" fill="none"/>
+<text x="0" y="20" font-size="15" font-weight="650" fill="#E2E8F0">Finding the right files, on 42 sealed instances</text>
+<text x="0" y="40" font-size="12" fill="#94A3B8">File coverage against gold spans. Higher is better. Deterministic grading, no LLM judge.</text>
+<text x="186" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#E2E8F0">repowise get_answer</text>
+<rect x="196" y="62" width="430" height="24" rx="3" fill="#18202E"/>
+<rect x="196" y="62" width="376.7" height="24" rx="3" fill="#F59520"/>
+<text x="716" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#E2E8F0">0.876</text>
+<text x="186" y="116" text-anchor="end" font-size="12.5" font-weight="650" fill="#E2E8F0">repowise search_codebase</text>
+<rect x="196" y="100" width="430" height="24" rx="3" fill="#18202E"/>
+<rect x="196" y="100" width="319.1" height="24" rx="3" fill="#B4741F"/>
+<text x="716" y="116" text-anchor="end" font-size="12.5" font-weight="650" fill="#E2E8F0">0.742</text>
+<text x="186" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">CodeGraph</text>
+<rect x="196" y="138" width="430" height="24" rx="3" fill="#18202E"/>
+<rect x="196" y="138" width="262.3" height="24" rx="3" fill="#334155"/>
+<text x="716" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#E2E8F0">0.610</text>
+<text x="186" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">Graphify</text>
+<rect x="196" y="176" width="430" height="24" rx="3" fill="#18202E"/>
+<rect x="196" y="176" width="234.8" height="24" rx="3" fill="#334155"/>
+<text x="716" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#E2E8F0">0.546</text>
+<text x="186" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">code-review-graph</text>
+<rect x="196" y="214" width="430" height="24" rx="3" fill="#18202E"/>
+<rect x="196" y="214" width="191.3" height="24" rx="3" fill="#334155"/>
+<text x="716" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#E2E8F0">0.445</text>
+<text x="186" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#94A3B8">cocoindex</text>
+<rect x="196" y="252" width="430" height="24" rx="3" fill="#18202E"/>
+<rect x="196" y="252" width="155.2" height="24" rx="3" fill="#334155"/>
+<text x="716" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#E2E8F0">0.361</text>
+<text x="0" y="316" font-size="11" fill="#94A3B8">n=42 (cocoindex n=41). Sign test vs CodeGraph p = 0.00004. Sealed half, evaluated once.</text>
+</svg>
diff --git a/.github/assets/bench/file-coverage.svg b/.github/assets/bench/file-coverage.svg
new file mode 100644
index 000000000..b8fac1861
--- /dev/null
+++ b/.github/assets/bench/file-coverage.svg
@@ -0,0 +1,30 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="720" height="324" viewBox="0 0 720 324" font-family="-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,Arial,sans-serif" role="img" aria-label="Finding the right files, on 42 sealed instances">
+<rect width="720" height="324" fill="none"/>
+<text x="0" y="20" font-size="15" font-weight="650" fill="#0A0A0A">Finding the right files, on 42 sealed instances</text>
+<text x="0" y="40" font-size="12" fill="#64748B">File coverage against gold spans. Higher is better. Deterministic grading, no LLM judge.</text>
+<text x="186" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#0A0A0A">repowise get_answer</text>
+<rect x="196" y="62" width="430" height="24" rx="3" fill="#F1F5F9"/>
+<rect x="196" y="62" width="376.7" height="24" rx="3" fill="#F59520"/>
+<text x="716" y="78" text-anchor="end" font-size="12.5" font-weight="650" fill="#0A0A0A">0.876</text>
+<text x="186" y="116" text-anchor="end" font-size="12.5" font-weight="650" fill="#0A0A0A">repowise search_codebase</text>
+<rect x="196" y="100" width="430" height="24" rx="3" fill="#F1F5F9"/>
+<rect x="196" y="100" width="319.1" height="24" rx="3" fill="#F6B15C"/>
+<text x="716" y="116" text-anchor="end" font-size="12.5" font-weight="650" fill="#0A0A0A">0.742</text>
+<text x="186" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">CodeGraph</text>
+<rect x="196" y="138" width="430" height="24" rx="3" fill="#F1F5F9"/>
+<rect x="196" y="138" width="262.3" height="24" rx="3" fill="#CBD5E1"/>
+<text x="716" y="154" text-anchor="end" font-size="12.5" font-weight="400" fill="#0A0A0A">0.610</text>
+<text x="186" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">Graphify</text>
+<rect x="196" y="176" width="430" height="24" rx="3" fill="#F1F5F9"/>
+<rect x="196" y="176" width="234.8" height="24" rx="3" fill="#CBD5E1"/>
+<text x="716" y="192" text-anchor="end" font-size="12.5" font-weight="400" fill="#0A0A0A">0.546</text>
+<text x="186" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">code-review-graph</text>
+<rect x="196" y="214" width="430" height="24" rx="3" fill="#F1F5F9"/>
+<rect x="196" y="214" width="191.3" height="24" rx="3" fill="#CBD5E1"/>
+<text x="716" y="230" text-anchor="end" font-size="12.5" font-weight="400" fill="#0A0A0A">0.445</text>
+<text x="186" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#64748B">cocoindex</text>
+<rect x="196" y="252" width="430" height="24" rx="3" fill="#F1F5F9"/>
+<rect x="196" y="252" width="155.2" height="24" rx="3" fill="#CBD5E1"/>
+<text x="716" y="268" text-anchor="end" font-size="12.5" font-weight="400" fill="#0A0A0A">0.361</text>
+<text x="0" y="316" font-size="11" fill="#64748B">n=42 (cocoindex n=41). Sign test vs CodeGraph p = 0.00004. Sealed half, evaluated once.</text>
+</svg>
diff --git a/README.md b/README.md
index f15db1dfc..1cc491070 100644
--- a/README.md
+++ b/README.md
@@ -43,14 +43,24 @@
### Your AI agent burns most of its budget rediscovering your codebase. Index it once, and it never has to again.
-### One index. A dependency graph, a generated wiki, mined git history, architectural decisions, and a defect-validated health score.
-
-<sub><strong>−31.6%</strong> of an agent's own output tokens across 43 questions (p&lt;0.0001), reached in
-3.8 tool calls where a bare agent needed 7.2 · loading one commit's context costs 393 tokens
-instead of 13,984 raw · defect-risk <strong>ROC AUC 0.737</strong> across 21 repos and 9 languages, scored
-leakage-free, and 0.76–0.78 on a public dataset that played no part in calibration · every
-layer computed with <strong>zero LLM calls</strong>. We publish the rows we lose.
-<a href="docs/BENCHMARKS.md"><strong>See how the others did →</strong></a><br />
+<table align="center">
+<tr>
+<td align="center" width="250"><h2>#1 of 6</h2></td>
+<td align="center" width="250"><h2>−31.6%</h2></td>
+<td align="center" width="250"><h2>97%</h2></td>
+</tr>
+<tr>
+<td align="center" valign="top"><sub><strong>at finding the right files.</strong><br />0.876 file coverage against the<br />next tool's 0.610, on a <strong>sealed</strong><br />42-instance split. <em>p=0.00004</em></sub></td>
+<td align="center" valign="top"><sub><strong>of your agent's own output tokens,</strong><br />reached in 3.8 tool calls where a<br />bare agent needed 7.2. <em>n=43,<br />p&lt;0.0001, leaner on 37 of 44</em></sub></td>
+<td align="center" valign="top"><sub><strong>fewer tokens to load a commit.</strong><br />393 instead of 13,984 raw, counted<br />with deterministic tiktoken across<br />30 commits. <em>35.6x, pooled</em></sub></td>
+</tr>
+</table>
+
+<sub>Measured head to head against the open-source agent-context field, on instances held out
+from every improvement round. Defect risk validated separately at <strong>ROC AUC 0.737</strong>
+across 21 repos and 9 languages, leakage-free. Every layer computed with <strong>zero LLM
+calls</strong>. <strong>We publish the rows we lose</strong>, and we are the slowest indexer here.
+<a href="docs/BENCHMARKS.md"><strong>All of it, including the losses →</strong></a><br />
Free and self-hosted, runs on your machine, and the first index needs no API key.</sub>
<picture>
@@ -527,9 +537,20 @@ opt-in tools, and the full reference: **[docs/agent/MCP_TOOLS.md →](docs/agent
## Measured against the field
-Three numbers, each with its sample size and its test. The tools here are the
-open-source agent-context tools we ran head to head, and the full page carries
-the rows we lose beside the rows we win.
+Six open-source agent-context tools, the same repositories, the same pinned
+commits, the same questions, each one given its own full advertised tool surface.
+The full page carries the rows we lose beside the rows we win.
+
+<div align="center">
+<picture>
+ <source media="(prefers-color-scheme: dark)" srcset=".github/assets/bench/file-coverage-dark.svg" />
+ <img src=".github/assets/bench/file-coverage.svg" alt="File coverage on 42 sealed ContextBench instances: repowise get_answer 0.876, repowise search_codebase 0.742, CodeGraph 0.610, Graphify 0.546, code-review-graph 0.445, cocoindex 0.361" width="100%" />
+</picture>
+<picture>
+ <source media="(prefers-color-scheme: dark)" srcset=".github/assets/bench/agent-output-tokens-dark.svg" />
+ <img src=".github/assets/bench/agent-output-tokens.svg" alt="Output tokens an agent writes to reach an answer across 43 django questions on Codex: repowise 1,250, CodeGraph 1,383, Serena 1,550, Graphify 1,658, code-review-graph 1,710, bare agent 1,828" width="100%" />
+</picture>
+</div>
- **Finds the right files.** 0.876 file coverage against CodeGraph's 0.610 on a
**sealed** 42-instance split, held out from every improvement round. 19 wins,
diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
index 7cd6f0ffe..d306e886d 100644
--- a/docs/BENCHMARKS.md
+++ b/docs/BENCHMARKS.md
@@ -1,92 +1,60 @@
# Benchmarks
-Every number repowise publishes, with its sample size, its test, and a link to
-the raw data. The harnesses and full reports live in
-**[repowise-bench](https://github.com/repowise-dev/repowise-bench)**, and nothing
-here is measured on a private corpus.
+Every number repowise publishes, with its sample size, its test, and a link to the
+raw data. Nothing here is measured on a private corpus. The harnesses, the
+pre-registrations and every graded cell live in
+**[repowise-bench](https://github.com/repowise-dev/repowise-bench)**, which is also
+where the depth is: this page is the summary, that repository is the evidence.
-## Why this page is built the way it is
-
-In July 2026, JetBrains took two popular token-saving tools and reran their
-headline claims on real agent work with Sonnet 5.
-
-| Tool | Advertised saving | What JetBrains measured |
-|---|---|---|
-| [Caveman](https://blog.jetbrains.com/ai/2026/07/speak-to-ai-agents-like-cavemen-tosave-tokens/) | 65% | **8.5%** |
-| [RTK](https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings/) | 60 to 90% | **7.6% more expensive** at low reasoning effort (p = 0.004), and no change at high effort |
-
-Greptile's 82% became 45% under Augment's rerun. The pattern is consistent
-enough to be a rule: in this category, the advertised number and the reran
-number are different numbers.
-
-They are different for a reason that is worth understanding before you read
-anything below, because it is the single most common way a token claim goes
-wrong.
-
-**Measuring one context load is easy. Measuring an agent loop is hard.** You can
-show that your representation of a file is smaller than the file. That is a real
-measurement and it is the one almost everybody publishes. It is also not the
-question anyone actually has, which is whether an agent given your tool finishes
-the job having done less work. Agents re-read, backtrack, re-plan and re-explore.
-A compression that looks like 90% on one payload routinely nets out near zero
-across a real session, and can go negative when the agent has to work harder to
-recover what you compressed away.
-
-**This page publishes both numbers and labels which is which.** Section 3 is our
-one-context-load figure, the easy one. Section 2 is the agent-loop figure,
-measured the way JetBrains measured: real agent, real repository, real tool use,
-against a control. Section 2 is the one that matters, and it is deliberately the
-smaller number.
-
-That is also why this page prints n beside every mean, states which tool
-produced each number, and publishes the rows we lose. It is built to survive a
-rerun, because in this field that is the only property that turns out to matter.
-
-So we reran our own headline ourselves, on a second agent harness, and published
-what came back. It moved a number we had been quoting, and section 2 now
-carries both results and the correction.
-
-## What we measured, and against whom
-
-repowise builds [five intelligence layers](layers/INTELLIGENCE_LAYERS.md) from
-one index, so there is no single competitor to measure against. Different tools
-overlap on different layers, and this table says exactly which.
+**Read this page in one minute:** the two charts below are the two results that
+matter. Everything after them is the sample size, the caveat and the row we lose.
| Layer | Measured against | Result |
|---|---|---|
-| Finding the right files | CodeGraph, Graphify, code-review-graph | [§1](#1-finding-the-right-files) **we win**, n=42 held out, p=0.00004 |
-| Work saved in a real agent loop | CodeGraph, Serena, Graphify, code-review-graph, bare agent | [§2](#2-what-changes-in-a-real-agent-loop) **we win**, n=43 on Codex at p&lt;0.0001, and the only tool to clear the bar on both agent harnesses we tried |
+| Finding the right files | CodeGraph, Graphify, code-review-graph, cocoindex | [§1](#1-finding-the-right-files) **we win**, n=42 held out, p=0.00004 |
+| Work saved in a real agent loop | the same field plus Serena and a bare agent | [§2](#2-what-changes-in-a-real-agent-loop) **we win** on all three agent harnesses we tried, n=43 at p&lt;0.0001 |
| Loading one commit's context | naive file reads, `git diff` | [§3](#3-loading-one-commits-context-the-easy-number) **35.6x** fewer tokens than naive |
| Command-output compression | RTK | [§4](#4-command-output-compression) **not measured head to head** |
-| Code health and defect prediction | CodeScene | [§5](#5-code-health-predicts-defects) **we win**, p=0.003 |
+| Code health and defect prediction | CodeScene | [§5](#5-code-health-predicts-defects) **we win** on recall and effort-aware ranking, p=0.003 |
| Indexing time | CodeGraph, Graphify, code-review-graph | [§6](#6-indexing-time-the-row-we-lose) **we lose**, 22x, because we build four more layers in the same pass |
| Documentation generation | DeepWiki, Google Code Wiki, Swimm | **not measured** |
| PR review | CodeRabbit, Greptile | **not measured** |
The last two rows are capability comparisons, not measurements, and they live in
the [README's feature table](../README.md#how-it-compares-on-capability) where a
-reader can tell the difference. We would rather say "not measured" than let a
-checkmark do a number's job. **§4 carries the same label for the same reason**:
-RTK does exactly what `repowise distill` does, and we have not run the two
-against each other.
+reader can tell the difference. **§4 carries the same label for the same reason:**
+RTK does exactly what `repowise distill` does, and we have not run the two against
+each other. We would rather write "not measured" than let a checkmark do a
+number's job.
+
+---
+
+<div align="center">
+<picture>
+ <source media="(prefers-color-scheme: dark)" srcset="../.github/assets/bench/file-coverage-dark.svg" />
+ <img src="../.github/assets/bench/file-coverage.svg" alt="File coverage on 42 sealed ContextBench instances: repowise get_answer 0.876, repowise search_codebase 0.742, CodeGraph 0.610, Graphify 0.546, code-review-graph 0.445, cocoindex 0.361" width="100%" />
+</picture>
+<picture>
+ <source media="(prefers-color-scheme: dark)" srcset="../.github/assets/bench/agent-output-tokens-dark.svg" />
+ <img src="../.github/assets/bench/agent-output-tokens.svg" alt="Output tokens an agent writes to reach an answer across 43 django questions on Codex: repowise 1,250, CodeGraph 1,383, Serena 1,550, Graphify 1,658, code-review-graph 1,710, bare agent 1,828 baseline" width="100%" />
+</picture>
+</div>
---
## 1. Finding the right files
-Before a tool can save an agent any work, it has to point at the right code.
-This section measures only that.
+Before a tool can save an agent any work, it has to point at the right code. This
+section measures only that.
-**Grading here is deterministic. No LLM judge is involved anywhere in this
-number**, which makes it the most reproducible result on the page. ContextBench
-ships gold file spans; a tool either returns them or it does not.
+**Grading is deterministic. No LLM judge is involved anywhere in this number**,
+which makes it the most reproducible result on the page. ContextBench ships gold
+file spans; a tool either returns them or it does not.
**Every number below comes from instances this work has never seen.** The 112
instances were split 70 / 42 by instance id, **pinned before any of it started**,
and the 42 were kept sealed until the final measurement.
-### The numbers, on the 42 sealed instances
-
| Tool | File coverage | n | Precision | Files served |
|---|---:|---:|---:|---:|
| **repowise** (`get_answer`) | **0.876** | 42 | 0.087 | 19.2 |
@@ -94,23 +62,37 @@ and the 42 were kept sealed until the final measurement.
| CodeGraph | 0.610 | 42 | 0.093 | 14.0 |
| Graphify | 0.546 | 42 | 0.033 | 34.5 |
| code-review-graph | 0.445 | 42 | 0.240 | 5.4 |
+| cocoindex | 0.361 | 41 | 0.092 | 7.1 |
Head to head per instance against CodeGraph: `get_answer` **19 wins, 1 loss, 22
ties, sign test p = 0.00004**. `search_codebase` **13 wins, 3 losses, 26 ties,
p = 0.021**.
**These are two different tools with two different profiles, and we would rather
-say so than average them into one claim.** `get_answer` finds the most, from a
-list of ~19 files. `search_codebase` finds fewer but is the most efficient per
-file served: **0.742 from 8.2 files**, better coverage-per-file than anything
-else in the table. If you are paying by the token, that is the row to read.
-
-### How this number moved, and why it is not benchmark tuning
-
-We first ran this and came **last, at 0.228**. We published that. The cause
-turned out to be a bug: a query-time gate was discarding most candidates before
-ranking ever happened. Fixing that path is what moved the number, and it is a
-fix any user of the tool gets, not a change shaped around these questions.
+say so than average them into one claim.** `get_answer` finds the most, from a list
+of ~19 files. `search_codebase` finds fewer but is the most efficient per file
+served: **0.742 from 8.2 files**, better coverage-per-file than anything else in
+the table. If you are paying by the token, that is the row to read.
+
+**Precision is not our column.** code-review-graph's 0.240 is nearly three times
+ours. Part of that is mechanical, since precision rises for whoever returns fewest
+files, which is exactly why files-served is a column here and not a footnote.
+
+**The cocoindex row is not from the same sitting**, and it is printed rather than
+smoothed away: measured 2026-08-09 against the other five from 2026-08-02 to
+2026-08-06, same instances, same gold spans, same deterministic grading. Its n is
+41 because one named instance never answered even when queried alone, so it is
+excluded rather than counted as a zero, which makes its row *better* than counting
+it would (0.361 against 0.353). Its placing was
+[pre-registered before its index existed](https://github.com/repowise-dev/repowise-bench/tree/master/configs),
+including the prediction that it would come last.
+
+### Why this is not benchmark tuning
+
+We first ran this and came **last, at 0.228**. We published that. The cause was a
+query-time gate discarding most candidates before ranking ever happened. Fixing
+that path is what moved the number, and it is a fix any user gets, not a change
+shaped around these questions.
The check on that claim is the split, and it points the right way:
@@ -121,57 +103,43 @@ The check on that claim is the split, and it points the right way:
| CodeGraph | **0.6093** | **0.6095** |
**Overfitting makes the unseen half score worse. Ours scores better**, on both
-tools. And CodeGraph, which nobody tuned against either half, scores the same on
-both to three decimal places, so the two halves are equally hard and the gap is
-about the tool rather than the questions.
-
-**We do not quote a pooled 112-instance figure**, though it is easy to compute
-and would be 0.835. Averaging the halves loses the only number that matters,
-which is how the tool does on instances it has never seen.
+tools. CodeGraph, which nobody tuned against either half, scores the same on both
+to three decimal places, so the two halves are equally hard and the gap is about
+the tool rather than the questions.
-### What it cost to produce
-
-Every arm builds its **own** index of **every** instance's repository at that
-instance's own `base_commit`. Nothing is shared between arms, nothing is cached
-across instances, and a stale checkout is a wrong answer rather than a fast one.
-Across the rung-8 matrix that is **748 index builds and roughly 78 machine-hours
-of indexing for 1,129 graded (instance, arm) cells.**
+**We do not quote a pooled 112-instance figure**, though it is easy to compute and
+would be 0.835. Averaging the halves loses the only number that matters.
This is retrieval, not task success. It says we find the right files, not that an
-agent using us writes better code. That is what section 2 is for.
-
-Measured on repowise at commit `081a59fa` (between v0.37.0 and v0.38.0).
+agent using us writes better code. That is what §2 is for. Producing it cost
+**748 index builds and roughly 78 machine-hours**, because every arm builds its own
+index of every instance at that instance's own `base_commit`, with nothing shared
+and nothing cached.
-Raw data and harness:
-**[bakeoff\_2026\_08/rung8](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08/rung8)**
+Measured on repowise `081a59fa` (between v0.37.0 and v0.38.0). Raw cells:
+**[rung8](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08/rung8)**.
---
## 2. What changes in a real agent loop
-This is the section modelled on the JetBrains reruns, and the one we would ask a
-skeptic to read first.
+The section a skeptic should read first, and the one modelled on the JetBrains
+reruns described [at the bottom of this page](#how-to-read-a-number-on-this-page).
-Every question in `django/django`'s question set, 48 of them, spanning five
-question shapes. Six arms: repowise, four competing tools, and a bare agent with
-no tools at all. Every arm got a byte-identical prompt, its full advertised tool
-surface, and a freshly built index on the same pinned commit. The bare-agent
-control was verified free of any local hooks, so it is a real control and not a
-contaminated one.
+Every question in `django/django`'s question set, 48 of them, across five question
+shapes. Six arms: repowise, four competing tools, and a bare agent with no tools.
+Byte-identical prompt, full advertised tool surface per arm, a freshly built index
+on the same pinned commit, and a bare control verified free of local hooks.
-Two things have to be true for a tool to be worth mounting. The agent has to
-actually call it, and the loop has to get leaner when it does. One table, both
-questions.
+Two things have to be true for a tool to be worth mounting: the agent has to
+actually call it, and the loop has to get leaner when it does.
-We ran this on two agent harnesses, because the answer turned out to depend on
-the harness as much as on the tools. The main result is on Codex, where every
-tool in the field actually gets used, so the comparison is between the tools
-rather than between agents that ignored them. Claude Code follows as a second
-proof point.
+**We ran this on three agent harnesses, because the answer depends on the harness
+as much as on the tools.**
### The main run: 48 questions on Codex (`gpt-5.6-sol`)
-Every tool called on every question, so this is a like-for-like comparison.
+Every tool called on every question, so this is like-for-like.
| Tool | Agent used it | Output tokens | vs bare agent | Tool calls | Leaner on | p |
|---|---:|---:|---:|---:|---:|---:|
@@ -182,26 +150,30 @@ Every tool called on every question, so this is a like-for-like comparison.
| code-review-graph | 43 / 43 | 1,710 | -6.0% | 7.2 | 26 of 43 | 0.046 |
| *bare agent (control)* | 0 / 44 | 1,828 | baseline | 7.2 | n/a | n/a |
-**repowise leaves the agent with the least work to do, and gets there in the
-fewest steps.** A third less output than working with no tool at all, reached in
-**3.8 tool calls against the bare agent's 7.2**. One answered question replacing
-roughly six greps, visible directly in the call counts rather than inferred.
+A third less output than working with no tool at all, reached in **3.8 tool calls
+against the bare agent's 7.2**, opening 3.0 files instead of 7.2. One answered
+question replacing roughly six greps, visible in the call counts rather than
+inferred.
-Correcting for testing five tools at once, three reductions are solid and two
-are marginal. **CodeGraph is a genuine second at -24.4%**, and the honest reading
-is that we lead a field in which more than one tool works, not that we are the
-only one that does.
+Correcting for testing five tools at once, three reductions are solid and two are
+marginal. **CodeGraph is a genuine second at -24.4%**, and the honest reading is
+that we lead a field in which more than one tool works. Serena is the counter-case:
+it writes less than the bare agent while calling tools **42% more often**. Busier,
+not leaner.
-Serena is the interesting counter-case: it writes less than the bare agent while
-calling tools **42% more often**. Busier, not leaner.
+5 of the 48 questions are missing from every arm equally because the run hit an API
+usage cap. Paired comparisons are unaffected; the figures are over the 43 all six
+arms completed.
-5 of the 48 questions are missing from every arm equally, because the run hit an
-API usage cap near the end. Paired comparisons are unaffected and the figures
-above are over the 43 questions all six arms completed.
+**The saving grows with the work.** Splitting at the median by how much the bare
+agent needed: easier half 27.2%, harder half **34.3%**, correlation +0.379.
+Pre-computed structure replaces exploration, and harder questions contain more
+exploration to replace. That split is post-hoc and the pre-registered comparisons
+are the stronger evidence.
-### The second proof point: 15 questions on Claude Code (`claude-sonnet-5`)
+### Second harness: 15 questions on Claude Code (`claude-sonnet-5`)
-| Tool | Tools advertised | Schema cost (chars) | Agent used it | Output tokens | vs bare agent | Leaner on | p |
+| Tool | Tools advertised | Schema cost (chars) | Agent used it | Output tokens | vs bare | Leaner on | p |
|---|---:|---:|---:|---:|---:|---:|---:|
| **repowise** | 10 | 17,561 | **15 / 15** | **2,420** | **-15.9%** | **12 of 15** | **0.035** |
| CodeGraph | 1 | 1,567 | 13 / 15 | 2,540 | -11.7% | 10 of 15 | 0.302 |
@@ -210,202 +182,102 @@ above are over the 43 questions all six arms completed.
| Graphify | 10 | 5,482 | 3 / 15 | 2,878 | 0.0% | 7 of 15 | 1.000 |
| *bare agent (control)* | 0 | 0 | n/a | 2,877 | baseline | n/a | n/a |
-**repowise is the only tool that clears the bar on both harnesses**, and it is
-the same direction and the same mechanism in each.
-
-The "agent used it" column is doing something different here, and it is the most
-interesting number on this page. Under Claude Code most of these tools were
-barely called at all: code-review-graph never once, Graphify three times in
-fifteen. Nothing was different about the servers, the questions or the indexes
-between the two runs. Claude Code loads MCP tool schemas on demand, so the agent
-has to go looking before it can call anything, and frequently never does.
-
-**Treat that column as unstable, including our own 15 of 15.** Rerunning the
-same setup on later days returned 4 of 15 and then 3 of 15 for us, and 2 of 14
-for CodeGraph. It is a property of the pairing of tool and harness on a given
-day, not of the tool.
-
-### Is that adoption collapse the model or the harness? Inconclusive.
-
-The same 15 questions on Opus, everything else held fixed, against bands set
-before the run: **12 or more means the model**, 6 or fewer means **the harness's
-schema deferral**, 7 to 11 is inconclusive.
-
-**It came back at 7.** Published as inconclusive.
-
-The mechanism is more useful than the verdict. Opus goes looking on **11 of 15**
-against Sonnet's 13 of 30, then declines about a third of the times it looks. So
-schema deferral is part of the story and not all of it. Ordering across the three
-instruments is Sonnet 3 to 4, Opus 7, Codex 15.
-
-**That run's token column also fails its own control**, at -9.3% when the tool
-was called against -10.7% when it was not, so no token figure is quoted from it
-for any tool including ours. Same class of artifact as the 43%-cheaper dollar
-control above, from position and variance at n=15 rather than caching. The Codex
-run is unaffected: different harness, three times the `n`, every tool called on
-every question.
-
-### How to read those tables
-
-**Agent used it** counts the questions where the agent made at least one call the
-server actually answered. A tool the agent never reaches for is not a tool it
-has, whatever the feature list says. code-review-graph advertises 30 tools over
-a built, embedded graph of 40,904 nodes and 380,168 edges, and under Claude Code
-the agent never called it once across 15 questions. Under Codex it called it on
-all 15. **That number is a fact about the pairing of tool and harness, not about
-the tool**, which is why it appears twice above and never as a single figure.
-
-**Output tokens** is how much the agent itself writes to reach an answer: its
-reasoning, its tool calls, its final reply. Lower means it went in a straighter
-line. This is the honest measure of work saved, for reasons in the box below.
-
-**Leaner on** is the plain-language version of the statistic: on how many
-questions did this tool beat the bare agent, head to head. 37 of 44 is not
-something a coin does. Roughly half is.
-
-**Tool calls** is how many separate actions the agent took. It is the clearest
-view of the mechanism: on Codex the repowise agent finished in 3.8 steps where
-the bare agent needed 7.2, and it opened 3.0 files instead of 7.2. Those move
-together because they are the same effect, which is work done once, offline,
-that the agent would otherwise redo on every query.
-
-### Why this section reports tokens and not dollars
-
-Dollar cost per question is the number every tool in this category wants to
-quote, and it is close to meaningless as a measure of a tool. Here is the control
-that convinced us, run in our own harness.
-
-**Under Claude Code, code-review-graph never called its server across all 15
-questions.** It is behaviourally identical to the bare agent, carrying an extra
-28,118 characters of tool schema that should make it cost *more*. Measured on
-dollars, it came out
-**43% cheaper than the bare agent**. A tool that did nothing produced a
-best-in-class saving.
-
-The cause is prompt caching. Cached tokens bill at a fraction of fresh ones, so
-whichever arm happens to run first pays to warm the cache and every arm after it
-reads it cheaply. In our run the correlation between an arm's position in the
-cycle and its dollar cost was **-0.487**. That is not a property of any tool. It
-is a property of the schedule.
-
-**Output tokens are immune to this.** They are never cached, and their
-correlation with run position is **+0.010**, which is nothing. So that is what we
-report. It is a smaller and less impressive number than the dollar figure would
-have been, and it is the one that survives.
-
-If you see a token-savings claim in this category that does not say whether it
-controls for cache state and arm ordering, this is the first thing to ask about.
-
-### Where the saving is largest
-
-The effect is not uniform. **repowise saves more on questions that require
-touching more of the codebase.** Splitting the 48-question Codex run at the
-median, by how much work the bare agent needed:
-
-| | n | Bare agent output | Tokens saved | % saved |
-|---|---:|---:|---:|---:|
-| easier half | 22 | 1,377 | 374 | 27.2% |
-| harder half | 22 | 2,279 | **781** | **34.3%** |
-
-The harder half saves **more than twice as many tokens per question**, and the
-correlation between how much work a question demands and how much we save is
-**+0.379**. The same pattern appeared on the smaller Claude Code run, so it has
-now shown up twice.
-
-The mechanism is that pre-computed structure replaces exploration, and harder
-questions contain more exploration to replace.
-
-**One honest limit.** Every question here is answered in a single session of
-roughly four to seven turns. **We have not measured a long multi-hour task such
-as designing a feature across many files, and we will not imply a number for
-one.** The reasonable expectation is that a saving which scales with exploration
-keeps scaling when there is more exploration to do, but that is an argument from
-mechanism rather than a result, and it stays labelled as one until we run it.
-
-### What we will not claim from this run
-
-- **This is a work-saved result, not a quality result.** A blind judge scored
- every tool in the field, ours included, a fraction **below** the bare agent on
- the 48-question run, in a range of 0.04 to 0.25 points on a 10-point scale.
- None of those differences is statistically distinguishable from zero, and all
- of them are smaller than the 0.69 points by which this benchmark moves when we
- rerun it unchanged. So the correct reading is that no tool here measurably
- changed answer quality in either direction. Ours is at the low end of that
- band, we are watching it, and we will say so if it turns into a real effect.
- "No significant difference" is not the same as parity, and an equivalence
- claim needs a test we have not run.
-- **The two quality columns cannot be compared with each other.** They were
- graded by different judges, because grading a model with a judge from its own
- family is a known bias and avoiding it means the judge changes when the agent
- does. So no quality number in one table may be subtracted from one in the other.
-- **Not a universal saving.** One repository, one commit, one prompt. Two
- harnesses, which is two more than most published numbers in this category, and
- still not a constant. Treat each figure as that configuration's result.
-- **Adoption is not a stable property of a tool.** We used to read the "agent
- used it" column as a design result about how well we had named and shaped our
- tools. It does not support that. Repeating the Claude Code run with nothing
- changed on anyone's side moved us from 15 of 15 to 4 of 15 to 3 of 15, and
- CodeGraph from 13 of 15 to 2 of 14. Switching harness moved every tool in the
- field to called-on-every-question. **Whether an agent calls a codebase server
- at all depends more on the harness than on the server.** Any adoption figure, ours included,
- is only meaningful with its harness and its date attached.
-
-### What producing these tables cost
-
-**471 agent runs, about 13 hours of machine time, roughly $44 of API spend.**
-
-| | runs | wall clock | API spend |
-|---|---:|---:|---:|
-| Codex, 48 questions x 6 tools | 261 | 4.9h | $17.62 |
-| Codex, earlier 15-question run | 90 | 1.7h | $6.08 |
-| Codex, proof-of-life checks and a second language | 30 | 0.5h | $1.58 |
-| Claude Code with Sonnet, 15 questions x 6 tools | 90 | 1.8h | $18.77 |
-| Claude Code with Opus, the promised rerun | 90 | 2.1h | $28.57 |
-
-The two harnesses' dollar figures are not comparable and are given only as what
-each cost to produce: Claude Code reports its own spend, while Codex reports
-token counts only, so its figure is computed from published list rates.
-
-**Before any of that, every tool got a fresh index built from scratch** on the
-same pinned commit, **11.3 minutes in total**: repowise 363.6s, Graphify 102.7s,
-code-review-graph 54.3s, CodeGraph 15.6s. Serena indexes on demand. We are the
-slowest of the four and [say so in §6](#6-indexing-time-the-row-we-lose). Those
-indexes were then reused by every later run, so the second harness cost agent
-time only.
-
-Two things are not in that table and are most of the real effort. **The
-competitor setups**:
-[code-review-graph](https://github.com/repowise-dev/repowise-bench/blob/master/head-to-head/arms/code-review-graph.md)
-alone needs three steps that are not in its README, each of which produces a
-clean, plausible zero when missed, and getting
-[Serena](https://github.com/repowise-dev/repowise-bench/blob/master/head-to-head/arms/serena.md)
-to answer anything at all needs an explicit project activation. A tool scoring
-zero because we set it up wrong is not a result, and finding that out is most of
-what this work is. The setup each tool needs is written up per tool.
-
-**And the checks that come before any number is allowed to count.** Every arm is
-probed before each run to confirm its server actually answers, and a control
-that checks it can also correctly report a tool as unused, so a broken detector
-cannot quietly pass everything. Roughly a third of the total runs above are
-those checks and repeats rather than headline numbers.
-
-Raw data, every cell including the failures:
+The "agent used it" column is the interesting one. Under Claude Code most of these
+tools were barely called at all: code-review-graph never once, Graphify three times
+in fifteen. Nothing differed about the servers, questions or indexes between the
+two harnesses. Claude Code loads MCP tool schemas on demand, so the agent has to go
+looking before it can call anything, and frequently never does.
+
+**Treat that column as unstable, including our own 15 of 15.** Reruns on later days
+returned 4 of 15 and then 3 of 15 for us, and 2 of 14 for CodeGraph.
+
+**Is the collapse the model or the harness? Inconclusive.** The same 15 questions on
+Opus, against bands fixed before the run (12+ means the model, 6 or fewer means
+schema deferral, 7 to 11 inconclusive) **came back at 7**. Opus goes looking on 11
+of 15 and declines about a third of the times it looks, so deferral is part of the
+story and not all of it. **That run's token column fails its own control** (-9.3%
+when the tool was called against -10.7% when it was not), so no token figure is
+quoted from it for any tool including ours.
+
+### Third harness: a small model on your own hardware
+
+`qwen3:8b` under Ollama, driven by opencode, same 15 django questions, same seed.
+Inference cost is zero, so the question becomes "is it faster, and is it better".
+
+**Every cell called its tool: 15 of 15 on both rows**, against 7 of 15 for Opus on
+this identical draw. Adoption ordering across four instruments is Sonnet 3 to 4,
+Opus 7, Codex 15, local `qwen3:8b` 15.
+
+| Row | Agent used it | Output tokens | vs bare | Leaner on | p | Wall clock | vs bare |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| **repowise, full surface** | **15 / 15** | **1,319** | **-40.8%** | **15 of 15** | **0.00006** | **117s** | **-27.5%** |
+| **repowise, local-only tools** | **15 / 15** | **1,172** | **-47.9%** | **15 of 15** | **0.00006** | **96s** | **-41.5%** |
+| *bare agent (control)* | 0 / 15 | 2,336 | baseline | n/a | n/a | 171s | baseline |
+
+**Two repowise rows, never combined.** `get_answer` writes its answer using a hosted
+model, so a row using it is not a local-only result. The second row switches it off
+and leaves only tools that run against the local index. We verified the restriction
+holds rather than assuming it: instructed directly and repeatedly to call
+`get_answer`, that agent could not reach it in any of its 15 cells.
+
+**On a local model the win shows up as time, and the mechanism differs from the
+hosted case.** repowise roughly doubles the tokens fed in on a single step while
+cutting steps from 3.3 to 2.1 and halving what the model writes. Reading a large
+payload once is cheap on a GPU; generating text token-by-token across several
+rounds is not.
+
+**We are not claiming a quality improvement from this run.** The full surface scored
++1.32 against the bare agent on a 0 to 10 judge scale, above the judge's measured
+0.69 noise, but the win-loss count is 10 to 5 at p = 0.30 and removing the single
+best question drops it to +0.99. The local-only row is **+0.20, a null**. The
+defensible statement is narrower: **the local-only configuration answers about as
+well as a bare agent in roughly half the wall clock, on hardware you already own.**
+
+### What this section will not claim
+
+- **This is work saved, not quality.** A blind judge scored every tool in the field,
+ ours included, a fraction **below** the bare agent on the 48-question run, in a
+ range of 0.04 to 0.25 points on a 10-point scale. All are smaller than the 0.69
+ points by which this benchmark moves when rerun unchanged. No tool here measurably
+ changed answer quality in either direction. Ours is at the low end of that band
+ and we will say so if it becomes a real effect. "No significant difference" is not
+ parity, and an equivalence claim needs a test we have not run.
+- **The quality columns cannot be compared across tables.** Different judges, because
+ grading a model with a judge from its own family is a known bias.
+- **Not a universal saving.** One repository, one commit, one prompt. Three
+ harnesses, which is more than most published numbers in this category, and still
+ not a constant.
+- **Adoption is not a stable property of a tool.** Whether an agent calls a codebase
+ server at all depends more on the harness than on the server. Any adoption figure,
+ ours included, is only meaningful with its harness and its date attached.
+- **No dollar figure**, and the control that retired it is the most useful thing we
+ ran. code-review-graph, having never called its server once in 15 questions and
+ carrying 28,118 extra characters of schema that should cost *more*, measured
+ **43% cheaper than the bare agent**. The cause is prompt caching: whichever arm
+ runs first warms it. An arm's position in the cycle correlated **-0.487** with its
+ dollar cost. Output tokens are never cached and correlate **+0.010**, so those are
+ what we publish. If you see a token claim here that does not say whether it
+ controls for cache state and arm ordering, ask about that first.
+
+**What producing these tables cost: 471 agent runs, about 13 hours of machine time,
+roughly $44 of API spend**, plus 11.3 minutes of fresh index builds for every tool.
+Roughly a third of those runs are gates and repeats rather than headline numbers.
+Full breakdown, and the per-tool setup traps that were most of the real effort, in
+[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head).
+
+Raw data, every cell including failures:
**[rung9](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08/rung9)**
-(the 48-question Codex run) and
+(48-question Codex) and
**[rung6](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08/rung6)**
-(the 15-question runs and the proof-of-life checks).
+(the 15-question runs).
---
## 3. Loading one commit's context, the easy number
This is the measurement almost every tool in this category publishes, and we are
-labelling it as such. It is a real measurement. It is not section 2, and it
-should not be read as though it were.
-
-How many tokens does it take to load one commit's context? Measured over the 30
-most recent non-merge commits of `pallets/flask`, counted with deterministic
-`tiktoken` (`cl100k_base`):
+labelling it as such. Over the 30 most recent non-merge commits of `pallets/flask`,
+counted with deterministic `tiktoken` (`cl100k_base`):
| Strategy | Tokens per commit |
|---|---:|
@@ -413,22 +285,21 @@ most recent non-merge commits of `pallets/flask`, counted with deterministic
| `git diff` only | 1,408 |
| **`get_context`** | **393** |
-**35.6x fewer than naive, pooled**, 29.3x as a mean of per-commit ratios, and
-3.6x pooled against `git diff`.
+**35.6x fewer than naive, pooled**, 29.3x as a mean of per-commit ratios, and 3.6x
+pooled against `git diff`.
-**Lead with the pooled figure.** Pooled is sum-of-tokens over sum-of-tokens, so
-it weights each commit by the tokens actually at stake. A mean of per-commit
-ratios does not: a one-line commit where `get_context` returns 40 tokens
-contributes a huge ratio that counts equally against a commit saving a hundred
-thousand. That is exactly how a 35x becomes a 209x in a press release, and it is
-why the pooled number is the one we print first.
+**Lead with the pooled figure.** Pooled is sum over sum, so it weights each commit by
+the tokens actually at stake. A mean of per-commit ratios does not: a one-line commit
+where `get_context` returns 40 tokens contributes a huge ratio that counts equally
+against one saving a hundred thousand. That is how a 35x becomes a 209x in a press
+release.
-**What this number does not tell you.** It is one payload, not a session. It says
-our representation of a commit is smaller than the commit. It does not say an
-agent finishes faster, and the honest version of that question is section 2,
-where the answer is a much more modest 15.9%.
+**What this does not tell you.** It is one payload, not a session. It says our
+representation of a commit is smaller than the commit, not that an agent finishes
+faster. The honest version of that question is §2, where the answer on the same
+harness is a much more modest 15.9%.
-Raw CSV committed alongside the harness:
+Raw CSV:
**[token\_efficiency\_flask30](https://github.com/repowise-dev/repowise-bench/blob/master/results/bakeoff_2026_08/rung1/token_efficiency_flask30_2026-08-01.csv)**
---
@@ -443,30 +314,26 @@ errors first, exit code preserved, every omission recoverable through an inline
|---|---:|---:|---|
| `pytest -q` (11 failures) | 3,374 | 1,317 | **61%**, all 11 `FAILED` lines kept |
| `git log -50` | 3,064 | 331 | **89%** |
-| `git diff` (30 commits) | 62,833 | 8,635 | **86%** |
+| `git diff` (30 commits) | 62,833 | 8,635 | 86%, **unsupported, see below** |
| `git log --oneline -30` | 321 | 321 | 0%, already compact |
| `git status` (clean) | 83 | 83 | 0%, too small to distill |
The two 0% rows are the net-positive guard working: distill never inflates small
-output. One run per command on one repository, so these are point measurements,
-not a distribution. Reduction is also not comprehension: the bytes removed are
-measured, and the evidence they were safe to remove is narrower, being preserved
-failure lines plus CI-asserted zero-error-line-loss fixtures.
+output. One run per command on one repository, so these are point measurements.
+Reduction is also not comprehension: the bytes removed are measured, the evidence
+they were safe to remove is narrower.
+
+**One row above is inflated, by the mechanism that broke RTK's number.** A saving may
+only count output the agent would actually have received, and the host truncates a
+command result at 30,000 characters. `git diff`'s 62,833 raw tokens is eight times
+that cap and predates it. **Treat that 86% as unsupported until re-measured.**
+`pytest` and `git log` sit under the cap and are unaffected.
**There is a comparable tool and we have not run it.**
-[RTK](https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings/) does
-this and essentially only this. It is also the tool in the table at the top of
-this page whose advertised 60 to 90% came back **7.6% more expensive** under
-JetBrains' rerun. So the table above is a before-and-after of our own output, not
-a head to head, and a before-and-after is the weaker design. Only a paired run
-against RTK would settle it.
-
-**One row above is inflated, by the mechanism that broke RTK's number.** A saving
-may only count output the agent would actually have received, and a host
-truncates a command result at 30,000 characters. The engine applies that cap;
-`git diff`'s 62,833 raw tokens is eight times it and predates it. **Treat that
-86% as unsupported until re-measured.** `pytest` and `git log` sit under the cap
-and are unaffected.
+[RTK](https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings/) does this
+and essentially only this. It is also the tool whose advertised 60 to 90% came back
+**7.6% more expensive** under JetBrains' rerun. So this table is a before-and-after
+of our own output, which is the weaker design. Only a paired run would settle it.
Full guide: **[docs/agent/DISTILL.md](agent/DISTILL.md)**
@@ -479,23 +346,22 @@ break. Scores are taken at a historical commit, bug fixes are counted over the
following six months, and nothing after the scoring commit feeds the score.
Across **21 repositories, 9 languages, 2,826 files**: **ROC AUC 0.737**
-(95% CI 0.683 to 0.787), ranging from 0.55 to 0.86 across individual repos. It
-beats recent churn by +0.100 AUC and prior-defect history by +0.117 (DeLong
-p < 1e-9). On PROMISE/jEdit, a dataset it never saw and which carries no git
-history at all, it holds at 0.76 to 0.78.
-
-It is **not** better than raw file size at discrimination: LOC-only scores 0.742
-against our 0.737 (p = 0.92, a tie). Where it wins is effort-aware ranking,
-Popt +0.134 (95% CI +0.080 to +0.198) — same discrimination as counting lines,
-much better at ordering a fixed review budget, and unlike a line count it says
-why. Holding size fixed, within-band AUC runs 0.525 / 0.572 / 0.593 / 0.718
-across NLOC quartiles: the signal survives cleanly only in the largest quartile,
-and a purpose-built positive control confirms that collapse is a real absence
-rather than too few positives to detect one.
-
-**Against CodeScene**, the closest commercial product and the only other vendor
-in this category with a published empirical defect study. Both tools scored the
-same 2,770 files at the same leakage-free commit against the same labels:
+(95% CI 0.683 to 0.787), ranging 0.55 to 0.86 across individual repos. It beats
+recent churn by +0.100 AUC and prior-defect history by +0.117 (DeLong p < 1e-9). On
+PROMISE/jEdit, a dataset it never saw which carries no git history at all, it holds
+at **0.76 to 0.78**.
+
+**It is not better than raw file size at discrimination.** LOC-only scores 0.742
+against our 0.737 (p = 0.92, a tie). Where it wins is effort-aware ranking, Popt
++0.134 (95% CI +0.080 to +0.198): same discrimination as counting lines, much better
+at ordering a fixed review budget, and unlike a line count it says why. Holding size
+fixed, within-band AUC runs 0.525 / 0.572 / 0.593 / 0.718 across NLOC quartiles, so
+the signal survives cleanly only in the largest quartile, and a positive control
+confirms that collapse is a real absence rather than too few positives to detect one.
+
+**Against CodeScene**, the closest commercial product and the only other vendor in
+this category with a published empirical defect study. Both tools scored the same
+2,770 files at the same leakage-free commit against the same labels:
| Paired test | repowise | CodeScene | |
|---|---:|---:|---|
@@ -505,34 +371,23 @@ same 2,770 files at the same leakage-free commit against the same labels:
| Discrimination (ROC AUC) | 0.731 | 0.705 | p = 0.054, marginal |
| Precision at a 20%-of-lines review budget | 0.580 | **0.636** | p = 0.64, a tie |
-Ranking by repowise health surfaces **2.3x the defects under a fixed review
-budget**, at Popt +0.144 and recall +0.098, both p = 0.003, paired.
-
-**Where CodeScene is ahead, and why it is a real choice.** Its nominal precision
-lead is not statistically significant, but the behaviour behind it is worth
-having: CodeScene flags about **27 files** where we flag **132**. That is a more
-conservative threshold, trading recall for a short list a team will actually work
-through. If what you want is a handful of files to fix this quarter rather than
-the ranking that catches the most defects, that operating point is the better
-one, and it is a deliberate design choice rather than a weaker model. Our AUC
-edge is also **marginal**, not significant at 0.05, and we would rather say so
-than round it up. So is our raw defect-density lead: 16.9x against 14.2x, but
-p = 0.65 on a heavy tail. The size-normalized version of that row, 2.18x against
-0.56x, is the one that reaches significance and the one to cite.
-
-**The axis we lost outright.** CodeScene's "Code Red" study reports a Pearson
-correlation of **−0.58** between Code Health and mean issue-resolution time, on
-proprietary Jira cycle-time data. We tried to replicate that on open data and
-could not. Across 17 repositories and 271 files, GitHub PR merge time correlated
-with health at **−0.09** (95% CI −0.19 to +0.14), and six queue-independent
-effort signals — commit span, commit count, review rounds, changes-requested,
-commits after first review, review-comment density — all came back flat with
-every interval spanning zero. The likely reason is structural: GitHub merge time
-largely measures maintainer review-queue availability, not how hard a change was.
-An early three-repo slice did show a review-rounds correlation of about −0.29,
-and it did not survive expanding the corpus, so we treat it as small-sample
-noise. **The business-impact axis remains CodeScene's, unreplicated on open
-data.**
+Ranking by repowise health surfaces **2.3x the defects under a fixed review budget**.
+
+**Where CodeScene is ahead, and why it is a real choice.** Its precision lead is not
+significant, but the behaviour behind it is worth having: it flags about **27 files**
+where we flag **132**, trading recall for a short list a team will actually work
+through. That is a deliberate operating point, not a weaker model. Our AUC edge is
+**marginal**, and so is our raw defect-density lead (16.9x against 14.2x, p = 0.65 on
+a heavy tail); the size-normalized version is the one that reaches significance and
+the one to cite.
+
+**The axis we lost outright.** CodeScene's "Code Red" study reports a **-0.58**
+correlation between Code Health and mean issue-resolution time, on proprietary Jira
+data. We could not replicate it on open data: across 17 repositories and 271 files,
+GitHub PR merge time correlated at **-0.09** (95% CI -0.19 to +0.14), and six
+queue-independent effort signals all came back flat with every interval spanning
+zero. Merge time on GitHub largely measures maintainer review-queue availability.
+**The business-impact axis remains CodeScene's, unreplicated on open data.**
Reports:
**[BENCHMARK\_REPORT.md](https://github.com/repowise-dev/repowise-bench/blob/master/health-defect/BENCHMARK_REPORT.md)** ·
@@ -543,10 +398,7 @@ Reports:
## 6. Indexing time, the row we lose
We are the slowest indexer in the field, on every repo we measured, and it is not
-close. **The reason is not an optimisation we forgot: the tools we are measured
-against build a call graph, and in the same pass we also mine git history,
-generate and embed documentation, extract decision records and score code
-health.** On `django/django`:
+close. On `django/django`:
| Tool | Index time | What it builds |
|---|---:|---|
@@ -556,29 +408,32 @@ health.** On `django/django`:
| **repowise** (`--no-prose`) | **366.8s** | five layers, below |
| **repowise** (default, prose on) | **1,058s** | five layers plus generated documentation |
-That is **22x** CodeGraph like for like, and **135x** with prose on, which is what
-a default `repowise init` actually costs you. Both numbers ship, and the 22x is
-not the user-facing one.
-
-**Here is what the extra time buys.** The tools above build a call graph. In the
-same run, repowise builds five layers over the same codebase:
+That is **22x** CodeGraph like for like, and **135x** with prose on, which is what a
+default `repowise init` actually costs you. Both numbers ship, and the 22x is not the
+user-facing one.
-| Layer | On django, in that run |
-|---|---|
-| **Graph** | 36,485 nodes, 90,477 edges, 31,384 symbols, plus PageRank, betweenness, Leiden communities and execution-flow tracing |
-| **Git** | full history mined across 2,630 files: hotspots, ownership, co-change pairs, bus factor |
-| **Documentation** | 3,392 wiki pages rendered and embedded for natural-language search |
-| **Decisions** | architectural decision records mined from history and sessions |
-| **Code health** | 5,317 findings, plus 155 unreachable files and 98 unused exports |
+**Here is what the extra time buys.** The tools above build a call graph. In the same
+run, repowise builds: a **graph** of 36,485 nodes, 90,477 edges and 31,384 symbols
+plus PageRank, betweenness, Leiden communities and execution-flow tracing; **git**
+history mined across 2,630 files for hotspots, ownership, co-change pairs and bus
+factor; **3,392 wiki pages** rendered and embedded for natural-language search;
+**architectural decisions** mined from history and sessions; and **code health**,
+5,317 findings plus 155 unreachable files and 98 unused exports.
So "22x slower" and "the index contains categorically more" are both true, and
-neither one cancels the other. If all you want is a call graph, CodeGraph builds
-one in 16 seconds and you should use it. The comparison that would be dishonest
-is quoting the ratio without the column beside it, which is why the column is
-here.
+neither cancels the other. If all you want is a call graph, CodeGraph builds one in
+16 seconds and you should use it. The comparison that would be dishonest is quoting
+the ratio without the column beside it, which is why the column is here.
It is also a one-time cost. Updates after the first index are incremental.
+A fitted build-cost curve for five tools across a 12x repository-size range, which
+nobody in this field had published, is in
+[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head#the-result-nobody-in-the-field-had-published-a-build-cost-curve).
+Our exponent is sublinear at 0.906, and we publish that as sublinear **work** rather
+than efficiency, because symbol density halves across the range. That is a product
+finding against us.
+
---
## Limits
@@ -588,41 +443,96 @@ Beyond the ones stated in each section:
- **Every number here is Python or Go.** A JavaScript/TypeScript corpus
(`mui/material-ui`, six tools, a 12x size range) is built and half graded, but
**its sealed half is unrun and nothing from it is quoted here.** Publishing the
- development half is what that split exists to prevent, so the row arrives when the sealed
- half is evaluated, once.
-- **We index code files only, which depresses our own coverage on repositories
- with documentation.** Deliberate: on doc-heavy repositories the docs outweigh
- the code, and indexing them polluted the index and degraded retrieval for the
- code questions the tool exists to answer. The cost is invisible in a coverage
- figure. On the JavaScript corpus above, **21% of gold files are `.md` or
- `.json`**, unreachable to us by construction rather than by ranking. A tool
- making the opposite trade retrieves some of them and pays for it on code.
-- **§2 is one repository**, `django/django` at one commit, which is in every
- model's training data.
-- **§2 measures single-session questions**, roughly nine turns each. Nothing on
- this page measures a long multi-hour engineering task.
-- **§1 quotes only the sealed half.** Pooling both halves gives a stronger p and
- we do not quote it. The other half appears in §1 as a check on the sealed
- number, never as the result.
-- **§2's difficulty split is post-hoc.** The median split was chosen after seeing
- the data. The pre-registered comparisons on this page are stronger evidence.
-- **§4 has no head-to-head and one row that needs re-measuring.** See that
- section.
-
-## Method and provenance
-
-The method, drawn end to end with every gate and the failure that put it there,
-is **[THE\_LOOP.md](https://github.com/repowise-dev/repowise-bench/blob/master/head-to-head/THE_LOOP.md)**.
-One page per tool in the field, with its setup traps, is
-**[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head)**.
-The pre-registration files with their commit timestamps, the arm-parity rules and
-the statistical tests are in
-**[repowise-bench](https://github.com/repowise-dev/repowise-bench)**, which also
-holds every raw run permanently, including the invalidated ones with their
-invalidation notes attached.
+ development half is what that split exists to prevent, so the row arrives when the
+ sealed half is evaluated, once.
+- **We index code files only, which depresses our own coverage on repositories with
+ documentation.** Deliberate: on doc-heavy repositories the docs outweigh the code,
+ and indexing them polluted the index and degraded retrieval for the code questions
+ the tool exists to answer. On the JavaScript corpus above, **21% of gold files are
+ `.md` or `.json`**, unreachable to us by construction rather than by ranking.
+ cocoindex makes the opposite trade, is the only arm of six to retrieve any
+ documentation gold, and pays for it on code gold.
+- **§2 is one repository**, `django/django` at one commit, which is in every model's
+ training data.
+- **§2 measures single-session questions**, roughly four to nine turns each. Nothing
+ on this page measures a long multi-hour engineering task, and we will not imply a
+ number for one.
+- **§1 quotes only the sealed half.** Pooling both halves gives a stronger p and we
+ do not quote it.
+- **§2's difficulty split is post-hoc.** The pre-registered comparisons are stronger
+ evidence.
+- **§4 has no head-to-head and one row that needs re-measuring.**
+- **§5's signal is weak among files of similar size**, and a prior-defects baseline
+ still beats us on Popt by 0.085 even while losing on AUC.
+
+---
+
+## How to read a number on this page
+
+In July 2026 JetBrains took two popular token-saving tools and reran their headline
+claims on real agent work.
+
+| Tool | Advertised saving | What JetBrains measured |
+|---|---|---|
+| [Caveman](https://blog.jetbrains.com/ai/2026/07/speak-to-ai-agents-like-cavemen-tosave-tokens/) | 65% | **8.5%** |
+| [RTK](https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings/) | 60 to 90% | **7.6% more expensive** at low reasoning effort (p = 0.004) |
+
+Greptile's 82% became 45% under Augment's rerun. The pattern is consistent enough to
+be a rule: in this category, the advertised number and the reran number are different
+numbers. They differ for one reason.
+
+**Measuring one context load is easy. Measuring an agent loop is hard.** Showing that
+your representation of a file is smaller than the file is a real measurement, and the
+one almost everybody publishes. It is not the question anyone has, which is whether
+an agent given your tool finishes the job having done less work. Agents re-read,
+backtrack, re-plan and re-explore, so a compression that looks like 90% on one
+payload routinely nets out near zero across a session, and can go negative when the
+agent works harder to recover what you compressed away.
+
+**This page publishes both and labels which is which.** §3 is the easy
+one-context-load figure. §2 is the agent-loop figure, measured the way JetBrains
+measured, and deliberately the smaller number. We reran our own headline on a second
+and then a third harness and published what came back: it moved a number we had been
+quoting, and §2 carries the correction.
+
+Four rules apply to every number here, and each exists because breaking it produced a
+wrong published figure at least once:
+
+- **Pre-register before spending**, as its own commit, so a favourable result cannot
+ become a different question afterwards.
+- **Seal a half**, split by instance id before any work begins, evaluated once.
+- **Precision and files-served beside coverage**, never averaged into one figure.
+- **Prove an arm was alive and its extractor works before recording a zero.** A dead
+ server, a wrong tool name and a broken output parser all score exactly like a bad
+ tool.
+
+The full method, drawn end to end with every gate and the failure that put it there,
+is
+**[THE\_LOOP.md](https://github.com/repowise-dev/repowise-bench/blob/master/head-to-head/THE_LOOP.md)**.
+
+---
+
+## Where the depth is
+
+This page is the summary. Stop wherever you like; each level is the evidence for the
+one above it.
+
+| Level | What is there |
+|---|---|
+| **[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head)** | Who wins what, the build-cost curve, and what each index can rank at all |
+| **[arms/](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head/arms)** | One page per competitor: what it is, what it serves, and every setup trap. Four of six have a step that produces a clean zero when missed |
+| **[THE\_LOOP.md](https://github.com/repowise-dev/repowise-bench/blob/master/head-to-head/THE_LOOP.md)** | The method and all nine gates, each named with the failure that created it |
+| **[configs/arms.yaml](https://github.com/repowise-dev/repowise-bench/blob/master/configs/arms.yaml)** | Every launch command, allowlisted tool and exclusion with its reason. Read this if you think an arm was set up unfairly |
+| **[configs/\*.PREREGISTRATION.md](https://github.com/repowise-dev/repowise-bench/tree/master/configs)** | One per scored run, each committed before its run spent anything |
+| **[results/bakeoff\_2026\_08](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08)** | Every graded cell and every verbatim response, including the runs we invalidated, with their invalidation notes attached |
+| **[repro/README.md](https://github.com/repowise-dev/repowise-bench/blob/master/repro/README.md)** | Per claim: what it costs to reproduce, how long it takes, and which ones need credentials we cannot hand you |
+
+**Found a problem with one of these numbers, or want your tool in the field?** Adding
+a competitor is a YAML block, no Python and no runner change. See
+[CONTRIBUTING.md](https://github.com/repowise-dev/repowise-bench/blob/master/CONTRIBUTING.md).
Tool versions as measured: CodeGraph 1.5.0, Graphify 0.9.31, Serena 1.6.2.dev0,
-code-review-graph 2.3.7.
+code-review-graph 2.3.7, cocoindex as of 2026-08-09.
## See also