Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.
Python aicode-complexitycode-healthcode-intelligencecode-qualitydead-codedeveloper-toolsdocumentationgit-analyticsmcpopen-sourcepythonrefactoringstatic-analysistechnical-debt
docs(benchmarks): reconcile the precision table with the rows it is built from (#1928)
Raghav Chamadiyad416aca99135
diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.mdindex 2bcc9c6ed..c3318756b 100644--- a/docs/BENCHMARKS.md+++ b/docs/BENCHMARKS.md@@ -12,10 +12,15 @@ says **not measured** rather than carrying a checkmark.## The scoreboard+**Six of the thirteen rows below are wins against a named competitor. Two are+losses, one is level, three are not measured, and one is a ratio against no tool+at all.** They sit in one table on purpose: a page that reports only its wins is+marketing with a sample size attached.+| Question | Measured against | Result ||---|---|---|| **Are the call edges true, judged by a compiler** | CodeGraph, codebase-memory-mcp, Graphify, code-review-graph | [**we win**](#is-the-call-graph-correct). No tool that finds as much of the graph gets more of it right, in all 7 cells |-| **Are the call edges true, hand-graded from source** | CodeGraph | [**we win**](#7-edge-precision), 84.8% against 57.0%, 540 rows read on both sides |+| **Are the call edges true, hand-graded from source** | CodeGraph | [**we win**](#7-edge-precision), 85.7% against 58.6%, 560 rows read on both sides || **Finding the right files** | CodeGraph, Graphify, code-review-graph, cocoindex | [**we win**](#1-finding-the-right-files), n=42 held out, p=0.00004 || **Work saved in a real agent loop** | the same field plus Serena and a bare agent | [**we win**](#2-what-changes-in-a-real-agent-loop) on all three agent harnesses we tried, n=43 at p<0.0001 || **Does the health score predict real bugs** | CodeScene | [**we win**](#5-code-health-predicts-defects) on recall and effort-aware ranking, p=0.003, and lose the business-impact axis outright |@@ -28,18 +33,14 @@ says **not measured** rather than carrying a checkmark.| **Documentation generation** | DeepWiki, Google Code Wiki, Swimm | **not measured** || **PR review** | CodeRabbit, Greptile | **not measured** |-**The two cost rows are different questions and are the easiest thing on this page-to misread.** Building the call graph, we are the lightest tool measured and about-as fast as the fastest. Building the *whole index*, we are 22x slower than a tool-that only builds a call graph, because by then we have also built the git history-layer, the wiki, the decisions and the health pass. Both are on the page with their-denominators stated.--The three **not measured** rows are capability comparisons rather than-measurements, and they live in the-[README's feature table](../README.md#how-it-compares-on-capability) where a reader-can tell the difference. We would rather write "not measured" than let a checkmark-do a number's job.+**The two cost rows are different questions and are the easiest thing here to+misread.** Building the call graph, we are the lightest tool measured and about as+fast as the fastest. Building the *whole index*, we are 22x slower than a tool+that only builds a call graph, because by then we have also built the git history,+the wiki, the decisions and the health pass. The **not measured** rows are+capability comparisons rather than measurements, and live in the+[README's feature table](../README.md#how-it-compares-on-capability); we would+rather write that than let a checkmark do a number's job.<div align="center"><picture>@@ -112,11 +113,10 @@ CodeGraph and codebase-memory-mcp, we are the most precise in seven of seven, anthat narrower claim should always carry its label.**What the oracle adds is the price of a bigger map, not an excuse for ours.** On-syft, more than a third of what the coverage leader emits is a call the Go compiler-says does not exist. And the highest precision anywhere on this page, 0.997, comes-from a graph holding **17% of the calls in the repository**; on gitleaks that same-tool stored 4,367 call rows of which **76** resolve to anything, and a graph that-small is very hard to be wrong in and not much use to walk.+syft, more than a third of what the coverage leader emits is a call the Go+compiler says does not exist, and the highest precision anywhere on this page,+0.997, comes from a graph holding **17% of the calls in the repository**. A graph+that small is very hard to be wrong in and not much use to walk.**Graphify's result is the one worth taking seriously** and should not be lumped inwith that: 89% recall against our 95% on gitleaks is a narrow, real trade.@@ -194,41 +194,49 @@ missed:## The same question, hand-graded across nine languages-The oracle covers two languages. This one covers nine, on both sides, by the same-method: 30 rows per language per tool, seed 2026, stratified by resolution-strategy, every row read from source with its imports and enclosing scope open.+**Two sections ask whether the edges are true, because there are two ways to+find out and each covers what the other cannot.** Above, a compiler is the judge+across five tools, which is unarguable but reaches only two languages. Here,+people read the source across nine languages and two tools, which reaches far+wider but is graded by us. Where both exist they agree to within about a point.++30 rows per language per tool, seed 2026, stratified by resolution strategy,+every row read from source with its imports and enclosing scope open.+Java is the one cell read at 40 rows rather than 30, because it was a single+repository and that repository turned out to be an outlier; it was widened to a+second one.| | correct / n | 95% CI ||---|---|---|-| **repowise** | **229/270 = 84.8%** | [80.0, 88.6] |-| **CodeGraph 1.5.0** | **154/270 = 57.0%** | [51.1, 62.8] |+| **repowise** | **240/280 = 85.7%** | [81.1, 89.3] |+| **CodeGraph 1.5.0** | **164/280 = 58.6%** | [52.7, 64.2] || Language | repowise | CodeGraph | ||---|---|---|---|| typescript | 29/30 | 7/30 | separates || go | 29/30 | 29/30 | tie |-| csharp | 28/30 | 20/30 | separates |+| csharp | 30/30 | 20/30 | separates || python | 28/30 | 19/30 | separates || kotlin | 27/30 | 13/30 | separates || swift | 23/30 | 19/30 | tie |-| cpp | 23/30 | 16/30 | tie |+| cpp | 22/30 | 16/30 | tie || rust | 22/30 | 13/30 | tie |-| java | 20/30 | 18/30 | tie |+| java *(n=40)* | 30/40 | 28/40 | tie |-**Read our number the other way round: roughly fifteen percent of our call edges-are wrong.** That is the number we plan against, and rust, java and cpp are where+**Read our number the other way round: roughly fourteen percent of our call edges+are wrong.** That is the number we plan against, and rust, cpp and java are whereit concentrates.**Four of nine cells separate. Five are ties and we report them as ties.** At n=30the interval runs about ±16 points near 60%, so a point-estimate gap inside two-overlapping intervals is not a win. C++ is a tie despite looking like a 23-point+overlapping intervals is not a win. C++ is a tie despite looking like a 20-pointlead.<details><summary><b>Method, limits, and what this does not show</b></summary>**One repository goes clearly to them.** On `seastar` CodeGraph reads 6/10 against-our 4/10, the only repository in the audit, on any language, where they beat us on+our 5/10, the only repository in the audit, on any language, where they beat us ona clear margin. Our misses there are chained calls on an untyped receiver; theyinfer the callee's declared return type and validate against it, so a failedinference costs them an edge instead of buying them a wrong one. On `aria2` both@@ -240,18 +248,21 @@ weak thing, and it is exactly what the oracle section above exists to check. Thetwo agree where both exist, to within about a point, but only the oracle has ananswer key we did not produce.-Cells were measured at different commits, and the staleness runs **conservative**:-every resolver change in between only removes wrong edges and gained zero, so 84.8%-is a floor. No cell was measured on the 0.44.0 release; the full table pins a-commit per cell.+**All nine cells are measured at one commit**, and the resolver has moved since.+Every corpus repository's call population was diffed site by site against the+current tip: three cells are byte-identical, six moved, and **of the 280 graded+rows twelve moved — every one of them a row graded `wrong`, with no row graded+`correct` moving at all.** Re-reading at the tip can therefore only raise the+number, so 85.7% is a floor. The table is not redrawn here because a cell whose+population moved needs a fresh proportional draw rather than a twelve-row patch.+No cell was measured on the 0.44.0 release.-**All 540 graded rows are published**, one file per cell, each row carrying the call+**All 600 graded rows are published**, one file per cell, each row carrying the callsite, the declaration the tool bound it to, the verdict and the reason it wasgiven. A script in the same directory rebuilds every table above from them and-fails if it disagrees. One cell of the eighteen, rust on our side, ships the 30-sites that were read without their per-row verdicts, because that cell was re-read-on a fresh draw and the verdicts of that read were never written down; the file-says so on every row.+fails if it disagrees. 560 of those rows enter the pooled figures: C++ is read at+50 rows per side and enters at a seeded 30 so that it carries the same weight as+every other language. Every row on the page now carries its own verdict.**Precision is one of two halves.** We lead no recall cell against the oracle, andlose cross-file coverage on 15 of 35 repositories in the same bench. A precision@@ -373,13 +384,11 @@ inferred.a field in which more than one tool works. Correcting for testing five tools atonce, three reductions are solid and two are marginal.-**We ran this on three agent harnesses**, because the answer depends on the harness-as much as on the tools. On Claude Code (15 questions, `claude-sonnet-5`) we-measured **-15.9%** and were the only arm the agent reliably called. On a local-`qwen3:8b` under Ollama, where inference is free and the win shows up as time-instead, the local-only tool set ran **-47.9% output tokens in -41.5% wall clock**.-Both tables are in the fold below with their controls, including the one whose-token column fails its own control and is therefore quoted for nobody.+**We ran this on three agent harnesses**, because the answer depends on the+harness as much as on the tools: Codex above, Claude Code, and a local `qwen3:8b`+under Ollama where inference is free and the win shows up as time instead. Both+other tables are in the fold with their controls, including the one whose token+column fails its own control and is therefore quoted for nobody.<details><summary><b>Method, limits, and what this does not show</b></summary>@@ -592,10 +601,9 @@ reaches **5,523 MB** on the same repository where we use 468 MB. That is thedifference between running in a normal CI container and not.**Speed is level, not a win, and this page used to claim otherwise.** CodeGraph is-fastest on 16 repositories to our 14. The split is size: we lead under 1,000 files,-2.04s to 2.37s, and trail above it, 10.63s to 8.86s. We win the middle and lose the-tail, and the reason is that we do more resolution work per file. The other half of-that trade is the precision table above.+fastest on 16 repositories to our 14: we lead under 1,000 files and trail above+it, because we do more resolution work per file. The other half of that trade is+the precision table above.### Building the whole index@@ -613,14 +621,11 @@ field on every repository measured. On `django/django`:That is **22x** CodeGraph like for like and **135x** with prose on. Both numbersship and the 22x is not the user-facing one.-**The 22x is not a graph number.** By the time that clock stops, the call graph has-been built in about 3 seconds and the rest of the run has built four more things:-a **graph** of 36,485 nodes, 90,477 edges and 31,384 symbols plus PageRank,-betweenness, Leiden communities and execution-flow tracing; **git history** mined-across 2,630 files for hotspots, ownership, co-change pairs and bus factor; **3,392-wiki pages** rendered and embedded for natural-language search; **architectural-decisions** mined from history and sessions; and **code health**, 5,317 findings-plus 155 unreachable files and 98 unused exports.+**The 22x is not a graph number.** The call graph itself is done in about 3+seconds. The rest of that clock builds four more layers on `django/django`: a+90,477-edge graph with communities and flow tracing, git history mined across+2,630 files, **3,392 wiki pages** embedded for natural-language search,+architectural decisions, and code health at **5,317 findings**.So "22x slower" and "the index contains categorically more" are both true andneither cancels the other. **If a call graph is all you want, CodeGraph builds one@@ -752,7 +757,7 @@ Raw CSV:## Limits that apply to the whole page<details>-<summary><b>Six limits that are not specific to any one section</b></summary>+<summary><b>Four limits that are not specific to any one section</b></summary>- **Every retrieval number here is Python or Go.** The graph sections are wider, atnine languages hand-graded and two compiler-graded, and they are the only@@ -772,10 +777,6 @@ Raw CSV:is in every model's training data, and it measures single-session questions ofroughly four to nine turns each. **Nothing on this page measures a longmulti-hour engineering task**, and we will not imply a number for one.-- **Precision is one of two halves.** We lead no recall cell against the oracle and- lose cross-file coverage on 15 of 35 repositories in the same bench.-- **The health signal is weak among files of similar size**, and it ties raw line- count on discrimination.- **Command-output compression has no head-to-head and one row that needsre-measuring.**