Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.
Python aicode-complexitycode-healthcode-intelligencecode-qualitydead-codedeveloper-toolsdocumentationgit-analyticsmcpopen-sourcepythonrefactoringstatic-analysistechnical-debt
docs: publish the graph precision result, judged by a compiler across five tools (#1738)
Raghav Chamadiya6cd8dddeb72f
diff --git a/README.md b/README.mdindex 9613b590e..6680f6d44 100644--- a/README.md+++ b/README.md@@ -144,7 +144,8 @@ from the CLI or right in the dashboard with the cost shown before you confirm.during doc generation needs a provider.)Full detail on every layer: **[docs/layers/INTELLIGENCE_LAYERS.md →](docs/layers/INTELLIGENCE_LAYERS.md)**-How the graph resolves an edge, and how much to trust one:+How the graph sees a call, how it resolves one, how much to trust the result, and+what a compiler says when it grades the whole thing:**[docs/layers/GRAPH.md →](docs/layers/GRAPH.md)**---@@ -651,6 +652,8 @@ agent over MCP.| **Output tokens vs a bare agent** *([measured](docs/BENCHMARKS.md#2-what-changes-in-a-real-agent-loop), n=43)* | ✅ **-31.6%** | -24.4% | -14.8% | not measured || **Index time, django** *([measured](docs/BENCHMARKS.md#6-indexing-time-the-row-we-lose))* | ⚠️ **366.8s**, slowest here | ✅ **16.4s** | not measured | n/a, cloud || | *one-time; updates after it are incremental* | | | |+| **Call-edge precision** *([measured](docs/BENCHMARKS.md#7-edge-precision), 540 rows hand-graded from source)* | ✅ **84.8%** | 57.0% | not measured | not measured |+| **Call-edge precision, judged by a compiler** *([measured](docs/BENCHMARKS.md#8-the-same-question-against-an-answer-key-we-do-not-control), 5 tools, 7 cells, 37,853 edges)* | ✅ **nothing that finds as much gets more of it right**, 7 of 7 | lower precision in 7, and lower recall in 5 | not measured | not measured || Generated documentation | ✅ | ❌ | ❌ | ✅ || Proactive agent hooks | ✅ Claude + Codex | ❌ | ❌ | ❌ || Auto-generated AI instructions (`CLAUDE.md`, `AGENTS.md`) | ✅ | ❌ | ❌ | ❌ |@@ -664,6 +667,25 @@ you need, that is the right trade. With prose generation on, which is what adefault `repowise init` actually costs, it is **135x**. Graphify andcode-review-graph were in the same measured field and are on the benchmarks page.+The precision row cuts the other way and is worth stating as plainly: of the call+edges we draw, **about fifteen percent are wrong**, and on `seastar` CodeGraph+grades better than we do. Nine languages were read on both sides, four separate,+five are statistical ties.++The compiler row exists because we graded the hand-read one ourselves. On Go and+TypeScript the answer key is the Go team's own RTA call graph and the `tsc`+checker's own resolution, which we neither wrote nor can tune.++**Read that row carefully, because it is a claim about two numbers.** Precision+alone is easy to win by drawing almost nothing, and two of the five tools score+above us that way, one of them at 0.997 from a graph holding 17% of the calls in+the repository. Recall alone is easy to win by drawing everything, and the tool+that leads it emits, on the largest repository measured, more than a third of its+edges as calls that do not exist. What we claim is the pair: **in all seven cells,+no tool that recovers as much of the call graph as we do gets more of it right.**+The column we lose is still there and is still ours to lose: **the tool with the+highest recall in every Go cell is not us.**+<sub>Measured against CodeGraph 1.5.0, Graphify 0.9.31, Serena 1.6.2.dev0,code-review-graph 2.3.7, on repowise `081a59fa` (between v0.37.0 and v0.38.0),August 2026. Unmarked rows are capability presence, not measurements.</sub>diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.mdindex d306e886d..8b1332c4a 100644--- a/docs/BENCHMARKS.md+++ b/docs/BENCHMARKS.md@@ -6,8 +6,10 @@ pre-registrations and every graded cell live in**[repowise-bench](https://github.com/repowise-dev/repowise-bench)**, which is alsowhere the depth is: this page is the summary, that repository is the evidence.-**Read this page in one minute:** the two charts below are the two results that-matter. Everything after them is the sample size, the caveat and the row we lose.+**Read this page in one minute:** the two charts below are the two retrieval+results, and [§8](#8-the-same-question-against-an-answer-key-we-do-not-control)+is the graph result, where a compiler rather than we decided who was right.+Everything else is the sample size, the caveat and the row we lose.| Layer | Measured against | Result ||---|---|---|@@ -17,6 +19,8 @@ matter. Everything after them is the sample size, the caveat and the row we lose| Command-output compression | RTK | [§4](#4-command-output-compression) **not measured head to head** || Code health and defect prediction | CodeScene | [§5](#5-code-health-predicts-defects) **we win** on recall and effort-aware ranking, p=0.003 || Indexing time | CodeGraph, Graphify, code-review-graph | [§6](#6-indexing-time-the-row-we-lose) **we lose**, 22x, because we build four more layers in the same pass |+| Are the call edges true | CodeGraph | [§7](#7-edge-precision) **we win**, 84.8% against 57.0%, 540 rows hand-graded from source |+| The same question, judged by a compiler | CodeGraph, codebase-memory-mcp, Graphify, code-review-graph | [§8](#8-the-same-question-against-an-answer-key-we-do-not-control) **no tool that finds as much of the graph gets more of it right**, 7 of 7 cells, and the recall column we lose || Documentation generation | DeepWiki, Google Code Wiki, Swimm | **not measured** || PR review | CodeRabbit, Greptile | **not measured** |@@ -436,11 +440,237 @@ finding against us.---+## 7. Edge precision++Every other row on this page counts things. This one asks whether they are **true**.++A resolver that guesses aggressively wins a coverage table and a raw edge count while+sending its reader to the wrong function. So we hand-graded call edges from source, on+both sides, by the same method: 30 rows per language per tool, seed 2026, stratified by+resolution strategy, every row read with its imports and enclosing scope open.++| | correct / n | 95% CI |+|---|---|---|+| **repowise** | **229/270 = 84.8%** | [80.0, 88.6] |+| **CodeGraph 1.5.0** | **154/270 = 57.0%** | [51.1, 62.8] |++Per language, nine languages, both sides read:++| Language | repowise | CodeGraph | |+|---|---|---|---|+| typescript | 29/30 | 7/30 | separates |+| go | 29/30 | 29/30 | tie |+| csharp | 28/30 | 20/30 | separates |+| python | 28/30 | 19/30 | separates |+| kotlin | 27/30 | 13/30 | separates |+| swift | 23/30 | 19/30 | tie |+| cpp | 23/30 | 16/30 | tie |+| rust | 22/30 | 13/30 | tie |+| java | 20/30 | 18/30 | tie |++**Read our number the other way round: roughly fifteen percent of our call edges are+wrong.** That is the number we plan against, and rust, java and cpp are where it+concentrates.++**Four of nine cells separate. Five are ties and we report them as ties.** At n=30 the+interval runs about ±16 points near 60%, so a point-estimate gap inside two overlapping+intervals is not a win. C++ is a tie despite looking like a 23-point lead.++**One repository goes clearly to them.** On `seastar` CodeGraph reads 6/10 against our+4/10, the only repository in the audit, on any language, where they beat us on a clear+margin. Our misses there are chained calls on an untyped receiver; they infer the+callee's declared return type and validate against it, so a failed inference costs them+an edge instead of buying them a wrong one. On `aria2` both sides read 10/10 and they+resolve 24,950 distinct call edges to our 9,486. Precision is not the only reading.++Cells were measured at different commits, and the staleness runs **conservative**: every+resolver change in between only removes wrong edges and gained zero, so 84.8% is a floor.+No cell was measured on the 0.44.0 release; the full table pins a commit per cell.++**All 540 graded rows are published**, one file per cell, each row carrying the call site,+the declaration the tool bound it to, the verdict and the reason it was given. A script in+the same directory rebuilds every table above from them and fails if it disagrees. One cell+of the eighteen, rust on our side, ships the 30 sites that were read without their per-row+verdicts, because that cell was re-read on a fresh draw and the verdicts of that read were+never written down; the file says so on every row.+**[graph/experiments/g1-edge-precision/rows](https://github.com/repowise-dev/repowise-bench/tree/master/graph/experiments/g1-edge-precision/rows)**.++---++## 8. The same question, against an answer key we do not control++Section 7 is hand-graded, by us, on both sides. That is the strongest form of a+weak thing: **every graph-quality number in this field, ours included, is scored+against something the publisher controls.** A tool that is confidently wrong in a+consistent way scores well and the reader has no way to check.++So we re-asked the question with the answer key taken out of our hands. On Go the+oracle is the **Go team's own RTA call graph** from+`golang.org/x/tools/go/callgraph/rta`, over the fully type-checked program. On+TypeScript it is the **`tsc` type checker's own resolution** of every call site.+We did not write either one, we cannot tune either one, and anyone with the+toolchain can regenerate both. Five tools, seven cells, five repositories,+**37,853 oracle edges**.++### Why precision alone is not the claim, and neither is recall++Two numbers, and you need both, because each one on its own has a cheap way to+win.++- **Recall** is the share of the real call graph a tool found. Draw an edge+ between everything and you score 1.000.+- **Precision** is the share of a tool's edges that are real. Draw one edge you+ are sure of and you score 1.000.++Both failure modes are in the table below, measured, not hypothetical. One tool+draws 12,533 edges on syft and more than a third of them are calls the compiler+says do not exist. Another scores the highest precision on this whole page,+0.997, from a graph containing **17% of the calls in the repository**: on+gitleaks it stored 4,367 call rows of which 76 resolve to anything, and a graph+that small is very hard to be wrong in and not much use to walk.++**So the only meaningful reading is the pair**, and it is the pair we claim:++> **In all seven cells, no tool that recovers as much of the call graph as we do+> gets more of it right.**++That sentence names no threshold, which is what makes it worth something. It+cannot be tuned by picking a cutoff, and adding a competitor can only break it.+Two competitors were added to this experiment after it was first written, and it+held in all seven cells.++**Precision: of the call edges a tool emits, the share the compiler confirms.**++| cell | repowise | CodeGraph 1.5.0 | codebase-memory-mcp 0.10.8 | Graphify 0.9.31 | code-review-graph 2.3.7 |+|---|---|---|---|---|---|+| cobra (with tests) | 0.972 | 0.929 | 0.912 | 0.971 | **0.997** |+| gitleaks (no tests) | 0.976 | 0.972 | 0.934 | **0.997** | 0.759 |+| gitleaks (with tests) | 0.974 | 0.971 | 0.922 | **0.995** | 0.800 |+| syft (no tests) | 0.943 | 0.872 | 0.635 | 0.771 | **0.968** |+| syft (with tests) | 0.950 | 0.864 | 0.673 | 0.802 | **0.966** |+| zod (no tests) | **0.992** | 0.729 | 0.987 | 0.825 | 0.932 |+| hono (no tests) | 0.977 | 0.805 | 0.949 | 0.980 | 0.966 |++**Recall: of the edges the compiler has, the share the tool found.**++| cell | repowise | CodeGraph | codebase-memory-mcp | Graphify | code-review-graph |+|---|---|---|---|---|---|+| cobra (with tests) | 0.684 | **0.763** | 0.743 | 0.433 | 0.174 |+| gitleaks (no tests) | 0.955 | 0.920 | **0.967** | 0.886 | 0.026 |+| gitleaks (with tests) | 0.914 | 0.895 | **0.945** | 0.832 | 0.032 |+| syft (no tests) | 0.513 | 0.508 | **0.542** | 0.447 | 0.201 |+| syft (with tests) | 0.322 | 0.338 | **0.361** | 0.273 | 0.086 |+| zod (no tests) | **0.703** | 0.373 | 0.694 | 0.248 | 0.652 |+| hono (no tests) | **0.731** | 0.684 | 0.686 | 0.688 | 0.691 |++Read a row across both tables and the trade is visible in every one. The tools+above us on precision are below us on recall, every time. The tool above us on+recall is below us on precision, every time. **Nobody is above us on both, in any+cell.** Intervals, per-cell artifacts and the full method are linked at the end+of this section; ties are ties and are marked as ties there.++**The weaker readings, so nobody has to infer them.** We are the most precise arm+outright in one cell of seven and tied for it in one more. We are beaten on+precision in five, by code-review-graph on cobra and both syft cells and by+Graphify on both gitleaks cells. Against the two tools this experiment started+with, CodeGraph and codebase-memory-mcp, we are the most precise in seven of+seven, and that narrower claim should always carry its label.++This also does something section 7 cannot: it removes the n=30 ceiling.+Hand-grading caps a precision estimate at roughly ±13 points near a 95% rate. An+oracle grades every edge, so n becomes the size of the repository.++**The two methods agree.** On Go, section 7 read 29/30 = 96.7% by hand for us and+29/30 = 96.7% for CodeGraph. The Go compiler, over roughly 1,600 edges on the same+repository, says 97.6% and 97.2%. Two unrelated methods, one person reading source+and one type checker, land within about a point on both arms. That is the result+we care about most and it is not a competitive one.++### The column we lose++We lead recall in the two TypeScript cells and in **none of the five Go cells**,+where codebase-memory-mcp leads four and CodeGraph the fifth. On cross-file+coverage over 35 repositories, codebase-memory-mcp separates from us on 15 and we+separate on none. Those are real losses and they are not softened here.++What the oracle adds is the price of that lead rather than an excuse for ours: on+syft, more than a third of what that tool emits is a call the Go compiler says+does not exist. Coverage rewards drawing edges and never asks whether they are+real, which is why no page in that benchmark publishes a coverage number without+a precision number beside it.++**Where our own miss goes**, decomposed rather than waved at. On syft without+tests we miss 3,846 of the oracle's 7,898 edges: 44% of that miss is dynamic+dispatch alone and a further 39% is dispatch with a closure at one end, and the+two buckets overlap so neither is the whole gap. Interface dispatch is the+ceiling and nobody in the comparison has cleared it, at 6.5 distinct possible+targets per call site. Matching that recall means emitting six edges where one is+right, which is the behaviour the precision table charges other tools for.++**Do not compare recall across rows.** It swings from 0.03 to 0.97, driven by how+many entry points the oracle had (4 on gitleaks, 268 on syft-with-tests), not by+tool quality. Only within-row comparisons carry meaning and a pooled recall over+these cells would be meaningless.++### Limits++- **Two languages, seven cells, five repositories.** This is not a nine-language+ claim and must not be quoted as one. Section 7 is the nine-language number.+- **A contradicted edge is very strong evidence, not proof.** RTA is unsound under+ reflection and `go:linkname`, so a genuinely dynamic edge can land in that+ bucket. It applies to all five arms equally and the gaps are far too large to+ be explained by it, which is why the metric is named *precision against the+ oracle* rather than precision.+- **Edges the oracle cannot speak about are charged to nobody** and reported at+ full size. That bucket is 0.4% to 11.1% of a tool's output on the Go cells and+ 16% to 46% on the TypeScript ones, where dependencies are not installed in the+ pinned corpus.+- **A library has no `main`, so RTA has no roots**; cobra is analysed through its+ test binaries only.+- The two variants of a repository answer different questions. Report both or say+ which one you used.+- **Two oracle languages, and the programme stops at two.** C#, Java, Kotlin and+ C++ each need a toolchain installed and a working build per repository, none of+ which exists on the measurement machine. Rust has the toolchain and no sound+ call-graph tool exists for it. Python, Ruby and PHP admit no oracle even in+ principle. So section 7 is the permanent method on those languages rather than+ a stopgap, and nothing here should be read as a claim about them.+- **The TypeScript cells exclude test files, and the with-tests variants are+ void.** A test file imports its own package by name, the pinned corpus has no+ dependencies installed, and roughly a third of call sites go unresolvable,+ which takes the unjudged bucket past three quarters. Both variants were run and+ neither is quoted anywhere. Installing the dependency trees would fix it and+ would also have to go to a scratch copy, since `node_modules/` inside the+ corpus changes what every tool walks.+- **The three competitors here are priced on these two languages only.**+ codebase-memory-mcp, Graphify and code-review-graph have no precision figure of+ any kind on the other nine languages in the coverage bench, and none of the+ three was ever entered into the section 7 audit in either direction. Section 7+ is a two-tool claim and section 8 a five-tool one, and neither transfers to the+ other's languages.+- **Two of the five arms are read through an adapter we wrote**, and both now beat+ us on precision in some cell, so the reading matters. Graphify tags 93% of its+ call edges `INFERRED` rather than AST-certain and we score all of them, which is+ the choice least favourable to us. code-review-graph stores unresolved callees+ beside resolved ones and we score only the resolved, which is the choice most+ favourable to it: on gitleaks that is 76 edges out of 4,367 stored rows. Both+ choices are argued in the benchmark and a reader who disagrees can recompute+ either from the artifacts.++Full method, per-cell artifacts, the twenty hand-confirmed identities the protocol+required, and the graded pre-registration including the two predictions that+missed:+**[graph/experiments/g4-oracle-anchored](https://github.com/repowise-dev/repowise-bench/tree/master/graph/experiments/g4-oracle-anchored)**.++---+## LimitsBeyond the ones stated in each section:-- **Every number here is Python or Go.** A JavaScript/TypeScript corpus+- **Every retrieval number here is Python or Go.** The graph sections are wider,+ at nine languages hand-graded and two compiler-graded, and they are the only+ sections that are. A JavaScript/TypeScript corpus(`mui/material-ui`, six tools, a 12x size range) is built and half graded, but**its sealed half is unrun and nothing from it is quoted here.** Publishing thedevelopment half is what that split exists to prevent, so the row arrives when the@@ -464,6 +694,11 @@ Beyond the ones stated in each section:- **§4 has no head-to-head and one row that needs re-measuring.**- **§5's signal is weak among files of similar size**, and a prior-defects baselinestill beats us on Popt by 0.085 even while losing on AUC.+- **§7 was graded by us on both sides.** That is what §8 exists to check, and the two+ agree where both exist, but only §8 has an answer key we did not produce.+- **§7 and §8 measure precision, which is one of two halves.** We lead no recall cell+ in §8 and lose cross-file coverage on 15 of 35 repositories in the same bench. A+ precision win is a claim about the edges a tool draws, never about how many.---@@ -520,6 +755,7 @@ one above it.| Level | What is there ||---|---|| **[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head)** | Who wins what, the build-cost curve, and what each index can rank at all |+| **[graph/](https://github.com/repowise-dev/repowise-bench/tree/master/graph)** | The graph-quality bench: edge precision hand-graded on both sides across nine languages against one competitor, precision and recall against a compiler oracle on Go and TypeScript against two, and cross-file coverage, adversarial invariance and build cost across five || **[arms/](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head/arms)** | One page per competitor: what it is, what it serves, and every setup trap. Four of six have a step that produces a clean zero when missed || **[THE\_LOOP.md](https://github.com/repowise-dev/repowise-bench/blob/master/head-to-head/THE_LOOP.md)** | The method and all nine gates, each named with the failure that created it || **[configs/arms.yaml](https://github.com/repowise-dev/repowise-bench/blob/master/configs/arms.yaml)** | Every launch command, allowlisted tool and exclusion with its reason. Read this if you think an arm was set up unfairly |@@ -532,7 +768,7 @@ a competitor is a YAML block, no Python and no runner change. See[CONTRIBUTING.md](https://github.com/repowise-dev/repowise-bench/blob/master/CONTRIBUTING.md).Tool versions as measured: CodeGraph 1.5.0, Graphify 0.9.31, Serena 1.6.2.dev0,-code-review-graph 2.3.7, cocoindex as of 2026-08-09.+code-review-graph 2.3.7, cocoindex as of 2026-08-09, codebase-memory-mcp 0.10.8.## See alsodiff --git a/docs/layers/GRAPH.md b/docs/layers/GRAPH.mdindex bd9a1a4f9..a33521692 100644--- a/docs/layers/GRAPH.md+++ b/docs/layers/GRAPH.md@@ -14,9 +14,12 @@ that **every edge carries its own evidence**.<img src="https://img.shields.io/badge/19-languages-F59520?style=flat-square&labelColor=0A0A0A" alt="19 languages" /><img src="https://img.shields.io/badge/22-framework_detectors-7F52FF?style=flat-square&labelColor=0A0A0A" alt="22 framework detectors" /><img src="https://img.shields.io/badge/0-LLM_calls-1E293B?style=flat-square&labelColor=0A0A0A" alt="zero LLM calls" />+ <img src="https://img.shields.io/badge/compiler_graded-7_of_7_cells_undominated-DC2626?style=flat-square&labelColor=0A0A0A" alt="no tool is both more precise and more complete, in 7 of 7 compiler-graded cells" /></p>**Contents:** [The problem with a plain arrow](#the-problem-with-a-plain-arrow) ·+[Two stages, and they fail differently](#two-stages-and-they-fail-differently) ·+[How good is it, and how we know](#how-good-is-it-and-how-we-know) ·[What is in the graph](#what-is-in-the-graph) ·[Every edge says how it got there](#every-edge-says-how-it-got-there) ·[Typing the receiver](#typing-the-receiver) ·@@ -56,6 +59,160 @@ which one happened, every time**.---+## Two stages, and they fail differently++An edge is two claims made by two different pieces of machinery, and the reason+to keep them apart is that they break in opposite directions.++**Stage one: capture.** Before anything can be resolved, the parser has to+notice that a call was written at all. That is a tree-sitter query per language,+listing the source shapes that count as a call site:++```scheme+; queries/go.scm -- Method call: obj.Method(args)+(call_expression+ function: (selector_expression+ operand: (identifier) @call.receiver+ field: (field_identifier) @call.target+ )+ arguments: (argument_list) @call.arguments+) @call.site+```++There is one of those files per language, and a shape that is not in it is a+call the graph will never contain. Go alone lists a plain call, a method call, a+package-qualified call, a chained call and a function passed as an argument, and+that last one is captured deliberately as a *reference* rather than a call,+because passing a handler is not invoking it.++A shape no query matches is invisible. Not low confidence, not unresolved:+**absent**. Nothing downstream can recover it, because nothing downstream knows+the call was there.++**Stage two: resolution.** Given a captured site, work out what the name points+at. `repo.save(draft)` hands you the name `save` and a receiver spelled `repo`,+and the job is to turn that into one declaration in one file. This is where the+29 origins below live, and it is the `user.save()` problem from the section+above.++| | fails when | costs you | how you find out |+|---|---|---|---|+| **capture** | nobody wrote a query for that call shape | **recall.** the edge does not exist | nothing internal can tell you, so it takes an outside answer key |+| **resolution** | the receiver cannot be typed, or the name is ambiguous | **precision** if it guesses, **recall** if it declines | the origin on the edge names the strategy that answered, so a wrong class is found and fixed once rather than per call site |++That asymmetry sets the whole design. A missed capture is silent, so it is+measured against a compiler rather than against ourselves. A bad resolution is+loud, so every edge is stamped with the strategy that produced it and nothing is+allowed to launder a guess into a fact.++**And at the bottom of the ladder repowise declines rather than guesses.** That+is a choice with a price, paid in the recall column below, and it is the right+way round: a missing arrow looks like missing information, a wrong arrow looks+like an answer.++---++## How good is it, and how we know++Two readings of the same question, and we graded only one of them.++### The one we did not grade++On Go the answer key is the **Go team's own RTA call graph** from+`golang.org/x/tools`, computed over the fully type-checked program. On TypeScript+it is the **`tsc` checker's own resolution** of every call site. We wrote+neither, we can tune neither, and anyone with the toolchain can regenerate both.++Seven cells, five repositories, **five tools**, 37,853 oracle edges. Of the call+edges we emit, the share the compiler confirms runs **0.943 to 0.992** per cell.++That number on its own is not the claim, because precision on its own has a cheap+way to win: draw one edge you are certain of and you score 1.000. Two of the five+tools do a version of exactly that. One scores 0.997 on cobra, the highest figure+in the whole experiment, from a graph holding **17% of the calls in the+repository**. Another takes both gitleaks cells from a graph that+finds 89% of the calls where we find 95%. Meanwhile the tool with the best recall emits, on the largest+repository measured, more than a third of its edges as calls the compiler says do+not exist.++**Recall alone is gameable in the other direction and precision alone in this+one, so the claim is the pair:**++> In all seven cells, **no tool that recovers as much of the call graph as we do+> gets more of it right.**++It names no threshold, so it cannot be tuned, and a new competitor can only break+it. Two were added after it was written and it held in all seven cells.++The weaker readings, so nobody has to infer them: most precise outright in one+cell, tied in one more, beaten in five by tools drawing much smaller graphs. And+against the two tools the experiment started with, most precise in seven of+seven, which is the narrower claim it should always be labelled as.++**Two languages, and only two.** C#, Java, Kotlin and C++ each need a toolchain+installed and a working build per repository, and nobody has done that here.+Python, Ruby and PHP can never have an oracle at all, because what a call+resolves to can change at runtime. That is a fact about those languages rather+than a gap in the harness, and it is why the hand-graded reading below is+permanent rather than a stopgap.++[The cells, the method and the graded pre-registration](../BENCHMARKS.md#8-the-same-question-against-an-answer-key-we-do-not-control)++### The one we did++Nine languages, 30 call edges per language per tool, every row opened in its own+file with its imports and enclosing scope, then the target declaration opened+too. **229 of 270 correct for us, 154 of 270 for CodeGraph 1.5.0**, intervals+disjoint. Four of the nine cells separate and five are ties, reported as ties.++Read our own number the other way round: **roughly fifteen percent of our call+edges are wrong**, concentrated in java, rust and cpp. That is the figure to plan+against, and it is a floor rather than a best case, because every resolver change+since the earliest rows were graded only removes wrong edges.++**The two readings agree.** On Go the hand grade says 96.7% for us and the+compiler says 97.6%, over roughly 1,600 edges rather than 30 rows. A person+reading source and a type checker landing within about a point of each other is+the strongest available evidence that the hand-graded half is accurate rather+than self-serving, and it is the result here we care about most.++[The nine cells, and all 540 graded rows with the reason each was given](../BENCHMARKS.md#7-edge-precision)++### The column we lose++Recall is the other half of the same question, and we do not lead it.++Across the five Go cells our recall runs 0.32 to 0.96 and **we lead none of+them**; codebase-memory-mcp leads four and CodeGraph the fifth. On cross-file+coverage over 35 repositories the same tool separates from us on 15 and we+separate on none. We do lead recall in both TypeScript cells, and we lead it over+the two tools that beat us on precision in every cell, which is the same trade+seen from the other side.++The oracle explains the trade rather than excusing it. **That tool recovers more+of the true call graph and emits far more that is not in it**: on the largest Go+repository measured, more than a third of what it emits is a call the compiler+says does not exist. Coverage rewards drawing edges and never asks whether they+are real, which is why no page in that benchmark prints a coverage number without+a precision number beside it.++Where our own miss actually goes, decomposed on one cell rather than waved at.+On syft without tests we miss **3,846 of the oracle's 7,898 edges**, and+**44% of that miss is dynamic dispatch alone, with a further 39% dispatch with a+closure at one end**. The two buckets overlap, so neither is the whole gap. Interface dispatch is the ceiling and+nobody in the comparison has cleared it: of 3,303 dispatch edges we match 12,+CodeGraph 35, codebase-memory-mcp 81, at 6.5 distinct possible targets per call+site. Matching that recall means emitting six edges where one is right, which is+the behaviour the precision table charges the other tool for.++The obvious cheap fix was priced and refused: giving Go `func` literals a symbol+would recover **50** static edges on that cell, not the 1,309 the raw closure+count suggests, because the rest need the dispatch ceiling cleared first.++[Both tables, the recall decomposition and what it would cost to close](https://github.com/repowise-dev/repowise-bench/tree/master/graph/experiments/g4-oracle-anchored)++---+## What is in the graph**Two tiers of node.** Files and packages on one tier; functions, classes,@@ -269,6 +426,23 @@ graph, which is why the graph is reproducible and why indexing needs no API key.- **Method-level dead-code detection is not shipped**, for any language. It wasmeasured, precision failed, and shipping it would have meant confidentlyrecommending deletions that were wrong.+- **Roughly fifteen percent of our call edges are wrong**, and **we lead no Go+ recall cell**. Both are measured, both are above under [how good is it, and how+ we know](#how-good-is-it-and-how-we-know), and neither is buried down here.+- **Two competing tools are more precise than us in five of seven oracle cells.**+ Each of them draws a much smaller graph, which is the whole reason, and it is+ stated above rather than left out. A precision figure quoted without the recall+ beside it is a misuse of this data, including by us.+- **The compiler-graded reading covers two languages.** Go and TypeScript are the+ only ones with an oracle, so on the other seventeen the precision figure is the+ hand-graded one, at 30 rows per language. That is a smaller n and a method we+ ran ourselves; the two agree to within a point where both exist, which is the+ reason to trust the half where only one does.+- **A competitor's coverage lead is only priced on two languages too.** The tool+ that beats us on cross-file coverage across 35 repositories has an+ oracle-anchored precision figure on go and typescript alone. That its extra+ edges are mostly wrong is measured there and inferred elsewhere, and the+ benchmark says so rather than generalising quietly.---@@ -280,3 +454,5 @@ graph, which is why the graph is reproducible and why indexing needs no API key.- [DEAD_CODE.md](DEAD_CODE.md) · how reachability becomes a confidence-tiered report- [CHANGE_RISK.md](CHANGE_RISK.md) · how the graph feeds a per-change risk score- [reference/COMPUTED_GLOSSARY.md](../reference/COMPUTED_GLOSSARY.md) · every derived metric, defined+- [BENCHMARKS.md §7](../BENCHMARKS.md#7-edge-precision) and [§8](../BENCHMARKS.md#8-the-same-question-against-an-answer-key-we-do-not-control) · the precision numbers on this page, with their sample sizes and intervals+- [repowise-bench/graph](https://github.com/repowise-dev/repowise-bench/tree/master/graph) · the harnesses, the five arms, the graded rows and the pre-registrations behind all of it