Skip to content

Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.

docs: add a roadmap, surface the enterprise docs, and restructure the benchmarks page (#1766)

Raghav Chamadiya689132bf7d82
diff --git a/README.md b/README.md
index 6680f6d44..39fe5d1b2 100644
--- a/README.md
+++ b/README.md
@@ -37,7 +37,8 @@
<a href="#the-ten-mcp-tools">MCP tools</a> ·
<a href="#measured-against-the-field">Benchmarks</a> ·
<a href="#how-it-compares-on-capability">Comparison</a> ·
- <a href="#for-teams--enterprises">Teams</a>
+ <a href="#for-teams-and-enterprises">Enterprise</a> ·
+ <a href="ROADMAP.md">Roadmap</a>
</sub></p>
---
@@ -64,6 +65,11 @@ calls</strong>. <strong>We publish the rows we lose</strong>, and we are the slo
<a href="docs/BENCHMARKS.md"><strong>All of it, including the losses →</strong></a><br />
Free and self-hosted, runs on your machine, and the first index needs no API key.</sub>
+<p align="center"><sub>
+Runs entirely on your infrastructure · <strong>zero LLM calls in every analysis layer</strong> ·
+AGPL-3.0 or commercial · <a href="#for-teams-and-enterprises"><strong>For enterprises →</strong></a>
+</sub></p>
+
<picture>
<source media="(prefers-color-scheme: dark)" srcset=".github/assets/one-index-dark.svg" />
<img src=".github/assets/one-index.svg" alt="One index producing code health, a dependency graph, git history, generated docs, architectural decisions, and ten MCP tools" width="100%" />
@@ -140,8 +146,9 @@ layers and renders every wiki page from your code's structure, with no API key a
no spend. Convert any part of it to LLM-written prose whenever you want, one page,
one directory, or a ranked coverage slice at a time, and pay only for what you pick,
from the CLI or right in the dashboard with the cost shown before you confirm.
-(Seven of the eight decision sources are deterministic too; only the one harvested
-during doc generation needs a provider.)
+(Six of the seven decision sources are deterministic too; only comment
+archaeology, which reads rationale prose on high-centrality code, needs a
+provider.)
Full detail on every layer: **[docs/layers/INTELLIGENCE_LAYERS.md →](docs/layers/INTELLIGENCE_LAYERS.md)**
How the graph sees a call, how it resolves one, how much to trust the result, and
@@ -189,8 +196,9 @@ Three deterministic signals, all computed from the graph and git history, no LLM
Doc, test and config commits are filtered out so the count means what it says, and a
file with a run of recent fixes gets flagged as a bug magnet while you edit it.
([reference →](docs/layers/BUG_HISTORY.md))
-- **Test intelligence.** Ingest coverage, find untested hotspots, and run only the
- tests a diff actually exercises with `repowise impacted-tests HEAD~1`.
+- **[Test intelligence](#which-tests-cover-this-file-without-a-coverage-report).** Which
+ tests reach a file and which ones a diff actually exercises, from the call graph,
+ with or without a coverage report.
([reference →](docs/layers/TEST_INTELLIGENCE.md))
Plus the free **[Repowise PR Bot](#the-pr-bot)**, which puts all of it on every pull
@@ -198,6 +206,34 @@ request. Zero LLM calls.
---
+## Which tests cover this file, without a coverage report
+
+Ingest LCOV, Cobertura or Clover and you get the measured answer. **Most
+repositories never produce one**, so the graph answers instead: a test file that
+imports a source file *reaches* it, which is a recorded edge rather than the
+name-shaped guess everything else falls back to.
+
+That fallback fails in both directions, and this repo is the proof. Five of its
+six worst bug-magnet files have no test named for them and read as untested while
+the graph names 3 to 23 test files each. The sixth is worse: matching on basename
+paired the *health* engine with the *distill* engine's tests and called it tested.
+
+```bash
+repowise impacted-tests main..HEAD # only the tests this diff actually exercises
+repowise health # untested hotspots, now graph-aware
+```
+
+<sub>Dogfooded against a real <code>coverage run --contexts=test</code>:
+<strong>95.7% precision</strong> on what reaches a file and <strong>97.5% on the
+run list</strong>, at a 100% hit rate, against 72.1% and 94.8% for the one-hop
+import walk this replaced. The two tiers are never averaged: rows are stamped
+<code>basis: "measured"</code> or <code>"inferred"</code>, measured wins outright
+where both can answer, and the inferred tier may never produce a percentage.
+Sound as a floor, unsound as a quantity, and labelled so.
+<a href="docs/layers/TEST_INTELLIGENCE.md"><strong>Test intelligence →</strong></a></sub>
+
+---
+
## The PR bot
Install the [GitHub App](https://github.com/apps/repowise-bot) and the index shows up
@@ -407,7 +443,11 @@ the orchestrators. Full matrix and the contributor recipe:
## Supported languages
-**19 languages parsed to AST · 13 at the Full tier · framework-aware across all of them.**
+**19 languages parsed to AST · 35 on a five-rung ladder · framework-aware across
+all of them.**
+
+"Do you support X" has five useful answers, not two, so languages land on a
+ladder and every rung says what it buys you.
<p>
<strong>Full tier &nbsp;</strong>
@@ -436,6 +476,23 @@ the orchestrators. Full matrix and the contributor recipe:
<img src="https://img.shields.io/badge/Luau-00A2FF?style=flat-square&logo=lua&logoColor=white" alt="Luau" />
</p>
+Below those two rungs the ladder keeps going, and a language on a lower rung is
+still doing real work rather than being ignored:
+
+| Rung | Languages | What you get |
+|---|---|---|
+| **Full** (13) | Python · TypeScript · JavaScript · Svelte · Vue · Java · Kotlin · Go · Rust · C++ · C# · Scala · Ruby | The whole pipeline: AST symbols, import resolution, a resolved call graph, heritage, docstrings, framework edges, **and code-health markers** |
+| **Good** (5) | C · Swift · PHP · Dart · Object Pascal | All of the above except the full health suite |
+| **Partial** (1) | Luau / Roblox | AST symbols and `require()` resolution, Rojo and `.luaurc` aware |
+| | | ⎯⎯ *tree-sitter parsing stops here; the rungs below come from git and imports* ⎯⎯ |
+| **Lightweight** (7) | Elixir · Clojure · Haskell · Lean 4 · Erlang · F# · HTML | A real file-to-file import graph, and no symbol-level claims |
+| **Structural** (9) | Objective-C · R · Zig · Julia · Elm · OCaml · Crystal · Nim · D | Git history: blame, hotspots, co-change, ownership, bug history |
+
+**Every language ships in the open-source distribution.** None is gated behind
+the commercial licence, and none will be. Languages on the way up the ladder,
+including **COBOL**, are on the
+**[roadmap →](ROADMAP.md#languages)**.
+
SQL and dbt projects get real `ref()` / `source()` lineage, shell scripts get
function-level symbols, HTML pages contribute their `<script src>` / `<link href>`
dependencies (including `index.html` → `src/main.ts`), and OpenAPI, Protobuf,
@@ -763,26 +820,57 @@ one index, self-hostable and open source. Full side-by-side comparisons:
| **Team leads** | Know which PRs to worry about before you merge: change-risk scoring plus the free [Repowise PR Bot](https://github.com/apps/repowise-bot). [For team leads →](https://www.repowise.dev/for/teams) |
| **Engineering leaders** | See how much of your code AI wrote and whether it is healthy: agent provenance, health trends and bus factor, straight from git history. [For engineering leaders →](https://www.repowise.dev/for/engineering-leaders) |
| **Security & compliance** | Reachability-aware CVE triage, secret detection across full git history, and SBOM, on your real dependency graph. [For security →](https://www.repowise.dev/for/security) · [security review →](docs/business/SECURITY_COMPLIANCE.md) |
-| **Enterprises** | On-prem and air-gapped, SSO/SCIM, commercial licensing with no AGPL obligation, IP indemnification. [For enterprise →](https://www.repowise.dev/for/enterprise) · [docs/business/COMMERCIAL.md](docs/business/COMMERCIAL.md) |
+| **Enterprises** | On-prem and air-gapped, SSO/SCIM, commercial licensing with no AGPL obligation, IP indemnification. [For enterprise →](https://www.repowise.dev/for/enterprise) · [what we're building next →](ROADMAP.md) · [docs/business/COMMERCIAL.md](docs/business/COMMERCIAL.md) |
---
-## For teams & enterprises
+<a id="for-teams-and-enterprises"></a>
+
+## For teams and enterprises
+
+The reason this passes a security review is structural rather than contractual:
+**every analysis layer computes with zero LLM calls.** The graph, git history,
+code health, change risk, dead code and the PR bot make no model calls at all, so
+in the deterministic path there is no provider to trust, no retention policy to
+read, and nothing to prompt-inject. Documentation prose is the one layer that
+uses a model, it is opt-in, and it runs on your key or fully offline through
+Ollama.
+
+| | |
+|---|---|
+| **Deployment** | Self-hosted from `pip install`, or containers on your own infrastructure: API server, indexer workers and dashboard, backed by Postgres and LanceDB or pgvector. Reference topology is Kubernetes, and the same containers run on Nomad or plain Docker. Air-gapped mode requires no outbound connectivity. *(Helm chart and air-gap bundle: [planned](ROADMAP.md#enterprise-operations))* |
+| **Data boundary** | Source never leaves your network. BYOK against your own Anthropic, OpenAI or Azure OpenAI contract, or fully offline. The provider choice is **per repository**, so sensitive repos run offline while others use a hosted model. Stored: the graph, non-reversible embeddings, wiki pages, git metadata. Raw source is processed transiently and never persisted. |
+| **Compliance and audit** | CycloneDX 1.6 SBOM with VEX export and per-snapshot diffs · PCI-DSS 4.0 and SOC 2 control-coverage reports with per-control evidence · insert-only audit trail with JSON/CSV export and a signed-webhook SIEM stream · CVE triage ranked by KEV, EPSS and whether your code actually imports the package. |
+| **Contract** | Commercial licence removes the AGPL obligation for embedding repowise in your own platform · IP indemnification · defensive patent grant · named support contact with a response-time SLA and a quarterly architecture review · per-seat, per-repo or enterprise-wide pricing. |
+
+**Past one repository.** Workspaces index an estate as one unit: API contracts
+matched producer to consumer so a breaking change is caught before it ships,
+cross-repo co-change, and one federated MCP endpoint that answers across all of
+it. *(Estate-scale dashboards: [in development](ROADMAP.md#multi-repo-and-workspace).)*
+
+**Not on git?** Only the history layer needs a commit log. Point `repowise init`
+at a plain directory, an export, or a Perforce or SVN workspace and the graph,
+documentation, decisions and code-health layers all build normally; what is
+missing is hotspots, ownership, co-change and bug history until the history layer
+learns to read your system.
+*([Perforce, SVN, Endevor and ChangeMan on the roadmap →](ROADMAP.md#source-control-beyond-git))*
+
+**Everything above is labelled GA, in development, or planned, line by line, in
+[COMMERCIAL.md](docs/business/COMMERCIAL.md).** Some of it is shipping today and
+some of it is not, and a comparison table that blurred the two would not survive
+the first procurement call. Sequencing against your timeline is part of the
+commercial conversation, so the items that matter most to you can be prioritized.
[**repowise.dev**](https://www.repowise.dev) is the same engine, fully managed, at
-feature parity with self-hosted: every CLI command, every MCP tool, the whole
-dashboard. We run it on our own codebase in the open:
+feature parity with self-hosted. We run it on our own codebase in the open:
[live snapshot →](https://www.repowise.dev/s/5a6b93fa9a69) ·
[explore public repos →](https://www.repowise.dev/explore).
-On top of self-hosting: managed deploys and webhooks with auto re-index on every
-commit, a hosted MCP endpoint so any client can point at one URL with no local server,
-a CVE-aware security layer, cross-repo intelligence at scale, and integrations (Slack,
-Jira/Linear, Confluence/Notion, PagerDuty) *(rolling out)*.
-
-What is GA versus in development, on-prem topology, SSO/SCIM/RBAC and pricing:
-**[docs/business/COMMERCIAL.md](docs/business/COMMERCIAL.md)** ·
-[Get in touch →](https://www.repowise.dev/#contact)
+**[Commercial detail and pricing models →](docs/business/COMMERCIAL.md)** ·
+**[Security review pack →](docs/business/SECURITY_COMPLIANCE.md)** ·
+**[Roadmap →](ROADMAP.md)** ·
---
diff --git a/ROADMAP.md b/ROADMAP.md
new file mode 100644
index 000000000..c1287e635
--- /dev/null
+++ b/ROADMAP.md
@@ -0,0 +1,195 @@
+# Roadmap
+
+What we are building next, and what we have decided not to build.
+
+**There are no dates on this page.** We shipped twelve minor releases in the
+thirty-one days to 2026-08-18, roughly one every two and a half days. Any
+quarter printed here would be wrong in the slow direction, and a roadmap that is
+wrong in the slow direction is worse than one carrying no dates at all. What we
+commit to is the status column. The [changelog](docs/CHANGELOG.md) is the
+evidence, and it is the fastest way to check whether this page is honest.
+
+Items disappear from this page when they ship. If you are looking for something
+that used to be here, it is in the changelog.
+
+| Status | What it means |
+|---|---|
+| **In development** | Being worked on now. Code exists, it is not finished. |
+| **Planned** | Committed to, not started. |
+| **Exploring** | We think it is a good idea and have not committed to it. Tell us if you need it, that is what moves a row up. |
+
+The same vocabulary describes the commercial surface in
+[docs/business/COMMERCIAL.md](docs/business/COMMERCIAL.md), where each capability
+also carries a **GA** marker when it is available today.
+
+---
+
+## Languages
+
+**Every language repowise supports ships in the open-source distribution under
+AGPL-3.0.** No language sits behind the commercial licence, and none will. If
+your stack is on this list, the support arrives in `pip install repowise`.
+
+Languages land on a five-rung ladder rather than a yes/no list, because "do you
+support X" has five different useful answers. Each rung is defined, with what it
+buys you, in
+[docs/layers/LANGUAGE_SUPPORT.md](docs/layers/LANGUAGE_SUPPORT.md). Today that
+ladder covers **35 languages, 19 of them parsed to a full AST**.
+
+### New languages
+
+| Language | Status | Why it is on the list |
+|---|---|---|
+| **COBOL** | **In development** | Asked for by enterprises with mainframe estates evaluating repowise. The codebases that most need an institutional-memory layer are often the ones whose authors have already retired. |
+| **VB.NET** | Planned | Large, long-lived line-of-business estates, usually sitting beside the C# we already treat at Full tier. |
+| **PL/SQL** | Planned | We already parse SQL through sqlglot. Packages, procedures and triggers are where the business logic actually lives. |
+| **ABAP** | Exploring | The same argument as COBOL, in SAP estates. |
+| **RPG / AS400** | Exploring | Named alongside COBOL often enough to track. |
+| **Fortran** | Exploring | Scientific and engineering codebases with long lifespans and thin documentation. |
+| **Apex** | Planned | Salesforce estates, where the org is often the least documented system a company runs. Apex is Java-shaped, and Lightning Web Components are already covered as JavaScript and HTML. |
+| **Ada** | Exploring | Defense and aerospace, long-lived and thinly documented, and usually in the same estates asking about COBOL. |
+| **Objective-C** | Planned | Currently Structural, meaning history only. The grammar is mature, so this is a climb up the ladder rather than new ground. |
+| **Elixir**, **F#** | Planned | Currently Lightweight, meaning file-to-file imports. Both grammars are already on PyPI, so these are AST upgrades rather than new ground. |
+| **Template dialects** (Django/Jinja, ERB, Blade, Thymeleaf, Go templates) | Planned | Today they parse cleanly as HTML and yield nothing, because `{% extends "base.html" %}` is plain text to an HTML parser. A stated ceiling, not an oversight. |
+
+### Deepening languages we already parse
+
+| Language | Status | Next |
+|---|---|---|
+| Vue, Svelte | In development | Options-API members, `{#each}` and `v-for` head bindings, `.svelte.ts` rune modules |
+| C#, Scala, Ruby | Planned | Dataflow dialects, which is what Extract Method needs in order to lift a span safely |
+| Kotlin | Exploring | Dataflow is blocked on the grammar and needs either a grammar upgrade or a text-based jump seam. We would rather say blocked than planned. |
+| C | Planned | It shares the C++ grammar for parsing but reaches no health walker map, so it gets graph coverage without markers |
+| SQL / dbt | Planned | Column-level blast radius |
+| Object Pascal | Planned | Assertion and performance markers, a dedicated `uses` resolver |
+
+### Need a language that is not here
+
+Adding one touches per-language subpackages rather than the parser core, which
+is what makes this tractable at the rate above. Open an issue and we will scope
+it in public. Commercial customers can have a language or framework built and
+maintained by us as a line item, which is
+[GA today](docs/business/COMMERCIAL.md#54-enterprise-operations), and the result
+still ships to everyone under AGPL.
+
+---
+
+## Source control beyond git
+
+Plenty of the code most worth indexing has never been near a git remote. It sits
+in Perforce, in Subversion, or on a mainframe under a change-management system
+older than most of the people maintaining it.
+
+**A repository with no git at all already indexes today.** Only one of the five
+layers is derived from history; the graph, documentation, decisions and
+code-health layers are all computed from the working tree. Point `repowise init`
+at a plain directory, an export, or a Perforce or SVN workspace and those four
+build normally. What is missing is the history-derived half: hotspots, ownership,
+co-change, bus factor, bug history, and change risk, all of which need a commit
+log to mine.
+
+Closing that gap means teaching the history layer to read something other than
+`git log`. The signals above it do not care where the history came from once it
+is normalized into changes, authors, timestamps and touched files, which is the
+shape every one of these systems has.
+
+| System | Status | Shape of the work |
+|---|---|---|
+| **Perforce Helix Core** | Planned | Changelists map to commits almost directly, and `p4 filelog` and `p4 annotate` cover history and blame. Common in games, hardware, embedded and anywhere the repo carries large binaries. |
+| **Subversion** | Planned | Revisions map cleanly; `svn log` and `svn blame` are close analogues. Still the system of record in a lot of long-lived estates. |
+| **TFVC (Azure DevOps)** | Exploring | Changesets, for shops on Azure DevOps that never moved to its git repos. |
+| **Mercurial** | Exploring | Changesets map cleanly, and the work is small. Ordered by demand rather than difficulty. |
+| **CA Endevor SCM** (Broadcom) | Exploring | Not a commit DAG. Endevor tracks elements moving through a stage and environment promotion hierarchy, so the practical shape is an export to a filesystem plus a history adapter that reads element history rather than a log. Pairs with the COBOL estates above. |
+| **ChangeMan ZMF** (OpenText) | Exploring | The same argument, with packages instead of elements. |
+
+**Ask, do not guess, if your system is not listed.** The seam is the same for all
+of them, and what decides the order is which ones customers actually run.
+
+---
+
+## Multi-repo and workspace
+
+Real systems are not one repository, and the interesting failures live in the
+gaps between them.
+
+| Item | Status | What it adds |
+|---|---|---|
+| Cross-repo intelligence at scale | In development | Hotspots, dead code and ownership across a whole estate with centralized dashboards, beyond the local-workspace scope already shipping in open source |
+| Column-level blast radius for SQL / dbt | Planned | Lineage is table-level today through `ref()` and `source()`. Column-level answers "who breaks if I drop this field" |
+| Custom decision policies | Planned | Required-reviewer rules, mandatory `get_why` checks on governed paths, and merge gating tied to findings |
+
+---
+
+## Integrations
+
+These sit on the hosted platform and the commercial licence. Current
+availability, including everything that is GA today, is in
+[COMMERCIAL.md §5.2](docs/business/COMMERCIAL.md#52-workflow-integrations-rolling-out).
+
+| Item | Status |
+|---|---|
+| GitHub Enterprise, Azure DevOps, GitLab self-managed, Bitbucket | In development |
+| SAML / OIDC SSO (Okta, Entra ID, Auth0, Google Workspace) and SCIM provisioning | In development |
+| Slack and Microsoft Teams beyond security alerting: hotspot drift, bus-factor warnings, decision staleness, routed by ownership | In development |
+| Jira risk-finding links and PR-impact auto-comments on linked issues | Planned |
+| Native SIEM connectors (Splunk, Datadog, Elastic) beyond the signed-webhook stream | Planned |
+| PagerDuty | Planned |
+| SPDX SBOM output and cross-format conversion | Planned |
+| ISO 27001 Annex A and GDPR / data-residency control mappings | Planned |
+
+On that last row: we would rather ship two solid compliance mappings than four
+shallow ones, which is why PCI-DSS 4.0 and SOC 2 are GA and these are not.
+
+---
+
+## Enterprise operations
+
+| Item | Status |
+|---|---|
+| Role-based access control at repo, module and decision level | Planned |
+| Air-gapped install bundle with bundled grammars, embedding model and optional Ollama runtime | Planned |
+| Helm chart for the reference topology | Planned |
+| Multi-tenant deployment | Planned |
+| Backup and restore, point-in-time snapshots of the intelligence layers | Planned |
+| Audit-trail coverage beyond the security surface, covering decisions and overrides | In development |
+| Engineering leader dashboard: bus-factor trends, ownership drift, decision-staleness curves, scheduled digests | In development |
+
+---
+
+## Core intelligence
+
+The half of the roadmap that has nothing to do with procurement.
+
+| Item | Status | Note |
+|---|---|---|
+| Interface-dispatch recall | Exploring | The measured ceiling on our call-graph recall. On `syft`, 44% of what we miss is dynamic dispatch alone and a further 39% is dispatch with a closure at one end. Nobody in the [measured field](docs/BENCHMARKS.md#8-the-same-question-against-an-answer-key-we-do-not-control) has cleared it, at 6.5 possible targets per call site, and matching that recall naively means emitting six edges where one is right. |
+| A sealed JavaScript / TypeScript retrieval corpus | In development | Built and half graded. The sealed half is unrun, and nothing from it is quoted anywhere until it is evaluated, once. |
+| More compiler oracles for graph precision | Exploring | Go and TypeScript are done. C#, Java, Kotlin and C++ each need a toolchain and a working build per repository; Rust has the toolchain and no sound call-graph tool exists for it; Python, Ruby and PHP admit no oracle even in principle. The honest status is "probably not", not "planned". |
+| Session intelligence harvesting | Planned | Architectural decisions surfaced from AI coding sessions and proposed to the team knowledge base, so knowledge generated during agent work does not evaporate when the session ends |
+
+---
+
+## What we are not building
+
+- **An LLM-based PR reviewer.** The [PR bot](README.md#the-pr-bot) makes zero
+ model calls per review, and that is the product rather than a limitation:
+ nothing to hallucinate, nothing to prompt-inject, and pushing the same diff
+ twice produces the same review twice. CodeRabbit and Greptile do the other
+ thing, and do it well. If you want intent-reading review, run one of those
+ alongside this.
+- **Runtime or APM data.** Everything here is derived from source and git
+ history. A performance finding we report is a static one, and we label it as
+ static.
+- **Any language behind a paywall.** Stated above, repeated here because it is
+ the question we get asked most.
+- **Dates.** See the top of this page.
+
+---
+
+## Ask for something
+
+- Open an [issue](https://github.com/repowise-dev/repowise/issues) and say what
+ you need and why. Demand is what moves an Exploring row to Planned, and we say
+ so rather than implying the order is purely technical.
+- Commercial and procurement timelines: [[email protected]](mailto:[email protected])
+- Security review: [[email protected]](mailto:[email protected])
diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
index 8b1332c4a..5d8056002 100644
--- a/docs/BENCHMARKS.md
+++ b/docs/BENCHMARKS.md
@@ -6,32 +6,30 @@ pre-registrations and every graded cell live in
**[repowise-bench](https://github.com/repowise-dev/repowise-bench)**, which is also
where the depth is: this page is the summary, that repository is the evidence.
-**Read this page in one minute:** the two charts below are the two retrieval
-results, and [§8](#8-the-same-question-against-an-answer-key-we-do-not-control)
-is the graph result, where a compiler rather than we decided who was right.
-Everything else is the sample size, the caveat and the row we lose.
+Each section leads with the result; method and caveats are one fold down, and
+nothing put there changes a headline. Where we have not run a comparison, the row
+says **not measured** rather than carrying a checkmark.
-| Layer | Measured against | Result |
-|---|---|---|
-| Finding the right files | CodeGraph, Graphify, code-review-graph, cocoindex | [§1](#1-finding-the-right-files) **we win**, n=42 held out, p=0.00004 |
-| Work saved in a real agent loop | the same field plus Serena and a bare agent | [§2](#2-what-changes-in-a-real-agent-loop) **we win** on all three agent harnesses we tried, n=43 at p&lt;0.0001 |
-| Loading one commit's context | naive file reads, `git diff` | [§3](#3-loading-one-commits-context-the-easy-number) **35.6x** fewer tokens than naive |
-| Command-output compression | RTK | [§4](#4-command-output-compression) **not measured head to head** |
-| Code health and defect prediction | CodeScene | [§5](#5-code-health-predicts-defects) **we win** on recall and effort-aware ranking, p=0.003 |
-| Indexing time | CodeGraph, Graphify, code-review-graph | [§6](#6-indexing-time-the-row-we-lose) **we lose**, 22x, because we build four more layers in the same pass |
-| Are the call edges true | CodeGraph | [§7](#7-edge-precision) **we win**, 84.8% against 57.0%, 540 rows hand-graded from source |
-| The same question, judged by a compiler | CodeGraph, codebase-memory-mcp, Graphify, code-review-graph | [§8](#8-the-same-question-against-an-answer-key-we-do-not-control) **no tool that finds as much of the graph gets more of it right**, 7 of 7 cells, and the recall column we lose |
-| Documentation generation | DeepWiki, Google Code Wiki, Swimm | **not measured** |
-| PR review | CodeRabbit, Greptile | **not measured** |
-
-The last two rows are capability comparisons, not measurements, and they live in
-the [README's feature table](../README.md#how-it-compares-on-capability) where a
-reader can tell the difference. **§4 carries the same label for the same reason:**
-RTK does exactly what `repowise distill` does, and we have not run the two against
-each other. We would rather write "not measured" than let a checkmark do a
-number's job.
+## The scoreboard
----
+| Question | Measured against | Result |
+|---|---|---|
+| **Are the call edges true, judged by a compiler** | CodeGraph, codebase-memory-mcp, Graphify, code-review-graph | [**no tool that finds as much of the graph gets more of it right**](#is-the-call-graph-correct), 7 of 7 cells, plus the recall column we lose |
+| **Are the call edges true, hand-graded from source** | CodeGraph | [**we win**](#7-edge-precision), 84.8% against 57.0%, 540 rows read on both sides |
+| **Finding the right files** | CodeGraph, Graphify, code-review-graph, cocoindex | [**we win**](#1-finding-the-right-files), n=42 held out, p=0.00004 |
+| **Work saved in a real agent loop** | the same field plus Serena and a bare agent | [**we win**](#2-what-changes-in-a-real-agent-loop) on all three agent harnesses we tried, n=43 at p&lt;0.0001 |
+| **Does the health score predict real bugs** | CodeScene | [**we win**](#5-code-health-predicts-defects) on recall and effort-aware ranking, p=0.003, and lose the business-impact axis outright |
+| **Loading one commit's context** | naive file reads, `git diff` | [**35.6x**](#loading-one-commits-context-the-easy-number) fewer tokens than naive |
+| **Indexing time** | CodeGraph, Graphify, code-review-graph | [**we lose**](#6-indexing-time-the-row-we-lose), 22x, because we build four more layers in the same pass |
+| **Command-output compression** | RTK | [**not measured**](#command-output-compression) head to head |
+| **Documentation generation** | DeepWiki, Google Code Wiki, Swimm | **not measured** |
+| **PR review** | CodeRabbit, Greptile | **not measured** |
+
+The three **not measured** rows are capability comparisons rather than
+measurements, and they live in the
+[README's feature table](../README.md#how-it-compares-on-capability) where a reader
+can tell the difference. We would rather write "not measured" than let a checkmark
+do a number's job.
<div align="center">
<picture>
@@ -46,14 +44,250 @@ number's job.
---
-## 1. Finding the right files
+<a id="8-the-same-question-against-an-answer-key-we-do-not-control"></a>
+<a id="is-the-call-graph-correct"></a>
+
+## Is the call graph correct
+
+Every other section on this page counts things. This one asks whether they are
+**true**, and it is the result we would keep if we could keep only one.
+
+**Every graph-quality number in this field, ours included, is normally scored
+against something the publisher controls**, so a tool that is confidently wrong in
+a consistent way scores well and the reader has no way to check. We re-asked the
+question with the answer key taken out of our hands: on Go the oracle is the **Go
+team's own RTA call graph** from `golang.org/x/tools/go/callgraph/rta` over the
+fully type-checked program, and on TypeScript it is the **`tsc` type checker's own
+resolution** of every call site. We did not write either one, we cannot tune either
+one, and anyone with the toolchain can regenerate both. Five tools, seven cells,
+five repositories, **37,853 oracle edges**.
+
+### Two numbers, and you need both
+
+Each one alone has a cheap way to win, and both failure modes are in the tables
+below rather than hypothetical.
+
+- **Recall** is the share of the real call graph a tool found. Draw an edge between
+ everything and you score 1.000. One tool draws 12,533 edges on `syft` and more
+ than a third of them are calls the compiler says do not exist.
+- **Precision** is the share of a tool's edges that are real. Draw one edge you are
+ sure of and you score 1.000. The highest precision on this whole page, 0.997,
+ comes from a graph containing **17% of the calls in the repository**; on
+ `gitleaks` that tool stored 4,367 call rows of which 76 resolve to anything, and
+ a graph that small is very hard to be wrong in and not much use to walk.
+
+**So the only meaningful reading is the pair, and it is the pair we claim:**
+
+> **In all seven cells, no tool that recovers as much of the call graph as we do
+> gets more of it right.**
+
+That sentence names no threshold, which is what makes it worth something. It cannot
+be tuned by picking a cutoff, and adding a competitor can only break it. Two
+competitors were added to this experiment after it was first written, and it held
+in all seven cells.
+
+**Precision: of the call edges a tool emits, the share the compiler confirms.**
+
+| cell | repowise | CodeGraph 1.5.0 | codebase-memory-mcp 0.10.8 | Graphify 0.9.31 | code-review-graph 2.3.7 |
+|---|---|---|---|---|---|
+| cobra (with tests) | 0.972 | 0.929 | 0.912 | 0.971 | **0.997** |
+| gitleaks (no tests) | 0.976 | 0.972 | 0.934 | **0.997** | 0.759 |
+| gitleaks (with tests) | 0.974 | 0.971 | 0.922 | **0.995** | 0.800 |
+| syft (no tests) | 0.943 | 0.872 | 0.635 | 0.771 | **0.968** |
+| syft (with tests) | 0.950 | 0.864 | 0.673 | 0.802 | **0.966** |
+| zod (no tests) | **0.992** | 0.729 | 0.987 | 0.825 | 0.932 |
+| hono (no tests) | 0.977 | 0.805 | 0.949 | 0.980 | 0.966 |
+
+**Recall: of the edges the compiler has, the share the tool found.**
+
+| cell | repowise | CodeGraph | codebase-memory-mcp | Graphify | code-review-graph |
+|---|---|---|---|---|---|
+| cobra (with tests) | 0.684 | **0.763** | 0.743 | 0.433 | 0.174 |
+| gitleaks (no tests) | 0.955 | 0.920 | **0.967** | 0.886 | 0.026 |
+| gitleaks (with tests) | 0.914 | 0.895 | **0.945** | 0.832 | 0.032 |
+| syft (no tests) | 0.513 | 0.508 | **0.542** | 0.447 | 0.201 |
+| syft (with tests) | 0.322 | 0.338 | **0.361** | 0.273 | 0.086 |
+| zod (no tests) | **0.703** | 0.373 | 0.694 | 0.248 | 0.652 |
+| hono (no tests) | **0.731** | 0.684 | 0.686 | 0.688 | 0.691 |
+
+Read a row across both tables and the trade is visible in every one. The tools
+above us on precision are below us on recall, every time. The tool above us on
+recall is below us on precision, every time. **Nobody is above us on both, in any
+cell.**
+
+**The weaker readings, so nobody has to infer them.** We are the most precise arm
+outright in **one cell of seven** and tied for it in one more. We are beaten on
+precision in **five**: by code-review-graph on cobra and both syft cells, and by
+Graphify on both gitleaks cells. Against the two tools this experiment started
+with, CodeGraph and codebase-memory-mcp, we are the most precise in seven of seven,
+and that narrower claim should always carry its label.
+
+**The column we lose.** We lead recall in the two TypeScript cells and in **none of
+the five Go cells**, where codebase-memory-mcp leads four and CodeGraph the fifth.
+On cross-file coverage over 35 repositories, codebase-memory-mcp separates from us
+on 15 and we separate on none. Those are real losses and they are not softened
+here. What the oracle adds is the price of that lead rather than an excuse for
+ours: on syft, more than a third of what that tool emits is a call the Go compiler
+says does not exist.
+
+<details>
+<summary><b>Method, limits, and what this does not show</b></summary>
+
+**The two methods agree.** On Go, the hand-graded audit below read 29/30 = 96.7%
+for us and 29/30 = 96.7% for CodeGraph. The Go compiler, over roughly 1,600 edges
+on the same repository, says **97.6% and 97.2%**. Two unrelated methods, one person
+reading source and one type checker, land within about a point on both arms. That
+is the result we care about most, and it is not a competitive one. It also removes
+the n=30 ceiling: hand-grading caps a precision estimate at roughly ±13 points near
+a 95% rate, while an oracle grades every edge, so n becomes the size of the
+repository.
+
+**Where our own miss goes**, decomposed rather than waved at. On syft without tests
+we miss 3,846 of the oracle's 7,898 edges: **44% of that miss is dynamic dispatch
+alone** and a further **39% is dispatch with a closure at one end**, and the two
+buckets overlap so neither is the whole gap. Interface dispatch is the ceiling and
+nobody in the comparison has cleared it, at 6.5 distinct possible targets per call
+site. Matching that recall means emitting six edges where one is right, which is
+the behaviour the precision table charges other tools for.
+
+**Do not compare recall across rows.** It swings from 0.03 to 0.97, driven by how
+many entry points the oracle had (4 on gitleaks, 268 on syft-with-tests), not by
+tool quality. Only within-row comparisons carry meaning, and a pooled recall over
+these cells would be meaningless.
+
+Further limits:
+
+- **Two languages, seven cells, five repositories.** This is not a nine-language
+ claim and must not be quoted as one. The hand-graded audit below is the
+ nine-language number.
+- **A contradicted edge is very strong evidence, not proof.** RTA is unsound under
+ reflection and `go:linkname`, so a genuinely dynamic edge can land in that
+ bucket. It applies to all five arms equally and the gaps are far too large to be
+ explained by it, which is why the metric is named *precision against the oracle*
+ rather than precision.
+- **Edges the oracle cannot speak about are charged to nobody** and reported at
+ full size. That bucket is 0.4% to 11.1% of a tool's output on the Go cells and
+ 16% to 46% on the TypeScript ones, where dependencies are not installed in the
+ pinned corpus.
+- **A library has no `main`, so RTA has no roots**; cobra is analysed through its
+ test binaries only.
+- The two variants of a repository answer different questions. Report both or say
+ which one you used.
+- **Two oracle languages, and the programme stops at two.** C#, Java, Kotlin and
+ C++ each need a toolchain installed and a working build per repository, none of
+ which exists on the measurement machine. Rust has the toolchain and no sound
+ call-graph tool exists for it. Python, Ruby and PHP admit no oracle even in
+ principle. So the hand-graded audit is the permanent method on those languages
+ rather than a stopgap, and nothing here should be read as a claim about them.
+- **The TypeScript cells exclude test files, and the with-tests variants are
+ void.** A test file imports its own package by name, the pinned corpus has no
+ dependencies installed, and roughly a third of call sites go unresolvable, which
+ takes the unjudged bucket past three quarters. Both variants were run and neither
+ is quoted anywhere.
+- **The three newer competitors are priced on these two languages only.**
+ codebase-memory-mcp, Graphify and code-review-graph have no precision figure of
+ any kind on the other nine languages, and none was ever entered into the
+ hand-graded audit in either direction. That audit is a two-tool claim and this one
+ a five-tool claim, and neither transfers to the other's languages.
+- **Two of the five arms are read through an adapter we wrote**, and both now beat
+ us on precision in some cell, so the reading matters. Graphify tags 93% of its
+ call edges `INFERRED` rather than AST-certain and we score all of them, which is
+ the choice least favourable to us. code-review-graph stores unresolved callees
+ beside resolved ones and we score only the resolved, which is the choice most
+ favourable to it: on gitleaks that is 76 edges out of 4,367 stored rows. Both
+ choices are argued in the benchmark and a reader who disagrees can recompute
+ either from the artifacts.
+
+Full method, per-cell artifacts, the twenty hand-confirmed identities the protocol
+required, and the graded pre-registration including the two predictions that
+missed:
+**[graph/experiments/g4-oracle-anchored](https://github.com/repowise-dev/repowise-bench/tree/master/graph/experiments/g4-oracle-anchored)**.
+
+</details>
+
+---
+
+<a id="7-edge-precision"></a>
+
+## The same question, hand-graded across nine languages
+
+The oracle covers two languages. This one covers nine, on both sides, by the same
+method: 30 rows per language per tool, seed 2026, stratified by resolution
+strategy, every row read from source with its imports and enclosing scope open.
+
+| | correct / n | 95% CI |
+|---|---|---|
+| **repowise** | **229/270 = 84.8%** | [80.0, 88.6] |
+| **CodeGraph 1.5.0** | **154/270 = 57.0%** | [51.1, 62.8] |
+
+| Language | repowise | CodeGraph | |
+|---|---|---|---|
+| typescript | 29/30 | 7/30 | separates |
+| go | 29/30 | 29/30 | tie |
+| csharp | 28/30 | 20/30 | separates |
+| python | 28/30 | 19/30 | separates |
+| kotlin | 27/30 | 13/30 | separates |
+| swift | 23/30 | 19/30 | tie |
+| cpp | 23/30 | 16/30 | tie |
+| rust | 22/30 | 13/30 | tie |
+| java | 20/30 | 18/30 | tie |
+
+**Read our number the other way round: roughly fifteen percent of our call edges
+are wrong.** That is the number we plan against, and rust, java and cpp are where
+it concentrates.
+
+**Four of nine cells separate. Five are ties and we report them as ties.** At n=30
+the interval runs about ±16 points near 60%, so a point-estimate gap inside two
+overlapping intervals is not a win. C++ is a tie despite looking like a 23-point
+lead.
+
+<details>
+<summary><b>Method, limits, and what this does not show</b></summary>
+
+**One repository goes clearly to them.** On `seastar` CodeGraph reads 6/10 against
+our 4/10, the only repository in the audit, on any language, where they beat us on
+a clear margin. Our misses there are chained calls on an untyped receiver; they
+infer the callee's declared return type and validate against it, so a failed
+inference costs them an edge instead of buying them a wrong one. On `aria2` both
+sides read 10/10 and they resolve 24,950 distinct call edges to our 9,486.
+Precision is not the only reading.
+
+**This audit was graded by us, on both sides.** That is the strongest form of a
+weak thing, and it is exactly what the oracle section above exists to check. The
+two agree where both exist, to within about a point, but only the oracle has an
+answer key we did not produce.
+
+Cells were measured at different commits, and the staleness runs **conservative**:
+every resolver change in between only removes wrong edges and gained zero, so 84.8%
+is a floor. No cell was measured on the 0.44.0 release; the full table pins a
+commit per cell.
+
+**All 540 graded rows are published**, one file per cell, each row carrying the call
+site, the declaration the tool bound it to, the verdict and the reason it was
+given. A script in the same directory rebuilds every table above from them and
+fails if it disagrees. One cell of the eighteen, rust on our side, ships the 30
+sites that were read without their per-row verdicts, because that cell was re-read
+on a fresh draw and the verdicts of that read were never written down; the file
+says so on every row.
+
+**Precision is one of two halves.** We lead no recall cell against the oracle, and
+lose cross-file coverage on 15 of 35 repositories in the same bench. A precision
+win is a claim about the edges a tool draws, never about how many.
+
+**[graph/experiments/g1-edge-precision/rows](https://github.com/repowise-dev/repowise-bench/tree/master/graph/experiments/g1-edge-precision/rows)**
+
+</details>
+
+---
+
+<a id="1-finding-the-right-files"></a>
-Before a tool can save an agent any work, it has to point at the right code. This
-section measures only that.
+## Finding the right files
-**Grading is deterministic. No LLM judge is involved anywhere in this number**,
-which makes it the most reproducible result on the page. ContextBench ships gold
-file spans; a tool either returns them or it does not.
+Before a tool can save an agent any work, it has to point at the right code.
+**Grading is deterministic and no LLM judge is involved anywhere in this number**,
+which makes it the most reproducible result on the page: ContextBench ships gold
+file spans, and a tool either returns them or it does not.
**Every number below comes from instances this work has never seen.** The 112
instances were split 70 / 42 by instance id, **pinned before any of it started**,
@@ -75,29 +309,20 @@ p = 0.021**.
**These are two different tools with two different profiles, and we would rather
say so than average them into one claim.** `get_answer` finds the most, from a list
of ~19 files. `search_codebase` finds fewer but is the most efficient per file
-served: **0.742 from 8.2 files**, better coverage-per-file than anything else in
+served, **0.742 from 8.2 files**, better coverage-per-file than anything else in
the table. If you are paying by the token, that is the row to read.
**Precision is not our column.** code-review-graph's 0.240 is nearly three times
ours. Part of that is mechanical, since precision rises for whoever returns fewest
files, which is exactly why files-served is a column here and not a footnote.
-**The cocoindex row is not from the same sitting**, and it is printed rather than
-smoothed away: measured 2026-08-09 against the other five from 2026-08-02 to
-2026-08-06, same instances, same gold spans, same deterministic grading. Its n is
-41 because one named instance never answered even when queried alone, so it is
-excluded rather than counted as a zero, which makes its row *better* than counting
-it would (0.361 against 0.353). Its placing was
-[pre-registered before its index existed](https://github.com/repowise-dev/repowise-bench/tree/master/configs),
-including the prediction that it would come last.
-
-### Why this is not benchmark tuning
-
-We first ran this and came **last, at 0.228**. We published that. The cause was a
-query-time gate discarding most candidates before ranking ever happened. Fixing
-that path is what moved the number, and it is a fix any user gets, not a change
-shaped around these questions.
+<details>
+<summary><b>Method, limits, and what this does not show</b></summary>
+**Why this is not benchmark tuning.** We first ran this and came **last, at
+0.228**. We published that. The cause was a query-time gate discarding most
+candidates before ranking ever happened. Fixing that path is what moved the number,
+and it is a fix any user gets rather than a change shaped around these questions.
The check on that claim is the split, and it points the right way:
| | other half (n=70) | **sealed half (n=42)** |
@@ -106,44 +331,46 @@ The check on that claim is the split, and it points the right way:
| repowise (`search_codebase`) | 0.684 | 0.742 |
| CodeGraph | **0.6093** | **0.6095** |
-**Overfitting makes the unseen half score worse. Ours scores better**, on both
-tools. CodeGraph, which nobody tuned against either half, scores the same on both
-to three decimal places, so the two halves are equally hard and the gap is about
-the tool rather than the questions.
+Overfitting makes the unseen half score worse. Ours scores better, on both tools.
+CodeGraph, which nobody tuned against either half, scores the same on both to three
+decimal places, so the two halves are equally hard and the gap is about the tool
+rather than the questions. **We do not quote a pooled 112-instance figure**, though
+it is easy to compute and would be 0.835. Averaging the halves loses the only
+number that matters.
-**We do not quote a pooled 112-instance figure**, though it is easy to compute and
-would be 0.835. Averaging the halves loses the only number that matters.
+**The cocoindex row is not from the same sitting**, and it is printed rather than
+smoothed away: measured 2026-08-09 against the other five from 2026-08-02 to
+2026-08-06, same instances, same gold spans, same deterministic grading. Its n is
+41 because one named instance never answered even when queried alone, so it is
+excluded rather than counted as a zero, which makes its row *better* than counting
+it would (0.361 against 0.353). Its placing was
+[pre-registered before its index existed](https://github.com/repowise-dev/repowise-bench/tree/master/configs),
+including the prediction that it would come last.
-This is retrieval, not task success. It says we find the right files, not that an
-agent using us writes better code. That is what §2 is for. Producing it cost
-**748 index builds and roughly 78 machine-hours**, because every arm builds its own
-index of every instance at that instance's own `base_commit`, with nothing shared
-and nothing cached.
+**This is retrieval, not task success.** It says we find the right files, not that
+an agent using us writes better code.
Measured on repowise `081a59fa` (between v0.37.0 and v0.38.0). Raw cells:
**[rung8](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08/rung8)**.
----
-
-## 2. What changes in a real agent loop
+</details>
-The section a skeptic should read first, and the one modelled on the JetBrains
-reruns described [at the bottom of this page](#how-to-read-a-number-on-this-page).
-
-Every question in `django/django`'s question set, 48 of them, across five question
-shapes. Six arms: repowise, four competing tools, and a bare agent with no tools.
-Byte-identical prompt, full advertised tool surface per arm, a freshly built index
-on the same pinned commit, and a bare control verified free of local hooks.
+---
-Two things have to be true for a tool to be worth mounting: the agent has to
-actually call it, and the loop has to get leaner when it does.
+<a id="2-what-changes-in-a-real-agent-loop"></a>
-**We ran this on three agent harnesses, because the answer depends on the harness
-as much as on the tools.**
+## What changes in a real agent loop
-### The main run: 48 questions on Codex (`gpt-5.6-sol`)
+The section a skeptic should read first. Every question in `django/django`'s
+question set, 48 of them, across five question shapes. Six arms: repowise, four
+competing tools, and a bare agent with no tools. Byte-identical prompt, full
+advertised tool surface per arm, a freshly built index on the same pinned commit,
+and a bare control verified free of local hooks. Two things have to be true for a
+tool to be worth mounting: the agent has to actually call it, and the loop has to
+get leaner when it does.
-Every tool called on every question, so this is like-for-like.
+**48 questions on Codex (`gpt-5.6-sol`).** Every tool called on every question, so
+this is like for like.
| Tool | Agent used it | Output tokens | vs bare agent | Tool calls | Leaner on | p |
|---|---:|---:|---:|---:|---:|---:|
@@ -159,11 +386,20 @@ against the bare agent's 7.2**, opening 3.0 files instead of 7.2. One answered
question replacing roughly six greps, visible in the call counts rather than
inferred.
-Correcting for testing five tools at once, three reductions are solid and two are
-marginal. **CodeGraph is a genuine second at -24.4%**, and the honest reading is
-that we lead a field in which more than one tool works. Serena is the counter-case:
-it writes less than the bare agent while calling tools **42% more often**. Busier,
-not leaner.
+**CodeGraph is a genuine second at -24.4%**, and the honest reading is that we lead
+a field in which more than one tool works. Correcting for testing five tools at
+once, three reductions are solid and two are marginal.
+
+**We ran this on three agent harnesses**, because the answer depends on the harness
+as much as on the tools. On Claude Code (15 questions, `claude-sonnet-5`) we
+measured **-15.9%** and were the only arm the agent reliably called. On a local
+`qwen3:8b` under Ollama, where inference is free and the win shows up as time
+instead, the local-only tool set ran **-47.9% output tokens in -41.5% wall clock**.
+Both tables are in the fold below with their controls, including the one whose
+token column fails its own control and is therefore quoted for nobody.
+
+<details>
+<summary><b>Method, limits, and what this does not show</b></summary>
5 of the 48 questions are missing from every arm equally because the run hit an API
usage cap. Paired comparisons are unaffected; the figures are over the 43 all six
@@ -175,7 +411,7 @@ Pre-computed structure replaces exploration, and harder questions contain more
exploration to replace. That split is post-hoc and the pre-registered comparisons
are the stronger evidence.
-### Second harness: 15 questions on Claude Code (`claude-sonnet-5`)
+**Second harness: 15 questions on Claude Code (`claude-sonnet-5`).**
| Tool | Tools advertised | Schema cost (chars) | Agent used it | Output tokens | vs bare | Leaner on | p |
|---|---:|---:|---:|---:|---:|---:|---:|
@@ -186,31 +422,21 @@ are the stronger evidence.
| Graphify | 10 | 5,482 | 3 / 15 | 2,878 | 0.0% | 7 of 15 | 1.000 |
| *bare agent (control)* | 0 | 0 | n/a | 2,877 | baseline | n/a | n/a |
-The "agent used it" column is the interesting one. Under Claude Code most of these
-tools were barely called at all: code-review-graph never once, Graphify three times
-in fifteen. Nothing differed about the servers, questions or indexes between the
-two harnesses. Claude Code loads MCP tool schemas on demand, so the agent has to go
-looking before it can call anything, and frequently never does.
-
-**Treat that column as unstable, including our own 15 of 15.** Reruns on later days
-returned 4 of 15 and then 3 of 15 for us, and 2 of 14 for CodeGraph.
-
-**Is the collapse the model or the harness? Inconclusive.** The same 15 questions on
-Opus, against bands fixed before the run (12+ means the model, 6 or fewer means
-schema deferral, 7 to 11 inconclusive) **came back at 7**. Opus goes looking on 11
-of 15 and declines about a third of the times it looks, so deferral is part of the
-story and not all of it. **That run's token column fails its own control** (-9.3%
-when the tool was called against -10.7% when it was not), so no token figure is
-quoted from it for any tool including ours.
-
-### Third harness: a small model on your own hardware
-
-`qwen3:8b` under Ollama, driven by opencode, same 15 django questions, same seed.
-Inference cost is zero, so the question becomes "is it faster, and is it better".
-
-**Every cell called its tool: 15 of 15 on both rows**, against 7 of 15 for Opus on
-this identical draw. Adoption ordering across four instruments is Sonnet 3 to 4,
-Opus 7, Codex 15, local `qwen3:8b` 15.
+The "agent used it" column is the interesting one: under Claude Code most of these
+tools were barely called at all, code-review-graph never once and Graphify three
+times in fifteen. Nothing differed about the servers, questions or indexes between
+the two harnesses. Claude Code loads MCP tool schemas on demand, so the agent has
+to go looking before it can call anything, and frequently never does. **Treat that
+column as unstable, including our own 15 of 15.** Reruns on later days returned 4
+of 15 and then 3 of 15 for us, and 2 of 14 for CodeGraph. Whether the collapse is
+the model or the harness is **inconclusive**: the same 15 questions on Opus,
+against bands fixed before the run (12+ means the model, 6 or fewer means schema
+deferral, 7 to 11 inconclusive), came back at 7. That run's token column fails its
+own control (-9.3% when the tool was called against -10.7% when it was not), so no
+token figure is quoted from it for any tool including ours.
+
+**Third harness: `qwen3:8b` under Ollama via opencode**, same 15 questions, same
+seed. Every cell called its tool, 15 of 15 on both rows.
| Row | Agent used it | Output tokens | vs bare | Leaner on | p | Wall clock | vs bare |
|---|---:|---:|---:|---:|---:|---:|---:|
@@ -218,151 +444,76 @@ Opus 7, Codex 15, local `qwen3:8b` 15.
| **repowise, local-only tools** | **15 / 15** | **1,172** | **-47.9%** | **15 of 15** | **0.00006** | **96s** | **-41.5%** |
| *bare agent (control)* | 0 / 15 | 2,336 | baseline | n/a | n/a | 171s | baseline |
-**Two repowise rows, never combined.** `get_answer` writes its answer using a hosted
-model, so a row using it is not a local-only result. The second row switches it off
-and leaves only tools that run against the local index. We verified the restriction
-holds rather than assuming it: instructed directly and repeatedly to call
-`get_answer`, that agent could not reach it in any of its 15 cells.
-
-**On a local model the win shows up as time, and the mechanism differs from the
-hosted case.** repowise roughly doubles the tokens fed in on a single step while
-cutting steps from 3.3 to 2.1 and halving what the model writes. Reading a large
-payload once is cheap on a GPU; generating text token-by-token across several
-rounds is not.
-
-**We are not claiming a quality improvement from this run.** The full surface scored
-+1.32 against the bare agent on a 0 to 10 judge scale, above the judge's measured
-0.69 noise, but the win-loss count is 10 to 5 at p = 0.30 and removing the single
-best question drops it to +0.99. The local-only row is **+0.20, a null**. The
-defensible statement is narrower: **the local-only configuration answers about as
-well as a bare agent in roughly half the wall clock, on hardware you already own.**
-
-### What this section will not claim
-
-- **This is work saved, not quality.** A blind judge scored every tool in the field,
- ours included, a fraction **below** the bare agent on the 48-question run, in a
- range of 0.04 to 0.25 points on a 10-point scale. All are smaller than the 0.69
- points by which this benchmark moves when rerun unchanged. No tool here measurably
- changed answer quality in either direction. Ours is at the low end of that band
- and we will say so if it becomes a real effect. "No significant difference" is not
- parity, and an equivalence claim needs a test we have not run.
-- **The quality columns cannot be compared across tables.** Different judges, because
- grading a model with a judge from its own family is a known bias.
+**Two repowise rows, never combined.** `get_answer` writes its answer using a
+hosted model, so a row using it is not a local-only result. The second row switches
+it off and leaves only tools that run against the local index. We verified the
+restriction holds rather than assuming it: instructed directly and repeatedly to
+call `get_answer`, that agent could not reach it in any of its 15 cells. repowise
+roughly doubles the tokens fed in on a single step while cutting steps from 3.3 to
+2.1 and halving what the model writes, which is why the win shows up as time:
+reading a large payload once is cheap on a GPU, generating text token-by-token
+across several rounds is not.
+
+**What this section will not claim:**
+
+- **This is work saved, not quality.** A blind judge scored every tool in the
+ field, ours included, a fraction **below** the bare agent on the 48-question run,
+ in a range of 0.04 to 0.25 points on a 10-point scale, all smaller than the 0.69
+ points by which this benchmark moves when rerun unchanged. No tool here
+ measurably changed answer quality in either direction. Ours is at the low end of
+ that band and we will say so if it becomes a real effect. "No significant
+ difference" is not parity, and an equivalence claim needs a test we have not run.
+ On the local run the full surface scored +1.32 against the bare agent, above the
+ judge's measured 0.69 noise, but the win-loss count is 10 to 5 at p = 0.30 and
+ removing the single best question drops it to +0.99; the local-only row is
+ **+0.20, a null**. The defensible statement is narrower: the local-only
+ configuration answers about as well as a bare agent in roughly half the wall
+ clock, on hardware you already own.
+- **The quality columns cannot be compared across tables**, because grading a model
+ with a judge from its own family is a known bias, so the judges differ.
- **Not a universal saving.** One repository, one commit, one prompt. Three
harnesses, which is more than most published numbers in this category, and still
not a constant.
-- **Adoption is not a stable property of a tool.** Whether an agent calls a codebase
- server at all depends more on the harness than on the server. Any adoption figure,
- ours included, is only meaningful with its harness and its date attached.
+- **Adoption is not a stable property of a tool.** Whether an agent calls a
+ codebase server at all depends more on the harness than on the server. Any
+ adoption figure, ours included, is only meaningful with its harness and its date
+ attached. Ordering across four instruments was Sonnet 3 to 4, Opus 7, Codex 15,
+ local `qwen3:8b` 15.
- **No dollar figure**, and the control that retired it is the most useful thing we
ran. code-review-graph, having never called its server once in 15 questions and
carrying 28,118 extra characters of schema that should cost *more*, measured
**43% cheaper than the bare agent**. The cause is prompt caching: whichever arm
runs first warms it. An arm's position in the cycle correlated **-0.487** with its
dollar cost. Output tokens are never cached and correlate **+0.010**, so those are
- what we publish. If you see a token claim here that does not say whether it
+ what we publish. If you see a token claim anywhere that does not say whether it
controls for cache state and arm ordering, ask about that first.
-**What producing these tables cost: 471 agent runs, about 13 hours of machine time,
-roughly $44 of API spend**, plus 11.3 minutes of fresh index builds for every tool.
-Roughly a third of those runs are gates and repeats rather than headline numbers.
-Full breakdown, and the per-tool setup traps that were most of the real effort, in
-[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head).
-
Raw data, every cell including failures:
**[rung9](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08/rung9)**
(48-question Codex) and
**[rung6](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08/rung6)**
-(the 15-question runs).
-
----
-
-## 3. Loading one commit's context, the easy number
-
-This is the measurement almost every tool in this category publishes, and we are
-labelling it as such. Over the 30 most recent non-merge commits of `pallets/flask`,
-counted with deterministic `tiktoken` (`cl100k_base`):
-
-| Strategy | Tokens per commit |
-|---|---:|
-| naive, full contents of every changed file | 13,984 |
-| `git diff` only | 1,408 |
-| **`get_context`** | **393** |
-
-**35.6x fewer than naive, pooled**, 29.3x as a mean of per-commit ratios, and 3.6x
-pooled against `git diff`.
-
-**Lead with the pooled figure.** Pooled is sum over sum, so it weights each commit by
-the tokens actually at stake. A mean of per-commit ratios does not: a one-line commit
-where `get_context` returns 40 tokens contributes a huge ratio that counts equally
-against one saving a hundred thousand. That is how a 35x becomes a 209x in a press
-release.
-
-**What this does not tell you.** It is one payload, not a session. It says our
-representation of a commit is smaller than the commit, not that an agent finishes
-faster. The honest version of that question is §2, where the answer on the same
-harness is a much more modest 15.9%.
+(the 15-question runs). What producing them cost in runs, machine hours and API
+spend, plus the per-tool setup traps that were most of the real effort:
+[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head).
-Raw CSV:
-**[token\_efficiency\_flask30](https://github.com/repowise-dev/repowise-bench/blob/master/results/bakeoff_2026_08/rung1/token_efficiency_flask30_2026-08-01.csv)**
+</details>
---
-## 4. Command-output compression
-
-`repowise distill <cmd>` compresses command output *before* the agent reads it:
-errors first, exit code preserved, every omission recoverable through an inline
-`[repowise#<ref>]` marker.
-
-| Command | Raw tokens | Distilled | Saved |
-|---|---:|---:|---|
-| `pytest -q` (11 failures) | 3,374 | 1,317 | **61%**, all 11 `FAILED` lines kept |
-| `git log -50` | 3,064 | 331 | **89%** |
-| `git diff` (30 commits) | 62,833 | 8,635 | 86%, **unsupported, see below** |
-| `git log --oneline -30` | 321 | 321 | 0%, already compact |
-| `git status` (clean) | 83 | 83 | 0%, too small to distill |
-
-The two 0% rows are the net-positive guard working: distill never inflates small
-output. One run per command on one repository, so these are point measurements.
-Reduction is also not comprehension: the bytes removed are measured, the evidence
-they were safe to remove is narrower.
-
-**One row above is inflated, by the mechanism that broke RTK's number.** A saving may
-only count output the agent would actually have received, and the host truncates a
-command result at 30,000 characters. `git diff`'s 62,833 raw tokens is eight times
-that cap and predates it. **Treat that 86% as unsupported until re-measured.**
-`pytest` and `git log` sit under the cap and are unaffected.
+<a id="5-code-health-predicts-defects"></a>
-**There is a comparable tool and we have not run it.**
-[RTK](https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings/) does this
-and essentially only this. It is also the tool whose advertised 60 to 90% came back
-**7.6% more expensive** under JetBrains' rerun. So this table is a before-and-after
-of our own output, which is the weaker design. Only a paired run would settle it.
-
-Full guide: **[docs/agent/DISTILL.md](agent/DISTILL.md)**
-
----
-
-## 5. Code health predicts defects
+## Code health predicts defects
A health score is worth something only if the files it flags are the files that
break. Scores are taken at a historical commit, bug fixes are counted over the
following six months, and nothing after the scoring commit feeds the score.
-Across **21 repositories, 9 languages, 2,826 files**: **ROC AUC 0.737**
-(95% CI 0.683 to 0.787), ranging 0.55 to 0.86 across individual repos. It beats
-recent churn by +0.100 AUC and prior-defect history by +0.117 (DeLong p < 1e-9). On
+Across **21 repositories, 9 languages, 2,826 files**: **ROC AUC 0.737** (95% CI
+0.683 to 0.787), ranging 0.55 to 0.86 across individual repos. It beats recent
+churn by +0.100 AUC and prior-defect history by +0.117 (DeLong p < 1e-9). On
PROMISE/jEdit, a dataset it never saw which carries no git history at all, it holds
at **0.76 to 0.78**.
-**It is not better than raw file size at discrimination.** LOC-only scores 0.742
-against our 0.737 (p = 0.92, a tie). Where it wins is effort-aware ranking, Popt
-+0.134 (95% CI +0.080 to +0.198): same discrimination as counting lines, much better
-at ordering a fixed review budget, and unlike a line count it says why. Holding size
-fixed, within-band AUC runs 0.525 / 0.572 / 0.593 / 0.718 across NLOC quartiles, so
-the signal survives cleanly only in the largest quartile, and a positive control
-confirms that collapse is a real absence rather than too few positives to detect one.
-
**Against CodeScene**, the closest commercial product and the only other vendor in
this category with a published empirical defect study. Both tools scored the same
2,770 files at the same leakage-free commit against the same labels:
@@ -375,15 +526,13 @@ this category with a published empirical defect study. Both tools scored the sam
| Discrimination (ROC AUC) | 0.731 | 0.705 | p = 0.054, marginal |
| Precision at a 20%-of-lines review budget | 0.580 | **0.636** | p = 0.64, a tie |
-Ranking by repowise health surfaces **2.3x the defects under a fixed review budget**.
+Ranking by repowise health surfaces **2.3x the defects under a fixed review
+budget**.
-**Where CodeScene is ahead, and why it is a real choice.** Its precision lead is not
-significant, but the behaviour behind it is worth having: it flags about **27 files**
-where we flag **132**, trading recall for a short list a team will actually work
-through. That is a deliberate operating point, not a weaker model. Our AUC edge is
-**marginal**, and so is our raw defect-density lead (16.9x against 14.2x, p = 0.65 on
-a heavy tail); the size-normalized version is the one that reaches significance and
-the one to cite.
+**Where CodeScene is ahead, and why it is a real choice.** Its precision lead is
+not significant, but the behaviour behind it is worth having: it flags about **27
+files** where we flag **132**, trading recall for a short list a team will actually
+work through. That is a deliberate operating point, not a weaker model.
**The axis we lost outright.** CodeScene's "Code Red" study reports a **-0.58**
correlation between Code Health and mean issue-resolution time, on proprietary Jira
@@ -393,13 +542,42 @@ queue-independent effort signals all came back flat with every interval spanning
zero. Merge time on GitHub largely measures maintainer review-queue availability.
**The business-impact axis remains CodeScene's, unreplicated on open data.**
+<details>
+<summary><b>Method, limits, and what this does not show</b></summary>
+
+**It is not better than raw file size at discrimination.** LOC-only scores 0.742
+against our 0.737 (p = 0.92, a tie). Where it wins is effort-aware ranking, Popt
++0.134 (95% CI +0.080 to +0.198): the same discrimination as counting lines, much
+better at ordering a fixed review budget, and unlike a line count it says why.
+
+**The signal is weak among files of similar size.** Holding size fixed, within-band
+AUC runs 0.525 / 0.572 / 0.593 / 0.718 across NLOC quartiles, so it survives
+cleanly only in the largest quartile, and a positive control confirms that collapse
+is a real absence rather than too few positives to detect one. A prior-defects
+baseline also still beats us on Popt by 0.085 even while losing on AUC.
+
+Our AUC edge over CodeScene is **marginal**, and so is our raw defect-density lead
+(16.9x against 14.2x, p = 0.65 on a heavy tail); the size-normalized version is the
+one that reaches significance and the one to cite.
+
+Marker weights are **calibrated against a real defect corpus, not hand-tuned**:
+every file scored at a commit preceding the bug window so nothing leaks backward,
+and an L2-logistic fit with file size as an explicit control, so a marker only
+earns weight for defect lift *beyond* being big. Only **26** of the 49 detectors
+are permitted to move the defect number, because that is the number carrying these
+accuracy claims.
+
Reports:
**[BENCHMARK\_REPORT.md](https://github.com/repowise-dev/repowise-bench/blob/master/health-defect/BENCHMARK_REPORT.md)** ·
**[COMPARISON\_REPORT.md](https://github.com/repowise-dev/repowise-bench/blob/master/health-defect/COMPARISON_REPORT.md)**
+</details>
+
---
-## 6. Indexing time, the row we lose
+<a id="6-indexing-time-the-row-we-lose"></a>
+
+## Indexing time, the row we lose
We are the slowest indexer in the field, on every repo we measured, and it is not
close. On `django/django`:
@@ -413,292 +591,151 @@ close. On `django/django`:
| **repowise** (default, prose on) | **1,058s** | five layers plus generated documentation |
That is **22x** CodeGraph like for like, and **135x** with prose on, which is what a
-default `repowise init` actually costs you. Both numbers ship, and the 22x is not the
-user-facing one.
+default `repowise init` actually costs you. Both numbers ship, and the 22x is not
+the user-facing one.
-**Here is what the extra time buys.** The tools above build a call graph. In the same
-run, repowise builds: a **graph** of 36,485 nodes, 90,477 edges and 31,384 symbols
-plus PageRank, betweenness, Leiden communities and execution-flow tracing; **git**
-history mined across 2,630 files for hotspots, ownership, co-change pairs and bus
-factor; **3,392 wiki pages** rendered and embedded for natural-language search;
-**architectural decisions** mined from history and sessions; and **code health**,
-5,317 findings plus 155 unreachable files and 98 unused exports.
+**Here is what the extra time buys.** The tools above build a call graph. In the
+same run, repowise builds: a **graph** of 36,485 nodes, 90,477 edges and 31,384
+symbols plus PageRank, betweenness, Leiden communities and execution-flow tracing;
+**git** history mined across 2,630 files for hotspots, ownership, co-change pairs
+and bus factor; **3,392 wiki pages** rendered and embedded for natural-language
+search; **architectural decisions** mined from history and sessions; and **code
+health**, 5,317 findings plus 155 unreachable files and 98 unused exports.
So "22x slower" and "the index contains categorically more" are both true, and
neither cancels the other. If all you want is a call graph, CodeGraph builds one in
16 seconds and you should use it. The comparison that would be dishonest is quoting
-the ratio without the column beside it, which is why the column is here.
+the ratio without the column beside it, which is why the column is here. It is also
+a one-time cost: updates after the first index are incremental.
-It is also a one-time cost. Updates after the first index are incremental.
+<details>
+<summary><b>Method, limits, and what this does not show</b></summary>
A fitted build-cost curve for five tools across a 12x repository-size range, which
nobody in this field had published, is in
[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head#the-result-nobody-in-the-field-had-published-a-build-cost-curve).
-Our exponent is sublinear at 0.906, and we publish that as sublinear **work** rather
-than efficiency, because symbol density halves across the range. That is a product
-finding against us.
-
----
-
-## 7. Edge precision
-
-Every other row on this page counts things. This one asks whether they are **true**.
-
-A resolver that guesses aggressively wins a coverage table and a raw edge count while
-sending its reader to the wrong function. So we hand-graded call edges from source, on
-both sides, by the same method: 30 rows per language per tool, seed 2026, stratified by
-resolution strategy, every row read with its imports and enclosing scope open.
-
-| | correct / n | 95% CI |
-|---|---|---|
-| **repowise** | **229/270 = 84.8%** | [80.0, 88.6] |
-| **CodeGraph 1.5.0** | **154/270 = 57.0%** | [51.1, 62.8] |
-
-Per language, nine languages, both sides read:
-
-| Language | repowise | CodeGraph | |
-|---|---|---|---|
-| typescript | 29/30 | 7/30 | separates |
-| go | 29/30 | 29/30 | tie |
-| csharp | 28/30 | 20/30 | separates |
-| python | 28/30 | 19/30 | separates |
-| kotlin | 27/30 | 13/30 | separates |
-| swift | 23/30 | 19/30 | tie |
-| cpp | 23/30 | 16/30 | tie |
-| rust | 22/30 | 13/30 | tie |
-| java | 20/30 | 18/30 | tie |
+Our exponent is sublinear at 0.906, and we publish that as sublinear **work**
+rather than efficiency, because symbol density halves across the range. That is a
+product finding against us.
-**Read our number the other way round: roughly fifteen percent of our call edges are
-wrong.** That is the number we plan against, and rust, java and cpp are where it
-concentrates.
-
-**Four of nine cells separate. Five are ties and we report them as ties.** At n=30 the
-interval runs about ±16 points near 60%, so a point-estimate gap inside two overlapping
-intervals is not a win. C++ is a tie despite looking like a 23-point lead.
-
-**One repository goes clearly to them.** On `seastar` CodeGraph reads 6/10 against our
-4/10, the only repository in the audit, on any language, where they beat us on a clear
-margin. Our misses there are chained calls on an untyped receiver; they infer the
-callee's declared return type and validate against it, so a failed inference costs them
-an edge instead of buying them a wrong one. On `aria2` both sides read 10/10 and they
-resolve 24,950 distinct call edges to our 9,486. Precision is not the only reading.
-
-Cells were measured at different commits, and the staleness runs **conservative**: every
-resolver change in between only removes wrong edges and gained zero, so 84.8% is a floor.
-No cell was measured on the 0.44.0 release; the full table pins a commit per cell.
-
-**All 540 graded rows are published**, one file per cell, each row carrying the call site,
-the declaration the tool bound it to, the verdict and the reason it was given. A script in
-the same directory rebuilds every table above from them and fails if it disagrees. One cell
-of the eighteen, rust on our side, ships the 30 sites that were read without their per-row
-verdicts, because that cell was re-read on a fresh draw and the verdicts of that read were
-never written down; the file says so on every row.
-**[graph/experiments/g1-edge-precision/rows](https://github.com/repowise-dev/repowise-bench/tree/master/graph/experiments/g1-edge-precision/rows)**.
+</details>
---
-## 8. The same question, against an answer key we do not control
+<a id="4-command-output-compression"></a>
+<a id="command-output-compression"></a>
-Section 7 is hand-graded, by us, on both sides. That is the strongest form of a
-weak thing: **every graph-quality number in this field, ours included, is scored
-against something the publisher controls.** A tool that is confidently wrong in a
-consistent way scores well and the reader has no way to check.
+## Command-output compression
+
+`repowise distill <cmd>` compresses command output *before* the agent reads it:
+errors first, exit code preserved, every omission recoverable through an inline
+`[repowise#<ref>]` marker.
-So we re-asked the question with the answer key taken out of our hands. On Go the
-oracle is the **Go team's own RTA call graph** from
-`golang.org/x/tools/go/callgraph/rta`, over the fully type-checked program. On
-TypeScript it is the **`tsc` type checker's own resolution** of every call site.
-We did not write either one, we cannot tune either one, and anyone with the
-toolchain can regenerate both. Five tools, seven cells, five repositories,
-**37,853 oracle edges**.
+| Command | Raw tokens | Distilled | Saved |
+|---|---:|---:|---|
+| `pytest -q` (11 failures) | 3,374 | 1,317 | **61%**, all 11 `FAILED` lines kept |
+| `git log -50` | 3,064 | 331 | **89%** |
+| `git diff` (30 commits) | 62,833 | 8,635 | 86%, **unsupported, see below** |
+| `git log --oneline -30` | 321 | 321 | 0%, already compact |
+| `git status` (clean) | 83 | 83 | 0%, too small to distill |
-### Why precision alone is not the claim, and neither is recall
+**One row above is inflated, by the mechanism that broke RTK's number.** A saving
+may only count output the agent would actually have received, and the host
+truncates a command result at 30,000 characters. `git diff`'s 62,833 raw tokens is
+eight times that cap and predates it. **Treat that 86% as unsupported until
+re-measured.** `pytest` and `git log` sit under the cap and are unaffected.
-Two numbers, and you need both, because each one on its own has a cheap way to
-win.
+<details>
+<summary><b>Method, limits, and what this does not show</b></summary>
-- **Recall** is the share of the real call graph a tool found. Draw an edge
- between everything and you score 1.000.
-- **Precision** is the share of a tool's edges that are real. Draw one edge you
- are sure of and you score 1.000.
+**There is a comparable tool and we have not run it.**
+[RTK](https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings/) does
+this and essentially only this. It is also the tool whose advertised 60 to 90% came
+back **7.6% more expensive** under JetBrains' rerun. So this table is a
+before-and-after of our own output, which is the weaker design. Only a paired run
+would settle it.
-Both failure modes are in the table below, measured, not hypothetical. One tool
-draws 12,533 edges on syft and more than a third of them are calls the compiler
-says do not exist. Another scores the highest precision on this whole page,
-0.997, from a graph containing **17% of the calls in the repository**: on
-gitleaks it stored 4,367 call rows of which 76 resolve to anything, and a graph
-that small is very hard to be wrong in and not much use to walk.
+The two 0% rows are the net-positive guard working: distill never inflates small
+output. One run per command on one repository, so these are point measurements.
+Reduction is also not comprehension: the bytes removed are measured, the evidence
+they were safe to remove is narrower.
-**So the only meaningful reading is the pair**, and it is the pair we claim:
+Full guide: **[docs/agent/DISTILL.md](agent/DISTILL.md)**
-> **In all seven cells, no tool that recovers as much of the call graph as we do
-> gets more of it right.**
+</details>
-That sentence names no threshold, which is what makes it worth something. It
-cannot be tuned by picking a cutoff, and adding a competitor can only break it.
-Two competitors were added to this experiment after it was first written, and it
-held in all seven cells.
+---
-**Precision: of the call edges a tool emits, the share the compiler confirms.**
+<a id="3-loading-one-commits-context-the-easy-number"></a>
+<a id="loading-one-commits-context-the-easy-number"></a>
-| cell | repowise | CodeGraph 1.5.0 | codebase-memory-mcp 0.10.8 | Graphify 0.9.31 | code-review-graph 2.3.7 |
-|---|---|---|---|---|---|
-| cobra (with tests) | 0.972 | 0.929 | 0.912 | 0.971 | **0.997** |
-| gitleaks (no tests) | 0.976 | 0.972 | 0.934 | **0.997** | 0.759 |
-| gitleaks (with tests) | 0.974 | 0.971 | 0.922 | **0.995** | 0.800 |
-| syft (no tests) | 0.943 | 0.872 | 0.635 | 0.771 | **0.968** |
-| syft (with tests) | 0.950 | 0.864 | 0.673 | 0.802 | **0.966** |
-| zod (no tests) | **0.992** | 0.729 | 0.987 | 0.825 | 0.932 |
-| hono (no tests) | 0.977 | 0.805 | 0.949 | 0.980 | 0.966 |
+## Loading one commit's context, the easy number
-**Recall: of the edges the compiler has, the share the tool found.**
+This is the measurement almost every tool in this category publishes, and we are
+labelling it as such. Over the 30 most recent non-merge commits of `pallets/flask`,
+counted with deterministic `tiktoken` (`cl100k_base`):
-| cell | repowise | CodeGraph | codebase-memory-mcp | Graphify | code-review-graph |
-|---|---|---|---|---|---|
-| cobra (with tests) | 0.684 | **0.763** | 0.743 | 0.433 | 0.174 |
-| gitleaks (no tests) | 0.955 | 0.920 | **0.967** | 0.886 | 0.026 |
-| gitleaks (with tests) | 0.914 | 0.895 | **0.945** | 0.832 | 0.032 |
-| syft (no tests) | 0.513 | 0.508 | **0.542** | 0.447 | 0.201 |
-| syft (with tests) | 0.322 | 0.338 | **0.361** | 0.273 | 0.086 |
-| zod (no tests) | **0.703** | 0.373 | 0.694 | 0.248 | 0.652 |
-| hono (no tests) | **0.731** | 0.684 | 0.686 | 0.688 | 0.691 |
+| Strategy | Tokens per commit |
+|---|---:|
+| naive, full contents of every changed file | 13,984 |
+| `git diff` only | 1,408 |
+| **`get_context`** | **393** |
-Read a row across both tables and the trade is visible in every one. The tools
-above us on precision are below us on recall, every time. The tool above us on
-recall is below us on precision, every time. **Nobody is above us on both, in any
-cell.** Intervals, per-cell artifacts and the full method are linked at the end
-of this section; ties are ties and are marked as ties there.
+**35.6x fewer than naive, pooled**, 29.3x as a mean of per-commit ratios, and 3.6x
+pooled against `git diff`.
-**The weaker readings, so nobody has to infer them.** We are the most precise arm
-outright in one cell of seven and tied for it in one more. We are beaten on
-precision in five, by code-review-graph on cobra and both syft cells and by
-Graphify on both gitleaks cells. Against the two tools this experiment started
-with, CodeGraph and codebase-memory-mcp, we are the most precise in seven of
-seven, and that narrower claim should always carry its label.
-
-This also does something section 7 cannot: it removes the n=30 ceiling.
-Hand-grading caps a precision estimate at roughly ±13 points near a 95% rate. An
-oracle grades every edge, so n becomes the size of the repository.
-
-**The two methods agree.** On Go, section 7 read 29/30 = 96.7% by hand for us and
-29/30 = 96.7% for CodeGraph. The Go compiler, over roughly 1,600 edges on the same
-repository, says 97.6% and 97.2%. Two unrelated methods, one person reading source
-and one type checker, land within about a point on both arms. That is the result
-we care about most and it is not a competitive one.
-
-### The column we lose
-
-We lead recall in the two TypeScript cells and in **none of the five Go cells**,
-where codebase-memory-mcp leads four and CodeGraph the fifth. On cross-file
-coverage over 35 repositories, codebase-memory-mcp separates from us on 15 and we
-separate on none. Those are real losses and they are not softened here.
-
-What the oracle adds is the price of that lead rather than an excuse for ours: on
-syft, more than a third of what that tool emits is a call the Go compiler says
-does not exist. Coverage rewards drawing edges and never asks whether they are
-real, which is why no page in that benchmark publishes a coverage number without
-a precision number beside it.
-
-**Where our own miss goes**, decomposed rather than waved at. On syft without
-tests we miss 3,846 of the oracle's 7,898 edges: 44% of that miss is dynamic
-dispatch alone and a further 39% is dispatch with a closure at one end, and the
-two buckets overlap so neither is the whole gap. Interface dispatch is the
-ceiling and nobody in the comparison has cleared it, at 6.5 distinct possible
-targets per call site. Matching that recall means emitting six edges where one is
-right, which is the behaviour the precision table charges other tools for.
+**What this does not tell you.** It is one payload, not a session. It says our
+representation of a commit is smaller than the commit, not that an agent finishes
+faster. The honest version of that question is the agent loop above, where the
+answer on the same harness is a much more modest 15.9%.
-**Do not compare recall across rows.** It swings from 0.03 to 0.97, driven by how
-many entry points the oracle had (4 on gitleaks, 268 on syft-with-tests), not by
-tool quality. Only within-row comparisons carry meaning and a pooled recall over
-these cells would be meaningless.
+<details>
+<summary><b>Method, limits, and what this does not show</b></summary>
-### Limits
+**Lead with the pooled figure.** Pooled is sum over sum, so it weights each commit
+by the tokens actually at stake. A mean of per-commit ratios does not: a one-line
+commit where `get_context` returns 40 tokens contributes a huge ratio that counts
+equally against one saving a hundred thousand. That is how a 35x becomes a 209x in
+a press release.
-- **Two languages, seven cells, five repositories.** This is not a nine-language
- claim and must not be quoted as one. Section 7 is the nine-language number.
-- **A contradicted edge is very strong evidence, not proof.** RTA is unsound under
- reflection and `go:linkname`, so a genuinely dynamic edge can land in that
- bucket. It applies to all five arms equally and the gaps are far too large to
- be explained by it, which is why the metric is named *precision against the
- oracle* rather than precision.
-- **Edges the oracle cannot speak about are charged to nobody** and reported at
- full size. That bucket is 0.4% to 11.1% of a tool's output on the Go cells and
- 16% to 46% on the TypeScript ones, where dependencies are not installed in the
- pinned corpus.
-- **A library has no `main`, so RTA has no roots**; cobra is analysed through its
- test binaries only.
-- The two variants of a repository answer different questions. Report both or say
- which one you used.
-- **Two oracle languages, and the programme stops at two.** C#, Java, Kotlin and
- C++ each need a toolchain installed and a working build per repository, none of
- which exists on the measurement machine. Rust has the toolchain and no sound
- call-graph tool exists for it. Python, Ruby and PHP admit no oracle even in
- principle. So section 7 is the permanent method on those languages rather than
- a stopgap, and nothing here should be read as a claim about them.
-- **The TypeScript cells exclude test files, and the with-tests variants are
- void.** A test file imports its own package by name, the pinned corpus has no
- dependencies installed, and roughly a third of call sites go unresolvable,
- which takes the unjudged bucket past three quarters. Both variants were run and
- neither is quoted anywhere. Installing the dependency trees would fix it and
- would also have to go to a scratch copy, since `node_modules/` inside the
- corpus changes what every tool walks.
-- **The three competitors here are priced on these two languages only.**
- codebase-memory-mcp, Graphify and code-review-graph have no precision figure of
- any kind on the other nine languages in the coverage bench, and none of the
- three was ever entered into the section 7 audit in either direction. Section 7
- is a two-tool claim and section 8 a five-tool one, and neither transfers to the
- other's languages.
-- **Two of the five arms are read through an adapter we wrote**, and both now beat
- us on precision in some cell, so the reading matters. Graphify tags 93% of its
- call edges `INFERRED` rather than AST-certain and we score all of them, which is
- the choice least favourable to us. code-review-graph stores unresolved callees
- beside resolved ones and we score only the resolved, which is the choice most
- favourable to it: on gitleaks that is 76 edges out of 4,367 stored rows. Both
- choices are argued in the benchmark and a reader who disagrees can recompute
- either from the artifacts.
+Raw CSV:
+**[token\_efficiency\_flask30](https://github.com/repowise-dev/repowise-bench/blob/master/results/bakeoff_2026_08/rung1/token_efficiency_flask30_2026-08-01.csv)**
-Full method, per-cell artifacts, the twenty hand-confirmed identities the protocol
-required, and the graded pre-registration including the two predictions that
-missed:
-**[graph/experiments/g4-oracle-anchored](https://github.com/repowise-dev/repowise-bench/tree/master/graph/experiments/g4-oracle-anchored)**.
+</details>
---
-## Limits
-
-Beyond the ones stated in each section:
-
-- **Every retrieval number here is Python or Go.** The graph sections are wider,
- at nine languages hand-graded and two compiler-graded, and they are the only
- sections that are. A JavaScript/TypeScript corpus
- (`mui/material-ui`, six tools, a 12x size range) is built and half graded, but
- **its sealed half is unrun and nothing from it is quoted here.** Publishing the
- development half is what that split exists to prevent, so the row arrives when the
- sealed half is evaluated, once.
-- **We index code files only, which depresses our own coverage on repositories with
- documentation.** Deliberate: on doc-heavy repositories the docs outweigh the code,
- and indexing them polluted the index and degraded retrieval for the code questions
- the tool exists to answer. On the JavaScript corpus above, **21% of gold files are
- `.md` or `.json`**, unreachable to us by construction rather than by ranking.
- cocoindex makes the opposite trade, is the only arm of six to retrieve any
- documentation gold, and pays for it on code gold.
-- **§2 is one repository**, `django/django` at one commit, which is in every model's
- training data.
-- **§2 measures single-session questions**, roughly four to nine turns each. Nothing
- on this page measures a long multi-hour engineering task, and we will not imply a
- number for one.
-- **§1 quotes only the sealed half.** Pooling both halves gives a stronger p and we
- do not quote it.
-- **§2's difficulty split is post-hoc.** The pre-registered comparisons are stronger
- evidence.
-- **§4 has no head-to-head and one row that needs re-measuring.**
-- **§5's signal is weak among files of similar size**, and a prior-defects baseline
- still beats us on Popt by 0.085 even while losing on AUC.
-- **§7 was graded by us on both sides.** That is what §8 exists to check, and the two
- agree where both exist, but only §8 has an answer key we did not produce.
-- **§7 and §8 measure precision, which is one of two halves.** We lead no recall cell
- in §8 and lose cross-file coverage on 15 of 35 repositories in the same bench. A
- precision win is a claim about the edges a tool draws, never about how many.
+## Limits that apply to the whole page
+
+<details>
+<summary><b>Six limits that are not specific to any one section</b></summary>
+
+- **Every retrieval number here is Python or Go.** The graph sections are wider, at
+ nine languages hand-graded and two compiler-graded, and they are the only
+ sections that are. A JavaScript/TypeScript corpus (`mui/material-ui`, six tools,
+ a 12x size range) is built and half graded, but **its sealed half is unrun and
+ nothing from it is quoted here.** Publishing the development half is what that
+ split exists to prevent, so the row arrives when the sealed half is evaluated,
+ once.
+- **We index code files only, which depresses our own coverage on repositories
+ with documentation.** Deliberate: on doc-heavy repositories the docs outweigh the
+ code, and indexing them polluted the index and degraded retrieval for the code
+ questions the tool exists to answer. On the JavaScript corpus above, **21% of
+ gold files are `.md` or `.json`**, unreachable to us by construction rather than
+ by ranking. cocoindex makes the opposite trade, is the only arm of six to
+ retrieve any documentation gold, and pays for it on code gold.
+- **The agent-loop section is one repository**, `django/django` at one commit, which
+ is in every model's training data, and it measures single-session questions of
+ roughly four to nine turns each. **Nothing on this page measures a long
+ multi-hour engineering task**, and we will not imply a number for one.
+- **Precision is one of two halves.** We lead no recall cell against the oracle and
+ lose cross-file coverage on 15 of 35 repositories in the same bench.
+- **The health signal is weak among files of similar size**, and it ties raw line
+ count on discrimination.
+- **Command-output compression has no head-to-head and one row that needs
+ re-measuring.**
+
+</details>
---
@@ -712,66 +749,55 @@ claims on real agent work.
| [Caveman](https://blog.jetbrains.com/ai/2026/07/speak-to-ai-agents-like-cavemen-tosave-tokens/) | 65% | **8.5%** |
| [RTK](https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings/) | 60 to 90% | **7.6% more expensive** at low reasoning effort (p = 0.004) |
-Greptile's 82% became 45% under Augment's rerun. The pattern is consistent enough to
-be a rule: in this category, the advertised number and the reran number are different
-numbers. They differ for one reason.
-
-**Measuring one context load is easy. Measuring an agent loop is hard.** Showing that
-your representation of a file is smaller than the file is a real measurement, and the
-one almost everybody publishes. It is not the question anyone has, which is whether
-an agent given your tool finishes the job having done less work. Agents re-read,
-backtrack, re-plan and re-explore, so a compression that looks like 90% on one
-payload routinely nets out near zero across a session, and can go negative when the
-agent works harder to recover what you compressed away.
-
-**This page publishes both and labels which is which.** §3 is the easy
-one-context-load figure. §2 is the agent-loop figure, measured the way JetBrains
-measured, and deliberately the smaller number. We reran our own headline on a second
-and then a third harness and published what came back: it moved a number we had been
-quoting, and §2 carries the correction.
-
-Four rules apply to every number here, and each exists because breaking it produced a
-wrong published figure at least once:
-
-- **Pre-register before spending**, as its own commit, so a favourable result cannot
- become a different question afterwards.
-- **Seal a half**, split by instance id before any work begins, evaluated once.
-- **Precision and files-served beside coverage**, never averaged into one figure.
-- **Prove an arm was alive and its extractor works before recording a zero.** A dead
- server, a wrong tool name and a broken output parser all score exactly like a bad
- tool.
-
-The full method, drawn end to end with every gate and the failure that put it there,
-is
+Greptile's 82% became 45% under Augment's rerun. The pattern is consistent enough
+to be a rule: in this category, the advertised number and the reran number are
+different numbers. **Measuring one context load is easy. Measuring an agent loop is
+hard.** Agents re-read, backtrack, re-plan and re-explore, so a compression that
+looks like 90% on one payload routinely nets out near zero across a session.
+
+This page publishes both and labels which is which. We reran our own headline on a
+second and then a third harness and published what came back: it moved a number we
+had been quoting, and the agent-loop section carries the correction.
+
+Four rules apply to every number here, each because breaking it produced a wrong
+published figure at least once. **Pre-register before spending**, as its own commit,
+so a favourable result cannot become a different question afterwards. **Seal a
+half**, split by instance id before any work begins, evaluated once. **Precision and
+files-served beside coverage**, never averaged into one figure. **Prove an arm was
+alive and its extractor works before recording a zero**, because a dead server, a
+wrong tool name and a broken output parser all score exactly like a bad tool. The
+method end to end, with every gate and the failure that put it there:
**[THE\_LOOP.md](https://github.com/repowise-dev/repowise-bench/blob/master/head-to-head/THE_LOOP.md)**.
---
## Where the depth is
-This page is the summary. Stop wherever you like; each level is the evidence for the
-one above it.
+This page is the summary. Everything below it is the evidence, in
+**[repowise-bench](https://github.com/repowise-dev/repowise-bench)**: what each run
+cost in machine hours and API spend, the build-cost curve, one page per competitor
+with every setup trap, all nine method gates, every launch command and exclusion
+with its reason, one pre-registration per scored run committed before that run spent
+anything, and every graded cell including the runs we invalidated with their
+invalidation notes attached.
-| Level | What is there |
+| Start here | For |
|---|---|
-| **[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head)** | Who wins what, the build-cost curve, and what each index can rank at all |
-| **[graph/](https://github.com/repowise-dev/repowise-bench/tree/master/graph)** | The graph-quality bench: edge precision hand-graded on both sides across nine languages against one competitor, precision and recall against a compiler oracle on Go and TypeScript against two, and cross-file coverage, adversarial invariance and build cost across five |
-| **[arms/](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head/arms)** | One page per competitor: what it is, what it serves, and every setup trap. Four of six have a step that produces a clean zero when missed |
-| **[THE\_LOOP.md](https://github.com/repowise-dev/repowise-bench/blob/master/head-to-head/THE_LOOP.md)** | The method and all nine gates, each named with the failure that created it |
-| **[configs/arms.yaml](https://github.com/repowise-dev/repowise-bench/blob/master/configs/arms.yaml)** | Every launch command, allowlisted tool and exclusion with its reason. Read this if you think an arm was set up unfairly |
-| **[configs/\*.PREREGISTRATION.md](https://github.com/repowise-dev/repowise-bench/tree/master/configs)** | One per scored run, each committed before its run spent anything |
-| **[results/bakeoff\_2026\_08](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08)** | Every graded cell and every verbatim response, including the runs we invalidated, with their invalidation notes attached |
+| **[graph/](https://github.com/repowise-dev/repowise-bench/tree/master/graph)** | The graph-quality bench: hand-graded precision across nine languages, the compiler oracle on Go and TypeScript, cross-file coverage and build cost across five tools |
+| **[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head)** | The retrieval and agent-loop bench: who wins what, what each index can rank at all, and what it all cost to produce |
+| **[results/bakeoff\_2026\_08](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08)** | Every graded cell and every verbatim response |
| **[repro/README.md](https://github.com/repowise-dev/repowise-bench/blob/master/repro/README.md)** | Per claim: what it costs to reproduce, how long it takes, and which ones need credentials we cannot hand you |
-**Found a problem with one of these numbers, or want your tool in the field?** Adding
-a competitor is a YAML block, no Python and no runner change. See
+**Found a problem with one of these numbers, or want your tool in the field?**
+Adding a competitor is a YAML block, no Python and no runner change. See
[CONTRIBUTING.md](https://github.com/repowise-dev/repowise-bench/blob/master/CONTRIBUTING.md).
-Tool versions as measured: CodeGraph 1.5.0, Graphify 0.9.31, Serena 1.6.2.dev0,
-code-review-graph 2.3.7, cocoindex as of 2026-08-09, codebase-memory-mcp 0.10.8.
+<sub>Tool versions as measured: CodeGraph 1.5.0, Graphify 0.9.31, Serena 1.6.2.dev0,
+code-review-graph 2.3.7, cocoindex as of 2026-08-09, codebase-memory-mcp 0.10.8.</sub>
## See also
- [The five intelligence layers](layers/INTELLIGENCE_LAYERS.md)
- [Code health methodology](layers/CODE_HEALTH.md)
- [MCP tool reference](agent/MCP_TOOLS.md)
+- [Roadmap](../ROADMAP.md), including the interface-dispatch recall ceiling
diff --git a/docs/README.md b/docs/README.md
index 370247479..0a2baa527 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -20,7 +20,7 @@ your agent in under five minutes, with no API key.
| **Living in an editor** | [VS Code extension](agent/VSCODE.md) | [Codex](agent/CODEX.md) · [opencode](agent/OPENCODE.md) |
| **A team lead watching what ships** | [Change risk](layers/CHANGE_RISK.md) | [Code health](layers/CODE_HEALTH.md) · [Bug history](layers/BUG_HISTORY.md) |
| **Running many repos** | [Workspaces](scale/WORKSPACES.md) | [Worktrees](scale/WORKTREES.md) · [Auto-sync](scale/AUTO_SYNC.md) |
-| **Evaluating repowise to buy it** | [Commercial](business/COMMERCIAL.md) | [Security & compliance](business/SECURITY_COMPLIANCE.md) · [Benchmarks](BENCHMARKS.md) |
+| **Evaluating repowise to buy it** | [Commercial](business/COMMERCIAL.md) | [Security & compliance](business/SECURITY_COMPLIANCE.md) · [Benchmarks](BENCHMARKS.md) · [Roadmap](../ROADMAP.md) |
| **Contributing** | [Architecture](architecture/README.md) | [CONTRIBUTING](../.github/CONTRIBUTING.md) |
---
@@ -58,7 +58,7 @@ your agent in under five minutes, with no API key.
| [layers/GRAPH.md](layers/GRAPH.md) | The dependency graph: what is in it, how every edge is resolved, and how much to trust each one |
| [layers/DECISIONS.md](layers/DECISIONS.md) | Architectural decisions mined from your repo and from your own agent sessions |
| [layers/DEAD_CODE.md](layers/DEAD_CODE.md) | Unreachable files, unused exports, and zombie packages by confidence tier |
-| [layers/LANGUAGE_SUPPORT.md](layers/LANGUAGE_SUPPORT.md) | What works per language, across 19 parsed languages and 13 at the Full tier |
+| [layers/LANGUAGE_SUPPORT.md](layers/LANGUAGE_SUPPORT.md) | What works per language: 19 parsed to a full AST, 35 on the five-rung ladder |
| [layers/WIKI.md](layers/WIKI.md) | The generated wiki: page types, what `update` re-renders, styles, output language |
## Scale it
@@ -86,7 +86,7 @@ your agent in under five minutes, with no API key.
| Doc | What it covers |
|-----|----------------|
-| [BENCHMARKS.md](BENCHMARKS.md) | Agent-efficiency, distillation, and defect-prediction results, each with what it does not show |
+| [BENCHMARKS.md](BENCHMARKS.md) | Every published number with its sample size and its test: call-graph precision judged by a compiler we do not control, retrieval against the agent-context field, defect prediction against CodeScene, distillation, and the rows we lose |
## Teams & business
@@ -94,6 +94,7 @@ your agent in under five minutes, with no API key.
|-----|----------------|
| [business/COMMERCIAL.md](business/COMMERCIAL.md) | Hosted tier, enterprise (on-prem, SSO/SCIM), and commercial licensing |
| [business/SECURITY_COMPLIANCE.md](business/SECURITY_COMPLIANCE.md) | What leaves your machine, what gets stored, and the answers your security team wants |
+| [ROADMAP.md](../ROADMAP.md) | What we are building next, what we are not building, and why there are no dates on it |
## Architecture & internals
diff --git a/docs/business/COMMERCIAL.md b/docs/business/COMMERCIAL.md
index fa60337d6..08f35885e 100644
--- a/docs/business/COMMERCIAL.md
+++ b/docs/business/COMMERCIAL.md
@@ -19,7 +19,7 @@ models. Specific pricing figures are provided in a separate proposal.
## 1. Who the commercial license is for
The open-source distribution covers the full developer-experience surface: the five
-intelligence layers, the eleven MCP tools, multi-repo workspaces, the local dashboard,
+intelligence layers, the ten MCP tools, multi-repo workspaces, the local dashboard,
auto-sync, and auto-generated `CLAUDE.md`. That is everything an engineer or a team
needs to make their AI coding agents codebase-aware.
@@ -41,21 +41,30 @@ scale, in a regulated or security-sensitive environment**:
All of the following ship in `pip install repowise` today, free for internal use.
-- **Five intelligence layers**: Graph (tree-sitter AST across 19 languages, two-tier
- dependency graph, call resolution, heritage extraction, Leiden communities,
- PageRank / betweenness / SCC), Git (hotspots, ownership, co-change pairs, bus
- factor, significant commits, contributor profiles, module health), Documentation
- (LLM-generated wiki, freshness scoring, RAG search), Decision (architectural
+- **[Five intelligence layers](../layers/INTELLIGENCE_LAYERS.md)**: Graph
+ (tree-sitter AST across 19 languages, two-tier dependency graph, call
+ resolution, heritage extraction, Leiden communities, PageRank / betweenness /
+ SCC), Git (hotspots, ownership, co-change pairs, bus factor, significant
+ commits, contributor profiles, module health), Documentation (a wiki page per
+ module and file, freshness scoring, hybrid search), Decision (architectural
decision records linked to graph nodes, staleness tracking), and Code Health
(49 deterministic detectors, 1–10 score per file, coverage ingestion, trend
- alerts).
+ alerts). Full detail on each, and what every layer costs to build:
+ **[INTELLIGENCE_LAYERS.md](../layers/INTELLIGENCE_LAYERS.md)**.
+- **Zero LLM calls in every analysis layer.** Graph, git, code health, change
+ risk, dead code and the PR bot make no model calls at all, and the whole wiki
+ renders from code structure with no API key (`repowise init --no-prose`).
+ Model-written prose is opt-in, priced before you confirm it, and runs on your
+ own key or fully offline through Ollama. Six of the seven decision sources are
+ deterministic as well; only comment archaeology, which reads rationale prose on
+ high-centrality code, needs a provider.
- **Ten task-shaped MCP tools**: `get_overview`, `get_answer`, `get_context`,
`get_symbol`, `search_codebase`, `get_risk`, `get_change_risk`, `get_why`,
- `get_dead_code`, `get_health`. Benchmarked at **−33.5 % cost per question**
- against a bare agent, cheaper on 13 of 15 questions (n=15, p=0.007), and
- **−49 % to −70 % tool calls** across paired runs on `pallets/flask` and
- `scikit-learn`. See [docs/BENCHMARKS.md](../BENCHMARKS.md) for the sample
- sizes, the tests, and the runs where the cost saving did not replicate.
+ `get_dead_code`, `get_health`, plus `list_repos` for workspace routing.
+ Measured at **−31.6 % of the agent's own output tokens** against a bare agent
+ on 43 questions (p < 0.0001), leaner on 37 of 44, reached in 3.8 tool calls
+ where the bare agent needed 7.2. See [docs/BENCHMARKS.md](../BENCHMARKS.md) for
+ the sample sizes, all three agent harnesses, and the rows where we lose.
- **Multi-repo workspace intelligence**: cross-repo co-changes, API contract
extraction (HTTP / gRPC / topics) with provider↔consumer matching, package
dependency mapping, federated MCP queries (`repo="all"`), workspace dashboard and
@@ -70,6 +79,13 @@ All of the following ship in `pip install repowise` today, free for internal use
Security (local pattern scan), Knowledge Map, and the workspace views.
- **Dead-code detection**: pure graph traversal, confidence-tiered, framework-aware
(ASP.NET, Django, FastAPI, Flask, Rails, Laravel), dynamic-import aware.
+- **Test intelligence, from a coverage report and from the call graph.** Ingests
+ LCOV / Cobertura / Clover like a coverage service, then does the half a coverage
+ service structurally cannot: answers *which tests reach this file* and *which
+ tests does this diff exercise* **with no report at all**, at **95.7% and 97.5%
+ precision** measured against a real `coverage run --contexts=test`. Measured and
+ inferred rows are labelled and never averaged. Zero LLM calls, no CI integration.
+ ([TEST_INTELLIGENCE.md](../layers/TEST_INTELLIGENCE.md))
- **Privacy** (self-hosted): source never leaves your infrastructure, BYOK or fully
offline via Ollama. Anonymous, opt-out usage telemetry (command names and coarse
environment only, never code, paths or repo names) can be turned off with
@@ -83,14 +99,33 @@ All of the following ship in `pip install repowise` today, free for internal use
## 3. First-class language coverage
-Repowise treats **13 languages at Full tier** (Python, TypeScript, JavaScript, Svelte,
-Vue, Java, Kotlin, Go, Rust, C++, **C#**, Scala, and Ruby) with AST parsing, import
-resolution, named bindings, call resolution, heritage extraction, multi-project
-workspace resolvers, framework-aware edges, per-language dynamic-hint extractors, and
-code-health markers. A further 5 languages (C, Swift, PHP, Dart, Object Pascal/Delphi)
-sit at Good tier,
-and Luau is partial. SQL/dbt, shell, HTML, and the config formats are handled by
-dedicated extractors on top of that.
+Repowise parses **19 languages to a full AST** and places **35 on a five-rung
+ladder**, so "do you support X" gets the rung as its answer rather than a yes or a
+no. Both numbers matter and neither is worth quoting alone.
+
+The 19 are the top three rungs. At **Full tier sit 13** (Python, TypeScript,
+JavaScript, Svelte, Vue, Java, Kotlin, Go, Rust, C++, **C#**, Scala, and Ruby) with
+AST parsing, import resolution, named bindings, call resolution, heritage
+extraction, multi-project workspace resolvers, framework-aware edges, per-language
+dynamic-hint extractors, and code-health markers. A further **5 at Good tier** (C,
+Swift, PHP, Dart, Object Pascal/Delphi) get all of that except the full health
+suite, and Luau is partial. SQL/dbt, shell, HTML, and the config formats are
+handled by dedicated extractors on top of that.
+
+Below the AST line the ladder continues rather than stopping: **7 at Lightweight**
+(Elixir, Clojure, Haskell, Lean 4, Erlang, F#, HTML) get a real file-to-file import
+graph with no symbol-level claims, and **9 at Structural** (Objective-C, R, Zig,
+Julia, Elm, OCaml, Crystal, Nim, D) get the git intelligence layer only: blame,
+hotspots, co-change, ownership and bug history, with no parsing. Stating the rung
+per language is deliberate, because a support claim that averages these together
+would not survive a technical evaluation. Full ladder:
+[LANGUAGE_SUPPORT.md](../layers/LANGUAGE_SUPPORT.md).
+
+**Languages are never gated.** Every one of them ships in the AGPL distribution,
+including the ones still climbing. What is on the way up, **COBOL** among them,
+is on the public [roadmap](../../ROADMAP.md#languages). Where you need a language
+or framework that is not there, §5.4 covers having it built and maintained by us
+as a commercial line item, and the result still ships to everyone under AGPL.
For estates built on a particular stack, the relevant Full-tier capabilities are
worth calling out. For **.NET**, as one example:
@@ -122,12 +157,13 @@ the items that matter most to you can be prioritized.
| Capability | Open Source (AGPL) | Commercial License |
|------------|:------------------:|:------------------:|
| Five intelligence layers | ✅ | ✅ |
-| Eleven MCP tools | ✅ | ✅ |
+| Ten task-shaped MCP tools (plus `list_repos`) | ✅ | ✅ |
| Multi-repo workspaces | ✅ | ✅ |
| Full-tier language support (incl. C# / .NET) | ✅ | ✅ |
| Local dashboard (incl. local security pattern scan) | ✅ | ✅ |
| Auto-sync (hooks, watcher, webhooks) | ✅ | ✅ |
| Auto-generated CLAUDE.md | ✅ | ✅ |
+| Test intelligence (coverage ingestion **and** the graph-inferred test map) | ✅ | ✅ |
| Local full-history secret scan (`repowise security scan --history`) | ✅ | ✅ |
| Graph-aware enhanced security scanning | — | ✅ *(GA on hosted)* |
| Language-specific security rulesets | — | ✅ *(dev)* |
diff --git a/docs/layers/INTELLIGENCE_LAYERS.md b/docs/layers/INTELLIGENCE_LAYERS.md
index 3892c8266..2c071d46c 100644
--- a/docs/layers/INTELLIGENCE_LAYERS.md
+++ b/docs/layers/INTELLIGENCE_LAYERS.md
@@ -33,8 +33,10 @@ searchable in natural language.
- [Auto-sync](#auto-sync-five-ways-to-stay-current)
- [Auto-generated CLAUDE.md](#auto-generated-claudemd)
-Everything below is computed **without model calls**. An LLM is an optional
-upgrade for prose quality in the docs layer, never a requirement for the index.
+Everything below is computed **without model calls**, with two named exceptions,
+both optional and both skipped entirely on a keyless index: prose quality in the
+docs layer, and comment archaeology in the decision layer. Neither is a
+requirement for the index.
---
@@ -147,7 +149,10 @@ Architectural decisions mined at index time from **five sources**: ADR files
(Nygard/MADR), PR and squash-commit bodies, inline markers, git archaeology, and
centrality-bounded code comments. Two further capture paths exist for humans and
agents: `repowise decision add`, and mining your Claude Code or Codex session
-transcripts.
+transcripts. **Seven sources in total, six of them deterministic**; comment
+archaeology is the one that reads prose with a model, and it is skipped on a
+keyless index. Three further sources were retired for producing records nobody
+ever acted on, which [`DECISIONS.md`](DECISIONS.md) names.
```python
# WHY: JWT chosen over sessions — API must be stateless for k8s horizontal scaling
@@ -240,13 +245,22 @@ Reference: [`CHANGE_RISK.md`](CHANGE_RISK.md).
## Test Intelligence
Which tests actually exercise the code you changed, which changed files have no
-guarding test at all, and per-file coverage merged across every test that
-touches it. Coverage ingests from LCOV, Cobertura, Clover or normalized JSON and
-feeds the code-health coverage markers.
-
-The practical payoff is `tests_to_run` in `get_risk()` PR mode: a
-coverage-backed list rather than a guess, so an agent runs the tests that can
-actually catch its change.
+guarding test at all, and per-file coverage merged across every test that touches
+it. Coverage ingests from LCOV, Cobertura, Clover or normalized JSON and feeds the
+code-health coverage markers.
+
+**It answers with or without that ingest.** Most repositories never produce a
+report, so where one is missing the layer walks the call graph instead: a test
+whose calls reach a file executes it, three hops by default, filtered to the call
+edges the graph is confident in. Measured against a real
+`coverage run --contexts=test`, that reaches **95.7% precision** on what covers a
+file and **97.5% on the run list**. Rows are stamped `basis: "measured"` or
+`basis: "inferred"`, the measured tier wins outright wherever it can answer, the
+two are never averaged, and the inferred tier may never emit a percentage.
+
+The practical payoff is `tests_to_run` in `get_risk()` PR mode: a real list rather
+than a guess, so an agent runs the tests that can actually catch its change, on
+repositories that have never configured coverage at all.
Reference: [`TEST_INTELLIGENCE.md`](TEST_INTELLIGENCE.md).
@@ -289,8 +303,9 @@ repowise dead-code
```
Conservative by design. `safe_to_delete` requires confidence ≥ 0.70 and excludes
-17 dynamically-loaded naming patterns (`*Plugin`, `*Handler`, `*Middleware`,
-`register_*`, `on_*`, `*_route`, `*_callback` and more). Dynamic-import
+14 dynamically-loaded naming patterns (`*Plugin`, `*Handler`, `*Adapter`,
+`*Middleware`, `*Mixin`, `*Command`, `register_*`, `on_*`, `*_view`,
+`*_endpoint`, `*_route`, `*_callback`, `*_signal`, `*_task`). Dynamic-import
detection and a per-language framework-convention registry further cut false
positives. repowise surfaces candidates; engineers decide.
diff --git a/docs/layers/LANGUAGE_SUPPORT.md b/docs/layers/LANGUAGE_SUPPORT.md
index b44f7b67a..61fd34466 100644
--- a/docs/layers/LANGUAGE_SUPPORT.md
+++ b/docs/layers/LANGUAGE_SUPPORT.md
@@ -1,9 +1,11 @@
# Language Support
-**19 languages parsed to a full AST · 13 at the Full tier · framework-aware
-across all of them.** Everything else in your repo still appears in the wiki and
-is tracked through git history. This page is the "what works for my language
-today" reference.
+**19 languages parsed to a full AST · 35 on the five-rung ladder ·
+framework-aware across all of them.** "Do you support X" has five useful answers
+rather than two, so every language lands on a rung and the rung says what it
+buys you. Everything else in your repo still appears in the wiki and is tracked
+through git history. This page is the "what works for my language today"
+reference.
<p>
<strong>Full tier &nbsp;</strong>
@@ -49,11 +51,17 @@ produce meaningful output.
| Tier | Languages | What you get |
|------|-----------|--------------|
-| **Full** | Python · TypeScript · JavaScript · Svelte · Vue · Java · Kotlin · Go · Rust · C++ · C# · Scala · Ruby | The whole pipeline: AST symbols, import resolution, a resolved call graph, heritage, docstrings, framework edges, **and code-health markers** |
-| **Good** | C · Swift · PHP · Dart · Object Pascal | Everything above except the full health suite. Dart and Object Pascal *do* get health markers; C, Swift and PHP don't yet |
-| **Partial** | Luau / Roblox | AST symbols and `require()` resolution (Rojo / `.luaurc` aware). No health markers yet |
-| **Lightweight** | Elixir · Clojure · Haskell · Lean 4 · Erlang · F# · HTML | A real file-to-file import graph, no symbol-level claims |
-| **Structural** | Objective-C · R · Zig · Julia · Elm · OCaml · Crystal · Nim · D | Git history only: blame, hotspots, co-change. No AST parsing |
+| **Full** (13) | Python · TypeScript · JavaScript · Svelte · Vue · Java · Kotlin · Go · Rust · C++ · C# · Scala · Ruby | The whole pipeline: AST symbols, import resolution, a resolved call graph, heritage, docstrings, framework edges, **and code-health markers** |
+| **Good** (5) | C · Swift · PHP · Dart · Object Pascal | Everything above except the full health suite. Dart and Object Pascal *do* get health markers; C, Swift and PHP don't yet |
+| **Partial** (1) | Luau / Roblox | AST symbols and `require()` resolution (Rojo / `.luaurc` aware). No health markers yet |
+| | | ⎯⎯ *tree-sitter parsing stops here. The rungs below are derived from git and imports, not from an AST.* ⎯⎯ |
+| **Lightweight** (7) | Elixir · Clojure · Haskell · Lean 4 · Erlang · F# · HTML | A real file-to-file import graph, no symbol-level claims |
+| **Structural** (9) | Objective-C · R · Zig · Julia · Elm · OCaml · Crystal · Nim · D | Git history only: blame, hotspots, co-change. No AST parsing |
+
+The first three rungs are the **19 languages parsed to a full AST**; all five are
+the **35** on the ladder. Both numbers are worth stating and neither is worth
+stating alone, so if you only take one thing from this page, take the rung your
+language sits on rather than either count.
[**SQL / dbt**](#sql--dbt) and [**shell**](#shell) sit outside this ladder on
purpose: each has a coverage shape the tiers cannot describe. SQL is parsed by
diff --git a/docs/layers/TEST_INTELLIGENCE.md b/docs/layers/TEST_INTELLIGENCE.md
index 1bde1f112..5de31161c 100644
--- a/docs/layers/TEST_INTELLIGENCE.md
+++ b/docs/layers/TEST_INTELLIGENCE.md
@@ -242,7 +242,7 @@ matching filenames.
Matching filenames fails both ways, and this repository is the proof. Of its six
worst bug-magnet files, five have no file named for them anywhere under `tests/`
-and so read as untested, while the graph names the tests that import them:
+and so read as untested, while the graph names the tests that reach them:
| File | Filename convention | Graph |
|---|---|---|
@@ -257,57 +257,94 @@ The last row is the worse failure. The convention matches on basename alone, so
it paired the health engine with the *distill* engine's tests (a different
subsystem) and called the file tested on the strength of a name collision.
-Across the whole index the effect is large: 74 of repowise's 110 standing
-`untested_hotspot` findings are cleared once the graph is consulted, 18 of them
-graded critical.
+Across the whole index the effect is large. Against the **80** standing
+`untested_hotspot` findings the filename convention leaves behind, the call graph
+clears **32** of them, 22 high and 3 critical, where the one-hop import walk this
+tier used to run clears 11.
-The dependency graph already holds a weaker version of the same relation. A test
-file that imports a source file **reaches** it. That is not proof it executes it,
-and the layer never claims otherwise, but it is a recorded edge rather than a
-name-shaped guess, and it needs no setup at all.
+The graph already holds the relation. A test whose **calls reach** a source file
+executes it, and that is a recorded edge rather than a name-shaped guess. It needs
+no setup at all.
### What it is allowed to claim
| | Measured map | Inferred map |
|---|---|---|
-| Comes from | a coverage report you ingested | the import graph, already indexed |
+| Comes from | a coverage report you ingested | the call graph, already indexed |
| Granularity | lines | files |
-| Proves | this test executed these lines | this test imports this file |
+| Proves | this test executed these lines | this test's calls reach this file |
| Decays | yes, see the coverage age report | no |
| May produce a percentage | yes | **never** |
| Labelled | `basis: "measured"` | `basis: "inferred"` |
-The two are shown side by side and are never averaged into one number. Where
-both can answer, the measured one wins outright and the graph is not consulted;
-the inferred tier only fills silence. Its error is one-sided and known: importing
-a module pulls in every symbol it defines while the test body may touch one
-function, so it **over-claims**. That makes it sound as a floor ("something
-reaches this, do not call it untested") and unsound as a quantity.
-
-### How far it walks
-
-One hop by default: a test importing the file. That default was measured, not
-chosen, by dogfooding against a real `coverage run --contexts=test` on this
-repository over a slice where per-test attribution was complete and both sides
-saw the same test files:
-
-| `max_depth` | reach precision | reach recall | run-list precision | run-list hit rate |
-|---|---|---|---|---|
-| **1** (default) | **72.1%** | 30.4% | **93.1%** | 96.8% |
-| 2 | 40.7% | 45.1% | 73.8% | 97.9% |
-| 3 | 17.1% | 45.1% | n/a | n/a |
-
-Both uses want precision: a false "something reaches this" suppresses a real
-untested-hotspot finding, and a false entry in a run-list sends someone to run a
-test that cannot fail for their change. The extra recall a second hop buys is not
-worth halving either, and by depth 3 the closure has begun to blow up: 269
-claims for the same 46 confirmations.
-
-Depth 1's recall already beats the filename convention it sits beside (30.4%
-against 23.5% on the same set) while resting on an edge rather than a name, and
-the two are unioned rather than ranked, so the shipped floor is better than
-either alone. A repository whose tests import a package facade rather than the
-module under test will want a second hop; it is a keyword argument on both walks.
+The two are shown side by side and are never averaged into one number. Where both
+can answer, the measured one wins outright and the graph is not consulted; the
+inferred tier only fills silence. Its error is one-sided and known, so it is sound
+as a floor ("something reaches this, do not call it untested") and unsound as a
+quantity.
+
+### The call graph leads, the import graph fills silence
+
+The tier first shipped walking **import** edges one hop, which was the best of the
+import-based options and still the wrong graph: a test that imports a module is
+weak evidence it runs it. Release 0.44.0 made the call graph good enough to lead
+with, so it does.
+
+Dogfooded against a real `coverage run --contexts=test` over a slice where
+per-test attribution is complete, both sides seeing the same 37 test files and 159
+provably-executed production files:
+
+**Forward, "what reaches this file" (suppresses `untested_hotspot`):**
+
+| Walk | claims | correct | precision | recall |
+|---|---:|---:|---:|---:|
+| import graph, 1 hop (what shipped before) | 43 | 31 | 72.1% | 19.5% |
+| call graph, 3 hops | 48 | 44 | 91.7% | 27.7% |
+| **call graph, 3 hops, filtered** (ships) | 46 | 44 | **95.7%** | **27.7%** |
+| both unioned | 57 | 45 | 78.9% | 28.3% |
+
+**Reverse, "which tests do I run" (`tests_to_run`, `impacted_tests`):**
+
+| Walk | targets | hit rate | precision |
+|---|---:|---:|---:|
+| import graph, 1 hop (what shipped before) | 32 | 96.9% | 94.8% |
+| call graph, 3 hops, filtered | 46 | 100.0% | 97.5% |
+| both unioned | 47 | 100.0% | 95.8% |
+| **call graph, else import graph** (ships) | 47 | **100.0%** | **97.5%** |
+
+The call graph wins on both axes, so it leads. **The two tiers are combined
+differently per direction, because the measurements differ.** Unioning costs the
+forward walk 16.8 points of precision for 0.6 of recall, and a false "something
+reaches this" hides a real gap, so forward is call edges only. Reverse falls back
+instead of unioning, spending the import tier only on targets the call graph left
+silent, which answers one more target at identical precision.
+
+**Depth 3 is where the call walk saturates**: 3, 4 and 5 hops return the same 48
+claims and 44 confirmations, so the ceiling is the call graph's capture rate rather
+than the depth. The import tier keeps its measured default of one hop.
+
+**"Filtered" is `resolution_origin` doing work.** Not every call edge is equally
+trustworthy, and this is the first thing to read the graph's own confidence
+vocabulary: `global_unique`, which binds a name to the only symbol carrying it
+repo-wide and which the vocabulary itself scores 0.50, is dropped. That buys **4.0
+points of forward precision and 1.1 of reverse at no cost to recall**. A confidence
+floor of 0.90 was tried instead and cut recall from 27.7% to 23.3%.
+
+**Why recall reads low at 27.7%**, which is a property of the truth set rather than
+the walk: `coverage` records a line as run whether a test called into it or Python
+merely evaluated the module body on import, so the truth set cannot tell an
+imported file from an exercised one. Splitting the 159 truth files by how much of
+what ran was inside a function body: 39 ran nothing inside one at all (0% recall,
+correctly), 19 ran under a quarter inside (0%), 45 a quarter to three quarters
+(7%), and 56 over three quarters (**77%**). Of the 13 missed in that last group, 11
+are alembic migrations the framework invokes by naming convention with no static
+caller anywhere.
+
+<sub>Reachability walks <code>EXECUTION_EDGE_TYPES</code>, which is the reachability
+view minus <code>references</code> and <code>reads</code>: those record a mention
+rather than a transfer of control. Dead code asks "is this used", which a mention
+answers; this asks "is this run", which it does not. Both depths are keyword
+arguments on the walks.</sub>
### Nothing is stored