Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.
Python aicode-complexitycode-healthcode-intelligencecode-qualitydead-codedeveloper-toolsdocumentationgit-analyticsmcpopen-sourcepythonrefactoringstatic-analysistechnical-debt
docs: add a roadmap, surface the enterprise docs, and restructure the benchmarks page (#1766)
Raghav Chamadiya689132bf7d82
diff --git a/README.md b/README.mdindex 6680f6d44..39fe5d1b2 100644--- a/README.md+++ b/README.md@@ -37,7 +37,8 @@<a href="#the-ten-mcp-tools">MCP tools</a> ·<a href="#measured-against-the-field">Benchmarks</a> ·<a href="#how-it-compares-on-capability">Comparison</a> ·- <a href="#for-teams--enterprises">Teams</a>+ <a href="#for-teams-and-enterprises">Enterprise</a> ·+ <a href="ROADMAP.md">Roadmap</a></sub></p>---@@ -64,6 +65,11 @@ calls</strong>. <strong>We publish the rows we lose</strong>, and we are the slo<a href="docs/BENCHMARKS.md"><strong>All of it, including the losses →</strong></a><br />Free and self-hosted, runs on your machine, and the first index needs no API key.</sub>+<p align="center"><sub>+Runs entirely on your infrastructure · <strong>zero LLM calls in every analysis layer</strong> ·+AGPL-3.0 or commercial · <a href="#for-teams-and-enterprises"><strong>For enterprises →</strong></a>+</sub></p>+<picture><source media="(prefers-color-scheme: dark)" srcset=".github/assets/one-index-dark.svg" /><img src=".github/assets/one-index.svg" alt="One index producing code health, a dependency graph, git history, generated docs, architectural decisions, and ten MCP tools" width="100%" />@@ -140,8 +146,9 @@ layers and renders every wiki page from your code's structure, with no API key ano spend. Convert any part of it to LLM-written prose whenever you want, one page,one directory, or a ranked coverage slice at a time, and pay only for what you pick,from the CLI or right in the dashboard with the cost shown before you confirm.-(Seven of the eight decision sources are deterministic too; only the one harvested-during doc generation needs a provider.)+(Six of the seven decision sources are deterministic too; only comment+archaeology, which reads rationale prose on high-centrality code, needs a+provider.)Full detail on every layer: **[docs/layers/INTELLIGENCE_LAYERS.md →](docs/layers/INTELLIGENCE_LAYERS.md)**How the graph sees a call, how it resolves one, how much to trust the result, and@@ -189,8 +196,9 @@ Three deterministic signals, all computed from the graph and git history, no LLMDoc, test and config commits are filtered out so the count means what it says, and afile with a run of recent fixes gets flagged as a bug magnet while you edit it.([reference →](docs/layers/BUG_HISTORY.md))-- **Test intelligence.** Ingest coverage, find untested hotspots, and run only the- tests a diff actually exercises with `repowise impacted-tests HEAD~1`.+- **[Test intelligence](#which-tests-cover-this-file-without-a-coverage-report).** Which+ tests reach a file and which ones a diff actually exercises, from the call graph,+ with or without a coverage report.([reference →](docs/layers/TEST_INTELLIGENCE.md))Plus the free **[Repowise PR Bot](#the-pr-bot)**, which puts all of it on every pull@@ -198,6 +206,34 @@ request. Zero LLM calls.---+## Which tests cover this file, without a coverage report++Ingest LCOV, Cobertura or Clover and you get the measured answer. **Most+repositories never produce one**, so the graph answers instead: a test file that+imports a source file *reaches* it, which is a recorded edge rather than the+name-shaped guess everything else falls back to.++That fallback fails in both directions, and this repo is the proof. Five of its+six worst bug-magnet files have no test named for them and read as untested while+the graph names 3 to 23 test files each. The sixth is worse: matching on basename+paired the *health* engine with the *distill* engine's tests and called it tested.++```bash+repowise impacted-tests main..HEAD # only the tests this diff actually exercises+repowise health # untested hotspots, now graph-aware+```++<sub>Dogfooded against a real <code>coverage run --contexts=test</code>:+<strong>95.7% precision</strong> on what reaches a file and <strong>97.5% on the+run list</strong>, at a 100% hit rate, against 72.1% and 94.8% for the one-hop+import walk this replaced. The two tiers are never averaged: rows are stamped+<code>basis: "measured"</code> or <code>"inferred"</code>, measured wins outright+where both can answer, and the inferred tier may never produce a percentage.+Sound as a floor, unsound as a quantity, and labelled so.+<a href="docs/layers/TEST_INTELLIGENCE.md"><strong>Test intelligence →</strong></a></sub>++---+## The PR botInstall the [GitHub App](https://github.com/apps/repowise-bot) and the index shows up@@ -407,7 +443,11 @@ the orchestrators. Full matrix and the contributor recipe:## Supported languages-**19 languages parsed to AST · 13 at the Full tier · framework-aware across all of them.**+**19 languages parsed to AST · 35 on a five-rung ladder · framework-aware across+all of them.**++"Do you support X" has five useful answers, not two, so languages land on a+ladder and every rung says what it buys you.<p><strong>Full tier </strong>@@ -436,6 +476,23 @@ the orchestrators. Full matrix and the contributor recipe:<img src="https://img.shields.io/badge/Luau-00A2FF?style=flat-square&logo=lua&logoColor=white" alt="Luau" /></p>+Below those two rungs the ladder keeps going, and a language on a lower rung is+still doing real work rather than being ignored:++| Rung | Languages | What you get |+|---|---|---|+| **Full** (13) | Python · TypeScript · JavaScript · Svelte · Vue · Java · Kotlin · Go · Rust · C++ · C# · Scala · Ruby | The whole pipeline: AST symbols, import resolution, a resolved call graph, heritage, docstrings, framework edges, **and code-health markers** |+| **Good** (5) | C · Swift · PHP · Dart · Object Pascal | All of the above except the full health suite |+| **Partial** (1) | Luau / Roblox | AST symbols and `require()` resolution, Rojo and `.luaurc` aware |+| | | ⎯⎯ *tree-sitter parsing stops here; the rungs below come from git and imports* ⎯⎯ |+| **Lightweight** (7) | Elixir · Clojure · Haskell · Lean 4 · Erlang · F# · HTML | A real file-to-file import graph, and no symbol-level claims |+| **Structural** (9) | Objective-C · R · Zig · Julia · Elm · OCaml · Crystal · Nim · D | Git history: blame, hotspots, co-change, ownership, bug history |++**Every language ships in the open-source distribution.** None is gated behind+the commercial licence, and none will be. Languages on the way up the ladder,+including **COBOL**, are on the+**[roadmap →](ROADMAP.md#languages)**.+SQL and dbt projects get real `ref()` / `source()` lineage, shell scripts getfunction-level symbols, HTML pages contribute their `<script src>` / `<link href>`dependencies (including `index.html` → `src/main.ts`), and OpenAPI, Protobuf,@@ -763,26 +820,57 @@ one index, self-hostable and open source. Full side-by-side comparisons:| **Team leads** | Know which PRs to worry about before you merge: change-risk scoring plus the free [Repowise PR Bot](https://github.com/apps/repowise-bot). [For team leads →](https://www.repowise.dev/for/teams) || **Engineering leaders** | See how much of your code AI wrote and whether it is healthy: agent provenance, health trends and bus factor, straight from git history. [For engineering leaders →](https://www.repowise.dev/for/engineering-leaders) || **Security & compliance** | Reachability-aware CVE triage, secret detection across full git history, and SBOM, on your real dependency graph. [For security →](https://www.repowise.dev/for/security) · [security review →](docs/business/SECURITY_COMPLIANCE.md) |-| **Enterprises** | On-prem and air-gapped, SSO/SCIM, commercial licensing with no AGPL obligation, IP indemnification. [For enterprise →](https://www.repowise.dev/for/enterprise) · [docs/business/COMMERCIAL.md](docs/business/COMMERCIAL.md) |+| **Enterprises** | On-prem and air-gapped, SSO/SCIM, commercial licensing with no AGPL obligation, IP indemnification. [For enterprise →](https://www.repowise.dev/for/enterprise) · [what we're building next →](ROADMAP.md) · [docs/business/COMMERCIAL.md](docs/business/COMMERCIAL.md) |----## For teams & enterprises+<a id="for-teams-and-enterprises"></a>++## For teams and enterprises++The reason this passes a security review is structural rather than contractual:+**every analysis layer computes with zero LLM calls.** The graph, git history,+code health, change risk, dead code and the PR bot make no model calls at all, so+in the deterministic path there is no provider to trust, no retention policy to+read, and nothing to prompt-inject. Documentation prose is the one layer that+uses a model, it is opt-in, and it runs on your key or fully offline through+Ollama.++| | |+|---|---|+| **Deployment** | Self-hosted from `pip install`, or containers on your own infrastructure: API server, indexer workers and dashboard, backed by Postgres and LanceDB or pgvector. Reference topology is Kubernetes, and the same containers run on Nomad or plain Docker. Air-gapped mode requires no outbound connectivity. *(Helm chart and air-gap bundle: [planned](ROADMAP.md#enterprise-operations))* |+| **Data boundary** | Source never leaves your network. BYOK against your own Anthropic, OpenAI or Azure OpenAI contract, or fully offline. The provider choice is **per repository**, so sensitive repos run offline while others use a hosted model. Stored: the graph, non-reversible embeddings, wiki pages, git metadata. Raw source is processed transiently and never persisted. |+| **Compliance and audit** | CycloneDX 1.6 SBOM with VEX export and per-snapshot diffs · PCI-DSS 4.0 and SOC 2 control-coverage reports with per-control evidence · insert-only audit trail with JSON/CSV export and a signed-webhook SIEM stream · CVE triage ranked by KEV, EPSS and whether your code actually imports the package. |+| **Contract** | Commercial licence removes the AGPL obligation for embedding repowise in your own platform · IP indemnification · defensive patent grant · named support contact with a response-time SLA and a quarterly architecture review · per-seat, per-repo or enterprise-wide pricing. |++**Past one repository.** Workspaces index an estate as one unit: API contracts+matched producer to consumer so a breaking change is caught before it ships,+cross-repo co-change, and one federated MCP endpoint that answers across all of+it. *(Estate-scale dashboards: [in development](ROADMAP.md#multi-repo-and-workspace).)*++**Not on git?** Only the history layer needs a commit log. Point `repowise init`+at a plain directory, an export, or a Perforce or SVN workspace and the graph,+documentation, decisions and code-health layers all build normally; what is+missing is hotspots, ownership, co-change and bug history until the history layer+learns to read your system.+*([Perforce, SVN, Endevor and ChangeMan on the roadmap →](ROADMAP.md#source-control-beyond-git))*++**Everything above is labelled GA, in development, or planned, line by line, in+[COMMERCIAL.md](docs/business/COMMERCIAL.md).** Some of it is shipping today and+some of it is not, and a comparison table that blurred the two would not survive+the first procurement call. Sequencing against your timeline is part of the+commercial conversation, so the items that matter most to you can be prioritized.[**repowise.dev**](https://www.repowise.dev) is the same engine, fully managed, at-feature parity with self-hosted: every CLI command, every MCP tool, the whole-dashboard. We run it on our own codebase in the open:+feature parity with self-hosted. We run it on our own codebase in the open:[live snapshot →](https://www.repowise.dev/s/5a6b93fa9a69) ·[explore public repos →](https://www.repowise.dev/explore).-On top of self-hosting: managed deploys and webhooks with auto re-index on every-commit, a hosted MCP endpoint so any client can point at one URL with no local server,-a CVE-aware security layer, cross-repo intelligence at scale, and integrations (Slack,-Jira/Linear, Confluence/Notion, PagerDuty) *(rolling out)*.--What is GA versus in development, on-prem topology, SSO/SCIM/RBAC and pricing:-**[docs/business/COMMERCIAL.md](docs/business/COMMERCIAL.md)** ·-[Get in touch →](https://www.repowise.dev/#contact)+**[Commercial detail and pricing models →](docs/business/COMMERCIAL.md)** ·+**[Security review pack →](docs/business/SECURITY_COMPLIANCE.md)** ·+**[Roadmap →](ROADMAP.md)** ·+[[email protected]](mailto:[email protected]) ·+[[email protected]](mailto:[email protected])---diff --git a/ROADMAP.md b/ROADMAP.mdnew file mode 100644index 000000000..c1287e635--- /dev/null+++ b/ROADMAP.md@@ -0,0 +1,195 @@+# Roadmap++What we are building next, and what we have decided not to build.++**There are no dates on this page.** We shipped twelve minor releases in the+thirty-one days to 2026-08-18, roughly one every two and a half days. Any+quarter printed here would be wrong in the slow direction, and a roadmap that is+wrong in the slow direction is worse than one carrying no dates at all. What we+commit to is the status column. The [changelog](docs/CHANGELOG.md) is the+evidence, and it is the fastest way to check whether this page is honest.++Items disappear from this page when they ship. If you are looking for something+that used to be here, it is in the changelog.++| Status | What it means |+|---|---|+| **In development** | Being worked on now. Code exists, it is not finished. |+| **Planned** | Committed to, not started. |+| **Exploring** | We think it is a good idea and have not committed to it. Tell us if you need it, that is what moves a row up. |++The same vocabulary describes the commercial surface in+[docs/business/COMMERCIAL.md](docs/business/COMMERCIAL.md), where each capability+also carries a **GA** marker when it is available today.++---++## Languages++**Every language repowise supports ships in the open-source distribution under+AGPL-3.0.** No language sits behind the commercial licence, and none will. If+your stack is on this list, the support arrives in `pip install repowise`.++Languages land on a five-rung ladder rather than a yes/no list, because "do you+support X" has five different useful answers. Each rung is defined, with what it+buys you, in+[docs/layers/LANGUAGE_SUPPORT.md](docs/layers/LANGUAGE_SUPPORT.md). Today that+ladder covers **35 languages, 19 of them parsed to a full AST**.++### New languages++| Language | Status | Why it is on the list |+|---|---|---|+| **COBOL** | **In development** | Asked for by enterprises with mainframe estates evaluating repowise. The codebases that most need an institutional-memory layer are often the ones whose authors have already retired. |+| **VB.NET** | Planned | Large, long-lived line-of-business estates, usually sitting beside the C# we already treat at Full tier. |+| **PL/SQL** | Planned | We already parse SQL through sqlglot. Packages, procedures and triggers are where the business logic actually lives. |+| **ABAP** | Exploring | The same argument as COBOL, in SAP estates. |+| **RPG / AS400** | Exploring | Named alongside COBOL often enough to track. |+| **Fortran** | Exploring | Scientific and engineering codebases with long lifespans and thin documentation. |+| **Apex** | Planned | Salesforce estates, where the org is often the least documented system a company runs. Apex is Java-shaped, and Lightning Web Components are already covered as JavaScript and HTML. |+| **Ada** | Exploring | Defense and aerospace, long-lived and thinly documented, and usually in the same estates asking about COBOL. |+| **Objective-C** | Planned | Currently Structural, meaning history only. The grammar is mature, so this is a climb up the ladder rather than new ground. |+| **Elixir**, **F#** | Planned | Currently Lightweight, meaning file-to-file imports. Both grammars are already on PyPI, so these are AST upgrades rather than new ground. |+| **Template dialects** (Django/Jinja, ERB, Blade, Thymeleaf, Go templates) | Planned | Today they parse cleanly as HTML and yield nothing, because `{% extends "base.html" %}` is plain text to an HTML parser. A stated ceiling, not an oversight. |++### Deepening languages we already parse++| Language | Status | Next |+|---|---|---|+| Vue, Svelte | In development | Options-API members, `{#each}` and `v-for` head bindings, `.svelte.ts` rune modules |+| C#, Scala, Ruby | Planned | Dataflow dialects, which is what Extract Method needs in order to lift a span safely |+| Kotlin | Exploring | Dataflow is blocked on the grammar and needs either a grammar upgrade or a text-based jump seam. We would rather say blocked than planned. |+| C | Planned | It shares the C++ grammar for parsing but reaches no health walker map, so it gets graph coverage without markers |+| SQL / dbt | Planned | Column-level blast radius |+| Object Pascal | Planned | Assertion and performance markers, a dedicated `uses` resolver |++### Need a language that is not here++Adding one touches per-language subpackages rather than the parser core, which+is what makes this tractable at the rate above. Open an issue and we will scope+it in public. Commercial customers can have a language or framework built and+maintained by us as a line item, which is+[GA today](docs/business/COMMERCIAL.md#54-enterprise-operations), and the result+still ships to everyone under AGPL.++---++## Source control beyond git++Plenty of the code most worth indexing has never been near a git remote. It sits+in Perforce, in Subversion, or on a mainframe under a change-management system+older than most of the people maintaining it.++**A repository with no git at all already indexes today.** Only one of the five+layers is derived from history; the graph, documentation, decisions and+code-health layers are all computed from the working tree. Point `repowise init`+at a plain directory, an export, or a Perforce or SVN workspace and those four+build normally. What is missing is the history-derived half: hotspots, ownership,+co-change, bus factor, bug history, and change risk, all of which need a commit+log to mine.++Closing that gap means teaching the history layer to read something other than+`git log`. The signals above it do not care where the history came from once it+is normalized into changes, authors, timestamps and touched files, which is the+shape every one of these systems has.++| System | Status | Shape of the work |+|---|---|---|+| **Perforce Helix Core** | Planned | Changelists map to commits almost directly, and `p4 filelog` and `p4 annotate` cover history and blame. Common in games, hardware, embedded and anywhere the repo carries large binaries. |+| **Subversion** | Planned | Revisions map cleanly; `svn log` and `svn blame` are close analogues. Still the system of record in a lot of long-lived estates. |+| **TFVC (Azure DevOps)** | Exploring | Changesets, for shops on Azure DevOps that never moved to its git repos. |+| **Mercurial** | Exploring | Changesets map cleanly, and the work is small. Ordered by demand rather than difficulty. |+| **CA Endevor SCM** (Broadcom) | Exploring | Not a commit DAG. Endevor tracks elements moving through a stage and environment promotion hierarchy, so the practical shape is an export to a filesystem plus a history adapter that reads element history rather than a log. Pairs with the COBOL estates above. |+| **ChangeMan ZMF** (OpenText) | Exploring | The same argument, with packages instead of elements. |++**Ask, do not guess, if your system is not listed.** The seam is the same for all+of them, and what decides the order is which ones customers actually run.++---++## Multi-repo and workspace++Real systems are not one repository, and the interesting failures live in the+gaps between them.++| Item | Status | What it adds |+|---|---|---|+| Cross-repo intelligence at scale | In development | Hotspots, dead code and ownership across a whole estate with centralized dashboards, beyond the local-workspace scope already shipping in open source |+| Column-level blast radius for SQL / dbt | Planned | Lineage is table-level today through `ref()` and `source()`. Column-level answers "who breaks if I drop this field" |+| Custom decision policies | Planned | Required-reviewer rules, mandatory `get_why` checks on governed paths, and merge gating tied to findings |++---++## Integrations++These sit on the hosted platform and the commercial licence. Current+availability, including everything that is GA today, is in+[COMMERCIAL.md §5.2](docs/business/COMMERCIAL.md#52-workflow-integrations-rolling-out).++| Item | Status |+|---|---|+| GitHub Enterprise, Azure DevOps, GitLab self-managed, Bitbucket | In development |+| SAML / OIDC SSO (Okta, Entra ID, Auth0, Google Workspace) and SCIM provisioning | In development |+| Slack and Microsoft Teams beyond security alerting: hotspot drift, bus-factor warnings, decision staleness, routed by ownership | In development |+| Jira risk-finding links and PR-impact auto-comments on linked issues | Planned |+| Native SIEM connectors (Splunk, Datadog, Elastic) beyond the signed-webhook stream | Planned |+| PagerDuty | Planned |+| SPDX SBOM output and cross-format conversion | Planned |+| ISO 27001 Annex A and GDPR / data-residency control mappings | Planned |++On that last row: we would rather ship two solid compliance mappings than four+shallow ones, which is why PCI-DSS 4.0 and SOC 2 are GA and these are not.++---++## Enterprise operations++| Item | Status |+|---|---|+| Role-based access control at repo, module and decision level | Planned |+| Air-gapped install bundle with bundled grammars, embedding model and optional Ollama runtime | Planned |+| Helm chart for the reference topology | Planned |+| Multi-tenant deployment | Planned |+| Backup and restore, point-in-time snapshots of the intelligence layers | Planned |+| Audit-trail coverage beyond the security surface, covering decisions and overrides | In development |+| Engineering leader dashboard: bus-factor trends, ownership drift, decision-staleness curves, scheduled digests | In development |++---++## Core intelligence++The half of the roadmap that has nothing to do with procurement.++| Item | Status | Note |+|---|---|---|+| Interface-dispatch recall | Exploring | The measured ceiling on our call-graph recall. On `syft`, 44% of what we miss is dynamic dispatch alone and a further 39% is dispatch with a closure at one end. Nobody in the [measured field](docs/BENCHMARKS.md#8-the-same-question-against-an-answer-key-we-do-not-control) has cleared it, at 6.5 possible targets per call site, and matching that recall naively means emitting six edges where one is right. |+| A sealed JavaScript / TypeScript retrieval corpus | In development | Built and half graded. The sealed half is unrun, and nothing from it is quoted anywhere until it is evaluated, once. |+| More compiler oracles for graph precision | Exploring | Go and TypeScript are done. C#, Java, Kotlin and C++ each need a toolchain and a working build per repository; Rust has the toolchain and no sound call-graph tool exists for it; Python, Ruby and PHP admit no oracle even in principle. The honest status is "probably not", not "planned". |+| Session intelligence harvesting | Planned | Architectural decisions surfaced from AI coding sessions and proposed to the team knowledge base, so knowledge generated during agent work does not evaporate when the session ends |++---++## What we are not building++- **An LLM-based PR reviewer.** The [PR bot](README.md#the-pr-bot) makes zero+ model calls per review, and that is the product rather than a limitation:+ nothing to hallucinate, nothing to prompt-inject, and pushing the same diff+ twice produces the same review twice. CodeRabbit and Greptile do the other+ thing, and do it well. If you want intent-reading review, run one of those+ alongside this.+- **Runtime or APM data.** Everything here is derived from source and git+ history. A performance finding we report is a static one, and we label it as+ static.+- **Any language behind a paywall.** Stated above, repeated here because it is+ the question we get asked most.+- **Dates.** See the top of this page.++---++## Ask for something++- Open an [issue](https://github.com/repowise-dev/repowise/issues) and say what+ you need and why. Demand is what moves an Exploring row to Planned, and we say+ so rather than implying the order is purely technical.+- Commercial and procurement timelines: [[email protected]](mailto:[email protected])+- Security review: [[email protected]](mailto:[email protected])diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.mdindex 8b1332c4a..5d8056002 100644--- a/docs/BENCHMARKS.md+++ b/docs/BENCHMARKS.md@@ -6,32 +6,30 @@ pre-registrations and every graded cell live in**[repowise-bench](https://github.com/repowise-dev/repowise-bench)**, which is alsowhere the depth is: this page is the summary, that repository is the evidence.-**Read this page in one minute:** the two charts below are the two retrieval-results, and [§8](#8-the-same-question-against-an-answer-key-we-do-not-control)-is the graph result, where a compiler rather than we decided who was right.-Everything else is the sample size, the caveat and the row we lose.+Each section leads with the result; method and caveats are one fold down, and+nothing put there changes a headline. Where we have not run a comparison, the row+says **not measured** rather than carrying a checkmark.-| Layer | Measured against | Result |-|---|---|---|-| Finding the right files | CodeGraph, Graphify, code-review-graph, cocoindex | [§1](#1-finding-the-right-files) **we win**, n=42 held out, p=0.00004 |-| Work saved in a real agent loop | the same field plus Serena and a bare agent | [§2](#2-what-changes-in-a-real-agent-loop) **we win** on all three agent harnesses we tried, n=43 at p<0.0001 |-| Loading one commit's context | naive file reads, `git diff` | [§3](#3-loading-one-commits-context-the-easy-number) **35.6x** fewer tokens than naive |-| Command-output compression | RTK | [§4](#4-command-output-compression) **not measured head to head** |-| Code health and defect prediction | CodeScene | [§5](#5-code-health-predicts-defects) **we win** on recall and effort-aware ranking, p=0.003 |-| Indexing time | CodeGraph, Graphify, code-review-graph | [§6](#6-indexing-time-the-row-we-lose) **we lose**, 22x, because we build four more layers in the same pass |-| Are the call edges true | CodeGraph | [§7](#7-edge-precision) **we win**, 84.8% against 57.0%, 540 rows hand-graded from source |-| The same question, judged by a compiler | CodeGraph, codebase-memory-mcp, Graphify, code-review-graph | [§8](#8-the-same-question-against-an-answer-key-we-do-not-control) **no tool that finds as much of the graph gets more of it right**, 7 of 7 cells, and the recall column we lose |-| Documentation generation | DeepWiki, Google Code Wiki, Swimm | **not measured** |-| PR review | CodeRabbit, Greptile | **not measured** |--The last two rows are capability comparisons, not measurements, and they live in-the [README's feature table](../README.md#how-it-compares-on-capability) where a-reader can tell the difference. **§4 carries the same label for the same reason:**-RTK does exactly what `repowise distill` does, and we have not run the two against-each other. We would rather write "not measured" than let a checkmark do a-number's job.+## The scoreboard----+| Question | Measured against | Result |+|---|---|---|+| **Are the call edges true, judged by a compiler** | CodeGraph, codebase-memory-mcp, Graphify, code-review-graph | [**no tool that finds as much of the graph gets more of it right**](#is-the-call-graph-correct), 7 of 7 cells, plus the recall column we lose |+| **Are the call edges true, hand-graded from source** | CodeGraph | [**we win**](#7-edge-precision), 84.8% against 57.0%, 540 rows read on both sides |+| **Finding the right files** | CodeGraph, Graphify, code-review-graph, cocoindex | [**we win**](#1-finding-the-right-files), n=42 held out, p=0.00004 |+| **Work saved in a real agent loop** | the same field plus Serena and a bare agent | [**we win**](#2-what-changes-in-a-real-agent-loop) on all three agent harnesses we tried, n=43 at p<0.0001 |+| **Does the health score predict real bugs** | CodeScene | [**we win**](#5-code-health-predicts-defects) on recall and effort-aware ranking, p=0.003, and lose the business-impact axis outright |+| **Loading one commit's context** | naive file reads, `git diff` | [**35.6x**](#loading-one-commits-context-the-easy-number) fewer tokens than naive |+| **Indexing time** | CodeGraph, Graphify, code-review-graph | [**we lose**](#6-indexing-time-the-row-we-lose), 22x, because we build four more layers in the same pass |+| **Command-output compression** | RTK | [**not measured**](#command-output-compression) head to head |+| **Documentation generation** | DeepWiki, Google Code Wiki, Swimm | **not measured** |+| **PR review** | CodeRabbit, Greptile | **not measured** |++The three **not measured** rows are capability comparisons rather than+measurements, and they live in the+[README's feature table](../README.md#how-it-compares-on-capability) where a reader+can tell the difference. We would rather write "not measured" than let a checkmark+do a number's job.<div align="center"><picture>@@ -46,14 +44,250 @@ number's job.----## 1. Finding the right files+<a id="8-the-same-question-against-an-answer-key-we-do-not-control"></a>+<a id="is-the-call-graph-correct"></a>++## Is the call graph correct++Every other section on this page counts things. This one asks whether they are+**true**, and it is the result we would keep if we could keep only one.++**Every graph-quality number in this field, ours included, is normally scored+against something the publisher controls**, so a tool that is confidently wrong in+a consistent way scores well and the reader has no way to check. We re-asked the+question with the answer key taken out of our hands: on Go the oracle is the **Go+team's own RTA call graph** from `golang.org/x/tools/go/callgraph/rta` over the+fully type-checked program, and on TypeScript it is the **`tsc` type checker's own+resolution** of every call site. We did not write either one, we cannot tune either+one, and anyone with the toolchain can regenerate both. Five tools, seven cells,+five repositories, **37,853 oracle edges**.++### Two numbers, and you need both++Each one alone has a cheap way to win, and both failure modes are in the tables+below rather than hypothetical.++- **Recall** is the share of the real call graph a tool found. Draw an edge between+ everything and you score 1.000. One tool draws 12,533 edges on `syft` and more+ than a third of them are calls the compiler says do not exist.+- **Precision** is the share of a tool's edges that are real. Draw one edge you are+ sure of and you score 1.000. The highest precision on this whole page, 0.997,+ comes from a graph containing **17% of the calls in the repository**; on+ `gitleaks` that tool stored 4,367 call rows of which 76 resolve to anything, and+ a graph that small is very hard to be wrong in and not much use to walk.++**So the only meaningful reading is the pair, and it is the pair we claim:**++> **In all seven cells, no tool that recovers as much of the call graph as we do+> gets more of it right.**++That sentence names no threshold, which is what makes it worth something. It cannot+be tuned by picking a cutoff, and adding a competitor can only break it. Two+competitors were added to this experiment after it was first written, and it held+in all seven cells.++**Precision: of the call edges a tool emits, the share the compiler confirms.**++| cell | repowise | CodeGraph 1.5.0 | codebase-memory-mcp 0.10.8 | Graphify 0.9.31 | code-review-graph 2.3.7 |+|---|---|---|---|---|---|+| cobra (with tests) | 0.972 | 0.929 | 0.912 | 0.971 | **0.997** |+| gitleaks (no tests) | 0.976 | 0.972 | 0.934 | **0.997** | 0.759 |+| gitleaks (with tests) | 0.974 | 0.971 | 0.922 | **0.995** | 0.800 |+| syft (no tests) | 0.943 | 0.872 | 0.635 | 0.771 | **0.968** |+| syft (with tests) | 0.950 | 0.864 | 0.673 | 0.802 | **0.966** |+| zod (no tests) | **0.992** | 0.729 | 0.987 | 0.825 | 0.932 |+| hono (no tests) | 0.977 | 0.805 | 0.949 | 0.980 | 0.966 |++**Recall: of the edges the compiler has, the share the tool found.**++| cell | repowise | CodeGraph | codebase-memory-mcp | Graphify | code-review-graph |+|---|---|---|---|---|---|+| cobra (with tests) | 0.684 | **0.763** | 0.743 | 0.433 | 0.174 |+| gitleaks (no tests) | 0.955 | 0.920 | **0.967** | 0.886 | 0.026 |+| gitleaks (with tests) | 0.914 | 0.895 | **0.945** | 0.832 | 0.032 |+| syft (no tests) | 0.513 | 0.508 | **0.542** | 0.447 | 0.201 |+| syft (with tests) | 0.322 | 0.338 | **0.361** | 0.273 | 0.086 |+| zod (no tests) | **0.703** | 0.373 | 0.694 | 0.248 | 0.652 |+| hono (no tests) | **0.731** | 0.684 | 0.686 | 0.688 | 0.691 |++Read a row across both tables and the trade is visible in every one. The tools+above us on precision are below us on recall, every time. The tool above us on+recall is below us on precision, every time. **Nobody is above us on both, in any+cell.**++**The weaker readings, so nobody has to infer them.** We are the most precise arm+outright in **one cell of seven** and tied for it in one more. We are beaten on+precision in **five**: by code-review-graph on cobra and both syft cells, and by+Graphify on both gitleaks cells. Against the two tools this experiment started+with, CodeGraph and codebase-memory-mcp, we are the most precise in seven of seven,+and that narrower claim should always carry its label.++**The column we lose.** We lead recall in the two TypeScript cells and in **none of+the five Go cells**, where codebase-memory-mcp leads four and CodeGraph the fifth.+On cross-file coverage over 35 repositories, codebase-memory-mcp separates from us+on 15 and we separate on none. Those are real losses and they are not softened+here. What the oracle adds is the price of that lead rather than an excuse for+ours: on syft, more than a third of what that tool emits is a call the Go compiler+says does not exist.++<details>+<summary><b>Method, limits, and what this does not show</b></summary>++**The two methods agree.** On Go, the hand-graded audit below read 29/30 = 96.7%+for us and 29/30 = 96.7% for CodeGraph. The Go compiler, over roughly 1,600 edges+on the same repository, says **97.6% and 97.2%**. Two unrelated methods, one person+reading source and one type checker, land within about a point on both arms. That+is the result we care about most, and it is not a competitive one. It also removes+the n=30 ceiling: hand-grading caps a precision estimate at roughly ±13 points near+a 95% rate, while an oracle grades every edge, so n becomes the size of the+repository.++**Where our own miss goes**, decomposed rather than waved at. On syft without tests+we miss 3,846 of the oracle's 7,898 edges: **44% of that miss is dynamic dispatch+alone** and a further **39% is dispatch with a closure at one end**, and the two+buckets overlap so neither is the whole gap. Interface dispatch is the ceiling and+nobody in the comparison has cleared it, at 6.5 distinct possible targets per call+site. Matching that recall means emitting six edges where one is right, which is+the behaviour the precision table charges other tools for.++**Do not compare recall across rows.** It swings from 0.03 to 0.97, driven by how+many entry points the oracle had (4 on gitleaks, 268 on syft-with-tests), not by+tool quality. Only within-row comparisons carry meaning, and a pooled recall over+these cells would be meaningless.++Further limits:++- **Two languages, seven cells, five repositories.** This is not a nine-language+ claim and must not be quoted as one. The hand-graded audit below is the+ nine-language number.+- **A contradicted edge is very strong evidence, not proof.** RTA is unsound under+ reflection and `go:linkname`, so a genuinely dynamic edge can land in that+ bucket. It applies to all five arms equally and the gaps are far too large to be+ explained by it, which is why the metric is named *precision against the oracle*+ rather than precision.+- **Edges the oracle cannot speak about are charged to nobody** and reported at+ full size. That bucket is 0.4% to 11.1% of a tool's output on the Go cells and+ 16% to 46% on the TypeScript ones, where dependencies are not installed in the+ pinned corpus.+- **A library has no `main`, so RTA has no roots**; cobra is analysed through its+ test binaries only.+- The two variants of a repository answer different questions. Report both or say+ which one you used.+- **Two oracle languages, and the programme stops at two.** C#, Java, Kotlin and+ C++ each need a toolchain installed and a working build per repository, none of+ which exists on the measurement machine. Rust has the toolchain and no sound+ call-graph tool exists for it. Python, Ruby and PHP admit no oracle even in+ principle. So the hand-graded audit is the permanent method on those languages+ rather than a stopgap, and nothing here should be read as a claim about them.+- **The TypeScript cells exclude test files, and the with-tests variants are+ void.** A test file imports its own package by name, the pinned corpus has no+ dependencies installed, and roughly a third of call sites go unresolvable, which+ takes the unjudged bucket past three quarters. Both variants were run and neither+ is quoted anywhere.+- **The three newer competitors are priced on these two languages only.**+ codebase-memory-mcp, Graphify and code-review-graph have no precision figure of+ any kind on the other nine languages, and none was ever entered into the+ hand-graded audit in either direction. That audit is a two-tool claim and this one+ a five-tool claim, and neither transfers to the other's languages.+- **Two of the five arms are read through an adapter we wrote**, and both now beat+ us on precision in some cell, so the reading matters. Graphify tags 93% of its+ call edges `INFERRED` rather than AST-certain and we score all of them, which is+ the choice least favourable to us. code-review-graph stores unresolved callees+ beside resolved ones and we score only the resolved, which is the choice most+ favourable to it: on gitleaks that is 76 edges out of 4,367 stored rows. Both+ choices are argued in the benchmark and a reader who disagrees can recompute+ either from the artifacts.++Full method, per-cell artifacts, the twenty hand-confirmed identities the protocol+required, and the graded pre-registration including the two predictions that+missed:+**[graph/experiments/g4-oracle-anchored](https://github.com/repowise-dev/repowise-bench/tree/master/graph/experiments/g4-oracle-anchored)**.++</details>++---++<a id="7-edge-precision"></a>++## The same question, hand-graded across nine languages++The oracle covers two languages. This one covers nine, on both sides, by the same+method: 30 rows per language per tool, seed 2026, stratified by resolution+strategy, every row read from source with its imports and enclosing scope open.++| | correct / n | 95% CI |+|---|---|---|+| **repowise** | **229/270 = 84.8%** | [80.0, 88.6] |+| **CodeGraph 1.5.0** | **154/270 = 57.0%** | [51.1, 62.8] |++| Language | repowise | CodeGraph | |+|---|---|---|---|+| typescript | 29/30 | 7/30 | separates |+| go | 29/30 | 29/30 | tie |+| csharp | 28/30 | 20/30 | separates |+| python | 28/30 | 19/30 | separates |+| kotlin | 27/30 | 13/30 | separates |+| swift | 23/30 | 19/30 | tie |+| cpp | 23/30 | 16/30 | tie |+| rust | 22/30 | 13/30 | tie |+| java | 20/30 | 18/30 | tie |++**Read our number the other way round: roughly fifteen percent of our call edges+are wrong.** That is the number we plan against, and rust, java and cpp are where+it concentrates.++**Four of nine cells separate. Five are ties and we report them as ties.** At n=30+the interval runs about ±16 points near 60%, so a point-estimate gap inside two+overlapping intervals is not a win. C++ is a tie despite looking like a 23-point+lead.++<details>+<summary><b>Method, limits, and what this does not show</b></summary>++**One repository goes clearly to them.** On `seastar` CodeGraph reads 6/10 against+our 4/10, the only repository in the audit, on any language, where they beat us on+a clear margin. Our misses there are chained calls on an untyped receiver; they+infer the callee's declared return type and validate against it, so a failed+inference costs them an edge instead of buying them a wrong one. On `aria2` both+sides read 10/10 and they resolve 24,950 distinct call edges to our 9,486.+Precision is not the only reading.++**This audit was graded by us, on both sides.** That is the strongest form of a+weak thing, and it is exactly what the oracle section above exists to check. The+two agree where both exist, to within about a point, but only the oracle has an+answer key we did not produce.++Cells were measured at different commits, and the staleness runs **conservative**:+every resolver change in between only removes wrong edges and gained zero, so 84.8%+is a floor. No cell was measured on the 0.44.0 release; the full table pins a+commit per cell.++**All 540 graded rows are published**, one file per cell, each row carrying the call+site, the declaration the tool bound it to, the verdict and the reason it was+given. A script in the same directory rebuilds every table above from them and+fails if it disagrees. One cell of the eighteen, rust on our side, ships the 30+sites that were read without their per-row verdicts, because that cell was re-read+on a fresh draw and the verdicts of that read were never written down; the file+says so on every row.++**Precision is one of two halves.** We lead no recall cell against the oracle, and+lose cross-file coverage on 15 of 35 repositories in the same bench. A precision+win is a claim about the edges a tool draws, never about how many.++**[graph/experiments/g1-edge-precision/rows](https://github.com/repowise-dev/repowise-bench/tree/master/graph/experiments/g1-edge-precision/rows)**++</details>++---++<a id="1-finding-the-right-files"></a>-Before a tool can save an agent any work, it has to point at the right code. This-section measures only that.+## Finding the right files-**Grading is deterministic. No LLM judge is involved anywhere in this number**,-which makes it the most reproducible result on the page. ContextBench ships gold-file spans; a tool either returns them or it does not.+Before a tool can save an agent any work, it has to point at the right code.+**Grading is deterministic and no LLM judge is involved anywhere in this number**,+which makes it the most reproducible result on the page: ContextBench ships gold+file spans, and a tool either returns them or it does not.**Every number below comes from instances this work has never seen.** The 112instances were split 70 / 42 by instance id, **pinned before any of it started**,@@ -75,29 +309,20 @@ p = 0.021**.**These are two different tools with two different profiles, and we would rathersay so than average them into one claim.** `get_answer` finds the most, from a listof ~19 files. `search_codebase` finds fewer but is the most efficient per file-served: **0.742 from 8.2 files**, better coverage-per-file than anything else in+served, **0.742 from 8.2 files**, better coverage-per-file than anything else inthe table. If you are paying by the token, that is the row to read.**Precision is not our column.** code-review-graph's 0.240 is nearly three timesours. Part of that is mechanical, since precision rises for whoever returns fewestfiles, which is exactly why files-served is a column here and not a footnote.-**The cocoindex row is not from the same sitting**, and it is printed rather than-smoothed away: measured 2026-08-09 against the other five from 2026-08-02 to-2026-08-06, same instances, same gold spans, same deterministic grading. Its n is-41 because one named instance never answered even when queried alone, so it is-excluded rather than counted as a zero, which makes its row *better* than counting-it would (0.361 against 0.353). Its placing was-[pre-registered before its index existed](https://github.com/repowise-dev/repowise-bench/tree/master/configs),-including the prediction that it would come last.--### Why this is not benchmark tuning--We first ran this and came **last, at 0.228**. We published that. The cause was a-query-time gate discarding most candidates before ranking ever happened. Fixing-that path is what moved the number, and it is a fix any user gets, not a change-shaped around these questions.+<details>+<summary><b>Method, limits, and what this does not show</b></summary>+**Why this is not benchmark tuning.** We first ran this and came **last, at+0.228**. We published that. The cause was a query-time gate discarding most+candidates before ranking ever happened. Fixing that path is what moved the number,+and it is a fix any user gets rather than a change shaped around these questions.The check on that claim is the split, and it points the right way:| | other half (n=70) | **sealed half (n=42)** |@@ -106,44 +331,46 @@ The check on that claim is the split, and it points the right way:| repowise (`search_codebase`) | 0.684 | 0.742 || CodeGraph | **0.6093** | **0.6095** |-**Overfitting makes the unseen half score worse. Ours scores better**, on both-tools. CodeGraph, which nobody tuned against either half, scores the same on both-to three decimal places, so the two halves are equally hard and the gap is about-the tool rather than the questions.+Overfitting makes the unseen half score worse. Ours scores better, on both tools.+CodeGraph, which nobody tuned against either half, scores the same on both to three+decimal places, so the two halves are equally hard and the gap is about the tool+rather than the questions. **We do not quote a pooled 112-instance figure**, though+it is easy to compute and would be 0.835. Averaging the halves loses the only+number that matters.-**We do not quote a pooled 112-instance figure**, though it is easy to compute and-would be 0.835. Averaging the halves loses the only number that matters.+**The cocoindex row is not from the same sitting**, and it is printed rather than+smoothed away: measured 2026-08-09 against the other five from 2026-08-02 to+2026-08-06, same instances, same gold spans, same deterministic grading. Its n is+41 because one named instance never answered even when queried alone, so it is+excluded rather than counted as a zero, which makes its row *better* than counting+it would (0.361 against 0.353). Its placing was+[pre-registered before its index existed](https://github.com/repowise-dev/repowise-bench/tree/master/configs),+including the prediction that it would come last.-This is retrieval, not task success. It says we find the right files, not that an-agent using us writes better code. That is what §2 is for. Producing it cost-**748 index builds and roughly 78 machine-hours**, because every arm builds its own-index of every instance at that instance's own `base_commit`, with nothing shared-and nothing cached.+**This is retrieval, not task success.** It says we find the right files, not that+an agent using us writes better code.Measured on repowise `081a59fa` (between v0.37.0 and v0.38.0). Raw cells:**[rung8](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08/rung8)**.------## 2. What changes in a real agent loop+</details>-The section a skeptic should read first, and the one modelled on the JetBrains-reruns described [at the bottom of this page](#how-to-read-a-number-on-this-page).--Every question in `django/django`'s question set, 48 of them, across five question-shapes. Six arms: repowise, four competing tools, and a bare agent with no tools.-Byte-identical prompt, full advertised tool surface per arm, a freshly built index-on the same pinned commit, and a bare control verified free of local hooks.+----Two things have to be true for a tool to be worth mounting: the agent has to-actually call it, and the loop has to get leaner when it does.+<a id="2-what-changes-in-a-real-agent-loop"></a>-**We ran this on three agent harnesses, because the answer depends on the harness-as much as on the tools.**+## What changes in a real agent loop-### The main run: 48 questions on Codex (`gpt-5.6-sol`)+The section a skeptic should read first. Every question in `django/django`'s+question set, 48 of them, across five question shapes. Six arms: repowise, four+competing tools, and a bare agent with no tools. Byte-identical prompt, full+advertised tool surface per arm, a freshly built index on the same pinned commit,+and a bare control verified free of local hooks. Two things have to be true for a+tool to be worth mounting: the agent has to actually call it, and the loop has to+get leaner when it does.-Every tool called on every question, so this is like-for-like.+**48 questions on Codex (`gpt-5.6-sol`).** Every tool called on every question, so+this is like for like.| Tool | Agent used it | Output tokens | vs bare agent | Tool calls | Leaner on | p ||---|---:|---:|---:|---:|---:|---:|@@ -159,11 +386,20 @@ against the bare agent's 7.2**, opening 3.0 files instead of 7.2. One answeredquestion replacing roughly six greps, visible in the call counts rather thaninferred.-Correcting for testing five tools at once, three reductions are solid and two are-marginal. **CodeGraph is a genuine second at -24.4%**, and the honest reading is-that we lead a field in which more than one tool works. Serena is the counter-case:-it writes less than the bare agent while calling tools **42% more often**. Busier,-not leaner.+**CodeGraph is a genuine second at -24.4%**, and the honest reading is that we lead+a field in which more than one tool works. Correcting for testing five tools at+once, three reductions are solid and two are marginal.++**We ran this on three agent harnesses**, because the answer depends on the harness+as much as on the tools. On Claude Code (15 questions, `claude-sonnet-5`) we+measured **-15.9%** and were the only arm the agent reliably called. On a local+`qwen3:8b` under Ollama, where inference is free and the win shows up as time+instead, the local-only tool set ran **-47.9% output tokens in -41.5% wall clock**.+Both tables are in the fold below with their controls, including the one whose+token column fails its own control and is therefore quoted for nobody.++<details>+<summary><b>Method, limits, and what this does not show</b></summary>5 of the 48 questions are missing from every arm equally because the run hit an APIusage cap. Paired comparisons are unaffected; the figures are over the 43 all six@@ -175,7 +411,7 @@ Pre-computed structure replaces exploration, and harder questions contain moreexploration to replace. That split is post-hoc and the pre-registered comparisonsare the stronger evidence.-### Second harness: 15 questions on Claude Code (`claude-sonnet-5`)+**Second harness: 15 questions on Claude Code (`claude-sonnet-5`).**| Tool | Tools advertised | Schema cost (chars) | Agent used it | Output tokens | vs bare | Leaner on | p ||---|---:|---:|---:|---:|---:|---:|---:|@@ -186,31 +422,21 @@ are the stronger evidence.| Graphify | 10 | 5,482 | 3 / 15 | 2,878 | 0.0% | 7 of 15 | 1.000 || *bare agent (control)* | 0 | 0 | n/a | 2,877 | baseline | n/a | n/a |-The "agent used it" column is the interesting one. Under Claude Code most of these-tools were barely called at all: code-review-graph never once, Graphify three times-in fifteen. Nothing differed about the servers, questions or indexes between the-two harnesses. Claude Code loads MCP tool schemas on demand, so the agent has to go-looking before it can call anything, and frequently never does.--**Treat that column as unstable, including our own 15 of 15.** Reruns on later days-returned 4 of 15 and then 3 of 15 for us, and 2 of 14 for CodeGraph.--**Is the collapse the model or the harness? Inconclusive.** The same 15 questions on-Opus, against bands fixed before the run (12+ means the model, 6 or fewer means-schema deferral, 7 to 11 inconclusive) **came back at 7**. Opus goes looking on 11-of 15 and declines about a third of the times it looks, so deferral is part of the-story and not all of it. **That run's token column fails its own control** (-9.3%-when the tool was called against -10.7% when it was not), so no token figure is-quoted from it for any tool including ours.--### Third harness: a small model on your own hardware--`qwen3:8b` under Ollama, driven by opencode, same 15 django questions, same seed.-Inference cost is zero, so the question becomes "is it faster, and is it better".--**Every cell called its tool: 15 of 15 on both rows**, against 7 of 15 for Opus on-this identical draw. Adoption ordering across four instruments is Sonnet 3 to 4,-Opus 7, Codex 15, local `qwen3:8b` 15.+The "agent used it" column is the interesting one: under Claude Code most of these+tools were barely called at all, code-review-graph never once and Graphify three+times in fifteen. Nothing differed about the servers, questions or indexes between+the two harnesses. Claude Code loads MCP tool schemas on demand, so the agent has+to go looking before it can call anything, and frequently never does. **Treat that+column as unstable, including our own 15 of 15.** Reruns on later days returned 4+of 15 and then 3 of 15 for us, and 2 of 14 for CodeGraph. Whether the collapse is+the model or the harness is **inconclusive**: the same 15 questions on Opus,+against bands fixed before the run (12+ means the model, 6 or fewer means schema+deferral, 7 to 11 inconclusive), came back at 7. That run's token column fails its+own control (-9.3% when the tool was called against -10.7% when it was not), so no+token figure is quoted from it for any tool including ours.++**Third harness: `qwen3:8b` under Ollama via opencode**, same 15 questions, same+seed. Every cell called its tool, 15 of 15 on both rows.| Row | Agent used it | Output tokens | vs bare | Leaner on | p | Wall clock | vs bare ||---|---:|---:|---:|---:|---:|---:|---:|@@ -218,151 +444,76 @@ Opus 7, Codex 15, local `qwen3:8b` 15.| **repowise, local-only tools** | **15 / 15** | **1,172** | **-47.9%** | **15 of 15** | **0.00006** | **96s** | **-41.5%** || *bare agent (control)* | 0 / 15 | 2,336 | baseline | n/a | n/a | 171s | baseline |-**Two repowise rows, never combined.** `get_answer` writes its answer using a hosted-model, so a row using it is not a local-only result. The second row switches it off-and leaves only tools that run against the local index. We verified the restriction-holds rather than assuming it: instructed directly and repeatedly to call-`get_answer`, that agent could not reach it in any of its 15 cells.--**On a local model the win shows up as time, and the mechanism differs from the-hosted case.** repowise roughly doubles the tokens fed in on a single step while-cutting steps from 3.3 to 2.1 and halving what the model writes. Reading a large-payload once is cheap on a GPU; generating text token-by-token across several-rounds is not.--**We are not claiming a quality improvement from this run.** The full surface scored-+1.32 against the bare agent on a 0 to 10 judge scale, above the judge's measured-0.69 noise, but the win-loss count is 10 to 5 at p = 0.30 and removing the single-best question drops it to +0.99. The local-only row is **+0.20, a null**. The-defensible statement is narrower: **the local-only configuration answers about as-well as a bare agent in roughly half the wall clock, on hardware you already own.**--### What this section will not claim--- **This is work saved, not quality.** A blind judge scored every tool in the field,- ours included, a fraction **below** the bare agent on the 48-question run, in a- range of 0.04 to 0.25 points on a 10-point scale. All are smaller than the 0.69- points by which this benchmark moves when rerun unchanged. No tool here measurably- changed answer quality in either direction. Ours is at the low end of that band- and we will say so if it becomes a real effect. "No significant difference" is not- parity, and an equivalence claim needs a test we have not run.-- **The quality columns cannot be compared across tables.** Different judges, because- grading a model with a judge from its own family is a known bias.+**Two repowise rows, never combined.** `get_answer` writes its answer using a+hosted model, so a row using it is not a local-only result. The second row switches+it off and leaves only tools that run against the local index. We verified the+restriction holds rather than assuming it: instructed directly and repeatedly to+call `get_answer`, that agent could not reach it in any of its 15 cells. repowise+roughly doubles the tokens fed in on a single step while cutting steps from 3.3 to+2.1 and halving what the model writes, which is why the win shows up as time:+reading a large payload once is cheap on a GPU, generating text token-by-token+across several rounds is not.++**What this section will not claim:**++- **This is work saved, not quality.** A blind judge scored every tool in the+ field, ours included, a fraction **below** the bare agent on the 48-question run,+ in a range of 0.04 to 0.25 points on a 10-point scale, all smaller than the 0.69+ points by which this benchmark moves when rerun unchanged. No tool here+ measurably changed answer quality in either direction. Ours is at the low end of+ that band and we will say so if it becomes a real effect. "No significant+ difference" is not parity, and an equivalence claim needs a test we have not run.+ On the local run the full surface scored +1.32 against the bare agent, above the+ judge's measured 0.69 noise, but the win-loss count is 10 to 5 at p = 0.30 and+ removing the single best question drops it to +0.99; the local-only row is+ **+0.20, a null**. The defensible statement is narrower: the local-only+ configuration answers about as well as a bare agent in roughly half the wall+ clock, on hardware you already own.+- **The quality columns cannot be compared across tables**, because grading a model+ with a judge from its own family is a known bias, so the judges differ.- **Not a universal saving.** One repository, one commit, one prompt. Threeharnesses, which is more than most published numbers in this category, and stillnot a constant.-- **Adoption is not a stable property of a tool.** Whether an agent calls a codebase- server at all depends more on the harness than on the server. Any adoption figure,- ours included, is only meaningful with its harness and its date attached.+- **Adoption is not a stable property of a tool.** Whether an agent calls a+ codebase server at all depends more on the harness than on the server. Any+ adoption figure, ours included, is only meaningful with its harness and its date+ attached. Ordering across four instruments was Sonnet 3 to 4, Opus 7, Codex 15,+ local `qwen3:8b` 15.- **No dollar figure**, and the control that retired it is the most useful thing weran. code-review-graph, having never called its server once in 15 questions andcarrying 28,118 extra characters of schema that should cost *more*, measured**43% cheaper than the bare agent**. The cause is prompt caching: whichever armruns first warms it. An arm's position in the cycle correlated **-0.487** with itsdollar cost. Output tokens are never cached and correlate **+0.010**, so those are- what we publish. If you see a token claim here that does not say whether it+ what we publish. If you see a token claim anywhere that does not say whether itcontrols for cache state and arm ordering, ask about that first.-**What producing these tables cost: 471 agent runs, about 13 hours of machine time,-roughly $44 of API spend**, plus 11.3 minutes of fresh index builds for every tool.-Roughly a third of those runs are gates and repeats rather than headline numbers.-Full breakdown, and the per-tool setup traps that were most of the real effort, in-[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head).-Raw data, every cell including failures:**[rung9](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08/rung9)**(48-question Codex) and**[rung6](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08/rung6)**-(the 15-question runs).-------## 3. Loading one commit's context, the easy number--This is the measurement almost every tool in this category publishes, and we are-labelling it as such. Over the 30 most recent non-merge commits of `pallets/flask`,-counted with deterministic `tiktoken` (`cl100k_base`):--| Strategy | Tokens per commit |-|---|---:|-| naive, full contents of every changed file | 13,984 |-| `git diff` only | 1,408 |-| **`get_context`** | **393** |--**35.6x fewer than naive, pooled**, 29.3x as a mean of per-commit ratios, and 3.6x-pooled against `git diff`.--**Lead with the pooled figure.** Pooled is sum over sum, so it weights each commit by-the tokens actually at stake. A mean of per-commit ratios does not: a one-line commit-where `get_context` returns 40 tokens contributes a huge ratio that counts equally-against one saving a hundred thousand. That is how a 35x becomes a 209x in a press-release.--**What this does not tell you.** It is one payload, not a session. It says our-representation of a commit is smaller than the commit, not that an agent finishes-faster. The honest version of that question is §2, where the answer on the same-harness is a much more modest 15.9%.+(the 15-question runs). What producing them cost in runs, machine hours and API+spend, plus the per-tool setup traps that were most of the real effort:+[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head).-Raw CSV:-**[token\_efficiency\_flask30](https://github.com/repowise-dev/repowise-bench/blob/master/results/bakeoff_2026_08/rung1/token_efficiency_flask30_2026-08-01.csv)**+</details>----## 4. Command-output compression--`repowise distill <cmd>` compresses command output *before* the agent reads it:-errors first, exit code preserved, every omission recoverable through an inline-`[repowise#<ref>]` marker.--| Command | Raw tokens | Distilled | Saved |-|---|---:|---:|---|-| `pytest -q` (11 failures) | 3,374 | 1,317 | **61%**, all 11 `FAILED` lines kept |-| `git log -50` | 3,064 | 331 | **89%** |-| `git diff` (30 commits) | 62,833 | 8,635 | 86%, **unsupported, see below** |-| `git log --oneline -30` | 321 | 321 | 0%, already compact |-| `git status` (clean) | 83 | 83 | 0%, too small to distill |--The two 0% rows are the net-positive guard working: distill never inflates small-output. One run per command on one repository, so these are point measurements.-Reduction is also not comprehension: the bytes removed are measured, the evidence-they were safe to remove is narrower.--**One row above is inflated, by the mechanism that broke RTK's number.** A saving may-only count output the agent would actually have received, and the host truncates a-command result at 30,000 characters. `git diff`'s 62,833 raw tokens is eight times-that cap and predates it. **Treat that 86% as unsupported until re-measured.**-`pytest` and `git log` sit under the cap and are unaffected.+<a id="5-code-health-predicts-defects"></a>-**There is a comparable tool and we have not run it.**-[RTK](https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings/) does this-and essentially only this. It is also the tool whose advertised 60 to 90% came back-**7.6% more expensive** under JetBrains' rerun. So this table is a before-and-after-of our own output, which is the weaker design. Only a paired run would settle it.--Full guide: **[docs/agent/DISTILL.md](agent/DISTILL.md)**-------## 5. Code health predicts defects+## Code health predicts defectsA health score is worth something only if the files it flags are the files thatbreak. Scores are taken at a historical commit, bug fixes are counted over thefollowing six months, and nothing after the scoring commit feeds the score.-Across **21 repositories, 9 languages, 2,826 files**: **ROC AUC 0.737**-(95% CI 0.683 to 0.787), ranging 0.55 to 0.86 across individual repos. It beats-recent churn by +0.100 AUC and prior-defect history by +0.117 (DeLong p < 1e-9). On+Across **21 repositories, 9 languages, 2,826 files**: **ROC AUC 0.737** (95% CI+0.683 to 0.787), ranging 0.55 to 0.86 across individual repos. It beats recent+churn by +0.100 AUC and prior-defect history by +0.117 (DeLong p < 1e-9). OnPROMISE/jEdit, a dataset it never saw which carries no git history at all, it holdsat **0.76 to 0.78**.-**It is not better than raw file size at discrimination.** LOC-only scores 0.742-against our 0.737 (p = 0.92, a tie). Where it wins is effort-aware ranking, Popt-+0.134 (95% CI +0.080 to +0.198): same discrimination as counting lines, much better-at ordering a fixed review budget, and unlike a line count it says why. Holding size-fixed, within-band AUC runs 0.525 / 0.572 / 0.593 / 0.718 across NLOC quartiles, so-the signal survives cleanly only in the largest quartile, and a positive control-confirms that collapse is a real absence rather than too few positives to detect one.-**Against CodeScene**, the closest commercial product and the only other vendor inthis category with a published empirical defect study. Both tools scored the same2,770 files at the same leakage-free commit against the same labels:@@ -375,15 +526,13 @@ this category with a published empirical defect study. Both tools scored the sam| Discrimination (ROC AUC) | 0.731 | 0.705 | p = 0.054, marginal || Precision at a 20%-of-lines review budget | 0.580 | **0.636** | p = 0.64, a tie |-Ranking by repowise health surfaces **2.3x the defects under a fixed review budget**.+Ranking by repowise health surfaces **2.3x the defects under a fixed review+budget**.-**Where CodeScene is ahead, and why it is a real choice.** Its precision lead is not-significant, but the behaviour behind it is worth having: it flags about **27 files**-where we flag **132**, trading recall for a short list a team will actually work-through. That is a deliberate operating point, not a weaker model. Our AUC edge is-**marginal**, and so is our raw defect-density lead (16.9x against 14.2x, p = 0.65 on-a heavy tail); the size-normalized version is the one that reaches significance and-the one to cite.+**Where CodeScene is ahead, and why it is a real choice.** Its precision lead is+not significant, but the behaviour behind it is worth having: it flags about **27+files** where we flag **132**, trading recall for a short list a team will actually+work through. That is a deliberate operating point, not a weaker model.**The axis we lost outright.** CodeScene's "Code Red" study reports a **-0.58**correlation between Code Health and mean issue-resolution time, on proprietary Jira@@ -393,13 +542,42 @@ queue-independent effort signals all came back flat with every interval spanningzero. Merge time on GitHub largely measures maintainer review-queue availability.**The business-impact axis remains CodeScene's, unreplicated on open data.**+<details>+<summary><b>Method, limits, and what this does not show</b></summary>++**It is not better than raw file size at discrimination.** LOC-only scores 0.742+against our 0.737 (p = 0.92, a tie). Where it wins is effort-aware ranking, Popt++0.134 (95% CI +0.080 to +0.198): the same discrimination as counting lines, much+better at ordering a fixed review budget, and unlike a line count it says why.++**The signal is weak among files of similar size.** Holding size fixed, within-band+AUC runs 0.525 / 0.572 / 0.593 / 0.718 across NLOC quartiles, so it survives+cleanly only in the largest quartile, and a positive control confirms that collapse+is a real absence rather than too few positives to detect one. A prior-defects+baseline also still beats us on Popt by 0.085 even while losing on AUC.++Our AUC edge over CodeScene is **marginal**, and so is our raw defect-density lead+(16.9x against 14.2x, p = 0.65 on a heavy tail); the size-normalized version is the+one that reaches significance and the one to cite.++Marker weights are **calibrated against a real defect corpus, not hand-tuned**:+every file scored at a commit preceding the bug window so nothing leaks backward,+and an L2-logistic fit with file size as an explicit control, so a marker only+earns weight for defect lift *beyond* being big. Only **26** of the 49 detectors+are permitted to move the defect number, because that is the number carrying these+accuracy claims.+Reports:**[BENCHMARK\_REPORT.md](https://github.com/repowise-dev/repowise-bench/blob/master/health-defect/BENCHMARK_REPORT.md)** ·**[COMPARISON\_REPORT.md](https://github.com/repowise-dev/repowise-bench/blob/master/health-defect/COMPARISON_REPORT.md)**+</details>+----## 6. Indexing time, the row we lose+<a id="6-indexing-time-the-row-we-lose"></a>++## Indexing time, the row we loseWe are the slowest indexer in the field, on every repo we measured, and it is notclose. On `django/django`:@@ -413,292 +591,151 @@ close. On `django/django`:| **repowise** (default, prose on) | **1,058s** | five layers plus generated documentation |That is **22x** CodeGraph like for like, and **135x** with prose on, which is what a-default `repowise init` actually costs you. Both numbers ship, and the 22x is not the-user-facing one.+default `repowise init` actually costs you. Both numbers ship, and the 22x is not+the user-facing one.-**Here is what the extra time buys.** The tools above build a call graph. In the same-run, repowise builds: a **graph** of 36,485 nodes, 90,477 edges and 31,384 symbols-plus PageRank, betweenness, Leiden communities and execution-flow tracing; **git**-history mined across 2,630 files for hotspots, ownership, co-change pairs and bus-factor; **3,392 wiki pages** rendered and embedded for natural-language search;-**architectural decisions** mined from history and sessions; and **code health**,-5,317 findings plus 155 unreachable files and 98 unused exports.+**Here is what the extra time buys.** The tools above build a call graph. In the+same run, repowise builds: a **graph** of 36,485 nodes, 90,477 edges and 31,384+symbols plus PageRank, betweenness, Leiden communities and execution-flow tracing;+**git** history mined across 2,630 files for hotspots, ownership, co-change pairs+and bus factor; **3,392 wiki pages** rendered and embedded for natural-language+search; **architectural decisions** mined from history and sessions; and **code+health**, 5,317 findings plus 155 unreachable files and 98 unused exports.So "22x slower" and "the index contains categorically more" are both true, andneither cancels the other. If all you want is a call graph, CodeGraph builds one in16 seconds and you should use it. The comparison that would be dishonest is quoting-the ratio without the column beside it, which is why the column is here.+the ratio without the column beside it, which is why the column is here. It is also+a one-time cost: updates after the first index are incremental.-It is also a one-time cost. Updates after the first index are incremental.+<details>+<summary><b>Method, limits, and what this does not show</b></summary>A fitted build-cost curve for five tools across a 12x repository-size range, whichnobody in this field had published, is in[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head#the-result-nobody-in-the-field-had-published-a-build-cost-curve).-Our exponent is sublinear at 0.906, and we publish that as sublinear **work** rather-than efficiency, because symbol density halves across the range. That is a product-finding against us.-------## 7. Edge precision--Every other row on this page counts things. This one asks whether they are **true**.--A resolver that guesses aggressively wins a coverage table and a raw edge count while-sending its reader to the wrong function. So we hand-graded call edges from source, on-both sides, by the same method: 30 rows per language per tool, seed 2026, stratified by-resolution strategy, every row read with its imports and enclosing scope open.--| | correct / n | 95% CI |-|---|---|---|-| **repowise** | **229/270 = 84.8%** | [80.0, 88.6] |-| **CodeGraph 1.5.0** | **154/270 = 57.0%** | [51.1, 62.8] |--Per language, nine languages, both sides read:--| Language | repowise | CodeGraph | |-|---|---|---|---|-| typescript | 29/30 | 7/30 | separates |-| go | 29/30 | 29/30 | tie |-| csharp | 28/30 | 20/30 | separates |-| python | 28/30 | 19/30 | separates |-| kotlin | 27/30 | 13/30 | separates |-| swift | 23/30 | 19/30 | tie |-| cpp | 23/30 | 16/30 | tie |-| rust | 22/30 | 13/30 | tie |-| java | 20/30 | 18/30 | tie |+Our exponent is sublinear at 0.906, and we publish that as sublinear **work**+rather than efficiency, because symbol density halves across the range. That is a+product finding against us.-**Read our number the other way round: roughly fifteen percent of our call edges are-wrong.** That is the number we plan against, and rust, java and cpp are where it-concentrates.--**Four of nine cells separate. Five are ties and we report them as ties.** At n=30 the-interval runs about ±16 points near 60%, so a point-estimate gap inside two overlapping-intervals is not a win. C++ is a tie despite looking like a 23-point lead.--**One repository goes clearly to them.** On `seastar` CodeGraph reads 6/10 against our-4/10, the only repository in the audit, on any language, where they beat us on a clear-margin. Our misses there are chained calls on an untyped receiver; they infer the-callee's declared return type and validate against it, so a failed inference costs them-an edge instead of buying them a wrong one. On `aria2` both sides read 10/10 and they-resolve 24,950 distinct call edges to our 9,486. Precision is not the only reading.--Cells were measured at different commits, and the staleness runs **conservative**: every-resolver change in between only removes wrong edges and gained zero, so 84.8% is a floor.-No cell was measured on the 0.44.0 release; the full table pins a commit per cell.--**All 540 graded rows are published**, one file per cell, each row carrying the call site,-the declaration the tool bound it to, the verdict and the reason it was given. A script in-the same directory rebuilds every table above from them and fails if it disagrees. One cell-of the eighteen, rust on our side, ships the 30 sites that were read without their per-row-verdicts, because that cell was re-read on a fresh draw and the verdicts of that read were-never written down; the file says so on every row.-**[graph/experiments/g1-edge-precision/rows](https://github.com/repowise-dev/repowise-bench/tree/master/graph/experiments/g1-edge-precision/rows)**.+</details>----## 8. The same question, against an answer key we do not control+<a id="4-command-output-compression"></a>+<a id="command-output-compression"></a>-Section 7 is hand-graded, by us, on both sides. That is the strongest form of a-weak thing: **every graph-quality number in this field, ours included, is scored-against something the publisher controls.** A tool that is confidently wrong in a-consistent way scores well and the reader has no way to check.+## Command-output compression++`repowise distill <cmd>` compresses command output *before* the agent reads it:+errors first, exit code preserved, every omission recoverable through an inline+`[repowise#<ref>]` marker.-So we re-asked the question with the answer key taken out of our hands. On Go the-oracle is the **Go team's own RTA call graph** from-`golang.org/x/tools/go/callgraph/rta`, over the fully type-checked program. On-TypeScript it is the **`tsc` type checker's own resolution** of every call site.-We did not write either one, we cannot tune either one, and anyone with the-toolchain can regenerate both. Five tools, seven cells, five repositories,-**37,853 oracle edges**.+| Command | Raw tokens | Distilled | Saved |+|---|---:|---:|---|+| `pytest -q` (11 failures) | 3,374 | 1,317 | **61%**, all 11 `FAILED` lines kept |+| `git log -50` | 3,064 | 331 | **89%** |+| `git diff` (30 commits) | 62,833 | 8,635 | 86%, **unsupported, see below** |+| `git log --oneline -30` | 321 | 321 | 0%, already compact |+| `git status` (clean) | 83 | 83 | 0%, too small to distill |-### Why precision alone is not the claim, and neither is recall+**One row above is inflated, by the mechanism that broke RTK's number.** A saving+may only count output the agent would actually have received, and the host+truncates a command result at 30,000 characters. `git diff`'s 62,833 raw tokens is+eight times that cap and predates it. **Treat that 86% as unsupported until+re-measured.** `pytest` and `git log` sit under the cap and are unaffected.-Two numbers, and you need both, because each one on its own has a cheap way to-win.+<details>+<summary><b>Method, limits, and what this does not show</b></summary>-- **Recall** is the share of the real call graph a tool found. Draw an edge- between everything and you score 1.000.-- **Precision** is the share of a tool's edges that are real. Draw one edge you- are sure of and you score 1.000.+**There is a comparable tool and we have not run it.**+[RTK](https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings/) does+this and essentially only this. It is also the tool whose advertised 60 to 90% came+back **7.6% more expensive** under JetBrains' rerun. So this table is a+before-and-after of our own output, which is the weaker design. Only a paired run+would settle it.-Both failure modes are in the table below, measured, not hypothetical. One tool-draws 12,533 edges on syft and more than a third of them are calls the compiler-says do not exist. Another scores the highest precision on this whole page,-0.997, from a graph containing **17% of the calls in the repository**: on-gitleaks it stored 4,367 call rows of which 76 resolve to anything, and a graph-that small is very hard to be wrong in and not much use to walk.+The two 0% rows are the net-positive guard working: distill never inflates small+output. One run per command on one repository, so these are point measurements.+Reduction is also not comprehension: the bytes removed are measured, the evidence+they were safe to remove is narrower.-**So the only meaningful reading is the pair**, and it is the pair we claim:+Full guide: **[docs/agent/DISTILL.md](agent/DISTILL.md)**-> **In all seven cells, no tool that recovers as much of the call graph as we do-> gets more of it right.**+</details>-That sentence names no threshold, which is what makes it worth something. It-cannot be tuned by picking a cutoff, and adding a competitor can only break it.-Two competitors were added to this experiment after it was first written, and it-held in all seven cells.+----**Precision: of the call edges a tool emits, the share the compiler confirms.**+<a id="3-loading-one-commits-context-the-easy-number"></a>+<a id="loading-one-commits-context-the-easy-number"></a>-| cell | repowise | CodeGraph 1.5.0 | codebase-memory-mcp 0.10.8 | Graphify 0.9.31 | code-review-graph 2.3.7 |-|---|---|---|---|---|---|-| cobra (with tests) | 0.972 | 0.929 | 0.912 | 0.971 | **0.997** |-| gitleaks (no tests) | 0.976 | 0.972 | 0.934 | **0.997** | 0.759 |-| gitleaks (with tests) | 0.974 | 0.971 | 0.922 | **0.995** | 0.800 |-| syft (no tests) | 0.943 | 0.872 | 0.635 | 0.771 | **0.968** |-| syft (with tests) | 0.950 | 0.864 | 0.673 | 0.802 | **0.966** |-| zod (no tests) | **0.992** | 0.729 | 0.987 | 0.825 | 0.932 |-| hono (no tests) | 0.977 | 0.805 | 0.949 | 0.980 | 0.966 |+## Loading one commit's context, the easy number-**Recall: of the edges the compiler has, the share the tool found.**+This is the measurement almost every tool in this category publishes, and we are+labelling it as such. Over the 30 most recent non-merge commits of `pallets/flask`,+counted with deterministic `tiktoken` (`cl100k_base`):-| cell | repowise | CodeGraph | codebase-memory-mcp | Graphify | code-review-graph |-|---|---|---|---|---|---|-| cobra (with tests) | 0.684 | **0.763** | 0.743 | 0.433 | 0.174 |-| gitleaks (no tests) | 0.955 | 0.920 | **0.967** | 0.886 | 0.026 |-| gitleaks (with tests) | 0.914 | 0.895 | **0.945** | 0.832 | 0.032 |-| syft (no tests) | 0.513 | 0.508 | **0.542** | 0.447 | 0.201 |-| syft (with tests) | 0.322 | 0.338 | **0.361** | 0.273 | 0.086 |-| zod (no tests) | **0.703** | 0.373 | 0.694 | 0.248 | 0.652 |-| hono (no tests) | **0.731** | 0.684 | 0.686 | 0.688 | 0.691 |+| Strategy | Tokens per commit |+|---|---:|+| naive, full contents of every changed file | 13,984 |+| `git diff` only | 1,408 |+| **`get_context`** | **393** |-Read a row across both tables and the trade is visible in every one. The tools-above us on precision are below us on recall, every time. The tool above us on-recall is below us on precision, every time. **Nobody is above us on both, in any-cell.** Intervals, per-cell artifacts and the full method are linked at the end-of this section; ties are ties and are marked as ties there.+**35.6x fewer than naive, pooled**, 29.3x as a mean of per-commit ratios, and 3.6x+pooled against `git diff`.-**The weaker readings, so nobody has to infer them.** We are the most precise arm-outright in one cell of seven and tied for it in one more. We are beaten on-precision in five, by code-review-graph on cobra and both syft cells and by-Graphify on both gitleaks cells. Against the two tools this experiment started-with, CodeGraph and codebase-memory-mcp, we are the most precise in seven of-seven, and that narrower claim should always carry its label.--This also does something section 7 cannot: it removes the n=30 ceiling.-Hand-grading caps a precision estimate at roughly ±13 points near a 95% rate. An-oracle grades every edge, so n becomes the size of the repository.--**The two methods agree.** On Go, section 7 read 29/30 = 96.7% by hand for us and-29/30 = 96.7% for CodeGraph. The Go compiler, over roughly 1,600 edges on the same-repository, says 97.6% and 97.2%. Two unrelated methods, one person reading source-and one type checker, land within about a point on both arms. That is the result-we care about most and it is not a competitive one.--### The column we lose--We lead recall in the two TypeScript cells and in **none of the five Go cells**,-where codebase-memory-mcp leads four and CodeGraph the fifth. On cross-file-coverage over 35 repositories, codebase-memory-mcp separates from us on 15 and we-separate on none. Those are real losses and they are not softened here.--What the oracle adds is the price of that lead rather than an excuse for ours: on-syft, more than a third of what that tool emits is a call the Go compiler says-does not exist. Coverage rewards drawing edges and never asks whether they are-real, which is why no page in that benchmark publishes a coverage number without-a precision number beside it.--**Where our own miss goes**, decomposed rather than waved at. On syft without-tests we miss 3,846 of the oracle's 7,898 edges: 44% of that miss is dynamic-dispatch alone and a further 39% is dispatch with a closure at one end, and the-two buckets overlap so neither is the whole gap. Interface dispatch is the-ceiling and nobody in the comparison has cleared it, at 6.5 distinct possible-targets per call site. Matching that recall means emitting six edges where one is-right, which is the behaviour the precision table charges other tools for.+**What this does not tell you.** It is one payload, not a session. It says our+representation of a commit is smaller than the commit, not that an agent finishes+faster. The honest version of that question is the agent loop above, where the+answer on the same harness is a much more modest 15.9%.-**Do not compare recall across rows.** It swings from 0.03 to 0.97, driven by how-many entry points the oracle had (4 on gitleaks, 268 on syft-with-tests), not by-tool quality. Only within-row comparisons carry meaning and a pooled recall over-these cells would be meaningless.+<details>+<summary><b>Method, limits, and what this does not show</b></summary>-### Limits+**Lead with the pooled figure.** Pooled is sum over sum, so it weights each commit+by the tokens actually at stake. A mean of per-commit ratios does not: a one-line+commit where `get_context` returns 40 tokens contributes a huge ratio that counts+equally against one saving a hundred thousand. That is how a 35x becomes a 209x in+a press release.-- **Two languages, seven cells, five repositories.** This is not a nine-language- claim and must not be quoted as one. Section 7 is the nine-language number.-- **A contradicted edge is very strong evidence, not proof.** RTA is unsound under- reflection and `go:linkname`, so a genuinely dynamic edge can land in that- bucket. It applies to all five arms equally and the gaps are far too large to- be explained by it, which is why the metric is named *precision against the- oracle* rather than precision.-- **Edges the oracle cannot speak about are charged to nobody** and reported at- full size. That bucket is 0.4% to 11.1% of a tool's output on the Go cells and- 16% to 46% on the TypeScript ones, where dependencies are not installed in the- pinned corpus.-- **A library has no `main`, so RTA has no roots**; cobra is analysed through its- test binaries only.-- The two variants of a repository answer different questions. Report both or say- which one you used.-- **Two oracle languages, and the programme stops at two.** C#, Java, Kotlin and- C++ each need a toolchain installed and a working build per repository, none of- which exists on the measurement machine. Rust has the toolchain and no sound- call-graph tool exists for it. Python, Ruby and PHP admit no oracle even in- principle. So section 7 is the permanent method on those languages rather than- a stopgap, and nothing here should be read as a claim about them.-- **The TypeScript cells exclude test files, and the with-tests variants are- void.** A test file imports its own package by name, the pinned corpus has no- dependencies installed, and roughly a third of call sites go unresolvable,- which takes the unjudged bucket past three quarters. Both variants were run and- neither is quoted anywhere. Installing the dependency trees would fix it and- would also have to go to a scratch copy, since `node_modules/` inside the- corpus changes what every tool walks.-- **The three competitors here are priced on these two languages only.**- codebase-memory-mcp, Graphify and code-review-graph have no precision figure of- any kind on the other nine languages in the coverage bench, and none of the- three was ever entered into the section 7 audit in either direction. Section 7- is a two-tool claim and section 8 a five-tool one, and neither transfers to the- other's languages.-- **Two of the five arms are read through an adapter we wrote**, and both now beat- us on precision in some cell, so the reading matters. Graphify tags 93% of its- call edges `INFERRED` rather than AST-certain and we score all of them, which is- the choice least favourable to us. code-review-graph stores unresolved callees- beside resolved ones and we score only the resolved, which is the choice most- favourable to it: on gitleaks that is 76 edges out of 4,367 stored rows. Both- choices are argued in the benchmark and a reader who disagrees can recompute- either from the artifacts.+Raw CSV:+**[token\_efficiency\_flask30](https://github.com/repowise-dev/repowise-bench/blob/master/results/bakeoff_2026_08/rung1/token_efficiency_flask30_2026-08-01.csv)**-Full method, per-cell artifacts, the twenty hand-confirmed identities the protocol-required, and the graded pre-registration including the two predictions that-missed:-**[graph/experiments/g4-oracle-anchored](https://github.com/repowise-dev/repowise-bench/tree/master/graph/experiments/g4-oracle-anchored)**.+</details>----## Limits--Beyond the ones stated in each section:--- **Every retrieval number here is Python or Go.** The graph sections are wider,- at nine languages hand-graded and two compiler-graded, and they are the only- sections that are. A JavaScript/TypeScript corpus- (`mui/material-ui`, six tools, a 12x size range) is built and half graded, but- **its sealed half is unrun and nothing from it is quoted here.** Publishing the- development half is what that split exists to prevent, so the row arrives when the- sealed half is evaluated, once.-- **We index code files only, which depresses our own coverage on repositories with- documentation.** Deliberate: on doc-heavy repositories the docs outweigh the code,- and indexing them polluted the index and degraded retrieval for the code questions- the tool exists to answer. On the JavaScript corpus above, **21% of gold files are- `.md` or `.json`**, unreachable to us by construction rather than by ranking.- cocoindex makes the opposite trade, is the only arm of six to retrieve any- documentation gold, and pays for it on code gold.-- **§2 is one repository**, `django/django` at one commit, which is in every model's- training data.-- **§2 measures single-session questions**, roughly four to nine turns each. Nothing- on this page measures a long multi-hour engineering task, and we will not imply a- number for one.-- **§1 quotes only the sealed half.** Pooling both halves gives a stronger p and we- do not quote it.-- **§2's difficulty split is post-hoc.** The pre-registered comparisons are stronger- evidence.-- **§4 has no head-to-head and one row that needs re-measuring.**-- **§5's signal is weak among files of similar size**, and a prior-defects baseline- still beats us on Popt by 0.085 even while losing on AUC.-- **§7 was graded by us on both sides.** That is what §8 exists to check, and the two- agree where both exist, but only §8 has an answer key we did not produce.-- **§7 and §8 measure precision, which is one of two halves.** We lead no recall cell- in §8 and lose cross-file coverage on 15 of 35 repositories in the same bench. A- precision win is a claim about the edges a tool draws, never about how many.+## Limits that apply to the whole page++<details>+<summary><b>Six limits that are not specific to any one section</b></summary>++- **Every retrieval number here is Python or Go.** The graph sections are wider, at+ nine languages hand-graded and two compiler-graded, and they are the only+ sections that are. A JavaScript/TypeScript corpus (`mui/material-ui`, six tools,+ a 12x size range) is built and half graded, but **its sealed half is unrun and+ nothing from it is quoted here.** Publishing the development half is what that+ split exists to prevent, so the row arrives when the sealed half is evaluated,+ once.+- **We index code files only, which depresses our own coverage on repositories+ with documentation.** Deliberate: on doc-heavy repositories the docs outweigh the+ code, and indexing them polluted the index and degraded retrieval for the code+ questions the tool exists to answer. On the JavaScript corpus above, **21% of+ gold files are `.md` or `.json`**, unreachable to us by construction rather than+ by ranking. cocoindex makes the opposite trade, is the only arm of six to+ retrieve any documentation gold, and pays for it on code gold.+- **The agent-loop section is one repository**, `django/django` at one commit, which+ is in every model's training data, and it measures single-session questions of+ roughly four to nine turns each. **Nothing on this page measures a long+ multi-hour engineering task**, and we will not imply a number for one.+- **Precision is one of two halves.** We lead no recall cell against the oracle and+ lose cross-file coverage on 15 of 35 repositories in the same bench.+- **The health signal is weak among files of similar size**, and it ties raw line+ count on discrimination.+- **Command-output compression has no head-to-head and one row that needs+ re-measuring.**++</details>---@@ -712,66 +749,55 @@ claims on real agent work.| [Caveman](https://blog.jetbrains.com/ai/2026/07/speak-to-ai-agents-like-cavemen-tosave-tokens/) | 65% | **8.5%** || [RTK](https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings/) | 60 to 90% | **7.6% more expensive** at low reasoning effort (p = 0.004) |-Greptile's 82% became 45% under Augment's rerun. The pattern is consistent enough to-be a rule: in this category, the advertised number and the reran number are different-numbers. They differ for one reason.--**Measuring one context load is easy. Measuring an agent loop is hard.** Showing that-your representation of a file is smaller than the file is a real measurement, and the-one almost everybody publishes. It is not the question anyone has, which is whether-an agent given your tool finishes the job having done less work. Agents re-read,-backtrack, re-plan and re-explore, so a compression that looks like 90% on one-payload routinely nets out near zero across a session, and can go negative when the-agent works harder to recover what you compressed away.--**This page publishes both and labels which is which.** §3 is the easy-one-context-load figure. §2 is the agent-loop figure, measured the way JetBrains-measured, and deliberately the smaller number. We reran our own headline on a second-and then a third harness and published what came back: it moved a number we had been-quoting, and §2 carries the correction.--Four rules apply to every number here, and each exists because breaking it produced a-wrong published figure at least once:--- **Pre-register before spending**, as its own commit, so a favourable result cannot- become a different question afterwards.-- **Seal a half**, split by instance id before any work begins, evaluated once.-- **Precision and files-served beside coverage**, never averaged into one figure.-- **Prove an arm was alive and its extractor works before recording a zero.** A dead- server, a wrong tool name and a broken output parser all score exactly like a bad- tool.--The full method, drawn end to end with every gate and the failure that put it there,-is+Greptile's 82% became 45% under Augment's rerun. The pattern is consistent enough+to be a rule: in this category, the advertised number and the reran number are+different numbers. **Measuring one context load is easy. Measuring an agent loop is+hard.** Agents re-read, backtrack, re-plan and re-explore, so a compression that+looks like 90% on one payload routinely nets out near zero across a session.++This page publishes both and labels which is which. We reran our own headline on a+second and then a third harness and published what came back: it moved a number we+had been quoting, and the agent-loop section carries the correction.++Four rules apply to every number here, each because breaking it produced a wrong+published figure at least once. **Pre-register before spending**, as its own commit,+so a favourable result cannot become a different question afterwards. **Seal a+half**, split by instance id before any work begins, evaluated once. **Precision and+files-served beside coverage**, never averaged into one figure. **Prove an arm was+alive and its extractor works before recording a zero**, because a dead server, a+wrong tool name and a broken output parser all score exactly like a bad tool. The+method end to end, with every gate and the failure that put it there:**[THE\_LOOP.md](https://github.com/repowise-dev/repowise-bench/blob/master/head-to-head/THE_LOOP.md)**.---## Where the depth is-This page is the summary. Stop wherever you like; each level is the evidence for the-one above it.+This page is the summary. Everything below it is the evidence, in+**[repowise-bench](https://github.com/repowise-dev/repowise-bench)**: what each run+cost in machine hours and API spend, the build-cost curve, one page per competitor+with every setup trap, all nine method gates, every launch command and exclusion+with its reason, one pre-registration per scored run committed before that run spent+anything, and every graded cell including the runs we invalidated with their+invalidation notes attached.-| Level | What is there |+| Start here | For ||---|---|-| **[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head)** | Who wins what, the build-cost curve, and what each index can rank at all |-| **[graph/](https://github.com/repowise-dev/repowise-bench/tree/master/graph)** | The graph-quality bench: edge precision hand-graded on both sides across nine languages against one competitor, precision and recall against a compiler oracle on Go and TypeScript against two, and cross-file coverage, adversarial invariance and build cost across five |-| **[arms/](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head/arms)** | One page per competitor: what it is, what it serves, and every setup trap. Four of six have a step that produces a clean zero when missed |-| **[THE\_LOOP.md](https://github.com/repowise-dev/repowise-bench/blob/master/head-to-head/THE_LOOP.md)** | The method and all nine gates, each named with the failure that created it |-| **[configs/arms.yaml](https://github.com/repowise-dev/repowise-bench/blob/master/configs/arms.yaml)** | Every launch command, allowlisted tool and exclusion with its reason. Read this if you think an arm was set up unfairly |-| **[configs/\*.PREREGISTRATION.md](https://github.com/repowise-dev/repowise-bench/tree/master/configs)** | One per scored run, each committed before its run spent anything |-| **[results/bakeoff\_2026\_08](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08)** | Every graded cell and every verbatim response, including the runs we invalidated, with their invalidation notes attached |+| **[graph/](https://github.com/repowise-dev/repowise-bench/tree/master/graph)** | The graph-quality bench: hand-graded precision across nine languages, the compiler oracle on Go and TypeScript, cross-file coverage and build cost across five tools |+| **[head-to-head](https://github.com/repowise-dev/repowise-bench/tree/master/head-to-head)** | The retrieval and agent-loop bench: who wins what, what each index can rank at all, and what it all cost to produce |+| **[results/bakeoff\_2026\_08](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08)** | Every graded cell and every verbatim response || **[repro/README.md](https://github.com/repowise-dev/repowise-bench/blob/master/repro/README.md)** | Per claim: what it costs to reproduce, how long it takes, and which ones need credentials we cannot hand you |-**Found a problem with one of these numbers, or want your tool in the field?** Adding-a competitor is a YAML block, no Python and no runner change. See+**Found a problem with one of these numbers, or want your tool in the field?**+Adding a competitor is a YAML block, no Python and no runner change. See[CONTRIBUTING.md](https://github.com/repowise-dev/repowise-bench/blob/master/CONTRIBUTING.md).-Tool versions as measured: CodeGraph 1.5.0, Graphify 0.9.31, Serena 1.6.2.dev0,-code-review-graph 2.3.7, cocoindex as of 2026-08-09, codebase-memory-mcp 0.10.8.+<sub>Tool versions as measured: CodeGraph 1.5.0, Graphify 0.9.31, Serena 1.6.2.dev0,+code-review-graph 2.3.7, cocoindex as of 2026-08-09, codebase-memory-mcp 0.10.8.</sub>## See also- [The five intelligence layers](layers/INTELLIGENCE_LAYERS.md)- [Code health methodology](layers/CODE_HEALTH.md)- [MCP tool reference](agent/MCP_TOOLS.md)+- [Roadmap](../ROADMAP.md), including the interface-dispatch recall ceilingdiff --git a/docs/README.md b/docs/README.mdindex 370247479..0a2baa527 100644--- a/docs/README.md+++ b/docs/README.md@@ -20,7 +20,7 @@ your agent in under five minutes, with no API key.| **Living in an editor** | [VS Code extension](agent/VSCODE.md) | [Codex](agent/CODEX.md) · [opencode](agent/OPENCODE.md) || **A team lead watching what ships** | [Change risk](layers/CHANGE_RISK.md) | [Code health](layers/CODE_HEALTH.md) · [Bug history](layers/BUG_HISTORY.md) || **Running many repos** | [Workspaces](scale/WORKSPACES.md) | [Worktrees](scale/WORKTREES.md) · [Auto-sync](scale/AUTO_SYNC.md) |-| **Evaluating repowise to buy it** | [Commercial](business/COMMERCIAL.md) | [Security & compliance](business/SECURITY_COMPLIANCE.md) · [Benchmarks](BENCHMARKS.md) |+| **Evaluating repowise to buy it** | [Commercial](business/COMMERCIAL.md) | [Security & compliance](business/SECURITY_COMPLIANCE.md) · [Benchmarks](BENCHMARKS.md) · [Roadmap](../ROADMAP.md) || **Contributing** | [Architecture](architecture/README.md) | [CONTRIBUTING](../.github/CONTRIBUTING.md) |---@@ -58,7 +58,7 @@ your agent in under five minutes, with no API key.| [layers/GRAPH.md](layers/GRAPH.md) | The dependency graph: what is in it, how every edge is resolved, and how much to trust each one || [layers/DECISIONS.md](layers/DECISIONS.md) | Architectural decisions mined from your repo and from your own agent sessions || [layers/DEAD_CODE.md](layers/DEAD_CODE.md) | Unreachable files, unused exports, and zombie packages by confidence tier |-| [layers/LANGUAGE_SUPPORT.md](layers/LANGUAGE_SUPPORT.md) | What works per language, across 19 parsed languages and 13 at the Full tier |+| [layers/LANGUAGE_SUPPORT.md](layers/LANGUAGE_SUPPORT.md) | What works per language: 19 parsed to a full AST, 35 on the five-rung ladder || [layers/WIKI.md](layers/WIKI.md) | The generated wiki: page types, what `update` re-renders, styles, output language |## Scale it@@ -86,7 +86,7 @@ your agent in under five minutes, with no API key.| Doc | What it covers ||-----|----------------|-| [BENCHMARKS.md](BENCHMARKS.md) | Agent-efficiency, distillation, and defect-prediction results, each with what it does not show |+| [BENCHMARKS.md](BENCHMARKS.md) | Every published number with its sample size and its test: call-graph precision judged by a compiler we do not control, retrieval against the agent-context field, defect prediction against CodeScene, distillation, and the rows we lose |## Teams & business@@ -94,6 +94,7 @@ your agent in under five minutes, with no API key.|-----|----------------|| [business/COMMERCIAL.md](business/COMMERCIAL.md) | Hosted tier, enterprise (on-prem, SSO/SCIM), and commercial licensing || [business/SECURITY_COMPLIANCE.md](business/SECURITY_COMPLIANCE.md) | What leaves your machine, what gets stored, and the answers your security team wants |+| [ROADMAP.md](../ROADMAP.md) | What we are building next, what we are not building, and why there are no dates on it |## Architecture & internalsdiff --git a/docs/business/COMMERCIAL.md b/docs/business/COMMERCIAL.mdindex fa60337d6..08f35885e 100644--- a/docs/business/COMMERCIAL.md+++ b/docs/business/COMMERCIAL.md@@ -19,7 +19,7 @@ models. Specific pricing figures are provided in a separate proposal.## 1. Who the commercial license is forThe open-source distribution covers the full developer-experience surface: the five-intelligence layers, the eleven MCP tools, multi-repo workspaces, the local dashboard,+intelligence layers, the ten MCP tools, multi-repo workspaces, the local dashboard,auto-sync, and auto-generated `CLAUDE.md`. That is everything an engineer or a teamneeds to make their AI coding agents codebase-aware.@@ -41,21 +41,30 @@ scale, in a regulated or security-sensitive environment**:All of the following ship in `pip install repowise` today, free for internal use.-- **Five intelligence layers**: Graph (tree-sitter AST across 19 languages, two-tier- dependency graph, call resolution, heritage extraction, Leiden communities,- PageRank / betweenness / SCC), Git (hotspots, ownership, co-change pairs, bus- factor, significant commits, contributor profiles, module health), Documentation- (LLM-generated wiki, freshness scoring, RAG search), Decision (architectural+- **[Five intelligence layers](../layers/INTELLIGENCE_LAYERS.md)**: Graph+ (tree-sitter AST across 19 languages, two-tier dependency graph, call+ resolution, heritage extraction, Leiden communities, PageRank / betweenness /+ SCC), Git (hotspots, ownership, co-change pairs, bus factor, significant+ commits, contributor profiles, module health), Documentation (a wiki page per+ module and file, freshness scoring, hybrid search), Decision (architecturaldecision records linked to graph nodes, staleness tracking), and Code Health(49 deterministic detectors, 1–10 score per file, coverage ingestion, trend- alerts).+ alerts). Full detail on each, and what every layer costs to build:+ **[INTELLIGENCE_LAYERS.md](../layers/INTELLIGENCE_LAYERS.md)**.+- **Zero LLM calls in every analysis layer.** Graph, git, code health, change+ risk, dead code and the PR bot make no model calls at all, and the whole wiki+ renders from code structure with no API key (`repowise init --no-prose`).+ Model-written prose is opt-in, priced before you confirm it, and runs on your+ own key or fully offline through Ollama. Six of the seven decision sources are+ deterministic as well; only comment archaeology, which reads rationale prose on+ high-centrality code, needs a provider.- **Ten task-shaped MCP tools**: `get_overview`, `get_answer`, `get_context`,`get_symbol`, `search_codebase`, `get_risk`, `get_change_risk`, `get_why`,- `get_dead_code`, `get_health`. Benchmarked at **−33.5 % cost per question**- against a bare agent, cheaper on 13 of 15 questions (n=15, p=0.007), and- **−49 % to −70 % tool calls** across paired runs on `pallets/flask` and- `scikit-learn`. See [docs/BENCHMARKS.md](../BENCHMARKS.md) for the sample- sizes, the tests, and the runs where the cost saving did not replicate.+ `get_dead_code`, `get_health`, plus `list_repos` for workspace routing.+ Measured at **−31.6 % of the agent's own output tokens** against a bare agent+ on 43 questions (p < 0.0001), leaner on 37 of 44, reached in 3.8 tool calls+ where the bare agent needed 7.2. See [docs/BENCHMARKS.md](../BENCHMARKS.md) for+ the sample sizes, all three agent harnesses, and the rows where we lose.- **Multi-repo workspace intelligence**: cross-repo co-changes, API contractextraction (HTTP / gRPC / topics) with provider↔consumer matching, packagedependency mapping, federated MCP queries (`repo="all"`), workspace dashboard and@@ -70,6 +79,13 @@ All of the following ship in `pip install repowise` today, free for internal useSecurity (local pattern scan), Knowledge Map, and the workspace views.- **Dead-code detection**: pure graph traversal, confidence-tiered, framework-aware(ASP.NET, Django, FastAPI, Flask, Rails, Laravel), dynamic-import aware.+- **Test intelligence, from a coverage report and from the call graph.** Ingests+ LCOV / Cobertura / Clover like a coverage service, then does the half a coverage+ service structurally cannot: answers *which tests reach this file* and *which+ tests does this diff exercise* **with no report at all**, at **95.7% and 97.5%+ precision** measured against a real `coverage run --contexts=test`. Measured and+ inferred rows are labelled and never averaged. Zero LLM calls, no CI integration.+ ([TEST_INTELLIGENCE.md](../layers/TEST_INTELLIGENCE.md))- **Privacy** (self-hosted): source never leaves your infrastructure, BYOK or fullyoffline via Ollama. Anonymous, opt-out usage telemetry (command names and coarseenvironment only, never code, paths or repo names) can be turned off with@@ -83,14 +99,33 @@ All of the following ship in `pip install repowise` today, free for internal use## 3. First-class language coverage-Repowise treats **13 languages at Full tier** (Python, TypeScript, JavaScript, Svelte,-Vue, Java, Kotlin, Go, Rust, C++, **C#**, Scala, and Ruby) with AST parsing, import-resolution, named bindings, call resolution, heritage extraction, multi-project-workspace resolvers, framework-aware edges, per-language dynamic-hint extractors, and-code-health markers. A further 5 languages (C, Swift, PHP, Dart, Object Pascal/Delphi)-sit at Good tier,-and Luau is partial. SQL/dbt, shell, HTML, and the config formats are handled by-dedicated extractors on top of that.+Repowise parses **19 languages to a full AST** and places **35 on a five-rung+ladder**, so "do you support X" gets the rung as its answer rather than a yes or a+no. Both numbers matter and neither is worth quoting alone.++The 19 are the top three rungs. At **Full tier sit 13** (Python, TypeScript,+JavaScript, Svelte, Vue, Java, Kotlin, Go, Rust, C++, **C#**, Scala, and Ruby) with+AST parsing, import resolution, named bindings, call resolution, heritage+extraction, multi-project workspace resolvers, framework-aware edges, per-language+dynamic-hint extractors, and code-health markers. A further **5 at Good tier** (C,+Swift, PHP, Dart, Object Pascal/Delphi) get all of that except the full health+suite, and Luau is partial. SQL/dbt, shell, HTML, and the config formats are+handled by dedicated extractors on top of that.++Below the AST line the ladder continues rather than stopping: **7 at Lightweight**+(Elixir, Clojure, Haskell, Lean 4, Erlang, F#, HTML) get a real file-to-file import+graph with no symbol-level claims, and **9 at Structural** (Objective-C, R, Zig,+Julia, Elm, OCaml, Crystal, Nim, D) get the git intelligence layer only: blame,+hotspots, co-change, ownership and bug history, with no parsing. Stating the rung+per language is deliberate, because a support claim that averages these together+would not survive a technical evaluation. Full ladder:+[LANGUAGE_SUPPORT.md](../layers/LANGUAGE_SUPPORT.md).++**Languages are never gated.** Every one of them ships in the AGPL distribution,+including the ones still climbing. What is on the way up, **COBOL** among them,+is on the public [roadmap](../../ROADMAP.md#languages). Where you need a language+or framework that is not there, §5.4 covers having it built and maintained by us+as a commercial line item, and the result still ships to everyone under AGPL.For estates built on a particular stack, the relevant Full-tier capabilities areworth calling out. For **.NET**, as one example:@@ -122,12 +157,13 @@ the items that matter most to you can be prioritized.| Capability | Open Source (AGPL) | Commercial License ||------------|:------------------:|:------------------:|| Five intelligence layers | ✅ | ✅ |-| Eleven MCP tools | ✅ | ✅ |+| Ten task-shaped MCP tools (plus `list_repos`) | ✅ | ✅ || Multi-repo workspaces | ✅ | ✅ || Full-tier language support (incl. C# / .NET) | ✅ | ✅ || Local dashboard (incl. local security pattern scan) | ✅ | ✅ || Auto-sync (hooks, watcher, webhooks) | ✅ | ✅ || Auto-generated CLAUDE.md | ✅ | ✅ |+| Test intelligence (coverage ingestion **and** the graph-inferred test map) | ✅ | ✅ || Local full-history secret scan (`repowise security scan --history`) | ✅ | ✅ || Graph-aware enhanced security scanning | — | ✅ *(GA on hosted)* || Language-specific security rulesets | — | ✅ *(dev)* |diff --git a/docs/layers/INTELLIGENCE_LAYERS.md b/docs/layers/INTELLIGENCE_LAYERS.mdindex 3892c8266..2c071d46c 100644--- a/docs/layers/INTELLIGENCE_LAYERS.md+++ b/docs/layers/INTELLIGENCE_LAYERS.md@@ -33,8 +33,10 @@ searchable in natural language.- [Auto-sync](#auto-sync-five-ways-to-stay-current)- [Auto-generated CLAUDE.md](#auto-generated-claudemd)-Everything below is computed **without model calls**. An LLM is an optional-upgrade for prose quality in the docs layer, never a requirement for the index.+Everything below is computed **without model calls**, with two named exceptions,+both optional and both skipped entirely on a keyless index: prose quality in the+docs layer, and comment archaeology in the decision layer. Neither is a+requirement for the index.---@@ -147,7 +149,10 @@ Architectural decisions mined at index time from **five sources**: ADR files(Nygard/MADR), PR and squash-commit bodies, inline markers, git archaeology, andcentrality-bounded code comments. Two further capture paths exist for humans andagents: `repowise decision add`, and mining your Claude Code or Codex session-transcripts.+transcripts. **Seven sources in total, six of them deterministic**; comment+archaeology is the one that reads prose with a model, and it is skipped on a+keyless index. Three further sources were retired for producing records nobody+ever acted on, which [`DECISIONS.md`](DECISIONS.md) names.```python# WHY: JWT chosen over sessions — API must be stateless for k8s horizontal scaling@@ -240,13 +245,22 @@ Reference: [`CHANGE_RISK.md`](CHANGE_RISK.md).## Test IntelligenceWhich tests actually exercise the code you changed, which changed files have no-guarding test at all, and per-file coverage merged across every test that-touches it. Coverage ingests from LCOV, Cobertura, Clover or normalized JSON and-feeds the code-health coverage markers.--The practical payoff is `tests_to_run` in `get_risk()` PR mode: a-coverage-backed list rather than a guess, so an agent runs the tests that can-actually catch its change.+guarding test at all, and per-file coverage merged across every test that touches+it. Coverage ingests from LCOV, Cobertura, Clover or normalized JSON and feeds the+code-health coverage markers.++**It answers with or without that ingest.** Most repositories never produce a+report, so where one is missing the layer walks the call graph instead: a test+whose calls reach a file executes it, three hops by default, filtered to the call+edges the graph is confident in. Measured against a real+`coverage run --contexts=test`, that reaches **95.7% precision** on what covers a+file and **97.5% on the run list**. Rows are stamped `basis: "measured"` or+`basis: "inferred"`, the measured tier wins outright wherever it can answer, the+two are never averaged, and the inferred tier may never emit a percentage.++The practical payoff is `tests_to_run` in `get_risk()` PR mode: a real list rather+than a guess, so an agent runs the tests that can actually catch its change, on+repositories that have never configured coverage at all.Reference: [`TEST_INTELLIGENCE.md`](TEST_INTELLIGENCE.md).@@ -289,8 +303,9 @@ repowise dead-code```Conservative by design. `safe_to_delete` requires confidence ≥ 0.70 and excludes-17 dynamically-loaded naming patterns (`*Plugin`, `*Handler`, `*Middleware`,-`register_*`, `on_*`, `*_route`, `*_callback` and more). Dynamic-import+14 dynamically-loaded naming patterns (`*Plugin`, `*Handler`, `*Adapter`,+`*Middleware`, `*Mixin`, `*Command`, `register_*`, `on_*`, `*_view`,+`*_endpoint`, `*_route`, `*_callback`, `*_signal`, `*_task`). Dynamic-importdetection and a per-language framework-convention registry further cut falsepositives. repowise surfaces candidates; engineers decide.diff --git a/docs/layers/LANGUAGE_SUPPORT.md b/docs/layers/LANGUAGE_SUPPORT.mdindex b44f7b67a..61fd34466 100644--- a/docs/layers/LANGUAGE_SUPPORT.md+++ b/docs/layers/LANGUAGE_SUPPORT.md@@ -1,9 +1,11 @@# Language Support-**19 languages parsed to a full AST · 13 at the Full tier · framework-aware-across all of them.** Everything else in your repo still appears in the wiki and-is tracked through git history. This page is the "what works for my language-today" reference.+**19 languages parsed to a full AST · 35 on the five-rung ladder ·+framework-aware across all of them.** "Do you support X" has five useful answers+rather than two, so every language lands on a rung and the rung says what it+buys you. Everything else in your repo still appears in the wiki and is tracked+through git history. This page is the "what works for my language today"+reference.<p><strong>Full tier </strong>@@ -49,11 +51,17 @@ produce meaningful output.| Tier | Languages | What you get ||------|-----------|--------------|-| **Full** | Python · TypeScript · JavaScript · Svelte · Vue · Java · Kotlin · Go · Rust · C++ · C# · Scala · Ruby | The whole pipeline: AST symbols, import resolution, a resolved call graph, heritage, docstrings, framework edges, **and code-health markers** |-| **Good** | C · Swift · PHP · Dart · Object Pascal | Everything above except the full health suite. Dart and Object Pascal *do* get health markers; C, Swift and PHP don't yet |-| **Partial** | Luau / Roblox | AST symbols and `require()` resolution (Rojo / `.luaurc` aware). No health markers yet |-| **Lightweight** | Elixir · Clojure · Haskell · Lean 4 · Erlang · F# · HTML | A real file-to-file import graph, no symbol-level claims |-| **Structural** | Objective-C · R · Zig · Julia · Elm · OCaml · Crystal · Nim · D | Git history only: blame, hotspots, co-change. No AST parsing |+| **Full** (13) | Python · TypeScript · JavaScript · Svelte · Vue · Java · Kotlin · Go · Rust · C++ · C# · Scala · Ruby | The whole pipeline: AST symbols, import resolution, a resolved call graph, heritage, docstrings, framework edges, **and code-health markers** |+| **Good** (5) | C · Swift · PHP · Dart · Object Pascal | Everything above except the full health suite. Dart and Object Pascal *do* get health markers; C, Swift and PHP don't yet |+| **Partial** (1) | Luau / Roblox | AST symbols and `require()` resolution (Rojo / `.luaurc` aware). No health markers yet |+| | | ⎯⎯ *tree-sitter parsing stops here. The rungs below are derived from git and imports, not from an AST.* ⎯⎯ |+| **Lightweight** (7) | Elixir · Clojure · Haskell · Lean 4 · Erlang · F# · HTML | A real file-to-file import graph, no symbol-level claims |+| **Structural** (9) | Objective-C · R · Zig · Julia · Elm · OCaml · Crystal · Nim · D | Git history only: blame, hotspots, co-change. No AST parsing |++The first three rungs are the **19 languages parsed to a full AST**; all five are+the **35** on the ladder. Both numbers are worth stating and neither is worth+stating alone, so if you only take one thing from this page, take the rung your+language sits on rather than either count.[**SQL / dbt**](#sql--dbt) and [**shell**](#shell) sit outside this ladder onpurpose: each has a coverage shape the tiers cannot describe. SQL is parsed bydiff --git a/docs/layers/TEST_INTELLIGENCE.md b/docs/layers/TEST_INTELLIGENCE.mdindex 1bde1f112..5de31161c 100644--- a/docs/layers/TEST_INTELLIGENCE.md+++ b/docs/layers/TEST_INTELLIGENCE.md@@ -242,7 +242,7 @@ matching filenames.Matching filenames fails both ways, and this repository is the proof. Of its sixworst bug-magnet files, five have no file named for them anywhere under `tests/`-and so read as untested, while the graph names the tests that import them:+and so read as untested, while the graph names the tests that reach them:| File | Filename convention | Graph ||---|---|---|@@ -257,57 +257,94 @@ The last row is the worse failure. The convention matches on basename alone, soit paired the health engine with the *distill* engine's tests (a differentsubsystem) and called the file tested on the strength of a name collision.-Across the whole index the effect is large: 74 of repowise's 110 standing-`untested_hotspot` findings are cleared once the graph is consulted, 18 of them-graded critical.+Across the whole index the effect is large. Against the **80** standing+`untested_hotspot` findings the filename convention leaves behind, the call graph+clears **32** of them, 22 high and 3 critical, where the one-hop import walk this+tier used to run clears 11.-The dependency graph already holds a weaker version of the same relation. A test-file that imports a source file **reaches** it. That is not proof it executes it,-and the layer never claims otherwise, but it is a recorded edge rather than a-name-shaped guess, and it needs no setup at all.+The graph already holds the relation. A test whose **calls reach** a source file+executes it, and that is a recorded edge rather than a name-shaped guess. It needs+no setup at all.### What it is allowed to claim| | Measured map | Inferred map ||---|---|---|-| Comes from | a coverage report you ingested | the import graph, already indexed |+| Comes from | a coverage report you ingested | the call graph, already indexed || Granularity | lines | files |-| Proves | this test executed these lines | this test imports this file |+| Proves | this test executed these lines | this test's calls reach this file || Decays | yes, see the coverage age report | no || May produce a percentage | yes | **never** || Labelled | `basis: "measured"` | `basis: "inferred"` |-The two are shown side by side and are never averaged into one number. Where-both can answer, the measured one wins outright and the graph is not consulted;-the inferred tier only fills silence. Its error is one-sided and known: importing-a module pulls in every symbol it defines while the test body may touch one-function, so it **over-claims**. That makes it sound as a floor ("something-reaches this, do not call it untested") and unsound as a quantity.--### How far it walks--One hop by default: a test importing the file. That default was measured, not-chosen, by dogfooding against a real `coverage run --contexts=test` on this-repository over a slice where per-test attribution was complete and both sides-saw the same test files:--| `max_depth` | reach precision | reach recall | run-list precision | run-list hit rate |-|---|---|---|---|---|-| **1** (default) | **72.1%** | 30.4% | **93.1%** | 96.8% |-| 2 | 40.7% | 45.1% | 73.8% | 97.9% |-| 3 | 17.1% | 45.1% | n/a | n/a |--Both uses want precision: a false "something reaches this" suppresses a real-untested-hotspot finding, and a false entry in a run-list sends someone to run a-test that cannot fail for their change. The extra recall a second hop buys is not-worth halving either, and by depth 3 the closure has begun to blow up: 269-claims for the same 46 confirmations.--Depth 1's recall already beats the filename convention it sits beside (30.4%-against 23.5% on the same set) while resting on an edge rather than a name, and-the two are unioned rather than ranked, so the shipped floor is better than-either alone. A repository whose tests import a package facade rather than the-module under test will want a second hop; it is a keyword argument on both walks.+The two are shown side by side and are never averaged into one number. Where both+can answer, the measured one wins outright and the graph is not consulted; the+inferred tier only fills silence. Its error is one-sided and known, so it is sound+as a floor ("something reaches this, do not call it untested") and unsound as a+quantity.++### The call graph leads, the import graph fills silence++The tier first shipped walking **import** edges one hop, which was the best of the+import-based options and still the wrong graph: a test that imports a module is+weak evidence it runs it. Release 0.44.0 made the call graph good enough to lead+with, so it does.++Dogfooded against a real `coverage run --contexts=test` over a slice where+per-test attribution is complete, both sides seeing the same 37 test files and 159+provably-executed production files:++**Forward, "what reaches this file" (suppresses `untested_hotspot`):**++| Walk | claims | correct | precision | recall |+|---|---:|---:|---:|---:|+| import graph, 1 hop (what shipped before) | 43 | 31 | 72.1% | 19.5% |+| call graph, 3 hops | 48 | 44 | 91.7% | 27.7% |+| **call graph, 3 hops, filtered** (ships) | 46 | 44 | **95.7%** | **27.7%** |+| both unioned | 57 | 45 | 78.9% | 28.3% |++**Reverse, "which tests do I run" (`tests_to_run`, `impacted_tests`):**++| Walk | targets | hit rate | precision |+|---|---:|---:|---:|+| import graph, 1 hop (what shipped before) | 32 | 96.9% | 94.8% |+| call graph, 3 hops, filtered | 46 | 100.0% | 97.5% |+| both unioned | 47 | 100.0% | 95.8% |+| **call graph, else import graph** (ships) | 47 | **100.0%** | **97.5%** |++The call graph wins on both axes, so it leads. **The two tiers are combined+differently per direction, because the measurements differ.** Unioning costs the+forward walk 16.8 points of precision for 0.6 of recall, and a false "something+reaches this" hides a real gap, so forward is call edges only. Reverse falls back+instead of unioning, spending the import tier only on targets the call graph left+silent, which answers one more target at identical precision.++**Depth 3 is where the call walk saturates**: 3, 4 and 5 hops return the same 48+claims and 44 confirmations, so the ceiling is the call graph's capture rate rather+than the depth. The import tier keeps its measured default of one hop.++**"Filtered" is `resolution_origin` doing work.** Not every call edge is equally+trustworthy, and this is the first thing to read the graph's own confidence+vocabulary: `global_unique`, which binds a name to the only symbol carrying it+repo-wide and which the vocabulary itself scores 0.50, is dropped. That buys **4.0+points of forward precision and 1.1 of reverse at no cost to recall**. A confidence+floor of 0.90 was tried instead and cut recall from 27.7% to 23.3%.++**Why recall reads low at 27.7%**, which is a property of the truth set rather than+the walk: `coverage` records a line as run whether a test called into it or Python+merely evaluated the module body on import, so the truth set cannot tell an+imported file from an exercised one. Splitting the 159 truth files by how much of+what ran was inside a function body: 39 ran nothing inside one at all (0% recall,+correctly), 19 ran under a quarter inside (0%), 45 a quarter to three quarters+(7%), and 56 over three quarters (**77%**). Of the 13 missed in that last group, 11+are alembic migrations the framework invokes by naming convention with no static+caller anywhere.++<sub>Reachability walks <code>EXECUTION_EDGE_TYPES</code>, which is the reachability+view minus <code>references</code> and <code>reads</code>: those record a mention+rather than a transfer of control. Dead code asks "is this used", which a mention+answers; this asks "is this run", which it does not. Both depths are keyword+arguments on the walks.</sub>### Nothing is stored