Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.
Python aicode-complexitycode-healthcode-intelligencecode-qualitydead-codedeveloper-toolsdocumentationgit-analyticsmcpopen-sourcepythonrefactoringstatic-analysistechnical-debt
docs: publish the measured results, and compare against real peers (#1287)
Raghav Chamadiya6e640787488e
diff --git a/README.md b/README.mdindex 4eef349bd..411c6897e 100644--- a/README.md+++ b/README.md@@ -34,7 +34,8 @@<a href="#past-one-repo">Workspaces</a> ·<a href="#quickstart-under-5-minutes-no-api-key">Quickstart</a> ·<a href="#the-ten-mcp-tools">MCP tools</a> ·- <a href="#how-it-compares">Comparison</a> ·+ <a href="#measured-against-the-field">Benchmarks</a> ·+ <a href="#how-it-compares-on-capability">Comparison</a> ·<a href="#for-teams--enterprises">Teams</a></sub></p>@@ -507,32 +508,90 @@ opt-in tools, and the full reference: **[docs/agent/MCP_TOOLS.md →](docs/agent----## How it compares--| | repowise | Google Code Wiki | DeepWiki | Swimm | CodeScene |-|---|---|---|---|---|---|-| Self-hostable, open source | ✅ AGPL-3.0 | ❌ cloud only | ❌ cloud only | ❌ Enterprise only | ✅ Docker |-| Private repo, no cloud | ✅ | ❌ in development | ❌ OSS forks only | ✅ Enterprise tier | ✅ |-| Auto-generated documentation | ✅ | ✅ Gemini | ✅ | ✅ PR2Doc | ❌ |-| MCP server for AI agents | ✅ 10 tools | ❌ | ✅ 3 tools | ✅ | ✅ |-| Proactive agent hooks | ✅ Claude + Codex | ❌ | ❌ | ❌ | ❌ |-| Auto-generated AI instructions (`CLAUDE.md`, `AGENTS.md`) | ✅ | ❌ | ❌ | ❌ | ❌ |-| Command-output distillation | ✅ reversible | ❌ | ❌ | ❌ | ❌ |-| Learns from your usage (session-mined decisions, demand-weighted docs) | ✅ | ❌ | ❌ | ❌ | ❌ |-| Code health score (1-10) | ✅ 25 markers | ❌ | ❌ | ❌ | ✅ 25-30 |-| Brain Method / LCOM4 / god class | ✅ | ❌ | ❌ | ❌ | ✅ |-| Test-coverage intelligence | ✅ LCOV/Cobertura/Clover | ❌ | ❌ | ❌ | ❌ |-| Untested-hotspot detection | ✅ coverage × hotspot | ❌ | ❌ | ❌ | ❌ |-| Health trend + declining alerts | ✅ rolling snapshots | ❌ | ❌ | ❌ | ✅ |-| Concrete cross-file refactoring plans | ✅ graph-aware + blast radius | ❌ | ❌ | ❌ | ⚠️ within-function only |-| Dataflow-verified within-function plans | ✅ CFG + reaching definitions | ❌ | ❌ | ❌ | ⚠️ LLM-generated, unverified |-| Git intelligence (hotspots, ownership, co-change) | ✅ | ❌ | ❌ | ❌ | ✅ |-| Pre-merge change-risk scoring | ✅ 0-10 + directives | ❌ | ❌ | ❌ | ✅ |-| Bus factor analysis | ✅ | ❌ | ❌ | ❌ | ✅ |-| Dead code detection | ✅ | ❌ | ❌ | ❌ | ❌ |-| Architectural decision records | ✅ | ❌ | ❌ | ❌ | ❌ |-| Multi-repo workspace intelligence | ✅ contracts, co-change, federated MCP | ❌ | ❌ | ❌ | ❌ |-| Local dashboard | ✅ | ❌ | ❌ | ❌ IDE only | ✅ |+## Measured against the field++Three numbers, each with its sample size and its test. The tools here are the+open-source agent-context tools we ran head to head, and the full page carries+the rows we lose beside the rows we win.++- **Finds the right files.** 0.746 file coverage against CodeGraph's 0.610 on a+ sealed 42-instance split evaluated once, while serving 8.1 files against 14.0.+ Deterministic grading, no LLM judge. *n=42, sign test p=0.021.*+- **Cheaper in a real agent loop.** -33.5% cost per question against a bare+ agent, cheaper on 13 of 15 questions. *n=15, p=0.007.*+- **Agents actually call it.** Adopted in 15 of 15 cells, against CodeGraph 13,+ Serena 4, Graphify 3, and code-review-graph 0 with 30 tools advertised.++**[The full results, the methodology, and the rows we lose →](docs/BENCHMARKS.md)**++---++## How it compares on capability++No single product competes with all of this, so there is no single table. Three+axes, three sets of real peers. Rows marked *measured* are head-to-head numbers,+and they link to **[docs/BENCHMARKS.md](docs/BENCHMARKS.md)** where the sample+sizes, the tests and the rows we lose all live.++### As an agent context layer++Against the tools doing the same job: index a repository, serve it to a coding+agent over MCP.++| | repowise | CodeGraph | Serena | DeepWiki |+|---|---|---|---|---|+| Self-hostable, open source | ✅ AGPL-3.0 | ✅ | ✅ | ❌ cloud only |+| Private repo, no cloud | ✅ | ✅ | ✅ | ❌ OSS forks only |+| MCP tools served | 10 | 1 | 29 | 3 |+| **Finds the gold files** *([measured](docs/BENCHMARKS.md#1-finding-the-right-files), n=42)* | ✅ **0.746** | 0.610 | not in this run | not measured |+| **The agent actually calls it** *([measured](docs/BENCHMARKS.md#3-whether-agents-call-the-tools-at-all), n=15)* | ✅ **15/15** | 13/15 | 4/15 | not measured |+| **Index time, django** *([measured](docs/BENCHMARKS.md#6-indexing-time-the-row-we-lose))* | ⚠️ **366.8s**, slowest here | ✅ **16.4s** | not measured | n/a, cloud |+| Generated documentation | ✅ | ❌ | ❌ | ✅ |+| Proactive agent hooks | ✅ Claude + Codex | ❌ | ❌ | ❌ |+| Auto-generated AI instructions (`CLAUDE.md`, `AGENTS.md`) | ✅ | ❌ | ❌ | ❌ |+| Command-output distillation | ✅ reversible | ❌ | ❌ | ❌ |+| Learns from your usage (session-mined decisions, demand-weighted docs) | ✅ | ❌ | ❌ | ❌ |+| Architectural decision records | ✅ | ❌ | ❌ | ❌ |+| Multi-repo workspace intelligence | ✅ contracts, co-change, federated MCP | ❌ | ❌ | ❌ |++CodeGraph builds its index **22x faster than we do**, and if a call graph is all+you need, that is the right trade. Graphify and code-review-graph were in the same+measured field and are on the benchmarks page.++### As a code health tool++| | repowise | CodeScene |+|---|---|---|+| Self-hostable, open source | ✅ AGPL-3.0 | ✅ Docker |+| Code health score (1-10) | ✅ 25 markers | ✅ 25-30 |+| Brain Method / LCOM4 / god class | ✅ | ✅ |+| **Defects found at a 20% review budget** *([measured](docs/BENCHMARKS.md#5-code-health-predicts-defects), 2,770 files)* | ✅ **0.173** | 0.074 |+| **Effort-aware ranking, Popt** *(measured, p=0.003)* | ✅ **0.607** | 0.462 |+| **Precision at that budget** *(measured)* | 0.580 | ✅ **0.636**, a shorter list |+| Defect-prediction AUC, published and reproducible | ✅ 0.74 over 21 repos | ✅ Code Red study |+| Git intelligence (hotspots, ownership, co-change) | ✅ | ✅ |+| Pre-merge change-risk scoring | ✅ 0-10 + directives | ✅ |+| Health trend + declining alerts | ✅ rolling snapshots | ✅ |+| Bus factor analysis | ✅ | ✅ |+| Concrete cross-file refactoring plans | ✅ graph-aware + blast radius | ⚠️ within-function only |+| Dataflow-verified within-function plans | ✅ CFG + reaching definitions | ⚠️ LLM-generated, unverified |+| Test-coverage intelligence | ✅ LCOV/Cobertura/Clover | ❌ |+| Untested-hotspot detection | ✅ coverage × hotspot | ❌ |+| Dead code detection | ✅ | ❌ |+| Serves it to an AI agent over MCP | ✅ | ✅ |+| Local dashboard | ✅ | ✅ |++CodeScene is the only other vendor in this category with a published empirical+defect study, which is why it is the one we ran head to head against. It flags+about 27 files where we flag 132, so if you want a short list to action rather+than the ranking that catches the most defects, its threshold is the better fit.++### Documentation generators++DeepWiki, Google Code Wiki and Swimm generate documentation from a repository,+which overlaps one of our five layers. **We have not measured against them**, so+there is no table here rather than a table of checkmarks. DeepWiki appears above+because it also serves an agent over MCP, which is a job we can be measured on.### The PR bot, against the LLM review botsdiff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.mdindex 1b203c49f..aab7eed28 100644--- a/docs/BENCHMARKS.md+++ b/docs/BENCHMARKS.md@@ -1,292 +1,299 @@# Benchmarks-Every number repowise publishes, with the method behind it and the limits it-carries. Three studies:+Every number repowise publishes, with its sample size, its test, and a link to+the raw data. The harnesses and full reports live in+**[repowise-bench](https://github.com/repowise-dev/repowise-bench)**, and nothing+here is measured on a private corpus.++Three headline claims in this category collapsed when someone else reran them:+RTK's 60-90% became 7.6% *worse*, Caveman's 65% became 8.5% under JetBrains,+Greptile's 82% became 45% under Augment's rerun. That is the reason this page+prints n beside every mean, states which tool produced each number, and publishes+the rows we lose. It is built to survive a rerun, because in this field that is+the only property that turns out to matter.++## What we measured, and against whom++repowise builds [five intelligence layers](layers/INTELLIGENCE_LAYERS.md) from+one index, so there is no single competitor to measure against. Different tools+overlap on different layers, and this table says exactly which.++| Layer | Measured against | Result |+|---|---|---|+| Finding the right files | CodeGraph, Graphify, code-review-graph | [§1](#1-finding-the-right-files) **we win**, n=42, p=0.021 |+| Cost in a real agent loop | CodeGraph, Serena, Graphify, code-review-graph, bare agent | [§2](#2-cost-in-a-real-agent-loop) **we win**, n=15, p=0.007 |+| Whether agents call the tools at all | same five | [§3](#3-whether-agents-call-the-tools-at-all) **we win**, 15/15 |+| Command-output compression | no comparable tool in the field | [§4](#4-command-output-compression) |+| Code health and defect prediction | CodeScene | [§5](#5-code-health-predicts-defects) **we win**, p=0.003 |+| Indexing time | CodeGraph, Graphify, code-review-graph | [§6](#6-indexing-time-the-row-we-lose) **we lose**, 22x |+| Documentation generation | DeepWiki, Google Code Wiki, Swimm | **not measured** |+| PR review | CodeRabbit, Greptile | **not measured** |++The last two rows are capability comparisons, not measurements, and they live in+the [README's feature table](../README.md#how-it-compares-on-capability) where a+reader can tell the difference. We would rather say "not measured" than let a+checkmark do a number's job.-1. [Agent efficiency](#1-agent-efficiency) (context, file reads, tool calls)-2. [Distill](#2-distill-command-output-compression) (command-output compression)-3. [Code health predicts defects](#3-code-health-predicts-defects) (defect ranking, plus a CodeScene head-to-head)+---++## 1. Finding the right files++**Grading here is deterministic. No LLM judge is involved anywhere in this+number**, which makes it the most reproducible result on the page. ContextBench+ships gold file spans; a tool either returns them or it does not.++The 112 instances were split into a 70-instance development half and a+42-instance sealed half, **pinned by instance id before any tuning work started**.+The sealed half was evaluated once, at publication. That protocol is the whole+reason the number below is worth reading.++| Tool | File coverage | n | Precision | Files served |+|---|---:|---:|---:|---:|+| **repowise** (`search_codebase`) | **0.746** | 42 | **0.168** | **8.1** |+| CodeGraph | 0.610 | 42 | 0.093 | 14.0 |+| repowise (`get_answer`) | 0.597 | 40 | 0.113 | 12.1 |+| Graphify | 0.546 | 42 | 0.033 | 34.5 |+| code-review-graph | 0.445 | 42 | 0.240 | 5.4 |++We find more of the right files while serving **fewer files overall**, which is+the combination that matters for an agent paying by the token. Head to head+against CodeGraph per instance: **13 wins, 3 losses, 26 ties, sign test+p = 0.021**.-The harnesses, raw run logs, and full reports live in-**[repowise-bench](https://github.com/repowise-dev/repowise-bench)**. Nothing-here is measured on a private corpus: every study runs on public codebases so-you can re-run it.+**Our synthesis tool ties CodeGraph.** `get_answer` scores 0.597 on the same+sealed half, an honest tie (11-10-19, p = 1.000). Anyone reading the table would+spot it, so we would rather say it: the retrieval win belongs to+`search_codebase`, and `get_answer` is where our own next round of work is aimed.-Two ground rules for how we report:+This is retrieval, not task success. It says we find the right files, not that an+agent using us writes better code.-- **The limits ship with the wins.** Each study has a "What this does not show"- section, and it is the section we would want to read first if someone else- published these numbers.-- **Token savings are not automatically dollar savings.** Agent-side prompt- caching now mutes the cost delta on repeated context, even where token counts- drop sharply. We report tokens, reads, and calls because those are what the- measurements actually establish.+Raw data and harness:+**[bakeoff\_2026\_08/rung8](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08/rung8)**----## 1 · Agent efficiency+## 2. Cost in a real agent loop-Most of a coding agent's spend goes to exploration: greping for symbols, reading-candidate files, re-reading them as the context window fills. repowise does that-work once, offline, so the agent skips it on every query.+Fifteen questions on `django/django`, stratified across five question shapes,+drawn and pre-registered before any money was spent. Every arm got a+byte-identical prompt and its full advertised tool surface, and the bare-agent+control was verified free of the operator's own hooks.-Paired SWE-QA runs on real repositories, same model and same harness, with and+| Arm | Cost per question | vs bare agent | Cheaper on | n |+|---|---:|---:|---:|---:|+| **repowise** | **$0.2068** | **-33.5%** | **13 of 15** | 15 |+| CodeGraph | | -27.0% | | 15 |++**p = 0.007**, sign test, two-sided. This is the cost claim, and it is the only+significant result in the run.++Two things we will not claim from it:++- **Not quality parity.** repowise scored +0.13 on the judge and CodeGraph+ -0.48, but the judge's two graders disagree by 0.46 points on the *same*+ answers, which is larger than every per-arm effect in the run. The quality+ column is inside the instrument's noise, and "no significant difference" is not+ parity. An equivalence claim needs a TOST that we have not run.+- **Not a universal saving.** This is n=15 on one repository, at one commit,+ under one prompt and one model. Cost deltas move with all four, and other runs+ in [repowise-bench](https://github.com/repowise-dev/repowise-bench) land+ elsewhere. Treat -33.5% as this configuration's result, not a constant.++### What the agent stops doing++The mechanism behind the saving is context substitution: work done once, offline,+that the agent would otherwise redo on every query. Measured over paired SWE-QA+runs on `pallets/flask` and `scikit-learn`, same model and same harness, with andwithout repowise's MCP tools:| Measure | Result ||---|---|-| Tokens to load context | up to **−96%** |-| File reads | **−69% to −89%** |-| Tool calls | **−49% to −70%** |-| Answer quality | at parity with raw exploration |+| Tokens to load context | up to **-96%** |+| File reads | **-69% to -89%** |+| Tool calls | **-49% to -70%** |-The mechanism is context substitution, not a smaller model or a shorter answer.Loading one commit's context through `get_context` costs **2,391 tokens against-64,039** raw, roughly 27x fewer. On a long multi-step investigation the effect-compounds: **−41% of the context re-read across the whole session**, because the-agent is not re-reading files it already saw to recover a detail.+64,039** read raw, roughly 27x fewer, and over a long investigation the effect+compounds to **-41% of the context re-read across the session**.++These are token and call counts, and they are not the same as dollars: agent-side+prompt caching mutes the cost delta on repeated context even where token counts+drop sharply. We report what these runs establish, which is the exploration the+agent no longer performs.Reports:-[flask48](https://github.com/repowise-dev/repowise-bench/blob/master/BENCHMARK_REPORT_FLASK48.md) ·-[flask v3](https://github.com/repowise-dev/repowise-bench/blob/master/BENCHMARK_REPORT_FLASK_V3.md) ·-[sklearn48](https://github.com/repowise-dev/repowise-bench/blob/master/BENCHMARK_REPORT_SKLEARN48.md)--An earlier cut of the same paired setup is quoted in-[COMMERCIAL.md](business/COMMERCIAL.md) as −36% cost / −49% tool calls on-`pallets/flask` and −29% cost / −70% tool calls on `scikit-learn`, both at-parity answer quality.--### How to reproduce--1. Clone [repowise-bench](https://github.com/repowise-dev/repowise-bench) and- the target repository (`pallets/flask` or `scikit-learn`) at the pinned- commit named in the report you want to reproduce.-2. Index the target: `repowise init <path>`.-3. Run the harness twice over the same SWE-QA question set with the same model:- once with the repowise MCP server registered, once without. Each report names- its own question set, model, and pinned commit.-4. Compare the per-run token, file-read, and tool-call totals the harness emits.- The answer-quality check is a graded comparison of the two answer sets, also- described in the report.--The `get_context` figure is the cheapest thing to check on your own repo: run-`get_context` on a commit and compare the response size against the raw bytes of-the files it covers.--### What this does not show--- **Not a SWE-bench-style task-completion result.** These runs measure the cost- and shape of answering questions about a codebase, not the rate at which an- agent lands a correct patch.-- **Answer quality is "at parity", not "better".** The claim is that the agent- reaches the same quality of answer for far less exploration. If a benchmark- showed a quality gain, we would report a quality gain.-- **The ranges are wide because the repos differ.** "up to −96%" is a ceiling- observed on a specific context-loading comparison, not the average case. Read- the per-repo reports for the distribution.-- **Prompt caching changes the economics.** With agent-side caching, a large- repeated context can be cheap in dollars even when it is expensive in tokens.- Token reduction still buys you context-window headroom and fewer round trips;- it no longer buys a proportional dollar saving.-- **Same model, same harness, one vendor's agent.** We have not shown the effect- transfers unchanged to every agent framework.+**[flask48](https://github.com/repowise-dev/repowise-bench/blob/master/BENCHMARK_REPORT_FLASK48.md)** ·+**[flask v3](https://github.com/repowise-dev/repowise-bench/blob/master/BENCHMARK_REPORT_FLASK_V3.md)** ·+**[sklearn48](https://github.com/repowise-dev/repowise-bench/blob/master/BENCHMARK_REPORT_SKLEARN48.md)**++Raw data for the stratified run:+**[bakeoff\_2026\_08/rung6](https://github.com/repowise-dev/repowise-bench/tree/master/results/bakeoff_2026_08/rung6)**----## 2 · Distill: command-output compression+## 3. Whether agents call the tools at all-`repowise distill <cmd>` compresses command output *before* the agent reads it:-errors first, exit code preserved, and every omission recoverable through an-inline `[repowise#<ref>]` marker (`repowise expand <ref>`).+Same 15 questions, same agent, same neutral prompt, every server verified alive+and serving its full advertised surface. This counts the cells where the agent+issued at least one call the server answered.++| Tool | Tools advertised | Schema cost (chars) | Cells adopted |+|---|---:|---:|---:|+| **repowise** | 10 | 17,561 | **15 / 15** |+| CodeGraph | 1 | 1,567 | 13 / 15 |+| Serena | 29 | 29,050 | 4 / 15 |+| Graphify | 10 | 5,482 | 3 / 15 |+| code-review-graph | 30 | 28,118 | **0 / 15** |++Tool counts are the surface each server advertises. Schema cost is measured on+the exact build used in the run.++code-review-graph advertises 30 tools over a built, embedded graph of 40,904+nodes and 380,168 edges, and the agent never called it once. A capability an+agent does not reach for is not a capability.++**Our own caveat, because it belongs to us to say:** this is an advantage of+naming and surface design, not of retrieval quality, and adoption is clearly+**not** ordered by surface size. We serve 10 tools and get called 15/15;+CodeGraph serves 1 and gets called 13/15; Serena serves 29 and gets called 4/15.+Designing tools an agent picks up is a real skill and this table measures it, but+it measures nothing about what comes back.-Paired runs on a public OSS repository (microdot), one run per command, tokens-estimated at chars/4, which is the same estimator the savings ledger uses:+---++## 4. Command-output compression++`repowise distill <cmd>` compresses command output *before* the agent reads it:+errors first, exit code preserved, every omission recoverable through an inline+`[repowise#<ref>]` marker.| Command | Raw tokens | Distilled | Saved ||---|---:|---:|---|-| `pytest -q` (11 failures) | 3,374 | 1,317 | **61%**, all 11 `FAILED` lines preserved |+| `pytest -q` (11 failures) | 3,374 | 1,317 | **61%**, all 11 `FAILED` lines kept || `git log -50` | 3,064 | 331 | **89%** |-| `git diff` (30 commits of history) | 62,833 | 8,635 | **86%** |-| `git log --oneline -30` | 321 | 321 | 0%, already compact, passed through |-| `git status` (clean tree) | 83 | 83 | 0%, too small to distill, passed through |--The two 0% rows are the net-positive guard doing its job: distill never inflates-small output. In an end-to-end spot-check on the same repo with a seeded-11-failure bug, the agent reached the identical root-cause line and fix from-distilled output as from raw. Across the fixture suite, the core filters hold a-median of at least 60% reduction on test/build/lint output with zero error-line-loss, asserted in CI.--Full guide: **[docs/agent/DISTILL.md](agent/DISTILL.md)**.--### How to reproduce--1. Clone microdot (or any repo with a test suite) and index it with- `repowise init`.-2. For each command, capture the raw output and the distilled output:-- ```bash- pytest -q > raw.txt 2>&1- repowise distill pytest -q > distilled.txt 2>&1- ```--3. Compare sizes with the chars/4 estimator. `repowise saved` reports the same- accounting cumulatively across a session.-4. To reproduce the failure-preservation claim, diff the `FAILED` lines in- `raw.txt` against `distilled.txt`. They should match exactly.--### What this does not show--- **One run per command, one repository.** These are point measurements, not a- distribution over many repos with confidence intervals. Your ratios will- differ with your test suite's verbosity and your diff sizes.-- **Reduction is not comprehension.** The 61/89/86% figures measure bytes- removed. The evidence that the removal is safe is narrower: preserved failure- lines, the CI-asserted zero-error-line-loss fixtures, and a single end-to-end- agent spot-check. One spot-check is an existence proof, not a rate.-- **`git diff` compresses hardest because diffs are the most redundant input.**- Do not read 86% as the expected saving on arbitrary commands.-- **Token estimation is chars/4, not a tokenizer.** It is consistent across the- raw and distilled sides, so the ratio is sound, but the absolute counts are- approximations.-- **Dollar savings again depend on caching.** `repowise saved` prices tokens at- your agent's model rate; a cached context reduces the real-world delta.+| `git diff` (30 commits) | 62,833 | 8,635 | **86%** |+| `git log --oneline -30` | 321 | 321 | 0%, already compact |+| `git status` (clean) | 83 | 83 | 0%, too small to distill |++The two 0% rows are the net-positive guard working: distill never inflates small+output. One run per command on one repository, so these are point measurements,+not a distribution. Reduction is also not comprehension: the bytes removed are+measured, and the evidence they were safe to remove is narrower, being preserved+failure lines plus CI-asserted zero-error-line-loss fixtures.++Full guide: **[docs/agent/DISTILL.md](agent/DISTILL.md)**----## 3 · Code health predicts defects--The health score is worth something only if the files it flags are the files-that actually break. Scores are collected at a historical commit (T0),-bug-fixing commits are counted over the following six months, and the two are-correlated with strictly no leakage: nothing after T0 feeds the score.--### Cross-project validation--Across **21 open-source repositories spanning 9 languages** and 2,826 files-(the study predates the promotion of Scala and Ruby to the Full tier, so it-covers 9 of today's 11):--- **Mean ROC AUC 0.74** (95% CI 0.68 to 0.79) at identifying the files that go- on to receive bug fixes, reaching **0.90** on individual repos. ROC AUC is the- probability the score ranks a known-buggy file worse than a clean one: 0.5 is- a coin flip, 1.0 is perfect.-- **Survives controlling for file size**: partial Spearman rho = −0.16, so the- signal is not simply "flag the big files".-- **Out-discriminates the obvious baselines**: +0.10 AUC over recent churn and- +0.12 AUC over prior-defect history, DeLong p < 1e-9.-- **Holds on an external dataset it never saw**: PROMISE/jEdit CK-metrics, AUC- **0.76 to 0.78**, within about 0.03 of that dataset's own tuned model. This- held-out result is the main evidence the markers are not overfit to the- calibration corpus.--Full report:-**[health-defect/BENCHMARK_REPORT.md](https://github.com/repowise-dev/repowise-bench/blob/master/health-defect/BENCHMARK_REPORT.md)**--### CodeScene head-to-head--CodeScene is the closest commercial product on code health and the only vendor-in the category with a published empirical defect study (the "Code Red"-correlation study, Tornhill and Borg, TechDebt 2022). Both tools were run over-the **same 2,770 files across 9 languages**, scored at the same leakage-free-commit against the same defect labels:--| Axis (paired tests) | repowise | CodeScene |-|---|---:|---:|-| Recall at a 20%-of-lines review budget | **0.173** | 0.074 |-| Effort-aware ranking (Popt) | **0.607** | 0.462 |-| Defect density, size-normalized (defects/KLOC, Alert:Healthy) | **2.18x** | 0.56x |+## 5. Code health predicts defects++A health score is worth something only if the files it flags are the files that+break. Scores are taken at a historical commit, bug fixes are counted over the+following six months, and nothing after the scoring commit feeds the score.++Across **21 repositories, 9 languages, 2,826 files**: **ROC AUC 0.74**+(95% CI 0.68 to 0.79), reaching 0.90 on individual repos. It survives controlling+for file size, so it is not simply flagging the big files, and it beats recent+churn by +0.10 AUC and prior-defect history by +0.12 (DeLong p < 1e-9). On+PROMISE/jEdit, a dataset it never saw, it holds at 0.76 to 0.78.++**Against CodeScene**, the closest commercial product and the only other vendor+in this category with a published empirical defect study. Both tools scored the+same 2,770 files at the same leakage-free commit against the same labels:++| Paired test | repowise | CodeScene | |+|---|---:|---:|---|+| Recall at a 20%-of-lines review budget | **0.173** | 0.074 | p = 0.003 |+| Effort-aware ranking (Popt) | **0.607** | 0.462 | p = 0.003 |+| Defect density, size-normalized (Alert:Healthy) | **2.18x** | 0.56x | p = 0.003 |+| Discrimination (ROC AUC) | 0.731 | 0.705 | p = 0.054, marginal |+| Precision at a 20%-of-lines review budget | 0.580 | **0.636** | p = 0.64, a tie |Ranking by repowise health surfaces **2.3x the defects under a fixed review-budget**. The deltas are Popt +0.144 and recall +0.098, both p = 0.003, paired-and significant.--Full methodology and confidence intervals:-**[health-defect/COMPARISON_REPORT.md](https://github.com/repowise-dev/repowise-bench/blob/master/health-defect/COMPARISON_REPORT.md)**.--### The per-repo self-check--Separate from the cross-project study, every index prints a check against your-own history:--```-Does the score find the bugs? 16/20 lowest-health files had a bug fix in the-last 6 months, 3.3x the 24% baseline (80% vs 24%).-```--Agents can read the same block over MCP with `get_health(include=["accuracy"])`.-It stays silent on repos with too little history to be honest (fewer than 25-scored files, or fewer than 5 recently-fixed files). Details in-[docs/layers/CODE_HEALTH.md](layers/CODE_HEALTH.md#does-the-score-find-the-bugs).--### How to reproduce--1. Clone [repowise-bench](https://github.com/repowise-dev/repowise-bench) and- open `health-defect/`. The corpus list, the T0 commit per repo, and the- bug-window definition are all pinned there.-2. The harness checks out each repo at T0, runs the health pass, then labels- files from `fix:` commits in the following six months and computes ROC AUC,- Popt, recall@20%, and the DeLong comparisons against the churn and- prior-defect baselines.-3. The PROMISE/jEdit arm needs no repowise index at all: it scores the published- CK-metrics dataset with the same marker weights.-4. The CodeScene arm requires a CodeScene account. The comparison is restricted- to the 2,770 files both tools scored, so it can be re-derived from the two- exported score sets plus the shared label file.-5. For the per-repo self-check, just run `repowise init` then `repowise health`- on any repo with enough history.--### What this does not show--- **The per-repo callout is an association, not a forward prediction.**- `prior_defect` is itself one (down-weighted) input to the score, so the- "16/20 lowest-health files" line is measured on the indexed history. The- cross-project study is the leakage-free one.-- **Within a size band the signal is weak.** Among files of similar size the AUC- sits near 0.49, so part of the headline number is that larger files carry more- risk. We report this because it is the most important caveat on the AUC.-- **A prior-defects baseline still wins on effort.** Under a fixed review budget- that baseline finds bugs slightly more efficiently than the repowise score- (Popt by 0.085), even though the score out-discriminates it on AUC.-- **The CodeScene comparison is scoped.** It covers 2,770 shared files at one- leakage-free commit. The significant wins are Popt, recall@20%, and defect- density; the ROC AUC edge is marginal (+0.026, p = 0.054) and precision@20% is- a tie, not a win. The operating points also differ: CodeScene flags about 27- files where repowise flags 132, a more conservative threshold, which is why- its precision reads higher. This is not an unqualified "better than- CodeScene", and it says nothing about CodeScene's breadth (28+ languages,- knowledge maps, off-boarding simulation).-- **"Bug fix" means a `fix:` commit touching the file.** That is a proxy for a- defect, and it inherits every quirk of the corpus repos' commit hygiene.-- **Maintainability and performance are not validated this way.**- Maintainability weights are expert-set and the performance signal is- high-precision, low-recall and advisory. Only the defect pillar carries these- numbers, which is exactly why repowise refuses to blend the three into one- headline score.--### Related: static performance risk--The performance pillar has its own, separate benchmark. On a 12,000-file corpus,-standard linters (clippy, ruff `PERF`, ESLint, golangci-lint) found **0** of the-cross-function I/O-in-loop cases, while repowise surfaced 557 findings, about 90-of them spanning function boundaries, with 98% in categories ruff has no rule-for. Findings are ordered by impact rather than raw count (NDCG 0.755 against-0.292 for severity-only). Hand-labeled `io_in_loop` precision on an 11-repo OSS-corpus: Go 96.7%, TypeScript 100%, Python 96.2%. One caveat travels with it: the-Rust dialect was new when the benchmark ran and clippy could not be built-end-to-end on the corpus under Windows, so the Rust comparison is-catalogue-level, not a measured head-to-head. Data and method:-[perf-detection](https://github.com/repowise-dev/repowise-bench/tree/master/perf-detection).+budget**, at Popt +0.144 and recall +0.098, both p = 0.003, paired.++**Where CodeScene is ahead, and why it is a real choice.** Its nominal precision+lead is not statistically significant, but the behaviour behind it is worth+having: CodeScene flags about **27 files** where we flag **132**. That is a more+conservative threshold, trading recall for a short list a team will actually work+through. If what you want is a handful of files to fix this quarter rather than+the ranking that catches the most defects, that operating point is the better+one, and it is a deliberate design choice rather than a weaker model. Our AUC+edge is also **marginal**, not significant at 0.05, and we would rather say so+than round it up.++Reports:+**[BENCHMARK\_REPORT.md](https://github.com/repowise-dev/repowise-bench/blob/master/health-defect/BENCHMARK_REPORT.md)** ·+**[COMPARISON\_REPORT.md](https://github.com/repowise-dev/repowise-bench/blob/master/health-defect/COMPARISON_REPORT.md)**++---++## 6. Indexing time, the row we lose++We are the slowest indexer in the field, on every repo we measured, and it is not+close. On `django/django`:++| Tool | Index time | What it builds |+|---|---:|---|+| CodeGraph | 16.4s | call graph |+| code-review-graph | 44.8s | call graph |+| Graphify | 141.5s | call graph, communities |+| **repowise** (`--no-prose`) | **366.8s** | five layers, below |+| **repowise** (default, prose on) | **1,058s** | five layers plus generated documentation |++That is **22x** CodeGraph like for like, and **135x** with prose on, which is what+a default `repowise init` actually costs you. Both numbers ship, and the 22x is+not the user-facing one.++**Here is what the extra time buys.** The tools above build a call graph. In the+same run, repowise builds five layers over the same codebase:++| Layer | On django, in that run |+|---|---|+| **Graph** | 36,485 nodes, 90,477 edges, 31,384 symbols, plus PageRank, betweenness, Leiden communities and execution-flow tracing |+| **Git** | full history mined across 2,630 files: hotspots, ownership, co-change pairs, bus factor |+| **Documentation** | 3,392 wiki pages rendered and embedded for natural-language search |+| **Decisions** | architectural decision records mined from history and sessions |+| **Code health** | 5,317 findings, plus 155 unreachable files and 98 unused exports |++So "22x slower" and "the index contains categorically more" are both true, and+neither one cancels the other. If all you want is a call graph, CodeGraph builds+one in 16 seconds and you should use it. The comparison that would be dishonest+is quoting the ratio without the column beside it, which is why the column is+here.++It is also a one-time cost. Updates after the first index are incremental.---+## Limits++Beyond the ones stated in each section:++- **Python and Go only.** No TypeScript or JavaScript row appears anywhere on+ this page. That was a scope choice made for instance density in the benchmark+ corpus, not a statement about language support.+- **§2 and §3 are one repository**, `django/django` at one commit, which is in+ every model's training data.+- **§1's development half is not a headline.** Pooling the development and sealed+ halves gives a stronger p, and we do not quote it, because the development half+ is the set the work was built against.++## Method and provenance++The full methodology, the pre-registration files with their commit timestamps,+the arm-parity rules, the statistical tests, and the list of measurement traps+that produced wrong numbers before we caught them all live in+**[repowise-bench](https://github.com/repowise-dev/repowise-bench)**. That+repository also holds every raw run, kept permanently, including the invalidated+ones with their invalidation notes attached.++Tool versions as measured: CodeGraph 1.5.0, Graphify 0.9.31, Serena 1.6.2.dev0,+code-review-graph 2.3.7.+## See also-- [docs/layers/CODE_HEALTH.md](layers/CODE_HEALTH.md): the score, the 25- markers, the bands, and the validation section in full.-- [docs/agent/DISTILL.md](agent/DISTILL.md): distill filters, the omission- store, and the hook.-- [docs/agent/MCP_TOOLS.md](agent/MCP_TOOLS.md): the tools the agent-efficiency- study exercises.-- [repowise-bench](https://github.com/repowise-dev/repowise-bench): harnesses,- raw logs, and every full report.+- [The five intelligence layers](layers/INTELLIGENCE_LAYERS.md)+- [Code health methodology](layers/CODE_HEALTH.md)+- [MCP tool reference](agent/MCP_TOOLS.md)diff --git a/docs/business/COMMERCIAL.md b/docs/business/COMMERCIAL.mdindex 4377123ec..5cea23ee8 100644--- a/docs/business/COMMERCIAL.md+++ b/docs/business/COMMERCIAL.md@@ -51,10 +51,11 @@ All of the following ship in `pip install repowise` today, free for internal usealerts).- **Ten task-shaped MCP tools** — `get_overview`, `get_answer`, `get_context`,`get_symbol`, `search_codebase`, `get_risk`, `get_change_risk`, `get_why`,- `get_dead_code`, `get_health`. Benchmarked at **−36 % cost / −49 % tool calls**- on `pallets/flask`- and **−29 % cost / −70 % tool calls** on `scikit-learn` versus a strong baseline- agent, at parity answer quality — see [repowise-bench](https://github.com/repowise-dev/repowise-bench).+ `get_dead_code`, `get_health`. Benchmarked at **−33.5 % cost per question**+ against a bare agent, cheaper on 13 of 15 questions (n=15, p=0.007), and+ **−49 % to −70 % tool calls** across paired runs on `pallets/flask` and+ `scikit-learn`. See [docs/BENCHMARKS.md](../BENCHMARKS.md) for the sample+ sizes, the tests, and the runs where the cost saving did not replicate.- **Multi-repo workspace intelligence** — cross-repo co-changes, API contractextraction (HTTP / gRPC / topics) with provider↔consumer matching, packagedependency mapping, federated MCP queries (`repo="all"`), workspace dashboard and