Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.
Python aicode-complexitycode-healthcode-intelligencecode-qualitydead-codedeveloper-toolsdocumentationgit-analyticsmcpopen-sourcepythonrefactoringstatic-analysistechnical-debt
feat(health): de-stack history findings before the history cap (#4129)
Raghav Chamadiya2185c45723f5
diff --git a/docs/architecture/code-health.md b/docs/architecture/code-health.mdindex e8a0bdd64..5f9de678f 100644--- a/docs/architecture/code-health.md+++ b/docs/architecture/code-health.md@@ -339,7 +339,9 @@ number.‡ A ceiling. The live cap is `history_cap(structure)`, `min(3.5, 1.0 + structure)`,where `structure` is the file's capped deduction from every other defect-category, so git history alone costs a file at most 1.0.+category, so git history alone costs a file at most 1.0. Before that cap, the+history findings de-stack: the largest counts whole and each other one at 0.25+(`destacked`), since a busy file trips several history markers for one cause.`hidden_coupling` is advisory: still detected, stored and listed, it deductsnothing. On its own it ranks defect-prone files near chance (AUC about 0.55), anddiff --git a/docs/layers/CODE_HEALTH.md b/docs/layers/CODE_HEALTH.mdindex d1856ee68..d43e813eb 100644--- a/docs/layers/CODE_HEALTH.md+++ b/docs/layers/CODE_HEALTH.md@@ -215,7 +215,9 @@ governance, a contradictory decision) come from theHistory amplifies code risk and does not stand alone. Git history costs a file atmost 1.0 point, plus one point for each point its code-shape findings deduct, upto 3.5. A simple file that changes often reads as busy, not broken, and never-drops below 9.0 from history alone.+drops below 9.0 from history alone. Several history findings on one file count+the largest in full and the rest at a quarter, because they usually describe the+same burst of activity.| Detector | Fires when ||---|---|diff --git a/packages/core/src/repowise/core/analysis/health/aggregation.py b/packages/core/src/repowise/core/analysis/health/aggregation.pyindex 11361acdd..b0574d2aa 100644--- a/packages/core/src/repowise/core/analysis/health/aggregation.py+++ b/packages/core/src/repowise/core/analysis/health/aggregation.py@@ -13,6 +13,7 @@ from __future__ import annotationsfrom collections.abc import Iterable, Mapping, Sequencefrom typing import Any+from .governance import GOVERNANCE_BIOMARKERSfrom .models import Severity, primary_findingfrom .ranking import worst_metricfrom .rows import detail_map, field, scored_rows@@ -24,6 +25,7 @@ from .scoring import (biomarker_category,biomarker_weight,deduction_split,+ destacked,history_cap,is_advisory,severity_deduction,@@ -251,6 +253,13 @@ def primary_and_magnitude_by_file(findings: Iterable[Any]) -> dict[str, dict[strreturn {path: primary_and_magnitude(fs) for path, fs in by_file.items()}+def _scored_raw(finding: Any) -> float:+ """:func:`finding_raw_deduction`, or 0 for a governance finding the score pass never saw."""+ if field(finding, "biomarker_type", None) in GOVERNANCE_BIOMARKERS:+ return 0.0+ return finding_raw_deduction(finding)++def score_breakdown(findings: Sequence[Any]) -> dict[str, Any]:"""Reconstruct one file's per-category deductions from its open findings.@@ -258,7 +267,8 @@ def score_breakdown(findings: Sequence[Any]) -> dict[str, Any]:already-weighted-and-capped value the scorer produced at index time — so thebreakdown reproduces the file's score instead of re-deriving it. Only theraw (pre-cap) figure is reconstructed, with the same ``base x weight``- formula, so a capped category is honest about how much it shed.+ formula and the same history de-stacking, so a capped category is honest+ about how much it shed.Categories are emitted in :data:`CATEGORY_CAPS` order and a category with nofindings is omitted, not zero-filled.@@ -280,8 +290,10 @@ def score_breakdown(findings: Sequence[Any]) -> dict[str, Any]:entries = per_category.get(category, [])if not entries:continue- cap = history_cap(structure) if category == HISTORY_CATEGORY else static_cap- raw_each = [finding_raw_deduction(f) for f in entries]+ raw_each = [_scored_raw(f) for f in entries]+ cap = static_cap+ if category == HISTORY_CATEGORY:+ cap, raw_each = history_cap(structure), destacked(raw_each)applied_each = [float(field(f, "health_impact", 0.0) or 0.0) for f in entries]raw_sum = sum(raw_each)applied_sum = sum(applied_each)diff --git a/packages/core/src/repowise/core/analysis/health/engine.py b/packages/core/src/repowise/core/analysis/health/engine.pyindex 576842520..365916644 100644--- a/packages/core/src/repowise/core/analysis/health/engine.py+++ b/packages/core/src/repowise/core/analysis/health/engine.py@@ -102,6 +102,7 @@ log = structlog.get_logger(__name__)# moves. Parameter counts exclude every comment kind and discover C/C++# declarators and Kotlin lists; cached walks refresh the total, typed and scalar# counts, enabling primitive_obsession in those languages.+# History findings de-stack before the history cap (``scoring.destacked``).## v40: an Extract Method plan names its helper with a placeholder# (``extracted_helper``, numbered when taken in scope) and asks the agent applyingdiff --git a/packages/core/src/repowise/core/analysis/health/scoring.py b/packages/core/src/repowise/core/analysis/health/scoring.pyindex 91cf8d6cc..4e931b185 100644--- a/packages/core/src/repowise/core/analysis/health/scoring.py+++ b/packages/core/src/repowise/core/analysis/health/scoring.py@@ -1,25 +1,14 @@"""Per-file score aggregation + repo-level KPIs.-Each file starts at 10.0. Biomarker findings deduct; deductions are-capped per category so no single category can drive the score below the-cap. Final score is clamped to [1.0, 10.0].--The recalibrated category caps (plan §3.1):-- organizational -> -3.5 # ceiling; the live cap is history_cap()- structural_complexity -> -2.5 # was -3.5- test_coverage -> -2.0- size_and_complexity -> -1.5 # was -2.0- duplication -> -1.0 # was -1.5--Per-biomarker weight multipliers (plan §3.2 Option A) are applied to-the per-finding raw deduction *before* category capping, so the strongest-empirical predictors are no longer suppressed by uniform severity values.+Each file starts at 10.0. Biomarker findings deduct; each finding's severity+deduction is scaled by its biomarker's weight, then capped per category+(:data:`CATEGORY_CAPS`; history by :func:`history_cap`). The final score is+clamped to [1.0, 10.0]."""from __future__ import annotations-from collections.abc import Callable, Iterable, Mapping+from collections.abc import Callable, Iterable, Mapping, Sequencefrom dataclasses import replacefrom typing import TYPE_CHECKING, Any@@ -90,11 +79,13 @@ HISTORY_CATEGORY = "organizational"# broken. History therefore amplifies code risk rather than standing alone: its# cap is 1.0 plus one point per point of structure deduction (every other defect# category, already capped), up to the category's 3.5. A file with no code-shape-# finding keeps at least 9.0. Checked against the defect benchmark: no-# measurable predictive cost. The weights are unchanged; fit the base and slope-# with them in the next calibration pass.+# finding keeps at least 9.0. Against a flat 3.5 cap this costs about 0.009+# defect AUC; it is kept so a refactor can lift a file that history holds down.HISTORY_CAP_BASE: float = 1.0HISTORY_CAP_PER_STRUCTURE: float = 1.0+# History markers restate one another (a busy file trips churn, entropy and+# scatter together), so after the largest each one adds only this share.+HISTORY_STACK_SHARE: float = 0.25def history_cap(structure: float) -> float:@@ -106,6 +97,14 @@ def history_cap(structure: float) -> float:)+def destacked(values: Sequence[float]) -> list[float]:+ """*values* with the largest kept whole and every other one scaled by :data:`HISTORY_STACK_SHARE`."""+ if not values:+ return []+ top = max(range(len(values)), key=values.__getitem__)+ return [v if i == top else v * HISTORY_STACK_SHARE for i, v in enumerate(values)]++# Per-biomarker deduction by severity. The scorer caps the per-category# total at the value in ``CATEGORY_CAPS``._SEVERITY_DEDUCTION: dict[Severity, float] = {@@ -125,9 +124,8 @@ _SEVERITY_DEDUCTION: dict[Severity, float] = {# L2-regularized logistic regression of "received a bug-fix in (T0,T1]" on the# per-biomarker hits PLUS an explicit NLOC control column - so each weight# reflects the biomarker's defect lift *beyond file size*. Cross-project-# (leave-one-repo-out) pooled OOF AUC ~ 0.70. The fit is reproduced by-# `local-stash/calibrate_health_weights.py`; the runtime stays pure--# deterministic / zero-LLM - only these learned constants ship.+# (leave-one-repo-out) pooled OOF AUC ~ 0.70. Method and later re-fits:+# docs/architecture/code-health.md. Only these learned constants ship.## Mapping policy ("balanced"): positive, well-measured predictors are scaled# into [1.0, 1.8] ∝ coefficient; biomarkers that fired widely but were weak/@@ -152,15 +150,10 @@ _BIOMARKER_WEIGHT_MULTIPLIER: dict[str, float] = {"complex_method": 1.21,"function_hotspot": 1.16,"god_class": 1.13,- # Prior-defect history (recent bug-fix count). NEUTRAL weight by design, not- # a boost: on the 13-repo corpus its calibrated coefficient is ~+0.02 (no lift- # beyond size + the existing process signals - it correlates +0.59 with- # change_entropy, +0.38 with churn) and the effort-aware Popt gain is +0.01- # with bootstrap CIs straddling zero. So it ships as an interpretable,- # leakage-free finding ("bug-fixed N times recently" - directly actionable,- # and it uniquely flags a handful of files the other signals miss), NOT as a- # calibrated predictor. See local-stash/{calibrate_health_weights,- # measure_prior_defect,diagnose_prior_defect}.py + the progress doc Phase 8.5.+ # Prior-defect history (recent bug-fix count). Neutral weight: its 21-repo+ # coefficient is positive (+0.13 keyword labels, CI spans zero; +0.23 SZZ+ # labels) but it correlates with change_entropy and churn, so it is kept at+ # 1.0 as an interpretable finding rather than boosted."prior_defect": 1.0,# --- kept at prior: benchmark could not fairly measure these ---"untested_hotspot": 1.3, # benchmark ingests no coverage (has_test_file fallback only)@@ -631,6 +624,50 @@ def performance_category(name: str) -> str:return _PERFORMANCE_CATEGORY.get(name, "performance")+# Per category: ``(index into the scored results, weighted deduction)``.+_Entries = dict[str, list[tuple[int, float]]]+++def _weighted_by_category(+ results_list: list[BiomarkerResult],+ weight_fn: Callable[[BiomarkerResult], float],+ category_fn: Callable[[str], str],+) -> _Entries:+ """Each finding's ``base x weight``, grouped by category."""+ raw: _Entries = {}+ for idx, r in enumerate(results_list):+ # A continuous ``deduction`` override (coverage scaled by the uncovered+ # fraction) takes the place of the severity table, so its impact stays linear.+ base = r.deduction if r.deduction is not None else severity_deduction(r.severity)+ raw.setdefault(category_fn(r.biomarker_type), []).append((idx, base * weight_fn(r)))+ return raw+++def _condition(+ raw: _Entries, caps: dict[str, float], conditioned_cap: tuple[str, Callable[[float], float]]+) -> dict[str, float]:+ """De-stack the conditioned category in place and return its live cap."""+ cond_cat, cap_fn = conditioned_cap+ others = sum(+ min(sum(d for _, d in entries), caps.get(cat, 1.0))+ for cat, entries in raw.items()+ if cat != cond_cat+ )+ if cond_cat in raw:+ idxs, values = zip(*raw[cond_cat], strict=True)+ raw[cond_cat] = list(zip(idxs, destacked(values), strict=True))+ return {cond_cat: cap_fn(others)}+++def _capped(entries: list[tuple[int, float]], cap: float, per_result: list[float]) -> float:+ """Write each entry's share of the capped category total; return that total."""+ cat_sum = sum(d for _, d in entries)+ scale = 1.0 if cat_sum <= cap else cap / cat_sum+ for idx, d in entries:+ per_result[idx] = d * scale+ return min(cat_sum, cap)++def _score_dimension(results_list: list[BiomarkerResult],weight_fn: Callable[[BiomarkerResult], float],@@ -644,46 +681,18 @@ def _score_dimension(category, cap each category, clamp to ``[1.0, 10.0]``. Every dimension runsthe identical algorithm against its own weight / category / cap tables.- *conditioned_cap* ``(category, cap_fn)`` replaces that one category's cap- with ``cap_fn(sum of every other category's capped total)``; only the- defect dimension uses it, for the history category.+ *conditioned_cap* ``(category, cap_fn)`` de-stacks that one category's+ findings (:func:`destacked`) and replaces its cap with ``cap_fn(sum of every+ other category's capped total)``; only the defect dimension uses it, for the+ history category."""- raw: dict[str, list[tuple[int, float]]] = {}- for idx, r in enumerate(results_list):- cat = category_fn(r.biomarker_type)- # A continuous ``deduction`` override (e.g. coverage scaled by the- # uncovered fraction) takes the place of the discrete severity table;- # both paths are then weighted and category-capped identically, so the- # per-finding ``health_impact`` stays linear and attributable.- base = r.deduction if r.deduction is not None else severity_deduction(r.severity)- weighted = base * weight_fn(r)- raw.setdefault(cat, []).append((idx, weighted))-+ raw = _weighted_by_category(results_list, weight_fn, category_fn)+ cap_for = _condition(raw, caps, conditioned_cap) if conditioned_cap else {}per_result = [0.0] * len(results_list)- total = 0.0- cap_for: dict[str, float] = {}- if conditioned_cap is not None:- cond_cat, cap_fn = conditioned_cap- others = sum(- min(sum(d for _, d in entries), caps.get(cat, 1.0))- for cat, entries in raw.items()- if cat != cond_cat- )- cap_for[cond_cat] = cap_fn(others)- for cat, entries in raw.items():- cap = cap_for.get(cat, caps.get(cat, 1.0))- cat_sum = sum(d for _, d in entries)- if cat_sum <= cap:- for idx, d in entries:- per_result[idx] = d- total += cat_sum- else:- # Scale down proportionally so the cap is respected.- scale = cap / cat_sum if cat_sum > 0 else 0.0- for idx, d in entries:- per_result[idx] = d * scale- total += cap-+ total = sum(+ _capped(entries, cap_for.get(cat, caps.get(cat, 1.0)), per_result)+ for cat, entries in raw.items()+ )score = max(SCORE_FLOOR, min(SCORE_MAX, SCORE_MAX - total))return score, per_resultdiff --git a/tests/unit/health/test_aggregation_contract.py b/tests/unit/health/test_aggregation_contract.pyindex a7de8d712..1cfbffe41 100644--- a/tests/unit/health/test_aggregation_contract.py+++ b/tests/unit/health/test_aggregation_contract.py@@ -268,6 +268,35 @@ def test_the_history_category_reports_the_cap_it_was_scored_under() -> None:assert by_cat[HISTORY_CATEGORY]["capped"] is True+def test_destacked_history_under_its_cap_is_not_flagged_capped() -> None:+ """History findings de-stack before the cap; that alone is not a cap binding."""+ from repowise.core.analysis.health.scoring import HISTORY_CATEGORY++ shape = _Finding(_category_biomarker("structural_complexity"), health_impact=2.5)+ top = _Finding("prior_defect", severity="critical", health_impact=2.0)+ rest = _Finding("prior_defect", severity="high", health_impact=0.3)+ history = {c["category"]: c for c in score_breakdown([shape, top, rest])["categories"]}[+ HISTORY_CATEGORY+ ]+ assert history["raw_deduction"] == 2.3+ assert [f["raw_impact"] for f in history["findings"]] == [2.0, 0.3]+ assert history["capped"] is False+++def test_a_governance_finding_does_not_reshape_the_history_breakdown() -> None:+ """Governance findings are written after scoring, so they add no raw history."""+ from repowise.core.analysis.health.scoring import HISTORY_CATEGORY++ shape = _Finding(_category_biomarker("structural_complexity"), health_impact=2.5)+ real = _Finding("prior_defect", severity="high", health_impact=1.2)+ governance = _Finding("ungoverned_hotspot", severity="critical", health_impact=0.0)+ history = {c["category"]: c for c in score_breakdown([shape, real, governance])["categories"]}[+ HISTORY_CATEGORY+ ]+ assert [f["raw_impact"] for f in history["findings"]] == [1.2, 0.0]+ assert history["capped"] is False++def test_a_category_with_no_findings_is_absent_rather_than_zero_filled() -> None:biomarker = _category_biomarker("structural_complexity")out = score_breakdown([_Finding(biomarker, health_impact=1.0)])diff --git a/tests/unit/health/test_scoring_dimensions.py b/tests/unit/health/test_scoring_dimensions.pyindex 410b64720..c6d0c25da 100644--- a/tests/unit/health/test_scoring_dimensions.py+++ b/tests/unit/health/test_scoring_dimensions.py@@ -50,6 +50,8 @@ _LEGACY_CATEGORY_CAPS = {# The structure-conditioned history cap, frozen here._HISTORY_CAP_BASE = 1.0_HISTORY_CAP_PER_STRUCTURE = 1.0+# History de-stacking: the largest history finding whole, the rest at this share.+_HISTORY_STACK_SHARE = 0.25_LEGACY_SEVERITY_DEDUCTION = {Severity.LOW: 0.3,@@ -121,6 +123,7 @@ _LEGACY_CATEGORY = {def _legacy_score_file(results: list[BiomarkerResult]) -> float:"""The pre-split algorithm, verbatim, against the frozen tables above."""raw: dict[str, float] = {}+ history: list[float] = []for r in results:cat = _LEGACY_CATEGORY.get(r.biomarker_type, "size_and_complexity")base = (@@ -129,7 +132,13 @@ def _legacy_score_file(results: list[BiomarkerResult]) -> float:else _LEGACY_SEVERITY_DEDUCTION.get(r.severity, 0.5))weighted = base * _LEGACY_WEIGHT_MULTIPLIER.get(r.biomarker_type, 1.0)- raw[cat] = raw.get(cat, 0.0) + weighted+ if cat == "organizational":+ history.append(weighted)+ else:+ raw[cat] = raw.get(cat, 0.0) + weighted+ if history:+ top = max(history)+ raw["organizational"] = top + _HISTORY_STACK_SHARE * (sum(history) - top)structure = sum(min(cat_sum, _LEGACY_CATEGORY_CAPS.get(cat, 1.0))diff --git a/tests/unit/health/test_scoring_snapshot.py b/tests/unit/health/test_scoring_snapshot.pyindex 74d296637..5bc137294 100644--- a/tests/unit/health/test_scoring_snapshot.py+++ b/tests/unit/health/test_scoring_snapshot.py@@ -239,7 +239,8 @@ def test_organizational_cap_bounds_stream():def test_history_cap_follows_structure():"""Each point of code-shape deduction lets history deduct one more, to 3.5."""- history = [_result("prior_defect", Severity.CRITICAL) for _ in range(3)] # 6.0 raw+ # 2.0 + 0.25 x 10.0 = 4.5 de-stacked+ history = [_result("prior_defect", Severity.CRITICAL) for _ in range(6)]# structure 2.5 (low_cohesion x 3 -> structural cap) -> history cap 3.5shaped = [*history, *(_result("low_cohesion", Severity.CRITICAL) for _ in range(3))]scores, _ = score_file(shaped)@@ -252,6 +253,19 @@ def test_history_cap_follows_structure():assert round(sum(ded), 6) == round(0.847 + 1.847, 6)+def test_history_markers_destack():+ """The largest history finding counts whole, every other one at a quarter."""+ shaped = [_result("low_cohesion", Severity.CRITICAL) for _ in range(3)] # structure 2.5+ history = [+ _result("prior_defect", Severity.CRITICAL), # 2.0+ _result("change_entropy", Severity.HIGH), # 1.2 x 1.51 = 1.812+ ]+ scores, ded = score_file([*shaped, *history])+ # 2.0 + 0.25 x 1.812 = 2.453, under the 3.5 cap+ assert round(scores["defect"], 6) == round(10.0 - 2.5 - 2.453, 6)+ assert [round(d, 6) for d in ded[3:]] == [2.0, 0.453]++def test_improving_code_never_lowers_score():"""Monotone in structure at fixed history: removing a code-shape findingcan only raise the score."""