Skip to content

Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.

feat(health): let a called function's assertions reach its caller (#2399)

Raghav Chamadiyac1a3756c17e5
diff --git a/docs/layers/CODE_HEALTH.md b/docs/layers/CODE_HEALTH.md
index f9a81b35c..3fc2cf78a 100644
--- a/docs/layers/CODE_HEALTH.md
+++ b/docs/layers/CODE_HEALTH.md
@@ -454,9 +454,15 @@ whether an assertion observes a double or production output, which is a dataflow
question the pass does not ask — and on the TypeScript sample that one question
accounted for every false positive.
-`assertion_free_test` measures **52%** on TypeScript and **53%** on Python, and
-does not report on Go or Java at all; the per-language reasoning is in
-[LANGUAGE_SUPPORT.md](LANGUAGE_SUPPORT.md#code-health-coverage).
+`assertion_free_test` measures **51%** on TypeScript (51 findings, the complete
+population of two corpora) and **86%** on Python (29 hand-labelled, a systematic
+sample of 172), and does not report on Go or Java at all; the per-language
+reasoning is in
+[LANGUAGE_SUPPORT.md](LANGUAGE_SUPPORT.md#code-health-coverage). The gap between
+the two is one fact: a test that delegates its oracle to a helper in the same
+file is resolved, and a JS/TS suite keeps its helpers in another file. The
+marker stays advisory on both, and twenty-nine items are too few to settle
+whether Python clears the bar.
It asks a question with a yes-or-no answer rather than a
threshold, which is why it can be stated plainly: a test case whose assertion
diff --git a/docs/layers/LANGUAGE_SUPPORT.md b/docs/layers/LANGUAGE_SUPPORT.md
index 6d8eb8ec9..4ae153b9c 100644
--- a/docs/layers/LANGUAGE_SUPPORT.md
+++ b/docs/layers/LANGUAGE_SUPPORT.md
@@ -441,11 +441,29 @@ for Java. A language with no row classifies nothing. Go and Java are classified
and counted, which is how their precision below was measured, but the marker
does not report on them — the shipping set is a constant in the detector.
-Measured precision, hand-labelled: **52%** on TypeScript (52 findings, the
-complete population of two corpora) and **53%** on Python (30, a systematic
-sample of 194). Both are below the 70% house bar and above the 40% at which
-`mock_saturated_test` shipped TypeScript, and they ship on the same line: the
-false positives are one family the marker declares, described below.
+Measured precision, hand-labelled: **51%** on TypeScript (51 findings, the
+complete population of two corpora) and **86%** on Python (29 findings, a
+systematic sample of 172). The marker ships advisory on both, because the bar
+below is a property of the marker rather than of one language, and TypeScript
+is at 51%.
+
+Twenty-nine items cannot settle a 70% bar: 86% on that sample carries a 95%
+interval of roughly 68% to 96%, whose lower bound sits below the bar it appears
+to clear. The reading is consistent with clearing it and does not demonstrate
+it. What moved Python is that a test's oracle now reaches it from a function it
+calls in the same file, described below. The same change barely moved
+TypeScript, for a reason worth recording: a JS/TS suite keeps its helpers in a
+separate module, so 19 of its 25 remaining false positives delegate across a
+file boundary rather than within one.
+
+An earlier pass published 53% for Python. Re-labelling the pre-change findings
+of the same corpus, drawn the same way and read against this pass's rubric,
+gives 68.8% (22 of 32), so the gain below is that 68.8 to 86.2 and the 53% is
+superseded rather than contradicted. The two passes are not reconciled: 15
+points of the difference is labelling, on samples of about thirty, and the
+rubric behind the earlier number was not recorded. TypeScript read 52% then and
+51% here, which is the evidence that the two rubrics are close and Python's gap
+is sample noise rather than a changed standard.
A **mock verification counts as an assertion for this marker and not for the one
above**, which is the deliberate inversion described in CODE_HEALTH.md. It
@@ -491,8 +509,41 @@ from the test that calls it:
asserts on the test's behalf. Hand-labelled at **33%** over 18 findings, and 10
of its 12 false positives were this family.
-Separating them needs the assertions of a called function to reach its caller,
-which is a cross-function question this pass does not ask.
+Separating them needs the assertions of a called function to reach its caller.
+Within a single file that now happens. Neither language is unblocked by it: Go
+hands `*testing.T` to a package-level helper and Java inherits a base-class
+method, and both live in another file, so re-admitting either is a measurement
+of its own.
+
+**A called function's assertions reach its caller, inside one file.** The
+assertion count is per function, so a test that hands its checks to a helper
+beside it read as checking nothing. That was the largest false-positive family
+in every language the marker reports on. The walk now records the names each
+function calls, and the marker resolves them against same-file functions that
+assert. It counts nothing and no calibrated marker reads it.
+
+What it suppresses, by corpus:
+
+| Corpus | Findings | Test cases | Rate |
+|---|---|---|---|
+| 2,012-file Python | 763 to 501 | 17,966 | 4.2% to 2.8% |
+| 794-file Python | 194 to 172 | 5,212 | 3.7% to 3.3% |
+| 374-file TypeScript | 35 to 33 | 7,354 | 0.5% to 0.4% |
+| 562-file TypeScript | 18 to 18 | 4,761 | 0.4% to 0.4% |
+
+Six suppressions were read by hand across the two Python corpora and each is
+the declared family: a test whose whole body delegates to a helper that asserts,
+or one that parametrises a sibling test by calling it. The resolution matches
+names against same-file names and drops the receiver, so a call to an imported
+function sharing a local helper's name, or a method on a test-local double
+sharing a module-level function's name, suppresses wrongly. Both hide a finding
+rather than invent one. No instance turned up in the corpora above, and the
+shape is pinned by a test.
+
+What it deliberately does not reach is anything needing the call graph, which
+this pass does not consult: `super().test_x(...)`, a package-level Go helper
+handed the test handle, and the shared JS/TS fixture module that is most of
+TypeScript's remaining error.
**A `.tsx` file is read with the JSX grammar.** It arrives tagged `typescript`,
and the grammar that tag selects fails on the first `<Component />`; everything
diff --git a/packages/core/src/repowise/core/analysis/health/biomarkers/assertion_free_test.py b/packages/core/src/repowise/core/analysis/health/biomarkers/assertion_free_test.py
index fbb311177..075276b06 100644
--- a/packages/core/src/repowise/core/analysis/health/biomarkers/assertion_free_test.py
+++ b/packages/core/src/repowise/core/analysis/health/biomarkers/assertion_free_test.py
@@ -33,10 +33,11 @@ from .base import BiomarkerResult, FileContext
#: Languages the marker reports on. Go and Java are classified by
#: ``complexity/test_case.py`` and counted like any other, and are deliberately
-#: absent here: both conventionally hand the oracle to a helper the test calls,
-#: and the assertion count is per function, so the helper's assertions are
-#: invisible from its caller. On Go that is not a false-positive family the
-#: marker could declare and live with, it is nearly the whole population.
+#: absent here: both conventionally hand the oracle to a helper the test calls.
+#: ``_asserting_names`` now resolves that within a file, but the idiom reaches
+#: across one in both languages, as a package-level Go helper handed ``*testing
+#: .T`` and a Java base-class method. Re-admitting either is a measurement of
+#: its own, not a consequence of this one.
#: Figures and the reasoning: LANGUAGE_SUPPORT.md#code-health-coverage.
SHIPPING_LANGUAGES = frozenset({"javascript", "python", "tsx", "typescript"})
@@ -53,6 +54,7 @@ class AssertionFreeTestDetector:
out: list[BiomarkerResult] = []
# ``all_functions``, not ``function_metrics``: name-keying collapses a
# file's anonymous ``it`` callbacks into one row. See ``FileContext``.
+ oracles = _asserting_names(ctx)
for fn in ctx.all_functions:
# ``is_test_case`` is the whole gate. Every other test-quality
# marker can lean on "it has assertions, so it is a test"; this one
@@ -61,6 +63,8 @@ class AssertionFreeTestDetector:
continue
if fn.assertion_count or fn.verification_count:
continue
+ if fn.called_names & oracles:
+ continue
out.append(
BiomarkerResult(
biomarker_type=self.name,
@@ -75,4 +79,32 @@ class AssertionFreeTestDetector:
return out
+def _asserting_names(ctx: FileContext) -> frozenset[str]:
+ """Names of functions in this file that do check something, lowercased.
+
+ A test whose own assertion count is zero has still checked something if it
+ handed the job to a helper that asserts, and the count is per function, so
+ the helper's assertions are otherwise invisible from the test that calls
+ it. That is the largest false-positive family this marker has in every
+ language it reports on.
+
+ Names only, and the whole file is one namespace: the receiver is dropped,
+ so a helper on one class suppresses a same-named helper on another, and a
+ method on a test-local double suppresses against a module-level function of
+ that name. Both directions hide a real finding rather than invent one,
+ which is the tolerable way round for an advisory marker, but neither is
+ free. A helper nested inside the test body is not collected as a function
+ at all, so it suppresses nothing.
+
+ Same file only. ``super().test_x(...)``, a package-level Go helper and a
+ shared JS fixture all live elsewhere and would need the call graph, which
+ this pass does not consult, so those stay false positives.
+ """
+ return frozenset(
+ fn.name.lower()
+ for fn in ctx.all_functions
+ if fn.assertion_count or fn.verification_count
+ )
+
+
BIOMARKER = AssertionFreeTestDetector()
diff --git a/packages/core/src/repowise/core/analysis/health/complexity/assertions.py b/packages/core/src/repowise/core/analysis/health/complexity/assertions.py
index 8e093d0b8..197431e2f 100644
--- a/packages/core/src/repowise/core/analysis/health/complexity/assertions.py
+++ b/packages/core/src/repowise/core/analysis/health/complexity/assertions.py
@@ -263,8 +263,8 @@ def _is_assertion_statement(
def _collect_assertion_facts(
body_node: Node, lmap: LanguageNodeMap, dialect: AssertDialect | None = None
-) -> tuple[list[tuple[int, int, int]], int, int]:
- """``(blocks, total, verifications)`` assertion facts for one function body.
+) -> tuple[list[tuple[int, int, int]], int, int, frozenset[str]]:
+ """``(blocks, total, verifications, called)`` facts for one function body.
*blocks* are runs of ≥2 consecutive **narrow-tier** assertion statements,
each recorded as ``(start_line, end_line, count)``. Runs are found per
@@ -284,12 +284,22 @@ def _collect_assertion_facts(
*verifications* counts mock verifications, on the same block-level gate as
*total* and in neither of the other two counts. See the module docstring.
+
+ *called* is every name called directly in this body, lowercased, excluding
+ nested function bodies on the same rule the assertion scan uses. It rides
+ along on this traversal because the traversal already reaches every call
+ node; collecting it separately would mean walking every function body
+ twice. It counts nothing and so cannot move a calibrated marker. Empty
+ under the early return above, which is fine only because the one reader
+ ships no language that takes it.
"""
if not lmap.assert_kinds and not lmap.assert_call_kinds:
- return [], 0, 0
+ return [], 0, 0, frozenset()
blocks: list[tuple[int, int, int]] = []
total = 0
verifications = 0
+ called: set[str] = set()
+ call_kinds = lmap.call_kinds or lmap.assert_call_kinds
def _scan_siblings(parent: Node, *, count_total: bool) -> None:
nonlocal total, verifications
@@ -328,6 +338,10 @@ def _collect_assertion_facts(
or node.type in lmap.lambda_kinds
)
_scan_siblings(node, count_total=counts_here)
+ if node.type in call_kinds:
+ names = _callee_names(node)
+ if names is not None:
+ called.add(names[0])
for child in node.children:
if child.type in lmap.function_kinds:
continue # nested fn, not collected as its own entry either
@@ -347,4 +361,4 @@ def _collect_assertion_facts(
verifications += 1
elif tier != _NOT_ASSERTION:
total += 1
- return blocks, total, verifications
+ return blocks, total, verifications, frozenset(called)
diff --git a/packages/core/src/repowise/core/analysis/health/complexity/models.py b/packages/core/src/repowise/core/analysis/health/complexity/models.py
index 993301a4f..d6973fbf4 100644
--- a/packages/core/src/repowise/core/analysis/health/complexity/models.py
+++ b/packages/core/src/repowise/core/analysis/health/complexity/models.py
@@ -61,6 +61,11 @@ class FunctionComplexity:
# True when this function is a test case its framework would run, as
# opposed to a helper or fixture beside it. ``complexity/test_case.py``.
is_test_case: bool = False
+ # Every name called directly in this function's own body, lowercased,
+ # excluding nested function bodies. Empty for a language with no assertion
+ # vocabulary row, because it rides on that traversal. It answers one
+ # question: did this function hand its work to something else in the file?
+ called_names: frozenset[str] = frozenset()
def __post_init__(self) -> None:
if self.complex_conditions is None:
diff --git a/packages/core/src/repowise/core/analysis/health/complexity/walker.py b/packages/core/src/repowise/core/analysis/health/complexity/walker.py
index 522730b64..7640885f1 100644
--- a/packages/core/src/repowise/core/analysis/health/complexity/walker.py
+++ b/packages/core/src/repowise/core/analysis/health/complexity/walker.py
@@ -161,7 +161,7 @@ def walk_file(
for fn_node in _collect_function_nodes(tree.root_node, lmap):
body = fn_node.child_by_field_name("body") or fn_node
ccn, max_nest, cognitive, bumps, conditions = _walk_function_body(body, lmap)
- assertion_blocks, assertion_count, verifications = _collect_assertion_facts(
+ assertion_blocks, assertion_count, verifications, called = _collect_assertion_facts(
body, lmap, asserts
)
name = _find_function_entry_name(fn_node, lmap)
@@ -181,6 +181,7 @@ def walk_file(
verification_count=verifications,
mock_setup_count=_count_mock_setup(fn_node, body, lmap, mock_dialect, asserts),
is_test_case=is_test_case(fn_node, name, language),
+ called_names=called,
)
functions.append(fc)
fc_by_node_id[fn_node.id] = fc
diff --git a/packages/core/src/repowise/core/analysis/health/engine.py b/packages/core/src/repowise/core/analysis/health/engine.py
index e6013779c..5f9529b57 100644
--- a/packages/core/src/repowise/core/analysis/health/engine.py
+++ b/packages/core/src/repowise/core/analysis/health/engine.py
@@ -100,7 +100,13 @@ log = structlog.get_logger(__name__)
# Not a licence to move a calibrated scoring weight — those are frozen
# independently of this stamp.
#
-# Current stamp: the health pass reads a ``.tsx`` file with the JSX grammar.
+# Current stamp: the walker records one new ``FunctionComplexity`` field,
+# ``called_names``, which a cached v16 walk does not carry at all. It feeds the
+# advisory ``assertion_free_test``, which no longer calls a test assertion-free
+# when it handed its checks to a function in the same file that asserts. It
+# counts nothing, so no calibrated marker reads it and no score moves.
+#
+# v16: the health pass reads a ``.tsx`` file with the JSX grammar.
# It arrives tagged ``typescript``, and the grammar that tag selects errors on
# the first ``<Component />``, so a cached v15 walk stored an ERROR-recovered
# tree. Three things stored under it change for ``.tsx`` files and nothing
@@ -156,7 +162,7 @@ log = structlog.get_logger(__name__)
# forms. Files that were counted untested and are not become tested, which
# moves untested-hotspot findings and the scores that carry them, on every
# language with a prefix or spec convention rather than Ruby alone.
-HEALTH_ANALYZER_VERSION = 16
+HEALTH_ANALYZER_VERSION = 17
# Method-level smells that make the dataflow / Extract Method pass worthwhile.
# Only files carrying one of these get a CFG + def/use + reaching pass built.
diff --git a/tests/unit/health/test_assertion_free_test.py b/tests/unit/health/test_assertion_free_test.py
index 48bee2f8f..c4715aef4 100644
--- a/tests/unit/health/test_assertion_free_test.py
+++ b/tests/unit/health/test_assertion_free_test.py
@@ -448,3 +448,104 @@ def test_go_and_java_are_classified_but_never_reported() -> None:
assert _counts(java, "src/test/java/FooTest.java", "java")["bare"][2] is True
assert _flagged(go, "pkg/thing_test.go", "go") == []
assert _flagged(java, "src/test/java/FooTest.java", "java") == []
+
+
+def test_a_delegated_oracle_in_the_same_file_counts() -> None:
+ _require("python")
+ # The largest false-positive family: the test hands its checks to a helper
+ # beside it, and the assertion count is per function.
+ source = (
+ "def _check_user(u):\n"
+ " assert u.name\n"
+ " assert u.age > 0\n"
+ "\n"
+ "def test_delegates():\n"
+ " _check_user(build_user())\n"
+ "\n"
+ "def test_bare():\n"
+ " build_user()\n"
+ )
+ assert _flagged(source) == ["test_bare"]
+
+
+def test_a_delegated_oracle_through_a_receiver_counts() -> None:
+ _require("python")
+ # Same shape stated as a method call. The callee name is what resolves,
+ # never the receiver.
+ source = (
+ "class TestUser:\n"
+ " def _check(self, u):\n"
+ " assert u.name\n"
+ "\n"
+ " def test_delegates(self):\n"
+ " self._check(build_user())\n"
+ "\n"
+ " def test_bare(self):\n"
+ " build_user()\n"
+ )
+ assert _flagged(source) == ["test_bare"]
+
+
+def test_a_helper_that_asserts_nothing_suppresses_nothing() -> None:
+ _require("python")
+ # The call is not the point; the callee having an oracle is.
+ source = (
+ "def _build(u):\n"
+ " return User(u)\n"
+ "\n"
+ "def test_calls_a_plain_helper():\n"
+ " _build(1)\n"
+ )
+ assert _flagged(source) == ["test_calls_a_plain_helper"]
+
+
+def test_a_helper_in_another_file_is_not_resolved() -> None:
+ _require("python")
+ # Resolution is intra-file by construction: this pass has no call graph, so
+ # an imported helper stays a false positive rather than a guess.
+ source = "from .helpers import check_user\n\ndef test_imported():\n check_user(1)\n"
+ assert _flagged(source) == ["test_imported"]
+
+
+def test_a_delegated_oracle_counts_for_a_jest_callback() -> None:
+ _require("typescript")
+ # The helper must not look like an assertion itself, or the counter would
+ # have reached it without any of this.
+ source = (
+ "function renderedOk(r) {\n"
+ " expect(r.ok).toBe(true);\n"
+ "}\n"
+ 'describe("s", () => {\n'
+ ' it("delegates", () => { renderedOk(run()); });\n'
+ ' it("bare", () => { run(); });\n'
+ "});\n"
+ )
+ assert _flagged(source, "src/x.test.ts", "typescript") == ["it callback"]
+ counts = _counts(source, "src/x.test.ts", "typescript")
+ assert counts["renderedOk"][0] == 1
+
+
+
+def test_resolution_is_by_name_and_a_second_class_shares_it() -> None:
+ _require("python")
+ # The limit, pinned so it is a known shape rather than a surprise: the
+ # receiver is dropped, so the whole file is one namespace. TestB._check
+ # checks nothing, and TestA._check suppresses it. This hides a finding
+ # rather than inventing one, which is the tolerable direction, and the fix
+ # is a class-aware resolution this pass does not do.
+ source = (
+ "class TestA:\n"
+ " def _check(self, r):\n"
+ " assert r == 1\n"
+ "\n"
+ " def test_a(self):\n"
+ " self._check(f())\n"
+ "\n"
+ "class TestB:\n"
+ " def _check(self, r):\n"
+ " pass\n"
+ "\n"
+ " def test_b(self):\n"
+ " self._check(g())\n"
+ )
+ assert _flagged(source) == []
diff --git a/tests/unit/health/test_model_version_coupling.py b/tests/unit/health/test_model_version_coupling.py
index d9ed019a1..6d1a6325f 100644
--- a/tests/unit/health/test_model_version_coupling.py
+++ b/tests/unit/health/test_model_version_coupling.py
@@ -22,7 +22,7 @@ from repowise.core.analysis.health.refactoring.identity import (
# The three stamps as they stand together. Moving either identity model means
# moving the analyzer with it, and updating this tuple in the same commit.
-_STAMPS = (16, 2, 2)
+_STAMPS = (17, 2, 2)
def test_the_identity_models_and_the_analyzer_stamp_are_pinned_together() -> None: