Skip to content

Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.

feat(health): answer "what tests this" from the call graph, not imports (#1755)

Raghav Chamadiya68b29bde64f5
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 107140c1b..4b9ba1015 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -16,43 +16,55 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **A test-to-code map that needs no coverage report.** The per-test map only
ever existed if you ingested a coverage report with contexts, so on most
repositories `tests_to_run` was empty, `impacted_tests` said "run the full
- suite", and `untested_hotspot` fell back to matching filenames. The dependency
- graph already records which test files import which source files, and that
+ suite", and `untested_hotspot` fell back to matching filenames. The call graph
+ already records which test can execute into which source file, and that
relation now answers when the measured one cannot. It is labelled `inferred`
everywhere, never blended with measured coverage, and never turned into a
percentage. Nothing is stored: it is a bounded walk over rows already indexed,
- measured at 27 ms once per health run on a 3,700-file repository.
+ measured at 63 ms once per health run on a 3,700-file repository.
+
+ The call graph is the primary signal and the import graph the weaker fallback,
+ which is a measured choice rather than a preference. Dogfooded against a real
+ `coverage run --contexts=test` over a slice where per-test attribution is
+ complete, with both sides seeing the same 37 test files: walking imports one
+ hop scored 72.1% precision at 19.5% recall, walking calls three hops scored
+ 91.7% at 27.7%, and dropping the one call-resolution strategy that only
+ matches a name repo-wide (`global_unique`) took that to 95.7% at no cost to
+ recall. Unioning the two is worse than either lead, so the import tier is
+ spent only on the files the call graph says nothing about, where it answers
+ one more of them at no loss of precision.
New fields: `tests_to_run_basis` on `get_risk`'s directive
(`measured` / `inferred` / `none`), `basis` and a `status: "inferred"` on
`get_change_risk`'s `impacted_tests`, and a `via` marker on every
- `repowise impacted-tests` candidate.
+ `repowise impacted-tests` candidate saying which tier answered.
### Changed
-- **`untested_hotspot` stops accusing files that six tests import.** With no
+- **`untested_hotspot` stops accusing files that the tests run.** With no
coverage ingested it fired on any hotspot without a *paired test file*, which
is a filename convention, so a suite that names its tests for behaviour
- satisfied nothing. A test reaching the file in the import graph now suppresses
- it too. On this repository five of the six worst bug-magnet files had no test
- named for them and read as untested; the sixth, `analysis/health/engine.py`,
- was called tested because the convention matched `distill/test_engine.py` on
- basename alone. The same floor now runs under `get_risk`'s `missing_tests`,
- which had two separate filename heuristics that disagreed.
-
- Measured on repowise's own index: 74 of its 110 standing `untested_hotspot`
- findings are cleared, 18 of them graded critical. The persisted
- `has_test_file` widens to match, so the file table stops labelling those same
- files "untested" while the biomarker says nothing.
+ satisfied nothing. A test whose calls reach the file now suppresses it too. On
+ this repository five of the six worst bug-magnet files had no test named for
+ them and read as untested; the sixth, `analysis/health/engine.py`, was called
+ tested because the convention matched `distill/test_engine.py` on basename
+ alone. The same floor now runs under `get_risk`'s `missing_tests`, which had
+ two separate filename heuristics that disagreed.
+
+ Measured on repowise's own index, against the 80 standing `untested_hotspot`
+ findings the filename convention leaves behind: the call graph clears 32 of
+ them, 22 graded high and 3 critical, where a one-hop import walk clears 11.
+ The persisted `has_test_file` widens to match, so the file table stops
+ labelling those same files "untested" while the biomarker says nothing.
Both stored values are wrong on an index built before this, so
- `HEALTH_ANALYZER_VERSION` moves to 2 and the next `repowise update` with
+ `HEALTH_ANALYZER_VERSION` moves to 3 and the next `repowise update` with
changed files re-scores health rather than waiting out the decay timer.
- **`repowise impacted-tests --format json` renames `guessed_tests` to
`inferred_tests`.** The bucket no longer holds only filename guesses, so the
- old name described the wrong thing; each entry carries `via`
- (`import-graph` or `filename-pattern`) to say which tier answered.
+ old name described the wrong thing; each entry carries `via` (`call-graph`,
+ `import-graph` or `filename-pattern`) to say which tier answered.
---
diff --git a/packages/cli/src/repowise/cli/agent_targets/_data/codex_prompts/repowise-impacted-tests.md b/packages/cli/src/repowise/cli/agent_targets/_data/codex_prompts/repowise-impacted-tests.md
index 33f05b2f7..0e8aedd62 100644
--- a/packages/cli/src/repowise/cli/agent_targets/_data/codex_prompts/repowise-impacted-tests.md
+++ b/packages/cli/src/repowise/cli/agent_targets/_data/codex_prompts/repowise-impacted-tests.md
@@ -39,8 +39,9 @@ repowise impacted-tests main..HEAD --format list | xargs pytest
"no tests needed" result.
- A changed file with no coverage rows gets *candidates* instead, in a table
headed "NOT coverage-backed", each carrying a `via` marker: `changed-test`
- (the changed file is itself a test), `import-graph` (a test file that reaches
- it, no ingest needed), or `filename-pattern` (a name-shaped guess). Report
+ (the changed file is itself a test), `call-graph` or `import-graph` (a test
+ file that reaches it, no ingest needed), or `filename-pattern` (a name-shaped
+ guess). Report
them as candidates. Only `via: coverage` proves a test executed the change.
- A file none of those can speak to is "unknown, run the full suite".
- For a whole-change defect-risk score, use `/prompts:repowise-risk`. For per-file
diff --git a/packages/cli/src/repowise/cli/commands/impacted_tests_cmd.py b/packages/cli/src/repowise/cli/commands/impacted_tests_cmd.py
index cdfc457ab..65945cfbe 100644
--- a/packages/cli/src/repowise/cli/commands/impacted_tests_cmd.py
+++ b/packages/cli/src/repowise/cli/commands/impacted_tests_cmd.py
@@ -7,9 +7,9 @@ the per-test test-to-code map built by ``repowise coverage add``. The output is
Honest about what it knows:
- a changed file with per-test coverage -> the exact covering tests;
- - a changed file with no coverage rows -> candidate tests the import graph
- shows reaching it, or failing that a filename-pattern guess, each labelled
- with which one answered and none of them claimed as coverage;
+ - a changed file with no coverage rows -> candidate tests the dependency
+ graph shows reaching it, or failing that a filename-pattern guess, each
+ labelled with which one answered and none of them claimed as coverage;
- a new file with neither -> "unknown, run the full suite" (never implied
as "no tests needed").
@@ -161,16 +161,21 @@ async def _resolve_impacted(
No coverage row, so fall back to naming candidates. A changed file that
is itself a test is its own candidate (``via="changed-test"``) - it used
to be reported as a file with no test, which is true and useless. Then
- the import graph (``via="import-graph"``): a test file that reaches this
- one is a recorded edge, and it finds suites whose tests are named for
- behaviour rather than for the file. The filename pattern answers when
- the graph is silent (``via="filename-pattern"``). All are file-level and
- all over-claim; none may be read as coverage.
+ the dependency graph, which reports which tier answered: a test whose
+ calls reach the file (``via="call-graph"``) or, only where no call edge
+ does, a test that imports it (``via="import-graph"``). Both are recorded
+ edges, and they find suites whose tests are named for behaviour rather
+ than for the file. The filename pattern answers when the graph is silent
+ (``via="filename-pattern"``). All are file-level and all over-claim;
+ none may be read as coverage.
``unknown``
Nothing said anything. Run the full suite.
"""
from repowise.core.analysis.health.coverage import paired_test_file
- from repowise.core.analysis.test_reachability import load_test_files, tests_reaching
+ from repowise.core.analysis.test_reachability import (
+ load_test_files,
+ tests_reaching_by_tier,
+ )
from repowise.core.persistence.crud import tests_covering
covered: dict[str, dict] = out["covered"]
@@ -195,7 +200,7 @@ async def _resolve_impacted(
# is what makes this cheap, and splitting it would run the same query once
# per changed path.
try:
- reaching = await tests_reaching(session, repo_id, uncovered)
+ reaching = await tests_reaching_by_tier(session, repo_id, uncovered)
test_files = await load_test_files(session, repo_id)
except Exception:
reaching, test_files = {}, set()
@@ -210,11 +215,11 @@ async def _resolve_impacted(
}
)
continue
- graph_tests = reaching.get(source_file) or []
- if graph_tests:
+ reached = reaching.get(source_file)
+ if reached:
out["inferred"].extend(
- {"source_file": source_file, "test_file": t, "via": "import-graph"}
- for t in graph_tests
+ {"source_file": source_file, "test_file": t, "via": reached.via}
+ for t in reached.tests
)
continue
guess = paired_test_file(source_file, repo_keys)
@@ -301,7 +306,7 @@ def _render_table(result: dict) -> None:
"[yellow]No test-to-code map ingested.[/yellow] Run "
"[cyan]repowise coverage add[/cyan] on a coverage.py report written with "
"[cyan]coverage run --contexts=test[/cyan] to get exact impacted tests. "
- "Falling back to the import graph and filename patterns below."
+ "Falling back to the dependency graph and filename patterns below."
)
covered = result["covered"]
@@ -331,7 +336,8 @@ def _render_table(result: dict) -> None:
console.print(
f"[yellow]{len(result['inferred'])} candidate(s)[/yellow] for {files} file(s) with "
"no coverage. [dim]changed-test[/dim] = the changed file is itself a test; "
- "[dim]import-graph[/dim] = a test file that reaches this one; "
+ "[dim]call-graph[/dim] = a test whose calls reach this file; "
+ "[dim]import-graph[/dim] = a test that only imports it; "
"[dim]filename-pattern[/dim] = a name-shaped guess. None proves execution - "
"verify they exercise the change."
)
diff --git a/packages/core/src/repowise/core/analysis/health/biomarkers/base.py b/packages/core/src/repowise/core/analysis/health/biomarkers/base.py
index ee10148ff..554458d3c 100644
--- a/packages/core/src/repowise/core/analysis/health/biomarkers/base.py
+++ b/packages/core/src/repowise/core/analysis/health/biomarkers/base.py
@@ -46,13 +46,14 @@ class FileContext:
# Per-file git metadata (may be empty when git indexing skipped).
git_meta: dict[str, Any] = field(default_factory=dict)
# Graph-derived signals.
- # True when a test file reaches this one through the import graph, within
- # ``test_reachability.DEFAULT_MAX_DEPTH`` hops. Distinct from
+ # True when a test file can execute into this one along the call graph,
+ # within ``test_reachability.DEFAULT_CALL_DEPTH`` hops. Distinct from
# ``has_test_file``, which is a filename convention: this is a recorded
# edge, so it finds behaviour-named tests the convention cannot, and it
- # over-claims, since importing a module is not exercising it. Sound as a
- # floor ("something tests this"), never as a coverage quantity. ``False``
- # when no graph was available - the documented "no signal" outcome.
+ # over-claims, since control reaching a file is not a run exercising it.
+ # Sound as a floor ("something tests this"), never as a coverage quantity.
+ # ``False`` when no graph was available - the documented "no signal"
+ # outcome.
reached_by_tests: bool = False
dependents_count: int = 0
# Repo-wide 80th percentile of file-level in-degree (dependents),
diff --git a/packages/core/src/repowise/core/analysis/health/biomarkers/untested_hotspot.py b/packages/core/src/repowise/core/analysis/health/biomarkers/untested_hotspot.py
index 9bf0da37d..8cfc89959 100644
--- a/packages/core/src/repowise/core/analysis/health/biomarkers/untested_hotspot.py
+++ b/packages/core/src/repowise/core/analysis/health/biomarkers/untested_hotspot.py
@@ -5,7 +5,7 @@ Fires when a file is **all three** of:
- a *hotspot* (``git_meta['is_hotspot']`` true OR commit_count_90d ≥ 8)
- under-tested (``line_coverage_pct`` < 40 when coverage is available,
OR, when it isn't, neither a paired test file nor a test reaching it
- through the import graph)
+ through the call graph)
- centrally depended on (``dependents_count`` ≥ 4 OR temporal_hotspot
in the top decile)
@@ -14,9 +14,10 @@ file has.
When no coverage report has been ingested the biomarker falls back to two
signals that both mean "something tests this": the paired-test-file naming
-convention, and ``reached_by_tests``: a test file importing this one, recorded
-in the dependency graph. Either suppresses the finding. Both over-claim relative to real execution, which is the right
-direction of error for a check whose output is an accusation.
+convention, and ``reached_by_tests``, a test file whose calls reach this one in
+the dependency graph. Either suppresses the finding. Both over-claim relative to
+real execution, which is the right direction of error for a check whose output
+is an accusation.
"""
from __future__ import annotations
diff --git a/packages/core/src/repowise/core/analysis/health/engine.py b/packages/core/src/repowise/core/analysis/health/engine.py
index 7204c15c3..580b88677 100644
--- a/packages/core/src/repowise/core/analysis/health/engine.py
+++ b/packages/core/src/repowise/core/analysis/health/engine.py
@@ -38,8 +38,8 @@ from ...ingestion.package_roots import module_for as _module_for
from ...ingestion.package_roots import package_roots_from_paths as _package_roots
from ...ingestion.package_roots import scan_package_roots as _scan_package_roots
from ..test_reachability import (
+ call_graph_from_graph,
files_reached_by_tests,
- forward_dependencies_from_graph,
)
from .biomarkers import FileContext, detect_all
from .biomarkers.base import HasEdge
@@ -92,7 +92,9 @@ log = structlog.get_logger(__name__)
# ``has_test_file`` widened to match it. Both stored values are wrong on an
# index built before that, and wrong in the direction that accuses a tested
# file, so they should not wait out the decay timer.
-HEALTH_ANALYZER_VERSION = 2
+# 3: that signal moved from the import graph to the call graph, so the same two
+# values are wrong again on an index stamped 2 - this time in both directions.
+HEALTH_ANALYZER_VERSION = 3
# Method-level smells that make the dataflow / Extract Method pass worthwhile.
# Only files carrying one of these get a CFG + def/use + reaching pass built.
@@ -336,24 +338,24 @@ class HealthAnalyzer:
self._tests_reach_cache: set[str] | None = None
def _files_reached_by_tests(self) -> set[str]:
- """Non-test files that some test file reaches through the import graph.
+ """Non-test files that some test file can execute into, per the call graph.
The graph-backed half of "is this file tested". Computed once per
analyzer over the whole file set (one multi-source walk, not one per
file), because every ``_evaluate_file`` call asks the same question of
the same graph.
- Inferred, and it over-claims: a test importing a module reaches every
- file that module depends on, whether or not the test body exercises
- them. Consumers must use it only as a floor - "something tests this" -
- never as a coverage quantity. See ``analysis.test_reachability``.
+ Inferred, and it over-claims: a call edge says control *can* reach the
+ file, not that a given run did. Consumers must use it only as a floor -
+ "something tests this" - never as a coverage quantity. See
+ ``analysis.test_reachability``.
"""
if self._tests_reach_cache is None:
test_files = {
pf.file_info.path for pf in self.parsed_files if pf.file_info.is_test
}
self._tests_reach_cache = files_reached_by_tests(
- forward_dependencies_from_graph(self.graph), test_files
+ call_graph_from_graph(self.graph), test_files
)
return self._tests_reach_cache
@@ -950,10 +952,9 @@ class HealthAnalyzer:
or fcx.has_inline_tests,
# Kept separate from ``has_test_file`` on purpose. That flag means
# "a file named like this one's test exists"; this one means "the
- # graph records a test reaching this file". They disagree often -
- # measured here, the graph finds 984 files the naming convention
- # misses - and collapsing them would leave no way to say which
- # signal answered, or that one of them over-claims.
+ # call graph records a test reaching this file". They disagree
+ # often, and collapsing them would leave no way to say which signal
+ # answered, or that one of them over-claims.
reached_by_tests=file_path in self._files_reached_by_tests(),
module=module,
function_metrics=fn_metrics,
diff --git a/packages/core/src/repowise/core/analysis/pr_blast.py b/packages/core/src/repowise/core/analysis/pr_blast.py
index 1d1bcc309..af91f67a7 100644
--- a/packages/core/src/repowise/core/analysis/pr_blast.py
+++ b/packages/core/src/repowise/core/analysis/pr_blast.py
@@ -274,11 +274,12 @@ class PRBlastRadiusAnalyzer:
1. A per-test coverage row (from ``repowise coverage add``) is
execution-*proof*: never a gap.
- 2. A test file reaching it in the import graph is evidence, not proof -
- importing a module is not exercising it - but it is a recorded edge
- rather than a guess, and it finds the suites that name their tests
- for behaviour instead of for the file under test. Used only as this
- floor; nothing downstream reads a coverage figure off it.
+ 2. A test file reaching it in the dependency graph is evidence, not
+ proof - control reaching a file is not a run exercising it - but it
+ is a recorded edge rather than a guess, and it finds the suites that
+ name their tests for behaviour instead of for the file under test.
+ Used only as this floor; nothing downstream reads a coverage figure
+ off it.
3. Otherwise the filename pattern (test_<name>, <name>_test,
<name>.spec.*) - an honest "unknown", never asserted as untested.
@@ -364,11 +365,11 @@ class PRBlastRadiusAnalyzer:
a real diff.
Falls back to the graph when no coverage report has been ingested, which
- is most repositories: a test file that imports a changed file is a
+ is most repositories: a test file that reaches a changed file is a
candidate worth running. ``basis`` says which of the two answered, and
the two are never merged. A run-list the coverage map proved and one the
- import graph suggests are different claims, and averaging them would
- leave the reader unable to tell a measured test from a guessed one. The
+ graph suggests are different claims, and averaging them would leave the
+ reader unable to tell a measured test from a guessed one. The
inferred list names test *files* rather than test ids, because reaching
is a file-level fact; both forms are runnable arguments to pytest.
diff --git a/packages/core/src/repowise/core/analysis/test_reachability.py b/packages/core/src/repowise/core/analysis/test_reachability.py
index f4ce50c9b..955259c71 100644
--- a/packages/core/src/repowise/core/analysis/test_reachability.py
+++ b/packages/core/src/repowise/core/analysis/test_reachability.py
@@ -9,115 +9,260 @@ Matching filenames fails both ways, and this repository is the proof. Of its six
worst bug-magnet files, five - ``call_resolver.py``, ``dead_code/analyzer.py``,
``pipeline/persist.py``, ``tool_answer/answer.py``, ``pr_blast.py`` - have no
file named for them anywhere under ``tests/`` and so read as untested, while the
-graph names 7, 9, 23, 18 and 3 test files importing them. The sixth,
+graph names the test files that reach them. The sixth,
``analysis/health/engine.py``, is worse: the convention matches on basename
alone, so it paired with ``tests/unit/distill/test_engine.py`` - a different
engine, in a different subsystem - and called the file tested on the strength of
-a name collision. The graph names the six ``tests/unit/health/`` files that
-actually import it.
-
-The graph already knows the answer in the shape it can honestly give: a test
-file that imports a source file *reaches* it. This module reads that relation.
-It is a second signal beside the measured one, never a replacement and never
-averaged with it.
+a name collision.
+
+The graph already knows the answer. This module reads that relation. It is a
+second signal beside the measured one, never a replacement and never averaged
+with it.
+
+Which graph, and why
+--------------------
+Two graphs can answer "does a test reach this file", and they are not equally
+good at it.
+
+The **call graph** is the primary signal: seed the symbols a test file declares,
+walk ``EXECUTION_EDGE_TYPES`` forward, and every symbol reached is one the test
+can actually run. This is the closer model of the question - the claim is about
+execution, and these edges are execution.
+
+The **import graph** is the broader, weaker tier: a test file that imports a
+source file references it, which is real evidence and a much cruder one. A test
+importing a module pulls in every symbol the module defines while the test body
+may touch one function.
+
+Measured against a real ``coverage run --contexts=test`` on this repository,
+over the slice where per-test attribution is complete and both sides see the
+same 37 test files (159 production files provably executed):
+
+============================ ======= ======= ========= =======
+forward signal claims correct precision recall
+============================ ======= ======= ========= =======
+import graph, 1 hop 43 31 72.1% 19.5%
+call graph, 3 hops 48 44 91.7% 27.7%
+call graph, 3 hops, filtered 46 44 95.7% 27.7%
+both unioned 57 45 78.9% 28.3%
+============================ ======= ======= ========= =======
+
+The call graph wins on both axes, so it leads. Unioning the import graph into it
+buys 0.6 points of recall for 16.8 points of precision, and a false "something
+reaches this" suppresses a real untested-hotspot finding, so the forward walk
+does not union - it is call edges only.
+
+The reverse walk combines them differently, because there the import tier can be
+spent only on the targets the call graph said nothing about:
+
+============================= ======= ====== =========
+reverse signal targets hit precision
+============================= ======= ====== =========
+import graph, 1 hop 32 96.9% 94.8%
+call graph, 3 hops, filtered 46 100.0% 97.5%
+both unioned 47 100.0% 95.8%
+call graph, else import graph 47 100.0% 97.5%
+============================= ======= ====== =========
+
+Falling back is free where unioning is not: it answers one more target at
+identical precision, because the second tier never speaks over the first.
+
+Both tables compare the signals uncapped and over the same test set, which is
+what makes them a choice between signals rather than two different experiments.
+End to end through this module, with the walk seeing the repository's whole test
+set and ``MAX_TESTS_PER_TARGET`` applied, the reverse walk scores 100.0% hit and
+95.5% precision on the same slice; the gap is the cap truncating a repo-wide
+list, not a difference in the walk.
+
+Depth, and why 3
+----------------
+The call walk saturates at 3 hops - 3, 4 and 5 return the same 48 claims and 44
+confirmations - so the recall ceiling is the call graph's own capture rate, not
+the depth, and there is nothing to buy by walking further. The import walk keeps
+its measured default of 1: a blanket second hop was tested and recovers 20 of
+128 misses while adding 50 wrong claims, and both a facade-only second hop
+through ``__init__.py`` (3 recovered, 5 added) and ``conftest.py`` transitivity
+(zero recovered) were tested and disproved. What the import graph misses is
+transitive *execution*, which import edges structurally cannot see, which is
+what the call graph is for.
+
+Why recall reads low, and why that is not the graph's fault
+-----------------------------------------------------------
+27.7% looks poor for a graph this dense, and the reason is what the ground truth
+counts. ``coverage`` records a line as run whether a test called into it or
+Python merely evaluated the module body on import, so a file that was only
+imported is indistinguishable in the truth set from one a test drove. Splitting
+the 159 truth files by how much of what ran was *inside a function body*:
+
+================================= ======= ======= ========
+what actually ran in the file files found recall
+================================= ======= ======= ========
+nothing inside any function body 39 0 0%
+under a quarter inside bodies 19 0 0%
+a quarter to three quarters 45 3 7%
+over three quarters inside bodies 56 43 77%
+================================= ======= ======= ========
+
+A quarter of the "covered" files never ran a single line inside a function. The
+walk is right to miss those: nothing called them, and claiming otherwise is the
+import graph's error, which is what it costs 16.8 points of precision to make.
+
+Of the 13 missed in the bottom row, 11 are ``alembic/versions/*`` migrations,
+whose ``upgrade``/``downgrade`` the framework invokes by naming convention with
+no static caller anywhere in the repository. Excluding those, the walk finds 43
+of 45 files that a test genuinely exercised and that anything statically calls.
+Chasing the headline number by widening the walk trades that away.
+
+Confidence, and why one origin is dropped
+-----------------------------------------
+``graph_edges.resolution_origin`` records which strategy resolved a call edge,
+and they are not equally trustworthy. ``global_unique`` binds a name to the only
+symbol carrying it anywhere in the repository, which is a guess, and it is the
+one origin the vocabulary itself scores at 0.50. Dropping it costs no recall at
+all and buys 4.0 points of forward precision (91.7% -> 95.7%) and 1.1 of reverse
+(96.4% -> 97.5%). Nothing else earns its keep: filtering ``receiver_global`` too
+changes no number, and a confidence floor of 0.90 cuts recall from 27.7% to
+23.3%.
What it claims, and what it does not
------------------------------------
-Reaching is not executing. A test that imports a module pulls in every symbol
-the module defines, while the test body may touch one function; the inferred
-map therefore **over-claims**, and the direction of the error is known and
-one-sided:
+Reaching is not executing. A call edge says control *can* flow, not that a given
+test run did, so the inferred map **over-claims**, and the direction of the
+error is known and one-sided:
* Sound as a floor. "Some test reaches this file, so do not call it untested"
- is safe: the import is recorded, not guessed.
+ is safe: the edge is recorded, not guessed.
* Unsound as a quantity. No percentage may be derived from it. There is no line
attribution here at all - reaching is a file-level fact - so a caller that
needs "are these changed *lines* covered" must use the measured map or say it
does not know.
Every surface that consumes this labels the result ``inferred``. Nothing here
-writes to ``test_coverage``, and no row it produces is stored: the relation is
-a bounded walk over ``graph_edges``/``graph_nodes``, which are already indexed
-and already fresh. Materialising it would cost a transitive closure - on this
-repository tests reach 1,630 of 2,509 production files, so the stored form is
-O(tests x sources) rows that go stale the moment the graph moves, to answer a
-query that is a breadth-first search over data already in the database.
-
-Depth
------
-``max_depth`` bounds the chain, and the default of 1 was measured rather than
-picked. Dogfooded against a real ``coverage run --contexts=test`` on this
-repository, over a slice where per-test attribution was complete and both sides
-saw the same 38 test files:
-
-=========== ========= ========= ========== ==========
-max_depth reach reach run-list run-list
- precision recall precision hit rate
-=========== ========= ========= ========== ==========
-1 72.1% 30.4% 93.1% 96.8%
-2 40.7% 45.1% 73.8% 97.9%
-3 17.1% 45.1% - -
-=========== ========= ========= ========== ==========
-
-Both uses want precision. A false "something reaches this" suppresses a real
-untested-hotspot finding, and a false entry in a run-list sends someone to run
-a test that cannot fail for their change; the extra recall a second hop buys is
-not worth halving either. Depth 3 is where the closure starts to blow up (269
-claims for the same 46 confirmations) and is never a sensible default.
-
-Recall at depth 1 already beats the filename convention it sits beside (30.4%
-against 23.5% on the same set) while resting on a recorded edge instead of a
-name, and the two are unioned rather than ranked, so the shipped floor is
-better than either.
-
-A repository whose tests import a package facade rather than the module under
-test will want ``max_depth=2``; it is a keyword argument on both walks for
-exactly that reason.
+writes to ``test_coverage``, and no row it produces is stored: the relation is a
+bounded walk over ``graph_edges``/``graph_nodes``, which are already indexed and
+already fresh. Materialising it would cost a transitive closure that goes stale
+the moment the graph moves, to answer a query that is a breadth-first search
+over data already in the database.
"""
from __future__ import annotations
-from collections import deque
-from typing import Any
+from dataclasses import dataclass
+from typing import Any, Literal
from sqlalchemy import select, text
from sqlalchemy.ext.asyncio import AsyncSession
-from repowise.core.ingestion.models import FILE_DEPENDENCY_EDGE_TYPES
+from repowise.core.ingestion.models import (
+ EXECUTION_EDGE_TYPES,
+ FILE_DEPENDENCY_EDGE_TYPES,
+)
from repowise.core.persistence.models import GraphNode
-# How many dependency hops a test may take and still be said to reach a file.
-# 1 - a test imports the file - is the measured choice, not a conservative
-# guess; see the Depth section above for the numbers behind it.
+# How many call hops a test may take and still be said to reach a file. The walk
+# saturates here; see the Depth section above.
+DEFAULT_CALL_DEPTH = 3
+
+# How many import hops the weaker fallback tier may take. Still the measured
+# choice; a blanket second hop costs more than it buys.
DEFAULT_MAX_DEPTH = 1
+# Call edges whose resolver only matched a name. See the Confidence section.
+UNRELIABLE_CALL_ORIGINS = frozenset({"global_unique"})
+
# Cap on how many test files one target reports. The consumers all cut their
-# own lists shorter; the cap exists so a facade module imported by every test
-# in the suite cannot produce an unbounded intermediate.
+# own lists shorter; the cap exists so a helper called from every test in the
+# suite cannot produce an unbounded intermediate.
MAX_TESTS_PER_TARGET = 50
+# Which tier answered. The CLI prints this so a reader can tell "this test runs
+# into the file" from the weaker "this test imports it".
+ReachedVia = Literal["call-graph", "import-graph"]
+
__all__ = [
+ "DEFAULT_CALL_DEPTH",
"DEFAULT_MAX_DEPTH",
"MAX_TESTS_PER_TARGET",
+ "UNRELIABLE_CALL_ORIGINS",
+ "CallGraphView",
+ "ReachedBy",
+ "call_graph_from_graph",
"files_reached_by_tests",
- "forward_dependencies_from_graph",
"load_test_files",
"tests_reaching",
+ "tests_reaching_by_tier",
]
+@dataclass(frozen=True)
+class CallGraphView:
+ """The two adjacencies a call-graph reachability walk needs.
+
+ ``declares`` bridges the file layer to the symbol layer - nothing points
+ from a symbol back to a file, so without it a walk cannot start at a test
+ file. ``calls`` is the symbol-to-symbol adjacency, already origin-filtered.
+ """
+
+ declares: dict[str, set[str]]
+ calls: dict[str, set[str]]
+
+
+@dataclass(frozen=True)
+class ReachedBy:
+ """Tests reaching one target, and which tier found them."""
+
+ tests: list[str]
+ via: ReachedVia
+
+
+def _file_of(symbol_id: str) -> str:
+ """The file a symbol node belongs to. Node ids are ``path::Name``."""
+ return symbol_id.split("::", 1)[0]
+
+
+def call_graph_from_graph(graph: Any) -> CallGraphView:
+ """Build the call-reachability view from an in-memory NetworkX graph.
+
+ ``defines`` alone bridges files to symbols: measured across this repository
+ it reaches every symbol, class members included, so also walking
+ ``has_method`` would add a second containment layer for nothing.
+
+ Returns an empty view for ``None`` - the health engine runs without a graph
+ on some paths, and the documented outcome there is "no signal".
+ """
+ declares: dict[str, set[str]] = {}
+ calls: dict[str, set[str]] = {}
+ if graph is None:
+ return CallGraphView(declares, calls)
+ try:
+ edges = graph.edges(data=True)
+ except Exception:
+ return CallGraphView(declares, calls)
+ for src, dst, data in edges:
+ attrs = data or {}
+ edge_type = attrs.get("edge_type")
+ if edge_type == "defines":
+ declares.setdefault(src, set()).add(dst)
+ elif (
+ edge_type in EXECUTION_EDGE_TYPES
+ and attrs.get("resolution_origin") not in UNRELIABLE_CALL_ORIGINS
+ ):
+ calls.setdefault(src, set()).add(dst)
+ return CallGraphView(declares, calls)
+
+
def files_reached_by_tests(
- forward_deps: dict[str, set[str]],
+ view: CallGraphView,
test_files: set[str],
*,
- max_depth: int = DEFAULT_MAX_DEPTH,
+ max_depth: int = DEFAULT_CALL_DEPTH,
) -> set[str]:
- """Every non-test file some test reaches within *max_depth* hops.
+ """Every non-test file some test can execute into within *max_depth* hops.
- One multi-source breadth-first search from the whole test set at once, so
- the cost is O(V+E) however many test files there are - this is the batch
- question ("which files are reached at all"), not the attributed one.
- *forward_deps* maps a file to the files it depends on.
+ One multi-source breadth-first search from every symbol the test files
+ declare, so the cost is O(V+E) however many test files there are - this is
+ the batch question ("which files are reached at all"), not the attributed
+ one.
Returns the reached set with the test files themselves removed: a test does
not need a test.
@@ -125,43 +270,24 @@ def files_reached_by_tests(
if not test_files or max_depth < 1:
return set()
- seen = set(test_files)
- frontier = set(test_files)
+ frontier: set[str] = set()
+ for path in test_files:
+ frontier |= view.declares.get(path, frozenset())
+ seen = set(frontier)
+ # The seeds are what the tests *declare*, not what they reach. Counting them
+ # would make containment alone a claim of reaching.
+ reached: set[str] = set()
for _ in range(max_depth):
nxt: set[str] = set()
for node in frontier:
- nxt |= forward_deps.get(node, frozenset())
+ nxt |= view.calls.get(node, frozenset())
nxt -= seen
if not nxt:
break
seen |= nxt
+ reached |= nxt
frontier = nxt
- return seen - test_files
-
-
-def forward_dependencies_from_graph(graph: Any) -> dict[str, set[str]]:
- """Build the file -> depends-on map from an in-memory NetworkX graph.
-
- Restricted to :data:`FILE_DEPENDENCY_EDGE_TYPES`, the codified answer to
- "which edges mean one file depends on another". Containment edges are the
- ones that matter to exclude: with ``defines`` in, a walk leaves the file
- layer into symbol nodes and every file defining a called symbol reads as a
- dependency of the caller's *file*.
-
- Returns an empty map for ``None`` - the health engine runs without a graph
- on some paths, and the documented outcome there is "no signal".
- """
- out: dict[str, set[str]] = {}
- if graph is None:
- return out
- try:
- edges = graph.edges(data=True)
- except Exception:
- return out
- for src, dst, data in edges:
- if (data or {}).get("edge_type") in FILE_DEPENDENCY_EDGE_TYPES:
- out.setdefault(src, set()).add(dst)
- return out
+ return {_file_of(symbol) for symbol in reached} - test_files
async def load_test_files(session: AsyncSession, repo_id: str) -> set[str]:
@@ -180,56 +306,153 @@ async def tests_reaching(
repo_id: str,
targets: list[str],
*,
- max_depth: int = DEFAULT_MAX_DEPTH,
+ call_depth: int = DEFAULT_CALL_DEPTH,
+ import_depth: int = DEFAULT_MAX_DEPTH,
) -> dict[str, list[str]]:
"""Test files that reach each of *targets*, keyed by target path.
- The reverse of :func:`files_reached_by_tests`, and attributed: the walk
- runs backwards along dependency edges from the targets, carrying the seed
- each visited node came from, so the answer says *which* target each test
- guards rather than only that some test does.
-
- Attribution is why this is a separate walk rather than a filter over the
- forward one. It stays cheap because the seed set is a change's files, not
- the repository: one ``IN`` query per depth level, at most *max_depth* of
- them, which is the shape ``pr_blast._transitive_affected`` already uses.
-
Targets nothing reaches are absent from the result. An empty dict is
"no test reaches these"; the caller decides whether that reads as untested
or as unknown.
"""
+ found = await tests_reaching_by_tier(
+ session, repo_id, targets, call_depth=call_depth, import_depth=import_depth
+ )
+ return {target: reached.tests for target, reached in found.items()}
+
+
+async def tests_reaching_by_tier(
+ session: AsyncSession,
+ repo_id: str,
+ targets: list[str],
+ *,
+ call_depth: int = DEFAULT_CALL_DEPTH,
+ import_depth: int = DEFAULT_MAX_DEPTH,
+) -> dict[str, ReachedBy]:
+ """:func:`tests_reaching`, also saying which tier answered each target.
+
+ The call walk runs first; the import walk is then seeded with only the
+ targets it left unanswered, so the weaker tier never speaks over the
+ stronger one and costs nothing where the stronger one already spoke.
+
+ Both walks are attributed: they run backwards from the targets carrying the
+ seed each visited node came from, so the answer says *which* target each
+ test guards rather than only that some test does. That is why these are
+ separate walks rather than a filter over the forward one, and it stays cheap
+ because the seed set is a change's files, not the repository - one ``IN``
+ query per level, which is the shape ``pr_blast._transitive_affected``
+ already uses.
+ """
seeds = sorted({t for t in targets if t})
- if not seeds or max_depth < 1:
+ if not seeds:
return {}
test_files = await load_test_files(session, repo_id)
if not test_files:
return {}
- # node -> the seeds it was reached from. A node can serve several seeds (one
- # shared helper changed alongside its caller), and dropping that would
- # report the test against an arbitrary one of them.
+ out: dict[str, ReachedBy] = {}
+ if call_depth >= 1:
+ found = await _call_reaching(session, repo_id, seeds, test_files, call_depth)
+ for seed, tests in found.items():
+ out[seed] = ReachedBy(_cut(tests), "call-graph")
+
+ unanswered = [seed for seed in seeds if seed not in out]
+ if unanswered and import_depth >= 1:
+ found = await _import_reaching(
+ session, repo_id, unanswered, test_files, import_depth
+ )
+ for seed, tests in found.items():
+ out[seed] = ReachedBy(_cut(tests), "import-graph")
+ return out
+
+
+def _cut(tests: set[str]) -> list[str]:
+ return sorted(tests)[:MAX_TESTS_PER_TARGET]
+
+
+async def _call_reaching(
+ session: AsyncSession,
+ repo_id: str,
+ seeds: list[str],
+ test_files: set[str],
+ max_depth: int,
+) -> dict[str, set[str]]:
+ """Tests that can execute into each seed file, walking call edges backwards.
+
+ Starts one layer below the other walk: the seeds are files and call edges
+ join symbols, so the first query resolves each seed to the symbols it
+ declares, and the walk carries the seed from there.
+ """
+ declared = await _edges_from(session, repo_id, seeds, ["defines"])
+ origins: dict[str, set[str]] = {}
+ for seed, symbol in declared:
+ origins.setdefault(symbol, set()).add(seed)
+ if not origins:
+ return {}
+
+ found: dict[str, set[str]] = {}
+ frontier = list(origins)
+ for _ in range(max_depth):
+ if not frontier:
+ break
+ level, frontier = frontier, []
+ queued: set[str] = set()
+ for caller, callee in await _edges_into(
+ session,
+ repo_id,
+ level,
+ sorted(EXECUTION_EDGE_TYPES),
+ UNRELIABLE_CALL_ORIGINS,
+ ):
+ carried = origins.get(callee, frozenset())
+ if not carried:
+ continue
+ owner = _file_of(caller)
+ if owner in test_files:
+ for seed in carried:
+ found.setdefault(seed, set()).add(owner)
+ # A test is a leaf. Walking through one would let "test A calls
+ # shared helper B" drag B's unrelated targets in.
+ continue
+ known = origins.setdefault(caller, set())
+ fresh = carried - known
+ if not fresh:
+ continue
+ known |= fresh
+ if caller not in queued:
+ queued.add(caller)
+ frontier.append(caller)
+ return found
+
+
+async def _import_reaching(
+ session: AsyncSession,
+ repo_id: str,
+ seeds: list[str],
+ test_files: set[str],
+ max_depth: int,
+) -> dict[str, set[str]]:
+ """Tests that import each seed file, directly or within *max_depth* hops."""
origins: dict[str, set[str]] = {s: {s} for s in seeds}
found: dict[str, set[str]] = {}
seed_set = set(seeds)
- frontier: deque[str] = deque(seeds)
+ frontier = list(seeds)
for _ in range(max_depth):
- level = list(frontier)
- frontier.clear()
- queued: set[str] = set()
- if not level:
+ if not frontier:
break
- for dependent, dependency in await _dependents_of(session, repo_id, level):
+ level, frontier = frontier, []
+ queued: set[str] = set()
+ for dependent, dependency in await _edges_into(
+ session, repo_id, level, sorted(FILE_DEPENDENCY_EDGE_TYPES), frozenset()
+ ):
carried = origins.get(dependency, frozenset())
if not carried:
continue
if dependent in test_files:
for seed in carried:
found.setdefault(seed, set()).add(dependent)
- # A test file is a leaf for this walk. Nothing imports a test,
- # and treating one as an intermediary would let "test A imports
- # test helper B" drag B's unrelated targets in.
continue
if dependent in seed_set:
continue
@@ -244,35 +467,68 @@ async def tests_reaching(
if dependent not in queued:
queued.add(dependent)
frontier.append(dependent)
+ return found
+
- return {seed: sorted(tests)[:MAX_TESTS_PER_TARGET] for seed, tests in found.items() if tests}
+def _in_clause(prefix: str, values: list[str], params: dict[str, Any]) -> str:
+ """Bind *values* as ``:prefix0, :prefix1, ...`` and return the clause body."""
+ params.update({f"{prefix}{i}": v for i, v in enumerate(values)})
+ return ",".join(f":{prefix}{i}" for i in range(len(values)))
-async def _dependents_of(
- session: AsyncSession, repo_id: str, paths: list[str]
+async def _edges_from(
+ session: AsyncSession, repo_id: str, sources: list[str], edge_types: list[str]
) -> list[tuple[str, str]]:
- """``(dependent, dependency)`` pairs for every file that depends on *paths*.
+ """``(source, target)`` pairs leaving *sources* along *edge_types*."""
+ if not sources:
+ return []
+ params: dict[str, Any] = {"repo_id": repo_id}
+ src = _in_clause("s", sources, params)
+ ets = _in_clause("e", edge_types, params)
+ rows = await session.execute(
+ text(
+ "SELECT DISTINCT source_node_id, target_node_id FROM graph_edges "
+ "WHERE repository_id = :repo_id "
+ f"AND source_node_id IN ({src}) AND edge_type IN ({ets})"
+ ),
+ params,
+ )
+ return [(s, t) for s, t in rows]
+
+
+async def _edges_into(
+ session: AsyncSession,
+ repo_id: str,
+ targets: list[str],
+ edge_types: list[str],
+ excluded_origins: frozenset[str],
+) -> list[tuple[str, str]]:
+ """``(source, target)`` pairs arriving at *targets* along *edge_types*.
Raw text SQL with an ``IN`` list, matching ``pr_blast._transitive_affected``:
the edge table is the one place a per-level parameterised ``IN`` beats
loading every row, and the two walks should not disagree about how they read
it.
"""
- if not paths:
+ if not targets:
return []
- dep_types = sorted(FILE_DEPENDENCY_EDGE_TYPES)
- ph = ",".join(f":p{i}" for i in range(len(paths)))
- et = ",".join(f":e{i}" for i in range(len(dep_types)))
params: dict[str, Any] = {"repo_id": repo_id}
- params.update({f"p{i}": v for i, v in enumerate(paths)})
- params.update({f"e{i}": v for i, v in enumerate(dep_types)})
+ tgt = _in_clause("p", targets, params)
+ ets = _in_clause("e", edge_types, params)
+ origin_filter = ""
+ if excluded_origins:
+ # NULL means the row predates the vocabulary, not "unknown", so it has
+ # to survive the filter, and a bare NOT IN would drop it.
+ bad = _in_clause("o", sorted(excluded_origins), params)
+ origin_filter = (
+ f" AND (resolution_origin IS NULL OR resolution_origin NOT IN ({bad}))"
+ )
rows = await session.execute(
text(
- f"SELECT DISTINCT source_node_id, target_node_id FROM graph_edges "
- f"WHERE repository_id = :repo_id "
- f"AND target_node_id IN ({ph}) "
- f"AND edge_type IN ({et})"
+ "SELECT DISTINCT source_node_id, target_node_id FROM graph_edges "
+ "WHERE repository_id = :repo_id "
+ f"AND target_node_id IN ({tgt}) AND edge_type IN ({ets}){origin_filter}"
),
params,
)
- return [(src, tgt) for src, tgt in rows]
+ return [(s, t) for s, t in rows]
diff --git a/packages/core/src/repowise/core/ingestion/models.py b/packages/core/src/repowise/core/ingestion/models.py
index e8f90fcb2..2ad76d619 100644
--- a/packages/core/src/repowise/core/ingestion/models.py
+++ b/packages/core/src/repowise/core/ingestion/models.py
@@ -499,6 +499,17 @@ SYMBOL_USE_EDGE_TYPES: frozenset[str] = frozenset(
)
+# Symbol → symbol edges along which control can actually reach the target, for
+# the question "would running this test execute that code?". The reachability
+# view minus the two that record a mention rather than a transfer of control:
+# `references` is a name sitting in a dispatch table and `reads` is a field
+# access, and neither runs the thing it names. Narrower than
+# SYMBOL_USE_EDGE_TYPES on purpose — dead code asks "is this used", which a
+# mention answers, and the inferred test map asks "is this run", which it does
+# not.
+EXECUTION_EDGE_TYPES: frozenset[str] = SYMBOL_USE_EDGE_TYPES - {"references", "reads"}
+
+
# "Does anything use this symbol at all?" — the reachability view. Adds
# ``type_use`` to the symbol set: a C#-style type reference is a real use, and
# the dead-code analyzer and the C/C++ reachability pass both asked for exactly
diff --git a/packages/server/src/repowise/server/mcp_server/tool_change_risk.py b/packages/server/src/repowise/server/mcp_server/tool_change_risk.py
index 56b90a5a1..76b42c582 100644
--- a/packages/server/src/repowise/server/mcp_server/tool_change_risk.py
+++ b/packages/server/src/repowise/server/mcp_server/tool_change_risk.py
@@ -61,7 +61,7 @@ async def get_change_risk(
``impacted_tests`` names the tests for this change; ``missing_tests`` the
uncovered lines. ``basis``: ``measured`` = a coverage map proves those tests
- run the changed *lines*; ``inferred`` = test files the import graph shows
+ run the changed *lines*; ``inferred`` = test files the graph shows
reaching it (candidates, file-level, no ingest needed). No map is never
"untested": ``status`` ``no_map`` means run the suite. ``prior_fixes``
counts past fixes whose lines overlap this diff.
@@ -425,10 +425,11 @@ async def _inferred_impacted(
"""Graph-inferred candidates for a repo with no coverage map.
"Run the full suite" is correct but useless on the repositories that have
- no coverage report, which is most of them. The import graph can narrow it:
- a test file that reaches a changed file is worth running first. That is a candidate list and is labelled one - ``basis`` is
- ``"inferred"`` and ``map_present`` stays False, so nothing here can be read
- as the line-precise measured answer.
+ no coverage report, which is most of them. The dependency graph can narrow
+ it: a test file that reaches a changed file is worth running first. That is
+ a candidate list and is labelled one - ``basis`` is ``"inferred"`` and
+ ``map_present`` stays False, so nothing here can be read as the line-precise
+ measured answer.
Deliberately file-level and line-blind. Reaching carries no line
attribution, so ``missing_tests`` stays empty rather than being filled from
@@ -438,7 +439,7 @@ async def _inferred_impacted(
from repowise.core.analysis.test_reachability import tests_reaching
hint = (
- "Inferred from the import graph, not measured. For the line-precise "
+ "Inferred from the dependency graph, not measured. For the line-precise "
"answer build the map with `coverage run --contexts=test` then "
"`repowise coverage add`."
)
@@ -462,7 +463,7 @@ async def _inferred_impacted(
"total": total,
"truncated": total > _IMPACTED_TESTS_LIMIT,
"summary": (
- f"{total} test file(s) reach the changed files via the import graph"
+ f"{total} test file(s) reach the changed files in the graph"
+ (
f"; showing first {_IMPACTED_TESTS_LIMIT}"
if total > _IMPACTED_TESTS_LIMIT
diff --git a/packages/server/src/repowise/server/mcp_server/tool_risk/assessment.py b/packages/server/src/repowise/server/mcp_server/tool_risk/assessment.py
index 94df6b996..e1d08de8b 100644
--- a/packages/server/src/repowise/server/mcp_server/tool_risk/assessment.py
+++ b/packages/server/src/repowise/server/mcp_server/tool_risk/assessment.py
@@ -187,9 +187,9 @@ async def _check_test_gap(session: AsyncSession, repo_id: str, target: str) -> b
1. A per-test coverage row (from ``repowise coverage add``) is
execution-proof: never a gap.
- 2. A test file reaching it in the import graph is evidence, not proof, but a
- recorded edge rather than a guess - it catches the suites whose tests are
- named for behaviour rather than for the file under test.
+ 2. A test file reaching it in the dependency graph is evidence, not proof,
+ but a recorded edge rather than a guess - it catches the suites whose
+ tests are named for behaviour rather than for the file under test.
3. Otherwise the filename pattern (test_<name>, <name>_test, <name>.spec.*)
- an honest "unknown", never asserted as untested.
diff --git a/tests/unit/analysis/test_test_reachability.py b/tests/unit/analysis/test_test_reachability.py
index 17957f76f..7b706593b 100644
--- a/tests/unit/analysis/test_test_reachability.py
+++ b/tests/unit/analysis/test_test_reachability.py
@@ -1,10 +1,10 @@
"""Static test-to-code reachability: the pure walks over an adjacency map.
-A test file that imports a source file, directly or within ``max_depth`` hops,
-*reaches* it. Pins the edge-type filter (containment must not leak the walk into
-symbol nodes) and the measured depth default. The attributed reverse walk needs
-a database, so it lives in ``tests/unit/persistence`` beside the session
-fixture.
+A test file whose calls reach a source file, within ``max_depth`` hops,
+*reaches* it. Pins the edge-type filter (containment must bridge files to
+symbols and nothing else may), the ``resolution_origin`` filter, and the
+measured depth default. The attributed reverse walk needs a database, so it
+lives in ``tests/unit/persistence`` beside the session fixture.
"""
from __future__ import annotations
@@ -12,64 +12,99 @@ from __future__ import annotations
import pytest
from repowise.core.analysis.test_reachability import (
- DEFAULT_MAX_DEPTH,
+ DEFAULT_CALL_DEPTH,
+ CallGraphView,
+ call_graph_from_graph,
files_reached_by_tests,
- forward_dependencies_from_graph,
)
+
+def _view(declares, calls):
+ return CallGraphView(declares=declares, calls=calls)
+
+
# ---------------------------------------------------------------------------
# files_reached_by_tests: the batch "is anything testing this" walk
# ---------------------------------------------------------------------------
-def test_direct_import_is_reached():
- fwd = {"tests/test_a.py": {"src/a.py"}}
- assert files_reached_by_tests(fwd, {"tests/test_a.py"}) == {"src/a.py"}
+def test_a_test_that_calls_into_a_file_reaches_it():
+ view = _view(
+ {"tests/test_a.py": {"tests/test_a.py::test_x"}},
+ {"tests/test_a.py::test_x": {"src/a.py::run"}},
+ )
+ assert files_reached_by_tests(view, {"tests/test_a.py"}) == {"src/a.py"}
-def test_the_default_is_one_hop():
- """A second hop is opt-in, because it was measured to cost more than it buys.
+def test_the_default_is_three_hops():
+ """The call walk saturates at 3, so 3 is the default rather than a cap.
- Dogfooded against a real coverage run, depth 2 halved reach precision
- (72.1% -> 40.7%) and cut run-list precision from 93.1% to 73.8%. Repos whose
- tests import a package facade pass ``max_depth=2`` explicitly.
+ Dogfooded against a real coverage run, hops 3, 4 and 5 returned the same 48
+ claims and 44 confirmations: the recall ceiling is the call graph's own
+ capture rate, not the depth.
"""
- assert DEFAULT_MAX_DEPTH == 1
- fwd = {"tests/test_a.py": {"src/__init__.py"}, "src/__init__.py": {"src/a.py"}}
- assert files_reached_by_tests(fwd, {"tests/test_a.py"}) == {"src/__init__.py"}
- assert files_reached_by_tests(fwd, {"tests/test_a.py"}, max_depth=2) == {
- "src/__init__.py",
- "src/a.py",
+ assert DEFAULT_CALL_DEPTH == 3
+ view = _view(
+ {"t.py": {"t.py::test"}},
+ {
+ "t.py::test": {"a.py::f"},
+ "a.py::f": {"b.py::g"},
+ "b.py::g": {"c.py::h"},
+ "c.py::h": {"d.py::i"},
+ },
+ )
+ assert files_reached_by_tests(view, {"t.py"}) == {"a.py", "b.py", "c.py"}
+ assert files_reached_by_tests(view, {"t.py"}, max_depth=4) == {
+ "a.py",
+ "b.py",
+ "c.py",
+ "d.py",
}
def test_depth_bound_is_respected():
- fwd = {"t.py": {"a.py"}, "a.py": {"b.py"}, "b.py": {"c.py"}}
- assert files_reached_by_tests(fwd, {"t.py"}, max_depth=1) == {"a.py"}
- assert files_reached_by_tests(fwd, {"t.py"}, max_depth=2) == {"a.py", "b.py"}
- assert files_reached_by_tests(fwd, {"t.py"}, max_depth=3) == {"a.py", "b.py", "c.py"}
+ view = _view(
+ {"t.py": {"t.py::test"}},
+ {"t.py::test": {"a.py::f"}, "a.py::f": {"b.py::g"}},
+ )
+ assert files_reached_by_tests(view, {"t.py"}, max_depth=1) == {"a.py"}
+ assert files_reached_by_tests(view, {"t.py"}, max_depth=2) == {"a.py", "b.py"}
def test_test_files_are_excluded_from_the_result():
- # A test does not need a test, so a helper test module that another test
- # imports must not come back as a file needing coverage.
- fwd = {"tests/test_a.py": {"tests/helpers.py", "src/a.py"}}
- reached = files_reached_by_tests(fwd, {"tests/test_a.py", "tests/helpers.py"})
+ # A test does not need a test, so a helper test module another test calls
+ # must not come back as a file needing coverage.
+ view = _view(
+ {"tests/test_a.py": {"tests/test_a.py::test_x"}},
+ {"tests/test_a.py::test_x": {"tests/helpers.py::build", "src/a.py::run"}},
+ )
+ reached = files_reached_by_tests(view, {"tests/test_a.py", "tests/helpers.py"})
assert reached == {"src/a.py"}
def test_cycle_terminates():
- fwd = {"t.py": {"a.py"}, "a.py": {"b.py"}, "b.py": {"a.py"}}
- assert files_reached_by_tests(fwd, {"t.py"}, max_depth=10) == {"a.py", "b.py"}
+ view = _view(
+ {"t.py": {"t.py::test"}},
+ {"t.py::test": {"a.py::f"}, "a.py::f": {"b.py::g"}, "b.py::g": {"a.py::f"}},
+ )
+ assert files_reached_by_tests(view, {"t.py"}, max_depth=10) == {"a.py", "b.py"}
+
+
+def test_a_test_declaring_nothing_reaches_nothing():
+ # No ``defines`` edge means no way into the symbol layer, which is the
+ # documented "no signal" outcome rather than an error.
+ view = _view({}, {"t.py::test": {"a.py::f"}})
+ assert files_reached_by_tests(view, {"t.py"}) == set()
@pytest.mark.parametrize("tests,depth", [(set(), 2), ({"t.py"}, 0)])
def test_degenerate_inputs_return_empty(tests, depth):
- assert files_reached_by_tests({"t.py": {"a.py"}}, tests, max_depth=depth) == set()
+ view = _view({"t.py": {"t.py::test"}}, {"t.py::test": {"a.py::f"}})
+ assert files_reached_by_tests(view, tests, max_depth=depth) == set()
# ---------------------------------------------------------------------------
-# forward_dependencies_from_graph: the edge-type filter
+# call_graph_from_graph: the edge-type and resolution-origin filters
# ---------------------------------------------------------------------------
@@ -81,21 +116,41 @@ class _FakeGraph:
return [(s, t, d) for s, t, d in self._edges]
-def test_only_file_dependency_edges_are_followed():
- # ``defines`` is the one that matters: with containment in, the walk leaves
- # the file layer into symbol nodes, and every file defining a called symbol
- # reads as a dependency of the caller's file.
+def test_containment_bridges_files_to_symbols_and_nothing_else():
graph = _FakeGraph(
[
- ("tests/test_a.py", "src/a.py", {"edge_type": "imports"}),
("tests/test_a.py", "tests/test_a.py::test_x", {"edge_type": "defines"}),
- ("src/a.py", "src/b.py", {"edge_type": "co_changes"}),
- ("src/a.py", "src/c.py", {"edge_type": "type_use"}),
+ ("tests/test_a.py::test_x", "src/a.py::run", {"edge_type": "calls"}),
+ # Not execution: an import is the weaker tier and is walked only by
+ # the reverse fallback, and a co-change is history rather than code.
+ ("tests/test_a.py", "src/b.py", {"edge_type": "imports"}),
+ ("src/a.py", "src/c.py", {"edge_type": "co_changes"}),
+ # Naming a symbol is not running it.
+ ("src/a.py::run", "src/d.py::handler", {"edge_type": "references"}),
+ ]
+ )
+ view = call_graph_from_graph(graph)
+ assert view.declares == {"tests/test_a.py": {"tests/test_a.py::test_x"}}
+ assert view.calls == {"tests/test_a.py::test_x": {"src/a.py::run"}}
+
+
+def test_name_only_resolutions_are_dropped():
+ """``global_unique`` binds a name to the only symbol carrying it: a guess.
+
+ Dropping it cost no recall and bought 4.0 points of forward precision
+ (91.7% -> 95.7%) on the dogfooded slice.
+ """
+ graph = _FakeGraph(
+ [
+ ("t.py::test", "a.py::f", {"edge_type": "calls", "resolution_origin": "global_unique"}),
+ ("t.py::test", "b.py::g", {"edge_type": "calls", "resolution_origin": "same_file"}),
+ # NULL means the row predates the vocabulary, not "unknown origin".
+ ("t.py::test", "c.py::h", {"edge_type": "calls"}),
]
)
- fwd = forward_dependencies_from_graph(graph)
- assert fwd == {"tests/test_a.py": {"src/a.py"}, "src/a.py": {"src/c.py"}}
+ assert call_graph_from_graph(graph).calls == {"t.py::test": {"b.py::g", "c.py::h"}}
def test_missing_graph_is_no_signal_not_an_error():
- assert forward_dependencies_from_graph(None) == {}
+ view = call_graph_from_graph(None)
+ assert view.declares == {} and view.calls == {}
diff --git a/tests/unit/health/test_reachability_wiring.py b/tests/unit/health/test_reachability_wiring.py
index 4316dba51..8cbe36941 100644
--- a/tests/unit/health/test_reachability_wiring.py
+++ b/tests/unit/health/test_reachability_wiring.py
@@ -1,4 +1,4 @@
-"""The health engine reads ``reached_by_tests`` off the real graph.
+"""The health engine reads ``reached_by_tests`` off the real call graph.
The helper walks are unit-tested in ``tests/unit/analysis``; what this pins is
the wiring, because the walk can be right while the engine still hands the
@@ -34,10 +34,21 @@ def _tree(tmp_path):
]
-def _graph(*edges):
+def _graph(*, calls=False, extra=None):
+ """The tree's graph, optionally with the test's call into ``parser.py``.
+
+ Files join to their symbols by ``defines`` and symbols to each other by
+ ``calls``, so clearing the finding takes all three edges, not one.
+ """
g = nx.DiGraph()
- for src, dst in edges:
- g.add_edge(src, dst, edge_type="imports")
+ g.add_edge("tests/test_round_trips.py", "tests/test_round_trips.py::test_it", edge_type="defines")
+ g.add_edge("src/parser.py", "src/parser.py::parse", edge_type="defines")
+ if calls:
+ g.add_edge(
+ "tests/test_round_trips.py::test_it", "src/parser.py::parse", edge_type="calls"
+ )
+ if extra:
+ g.add_edge(*extra[:2], edge_type=extra[2])
# Dependents for the centrality gate; untested_hotspot wants at least four.
for i in range(6):
g.add_edge(f"src/consumer_{i}.py", "src/parser.py", edge_type="imports")
@@ -53,10 +64,11 @@ def _findings(parsed, graph):
return [f for f in report.findings if f.biomarker_type == "untested_hotspot"]
-def test_a_test_importing_the_file_clears_the_untested_hotspot(tmp_path):
+def test_a_test_calling_into_the_file_clears_the_untested_hotspot(tmp_path):
parsed = _tree(tmp_path)
- graph = _graph(("tests/test_round_trips.py", "src/parser.py"))
- assert [f for f in _findings(parsed, graph) if f.file_path == "src/parser.py"] == []
+ assert [
+ f for f in _findings(parsed, _graph(calls=True)) if f.file_path == "src/parser.py"
+ ] == []
def test_without_that_edge_the_same_tree_still_fires(tmp_path):
@@ -66,12 +78,16 @@ def test_without_that_edge_the_same_tree_still_fires(tmp_path):
assert [f.file_path for f in fired if f.file_path == "src/parser.py"] == ["src/parser.py"]
[email protected]("edge_type", ["co_changes", "defines"])
-def test_a_non_dependency_edge_does_not_clear_it(tmp_path, edge_type):
- """Changing together, or containment, is not testing."""
[email protected]("edge_type", ["co_changes", "defines", "imports", "references"])
+def test_a_non_execution_edge_does_not_clear_it(tmp_path, edge_type):
+ """Changing together, containment, importing and naming are not executing.
+
+ ``imports`` is the one that changed: it clears the finding no longer. On the
+ dogfooded slice unioning the import graph into this walk bought 0.6 points
+ of recall for 16.8 of precision, and each false clear hides a real gap.
+ """
parsed = _tree(tmp_path)
- graph = _graph()
- graph.add_edge("tests/test_round_trips.py", "src/parser.py", edge_type=edge_type)
+ graph = _graph(extra=("tests/test_round_trips.py", "src/parser.py", edge_type))
fired = [f.file_path for f in _findings(parsed, graph) if f.file_path == "src/parser.py"]
assert fired == ["src/parser.py"]
@@ -102,9 +118,8 @@ def test_the_stored_metric_agrees_with_the_biomarker(tmp_path):
disagreement, one layer further out.
"""
parsed = _tree(tmp_path)
- graph = _graph(("tests/test_round_trips.py", "src/parser.py"))
report = HealthAnalyzer(
- graph,
+ _graph(calls=True),
parsed_files=parsed,
git_meta_map={"src/parser.py": dict(_HOTSPOT_META)},
).analyze()
diff --git a/tests/unit/persistence/test_test_reachability_walk.py b/tests/unit/persistence/test_test_reachability_walk.py
index 49242b739..990ba4c59 100644
--- a/tests/unit/persistence/test_test_reachability_walk.py
+++ b/tests/unit/persistence/test_test_reachability_walk.py
@@ -2,8 +2,9 @@
Runs against real ``graph_nodes`` / ``graph_edges`` rows rather than a stub,
because the walk's whole job is reading those two tables correctly - the edge
-type filter, the depth bound, and the per-seed attribution that lets one walk
-serve every file in a diff.
+type filter, the ``resolution_origin`` filter, the depth bound, the two-tier
+fallback, and the per-seed attribution that lets one walk serve every file in a
+diff.
``tests_reaching`` is imported under an alias: pytest collects module-level
names matching ``test*``, and the unaliased import would be collected as a test
@@ -13,62 +14,137 @@ and error on its missing arguments.
from __future__ import annotations
from repowise.core.analysis.test_reachability import tests_reaching as reaching
+from repowise.core.analysis.test_reachability import tests_reaching_by_tier as by_tier
from repowise.core.persistence.models import GraphEdge, GraphNode
from tests.unit.persistence.helpers import insert_repo
async def _seed(session, repo_id, *, nodes, edges):
+ """Seed file nodes and edges. An edge is ``(src, dst, type[, origin])``."""
for path, is_test in nodes.items():
session.add(
GraphNode(
repository_id=repo_id, node_id=path, node_type="file", is_test=is_test
)
)
- for src, dst, etype in edges:
+ # Deduped: two ``_calls`` into the same source file both declare its
+ # symbol, and graph_edges is unique on (repo, src, dst, type).
+ for src, dst, etype, *origin in dict.fromkeys(
+ (e[0], e[1], e[2], e[3] if len(e) > 3 else None) for e in edges
+ ):
session.add(
GraphEdge(
repository_id=repo_id,
source_node_id=src,
target_node_id=dst,
edge_type=etype,
+ resolution_origin=origin[0] if origin else None,
)
)
await session.flush()
-async def test_names_the_tests_that_import_a_changed_file(async_session):
+def _calls(test_file, source_file, *, origin=None):
+ """The three edges that put one call from a test into a source file.
+
+ A file is joined to its symbols by ``defines`` and symbols to each other by
+ ``calls``, so the shortest real path from a test file to a source file is
+ two containment edges bridging one call edge.
+ """
+ return [
+ (test_file, f"{test_file}::test_it", "defines"),
+ (source_file, f"{source_file}::run", "defines"),
+ (f"{test_file}::test_it", f"{source_file}::run", "calls", origin),
+ ]
+
+
+async def test_names_the_tests_that_call_a_changed_file(async_session):
repo = await insert_repo(async_session)
await _seed(
async_session,
repo.id,
nodes={"tests/test_a.py": True, "tests/test_z.py": True, "src/a.py": False},
- edges=[
- ("tests/test_a.py", "src/a.py", "imports"),
- ("tests/test_z.py", "src/z.py", "imports"),
- ],
+ edges=[*_calls("tests/test_a.py", "src/a.py")],
)
assert await reaching(async_session, repo.id, ["src/a.py"]) == {
"src/a.py": ["tests/test_a.py"]
}
-async def test_finds_a_behaviour_named_test_two_hops_out(async_session):
- """Opt-in second hop, for repos whose tests go through a package facade."""
+async def test_transitive_execution_is_found(async_session):
+ """The whole reason for the call graph: a test two calls away from the file.
+
+ The import graph structurally cannot see this - the test imports the facade,
+ not the parser - and a deeper import hop was measured to add more wrong
+ claims than right ones.
+ """
repo = await insert_repo(async_session)
await _seed(
async_session,
repo.id,
nodes={"tests/test_round_trips.py": True, "src/api.py": False, "src/parser.py": False},
edges=[
- ("tests/test_round_trips.py", "src/api.py", "imports"),
- ("src/api.py", "src/parser.py", "imports"),
+ ("tests/test_round_trips.py", "tests/test_round_trips.py::test_it", "defines"),
+ ("src/api.py", "src/api.py::load", "defines"),
+ ("src/parser.py", "src/parser.py::parse", "defines"),
+ ("tests/test_round_trips.py::test_it", "src/api.py::load", "calls"),
+ ("src/api.py::load", "src/parser.py::parse", "calls"),
],
)
- result = await reaching(async_session, repo.id, ["src/parser.py"], max_depth=2)
- assert result == {"src/parser.py": ["tests/test_round_trips.py"]}
+ assert await reaching(async_session, repo.id, ["src/parser.py"]) == {
+ "src/parser.py": ["tests/test_round_trips.py"]
+ }
+ # One hop stops at the facade, so the depth bound is real.
+ assert await reaching(async_session, repo.id, ["src/parser.py"], call_depth=1) == {}
+
+
+async def test_name_only_call_resolutions_are_not_evidence(async_session):
+ """``global_unique`` matched a name repo-wide, which is a guess, not an edge
+ anyone should be sent to run a test on."""
+ repo = await insert_repo(async_session)
+ await _seed(
+ async_session,
+ repo.id,
+ nodes={"tests/test_a.py": True, "src/a.py": False},
+ edges=[*_calls("tests/test_a.py", "src/a.py", origin="global_unique")],
+ )
+ assert await reaching(async_session, repo.id, ["src/a.py"]) == {}
- # The default stops at one hop, so the facade case is not claimed by default.
- assert await reaching(async_session, repo.id, ["src/parser.py"]) == {}
+
+async def test_the_import_tier_answers_only_where_calls_are_silent(async_session):
+ """Fallback, not union: measured at 97.5% precision either way, so the
+ weaker tier is free as long as it never speaks over the stronger one."""
+ repo = await insert_repo(async_session)
+ await _seed(
+ async_session,
+ repo.id,
+ nodes={
+ "tests/test_called.py": True,
+ "tests/test_imports_only.py": True,
+ "src/called.py": False,
+ "src/imported.py": False,
+ },
+ edges=[
+ *_calls("tests/test_called.py", "src/called.py"),
+ # Also imports the file it calls: the import tier must not add
+ # itself on top of the call tier's answer.
+ ("tests/test_imports_only.py", "src/called.py", "imports"),
+ ("tests/test_imports_only.py", "src/imported.py", "imports"),
+ ],
+ )
+ result = await by_tier(
+ async_session, repo.id, ["src/called.py", "src/imported.py"]
+ )
+ assert result["src/called.py"].tests == ["tests/test_called.py"]
+ assert result["src/called.py"].via == "call-graph"
+ assert result["src/imported.py"].tests == ["tests/test_imports_only.py"]
+ assert result["src/imported.py"].via == "import-graph"
+
+ # Opting the import tier out leaves the file the call graph cannot reach.
+ calls_only = await reaching(
+ async_session, repo.id, ["src/called.py", "src/imported.py"], import_depth=0
+ )
+ assert set(calls_only) == {"src/called.py"}
async def test_attribution_is_per_changed_file(async_session):
@@ -85,10 +161,10 @@ async def test_attribution_is_per_changed_file(async_session):
"src/b.py": False,
},
edges=[
- ("tests/test_a.py", "src/a.py", "imports"),
- ("tests/test_b.py", "src/b.py", "imports"),
- ("tests/test_both.py", "src/a.py", "imports"),
- ("tests/test_both.py", "src/b.py", "imports"),
+ *_calls("tests/test_a.py", "src/a.py"),
+ *_calls("tests/test_b.py", "src/b.py"),
+ *_calls("tests/test_both.py", "src/a.py"),
+ ("tests/test_both.py::test_it", "src/b.py::run", "calls"),
],
)
assert await reaching(async_session, repo.id, ["src/a.py", "src/b.py"]) == {
@@ -98,7 +174,7 @@ async def test_attribution_is_per_changed_file(async_session):
async def test_a_test_file_is_a_leaf(async_session):
- """A shared test helper must not drag its other importers' targets in."""
+ """A shared test helper must not drag its other callers' targets in."""
repo = await insert_repo(async_session)
await _seed(
async_session,
@@ -109,13 +185,15 @@ async def test_a_test_file_is_a_leaf(async_session):
"src/a.py": False,
},
edges=[
- ("tests/helpers.py", "src/a.py", "imports"),
- # test_unrelated imports the helper, not src/a.py.
- ("tests/test_unrelated.py", "tests/helpers.py", "imports"),
+ *_calls("tests/helpers.py", "src/a.py"),
+ ("tests/test_unrelated.py", "tests/test_unrelated.py::test_it", "defines"),
+ # test_unrelated calls the helper, not src/a.py.
+ ("tests/test_unrelated.py::test_it", "tests/helpers.py::test_it", "calls"),
],
)
- result = await reaching(async_session, repo.id, ["src/a.py"])
- assert result == {"src/a.py": ["tests/helpers.py"]}
+ assert await reaching(async_session, repo.id, ["src/a.py"]) == {
+ "src/a.py": ["tests/helpers.py"]
+ }
async def test_co_change_edges_are_not_reachability(async_session):
@@ -136,7 +214,7 @@ async def test_unreached_file_is_absent_not_empty(async_session):
async_session,
repo.id,
nodes={"tests/test_a.py": True, "src/a.py": False, "src/lonely.py": False},
- edges=[("tests/test_a.py", "src/a.py", "imports")],
+ edges=[*_calls("tests/test_a.py", "src/a.py")],
)
result = await reaching(async_session, repo.id, ["src/a.py", "src/lonely.py"])
assert "src/lonely.py" not in result
@@ -148,6 +226,6 @@ async def test_repo_with_no_test_nodes_returns_empty(async_session):
async_session,
repo.id,
nodes={"src/a.py": False, "src/b.py": False},
- edges=[("src/b.py", "src/a.py", "imports")],
+ edges=[*_calls("src/b.py", "src/a.py")],
)
assert await reaching(async_session, repo.id, ["src/a.py"]) == {}
diff --git a/tests/unit/server/mcp/test_change_risk.py b/tests/unit/server/mcp/test_change_risk.py
index f373bcd46..cbb625af5 100644
--- a/tests/unit/server/mcp/test_change_risk.py
+++ b/tests/unit/server/mcp/test_change_risk.py
@@ -306,7 +306,10 @@ async def test_impacted_tests_falls_back_to_the_graph_without_a_map(tmp_path, mo
assert it["map_present"] is False
assert it["tests"] == ["tests/test_round_trips.py"]
assert it["missing_tests"]["untested_changes"] == []
- assert "import graph" in it["summary"]
+ # Answered by the import tier: there is no call edge here, which is exactly
+ # when the weaker tier is allowed to speak.
+ assert "reach the changed files in the graph" in it["summary"]
+ assert "not measured" in it["summary"]
@pytest.mark.asyncio