Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.
Python aicode-complexitycode-healthcode-intelligencecode-qualitydead-codedeveloper-toolsdocumentationgit-analyticsmcpopen-sourcepythonrefactoringstatic-analysistechnical-debt
fix(health): teach the Python perf dialect that pathlib is filesystem I/O (#1269)
Raghav Chamadiya49a1795c3f2e
diff --git a/docs/layers/CODE_HEALTH.md b/docs/layers/CODE_HEALTH.mdindex 5b6d2522d..3285dfa46 100644--- a/docs/layers/CODE_HEALTH.md+++ b/docs/layers/CODE_HEALTH.md@@ -282,6 +282,46 @@ corpus: Go 96.7%, TypeScript 100%, Python 96.2% hand-labeled precision; the`*sql.Rows.Scan` cursor FP were caught and fixed by that validation.`string_concat_in_loop` is validated at 100% (26/26) after a reset-per-iterationguard (an accumulator re-initialized each iteration is bounded, not O(n²)).+Python's filesystem lexicon covers the round-trip verbs of `pathlib`+(`read_text` / `read_bytes` / `write_text` / `write_bytes`, matched on the method+name because only a `Path` spells them and the receiver is usually a plain+local), plus `os` and `shutil` verbs gated on the literal module root. Metadata+syscalls (`exists` / `is_file` / `stat` / `glob`) are deliberately excluded: a+stat per iteration is cheap and usually correct, so flagging it would be the+obvious FP class. The module-rooted half carries a ceiling worth knowing —+the root name resolves to the *leftmost* identifier of the callee chain, so+`os.path.join(a, b).verb()` also arrives as `root == "os"`; the verb set+therefore admits no name a `str` / `list` / `dict` answers to (`replace` is+omitted for exactly this reason, at no recall cost since `os.rename` covers it).++Because `hot_path_sync_io` fires *outside* any loop, it reads a narrower set+still (`hot_path_excluded_methods`): anything whose cost is bounded by one inode+— a single file read/write, one unlink/mkdir/rename, one directory listing — is+not a finding there, while an unbounded `os.walk`, `shutil.rmtree` or+`copytree` remains one. Keeping those two sets distinct is deliberate: a+per-language lexicon and a separately-calibrated marker should not widen+together just because they share a boundary kind.++Hand-labeled precision for the Python filesystem verbs is **87.6%** (163/186+non-test findings, every one labeled rather than sampled: 86.7% on this repo,+95% on Django). Where it loses precision is worth stating exactly, because it is+not the lexicon — same-function findings are near-perfect, and every false+positive came from a *cross-function* chain hitting one of three pre-existing+limits: a **memoized callee** (`get_or_build_*` index builders, `lru_cache`, a+`if path not in text_cache` guard) where the read runs once rather than per+iteration; a **first-hop name collision** (`set.update`, `sqlalchemy.update`+matched by name); and a **pre-read fast path** whose only caller always supplies+the text, leaving the disk branch unreachable. Memoization is already on the+documented static-blind list; what this change did was make far more of those+chains reachable by making their terminal sink visible. Narrowing that is the+top follow-up for this detector.++**Actionability caveat, pillar-wide.** `io_in_loop` reports a per-iteration I/O+round-trip; it does not claim the work is avoidable. Database and network+findings are usually batchable, but filesystem ones often are not — deleting N+files genuinely needs N unlinks. On Django, 16 of 19 true filesystem findings+were irreducible per-item work and 3 were batchable. The finding is still+correct and still tells you where the time goes; it is not, on its own, a defect.`nested_loop_quadratic` now fires only on a same-collection shape (two nestedloops over the same collection = all-pairs O(n²)) instead of raw nesting depth;that makes it precision-safe-by-construction but rare, so it stays advisory-only,diff --git a/packages/core/src/repowise/core/analysis/health/complexity/perf_walk.py b/packages/core/src/repowise/core/analysis/health/complexity/perf_walk.pyindex 968013390..d4c64a1b8 100644--- a/packages/core/src/repowise/core/analysis/health/complexity/perf_walk.py+++ b/packages/core/src/repowise/core/analysis/health/complexity/perf_walk.py@@ -394,13 +394,17 @@ def _collect_perf_hits(and not awaitedand misc[1] is Noneand kind in _HOT_PATH_SINK_KINDS+ and method not in dialect.hot_path_excluded_methods):# An inherently-blocking (non-awaited subprocess / fs /# sync-network) sink outside any loop. Noisy everywhere,# so record it as a fact; the engine emits# ``hot_path_sync_io`` only for a hot, request-reachable# function (centrality gate). ``db`` is excluded — see- # ``_HOT_PATH_SINK_KINDS``.+ # ``_HOT_PATH_SINK_KINDS``; point-sized reads/writes are+ # excluded per-dialect — see ``hot_path_excluded_methods``+ # (outside a loop their cost is bounded, so they belong to+ # the loop markers only).misc[1] = kindmisc[2] = lineif do_lock_io and lock_depth >= 1:diff --git a/packages/core/src/repowise/core/analysis/health/perf/dialects/base.py b/packages/core/src/repowise/core/analysis/health/perf/dialects/base.pyindex 6a256a773..cf7abbbf9 100644--- a/packages/core/src/repowise/core/analysis/health/perf/dialects/base.py+++ b/packages/core/src/repowise/core/analysis/health/perf/dialects/base.py@@ -108,6 +108,19 @@ class BasePerfDialect:string_literal_kinds: frozenset[str] = frozenset()aug_assign_kinds: frozenset[str] = frozenset()+ #: Sink methods that are genuine I/O boundaries but must NOT make a function+ #: a ``hot_path_sync_io`` candidate.+ #:+ #: That marker fires OUTSIDE any loop, so it only earns its keep on work+ #: whose cost is unbounded — spawning a subprocess, walking a whole tree.+ #: It was gated at 11/11 back when a dialect's ``filesystem`` kind meant+ #: essentially one call, so widening a per-language filesystem lexicon+ #: silently widens this separately-calibrated marker too. Listing the+ #: point-sized reads/writes here keeps the two decoupled: they stay real+ #: sinks for every loop-based marker (where per-iteration cost is the whole+ #: point) while a single bounded file touch in a hot function stays quiet.+ hot_path_excluded_methods: frozenset[str] = frozenset()+# -- callee extraction (the per-grammar seam) -----------------------------def callee_root_name(self, call_node: Node) -> str | None:diff --git a/packages/core/src/repowise/core/analysis/health/perf/dialects/python.py b/packages/core/src/repowise/core/analysis/health/perf/dialects/python.pyindex e2a7ac06f..02b9ad34b 100644--- a/packages/core/src/repowise/core/analysis/health/perf/dialects/python.py+++ b/packages/core/src/repowise/core/analysis/health/perf/dialects/python.py@@ -42,6 +42,70 @@ PY_SUBPROC_METHODS: frozenset[str] = frozenset({"run", "call", "check_call", "check_output", "Popen"})+# Filesystem round-trips, in two strata for the same reason the DB verbs are+# stratified: how much evidence the name alone carries.+#+# ``PY_FS_METHODS`` are matched on the method name alone, because in this+# ecosystem only a ``pathlib.Path`` spells them — the same reasoning the TS+# dialect applies to ``readFileSync``. That matters because the dominant real+# shape binds the path to a local first (``src_path.read_text()``), so the+# receiver is an ordinary variable and ``io_names`` cannot classify it; keying+# on the module would miss nearly every genuine site. The Kotlin dialect+# deliberately excludes its ``readText`` / ``readBytes`` homonyms because+# kotlinx-io and Ktor mirror those names on in-memory buffers; Python has no+# such homonym (``io.StringIO`` and every stream type spell it ``.read()``),+# which is what makes the bare-name match safe here and not there.+#+# Metadata syscalls (``exists`` / ``is_file`` / ``is_dir`` / ``stat`` /+# ``glob``) are deliberately absent. A stat in a loop is cheap and usually+# correct, so flagging it is the obvious false-positive class — the same call+# the C++ dialect makes when it excludes buffered stdio.+PY_FS_METHODS: frozenset[str] = frozenset(+ {"read_text", "read_bytes", "write_text", "write_bytes"}+)+# ``os`` and ``shutil`` verbs, gated on the literal module root the way+# ``subprocess`` is. ``os`` is far too broad to classify wholesale (``os.environ``+# / ``os.getpid`` / ``os.path.join`` are not I/O), so it never goes through+# ``io_names``; the root name plus an explicit verb set is the whole gate.+#+# CEILING: ``callee_root_name`` resolves the LEFTMOST identifier of the whole+# callee chain, so ``os.path.join(a, b).verb()`` also arrives here as+# ``root == "os"``. The verb set must therefore hold no name that a ``str`` /+# ``list`` / ``dict`` also answers to. ``replace`` was the one that did, and it+# was a live false positive on the canonical Windows path idiom+# (``os.path.join(root, name).replace("\\", "/")`` — five instances across the+# OSS corpus, each yielding a bogus ``io_in_loop`` AND a ``nested_loop_with_io``).+# It is omitted at near-zero recall cost: ``os.replace`` is atomic ``os.rename``,+# which is already listed. Upgrade path if this set ever needs a colliding verb:+# thread the callee's dotted segment count into ``sink_kind`` and require+# exactly two, which is the real invariant being approximated here.+PY_OS_FS_METHODS: frozenset[str] = frozenset(+ {+ "walk",+ "listdir",+ "scandir",+ "remove",+ "unlink",+ "rename",+ "makedirs",+ "mkdir",+ "rmdir",+ "removedirs",+ }+)+PY_SHUTIL_FS_METHODS: frozenset[str] = frozenset(+ {+ "copy",+ "copy2",+ "copyfile",+ "copytree",+ "move",+ "rmtree",+ "make_archive",+ "unpack_archive",+ }+)+# Python string-literal node kinds (f-strings parse as ``string`` too)._PY_STRING_KINDS: frozenset[str] = frozenset({"string", "concatenated_string"})_PY_AUG_ASSIGN_KINDS: frozenset[str] = frozenset({"augmented_assignment"})@@ -118,6 +182,19 @@ class PythonPerfDialect(BasePerfDialect):string_literal_kinds = _PY_STRING_KINDSaug_assign_kinds = _PY_AUG_ASSIGN_KINDS+ # Everything whose cost is bounded by ONE inode: a single file read/write, a+ # single unlink/mkdir/rename, one directory listing. Outside a loop that is+ # ordinary work, not a latency risk, so it must not reach ``hot_path_sync_io``+ # (see ``BasePerfDialect.hot_path_excluded_methods``). What survives is the+ # unbounded set — a whole-tree walk, a recursive copy/delete, an archive+ # round-trip — plus the bare ``open(...)`` builtin the marker was originally+ # calibrated on. Keyed on the method name alone, so a same-named non-``os``+ # receiver is excluded too: recall-only, never a false positive.+ hot_path_excluded_methods = (+ PY_FS_METHODS+ | (PY_OS_FS_METHODS - {"walk"})+ | (PY_SHUTIL_FS_METHODS - {"copytree", "rmtree", "make_archive", "unpack_archive"})+ )def sink_kind(self,@@ -159,6 +236,16 @@ class PythonPerfDialect(BasePerfDialect):return "subprocess"if root == "open" and method == "open": # bare ``open(...)`` builtinreturn "filesystem"+ if is_attribute and method in PY_FS_METHODS:+ # ``p.read_text()`` / ``Path(x).write_bytes(...)``. ``is_attribute``+ # mirrors the DB branches: pathlib always spells these as a member+ # call, so requiring it costs no recall and drops a bare local+ # helper that happens to share the name.+ return "filesystem"+ if root == "os" and method in PY_OS_FS_METHODS:+ return "filesystem"+ if root == "shutil" and method in PY_SHUTIL_FS_METHODS:+ return "filesystem"if method == "urlopen":return "network"if is_attribute and root_kind == "network" and method in HTTP_VERBS:diff --git a/tests/unit/health/test_perf_dialects.py b/tests/unit/health/test_perf_dialects.pyindex e67b3f47e..0097527df 100644--- a/tests/unit/health/test_perf_dialects.py+++ b/tests/unit/health/test_perf_dialects.py@@ -502,6 +502,165 @@ def test_python_pandas_iterrows_in_loop():assert not any(k == "pandas_iterrows_in_loop" for k, _ in _hits("python", plain))+def test_python_pathlib_round_trips_are_filesystem_sinks():+ """``p.read_text()`` in a loop is per-iteration file I/O.++ Python's filesystem lexicon recognised only the bare ``open(...)`` builtin,+ so the idiom the whole ecosystem actually uses was invisible: a loop full of+ ``Path.read_text()`` scored a perfect 10/10 on the performance dimension.+ Matched on the method name alone because only a ``pathlib.Path`` spells+ these — the receiver is usually a plain local (``src_path.read_text()``),+ so there is no import to classify.+ """+ for method in ("read_text", "read_bytes", "write_text", "write_bytes"):+ src = f"def f(ps):\n for p in ps:\n p.{method}()\n"+ assert ("io_in_loop", "filesystem") in _hits("python", src), method+ # The constructor-receiver form resolves the same way.+ ctor = "from pathlib import Path\ndef f(ps):\n for p in ps:\n Path(p).read_text()\n"+ assert ("io_in_loop", "filesystem") in _hits("python", ctor)+++def test_python_metadata_syscalls_are_not_filesystem_sinks():+ """``exists`` / ``stat`` / ``glob`` in a loop are cheap and usually correct.++ The deliberate precision boundary: flagging a metadata probe per iteration+ is the obvious false-positive class, so only round-trip verbs are sinks.+ Mirrors the C++ dialect excluding buffered stdio.+ """+ for method in ("exists", "is_file", "is_dir", "stat", "glob"):+ src = f"def f(ps):\n for p in ps:\n p.{method}()\n"+ assert not any(k == "io_in_loop" for k, _ in _hits("python", src)), method+++def test_python_os_and_shutil_verbs_gated_on_the_module_root():+ """``os``/``shutil`` verbs need the literal module root, never ``io_names``.++ ``os`` is far too broad to classify wholesale, so the root name plus an+ explicit verb set is the whole gate — the same shape ``subprocess`` uses.+ """+ assert ("io_in_loop", "filesystem") in _hits(+ "python", "import os\ndef f(ps):\n for p in ps:\n os.listdir(p)\n"+ )+ assert ("io_in_loop", "filesystem") in _hits(+ "python", "import shutil\ndef f(ps):\n for p in ps:\n shutil.rmtree(p)\n"+ )+ # Non-I/O members of the same module never fire.+ assert not any(+ k == "io_in_loop"+ for k, _ in _hits("python", "import os\ndef f(ps):\n for p in ps:\n os.getpid()\n")+ )+ # ``remove`` / ``move`` / ``replace`` are ordinary collection verbs off the+ # module root, so an unrelated receiver must stay silent.+ for src in (+ "def f(xs, q):\n for x in xs:\n q.remove(x)\n",+ "def f(xs, q):\n for x in xs:\n q.replace(x, 1)\n",+ ):+ assert not any(k == "io_in_loop" for k, _ in _hits("python", src))+++def test_python_chained_os_call_is_not_a_filesystem_sink():+ """``os.path.join(a, b).replace(...)`` is string work, not file I/O.++ ``callee_root_name`` resolves the LEFTMOST identifier of the whole callee+ chain, so a call on the *result* of an ``os.*`` expression also arrives at+ ``sink_kind`` as ``root == "os"``. The canonical Windows path idiom below+ therefore looked like a filesystem sink, and produced both a bogus+ ``io_in_loop`` and a bogus (un-gated) ``nested_loop_with_io``. The verb set+ must hold no name a ``str`` / ``list`` / ``dict`` also answers to.+ """+ src = (+ "import os\n"+ "def f(top):\n"+ " for root, dirs, files in os.walk(top):\n"+ " for name in files:\n"+ " rel = os.path.join(root, name).replace('\\\\', '/')\n"+ )+ hits = _hits("python", src)+ assert not any(k == "io_in_loop" for k, _ in hits), hits+ assert not any(k == "nested_loop_with_io" for k, _ in hits), hits+ # Other chained-root shapes off the same ceiling.+ for expr in ("os.environ.get(k).replace('a', 'b')", "os.path.relpath(p).replace('a', 'b')"):+ assert not any(+ k == "io_in_loop" for k, _ in _hits("python", f"import os\ndef f(xs):\n for x in xs:\n {expr}\n")+ ), expr+ # The genuine two-segment call is unaffected.+ assert ("io_in_loop", "filesystem") in _hits(+ "python", "import os\ndef f(ps):\n for p in ps:\n os.rename(p, p)\n"+ )+++def test_python_point_reads_do_not_become_hot_path_candidates():+ """Widening the fs lexicon must not widen ``hot_path_sync_io`` with it.++ That marker fires outside any loop, so it was gated (11/11) on unbounded+ work: subprocess spawns and whole-tree walks. A single ``p.write_text()``+ in a hot function is bounded and ordinary. Without the decoupling this+ change alone added 118 such candidates on the dogfood corpus.+ """++ def hot_fact(src):+ fc = walk_file("t.py", "python", src.encode())+ return [f.blocking_sink_kind for f in fc.perf_fn_facts if f.blocking_sink_kind]++ # Point-sized reads/writes: real sinks, but not hot-path candidates.+ for method in ("read_text", "read_bytes", "write_text", "write_bytes"):+ assert hot_fact(f"def f(p):\n p.{method}()\n") == [], method+ # Single-inode ``os`` / ``shutil`` work is the same bounded cost class and+ # must be excluded too — 177 such candidates on the OSS corpus otherwise+ # came through the door this attribute exists to close.+ for expr in (+ "os.remove(p)",+ "os.unlink(p)",+ "os.makedirs(p)",+ "os.mkdir(p)",+ "os.rmdir(p)",+ "os.rename(p, p)",+ "os.listdir(p)",+ "os.scandir(p)",+ "shutil.copyfile(p, p)",+ "shutil.copy(p, p)",+ "shutil.move(p, p)",+ ):+ src = f"import os\nimport shutil\ndef f(p):\n {expr}\n"+ assert hot_fact(src) == [], expr+ # Unbounded work keeps its candidacy, as does the bare ``open`` builtin the+ # marker was originally calibrated on.+ for expr in ("os.walk(p)", "shutil.rmtree(p)", "shutil.copytree(p, p)"):+ src = f"import os\nimport shutil\ndef f(p):\n {expr}\n"+ assert hot_fact(src) == ["filesystem"], expr+ assert hot_fact("def f(p):\n open(p).read()\n") == ["filesystem"]+ assert hot_fact("import subprocess\ndef f(p):\n subprocess.run([p])\n") == ["subprocess"]+ # Excluding a verb from hot-path candidacy must NOT cost it the bare-sink+ # fact the cross-function N+1 bridge runs on.+ fc = walk_file("t.py", "python", b"import os\ndef f(p):\n os.remove(p)\n")+ assert [f.bare_sink_kind for f in fc.perf_fn_facts] == ["filesystem"]+ # Still a per-iteration finding when it IS in a loop.+ assert ("io_in_loop", "filesystem") in _hits(+ "python", "def f(ps):\n for p in ps:\n p.write_text('x')\n"+ )+++def test_python_filesystem_sink_reaches_the_cross_function_bridge():+ """A bare fs sink makes its function a cross-function reachability target.++ The shape that motivated this: a per-file loop calls a small reader helper+ that holds the actual ``read_bytes``. The loop and the I/O are in different+ functions, so only the ``PerfFnFacts.bare_sink_kind`` fact can carry it.+ """+ src = (+ "from pathlib import Path\n"+ "def read_source(p):\n"+ " return Path(p).read_bytes()\n"+ "def scan(paths):\n"+ " for p in paths:\n"+ " read_source(p)\n"+ )+ fc = walk_file("t.py", "python", src.encode())+ facts = {f.function: f for f in fc.perf_fn_facts}+ assert facts["read_source"].bare_sink_kind == "filesystem"+ assert "read_source" in dict(facts["scan"].loop_call_targets)++def test_ts_json_parse_in_loop():src = "function f(xs){ for (const x of xs) { const c = JSON.parse(JSON.stringify(x)); } }"assert ("json_parse_in_loop", "") in _hits("typescript", src)