Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.
Python aicode-complexitycode-healthcode-intelligencecode-qualitydead-codedeveloper-toolsdocumentationgit-analyticsmcpopen-sourcepythonrefactoringstatic-analysistechnical-debt
feat(cobol): resolve COPY and EXEC SQL INCLUDE members to copybooks (#4435)
Raghav Chamadiya954411857c49
diff --git a/docs/architecture/language-support.md b/docs/architecture/language-support.mdindex 41610f774..2cbc873bd 100644--- a/docs/architecture/language-support.md+++ b/docs/architecture/language-support.md@@ -728,7 +728,7 @@ dependency surface their syntax supports.| Good | **PHP** | `.php` | `use` declarations through PSR-4 from every `composer.json`, same-namespace and fully qualified class references || Good | **Dart** | `.dart` | `import` / `export` / `part`, `package:` via every `pubspec.yaml` || Good | **Object Pascal** | `.pas` `.pp` `.dpr` `.dpk` `.lpr` | `uses` clauses via unit-name to file-stem match; project files as entry points |-| Good | **COBOL** | `.cbl` `.cob` `.cobol` `.cpy` | Literal `CALL` and `PERFORM` targets; dynamic calls and `COPY` stay silent |+| Good | **COBOL** | `.cbl` `.cob` `.cobol` `.cpy` | `COPY` members to copybooks; literal `CALL` and `PERFORM` targets; dynamic calls stay silent || Good | **GDScript** | `.gd` | `preload` / `load` / `extends "res://..."` from the nearest `project.godot` || Good | **VB.NET** | `.vb` | `Imports` through the same MSBuild project index C# uses || Good | **Elixir** | `.ex` `.exs` | `alias` / `import` / `require` / `use` against a `defmodule` index, Mix path convention as fallback |@@ -1067,8 +1067,8 @@ point. A bundle two sfdx projects declare goes to the nearer one; a tie gets noboth, since props are set by the parent as markup attributes.- **Object Pascal type kinds collapse** to `kind="class"`, and its `extends`vs `implements` split follows the `I`-prefix naming convention.-- **COBOL is COBOL-85 shaped.** `COPY` does not create an edge; dynamic program- names and dialect extensions stay silent.+- **COBOL dynamic program names stay silent**, and free-format source is read+ without column handling.- **Razor has no import edges**, because every directive is blanked by theprojection.- **GDScript string dispatch is unresolved**, and a script without `class_name`diff --git a/docs/layers/LANGUAGE_SUPPORT.md b/docs/layers/LANGUAGE_SUPPORT.mdindex b011e3708..c3cf6c62c 100644--- a/docs/layers/LANGUAGE_SUPPORT.md+++ b/docs/layers/LANGUAGE_SUPPORT.md@@ -68,7 +68,7 @@ listed per language and framework inEvery Full and Good language has a dedicated import resolver except ObjectPascal and Objective-C, which resolve through a generic unit-name or header-stem-match, and COBOL, which resolves literal program calls instead of imports.+match.Scala, Elixir and F# have dedicated import resolvers with known gaps; see[Limits](#limits).@@ -79,7 +79,8 @@ The COBOL and Apex grammars download on first use; offline, setwhether it is cached.COBOL is read in fixed reference format, with `EXEC SQL/CICS/DLI` blocks and`COPY` statements masked before parsing; a copybook parses as a data or-procedure fragment.+procedure fragment. `COPY` and `EXEC SQL INCLUDE` members resolve to the+nearest copybook with that name; system copybooks such as `DFHAID` stay external.Apex (`.cls`, `.trigger`, `.apex`) is parsed for symbols, heritage and calls,resolved case-insensitively within each sfdx project's namespace; its dead-codefindings are review-only. Apex test coverage from the Salesforce CLI is read as a@@ -275,8 +276,7 @@ Per-marker detail: [CODE_HEALTH.md](CODE_HEALTH.md).- Razor has no import edges.- Scala import resolution is partial, and the Elixir and F# import resolvershave known gaps.-- COBOL support is COBOL-85 shaped: `COPY`, dynamic program names and dialect- extensions produce no edges.+- COBOL dynamic program names and free-format source produce no edges.- C has no performance risk support.The full list per language, with the mechanics behind each gap, is in@@ -291,7 +291,7 @@ The full list per language, with the mechanics behind each gap, is in| Scala, Ruby | Extract Method (dataflow dialect) || GDScript | Health markers (complexity, performance, dataflow) || Object Pascal | A dedicated `uses` resolver |-| COBOL | Copybook resolution, dialect coverage, health markers |+| COBOL | Dialect coverage, health markers || VB.NET, Elixir | Health markers || F#, Objective-C | Health markers beyond complexity || SQL / dbt | Column-level blast radius |diff --git a/packages/core/src/repowise/core/ingestion/cobol/copybooks.py b/packages/core/src/repowise/core/ingestion/cobol/copybooks.pynew file mode 100644index 000000000..82c7eec89--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/cobol/copybooks.py@@ -0,0 +1,34 @@+"""``COPY`` and ``EXEC SQL INCLUDE`` statements as graph imports."""++from __future__ import annotations++from ..models import Import+from .source import code_view+from .statements import copy_statements+++def copy_imports(source: bytes) -> list[Import]:+ """One import per copied member, read from the unmasked source."""+ imports: list[Import] = []+ seen: set[str] = set()+ for statement in copy_statements(code_view(source)):+ # ``LIB/MEMBER`` lets the resolver prefer a library directory.+ module = (+ f"{statement.library}/{statement.member}" if statement.library else statement.member+ )+ if module in seen:+ continue+ seen.add(module)+ verb = "EXEC SQL INCLUDE" if statement.include else "COPY"+ library = f" OF {statement.library}" if statement.library else ""+ imports.append(+ Import(+ raw_statement=f"{verb} {statement.member}{library}",+ module_path=module,+ # A copybook is textually included: every name in it is in scope.+ imported_names=["*"],+ is_relative=False,+ resolved_file=None,+ )+ )+ return importsdiff --git a/packages/core/src/repowise/core/ingestion/language_configs.py b/packages/core/src/repowise/core/ingestion/language_configs.pyindex a02b02b64..3e1a66e20 100644--- a/packages/core/src/repowise/core/ingestion/language_configs.py+++ b/packages/core/src/repowise/core/ingestion/language_configs.py@@ -674,7 +674,7 @@ LANGUAGE_CONFIGS: dict[str, LanguageConfig] = {"paragraph_header": "function","data_description": "variable",},- # COPY/include resolution is intentionally outside this good-tier cut.+ # COPY is masked before parsing; parse_file reads it from the source.import_node_types=[],export_node_types=[],visibility_fn=public_by_default,diff --git a/packages/core/src/repowise/core/ingestion/languages/specs/cobol.py b/packages/core/src/repowise/core/ingestion/languages/specs/cobol.pyindex 83e809719..881601064 100644--- a/packages/core/src/repowise/core/ingestion/languages/specs/cobol.py+++ b/packages/core/src/repowise/core/ingestion/languages/specs/cobol.py@@ -16,7 +16,7 @@ SPEC = LanguageSpec(# JCL, schedulers and dynamic CALL data-items invoke programs outside the# source graph. Without those inputs, unused/unreachable claims are unsound.dead_code_exempt=True,- import_support="none",+ import_support="full",entry_point_patterns=(),manifest_files=(),shebang_tokens=(),diff --git a/packages/core/src/repowise/core/ingestion/parser/__init__.py b/packages/core/src/repowise/core/ingestion/parser/__init__.pyindex e6a8a611e..248c26a56 100644--- a/packages/core/src/repowise/core/ingestion/parser/__init__.py+++ b/packages/core/src/repowise/core/ingestion/parser/__init__.py@@ -167,12 +167,14 @@ class ASTParser:or _is_fsharp_signature(file_info)):return self.parse_file(file_info, source).imports+ original_source = sourcesource = prepare_source(lang, source, path=file_info.path)root, _, grammar_tag, language = _parse_tree(lang, language, grammar_tag, source)query = _load_compiled_query(lang, grammar_tag)matches = _run_query(query, root) if query is not None else []src = source.decode("utf-8", errors="replace")- return _extract_imports(matches, config, file_info, src)+ imports = _extract_imports(matches, config, file_info, src)+ return imports + _masked_imports(lang, original_source)def parse_file(self, file_info: FileInfo, source: bytes) -> ParsedFile:"""Parse *source* bytes and return a fully populated ParsedFile."""@@ -221,6 +223,7 @@ class ASTParser:)_restore_kotlin_annotations(symbols, recovered_annotations)imports = _extract_imports(matches, config, file_info, src)+ imports.extend(_masked_imports(lang, original_source))calls = _extract_calls(matches, config, file_info, src, symbols)# An SFC's markup tags (``<Foo />``) are component calls the TS buffer lost.calls.extend(component_call_sites(lang, original_source, symbols))@@ -293,6 +296,15 @@ def _parse_without_grammar()+def _masked_imports(lang: str, original_source: bytes) -> list[Import]:+ """Imports ``prepare_source`` masked from the grammar, read from the original bytes."""+ if lang == "cobol":+ from ..cobol.copybooks import copy_imports++ return copy_imports(original_source)+ return []++def _warn_symbol_bloat(file_info: FileInfo, symbols: list[Symbol]) -> None:if len(symbols) > _SYMBOL_COUNT_WARN_THRESHOLD:log.warning(diff --git a/packages/core/src/repowise/core/ingestion/resolvers/__init__.py b/packages/core/src/repowise/core/ingestion/resolvers/__init__.pyindex a580e99e3..4fc36ca9e 100644--- a/packages/core/src/repowise/core/ingestion/resolvers/__init__.py+++ b/packages/core/src/repowise/core/ingestion/resolvers/__init__.py@@ -5,6 +5,7 @@ from __future__ import annotationsfrom collections.abc import Callablefrom .clojure import resolve_clojure_import+from .cobol import resolve_cobol_importfrom .context import ResolverContextfrom .cpp import resolve_cpp_importfrom .csharp import resolve_csharp_import@@ -81,6 +82,8 @@ _RESOLVERS: dict[str, ResolverFn] = {"html": resolve_html_asset,# VB.NET Imports resolve through the same DotNetProjectIndex as C#."vbnet": resolve_vbnet_import,+ # COPY / EXEC SQL INCLUDE members, matched to copybooks by stem.+ "cobol": resolve_cobol_import,}diff --git a/packages/core/src/repowise/core/ingestion/resolvers/cobol.py b/packages/core/src/repowise/core/ingestion/resolvers/cobol.pynew file mode 100644index 000000000..3cd1919ab--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/resolvers/cobol.py@@ -0,0 +1,47 @@+"""COBOL ``COPY`` member resolution against copybook files in the repo.++A member resolves case-insensitively to a COBOL file with that stem, a+copybook extension first, then one under the named library, then the one+nearest the copying file. An unresolved member (``DFHAID``, a vendor or+system copybook) becomes an external node, never a guess.+"""++from __future__ import annotations++import posixpath+from typing import TYPE_CHECKING++from ..languages.registry import REGISTRY++if TYPE_CHECKING:+ from .context import ResolverContext++_COPYBOOK_SUFFIXES = frozenset({".cpy"})+++def resolve_cobol_import(module_path: str, importer_path: str, ctx: ResolverContext) -> str | None:+ library, _, member = module_path.rpartition("/")+ spec = REGISTRY.get("cobol")+ extensions = spec.extensions if spec else _COPYBOOK_SUFFIXES+ candidates = [+ path+ for path in ctx.stem_map.get(member.lower(), ())+ if posixpath.splitext(path)[1].lower() in extensions and path != importer_path+ ]+ if not candidates:+ return ctx.add_external_node(f"copybook:{member}")+ importer_dir = posixpath.dirname(importer_path).split("/")+ order = {path: index for index, path in enumerate(candidates)}++ def rank(path: str) -> tuple[int, int, int, int]:+ parts = path.split("/")+ shared = 0+ for a, b in zip(parts[:-1], importer_dir, strict=False):+ if a != b:+ break+ shared += 1+ in_library = bool(library) and library.lower() in (p.lower() for p in parts[:-1])+ is_copybook = posixpath.splitext(path)[1].lower() in _COPYBOOK_SUFFIXES+ return (not is_copybook, not in_library, -shared, order[path])++ return min(candidates, key=rank)diff --git a/tests/unit/ingestion/test_cobol_resolver.py b/tests/unit/ingestion/test_cobol_resolver.pynew file mode 100644index 000000000..73f24d453--- /dev/null+++ b/tests/unit/ingestion/test_cobol_resolver.py@@ -0,0 +1,69 @@+"""COBOL ``COPY`` members resolve to copybooks by stem, nearest first, never by guess."""++from __future__ import annotations++import networkx as nx++from repowise.core.ingestion.cobol.copybooks import copy_imports+from repowise.core.ingestion.graph import build_stem_map+from repowise.core.ingestion.resolvers.cobol import resolve_cobol_import+from repowise.core.ingestion.resolvers.context import ResolverContext+++def _ctx(*paths: str) -> ResolverContext:+ path_set = set(paths)+ return ResolverContext(path_set=path_set, stem_map=build_stem_map(path_set), graph=nx.DiGraph())+++def test_a_member_resolves_case_insensitively_to_its_copybook() -> None:+ ctx = _ctx("app/cbl/COSGN00C.cbl", "app/cpy/cocom01y.cpy")++ assert resolve_cobol_import("COCOM01Y", "app/cbl/COSGN00C.cbl", ctx) == "app/cpy/cocom01y.cpy"+++def test_a_copybook_beats_a_bms_map_or_program_with_the_same_stem() -> None:+ ctx = _ctx("app/bms/COSGN00.bms", "app/cpy-bms/COSGN00.CPY", "app/cbl/COSGN00.cbl", "a/X.cbl")++ assert resolve_cobol_import("COSGN00", "a/X.cbl", ctx) == "app/cpy-bms/COSGN00.CPY"+++def test_the_copybook_nearest_the_copying_program_wins() -> None:+ ctx = _ctx("app/cpy/CVACT01Y.cpy", "app/sub/cpy/CVACT01Y.cpy", "app/sub/cbl/P.cbl")++ assert resolve_cobol_import("CVACT01Y", "app/sub/cbl/P.cbl", ctx) == "app/sub/cpy/CVACT01Y.cpy"+++def test_a_named_library_directory_is_preferred() -> None:+ ctx = _ctx("lib1/MEMBER.cpy", "LIB2/MEMBER.cpy", "src/P.cbl")++ assert resolve_cobol_import("LIB2/MEMBER", "src/P.cbl", ctx) == "LIB2/MEMBER.cpy"+++def test_a_system_copybook_is_external_not_guessed() -> None:+ ctx = _ctx("app/cbl/P.cbl", "app/docs/DFHAID.md")++ assert resolve_cobol_import("DFHAID", "app/cbl/P.cbl", ctx) == "external:copybook:DFHAID"+ assert "external:copybook:DFHAID" in ctx.graph+++def test_copy_and_sql_include_become_one_import_per_member() -> None:+ source = (+ b" IDENTIFICATION DIVISION.\n"+ b" PROGRAM-ID. P.\n"+ b" DATA DIVISION.\n"+ b" WORKING-STORAGE SECTION.\n"+ b" COPY COCOM01Y.\n"+ b" COPY COCOM01Y.\n"+ b" COPY MEMBER1 OF MYLIB.\n"+ b" COPY MEMBER1 OF OTHERLIB.\n"+ b" EXEC SQL INCLUDE DCLTRTYP END-EXEC.\n"+ )++ found = [(i.module_path, i.raw_statement, i.imported_names) for i in copy_imports(source)]++ assert found == [+ ("COCOM01Y", "COPY COCOM01Y", ["*"]),+ ("MYLIB/MEMBER1", "COPY MEMBER1 OF MYLIB", ["*"]),+ ("OTHERLIB/MEMBER1", "COPY MEMBER1 OF OTHERLIB", ["*"]),+ ("DCLTRTYP", "EXEC SQL INCLUDE DCLTRTYP", ["*"]),+ ]diff --git a/tests/unit/ingestion/test_language_capabilities.py b/tests/unit/ingestion/test_language_capabilities.pyindex eb0fe0dd6..cad68cb07 100644--- a/tests/unit/ingestion/test_language_capabilities.py+++ b/tests/unit/ingestion/test_language_capabilities.py@@ -161,6 +161,8 @@ _FULL = {# through the same SFC pass as svelte, so its imports are ordinary ESM."astro","c",+ # COPY members resolve to copybooks by stem, nearest first.+ "cobol","cpp","csharp",# dart promoted with the tree-sitter grammar (the regex tier stays as