Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.
Python aicode-complexitycode-healthcode-intelligencecode-qualitydead-codedeveloper-toolsdocumentationgit-analyticsmcpopen-sourcepythonrefactoringstatic-analysistechnical-debt
refactor(parser): split parser.py into a package; per-language import extractors on LanguageConfig (#4042)
Raghav Chamadiya4622beea4876
diff --git a/docs/architecture/ARCHITECTURE.md b/docs/architecture/ARCHITECTURE.mdindex ac2a7cbe2..009108ac5 100644--- a/docs/architecture/ARCHITECTURE.md+++ b/docs/architecture/ARCHITECTURE.md@@ -113,7 +113,7 @@ repowise/│ │ ├── src/repowise/core/│ │ │ ├── ingestion/│ │ │ │ ├── traverser.py # file tree walking + gitignore + per-dir .repowiseIgnore + extra exclude patterns-│ │ │ │ ├── parser.py # ASTParser: one class, all languages; _extract_calls(), _extract_import_bindings()+│ │ │ │ ├── parser/ # ASTParser: one class, all languages; one module per pass (symbols, imports, calls, references)│ │ │ │ ├── parsers/ # per-language parser helpers│ │ │ │ ├── graph.py # NetworkX dep graph builder; add_file(), _resolve_calls(), file_subgraph()│ │ │ │ ├── call_resolver.py # CallResolver: origin-stamped call resolution@@ -1594,7 +1594,7 @@ each with the file it registers in and what you lose by skipping it.Two things worth knowing before you start:-- **You never edit `parser.py` or `registry.py`.** Language identity goes in+- **You never edit `parser/` or `registry.py`.** Language identity goes in`ingestion/languages/specs/<lang>.py` and is slotted into `ALL_SPECS`; parserconfiguration goes in `ingestion/language_configs.py`. The one unavoidablecore edit is adding your tag to the `LanguageTag` Literal indiff --git a/docs/architecture/language-support.md b/docs/architecture/language-support.mdindex ad2b9b572..120984335 100644--- a/docs/architecture/language-support.md+++ b/docs/architecture/language-support.md@@ -10,7 +10,7 @@ centralised `LanguageRegistry`; per-language extraction logic lives incall-resolution strategy lives behind a seam in `call_resolver.py`. Adding alanguage means dropping one file into each relevant subpackage and registering itin that subpackage's dispatcher. No edits are needed to `graph.py`,-`traverser.py`, or any analysis core file. `parser.py` holds a short per-language+`traverser.py`, or any analysis core file. The `parser/` package holds a short per-languageblock for a handful of languages whose grammar shape the generic path cannotread (C++ qualified definitions, Dart mixins, Object Pascal out-of-line methods,[Elixir](#elixir), [F#](#f)); each is a few lines calling into a helper, and a@@ -151,7 +151,8 @@ ingestion/django.py python_imports.py node.py dotnet.pyspring.py ruby.py php.py scala.py swift.py c.py cpp.py luau.py go.py jvm.pysfc_source.py # Multi-language-file (SFC) projection (see below)- parser.py # ASTParser (language-agnostic orchestration)+ parser/ # ASTParser (language-agnostic orchestration), one module per pass+ extractors/imports/ # Per-language import rules (LanguageConfig.import_extractor)graph.py # GraphBuilder (import/call/heritage resolution)analysis/@@ -310,7 +311,9 @@ receiver typing (step 9).Add a parser configuration to `LANGUAGE_CONFIGS` in`packages/core/src/repowise/core/ingestion/language_configs.py` (re-exported from-`parser.py` for back-compat):+the `parser` package for back-compat). A language whose imports need more than+the generic shape also sets `import_extractor` to an `ImportExtractor` from+`extractors/imports/<lang>.py`:```python"mylang": LanguageConfig(@@ -629,8 +632,8 @@ corrupt every unit named after it in the same clause. Its sanitizer(`prepare_pascal_source` in `ingestion/lang_helpers/source_prep.py`) is gated on `path`'sextension (that syntax is invalid in a plain `.pas`/`.pp` unit file) and is ano-op everywhere else, same contract as the `_LOCATORS` path. Registering a-sanitizer this way, rather than as an if-block in `parser.py`, means it stays-"zero changes to `parser.py`" for a new language and the health walkers get the+sanitizer this way, rather than as an if-block in the parser, means it stays+"zero changes to the parser" for a new language and the health walkers get thesame clean projection the ingestion parser does.### The locator registry@@ -777,7 +780,7 @@ type body and a member the way Elixir shares `call` across everything, so itsper-language block is about extending and locating rather than reclassifying:- **A `let rec f ... and g ...` group is one grammar node for every clause**, so- `@symbol.def` sits on each clause's left-hand side and `parser.py` extends its+ `@symbol.def` sits on each clause's left-hand side and the parser extends itsspan forward to the next clause (or the file end), stepping over a return-typeannotation sitting between the two. The same left-hand side node also marks abinding local to another binding's body, which the generic callable-ancestordiff --git a/docs/reference/COMPUTED_GLOSSARY.md b/docs/reference/COMPUTED_GLOSSARY.mdindex d6e03329c..6fb15c7d1 100644--- a/docs/reference/COMPUTED_GLOSSARY.md+++ b/docs/reference/COMPUTED_GLOSSARY.md@@ -11,7 +11,7 @@ workspace overlays, MCP responses, and CLI output.| Area | Main code paths | What gets computed || --- | --- | --- |-| Traversal and parsing | `packages/core/src/repowise/core/ingestion/traverser.py`, `packages/core/src/repowise/core/ingestion/parser.py`, `packages/core/src/repowise/core/ingestion/models.py` | Files, languages, entry points, symbols, imports, exports, calls, inheritance, parse errors, content hashes |+| Traversal and parsing | `packages/core/src/repowise/core/ingestion/traverser.py`, `packages/core/src/repowise/core/ingestion/parser/`, `packages/core/src/repowise/core/ingestion/models.py` | Files, languages, entry points, symbols, imports, exports, calls, inheritance, parse errors, content hashes || Graph construction | `packages/core/src/repowise/core/ingestion/graph.py`, `call_resolver.py`, `heritage_resolver.py`, `framework_edges.py`, `dynamic_hints/` | File and symbol nodes, import/call/heritage/framework/dynamic/co-change edges, centrality, SCCs, communities, execution flows || Git intelligence | `packages/core/src/repowise/core/ingestion/git_indexer/` | Churn, ownership, hotspots, bus factor, co-change partners, significant commits, temporal scores, rename and merge signals || Analysis | `packages/core/src/repowise/core/analysis/` | Dead-code findings, decision records, decision staleness, security findings, PR blast radius, execution flows, communities |@@ -24,7 +24,7 @@ workspace overlays, MCP responses, and CLI output.| Term | Definition | Computed by | Example || --- | --- | --- | --- |-| Includable source file | A file that survives ignore rules, blocked patterns, size limit, binary detection, generated-file detection, and language detection. | `FileTraverser._build_file_info()` | `packages/core/src/repowise/core/ingestion/parser.py` |+| Includable source file | A file that survives ignore rules, blocked patterns, size limit, binary detection, generated-file detection, and language detection. | `FileTraverser._build_file_info()` | `packages/core/src/repowise/core/ingestion/traverser.py` || `FileInfo` | Per-file metadata used by the parser and graph builder. | `FileTraverser.traverse()` | `{path: "src/app.py", language: "python", is_test: false, is_entry_point: true}` || Language tag | Canonical language value from file extension, special filename, or shebang. | `ingestion/models.py`, `traverser.py`, `languages/registry.py` | `python`, `typescript`, `go`, `terraform`, `openapi`, `unknown` || Test file flag | Whether a file looks like a test/spec/fixture file. | `FileTraverser._build_file_info()` and community/test-gap helpers | `tests/test_auth.py -> is_test=true` |diff --git a/packages/core/src/repowise/core/ingestion/extractors/imports/__init__.py b/packages/core/src/repowise/core/ingestion/extractors/imports/__init__.pynew file mode 100644index 000000000..de39aceb0--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/extractors/imports/__init__.py@@ -0,0 +1,36 @@+"""Per-language import extraction, one ``ImportExtractor`` per language.++Each language's rules live in ``<lang>.py``; ``LanguageConfig.import_extractor``+wires them in, and a language without one takes ``GENERIC_IMPORTS``. This+package never imports the parser, so ``language_configs`` can import it.+"""++from ._types import ImportExtractor+from .dart import DART_IMPORTS+from .elixir import ELIXIR_IMPORTS+from .fsharp import FSHARP_IMPORTS+from .generic import GENERIC_IMPORTS+from .jvm import JVM_IMPORTS+from .pascal import PASCAL_IMPORTS+from .php import PHP_IMPORTS+from .python import PYTHON_IMPORTS+from .rust import RUST_IMPORTS+from .scala import SCALA_IMPORTS+from .swift import SWIFT_IMPORTS+from .ts_js import TS_JS_IMPORTS++__all__ = [+ "DART_IMPORTS",+ "ELIXIR_IMPORTS",+ "FSHARP_IMPORTS",+ "GENERIC_IMPORTS",+ "JVM_IMPORTS",+ "PASCAL_IMPORTS",+ "PHP_IMPORTS",+ "PYTHON_IMPORTS",+ "RUST_IMPORTS",+ "SCALA_IMPORTS",+ "SWIFT_IMPORTS",+ "TS_JS_IMPORTS",+ "ImportExtractor",+]diff --git a/packages/core/src/repowise/core/ingestion/extractors/imports/_types.py b/packages/core/src/repowise/core/ingestion/extractors/imports/_types.pynew file mode 100644index 000000000..406ae4e46--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/extractors/imports/_types.py@@ -0,0 +1,30 @@+"""The per-language hooks that turn import query matches into ``Import`` records."""++from __future__ import annotations++from collections.abc import Callable+from dataclasses import dataclass++from tree_sitter import Node++from ...models import Import++# (stmt_node, module_node, module_text, raw, language, src) -> imports+StatementFn = Callable[[Node, Node, str, str, str, str], list[Import]]+++@dataclass(frozen=True)+class ImportExtractor:+ """One language's import rules; a ``None`` hook takes the shared default.++ ``statement`` maps one deduplicated statement to its imports.+ ``deduped`` replaces the shared raw-text dedup for languages that key a+ statement their own way, given a per-file ``seen`` set; those imports+ keep the default load kind. ``dedup_key`` refines the raw-text key, and+ ``finalize`` rewrites the file's whole import list once.+ """++ statement: StatementFn+ deduped: Callable[[Node, Node, str, set[str]], list[Import]] | None = None+ dedup_key: Callable[[Node, str, str], str] | None = None+ finalize: Callable[[list[Import]], list[Import]] | None = Nonediff --git a/packages/core/src/repowise/core/ingestion/extractors/imports/dart.py b/packages/core/src/repowise/core/ingestion/extractors/imports/dart.pynew file mode 100644index 000000000..5603a896d--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/extractors/imports/dart.py@@ -0,0 +1,39 @@+"""Dart ``import`` / ``export`` / ``part`` directives."""++from __future__ import annotations++from tree_sitter import Node++from ...models import Import+from ..bindings import extract_import_bindings+from ._types import ImportExtractor+++def _dart_import(+ stmt_node: Node, module_node: Node, module_text: str, raw: str, src: str+) -> Import:+ """Dart: URIs are relative unless schemed, ``export`` re-exports, and the+ dotted ``part of library.name;`` form resolves through ``library:``.+ """+ module_path = module_text+ if module_node.type == "dotted_identifier_list":+ module_path = f"library:{module_text}"+ imported_names, bindings = extract_import_bindings(stmt_node, src, "dart")+ return Import(+ raw_statement=raw,+ module_path=module_path,+ imported_names=imported_names,+ is_relative=not module_path.startswith(("package:", "dart:", "library:")),+ resolved_file=None,+ bindings=bindings,+ is_reexport=stmt_node.type == "library_export",+ )+++def _dart_statement(+ stmt_node: Node, module_node: Node, module_text: str, raw: str, language: str, src: str+) -> list[Import]:+ return [_dart_import(stmt_node, module_node, module_text, raw, src)]+++DART_IMPORTS = ImportExtractor(statement=_dart_statement)diff --git a/packages/core/src/repowise/core/ingestion/extractors/imports/elixir.py b/packages/core/src/repowise/core/ingestion/extractors/imports/elixir.pynew file mode 100644index 000000000..d2af76f14--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/extractors/imports/elixir.py@@ -0,0 +1,44 @@+"""Elixir ``alias`` / ``import`` / ``require`` / ``use`` directives."""++from __future__ import annotations++from tree_sitter import Node++from ...models import Import+from ..bindings.elixir import elixir_import_modules+from ..helpers import node_text+from ._types import ImportExtractor+from .generic import generic_statement+++def _elixir_imports(+ stmt_node: Node, module_node: Node, src: str, seen_modules: set[str]+) -> list[Import]:+ """Elixir directives, one Import per module and deduped by module path.++ ``alias Foo.{Bar, Baz}`` names two modules in one statement. ``import``+ pulls in every public function (``*``); alias/require/use bind the module.+ """+ raw = node_text(stmt_node, src).split("\n", 1)[0].strip()+ directive_node = stmt_node.child_by_field_name("target")+ directive = node_text(directive_node, src).strip() if directive_node else ""+ imports: list[Import] = []+ for module_path in elixir_import_modules(module_node, src):+ if module_path in seen_modules:+ continue+ seen_modules.add(module_path)+ imports.append(+ Import(+ raw_statement=raw,+ module_path=module_path,+ imported_names=["*"] if directive == "import" else [],+ is_relative=False,+ resolved_file=None,+ bindings=[],+ is_reexport=False,+ )+ )+ return imports+++ELIXIR_IMPORTS = ImportExtractor(statement=generic_statement, deduped=_elixir_imports)diff --git a/packages/core/src/repowise/core/ingestion/extractors/imports/fsharp.py b/packages/core/src/repowise/core/ingestion/extractors/imports/fsharp.pynew file mode 100644index 000000000..d7a9ca49e--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/extractors/imports/fsharp.py@@ -0,0 +1,47 @@+"""F# ``open`` declarations."""++from __future__ import annotations++from tree_sitter import Node++from ...models import Import+from ..helpers import node_text+from ._types import ImportExtractor+from .generic import generic_statement+++def _fsharp_imports(+ stmt_node: Node, module_node: Node, src: str, seen_raws: set[str]+) -> list[Import]:+ """F# ``open``: every public name of the module (``*``), which bare-name+ call resolution reads. ``open type A.B.T`` binds the type's static members,+ so the module the file depends on is ``A.B``.+ """+ raw = node_text(stmt_node, src).strip()+ if raw in seen_raws:+ return []+ seen_raws.add(raw)+ module_path = node_text(module_node, src).strip()+ if not module_path:+ return []+ names: list[str] = ["*"]+ if any(child.type == "type" for child in stmt_node.children):+ head, _, type_name = module_path.rpartition(".")+ if head:+ module_path, names = head, [type_name]+ else:+ names = []+ return [+ Import(+ raw_statement=raw,+ module_path=module_path,+ imported_names=names,+ is_relative=False,+ resolved_file=None,+ bindings=[],+ is_reexport=False,+ )+ ]+++FSHARP_IMPORTS = ImportExtractor(statement=generic_statement, deduped=_fsharp_imports)diff --git a/packages/core/src/repowise/core/ingestion/extractors/imports/generic.py b/packages/core/src/repowise/core/ingestion/extractors/imports/generic.pynew file mode 100644index 000000000..e6671c707--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/extractors/imports/generic.py@@ -0,0 +1,48 @@+"""The import of a statement no language-specific shape claims."""++from __future__ import annotations++from tree_sitter import Node++from ...models import Import+from ..bindings import extract_import_bindings+from ._types import ImportExtractor+++def _generic_import(+ stmt_node: Node,+ module_text: str,+ raw: str,+ language: str,+ src: str,+ *,+ is_reexport: bool = False,+ is_module_declaration: bool = False,+) -> Import:+ """The single Import of a statement no language-specific shape claims."""+ imported_names, bindings = extract_import_bindings(stmt_node, src, language)+ is_relative = (+ module_text.startswith(".")+ or module_text.startswith("./")+ or module_text.startswith(("self::", "super::", "crate::"))+ )+ # ``type_only`` / ``deferred`` are set by the caller from ``import_load_kind``.+ return Import(+ raw_statement=raw,+ module_path=module_text,+ imported_names=imported_names,+ is_relative=is_relative,+ resolved_file=None,+ bindings=bindings,+ is_reexport=is_reexport,+ is_module_declaration=is_module_declaration,+ )+++def generic_statement(+ stmt_node: Node, module_node: Node, module_text: str, raw: str, language: str, src: str+) -> list[Import]:+ return [_generic_import(stmt_node, module_text, raw, language, src)]+++GENERIC_IMPORTS = ImportExtractor(statement=generic_statement)diff --git a/packages/core/src/repowise/core/ingestion/extractors/imports/jvm.py b/packages/core/src/repowise/core/ingestion/extractors/imports/jvm.pynew file mode 100644index 000000000..d4c0736d9--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/extractors/imports/jvm.py@@ -0,0 +1,24 @@+"""Java and Kotlin imports."""++from __future__ import annotations++from tree_sitter import Node++from ...models import Import+from ._types import ImportExtractor+from .generic import _generic_import+++def _jvm_statement(+ stmt_node: Node, module_node: Node, module_text: str, raw: str, language: str, src: str+) -> list[Import]:+ # The query captures the scoped identifier only, so ``import com.foo.*``+ # arrives as ``com.foo``. Restore the ``.*``.+ if not module_text.endswith("*"):+ stmt_text = raw.rstrip().rstrip(";").rstrip()+ if stmt_text.endswith(".*"):+ module_text += ".*"+ return [_generic_import(stmt_node, module_text, raw, language, src)]+++JVM_IMPORTS = ImportExtractor(statement=_jvm_statement)diff --git a/packages/core/src/repowise/core/ingestion/extractors/imports/pascal.py b/packages/core/src/repowise/core/ingestion/extractors/imports/pascal.pynew file mode 100644index 000000000..e7f32602f--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/extractors/imports/pascal.py@@ -0,0 +1,41 @@+"""Pascal ``uses`` clauses."""++from __future__ import annotations++from tree_sitter import Node++from ...models import Import+from ..helpers import node_text+from ._types import ImportExtractor+from .generic import generic_statement+++def _pascal_imports(+ stmt_node: Node, module_node: Node, src: str, seen_units: set[str]+) -> list[Import]:+ """One Pascal ``uses`` unit, deduped case-insensitively across the file.++ pascal.scm fires once per unit name, so a 3-unit clause arrives as 3+ matches sharing one statement span: deduping by raw text would drop all+ but the first. A unit named in both ``uses`` clauses is one dependency.+ ``uses`` exposes the unit's whole interface, hence the ``*`` wildcard.+ """+ raw = node_text(stmt_node, src).strip()+ unit_name = node_text(module_node, src).strip()+ if not unit_name or unit_name.lower() in seen_units:+ return []+ seen_units.add(unit_name.lower())+ return [+ Import(+ raw_statement=raw,+ module_path=unit_name,+ imported_names=["*"],+ is_relative=False,+ resolved_file=None,+ bindings=[],+ is_reexport=False,+ )+ ]+++PASCAL_IMPORTS = ImportExtractor(statement=generic_statement, deduped=_pascal_imports)diff --git a/packages/core/src/repowise/core/ingestion/extractors/imports/php.py b/packages/core/src/repowise/core/ingestion/extractors/imports/php.pynew file mode 100644index 000000000..58d020f5a--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/extractors/imports/php.py@@ -0,0 +1,37 @@+"""PHP ``use`` declarations."""++from __future__ import annotations++from tree_sitter import Node++from ...models import Import, NamedBinding+from ..bindings.php import php_class_name, php_use_clauses+from ._types import ImportExtractor+from .generic import _generic_import+++def _php_imports(stmt_node: Node, raw: str, src: str) -> list[Import]:+ """PHP: one ``use`` declaration can name several classes, each its own file."""+ return [+ Import(+ raw_statement=raw,+ module_path=fqn,+ imported_names=[php_class_name(fqn)],+ is_relative=False,+ resolved_file=None,+ bindings=[NamedBinding(local_name=local, exported_name=fqn, source_file=None)],+ is_reexport=False,+ )+ for fqn, local in php_use_clauses(stmt_node, src)+ ]+++def _php_statement(+ stmt_node: Node, module_node: Node, module_text: str, raw: str, language: str, src: str+) -> list[Import]:+ if stmt_node.type == "namespace_use_declaration":+ return _php_imports(stmt_node, raw, src)+ return [_generic_import(stmt_node, module_text, raw, language, src)]+++PHP_IMPORTS = ImportExtractor(statement=_php_statement)diff --git a/packages/core/src/repowise/core/ingestion/extractors/imports/python.py b/packages/core/src/repowise/core/ingestion/extractors/imports/python.pynew file mode 100644index 000000000..cf943b8bc--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/extractors/imports/python.py@@ -0,0 +1,9 @@+"""Python imports."""++from __future__ import annotations++from ..bindings.python import expand_bare_relative_imports+from ._types import ImportExtractor+from .generic import generic_statement++PYTHON_IMPORTS = ImportExtractor(statement=generic_statement, finalize=expand_bare_relative_imports)diff --git a/packages/core/src/repowise/core/ingestion/extractors/imports/rust.py b/packages/core/src/repowise/core/ingestion/extractors/imports/rust.pynew file mode 100644index 000000000..b18fc68da--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/extractors/imports/rust.py@@ -0,0 +1,127 @@+"""Rust ``use`` declarations, ``mod`` items and module-declaring macros."""++from __future__ import annotations++import re++from tree_sitter import Node++from ...models import Import, NamedBinding+from ..bindings.rust import expand_rust_use_tree, macro_mod_imports, rust_use_argument+from ..helpers import node_text+from ._types import ImportExtractor+from .generic import _generic_import+++def _rust_use_imports(stmt_node: Node, module_text: str, raw: str, src: str) -> list[Import]:+ """Rust: one Import per leaf of the use tree, each with its own path.++ ``use crate::{a::B, c::D}`` names two modules, and the joined string+ resolves to neither. ``super`` hops that only climb out of inline+ ``mod`` blocks stay inside this file, so they are dropped here and the+ resolver sees the path the file itself would write.+ """+ arg_node = rust_use_argument(stmt_node)+ leaves = expand_rust_use_tree(arg_node, src) if arg_node is not None else []+ inline_depth = _rust_inline_mod_depth(stmt_node)+ # ``pub use`` re-exports the module.+ is_reexport = any(child.type == "visibility_modifier" for child in stmt_node.children)+ if len(leaves) <= 1 and "{" not in module_text and not inline_depth:+ return [_generic_import(stmt_node, module_text, raw, "rust", src, is_reexport=is_reexport)]+ imports = []+ for path, local, exported in leaves:+ path = _strip_inline_super(path, inline_depth)+ imports.append(+ Import(+ raw_statement=raw,+ module_path=path,+ imported_names=[local],+ is_relative=path.startswith(("self::", "super::", "crate::")),+ resolved_file=None,+ bindings=[NamedBinding(local_name=local, exported_name=exported, source_file=None)],+ is_reexport=is_reexport,+ )+ )+ return imports+++def _rust_inline_mod_depth(node: Node) -> int:+ """How many inline ``mod name { ... }`` blocks enclose *node* in its file."""+ depth = 0+ parent = node.parent+ while parent is not None:+ if parent.type == "mod_item":+ depth += 1+ parent = parent.parent+ return depth+++def _strip_inline_super(path: str, inline_depth: int) -> str:+ """Drop the ``super`` hops of *path* that stay inside the file.++ In ``mod tests { use super::*; }`` the ``super`` is this file's own+ module, so the path is ``self::*``, not the parent module's glob.+ """+ if not inline_depth:+ return path+ segments = path.split("::")+ hops = 0+ while hops < len(segments) and segments[hops] == "super":+ hops += 1+ if not hops:+ return path+ if hops <= inline_depth:+ return "::".join(["self", *segments[hops:]])+ return "::".join(segments[inline_depth:])+++def _rust_mod_path_attribute(stmt_node: Node, src: str) -> str | None:+ """The ``#[path = "..."]`` override on a Rust ``mod`` item, if any.++ Outer attributes are preceding siblings of the item, not children.+ """+ parent = stmt_node.parent+ if parent is None:+ return None+ siblings = parent.children+ for j, sib in enumerate(siblings):+ if sib.id != stmt_node.id:+ continue+ # Walk backward through preceding attribute_item siblings+ k = j - 1+ while k >= 0 and siblings[k].type == "attribute_item":+ path_match = re.search(r'path\s*=\s*"([^"]+)"', node_text(siblings[k], src))+ if path_match:+ return path_match.group(1)+ k -= 1+ return None+ return None+++def _rust_statement(+ stmt_node: Node, module_node: Node, module_text: str, raw: str, language: str, src: str+) -> list[Import]:+ if stmt_node.type == "use_declaration":+ return _rust_use_imports(stmt_node, module_text, raw, src)+ if stmt_node.type == "macro_invocation":+ # Most top-level calls (``s! { struct ... }``) declare no module.+ return macro_mod_imports(module_node, raw) if "mod" in raw else []+ if stmt_node.type == "mod_item":+ module_text = _rust_mod_path_attribute(stmt_node, src) or module_text+ return [+ _generic_import(stmt_node, module_text, raw, language, src, is_module_declaration=True)+ ]+ return [_generic_import(stmt_node, module_text, raw, language, src)]+++def _rust_dedup_key(stmt_node: Node, raw: str, src: str) -> str:+ # A ``mod`` item's text excludes its outer attributes, so two ``#[cfg]``-gated+ # ``mod imp;`` items with different ``#[path]`` files would share one key.+ if stmt_node.type == "mod_item":+ mod_path_attr = _rust_mod_path_attribute(stmt_node, src)+ if mod_path_attr is not None:+ return f"{raw}|path={mod_path_attr}"+ return raw+++RUST_IMPORTS = ImportExtractor(statement=_rust_statement, dedup_key=_rust_dedup_key)diff --git a/packages/core/src/repowise/core/ingestion/extractors/imports/scala.py b/packages/core/src/repowise/core/ingestion/extractors/imports/scala.pynew file mode 100644index 000000000..529f7a63c--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/extractors/imports/scala.py@@ -0,0 +1,52 @@+"""Scala import declarations."""++from __future__ import annotations++from tree_sitter import Node++from ...models import Import, NamedBinding+from ..bindings.scala import expand_scala_import_clauses+from ._types import ImportExtractor+from .generic import _generic_import+++def _scala_imports(stmt_node: Node, raw: str, src: str) -> list[Import]:+ """Scala: one Import per selected name, with full dotted paths.++ The query's ``(identifier)`` capture is only the first path segment, and+ one declaration can hold several clauses, brace selectors, renames and+ wildcards.+ """+ imports: list[Import] = []+ for clause_path, clause_names in expand_scala_import_clauses(stmt_node, src):+ local = clause_names[0]+ exported = None if local == "*" else clause_path.rsplit(".", 1)[-1]+ imports.append(+ Import(+ raw_statement=raw,+ module_path=clause_path,+ imported_names=clause_names,+ is_relative=False,+ resolved_file=None,+ bindings=[+ NamedBinding(+ local_name=local,+ exported_name=exported,+ source_file=None,+ )+ ],+ is_reexport=False,+ )+ )+ return imports+++def _scala_statement(+ stmt_node: Node, module_node: Node, module_text: str, raw: str, language: str, src: str+) -> list[Import]:+ if stmt_node.type == "import_declaration":+ return _scala_imports(stmt_node, raw, src)+ return [_generic_import(stmt_node, module_text, raw, language, src)]+++SCALA_IMPORTS = ImportExtractor(statement=_scala_statement)diff --git a/packages/core/src/repowise/core/ingestion/extractors/imports/swift.py b/packages/core/src/repowise/core/ingestion/extractors/imports/swift.pynew file mode 100644index 000000000..9f66714c3--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/extractors/imports/swift.py@@ -0,0 +1,20 @@+"""Swift imports."""++from __future__ import annotations++from tree_sitter import Node++from ...models import Import+from ._types import ImportExtractor+from .generic import _generic_import+++def _swift_statement(+ stmt_node: Node, module_node: Node, module_text: str, raw: str, language: str, src: str+) -> list[Import]:+ # ``@_exported import`` re-exports the module.+ is_reexport = raw.startswith("@_exported")+ return [_generic_import(stmt_node, module_text, raw, language, src, is_reexport=is_reexport)]+++SWIFT_IMPORTS = ImportExtractor(statement=_swift_statement)diff --git a/packages/core/src/repowise/core/ingestion/extractors/imports/ts_js.py b/packages/core/src/repowise/core/ingestion/extractors/imports/ts_js.pynew file mode 100644index 000000000..40db63b1a--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/extractors/imports/ts_js.py@@ -0,0 +1,82 @@+"""TypeScript / JavaScript imports: ESM, dynamic ``import()`` and CommonJS ``require``.++Also serves the SFC tags (svelte, vue, astro), whose script blocks parse as TS.+"""++from __future__ import annotations++from tree_sitter import Node++from ...models import Import+from ..bindings.ts_js import (+ cjs_statement_is_reexport,+ collect_cjs_requires,+ dynamic_import_bindings,+)+from ._types import ImportExtractor+from .generic import _generic_import++# CommonJS is read for plain TS/JS files only, not the SFC tags.+_CJS_LANGUAGES = ("javascript", "typescript")+_CJS_STATEMENTS = ("assignment_expression", "call_expression")+++def _is_dynamic_esm_import(stmt_node: Node) -> bool:+ if stmt_node.type != "call_expression":+ return False+ fn = stmt_node.child_by_field_name("function")+ return fn is not None and fn.type == "import"+++def _cjs_imports(stmt_node: Node, raw: str, src: str) -> list[Import]:+ """CommonJS: every ``require()`` in the statement, re-exports marked.++ A hub like ``Object.assign(module.exports, require('./a'), ...)`` is+ several imports, and ``module.exports`` shapes are barrels.+ """+ cjs_reexport = cjs_statement_is_reexport(stmt_node, src)+ return [+ Import(+ raw_statement=raw,+ module_path=cjs_module,+ imported_names=["*"] if cjs_reexport else [],+ is_relative=cjs_module.startswith("."),+ resolved_file=None,+ bindings=[],+ is_reexport=cjs_reexport,+ )+ for cjs_module in collect_cjs_requires(stmt_node, src)+ ]+++def _ts_js_statement(+ stmt_node: Node, module_node: Node, module_text: str, raw: str, language: str, src: str+) -> list[Import]:+ if _is_dynamic_esm_import(stmt_node):+ # ``import('./mod')`` binds a module namespace at runtime, so it is a+ # wildcard, which keeps the target's exports live. The names it is+ # destructured into still bind, for call resolution.+ return [+ Import(+ raw_statement=raw,+ module_path=module_text,+ imported_names=["*"],+ is_relative=module_text.startswith("."),+ resolved_file=None,+ bindings=dynamic_import_bindings(stmt_node, src),+ is_reexport=False,+ )+ ]+ if language in _CJS_LANGUAGES and stmt_node.type in _CJS_STATEMENTS:+ return _cjs_imports(stmt_node, raw, src)+ return [_generic_import(stmt_node, module_text, raw, language, src)]+++def _ts_js_dedup_key(stmt_node: Node, raw: str, src: str) -> str:+ # Two ``import('./m')`` calls share their text but not the names each one binds.+ if _is_dynamic_esm_import(stmt_node):+ return f"{raw}|at={stmt_node.start_byte}"+ return raw+++TS_JS_IMPORTS = ImportExtractor(statement=_ts_js_statement, dedup_key=_ts_js_dedup_key)diff --git a/packages/core/src/repowise/core/ingestion/lang_helpers/__init__.py b/packages/core/src/repowise/core/ingestion/lang_helpers/__init__.pyindex 83c12b9b1..193d58a71 100644--- a/packages/core/src/repowise/core/ingestion/lang_helpers/__init__.py+++ b/packages/core/src/repowise/core/ingestion/lang_helpers/__init__.py@@ -1,4 +1,4 @@"""Per-language tree-sitter helpers the ingestion parser calls.-``parser_helpers`` re-exports every name ``parser.py`` imports from here.+``parser_helpers`` re-exports every name the ``parser`` package imports from here."""diff --git a/packages/core/src/repowise/core/ingestion/lang_helpers/elixir.py b/packages/core/src/repowise/core/ingestion/lang_helpers/elixir.pyindex b0db4bf37..7f6115230 100644--- a/packages/core/src/repowise/core/ingestion/lang_helpers/elixir.py+++ b/packages/core/src/repowise/core/ingestion/lang_helpers/elixir.py@@ -226,7 +226,7 @@ def _elixir_is_template_definition(def_node: Node, src: str) -> bool:def _elixir_module_parent(def_node: Node, src: str) -> str | None:"""Return the enclosing ``defmodule``/``defprotocol`` name, or None.- The generic nesting walk in ``ASTParser._find_parent`` reads+ The generic nesting walk in ``parser.symbols._find_parent`` reads``ancestor.child_by_field_name("name")``, and an Elixir ``call`` has no``name`` field -- the module name is the first argument of a call to``defmodule``. Nesting is the only thing that says which module adiff --git a/packages/core/src/repowise/core/ingestion/language_configs.py b/packages/core/src/repowise/core/ingestion/language_configs.pyindex 4571659c8..168ed6ac8 100644--- a/packages/core/src/repowise/core/ingestion/language_configs.py+++ b/packages/core/src/repowise/core/ingestion/language_configs.py@@ -1,9 +1,8 @@"""Per-language parser configuration data.``LanguageConfig`` plus the declarative ``LANGUAGE_CONFIGS`` table that-drives :class:`~repowise.core.ingestion.parser.ASTParser`. Extracted from-``parser.py`` so the parser module holds behaviour and this module holds-the per-language data. The parser keeps re-exporting both names, so+drives :class:`~repowise.core.ingestion.parser.ASTParser`: the parser holds+behaviour, this module the per-language data. The parser keeps re-exporting both names, so``from ...parser import LANGUAGE_CONFIGS, LanguageConfig`` still works."""@@ -19,6 +18,21 @@ from .extractors.cobol import (normalize_cobol_call_target,normalize_cobol_symbol_name,)+from .extractors.imports import (+ DART_IMPORTS,+ ELIXIR_IMPORTS,+ FSHARP_IMPORTS,+ GENERIC_IMPORTS,+ JVM_IMPORTS,+ PASCAL_IMPORTS,+ PHP_IMPORTS,+ PYTHON_IMPORTS,+ RUST_IMPORTS,+ SCALA_IMPORTS,+ SWIFT_IMPORTS,+ TS_JS_IMPORTS,+ ImportExtractor,+)from .extractors.visibility import (csharp_visibility,dart_visibility,@@ -98,9 +112,14 @@ class LanguageConfig:# ``modifier``, Kotlin ``modifiers``), read into ``Symbol.modifiers``.modifier_node_types: frozenset[str] = field(default_factory=frozenset)+ # How this language's import matches become ``Import`` records; see+ # ``extractors.imports``.+ import_extractor: ImportExtractor = GENERIC_IMPORTS+LANGUAGE_CONFIGS: dict[str, LanguageConfig] = {"python": LanguageConfig(+ import_extractor=PYTHON_IMPORTS,symbol_node_types={"function_definition": "function","class_definition": "class",@@ -115,6 +134,7 @@ LANGUAGE_CONFIGS: dict[str, LanguageConfig] = {parent_class_types=frozenset({"class_definition"}),),"typescript": LanguageConfig(+ import_extractor=TS_JS_IMPORTS,symbol_node_types={"function_declaration": "function","generator_function_declaration": "function",@@ -148,6 +168,7 @@ LANGUAGE_CONFIGS: dict[str, LanguageConfig] = {declaration_node_types=frozenset({"function_signature", "method_signature"}),),"javascript": LanguageConfig(+ import_extractor=TS_JS_IMPORTS,symbol_node_types={"function_declaration": "function","generator_function_declaration": "function",@@ -178,6 +199,7 @@ LANGUAGE_CONFIGS: dict[str, LanguageConfig] = {parent_class_types=frozenset(),),"rust": LanguageConfig(+ import_extractor=RUST_IMPORTS,symbol_node_types={"function_item": "function","function_signature_item": "function",@@ -212,6 +234,7 @@ LANGUAGE_CONFIGS: dict[str, LanguageConfig] = {reference_call_node_types=frozenset({"macro_invocation"}),),"java": LanguageConfig(+ import_extractor=JVM_IMPORTS,symbol_node_types={"class_declaration": "class","interface_declaration": "interface",@@ -280,6 +303,7 @@ LANGUAGE_CONFIGS: dict[str, LanguageConfig] = {declaration_node_types=frozenset({"declaration"}),),"kotlin": LanguageConfig(+ import_extractor=JVM_IMPORTS,symbol_node_types={"function_declaration": "function","class_declaration": "class",@@ -374,6 +398,7 @@ LANGUAGE_CONFIGS: dict[str, LanguageConfig] = {),),"swift": LanguageConfig(+ import_extractor=SWIFT_IMPORTS,symbol_node_types={"class_declaration": "class","protocol_declaration": "interface",@@ -390,6 +415,7 @@ LANGUAGE_CONFIGS: dict[str, LanguageConfig] = {parent_class_types=frozenset({"class_declaration", "protocol_declaration"}),),"dart": LanguageConfig(+ import_extractor=DART_IMPORTS,symbol_node_types={"class_definition": "class","mixin_declaration": "class",@@ -419,6 +445,7 @@ LANGUAGE_CONFIGS: dict[str, LanguageConfig] = {),),"scala": LanguageConfig(+ import_extractor=SCALA_IMPORTS,symbol_node_types={"class_definition": "class","trait_definition": "trait",@@ -438,6 +465,7 @@ LANGUAGE_CONFIGS: dict[str, LanguageConfig] = {parent_class_types=frozenset({"class_definition", "trait_definition", "object_definition"}),),"php": LanguageConfig(+ import_extractor=PHP_IMPORTS,symbol_node_types={"class_declaration": "class","interface_declaration": "interface",@@ -457,6 +485,7 @@ LANGUAGE_CONFIGS: dict[str, LanguageConfig] = {),),"elixir": LanguageConfig(+ import_extractor=ELIXIR_IMPORTS,symbol_node_types={# Elixir's one node kind for every definition. "module" is a# placeholder, refined per keyword in refine_elixir_call_kind --@@ -478,6 +507,7 @@ LANGUAGE_CONFIGS: dict[str, LanguageConfig] = {parent_class_types=frozenset(),),"fsharp": LanguageConfig(+ import_extractor=FSHARP_IMPORTS,symbol_node_types={# `module Foo.Bar` / `namespace Foo.Bar` head the file;# `module X =` nests inside one.@@ -487,7 +517,7 @@ LANGUAGE_CONFIGS: dict[str, LanguageConfig] = {# The captured node is the binding's left-hand side, not the# enclosing function_or_value_defn: that one node holds every# clause of a `let rec f ... and g ...` group, so capturing it- # would give every clause the same span. parser.py extends each+ # would give every clause the same span. the parser extends each# symbol over its own clause and filters the nested ones."function_declaration_left": "function","value_declaration_left": "variable",@@ -516,7 +546,7 @@ LANGUAGE_CONFIGS: dict[str, LanguageConfig] = {parent_extraction="nesting",# Deliberately empty: no F# type node carries a `name` field, so the# generic walk in _find_parent would match an ancestor and then read- # nothing off it. The parent walk is language-gated in parser.py --+ # nothing off it. The parent walk is language-gated in the parser --# see _fsharp_parent_name.parent_class_types=frozenset(),),@@ -567,12 +597,13 @@ LANGUAGE_CONFIGS: dict[str, LanguageConfig] = {),),"pascal": LanguageConfig(+ import_extractor=PASCAL_IMPORTS,symbol_node_types={# declType wraps class/record/interface/helper/enum/set/array/alias# in one node shape (see spec docstring) -- "class" is the closest# single bucket, same tradeoff Go makes for `type_spec`: "struct".# A real kind refinement (peek at the `type` field to distinguish- # class/struct/interface/enum) would need a hook parser.py doesn't+ # class/struct/interface/enum) would need a hook the parser doesn't# currently expose per-language the way it does for Go/Kotlin --# flagged as a follow-up, not attempted here."declType": "class",@@ -597,7 +628,7 @@ LANGUAGE_CONFIGS: dict[str, LanguageConfig] = {visibility_fn=public_by_default,parent_extraction="nesting",# NOT {"declClass", "declIntf", "declHelper"} -- see the spec docstring.- # ASTParser._find_parent walks ancestors checking `ancestor.type in+ # parser.symbols._find_parent walks ancestors checking `ancestor.type in# parent_class_types` then reads `ancestor.child_by_field_name("name")`;# only declType carries a name field, so pointing this at the inner# class/interface/helper node would silently return no parent for everydiff --git a/packages/core/src/repowise/core/ingestion/languages/specs/pascal.py b/packages/core/src/repowise/core/ingestion/languages/specs/pascal.pyindex c0df0cf61..2b1eaf363 100644--- a/packages/core/src/repowise/core/ingestion/languages/specs/pascal.py+++ b/packages/core/src/repowise/core/ingestion/languages/specs/pascal.py@@ -10,7 +10,7 @@ heritage_node_types / parent_class_types are both "declType", not the moreobvious "declClass"/"declIntf"/"declHelper". Pascal nests the actual classbody one level inside a `Name = <class|interface|helper body>;` wrapper(`declType`), and only the wrapper carries a `name` field -- `declClass`-itself has none. `ASTParser._find_parent` looks up+itself has none. `parser.symbols._find_parent` looks up`ancestor.child_by_field_name("name")` on whatever ancestor type matches`parent_class_types`, so pointing it at "declClass" would silently returnno parent name for every method. See extractors/heritage/pascal.py for the@@ -48,8 +48,8 @@ SPEC = LanguageSpec(shebang_tokens=(),# Common RTL calls worth excluding from the project call graph, spelled# the way the RTL itself declares them (Pascal is case-insensitive, but- # `ASTParser._extract_calls` does a case-sensitive `target_name in- # _call_builtins` check against the raw source text, so entries need to+ # `parser.calls._extract_calls` does a case-sensitive `target_name in+ # call_builtins` check against the raw source text, so entries need to# match real-world casing, not lowercased). This list is intentionally# conservative -- System-unit built-ins only; framework calls (VCL/FMX/# LCL) are ordinary project dependencies and should still show up asdiff --git a/packages/core/src/repowise/core/ingestion/parser.py b/packages/core/src/repowise/core/ingestion/parser.pydeleted file mode 100644index 78557e337..000000000--- a/packages/core/src/repowise/core/ingestion/parser.py+++ /dev/null@@ -1,2452 +0,0 @@-"""Unified AST parser — one class for all languages.--Architecture-============-Per-language differences live in two places:- 1. ``packages/core/queries/<lang>.scm`` — tree-sitter S-expression queries- that capture symbols and imports using consistent capture-name conventions.- 2. ``LANGUAGE_CONFIGS`` dict in this module — a ``LanguageConfig`` per language- that maps node types to symbol kinds, defines visibility rules, etc.--``ASTParser`` itself contains *no* if/elif language branches. Adding support-for a new language means writing one ``.scm`` file and one ``LanguageConfig``-entry. No Python class, no new module.--Capture-name conventions (shared across ALL .scm files):- @symbol.def — the full definition node (line numbers, kind lookup)- @symbol.name — name identifier- @symbol.params — parameter list (optional)- @symbol.modifiers — decorators / visibility modifiers (optional)- @symbol.receiver — Go method receiver (optional, used for parent detection)- @import.statement — full import node- @import.module — module path being imported-"""--from __future__ import annotations--import re-import sys-from collections.abc import Iterable-from dataclasses import replace-from functools import cache-from pathlib import Path--import structlog-from tree_sitter import Language, Node, Parser--from .cpp_export_macros import (- CppExportTypes,- _cpp_export_macro_parent,- _cpp_normalize_identifier,- _CppExportType,- _is_bodiless_cpp_type,- collect_cpp_export_types,-)-from .extractors import (- build_signature,- extract_go_receiver_type,- extract_heritage,- extract_import_bindings,- extract_module_docstring,- extract_symbol_docstring,- node_text,- refine_elixir_call_kind,- refine_fsharp_type_kind,- refine_go_type_kind,- refine_kotlin_class_kind,- refine_pascal_type_kind,- refine_swift_class_kind,-)-from .extractors.bindings.elixir import elixir_import_modules-from .extractors.bindings.python import expand_bare_relative_imports-from .extractors.bindings.ts_js import (- declarator_binds_callable,- declarator_value_is_module_ref,- dynamic_import_bindings,-)-from .extractors.synthetic_symbols import extract_synthetic_symbols-from .extractors.synthetic_symbols.csharp_extension_blocks import drop_extension_block_symbols-from .extractors.visibility import (- py_module_all_names,- refine_cpp_visibility,- refine_csharp_visibility,- refine_java_visibility,- refine_py_visibility,- refine_rust_visibility,- refine_ts_visibility,- ts_deferred_export_names,- ts_export_aliases,-)-from .language_configs import LANGUAGE_CONFIGS, LanguageConfig-from .languages.registry import REGISTRY as _LANG_REGISTRY-from .models import (- CallReceiver,- CallSite,- FileInfo,- Import,- ParsedFile,- Symbol,- TypeReference,- combine_load_kinds,- compute_content_hash,-)-from .parser_helpers import (- TYPE_HEAD_EXTRACTORS,- _argument_literal_types,- _argument_names,- _build_qualified_name,- _classify_param_origin,- _collect_error_nodes,- _count_arguments,- _csharp_type_parameter_count,- _dedupe_objc_interface_symbols,- _dedupe_pascal_interface_symbols,- _elixir_call_is_definitional,- _elixir_is_template_definition,- _elixir_module_parent,- _elixir_symbol_name,- _find_enclosing_symbol,- _fsharp_binding_end_line,- _fsharp_binding_has_params,- _fsharp_binding_is_nested,- _fsharp_parent_is_type,- _fsharp_parent_name,- _has_callable_ancestor,- _head_type_identifier,- _is_async_node,- _kotlin_receiver_fields,- _objc_call_is_block_variable,- _objc_container_node,- _objc_container_parent,- _objc_is_macro_enum,- _objc_message_selector,- _objc_symbol_name,- _overload_arguments,- _qualified_cpp_parent,- _qualified_pascal_parent,- _run_query,- _rust_shadowed_by_type_param,- _ts_nested_object_method_owner,- follow_renamed_hosts,- import_load_kind,- in_anonymous_class,- is_anonymous_member,- scope_segment,-)-from .python_local_refs import local_refs_from_tree, parse_python_module-from .python_overload import is_python_overload as _is_python_overload-from .python_registrars import registering_functions-from .python_tasks import is_fire_and_forget-from .sfc_source import component_call_sites, prepare_source-from .special_handlers import SPECIAL_HANDLER_LANGUAGES, parse_special-from .symbol_identity import disambiguate_colliding_ids, symbol_discriminator--log = structlog.get_logger(__name__)--# Any single file emitting more than this many symbols is almost-# certainly machine-generated (large gRPC service contracts, OpenAPI-# bindings, SQL schema bindings). Warn rather than truncate — operators-# can decide whether to add the file to ``_NEVER_FLAG_PATTERNS`` or to-# exclude it via traversal.-_SYMBOL_COUNT_WARN_THRESHOLD = 500--QUERIES_DIR = Path(__file__).parent / "queries"--# Node types whose .scm patterns are anchored at module/program level-# (constants and module variables). They can never be function-local, so-# the callable-ancestor filter must not apply — and for TS/JS declarators-# it would misfire on the parent lexical_declaration kind mapping.-_MODULE_ANCHORED_NODE_TYPES = frozenset({"assignment", "variable_declarator"})--# Languages whose source reaches the parser as TypeScript/JavaScript. The-# SFC tags are here because ``sfc_source`` projects their <script> blocks into-# a TS buffer at identical offsets, so every TS/JS code path applies verbatim.-_TS_JS_LANGUAGES = ("typescript", "javascript", "svelte", "vue", "astro")--# Languages whose query defines the ``@reference.*`` captures. Every other-# language would only scan the whole match list to find nothing, so the check-# is here rather than inside ``_extract_references``. The SFC tags reuse the-# TS captures but stay out: resolution has no local-shadow scan for them.-_REFERENCE_LANGUAGES = ("cpp", "c", "go", "rust", "kotlin", "typescript", "javascript", "python")--# Languages where a bare name resolves only through the file's own symbols or-# a name it imports, so any other bare name cannot pass the reference floor.-_NAMED_BINDING_LANGUAGES = frozenset({"typescript", "javascript", "python"})---def _bare_reference_names(- language: str, symbols: list[Symbol], imports: list[Import]-) -> frozenset[str] | None:- """Names a receiver-less reference in this file could resolve to, or None for any.-- Most bare identifiers in a value position are locals; dropping them here- skips their per-site work. A wildcard import keeps every name.- """- if language not in _NAMED_BINDING_LANGUAGES:- return None- imported = {name for imp in imports for name in imp.local_names}- if "*" in imported:- return None- return frozenset(imported.union(s.name for s in symbols))---def _call_receiver_from_node(node: Node, src: str) -> CallReceiver | None:- """Describe an inner call captured as another call's receiver.-- The queries identify the complete inner AST node. This helper reads only- named tree-sitter fields, so nested arguments and formatting cannot change- which call is carried into resolution.- """-- if node.type == "identifier":- return _dart_call_receiver(node, src)- arguments = node.child_by_field_name("arguments")- argument_count = _count_arguments(arguments) if arguments is not None else None-- target = node.child_by_field_name("name")- receiver = node.child_by_field_name("object")- if target is None:- function = node.child_by_field_name("function")- if function is None:- return None- if function.type in ("identifier", "property_identifier", "field_identifier"):- target = function- receiver = None- elif function.type == "generic_name":- target = function.child_by_field_name("name") or next(- (child for child in function.named_children if child.type == "identifier"), None- )- receiver = None- elif function.type == "attribute":- # Python ``obj.m()``: the receiver is kept as a plain dotted path,- # and a call on anything else is not carried at all.- target = function.child_by_field_name("attribute")- obj = function.child_by_field_name("object")- path = (- _receiver_text("python", obj, src)- if obj is not None and obj.type in ("identifier", "attribute")- else None- )- if target is None or path is None:- return None- return CallReceiver(_node_text(target, src).strip(), path, argument_count)- else:- target = (- function.child_by_field_name("name")- or function.child_by_field_name("property")- or function.child_by_field_name("field")- )- receiver = (- function.child_by_field_name("expression")- or function.child_by_field_name("object")- or function.child_by_field_name("argument")- or function.child_by_field_name("operand")- )-- if target is None:- return None- target_name = _node_text(target, src).strip()- if not target_name:- return None-- receiver_name = None- if receiver is not None and receiver.type in ("identifier", "this"):- receiver_name = _node_text(receiver, src).strip() or None- return CallReceiver(target_name, receiver_name, argument_count)---def _dart_call_receiver(name: Node, src: str) -> CallReceiver | None:- """The Dart call whose *name* the query captured: ``a.b(..)``, ``B(..)``.-- A Dart call is a flat selector run with no call node, so the name's- siblings are read: the selector after it holds the arguments, and the- node before a ``.b`` selector is its receiver.- """- if name.parent.type == "unconditional_assignable_selector":- head = name.parent.parent- prev = head.prev_named_sibling- if prev is not None and any(c.type == "argument_part" for c in prev.named_children):- receiver = None- else:- receiver = _node_text(prev, src).strip() if prev is not None else None- else:- head, receiver = name, None- return CallReceiver(- _node_text(name, src).strip(), receiver, _dart_argument_count(head.next_named_sibling)- )---def _dart_argument_count(selector: Node | None) -> int | None:- """The arguments a Dart ``(selector (argument_part (arguments)))`` passes."""- parts = selector.named_children if selector is not None else ()- found = [arg for part in parts for arg in part.named_children if arg.type == "arguments"]- return _count_arguments(found[0]) if found else None---@cache-def _compile_query(lang: str, grammar_tag: str | None = None) -> tuple[object | None, str | None]:- """Compile and return (Query, None) or (None, error_str).-- Cached process-wide so preflight and parsing workers never recompile the same query.- """- grammar = grammar_tag or lang- language = _get_language(grammar)- if language is None:- return None, None-- # The spec names the query file, so a language can reuse another's- # queries wholesale (svelte -> typescript.scm). Every other spec declares- # ``<tag>.scm``, which is what the default preserves.- spec = _LANG_REGISTRY.get(lang)- scm_name = (spec.scm_file if spec and spec.scm_file else None) or f"{lang}.scm"- scm_path = QUERIES_DIR / scm_name- if not scm_path.exists():- log.debug("No .scm query file found", language=lang, path=str(scm_path))- return None, None-- scm_text = scm_path.read_text(encoding="utf-8")- # Grammar-variant-specific additions (e.g. JSX node captures that are- # only valid against the ``tsx`` grammar but not the plain ``typescript``- # one). Appended to the base SCM only when the variant scm file exists.- if grammar_tag and grammar_tag != lang:- extra_scm = QUERIES_DIR / f"{grammar_tag}.scm"- if extra_scm.exists():- scm_text = scm_text + "\n" + extra_scm.read_text(encoding="utf-8")- try:- from tree_sitter import Query # type: ignore[attr-defined]-- return Query(language, scm_text), None- except Exception as exc:- return None, str(exc)---@cache-def _load_compiled_query(lang: str, grammar_tag: str | None = None) -> object | None:- """Process-wide cache of compiled tree-sitter Query objects.-- Compiling `.scm` queries is non-trivial; in process-pool parsing each worker- would otherwise recompile per file. ``grammar_tag`` may differ from- ``lang`` when a language reuses another's grammar at a different- variant — e.g. ``.tsx`` files reuse ``typescript.scm`` but must bind- to the JSX-aware ``tsx`` grammar so React components don't drown in- ERROR nodes.- """- query, err = _compile_query(lang, grammar_tag)- if err is not None:- log.warning("Failed to compile query", language=lang, error=err)- return query---# Languages that intentionally have no AST parser. Derived from the-# centralised LanguageRegistry — only non-code passthrough languages are-# included (not the extra git-blame-only languages).--# Excludes "openapi" (handled by special_handlers) and "unknown".-_PASSTHROUGH_LANGUAGES: frozenset[str] = _LANG_REGISTRY.unparseable_data_languages()--# ----------------------------------------------------------------------------# Language registry — maps language tag → tree-sitter Language object-# ------------------------------------------------------------------------------def _build_language_registry() -> dict[str, Language]:- """Lazily load installed tree-sitter language packages.-- Driven by ``LanguageSpec.grammar_package`` / ``grammar_loader`` /- ``shares_grammar_with`` from the centralised registry.- """- registry: dict[str, Language] = {}-- for spec in _LANG_REGISTRY.all_specs():- # Languages that share another's grammar (e.g. C → cpp)- if spec.shares_grammar_with:- shared = registry.get(spec.shares_grammar_with)- if shared:- registry[spec.tag] = shared- continue-- if not spec.grammar_package:- continue-- try:- mod = __import__(spec.grammar_package)- loader_fn = getattr(mod, spec.grammar_loader)- loaded = loader_fn(*spec.grammar_loader_args)- # Standalone grammar wheels return a PyCapsule; shared grammar- # packs may return the fully constructed Language directly.- lang_obj = loaded if isinstance(loaded, Language) else Language(loaded)- registry[spec.tag] = lang_obj- except Exception as exc:- log.debug(- "tree-sitter language unavailable",- language=spec.tag,- reason=str(exc),- )-- # TypeScript's tsx variant — special case: same package, different loader- if "typescript" in registry and "tsx" not in registry:- try:- import tree_sitter_typescript as _ts_mod-- registry["tsx"] = Language(_ts_mod.language_tsx())- except Exception as exc:- log.debug("tree-sitter language unavailable", language="tsx", reason=str(exc))-- return registry---_LANGUAGE_REGISTRY: dict[str, Language] = {}--# Languages already reported as having a config but no installed grammar, so-# the report is one line per language per process instead of one per file. A-# repo with a few thousand shell scripts otherwise emitted a few thousand-# identical lines, all saying the same three facts.-_MISSING_GRAMMAR_REPORTED: set[str] = set()---def missing_grammar_languages(language_tags: Iterable[str]) -> list[str]:- """Of *language_tags*, those that parse via tree-sitter but have no grammar.-- Answers the question once, in the parent, before up to eight spawned- workers each rediscover it and log their own copy of the answer at a level- the CLI discards anyway.-- Deliberately uses ``find_spec`` rather than importing: the whole point is- to stay cheap enough to run on every index. Building the real registry here- would import every tree-sitter package into the parent process, which is- memory the parse pool is about to need for something else.-- A tag with no :data:`LANGUAGE_CONFIGS` entry is not a gap — nothing claims- to parse it — so it is skipped rather than reported.-- Ceiling: "importable" is not "loadable". A grammar whose compiled ABI does- not match the installed ``tree_sitter`` imports fine and then raises inside- ``Language()``, which this cannot see, so that case reports nothing here- and stays a per-worker debug line. Reporting it properly means loading the- grammars, which is the cost this function exists to avoid.- """- import importlib.util-- specs = {spec.tag: spec for spec in _LANG_REGISTRY.all_specs()}- missing: list[str] = []- for tag in language_tags:- if tag not in LANGUAGE_CONFIGS:- continue- spec = specs.get(tag)- if spec is None:- continue- package = spec.grammar_package- if not package and spec.shares_grammar_with:- shared = specs.get(spec.shares_grammar_with)- package = shared.grammar_package if shared else None- if not package:- continue- try:- if importlib.util.find_spec(package) is None:- missing.append(tag)- except (ImportError, ValueError):- missing.append(tag)- return sorted(missing)---def failed_query_languages(language_tags: Iterable[str]) -> list[tuple[str, str]]:- """Of *language_tags*, those whose tree-sitter queries fail to compile.-- Scoped to what traversal discovered in the repo. Only checks languages with- a registered LanguageConfig and query file. Best-effort and bounded.- """- failed: list[tuple[str, str]] = []- seen: set[str] = set()-- for tag in language_tags:- if tag in seen or tag not in LANGUAGE_CONFIGS:- continue- seen.add(tag)-- _, err = _compile_query(tag)- if err is not None:- failed.append((tag, err))- continue-- # Check grammar variants if applicable (e.g. tsx for typescript)- if tag == "typescript":- _, tsx_err = _compile_query("typescript", grammar_tag="tsx")- if tsx_err is not None:- failed.append((tag, f"tsx variant: {tsx_err}"))-- return sorted(failed, key=lambda x: x[0])---def _get_language(tag: str) -> Language | None:- global _LANGUAGE_REGISTRY- if not _LANGUAGE_REGISTRY:- _LANGUAGE_REGISTRY = _build_language_registry()- return _LANGUAGE_REGISTRY.get(tag)---def grammar_tag_for(language: str, path: str) -> str:- """The grammar a file is read with, which is not always its language tag.-- A ``.tsx`` file arrives tagged ``typescript``, and tree-sitter-typescript's- default grammar errors on every ``<Component />``. Only the grammar moves:- the language tag keeps selecting dialects and vocabularies, several of- which (``mocks/lexicon.py`` among them) carry no ``tsx`` row and would- silently degrade if handed one.- """- # Case-folded, because the extension table that tagged the file is.- if language == "typescript" and path.lower().endswith(".tsx"):- return "tsx"- return language---# Private alias for internal use (kept for compatibility with _find_parent)-_node_text = node_text---# The receiver captures that are themselves a member access, by language (the-# queries widen only these two), and what a plain dotted path of names looks-# like once whitespace is gone.-_PATH_RECEIVERS = frozenset({("typescript", "member_expression"), ("python", "attribute")})-_DOTTED_PATH = re.compile(r"[A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*)+")---def _receiver_text(language: str, node: Node, src: str) -> str | None:- """The receiver's text, or None to drop the site.-- A dotted receiver (``this.a.b.m()``) is kept only as a plain path of- names: a call, subscript or optional hop inside it types nothing, and- dropping the site keeps it off the bare-call tiers.- """- sibling = node.next_named_sibling- if language == "java" and node.type != "super" and getattr(sibling, "type", None) == "super":- # ``S.super.m()``: ``S`` qualifies the ``super`` its own pattern captures.- return None- text = _node_text(node, src).strip()- if (language, node.type) not in _PATH_RECEIVERS:- return text- path = "".join(text.split())- return path if _DOTTED_PATH.fullmatch(path) else None---def _normalize_php_receiver(text: str) -> str:- """Spell PHP's receiver the way the resolver's strategies expect.-- `self::` and `static::` are the same dispatch as `$this`, and the self/this- strategy tests `in ("self", "this")` — so without this the whole implicit-- receiver population misses. `parent::` needs the heritage walk and is left- to miss rather than guessed at.- """- name = text.lstrip("$")- return "this" if name in ("self", "static") else name---# ----------------------------------------------------------------------------# ASTParser-# ------------------------------------------------------------------------------def _match_identity(- capture_dict: dict,- config: LanguageConfig,- language: str,- src: str,- cpp_exports: CppExportTypes,- seen: set[tuple[int, str]],-) -> tuple[Node, list[Node], str, _CppExportType | None, int] | None:- """``(def_node, name_nodes, name, export_type, start_line)`` for a new symbol match.-- None drops the match: no definition or name, a recovery match this file- does not support, a suppressed export macro, or a duplicate. The- ``(start_line, name)`` key is recorded in *seen* here.- """- def_nodes = capture_dict.get("symbol.def", [])- name_nodes = capture_dict.get("symbol.name", [])- captured_export_type_nodes = capture_dict.get("symbol.cpp_export_type", [])-- if (- captured_export_type_nodes- and captured_export_type_nodes[0].id not in cpp_exports.capture_ids- ):- # The query also sees ordinary ``struct Tag variable;`` forms;- # discard only the unsupported recovery match.- return None-- if not def_nodes or not name_nodes:- return None-- def_node = def_nodes[0]- name = _language_symbol_name(- language, def_node, config.symbol_name_fn(_node_text(name_nodes[0], src), def_node.type), src- )- if name is None:- return None-- export_type = cpp_exports.defs.get(def_node.id)- if export_type is not None and name != export_type.name:- # The ordinary struct/class query sees the same specifier, but- # tree-sitter calls the export macro its name. Keep only the- # dedicated match whose name is the outer declarator.- return None-- if def_node.type == "preproc_def" and (- _cpp_normalize_identifier(name) in cpp_exports.macro_names- or def_node.id in cpp_exports.macro_def_ids- ):- # Body-form macros are suppressed by name as before #1901;- # ambiguous forward declarations suppress only the exact- # active definition that made recovery safe.- return None-- start_line = def_node.start_point[0] + 1- if export_type is not None:- start_line = export_type.range_node.start_point[0] + 1- dedup_key = (start_line, name)- if dedup_key in seen:- return None- seen.add(dedup_key)- return def_node, name_nodes, name, export_type, start_line---_TS_OVERLOAD_SIGNATURES = frozenset({"function_signature", "method_signature"})---def _keep_ts_overloads_beside_a_body(symbols: list[Symbol], node_types: list[str]) -> list[Symbol]:- """Drop a bodiless TS signature whose id has no implementation in the file.-- That is an ambient declaration (``declare function``, a ``.d.ts``), not an- overload, and it never was a symbol.- """- bodied = {s.id for s, t in zip(symbols, node_types, strict=True) if t not in _TS_OVERLOAD_SIGNATURES}- return [- s- for s, t in zip(symbols, node_types, strict=True)- if t not in _TS_OVERLOAD_SIGNATURES or s.id in bodied- ]---def _drop_swift_extensions_of_local_types(symbols: list[Symbol]) -> list[Symbol]:- """Drop an ``extension`` of a type the same file declares.-- It shares the type's id, so it would replace the type's node and kind; its- members already name the type as their parent.- """- declared = {s.id for s in symbols if s.kind != "impl"}- return [s for s in symbols if s.kind != "impl" or s.id not in declared]---def _is_declaration(- def_node: Node,- config: LanguageConfig,- language: str,- export_type: _CppExportType | None,- src: str,-) -> bool:- node_type = def_node.type- if node_type in config.declaration_node_types:- return True- if language == "python":- return _is_python_overload(def_node, src)- if export_type is not None:- return export_type.is_forward_declaration- return _is_bodiless_cpp_type(language, node_type, def_node)---def _objc_call_target(- site_node: Node, target_node: Node, target_name: str, has_receiver: bool, src: str-) -> str | None:- """Objective-C call target name, or None to drop the match."""- # A message send binds one `method:` child per keyword, so- # `[view setTitle:t forState:s]` matches the query twice. Join the whole- # selector on the first match and drop the rest.- if site_node.type == "message_expression":- return _objc_message_selector(site_node, target_node, src)- # A block variable invoked with C call syntax (`completionBlock(hit)`) is- # not a call to a same-named function or property.- if not has_receiver and _objc_call_is_block_variable(site_node, target_name, src):- return None- return target_name---def _jsx_supplied_props(site_node: Node, src: str) -> frozenset[str] | None:- """Props written on a JSX element, or None when not JSX or a spread hides them."""- if site_node.type not in ("jsx_self_closing_element", "jsx_opening_element"):- return None- props_set: set[str] = set()- for child in site_node.children:- if child.type == "jsx_attribute":- name = next(- (sub for sub in child.children if sub.type in ("property_identifier", "identifier")),- None,- )- if name is not None:- props_set.add(_node_text(name, src))- elif child.type == "jsx_expression" and any(- sub.type == "spread_element" for sub in child.children- ):- return None- return frozenset(props_set)---def _fold_receiverless_twins(calls: list[tuple[tuple[int, int, str], CallSite]]) -> list[CallSite]:- """Fold the receiver-less copy of a member call into the copy with a receiver.-- A grammar's bare-call pattern that leaves the object unconstrained (Java's- ``method_invocation``, Ruby's ``call``) matches ``obj.m()`` too. Resolved on- its own, that copy binds ``m`` by bare name beside whatever ``obj``'s type- answers, so it survives only as the member call's ``bare_name_fallback``.- Keyed on the call node, not the line, so a bare ``m(obj.m())`` keeps its own- site.- """- member_sites = {site for site, call in calls if not _is_bare(call)}- bare_sites = {site for site, call in calls if site in member_sites and _is_bare(call)}- folded: list[CallSite] = []- for site, call in calls:- if _is_bare(call):- if site not in member_sites:- folded.append(call)- elif call.receiver_name and site in bare_sites:- folded.append(replace(call, bare_name_fallback=True))- else:- folded.append(call)- return folded---def _is_bare(call: CallSite) -> bool:- return not (call.receiver_name or call.receiver_call)---def _dedupe_calls(calls: list[CallSite]) -> list[CallSite]:- """One call per ``(line, target, receiver)``, keeping the richest record.-- Two scoped-call patterns can match the same two-part call (one keeps the- qualifier), so the richer record wins whichever order they arrive in.- """- deduplicated: dict[tuple[int, str, str | None], CallSite] = {}- for call in calls:- key = (call.line, call.target_name, call.receiver_name)- existing = deduplicated.get(key)- if (- existing is None- or (existing.receiver_call is None and call.receiver_call is not None)- or (existing.scope_name is None and call.scope_name is not None)- ):- deduplicated[key] = call- return list(deduplicated.values())---_FSHARP_BINDING_NODE_TYPES = ("function_declaration_left", "value_declaration_left")-_DART_FUNCTION_NODE_TYPES = ("function_signature", "getter_signature", "setter_signature")---def _is_fsharp_binding(language: str, node_type: str) -> bool:- return language == "fsharp" and node_type in _FSHARP_BINDING_NODE_TYPES---def _language_symbol_name(language: str, def_node: Node, name: str, src: str) -> str | None:- """Apply a language's own naming rule; None drops the match."""- if not name:- return None- if language == "elixir":- # A `def` inside `quote do ... end` is macro body, not a- # definition of the module that writes it.- if _elixir_is_template_definition(def_node, src):- return None- # `defimpl Proto, for: Type` is named for the module the- # compiler generates, not for the protocol alone.- return _elixir_symbol_name(def_node, name, src)- if language == "objectivec":- # `typedef NS_ENUM(NSInteger, Kind) { ... }` has no grammar- # rule, so only its enum *cases* survive as declarators and- # each would become a symbol named as though it were the type.- if _objc_is_macro_enum(def_node, src):- return None- # A method is named by its whole selector- # (`initWithName:age:`), which no single node holds, and a- # category by the class it extends plus its own name.- return _objc_symbol_name(def_node, name, src)- return name---def _refine_symbol_kind(- kind: str, def_node: Node, config: LanguageConfig, language: str, src: str-) -> str:- """Narrow a node-type kind where one node shape spells several kinds."""- node_type = def_node.type- # A value binding that carries parameter patterns beside its- # name is a function the grammar reparsed as a value because- # of its return-type annotation.- if (- language == "fsharp"- and node_type == "value_declaration_left"- and _fsharp_binding_has_params(def_node)- ):- kind = "function"-- # Refine "struct" kind for Go type_spec (check if struct or interface body)- if kind == "struct" and config.parent_extraction == "receiver":- kind = refine_go_type_kind(def_node, src)-- # Refine "class" kind for Kotlin (interface / enum class share class_declaration)- if kind == "class" and language == "kotlin" and node_type == "class_declaration":- kind = refine_kotlin_class_kind(def_node)-- if kind == "class" and language == "swift" and node_type == "class_declaration":- kind = refine_swift_class_kind(def_node)-- # Pascal ``declType`` wraps class, record, interface, enum and more.- if kind == "class" and language == "pascal" and node_type == "declType":- kind = refine_pascal_type_kind(def_node)-- # Elixir: every definition is a ``call``; its keyword names the kind.- if language == "elixir" and node_type == "call":- kind = refine_elixir_call_kind(def_node, src)-- # F# writes a class, a struct and an interface with the same- # ``anon_type_defn`` node; only the body says which it is.- if kind == "class" and language == "fsharp" and node_type == "anon_type_defn":- kind = refine_fsharp_type_kind(def_node)- return kind---def _symbol_end_line(- def_node: Node,- config: LanguageConfig,- language: str,- export_type: _CppExportType | None,-) -> int | None:- """The symbol's last line, or None for a Dart local function (not a symbol)."""- node_type = def_node.type- end_line = def_node.end_point[0] + 1- if config.symbol_end_line_fn is not None:- end_line = config.symbol_end_line_fn(def_node, end_line)- if export_type is not None:- end_line = export_type.range_node.end_point[0] + 1- # F#: the captured left-hand side stops at the parameters; extend over the body.- if _is_fsharp_binding(language, node_type):- end_line = _fsharp_binding_end_line(def_node)- if language == "dart" and node_type in _DART_FUNCTION_NODE_TYPES:- return _dart_function_end_line(def_node, end_line)- return end_line---def _dart_function_end_line(def_node: Node, end_line: int) -> int | None:- """Dart: a function's body is a sibling of its signature.-- Local functions nested inside another function's body have no callable- *ancestor* (the enclosing signature is a sibling), so they are dropped- here (None); and the line range extends to the trailing body sibling or- call-site attribution stops at the signature line.- """- ancestor = def_node.parent- while ancestor is not None:- if ancestor.type in ("function_body", "function_expression"):- return None- ancestor = ancestor.parent- anchor = def_node- if def_node.parent is not None and def_node.parent.type == "method_signature":- anchor = def_node.parent- body_sibling = anchor.next_named_sibling- if body_sibling is not None and body_sibling.type == "function_body":- return body_sibling.end_point[0] + 1- return end_line---def _module_binding_kind(def_node: Node, name: str, language: str, src: str) -> str | None:- """Kind of a module-level assignment, or None when it only binds a module.-- SCREAMING_CASE names are constants by convention; the rest are module- variables (singletons like ``app = FastAPI()``, registries, caches).- ``str.isupper()`` requires at least one cased char, so names with no- letters (``_``, ``__all__``) fall to "variable" rather than being- mislabelled constants by ``name == name.upper()``.- """- # TS/JS: ``const svc = require('./svc')`` binds a module (already an- # import), so it is not a symbol.- if language in _TS_JS_LANGUAGES and declarator_value_is_module_ref(def_node, src):- return None- # A structurally callable value is a function whatever its name says.- callable_kind = (- declarator_binds_callable(def_node, src) if language in _TS_JS_LANGUAGES else None- )- return callable_kind or ("constant" if name.isupper() else "variable")---def _attribute_decorators(def_node: Node, language: str, src: str) -> list[str]:- """Attribute texts that act as decorators (Rust, C#, C/C++), brackets stripped.-- They land beside ``@`` decorators so one deprecation check reads them all.- """- if language == "rust":- return _rust_outer_attributes(def_node, src)- # C#: [Obsolete] / [System.Obsolete] are ``attribute_list`` nodes.- # In tree-sitter-c-sharp the attribute_list is child[0] of the- # declaration node itself (method_declaration, class_declaration, etc.),- # NOT a preceding sibling in the class body.- if language == "csharp":- return _leading_child_attributes(def_node, "attribute_list", "[", "]", src)- # C/C++: [[deprecated]] / [[deprecated("reason")]] are- # ``attribute_declaration`` nodes, child[0] of function_definition itself- # (NOT a preceding sibling at translation_unit level).- if language in ("cpp", "c"):- return _leading_child_attributes(def_node, "attribute_declaration", "[[", "]]", src)- if language == "kotlin":- return _kotlin_detached_annotation(def_node, src)- return []---def _kotlin_detached_annotation(def_node: Node, src: str) -> list[str]:- """Kotlin: annotations the grammar split off a top-level declaration.-- When the last annotation on its own line before a bare ``fun`` or- ``object`` takes one argument (``@Suppress("unused")``), the argument list- reads as a parenthesized expression: the annotations become an- ``annotated_expression`` chain beside a declaration with no modifiers.- """- if def_node.parent is None or def_node.parent.type != "source_file":- return []- sib = def_node.prev_sibling- while sib is not None and sib.type in ("line_comment", "block_comment"):- sib = sib.prev_sibling- return [_node_text(sib, src).strip()] if _is_split_annotation_chain(sib) else []---def _is_split_annotation_chain(node: Node | None) -> bool:- """Nested ``annotated_expression`` nodes of annotations, the last one's argument at the tail."""- while node is not None and node.type == "annotated_expression":- kinds = [c.type for c in node.named_children]- if len(kinds) != 2 or kinds[0] != "annotation":- return False- if kinds[1] == "parenthesized_expression":- return True- node = node.named_children[1]- return False---def _declared_modifiers(def_node: Node, node_types: frozenset[str], src: str) -> tuple[str, ...]:- """Lowercased keyword modifiers written in *def_node*'s modifier children.-- Keywords are the anonymous leaves of those children (C# ``modifier`` >- ``override``, Kotlin ``modifiers`` > ``member_modifier`` > ``override``) or- a named leaf that is itself a modifier (VB.NET ``modifier`` ``Overrides``).- Named leaves such as Scala's ``protected[this]`` qualifier are not- keywords. Annotations and attributes nested among them are skipped: they- are decorators, not modifiers. The words are interned: a handful of- distinct values repeat on every symbol.- """- if not node_types:- return ()- words: list[str] = []- stack = [c for c in reversed(def_node.children) if c.type in node_types]- while stack:- node = stack.pop()- if "annotation" in node.type or "attribute" in node.type:- continue- if node.child_count == 0:- text = _node_text(node, src).strip()- keyword = not node.is_named or "modifier" in node.type- if keyword and text.isalpha():- words.append(sys.intern(text.lower()))- continue- stack.extend(reversed(node.children))- return tuple(words)---def _rust_outer_attributes(def_node: Node, src: str) -> list[str]:- """Rust: outer attributes (#[...]) are preceding siblings of the item."""- attrs: list[str] = []- if def_node.parent is None:- return attrs- siblings = def_node.parent.children- for j, sib in enumerate(siblings):- if sib.id != def_node.id:- continue- k = j - 1- while k >= 0 and siblings[k].type == "attribute_item":- attr_text = _node_text(siblings[k], src).strip()- # Strip #[ and ] to get the inner attribute text- if attr_text.startswith("#[") and attr_text.endswith("]"):- attrs.append(attr_text[2:-1])- k -= 1- break- return attrs---def _leading_child_attributes(- def_node: Node, node_type: str, opener: str, closer: str, src: str-) -> list[str]:- """Inner text of the attribute children leading *def_node*, until the first other child.-- A comment between attribute lists does not end the run, so an attribute- placed after one is still recorded.- """- attrs: list[str] = []- for child in def_node.children:- if child.type != node_type:- if child.type == "comment":- continue- break- attr_text = _node_text(child, src).strip()- if attr_text.startswith(opener) and attr_text.endswith(closer):- attrs.append(attr_text[len(opener) : -len(closer)])- return attrs---def _refine_visibility(- def_node: Node,- language: str,- visibility: str,- name: str,- ts_deferred_exports: frozenset[str] | None,- py_all_names: frozenset[str] | None,- src: str,-) -> tuple[str, bool]:- """``(visibility, is_exported_symbol)`` after the language's AST-context rules."""- # C/C++ visibility is dictated by AST context (access- # specifiers / storage class / export attributes), not by- # modifier text. Refine after the generic fn ran. Objective-C's- # grammar is a superset of C's, so the same shapes apply to- # plain C code inside a ``.m`` file.- if language in ("cpp", "c", "objectivec"):- return refine_cpp_visibility(def_node, visibility, src)- # C#: an unmodified declaration's default depends on what encloses- # it, which the modifier-text fn cannot see.- if language == "csharp":- return refine_csharp_visibility(def_node, visibility), False- # Java: no access keyword means package-private, except inside an- # interface or annotation body, which the modifier-text fn cannot see.- if language == "java":- return refine_java_visibility(def_node), False- # TS/JS: a top-level declaration is only public when exported —- # inline, via ``export { x }`` lists, or ``export default x``.- if language in _TS_JS_LANGUAGES:- return refine_ts_visibility(def_node, visibility, name, ts_deferred_exports), False- # Python: a literal ``__all__`` raises the names it lists to public; it- # never demotes the ones it omits. GDScript shares ``py_visibility`` but- # has no ``__all__``, so the gate is on the language, not the fn.- if language == "python":- return refine_py_visibility(def_node, visibility, name, py_all_names), False- # Rust: a trait's items may not write ``pub`` of their own, so the- # trait's modifier is the only place their visibility is stated.- if language == "rust":- return refine_rust_visibility(def_node, visibility, src), False- return visibility, False---def _dart_mixin_parent(def_node: Node, config: LanguageConfig, src: str) -> str | None:- """Name of the Dart ``mixin`` enclosing *def_node*, if it is the nearest type."""- ancestor = def_node.parent- while ancestor is not None:- if ancestor.type == "mixin_declaration":- ident = next((c for c in ancestor.children if c.type == "identifier"), None)- return _node_text(ident, src) if ident is not None else None- if ancestor.type in config.parent_class_types:- return None- ancestor = ancestor.parent- return None---def _pascal_imports(- stmt_node: Node, module_node: Node, src: str, seen_units: set[str]-) -> list[Import]:- """One Pascal ``uses`` unit, deduped case-insensitively across the file.-- pascal.scm fires once per unit name, so a 3-unit clause arrives as 3- matches sharing one statement span: deduping by raw text would drop all- but the first. A unit named in both ``uses`` clauses is one dependency.- ``uses`` exposes the unit's whole interface, hence the ``*`` wildcard.- """- raw = _node_text(stmt_node, src).strip()- unit_name = _node_text(module_node, src).strip()- if not unit_name or unit_name.lower() in seen_units:- return []- seen_units.add(unit_name.lower())- return [- Import(- raw_statement=raw,- module_path=unit_name,- imported_names=["*"],- is_relative=False,- resolved_file=None,- bindings=[],- is_reexport=False,- )- ]---def _elixir_imports(- stmt_node: Node, module_node: Node, src: str, seen_modules: set[str]-) -> list[Import]:- """Elixir directives, one Import per module and deduped by module path.-- ``alias Foo.{Bar, Baz}`` names two modules in one statement. ``import``- pulls in every public function (``*``); alias/require/use bind the module.- """- raw = _node_text(stmt_node, src).split("\n", 1)[0].strip()- directive_node = stmt_node.child_by_field_name("target")- directive = _node_text(directive_node, src).strip() if directive_node else ""- imports: list[Import] = []- for module_path in elixir_import_modules(module_node, src):- if module_path in seen_modules:- continue- seen_modules.add(module_path)- imports.append(- Import(- raw_statement=raw,- module_path=module_path,- imported_names=["*"] if directive == "import" else [],- is_relative=False,- resolved_file=None,- bindings=[],- is_reexport=False,- )- )- return imports---def _fsharp_imports(- stmt_node: Node, module_node: Node, src: str, seen_raws: set[str]-) -> list[Import]:- """F# ``open``: every public name of the module (``*``), which bare-name- call resolution reads. ``open type A.B.T`` binds the type's static members,- so the module the file depends on is ``A.B``.- """- raw = _node_text(stmt_node, src).strip()- if raw in seen_raws:- return []- seen_raws.add(raw)- module_path = _node_text(module_node, src).strip()- if not module_path:- return []- names: list[str] = ["*"]- if any(child.type == "type" for child in stmt_node.children):- head, _, type_name = module_path.rpartition(".")- if head:- module_path, names = head, [type_name]- else:- names = []- return [- Import(- raw_statement=raw,- module_path=module_path,- imported_names=names,- is_relative=False,- resolved_file=None,- bindings=[],- is_reexport=False,- )- ]---def _statement_imports(- stmt_node: Node,- module_node: Node,- module_text: str,- raw: str,- language: str,- src: str,-) -> list[Import]:- """The imports one deduplicated statement declares."""- if language == "scala" and stmt_node.type == "import_declaration":- return _scala_imports(stmt_node, raw, src)- if language == "php" and stmt_node.type == "namespace_use_declaration":- return _php_imports(stmt_node, raw, src)- if language == "dart":- return [_dart_import(stmt_node, module_node, module_text, raw, src)]- if language == "rust" and stmt_node.type == "use_declaration":- return _rust_use_imports(stmt_node, module_text, raw, src)- if language == "rust" and stmt_node.type == "macro_invocation":- from .extractors.bindings.rust import macro_mod_imports-- # Most top-level calls (``s! { struct ... }``) declare no module.- return macro_mod_imports(module_node, raw) if "mod" in raw else []- if language in _TS_JS_LANGUAGES and _is_dynamic_esm_import(stmt_node):- # ``import('./mod')`` binds a module namespace at runtime, so it is a- # wildcard, which keeps the target's exports live. The names it is- # destructured into still bind, for call resolution.- return [- Import(- raw_statement=raw,- module_path=module_text,- imported_names=["*"],- is_relative=module_text.startswith("."),- resolved_file=None,- bindings=dynamic_import_bindings(stmt_node, src),- is_reexport=False,- )- ]- if language in ("javascript", "typescript") and stmt_node.type in (- "assignment_expression",- "call_expression",- ):- return _cjs_imports(stmt_node, raw, src)- return [_generic_import(stmt_node, module_text, raw, language, src)]---def _scala_imports(stmt_node: Node, raw: str, src: str) -> list[Import]:- """Scala: one Import per selected name, with full dotted paths.-- The query's ``(identifier)`` capture is only the first path segment, and- one declaration can hold several clauses, brace selectors, renames and- wildcards.- """- from .extractors.bindings.scala import expand_scala_import_clauses- from .models import NamedBinding-- imports: list[Import] = []- for clause_path, clause_names in expand_scala_import_clauses(stmt_node, src):- local = clause_names[0]- exported = None if local == "*" else clause_path.rsplit(".", 1)[-1]- imports.append(- Import(- raw_statement=raw,- module_path=clause_path,- imported_names=clause_names,- is_relative=False,- resolved_file=None,- bindings=[- NamedBinding(- local_name=local,- exported_name=exported,- source_file=None,- )- ],- is_reexport=False,- )- )- return imports---def _php_imports(stmt_node: Node, raw: str, src: str) -> list[Import]:- """PHP: one ``use`` declaration can name several classes, each its own file."""- from .extractors.bindings.php import php_class_name, php_use_clauses- from .models import NamedBinding-- return [- Import(- raw_statement=raw,- module_path=fqn,- imported_names=[php_class_name(fqn)],- is_relative=False,- resolved_file=None,- bindings=[NamedBinding(local_name=local, exported_name=fqn, source_file=None)],- is_reexport=False,- )- for fqn, local in php_use_clauses(stmt_node, src)- ]---def _dart_import(- stmt_node: Node, module_node: Node, module_text: str, raw: str, src: str-) -> Import:- """Dart: URIs are relative unless schemed, ``export`` re-exports, and the- dotted ``part of library.name;`` form resolves through ``library:``.- """- module_path = module_text- if module_node.type == "dotted_identifier_list":- module_path = f"library:{module_text}"- imported_names, bindings = extract_import_bindings(stmt_node, src, "dart")- return Import(- raw_statement=raw,- module_path=module_path,- imported_names=imported_names,- is_relative=not module_path.startswith(("package:", "dart:", "library:")),- resolved_file=None,- bindings=bindings,- is_reexport=stmt_node.type == "library_export",- )---def _is_dynamic_esm_import(stmt_node: Node) -> bool:- if stmt_node.type != "call_expression":- return False- fn = stmt_node.child_by_field_name("function")- return fn is not None and fn.type == "import"---def _cjs_imports(stmt_node: Node, raw: str, src: str) -> list[Import]:- """CommonJS: every ``require()`` in the statement, re-exports marked.-- A hub like ``Object.assign(module.exports, require('./a'), ...)`` is- several imports, and ``module.exports`` shapes are barrels.- """- from .extractors.bindings.ts_js import (- cjs_statement_is_reexport,- collect_cjs_requires,- )-- cjs_reexport = cjs_statement_is_reexport(stmt_node, src)- return [- Import(- raw_statement=raw,- module_path=cjs_module,- imported_names=["*"] if cjs_reexport else [],- is_relative=cjs_module.startswith("."),- resolved_file=None,- bindings=[],- is_reexport=cjs_reexport,- )- for cjs_module in collect_cjs_requires(stmt_node, src)- ]---def _generic_import(- stmt_node: Node, module_text: str, raw: str, language: str, src: str-) -> Import:- """The single Import of a statement no language-specific shape claims."""- is_module_declaration = language == "rust" and stmt_node.type == "mod_item"- if is_module_declaration:- module_text = _rust_mod_path_attribute(stmt_node, src) or module_text-- # JVM wildcard imports: the query captures the scoped identifier only,- # so ``import com.foo.*`` arrives as ``com.foo``. Restore the ``.*``.- if language in ("java", "kotlin") and not module_text.endswith("*"):- stmt_text = raw.rstrip().rstrip(";").rstrip()- if stmt_text.endswith(".*"):- module_text += ".*"-- # Language-specific import name + binding extraction- imported_names, bindings = extract_import_bindings(stmt_node, src, language)- is_relative = (- module_text.startswith(".")- or module_text.startswith("./")- or module_text.startswith(("self::", "super::", "crate::"))- )- # ``type_only`` / ``deferred`` are set by the caller from ``import_load_kind``.- return Import(- raw_statement=raw,- module_path=module_text,- imported_names=imported_names,- is_relative=is_relative,- resolved_file=None,- bindings=bindings,- is_reexport=_is_reexport_import(stmt_node, raw, language),- is_module_declaration=is_module_declaration,- )---def _rust_use_imports(stmt_node: Node, module_text: str, raw: str, src: str) -> list[Import]:- """Rust: one Import per leaf of the use tree, each with its own path.-- ``use crate::{a::B, c::D}`` names two modules, and the joined string- resolves to neither. ``super`` hops that only climb out of inline- ``mod`` blocks stay inside this file, so they are dropped here and the- resolver sees the path the file itself would write.- """- from .extractors.bindings.rust import expand_rust_use_tree, rust_use_argument- from .models import NamedBinding-- arg_node = rust_use_argument(stmt_node)- leaves = expand_rust_use_tree(arg_node, src) if arg_node is not None else []- inline_depth = _rust_inline_mod_depth(stmt_node)- is_reexport = _is_reexport_import(stmt_node, raw, "rust")- if len(leaves) <= 1 and "{" not in module_text and not inline_depth:- return [_generic_import(stmt_node, module_text, raw, "rust", src)]- imports = []- for path, local, exported in leaves:- path = _strip_inline_super(path, inline_depth)- imports.append(- Import(- raw_statement=raw,- module_path=path,- imported_names=[local],- is_relative=path.startswith(("self::", "super::", "crate::")),- resolved_file=None,- bindings=[NamedBinding(local_name=local, exported_name=exported, source_file=None)],- is_reexport=is_reexport,- )- )- return imports---def _rust_inline_mod_depth(node: Node) -> int:- """How many inline ``mod name { ... }`` blocks enclose *node* in its file."""- depth = 0- parent = node.parent- while parent is not None:- if parent.type == "mod_item":- depth += 1- parent = parent.parent- return depth---def _strip_inline_super(path: str, inline_depth: int) -> str:- """Drop the ``super`` hops of *path* that stay inside the file.-- In ``mod tests { use super::*; }`` the ``super`` is this file's own- module, so the path is ``self::*``, not the parent module's glob.- """- if not inline_depth:- return path- segments = path.split("::")- hops = 0- while hops < len(segments) and segments[hops] == "super":- hops += 1- if not hops:- return path- if hops <= inline_depth:- return "::".join(["self", *segments[hops:]])- return "::".join(segments[inline_depth:])---def _rust_mod_path_attribute(stmt_node: Node, src: str) -> str | None:- """The ``#[path = "..."]`` override on a Rust ``mod`` item, if any.-- Outer attributes are preceding siblings of the item, not children.- """- parent = stmt_node.parent- if parent is None:- return None- siblings = parent.children- for j, sib in enumerate(siblings):- if sib.id != stmt_node.id:- continue- # Walk backward through preceding attribute_item siblings- k = j - 1- while k >= 0 and siblings[k].type == "attribute_item":- path_match = re.search(r'path\s*=\s*"([^"]+)"', _node_text(siblings[k], src))- if path_match:- return path_match.group(1)- k -= 1- return None- return None---def _is_reexport_import(stmt_node: Node, raw: str, language: str) -> bool:- """``pub use`` (Rust) and ``@_exported import`` (Swift) re-export the module."""- if language == "rust" and stmt_node.type == "use_declaration":- return any(child.type == "visibility_modifier" for child in stmt_node.children)- return language == "swift" and raw.startswith("@_exported")---def _parse_tree(- lang: str, language: Language, grammar_tag: str, source: bytes-) -> tuple[Node, list[str], str, Language]:- """``(root, parse errors, grammar tag, language)``; a JSX-bearing ``.ts`` retries as TSX."""- parser = Parser(language)- tree = parser.parse(source)- root = tree.root_node-- parse_errors = _collect_error_nodes(root)-- # Adaptive TSX grammar fallback: if a .ts file contains JSX markup,- # tree-sitter-typescript produces ERROR nodes. Re-parse using the TSX- # grammar ONLY IF:- # 1. Initial parse yielded error nodes (parse_errors is non-empty)- # 2. The file projects to TypeScript and is not already on tsx- # 3. Source contains JSX-specific closing tokens (b"/>" or b"</")- # A .vue render function may be written in JSX- # (``vnodes.push(<i class={c} />)``), which is the single TS parse- # failure across a 1,593-file .vue corpus. ``source`` here is the- # markup-blanked projection, so the template's own ``</`` and ``/>``- # are already spaces — the token test still keys on real JSX only.- # The swap is strictly safe: it only takes effect when TSX yields- # FEWER errors than the first parse.- if (- parse_errors- and lang in ("typescript", "vue")- and grammar_tag != "tsx"- and (b"/>" in source or b"</" in source)- ):- tsx_language = _get_language("tsx")- if tsx_language is not None:- tsx_tree = Parser(tsx_language).parse(source)- tsx_errors = _collect_error_nodes(tsx_tree.root_node)- if len(tsx_errors) < len(parse_errors):- tree = tsx_tree- root = tree.root_node- parse_errors = tsx_errors- # Both grammar_tag AND language must be reassigned.- # grammar_tag is consumed immediately below by- # the caller's _get_query(lang, language, grammar_tag), which- # appends tsx.scm to the base typescript.scm query.- # That append is what supplies the- # jsx_opening_element / jsx_self_closing_element captures- # that restore JSX component call-site edges.- # Reassigning only ``tree`` / ``root`` would fix parse- # errors but leave those edges missing — the dead-code- # false-positive would remain.- grammar_tag = "tsx"- language = tsx_language- return root, parse_errors, grammar_tag, language---def _parent_type_name(ancestor: Node, src: str) -> str | None:- """The name a class/impl ancestor gives its members, without type arguments."""- name_node = ancestor.child_by_field_name("name") or (- ancestor.child_by_field_name("type") # Rust impl_item- )- if name_node is None:- return None- # For Rust impl blocks with generic types (e.g. impl<T> Foo<T>),- # extract only the base type name, not the full generic signature.- if name_node.type in ("generic_type", "scoped_type_identifier"):- inner = name_node.child_by_field_name(- "type" if name_node.type == "generic_type" else "name"- )- if inner and inner.type == "type_identifier":- name_node = inner- return _node_text(name_node, src)---def _nesting_parent(def_node: Node, config: LanguageConfig, src: str) -> str | None:- """The class/impl ancestor naming *def_node*'s parent.-- An anonymous class is a class of its own, so its members gain segments- below it: ``Gson::FILTER::Filter::check``.- """- segments: list[str] = []- child, ancestor = def_node, def_node.parent- while ancestor is not None:- if ancestor.type in config.parent_class_types:- name = _parent_type_name(ancestor, src)- if name is not None:- return "::".join([name, *reversed(segments)])- else:- segment = scope_segment(- ancestor, child, bool(segments), config.anonymous_class_types, src- )- if segment is not None:- segments.append(segment)- child, ancestor = ancestor, ancestor.parent- # A Kotlin top-level function's object expression: ``f::Foo``.- return "::".join(reversed(segments)) or None---def _java_enum_constant(def_node: Node, config: LanguageConfig, src: str) -> str | None:- """The enum constant whose body declares *def_node*, before any enclosing type.-- An anonymous class is such a type: ``_find_parent`` names its members.- """- ancestor = def_node.parent- while (- ancestor is not None- and ancestor.type not in config.parent_class_types- and ancestor.type not in config.anonymous_class_types- ):- if ancestor.type == "enum_constant":- name = ancestor.child_by_field_name("name")- return _node_text(name, src) if name else None- ancestor = ancestor.parent- return None---class ASTParser:- """Unified AST parser — works for all languages via .scm query files.-- Usage::-- parser = ASTParser()- parsed = parser.parse_file(file_info, source_bytes)-- Adding a new language:- 1. Write ``packages/core/queries/<lang>.scm``- 2. Add one entry to ``LANGUAGE_CONFIGS``- That's it. No Python class, no new module.- """-- def __init__(self) -> None:- pass-- def parse_imports(self, file_info: FileInfo, source: bytes) -> list[Import]:- """Only *source*'s imports: :meth:`parse_file`'s grammar, query and import pass.-- Symbols, calls and references are not extracted, which is most of a- parse. A language without a tree-sitter path takes the full parse.- """- lang = file_info.language- config = LANGUAGE_CONFIGS.get(lang)- grammar_tag = grammar_tag_for(lang, file_info.path)- language = _get_language(grammar_tag)- signature_file = lang == "fsharp" and file_info.path.endswith(".fsi")- if (- lang in SPECIAL_HANDLER_LANGUAGES- or config is None- or language is None- or signature_file- ):- return self.parse_file(file_info, source).imports- source = prepare_source(lang, source, path=file_info.path)- root, _, grammar_tag, language = _parse_tree(lang, language, grammar_tag, source)- query = self._get_query(lang, language, grammar_tag)- matches = _run_query(query, root) if query is not None else []- src = source.decode("utf-8", errors="replace")- return self._extract_imports(matches, config, file_info, src)-- def parse_file(self, file_info: FileInfo, source: bytes) -> ParsedFile:- """Parse *source* bytes and return a fully populated ParsedFile."""- lang = file_info.language- content_hash = compute_content_hash(source)-- # Non-tree-sitter formats (OpenAPI, Dockerfile, Makefile, SQL) parse- # via dedicated handlers. Checked before the grammar lookup: none of- # these tags carry a LanguageConfig, so the no-grammar fallback below- # would otherwise swallow them.- if lang in SPECIAL_HANDLER_LANGUAGES:- parsed = parse_special(file_info, source, lang)- parsed.content_hash = content_hash- return parsed-- config = LANGUAGE_CONFIGS.get(lang)- # .tsx needs the JSX-aware grammar: the default one's ERROR-node- # recovery hoists nested helpers (handlers defined inside component- # bodies) to the top level.- grammar_tag = grammar_tag_for(lang, file_info.path)- language = _get_language(grammar_tag)-- # tree-sitter-fsharp ships a second grammar (``language_signature``)- # for .fsi signature files, and a spec loads exactly one. Read with- # the implementation grammar, a signature file's ``val`` and member- # signatures land in ERROR recovery, which hoists whatever the- # recovery invents into the symbol list. The regex tier still gives- # these files their ``open`` imports, which is all a signature file- # contributes that another file does not also state.- signature_file = lang == "fsharp" and file_info.path.endswith(".fsi")-- if config is None or language is None or signature_file:- if config is not None and language is None and lang not in _MISSING_GRAMMAR_REPORTED:- # Once per language, not once per file: the fact is about the- # environment, and it does not become truer on the four- # thousandth shell script.- _MISSING_GRAMMAR_REPORTED.add(lang)- log.debug("tree-sitter grammar unavailable", language=lang)- # Languages without a grammar may still carry regex-tier import- # extraction (their specs declare import_support="partial");- # symbols stay empty — the regex tier claims no symbol knowledge.- from .lightweight_imports import extract_lightweight_imports-- return ParsedFile(- file_info=file_info,- symbols=[],- imports=extract_lightweight_imports(file_info, source),- exports=[],- docstring=None,- parse_errors=[],- content_hash=content_hash,- )-- # An SFC (.svelte, .vue) is three languages in one file.- # ``prepare_source`` blanks the markup and <style> so what reaches the- # TypeScript grammar is valid TS at byte-identical offsets — no offset- # translation is needed anywhere downstream. A no-op for every other- # language without a registered locator/sanitizer -- Pascal's- # sanitizers (project-file `in '...'` clauses, ERROR-node blanking)- # live behind the same hook; see prepare_pascal_source in- # parser_helpers.py for why they're wired in here rather than as- # ad-hoc if-blocks.- # ``content_hash`` above deliberately hashes the ORIGINAL bytes, so- # incremental update still tracks the real file.- original_source = source- source = prepare_source(lang, source, path=file_info.path)-- root, parse_errors, grammar_tag, language = _parse_tree(lang, language, grammar_tag, source)- src = source.decode("utf-8", errors="replace")-- query = self._get_query(lang, language, grammar_tag)-- # Execute the compiled query ONCE per file. The five extraction- # passes below all consume the same capture dicts read-only;- # re-running ``cursor.matches()`` per pass multiplied the most- # expensive part of parsing by five.- matches = _run_query(query, root) if query is not None else []-- symbols = self._extract_symbols(matches, config, file_info, src)- # Per-language synthetic-symbol pass — recognises source-generator- # attributes (e.g. CommunityToolkit.Mvvm) and adds the symbols the- # generator would emit at compile time. No-op for languages- # without a registered extractor.- synthetic = extract_synthetic_symbols(root, src, file_info)- if synthetic:- # Keyed with the discriminator too, so a record's synthetic- # canonical constructor survives beside an explicit overload of- # another arity. A language without one dedupes on the id alone.- existing = {(s.id, symbol_discriminator(s)) for s in symbols}- symbols.extend(s for s in synthetic if (s.id, symbol_discriminator(s)) not in existing)- if lang == "csharp":- # The grammar reads a C# 14 extension block as a constructor- # named ``extension``; its members were added above instead.- symbols = drop_extension_block_symbols(symbols, root, src)- # Before calls and references are extracted, so both attribute to the- # final ids. Anonymous members go last, once the overloads holding them- # carry their discriminators.- anonymous = [s for s in symbols if is_anonymous_member(s)]- disambiguate_colliding_ids([s for s in symbols if not is_anonymous_member(s)], lang)- follow_renamed_hosts(symbols)- disambiguate_colliding_ids(anonymous, lang)- imports = self._extract_imports(matches, config, file_info, src)- calls = self._extract_calls(matches, config, file_info, src, symbols)- # An SFC instantiates a component by writing its tag in the markup- # (``<Foo />``), which the blanked TS buffer no longer contains. Mint- # those call sites from the markup grammar directly — the same way- # tsx.scm turns ``<Component />`` into a call for React. Returns [] for- # every non-SFC language.- calls.extend(component_call_sites(lang, original_source, symbols))- references = (- self._extract_references(matches, file_info, src, symbols, imports)- if lang in _REFERENCE_LANGUAGES- else []- )- heritage = extract_heritage(matches, config, file_info, src)- exports = self._derive_exports(symbols, config)- export_aliases = ts_export_aliases(src) if lang in _TS_JS_LANGUAGES else {}- docstring = extract_module_docstring(root, src, lang)- type_refs = self._extract_type_refs(matches, src, lang)-- # Same-file reference rescue (Python only): top-level symbols used- # elsewhere in their own module in a non-call / non-import position- # (callable passed as an arg, type annotation, decorator, default)- # carry no graph edge, so the dead-code unused-export pass would flag- # them. Stamp the referenced names so the analyzer can rescue them.- local_refs, registrars = (- _python_module_facts(src, symbols) if lang == "python" else (frozenset(), frozenset())- )-- if len(symbols) > _SYMBOL_COUNT_WARN_THRESHOLD:- log.warning(- "parser.symbol_bloat",- path=file_info.path,- language=lang,- symbol_count=len(symbols),- threshold=_SYMBOL_COUNT_WARN_THRESHOLD,- )-- return ParsedFile(- file_info=file_info,- symbols=symbols,- imports=imports,- exports=exports,- export_aliases=export_aliases,- calls=calls,- heritage=heritage,- docstring=docstring,- parse_errors=parse_errors,- content_hash=content_hash,- type_refs=type_refs,- local_refs=local_refs,- registrars=registrars,- references=references,- )-- # ------------------------------------------------------------------- # Query loading- # -------------------------------------------------------------------- def _get_query(- self, lang: str, language: Language, grammar_tag: str | None = None- ) -> object | None:- """Load and cache the compiled tree-sitter Query for *lang*."""- return _load_compiled_query(lang, grammar_tag)-- # ------------------------------------------------------------------- # Symbol extraction- # -------------------------------------------------------------------- def _extract_symbols(- self,- matches: list[dict],- config: LanguageConfig,- file_info: FileInfo,- src: str,- ) -> list[Symbol]:- language = file_info.language- symbols: list[Symbol] = []- seen: set[tuple[int, str]] = set() # (start_line, name) — dedup decorated dupes- # Parallel to ``symbols`` (same indices) -- consumed for Pascal and- # Objective-C, to dedupe interface-declaration vs. implementation- # method pairs after the loop, and for TS overload signatures. See- # _dedupe_pascal_interface_symbols / _dedupe_objc_interface_symbols.- node_types: list[str] = []- # Also parallel to ``symbols``, Objective-C only: which of @interface /- # @implementation / @protocol declared each member, so a protocol's- # method is never deduped against a same-named class method.- objc_container_kinds: list[str | None] = []-- # Deferred-export names (``export { x }`` / ``export default x``),- # computed once per file for the TS/JS visibility refinement.- ts_deferred_exports: frozenset[str] | None = None- if language in _TS_JS_LANGUAGES:- ts_deferred_exports = ts_deferred_export_names(src)- # Literal module ``__all__`` names, once per file for the Python- # refinement. None means no signal (built at runtime or absent).- py_all_names: frozenset[str] | None = None- if language == "python":- py_all_names = py_module_all_names(src)- cpp_exports = (- collect_cpp_export_types(matches, src) if language == "cpp" else CppExportTypes()- )-- for capture_dict in matches:- built = self._symbol_from_match(- capture_dict,- config,- file_info,- src,- cpp_exports,- ts_deferred_exports,- py_all_names,- seen,- )- if built is None:- continue- symbol, def_node = built- symbols.append(symbol)- node_types.append(def_node.type)- if language == "objectivec":- container = _objc_container_node(def_node, config.parent_class_types)- objc_container_kinds.append(container.type if container else None)-- if language == "pascal":- symbols = _dedupe_pascal_interface_symbols(symbols, node_types)-- if language in _TS_JS_LANGUAGES:- symbols = _keep_ts_overloads_beside_a_body(symbols, node_types)-- if language == "swift":- symbols = _drop_swift_extensions_of_local_types(symbols)-- # A .m file routinely declares its private methods in a class- # extension and defines them below in the @implementation, which- # builds each symbol id twice in one file.- if language == "objectivec":- symbols = _dedupe_objc_interface_symbols(symbols, node_types, objc_container_kinds)-- return symbols-- def _symbol_from_match(- self,- capture_dict: dict,- config: LanguageConfig,- file_info: FileInfo,- src: str,- cpp_exports: CppExportTypes,- ts_deferred_exports: frozenset[str] | None,- py_all_names: frozenset[str] | None,- seen: set[tuple[int, str]],- ) -> tuple[Symbol, Node] | None:- """One query match as a symbol and its definition node, or None to drop it.-- Records the match's ``(start_line, name)`` in *seen* once it passes the- name checks, so a later duplicate is dropped even if this one is.- """- language = file_info.language- identity = _match_identity(capture_dict, config, language, src, cpp_exports, seen)- if identity is None:- return None- def_node, name_nodes, name, export_type, start_line = identity-- node_type = def_node.type- # A TS/JS object-literal method inside an anonymous callback is kept,- # qualified by the binding that owns its object.- object_owner = (- _ts_nested_object_method_owner(def_node, src, config.symbol_node_types)- if language in _TS_JS_LANGUAGES- else None- )- kind = self._symbol_kind(- def_node,- config,- language,- src,- cpp_exports.container_ids,- # An anonymous class member is a method of its own class, even- # inside another method's body; ``_find_parent`` names it.- keep_nested=object_owner is not None- or in_anonymous_class(def_node, config.anonymous_class_types),- )- if kind is None:- return None-- end_line = _symbol_end_line(def_node, config, language, export_type)- if end_line is None:- return None-- if node_type in _MODULE_ANCHORED_NODE_TYPES:- kind = _module_binding_kind(def_node, name, language, src)- if kind is None:- return None-- modifier_texts = [_node_text(m, src) for m in capture_dict.get("symbol.modifiers", [])]- visibility, is_exported_symbol = _refine_visibility(- def_node,- language,- config.visibility_fn(name, modifier_texts),- name,- ts_deferred_exports,- py_all_names,- src,- )-- parent_name = object_owner or self._resolve_parent_name(- def_node,- config,- capture_dict.get("symbol.receiver", []),- name_nodes,- language,- src,- no_export_type=export_type is None,- export_type_parents=cpp_exports.parents,- )-- # A ``field_declaration`` cannot occur outside a class body, so a- # missing parent means the class did not parse. Grammar recovery,- # not a member function.- if node_type == "function_declarator" and parent_name is None:- return None-- # Upgrade function → method when a parent class is detected.- # F#: a nested module is a parent too (for id uniqueness), but it- # is not a type, so a `let` inside one stays a function.- if parent_name and kind == "function" and (- language != "fsharp" or _fsharp_parent_is_type(def_node)- ):- kind = "method"-- params_nodes = capture_dict.get("symbol.params", [])- params_text = _node_text(params_nodes[0], src) if params_nodes else ""- # The id scope adds a Java enum constant to its parent, so each- # constant's override is its own node while the enum stays its owner.- constant = _java_enum_constant(def_node, config, src) if language == "java" else None- scope = f"{parent_name}::{constant}" if parent_name and constant else parent_name- sym_id = f"{file_info.path}::{scope}::{name}" if scope else f"{file_info.path}::{name}"- symbol = Symbol(- id=sym_id,- name=name,- qualified_name=_build_qualified_name(file_info.path, scope, name),- kind=kind, # type: ignore[arg-type]- signature=build_signature(node_type, name, params_text, def_node, src),- start_line=start_line,- end_line=end_line,- docstring=extract_symbol_docstring(def_node, src, language),- decorators=(- [m for m in modifier_texts if m.startswith("@")]- + _attribute_decorators(def_node, language, src)- ),- visibility=visibility, # type: ignore[arg-type]- is_async=_is_async_node(def_node, src),- language=language,- parent_name=parent_name,- is_exported_symbol=is_exported_symbol,- is_declaration=_is_declaration(def_node, config, language, export_type, src),- type_parameter_count=(- _csharp_type_parameter_count(def_node) if language == "csharp" else None- ),- **_kotlin_receiver_fields(def_node, params_nodes, language, name),- modifiers=_declared_modifiers(def_node, config.modifier_node_types, src),- )- return symbol, def_node-- def _symbol_kind(- self,- def_node: Node,- config: LanguageConfig,- language: str,- src: str,- container_ids: frozenset[int],- *,- keep_nested: bool = False,- ) -> str | None:- """The symbol kind for *def_node*, or None when it is not a symbol here."""- node_type = def_node.type- kind = config.symbol_node_types.get(node_type)- if kind is None:- return None-- # Only module-level and class-body members are symbols; the query is- # recursive, so defs nested in a callable are dropped. Module-anchored- # node types only match at module level and skip the check, as does a- # def the caller has placed already (``keep_nested``).- if (- not keep_nested- and node_type not in _MODULE_ANCHORED_NODE_TYPES- and _has_callable_ancestor(def_node, config.symbol_node_types, container_ids)- ):- return None-- # F#: a ``let`` nested in another binding has no callable ancestor, so- # it is checked separately. A ``let`` in a type body is a field and stays.- if _is_fsharp_binding(language, node_type) and _fsharp_binding_is_nested(def_node):- return None- return _refine_symbol_kind(kind, def_node, config, language, src)-- def _resolve_parent_name(- self,- def_node: Node,- config: LanguageConfig,- receiver_nodes: list[Node],- name_nodes: list[Node],- language: str,- src: str,- *,- no_export_type: bool,- export_type_parents: dict[int, str],- ) -> str | None:- """The enclosing type or module name, trying each language's own shape."""- parent_name = self._find_parent(def_node, config, receiver_nodes, src)- if parent_name is not None:- return parent_name-- if language == "cpp" and no_export_type:- parent_name = _cpp_export_macro_parent(def_node, export_type_parents)- if parent_name is not None:- return parent_name-- # Dart mixin_declaration exposes no ``name`` field, so- # ``_find_parent``'s field lookup misses mixin members.- if language == "dart":- return _dart_mixin_parent(def_node, config, src)-- # F#: no type node carries a ``name`` field -- the name hangs off- # a ``type_name`` child -- so the generic walk finds the ancestor- # and then reads nothing off it.- if language == "fsharp":- return _fsharp_parent_name(def_node, src)-- # C/C++ out-of-line ``void Foo::method()`` names its class in the scope.- if language in ("cpp", "c") and name_nodes:- return _qualified_cpp_parent(name_nodes[0], src)-- # Pascal out-of-line ``function TFoo.Bar`` names its class in the header.- if language == "pascal" and name_nodes:- return _qualified_pascal_parent(name_nodes[0], src)-- # Elixir: ``defmodule`` is a ``call``; the name is its first argument.- if language == "elixir":- return _elixir_module_parent(def_node, src)-- # Objective-C containers name themselves with a bare first identifier.- if language == "objectivec":- return _objc_container_parent(def_node, config.parent_class_types, src)- return None-- def _find_parent(- self,- def_node: Node,- config: LanguageConfig,- receiver_nodes: list[Node],- src: str,- ) -> str | None:- """Determine the parent class/type for a symbol."""- if config.parent_extraction == "receiver":- # Go: extract type name from receiver parameter list- if receiver_nodes:- return extract_go_receiver_type(_node_text(receiver_nodes[0], src))- return None-- if config.parent_extraction in ("nesting", "impl"):- return _nesting_parent(def_node, config, src)-- return None # "none" mode-- # ------------------------------------------------------------------- # Import extraction- # -------------------------------------------------------------------- def _extract_imports(- self,- matches: list[dict],- config: LanguageConfig,- file_info: FileInfo,- src: str,- ) -> list[Import]:- language = file_info.language- imports: list[Import] = []- seen_raws: set[str] = set()- # Statement key -> the Imports it produced, so a repeat folds its load- # kind into them rather than adding a duplicate.- seen_imports: dict[str, list[Import]] = {}- seen_pascal_units: set[str] = set()- seen_elixir_modules: set[str] = set()-- for capture_dict in matches:- stmt_nodes = capture_dict.get("import.statement", [])- module_nodes = capture_dict.get("import.module", [])-- if not stmt_nodes or not module_nodes:- continue-- stmt_node = stmt_nodes[0]-- # These three dedupe on their own key, before the raw-statement- # dedup below.- if language == "pascal":- imports.extend(_pascal_imports(stmt_node, module_nodes[0], src, seen_pascal_units))- continue- if language == "elixir":- imports.extend(- _elixir_imports(stmt_node, module_nodes[0], src, seen_elixir_modules)- )- continue- if language == "fsharp":- imports.extend(_fsharp_imports(stmt_node, module_nodes[0], src, seen_raws))- continue-- raw = _node_text(stmt_node, src).strip()- # A Rust ``mod`` item's text excludes its outer attributes, so two- # ``#[cfg]``-gated ``mod imp;`` declarations targeting different- # ``#[path]`` files collide on the same raw text and the second- # file silently loses its import edge (and is then reported as- # dead code). Qualify the dedup key with the path override.- dedup_key = raw- if language == "rust" and stmt_node.type == "mod_item":- mod_path_attr = _rust_mod_path_attribute(stmt_node, src)- if mod_path_attr is not None:- dedup_key = f"{raw}|path={mod_path_attr}"- # Two ``import('./m')`` calls share their text but not the names- # each one binds.- if language in _TS_JS_LANGUAGES and _is_dynamic_esm_import(stmt_node):- dedup_key = f"{raw}|at={stmt_node.start_byte}"- kind = import_load_kind(stmt_node, language, src)- if dedup_key in seen_imports:- # A lazy copy must not hide a load-time twin: runtime wins.- for imp in seen_imports[dedup_key]:- imp.type_only, imp.deferred = combine_load_kinds(- (imp.type_only, imp.deferred), kind- )- continue- seen_imports[dedup_key] = []-- module_text = _node_text(module_nodes[0], src).strip().strip("\"'` ")- if not module_text:- continue-- found = _statement_imports(stmt_node, module_nodes[0], module_text, raw, language, src)- for imp in found:- imp.type_only, imp.deferred = kind- seen_imports[dedup_key] = found- imports.extend(found)-- if language == "python":- imports = expand_bare_relative_imports(imports)-- return imports-- # ------------------------------------------------------------------- # Call extraction- # -------------------------------------------------------------------- def _extract_calls(- self,- matches: list[dict],- config: LanguageConfig,- file_info: FileInfo,- src: str,- symbols: list[Symbol],- ) -> list[CallSite]:- """Extract function/method call sites from the AST."""- from .language_data import get_builtin_calls-- _call_builtins = get_builtin_calls(file_info.language)-- symbol_ranges = sorted(- [(s.start_line, s.end_line, s.id) for s in symbols],- key=lambda t: (t[0], -t[1]),- )-- # Each call as ``(site, call)``; see ``_fold_receiverless_twins``.- calls: list[tuple[tuple[int, int, str], CallSite]] = []-- for capture_dict in matches:- site_nodes = capture_dict.get("call.site", [])- target_nodes = capture_dict.get("call.target", [])- arg_nodes = capture_dict.get("call.arguments", [])- receiver_nodes = capture_dict.get("call.receiver", [])- receiver_call_nodes = capture_dict.get("call.receiver_call", [])- scope_nodes = capture_dict.get("call.scope", [])-- if not site_nodes or not target_nodes:- continue-- site_node = site_nodes[0]- target_node = target_nodes[0]- target_name = config.call_target_name_fn(- _node_text(target_node, src).strip(), target_node.type- )- if not target_name:- continue-- if file_info.language == "objectivec":- target_name = _objc_call_target(- site_node, target_node, target_name, bool(receiver_nodes), src- )- if target_name is None:- continue-- if target_name in _call_builtins:- continue-- # Elixir: a definition head (`add(a, b)` in `def add(a, b)`) and a- # module attribute (`@doc "..."`) are both ``call`` nodes, and no- # query predicate can see the parent that tells them apart. Left- # in, every function in the repo would call itself.- if file_info.language == "elixir" and _elixir_call_is_definitional(site_node, src):- continue-- line = site_node.start_point[0] + 1- receiver_name = None- if receiver_nodes:- receiver_name = _receiver_text(file_info.language, receiver_nodes[0], src)- if receiver_name is None:- continue- if receiver_name and file_info.language == "php":- receiver_name = _normalize_php_receiver(receiver_name)- # F#: a dotted static path (``Path.Combine(a, b)``) collapses into- # one identifier node in this grammar -- there is no dot node to- # capture a receiver from -- so the split happens on the text.- # After the builtin check above, so a name is filtered as written.- if file_info.language == "fsharp" and receiver_name is None and "." in target_name:- receiver_part, _, target_name = target_name.rpartition(".")- receiver_name = receiver_part.strip()- target_name = target_name.strip()- if not target_name:- continue- receiver_call = (- _call_receiver_from_node(receiver_call_nodes[0], src)- if receiver_call_nodes- else None- )- scope_name = _node_text(scope_nodes[0], src).strip() if scope_nodes else None-- arg_count = _count_arguments(arg_nodes[0]) if arg_nodes else None- caller_id = _find_enclosing_symbol(line, symbol_ranges)- arguments = _overload_arguments(file_info.language, arg_nodes)-- calls.append(- (- (site_node.start_byte, site_node.end_byte, target_name),- CallSite(- target_name=target_name,- receiver_name=receiver_name,- caller_symbol_id=caller_id,- line=line,- argument_count=arg_count,- receiver_call=receiver_call,- argument_types=_argument_literal_types(arguments, src),- argument_names=_argument_names(arguments, src),- scope_name=scope_name,- edge_type=(- "references"- if site_node.type in config.reference_call_node_types- else "calls"- ),- supplied_props=_jsx_supplied_props(site_node, src),- # Ceiling: schedulers matched by name, the task followed- # within its own function only (see ``python_tasks``).- spawned=(- file_info.language == "python" and is_fire_and_forget(site_node, src)- ),- ),- )- )- return _dedupe_calls(_fold_receiverless_twins(calls))-- def _extract_references(- self,- matches: list[dict],- file_info: FileInfo,- src: str,- symbols: list[Symbol],- imports: list[Import],- ) -> list[CallSite]:- """Extract sites that name a function without calling it.-- Six shapes carry a function by name and never call it where a parser- can see: a dispatch-table entry, a callback field, an argument to a- registration macro, a func value in argument position, a struct-field- initialiser, and a ``::`` callable reference. Each leaves the named- function with no inbound edge, which read as a ``safe_to_delete``- unused export and took out whole handler and interop layers (#1602).- TS/JS and Python capture any value position: an argument, a list or- object entry, a return, an assignment and a default export.-- ``@reference.receiver`` is optional; capturing it is what lets- ``_add_reference_edges`` restrict a bare name to free functions and- allow a qualified name to reach a method.-- Self-gating on the reference captures, so a language whose query- defines none produces nothing and pays two dict lookups. Two guards- keep the broad syntactic positions from claiming ordinary code:-- * A macro argument requires a SCREAMING_CASE callee and must be that- macro's only argument. Uppercase alone is not enough: assertion- macros are spelled the same way and take values, so ``EXPECT_EQ(- capacity, 100)`` bound a local to whatever free function shared its- name. Registering something registers one thing, which separates- ``BENCHMARK(BM_Foo)`` from ``TEST(SuiteName, TestName)``.- * A table entry must sit outside any function body. A dispatch table is- a file-, namespace- or class-scope aggregate; the same braces inside- a function are a constructor member-init or a local aggregate, where- ``{data, size}`` names parameters. Measured on fmt, admitting those- turned ``data``, ``size``, ``begin``, ``end``, ``capacity`` and- ``buffer`` into edges, every one of them wrong.-- Whether the name denotes a function at all is settled later, at- resolution, where the symbol kind is known.- """- from .language_data import get_builtin_calls-- builtins = get_builtin_calls(file_info.language)-- symbol_ranges = sorted(- [(s.start_line, s.end_line, s.id) for s in symbols],- key=lambda t: (t[0], -t[1]),- )- callable_ids = {s.id for s in symbols if s.kind in ("function", "method")}- bindable = _bare_reference_names(file_info.language, symbols, imports)-- references: list[CallSite] = []- seen: set[tuple[int, str, str | None]] = set()-- for capture_dict in matches:- plain_nodes = capture_dict.get("reference.name", [])- table_nodes = capture_dict.get("reference.table", [])- if not plain_nodes and not table_nodes:- continue-- receiver_nodes = capture_dict.get("reference.receiver", [])- receiver = (- _node_text(receiver_nodes[0], src).strip() if receiver_nodes else None- ) or None-- via_nodes = capture_dict.get("reference.via", [])- if via_nodes:- via = _node_text(via_nodes[0], src).strip()- if not via or not via.isupper():- continue- arg_list = plain_nodes[0].parent if plain_nodes else None- if arg_list is None or len(arg_list.named_children) != 1:- continue-- candidates = [(node, False) for node in plain_nodes]- candidates += [(node, True) for node in table_nodes]- for name_node, is_table in candidates:- name = _node_text(name_node, src).strip()- if not name or name in builtins:- continue- if bindable is not None and receiver is None and name not in bindable:- continue- line = name_node.start_point[0] + 1- enclosing = _find_enclosing_symbol(line, symbol_ranges)- if is_table and enclosing in callable_ids:- continue- if (line, name, receiver) in seen:- continue- seen.add((line, name, receiver))- references.append(- CallSite(- target_name=name,- receiver_name=receiver,- caller_symbol_id=enclosing,- line=line,- argument_count=None,- )- )-- return references-- # ------------------------------------------------------------------- # Export derivation- # -------------------------------------------------------------------- def _derive_exports(- self,- symbols: list[Symbol],- config: LanguageConfig,- ) -> list[str]:- """Derive the list of exported names from parsed symbols.-- Note on TS/JS: export visibility is resolved upstream during symbol- extraction by ``refine_ts_visibility()``, which demotes non-exported- declarations to private. This expression relies on that classification to- filter top-level public symbols accurately for TS/JS.- """- return [s.name for s in symbols if s.visibility == "public" and s.parent_name is None]-- # ------------------------------------------------------------------- # Type reference extraction (non-import positions)- # -------------------------------------------------------------------- def _extract_type_refs(- self,- matches: list[dict],- src: str,- lang: str = "",- ) -> list[TypeReference]:- """Collect ``@param.type`` captures into TypeReference records.-- C# emits these from constructor / method / delegate / primary-ctor- parameter types; Go emits them from parameter, struct-field, return,- and composite-literal type positions (see ``go.scm``). The graph- builder resolves each reference to a defining file via the- language-specific resolver index and emits a file-level edge.-- The head-identifier extractor is language-specific (Go unwraps- ``*T`` / ``[]T`` / ``map[K]V`` / ``pkg.T``); see- ``TYPE_HEAD_EXTRACTORS``. Capture origin is inferred from the- enclosing node: ``constructor_declaration`` → ``ctor_param``,- ``method_declaration`` → ``method_param`` (C#);- ``field_declaration`` → ``field_type``, ``composite_literal`` →- ``composite_literal`` (Go).- """- head_of = TYPE_HEAD_EXTRACTORS.get(lang, _head_type_identifier)-- refs: list[TypeReference] = []- seen: set[tuple[str, int]] = set()-- for capture_dict in matches:- type_nodes = capture_dict.get("param.type", [])- if not type_nodes:- continue- for type_node in type_nodes:- head = head_of(type_node, src)- if not head:- continue- # ``struct Wrapper<Item> { value: Item }`` binds Item as a type- # parameter, and the capture cannot tell that from a reference- # to a real ``struct Item``. The head extractor drops a- # single-letter ``T`` but not a named one, so the shadow has to- # be read off the enclosing item's ``type_parameters``.- if lang == "rust" and _rust_shadowed_by_type_param(type_node, head, src):- continue- line = type_node.start_point[0] + 1- key = (head, line)- if key in seen:- continue- seen.add(key)- origin = _classify_param_origin(type_node)- refs.append(TypeReference(type_name=head, line=line, origin=origin))-- return refs---# ----------------------------------------------------------------------------# Convenience function-# -----------------------------------------------------------------------------_DEFAULT_PARSER: ASTParser | None = None---def _python_module_facts(- src: str, symbols: list[Symbol]-) -> tuple[frozenset[str], frozenset[str]]:- """``(local_refs, registrars)`` for a Python module, from one ``ast`` parse.-- See ``python_local_refs`` and ``python_registrars``.- """- top_level_names = {s.name for s in symbols if s.name and not s.parent_name}- tree = parse_python_module(src) if top_level_names else None- if tree is None:- return frozenset(), frozenset()- nested_classes = {- (s.parent_name, s.name) for s in symbols if s.kind == "class" and s.parent_name- }- return (- local_refs_from_tree(tree, top_level_names, nested_classes),- registering_functions(tree),- )---def parse_file(file_info: FileInfo, source: bytes) -> ParsedFile:- """Module-level convenience: parse a file using the default ASTParser."""- global _DEFAULT_PARSER- if _DEFAULT_PARSER is None:- _DEFAULT_PARSER = ASTParser()- return _DEFAULT_PARSER.parse_file(file_info, source)diff --git a/packages/core/src/repowise/core/ingestion/parser/__init__.py b/packages/core/src/repowise/core/ingestion/parser/__init__.pynew file mode 100644index 000000000..aa5023817--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/parser/__init__.py@@ -0,0 +1,324 @@+"""Unified AST parser — one class for all languages.++Architecture+============+Per-language differences live in two places:+ 1. ``packages/core/queries/<lang>.scm`` — tree-sitter S-expression queries+ that capture symbols and imports using consistent capture-name conventions.+ 2. ``LANGUAGE_CONFIGS`` (``language_configs.py``) — a ``LanguageConfig`` per+ language that maps node types to symbol kinds, defines visibility rules,+ and names its import extractor (``extractors/imports/<lang>.py``).++The package splits one parse into its passes: ``grammars`` (grammar and query+loading, the tree-sitter parse), ``symbols`` and ``symbol_rules``, ``imports``,+``calls`` and ``references``. Every name the old single module defined is+re-exported here.++Capture-name conventions (shared across ALL .scm files):+ @symbol.def — the full definition node (line numbers, kind lookup)+ @symbol.name — name identifier+ @symbol.params — parameter list (optional)+ @symbol.modifiers — decorators / visibility modifiers (optional)+ @symbol.receiver — Go method receiver (optional, used for parent detection)+ @import.statement — full import node+ @import.module — module path being imported+"""++# ruff: noqa: F401++from __future__ import annotations++from tree_sitter import Language++from ..extractors import extract_heritage, extract_module_docstring+from ..extractors.imports.dart import _dart_import+from ..extractors.imports.elixir import _elixir_imports+from ..extractors.imports.fsharp import _fsharp_imports+from ..extractors.imports.generic import _generic_import+from ..extractors.imports.pascal import _pascal_imports+from ..extractors.imports.php import _php_imports+from ..extractors.imports.rust import (+ _rust_inline_mod_depth,+ _rust_mod_path_attribute,+ _rust_use_imports,+ _strip_inline_super,+)+from ..extractors.imports.scala import _scala_imports+from ..extractors.imports.ts_js import _cjs_imports, _is_dynamic_esm_import+from ..extractors.visibility import ts_export_aliases+from ..language_configs import LANGUAGE_CONFIGS, LanguageConfig+from ..models import FileInfo, Import, ParsedFile, Symbol, compute_content_hash+from ..parser_helpers import _run_query+from ..python_local_refs import local_refs_from_tree, parse_python_module+from ..python_registrars import registering_functions+from ..sfc_source import component_call_sites, prepare_source+from ..special_handlers import SPECIAL_HANDLER_LANGUAGES, parse_special+from .calls import (+ _DOTTED_PATH,+ _PATH_RECEIVERS,+ _call_receiver_from_node,+ _dart_argument_count,+ _dart_call_receiver,+ _dedupe_calls,+ _extract_calls,+ _fold_receiverless_twins,+ _is_bare,+ _jsx_supplied_props,+ _normalize_php_receiver,+ _objc_call_target,+ _receiver_text,+)+from .grammars import (+ _LANGUAGE_REGISTRY,+ _MISSING_GRAMMAR_REPORTED,+ _PASSTHROUGH_LANGUAGES,+ QUERIES_DIR,+ _build_language_registry,+ _compile_query,+ _get_language,+ _load_compiled_query,+ _parse_tree,+ failed_query_languages,+ grammar_tag_for,+ log,+ missing_grammar_languages,+)+from .imports import _extract_imports+from .references import (+ _NAMED_BINDING_LANGUAGES,+ _REFERENCE_LANGUAGES,+ _bare_reference_names,+ _extract_references,+ _extract_type_refs,+)+from .symbol_rules import (+ _DART_FUNCTION_NODE_TYPES,+ _FSHARP_BINDING_NODE_TYPES,+ _TS_JS_LANGUAGES,+ _TS_OVERLOAD_SIGNATURES,+ _attribute_decorators,+ _dart_function_end_line,+ _dart_mixin_parent,+ _declared_modifiers,+ _drop_swift_extensions_of_local_types,+ _is_declaration,+ _is_fsharp_binding,+ _is_split_annotation_chain,+ _keep_ts_overloads_beside_a_body,+ _kotlin_detached_annotation,+ _language_symbol_name,+ _leading_child_attributes,+ _module_binding_kind,+ _node_text,+ _refine_symbol_kind,+ _refine_visibility,+ _rust_outer_attributes,+ _symbol_end_line,+)+from .symbols import (+ _MODULE_ANCHORED_NODE_TYPES,+ _extract_symbols,+ _finalize_symbols,+ _java_enum_constant,+ _match_identity,+ _nesting_parent,+ _parent_type_name,+)++# A file with more symbols than this is almost certainly generated; warn, never truncate.+_SYMBOL_COUNT_WARN_THRESHOLD = 500+++class ASTParser:+ """Unified AST parser — works for all languages via .scm query files.++ Usage::++ parser = ASTParser()+ parsed = parser.parse_file(file_info, source_bytes)++ Adding a new language:+ 1. Write ``packages/core/queries/<lang>.scm``+ 2. Add one entry to ``LANGUAGE_CONFIGS``+ That's it. No Python class, no new module.+ """++ def __init__(self) -> None:+ pass++ def parse_imports(self, file_info: FileInfo, source: bytes) -> list[Import]:+ """Only *source*'s imports: :meth:`parse_file`'s grammar, query and import pass.++ Symbols, calls and references are not extracted, which is most of a+ parse. A language without a tree-sitter path takes the full parse.+ """+ lang = file_info.language+ config = LANGUAGE_CONFIGS.get(lang)+ grammar_tag = grammar_tag_for(lang, file_info.path)+ language = _get_language(grammar_tag)+ if (+ lang in SPECIAL_HANDLER_LANGUAGES+ or config is None+ or language is None+ or _is_fsharp_signature(file_info)+ ):+ return self.parse_file(file_info, source).imports+ source = prepare_source(lang, source, path=file_info.path)+ root, _, grammar_tag, language = _parse_tree(lang, language, grammar_tag, source)+ query = _load_compiled_query(lang, grammar_tag)+ matches = _run_query(query, root) if query is not None else []+ src = source.decode("utf-8", errors="replace")+ return _extract_imports(matches, config, file_info, src)++ def parse_file(self, file_info: FileInfo, source: bytes) -> ParsedFile:+ """Parse *source* bytes and return a fully populated ParsedFile."""+ lang = file_info.language+ content_hash = compute_content_hash(source)++ # Non-tree-sitter formats (OpenAPI, Dockerfile, Makefile, SQL) carry no+ # LanguageConfig, so they are routed before the no-grammar fallback.+ if lang in SPECIAL_HANDLER_LANGUAGES:+ parsed = parse_special(file_info, source, lang)+ parsed.content_hash = content_hash+ return parsed++ config = LANGUAGE_CONFIGS.get(lang)+ grammar_tag = grammar_tag_for(lang, file_info.path)+ language = _get_language(grammar_tag)+ if config is None or language is None or _is_fsharp_signature(file_info):+ return _parse_without_grammar(file_info, source, config, language, content_hash)++ # An SFC (.svelte, .vue) is blanked to its script at byte-identical+ # offsets; ``content_hash`` above keeps hashing the original bytes.+ original_source = source+ source = prepare_source(lang, source, path=file_info.path)++ root, parse_errors, grammar_tag, language = _parse_tree(lang, language, grammar_tag, source)+ src = source.decode("utf-8", errors="replace")++ # Execute the compiled query ONCE per file; every pass below reads+ # the same capture dicts.+ query = _load_compiled_query(lang, grammar_tag)+ matches = _run_query(query, root) if query is not None else []++ symbols = _finalize_symbols(+ _extract_symbols(matches, config, file_info, src), root, src, file_info+ )+ imports = _extract_imports(matches, config, file_info, src)+ calls = _extract_calls(matches, config, file_info, src, symbols)+ # An SFC's markup tags (``<Foo />``) are component calls the TS buffer lost.+ calls.extend(component_call_sites(lang, original_source, symbols))+ references = (+ _extract_references(matches, file_info, src, symbols, imports)+ if lang in _REFERENCE_LANGUAGES+ else []+ )+ # Python names used in their own module outside a call or import,+ # stamped so dead-code can rescue them.+ local_refs, registrars = (+ _python_module_facts(src, symbols) if lang == "python" else (frozenset(), frozenset())+ )+ _warn_symbol_bloat(file_info, symbols)++ return ParsedFile(+ file_info=file_info,+ symbols=symbols,+ imports=imports,+ exports=_derive_exports(symbols),+ export_aliases=ts_export_aliases(src) if lang in _TS_JS_LANGUAGES else {},+ calls=calls,+ heritage=extract_heritage(matches, config, file_info, src),+ docstring=extract_module_docstring(root, src, lang),+ parse_errors=parse_errors,+ content_hash=content_hash,+ type_refs=_extract_type_refs(matches, src, lang),+ local_refs=local_refs,+ registrars=registrars,+ references=references,+ )+++def _is_fsharp_signature(file_info: FileInfo) -> bool:+ """An F# ``.fsi`` signature file needs a grammar the spec does not load.++ Read with the implementation grammar, its ``val`` signatures land in+ ERROR recovery; the regex tier still gives it its ``open`` imports.+ """+ return file_info.language == "fsharp" and file_info.path.endswith(".fsi")+++def _parse_without_grammar(+ file_info: FileInfo,+ source: bytes,+ config: LanguageConfig | None,+ language: Language | None,+ content_hash: str,+) -> ParsedFile:+ """Regex-tier imports and no symbols, for a file with no usable grammar."""+ lang = file_info.language+ if config is not None and language is None and lang not in _MISSING_GRAMMAR_REPORTED:+ # Once per language, not once per file: the fact is about the environment.+ _MISSING_GRAMMAR_REPORTED.add(lang)+ log.debug("tree-sitter grammar unavailable", language=lang)+ from ..lightweight_imports import extract_lightweight_imports++ return ParsedFile(+ file_info=file_info,+ symbols=[],+ imports=extract_lightweight_imports(file_info, source),+ exports=[],+ docstring=None,+ parse_errors=[],+ content_hash=content_hash,+ )+++def _warn_symbol_bloat(file_info: FileInfo, symbols: list[Symbol]) -> None:+ if len(symbols) > _SYMBOL_COUNT_WARN_THRESHOLD:+ log.warning(+ "parser.symbol_bloat",+ path=file_info.path,+ language=file_info.language,+ symbol_count=len(symbols),+ threshold=_SYMBOL_COUNT_WARN_THRESHOLD,+ )+++def _derive_exports(symbols: list[Symbol]) -> list[str]:+ """Top-level public symbol names.++ TS/JS visibility was already refined to "exported" during symbol extraction.+ """+ return [s.name for s in symbols if s.visibility == "public" and s.parent_name is None]+++_DEFAULT_PARSER: ASTParser | None = None+++def _python_module_facts(+ src: str, symbols: list[Symbol]+) -> tuple[frozenset[str], frozenset[str]]:+ """``(local_refs, registrars)`` for a Python module, from one ``ast`` parse.++ See ``python_local_refs`` and ``python_registrars``.+ """+ top_level_names = {s.name for s in symbols if s.name and not s.parent_name}+ tree = parse_python_module(src) if top_level_names else None+ if tree is None:+ return frozenset(), frozenset()+ nested_classes = {+ (s.parent_name, s.name) for s in symbols if s.kind == "class" and s.parent_name+ }+ return (+ local_refs_from_tree(tree, top_level_names, nested_classes),+ registering_functions(tree),+ )+++def parse_file(file_info: FileInfo, source: bytes) -> ParsedFile:+ """Module-level convenience: parse a file using the default ASTParser."""+ global _DEFAULT_PARSER+ if _DEFAULT_PARSER is None:+ _DEFAULT_PARSER = ASTParser()+ return _DEFAULT_PARSER.parse_file(file_info, source)+diff --git a/packages/core/src/repowise/core/ingestion/parser/calls.py b/packages/core/src/repowise/core/ingestion/parser/calls.pynew file mode 100644index 000000000..d52086c9d--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/parser/calls.py@@ -0,0 +1,372 @@+"""Call-site extraction: what each call names, on which receiver, from which symbol."""++from __future__ import annotations++import re+from dataclasses import replace++from tree_sitter import Node++from ..extractors import node_text+from ..language_configs import LanguageConfig+from ..models import CallReceiver, CallSite, FileInfo, Symbol+from ..parser_helpers import (+ _argument_literal_types,+ _argument_names,+ _count_arguments,+ _elixir_call_is_definitional,+ _find_enclosing_symbol,+ _objc_call_is_block_variable,+ _objc_message_selector,+ _overload_arguments,+)+from ..python_tasks import is_fire_and_forget++_node_text = node_text++def _call_receiver_from_node(node: Node, src: str) -> CallReceiver | None:+ """Describe an inner call captured as another call's receiver.++ The queries identify the complete inner AST node. This helper reads only+ named tree-sitter fields, so nested arguments and formatting cannot change+ which call is carried into resolution.+ """++ if node.type == "identifier":+ return _dart_call_receiver(node, src)+ arguments = node.child_by_field_name("arguments")+ argument_count = _count_arguments(arguments) if arguments is not None else None++ target = node.child_by_field_name("name")+ receiver = node.child_by_field_name("object")+ if target is None:+ function = node.child_by_field_name("function")+ if function is None:+ return None+ if function.type == "attribute":+ return _python_attribute_receiver(function, src, argument_count)+ target, receiver = _function_target(function)++ if target is None:+ return None+ target_name = _node_text(target, src).strip()+ if not target_name:+ return None++ receiver_name = None+ if receiver is not None and receiver.type in ("identifier", "this"):+ receiver_name = _node_text(receiver, src).strip() or None+ return CallReceiver(target_name, receiver_name, argument_count)+++def _function_target(function: Node) -> tuple[Node | None, Node | None]:+ """``(target, receiver)`` named by an inner call's ``function`` field."""+ if function.type in ("identifier", "property_identifier", "field_identifier"):+ return function, None+ if function.type == "generic_name":+ target = function.child_by_field_name("name") or next(+ (child for child in function.named_children if child.type == "identifier"), None+ )+ return target, None+ target = (+ function.child_by_field_name("name")+ or function.child_by_field_name("property")+ or function.child_by_field_name("field")+ )+ receiver = (+ function.child_by_field_name("expression")+ or function.child_by_field_name("object")+ or function.child_by_field_name("argument")+ or function.child_by_field_name("operand")+ )+ return target, receiver+++def _python_attribute_receiver(+ function: Node, src: str, argument_count: int | None+) -> CallReceiver | None:+ """Python ``obj.m()``: the receiver is kept as a plain dotted path, and a+ call on anything else is not carried at all.+ """+ target = function.child_by_field_name("attribute")+ obj = function.child_by_field_name("object")+ path = (+ _receiver_text("python", obj, src)+ if obj is not None and obj.type in ("identifier", "attribute")+ else None+ )+ if target is None or path is None:+ return None+ return CallReceiver(_node_text(target, src).strip(), path, argument_count)+++def _dart_call_receiver(name: Node, src: str) -> CallReceiver | None:+ """The Dart call whose *name* the query captured: ``a.b(..)``, ``B(..)``.++ A Dart call is a flat selector run with no call node, so the name's+ siblings are read: the selector after it holds the arguments, and the+ node before a ``.b`` selector is its receiver.+ """+ if name.parent.type == "unconditional_assignable_selector":+ head = name.parent.parent+ prev = head.prev_named_sibling+ if prev is not None and any(c.type == "argument_part" for c in prev.named_children):+ receiver = None+ else:+ receiver = _node_text(prev, src).strip() if prev is not None else None+ else:+ head, receiver = name, None+ return CallReceiver(+ _node_text(name, src).strip(), receiver, _dart_argument_count(head.next_named_sibling)+ )+++def _dart_argument_count(selector: Node | None) -> int | None:+ """The arguments a Dart ``(selector (argument_part (arguments)))`` passes."""+ parts = selector.named_children if selector is not None else ()+ found = [arg for part in parts for arg in part.named_children if arg.type == "arguments"]+ return _count_arguments(found[0]) if found else None++++# The receiver captures that are themselves a member access, by language (the+# queries widen only these two), and what a plain dotted path of names looks+# like once whitespace is gone.+_PATH_RECEIVERS = frozenset({("typescript", "member_expression"), ("python", "attribute")})+_DOTTED_PATH = re.compile(r"[A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*)+")+++def _receiver_text(language: str, node: Node, src: str) -> str | None:+ """The receiver's text, or None to drop the site.++ A dotted receiver (``this.a.b.m()``) is kept only as a plain path of+ names: a call, subscript or optional hop inside it types nothing, and+ dropping the site keeps it off the bare-call tiers.+ """+ sibling = node.next_named_sibling+ if language == "java" and node.type != "super" and getattr(sibling, "type", None) == "super":+ # ``S.super.m()``: ``S`` qualifies the ``super`` its own pattern captures.+ return None+ text = _node_text(node, src).strip()+ if (language, node.type) not in _PATH_RECEIVERS:+ return text+ path = "".join(text.split())+ return path if _DOTTED_PATH.fullmatch(path) else None+++def _normalize_php_receiver(text: str) -> str:+ """Spell PHP's receiver the way the resolver's strategies expect.++ `self::` and `static::` are the same dispatch as `$this`, and the self/this+ strategy tests `in ("self", "this")` — so without this the whole implicit-+ receiver population misses. `parent::` needs the heritage walk and is left+ to miss rather than guessed at.+ """+ name = text.lstrip("$")+ return "this" if name in ("self", "static") else name+++def _objc_call_target(+ site_node: Node, target_node: Node, target_name: str, has_receiver: bool, src: str+) -> str | None:+ """Objective-C call target name, or None to drop the match."""+ # A message send binds one `method:` child per keyword, so+ # `[view setTitle:t forState:s]` matches the query twice. Join the whole+ # selector on the first match and drop the rest.+ if site_node.type == "message_expression":+ return _objc_message_selector(site_node, target_node, src)+ # A block variable invoked with C call syntax (`completionBlock(hit)`) is+ # not a call to a same-named function or property.+ if not has_receiver and _objc_call_is_block_variable(site_node, target_name, src):+ return None+ return target_name+++def _jsx_supplied_props(site_node: Node, src: str) -> frozenset[str] | None:+ """Props written on a JSX element, or None when not JSX or a spread hides them."""+ if site_node.type not in ("jsx_self_closing_element", "jsx_opening_element"):+ return None+ props_set: set[str] = set()+ for child in site_node.children:+ if child.type == "jsx_attribute":+ name = next(+ (sub for sub in child.children if sub.type in ("property_identifier", "identifier")),+ None,+ )+ if name is not None:+ props_set.add(_node_text(name, src))+ elif child.type == "jsx_expression" and any(+ sub.type == "spread_element" for sub in child.children+ ):+ return None+ return frozenset(props_set)+++def _fold_receiverless_twins(calls: list[tuple[tuple[int, int, str], CallSite]]) -> list[CallSite]:+ """Fold the receiver-less copy of a member call into the copy with a receiver.++ A grammar's bare-call pattern that leaves the object unconstrained (Java's+ ``method_invocation``, Ruby's ``call``) matches ``obj.m()`` too. Resolved on+ its own, that copy binds ``m`` by bare name beside whatever ``obj``'s type+ answers, so it survives only as the member call's ``bare_name_fallback``.+ Keyed on the call node, not the line, so a bare ``m(obj.m())`` keeps its own+ site.+ """+ member_sites = {site for site, call in calls if not _is_bare(call)}+ bare_sites = {site for site, call in calls if site in member_sites and _is_bare(call)}+ folded: list[CallSite] = []+ for site, call in calls:+ if _is_bare(call):+ if site not in member_sites:+ folded.append(call)+ elif call.receiver_name and site in bare_sites:+ folded.append(replace(call, bare_name_fallback=True))+ else:+ folded.append(call)+ return folded+++def _is_bare(call: CallSite) -> bool:+ return not (call.receiver_name or call.receiver_call)+++def _dedupe_calls(calls: list[CallSite]) -> list[CallSite]:+ """One call per ``(line, target, receiver)``, keeping the richest record.++ Two scoped-call patterns can match the same two-part call (one keeps the+ qualifier), so the richer record wins whichever order they arrive in.+ """+ deduplicated: dict[tuple[int, str, str | None], CallSite] = {}+ for call in calls:+ key = (call.line, call.target_name, call.receiver_name)+ existing = deduplicated.get(key)+ if (+ existing is None+ or (existing.receiver_call is None and call.receiver_call is not None)+ or (existing.scope_name is None and call.scope_name is not None)+ ):+ deduplicated[key] = call+ return list(deduplicated.values())+++def _extract_calls(+ matches: list[dict],+ config: LanguageConfig,+ file_info: FileInfo,+ src: str,+ symbols: list[Symbol],+) -> list[CallSite]:+ """Extract function/method call sites from the AST."""+ from ..language_data import get_builtin_calls++ call_builtins = get_builtin_calls(file_info.language)++ symbol_ranges = sorted(+ [(s.start_line, s.end_line, s.id) for s in symbols],+ key=lambda t: (t[0], -t[1]),+ )++ # Each call as ``(site, call)``; see ``_fold_receiverless_twins``.+ calls: list[tuple[tuple[int, int, str], CallSite]] = []+ for capture_dict in matches:+ found = _call_from_match(+ capture_dict, config, file_info.language, src, call_builtins, symbol_ranges+ )+ if found is not None:+ calls.append(found)+ return _dedupe_calls(_fold_receiverless_twins(calls))+++def _call_from_match(+ capture_dict: dict,+ config: LanguageConfig,+ language: str,+ src: str,+ call_builtins: frozenset[str],+ symbol_ranges: list[tuple[int, int, str]],+) -> tuple[tuple[int, int, str], CallSite] | None:+ """One call match as ``(site key, CallSite)``, or None to drop it."""+ site_nodes = capture_dict.get("call.site", [])+ target_nodes = capture_dict.get("call.target", [])+ if not site_nodes or not target_nodes:+ return None++ site_node = site_nodes[0]+ names = _call_names(+ capture_dict, site_node, target_nodes[0], config, language, src, call_builtins+ )+ if names is None:+ return None+ target_name, receiver_name = names+ # F#: a dotted static path (``Path.Combine(a, b)``) is one identifier node,+ # so it splits on the text, after the builtin check above.+ if language == "fsharp" and receiver_name is None and "." in target_name:+ receiver_part, _, target_name = target_name.rpartition(".")+ receiver_name = receiver_part.strip()+ target_name = target_name.strip()+ if not target_name:+ return None++ arg_nodes = capture_dict.get("call.arguments", [])+ receiver_call_nodes = capture_dict.get("call.receiver_call", [])+ scope_nodes = capture_dict.get("call.scope", [])+ line = site_node.start_point[0] + 1+ arguments = _overload_arguments(language, arg_nodes)+ call = CallSite(+ target_name=target_name,+ receiver_name=receiver_name,+ caller_symbol_id=_find_enclosing_symbol(line, symbol_ranges),+ line=line,+ argument_count=_count_arguments(arg_nodes[0]) if arg_nodes else None,+ receiver_call=(+ _call_receiver_from_node(receiver_call_nodes[0], src) if receiver_call_nodes else None+ ),+ argument_types=_argument_literal_types(arguments, src),+ argument_names=_argument_names(arguments, src),+ scope_name=_node_text(scope_nodes[0], src).strip() if scope_nodes else None,+ edge_type=(+ "references" if site_node.type in config.reference_call_node_types else "calls"+ ),+ supplied_props=_jsx_supplied_props(site_node, src),+ # Ceiling: schedulers matched by name, the task followed within its+ # own function only (see ``python_tasks``).+ spawned=language == "python" and is_fire_and_forget(site_node, src),+ )+ return (site_node.start_byte, site_node.end_byte, target_name), call+++def _call_names(+ capture_dict: dict,+ site_node: Node,+ target_node: Node,+ config: LanguageConfig,+ language: str,+ src: str,+ call_builtins: frozenset[str],+) -> tuple[str, str | None] | None:+ """``(target_name, receiver_name)`` as written, or None to drop the site."""+ target_name = config.call_target_name_fn(_node_text(target_node, src).strip(), target_node.type)+ if not target_name:+ return None+ receiver_nodes = capture_dict.get("call.receiver", [])+ if language == "objectivec":+ target_name = _objc_call_target(+ site_node, target_node, target_name, bool(receiver_nodes), src+ )+ if target_name is None:+ return None+ if target_name in call_builtins:+ return None+ # Elixir: a definition head (`add(a, b)` in `def add(a, b)`) and a module+ # attribute (`@doc "..."`) are both ``call`` nodes; left in, every function+ # would call itself.+ if language == "elixir" and _elixir_call_is_definitional(site_node, src):+ return None+ receiver_name = None+ if receiver_nodes:+ receiver_name = _receiver_text(language, receiver_nodes[0], src)+ if receiver_name is None:+ return None+ if receiver_name and language == "php":+ receiver_name = _normalize_php_receiver(receiver_name)+ return target_name, receiver_namediff --git a/packages/core/src/repowise/core/ingestion/parser/grammars.py b/packages/core/src/repowise/core/ingestion/parser/grammars.pynew file mode 100644index 000000000..a72a9a9a6--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/parser/grammars.py@@ -0,0 +1,269 @@+"""Grammar loading, query compilation and the tree-sitter parse itself."""++from __future__ import annotations++from collections.abc import Iterable+from functools import cache+from pathlib import Path++import structlog+from tree_sitter import Language, Node, Parser++from ..language_configs import LANGUAGE_CONFIGS+from ..languages.registry import REGISTRY as _LANG_REGISTRY+from ..parser_helpers import _collect_error_nodes++# One logger for the whole package, under the package's public name.+log = structlog.get_logger("repowise.core.ingestion.parser")++QUERIES_DIR = Path(__file__).parent.parent / "queries"+++@cache+def _compile_query(lang: str, grammar_tag: str | None = None) -> tuple[object | None, str | None]:+ """Compile and return (Query, None) or (None, error_str).++ Cached process-wide so preflight and parsing workers never recompile the same query.+ """+ grammar = grammar_tag or lang+ language = _get_language(grammar)+ if language is None:+ return None, None++ # The spec names the query file, so a language can reuse another's+ # queries wholesale (svelte -> typescript.scm). Every other spec declares+ # ``<tag>.scm``, which is what the default preserves.+ spec = _LANG_REGISTRY.get(lang)+ scm_name = (spec.scm_file if spec and spec.scm_file else None) or f"{lang}.scm"+ scm_path = QUERIES_DIR / scm_name+ if not scm_path.exists():+ log.debug("No .scm query file found", language=lang, path=str(scm_path))+ return None, None++ scm_text = scm_path.read_text(encoding="utf-8")+ # Grammar-variant-specific additions (e.g. JSX node captures that are+ # only valid against the ``tsx`` grammar but not the plain ``typescript``+ # one). Appended to the base SCM only when the variant scm file exists.+ if grammar_tag and grammar_tag != lang:+ extra_scm = QUERIES_DIR / f"{grammar_tag}.scm"+ if extra_scm.exists():+ scm_text = scm_text + "\n" + extra_scm.read_text(encoding="utf-8")+ try:+ from tree_sitter import Query # type: ignore[attr-defined]++ return Query(language, scm_text), None+ except Exception as exc:+ return None, str(exc)+++@cache+def _load_compiled_query(lang: str, grammar_tag: str | None = None) -> object | None:+ """Process-wide cache of compiled tree-sitter Query objects.++ Compiling `.scm` queries is non-trivial; in process-pool parsing each worker+ would otherwise recompile per file. ``grammar_tag`` may differ from+ ``lang`` when a language reuses another's grammar at a different+ variant — e.g. ``.tsx`` files reuse ``typescript.scm`` but must bind+ to the JSX-aware ``tsx`` grammar so React components don't drown in+ ERROR nodes.+ """+ query, err = _compile_query(lang, grammar_tag)+ if err is not None:+ log.warning("Failed to compile query", language=lang, error=err)+ return query++++# Languages that intentionally have no AST parser. Derived from the+# centralised LanguageRegistry — only non-code passthrough languages are+# included (not the extra git-blame-only languages).++# Excludes "openapi" (handled by special_handlers) and "unknown".+_PASSTHROUGH_LANGUAGES: frozenset[str] = _LANG_REGISTRY.unparseable_data_languages()+++# ---------------------------------------------------------------------------+# Language registry — maps language tag → tree-sitter Language object+# ---------------------------------------------------------------------------+++def _build_language_registry() -> dict[str, Language]:+ """Lazily load installed tree-sitter language packages.++ Driven by ``LanguageSpec.grammar_package`` / ``grammar_loader`` /+ ``shares_grammar_with`` from the centralised registry.+ """+ registry: dict[str, Language] = {}++ for spec in _LANG_REGISTRY.all_specs():+ # Languages that share another's grammar (e.g. C → cpp)+ if spec.shares_grammar_with:+ shared = registry.get(spec.shares_grammar_with)+ if shared:+ registry[spec.tag] = shared+ continue++ if not spec.grammar_package:+ continue++ try:+ mod = __import__(spec.grammar_package)+ loader_fn = getattr(mod, spec.grammar_loader)+ loaded = loader_fn(*spec.grammar_loader_args)+ # Standalone grammar wheels return a PyCapsule; shared grammar+ # packs may return the fully constructed Language directly.+ lang_obj = loaded if isinstance(loaded, Language) else Language(loaded)+ registry[spec.tag] = lang_obj+ except Exception as exc:+ log.debug(+ "tree-sitter language unavailable",+ language=spec.tag,+ reason=str(exc),+ )++ # TypeScript's tsx variant — special case: same package, different loader+ if "typescript" in registry and "tsx" not in registry:+ try:+ import tree_sitter_typescript as _ts_mod++ registry["tsx"] = Language(_ts_mod.language_tsx())+ except Exception as exc:+ log.debug("tree-sitter language unavailable", language="tsx", reason=str(exc))++ return registry+++_LANGUAGE_REGISTRY: dict[str, Language] = {}++# Languages already reported as having a config but no installed grammar:+# one line per language per process, not one per file.+_MISSING_GRAMMAR_REPORTED: set[str] = set()+++def missing_grammar_languages(language_tags: Iterable[str]) -> list[str]:+ """Of *language_tags*, those that parse via tree-sitter but have no grammar.++ Answers the question once, in the parent, before up to eight spawned+ workers each rediscover it and log their own copy of the answer at a level+ the CLI discards anyway.++ Deliberately uses ``find_spec`` rather than importing: the whole point is+ to stay cheap enough to run on every index. Building the real registry here+ would import every tree-sitter package into the parent process, which is+ memory the parse pool is about to need for something else.++ A tag with no :data:`LANGUAGE_CONFIGS` entry is not a gap — nothing claims+ to parse it — so it is skipped rather than reported.++ Ceiling: "importable" is not "loadable". A grammar whose compiled ABI does+ not match the installed ``tree_sitter`` imports fine and then raises inside+ ``Language()``, which this cannot see, so that case reports nothing here+ and stays a per-worker debug line. Reporting it properly means loading the+ grammars, which is the cost this function exists to avoid.+ """+ import importlib.util++ specs = {spec.tag: spec for spec in _LANG_REGISTRY.all_specs()}+ missing: list[str] = []+ for tag in language_tags:+ if tag not in LANGUAGE_CONFIGS:+ continue+ spec = specs.get(tag)+ if spec is None:+ continue+ package = spec.grammar_package+ if not package and spec.shares_grammar_with:+ shared = specs.get(spec.shares_grammar_with)+ package = shared.grammar_package if shared else None+ if not package:+ continue+ try:+ if importlib.util.find_spec(package) is None:+ missing.append(tag)+ except (ImportError, ValueError):+ missing.append(tag)+ return sorted(missing)+++def failed_query_languages(language_tags: Iterable[str]) -> list[tuple[str, str]]:+ """Of *language_tags*, those whose tree-sitter queries fail to compile.++ Scoped to what traversal discovered in the repo. Only checks languages with+ a registered LanguageConfig and query file. Best-effort and bounded.+ """+ failed: list[tuple[str, str]] = []+ seen: set[str] = set()++ for tag in language_tags:+ if tag in seen or tag not in LANGUAGE_CONFIGS:+ continue+ seen.add(tag)++ _, err = _compile_query(tag)+ if err is not None:+ failed.append((tag, err))+ continue++ # Check grammar variants if applicable (e.g. tsx for typescript)+ if tag == "typescript":+ _, tsx_err = _compile_query("typescript", grammar_tag="tsx")+ if tsx_err is not None:+ failed.append((tag, f"tsx variant: {tsx_err}"))++ return sorted(failed, key=lambda x: x[0])+++def _get_language(tag: str) -> Language | None:+ # Filled in place, never rebound: the package re-exports this dict.+ if not _LANGUAGE_REGISTRY:+ _LANGUAGE_REGISTRY.update(_build_language_registry())+ return _LANGUAGE_REGISTRY.get(tag)+++def grammar_tag_for(language: str, path: str) -> str:+ """The grammar a file is read with, which is not always its language tag.++ A ``.tsx`` file arrives tagged ``typescript``, and tree-sitter-typescript's+ default grammar errors on every ``<Component />``. Only the grammar moves:+ the language tag keeps selecting dialects and vocabularies, several of+ which (``mocks/lexicon.py`` among them) carry no ``tsx`` row and would+ silently degrade if handed one.+ """+ # Case-folded, because the extension table that tagged the file is.+ if language == "typescript" and path.lower().endswith(".tsx"):+ return "tsx"+ return language+++def _parse_tree(+ lang: str, language: Language, grammar_tag: str, source: bytes+) -> tuple[Node, list[str], str, Language]:+ """``(root, parse errors, grammar tag, language)``; a JSX-bearing ``.ts`` retries as TSX."""+ parser = Parser(language)+ tree = parser.parse(source)+ root = tree.root_node++ parse_errors = _collect_error_nodes(root)++ # A .ts file (or a .vue render function) holding JSX errors under the+ # TypeScript grammar; retry as TSX and keep it only when it errors less.+ # ``source`` is the markup-blanked projection, so the token test sees real JSX.+ if (+ parse_errors+ and lang in ("typescript", "vue")+ and grammar_tag != "tsx"+ and (b"/>" in source or b"</" in source)+ ):+ tsx_language = _get_language("tsx")+ if tsx_language is not None:+ tsx_tree = Parser(tsx_language).parse(source)+ tsx_errors = _collect_error_nodes(tsx_tree.root_node)+ if len(tsx_errors) < len(parse_errors):+ tree = tsx_tree+ root = tree.root_node+ parse_errors = tsx_errors+ # The caller's query appends tsx.scm for the tsx tag, which+ # supplies the JSX component call sites, so both must change.+ grammar_tag = "tsx"+ language = tsx_language+ return root, parse_errors, grammar_tag, languagediff --git a/packages/core/src/repowise/core/ingestion/parser/imports.py b/packages/core/src/repowise/core/ingestion/parser/imports.pynew file mode 100644index 000000000..b1b198f7f--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/parser/imports.py@@ -0,0 +1,65 @@+"""Import extraction: the shared dedup loop over a file's import matches.++What each statement yields is the language's ``LanguageConfig.import_extractor``+(``extractors.imports``); this loop only dedupes and stamps load kinds.+"""++from __future__ import annotations++from ..extractors import node_text+from ..language_configs import LanguageConfig+from ..models import FileInfo, Import, combine_load_kinds+from ..parser_helpers import import_load_kind+++def _extract_imports(+ matches: list[dict],+ config: LanguageConfig,+ file_info: FileInfo,+ src: str,+) -> list[Import]:+ language = file_info.language+ rules = config.import_extractor+ imports: list[Import] = []+ # Statement key -> the Imports it produced, so a repeat folds its load+ # kind into them rather than adding a duplicate.+ seen_imports: dict[str, list[Import]] = {}+ seen_own: set[str] = set()++ for capture_dict in matches:+ stmt_nodes = capture_dict.get("import.statement", [])+ module_nodes = capture_dict.get("import.module", [])++ if not stmt_nodes or not module_nodes:+ continue++ stmt_node = stmt_nodes[0]+ if rules.deduped is not None:+ imports.extend(rules.deduped(stmt_node, module_nodes[0], src, seen_own))+ continue++ raw = node_text(stmt_node, src).strip()+ dedup_key = rules.dedup_key(stmt_node, raw, src) if rules.dedup_key else raw+ kind = import_load_kind(stmt_node, language, src)+ if dedup_key in seen_imports:+ # A lazy copy must not hide a load-time twin: runtime wins.+ for imp in seen_imports[dedup_key]:+ imp.type_only, imp.deferred = combine_load_kinds(+ (imp.type_only, imp.deferred), kind+ )+ continue+ seen_imports[dedup_key] = []++ module_text = node_text(module_nodes[0], src).strip().strip("\"'` ")+ if not module_text:+ continue++ found = rules.statement(stmt_node, module_nodes[0], module_text, raw, language, src)+ for imp in found:+ imp.type_only, imp.deferred = kind+ seen_imports[dedup_key] = found+ imports.extend(found)++ if rules.finalize is not None:+ imports = rules.finalize(imports)+ return importsdiff --git a/packages/core/src/repowise/core/ingestion/parser/references.py b/packages/core/src/repowise/core/ingestion/parser/references.pynew file mode 100644index 000000000..1b153b04a--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/parser/references.py@@ -0,0 +1,215 @@+"""Reference sites (a function named without a call) and type references."""++from __future__ import annotations++from dataclasses import dataclass, field++from tree_sitter import Node++from ..extractors import node_text+from ..models import CallSite, FileInfo, Import, Symbol, TypeReference+from ..parser_helpers import (+ TYPE_HEAD_EXTRACTORS,+ _classify_param_origin,+ _find_enclosing_symbol,+ _head_type_identifier,+ _rust_shadowed_by_type_param,+)++_node_text = node_text++# Languages whose query defines the ``@reference.*`` captures. The SFC tags+# reuse the TS captures but stay out: resolution has no local-shadow scan for them.+_REFERENCE_LANGUAGES = ("cpp", "c", "go", "rust", "kotlin", "typescript", "javascript", "python")++# Languages where a bare name resolves only through the file's own symbols or+# a name it imports, so any other bare name cannot pass the reference floor.+_NAMED_BINDING_LANGUAGES = frozenset({"typescript", "javascript", "python"})+++def _bare_reference_names(+ language: str, symbols: list[Symbol], imports: list[Import]+) -> frozenset[str] | None:+ """Names a receiver-less reference in this file could resolve to, or None for any.++ Most bare identifiers in a value position are locals; dropping them here+ skips their per-site work. A wildcard import keeps every name.+ """+ if language not in _NAMED_BINDING_LANGUAGES:+ return None+ imported = {name for imp in imports for name in imp.local_names}+ if "*" in imported:+ return None+ return frozenset(imported.union(s.name for s in symbols))+++def _extract_references(+ matches: list[dict],+ file_info: FileInfo,+ src: str,+ symbols: list[Symbol],+ imports: list[Import],+) -> list[CallSite]:+ """Extract sites that name a function without calling it.++ Six shapes carry a function by name and never call it where a parser+ can see: a dispatch-table entry, a callback field, an argument to a+ registration macro, a func value in argument position, a struct-field+ initialiser, and a ``::`` callable reference. Each leaves the named+ function with no inbound edge, which read as a ``safe_to_delete``+ unused export and took out whole handler and interop layers.+ TS/JS and Python capture any value position: an argument, a list or+ object entry, a return, an assignment and a default export.++ ``@reference.receiver`` is optional; capturing it is what lets+ ``_add_reference_edges`` restrict a bare name to free functions and+ allow a qualified name to reach a method.++ Self-gating on the reference captures, so a language whose query+ defines none produces nothing and pays two dict lookups. Two guards+ keep the broad syntactic positions from claiming ordinary code:++ * A macro argument requires a SCREAMING_CASE callee and must be that+ macro's only argument. Uppercase alone is not enough: assertion+ macros are spelled the same way and take values, so ``EXPECT_EQ(+ capacity, 100)`` bound a local to whatever free function shared its+ name. Registering something registers one thing, which separates+ ``BENCHMARK(BM_Foo)`` from ``TEST(SuiteName, TestName)``.+ * A table entry must sit outside any function body. A dispatch table is+ a file-, namespace- or class-scope aggregate; the same braces inside+ a function are a constructor member-init or a local aggregate, where+ ``{data, size}`` names parameters, not functions.++ Whether the name denotes a function at all is settled later, at+ resolution, where the symbol kind is known.+ """+ from ..language_data import get_builtin_calls++ scope = _ReferenceScope(+ builtins=get_builtin_calls(file_info.language),+ bindable=_bare_reference_names(file_info.language, symbols, imports),+ symbol_ranges=sorted(+ [(s.start_line, s.end_line, s.id) for s in symbols],+ key=lambda t: (t[0], -t[1]),+ ),+ callable_ids={s.id for s in symbols if s.kind in ("function", "method")},+ )+ references: list[CallSite] = []++ for capture_dict in matches:+ plain_nodes = capture_dict.get("reference.name", [])+ table_nodes = capture_dict.get("reference.table", [])+ if not plain_nodes and not table_nodes:+ continue++ receiver_nodes = capture_dict.get("reference.receiver", [])+ receiver = (+ _node_text(receiver_nodes[0], src).strip() if receiver_nodes else None+ ) or None+ if not _macro_argument_allowed(capture_dict, plain_nodes, src):+ continue++ candidates = [(node, False) for node in plain_nodes]+ candidates += [(node, True) for node in table_nodes]+ for name_node, is_table in candidates:+ site = scope.site(name_node, is_table, receiver, src)+ if site is not None:+ references.append(site)++ return references+++def _macro_argument_allowed(capture_dict: dict, plain_nodes: list[Node], src: str) -> bool:+ """A registration-macro argument needs a SCREAMING_CASE callee and to be its only argument."""+ via_nodes = capture_dict.get("reference.via", [])+ if not via_nodes:+ return True+ via = _node_text(via_nodes[0], src).strip()+ if not via or not via.isupper():+ return False+ arg_list = plain_nodes[0].parent if plain_nodes else None+ return arg_list is not None and len(arg_list.named_children) == 1+++@dataclass+class _ReferenceScope:+ """Per-file state the reference sites of one file are filtered against."""++ builtins: frozenset[str]+ bindable: frozenset[str] | None+ symbol_ranges: list[tuple[int, int, str]]+ callable_ids: set[str]+ seen: set[tuple[int, str, str | None]] = field(default_factory=set)++ def site(+ self, name_node: Node, is_table: bool, receiver: str | None, src: str+ ) -> CallSite | None:+ """The reference *name_node* makes, or None when it is filtered or repeated."""+ name = _node_text(name_node, src).strip()+ if not name or name in self.builtins:+ return None+ if self.bindable is not None and receiver is None and name not in self.bindable:+ return None+ line = name_node.start_point[0] + 1+ enclosing = _find_enclosing_symbol(line, self.symbol_ranges)+ if is_table and enclosing in self.callable_ids:+ return None+ if (line, name, receiver) in self.seen:+ return None+ self.seen.add((line, name, receiver))+ return CallSite(+ target_name=name,+ receiver_name=receiver,+ caller_symbol_id=enclosing,+ line=line,+ argument_count=None,+ )+++def _extract_type_refs(+ matches: list[dict],+ src: str,+ lang: str = "",+) -> list[TypeReference]:+ """Collect ``@param.type`` captures into TypeReference records.++ C# emits these from constructor / method / delegate / primary-ctor+ parameter types; Go emits them from parameter, struct-field, return,+ and composite-literal type positions (see ``go.scm``). The graph+ builder resolves each reference to a defining file via the+ language-specific resolver index and emits a file-level edge.++ The head-identifier extractor is language-specific (Go unwraps+ ``*T`` / ``[]T`` / ``map[K]V`` / ``pkg.T``); see+ ``TYPE_HEAD_EXTRACTORS``. Capture origin is inferred from the+ enclosing node: ``constructor_declaration`` → ``ctor_param``,+ ``method_declaration`` → ``method_param`` (C#);+ ``field_declaration`` → ``field_type``, ``composite_literal`` →+ ``composite_literal`` (Go).+ """+ head_of = TYPE_HEAD_EXTRACTORS.get(lang, _head_type_identifier)++ refs: list[TypeReference] = []+ seen: set[tuple[str, int]] = set()++ for capture_dict in matches:+ type_nodes = capture_dict.get("param.type", [])+ if not type_nodes:+ continue+ for type_node in type_nodes:+ head = head_of(type_node, src)+ if not head:+ continue+ # ``struct Wrapper<Item>`` binds Item as a type parameter; the+ # capture cannot tell it from a real ``struct Item``.+ if lang == "rust" and _rust_shadowed_by_type_param(type_node, head, src):+ continue+ line = type_node.start_point[0] + 1+ key = (head, line)+ if key in seen:+ continue+ seen.add(key)+ origin = _classify_param_origin(type_node)+ refs.append(TypeReference(type_name=head, line=line, origin=origin))++ return refsdiff --git a/packages/core/src/repowise/core/ingestion/parser/symbol_rules.py b/packages/core/src/repowise/core/ingestion/parser/symbol_rules.pynew file mode 100644index 000000000..8e516fda2--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/parser/symbol_rules.py@@ -0,0 +1,397 @@+"""Per-language symbol rules: names, kinds, line spans, decorators, modifiers, visibility."""++from __future__ import annotations++import sys++from tree_sitter import Node++from ..cpp_export_macros import _CppExportType, _is_bodiless_cpp_type+from ..extractors import (+ node_text,+ refine_elixir_call_kind,+ refine_fsharp_type_kind,+ refine_go_type_kind,+ refine_kotlin_class_kind,+ refine_pascal_type_kind,+ refine_swift_class_kind,+)+from ..extractors.bindings.ts_js import declarator_binds_callable, declarator_value_is_module_ref+from ..extractors.visibility import (+ refine_cpp_visibility,+ refine_csharp_visibility,+ refine_java_visibility,+ refine_py_visibility,+ refine_rust_visibility,+ refine_ts_visibility,+)+from ..language_configs import LanguageConfig+from ..models import Symbol+from ..parser_helpers import (+ _elixir_is_template_definition,+ _elixir_symbol_name,+ _fsharp_binding_end_line,+ _fsharp_binding_has_params,+ _objc_is_macro_enum,+ _objc_symbol_name,+)+from ..python_overload import is_python_overload as _is_python_overload++_node_text = node_text++# Languages whose source reaches the parser as TypeScript/JavaScript. The+# SFC tags are here because ``sfc_source`` projects their <script> blocks into+# a TS buffer at identical offsets, so every TS/JS code path applies verbatim.+_TS_JS_LANGUAGES = ("typescript", "javascript", "svelte", "vue", "astro")+++_TS_OVERLOAD_SIGNATURES = frozenset({"function_signature", "method_signature"})+++def _keep_ts_overloads_beside_a_body(symbols: list[Symbol], node_types: list[str]) -> list[Symbol]:+ """Drop a bodiless TS signature whose id has no implementation in the file.++ That is an ambient declaration (``declare function``, a ``.d.ts``), not an+ overload, and it never was a symbol.+ """+ bodied = {s.id for s, t in zip(symbols, node_types, strict=True) if t not in _TS_OVERLOAD_SIGNATURES}+ return [+ s+ for s, t in zip(symbols, node_types, strict=True)+ if t not in _TS_OVERLOAD_SIGNATURES or s.id in bodied+ ]+++def _drop_swift_extensions_of_local_types(symbols: list[Symbol]) -> list[Symbol]:+ """Drop an ``extension`` of a type the same file declares.++ It shares the type's id, so it would replace the type's node and kind; its+ members already name the type as their parent.+ """+ declared = {s.id for s in symbols if s.kind != "impl"}+ return [s for s in symbols if s.kind != "impl" or s.id not in declared]+++def _is_declaration(+ def_node: Node,+ config: LanguageConfig,+ language: str,+ export_type: _CppExportType | None,+ src: str,+) -> bool:+ node_type = def_node.type+ if node_type in config.declaration_node_types:+ return True+ if language == "python":+ return _is_python_overload(def_node, src)+ if export_type is not None:+ return export_type.is_forward_declaration+ return _is_bodiless_cpp_type(language, node_type, def_node)++++_FSHARP_BINDING_NODE_TYPES = ("function_declaration_left", "value_declaration_left")+_DART_FUNCTION_NODE_TYPES = ("function_signature", "getter_signature", "setter_signature")+++def _is_fsharp_binding(language: str, node_type: str) -> bool:+ return language == "fsharp" and node_type in _FSHARP_BINDING_NODE_TYPES+++def _language_symbol_name(language: str, def_node: Node, name: str, src: str) -> str | None:+ """Apply a language's own naming rule; None drops the match."""+ if not name:+ return None+ if language == "elixir":+ # A `def` inside `quote do ... end` is macro body, not a+ # definition of the module that writes it.+ if _elixir_is_template_definition(def_node, src):+ return None+ # `defimpl Proto, for: Type` is named for the module the+ # compiler generates, not for the protocol alone.+ return _elixir_symbol_name(def_node, name, src)+ if language == "objectivec":+ # `typedef NS_ENUM(NSInteger, Kind) { ... }` has no grammar+ # rule, so only its enum *cases* survive as declarators and+ # each would become a symbol named as though it were the type.+ if _objc_is_macro_enum(def_node, src):+ return None+ # A method is named by its whole selector+ # (`initWithName:age:`), which no single node holds, and a+ # category by the class it extends plus its own name.+ return _objc_symbol_name(def_node, name, src)+ return name+++# ``(language, node_type)`` whose "class" kind the body refines: Kotlin+# interface / enum class share class_declaration, Pascal ``declType`` wraps+# class, record, interface, enum and more, and F# writes a class, a struct+# and an interface with the same ``anon_type_defn`` node.+_CLASS_KIND_REFINERS = {+ ("kotlin", "class_declaration"): refine_kotlin_class_kind,+ ("swift", "class_declaration"): refine_swift_class_kind,+ ("pascal", "declType"): refine_pascal_type_kind,+ ("fsharp", "anon_type_defn"): refine_fsharp_type_kind,+}+++def _refine_symbol_kind(+ kind: str, def_node: Node, config: LanguageConfig, language: str, src: str+) -> str:+ """Narrow a node-type kind where one node shape spells several kinds."""+ node_type = def_node.type+ # A value binding that carries parameter patterns beside its+ # name is a function the grammar reparsed as a value because+ # of its return-type annotation.+ if (+ language == "fsharp"+ and node_type == "value_declaration_left"+ and _fsharp_binding_has_params(def_node)+ ):+ kind = "function"++ # Refine "struct" kind for Go type_spec (check if struct or interface body)+ if kind == "struct" and config.parent_extraction == "receiver":+ kind = refine_go_type_kind(def_node, src)++ if kind == "class":+ refine = _CLASS_KIND_REFINERS.get((language, node_type))+ if refine is not None:+ kind = refine(def_node)++ # Elixir: every definition is a ``call``; its keyword names the kind.+ if language == "elixir" and node_type == "call":+ kind = refine_elixir_call_kind(def_node, src)+ return kind+++def _symbol_end_line(+ def_node: Node,+ config: LanguageConfig,+ language: str,+ export_type: _CppExportType | None,+) -> int | None:+ """The symbol's last line, or None for a Dart local function (not a symbol)."""+ node_type = def_node.type+ end_line = def_node.end_point[0] + 1+ if config.symbol_end_line_fn is not None:+ end_line = config.symbol_end_line_fn(def_node, end_line)+ if export_type is not None:+ end_line = export_type.range_node.end_point[0] + 1+ # F#: the captured left-hand side stops at the parameters; extend over the body.+ if _is_fsharp_binding(language, node_type):+ end_line = _fsharp_binding_end_line(def_node)+ if language == "dart" and node_type in _DART_FUNCTION_NODE_TYPES:+ return _dart_function_end_line(def_node, end_line)+ return end_line+++def _dart_function_end_line(def_node: Node, end_line: int) -> int | None:+ """Dart: a function's body is a sibling of its signature.++ Local functions nested inside another function's body have no callable+ *ancestor* (the enclosing signature is a sibling), so they are dropped+ here (None); and the line range extends to the trailing body sibling or+ call-site attribution stops at the signature line.+ """+ ancestor = def_node.parent+ while ancestor is not None:+ if ancestor.type in ("function_body", "function_expression"):+ return None+ ancestor = ancestor.parent+ anchor = def_node+ if def_node.parent is not None and def_node.parent.type == "method_signature":+ anchor = def_node.parent+ body_sibling = anchor.next_named_sibling+ if body_sibling is not None and body_sibling.type == "function_body":+ return body_sibling.end_point[0] + 1+ return end_line+++def _module_binding_kind(def_node: Node, name: str, language: str, src: str) -> str | None:+ """Kind of a module-level assignment, or None when it only binds a module.++ SCREAMING_CASE names are constants by convention; the rest are module+ variables (singletons like ``app = FastAPI()``, registries, caches).+ ``str.isupper()`` requires at least one cased char, so names with no+ letters (``_``, ``__all__``) fall to "variable" rather than being+ mislabelled constants by ``name == name.upper()``.+ """+ # TS/JS: ``const svc = require('./svc')`` binds a module (already an+ # import), so it is not a symbol.+ if language in _TS_JS_LANGUAGES and declarator_value_is_module_ref(def_node, src):+ return None+ # A structurally callable value is a function whatever its name says.+ callable_kind = (+ declarator_binds_callable(def_node, src) if language in _TS_JS_LANGUAGES else None+ )+ return callable_kind or ("constant" if name.isupper() else "variable")+++def _attribute_decorators(def_node: Node, language: str, src: str) -> list[str]:+ """Attribute texts that act as decorators (Rust, C#, C/C++), brackets stripped.++ They land beside ``@`` decorators so one deprecation check reads them all.+ """+ if language == "rust":+ return _rust_outer_attributes(def_node, src)+ # C#: ``attribute_list`` is child[0] of the declaration itself, not a+ # preceding sibling in the class body.+ if language == "csharp":+ return _leading_child_attributes(def_node, "attribute_list", "[", "]", src)+ # C/C++: [[deprecated]] / [[deprecated("reason")]] are+ # ``attribute_declaration`` nodes, child[0] of function_definition itself+ # (NOT a preceding sibling at translation_unit level).+ if language in ("cpp", "c"):+ return _leading_child_attributes(def_node, "attribute_declaration", "[[", "]]", src)+ if language == "kotlin":+ return _kotlin_detached_annotation(def_node, src)+ return []+++def _kotlin_detached_annotation(def_node: Node, src: str) -> list[str]:+ """Kotlin: annotations the grammar split off a top-level declaration.++ When the last annotation on its own line before a bare ``fun`` or+ ``object`` takes one argument (``@Suppress("unused")``), the argument list+ reads as a parenthesized expression: the annotations become an+ ``annotated_expression`` chain beside a declaration with no modifiers.+ """+ if def_node.parent is None or def_node.parent.type != "source_file":+ return []+ sib = def_node.prev_sibling+ while sib is not None and sib.type in ("line_comment", "block_comment"):+ sib = sib.prev_sibling+ return [_node_text(sib, src).strip()] if _is_split_annotation_chain(sib) else []+++def _is_split_annotation_chain(node: Node | None) -> bool:+ """Nested ``annotated_expression`` nodes of annotations, the last one's argument at the tail."""+ while node is not None and node.type == "annotated_expression":+ kinds = [c.type for c in node.named_children]+ if len(kinds) != 2 or kinds[0] != "annotation":+ return False+ if kinds[1] == "parenthesized_expression":+ return True+ node = node.named_children[1]+ return False+++def _declared_modifiers(def_node: Node, node_types: frozenset[str], src: str) -> tuple[str, ...]:+ """Lowercased keyword modifiers written in *def_node*'s modifier children.++ Keywords are the anonymous leaves of those children (C# ``modifier`` >+ ``override``, Kotlin ``modifiers`` > ``member_modifier`` > ``override``) or+ a named leaf that is itself a modifier (VB.NET ``modifier`` ``Overrides``).+ Named leaves such as Scala's ``protected[this]`` qualifier are not+ keywords. Annotations and attributes nested among them are skipped: they+ are decorators, not modifiers. The words are interned: a handful of+ distinct values repeat on every symbol.+ """+ if not node_types:+ return ()+ words: list[str] = []+ stack = [c for c in reversed(def_node.children) if c.type in node_types]+ while stack:+ node = stack.pop()+ if "annotation" in node.type or "attribute" in node.type:+ continue+ if node.child_count == 0:+ text = _node_text(node, src).strip()+ keyword = not node.is_named or "modifier" in node.type+ if keyword and text.isalpha():+ words.append(sys.intern(text.lower()))+ continue+ stack.extend(reversed(node.children))+ return tuple(words)+++def _rust_outer_attributes(def_node: Node, src: str) -> list[str]:+ """Rust: outer attributes (#[...]) are preceding siblings of the item."""+ attrs: list[str] = []+ if def_node.parent is None:+ return attrs+ siblings = def_node.parent.children+ for j, sib in enumerate(siblings):+ if sib.id != def_node.id:+ continue+ k = j - 1+ while k >= 0 and siblings[k].type == "attribute_item":+ attr_text = _node_text(siblings[k], src).strip()+ # Strip #[ and ] to get the inner attribute text+ if attr_text.startswith("#[") and attr_text.endswith("]"):+ attrs.append(attr_text[2:-1])+ k -= 1+ break+ return attrs+++def _leading_child_attributes(+ def_node: Node, node_type: str, opener: str, closer: str, src: str+) -> list[str]:+ """Inner text of the attribute children leading *def_node*, until the first other child.++ A comment between attribute lists does not end the run, so an attribute+ placed after one is still recorded.+ """+ attrs: list[str] = []+ for child in def_node.children:+ if child.type != node_type:+ if child.type == "comment":+ continue+ break+ attr_text = _node_text(child, src).strip()+ if attr_text.startswith(opener) and attr_text.endswith(closer):+ attrs.append(attr_text[len(opener) : -len(closer)])+ return attrs+++def _refine_visibility(+ def_node: Node,+ language: str,+ visibility: str,+ name: str,+ ts_deferred_exports: frozenset[str] | None,+ py_all_names: frozenset[str] | None,+ src: str,+) -> tuple[str, bool]:+ """``(visibility, is_exported_symbol)`` after the language's AST-context rules."""+ # C/C++ (and plain C inside Objective-C) visibility comes from AST+ # context: access specifiers, storage class, export attributes.+ if language in ("cpp", "c", "objectivec"):+ return refine_cpp_visibility(def_node, visibility, src)+ # C#: an unmodified declaration's default depends on what encloses+ # it, which the modifier-text fn cannot see.+ if language == "csharp":+ return refine_csharp_visibility(def_node, visibility), False+ # Java: no access keyword means package-private, except inside an+ # interface or annotation body, which the modifier-text fn cannot see.+ if language == "java":+ return refine_java_visibility(def_node), False+ # TS/JS: a top-level declaration is only public when exported —+ # inline, via ``export { x }`` lists, or ``export default x``.+ if language in _TS_JS_LANGUAGES:+ return refine_ts_visibility(def_node, visibility, name, ts_deferred_exports), False+ # Python: a literal ``__all__`` raises the names it lists to public; it+ # never demotes the ones it omits. GDScript shares ``py_visibility`` but+ # has no ``__all__``, so the gate is on the language, not the fn.+ if language == "python":+ return refine_py_visibility(def_node, visibility, name, py_all_names), False+ # Rust: a trait's items may not write ``pub`` of their own, so the+ # trait's modifier is the only place their visibility is stated.+ if language == "rust":+ return refine_rust_visibility(def_node, visibility, src), False+ return visibility, False+++def _dart_mixin_parent(def_node: Node, config: LanguageConfig, src: str) -> str | None:+ """Name of the Dart ``mixin`` enclosing *def_node*, if it is the nearest type."""+ ancestor = def_node.parent+ while ancestor is not None:+ if ancestor.type == "mixin_declaration":+ ident = next((c for c in ancestor.children if c.type == "identifier"), None)+ return _node_text(ident, src) if ident is not None else None+ if ancestor.type in config.parent_class_types:+ return None+ ancestor = ancestor.parent+ return Nonediff --git a/packages/core/src/repowise/core/ingestion/parser/symbols.py b/packages/core/src/repowise/core/ingestion/parser/symbols.pynew file mode 100644index 000000000..49f064aa1--- /dev/null+++ b/packages/core/src/repowise/core/ingestion/parser/symbols.py@@ -0,0 +1,541 @@+"""Symbol extraction: one query match to one ``Symbol``, and the parent it belongs to."""++from __future__ import annotations++from dataclasses import dataclass, field++from tree_sitter import Node++from ..cpp_export_macros import (+ CppExportTypes,+ _cpp_export_macro_parent,+ _cpp_normalize_identifier,+ _CppExportType,+ collect_cpp_export_types,+)+from ..extractors import (+ build_signature,+ extract_go_receiver_type,+ extract_symbol_docstring,+ node_text,+)+from ..extractors.synthetic_symbols import extract_synthetic_symbols+from ..extractors.synthetic_symbols.csharp_extension_blocks import drop_extension_block_symbols+from ..extractors.visibility import py_module_all_names, ts_deferred_export_names+from ..language_configs import LanguageConfig+from ..models import FileInfo, Symbol+from ..parser_helpers import (+ _build_qualified_name,+ _csharp_type_parameter_count,+ _dedupe_objc_interface_symbols,+ _dedupe_pascal_interface_symbols,+ _elixir_module_parent,+ _fsharp_binding_is_nested,+ _fsharp_parent_is_type,+ _fsharp_parent_name,+ _has_callable_ancestor,+ _is_async_node,+ _kotlin_receiver_fields,+ _objc_container_node,+ _objc_container_parent,+ _qualified_cpp_parent,+ _qualified_pascal_parent,+ _ts_nested_object_method_owner,+ follow_renamed_hosts,+ in_anonymous_class,+ is_anonymous_member,+ scope_segment,+)+from ..symbol_identity import disambiguate_colliding_ids, symbol_discriminator+from .symbol_rules import (+ _TS_JS_LANGUAGES,+ _attribute_decorators,+ _dart_mixin_parent,+ _declared_modifiers,+ _drop_swift_extensions_of_local_types,+ _is_declaration,+ _is_fsharp_binding,+ _keep_ts_overloads_beside_a_body,+ _language_symbol_name,+ _module_binding_kind,+ _refine_symbol_kind,+ _refine_visibility,+ _symbol_end_line,+)++_node_text = node_text++# Node types whose .scm patterns are anchored at module level, so the+# callable-ancestor filter must not apply to them.+_MODULE_ANCHORED_NODE_TYPES = frozenset({"assignment", "variable_declarator"})+++def _match_identity(+ capture_dict: dict,+ config: LanguageConfig,+ language: str,+ src: str,+ cpp_exports: CppExportTypes,+ seen: set[tuple[int, str]],+) -> tuple[Node, list[Node], str, _CppExportType | None, int] | None:+ """``(def_node, name_nodes, name, export_type, start_line)`` for a new symbol match.++ None drops the match: no definition or name, a recovery match this file+ does not support, a suppressed export macro, or a duplicate. The+ ``(start_line, name)`` key is recorded in *seen* here.+ """+ def_nodes = capture_dict.get("symbol.def", [])+ name_nodes = capture_dict.get("symbol.name", [])+ captured_export_type_nodes = capture_dict.get("symbol.cpp_export_type", [])++ if (+ captured_export_type_nodes+ and captured_export_type_nodes[0].id not in cpp_exports.capture_ids+ ):+ # The query also sees ordinary ``struct Tag variable;`` forms;+ # discard only the unsupported recovery match.+ return None++ if not def_nodes or not name_nodes:+ return None++ def_node = def_nodes[0]+ name = _language_symbol_name(+ language, def_node, config.symbol_name_fn(_node_text(name_nodes[0], src), def_node.type), src+ )+ if name is None:+ return None++ export_type = cpp_exports.defs.get(def_node.id)+ if export_type is not None and name != export_type.name:+ # The ordinary struct/class query sees the same specifier, but+ # tree-sitter calls the export macro its name. Keep only the+ # dedicated match whose name is the outer declarator.+ return None++ if def_node.type == "preproc_def" and (+ _cpp_normalize_identifier(name) in cpp_exports.macro_names+ or def_node.id in cpp_exports.macro_def_ids+ ):+ # Body-form macros are suppressed by name;+ # ambiguous forward declarations suppress only the exact+ # active definition that made recovery safe.+ return None++ start_line = def_node.start_point[0] + 1+ if export_type is not None:+ start_line = export_type.range_node.start_point[0] + 1+ dedup_key = (start_line, name)+ if dedup_key in seen:+ return None+ seen.add(dedup_key)+ return def_node, name_nodes, name, export_type, start_line+++def _parent_type_name(ancestor: Node, src: str) -> str | None:+ """The name a class/impl ancestor gives its members, without type arguments."""+ name_node = ancestor.child_by_field_name("name") or (+ ancestor.child_by_field_name("type") # Rust impl_item+ )+ if name_node is None:+ return None+ # For Rust impl blocks with generic types (e.g. impl<T> Foo<T>),+ # extract only the base type name, not the full generic signature.+ if name_node.type in ("generic_type", "scoped_type_identifier"):+ inner = name_node.child_by_field_name(+ "type" if name_node.type == "generic_type" else "name"+ )+ if inner and inner.type == "type_identifier":+ name_node = inner+ return _node_text(name_node, src)+++def _nesting_parent(def_node: Node, config: LanguageConfig, src: str) -> str | None:+ """The class/impl ancestor naming *def_node*'s parent.++ An anonymous class is a class of its own, so its members gain segments+ below it: ``Gson::FILTER::Filter::check``.+ """+ segments: list[str] = []+ child, ancestor = def_node, def_node.parent+ while ancestor is not None:+ if ancestor.type in config.parent_class_types:+ name = _parent_type_name(ancestor, src)+ if name is not None:+ return "::".join([name, *reversed(segments)])+ else:+ segment = scope_segment(+ ancestor, child, bool(segments), config.anonymous_class_types, src+ )+ if segment is not None:+ segments.append(segment)+ child, ancestor = ancestor, ancestor.parent+ # A Kotlin top-level function's object expression: ``f::Foo``.+ return "::".join(reversed(segments)) or None+++def _java_enum_constant(def_node: Node, config: LanguageConfig, src: str) -> str | None:+ """The enum constant whose body declares *def_node*, before any enclosing type.++ An anonymous class is such a type: ``_find_parent`` names its members.+ """+ ancestor = def_node.parent+ while (+ ancestor is not None+ and ancestor.type not in config.parent_class_types+ and ancestor.type not in config.anonymous_class_types+ ):+ if ancestor.type == "enum_constant":+ name = ancestor.child_by_field_name("name")+ return _node_text(name, src) if name else None+ ancestor = ancestor.parent+ return None+++def _extract_symbols(+ matches: list[dict],+ config: LanguageConfig,+ file_info: FileInfo,+ src: str,+) -> list[Symbol]:+ language = file_info.language+ symbols: list[Symbol] = []+ # Parallel to ``symbols`` (same indices) -- consumed for Pascal and+ # Objective-C, to dedupe interface-declaration vs. implementation+ # method pairs after the loop, and for TS overload signatures. See+ # _dedupe_pascal_interface_symbols / _dedupe_objc_interface_symbols.+ node_types: list[str] = []+ # Also parallel to ``symbols``, Objective-C only: which of @interface /+ # @implementation / @protocol declared each member, so a protocol's+ # method is never deduped against a same-named class method.+ objc_container_kinds: list[str | None] = []++ # Deferred-export names (``export { x }`` / ``export default x``),+ # computed once per file for the TS/JS visibility refinement.+ ts_deferred_exports: frozenset[str] | None = None+ if language in _TS_JS_LANGUAGES:+ ts_deferred_exports = ts_deferred_export_names(src)+ # Literal module ``__all__`` names, once per file for the Python+ # refinement. None means no signal (built at runtime or absent).+ py_all_names: frozenset[str] | None = None+ if language == "python":+ py_all_names = py_module_all_names(src)+ cpp_exports = (+ collect_cpp_export_types(matches, src) if language == "cpp" else CppExportTypes()+ )++ file = _SymbolPass(config, file_info, src, cpp_exports, ts_deferred_exports, py_all_names)+ for capture_dict in matches:+ built = _symbol_from_match(capture_dict, file)+ if built is None:+ continue+ symbol, def_node = built+ symbols.append(symbol)+ node_types.append(def_node.type)+ if language == "objectivec":+ container = _objc_container_node(def_node, config.parent_class_types)+ objc_container_kinds.append(container.type if container else None)++ if language == "pascal":+ symbols = _dedupe_pascal_interface_symbols(symbols, node_types)++ if language in _TS_JS_LANGUAGES:+ symbols = _keep_ts_overloads_beside_a_body(symbols, node_types)++ if language == "swift":+ symbols = _drop_swift_extensions_of_local_types(symbols)++ # A .m file routinely declares its private methods in a class+ # extension and defines them below in the @implementation, which+ # builds each symbol id twice in one file.+ if language == "objectivec":+ symbols = _dedupe_objc_interface_symbols(symbols, node_types, objc_container_kinds)++ return symbols++@dataclass+class _SymbolPass:+ """Per-file inputs every symbol match of one file is read against."""++ config: LanguageConfig+ file_info: FileInfo+ src: str+ cpp_exports: CppExportTypes+ ts_deferred_exports: frozenset[str] | None+ py_all_names: frozenset[str] | None+ # (start_line, name) of every match kept so far: dedups decorated twins.+ seen: set[tuple[int, str]] = field(default_factory=set)+++def _symbol_from_match(capture_dict: dict, file: _SymbolPass) -> tuple[Symbol, Node] | None:+ """One query match as a symbol and its definition node, or None to drop it.++ Records the match's ``(start_line, name)`` in ``file.seen`` once it passes+ the name checks, so a later duplicate is dropped even if this one is.+ """+ config, file_info, src = file.config, file.file_info, file.src+ language = file_info.language+ identity = _match_identity(capture_dict, config, language, src, file.cpp_exports, file.seen)+ if identity is None:+ return None+ def_node, name_nodes, name, export_type, start_line = identity++ node_type = def_node.type+ # A TS/JS object-literal method inside an anonymous callback is kept,+ # qualified by the binding that owns its object.+ object_owner = (+ _ts_nested_object_method_owner(def_node, src, config.symbol_node_types)+ if language in _TS_JS_LANGUAGES+ else None+ )+ spanned = _kind_and_end_line(def_node, name, export_type, object_owner is not None, file)+ if spanned is None:+ return None+ kind, end_line = spanned++ modifier_texts = [_node_text(m, src) for m in capture_dict.get("symbol.modifiers", [])]+ visibility, is_exported_symbol = _refine_visibility(+ def_node,+ language,+ config.visibility_fn(name, modifier_texts),+ name,+ file.ts_deferred_exports,+ file.py_all_names,+ src,+ )++ parent_name = object_owner or _resolve_parent_name(+ def_node,+ config,+ capture_dict.get("symbol.receiver", []),+ name_nodes,+ language,+ src,+ no_export_type=export_type is None,+ export_type_parents=file.cpp_exports.parents,+ )++ # A ``field_declaration`` cannot occur outside a class body, so a+ # missing parent means the class did not parse. Grammar recovery,+ # not a member function.+ if node_type == "function_declarator" and parent_name is None:+ return None++ params_nodes = capture_dict.get("symbol.params", [])+ params_text = _node_text(params_nodes[0], src) if params_nodes else ""+ scope = _id_scope(def_node, parent_name, config, language, src)+ sym_id = f"{file_info.path}::{scope}::{name}" if scope else f"{file_info.path}::{name}"+ symbol = Symbol(+ id=sym_id,+ name=name,+ qualified_name=_build_qualified_name(file_info.path, scope, name),+ kind=_method_or_kind(kind, parent_name, def_node, language), # type: ignore[arg-type]+ signature=build_signature(node_type, name, params_text, def_node, src),+ start_line=start_line,+ end_line=end_line,+ docstring=extract_symbol_docstring(def_node, src, language),+ decorators=(+ [m for m in modifier_texts if m.startswith("@")]+ + _attribute_decorators(def_node, language, src)+ ),+ visibility=visibility, # type: ignore[arg-type]+ is_async=_is_async_node(def_node, src),+ language=language,+ parent_name=parent_name,+ is_exported_symbol=is_exported_symbol,+ is_declaration=_is_declaration(def_node, config, language, export_type, src),+ type_parameter_count=(+ _csharp_type_parameter_count(def_node) if language == "csharp" else None+ ),+ **_kotlin_receiver_fields(def_node, params_nodes, language, name),+ modifiers=_declared_modifiers(def_node, config.modifier_node_types, src),+ )+ return symbol, def_node+++def _kind_and_end_line(+ def_node: Node,+ name: str,+ export_type: _CppExportType | None,+ has_object_owner: bool,+ file: _SymbolPass,+) -> tuple[str, int] | None:+ """``(kind, end_line)`` of a match, or None when it is not a symbol here."""+ config, language, src = file.config, file.file_info.language, file.src+ kind = _symbol_kind(+ def_node,+ config,+ language,+ src,+ file.cpp_exports.container_ids,+ # An anonymous class member is a method of its own class, even+ # inside another method's body; ``_find_parent`` names it.+ keep_nested=has_object_owner or in_anonymous_class(def_node, config.anonymous_class_types),+ )+ if kind is None:+ return None++ end_line = _symbol_end_line(def_node, config, language, export_type)+ if end_line is None:+ return None++ if def_node.type in _MODULE_ANCHORED_NODE_TYPES:+ kind = _module_binding_kind(def_node, name, language, src)+ if kind is None:+ return None+ return kind, end_line+++def _method_or_kind(kind: str, parent_name: str | None, def_node: Node, language: str) -> str:+ """Upgrade a function with a parent to a method.++ F#: a nested module is a parent too (for id uniqueness), but it is not a+ type, so a ``let`` inside one stays a function.+ """+ if parent_name and kind == "function" and (+ language != "fsharp" or _fsharp_parent_is_type(def_node)+ ):+ return "method"+ return kind+++def _id_scope(+ def_node: Node, parent_name: str | None, config: LanguageConfig, language: str, src: str+) -> str | None:+ """The id scope: the parent, plus a Java enum constant whose body declares the symbol.++ Each constant's override is then its own node while the enum stays its owner.+ """+ constant = _java_enum_constant(def_node, config, src) if language == "java" else None+ return f"{parent_name}::{constant}" if parent_name and constant else parent_name+++def _symbol_kind(+ def_node: Node,+ config: LanguageConfig,+ language: str,+ src: str,+ container_ids: frozenset[int],+ *,+ keep_nested: bool = False,+) -> str | None:+ """The symbol kind for *def_node*, or None when it is not a symbol here."""+ node_type = def_node.type+ kind = config.symbol_node_types.get(node_type)+ if kind is None:+ return None++ # Only module-level and class-body members are symbols; the query is+ # recursive, so defs nested in a callable are dropped. Module-anchored+ # node types only match at module level and skip the check, as does a+ # def the caller has placed already (``keep_nested``).+ if (+ not keep_nested+ and node_type not in _MODULE_ANCHORED_NODE_TYPES+ and _has_callable_ancestor(def_node, config.symbol_node_types, container_ids)+ ):+ return None++ # F#: a ``let`` nested in another binding has no callable ancestor, so+ # it is checked separately. A ``let`` in a type body is a field and stays.+ if _is_fsharp_binding(language, node_type) and _fsharp_binding_is_nested(def_node):+ return None+ return _refine_symbol_kind(kind, def_node, config, language, src)+++def _resolve_parent_name(+ def_node: Node,+ config: LanguageConfig,+ receiver_nodes: list[Node],+ name_nodes: list[Node],+ language: str,+ src: str,+ *,+ no_export_type: bool,+ export_type_parents: dict[int, str],+) -> str | None:+ """The enclosing type or module name, trying each language's own shape."""+ parent_name = _find_parent(def_node, config, receiver_nodes, src)+ if parent_name is not None:+ return parent_name++ if language == "cpp" and no_export_type:+ parent_name = _cpp_export_macro_parent(def_node, export_type_parents)+ if parent_name is not None:+ return parent_name++ # Dart mixin_declaration exposes no ``name`` field, so+ # ``_find_parent``'s field lookup misses mixin members.+ if language == "dart":+ return _dart_mixin_parent(def_node, config, src)++ # F#: no type node carries a ``name`` field -- the name hangs off+ # a ``type_name`` child -- so the generic walk finds the ancestor+ # and then reads nothing off it.+ if language == "fsharp":+ return _fsharp_parent_name(def_node, src)++ # C/C++ out-of-line ``void Foo::method()`` names its class in the scope.+ if language in ("cpp", "c") and name_nodes:+ return _qualified_cpp_parent(name_nodes[0], src)++ # Pascal out-of-line ``function TFoo.Bar`` names its class in the header.+ if language == "pascal" and name_nodes:+ return _qualified_pascal_parent(name_nodes[0], src)++ # Elixir: ``defmodule`` is a ``call``; the name is its first argument.+ if language == "elixir":+ return _elixir_module_parent(def_node, src)++ # Objective-C containers name themselves with a bare first identifier.+ if language == "objectivec":+ return _objc_container_parent(def_node, config.parent_class_types, src)+ return None+++def _find_parent(+ def_node: Node,+ config: LanguageConfig,+ receiver_nodes: list[Node],+ src: str,+) -> str | None:+ """Determine the parent class/type for a symbol."""+ if config.parent_extraction == "receiver":+ # Go: extract type name from receiver parameter list+ if receiver_nodes:+ return extract_go_receiver_type(_node_text(receiver_nodes[0], src))+ return None++ if config.parent_extraction in ("nesting", "impl"):+ return _nesting_parent(def_node, config, src)++ return None # "none" mode+++def _finalize_symbols(+ symbols: list[Symbol], root: Node, src: str, file_info: FileInfo+) -> list[Symbol]:+ """Add generator-emitted symbols and settle colliding ids.++ Runs before calls and references are extracted, so both attribute to+ the final ids.+ """+ lang = file_info.language+ synthetic = extract_synthetic_symbols(root, src, file_info)+ if synthetic:+ # Keyed with the discriminator too, so a record's synthetic+ # canonical constructor survives beside an explicit overload of+ # another arity. A language without one dedupes on the id alone.+ existing = {(s.id, symbol_discriminator(s)) for s in symbols}+ symbols.extend(s for s in synthetic if (s.id, symbol_discriminator(s)) not in existing)+ if lang == "csharp":+ # The grammar reads a C# 14 extension block as a constructor+ # named ``extension``; its members were added above instead.+ symbols = drop_extension_block_symbols(symbols, root, src)+ # Anonymous members go last, once the overloads holding them carry+ # their discriminators.+ anonymous = [s for s in symbols if is_anonymous_member(s)]+ disambiguate_colliding_ids([s for s in symbols if not is_anonymous_member(s)], lang)+ follow_renamed_hosts(symbols)+ disambiguate_colliding_ids(anonymous, lang)+ return symbolsdiff --git a/packages/core/src/repowise/core/ingestion/parser_helpers.py b/packages/core/src/repowise/core/ingestion/parser_helpers.pyindex ba5ef444c..e41e90971 100644--- a/packages/core/src/repowise/core/ingestion/parser_helpers.py+++ b/packages/core/src/repowise/core/ingestion/parser_helpers.py@@ -1,7 +1,7 @@"""Stateless AST helper functions used by :class:`~.parser.ASTParser`.Language-neutral tree-sitter utilities live here; per-language helpers live in-:mod:`.lang_helpers` and are re-exported so ``parser.py`` imports one module.+:mod:`.lang_helpers` and are re-exported so the ``parser`` package imports one module.No imports from ``parser``, so there is no import cycle."""diff --git a/tests/unit/ingestion/parser/test_package_layout.py b/tests/unit/ingestion/parser/test_package_layout.pynew file mode 100644index 000000000..afef3b676--- /dev/null+++ b/tests/unit/ingestion/parser/test_package_layout.py@@ -0,0 +1,65 @@+"""The parser package keeps its old import surface and wires every language's import rules."""++from __future__ import annotations++import pytest++import repowise.core.ingestion.parser as parser_pkg+from repowise.core.ingestion.extractors.imports import GENERIC_IMPORTS, ImportExtractor+from repowise.core.ingestion.language_configs import LANGUAGE_CONFIGS++# Names other code and open branches import from ``repowise.core.ingestion.parser``.+_KEPT_NAMES = (+ "ASTParser",+ "LANGUAGE_CONFIGS",+ "LanguageConfig",+ "QUERIES_DIR",+ "_PASSTHROUGH_LANGUAGES",+ "_compile_query",+ "_get_language",+ "_load_compiled_query",+ "_run_query",+ "failed_query_languages",+ "grammar_tag_for",+ "log",+ "missing_grammar_languages",+ "parse_file",+ "_call_receiver_from_node",+ "_dedupe_calls",+ "_fold_receiverless_twins",+ "_receiver_text",+ "_match_identity",+ "_refine_symbol_kind",+ "_refine_visibility",+ "_nesting_parent",+ "_bare_reference_names",+ "_generic_import",+ "_rust_use_imports",+ "_cjs_imports",+ "_pascal_imports",+)++[email protected]("name", _KEPT_NAMES)+def test_old_names_stay_importable(name: str) -> None:+ assert hasattr(parser_pkg, name)+++def test_queries_dir_points_at_the_query_files() -> None:+ assert (parser_pkg.QUERIES_DIR / "python.scm").is_file()+++def test_every_config_carries_an_import_extractor() -> None:+ for tag, config in LANGUAGE_CONFIGS.items():+ assert isinstance(config.import_extractor, ImportExtractor), tag+ assert LANGUAGE_CONFIGS["go"].import_extractor is GENERIC_IMPORTS+ typescript = LANGUAGE_CONFIGS["typescript"].import_extractor+ assert LANGUAGE_CONFIGS["svelte"].import_extractor is typescript+++def test_language_registry_export_is_the_live_registry() -> None:+ from repowise.core.ingestion.parser import grammars++ parser_pkg._get_language("python")+ assert parser_pkg._LANGUAGE_REGISTRY is grammars._LANGUAGE_REGISTRY+ assert "python" in parser_pkg._LANGUAGE_REGISTRY