Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
fix/Dependency-forward-refs · L-260929-749d49 (#1299)
Louis Choquel68ff6052205f
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 34b262b6d..01bd93e4d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -6,10 +6,12 @@- **`read_scope` is a required run parameter (Breaking)**: `RunMetadata`, `pipeline_run_setup`, `prepare_pipe_job`, `PipelexKernel.make`, `shape_inputs`, `InputShaper.shape`, `WorkingMemoryFactory.make_from_pipeline_inputs`, `StuffFactory.make_stuff_from_stuff_content_or_data`, `normalize_data_urls_to_storage` and the runtime bridge's `PipelexPipeRunInput` take a `read_scope` with no default: a host serving several tenants passes the tenant's prefix, which must contain the run's `storage_scope`, and `None` states that the run is unscoped. `PipelexMTHDSProtocol` defaults it to `None`, which is the truth on a laptop.- **Dry-run mocks of images and documents carry readable URLs**: a mocked `ImageContent` carries one of `inference.dry_run.image_urls` and a mocked `DocumentContent` a blank one-page PDF as a `data:` URL, where both carried a random string that reads as a local path; and a model nested two levels or more inside a mocked structure now gets the same field providers as a shallower one.+- **A structure field naming a concept that cannot be found refuses the load (Breaking)**: a concept whose field names a concept that no bundle it can see declares now raises `LibraryLoadingError` at load, naming the concept, its bundle and the missing structure class, where it loaded and failed with pydantic's "not fully defined" error at first use. A concept declared only by a bundle loaded afterwards cannot be found either. `load_from_crate` does not refuse, since a crate carries no dependency package, and neither does a method package's own load, since its own dependencies are never loaded; a concept holding a method package's concept left incomplete that way is not refused either.- **PipeCompose, PipeSearch and PipeImgGen refuse an input they never read (Breaking)**: as PipeLLM already did, validation refuses a declared input that none of the pipe's templates reads (the template or the construct of a PipeCompose, the prompt of a PipeSearch, the prompt or negative prompt of a PipeImgGen) as `extraneous_input_variable`, naming the input: reference it in a template, or remove it from `inputs`. A variable a PipeImgGen's `negative_prompt` reads without declaring it is refused when the bundle is parsed, and `UnusedInputError`, which nothing raised, is removed.### Fixed+- **Concepts holding other concepts are complete**: a method package's concept holding another of the package's concepts, a field typed by a method package's concept (`address->domain.Concept`), a field typed by a concept that an earlier load put into the same library, a field typed by a concept backed by a Python class (`structure = "ClassName"`), and a field typed `native.Anything`, which holds any value, now resolve, where each loaded and failed with pydantic's "not fully defined" error at first use. A method package whose concepts form a cycle is refused as a bundle's own concepts are, naming the package's bundle.- **The input errors of the template operators carry their type and names**: a missing or unread input of a PipeLLM, PipeCompose, PipeSearch or PipeImgGen is reported with its `error_type` and with `variable_names` holding the plain names, sorted. A PipeCompose template reading an undeclared variable was an uncategorized item, the PipeSearch and PipeImgGen items carried no names, and PipeLLM's unread names arrived quoted (`'platform'`).- **A template read through a subscript, a call or a `set` counts as a read**: an input a template reads only as `{{ items[0].text }}`, `{{ (items|first).text }}` or `{{ record['meta'].title }}`, or reassigns with `{% set topic = topic|trim %}`, is seen as read, where PipeLLM refused it as unread, and such a read of an undeclared variable is refused as missing. A read placed before a top-level `{% set %}` of the same name, and a `for` loop's iterable or `else` branch, read the input. The name a top-level `{% set %}…{% endset %}` block assigns, a call to one of Jinja's globals such as `range(count)` or `namespace()`, and `caller`, `varargs` and `kwargs` inside a macro are no longer demanded as inputs, while an input named like a global, such as `range`, is read wherever the template reads the name without calling it. The images and documents PipeLLM attaches, and the reference images PipeImgGen attaches, are found under the same scopes, so an image read after `{% set image = image %}` or as the iterable of a loop named after it is attached.- **A generated object's storage key names its own format**: the extension of a key under `generated/` follows the whole MIME type the object is stored under, so a fetched `image/svg+xml` is stored as `.svg`, and a type with no extension of its own gets `.bin`, where both used to get `.jpg`. The MIME type's parameters and case no longer change the extension.diff --git a/docs/building-methods/concepts/inline-structures.md b/docs/building-methods/concepts/inline-structures.mdindex f728a2601..85aa9eba9 100644--- a/docs/building-methods/concepts/inline-structures.md+++ b/docs/building-methods/concepts/inline-structures.md@@ -267,6 +267,15 @@ lead = { type = "concept", concept_ref = "myapp.Employee", description = "Team lPipelex automatically handles concept dependencies. You can define concepts in any order—Pipelex loads them in the correct sequence based on their relationships.+A field may name a concept declared in the same bundle, in another bundle loaded into the same library (in the same load or an earlier one), or in a method package the bundle depends on, written with the package's address: `concept_ref = "github.com/acme/crm-methods/crm->crm.Customer"`. A method package is loaded when one of the bundle's pipes references it, so a field alone does not bring a package in. A concept in a method package sees only that package's own concepts.++Two things refuse the load rather than failing later, when the concept is first used:++- **A cycle between concepts**, such as `A` holding a `B` that holds an `A`, including a concept that holds itself.+- **A field naming a concept that cannot be found**, because no bundle the concept can see declares it. A concept declared only by a bundle loaded afterwards cannot be found either.++A method package's own dependencies are never loaded, so a concept in the package that names one of their concepts is not refused. It fails when first used, and so does any concept holding it.+## Choice Fields (Enums)For fields that should only accept specific values, use the `choices` property. When using `choices`, you don't need to specify a `type`.diff --git a/docs/features/validation-dry-run.md b/docs/features/validation-dry-run.mdindex 4bc251efe..42fb7cf4b 100644--- a/docs/features/validation-dry-run.md+++ b/docs/features/validation-dry-run.md@@ -16,7 +16,7 @@ Pipelex provides multiple layers of validation to catch issues before they costCheck pipeline syntax, structure, and compatibility without execution:- **Syntax validation** — Catch MTHDS language errors via plxt linting-- **Structure validation** — Resolve pipe and concept references in the loaded library, and verify each pipe's declared inputs against the concepts that operator accepts — plus a controller's declared output against what it actually produces (for a `PipeSequence`, its last step's output concept and multiplicity)+- **Structure validation** — Resolve pipe and concept references in the loaded library, refuse concepts whose structures form a cycle or whose fields name a concept that cannot be found, and verify each pipe's declared inputs against the concepts that operator accepts — plus a controller's declared output against what it actually produces (for a `PipeSequence`, its last step's output concept and multiplicity)- **Input validation** — Ensure required inputs are provided and correctly typed, and that every variable a step reads is bound by the time that step runs- **Declared inputs are read** — For `PipeLLM`, `PipeCompose`, `PipeSearch` and `PipeImgGen`, every variable a template reads must be declared in `inputs` (`missing_input_variable`), and every declared input must be read by one of the pipe's templates (`extraneous_input_variable`): the prompt and system prompt of a `PipeLLM`, the template or the construct of a `PipeCompose`, the prompt of a `PipeSearch`, the prompt and negative prompt of a `PipeImgGen`. An unread input is refused on the pipe that declares it, before any controller above it is asked to supply itdiff --git a/docs/under-the-hood/runtime-bridge-and-transport.md b/docs/under-the-hood/runtime-bridge-and-transport.mdindex 106c67b76..eaa078943 100644--- a/docs/under-the-hood/runtime-bridge-and-transport.md+++ b/docs/under-the-hood/runtime-bridge-and-transport.md@@ -77,7 +77,9 @@ Sending a pipe run to a separate worker process introduces three challenges thatThe submitter builds a `LibraryCrate` from its loaded library (`pipelex/libraries/library_crate_factory.py`) and attaches it to the `PipeJob`. On the worker, the job's entry point loads the crate into a scoped library, registering the bundle's dynamic concept classes, so `get_required_pipe()` resolves every pipe at every level of nesting. Because a distributed run can fan out into nested jobs (a controller's child pipes), every nested job carries the same crate; loading is idempotent via the fingerprint, so re-receiving a crate already loaded on a worker is a no-op.-A load ends by validating the whole library (`Library.validate_library`), which checks every pipe against the concepts and pipes it depends on. That check is pure — it raises or returns, and leaves nothing behind that a run later reads — and the crate a worker receives was built from a library the submitter already loaded and validated. So a worker running the same pipelex version as its submitter can call `load_from_crate(library_id=…, crate=…, is_crate_prevalidated=True)`, which skips that final validation and nothing else: fingerprint idempotency, domain and concept loading with the registration of the dynamic classes, the concept-cycle check, pipe construction with each pipe's static validation, and source tracking all still run. The precondition belongs to the host: pass it only where submitter and worker ship on the same pipelex, as a deployment that builds them from one lockfile does. A crate from anywhere else keeps the default, validating load.+A load ends by validating the whole library (`Library.validate_library`), which checks every pipe against the concepts and pipes it depends on. That check is pure — it raises or returns, and leaves nothing behind that a run later reads — and the crate a worker receives was built from a library the submitter already loaded and validated. So a worker running the same pipelex version as its submitter can call `load_from_crate(library_id=…, crate=…, is_crate_prevalidated=True)`, which skips that final validation and nothing else: fingerprint idempotency, domain and concept loading with the registration of the dynamic classes, the resolution of their references to one another, the concept-cycle check, pipe construction with each pipe's static validation, and source tracking all still run. The precondition belongs to the host: pass it only where submitter and worker ship on the same pipelex, as a deployment that builds them from one lockfile does. A crate from anywhere else keeps the default, validating load.++A crate carries no method package the library depends on, so a structure field typed by a dependency's concept cannot be resolved on the worker. A crate load therefore never refuses a structure class it could not complete, whichever the flag: the refusal of a field naming a concept that cannot be found belongs to the load from bundles, which resolves the dependencies first, and the submitter's library went through it. Such a class fails only if a run on the worker uses it, as would the dependency's own pipes, which the crate does not carry either.---diff --git a/pipelex/core/concepts/structure_generation/generator.py b/pipelex/core/concepts/structure_generation/generator.pyindex b22e5db54..b8f1a265c 100644--- a/pipelex/core/concepts/structure_generation/generator.py+++ b/pipelex/core/concepts/structure_generation/generator.py@@ -364,7 +364,8 @@ class StructureGenerator:The concept ref is already normalized by the resolved-field layer (natives flagged, barerefs promoted to the local domain). Emits:- - for native concepts: an import of the native content class and its class name;+ - for native concepts: an import of the native content class and its class name, or `Any` for+ `native.Anything`, which has no content class;- otherwise: a domain-qualified forward reference (resolved via model_rebuild at runtime)."""concept_ref = resolved_type.concept_ref@@ -382,6 +383,9 @@ class StructureGenerator:if structure_class:self.imports.add(f"from {structure_class.__module__} import {structure_class.__name__}")return structure_class.__name__+ # `native.Anything` has no content class: a field holding it holds any value, and a forward+ # reference to its code would name a class no namespace has.+ return "Any"concept_code = extract_concept_code_from_concept_ref_or_code(concept_ref)parsed_ref = QualifiedRef.parse_stripping_cross_package(concept_ref)diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.pyindex e564c66ee..d00eb0b6e 100644--- a/pipelex/libraries/library_manager.py+++ b/pipelex/libraries/library_manager.py@@ -3,7 +3,7 @@ from collections.abc import Generatorfrom contextlib import ExitStack, contextmanagerfrom graphlib import CycleError, TopologicalSorterfrom pathlib import Path-from typing import TYPE_CHECKING+from typing import TYPE_CHECKING, ForwardRef, Literal, get_args, get_originfrom kajson.class_registry import ClassRegistryfrom kajson.kajson_manager import KajsonManager@@ -21,6 +21,7 @@ from pipelex.cogt.exceptions import ModelChoiceNotFoundErrorfrom pipelex.config import is_pipe_func_sandbox_hostedfrom pipelex.core.concepts.concept_blueprint import ConceptBlueprintfrom pipelex.core.concepts.concept_factory import ConceptFactory+from pipelex.core.concepts.helpers import make_qualified_structure_class_namefrom pipelex.core.concepts.native.concept_native import NativeConceptCodefrom pipelex.core.domains.domain_blueprint import DomainBlueprintfrom pipelex.core.domains.domain_factory import DomainFactory@@ -60,7 +61,7 @@ from pipelex.system.registries.func_registry_utils import FuncRegistryUtilsfrom pipelex.tools.misc.semver import SemVerError, parse_constraint, parse_version, version_satisfiesif TYPE_CHECKING:- from collections.abc import Mapping+ from collections.abc import Iterable, Mappingfrom pipelex.core.concepts.concept import Conceptfrom pipelex.core.domains.domain import Domain@@ -118,6 +119,34 @@ def _dependency_bundle_source(*, package_address: str, package_root: Path, mthdsreturn f"{package_address}/{mthds_path.name}"+def _dependency_entries_first(*, concepts_by_key: "Mapping[str, Concept]") -> list["Concept"]:+ """A library's concepts with its dependencies' aliased entries (`alias->domain.Code`) before its own.++ The structure-class rebuild lets a later concept win a class name two concepts share, so this order is what makes+ a main-package concept win over a dependency's of the same `domain.Code`, whichever batch loaded each.+ """+ dependency_entries = [concept for key, concept in concepts_by_key.items() if QualifiedRef.has_cross_package_prefix(key)]+ own_entries = [concept for key, concept in concepts_by_key.items() if not QualifiedRef.has_cross_package_prefix(key)]+ return [*dependency_entries, *own_entries]+++def _forward_refs_missing_from(*, annotation: object, namespace: "Mapping[str, type]") -> list[str]:+ """The structure class names an annotation holds as forward references that the namespace lacks.++ The generator spells a field typed by a concept as a quoted name, `"domain__Code"`, or `"domain__Code | None"` for+ an optional one, alone or inside a `list[...]` or a `dict[...]`. A rebuild that fails leaves those names as they+ were written, a `ForwardRef` or a plain string, so a name is missing when the namespace has no entry for it.+ """+ if isinstance(annotation, ForwardRef):+ annotation = annotation.__forward_arg__+ if isinstance(annotation, str):+ names = [name.strip() for name in annotation.split("|")]+ return [name for name in names if name != "None" and name not in namespace]+ if get_origin(annotation) is Literal:+ return []+ return [name for arg in get_args(annotation) for name in _forward_refs_missing_from(annotation=arg, namespace=namespace)]++def _authored_model_field(*, step_role: StepRole) -> str:"""The field of the authored ``preliminary_text`` PipeLLM whose model a synthetic helper was given."""match step_role:@@ -537,7 +566,14 @@ class LibraryManager(LibraryManagerAbstract):Note: This method does NOT resolve cross-package address-based dependencies.Callers must handle dependency loading before calling this method (e.g. via_load_address_based_dependencies). The load_from_blueprints method does this- automatically before delegating here.+ automatically before delegating to the same load.++ A crate carries no dependency packages, so a crate loaded here, typically one transported to a worker+ or a sandbox, cannot tell a structure field naming a dependency's concept, which the transport left+ behind, from a field naming nothing. Its structure classes are rebuilt against what the library holds,+ and one still incomplete is left to fail at first use rather than refusing the load: the refusal+ belongs to load_from_blueprints, the load that resolves the bundle's dependencies, and the library a+ crate comes from went through it.Args:library_id: The library to load into@@ -547,13 +583,29 @@ class LibraryManager(LibraryManagerAbstract):crate, handed to the worker that executes it. Library validation is a pure check (it writesnothing a run later reads), so skipping it changes no behaviour, only the cost of reaching averdict already reached. Everything else still happens: fingerprint idempotency, domain and- concept loading with class registration, the concept-cycle check, pipe construction with each- pipe's static validation, and source tracking. Leave it False for any crate whose library was- not validated by this pipelex version.+ concept loading with class registration, the resolution of the structure classes' references+ to one another, the concept-cycle check, pipe construction with each pipe's static validation,+ and source tracking. Leave it False for any crate whose library was not validated by this+ pipelex version.Returns:List of all pipes that were loaded, or empty list if already loaded"""+ return self._load_crate(library_id=library_id, crate=crate, is_crate_prevalidated=is_crate_prevalidated, refuses_unresolved_structures=False)++ def _load_crate(+ self, *, library_id: str, crate: LibraryCrate, is_crate_prevalidated: bool, refuses_unresolved_structures: bool+ ) -> list[PipeAbstract]:+ """The load behind load_from_crate and load_from_blueprints.++ Args:+ library_id: The library to load into+ crate: The LibraryCrate to load+ is_crate_prevalidated: As load_from_crate describes it+ refuses_unresolved_structures: Whether a structure class still incomplete once the concepts are loaded+ refuses the load. True only where the load resolved the bundle's dependencies first, so that a+ reference which resolves to nothing is the bundle's own fault.+ """# Bind the target library as current for the whole load: PipeFactory and the concept# factories resolve concepts and the class registry through the ambient current library,# so a load into a non-current library must not read the caller's binding.@@ -585,11 +637,14 @@ class LibraryManager(LibraryManagerAbstract):all_concepts = self._load_concepts_from_crate(crate.concepts)library.concept_library.add_concepts(concepts=all_concepts)- # Resolve forward references in dynamically generated structure classes- self._rebuild_models_with_forward_refs(all_concepts)-- # Detect cycles in concept references (A -> B -> A is forbidden)- self._detect_concept_cycles(all_concepts)+ # A main-package concept's forward references may name any concept the library holds: this batch's,+ # an earlier batch's, or a loaded dependency's aliased entry.+ self._run_concept_stage(+ loaded_concepts=all_concepts,+ visible_concepts=_dependency_entries_first(concepts_by_key=library.concept_library.root),+ concept_sources=crate.source_map,+ refuses_unresolved_structures=refuses_unresolved_structures,+ )# Precompute domain -> concept local codes mapping. VESTIGIAL on this path — see the note# on the qualification pass below: every io ref reaching PipeFactory is dotted, and the@@ -697,8 +752,9 @@ class LibraryManager(LibraryManagerAbstract):already_loaded_concept_refs=set(library.concept_library.root.keys()),)- # Load from crate (domains, concepts, pipes, validation)- all_pipes = self.load_from_crate(library_id=library_id, crate=crate)+ # Load from crate (domains, concepts, pipes, validation). The dependencies were loaded above, so a+ # structure reference that still resolves to nothing is the bundle's own, and refuses the load.+ all_pipes = self._load_crate(library_id=library_id, crate=crate, is_crate_prevalidated=False, refuses_unresolved_structures=True)# Also record the aggregate crate fingerprint: get_crate() rebuilds one crate from# ALL accumulated blueprints, so once the library holds more than one batch its@@ -719,7 +775,7 @@ class LibraryManager(LibraryManagerAbstract):Concepts that refine other concepts are loaded after their base concepts,ensuring the base class is always registered before the refining class.Forward references between concepts (for structure fields) are resolved- later by _rebuild_models_with_forward_refs().+ later by _run_concept_stage().Args:blueprints: List of parsed MTHDS blueprints to load@@ -1170,9 +1226,16 @@ class LibraryManager(LibraryManagerAbstract):all_domains.append(domain)child_library.domain_library.add_domains(domains=all_domains)- # Load concepts into child library+ # Load concepts into child library. A dependency's forward references may name its own concepts and+ # nothing else: it cannot name its consumer, its own dependencies are not loaded, and a wider namespace would+ # let a reference to a domain it does not declare bind silently to a consumer's class of that name. Because its+ # own dependencies are not loaded, a structure class still incomplete does not refuse the consumer's load: it+ # may name a concept of one of them, which is no fault of the package, and it fails at first use instead.dep_concepts = self._load_concepts_from_blueprints(dep_blueprints)child_library.concept_library.add_concepts(concepts=dep_concepts)+ self._run_concept_stage(+ loaded_concepts=dep_concepts, visible_concepts=dep_concepts, concept_sources=crate.source_map, refuses_unresolved_structures=False+ )# Collect main_pipes for auto-exportmain_pipes: set[str] = set()@@ -1408,38 +1471,123 @@ class LibraryManager(LibraryManagerAbstract):for blueprint in blueprints:self._remove_from_blueprint(library_id=library_id, blueprint=blueprint)- def _rebuild_models_with_forward_refs(self, concepts: list["Concept"]) -> None:- """Rebuild Pydantic models to resolve forward references.+ def _run_concept_stage(+ self,+ *,+ loaded_concepts: list["Concept"],+ visible_concepts: "Iterable[Concept]",+ concept_sources: "Mapping[str, str]",+ refuses_unresolved_structures: bool,+ ) -> None:+ """The steps that follow adding a batch of concepts to a library, shared by both load paths.- When dynamically generated classes have forward references (e.g., `customer: "Customer"`),- Python's get_type_hints() cannot resolve them because the referenced classes are not- in any accessible namespace. This method builds a namespace with all structure classes- and calls model_rebuild() to resolve the forward references.+ The crate load and `_load_single_dependency` both call this, and nothing else does: the dependency loader+ once hand-copied these steps and dropped two of them, so a dependency's forward references were never+ resolved and its concept cycles never checked. The order is load-bearing. The cycle check reads the resolved+ field annotations, and an unresolved forward reference has no `__name__`, which is how an unrebuilt reference+ hides from it, so the rebuild and its refusal come first.Args:- concepts: List of concepts that were just loaded+ loaded_concepts: The concepts this batch loaded, whose structure classes are rebuilt and checked+ visible_concepts: Every concept those classes' forward references may name, the loaded ones included;+ where two share a class name, the later one wins+ concept_sources: concept_ref -> the bundle that declared it, for the refusal's message+ refuses_unresolved_structures: Whether a generated structure class still incomplete refuses the load"""- # Build namespace with all structure class names- namespace: dict[str, type] = {}- class_registry = get_class_registry()+ self._rebuild_structure_classes(+ loaded_concepts=loaded_concepts,+ visible_concepts=visible_concepts,+ concept_sources=concept_sources,+ refuses_unresolved_structures=refuses_unresolved_structures,+ )+ self._detect_concept_cycles(loaded_concepts, concept_sources=concept_sources)- for concept in concepts:- structure_class = class_registry.get_class(name=concept.structure_class_name)- if structure_class is not None:+ def _rebuild_structure_classes(+ self,+ *,+ loaded_concepts: list["Concept"],+ visible_concepts: "Iterable[Concept]",+ concept_sources: "Mapping[str, str]",+ refuses_unresolved_structures: bool,+ ) -> None:+ """Resolve the forward references of the loaded concepts' structure classes, and refuse any left unresolved.++ A generated structure class names another concept's class as a forward reference, spelled from the concept+ ref whatever class the concept actually has (`"invented_notes__Note"`), and pydantic resolves it only when the+ class is rebuilt against a namespace holding that name. So the namespace keys every visible concept's class+ both by its own name and by that spelling, which differ for a concept backed by a Python class+ (`structure = "CustomerPayload"`). The loaded concepts' classes are keyed by their bare concept code too: the+ generator emits a bare code only for a ref with no domain, and a bare code is ambiguous across domains, so+ that key must not widen with the rest.++ Where two visible concepts share a key, the later one wins, and the callers order the main package after its+ dependencies. A dependency concept and a main-package concept with the same `domain.Code` share that key, and+ the main package's class is the one a reference resolves to. That clash is L-260929-0584cc's.++ A generated class whose own fields name a class the namespace lacks refuses the load, when the caller asks for+ it, instead of loading and failing at first use. A class whose own fields all resolve can still be incomplete,+ because pydantic builds a held class that is incomplete inline, and the held class is the one to blame: the+ load of a dependency's class or of a Python class chose not to refuse it, so the holder is left for first use+ too, and a generated class of this batch is refused on its own. Only generated classes are checked, and a+ generated class is recognised by its name: the concept factory names every class it generates+ `make_qualified_structure_class_name(domain, code)`, while a class registered from Python for a concept keeps+ its own name. Such a class, like a native one, is left to its own module's resolution: its rebuild failing is+ only logged, as before.+ """+ class_registry = get_class_registry()+ namespace: dict[str, type] = {}+ for concept in [*visible_concepts, *loaded_concepts]:+ if structure_class := class_registry.get_class(name=concept.structure_class_name):namespace[concept.structure_class_name] = structure_class- # Also add by concept code in case the forward ref uses the code+ namespace[make_qualified_structure_class_name(domain_code=concept.domain_code, concept_code=concept.code)] = structure_class+ for concept in loaded_concepts:+ if structure_class := class_registry.get_class(name=concept.structure_class_name):namespace[concept.code] = structure_class- # Rebuild each model with the shared namespace- for concept in concepts:+ refusals: list[str] = []+ for concept in loaded_concepts:structure_class = class_registry.get_class(name=concept.structure_class_name)- if structure_class is not None and issubclass(structure_class, BaseModel):- try:- structure_class.model_rebuild(_types_namespace=namespace)- except (NameError, PydanticUserError) as exc:- log.debug(f"Could not rebuild model for {concept.concept_ref}: {exc}")+ if structure_class is None or not issubclass(structure_class, BaseModel):+ continue+ rebuild_error: NameError | PydanticUserError | None = None+ try:+ structure_class.model_rebuild(_types_namespace=namespace)+ except (NameError, PydanticUserError) as exc:+ rebuild_error = exc+ if structure_class.__pydantic_complete__:+ continue+ is_generated = concept.structure_class_name == make_qualified_structure_class_name(+ domain_code=concept.domain_code, concept_code=concept.code+ )+ missing_names = sorted(+ {+ name+ for field_info in structure_class.model_fields.values()+ for name in _forward_refs_missing_from(annotation=field_info.annotation, namespace=namespace)+ }+ )+ if not (is_generated and refuses_unresolved_structures and missing_names):+ log.debug(f"The structure class of {concept.concept_ref} stays incomplete, and will fail at first use: {rebuild_error}")+ continue+ source = concept_sources.get(concept.concept_ref)+ where = f" (declared in '{source}')" if source else ""+ quoted_names = ", ".join(f"'{name}'" for name in missing_names)+ refusals.append(+ f"Concept '{concept.concept_ref}'{where} cannot be built: its structure names the class {quoted_names}, "+ "and no concept it can see has it."+ if len(missing_names) == 1+ else f"Concept '{concept.concept_ref}'{where} cannot be built: its structure names the classes {quoted_names}, "+ "and no concept it can see has them."+ )++ if refusals:+ refusals.append(+ "A structure field may name a concept declared by a bundle already loaded, or by a method package that "+ "one of the bundle's pipes references (`address->domain.Concept`)."+ )+ raise LibraryLoadingError(" ".join(refusals))- def _detect_concept_cycles(self, concepts: list["Concept"]) -> None:+ def _detect_concept_cycles(self, concepts: list["Concept"], *, concept_sources: "Mapping[str, str]") -> None:"""Detect cycles in concept references and raise an error if found.Cycles like A -> B -> A are forbidden because they create infinite recursion@@ -1448,6 +1596,8 @@ class LibraryManager(LibraryManagerAbstract):Args:concepts: List of concepts to check for cycles+ concept_sources: concept_ref -> the bundle that declared it, which the refusal names, since a method+ package's refs carry no address and would not say which package the cycle is in"""# TODO: Refactor to inspect ConceptStructureBlueprint directly (concept_ref and item_concept_ref fields)# instead of the generated Python types. This would be more direct and wouldn't depend on how types@@ -1511,7 +1661,9 @@ class LibraryManager(LibraryManagerAbstract):cycle_start = path.index(concept_ref)cycle = [*path[cycle_start:], concept_ref]cycle_str = " -> ".join(cycle)- msg = f"Cycle detected in concept references: {cycle_str}"+ source = concept_sources.get(concept_ref)+ where = f" (declared in '{source}')" if source else ""+ msg = f"Cycle detected in concept references: {cycle_str}{where}"raise LibraryLoadingError(msg)# Find the concept by refdiff --git a/subject_grants.toml b/subject_grants.tomlindex 652d710df..740e9d544 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -3812,10 +3812,6 @@ rationale = "Verb-object: loads the dependency library object; context stays keyparam = "library_id"rationale = "Verb-object: pops and tears down the library named by the id; single operand."-["pipelex/libraries/library_manager.py::LibraryManager._rebuild_models_with_forward_refs"]-param = "concepts"-rationale = "Verb-object: rebuilds the given concepts' models; single operand."-["pipelex/libraries/library_manager.py::LibraryManager._remove_concepts_from_blueprint"]param = "blueprint"rationale = "Verb-object: strips concepts from the blueprint; single operand."diff --git a/tests/integration/pipelex/libraries/test_class_backed_concept_references.py b/tests/integration/pipelex/libraries/test_class_backed_concept_references.pynew file mode 100644index 000000000..2397e9a7c--- /dev/null+++ b/tests/integration/pipelex/libraries/test_class_backed_concept_references.py@@ -0,0 +1,87 @@+"""A structure field naming a concept backed by a Python class resolves to that class.++The generator spells a field's forward reference from the concept ref (`shop__Customer`), whatever class the+concept actually has, so the rebuild must find a class-backed concept under that spelling as well as under the+class's own name. And a Python class whose own rebuild fails is left to its module, never refusing the load.+"""++from collections.abc import Callable+from pathlib import Path++from pipelex.interpreter_hub import get_concept_library, get_library_manager++CUSTOMER_PAYLOAD_PY = """+from pipelex.core.stuffs.structured_content import StructuredContent+++class InventedCustomerPayload(StructuredContent):+ name: str+"""++CLASS_BACKED_MTHDS = """+domain = "invented_shop"+description = "A concept backed by a Python class, held by an inline structure"++[concept.Customer]+description = "A customer"+structure = "InventedCustomerPayload"++[concept.Order]+description = "An order"++[concept.Order.structure]+customer = { type = "concept", concept_ref = "invented_shop.Customer", description = "The customer", required = true }+"""++# `weird` is typed by a plain class pydantic cannot build a schema for, so rebuilding it raises a+# `PydanticSchemaGenerationError`; the class was unusable before the rebuild, and stays the module's concern.+UNBUILDABLE_PY = """+from pipelex.core.stuffs.structured_content import StructuredContent+++class InventedOpaqueThing:+ pass+++class InventedUnbuildableRecord(StructuredContent):+ order: "invented_ledger__Entry | None" = None+ weird: InventedOpaqueThing | None = None+"""++UNBUILDABLE_MTHDS = """+domain = "invented_ledger"+description = "A concept backed by a Python class whose rebuild fails"++[concept.Record]+description = "A record"+structure = "InventedUnbuildableRecord"++[concept.Entry]+description = "An entry"++[concept.Entry.structure]+number = { type = "text", description = "The entry's number", required = true }+"""+++class TestClassBackedConceptReferences:+ def test_field_naming_a_class_backed_concept_is_fully_defined(self, tmp_path: Path, load_empty_library: Callable[[], str]):+ (tmp_path / "invented_customer_payload.py").write_text(CUSTOMER_PAYLOAD_PY, encoding="utf-8")+ (tmp_path / "shop.mthds").write_text(CLASS_BACKED_MTHDS, encoding="utf-8")++ library_id = load_empty_library()+ get_library_manager().load_libraries(library_id=library_id, library_dirs=[tmp_path])++ concept_library = get_concept_library()+ order_class = concept_library.get_structure_class(concept=concept_library.get_required_concept("invented_shop.Order"))+ order = order_class.model_validate({"customer": {"name": "Invented customer"}})+ assert order.model_dump()["customer"] == {"name": "Invented customer"}++ def test_python_class_whose_rebuild_fails_does_not_refuse_the_load(self, tmp_path: Path, load_empty_library: Callable[[], str]):+ (tmp_path / "invented_unbuildable_record.py").write_text(UNBUILDABLE_PY, encoding="utf-8")+ (tmp_path / "ledger.mthds").write_text(UNBUILDABLE_MTHDS, encoding="utf-8")++ library_id = load_empty_library()+ get_library_manager().load_libraries(library_id=library_id, library_dirs=[tmp_path])++ assert get_concept_library().get_required_concept("invented_ledger.Record").structure_class_name == "InventedUnbuildableRecord"diff --git a/tests/integration/pipelex/libraries/test_concept_to_concept_references.py b/tests/integration/pipelex/libraries/test_concept_to_concept_references.pyindex 9502e5c45..6fb4afb49 100644--- a/tests/integration/pipelex/libraries/test_concept_to_concept_references.py+++ b/tests/integration/pipelex/libraries/test_concept_to_concept_references.py@@ -7,6 +7,7 @@ from pathlib import Pathimport pytestfrom pipelex.interpreter_hub import get_concept_library, get_library_manager+from pipelex.libraries.exceptions import LibraryLoadingErrorclass TestConceptToConceptReferences:@@ -455,3 +456,111 @@ amount = { type = "number", description = "Invoice amount" }assert customer_concept is not Noneassert invoice_concept is not None++ def test_reference_into_an_earlier_load_batch(self, load_empty_library: Callable[[], str]):+ """A concept loaded in a second batch, holding a list of a concept from the first batch, is fully defined."""+ notes_mthds = """+domain = "invented_notes"+description = "The first batch"++[concept.Note]+description = "One invented note"++[concept.Note.structure]+title = { type = "text", description = "The note's title", required = true }+"""++ search_mthds = """+domain = "invented_search"+description = "The second batch, into the same library"++[concept.NoteSearch]+description = "The notes a search found"++[concept.NoteSearch.structure]+notes = { type = "list", item_type = "concept", item_concept_ref = "invented_notes.Note", description = "The notes found", required = true }+"""++ with tempfile.TemporaryDirectory() as notes_dir, tempfile.TemporaryDirectory() as search_dir:+ (Path(notes_dir) / "notes.mthds").write_text(notes_mthds, encoding="utf-8")+ (Path(search_dir) / "search.mthds").write_text(search_mthds, encoding="utf-8")++ library_id = load_empty_library()+ library_manager = get_library_manager()+ library_manager.load_libraries(library_id=library_id, library_dirs=[Path(notes_dir)])+ library_manager.load_libraries(library_id=library_id, library_dirs=[Path(search_dir)])++ note_search_concept = get_concept_library().get_required_concept("invented_search.NoteSearch")+ note_search_class = get_concept_library().get_structure_class(concept=note_search_concept)++ note_search = note_search_class.model_validate({"notes": [{"title": "Invented note one"}]})+ assert note_search.model_dump()["notes"] == [{"title": "Invented note one"}]++ def test_reference_to_a_concept_no_bundle_declares_is_refused(self, load_empty_library: Callable[[], str]):+ """A field naming a concept that no loaded bundle declares refuses the load, naming the concept and the missing class.++ `NoteDigest` only holds `NoteSearch`, whose class cannot be built, so the refusal names `NoteSearch` alone, and+ `NoteSearch`'s optional field naming `Query`, which is declared, is not named either.+ """+ mthds_content = """+domain = "invented_search"+description = "A concept holding notes from a domain nothing declares"++[concept.NoteSearch]+description = "The notes a search found"++[concept.NoteSearch.structure]+notes = { type = "list", item_type = "concept", item_concept_ref = "invented_elsewhere.Note", description = "The notes found", required = true }+query = { type = "concept", concept_ref = "invented_search.Query", description = "The query searched for" }++[concept.Query]+description = "A search query"++[concept.Query.structure]+text = { type = "text", description = "The query's text", required = true }++[concept.NoteDigest]+description = "A digest of a search"++[concept.NoteDigest.structure]+search = { type = "concept", concept_ref = "invented_search.NoteSearch", description = "The search digested" }+"""++ with tempfile.TemporaryDirectory() as tmp_dir:+ (Path(tmp_dir) / "search.mthds").write_text(mthds_content, encoding="utf-8")++ library_id = load_empty_library()+ library_manager = get_library_manager()++ with pytest.raises(LibraryLoadingError) as exc_info:+ library_manager.load_libraries(library_id=library_id, library_dirs=[Path(tmp_dir)])++ message = str(exc_info.value)+ assert "invented_search.NoteSearch" in message+ assert "invented_elsewhere__Note" in message+ assert "invented_search.NoteDigest" not in message+ assert "invented_search__Query" not in message++ def test_field_typed_anything_holds_any_value(self, load_empty_library: Callable[[], str]):+ """`native.Anything` has no content class, so a field typed by it is spelled `Any`, not a reference nothing resolves."""+ mthds_content = """+domain = "invented_box"+description = "A box holding anything"++[concept.Holder]+description = "A box holding one value of any kind"++[concept.Holder.structure]+payload = { type = "concept", concept_ref = "native.Anything", description = "The value held", required = true }+"""++ with tempfile.TemporaryDirectory() as tmp_dir:+ (Path(tmp_dir) / "box.mthds").write_text(mthds_content, encoding="utf-8")++ library_id = load_empty_library()+ get_library_manager().load_libraries(library_id=library_id, library_dirs=[Path(tmp_dir)])++ holder_concept = get_concept_library().get_required_concept("invented_box.Holder")+ holder_class = get_concept_library().get_structure_class(concept=holder_concept)+ holder = holder_class.model_validate({"payload": {"invented_key": [1, 2]}})+ assert holder.model_dump()["payload"] == {"invented_key": [1, 2]}diff --git a/tests/integration/pipelex/libraries/test_data.py b/tests/integration/pipelex/libraries/test_data.pynew file mode 100644index 000000000..cce488a16--- /dev/null+++ b/tests/integration/pipelex/libraries/test_data.py@@ -0,0 +1,99 @@+from typing import ClassVar+++class InstalledNotesPackageTestData:+ """A consumer bundle and an installed method package it references by address; nothing here is real."""++ METHOD_NAME: ClassVar[str] = "notes"+ DEP_ALIAS: ClassVar[str] = "github.com/invented/notes-lib/notes"++ DEP_MANIFEST: ClassVar[str] = """[package]+name = "notes"+address = "github.com/invented/notes-lib"+version = "1.0.0"+description = "An invented notes library"++[exports.invented_notes]+pipes = ["find_notes"]+"""++ #: The dependency's `NoteSearch` holds a list of its `Note`.+ DEP_BUNDLE: ClassVar[str] = """domain = "invented_notes"+description = "An invented notes library"++[concept.Note]+description = "One invented note"++[concept.Note.structure]+title = { type = "text", description = "The note's title", required = true }++[concept.NoteSearch]+description = "The notes a search found"++[concept.NoteSearch.structure]+notes = { type = "list", item_type = "concept", item_concept_ref = "invented_notes.Note", description = "The notes found", required = true }++[pipe.find_notes]+type = "PipeLLM"+description = "Find the notes about a query"+inputs = { query = "Text" }+output = "NoteSearch"+prompt = "Find the notes about $query."+"""++ #: The same package, whose `Note` also names a concept of the package's own dependency, which is never loaded.+ DEP_BUNDLE_NAMING_ITS_OWN_DEPENDENCY: ClassVar[str] = """domain = "invented_notes"+description = "An invented notes library that depends on another package"++[concept.Note]+description = "One invented note, citing a source from another package"++[concept.Note.structure]+title = { type = "text", description = "The note's title", required = true }+source = { type = "concept", concept_ref = "github.com/invented/sources-lib/sources->invented_sources.Source", description = "Where it came from" }++[concept.NoteSearch]+description = "The notes a search found"++[concept.NoteSearch.structure]+notes = { type = "list", item_type = "concept", item_concept_ref = "invented_notes.Note", description = "The notes found", required = true }++[pipe.find_notes]+type = "PipeLLM"+description = "Find the notes about a query"+inputs = { query = "Text" }+output = "NoteSearch"+prompt = "Find the notes about $query."+"""++ #: The consumer's `Digest` has a field typed by the dependency's `Note`, and a field with choices, whose strings name+ #: no class. Its sequence steps into the dependency's pipe, which is what makes the loader discover the package: a+ #: concept ref alone does not (L-260929-9c8eac).+ CONSUMER_BUNDLE: ClassVar[str] = """domain = "invented_consumer"+description = "A consumer of the invented notes library"++[concept.Digest]+description = "A digest built around one note"++[concept.Digest.structure]+note = { type = "concept", concept_ref = "github.com/invented/notes-lib/notes->invented_notes.Note", description = "The note", required = true }+summary = { type = "text", description = "What the note says", required = true }+tone = { choices = ["brief", "detailed"], description = "How long the digest runs" }++[pipe.digest_notes]+type = "PipeSequence"+description = "Find the invented notes about a query, then digest them"+inputs = { query = "Text" }+output = "Digest"+steps = [+ { pipe = "github.com/invented/notes-lib/notes->invented_notes.find_notes", result = "found_notes" },+ { pipe = "write_digest", result = "digest" },+]++[pipe.write_digest]+type = "PipeLLM"+description = "Digest the notes found about a query"+inputs = { query = "Text" }+output = "Digest"+prompt = "Digest the notes about $query."+"""diff --git a/tests/integration/pipelex/libraries/test_dependency_concept_field_refs.py b/tests/integration/pipelex/libraries/test_dependency_concept_field_refs.pynew file mode 100644index 000000000..617b4d405--- /dev/null+++ b/tests/integration/pipelex/libraries/test_dependency_concept_field_refs.py@@ -0,0 +1,100 @@+"""A consumer's structure field typed by an installed method package's concept resolves to that package's class.++The forward reference a consumer's generated class holds names the dependency's class, which the main path's+concept stage used to leave out of its namespace, so the consumer's class loaded "not fully defined". A crate of the+consumer's library carries no dependency, so a worker loading it cannot resolve that field, and must not refuse it.+"""++from pathlib import Path++from pytest_mock import MockerFixture++from pipelex.interpreter_hub import get_concept_library, get_library_manager, scoped_current_library+from tests.integration.pipelex.libraries.test_data import InstalledNotesPackageTestData+++def _write_consumer_and_its_dependency(*, root: Path, dep_bundle: str = InstalledNotesPackageTestData.DEP_BUNDLE) -> Path:+ """Install the notes package under `.mthds/methods/` beside the consumer's bundle, and return the consumer's bundle file."""+ dep_dir = root / ".mthds" / "methods" / InstalledNotesPackageTestData.METHOD_NAME+ dep_dir.mkdir(parents=True)+ (dep_dir / "METHODS.toml").write_text(InstalledNotesPackageTestData.DEP_MANIFEST, encoding="utf-8")+ (dep_dir / "invented_notes.mthds").write_text(dep_bundle, encoding="utf-8")+ consumer_bundle_file = root / "invented_consumer.mthds"+ consumer_bundle_file.write_text(InstalledNotesPackageTestData.CONSUMER_BUNDLE, encoding="utf-8")+ return consumer_bundle_file+++class TestDependencyConceptFieldRefs:+ def test_consumer_field_typed_by_a_dependency_concept_is_fully_defined(self, tmp_path: Path, mocker: MockerFixture) -> None:+ """The consumer's `Digest` resolves its field to the dependency's `Note`, and the dependency's `NoteSearch` its list."""+ mocker.patch("pipelex.cli.installed_methods.GLOBAL_METHODS_DIR", tmp_path / "global-methods")+ mocker.patch("pipelex.cli.installed_methods.PROJECT_METHODS_DIR", tmp_path / "project-methods")+ mocker.patch("pipelex.methods.fetch_on_miss.is_method_fetch_on_miss_enabled", return_value=False)+ consumer_bundle_file = _write_consumer_and_its_dependency(root=tmp_path)+ library_manager = get_library_manager()+ library_id, library = library_manager.open_library()+ try:+ library_manager.load_libraries(library_id=library_id, library_file_paths=[consumer_bundle_file])++ assert InstalledNotesPackageTestData.DEP_ALIAS in library.dependency_libraries+ with scoped_current_library(library_id=library_id):+ concept_library = get_concept_library()+ digest_class = concept_library.get_structure_class(concept=concept_library.get_required_concept("invented_consumer.Digest"))+ note_search_class = concept_library.get_structure_class(+ concept=concept_library.get_required_concept(f"{InstalledNotesPackageTestData.DEP_ALIAS}->invented_notes.NoteSearch")+ )++ digest = digest_class.model_validate({"note": {"title": "Invented note one"}, "summary": "It says one thing"})+ note_search = note_search_class.model_validate({"notes": [{"title": "Invented note one"}]})+ finally:+ library_manager.teardown(library_id=library_id)++ assert digest.model_dump()["note"] == {"title": "Invented note one"}+ assert note_search.model_dump()["notes"] == [{"title": "Invented note one"}]++ def test_a_gap_in_the_dependency_s_own_dependencies_does_not_refuse_the_consumer(self, tmp_path: Path, mocker: MockerFixture) -> None:+ """The dependency's `Note` names a package the dependency's load never loads: the consumer's `Digest`, which holds+ that `Note`, inherits the gap and is left for first use rather than refusing the consumer's load.+ """+ mocker.patch("pipelex.cli.installed_methods.GLOBAL_METHODS_DIR", tmp_path / "global-methods")+ mocker.patch("pipelex.cli.installed_methods.PROJECT_METHODS_DIR", tmp_path / "project-methods")+ mocker.patch("pipelex.methods.fetch_on_miss.is_method_fetch_on_miss_enabled", return_value=False)+ consumer_bundle_file = _write_consumer_and_its_dependency(+ root=tmp_path, dep_bundle=InstalledNotesPackageTestData.DEP_BUNDLE_NAMING_ITS_OWN_DEPENDENCY+ )+ library_manager = get_library_manager()+ library_id, library = library_manager.open_library()+ try:+ library_manager.load_libraries(library_id=library_id, library_file_paths=[consumer_bundle_file])++ with scoped_current_library(library_id=library_id):+ concept_library = get_concept_library()+ digest_class = concept_library.get_structure_class(concept=concept_library.get_required_concept("invented_consumer.Digest"))+ pipe_codes = set(library.pipe_library.get_pipes_dict())+ finally:+ library_manager.teardown(library_id=library_id)++ assert "invented_consumer.write_digest" in pipe_codes+ assert not digest_class.__pydantic_complete__++ def test_a_crate_of_the_consumer_loads_without_its_dependency(self, tmp_path: Path, mocker: MockerFixture) -> None:+ """A worker loading the consumer's crate, which carries no dependency, loads it rather than refusing the field it cannot resolve."""+ mocker.patch("pipelex.cli.installed_methods.GLOBAL_METHODS_DIR", tmp_path / "global-methods")+ mocker.patch("pipelex.cli.installed_methods.PROJECT_METHODS_DIR", tmp_path / "project-methods")+ mocker.patch("pipelex.methods.fetch_on_miss.is_method_fetch_on_miss_enabled", return_value=False)+ consumer_bundle_file = _write_consumer_and_its_dependency(root=tmp_path)+ library_manager = get_library_manager()+ library_id, _ = library_manager.open_library()+ worker_library_id, worker_library = library_manager.open_library()+ try:+ library_manager.load_libraries(library_id=library_id, library_file_paths=[consumer_bundle_file])+ crate = library_manager.get_crate(library_id=library_id)+ assert crate is not None++ library_manager.load_from_crate(library_id=worker_library_id, crate=crate)++ assert "invented_consumer.write_digest" in worker_library.pipe_library.get_pipes_dict()+ assert not worker_library.dependency_libraries+ finally:+ library_manager.teardown(library_id=worker_library_id)+ library_manager.teardown(library_id=library_id)diff --git a/tests/unit/pipelex/codegen/test_ts_zod_wire_agreement.py b/tests/unit/pipelex/codegen/test_ts_zod_wire_agreement.pyindex d43cc91d2..40a4b1022 100644--- a/tests/unit/pipelex/codegen/test_ts_zod_wire_agreement.py+++ b/tests/unit/pipelex/codegen/test_ts_zod_wire_agreement.py@@ -103,7 +103,7 @@ class TestTsZodWireAgreement:classes[concept.structure_class_name] = registry.get_required_subclass(name=concept.structure_class_name, base_class=StuffContent)classes[code] = classes[concept.structure_class_name]# A concept reference is generated as a quoted forward ref, so the classes need the same rebuild- # against a shared namespace the library manager performs on load (`_rebuild_models_with_forward_refs`).+ # against a shared namespace the library manager performs on load (`_rebuild_structure_classes`).for structure_class in dict.fromkeys(classes.values()):structure_class.model_rebuild(_types_namespace=classes)diff --git a/tests/unit/pipelex/libraries/test_dependency_structure_classes.py b/tests/unit/pipelex/libraries/test_dependency_structure_classes.pynew file mode 100644index 000000000..4eebc5bbc--- /dev/null+++ b/tests/unit/pipelex/libraries/test_dependency_structure_classes.py@@ -0,0 +1,159 @@+"""A dependency package's structure classes go through the same concept stage as the main package's.++A concept whose structure holds another concept gets a generated class with a forward reference to the other+concept's class, which the loader resolves after loading the concepts. The dependency loader used to skip that+step, and the concept-cycle check with it: a dependency concept holding a list of another of its concepts loaded+"not fully defined" and failed at first use, and a dependency whose concepts formed a cycle loaded without complaint.+"""++from pathlib import Path++import pytest+from mthds.package.dependency_resolver import ResolvedDependency+from mthds.package.manifest.schema import MethodsManifest++from pipelex.interpreter_hub import get_library_manager, scoped_current_library+from pipelex.libraries.exceptions import LibraryLoadingError+from pipelex.libraries.library_manager import LibraryManager+from pipelex.system.registries.class_registry_access import get_class_registry++DEP_ALIAS = "invented_notes_dep"+DEP_ADDRESS = "github.com/invented/notes-lib"++# `NoteSearch` holds a list of `Note`, a forward reference the concept stage must resolve.+DEP_NOTES_MTHDS = """\+domain = "invented_notes"+description = "An invented notes library"++[concept.Note]+description = "One invented note"++[concept.Note.structure]+title = { type = "text", description = "The note's title", required = true }++[concept.NoteSearch]+description = "The notes a search found"++[concept.NoteSearch.structure]+notes = { type = "list", item_type = "concept", item_concept_ref = "invented_notes.Note", description = "The notes found", required = true }+"""++# The same two concepts, with `Note` pointing back at `NoteSearch`: a cycle the main path refuses.+DEP_CYCLIC_NOTES_MTHDS = """\+domain = "invented_notes"+description = "An invented notes library whose concepts form a cycle"++[concept.Note]+description = "One invented note, which points back at the search that found it"++[concept.Note.structure]+title = { type = "text", description = "The note's title", required = true }+found_by = { type = "concept", concept_ref = "invented_notes.NoteSearch", description = "The search that found it" }++[concept.NoteSearch]+description = "The notes a search found"++[concept.NoteSearch.structure]+notes = { type = "list", item_type = "concept", item_concept_ref = "invented_notes.Note", description = "The notes found", required = true }+"""++# `Note` holds a concept of the package's own dependency, which a dependency load never loads.+DEP_WITH_ITS_OWN_DEPENDENCY_MTHDS = """\+domain = "invented_notes"+description = "An invented notes library that depends on another package"++[concept.Note]+description = "One invented note, citing a source from another package"++[concept.Note.structure]+title = { type = "text", description = "The note's title", required = true }+source = { type = "concept", concept_ref = "github.com/invented/sources-lib/sources->invented_sources.Source", description = "Where it came from" }++[concept.Tag]+description = "A tag"++[concept.Tag.structure]+label = { type = "text", description = "The tag's label", required = true }+"""+++def _get_library_manager() -> LibraryManager:+ """The hub's manager, whose libraries the class registry resolves against, so each test's classes die with its library."""+ library_manager = get_library_manager()+ assert isinstance(library_manager, LibraryManager)+ return library_manager+++def _make_resolved_dep(*, tmp_path: Path, mthds_content: str) -> ResolvedDependency:+ mthds_file = tmp_path / "notes.mthds"+ mthds_file.write_text(mthds_content, encoding="utf-8")+ return ResolvedDependency(+ alias=DEP_ALIAS,+ address=DEP_ADDRESS,+ manifest=MethodsManifest(address=DEP_ADDRESS, version="1.0.0", description="An invented notes library"),+ package_root=tmp_path,+ mthds_files=[mthds_file],+ # None => no export filter, every pipe is public.+ exported_pipe_codes=None,+ )+++class TestDependencyStructureClasses:+ def test_a_concept_holding_a_list_of_another_is_fully_defined(self, tmp_path: Path) -> None:+ """A dependency's `NoteSearch` resolves its forward reference to `Note`, so it validates and renders a schema."""+ library_manager = _get_library_manager()+ library_id, library = library_manager.open_library()+ try:+ with scoped_current_library(library_id=library_id):+ library_manager._load_single_dependency( # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+ library=library,+ resolved_dep=_make_resolved_dep(tmp_path=tmp_path, mthds_content=DEP_NOTES_MTHDS),+ package_address=DEP_ADDRESS,+ )+ note_search_concept = library.dependency_libraries[DEP_ALIAS].concept_library.get_required_concept(+ concept_ref="invented_notes.NoteSearch"+ )+ note_search_class = get_class_registry().get_required_base_model(name=note_search_concept.structure_class_name)++ note_search = note_search_class.model_validate({"notes": [{"title": "Invented note one"}]})+ schema = note_search_class.model_json_schema()+ finally:+ library_manager.teardown(library_id=library_id)++ assert note_search_class.__pydantic_complete__+ assert note_search.model_dump()["notes"] == [{"title": "Invented note one"}]+ assert "notes" in schema["properties"]++ def test_concepts_forming_a_cycle_are_refused(self, tmp_path: Path) -> None:+ """A dependency whose concepts form a cycle is refused with the message the main path gives, naming the package's bundle."""+ library_manager = _get_library_manager()+ library_id, library = library_manager.open_library()+ cycle_refusal = r"Cycle detected in concept references: .* \(declared in 'github\.com/invented/notes-lib/notes\.mthds'\)"+ try:+ with scoped_current_library(library_id=library_id), pytest.raises(LibraryLoadingError, match=cycle_refusal):+ library_manager._load_single_dependency( # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+ library=library,+ resolved_dep=_make_resolved_dep(tmp_path=tmp_path, mthds_content=DEP_CYCLIC_NOTES_MTHDS),+ package_address=DEP_ADDRESS,+ )+ finally:+ library_manager.teardown(library_id=library_id)++ def test_a_concept_naming_its_own_dependency_does_not_refuse_the_load(self, tmp_path: Path) -> None:+ """A dependency's own dependencies are never loaded, so a concept naming one of theirs is left for first use, not refused."""+ library_manager = _get_library_manager()+ library_id, library = library_manager.open_library()+ try:+ with scoped_current_library(library_id=library_id):+ library_manager._load_single_dependency( # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+ library=library,+ resolved_dep=_make_resolved_dep(tmp_path=tmp_path, mthds_content=DEP_WITH_ITS_OWN_DEPENDENCY_MTHDS),+ package_address=DEP_ADDRESS,+ )+ tag_concept = library.dependency_libraries[DEP_ALIAS].concept_library.get_required_concept(concept_ref="invented_notes.Tag")+ tag_class = get_class_registry().get_required_base_model(name=tag_concept.structure_class_name)+ tag = tag_class.model_validate({"label": "Invented tag"})+ finally:+ library_manager.teardown(library_id=library_id)++ assert tag.model_dump() == {"label": "Invented tag"}