Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Consumer-pipe-shadows-dependencys · L-260929-7dcd75 (#1341)

Louis Choqueld75d4e10c2bb
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0e0927c5b..39a4cb0f6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,19 @@
- **The hosted backend's errors and messages no longer carry a codename (Breaking)**: its error classes are renamed from `Manifold*` to `PipelexHosted*`, such as `PipelexHostedError` and `PipelexHostedCredentialsError`, with their reference pages, while each keeps the `type` and `title` of its generic family. Its messages speak of "the Pipelex service", and the two that refuse a missing endpoint or key name the backend's `endpoint` and `api_key` settings rather than an environment variable. Its provider package moves to `pipelex.providers.pipelex_hosted`, `extract_manifold_metadata` is now `extract_pipelex_hosted_metadata`, and the enum member `PipelexBackend.MANIFOLD` is now `PipelexBackend.PIPELEX_HOSTED`, with its value `pipelex_manifold` unchanged.
+### Added
+
+- **A reference to a package's private pipe is refused as unexported**: a consumer naming a pipe that a loaded dependency package declares but does not export is refused at load with the new `PipeValidationErrorType` value `unexported_pipe_dependency`, whose item names the referring pipe in `pipe_code` and the reference in `missing_pipe_code`.
+
+### Fixed
+
+- **A dependency package's pipes call their own sub-pipes**: a sequence, parallel, condition or batch inside an installed package now reaches the package's own pipe, stored as `alias->domain.code`, where it used to look the plain `domain.code` up in the consumer's library. A package whose controllers call its own pipes, such as `github.com/Pipelex/methods/documents` from 0.1.3, no longer fails to load with `PipeNotFoundError`, and a consumer pipe of the same domain and code is no longer run in the package's place.
+- **An exported pipe can call its package's private pipes (Breaking)**: a package with `[exports]` loads every pipe its exported pipes and `main_pipe`s reach, not the exported ones alone, and a private pipe nothing public reaches is not built. Exports are read by domain, so exporting `x` from one domain no longer exports a pipe named `x` in another, and `PipeLibrary.add_dependency_pipe` takes `is_exported`.
+- **A missing pipe of a loaded package is refused at load**: a reference to a pipe a loaded dependency package does not have is refused as `unresolved_pipe_dependency`, naming the package, where it was skipped and reported as an extraneous input with a wrong suggested fix. A dependency's own reference to another package is not checked against the consumer's packages, since a dependency's dependencies are not loaded.
+- **A pipe wrapping another package's pipe of the same domain and code reports its inputs**: the recursion guards of the needed-inputs, signature and file-input walks tell two pipes of the same `domain.code` apart by object rather than by `pipe_ref`. A consumer pipe calling a dependency's pipe of its own domain and code, or one package's pipe calling another's, was taken for an already-visited pipe and reported no inputs, so a correct bundle was refused with an extraneous-input error.
+- **A bare reference into a package ignores its private pipes when an exported one matches**: a consumer's `alias->code` that matches an exported pipe and a private pipe of the same code in another domain resolves to the exported one, where it was refused as ambiguous.
+- **A dependency pipe that fails to build is refused naming why**: a reference to a pipe the package declares but could not build, such as a `PipeFunc` naming an unregistered function, is refused with that build error, where it was reported as a pipe the package does not have. A bare `alias->code` matching such a pipe is refused the same way, rather than resolved to another pipe of that code.
+
## [v0.73.0] - 2026-10-05
### Added
diff --git a/docs/api-server/openapi/pipelex-api.openapi.yaml b/docs/api-server/openapi/pipelex-api.openapi.yaml
index f1e5295b3..0634d5a63 100644
--- a/docs/api-server/openapi/pipelex-api.openapi.yaml
+++ b/docs/api-server/openapi/pipelex-api.openapi.yaml
@@ -5521,6 +5521,7 @@ components:
- native_concept_redeclaration
- unresolved_concept
- unresolved_pipe_dependency
+ - unexported_pipe_dependency
- unknown_model
- unknown_validation_error
title: PipeValidationErrorType
diff --git a/docs/building-methods/packages.md b/docs/building-methods/packages.md
index f68b7c2d3..c48ed1adb 100644
--- a/docs/building-methods/packages.md
+++ b/docs/building-methods/packages.md
@@ -57,8 +57,8 @@ The `[exports]` section controls which pipes are visible outside the package.
### Default Behavior
-- **Without `METHODS.toml`**: local bundles are still available to your project
-- **With `METHODS.toml`**: pipes are private by default — only pipes listed in `[exports]` (and each bundle's `main_pipe`) are visible from outside
+- **Without `METHODS.toml`, or with one that declares no `[exports]`**: every pipe of the package is visible from outside
+- **With an `[exports]` section**: pipes are private by default — only the pipes listed in `[exports]` (and each bundle's `main_pipe`) are visible from outside
### Declaring Exports
@@ -72,7 +72,26 @@ pipes = ["extract_clause", "analyze_contract"]
pipes = ["compute_weighted_score"]
```
-This manifest exports two pipes from `legal.contracts` and one pipe from `scoring`.
+This manifest exports two pipes from `legal.contracts` and one pipe from `scoring`. Exports are read by domain: exporting `compute_weighted_score` from `scoring` does not export a pipe of the same name declared in another domain of the package.
+
+### Pipes Call Their Own Package
+
+Inside a package, a pipe calls the package's own pipes, private ones included. A sequence exported by the package can call a helper the manifest does not export, and that helper is loaded with it. A private pipe that no exported pipe calls is not loaded at all, so a problem in it never stops a consumer from loading the package.
+
+A reference written inside a package always reaches that package's pipe. If the consumer declares a pipe with the same domain and code, or another loaded package does, the package's own call is never redirected to it.
+
+### What a Consumer Is Refused
+
+When a consumer references a loaded package, two mistakes are refused when the consumer loads, rather than surfacing later as an unrelated error:
+
+- **A pipe the package does not have** is refused as an unresolved pipe dependency (`unresolved_pipe_dependency`), naming the reference and the package. The same refusal names the error when the package declares the pipe but it failed to build.
+- **A pipe the package does not export** is refused as an unexported pipe dependency (`unexported_pipe_dependency`). The pipe exists, so the remedy is to call one of the package's exported pipes, or to export this one in the package's manifest.
+
+These checks apply to the consumer's references and to a package's references to its own pipes. A package's reference to one of its own dependencies is not checked when the consumer loads, because a package's dependencies are not loaded with it.
+
+A reference by bare code, `alias->code`, ignores the package's private pipes when it also matches an exported one, so a private helper sharing an exported pipe's code in another domain does not make the reference ambiguous.
+
+Naming a private pipe by hand, as the pipe to run at the command line or in an API request, is not a reference from inside a method, so `[exports]` does not apply to it: a loaded private pipe can be run that way by its `alias->domain.code`.
## Cross-Package References
diff --git a/docs/under-the-hood/build-time-elaboration.md b/docs/under-the-hood/build-time-elaboration.md
index 8592b5410..d1c3a1126 100644
--- a/docs/under-the-hood/build-time-elaboration.md
+++ b/docs/under-the-hood/build-time-elaboration.md
@@ -55,7 +55,7 @@ Practical consequences:
- Within a single process, after `BundleElaborator.elaborate(...)`, the metadata is available everywhere the bundle goes.
- Across any serialization boundary — kajson dump, library cache, Temporal payload, MTHDS export — the metadata is gone. Downstream consumers that need it today have to re-elaborate.
-- The dependency loader handles one specific consequence: when a manifest restricts exports, synthetic helpers of exported parents are still loaded, even though they are never named in the manifest. This is implemented inline in `LibraryManager._load_single_dependency`.
+- The dependency loader does not need the metadata: when a manifest restricts exports, the synthetic helpers of an exported parent are still loaded, though the manifest never names them, because the parent's steps call them. `LibraryManager._load_single_dependency` loads every pipe a package's public pipes reach through its own references, as private pipes of the package.
When a future consumer (graph viewer, persistent observability store) wants the metadata across boundaries, dropping `exclude=True` is the deliberate next step.
diff --git a/pipelex/cli/dev_cli/commands/generate_corpus_vocabulary_cmd.py b/pipelex/cli/dev_cli/commands/generate_corpus_vocabulary_cmd.py
index 0ab1683c0..f8598f441 100644
--- a/pipelex/cli/dev_cli/commands/generate_corpus_vocabulary_cmd.py
+++ b/pipelex/cli/dev_cli/commands/generate_corpus_vocabulary_cmd.py
@@ -157,6 +157,13 @@ _ERROR_TYPE_EXCLUSIONS: dict[ValidationErrorType, str] = {
"a caller — so there is no `expected_error` an entry could carry. Revisit if the manifest ever grows a "
"warnings axis."
),
+ PipeValidationErrorType.UNEXPORTED_PIPE_DEPENDENCY: (
+ "It refuses a reference to a pipe an installed dependency package loads but does not export, so the "
+ "bundle that produces it needs that package installed beside it, with a manifest declaring `[exports]`. "
+ "A focused entry is one self-contained bundle validated where nothing is installed, and an "
+ "address-based reference to a package that is not installed fails as a missing method instead. "
+ "Revisit if an entry can ever carry the packages it depends on."
+ ),
HintLintErrorType.HINT_UNKNOWN_KEY: _ADVISORY_HINT_LINT_EXCLUSION,
HintLintErrorType.HINT_UNKNOWN_INTENT: _ADVISORY_HINT_LINT_EXCLUSION,
HintLintErrorType.HINT_INAPPLICABLE_INTENT: _ADVISORY_HINT_LINT_EXCLUSION,
diff --git a/pipelex/libraries/crate_qualification.py b/pipelex/libraries/crate_qualification.py
index 1738a7a03..63df8e757 100644
--- a/pipelex/libraries/crate_qualification.py
+++ b/pipelex/libraries/crate_qualification.py
@@ -23,6 +23,13 @@ Two kinds of ref are deliberately left alone: cross-package (`alias->…`) refs,
is the packaging project's design work, and `SpecialOutcome` values (`fail` / `continue`), which are
condition outcomes rather than pipe refs.
+**A dependency package's own pipe refs carry its alias.** The dependency loader passes `package_alias`,
+and every pipe ref the package wrote without an alias then becomes `alias->domain.code`, the key the
+consumer's library holds that pipe under (`PipeLibrary.add_dependency_pipe`). Every reader of a sub-pipe
+ref — validation, the pre-run walks, execution — looks it up in the consumer's library, so stored as a
+plain `domain.code` it would find nothing, or the consumer's own pipe of the same `domain.code`. Concept
+refs are not aliased: a dependency's concepts are bound when its pipes are built, under `domain.Code`.
+
The pass is idempotent: an already-qualified ref is returned unchanged, so running it twice over the
same crate is the same as running it once. It does not mutate the crate it is given.
@@ -64,9 +71,14 @@ class QualifiedCrateContent(NamedTuple):
pipes: dict[str, PipeBlueprintUnion]
-def qualify_crate(crate: LibraryCrate) -> QualifiedCrateContent:
+def qualify_crate(crate: LibraryCrate, *, package_alias: str | None = None) -> QualifiedCrateContent:
"""Qualify every in-body reference of a merged, key-qualified crate.
+ Args:
+ crate: The merged, key-qualified crate.
+ package_alias: The alias a dependency package is loaded under, when the crate is that package's:
+ every pipe ref it wrote without an alias is then stored as `alias->domain.code`.
+
Raises:
CrateNormalizationError: A crate key is not domain-qualified — the pass was handed something
that is not a merged, key-qualified crate.
@@ -76,7 +88,8 @@ def qualify_crate(crate: LibraryCrate) -> QualifiedCrateContent:
}
pipes: dict[str, PipeBlueprintUnion] = {
- pipe_ref: _qualify_pipe_blueprint(blueprint, owner_domain=_domain_of(pipe_ref)) for pipe_ref, blueprint in crate.pipes.items()
+ pipe_ref: _qualify_pipe_blueprint(blueprint, owner_domain=_domain_of(pipe_ref), package_alias=package_alias)
+ for pipe_ref, blueprint in crate.pipes.items()
}
return QualifiedCrateContent(concepts=concepts, pipes=pipes)
@@ -147,7 +160,7 @@ def _qualify_slot_value(slot_value: "str | InputSlotBlueprint", *, domain: str)
return slot_value.model_copy(update={"concept": _qualify_io_ref(slot_value.concept, domain=domain)})
-def _qualify_pipe_blueprint(blueprint: PipeBlueprintUnion, *, owner_domain: str) -> PipeBlueprintUnion:
+def _qualify_pipe_blueprint(blueprint: PipeBlueprintUnion, *, owner_domain: str, package_alias: str | None) -> PipeBlueprintUnion:
updates: dict[str, object] = {}
if blueprint.inputs:
updates["inputs"] = {name: _qualify_slot_value(value, domain=owner_domain) for name, value in blueprint.inputs.items()}
@@ -155,7 +168,7 @@ def _qualify_pipe_blueprint(blueprint: PipeBlueprintUnion, *, owner_domain: str)
updates["output"] = _qualify_io_ref(blueprint.output, domain=owner_domain)
def qualify(*, pipe_ref: str) -> str:
- return _qualify_pipe_ref(pipe_ref, owner_domain=owner_domain)
+ return _qualify_pipe_ref(pipe_ref, owner_domain=owner_domain, package_alias=package_alias)
def qualify_outcome(*, outcome: str) -> str:
"""A condition outcome may be a `SpecialOutcome` instead of a pipe ref.
@@ -167,7 +180,7 @@ def _qualify_pipe_blueprint(blueprint: PipeBlueprintUnion, *, owner_domain: str)
"""
if outcome in SpecialOutcome.value_list():
return outcome
- return _qualify_pipe_ref(outcome, owner_domain=owner_domain)
+ return _qualify_pipe_ref(outcome, owner_domain=owner_domain, package_alias=package_alias)
match blueprint:
case PipeSequenceBlueprint():
@@ -210,7 +223,7 @@ def _render_ref_with_markers(concept_ref: str, *, parsed: MultiplicityParseResul
return f"{concept_ref}{suffix}{presence_symbol(presence=parsed.presence)}"
-def _qualify_pipe_ref(pipe_ref: str, *, owner_domain: str) -> str:
+def _qualify_pipe_ref(pipe_ref: str, *, owner_domain: str, package_alias: str | None) -> str:
"""Qualify one in-body pipe ref to the domain of the pipe that wrote it.
The exact twin of `_qualify_concept_ref`, and unconditional: a bare code is the owner domain's,
@@ -220,9 +233,13 @@ def _qualify_pipe_ref(pipe_ref: str, *, owner_domain: str) -> str:
A ref that names nothing is not this function's problem. It cannot be: the pass sees one load
batch, and a batch may reference a pipe a prior batch put in the same domain. Dependency
validation sees the whole live library and reports the miss there, naming the qualified ref.
+
+ Under `package_alias` the qualified ref is prefixed with the alias, since that is the key the
+ consumer's library holds a dependency's pipe under.
"""
if QualifiedRef.has_cross_package_prefix(pipe_ref):
- return pipe_ref # cross-package deferred
- if "." in pipe_ref:
- return pipe_ref
- return f"{owner_domain}.{pipe_ref}"
+ return pipe_ref # cross-package deferred; an aliased ref is also what makes the pass idempotent under `package_alias`
+ qualified = pipe_ref if "." in pipe_ref else f"{owner_domain}.{pipe_ref}"
+ if package_alias is None:
+ return qualified
+ return f"{package_alias}->{qualified}"
diff --git a/pipelex/libraries/library.py b/pipelex/libraries/library.py
index 92b42ee15..b057aadb9 100644
--- a/pipelex/libraries/library.py
+++ b/pipelex/libraries/library.py
@@ -81,6 +81,32 @@ def _describe_unresolved_pipe_dependency(
return " ".join(lines)
+def _describe_unexported_pipe_dependency(*, referring_pipe_key: str, private_ref: str, package_alias: str) -> str:
+ return (
+ f"Pipe '{referring_pipe_key}' references '{private_ref}', which the package loaded as '{package_alias}' does not export. "
+ "A package's private pipes are called by its own pipes only: call one of its exported pipes, or export this one in the package."
+ )
+
+
+def _pipe_dependency_refusal(*, pipe: PipeController, missing_ref: str, error_type: PipeValidationErrorType, message: str) -> LibraryLoadingError:
+ """The refusal of a controller's sub-pipe reference, carrying a structured item.
+
+ The item makes the refusal a categorized `pipe_validation` item: `pipe_code` is the referring controller and
+ `missing_pipe_code` the reference that does not resolve, or is not visible, so a machine consumer reads both
+ without parsing the message. LibraryLoadingError rides the existing `except LibraryError` forwarding in
+ translate_to_validate_bundle_error.
+ """
+ dependency_error_data = PipesAndConceptValidationErrorData(
+ error_type=error_type,
+ domain_code=pipe.domain_code,
+ pipe_code=pipe.code,
+ missing_pipe_code=missing_ref,
+ message=message,
+ field_path=f"pipe.{pipe.code}",
+ )
+ return LibraryLoadingError(message=message, pipe_concept_validation_errors=[dependency_error_data])
+
+
class Library(BaseModel):
"""A Library bundles together domain, concept, and pipe libraries for a specific context.
@@ -201,14 +227,9 @@ class Library(BaseModel):
# Validate pipe dependencies exist for pipe controllers
if isinstance(pipe, PipeController):
for sub_pipe_code in pipe.pipe_dependencies():
- # Cross-package refs that aren't loaded are validated at package level, not library level
- if QualifiedRef.has_cross_package_prefix(sub_pipe_code) and self.pipe_library.get_optional_pipe(sub_pipe_code) is None:
+ if QualifiedRef.has_cross_package_prefix(sub_pipe_code):
+ self._validate_cross_package_pipe_dependency(pipe=pipe, pipe_key=pipe_key, dep_alias=dep_alias, sub_pipe_code=sub_pipe_code)
continue
- # For dependency pipes, look up bare sub-pipe codes in the child library
- if dep_alias is not None and not QualifiedRef.has_cross_package_prefix(sub_pipe_code):
- child_library = self.dependency_libraries.get(dep_alias)
- if child_library is not None and child_library.pipe_library.get_optional_pipe(sub_pipe_code) is not None:
- continue
try:
self.pipe_library.get_required_pipe(pipe_code=sub_pipe_code)
except PipeLibraryError as pipe_error:
@@ -220,20 +241,9 @@ class Library(BaseModel):
candidates=bare_code_candidates,
cause=pipe_error,
)
- # Carry a structured item so an unresolved dependency surfaces as a categorized
- # `pipe_validation` item: `pipe_code` is the referencing controller and
- # `missing_pipe_code` is the dependency that does not resolve, so a machine
- # consumer can read both without parsing the message. LibraryLoadingError rides
- # the existing `except LibraryError` forwarding in translate_to_validate_bundle_error.
- dependency_error_data = PipesAndConceptValidationErrorData(
- error_type=PipeValidationErrorType.UNRESOLVED_PIPE_DEPENDENCY,
- domain_code=pipe.domain_code,
- pipe_code=pipe.code,
- missing_pipe_code=sub_pipe_code,
- message=msg,
- field_path=f"pipe.{pipe.code}",
- )
- raise LibraryLoadingError(message=msg, pipe_concept_validation_errors=[dependency_error_data]) from pipe_error
+ raise _pipe_dependency_refusal(
+ pipe=pipe, missing_ref=sub_pipe_code, error_type=PipeValidationErrorType.UNRESOLVED_PIPE_DEPENDENCY, message=msg
+ ) from pipe_error
for pipe in self.pipe_library.root.values():
# Skip full validation for pipe controllers with unresolved cross-package dependencies
@@ -241,6 +251,65 @@ class Library(BaseModel):
continue
pipe.validate_with_libraries()
+ def _validate_cross_package_pipe_dependency(self, *, pipe: PipeController, pipe_key: str, dep_alias: str | None, sub_pipe_code: str) -> None:
+ """Refuse a reference into a dependency package that names a pipe the package does not have or does not export.
+
+ A dependency's own references carry its alias (`crate_qualification.qualify_crate`), so this covers a
+ package's calls to its own pipes as well as a consumer's calls into it. A reference whose alias names no
+ loaded package is tolerated: that is a crate loaded without its packages, as a worker loads one, and the
+ package is validated where it is loaded. So is a dependency's reference to another package: a dependency's own
+ dependencies are never loaded, and its alias for one may name a different package the consumer loaded.
+
+ A reference naming a pipe the package declares but could not build is refused naming that failure before
+ anything is looked up, so a bare `alias->code` never falls through to another pipe of the same code.
+
+ Raises:
+ LibraryLoadingError: The package is loaded and has no such pipe, withholds it, or holds it as private
+ while the referring pipe is not the package's own.
+ """
+ ref_alias, remainder = QualifiedRef.split_cross_package_ref(sub_pipe_code)
+ if ref_alias not in self.dependency_libraries or (dep_alias is not None and ref_alias != dep_alias):
+ return
+ unbuilt_reason = self.pipe_library.unbuilt_dependency_pipe_reason(pipe_code=sub_pipe_code)
+ if unbuilt_reason is not None:
+ msg = (
+ f"Pipe '{pipe_key}' references '{sub_pipe_code}', which the package loaded as '{ref_alias}' declares "
+ f"but could not build: {unbuilt_reason}"
+ )
+ raise _pipe_dependency_refusal(
+ pipe=pipe, missing_ref=sub_pipe_code, error_type=PipeValidationErrorType.UNRESOLVED_PIPE_DEPENDENCY, message=msg
+ )
+ try:
+ sub_pipe = self.pipe_library.get_optional_pipe(pipe_code=sub_pipe_code)
+ except PipeLibraryError as pipe_error:
+ msg = _describe_unresolved_pipe_dependency(referring_pipe_ref=pipe.pipe_ref, missing_ref=sub_pipe_code, candidates={}, cause=pipe_error)
+ raise _pipe_dependency_refusal(
+ pipe=pipe, missing_ref=sub_pipe_code, error_type=PipeValidationErrorType.UNRESOLVED_PIPE_DEPENDENCY, message=msg
+ ) from pipe_error
+ if sub_pipe is None:
+ if self.pipe_library.is_withheld_dependency_pipe(pipe_code=sub_pipe_code):
+ raise _pipe_dependency_refusal(
+ pipe=pipe,
+ missing_ref=sub_pipe_code,
+ error_type=PipeValidationErrorType.UNEXPORTED_PIPE_DEPENDENCY,
+ message=_describe_unexported_pipe_dependency(referring_pipe_key=pipe_key, private_ref=sub_pipe_code, package_alias=ref_alias),
+ )
+ msg = (
+ f"Pipe '{pipe_key}' references '{sub_pipe_code}', which does not exist. "
+ f"The package loaded as '{ref_alias}' has no pipe '{remainder}'."
+ )
+ raise _pipe_dependency_refusal(
+ pipe=pipe, missing_ref=sub_pipe_code, error_type=PipeValidationErrorType.UNRESOLVED_PIPE_DEPENDENCY, message=msg
+ )
+ # A package's private pipe is visible to that package's own pipes, and to nothing else.
+ if ref_alias != dep_alias and self.pipe_library.is_private_dependency_pipe(pipe_key=f"{ref_alias}->{sub_pipe.pipe_ref}"):
+ raise _pipe_dependency_refusal(
+ pipe=pipe,
+ missing_ref=sub_pipe_code,
+ error_type=PipeValidationErrorType.UNEXPORTED_PIPE_DEPENDENCY,
+ message=_describe_unexported_pipe_dependency(referring_pipe_key=pipe_key, private_ref=sub_pipe_code, package_alias=ref_alias),
+ )
+
def _index_pipe_refs_by_bare_code(self) -> dict[str, list[str]]:
"""Group every pipe ref in this library by its bare code, for failure-path diagnostics only.
diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.py
index 46ab8bf95..b2337926b 100644
--- a/pipelex/libraries/library_manager.py
+++ b/pipelex/libraries/library_manager.py
@@ -52,7 +52,7 @@ from pipelex.methods.structures_check import ensure_no_structured_content_in_lib
from pipelex.mthds_parsing.exceptions import MthdsParserError
from pipelex.mthds_parsing.handle_pipe_errors import categorize_pipe_validation_error
from pipelex.mthds_parsing.parser import MthdsParser
-from pipelex.mthds_parsing.pipelex_bundle_blueprint import ElaborationMetadata, PipelexBundleBlueprint, StepRole
+from pipelex.mthds_parsing.pipelex_bundle_blueprint import ElaborationMetadata, PipeBlueprintUnion, PipelexBundleBlueprint, StepRole
from pipelex.pipe_machinery.pipe_abstract import PipeAbstract
from pipelex.pipe_machinery.pipe_factory import PipeFactory
from pipelex.runtime_hub import get_class_registry
@@ -119,6 +119,60 @@ def _dependency_bundle_source(*, package_address: str, package_root: Path, mthds
return f"{package_address}/{mthds_path.name}"
+def _public_dependency_pipe_refs(*, resolved_dep: ResolvedDependency, blueprints: list[PipelexBundleBlueprint]) -> set[str] | None:
+ """The `domain.code` of every pipe a dependency makes public, or `None` when it declares no exports and all are public.
+
+ A package's public pipes are the ones its manifest exports, read by domain so that exporting `a.x` does not
+ also export a `b.x` of another domain, and the `main_pipe` of each of its bundles.
+ """
+ if resolved_dep.exported_pipe_codes is None:
+ return None
+ public_pipe_refs: set[str] = set()
+ if resolved_dep.manifest is not None:
+ for domain_path, domain_exports in resolved_dep.manifest.exports.items():
+ public_pipe_refs.update(f"{domain_path}.{pipe_code}" for pipe_code in domain_exports.pipes)
+ for blueprint in blueprints:
+ if blueprint.main_pipe:
+ public_pipe_refs.add(f"{blueprint.domain}.{blueprint.main_pipe}")
+ return public_pipe_refs
+
+
+def reachable_dependency_pipe_refs(
+ *, public_pipe_refs: set[str], qualified_pipes: "Mapping[str, PipeBlueprintUnion]", package_alias: str
+) -> set[str]:
+ """The `domain.code` of every pipe of a dependency its public pipes reach through the package's own references.
+
+ The pipes are the package's qualified blueprints, whose in-package refs carry `package_alias`. A public pipe's
+ private helpers, a build-time elaboration's synthetic helpers among them, travel with it; a private pipe nothing
+ public reaches is left out. A ref to another package, or to a pipe the package does not declare, is not followed:
+ validation reports the latter. A ref written `alias->code`, with no domain, is followed as lookup resolves it, by code
+ within the package: to the public pipes of that code when there are some among several, else to every pipe of it,
+ lookup reporting the ambiguity when more than one remains.
+ """
+ reachable: set[str] = set()
+ pending = [pipe_ref for pipe_ref in public_pipe_refs if pipe_ref in qualified_pipes]
+ while pending:
+ pipe_ref = pending.pop()
+ if pipe_ref in reachable:
+ continue
+ reachable.add(pipe_ref)
+ for dependency_ref in qualified_pipes[pipe_ref].pipe_dependencies:
+ if not QualifiedRef.has_cross_package_prefix(dependency_ref):
+ continue
+ ref_alias, in_package_ref = QualifiedRef.split_cross_package_ref(dependency_ref)
+ if ref_alias != package_alias:
+ continue
+ if "." in in_package_ref:
+ if in_package_ref in qualified_pipes:
+ pending.append(in_package_ref)
+ else:
+ candidates = [candidate for candidate in qualified_pipes if candidate.rsplit(".", 1)[-1] == in_package_ref]
+ # Lookup lets private pipes only break a tie (`PipeLibrary.get_optional_pipe`), so one public match is the one reached.
+ public_candidates = [candidate for candidate in candidates if candidate in public_pipe_refs]
+ pending.extend(public_candidates if len(candidates) > 1 and public_candidates else candidates)
+ return reachable
+
+
def _dependency_entries_first(*, concepts_by_key: "Mapping[str, Concept]") -> list["Concept"]:
"""A library's concepts with its dependencies' aliased entries (`alias->domain.Code`) before its own.
@@ -1245,41 +1299,10 @@ class LibraryManager(LibraryManagerAbstract):
loaded_concepts=dep_concepts, visible_concepts=dep_concepts, concept_sources=crate.source_map, refuses_unresolved_structures=False
)
- # Collect main_pipes for auto-export
- main_pipes: set[str] = set()
- for blueprint in dep_blueprints:
- if blueprint.main_pipe:
- main_pipes.add(blueprint.main_pipe)
-
- # Determine if we filter by exports or load all.
- # exported_pipe_codes is None when no manifest exists (all pipes public),
- # or a set (possibly empty) when a manifest defines exports.
- if resolved_dep.exported_pipe_codes is None:
- # No manifest: all pipes are public, no filtering
- has_exports = False
- all_exported: set[str] = set()
- else:
- # Manifest exists: filter to exported pipes + main_pipes
- has_exports = True
- all_exported = resolved_dep.exported_pipe_codes | main_pipes
- # Synthetic helpers from build-time elaboration (e.g. `<code>__draft_text` and
- # `<code>__structure` produced by `structuring_method = preliminary_text`) are
- # private to their parent pipe and never listed in the manifest. When the parent
- # is exported, its helpers must travel with it — otherwise the wrapping
- # PipeSequence references unresolved pipe codes at runtime.
- # Note: `parent_pipe_code` and `synthetic_code` are bare codes within a single
- # bundle — `BundleElaborator` writes them that way today. If two bundles in the
- # same dep ever ship the same bare pipe code, this lookup would conflate their
- # helpers. The downstream factory would then fail on duplicate registration, so
- # the failure mode is loud rather than silent.
- synthetic_helpers: set[str] = set()
- for blueprint in dep_blueprints:
- if not blueprint.elaboration_metadata:
- continue
- for synthetic_code, meta in blueprint.elaboration_metadata.items():
- if meta.parent_pipe_code in all_exported:
- synthetic_helpers.add(synthetic_code)
- all_exported |= synthetic_helpers
+ # What a consumer may reference: `None` when the package declares no exports, so every pipe is public.
+ public_pipe_refs = _public_dependency_pipe_refs(resolved_dep=resolved_dep, blueprints=dep_blueprints)
+ # The `domain.code` of each pipe meant to load that failed to build, with why, for a reference to it to name.
+ unbuilt_pipe_reasons: dict[str, str] = {}
# Temporarily register dep concepts in main library for pipe construction
# (PipeFactory resolves concepts through the hub's current library)
@@ -1302,11 +1325,20 @@ class LibraryManager(LibraryManagerAbstract):
# Load exported pipes (reconciled by the crate) into child library, ensuring temp concepts
# are always cleaned up even if an unexpected exception occurs
try:
- # Same qualification the main load path applies: a dependency package's own in-body refs
- # are its own domain's, and its child library is keyed by qualified pipe_ref. Inside the
+ # The main load path's qualification, plus this load's alias: a dependency package's own
+ # in-body refs are its own domain's, and each is stored as `alias->domain.code`, the key
+ # the consumer's library holds that pipe under, so every reader of the ref (validation,
+ # the pre-run walks, execution) reaches the package's pipe and never a consumer pipe of
+ # the same `domain.code`. The child library itself stays keyed by `domain.code`. Inside the
# try: qualification can raise on malformed refs, and the temp concepts must still be
# removed from the main library.
- qualified_dep_pipes = qualify_crate(crate).pipes
+ qualified_dep_pipes = qualify_crate(crate, package_alias=alias).pipes
+ if public_pipe_refs is None:
+ loaded_pipe_refs = set(qualified_dep_pipes)
+ else:
+ loaded_pipe_refs = reachable_dependency_pipe_refs(
+ public_pipe_refs=public_pipe_refs, qualified_pipes=qualified_dep_pipes, package_alias=alias
+ )
for pipe_ref, pipe_blueprint in qualified_dep_pipes.items():
parsed_pipe = QualifiedRef.parse_pipe_ref(raw=pipe_ref)
if parsed_pipe.domain_path is None:
@@ -1314,8 +1346,8 @@ class LibraryManager(LibraryManagerAbstract):
raise PipeLibraryError(msg)
domain_code = parsed_pipe.domain_path
pipe_code = parsed_pipe.local_code
- # If manifest has exports, only load exported pipes
- if has_exports and pipe_code not in all_exported:
+ # A private pipe nothing public reaches is never built, so a broken one cannot refuse the consumer's load.
+ if pipe_ref not in loaded_pipe_refs:
continue
dependency_source = crate.source_map.get(pipe_ref)
# A pipe that reads a file beside its bundle (a PipeDocGen `template_file`) finds it from the
@@ -1342,6 +1374,7 @@ class LibraryManager(LibraryManagerAbstract):
child_library.pipe_library.add_new_pipe(pipe=pipe)
except ValidationError as exc:
log.warning(f"Could not load dependency '{alias}' pipe '{pipe_code}': {exc}")
+ unbuilt_pipe_reasons[pipe_ref] = "; ".join(str(error["msg"]) for error in exc.errors()) or str(exc)
finally:
# Remove temporary concept entries from main library
library.concept_library.remove_concepts_by_concept_refs(concept_refs=temp_concept_refs)
@@ -1354,7 +1387,11 @@ class LibraryManager(LibraryManagerAbstract):
library.concept_library.add_dependency_concept(alias=alias, concept=concept)
for pipe in child_library.pipe_library.get_pipes():
- library.pipe_library.add_dependency_pipe(alias=alias, pipe=pipe)
+ is_exported = public_pipe_refs is None or pipe.pipe_ref in public_pipe_refs
+ library.pipe_library.add_dependency_pipe(alias=alias, pipe=pipe, is_exported=is_exported)
+ library.pipe_library.add_withheld_dependency_pipes(alias=alias, pipe_refs=set(qualified_dep_pipes) - loaded_pipe_refs)
+ for pipe_ref, reason in unbuilt_pipe_reasons.items():
+ library.pipe_library.add_unbuilt_dependency_pipe(alias=alias, pipe_ref=pipe_ref, reason=reason)
log.verbose(f"Loaded dependency '{alias}': {len(dep_concepts)} concepts, pipes from {len(dep_blueprints)} bundles")
diff --git a/pipelex/libraries/pipe/pipe_library.py b/pipelex/libraries/pipe/pipe_library.py
index 6407cac2c..478697e96 100644
--- a/pipelex/libraries/pipe/pipe_library.py
+++ b/pipelex/libraries/pipe/pipe_library.py
@@ -1,7 +1,7 @@
from itertools import groupby
from typing import Self
-from pydantic import RootModel
+from pydantic import PrivateAttr, RootModel
from typing_extensions import override
from pipelex import pretty_print
@@ -15,6 +15,16 @@ PipeLibraryRoot = dict[str, PipeAbstract]
class PipeLibrary(RootModel[PipeLibraryRoot], PipeLibraryAbstract):
+ # The aliased keys (`alias->domain.code`) of dependency pipes their package does not export: loaded because a
+ # public pipe of the package calls them, and visible to that package's pipes only.
+ _private_dependency_keys: set[str] = PrivateAttr(default_factory=set)
+ # The aliased keys of dependency pipes their package declares but neither exports nor calls from a public pipe:
+ # never built, and recorded only so a reference to one is refused as unexported rather than as missing.
+ _withheld_dependency_keys: set[str] = PrivateAttr(default_factory=set)
+ # The aliased keys of dependency pipes the package meant to load but that failed to build, with the reason, so a
+ # reference to one is refused naming that failure rather than as a pipe the package does not have.
+ _unbuilt_dependency_reasons: dict[str, str] = PrivateAttr(default_factory=dict)
+
@override
def setup(self):
pass
@@ -22,6 +32,9 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeLibraryAbstract):
@override
def teardown(self):
self.root = {}
+ self._private_dependency_keys = set()
+ self._withheld_dependency_keys = set()
+ self._unbuilt_dependency_reasons = {}
@override
def reset(self):
@@ -81,9 +94,14 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeLibraryAbstract):
pipe = self.root.get(aliased_key)
if pipe is not None:
return pipe
- # Bare code remainder — search aliased entries matching the bare code
+ # Bare code remainder — search aliased entries matching the bare code. A package's private pipes only
+ # break a tie: they are invisible to a consumer, so one exported match among several is the one meant,
+ # and a lone private match is returned for validation to refuse as unexported.
if "." not in remainder:
- matches = [val for key, val in self.root.items() if key.startswith(f"{alias}->") and val.code == remainder]
+ matched = {key: val for key, val in self.root.items() if key.startswith(f"{alias}->") and val.code == remainder}
+ matches = list(matched.values())
+ if len(matches) > 1:
+ matches = [val for key, val in matched.items() if key not in self._private_dependency_keys] or matches
if len(matches) == 1:
return matches[0]
if len(matches) > 1:
@@ -105,7 +123,8 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeLibraryAbstract):
It deliberately does **not** consult `[exports]`. Package visibility governs what one method
may reference from inside another; a pipe someone names by hand at an entry point is not an
- in-body reference, so the rule does not apply to it.
+ in-body reference, so the rule does not apply to it. A dependency's private pipe that a public
+ one calls is loaded, so it is reachable here by its aliased key.
Aliased dependency entries are excluded from the search. Without that, installing an
unrelated package could make a host pipe's bare code ambiguous — reintroducing, through this
@@ -154,18 +173,69 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeLibraryAbstract):
raise EntryPipeNotFoundError(msg)
return the_pipe
- def add_dependency_pipe(self, *, alias: str, pipe: PipeAbstract) -> None:
+ def add_dependency_pipe(self, *, alias: str, pipe: PipeAbstract, is_exported: bool) -> None:
"""Add a pipe from a dependency package with an aliased key.
Args:
alias: The dependency alias
pipe: The pipe to add
+ is_exported: Whether the package makes the pipe public. A private one is there because a public pipe
+ of the package calls it, and validation refuses a reference to it from outside the package.
"""
key = f"{alias}->{pipe.pipe_ref}"
if key in self.root:
msg = f"Dependency pipe '{key}' already exists in the library"
raise PipeLibraryError(msg)
self.root[key] = pipe
+ if not is_exported:
+ self._private_dependency_keys.add(key)
+
+ def is_private_dependency_pipe(self, *, pipe_key: str) -> bool:
+ """Whether `pipe_key`, an aliased key `alias->domain.code`, holds a dependency pipe its package does not export."""
+ return pipe_key in self._private_dependency_keys
+
+ def add_withheld_dependency_pipes(self, *, alias: str, pipe_refs: set[str]) -> None:
+ """Record the `domain.code` of the pipes a dependency declares but does not load, being private and unreached."""
+ self._withheld_dependency_keys.update(f"{alias}->{pipe_ref}" for pipe_ref in pipe_refs)
+
+ def is_withheld_dependency_pipe(self, *, pipe_code: str) -> bool:
+ """Whether a cross-package reference names a pipe its package declares but withholds, as private and unreached.
+
+ The reference may name the pipe as `alias->domain.code` or as `alias->code`, as `get_optional_pipe` accepts.
+ """
+ return bool(self._dependency_keys_named_by(pipe_code=pipe_code, keys=self._withheld_dependency_keys))
+
+ def add_unbuilt_dependency_pipe(self, *, alias: str, pipe_ref: str, reason: str) -> None:
+ """Record a dependency pipe, by its `domain.code`, that failed to build, and why."""
+ self._unbuilt_dependency_reasons[f"{alias}->{pipe_ref}"] = reason
+
+ def unbuilt_dependency_pipe_reason(self, *, pipe_code: str) -> str | None:
+ """Why the dependency pipe a cross-package reference names failed to build, or `None` when none it names failed.
+
+ The reference may name the pipe as `alias->domain.code` or as `alias->code`, as `get_optional_pipe` accepts. A
+ bare code matching a pipe that failed is answered even when another pipe of that code loaded, since the failed
+ one may be the one meant.
+ """
+ keys = self._dependency_keys_named_by(pipe_code=pipe_code, keys=set(self._unbuilt_dependency_reasons))
+ if not keys:
+ return None
+ return (
+ "; ".join(f"'{key}': {self._unbuilt_dependency_reasons[key]}" for key in keys)
+ if len(keys) > 1
+ else self._unbuilt_dependency_reasons[keys[0]]
+ )
+
+ @classmethod
+ def _dependency_keys_named_by(cls, *, pipe_code: str, keys: set[str]) -> list[str]:
+ """The aliased keys among `keys` that a cross-package reference names, by its full key or by its bare code."""
+ if pipe_code in keys:
+ return [pipe_code]
+ if not QualifiedRef.has_cross_package_prefix(pipe_code):
+ return []
+ alias, remainder = QualifiedRef.split_cross_package_ref(pipe_code)
+ if "." in remainder:
+ return []
+ return sorted(key for key in keys if key.startswith(f"{alias}->") and key.endswith(f".{remainder}"))
@override
def get_required_pipe(self, pipe_code: str) -> PipeAbstract:
@@ -196,6 +266,7 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeLibraryAbstract):
for pipe_ref in pipe_refs:
if pipe_ref in self.root:
del self.root[pipe_ref]
+ self._private_dependency_keys.discard(pipe_ref)
@override
def pretty_list_pipes(self) -> None:
diff --git a/pipelex/pipe_controllers/condition/pipe_condition.py b/pipelex/pipe_controllers/condition/pipe_condition.py
index 863a41739..a28d561d9 100644
--- a/pipelex/pipe_controllers/condition/pipe_condition.py
+++ b/pipelex/pipe_controllers/condition/pipe_condition.py
@@ -70,11 +70,11 @@ class PipeCondition(PipeController):
visited_pipes = set()
# If we've already visited this pipe, stop recursion
- if self.pipe_ref in visited_pipes:
+ if self.visit_key in visited_pipes:
return InputStuffSpecsFactory.make_empty()
# Add this pipe to visited set for recursive calls
- visited_pipes_with_current = visited_pipes | {self.pipe_ref}
+ visited_pipes_with_current = visited_pipes | {self.visit_key}
needed_inputs = InputStuffSpecsFactory.make_empty()
diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel.py b/pipelex/pipe_controllers/parallel/pipe_parallel.py
index 1727f8b3f..31d5deb0b 100644
--- a/pipelex/pipe_controllers/parallel/pipe_parallel.py
+++ b/pipelex/pipe_controllers/parallel/pipe_parallel.py
@@ -84,11 +84,11 @@ class PipeParallel(PipeController):
visited_pipes = set()
# If we've already visited this pipe, stop recursion
- if self.pipe_ref in visited_pipes:
+ if self.visit_key in visited_pipes:
return InputStuffSpecsFactory.make_empty()
# Add this pipe to visited set for recursive calls
- visited_pipes_with_current = visited_pipes | {self.pipe_ref}
+ visited_pipes_with_current = visited_pipes | {self.visit_key}
needed_inputs = InputStuffSpecsFactory.make_empty()
diff --git a/pipelex/pipe_controllers/sequence/pipe_sequence.py b/pipelex/pipe_controllers/sequence/pipe_sequence.py
index b4f1bf273..1a4911813 100644
--- a/pipelex/pipe_controllers/sequence/pipe_sequence.py
+++ b/pipelex/pipe_controllers/sequence/pipe_sequence.py
@@ -303,11 +303,11 @@ class PipeSequence(PipeController):
visited_pipes = set()
# If we've already visited this pipe, stop recursion
- if self.pipe_ref in visited_pipes:
+ if self.visit_key in visited_pipes:
return InputStuffSpecsFactory.make_empty()
# Add this pipe to visited set for recursive calls
- visited_pipes_with_current = visited_pipes | {self.pipe_ref}
+ visited_pipes_with_current = visited_pipes | {self.visit_key}
needed_inputs = InputStuffSpecsFactory.make_empty()
generated_outputs: set[str] = set()
diff --git a/pipelex/pipe_machinery/pipe_abstract.py b/pipelex/pipe_machinery/pipe_abstract.py
index 4de2593f4..f14685448 100644
--- a/pipelex/pipe_machinery/pipe_abstract.py
+++ b/pipelex/pipe_machinery/pipe_abstract.py
@@ -107,6 +107,16 @@ class PipeAbstract(ABC, BaseModel):
"""Domain-qualified pipe reference, e.g. 'scoring.compute_score'."""
return f"{self.domain_code}.{self.code}"
+ @property
+ def visit_key(self) -> str:
+ """This pipe's entry in a recursion guard's visited set, unique to this pipe object.
+
+ Not `pipe_ref`: a consumer and a dependency package, or two dependency packages, may each declare a pipe
+ of the same `domain.code`, and a guard keyed by it would take one for the other and cut a walk short.
+ A library holds one object per pipe, so a cycle returns to the same object.
+ """
+ return f"{self.pipe_ref}@{id(self):x}"
+
@property
def pipe_type(self) -> str:
return self.__class__.__name__
@@ -596,7 +606,7 @@ class PipeAbstract(ABC, BaseModel):
"""Return the stuff specs that are needed for the pipe to run.
Args:
- visited_pipes: Set of pipe codes currently being processed to prevent infinite recursion.
+ visited_pipes: The `visit_key` of each pipe currently being processed, to prevent infinite recursion.
If None, starts recursion detection with an empty set.
Returns:
diff --git a/pipelex/pipe_signature/signature_walk.py b/pipelex/pipe_signature/signature_walk.py
index de9910765..09c9b7c08 100644
--- a/pipelex/pipe_signature/signature_walk.py
+++ b/pipelex/pipe_signature/signature_walk.py
@@ -16,13 +16,14 @@ def collect_signature_refs(pipe: PipeAbstract, *, visited: set[str] | None = Non
Walks `pipe.pipe_dependencies()`, resolving each dependency via `get_optional_pipe`.
Unresolved dependencies are skipped silently. Cycles are broken via the `visited`
- set keyed by qualified pipe_ref.
+ set keyed by each pipe's `visit_key`, so a consumer pipe and a dependency's pipe of the
+ same `domain.code` are both walked.
"""
if visited is None:
visited = set()
- if pipe.pipe_ref in visited:
+ if pipe.visit_key in visited:
return set()
- visited.add(pipe.pipe_ref)
+ visited.add(pipe.visit_key)
found: set[str] = set()
if pipe.is_signature:
@@ -41,23 +42,27 @@ def collect_signature_paths(
pipe: PipeAbstract,
*,
current_path: list[str] | None = None,
+ visiting: frozenset[str] | None = None,
paths: dict[str, list[str]] | None = None,
) -> dict[str, list[str]]:
"""Return mapping from signature pipe_ref to the longest dep chain that reaches it.
Each value is the ordered list of controller pipe_refs traversed (entry-point first).
- Keys and path entries are qualified pipe_refs. Companion of `collect_signature_refs`
+ Keys and path entries are qualified pipe_refs; `visiting` holds the `visit_key` of each pipe on
+ the active path, which breaks cycles without taking one pipe for another of the same `domain.code`. Companion of `collect_signature_refs`
used to render the dep chain that reaches each unimplemented signature. When a signature
is reachable by several paths (a diamond), the longest — most informative — chain is kept.
"""
if current_path is None:
current_path = []
+ if visiting is None:
+ visiting = frozenset()
if paths is None:
paths = {}
# Cycle break: a pipe already on the active dependency path is a back edge — stop.
# Unlike a global visited set, this still explores diamonds (distinct paths that
# re-converge on the same signature), so we can keep the longest, most informative chain.
- if pipe.pipe_ref in current_path:
+ if pipe.visit_key in visiting:
return paths
if pipe.is_signature:
existing = paths.get(pipe.pipe_ref)
@@ -69,5 +74,5 @@ def collect_signature_paths(
dep_pipe = get_optional_pipe(pipe_code=dep_code)
if dep_pipe is None:
continue
- collect_signature_paths(pipe=dep_pipe, current_path=next_path, paths=paths)
+ collect_signature_paths(pipe=dep_pipe, current_path=next_path, visiting=visiting | {pipe.visit_key}, paths=paths)
return paths
diff --git a/pipelex/pipeline/file_input_consumers.py b/pipelex/pipeline/file_input_consumers.py
index a13f8c0b1..034bf0718 100644
--- a/pipelex/pipeline/file_input_consumers.py
+++ b/pipelex/pipeline/file_input_consumers.py
@@ -287,9 +287,9 @@ def _resolve_model_specs(*, model_reference: str, model_type: ModelType, visited
def _visit(*, pipe: PipeAbstract, frame: _Frame, is_conditional: bool, visiting: frozenset[str], consumers: list[FileInputConsumer]) -> None:
"""Visit one pipe with the frame of tracked variables it sees, recording the consumers it holds."""
- if pipe.pipe_ref in visiting or not frame:
+ if pipe.visit_key in visiting or not frame:
return
- visiting |= {pipe.pipe_ref}
+ visiting |= {pipe.visit_key}
if isinstance(pipe, PipeSequence):
_visit_sequence(sequence=pipe, frame=frame, is_conditional=is_conditional, visiting=visiting, consumers=consumers)
elif isinstance(pipe, PipeParallel):
@@ -371,9 +371,9 @@ def _pipe_writes(*, pipe: PipeAbstract, visiting: frozenset[str]) -> _Writes:
only its branch results, when it adds each of them. A batch runs on copies. An operator writes
only its result.
"""
- if pipe.pipe_ref in visiting:
+ if pipe.visit_key in visiting:
return _NO_WRITES
- visiting |= {pipe.pipe_ref}
+ visiting |= {pipe.visit_key}
if isinstance(pipe, PipeSequence):
return _merge_writes(
writes=[
diff --git a/pipelex/test_extras/mthds_corpus/vocabulary.toml b/pipelex/test_extras/mthds_corpus/vocabulary.toml
index dea67ae3b..bbae073b8 100644
--- a/pipelex/test_extras/mthds_corpus/vocabulary.toml
+++ b/pipelex/test_extras/mthds_corpus/vocabulary.toml
@@ -192,6 +192,10 @@ fails_at = "runtime"
code = "unresolved_pipe_dependency"
fails_at = "runtime"
+[error.unexported_pipe_dependency]
+code = "unexported_pipe_dependency"
+excluded = "It refuses a reference to a pipe an installed dependency package loads but does not export, so the bundle that produces it needs that package installed beside it, with a manifest declaring `[exports]`. A focused entry is one self-contained bundle validated where nothing is installed, and an address-based reference to a package that is not installed fails as a missing method instead. Revisit if an entry can ever carry the packages it depends on."
+
[error.unknown_model]
code = "unknown_model"
fails_at = "runtime"
diff --git a/pipelex/validation_error_types.py b/pipelex/validation_error_types.py
index e48a79866..9ae9935a8 100644
--- a/pipelex/validation_error_types.py
+++ b/pipelex/validation_error_types.py
@@ -93,6 +93,9 @@ class PipeValidationErrorType(StrEnum):
# merged library (a referenced concept or dependency pipe does not resolve).
UNRESOLVED_CONCEPT = "unresolved_concept"
UNRESOLVED_PIPE_DEPENDENCY = "unresolved_pipe_dependency"
+ # A reference from outside a dependency package to a pipe the package loads but does not export: the pipe
+ # exists, so the remedy is to export it in the package or to call one of its public pipes.
+ UNEXPORTED_PIPE_DEPENDENCY = "unexported_pipe_dependency"
# A pipe's model field names a handle, alias, preset or waterfall its model deck does not define,
# refused when the pipe is built. The item carries the field's path, the reference as written, the
@@ -131,6 +134,7 @@ class PipeValidationErrorType(StrEnum):
| PipeValidationErrorType.NATIVE_CONCEPT_REDECLARATION
| PipeValidationErrorType.UNRESOLVED_CONCEPT
| PipeValidationErrorType.UNRESOLVED_PIPE_DEPENDENCY
+ | PipeValidationErrorType.UNEXPORTED_PIPE_DEPENDENCY
| PipeValidationErrorType.UNKNOWN_MODEL
| PipeValidationErrorType.UNKNOWN_VALIDATION_ERROR
):
@@ -162,6 +166,7 @@ class PipeValidationErrorType(StrEnum):
| PipeValidationErrorType.NATIVE_CONCEPT_REDECLARATION
| PipeValidationErrorType.UNRESOLVED_CONCEPT
| PipeValidationErrorType.UNRESOLVED_PIPE_DEPENDENCY
+ | PipeValidationErrorType.UNEXPORTED_PIPE_DEPENDENCY
| PipeValidationErrorType.UNKNOWN_MODEL
| PipeValidationErrorType.UNKNOWN_VALIDATION_ERROR
):
@@ -200,6 +205,7 @@ class PipeValidationErrorType(StrEnum):
| PipeValidationErrorType.NATIVE_CONCEPT_REDECLARATION
| PipeValidationErrorType.UNRESOLVED_CONCEPT
| PipeValidationErrorType.UNRESOLVED_PIPE_DEPENDENCY
+ | PipeValidationErrorType.UNEXPORTED_PIPE_DEPENDENCY
| PipeValidationErrorType.UNKNOWN_MODEL
| PipeValidationErrorType.UNKNOWN_VALIDATION_ERROR
):
@@ -232,6 +238,7 @@ class PipeValidationErrorType(StrEnum):
| PipeValidationErrorType.INPUT_PRESENCE_VACUOUS
| PipeValidationErrorType.UNRESOLVED_CONCEPT
| PipeValidationErrorType.UNRESOLVED_PIPE_DEPENDENCY
+ | PipeValidationErrorType.UNEXPORTED_PIPE_DEPENDENCY
| PipeValidationErrorType.UNKNOWN_MODEL
| PipeValidationErrorType.UNKNOWN_VALIDATION_ERROR
):
@@ -269,6 +276,7 @@ class PipeValidationErrorType(StrEnum):
| PipeValidationErrorType.NATIVE_CONCEPT_REDECLARATION
| PipeValidationErrorType.UNRESOLVED_CONCEPT
| PipeValidationErrorType.UNRESOLVED_PIPE_DEPENDENCY
+ | PipeValidationErrorType.UNEXPORTED_PIPE_DEPENDENCY
| PipeValidationErrorType.UNKNOWN_MODEL
| PipeValidationErrorType.UNKNOWN_VALIDATION_ERROR
):
@@ -302,6 +310,7 @@ class PipeValidationErrorType(StrEnum):
| PipeValidationErrorType.NATIVE_CONCEPT_REDECLARATION
| PipeValidationErrorType.UNRESOLVED_CONCEPT
| PipeValidationErrorType.UNRESOLVED_PIPE_DEPENDENCY
+ | PipeValidationErrorType.UNEXPORTED_PIPE_DEPENDENCY
| PipeValidationErrorType.UNKNOWN_VALIDATION_ERROR
):
return False
diff --git a/tests/integration/pipelex/libraries/installed_packages.py b/tests/integration/pipelex/libraries/installed_packages.py
new file mode 100644
index 000000000..e93c733af
--- /dev/null
+++ b/tests/integration/pipelex/libraries/installed_packages.py
@@ -0,0 +1,56 @@
+"""Helpers that install invented method packages under a temporary `.mthds/methods/` and write a consumer beside them."""
+
+from pathlib import Path
+
+from pytest_mock import MockerFixture
+
+from pipelex.libraries.exceptions import LibraryLoadingError
+from pipelex.libraries.library import Library
+from pipelex.pipe_controllers.sequence.pipe_sequence import PipeSequence
+from pipelex.validation_error_types import PipeValidationErrorType
+from tests.integration.pipelex.libraries.test_data import ProbePackageTestData
+
+
+def isolate_installed_methods(*, mocker: MockerFixture, root: Path) -> None:
+ """Point package discovery at `root` and disable fetch-on-miss, so only the packages a test installs exist."""
+ mocker.patch("pipelex.cli.installed_methods.GLOBAL_METHODS_DIR", root / "global-methods")
+ mocker.patch("pipelex.cli.installed_methods.PROJECT_METHODS_DIR", root / "project-methods")
+ mocker.patch("pipelex.methods.fetch_on_miss.is_method_fetch_on_miss_enabled", return_value=False)
+
+
+def install_package(*, root: Path, method_name: str, manifest: str, bundles: dict[str, str]) -> None:
+ package_dir = root / ".mthds" / "methods" / method_name
+ package_dir.mkdir(parents=True)
+ (package_dir / "METHODS.toml").write_text(manifest, encoding="utf-8")
+ for file_name, bundle in bundles.items():
+ (package_dir / file_name).write_text(bundle, encoding="utf-8")
+
+
+def install_probe(
+ *,
+ root: Path,
+ manifest: str = ProbePackageTestData.MANIFEST_EXPORTING_EVERYTHING,
+ bundle: str = ProbePackageTestData.DEP_BUNDLE,
+) -> None:
+ install_package(root=root, method_name=ProbePackageTestData.METHOD_NAME, manifest=manifest, bundles={"probe_dep.mthds": bundle})
+
+
+def write_consumer(*, root: Path, bundles: dict[str, str]) -> list[Path]:
+ paths: list[Path] = []
+ for file_name, bundle in bundles.items():
+ path = root / file_name
+ path.write_text(bundle, encoding="utf-8")
+ paths.append(path)
+ return paths
+
+
+def sole_step_target(*, library: Library, pipe_key: str) -> str:
+ """The pipe ref the single step of the sequence held under `pipe_key` names."""
+ pipe = library.pipe_library.get_required_pipe(pipe_code=pipe_key)
+ assert isinstance(pipe, PipeSequence)
+ assert len(pipe.sequential_sub_pipes) == 1
+ return pipe.sequential_sub_pipes[0].pipe_code
+
+
+def refusal_types(exc: LibraryLoadingError) -> list[PipeValidationErrorType | None]:
+ return [item.error_type for item in exc.pipe_concept_validation_errors or []]
diff --git a/tests/integration/pipelex/libraries/test_data.py b/tests/integration/pipelex/libraries/test_data.py
index cce488a16..800721dfe 100644
--- a/tests/integration/pipelex/libraries/test_data.py
+++ b/tests/integration/pipelex/libraries/test_data.py
@@ -97,3 +97,267 @@ inputs = { query = "Text" }
output = "Digest"
prompt = "Digest the notes about $query."
"""
+
+
+class ProbePackageTestData:
+ """An installed method package whose sequence calls its own helper, and consumers of it; nothing here is real.
+
+ The helpers' descriptions say whose they are, so a graph or a resolved pipe tells which one a call reached.
+ """
+
+ METHOD_NAME: ClassVar[str] = "probe"
+ DEP_ALIAS: ClassVar[str] = "github.com/invented/probe-lib/probe"
+
+ MANIFEST_EXPORTING_EVERYTHING: ClassVar[str] = """[package]
+name = "probe"
+address = "github.com/invented/probe-lib"
+version = "1.0.0"
+description = "An invented probe package"
+
+[exports.probe_dep]
+pipes = ["entry", "helper"]
+"""
+
+ MANIFEST_EXPORTING_ONLY_THE_ENTRY: ClassVar[str] = """[package]
+name = "probe"
+address = "github.com/invented/probe-lib"
+version = "1.0.0"
+description = "An invented probe package"
+
+[exports.probe_dep]
+pipes = ["entry"]
+"""
+
+ #: `entry` calls its own `helper` by bare code, the way a package is authored. `orphan` is reached by nothing,
+ #: and names a model no deck defines, so building it would refuse.
+ DEP_BUNDLE: ClassVar[str] = """domain = "probe_dep"
+description = "An invented probe package"
+
+[pipe.entry]
+type = "PipeSequence"
+description = "Run the dependency's own helper"
+inputs = { data = "Text" }
+output = "Text"
+steps = [{ pipe = "helper", result = "helped" }]
+
+[pipe.helper]
+type = "PipeLLM"
+description = "DEPENDENCY helper"
+inputs = { data = "Text" }
+output = "Text"
+prompt = "DEPENDENCY helper: $data"
+
+[pipe.orphan]
+type = "PipeLLM"
+description = "A private pipe nothing reaches"
+inputs = { data = "Text" }
+output = "Text"
+model = "no_such_model_anywhere"
+prompt = "Orphan: $data"
+"""
+
+ #: The same package with `entry` naming its helper through the package's own alias and no domain, `alias->helper`.
+ DEP_BUNDLE_CALLING_ITS_HELPER_BY_ALIAS: ClassVar[str] = DEP_BUNDLE.replace(
+ 'steps = [{ pipe = "helper", result = "helped" }]',
+ f'steps = [{{ pipe = "{DEP_ALIAS}->helper", result = "helped" }}]',
+ )
+
+ #: The same package with `helper` a PipeFunc naming a function no registry holds, so it fails to build.
+ DEP_BUNDLE_WITH_AN_UNBUILDABLE_HELPER: ClassVar[str] = DEP_BUNDLE.replace(
+ 'type = "PipeLLM"\ndescription = "DEPENDENCY helper"\ninputs = { data = "Text" }\n'
+ 'output = "Text"\nprompt = "DEPENDENCY helper: $data"',
+ 'type = "PipeFunc"\ndescription = "DEPENDENCY helper"\ninputs = { data = "Text" }\n'
+ 'output = "Text"\nfunction_name = "no_such_function_anywhere"',
+ )
+
+ #: A consumer whose sequence calls the package's entry.
+ CONSUMER_BUNDLE: ClassVar[str] = """domain = "probe_consumer"
+description = "A consumer of the invented probe package"
+main_pipe = "go"
+
+[pipe.go]
+type = "PipeSequence"
+description = "Call the probe package"
+inputs = { data = "Text" }
+output = "Text"
+steps = [{ pipe = "github.com/invented/probe-lib/probe->probe_dep.entry", result = "out" }]
+"""
+
+ #: A second consumer bundle declaring the dependency's `domain.code` for its own helper.
+ CONSUMER_HELPER_BUNDLE: ClassVar[str] = """domain = "probe_dep"
+description = "The consumer's own pipes in a domain named like the dependency's"
+
+[pipe.helper]
+type = "PipeLLM"
+description = "CONSUMER helper"
+inputs = { data = "Text" }
+output = "Text"
+prompt = "CONSUMER helper: $data"
+"""
+
+ #: A consumer pipe of the dependency's own `domain.code`, `probe_dep.entry`, wrapping the dependency's entry.
+ CONSUMER_ENTRY_WRAPPING_THE_DEPENDENCY_BUNDLE: ClassVar[str] = """domain = "probe_dep"
+description = "The consumer's own entry, in a domain named like the dependency's"
+
+[pipe.entry]
+type = "PipeSequence"
+description = "Wrap the dependency's entry of the same domain and code"
+inputs = { data = "Text" }
+output = "Text"
+steps = [{ pipe = "github.com/invented/probe-lib/probe->probe_dep.entry", result = "out" }]
+"""
+
+ @classmethod
+ def consumer_calling(cls, *, pipe_ref: str) -> str:
+ """A consumer whose single step names `pipe_ref`."""
+ return f"""domain = "probe_consumer"
+description = "A consumer naming one pipe of the probe package"
+
+[pipe.go]
+type = "PipeSequence"
+description = "Call one pipe"
+inputs = {{ data = "Text" }}
+output = "Text"
+steps = [{{ pipe = "{pipe_ref}", result = "out" }}]
+"""
+
+
+class TwoPackagesSharingADomainTestData:
+ """Two installed packages that both declare `shared.entry` calling `shared.helper`; nothing here is real."""
+
+ PACKAGES: ClassVar[dict[str, str]] = {"left": "github.com/invented/left-lib/left", "right": "github.com/invented/right-lib/right"}
+
+ @classmethod
+ def manifest(cls, *, name: str) -> str:
+ return f"""[package]
+name = "{name}"
+address = "github.com/invented/{name}-lib"
+version = "1.0.0"
+description = "An invented package named {name}"
+"""
+
+ @classmethod
+ def bundle(cls, *, name: str) -> str:
+ return f"""domain = "shared"
+description = "The {name} package's shared domain"
+
+[pipe.entry]
+type = "PipeSequence"
+description = "Run the {name} package's helper"
+inputs = {{ data = "Text" }}
+output = "Text"
+steps = [{{ pipe = "helper", result = "helped" }}]
+
+[pipe.helper]
+type = "PipeLLM"
+description = "{name.upper()} helper"
+inputs = {{ data = "Text" }}
+output = "Text"
+prompt = "{name} helper: $data"
+"""
+
+ CONSUMER_BUNDLE: ClassVar[str] = """domain = "two_consumer"
+description = "A consumer of both invented packages"
+
+[pipe.go]
+type = "PipeSequence"
+description = "Call both packages"
+inputs = { data = "Text" }
+output = "Text"
+steps = [
+ { pipe = "github.com/invented/left-lib/left->shared.entry", result = "from_left" },
+ { pipe = "github.com/invented/right-lib/right->shared.entry", result = "out" },
+]
+"""
+
+
+class TwoDomainPackageTestData:
+ """An installed package with two domains declaring the same pipe code, exporting only one of them; nothing here is real."""
+
+ METHOD_NAME: ClassVar[str] = "twodom"
+ DEP_ALIAS: ClassVar[str] = "github.com/invented/twodom-lib/twodom"
+
+ MANIFEST: ClassVar[str] = """[package]
+name = "twodom"
+address = "github.com/invented/twodom-lib"
+version = "1.0.0"
+description = "An invented two-domain package"
+
+[exports.dom_a]
+pipes = ["x"]
+"""
+
+ BUNDLE_A: ClassVar[str] = """domain = "dom_a"
+description = "The exported domain"
+
+[pipe.x]
+type = "PipeSequence"
+description = "Call the other domain's x"
+inputs = { data = "Text" }
+output = "Text"
+steps = [{ pipe = "dom_b.x", result = "out" }]
+"""
+
+ BUNDLE_B: ClassVar[str] = """domain = "dom_b"
+description = "The private domain"
+
+[pipe.x]
+type = "PipeLLM"
+description = "The private domain's x"
+inputs = { data = "Text" }
+output = "Text"
+prompt = "x: $data"
+"""
+
+
+class FailedExportWithAPrivateNamesakeTestData:
+ """A probe package whose exported `dom_a.x` fails to build while a private `dom_b.x` of the same code loads; nothing here is real.
+
+ `dom_a.entry` names `x` through the package's own alias and no domain, and `dom_a.other` reaches the private `dom_b.x`.
+ """
+
+ MANIFEST: ClassVar[str] = """[package]
+name = "probe"
+address = "github.com/invented/probe-lib"
+version = "1.0.0"
+description = "An invented probe package"
+
+[exports.dom_a]
+pipes = ["x", "entry", "other"]
+"""
+
+ BUNDLE_A: ClassVar[str] = """domain = "dom_a"
+description = "The exported domain"
+
+[pipe.x]
+type = "PipeFunc"
+description = "The exported x, which fails to build"
+inputs = { data = "Text" }
+output = "Text"
+function_name = "no_such_function_anywhere"
+
+[pipe.entry]
+type = "PipeSequence"
+description = "Call x by the package's alias and its bare code"
+inputs = { data = "Text" }
+output = "Text"
+steps = [{ pipe = "github.com/invented/probe-lib/probe->x", result = "out" }]
+
+[pipe.other]
+type = "PipeSequence"
+description = "Reach the private domain's x"
+inputs = { data = "Text" }
+output = "Text"
+steps = [{ pipe = "dom_b.x", result = "out" }]
+"""
+
+ BUNDLE_B: ClassVar[str] = """domain = "dom_b"
+description = "The private domain"
+
+[pipe.x]
+type = "PipeLLM"
+description = "The private x"
+inputs = { data = "Text" }
+output = "Text"
+prompt = "private x: $data"
+"""
diff --git a/tests/integration/pipelex/libraries/test_dependency_pipe_scope.py b/tests/integration/pipelex/libraries/test_dependency_pipe_scope.py
new file mode 100644
index 000000000..e5dab948b
--- /dev/null
+++ b/tests/integration/pipelex/libraries/test_dependency_pipe_scope.py
@@ -0,0 +1,259 @@
+"""A dependency package's pipes call their own sub-pipes, never the consumer's, and its private pipes stay private.
+
+A dependency is registered in the consumer's library under `alias->domain.code`, and every reader of a sub-pipe ref —
+validation, the pre-run walks, execution — looks the ref up in the consumer's library. So a ref the package wrote
+must be stored in that aliased form: stored as `domain.code`, it finds nothing, and the consumer's load fails, or it
+finds the consumer's own pipe of the same `domain.code` and runs that one in the dependency's place, silently.
+
+The packages are invented and installed under a temporary `.mthds/methods/`, with fetch-on-miss disabled.
+"""
+
+from pathlib import Path
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.interpreter_hub import get_library_manager, scoped_current_library
+from pipelex.libraries.exceptions import LibraryLoadingError
+from pipelex.pipe_run.dry_run_in_process import dry_run_pipe_in_process
+from pipelex.validation_error_types import PipeValidationErrorType
+from tests.integration.pipelex.libraries.installed_packages import (
+ install_package,
+ install_probe,
+ isolate_installed_methods,
+ refusal_types,
+ sole_step_target,
+ write_consumer,
+)
+from tests.integration.pipelex.libraries.test_data import (
+ FailedExportWithAPrivateNamesakeTestData,
+ ProbePackageTestData,
+ TwoPackagesSharingADomainTestData,
+)
+
+PROBE_ALIAS = ProbePackageTestData.DEP_ALIAS
+
+
+class TestDependencyPipeScope:
+ @pytest.mark.asyncio(loop_scope="class")
+ async def test_a_dependency_runs_its_own_helper_not_the_consumer_s(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """The consumer declares `probe_dep.helper` too: the dependency's entry still reaches its own helper."""
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_probe(root=tmp_path)
+ consumer_files = write_consumer(
+ root=tmp_path,
+ bundles={
+ "consumer.mthds": ProbePackageTestData.CONSUMER_BUNDLE,
+ "consumer_probe_dep.mthds": ProbePackageTestData.CONSUMER_HELPER_BUNDLE,
+ },
+ )
+ library_manager = get_library_manager()
+ library_id, library = library_manager.open_library()
+ try:
+ library_manager.load_libraries(library_id=library_id, library_file_paths=consumer_files)
+
+ step_target = sole_step_target(library=library, pipe_key=f"{PROBE_ALIAS}->probe_dep.entry")
+ resolved_helper = library.pipe_library.get_required_pipe(pipe_code=step_target)
+ dependency_helper = library.dependency_libraries[PROBE_ALIAS].pipe_library.get_required_pipe(pipe_code="probe_dep.helper")
+ assert resolved_helper is dependency_helper
+ assert resolved_helper.description == "DEPENDENCY helper"
+
+ with scoped_current_library(library_id=library_id):
+ graph_spec = await dry_run_pipe_in_process(
+ pipe=library.pipe_library.get_required_pipe(pipe_code="probe_consumer.go"),
+ library_id=library_id,
+ )
+ finally:
+ library_manager.teardown(library_id=library_id)
+
+ graph_json = graph_spec.model_dump_json()
+ assert "DEPENDENCY helper" in graph_json
+ assert "CONSUMER helper" not in graph_json
+
+ def test_a_dependency_with_an_internal_controller_loads_alone(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """With nothing of the dependency's domain in the consumer, the load succeeds and the entry needs the helper's input."""
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_probe(root=tmp_path)
+ consumer_files = write_consumer(root=tmp_path, bundles={"consumer.mthds": ProbePackageTestData.CONSUMER_BUNDLE})
+ library_manager = get_library_manager()
+ library_id, library = library_manager.open_library()
+ try:
+ library_manager.load_libraries(library_id=library_id, library_file_paths=consumer_files)
+
+ with scoped_current_library(library_id=library_id):
+ entry = library.pipe_library.get_required_pipe(pipe_code=f"{PROBE_ALIAS}->probe_dep.entry")
+ needed_input_names = set(entry.needed_inputs().root)
+ finally:
+ library_manager.teardown(library_id=library_id)
+
+ assert needed_input_names == {"data"}
+
+ def test_two_packages_declaring_the_same_domain_code_each_call_their_own(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ for name in TwoPackagesSharingADomainTestData.PACKAGES:
+ install_package(
+ root=tmp_path,
+ method_name=name,
+ manifest=TwoPackagesSharingADomainTestData.manifest(name=name),
+ bundles={"shared.mthds": TwoPackagesSharingADomainTestData.bundle(name=name)},
+ )
+ consumer_files = write_consumer(root=tmp_path, bundles={"consumer.mthds": TwoPackagesSharingADomainTestData.CONSUMER_BUNDLE})
+ library_manager = get_library_manager()
+ library_id, library = library_manager.open_library()
+ try:
+ library_manager.load_libraries(library_id=library_id, library_file_paths=consumer_files)
+
+ reached: dict[str, str] = {}
+ for name, alias in TwoPackagesSharingADomainTestData.PACKAGES.items():
+ step_target = sole_step_target(library=library, pipe_key=f"{alias}->shared.entry")
+ reached[name] = library.pipe_library.get_required_pipe(pipe_code=step_target).description
+ finally:
+ library_manager.teardown(library_id=library_id)
+
+ assert reached == {"left": "LEFT helper", "right": "RIGHT helper"}
+
+ def test_a_missing_pipe_of_a_loaded_dependency_is_refused(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """A typo in a reference to a loaded package is reported as what it is, not as an extraneous input."""
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_probe(root=tmp_path)
+ missing_ref = f"{PROBE_ALIAS}->probe_dep.nope"
+ consumer_files = write_consumer(
+ root=tmp_path,
+ bundles={
+ "consumer.mthds": ProbePackageTestData.CONSUMER_BUNDLE,
+ "typo.mthds": ProbePackageTestData.consumer_calling(pipe_ref=missing_ref).replace("probe_consumer", "probe_typo"),
+ },
+ )
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ try:
+ with pytest.raises(LibraryLoadingError) as exc_info:
+ library_manager.load_libraries(library_id=library_id, library_file_paths=consumer_files)
+ finally:
+ library_manager.teardown(library_id=library_id)
+
+ items = exc_info.value.pipe_concept_validation_errors or []
+ assert [(item.error_type, item.pipe_code, item.missing_pipe_code) for item in items] == [
+ (PipeValidationErrorType.UNRESOLVED_PIPE_DEPENDENCY, "go", missing_ref)
+ ]
+ assert "has no pipe" in str(exc_info.value)
+
+ def test_a_crate_of_the_consumer_loads_without_its_dependency(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """A worker loads the consumer's crate with no package loaded: a reference into an alias never loaded is still tolerated."""
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_probe(root=tmp_path)
+ consumer_files = write_consumer(root=tmp_path, bundles={"consumer.mthds": ProbePackageTestData.CONSUMER_BUNDLE})
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ worker_library_id, worker_library = library_manager.open_library()
+ try:
+ library_manager.load_libraries(library_id=library_id, library_file_paths=consumer_files)
+ crate = library_manager.get_crate(library_id=library_id)
+ assert crate is not None
+
+ library_manager.load_from_crate(library_id=worker_library_id, crate=crate)
+
+ assert "probe_consumer.go" in worker_library.pipe_library.get_pipes_dict()
+ assert not worker_library.dependency_libraries
+ finally:
+ library_manager.teardown(library_id=worker_library_id)
+ library_manager.teardown(library_id=library_id)
+
+ def test_a_consumer_pipe_wrapping_a_dependency_pipe_of_the_same_domain_code_needs_its_inputs(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """The walk over needed inputs must not take the dependency's `probe_dep.entry` for the consumer's, already visited."""
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_probe(root=tmp_path)
+ consumer_files = write_consumer(
+ root=tmp_path, bundles={"consumer_probe_dep.mthds": ProbePackageTestData.CONSUMER_ENTRY_WRAPPING_THE_DEPENDENCY_BUNDLE}
+ )
+ library_manager = get_library_manager()
+ library_id, library = library_manager.open_library()
+ try:
+ library_manager.load_libraries(library_id=library_id, library_file_paths=consumer_files)
+
+ with scoped_current_library(library_id=library_id):
+ needed_input_names = set(library.pipe_library.get_required_pipe(pipe_code="probe_dep.entry").needed_inputs().root)
+ finally:
+ library_manager.teardown(library_id=library_id)
+
+ assert needed_input_names == {"data"}
+
+ def test_a_dependency_pipe_that_fails_to_build_is_refused_naming_why(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """The package declares `helper`, so a reference to it is not to a pipe the package lacks: the refusal says it failed to build."""
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_probe(root=tmp_path, bundle=ProbePackageTestData.DEP_BUNDLE_WITH_AN_UNBUILDABLE_HELPER)
+ consumer_files = write_consumer(root=tmp_path, bundles={"consumer.mthds": ProbePackageTestData.CONSUMER_BUNDLE})
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ try:
+ with pytest.raises(LibraryLoadingError) as exc_info:
+ library_manager.load_libraries(library_id=library_id, library_file_paths=consumer_files)
+ finally:
+ library_manager.teardown(library_id=library_id)
+
+ assert refusal_types(exc_info.value) == [PipeValidationErrorType.UNRESOLVED_PIPE_DEPENDENCY]
+ assert "could not build" in str(exc_info.value)
+ assert "no_such_function_anywhere" in str(exc_info.value)
+ assert "has no pipe" not in str(exc_info.value)
+
+ def test_a_bare_self_reference_to_a_failed_exported_pipe_never_falls_to_a_private_namesake(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """`alias->x` names an exported `dom_a.x` that failed to build: the load is refused, never resolved to the private `dom_b.x`."""
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_package(
+ root=tmp_path,
+ method_name=ProbePackageTestData.METHOD_NAME,
+ manifest=FailedExportWithAPrivateNamesakeTestData.MANIFEST,
+ bundles={
+ "dom_a.mthds": FailedExportWithAPrivateNamesakeTestData.BUNDLE_A,
+ "dom_b.mthds": FailedExportWithAPrivateNamesakeTestData.BUNDLE_B,
+ },
+ )
+ consumer_files = write_consumer(
+ root=tmp_path, bundles={"consumer.mthds": ProbePackageTestData.consumer_calling(pipe_ref=f"{PROBE_ALIAS}->dom_a.entry")}
+ )
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ try:
+ with pytest.raises(LibraryLoadingError) as exc_info:
+ library_manager.load_libraries(library_id=library_id, library_file_paths=consumer_files)
+ finally:
+ library_manager.teardown(library_id=library_id)
+
+ assert refusal_types(exc_info.value) == [PipeValidationErrorType.UNRESOLVED_PIPE_DEPENDENCY]
+ assert "could not build" in str(exc_info.value)
+
+ def test_a_dependency_s_reference_to_another_package_is_not_validated_against_the_consumer_s(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """A package's own dependencies are never loaded, so the consumer's package of the same alias says nothing about its reference."""
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ left_alias = TwoPackagesSharingADomainTestData.PACKAGES["left"]
+ install_probe(
+ root=tmp_path,
+ bundle=ProbePackageTestData.DEP_BUNDLE.replace(
+ '{ pipe = "helper", result = "helped" }',
+ f'{{ pipe = "helper", result = "helped" }}, {{ pipe = "{left_alias}->shared.not_in_left", result = "further" }}',
+ ),
+ )
+ install_package(
+ root=tmp_path,
+ method_name="left",
+ manifest=TwoPackagesSharingADomainTestData.manifest(name="left"),
+ bundles={"shared.mthds": TwoPackagesSharingADomainTestData.bundle(name="left")},
+ )
+ consumer_files = write_consumer(
+ root=tmp_path,
+ bundles={
+ "consumer.mthds": ProbePackageTestData.CONSUMER_BUNDLE,
+ "left.mthds": ProbePackageTestData.consumer_calling(pipe_ref=f"{left_alias}->shared.entry").replace(
+ "probe_consumer", "left_consumer"
+ ),
+ },
+ )
+ library_manager = get_library_manager()
+ library_id, library = library_manager.open_library()
+ try:
+ library_manager.load_libraries(library_id=library_id, library_file_paths=consumer_files)
+ loaded_aliases = set(library.dependency_libraries)
+ finally:
+ library_manager.teardown(library_id=library_id)
+
+ assert loaded_aliases == {PROBE_ALIAS, left_alias}
diff --git a/tests/integration/pipelex/libraries/test_dependency_private_pipes.py b/tests/integration/pipelex/libraries/test_dependency_private_pipes.py
new file mode 100644
index 000000000..922af576a
--- /dev/null
+++ b/tests/integration/pipelex/libraries/test_dependency_private_pipes.py
@@ -0,0 +1,197 @@
+"""A dependency package's private pipes travel with the exported pipes that call them, and stay private to the package.
+
+A package's public pipes are the ones its manifest exports, by domain, and its bundles' `main_pipe`. Whatever they reach
+inside the package is loaded with them; a private pipe nothing public reaches is not built at all. A consumer referencing
+a private pipe, loaded or withheld, is refused at load with `UNEXPORTED_PIPE_DEPENDENCY`.
+
+The packages are invented and installed under a temporary `.mthds/methods/`, with fetch-on-miss disabled.
+"""
+
+from pathlib import Path
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.interpreter_hub import get_library_manager, scoped_current_library
+from pipelex.libraries.exceptions import LibraryLoadingError
+from pipelex.pipe_run.dry_run_in_process import dry_run_pipe_in_process
+from pipelex.validation_error_types import PipeValidationErrorType
+from tests.integration.pipelex.libraries.installed_packages import (
+ install_package,
+ install_probe,
+ isolate_installed_methods,
+ refusal_types,
+ write_consumer,
+)
+from tests.integration.pipelex.libraries.test_data import ProbePackageTestData, TwoDomainPackageTestData
+
+PROBE_ALIAS = ProbePackageTestData.DEP_ALIAS
+
+
+class TestDependencyPrivatePipes:
+ @pytest.mark.asyncio(loop_scope="class")
+ async def test_an_exported_controller_runs_its_private_helper(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_probe(root=tmp_path, manifest=ProbePackageTestData.MANIFEST_EXPORTING_ONLY_THE_ENTRY)
+ consumer_files = write_consumer(root=tmp_path, bundles={"consumer.mthds": ProbePackageTestData.CONSUMER_BUNDLE})
+ library_manager = get_library_manager()
+ library_id, library = library_manager.open_library()
+ try:
+ library_manager.load_libraries(library_id=library_id, library_file_paths=consumer_files)
+
+ assert "probe_dep.helper" in library.dependency_libraries[PROBE_ALIAS].pipe_library.get_pipes_dict()
+ with scoped_current_library(library_id=library_id):
+ graph_spec = await dry_run_pipe_in_process(
+ pipe=library.pipe_library.get_required_pipe(pipe_code="probe_consumer.go"),
+ library_id=library_id,
+ )
+ finally:
+ library_manager.teardown(library_id=library_id)
+
+ assert "DEPENDENCY helper" in graph_spec.model_dump_json()
+
+ def test_a_consumer_cannot_reference_a_private_helper(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_probe(root=tmp_path, manifest=ProbePackageTestData.MANIFEST_EXPORTING_ONLY_THE_ENTRY)
+ private_ref = f"{PROBE_ALIAS}->probe_dep.helper"
+ consumer_files = write_consumer(
+ root=tmp_path,
+ bundles={
+ "consumer.mthds": ProbePackageTestData.CONSUMER_BUNDLE,
+ "sneaky.mthds": ProbePackageTestData.consumer_calling(pipe_ref=private_ref).replace("probe_consumer", "probe_sneaky"),
+ },
+ )
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ try:
+ with pytest.raises(LibraryLoadingError) as exc_info:
+ library_manager.load_libraries(library_id=library_id, library_file_paths=consumer_files)
+ finally:
+ library_manager.teardown(library_id=library_id)
+
+ items = exc_info.value.pipe_concept_validation_errors or []
+ assert [(item.error_type, item.pipe_code, item.missing_pipe_code) for item in items] == [
+ (PipeValidationErrorType.UNEXPORTED_PIPE_DEPENDENCY, "go", private_ref)
+ ]
+
+ @pytest.mark.parametrize("orphan_ref", [f"{PROBE_ALIAS}->probe_dep.orphan", f"{PROBE_ALIAS}->orphan"])
+ def test_a_reference_to_a_withheld_private_pipe_is_refused_as_unexported(self, tmp_path: Path, mocker: MockerFixture, orphan_ref: str) -> None:
+ """`orphan` is never built, but the package declares it: a reference to it is unexported, not missing."""
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_probe(root=tmp_path, manifest=ProbePackageTestData.MANIFEST_EXPORTING_ONLY_THE_ENTRY)
+ consumer_files = write_consumer(
+ root=tmp_path,
+ bundles={
+ "consumer.mthds": ProbePackageTestData.CONSUMER_BUNDLE,
+ "sneaky.mthds": ProbePackageTestData.consumer_calling(pipe_ref=orphan_ref).replace("probe_consumer", "probe_sneaky"),
+ },
+ )
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ try:
+ with pytest.raises(LibraryLoadingError) as exc_info:
+ library_manager.load_libraries(library_id=library_id, library_file_paths=consumer_files)
+ finally:
+ library_manager.teardown(library_id=library_id)
+
+ assert refusal_types(exc_info.value) == [PipeValidationErrorType.UNEXPORTED_PIPE_DEPENDENCY]
+ assert "does not export" in str(exc_info.value)
+
+ def test_a_private_pipe_nothing_public_reaches_is_not_loaded(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """`orphan` names a model no deck defines: it is never built, so it cannot refuse the consumer's load."""
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_probe(root=tmp_path, manifest=ProbePackageTestData.MANIFEST_EXPORTING_ONLY_THE_ENTRY)
+ consumer_files = write_consumer(root=tmp_path, bundles={"consumer.mthds": ProbePackageTestData.CONSUMER_BUNDLE})
+ library_manager = get_library_manager()
+ library_id, library = library_manager.open_library()
+ try:
+ library_manager.load_libraries(library_id=library_id, library_file_paths=consumer_files)
+ child_pipe_refs = set(library.dependency_libraries[PROBE_ALIAS].pipe_library.get_pipes_dict())
+ finally:
+ library_manager.teardown(library_id=library_id)
+
+ assert child_pipe_refs == {"probe_dep.entry", "probe_dep.helper"}
+
+ def test_exports_are_scoped_by_domain(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """Exporting `dom_a.x` does not export `dom_b.x`, though both are named `x`."""
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_package(
+ root=tmp_path,
+ method_name=TwoDomainPackageTestData.METHOD_NAME,
+ manifest=TwoDomainPackageTestData.MANIFEST,
+ bundles={"dom_a.mthds": TwoDomainPackageTestData.BUNDLE_A, "dom_b.mthds": TwoDomainPackageTestData.BUNDLE_B},
+ )
+ private_ref = f"{TwoDomainPackageTestData.DEP_ALIAS}->dom_b.x"
+ consumer_files = write_consumer(
+ root=tmp_path,
+ bundles={
+ "public.mthds": ProbePackageTestData.consumer_calling(pipe_ref=f"{TwoDomainPackageTestData.DEP_ALIAS}->dom_a.x"),
+ "sneaky.mthds": ProbePackageTestData.consumer_calling(pipe_ref=private_ref).replace("probe_consumer", "probe_sneaky"),
+ },
+ )
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ try:
+ with pytest.raises(LibraryLoadingError) as exc_info:
+ library_manager.load_libraries(library_id=library_id, library_file_paths=consumer_files)
+ finally:
+ library_manager.teardown(library_id=library_id)
+
+ assert refusal_types(exc_info.value) == [PipeValidationErrorType.UNEXPORTED_PIPE_DEPENDENCY]
+
+ def test_the_entry_door_reaches_a_private_helper_by_hand(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """`[exports]` governs references from inside a method, not a pipe a person names at a CLI argument or an API field."""
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_probe(root=tmp_path, manifest=ProbePackageTestData.MANIFEST_EXPORTING_ONLY_THE_ENTRY)
+ consumer_files = write_consumer(root=tmp_path, bundles={"consumer.mthds": ProbePackageTestData.CONSUMER_BUNDLE})
+ library_manager = get_library_manager()
+ library_id, library = library_manager.open_library()
+ try:
+ library_manager.load_libraries(library_id=library_id, library_file_paths=consumer_files)
+ helper = library.pipe_library.get_required_entry_pipe(pipe_code=f"{PROBE_ALIAS}->probe_dep.helper")
+ finally:
+ library_manager.teardown(library_id=library_id)
+
+ assert helper.description == "DEPENDENCY helper"
+
+ def test_a_private_helper_named_through_the_package_alias_travels_with_its_caller(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """`entry` names its private helper as `alias->helper`, with no domain: the helper is still loaded, and the call allowed."""
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_probe(
+ root=tmp_path,
+ manifest=ProbePackageTestData.MANIFEST_EXPORTING_ONLY_THE_ENTRY,
+ bundle=ProbePackageTestData.DEP_BUNDLE_CALLING_ITS_HELPER_BY_ALIAS,
+ )
+ consumer_files = write_consumer(root=tmp_path, bundles={"consumer.mthds": ProbePackageTestData.CONSUMER_BUNDLE})
+ library_manager = get_library_manager()
+ library_id, library = library_manager.open_library()
+ try:
+ library_manager.load_libraries(library_id=library_id, library_file_paths=consumer_files)
+ child_pipe_refs = set(library.dependency_libraries[PROBE_ALIAS].pipe_library.get_pipes_dict())
+ finally:
+ library_manager.teardown(library_id=library_id)
+
+ assert child_pipe_refs == {"probe_dep.entry", "probe_dep.helper"}
+
+ def test_a_private_pipe_sharing_an_exported_pipe_s_code_does_not_make_a_bare_reference_ambiguous(
+ self, tmp_path: Path, mocker: MockerFixture
+ ) -> None:
+ """`dom_a.x` is exported and calls the private `dom_b.x`: a consumer's `alias->x` means the exported one."""
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_package(
+ root=tmp_path,
+ method_name=TwoDomainPackageTestData.METHOD_NAME,
+ manifest=TwoDomainPackageTestData.MANIFEST,
+ bundles={"dom_a.mthds": TwoDomainPackageTestData.BUNDLE_A, "dom_b.mthds": TwoDomainPackageTestData.BUNDLE_B},
+ )
+ bare_ref = f"{TwoDomainPackageTestData.DEP_ALIAS}->x"
+ consumer_files = write_consumer(root=tmp_path, bundles={"consumer.mthds": ProbePackageTestData.consumer_calling(pipe_ref=bare_ref)})
+ library_manager = get_library_manager()
+ library_id, library = library_manager.open_library()
+ try:
+ library_manager.load_libraries(library_id=library_id, library_file_paths=consumer_files)
+ resolved = library.pipe_library.get_required_pipe(pipe_code=bare_ref)
+ finally:
+ library_manager.teardown(library_id=library_id)
+
+ assert resolved.pipe_ref == "dom_a.x"
diff --git a/tests/unit/pipelex/libraries/test_crate_qualification.py b/tests/unit/pipelex/libraries/test_crate_qualification.py
index 02ebd6f3b..ff434a639 100644
--- a/tests/unit/pipelex/libraries/test_crate_qualification.py
+++ b/tests/unit/pipelex/libraries/test_crate_qualification.py
@@ -15,7 +15,7 @@ qualified ref out of the result for that reason.
import pytest
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
-from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldType
+from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint
from pipelex.core.domains.domain_blueprint import DomainBlueprint
from pipelex.libraries.crate_qualification import QualifiedCrateContent, qualify_crate
from pipelex.libraries.exceptions import CrateNormalizationError
@@ -27,57 +27,7 @@ from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParall
from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
-
-
-def _crate() -> LibraryCrate:
- """A crate carrying one bare ref of every kind the pass advertises."""
- return LibraryCrate(
- concepts={
- "alpha.Category": "a category",
- "alpha.Report": ConceptBlueprint(
- description="a report",
- structure={
- "label": ConceptStructureBlueprint(
- description="the label",
- type=ConceptStructureBlueprintFieldType.CONCEPT,
- concept_ref="Category",
- ),
- "tags": ConceptStructureBlueprint(
- description="the tags",
- type=ConceptStructureBlueprintFieldType.LIST,
- item_type=ConceptStructureBlueprintFieldType.CONCEPT,
- item_concept_ref="Category",
- ),
- },
- ),
- "alpha.Detailed": ConceptBlueprint(description="a detailed report", refines="Report"),
- },
- pipes={
- "alpha.leaf": PipeLLMBlueprint(description="leaf", inputs={"item": "Category"}, output="Report", prompt="use $item"),
- "alpha.seq": PipeSequenceBlueprint(description="seq", output="Report", steps=[SubPipeBlueprint(pipe="leaf")]),
- "alpha.par": PipeParallelBlueprint(
- description="par",
- output="Composite",
- branches=[SubPipeBlueprint(pipe="leaf", result="one")],
- ),
- "alpha.cond": PipeConditionBlueprint(
- description="cond",
- output="Report",
- expression="x",
- outcomes={"hit": "leaf"},
- default_outcome="leaf",
- ),
- "alpha.batched": PipeBatchBlueprint(
- description="batch",
- inputs={"items": "Category[]"},
- output="Report[]",
- branch_pipe_code="leaf",
- input_list_name="items",
- input_item_name="item",
- ),
- },
- domains={"alpha": DomainBlueprint(code="alpha", description="alpha domain")},
- )
+from tests.unit.pipelex.libraries.test_data import CrateQualificationTestData
def _report_structure(result: QualifiedCrateContent) -> dict[str, ConceptStructureBlueprint]:
@@ -91,49 +41,49 @@ class TestCrateQualification:
# --- every ref kind the pass advertises, read back out of the result ---
def test_concept_refines(self):
- detailed = qualify_crate(_crate()).concepts["alpha.Detailed"]
+ detailed = qualify_crate(CrateQualificationTestData.crate()).concepts["alpha.Detailed"]
assert isinstance(detailed, ConceptBlueprint)
assert detailed.refines == "alpha.Report"
def test_structure_field_refs(self):
- structure = _report_structure(qualify_crate(_crate()))
+ structure = _report_structure(qualify_crate(CrateQualificationTestData.crate()))
assert structure["label"].concept_ref == "alpha.Category"
assert structure["tags"].item_concept_ref == "alpha.Category"
def test_pipe_io_refs(self):
- leaf = qualify_crate(_crate()).pipes["alpha.leaf"]
+ leaf = qualify_crate(CrateQualificationTestData.crate()).pipes["alpha.leaf"]
assert leaf.inputs == {"item": "alpha.Category"}
assert leaf.output == "alpha.Report"
def test_multiplicity_markers_survive_qualification(self):
- batched = qualify_crate(_crate()).pipes["alpha.batched"]
+ batched = qualify_crate(CrateQualificationTestData.crate()).pipes["alpha.batched"]
assert batched.inputs == {"items": "alpha.Category[]"}
assert batched.output == "alpha.Report[]"
def test_sequence_step_refs(self):
- sequence = qualify_crate(_crate()).pipes["alpha.seq"]
+ sequence = qualify_crate(CrateQualificationTestData.crate()).pipes["alpha.seq"]
assert isinstance(sequence, PipeSequenceBlueprint)
assert [step.pipe for step in sequence.steps] == ["alpha.leaf"]
def test_parallel_branch_refs(self):
- parallel = qualify_crate(_crate()).pipes["alpha.par"]
+ parallel = qualify_crate(CrateQualificationTestData.crate()).pipes["alpha.par"]
assert isinstance(parallel, PipeParallelBlueprint)
assert [branch.pipe for branch in parallel.branches] == ["alpha.leaf"]
def test_condition_outcome_refs(self):
- condition = qualify_crate(_crate()).pipes["alpha.cond"]
+ condition = qualify_crate(CrateQualificationTestData.crate()).pipes["alpha.cond"]
assert isinstance(condition, PipeConditionBlueprint)
assert condition.outcomes == {"hit": "alpha.leaf"}
assert condition.default_outcome == "alpha.leaf"
def test_batch_branch_ref(self):
- batched = qualify_crate(_crate()).pipes["alpha.batched"]
+ batched = qualify_crate(CrateQualificationTestData.crate()).pipes["alpha.batched"]
assert isinstance(batched, PipeBatchBlueprint)
assert batched.branch_pipe_code == "alpha.leaf"
def test_string_described_concept_passes_through(self):
"""A description holds no refs — it must come back as the same string, not a blueprint."""
- assert qualify_crate(_crate()).concepts["alpha.Category"] == "a category"
+ assert qualify_crate(CrateQualificationTestData.crate()).concepts["alpha.Category"] == "a category"
# --- the rule itself: a bare ref is its own domain's, whoever else declares the code ---
@@ -144,7 +94,7 @@ class TestCrateQualification:
only one domain exists, so every other test in this module would pass just as happily under
the deleted rule. This one is what pins the rule that was actually chosen.
"""
- crate = _crate()
+ crate = CrateQualificationTestData.crate()
crate.pipes["beta.only_here"] = PipeLLMBlueprint(description="beta's pipe", output="Text", prompt="go")
crate.pipes["alpha.calls_sibling"] = PipeSequenceBlueprint(
description="calls a code only beta declares",
@@ -161,7 +111,7 @@ class TestCrateQualification:
"""Ambiguity was a consequence of searching. Nothing is searched for, so nothing is ambiguous —
an unrelated domain declaring `leaf` cannot change what `alpha.seq` means.
"""
- crate = _crate()
+ crate = CrateQualificationTestData.crate()
crate.pipes["beta.leaf"] = PipeLLMBlueprint(description="beta's own leaf", output="Text", prompt="go")
crate.domains["beta"] = DomainBlueprint(code="beta", description="beta domain")
@@ -171,7 +121,7 @@ class TestCrateQualification:
def test_cross_package_refs_pass_through_untouched(self):
"""`alias->…` refs are the packaging project's to canonicalize; this pass must not touch them."""
- crate = _crate()
+ crate = CrateQualificationTestData.crate()
crate.pipes["alpha.calls_dependency"] = PipeSequenceBlueprint(
description="calls into a dependency",
output="Text",
@@ -188,7 +138,7 @@ class TestCrateQualification:
Exempting it everywhere — which is what the pass did until a review caught it — leaves a real
reference unqualified, and under the strict lookup an unqualified ref resolves to nothing.
"""
- crate = _crate()
+ crate = CrateQualificationTestData.crate()
crate.pipes["alpha.continue"] = PipeLLMBlueprint(description="a pipe genuinely named continue", output="Text", prompt="go")
crate.pipes["alpha.calls_continue"] = PipeSequenceBlueprint(
description="a step naming that pipe",
@@ -202,7 +152,7 @@ class TestCrateQualification:
def test_special_outcomes_pass_through_untouched(self):
"""`fail` / `continue` are outcomes, not pipe refs — in a condition, and only there."""
- crate = _crate()
+ crate = CrateQualificationTestData.crate()
crate.pipes["alpha.route_special"] = PipeConditionBlueprint(
description="route to special outcomes",
inputs={"data": "Category"},
@@ -221,7 +171,7 @@ class TestCrateQualification:
def test_input_crate_is_not_mutated(self):
"""The pass runs on a crate the caller still holds — in the library build, one it has cached."""
- crate = _crate()
+ crate = CrateQualificationTestData.crate()
qualify_crate(crate)
sequence = crate.pipes["alpha.seq"]
@@ -236,8 +186,8 @@ class TestCrateQualification:
"""Qualifying an already-qualified ref must be a no-op — `alpha.leaf` must not become
`alpha.alpha.leaf`. The pass runs in more than one place, so a crate can meet it twice.
"""
- once = qualify_crate(_crate())
- twice = qualify_crate(_crate().model_copy(update={"concepts": once.concepts, "pipes": once.pipes}))
+ once = qualify_crate(CrateQualificationTestData.crate())
+ twice = qualify_crate(CrateQualificationTestData.crate().model_copy(update={"concepts": once.concepts, "pipes": once.pipes}))
assert twice.concepts == once.concepts
assert twice.pipes == once.pipes
# Anti-vacuity: two no-ops would also compare equal, so pin that the compared value is the
diff --git a/tests/unit/pipelex/libraries/test_crate_qualification_under_package_alias.py b/tests/unit/pipelex/libraries/test_crate_qualification_under_package_alias.py
new file mode 100644
index 000000000..ee660d094
--- /dev/null
+++ b/tests/unit/pipelex/libraries/test_crate_qualification_under_package_alias.py
@@ -0,0 +1,80 @@
+from pipelex.libraries.crate_qualification import qualify_crate
+from pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint
+from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint
+from pipelex.pipe_controllers.condition.special_outcome import SpecialOutcome
+from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
+from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
+from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
+from tests.unit.pipelex.libraries.test_data import CrateQualificationTestData
+
+
+class TestCrateQualificationUnderAPackageAlias:
+ """With `package_alias`, a dependency's own pipe refs become the key the consumer's library holds them under.
+
+ A dependency's pipes are registered in the consumer's library as `alias->domain.code`, and every reader of a
+ sub-pipe ref looks it up there. So a ref the package wrote must be stored in that form, or it finds nothing, or
+ it finds the consumer's own pipe of the same `domain.code`.
+ """
+
+ ALIAS = "github.com/invented/alpha-lib/alpha"
+
+ def _qualified_pipes(self) -> dict[str, object]:
+ crate = CrateQualificationTestData.crate()
+ crate.pipes["alpha.route_special"] = PipeConditionBlueprint(
+ description="route to a special outcome",
+ inputs={"data": "Category"},
+ output="Report",
+ expression="x",
+ outcomes={"stop": SpecialOutcome.FAIL, "hit": "leaf"},
+ default_outcome="leaf",
+ )
+ crate.pipes["alpha.calls_other_package"] = PipeSequenceBlueprint(
+ description="calls a package of its own",
+ output="Text",
+ steps=[SubPipeBlueprint(pipe="other->beta.helper")],
+ )
+ return dict(qualify_crate(crate, package_alias=self.ALIAS).pipes)
+
+ def test_every_controller_kind_is_aliased(self):
+ pipes = self._qualified_pipes()
+ aliased_leaf = f"{self.ALIAS}->alpha.leaf"
+
+ sequence = pipes["alpha.seq"]
+ assert isinstance(sequence, PipeSequenceBlueprint)
+ assert sequence.steps[0].pipe == aliased_leaf
+ parallel = pipes["alpha.par"]
+ assert isinstance(parallel, PipeParallelBlueprint)
+ assert parallel.branches[0].pipe == aliased_leaf
+ condition = pipes["alpha.cond"]
+ assert isinstance(condition, PipeConditionBlueprint)
+ assert condition.outcomes == {"hit": aliased_leaf}
+ assert condition.default_outcome == aliased_leaf
+ batch = pipes["alpha.batched"]
+ assert isinstance(batch, PipeBatchBlueprint)
+ assert batch.branch_pipe_code == aliased_leaf
+
+ def test_concept_refs_are_not_aliased(self):
+ """Only pipe refs move: a dependency's concepts are bound at construction, under their `domain.Code`."""
+ leaf = self._qualified_pipes()["alpha.leaf"]
+ assert isinstance(leaf, PipeLLMBlueprint)
+ assert leaf.output == "alpha.Report"
+
+ def test_special_outcomes_and_already_aliased_refs_are_left_alone(self):
+ pipes = self._qualified_pipes()
+ condition = pipes["alpha.route_special"]
+ assert isinstance(condition, PipeConditionBlueprint)
+ assert condition.outcomes == {"stop": SpecialOutcome.FAIL, "hit": f"{self.ALIAS}->alpha.leaf"}
+ other = pipes["alpha.calls_other_package"]
+ assert isinstance(other, PipeSequenceBlueprint)
+ assert other.steps[0].pipe == "other->beta.helper"
+
+ def test_is_idempotent(self):
+ once = qualify_crate(CrateQualificationTestData.crate(), package_alias=self.ALIAS)
+ twice = qualify_crate(
+ CrateQualificationTestData.crate().model_copy(update={"concepts": once.concepts, "pipes": once.pipes}), package_alias=self.ALIAS
+ )
+ assert twice.pipes == once.pipes
+ sequence = twice.pipes["alpha.seq"]
+ assert isinstance(sequence, PipeSequenceBlueprint)
+ assert sequence.steps[0].pipe == f"{self.ALIAS}->alpha.leaf"
diff --git a/tests/unit/pipelex/libraries/test_data.py b/tests/unit/pipelex/libraries/test_data.py
new file mode 100644
index 000000000..5374fd063
--- /dev/null
+++ b/tests/unit/pipelex/libraries/test_data.py
@@ -0,0 +1,65 @@
+from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
+from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldType
+from pipelex.core.domains.domain_blueprint import DomainBlueprint
+from pipelex.libraries.library_crate import LibraryCrate
+from pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint
+from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint
+from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
+from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
+from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
+
+
+class CrateQualificationTestData:
+ """A crate for the qualification pass, built fresh on each call since a test may change it."""
+
+ @classmethod
+ def crate(cls) -> LibraryCrate:
+ """A crate carrying one bare ref of every kind the pass advertises."""
+ return LibraryCrate(
+ concepts={
+ "alpha.Category": "a category",
+ "alpha.Report": ConceptBlueprint(
+ description="a report",
+ structure={
+ "label": ConceptStructureBlueprint(
+ description="the label",
+ type=ConceptStructureBlueprintFieldType.CONCEPT,
+ concept_ref="Category",
+ ),
+ "tags": ConceptStructureBlueprint(
+ description="the tags",
+ type=ConceptStructureBlueprintFieldType.LIST,
+ item_type=ConceptStructureBlueprintFieldType.CONCEPT,
+ item_concept_ref="Category",
+ ),
+ },
+ ),
+ "alpha.Detailed": ConceptBlueprint(description="a detailed report", refines="Report"),
+ },
+ pipes={
+ "alpha.leaf": PipeLLMBlueprint(description="leaf", inputs={"item": "Category"}, output="Report", prompt="use $item"),
+ "alpha.seq": PipeSequenceBlueprint(description="seq", output="Report", steps=[SubPipeBlueprint(pipe="leaf")]),
+ "alpha.par": PipeParallelBlueprint(
+ description="par",
+ output="Composite",
+ branches=[SubPipeBlueprint(pipe="leaf", result="one")],
+ ),
+ "alpha.cond": PipeConditionBlueprint(
+ description="cond",
+ output="Report",
+ expression="x",
+ outcomes={"hit": "leaf"},
+ default_outcome="leaf",
+ ),
+ "alpha.batched": PipeBatchBlueprint(
+ description="batch",
+ inputs={"items": "Category[]"},
+ output="Report[]",
+ branch_pipe_code="leaf",
+ input_list_name="items",
+ input_item_name="item",
+ ),
+ },
+ domains={"alpha": DomainBlueprint(code="alpha", description="alpha domain")},
+ )
diff --git a/tests/unit/pipelex/libraries/test_dependency_pipe_reachability.py b/tests/unit/pipelex/libraries/test_dependency_pipe_reachability.py
new file mode 100644
index 000000000..6eae4a2a6
--- /dev/null
+++ b/tests/unit/pipelex/libraries/test_dependency_pipe_reachability.py
@@ -0,0 +1,106 @@
+from typing import TYPE_CHECKING
+
+from pipelex.libraries.library_manager import reachable_dependency_pipe_refs
+from pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint
+from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint
+from pipelex.pipe_controllers.condition.special_outcome import SpecialOutcome
+from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
+from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
+from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
+
+if TYPE_CHECKING:
+ from pipelex.mthds_parsing.pipelex_bundle_blueprint import PipeBlueprintUnion
+
+ALIAS = "github.com/invented/alpha-lib/alpha"
+
+
+def _leaf(*, description: str) -> PipeLLMBlueprint:
+ return PipeLLMBlueprint(description=description, inputs={"data": "Text"}, output="Text", prompt="use $data")
+
+
+def _sequence(*step_refs: str) -> PipeSequenceBlueprint:
+ return PipeSequenceBlueprint(description="seq", output="Text", steps=[SubPipeBlueprint(pipe=step_ref) for step_ref in step_refs])
+
+
+class TestDependencyPipeReachability:
+ """The pipes a dependency loads are its public ones and whatever they reach through the package's own references."""
+
+ def test_every_controller_kind_is_followed_across_several_hops(self):
+ pipes: dict[str, PipeBlueprintUnion] = {
+ "alpha.entry": _sequence(f"{ALIAS}->alpha.router"),
+ "alpha.router": PipeConditionBlueprint(
+ description="route",
+ inputs={"data": "Text"},
+ output="Text",
+ expression="data",
+ outcomes={"fan": f"{ALIAS}->alpha.fan", "stop": SpecialOutcome.FAIL},
+ default_outcome=f"{ALIAS}->alpha.fallback",
+ ),
+ "alpha.fan": PipeParallelBlueprint(
+ description="fan",
+ output="Composite",
+ branches=[SubPipeBlueprint(pipe=f"{ALIAS}->alpha.batched", result="one")],
+ ),
+ "alpha.batched": PipeBatchBlueprint(
+ description="batch",
+ inputs={"items": "Text[]"},
+ output="Text[]",
+ branch_pipe_code=f"{ALIAS}->alpha.leaf",
+ input_list_name="items",
+ input_item_name="item",
+ ),
+ "alpha.fallback": _leaf(description="fallback"),
+ "alpha.leaf": _leaf(description="leaf"),
+ "alpha.orphan": _leaf(description="orphan"),
+ }
+
+ reachable = reachable_dependency_pipe_refs(public_pipe_refs={"alpha.entry"}, qualified_pipes=pipes, package_alias=ALIAS)
+
+ assert reachable == {"alpha.entry", "alpha.router", "alpha.fan", "alpha.batched", "alpha.fallback", "alpha.leaf"}
+
+ def test_a_cycle_terminates(self):
+ pipes: dict[str, PipeBlueprintUnion] = {
+ "alpha.ping": _sequence(f"{ALIAS}->alpha.pong"),
+ "alpha.pong": _sequence(f"{ALIAS}->alpha.ping", f"{ALIAS}->alpha.leaf"),
+ "alpha.leaf": _leaf(description="leaf"),
+ }
+
+ reachable = reachable_dependency_pipe_refs(public_pipe_refs={"alpha.ping"}, qualified_pipes=pipes, package_alias=ALIAS)
+
+ assert reachable == {"alpha.ping", "alpha.pong", "alpha.leaf"}
+
+ def test_a_ref_without_its_domain_reaches_every_pipe_of_that_code(self):
+ """`alias->leaf` resolves by code within the package; where two domains declare it, both are kept for lookup to refuse."""
+ pipes: dict[str, PipeBlueprintUnion] = {
+ "alpha.entry": _sequence(f"{ALIAS}->leaf"),
+ "alpha.leaf": _leaf(description="alpha leaf"),
+ "beta.leaf": _leaf(description="beta leaf"),
+ "beta.other": _leaf(description="unreached"),
+ }
+
+ reachable = reachable_dependency_pipe_refs(public_pipe_refs={"alpha.entry"}, qualified_pipes=pipes, package_alias=ALIAS)
+
+ assert reachable == {"alpha.entry", "alpha.leaf", "beta.leaf"}
+
+ def test_refs_outside_the_package_or_to_undeclared_pipes_are_not_followed(self):
+ pipes: dict[str, PipeBlueprintUnion] = {
+ "alpha.entry": _sequence("other->alpha.leaf", f"{ALIAS}->alpha.missing", f"{ALIAS}->missing"),
+ "alpha.leaf": _leaf(description="leaf"),
+ }
+
+ reachable = reachable_dependency_pipe_refs(public_pipe_refs={"alpha.entry", "alpha.not_declared"}, qualified_pipes=pipes, package_alias=ALIAS)
+
+ assert reachable == {"alpha.entry"}
+
+ def test_a_ref_without_its_domain_follows_only_the_exported_match_when_there_is_one(self):
+ """Lookup resolves `alias->x` to the exported `alpha.x`, so the private `beta.x` is never reached, nor built."""
+ pipes: dict[str, PipeBlueprintUnion] = {
+ "alpha.entry": _sequence(f"{ALIAS}->x"),
+ "alpha.x": _leaf(description="exported x"),
+ "beta.x": _sequence(f"{ALIAS}->beta.missing"),
+ }
+
+ reachable = reachable_dependency_pipe_refs(public_pipe_refs={"alpha.entry", "alpha.x"}, qualified_pipes=pipes, package_alias=ALIAS)
+
+ assert reachable == {"alpha.entry", "alpha.x"}
diff --git a/tests/unit/pipelex/libraries/test_dependency_preliminary_text_export.py b/tests/unit/pipelex/libraries/test_dependency_preliminary_text_export.py
index a6f287b05..f8fe007a0 100644
--- a/tests/unit/pipelex/libraries/test_dependency_preliminary_text_export.py
+++ b/tests/unit/pipelex/libraries/test_dependency_preliminary_text_export.py
@@ -1,5 +1,5 @@
"""Tests that synthetic helpers from build-time elaboration are loaded alongside their parent
-when a dependency manifest restricts exports.
+when a dependency manifest restricts exports: the exported closure follows the parent's steps.
Without this behavior, exporting a `preliminary_text` PipeLLM would let the parent pipe load
into the consumer library while its `__draft_text` and `__structure` helpers are filtered out,
@@ -9,7 +9,7 @@ producing a runtime resolution failure inside the wrapping PipeSequence.
from pathlib import Path
from mthds.package.dependency_resolver import ResolvedDependency
-from mthds.package.manifest.schema import MethodsManifest
+from mthds.package.manifest.schema import DomainExports, MethodsManifest
from pytest_mock import MockerFixture
from pipelex.interpreter_hub import get_library_manager
@@ -40,10 +40,12 @@ class TestDependencyPreliminaryTextExport:
mthds_file = tmp_path / "review_dep.mthds"
mthds_file.write_text(DEP_MTHDS, encoding="utf-8")
+ # The public set is read by domain from the manifest's `[exports]`, so the manifest declares what `exported` names.
manifest = MethodsManifest(
address="github.com/org/review-dep",
version="1.0.0",
description="A dep package",
+ exports={"review_dep": DomainExports(pipes=sorted(exported))} if exported else {},
)
return ResolvedDependency(
alias="review_dep",
@@ -61,8 +63,8 @@ class TestDependencyPreliminaryTextExport:
) -> None:
"""When the manifest exports `make_review`, both synthetic helpers must also load.
- The parent is a PipeSequence post-elaboration; it references the helpers by bare code,
- so they must be present in the child library or the consumer crashes at run time.
+ The parent is a PipeSequence post-elaboration that calls the helpers, so the exported closure
+ carries them, or the consumer would crash at run time.
"""
# Hub interaction is required for PipeFactory to resolve concepts during the dep load.
# We use the live hub but a fresh empty library to keep the test isolated.
diff --git a/tests/unit/pipelex/libraries/test_dependency_ref_qualification.py b/tests/unit/pipelex/libraries/test_dependency_ref_qualification.py
index e5b8242cc..ba6497695 100644
--- a/tests/unit/pipelex/libraries/test_dependency_ref_qualification.py
+++ b/tests/unit/pipelex/libraries/test_dependency_ref_qualification.py
@@ -1,17 +1,15 @@
-"""A dependency package's own in-body refs are qualified to the dependency's own domain.
+"""A dependency package's own in-body refs are qualified to its own domain, under the alias it is loaded as.
The dependency loader builds its own crate and its own child library — it does not go through
`load_from_crate` — so it is a second, independent crate-to-pipes path, and the qualification pass
-has to be wired into both. A miss on this one is invisible: every other test in this suite loads a
-single-domain library, where a bare ref and an owner-qualified ref name the same pipe, so nothing
-observes the difference. Removing the pass from the dependency path reddens *nothing* in the whole
-suite without this module — which is exactly why it exists.
-
-What this does NOT claim: that a dependency's bare sub-pipe ref *resolves* at run time. It does not,
-and that is a pre-existing defect deferred to the packaging project (execution consults the host
-library unconditionally, with no package scope). The claim here is narrower and is the part this
-change owns — the ref stored on the built pipe is the dependency's own qualified ref, which is what
-makes the eventual package-scoped lookup a direct key hit in the child library.
+has to be wired into both. A miss on this one is invisible to every test that loads a single-domain
+library, where a bare ref and an owner-qualified ref name the same pipe — which is why this module
+exists.
+
+The ref stored on the built pipe is `alias->domain.code`, the key the consumer's library holds the
+dependency's pipe under, so every reader of the ref reaches the package's own pipe and never a
+consumer pipe of the same `domain.code`. That the ref resolves end to end, at load and in a dry run,
+is pinned by `tests/integration/pipelex/libraries/test_dependency_pipe_scope.py`.
"""
from pathlib import Path
@@ -101,5 +99,6 @@ class TestDependencyRefQualification:
render_chart = child_library.pipe_library.get_required_pipe("charts_dep.render_chart")
# Its own domain's helper — not the sibling's, and not a bare code the strict lookup would
- # never resolve.
- assert render_chart.pipe_dependencies() == {"charts_dep.prepare_series"}
+ # never resolve — under the alias, the key the consumer's library holds it under.
+ assert render_chart.pipe_dependencies() == {"charts_dep->charts_dep.prepare_series"}
+ assert library.pipe_library.get_required_pipe(pipe_code="charts_dep->charts_dep.prepare_series").domain_code == "charts_dep"
diff --git a/tests/unit/pipelex/libraries/test_pipe_library_lookup.py b/tests/unit/pipelex/libraries/test_pipe_library_lookup.py
index 9eff00d71..92ff9160f 100644
--- a/tests/unit/pipelex/libraries/test_pipe_library_lookup.py
+++ b/tests/unit/pipelex/libraries/test_pipe_library_lookup.py
@@ -119,8 +119,8 @@ class TestPipeLibraryLookup:
`__cause__`.
"""
library = PipeLibrary.make_empty()
- library.add_dependency_pipe(alias="lib", pipe=_make_stub_pipe(mocker, code="compute_score", domain_code="scoring"))
- library.add_dependency_pipe(alias="lib", pipe=_make_stub_pipe(mocker, code="compute_score", domain_code="analytics"))
+ library.add_dependency_pipe(alias="lib", pipe=_make_stub_pipe(mocker, code="compute_score", domain_code="scoring"), is_exported=True)
+ library.add_dependency_pipe(alias="lib", pipe=_make_stub_pipe(mocker, code="compute_score", domain_code="analytics"), is_exported=True)
with pytest.raises(EntryPipeAmbiguousError) as exc_info:
library.get_optional_entry_pipe("lib->compute_score")
@@ -153,8 +153,8 @@ class TestPipeLibraryLookup:
bundle — so the entry classification must not leak onto them.
"""
library = PipeLibrary.make_empty()
- library.add_dependency_pipe(alias="lib", pipe=_make_stub_pipe(mocker, code="compute_score", domain_code="scoring"))
- library.add_dependency_pipe(alias="lib", pipe=_make_stub_pipe(mocker, code="compute_score", domain_code="analytics"))
+ library.add_dependency_pipe(alias="lib", pipe=_make_stub_pipe(mocker, code="compute_score", domain_code="scoring"), is_exported=True)
+ library.add_dependency_pipe(alias="lib", pipe=_make_stub_pipe(mocker, code="compute_score", domain_code="analytics"), is_exported=True)
with pytest.raises(PipeLibraryError) as exc_info:
library.get_optional_pipe("lib->compute_score")
@@ -179,7 +179,7 @@ class TestPipeLibraryLookup:
library = PipeLibrary.make_empty()
host_pipe = _make_stub_pipe(mocker, code="compute_score", domain_code="scoring")
library.root["scoring.compute_score"] = host_pipe
- library.add_dependency_pipe(alias="lib", pipe=_make_stub_pipe(mocker, code="compute_score", domain_code="vendor"))
+ library.add_dependency_pipe(alias="lib", pipe=_make_stub_pipe(mocker, code="compute_score", domain_code="vendor"), is_exported=True)
assert library.get_optional_entry_pipe("compute_score") is host_pipe
@@ -227,7 +227,7 @@ class TestPipeLibraryLookup:
"""Cross-package ref with domain-qualified remainder resolves correctly."""
library = PipeLibrary.make_empty()
mock_pipe = _make_stub_pipe(mocker, code="compute_score", domain_code="scoring")
- library.add_dependency_pipe(alias="lib", pipe=mock_pipe)
+ library.add_dependency_pipe(alias="lib", pipe=mock_pipe, is_exported=True)
result = library.get_optional_pipe("lib->scoring.compute_score")
assert result is mock_pipe
@@ -235,7 +235,7 @@ class TestPipeLibraryLookup:
"""Cross-package ref with wrong domain returns None."""
library = PipeLibrary.make_empty()
mock_pipe = _make_stub_pipe(mocker, code="compute_score", domain_code="scoring")
- library.add_dependency_pipe(alias="lib", pipe=mock_pipe)
+ library.add_dependency_pipe(alias="lib", pipe=mock_pipe, is_exported=True)
result = library.get_optional_pipe("lib->wrong_domain.compute_score")
assert result is None
@@ -243,7 +243,7 @@ class TestPipeLibraryLookup:
"""Cross-package ref with bare code resolves when unambiguous."""
library = PipeLibrary.make_empty()
mock_pipe = _make_stub_pipe(mocker, code="compute_score", domain_code="scoring")
- library.add_dependency_pipe(alias="lib", pipe=mock_pipe)
+ library.add_dependency_pipe(alias="lib", pipe=mock_pipe, is_exported=True)
result = library.get_optional_pipe("lib->compute_score")
assert result is mock_pipe
@@ -338,8 +338,8 @@ class TestPipeLibraryOwnPipes:
own = _make_stub_pipe(mocker, code="compute_score", domain_code="scoring")
library.add_new_pipe(own)
# A dependency sharing the host's `domain.code`: keyed by alias, so it lands beside the host's.
- library.add_dependency_pipe(alias="lib", pipe=_make_stub_pipe(mocker, code="compute_score", domain_code="scoring"))
- library.add_dependency_pipe(alias="lib", pipe=_make_stub_pipe(mocker, code="other", domain_code="analytics"))
+ library.add_dependency_pipe(alias="lib", pipe=_make_stub_pipe(mocker, code="compute_score", domain_code="scoring"), is_exported=True)
+ library.add_dependency_pipe(alias="lib", pipe=_make_stub_pipe(mocker, code="other", domain_code="analytics"), is_exported=True)
assert library.get_own_pipes() == [own]
assert len(library.get_pipes()) == 3
diff --git a/tests/unit/pipelex/migration/test_replay_properties.py b/tests/unit/pipelex/migration/test_replay_properties.py
index 6f8d4e788..69bdf15b9 100644
--- a/tests/unit/pipelex/migration/test_replay_properties.py
+++ b/tests/unit/pipelex/migration/test_replay_properties.py
@@ -45,6 +45,9 @@ PROPERTY_SETTINGS = settings(max_examples=MAX_EXAMPLES, deadline=None)
"""Bounded on purpose. `deadline=None` because these run under `xdist`, where a per-example
wall-clock deadline measures machine contention rather than anything about the engine."""
+FIND_SETTINGS = settings(max_examples=2000, deadline=None)
+"""`find`'s own search budget, with the deadline off for the same reason as `PROPERTY_SETTINGS`."""
+
REAL_SURFACES = build_config_surface_registry().surfaces
REAL_SURFACE_IDS = [surface.surface_id for surface in REAL_SURFACES]
@@ -214,7 +217,7 @@ class TestTheReplayProperties:
@pytest.mark.parametrize("surface", REAL_SURFACES, ids=REAL_SURFACE_IDS)
def test_the_generator_does_not_merely_re_emit_the_reference_document(self, surface: Surface) -> None:
"""The other direction of vacuity. A sampler of one document satisfies every property."""
- found = find(_documents_for(surface=surface), lambda generated: bool(generated.mutations))
+ found = find(_documents_for(surface=surface), lambda generated: bool(generated.mutations), settings=FIND_SETTINGS)
assert found.mutations
@@ -240,7 +243,7 @@ class TestTheReplayProperties:
Asking it of the real surfaces would couple this test to whichever fields they happen to
have today, and the service surface has no enumerated field at all.
"""
- found = find(_documents_for(surface=SYNTHETIC_SURFACE), lambda generated: mutation in generated.mutations)
+ found = find(_documents_for(surface=SYNTHETIC_SURFACE), lambda generated: mutation in generated.mutations, settings=FIND_SETTINGS)
assert mutation in found.mutations
@@ -284,7 +287,11 @@ class TestTheReplayProperties:
the injector ever stopped injecting, both properties would stay green over a sampler of
documents the ledger never touches.
"""
- found = find(_documents_at_an_older_shape(), lambda text: replay_ledger_over_text(ledger=SYNTHETIC_LEDGER, text=text).did_change_document)
+ found = find(
+ _documents_at_an_older_shape(),
+ lambda text: replay_ledger_over_text(ledger=SYNTHETIC_LEDGER, text=text).did_change_document,
+ settings=FIND_SETTINGS,
+ )
assert replay_ledger_over_text(ledger=SYNTHETIC_LEDGER, text=found).did_change_document
@@ -301,7 +308,7 @@ class TestTheReplayProperties:
applied = [op for step in replay.steps for op in step.applied_ops]
return not replay.blocked and any(isinstance(op, RenameTableKeyOp) for op in applied) and any(isinstance(op, MoveKeyOp) for op in applied)
- found = find(_documents_at_an_older_shape(), rename_and_move_applied)
+ found = find(_documents_at_an_older_shape(), rename_and_move_applied, settings=FIND_SETTINGS)
assert rename_and_move_applied(found)
@@ -346,5 +353,6 @@ class TestTheReplayProperties:
found = find(
_documents_for(surface=SYNTHETIC_SURFACE),
lambda generated: replay_ledger_over_text(ledger=illegal, text=generated.text).text != generated.text,
+ settings=FIND_SETTINGS,
)
assert DocumentMutation.SWAPPED_ENUM_MEMBER in found.mutations