Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Package-vendoring · L-260904-df514c (#1355)

Louis Choquel5b3b76ec506b
diff --git a/.drift/acks/cli-docs.toml b/.drift/acks/cli-docs.toml
index 27ac4e7cb..fb05b170f 100644
--- a/.drift/acks/cli-docs.toml
+++ b/.drift/acks/cli-docs.toml
@@ -1,8 +1,8 @@
contract = "cli-docs"
-digest = "sha256:99306940c616bb57196113a07e173e04a4bb173fe716eec33de645d99ad3c2b8"
-reviewed_by = "Claude (Opus 5.5)"
-reviewed_at = "2026-10-06T00:00:23Z"
-rationale = "clean-pass: the show backends change only escapes the endpoint and variable names before Rich renders them, so a bracketed name prints as written, which is what docs/tools/cli/show.md already says the command shows; the run core change rewrites a comment only. Re-read run.md's dry-run claims (no credentials needed) and show.md's no-resolution paragraph against the code: both hold. pipelex/cli/agent_cli/CLAUDE.md names no surface either change touched."
+digest = "sha256:b48cd9256da89788502648136e4b3a6a5a4be0b853d6790d4178ca8091fd21a9"
+reviewed_by = "Louis Choquel"
+reviewed_at = "2026-10-06T10:13:22Z"
+rationale = "clean-pass: installed_methods.py now validates the provenance sidecar from its bytes, so a sidecar that is not UTF-8 is read as no provenance like an unparsable one, instead of raising. Re-read docs/tools/cli/ (run-by-address.md prints the provenance of a fetch, which is unchanged) and pipelex/cli/agent_cli/CLAUDE.md; neither describes how discovery treats an unreadable sidecar, so nothing is stale."
[trigger_files]
"pipelex/cli/__init__.py" = "blob:8b137891791fe96927ad78e64b0aad7bded08bdc"
@@ -119,7 +119,7 @@ rationale = "clean-pass: the show backends change only escapes the endpoint and
"pipelex/cli/deck_notice.py" = "blob:542877125e89e35b02bc969cfee952155b0d7c2c"
"pipelex/cli/error_handlers.py" = "blob:dd01a16295c17cc06378d3290ad5e60c27de986f"
"pipelex/cli/exceptions.py" = "blob:1d51c3a53ddd3f6cf8de566fa2ade3fa75ab7a3a"
-"pipelex/cli/installed_methods.py" = "blob:09197f47baf8d7251d28d148a0544b4a23ee7365"
+"pipelex/cli/installed_methods.py" = "blob:f7818b7857a8350389cf196e32630e7b3c51818c"
"pipelex/cli/method_resolver.py" = "blob:afaf77b2d8b6a3467d6ab1f355b1ee78a71eafa7"
"pipelex/cli/readiness.py" = "blob:6d185324ed0784663e1dc78d1194a28312c54798"
"pipelex/cli/version_report.py" = "blob:f4a2311bb629bd77196ea2f6b031c166db0d362a"
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a43d751c1..afcf2e2f2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,10 +2,21 @@
## [Unreleased]
+### Added
+
+- **A run request carries the method packages its bundle calls (Breaking)**: a `files` or `bundle_b64` request to `/v1/execute` or `/v1/start` may ship each package its bundle calls by address under `.mthds/methods/<name>/` at its root, with its `METHODS.toml`, and the server searches them before its installed copies and fetch-on-miss, for that run alone, so a shipped package wins over an installed copy and runs with fetching off; a package fetched by `method_ref` that ships its own `.mthds/methods/` is handled the same way. Those files no longer join the bundle's own `.mthds` content or its library directory, any other `.mthds` path in a bundle is refused with a `422 InvalidBundle` naming the entry, as are two shipped packages declaring the same address and name without regard to case, and on a sandbox-hosted deployment a shipped package declaring a Python structure class is refused with `403 MethodStructuresRefusedError`. The engine takes these directories as the new keyword-only `methods_dirs` of `PipelexMTHDSProtocol`, `pipeline_run_setup`, `acquire_library` and `LibraryManagerAbstract.load_from_blueprints`, which every implementation of the abstract must accept.
+
### Changed
- **The documentation no longer describes the Pipelex Gateway**: **Configure AI Providers** offers your own provider keys or a local model, says that the shipped defaults run as they are on OpenAI or Azure OpenAI and must be repointed for any other provider, OpenRouter included, and points to the Quick Start for running on the hosted API; the MTHDS language tutorial no longer runs `pipelex login`. The telemetry, logging, feature and under-the-hood pages describe only the telemetry streams you configure yourself and the backends the runtime ships, without the Gateway stream, its backend or its plugin.
-- **A call into a method package that cannot be resolved is a verdict item (Breaking)**: when a bundle names a pipe of another package by its address and the package is not installed and cannot be fetched (fetch-on-miss off, an address this runtime cannot fetch, a failed fetch), the load is refused with one item of the new `PipeValidationErrorType` value `unresolved_package_dependency` per package, on the first pipe naming it, with the reference as written in `missing_pipe_code`, where it raised `MethodFetchDisabledError` or `MethodDependencyFetchError` and the API answered a no-verdict `500`. `POST /v1/validate` now answers `200` with `is_valid: false` and the run routes a `422`, every package of the bundle is tried before the refusal, a failed clone's message no longer names the runner's temporary clone directory, and the remedies name no directory on the host. The MTHDS Test Corpus covers it as `invalid_unresolved_package_dependency`.
+- **A call into a method package that cannot be resolved is a verdict item (Breaking)**: when a bundle names a pipe of another package by its address and the package is not installed and cannot be fetched (fetch-on-miss off, an address this runtime cannot fetch, a failed fetch), the load is refused with one item of the new `PipeValidationErrorType` value `unresolved_package_dependency` per package, on the first pipe naming it, with the reference as written in `missing_pipe_code`, where it raised `MethodFetchDisabledError` or `MethodDependencyFetchError` and the API answered a no-verdict `500`. `POST /v1/validate` now answers `200` with `is_valid: false` and the run routes a `422`, every package of the bundle is tried before the refusal, a failed clone's message no longer names the runner's temporary clone directory, and the remedies name no directory on the host and include shipping the package with the bundle under `.mthds/methods/<name>/`. The MTHDS Test Corpus covers it as `invalid_unresolved_package_dependency`.
+- **A package bundle that does not parse refuses the load (Breaking)**: when a bundle of a package the consumer calls by address does not parse, the load is refused with the parser's items, each naming the file as `<address>/<path inside the package>` and never by its path on the host, whether the package was shipped with the bundle, installed or fetched. It used to be skipped with a warning, so the caller learned of it from a later refusal about a pipe.
+
+### Fixed
+
+- **A provenance sidecar that is not UTF-8 no longer breaks method discovery**: an installed or shipped method package whose `.provenance.json` is not valid UTF-8 is discovered with no recorded provenance, as one that does not parse already was, where discovery raised `UnicodeDecodeError` and a run calling the package answered a `500`.
+- **A `files` entry holding a lone surrogate is refused as an invalid bundle**: a name or a text in a run request's `files` map holding an escaped lone surrogate such as `"\ud800"`, which JSON carries but UTF-8 cannot encode, is refused with a `422 InvalidBundle` naming the entry, where the server answered a `500`.
+- **A bundle naming one path twice is refused**: a run request whose bundle holds two entries for one file, spelled differently in a `files` map (`a/./b` beside `a/b`) or twice in a zip, or a file that is also the directory of another entry, is refused with a `422 InvalidBundle` naming the path. The shipped packages' manifest check read the first copy while the engine read the last, so two packages of one address could get past it, and the second shape answered a `500`.
## [v0.75.0] - 2026-10-06
diff --git a/api/pipelex_api/bundle.py b/api/pipelex_api/bundle.py
index 1333ea153..402debfd5 100644
--- a/api/pipelex_api/bundle.py
+++ b/api/pipelex_api/bundle.py
@@ -13,7 +13,16 @@ form, enforces the ingest guards (both transport forms are refused together; a
hard file-count and total-size ceiling; per-entry path-safety against absolute
paths and `..` traversal; a zip-bomb guard that bounds actual decompression),
writes the surviving files into a fresh temp directory, and hands that directory
-back so the runner can load it via `library_dirs`. In a sandbox-hosted
+back so the runner can load it via `library_dirs`.
+
+A bundle may also ship the method packages it calls by address, each under
+`.mthds/methods/<name>/` at its root with its `METHODS.toml` there, the layout the
+standard names the local method cache. `partition_bundle_entries` sorts the entries
+into the bundle's own `.mthds` files, its other files, and those packages, refusing
+every other `.mthds` path with a `422 InvalidBundle` naming the entry, and two
+packages declaring the same address with one naming both directories, and
+`materialized_methods_dir` writes the packages into a temp directory of their own,
+`<tmp>/<name>/…`, which the runner hands the engine as `methods_dirs`. In a sandbox-hosted
deployment the load path reads every `.py` as source text, never importing it:
it refuses a bundle whose Python declares a structure class
(`MethodStructuresRefusedError`, a 403) and captures the rest onto the crate for
@@ -34,8 +43,11 @@ import zipfile
from contextlib import contextmanager
from io import BytesIO
from pathlib import Path, PurePosixPath
-from typing import TYPE_CHECKING, NamedTuple
+from typing import TYPE_CHECKING, NamedTuple, NoReturn
+from mthds.package.discovery import MANIFEST_FILENAME
+from mthds.package.exceptions import ManifestError
+from mthds.package.manifest.parser import parse_methods_toml
from pipelex import log
from pydantic import ConfigDict
from pydantic.dataclasses import dataclass
@@ -152,22 +164,56 @@ def _entries_from_zip(bundle_b64: str) -> list[tuple[PurePosixPath, bytes]]:
def _entries_from_files(files: dict[str, str]) -> list[tuple[PurePosixPath, bytes]]:
- """Validate a {relpath: text} map and return its (safe relpath, bytes) entries."""
+ r"""Validate a {relpath: text} map and return its (safe relpath, bytes) entries.
+
+ JSON can carry a lone surrogate as an escape (`"\ud800"`), which has no UTF-8 form, so an entry whose name or
+ content holds one is refused with a `422` naming it rather than failing later, when it is encoded or written.
+ """
_guard_count(len(files))
entries: list[tuple[PurePosixPath, bytes]] = []
total_bytes = 0
for name, content in files.items():
+ try:
+ name.encode("utf-8")
+ except UnicodeEncodeError:
+ msg = f"Bundle entry {name!r} has a name holding a lone surrogate, which is not valid Unicode text"
+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)
relpath = _safe_relpath(name)
if not relpath.parts:
msg = f"Bundle entry {name!r} has no filename"
raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)
- data = content.encode("utf-8")
+ try:
+ data = content.encode("utf-8")
+ except UnicodeEncodeError:
+ msg = f"Bundle entry {name!r} holds a lone surrogate, which is not valid Unicode text"
+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)
total_bytes += len(data)
_guard_running_total(total_bytes)
entries.append((relpath, data))
return entries
+def _refuse_colliding_paths(entries: list[tuple[PurePosixPath, bytes]]) -> None:
+ """Refuse two entries of one path, and an entry that is also the directory of another.
+
+ Entry names are normalized (`a/./b` and `a//b` are `a/b`) and a zip may hold one member name twice, so two entries
+ can name one file: a check reading the first copy, as the shipped packages' manifest check does, would pass content
+ that differs from the last copy, which is the one written to disk and read by the engine. A file that another entry
+ is placed under could not be written at all.
+ """
+ paths: set[PurePosixPath] = set()
+ for relpath, _ in entries:
+ if relpath in paths:
+ msg = f"Bundle has more than one entry for the path '{relpath.as_posix()}'; each file may appear only once"
+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)
+ paths.add(relpath)
+ for relpath, _ in entries:
+ for parent in relpath.parents:
+ if parent in paths:
+ msg = f"Bundle entry '{parent.as_posix()}' is a file, and also the directory of the entry '{relpath.as_posix()}'"
+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)
+
+
class ParsedBundle(NamedTuple):
"""A decoded-and-validated bundle held in memory, NOT yet written to disk.
@@ -189,8 +235,9 @@ def parse_bundle(*, bundle_b64: str | None, files: dict[str, str] | None) -> Par
Exactly one of `bundle_b64` / `files` must be supplied; supplying both is a
caller mistake (they are the same content in two forms) and is refused. All
- ingest guards (base64, size, count, path-safety, zip-bomb) run here, so the
- caller can inspect `has_python_sources` and reject BEFORE materializing to disk.
+ ingest guards (base64, size, count, path-safety, zip-bomb, colliding paths) run
+ here, so the caller can inspect `has_python_sources` and reject BEFORE
+ materializing to disk.
"""
if bundle_b64 is not None and files is not None:
msg = "Provide either bundle_b64 or files, not both"
@@ -202,17 +249,148 @@ def parse_bundle(*, bundle_b64: str | None, files: dict[str, str] | None) -> Par
else:
msg = "No bundle supplied (bundle_b64 and files are both absent)"
raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)
+ _refuse_colliding_paths(entries)
return ParsedBundle(entries=tuple(entries))
+# Where a bundle ships the method packages it calls: `.mthds/methods/<name>/`, at the bundle's root and nowhere else.
+_MTHDS_DIR_NAME = ".mthds"
+_METHODS_DIR_NAME = "methods"
+_VENDORING_LAYOUT = (
+ "A method package shipped with the bundle goes under '.mthds/methods/<name>/' at the bundle's root, "
+ "with its manifest at '.mthds/methods/<name>/METHODS.toml'."
+)
+
+
+class VendoredMethodPackage(NamedTuple):
+ """A method package a bundle ships under `.mthds/methods/<name>/`, held in memory."""
+
+ name: str
+ """The package's directory name under `.mthds/methods/`."""
+
+ full_address: str
+ """The address its manifest declares, `address/name`, as a reference names it and a refusal names the package."""
+
+ entries: tuple[tuple[PurePosixPath, bytes], ...]
+ """Every file of the package, by its path relative to the package directory."""
+
+
+class BundlePartition(NamedTuple):
+ """A bundle's entries sorted into its own `.mthds` files, its other files, and the packages it ships."""
+
+ mthds_entries: tuple[tuple[PurePosixPath, bytes], ...]
+ library_entries: tuple[tuple[PurePosixPath, bytes], ...]
+ vendored_packages: tuple[VendoredMethodPackage, ...]
+
+
+def _refuse_entry(*, owner: str, relpath: PurePosixPath, reason: str) -> NoReturn:
+ msg = f"{owner} entry '{relpath.as_posix()}' {reason}. {_VENDORING_LAYOUT}"
+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)
+
+
+def _vendored_package(*, owner: str, name: str, entries: list[tuple[PurePosixPath, bytes]]) -> VendoredMethodPackage:
+ """Check one shipped package's directory and read the address its manifest declares."""
+ package_label = f"'.mthds/methods/{name}/'"
+ if name.startswith("."):
+ msg = f"{owner} ships {package_label}, a hidden directory, which is never searched for a package. {_VENDORING_LAYOUT}"
+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)
+ manifest_bytes = next((data for relpath, data in entries if relpath == PurePosixPath(MANIFEST_FILENAME)), None)
+ if manifest_bytes is None:
+ msg = f"{owner} ships {package_label} without a {MANIFEST_FILENAME}, so no reference could find it. {_VENDORING_LAYOUT}"
+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)
+ manifest_path = f".mthds/methods/{name}/{MANIFEST_FILENAME}"
+ try:
+ manifest = parse_methods_toml(manifest_bytes.decode("utf-8"))
+ except UnicodeDecodeError:
+ msg = f"{owner} entry '{manifest_path}' is not valid UTF-8. {_VENDORING_LAYOUT}"
+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)
+ except ManifestError as exc:
+ msg = f"{owner} entry '{manifest_path}' is not a valid method package manifest: {exc.message} {_VENDORING_LAYOUT}"
+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)
+ # The name a reference matches, as the engine's discovery gives it: the manifest's, else the directory's.
+ package_name = manifest.name if manifest.name is not None else name
+ return VendoredMethodPackage(name=name, full_address=f"{manifest.address}/{package_name}", entries=tuple(entries))
+
+
+def _refuse_shared_identities(*, owner: str, packages: tuple[VendoredMethodPackage, ...]) -> None:
+ """Refuse shipped packages a reference could not tell apart, rather than let the directory order pick one.
+
+ A reference is matched against a package's full address without regard to case, as the engine's lookup compares it,
+ so two packages whose full addresses differ only in case are one identity too.
+ """
+ by_identity: dict[str, list[VendoredMethodPackage]] = {}
+ for package in packages:
+ by_identity.setdefault(package.full_address.casefold(), []).append(package)
+ for sharing in by_identity.values():
+ if len(sharing) < 2:
+ continue
+ directories = ", ".join(f"'.mthds/methods/{package.name}/'" for package in sharing)
+ msg = (
+ f"{owner} ships several method packages declaring the address '{sharing[0].full_address}': {directories}. "
+ "A reference matches a package by its address without regard to case, so it could not tell them apart: "
+ "ship one package per address."
+ )
+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)
+
+
+def partition_bundle_entries(entries: tuple[tuple[PurePosixPath, bytes], ...], *, owner: str = "Bundle") -> BundlePartition:
+ """Sort a bundle's entries into its own `.mthds` files, its other files, and the method packages it ships.
+
+ An entry under `.mthds/methods/<name>/`, whatever its extension, belongs to the shipped package `<name>`, which
+ leaves the bundle's own content and its library directory. Every other `.mthds` path component is refused with a
+ `422 InvalidBundle` naming the entry rather than dropped, since nothing could ever find what it holds: a file
+ elsewhere under `.mthds/`, a `.mthds/methods/` store below the root, a file directly under `.mthds/methods/`, and a
+ `.mthds/` inside a shipped package, whose own dependencies are not loaded. A shipped package must be a directory
+ whose name does not start with a dot and whose `METHODS.toml` parses, and the address that manifest declares is read
+ here, to name the package in a refusal. Two shipped packages declaring the same full address, compared without regard
+ to case as a reference is matched, are refused too, naming the address and both directories, since nothing but their
+ directory order would decide which one answers. Counting the bundle's own `.mthds` files is the caller's: a bundle
+ whose only `.mthds` files are shipped ones has no content of its own.
+
+ Args:
+ entries: The bundle's `(safe relpath, bytes)` entries.
+ owner: How a refusal names the bundle, for a method package fetched by address as for a request's bundle.
+
+ Returns:
+ The three groups, each in the bundle's entry order.
+ """
+ mthds_entries: list[tuple[PurePosixPath, bytes]] = []
+ library_entries: list[tuple[PurePosixPath, bytes]] = []
+ vendored_entries: dict[str, list[tuple[PurePosixPath, bytes]]] = {}
+ for relpath, data in entries:
+ parts = relpath.parts
+ if parts[0] == _MTHDS_DIR_NAME:
+ if len(parts) < 4 or parts[1] != _METHODS_DIR_NAME:
+ _refuse_entry(owner=owner, relpath=relpath, reason="is under '.mthds/' but not inside a shipped method package")
+ package_relpath = PurePosixPath(*parts[3:])
+ if _MTHDS_DIR_NAME in package_relpath.parts:
+ _refuse_entry(
+ owner=owner,
+ relpath=relpath,
+ reason="is a '.mthds/' directory inside a shipped method package, whose own dependencies are not loaded",
+ )
+ vendored_entries.setdefault(parts[2], []).append((package_relpath, data))
+ elif _MTHDS_DIR_NAME in parts:
+ _refuse_entry(owner=owner, relpath=relpath, reason="is under a '.mthds/' directory below the bundle's root, which is never searched")
+ elif relpath.suffix == ".mthds":
+ mthds_entries.append((relpath, data))
+ else:
+ library_entries.append((relpath, data))
+ vendored_packages = tuple(
+ _vendored_package(owner=owner, name=name, entries=package_entries) for name, package_entries in vendored_entries.items()
+ )
+ _refuse_shared_identities(owner=owner, packages=vendored_packages)
+ return BundlePartition(mthds_entries=tuple(mthds_entries), library_entries=tuple(library_entries), vendored_packages=vendored_packages)
+
+
@contextmanager
-def materialize_parsed(parsed: ParsedBundle) -> Generator[MaterializedBundle, None, None]:
+def materialize_parsed(parsed: ParsedBundle, *, prefix: str = "pipelex-bundle-") -> Generator[MaterializedBundle, None, None]:
"""Write an already-parsed bundle into a fresh temp directory, cleaned up on exit.
The yielded `MaterializedBundle.directory` is safe to pass as a `library_dirs`
entry; it is removed when the context exits, on both the happy and error path.
"""
- directory = Path(tempfile.mkdtemp(prefix="pipelex-bundle-"))
+ directory = Path(tempfile.mkdtemp(prefix=prefix))
try:
root = directory.resolve()
relpaths: list[str] = []
@@ -231,6 +409,18 @@ def materialize_parsed(parsed: ParsedBundle) -> Generator[MaterializedBundle, No
shutil.rmtree(directory, ignore_errors=True)
+@contextmanager
+def materialized_methods_dir(packages: tuple[VendoredMethodPackage, ...]) -> Generator[Path, None, None]:
+ """Write shipped method packages into a temp directory of their own, as `<tmp>/<name>/…`, cleaned up on exit.
+
+ The directory is laid out like `.mthds/methods/`, one directory per package, which is what the engine's
+ `methods_dirs` take, and its path holds no `.mthds/methods` a walk-up from a bundle could take for a store.
+ """
+ entries = tuple((PurePosixPath(package.name) / relpath, data) for package in packages for relpath, data in package.entries)
+ with materialize_parsed(ParsedBundle(entries=entries), prefix="pipelex-methods-") as materialized:
+ yield materialized.directory
+
+
@contextmanager
def materialized_bundle(*, bundle_b64: str | None, files: dict[str, str] | None) -> Generator[MaterializedBundle, None, None]:
"""Parse AND materialize a bundle in one step (convenience for callers that don't gate).
diff --git a/api/pipelex_api/method_source.py b/api/pipelex_api/method_source.py
index 4e326ce85..7dd299574 100644
--- a/api/pipelex_api/method_source.py
+++ b/api/pipelex_api/method_source.py
@@ -12,11 +12,15 @@ manifest-identity package location, the bounds, and the structures check all liv
files travel as `mthds_contents` (paired with their real relative paths as
`mthds_sources`, so diagnostics carry true per-file labels), and only the non-`.mthds`
files are materialized into a temporary `library_dirs` entry — exactly the split the
- method-bundle transport uses, so a fetched package runs the same proven path.
+ method-bundle transport uses, so a fetched package runs the same proven path. The
+ packages it ships under its own `.mthds/methods/<name>/` are partitioned out of both, as
+ a bundle's are, into a temporary `methods_dirs` entry: the run routes hand it to the
+ engine, and `/validate` does not, resolving such a dependency through the installed store
+ or fetch-on-miss.
- The tooling routes (`/resolve`, `/codegen`, `/pipe-io`) use
- :func:`fetch_method_mthds_files`: only the `.mthds` files, as `files[]` items, paired with
- the manifest's `main_pipe` so the per-pipe route defaults its selector exactly as a run
- does. No Python ever loads there, so the execution-locus gate does not apply.
+ :func:`fetch_method_mthds_files`: only the package's own `.mthds` files, as `files[]` items,
+ paired with the manifest's `main_pipe` so the per-pipe route defaults its selector exactly
+ as a run does. No Python ever loads there, so the execution-locus gate does not apply.
The security gate (packaging invariant 7 — execution locus decides): `.mthds` content is
data, always acceptable. On a deployment that is NOT sandbox-hosted, a fetched package
@@ -34,9 +38,9 @@ can never race a running pipeline.
"""
from collections.abc import Generator
-from contextlib import contextmanager
+from contextlib import ExitStack, contextmanager
from pathlib import Path, PurePosixPath
-from typing import NamedTuple
+from typing import NamedTuple, NoReturn
from mthds.package.discovery import MANIFEST_FILENAME
from pipelex import log
@@ -46,7 +50,7 @@ from pipelex.methods.method_ref import parse_method_ref
from pipelex.methods.structures_check import ensure_no_structured_content_python
from pydantic import BaseModel, ConfigDict, Field
-from pipelex_api.bundle import ParsedBundle, materialize_parsed
+from pipelex_api.bundle import BundlePartition, ParsedBundle, materialize_parsed, materialized_methods_dir, partition_bundle_entries
from pipelex_api.error_types import ErrorType
from pipelex_api.errors import raise_forbidden, raise_validation_error
from pipelex_api.limits import MAX_MTHDS_FILE_BYTES
@@ -64,6 +68,9 @@ class FetchedMethodSource(BaseModel):
mthds_contents: list[str]
mthds_sources: list[str]
library_dirs: list[str] | None = None
+ methods_dirs: list[Path] | None = Field(
+ default=None, description="The methods directory holding the packages the package ships under its own `.mthds/methods/`."
+ )
main_pipe: str | None = Field(default=None, description="The manifest's declared entry pipe; a request `pipe_code` overrides it.")
provenance: MethodProvenance
@@ -106,17 +113,36 @@ def _package_files(package: FetchedMethodPackage) -> list[Path]:
return files
-def _read_mthds_text(file_path: Path, *, relative: str, package_address: str) -> str:
+def _decode_mthds_text(data: bytes, *, relative: str, package_address: str) -> str:
try:
- content = file_path.read_text(encoding="utf-8")
+ content = data.decode("utf-8")
except UnicodeDecodeError:
raise_validation_error(message=f"File '{relative}' in method package '{package_address}' is not valid UTF-8.")
- if len(content.encode("utf-8")) > MAX_MTHDS_FILE_BYTES:
+ if len(data) > MAX_MTHDS_FILE_BYTES:
msg = f"File '{relative}' in method package '{package_address}' exceeds the {MAX_MTHDS_FILE_BYTES // 1024} KiB per-file limit."
raise_validation_error(message=msg)
return content
+def _partition_package(package: FetchedMethodPackage, *, files: list[Path]) -> BundlePartition:
+ """Sort a fetched package's files as a bundle's are: its own `.mthds`, its other files, and the packages it ships.
+
+ The refusals are a bundle's too: a `.mthds` path no reference could find is a `422` naming the file.
+ """
+ entries = tuple((PurePosixPath(file_path.relative_to(package.package_dir).as_posix()), file_path.read_bytes()) for file_path in files)
+ return partition_bundle_entries(entries, owner=f"Method package '{package.full_address}'")
+
+
+def _refuse_no_own_mthds(package: FetchedMethodPackage, *, partition: BundlePartition) -> NoReturn:
+ msg = f"Method package '{package.full_address}' contains no .mthds file."
+ if partition.vendored_packages:
+ msg = (
+ f"Method package '{package.full_address}' contains no .mthds file of its own: "
+ "the files under '.mthds/methods/' are the packages it ships."
+ )
+ raise_validation_error(message=msg)
+
+
def _apply_execution_locus_gate(package: FetchedMethodPackage, *, python_relpaths: list[str]) -> None:
"""Refuse Python that would execute where it must not (see the module docstring)."""
if not python_relpaths:
@@ -131,10 +157,11 @@ def _apply_execution_locus_gate(package: FetchedMethodPackage, *, python_relpath
def fetched_method_source(method_ref: str) -> Generator[FetchedMethodSource, None, None]:
"""Fetch a `method_ref`'s package and shape it for the run/validate path.
- Yields the package's `.mthds` files as `(mthds_contents, mthds_sources)` pairs and its
+ Yields the package's `.mthds` files as `(mthds_contents, mthds_sources)` pairs, its
non-`.mthds` files (PipeFunc `.py`, `requirements.txt`, …) materialized into a temporary
- `library_dirs` entry, cleaned up on exit. The execution-locus gate runs before anything
- touches disk.
+ `library_dirs` entry, and the packages it ships under its own `.mthds/methods/<name>/`
+ materialized into a temporary `methods_dirs` entry, each cleaned up on exit. The
+ execution-locus gate runs before anything touches disk.
Args:
method_ref: The raw `method_ref` string from the request.
@@ -148,37 +175,35 @@ def fetched_method_source(method_ref: str) -> Generator[FetchedMethodSource, Non
package = _fetch_package(method_ref)
files = _package_files(package)
python_relpaths = [file_path.relative_to(package.package_dir).as_posix() for file_path in files if file_path.suffix == ".py"]
+ # The whole package, the packages it ships included, so a structure class anywhere in it is refused before anything is copied.
_apply_execution_locus_gate(package, python_relpaths=python_relpaths)
+ partition = _partition_package(package, files=files)
mthds_contents: list[str] = []
mthds_sources: list[str] = []
- other_entries: list[tuple[PurePosixPath, bytes]] = []
- for file_path in files:
- relative = file_path.relative_to(package.package_dir).as_posix()
- if file_path.suffix == ".mthds":
- mthds_contents.append(_read_mthds_text(file_path, relative=relative, package_address=package.full_address))
- mthds_sources.append(relative)
- elif file_path.name != MANIFEST_FILENAME:
- # The manifest is already consumed (identity + `main_pipe`); materializing it into
- # the library dir would hand the local loader a package boundary it must not see.
- other_entries.append((PurePosixPath(relative), file_path.read_bytes()))
+ for relpath, data in partition.mthds_entries:
+ relative = relpath.as_posix()
+ mthds_contents.append(_decode_mthds_text(data, relative=relative, package_address=package.full_address))
+ mthds_sources.append(relative)
if not mthds_contents:
- raise_validation_error(message=f"Method package '{package.full_address}' contains no .mthds file.")
-
- if not other_entries:
+ _refuse_no_own_mthds(package, partition=partition)
+ # The manifest is already consumed (identity + `main_pipe`); materializing it into the library dir would hand the
+ # local loader a package boundary it must not see.
+ other_entries = tuple((relpath, data) for relpath, data in partition.library_entries if relpath.name != MANIFEST_FILENAME)
+
+ with ExitStack() as stack:
+ library_dirs: list[str] | None = None
+ if other_entries:
+ library_bundle = stack.enter_context(materialize_parsed(ParsedBundle(entries=other_entries)))
+ library_dirs = [str(library_bundle.directory)]
+ methods_dirs: list[Path] | None = None
+ if partition.vendored_packages:
+ methods_dirs = [stack.enter_context(materialized_methods_dir(partition.vendored_packages))]
yield FetchedMethodSource(
mthds_contents=mthds_contents,
mthds_sources=mthds_sources,
- library_dirs=None,
- main_pipe=package.manifest.main_pipe,
- provenance=package.provenance,
- )
- return
- with materialize_parsed(ParsedBundle(entries=tuple(other_entries))) as bundle:
- yield FetchedMethodSource(
- mthds_contents=mthds_contents,
- mthds_sources=mthds_sources,
- library_dirs=[str(bundle.directory)],
+ library_dirs=library_dirs,
+ methods_dirs=methods_dirs,
main_pipe=package.manifest.main_pipe,
provenance=package.provenance,
)
@@ -188,9 +213,11 @@ def fetch_method_mthds_files(method_ref: str) -> FetchedMthdsFiles:
"""Fetch a `method_ref`'s package and return its `.mthds` files as `files[]` items, plus its `main_pipe`.
The tooling-route shape: each item pairs the file's content with its real relative path
- as `source`, so crate provenance and diagnostics carry true per-file labels. Only
- `.mthds` data travels — the package's Python (if any) never loads on these routes, so
- the execution-locus gate does not apply here. The manifest's `main_pipe` rides beside the
+ as `source`, so crate provenance and diagnostics carry true per-file labels. Only the
+ package's own `.mthds` data travels, never that of a package it ships under its
+ `.mthds/methods/`, since these routes resolve no address-based dependency. The package's
+ Python (if any) never loads on these routes, so the execution-locus gate does not apply
+ here. The manifest's `main_pipe` rides beside the
files so a per-pipe projection can default its selector the way a run does — the manifest
is the package author's declaration of the entry pipe, and dropping it here would make
`main_pipe` buy them nothing on the tooling routes.
@@ -203,13 +230,12 @@ def fetch_method_mthds_files(method_ref: str) -> FetchedMthdsFiles:
ApiError: 422 for a package with no `.mthds` file or one this server cannot accept.
"""
package = _fetch_package(method_ref)
+ partition = _partition_package(package, files=_package_files(package))
items: list[MthdsFileItem] = []
- for file_path in _package_files(package):
- if file_path.suffix != ".mthds":
- continue
- relative = file_path.relative_to(package.package_dir).as_posix()
- content = _read_mthds_text(file_path, relative=relative, package_address=package.full_address)
+ for relpath, data in partition.mthds_entries:
+ relative = relpath.as_posix()
+ content = _decode_mthds_text(data, relative=relative, package_address=package.full_address)
items.append(MthdsFileItem(content=content, source=relative))
if not items:
- raise_validation_error(message=f"Method package '{package.full_address}' contains no .mthds file.")
+ _refuse_no_own_mthds(package, partition=partition)
return FetchedMthdsFiles(files=items, main_pipe=package.manifest.main_pipe)
diff --git a/api/pipelex_api/routes/pipelex/pipeline.py b/api/pipelex_api/routes/pipelex/pipeline.py
index 6516412e3..431d48030 100644
--- a/api/pipelex_api/routes/pipelex/pipeline.py
+++ b/api/pipelex_api/routes/pipelex/pipeline.py
@@ -2,8 +2,8 @@ from __future__ import annotations
import hashlib
import hmac
-from contextlib import contextmanager
-from pathlib import Path, PurePosixPath
+from contextlib import ExitStack, contextmanager
+from pathlib import Path
from typing import TYPE_CHECKING, Annotated, Any, NamedTuple, cast
from fastapi import APIRouter, Depends, Request
@@ -11,6 +11,7 @@ from fastapi.responses import JSONResponse
from mthds.protocol.exceptions import PipelineRequestError
from pipelex.config import get_config, is_pipe_func_sandbox_hosted
from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.methods.structures_check import ensure_no_structured_content_python
from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, StorageTarget, WebhookTarget
from pipelex.pipe_run.pipe_run_protocol import PipeRunProtocol
from pipelex.pipeline.pipeline_response import PipelexRunResultExecute, PipelexRunResultStart, RunState
@@ -26,7 +27,14 @@ from pydantic import ValidationError
from typing_extensions import override
from pipelex_api.api_config import get_api_config, resolve_orchestration_mode
-from pipelex_api.bundle import ParsedBundle, materialize_parsed, parse_bundle
+from pipelex_api.bundle import (
+ ParsedBundle,
+ VendoredMethodPackage,
+ materialize_parsed,
+ materialized_methods_dir,
+ parse_bundle,
+ partition_bundle_entries,
+)
from pipelex_api.error_types import ErrorType
from pipelex_api.errors import raise_bad_request, raise_forbidden, raise_validation_error
from pipelex_api.json_body import decode_json_body
@@ -407,6 +415,8 @@ class ApiRunner(PipelexMTHDSProtocol):
# here, so a group left out of this call is dropped with no error
# and the run's spans lose their groups while the ack still says 202.
extras=self.extras,
+ # The packages the request ships, which the base `execute` threads too.
+ methods_dirs=self.methods_dirs,
pipeline_run_id=pipeline_run_id,
request_id=request_id,
)
@@ -685,31 +695,58 @@ async def _parse_request(request: Request) -> tuple[RunRequest, PipelineApiExtra
return run_request, extras
+class _BundleRunSource(NamedTuple):
+ """What a request's run source resolves to, before the routes add its entry pipe and provenance."""
+
+ mthds_contents: list[str] | None
+ library_dirs: list[str] | None
+ methods_dirs: list[Path] | None
+
+
+def _ensure_vendored_packages_hold_no_structures(*, methods_dir: Path, packages: tuple[VendoredMethodPackage, ...]) -> None:
+ """Refuse a shipped package whose Python declares a structure class, as a fetched package is on this deployment.
+
+ The library load refuses the bundle's own structure classes, but never reads a shipped package's Python, which is
+ not in a library directory. Each package is named by the address its manifest declares, and each file by its path
+ inside the package, never by the temporary directory.
+ """
+ for package in packages:
+ if any(relpath.suffix == ".py" for relpath, _ in package.entries):
+ ensure_no_structured_content_python(package_dir=methods_dir / package.name, package_address=package.full_address)
+
+
@contextmanager
-def _bundle_run_source(run_request: RunRequest) -> Generator[tuple[list[str] | None, list[str] | None], None, None]:
- """Resolve a run's `(mthds_contents, library_dirs)` from the request.
+def _bundle_run_source(run_request: RunRequest) -> Generator[_BundleRunSource, None, None]:
+ """Resolve a run's `(mthds_contents, library_dirs, methods_dirs)` from the request.
- A method bundle KEEPS the proven run path rather than replacing it: its `.mthds`
+ A method bundle KEEPS the proven run path rather than replacing it: its own `.mthds`
text travels as `mthds_contents` — so `main_pipe` resolves exactly as it does for a
- plain (non-bundle) run — and ONLY the non-`.mthds` files (custom PipeFunc `.py`,
+ plain (non-bundle) run — and its other files (custom PipeFunc `.py`,
`requirements.txt`) are materialized into a temp `library_dirs` entry, where the
load path reads them without importing them. This mirrors "a normal run, plus the Python", instead of
handing the engine a bare directory with no `mthds_contents` (which never resolves
`main_pipe`, since that is only derived from `mthds_contents`).
- No bundle → yields the request's own `mthds_contents` and no library dir (the
- classic path, unchanged). Bundle with no non-`.mthds` files → yields the `.mthds`
- texts and no library dir. The temp dir (when created) is cleaned up on exit.
-
- Security gate (decision 5): a bundle that ships custom Python (`.py`) is only
- honored on a sandbox-hosted deployment, where the load path reads the source
- without importing it and refuses, with a 403 `MethodStructuresRefusedError`, a
- bundle whose Python declares a structure class; PipeFunc source is captured for
- the sandbox. On a non-hosted deployment, running that code would import it
- in-process — refused with a 403 rather than executing untrusted code.
+ The method packages the bundle ships under `.mthds/methods/<name>/` belong to neither:
+ every file of theirs is written into a temp directory of its own, as `<tmp>/<name>/…`,
+ which the engine searches first for an address-based reference and never writes, so a
+ shipped package wins over an installed copy and ends with the request. The other `.mthds`
+ paths are refused with a `422` naming the entry (`partition_bundle_entries`), and so are two
+ shipped packages declaring the same address and a bundle whose only `.mthds` files are shipped ones.
+
+ No bundle → yields the request's own `mthds_contents` and no directory (the classic path,
+ unchanged). Each temp dir is created only when it has files, and cleaned up on exit.
+
+ Security gate (decision 5): a bundle that ships custom Python (`.py`), in a shipped package
+ or not, is only honored on a sandbox-hosted deployment, where the load path reads the bundle's
+ own source without importing it and refuses, with a 403 `MethodStructuresRefusedError`, a
+ bundle whose Python declares a structure class; PipeFunc source is captured for the sandbox.
+ A shipped package's Python is checked the same way, as a fetched package's is, and never runs.
+ On a non-hosted deployment, running that code would import it in-process — refused with a 403
+ rather than executing untrusted code.
"""
if run_request.bundle_b64 is None and run_request.files is None:
- yield run_request.mthds_contents, None
+ yield _BundleRunSource(mthds_contents=run_request.mthds_contents, library_dirs=None, methods_dirs=None)
return
# Parse + guard in memory FIRST, then apply the sandbox-hosted gate BEFORE any disk write —
# a bundle destined for a 403 on a non-hosted deployment never touches the filesystem.
@@ -717,25 +754,36 @@ def _bundle_run_source(run_request: RunRequest) -> Generator[tuple[list[str] | N
if parsed.has_python_sources and not is_pipe_func_sandbox_hosted():
msg = "This bundle ships custom Python (.py); running it requires a sandbox-hosted deployment."
raise_forbidden(message=msg, error_type=ErrorType.CUSTOM_CODE_REQUIRES_SANDBOX)
- # Split: `.mthds` text → `mthds_contents` (the proven main_pipe path); everything else
- # (`.py`, `requirements.txt`) → a temp `library_dirs` entry the load path source-captures.
+ # Split: the bundle's own `.mthds` text → `mthds_contents` (the proven main_pipe path); its other files
+ # (`.py`, `requirements.txt`) → a temp `library_dirs` entry the load path source-captures; the packages it
+ # ships → a temp methods directory.
+ partition = partition_bundle_entries(parsed.entries)
mthds_contents: list[str] = []
- other_entries: list[tuple[PurePosixPath, bytes]] = []
- for relpath, content in parsed.entries:
- if str(relpath).endswith(".mthds"):
- try:
- mthds_contents.append(content.decode("utf-8"))
- except UnicodeDecodeError:
- raise_validation_error(message=f"Bundle .mthds file '{relpath}' is not valid UTF-8.", error_type=ErrorType.INVALID_BUNDLE)
- else:
- other_entries.append((relpath, content))
+ for relpath, content in partition.mthds_entries:
+ try:
+ mthds_contents.append(content.decode("utf-8"))
+ except UnicodeDecodeError:
+ raise_validation_error(message=f"Bundle .mthds file '{relpath}' is not valid UTF-8.", error_type=ErrorType.INVALID_BUNDLE)
if not mthds_contents:
- raise_validation_error(message="Method bundle contains no .mthds file.", error_type=ErrorType.INVALID_BUNDLE)
- if not other_entries:
- yield mthds_contents, None
- return
- with materialize_parsed(ParsedBundle(entries=tuple(other_entries))) as bundle:
- yield mthds_contents, [str(bundle.directory)]
+ msg = "Method bundle contains no .mthds file."
+ if partition.vendored_packages:
+ msg = (
+ "Method bundle contains no .mthds file of its own: the files under '.mthds/methods/<name>/' are the method "
+ "packages it ships, and the bundle's own .mthds files go outside '.mthds/'."
+ )
+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)
+ with ExitStack() as stack:
+ library_dirs: list[str] | None = None
+ if partition.library_entries:
+ library_bundle = stack.enter_context(materialize_parsed(ParsedBundle(entries=partition.library_entries)))
+ library_dirs = [str(library_bundle.directory)]
+ methods_dirs: list[Path] | None = None
+ if partition.vendored_packages:
+ methods_dir = stack.enter_context(materialized_methods_dir(partition.vendored_packages))
+ if is_pipe_func_sandbox_hosted():
+ _ensure_vendored_packages_hold_no_structures(methods_dir=methods_dir, packages=partition.vendored_packages)
+ methods_dirs = [methods_dir]
+ yield _BundleRunSource(mthds_contents=mthds_contents, library_dirs=library_dirs, methods_dirs=methods_dirs)
class _ResolvedRunSource(NamedTuple):
@@ -743,6 +791,7 @@ class _ResolvedRunSource(NamedTuple):
mthds_contents: list[str] | None
library_dirs: list[str] | None
+ methods_dirs: list[Path] | None
pipe_code: str | None
method_provenance: MethodProvenance | None
@@ -753,7 +802,8 @@ def _run_source(run_request: RunRequest) -> Generator[_ResolvedRunSource, None,
A `method_ref` resolves through `pipelex_api.method_source.fetched_method_source` (fetch → locate →
execution-locus gate → materialize) into exactly the shape a bundle produces — `.mthds` text
- as `mthds_contents`, non-`.mthds` files in a temp `library_dirs` entry — so the engine runs
+ as `mthds_contents`, non-`.mthds` files in a temp `library_dirs` entry, the packages the package
+ ships under its own `.mthds/methods/` in a temp `methods_dirs` entry — so the engine runs
the same proven path. The entry pipe defaults to the fetched manifest's `main_pipe`; a
request `pipe_code` overrides it. Provenance `(address, tag, commit_sha)` rides back so the
routes can put it on the response. The other two forms delegate to `_bundle_run_source`,
@@ -764,14 +814,16 @@ def _run_source(run_request: RunRequest) -> Generator[_ResolvedRunSource, None,
yield _ResolvedRunSource(
mthds_contents=fetched.mthds_contents,
library_dirs=fetched.library_dirs,
+ methods_dirs=fetched.methods_dirs,
pipe_code=run_request.pipe_code or fetched.main_pipe,
method_provenance=fetched.provenance,
)
return
- with _bundle_run_source(run_request) as (mthds_contents, library_dirs):
+ with _bundle_run_source(run_request) as bundle_source:
yield _ResolvedRunSource(
- mthds_contents=mthds_contents,
- library_dirs=library_dirs,
+ mthds_contents=bundle_source.mthds_contents,
+ library_dirs=bundle_source.library_dirs,
+ methods_dirs=bundle_source.methods_dirs,
pipe_code=run_request.pipe_code,
method_provenance=None,
)
@@ -825,6 +877,7 @@ async def execute(request: Request) -> JSONResponse:
read_scope=scopes.read_scope,
extras=extras.analytics_groups,
library_dirs=source.library_dirs,
+ methods_dirs=source.methods_dirs,
)
response = await runner.execute(
pipe_code=source.pipe_code,
@@ -916,6 +969,7 @@ async def start(
read_scope=scopes.read_scope,
extras=extras.analytics_groups,
library_dirs=source.library_dirs,
+ methods_dirs=source.methods_dirs,
)
start_result = await runner.start(
pipe_code=source.pipe_code,
diff --git a/api/pipelex_api/schemas/models.py b/api/pipelex_api/schemas/models.py
index 8fbf31574..a45c68312 100644
--- a/api/pipelex_api/schemas/models.py
+++ b/api/pipelex_api/schemas/models.py
@@ -71,7 +71,10 @@ class RunRequest(BaseModel):
bundle_b64: PIPELEX-API EXTENSION — base64-encoded zip of the whole method
bundle (`.mthds` + `.py` + `structures/*.py` + `requirements.txt`). Lets a
caller ship custom PipeFunc Python alongside the method; materialized into a
- temporary library directory before the run. Mutually exclusive with `files`.
+ temporary library directory before the run. The method packages the bundle
+ calls by address may travel with it under `.mthds/methods/<name>/`, each with
+ its `METHODS.toml`, and are searched before the runner's installed copies.
+ Mutually exclusive with `files`.
files: PIPELEX-API EXTENSION — the bundle as a `{relative_path: text}` map (the
unzipped equivalent of `bundle_b64`). Mutually exclusive with `bundle_b64`.
method_ref: PIPELEX-API EXTENSION — run a published method by address instead of
@@ -95,6 +98,8 @@ class RunRequest(BaseModel):
"PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — base64-encoded zip of the whole "
"method bundle (`.mthds` + `.py` + `structures/*.py` + `requirements.txt`), materialized into a "
"temporary library directory before the run so custom PipeFunc Python travels with the method. "
+ "The method packages the bundle calls by address may travel with it under `.mthds/methods/<name>/`, "
+ "each with its `METHODS.toml`; they are searched before the runner's installed copies, for this run alone. "
"Mutually exclusive with `files`. Custom `.py` is only honored on a sandbox-hosted deployment."
),
)
diff --git a/api/tests/unit/_constants.py b/api/tests/unit/_constants.py
index 929656c62..5f1a98cac 100644
--- a/api/tests/unit/_constants.py
+++ b/api/tests/unit/_constants.py
@@ -307,6 +307,39 @@ output = "Text"
steps = [{{ pipe = "{UNRESOLVED_PACKAGE_REFERENCE}", result = "out" }}]
"""
+# The invented package `UNRESOLVED_PACKAGE_MTHDS` calls, as a request ships it under `.mthds/methods/probe/` beside its
+# bundle. Its entry composes a text that says whose copy it is, which a dry run renders too, so a run's output tells which
+# copy answered the call: a test installing a competing copy in the store replaces `VENDORED_PROBE_MARKER` with another one.
+VENDORED_PROBE_ADDRESS = "github.com/invented/probe-lib/probe"
+VENDORED_PROBE_MARKER = "VENDORED copy"
+VENDORED_PROBE_MANIFEST = """\
+[package]
+name = "probe"
+address = "github.com/invented/probe-lib"
+version = "0.1.0"
+description = "An invented package a request ships with its bundle"
+
+[exports.probe_dep]
+pipes = ["entry"]
+"""
+VENDORED_PROBE_MTHDS = f"""\
+domain = "probe_dep"
+
+[pipe.entry]
+type = "PipeCompose"
+description = "Say which copy of the package answered"
+inputs = {{ text = "Text" }}
+output = "Text"
+template = "{VENDORED_PROBE_MARKER}: $text"
+"""
+
+# The `files` map of a bundle calling the invented package and shipping it: the bundle's own file, then the package.
+VENDORED_PROBE_FILES: dict[str, str] = {
+ "bundle.mthds": UNRESOLVED_PACKAGE_MTHDS,
+ ".mthds/methods/probe/METHODS.toml": VENDORED_PROBE_MANIFEST,
+ ".mthds/methods/probe/probe.mthds": VENDORED_PROBE_MTHDS,
+}
+
# A bundle that loads but whose run fails one step down, in a live run and a dry run alike since no
# pipe calls a model: `review_topic` runs the parallel `analyze_topic`, whose branch `draft_idea` gives
# one `Idea` for the field `ideas`, which `TopicReview` declares as a list. The combine of the branch
diff --git a/api/tests/unit/test_pipeline_bundle.py b/api/tests/unit/test_pipeline_bundle.py
index 61dd87175..8dd3183b6 100644
--- a/api/tests/unit/test_pipeline_bundle.py
+++ b/api/tests/unit/test_pipeline_bundle.py
@@ -4,16 +4,20 @@ The pipeline runner is mocked (as in `test_pipeline_routes`); these assert that
the API layer KEEPS the proven run path for a bundle — the bundle's `.mthds` text
is passed as `mthds_contents` (so the engine resolves `main_pipe` exactly as for a
plain run) — while ONLY the non-`.mthds` files (custom PipeFunc `.py`, etc.) are
-materialized into a temporary `library_dirs` directory for source capture. They
-also assert the custom-Python sandbox gate and the both-forms guard.
+materialized into a temporary `library_dirs` directory for source capture, while the
+method packages it ships under `.mthds/methods/<name>/` leave both and are written
+into their own temporary `methods_dirs` directory. They also assert the custom-Python
+sandbox gate, the both-forms guard, and the shapes of `.mthds/` content refused.
"""
import base64
import io
+import json
import zipfile
from pathlib import Path
from typing import Any
+import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from pipelex.pipeline.pipeline_response import PipelexRunResultStart, RunState
@@ -21,15 +25,121 @@ from pytest_mock import MockerFixture
from pipelex_api.exception_handlers import register_exception_handlers
from pipelex_api.routes.pipelex.pipeline import router as pipeline_router
-from tests.unit._constants import VALID_MTHDS
+from tests.unit._constants import (
+ UNRESOLVED_PACKAGE_MTHDS,
+ VALID_MTHDS,
+ VENDORED_PROBE_ADDRESS,
+ VENDORED_PROBE_FILES,
+ VENDORED_PROBE_MANIFEST,
+ VENDORED_PROBE_MTHDS,
+)
_PIPE_FUNC_PY = "def echo(working_memory):\n return 'hi'\n"
+# Each `.mthds/` shape a request is refused for, with the text the refusal must name: the entry, or the package directory.
+_REFUSED_VENDORING_SHAPES: dict[str, tuple[dict[str, str], str]] = {
+ "a file under .mthds/ outside methods/": ({"bundle.mthds": UNRESOLVED_PACKAGE_MTHDS, ".mthds/foo": "x"}, ".mthds/foo"),
+ "a .mthds/methods/ store below the root": (
+ {"bundle.mthds": UNRESOLVED_PACKAGE_MTHDS, "sub/.mthds/methods/probe/probe.mthds": VENDORED_PROBE_MTHDS},
+ "sub/.mthds/methods/probe/probe.mthds",
+ ),
+ "a file directly under .mthds/methods/": (
+ {"bundle.mthds": UNRESOLVED_PACKAGE_MTHDS, ".mthds/methods/probe.mthds": VENDORED_PROBE_MTHDS},
+ ".mthds/methods/probe.mthds",
+ ),
+ "a package shipping its own .mthds/ store": (
+ {
+ **VENDORED_PROBE_FILES,
+ ".mthds/methods/probe/.mthds/methods/inner/inner.mthds": VENDORED_PROBE_MTHDS,
+ },
+ ".mthds/methods/probe/.mthds/methods/inner/inner.mthds",
+ ),
+ "a package directory without METHODS.toml": (
+ {"bundle.mthds": UNRESOLVED_PACKAGE_MTHDS, ".mthds/methods/probe/probe.mthds": VENDORED_PROBE_MTHDS},
+ ".mthds/methods/probe/",
+ ),
+ "a package whose METHODS.toml is not a manifest": (
+ {**VENDORED_PROBE_FILES, ".mthds/methods/probe/METHODS.toml": "[package]\nname = 'probe'\n"},
+ ".mthds/methods/probe/METHODS.toml",
+ ),
+ "a hidden package directory": (
+ {
+ "bundle.mthds": UNRESOLVED_PACKAGE_MTHDS,
+ ".mthds/methods/.probe/METHODS.toml": VENDORED_PROBE_MANIFEST,
+ ".mthds/methods/.probe/probe.mthds": VENDORED_PROBE_MTHDS,
+ },
+ ".mthds/methods/.probe/",
+ ),
+ "a request whose only .mthds files are vendored": (
+ {".mthds/methods/probe/METHODS.toml": VENDORED_PROBE_MANIFEST, ".mthds/methods/probe/probe.mthds": VENDORED_PROBE_MTHDS},
+ "no .mthds file of its own",
+ ),
+}
+
+
+def _probe_manifest(*, address: str, name: str | None) -> str:
+ """A manifest exporting the probe package's entry, declaring the address and, when given, the name."""
+ name_line = f'name = "{name}"\n' if name is not None else ""
+ package = f'[package]\n{name_line}address = "{address}"\nversion = "0.1.0"\ndescription = "A shipped package"\n'
+ return f'{package}\n[exports.probe_dep]\npipes = ["entry"]\n'
+
+
+def _shipped_packages(manifests: dict[str, str]) -> dict[str, str]:
+ """A bundle calling the probe package and shipping one package per `{directory name: manifest}`."""
+ files = {"bundle.mthds": UNRESOLVED_PACKAGE_MTHDS}
+ for directory, manifest in manifests.items():
+ files[f".mthds/methods/{directory}/METHODS.toml"] = manifest
+ files[f".mthds/methods/{directory}/probe.mthds"] = VENDORED_PROBE_MTHDS
+ return files
+
+
+# Two shipped packages a reference could not tell apart, each with the directories the refusal must name. The identity
+# is the one a reference is matched against: the address, then the manifest's name or else the directory's, without case.
+_DUPLICATE_IDENTITY_BUNDLES: dict[str, tuple[dict[str, str], tuple[str, ...]]] = {
+ "the same manifest twice": (_shipped_packages({"a": VENDORED_PROBE_MANIFEST, "b": VENDORED_PROBE_MANIFEST}), ("a", "b")),
+ "one named by its directory, in another case": (
+ _shipped_packages(
+ {
+ "first": _probe_manifest(address="github.com/invented/probe-lib", name="probe"),
+ "Probe": _probe_manifest(address="github.com/Invented/Probe-Lib", name=None),
+ }
+ ),
+ ("first", "Probe"),
+ ),
+}
+
+
+# Entries naming one file twice once their names are normalized, or a file that is another entry's directory, each with
+# the path the refusal must name. Each name is spelled differently, so the entries fit a `files` map as well as a zip.
+_COLLIDING_ENTRIES: dict[str, tuple[list[tuple[str, str]], str]] = {
+ "a file named twice": ([("main.mthds", VALID_MTHDS), ("notes.txt", "a"), ("./notes.txt", "b")], "'notes.txt'"),
+ "a shipped manifest named twice": (
+ [
+ *VENDORED_PROBE_FILES.items(),
+ (".mthds/methods/probe//METHODS.toml", _probe_manifest(address="github.com/invented/decoy", name="probe")),
+ ],
+ "'.mthds/methods/probe/METHODS.toml'",
+ ),
+ "a file that is another entry's directory": (
+ [("main.mthds", VALID_MTHDS), ("helper.txt", "x"), ("helper.txt/inner.txt", "y")],
+ "'helper.txt'",
+ ),
+}
+
+
+def _files_under(directory: Path) -> list[str]:
+ return sorted(path.relative_to(directory).as_posix() for path in directory.rglob("*") if path.is_file())
+
def _zip_b64(files: dict[str, str]) -> str:
+ return _zip_b64_of_entries(list(files.items()))
+
+
+def _zip_b64_of_entries(entries: list[tuple[str, str]]) -> str:
+ """A zip of the entries in order, which, unlike a `files` map, may hold one member name twice."""
buffer = io.BytesIO()
with zipfile.ZipFile(buffer, "w", zipfile.ZIP_DEFLATED) as archive:
- for name, content in files.items():
+ for name, content in entries:
archive.writestr(name, content)
return base64.b64encode(buffer.getvalue()).decode("ascii")
@@ -60,26 +170,29 @@ def _build_client(mocker: MockerFixture) -> tuple[TestClient, dict[str, Any]]:
}
fake_execute_response.pipe_output.tokens_usages = None
- def _record(library_dirs: list[str] | None, run_kwargs: dict[str, Any]) -> None:
+ def _record(runner_kwargs: dict[str, Any], run_kwargs: dict[str, Any]) -> None:
+ library_dirs: list[str] | None = runner_kwargs.get("library_dirs")
+ methods_dirs: list[Path] | None = runner_kwargs.get("methods_dirs")
snapshot["library_dirs"] = library_dirs
+ snapshot["methods_dirs"] = methods_dirs
snapshot["mthds_contents"] = run_kwargs.get("mthds_contents")
if library_dirs:
root = Path(library_dirs[0])
snapshot["dir_exists"] = root.exists()
- snapshot["files"] = sorted(path.relative_to(root).as_posix() for path in root.rglob("*") if path.is_file())
+ snapshot["files"] = _files_under(root)
else:
snapshot["files"] = None
+ snapshot["methods_files"] = [_files_under(Path(methods_dir)) for methods_dir in methods_dirs] if methods_dirs else None
def _make_runner(**kwargs: Any) -> Any:
- library_dirs: list[str] | None = kwargs.get("library_dirs")
runner = mocker.MagicMock()
async def _execute(**run_kwargs: Any) -> Any:
- _record(library_dirs, run_kwargs)
+ _record(kwargs, run_kwargs)
return fake_execute_response
async def _start(**run_kwargs: Any) -> Any:
- _record(library_dirs, run_kwargs)
+ _record(kwargs, run_kwargs)
return PipelexRunResultStart(
pipeline_run_id="run-1",
created_at="2026-01-15T12:00:00Z",
@@ -154,6 +267,27 @@ class TestPipelineBundle:
assert response.status_code == 422
assert response.json()["error_type"] == "InvalidBundle"
+ @pytest.mark.parametrize(
+ ("files", "named"),
+ [
+ pytest.param({"main.mthds": VALID_MTHDS, "notes.txt": "\ud800"}, "'notes.txt'", id="in-content"),
+ pytest.param({"main.mthds": VALID_MTHDS, "notes\ud800.txt": "x"}, r"'notes\ud800.txt'", id="in-name"),
+ ],
+ )
+ def test_a_files_entry_that_is_not_utf8_text_is_refused(self, mocker: MockerFixture, files: dict[str, str], named: str):
+ """A lone surrogate rides JSON as an escape but has no UTF-8 form, so the entry is refused, never written."""
+ client, snapshot = _build_client(mocker)
+ # The escaped `\ud800` is valid JSON text, which `json=` would refuse to encode.
+ raw_body = json.dumps({"files": files, "inputs": {"text": "hi"}})
+
+ response = client.post("/v1/execute", content=raw_body, headers={"content-type": "application/json"})
+
+ assert response.status_code == 422, response.text
+ problem = response.json()
+ assert problem["error_type"] == "InvalidBundle"
+ assert named in problem["detail"]
+ assert snapshot == {}, "the runner was reached"
+
def test_start_bundle_rides_mthds_contents(self, mocker: MockerFixture):
client, snapshot = _build_client(mocker)
response = client.post("/v1/start", json={"files": {"main.mthds": VALID_MTHDS}, "inputs": {"text": "hi"}})
@@ -169,3 +303,112 @@ class TestPipelineBundle:
)
assert response.status_code == 422
assert "mutually exclusive" in response.text
+
+ @pytest.mark.parametrize("transport", ["files", "bundle_b64"])
+ @pytest.mark.parametrize("path", ["/v1/execute", "/v1/start"])
+ def test_a_vendored_package_gets_its_own_methods_dir(self, mocker: MockerFixture, transport: str, path: str):
+ """A package under `.mthds/methods/<name>/` leaves `mthds_contents` and the library dir for a methods dir of its own."""
+ mocker.patch("pipelex_api.routes.pipelex.pipeline.is_pipe_func_sandbox_hosted", return_value=True)
+ client, snapshot = _build_client(mocker)
+ files = {**VENDORED_PROBE_FILES, "pipe_func.py": _PIPE_FUNC_PY}
+ body: dict[str, Any] = {"files": files} if transport == "files" else {"bundle_b64": _zip_b64(files)}
+
+ response = client.post(path, json={**body, "inputs": {"text": "hi"}})
+
+ assert response.status_code in {200, 202}, response.text
+ # The bundle's own file alone is the content; the package's bundle is not the caller's own domain.
+ assert snapshot["mthds_contents"] == [UNRESOLVED_PACKAGE_MTHDS]
+ # The library dir holds the bundle's own Python, and nothing of the package.
+ assert snapshot["files"] == ["pipe_func.py"]
+ # The package is in its own methods dir, laid out as `<name>/…`, a path no walk-up could take for a store.
+ (methods_dir,) = snapshot["methods_dirs"]
+ assert snapshot["methods_files"] == [["probe/METHODS.toml", "probe/probe.mthds"]]
+ assert ".mthds/methods" not in Path(methods_dir).as_posix()
+ # Cleaned once the request returns, like the library dir.
+ assert not Path(methods_dir).exists()
+
+ def test_a_bundle_shipping_no_package_gets_no_methods_dir(self, mocker: MockerFixture):
+ client, snapshot = _build_client(mocker)
+ response = client.post("/v1/execute", json={"files": {"main.mthds": VALID_MTHDS}, "inputs": {"text": "hi"}})
+ assert response.status_code == 200
+ assert snapshot["methods_dirs"] is None
+
+ @pytest.mark.parametrize("transport", ["files", "bundle_b64"])
+ @pytest.mark.parametrize("shape", list(_REFUSED_VENDORING_SHAPES))
+ def test_a_misplaced_mthds_entry_is_refused(self, mocker: MockerFixture, transport: str, shape: str):
+ """A `.mthds` path no reference could find is refused with the layout to use, never dropped."""
+ files, named = _REFUSED_VENDORING_SHAPES[shape]
+ client, snapshot = _build_client(mocker)
+ body: dict[str, Any] = {"files": files} if transport == "files" else {"bundle_b64": _zip_b64(files)}
+
+ response = client.post("/v1/execute", json={**body, "inputs": {"text": "hi"}})
+
+ assert response.status_code == 422, response.text
+ problem = response.json()
+ assert problem["error_type"] == "InvalidBundle"
+ assert named in problem["detail"]
+ assert ".mthds/methods/<name>/" in problem["detail"]
+ assert snapshot == {}, "the runner was reached"
+
+ @pytest.mark.parametrize("transport", ["files", "bundle_b64"])
+ @pytest.mark.parametrize("bundle", list(_DUPLICATE_IDENTITY_BUNDLES))
+ def test_two_shipped_packages_of_one_identity_are_refused(self, mocker: MockerFixture, transport: str, bundle: str):
+ """Which of two packages a reference would match is no choice to make by directory order: the bundle is refused."""
+ files, directories = _DUPLICATE_IDENTITY_BUNDLES[bundle]
+ client, snapshot = _build_client(mocker)
+ body: dict[str, Any] = {"files": files} if transport == "files" else {"bundle_b64": _zip_b64(files)}
+
+ response = client.post("/v1/execute", json={**body, "inputs": {"text": "hi"}})
+
+ assert response.status_code == 422, response.text
+ problem = response.json()
+ assert problem["error_type"] == "InvalidBundle"
+ assert VENDORED_PROBE_ADDRESS.casefold() in problem["detail"].casefold()
+ for directory in directories:
+ assert f"'.mthds/methods/{directory}/'" in problem["detail"]
+ assert snapshot == {}, "the runner was reached"
+
+ @pytest.mark.parametrize("transport", ["files", "bundle_b64"])
+ @pytest.mark.parametrize("case", list(_COLLIDING_ENTRIES))
+ def test_entries_colliding_on_one_path_are_refused(self, mocker: MockerFixture, transport: str, case: str):
+ """Two entries of one file would let a check read one copy while the last one written lands on disk: refused."""
+ entries, named = _COLLIDING_ENTRIES[case]
+ client, snapshot = _build_client(mocker)
+ body: dict[str, Any] = {"files": dict(entries)} if transport == "files" else {"bundle_b64": _zip_b64_of_entries(entries)}
+
+ response = client.post("/v1/execute", json={**body, "inputs": {"text": "hi"}})
+
+ assert response.status_code == 422, response.text
+ problem = response.json()
+ assert problem["error_type"] == "InvalidBundle"
+ assert named in problem["detail"]
+ assert snapshot == {}, "the runner was reached"
+
+ def test_a_zip_member_named_twice_is_refused(self, mocker: MockerFixture):
+ """A zip may hold one member name twice, spelled the same."""
+ client, snapshot = _build_client(mocker)
+ with pytest.warns(UserWarning, match="Duplicate name"):
+ bundle_b64 = _zip_b64_of_entries([("main.mthds", VALID_MTHDS), ("notes.txt", "a"), ("notes.txt", "b")])
+
+ response = client.post("/v1/execute", json={"bundle_b64": bundle_b64, "inputs": {"text": "hi"}})
+
+ assert response.status_code == 422, response.text
+ problem = response.json()
+ assert problem["error_type"] == "InvalidBundle"
+ assert "'notes.txt'" in problem["detail"]
+ assert snapshot == {}, "the runner was reached"
+
+ def test_two_shipped_packages_of_one_repository_are_both_shipped(self, mocker: MockerFixture):
+ """Packages sharing an address but not a name are two identities, as two packages of one library repository are."""
+ files = _shipped_packages(
+ {
+ "probe": VENDORED_PROBE_MANIFEST,
+ "sibling": _probe_manifest(address="github.com/invented/probe-lib", name="sibling"),
+ }
+ )
+ client, snapshot = _build_client(mocker)
+
+ response = client.post("/v1/execute", json={"files": files, "inputs": {"text": "hi"}})
+
+ assert response.status_code == 200, response.text
+ assert snapshot["methods_files"] == [["probe/METHODS.toml", "probe/probe.mthds", "sibling/METHODS.toml", "sibling/probe.mthds"]]
diff --git a/api/tests/unit/test_vendored_method_packages.py b/api/tests/unit/test_vendored_method_packages.py
new file mode 100644
index 000000000..7ad968a05
--- /dev/null
+++ b/api/tests/unit/test_vendored_method_packages.py
@@ -0,0 +1,311 @@
+"""A request carries the method packages its bundle calls, under `.mthds/methods/<name>/` at the bundle's root.
+
+The routes and the runner are real, and so is the in-process orchestrator of the run tests: what the run answers is
+what the engine resolved. Fetch-on-miss is off and the installed store is an empty temporary directory, so the shipped
+package is the only one that can answer, or, where a test installs a competing copy, the one that must win. The
+invented package's entry composes a text naming its copy, which a dry run renders too, so the output tells which ran.
+"""
+
+import base64
+import io
+import json
+import tempfile
+import zipfile
+from collections.abc import Callable, Generator, Mapping
+from pathlib import Path
+from typing import Any
+
+import pytest
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+from pipelex.base_exceptions import DisclosureMode
+from pipelex.config import get_config
+from pipelex.pipe_run.delivery_assignment import DeliveryAssignment
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.plugins.orchestrator_registry import OrchestratorRegistry
+from pipelex.runtime_bridge.payloads import PipelexPipeDispatchAck, PipelexPipeRunOutput
+from pytest_mock import MockerFixture
+
+from pipelex_api.api_config import ApiConfig
+from pipelex_api.exception_handlers import register_exception_handlers
+from pipelex_api.method_source import fetch_method_mthds_files, fetched_method_source
+from pipelex_api.routes import router as api_router
+from tests.unit._constants import (
+ STUB_METHOD_ADDRESS,
+ UNRESOLVED_PACKAGE_MTHDS,
+ UNRESOLVED_PACKAGE_REFERENCE,
+ VENDORED_PROBE_ADDRESS,
+ VENDORED_PROBE_FILES,
+ VENDORED_PROBE_MANIFEST,
+ VENDORED_PROBE_MARKER,
+ VENDORED_PROBE_MTHDS,
+)
+
+_PIPELINE_NS = "pipelex_api.routes.pipelex.pipeline"
+_METHOD_REF = f"{STUB_METHOD_ADDRESS}@v0.1.0"
+_INSTALLED_MARKER = "INSTALLED copy"
+
+# The shipped package's bundle, made unparsable two ways: as TOML, and as a blueprint whose `main_pipe` is no pipe code.
+_UNPARSABLE_PROBE_BUNDLES: dict[str, str] = {
+ "toml syntax": 'domain = "probe_dep"\n\n[pipe.entry\ntype = "PipeLLM"\n',
+ "invalid blueprint": VENDORED_PROBE_MTHDS.replace('domain = "probe_dep"', 'domain = "probe_dep"\nmain_pipe = "Not A Pipe Code!"'),
+}
+
+
+def _structures_py(*, sentinel: Path) -> str:
+ """A structure class whose module, if it were ever imported, would write the sentinel file."""
+ return f"""\
+from pathlib import Path
+
+from pipelex.core.stuffs.structured_content import StructuredContent
+
+Path({str(sentinel)!r}).write_text("imported", encoding="utf-8")
+
+
+class Invoice(StructuredContent):
+ total: float
+"""
+
+
+class _RecordingOrchestrator:
+ """An async-capable orchestrator that acknowledges a start and records what each dispatched job's library holds."""
+
+ supports_fire_and_forget = True
+
+ def __init__(self) -> None:
+ self.dispatches: list[str] = []
+
+ async def execute(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeRunOutput: # noqa: ARG002
+ msg = "These tests dispatch through /start only."
+ raise AssertionError(msg)
+
+ async def start(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeDispatchAck: # noqa: ARG002
+ self.dispatches.append(pipe_job.pipe.code)
+ return PipelexPipeDispatchAck(pipeline_run_id="run-1", workflow_id="wf-1")
+
+
[email protected](name="sandbox_hosted_mode")
+def sandbox_hosted_mode_fixture() -> Generator[None, None, None]:
+ """Select a non-`direct` PipeFunc execution mode, which makes the deployment sandbox-hosted for the route and the loader alike."""
+ pipe_func_config = get_config().interpreter.pipe_func
+ previous = pipe_func_config.execution_mode
+ pipe_func_config.execution_mode = "sandbox"
+ try:
+ yield
+ finally:
+ pipe_func_config.execution_mode = previous
+
+
+def _build_client(
+ mocker: MockerFixture,
+ *,
+ tmp_path: Path,
+ orchestrator: _RecordingOrchestrator | None = None,
+ disclosure_mode: DisclosureMode = DisclosureMode.STRICT,
+) -> TestClient:
+ mocker.patch("pipelex.cli.installed_methods.GLOBAL_METHODS_DIR", tmp_path / "global-methods")
+ mocker.patch("pipelex.cli.installed_methods.PROJECT_METHODS_DIR", tmp_path / "project-methods")
+ mocker.patch("pipelex.methods.fetch_on_miss.is_method_fetch_on_miss_enabled", return_value=False)
+ mocker.patch("pipelex.methods.fetch_on_miss.fetch_method_package", side_effect=AssertionError("nothing may be fetched"))
+ mocker.patch(
+ f"{_PIPELINE_NS}.get_api_config", return_value=ApiConfig(orchestration_mode="direct", allow_request_orchestration_mode_override=False)
+ )
+ if orchestrator is not None:
+ mocker.patch(f"{_PIPELINE_NS}.get_orchestrator_registry", return_value=OrchestratorRegistry({"direct": orchestrator}))
+ app = FastAPI()
+ app.include_router(api_router, prefix="/v1")
+ register_exception_handlers(app, disclosure_mode=disclosure_mode)
+ return TestClient(app)
+
+
+def _install_competing_probe(*, tmp_path: Path) -> None:
+ """Install the same package into the isolated store, its entry describing itself as the installed copy."""
+ package_dir = tmp_path / "global-methods" / "probe"
+ package_dir.mkdir(parents=True)
+ (package_dir / "METHODS.toml").write_text(VENDORED_PROBE_MANIFEST, encoding="utf-8")
+ (package_dir / "probe.mthds").write_text(VENDORED_PROBE_MTHDS.replace(VENDORED_PROBE_MARKER, _INSTALLED_MARKER), encoding="utf-8")
+
+
+def _run_output(body: dict[str, Any]) -> str:
+ """The run's working memory as text, where the composed answer names the copy of the package that ran."""
+ return json.dumps(body["pipe_output"]["working_memory"])
+
+
+def _execute_vendored(client: TestClient, *, files: dict[str, str]) -> Any:
+ return client.post("/v1/execute", json={"files": files, "inputs": {"text": "hello"}})
+
+
+def _zip_b64(files: Mapping[str, str | bytes]) -> str:
+ """A `bundle_b64` of the files, which, unlike a `files` map, can carry bytes that are not text."""
+ buffer = io.BytesIO()
+ with zipfile.ZipFile(buffer, "w", zipfile.ZIP_DEFLATED) as archive:
+ for name, content in files.items():
+ archive.writestr(name, content)
+ return base64.b64encode(buffer.getvalue()).decode("ascii")
+
+
+class TestVendoredMethodPackages:
+ def test_a_shipped_package_answers_the_call_with_fetching_off(self, mocker: MockerFixture, tmp_path: Path):
+ client = _build_client(mocker, tmp_path=tmp_path)
+
+ response = _execute_vendored(client, files=VENDORED_PROBE_FILES)
+
+ assert response.status_code == 200, response.text
+ body = response.json()
+ assert body["state"] == "COMPLETED"
+ assert f"{VENDORED_PROBE_MARKER}: hello" in _run_output(body)
+ # The package ends with the request: nothing was installed into the store.
+ assert not (tmp_path / "global-methods").exists()
+
+ def test_the_shipped_copy_wins_over_an_installed_one(self, mocker: MockerFixture, tmp_path: Path):
+ client = _build_client(mocker, tmp_path=tmp_path)
+ _install_competing_probe(tmp_path=tmp_path)
+
+ response = _execute_vendored(client, files=VENDORED_PROBE_FILES)
+
+ assert response.status_code == 200, response.text
+ run_output = _run_output(response.json())
+ assert f"{VENDORED_PROBE_MARKER}: hello" in run_output
+ assert _INSTALLED_MARKER not in run_output
+
+ def test_a_shipped_provenance_sidecar_that_is_not_utf8_is_ignored(self, mocker: MockerFixture, tmp_path: Path):
+ """A sidecar the server cannot read records no provenance, as an unparsable one does, and the shipped copy runs."""
+ client = _build_client(mocker, tmp_path=tmp_path)
+ bundle_b64 = _zip_b64({**VENDORED_PROBE_FILES, ".mthds/methods/probe/.provenance.json": b"\xff\xfe garbage"})
+
+ response = client.post("/v1/execute", json={"bundle_b64": bundle_b64, "inputs": {"text": "hello"}})
+
+ assert response.status_code == 200, response.text
+ assert f"{VENDORED_PROBE_MARKER}: hello" in _run_output(response.json())
+
+ def test_start_hands_the_job_a_library_holding_the_shipped_package(self, mocker: MockerFixture, tmp_path: Path):
+ orchestrator = _RecordingOrchestrator()
+ client = _build_client(mocker, tmp_path=tmp_path, orchestrator=orchestrator)
+
+ response = client.post("/v1/start", json={"files": VENDORED_PROBE_FILES, "inputs": {"text": "hello"}})
+
+ assert response.status_code == 202, response.text
+ assert orchestrator.dispatches == ["echo"]
+
+ @pytest.mark.parametrize("broken", list(_UNPARSABLE_PROBE_BUNDLES))
+ def test_a_shipped_bundle_that_does_not_parse_is_the_verdict(self, mocker: MockerFixture, tmp_path: Path, broken: str):
+ """The package is the caller's own content, so its parse failure is the verdict, located by address, never by host path."""
+ client = _build_client(mocker, tmp_path=tmp_path, disclosure_mode=DisclosureMode.STRICT)
+ files = {**VENDORED_PROBE_FILES, ".mthds/methods/probe/probe.mthds": _UNPARSABLE_PROBE_BUNDLES[broken]}
+
+ response = _execute_vendored(client, files=files)
+
+ assert response.status_code == 422, response.text
+ body = response.json()
+ assert body["error_type"] == "ValidateBundleError"
+ items = body["validation_errors"]
+ assert items, body
+ assert {item["source"] for item in items} == {f"{VENDORED_PROBE_ADDRESS}/probe.mthds"}
+ assert VENDORED_PROBE_ADDRESS in body["detail"]
+ dump = json.dumps(body)
+ assert "pipelex-methods-" not in dump
+ for temporary_root in (tempfile.gettempdir(), str(Path(tempfile.gettempdir()).resolve())):
+ assert temporary_root not in dump
+
+ @pytest.mark.usefixtures("sandbox_hosted_mode")
+ def test_a_shipped_structure_class_is_refused_unimported(self, mocker: MockerFixture, tmp_path: Path):
+ sentinel = tmp_path / "structures-module-ran"
+ orchestrator = _RecordingOrchestrator()
+ client = _build_client(mocker, tmp_path=tmp_path, orchestrator=orchestrator)
+ files = {**VENDORED_PROBE_FILES, ".mthds/methods/probe/structures.py": _structures_py(sentinel=sentinel)}
+
+ response = client.post("/v1/start", json={"files": files, "inputs": {"text": "hello"}})
+
+ assert response.status_code == 403, response.text
+ body = response.json()
+ assert body["error_type"] == "MethodStructuresRefusedError"
+ # Named as the fetched package's refusal is: by the package's address, and the file by its path inside it.
+ assert f"Method package '{VENDORED_PROBE_ADDRESS}'" in body["detail"]
+ assert "structures.py defines Invoice" in body["detail"]
+ assert not sentinel.exists(), "the structure module was imported"
+ assert orchestrator.dispatches == []
+
+ def test_a_method_ref_package_runs_the_dependency_it_ships(
+ self, mocker: MockerFixture, tmp_path: Path, install_method_package: Callable[..., Path]
+ ):
+ install_method_package(files={"smoke.mthds": UNRESOLVED_PACKAGE_MTHDS, **_package_vendoring_files()})
+ client = _build_client(mocker, tmp_path=tmp_path)
+
+ response = client.post("/v1/execute", json={"method_ref": _METHOD_REF, "inputs": {"text": "hello"}})
+
+ assert response.status_code == 200, response.text
+ assert f"{VENDORED_PROBE_MARKER}: hello" in _run_output(response.json())
+
+ def test_a_method_ref_package_shipping_one_identity_twice_is_refused(
+ self, mocker: MockerFixture, tmp_path: Path, install_method_package: Callable[..., Path]
+ ):
+ shipped_twice = {
+ f".mthds/methods/{directory}/{relpath.removeprefix('.mthds/methods/probe/')}": content
+ for directory in ("a", "b")
+ for relpath, content in _package_vendoring_files().items()
+ }
+ install_method_package(files={"smoke.mthds": UNRESOLVED_PACKAGE_MTHDS, **shipped_twice})
+ client = _build_client(mocker, tmp_path=tmp_path)
+
+ response = client.post("/v1/execute", json={"method_ref": _METHOD_REF, "inputs": {"text": "hello"}})
+
+ assert response.status_code == 422, response.text
+ body = response.json()
+ assert body["error_type"] == "InvalidBundle"
+ assert f"Method package '{STUB_METHOD_ADDRESS}'" in body["detail"]
+ assert VENDORED_PROBE_ADDRESS in body["detail"]
+ assert "'.mthds/methods/a/'" in body["detail"]
+ assert "'.mthds/methods/b/'" in body["detail"]
+
+ def test_a_method_ref_package_keeps_its_shipped_files_apart(self, install_method_package: Callable[..., Path]):
+ install_method_package(files={"smoke.mthds": UNRESOLVED_PACKAGE_MTHDS, **_package_vendoring_files()})
+
+ with fetched_method_source(_METHOD_REF) as fetched:
+ assert fetched.mthds_sources == ["smoke.mthds"]
+ assert fetched.mthds_contents == [UNRESOLVED_PACKAGE_MTHDS]
+ assert fetched.library_dirs is None
+ assert fetched.methods_dirs is not None
+ (methods_dir,) = fetched.methods_dirs
+ assert sorted(path.relative_to(methods_dir).as_posix() for path in methods_dir.rglob("*") if path.is_file()) == [
+ "probe/METHODS.toml",
+ "probe/probe.mthds",
+ ]
+ assert not methods_dir.exists()
+
+ tooling = fetch_method_mthds_files(_METHOD_REF)
+ assert [item.source for item in tooling.files] == ["smoke.mthds"]
+
+ def test_validate_by_method_ref_validates_only_the_package_s_own_content(
+ self, mocker: MockerFixture, tmp_path: Path, install_method_package: Callable[..., Path]
+ ):
+ """`/v1/validate` leaves a shipped package out of the content; the dependency resolves from the installed store there."""
+ install_method_package(files={"smoke.mthds": UNRESOLVED_PACKAGE_MTHDS, **_package_vendoring_files()})
+ client = _build_client(mocker, tmp_path=tmp_path)
+ _install_competing_probe(tmp_path=tmp_path)
+
+ response = client.post("/v1/validate", json={"method_ref": _METHOD_REF})
+
+ assert response.status_code == 200, response.text
+ body = response.json()
+ assert body["is_valid"] is True, body
+ assert body["mthds_contents"] == [UNRESOLVED_PACKAGE_MTHDS]
+
+ def test_validate_by_method_ref_does_not_search_the_shipped_package(
+ self, mocker: MockerFixture, tmp_path: Path, install_method_package: Callable[..., Path]
+ ):
+ install_method_package(files={"smoke.mthds": UNRESOLVED_PACKAGE_MTHDS, **_package_vendoring_files()})
+ client = _build_client(mocker, tmp_path=tmp_path)
+
+ response = client.post("/v1/validate", json={"method_ref": _METHOD_REF})
+
+ assert response.status_code == 200, response.text
+ body = response.json()
+ assert body["is_valid"] is False
+ assert [(item["error_type"], item["missing_pipe_code"]) for item in body["validation_errors"]] == [
+ ("unresolved_package_dependency", UNRESOLVED_PACKAGE_REFERENCE)
+ ]
+
+
+def _package_vendoring_files() -> dict[str, str]:
+ """The shipped package's files, without the consumer's bundle `VENDORED_PROBE_FILES` also holds."""
+ return {relpath: content for relpath, content in VENDORED_PROBE_FILES.items() if relpath.startswith(".mthds/")}
diff --git a/docs/api-server/codegen.md b/docs/api-server/codegen.md
index 29f012059..dd6adf4a4 100644
--- a/docs/api-server/codegen.md
+++ b/docs/api-server/codegen.md
@@ -11,7 +11,7 @@ Both endpoints speak the [`POST /v1/validate`](pipe-validate.md) verdict discipl
Both endpoints accept the same closure selector — exactly one of:
- `files` (list, content-passing): inline MTHDS bundles, each `{ "content": "...", "source": "optional/logical/path.mthds" }`. `source` threads onto diagnostics and the crate's `source_map`.
-- `method_ref` (string): a reference to a published method. The **address form** — `github.com/<owner>/<repo>` plus an optional package selector and `@<tag>` — is resolved server-side through the same fetch path as a [`method_ref` run](pipe-run.md#running-a-method-by-address-method_ref): the package's `.mthds` files feed the closure, each with its real relative path as `source` (so diagnostics and the crate's `source_map` carry true per-file labels). Only `.mthds` data travels on these routes — the package's Python (if any) never loads here. The **registry form** (any non-address reference) stays reserved: the server answers **`501`** (`MethodRefNotSupported`) until method-registry resolution lands.
+- `method_ref` (string): a reference to a published method. The **address form** — `github.com/<owner>/<repo>` plus an optional package selector and `@<tag>` — is resolved server-side through the same fetch path as a [`method_ref` run](pipe-run.md#running-a-method-by-address-method_ref): the package's own `.mthds` files feed the closure, each with its real relative path as `source` (so diagnostics and the crate's `source_map` carry true per-file labels), and the files of a method package it ships under `.mthds/methods/<name>/` do not. Only `.mthds` data travels on these routes — the package's Python (if any) never loads here. The **registry form** (any non-address reference) stays reserved: the server answers **`501`** (`MethodRefNotSupported`) until method-registry resolution lands.
Providing neither or both is a request-shape `422`.
diff --git a/docs/api-server/openapi/pipelex-api.openapi.yaml b/docs/api-server/openapi/pipelex-api.openapi.yaml
index 46284b321..b51ed3557 100644
--- a/docs/api-server/openapi/pipelex-api.openapi.yaml
+++ b/docs/api-server/openapi/pipelex-api.openapi.yaml
@@ -149,7 +149,9 @@ paths:
title: Bundle B64
description: PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — base64-encoded zip of the whole method
bundle (`.mthds` + `.py` + `structures/*.py` + `requirements.txt`), materialized into a temporary library
- directory before the run so custom PipeFunc Python travels with the method. Mutually exclusive with `files`.
+ directory before the run so custom PipeFunc Python travels with the method. The method packages the bundle
+ calls by address may travel with it under `.mthds/methods/<name>/`, each with its `METHODS.toml`; they
+ are searched before the runner's installed copies, for this run alone. Mutually exclusive with `files`.
Custom `.py` is only honored on a sandbox-hosted deployment.
files:
anyOf:
@@ -403,7 +405,9 @@ paths:
title: Bundle B64
description: PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — base64-encoded zip of the whole method
bundle (`.mthds` + `.py` + `structures/*.py` + `requirements.txt`), materialized into a temporary library
- directory before the run so custom PipeFunc Python travels with the method. Mutually exclusive with `files`.
+ directory before the run so custom PipeFunc Python travels with the method. The method packages the bundle
+ calls by address may travel with it under `.mthds/methods/<name>/`, each with its `METHODS.toml`; they
+ are searched before the runner's installed copies, for this run alone. Mutually exclusive with `files`.
Custom `.py` is only honored on a sandbox-hosted deployment.
files:
anyOf:
diff --git a/docs/api-server/pipe-run.md b/docs/api-server/pipe-run.md
index 12ab74dc0..6d5e11463 100644
--- a/docs/api-server/pipe-run.md
+++ b/docs/api-server/pipe-run.md
@@ -35,7 +35,7 @@ Execute a Pipelex pipeline with flexible inputs and wait for completion.
- `output_name` (string, optional): Name for the output slot
- `output_multiplicity` (string, optional): Output multiplicity setting (`"single"`, `"variable"`, or a specific number)
- `dynamic_output_concept_ref` (string, optional): Override output concept ref
-- `bundle_b64` (string, optional): **Pipelex-API extension.** Base64-encoded zip of a whole method bundle — see [Shipping a method bundle](#shipping-a-method-bundle-custom-pipefunc). Mutually exclusive with `files`.
+- `bundle_b64` (string, optional): **Pipelex-API extension.** Base64-encoded zip of a whole method bundle, which may carry the method packages it calls — see [Shipping a method bundle](#shipping-a-method-bundle-custom-pipefunc). Mutually exclusive with `files`.
- `files` (dict[str, str], optional): **Pipelex-API extension.** The same bundle as a `{relative_path: text}` map (the unzipped equivalent of `bundle_b64`). Mutually exclusive with `bundle_b64`.
- `method_ref` (string, optional): **Pipelex-API extension.** Run a published method by address instead of inline source — see [Running a method by address](#running-a-method-by-address-method_ref). Mutually exclusive with `mthds_contents` and with a method bundle; `pipe_code` may accompany it to override the package's `main_pipe`.
- `storage_scope` (string, optional): **Pipelex-API extension.** The prefix every object this run writes lands under — see [Host-supplied run context](#host-supplied-run-context-storage_scope-and-analytics_groups).
@@ -115,7 +115,7 @@ Start a pipeline execution and get its `pipeline_run_id` back with a `202` ack.
- `output_name` (string, optional): Name for the output slot
- `output_multiplicity` (string, optional): Output multiplicity setting (`"single"`, `"variable"`, or a specific number)
- `dynamic_output_concept_ref` (string, optional): Override output concept ref
-- `bundle_b64` (string, optional): **Pipelex-API extension.** Base64-encoded zip of a whole method bundle — see [Shipping a method bundle](#shipping-a-method-bundle-custom-pipefunc). Mutually exclusive with `files`.
+- `bundle_b64` (string, optional): **Pipelex-API extension.** Base64-encoded zip of a whole method bundle, which may carry the method packages it calls — see [Shipping a method bundle](#shipping-a-method-bundle-custom-pipefunc). Mutually exclusive with `files`.
- `files` (dict[str, str], optional): **Pipelex-API extension.** The same bundle as a `{relative_path: text}` map (the unzipped equivalent of `bundle_b64`). Mutually exclusive with `bundle_b64`.
- `method_ref` (string, optional): **Pipelex-API extension.** Run a published method by address instead of inline source — see [Running a method by address](#running-a-method-by-address-method_ref). Mutually exclusive with `mthds_contents` and with a method bundle; `pipe_code` may accompany it to override the package's `main_pipe`.
- `storage_scope` (string, optional): **Pipelex-API extension.** The prefix every object this run writes lands under — see [Host-supplied run context](#host-supplied-run-context-storage_scope-and-analytics_groups).
@@ -232,12 +232,40 @@ The server materializes the bundle into a temporary library directory for the ru
- A hard **file-count** ceiling (`MAX_BUNDLE_FILES`) and a **total decompressed-size** ceiling (`MAX_BUNDLE_TOTAL_KIB`) → `413 PayloadTooLarge`. The zip path bounds actual decompression, so a zip bomb cannot expand past the ceiling, and an oversized `bundle_b64` is refused on its encoded length *before* it is decoded into memory.
- **Path safety:** entry names that are absolute, use `..` traversal, use backslashes, or carry a Windows drive/`:` form → `422 InvalidBundle`.
-- Supplying **both** `bundle_b64` and `files`, an empty bundle, or a corrupt zip → `422 InvalidBundle`; invalid base64 → `400 InvalidBase64`.
+- **One entry per path:** two entries naming one file once their names are normalized (`a/./b` and `a//b` are `a/b`), which a zip can also hold as one member name twice, and a file that is also the directory of another entry → `422 InvalidBundle` naming the path.
+- Supplying **both** `bundle_b64` and `files`, an empty bundle, a corrupt zip, or a `files` entry whose name or text holds a lone surrogate (which JSON can escape but UTF-8 cannot encode) → `422 InvalidBundle`; invalid base64 → `400 InvalidBase64`.
**Sandbox-hosted only for custom Python.** A bundle that ships any `.py` is honored **only on a sandbox-hosted deployment**, where the load path reads the source without importing it and execution happens in an isolated sandbox. On a non-hosted deployment such a bundle is refused with `403 CustomCodeRequiresSandbox` — running caller-supplied code in-process is never done implicitly. A bundle that carries only `.mthds` (no `.py`) is accepted on any deployment.
**No Python structure classes.** A structure class (a `StructuredContent` subclass) can only be used by importing its module into the runner's own process, which a sandbox-hosted deployment never does, so a bundle whose Python declares one is refused with a `403` (`MethodStructuresRefusedError`) naming each file and class, before anything is loaded. The rule is the same however the method arrives: inline as a bundle, or fetched by `method_ref`. Declare the types as MTHDS concepts with inline structures instead; a PipeFunc returns them by importing the classes the sandbox generates from those concepts (`from structures import <domain>__<Concept>`). The `structures` module that `pipelex build structures` writes is accepted as long as it is left as generated, and it is not sent to the sandbox, which generates its own.
+### Shipping the method packages the bundle calls
+
+A bundle can call a pipe of another method package by its address, such as `github.com/Pipelex/methods/documents->documents.extract_page_contents_and_views`. A `files` or `bundle_b64` request can carry that package with the bundle, so the run needs no copy installed on the server and no fetch. Put each package under `.mthds/methods/<name>/` at the bundle's root, with its `METHODS.toml` at `.mthds/methods/<name>/METHODS.toml`, which is the layout of a local method store:
+
+```text
+bundle.mthds
+.mthds/methods/documents/METHODS.toml
+.mthds/methods/documents/documents.mthds
+```
+
+- **A reference finds the package by its manifest.** The package that answers is the one whose `METHODS.toml` declares the referenced address and name, whatever its directory is called. A reference pinned with `@<tag>` is answered by the shipped copy too, and the server logs a warning when the copy's `version` is not that tag.
+- **The shipped copy comes first.** The server searches the shipped packages before the copies installed on it and before fetch-on-miss, so a shipped package wins over an installed copy of the same address, and it runs even where fetching is disabled or the address is not one the server can fetch. It serves this run alone: nothing is installed.
+- **It is not the bundle's own content.** A shipped package's files are kept out of the bundle's `.mthds` content and out of its library directory, so its domains never merge with the bundle's, and a bundle whose only `.mthds` files are shipped ones is refused as having no `.mthds` file of its own.
+- **Its Python never runs in the server's process.** A `.py` file in a shipped package counts for the sandbox gate like any other, so a deployment that is not sandbox-hosted refuses the bundle with `403 CustomCodeRequiresSandbox`. On a sandbox-hosted deployment, a shipped package whose Python declares a structure class is refused with `403 MethodStructuresRefusedError` naming the package's address, as a package fetched by address is.
+- **A shipped bundle that does not parse refuses the run** with the `422` verdict, whose items name the file as `<address>/<path inside the package>`, never by a path on the server.
+
+Any other `.mthds` path in the bundle is refused with `422 InvalidBundle`, naming the entry and the expected layout, rather than dropped, and so are shipped packages a reference could not tell apart:
+
+- a file under `.mthds/` outside `.mthds/methods/<name>/`, such as `.mthds/foo` or a file directly under `.mthds/methods/`;
+- a `.mthds/methods/` directory below the bundle's root, such as `sub/.mthds/methods/…`;
+- a `.mthds/` directory inside a shipped package, since a package's own dependencies are not loaded;
+- a package directory whose name starts with a dot;
+- a package directory without a `METHODS.toml`, or whose `METHODS.toml` is not a valid manifest;
+- two package directories whose manifests declare the same address and name, compared without regard to case as a reference is matched, the name being the directory's when the manifest has none. The refusal names the address and each directory, since nothing but their order would decide which one answers.
+
+Shipping packages is a feature of the run routes. `POST /v1/validate` takes `mthds_contents` or `method_ref`, and a package reference in the validated content resolves there through the server's installed copies and fetch-on-miss only.
+
---
## Running a method by address (`method_ref`)
@@ -253,7 +281,7 @@ Both `/execute` and `/start` accept **`method_ref`** — a globally resolvable a
**The reference grammar is `<address>[@<tag>]`.** The address is `github.com/<owner>/<repo>` optionally followed by a package selector within the repository (`github.com` only for now). A bare address means the repository's default branch at HEAD; `@<tag>` pins the repository at that **git tag** (recommended form `vX.Y.Z`) — branch names are refused, so a reference can never silently track a moving branch. Full browser URLs (`https://github.com/...`, including `/tree/<branch>/...` deep links) are accepted and normalized.
-**Resolution.** The server fetches the repository (shallow clone, at the tag when one is named), records the **commit SHA** of what was fetched, and locates the requested package **by manifest identity**: it scans the clone for `METHODS.toml` manifests and selects the one whose declared address (plus `name`, for a package in a library repo) equals the requested address — never by directory path. The package's `.mthds` files then run exactly like an inline submission; its non-`.mthds` files travel like a bundle's.
+**Resolution.** The server fetches the repository (shallow clone, at the tag when one is named), records the **commit SHA** of what was fetched, and locates the requested package **by manifest identity**: it scans the clone for `METHODS.toml` manifests and selects the one whose declared address (plus `name`, for a package in a library repo) equals the requested address — never by directory path. The package's `.mthds` files then run exactly like an inline submission; its non-`.mthds` files travel like a bundle's, and so do the packages it ships under its own `.mthds/methods/<name>/` ([Shipping the method packages the bundle calls](#shipping-the-method-packages-the-bundle-calls)), under the same refusals. `POST /v1/validate` by `method_ref` keeps those shipped packages out of the validated content without searching them, so a dependency the package ships resolves there through the installed copies and fetch-on-miss only.
**Entry pipe.** The pipe to run defaults to the package manifest's `main_pipe`; an explicit `pipe_code` in the request overrides it (to run another pipe from the fetched package).
diff --git a/docs/api-server/pipe-validate.md b/docs/api-server/pipe-validate.md
index 354c0a145..daf9b21f1 100644
--- a/docs/api-server/pipe-validate.md
+++ b/docs/api-server/pipe-validate.md
@@ -17,7 +17,7 @@ Validate MTHDS content by parsing, loading, and dry-running pipes without execut
**Request Fields:**
- `mthds_contents` (list[str], optional): MTHDS contents to validate (always an array, even for a single file). Exactly one of `mthds_contents` / `method_ref` must be provided
-- `method_ref` (string, optional): **Pipelex-API extension.** Validate a published method by address — `github.com/<owner>/<repo>` plus an optional package selector and `@<tag>` — resolved by this server through the same fetch path as a [`method_ref` run](pipe-run.md#running-a-method-by-address-method_ref): the repository is fetched at the tag, the package located by manifest identity, and its `.mthds` files validated with their **real relative paths** feeding the per-file sources (so `validation_errors[].source` names the owning file). Exactly one of `mthds_contents` / `method_ref`; `mthds_sources` cannot accompany it
+- `method_ref` (string, optional): **Pipelex-API extension.** Validate a published method by address — `github.com/<owner>/<repo>` plus an optional package selector and `@<tag>` — resolved by this server through the same fetch path as a [`method_ref` run](pipe-run.md#running-a-method-by-address-method_ref): the repository is fetched at the tag, the package located by manifest identity, and its `.mthds` files validated with their **real relative paths** feeding the per-file sources (so `validation_errors[].source` names the owning file). The method packages the package ships under its own `.mthds/methods/<name>/` are not part of the validated content, and validation does not search them: a call into one resolves here through the server's installed copies and fetch-on-miss only, while a [run](pipe-run.md#shipping-the-method-packages-the-bundle-calls) uses the shipped copy first. Exactly one of `mthds_contents` / `method_ref`; `mthds_sources` cannot accompany it
- `allow_signatures` (boolean, optional, default `false`): controls only the **sweep mechanics** for `PipeSignature` placeholders — whether signature pipes are mock-run during the dry-run sweep and therefore listed in `validated_pipes`. It does **not** change the verdict: an unimplemented signature is never a rejection, it is a *runnability fact* reported via `pending_signatures` + `is_runnable` in both modes
- `mthds_sources` (list[str] | null, optional): per-file sources, parallel to `mthds_contents` — see [Sourcing submitted files](#sourcing-submitted-files). When present it must match `mthds_contents` in length (a mismatch is a 422 request error)
- `render` (list[str], optional, default `[]`): opt-in *rendered text* to attach to the verdict — see [Opt-in extras](#opt-in-extras-render-and-views)
diff --git a/docs/building-methods/packages.md b/docs/building-methods/packages.md
index d5d78d892..efc90474a 100644
--- a/docs/building-methods/packages.md
+++ b/docs/building-methods/packages.md
@@ -130,11 +130,13 @@ steps = [
]
```
-An address-based reference resolves against the **installed methods** — `~/.mthds/methods/` (global) and `.mthds/methods/` (project-local), matched by the manifest's `address` + `name`, case-insensitively. When no installed method matches, Pipelex **fetches on miss**: it fetches the package by address — honoring an `@<tag>` pin, through the same grammar, bounds, and tags-only rules as a direct fetch — installs it into `~/.mthds/methods/`, records the fetch provenance (address, tag, and the commit SHA of what was actually cloned) in a `.provenance.json` sidecar beside the manifest, and loads it so resolution proceeds. The next load finds the installed copy and never touches the network again.
+An address-based reference resolves against the **installed methods** — `~/.mthds/methods/` (global), `.mthds/methods/` in the working directory (project-local), and a `.mthds/methods/` directory beside the bundle being loaded or in a directory above it — matched by the manifest's `address` + `name`, case-insensitively. When no installed method matches, Pipelex **fetches on miss**: it fetches the package by address — honoring an `@<tag>` pin, through the same grammar, bounds, and tags-only rules as a direct fetch — installs it into `~/.mthds/methods/`, records the fetch provenance (address, tag, and the commit SHA of what was actually cloned) in a `.provenance.json` sidecar beside the manifest, and loads it so resolution proceeds. The next load finds the installed copy and never touches the network again.
A few behaviors worth knowing:
-- **Installed wins.** Once a method is installed, it is used as-is — including when a reference pins a different `@<tag>` than what is installed (a warning tells you so). Remove the installed directory to re-fetch.
+- **Installed wins over a fetch.** Once a method is installed, it is used as-is — including when a reference pins a different `@<tag>` than what is installed (a warning tells you so). Remove the installed directory to re-fetch.
+- **A package can travel with the bundle.** A bundle sent over HTTP to a Pipelex API server as `files` or `bundle_b64` can carry the packages it calls under `.mthds/methods/<name>/` at its root, each with its `METHODS.toml`. The server searches them before its installed copies and before fetch-on-miss, for that run alone, so a shipped package wins over an installed copy and runs where fetching is disabled; see [Shipping the method packages the bundle calls](../api-server/pipe-run.md#shipping-the-method-packages-the-bundle-calls). On disk, the same layout beside the bundle is one of the local stores above.
+- **A package bundle that does not parse refuses the load.** When a bundle of a referenced package does not parse, the bundle calling it is refused with the parser's items, each naming the file as `<address>/<path inside the package>`, never by its path on the machine, whether the package was shipped with the bundle, installed or fetched.
- **A miss that cannot be bridged refuses the bundle, never a silent pass.** Fetch disabled, an address this runtime cannot fetch, or a failed fetch (a repository that does not exist, a `@tag` that is not a tag, a repository holding no package at that address) refuses the load with one `unresolved_package_dependency` item per package, located on the first pipe that calls it, with the reference as written in `missing_pipe_code` and the reason and the remedy in its message. It is the same invalid verdict as any other mistake in the bundle: `pipelex validate` reports it, `POST /v1/validate` answers `200` with `is_valid: false`, and a run is refused with a `422` before any pipe runs. Every package of the bundle is tried before the refusal, so two bad references give two items.
- **Hosted parity warning.** A fetched method declaring Python structure classes runs locally but triggers the same warning as a direct CLI fetch: hosted execution accepts MTHDS concepts and sandboxed PipeFuncs, not in-process Python. On a sandbox-hosted deployment the fetch refuses such a package outright, with the same rule-naming error. The refusal is not specific to fetching: a sandbox-hosted deployment refuses every method whose Python declares structure classes, however it arrived (see [Python classes](concepts/python-classes.md)).
@@ -149,7 +151,7 @@ fetch_on_miss = false
export PIPELEX_METHODS_FETCH_ON_MISS=0
```
-With fetching disabled, a miss refuses the bundle with that item, whose message names the address and says to install the package where the runtime runs (for example with `mthds install <address>`). See [Methods Configuration](../configuration/config-practical/methods-config.md).
+With fetching disabled, a miss refuses the bundle with that item, whose message names the address and says to ship the package with the bundle under `.mthds/methods/<name>/` or to install it where the runtime runs (for example with `mthds install <address>`). See [Methods Configuration](../configuration/config-practical/methods-config.md).
### Example Layout
diff --git a/docs/configuration/config-practical/methods-config.md b/docs/configuration/config-practical/methods-config.md
index 69f459c64..2110ba07d 100644
--- a/docs/configuration/config-practical/methods-config.md
+++ b/docs/configuration/config-practical/methods-config.md
@@ -15,7 +15,7 @@ class MethodsConfig(ConfigModel):
### Fields
-- `fetch_on_miss`: When a bundle references another method by address (`github.com/...->domain.pipe`) and no installed method matches, fetch the package by address and install it into `~/.mthds/methods/` so the load can proceed. When disabled, such a miss refuses the bundle with an `unresolved_package_dependency` item naming the address and how to install the package, and the network is never touched at load time.
+- `fetch_on_miss`: When a bundle references another method by address (`github.com/...->domain.pipe`) and no installed method matches, fetch the package by address and install it into `~/.mthds/methods/` so the load can proceed. When disabled, such a miss refuses the bundle with an `unresolved_package_dependency` item naming the address and how to ship the package with the bundle or install it, and the network is never touched at load time.
## Example Configuration
diff --git a/pipelex/cli/installed_methods.py b/pipelex/cli/installed_methods.py
index 09197f47b..f7818b785 100644
--- a/pipelex/cli/installed_methods.py
+++ b/pipelex/cli/installed_methods.py
@@ -63,11 +63,11 @@ def _read_provenance(*, method_dir: Path) -> MethodProvenance | None:
provenance_path = method_dir / PROVENANCE_FILENAME
if not provenance_path.is_file():
return None
- content = provenance_path.read_text(encoding="utf-8")
try:
- return MethodProvenance.model_validate_json(content)
+ # Validated from the bytes, so pydantic reports a sidecar that is not UTF-8 as invalid JSON too.
+ return MethodProvenance.model_validate_json(provenance_path.read_bytes())
except ValidationError:
- # A hand-edited or stale sidecar must not break discovery — the method is still usable,
+ # A hand-edited, stale or corrupt sidecar must not break discovery — the method is still usable,
# it just has no recorded provenance.
return None
diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.py
index 6e3dbd601..3d0c2daab 100644
--- a/pipelex/libraries/library_manager.py
+++ b/pipelex/libraries/library_manager.py
@@ -24,7 +24,7 @@ from pipelex.core.concepts.helpers import make_qualified_structure_class_name
from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.domains.domain_blueprint import DomainBlueprint
from pipelex.core.domains.domain_factory import DomainFactory
-from pipelex.core.exceptions import PipesAndConceptValidationErrorData
+from pipelex.core.exceptions import PipelexBundleBlueprintValidationErrorData, PipesAndConceptValidationErrorData
from pipelex.core.pipes.exceptions import PipeLoadRefusalError, PipeOperatorModelChoiceError
from pipelex.core.qualified_ref import QualifiedRef
from pipelex.core.stuffs.structured_content import StructuredContent
@@ -49,7 +49,7 @@ from pipelex.libraries.library_utils import (
from pipelex.libraries.pipe.exceptions import PipeLibraryError
from pipelex.libraries.visibility_utils import check_visibility_for_blueprints, make_visibility_checker
from pipelex.methods.exceptions import MethodDependencyFetchError, MethodFetchDisabledError
-from pipelex.methods.fetch_on_miss import resolve_address_based_method
+from pipelex.methods.fetch_on_miss import find_vendored_method, resolve_address_based_method
from pipelex.methods.structures_check import ensure_no_structured_content_in_library_sources, is_generated_structures_module
from pipelex.mthds_parsing.exceptions import MthdsParserError
from pipelex.mthds_parsing.handle_pipe_errors import categorize_pipe_validation_error
@@ -128,6 +128,44 @@ def _dependency_bundle_source(*, package_address: str, package_root: Path, mthds
return f"{package_address}/{mthds_path.name}"
+def _parse_dependency_bundle(*, mthds_path: Path, bundle_source: str, package_address: str) -> PipelexBundleBlueprint:
+ """Parse one bundle of a dependency package under its caller-facing source, refusing the load when it does not parse.
+
+ The bundle is parsed from its text under ``bundle_source`` rather than from its path, so the blueprint, every item
+ a parse failure gives and every message the parser writes name the bundle by the package's address and its path
+ inside the package: parsing from the path would put the host's path into a TOML error's message and into the
+ blueprint's own messages. A file discovery has just listed that cannot be read is the host's fault and propagates.
+
+ Args:
+ mthds_path: The bundle's file on this host.
+ bundle_source: The bundle's caller-facing name, ``<package_address>/<path inside the package>``.
+ package_address: The package's address, which the refusal's summary names.
+
+ Returns:
+ The parsed blueprint, its ``source`` the caller-facing name.
+
+ Raises:
+ MthdsParserError: The bundle is not UTF-8 or does not parse; each item carries ``bundle_source`` as its source,
+ and the summary names the bundle and the package.
+ """
+ try:
+ mthds_content = mthds_path.read_text(encoding="utf-8")
+ except UnicodeDecodeError as exc:
+ msg = f"Bundle '{bundle_source}' of method package '{package_address}' is not valid UTF-8."
+ raise MthdsParserError(msg, validation_errors=[PipelexBundleBlueprintValidationErrorData(source=bundle_source, message=msg)]) from exc
+ try:
+ return MthdsParser.make_pipelex_bundle_blueprint(mthds_content=mthds_content, mthds_source=bundle_source)
+ except MthdsParserError as exc:
+ # Each item names the bundle in its text too: the verdict's summary is an item's message, and a caller reading
+ # it must learn that the fault is in a package the bundle calls, not in the bundle itself.
+ context = f"Bundle '{bundle_source}' of method package '{package_address}' does not parse"
+ items = [item.model_copy(update={"source": bundle_source, "message": f"{context}: {item.message}"}) for item in exc.validation_errors]
+ msg = f"{context}: {exc.message}"
+ raise MthdsParserError(
+ msg, validation_errors=items or [PipelexBundleBlueprintValidationErrorData(source=bundle_source, message=msg)]
+ ) from exc
+
+
def _public_dependency_pipe_refs(*, resolved_dep: ResolvedDependency, blueprints: list[PipelexBundleBlueprint]) -> set[str] | None:
"""The `domain.code` of every pipe a dependency makes public, or `None` when it declares no exports and all are public.
@@ -780,7 +818,9 @@ class LibraryManager(LibraryManagerAbstract):
return all_pipes
@override
- def load_from_blueprints(self, *, library_id: str, blueprints: list[PipelexBundleBlueprint]) -> list[PipeAbstract]:
+ def load_from_blueprints(
+ self, *, library_id: str, blueprints: list[PipelexBundleBlueprint], methods_dirs: list[Path] | None = None
+ ) -> list[PipeAbstract]:
"""Load domains, concepts, and pipes from a list of blueprints.
Delegates through LibraryCrate: builds a crate from blueprints, then loads from the crate.
@@ -789,6 +829,9 @@ class LibraryManager(LibraryManagerAbstract):
Args:
library_id: The ID of the library to load into
blueprints: List of parsed MTHDS blueprints to load
+ methods_dirs: Directories laid out like ``.mthds/methods/`` holding packages for this load alone, such as
+ the ones a request ships with its bundle. An address-based reference is looked up there first, and a
+ package found there is never installed anywhere.
Returns:
List of all pipes that were loaded
@@ -805,6 +848,7 @@ class LibraryManager(LibraryManagerAbstract):
self._load_address_based_dependencies(
library_id=library_id,
blueprints=blueprints,
+ methods_dirs=methods_dirs,
)
# Build the crate (merges, qualifies, detects duplicates)
@@ -1238,10 +1282,17 @@ class LibraryManager(LibraryManagerAbstract):
path on the host (see ``_dependency_bundle_source``), so a refusal inside the dependency names it
that way on every surface.
+ A bundle of the dependency that does not parse refuses the load (``_parse_dependency_bundle``): a
+ package shipped with a request is the caller's own content, and a fetched package's author is owed
+ the same report rather than a later refusal saying the package lacks a pipe.
+
Args:
library: The main library to load into
resolved_dep: The resolved dependency info
package_address: The dependency's address as a reference names it, without any ``@<tag>``
+
+ Raises:
+ MthdsParserError: A bundle of the dependency does not parse, its items named by the bundle's source.
"""
alias = resolved_dep.alias
@@ -1250,13 +1301,8 @@ class LibraryManager(LibraryManagerAbstract):
# Where each bundle is on this host, by its caller-facing source, for the pipes that read a file beside it.
bundle_files_by_source: dict[str, Path] = {}
for mthds_path in resolved_dep.mthds_files:
- try:
- blueprint = MthdsParser.make_pipelex_bundle_blueprint(bundle_path=mthds_path)
- except (FileNotFoundError, MthdsParserError) as exc:
- log.warning(f"Could not parse dependency '{alias}' bundle '{mthds_path}': {exc}")
- continue
bundle_source = _dependency_bundle_source(package_address=package_address, package_root=resolved_dep.package_root, mthds_path=mthds_path)
- blueprint.source = bundle_source
+ blueprint = _parse_dependency_bundle(mthds_path=mthds_path, bundle_source=bundle_source, package_address=package_address)
bundle_files_by_source[bundle_source] = mthds_path
dep_blueprints.append(blueprint)
@@ -1409,15 +1455,17 @@ class LibraryManager(LibraryManagerAbstract):
*,
library_id: str,
blueprints: list[PipelexBundleBlueprint],
+ methods_dirs: list[Path] | None = None,
) -> None:
"""Scan blueprints for cross-package pipe refs with address-based aliases and load them.
Collects all cross-package pipe references from controller blueprints
(sequences, batches, conditions, parallels), identifies those whose
alias contains '/' (i.e. a full package address), and loads each
- unique address-based dependency. A dependency missing from the
- installed methods is fetched by address (honoring an ``@<tag>`` pin)
- and installed when fetch-on-miss is enabled.
+ unique address-based dependency. A dependency is looked up first among
+ the packages of ``methods_dirs``, the load's own, then among the
+ installed methods; one missing from both is fetched by address
+ (honoring an ``@<tag>`` pin) and installed when fetch-on-miss is enabled.
A package that cannot be resolved is the caller's to fix, since the reference is part of their bundle: every
address is tried, then the load is refused with one ``unresolved_package_dependency`` item per address that
@@ -1431,6 +1479,7 @@ class LibraryManager(LibraryManagerAbstract):
Args:
library_id: The library to load into
blueprints: The parsed bundle blueprints to scan
+ methods_dirs: The load's own methods directories, searched before everything else and never written
Raises:
LibraryLoadingError: One or more referenced packages could not be resolved.
@@ -1461,6 +1510,7 @@ class LibraryManager(LibraryManagerAbstract):
library=library,
full_address=full_address,
extra_search_dirs=extra_search_dirs,
+ methods_dirs=methods_dirs,
)
except (MethodFetchDisabledError, MethodDependencyFetchError) as exc:
# The item carries the caller-facing text; the host's log keeps the whole cause chain, git's raw output
@@ -1494,25 +1544,29 @@ class LibraryManager(LibraryManagerAbstract):
*,
full_address: str,
extra_search_dirs: list[Path] | None = None,
+ methods_dirs: list[Path] | None = None,
) -> None:
"""Load a method package on demand using its full address, fetching it on a miss.
- Resolves the address to an installed method — fetching and installing the package
- (honoring an ``@<tag>`` pin) when no installed method matches and fetch-on-miss is
- enabled — builds a ResolvedDependency, and delegates to _load_single_dependency()
- to load it as a child library with the full address as alias.
+ Looks the address up among the packages of ``methods_dirs`` first, which win over an installed copy of the same
+ address since they are the ones the caller chose, and on a miss resolves it to an installed method, fetching
+ and installing the package (honoring an ``@<tag>`` pin) when no installed method matches and fetch-on-miss is
+ enabled. It then builds a ResolvedDependency and delegates to _load_single_dependency() to load it as a child
+ library with the full address as alias.
Args:
library: The main library to load into
full_address: The full package address (e.g. "github.com/Pipelex/methods/documents")
extra_search_dirs: Additional .mthds/methods/ directories to scan
+ methods_dirs: The load's own methods directories, searched before everything else and never written
Raises:
- MethodRefError: The address resolves to no installed method and could not be
+ MethodRefError: The address resolves to no vendored or installed method and could not be
fetched — fetch-on-miss disabled, an unfetchable address, a failed fetch,
or a failed install. Never a silent pass.
"""
- installed = resolve_address_based_method(full_address=full_address, extra_search_dirs=extra_search_dirs)
+ vendored = find_vendored_method(full_address=full_address, methods_dirs=methods_dirs) if methods_dirs else None
+ installed = vendored or resolve_address_based_method(full_address=full_address, extra_search_dirs=extra_search_dirs)
exported_pipe_codes = determine_exported_pipes(manifest=installed.manifest)
diff --git a/pipelex/libraries/library_manager_abstract.py b/pipelex/libraries/library_manager_abstract.py
index b83e16bed..7de66929a 100644
--- a/pipelex/libraries/library_manager_abstract.py
+++ b/pipelex/libraries/library_manager_abstract.py
@@ -133,8 +133,17 @@ class LibraryManagerAbstract(ABC):
"""
@abstractmethod
- def load_from_blueprints(self, *, library_id: str, blueprints: list[PipelexBundleBlueprint]) -> list[PipeAbstract]:
- pass
+ def load_from_blueprints(
+ self, *, library_id: str, blueprints: list[PipelexBundleBlueprint], methods_dirs: list[Path] | None = None
+ ) -> list[PipeAbstract]:
+ """Load the blueprints into the library, resolving their address-based dependencies.
+
+ Args:
+ library_id: The library to load into
+ blueprints: The parsed bundles to load
+ methods_dirs: Directories laid out like ``.mthds/methods/`` holding packages for this load alone, searched
+ for an address-based dependency before the installed stores and fetch-on-miss, and never written
+ """
@abstractmethod
def _remove_from_blueprint(self, *, library_id: str, blueprint: PipelexBundleBlueprint) -> None:
diff --git a/pipelex/methods/fetch_on_miss.py b/pipelex/methods/fetch_on_miss.py
index 80af720ee..ac42fbcbd 100644
--- a/pipelex/methods/fetch_on_miss.py
+++ b/pipelex/methods/fetch_on_miss.py
@@ -7,14 +7,19 @@ fetch), installs it into the installed-methods store (``~/.mthds/methods/``) wit
provenance recorded, and hands it back so library loading can proceed. A miss that cannot be
bridged — fetch disabled, an unfetchable address, a failed fetch — raises a diagnostic that
names the address and the remedy; it is never a silent pass.
+
+A load can also be handed methods directories of its own, holding the packages a request ships
+with its bundle under `.mthds/methods/<name>/`. They are looked up first, by the same manifest
+identity, and only searched, never written (:func:`find_vendored_method`).
"""
import shutil
import tempfile
from pathlib import Path
+from typing import NamedTuple
from pipelex import log
-from pipelex.cli.installed_methods import InstalledMethod, find_method_by_full_address, install_method_package
+from pipelex.cli.installed_methods import InstalledMethod, discover_installed_methods, find_method_by_full_address, install_method_package
from pipelex.config import METHODS_FETCH_ON_MISS_ENV_VAR, is_method_fetch_on_miss_enabled, is_pipe_func_sandbox_hosted
from pipelex.methods.exceptions import (
MethodDependencyFetchError,
@@ -35,6 +40,58 @@ from pipelex.methods.structures_check import (
# The remedy every refusal ends with. It names no directory: on a host, the runtime's own store is not the caller's to write.
MANUAL_INSTALL_HINT = "install the package where this runtime runs (for example with `mthds install <address>`)"
+# The remedy a caller can always apply, a hosted one included: the bundle carries the package, which the runtime then
+# finds before its own store, beside a bundle on disk as in a bundle sent over HTTP.
+VENDORING_HINT = "ship the package with the bundle under `.mthds/methods/<name>/`"
+
+
+class _LookupAddress(NamedTuple):
+ """The address a reference is looked up by, with the parsed reference or the reason it did not parse."""
+
+ address: str
+ ref: MethodRef | None
+ parse_error: MethodRefParseError | None
+
+
+def _lookup_address(*, full_address: str) -> _LookupAddress:
+ """Strip any ``@<tag>`` from a reference for the lookup: an installed or vendored copy is keyed by its address alone."""
+ if not looks_like_method_ref(full_address):
+ return _LookupAddress(address=full_address, ref=None, parse_error=None)
+ try:
+ ref = parse_method_ref(full_address)
+ except MethodRefParseError as exc:
+ return _LookupAddress(address=full_address, ref=None, parse_error=exc)
+ return _LookupAddress(address=ref.address, ref=ref, parse_error=None)
+
+
+def find_vendored_method(*, full_address: str, methods_dirs: list[Path]) -> InstalledMethod | None:
+ """Look an address-based reference up among the packages a load was handed, and nowhere else.
+
+ ``methods_dirs`` are laid out like ``.mthds/methods/`` (one directory per package, its ``METHODS.toml`` at its
+ root) and hold the packages a request ships with its bundle. A package matches by the manifest identity the
+ installed stores are matched by, any ``@<tag>`` stripped, so a vendored copy answers an address this runtime could
+ not fetch too. The installed stores are not read, and nothing is fetched or installed. A ``@<tag>`` pin answered by
+ a vendored copy uses the copy, as an installed copy is used, with a warning unless the copy's version is that tag.
+
+ Args:
+ full_address: The address-based alias as written in the bundle, with any ``@<tag>``.
+ methods_dirs: The load's own methods directories.
+
+ Returns:
+ The vendored package the reference names, or ``None`` when none of the directories holds it.
+ """
+ lookup = _lookup_address(full_address=full_address)
+ vendored_methods = discover_installed_methods(include_global=False, include_project=False, extra_search_dirs=methods_dirs)
+ vendored = find_method_by_full_address(lookup.address, methods=vendored_methods)
+ if vendored is not None and lookup.ref is not None and lookup.ref.tag is not None:
+ version = vendored.manifest.version
+ if lookup.ref.tag not in {version, f"v{version}"}:
+ log.warning(
+ f"Method '{lookup.ref.address}' is shipped with the request at version {version} while the reference pins "
+ f"'@{lookup.ref.tag}'; using the shipped copy."
+ )
+ return vendored
+
def _warn_on_tag_mismatch(*, installed: InstalledMethod, ref: MethodRef | None) -> None:
"""Warn when a tag-pinned reference resolves to an installed copy that is not that tag."""
@@ -86,17 +143,11 @@ def resolve_address_based_method(
the install target being occupied by a different package that shares the bare
directory name (never silently loaded, never silently overwritten).
"""
- ref: MethodRef | None = None
- parse_error: MethodRefParseError | None = None
- lookup_address = full_address
- if looks_like_method_ref(full_address):
- try:
- ref = parse_method_ref(full_address)
- lookup_address = ref.address
- except MethodRefParseError as exc:
- parse_error = exc
+ lookup = _lookup_address(full_address=full_address)
+ ref = lookup.ref
+ parse_error = lookup.parse_error
- installed = find_method_by_full_address(lookup_address, extra_search_dirs=extra_search_dirs)
+ installed = find_method_by_full_address(lookup.address, extra_search_dirs=extra_search_dirs)
if installed is not None:
_warn_on_tag_mismatch(installed=installed, ref=ref)
return installed
@@ -110,13 +161,13 @@ def resolve_address_based_method(
if ref is None:
msg = (
f"Method '{full_address}' is referenced but not installed, and this runtime cannot fetch it: only github.com/... addresses "
- f"are fetchable. Correct the address, or {MANUAL_INSTALL_HINT}."
+ f"are fetchable. Correct the address, {VENDORING_HINT}, or {MANUAL_INSTALL_HINT}."
)
raise MethodDependencyFetchError(msg)
if not is_method_fetch_on_miss_enabled():
msg = (
f"Method '{ref.ref_str}' is referenced but not installed, and fetch-on-miss is disabled on this runtime, so it does not "
- f"fetch it. Correct the address if it is wrong, or {MANUAL_INSTALL_HINT}. Fetch-on-miss is enabled by "
+ f"fetch it. Correct the address if it is wrong, {VENDORING_HINT}, or {MANUAL_INSTALL_HINT}. Fetch-on-miss is enabled by "
f"`fetch_on_miss = true` under [interpreter.methods] in the runtime's pipelex.toml, or {METHODS_FETCH_ON_MISS_ENV_VAR}=1."
)
raise MethodFetchDisabledError(msg)
@@ -133,7 +184,7 @@ def resolve_address_based_method(
except MethodRefError as exc:
msg = (
f"Method '{ref.ref_str}' is referenced but not installed, and fetching it failed: {exc} "
- f"Correct the address or the tag, or {MANUAL_INSTALL_HINT}."
+ f"Correct the address or the tag, {VENDORING_HINT}, or {MANUAL_INSTALL_HINT}."
)
raise MethodDependencyFetchError(msg) from exc
diff --git a/pipelex/pipeline/execution_seams.py b/pipelex/pipeline/execution_seams.py
index 1dc7f9a9f..25650fc26 100644
--- a/pipelex/pipeline/execution_seams.py
+++ b/pipelex/pipeline/execution_seams.py
@@ -68,6 +68,7 @@ def acquire_library(
mthds_contents: list[str] | None = None,
bundle_uris: list[str] | None = None,
library_dirs_are_callers: bool = False,
+ methods_dirs: list[Path] | None = None,
) -> tuple[str, str | None]:
"""Set the current library, open it, and load dirs + blueprints into it.
@@ -96,6 +97,11 @@ def acquire_library(
the host's (installed libraries, a temporary directory of shipped Python): a fault there is not
the caller's to fix and its paths are not the caller's to read, so by default they load untranslated.
+ ``methods_dirs`` are directories laid out like ``.mthds/methods/`` holding the packages the caller ships with the
+ ``mthds_contents`` (on a host, the ones a request carries under ``.mthds/methods/<name>/``). An address-based
+ reference of the contents is looked up there first, before the installed stores and fetch-on-miss, and a package
+ found there is never installed; the bundles of a package are named by its address, never by these directories.
+
Returns the ``library_id`` and the bundle's qualified ``main_pipe`` ref
(``domain.pipe_code``) when an ``mthds_contents`` bundle declares one, else
``None``. Pipe resolution is the caller's responsibility.
@@ -148,7 +154,7 @@ def acquire_library(
blueprints_to_load.append(blueprint)
if blueprints_to_load:
- library_manager.load_from_blueprints(library_id=library_id, blueprints=blueprints_to_load)
+ library_manager.load_from_blueprints(library_id=library_id, blueprints=blueprints_to_load, methods_dirs=methods_dirs)
# Qualify main_pipe with domain to avoid ambiguity when multiple domains define pipes with
# the same code — via the one shared selection rule (first declaring main_pipe, else first).
diff --git a/pipelex/pipeline/pipeline_run_setup.py b/pipelex/pipeline/pipeline_run_setup.py
index ec4e96179..5d6bfd52a 100644
--- a/pipelex/pipeline/pipeline_run_setup.py
+++ b/pipelex/pipeline/pipeline_run_setup.py
@@ -69,6 +69,7 @@ async def pipeline_run_setup(
request_id: str | None = None,
inputs_base_dir: Path | None = None,
library_dirs_are_callers: bool = False,
+ methods_dirs: list[Path] | None = None,
) -> tuple[PipeJob, str, str]:
"""Set up a pipeline for execution.
@@ -169,6 +170,11 @@ async def pipeline_run_setup(
loading them is the caller's invalid bundle. ``False`` (a host's own directories) loads them
untranslated. The ``mthds_contents`` are always the caller's: a refusal while loading them is
always the ``ValidateBundleError`` verdict. See :func:`acquire_library`.
+ methods_dirs:
+ Directories laid out like ``.mthds/methods/`` holding the method packages the caller ships with
+ ``mthds_contents``, for this run alone. An address-based reference is looked up there before the
+ installed stores and fetch-on-miss, and a package found there is never installed. See
+ :func:`acquire_library`.
Returns:
-------
@@ -298,6 +304,7 @@ async def pipeline_run_setup(
mthds_contents=mthds_contents,
bundle_uris=bundle_uris,
library_dirs_are_callers=library_dirs_are_callers,
+ methods_dirs=methods_dirs,
)
library_acquired = True
diff --git a/pipelex/pipeline/runner.py b/pipelex/pipeline/runner.py
index 7d772d6c6..0ecffd6ef 100644
--- a/pipelex/pipeline/runner.py
+++ b/pipelex/pipeline/runner.py
@@ -192,10 +192,15 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):
# them is the caller's invalid bundle; a host's own otherwise, loaded untranslated. See
# `acquire_library`.
library_dirs_are_callers: bool = False,
+ # Directories laid out like `.mthds/methods/` holding the method packages a caller ships with the contents it
+ # runs, searched first for an address-based reference and never written. A run reads them; `validate` does
+ # not, and resolves such a reference through the installed stores and fetch-on-miss only.
+ methods_dirs: list[Path] | None = None,
):
self.library_id = library_id
self.library_dirs = library_dirs
self.library_dirs_are_callers = library_dirs_are_callers
+ self.methods_dirs = methods_dirs
self.bundle_uris = bundle_uris
self.pipe_run_mode = pipe_run_mode
self.is_mock_usage = is_mock_usage
@@ -331,6 +336,7 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):
extras=self.extras,
inputs_base_dir=self.inputs_base_dir,
library_dirs_are_callers=self.library_dirs_are_callers,
+ methods_dirs=self.methods_dirs,
request_id=request_id,
)
effective_pipe_run = self._pipe_run or get_pipe_run()
diff --git a/tests/integration/pipelex/libraries/installed_packages.py b/tests/integration/pipelex/libraries/installed_packages.py
index f24c6a353..21dce5b54 100644
--- a/tests/integration/pipelex/libraries/installed_packages.py
+++ b/tests/integration/pipelex/libraries/installed_packages.py
@@ -35,6 +35,25 @@ def install_probe(
install_package(root=root, method_name=ProbePackageTestData.METHOD_NAME, manifest=manifest, bundles={"probe_dep.mthds": bundle})
+def vendor_probe(
+ *,
+ methods_dir: Path,
+ manifest: str = ProbePackageTestData.MANIFEST_EXPORTING_EVERYTHING,
+ bundle: str = ProbePackageTestData.DEP_BUNDLE,
+) -> Path:
+ """Write the probe package into a methods directory, laid out as `<methods_dir>/probe/…`, and return the directory."""
+ package_dir = methods_dir / ProbePackageTestData.METHOD_NAME
+ package_dir.mkdir(parents=True)
+ (package_dir / "METHODS.toml").write_text(manifest, encoding="utf-8")
+ (package_dir / "probe_dep.mthds").write_text(bundle, encoding="utf-8")
+ return methods_dir
+
+
+def install_probe_globally(*, root: Path, bundle: str = ProbePackageTestData.DEP_BUNDLE) -> None:
+ """Install the probe package into the global store `isolate_installed_methods` points at under `root`."""
+ vendor_probe(methods_dir=root / "global-methods", bundle=bundle)
+
+
def write_consumer(*, root: Path, bundles: dict[str, str]) -> list[Path]:
paths: list[Path] = []
for file_name, bundle in bundles.items():
diff --git a/tests/integration/pipelex/libraries/test_unparsable_dependency_bundle.py b/tests/integration/pipelex/libraries/test_unparsable_dependency_bundle.py
new file mode 100644
index 000000000..12de3c0f0
--- /dev/null
+++ b/tests/integration/pipelex/libraries/test_unparsable_dependency_bundle.py
@@ -0,0 +1,69 @@
+"""A bundle of a dependency package that does not parse refuses the load, named by the package's address.
+
+The bundles of a package the consumer calls by address load with it, and one that does not parse is a fault the
+caller can see and report: a shipped package is the caller's own content, and a fetched or installed package's author
+needs to learn the package is broken, not that it lacks the pipe called. The refusal is the parser's verdict, each
+item's `source` the package's address and the bundle's path inside it, and nothing in it names a path on the host.
+
+The packages are invented; every test isolates the installed store, and none reaches the network.
+"""
+
+import json
+from pathlib import Path
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.base_exceptions import DisclosureMode
+from pipelex.pipeline.exceptions import ValidateBundleError
+from pipelex.pipeline.execution_seams import acquire_library
+from pipelex.pipeline.validate_bundle import validate_bundle
+from tests.integration.pipelex.libraries.installed_packages import install_probe_globally, isolate_installed_methods, vendor_probe
+from tests.integration.pipelex.libraries.test_data import ProbePackageTestData
+
+PROBE_ALIAS = ProbePackageTestData.DEP_ALIAS
+
+# The probe package's bundle, made unparsable as TOML and as a blueprint.
+UNPARSABLE_BUNDLES: dict[str, str] = {
+ "toml syntax": 'domain = "probe_dep"\n\n[pipe.entry\ntype = "PipeLLM"\n',
+ "invalid blueprint": ProbePackageTestData.DEP_BUNDLE.replace(
+ 'domain = "probe_dep"', 'domain = "probe_dep"\nmain_pipe = "Not A Pipe Code!"'
+ ),
+}
+
+
+def _assert_names_no_host_path(*, verdict: ValidateBundleError, tmp_path: Path) -> None:
+ strict_dump = json.dumps(verdict.to_error_report().to_dict(disclosure_mode=DisclosureMode.STRICT))
+ for host_path in (str(tmp_path), str(tmp_path.resolve())):
+ assert host_path not in strict_dump
+
+
+class TestUnparsableDependencyBundle:
+ @pytest.mark.parametrize("broken", list(UNPARSABLE_BUNDLES))
+ def test_a_shipped_bundle_that_does_not_parse_refuses_the_load(self, tmp_path: Path, mocker: MockerFixture, broken: str) -> None:
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ methods_dir = vendor_probe(methods_dir=tmp_path / "request-methods", bundle=UNPARSABLE_BUNDLES[broken])
+
+ with pytest.raises(ValidateBundleError) as exc_info:
+ acquire_library(library_id="", mthds_contents=[ProbePackageTestData.CONSUMER_BUNDLE], methods_dirs=[methods_dir])
+
+ verdict = exc_info.value
+ items = verdict.to_error_report().validation_errors or []
+ assert items, verdict.message
+ assert {item.source for item in items} == {f"{PROBE_ALIAS}/probe_dep.mthds"}
+ assert PROBE_ALIAS in verdict.message
+ _assert_names_no_host_path(verdict=verdict, tmp_path=tmp_path)
+
+ @pytest.mark.asyncio(loop_scope="class")
+ async def test_an_installed_bundle_that_does_not_parse_refuses_the_load(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """A fetched or installed package's author gets the same report: the parse failure, never a pipe the package lacks."""
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_probe_globally(root=tmp_path, bundle=UNPARSABLE_BUNDLES["toml syntax"])
+
+ with pytest.raises(ValidateBundleError) as exc_info:
+ await validate_bundle(mthds_contents=[ProbePackageTestData.CONSUMER_BUNDLE])
+
+ verdict = exc_info.value
+ items = verdict.to_error_report().validation_errors or []
+ assert [(item.source, item.line) for item in items] == [(f"{PROBE_ALIAS}/probe_dep.mthds", 3)]
+ _assert_names_no_host_path(verdict=verdict, tmp_path=tmp_path)
diff --git a/tests/integration/pipelex/libraries/test_vendored_methods_dirs.py b/tests/integration/pipelex/libraries/test_vendored_methods_dirs.py
new file mode 100644
index 000000000..77655c53b
--- /dev/null
+++ b/tests/integration/pipelex/libraries/test_vendored_methods_dirs.py
@@ -0,0 +1,117 @@
+"""A load's own methods directories answer a reference by address first, and are never written.
+
+A host that receives a bundle with the packages it calls hands the engine their directory, laid out like
+`.mthds/methods/`, as `methods_dirs`. A reference is looked up there by manifest identity before the installed stores
+and fetch-on-miss, so the shipped copy answers even with fetching off, wins over an installed copy of the same
+address, and is never installed anywhere.
+
+The packages are invented; every test isolates the installed store, and none reaches the network.
+"""
+
+import logging
+from pathlib import Path
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.config import get_config
+from pipelex.interpreter_hub import get_library_manager
+from pipelex.libraries.library import Library
+from pipelex.mthds_parsing.parser import MthdsParser
+from pipelex.pipeline.runner import PipelexMTHDSProtocol
+from pipelex.system.pipe_run_mode import PipeRunMode
+from tests.integration.pipelex.libraries.installed_packages import install_probe_globally, isolate_installed_methods, vendor_probe
+from tests.integration.pipelex.libraries.test_data import ProbePackageTestData
+
+PROBE_ALIAS = ProbePackageTestData.DEP_ALIAS
+VENDORED_HELPER = "DEPENDENCY helper"
+INSTALLED_HELPER = "INSTALLED helper"
+INSTALLED_BUNDLE = ProbePackageTestData.DEP_BUNDLE.replace(VENDORED_HELPER, INSTALLED_HELPER)
+
+
+def _load_consumer(*, consumer: str, methods_dirs: list[Path] | None) -> tuple[str, Library]:
+ """Open a library and load the consumer into it; the caller tears it down."""
+ library_manager = get_library_manager()
+ library_id, library = library_manager.open_library()
+ blueprint = MthdsParser.make_pipelex_bundle_blueprint(mthds_content=consumer)
+ library_manager.load_from_blueprints(library_id=library_id, blueprints=[blueprint], methods_dirs=methods_dirs)
+ return library_id, library
+
+
+def _helper_description(*, library: Library, alias: str = PROBE_ALIAS) -> str | None:
+ return library.dependency_libraries[alias].pipe_library.get_required_pipe(pipe_code="probe_dep.helper").description
+
+
+class TestVendoredMethodsDirs:
+ def test_a_shipped_package_answers_the_reference_with_fetching_off(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ resolve_mock = mocker.patch("pipelex.libraries.library_manager.resolve_address_based_method")
+ methods_dir = vendor_probe(methods_dir=tmp_path / "request-methods")
+
+ library_id, library = _load_consumer(consumer=ProbePackageTestData.CONSUMER_BUNDLE, methods_dirs=[methods_dir])
+ try:
+ assert _helper_description(library=library) == VENDORED_HELPER
+ finally:
+ get_library_manager().teardown(library_id=library_id)
+ # A hit goes neither to the installed stores nor to fetch-on-miss, and nothing is installed.
+ resolve_mock.assert_not_called()
+ assert not (tmp_path / "global-methods").exists()
+
+ def test_the_shipped_copy_wins_over_an_installed_one(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_probe_globally(root=tmp_path, bundle=INSTALLED_BUNDLE)
+ methods_dir = vendor_probe(methods_dir=tmp_path / "request-methods")
+
+ library_id, library = _load_consumer(consumer=ProbePackageTestData.CONSUMER_BUNDLE, methods_dirs=[methods_dir])
+ try:
+ assert _helper_description(library=library) == VENDORED_HELPER
+ finally:
+ get_library_manager().teardown(library_id=library_id)
+
+ def test_a_miss_in_the_shipped_packages_goes_on_to_the_installed_store(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_probe_globally(root=tmp_path, bundle=INSTALLED_BUNDLE)
+ empty_methods_dir = tmp_path / "request-methods"
+ empty_methods_dir.mkdir()
+
+ library_id, library = _load_consumer(consumer=ProbePackageTestData.CONSUMER_BUNDLE, methods_dirs=[empty_methods_dir])
+ try:
+ assert _helper_description(library=library) == INSTALLED_HELPER
+ finally:
+ get_library_manager().teardown(library_id=library_id)
+
+ def test_a_tag_pin_answered_by_a_shipped_copy_uses_it_and_says_so(
+ self, tmp_path: Path, mocker: MockerFixture, caplog: pytest.LogCaptureFixture
+ ) -> None:
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ methods_dir = vendor_probe(methods_dir=tmp_path / "request-methods")
+ pinned_alias = f"{PROBE_ALIAS}@v9.9.9"
+
+ with caplog.at_level(logging.WARNING):
+ library_id, library = _load_consumer(
+ consumer=ProbePackageTestData.consumer_calling(pipe_ref=f"{pinned_alias}->probe_dep.entry"), methods_dirs=[methods_dir]
+ )
+ try:
+ assert _helper_description(library=library, alias=pinned_alias) == VENDORED_HELPER
+ finally:
+ get_library_manager().teardown(library_id=library_id)
+ warnings = [record.message for record in caplog.records if record.levelno == logging.WARNING and "@v9.9.9" in record.message]
+ assert len(warnings) == 1, warnings
+ assert PROBE_ALIAS in warnings[0]
+ assert "1.0.0" in warnings[0]
+
+ @pytest.mark.asyncio(loop_scope="class")
+ async def test_the_runner_protocol_runs_the_shipped_dependency(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ isolate_installed_methods(mocker=mocker, root=tmp_path)
+ install_probe_globally(root=tmp_path, bundle=INSTALLED_BUNDLE)
+ methods_dir = vendor_probe(methods_dir=tmp_path / "request-methods")
+ execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=True)
+ runner = PipelexMTHDSProtocol(pipe_run_mode=PipeRunMode.DRY, execution_config=execution_config, methods_dirs=[methods_dir])
+
+ response = await runner.execute(pipe_code="go", mthds_contents=[ProbePackageTestData.CONSUMER_BUNDLE], inputs={"data": "hello"})
+
+ graph_spec = response.pipe_output.graph_spec
+ assert graph_spec is not None
+ descriptions = {node.description for node in graph_spec.nodes}
+ assert VENDORED_HELPER in descriptions
+ assert INSTALLED_HELPER not in descriptions
diff --git a/tests/unit/pipelex/cli/test_install_method_package.py b/tests/unit/pipelex/cli/test_install_method_package.py
index 68e6e9902..ba66c6d87 100644
--- a/tests/unit/pipelex/cli/test_install_method_package.py
+++ b/tests/unit/pipelex/cli/test_install_method_package.py
@@ -83,6 +83,28 @@ class TestInstallMethodPackage:
assert installed.provenance is None
assert not (installed.path / PROVENANCE_FILENAME).exists()
+ @pytest.mark.parametrize(
+ "sidecar",
+ [
+ pytest.param(b"\xff\xfe garbage", id="not-utf8"),
+ pytest.param(b"{not json", id="not-json"),
+ pytest.param(b'{"address": "x"}', id="not-provenance"),
+ ],
+ )
+ def test_discovery_reads_an_unreadable_sidecar_as_no_provenance(self, tmp_path: Path, sidecar: bytes) -> None:
+ """A sidecar discovery cannot read, whatever the reason, leaves the method usable with no recorded provenance."""
+ package_dir = _make_package_dir(tmp_path)
+ methods_dir = tmp_path / "methods"
+ installed = install_method_package(package_dir=package_dir, name="scoring", provenance=PROVENANCE, methods_dir=methods_dir)
+ (installed.path / PROVENANCE_FILENAME).write_bytes(sidecar)
+
+ methods = discover_installed_methods(include_global=False, include_project=False, extra_search_dirs=[methods_dir])
+ found = find_method_by_full_address(FULL_ADDRESS, methods=methods)
+
+ assert found is not None
+ assert found.path == installed.path
+ assert found.provenance is None
+
def test_install_refuses_a_target_occupied_by_a_non_package(self, tmp_path: Path) -> None:
"""An existing target directory that is not a method package is never overwritten."""
package_dir = _make_package_dir(tmp_path)
diff --git a/tests/unit/pipelex/methods/test_fetch_on_miss.py b/tests/unit/pipelex/methods/test_fetch_on_miss.py
index db221a5f4..dafd44d29 100644
--- a/tests/unit/pipelex/methods/test_fetch_on_miss.py
+++ b/tests/unit/pipelex/methods/test_fetch_on_miss.py
@@ -116,6 +116,7 @@ class TestResolveAddressBasedMethod:
assert "PIPELEX_METHODS_FETCH_ON_MISS" in message
assert "fetch_on_miss" in message
assert "install the package where this runtime runs" in message
+ assert "ship the package with the bundle under `.mthds/methods/<name>/`" in message
assert "~/.mthds" not in message
fetch_mock.assert_not_called()
@@ -134,13 +135,15 @@ class TestResolveAddressBasedMethod:
assert "fetching it failed" in message
assert "clone timed out" in message
assert "Correct the address or the tag" in message
+ assert "ship the package with the bundle under `.mthds/methods/<name>/`" in message
assert exc_info.value.__cause__ is fetch_error
@pytest.mark.usefixtures("isolated_methods_dirs")
def test_miss_with_unfetchable_address_raises_a_diagnostic(self) -> None:
"""An address-based alias that is not a fetchable reference gets a diagnostic, not a silent pass."""
- with pytest.raises(MethodDependencyFetchError, match="this runtime cannot fetch it"):
+ with pytest.raises(MethodDependencyFetchError, match="this runtime cannot fetch it") as exc_info:
resolve_address_based_method(full_address="example.com/foo/bar")
+ assert "ship the package with the bundle under `.mthds/methods/<name>/`" in str(exc_info.value)
def test_miss_fetches_installs_and_returns_the_method(self, isolated_methods_dirs: Path, tmp_path: Path, mocker: MockerFixture) -> None:
"""The full miss path: fetch, install with provenance, return — and the next resolve hits the install."""