Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feature/Api-routes-take-files · L-261016-fd5abd (#1447)
Louis Choquel745aca9d7ddd
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 73373a539..663884956 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,10 +2,24 @@## [Unreleased]+### Added++- **`resolve_crate_from_method_dir`**: `pipelex.pipeline.resolve_bundle` resolves a method's tree a host wrote out to a directory into the normalized crate, loading it as the caller's library directory the way `validate_bundle` loads a `method_dir`, with `methods_dirs` holding the packages shipped with it, under the same `ValidateBundleError` verdict and loaded-on-success contract as `resolve_crate_from_contents`. The resolution is static: a package the method calls by its address is loaded from `methods_dirs` alone, never from the methods installed on the host or by a fetch, and one they do not hold is an `unresolved_package_dependency` item of the verdict.+- **`address_dependency_sources`**: `LibraryManager.load_libraries` and `load_from_blueprints` take an `AddressDependencySources` (`pipelex.methods.address_dependencies`), `ANYWHERE` by default, which looks for a package called by its address among the load's `methods_dirs`, then the installed methods, then fetches it when fetch-on-miss is on, or `SHIPPED`, which looks only in `methods_dirs` and reports a package they do not hold as an `unresolved_package_dependency` item raised from the new `MethodDependencyNotShippedError`.+### Changed+- **`/v1/resolve`, `/v1/codegen` and `/v1/pipe-io` take the method tree (Breaking)**: the three routes take the method as `/v1/validate` and the run routes do, `files` being the MTHDS protocol's map of file entries with `dependencies`, `resources` and `read_scope` beside it, or a `method_ref` whose package's whole tree is loaded with the packages it ships, so a call into another package by its address resolves when that package was shipped, while one nothing shipped is the invalid verdict, never looked up among the runner's installed methods or fetched, and a `METHODS.toml` at the root of a sent tree names the default pipe of `/v1/pipe-io`, which reads it only for a request naming no pipe. A `files` list of `{content, source}` items is refused with a `422` `InvalidMethodFiles`, the tree's refusals and the custom-Python `403`s are those of `/v1/validate`, `include_files` echoes the method's `.mthds` bundles as a map from tree path to text, and the request models `MthdsFilesRequest`, `MthdsFileItem` and `MthdsPipeRequest` are removed. The routes load the method in a worker thread, so a slow load never holds up the server's other requests.- **A bundle validator receives the read scope, plugin API version 8 (Breaking)**: `BundleValidatorProtocol.validate_bundles` takes `read_scope`, the scope the host named for the validation, `None` when it named none, so a validator dispatching the validation to another process hands the method's trees to its worker through that scope's blob store and the worker reads them by digest under it, as a run's worker reads under the run's read scope. The API server's `POST /v1/validate` hands its validator the request's `read_scope` on every arm, never the caller's own id, and `PLUGIN_API_VERSION` moves to 8, so a plugin declaring `targets_api = 7` is refused at discovery with `PluginApiVersionMismatchError`.+### Fixed++- **Load errors name a written tree's bundles by their tree paths**: an error raised once the bundles of a tree the server wrote out have parsed, such as a concept declared in two bundles, names them by their tree paths instead of the directory they were written to, so a verdict of `/v1/validate`, a run or a crate route no longer discloses a path on the server.++### Security++- **A `jwt` caller's `read_scope` is bound to its own id (Breaking)**: on a deployment with `AUTH_MODE=jwt`, every route taking a `read_scope` (the run routes, `/v1/validate`, `/v1/resolve`, `/v1/codegen`, `/v1/pipe-io` and `POST /v1/blobs/missing`) refuses a scope that does not lie under the caller's token `user_id` with a `403 ReadScopeForbidden`, so a caller holding its own token can no longer read another caller's blobs or run under another's scope. With `AUTH_MODE=none` or `api_key`, the caller is a trusted host or a single tenant, and the scope is taken as before.+## [v0.88.0] - 2026-10-16### Addeddiff --git a/api/CLAUDE.md b/api/CLAUDE.mdindex 4f6f4d032..20c6f961c 100644--- a/api/CLAUDE.md+++ b/api/CLAUDE.md@@ -32,10 +32,11 @@ pipelex_api/pipeline.py # POST /v1/execute, /v1/start (MTHDS Protocol run routes)blobs.py # POST /v1/blobs/missing (MTHDS Protocol blob route; the signed upload targets)validate.py # POST /v1/validate- resolve.py # POST /v1/resolve (closure → normalized crate)- pipe_io.py # POST /v1/pipe-io (closure → one pipe's, or every pipe's, I/O artifacts; no dry run)+ method_selection.py # The method tree a diagnostic request names (`files` or `method_ref`), read as the run routes read it+ resolve.py # POST /v1/resolve (method tree → normalized crate)+ pipe_io.py # POST /v1/pipe-io (method tree → one pipe's, or every pipe's, I/O artifacts; no dry run)codegen.py # POST /v1/codegen (crate → typed artifacts)- crate_ops.py # Closure resolution, the pipe selection chain, and the invalid-arm envelope the crate routes share+ crate_ops.py # Writing and resolving the method's tree, the pipe selection chain, and the invalid-arm envelope the crate routes sharetools.py # POST /v1/lint, /v1/format (editor tooling)agent/ # GET /v1/models, GET /v1/models/check (the model reference check)tests/diff --git a/api/pipelex_api/error_types.py b/api/pipelex_api/error_types.pyindex 7149dc4e6..422dd7914 100644--- a/api/pipelex_api/error_types.py+++ b/api/pipelex_api/error_types.py@@ -48,6 +48,9 @@ class ErrorType(StrEnum):# A run request's `read_scope` extra is not one to three path-safe segments, or the run's storage# scope does not lie under it, so the run could not read back what it writes.INVALID_READ_SCOPE = "InvalidReadScope"+ # A caller authenticated by its own token (`AUTH_MODE=jwt`) sent a `read_scope` that does not lie under its own id.+ # A 403: the scope is well formed, but this caller may not read under it.+ READ_SCOPE_FORBIDDEN = "ReadScopeForbidden"# A run request's `analytics_groups` extra is not a mapping of group type to group key within# the runtime's charset and entry cap (`pipelex.system.run_extras`).INVALID_ANALYTICS_GROUPS = "InvalidAnalyticsGroups"diff --git a/api/pipelex_api/method_source.py b/api/pipelex_api/method_source.pyindex 9266bc725..60932249c 100644--- a/api/pipelex_api/method_source.py+++ b/api/pipelex_api/method_source.py@@ -19,16 +19,10 @@ The packages a package ships under its own `.mthds/methods/<name>/`, the layoutits tree (a dot directory), and each becomes a dependency tree of its own, keyed by the address its manifest declares,as a dependency a request sends in `dependencies` is.-The routes use it two ways:--- The run routes and `/validate` use :func:`fetched_method`: the tree and its dependency trees, read in full, which the- routes write out and load as a request's `files` are (`pipelex_api.method_tree_request`), so a fetched package and a- sent method run the same path, the files a step reads, such as a `PipeDocGen` template, included.-- The tooling routes (`/resolve`, `/codegen`, `/pipe-io`) use :func:`fetch_method_mthds_files`: the tree's `.mthds`- files, as `files[]` items labelled with their tree paths, paired with the manifest's `main_pipe` so the per-pipe route- defaults its selector as a run does. It reads only what the membership rule needs to tell which `.mthds` files are- the tree's, never the shipped packages, so it neither stages nor hashes the rest of the package. No Python ever loads- there, so the execution-locus gate does not apply.+Every route taking a `method_ref` — the run routes, `/validate`, and the crate routes `/resolve`, `/codegen` and+`/pipe-io` — uses :func:`fetched_method`: the tree and its dependency trees, read in full, which the routes write out+and load as a request's `files` are (`pipelex_api.method_tree_request`), so a fetched package and a sent method run the+same path, the files a step reads, such as a `PipeDocGen` template, included.The security gate (packaging invariant 7 — execution locus decides) is the request tree's(`pipelex_api.method_tree_request.ensure_python_allowed`): on a deployment that is not sandbox-hosted, a package@@ -44,21 +38,17 @@ import osimport shutilimport subprocessimport tempfile-from collections.abc import Callable, Generator+from collections.abc import Generatorfrom contextlib import contextmanagerfrom pathlib import Pathfrom typing import NamedTuple-from mthds.method_tree.discovery import BUNDLE_SUFFIX, GITIGNORE_NAME, MANIFEST_NAMEfrom pipelex import logfrom pipelex.libraries.method_trees import list_package_method_treefrom pipelex.methods.exceptions import MethodFetchErrorfrom pipelex.methods.fetching import FetchedMethodPackage, MethodProvenancefrom pipelex.methods.method_ref import parse_method_ref-from pipelex_api.error_types import ErrorType-from pipelex_api.errors import raise_validation_error-from pipelex_api.limits import MAX_MTHDS_FILE_BYTESfrom pipelex_api.method_cache import get_method_clone_cachefrom pipelex_api.method_tree_request import (RequestMethod,@@ -69,7 +59,6 @@ from pipelex_api.method_tree_request import (ensure_tree_within_limits,is_package_dir_name,)-from pipelex_api.schemas.models import MthdsFileItemGIT_TIMEOUT_SECONDS = 60# The git modes of a tree entry this reader treats apart from a file: an executable file, a symbolic link, and a submodule's commit.@@ -78,9 +67,6 @@ _GIT_MODE_SYMLINK = "120000"_GIT_MODE_SUBMODULE = "160000"# Where a package ships the method packages it calls, the layout of the local method cache._VENDORED_METHODS_DIR = Path(".mthds") / "methods"-# The files the membership rule reads to judge an entry beside the entry's own name: the patterns of a `.gitignore` file-# and of the manifest, a manifest marking a nested package, and a `pyvenv.cfg` marking a virtual environment.-_MEMBERSHIP_RULE_NAMES = frozenset({GITIGNORE_NAME, MANIFEST_NAME, "pyvenv.cfg"})class FetchedMethod(NamedTuple):@@ -96,16 +82,6 @@ class FetchedMethod(NamedTuple):return f"'{self.package.full_address}'"-class FetchedMthdsFiles(NamedTuple):- """A fetched package shaped for the tooling path: its `.mthds` files plus the manifest's entry pipe."""-- files: list[MthdsFileItem]- """The package's `.mthds` files as `files[]` items, each labelled with its tree path."""-- main_pipe: str | None- """The manifest's declared `main_pipe` (a bare pipe code), or None when the manifest declares none."""--class _GitTreeEntry(NamedTuple):mode: strobject_sha: str@@ -200,18 +176,13 @@ def _read_git_objects(*, package: FetchedMethodPackage, object_shas: list[str])return contents-def _stage_package(*, package: FetchedMethodPackage, staging_dir: Path, keeps: Callable[[str], bool] | None = None) -> None:+def _stage_package(*, package: FetchedMethodPackage, staging_dir: Path) -> None:"""Write the package's directory as its commit stores it into `staging_dir`, from the clone's git objects.A symbolic link is written as one, for the tree's lister to refuse unless the membership rule excludes it, and asubmodule contributes no file.-- Args:- package: The fetched package.- staging_dir: The directory to write it into.- keeps: Which paths, relative to the package's directory, to write; every path when None."""- entries = [entry for entry in _package_tree_entries(package=package) if keeps is None or keeps(entry.path)]+ entries = _package_tree_entries(package=package)blobs = _read_git_objects(package=package, object_shas=[entry.object_sha for entry in entries if entry.mode != _GIT_MODE_SUBMODULE])for entry in entries:if entry.mode == _GIT_MODE_SUBMODULE:@@ -241,7 +212,15 @@ def _read_listed_tree(*, root: Path, package_address: str, label: str) -> Requesdef _read_package_method(*, package: FetchedMethodPackage) -> RequestMethod:- """The package's tree and the dependency trees it ships, read from the commit's git objects."""+ """The package's tree and the dependency trees it ships, read from the commit's git objects.++ The staging directory is read into memory and removed here, and a route loading the method writes the tree out a+ second time (`written_method`), rather than loading the staged directory in place. The staged copy is the commit+ as git holds it: files the membership rule leaves out of the tree, symlinks, and entries under `.mthds/methods/`+ that are no package all stay in it, so loading it would load more than the tree. The tree is needed in memory+ anyway, for the Python policy, the limits, the `include_files` echo and the blob store, and every method a route+ loads, sent or fetched, is then written by the one `written_method`, which owns its teardown.+ """staging_dir = Path(tempfile.mkdtemp(prefix="pipelex-package-"))try:package_root = staging_dir / "package"@@ -292,27 +271,6 @@ def _fetched_package(method_ref: str) -> Generator[FetchedMethodPackage, None, Nyield package-def _keeps_for_bundle_listing(path: str) -> bool:- """Whether a package's file is one the tooling read writes: a `.mthds` file, or one the membership rule reads."""- name = path.rsplit("/", 1)[-1]- return path.endswith(BUNDLE_SUFFIX) or name in _MEMBERSHIP_RULE_NAMES---def _read_package_bundles(*, package: FetchedMethodPackage) -> dict[str, bytes]:- """The `.mthds` files of the package's tree by tree path, read without staging or hashing the rest of the package.-- Only the `.mthds` files and the files the membership rule reads are written out, so the lister keeps exactly the- `.mthds` files the whole tree would keep. The packages the package ships stay out, being under a dot directory.- """- staging_dir = Path(tempfile.mkdtemp(prefix="pipelex-package-bundles-"))- try:- _stage_package(package=package, staging_dir=staging_dir, keeps=_keeps_for_bundle_listing)- tree = list_package_method_tree(root=staging_dir, package_address=package.full_address)- return {tree_path: tree.file_location(tree_path).read_bytes() for tree_path in tree.tree_paths if tree_path.endswith(BUNDLE_SUFFIX)}- finally:- shutil.rmtree(staging_dir, ignore_errors=True)--def fetched_method(method_ref: str) -> FetchedMethod:"""Fetch a `method_ref`'s package and read it as a method tree, with the dependency trees it ships.@@ -329,37 +287,3 @@ def fetched_method(method_ref: str) -> FetchedMethod:request_method = _read_package_method(package=package)provenance = package.provenance.model_copy(update={"tree_digest": request_method.tree.tree.digest})return FetchedMethod(request_method=request_method, package=package, provenance=provenance)---def fetch_method_mthds_files(method_ref: str) -> FetchedMthdsFiles:- """Fetch a `method_ref`'s package and return its tree's `.mthds` files as `files[]` items, plus its `main_pipe`.-- The tooling-route shape: each item pairs the file's content with its tree path as `source`, so crate provenance and- diagnostics carry true per-file labels. Only the package's own `.mthds` files travel, never those of a package it- ships under its `.mthds/methods/`, since these routes resolve no address-based dependency. The manifest's `main_pipe`- rides beside the files so a per-pipe projection can default its selector the way a run does.-- Args:- method_ref: The raw `method_ref` string from the request.-- Raises:- MethodRefError subclasses: parse, fetch, location, and bounds failures.- MethodTreeLoadError: The standard refused the package's tree.- ApiError: 422 for a package with no `.mthds` file or one this server cannot accept.- """- with _fetched_package(method_ref) as package:- bundles = _read_package_bundles(package=package)- label = f"'{package.full_address}'"- items: list[MthdsFileItem] = []- for tree_path, data in bundles.items():- try:- content = data.decode("utf-8")- except UnicodeDecodeError:- raise_validation_error(message=f"File '{tree_path}' in method package {label} is not valid UTF-8.")- if len(data) > MAX_MTHDS_FILE_BYTES:- msg = f"File '{tree_path}' in method package {label} exceeds the {MAX_MTHDS_FILE_BYTES // 1024} KiB per-file limit."- raise_validation_error(message=msg)- items.append(MthdsFileItem(content=content, source=tree_path))- if not items:- raise_validation_error(message=f"Method package {label} contains no {BUNDLE_SUFFIX} file.", error_type=ErrorType.INVALID_BUNDLE)- return FetchedMthdsFiles(files=items, main_pipe=package.manifest.main_pipe)diff --git a/api/pipelex_api/method_tree_request.py b/api/pipelex_api/method_tree_request.pyindex 27798512f..ba00488b9 100644--- a/api/pipelex_api/method_tree_request.py+++ b/api/pipelex_api/method_tree_request.py@@ -100,6 +100,17 @@ class RequestTree(NamedTuple):def bundle_paths(self) -> list[str]:return [tree_path for tree_path in self.tree.tree_paths if tree_path.endswith(BUNDLE_SUFFIX)]+ @property+ def bundle_texts(self) -> dict[str, str]:+ """The text of each `.mthds` bundle of the tree by its tree path, in tree path order, as a valid verdict echoes them.++ Decoded strictly, so it is read only once the bundles loaded: a bundle that is not UTF-8 never loads.++ Raises:+ UnicodeDecodeError: A bundle is not UTF-8 text, which a tree that loaded never holds.+ """+ return {tree_path: self.contents[tree_path].decode("utf-8") for tree_path in self.bundle_paths}+@classmethoddef make_from_contents(cls, *, contents: dict[str, bytes]) -> "RequestTree":"""The tree these files make, each at its tree path, already judged as a tree's files."""diff --git a/api/pipelex_api/openapi_responses.py b/api/pipelex_api/openapi_responses.pyindex 9ca2c46cc..f6e349aa3 100644--- a/api/pipelex_api/openapi_responses.py+++ b/api/pipelex_api/openapi_responses.py@@ -124,7 +124,22 @@ PROBLEM_403_RUN_POLICY: dict[str, Any] = _problem("`MethodStructuresRefusedError` — the method's Python declares structure classes (`StructuredContent` ""subclasses), whether it arrived as a bundle (`files` or `bundle_b64`) or as a fetched `method_ref` package: ""a sandbox-hosted deployment imports no caller Python into its own process, so it refuses them before "- "loading anything (express the types as MTHDS concepts instead).",+ "loading anything (express the types as MTHDS concepts instead); or `ReadScopeForbidden` — on a `jwt` deployment, "+ "the request sent a `read_scope` that does not lie under the caller's own `user_id`.",+)++PROBLEM_403_METHOD_PYTHON_POLICY: dict[str, Any] = _problem(+ "The run routes' Python policy, applied to the method's tree: `CustomCodeRequiresSandbox` — the method (sent in "+ "`files`, or a fetched `method_ref` package) ships custom Python and this deployment is not sandbox-hosted; or "+ "`MethodStructuresRefusedError` — the method's Python declares structure classes (`StructuredContent` subclasses), "+ "which a sandbox-hosted deployment never imports into its own process (express the types as MTHDS concepts instead). "+ "Also `ReadScopeForbidden` — on a `jwt` deployment, the request sent a `read_scope` that does not lie under the "+ "caller's own `user_id`.",+)++PROBLEM_403_READ_SCOPE: dict[str, Any] = _problem(+ "`ReadScopeForbidden` — on a `jwt` deployment, the request sent a `read_scope` that does not lie under the caller's "+ "own `user_id`, the only scope its token may read.",)PROBLEM_409_DUPLICATE_RUN: dict[str, Any] = _problem(@@ -174,9 +189,9 @@ PROBLEM_403_BLOB_UPLOAD: dict[str, Any] = _problem()PROBLEM_501_METHOD_REF: dict[str, Any] = _problem(- "`MethodRefNotSupported` — the request selected its closure by a **registry-form** `method_ref` (not a "+ "`MethodRefNotSupported` — the request selected its method by a **registry-form** `method_ref` (not a ""`github.com/...` address), and no server-side method registry resolves those yet. Use an address-form reference "- "or submit inline `files[]` instead.",+ "or send the method's tree in `files` instead.",)diff --git a/api/pipelex_api/routes/pipelex/blobs.py b/api/pipelex_api/routes/pipelex/blobs.pyindex 7b1c4fd7b..74f798956 100644--- a/api/pipelex_api/routes/pipelex/blobs.py+++ b/api/pipelex_api/routes/pipelex/blobs.py@@ -4,7 +4,8 @@ A sender about to send a method by hash first asks which of its files' bytes thimissing ones to the targets the answer names (`pipelex_api.routes.blob_upload`), then sends the request with hashentries. The blobs live in the runner's blob store (`pipelex_api.blob_store`), under the caller's scope: the request's`read_scope` query parameter when a multi-tenant host sends one for the organization it serves, else the caller's own-id, the scope a run's hash entries are resolved in.+id, the scope a run's hash entries are resolved in. A caller holding its own token (`AUTH_MODE=jwt`) may name only a+scope under its own id (`bound_read_scope`)."""from typing import Annotated@@ -19,6 +20,7 @@ from pipelex_api.error_types import ErrorTypefrom pipelex_api.errors import raise_validation_errorfrom pipelex_api.json_body import JsonBodyRoutefrom pipelex_api.method_tree_request import ensure_within_tree_limits+from pipelex_api.openapi_responses import PROBLEM_403_READ_SCOPEfrom pipelex_api.routes.pipelex.pipeline import resolve_blob_scoperouter = APIRouter(tags=["blobs"], route_class=JsonBodyRoute)@@ -89,6 +91,7 @@ def _unique_blobs(*, blobs: list[BlobRef]) -> dict[str, int]:"/blobs/missing",summary="Which of these blobs the runner lacks, and where to upload each.",openapi_extra={"x-mthds-protocol": True},+ responses={403: PROBLEM_403_READ_SCOPE},)async def find_missing_blobs(request: Request,diff --git a/api/pipelex_api/routes/pipelex/codegen.py b/api/pipelex_api/routes/pipelex/codegen.pyindex 115e92cfc..afeb2bebd 100644--- a/api/pipelex_api/routes/pipelex/codegen.py+++ b/api/pipelex_api/routes/pipelex/codegen.py@@ -1,13 +1,13 @@from enum import StrEnumfrom typing import Annotated, Literal, Self, Union-from fastapi import APIRouter+from fastapi import APIRouter, Requestfrom fastapi.responses import JSONResponse+from pipelex.base_exceptions import ErrorReportfrom pipelex.codegen.emission import build_stamped_projectionfrom pipelex.codegen.emitters.target import CodegenKind, CodegenTargetfrom pipelex.codegen.emitters.types_emitter import emit_typesfrom pipelex.codegen.lock import CODEGEN_LOCK_FILENAME-from pipelex.pipeline.exceptions import ValidateBundleErrorfrom pipelex.pipeline.usage_stats_reporting import CodeGenerationReport, usage_stats_codegen_targetfrom pipelex.system.usage_stats.usage_stats_contract import UsageStatsCodegenArtifactfrom pipelex.tools.misc.package_utils import get_package_version@@ -15,15 +15,15 @@ from pipelex.tools.typing.pydantic_utils import empty_list_factory_offrom pydantic import BaseModel, Field, model_validatorfrom pipelex_api.json_body import JsonBodyRoute-from pipelex_api.openapi_responses import PROBLEM_404_METHOD_PACKAGE, PROBLEM_501_METHOD_REF+from pipelex_api.openapi_responses import PROBLEM_403_METHOD_PYTHON_POLICY, PROBLEM_404_METHOD_PACKAGE, PROBLEM_501_METHOD_REFfrom pipelex_api.routes.pipelex.crate_ops import (CrateInvalidReport,GeneratedArtifact,+ ResolvedMethod,invalid_crate_report_response,- resolve_requested_crate,- teardown_current_library,+ project_resolved_method,)-from pipelex_api.schemas.models import MthdsFilesRequest+from pipelex_api.schemas.models import MethodTreeRequestrouter = APIRouter(tags=["codegen"], route_class=JsonBodyRoute)@@ -50,8 +50,8 @@ class CodegenRouteKind(StrEnum):return CodegenKind.TYPES-class CodegenRequest(MthdsFilesRequest):- """The codegen request: the shared closure selector plus the two explicit projection axes."""+class CodegenRequest(MethodTreeRequest):+ """The codegen request: the crate routes' method selector plus the two explicit projection axes."""kind: CodegenRouteKind = Field(..., description="What to project. `types` projects the crate's concept set into typed models.")target: CodegenTarget = Field(@@ -112,17 +112,17 @@ CodegenResponse = Annotated[Union[CodegenValidReport, CrateInvalidReport], Field@router.post("/codegen",response_model=CodegenResponse,- # On top of the composite router's shared 401/413/422/500: the `method_ref` closure selector- # the envelope accepts but no server-side method registry resolves yet (shared with `/resolve`).- responses={404: PROBLEM_404_METHOD_PACKAGE, 501: PROBLEM_501_METHOD_REF},+ # On top of the composite router's shared 401/413/422/500: the Python policy's 403s and the+ # `method_ref` fetch outcomes, shared with `/resolve`.+ responses={403: PROBLEM_403_METHOD_PYTHON_POLICY, 404: PROBLEM_404_METHOD_PACKAGE, 501: PROBLEM_501_METHOD_REF},# NOT tagged `x-mthds-protocol` — a Pipelex API extension, like `/resolve`. The MTHDS standard# specifies the crate this reads (the Library Crate Format); it specifies no type projection, so# every `target` here — `ts-zod` and `python-pydantic` no less than `python-structures` — is ours.)-async def codegen_mthds(request_data: CodegenRequest) -> JSONResponse:- """Generate typed artifacts from a library closure (Pipelex API extension).+async def codegen_mthds(request: Request, request_data: CodegenRequest) -> JSONResponse:+ """Generate typed artifacts from a method (Pipelex API extension).- Resolves the closure to its normalized crate (exactly like `POST /resolve`), then projects it+ Resolves the method to its normalized crate (exactly like `POST /resolve`), then projects itthrough the requested `kind`/`target` axes and returns the **stamped** artifact set plus its`codegen.lock` — everything a client needs to materialize a byte-identical local projectionand run the offline drift check. There is deliberately **no** server-side check route: the@@ -134,27 +134,25 @@ async def codegen_mthds(request_data: CodegenRequest) -> JSONResponse:- **Invalid verdict (200, `is_valid: false`):** the library could not be parsed, loaded, orvalidated — `validation_errors[]` from pipelex's one shared builder; no artifacts exist.- **No verdict (non-2xx):** an unknown projection `kind`/`target`, a `pipe_ref` on a- concept-set-wide kind, or a malformed closure selector is a request-shape 422 problem+json;- `method_ref` is a 501 until server-side method registry resolution exists; auth is 401/403;- server fault is 5xx.+ concept-set-wide kind, or a malformed method selector is a request-shape 422 problem+json; the+ method's tree is refused as `/resolve` refuses it (422 `InvalidMethodFiles` or `MissingBlobs`,+ the Python policy's 403s, the `method_ref` fetch outcomes, 501 for a registry-form+ `method_ref`); auth is 401/403; server fault is 5xx."""with CodeGenerationReport(artifact=UsageStatsCodegenArtifact.TYPES, target=usage_stats_codegen_target(target=request_data.target)) as generation_report:- return await _generate_codegen_response(request_data=request_data, generation_report=generation_report)+ return await _generate_codegen_response(request=request, request_data=request_data, generation_report=generation_report)-async def _generate_codegen_response(*, request_data: CodegenRequest, generation_report: CodeGenerationReport) -> JSONResponse:+async def _generate_codegen_response(*, request: Request, request_data: CodegenRequest, generation_report: CodeGenerationReport) -> JSONResponse:"""The body of `codegen_mthds`, timed by `generation_report`, which an invalid verdict declares failed."""- try:- # `types` is concept-set-wide: no pipe is selected, so the manifest's `main_pipe` beside the crate is unused.- # A future per-pipe kind takes it from the `ResolvedClosure` and hands it to `resolve_requested_pipe`.- crate = (await resolve_requested_crate(request_data)).crate- except ValidateBundleError as validate_error:- generation_report.declare_failed()- return invalid_crate_report_response(validate_error.to_error_report())- generation_report.describe(crate=crate)- try:++ def _projection_response(resolution: ResolvedMethod) -> JSONResponse:+ # `types` is concept-set-wide: no pipe is selected, so the method's manifest `main_pipe` is unused. A future+ # per-pipe kind reads it off the resolution and hands it to `resolve_requested_pipe`.+ crate = resolution.crate+ generation_report.describe(crate=crate)emitted = emit_types(crate, target=request_data.target)projection = build_stamped_projection(emitted,@@ -172,5 +170,9 @@ async def _generate_codegen_response(*, request_data: CodegenRequest, generationlock=projection.lock_content,)return JSONResponse(content=report.model_dump(mode="json", by_alias=True))- finally:- teardown_current_library()++ outcome = await project_resolved_method(request=request, request_data=request_data, project=_projection_response)+ if isinstance(outcome, ErrorReport):+ generation_report.declare_failed()+ return invalid_crate_report_response(outcome)+ return outcomediff --git a/api/pipelex_api/routes/pipelex/crate_ops.py b/api/pipelex_api/routes/pipelex/crate_ops.pyindex b21c6f26b..8ff895a61 100644--- a/api/pipelex_api/routes/pipelex/crate_ops.py+++ b/api/pipelex_api/routes/pipelex/crate_ops.py@@ -1,21 +1,33 @@"""Shared crate-resolution plumbing for the `/resolve`, `/codegen` and `/pipe-io` routes.-They all select a closure the same way (inline `files[]` XOR `method_ref`), resolve it through the-same engine core (`pipelex.pipeline.resolve_bundle.resolve_crate_from_contents`), and speak the same-verdict vocabulary as `POST /validate`: a produced verdict is a 200 discriminated on `is_valid`, with-the invalid arm carrying the structured `validation_errors[]` built by pipelex's one shared builder.-This module holds the pieces they share so the envelopes cannot drift.+They all take a method the same way `/validate` and the run routes do — a tree sent in `files`, with its+`dependencies` and `resources`, XOR an address-form `method_ref` — read it with the same machinery+(`pipelex_api.routes.pipelex.method_selection`), write it out as a run writes it+(`pipelex_api.method_tree_request.written_method`), and resolve it through the same engine core+(`pipelex.pipeline.resolve_bundle.resolve_crate_from_method_dir`). The resolution is static: a package the method+calls by its address loads only when the request shipped it, in `dependencies` or under a fetched package's+`.mthds/methods/`, and one nothing shipped is the invalid verdict, never a lookup among the methods installed on this+server or a fetch. They speak the same verdict vocabulary as+`POST /validate`: a produced verdict is a 200 discriminated on `is_valid`, with the invalid arm carrying the+structured `validation_errors[]` built by pipelex's one shared builder. This module holds the pieces they share so+the envelopes cannot drift.The invalid arm here is the *crate* verdict: it deliberately omits `/validate`'s runnability facts(`pending_signatures`, `is_runnable`) — resolution is static (no dry-run sweep, matching`pipelex resolve`), so runnability is not part of its vocabulary. The per-pipe route (`/pipe-io`)rides that same static core: it reads the selected pipe's *declared* IO with no dry-run sweep.++Writing the tree, loading it, projecting the crate and tearing the library down are blocking work, so they run in one+worker thread (`project_resolved_method`), never on the event loop: a slow resolution never stalls the server's other+requests, and the current library, a context variable, is bound, read and cleared in that one thread's context."""import asyncio+from collections.abc import Callablefrom enum import StrEnum-from typing import Literal, NamedTuple, NoReturn+from typing import Literal, NamedTuple, NoReturn, TypeVar+from fastapi import Requestfrom fastapi.responses import JSONResponsefrom pipelex.base_exceptions import ErrorReport, ValidationErrorItemfrom pipelex.cogt.inference.error_classification import UserAction, UserActionKind@@ -24,15 +36,19 @@ from pipelex.libraries.library_crate import LibraryCratefrom pipelex.libraries.pipe.exceptions import EntryPipeAmbiguousError, EntryPipeNotFoundError, PipeNotFoundErrorfrom pipelex.methods.method_ref import looks_like_method_reffrom pipelex.pipe_machinery.pipe_abstract import PipeAbstract-from pipelex.pipeline.resolve_bundle import resolve_crate_from_contents+from pipelex.pipeline.exceptions import ValidateBundleError+from pipelex.pipeline.resolve_bundle import resolve_crate_from_method_dirfrom pipelex.tools.typing.pydantic_utils import empty_list_factory_offrom pydantic import BaseModel, Field-from pipelex_api.bundle_resources import decode_bundle_resourcesfrom pipelex_api.error_types import ErrorTypefrom pipelex_api.errors import raise_not_implemented-from pipelex_api.method_source import fetch_method_mthds_files-from pipelex_api.schemas.models import MthdsFileItem, MthdsFilesRequest+from pipelex_api.method_tree_request import RequestMethod, WrittenMethod, manifest_of, written_method+from pipelex_api.routes.pipelex.method_selection import select_request_method+from pipelex_api.schemas.models import MethodTreeRequest++#: What a route makes of a resolved method, in the worker thread that holds it loaded.+ProjectionT = TypeVar("ProjectionT")class GeneratedArtifact(BaseModel):@@ -74,90 +90,96 @@ def invalid_crate_report_response(error_report: ErrorReport) -> JSONResponse:return JSONResponse(content=invalid_report.model_dump(mode="json", serialize_as_any=True, by_alias=True, exclude_none=True))-class SelectedFiles(NamedTuple):- """What the closure selector resolved to: the files, plus the fetched manifest's entry pipe when there is one."""+class ResolvedMethod(NamedTuple):+ """A method resolved to its crate: the normalized crate, and the method's trees as the request named them."""- files: list[MthdsFileItem]- """The `.mthds` files forming the closure — inline `files[]` verbatim, or the fetched package's."""+ crate: LibraryCrate+ """The normalized library crate the method's tree resolved to."""- manifest_main_pipe: str | None- """The fetched package manifest's `main_pipe` (a bare pipe code). Always None for inline `files[]`, which carry no manifest."""+ request_method: RequestMethod+ """The method's tree and its dependency trees, read in full, so a route echoes its bundles without reading it again."""+ method_label: str+ """How the entry rule's messages name the method: as sent in the request, or by the fetched package's address."""-async def selected_files(request_data: MthdsFilesRequest) -> SelectedFiles:- """The files the closure selector names: inline `files[]`, or an address `method_ref`'s package.+ @property+ def manifest_main_pipe(self) -> str | None:+ """The `main_pipe` the `METHODS.toml` at the method tree's root declares, or None without one.- Every route on the `files[]` envelope reaches it through `resolve_requested_crate`, so they all- give the same answer on every selector.+ A fetched package's manifest and a manifest sent in `files` alike, since both sit at the root of the tree.- An **address-form** `method_ref` (`github.com/<owner>/<repo>[/<selector>][@<tag>]`) resolves- through the same fetch path the run routes use: the package's `.mthds` files come back as- `files[]` items with their real relative paths as per-file sources, and the manifest's- `main_pipe` rides beside them so the per-pipe route can default its selector the way a run- by address does. Only `.mthds` data travels — the package's Python (if any) never loads- on these routes. The **registry form** (any non-address reference) stays reserved and keeps- its 501 until the packaging program's registry phase.+ Raises:+ MethodEntryError: The manifest is not UTF-8 text, or not a valid manifest (a 422 in the input domain).+ """+ manifest = manifest_of(tree=self.request_method.tree, label=f"the method {self.method_label}")+ return manifest.main_pipe if manifest is not None else None- Raises:- ApiError: 501 for a registry-form `method_ref`; 422 for a fetched package this server- cannot accept.- MethodRefError subclasses: parse, fetch, location, and bounds failures on the address- form, rendered as `problem+json` by the global handler.++def _resolve_written_method(*, written: WrittenMethod) -> LibraryCrate | ErrorReport:+ """The crate of a method written out, or the invalid verdict when it does not resolve."""+ try:+ return resolve_crate_from_method_dir(method_dir=written.method_dir, methods_dirs=written.methods_dirs)+ except ValidateBundleError as validate_error:+ # The produced negative verdict: the core already tore its library down.+ return validate_error.to_error_report()+++def _project_written_method(+ *,+ request_method: RequestMethod,+ method_label: str,+ project: Callable[[ResolvedMethod], ProjectionT],+) -> ProjectionT | ErrorReport:+ """Write the method out, resolve it, hand it to `project` while it is loaded, and release everything: blocking work.++ Runs whole in one worker thread, so the library the core binds as current in this thread's context is the one+ `project` reads and the one torn down here, however `project` exits."""- if request_data.method_ref is not None:- if looks_like_method_ref(request_data.method_ref):- # The fetch runs git and reads the package from disk, so it runs off the event loop, as a run's does.- fetched = await asyncio.to_thread(fetch_method_mthds_files, request_data.method_ref)- return SelectedFiles(files=fetched.files, manifest_main_pipe=fetched.main_pipe)+ with written_method(request_method=request_method) as written:+ resolution = _resolve_written_method(written=written)+ if isinstance(resolution, ErrorReport):+ return resolution+ try:+ return project(ResolvedMethod(crate=resolution, request_method=request_method, method_label=method_label))+ finally:+ teardown_current_library()+++async def project_resolved_method(+ *,+ request: Request,+ request_data: MethodTreeRequest,+ project: Callable[[ResolvedMethod], ProjectionT],+) -> ProjectionT | ErrorReport:+ """Resolve the method the request names into its normalized crate, and return what `project` makes of it.++ The method is read as `/validate` reads it (`select_request_method`), then written out as a run writes it and loaded+ alone as the caller's own library directory, its dependency trees as the packages it ships, in a worker thread.+ `project` runs in that thread with the freshly opened library loaded and current, so it can read live pipes, and the+ written tree held; both are released when it returns or raises. A method that does not resolve returns its invalid+ verdict instead, an `ErrorReport` carrying the structured `validation_errors`, with `project` never called and+ nothing left loaded.++ Raises (as problem documents): 501 for a registry-form `method_ref`; the 422s and 403s of `select_request_method`;+ the address-form fetch and location failures (`MethodRefError` subclasses); and whatever `project` raises, such as+ a selection refusal.+ """+ if request_data.method_ref is not None and not looks_like_method_ref(request_data.method_ref):raise_not_implemented("Registry-form method_ref resolution is not available on this server: no method registry is wired. ""Use an address-form reference (a `github.com/...` address, e.g. `github.com/Pipelex/methods/[email protected]`) "- "or submit inline `files[]`.",+ "or send the method's tree in `files`.",error_type=ErrorType.METHOD_REF_NOT_SUPPORTED,)- return SelectedFiles(files=request_data.files or [], manifest_main_pipe=None)---class ResolvedClosure(NamedTuple):- """A resolved closure: its normalized crate, the files it was resolved from, and the fetched manifest's entry pipe."""-- crate: LibraryCrate- """The normalized library crate the closure resolved to."""-- files: list[MthdsFileItem]- """The `.mthds` files the crate was resolved from, carried through from `selected_files` so no route fetches twice."""-- manifest_main_pipe: str | None- """The fetched package manifest's `main_pipe`, carried through from `selected_files` (None for inline `files[]`)."""---async def resolve_requested_crate(request_data: MthdsFilesRequest) -> ResolvedClosure:- """Resolve the request's closure selector into a normalized library crate.-- Inherits the engine core's **loaded-on-success contract**: on success the freshly opened- library is loaded and current (so a route can read live pipes from it) and the route owns its- teardown — call `teardown_current_library()` in a `finally`. On failure the core has already- torn down and restored.-- The fetched manifest's `main_pipe` (when the selector was a `method_ref`) rides beside the- crate so a per-pipe route can hand it to `resolve_requested_pipe` — the crate itself only- knows the domains' own `main_pipe` declarations. The selected files ride beside it too, so a- route that echoes the closure (`/pipe-io` with `include_files`) reads them here rather than- calling `selected_files` again, which would fetch a `method_ref` a second time.-- Raises:- ValidateBundleError: the produced negative verdict (route maps it to the 200 invalid arm).- ApiError: 501 for a registry-form `method_ref` (see `selected_files`).- MethodRefError subclasses: address-form fetch/location failures (see `selected_files`).- """- selection = await selected_files(request_data)- crate = resolve_crate_from_contents(- mthds_contents=[item.content for item in selection.files],- mthds_sources=[item.source for item in selection.files],- # The files the closure's steps read, such as a PipeDocGen template file, which the load reads.- submitted_files=decode_bundle_resources(resources=request_data.resources),+ selected = await select_request_method(+ request=request, method_ref=request_data.method_ref, resources=request_data.resources, read_scope=request_data.read_scope+ )+ return await asyncio.to_thread(+ _project_written_method,+ request_method=selected.request_method,+ method_label=selected.method_label,+ project=project,)- return ResolvedClosure(crate=crate, files=selection.files, manifest_main_pipe=selection.manifest_main_pipe)class RequestedPipe(NamedTuple):@@ -167,7 +189,7 @@ class RequestedPipe(NamedTuple):"""The qualified `domain.pipe_code` actually selected — always qualified, whatever the request spelled."""pipe: PipeAbstract- """The live pipe, read from the library `resolve_requested_crate` left loaded + current."""+ """The live pipe, read from the library `project_resolved_method` holds loaded + current."""class PipeSelectionMissKind(StrEnum):@@ -209,7 +231,7 @@ def resolve_requested_pipe(crate: LibraryCrate, *, pipe_ref: str | None, manifesanswer for the same `pipe_code`. A chain finding no entry pipe or several has no run-route twin tomatch: this route refuses both the way the pipe-selector design classifies them.- Must be called while the library `resolve_requested_crate` opened is still loaded + current.+ Must be called from the `project` of `project_resolved_method`, while its library is loaded + current."""if pipe_ref is not None:selected = _select_entry_pipe(pipe_ref, origin=_SelectorOrigin.REQUEST)@@ -246,15 +268,15 @@ def _raise_selection_refusal(miss: PipeSelectionMiss) -> NoReturn:def select_default_pipe(crate: LibraryCrate, *, manifest_main_pipe: str | None) -> RequestedPipe | PipeSelectionMiss:"""The selection chain without the request's `pipe_ref`: the method's own entry pipe, or why there is none.- The precedence is the run routes' (`pipeline.py`: `pipe_code or fetched.main_pipe`): a fetched- package's manifest `main_pipe` is the package author's declared entry pipe and is taken first;- only without one does the closure's own declaration decide. That last step mirrors `pipelex- codegen inputs` (`inputs_cmd.py::_default_main_pipe_ref`): the single declared `main_pipe`, with- **both** un-defaultable arms missed — a closure declaring none, and one declaring several across- domains. The chain stops at the first link present, so a manifest `main_pipe` the closure does- not declare, or declares in several domains, is a miss rather than a fall-through to the- closure's declarations. Inline `files[]` carry no manifest, so for them the chain is exactly the- closure-declared default.+ The precedence is the run routes' (the protocol's entry rule, `select_tree_entry`): the `main_pipe`+ of the `METHODS.toml` at the method tree's root, a fetched package's or one sent in `files`, is the+ method author's declared entry pipe and is taken first; only without one does the closure's own+ declaration decide. That last step mirrors `pipelex codegen inputs`+ (`inputs_cmd.py::_default_main_pipe_ref`): the single declared `main_pipe`, with **both**+ un-defaultable arms missed — a closure declaring none, and one declaring several across domains.+ The chain stops at the first link present, so a manifest `main_pipe` the closure does not+ declare, or declares in several domains, is a miss rather than a fall-through to the closure's+ declarations. A tree with no manifest at its root gets exactly the closure-declared default.It returns the miss instead of raising, so the one chain serves both `resolve_requested_pipe`(which refuses with the miss's message) and `/pipe-io`'s `default_pipe_ref` (which states `null`),@@ -262,7 +284,7 @@ def select_default_pipe(crate: LibraryCrate, *, manifest_main_pipe: str | None)`/validate`'s `default_pipe_ref`, which is the run default and names the first of severaldeclaring domains where this chain refuses to choose.- Must be called while the library `resolve_requested_crate` opened is still loaded + current.+ Must be called from the `project` of `project_resolved_method`, while its library is loaded + current."""if manifest_main_pipe:return _select_entry_pipe(manifest_main_pipe, origin=_SelectorOrigin.MANIFEST)@@ -308,8 +330,8 @@ def _select_entry_pipe(selector: str, *, origin: _SelectorOrigin) -> RequestedPiscope = "the submitted closure"case _SelectorOrigin.MANIFEST:# The caller never spelled this selector — say where it came from, or the 422 names a pipe out of nowhere.- described = f"Pipe '{selector}' (the fetched package manifest's `main_pipe`)"- scope = "the package's closure"+ described = f"Pipe '{selector}' (the `main_pipe` of the method's manifest)"+ scope = "the method's closure"case _SelectorOrigin.CLOSURE:described = f"Pipe '{selector}' (the closure's declared `main_pipe`)"scope = "the submitted closure"@@ -326,7 +348,7 @@ def _select_entry_pipe(selector: str, *, origin: _SelectorOrigin) -> RequestedPidef teardown_current_library() -> None:- """Tear down the library `resolve_requested_crate` left loaded + current (success-path cleanup)."""+ """Tear down the library the resolution left loaded + current, once its projection is done."""library_id = get_current_library_id_or_none()clear_current_library()if library_id is not None:diff --git a/api/pipelex_api/routes/pipelex/method_selection.py b/api/pipelex_api/routes/pipelex/method_selection.pynew file mode 100644index 000000000..a194e480e--- /dev/null+++ b/api/pipelex_api/routes/pipelex/method_selection.py@@ -0,0 +1,85 @@+"""The method tree a diagnostic request names, read the one way `/validate` and the crate routes share.++`/validate`, `/resolve`, `/codegen` and `/pipe-io` each take a method as a tree: sent in `files`, with the+`dependencies` it carries and the `resources` that join it, or fetched by an address-form `method_ref`. This module reads+either into a `RequestMethod` with the run routes' own machinery (`pipelex_api.method_tree_request`,+`pipelex_api.method_source.fetched_method`), so the four routes accept, refuse and resolve a tree exactly as the run+routes do: the same limits, the same `InvalidMethodFiles` and `MissingBlobs` refusals, the same fetch, and the same+Python policy, a `.py` file refused with a 403 `CustomCodeRequiresSandbox` unless the deployment is sandbox-hosted.++A sent tree's hash entries resolve in the blob store of the request's `read_scope`, else of the caller's own id+(`resolve_blob_scope`), and the tree is recorded in that store only under a `read_scope` the host named: without one, the+fallback scope is the caller's own id, a prefix no host lists or erases, where a diagnostic request must leave nothing.+"""++import asyncio+from typing import NamedTuple++from fastapi import Request++from pipelex_api.blob_store import RunnerBlobStore+from pipelex_api.bundle_resources import decode_bundle_resources+from pipelex_api.errors import raise_validation_error+from pipelex_api.method_source import fetched_method+from pipelex_api.method_tree_request import (+ REQUEST_METHOD_LABEL,+ RequestMethod,+ ensure_method_holds_a_bundle,+ ensure_python_allowed,+ read_request_method,+ record_request_method,+ tree_documents_of,+ with_resources,+)+from pipelex_api.routes.pipelex.pipeline import resolve_blob_scope++# How the refusals of a sent method name it, as the run routes name it.+_SENT_METHOD_LABEL = "This method"+++class SelectedMethod(NamedTuple):+ """A method tree a request named, read in full, and how the entry rule's messages name it."""++ request_method: RequestMethod+ method_label: str+ """`REQUEST_METHOD_LABEL` for a tree sent in `files`, the fetched package's quoted address for a `method_ref`."""+++async def select_request_method(+ *,+ request: Request,+ method_ref: str | None,+ resources: dict[str, str] | None,+ read_scope: str | None,+) -> SelectedMethod:+ """The method tree the request names: the package a `method_ref` fetches, else the tree sent in `files`.++ The request envelope already refused a body naming neither or both, so a request without a `method_ref` sends+ `files`, read from the raw JSON text of its body (`tree_documents_of`).++ Raises (as problem documents): the refusals of `read_request_method` and `with_resources` (422s), the+ `method_ref` fetch outcomes (`fetched_method`), a 422 `InvalidBundle` for a method whose own tree holds no `.mthds`+ file, and the Python policy's 403s (`ensure_python_allowed`).++ The Python policy holds for the crate routes too, although they run nothing: on a deployment that is not+ sandbox-hosted, loading the library imports every `.py` of the tree declaring a `StructuredContent` subclass+ (`LibraryManager.load_libraries`), which runs that file's module-level code in this process. Which files those are+ is a static scan's call, so the policy refuses every `.py` there, exactly as a run and `/v1/validate` do.+ """+ if method_ref is not None:+ fetched = await asyncio.to_thread(fetched_method, method_ref)+ package_label = f"Method package {fetched.label}"+ ensure_method_holds_a_bundle(request_method=fetched.request_method, label=package_label)+ ensure_python_allowed(request_method=fetched.request_method, label=package_label)+ return SelectedMethod(request_method=fetched.request_method, method_label=fetched.label)+ blob_store = RunnerBlobStore.make_for_scope(scope=resolve_blob_scope(request, read_scope=read_scope))+ sent_method = await read_request_method(documents=tree_documents_of(body=await request.body()), blob_store=blob_store)+ if sent_method is None:+ # Unreachable: the request envelope already refused a body naming no method.+ raise_validation_error(message="This request sends no method tree in `files` and no `method_ref`.")+ request_method = with_resources(request_method=sent_method, resources=decode_bundle_resources(resources=resources))+ ensure_method_holds_a_bundle(request_method=request_method, label=_SENT_METHOD_LABEL)+ ensure_python_allowed(request_method=request_method, label=_SENT_METHOD_LABEL)+ if read_scope is not None:+ await record_request_method(request_method=request_method, blob_store=blob_store)+ return SelectedMethod(request_method=request_method, method_label=REQUEST_METHOD_LABEL)diff --git a/api/pipelex_api/routes/pipelex/pipe_io.py b/api/pipelex_api/routes/pipelex/pipe_io.pyindex bb0c346b1..debf3559b 100644--- a/api/pipelex_api/routes/pipelex/pipe_io.py+++ b/api/pipelex_api/routes/pipelex/pipe_io.py@@ -1,28 +1,28 @@from typing import TYPE_CHECKING, Annotated, Any, Literal, Union-from fastapi import APIRouter+from fastapi import APIRouter, Requestfrom fastapi.responses import JSONResponse+from pipelex.base_exceptions import ErrorReportfrom pipelex.interpreter_hub import get_own_pipes, get_pipe_libraryfrom pipelex.pipeline.build_pipe_io_artifacts import build_pipe_io_artifacts-from pipelex.pipeline.exceptions import ValidateBundleError+from pipelex.pipeline.exceptions import MethodEntryErrorfrom pipelex.pipeline.input_form import InputForm, OutputFormfrom pipelex.pipeline.pipe_io_contracts import PipeIOContractsfrom pipelex.pipeline.validate_bundle import build_pending_signatures-from pipelex.tools.typing.pydantic_utils import empty_list_factory_offrom pydantic import BaseModel, Fieldfrom pipelex_api.json_body import JsonBodyRoute-from pipelex_api.openapi_responses import PROBLEM_404_METHOD_PACKAGE, PROBLEM_501_METHOD_REF+from pipelex_api.openapi_responses import PROBLEM_403_METHOD_PYTHON_POLICY, PROBLEM_404_METHOD_PACKAGE, PROBLEM_501_METHOD_REFfrom pipelex_api.routes.pipelex.crate_ops import (CrateInvalidReport,RequestedPipe,+ ResolvedMethod,invalid_crate_report_response,- resolve_requested_crate,+ project_resolved_method,resolve_requested_pipe,select_default_pipe,- teardown_current_library,)-from pipelex_api.schemas.models import MthdsFileItem, MthdsPipeRequest+from pipelex_api.schemas.models import MethodTreePipeRequestif TYPE_CHECKING:from pipelex.pipe_machinery.pipe_abstract import PipeAbstract@@ -30,8 +30,8 @@ if TYPE_CHECKING:router = APIRouter(tags=["pipe-io"], route_class=JsonBodyRoute)-class PipeIORequest(MthdsPipeRequest):- """The pipe I/O request: the crate routes' closure selector and pipe selector, plus the whole-method and echo opt-ins.+class PipeIORequest(MethodTreePipeRequest):+ """The pipe I/O request: the crate routes' method selector and pipe selector, plus the whole-method and echo opt-ins.The hosted catalog selector is replaced by `files` before the request reaches the runner, so thisrequest has no field for it. It takes no `views` either, since the valid arm always carries all@@ -53,9 +53,10 @@ class PipeIORequest(MthdsPipeRequest):include_files: bool = Field(default=False,description=(- "Echo the resolved closure's `.mthds` files on the valid arm, in the request's own `files[]` shape: the request's "- "files for inline `files[]`, the fetched package's `.mthds` files under their package-relative paths for a "- "`method_ref`. Absent from the response unless true."+ "Echo the method's `.mthds` bundles on the valid arm, as a map of file entries in the request's own `files` shape, "+ "each bundle's tree path to its text: the bundles of the tree sent in `files`, whatever entry carried them, or "+ "those of the fetched package's tree for a `method_ref`. Its other files and its dependencies are not echoed. "+ "Absent from the response unless true."),)@@ -97,9 +98,12 @@ class PipeIOValidReport(BaseModel):is_runnable: bool = Field(..., description="`not pending_signatures`, exactly as on `/validate`. No dry run backs it.")# Never serialized from the model: the route attaches the echo itself, and only when asked, so the# field is absent — not empty, not null — otherwise. Declared here so the published schema names it.- files: list[MthdsFileItem] = Field(- default_factory=empty_list_factory_of(MthdsFileItem),- description="The resolved closure's `.mthds` files, in the request's `files[]` shape. Present only with `include_files: true`.",+ files: dict[str, str] = Field(+ default_factory=dict,+ description=(+ "The method's `.mthds` bundles, each tree path to its text, a map of file entries as the request's `files` is. "+ "Present only with `include_files: true`."+ ),)@@ -111,19 +115,19 @@ PipeIOResponse = Annotated[Union[PipeIOValidReport, CrateInvalidReport], Field(d@router.post("/pipe-io",response_model=PipeIOResponse,- # On top of the composite router's shared 401/413/422/500: the `method_ref` fetch outcomes it- # shares with `/resolve`. No structures 403: the route reads only a package's `.mthds` files.- responses={404: PROBLEM_404_METHOD_PACKAGE, 501: PROBLEM_501_METHOD_REF},+ # On top of the composite router's shared 401/413/422/500: the Python policy's 403s and the+ # `method_ref` fetch outcomes it shares with `/resolve` and `/codegen`.+ responses={403: PROBLEM_403_METHOD_PYTHON_POLICY, 404: PROBLEM_404_METHOD_PACKAGE, 501: PROBLEM_501_METHOD_REF},# NOT tagged `x-mthds-protocol`: it carries the standard's artifacts under their neutral names,# but the route is a Pipelex API extension, and the flag marks the standard's five operations alone.)-async def pipe_io(request_data: PipeIORequest) -> JSONResponse:+async def pipe_io(request: Request, request_data: PipeIORequest) -> JSONResponse:"""Return a method's pipe I/O contracts, input form and output form, with no dry run (Pipelex API extension).- The closure resolves through the static core `/resolve` rides, a pipe is selected the way the- per-pipe routes select one, and the three artifacts are derived with `build_pipe_io_artifacts`,- the builder `/validate` and every run call — so a call costs one load and one derivation where- `/validate` dry-runs every pipe.+ The method is read as `/validate` reads a method tree and resolves through the static core+ `/resolve` rides, a pipe is selected the way the per-pipe routes select one, and the three+ artifacts are derived with `build_pipe_io_artifacts`, the builder `/validate` and every run call —+ so a call costs one load and one derivation where `/validate` dry-runs every pipe.Response contract (the `/validate` discipline):@@ -132,55 +136,83 @@ async def pipe_io(request_data: PipeIORequest) -> JSONResponse:`include_files` is true.- **Invalid verdict (200, `is_valid: false`):** the closure could not be parsed, loaded, orstatically validated — the crate verdict, carrying no artifact, no selection and no files.- - **No verdict (non-2xx):** a malformed body and neither or both closure selectors are- request-shape 422s (`ValidationError`); a selection refusal is an input 422 named for its failure,+ - **No verdict (non-2xx):** a malformed body and neither or both method selectors are+ request-shape 422s (`ValidationError`); a `files` map that is not a method tree is a 422+ `InvalidMethodFiles`, and hash entries this runner does not hold one 422 `MissingBlobs`, as on+ `/validate`; custom Python on a deployment that is not sandbox-hosted, or a Python structure+ class, is a 403; a selection refusal is an input 422 named for its failure,`EntryPipeNotFoundError` for an unknown `pipe_ref` or a single-pipe request whose chain finds noentry pipe, `EntryPipeAmbiguousError` for an ambiguous one or a chain that finds several; the`method_ref` fetch outcomes are those of `/resolve` (404, 422, 501);an artifact that cannot be derived (`PipeIOContractError`) is a 500. All RFC 7807`application/problem+json` via the global handlers."""+ outcome = await project_resolved_method(+ request=request,+ request_data=request_data,+ project=lambda resolution: _pipe_io_response(resolution=resolution, request_data=request_data),+ )+ if isinstance(outcome, ErrorReport):+ return invalid_crate_report_response(outcome)+ return outcome+++def _manifest_main_pipe_if_readable(*, resolution: ResolvedMethod) -> tuple[str | None, bool]:+ """The manifest's `main_pipe`, and whether the manifest could be read, for a request that named its pipe.++ Such a request selects its pipe without the manifest, so a `METHODS.toml` that cannot be read refuses nothing: it+ only leaves the method's own default unstated.+ """try:- resolved = await resolve_requested_crate(request_data)- except ValidateBundleError as validate_error:- return invalid_crate_report_response(validate_error.to_error_report())- try:- default_pipe = select_default_pipe(resolved.crate, manifest_main_pipe=resolved.manifest_main_pipe)+ return resolution.manifest_main_pipe, True+ except MethodEntryError:+ return None, False+++def _pipe_io_response(*, resolution: ResolvedMethod, request_data: PipeIORequest) -> JSONResponse:+ """The valid arm for a resolved method, built in the worker thread that holds its library loaded and current."""+ # The manifest is read strictly only when the selection depends on it, which is when the request names no pipe.+ manifest_readable = True+ if request_data.pipe_ref is None:+ manifest_main_pipe = resolution.manifest_main_pipe+ else:+ manifest_main_pipe, manifest_readable = _manifest_main_pipe_if_readable(resolution=resolution)+ default_pipe_ref: str | None = None+ if manifest_readable:+ default_pipe = select_default_pipe(resolution.crate, manifest_main_pipe=manifest_main_pipe)default_pipe_ref = default_pipe.ref if isinstance(default_pipe, RequestedPipe) else None- pipe_ref: str | None- described_pipes: list[PipeAbstract]- if request_data.all_pipes:- # A whole-method answer needs no selection: a requested ref is still resolved (and refused- # when it names nothing), but a method with no single entry pipe is described all the same.- if request_data.pipe_ref is not None:- pipe_ref = resolve_requested_pipe(resolved.crate, pipe_ref=request_data.pipe_ref, manifest_main_pipe=resolved.manifest_main_pipe).ref- else:- pipe_ref = default_pipe_ref- # The closure's own pipes in load order, the order `/validate`'s report walks.- described_pipes = get_own_pipes()+ pipe_ref: str | None+ described_pipes: list[PipeAbstract]+ if request_data.all_pipes:+ # A whole-method answer needs no selection: a requested ref is still resolved (and refused+ # when it names nothing), but a method with no single entry pipe is described all the same.+ if request_data.pipe_ref is not None:+ pipe_ref = resolve_requested_pipe(resolution.crate, pipe_ref=request_data.pipe_ref, manifest_main_pipe=None).refelse:- requested = resolve_requested_pipe(resolved.crate, pipe_ref=request_data.pipe_ref, manifest_main_pipe=resolved.manifest_main_pipe)- pipe_ref = requested.ref- described_pipes = [requested.pipe]- # Raises `PipeIOContractError` when a pipe's input or output JSON Schema cannot be rendered;- # it reaches the global handler as a no-verdict 500, exactly as on `/validate`.- artifacts = build_pipe_io_artifacts(described_pipes)- pending_signatures = build_pending_signatures(get_pipe_library().get_pipes_dict())- report = PipeIOValidReport(- pipe_ref=pipe_ref,- pipe_io_contracts=artifacts.pipe_io_contracts,- input_form=artifacts.input_form,- output_form=artifacts.output_form,- default_pipe_ref=default_pipe_ref,- pending_signatures=pending_signatures,- is_runnable=not pending_signatures,- )- # Dumped as `/validate` dumps its valid arm, never with `exclude_none`: that would strip the- # contracts' `item_count: null` and break the byte equality with `/validate`'s maps.- content: dict[str, Any] = report.model_dump(mode="json", serialize_as_any=True, by_alias=True, exclude={"files"})- if request_data.include_files:- # Each file is echoed in the request's own shape: a `source` the request omitted stays omitted.- content["files"] = [item.model_dump(mode="json", exclude_none=True) for item in resolved.files]- return JSONResponse(content=content)- finally:- teardown_current_library()+ pipe_ref = default_pipe_ref+ # The closure's own pipes in load order, the order `/validate`'s report walks.+ described_pipes = get_own_pipes()+ else:+ requested = resolve_requested_pipe(resolution.crate, pipe_ref=request_data.pipe_ref, manifest_main_pipe=manifest_main_pipe)+ pipe_ref = requested.ref+ described_pipes = [requested.pipe]+ # Raises `PipeIOContractError` when a pipe's input or output JSON Schema cannot be rendered;+ # it reaches the global handler as a no-verdict 500, exactly as on `/validate`.+ artifacts = build_pipe_io_artifacts(described_pipes)+ pending_signatures = build_pending_signatures(get_pipe_library().get_pipes_dict())+ report = PipeIOValidReport(+ pipe_ref=pipe_ref,+ pipe_io_contracts=artifacts.pipe_io_contracts,+ input_form=artifacts.input_form,+ output_form=artifacts.output_form,+ default_pipe_ref=default_pipe_ref,+ pending_signatures=pending_signatures,+ is_runnable=not pending_signatures,+ )+ # Dumped as `/validate` dumps its valid arm, never with `exclude_none`: that would strip the+ # contracts' `item_count: null` and break the byte equality with `/validate`'s maps.+ content: dict[str, Any] = report.model_dump(mode="json", serialize_as_any=True, by_alias=True, exclude={"files"})+ if request_data.include_files:+ # Decoded strictly: every bundle loaded, so every bundle is UTF-8 text.+ content["files"] = resolution.request_method.tree.bundle_texts+ return JSONResponse(content=content)diff --git a/api/pipelex_api/routes/pipelex/pipeline.py b/api/pipelex_api/routes/pipelex/pipeline.pyindex b9f428ef4..fe76d61a9 100644--- a/api/pipelex_api/routes/pipelex/pipeline.py+++ b/api/pipelex_api/routes/pipelex/pipeline.py@@ -33,7 +33,7 @@ from pipelex_api.api_config import get_api_config, resolve_orchestration_modefrom pipelex_api.blob_store import RunnerBlobStorefrom pipelex_api.bundle_resources import decode_bundle_resourcesfrom pipelex_api.error_types import ErrorType-from pipelex_api.errors import raise_bad_request, raise_validation_error+from pipelex_api.errors import raise_bad_request, raise_forbidden, raise_validation_errorfrom pipelex_api.json_body import decode_json_bodyfrom pipelex_api.method_source import fetched_methodfrom pipelex_api.method_tree_request import (@@ -67,6 +67,7 @@ from pipelex_api.schemas.models import (PipelineApiExtras,RunRequest,)+from pipelex_api.security import AuthMode, get_auth_modeif TYPE_CHECKING:from collections.abc import AsyncGenerator@@ -133,21 +134,43 @@ def _resolve_storage_scope(request: Request, *, requested: str | None) -> str:return requested or get_request_user_id(request)+def bound_read_scope(request: Request, *, requested: str | None) -> str | None:+ """The read scope a request sent, once this caller may read under it; every route taking a `read_scope` reads it here.++ Who may name a read scope depends on who the caller is. With `AUTH_MODE=none` the caller is the host itself, a+ trusted proxy forwarding the user's identity (the hosted plane, which sends the organization it serves) or a+ single tenant, and with the shared `api_key` it is one tenant: a sent value is taken as it is. With `AUTH_MODE=jwt`+ the caller reaches this runner directly, its token's `user_id` its only identity, so it reads only under that id: a+ sent scope must be the id or lie under it, else the request is refused with a 403 `ReadScopeForbidden` before+ anything is read, so no caller reads another's blobs or runs under another's scope. A run's storage scope must lie+ under its read scope, so it is bound with it.+ """+ if requested is None:+ return None+ match get_auth_mode():+ case AuthMode.NONE | AuthMode.API_KEY:+ return requested+ case AuthMode.JWT:+ caller_id = get_request_user_id(request)+ if requested.split("/")[0] != caller_id:+ raise_forbidden(+ message=f"The read_scope {requested!r} does not lie under this caller's own id, the only scope its token may read.",+ error_type=ErrorType.READ_SCOPE_FORBIDDEN,+ )+ return requested++def _resolve_read_scope(request: Request, *, requested: str | None) -> str | None:"""What this run may read — the host's value, the caller's own id, or everything.- A sent value is the host's and is taken as it is. The fallback mirrors the storage- scope's (design DR2 of the read-scope campaign): when the field is omitted, a deployment- that identifies its callers scopes the run to the caller's own id, which is also where- their writes land by default, and a single-tenant deployment runs unscoped. That default- bounds a caller only where the host writes the body: a caller reaching a `jwt`- deployment directly can send any read scope, as it can send any storage scope. On a- multi-tenant host that forgets the field, the fallback fails closed: the host's own- storage scope lies under no caller's id, so the run is refused rather than reading- across tenants.+ A sent value is the host's, taken once this caller may read under it (`bound_read_scope`). The fallback mirrors the+ storage scope's (design DR2 of the read-scope campaign): when the field is omitted, a deployment that identifies its+ callers scopes the run to the caller's own id, which is also where their writes land by default, and a+ single-tenant deployment runs unscoped. On a multi-tenant host that forgets the field, the fallback fails closed:+ the host's own storage scope lies under no caller's id, so the run is refused rather than reading across tenants."""if requested is not None:- return requested+ return bound_read_scope(request, requested=requested)user: RequestUser | None = getattr(request.state, "user", None)return user.user_id if user else None@@ -156,10 +179,11 @@ def resolve_blob_scope(request: Request, *, read_scope: str | None) -> str:"""The scope of the blob store a request's hash entries resolve in: its read scope, else the caller's own id.The same scope a run reads under, so a hash entry resolves only in the caller's own blobs, never across tenants:- a multi-tenant host sends the organization it serves as the read scope, and a deployment that identifies no caller- is one tenant, `single-tenant`.+ a multi-tenant host sends the organization it serves as the read scope, a caller holding its own token may name+ only a scope under its id (`bound_read_scope`), and a deployment that identifies no caller is one tenant,+ `single-tenant`."""- return read_scope or get_request_user_id(request)+ return bound_read_scope(request, requested=read_scope) or get_request_user_id(request)class _RunScopes(NamedTuple):diff --git a/api/pipelex_api/routes/pipelex/resolve.py b/api/pipelex_api/routes/pipelex/resolve.pyindex fa99d6ae1..aa743bb56 100644--- a/api/pipelex_api/routes/pipelex/resolve.py+++ b/api/pipelex_api/routes/pipelex/resolve.py@@ -1,21 +1,16 @@import jsonfrom typing import Annotated, Any, Literal, Union-from fastapi import APIRouter+from fastapi import APIRouter, Requestfrom fastapi.responses import JSONResponse+from pipelex.base_exceptions import ErrorReportfrom pipelex.codegen.crate_encoding import encode_crate_json-from pipelex.pipeline.exceptions import ValidateBundleErrorfrom pydantic import BaseModel, Fieldfrom pipelex_api.json_body import JsonBodyRoute-from pipelex_api.openapi_responses import PROBLEM_404_METHOD_PACKAGE, PROBLEM_501_METHOD_REF-from pipelex_api.routes.pipelex.crate_ops import (- CrateInvalidReport,- invalid_crate_report_response,- resolve_requested_crate,- teardown_current_library,-)-from pipelex_api.schemas.models import MthdsFilesRequest+from pipelex_api.openapi_responses import PROBLEM_403_METHOD_PYTHON_POLICY, PROBLEM_404_METHOD_PACKAGE, PROBLEM_501_METHOD_REF+from pipelex_api.routes.pipelex.crate_ops import CrateInvalidReport, ResolvedMethod, invalid_crate_report_response, project_resolved_method+from pipelex_api.schemas.models import MethodTreeRequestrouter = APIRouter(tags=["resolve"], route_class=JsonBodyRoute)@@ -42,40 +37,43 @@ ResolveResponse = Annotated[Union[ResolveValidReport, CrateInvalidReport], Field@router.post("/resolve",response_model=ResolveResponse,- # On top of the composite router's shared 401/413/422/500: the `method_ref` closure selector- # the envelope accepts but no server-side method registry resolves yet.- responses={404: PROBLEM_404_METHOD_PACKAGE, 501: PROBLEM_501_METHOD_REF},+ # On top of the composite router's shared 401/413/422/500: the Python policy's 403s, the+ # `method_ref` package-not-found 404, and the registry-form `method_ref` no registry resolves yet.+ responses={403: PROBLEM_403_METHOD_PYTHON_POLICY, 404: PROBLEM_404_METHOD_PACKAGE, 501: PROBLEM_501_METHOD_REF},# NOT tagged `x-mthds-protocol`: the MTHDS Protocol is the five operations `execute`, `start`,# `validate`, `models`, `version`. `/resolve` is a Pipelex API extension. The *artifact* it# emits — the normalized library crate — IS standard-owned (the MTHDS Library Crate Format), so# its wire fields stay brand-neutral; the route that serves it over HTTP is ours.)-async def resolve_mthds(request_data: MthdsFilesRequest) -> JSONResponse:- """Resolve a library closure into its normalized crate (Pipelex API extension).+async def resolve_mthds(request: Request, request_data: MethodTreeRequest) -> JSONResponse:+ """Resolve a method into its normalized crate (Pipelex API extension).- Resolution is a first-class language operation alongside validation: assemble the closure from- the inline `files[]`, load + statically validate the library, and emit the **normalized- library crate** (fully qualified refs, refinement flattened, natives materialized, fingerprint- set) — the artifact shape the MTHDS standard specifies as the Library Crate Format. It runs no- dry-run sweep — runnability is `/validate`'s vocabulary.+ Resolution is a first-class language operation alongside validation: read the method's tree+ (`files`, or the package a `method_ref` fetches) exactly as `/validate` reads it, load + statically+ validate it as the caller's own library, and emit the **normalized library crate** (fully+ qualified refs, refinement flattened, natives materialized, fingerprint set) — the artifact+ shape the MTHDS standard specifies as the Library Crate Format. It runs no dry-run sweep —+ runnability is `/validate`'s vocabulary.Response contract (the `/validate` discipline):- **Valid verdict (200, `is_valid: true`):** the crate on the valid arm.- **Invalid verdict (200, `is_valid: false`):** the library could not be parsed, loaded, orvalidated — `validation_errors[]` from pipelex's one shared builder.- - **No verdict (non-2xx):** a malformed request body (neither/both closure selectors, an- over-limit file) is a request-shape 422; `method_ref` is a 501 until server-side method- registry resolution exists; auth is 401/403; server fault is 5xx. All RFC 7807- `application/problem+json` via the global handlers.+ - **No verdict (non-2xx):** a malformed request body (neither or both method selectors) is a+ request-shape 422; a `files` map that is not a method tree is a 422 `InvalidMethodFiles`, and+ hash entries this runner does not hold one 422 `MissingBlobs`; custom Python on a deployment+ that is not sandbox-hosted, or a Python structure class, is a 403; a registry-form `method_ref`+ is a 501 until server-side method registry resolution exists; auth is 401/403; server fault is+ 5xx. All RFC 7807 `application/problem+json` via the global handlers."""- try:- # Whole-closure route: no pipe is selected here, so the manifest's `main_pipe` beside the crate is unused.- crate = (await resolve_requested_crate(request_data)).crate- except ValidateBundleError as validate_error:- return invalid_crate_report_response(validate_error.to_error_report())- try:- report = ResolveValidReport(crate=json.loads(encode_crate_json(crate)))- return JSONResponse(content=report.model_dump(mode="json", by_alias=True))- finally:- teardown_current_library()+ outcome = await project_resolved_method(request=request, request_data=request_data, project=_crate_response)+ if isinstance(outcome, ErrorReport):+ return invalid_crate_report_response(outcome)+ return outcome+++def _crate_response(resolution: ResolvedMethod) -> JSONResponse:+ """The valid arm carrying the resolved method's crate, encoded in the worker thread that resolved it."""+ report = ResolveValidReport(crate=json.loads(encode_crate_json(resolution.crate)))+ return JSONResponse(content=report.model_dump(mode="json", by_alias=True))diff --git a/api/pipelex_api/routes/pipelex/validate.py b/api/pipelex_api/routes/pipelex/validate.pyindex d1e317659..50d0b476f 100644--- a/api/pipelex_api/routes/pipelex/validate.py+++ b/api/pipelex_api/routes/pipelex/validate.py@@ -1,4 +1,3 @@-import asynciofrom enum import StrEnumfrom typing import Annotated, Literal, Self, Union@@ -15,26 +14,13 @@ from pipelex.system.storage_scope import validate_read_scopefrom pipelex.tools.typing.pydantic_utils import empty_list_factory_offrom pydantic import BaseModel, Field, field_validator, model_validator-from pipelex_api.blob_store import RunnerBlobStorefrom pipelex_api.bundle_resources import decode_bundle_resources-from pipelex_api.errors import raise_validation_errorfrom pipelex_api.exception_handlers import problem_response_from_error_reportfrom pipelex_api.json_body import JsonBodyRoute-from pipelex_api.method_source import fetched_method-from pipelex_api.method_tree_request import (- REQUEST_METHOD_LABEL,- RequestMethod,- ensure_method_holds_a_bundle,- ensure_python_allowed,- read_request_method,- record_request_method,- select_tree_entry,- tree_documents_of,- with_resources,- written_method,-)+from pipelex_api.method_tree_request import RequestMethod, select_tree_entry, written_methodfrom pipelex_api.openapi_responses import PROBLEM_403_RUN_POLICY, PROBLEM_404_METHOD_PACKAGE-from pipelex_api.routes.pipelex.pipeline import ApiRunner, get_request_user_id, resolve_blob_scope+from pipelex_api.routes.pipelex.method_selection import select_request_method+from pipelex_api.routes.pipelex.pipeline import ApiRunner, bound_read_scope, get_request_user_idfrom pipelex_api.schemas.models import RESOURCES_FIELD_DESCRIPTION, CallerAnalyticsGroupsMixin, MthdsContentsRequestrouter = APIRouter(tags=["validate"], route_class=JsonBodyRoute)@@ -189,12 +175,6 @@ def _effective_default_pipe_ref(report: PipelexValidationReport) -> str | None:return PipeFactory.make_pipe_ref_with_domain(domain_code=blueprint.domain, pipe_code=blueprint.main_pipe)-def _bundle_texts(*, request_method: RequestMethod) -> list[str]:- """The texts of a method tree's `.mthds` bundles, in tree path order, as the valid verdict echoes them."""- tree = request_method.tree- return [tree.contents[tree_path].decode("utf-8", errors="replace") for tree_path in tree.bundle_paths]--def _optional_tree_entry(*, request_method: RequestMethod, method_label: str) -> MethodEntry | None:"""The method's entry pipe, or None when it has none or a bundle does not parse, for the verdict to say why.@@ -383,14 +363,15 @@ async def validate_mthds(request: Request, request_data: ValidateRequest) -> JSOuser_id = get_request_user_id(request)# The scope the host named, which the validator reads the method under: a dispatched validator hands the method's# trees to its worker through that scope's blob store. Never the caller's id a run falls back to, a prefix no host- # lists or erases, where a validation must leave nothing stored.- read_scope = request_data.read_scope+ # lists or erases, where a validation must leave nothing stored. Bound to the caller on every arm, the contents arm+ # included, which reads no blob store of its own (`bound_read_scope`).+ read_scope = bound_read_scope(request, requested=request_data.read_scope)# Verdict-as-value: the runner resolves the orchestration mode and dispatches through the bundle# validator registry, returning either a validation verdict or a classified fault report.# Only `ErrorReport`s with validation diagnostics are invalid-bundle verdicts (→ 200# InvalidReport); backend/config/runtime reports keep the global problem+json mapping.+ validated_method: RequestMethod | None = Noneif request_data.mthds_contents is not None:- validated_contents = request_data.mthds_contentsverdict = await ApiRunner(user_id=user_id,extras=request_data.analytics_groups,@@ -407,32 +388,15 @@ async def validate_mthds(request: Request, request_data: ValidateRequest) -> JSOtree_default_pipe_ref: str | None = Noneelse:# A method tree: sent as `files`, or fetched by `method_ref` from its commit's git objects through the same path- # as a `method_ref` run. Every failure to obtain the tree — a map that is not a tree, missing blobs, a fetch, a- # package location, the custom-Python policy — raises OUT of this block as a non-2xx problem+json: no verdict- # was produced, so it is never rendered as `is_valid: false` (that verdict is about the method's content).- request_method: RequestMethod- if request_data.method_ref is not None:- fetched = await asyncio.to_thread(fetched_method, request_data.method_ref)- request_method = fetched.request_method- method_label = fetched.label- ensure_method_holds_a_bundle(request_method=request_method, label=f"Method package {fetched.label}")- ensure_python_allowed(request_method=request_method, label=f"Method package {fetched.label}")- else:- blob_store = RunnerBlobStore.make_for_scope(scope=resolve_blob_scope(request, read_scope=read_scope))- sent_method = await read_request_method(documents=tree_documents_of(body=await request.body()), blob_store=blob_store)- if sent_method is None:- # Unreachable: the envelope's selector validator already 422'd a body with no `files`.- raise_validation_error(message="provide exactly one of `mthds_contents`, `files` or `method_ref`")- request_method = with_resources(request_method=sent_method, resources=decode_bundle_resources(resources=request_data.resources))- method_label = REQUEST_METHOD_LABEL- ensure_method_holds_a_bundle(request_method=request_method, label="This method")- ensure_python_allowed(request_method=request_method, label="This method")- # A dry run records the tree only under the scope a host named: without one, the fallback scope is the- # caller's own id, a prefix no host lists or erases, and a validation must leave nothing stored there. Under- # one, the tree is held there before the validator is called, so a dispatched validator names it by digest.- if read_scope is not None:- await record_request_method(request_method=request_method, blob_store=blob_store)- validated_contents = _bundle_texts(request_method=request_method)+ # as a `method_ref` run, read the way the crate routes read it (`select_request_method`). Every failure to obtain+ # the tree — a map that is not a tree, missing blobs, a fetch, a package location, the custom-Python policy —+ # raises OUT of this block as a non-2xx problem+json: no verdict was produced, so it is never rendered as+ # `is_valid: false` (that verdict is about the method's content). A sent tree is recorded only under the scope a+ # host named, before the validator is called, so a dispatched validator names it by digest.+ request_method, method_label = await select_request_method(+ request=request, method_ref=request_data.method_ref, resources=request_data.resources, read_scope=read_scope+ )+ validated_method = request_method# The entry pipe of the tree, the one a selector-less run executes, is the graph's pipe and the default the# verdict reports; a tree with none still validates.entry = _optional_tree_entry(request_method=request_method, method_label=method_label)@@ -459,6 +423,8 @@ async def validate_mthds(request: Request, request_data: ValidateRequest) -> JSOreturn _invalid_report_response(verdict, requested_formats=requested_formats)return problem_response_from_error_report(verdict, request=request)report = verdict+ # The echo of a tree is its bundles' text, decoded strictly now that the valid verdict says they loaded.+ validated_contents = request_data.mthds_contents if validated_method is None else list(validated_method.tree.bundle_texts.values())# Splat the report's own field/value pairs so a future canonical field rides the wire# automatically — the wrapper never enumerates (and silently drops) report fields. `is_valid`diff --git a/api/pipelex_api/schemas/models.py b/api/pipelex_api/schemas/models.pyindex 3fd57e0fe..7a8af935f 100644--- a/api/pipelex_api/schemas/models.py+++ b/api/pipelex_api/schemas/models.py@@ -587,84 +587,80 @@ class PipelexApiStartResponse(PipelexRunResultStart):method_provenance: MethodProvenance | None = Field(default=None, description=_METHOD_PROVENANCE_DESCRIPTION)-class MthdsFileItem(BaseModel):- """One inline MTHDS bundle: its content plus an optional logical source for diagnostics.-- The `files[]` envelope pairs each content with its source in one entry (the shape the codegen- spec pins for the resolve/codegen routes), unlike the legacy parallel- `mthds_contents[]`/`mthds_sources[]` lists on `/validate`.- """-- content: str = Field(..., description="MTHDS bundle content.")- source: str | None = Field(- default=None,- max_length=1024,- description=(- "Optional logical source for this bundle (e.g. its path relative to the submitted directory). "- "Threaded onto the blueprint so server-side diagnostics carry a `source` pointing at the owning file."- ),- )-- @field_validator("content")- @classmethod- def _bound_content(cls, value: str) -> str:- _ensure_mthds_file_within_bytes_limit(value)- return value+_TREE_READ_SCOPE_DESCRIPTION = (+ "PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — the scope the request reads the method under, as a run's "+ "`read_scope`: a multi-tenant host sends the one of the organization it serves. A hash entry of `files` resolves in "+ "that scope's blob store, and the tree the request read is recorded there, so a later request can send it by hash. "+ "Omit it and a hash entry resolves in the caller's own id's store, and nothing is recorded."+)-class MthdsFilesRequest(BaseModel):- """Shared closure selector for the resolve/codegen routes: inline `files[]` XOR a `method_ref`.+class MethodTreeRequest(BaseModel):+ """The method selector of the crate routes (`/resolve`, `/codegen`, `/pipe-io`): a tree sent in `files` XOR a `method_ref`.- Exactly one of the two must be provided (spec'd envelope) — both or neither is a request-shape- 422. An **address-form** `method_ref` (`github.com/<owner>/<repo>[/<selector>][@<tag>]`) is- resolved by this server: the repository is fetched at the tag, the package is located by- manifest identity, and its `.mthds` files feed the closure with their real relative paths as- per-file sources. The **registry form** (any non-address reference) stays reserved and answers- 501 until server-side method-registry resolution exists.+ Exactly one of the two must be provided — both or neither is a request-shape 422. The tree is the MTHDS standard's+ map of file entries, read from the request's raw JSON text exactly as `/validate` and the run routes read it+ (`pipelex_api.method_tree_request`), with the `dependencies` it carries and the `resources` that join it. An+ **address-form** `method_ref` (`github.com/<owner>/<repo>[/<selector>][@<tag>]`) is resolved by this server through+ the run routes' fetch path: the package's tree is read from its commit's git objects, with the packages it ships+ under `.mthds/methods/`. The **registry form** (any non-address reference) stays reserved and answers 501 until+ server-side method-registry resolution exists. Either way the method is held to the run routes' Python policy:+ custom `.py` only on a sandbox-hosted deployment, and no Python structure class at all."""- files: list[MthdsFileItem] | None = Field(+ # Documents the wire only: the route reads the map from the request's raw JSON text, never from this field.+ files: Annotated[dict[str, FileEntry] | None, SkipValidation] = Field(default=None,- min_length=1,- max_length=MAX_MTHDS_FILES_PER_REQUEST,- description="Inline MTHDS bundles forming the closure to resolve (content-passing — no server-side path reads).",+ description=f"{FILES_FIELD_DESCRIPTION} Exactly one of `files` / `method_ref`.",)+ dependencies: Annotated[dict[str, dict[str, FileEntry]] | None, SkipValidation] = Field(default=None, description=DEPENDENCIES_FIELD_DESCRIPTION)method_ref: str | None = Field(default=None,min_length=1,max_length=MAX_METHOD_REF_LEN,description=(- "Reference to a published method, resolving to its package's `.mthds` files. Address form "- "(`github.com/<owner>/<repo>[/<selector>][@<tag>]`) is fetched and resolved server-side; the registry "- "form stays reserved (501 until a method registry exists)."+ "PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — a published method by reference, instead of its tree. "+ "Address form (`github.com/<owner>/<repo>[/<selector>][@<tag>]`) is fetched and read as a `files` tree is, "+ "through the same path as a `method_ref` run; the registry form stays reserved (501 until a method registry "+ "exists). Exactly one of `files` / `method_ref`."),)-resources: dict[str, str] | None = Field(default=None, description=RESOURCES_FIELD_DESCRIPTION)+ read_scope: str | None = Field(default=None, description=_TREE_READ_SCOPE_DESCRIPTION)++ @field_validator("read_scope")+ @classmethod+ def _validate_read_scope(cls, value: str | None) -> str | None:+ if value is None:+ return None+ return validate_read_scope(value=value)@model_validator(mode="after")def _exactly_one_selector(self) -> Self:if (self.files is None) == (self.method_ref is None):msg = "provide exactly one of `files` or `method_ref`"raise ValueError(msg)+ if self.dependencies is not None and self.files is None:+ msg = "`dependencies` accompany the method's own `files`"+ raise ValueError(msg)if self.resources and self.method_ref is not None:- msg = "resources accompany inline `files`; a method_ref's files are the package's own"+ msg = "resources accompany a method sent in `files`; a method_ref's files are the package's own"raise ValueError(msg)return self-class MthdsPipeRequest(MthdsFilesRequest):- """The base of a per-pipe route's request (`/pipe-io`): the closure selector plus a pipe selector.+class MethodTreePipeRequest(MethodTreeRequest):+ """The base of a per-pipe route's request (`/pipe-io`): the method selector plus a pipe selector.The pipe selector is the **qualified** ref `domain.pipe_code`, mirroring `pipelex codegen inputs- --pipe`. It is optional, and an omitted selector defaults with the run routes' precedence: on a- `method_ref` request, to the fetched package manifest's `main_pipe` (the package author's declared- entry pipe); otherwise to the closure's declared `main_pipe`, which is what a single-bundle caller- almost always wants. A closure that declares no `main_pipe` — or several, across domains — cannot- be defaulted from its own declarations, so when no manifest `main_pipe` settles it a route that- needs one pipe refuses the omitted selector with an input 422, `EntryPipeNotFoundError` or- `EntryPipeAmbiguousError` (`crate_ops.resolve_requested_pipe`), never the request-shape- `ValidationError` of a malformed body.+ --pipe`. It is optional, and an omitted selector defaults with the run routes' precedence: to the+ `main_pipe` of the `METHODS.toml` at the method tree's root (a fetched package's manifest, or one+ sent in `files`), the method author's declared entry pipe; otherwise to the closure's declared+ `main_pipe`, which is what a single-bundle caller almost always wants. A closure that declares no+ `main_pipe` — or several, across domains — cannot be defaulted from its own declarations, so when+ no manifest `main_pipe` settles it a route that needs one pipe refuses the omitted selector with an+ input 422, `EntryPipeNotFoundError` or `EntryPipeAmbiguousError`+ (`crate_ops.resolve_requested_pipe`), never the request-shape `ValidationError` of a malformed body."""pipe_ref: str | None = Field(@@ -672,9 +668,9 @@ class MthdsPipeRequest(MthdsFilesRequest):min_length=1,max_length=MAX_PIPE_CODE_LEN,description=(- "Qualified pipe ref (`domain.pipe_code`) to select. Optional — defaults to the fetched package manifest's "- "`main_pipe` on a `method_ref` request, else to the closure's declared `main_pipe`; a closure declaring none, "- "or several, with no manifest `main_pipe` to settle it, requires it unless `all_pipes` is set."+ "Qualified pipe ref (`domain.pipe_code`) to select. Optional — defaults to the `main_pipe` of the `METHODS.toml` "+ "at the method tree's root, else to the closure's declared `main_pipe`; a closure declaring none, or several, "+ "with no manifest `main_pipe` to settle it, requires it unless `all_pipes` is set."),)@@ -690,16 +686,13 @@ so the static crate routes do not accept it."""class MthdsContentsRequest(BaseModel):"""The `POST /validate` request envelope: one of `mthds_contents`, `files` and `method_ref`, plus the sweep's `allow_signatures`.- This is the flat-list envelope. `/validate` uses it through this class (its `ValidateRequest`- subclass adds a parallel `mthds_sources` for per-file source labels), and the run routes- `/execute` and `/start` carry the same `mthds_contents` field independently on `RunRequest`.- The newer Pipelex-API extension routes (`/resolve`, `/codegen`, `/pipe-io`) use- `MthdsFilesRequest` instead (`files[]` pairing each content with its `source`, XOR a- `method_ref`), which folds the source label into each entry. `/validate` keeps the flat-list- shape deliberately: it is an MTHDS Protocol route, so changing its inline envelope is a- protocol-level decision owned by the `mthds/` spec, not this server's to take. `method_ref` is- this server's layer-2 extension beside that inline shape — an address is resolved by the- runner, so validate-by-address belongs here (the hosted-only `method_id` does not).+ `/validate` uses it through this class (its `ValidateRequest` subclass adds a parallel+ `mthds_sources` for per-file source labels), and the run routes `/execute` and `/start` carry the+ same `mthds_contents` and `files` fields independently on `RunRequest`. The Pipelex-API extension+ routes (`/resolve`, `/codegen`, `/pipe-io`) take the same `files` tree through+ `MethodTreeRequest`, XOR a `method_ref`, with no inline `mthds_contents`. `method_ref` is this+ server's layer-2 extension — an address is resolved by the runner, so validate-by-address+ belongs here (the hosted-only `method_id` does not)."""mthds_contents: list[str] | None = Field(diff --git a/api/tests/unit/test_bundle_resources_routes.py b/api/tests/unit/test_bundle_resources_routes.pyindex 4a7fc5943..eb8610bc7 100644--- a/api/tests/unit/test_bundle_resources_routes.py+++ b/api/tests/unit/test_bundle_resources_routes.py@@ -2,7 +2,8 @@Pins the wire across the routes that load inline contents: the files reach the step that reads them, a file no stepnames is refused as the request's fault, a path that cannot name a file inside the bundle is refused, and the count,-per-file and total limits are enforced before anything loads. A run handed to an orchestrator other than `direct` names+per-file and total limits are enforced before anything loads. Beside a method tree, which the crate routes take alone,+the files join the tree instead. A run handed to an orchestrator other than `direct` namesthe tree of the submitted files by its digest, in its crate and its run metadata, and carries none of their bytes.The hermetic API environment has no document engine, so the step's engine is stubbed and the dry print's template check@@ -181,16 +182,16 @@ def _zip_b64(files: Mapping[str, str | bytes]) -> str:def _contents_body(*, route: str, resources: dict[str, str], mthds: str = _INVOICE_MTHDS) -> dict[str, Any]:- """The route's body carrying the bundle inline with the given resources: the crate routes take `files[]`."""+ """The route's body carrying the bundle with the given resources: inline where the route takes contents, else as a tree."""match route:case "/v1/validate":return {"mthds_contents": [mthds], "resources": resources}case "/v1/execute":return {"mthds_contents": [mthds], "inputs": _INVOICE_INPUTS, "resources": resources}case "/v1/codegen":- return {"files": [{"content": mthds, "source": "invoicing.mthds"}], "kind": "types", "target": "ts-zod", "resources": resources}+ return {"files": {"invoicing.mthds": mthds}, "kind": "types", "target": "ts-zod", "resources": resources}case _:- return {"files": [{"content": mthds, "source": "invoicing.mthds"}], "resources": resources}+ return {"files": {"invoicing.mthds": mthds}, "resources": resources}def _assert_problem(response: Any, *, status: int, error_type: str) -> str:@@ -229,7 +230,7 @@ class TestBundleResourcesRoutes:assert body["is_valid"] is Falseassert "which was not supplied with its bundle" in str(body["validation_errors"])- @pytest.mark.parametrize("route", ["/v1/validate", "/v1/execute", "/v1/pipe-io", "/v1/resolve", "/v1/codegen"])+ @pytest.mark.parametrize("route", ["/v1/validate", "/v1/execute"])def test_a_file_no_step_names_is_refused_on_every_route_that_loads_contents(self, mocker: MockerFixture, route: str) -> None:_stub_the_document_engine(mocker)body = _contents_body(route=route, resources={_TEMPLATE_PATH: _b64(_TEMPLATE_BYTES), "templates/logo.png": _b64(b"png")})@@ -300,21 +301,16 @@ class TestBundleResourcesRoutes:assert "method_ref" in detail- @pytest.mark.parametrize("route", ["/v1/validate", "/v1/pipe-io"])- def test_a_source_label_naming_a_file_on_the_runner_never_reads_beside_it(self, mocker: MockerFixture, tmp_path: Path, route: str) -> None:+ def test_a_source_label_naming_a_file_on_the_runner_never_reads_beside_it(self, mocker: MockerFixture, tmp_path: Path) -> None:"""A caller's absolute source naming a real bundle file, with the template file beside it, reaches nothing on the runner's disk."""read_templates = _stub_the_document_engine(mocker)bundle_path = tmp_path / "invoicing.mthds"bundle_path.write_text(_INVOICE_MTHDS, encoding="utf-8")(tmp_path / "templates").mkdir()(tmp_path / "templates" / "invoice.docx").write_bytes(_TEMPLATE_BYTES)- body: dict[str, Any] = (- {"mthds_contents": [_INVOICE_MTHDS], "mthds_sources": [str(bundle_path)]}- if route == "/v1/validate"- else {"files": [{"content": _INVOICE_MTHDS, "source": str(bundle_path)}]}- )+ body: dict[str, Any] = {"mthds_contents": [_INVOICE_MTHDS], "mthds_sources": [str(bundle_path)]}- response = _build_client().post(route, json=body)+ response = _build_client().post("/v1/validate", json=body)assert response.status_code == 200, response.textverdict = response.json()@@ -322,6 +318,23 @@ class TestBundleResourcesRoutes:assert "which was not supplied with its bundle" in str(verdict["validation_errors"])assert read_templates == []+ @pytest.mark.parametrize("route", ["/v1/pipe-io", "/v1/resolve", "/v1/codegen"])+ @pytest.mark.parametrize("sends_the_template", [True, False])+ def test_resources_join_the_tree_a_crate_route_reads(self, mocker: MockerFixture, route: str, sends_the_template: bool) -> None:+ """Beside a tree, `resources` are files of it: the step's template is found there, and a file no step names is kept."""+ _stub_the_document_engine(mocker)+ resources = {"notes/readme.txt": _b64(b"kept beside the method")}+ if sends_the_template:+ resources[_TEMPLATE_PATH] = _b64(_TEMPLATE_BYTES)++ response = _build_client().post(route, json=_contents_body(route=route, resources=resources))++ assert response.status_code == 200, response.text+ verdict = response.json()+ assert verdict["is_valid"] is sends_the_template, response.text+ if not sends_the_template:+ assert _TEMPLATE_PATH in json.dumps(verdict["validation_errors"])+def test_resources_with_no_bundle_to_read_them_are_refused(self) -> None:response = _build_client().post("/v1/execute", json={"pipe_code": "print_invoice", "resources": {_TEMPLATE_PATH: _b64(_TEMPLATE_BYTES)}})diff --git a/api/tests/unit/test_codegen_route.py b/api/tests/unit/test_codegen_route.pyindex 69aeff2a2..535bb9269 100644--- a/api/tests/unit/test_codegen_route.py+++ b/api/tests/unit/test_codegen_route.py@@ -1,12 +1,14 @@"""Envelope tests for `POST /v1/codegen` — the typed-artifact projection route.Pins `conformance/specs/pipelex-codegen.md#route-envelopes`, in the `conformance` repo beside the tests-that verify it: the two explicit projection axes (`kind`, `target`); a produced verdict is a 200+that verify it: the method's tree as the protocol's `files` map XOR a `method_ref`, read as `/v1/resolve`+reads it; the two explicit projection axes (`kind`, `target`); a produced verdict is a 200discriminated on `is_valid` with the stamped artifacts + lock on the valid arm; an unknown`kind`/`target` is a request-shape 422 problem+json; the artifacts a client writes verbatim pass theoffline `codegen check` byte-for-byte."""+from collections.abc import Callablefrom pathlib import Pathimport pytest@@ -19,7 +21,7 @@ from pytest_mock import MockerFixturefrom pipelex_api.exception_handlers import register_exception_handlersfrom pipelex_api.routes import router as api_router-from tests.unit._constants import INVALID_MAIN_PIPE_MTHDS, VALID_MTHDS+from tests.unit._constants import INVALID_MAIN_PIPE_MTHDS, STUB_METHOD_ADDRESS, VALID_MTHDSdef _build_client() -> TestClient:@@ -42,7 +44,7 @@ class TestCodegenRoute:client = _build_client()response = client.post("/v1/codegen",- json={"files": [{"content": VALID_MTHDS}], "kind": "types", "target": target},+ json={"files": {"main.mthds": VALID_MTHDS}, "kind": "types", "target": target},)assert response.status_code == 200, response.textbody = response.json()@@ -67,7 +69,7 @@ class TestCodegenRoute:client = _build_client()response = client.post("/v1/codegen",- json={"files": [{"content": VALID_MTHDS}], "kind": "types", "target": "python-pydantic"},+ json={"files": {"main.mthds": VALID_MTHDS}, "kind": "types", "target": "python-pydantic"},)assert response.status_code == 200, response.textbody = response.json()@@ -82,7 +84,7 @@ class TestCodegenRoute:client = _build_client()response = client.post("/v1/codegen",- json={"files": [{"content": INVALID_MAIN_PIPE_MTHDS}], "kind": "types", "target": "ts-zod"},+ json={"files": {"broken.mthds": INVALID_MAIN_PIPE_MTHDS}, "kind": "types", "target": "ts-zod"},)assert response.status_code == 200, response.textbody = response.json()@@ -102,13 +104,31 @@ class TestCodegenRoute:)def test_request_shape_errors_are_422_problem_json(self, payload_patch: dict[str, str]):client = _build_client()- payload: dict[str, object] = {"files": [{"content": VALID_MTHDS}], "kind": "types", "target": "ts-zod"}+ payload: dict[str, object] = {"files": {"main.mthds": VALID_MTHDS}, "kind": "types", "target": "ts-zod"}payload.update(payload_patch)response = client.post("/v1/codegen", json=payload)assert response.status_code == 422, response.textassert response.headers["content-type"] == "application/problem+json"assert response.json()["error_type"] == "ValidationError"+ def test_an_address_method_ref_projects_the_fetched_package(self, install_method_package: Callable[..., Path]):+ install_method_package(files={"documents.mthds": VALID_MTHDS})+ client = _build_client()+ by_ref = client.post("/v1/codegen", json={"method_ref": f"{STUB_METHOD_ADDRESS}@v0.1.0", "kind": "types", "target": "ts-zod"})+ by_tree = client.post("/v1/codegen", json={"files": {"documents.mthds": VALID_MTHDS}, "kind": "types", "target": "ts-zod"})+ assert by_ref.status_code == 200, by_ref.text+ assert by_ref.json()["is_valid"] is True+ # The package's bundle and the same bundle sent in `files` resolve to one crate, so to one projection.+ assert by_ref.json()["crate_fingerprint"] == by_tree.json()["crate_fingerprint"]+ assert by_ref.json()["artifacts"] == by_tree.json()["artifacts"]++ def test_the_retired_list_shape_is_refused_as_no_method_tree(self):+ client = _build_client()+ response = client.post("/v1/codegen", json={"files": [{"content": VALID_MTHDS}], "kind": "types", "target": "ts-zod"})+ assert response.status_code == 422, response.text+ assert response.headers["content-type"] == "application/problem+json"+ assert response.json()["error_type"] == "InvalidMethodFiles"+def test_method_ref_is_501_problem_json_until_registry_exists(self):client = _build_client()response = client.post("/v1/codegen", json={"method_ref": "acme/methods/x", "kind": "types", "target": "ts-zod"})@@ -124,15 +144,15 @@ class TestCodegenRoute:teardown_spy = mocker.spy(library_manager, "teardown")client = _build_client()- response = client.post("/v1/codegen", json={"files": [{"content": VALID_MTHDS}], "kind": "types", "target": "python-pydantic"})+ response = client.post("/v1/codegen", json={"files": {"main.mthds": VALID_MTHDS}, "kind": "types", "target": "python-pydantic"})assert response.status_code == 200, response.textassert open_spy.call_count >= 1assert open_spy.call_count == teardown_spy.call_countdef test_codegen_tears_down_library_when_emission_raises(self, mocker: MockerFixture):- # Leak canary: a failure in the success window — after resolve_requested_crate left its- # library loaded + current, before the finally — must NOT leak that library. The synthetic+ # Leak canary: a failure in the success window — after project_resolved_method's worker left+ # the library loaded + current, before its teardown in the finally — must NOT leak that library. The synthetic# RuntimeError propagates to the global Exception handler (500), and the conservation# property still holds: opens == teardowns.library_manager = get_library_manager()@@ -149,7 +169,7 @@ class TestCodegenRoute:app.include_router(api_router, prefix="/v1")register_exception_handlers(app)client = TestClient(app, raise_server_exceptions=False)- response = client.post("/v1/codegen", json={"files": [{"content": VALID_MTHDS}], "kind": "types", "target": "python-pydantic"})+ response = client.post("/v1/codegen", json={"files": {"main.mthds": VALID_MTHDS}, "kind": "types", "target": "python-pydantic"})assert response.status_code == 500assert open_spy.call_count >= 1diff --git a/api/tests/unit/test_crate_route_concurrency.py b/api/tests/unit/test_crate_route_concurrency.pynew file mode 100644index 000000000..3fd7b3053--- /dev/null+++ b/api/tests/unit/test_crate_route_concurrency.py@@ -0,0 +1,63 @@+"""A crate route resolves off the event loop, so a stalled resolution never holds up the server's other requests.++`project_resolved_method` writes, loads, projects and tears the method down in one worker thread. Were any of it to+run on the event loop again, the stalled resolution below would freeze every other request on the worker until it+gave up, and the request sent meanwhile could not complete. The stall happens before the first load, so the two+requests never load a library at the same time.+"""++import asyncio+import threading+from pathlib import Path++import httpx+import pytest+from fastapi import FastAPI+from pipelex.libraries.library_crate import LibraryCrate+from pipelex.pipeline.resolve_bundle import resolve_crate_from_method_dir+from pytest_mock import MockerFixture++from pipelex_api.exception_handlers import register_exception_handlers+from pipelex_api.routes import router as api_router+from tests.unit._constants import VALID_MTHDS++#: How long the stalled resolution waits to be released before it gives up, which bounds a regression's run.+_STALL_TIMEOUT_SECONDS = 10.0+_REQUEST_TIMEOUT_SECONDS = 30.0+++class TestCrateRouteConcurrency:+ @pytest.mark.asyncio+ async def test_an_unrelated_request_completes_while_a_resolution_is_stalled(self, mocker: MockerFixture):+ entered = threading.Event()+ release = threading.Event()++ def _resolve_after_a_stall(*, method_dir: Path, methods_dirs: list[Path] | None = None) -> LibraryCrate:+ # Only the first resolution stalls; every later one, the unrelated request's included, resolves at once.+ if not entered.is_set():+ entered.set()+ if not release.wait(timeout=_STALL_TIMEOUT_SECONDS):+ msg = "the stalled resolution was never released"+ raise AssertionError(msg)+ return resolve_crate_from_method_dir(method_dir=method_dir, methods_dirs=methods_dirs)++ mocker.patch("pipelex_api.routes.pipelex.crate_ops.resolve_crate_from_method_dir", side_effect=_resolve_after_a_stall)+ app = FastAPI()+ app.include_router(api_router, prefix="/v1")+ register_exception_handlers(app)+ body = {"files": {"main.mthds": VALID_MTHDS}}++ async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://runner") as client:+ stalled = asyncio.create_task(client.post("/v1/resolve", json=body))+ assert await asyncio.to_thread(entered.wait, _STALL_TIMEOUT_SECONDS), "the first resolution never started"+ try:+ unrelated = await asyncio.wait_for(client.post("/v1/resolve", json=body), timeout=_REQUEST_TIMEOUT_SECONDS)+ assert not stalled.done(), "the stalled resolution answered before it was released"+ finally:+ release.set()+ first = await asyncio.wait_for(stalled, timeout=_REQUEST_TIMEOUT_SECONDS)++ assert unrelated.status_code == 200, unrelated.text+ assert unrelated.json()["is_valid"] is True, unrelated.text+ assert first.status_code == 200, first.text+ assert first.json()["is_valid"] is True, first.textdiff --git a/api/tests/unit/test_openapi_contract.py b/api/tests/unit/test_openapi_contract.pyindex 3427fd7e7..b44bff206 100644--- a/api/tests/unit/test_openapi_contract.py+++ b/api/tests/unit/test_openapi_contract.py@@ -62,9 +62,9 @@ ROUTE_EXTRA_STATUSES = {("/v1/execute", "post"): (403, 429),("/v1/start", "post"): (400, 403, 409, 501),("/v1/validate", "post"): (403,),- ("/v1/resolve", "post"): (404, 501),- ("/v1/pipe-io", "post"): (404, 501),- ("/v1/codegen", "post"): (501,),+ ("/v1/resolve", "post"): (403, 404, 501),+ ("/v1/pipe-io", "post"): (403, 404, 501),+ ("/v1/codegen", "post"): (403, 404, 501),}@@ -156,13 +156,13 @@ class TestOpenApiErrorContract:assert "method_id" not in json.dumps(openapi_schema)def test_pipe_io_request_declares_no_hosted_selector_and_no_views(self, openapi_schema: dict[str, Any]):- """The platform resolves the hosted `method_id` into `files[]` before forwarding, so the runner+ """The platform resolves the hosted `method_id` into `files` before forwarding, so the runnernever declares it; and the route always carries all three artifacts, so it takes no `views`."""schemas = openapi_schema["components"]["schemas"]body_ref = openapi_schema["paths"]["/v1/pipe-io"]["post"]["requestBody"]["content"]["application/json"]["schema"]["$ref"]properties = set(schemas[body_ref.rsplit("/", 1)[-1]]["properties"])- assert properties >= {"files", "method_ref", "pipe_ref", "all_pipes", "include_files"}+ assert properties >= {"files", "dependencies", "method_ref", "resources", "read_scope", "pipe_ref", "all_pipes", "include_files"}assert "method_id" not in propertiesassert "views" not in propertiesdiff --git a/api/tests/unit/test_pipe_io_route.py b/api/tests/unit/test_pipe_io_route.pyindex b4a035606..7b8359c5b 100644--- a/api/tests/unit/test_pipe_io_route.py+++ b/api/tests/unit/test_pipe_io_route.py@@ -1,14 +1,16 @@"""Tests for `POST /v1/pipe-io` — a method's three I/O artifacts off the resolved crate, with no dry run.Pins `conformance/specs/pipelex-codegen.md#the-pipe-io-route`, in the `conformance` repo beside the-tests that verify it: the route resolves the closure through the crate routes' static core, selects a-pipe with the per-pipe routes' chain, and returns `pipe_io_contracts`, `input_form` and `output_form`-keyed by qualified `pipe_ref`, beside `pipe_ref`, `default_pipe_ref`, `pending_signatures`,-`is_runnable` and, on request, the closure's files. The pinning cases hold each artifact equal to-`/v1/validate`'s same-named view for the same closure and pipe; the teardown cases hold the-loaded-on-success contract on every exit.+tests that verify it: the route takes the method's tree as the protocol's `files` map XOR a+`method_ref`, resolves it through the crate routes' static core, selects a pipe with the per-pipe+routes' chain, and returns `pipe_io_contracts`, `input_form` and `output_form` keyed by qualified+`pipe_ref`, beside `pipe_ref`, `default_pipe_ref`, `pending_signatures`, `is_runnable` and, on+request, the method's bundles. The pinning cases hold each artifact equal to `/v1/validate`'s+same-named view for the same tree and pipe; the teardown cases hold the loaded-on-success contract on+every exit."""+import base64from collections.abc import Callablefrom pathlib import Pathfrom typing import Any@@ -53,6 +55,8 @@ ARTIFACT_FIELDS = ("pipe_io_contracts", "input_form", "output_form")REQUEST_SHAPE_ERROR = "ValidationError"PIPE_NOT_FOUND_ERROR = "EntryPipeNotFoundError"PIPE_AMBIGUOUS_ERROR = "EntryPipeAmbiguousError"+# A `files` member that is no method tree, refused by the standard's reader as on `/v1/validate`.+INVALID_METHOD_FILES_ERROR = "InvalidMethodFiles"# Every field only the valid arm carries: the invalid arm is the crate verdict alone.VALID_ARM_ONLY_FIELDS = ("pipe_ref", "default_pipe_ref", *ARTIFACT_FIELDS, "pending_signatures", "is_runnable", "files")@@ -65,9 +69,9 @@ def _build_client() -> TestClient:return TestClient(app)-def _files(*contents: str) -> list[dict[str, str]]:- """Inline `files[]` items, each with a distinct `source` so `/v1/validate` can be sent the same labels."""- return [{"content": content, "source": f"bundle_{index}.mthds"} for index, content in enumerate(contents)]+def _files(*contents: str) -> dict[str, str]:+ """A method tree holding each bundle at its own tree path, as text entries of a `files` map."""+ return {f"bundle_{index}.mthds": content for index, content in enumerate(contents)}def _valid_arm(client: TestClient, payload: dict[str, Any]) -> dict[str, Any]:@@ -272,22 +276,46 @@ class TestPipeIoRoute:body = _valid_arm(client, payload)assert "files" not in body- def test_include_files_echoes_the_inline_files_in_the_request_shape(self):- # A `source` the request omitted stays omitted: the echo is the request's own shape.+ def test_include_files_echoes_the_trees_bundles_as_a_map_of_text_entries(self):+ # The echo is the request's own shape, a map of file entries: each bundle as text under its tree path, whatever+ # entry carried it, and none of the tree's other files.client = _build_client()- files = [{"content": VALID_MTHDS, "source": "main.mthds"}, {"content": SIBLING_MTHDS}]+ files: dict[str, Any] = {+ "main.mthds": VALID_MTHDS,+ "steps/wrap.mthds": {"base64": base64.b64encode(SIBLING_MTHDS.encode("utf-8")).decode("ascii")},+ "notes.txt": "Not a bundle",+ }body = _valid_arm(client, {"files": files, "include_files": True})- assert body["files"] == files+ assert body["files"] == {"main.mthds": VALID_MTHDS, "steps/wrap.mthds": SIBLING_MTHDS}- def test_include_files_echoes_a_method_refs_mthds_files_with_relative_paths(self, install_method_package: Callable[..., Path]):+ def test_include_files_echoes_a_method_refs_bundles_under_their_tree_paths(self, install_method_package: Callable[..., Path]):# Only the package's `.mthds` files travel, each under its path relative to the package root.install_method_package(files={"documents.mthds": VALID_MTHDS, "steps/wrap.mthds": SIBLING_MTHDS, "README.md": "# Not a bundle\n"})client = _build_client()body = _valid_arm(client, {"method_ref": STUB_METHOD_REF, "include_files": True})- assert sorted(body["files"], key=lambda item: item["source"]) == [- {"content": VALID_MTHDS, "source": "documents.mthds"},- {"content": SIBLING_MTHDS, "source": "steps/wrap.mthds"},- ]+ assert body["files"] == {"documents.mthds": VALID_MTHDS, "steps/wrap.mthds": SIBLING_MTHDS}++ def test_a_manifest_sent_in_the_tree_names_the_default_pipe(self):+ # A `METHODS.toml` at the root of a sent tree is the method's manifest, as a fetched package's is.+ client = _build_client()+ files = {"METHODS.toml": STUB_METHOD_MANIFEST_NOMAIN_ENTRY, "documents.mthds": NO_MAIN_PIPE_MTHDS}+ body = _valid_arm(client, {"files": files})+ assert body["pipe_ref"] == "nomain.echo"+ assert body["default_pipe_ref"] == "nomain.echo"++ @pytest.mark.parametrize("all_pipes", [False, True])+ def test_a_manifest_that_does_not_parse_refuses_only_a_request_naming_no_pipe(self, all_pipes: bool):+ """A request naming its pipe never reads the manifest to select it, so a broken one only leaves the default unstated."""+ client = _build_client()+ files = {"METHODS.toml": "not toml [", "main.mthds": VALID_MTHDS}++ named = _valid_arm(client, {"files": files, "pipe_ref": "smoke.echo", "all_pipes": all_pipes})+ assert named["pipe_ref"] == "smoke.echo"+ assert "default_pipe_ref" in named+ assert named["default_pipe_ref"] is None++ unnamed = client.post(PIPE_IO_PATH, json={"files": files, "all_pipes": all_pipes})+ _assert_input_422(unnamed, error_type="MethodEntryError")@pytest.mark.parametrize("request_fields",@@ -301,7 +329,7 @@ class TestPipeIoRoute:# The closure is resolved before any pipe is selected, so even a pipe_ref naming nothing# answers the invalid verdict rather than a 422.client = _build_client()- response = client.post(PIPE_IO_PATH, json={"files": [{"content": INVALID_MAIN_PIPE_MTHDS, "source": "broken.mthds"}], **request_fields})+ response = client.post(PIPE_IO_PATH, json={"files": {"broken.mthds": INVALID_MAIN_PIPE_MTHDS}, **request_fields})assert response.status_code == 200, response.textbody = response.json()assert body["is_valid"] is False@@ -364,23 +392,29 @@ class TestPipeIoRoute:[pytest.param({}, id="neither"),pytest.param({"pipe_ref": "smoke.echo"}, id="neither-with-pipe-ref"),- pytest.param({"files": [{"content": VALID_MTHDS}], "method_ref": STUB_METHOD_REF}, id="both"),+ pytest.param({"files": {"main.mthds": VALID_MTHDS}, "method_ref": STUB_METHOD_REF}, id="both"),],)def test_closure_selector_xor_is_a_request_shape_422(self, payload: dict[str, Any]):client = _build_client()_assert_input_422(client.post(PIPE_IO_PATH, json=payload), error_type=REQUEST_SHAPE_ERROR)+ def test_an_out_of_bounds_pipe_ref_is_a_request_shape_422(self):+ client = _build_client()+ payload = {"files": {"main.mthds": VALID_MTHDS}, "pipe_ref": "x" * (MAX_PIPE_CODE_LEN + 1)}+ _assert_input_422(client.post(PIPE_IO_PATH, json=payload), error_type=REQUEST_SHAPE_ERROR)+@pytest.mark.parametrize(- "payload",+ "files",[- pytest.param({"files": []}, id="empty-files"),- pytest.param({"files": [{"content": VALID_MTHDS}], "pipe_ref": "x" * (MAX_PIPE_CODE_LEN + 1)}, id="oversized-pipe-ref"),+ pytest.param({}, id="empty-map"),+ pytest.param([], id="empty-list"),+ pytest.param([{"content": VALID_MTHDS, "source": "main.mthds"}], id="the-retired-list-shape"),],)- def test_out_of_bounds_selectors_are_a_request_shape_422(self, payload: dict[str, Any]):+ def test_a_files_member_that_is_no_method_tree_is_invalid_method_files(self, files: Any):client = _build_client()- _assert_input_422(client.post(PIPE_IO_PATH, json=payload), error_type=REQUEST_SHAPE_ERROR)+ _assert_input_422(client.post(PIPE_IO_PATH, json={"files": files}), error_type=INVALID_METHOD_FILES_ERROR)def test_a_bare_pipe_code_resolves_but_is_answered_qualified(self):# The engine's entry lookup still accepts a bare request code, falling back across domains, until@@ -425,18 +459,13 @@ class TestPipeIoRoute:)def test_artifacts_equal_validate_views(self, contents: tuple[str, ...], request_fields: dict[str, Any]):# The pinning test: each map equals `/v1/validate`'s same-named view restricted to the same- # keys. Validate is asked first, so a closure it would refuse fails here rather than passing- # vacuously; `allow_signatures` lets it accept the signature closures this route accepts.+ # keys, for the same tree. Validate is asked first, so a tree it would refuse fails here rather+ # than passing vacuously; `allow_signatures` lets it accept the signature trees this route accepts.files = _files(*contents)client = _build_client()validate_response = client.post("/v1/validate",- json={- "mthds_contents": [item["content"] for item in files],- "mthds_sources": [item["source"] for item in files],- "views": ["input_form", "output_form"],- "allow_signatures": True,- },+ json={"files": files, "views": ["input_form", "output_form"], "allow_signatures": True},)assert validate_response.status_code == 200, validate_response.textvalidate_body = validate_response.json()@@ -471,10 +500,15 @@ class TestPipeIoRoute:@pytest.mark.parametrize(("payload", "expected_status"),[- pytest.param({"files": [{"content": VALID_MTHDS}]}, 200, id="valid-arm"),- pytest.param({"files": [{"content": NO_MAIN_PIPE_MTHDS}], "all_pipes": True, "include_files": True}, 200, id="all-pipes"),- pytest.param({"files": [{"content": VALID_MTHDS}], "pipe_ref": "smoke.not_a_pipe"}, 422, id="selection-422"),- pytest.param({"files": [{"content": NO_MAIN_PIPE_MTHDS}]}, 422, id="no-entry-pipe-422"),+ pytest.param({"files": {"main.mthds": VALID_MTHDS}}, 200, id="valid-arm"),+ pytest.param({"files": {"main.mthds": NO_MAIN_PIPE_MTHDS}, "all_pipes": True, "include_files": True}, 200, id="all-pipes"),+ pytest.param({"files": {"main.mthds": VALID_MTHDS}, "pipe_ref": "smoke.not_a_pipe"}, 422, id="selection-422"),+ pytest.param({"files": {"main.mthds": NO_MAIN_PIPE_MTHDS}}, 422, id="no-entry-pipe-422"),+ # A manifest that does not parse is the method entry rule's refusal, as on a run naming no pipe.+ pytest.param({"files": {"METHODS.toml": "not toml [", "main.mthds": VALID_MTHDS}}, 422, id="invalid-manifest-422"),+ pytest.param(+ {"files": {"METHODS.toml": "not toml [", "main.mthds": VALID_MTHDS}, "pipe_ref": "smoke.echo"}, 200, id="invalid-manifest-named-pipe"+ ),],)def test_every_exit_tears_its_library_down(self, mocker: MockerFixture, payload: dict[str, Any], expected_status: int):@@ -504,7 +538,7 @@ class TestPipeIoRoute:teardown_spy = mocker.spy(library_manager, "teardown")client = _build_client()- response = client.post(PIPE_IO_PATH, json={"files": [{"content": VALID_MTHDS}]})+ response = client.post(PIPE_IO_PATH, json={"files": {"main.mthds": VALID_MTHDS}})assert response.status_code == 500, response.textassert response.headers["content-type"] == "application/problem+json"diff --git a/api/tests/unit/test_read_scope_binding.py b/api/tests/unit/test_read_scope_binding.pynew file mode 100644index 000000000..cd283b583--- /dev/null+++ b/api/tests/unit/test_read_scope_binding.py@@ -0,0 +1,111 @@+"""A caller holding its own token reads only under its own id, on every route that takes a `read_scope`.++With `AUTH_MODE=jwt` the caller reaches the runner directly, so the read scope it sends is its own claim, not a trusted+host's: a scope lying under another id is refused with a 403 `ReadScopeForbidden` before anything is read, whether the+route resolves hash entries in that scope's blob store, hands it to a validator or runs under it. On the other modes+the caller is the host or one tenant, so the same scope is taken as it is.+"""++from collections.abc import Awaitable, Callable+from typing import Any++import pytest+from fastapi import FastAPI, Request, Response+from fastapi.testclient import TestClient+from mthds.method_tree.digest import sha256_hex+from pytest_mock import MockerFixture++from pipelex_api.api_config import ApiConfig+from pipelex_api.exception_handlers import register_exception_handlers+from pipelex_api.routes import router as api_router+from pipelex_api.routes.blob_upload import router as blob_upload_router+from pipelex_api.security import AUTH_MODE_ENV_VAR, RequestUser+from tests.unit._constants import VALID_MTHDS++_PIPELINE_NS = "pipelex_api.routes.pipelex.pipeline"+_CALLER_ID = "user_a"+_OTHER_ID = "user_b"+_BLOB = b"a file of the caller's method"+++def _build_client(mocker: MockerFixture) -> TestClient:+ """An app whose every request is authenticated as `_CALLER_ID`, the shape `verify_jwt` binds."""+ mocker.patch(+ f"{_PIPELINE_NS}.get_api_config", return_value=ApiConfig(orchestration_mode="direct", allow_request_orchestration_mode_override=False)+ )+ app = FastAPI()++ async def _bind_user(request: Request, call_next: Callable[[Request], Awaitable[Response]]) -> Response:+ request.state.user = RequestUser(user_id=_CALLER_ID)+ return await call_next(request)++ app.middleware("http")(_bind_user)+ app.include_router(api_router, prefix="/v1")+ # The upload targets `/v1/blobs/missing` signs name this route.+ app.include_router(blob_upload_router, prefix="/v1")+ register_exception_handlers(app)+ return TestClient(app)+++def _post(client: TestClient, *, route: str, read_scope: str) -> Any:+ """Each route taking a `read_scope`, sent the given one."""+ match route:+ case "/v1/blobs/missing":+ body: dict[str, Any] = {"blobs": [{"sha256": sha256_hex(_BLOB), "size": len(_BLOB)}]}+ return client.post(route, json=body, params={"read_scope": read_scope})+ case "/v1/validate-contents":+ return client.post("/v1/validate", json={"mthds_contents": [VALID_MTHDS], "read_scope": read_scope})+ case "/v1/validate" | "/v1/resolve" | "/v1/pipe-io":+ return client.post(route, json={"files": {"main.mthds": VALID_MTHDS}, "read_scope": read_scope})+ case "/v1/codegen":+ return client.post(route, json={"files": {"main.mthds": VALID_MTHDS}, "kind": "types", "target": "ts-zod", "read_scope": read_scope})+ case "/v1/execute":+ body = {"pipe_code": "echo", "files": {"main.mthds": VALID_MTHDS}, "inputs": {"text": "hello"}, "dry_run": True}+ return client.post(route, json={**body, "storage_scope": f"{read_scope}/run_1", "read_scope": read_scope})+ case _:+ msg = f"no request shape for {route}"+ raise AssertionError(msg)+++_ROUTES = ["/v1/blobs/missing", "/v1/validate-contents", "/v1/validate", "/v1/resolve", "/v1/pipe-io", "/v1/codegen", "/v1/execute"]+++class TestReadScopeBinding:+ @pytest.mark.parametrize("route", _ROUTES)+ @pytest.mark.parametrize("read_scope", [_OTHER_ID, f"{_OTHER_ID}/{_CALLER_ID}", f"{_CALLER_ID}x"])+ def test_a_jwt_caller_is_refused_a_scope_outside_its_own_id(+ self, mocker: MockerFixture, monkeypatch: pytest.MonkeyPatch, route: str, read_scope: str+ ) -> None:+ monkeypatch.setenv(AUTH_MODE_ENV_VAR, "jwt")+ client = _build_client(mocker)++ response = _post(client, route=route, read_scope=read_scope)++ assert response.status_code == 403, response.text+ assert response.headers["content-type"] == "application/problem+json"+ problem = response.json()+ assert problem["error_type"] == "ReadScopeForbidden", response.text+ assert repr(read_scope) in problem["detail"]++ @pytest.mark.parametrize("route", _ROUTES)+ @pytest.mark.parametrize("read_scope", [_CALLER_ID, f"{_CALLER_ID}/org_x"])+ def test_a_jwt_caller_reads_under_its_own_id(self, mocker: MockerFixture, monkeypatch: pytest.MonkeyPatch, route: str, read_scope: str) -> None:+ monkeypatch.setenv(AUTH_MODE_ENV_VAR, "jwt")+ client = _build_client(mocker)++ response = _post(client, route=route, read_scope=read_scope)++ assert response.status_code == 200, response.text++ @pytest.mark.parametrize("route", _ROUTES)+ @pytest.mark.parametrize("auth_mode", ["none", "api_key"])+ def test_a_host_or_a_single_tenant_names_any_scope(+ self, mocker: MockerFixture, monkeypatch: pytest.MonkeyPatch, route: str, auth_mode: str+ ) -> None:+ """A trusted host sends the organization it serves, which lies under no user's id."""+ monkeypatch.setenv(AUTH_MODE_ENV_VAR, auth_mode)+ client = _build_client(mocker)++ response = _post(client, route=route, read_scope=_OTHER_ID)++ assert response.status_code == 200, response.textdiff --git a/api/tests/unit/test_resolve_route.py b/api/tests/unit/test_resolve_route.pyindex 6b6d2136d..a9f451fb2 100644--- a/api/tests/unit/test_resolve_route.py+++ b/api/tests/unit/test_resolve_route.py@@ -1,39 +1,62 @@"""Envelope tests for `POST /v1/resolve` — the normalized-crate resolution route.Pins `conformance/specs/pipelex-codegen.md#route-envelopes`, in the `conformance` repo beside the tests-that verify it: the request accepts inline `files[]` XOR a `method_ref`; a produced verdict is a 200-discriminated on `is_valid` with the crate on the valid arm; request-shape errors are 422-problem+json; an address-form `method_ref` resolves the fetched package's `.mthds` files into the-closure, while the registry form keeps its honest 501 until the method registry exists.+that verify it: the request takes the method's tree as the protocol's `files` map XOR a `method_ref`,+read exactly as `/v1/validate` and the run routes read it; a produced verdict is a 200 discriminated on+`is_valid` with the crate on the valid arm; request-shape errors are 422 problem+json; an address-form+`method_ref` resolves the fetched package's tree, while the registry form keeps its honest 501 until the+method registry exists."""+import base64from collections.abc import Callablefrom pathlib import Path+from typing import Any+import pytestfrom fastapi import FastAPIfrom fastapi.testclient import TestClient+from mthds.method_tree.digest import sha256_hexfrom mthds.package.manifest.schema import MTHDS_STANDARD_VERSIONfrom pipelex.interpreter_hub import get_library_managerfrom pytest_mock import MockerFixturefrom pipelex_api.exception_handlers import register_exception_handlers+from pipelex_api.limits import MAX_METHOD_TREE_FILE_BYTESfrom pipelex_api.routes import router as api_router+from pipelex_api.routes.blob_upload import router as blob_upload_routerfrom tests.unit._constants import INVALID_MAIN_PIPE_MTHDS, SIBLING_MTHDS, VALID_MTHDS+RESOLVE_PATH = "/v1/resolve"+def _build_client() -> TestClient:app = FastAPI()app.include_router(api_router, prefix="/v1")+ app.include_router(blob_upload_router, prefix="/v1")register_exception_handlers(app)return TestClient(app)+def _hash_entry(data: bytes) -> dict[str, Any]:+ return {"sha256": sha256_hex(data), "size": len(data)}+++def _assert_problem(response: Any, *, status: int, error_type: str) -> dict[str, Any]:+ assert response.status_code == status, response.text+ assert response.headers["content-type"] == "application/problem+json"+ problem: dict[str, Any] = response.json()+ assert problem["error_type"] == error_type, response.text+ assert "is_valid" not in problem+ return problem++class TestResolveRoute:- def test_valid_files_return_200_crate_on_valid_arm(self):+ def test_a_tree_resolves_to_the_crate_on_the_valid_arm(self):client = _build_client()response = client.post(- "/v1/resolve",- json={"files": [{"content": VALID_MTHDS, "source": "main.mthds"}, {"content": SIBLING_MTHDS, "source": "sibling.mthds"}]},+ RESOLVE_PATH,+ json={"files": {"main.mthds": VALID_MTHDS, "steps/sibling.mthds": SIBLING_MTHDS, "README.md": "# Not a bundle\n"}},)assert response.status_code == 200, response.textbody = response.json()@@ -45,63 +68,107 @@ class TestResolveRoute:assert "smoke.echo" in crate["pipes"]assert "smoke.wrap_echo" in crate["pipes"]assert "native.Text" in crate["concepts"], "referenced natives must be materialized into the crate"- # Sources threaded per file[] entry surface in the crate's provenance map.+ # Each bundle's tree path surfaces in the crate's provenance map.assert crate["source_map"]["smoke.echo"] == "main.mthds"- assert crate["source_map"]["smoke.wrap_echo"] == "sibling.mthds"+ assert crate["source_map"]["smoke.wrap_echo"] == "steps/sibling.mthds"- def test_invalid_files_return_200_invalid_arm_with_structured_errors(self):+ def test_every_entry_form_reads_the_same_crate(self):+ # A bundle inline, in Base64 or by hash is the same file of the same tree, so the crate is the same.+ client = _build_client()+ data = VALID_MTHDS.encode("utf-8")+ missing = client.post("/v1/blobs/missing", json={"blobs": [_hash_entry(data)]})+ assert missing.status_code == 200, missing.text+ (blob,) = missing.json()["missing"]+ uploaded = client.request(blob["upload"]["method"], blob["upload"]["url"], content=data, headers=blob["upload"]["headers"])+ assert uploaded.status_code == 204, uploaded.text++ fingerprints: set[str] = set()+ for entry in (VALID_MTHDS, {"base64": base64.b64encode(data).decode("ascii")}, _hash_entry(data)):+ response = client.post(RESOLVE_PATH, json={"files": {"main.mthds": entry}})+ assert response.status_code == 200, response.text+ fingerprints.add(response.json()["crate"]["fingerprint"])+ assert len(fingerprints) == 1++ def test_an_invalid_tree_is_the_invalid_arm_named_by_tree_path(self):client = _build_client()- response = client.post("/v1/resolve", json={"files": [{"content": INVALID_MAIN_PIPE_MTHDS, "source": "broken.mthds"}]})+ response = client.post(RESOLVE_PATH, json={"files": {"methods/broken.mthds": INVALID_MAIN_PIPE_MTHDS}})assert response.status_code == 200, response.textbody = response.json()assert body["is_valid"] is Falseassert body["validation_errors"], "an invalid verdict must carry a non-empty validation_errors[]"assert body["validation_errors"][0]["message"]- # The per-file source threads through to the diagnostics.- assert body["validation_errors"][0]["source"] == "broken.mthds"+ # The bundle's tree path threads through to the diagnostics.+ assert body["validation_errors"][0]["source"] == "methods/broken.mthds"# No crate exists on the invalid arm; the runnability facts are /validate vocabulary, absent here.assert "crate" not in bodyassert "is_runnable" not in bodydef test_address_method_ref_resolves_the_fetched_package(self, install_method_package: Callable[..., Path]):- # The address-form un-reserve (run-by-address MVP Phase 2): a `github.com/...` ref- # resolves through the same fetch path as a `method_ref` run, and the package's- # `.mthds` files feed the closure with their real relative paths as sources.+ # An address-form ref resolves through the same fetch path as a `method_ref` run, and the+ # package's bundles keep their tree paths as sources.install_method_package(files={"documents.mthds": VALID_MTHDS})client = _build_client()- response = client.post("/v1/resolve", json={"method_ref": "github.com/pipelex/methods/[email protected]"})+ response = client.post(RESOLVE_PATH, json={"method_ref": "github.com/pipelex/methods/[email protected]"})assert response.status_code == 200, response.textbody = response.json()assert body["is_valid"] is Truecrate = body["crate"]assert "smoke.echo" in crate["pipes"]- # The package's real file name rides the crate's provenance map.assert crate["source_map"]["smoke.echo"] == "documents.mthds"def test_registry_form_method_ref_keeps_the_501(self):# The registry form (any non-address reference) stays reserved for the packaging# program's closing phase — an honest 501, never a silent empty verdict.client = _build_client()- response = client.post("/v1/resolve", json={"method_ref": "acme/invoice-tools"})- assert response.status_code == 501, response.text- assert response.headers["content-type"] == "application/problem+json"- assert response.json()["error_type"] == "MethodRefNotSupported"-- def test_neither_or_both_selectors_is_422_problem_json(self):- client = _build_client()- neither = client.post("/v1/resolve", json={})- both = client.post("/v1/resolve", json={"files": [{"content": VALID_MTHDS}], "method_ref": "acme/x"})- for response in (neither, both):- assert response.status_code == 422, response.text- assert response.headers["content-type"] == "application/problem+json"- assert response.json()["error_type"] == "ValidationError"-- def test_oversized_file_is_422_problem_json(self):+ response = client.post(RESOLVE_PATH, json={"method_ref": "acme/invoice-tools"})+ _assert_problem(response, status=501, error_type="MethodRefNotSupported")++ @pytest.mark.parametrize(+ "payload",+ [+ pytest.param({}, id="neither"),+ pytest.param({"files": {"main.mthds": VALID_MTHDS}, "method_ref": "acme/x"}, id="both"),+ pytest.param({"method_ref": "acme/x", "dependencies": {}}, id="dependencies-without-files"),+ ],+ )+ def test_a_malformed_method_selector_is_a_request_shape_422(self, payload: dict[str, Any]):+ _assert_problem(_build_client().post(RESOLVE_PATH, json=payload), status=422, error_type="ValidationError")++ @pytest.mark.parametrize(+ ("files", "refusal_code"),+ [+ pytest.param([{"content": VALID_MTHDS, "source": "main.mthds"}], None, id="the-retired-list-shape"),+ pytest.param({}, "empty_tree", id="empty-map"),+ pytest.param({"../main.mthds": VALID_MTHDS}, "invalid_tree_path", id="path-leaving-the-tree"),+ pytest.param({".mthds/main.mthds": VALID_MTHDS}, "excluded_name", id="excluded-name"),+ ],+ )+ def test_a_files_member_that_is_no_method_tree_is_refused_as_validate_refuses_it(self, files: Any, refusal_code: str | None):client = _build_client()- response = client.post("/v1/resolve", json={"files": [{"content": "a" * (2 * 1024 * 1024)}]})- assert response.status_code == 422, response.text- assert response.headers["content-type"] == "application/problem+json"- assert response.json()["error_type"] == "ValidationError"+ problem = _assert_problem(client.post(RESOLVE_PATH, json={"files": files}), status=422, error_type="InvalidMethodFiles")+ validate_problem = _assert_problem(client.post("/v1/validate", json={"files": files}), status=422, error_type="InvalidMethodFiles")+ if refusal_code is not None:+ assert problem["tree_refusal"]["code"] == refusal_code+ assert problem.get("tree_refusal") == validate_problem.get("tree_refusal")++ def test_a_hash_the_runner_does_not_hold_is_a_422_naming_it(self):+ notes = b"notes the runner never received"+ response = _build_client().post(RESOLVE_PATH, json={"files": {"main.mthds": VALID_MTHDS, "notes.txt": _hash_entry(notes)}})+ problem = _assert_problem(response, status=422, error_type="MissingBlobs")+ assert problem["missing_blobs"] == [sha256_hex(notes)]++ def test_a_file_past_the_tree_limit_is_refused(self):+ response = _build_client().post(RESOLVE_PATH, json={"files": {"main.mthds": VALID_MTHDS, "big.txt": "a" * (MAX_METHOD_TREE_FILE_BYTES + 1)}})+ problem = _assert_problem(response, status=422, error_type="MethodTreeTooLarge")+ assert "'big.txt'" in problem["detail"]++ def test_python_in_a_tree_is_refused_where_it_could_not_run(self):+ response = _build_client().post(RESOLVE_PATH, json={"files": {"main.mthds": VALID_MTHDS, "helpers/tool.py": "VALUE = 1\n"}})+ _assert_problem(response, status=403, error_type="CustomCodeRequiresSandbox")++ def test_a_tree_without_a_bundle_is_refused(self):+ response = _build_client().post(RESOLVE_PATH, json={"files": {"notes.txt": "no bundle here"}})+ _assert_problem(response, status=422, error_type="InvalidBundle")def test_resolve_tears_down_its_library(self, mocker: MockerFixture):# The engine core leaves the library loaded + current on success (so /codegen can read live@@ -111,7 +178,7 @@ class TestResolveRoute:teardown_spy = mocker.spy(library_manager, "teardown")client = _build_client()- response = client.post("/v1/resolve", json={"files": [{"content": VALID_MTHDS}]})+ response = client.post(RESOLVE_PATH, json={"files": {"main.mthds": VALID_MTHDS}})assert response.status_code == 200, response.textassert open_spy.call_count >= 1diff --git a/api/tests/unit/test_vendored_method_packages.py b/api/tests/unit/test_vendored_method_packages.pyindex 708d9e4a1..2fbd4a917 100644--- a/api/tests/unit/test_vendored_method_packages.py+++ b/api/tests/unit/test_vendored_method_packages.py@@ -21,6 +21,7 @@ from fastapi import FastAPIfrom fastapi.testclient import TestClientfrom pipelex.base_exceptions import DisclosureModefrom pipelex.config import get_config+from pipelex.methods.address_dependencies import ADDRESS_DEPENDENCY_REFUSAL_MARKERfrom pipelex.pipe_run.delivery_assignment import DeliveryAssignmentfrom pipelex.pipe_run.pipe_job import PipeJobfrom pipelex.plugins.orchestrator_registry import OrchestratorRegistry@@ -30,7 +31,7 @@ from pytest_mock import MockerFixturefrom pipelex_api.api_config import ApiConfigfrom pipelex_api.errors import ApiErrorfrom pipelex_api.exception_handlers import register_exception_handlers-from pipelex_api.method_source import fetch_method_mthds_files, fetched_method+from pipelex_api.method_source import fetched_methodfrom pipelex_api.routes import router as api_routerfrom tests.unit._constants import (STUB_METHOD_ADDRESS,@@ -278,9 +279,6 @@ class TestVendoredMethodPackages:assert sorted(fetched.request_method.dependencies[VENDORED_PROBE_ADDRESS].contents) == ["METHODS.toml", "probe.mthds"]assert fetched.provenance.tree_digest == fetched.request_method.tree.tree.digest- tooling = fetch_method_mthds_files(_METHOD_REF)- assert [item.source for item in tooling.files] == ["smoke.mthds"]-def test_a_dot_directory_under_the_shipped_packages_is_no_package_of_a_method_ref(self, install_method_package: Callable[..., Path]):"""A fetched package and an archive hold `.mthds/methods/<name>/` to one rule: a dot directory there is no package.@@ -296,25 +294,86 @@ class TestVendoredMethodPackages:assert list(fetched.request_method.dependencies) == [VENDORED_PROBE_ADDRESS]- def test_the_tooling_read_keeps_the_tree_s_bundles_alone(self, install_method_package: Callable[..., Path]):- """The tooling routes read the `.mthds` files the tree keeps, without the shipped packages a run needs whole."""- no_manifest = {".mthds/methods/unfinished/draft.mthds": UNRESOLVED_PACKAGE_MTHDS}- install_method_package(- files={- "smoke.mthds": UNRESOLVED_PACKAGE_MTHDS,- "ignored/old.mthds": UNRESOLVED_PACKAGE_MTHDS,- ".gitignore": "ignored/\n",- "notes/readme.txt": "Not read by the tooling routes",- **no_manifest,- }- )-- tooling = fetch_method_mthds_files(_METHOD_REF)+ def test_a_method_ref_package_shipping_a_directory_without_a_manifest_is_refused(+ self, mocker: MockerFixture, tmp_path: Path, install_method_package: Callable[..., Path]+ ):+ """A directory under `.mthds/methods/` is a package the method ships, so one with no manifest is refused on every route."""+ install_method_package(files={"smoke.mthds": UNRESOLVED_PACKAGE_MTHDS, ".mthds/methods/unfinished/draft.mthds": UNRESOLVED_PACKAGE_MTHDS})+ client = _build_client(mocker, tmp_path=tmp_path)- assert [item.source for item in tooling.files] == ["smoke.mthds"]- assert tooling.main_pipe == "echo"with pytest.raises(ApiError, match="unfinished"):fetched_method(_METHOD_REF)+ response = client.post("/v1/pipe-io", json={"method_ref": _METHOD_REF})++ assert response.status_code == 422, response.text+ problem = response.json()+ assert problem["error_type"] == "InvalidMethodFiles"+ assert "unfinished" in problem["detail"]++ @pytest.mark.parametrize("route", ["/v1/pipe-io", "/v1/resolve", "/v1/codegen"])+ def test_a_crate_route_resolves_the_dependency_a_method_ref_ships(+ self, mocker: MockerFixture, tmp_path: Path, install_method_package: Callable[..., Path], route: str+ ):+ """The crate routes read a fetched package's tree with the packages it ships, as `/v1/validate` and a run do."""+ install_method_package(files={"smoke.mthds": UNRESOLVED_PACKAGE_MTHDS, **_package_vendoring_files()})+ client = _build_client(mocker, tmp_path=tmp_path)+ body: dict[str, Any] = {"method_ref": _METHOD_REF}+ if route == "/v1/codegen":+ body |= {"kind": "types", "target": "ts-zod"}++ response = client.post(route, json=body)++ assert response.status_code == 200, response.text+ assert response.json()["is_valid"] is True, response.text++ @pytest.mark.parametrize("route", ["/v1/pipe-io", "/v1/resolve", "/v1/codegen"])+ @pytest.mark.parametrize("sends_the_dependency", [True, False])+ def test_a_crate_route_resolves_a_dependency_sent_beside_the_tree(+ self, mocker: MockerFixture, tmp_path: Path, route: str, sends_the_dependency: bool+ ):+ """A dependency sent in `dependencies` answers the call; without it, the call is the invalid verdict `/v1/validate` gives."""+ client = _build_client(mocker, tmp_path=tmp_path)+ body = _tree_body(VENDORED_PROBE_FILES)+ if not sends_the_dependency:+ body.pop("dependencies")+ if route == "/v1/codegen":+ body |= {"kind": "types", "target": "ts-zod"}++ response = client.post(route, json=body)++ assert response.status_code == 200, response.text+ verdict = response.json()+ assert verdict["is_valid"] is sends_the_dependency, response.text+ if not sends_the_dependency:+ assert [(item["error_type"], item["missing_pipe_code"]) for item in verdict["validation_errors"]] == [+ ("unresolved_package_dependency", UNRESOLVED_PACKAGE_REFERENCE)+ ]++ @pytest.mark.parametrize("route", ["/v1/pipe-io", "/v1/resolve", "/v1/codegen"])+ def test_a_crate_route_never_looks_up_or_fetches_a_dependency_nothing_shipped(self, mocker: MockerFixture, tmp_path: Path, route: str):+ """Even with fetch-on-miss on, a crate route resolves statically, refusing the call in the words a hosted run matches."""+ client = _build_client(mocker, tmp_path=tmp_path)+ mocker.patch("pipelex.methods.fetch_on_miss.is_method_fetch_on_miss_enabled", return_value=True)+ lookup = mocker.patch(+ "pipelex.libraries.library_manager.resolve_address_based_method",+ side_effect=AssertionError("a crate route must not look up installed methods or fetch by address"),+ )+ body = _tree_body(VENDORED_PROBE_FILES)+ body.pop("dependencies")+ if route == "/v1/codegen":+ body |= {"kind": "types", "target": "ts-zod"}++ response = client.post(route, json=body)++ assert response.status_code == 200, response.text+ verdict = response.json()+ assert verdict["is_valid"] is False, response.text+ lookup.assert_not_called()+ [item] = verdict["validation_errors"]+ assert item["error_type"] == "unresolved_package_dependency"+ assert item["source"] == "bundle.mthds"+ assert f"{ADDRESS_DEPENDENCY_REFUSAL_MARKER} '{VENDORED_PROBE_ADDRESS}'" in item["message"]+ assert ADDRESS_DEPENDENCY_REFUSAL_MARKER in verdict["message"]def test_validate_by_method_ref_resolves_the_dependency_it_ships(self, mocker: MockerFixture, tmp_path: Path, install_method_package: Callable[..., Path]diff --git a/docs/api-server/codegen.md b/docs/api-server/codegen.mdindex ddbb7d3c9..a31472ebd 100644--- a/docs/api-server/codegen.md+++ b/docs/api-server/codegen.md@@ -4,18 +4,26 @@ Resolve a library closure into its **normalized library crate**, and project thaBoth are **Pipelex API extensions**, not MTHDS Protocol routes: they are not tagged `x-mthds-protocol` in the [OpenAPI artifact](openapi/pipelex-api.openapi.yaml), and an MTHDS runner is not required to serve them. The *crate* is a different matter — its shape is the standard's [Library Crate Format](https://mthds.ai), so the wire fields are brand-neutral and any MTHDS tool can read one. What is ours is the HTTP surface that produces it, and the type projection on top (the standard specifies none).-Both endpoints speak the [`POST /v1/validate`](pipe-validate.md) verdict discipline: a **produced verdict is always a `200`** discriminated on `is_valid`; the invalid arm carries the structured `validation_errors[]` from pipelex's one shared builder — each item optionally carrying a [`suggested_fix`](error-responses.md#suggested-fixes). Non-2xx is reserved for *no verdict could be produced*: request-shape errors (an unknown projection `kind`/`target`, a malformed closure selector) are `422` RFC 7807 `application/problem+json`, a registry-form `method_ref` is `501`, an address-form `method_ref` that fails to resolve maps per the [`method_ref` error table](pipe-run.md#running-a-method-by-address-method_ref), auth is `401`/`403`, server faults are `5xx`.+Both endpoints speak the [`POST /v1/validate`](pipe-validate.md) verdict discipline: a **produced verdict is always a `200`** discriminated on `is_valid`; the invalid arm carries the structured `validation_errors[]` from pipelex's one shared builder — each item optionally carrying a [`suggested_fix`](error-responses.md#suggested-fixes). Non-2xx is reserved for *no verdict could be produced*: request-shape errors (an unknown projection `kind`/`target`, a malformed method selector) and a method tree that cannot be read are `422` RFC 7807 `application/problem+json`, the custom-Python policy is a `403`, a registry-form `method_ref` is `501`, an address-form `method_ref` that fails to resolve maps per the [`method_ref` error table](pipe-run.md#running-a-method-by-address-method_ref), auth is `401`/`403`, server faults are `5xx`.-## Selecting the closure+## Selecting the method-Both endpoints accept the same closure selector — exactly one of:+Both endpoints take the method the way [`POST /v1/validate`](pipe-validate.md#validating-a-method-tree) and the [run routes](pipe-run.md#sending-a-method-as-a-tree) take it, as a method tree, read by the same machinery, so a tree one route accepts, refuses or resolves, the others accept, refuse or resolve the same way. The request names it with exactly one of:-- `files` (list, content-passing): inline MTHDS bundles, each `{ "content": "...", "source": "optional/logical/path.mthds" }`. `source` threads onto diagnostics and the crate's `source_map`.-- `method_ref` (string): a reference to a published method. The **address form** — `github.com/<owner>/<repo>` plus an optional package selector and `@<tag>` — is resolved server-side through the same fetch path as a [`method_ref` run](pipe-run.md#running-a-method-by-address-method_ref): the package's own `.mthds` files feed the closure, each with its real relative path as `source` (so diagnostics and the crate's `source_map` carry true per-file labels), and the files of a method package it ships under `.mthds/methods/<name>/` do not. Only `.mthds` data travels on these routes — the package's Python (if any) never loads here. The **registry form** (any non-address reference) stays reserved: the server answers **`501`** (`MethodRefNotSupported`) until method-registry resolution lands.+- `files` (object): the method as the MTHDS protocol's map of file entries, each key a tree path and each value the file's text, `{"base64": "…"}` or `{"sha256": "…", "size": n}` naming bytes this runner already holds in its [blob store](pipe-run.md#sending-files-by-hash). The tree is written whole and loaded as the caller's library directory: every `.mthds` bundle is named by its tree path, which threads onto diagnostics and the crate's `source_map`, and every other file is one a step may read, such as a `PipeDocGen` template beside its bundle.+- `method_ref` (string): a reference to a published method. The **address form**, `github.com/<owner>/<repo>` plus an optional package selector and `@<tag>`, is resolved server-side through the same fetch path as a [`method_ref` run](pipe-run.md#running-a-method-by-address-method_ref): the package's whole tree is read from the commit and loaded as a tree sent in `files` is, and the packages it ships under its own `.mthds/methods/<name>/` are loaded beside it, so a call into one resolves. The **registry form** (any non-address reference) stays reserved: the server answers **`501`** (`MethodRefNotSupported`) until method-registry resolution lands.-Providing neither or both is a request-shape `422`.+Providing neither or both is a request-shape `422`. Beside `files`, the request may also carry:-Beside inline `files`, both endpoints accept `resources`: the files the closure's steps read, such as a `PipeDocGen` step's `template_file`, keyed by the path the step names, each in standard base64, with the rules and refusals of [`/v1/validate`](pipe-validate.md#sending-the-files-the-steps-read-resources). A step that names a template file the request did not send cannot be built, so the closure is invalid. `resources` beside a `method_ref` is a request-shape `422`: a published method's files are its package's own.+- `dependencies` (object): the method packages the method calls by address, each a map of file entries keyed by its address, as on a [run](pipe-run.md#sending-the-method-packages-the-method-calls). They are loaded beside the method, never merged into it, so a cross-package call resolves without the server fetching or holding a copy. `dependencies` without `files` is a request-shape `422`.+- `resources` (object): files the method's steps read, such as a `PipeDocGen` step's `template_file`, keyed by the path from the tree's root, each in standard base64, joining the tree with the rules and refusals of [`/v1/validate`](pipe-validate.md#sending-the-files-the-steps-read-resources). A step that names a template file the tree does not hold cannot be built, so the method is invalid. `resources` beside a `method_ref` is a request-shape `422`: a published method's files are its package's own.+- `read_scope` (string): **Pipelex-API extension.** The scope the request reads the method under, as on `/v1/validate`: a hash entry of `files` resolves in that scope's blob store, else in the caller's own id's, and only a request carrying a `read_scope` records the tree it read in the blob store; [a `jwt` caller may send only a scope under its own id](pipe-run.md#host-supplied-run-context-storage_scope-and-analytics_groups), else a `403 ReadScopeForbidden`.++**The tree's refusals produce no verdict**, exactly as on `/v1/validate`: a key or entry the tree's rule refuses is a `422 InvalidMethodFiles` with its `tree_refusal` member, a hash the runner does not hold a `422 MissingBlobs`, a tree past its limits a `422 MethodTreeTooLarge`, and a tree holding no `.mthds` bundle of its own a `422 InvalidBundle`. A `files` member that is not a map, such as a list, is an `InvalidMethodFiles` too.++**Python follows the run routes' policy.** A projection runs nothing, but loading a method's library on a deployment that is not sandbox-hosted imports each of its `.py` files that declares a structure class (a `StructuredContent` subclass), which runs that file's module-level code in the server's process. So there a tree, sent or fetched, that holds any `.py` file is refused with a `403 CustomCodeRequiresSandbox` before anything is written. On a sandbox-hosted deployment, a tree whose Python declares a structure class is refused with a `403 MethodStructuresRefusedError`, as on a [run](pipe-run.md#sending-a-method-as-a-tree).++**A call by address resolves only from what was shipped.** A package the method calls by its address is loaded from the packages the request sends in `dependencies`, or that a fetched package ships under its `.mthds/methods/`, and from nowhere else: these routes never read a method installed on the runner and never fetch one. A call into a package nothing shipped is therefore the invalid verdict, one `unresolved_package_dependency` item per package, located on the pipe that calls it, whose message names it as `address-based dependency '<address>'`. The method is written out, loaded, projected and released in a worker thread, so a slow load never holds up the server's other requests.## Resolve@@ -27,10 +35,11 @@ Resolution is a first-class language operation alongside validation: the closure```json{- "files": [- { "content": "...bundle one...", "source": "main.mthds" },- { "content": "...bundle two...", "source": "steps.mthds" }- ]+ "files": {+ "main.mthds": "...bundle one...",+ "steps/extract.mthds": "...bundle two...",+ "templates/invoice.docx": { "sha256": "3e9b68b71cc667168399f0bccfb64b2d16314cdc0ed78fac4819aba0dda121a0", "size": 18342 }+ }}```@@ -71,7 +80,7 @@ An unknown `kind` or `target` is a request-shape `422` problem+json, never a `20```json{- "files": [{ "content": "...bundle..." }],+ "files": { "main.mthds": "...bundle..." },"kind": "types","target": "ts-zod"}diff --git a/docs/api-server/error-responses.md b/docs/api-server/error-responses.mdindex 41f3c35a3..436e081ea 100644--- a/docs/api-server/error-responses.md+++ b/docs/api-server/error-responses.md@@ -160,12 +160,12 @@ A few specific statuses bypass the domain mapping:- **400** — a well-formed request this deployment cannot serve. `error_type = "StartRequiresAsyncOrchestration"`: `POST /v1/start` is fire-and-forget by nature, and this deployment's orchestrator is blocking-only (the in-process `direct` default), so it refuses honestly rather than blocking and acking — use `POST /v1/execute` instead.- **400** — `error_type = "BadRequest"`: the client disconnected before its request body was received in full. The client is gone before the answer, so this one is seen in the server's logs, at warning level.- **401** — missing/invalid bearer token. `WWW-Authenticate: Bearer` is set. Only reachable when the deployment enables auth (`AUTH_MODE=api_key` or `AUTH_MODE=jwt`).-- **403** — authenticated but not authorized, or refused by deployment policy. The cases: a storage-ownership mismatch; `error_type = "OrchestrationModeOverrideForbidden"` — the request asked for an `orchestration_mode` this deployment does not allow overriding per request (`allow_request_orchestration_mode_override = false`); `error_type = "BlobUploadNotAuthorized"` — a `PUT /v1/blobs/{sha256}` without a valid upload target: its signature is missing, forged, expired, or signed for another hash or scope, so ask `POST /v1/blobs/missing` for a fresh target; `error_type = "CustomCodeRequiresSandbox"` — the method (a tree sent with the request, or a fetched `method_ref` package) holds custom Python `.py` files and this deployment is not sandbox-hosted, so running caller-supplied code in-process is refused; `error_type = "MethodStructuresRefusedError"` — a fetched `method_ref` package declares in-process Python structure classes (`StructuredContent` subclasses), which hosted execution always refuses, sandbox or not — express the types as MTHDS concepts with inline structures instead (see [Pipe Run](pipe-run.md#running-a-method-by-address-method_ref)).+- **403** — authenticated but not authorized, or refused by deployment policy. The cases: a storage-ownership mismatch; `error_type = "ReadScopeForbidden"` — on a `jwt` deployment, the request sent a `read_scope` that does not lie under the caller's own `user_id`, which is the only scope its token may read (see [Pipe Run](pipe-run.md#host-supplied-run-context-storage_scope-and-analytics_groups)); `error_type = "OrchestrationModeOverrideForbidden"` — the request asked for an `orchestration_mode` this deployment does not allow overriding per request (`allow_request_orchestration_mode_override = false`); `error_type = "BlobUploadNotAuthorized"` — a `PUT /v1/blobs/{sha256}` without a valid upload target: its signature is missing, forged, expired, or signed for another hash or scope, so ask `POST /v1/blobs/missing` for a fresh target; `error_type = "CustomCodeRequiresSandbox"` — the method (a tree sent with the request, or a fetched `method_ref` package) holds custom Python `.py` files and this deployment is not sandbox-hosted, so running caller-supplied code in-process is refused; `error_type = "MethodStructuresRefusedError"` — a fetched `method_ref` package declares in-process Python structure classes (`StructuredContent` subclasses), which hosted execution always refuses, sandbox or not — express the types as MTHDS concepts with inline structures instead (see [Pipe Run](pipe-run.md#running-a-method-by-address-method_ref)).- **404** — `error_type = "MethodPackageNotFoundError"`: the `method_ref` repository was fetched, but no package in it matches the requested address by manifest identity. The `detail` lists the package addresses the repository does contain, so the caller can correct the selector.- **409** — `error_type = "PipelineManagerAlreadyExistsError"`: the submitted `pipeline_run_id` is already registered for a run that is still in flight on this server. Completed and failed runs free their id, so this only fires for genuinely concurrent duplicates — resubmit after the in-flight run finishes, or pick a fresh id. Only `POST /v1/start` accepts a client-supplied `pipeline_run_id`, so only `/start` can produce it.- **413** — request body exceeds the configured size limit (`MAX_REQUEST_BODY_MIB`, 100 MiB by default), or a part of it exceeds its own: a `bundle_b64` archive over `MAX_BUNDLE_FILES` or `MAX_BUNDLE_TOTAL_KIB`, or `resources` over `MAX_RESOURCES_PER_REQUEST`, `MAX_RESOURCE_FILE_KIB` or `MAX_RESOURCES_TOTAL_KIB` (`error_type = "PayloadTooLarge"`).- **429** — an upstream inference provider rate-limited the run. `Retry-After` is set when the originating error carries `provider_metadata.retry_after_seconds`. Only `POST /v1/execute` runs inference, so only `/execute` can produce it.-- **501** — a request shape the published contract accepts but this server cannot serve. `error_type = "AsyncExecutionNotEnabledError"`: this deployment does not provide async pipeline execution (`POST /v1/start`). `error_type = "MethodRefNotSupported"`: the request selected its closure by a **registry-form** `method_ref` — not a `github.com/...` address — and no server-side method registry resolves those yet. Address-form references are fetched and resolved server-side (see [Pipe Run](pipe-run.md#running-a-method-by-address-method_ref)); use one of those, or submit inline `files[]`. Both 501s are permanent under the current deployment — do not retry.+- **501** — a request shape the published contract accepts but this server cannot serve. `error_type = "AsyncExecutionNotEnabledError"`: this deployment does not provide async pipeline execution (`POST /v1/start`). `error_type = "MethodRefNotSupported"`: the request selected its closure by a **registry-form** `method_ref` — not a `github.com/...` address — and no server-side method registry resolves those yet. Address-form references are fetched and resolved server-side (see [Pipe Run](pipe-run.md#running-a-method-by-address-method_ref)); use one of those, or send the method's tree in `files`. Both 501s are permanent under the current deployment — do not retry.The `method_ref` resolution errors are also mapped explicitly rather than through the domain table: `MethodRefParseError`, `MethodFetchError`, `MethodPackageAmbiguityError`, `MethodPackageTooLargeError` and `MethodPackageSymlinkError` are **422** (the caller can fix the reference, pin a tag, shrink the package, or remove the symlink it carries), `MethodPackageNotFoundError` is the **404** above, and `MethodStructuresRefusedError` is a **403**, whether the method was fetched by `method_ref` or sent as a tree. These pipelex `MethodRefError` subclasses do not classify an `error_domain`, so without the explicit mapping they would fall through to the 500 default — which would misreport caller-fixable failures as server faults.diff --git a/docs/api-server/index.md b/docs/api-server/index.mdindex be004e675..d9510cd5d 100644--- a/docs/api-server/index.md+++ b/docs/api-server/index.md@@ -164,7 +164,7 @@ Validate MTHDS content to ensure pipelines are correctly defined before executio### Pipe I/ORead a method's I/O artifacts without a validation. A Pipelex API extension carrying the standard's artifacts.-- `POST /v1/pipe-io` — The pipe I/O contracts, input form and output form of one pipe, or of every pipe with `all_pipes`, beside the method's entry pipe and its pending signatures; no dry run. Takes the same closure selector as `/v1/resolve`, and `include_files` echoes the closure's `.mthds` files+- `POST /v1/pipe-io` — The pipe I/O contracts, input form and output form of one pipe, or of every pipe with `all_pipes`, beside the method's entry pipe and its pending signatures; no dry run. Takes the method as a tree in `files` or by `method_ref`, as `/v1/resolve` and `/v1/validate` do, and `include_files` echoes the method's `.mthds` bundles by tree path[Learn more →](pipe-io.md)diff --git a/docs/api-server/openapi/pipelex-api.openapi.yaml b/docs/api-server/openapi/pipelex-api.openapi.yamlindex 23aae1dfc..81408383d 100644--- a/docs/api-server/openapi/pipelex-api.openapi.yaml+++ b/docs/api-server/openapi/pipelex-api.openapi.yaml@@ -470,7 +470,8 @@ paths:or `MethodStructuresRefusedError` — the method''s Python declares structure classes (`StructuredContent` subclasses),whether it arrived as a bundle (`files` or `bundle_b64`) or as a fetched `method_ref` package: a sandbox-hosteddeployment imports no caller Python into its own process, so it refuses them before loading anything (express- the types as MTHDS concepts instead).'+ the types as MTHDS concepts instead); or `ReadScopeForbidden` — on a `jwt` deployment, the request sent a `read_scope`+ that does not lie under the caller''s own `user_id`.'content:application/problem+json:schema:@@ -829,7 +830,8 @@ paths:or `MethodStructuresRefusedError` — the method''s Python declares structure classes (`StructuredContent` subclasses),whether it arrived as a bundle (`files` or `bundle_b64`) or as a fetched `method_ref` package: a sandbox-hosteddeployment imports no caller Python into its own process, so it refuses them before loading anything (express- the types as MTHDS concepts instead).'+ the types as MTHDS concepts instead); or `ReadScopeForbidden` — on a `jwt` deployment, the request sent a `read_scope`+ that does not lie under the caller''s own `user_id`.'content:application/problem+json:schema:@@ -955,7 +957,8 @@ paths:or `MethodStructuresRefusedError` — the method''s Python declares structure classes (`StructuredContent` subclasses),whether it arrived as a bundle (`files` or `bundle_b64`) or as a fetched `method_ref` package: a sandbox-hosteddeployment imports no caller Python into its own process, so it refuses them before loading anything (express- the types as MTHDS concepts instead).'+ the types as MTHDS concepts instead); or `ReadScopeForbidden` — on a `jwt` deployment, the request sent a `read_scope`+ that does not lie under the caller''s own `user_id`.'content:application/problem+json:schema:@@ -973,22 +976,25 @@ paths:tags:- resolvesummary: Resolve Mthds- description: "Resolve a library closure into its normalized crate (Pipelex API extension).\n\nResolution is a first-class\- \ language operation alongside validation: assemble the closure from\nthe inline `files[]`, load + statically validate\- \ the library, and emit the **normalized\nlibrary crate** (fully qualified refs, refinement flattened, natives materialized,\- \ fingerprint\nset) — the artifact shape the MTHDS standard specifies as the Library Crate Format. It runs no\ndry-run\- \ sweep — runnability is `/validate`'s vocabulary.\n\nResponse contract (the `/validate` discipline):\n\n- **Valid\- \ verdict (200, `is_valid: true`):** the crate on the valid arm.\n- **Invalid verdict (200, `is_valid: false`):**\- \ the library could not be parsed, loaded, or\n validated — `validation_errors[]` from pipelex's one shared builder.\n\- - **No verdict (non-2xx):** a malformed request body (neither/both closure selectors, an\n over-limit file) is a\- \ request-shape 422; `method_ref` is a 501 until server-side method\n registry resolution exists; auth is 401/403;\- \ server fault is 5xx. All RFC 7807\n `application/problem+json` via the global handlers."+ description: "Resolve a method into its normalized crate (Pipelex API extension).\n\nResolution is a first-class language\+ \ operation alongside validation: read the method's tree\n(`files`, or the package a `method_ref` fetches) exactly\+ \ as `/validate` reads it, load + statically\nvalidate it as the caller's own library, and emit the **normalized library\+ \ crate** (fully\nqualified refs, refinement flattened, natives materialized, fingerprint set) — the artifact\nshape\+ \ the MTHDS standard specifies as the Library Crate Format. It runs no dry-run sweep —\nrunnability is `/validate`'s\+ \ vocabulary.\n\nResponse contract (the `/validate` discipline):\n\n- **Valid verdict (200, `is_valid: true`):** the\+ \ crate on the valid arm.\n- **Invalid verdict (200, `is_valid: false`):** the library could not be parsed, loaded,\+ \ or\n validated — `validation_errors[]` from pipelex's one shared builder.\n- **No verdict (non-2xx):** a malformed\+ \ request body (neither or both method selectors) is a\n request-shape 422; a `files` map that is not a method tree\+ \ is a 422 `InvalidMethodFiles`, and\n hash entries this runner does not hold one 422 `MissingBlobs`; custom Python\+ \ on a deployment\n that is not sandbox-hosted, or a Python structure class, is a 403; a registry-form `method_ref`\n\+ \ is a 501 until server-side method registry resolution exists; auth is 401/403; server fault is\n 5xx. All RFC\+ \ 7807 `application/problem+json` via the global handlers."operationId: resolve_mthds_v1_resolve_postrequestBody:content:application/json:schema:- $ref: '#/components/schemas/MthdsFilesRequest'+ $ref: '#/components/schemas/MethodTreeRequest'required: trueresponses:'200':@@ -1038,6 +1044,17 @@ paths:application/problem+json:schema:$ref: '#/components/schemas/ProblemDocument'+ '403':+ description: 'The run routes'' Python policy, applied to the method''s tree: `CustomCodeRequiresSandbox` — the method+ (sent in `files`, or a fetched `method_ref` package) ships custom Python and this deployment is not sandbox-hosted;+ or `MethodStructuresRefusedError` — the method''s Python declares structure classes (`StructuredContent` subclasses),+ which a sandbox-hosted deployment never imports into its own process (express the types as MTHDS concepts instead).+ Also `ReadScopeForbidden` — on a `jwt` deployment, the request sent a `read_scope` that does not lie under the+ caller''s own `user_id`.'+ content:+ application/problem+json:+ schema:+ $ref: '#/components/schemas/ProblemDocument''404':description: '`MethodPackageNotFoundError` — the `method_ref` repository was fetched, but no package in it matchesthe requested address by manifest identity. The message lists the packages the repository does contain.'@@ -1046,9 +1063,9 @@ paths:schema:$ref: '#/components/schemas/ProblemDocument''501':- description: '`MethodRefNotSupported` — the request selected its closure by a **registry-form** `method_ref` (not+ description: '`MethodRefNotSupported` — the request selected its method by a **registry-form** `method_ref` (nota `github.com/...` address), and no server-side method registry resolves those yet. Use an address-form reference- or submit inline `files[]` instead.'+ or send the method''s tree in `files` instead.'content:application/problem+json:schema:@@ -1059,18 +1076,21 @@ paths:- pipe-iosummary: Pipe Iodescription: "Return a method's pipe I/O contracts, input form and output form, with no dry run (Pipelex API extension).\n\- \nThe closure resolves through the static core `/resolve` rides, a pipe is selected the way the\nper-pipe routes select\- \ one, and the three artifacts are derived with `build_pipe_io_artifacts`,\nthe builder `/validate` and every run\- \ call — so a call costs one load and one derivation where\n`/validate` dry-runs every pipe.\n\nResponse contract\- \ (the `/validate` discipline):\n\n- **Valid verdict (200, `is_valid: true`):** the three artifact maps keyed by qualified\- \ `pipe_ref`,\n beside `pipe_ref`, `default_pipe_ref`, `pending_signatures`, `is_runnable`, and `files` when\n `include_files`\- \ is true.\n- **Invalid verdict (200, `is_valid: false`):** the closure could not be parsed, loaded, or\n statically\- \ validated — the crate verdict, carrying no artifact, no selection and no files.\n- **No verdict (non-2xx):** a malformed\- \ body and neither or both closure selectors are\n request-shape 422s (`ValidationError`); a selection refusal is\- \ an input 422 named for its failure,\n `EntryPipeNotFoundError` for an unknown `pipe_ref` or a single-pipe request\- \ whose chain finds no\n entry pipe, `EntryPipeAmbiguousError` for an ambiguous one or a chain that finds several;\- \ the\n `method_ref` fetch outcomes are those of `/resolve` (404, 422, 501);\n an artifact that cannot be derived\- \ (`PipeIOContractError`) is a 500. All RFC 7807\n `application/problem+json` via the global handlers."+ \nThe method is read as `/validate` reads a method tree and resolves through the static core\n`/resolve` rides, a\+ \ pipe is selected the way the per-pipe routes select one, and the three\nartifacts are derived with `build_pipe_io_artifacts`,\+ \ the builder `/validate` and every run call —\nso a call costs one load and one derivation where `/validate` dry-runs\+ \ every pipe.\n\nResponse contract (the `/validate` discipline):\n\n- **Valid verdict (200, `is_valid: true`):** the\+ \ three artifact maps keyed by qualified `pipe_ref`,\n beside `pipe_ref`, `default_pipe_ref`, `pending_signatures`,\+ \ `is_runnable`, and `files` when\n `include_files` is true.\n- **Invalid verdict (200, `is_valid: false`):** the\+ \ closure could not be parsed, loaded, or\n statically validated — the crate verdict, carrying no artifact, no selection\+ \ and no files.\n- **No verdict (non-2xx):** a malformed body and neither or both method selectors are\n request-shape\+ \ 422s (`ValidationError`); a `files` map that is not a method tree is a 422\n `InvalidMethodFiles`, and hash entries\+ \ this runner does not hold one 422 `MissingBlobs`, as on\n `/validate`; custom Python on a deployment that is not\+ \ sandbox-hosted, or a Python structure\n class, is a 403; a selection refusal is an input 422 named for its failure,\n\+ \ `EntryPipeNotFoundError` for an unknown `pipe_ref` or a single-pipe request whose chain finds no\n entry pipe,\+ \ `EntryPipeAmbiguousError` for an ambiguous one or a chain that finds several; the\n `method_ref` fetch outcomes\+ \ are those of `/resolve` (404, 422, 501);\n an artifact that cannot be derived (`PipeIOContractError`) is a 500.\+ \ All RFC 7807\n `application/problem+json` via the global handlers."operationId: pipe_io_v1_pipe_io_postrequestBody:content:@@ -1126,6 +1146,17 @@ paths:application/problem+json:schema:$ref: '#/components/schemas/ProblemDocument'+ '403':+ description: 'The run routes'' Python policy, applied to the method''s tree: `CustomCodeRequiresSandbox` — the method+ (sent in `files`, or a fetched `method_ref` package) ships custom Python and this deployment is not sandbox-hosted;+ or `MethodStructuresRefusedError` — the method''s Python declares structure classes (`StructuredContent` subclasses),+ which a sandbox-hosted deployment never imports into its own process (express the types as MTHDS concepts instead).+ Also `ReadScopeForbidden` — on a `jwt` deployment, the request sent a `read_scope` that does not lie under the+ caller''s own `user_id`.'+ content:+ application/problem+json:+ schema:+ $ref: '#/components/schemas/ProblemDocument''404':description: '`MethodPackageNotFoundError` — the `method_ref` repository was fetched, but no package in it matchesthe requested address by manifest identity. The message lists the packages the repository does contain.'@@ -1134,9 +1165,9 @@ paths:schema:$ref: '#/components/schemas/ProblemDocument''501':- description: '`MethodRefNotSupported` — the request selected its closure by a **registry-form** `method_ref` (not+ description: '`MethodRefNotSupported` — the request selected its method by a **registry-form** `method_ref` (nota `github.com/...` address), and no server-side method registry resolves those yet. Use an address-form reference- or submit inline `files[]` instead.'+ or send the method''s tree in `files` instead.'content:application/problem+json:schema:@@ -1146,16 +1177,17 @@ paths:tags:- codegensummary: Codegen Mthds- description: "Generate typed artifacts from a library closure (Pipelex API extension).\n\nResolves the closure to its\- \ normalized crate (exactly like `POST /resolve`), then projects it\nthrough the requested `kind`/`target` axes and\- \ returns the **stamped** artifact set plus its\n`codegen.lock` — everything a client needs to materialize a byte-identical\- \ local projection\nand run the offline drift check. There is deliberately **no** server-side check route: the\ncheck\- \ is offline by design.\n\nResponse contract (the `/validate` discipline):\n\n- **Valid verdict (200, `is_valid: true`):**\+ description: "Generate typed artifacts from a method (Pipelex API extension).\n\nResolves the method to its normalized\+ \ crate (exactly like `POST /resolve`), then projects it\nthrough the requested `kind`/`target` axes and returns the\+ \ **stamped** artifact set plus its\n`codegen.lock` — everything a client needs to materialize a byte-identical local\+ \ projection\nand run the offline drift check. There is deliberately **no** server-side check route: the\ncheck is\+ \ offline by design.\n\nResponse contract (the `/validate` discipline):\n\n- **Valid verdict (200, `is_valid: true`):**\\ the artifacts + lock on the valid arm.\n- **Invalid verdict (200, `is_valid: false`):** the library could not be\\ parsed, loaded, or\n validated — `validation_errors[]` from pipelex's one shared builder; no artifacts exist.\n\- **No verdict (non-2xx):** an unknown projection `kind`/`target`, a `pipe_ref` on a\n concept-set-wide kind, or\- \ a malformed closure selector is a request-shape 422 problem+json;\n `method_ref` is a 501 until server-side method\- \ registry resolution exists; auth is 401/403;\n server fault is 5xx."+ \ a malformed method selector is a request-shape 422 problem+json; the\n method's tree is refused as `/resolve` refuses\+ \ it (422 `InvalidMethodFiles` or `MissingBlobs`,\n the Python policy's 403s, the `method_ref` fetch outcomes, 501\+ \ for a registry-form\n `method_ref`); auth is 401/403; server fault is 5xx."operationId: codegen_mthds_v1_codegen_postrequestBody:content:@@ -1211,6 +1243,17 @@ paths:application/problem+json:schema:$ref: '#/components/schemas/ProblemDocument'+ '403':+ description: 'The run routes'' Python policy, applied to the method''s tree: `CustomCodeRequiresSandbox` — the method+ (sent in `files`, or a fetched `method_ref` package) ships custom Python and this deployment is not sandbox-hosted;+ or `MethodStructuresRefusedError` — the method''s Python declares structure classes (`StructuredContent` subclasses),+ which a sandbox-hosted deployment never imports into its own process (express the types as MTHDS concepts instead).+ Also `ReadScopeForbidden` — on a `jwt` deployment, the request sent a `read_scope` that does not lie under the+ caller''s own `user_id`.'+ content:+ application/problem+json:+ schema:+ $ref: '#/components/schemas/ProblemDocument''404':description: '`MethodPackageNotFoundError` — the `method_ref` repository was fetched, but no package in it matchesthe requested address by manifest identity. The message lists the packages the repository does contain.'@@ -1219,9 +1262,9 @@ paths:schema:$ref: '#/components/schemas/ProblemDocument''501':- description: '`MethodRefNotSupported` — the request selected its closure by a **registry-form** `method_ref` (not+ description: '`MethodRefNotSupported` — the request selected its method by a **registry-form** `method_ref` (nota `github.com/...` address), and no server-side method registry resolves those yet. Use an address-form reference- or submit inline `files[]` instead.'+ or send the method''s tree in `files` instead.'content:application/problem+json:schema:@@ -1589,6 +1632,13 @@ paths:application/problem+json:schema:$ref: '#/components/schemas/ProblemDocument'+ '403':+ description: '`ReadScopeForbidden` — on a `jwt` deployment, the request sent a `read_scope` that does not lie under+ the caller''s own `user_id`, the only scope its token may read.'+ content:+ application/problem+json:+ schema:+ $ref: '#/components/schemas/ProblemDocument'x-mthds-protocol: true/:get:@@ -1898,14 +1948,41 @@ components:properties:files:anyOf:- - items:- $ref: '#/components/schemas/MthdsFileItem'- type: array- maxItems: 16- minItems: 1+ - additionalProperties:+ anyOf:+ - type: string+ - $ref: '#/components/schemas/Base64FileEntry'+ - $ref: '#/components/schemas/HashFileEntry'+ type: object- type: 'null'title: Files- description: Inline MTHDS bundles forming the closure to resolve (content-passing — no server-side path reads).+ description: 'The method as a map of file entries, the MTHDS standard''s form of a method tree that travels: each+ key is a tree path (relative, forward slashes, in Unicode NFC, no `.` or `..` segment, no segment starting with+ a dot), and each value is the file''s UTF-8 text, `{"base64": ...}` holding its bytes, or `{"sha256": ..., "size":+ ...}` naming bytes this runner already holds (uploaded with `POST /v1/blobs/missing` then `PUT /v1/blobs/{sha256}`).+ The map is read from the request''s JSON text: a key given twice, a key the membership rule keeps out of every+ tree (`.mthds/` included), colliding keys and a malformed entry are a 422 `InvalidMethodFiles` whose `tree_refusal`+ names the code and the paths, and hash entries the runner does not hold are one 422 `MissingBlobs` whose `missing_blobs`+ lists every such hash. Its `.mthds` files are the method''s bundles and every other file is one a step may read,+ such as a `PipeDocGen` template or a `PipeFunc` module. With no `pipe_code`, the run starts at the `main_pipe`+ of the `METHODS.toml` at the tree''s root, or else at the one `main_pipe` its bundles declare. Custom `.py` is+ only honored on a sandbox-hosted deployment. Exactly one of `files` / `method_ref`.'+ dependencies:+ anyOf:+ - additionalProperties:+ additionalProperties:+ anyOf:+ - type: string+ - $ref: '#/components/schemas/Base64FileEntry'+ - $ref: '#/components/schemas/HashFileEntry'+ type: object+ type: object+ - type: 'null'+ title: Dependencies+ description: 'PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — the method packages the method calls by address+ that this runner cannot fetch, such as one an author has not published, as `{"<address>": <map of file entries>}`:+ each value is a tree of its own, read as `files` is, with a `METHODS.toml` at its root declaring the address it+ is keyed by. They are searched before the runner''s installed copies, for this request alone. Accompanies `files`.'method_ref:anyOf:- type: string@@ -1913,8 +1990,10 @@ components:minLength: 1- type: 'null'title: Method Ref- description: Reference to a published method, resolving to its package's `.mthds` files. Address form (`github.com/<owner>/<repo>[/<selector>][@<tag>]`)- is fetched and resolved server-side; the registry form stays reserved (501 until a method registry exists).+ description: PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — a published method by reference, instead of+ its tree. Address form (`github.com/<owner>/<repo>[/<selector>][@<tag>]`) is fetched and read as a `files` tree+ is, through the same path as a `method_ref` run; the registry form stays reserved (501 until a method registry+ exists). Exactly one of `files` / `method_ref`.resources:anyOf:- additionalProperties:@@ -1932,6 +2011,15 @@ components:32 files, 5120 KiB each and 10240 KiB in all, decoded; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`,`MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`. The files make one method tree with the bundles, whichthe run names by its digest: no file''s bytes travel with the run, each being read where a step uses it.'+ read_scope:+ anyOf:+ - type: string+ - type: 'null'+ title: Read Scope+ description: 'PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — the scope the request reads the method under,+ as a run''s `read_scope`: a multi-tenant host sends the one of the organization it serves. A hash entry of `files`+ resolves in that scope''s blob store, and the tree the request read is recorded there, so a later request can+ send it by hash. Omit it and a hash entry resolves in the caller''s own id''s store, and nothing is recorded.'kind:$ref: '#/components/schemas/CodegenRouteKind'description: What to project. `types` projects the crate's concept set into typed models.@@ -1951,7 +2039,7 @@ components:- kind- targettitle: CodegenRequest- description: 'The codegen request: the shared closure selector plus the two explicit projection axes.'+ description: 'The codegen request: the crate routes'' method selector plus the two explicit projection axes.'CodegenRouteKind:type: stringenum:@@ -4021,6 +4109,103 @@ components:- commit_shatitle: MethodProvenancedescription: 'What a fetched-method run must record: address, tag, and the resolved commit SHA.'+ MethodTreeRequest:+ properties:+ files:+ anyOf:+ - additionalProperties:+ anyOf:+ - type: string+ - $ref: '#/components/schemas/Base64FileEntry'+ - $ref: '#/components/schemas/HashFileEntry'+ type: object+ - type: 'null'+ title: Files+ description: 'The method as a map of file entries, the MTHDS standard''s form of a method tree that travels: each+ key is a tree path (relative, forward slashes, in Unicode NFC, no `.` or `..` segment, no segment starting with+ a dot), and each value is the file''s UTF-8 text, `{"base64": ...}` holding its bytes, or `{"sha256": ..., "size":+ ...}` naming bytes this runner already holds (uploaded with `POST /v1/blobs/missing` then `PUT /v1/blobs/{sha256}`).+ The map is read from the request''s JSON text: a key given twice, a key the membership rule keeps out of every+ tree (`.mthds/` included), colliding keys and a malformed entry are a 422 `InvalidMethodFiles` whose `tree_refusal`+ names the code and the paths, and hash entries the runner does not hold are one 422 `MissingBlobs` whose `missing_blobs`+ lists every such hash. Its `.mthds` files are the method''s bundles and every other file is one a step may read,+ such as a `PipeDocGen` template or a `PipeFunc` module. With no `pipe_code`, the run starts at the `main_pipe`+ of the `METHODS.toml` at the tree''s root, or else at the one `main_pipe` its bundles declare. Custom `.py` is+ only honored on a sandbox-hosted deployment. Exactly one of `files` / `method_ref`.'+ dependencies:+ anyOf:+ - additionalProperties:+ additionalProperties:+ anyOf:+ - type: string+ - $ref: '#/components/schemas/Base64FileEntry'+ - $ref: '#/components/schemas/HashFileEntry'+ type: object+ type: object+ - type: 'null'+ title: Dependencies+ description: 'PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — the method packages the method calls by address+ that this runner cannot fetch, such as one an author has not published, as `{"<address>": <map of file entries>}`:+ each value is a tree of its own, read as `files` is, with a `METHODS.toml` at its root declaring the address it+ is keyed by. They are searched before the runner''s installed copies, for this request alone. Accompanies `files`.'+ method_ref:+ anyOf:+ - type: string+ maxLength: 512+ minLength: 1+ - type: 'null'+ title: Method Ref+ description: PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — a published method by reference, instead of+ its tree. Address form (`github.com/<owner>/<repo>[/<selector>][@<tag>]`) is fetched and read as a `files` tree+ is, through the same path as a `method_ref` run; the registry form stays reserved (501 until a method registry+ exists). Exactly one of `files` / `method_ref`.+ resources:+ anyOf:+ - additionalProperties:+ type: string+ type: object+ - type: 'null'+ title: Resources+ description: 'The files submitted with the bundle that its steps read, such as a PipeDocGen step''s `template_file`+ (an `.xlsx`, `.docx`, `.pptx` or `.html` template): a JSON object mapping each file''s bundle-relative path, exactly+ as the step names it, to the file''s bytes in standard base64. A path must be relative and written with forward+ slashes, with no `..` segment, backslash or colon. Beside inline `mthds_contents`, every bundle of the request+ is read as sitting in one directory, and every file must be named by a step, or the request is refused with a+ 422 `BundleResourceNotNamedError`. Beside a method tree, `files` or `bundle_b64`, each file joins the tree at+ its path from the tree''s root, where a file the tree already holds must have the same bytes. By default at most+ 32 files, 5120 KiB each and 10240 KiB in all, decoded; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`,+ `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`. The files make one method tree with the bundles, which+ the run names by its digest: no file''s bytes travel with the run, each being read where a step uses it.'+ read_scope:+ anyOf:+ - type: string+ - type: 'null'+ title: Read Scope+ description: 'PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — the scope the request reads the method under,+ as a run''s `read_scope`: a multi-tenant host sends the one of the organization it serves. A hash entry of `files`+ resolves in that scope''s blob store, and the tree the request read is recorded there, so a later request can+ send it by hash. Omit it and a hash entry resolves in the caller''s own id''s store, and nothing is recorded.'+ type: object+ title: MethodTreeRequest+ description: 'The method selector of the crate routes (`/resolve`, `/codegen`, `/pipe-io`): a tree sent in `files` XOR+ a `method_ref`.+++ Exactly one of the two must be provided — both or neither is a request-shape 422. The tree is the MTHDS standard''s++ map of file entries, read from the request''s raw JSON text exactly as `/validate` and the run routes read it++ (`pipelex_api.method_tree_request`), with the `dependencies` it carries and the `resources` that join it. An++ **address-form** `method_ref` (`github.com/<owner>/<repo>[/<selector>][@<tag>]`) is resolved by this server through++ the run routes'' fetch path: the package''s tree is read from its commit''s git objects, with the packages it ships++ under `.mthds/methods/`. The **registry form** (any non-address reference) stays reserved and answers 501 until++ server-side method-registry resolution exists. Either way the method is held to the run routes'' Python policy:++ custom `.py` only on a sandbox-hosted deployment, and no Python structure class at all.'MissingBlob:properties:sha256:@@ -4337,86 +4522,6 @@ components:is preserved within a parent and never across parents — see ``docs/migration-ledger.md``for the placement rule and for what happens to a moved table''s introducing comment.'- MthdsFileItem:- properties:- content:- type: string- title: Content- description: MTHDS bundle content.- source:- anyOf:- - type: string- maxLength: 1024- - type: 'null'- title: Source- description: Optional logical source for this bundle (e.g. its path relative to the submitted directory). Threaded- onto the blueprint so server-side diagnostics carry a `source` pointing at the owning file.- type: object- required:- - content- title: MthdsFileItem- description: 'One inline MTHDS bundle: its content plus an optional logical source for diagnostics.--- The `files[]` envelope pairs each content with its source in one entry (the shape the codegen-- spec pins for the resolve/codegen routes), unlike the legacy parallel-- `mthds_contents[]`/`mthds_sources[]` lists on `/validate`.'- MthdsFilesRequest:- properties:- files:- anyOf:- - items:- $ref: '#/components/schemas/MthdsFileItem'- type: array- maxItems: 16- minItems: 1- - type: 'null'- title: Files- description: Inline MTHDS bundles forming the closure to resolve (content-passing — no server-side path reads).- method_ref:- anyOf:- - type: string- maxLength: 512- minLength: 1- - type: 'null'- title: Method Ref- description: Reference to a published method, resolving to its package's `.mthds` files. Address form (`github.com/<owner>/<repo>[/<selector>][@<tag>]`)- is fetched and resolved server-side; the registry form stays reserved (501 until a method registry exists).- resources:- anyOf:- - additionalProperties:- type: string- type: object- - type: 'null'- title: Resources- description: 'The files submitted with the bundle that its steps read, such as a PipeDocGen step''s `template_file`- (an `.xlsx`, `.docx`, `.pptx` or `.html` template): a JSON object mapping each file''s bundle-relative path, exactly- as the step names it, to the file''s bytes in standard base64. A path must be relative and written with forward- slashes, with no `..` segment, backslash or colon. Beside inline `mthds_contents`, every bundle of the request- is read as sitting in one directory, and every file must be named by a step, or the request is refused with a- 422 `BundleResourceNotNamedError`. Beside a method tree, `files` or `bundle_b64`, each file joins the tree at- its path from the tree''s root, where a file the tree already holds must have the same bytes. By default at most- 32 files, 5120 KiB each and 10240 KiB in all, decoded; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`,- `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`. The files make one method tree with the bundles, which- the run names by its digest: no file''s bytes travel with the run, each being read where a step uses it.'- type: object- title: MthdsFilesRequest- description: 'Shared closure selector for the resolve/codegen routes: inline `files[]` XOR a `method_ref`.--- Exactly one of the two must be provided (spec''d envelope) — both or neither is a request-shape-- 422. An **address-form** `method_ref` (`github.com/<owner>/<repo>[/<selector>][@<tag>]`) is-- resolved by this server: the repository is fetched at the tag, the package is located by-- manifest identity, and its `.mthds` files feed the closure with their real relative paths as-- per-file sources. The **registry form** (any non-address reference) stays reserved and answers-- 501 until server-side method-registry resolution exists.'NodeIOSpec:properties:inputs:@@ -5446,14 +5551,41 @@ components:properties:files:anyOf:- - items:- $ref: '#/components/schemas/MthdsFileItem'- type: array- maxItems: 16- minItems: 1+ - additionalProperties:+ anyOf:+ - type: string+ - $ref: '#/components/schemas/Base64FileEntry'+ - $ref: '#/components/schemas/HashFileEntry'+ type: object- type: 'null'title: Files- description: Inline MTHDS bundles forming the closure to resolve (content-passing — no server-side path reads).+ description: 'The method as a map of file entries, the MTHDS standard''s form of a method tree that travels: each+ key is a tree path (relative, forward slashes, in Unicode NFC, no `.` or `..` segment, no segment starting with+ a dot), and each value is the file''s UTF-8 text, `{"base64": ...}` holding its bytes, or `{"sha256": ..., "size":+ ...}` naming bytes this runner already holds (uploaded with `POST /v1/blobs/missing` then `PUT /v1/blobs/{sha256}`).+ The map is read from the request''s JSON text: a key given twice, a key the membership rule keeps out of every+ tree (`.mthds/` included), colliding keys and a malformed entry are a 422 `InvalidMethodFiles` whose `tree_refusal`+ names the code and the paths, and hash entries the runner does not hold are one 422 `MissingBlobs` whose `missing_blobs`+ lists every such hash. Its `.mthds` files are the method''s bundles and every other file is one a step may read,+ such as a `PipeDocGen` template or a `PipeFunc` module. With no `pipe_code`, the run starts at the `main_pipe`+ of the `METHODS.toml` at the tree''s root, or else at the one `main_pipe` its bundles declare. Custom `.py` is+ only honored on a sandbox-hosted deployment. Exactly one of `files` / `method_ref`.'+ dependencies:+ anyOf:+ - additionalProperties:+ additionalProperties:+ anyOf:+ - type: string+ - $ref: '#/components/schemas/Base64FileEntry'+ - $ref: '#/components/schemas/HashFileEntry'+ type: object+ type: object+ - type: 'null'+ title: Dependencies+ description: 'PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — the method packages the method calls by address+ that this runner cannot fetch, such as one an author has not published, as `{"<address>": <map of file entries>}`:+ each value is a tree of its own, read as `files` is, with a `METHODS.toml` at its root declaring the address it+ is keyed by. They are searched before the runner''s installed copies, for this request alone. Accompanies `files`.'method_ref:anyOf:- type: string@@ -5461,8 +5593,10 @@ components:minLength: 1- type: 'null'title: Method Ref- description: Reference to a published method, resolving to its package's `.mthds` files. Address form (`github.com/<owner>/<repo>[/<selector>][@<tag>]`)- is fetched and resolved server-side; the registry form stays reserved (501 until a method registry exists).+ description: PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — a published method by reference, instead of+ its tree. Address form (`github.com/<owner>/<repo>[/<selector>][@<tag>]`) is fetched and read as a `files` tree+ is, through the same path as a `method_ref` run; the registry form stays reserved (501 until a method registry+ exists). Exactly one of `files` / `method_ref`.resources:anyOf:- additionalProperties:@@ -5480,6 +5614,15 @@ components:32 files, 5120 KiB each and 10240 KiB in all, decoded; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`,`MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`. The files make one method tree with the bundles, whichthe run names by its digest: no file''s bytes travel with the run, each being read where a step uses it.'+ read_scope:+ anyOf:+ - type: string+ - type: 'null'+ title: Read Scope+ description: 'PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — the scope the request reads the method under,+ as a run''s `read_scope`: a multi-tenant host sends the one of the organization it serves. A hash entry of `files`+ resolves in that scope''s blob store, and the tree the request read is recorded there, so a later request can+ send it by hash. Omit it and a hash entry resolves in the caller''s own id''s store, and nothing is recorded.'pipe_ref:anyOf:- type: string@@ -5487,9 +5630,9 @@ components:minLength: 1- type: 'null'title: Pipe Ref- description: Qualified pipe ref (`domain.pipe_code`) to select. Optional — defaults to the fetched package manifest's- `main_pipe` on a `method_ref` request, else to the closure's declared `main_pipe`; a closure declaring none, or- several, with no manifest `main_pipe` to settle it, requires it unless `all_pipes` is set.+ description: Qualified pipe ref (`domain.pipe_code`) to select. Optional — defaults to the `main_pipe` of the `METHODS.toml`+ at the method tree's root, else to the closure's declared `main_pipe`; a closure declaring none, or several, with+ no manifest `main_pipe` to settle it, requires it unless `all_pipes` is set.all_pipes:type: booleantitle: All Pipes@@ -5500,13 +5643,14 @@ components:include_files:type: booleantitle: Include Files- description: 'Echo the resolved closure''s `.mthds` files on the valid arm, in the request''s own `files[]` shape:- the request''s files for inline `files[]`, the fetched package''s `.mthds` files under their package-relative- paths for a `method_ref`. Absent from the response unless true.'+ description: 'Echo the method''s `.mthds` bundles on the valid arm, as a map of file entries in the request''s own+ `files` shape, each bundle''s tree path to its text: the bundles of the tree sent in `files`, whatever entry carried+ them, or those of the fetched package''s tree for a `method_ref`. Its other files and its dependencies are not+ echoed. Absent from the response unless true.'default: falsetype: objecttitle: PipeIORequest- description: 'The pipe I/O request: the crate routes'' closure selector and pipe selector, plus the whole-method and+ description: 'The pipe I/O request: the crate routes'' method selector and pipe selector, plus the whole-method andecho opt-ins.@@ -5567,12 +5711,12 @@ components:title: Is Runnabledescription: '`not pending_signatures`, exactly as on `/validate`. No dry run backs it.'files:- items:- $ref: '#/components/schemas/MthdsFileItem'- type: array+ additionalProperties:+ type: string+ type: objecttitle: Files- description: 'The resolved closure''s `.mthds` files, in the request''s `files[]` shape. Present only with `include_files:- true`.'+ description: 'The method''s `.mthds` bundles, each tree path to its text, a map of file entries as the request''s+ `files` is. Present only with `include_files: true`.'type: objectrequired:- pipe_refdiff --git a/docs/api-server/pipe-io.md b/docs/api-server/pipe-io.mdindex a00c612f2..4a1623c31 100644--- a/docs/api-server/pipe-io.md+++ b/docs/api-server/pipe-io.md@@ -12,27 +12,35 @@ It speaks the `/v1/validate` verdict discipline: a **produced verdict is always## Request-The body is the [crate routes' closure selector](codegen.md#selecting-the-closure) plus a pipe selector and two opt-ins:+The body names the method as the [crate routes](codegen.md#selecting-the-method) do, a method tree read exactly as [`/v1/validate`](pipe-validate.md#validating-a-method-tree) and the run routes read it, plus a pipe selector and two opt-ins:-- `files` (list) **or** `method_ref` (string), exactly one: the closure, as on `/v1/resolve`. Neither or both is a request-shape `422`.+- `files` (object) **or** `method_ref` (string), exactly one: the method, as on `/v1/resolve`. `files` is the MTHDS protocol's map of file entries, each key a tree path and each value the file's text, `{"base64": "…"}` or `{"sha256": "…", "size": n}`; `method_ref` an address-form reference to a published package, whose whole tree is fetched. Neither or both is a request-shape `422`.+- `dependencies` (object, optional): the method packages the method calls by address, each a map of file entries keyed by its address, as on a [run](pipe-run.md#sending-the-method-packages-the-method-calls). It accompanies `files` only.- `pipe_ref` (string, optional): the qualified ref `domain.pipe_code` of the pipe to describe. Omitted, the selection chain below decides.- `all_pipes` (boolean, default `false`): describe every pipe the closure loads instead of the selected one.-- `include_files` (boolean, default `false`): echo the resolved closure's `.mthds` files on the valid arm.-- `resources` (object, optional): the files the closure's steps read, such as a `PipeDocGen` step's `template_file`, keyed by the path the step names, each in standard base64, as on [`/v1/validate`](pipe-validate.md#sending-the-files-the-steps-read-resources). It accompanies `files` only. A string-loaded step whose template file was not sent makes the closure invalid.+- `include_files` (boolean, default `false`): echo the text of the method's own `.mthds` bundles on the valid arm.+- `resources` (object, optional): files the method's steps read, such as a `PipeDocGen` step's `template_file`, keyed by the path from the tree's root, each in standard base64, joining the tree as on [`/v1/validate`](pipe-validate.md#sending-the-files-the-steps-read-resources). It accompanies `files` only. A step whose template file the tree does not hold makes the method invalid.+- `read_scope` (string, optional): **Pipelex-API extension.** The scope the request reads the method under, as on `/v1/validate`: a hash entry of `files` resolves in that scope's blob store, and only a request carrying it records the tree it read there; [a `jwt` caller may send only a scope under its own id](pipe-run.md#host-supplied-run-context-storage_scope-and-analytics_groups), else a `403 ReadScopeForbidden`.-There is no `views` field, because the valid arm always carries all three artifacts. The hosted API also accepts a catalog `method_id`, which the platform resolves into `files[]` before forwarding, so this server never declares it.+There is no `views` field, because the valid arm always carries all three artifacts. The hosted API also accepts a catalog `method_id`, which the platform resolves into the method's `files` before forwarding, so this server never declares it.```json{- "files": [{ "content": "...bundle...", "source": "main.mthds" }],+ "files": {+ "METHODS.toml": "...manifest...",+ "main.mthds": "...bundle...",+ "templates/invoice.docx": { "base64": "UEsDBBQABgAIAAAAIQ..." }+ },"all_pipes": true,"include_files": true}```+The tree's refusals produce no verdict, as on `/v1/validate`: an `InvalidMethodFiles`, `MissingBlobs`, `MethodTreeTooLarge` or `InvalidBundle` `422`, and the custom-Python policy's `403`s (see [No verdict](#no-verdict)).+## Pipe selection-The selection chain takes the request's `pipe_ref`; else a fetched package's manifest `main_pipe`; else the closure's own `main_pipe` declaration, when exactly one domain declares one. The chain stops at the first link that is present, so a manifest `main_pipe` the closure does not declare, or declares in several domains, is a failed selection rather than a fall-through to the closure's declarations.+The selection chain takes the request's `pipe_ref`; else the `main_pipe` of the `METHODS.toml` at the tree's root, whether the tree was fetched by `method_ref` or sent in `files`; else the closure's own `main_pipe` declaration, when exactly one domain declares one. A root `METHODS.toml` the route cannot read as a manifest, being no UTF-8 text or no valid manifest, is a `422 MethodEntryError` when the request names no pipe. A request naming its pipe in `pipe_ref` never reads the manifest to select it, so there a manifest the route cannot read refuses nothing and only leaves `default_pipe_ref` a stated `null`. The chain stops at the first link that is present, so a manifest `main_pipe` the closure does not declare, or declares in several domains, is a failed selection rather than a fall-through to the closure's declarations.The closure is resolved first, so an invalid closure answers its invalid verdict whatever `pipe_ref` names. For a valid closure these selections are refused with an input `422` `problem+json`. Its `error_type` is the pipelex entry-lookup class that names the failure, never the `ValidationError` of a malformed request, so a client tells a selection refusal from a request-shape one by that field:@@ -59,30 +67,31 @@ A bare `pipe_ref` that matches one pipe (`echo` where `smoke.echo` is meant) is"default_pipe_ref": "smoke.echo","pending_signatures": [],"is_runnable": true,- "files": [{ "content": "...bundle...", "source": "main.mthds" }]+ "files": { "main.mthds": "...bundle..." }}```- `is_valid` (`true`): the closure parsed, loaded and passed static validation. It does **not** mean a dry run passed; this route runs none.- `pipe_ref` (string | null): the qualified ref the selection resolved, read off the resolved pipe and never echoed from the request. `null` only under `all_pipes` when nothing resolves.- `pipe_io_contracts`, `input_form`, `output_form` (objects keyed by qualified `pipe_ref`): the three artifacts, sharing one key set — the resolved `pipe_ref` alone by default, every pipe the closure loads under `all_pipes`.-- `default_pipe_ref` (string | null): the method's own entry pipe, the selection chain without the request's `pipe_ref`. A request that omits `pipe_ref` therefore always answers `pipe_ref == default_pipe_ref`. When the request named a pipe, or asked for `all_pipes`, and the chain finds no entry pipe or several, the field is a stated `null`.+- `default_pipe_ref` (string | null): the method's own entry pipe, the selection chain without the request's `pipe_ref`. A request that omits `pipe_ref` therefore always answers `pipe_ref == default_pipe_ref`. When the request named a pipe, or asked for `all_pipes`, and the chain finds no entry pipe or several, the field is a stated `null`, as it is when the request named a pipe and the root `METHODS.toml` cannot be read.- `pending_signatures` (list of strings) and `is_runnable` (boolean): exactly what they mean on `/v1/validate` — the qualified refs of the pipes still declared as signatures, and `not pending_signatures`. Both come from the loaded library with no dry run, so a method whose dry run would fail is still reported runnable here.-- `files` (list, only with `include_files: true`): the closure in the request's own `files[]` shape. For inline `files` it is the request's files echoed back; for a `method_ref` it is the fetched package's `.mthds` files, each `source` being the file's path relative to the package root. A package's other files are never included. Without `include_files` the field is absent, not empty.+- `files` (object, only with `include_files: true`): the text of each `.mthds` bundle of the method's own tree, keyed by its tree path. For a tree sent in `files`, it is the request's bundles, whatever entry form carried them; for a `method_ref`, the fetched package's bundles, each keyed by its path from the package root. The tree's other files and the bundles of its dependencies are never included. Without `include_files` the field is absent, not empty.-**The artifacts are pinned to `/v1/validate`.** For a closure both routes accept, each artifact map equals the same-named field of `/v1/validate`'s valid arm (requested with `views: ["input_form", "output_form"]`) restricted to the same keys: both routes derive them with the one builder and dump them the same way, keeping null members such as a contract's `item_count: null`. The two routes do not accept exactly the same closures: `/v1/validate` refuses a closure whose dry run fails, which this route accepts; this route refuses an address-based cross-package dependency, which `/v1/validate` loads; and `/v1/validate` by `method_ref` also loads a package's non-`.mthds` files, which this route never reads.+**The artifacts are pinned to `/v1/validate`.** For a closure both routes accept, each artifact map equals the same-named field of `/v1/validate`'s valid arm (requested with `views: ["input_form", "output_form"]`) restricted to the same keys: both routes derive them with the one builder and dump them the same way, keeping null members such as a contract's `item_count: null`. Both routes read the same tree, but they do not accept exactly the same methods: this route loads a package the method calls by its address only from what was shipped with it (see [Resolve & Codegen](codegen.md#selecting-the-method)), where `/v1/validate` also looks among the methods installed on the runner and fetches one on a miss; `/v1/validate` refuses a method whose dry run fails, which this route accepts, and checks the tree's Python by syntax and a `method_ref` package's `requirements.txt`, which this route does not; this route refuses a root `METHODS.toml` it cannot read when the request names no pipe, since its selection chain then reads the manifest, where `/v1/validate` still answers a verdict.`default_pipe_ref` is **not** `/v1/validate`'s field of the same name. That one is the run default: where several domains declare a `main_pipe`, it names the first declaring blueprint's, because a run would take it, while this route refuses to choose and states `null`.## Response (invalid verdict)-An invalid closure is the `200` crate verdict `/v1/resolve` gives: `is_valid: false`, the structured `validation_errors[]` and a `message`. It carries no artifacts, no `pipe_ref` or `default_pipe_ref`, no runnability facts and no `files`, whatever the request asked for.+An invalid closure is the `200` crate verdict `/v1/resolve` gives: `is_valid: false`, the structured `validation_errors[]` and a `message`. A call into a package nothing shipped is one, its `unresolved_package_dependency` item naming the package as `address-based dependency '<address>'`. It carries no artifacts, no `pipe_ref` or `default_pipe_ref`, no runnability facts and no `files`, whatever the request asked for.## No verdictEvery non-2xx is RFC 7807 `application/problem+json` (see [Error Responses](error-responses.md)):-- **`422`** for a malformed body, neither or both closure selectors and an over-limit file, all `ValidationError`, and for the selection refusals above, `EntryPipeNotFoundError` or `EntryPipeAmbiguousError`.-- **The `method_ref` fetch outcomes**, exactly as on `/v1/resolve`: `404` `MethodPackageNotFoundError` when no package in the fetched repository matches the address, `422` for a reference that does not parse or a fetch the server cannot accept, and `501` `MethodRefNotSupported` for a registry-form reference. The route reads only a package's `.mthds` files, so the custom-code and structures `403`s of `/v1/validate` and the run routes never arise here.+- **`422`** for a malformed body or neither or both method selectors, `ValidationError`; for a tree the runner cannot read, `InvalidMethodFiles`, `MissingBlobs`, `MethodTreeTooLarge` or `InvalidBundle`, as on [`/v1/resolve`](codegen.md#selecting-the-method); for an unreadable root manifest on a request naming no pipe, `MethodEntryError`; and for the selection refusals above, `EntryPipeNotFoundError` or `EntryPipeAmbiguousError`.+- **The `method_ref` fetch outcomes**, exactly as on `/v1/resolve`: `404` `MethodPackageNotFoundError` when no package in the fetched repository matches the address, `422` for a reference that does not parse or a fetch the server cannot accept, and `501` `MethodRefNotSupported` for a registry-form reference.+- **`403`** for the custom-Python policy of `/v1/validate` and the run routes, whose tree the route loads: `CustomCodeRequiresSandbox` for a tree holding a `.py` file on a deployment that is not sandbox-hosted, where loading the library would import each `.py` declaring a structure class and run its module-level code in the server's process, and `MethodStructuresRefusedError` for one whose Python declares a structure class.- **`401` / `403`** for authentication, and **`413`** for a body over the size limit.- **`500`** when an artifact cannot be derived: a pipe whose input or output JSON Schema cannot be rendered raises `PipeIOContractError`, which answers `500` as it does on `/v1/validate` — a fault of the tool, not a verdict about the method.diff --git a/docs/api-server/pipe-run.md b/docs/api-server/pipe-run.mdindex 534976a17..828b3390e 100644--- a/docs/api-server/pipe-run.md+++ b/docs/api-server/pipe-run.md@@ -267,7 +267,7 @@ The answer is `{"missing": [{"sha256": "…", "size": n, "upload": {"method": "P**The digest shortcut.** Once the runner has read a request's tree in full, it records the tree's listing beside its blobs, after every file it names is held, and an upload never records one. A validation records it only when its request carries a `read_scope`: without one, its scope would be the caller's own id, where a dry run must leave nothing stored. Asking `POST /v1/blobs/missing` with the digest of a recorded tree then answers an empty list in one lookup, however many files the tree has, when every blob listed is a file of that tree; a blob outside it is looked up as if no digest were sent.-**Scopes.** Blobs live in the deployment's configured storage, under `<scope>/blobs/<sha256>`, and recorded listings under `<scope>/trees/<digest>`. The scope is the request's `read_scope`, which a multi-tenant host sends for the organization it serves (a query parameter on `POST /v1/blobs/missing`, a body field on the run and validate routes), else the caller's own id, so a hash entry resolves only in the caller's own blobs and asking whether a hash is held tells a caller nothing of another's files. A hash entry the runner does not hold, at the size it states, is a `422 MissingBlobs` whose `missing_blobs` member lists the hashes to upload before sending the request again.+**Scopes.** Blobs live in the deployment's configured storage, under `<scope>/blobs/<sha256>`, and recorded listings under `<scope>/trees/<digest>`. The scope is the request's `read_scope`, which a multi-tenant host sends for the organization it serves (a query parameter on `POST /v1/blobs/missing`, a body field on the run and validate routes), else the caller's own id, so a hash entry resolves only in the caller's own blobs and asking whether a hash is held tells a caller nothing of another's files. On a `jwt` deployment a sent scope must lie under the caller's own id, else a `403 ReadScopeForbidden` (see [Host-supplied run context](#host-supplied-run-context-storage_scope-and-analytics_groups)). A hash entry the runner does not hold, at the size it states, is a `422 MissingBlobs` whose `missing_blobs` member lists the hashes to upload before sending the request again.**Signing key.** Each process draws its own signing key unless `BLOB_UPLOAD_SIGNING_KEY` is set. A deployment serving one API through several instances must set it, the same on every instance, since an upload may reach another instance than the one that named its target.@@ -334,7 +334,7 @@ Both `/execute` and `/start` accept **`method_ref`** — a globally resolvable a**Environment knobs.** The cache: `MAX_METHOD_CACHE_CLONES` (default 64), `MAX_METHOD_CACHE_TOTAL_KIB` (default 512 MiB), `MAX_METHOD_CACHE_AGE_HOURS` (default 24), `METHOD_CACHE_DIR` (default: a fixed directory under the system temp dir). The fetch itself is bounded by the pipelex runtime's fetched-package ceilings: `PIPELEX_MAX_FETCHED_PACKAGE_FILES` (default 256), `PIPELEX_MAX_FETCHED_PACKAGE_TOTAL_KIB` (default 8 MiB), and the manifest-scan bounds `PIPELEX_MAX_SCANNED_MANIFESTS` (default 100 manifests considered per fetched repository) and `PIPELEX_MAX_MANIFEST_FILE_KIB` (default 256 KiB per `METHODS.toml` read during package location). All are read once at process start.-`method_ref` also selects the method on the tooling routes: natively on [`POST /v1/validate`](pipe-validate.md), and as the closure selector on [`/resolve` and `/codegen`](codegen.md) (address form resolved; the registry form stays a `501`).+`method_ref` also selects the method on the tooling routes: on [`POST /v1/validate`](pipe-validate.md), [`/resolve` and `/codegen`](codegen.md#selecting-the-method) and [`/pipe-io`](pipe-io.md), which read the fetched package's tree as a run does (address form resolved; on the crate routes the registry form stays a `501`).---@@ -354,7 +354,7 @@ Both `/execute` and `/start` accept optional fields that describe the run's tena**`storage_scope`** is the prefix every object the run writes lands under: one to three path-safe segments, onto which the runtime composes its own leaves (`assets/`, `generated/`, `results/`, `payloads/`). Omit it and the run is scoped to the caller's own id, which is right for a single-tenant deployment and wrong for a multi-tenant one: a host serving many tenants must send it. Each segment is one or more characters from `A-Za-z0-9_-`, so an empty segment, a leading or trailing slash, a `.` or `..`, or a fourth segment is refused with a `422` whose `error_type` is `InvalidStorageScope`.-**`read_scope`** bounds what the run may read: every `pipelex-storage://` key the run reads, whether an input, a URL a pipe builds or an image it shows a model, must lie under it, and a scoped run reads no local path at all. It follows the same segment rule as `storage_scope`, and the storage scope must be the read scope or lie under it, since a run reads back what it writes: `org_acme` covers `org_acme/method_42/run_7`, while `org_other` does not. Omit it and the run is scoped to the caller's own id when the deployment identifies its callers, which is also where their writes land by default, and unscoped when the deployment is single-tenant. A host serving many tenants must send it: a host that sends its own `storage_scope` but no `read_scope` is refused, since that storage scope lies under no caller's id. A malformed read scope, a storage scope outside it, or the local storage scope `local` beside one is a `422` whose `error_type` is `InvalidReadScope`, answered before any method is fetched or library loaded. A read the scope does not allow is refused when the run makes it, with the `error_type` `UriReadRefusedError`, whose message says where the URL sat without quoting it. On `/execute` it is a `422`. On `/start` it is a synchronous `422` when an input breaks the rule, and a run that ends `FAILED` with that error in its completion callback when a pipe reaches for the URL during the run.+**`read_scope`** bounds what the run may read: every `pipelex-storage://` key the run reads, whether an input, a URL a pipe builds or an image it shows a model, must lie under it, and a scoped run reads no local path at all. It follows the same segment rule as `storage_scope`, and the storage scope must be the read scope or lie under it, since a run reads back what it writes: `org_acme` covers `org_acme/method_42/run_7`, while `org_other` does not. Omit it and the run is scoped to the caller's own id when the deployment identifies its callers, which is also where their writes land by default, and unscoped when the deployment is single-tenant. A host serving many tenants must send it: a host that sends its own `storage_scope` but no `read_scope` is refused, since that storage scope lies under no caller's id. A malformed read scope, a storage scope outside it, or the local storage scope `local` beside one is a `422` whose `error_type` is `InvalidReadScope`, answered before any method is fetched or library loaded. **On a `jwt` deployment the read scope is bound to the caller**: the caller reaches the runner with its own token rather than through a trusted host, so a read scope it sends must be its token's `user_id` or lie under it, else the request is a `403` whose `error_type` is `ReadScopeForbidden`, answered before anything is read. Every route taking a `read_scope` applies the same rule (`/v1/validate`, `/v1/resolve`, `/v1/codegen`, `/v1/pipe-io` and `POST /v1/blobs/missing` too), and since the storage scope must lie under the read scope, a `jwt` caller writes only under its own id as well. With `AUTH_MODE=none` and the shared `api_key`, the caller is a trusted host or a single tenant, and the scope it sends is taken as it is. A read the scope does not allow is refused when the run makes it, with the `error_type` `UriReadRefusedError`, whose message says where the URL sat without quoting it. On `/execute` it is a `422`. On `/start` it is a synchronous `422` when an input breaks the rule, and a run that ends `FAILED` with that error in its completion callback when a pipe reaches for the URL during the run.**`analytics_groups`** names the entities the run's telemetry belongs to, as a mapping of group type to group key. The runtime carries it on the run as its opaque `extras` and stamps it on every span of the run as the attribute `pipelex.run.extras`, so every OpenTelemetry exporter receives it, and the deployment's own PostHog stream, when `telemetry.toml` sets `mode = "identified"`, attaches it to each capture as PostHog groups. That is what lets a generation made for one of your customers appear inside that customer's organization instead of under one identity for the whole deployment. The rules are the runtime's own, imported rather than restated here:diff --git a/docs/api-server/pipe-validate.md b/docs/api-server/pipe-validate.mdindex c91c4ab92..98ca9a909 100644--- a/docs/api-server/pipe-validate.md+++ b/docs/api-server/pipe-validate.md@@ -20,7 +20,7 @@ Validate MTHDS content by parsing, loading, and dry-running pipes without execut- `files` (object, optional): The method as a map of file entries, the MTHDS standard's method tree, each value the file's text, `{"base64": "…"}` or `{"sha256": "…", "size": n}`, read as on a run — see [Validating a method tree](#validating-a-method-tree). Exactly one of `mthds_contents` / `files` / `method_ref`- `dependencies` (object, optional): **Pipelex-API extension.** The method packages the method calls by address that the runner cannot fetch, each a map of file entries keyed by its address, as on a [run](pipe-run.md#sending-the-method-packages-the-method-calls). Accompanies `files`- `method_ref` (string, optional): **Pipelex-API extension.** Validate a published method by address — `github.com/<owner>/<repo>` plus an optional package selector and `@<tag>` — resolved by this server through the same fetch path as a [`method_ref` run](pipe-run.md#running-a-method-by-address-method_ref): the repository is fetched at the tag, the package located by manifest identity, and its tree read from the commit's git objects and validated as a tree sent in `files` is — see [Validating a method tree](#validating-a-method-tree). Exactly one of `mthds_contents` / `files` / `method_ref`; `mthds_sources` cannot accompany it. A package holding Python and a `requirements.txt` has its requirements checked on this server against the package set of the deployment's PipeFunc sandbox before anything is validated, and a requirement the sandbox does not meet, or a line the standard refuses, makes the verdict `is_valid: false` with one item per line, whose `source` is `requirements.txt` and whose `line` is the line's number-- `read_scope` (string, optional): **Pipelex-API extension.** The scope the validation reads the method under, as a run's `read_scope`, which a multi-tenant host sends for the organization it serves. A hash entry of `files` resolves in that scope's blob store — see [Sending files by hash](pipe-run.md#sending-files-by-hash) — and the runner hands the scope to its validator, so a validation dispatched to a worker can hand it the method through that store — see [Where validation runs](#where-validation-runs). Omitted, a hash entry resolves in the caller's own id's store, but only a validation carrying it records the tree it read in the blob store, and the validator is handed no scope, so a dry run without one leaves nothing stored+- `read_scope` (string, optional): **Pipelex-API extension.** The scope the validation reads the method under, as a run's `read_scope`, which a multi-tenant host sends for the organization it serves; [a `jwt` caller may send only a scope under its own id](pipe-run.md#host-supplied-run-context-storage_scope-and-analytics_groups), else a `403 ReadScopeForbidden`. A hash entry of `files` resolves in that scope's blob store — see [Sending files by hash](pipe-run.md#sending-files-by-hash) — and the runner hands the scope to its validator, so a validation dispatched to a worker can hand it the method through that store — see [Where validation runs](#where-validation-runs). Omitted, a hash entry resolves in the caller's own id's store, but only a validation carrying it records the tree it read in the blob store, and the validator is handed no scope, so a dry run without one leaves nothing stored- `allow_signatures` (boolean, optional, default `false`): controls only the **sweep mechanics** for `PipeSignature` placeholders — whether signature pipes are mock-run during the dry-run sweep and therefore listed in `validated_pipes`. It does **not** change the verdict: an unimplemented signature is never a rejection, it is a *runnability fact* reported via `pending_signatures` + `is_runnable` in both modes- `mthds_sources` (list[str] | null, optional): per-file sources, parallel to `mthds_contents` — see [Sourcing submitted files](#sourcing-submitted-files). When present it must match `mthds_contents` in length (a mismatch is a 422 request error), and it accompanies `mthds_contents` alone, since a tree's files are named by their tree paths- `resources` (object | null, optional): **Pipelex-API extension.** The files the bundle's steps read, such as a `PipeDocGen` step's `template_file`, as a map from the path the step names to the file's bytes in standard base64 — see [Sending the files the steps read](#sending-the-files-the-steps-read-resources). It accompanies `mthds_contents` or `files`, whose tree each file joins; beside `method_ref` it is a 422@@ -170,7 +170,7 @@ A method sent in `files`, or fetched by `method_ref`, is validated whole, as it## Sending the files the steps read (`resources`)-A bundle sent as content has no directory, so a file one of its steps reads from beside the bundle, such as a `PipeDocGen` step's `template_file`, is sent with it in `resources`. The same field, with the same rules, is accepted by [`/v1/execute` and `/v1/start`](pipe-run.md), [`/v1/pipe-io`](pipe-io.md), and [`/v1/resolve` and `/v1/codegen`](codegen.md#selecting-the-closure).+A bundle sent as content has no directory, so a file one of its steps reads from beside the bundle, such as a `PipeDocGen` step's `template_file`, is sent with it in `resources`. The same field, with the same rules, is accepted by [`/v1/execute` and `/v1/start`](pipe-run.md), [`/v1/pipe-io`](pipe-io.md), and [`/v1/resolve` and `/v1/codegen`](codegen.md#selecting-the-method).```json{diff --git a/docs/errors/method-dependency-not-shipped-error.md b/docs/errors/method-dependency-not-shipped-error.mdnew file mode 100644index 000000000..030edbfca--- /dev/null+++ b/docs/errors/method-dependency-not-shipped-error.md@@ -0,0 +1,21 @@+---+title: "Method dependency not shipped"+description: "Reference for the `MethodDependencyNotShippedError` Pipelex error class."+---++<!-- pipelex:generated -->++# Method dependency not shipped++An address-referenced method the caller did not ship, under a resolution that reads nothing installed and fetches nothing.++| Field | Value |+|---|---|+| `error_type` | `MethodDependencyNotShippedError` |+| `title` | Method dependency not shipped |+| `type_uri` | `https://docs.pipelex.com/latest/errors/method-dependency-not-shipped-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.methods.exceptions` |+| Parent class | [`MethodRefError`](method-ref-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/platform-and-tooling.md b/docs/errors/platform-and-tooling.mdindex 2486727d5..859c2789e 100644--- a/docs/errors/platform-and-tooling.md+++ b/docs/errors/platform-and-tooling.md@@ -117,6 +117,7 @@ own page. Classes are grouped by subsystem.## Methods- [`MethodDependencyFetchError`](method-dependency-fetch-error.md) — Method dependency fetch+- [`MethodDependencyNotShippedError`](method-dependency-not-shipped-error.md) — Method dependency not shipped- [`MethodFetchDisabledError`](method-fetch-disabled-error.md) — Method fetch disabled- [`MethodFetchError`](method-fetch-error.md) — Method fetch- [`MethodInstallError`](method-install-error.md) — Method installdiff --git a/docs/under-the-hood/codegen-projections.md b/docs/under-the-hood/codegen-projections.mdindex e3bc98e37..03e75ea97 100644--- a/docs/under-the-hood/codegen-projections.md+++ b/docs/under-the-hood/codegen-projections.md@@ -169,8 +169,9 @@ The remaining findings are artifacts of *how* you lint, not of file content. Non## Serving the engine over HTTP-The same engine backs the `pipelex-api` routes (`POST /v1/resolve`, `POST /v1/codegen` and `POST /v1/pipe-io` — the route envelopes are documented on the API server's [Resolve & Codegen](../api-server/codegen.md) and [Pipe I/O](../api-server/pipe-io.md) pages). Two host-facing cores make that possible without any CLI plumbing:+The same engine backs the `pipelex-api` routes (`POST /v1/resolve`, `POST /v1/codegen` and `POST /v1/pipe-io` — the route envelopes are documented on the API server's [Resolve & Codegen](../api-server/codegen.md) and [Pipe I/O](../api-server/pipe-io.md) pages). Host-facing cores make that possible without any CLI plumbing:+- **`pipelex.pipeline.resolve_bundle.resolve_crate_from_method_dir`** resolves a method's tree the host received and wrote out, which is how the three routes resolve the tree a request sends in `files` or a `method_ref` fetches: the directory is loaded alone as the caller's library directory, exactly as `validate_bundle` loads a `method_dir`, so each bundle is named by its tree path while the host holds the tree in `WRITTEN_METHOD_TREES`, the other files are the ones its steps read, and the packages it ships (laid out like `.mthds/methods/`) are the only place an address-based reference is looked up: the resolution is static, so it reads no method installed on the host and fetches nothing, and a call into a package nothing shipped is an `unresolved_package_dependency` item of the verdict. The API server runs it in a worker thread, with the projection and the teardown. It shares the `ValidateBundleError` cascade and the loaded-on-success contract described next.- **`pipelex.pipeline.resolve_bundle.resolve_crate_from_contents`** resolves **in-memory** MTHDS contents (strings, with optional per-content sources) into the normalized crate. It mirrors `validate_bundle`'s in-memory arm — the same `translate_to_validate_bundle_error` cascade, so an invalid library raises the one shared `ValidateBundleError` and a resolve verdict cannot drift from a validate verdict — and the same **loaded-on-success contract**: the library is left loaded and current for the host to read live pipes from, and the host owns its teardown. Resolution is static (no dry-run sweep), matching `pipelex resolve`.- **`build_stamped_projection`** (above) gives the host the stamped artifact set plus the lock as pure content. A client that writes the served files and lock verbatim reproduces a local run byte-for-byte — the offline `codegen check` passes on the written tree exactly as it would locally. There is deliberately no server-side check route: the check is offline by design.diff --git a/pipelex/hosted/hosted_inputs.py b/pipelex/hosted/hosted_inputs.pyindex 7d2a73e20..5618108c0 100644--- a/pipelex/hosted/hosted_inputs.py+++ b/pipelex/hosted/hosted_inputs.py@@ -38,6 +38,7 @@ from pipelex_sdk.crate_models import CrateInvalidReport, MthdsFileItem, PipeIORefrom pipelex_sdk.upload import UploadRecordfrom pipelex.hosted.exceptions import HostedLocalFileUploadUnavailableError, HostedMethodInvalidError+from pipelex.methods.address_dependencies import ADDRESS_DEPENDENCY_REFUSAL_MARKERfrom pipelex.tools.uri.resolved_uri import ResolvedLocalPathfrom pipelex.tools.uri.uri_resolver import resolve_uri@@ -49,10 +50,9 @@ _HTTP_URL_PATTERN = re.compile(r"^https?://", re.IGNORECASE)_NOT_LOCAL_PREFIXES: tuple[str, ...] = ("pipelex-storage://", "data:")#: A `file://` URI, which a local run reads as the local path it names._FILE_URI_PREFIX = "file://"-#: What the crate core behind `POST /v1/pipe-io` says when it refuses a method calling another one by its address-#: (`pipelex/pipeline/resolve_bundle.py`). The run route fetches such a dependency; pipe-io cannot load it.-ADDRESS_DEPENDENCY_REFUSAL_MARKER = "address-based dependency"-#: The dependency's address, as that refusal quotes it.+#: The dependency's address, as the refusal of `POST /v1/pipe-io` quotes it when the method calls a package by its+#: address that the request did not ship (`ADDRESS_DEPENDENCY_REFUSAL_MARKER`). The run route fetches such a dependency;+#: pipe-io never fetches._REFUSED_DEPENDENCY_PATTERN = re.compile(re.escape(ADDRESS_DEPENDENCY_REFUSAL_MARKER) + r" '([^']+)'")diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.pyindex c7be4e507..7cfbfded2 100644--- a/pipelex/libraries/library_manager.py+++ b/pipelex/libraries/library_manager.py@@ -68,7 +68,8 @@ from pipelex.libraries.method_trees import (from pipelex.libraries.pipe.exceptions import PipeLibraryErrorfrom pipelex.libraries.visibility_utils import check_visibility_for_blueprints, make_visibility_checkerfrom pipelex.libraries.written_trees import WRITTEN_METHOD_TREES-from pipelex.methods.exceptions import MethodDependencyFetchError, MethodFetchDisabledError+from pipelex.methods.address_dependencies import AddressDependencySources, unshipped_address_dependency_message+from pipelex.methods.exceptions import MethodDependencyFetchError, MethodDependencyNotShippedError, MethodFetchDisabledErrorfrom pipelex.methods.fetch_on_miss import find_vendored_method, resolve_address_based_methodfrom pipelex.methods.structures_check import ensure_no_structured_content_in_library_sourcesfrom pipelex.mthds_parsing.exceptions import MthdsParserError@@ -687,9 +688,12 @@ class LibraryManager(LibraryManagerAbstract):library_dirs: list[Path] | None = None,library_file_paths: list[Path] | None = None,methods_dirs: list[Path] | None = None,+ address_dependency_sources: AddressDependencySources = AddressDependencySources.ANYWHERE,) -> list[PipeAbstract]:# `methods_dirs` are laid out like `.mthds/methods/` and hold the packages a caller ships with the method these# directories hold, searched first for an address-based reference of their bundles and never written.+ # `address_dependency_sources` says whether a reference they do not answer may be looked up on this host and+ # fetched (`ANYWHERE`), or is refused (`SHIPPED`, a static resolution of a caller's method).# Ensure libraries exist for this library_idif library_id not in self._libraries:msg = f"Trying to load a library that does not exist: '{library_id}'"@@ -784,7 +788,12 @@ class LibraryManager(LibraryManagerAbstract):"Loading the MTHDS files into the library",fields={"library_id": library_id, "mthds_paths": [str(bundle_file.path) for bundle_file in bundle_files]},)- return self._load_mthds_files_into_library(library_id=library_id, bundle_files=bundle_files, methods_dirs=methods_dirs)+ return self._load_mthds_files_into_library(+ library_id=library_id,+ bundle_files=bundle_files,+ methods_dirs=methods_dirs,+ address_dependency_sources=address_dependency_sources,+ )@overridedef is_crate_loaded(self, *, library_id: str, fingerprint: str) -> bool:@@ -1009,7 +1018,12 @@ class LibraryManager(LibraryManagerAbstract):@overridedef load_from_blueprints(- self, *, library_id: str, blueprints: list[PipelexBundleBlueprint], methods_dirs: list[Path] | None = None+ self,+ *,+ library_id: str,+ blueprints: list[PipelexBundleBlueprint],+ methods_dirs: list[Path] | None = None,+ address_dependency_sources: AddressDependencySources = AddressDependencySources.ANYWHERE,) -> list[PipeAbstract]:"""Load domains, concepts, and pipes from a list of blueprints.@@ -1022,6 +1036,8 @@ class LibraryManager(LibraryManagerAbstract):methods_dirs: Directories laid out like ``.mthds/methods/`` holding packages for this load alone, such asthe ones a request ships with its bundle. An address-based reference is looked up there first, and apackage found there is never installed anywhere.+ address_dependency_sources: Whether a reference ``methods_dirs`` do not answer is looked up among the methods+ installed on this host and fetched on a miss (``ANYWHERE``), or refused (``SHIPPED``).Returns:List of all pipes that were loaded@@ -1068,6 +1084,7 @@ class LibraryManager(LibraryManagerAbstract):library_id=library_id,blueprints=blueprints,methods_dirs=methods_dirs,+ address_dependency_sources=address_dependency_sources,)# Build the crate (merges, qualifies, detects duplicates)@@ -1273,7 +1290,12 @@ class LibraryManager(LibraryManagerAbstract):return live_treesdef _load_mthds_files_into_library(- self, *, library_id: str, bundle_files: list[LibraryBundleFile], methods_dirs: list[Path] | None = None+ self,+ *,+ library_id: str,+ bundle_files: list[LibraryBundleFile],+ methods_dirs: list[Path] | None = None,+ address_dependency_sources: AddressDependencySources = AddressDependencySources.ANYWHERE,) -> list[PipeAbstract]:"""Load MTHDS files into a specific library.@@ -1286,9 +1308,12 @@ class LibraryManager(LibraryManagerAbstract):library_id: The ID of the library to load intobundle_files: The bundle files to load, each the member of its method treemethods_dirs: Directories laid out like ``.mthds/methods/`` holding packages for this load alone+ address_dependency_sources: Where a package the bundles call by address may be found (``load_libraries``)"""valid_mthds_paths = [bundle_file.path for bundle_file in bundle_files]blueprints: list[PipelexBundleBlueprint] = []+ # Each bundle as a message names it, which for a tree a host wrote out is its tree path, as for its blueprint.+ bundle_sources: list[str] = []for bundle_file in bundle_files:mthds_file_path = bundle_file.path# A bundle of a tree a host wrote out for a caller is named by its tree path, never by the host's directory.@@ -1297,6 +1322,7 @@ class LibraryManager(LibraryManagerAbstract):if bundle_file.tree_bundle.tree_path is not None and WRITTEN_METHOD_TREES.holds_tree(tree=bundle_file.tree_bundle.tree)else str(mthds_file_path))+ bundle_sources.append(bundle_source)try:blueprint = MthdsParser.make_pipelex_bundle_blueprint(bundle_path=mthds_file_path, mthds_source=bundle_source)blueprint.place_in_tree(tree_bundle=bundle_file.tree_bundle)@@ -1342,7 +1368,12 @@ class LibraryManager(LibraryManagerAbstract):library.loaded_mthds_paths.append(resolved_path)try:- return self.load_from_blueprints(library_id=library_id, blueprints=blueprints, methods_dirs=methods_dirs)+ return self.load_from_blueprints(+ library_id=library_id,+ blueprints=blueprints,+ methods_dirs=methods_dirs,+ address_dependency_sources=address_dependency_sources,+ )except ValidationError as validation_error:# Categorize before wrapping. `translate_to_validate_bundle_error` has its own# `except ValidationError` arm that categorizes, but it never sees this error: wrapping it@@ -1353,7 +1384,7 @@ class LibraryManager(LibraryManagerAbstract):# reachable through `resolve_crate_from_contents` but not through a library directory.raise LibraryLoadingError(message=(- f"Could not load blueprints from {[str(pth) for pth in valid_mthds_paths]} because of: "+ f"Could not load blueprints from {bundle_sources} because of: "f"{report_validation_error(validation_error=validation_error).message}"),pipe_concept_validation_errors=categorize_pipe_validation_error(validation_error=validation_error),@@ -1367,7 +1398,7 @@ class LibraryManager(LibraryManagerAbstract):# `ConceptLibraryError` is a `LibraryLoadingError` and reaches here holding the per-reference# items that make an unresolved concept a categorized `unresolved_concept` rather than a# bare residual. Reconstructing it message-only silently dropped them.- msg = f"Could not load blueprints from {[str(pth) for pth in valid_mthds_paths]} because of: {ref_error}"+ msg = f"Could not load blueprints from {bundle_sources} because of: {ref_error}"if isinstance(ref_error, LibraryLoadingError):raise type(ref_error)(msg,@@ -1756,6 +1787,7 @@ class LibraryManager(LibraryManagerAbstract):library_id: str,blueprints: list[PipelexBundleBlueprint],methods_dirs: list[Path] | None = None,+ address_dependency_sources: AddressDependencySources = AddressDependencySources.ANYWHERE,) -> None:"""Scan blueprints for cross-package pipe refs with address-based aliases and load them.@@ -1766,6 +1798,10 @@ class LibraryManager(LibraryManagerAbstract):the packages of ``methods_dirs``, the load's own, then among theinstalled methods; one missing from both is fetched by address(honoring an ``@<tag>`` pin) and installed when fetch-on-miss is enabled.+ With ``AddressDependencySources.SHIPPED``, only ``methods_dirs`` are+ searched: nothing installed on this host is read and nothing is fetched,+ and an address they do not answer is unresolved, its item carrying+ ``unshipped_address_dependency_message``.A package that cannot be resolved is the caller's to fix, since the reference is part of their bundle: everyaddress is tried, then the load is refused with one ``unresolved_package_dependency`` item per address that@@ -1780,6 +1816,7 @@ class LibraryManager(LibraryManagerAbstract):library_id: The library to load intoblueprints: The parsed bundle blueprints to scanmethods_dirs: The load's own methods directories, searched before everything else and never written+ address_dependency_sources: Whether an address ``methods_dirs`` do not answer may be looked up on this hostRaises:LibraryLoadingError: One or more referenced packages could not be resolved.@@ -1798,8 +1835,13 @@ class LibraryManager(LibraryManagerAbstract):if not first_references:return- # Derive extra search dirs from bundle source paths- extra_search_dirs = find_methods_dirs_from_blueprints(blueprints)+ # Derive extra search dirs from bundle source paths, which only a load that may read this host's methods searches.+ extra_search_dirs: list[Path] | None+ match address_dependency_sources:+ case AddressDependencySources.SHIPPED:+ extra_search_dirs = None+ case AddressDependencySources.ANYWHERE:+ extra_search_dirs = find_methods_dirs_from_blueprints(blueprints)unresolved_items: list[PipesAndConceptValidationErrorData] = []for full_address, (reference, blueprint) in first_references.items():@@ -1812,8 +1854,9 @@ class LibraryManager(LibraryManagerAbstract):full_address=full_address,extra_search_dirs=extra_search_dirs,methods_dirs=methods_dirs,+ address_dependency_sources=address_dependency_sources,)- except (MethodFetchDisabledError, MethodDependencyFetchError) as exc:+ except (MethodFetchDisabledError, MethodDependencyFetchError, MethodDependencyNotShippedError) as exc:# The item carries the caller-facing text; the host's log keeps the whole cause chain, git's raw output# included, since a cause on the host (git missing, an unwritable clone directory) becomes an item too.causes = " <- ".join(f"{type(cause).__name__}: {cause}" for cause in iter_cause_chain(exc))@@ -1847,6 +1890,7 @@ class LibraryManager(LibraryManagerAbstract):full_address: str,extra_search_dirs: list[Path] | None = None,methods_dirs: list[Path] | None = None,+ address_dependency_sources: AddressDependencySources = AddressDependencySources.ANYWHERE,) -> None:"""Load a method package on demand using its full address, fetching it on a miss.@@ -1862,14 +1906,23 @@ class LibraryManager(LibraryManagerAbstract):full_address: The full package address (e.g. "github.com/Pipelex/methods/documents")extra_search_dirs: Additional .mthds/methods/ directories to scanmethods_dirs: The load's own methods directories, searched before everything else and never written+ address_dependency_sources: ``SHIPPED`` refuses an address ``methods_dirs`` do not answer, reading nothing+ installed on this host and fetching nothing; ``ANYWHERE`` falls back to the installed methods and a fetchRaises:MethodRefError: The address resolves to no vendored or installed method and could not befetched — fetch-on-miss disabled, an unfetchable address, a failed fetch,or a failed install. Never a silent pass.+ MethodDependencyNotShippedError: Under ``SHIPPED``, no package of ``methods_dirs`` answers the address."""vendored = find_vendored_method(full_address=full_address, methods_dirs=methods_dirs) if methods_dirs else None- installed = vendored or resolve_address_based_method(full_address=full_address, extra_search_dirs=extra_search_dirs)+ match address_dependency_sources:+ case AddressDependencySources.SHIPPED:+ if vendored is None:+ raise MethodDependencyNotShippedError(unshipped_address_dependency_message(alias=full_address))+ installed = vendored+ case AddressDependencySources.ANYWHERE:+ installed = vendored or resolve_address_based_method(full_address=full_address, extra_search_dirs=extra_search_dirs)exported_pipe_codes = determine_exported_pipes(manifest=installed.manifest)diff --git a/pipelex/libraries/library_manager_abstract.py b/pipelex/libraries/library_manager_abstract.pyindex 7ff05c58e..dab00680f 100644--- a/pipelex/libraries/library_manager_abstract.py+++ b/pipelex/libraries/library_manager_abstract.py@@ -6,6 +6,7 @@ from typing import TYPE_CHECKINGfrom kajson.class_registry import ClassRegistryfrom pipelex.libraries.library_crate import LibraryCrate+from pipelex.methods.address_dependencies import AddressDependencySourcesfrom pipelex.mthds_parsing.pipelex_bundle_blueprint import PipelexBundleBlueprintfrom pipelex.pipe_machinery.pipe_abstract import PipeAbstract@@ -162,7 +163,12 @@ class LibraryManagerAbstract(ABC):@abstractmethoddef load_from_blueprints(- self, *, library_id: str, blueprints: list[PipelexBundleBlueprint], methods_dirs: list[Path] | None = None+ self,+ *,+ library_id: str,+ blueprints: list[PipelexBundleBlueprint],+ methods_dirs: list[Path] | None = None,+ address_dependency_sources: AddressDependencySources = AddressDependencySources.ANYWHERE,) -> list[PipeAbstract]:"""Load the blueprints into the library, resolving their address-based dependencies.@@ -171,6 +177,8 @@ class LibraryManagerAbstract(ABC):blueprints: The parsed bundles to loadmethods_dirs: Directories laid out like ``.mthds/methods/`` holding packages for this load alone, searchedfor an address-based dependency before the installed stores and fetch-on-miss, and never written+ address_dependency_sources: ``SHIPPED`` searches ``methods_dirs`` alone, reading nothing installed on this host+ and fetching nothing; ``ANYWHERE`` falls back to the installed stores and fetch-on-miss"""@abstractmethod@@ -189,5 +197,6 @@ class LibraryManagerAbstract(ABC):library_dirs: list[Path] | None = None,library_file_paths: list[Path] | None = None,methods_dirs: list[Path] | None = None,+ address_dependency_sources: AddressDependencySources = AddressDependencySources.ANYWHERE,) -> list[PipeAbstract]:passdiff --git a/pipelex/methods/address_dependencies.py b/pipelex/methods/address_dependencies.pynew file mode 100644index 000000000..e0d63ff33--- /dev/null+++ b/pipelex/methods/address_dependencies.py@@ -0,0 +1,33 @@+"""Where a library load looks for the method packages its bundles call by their address.++A bundle calls a pipe of another method package by the package's address, such as+`github.com/Pipelex/methods/documents->documents.extract_page_contents_and_views`. A load finds that package among the+packages shipped with the method first (its `methods_dirs`), and, when it may, among the methods installed on this host,+fetching one over the network on a miss. A host resolving a caller's method statically, such as the API server's crate+routes, reads nothing of its own and fetches nothing: a package the caller did not ship is refused, with the refusal+`unshipped_address_dependency_message` words, which a client tells apart by `ADDRESS_DEPENDENCY_REFUSAL_MARKER`.+"""++from enum import StrEnum++#: The words every refusal of an address-based dependency nothing shipped carries, which a client matches to tell that+#: refusal apart from the method being invalid (`pipelex.hosted.hosted_inputs`).+ADDRESS_DEPENDENCY_REFUSAL_MARKER = "address-based dependency"+++class AddressDependencySources(StrEnum):+ """Where a load looks for a package its bundles call by address."""++ SHIPPED = "shipped"+ """The packages shipped with the method alone: nothing installed on this host is read, and nothing is fetched."""++ ANYWHERE = "anywhere"+ """The shipped packages first, then the methods installed on this host, then a fetch when fetch-on-miss is enabled."""+++def unshipped_address_dependency_message(*, alias: str) -> str:+ """The refusal of an address-based dependency a static resolution cannot load, since nothing shipped it."""+ return (+ f"In-memory resolve cannot load {ADDRESS_DEPENDENCY_REFUSAL_MARKER} '{alias}' from the host filesystem; "+ "provide the dependency contents in the request or use directory-based resolution."+ )diff --git a/pipelex/methods/exceptions.py b/pipelex/methods/exceptions.pyindex 8f0c1fdc6..ced3362ad 100644--- a/pipelex/methods/exceptions.py+++ b/pipelex/methods/exceptions.py@@ -56,3 +56,7 @@ class MethodFetchDisabledError(MethodRefError):class MethodDependencyFetchError(MethodRefError):"""An address-referenced method is not installed and fetching it failed (or its address is not fetchable)."""+++class MethodDependencyNotShippedError(MethodRefError):+ """An address-referenced method the caller did not ship, under a resolution that reads nothing installed and fetches nothing."""diff --git a/pipelex/pipeline/resolve_bundle.py b/pipelex/pipeline/resolve_bundle.pyindex 93bb85d5f..64dacf22f 100644--- a/pipelex/pipeline/resolve_bundle.py+++ b/pipelex/pipeline/resolve_bundle.py@@ -1,9 +1,19 @@-"""Resolve in-memory MTHDS contents into a normalized library crate (presentation-free core).+"""Resolve a method into a normalized library crate, for a host (presentation-free core).-This is the contents-based sibling of the CLI's directory-based crate loader-(`pipelex.cli.commands.crate_loading.load_normalized_crate`): a host that already holds the-bundle contents as strings — an HTTP API, an SDK — resolves them here without touching the-filesystem. Same engine, same verdict vocabulary: an invalid library raises the one shared+Two entry points, the host-facing siblings of the CLI's directory-based crate loader+(`pipelex.cli.commands.crate_loading.load_normalized_crate`):++- ``resolve_crate_from_method_dir`` resolves a method's tree a host received and wrote out, such as the tree the API+ server's crate routes write from a request's `files` map: the directory is loaded as the caller's own, the way+ ``validate_bundle`` loads a ``method_dir``, so its other files are the ones its steps read, and, while the host holds+ the tree in ``WRITTEN_METHOD_TREES``, its bundles are named by their tree paths.+- ``resolve_crate_from_contents`` resolves bundle contents held as strings, without touching the filesystem.++Both resolve statically from what the caller sent: a package the method calls by its address is loaded only from the+packages shipped with it, and one nothing shipped is refused with ``unshipped_address_dependency_message``, never looked+up among the methods installed on the host or fetched.++Same engine, same verdict vocabulary: an invalid library raises the one shared``ValidateBundleError`` (carrying the structured ``validation_errors`` items every surfacerenders), so a resolve verdict cannot drift from a validate verdict.@@ -12,7 +22,9 @@ no dry-run sweep, matching `pipelex resolve` — the crate is emitted from a loavalidated** library (D6), and runnability is `validate`'s vocabulary, not resolution's."""-from collections.abc import Mapping+from collections.abc import Generator, Mapping+from contextlib import contextmanager+from pathlib import Pathfrom typing import Sequencefrom mthds.package.manifest.schema import MTHDS_STANDARD_VERSION@@ -25,8 +37,10 @@ from pipelex.libraries.bundle_resources import check_submitted_filesfrom pipelex.libraries.crate_normalization import normalize_cratefrom pipelex.libraries.library_crate import LibraryCratefrom pipelex.libraries.pipe.exceptions import PipeLibraryError+from pipelex.methods.address_dependencies import AddressDependencySources, unshipped_address_dependency_messagefrom pipelex.mthds_parsing.parser import MthdsParserfrom pipelex.mthds_parsing.pipelex_bundle_blueprint import PipelexBundleBlueprint+from pipelex.pipeline.exceptions import ValidateBundleErrorfrom pipelex.pipeline.validate_bundle_translation import translate_to_validate_bundle_error@@ -72,11 +86,7 @@ def resolve_crate_from_contents(raise PipelexUnexpectedError(msg)library_manager = get_library_manager()- library_id, _ = library_manager.open_library()- success = False- prev_library_id = get_current_library_id_or_none()- try:- set_current_library(library_id=library_id)+ with _resolving_library() as library_id:content_sources: list[str | None] = list(mthds_sources) if mthds_sources is not None else [None] * len(mthds_contents)with translate_to_validate_bundle_error():# Contents belong to no method tree: their steps read only the files submitted with them.@@ -98,13 +108,73 @@ def resolve_crate_from_contents(# None crate is an internal invariant break, not a caller-facing verdict.msg = "library crate unavailable after a successful bundle load"raise PipelexUnexpectedError(msg)- normalized = normalize_crate(crate, mthds_version=MTHDS_STANDARD_VERSION)+ return normalize_crate(crate, mthds_version=MTHDS_STANDARD_VERSION)+++def resolve_crate_from_method_dir(*, method_dir: Path, methods_dirs: Sequence[Path] | None = None) -> LibraryCrate:+ """Load, validate, and normalize the library crate of a method's tree written out to a directory.++ The directory is loaded alone, as the caller's own library directory, exactly as ``validate_bundle`` loads a+ ``method_dir``: every bundle of the tree is loaded as a member of it, every other file is one a step may read, such+ as a ``PipeDocGen`` template, and ``methods_dirs``, laid out like ``.mthds/methods/``, hold the packages the caller+ ships with the method, the only place an address-based reference is looked up: one they do not answer is refused,+ since a static resolution reads no method installed on the host and fetches nothing. A tree the host holds in+ ``WRITTEN_METHOD_TREES`` (``pipelex.libraries.written_trees``) has its bundles named by their tree paths, so the+ diagnostics and the crate's ``source_map`` never name a path on the host. Resolution is static, as for contents: no+ dry run.++ Same **loaded-on-success contract** as ``resolve_crate_from_contents``: on success the library is left loaded and+ current, and the host owns its teardown; on failure it is torn down here and the caller's prior current library is+ restored.++ Args:+ method_dir: The directory of the method's tree, written as the tree holds it.+ methods_dirs: Directories laid out like ``.mthds/methods/``, holding the packages the method ships.++ Returns:+ The normalized library crate (fully qualified, natives materialized, fingerprint set).++ Raises:+ ValidateBundleError: the produced negative verdict — the method holds no bundle, calls a package by its address+ that nothing shipped, or its library could not be parsed, loaded, or validated; carries the structured+ validation-error items.+ CrateNormalizationError: internal invariant violation in the normalization pass — deliberately not translated.+ """+ library_manager = get_library_manager()+ with _resolving_library() as library_id:+ with translate_to_validate_bundle_error():+ library_manager.load_libraries(+ library_id=library_id,+ library_dirs=[method_dir],+ methods_dirs=list(methods_dirs) if methods_dirs is not None else None,+ address_dependency_sources=AddressDependencySources.SHIPPED,+ )+ crate = library_manager.get_crate(library_id)+ if crate is None:+ # A tree without a bundle loads nothing: the caller's method, so the caller's verdict, as `validate_bundle`+ # says of a method directory with no bundle.+ msg = "The method holds no .mthds bundle, so it has no crate to resolve."+ raise ValidateBundleError(message=msg)+ return normalize_crate(crate, mthds_version=MTHDS_STANDARD_VERSION)+++@contextmanager+def _resolving_library() -> Generator[str, None, None]:+ """Open a library and make it current for one resolution, which leaves it loaded on success.++ On failure the library is torn down and the caller's prior current library restored, FIRST, so the guarantee holds+ even when the teardown raises (the same ordering as ``validate_bundle``).+ """+ library_manager = get_library_manager()+ library_id, _ = library_manager.open_library()+ prev_library_id = get_current_library_id_or_none()+ success = False+ try:+ set_current_library(library_id=library_id)+ yield library_idsuccess = True- return normalizedfinally:if not success:- # Restore the caller's outer current-library FIRST so the safety guarantee holds- # even when teardown raises (same ordering rationale as validate_bundle).if prev_library_id is not None:set_current_library(library_id=prev_library_id)else:@@ -120,8 +190,5 @@ def _reject_address_based_dependencies(*, blueprints: list[PipelexBundleBlueprincontinuealias, _remainder = QualifiedRef.split_cross_package_ref(pipe_ref)if QualifiedRef.is_address_based_alias(alias):- msg = (- f"In-memory resolve cannot load address-based dependency '{alias}' from the host filesystem; "- "provide the dependency contents in the request or use directory-based resolution."- )+ msg = unshipped_address_dependency_message(alias=alias)raise PipeLibraryError(msg)diff --git a/tests/data/errors/error_identity.txt b/tests/data/errors/error_identity.txtindex dd46cbfe0..ca5dd2bb0 100644--- a/tests/data/errors/error_identity.txt+++ b/tests/data/errors/error_identity.txt@@ -184,6 +184,7 @@ LogSinkHeaderValueError | Log sink header value holds a line break | https://docLogSinkVariableError | Log sink variable did not resolve | https://docs.pipelex.com/latest/errors/log-sink-variable-error/MarkdownFormattingBudgetError | Markdown formatting budget exceeded | https://docs.pipelex.com/latest/errors/markdown-formatting-budget-error/MethodDependencyFetchError | Method dependency fetch | https://docs.pipelex.com/latest/errors/method-dependency-fetch-error/+MethodDependencyNotShippedError | Method dependency not shipped | https://docs.pipelex.com/latest/errors/method-dependency-not-shipped-error/MethodEntryError | Method entry | https://docs.pipelex.com/latest/errors/method-entry-error/MethodFetchDisabledError | Method fetch disabled | https://docs.pipelex.com/latest/errors/method-fetch-disabled-error/MethodFetchError | Method fetch | https://docs.pipelex.com/latest/errors/method-fetch-error/diff --git a/tests/integration/pipelex/pipeline/test_data.py b/tests/integration/pipelex/pipeline/test_data.pyindex 56b44f6e0..44c7177e3 100644--- a/tests/integration/pipelex/pipeline/test_data.py+++ b/tests/integration/pipelex/pipeline/test_data.py@@ -567,3 +567,65 @@ output = "Text"model = "$writing-creative"prompt = "Pitch this company in one sentence: $description""""+++class ResolveBundleTestData:+ """Bundles the host-facing resolve cores are pinned with."""++ MAIN_MTHDS: ClassVar[str] = """\+domain = "pipeline"+description = "Pipeline domain"++[concept.Report]+description = "A report"+structure.score = { description = "the score", type = "concept", concept_ref = "Score" }+structure.label = { description = "a label", type = "concept", concept_ref = "Text" }++[pipe.run_pipeline]+type = "PipeSequence"+description = "Run the pipeline"+inputs = { doc = "Text" }+output = "Score"+steps = [{ pipe = "compute_score", result = "score" }]+"""++ STEPS_MTHDS: ClassVar[str] = """\+domain = "pipeline"++[concept.Score]+description = "A score"+structure = { value = { description = "the value", type = "number" } }++[pipe.compute_score]+type = "PipeLLM"+description = "Compute a score"+inputs = { doc = "Text" }+output = "Score"+model = "$quick-reasoning"+prompt = "Compute a score from $doc"+"""++ # The pipe references a concept that no bundle declares — a load-stage library error, translated+ # by the shared cascade into a ValidateBundleError carrying structured validation errors.+ INVALID_MTHDS: ClassVar[str] = """\+domain = "broken"+description = "Broken domain"++[pipe.bad_pipe]+type = "PipeLLM"+description = "References an undeclared concept"+inputs = { doc = "NoSuchConcept" }+output = "Text"+prompt = "Do something with $doc"+"""++ ADDRESS_DEPENDENCY_MTHDS: ClassVar[str] = """\+domain = "host"++[pipe.run_host]+type = "PipeSequence"+description = "Run an installed dependency"+inputs = { doc = "Text" }+output = "Text"+steps = [{ pipe = "github.com/org/pkg->remote.process", result = "result" }]+"""diff --git a/tests/integration/pipelex/pipeline/test_resolve_bundle.py b/tests/integration/pipelex/pipeline/test_resolve_bundle.pyindex ceff01687..5bf1d710c 100644--- a/tests/integration/pipelex/pipeline/test_resolve_bundle.py+++ b/tests/integration/pipelex/pipeline/test_resolve_bundle.py@@ -13,68 +13,16 @@ from mthds.package.manifest.schema import MTHDS_STANDARD_VERSIONfrom pytest_mock import MockerFixturefrom pipelex.base_exceptions import PipelexUnexpectedError-from pipelex.hosted.hosted_inputs import ADDRESS_DEPENDENCY_REFUSAL_MARKERfrom pipelex.interpreter_hub import clear_current_library, get_current_library_id_or_none, get_library_manager+from pipelex.methods.address_dependencies import ADDRESS_DEPENDENCY_REFUSAL_MARKERfrom pipelex.pipeline.exceptions import ValidateBundleErrorfrom pipelex.pipeline.resolve_bundle import resolve_crate_from_contents+from tests.integration.pipelex.pipeline.test_data import ResolveBundleTestData-MAIN_MTHDS = """\-domain = "pipeline"-description = "Pipeline domain"--[concept.Report]-description = "A report"-structure.score = { description = "the score", type = "concept", concept_ref = "Score" }-structure.label = { description = "a label", type = "concept", concept_ref = "Text" }--[pipe.run_pipeline]-type = "PipeSequence"-description = "Run the pipeline"-inputs = { doc = "Text" }-output = "Score"-steps = [{ pipe = "compute_score", result = "score" }]-"""--STEPS_MTHDS = """\-domain = "pipeline"--[concept.Score]-description = "A score"-structure = { value = { description = "the value", type = "number" } }--[pipe.compute_score]-type = "PipeLLM"-description = "Compute a score"-inputs = { doc = "Text" }-output = "Score"-model = "$quick-reasoning"-prompt = "Compute a score from $doc"-"""--# The pipe references a concept that no bundle declares — a load-stage library error, translated-# by the shared cascade into a ValidateBundleError carrying structured validation errors.-INVALID_MTHDS = """\-domain = "broken"-description = "Broken domain"--[pipe.bad_pipe]-type = "PipeLLM"-description = "References an undeclared concept"-inputs = { doc = "NoSuchConcept" }-output = "Text"-prompt = "Do something with $doc"-"""--ADDRESS_DEPENDENCY_MTHDS = """\-domain = "host"--[pipe.run_host]-type = "PipeSequence"-description = "Run an installed dependency"-inputs = { doc = "Text" }-output = "Text"-steps = [{ pipe = "github.com/org/pkg->remote.process", result = "result" }]-"""+MAIN_MTHDS = ResolveBundleTestData.MAIN_MTHDS+STEPS_MTHDS = ResolveBundleTestData.STEPS_MTHDS+INVALID_MTHDS = ResolveBundleTestData.INVALID_MTHDS+ADDRESS_DEPENDENCY_MTHDS = ResolveBundleTestData.ADDRESS_DEPENDENCY_MTHDSclass TestResolveCrateFromContents:@@ -148,6 +96,6 @@ class TestResolveCrateFromContents:resolve_crate_from_contents(mthds_contents=[ADDRESS_DEPENDENCY_MTHDS], mthds_sources=["host.mthds"])discovery.assert_not_called()- # The items `/v1/pipe-io` sends in its invalid verdict carry the marker a hosted run tells this refusal apart by.+ # The verdict's items carry the marker a hosted run tells this refusal apart by, on a server answering it.validation_errors = exc_info.value.to_error_report().validation_errors or []assert any(ADDRESS_DEPENDENCY_REFUSAL_MARKER in item.message for item in validation_errors)diff --git a/tests/integration/pipelex/pipeline/test_resolve_crate_from_method_dir.py b/tests/integration/pipelex/pipeline/test_resolve_crate_from_method_dir.pynew file mode 100644index 000000000..1b1554421--- /dev/null+++ b/tests/integration/pipelex/pipeline/test_resolve_crate_from_method_dir.py@@ -0,0 +1,138 @@+"""Integration: ``resolve_crate_from_method_dir``, the host-facing resolve core over a method's written tree.++Pins the contract the HTTP crate routes build on: a tree a host wrote out and holds resolves to the normalized crate+with each bundle named by its tree path, an invalid tree, one holding no bundle or one calling a package nothing shipped+raises the one shared ``ValidateBundleError``, and the library lifecycle honors the loaded-on-success /+torn-down-on-failure contract.+"""++from collections.abc import Callable, Generator+from contextlib import contextmanager+from pathlib import Path++import pytest+from mthds.method_tree.digest import digest_of_contents+from mthds.package.manifest.schema import MTHDS_STANDARD_VERSION+from pytest_mock import MockerFixture++from pipelex.interpreter_hub import clear_current_library, get_current_library_id_or_none, get_library_manager+from pipelex.libraries.written_trees import WRITTEN_METHOD_TREES, write_method_tree+from pipelex.methods.address_dependencies import ADDRESS_DEPENDENCY_REFUSAL_MARKER+from pipelex.pipeline.exceptions import ValidateBundleError+from pipelex.pipeline.resolve_bundle import resolve_crate_from_method_dir+from pipelex.validation_error_types import PipeValidationErrorType+from tests.integration.pipelex.pipeline.test_data import ResolveBundleTestData+++@contextmanager+def _written_tree(*, destination: Path, files: dict[str, str]) -> Generator[Path, None, None]:+ """Write a tree out and hold it as written, as the API server does with the tree a request sends."""+ contents = {tree_path: content.encode("utf-8") for tree_path, content in files.items()}+ written = write_method_tree(contents=contents, destination=destination, expected_digest=digest_of_contents(contents))+ with WRITTEN_METHOD_TREES.holding(tree=written):+ yield destination+++class TestResolveCrateFromMethodDir:+ def test_a_written_tree_resolves_with_bundles_named_by_tree_path(self, load_empty_library: Callable[[], str], tmp_path: Path):+ """Every bundle of the tree joins the crate under its tree path, and the library is left loaded and current."""+ load_empty_library()+ library_manager = get_library_manager()+ files = {+ "main.mthds": ResolveBundleTestData.MAIN_MTHDS,+ "steps/scoring.mthds": ResolveBundleTestData.STEPS_MTHDS,+ "README.md": "# Not a bundle\n",+ }+ with _written_tree(destination=tmp_path / "method", files=files) as method_dir:+ crate = resolve_crate_from_method_dir(method_dir=method_dir)+ library_id = get_current_library_id_or_none()+ try:+ assert crate.fingerprint+ assert crate.mthds_version == MTHDS_STANDARD_VERSION+ assert "pipeline.run_pipeline" in crate.pipes+ assert "pipeline.compute_score" in crate.pipes+ assert "native.Text" in crate.concepts, "referenced natives must be materialized into the normalized crate"+ assert crate.source_map.get("pipeline.run_pipeline") == "main.mthds"+ assert crate.source_map.get("pipeline.compute_score") == "steps/scoring.mthds"+ assert library_id is not None+ finally:+ clear_current_library()+ if library_id is not None:+ library_manager.teardown(library_id=library_id)++ @pytest.mark.parametrize(+ ("files", "expects_items"),+ [+ pytest.param({"broken.mthds": ResolveBundleTestData.INVALID_MTHDS}, True, id="invalid-bundle"),+ pytest.param({"notes.txt": "no bundle here"}, False, id="no-bundle"),+ ],+ )+ def test_a_tree_with_no_crate_raises_the_shared_verdict_and_tears_down(+ self,+ load_empty_library: Callable[[], str],+ mocker: MockerFixture,+ tmp_path: Path,+ files: dict[str, str],+ expects_items: bool,+ ):+ """An invalid tree, or one holding no bundle, is the caller's verdict; the opened library is torn down."""+ load_empty_library()+ library_manager = get_library_manager()+ open_library_spy = mocker.spy(library_manager, "open_library")+ teardown_spy = mocker.spy(library_manager, "teardown")+ current_before = get_current_library_id_or_none()+ with _written_tree(destination=tmp_path / "method", files=files) as method_dir, pytest.raises(ValidateBundleError) as exc_info:+ resolve_crate_from_method_dir(method_dir=method_dir)++ if expects_items:+ validation_errors = exc_info.value.to_error_report().validation_errors or []+ assert validation_errors, "the shared builder must produce structured validation-error items"+ assert validation_errors[0].source == "broken.mthds"+ # Nothing of the verdict names the directory the host wrote the tree to, the report's own message included.+ report_text = exc_info.value.to_error_report().model_dump_json()+ assert str(tmp_path) not in report_text+ assert str(tmp_path.resolve()) not in report_text+ else:+ assert "holds no .mthds bundle" in exc_info.value.message+ opened_library_id, _ = open_library_spy.spy_return+ assert teardown_spy.call_args_list[-1].kwargs["library_id"] == opened_library_id+ assert get_current_library_id_or_none() == current_before++ def test_a_merge_error_names_the_bundles_by_their_tree_paths(self, load_empty_library: Callable[[], str], tmp_path: Path):+ """An error raised once the bundles parsed, such as one concept declared twice, names no directory of the host."""+ load_empty_library()+ files = {"first.mthds": ResolveBundleTestData.MAIN_MTHDS, "second.mthds": ResolveBundleTestData.MAIN_MTHDS}+ with _written_tree(destination=tmp_path / "method", files=files) as method_dir, pytest.raises(ValidateBundleError) as exc_info:+ resolve_crate_from_method_dir(method_dir=method_dir)++ report = exc_info.value.to_error_report()+ assert "['first.mthds', 'second.mthds']" in report.message+ report_text = report.model_dump_json()+ assert str(tmp_path) not in report_text+ assert str(tmp_path.resolve()) not in report_text++ def test_an_address_dependency_nothing_shipped_is_refused_without_a_lookup_or_a_fetch(+ self,+ load_empty_library: Callable[[], str],+ mocker: MockerFixture,+ tmp_path: Path,+ ):+ """A static resolution reads no method installed on the host and fetches nothing: it refuses the call."""+ load_empty_library()+ discovery = mocker.patch(+ "pipelex.libraries.library_manager.resolve_address_based_method",+ side_effect=AssertionError("a static resolution must not look up installed methods or fetch by address"),+ )+ files = {"host.mthds": ResolveBundleTestData.ADDRESS_DEPENDENCY_MTHDS}+ with _written_tree(destination=tmp_path / "method", files=files) as method_dir, pytest.raises(ValidateBundleError) as exc_info:+ resolve_crate_from_method_dir(method_dir=method_dir)++ discovery.assert_not_called()+ # The unresolved dependency `/v1/validate` reports, located on the pipe calling it, in the words of the contents+ # core's refusal, whose marker a hosted run tells it apart by.+ validation_errors = exc_info.value.to_error_report().validation_errors or []+ assert len(validation_errors) == 1, validation_errors+ item = validation_errors[0]+ assert item.error_type == PipeValidationErrorType.UNRESOLVED_PACKAGE_DEPENDENCY+ assert item.source == "host.mthds"+ assert f"{ADDRESS_DEPENDENCY_REFUSAL_MARKER} 'github.com/org/pkg'" in item.message